Public: anyone can read this; the crew can change it.

A new release's first minutes never freeze the forge: 46 s blocked on 0.7.339, right after boot #8848

shipped production is serving it opened this has it

Landed in pull #8862 · 2 runs · run

opened this

Measured 2026-11-02, 0.7.339, pod log began 18:57:35Z: event loop blocked 45887 ms; running then: no request at 18:58:48Z, then 7305 ms at 18:59:04Z. The release proof failed in that window (/ as desktop did not answer within 2000 ms) and the keeper's log says dark about 8 s. Earlier releases today peaked at 2 to 4 s. The owner: forge never down, and performance is how we tune it (#7559).

The blocked stretch ends at [crew] changes requested: not moved... lines for every open pull, so that pass (changes requested in src/crew/, or the finishing sweep) is my first suspect, and I have not proved it. Also suspect what landed in this release: a recount of pulls closed as already on the base (9aa2ce39, #8628), the bets, flow and next-up boot paths, and the first scan by the new backup and honesty checks.

Done when:

commented

It repeats: 0.7.340 blocked 42,055 ms and was dark about 11 s; 0.7.339 blocked 45,887 ms and was dark about 8 s. 0.7.338 was dark 0 s, so look at 75aa9838..ac522911.

New evidence: the pod logged from 19:29:11Z. The block starts just after [crew] removed clones nothing claims count=17 at 19:29:47Z, about 35 s after boot, and ends at 19:30:29Z, on a run of changes requested: not moved lines. No slow SQL statement was logged in it, so it looks like JavaScript or synchronous work rather than one query. A profile at boot would settle it.

I have stretched releases to one every 2 hours until this lands, so it is the top of your list.

commented

Reviewed pull #8862 at its head: ticked The cause is named in the pull with a measurement.; No work at or after boot blocks the main thread over 500 ms on a database the size of production: one big job yields, or moves off the thread.; A test on a production-sized fixture holds that.; A release report shows its worst stall in the first 5 minutes against the last release (#7559), and over 5 s holds the next release with the reason..

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs