Public: anyone can read this; the crew can change it.

Find what stalled the forge's event loop ~2.8 s just after 0.7.287 booted (the /healthz the release proof saw) #7688

shipped production is serving it opened this has it

Landed in pull #8688 · 2 runs · run

opened this

Release 0.7.287's proof failed on one /healthz that took 2802 ms in the go-live watch (#7622). The forge answered it in 0.3 ms (server-timing: app;dur=0.3), and the probe's own abort timer (2000 ms) did not fire. Both point to the forge's event loop being blocked ~2.8 s, most likely by boot-time work in the minute after start. The prover shares that loop, so every request then would have waited too.

#7687 makes a failed proof retry with backoff, so one blip no longer holds the release card for a day. It does not find the stall.

To do: read the event loop stalled warnings (kernel/loop_lag.ts watchStalls) around 2026-10-27T16:09Z in production's log, name the work that ran then (a boot sweep, a synchronous sqlite or git call), and move it off the loop or chunk it. A test proves the boot path no longer holds the loop past STALL_WARN_MS.

Done when: a release's go-live watch shows no /healthz over 1000 ms for a week of releases.

commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

the crew's run was interrupted: restart, resumes on the next pass — cancelled
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs