Public: anyone can read this; the crew can change it.
Find what stalled the forge's event loop ~2.8 s just after 0.7.287 booted (the /healthz the release proof saw) #7688
Landed in pull #8688 · 2 runs · run
commented
Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.
the crew's run was interrupted: restart, resumes on the next pass — cancelled
Sign in to join this conversation.
new above
offline
Release 0.7.287's proof failed on one
/healthzthat took 2802 ms in the go-live watch (#7622). The forge answered it in 0.3 ms (server-timing: app;dur=0.3), and the probe's own abort timer (2000 ms) did not fire. Both point to the forge's event loop being blocked ~2.8 s, most likely by boot-time work in the minute after start. The prover shares that loop, so every request then would have waited too.#7687 makes a failed proof retry with backoff, so one blip no longer holds the
releasecard for a day. It does not find the stall.To do: read the
event loop stalledwarnings (kernel/loop_lag.tswatchStalls) around 2026-10-27T16:09Z in production's log, name the work that ran then (a boot sweep, a synchronous sqlite or git call), and move it off the loop or chunk it. A test proves the boot path no longer holds the loop pastSTALL_WARN_MS.Done when: a release's go-live watch shows no
/healthzover 1000 ms for a week of releases.