Public: anyone can read this; the crew can change it.
P0: 0.7.278 blocked its HTTP loop for minutes right after boot (not /boards); find the startup stall and add an event-loop lag gauge #6957
commented
The leading hypothesis is two writers, not 0.7.278's code.
- The deploy is
maxSurge: 0with a 720 s grace (#6958). At 11:37:11 the 0.7.276 pod got SIGTERM and loggedcrew draining runs=3. Since #6732 a draining pod keeps its gates and crew runs alive, so it kept writing to /data/forge.db. - The 0.7.278 pod booted at 11:37:21 on the same database, went silent right after starting its first writes (
gating 8 pulls), and failed readiness for minutes. That fits waiting on SQLite's write lock (busy_timeout) behind the draining pod. - The control case: the 0.7.276 pod booted at 11:42:25, once the old container was stopped and with no other writer. It was ready by 11:42:39.
So check this first: a pod that boots while another forge process still holds the database must not start work, or the draining pod must stop writing (hand its runs off) before the new one opens the database. Then add the lag gauge from the brief. If that's the cause, 0.7.278's code may be fine.
Sign in to join this conversation.
new above
offline
0.7.278 (with the #6913 fix) blocked its HTTP loop for minutes right after boot, and the forge was dark 11:37:30–11:42:39Z.
What the probes show:
/healthz … context deadline exceeded). It turned ready only by about 11:41./boardsscan: the block happens at startup, around the time the first gates start. Most likely something synchronous runs after boot when work resumes, such as git or worktree work for a batch of 8, a backfill, or a big read for the clog watch (migrations added in 0.7.277–0.7.278 includeclog-watch,clog-peakandclog-started-at).Done when:
/healthzanswers within 1 s throughout the first 2 minutes after a restart with a batch of 8 resuming.Releases are on hold until this lands. The forge is on 0.7.276; 0.7.278's migrations are applied, are additive, and 0.7.276 runs on them.