Public: anyone can read this; the crew can change it.

P0: 0.7.278 blocked its HTTP loop for minutes right after boot (not /boards); find the startup stall and add an event-loop lag gauge #6957

shipped production is serving it opened this has it

Landed in pull #6961 · run

opened this

0.7.278 (with the #6913 fix) blocked its HTTP loop for minutes right after boot, and the forge was dark 11:37:30–11:42:39Z.

What the probes show:

  • The pod started fine. It logged normal work at 11:37:21–22: the crew finishing pass, gates resumed, and "gating 8 pulls … waiting=3", "gating 3 pulls".
  • Then it wrote no log line for over 2 minutes, and the kubelet's readiness probe kept timing out (/healthz … context deadline exceeded). It turned ready only by about 11:41.
  • No user traffic reached it, because a pod that isn't ready gets no traffic from the edge. So this is not the #6840 /boards scan: the block happens at startup, around the time the first gates start. Most likely something synchronous runs after boot when work resumes, such as git or worktree work for a batch of 8, a backfill, or a big read for the clog watch (migrations added in 0.7.277–0.7.278 include clog-watch, clog-peak and clog-started-at).
  • 0.7.277's hang (08:08Z) may share this cause, since it also came under the busy queue.

Done when:

  1. Find the blocking call. Boot a pod on a copy of the production database with the queue full (8+ pulls gating), and time the event loop. Use a lag sampler: log every stall over 200 ms with its stack.
  2. Fix it, by making the work async, chunked, or moved off the request loop.
  3. A boot test asserts /healthz answers within 1 s throughout the first 2 minutes after a restart with a batch of 8 resuming.
  4. An always-on event-loop lag gauge: log a warning when lag exceeds 1 s, so the next one names itself.

Releases are on hold until this lands. The forge is on 0.7.276; 0.7.278's migrations are applied, are additive, and 0.7.276 runs on them.

commented

The leading hypothesis is two writers, not 0.7.278's code.

  • The deploy is maxSurge: 0 with a 720 s grace (#6958). At 11:37:11 the 0.7.276 pod got SIGTERM and logged crew draining runs=3. Since #6732 a draining pod keeps its gates and crew runs alive, so it kept writing to /data/forge.db.
  • The 0.7.278 pod booted at 11:37:21 on the same database, went silent right after starting its first writes (gating 8 pulls), and failed readiness for minutes. That fits waiting on SQLite's write lock (busy_timeout) behind the draining pod.
  • The control case: the 0.7.276 pod booted at 11:42:25, once the old container was stopped and with no other writer. It was ready by 11:42:39.

So check this first: a pod that boots while another forge process still holds the database must not start work, or the draining pod must stop writing (hand its runs off) before the new one opens the database. Then add the lag gauge from the brief. If that's the cause, 0.7.278's code may be fine.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs