P0: 0.7.278 blocked its HTTP loop for minutes right after boot (not /boards); find the startup stall and add an event-loop lag gauge #6961

The 0.7.278 stall reproduces locally as SQLite lock contention between two forges. bun:sqlite waits for the write lock on the same thread that serves HTTP, for up to busy_timeout (15 s) per statement. So a forge booting beside one still draining on the same database freezes behind the old one's writes. This fits trevato's two-writers hypothesis. A minimal boot beside a process that holds the write lock reproduces the readiness timeouts, and this branch fixes it in two parts plus a gauge.

The new forge holds its work (src/runs/peer.ts, wired in src/main.ts). After the listener and apps are up, the boot checks for running runs held by another process whose lease was renewed in the last 30 s. While any exist, it waits before starting checks, the merge queue, the crew and the ticks. It stops waiting when they are gone, or after 720 s (the pod's grace period). Old forges already renew run leases, so this works with 0.7.276 as the draining pod. A signal during the wait stops the apps and exits cleanly. heldRuns in src/runs/lease.ts tells this process's own leases apart from the other forge's.

Telemetry never waits on the lock (withoutWaiting in src/kernel/db.ts). Every /healthz probe is counted, and those counts are flushed as database writes. Behind the other forge's lock, each flush waited up to 15 s and held up the next probe. Holding the forge's work alone does not fix that. Route timings (src/core/route_timing.ts), usage counts (src/core/usage_count.ts) and machine samples (src/boards/ring.ts) now try the lock once. If it is taken they keep their rows and retry on the next flush, which they already did when a write failed.

The next stall names itself:

• src/kernel/loop_lag.ts logs event loop stalled lateMs=… whenever the loop wakes more than a second late. It is on from the first line of the boot, and while draining.
• src/kernel/slow_sql.ts logs any statement on the forge's database that takes 200 ms or more, with the start of its SQL and its callers, at most ten lines a minute. This is how the boot test found the usage-count flush blocking the loop.

Tests:

• test/boot-beside-peer.test.ts boots src/main.ts beside a process that holds the write lock for 2.5 s out of every 2.65 s and keeps 8 running check leases fresh. Every /healthz probe must answer within 1 s. The forge must log the wait, and start its work once that process finishes.

- It passes with both fixes. - With both reverted, 7 probes time out at 1 s. - With only the hold removed, the probes stay fast, but the forge starts its work beside the other process.

• test/loop-stall.test.ts covers the stall warning, the slow-statement log (including a lock held by a real second process), lease counting and the wait's three outcomes.
• The type check and scripts/gate.ts are clean.
• test/ssh-port-taken.test.ts times out in this sandbox on main too; this change does not cause it.

Not done:

• I could not boot on a copy of the production database: the sandbox blocks /data/forge.db. The two log lines above are the first thing to look for on the next roll.
• The test's probe window is 20 s, not the brief's 2 minutes, to fit the check's 60 s per-test timeout.
• busy_timeout stays at 15 s for request writes. A person's write during the overlap can still wait on the old forge's lock, but probes no longer do.
• docs/ops/ROLL.md gets a new section on all this; its opening line still says maxSurge: 1, which belongs to #6958.

To try it: bun test test/boot-beside-peer.test.ts test/loop-stall.test.ts.

🤖 Generated with Claude Code

Closes #6957

The 0.7.278 stall reproduces locally as SQLite lock contention between two forges. bun:sqlite waits for the write lock on the same thread that serves HTTP, for up to busy_timeout (15 s) per statement. So a forge booting beside one still draining on the same database freezes behind the old one's writes. This fits trevato's two-writers hypothesis. A minimal boot beside a process that holds the write lock reproduces the readiness timeouts, and this branch fixes it in two parts plus a gauge.

The new forge holds its work (src/runs/peer.ts, wired in src/main.ts). After the listener and apps are up, the boot checks for running runs held by another process whose lease was renewed in the last 30 s. While any exist, it waits before starting checks, the merge queue, the crew and the ticks. It stops waiting when they are gone, or after 720 s (the pod's grace period). Old forges already renew run leases, so this works with 0.7.276 as the draining pod. A signal during the wait stops the apps and exits cleanly. heldRuns in src/runs/lease.ts tells this process's own leases apart from the other forge's.

Telemetry never waits on the lock (withoutWaiting in src/kernel/db.ts). Every /healthz probe is counted, and those counts are flushed as database writes. Behind the other forge's lock, each flush waited up to 15 s and held up the next probe. Holding the forge's work alone does not fix that. Route timings (src/core/route_timing.ts), usage counts (src/core/usage_count.ts) and machine samples (src/boards/ring.ts) now try the lock once. If it is taken they keep their rows and retry on the next flush, which they already did when a write failed.

The next stall names itself:

• src/kernel/loop_lag.ts logs event loop stalled lateMs=… whenever the loop wakes more than a second late. It is on from the first line of the boot, and while draining.
• src/kernel/slow_sql.ts logs any statement on the forge's database that takes 200 ms or more, with the start of its SQL and its callers, at most ten lines a minute. This is how the boot test found the usage-count flush blocking the loop.

Tests:

• test/boot-beside-peer.test.ts boots src/main.ts beside a process that holds the write lock for 2.5 s out of every 2.65 s and keeps 8 running check leases fresh. Every /healthz probe must answer within 1 s. The forge must log the wait, and start its work once that process finishes.

- It passes with both fixes. - With both reverted, 7 probes time out at 1 s. - With only the hold removed, the probes stay fast, but the forge starts its work beside the other process.

• test/loop-stall.test.ts covers the stall warning, the slow-statement log (including a lock held by a real second process), lease counting and the wait's three outcomes.
• The type check and scripts/gate.ts are clean.
• test/ssh-port-taken.test.ts times out in this sandbox on main too; this change does not cause it.

Not done:

• I could not boot on a copy of the production database: the sandbox blocks /data/forge.db. The two log lines above are the first thing to look for on the next roll.
• The test's probe window is 20 s, not the brief's 2 minutes, to fit the check's 60 s per-test timeout.
• busy_timeout stays at 15 s for request writes. A person's write during the overlap can still wait on the old forge's lock, but probes no longer do.
• docs/ops/ROLL.md gets a new section on all this; its opening line still says maxSurge: 1, which belongs to #6958.

To try it: bun test test/boot-beside-peer.test.ts test/loop-stall.test.ts.

🤖 Generated with Claude Code