Split from #7557 (the release keeper), from the CI research comment there: a roll should not pause queue work.
Now. Since #6732 the old forge keeps its gates, checks and crew runs going through its drain (FORGE_CHECKS_DRAIN_S, 660 s; the pod's grace is 720 s). Two writers stall each other on SQLite, so the new forge serves pages but holds its own queue work until the old one lets go (src/runs/peer.tswaitForPeer, up to 12 minutes).
Measured. #7557 records that hold per release as write_pause_ms on keeper_release. It shows on the /releases keeper card, the Maintenance board ("Write pause per release") and MCP release_keeper. The keeper's cadence waits at least 10 × the last pause before the next cut.
Done looks like. On SIGTERM the old forge stops writing within seconds. It ends its gates' and checks' steps and hands them back queued, to resume on the new forge, and it checkpoints crew runs instead of letting them run on. The new forge's waitForPeer then returns within 30 s. A test boots two forges on one data directory and proves the second starts its queue work under 30 s while the first still has a gate in flight. The Maintenance board's write pause stays under 30 s across three releases.
Split from #7557 (the release keeper), from the CI research comment there: a roll should not pause queue work.
Now. Since #6732 the old forge keeps its gates, checks and crew runs going through its drain (
FORGE_CHECKS_DRAIN_S, 660 s; the pod's grace is 720 s). Two writers stall each other on SQLite, so the new forge serves pages but holds its own queue work until the old one lets go (src/runs/peer.tswaitForPeer, up to 12 minutes).Measured. #7557 records that hold per release as
write_pause_msonkeeper_release. It shows on the /releases keeper card, the Maintenance board ("Write pause per release") and MCPrelease_keeper. The keeper's cadence waits at least 10 × the last pause before the next cut.Done looks like. On SIGTERM the old forge stops writing within seconds. It ends its gates' and checks' steps and hands them back queued, to resume on the new forge, and it checkpoints crew runs instead of letting them run on. The new forge's
waitForPeerthen returns within 30 s. A test boots two forges on one data directory and proves the second starts its queue work under 30 s while the first still has a gate in flight. The Maintenance board's write pause stays under 30 s across three releases.Files: src/main.ts (
drain), src/checks/service.ts (stop({ drain })), src/checks/queue.ts, src/runs/peer.ts, charts/app/templates/deployment.yaml.