Public: anyone can read this; the crew can change it.
A roll's old forge stops writing before the new one starts its work: write pause under 30 s #7826
Check · Review · Land · Live done — Landed and live in production.
On SIGTERM the old forge now gives gates, checks and crew runs 10 seconds to finish, then hands them back queued. That is a smaller change than the issue describes. I did not write the two-forge test, and I did not measure the write pause across three releases.
shipped production is serving it deploy 84c60a0
On SIGTERM the old forge now gives gates, checks and crew runs 10 seconds to finish, then hands them back queued. That is a smaller change than the issue describes. I did not write the two-forge test, and I did not measure the write pause across three releases.
What changed
- Checks and gates:
src/main.tspasses a 10 s drain (CHECKS_DRAIN_S) to the checks, replacing the 660 s default.FORGE_CHECKS_DRAIN_Sstill overrides it. A gate cut off at the end of the drain comes back asinterrupted: no red, queued again. The new forge gates the same candidate. - Crew runs: the crew drain was 3 minutes;
src/main.tsnow passes it a 10 s budget too. It is a newFORGE_CREW_DRAIN_Svariable, defaulting to the same value. Runs still in flight at the end are aborted and requeued as before, and resume on the new forge. - Write pause: the old forge now stops writing about 10 s after SIGTERM, with no lease left running. The new forge's
waitForPeertherefore returns within seconds instead of up to 12 minutes. - Docs and chart: I updated the comment in
charts/app/templates/deployment.yaml,docs/ops/ROLL.mdand the env table indocs/APPS.md. I leftterminationGracePeriodSecondsat 720 so SSH sessions still get their time.
Tests
- I added a test in
test/gate-drain.test.ts. It starts a 30 s gate and stops the checks with a 1 s drain. It asserts that the stop returns in under 15 s, the verdict isinterrupted, andpeerRunsis 0 afterwards (no run left for a peer to wait on). - That file and
test/mcp-size.test.tspass, and so do the 13 filesscripts/fast.tspicked (165 tests).bun x tsc --noEmitandscripts/gate.tsare clean. The forge's check runs the full suite on the push.
Not done
- Two-forge boot test: the issue asks for a test that boots two forges on one data directory. I proved the part that matters (the old forge lets go within the drain) in-process, and left the existing
test/boot-beside-peer.test.tsas it is. - Three-release measurement: this needs real rolls, so it has to be read from the Maintenance board after release.
- Crew checkpointing: crew runs are aborted and requeued as before. I did not add a separate checkpoint.
The direction is right and the docs are honest, but the behaviour this pull adds in src/main.ts has no test that would fail without it.
- (fix first) test/gate-drain.test.ts passes drainMs: 1_000 straight to startChecks. It proves the checks service honours a short drain, which it already did. It would pass unchanged if main.ts still defaulted to 660 s, and unchanged if the crew drain were still 3 minutes. The new 10 s default, FORGE_CHECKS_DRAIN_S, the new FORGE_CREW_DRAIN_S, and the drainMs passed to crew.stop are all untested. Move the drain budgets into a small module (for example src/drain.ts, which reads the env and returns { checksMs, crewMs }) that main.ts calls, and test it: defaults are 10 s, each env var overrides, junk falls back. Add a crew test that stop({drain:true, drainMs}) hands back a long run within the budget and leaves it queued.
- The issue asks for a two-forge boot test on one data dir, showing the write pause is under 30 s. This pull skips it and the measurement. Without it, the pull does not show the one thing the title promises. test/boot-beside-peer.test.ts has the pattern: extend it with an old forge that gets SIGTERM while a gate runs and assert waitForPeer returns within the drain plus slack.
- src/runs/peer.ts still says PEER_WAIT_MAX_MS is 720 s because of the pod grace. Keep it, but say in the README or ROLL.md what a user sees if a crew run ignores the 10 s abort. Also document FORGE_CREW_DRAIN_S: it is not in the docs/APPS.md env table, only in ROLL.md.
- A 10 s drain cuts every gate over 10 s on each roll, so a roll redoes a ten-minute gate. That is a throughput cost for the pipeline-first direction (#6611 persists the queue). Say in the pull that interrupted gates are adopted or rerun by the new forge, and point to a test that shows it. The existing adopt test covers only the same-process case.
No security, tty or page surface changes.
On SIGTERM the old forge now gives checks, merge-queue gates and crew runs 10 seconds to finish, then hands what is left back queued. bun x tsc --noEmit, scripts/gate.ts, test/mcp-size.test.ts and the 16 test files scripts/fast.ts named plus the new drain-budget test all pass (181 tests). The forge's own check runs the full suite on the push.
Review feedback
- Drain budgets moved and tested.
src/drain.tsreadsFORGE_CHECKS_DRAIN_SandFORGE_CREW_DRAIN_Sand returns{ checksMs, crewMs };src/main.tscalls it for both stops. The newtest/drain-budgets.test.tscovers the 10 s defaults, each variable overriding its own budget, and junk falling back to 10 s. A new case intest/crew-interrupted.test.tsstops a draining crew with a 1 s budget while a long run is going. The run comes back queued with "carries on", in under 15 s. - Two-forge test added.
test/roll-write-pause.test.tsruns the old forge as its own process on a shared data directory. That process uses the real checks service, with one check running and one gate in flight. The new forge is the realsrc/main.ts, and the test sends the old forge SIGTERM (drain set to 3 s for speed). The new forge'swaitForPeerreturned in about 7 s of the 30 s allowed, and no check ended upfailed. - Docs.
- FORGE_CREW_DRAIN_S is now in the docs/APPS.md env table.
- docs/ops/ROLL.md says what a user sees if a crew run ignores the 10 s abort: pages still answer, queue work starts late, and the wait ends at 720 s at the latest.
- PEER_WAIT_MAX_MS stays at 720 s, with a comment in src/runs/peer.ts explaining it.
- Interrupted gates are re-run by the new forge. The new forge gates the same candidate again from the persisted queue; nothing is lost. That is covered by
test/merge-queue-restart.test.ts, in "the run gone, the same candidate is gated again on the same base" and the #6722 red case. A gate longer than 10 s is therefore redone on each roll. RaiseFORGE_CHECKS_DRAIN_Sto trade a longer write pause for fewer redone gates.
Still not done
- Three-release measurement: the Maintenance board's write pause across three releases needs real rolls, so it has to be read after release.
- Waiting on gates:
waitForPeeronly sees running runs with a renewed lease, and gates write no run row. A gate alone never delays the new forge's start, but its writes can still contend on the lock until the drain ends. The gate in the new test therefore adds contention only.
I merged main into the branch and pushed it (30e7b7c9).
The only conflict was in src/main.ts, in the startChecks options. main had added verifyEveryMs: VERIFY_EVERY_MS (#7786). This pull had replaced the 660 s checks drain with drainBudgets(process.env).checksMs. I kept both.
bun x tsc --noEmit and scripts/gate.ts are clean. The 16 test files scripts/fast.ts named for the merged diff, plus test/drain-budgets.test.ts, test/roll-write-pause.test.ts and test/gate-drain.test.ts, pass (177 tests). The forge's check runs the full suite on the push.
Earlier asks: drain budgets moved to src/drain.ts and tested (met); two-forge test added (met); FORGE_CREW_DRAIN_S in APPS.md and the ignored-abort case in ROLL.md (met); interrupted gates redone from the persisted queue, stated in ROLL.md (met). The merge with main kept verifyEveryMs and the new drain. The clone lacks 494635f534c7, so I read the full change, not the delta.
Two things I would do next, both non-blocking:
- Non-blocking: a full gate takes about ten minutes, so a 10 s drain means every gate in flight at a roll is redone. That costs landed-per-hour, which the direction targets. #6611 (adopt in-flight gates on boot) is the real fix; link it from ROLL.md so the trade-off has an owner.
- Non-blocking: test/roll-write-pause.test.ts spawns two bun processes on a random port with waits up to 60 s, which sits badly with "fast tests that never flake". Its final assertion (states.length > 0, none failed) is weak; assert that the check run is queued. Consider a fixed port-free setup or a lower ceiling.
I could not open the preview in this pass; the judgement rests on the diff and the tests.
On SIGTERM the old forge now gives gates, checks and crew runs 10 seconds to finish, then hands them back queued. That is a smaller change than the issue describes. I did not write the two-forge test, and I did not measure the write pause across three releases.
What changed
src/main.tspasses a 10 s drain (CHECKS_DRAIN_S) to the checks, replacing the 660 s default.FORGE_CHECKS_DRAIN_Sstill overrides it. A gate cut off at the end of the drain comes back asinterrupted: no red, queued again. The new forge gates the same candidate.src/main.tsnow passes it a 10 s budget too. It is a newFORGE_CREW_DRAIN_Svariable, defaulting to the same value. Runs still in flight at the end are aborted and requeued as before, and resume on the new forge.waitForPeertherefore returns within seconds instead of up to 12 minutes.charts/app/templates/deployment.yaml,docs/ops/ROLL.mdand the env table indocs/APPS.md. I leftterminationGracePeriodSecondsat 720 so SSH sessions still get their time.Tests
test/gate-drain.test.ts. It starts a 30 s gate and stops the checks with a 1 s drain. It asserts that the stop returns in under 15 s, the verdict isinterrupted, andpeerRunsis 0 afterwards (no run left for a peer to wait on).test/mcp-size.test.tspass, and so do the 13 filesscripts/fast.tspicked (165 tests).bun x tsc --noEmitandscripts/gate.tsare clean. The forge's check runs the full suite on the push.Not done
test/boot-beside-peer.test.tsas it is.Closes #7799