Never down: the forge stays up through every release and every settings change #1712

This pull adds piece 4 of #873 (the rehearsal) and fixes a gap the rehearsal found. Pieces 1–3 were already closed (#874, #875, #932, #1589).

The gap. A forge with nothing to finish exited 34 ms after SIGTERM, and its listener closed with it. During a roll, routers keep sending to the old pod for up to a probe period after it leaves rotation, and the chart has no preStop hook. So every request in that window was refused, and the ingress returned it as a 502. In a local roll with the old forge exiting at once, 1,428 of 14,744 requests were refused.

Fix (src/http/health.ts, src/main.ts). On SIGTERM, the old forge keeps its listener open for FORGE_DRAIN_LINGER_S (default 10 s, 0–120; the chart probes readiness every 5 s). The draining itself is unchanged: /readyz still answers 503 straight away. An invalid value stops the forge at boot, and SIGINT (Ctrl-C) does not wait. The chart's 300 s grace period already covers the linger.

Rehearsal (scripts/rehearse-roll.ts, with its plain parts in scripts/rehearsal.ts).

• It starts the release rolled from (--old) on a copy of --from, or on an empty data directory if --from is not given.
• It starts the release rolled to (--new) beside it on the same copy and sends requests to /, /healthz and /explore the whole time.
• A forge joins rotation when its /readyz says 200 and leaves when it doesn't, as the Service does. The old forge gets SIGTERM once the new one is in.
• Exit 0 means zero refused, 1 means a 5xx or no answer (or nothing rolled), 2 means a bad argument or a forge that would not start.

docs/ops/ROLL.md now has this as step 3, between the surge check and the roll. Exit 1 is treated as a bug in the release, not a reason to ask for a dark roll.

Tests.

• test/rehearsal.test.ts checks the arguments, the rotation and the verdict, and runs two real 8-second rolls. With linger 0 the rehearsal must find refusals; with linger 2 it must find none. The second test fails without the main.ts change.
• test/health.test.ts checks how FORGE_DRAIN_LINGER_S is parsed.
• The type check passes; the rehearsal, health and roll tests pass (73 tests).
• A full local run with a 2 s linger: zero refused out of 15,625 requests.

Not done here.

• Piece 1's headroom (a larger droplet or a second node) is the owner's decision. Until there is headroom, the roll check from #1589 makes a roll that cannot surge wait for the owner's yes.
• The rehearsal runs the forges with SSH, apps and checks off. It proves HTTP stays up, not that SSH sessions or app previews carry over.
• The chart is unchanged: no preStop hook, because the linger lives in the process.

To try it: bun scripts/rehearse-roll.ts --seconds 10 --probe-ms 1000. Add --linger 0 to see the dark beat it closes.

🤖 Generated with Claude Code

Closes #873

This pull adds piece 4 of #873 (the rehearsal) and fixes a gap the rehearsal found. Pieces 1–3 were already closed (#874, #875, #932, #1589).

The gap. A forge with nothing to finish exited 34 ms after SIGTERM, and its listener closed with it. During a roll, routers keep sending to the old pod for up to a probe period after it leaves rotation, and the chart has no preStop hook. So every request in that window was refused, and the ingress returned it as a 502. In a local roll with the old forge exiting at once, 1,428 of 14,744 requests were refused.

Fix (src/http/health.ts, src/main.ts). On SIGTERM, the old forge keeps its listener open for FORGE_DRAIN_LINGER_S (default 10 s, 0–120; the chart probes readiness every 5 s). The draining itself is unchanged: /readyz still answers 503 straight away. An invalid value stops the forge at boot, and SIGINT (Ctrl-C) does not wait. The chart's 300 s grace period already covers the linger.

Rehearsal (scripts/rehearse-roll.ts, with its plain parts in scripts/rehearsal.ts).

• It starts the release rolled from (--old) on a copy of --from, or on an empty data directory if --from is not given.
• It starts the release rolled to (--new) beside it on the same copy and sends requests to /, /healthz and /explore the whole time.
• A forge joins rotation when its /readyz says 200 and leaves when it doesn't, as the Service does. The old forge gets SIGTERM once the new one is in.
• Exit 0 means zero refused, 1 means a 5xx or no answer (or nothing rolled), 2 means a bad argument or a forge that would not start.

docs/ops/ROLL.md now has this as step 3, between the surge check and the roll. Exit 1 is treated as a bug in the release, not a reason to ask for a dark roll.

Tests.

• test/rehearsal.test.ts checks the arguments, the rotation and the verdict, and runs two real 8-second rolls. With linger 0 the rehearsal must find refusals; with linger 2 it must find none. The second test fails without the main.ts change.
• It is now listed in GATE in scripts/gate.ts; the last push failed because it was in neither list. bun scripts/flake-hunt.ts admit test/rehearsal.test.ts was green 10 out of 10, in random order under load (seeds 2078416204, 995710730, 2872422569, 3984896075, 782921257, 1681166843, 2141414588, 3577141130, 1621723739, 4240540866).
• test/health.test.ts checks how FORGE_DRAIN_LINGER_S is parsed.
• The type check passes, and so do test/check-list.test.ts and the rehearsal tests.

Not done here.

• Piece 1's headroom (a larger droplet or a second node) is the owner's decision. Until there is headroom, the roll check from #1589 makes a roll that cannot surge wait for the owner's yes.
• The rehearsal runs the forges with SSH, apps and checks off. It proves HTTP stays up, not that SSH sessions or app previews carry over.
• The chart is unchanged: no preStop hook, because the linger lives in the process.

To try it: bun scripts/rehearse-roll.ts --seconds 10 --probe-ms 1000. Add --linger 0 to see the dark beat it closes.

🤖 Generated with Claude Code

A SIGTERM'd forge now keeps answering for FORGE_DRAIN_LINGER_S seconds (default 10, whole seconds from 0 to 120) while routers learn it is leaving. A roll rehearsal proves that a release can swap one forge for another with zero refused requests.

• Drain: src/http/health.ts gains drainLingerMs and startDraining. On SIGTERM, health reports the forge as draining while it keeps serving requests for the linger window, then the forge exits. Boot refuses a malformed FORGE_DRAIN_LINGER_S with a single line, the same way it refuses other bad configuration.
• Rehearsal: test/roll.test.ts runs two forges against one copied data directory and keeps requests flowing through the whole handover. It joins the gate in scripts/gate.ts and was green 10 times out of 10, in random order and under load.
• Merge with main: main moved boot's config checks into configProblem in kernel/preflight.ts (#1674), which caused a conflict in src/main.ts. The linger check now runs straight after configProblem in src/main.ts. A stale comment about FORGE_TRUSTED_PROXIES is gone, because main covers that check now.

Chosen not to do: the linger check is not part of configProblem, so bun src/main.ts --preflight does not yet reject a bad FORGE_DRAIN_LINGER_S. Moving it there would mean the kernel importing from http/, which belongs in a separate change.

After the merge the type check is clean and test/health.test.ts, test/release-preflight.test.ts and test/roll.test.ts pass (47 tests, 0 failures). I did not run the whole suite here; the forge's check runs it on this push.

To try it: run bun test test/roll.test.ts. Or start the forge, send it SIGTERM, and watch the health endpoint report draining while pages keep answering for 10 s.

🤖 Generated with Claude Code

The full gate on land was red and named no test; the pull is open again.