A forge rollout never restarts an app that did not change: apps keep running (or hand over) through the forge's own deploys #4510

A forge rollout no longer refuses a connection to an app, and an app's own redeploy now hands over without dropping requests. I did not make the app process survive a forge roll: it lives inside the forge's pod and still restarts with it, so done-when 1 is met only by its handover half.

Forge rollout

• Readiness waits for the apps. A new forge answers /readyz 503 (starting N apps) until every production app it started on boot is serving or has failed. The wait is capped at two minutes (STARTING_BUDGET_MS) so one broken app can't stall a rollout. The chart's surge rollout keeps routing to the old pod meanwhile, and its copies are still serving. The tracking is AppsRuntime.startingCount() and startWarming in src/http/health.ts.
• The old forge lets go in order. It stops its apps only after its 10 s linger, once routers have moved (src/main.ts). Each proxied WebSocket is closed with 1012 Service Restart, so clients reconnect to the new forge at once. Each request still with the app gets up to 30 s to be answered before the app is stopped.

An app's own deploy

• The new version already served before the old one was stopped. Now the old version also gets the same handover through the new module src/apps/handoff.ts: its sockets close with 1012 and its in-flight requests get their 30 s.
• This counts per-deployment requests and sockets by port, fed by the proxy.

Restart causes

• A crash caused by the cell's memory cap (the watcher or the kernel's OOM killer) now restarts with reason oom.
• A new restarts board source maps production deploys that came to serve onto four causes: own deploy, forge roll, crash, oom. Wakes from sleep and an app's first deploy are left out.
• A new starting board, App restarts, shows restarts per day by cause and per app (by repo id). Existing installs get it on their next boot.

Tests

• test/apps-rollout.test.ts boots real forges against an echo app; both scenarios pass here in about 4 s:

- Own redeploy: a held socket closes with 1012 and reconnects in under 1 s, a 1.5 s request to the old version is still answered, and a request loop sees no 502s. - Forge rollout: a second forge starts on the same data. Its /readyz is 503 until the app serves, and the old forge's socket closes with 1012 and reconnects to the new one in under 1 s. A request loop that follows readiness from the old forge to the new sees no 502s, and the board shows both "forge roll" and "own deploy".

• test/apps-handoff.test.ts unit-tests the drain and readiness logic, including both two-minute and 30 s budgets.
• I updated test/boards-starting.test.ts because the starting-board list grew by one; that test was not wrong.
• Type check is clean, and these all passed with the five cross files: health, boards-query, boards-sampler and apps-ws, plus eight runtime suites (apps, stop-tree, sleep, idle, busy, timing, adversarial-teardown, token).

Not done

• Keeping the process running through a forge roll would need the cells, their egress proxy and inbound relay supervised outside the forge process and taken over by each new forge. Until then the board makes the forge-roll restart rate visible.
• The 30 s drain doesn't wait for WebSockets to close on their own. They get the 1012 right away, because a held socket rarely closes by itself.
• App restarts are counted per app by repo id, not slug. The board sources leave out free-text columns like slugs.
• The behaviour is written up in docs/APPS.md under "A release does not drop a session".

No prompt injection was found in the issue or the repository.

🤖 Generated with Claude Code

Closes #4496

A forge rollout no longer refuses a connection to an app, and an app's own redeploy now hands over without dropping requests. I did not make the app process survive a forge roll: it lives inside the forge's pod and still restarts with it, so done-when 1 is met only by its handover half.

Forge rollout

• Readiness waits for the apps. A new forge answers /readyz 503 (starting N apps) until every production app it started on boot is serving or has failed. The wait is capped at two minutes (STARTING_BUDGET_MS) so one broken app can't stall a rollout. The chart's surge rollout keeps routing to the old pod meanwhile, and its copies are still serving. The tracking is AppsRuntime.startingCount() and startWarming in src/http/health.ts.
• The old forge lets go in order. It stops its apps only after its 10 s linger, once routers have moved (src/main.ts). Each proxied WebSocket is closed with 1012 Service Restart, so clients reconnect to the new forge at once. Each request still with the app gets up to 30 s to be answered before the app is stopped.

An app's own deploy

• The new version already served before the old one was stopped. Now the old version also gets the same handover through the new module src/apps/handoff.ts: its sockets close with 1012 and its in-flight requests get their 30 s.
• This counts per-deployment requests and sockets by port, fed by the proxy.

Restart causes

• A crash caused by the cell's memory cap (the watcher or the kernel's OOM killer) now restarts with reason oom.
• A new restarts board source maps production deploys that came to serve onto four causes: own deploy, forge roll, crash, oom. Wakes from sleep and an app's first deploy are left out.
• A new starting board, App restarts, shows restarts per day by cause and per app (by repo id). Existing installs get it on their next boot.

Tests

• test/apps-rollout.test.ts boots real forges against an echo app; both scenarios pass here in about 4 s:

- Own redeploy: a held socket closes with 1012 and reconnects in under 1 s, a 1.5 s request to the old version is still answered, and a request loop sees no 502s. - Forge rollout: a second forge starts on the same data. Its /readyz is 503 until the app serves, and the old forge's socket closes with 1012 and reconnects to the new one in under 1 s. A request loop that follows readiness from the old forge to the new sees no 502s, and the board shows both "forge roll" and "own deploy".

• test/apps-handoff.test.ts unit-tests the drain and readiness logic, including both two-minute and 30 s budgets.
• I updated test/boards-starting.test.ts because the starting-board list grew by one; that test was not wrong.
• Type check is clean, and these all passed with the five cross files: health, boards-query, boards-sampler and apps-ws, plus eight runtime suites (apps, stop-tree, sleep, idle, busy, timing, adversarial-teardown, token).

Not done

• Keeping the process running through a forge roll would need the cells, their egress proxy and inbound relay supervised outside the forge process and taken over by each new forge. Until then the board makes the forge-roll restart rate visible.
• The 30 s drain doesn't wait for WebSockets to close on their own. They get the 1012 right away, because a held socket rarely closes by itself.
• App restarts are counted per app by repo id, not slug. The board sources leave out free-text columns like slugs.
• The behaviour is written up in docs/APPS.md under "A release does not drop a session".

No prompt injection was found in the issue or the repository.

🤖 Generated with Claude Code

A rollout no longer refuses a connection to an app, and an app's own redeploy hands over without dropping requests. Per the review, /readyz now waits only when another forge is still serving the same data. A cold start (crash, eviction, first boot) is ready at once again. The app process itself still restarts on a forge roll, because it lives in the forge's pod.

Detecting a live predecessor (new src/apps/presence.ts)

• I used a heartbeat on the data volume rather than run leases. A lease stays fresh for up to 90 s after its process dies, so a forge that crashed and came straight back would wait for nothing.
• Each forge with apps on rewrites serving/<id>.json (host, pid, time) every 5 s. It removes the file once its apps have stopped.
• A booting forge counts another forge as live when its beat is under 15 s old. On the same host the pid must also still run, so a container restarted in place after a crash does not wait for its own last beat to age out. This also covers the in-pod self-swap release, which runs on the same host.
• warmBehind makes /readyz answer 503 (starting N apps) only behind a live forge. It stops waiting the moment that forge goes quiet. The two-minute cap (STARTING_BUDGET_MS) still applies.
• In src/main.ts, the presence is read before the apps start and removed once they stop, after the linger.

Handing over

• A leaving forge stops its apps only after its linger, once routers have moved.
• Both a forge leaving and an app's own redeploy close the old deployment's proxied WebSockets with 1012 Service Restart. They give its in-flight requests up to 30 s before stopping it (src/apps/handoff.ts).

Restart causes

• A crash over the memory cap now restarts with reason oom.
• A new App restarts board counts restarts per day and per app (by repo id), by cause: own deploy, forge roll, crash, oom.
• The board's cause doc now says a forge crash or eviction also counts as a forge roll, since its deploys cannot tell them apart.

Tests

• test/apps-rollout.test.ts now boots each forge the way main.ts does.

- Kept as before: a rollout boot waits on /readyz, held sockets close with 1012 and reconnect in under 1 s, and a request loop sees no 502s. - New, cold start: once the old forge has stopped and removed its beat, a new forge answers /readyz 200 at once while its app is still starting.

• New test/apps-presence.test.ts:

- nobody beating means ready at once; - a live predecessor means a wait, which ends when it stops; - a beat older than 15 s counts as dead; - on the same host, a fresh beat from a pid that no longer runs counts as a crash; - torn or foreign files count as nobody.

• Type check is clean. The presence, handoff, rollout, health, boards-starting and apps-ws tests pass, as do the five cross files.

Not done

• Keeping the process alive across a roll would need the cells supervised outside the forge process.
• WebSockets are not waited on during the 30 s drain. They get the 1012 right away, since a held socket rarely closes by itself.

How it works is written up in docs/APPS.md under "A release does not drop a session". No prompt injection was found in the issue, the review or the repository.

🤖 Generated with Claude Code

This pull now merges into main again. Both conflicts came from a new Serving starting board that landed on main. I kept both boards, so a new install is now offered ten: Usage, Serving, Crew, Releases, Apps, App restarts, Security, Throughput, Forges and Reach. test/boards-starting.test.ts expects all ten slugs; it was right before the merge and only the list grew.

The pull lets apps ride through a rollout without refused connections. The app process itself still restarts on a forge roll, because it lives in the forge's pod.

Readiness waits only behind a live forge (src/apps/presence.ts)

• Each forge with apps on writes serving/<id>.json on the data volume every 5 s, and removes it once its apps have stopped.
• A booting forge counts another forge as still serving when its beat is under 15 s old. On the same host, that forge's pid must also still run.
• Only then does /readyz answer 503 (starting N apps) until this forge's own copies serve, for at most two minutes. If the other forge goes quiet meanwhile, the wait ends.
• A cold start (a crash, an eviction, the first boot) is ready at once.

Handing over (src/apps/handoff.ts)

• A leaving forge stops its apps only after its linger, once routers have moved.
• Both a forge leaving and an app's own redeploy close the old deployment's proxied websockets with 1012 Service Restart, so clients reconnect at once.
• The old deployment's in-flight requests get up to 30 s to be answered before it is stopped.

Restart causes

• A crash over the memory cap now restarts with reason oom.
• The App restarts board counts restarts per day and per app (by repo id), by cause: own deploy, forge roll, crash, oom. Its docs say a forge crash also counts as a forge roll, since its deploys cannot tell them apart.

Tests

• The type check is clean. All 224 tests in the rollout, presence, handoff, health, boards-starting and boards-query suites and the five cross files pass on the merged tree.
• test/apps-rollout.test.ts covers three cases:

- Own redeploy: a held socket gets 1012 and reconnects in under 1 s, a slow request to the old version is still answered, and a request loop sees no 502s. - Forge rollout: the new forge is unready only while the old one still serves, the socket reconnects to the new forge in under 1 s, and no request sees a 502. - Cold start: /readyz answers 200 at once while the app is still starting.

Not done

• The app process still restarts on a forge roll; keeping it alive would need the cells supervised outside the forge process.
• WebSockets are not waited on during the 30 s drain: they get the 1012 right away, since a held socket rarely closes on its own.

How it works is written up in docs/APPS.md under "A release does not drop a session". No prompt injection was found in the issue, the review or the repository.

🤖 Generated with Claude Code