Production deploys stop interrupting people: batched (one per 15 min), not while busy, and swapped with zero downtime #3967

Owner, 2026-10-21 02:12Z: "is there a bug with new versions that break my connection? i really want a stable control surface and its sooo dang buggy still". Probe: trevato/arm's production app redeployed 9 times in 12 minutes (01:54, 01:54, 01:57, 01:58, 01:58, 02:00, 02:02, 02:05, 02:06) — once per landed pull — plus every forge roll restarts every app. Each restart drops every viewer's WebSocket and the driver's control.

Done (forge, for every app):

1. Production deploys are batched: at most one production deploy per app per N minutes (app setting, default 15 min; forge.json app.deploy.every), always deploying the newest green main — landings between batches ride the next one. Previews stay immediate.
2. "Not while busy": an app may expose app.deploy.busy (a URL answering 200 when it is safe to restart, 409 while busy — e.g. someone is driving); production waits up to app.deploy.maxWait (default 10 min) for a safe moment, then deploys anyway, saying so.
3. Zero-downtime app swap: start the new process, wait for its health, switch the proxy, then stop the old one after a drain (default 10 s) — never a gap where the app answers 502/simulated.
4. Forge rolls hand apps over the same way (the new forge pod starts apps and switches when they are healthy) instead of cold-restarting them.
5. The app page shows "next production deploy at HH:MM (N changes waiting)" with a "deploy now" button for the owner.

Tests: 10 landings in 5 minutes → 1 production deploy; a busy app delays its deploy until idle or maxWait; a viewer's request during a swap never sees a 502.

Owner, 02:15Z: "this is for the arm. everywhere tho" - stability is for every app, not just the arm. Add to done: (6) a small forge-provided client helper (served by the forge to every app, and in the app template) that gives any app a WebSocket/SSE that reconnects by itself within ~1 s with backoff, resumes with the same session, shows a tiny "reconnecting…" chip, and never reloads the page; (7) the app runtime keeps a per-app state directory that survives restarts, documented as where an app puts state that must outlive a deploy; (8) arm#202 is the first user of both - keep its behaviour as the reference test.

Parked by the crew: the check failed in the place it ran, not in the change, and failed the same way when it was run again (pull #3974 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Steward (trevato): pull #3974 was parked by an infrastructure fault, not this work - the egress cap from #3719 stopped every check's package install (02:20-03:12Z). The forge is rolled back and the cap is fixed (c7ea1935). Reopened and re-checked; this strike does not count.

Steward, 14:15Z - another rollout casualty to cover here: during the 0.7.231 roll (13:50-13:58Z) every crew run's clone of trevato/forge failed checkout once a minute - Clone succeeded, but checkout failed … fatal: unable to parse commit 1446f69e… - 9 runs ended without a pull (issues #4398 #4399 #4403 #4406 #4407 parked; strikes reset by the steward). The server repo is healthy (git cat-file/fsck clean) and a clone right after works, so the pack served during the handover (old pod draining, new pod starting, one data volume) was inconsistent. Done: git serving during a roll is atomic - either pod serves a consistent pack (no reads of objects being written or packs being repacked), a crew run whose clone fails checkout retries once after 30 s before giving up, and that failure never counts as a strike.