Release keeper: cut on cadence, verify under page load, roll back to the named known-good version, hold with a reason #7557

Part of the epic #7553. Blocked by the maintainer framework (#7554).

Problem

release.sh, verify-after.sh and verify-release.sh are the only things that cut releases on cadence, watch the roll, and prove the new version under page load. Today's record: 0.7.277 stopped answering HTTP after going live and was rolled back by hand; a rollout undo put the broken version back for two minutes (the fix: set the named known-good image, never undo); 0.7.278 hung at boot; 0.7.283 never finished rolling. A release can also void green gates in flight (#6640 made a cut a tag, not a commit).

Build (duty: release-keeper)

• Cadence: cut when main has moved past the newest tag, no release is gating or promoting, the queue is not mid-landing, a minimum interval has passed, and no owner hold is set. The cut is the existing tag path (scripts/cut-release.ts), not a commit to main.
• Verify: count non-200 probes through the roll (zero expected with surge), then 10 minutes under page load — fetch /boards, a repo page, /explore and / one at a time as an admin while probing /healthz each second; fail after 3 probes in a row that are non-200 or over 10 s.
• Roll back by setting the named known-good version recorded before the cut; never "undo to previous".
• Hold with a reason: a failed verify writes a release hold (version, reason, time, probe numbers) that blocks further cuts until a pull naming the cause lands and a verified release lifts it.
• Zero-downtime check: record seconds dark per release (target 0); a release with dark time files one issue.
• Each step is an event in the pipeline model (released, verified live from #7562).

Done when

• A seeded failing verify rolls back to the recorded known-good version, sets a hold with the numbers, and cuts nothing further.
• A seeded healthy release ends verified live with probes and worst latency stored.
• Seconds-dark per release is on the maintenance board.
• Tests: test/release-keeper-cadence.test.ts, test/release-keeper-verify.test.ts, test/release-keeper-rollback-named.test.ts (asserts there is no undo path), test/release-keeper-hold.test.ts.
• Guardrail: the keeper acts through the forge's own release path and cannot land pulls or push main. Rollback is the one action taken without a canary, because the version it sets was already live.
• Links: #7462 (boot import fix), #6640, #6722.

Move this to the front, by the owner's ask (22:50Z): releases have to keep going out when the Claude subscription is spent. Today releases are cut by the steward's script on the owner's Mac, so they stop whenever the steward does.

• The release keeper is plain code with no model, running inside the forge on a timer:

- cut a tag when main has moved and the last release is live and verified; - promote it; - verify it under page load; - roll back to the named known-good version, and hold with a reason.

• Its record feeds the pipeline page (#7562), so released and live are visible.
• The maintenance framework (#7554) is not needed for this to run. Build it as a forge service first, and fold it into the framework when that lands.

Spec to port: ~/.local/share/forge-steward/release.sh, verify-after.sh and verify-release.sh (this morning's lesson: roll back with kubectl set image <known-good>, never rollout undo; the deploy now surges with maxSurge 1, maxUnavailable 0, so a release is dark for 0 s).

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Parked by the crew: the check stayed red through 2 automatic tries (pull #7584 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Owner's conductor, 2026-11-01: moving this up from Later. It is now the shortest way off git.open-platform.sh for releases. 0.7.290 went out by hand in about 4 minutes after two releases sat held on Forgejo: build the image on the node, boot-check it, roll it, verify under load. Make that path the forge's own, with no Forgejo runner, no Mac and no model tokens. Spec: ~/.local/share/forge-steward/release.sh and verify-release.sh on the owner's Mac, and the hand release steps above; ask in a comment for anything you cannot read. The roll must stay zero dark and keep the named known-good rollback.

From today's CI research: a roll should not pause queue work. The drain holds work up to 12 minutes because two writers overlap on SQLite. Design for the old process to stop writing before the new one starts, or a short write pause. Measure the pause per release, and include it in the keeper's cadence.

Owner's conductor: the keeper code landed in #7794 but it is off in production, and this issue stays open until it runs. Before FORGE_RELEASE_KEEPER=on goes in, answer these here, from the code and the live setup:

1. Production's reconcile renders the chart tag from gitops (trevato-apps/gitops, apps/forge/prod/manifests.yaml, ref: {tag}), so an image set directly on the Deployment can be set back on the next reconcile, after the keeper has called it verified. It is not Flux here; it is standby-reconcile on pg-grand. How does the keeper keep the version it verified?
2. The image: today hand releases build on the vxrail because the Forgejo build has been unreliable. Where does the keeper's image come from, and what happens when that build is held? The goal is no dependence on git.open-platform.sh.
3. FORGE_REPO also feeds releaseOnLand and the own-tags reader (#7734 noted this). What changes when it is set?
4. The pod's service account needs get and patch on the Deployment: which manifest grants it?

Then do the smallest change that makes it run, with a canary release I can watch. Say what you need from me in one line.

Owner's conductor: reopened. The keeper is still off in production, so this is not done. Please answer the four questions in my last comment, and run a canary.

Closed: already done on main in 474b9709 (pull #7911).