Release keeper

The release keeper (#7557) cuts, promotes, verifies and, when it has to, rolls back the forge's own releases. It is plain code on the forge's own timer, with no model, so releases keep going out when no one is at the keyboard. It does what the steward's release.sh, verify-after.sh and verify-release.sh did. The code is src/keeper/.

What it does, each minute

  1. Cut on cadence (keeper/cadence.ts). It cuts when all of these hold:

- main has moved past the newest v* tag; - no keeper release is rolling or verifying; - the newest tag is live, or the keeper rolled it back; - no merge-queue gate is running; - no snapshot is being taken, unless it has run two hours and is stuck (a release drains the process the snapshot runs in; #9653); - the disk under the forge's data is less than 80% full (a release writes a new tree beside the live one; at 80% the tick says "the disk is N% full; releases pause at 80% until room is made" and cuts on the first tick with room; #11686); - 30 minutes have passed since the last cut, or ten times the last release's write pause when that is longer (the new forge's hold on its queue work at boot, kept on the release and shown on /releases, in release_keeper and on /maintenance; a 720 s pause waits 120 min; unmeasured leaves 30 min); - no hold stands.

Before anything moves, it records the known-good version and the exact image the Deployment runs. Then it makes the annotated tag v<version> (patch + 1) on main's head, with the same record scripts/cut-release.ts writes, and pushes the tag to the repository's origin when it has one. The cut is a tag, never a commit, so it voids no gate in flight (#6640). The row is written before the tag: if the forge stops between the two, the next tick sends the tag, and a refused tag drops the row (and the local tag) so nothing waits on a release that never left.

  1. Promote. It sets the Deployment's forge container to

<same image repository>:<version>. The chart rolls with maxSurge 1, maxUnavailable 0. If the image is not built yet, the new pod waits on its pull while the old one keeps serving. While the roll runs, the old forge probes /healthz through the public URL once a second and counts every probe that got no 200.

  1. Verify under page load (keeper/verify.ts). The new forge sees

that it is the version in flight and spends 10 minutes on two things at once: - it fetches /boards, the forge's repository page, /explore and /, one at a time, signed in as the first site admin (a session made for the verify and deleted after it); - it probes /healthz once a second.

When the ten minutes pass clean, it looks at a phone (keeper/covered.ts, #11855): /, the forge's repository page and /apps are opened at 390×844 in headless Chromium as the same admin, and the element at the centre of the screen must not be a dialog, a popover, a hidden element or a position: fixed layer over a quarter of the viewport. One that is fails the verify and rolls the release back, naming the page. A forge with no Chromium cannot look and says nothing failed.

A probe is bad when it is not a 200 or takes over 10 s. Three bad probes in a row fail the release.

  1. Verified live, or roll back and hold. On a failure, the keeper

first writes a hold with the version, the reason and the probe numbers. Then it sets the Deployment to the known-good image recorded at the cut, by name (keeper/rollback.ts). There is no "undo to previous": on 2026-10-27 a rollout undo put the broken version back for two minutes. A release that is not live 3 h after its cut is rolled back the same way.

  1. Prune (keeper/prune.ts, #11801). Once a release is verified live,

the keeper removes the deployer's stored releases beyond the newest 15, never the live one or the release's named known-good, however old. Under the self-swap keeper these are the trees under releases/<sha>/; the commits removed are pruned on release.verified_live (with pruneError when the store refused, and the release stays verified). The Deployment-patch deployer keeps no store the forge can reach, so it prunes nothing: there the pg-grand host timer forge-prune-images is still the only pruner, until that deployer gets a store behind the same ReleaseStore port (#11914). If the live, running or known-good release names no stored one, it prunes nothing and says so in pruneError. Landing's own prune of releases/ (keep FORGE_RELEASES_KEEP) never removes the keeper's newest release or its known-good tree. Removal runs inside the tick. Under the swap keeper only, this replaces forge-prune-images.

Every step is an event on the outbox: release.cut, release.released, release.verified_live, release.rolled_back and release.held.

Seconds dark counts the probe-seconds without a 200, through the roll and the verify together. The target is 0. A release with any dark seconds files one issue, Release <version> was dark for <n> s.

Holds

body says the hold by number (release hold 3). Naming the version alone (0.7.284) does not count: changelogs and revert notes mention it too. The keeper then cuts the fix, and the hold is lifted when that release verifies.

(release_hold {lift: <id>}, the card's Lift button, h then lift N) — one function, holdAs/liftAs, behind all three. It needs a person who may roll back (MCP: on an admin-scope token); an agent never can.

A hotfix does not wait (#11839)

release_now {reason} (MCP, admin scope), the Release now form on the /releases card: a person who may roll back (never an agent) asks for a cut at the next tick. It skips the 30-minute spacing, a running merge queue and a snapshot; it still waits for a release in flight to be proven, for a hold and for room on the disk. The request lapses after an hour if main had nothing to cut; the reason is kept on the release (hotfix) and shown on the card.

Where the time went (#11839)

Each release row keeps its stage timings: cut to live, build (checkout of the tree), proof (the preflight boot), switch (the new process taking the port) and verify. The swap deployer measures build, proof and switch; a stage not measured is left out. They show under the release on /releases and in release_keeper (stages), which also answers holding (what the tick last said stopped it: the disk guard, the spacing, a snapshot) and releaseNow.

Where to see it

read as the card, for the same people.

lists each release with its stage, the known-good version, the roll's non-200s, the probes and pages, the worst latency and the seconds dark.

charts seconds dark and worst latency per release, and releases by outcome.

Turning it on

The keeper is off unless all of these are true:

FORGE_KEEPER_DEPLOYMENT and FORGE_KEEPER_CONTAINER name the Deployment and its container (both default forge). The pod's service account needs get and patch on that Deployment; the chart does not grant them. The boot log says why it is off.

While the release tick has never reported, /releases (the "Release tick" row) and the terminal's keeper screen say whether the keeper is off and why (for example "the in-process release keeper is off: FORGE_RELEASE_KEEPER is not on"), or on but not yet ticked. The why names only a variable's state, never its value. If main is then ahead of production past the limit, the owner gets the main_ahead flow alarm with the same line, and the sentinel files no crew issue: no code change fixes a keeper that is off (#11924). An issue filed before stays open until production catches up.

Guardrail: the keeper writes only a tag, the Deployment's image, its own rows and issues. It has no account that may land a pull or push main. Rollback is the one action it takes without a canary, because the image it sets was already live.

Switching it on in production (#7947)

Nothing here is switched on yet: production config is the conductor's, and a run cannot reach it. The answers, from the code:

  1. Reconcile vs. keeper. The keeper re-sends its image only while a

release is rolling. Once it is verified, the ~15-minute standby reconcile (pg-grand) would set the older chart tag back. The fix is in gitops, not in a repair loop here: the reconcile must stop owning the forge container's image, so it never rolls the pod. The chart tag is then only the first image of a fresh Deployment.

  1. The image. The keeper pushes v<version>; the Forgejo release job on

git.open-platform.sh builds <repo>:<version>. While that build is held the new pod waits on its pull, the old pod serves (maxSurge 1, maxUnavailable 0), and the 3 h deadline (ROLL_DEADLINE_MS) rolls back by name and holds. Taking git.open-platform.sh off this path needs an in-cluster build Job started on the tag: a separate change.

  1. FORGE_REPO. release/own_tags.ts, /releases, the behind watch and

the landings read already run on it. releaseOnLand (checks/landing.ts) checks out and preflights each landed tree of that repository only, and swaps the process in only with FORGE_SELF_SWAP=1, which must stay unset in the cluster.

  1. The grant. No manifest in this repository grants it (charts/app is

the apps' chart). It is added in gitops.

The gitops change, in trevato-apps/gitops apps/forge/prod/manifests.yaml:

resourceNames: [forge], verbs: [get, patch], and a RoleBinding to the pod's service account;

For the conductor, in one line: set FORGE_RELEASE_KEEPER=on, grant the forge service account get/patch on deployments/forge, and stop standby-reconcile rendering the forge container's image.

Then the canary: watch one release cut, roll, verify and record on /releases, 0 s dark. Only after that retire the steward's tick release and its launchd job. Until then the Mac's tick stays.

Assumption: the keeper promotes by setting the image directly. Without the gitops change above, a reconcile can set it back after the keeper says verified; with it, nothing does. While a release is rolling the keeper sets the image again each tick, and if it never takes, the 3 h deadline rolls it back and holds.

The first five minutes (#8848)

A release is judged on how it behaves right after boot, when every sweep whose "last time" lives in memory runs once. The process keeps its worst event-loop stall inside its first 5 minutes (src/perf/boot_stall.ts, fed by the loop lag watch); the keeper writes it to the release row when the verify passes. /releases and the terminal show it beside the release before it. Over 5 s the release stays live (it answers) and a failed hold stands with the reason — the stall, the limit, the last release's — until a pull saying release hold N lands and a release verifies, as for any failed hold.

Cause of 0.7.339's 46 s: the review-repeat sweep (flow/repeats.ts, #8716) ran on the first tick and grouped a week of findings against every group so far, quadratically (51 s on the production-sized fixture of test/boot-no-stall.test.ts). It now finds groups through a word index and hands the thread back between slices (kernel/breathe.ts); the tick breathes between its passes.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs