Public: anyone can read this; the crew can change it.

Release keeper: cut on cadence, verify under page load, roll back to the named known-good version, hold with a reason #7584

Parked opened this in answer to
Closed without merging

torque/forge:torque/7557-release-keeper-cut-on-cadence-verify-und → main 8 commits fork 4 threads · 4 unresolved

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
34 files changed +1958 −6
README.md +2
@@ -483,6 +483,8 @@ What the crew's tokens bought is on the board (`/board/returns`), in MCP `return
483483
484484 The terminal's palette has `Signed in` for anyone signed in: your browsers, tokens and this terminal's key, with when each was last seen; `d` signs the selected browser out after a confirm, as `/settings` and MCP `sessions` do (`docs/CREW.md`, "Where you are signed in").
485485
486 The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases on the forge's own one-minute timer, with no model: it is off until `FORGE_RELEASE_KEEPER=on` (and the pod's service account may `get` and `patch` the forge Deployment). It tags main's head as `v<patch + 1>` when main has moved past a live tag, nothing is in flight, the merge queue is still, 30 minutes have passed since its last cut and no hold stands; it records the known-good version and image before the tag, sets the Deployment's image, and has the new version spend ten minutes under page load probing `/healthz` each second. Three bad probes in a row hold further cuts with the numbers and set the recorded known-good image back by name (never "undo"). A failed verify's hold lets the fix through once a pull saying `release hold <N>` lands, and that release verifying lifts it; an owner's hold is set and lifted with MCP `release_hold` (a person with an admin-scope token, never an agent). To see it: the **Release keeper** card on `/releases`, the palette's `Release keeper` in the terminal (the same read, gated as `/releases` is), the **Maintenance** board under `/boards`, and MCP `release_keeper`. How it works and how to turn it on: [`docs/features/release-keeper.md`](docs/features/release-keeper.md).
487
486488 `/changelog` lists the last 30 releases of `FORGE_REPO` (public only) with their pulls, for anyone; the terminal's palette has `Changelog` for the same list. In the terminal, `/` takes a word and keeps the releases whose version or pull titles contain it (`esc` clears it); Enter on a release lists its pulls, and Enter on a pull opens it (`docs/features/changelog.md`).
487489
488490 A repository admin cuts a wave from two doors, both through `cutWaveFor` (`src/actions/cut_wave.ts`): Settings → *Cut a wave*, and `c` on the terminal's waves screen, which says how many pulls would go in, asks `cut wave N? y/n` (`y` and `n` choose outright) and shows the draft pull's number (Enter opens it). Each opens a draft pull listing the pulls merged since the last wave; nothing merged, an earlier wave that does not parse, or a viewer who is not an admin is refused in the action's words, writing nothing.
docs/MCP.md +2
@@ -136,6 +136,8 @@ for a tunnel and a forge that sleeps.
136136 | `queue_gates` | `{ repo, limit? }`: the merge queue's recent gates, newest first (`limit` 1–100, default 20), as `GET …/queue/gates` answers: each gate's pulls, state, reason, counts and failing tests by name with their first error lines, a test red in two gates that share no pull flagged `flaky` | |
137137 | `quotas` | `{ key?, value?, dryRun? }` (forge admin): the per-account quotas `repositories`, `apps`, `storage` (MB), `previews`, `runs`, `checks`, each a number or null for none. `repositories` bounds repos on create, fork and copy; `apps` the apps made; `storage` the MB of repos plus app data, asked on push, create and fork; `previews` live previews; `runs` concurrent agent runs; `checks` concurrent checks. Each is asked before the thing is made or started and a forge admin is never limited; with `key` and `value` (`""` clears), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes |
138138 | `read_file` | a file (utf-8, or base64 for binary) or a directory listing at a ref, as GET …/contents; MP4/MOV/M4A files also carry about: {kind, seconds, width?, height?, says} | |
139 | `release_hold` | `{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent | yes |
140 | `release_keeper` | `{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases | |
139141 | `releases` | `{}`: the forge's kept releases newest first: short sha, version, the pull that landed it, when, and for the live one `live: true` and `since`; for whoever may read /releases, others get its refusal; no rollback, a person does that on the page | |
140142 | `repo` | `{ repo }`: one repository's description, visibility, autonomy, `audience { level, sentence }` and `provenance` (where a migrated one came from, `null` when hidden), the same as `GET /api/v1/repos/:owner/:name` | |
141143 | `repo_labels` | `{ repo, action?, name?, color?, id? }`: list (`list`, the default, paged by `limit` 1–200 and `cursor`, as `{ items, next_cursor }`; a read-only token may), create (`create`, `name`, `color` six hex digits) or remove (`remove`, `id`) the repository's labels, as `GET`/`POST …/labels` and `DELETE …/labels/:id`; create and remove need an admin token and a repository admin | yes |
docs/features/release-keeper.md added +102
@@ -0,0 +1,102 @@
1 # Release keeper
2
3 The release keeper (#7557) cuts, promotes, verifies and, when it has to,
4 rolls back the forge's own releases. It is plain code on the forge's own
5 timer, with no model, so releases keep going out when no one is at the
6 keyboard. It does what the steward's `release.sh`, `verify-after.sh` and
7 `verify-release.sh` did. The code is `src/keeper/`.
8
9 ## What it does, each minute
10
11 1. **Cut on cadence** (`keeper/cadence.ts`). It cuts when all of these hold:
12 - main has moved past the newest `v*` tag;
13 - no keeper release is rolling or verifying;
14 - the newest tag is live, or the keeper rolled it back;
15 - no merge-queue gate is running;
16 - 30 minutes have passed since the last cut;
17 - no hold stands.
18
19 Before anything moves, it records the known-good version and the exact
20 image the Deployment runs. Then it makes the annotated tag
21 `v<version>` (patch + 1) on main's head, with the same record
22 `scripts/cut-release.ts` writes, and pushes the tag to the repository's
23 origin when it has one. The cut is a tag, never a commit, so it voids no
24 gate in flight (#6640). The row is written before the tag: if the forge
25 stops between the two, the next tick sends the tag, and a refused tag
26 drops the row (and the local tag) so nothing waits on a release that
27 never left.
28 2. **Promote.** It sets the Deployment's forge container to
29 `<same image repository>:<version>`. The chart rolls with
30 `maxSurge 1, maxUnavailable 0`. If the image is not built yet, the new
31 pod waits on its pull while the old one keeps serving. While the roll
32 runs, the old forge probes `/healthz` through the public URL once a
33 second and counts every probe that got no 200.
34 3. **Verify under page load** (`keeper/verify.ts`). The new forge sees
35 that it is the version in flight and spends 10 minutes on two things at
36 once:
37 - it fetches `/boards`, the forge's repository page, `/explore` and `/`,
38 one at a time, signed in as the first site admin (a session made for
39 the verify and deleted after it);
40 - it probes `/healthz` once a second.
41
42 A probe is bad when it is not a 200 or takes over 10 s. Three bad probes
43 in a row fail the release.
44 4. **Verified live**, or **roll back and hold**. On a failure, the keeper
45 first writes a hold with the version, the reason and the probe numbers.
46 Then it sets the Deployment to the known-good image recorded at the cut,
47 by name (`keeper/rollback.ts`). There is no "undo to previous": on
48 2026-10-27 a `rollout undo` put the broken version back for two minutes.
49 A release that is not live 3 h after its cut is rolled back the same way.
50
51 Every step is an event on the outbox: `release.cut`, `release.released`,
52 `release.verified_live`, `release.rolled_back` and `release.held`.
53
54 **Seconds dark** counts the probe-seconds without a 200, through the roll
55 and the verify together. The target is 0. A release with any dark seconds
56 files one issue, `Release <version> was dark for <n> s`.
57
58 ## Holds
59
60 - **A failed verify's hold** blocks every cut. Land a pull whose title or
61 body says the hold by number (`release hold 3`). Naming the version alone
62 (`0.7.284`) does not count: changelogs and revert notes mention it too.
63 The keeper then cuts the fix, and the hold is lifted when that release
64 verifies.
65 - **An owner's hold** is set with MCP `release_hold {reason}` and lifted
66 with `release_hold {lift: <id>}`. It needs a person who may roll back, on
67 an admin-scope token; an agent never can.
68
69 ## Where to see it
70
71 - The terminal: **Release keeper** in the command palette shows the same
72 read as the card, for the same people.
73 - `/releases`: the **Release keeper** card says what holds the next cut and
74 lists each release with its stage, the known-good version, the roll's
75 non-200s, the probes and pages, the worst latency and the seconds dark.
76 - The **Maintenance** board (`/boards`, from the `keeperReleases` source)
77 charts seconds dark and worst latency per release, and releases by
78 outcome.
79 - MCP `release_keeper` returns the same data as JSON.
80
81 ## Turning it on
82
83 The keeper is off unless all of these are true:
84
85 - `FORGE_RELEASE_KEEPER=on`;
86 - `FORGE_REPO` names a repository on this forge;
87 - the forge runs in a cluster.
88
89 `FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER` name the Deployment
90 and its container (both default `forge`). The pod's service account needs
91 `get` and `patch` on that Deployment; the chart does not grant them. The
92 boot log says why it is off.
93
94 Guardrail: the keeper writes only a tag, the Deployment's image, its own
95 rows and issues. It has no account that may land a pull or push main.
96 Rollback is the one action it takes without a canary, because the image it
97 sets was already live.
98
99 Assumption: the keeper promotes by setting the image directly, as the
100 steward's rollback did. If Flux reconciles the image back, the keeper sets
101 it again on its next tick while the release is rolling. If it never takes,
102 the 3 h deadline rolls it back and holds.
src/boards/sources.ts +22
@@ -285,6 +285,28 @@ export const SOURCES: Readonly<Record<string, Source>> = {
285285 reports: "reports kept that day (production only)",
286286 },
287287 },
288 // The release keeper's releases (#7557), one row each: what the probes
289 // counted through the roll and the ten minutes under page load. Seconds
290 // dark is the target-zero number. The stop reason is free text, not listed.
291 keeperReleases: {
292 table: "keeper_release",
293 owner: "admin",
294 time: "cut_at",
295 columns: {
296 version: "text", stage: "text", cut_at: "time", live_at: "time", finished_at: "time",
297 roll_probes: "number", roll_non200: "number", probes: "number", pages: "number",
298 bad_in_row: "number", worst_ms: "number", seconds_dark: "number",
299 },
300 docs: {
301 stage: "rolling, verifying, verified (verified live) or rolled_back",
302 roll_non200: "/healthz probes, one a second, that got no 200 while the roll ran (target 0 with surge)",
303 probes: "/healthz probes in the ten minutes under page load",
304 pages: "pages fetched as an admin under load, one at a time",
305 bad_in_row: "the most bad probes in a row (non-200 or over 10 s); 3 fails the release",
306 worst_ms: "the slowest probe or page, ms",
307 seconds_dark: "probe-seconds with no 200, roll and verify together (target 0)",
308 },
309 },
288310 // Forges made by the one-click path (#2914): one row per spawn event
289311 // (`actions/spawn.ts`, `spawn/live.ts`). The forge's name and the failure's
290312 // words are not listed; the step is one of a fixed few.
src/boards/starting.ts +13 −1
@@ -1,5 +1,5 @@
11 /**
2 * The boards the forge offers a new install: Usage, Serving, Crew, Releases, Apps, App restarts, Security, Throughput, Forges and Reach. They are plain
2 * The boards the forge offers a new install: Usage, Serving, Crew, Releases, Maintenance, Apps, App restarts, Security, Throughput, Forges and Reach. They are plain
33 * boards in `forge/boards` once written, so a person edits or deletes them
44 * like any other; `.offered` (a JSON list of slugs, committed with the
55 * boards) records what was offered so a deletion sticks and an edit is
@@ -83,6 +83,18 @@ export const STARTING_BOARDS: Readonly<Record<string, Board>> = {
8383 panel("Decider time per hour (ms)", "line", { source: "decisions", aggregate: "avg", column: "ms", groupBy: HOUR, series: "kind" }, 2),
8484 ],
8585 },
86 // The release keeper (#7557): every release dark for 0 s is the target.
87 maintenance: {
88 title: "Maintenance",
89 range: "30d",
90 refresh: "1m",
91 panels: [
92 panel("Seconds dark per release", "bar", { source: "keeperReleases", aggregate: "max", column: "seconds_dark", groupBy: "version" }, 2),
93 panel("Worst latency per release (ms)", "bar", { source: "keeperReleases", aggregate: "max", column: "worst_ms", groupBy: "version" }, 2),
94 panel("Releases by outcome", "bar", { source: "keeperReleases", aggregate: "count", groupBy: "stage" }),
95 panel("Seconds dark (30 days)", "number", { source: "keeperReleases", aggregate: "sum", column: "seconds_dark" }),
96 ],
97 },
8698 apps: {
8799 title: "Apps",
88100 range: "7d",
src/core/event_audience.ts +2
@@ -176,6 +176,8 @@ export function classifiedAudience(kind: string): EventAudience | undefined {
176176 kind === "repo.crew_outsiders" ||
177177 // A clog starting names the forge's own pile-up: the owner's to hear (#6524).
178178 kind.startsWith("clog.") ||
179 // The release keeper's steps name images and probe numbers: the forge owner's (`keeper/keeper.ts`, #7557).
180 kind.startsWith("release.") ||
outdated

This inserts the release. clause between the 'file held' comment and the repo.file_held line it describes. Move it above that comment so each comment sits on its own clause.

179181 // A file held against new work is its admins' book-keeping (#6522).
180182 kind === "repo.file_held" ||
181183 kind === "repo.file_released"
src/core/migration-pins/20261027140000-release-keeper.pin added +1
@@ -0,0 +1,1 @@
1 e24c6b4f77b34f5ba3d6f043f2da2788dd0b2e239ebfd5970e12c3b99b1016be
src/core/migrations/20261027140000-release-keeper.ts added +39
@@ -0,0 +1,39 @@
1 // The release keeper (#7557): one row per release it cut, with the
2 // known-good image it rolls back to and what the probes counted, and the
3 // holds that stop further cuts. `kind` is `failed` (a verify that failed,
4 // lifted by a verified release after a pull naming it lands) or `owner`
5 // (set and lifted by an admin).
6 export default `
7 CREATE TABLE keeper_release (
8 id INTEGER PRIMARY KEY AUTOINCREMENT,
9 version TEXT NOT NULL UNIQUE,
10 sha TEXT NOT NULL,
11 known_good_version TEXT NOT NULL,
12 known_good_image TEXT NOT NULL,
13 stage TEXT NOT NULL,
14 cut_at TEXT NOT NULL,
15 live_at TEXT,
16 finished_at TEXT,
17 roll_probes INTEGER NOT NULL DEFAULT 0,
18 roll_non200 INTEGER NOT NULL DEFAULT 0,
19 probes INTEGER NOT NULL DEFAULT 0,
20 pages INTEGER NOT NULL DEFAULT 0,
21 bad_in_row INTEGER NOT NULL DEFAULT 0,
22 worst_ms INTEGER NOT NULL DEFAULT 0,
23 seconds_dark INTEGER NOT NULL DEFAULT 0,
24 reason TEXT NOT NULL DEFAULT ''
25 );
26 CREATE TABLE keeper_hold (
27 id INTEGER PRIMARY KEY AUTOINCREMENT,
28 kind TEXT NOT NULL,
29 version TEXT NOT NULL,
30 reason TEXT NOT NULL,
31 probes INTEGER NOT NULL DEFAULT 0,
32 bad_in_row INTEGER NOT NULL DEFAULT 0,
33 worst_ms INTEGER NOT NULL DEFAULT 0,
34 created_at TEXT NOT NULL,
35 cause_pull INTEGER,
36 lifted_at TEXT,
37 lifted_by TEXT
38 );
39 `;
src/http/mcp/tools/index.ts +2
@@ -106,6 +106,8 @@ export { tool as pulls } from "./pulls.ts";
106106 export { tool as queue_gates } from "./queue_gates.ts";
107107 export { tool as quotas } from "./quotas.ts";
108108 export { tool as read_file } from "./read_file.ts";
109 export { tool as release_hold } from "./release_hold.ts";
110 export { tool as release_keeper } from "./release_keeper.ts";
109111 export { tool as releases } from "./releases.ts";
110112 export { tool as repo } from "./repo.ts";
111113 export { tool as repo_labels } from "./repo_labels.ts";
src/http/mcp/tools/release_hold.ts added +36
@@ -0,0 +1,36 @@
1 import { mayRollBack } from "../../../release/gate.ts";
2 import { lift, listHolds, MAX_REASON, setHold } from "../../../keeper/hold.ts";
3 import { running } from "../../../kernel/version.ts";
4 import { bad, good, str, type Tool } from "../shared.ts";
5
6 export const tool: Tool = {
7 name: "release_hold",
8 doc: "`{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent",
9 description: "Hold or release the release keeper's cuts (admin).",
10 inputSchema: {
11 type: "object",
12 properties: { reason: { type: "string", maxLength: MAX_REASON }, lift: { type: "integer", minimum: 1 } },
13 },
14 writes: true,
15 needsAdminScope: true,
16 run(ctx, user, args) {
17 if (!mayRollBack(ctx.app, user))
18 return bad("forbidden", "holding releases is for a person who may write the forge's repository and administer it, not an agent");
19 const db = ctx.app.db;
20 const id = args["lift"];
21 if (id !== undefined) {
22 if (typeof id !== "number" || !Number.isInteger(id) || id < 1) return bad("invalid", "lift is a hold's id");
23 const hold = listHolds(db, 1000).find((h) => h.id === id);
24 if (hold === undefined || hold.liftedAt !== null) return bad("not_found", "no standing hold has that id");
25 if (hold.kind !== "owner") return bad("refused", "a failed verify's hold is lifted by a verified release after a pull saying \"release hold N\" lands");
26 lift(db, id, user.username, Date.now());
27 ctx.app.bus.append({ kind: "release.hold_lifted", actorId: user.id, payload: { hold: id } });
28 return good({ lifted: id });
29 }
30 const reason = str(args["reason"]);
31 if (reason === "" || reason.length > MAX_REASON) return bad("invalid", `reason is one line of 1 to ${MAX_REASON} characters`);
32 const hold = setHold(db, { kind: "owner", version: running().version, reason, at: Date.now() });
33 ctx.app.bus.append({ kind: "release.held", actorId: user.id, payload: { hold: hold.id, reason: hold.reason, owner: true } });
34 return good({ hold });
35 },
36 };
src/http/mcp/tools/release_keeper.ts added +22
@@ -0,0 +1,22 @@
1 import { mayRead } from "../../../release/gate.ts";
2 import { blocking, listHolds, standingHolds } from "../../../keeper/hold.ts";
3 import { listKeeperReleases } from "../../../keeper/record.ts";
4 import { bad, good, type Tool } from "../shared.ts";
5
6 export const tool: Tool = {
7 name: "release_keeper",
8 doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases",
9 description: "The release keeper's releases, holds and what blocks the next cut.",
10 inputSchema: { type: "object", properties: {} },
11 writes: false,
12 run(ctx, user) {
13 if (!mayRead(ctx.app, user))
14 return bad("forbidden", "the release history is for the forge's administrators and its repository's readers");
15 const db = ctx.app.db;
16 return good({
17 blocked: blocking(standingHolds(db)) ?? null,
18 releases: listKeeperReleases(db),
19 holds: listHolds(db),
20 });
21 },
22 };
src/keeper/cadence.ts added +65
@@ -0,0 +1,65 @@
1 /**
2 * When the release keeper cuts (#7557): plain data in, one answer out.
3 *
4 * cut ⇔ main ≠ the newest tag's commit
5 * ∧ no release of the keeper's is in flight (cut, rolling, verifying)
6 * ∧ the live version is the newest tag's, or the keeper rolled that
7 * tag back (nothing still promoting)
8 * ∧ no merge-queue gate is running (the queue is not mid-landing)
9 * ∧ now − the last cut ≥ the minimum interval
10 * ∧ no hold stands (a failed verify's, or an owner's)
11 *
12 * The first rule that refuses names itself, so the record says why not.
13 * The version is the newest tag's with its patch number plus one.
14 */
15
16 /** Two releases are at least this far apart: 30 minutes. */
17 export const MIN_INTERVAL_MS = 30 * 60_000;
18
19 export interface CadenceInput {
20 /** main's head commit. */
21 readonly main: string;
22 /** The newest release tag: its version and the commit it names. */
23 readonly newest: { readonly version: string; readonly sha: string } | undefined;
24 /** The version this forge runs. */
25 readonly live: string;
26 /** The keeper rolled the newest tag back: it is over, not promoting. */
27 readonly settled?: boolean;
28 /** A keeper release not yet verified or rolled back. */
29 readonly inFlight: boolean;
30 /** Merge-queue gates running now. */
31 readonly gating: number;
32 /** When the keeper last cut, epoch ms. */
33 readonly lastCutAt: number | undefined;
34 /** The reason of a standing hold, when one stands. */
35 readonly hold: string | undefined;
36 readonly now: number;
37 readonly minIntervalMs?: number;
38 }
39
40 export type Cadence = { readonly cut: true; readonly version: string } | { readonly cut: false; readonly why: string };
41
42 const SEMVER = /^(\d+)\.(\d+)\.(\d+)$/;
43
44 /** The next patch version: `0.7.283` → `0.7.284`; undefined when `version` is not major.minor.patch. */
45 export function nextVersion(version: string): string | undefined {
46 const m = SEMVER.exec(version);
47 if (m === null) return undefined;
48 return `${m[1]}.${m[2]}.${Number(m[3]) + 1}`;
49 }
50
51 export function decideCut(input: CadenceInput): Cadence {
52 const no = (why: string): Cadence => ({ cut: false, why });
53 if (input.hold !== undefined) return no(`held: ${input.hold}`);
54 if (input.newest === undefined) return no("no release tag to follow; cut the first one by hand");
55 if (input.main === input.newest.sha) return no(`main is still ${input.newest.version}`);
56 if (input.inFlight) return no("a release is still rolling or being verified");
57 if (input.live !== input.newest.version && input.settled !== true) return no(`${input.newest.version} is not live yet (live: ${input.live})`);
58 if (input.gating > 0) return no(`the merge queue is landing (${input.gating} gate${input.gating === 1 ? "" : "s"} running)`);
59 const wait = input.minIntervalMs ?? MIN_INTERVAL_MS;
60 if (input.lastCutAt !== undefined && input.now - input.lastCutAt < wait)
61 return no(`the last cut was ${Math.floor((input.now - input.lastCutAt) / 60_000)} min ago; the interval is ${wait / 60_000} min`);
62 const version = nextVersion(input.newest.version);
63 if (version === undefined) return no(`${input.newest.version} is not major.minor.patch`);
64 return { cut: true, version };
65 }
src/keeper/deploy.ts added +103
@@ -0,0 +1,103 @@
1 /**
2 * The release keeper's hands on the cluster (#7557): read the image the
3 * forge's Deployment runs, set it to a named image, and say whether a roll
4 * to an image has finished. That is all. There is no "undo to previous":
5 * on 2026-10-27 a `rollout undo` put the broken version back for two
6 * minutes, because "previous" is whatever the controller last held. A
7 * rollback here sets the image recorded before the cut, by name
8 * (`keeper/rollback.ts`).
9 *
10 * Spoken over `fetch` to the kube API with the pod's service account
11 * (`cluster/jobs.ts` `serviceAccount`), as a strategic-merge patch of the
12 * one container's image, the same request `kubectl set image` sends.
13 */
14
15 import type { Credentials } from "../cluster/jobs.ts";
16
17 export interface Deployer {
18 /** The image the Deployment's forge container names now. */
19 image(): Promise<string>;
20 /** Name this image on the forge container; the Deployment rolls to it with surge. */
21 setImage(image: string): Promise<void>;
22 /** The roll to `image` is done: every replica updated and available, none left of the old one. */
23 rolled(image: string): Promise<boolean>;
24 }
25
26 /** An image reference: registry/path:tag or @digest, no space, no flag. */
27 const IMAGE = /^[a-z0-9][a-z0-9._\-/:]{0,254}(@sha256:[a-f0-9]{64})?$/;
28 const LABEL = /^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$/;
29
30 export const isImage = (s: string): boolean => IMAGE.test(s) && !s.includes("..");
31
32 /** The image `image` names with its tag replaced by `version`: `reg:5000/o/forge:0.7.1` → `reg:5000/o/forge:0.7.2`. */
33 export function imageAt(image: string, version: string): string {
34 const repo = image.split("@")[0]!;
35 const slash = repo.lastIndexOf("/");
36 const colon = repo.lastIndexOf(":");
37 return `${colon > slash ? repo.slice(0, colon) : repo}:${version}`;
38 }
39
40 /** The tag an image names, or "" when it names none. */
41 export function tagOf(image: string): string {
42 const repo = image.split("@")[0]!;
43 const colon = repo.lastIndexOf(":");
44 return colon > repo.lastIndexOf("/") ? repo.slice(colon + 1) : "";
45 }
46
47 type Fetch = (url: string, init: RequestInit & { tls?: { ca: string } }) => Promise<Response>;
48
49 interface DeploymentJson {
50 metadata?: { generation?: number };
51 spec?: { replicas?: number; template?: { spec?: { containers?: { name?: string; image?: string }[] } } };
52 status?: { observedGeneration?: number; replicas?: number; updatedReplicas?: number; availableReplicas?: number };
53 }
54
55 export function kubeDeployer(input: {
56 credentials: Credentials;
57 deployment: string;
58 container: string;
59 fetch?: Fetch;
60 }): Deployer {
61 const { credentials: c, deployment, container } = input;
62 if (!LABEL.test(deployment) || !LABEL.test(container) || !LABEL.test(c.namespace))
63 throw new Error("deployment, container and namespace must be DNS labels");
64 const call = input.fetch ?? (fetch as unknown as Fetch);
65 const url = `${c.server}/apis/apps/v1/namespaces/${c.namespace}/deployments/${deployment}`;
66 const auth = { authorization: `Bearer ${c.token}` };
67 const tls = c.ca === undefined ? {} : { tls: { ca: c.ca } };
68 const read = async (): Promise<DeploymentJson> => {
69 const res = await call(url, { headers: auth, ...tls });
70 if (!res.ok) throw new Error(`reading deployment ${deployment}: HTTP ${res.status}`);
71 return (await res.json()) as DeploymentJson;
72 };
73 const imageOf = (d: DeploymentJson): string => {
74 const found = d.spec?.template?.spec?.containers?.find((k) => k.name === container)?.image;
75 if (found === undefined) throw new Error(`deployment ${deployment} has no container ${container}`);
76 return found;
77 };
78 return {
79 image: async () => imageOf(await read()),
80 async setImage(image) {
81 if (!isImage(image)) throw new Error("not an image reference");
82 const res = await call(url, {
83 method: "PATCH",
84 headers: { ...auth, "content-type": "application/strategic-merge-patch+json" },
85 body: JSON.stringify({ spec: { template: { spec: { containers: [{ name: container, image }] } } } }),
86 ...tls,
87 });
88 if (!res.ok) throw new Error(`setting ${deployment} to ${image}: HTTP ${res.status}`);
89 },
90 async rolled(image) {
91 const d = await read();
92 const want = d.spec?.replicas ?? 1;
93 const s = d.status ?? {};
94 return (
95 imageOf(d) === image &&
96 (s.observedGeneration ?? -1) >= (d.metadata?.generation ?? 0) &&
97 (s.updatedReplicas ?? 0) >= want &&
98 (s.availableReplicas ?? 0) >= want &&
99 (s.replicas ?? 0) <= want
100 );
101 },
102 };
103 }
src/keeper/hold.ts added +114
@@ -0,0 +1,114 @@
1 /**
2 * Release holds (#7557): while one stands, the keeper cuts nothing.
3 *
4 * - `failed`: a verify failed. It carries the version, the reason and the
5 * probe numbers. A pull that lands saying `release hold 4` is its cause —
6 * the hold by number, never the bare version, which changelogs and revert
7 * notes mention too. With a cause named the keeper may cut again; the
8 * hold is lifted when that release verifies.
9 * - `owner`: an admin set it, with a reason; an admin lifts it.
10 */
11
12 import type { Database } from "bun:sqlite";
13
14 export type HoldKind = "failed" | "owner";
15
16 export interface Hold {
17 readonly id: number;
18 readonly kind: HoldKind;
19 readonly version: string;
20 readonly reason: string;
21 readonly probes: number;
22 readonly badInRow: number;
23 readonly worstMs: number;
24 readonly createdAt: string;
25 readonly causePull: number | null;
26 readonly liftedAt: string | null;
27 readonly liftedBy: string | null;
28 }
29
30 interface Row {
31 id: number;
32 kind: HoldKind;
33 version: string;
34 reason: string;
35 probes: number;
36 bad_in_row: number;
37 worst_ms: number;
38 created_at: string;
39 cause_pull: number | null;
40 lifted_at: string | null;
41 lifted_by: string | null;
42 }
43
44 const toHold = (r: Row): Hold => ({
45 id: r.id,
46 kind: r.kind,
47 version: r.version,
48 reason: r.reason,
49 probes: r.probes,
50 badInRow: r.bad_in_row,
51 worstMs: r.worst_ms,
52 createdAt: r.created_at,
53 causePull: r.cause_pull,
54 liftedAt: r.lifted_at,
55 liftedBy: r.lifted_by,
56 });
57
58 /** A reason is one line a person reads: at most this long. */
59 export const MAX_REASON = 500;
60
61 export function setHold(
62 db: Database,
63 input: { kind: HoldKind; version: string; reason: string; probes?: number; badInRow?: number; worstMs?: number; at: number },
64 ): Hold {
65 const reason = input.reason.replace(/[\u0000-\u001f\u007f]+/g, " ").trim().slice(0, MAX_REASON);
66 return toHold(
67 db
68 .query<Row, [string, string, string, number, number, number, string]>(
69 `INSERT INTO keeper_hold (kind, version, reason, probes, bad_in_row, worst_ms, created_at)
70 VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) RETURNING *`,
71 )
72 .get(input.kind, input.version, reason, input.probes ?? 0, input.badInRow ?? 0, input.worstMs ?? 0, new Date(input.at).toISOString())!,
73 );
74 }
75
76 /** The holds not lifted, oldest first. */
77 export function standingHolds(db: Database): Hold[] {
78 return db.query<Row, []>(`SELECT * FROM keeper_hold WHERE lifted_at IS NULL ORDER BY id`).all().map(toHold);
79 }
80
81 /** What stops a cut, in one line; undefined when nothing does. A failed hold whose cause has landed lets the fix through. */
82 export function blocking(holds: readonly Hold[]): string | undefined {
83 const h = holds.find((x) => x.kind === "owner" || x.causePull === null);
84 if (h === undefined) return undefined;
outdated

A bare version string in any merged pull's title or body counts as the cause and unblocks cuts. Narrow this to an explicit release hold N reference (or a pull a rollback-capable person linked) and test that a mere mention does not lift it.

85 return h.kind === "owner"
86 ? `the owner holds releases: ${h.reason}`
87 : `${h.version} failed its verify (${h.reason}); waiting for a pull saying "release hold ${h.id}" to land`;
88 }
89
90 /** Does this pull's text say it is the hold's cause? Only `release hold N` does; a mere mention of the version does not. */
91 export function names(hold: Hold, text: string): boolean {
92 return new RegExp(`(^|[^0-9a-z])release hold #?${hold.id}(?![0-9])`, "i").test(text);
93 }
94
95 export function setCause(db: Database, id: number, pull: number): void {
96 db.query(`UPDATE keeper_hold SET cause_pull = ?2 WHERE id = ?1 AND cause_pull IS NULL`).run(id, pull);
97 }
98
99 export function lift(db: Database, id: number, by: string, at: number): boolean {
100 return db.query(`UPDATE keeper_hold SET lifted_at = ?3, lifted_by = ?2 WHERE id = ?1 AND lifted_at IS NULL`).run(id, by, new Date(at).toISOString())
101 .changes === 1;
102 }
103
104 /** A verified release lifts every failed hold whose cause has landed. */
105 export function liftFixed(db: Database, version: string, at: number): number {
106 return db
107 .query(`UPDATE keeper_hold SET lifted_at = ?2, lifted_by = ?1 WHERE kind = 'failed' AND cause_pull IS NOT NULL AND lifted_at IS NULL`)
108 .run(`verified ${version}`, new Date(at).toISOString()).changes;
109 }
110
111 /** Every hold, newest first. */
112 export function listHolds(db: Database, limit = 20): Hold[] {
113 return db.query<Row, [number]>(`SELECT * FROM keeper_hold ORDER BY id DESC LIMIT ?1`).all(limit).map(toHold);
114 }
src/keeper/keeper.ts added +226
@@ -0,0 +1,226 @@
1 /**
2 * The release keeper (#7557): plain code, no model, on the forge's own
3 * timer, so releases keep going out when no one is at the keyboard. It is
4 * the steward's `release.sh`, `verify-after.sh` and `verify-release.sh`
5 * moved inside:
6 *
7 * 1. **Cut** on cadence (`cadence.ts`): the known-good version and image
8 * are recorded, then the annotated tag `v<version>` is made on main's
9 * head — a tag, never a commit, so no gate in flight is voided (#6640).
10 * 2. **Promote**: the Deployment is set to the new version's image; it
11 * rolls with surge (`maxSurge 1, maxUnavailable 0`). The old forge
12 * probes `/healthz` each second while it still serves (`rollProbe`),
13 * so a roll that went dark says how long.
14 * 3. **Verify**: the new forge, finding itself the version in flight,
15 * spends ten minutes under page load (`verify.ts`).
16 * 4. **Verified live**, or **roll back** to the named known-good image
17 * (`rollback.ts`; there is no undo) and **hold** with the numbers
18 * (`hold.ts`). A release with seconds dark files one issue.
19 *
20 * Each step is an event on the outbox: `release.cut`, `release.released`,
21 * `release.verified_live`, `release.rolled_back`, `release.held`.
22 *
23 * Guardrail: the keeper's only writes are a tag, the Deployment's image,
24 * its own rows, and issues. It holds no account that may land a pull or
25 * push main, and it has no code path to either.
26 */
27
28 import type { Database } from "bun:sqlite";
29 import type { EventBus } from "../core/events.ts";
30 import { createIssue } from "../core/issues.ts";
31 import { decideCut } from "./cadence.ts";
32 import { imageAt, type Deployer } from "./deploy.ts";
33 import { blocking, liftFixed, names, setCause, setHold, standingHolds, type Hold } from "./hold.ts";
34 import { countRollProbe, dropCut, finish, getKeeperRelease, inFlight, keepTally, listKeeperReleases, markLive, recordCut, type KeeperRelease } from "./record.ts";
35 import { rollBackToNamed } from "./rollback.ts";
36 import type { Probe, Tally, Verdict } from "./verify.ts";
37
38 /** A release not live this long after its cut never rolled: 3 h (the tag's check may take 2 h, the build some more). */
39 export const ROLL_DEADLINE_MS = 3 * 3_600_000;
40
41 export interface KeeperPorts {
42 readonly db: Database;
43 readonly bus: EventBus;
44 /** The forge's own repository, for events and the issues the keeper files. */
45 readonly repo: { readonly id: string; readonly ownerId: string };
46 readonly now: () => number;
47 /** The version this process runs. */
48 readonly live: () => string;
49 readonly main: () => Promise<string | undefined>;
50 readonly newest: () => Promise<{ version: string; sha: string } | undefined>;
51 /** Merge-queue gates running now. */
52 readonly gating: () => number;
53 readonly deployer: Deployer;
54 /** Make the annotated tag `v<version>` on `sha` and send it where the release workflow starts; why not. */
55 readonly tag: (version: string, sha: string) => Promise<string | undefined>;
56 /** One `/healthz` probe through the public URL. */
57 readonly probe: () => Promise<Probe>;
58 /** Ten minutes under page load. */
59 readonly verify: (onProbe: (t: Tally) => void) => Promise<Verdict>;
60 readonly minIntervalMs?: number;
61 }
62
63 /** Pulls of the forge's repository merged since `since`: number, title and body. */
64 function landedSince(db: Database, repoId: string, since: string): { number: number; text: string }[] {
65 return db

The comment says 'unless one by its title is open' but this count has no state filter. Closed issues also suppress the report.

66 .query<{ number: number; title: string; body: string | null }, [string, string]>(
67 `SELECT issue.number, issue.title, issue.body FROM pull JOIN issue ON issue.id = pull.issue_id
68 WHERE issue.repo_id = ?1 AND pull.merged_at IS NOT NULL AND pull.merged_at > ?2 ORDER BY pull.merged_at LIMIT 500`,
69 )
70 .all(repoId, since)
71 .map((r) => ({ number: r.number, text: `${r.title}\n${r.body ?? ""}` }));
72 }
73
74 /** Name the landed pull that fixes each failed hold. */
75 function findCauses(p: KeeperPorts, holds: readonly Hold[]): void {
76 for (const h of holds) {
77 if (h.kind !== "failed" || h.causePull !== null) continue;
78 const fix = landedSince(p.db, p.repo.id, h.createdAt).find((x) => names(h, x.text));
79 if (fix !== undefined) setCause(p.db, h.id, fix.number);
80 }
81 }
82
83 /** One issue for a release that went dark, unless one by its title is open. */
84 function fileDark(p: KeeperPorts, r: KeeperRelease): number | undefined {
85 if (r.secondsDark <= 0) return undefined;
86 const title = `Release ${r.version} was dark for ${r.secondsDark} s`;
87 const open = p.db
88 .query<{ n: number }, [string, string]>(`SELECT COUNT(*) AS n FROM issue WHERE repo_id = ?1 AND title = ?2 AND state = 'open'`)
89 .get(p.repo.id, title)!.n;
90 if (open > 0) return undefined;
91 const made = createIssue(p.db, {
92 repoId: p.repo.id,
93 authorId: p.repo.ownerId,
94 title,
95 body:
96 `_Filed by the release keeper, not written by a person._\n\n` +
97 `${r.version} (${r.sha.slice(0, 12)}) did not answer \`/healthz\` with a 200 for ${r.secondsDark} probe-seconds: ` +
98 `${r.rollNon200} of ${r.rollProbes} during the roll, the rest of ${r.probes} under page load; worst ${r.worstMs} ms. ` +
99 `The target is 0 s: the forge rolls with \`maxSurge 1, maxUnavailable 0\`. ` +
100 `${r.stage === "rolled_back" ? `It was rolled back to ${r.knownGood.version}.` : "It stayed live."}\n\n` +
101 `Files: src/keeper/keeper.ts, charts/app/templates/deployment.yaml`,
102 });
103 if (!made.ok) return undefined;
104 p.bus.append({ kind: "issue.opened", repoId: p.repo.id, actorId: p.repo.ownerId, payload: { number: made.value.number } });
105 return made.value.number;
106 }
107
108 let verifying: number | undefined;
109
110 /** The verify of an in-flight release this process is the new version of, to its end. */
111 async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<string> {
112 if (verifying !== undefined) return `verifying ${r.version}`;
113 verifying = r.id;
114 try {
115 const verdict = await p.verify((t) => keepTally(p.db, r.id, t, r.rollNon200));
116 keepTally(p.db, r.id, verdict.tally, r.rollNon200);
117 const kept = getKeeperRelease(p.db, r.id)!;
118 const numbers = { version: r.version, probes: kept.probes, pages: kept.pages, worstMs: kept.worstMs, secondsDark: kept.secondsDark, rollNon200: kept.rollNon200 };
119 if (verdict.ok) {
120 finish(p.db, r.id, "verified", "", p.now());
121 const lifted = liftFixed(p.db, r.version, p.now());
122 p.bus.append({ kind: "release.verified_live", repoId: p.repo.id, payload: { ...numbers, holdsLifted: lifted } });
123 fileDark(p, getKeeperRelease(p.db, r.id)!);
124 return `${r.version} verified live: ${kept.probes} probes, worst ${kept.worstMs} ms, ${kept.secondsDark} s dark`;
125 }
126 return await rollBack(p, kept, verdict.reason);
127 } finally {
128 verifying = undefined;
129 }
130 }
131
132 /** Hold with the numbers, then set the named known-good image. The hold first: this process may be the one replaced. */
133 async function rollBack(p: KeeperPorts, r: KeeperRelease, reason: string): Promise<string> {
134 const hold = setHold(p.db, { kind: "failed", version: r.version, reason, probes: r.probes, badInRow: r.badInRow, worstMs: r.worstMs, at: p.now() });
135 p.bus.append({ kind: "release.held", repoId: p.repo.id, payload: { version: r.version, hold: hold.id, reason, probes: r.probes, badInRow: r.badInRow, worstMs: r.worstMs } });
136 const back = await rollBackToNamed(p.deployer, r.knownGood);
137 const line = back.ok ? `rolled back to ${r.knownGood.version} (${back.image})` : `rollback failed: ${back.line}`;
138 finish(p.db, r.id, "rolled_back", `${reason}; ${line}`, p.now());
139 p.bus.append({ kind: "release.rolled_back", repoId: p.repo.id, payload: { version: r.version, to: r.knownGood.version, image: r.knownGood.image, ok: back.ok } });
140 fileDark(p, getKeeperRelease(p.db, r.id)!);
141 return `${r.version} failed its verify: ${reason}; ${line}; hold ${hold.id} set`;
142 }
143
144 /** One step of the keeper; the line says what it did or why it did nothing. */
145 export async function tick(p: KeeperPorts): Promise<string> {
146 const holds = standingHolds(p.db);
147 findCauses(p, holds);
148 const flying = inFlight(p.db);
149 if (flying !== undefined) {
150 if (p.live() === flying.version) {
151 markLive(p.db, flying.id, p.now());
152 return verifyAndDecide(p, getKeeperRelease(p.db, flying.id)!);
153 }
154 // A cut recorded but never tagged (the process died between the two): tag it now, or drop it.
155 if (flying.stage === "rolling" && (await p.newest())?.version !== flying.version) {
156 const refused = await tagged(p, flying);
157 if (refused !== undefined) return refused;
158 }
159 if (flying.stage === "rolling" && p.now() - Date.parse(flying.cutAt) > ROLL_DEADLINE_MS)
160 return rollBack(p, flying, `${flying.version} was not live ${ROLL_DEADLINE_MS / 3_600_000} h after its cut`);
161 // A promote that did not take (the API refused, or something set the image back) is sent again.
162 const want = imageAt(flying.knownGood.image, flying.version);
163 if (flying.stage === "rolling" && (await p.deployer.image()) !== want) await promote(p, flying.version, want);
164 return `${flying.version} is ${flying.stage}`;
165 }
166 const main = await p.main();
167 const newest = await p.newest();
168 const last = listKeeperReleases(p.db, 1)[0];
169 const decision = decideCut({
170 main: main ?? "",
171 newest,
172 live: p.live(),
173 settled: newest !== undefined && rolledBack(p.db, newest.version),
174 inFlight: false,
175 gating: p.gating(),
176 lastCutAt: last === undefined ? undefined : Date.parse(last.cutAt),
177 hold: blocking(standingHolds(p.db)),
178 now: p.now(),
179 ...(p.minIntervalMs === undefined ? {} : { minIntervalMs: p.minIntervalMs }),
180 });
181 if (!decision.cut || main === undefined) return decision.cut ? "main is unreadable" : decision.why;
outdated

Tag is made and pushed here, before recordCut below. A crash between them strands the release (see review body, point 2). Record first, tag second.

182 const image = await p.deployer.image();
183 const knownGood = { version: p.live(), image };
184 // The row first, the tag second: a crash between them leaves a `rolling` row
185 // whose tag the next tick sends (`tagged`), never a tag the keeper has no row for.
186 const r = recordCut(p.db, { version: decision.version, sha: main, knownGood, at: p.now() });
187 const refused = await tagged(p, r);
188 if (refused !== undefined) return refused;
189 const sent = await promote(p, r.version, imageAt(image, r.version));
190 return `cut ${r.version} at ${main.slice(0, 12)}; ${sent}; known good: ${knownGood.version}`;
191 }
192
193 /** Make the release's tag; when it is refused the cut is dropped and the line says why. */
194 async function tagged(p: KeeperPorts, r: KeeperRelease): Promise<string | undefined> {
195 const refused = await p.tag(r.version, r.sha);
196 if (refused !== undefined) {
197 dropCut(p.db, r.id);
198 return `not cut: ${refused}`;
199 }
200 p.bus.append({ kind: "release.cut", repoId: p.repo.id, payload: { version: r.version, sha: r.sha, knownGood: r.knownGood.version, image: r.knownGood.image } });
201 return undefined;
202 }
203
204 /** Set the Deployment to the new version's image; it rolls with surge. */
205 async function promote(p: KeeperPorts, version: string, image: string): Promise<string> {
206 try {
207 await p.deployer.setImage(image);
208 } catch (error) {
209 return `promote to ${image} refused (${error instanceof Error ? error.message : String(error)}); sent again next tick`;
210 }
211 p.bus.append({ kind: "release.released", repoId: p.repo.id, payload: { version, image } });
212 return `promoted to ${image}`;
213 }
214
215 /** The keeper rolled this version back: its tag is settled, not still promoting. */
216 function rolledBack(db: Database, version: string): boolean {
217 return db.query<{ n: number }, [string]>(`SELECT COUNT(*) AS n FROM keeper_release WHERE version = ?1 AND stage = 'rolled_back'`).get(version)!.n > 0;
218 }
219
220 /** The old forge's probe while a release rolls: one `/healthz`, counted toward the roll. */
221 export async function rollProbe(p: Pick<KeeperPorts, "db" | "live" | "probe">): Promise<void> {
222 const r = inFlight(p.db);
223 if (r === undefined || r.stage !== "rolling" || p.live() === r.version) return;
224 const got = await p.probe();
225 countRollProbe(p.db, r.id, got.status === 200);
226 }
src/keeper/record.ts added +139
@@ -0,0 +1,139 @@
1 /**
2 * The keeper's releases (#7557): one row per version it cut, from the cut
3 * to `verified` or `rolled_back`, with the known-good image recorded before
4 * the cut and what the probes counted. Plain rows in, plain rows out.
5 *
6 * Stages: `rolling` (tagged and promoted; the old forge still serves),
7 * `verifying` (the new version answers; ten minutes under page load),
8 * `verified` (verified live), `rolled_back` (set back to the known-good
9 * image, a hold stands).
10 */
11
12 import type { Database } from "bun:sqlite";
13 import type { Tally } from "./verify.ts";
14
15 export type Stage = "rolling" | "verifying" | "verified" | "rolled_back";
16 export const IN_FLIGHT: readonly Stage[] = ["rolling", "verifying"];
17
18 export interface KeeperRelease {
19 readonly id: number;
20 readonly version: string;
21 readonly sha: string;
22 readonly knownGood: { readonly version: string; readonly image: string };
23 readonly stage: Stage;
24 readonly cutAt: string;
25 readonly liveAt: string | null;
26 readonly finishedAt: string | null;
27 readonly rollProbes: number;
28 readonly rollNon200: number;
29 readonly probes: number;
30 readonly pages: number;
31 readonly badInRow: number;
32 readonly worstMs: number;
33 readonly secondsDark: number;
34 readonly reason: string;
35 }
36
37 interface Row {
38 id: number;
39 version: string;
40 sha: string;
41 known_good_version: string;
42 known_good_image: string;
43 stage: Stage;
44 cut_at: string;
45 live_at: string | null;
46 finished_at: string | null;
47 roll_probes: number;
48 roll_non200: number;
49 probes: number;
50 pages: number;
51 bad_in_row: number;
52 worst_ms: number;
53 seconds_dark: number;
54 reason: string;
55 }
56
57 const toRelease = (r: Row): KeeperRelease => ({
58 id: r.id,
59 version: r.version,
60 sha: r.sha,
61 knownGood: { version: r.known_good_version, image: r.known_good_image },
62 stage: r.stage,
63 cutAt: r.cut_at,
64 liveAt: r.live_at,
65 finishedAt: r.finished_at,
66 rollProbes: r.roll_probes,
67 rollNon200: r.roll_non200,
68 probes: r.probes,
69 pages: r.pages,
70 badInRow: r.bad_in_row,
71 worstMs: r.worst_ms,
72 secondsDark: r.seconds_dark,
73 reason: r.reason,
74 });
75
76 const iso = (ms: number) => new Date(ms).toISOString();
77
78 /** The cut, written before the tag is pushed: the known-good image is on record before anything can need it. */
79 export function recordCut(
80 db: Database,
81 input: { version: string; sha: string; knownGood: { version: string; image: string }; at: number },
82 ): KeeperRelease {
83 return toRelease(
84 db
85 .query<Row, [string, string, string, string, string]>(
86 `INSERT INTO keeper_release (version, sha, known_good_version, known_good_image, stage, cut_at)
87 VALUES (?1, ?2, ?3, ?4, 'rolling', ?5) RETURNING *`,
88 )
89 .get(input.version, input.sha, input.knownGood.version, input.knownGood.image, iso(input.at))!,
90 );
91 }
92
93 /** A cut whose tag was refused never happened: its row goes, so the cadence is not left waiting on it. */
94 export function dropCut(db: Database, id: number): void {
95 db.query(`DELETE FROM keeper_release WHERE id = ?1 AND stage = 'rolling' AND live_at IS NULL`).run(id);
96 }
97
98 /** One more `/healthz` probe while the roll runs. */
99 export function countRollProbe(db: Database, id: number, ok: boolean): void {
100 db.query(
101 `UPDATE keeper_release SET roll_probes = roll_probes + 1, roll_non200 = roll_non200 + ?2,
102 seconds_dark = seconds_dark + ?2 WHERE id = ?1 AND stage = 'rolling'`,
103 ).run(id, ok ? 0 : 1);
104 }
105
106 /** The new version answers: the verify begins. Only one forge moves it, so a second caller gets false. */
107 export function markLive(db: Database, id: number, at: number): boolean {
108 return db.query(`UPDATE keeper_release SET stage = 'verifying', live_at = ?2 WHERE id = ?1 AND stage = 'rolling'`).run(id, iso(at))
109 .changes === 1;
110 }
111
112 /** The verify's numbers so far: the seconds dark it saw are added to the roll's. */
113 export function keepTally(db: Database, id: number, t: Tally, rollDark: number): void {
114 db.query(
115 `UPDATE keeper_release SET probes = ?2, pages = ?3, bad_in_row = ?4, worst_ms = ?5, seconds_dark = ?6 WHERE id = ?1`,
116 ).run(id, t.probes, t.pages, t.worstRun, t.worstMs, rollDark + t.secondsDark);
117 }
118
119 export function finish(db: Database, id: number, stage: "verified" | "rolled_back", reason: string, at: number): void {
120 db.query(`UPDATE keeper_release SET stage = ?2, reason = ?3, finished_at = ?4 WHERE id = ?1`).run(id, stage, reason, iso(at));
121 }
122
123 /** Newest first. */
124 export function listKeeperReleases(db: Database, limit = 20): KeeperRelease[] {
125 return db.query<Row, [number]>(`SELECT * FROM keeper_release ORDER BY id DESC LIMIT ?1`).all(limit).map(toRelease);
126 }
127
128 /** The release the keeper has not finished with, if any. */
129 export function inFlight(db: Database): KeeperRelease | undefined {
130 const row = db
131 .query<Row, []>(`SELECT * FROM keeper_release WHERE stage IN ('rolling', 'verifying') ORDER BY id DESC LIMIT 1`)
132 .get();
133 return row === null ? undefined : toRelease(row);
134 }
135
136 export function getKeeperRelease(db: Database, id: number): KeeperRelease | undefined {
137 const row = db.query<Row, [number]>(`SELECT * FROM keeper_release WHERE id = ?1`).get(id);
138 return row === null ? undefined : toRelease(row);
139 }
src/keeper/rollback.ts added +26
@@ -0,0 +1,26 @@
1 /**
2 * The one rollback the release keeper has (#7557): set the Deployment to
3 * the known-good image recorded before the cut, by name. Never "the
4 * previous revision": that is whatever the controller last held, and on
5 * 2026-10-27 it was the broken version. This is the one action the keeper
6 * takes without a canary, because the image it sets was already live.
7 */
8
9 import { isImage, type Deployer } from "./deploy.ts";
10
11 export interface KnownGood {
12 readonly version: string;
13 readonly image: string;
14 }
15
16 export type RolledBack = { readonly ok: true; readonly image: string } | { readonly ok: false; readonly line: string };
17
18 export async function rollBackToNamed(deployer: Deployer, good: KnownGood): Promise<RolledBack> {
19 if (!isImage(good.image)) return { ok: false, line: `the recorded known-good image of ${good.version} is not an image reference` };
20 try {
21 await deployer.setImage(good.image);
22 return { ok: true, image: good.image };
23 } catch (error) {
24 return { ok: false, line: `could not set ${good.image}: ${error instanceof Error ? error.message : String(error)}` };
25 }
26 }
src/keeper/verify.ts added +136
@@ -0,0 +1,136 @@
1 /**
2 * Proving a release live (#7557), as the steward's `verify-release.sh` did:
3 * for ten minutes, the pages people open are fetched one at a time as an
4 * admin (/boards, the forge's repository, /explore, /) while `/healthz` is
5 * probed once a second. A probe is bad when it is not a 200 or takes over
6 * 10 s; three bad probes in a row fail the release.
7 *
8 * Each probe stands for one second, so the seconds dark are the probes
9 * that got no 200 — a slow 200 is slow, not dark. The same tally counts
10 * the roll (`roll: true`), where the target is zero with surge.
11 *
12 * `tally` is plain data in, plain data out. `verifyUnderLoad` drives it
13 * with the fetch, clock and sleep it is handed, so a test runs ten minutes
14 * of probes in no time.
15 */
16
17 /** A probe slower than this is bad: 10 s. */
18 export const SLOW_MS = 10_000;
19 /** This many bad probes in a row fail a release. */
20 export const FAIL_AFTER = 3;
21 /** How long a release is held under page load: 10 minutes. */
22 export const VERIFY_MS = 10 * 60_000;
23 /** One `/healthz` probe a second. */
24 export const PROBE_EVERY_MS = 1000;
25
26 export interface Probe {
27 /** The HTTP status; 0 when nothing answered. */
28 readonly status: number;
29 readonly ms: number;
30 }
31
32 export interface Tally {
33 readonly probes: number;
34 readonly non200: number;
35 /** Bad probes in a row now, and the most there have been. */
36 readonly badInRow: number;
37 readonly worstRun: number;
38 readonly worstMs: number;
39 readonly secondsDark: number;
40 /** Pages fetched under load, and how many of them were not a 200. */
41 readonly pages: number;
42 readonly pagesBad: number;
43 }
44
45 export const EMPTY: Tally = { probes: 0, non200: 0, badInRow: 0, worstRun: 0, worstMs: 0, secondsDark: 0, pages: 0, pagesBad: 0 };
46
47 export const isBad = (p: Probe): boolean => p.status !== 200 || p.ms > SLOW_MS;
48
49 /** The tally with one more `/healthz` probe. */
50 export function tally(t: Tally, p: Probe): Tally {
51 const bad = isBad(p);
52 const badInRow = bad ? t.badInRow + 1 : 0;
53 return {
54 ...t,
55 probes: t.probes + 1,
56 non200: t.non200 + (p.status === 200 ? 0 : 1),
57 badInRow,
58 worstRun: Math.max(t.worstRun, badInRow),
59 worstMs: Math.max(t.worstMs, Math.round(p.ms)),
60 secondsDark: t.secondsDark + (p.status === 200 ? 0 : 1),
61 };
62 }
63
64 /** The tally with one more page fetched under load; its latency counts toward the worst. */
65 export function tallyPage(t: Tally, p: Probe): Tally {
66 return { ...t, pages: t.pages + 1, pagesBad: t.pagesBad + (isBad(p) ? 1 : 0), worstMs: Math.max(t.worstMs, Math.round(p.ms)) };
67 }
68
69 export const failed = (t: Tally): boolean => t.worstRun >= FAIL_AFTER;
70
71 /** Why a failed tally failed, with its numbers, in one line. */
72 export function failure(t: Tally): string {
73 return `${t.worstRun} bad /healthz probes in a row (non-200 or over ${SLOW_MS / 1000} s) after ${t.probes} probes; ${t.non200} non-200, worst ${t.worstMs} ms`;
74 }
75
76 export interface VerifyPorts {
77 /** One GET: the status (0 when nothing answered) and how long it took. */
78 readonly get: (path: string, asAdmin: boolean) => Promise<Probe>;
79 readonly now: () => number;
80 readonly sleep: (ms: number) => Promise<void>;
81 /** The pages fetched in turn, one at a time. */
82 readonly pages: readonly string[];
83 readonly durationMs?: number;
84 /** Told after each probe, so the record keeps up while the verify runs. */
85 readonly onProbe?: (t: Tally) => void;
86 }
87
88 export type Verdict = { readonly ok: boolean; readonly tally: Tally; readonly reason: string };
89
90 /** Probe `/healthz` each second under page load until the time is up or three bad probes in a row fail it. */
91 export async function verifyUnderLoad(ports: VerifyPorts, start: Tally = EMPTY): Promise<Verdict> {
92 const end = ports.now() + (ports.durationMs ?? VERIFY_MS);
93 let t = start;
94 let stop = false;
95 // The load: one page at a time, the next when the last has answered.
96 const load = (async () => {
97 for (let i = 0; !stop && ports.pages.length > 0; i++) {
98 const page = await ports.get(ports.pages[i % ports.pages.length]!, true);
99 t = tallyPage(t, page);
100 if (ports.now() >= end) break;
101 }
102 })();
103 while (ports.now() < end) {
104 const began = ports.now();
105 t = tally(t, await ports.get("/healthz", false));
106 ports.onProbe?.(t);
107 if (failed(t)) break;
108 await ports.sleep(Math.max(0, PROBE_EVERY_MS - (ports.now() - began)));
109 }
110 stop = true;
111 await load;
112 return failed(t) ? { ok: false, tally: t, reason: failure(t) } : { ok: true, tally: t, reason: "" };
113 }
114
115 /** A GET through `fetch` that never throws: a refused connection or a timeout is status 0. */
116 export function timedGet(
117 base: string,
118 headers: (asAdmin: boolean) => Record<string, string>,
119 fetcher: typeof fetch = fetch,
120 now: () => number = () => performance.now(),
121 ): (path: string, asAdmin: boolean) => Promise<Probe> {
122 return async (path, asAdmin) => {
123 const began = now();
124 try {
125 const res = await fetcher(new URL(path, base), {
126 headers: headers(asAdmin),
127 redirect: "manual",
128 signal: AbortSignal.timeout(SLOW_MS + 5000),
129 });
130 await res.arrayBuffer();
131 return { status: res.status, ms: now() - began };
132 } catch {
133 return { status: 0, ms: now() - began };
134 }
135 };
136 }
src/keeper/wire.ts added +130
@@ -0,0 +1,130 @@
1 /**
2 * The release keeper's ports, from a running forge (#7557): the forge's own
3 * repository for main and the tags, the merge queue for "mid-landing", the
4 * kube API for the Deployment, and the public URL for the probes.
5 *
6 * On when `FORGE_RELEASE_KEEPER=on`, `FORGE_REPO` names a repository here,
7 * and the forge runs in a cluster. The Deployment and its container are
8 * `FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER` (both default
9 * `forge`); the pod's service account needs `get` and `patch` on it.
10 */
11
12 import type { App } from "../app.ts";
13 import { originPush } from "../actions/branches.ts";
14 import { serviceAccount } from "../cluster/jobs.ts";
15 import { getRepo, type Repo } from "../core/repos.ts";
16 import { createSession, destroySession } from "../core/sessions.ts";
17 import { getUserById, type User } from "../core/users.ts";
18 import { gatesInFlight } from "../checks/queue.ts";
19 import { migrationFiles, NUMBERED } from "../core/schema.ts";
20 import { runGit } from "../git/env.ts";
21 import { repoDir } from "../git/repos.ts";
22 import { ownProbeHeaders } from "../kernel/probe.ts";
23 import { running } from "../kernel/version.ts";
24 import { compareVersions, readTags, recordFor } from "../release/shipped.ts";
25 import { kubeDeployer } from "./deploy.ts";
26 import type { KeeperPorts } from "./keeper.ts";
27 import { timedGet, verifyUnderLoad } from "./verify.ts";
28
29 const LABEL = /^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$/;
30 const OID = /^[0-9a-f]{40}$/;
31 const VERSION = /^\d+\.\d+\.\d+$/;
32 const ZERO = "0".repeat(40);
33
34 export type KeeperEnv = { on: true; deployment: string; container: string } | { on: false; why: string };
35
36 /** Whether the keeper runs, from the environment. */
37 export function keeperEnv(env: Record<string, string | undefined>): KeeperEnv {
38 if (env.FORGE_RELEASE_KEEPER !== "on") return { on: false, why: "FORGE_RELEASE_KEEPER is not on" };
39 if (!env.KUBERNETES_SERVICE_HOST) return { on: false, why: "not running in a cluster" };
40 const deployment = env.FORGE_KEEPER_DEPLOYMENT || "forge";
41 const container = env.FORGE_KEEPER_CONTAINER || "forge";
42 if (!LABEL.test(deployment) || !LABEL.test(container)) return { on: false, why: "FORGE_KEEPER_DEPLOYMENT and FORGE_KEEPER_CONTAINER must be DNS labels" };
43 return { on: true, deployment, container };
44 }
45
46 /** The first active site admin that is a person: the account the page load signs in as. */
47 function anAdmin(app: App): User | undefined {
48 const row = app.db
49 .query<{ id: string }, []>(`SELECT id FROM user WHERE is_admin = 1 AND is_agent = 0 ORDER BY created_at LIMIT 1`)
50 .get();
51 return row === null ? undefined : getUserById(app.db, row.id);
52 }
53
54 /** The pages a release is verified under, one at a time: the ones people open. */
55 export function loadPages(repo: Pick<Repo, "ownerName" | "name">): string[] {
56 return ["/boards", `/${encodeURIComponent(repo.ownerName)}/${encodeURIComponent(repo.name)}`, "/explore", "/"];
57 }
58
59 export function keeperPorts(app: App, input: { forgeUrl: string; deployment: string; container: string }): KeeperPorts | string {
60 const named = app.config.forgeRepo;
61 if (named === undefined) return "FORGE_REPO is not set";
62 const [owner = "", name = ""] = named.split("/");
63 const repo = getRepo(app.db, owner, name);
64 if (repo === undefined) return `FORGE_REPO names ${named}, which is no repository on this forge`;
65 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
66 const deployer = kubeDeployer({ credentials: serviceAccount(), deployment: input.deployment, container: input.container });
67 const probeHeaders = ownProbeHeaders(app.config.dataDir);
68 const at = (version: string) => runGit(["rev-parse", "--verify", "--quiet", `refs/tags/v${version}^{commit}`], { cwd: dir });
69 return {
70 db: app.db,
71 bus: app.bus,
72 repo: { id: repo.id, ownerId: repo.ownerId },
73 now: () => Date.now(),
74 live: () => running().version,
75 async main() {
76 const head = await runGit(["rev-parse", "--verify", "--quiet", `refs/heads/${repo.defaultBranch}^{commit}`], { cwd: dir });
77 return head.ok && OID.test(head.value) ? head.value : undefined;
78 },
79 async newest() {
80 const tags = await readTags(dir);
81 if (!tags.ok) return undefined;
82 const top = [...tags.value].filter((t) => VERSION.test(t.version)).sort((a, b) => compareVersions(b.version, a.version))[0];
83 if (top === undefined) return undefined;
84 const sha = await at(top.version);
85 return sha.ok && OID.test(sha.value) ? { version: top.version, sha: sha.value } : undefined;
86 },
87 gating: gatesInFlight,
88 deployer,
89 async tag(version, sha) {
90 if (!VERSION.test(version) || !OID.test(sha)) return "not a version and a commit";
91 // The record `scripts/cut-release.ts` writes. The migrations are this process's: the keeper cuts only
92 // when main has moved past a live tag, and each forge cuts the code it was built from at the earliest.
93 const record = JSON.stringify(recordFor(version, migrationFiles, NUMBERED));
94 const made = await runGit(["tag", "--annotate", "--file=-", `v${version}`, sha], {
95 cwd: dir,
96 stdin: `${record}\n`,
97 env: { GIT_COMMITTER_NAME: "release keeper", GIT_COMMITTER_EMAIL: "[email protected]" },
98 });
99 if (!made.ok) return made.error.stderr || "git refused the tag";
100 const tagOid = await runGit(["rev-parse", `refs/tags/v${version}`], { cwd: dir });
101 const owner = getUserById(app.db, repo.ownerId);
102 if (!tagOid.ok || owner === undefined) return undefined;
103 // A tag the origin did not take is taken back here too: the keeper drops the cut, and a
104 // tag left behind would be the newest, never live, and stall the cadence.
105 const untag = async (why: string) => (await runGit(["tag", "--delete", `v${version}`], { cwd: dir }), why);
106 const push = await originPush(app, owner, repo);
107 if (!push.ok) return untag(`the origin refused v${version}: ${push.error}`);
108 if (push.value === undefined) return undefined;
109 const sent = await push.value([{ ref: `refs/tags/v${version}`, oldOid: ZERO, newOid: tagOid.value }]);
110 return sent.ok ? undefined : untag(`pushing v${version} to the origin failed: ${sent.error}`);
111 },
112 probe: () => timedGet(input.forgeUrl, () => probeHeaders)("/healthz", false),
113 async verify(onProbe) {
114 const admin = anAdmin(app);
115 const session = admin === undefined ? undefined : createSession(app.db, admin.id);
116 const cookie = session === undefined ? {} : { cookie: `forge_session=${session.token}` };
117 try {
118 return await verifyUnderLoad({
119 get: timedGet(input.forgeUrl, (asAdmin) => (asAdmin ? cookie : probeHeaders)),
120 now: () => Date.now(),
121 sleep: (ms) => Bun.sleep(ms),
122 pages: loadPages(repo),
123 onProbe,
124 });
125 } finally {
126 if (session !== undefined) destroySession(app.db, session.token);
127 }
128 },
129 };
130 }
src/main.ts +22
@@ -68,6 +68,8 @@ import { vitals } from "./kernel/vitals.ts";
6868 import { proveRelease } from "./release/prove.ts";
6969 import { CHECK_EVERY_MS, setBehind, watchBehind } from "./release/behind.ts";
7070 import { ownTagsReader } from "./release/own_tags.ts";
71 import { rollProbe, tick as keeperTick } from "./keeper/keeper.ts";
72 import { keeperEnv, keeperPorts } from "./keeper/wire.ts";
7173 import { snapshotPlace } from "./actions/health.ts";
7274 import { capacityDefault } from "./core/capacity.ts";
7375 import { bootCounts, recordBoot, recordCleanExit } from "./kernel/boots.ts";
@@ -620,6 +622,26 @@ if (app.config.forgeRepo) {
620622 void watch.tick();
621623 setInterval(() => void watch.tick(), CHECK_EVERY_MS);
622624 }
625 // The release keeper (#7557): cut on cadence, promote, verify under page load, roll back to the named
626 // known-good image and hold. Off unless FORGE_RELEASE_KEEPER=on in a cluster (keeper/wire.ts).
627 const keeperOn = keeperEnv(process.env);
628 if (keeperOn.on) {
629 const ports = keeperPorts(app, { forgeUrl, deployment: keeperOn.deployment, container: keeperOn.container });
630 if (typeof ports === "string") app.log.warn("the release keeper is off", { why: ports });
631 else {
632 let said = "";
633 const step = () =>
634 void keeperTick(ports)
635 .then((line) => {
636 if (line !== said) app.log.info("release keeper", { line });
637 said = line;
638 })
639 .catch((error) => app.log.warn("release keeper step failed", { error: String(error) }));
640 step();
641 setInterval(step, 60_000);
642 setInterval(() => void rollProbe(ports).catch(() => {}), 1000);
643 }
644 } else app.log.info("the release keeper is off", { why: keeperOn.why });
623645 // Only an agent is on a crew (the `repo_crew_agents_only` trigger refuses anyone else). A person
624646 // put on one before that is reported here for the owner to take off — the
625647 // forge never removes it on its own.
src/tui/forge.ts +9
@@ -78,6 +78,7 @@ import { noteRead, setTelemetryGrant, telemetryReader } from "../actions/telemet
7878 import { queuesFor, systemFor } from "../system/telemetry.ts";
7979 import { live as liveMachine } from "../system/live.ts";
8080 import { changelogScreen } from "./screens/changelog.ts";
81 import { keeperScreen } from "./screens/keeper.ts";
8182 import { releasesScreen } from "./screens/releases.ts";
8283 import { mayRead } from "../release/gate.ts";
8384 import { fileHoldsPrompt } from "./screens/fileholds.ts";
@@ -399,6 +400,14 @@ export function forgeCommands(session: Session): (ctx: Ctx) => Command[] {
399400 hint: "the live release and the kept ones",
400401 run: () => ctx.push(releasesScreen(session, ctx)),
401402 });
403 // The release keeper's card (#7557), gated as /releases is.
404 if (mayRead(session.app, session.user))
405 commands.push({
406 id: "keeper",
407 title: "Release keeper",
408 hint: "what holds the next cut, and each release it cut with its probes",
409 run: () => ctx.push(keeperScreen(session, ctx)),
410 });
402411 // Stalled flow steps the viewer can unblock, as the home's cards show them (#1807).
403412 if (session.signedIn)
404413 commands.push({
src/tui/screens/keeper.ts added +89
@@ -0,0 +1,89 @@
1 /**
2 * The release keeper in the terminal (#7557), the /releases card's read:
3 * what holds the next cut, then each release the keeper cut with its stage,
4 * the known-good version it rolls back to and what the probes counted.
5 * Gated by `release/gate.ts` as the page is. Read only, as the card is; an
6 * owner's hold is set with MCP `release_hold`.
7 */
8
9 import { blocking, standingHolds } from "../../keeper/hold.ts";
10 import { listKeeperReleases, type KeeperRelease, type Stage } from "../../keeper/record.ts";
11 import { mayRead } from "../../release/gate.ts";
12 import type { Ctx, Screen } from "../app/app.ts";
13 import { binding } from "../app/keys.ts";
14 import type { Session } from "../session.ts";
15 import { style } from "../term/style.ts";
16 import { truncate, wrap } from "../term/width.ts";
17 import { fill, len, split } from "../ui/layout.ts";
18 import { emptyState, type Tone } from "../ui/widgets.ts";
19 import { ListState, ago, drawList, metaOf, type ListRow } from "./common.ts";
20
21 const STAGE: Record<Stage, { label: string; tone: Tone }> = {
22 rolling: { label: "rolling", tone: "default" },
23 verifying: { label: "verifying", tone: "default" },
24 verified: { label: "verified live", tone: "success" },
25 rolled_back: { label: "rolled back", tone: "danger" },
26 };
27
28 export interface KeeperRead {
29 /** What holds the next cut, or the line that says nothing does. */
30 readonly blocked: string;
31 readonly releases: readonly KeeperRelease[];
32 }
33
34 /** The card's content for this person; undefined for anyone the page would 404. */
35 export function readKeeper(f: Session): KeeperRead | undefined {
36 if (!mayRead(f.app, f.user)) return undefined;
37 return { blocked: blocking(standingHolds(f.db)) ?? "Nothing holds the next cut.", releases: listKeeperReleases(f.db) };
38 }
39
40 export function keeperRows(releases: readonly KeeperRelease[]): ListRow[] {
41 return releases.map((r) => ({
42 title: r.version,
43 tone: STAGE[r.stage].tone,
44 meta: metaOf(
45 STAGE[r.stage].label,
46 `cut ${ago(r.cutAt)}`,
47 `known good ${r.knownGood.version}`,
48 `roll ${r.rollNon200}/${r.rollProbes} non-200`,
49 `${r.probes} probes, ${r.pages} pages`,
50 `worst ${r.worstMs} ms`,
51 `${r.secondsDark} s dark`,
52 ),
53 }));
54 }
55
56 export function keeperScreen(f: Session, ctx: Ctx): Screen {
57 const state = new ListState();
58 let read: KeeperRead | undefined;
59 return {
60 title: "Release keeper",
61 keys: [binding(["j", "k"], "move"), binding(["esc", "q"], "close")],
62 load: async () => {
63 read = readKeeper(f);
64 state.clamp(read?.releases.length ?? 0);
65 ctx.invalidate();
66 },
67 render(area, s) {
68 const c = ctx.theme.c;
69 if (read === undefined) {
70 emptyState(s, area, c, { title: "Not yours to see", body: "The release keeper is for the forge's administrators and its repository's readers." });
71 return;
72 }
73 // What holds the cut ends with what to land ("release hold N"): wrapped, up to three lines, never cut short of it.
74 const lines = wrap(read.blocked, area.w).slice(0, 3);
75 const [head, body] = split(area, "v", [len(lines.length + 1), fill(1)]);
76 lines.forEach((l, i) => s.text(head!.x, head!.y + i, truncate(l, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w));
77 if (read.releases.length === 0) {
78 emptyState(s, body!, c, { title: "No keeper releases yet", body: "The keeper cuts when main has moved past a live tag (FORGE_RELEASE_KEEPER=on)." });
79 return;
80 }
81 drawList(s, body!, c, state, keeperRows(read.releases));
82 },
83 key(e) {
84 if (e.key === "esc" || e.key === "q") ctx.pop();
85 else state.key(e, read?.releases.length ?? 0);
86 return true;
87 },
88 };
89 }
src/web/pages/keeper-panel.ts added +42
@@ -0,0 +1,42 @@
1 /**
2 * The release keeper on /releases (#7557): what stops the next cut, and
3 * each release it cut with its stage, the known-good version it rolls back
4 * to, and what the probes counted. Nothing when it has done nothing.
5 */
6
7 import type { Database } from "bun:sqlite";
8 import { blocking, standingHolds } from "../../keeper/hold.ts";
9 import { listKeeperReleases, type Stage } from "../../keeper/record.ts";
10 import { html, when, type Html } from "../html.ts";
11 import { badge, card, time, type Tone } from "../ui.ts";
12
13 const STAGE: Record<Stage, { label: string; tone: Tone }> = {
14 rolling: { label: "rolling", tone: "attn" },
15 verifying: { label: "verifying", tone: "attn" },
16 verified: { label: "verified live", tone: "success" },
17 rolled_back: { label: "rolled back", tone: "danger" },
18 };
19
20 export function keeperPanel(db: Database): Html | undefined {
21 const releases = listKeeperReleases(db);
22 const blocked = blocking(standingHolds(db));
23 if (releases.length === 0 && blocked === undefined) return undefined;
24 const rows = releases.map((r) => {
25 const s = STAGE[r.stage];
26 return html`<tr>
27 <td>${r.version} ${badge(s.label, { tone: s.tone })}</td>
28 <td>${time(r.cutAt)}</td>
29 <td>${r.knownGood.version}</td>
30 <td>${r.rollNon200} of ${r.rollProbes}</td>
31 <td>${r.probes} / ${r.pages}</td>
32 <td>${r.worstMs} ms</td>
33 <td>${r.secondsDark} s</td>
34 </tr>${when(r.reason !== "", () => html`<tr><td colspan="7" class="small muted">${r.reason}</td></tr>`)}`;
35 });
36 return card(html`<h2>Release keeper</h2>
37 <p class="small ${blocked === undefined ? "muted" : ""}" id="keeper-blocked">${blocked ?? "Nothing holds the next cut."}</p>
38 ${when(rows.length > 0, () => html`<table class="table">
39 <thead><tr><th>Version</th><th>Cut</th><th>Known good</th><th>Roll non-200</th><th>Probes / pages</th><th>Worst</th><th>Dark</th></tr></thead>
40 <tbody>${rows}</tbody>
41 </table>`)}`);
42 }
src/web/pages/releases.ts +2 −1
@@ -19,6 +19,7 @@ import { readForm } from "../form.ts";
1919 import { html, when, type Html } from "../html.ts";
2020 import { flashTo, page } from "../layout.ts";
2121 import { badge, btn, card, empty, sha, time } from "../ui.ts";
22 import { keeperPanel } from "./keeper-panel.ts";
2223
2324 /** The live line (`liveSince`) for the home, or nothing when no release is known to be live. */
2425 export function liveLine(ctx: Ctx): Html | undefined {
@@ -81,7 +82,7 @@ export async function releasesPage(ctx: Ctx): Promise<Response> {
8182 const response = page({
8283 ctx,
8384 title: "Releases",
84 body: html`<div class="page-head"><h1>Releases</h1></div>${body}`,
85 body: html`<div class="page-head"><h1>Releases</h1></div>${keeperPanel(ctx.app.db)}${body}`,
8586 });
8687 notePhase(ctx.request, "render", performance.now() - rendering);
8788 return response;
test/boards-starting.test.ts +3 −3
@@ -36,10 +36,10 @@ test("a forge with no person to found the repo writes nothing", async () => {
3636 expect(await offerStartingBoards(bare)).toEqual([]);
3737 });
3838
39 test("a fresh boot writes the ten starting boards, and every panel runs", async () => {
40 expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
39 test("a fresh boot writes the eleven starting boards, and every panel runs", async () => {
40 expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "maintenance", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
4141 const boards = await listBoards(app.reposRoot);
42 expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
42 expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "maintenance", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
4343 let panels = 0;
4444 for (const { board } of boards) {
4545 for (const p of board.panels) {
test/helpers/keeper.ts added +102
@@ -0,0 +1,102 @@
1 /** A forge, its own repository and the release keeper's ports with a fake cluster and clock (#7557). */
2 import { mkdtempSync } from "node:fs";
3 import { tmpdir } from "node:os";
4 import { join } from "node:path";
5 import { createApp } from "../../src/app.ts";
6 import { createRepo } from "../../src/core/repos.ts";
7 import { createUser } from "../../src/core/users.ts";
8 import { id } from "../../src/kernel/id.ts";
9 import type { Deployer } from "../../src/keeper/deploy.ts";
10 import type { KeeperPorts } from "../../src/keeper/keeper.ts";
11 import { verifyUnderLoad, type Probe } from "../../src/keeper/verify.ts";
12
13 export const A = "a".repeat(40);
14 export const B = "b".repeat(40);
15 export const C = "c".repeat(40);
16
17 /** A Deployment that remembers every image set on it. Its revision history holds a broken image "undo" would bring back. */
18 export function fakeDeployer(image: string) {
19 const state = { image, set: [] as string[], history: ["reg.test/o/forge:0.7.277"] };
20 const deployer: Deployer = {
21 image: async () => state.image,
22 async setImage(next) {
23 state.history.push(state.image);
24 state.image = next;
25 state.set.push(next);
26 },
27 rolled: async (want) => state.image === want,
28 };
29 return { deployer, state };
30 }
31
32 export async function keeperForge(opts: { healthz?: (n: number) => Probe } = {}) {
33 const app = createApp({ dataDir: mkdtempSync(join(tmpdir(), "forge-keeper-")), port: 0, forgeRepo: "alice/forge" });
34 const u = await createUser(app.db, { username: "alice", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true, adminIfFirst: true });
35 if (!u.ok) throw new Error("user");
36 const repo = await createRepo(app.db, app.reposRoot, { ownerId: u.value.id, name: "forge", visibility: "public", description: "f" });
37 if (!repo.ok) throw new Error("repo");
38 const world = {
39 now: Date.parse("2026-10-27T12:00:00.000Z"),
40 live: "0.7.283",
41 main: B,
42 newest: { version: "0.7.283", sha: A } as { version: string; sha: string } | undefined,
43 gating: 0,
44 tags: [] as { version: string; sha: string }[],
45 /** When set, the tag is refused with this line. */
46 tagRefused: undefined as string | undefined,
47 /** Why the next tag is refused; undefined lets it through. */
48 refuseTag: undefined as string | undefined,
49 healthz: opts.healthz ?? (() => ({ status: 200, ms: 12 })),
50 pages: [] as string[],
51 };
52 const { deployer, state } = fakeDeployer("reg.test/o/forge:0.7.283");
53 let probes = 0;
54 const ports: KeeperPorts = {
55 db: app.db,
56 bus: app.bus,
57 repo: { id: repo.value.id, ownerId: u.value.id },
58 now: () => world.now,
59 live: () => world.live,
60 main: async () => world.main,
61 newest: async () => world.newest,
62 gating: () => world.gating,
63 deployer,
64 // As the forge's repository does: a tag made is the newest tag read back.
65 async tag(version, sha) {
66 if (world.refuseTag !== undefined) return world.refuseTag;
67 world.tags.push({ version, sha });
68 world.newest = { version, sha };
69 return undefined;
70 },
71 probe: async () => world.healthz(probes++),
72 verify: (onProbe) =>
73 verifyUnderLoad({
74 get: async (path) => {
75 if (path === "/healthz") {
76 const p = world.healthz(probes++);
77 world.now += p.ms;
78 return p;
79 }
80 world.pages.push(path);
81 world.now += 40;
82 return { status: 200, ms: 40 };
83 },
84 now: () => world.now,
85 sleep: async (ms) => {
86 world.now += ms;
87 },
88 pages: ["/boards", "/alice/forge", "/explore", "/"],
89 onProbe,
90 }),
91 };
92 /** A pull of the forge's repository, merged now. */
93 const land = (n: number, title: string, body = "") => {
94 const issueId = id();
95 const at = new Date(world.now).toISOString();
96 app.db.query(`INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at) VALUES (?1, ?2, ?3, 'pull', ?4, ?5, ?6, 'closed', ?7)`).run(issueId, repo.value.id, n, title, body, u.value.id, at);
97 app.db.query(`INSERT INTO pull (issue_id, base_branch, head_branch, base_oid, head_oid, head_repo_id, merged_at) VALUES (?1, 'main', ?2, '', '', ?3, ?4)`).run(issueId, `b/${n}`, repo.value.id, at);
98 };
99 const events = (kind: string) =>
100 app.db.query<{ payload: string }, [string]>(`SELECT payload FROM event WHERE kind = ?1 ORDER BY id`).all(kind).map((e) => JSON.parse(e.payload) as Record<string, unknown>);
101 return { app, user: u.value, repo: repo.value, world, ports, state, land, events };
102 }
test/mcp-write-scope.test.ts +1
@@ -31,6 +31,7 @@ const ADMIN_ARGS: Record<string, Record<string, unknown>> = {
3131 subscription_wake: { id: "u1" },
3232 subscription_headroom: { id: "u1", percent: 30 },
3333 prompts: { repo: "alice/game", action: "remove", id: "p1" },
34 release_hold: { reason: "freeze for the demo" },
3435 telemetry_grant: { agent: "robo", on: true },
3536 hooks: { repo: "alice/game", action: "add", url: "https://example.com/h", secret: "0123456789abcdef0123" },
3637 app_domains: { repo: "alice/game", action: "add", host: "game.example.com" },
test/mcp.test.ts +1
@@ -202,6 +202,7 @@ describe("the forge over MCP", () => {
202202 "edit",
203203 "close",
204204 "read_file",
205 "release_keeper",
205206 "write_file",
206207 "asks",
207208 "asks_act",
test/release-keeper-cadence.test.ts added +98
@@ -0,0 +1,98 @@
1 // When the release keeper cuts (#7557): each rule refuses on its own, and a cut is a tag, never a commit to main.
2 import { describe, expect, test } from "bun:test";
3 import { decideCut, MIN_INTERVAL_MS, nextVersion, type CadenceInput } from "../src/keeper/cadence.ts";
4 import { tick } from "../src/keeper/keeper.ts";
5 import { listKeeperReleases, recordCut } from "../src/keeper/record.ts";
6 import { A, B, keeperForge } from "./helpers/keeper.ts";
7
8 const NOW = Date.parse("2026-10-27T12:00:00.000Z");
9 const ready: CadenceInput = {
10 main: B,
11 newest: { version: "0.7.283", sha: A },
12 live: "0.7.283",
13 inFlight: false,
14 gating: 0,
15 lastCutAt: NOW - MIN_INTERVAL_MS,
16 hold: undefined,
17 now: NOW,
18 };
19
20 describe("cadence", () => {
21 test("the next version is the patch plus one", () => {
22 expect(nextVersion("0.7.283")).toBe("0.7.284");
23 expect(nextVersion("1.0.9")).toBe("1.0.10");
24 expect(nextVersion("0.7")).toBeUndefined();
25 });
26
27 test("main past a live tag, nothing in flight, the queue still, the interval passed, no hold: cut", () => {
28 expect(decideCut(ready)).toEqual({ cut: true, version: "0.7.284" });
29 });
30
31 test("each rule refuses alone, and says which", () => {
32 const why = (o: Partial<CadenceInput>) => {
33 const d = decideCut({ ...ready, ...o });
34 return d.cut ? "" : d.why;
35 };
36 expect(why({ main: A })).toBe("main is still 0.7.283");
37 expect(why({ inFlight: true })).toContain("still rolling");
38 expect(why({ live: "0.7.282" })).toBe("0.7.283 is not live yet (live: 0.7.282)");
39 expect(why({ gating: 2 })).toBe("the merge queue is landing (2 gates running)");
40 // One minute short of 30 is refused; exactly 30 passes (above).
41 expect(why({ lastCutAt: NOW - MIN_INTERVAL_MS + 60_000 })).toBe("the last cut was 29 min ago; the interval is 30 min");
42 expect(why({ hold: "owner: freeze for the demo" })).toBe("held: owner: freeze for the demo");
43 expect(why({ newest: undefined })).toContain("no release tag");
44 });
45
46 test("a tag the keeper rolled back is settled: the fix after it may be cut", () => {
47 expect(decideCut({ ...ready, newest: { version: "0.7.284", sha: A }, live: "0.7.283", settled: true })).toEqual({ cut: true, version: "0.7.285" });
48 });
49
50 test("the keeper's cut is a tag on main's head, and records the known-good version and image first", async () => {
51 const f = await keeperForge();
52 const line = await tick(f.ports);
53 expect(line).toContain("cut 0.7.284");
54 expect(f.world.tags).toEqual([{ version: "0.7.284", sha: B }]);
55 const [r] = listKeeperReleases(f.app.db);
56 expect(r).toMatchObject({ version: "0.7.284", sha: B, stage: "rolling", knownGood: { version: "0.7.283", image: "reg.test/o/forge:0.7.283" } });
57 expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284"]);
58 expect(f.events("release.cut")).toHaveLength(1);
59 expect(f.events("release.released")).toEqual([{ version: "0.7.284", image: "reg.test/o/forge:0.7.284" }]);
60 // While it rolls, a second tick cuts nothing more.
61 f.world.main = "d".repeat(40);
62 f.world.now += 2 * MIN_INTERVAL_MS;
63 expect(await tick(f.ports)).toBe("0.7.284 is rolling");
64 expect(f.world.tags).toHaveLength(1);
65 });
66
67 test("the queue mid-landing holds the cut", async () => {
68 const f = await keeperForge();
69 f.world.gating = 1;
70 expect(await tick(f.ports)).toContain("merge queue is landing");
71 expect(f.world.tags).toHaveLength(0);
72 expect(f.state.set).toHaveLength(0);
73 });
74
75 test("the cut is recorded before the tag: a crash between them is tagged on the next tick, not stranded", async () => {
76 const f = await keeperForge();
77 // The process died after the row and before the tag.
78 recordCut(f.app.db, { version: "0.7.284", sha: B, knownGood: { version: "0.7.283", image: "reg.test/o/forge:0.7.283" }, at: f.world.now });
79 expect(f.world.tags).toHaveLength(0);
80 expect(await tick(f.ports)).toBe("0.7.284 is rolling");
81 expect(f.world.tags).toEqual([{ version: "0.7.284", sha: B }]);
82 expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284"]);
83 // Tagged once: the next tick does not tag it again.
84 await tick(f.ports);
85 expect(f.world.tags).toHaveLength(1);
86 });
87
88 test("a refused tag drops the cut, so the cadence is not left waiting on a release that never was", async () => {
89 const f = await keeperForge();
90 f.world.refuseTag = "the origin refused v0.7.284";
91 expect(await tick(f.ports)).toBe("not cut: the origin refused v0.7.284");
92 expect(listKeeperReleases(f.app.db)).toHaveLength(0);
93 expect(f.state.set).toHaveLength(0);
94 expect(f.events("release.cut")).toHaveLength(0);
95 f.world.refuseTag = undefined;
96 expect(await tick(f.ports)).toContain("cut 0.7.284");
97 });
98 });
test/release-keeper-hold.test.ts added +105
@@ -0,0 +1,105 @@
1 // A failed verify holds further cuts until a pull naming the cause lands and a verified release lifts it;
2 // an owner's hold is set and lifted by an admin, never an agent (#7557).
3 import { describe, expect, test } from "bun:test";
4 import type { Ctx } from "../src/http/context.ts";
5 import { tool as holdTool } from "../src/http/mcp/tools/release_hold.ts";
6 import { tool as keeperTool } from "../src/http/mcp/tools/release_keeper.ts";
7 import { blocking, names, setHold, standingHolds, type Hold } from "../src/keeper/hold.ts";
8 import { tick } from "../src/keeper/keeper.ts";
9 import { listKeeperReleases } from "../src/keeper/record.ts";
10 import { keeperPanel } from "../src/web/pages/keeper-panel.ts";
11 import { keeperForge } from "./helpers/keeper.ts";
12
13 /** A forge whose 0.7.284 failed its verify and was rolled back to 0.7.283. */
14 async function heldForge() {
15 const f = await keeperForge();
16 await tick(f.ports);
17 f.world.live = "0.7.284";
18 f.world.healthz = (n) => (n >= 5 ? { status: 503, ms: 2 } : { status: 200, ms: 2 });
19 await tick(f.ports);
20 f.world.live = "0.7.283";
21 f.world.healthz = () => ({ status: 200, ms: 2 });
22 f.world.newest = { version: "0.7.284", sha: f.world.main };
23 f.world.main = "e".repeat(40);
24 f.world.now += 3_600_000;
25 return f;
26 }
27
28 describe("a failed verify's hold", () => {
29 test("names the version and the probe numbers, and blocks the next cut", async () => {
30 const f = await heldForge();
31 const [h] = standingHolds(f.app.db);
32 expect(h).toMatchObject({ kind: "failed", version: "0.7.284", probes: 8, badInRow: 3, causePull: null });
33 expect(await tick(f.ports)).toBe(`held: 0.7.284 failed its verify (${h!.reason}); waiting for a pull saying "release hold ${h!.id}" to land`);
34 });
35
36 test("a pull that does not name it changes nothing; one that does lets the fix through, and its verify lifts the hold", async () => {
37 const f = await heldForge();
38 const [h] = standingHolds(f.app.db);
39 f.land(10, "Tidy the explore page");
40 expect(await tick(f.ports)).toContain("held:");
41 // A changelog or a revert note mentions the version; that is no cause.
42 f.land(12, "Changelog for 0.7.284", "Reverts the 0.7.284 notes.");
43 expect(await tick(f.ports)).toContain("held:");
44 expect(standingHolds(f.app.db)[0]!.causePull).toBeNull();
45 f.land(11, "Fix the boot stall that sank 0.7.284", `Closes release hold ${h!.id}.`);
46 const line = await tick(f.ports);
47 expect(line).toContain("cut 0.7.285");
48 expect(standingHolds(f.app.db)[0]!.causePull).toBe(11);
49 // Not lifted yet: only a verified release lifts it.
50 expect(standingHolds(f.app.db)).toHaveLength(1);
51 f.world.live = "0.7.285";
52 expect(await tick(f.ports)).toContain("0.7.285 verified live");
53 expect(standingHolds(f.app.db)).toHaveLength(0);
54 expect(listKeeperReleases(f.app.db).map((r) => [r.version, r.stage])).toEqual([["0.7.285", "verified"], ["0.7.284", "rolled_back"]]);
55 expect(f.events("release.verified_live")[0]).toMatchObject({ version: "0.7.285", holdsLifted: 1 });
56 });
57
58 test("only `release hold N` names the cause: not the bare version, not another hold's number", () => {
59 const h = { id: 4, version: "0.7.284" } as Hold;
60 expect(names(h, "Fixes release hold #4")).toBe(true);
61 expect(names(h, "release hold 4")).toBe(true);
62 expect(names(h, "release hold 41")).toBe(false);
63 expect(names(h, "after 0.7.2840")).toBe(false);
64 expect(names(h, "after 10.7.284")).toBe(false);
65 expect(names(h, "0.7.284 hung at boot")).toBe(false);
66 expect(names(h, "prerelease hold 4")).toBe(false);
67 });
68
69 test("the /releases panel says what holds the next cut, with the numbers", async () => {
70 const f = await heldForge();
71 const out = String(keeperPanel(f.app.db));
72 expect(out).toContain("0.7.284 failed its verify");
73 expect(out).toContain("rolled back");
74 expect(out).toContain("0.7.283");
75 });
76 });
77
78 describe("an owner's hold", () => {
79 test("an admin sets it with a reason through MCP; the keeper cuts nothing; only an admin lifts it", async () => {
80 const f = await keeperForge();
81 const ctx = { app: f.app } as unknown as Ctx;
82 const set = await holdTool.run(ctx, f.user, { reason: "freeze for the launch demo" });
83 expect(set.ok).toBe(true);
84 expect(await tick(f.ports)).toBe("held: the owner holds releases: freeze for the launch demo");
85 expect(f.world.tags).toHaveLength(0);
86 const read = await keeperTool.run(ctx, f.user, {});
87 expect(read.ok && (read.value as { blocked: string }).blocked).toBe("the owner holds releases: freeze for the launch demo");
88 const agent = { ...f.user, kind: "agent" as const };
89 expect((await holdTool.run(ctx, agent, { lift: 1 })).ok).toBe(false);
90 expect((await holdTool.run(ctx, f.user, { lift: 1 })).ok).toBe(true);
91 expect(blocking(standingHolds(f.app.db))).toBeUndefined();
92 expect(await tick(f.ports)).toContain("cut 0.7.284");
93 });
94
95 test("an agent cannot set one, and a failed verify's hold is not lifted by hand", async () => {
96 const f = await keeperForge();
97 const ctx = { app: f.app } as unknown as Ctx;
98 expect((await holdTool.run(ctx, { ...f.user, kind: "agent" as const }, { reason: "stop" })).ok).toBe(false);
99 const h = setHold(f.app.db, { kind: "failed", version: "0.7.284", reason: "3 bad", at: f.world.now });
100 const tried = await holdTool.run(ctx, f.user, { lift: h.id });
101 expect(tried.ok).toBe(false);
102 expect(standingHolds(f.app.db)).toHaveLength(1);
103 expect((await holdTool.run(ctx, f.user, { reason: "" })).ok).toBe(false);
104 });
105 });
test/release-keeper-rollback-named.test.ts added +97
@@ -0,0 +1,97 @@
1 // The keeper rolls back by setting the known-good image recorded before the cut, by name; there is no undo path (#7557).
2 import { describe, expect, test } from "bun:test";
3 import { readdirSync, readFileSync } from "node:fs";
4 import { join } from "node:path";
5 import { imageAt, kubeDeployer, tagOf } from "../src/keeper/deploy.ts";
6 import { tick } from "../src/keeper/keeper.ts";
7 import { listKeeperReleases } from "../src/keeper/record.ts";
8 import { rollBackToNamed } from "../src/keeper/rollback.ts";
9 import { standingHolds } from "../src/keeper/hold.ts";
10 import { fakeDeployer, keeperForge } from "./helpers/keeper.ts";
11
12 const creds = { server: "https://kube.test", token: "t", namespace: "forge" };
13
14 describe("the named rollback", () => {
15 test("a seeded failing verify sets the recorded known-good image, holds with the numbers, and cuts nothing further", async () => {
16 const f = await keeperForge();
17 await tick(f.ports);
18 expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284"]);
19 // The new forge answers, then stops answering /healthz.
20 f.world.live = "0.7.284";
21 f.world.healthz = (n) => (n >= 20 ? { status: 503, ms: 4 } : { status: 200, ms: 4 });
22 const line = await tick(f.ports);
23 expect(line).toContain("0.7.284 failed its verify");
24 // The image set is the one recorded at the cut, by name — not the controller's "previous" (the broken 0.7.277 here).
25 expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284", "reg.test/o/forge:0.7.283"]);
26 expect(f.state.image).toBe("reg.test/o/forge:0.7.283");
27 const [r] = listKeeperReleases(f.app.db);
28 expect(r).toMatchObject({ stage: "rolled_back", probes: 23, badInRow: 3, secondsDark: 3 });
29 expect(r!.reason).toContain("rolled back to 0.7.283 (reg.test/o/forge:0.7.283)");
30 const [hold] = standingHolds(f.app.db);
31 expect(hold).toMatchObject({ kind: "failed", version: "0.7.284", probes: 23, badInRow: 3, worstMs: 40 });
32 expect(hold!.reason).toContain("3 bad /healthz probes in a row");
33 expect(f.events("release.rolled_back")).toEqual([{ version: "0.7.284", to: "0.7.283", image: "reg.test/o/forge:0.7.283", ok: true }]);
34 expect(f.events("release.held")).toHaveLength(1);
35 // Back on the known-good version, main moves on, time passes: nothing more is cut.
36 f.world.live = "0.7.283";
37 f.world.newest = { version: "0.7.284", sha: f.world.main };
38 f.world.main = "e".repeat(40);
39 f.world.now += 24 * 3_600_000;
40 expect(await tick(f.ports)).toContain("held: 0.7.284 failed its verify");
41 expect(f.world.tags).toHaveLength(1);
42 expect(f.state.set).toHaveLength(2);
43 });
44
45 test("a release not live three hours after its cut is set back by name, and held", async () => {
46 const f = await keeperForge();
47 await tick(f.ports);
48 f.world.now += 3 * 3_600_000 + 1;
49 expect(await tick(f.ports)).toContain("was not live 3 h after its cut");
50 expect(f.state.image).toBe("reg.test/o/forge:0.7.283");
51 expect(standingHolds(f.app.db)).toHaveLength(1);
52 });
53
54 test("a recorded image that is no image reference is refused, and nothing is set", async () => {
55 const { deployer, state } = fakeDeployer("reg.test/o/forge:0.7.284");
56 const r = await rollBackToNamed(deployer, { version: "0.7.283", image: "--help" });
57 expect(r.ok).toBe(false);
58 expect(state.set).toEqual([]);
59 });
60
61 test("there is no undo path: the deployer reads, sets a named image, or waits — and sends one PATCH naming it", async () => {
62 const sent: { url: string; method: string; body: string }[] = [];
63 const deployer = kubeDeployer({
64 credentials: creds,
65 deployment: "forge",
66 container: "forge",
67 fetch: async (url, init) => {
68 sent.push({ url, method: init.method ?? "GET", body: String(init.body ?? "") });
69 return Response.json({ spec: { replicas: 1, template: { spec: { containers: [{ name: "forge", image: "reg.test/o/forge:0.7.284" }] } } } });
70 },
71 });
72 expect(Object.keys(deployer).sort()).toEqual(["image", "rolled", "setImage"]);
73 const r = await rollBackToNamed(deployer, { version: "0.7.283", image: "reg.test/o/forge:0.7.283" });
74 expect(r.ok).toBe(true);
75 expect(sent).toEqual([
76 {
77 url: "https://kube.test/apis/apps/v1/namespaces/forge/deployments/forge",
78 method: "PATCH",
79 body: JSON.stringify({ spec: { template: { spec: { containers: [{ name: "forge", image: "reg.test/o/forge:0.7.283" }] } } } }),
80 },
81 ]);
82 // No code under src/keeper asks for a previous revision: no rollout undo, no rollback subresource, no revision number.
83 const dir = join(import.meta.dir, "../src/keeper");
84 for (const file of readdirSync(dir)) {
85 const code = readFileSync(join(dir, file), "utf8").replace(/\/\*[\s\S]*?\*\//g, "").replace(/\/\/.*$/gm, "");
86 expect(code.match(/undo|\/rollback(?!\.ts)|revision|previous/i)?.[0], file).toBeUndefined();
87 }
88 });
89
90 test("an image keeps its registry port when its tag changes", () => {
91 expect(imageAt("reg.test:5000/o/forge:0.7.283", "0.7.284")).toBe("reg.test:5000/o/forge:0.7.284");
92 expect(imageAt("reg.test:5000/o/forge", "0.7.284")).toBe("reg.test:5000/o/forge:0.7.284");
93 expect(imageAt(`reg.test/o/forge:0.7.283@sha256:${"f".repeat(64)}`, "0.7.284")).toBe("reg.test/o/forge:0.7.284");
94 expect(tagOf("reg.test:5000/o/forge:0.7.283")).toBe("0.7.283");
95 expect(tagOf("reg.test:5000/o/forge")).toBe("");
96 });
97 });
test/release-keeper-verify.test.ts added +140
@@ -0,0 +1,140 @@
1 // Proving a release live (#7557): ten minutes of /healthz once a second under page load; three bad in a row fail it.
2 import { describe, expect, test } from "bun:test";
3 import { tick, rollProbe } from "../src/keeper/keeper.ts";
4 import { listKeeperReleases } from "../src/keeper/record.ts";
5 import { EMPTY, FAIL_AFTER, SLOW_MS, tally, verifyUnderLoad, type Probe } from "../src/keeper/verify.ts";
6 import { keeperForge } from "./helpers/keeper.ts";
7
8 /** A clock that only the probes and the sleeps move. */
9 function clocked(healthz: (n: number) => Probe, pages: readonly string[] = [], page: Probe = { status: 200, ms: 0 }) {
10 let now = 0;
11 let n = 0;
12 let open = 0;
13 let mostOpen = 0;
14 const run = verifyUnderLoad({
15 get: async (path) => {
16 if (path === "/healthz") {
17 const p = healthz(n++);
18 now += p.ms;
19 return p;
20 }
21 open++;
22 mostOpen = Math.max(mostOpen, open);
23 await Promise.resolve();
24 now += page.ms;
25 open--;
26 return page;
27 },
28 now: () => now,
29 sleep: async (ms) => {
30 now += ms;
31 },
32 pages,
33 });
34 return { run, mostOpen: () => mostOpen };
35 }
36
37 describe("tally", () => {
38 test("a bad probe is a non-200 or one over 10 s; a slow 200 is slow, not dark", () => {
39 let t = tally(EMPTY, { status: 200, ms: SLOW_MS + 1 });
40 expect(t).toMatchObject({ probes: 1, non200: 0, badInRow: 1, worstMs: 10_001, secondsDark: 0 });
41 t = tally(t, { status: 502, ms: 3 });
42 expect(t).toMatchObject({ probes: 2, non200: 1, badInRow: 2, secondsDark: 1 });
43 t = tally(t, { status: 200, ms: 5 });
44 expect(t).toMatchObject({ badInRow: 0, worstRun: 2 });
45 // Exactly 10 s is not over it.
46 expect(tally(EMPTY, { status: 200, ms: SLOW_MS }).badInRow).toBe(0);
47 });
48 });
49
50 describe("under page load", () => {
51 test("a healthy release: 600 probes in 10 minutes, one a second, and it passes", async () => {
52 const v = await clocked(() => ({ status: 200, ms: 0 })).run;
53 expect(v.ok).toBe(true);
54 expect(v.tally.probes).toBe(600);
55 expect(v.tally.secondsDark).toBe(0);
56 });
57
58 test("two bad probes in a row pass; three fail at once, with the numbers", async () => {
59 const two = await clocked((n) => (n === 10 || n === 11 ? { status: 503, ms: 1 } : { status: 200, ms: 1 })).run;
60 expect(two.ok).toBe(true);
61 expect(two.tally).toMatchObject({ non200: 2, worstRun: 2, secondsDark: 2 });
62 const three = await clocked((n) => (n >= 10 && n < 10 + FAIL_AFTER ? { status: 0, ms: 1 } : { status: 200, ms: 1 })).run;
63 expect(three.ok).toBe(false);
64 expect(three.tally.probes).toBe(13);
65 expect(three.reason).toBe("3 bad /healthz probes in a row (non-200 or over 10 s) after 13 probes; 3 non-200, worst 1 ms");
66 });
67
68 test("three probes over 10 s fail it though each was a 200", async () => {
69 const v = await clocked((n) => ({ status: 200, ms: n >= 5 && n < 8 ? 10_500 : 2 })).run;
70 expect(v.ok).toBe(false);
71 expect(v.tally).toMatchObject({ non200: 0, worstMs: 10_500, secondsDark: 0 });
72 });
73
74 test("the pages are fetched one at a time, every one of them", async () => {
75 const pages = ["/boards", "/alice/forge", "/explore", "/"];
76 const c = clocked(() => ({ status: 200, ms: 0 }), pages, { status: 200, ms: 250 });
77 const v = await c.run;
78 expect(c.mostOpen()).toBe(1);
79 expect(v.tally.pages).toBeGreaterThan(pages.length);
80 expect(v.tally.worstMs).toBe(250);
81 });
82 });
83
84 describe("a seeded healthy release", () => {
85 test("ends verified live with its probes and worst latency stored, and an event", async () => {
86 const f = await keeperForge({ healthz: () => ({ status: 200, ms: 30 }) });
87 await tick(f.ports);
88 // The old forge counts the roll: zero non-200 with surge.
89 for (let i = 0; i < 5; i++) await rollProbe(f.ports);
90 f.world.live = "0.7.284";
91 const line = await tick(f.ports);
92 expect(line).toContain("0.7.284 verified live");
93 const [r] = listKeeperReleases(f.app.db);
94 expect(r!.stage).toBe("verified");
95 expect(r!.rollProbes).toBe(5);
96 expect(r!.rollNon200).toBe(0);
97 expect(r!.probes).toBeGreaterThan(500);
98 expect(r!.pages).toBeGreaterThan(0);
99 expect(r!.worstMs).toBe(40);
100 expect(r!.secondsDark).toBe(0);
101 expect(r!.liveAt).not.toBeNull();
102 const [ev] = f.events("release.verified_live");
103 expect(ev).toMatchObject({ version: "0.7.284", probes: r!.probes, worstMs: 40, secondsDark: 0 });
104 expect(f.world.pages.slice(0, 4)).toEqual(["/boards", "/alice/forge", "/explore", "/"]);
105 // Nothing went dark, so nothing is filed.
106 expect(f.app.db.query("SELECT COUNT(*) AS n FROM issue WHERE title LIKE 'Release % dark%'").get()).toEqual({ n: 0 });
107 });
108
109 test("a roll that went dark files one issue, and only one", async () => {
110 const f = await keeperForge({ healthz: () => ({ status: 200, ms: 5 }) });
111 await tick(f.ports);
112 const healthy = f.world.healthz;
113 f.world.healthz = () => ({ status: 0, ms: 1 });
114 for (let i = 0; i < 4; i++) await rollProbe(f.ports);
115 f.world.healthz = healthy;
116 f.world.live = "0.7.284";
117 await tick(f.ports);
118 const [r] = listKeeperReleases(f.app.db);
119 expect(r).toMatchObject({ stage: "verified", rollNon200: 4, secondsDark: 4 });
120 const filed = f.app.db.query<{ title: string; body: string }, []>("SELECT title, body FROM issue WHERE title LIKE 'Release %'").all();
121 expect(filed.map((i) => i.title)).toEqual(["Release 0.7.284 was dark for 4 s"]);
122 expect(filed[0]!.body).toContain("4 of 4 during the roll");
123 });
124
125 test("a closed issue about the same dark time does not stop the next report", async () => {
126 const f = await keeperForge({ healthz: () => ({ status: 200, ms: 5 }) });
127 f.app.db
128 .query(`INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at) VALUES ('old', ?1, 900, 'issue', 'Release 0.7.284 was dark for 4 s', '', ?2, 'closed', ?3)`)
129 .run(f.repo.id, f.user.id, new Date(f.world.now).toISOString());
130 await tick(f.ports);
131 const healthy = f.world.healthz;
132 f.world.healthz = () => ({ status: 0, ms: 1 });
133 for (let i = 0; i < 4; i++) await rollProbe(f.ports);
134 f.world.healthz = healthy;
135 f.world.live = "0.7.284";
136 await tick(f.ports);
137 const open = f.app.db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM issue WHERE title = 'Release 0.7.284 was dark for 4 s' AND state = 'open'").get();
138 expect(open).toEqual({ n: 1 });
139 });
140 });
test/tui-people-verify.test.ts +4 −1
@@ -59,7 +59,10 @@ describe("verify an email from the People screen", () => {
5959 try {
6060 expect(h.text()).toMatch(/v +verify/);
6161 expect(h.text()).toMatch(/bobby.*unverified/);
62 // newest first: bobby is the first row
62 // Newest first, but alice and bobby can share a created_at millisecond and then
63 // sort by name: put the cursor on bobby rather than assume his row.
64 if (!/▸ bobby/.test(h.text())) await h.press("j");
65 expect(h.text()).toMatch(/▸ bobby/);
6366 await h.press("v");
6467 expect(h.text()).toContain("verify bobby's email? y/n");
6568 await h.press("n");
test/tui-release-keeper.test.ts added +61
@@ -0,0 +1,61 @@
1 /**
2 * The release keeper in the terminal (#7557): the palette offers it to whoever
3 * may read /releases, and the screen shows the card's read — what holds the
4 * next cut, then each release with its stage, known-good version and numbers.
5 */
6 import { describe, expect, test } from "bun:test";
7 import { tick } from "../src/keeper/keeper.ts";
8 import type { User } from "../src/core/users.ts";
9 import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
10 import type { Ctx } from "../src/tui/app/app.ts";
11 import { keeperScreen, readKeeper } from "../src/tui/screens/keeper.ts";
12 import { headlessTerminal } from "../src/tui/term/terminal.ts";
13 import { quietLogs } from "./helpers/globals.ts";
14 import { keeperForge } from "./helpers/keeper.ts";
15 import { settleApp } from "./helpers/tui.ts";
16
17 quietLogs();
18
19 const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
20
21 /** A forge whose 0.7.284 failed its verify and was rolled back to 0.7.283. */
22 async function heldForge() {
23 const f = await keeperForge();
24 await tick(f.ports);
25 f.world.live = "0.7.284";
26 f.world.healthz = (n) => (n >= 5 ? { status: 503, ms: 2 } : { status: 200, ms: 2 });
27 await tick(f.ports);
28 return f;
29 }
30
31 describe("the release keeper screen", () => {
32 test("the palette offers it to an admin, not to a guest", async () => {
33 const f = await keeperForge();
34 const offered = (user?: User) => forgeCommands(new Session(f.app, identity(user)))({} as Ctx).some((c) => c.id === "keeper");
35 expect(offered(f.user)).toBe(true);
36 expect(offered(undefined)).toBe(false);
37 expect(readKeeper(new Session(f.app, identity(undefined)))).toBeUndefined();
38 });
39
40 test("it shows what holds the next cut and the rolled-back release with its numbers", async () => {
41 const f = await heldForge();
42 const terminal = headlessTerminal(160, 20);
43 const { app: tui, session } = createForgeApp(f.app, identity(f.user), { terminal, exitHooks: false });
44 tui.start();
45 tui.ctx.push(keeperScreen(session, tui.ctx));
46 await settleApp(tui);
47 const text = tui.screen.toText();
48 tui.stop?.();
49 expect(text).toContain("0.7.284 failed its verify");
50 expect(text.replace(/\s+/g, " ")).toContain(`a pull saying "release hold 1" to land`);
51 expect(text).toContain("rolled back");
52 expect(text).toContain("known good 0.7.283");
53 expect(text).toMatch(/8 probes/);
54 });
55
56 test("with nothing cut, it says nothing holds the next cut and that no releases are kept yet", async () => {
57 const f = await keeperForge();
58 const read = readKeeper(new Session(f.app, identity(f.user)));
59 expect(read).toEqual({ blocked: "Nothing holds the next cut.", releases: [] });
60 });
61 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs