trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns17g rEventsFlows

A change is live in two minutes, proven on a copy before anyone sees it, and… #11841

Merged masoncrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/11839-a-change-is-live-in-two-minutes-proven-o → main 4 commits fork 2 threads · 2 unresolved

  • Merged. mason/11839-a-change-is-live-in-two-minutes-proven-o was merged into main as fd4d15b.

Review

Sign in to review this change.
21 files changed +360 −23
docs/MCP.md +1
@@ -177,6 +177,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
177177 | `read_file` | — | Read a file or list a directory at a ref (default branch); over 5 MB answers tooLarge | 307 |
178178 | `release_hold` | — | Hold or release the release keeper's cuts (admin) | 202 |
179179 | `release_keeper` | — | The release keeper's releases, holds and what blocks the next cut | 192 |
180 | `release_now` | — | Cut a release now, with a reason (admin) | 176 |
180181 | `releases` | — | Kept releases, newest first | 253 |
181182 | `repo` | — | One repository you can read, as GET /api/v1/repos/:owner/:name | 223 |
182183 | `repo_labels` | — | Repo labels: list (default), create, update, remove (admin token) | 351 |
docs/features/release-keeper.md +18
@@ -72,6 +72,24 @@ files one issue, `Release <version> was dark for <n> s`.
7272 (`release_hold {lift: <id>}`, the card's Lift button, `h` then `lift N`) — one function, `holdAs`/`liftAs`, behind all three.
7373 It needs a person who may roll back (MCP: on an admin-scope token); an agent never can.
7474
75 ## A hotfix does not wait (#11839)
76
77 `release_now {reason}` (MCP, admin scope), the **Release now** form on the
78 `/releases` card: a person who may roll back (never an agent) asks for a cut at
79 the next tick. It skips the 30-minute spacing, a running merge queue and a
80 snapshot; it still waits for a release in flight to be proven, for a hold and
81 for room on the disk. The request lapses after an hour if main had nothing to
82 cut; the reason is kept on the release (`hotfix`) and shown on the card.
83
84 ## Where the time went (#11839)
85
86 Each release row keeps its stage timings: cut to live, build (checkout of the
87 tree), proof (the preflight boot), switch (the new process taking the port) and
88 verify. The swap deployer measures build, proof and switch; a stage not
89 measured is left out. They show under the release on `/releases` and in
90 `release_keeper` (`stages`), which also answers `holding` (what the tick last
91 said stopped it: the disk guard, the spacing, a snapshot) and `releaseNow`.
92
7593 ## Where to see it
7694
7795 - The terminal: **Release keeper** in the command palette shows the same
src/core/migration-pins/20261112140000-keeper-stages.pin added +1
@@ -0,0 +1,1 @@
1 1fa8037ac6e73c9523bd944562d207e3b4d67148b52faabb15a7fe2e09c0646d
src/core/migrations/20261112140000-keeper-stages.ts added +17
@@ -0,0 +1,17 @@
1 // The release keeper times the stages it owns (#11839): how long the tree took to check out (build),
2 // to pass its preflight boot (proof) and to be started beside the live one and take the port (switch),
3 // each null when that release did not measure it. `hotfix` is the reason a person cut the release now,
4 // outside the spacing; empty for a cut on cadence. `keeper_now` is a person's standing request to cut at
5 // the next tick, one row, consumed by the cut. Additive: new nullable columns and a new table.
6 export default `
7 ALTER TABLE keeper_release ADD COLUMN build_ms INTEGER;
8 ALTER TABLE keeper_release ADD COLUMN proof_ms INTEGER;
9 ALTER TABLE keeper_release ADD COLUMN switch_ms INTEGER;
10 ALTER TABLE keeper_release ADD COLUMN hotfix TEXT NOT NULL DEFAULT '';
11 CREATE TABLE IF NOT EXISTS keeper_now (
12 id INTEGER PRIMARY KEY CHECK (id = 1),
13 reason TEXT NOT NULL,
14 by TEXT NOT NULL,
15 at TEXT NOT NULL
16 );
17 `;
src/crew/conversation_confirm.ts +1
@@ -42,6 +42,7 @@ export const ASKING: Readonly<Record<string, Asking>> = {
4242 does: "hold or lift the release keeper",
4343 undo: (a) => (a["reason"] !== undefined ? "`release_hold` with `lift` and the hold's id lifts it" : undefined),
4444 },
45 release_now: { does: "cut a release now, outside the spacing" },
4546 // Archive and delete.
4647 archive: {
4748 does: "archive or wake a repository",
src/http/mcp/tools/index.ts +1
@@ -136,6 +136,7 @@ export { tool as quotas } from "./quotas.ts";
136136 export { tool as read_file } from "./read_file.ts";
137137 export { tool as release_hold } from "./release_hold.ts";
138138 export { tool as release_keeper } from "./release_keeper.ts";
139 export { tool as release_now } from "./release_now.ts";
139140 export { tool as releases } from "./releases.ts";
140141 export { tool as repo } from "./repo.ts";
141142 export { tool as repo_labels } from "./repo_labels.ts";
src/http/mcp/tools/release_now.ts added +16
@@ -0,0 +1,16 @@
1 import { MAX_REASON } from "../../../keeper/hold.ts";
2 import { releaseNowAs } from "../../../keeper/now.ts";
3 import { bad, good, str, type Tool } from "../shared.ts";
4
5 export const tool: Tool = {
sentinelcrew

Page and MCP have this; the terminal does not (both doors). Add the src/tui/ entry.
6 name: "release_now",
7 doc: "`{ reason }`: ask the release keeper (#7557) to cut main now, at its next tick (within a minute), outside the 30-minute spacing and past a running merge queue or snapshot, for a hotfix. The reason stays on the release. It still waits for a release in flight to be proven, for a hold and for room on the disk. A person who may roll back, with an admin-scope token, never an agent",
8 description: "Cut a release now, with a reason (admin).",
9 inputSchema: { type: "object", properties: { reason: { type: "string", maxLength: MAX_REASON } }, required: ["reason"] },
10 writes: true,
11 needsAdminScope: true,
12 run(ctx, user, args) {
13 const asked = releaseNowAs(ctx.app, user, str(args["reason"]));
14 return asked.ok ? good({ releaseNow: asked.value }) : bad(asked.error, asked.message);
15 },
16 };
src/http/server.ts +2 −1
@@ -178,7 +178,7 @@ import { chapterAct, chapterPage, chaptersPage } from "../web/pages/chapters.ts"
178178 import { betAct, betPage, betPropose, betsPage } from "../web/pages/bets.ts";
179179 import { changelogJson, changelogPage } from "../web/pages/changelog.ts";
180180 import { docsFile, docsFolder } from "../web/pages/docs_link.ts";
181 import { releaseHoldSubmit, liftSubmit, releasesPage, rollbackSubmit } from "../web/pages/releases.ts";
181 import { releaseHoldSubmit, releaseNowSubmit, liftSubmit, releasesPage, rollbackSubmit } from "../web/pages/releases.ts";
182182 import {
183183 changePassword,
184184 iconPng,
@@ -983,6 +983,7 @@ export function buildRouter(): Router<Handler> {
983983 router.add("GET", "/:owner/:name/api", appApiPage);
984984 router.add("POST", "/releases/:sha/rollback", rollbackSubmit);
985985 router.add("POST", "/releases/hold", releaseHoldSubmit);
986 router.add("POST", "/releases/now", releaseNowSubmit);
986987 router.add("POST", "/releases/hold/:id/lift", liftSubmit);
987988 router.add("GET", "/audit.csv", auditCsv);
988989 router.add("POST", "/audit/:owner/:name/probe", auditProbeSubmit);
src/keeper/cadence.ts +9 −3
@@ -57,6 +57,11 @@ export interface CadenceInput {
5757 readonly diskUsed?: number | undefined;
5858 readonly now: number;
5959 readonly minIntervalMs?: number;
60 /**
61 * A person asked for a release now (#11839): the merge queue, a snapshot and the
62 * spacing no longer hold the cut. A release in flight, a hold and a full disk still do.
63 */
64 readonly hotfix?: boolean;
6065 }
6166
6267 export type Cadence = { readonly cut: true; readonly version: string } | { readonly cut: false; readonly why: string };
@@ -77,13 +82,14 @@ export function decideCut(input: CadenceInput): Cadence {
7782 if (input.main === input.newest.sha) return no(`main is still ${input.newest.version}`);
7883 if (input.inFlight) return no("a release is still rolling or being verified");
7984 if (input.live !== input.newest.version && input.settled !== true) return no(`${input.newest.version} is not live yet (live: ${input.live})`);
80 if (input.gating > 0) return no(`the merge queue is landing (${input.gating} gate${input.gating === 1 ? "" : "s"} running)`);
81 if (input.snapshotMs !== undefined && input.snapshotMs < SNAPSHOT_WAIT_MS)
85 const hotfix = input.hotfix === true;
86 if (!hotfix && input.gating > 0) return no(`the merge queue is landing (${input.gating} gate${input.gating === 1 ? "" : "s"} running)`);
87 if (!hotfix && input.snapshotMs !== undefined && input.snapshotMs < SNAPSHOT_WAIT_MS)
8288 return no(`a snapshot is being taken (${Math.floor(input.snapshotMs / 60_000)} min in); a release would cut it short`);
8389 if (input.diskUsed !== undefined && input.diskUsed >= RELEASE_DISK_USED_MAX)
8490 return no(`the disk is ${Math.round(input.diskUsed * 100)}% full; releases pause at ${RELEASE_DISK_USED_MAX * 100}% until room is made`);
8591 const wait = input.minIntervalMs ?? MIN_INTERVAL_MS;
86 if (input.lastCutAt !== undefined && input.now - input.lastCutAt < wait)
92 if (!hotfix && input.lastCutAt !== undefined && input.now - input.lastCutAt < wait)
8793 return no(`the last cut was ${Math.floor((input.now - input.lastCutAt) / 60_000)} min ago; the interval is ${wait / 60_000} min`);
8894 const version = nextVersion(input.newest.version);
8995 if (version === undefined) return no(`${input.newest.version} is not major.minor.patch`);
src/keeper/keeper.ts +5 −1
@@ -33,6 +33,7 @@ import { createIssue } from "../core/issues.ts";
3333 import { decideCut } from "./cadence.ts";
3434 import { imageAt, type Deployer } from "./deploy.ts";
3535 import { blocking, liftFixed, names, setCause, setHold, standingHolds, type Hold } from "./hold.ts";
36 import { clearNow, pendingNow } from "./now.ts";
3637 import { stallReason, STALL_HOLD_MS } from "../perf/boot_stall.ts";
3738 import { countRollProbe, dropCut, finish, getKeeperRelease, inFlight, keepStall, keepTally, listKeeperReleases, markLive, releaseBefore, recordCut, type KeeperRelease } from "./record.ts";
3839 import { rollBackToNamed } from "./rollback.ts";
@@ -207,6 +208,7 @@ export async function tickStep(p: KeeperPorts): Promise<Step> {
207208 const main = await p.main();
208209 const newest = await p.newest();
209210 const last = listKeeperReleases(p.db, 1)[0];
211 const asked = pendingNow(p.db, p.now());
210212 const decision = decideCut({
211213 main: main ?? "",
212214 newest,
@@ -219,6 +221,7 @@ export async function tickStep(p: KeeperPorts): Promise<Step> {
219221 lastCutAt: last === undefined ? undefined : Date.parse(last.cutAt),
220222 hold: blocking(standingHolds(p.db)),
221223 now: p.now(),
224 ...(asked === undefined ? {} : { hotfix: true }),
222225 ...(p.minIntervalMs === undefined ? {} : { minIntervalMs: p.minIntervalMs }),
223226 });
224227 if (!decision.cut) return { line: decision.why, outcome: "skipped" };
@@ -230,7 +233,8 @@ export async function tickStep(p: KeeperPorts): Promise<Step> {
230233 const knownGood = { version: p.live(), image };
231234 // The row first, the tag second: a crash between them leaves a `rolling` row
232235 // whose tag the next tick sends (`tagged`), never a tag the keeper has no row for.
233 const r = recordCut(p.db, { version: decision.version, sha: main, knownGood, at: p.now() });
236 const r = recordCut(p.db, { version: decision.version, sha: main, knownGood, at: p.now(), ...(asked === undefined ? {} : { hotfix: `${asked.reason} (${asked.by})` }) });
237 if (asked !== undefined) clearNow(p.db);
234238 const tagRefused = await tagged(p, r);
235239 if (tagRefused !== undefined) return { line: tagRefused, outcome: "failed" };
236240 const sent = await promote(p, r.version, refFor(p, image, r.version, main));
src/keeper/landed.ts +47 −4
@@ -9,6 +9,7 @@
99 import type { Database } from "bun:sqlite";
1010 import { beatLine, lastBeat, SILENT_MS, type Beat } from "./beat.ts";
1111 import { blocking, listHolds, standingHolds, type Hold } from "./hold.ts";
12 import { pendingNow, type NowRequest } from "./now.ts";
1213 import { listKeeperReleases, type KeeperRelease } from "./record.ts";
1314
1415 /** When `sha` landed, epoch ms: its pull's merge, else the first time a landing checked it out. */
@@ -50,10 +51,48 @@ export function newestLive(db: Database): { version: string; sha: string; second
5051 }
5152
5253 /** A keeper release with its short commit and landed-to-live seconds (null until it answered): what the card, the terminal and MCP show. */
53 export type LandedRelease = KeeperRelease & { readonly commit: string; readonly landedToLiveSeconds: number | null };
54 export type LandedRelease = KeeperRelease & {
55 readonly commit: string;
56 readonly landedToLiveSeconds: number | null;
57 /** Where the release's time went (#11839): so the slow stage is named. */
58 readonly stages: Stages;
59 };
60
61 /** Each stage's milliseconds, null where it was not measured or has not happened. */
62 export interface Stages {
63 /** Tag cut to the new version answering. */
64 readonly cutToLiveMs: number | null;
65 readonly buildMs: number | null;
66 readonly proofMs: number | null;
67 readonly switchMs: number | null;
68 /** The new version answering to its verdict. */
69 readonly verifyMs: number | null;
70 }
71
72 const between = (from: string | null, to: string | null): number | null => {
73 if (from === null || to === null) return null;
74 const ms = Date.parse(to) - Date.parse(from);
75 return Number.isFinite(ms) ? Math.max(0, ms) : null;
76 };
77
78 export function stagesOf(r: Pick<KeeperRelease, "cutAt" | "liveAt" | "finishedAt" | "buildMs" | "proofMs" | "switchMs">): Stages {
79 return {
80 cutToLiveMs: between(r.cutAt, r.liveAt),
81 buildMs: r.buildMs,
82 proofMs: r.proofMs,
83 switchMs: r.switchMs,
84 verifyMs: between(r.liveAt, r.finishedAt),
85 };
86 }
87
88 /** "cut→live 1 min 2 s · build 4 s · proof 3 s · switch 1 s · verify 10 min 0 s"; a stage not measured is left out. */
89 export function stagesLine(s: Stages): string {
90 const w = (name: string, ms: number | null) => (ms === null ? [] : [`${name} ${secondsWords(Math.round(ms / 1000))}`]);
91 return [...w("cut→live", s.cutToLiveMs), ...w("build", s.buildMs), ...w("proof", s.proofMs), ...w("switch", s.switchMs), ...w("verify", s.verifyMs)].join(" · ");
92 }
5493
5594 export function withLandedToLive(db: Database, releases: readonly KeeperRelease[]): LandedRelease[] {
56 return releases.map((r) => ({ ...r, commit: r.sha.slice(0, 7), landedToLiveSeconds: landedToLive(db, r) ?? null }));
95 return releases.map((r) => ({ ...r, commit: r.sha.slice(0, 7), landedToLiveSeconds: landedToLive(db, r) ?? null, stages: stagesOf(r) }));
5796 }
5897
5998 /** The tick's last heartbeat with its sentence and whether it has gone quiet (#11507); null before the first. */
@@ -69,11 +108,15 @@ export function heartbeatOf(db: Database, now: Date = new Date()): Heartbeat | n
69108 export function keeperAnswer(
70109 db: Database,
71110 now: Date = new Date(),
72 ): { blocked: string | null; releases: LandedRelease[]; holds: Hold[]; heartbeat: Heartbeat | null } {
111 ): { blocked: string | null; holding: string | null; releaseNow: NowRequest | null; releases: LandedRelease[]; holds: Hold[]; heartbeat: Heartbeat | null } {
112 const heartbeat = heartbeatOf(db, now);
73113 return {
74114 blocked: blocking(standingHolds(db)) ?? null,
115 // What the tick last said stopped it (the disk guard, the spacing, a snapshot): visible, not only in a log (#11839).
116 holding: heartbeat !== null && heartbeat.outcome === "skipped" && !heartbeat.silent ? heartbeat.reason : null,
117 releaseNow: pendingNow(db, now.getTime()) ?? null,
75118 releases: withLandedToLive(db, listKeeperReleases(db)),
76119 holds: listHolds(db),
77 heartbeat: heartbeatOf(db, now),
120 heartbeat,
78121 };
79122 }
src/keeper/now.ts added +55
@@ -0,0 +1,55 @@
1 /**
2 * `release now` (#11839): a person who may roll back asks the keeper to cut at
3 * its next tick, outside the spacing, with a reason that stays on the release.
4 * A hotfix does not wait for the half hour, the merge queue or a snapshot; it
5 * still waits for a release in flight to be proven, for a hold, and for room
6 * on the disk. One row, consumed by the cut; one that is not consumed within
7 * {@link NOW_TTL_MS} lapses, so a request made when main had nothing new does
8 * not cut a release a day later. One function behind the MCP tool and the
9 * /releases card.
10 */
11
12 import type { Database } from "bun:sqlite";
13 import type { App } from "../app.ts";
14 import type { User } from "../core/users.ts";
15 import { mayRollBack } from "../release/gate.ts";
16 import { MAX_REASON, type OwnerHoldResult } from "./hold.ts";
17
18 export const NOW_TTL_MS = 60 * 60_000;
19
20 export interface NowRequest {
21 readonly reason: string;
22 readonly by: string;
23 readonly at: string;
24 }
25
26 /** The standing request, if one has not lapsed. */
27 export function pendingNow(db: Database, at: number): NowRequest | undefined {
28 const row = db.query<NowRequest, []>(`SELECT reason, by, at FROM keeper_now WHERE id = 1`).get();
29 if (row === null || at - Date.parse(row.at) > NOW_TTL_MS) return undefined;
30 return row;
31 }
32
33 export function clearNow(db: Database): void {
34 db.query(`DELETE FROM keeper_now WHERE id = 1`).run();
35 }
36
37 export function askNow(db: Database, reason: string, by: string, at: number): NowRequest {
38 const req = { reason, by, at: new Date(at).toISOString() };
39 db.query(
40 `INSERT INTO keeper_now (id, reason, by, at) VALUES (1, ?1, ?2, ?3)
41 ON CONFLICT (id) DO UPDATE SET reason = excluded.reason, by = excluded.by, at = excluded.at`,
42 ).run(req.reason, req.by, req.at);
43 return req;
44 }
45
46 /** Ask for a release now. Refused for an agent and for anyone who may not roll back; a refusal writes nothing. */
47 export function releaseNowAs(app: App, user: User | undefined, reason: string): OwnerHoldResult<NowRequest> {
48 if (user === undefined || !mayRollBack(app, user))
49 return { ok: false, error: "forbidden", message: "cutting a release now is for a person who may write the forge's repository and administer it, not an agent" };
50 const clean = reason.replace(/[\u0000-\u001f\u007f]+/g, " ").trim();
51 if (clean === "" || clean.length > MAX_REASON) return { ok: false, error: "invalid", message: `reason is one line of 1 to ${MAX_REASON} characters` };
52 const req = askNow(app.db, clean, user.username, Date.now());
53 app.bus.append({ kind: "release.now_asked", actorId: user.id, payload: { reason: clean } });
54 return { ok: true, value: req };
55 }
src/keeper/record.ts +29 −5
@@ -34,6 +34,12 @@ export interface KeeperRelease {
3434 /** The worst event-loop stall in the release's first five minutes, ms (#8848). */
3535 readonly worstStallMs: number;
3636 readonly reason: string;
37 /** Stage timings, ms; null when this release did not measure the stage (#11839). */
38 readonly buildMs: number | null;
39 readonly proofMs: number | null;
40 readonly switchMs: number | null;
41 /** Why a person cut this release now, outside the spacing; empty for a cut on cadence. */
42 readonly hotfix: string;
3743 }
3844
3945 interface Row {
@@ -55,6 +61,10 @@ interface Row {
5561 seconds_dark: number;
5662 worst_stall_ms: number;
5763 reason: string;
64 build_ms: number | null;
65 proof_ms: number | null;
66 switch_ms: number | null;
67 hotfix: string;
5868 }
5969
6070 const toRelease = (r: Row): KeeperRelease => ({
@@ -75,6 +85,10 @@ const toRelease = (r: Row): KeeperRelease => ({
7585 secondsDark: r.seconds_dark,
7686 worstStallMs: r.worst_stall_ms,
7787 reason: r.reason,
88 buildMs: r.build_ms,
89 proofMs: r.proof_ms,
90 switchMs: r.switch_ms,
91 hotfix: r.hotfix,
7892 });
7993
8094 const iso = (ms: number) => new Date(ms).toISOString();
@@ -82,15 +96,15 @@ const iso = (ms: number) => new Date(ms).toISOString();
8296 /** The cut, written before the tag is pushed: the known-good image is on record before anything can need it. */
8397 export function recordCut(
8498 db: Database,
85 input: { version: string; sha: string; knownGood: { version: string; image: string }; at: number },
99 input: { version: string; sha: string; knownGood: { version: string; image: string }; at: number; hotfix?: string },
86100 ): KeeperRelease {
87101 return toRelease(
88102 db
89 .query<Row, [string, string, string, string, string]>(
90 `INSERT INTO keeper_release (version, sha, known_good_version, known_good_image, stage, cut_at)
91 VALUES (?1, ?2, ?3, ?4, 'rolling', ?5) RETURNING *`,
103 .query<Row, [string, string, string, string, string, string]>(
104 `INSERT INTO keeper_release (version, sha, known_good_version, known_good_image, stage, cut_at, hotfix)
105 VALUES (?1, ?2, ?3, ?4, 'rolling', ?5, ?6) RETURNING *`,
92106 )
93 .get(input.version, input.sha, input.knownGood.version, input.knownGood.image, iso(input.at))!,
107 .get(input.version, input.sha, input.knownGood.version, input.knownGood.image, iso(input.at), input.hotfix ?? "")!,
94108 );
95109 }
96110
@@ -99,6 +113,16 @@ export function dropCut(db: Database, id: number): void {
99113 db.query(`DELETE FROM keeper_release WHERE id = ?1 AND stage = 'rolling' AND live_at IS NULL`).run(id);
100114 }
101115
116 /** What the deployer timed while promoting: kept on the release still rolling, so a rollback's swap never overwrites them. */
117 export function keepTimings(db: Database, id: number, t: { buildMs: number; proofMs: number; switchMs: number }): void {
118 db.query(`UPDATE keeper_release SET build_ms = ?2, proof_ms = ?3, switch_ms = ?4 WHERE id = ?1 AND stage = 'rolling'`).run(
119 id,
120 Math.max(0, Math.round(t.buildMs)),
121 Math.max(0, Math.round(t.proofMs)),
122 Math.max(0, Math.round(t.switchMs)),
123 );
124 }
125
102126 /** One more `/healthz` probe while the roll runs. */
103127 export function countRollProbe(db: Database, id: number, ok: boolean): void {
104128 db.query(
src/keeper/swapper.ts +12
@@ -57,6 +57,8 @@ export interface SwapperInput {
5757 readonly drain: () => void;
5858 readonly probe?: Readonly<Record<string, string>>;
5959 readonly budgetMs?: number;
60 /** Told how long each stage of a forward swap took, just before this process drains (#11839). */
61 readonly timed?: (t: { buildMs: number; proofMs: number; switchMs: number }) => void;
6062 readonly now?: () => number;
6163 }
6264
@@ -114,16 +116,24 @@ export function swapper(i: SwapperInput): Deployer {
114116 return err(why);
115117 };
116118
119 /** Milliseconds of the stages the last `prepare` ran; a stage it skipped (already admitted) counts 0. */
120 let stages = { buildMs: 0, proofMs: 0 };
121
117122 async function prepare(sha: string, forward: boolean): Promise<Result<string, string>> {
123 stages = { buildMs: 0, proofMs: 0 };
118124 const before = forward ? refused.get(sha) : undefined;
119125 if (before !== undefined) return err(before);
126 const built = now();
120127 const out = await i.checkout(sha);
128 stages.buildMs = now() - built;
121129 if (!out.ok) return err(`checkout failed: ${out.error}`);
122130 if (!forward) return out;
123131 if (admitted.has(sha)) return out;
124132 const deps = dependencyChange(out.value);
125133 if (deps !== undefined) return refuse(sha, deps);
134 const proving = now();
126135 const verdict = await i.preflight(out.value);
136 stages.proofMs = now() - proving;
127137 if (!verdict.ok) return refuse(sha, `preflight failed: ${verdict.line}`);
128138 admitted.add(sha);
129139 return out;
@@ -168,6 +178,7 @@ export function swapper(i: SwapperInput): Deployer {
168178 try {
169179 pin();
170180 const env = { ...i.env, FORGE_DATA: i.dataDir };
181 const switching = now();
171182 const started = await startRelease({
172183 dir: dir.value,
173184 sha,
@@ -179,6 +190,7 @@ export function swapper(i: SwapperInput): Deployer {
179190 });
180191 if (!started.ok) throw new Error(started.line);
181192 swapped = true;
193 if (!backward) i.timed?.({ ...stages, switchMs: now() - switching });
182194 } finally {
183195 endSwap(swapped);
184196 }
src/keeper/wire.ts +5
@@ -32,6 +32,7 @@ import { checkoutRelease, preflightRelease } from "../release/checkout.ts";
3232 import { handover } from "../release/swap.ts";
3333 import { snapshotsOf } from "../snapshot/service.ts";
3434 import { kubeDeployer } from "./deploy.ts";
35 import { inFlight, keepTimings } from "./record.ts";
3536 import { swapper } from "./swapper.ts";
3637 import type { KeeperPorts } from "./keeper.ts";
3738 import { timedGet, verifyUnderLoad } from "./verify.ts";
@@ -96,6 +97,10 @@ export function keeperPorts(app: App, input: { forgeUrl: string; deployment: str
9697 checkout: (sha) => checkoutRelease(app.config.dataDir, dir, sha),
9798 preflight: (tree) => preflightRelease(tree, app.config.dataDir),
9899 drain: () => handover.drain?.(),
100 timed: (t) => {
101 const flying = inFlight(app.db);
102 if (flying !== undefined) keepTimings(app.db, flying.id, t);
103 },
99104 probe: ownProbeHeaders(app.config.dataDir),
100105 ...(app.config.swapBudgetMs === undefined ? {} : { budgetMs: app.config.swapBudgetMs }),
101106 })
src/tui/screens/keeper.ts +16 −4
@@ -5,8 +5,9 @@
55 * its short commit, landed to live (#8042), the known-good version it rolls
66 * back to and what the probes counted.
77 * Gated by `release/gate.ts` as the page is. `h` holds releases with a
8 * reason and `h` with `lift N` lifts an owner's hold, through the same
9 * functions the card and MCP `release_hold` call.
8 * reason and `h` with `lift N` lifts an owner's hold; `now: reason` asks
9 * for a cut at the next tick (#11839), through the same functions the card and
10 * MCP `release_hold` and `release_now` call.
1011 */
1112
1213 import { aheadLine } from "../../flow/alarm_main_ahead.ts";
@@ -14,6 +15,7 @@ import { beatLine, lastBeat } from "../../keeper/beat.ts";
1415 import { stallWords } from "../../perf/boot_stall.ts";
1516 import { blocking, holdAs, liftAs, standingHolds } from "../../keeper/hold.ts";
1617 import { secondsWords, withLandedToLive, type LandedRelease } from "../../keeper/landed.ts";
18 import { releaseNowAs, pendingNow } from "../../keeper/now.ts";
1719 import { listKeeperReleases, type Stage } from "../../keeper/record.ts";
1820 import { mayRead, mayRollBack } from "../../release/gate.ts";
1921 import type { Ctx, Screen } from "../app/app.ts";
@@ -39,6 +41,8 @@ export interface KeeperRead {
3941 readonly heartbeat: string;
4042 /** Main green and ahead of production, and for how long; absent when it is not. */
4143 readonly ahead?: string;
44 /** A standing release-now request, said plainly; absent when there is none. */
45 readonly asked?: string;
4246 readonly releases: readonly LandedRelease[];
4347 }
4448
@@ -46,10 +50,12 @@ export interface KeeperRead {
4650 export function readKeeper(f: Session, now: Date = new Date()): KeeperRead | undefined {
4751 if (!mayRead(f.app, f.user)) return undefined;
4852 const ahead = aheadLine(f.db, now);
53 const asked = pendingNow(f.db, now.getTime());
4954 return {
5055 blocked: blocking(standingHolds(f.db)) ?? "Nothing holds the next cut.",
5156 heartbeat: beatLine(lastBeat(f.db), now),
5257 ...(ahead === undefined ? {} : { ahead }),
58 ...(asked === undefined ? {} : { asked: `Release now asked by ${asked.by}: ${asked.reason}` }),
5359 releases: withLandedToLive(f.db, listKeeperReleases(f.db)),
5460 };
5561 }
@@ -80,6 +86,11 @@ export function holdLine(f: Session, text: string): { ok: boolean; says: string
8086 const done = liftAs(f.app, f.user, Number(lifting[1]));
8187 return done.ok ? { ok: true, says: `Hold ${done.value} lifted.` } : { ok: false, says: done.message };
8288 }
89 const now = /^now:\s*(.*)$/is.exec(text.trim());
90 if (now !== null) {
91 const asked = releaseNowAs(f.app, f.user, now[1]!);
92 return asked.ok ? { ok: true, says: `Release asked for now: ${asked.value.reason}` } : { ok: false, says: asked.message };
93 }
8394 const held = holdAs(f.app, f.user, text);
8495 return held.ok ? { ok: true, says: `Releases held: ${held.value.reason}` } : { ok: false, says: held.message };
8596 }
@@ -94,7 +105,7 @@ export function keeperScreen(f: Session, ctx: Ctx): Screen {
94105 };
95106 const screen: Screen = {
96107 title: "Release keeper",
97 keys: [binding(["j", "k"], "move"), ...(mayRollBack(f.app, f.user) ? [binding(["h"], "hold / lift")] : []), binding(["esc", "q"], "close")],
108 keys: [binding(["j", "k"], "move"), ...(mayRollBack(f.app, f.user) ? [binding(["h"], "hold / lift / now")] : []), binding(["esc", "q"], "close")],
98109 load: async () => refresh(),
99110 render(area, s) {
100111 const c = ctx.theme.c;
@@ -107,6 +118,7 @@ export function keeperScreen(f: Session, ctx: Ctx): Screen {
107118 ...wrap(read.blocked, area.w).slice(0, 3),
108119 ...wrap(read.heartbeat, area.w).slice(0, 2),
109120 ...(read.ahead === undefined ? [] : wrap(read.ahead, area.w).slice(0, 2)),
121 ...(read.asked === undefined ? [] : wrap(read.asked, area.w).slice(0, 2)),
110122 ];
111123 const [head, body] = split(area, "v", [len(lines.length + 1), fill(1)]);
112124 lines.forEach((l, i) => s.text(head!.x, head!.y + i, truncate(l, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w));
@@ -122,7 +134,7 @@ export function keeperScreen(f: Session, ctx: Ctx): Screen {
122134 ctx.push(
123135 promptScreen(ctx, screen, {
124136 title: "Hold releases",
125 label: "a reason holds the next cut · lift N lifts an owner hold",
137 label: "a reason holds the next cut · lift N lifts a hold · now: reason cuts now",
126138 onSubmit: (text) => {
127139 const done = holdLine(f, text);
128140 if (!done.ok) return done.says;
src/web/pages/keeper-panel.ts +13 −4
@@ -10,7 +10,8 @@ import type { Database } from "bun:sqlite";
1010 import { aheadLine } from "../../flow/alarm_main_ahead.ts";
1111 import { beatFact, lastBeat } from "../../keeper/beat.ts";
1212 import { blocking, MAX_REASON, standingHolds } from "../../keeper/hold.ts";
13 import { secondsWords, withLandedToLive } from "../../keeper/landed.ts";
13 import { pendingNow } from "../../keeper/now.ts";
14 import { secondsWords, stagesLine, withLandedToLive } from "../../keeper/landed.ts";
1415 import { listKeeperReleases, type Stage } from "../../keeper/record.ts";
1516 import { stallWords } from "../../perf/boot_stall.ts";
1617 import { html, when, type Html } from "../html.ts";
@@ -23,12 +24,18 @@ const STAGE: Record<Stage, { label: string; tone: Tone }> = {
2324 rolled_back: { label: "rolled back", tone: "danger" },
2425 };
2526
27 /** One muted line under a release: its stages, why it was cut now, its keeper's reason. */
28 function notes(r: { stages: Parameters<typeof stagesLine>[0]; hotfix: string; reason: string }): string {
29 return [stagesLine(r.stages) === "" ? "" : `Stages: ${stagesLine(r.stages)}`, r.hotfix === "" ? "" : `Cut now: ${r.hotfix}`, r.reason].filter((x) => x !== "").join(" · ");
30 }
31
2632 /** `mayHold` is `mayRollBack`: the section then offers the owner's hold and its lift. */
2733 export function keeperPanel(db: Database, mayHold = false, now: Date = new Date()): Html | undefined {
2834 const releases = withLandedToLive(db, listKeeperReleases(db));
2935 const holds = standingHolds(db);
3036 const blocked = blocking(holds);
3137 const beat = lastBeat(db);
38 const asked = pendingNow(db, now.getTime());
3239 const ahead = aheadLine(db, now);
3340 if (releases.length === 0 && blocked === undefined && beat === null && ahead === undefined && !mayHold) return undefined;
3441 const rows = releases.map((r, i) => {
@@ -43,13 +50,14 @@ export function keeperPanel(db: Database, mayHold = false, now: Date = new Date(
4350 <td>${r.worstMs} ms</td>
4451 <td>${stallWords(r.worstStallMs, releases[i + 1]?.worstStallMs)}</td>
4552 <td>${r.secondsDark} s</td>
46 </tr>${when(r.reason !== "", () => html`<tr><td colspan="9" class="small muted">${r.reason}</td></tr>`)}`;
53 </tr>${when(notes(r) !== "", () => html`<tr><td colspan="9" class="small muted">${notes(r)}</td></tr>`)}`;
4754 });
4855 return html`<section class="settings-block" id="keeper">
4956 <h2>Release keeper</h2>
5057 <p class="small ${blocked === undefined ? "muted" : ""}" id="keeper-blocked">${blocked ?? "Nothing holds the next cut."}</p>
5158 <dl class="status-rows" id="keeper-facts">
5259 <div class="status-row"><dt>Release tick</dt><dd id="keeper-heartbeat">${beatFact(beat, now)}</dd></div>
60 ${when(asked !== undefined, () => html`<div class="status-row"><dt>Release now</dt><dd id="keeper-now">asked by ${asked!.by}: ${asked!.reason}</dd></div>`)}
5361 ${when(ahead !== undefined, () => html`<div class="status-row"><dt>Main vs production</dt><dd id="keeper-ahead">${ahead}</dd></div>`)}
5462 </dl>
5563 ${when(rows.length > 0, () => html`<div class="table-scroll" role="region" tabindex="0" aria-label="Keeper releases"><table class="table" data-sticky-lead>
@@ -60,7 +68,8 @@ export function keeperPanel(db: Database, mayHold = false, now: Date = new Date(
6068 (h) => html`<form method="post" action="/releases/hold/${h.id}/lift" class="small">Owner hold ${h.id}: ${h.reason} ${btn("Lift", { type: "submit" })}</form>`,
6169 )}
sentinelcrew outdated

Principles 1 and 3: a second form beside Hold, plus extra stacked rows per release. Consolidate.
6270 <form method="post" action="/releases/hold" class="small">
63 <label>Hold releases <input name="reason" maxlength="${MAX_REASON}" required placeholder="why"></label> ${btn("Hold", { type: "submit" })}
64 </form>`)}
71 <label>Reason <input name="reason" maxlength="${MAX_REASON}" required placeholder="why"></label> ${btn("Hold", { type: "submit" })} ${btn("Release now", { type: "submit", formaction: "/releases/now" })}
72 </form>
73 `)}
6574 </section>`;
6675 }
src/web/pages/releases.ts +11
@@ -20,6 +20,7 @@ import { html, when, type Html } from "../html.ts";
2020 import { flashTo, page } from "../layout.ts";
2121 import { badge, btn, row, sha, time } from "../ui.ts";
2222 import { holdAs, liftAs } from "../../keeper/hold.ts";
23 import { releaseNowAs } from "../../keeper/now.ts";
2324 import { keeperPanel } from "./keeper-panel.ts";
2425
2526 /** The live line (`liveSince`) for the home, or nothing when no release is known to be live. */
@@ -113,6 +114,16 @@ export async function releaseHoldSubmit(ctx: Ctx): Promise<Response> {
113114 : flashTo(ctx, "/releases", { tone: "danger", message: held.message });
114115 }
115116
117 /** POST /releases/now: cut at the keeper's next tick, with a reason, through the function MCP calls (#11839). */
118 export async function releaseNowSubmit(ctx: Ctx): Promise<Response> {
119 const form = await readForm(ctx, "rw");
120 if (form instanceof Response) return form;
121 const asked = releaseNowAs(ctx.app, ctx.auth.user, String(form.get("reason") ?? ""));
122 return asked.ok
123 ? flashTo(ctx, "/releases", { message: `Release asked for now: ${asked.value.reason}` })
124 : flashTo(ctx, "/releases", { tone: "danger", message: asked.message });
125 }
126
116127 /** POST /releases/hold/:id/lift: lift an owner's hold. */
117128 export async function liftSubmit(ctx: Ctx): Promise<Response> {
118129 const form = await readForm(ctx, "rw");
test/mcp-write-scope.test.ts +1
@@ -34,6 +34,7 @@ const ADMIN_ARGS: Record<string, Record<string, unknown>> = {
3434 subscription_headroom: { id: "u1", percent: 30 },
3535 prompts: { repo: "alice/game", action: "remove", id: "p1" },
3636 release_hold: { reason: "freeze for the demo" },
37 release_now: { reason: "hotfix for the demo" },
3738 telemetry_grant: { agent: "robo", on: true },
3839 hooks: { repo: "alice/game", action: "add", url: "https://example.com/h", secret: "0123456789abcdef0123" },
3940 app_domains: { repo: "alice/game", action: "add", host: "game.example.com" },
test/release-now.test.ts added +91
@@ -0,0 +1,91 @@
1 // A hotfix does not wait (#11839): `release now` cuts at the next tick past the spacing, the merge queue and a
2 // snapshot, never past a hold or a full disk; each release says where its time went; the keeper says what holds it.
3 import { describe, expect, test } from "bun:test";
4 import type { Ctx } from "../src/http/context.ts";
5 import { tool as keeperTool } from "../src/http/mcp/tools/release_keeper.ts";
6 import { tool as nowTool } from "../src/http/mcp/tools/release_now.ts";
7 import { decideCut, MIN_INTERVAL_MS, type CadenceInput } from "../src/keeper/cadence.ts";
8 import { setHold } from "../src/keeper/hold.ts";
9 import { stagesLine, stagesOf } from "../src/keeper/landed.ts";
10 import { tick } from "../src/keeper/keeper.ts";
11 import { stepWithBeat } from "../src/keeper/beat.ts";
12 import { keepTimings, listKeeperReleases } from "../src/keeper/record.ts";
13 import { pendingNow } from "../src/keeper/now.ts";
14 import { keeperPanel } from "../src/web/pages/keeper-panel.ts";
15 import { keeperForge, B, A } from "./helpers/keeper.ts";
16
17 const NOW = Date.parse("2026-11-12T12:00:00Z");
18 const base: CadenceInput = { main: B, newest: { version: "0.7.283", sha: A }, live: "0.7.283", inFlight: false, gating: 0, lastCutAt: NOW - 60_000, hold: undefined, now: NOW };
19
20 describe("decideCut with a hotfix", () => {
21 test("skips the spacing, the merge queue and a snapshot", () => {
22 expect(decideCut({ ...base, gating: 2, snapshotMs: 1000 }).cut).toBe(false);
23 expect(decideCut({ ...base, gating: 2, snapshotMs: 1000, hotfix: true })).toEqual({ cut: true, version: "0.7.284" });
24 expect(decideCut({ ...base, lastCutAt: NOW - MIN_INTERVAL_MS + 1 }).cut).toBe(false);
25 expect(decideCut({ ...base, hotfix: true }).cut).toBe(true);
26 });
27 test("still stops for a hold, a release in flight and a full disk", () => {
28 expect(decideCut({ ...base, hotfix: true, hold: "frozen" })).toMatchObject({ cut: false });
29 expect(decideCut({ ...base, hotfix: true, inFlight: true })).toMatchObject({ cut: false });
30 expect(decideCut({ ...base, hotfix: true, diskUsed: 0.9 })).toMatchObject({ cut: false });
31 });
32 });
33
34 describe("release now", () => {
35 test("a person asks, the next tick cuts inside the spacing and the reason stays on the release", async () => {
36 const f = await keeperForge();
37 const ctx = { app: f.app } as unknown as Ctx;
38 f.world.gating = 1;
39 expect(await tick(f.ports)).toContain("merge queue");
40 const asked = await nowTool.run(ctx, f.user, { reason: "covered pages on phones" });
41 expect(asked.ok).toBe(true);
42 expect(pendingNow(f.app.db, f.world.now)?.reason).toBe("covered pages on phones");
43 expect(await tick(f.ports)).toContain("cut 0.7.284");
44 expect(listKeeperReleases(f.app.db)[0]!.hotfix).toBe("covered pages on phones (alice)");
45 expect(pendingNow(f.app.db, f.world.now)).toBeUndefined();
46 });
47 test("an agent, an empty reason and a request that has lapsed cut nothing", async () => {
48 const f = await keeperForge();
49 const ctx = { app: f.app } as unknown as Ctx;
50 expect((await nowTool.run(ctx, { ...f.user, kind: "agent" as const }, { reason: "x" })).ok).toBe(false);
51 expect((await nowTool.run(ctx, f.user, { reason: " " })).ok).toBe(false);
52 expect(pendingNow(f.app.db, f.world.now)).toBeUndefined();
53 await nowTool.run(ctx, f.user, { reason: "later" });
54 expect(pendingNow(f.app.db, Date.now() + 2 * 3_600_000)).toBeUndefined();
55 });
56 test("a hold still stops it", async () => {
57 const f = await keeperForge();
58 setHold(f.app.db, { kind: "owner", version: "0.7.283", reason: "launch freeze", at: f.world.now });
59 await nowTool.run({ app: f.app } as unknown as Ctx, f.user, { reason: "urgent" });
60 expect(await tick(f.ports)).toContain("held:");
61 });
62 });
63
64 describe("what release_keeper shows", () => {
65 test("stage timings, the request, and what the tick last said held it", async () => {
66 const f = await keeperForge();
67 const ctx = { app: f.app } as unknown as Ctx;
68 f.world.gating = 1;
69 await stepWithBeat(f.app.db, async () => ({ line: "the merge queue is landing", outcome: "skipped" }), () => new Date(f.world.now));
70 f.world.gating = 0;
71 await nowTool.run(ctx, f.user, { reason: "fix" });
72 await tick(f.ports);
73 const r = listKeeperReleases(f.app.db)[0]!;
74 keepTimings(f.app.db, r.id, { buildMs: 4000, proofMs: 3000, switchMs: 900 });
75 const answer = await keeperTool.run(ctx, f.user, {});
76 expect(answer.ok).toBe(true);
77 const data = (answer as unknown as { value: Record<string, any> }).value ?? (answer as any).data ?? answer;
78 const rel = JSON.stringify(data);
79 expect(rel).toContain('"buildMs":4000');
80 expect(rel).toContain('"holding"');
81 expect(rel).toContain("the merge queue is landing");
82 const html = String(keeperPanel(f.app.db, true, new Date(f.world.now)));
83 expect(html).toContain("build 4 s");
84 expect(html).toContain("switch 1 s");
85 expect(html).toContain("/releases/now");
86 });
87 test("a stage not measured is left out of the line", () => {
88 const s = stagesOf({ cutAt: "2026-11-12T12:00:00Z", liveAt: "2026-11-12T12:01:10Z", finishedAt: "2026-11-12T12:11:10Z", buildMs: null, proofMs: null, switchMs: null });
89 expect(stagesLine(s)).toBe("cut→live 1 min 10 s · verify 10 min 0 s");
90 });
91 });
test/tui-release-keeper.test.ts +9 −1
@@ -8,7 +8,7 @@ import { tick } from "../src/keeper/keeper.ts";
88 import type { User } from "../src/core/users.ts";
99 import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
1010 import type { Ctx } from "../src/tui/app/app.ts";
11 import { keeperScreen, readKeeper } from "../src/tui/screens/keeper.ts";
11 import { holdLine, keeperScreen, readKeeper } from "../src/tui/screens/keeper.ts";
1212 import { headlessTerminal } from "../src/tui/term/terminal.ts";
1313 import { quietLogs } from "./helpers/globals.ts";
1414 import { keeperForge } from "./helpers/keeper.ts";
@@ -58,4 +58,12 @@ describe("the release keeper screen", () => {
5858 const read = readKeeper(new Session(f.app, identity(f.user)));
5959 expect(read).toEqual({ blocked: "Nothing holds the next cut.", heartbeat: "The release tick has never reported a heartbeat.", releases: [] });
6060 });
61
62 test("`now: reason` asks for a cut at the next tick through the hold prompt, and the card says so", async () => {
63 const f = await keeperForge();
64 const session = new Session(f.app, identity(f.user));
65 expect(holdLine(session, "now: hotfix for the login page")).toEqual({ ok: true, says: "Release asked for now: hotfix for the login page" });
66 expect(readKeeper(session)?.asked).toContain("hotfix for the login page");
67 expect(holdLine(session, "now: ").ok).toBe(false);
68 });
6169 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs