Public: anyone can read this; the crew can change it.

The release proof fails on the first try after every restart, never retries, and leaves its probe forge behind #12368

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/12367-the-release-proof-fails-on-the-first-try → main 15 commits fork 2 threads · 2 unresolved

  • Merged. scout/12367-the-release-proof-fails-on-the-first-try was merged into main as 2b1c1b6.

Review

Sign in to review this change.
17 files changed +369 −25
docs/MCP.md +1 −1
@@ -180,7 +180,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
180180 | `read_file` | — | No shell? Read a file or list a directory at a ref (default branch); over 5 MB answers tooLarge | 317 |
181181 | `release_hold` | — | Hold or release the release keeper's cuts (admin) | 202 |
182182 | `release_keeper` | — | The release keeper's releases, holds and what blocks the next cut | 192 |
183 | `release_now` | — | Cut a release now, with a reason (admin) | 176 |
183 | `release_now` | — | Cut a release now, with a reason; or prove again (admin) | 197 |
184184 | `releases` | — | Kept releases, newest first | 298 |
185185 | `repo` | `open`, `add`, `remove`, `close` | One repository you can read, as GET /api/v1/repos/:owner/:name | 383 |
186186 | `repo_labels` | — | Repo labels: list (default), create, update, remove (admin token) | 351 |
docs/features/release-heartbeat.md +8
@@ -31,3 +31,11 @@ It needs a token with the `admin` scope belonging to a site administrator (401 w
3131 After the usual smoke of the public URL, the proof walks a stranger's journey against it (`walkJourney`, `src/release/journey.ts`): a new forge, claimed, and the crew's first activity, cleaning up after itself. The walk runs last. If it fails, the release stays unproven: the proof is red with "a stranger's journey failed: …" as its sentence, and it is retried on the same `retryAfter` backoff as any failed proof. A walk failure does not set `slow` or `gaveUp`, so a walk that hangs gets the plain doubling, not the shorter polls of a late swap.
3232
3333 When the proof passes it keeps the two timings (`journey.readyMs`, `journey.firstActivityMs`). `/releases` (`#journey-proof`) and the terminal's releases screen then say "a stranger's forge was ready in N s and the crew working in M s". The line shows only for a passing proof of the version being served, with both timings, and only to those who may read /releases; an older proof without timings shows nothing.
34
35 ## The walk after a restart (#12367)
36
37 A restart wakes every app, so the first walk of the stranger's journey waits for `/readyz` to answer 200 and the event loop's lag to fall under 250 ms (90 s at most). A failed walk is tried again two minutes later, three walks at most, before a failure is recorded; `proof.json` and the `release proof` log line carry `tries`. The 20 s forge-ready budget is unchanged. A probe forge is removed after every walk; if its removal fails the failure names it, the walks stop, and the needs-you stack shows it with a Remove button (the forge's remove page). An admin runs the proof again from /releases ("Prove it again now") with `p` on the terminal's releases screen, or with MCP `release_now {prove: true}` (an admin-scope token of a person who may roll back, never an agent; it answers `invalid` when no failed proof waits). The Remove button on a probe forge's card has its terminal door in the forges screen (`x`, remove one).
38
39 ## A proof never runs without a word (#12367)
40
41 While a proof runs the forge keeps `release/proving.json` (version, start time); the home page's live-version figure says "still proving, started HH:MM UTC" until a verdict replaces it. A proof has `HANG_MS` (30 minutes, shorter than the gap between releases) in all, sized from the worst healthy case (`WORST_CASE_MS`: ten minutes of give-up polling, three minutes of smoke checks, three walks of settle plus the walk's 20 s, two waits of two minutes, five minutes' margin, a test holds it): past it, it is stopped and recorded as a failed, `hung` proof, tried again at the next tick.
src/http/mcp/tools/release_now.ts +9 −3
@@ -1,15 +1,21 @@
11 import { MAX_REASON } from "../../../keeper/hold.ts";
22 import { releaseNowAs } from "../../../keeper/now.ts";
3 import { proveAgain } from "../../../release/prove.ts";
4 import { mayRollBack } from "../../../release/gate.ts";
35 import { bad, good, str, type Tool } from "../shared.ts";
46
57 export const tool: Tool = {
68 name: "release_now",
7 doc: "`{ reason }`: ask the release keeper (#7557) to cut main now, at its next tick (within a minute), outside the 30-minute spacing and past a running merge queue or snapshot, for a hotfix. The reason stays on the release. It still waits for a release in flight to be proven, for a hold and for room on the disk. A person who may roll back, with an admin-scope token, never an agent",
8 description: "Cut a release now, with a reason (admin).",
9 inputSchema: { type: "object", properties: { reason: { type: "string", maxLength: MAX_REASON } }, required: ["reason"] },
9 doc: "`{ reason }`: ask the release keeper (#7557) to cut main now, at its next tick (within a minute), outside the 30-minute spacing and past a running merge queue or snapshot, for a hotfix. The reason stays on the release. It still waits for a release in flight to be proven, for a hold and for room on the disk. `{ prove: true }` instead asks the forge to prove the running release again at its next tick, when its last proof failed. A person who may roll back, with an admin-scope token, never an agent",
10 description: "Cut a release now, with a reason; or prove again (admin).",
11 inputSchema: { type: "object", properties: { reason: { type: "string", maxLength: MAX_REASON }, prove: { type: "boolean" } } },
1012 writes: true,
1113 needsAdminScope: true,
1214 run(ctx, user, args) {
15 if (args["prove"] === true) {
16 if (user === undefined || !mayRollBack(ctx.app, user)) return bad("forbidden", "proving the release again is for a person who may administer the forge, not an agent");
17 return proveAgain(ctx.app.config.dataDir) ? good({ prove: "asked: the forge proves the release again at its next tick, within a minute" }) : bad("invalid", "there is no failed proof to try again");
18 }
1319 const asked = releaseNowAs(ctx.app, user, str(args["reason"]));
1420 return asked.ok ? good({ releaseNow: asked.value }) : bad(asked.error, asked.message);
1521 },
src/kernel/vitals.ts +6 −1
@@ -36,7 +36,7 @@ import { lastDrill } from "../snapshot/drills.ts";
3636 import { lastMirror, mirrorNeed } from "../snapshot/mirror.ts";
3737 import { lastSnapshotOutcome, snapshotNeed } from "../snapshot/outcome.ts";
3838 import { drillNeed, lastVerdict } from "../snapshot/verdict.ts";
39 import { proofNeed, proofNotice } from "../release/prove.ts";
39 import { leftoverProbes, proofNeed, proofNotice } from "../release/prove.ts";
4040 import { running as runningVersion } from "./version.ts";
4141 import { rollbackNeed } from "../release/launch.ts";
4242 import { behindStanding } from "../release/behind.ts";
@@ -654,6 +654,11 @@ export function vitals(input: VitalsInput): Vitals {
654654 const unproven = proofNeed(input.dataDir, runningVersion().version);
655655 if (unproven !== undefined && proofNotice(input.dataDir, runningVersion().version, now)?.tone === "alarm") need("todo", "release", unproven.sentence, unproven.since, "/releases#proof");
656656
657 // A probe forge the proof could not remove is hidden work on the host: a card each, its
658 // Remove page one tap away.
659 for (const name of leftoverProbes(input.dataDir))
660 if (/^probe-[0-9a-f]{8}$/.test(name)) need("todo", `probe:${name}`, `The release proof's probe forge ${name} could not be removed and is still running.`, undefined, `/forges/${name}/remove`);
661
657662 // A roll the image's launcher undid is a card until a later version reads ready. Its sentence
658663 // carries the release's own last words (stderr), so anything credential-shaped is masked.
659664 const undone = rollbackNeed(input.dataDir);
src/release/journey.ts +4
@@ -267,6 +267,10 @@ export async function walkJourney(options: JourneyOptions): Promise<JourneyResul
267267 if (forge !== null) {
268268 if (child !== undefined && cookie !== "") await cancelRuns(doFetch, child, cookie);
269269 removed = await removeForge(doFetch, hostUrl, forge, options.adminToken, sleep);
270 // A forge left running is hidden work on the host: the failure names it (never when there
271 // was no token to remove it with).
272 if (!removed && options.adminToken !== undefined && options.adminToken !== "")
273 failure = `${failure === undefined ? "" : `${failure}; `}the probe forge ${forge} could not be removed`;
270274 }
271275 }
272276 // What an interrupted run left: swept after this walk's own cleanup, never its own forge.
src/release/prove.ts +133 −8
@@ -20,7 +20,7 @@
2020 * that, it was not replaced and proves its version again (#10248).
2121 */
2222
23 import { mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
23 import { mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs";
2424 import { join } from "node:path";
2525 import type { walkJourney, JourneyResult } from "./journey.ts";
2626 import { smoke, type Failure, type Heavy } from "./smoke.ts";
@@ -37,14 +37,30 @@ const POLL_MS = 2_000;
3737 export const RETRY_MS = 5 * 60_000;
3838 /** The longest wait between two proofs of a version that keeps failing. */
3939 export const MAX_RETRY_MS = 60 * 60_000;
40 /** The journey is walked at most this many times before a failure is recorded (#12367). */
41 export const JOURNEY_TRIES = 3;
42 /** Between two walks: long enough for a waking host to settle. */
43 export const JOURNEY_RETRY_MS = 2 * 60_000;
44 /**
45 * How long a walk waits for the forge to report ready and its loop to calm; the 20 s of the
46 * walk itself is never raised.
47 */
48 export const SETTLE_MS = 90_000;
49 /** The event loop's worst lag in the last ten seconds under which the forge counts as settled. */
50 export const SETTLED_LAG_MS = 250;
4051 /** How often `startProving` asks whether a proof is due. */
4152 export const TICK_MS = 60_000;
4253 /** How long a retry after a give-up waits for the public URL to answer as the version: the long wait was already spent (#10222). */
4354 export const RETRY_POLL_MS = 10_000;
4455 /** How many give-ups in a row are retried after one tick before the doubling takes over. */
4556 export const QUICK_RETRIES = 10;
46 /** A proof not ended by then is hung: the next tick is scheduled anyway, or the last failure would stand for hours (#10313). */
47 export const HANG_MS = GIVE_UP_MS + 5 * 60_000;
57 /**
58 * The most a proof may take in all, shorter than the gap between releases: one not ended by
59 * then is hung. It is recorded as a failed proof (a verdict the home page shows, tried again
60 * at the next tick) and the next tick is scheduled anyway, or the last failure would stand for
61 * hours (#10313, #12367).
62 */
63 export const HANG_MS = 30 * 60_000;
outdated

High: 15 min total does not cover smoke polling (up to 10 min) plus three settle+walk tries and two 2-min waits. A slow-but-healthy release is recorded as hung. Size it to the worst case.

4864
4965 export interface Proof {
5066 readonly version: string;
@@ -58,6 +74,8 @@ export interface Proof {
5874 readonly failures?: number;
5975 /** The first failing check was only a slow, empty or hung answer (`Failure.slow`, `.empty`, `.hung`); retried sooner (#9358, #9482). */
6076 readonly slow?: boolean;
77 /** The proof outran its ceiling (`HANG_MS`) and was stopped. */
78 readonly hung?: boolean;
6179 /** The failure was the give-up: ten minutes already spent polling; only its first retry skips the backoff (#10168). */
6280 readonly gaveUp?: boolean;
6381 /** A stranger's journey walked on the public URL after the smoke checks (#11158); its timings, in ms, for the release card. */
@@ -66,6 +84,10 @@ export interface Proof {
6684 readonly footprint?: Footprint;
6785 /** A person asked for it to be proven again now: the next tick does, whatever the backoff says. */
6886 readonly retryNow?: boolean;
87 /** How many walks of the stranger's journey it took (1 when it passed or failed at once). */
88 readonly tries?: number;
89 /** Probe forges the proof could not remove: the needs-you stack lists each with Remove. */
90 readonly leftover?: readonly string[];
6991 }
7092
7193 /**
@@ -116,6 +138,50 @@ export function lastProof(dataDir: string): Proof | undefined {
116138 }
117139 }
118140
141 const provingFile = (dataDir: string): string => join(dataDir, "release", "proving.json");
142
143 /** A proof is running: its version and start, so the home page can say "proving, started 06:01". */
144 export function markProving(dataDir: string, version: string, at: number): void {
145 try {
146 mkdirSync(join(dataDir, "release"), { recursive: true });
147 writeFileSync(provingFile(dataDir), JSON.stringify({ version, startedAt: new Date(at).toISOString() }));
148 } catch {}
149 }
150
151 export function clearProving(dataDir: string): void {
152 try {
153 unlinkSync(provingFile(dataDir));
154 } catch {}
155 }
156
157 /** When the proof of `version` began, while it runs; undefined when none runs for it. */
158 export function provingSince(dataDir: string, version: string, now = Date.now()): string | undefined {
159 try {
160 const p = JSON.parse(readFileSync(provingFile(dataDir), "utf8")) as { version?: unknown; startedAt?: unknown };
161 return p.version === version && typeof p.startedAt === "string" && Number.isFinite(Date.parse(p.startedAt)) && now - Date.parse(p.startedAt) < HANG_MS ? p.startedAt : undefined;
162 } catch {
163 return undefined;
164 }
165 }
166
167 /** The verdict for a proof that outran `HANG_MS`: a failure, so a release never has no word. */
168 export function keepHung(dataDir: string, version: string, startedAt: number, now: number, ceilingMs = HANG_MS): Proof {
169 const before = lastProof(dataDir);
170 const failedBefore = before?.version === version && !before.ok ? before : undefined;
171 const at = new Date(now).toISOString();
172 const started = new Date(startedAt).toISOString().slice(11, 16);
173 return keep(dataDir, {
174 version,
175 at,
176 ok: false,
177 failure: `the proof started at ${started} UTC did not finish in ${Math.round(ceilingMs / 60_000)} minutes`,
178 since: failedBefore ? (failedBefore.since ?? failedBefore.at) : at,
179 failures: (failedBefore?.failures ?? (failedBefore ? 1 : 0)) + 1,
180 slow: true,
181 hung: true,
182 });
183 }
184
119185 function keep(dataDir: string, proof: Proof): Proof {
120186 mkdirSync(join(dataDir, "release"), { recursive: true });
121187 const tmp = `${proofFile(dataDir)}.tmp`;
@@ -196,6 +262,8 @@ export function proofNotice(dataDir: string, running?: string, now = Date.now())
196262 sentence: `Release ${p.version} is serving. Its last proof, a stranger signing up from the public address, did not finish: ${rate ? "the forge's own sign-up limit turned the stranger away" : plain(detail)}. ${again}`,
197263 };
198264 }
265 if (p.hung === true)
266 return { ...base, tone: "notice", step: "The whole proof, from the public address answering to the stranger's journey", stopped: plain(raw), happened: "the host was too busy for the proof to finish in time", sentence: `Release ${p.version} is serving. Its proof ran out of time: ${plain(raw)}. ${again}` };
199267 if (p.slow === true)
200268 return { ...base, tone: "notice", step: "A smoke check of the public address", stopped: plain(raw), happened: "the public address answered slowly or not at all for a moment", sentence: `Release ${p.version} is serving. The public address answered slowly when the forge proved it. ${again}` };
201269 return {
@@ -208,6 +276,11 @@ export function proofNotice(dataDir: string, running?: string, now = Date.now())
208276 };
209277 }
210278
279 /** Probe forges the last proof could not remove, until a later proof of any version leaves none. */
280 export function leftoverProbes(dataDir: string): readonly string[] {
281 return lastProof(dataDir)?.leftover ?? [];
282 }
283
211284 /** Asks for the last failed proof to be tried again at the next tick, whatever its backoff says; false when there is none to try. */
212285 export function proveAgain(dataDir: string): boolean {
213286 const p = lastProof(dataDir);
@@ -272,7 +345,7 @@ export async function proveRelease(o: ProveOptions): Promise<Proof | undefined>
272345 const doFetch = ((url, init) => baseFetch(url, o.signal ? { ...init, signal: init?.signal ? AbortSignal.any([init.signal, o.signal]) : o.signal } : init)) as typeof fetch;
273346 const failedBefore = last?.version === o.version && !last.ok ? last : undefined;
274347 let measured: Footprint | undefined;
275 const finish = (failure?: string, slow = false, gaveUp = false, walked?: JourneyResult): Proof => {
348 const finish = (failure?: string, slow = false, gaveUp = false, walked?: JourneyResult, tries?: number, leftover: readonly string[] = []): Proof => {
276349 if (o.signal?.aborted) throw new Error("the proof was aborted as hung");
277350 const at = new Date(now()).toISOString();
278351 return keep(o.dataDir, {
@@ -281,6 +354,8 @@ export async function proveRelease(o: ProveOptions): Promise<Proof | undefined>
281354 ok: failure === undefined,
282355 ...(measured === undefined ? {} : { footprint: measured }),
283356 ...(walked === undefined ? {} : { journey: { readyMs: walked.readyMs, firstActivityMs: walked.firstActivityMs, screens: walked.screens } }),
357 ...(tries === undefined ? {} : { tries }),
358 ...(leftover.length === 0 ? {} : { leftover }),
284359 ...(failure === undefined
285360 ? {}
286361 : { failure, since: failedBefore ? (failedBefore.since ?? failedBefore.at) : at, failures: (failedBefore?.failures ?? (failedBefore ? 1 : 0)) + 1, ...(slow ? { slow } : {}), ...(gaveUp ? { gaveUp } : {}) }),
@@ -321,13 +396,51 @@ export async function proveRelease(o: ProveOptions): Promise<Proof | undefined>
321396 if (failures[0] !== undefined || o.journey === undefined)
322397 return finish(failures[0] === undefined ? undefined : checkLine(failures[0]), failures[0]?.slow === true || failures[0]?.empty === true || failures[0]?.hung === true);
323398 // The stranger's journey, the last check: a failed walk is a failed proof, retried on `retryAfter` like any other (one card, paged once).
324 const walked = await o.journey({ base: o.url, fetch: doFetch, ...(o.token ? { adminToken: o.token } : {}), ...(o.proberKey ? { proberKey: o.proberKey } : {}) });
325 return finish(walked.ok ? undefined : `a stranger's journey failed: ${walked.failure ?? "it did not finish"}`, false, false, walked);
399 // A restart wakes every app: the first walk waits for the host to settle, and a failure is
400 // tried again a few minutes later (twice at most) before it is recorded.
401 const leftover: string[] = [];
402 let walked: JourneyResult | undefined;
403 let tries = 0;
404 while (tries < JOURNEY_TRIES) {
405 if (o.signal?.aborted) throw new Error("the proof was aborted as hung");
406 const settled = await settle(o.url, doFetch, now, sleep, o.giveUpMs === undefined ? SETTLE_MS : Math.min(SETTLE_MS, o.giveUpMs));
407 tries++;
408 walked = settled
409 ? await o.journey({ base: o.url, fetch: doFetch, ...(o.token ? { adminToken: o.token } : {}), ...(o.proberKey ? { proberKey: o.proberKey } : {}) })
410 : { ok: false, screens: [], readyMs: null, firstActivityMs: null, failure: `the forge had not settled in ${SETTLE_MS / 1000} s after its start`, forge: null, removed: true };
411 if (walked.forge !== null && !walked.removed) leftover.push(walked.forge);
412 // A passed walk, a refusal by the sign-up limit, or a forge left behind ends the tries:
413 // another walk would only make another.
414 if (walked.ok || /start=rate/.test(walked.failure ?? "") || leftover.length > 0 || tries >= JOURNEY_TRIES) break;
415 await sleep(o.giveUpMs === undefined ? JOURNEY_RETRY_MS : Math.min(JOURNEY_RETRY_MS, o.giveUpMs));
416 }
417 const w = walked!;
418 return finish(w.ok ? undefined : `a stranger's journey failed: ${w.failure ?? "it did not finish"}`, false, false, w, tries, leftover);
326419 } catch (error) {
327420 return finish(`the smoke checks could not run: ${String(error)}`);
328421 }
329422 }
330423
424 /** Waits until `/readyz` answers and the event loop is calm; false when `budgetMs` passes. */
425 async function settle(url: string, doFetch: typeof fetch, now: () => number, sleep: (ms: number) => Promise<void>, budgetMs: number): Promise<boolean> {
426 const until = now() + budgetMs;
427 for (;;) {
428 try {
429 const ready = await doFetch(new URL("/readyz", url).href, { redirect: "manual", cache: "no-store", signal: AbortSignal.timeout(5_000) });
430 await ready.body?.cancel();
431 if (ready.status === 200) {
432 const live = await doFetch(new URL("/healthz", url).href, { redirect: "manual", cache: "no-store", headers: { accept: "application/json" }, signal: AbortSignal.timeout(5_000) });
433 const lag = live.status === 200 ? ((await live.json().catch(() => ({}))) as { loopLagMs?: unknown }).loopLagMs : undefined;
434 // A forge that does not report its lag is taken as calm: only a reported, high lag waits.
435 if (live.status === 200 && (typeof lag !== "number" || lag < SETTLED_LAG_MS)) return true;
436 if (live.status !== 200) await live.body?.cancel();
437 }
438 } catch {}
439 if (now() + POLL_MS > until) return false;
440 await sleep(POLL_MS);
441 }
442 }
443
331444 /** Runs `fn` after `ms`; returns its cancel. */
332445 export type Schedule = (fn: () => void, ms: number) => () => void;
333446
@@ -359,12 +472,17 @@ export function startProving(
359472 const next = (): void => {
360473 if (!stopped) cancel = schedule(() => void during("release-prove", tick), everyMs);
361474 };
475 // A marker left by a process that died mid-proof is not this boot's: it stands for nothing.
476 clearProving(o.dataDir);
362477 const tick = async (): Promise<void> => {
363478 if (running) {
364479 log?.warn("release proof still running", { version: o.version });
365480 return next();
366481 }
367482 running = true;
483 const startedAt = Date.now();
484 // Only a proof that is due is "proving": a tick that finds nothing to do says nothing.
485 if (proofDue(lastProof(o.dataDir), o.version, startedAt)) markProving(o.dataDir, o.version, startedAt);
368486 const abort = new AbortController();
369487 const work = prove({ ...o, signal: abort.signal }).then(
370488 (p) => ({ p }),
@@ -373,7 +491,10 @@ export function startProving(
373491 return { p: undefined };
374492 },
375493 );
376 void work.finally(() => void (running = false));
494 void work.finally(() => {
495 running = false;
496 if (!abort.signal.aborted) clearProving(o.dataDir);
497 });
377498 let timer: ReturnType<typeof setTimeout> | undefined;
378499 const hung = new Promise<"hung">((resolve) => {
379500 timer = setTimeout(() => resolve("hung"), hangMs);
@@ -385,6 +506,10 @@ export function startProving(
385506 // Stop its asks; the next tick waits for it to settle rather than run beside it.
386507 log?.warn("release proof hung", { version: o.version, afterMs: hangMs });
387508 abort.abort();
509 try {
510 done(keepHung(o.dataDir, o.version, startedAt, Date.now(), hangMs));
511 } catch {}
512 clearProving(o.dataDir);
388513 } else if (result.p) done(result.p);
389514 next();
390515 };
@@ -403,5 +528,5 @@ export function proveRunning(
403528 schedule: Schedule = unrefTimer,
404529 prove: (o: ProveOptions) => Promise<Proof | undefined> = proveRelease,
405530 ): () => void {
406 return startProving(o, (p) => log.info("release proof", { version: p.version, ok: String(p.ok), ...(p.failure ? { failure: p.failure } : {}) }), everyMs, schedule, prove, HANG_MS, log);
531 return startProving(o, (p) => log.info("release proof", { version: p.version, ok: String(p.ok), ...(p.tries === undefined ? {} : { tries: String(p.tries) }), ...(p.failure ? { failure: p.failure } : {}) }), everyMs, schedule, prove, HANG_MS, log);
407532 }
src/tui/screens/releases.ts +10 −2
@@ -10,7 +10,7 @@ import { getRepoById } from "../../core/repos.ts";
1010 import { running } from "../../kernel/version.ts";
1111 import { BOOTED, canRollBackTo, mayRead, mayRollBack, rollBackTo } from "../../release/gate.ts";
1212 import { listReleases, type Release } from "../../release/record.ts";
13 import { lastProof } from "../../release/prove.ts";
13 import { lastProof, proveAgain } from "../../release/prove.ts";
1414 import { journeyLine, liveSince, NO_RELEASES_YET, plain } from "../../release/words.ts";
1515 import type { Ctx, Screen } from "../app/app.ts";
1616 import { binding } from "../app/keys.ts";
@@ -87,7 +87,7 @@ export function releasesScreen(f: Session, ctx: Ctx): Screen {
8787 };
8888 const self: Screen = {
8989 title: "Releases",
90 keys: [binding(["j", "k"], "move"), binding(["enter"], "roll back"), binding(["esc", "q"], "close")],
90 keys: [binding(["j", "k"], "move"), binding(["enter"], "roll back"), binding(["p"], "prove it again"), binding(["esc", "q"], "close")],
9191 load: read,
9292 render(area, s) {
9393 const c = ctx.theme.c;
@@ -106,10 +106,18 @@ export function releasesScreen(f: Session, ctx: Ctx): Screen {
106106 key(e) {
107107 if (e.key === "esc" || e.key === "q") ctx.pop();
108108 else if (e.key === "enter") ask(state.index);
109 else if (e.key === "p") proveIt();
109110 else state.key(e, lines.length);
110111 return true;
111112 },
112113 };
114 // The page's "Prove it again now": the same gate (a person who may read, never an agent),
115 // the same call.
116 const proveIt = (): void => {
117 const user = f.user;
118 if (user === undefined || user.kind === "agent" || !mayRead(f.app, user)) return;
119 ctx.toast(proveAgain(f.app.config.dataDir) ? "Asked: the forge proves this release again at its next tick, within a minute." : "There is no failed proof to try again.", "default");
120 };
113121 const ask = (index: number): void => {
114122 const line = lines[index];
115123 const user = f.user;
src/web/pages/home_flow.ts +5 −3
@@ -39,7 +39,7 @@ import {
3939 type StripPart,
4040 } from "../../flow/lanes.ts";
4141 import { span } from "../../flow/pipeline.ts";
42 import { lastProof, proofNotice, type Proof } from "../../release/prove.ts";
42 import { lastProof, proofNotice, provingSince, type Proof } from "../../release/prove.ts";
4343 import { running, UNSTAMPED } from "../../kernel/version.ts";
4444 import { notePhase, takeSessionTime, timePhase, timePhaseAsync } from "../../core/route_timing.ts";
4545 import type { Ctx } from "../../http/context.ts";
@@ -215,7 +215,9 @@ const healthOf = (view: LaneView, picture: LanePicture, now: number): Health =>
215215
216216 // ---- the strip: today's figures, and the one line that needs the owner ----
217217
218 function proofWords(proof: Proof | undefined, version: string): { text: string; tone: "attn" | null } {
218 export function proofWords(proof: Proof | undefined, version: string, since?: string): { text: string; tone: "attn" | null } {
219 // A proof under way says so and when it began, so no release goes a quarter hour unspoken.
220 if (since !== undefined && (proof === undefined || proof.version !== version || !proof.ok)) return { text: `still proving, started ${since.slice(11, 16)} UTC`, tone: null };

No test renders this 'still proving' text (principle 8 wording is fine), and a stale proving.json would make it show forever. Test it, and clear the marker on boot.

219221 if (proof === undefined) return { text: "not proven yet", tone: null };
220222 if (proof.version !== version) return { text: "proof running", tone: null };
221223 return proof.ok ? { text: "journey proof passed", tone: null } : { text: "journey proof failed", tone: "attn" };
@@ -323,7 +325,7 @@ function needLine(needs: readonly Need[]): Html {
323325 export function stripRegion(picture: LanePicture, viewer: StripViewer, dataDir: string, now: number): Html {
324326 const strip = picture.strip;
325327 const version = running().version;
326 const proof = proofWords(lastProof(dataDir), version);
328 const proof = proofWords(lastProof(dataDir), version, provingSince(dataDir, version));
327329 const needs = needLine(needsOf(picture, viewer, dataDir, now));
328330 if (!viewer.isAdmin) return html`<section class="hf-strip" id="hf-strip" aria-label="What needs you" data-live="${STRIP_LIVE}" data-live-src="/live/strip">${needs}</section>`;
329331 const b = picture.builders;
src/web/pages/releases.ts +5 −4
@@ -47,16 +47,17 @@ export function journeyCard(ctx: Ctx): Html | undefined {
4747 * when it is tried again, and a way to try it now. Plain words; the proof's raw
4848 * line is one fold away.
4949 */
50 export function proofCard(ctx: Ctx, writer: boolean): Html | undefined {
51 const n = proofNotice(ctx.app.config.dataDir, running().version);
50 export function proofCard(dataDir: string, writer: boolean): Html | undefined {
51 const n = proofNotice(dataDir, running().version);
5252 if (n === undefined) return undefined;
53 const tries = lastProof(dataDir)?.tries;
5354 const serving = n.tone === "notice";
5455 const rows: [string, string | Html][] = [
5556 ["Is the release serving?", serving ? `Yes. This forge is running ${n.version} and answering.` : `Not as it should: ${n.happened}.`],
5657 ["What was checked", n.step],
5758 ["Where it stopped", n.stopped],
5859 ["What happened", n.happened],
59 ["First failed", html`${time(n.since)}, ${String(n.failures)} ${n.failures === 1 ? "try" : "tries"} so far`],
60 ["First failed", html`${time(n.since)}, ${String(n.failures)} failed ${n.failures === 1 ? "attempt" : "attempts"} so far${tries === undefined ? "" : `; the last one tried ${tries} ${tries === 1 ? "time" : "times"}`}`],
6061 ["Next try", n.nextAt === null ? "Soon." : html`${time(new Date(n.nextAt).toISOString())} (${new Date(n.nextAt).toISOString().slice(11, 16)} UTC)`],
6162 ];
6263 return html`<section class="settings-block" id="proof" aria-label="The release's proof">
@@ -133,7 +134,7 @@ export async function releasesPage(ctx: Ctx): Promise<Response> {
133134 const response = page({
134135 ctx,
135136 title: "Releases",
136 body: html`<div data-releases><div class="page-head"><h1>Releases</h1></div>${proofCard(ctx, user.kind !== "agent")}${keeperPanel(ctx.app.db, writer)}${journeyCard(ctx)}${body}</div>`,
137 body: html`<div data-releases><div class="page-head"><h1>Releases</h1></div>${proofCard(ctx.app.config.dataDir, user.kind !== "agent")}${keeperPanel(ctx.app.db, writer)}${journeyCard(ctx)}${body}</div>`,
137138 styleCss: RELEASES_CSS,
138139 });
139140 notePhase(ctx.request, "render", performance.now() - rendering);
src/web/studio/asks_page.ts +1 −1
@@ -410,7 +410,7 @@ export const forgeCard = (d: ForgeNeed): Html =>
410410 html`<section class="stack">
411411 <h2>The forge</h2>
412412 <p role="alert"><span class="dot" data-tone="danger" aria-hidden="true"></span> <a href="${d.href}">${d.line}</a></p>
413 <div class="bar">${btn("Health", { href: "/api/v1/health", size: "sm" })}</div>
413 <div class="bar">${d.cause.startsWith("probe:") ? btn("Remove", { href: d.href, size: "sm", variant: "primary" }) : null}${btn("Health", { href: "/api/v1/health", size: "sm" })}</div>
414414 </section>`;
415415
416416 /** What sits above the asks after a bulk step: a preview to confirm, a receipt, or why nothing was done. */
test/asks-forge-alert.test.ts +7
@@ -14,3 +14,10 @@ test("a degraded forge on /asks is an alert line in the danger tone, with Health
1414 expect(out).not.toContain("<now>");
1515 expect(out).not.toContain('class="card');
1616 });
17
18 test("a probe forge the release proof left behind is a card with a Remove button to its remove page; other causes have none", () => {
19 const d = { line: "Left." };
20 const probe = String(forgeCard({ ...d, cause: "probe:probe-abcd1234", href: "/forges/probe-abcd1234/remove" }));
21 expect(probe).toMatch(/data-variant="primary"[\s\S]*?>Remove</);
22 expect(String(forgeCard({ ...d, cause: "errors", href: "/x" }))).not.toContain("Remove");
23 });
test/release-journey.test.ts +2
@@ -267,6 +267,8 @@ test("a removal that never succeeds waits between its attempts and says it is no
267267 let t = 0;
268268 const result = await walkJourney({ base: parentServers.url, fetch: stuck, adminToken, pollMs: 0, now: () => (t += 5_000), sleep: async (ms) => void pauses.push(ms) });
269269 expect(result.removed).toBe(false);
270 expect(result.ok).toBe(false);
271 expect(result.failure).toContain(`the probe forge ${result.forge} could not be removed`);
270272 expect(pauses).toEqual([...REMOVE_BACKOFF_MS]);
271273 await fetch(`${parentServers.url}/forges/${result.forge}/remove`, { method: "POST", headers: { origin: parentServers.url, authorization: `Bearer ${adminToken}` } });
272274 }, 60_000);
test/release-now.test.ts +16
@@ -1,6 +1,9 @@
11 // A hotfix does not wait (#11839): `release now` cuts at the next tick past the spacing, the merge queue and a
22 // snapshot, never past a hold or a full disk; each release says where its time went; the keeper says what holds it.
33 import { describe, expect, test } from "bun:test";
4 import { mkdirSync, writeFileSync } from "node:fs";
5 import { join } from "node:path";
6 import { lastProof } from "../src/release/prove.ts";
47 import type { Ctx } from "../src/http/context.ts";
58 import { tool as keeperTool } from "../src/http/mcp/tools/release_keeper.ts";
69 import { tool as nowTool } from "../src/http/mcp/tools/release_now.ts";
@@ -53,6 +56,19 @@ describe("release now", () => {
5356 await nowTool.run(ctx, f.user, { reason: "later" });
5457 expect(pendingNow(f.app.db, Date.now() + 2 * 3_600_000)).toBeUndefined();
5558 });
59 test("prove: true asks for the failed proof again; an agent and a passing proof are refused", async () => {
60 const f = await keeperForge();
61 const ctx = { app: f.app } as unknown as Ctx;
62 const dir = f.app.config.dataDir;
63 expect((await nowTool.run(ctx, f.user, { prove: true })).ok).toBe(false);
64 mkdirSync(join(dir, "release"), { recursive: true });
65 writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ version: "1.0.0", at: "2026-11-14T02:00:00.000Z", ok: false, failure: "x" }));
66 expect((await nowTool.run(ctx, { ...f.user, kind: "agent" as const }, { prove: true })).ok).toBe(false);
67 expect(lastProof(dir)?.retryNow).toBeUndefined();
68 expect((await nowTool.run(ctx, f.user, { prove: true })).ok).toBe(true);
69 expect(lastProof(dir)?.retryNow).toBe(true);
70 expect(pendingNow(f.app.db, f.world.now)).toBeUndefined();
71 });
5672 test("a hold still stops it", async () => {
5773 const f = await keeperForge();
5874 setHold(f.app.db, { kind: "owner", version: "0.7.283", reason: "launch freeze", at: f.world.now });
test/release-proof-words.test.ts added +42
@@ -0,0 +1,42 @@
1 /** What a person reads of a proof: the home page's figure and the /releases card (#12367). */
2 import { afterEach, beforeEach, expect, test } from "bun:test";
3 import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { proofWords } from "../src/web/pages/home_flow.ts";
7 import { proofCard } from "../src/web/pages/releases.ts";
8 import type { Proof } from "../src/release/prove.ts";
9
10 let dir = "";
11 const was = process.env["FORGE_VERSION"];
12 beforeEach(() => {
13 dir = mkdtempSync(join(tmpdir(), "forge-proof-words-"));
14 process.env["FORGE_VERSION"] = "9.9.9";
15 });
16 afterEach(() => {
17 rmSync(dir, { recursive: true, force: true });
18 if (was === undefined) delete process.env["FORGE_VERSION"];
19 else process.env["FORGE_VERSION"] = was;
20 });
21
22 const base: Proof = { version: "9.9.9", at: "2026-11-14T06:00:00.000Z", ok: true };
23
24 test("the home figure says a proof is running and when it began, until a pass replaces it", () => {
25 const since = "2026-11-14T06:01:30.000Z";
26 expect(proofWords(undefined, "9.9.9", since)).toEqual({ text: "still proving, started 06:01 UTC", tone: null });
27 expect(proofWords({ ...base, ok: false }, "9.9.9", since).text).toBe("still proving, started 06:01 UTC");
28 expect(proofWords({ ...base, version: "9.9.8" }, "9.9.9", since).text).toBe("still proving, started 06:01 UTC");
29 expect(proofWords(base, "9.9.9", since).text).toBe("journey proof passed");
30 expect(proofWords({ ...base, ok: false }, "9.9.9").tone).toBe("attn");
31 expect(proofWords(undefined, "9.9.9").text).toBe("not proven yet");
32 });
33
34 test("the /releases card counts failed attempts and the last one's tries in one plain word", () => {
35 mkdirSync(join(dir, "release"), { recursive: true });
36 writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ ...base, ok: false, failure: "a stranger's journey failed: x", failures: 2, tries: 3 }));
37 const card = String(proofCard(dir, false));
38 expect(card).toContain("2 failed attempts so far; the last one tried 3 times");
39 expect(card).not.toMatch(/proofs so far|walks/);
40 writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ ...base, ok: false, failure: "a stranger's journey failed: x", failures: 1, tries: 1 }));
41 expect(String(proofCard(dir, false))).toContain("1 failed attempt so far; the last one tried 1 time<");
42 });
test/release-prove.test.ts +91 −1
@@ -9,7 +9,7 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test";
99 import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
1010 import { tmpdir } from "node:os";
1111 import { join } from "node:path";
12 import { GIVE_UP_MS, isNewer, lastProof, proofDue, proofNeed, proveRelease, proveRunning, QUICK_RETRIES, retryAfter, RETRY_MS, startProving, type Proof, type ProveOptions } from "../src/release/prove.ts";
12 import { GIVE_UP_MS, HANG_MS, JOURNEY_TRIES, JOURNEY_RETRY_MS, SETTLE_MS, markProving, isNewer, lastProof, leftoverProbes, proofDue, proofNeed, proofNotice, provingSince, proveRelease, proveRunning, QUICK_RETRIES, retryAfter, RETRY_MS, startProving, type Proof, type ProveOptions } from "../src/release/prove.ts";
1313 import { journeyLine, plain } from "../src/release/words.ts";
1414 import type { JourneyResult } from "../src/release/journey.ts";
1515 import { owned } from "../src/crew/incidents.ts";
@@ -23,6 +23,7 @@ function site(version: string, over: Record<string, () => Response> = {}) {
2323 hits.push(path);
2424 const made = over[path]?.();
2525 if (made) return made;
26 if (path === "/readyz") return new Response("ok");
2627 if (path === "/healthz") return new Response("ok", { headers: { "x-forge-version": version } });
2728 if (path === "/") return new Response(CSS, { headers: { "content-type": "text/html" } });
2829 if (path === "/a.css") return new Response("a{}", { headers: { "content-type": "text/css" } });
@@ -104,6 +105,54 @@ describe("proving a release", () => {
104105 expect(later?.failures).toBe(2);
105106 });
106107
108 test("a failed walk is tried twice more, and the proof records how many tries it took", async () => {
109 let walks = 0;
110 const flaky = async (): Promise<JourneyResult> => (++walks < 3 ? { ...(await walk({})()), ok: false, failure: "the forge was not ready in 20 s" } : walk({})());
111 const p = await run("4.1.0", site("4.1.0").fake, { journey: flaky });
112 expect(p?.ok).toBe(true);
113 expect(p?.tries).toBe(3);
114 walks = -10;
115 const q = await run("4.1.1", site("4.1.1").fake, { journey: flaky });
116 expect(q?.ok).toBe(false);
117 expect(q?.tries).toBe(3);
118 expect(walks).toBe(-7);
119 });
120
121 test("the first walk waits until the forge is ready and its loop is calm", async () => {
122 let ready = 0;
123 let walked = 0;
124 const { fake: base } = site("4.2.0");
125 const fake = (async (input: string | URL, init?: RequestInit) => {
126 const path = new URL(String(input)).pathname;
127 if (path === "/readyz") return ++ready < 3 ? new Response("starting 2 apps", { status: 503 }) : new Response("ok");
128 return base(input, init);
129 }) as unknown as typeof fetch;
130 const p = await run("4.2.0", fake, { journey: async () => (walked++, walk({})()) });
131 expect(ready).toBe(3);
132 expect(walked).toBe(1);
133 expect(p?.ok).toBe(true);
134 });
135
136 test("a forge that never settles fails the try without a walk", async () => {
137 let t = 0;
138 const fake = (async (input: string | URL, init?: RequestInit) =>
139 new URL(String(input)).pathname === "/readyz" ? new Response("starting", { status: 503 }) : site("4.3.0").fake(input, init)) as unknown as typeof fetch;
140 let walked = 0;
141 const p = await run("4.3.0", fake, { now: () => (t += 20_000), journey: async () => (walked++, walk({})()) });
142 expect(walked).toBe(0);
143 expect(p?.ok).toBe(false);
144 expect(p?.failure).toContain("had not settled");
145 });
146
147 test("a probe forge that could not be removed is named, ends the tries, and is kept as leftover", async () => {
148 let walks = 0;
149 const p = await run("4.4.0", site("4.4.0").fake, { journey: async () => (walks++, walk({ ok: false, failure: "the crew was not at work in 60 s; the probe forge probe-abcd1234 could not be removed", forge: "probe-abcd1234", removed: false })()) });
150 expect(walks).toBe(1);
151 expect(p?.failure).toContain("probe-abcd1234 could not be removed");
152 expect(p?.leftover).toEqual(["probe-abcd1234"]);
153 expect(leftoverProbes(dir)).toEqual(["probe-abcd1234"]);
154 });
155
107156 test("a passing walk keeps its timings, and they reach the card", async () => {
108157 const p = await run("4.0.1", site("4.0.1").fake, { journey: walk({}) });
109158 expect(p?.ok).toBe(true);
@@ -373,6 +422,12 @@ describe("proving a release", () => {
373422 await Bun.sleep(60);
374423 expect(signals[0]!.aborted).toBe(true);
375424 expect(warned).toEqual(["release proof hung"]);
425 // The hang is a verdict, not silence: a failed proof the home page and the card show (#12367).
426 const kept = lastProof(dir);
427 expect(kept).toMatchObject({ version: "4.2.0", ok: false, hung: true, slow: true });
428 expect(kept?.failure).toMatch(/did not finish in/);
429 expect(proofNotice(dir, "4.2.0")?.sentence).toMatch(/ran out of time/);
430 expect(provingSince(dir, "4.2.0")).toBeUndefined();
376431 expect(queued.map((q) => q.ms)).toEqual([60_000]);
377432 queued.shift()!.fn(); // the hung one has not settled: no second proof
378433 await Bun.sleep(5);
@@ -388,6 +443,41 @@ describe("proving a release", () => {
388443 stop();
389444 });
390445
446 test("while a proof runs the start time is kept for the home page, and cleared when it ends (#12367)", async () => {
447 let release!: () => void;
448 const prove = () => new Promise<Proof | undefined>((resolve) => (release = () => resolve(undefined)));
449 const stop = startProving({ dataDir: dir, version: "4.2.0", url: "https://forge.test" }, () => {}, 60_000, () => () => {}, prove);
450 await Bun.sleep(5); // real-timer: lets the proof promise settle
451 expect(provingSince(dir, "4.2.0")).toMatch(/^\d{4}-\d\d-\d\dT/);
452 expect(provingSince(dir, "4.2.1")).toBeUndefined();
453 release();
454 await Bun.sleep(5); // real-timer: lets the proof promise settle
455 expect(provingSince(dir, "4.2.0")).toBeUndefined();
456 stop();
457 });
458
459 test("the ceiling covers the worst healthy proof: give-up polling, smoke, every walk with its settle, the waits between (#12367)", () => {
460 const walks = JOURNEY_TRIES * (SETTLE_MS + 20_000) + (JOURNEY_TRIES - 1) * JOURNEY_RETRY_MS;
461 expect(HANG_MS).toBeGreaterThanOrEqual(GIVE_UP_MS + walks + 3 * 60_000);
462 });
463
464 test("a marker left by a process that died mid-proof is cleared at boot, and one older than the ceiling is not believed", async () => {
465 markProving(dir, "4.2.0", Date.now());
466 expect(provingSince(dir, "4.2.0")).toBeDefined();
467 markProving(dir, "4.2.0", Date.now() - HANG_MS - 1000);
468 expect(provingSince(dir, "4.2.0")).toBeUndefined();
469 markProving(dir, "4.2.0", Date.now());
470 // a failed version whose retry is not yet due: the boot finds nothing to prove and says nothing
471 writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ version: "4.2.0", at: new Date().toISOString(), ok: false, failure: "x", failures: 1 }));
472 let release!: () => void;
473 const prove = () => new Promise<Proof | undefined>((resolve) => (release = () => resolve(undefined)));
474 const stop = startProving({ dataDir: dir, version: "4.2.0", url: "https://forge.test" }, () => {}, 60_000, () => () => {}, prove);
475 await Bun.sleep(5); // real-timer: lets the boot tick settle
476 expect(provingSince(dir, "4.2.0")).toBeUndefined();
477 release();
478 stop();
479 });
480
391481 test("proveRelease aborted mid-wait stops asking and keeps no verdict (#10313)", async () => {
392482 const abort = new AbortController();
393483 const fetcher = ((_u: unknown, init?: RequestInit) =>
test/tui-releases.test.ts +20 −1
@@ -5,7 +5,7 @@
55 * repository, and says so when the forge does not swap itself.
66 */
77 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
8 import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
8 import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
99 import { tmpdir } from "node:os";
1010 import { join } from "node:path";
1111 import { createRepo } from "../src/actions/repos.ts";
@@ -159,6 +159,25 @@ describe("the releases screen", () => {
159159 }
160160 });
161161
162 test("p asks a failed proof to be tried again, for a reader who is a person; an agent or a stranger changes nothing", async () => {
163 const file = join(root, "release", "proof.json");
164 const failed = () => {
165 mkdirSync(join(root, "release"), { recursive: true });
166 writeFileSync(file, JSON.stringify({ version: "9.9.9", at: new Date().toISOString(), ok: false, failure: "x", failures: 1 }));
167 };
168 try {
169 expect(await open(alice, "p")).toContain("There is no failed proof to try again");
170 failed();
171 expect(await open(alice, "p")).toContain("proves this release again");
172 expect(JSON.parse(readFileSync(file, "utf8")).retryNow).toBe(true);
173 failed();
174 await open({ ...alice, kind: "agent" } as User, "p");
175 expect(JSON.parse(readFileSync(file, "utf8")).retryNow).toBeUndefined();
176 } finally {
177 rmSync(file, { force: true });
178 }
179 });
180
162181 test("the stranger's journey line: present with a passing proof of the serving version, absent without, nothing for one who may not read", () => {
163182 const was = process.env["FORGE_VERSION"];
164183 process.env["FORGE_VERSION"] = "9.9.9";
test/vitals-release.test.ts +9
@@ -81,3 +81,12 @@ test("a failed proof for the running version is still a card", () => {
8181 );
8282 runningAs("v0.7.479", () => expect(card()?.sentence).toStartWith("Release 0.7.479 is unproven"));
8383 });
84
85 test("a probe forge the proof could not remove is a card with its Remove page; none once a proof leaves none", () => {
86 keep({ version: "9.9.9", at: "2026-10-18T10:00:00.000Z", ok: false, failure: "a stranger's journey failed: x; the probe forge probe-abcd1234 could not be removed", leftover: ["probe-abcd1234"] });
87 const left = vitals({ db: app.db, signals: app.signals, dataDir }).degraded.find((d) => d.cause === "probe:probe-abcd1234");
88 expect(left).toMatchObject({ href: "/forges/probe-abcd1234/remove" });
89 expect(left?.sentence).toContain("probe-abcd1234");
90 keep({ version: "9.9.9", at: "2026-10-18T10:10:00.000Z", ok: true, tries: 2 });
91 expect(vitals({ db: app.db, signals: app.signals, dataDir }).degraded.some((d) => d.cause.startsWith("probe:"))).toBe(false);
92 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs