Public: anyone can read this; the crew can change it.
The release proof fails on the first try after every restart, never retries, and leaves its probe forge behind #12368
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/12367-the-release-proof-fails-on-the-first-try → main 15 commits fork 2 threads · 2 unresolved
17 files
changed +369
−25
docs/MCP.md +1 −1
| @@ -180,7 +180,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`; | ||
| 180 | 180 | | `read_file` | — | No shell? Read a file or list a directory at a ref (default branch); over 5 MB answers tooLarge | 317 | |
| 181 | 181 | | `release_hold` | — | Hold or release the release keeper's cuts (admin) | 202 | |
| 182 | 182 | | `release_keeper` | — | The release keeper's releases, holds and what blocks the next cut | 192 | |
| 183 | | `release_now` | — | Cut a release now, with a reason (admin) | 176 | | |
| 183 | | `release_now` | — | Cut a release now, with a reason; or prove again (admin) | 197 | | |
| 184 | 184 | | `releases` | — | Kept releases, newest first | 298 | |
| 185 | 185 | | `repo` | `open`, `add`, `remove`, `close` | One repository you can read, as GET /api/v1/repos/:owner/:name | 383 | |
| 186 | 186 | | `repo_labels` | — | Repo labels: list (default), create, update, remove (admin token) | 351 | |
docs/features/release-heartbeat.md +8
| @@ -31,3 +31,11 @@ It needs a token with the `admin` scope belonging to a site administrator (401 w | ||
| 31 | 31 | After the usual smoke of the public URL, the proof walks a stranger's journey against it (`walkJourney`, `src/release/journey.ts`): a new forge, claimed, and the crew's first activity, cleaning up after itself. The walk runs last. If it fails, the release stays unproven: the proof is red with "a stranger's journey failed: …" as its sentence, and it is retried on the same `retryAfter` backoff as any failed proof. A walk failure does not set `slow` or `gaveUp`, so a walk that hangs gets the plain doubling, not the shorter polls of a late swap. |
| 32 | 32 | |
| 33 | 33 | When the proof passes it keeps the two timings (`journey.readyMs`, `journey.firstActivityMs`). `/releases` (`#journey-proof`) and the terminal's releases screen then say "a stranger's forge was ready in N s and the crew working in M s". The line shows only for a passing proof of the version being served, with both timings, and only to those who may read /releases; an older proof without timings shows nothing. |
| 34 | ||
| 35 | ## The walk after a restart (#12367) | |
| 36 | ||
| 37 | A restart wakes every app, so the first walk of the stranger's journey waits for `/readyz` to answer 200 and the event loop's lag to fall under 250 ms (90 s at most). A failed walk is tried again two minutes later, three walks at most, before a failure is recorded; `proof.json` and the `release proof` log line carry `tries`. The 20 s forge-ready budget is unchanged. A probe forge is removed after every walk; if its removal fails the failure names it, the walks stop, and the needs-you stack shows it with a Remove button (the forge's remove page). An admin runs the proof again from /releases ("Prove it again now") with `p` on the terminal's releases screen, or with MCP `release_now {prove: true}` (an admin-scope token of a person who may roll back, never an agent; it answers `invalid` when no failed proof waits). The Remove button on a probe forge's card has its terminal door in the forges screen (`x`, remove one). | |
| 38 | ||
| 39 | ## A proof never runs without a word (#12367) | |
| 40 | ||
| 41 | While a proof runs the forge keeps `release/proving.json` (version, start time); the home page's live-version figure says "still proving, started HH:MM UTC" until a verdict replaces it. A proof has `HANG_MS` (30 minutes, shorter than the gap between releases) in all, sized from the worst healthy case (`WORST_CASE_MS`: ten minutes of give-up polling, three minutes of smoke checks, three walks of settle plus the walk's 20 s, two waits of two minutes, five minutes' margin, a test holds it): past it, it is stopped and recorded as a failed, `hung` proof, tried again at the next tick. |
src/http/mcp/tools/release_now.ts +9 −3
| @@ -1,15 +1,21 @@ | ||
| 1 | 1 | import { MAX_REASON } from "../../../keeper/hold.ts"; |
| 2 | 2 | import { releaseNowAs } from "../../../keeper/now.ts"; |
| 3 | import { proveAgain } from "../../../release/prove.ts"; | |
| 4 | import { mayRollBack } from "../../../release/gate.ts"; | |
| 3 | 5 | import { bad, good, str, type Tool } from "../shared.ts"; |
| 4 | 6 | |
| 5 | 7 | export const tool: Tool = { |
| 6 | 8 | name: "release_now", |
| 7 | doc: "`{ reason }`: ask the release keeper (#7557) to cut main now, at its next tick (within a minute), outside the 30-minute spacing and past a running merge queue or snapshot, for a hotfix. The reason stays on the release. It still waits for a release in flight to be proven, for a hold and for room on the disk. A person who may roll back, with an admin-scope token, never an agent", | |
| 8 | description: "Cut a release now, with a reason (admin).", | |
| 9 | inputSchema: { type: "object", properties: { reason: { type: "string", maxLength: MAX_REASON } }, required: ["reason"] }, | |
| 9 | doc: "`{ reason }`: ask the release keeper (#7557) to cut main now, at its next tick (within a minute), outside the 30-minute spacing and past a running merge queue or snapshot, for a hotfix. The reason stays on the release. It still waits for a release in flight to be proven, for a hold and for room on the disk. `{ prove: true }` instead asks the forge to prove the running release again at its next tick, when its last proof failed. A person who may roll back, with an admin-scope token, never an agent", | |
| 10 | description: "Cut a release now, with a reason; or prove again (admin).", | |
| 11 | inputSchema: { type: "object", properties: { reason: { type: "string", maxLength: MAX_REASON }, prove: { type: "boolean" } } }, | |
| 10 | 12 | writes: true, |
| 11 | 13 | needsAdminScope: true, |
| 12 | 14 | run(ctx, user, args) { |
| 15 | if (args["prove"] === true) { | |
| 16 | if (user === undefined || !mayRollBack(ctx.app, user)) return bad("forbidden", "proving the release again is for a person who may administer the forge, not an agent"); | |
| 17 | return proveAgain(ctx.app.config.dataDir) ? good({ prove: "asked: the forge proves the release again at its next tick, within a minute" }) : bad("invalid", "there is no failed proof to try again"); | |
| 18 | } | |
| 13 | 19 | const asked = releaseNowAs(ctx.app, user, str(args["reason"])); |
| 14 | 20 | return asked.ok ? good({ releaseNow: asked.value }) : bad(asked.error, asked.message); |
| 15 | 21 | }, |
src/kernel/vitals.ts +6 −1
| @@ -36,7 +36,7 @@ import { lastDrill } from "../snapshot/drills.ts"; | ||
| 36 | 36 | import { lastMirror, mirrorNeed } from "../snapshot/mirror.ts"; |
| 37 | 37 | import { lastSnapshotOutcome, snapshotNeed } from "../snapshot/outcome.ts"; |
| 38 | 38 | import { drillNeed, lastVerdict } from "../snapshot/verdict.ts"; |
| 39 | import { proofNeed, proofNotice } from "../release/prove.ts"; | |
| 39 | import { leftoverProbes, proofNeed, proofNotice } from "../release/prove.ts"; | |
| 40 | 40 | import { running as runningVersion } from "./version.ts"; |
| 41 | 41 | import { rollbackNeed } from "../release/launch.ts"; |
| 42 | 42 | import { behindStanding } from "../release/behind.ts"; |
| @@ -654,6 +654,11 @@ export function vitals(input: VitalsInput): Vitals { | ||
| 654 | 654 | const unproven = proofNeed(input.dataDir, runningVersion().version); |
| 655 | 655 | if (unproven !== undefined && proofNotice(input.dataDir, runningVersion().version, now)?.tone === "alarm") need("todo", "release", unproven.sentence, unproven.since, "/releases#proof"); |
| 656 | 656 | |
| 657 | // A probe forge the proof could not remove is hidden work on the host: a card each, its | |
| 658 | // Remove page one tap away. | |
| 659 | for (const name of leftoverProbes(input.dataDir)) | |
| 660 | if (/^probe-[0-9a-f]{8}$/.test(name)) need("todo", `probe:${name}`, `The release proof's probe forge ${name} could not be removed and is still running.`, undefined, `/forges/${name}/remove`); | |
| 661 | ||
| 657 | 662 | // A roll the image's launcher undid is a card until a later version reads ready. Its sentence |
| 658 | 663 | // carries the release's own last words (stderr), so anything credential-shaped is masked. |
| 659 | 664 | const undone = rollbackNeed(input.dataDir); |
src/release/journey.ts +4
| @@ -267,6 +267,10 @@ export async function walkJourney(options: JourneyOptions): Promise<JourneyResul | ||
| 267 | 267 | if (forge !== null) { |
| 268 | 268 | if (child !== undefined && cookie !== "") await cancelRuns(doFetch, child, cookie); |
| 269 | 269 | removed = await removeForge(doFetch, hostUrl, forge, options.adminToken, sleep); |
| 270 | // A forge left running is hidden work on the host: the failure names it (never when there | |
| 271 | // was no token to remove it with). | |
| 272 | if (!removed && options.adminToken !== undefined && options.adminToken !== "") | |
| 273 | failure = `${failure === undefined ? "" : `${failure}; `}the probe forge ${forge} could not be removed`; | |
| 270 | 274 | } |
| 271 | 275 | } |
| 272 | 276 | // What an interrupted run left: swept after this walk's own cleanup, never its own forge. |
src/release/prove.ts +133 −8
| @@ -20,7 +20,7 @@ | ||
| 20 | 20 | * that, it was not replaced and proves its version again (#10248). |
| 21 | 21 | */ |
| 22 | 22 | |
| 23 | import { mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; | |
| 23 | import { mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from "node:fs"; | |
| 24 | 24 | import { join } from "node:path"; |
| 25 | 25 | import type { walkJourney, JourneyResult } from "./journey.ts"; |
| 26 | 26 | import { smoke, type Failure, type Heavy } from "./smoke.ts"; |
| @@ -37,14 +37,30 @@ const POLL_MS = 2_000; | ||
| 37 | 37 | export const RETRY_MS = 5 * 60_000; |
| 38 | 38 | /** The longest wait between two proofs of a version that keeps failing. */ |
| 39 | 39 | export const MAX_RETRY_MS = 60 * 60_000; |
| 40 | /** The journey is walked at most this many times before a failure is recorded (#12367). */ | |
| 41 | export const JOURNEY_TRIES = 3; | |
| 42 | /** Between two walks: long enough for a waking host to settle. */ | |
| 43 | export const JOURNEY_RETRY_MS = 2 * 60_000; | |
| 44 | /** | |
| 45 | * How long a walk waits for the forge to report ready and its loop to calm; the 20 s of the | |
| 46 | * walk itself is never raised. | |
| 47 | */ | |
| 48 | export const SETTLE_MS = 90_000; | |
| 49 | /** The event loop's worst lag in the last ten seconds under which the forge counts as settled. */ | |
| 50 | export const SETTLED_LAG_MS = 250; | |
| 40 | 51 | /** How often `startProving` asks whether a proof is due. */ |
| 41 | 52 | export const TICK_MS = 60_000; |
| 42 | 53 | /** How long a retry after a give-up waits for the public URL to answer as the version: the long wait was already spent (#10222). */ |
| 43 | 54 | export const RETRY_POLL_MS = 10_000; |
| 44 | 55 | /** How many give-ups in a row are retried after one tick before the doubling takes over. */ |
| 45 | 56 | export const QUICK_RETRIES = 10; |
| 46 | /** A proof not ended by then is hung: the next tick is scheduled anyway, or the last failure would stand for hours (#10313). */ | |
| 47 | export const HANG_MS = GIVE_UP_MS + 5 * 60_000; | |
| 57 | /** | |
| 58 | * The most a proof may take in all, shorter than the gap between releases: one not ended by | |
| 59 | * then is hung. It is recorded as a failed proof (a verdict the home page shows, tried again | |
| 60 | * at the next tick) and the next tick is scheduled anyway, or the last failure would stand for | |
| 61 | * hours (#10313, #12367). | |
| 62 | */ | |
| 63 | export const HANG_MS = 30 * 60_000; | |
| 48 | 64 | |
| 49 | 65 | export interface Proof { |
| 50 | 66 | readonly version: string; |
| @@ -58,6 +74,8 @@ export interface Proof { | ||
| 58 | 74 | readonly failures?: number; |
| 59 | 75 | /** The first failing check was only a slow, empty or hung answer (`Failure.slow`, `.empty`, `.hung`); retried sooner (#9358, #9482). */ |
| 60 | 76 | readonly slow?: boolean; |
| 77 | /** The proof outran its ceiling (`HANG_MS`) and was stopped. */ | |
| 78 | readonly hung?: boolean; | |
| 61 | 79 | /** The failure was the give-up: ten minutes already spent polling; only its first retry skips the backoff (#10168). */ |
| 62 | 80 | readonly gaveUp?: boolean; |
| 63 | 81 | /** A stranger's journey walked on the public URL after the smoke checks (#11158); its timings, in ms, for the release card. */ |
| @@ -66,6 +84,10 @@ export interface Proof { | ||
| 66 | 84 | readonly footprint?: Footprint; |
| 67 | 85 | /** A person asked for it to be proven again now: the next tick does, whatever the backoff says. */ |
| 68 | 86 | readonly retryNow?: boolean; |
| 87 | /** How many walks of the stranger's journey it took (1 when it passed or failed at once). */ | |
| 88 | readonly tries?: number; | |
| 89 | /** Probe forges the proof could not remove: the needs-you stack lists each with Remove. */ | |
| 90 | readonly leftover?: readonly string[]; | |
| 69 | 91 | } |
| 70 | 92 | |
| 71 | 93 | /** |
| @@ -116,6 +138,50 @@ export function lastProof(dataDir: string): Proof | undefined { | ||
| 116 | 138 | } |
| 117 | 139 | } |
| 118 | 140 | |
| 141 | const provingFile = (dataDir: string): string => join(dataDir, "release", "proving.json"); | |
| 142 | ||
| 143 | /** A proof is running: its version and start, so the home page can say "proving, started 06:01". */ | |
| 144 | export function markProving(dataDir: string, version: string, at: number): void { | |
| 145 | try { | |
| 146 | mkdirSync(join(dataDir, "release"), { recursive: true }); | |
| 147 | writeFileSync(provingFile(dataDir), JSON.stringify({ version, startedAt: new Date(at).toISOString() })); | |
| 148 | } catch {} | |
| 149 | } | |
| 150 | ||
| 151 | export function clearProving(dataDir: string): void { | |
| 152 | try { | |
| 153 | unlinkSync(provingFile(dataDir)); | |
| 154 | } catch {} | |
| 155 | } | |
| 156 | ||
| 157 | /** When the proof of `version` began, while it runs; undefined when none runs for it. */ | |
| 158 | export function provingSince(dataDir: string, version: string, now = Date.now()): string | undefined { | |
| 159 | try { | |
| 160 | const p = JSON.parse(readFileSync(provingFile(dataDir), "utf8")) as { version?: unknown; startedAt?: unknown }; | |
| 161 | return p.version === version && typeof p.startedAt === "string" && Number.isFinite(Date.parse(p.startedAt)) && now - Date.parse(p.startedAt) < HANG_MS ? p.startedAt : undefined; | |
| 162 | } catch { | |
| 163 | return undefined; | |
| 164 | } | |
| 165 | } | |
| 166 | ||
| 167 | /** The verdict for a proof that outran `HANG_MS`: a failure, so a release never has no word. */ | |
| 168 | export function keepHung(dataDir: string, version: string, startedAt: number, now: number, ceilingMs = HANG_MS): Proof { | |
| 169 | const before = lastProof(dataDir); | |
| 170 | const failedBefore = before?.version === version && !before.ok ? before : undefined; | |
| 171 | const at = new Date(now).toISOString(); | |
| 172 | const started = new Date(startedAt).toISOString().slice(11, 16); | |
| 173 | return keep(dataDir, { | |
| 174 | version, | |
| 175 | at, | |
| 176 | ok: false, | |
| 177 | failure: `the proof started at ${started} UTC did not finish in ${Math.round(ceilingMs / 60_000)} minutes`, | |
| 178 | since: failedBefore ? (failedBefore.since ?? failedBefore.at) : at, | |
| 179 | failures: (failedBefore?.failures ?? (failedBefore ? 1 : 0)) + 1, | |
| 180 | slow: true, | |
| 181 | hung: true, | |
| 182 | }); | |
| 183 | } | |
| 184 | ||
| 119 | 185 | function keep(dataDir: string, proof: Proof): Proof { |
| 120 | 186 | mkdirSync(join(dataDir, "release"), { recursive: true }); |
| 121 | 187 | const tmp = `${proofFile(dataDir)}.tmp`; |
| @@ -196,6 +262,8 @@ export function proofNotice(dataDir: string, running?: string, now = Date.now()) | ||
| 196 | 262 | sentence: `Release ${p.version} is serving. Its last proof, a stranger signing up from the public address, did not finish: ${rate ? "the forge's own sign-up limit turned the stranger away" : plain(detail)}. ${again}`, |
| 197 | 263 | }; |
| 198 | 264 | } |
| 265 | if (p.hung === true) | |
| 266 | return { ...base, tone: "notice", step: "The whole proof, from the public address answering to the stranger's journey", stopped: plain(raw), happened: "the host was too busy for the proof to finish in time", sentence: `Release ${p.version} is serving. Its proof ran out of time: ${plain(raw)}. ${again}` }; | |
| 199 | 267 | if (p.slow === true) |
| 200 | 268 | return { ...base, tone: "notice", step: "A smoke check of the public address", stopped: plain(raw), happened: "the public address answered slowly or not at all for a moment", sentence: `Release ${p.version} is serving. The public address answered slowly when the forge proved it. ${again}` }; |
| 201 | 269 | return { |
| @@ -208,6 +276,11 @@ export function proofNotice(dataDir: string, running?: string, now = Date.now()) | ||
| 208 | 276 | }; |
| 209 | 277 | } |
| 210 | 278 | |
| 279 | /** Probe forges the last proof could not remove, until a later proof of any version leaves none. */ | |
| 280 | export function leftoverProbes(dataDir: string): readonly string[] { | |
| 281 | return lastProof(dataDir)?.leftover ?? []; | |
| 282 | } | |
| 283 | ||
| 211 | 284 | /** Asks for the last failed proof to be tried again at the next tick, whatever its backoff says; false when there is none to try. */ |
| 212 | 285 | export function proveAgain(dataDir: string): boolean { |
| 213 | 286 | const p = lastProof(dataDir); |
| @@ -272,7 +345,7 @@ export async function proveRelease(o: ProveOptions): Promise<Proof | undefined> | ||
| 272 | 345 | const doFetch = ((url, init) => baseFetch(url, o.signal ? { ...init, signal: init?.signal ? AbortSignal.any([init.signal, o.signal]) : o.signal } : init)) as typeof fetch; |
| 273 | 346 | const failedBefore = last?.version === o.version && !last.ok ? last : undefined; |
| 274 | 347 | let measured: Footprint | undefined; |
| 275 | const finish = (failure?: string, slow = false, gaveUp = false, walked?: JourneyResult): Proof => { | |
| 348 | const finish = (failure?: string, slow = false, gaveUp = false, walked?: JourneyResult, tries?: number, leftover: readonly string[] = []): Proof => { | |
| 276 | 349 | if (o.signal?.aborted) throw new Error("the proof was aborted as hung"); |
| 277 | 350 | const at = new Date(now()).toISOString(); |
| 278 | 351 | return keep(o.dataDir, { |
| @@ -281,6 +354,8 @@ export async function proveRelease(o: ProveOptions): Promise<Proof | undefined> | ||
| 281 | 354 | ok: failure === undefined, |
| 282 | 355 | ...(measured === undefined ? {} : { footprint: measured }), |
| 283 | 356 | ...(walked === undefined ? {} : { journey: { readyMs: walked.readyMs, firstActivityMs: walked.firstActivityMs, screens: walked.screens } }), |
| 357 | ...(tries === undefined ? {} : { tries }), | |
| 358 | ...(leftover.length === 0 ? {} : { leftover }), | |
| 284 | 359 | ...(failure === undefined |
| 285 | 360 | ? {} |
| 286 | 361 | : { failure, since: failedBefore ? (failedBefore.since ?? failedBefore.at) : at, failures: (failedBefore?.failures ?? (failedBefore ? 1 : 0)) + 1, ...(slow ? { slow } : {}), ...(gaveUp ? { gaveUp } : {}) }), |
| @@ -321,13 +396,51 @@ export async function proveRelease(o: ProveOptions): Promise<Proof | undefined> | ||
| 321 | 396 | if (failures[0] !== undefined || o.journey === undefined) |
| 322 | 397 | return finish(failures[0] === undefined ? undefined : checkLine(failures[0]), failures[0]?.slow === true || failures[0]?.empty === true || failures[0]?.hung === true); |
| 323 | 398 | // The stranger's journey, the last check: a failed walk is a failed proof, retried on `retryAfter` like any other (one card, paged once). |
| 324 | const walked = await o.journey({ base: o.url, fetch: doFetch, ...(o.token ? { adminToken: o.token } : {}), ...(o.proberKey ? { proberKey: o.proberKey } : {}) }); | |
| 325 | return finish(walked.ok ? undefined : `a stranger's journey failed: ${walked.failure ?? "it did not finish"}`, false, false, walked); | |
| 399 | // A restart wakes every app: the first walk waits for the host to settle, and a failure is | |
| 400 | // tried again a few minutes later (twice at most) before it is recorded. | |
| 401 | const leftover: string[] = []; | |
| 402 | let walked: JourneyResult | undefined; | |
| 403 | let tries = 0; | |
| 404 | while (tries < JOURNEY_TRIES) { | |
| 405 | if (o.signal?.aborted) throw new Error("the proof was aborted as hung"); | |
| 406 | const settled = await settle(o.url, doFetch, now, sleep, o.giveUpMs === undefined ? SETTLE_MS : Math.min(SETTLE_MS, o.giveUpMs)); | |
| 407 | tries++; | |
| 408 | walked = settled | |
| 409 | ? await o.journey({ base: o.url, fetch: doFetch, ...(o.token ? { adminToken: o.token } : {}), ...(o.proberKey ? { proberKey: o.proberKey } : {}) }) | |
| 410 | : { ok: false, screens: [], readyMs: null, firstActivityMs: null, failure: `the forge had not settled in ${SETTLE_MS / 1000} s after its start`, forge: null, removed: true }; | |
| 411 | if (walked.forge !== null && !walked.removed) leftover.push(walked.forge); | |
| 412 | // A passed walk, a refusal by the sign-up limit, or a forge left behind ends the tries: | |
| 413 | // another walk would only make another. | |
| 414 | if (walked.ok || /start=rate/.test(walked.failure ?? "") || leftover.length > 0 || tries >= JOURNEY_TRIES) break; | |
| 415 | await sleep(o.giveUpMs === undefined ? JOURNEY_RETRY_MS : Math.min(JOURNEY_RETRY_MS, o.giveUpMs)); | |
| 416 | } | |
| 417 | const w = walked!; | |
| 418 | return finish(w.ok ? undefined : `a stranger's journey failed: ${w.failure ?? "it did not finish"}`, false, false, w, tries, leftover); | |
| 326 | 419 | } catch (error) { |
| 327 | 420 | return finish(`the smoke checks could not run: ${String(error)}`); |
| 328 | 421 | } |
| 329 | 422 | } |
| 330 | 423 | |
| 424 | /** Waits until `/readyz` answers and the event loop is calm; false when `budgetMs` passes. */ | |
| 425 | async function settle(url: string, doFetch: typeof fetch, now: () => number, sleep: (ms: number) => Promise<void>, budgetMs: number): Promise<boolean> { | |
| 426 | const until = now() + budgetMs; | |
| 427 | for (;;) { | |
| 428 | try { | |
| 429 | const ready = await doFetch(new URL("/readyz", url).href, { redirect: "manual", cache: "no-store", signal: AbortSignal.timeout(5_000) }); | |
| 430 | await ready.body?.cancel(); | |
| 431 | if (ready.status === 200) { | |
| 432 | const live = await doFetch(new URL("/healthz", url).href, { redirect: "manual", cache: "no-store", headers: { accept: "application/json" }, signal: AbortSignal.timeout(5_000) }); | |
| 433 | const lag = live.status === 200 ? ((await live.json().catch(() => ({}))) as { loopLagMs?: unknown }).loopLagMs : undefined; | |
| 434 | // A forge that does not report its lag is taken as calm: only a reported, high lag waits. | |
| 435 | if (live.status === 200 && (typeof lag !== "number" || lag < SETTLED_LAG_MS)) return true; | |
| 436 | if (live.status !== 200) await live.body?.cancel(); | |
| 437 | } | |
| 438 | } catch {} | |
| 439 | if (now() + POLL_MS > until) return false; | |
| 440 | await sleep(POLL_MS); | |
| 441 | } | |
| 442 | } | |
| 443 | ||
| 331 | 444 | /** Runs `fn` after `ms`; returns its cancel. */ |
| 332 | 445 | export type Schedule = (fn: () => void, ms: number) => () => void; |
| 333 | 446 | |
| @@ -359,12 +472,17 @@ export function startProving( | ||
| 359 | 472 | const next = (): void => { |
| 360 | 473 | if (!stopped) cancel = schedule(() => void during("release-prove", tick), everyMs); |
| 361 | 474 | }; |
| 475 | // A marker left by a process that died mid-proof is not this boot's: it stands for nothing. | |
| 476 | clearProving(o.dataDir); | |
| 362 | 477 | const tick = async (): Promise<void> => { |
| 363 | 478 | if (running) { |
| 364 | 479 | log?.warn("release proof still running", { version: o.version }); |
| 365 | 480 | return next(); |
| 366 | 481 | } |
| 367 | 482 | running = true; |
| 483 | const startedAt = Date.now(); | |
| 484 | // Only a proof that is due is "proving": a tick that finds nothing to do says nothing. | |
| 485 | if (proofDue(lastProof(o.dataDir), o.version, startedAt)) markProving(o.dataDir, o.version, startedAt); | |
| 368 | 486 | const abort = new AbortController(); |
| 369 | 487 | const work = prove({ ...o, signal: abort.signal }).then( |
| 370 | 488 | (p) => ({ p }), |
| @@ -373,7 +491,10 @@ export function startProving( | ||
| 373 | 491 | return { p: undefined }; |
| 374 | 492 | }, |
| 375 | 493 | ); |
| 376 | void work.finally(() => void (running = false)); | |
| 494 | void work.finally(() => { | |
| 495 | running = false; | |
| 496 | if (!abort.signal.aborted) clearProving(o.dataDir); | |
| 497 | }); | |
| 377 | 498 | let timer: ReturnType<typeof setTimeout> | undefined; |
| 378 | 499 | const hung = new Promise<"hung">((resolve) => { |
| 379 | 500 | timer = setTimeout(() => resolve("hung"), hangMs); |
| @@ -385,6 +506,10 @@ export function startProving( | ||
| 385 | 506 | // Stop its asks; the next tick waits for it to settle rather than run beside it. |
| 386 | 507 | log?.warn("release proof hung", { version: o.version, afterMs: hangMs }); |
| 387 | 508 | abort.abort(); |
| 509 | try { | |
| 510 | done(keepHung(o.dataDir, o.version, startedAt, Date.now(), hangMs)); | |
| 511 | } catch {} | |
| 512 | clearProving(o.dataDir); | |
| 388 | 513 | } else if (result.p) done(result.p); |
| 389 | 514 | next(); |
| 390 | 515 | }; |
| @@ -403,5 +528,5 @@ export function proveRunning( | ||
| 403 | 528 | schedule: Schedule = unrefTimer, |
| 404 | 529 | prove: (o: ProveOptions) => Promise<Proof | undefined> = proveRelease, |
| 405 | 530 | ): () => void { |
| 406 | return startProving(o, (p) => log.info("release proof", { version: p.version, ok: String(p.ok), ...(p.failure ? { failure: p.failure } : {}) }), everyMs, schedule, prove, HANG_MS, log); | |
| 531 | return startProving(o, (p) => log.info("release proof", { version: p.version, ok: String(p.ok), ...(p.tries === undefined ? {} : { tries: String(p.tries) }), ...(p.failure ? { failure: p.failure } : {}) }), everyMs, schedule, prove, HANG_MS, log); | |
| 407 | 532 | } |
src/tui/screens/releases.ts +10 −2
| @@ -10,7 +10,7 @@ import { getRepoById } from "../../core/repos.ts"; | ||
| 10 | 10 | import { running } from "../../kernel/version.ts"; |
| 11 | 11 | import { BOOTED, canRollBackTo, mayRead, mayRollBack, rollBackTo } from "../../release/gate.ts"; |
| 12 | 12 | import { listReleases, type Release } from "../../release/record.ts"; |
| 13 | import { lastProof } from "../../release/prove.ts"; | |
| 13 | import { lastProof, proveAgain } from "../../release/prove.ts"; | |
| 14 | 14 | import { journeyLine, liveSince, NO_RELEASES_YET, plain } from "../../release/words.ts"; |
| 15 | 15 | import type { Ctx, Screen } from "../app/app.ts"; |
| 16 | 16 | import { binding } from "../app/keys.ts"; |
| @@ -87,7 +87,7 @@ export function releasesScreen(f: Session, ctx: Ctx): Screen { | ||
| 87 | 87 | }; |
| 88 | 88 | const self: Screen = { |
| 89 | 89 | title: "Releases", |
| 90 | keys: [binding(["j", "k"], "move"), binding(["enter"], "roll back"), binding(["esc", "q"], "close")], | |
| 90 | keys: [binding(["j", "k"], "move"), binding(["enter"], "roll back"), binding(["p"], "prove it again"), binding(["esc", "q"], "close")], | |
| 91 | 91 | load: read, |
| 92 | 92 | render(area, s) { |
| 93 | 93 | const c = ctx.theme.c; |
| @@ -106,10 +106,18 @@ export function releasesScreen(f: Session, ctx: Ctx): Screen { | ||
| 106 | 106 | key(e) { |
| 107 | 107 | if (e.key === "esc" || e.key === "q") ctx.pop(); |
| 108 | 108 | else if (e.key === "enter") ask(state.index); |
| 109 | else if (e.key === "p") proveIt(); | |
| 109 | 110 | else state.key(e, lines.length); |
| 110 | 111 | return true; |
| 111 | 112 | }, |
| 112 | 113 | }; |
| 114 | // The page's "Prove it again now": the same gate (a person who may read, never an agent), | |
| 115 | // the same call. | |
| 116 | const proveIt = (): void => { | |
| 117 | const user = f.user; | |
| 118 | if (user === undefined || user.kind === "agent" || !mayRead(f.app, user)) return; | |
| 119 | ctx.toast(proveAgain(f.app.config.dataDir) ? "Asked: the forge proves this release again at its next tick, within a minute." : "There is no failed proof to try again.", "default"); | |
| 120 | }; | |
| 113 | 121 | const ask = (index: number): void => { |
| 114 | 122 | const line = lines[index]; |
| 115 | 123 | const user = f.user; |
src/web/pages/home_flow.ts +5 −3
| @@ -39,7 +39,7 @@ import { | ||
| 39 | 39 | type StripPart, |
| 40 | 40 | } from "../../flow/lanes.ts"; |
| 41 | 41 | import { span } from "../../flow/pipeline.ts"; |
| 42 | import { lastProof, proofNotice, type Proof } from "../../release/prove.ts"; | |
| 42 | import { lastProof, proofNotice, provingSince, type Proof } from "../../release/prove.ts"; | |
| 43 | 43 | import { running, UNSTAMPED } from "../../kernel/version.ts"; |
| 44 | 44 | import { notePhase, takeSessionTime, timePhase, timePhaseAsync } from "../../core/route_timing.ts"; |
| 45 | 45 | import type { Ctx } from "../../http/context.ts"; |
| @@ -215,7 +215,9 @@ const healthOf = (view: LaneView, picture: LanePicture, now: number): Health => | ||
| 215 | 215 | |
| 216 | 216 | // ---- the strip: today's figures, and the one line that needs the owner ---- |
| 217 | 217 | |
| 218 | function proofWords(proof: Proof | undefined, version: string): { text: string; tone: "attn" | null } { | |
| 218 | export function proofWords(proof: Proof | undefined, version: string, since?: string): { text: string; tone: "attn" | null } { | |
| 219 | // A proof under way says so and when it began, so no release goes a quarter hour unspoken. | |
| 220 | if (since !== undefined && (proof === undefined || proof.version !== version || !proof.ok)) return { text: `still proving, started ${since.slice(11, 16)} UTC`, tone: null }; | |
No test renders this 'still proving' text (principle 8 wording is fine), and a stale proving.json would make it show forever. Test it, and clear the marker on boot. |
||
| 219 | 221 | if (proof === undefined) return { text: "not proven yet", tone: null }; |
| 220 | 222 | if (proof.version !== version) return { text: "proof running", tone: null }; |
| 221 | 223 | return proof.ok ? { text: "journey proof passed", tone: null } : { text: "journey proof failed", tone: "attn" }; |
| @@ -323,7 +325,7 @@ function needLine(needs: readonly Need[]): Html { | ||
| 323 | 325 | export function stripRegion(picture: LanePicture, viewer: StripViewer, dataDir: string, now: number): Html { |
| 324 | 326 | const strip = picture.strip; |
| 325 | 327 | const version = running().version; |
| 326 | const proof = proofWords(lastProof(dataDir), version); | |
| 328 | const proof = proofWords(lastProof(dataDir), version, provingSince(dataDir, version)); | |
| 327 | 329 | const needs = needLine(needsOf(picture, viewer, dataDir, now)); |
| 328 | 330 | if (!viewer.isAdmin) return html`<section class="hf-strip" id="hf-strip" aria-label="What needs you" data-live="${STRIP_LIVE}" data-live-src="/live/strip">${needs}</section>`; |
| 329 | 331 | const b = picture.builders; |
src/web/pages/releases.ts +5 −4
| @@ -47,16 +47,17 @@ export function journeyCard(ctx: Ctx): Html | undefined { | ||
| 47 | 47 | * when it is tried again, and a way to try it now. Plain words; the proof's raw |
| 48 | 48 | * line is one fold away. |
| 49 | 49 | */ |
| 50 | export function proofCard(ctx: Ctx, writer: boolean): Html | undefined { | |
| 51 | const n = proofNotice(ctx.app.config.dataDir, running().version); | |
| 50 | export function proofCard(dataDir: string, writer: boolean): Html | undefined { | |
| 51 | const n = proofNotice(dataDir, running().version); | |
| 52 | 52 | if (n === undefined) return undefined; |
| 53 | const tries = lastProof(dataDir)?.tries; | |
| 53 | 54 | const serving = n.tone === "notice"; |
| 54 | 55 | const rows: [string, string | Html][] = [ |
| 55 | 56 | ["Is the release serving?", serving ? `Yes. This forge is running ${n.version} and answering.` : `Not as it should: ${n.happened}.`], |
| 56 | 57 | ["What was checked", n.step], |
| 57 | 58 | ["Where it stopped", n.stopped], |
| 58 | 59 | ["What happened", n.happened], |
| 59 | ["First failed", html`${time(n.since)}, ${String(n.failures)} ${n.failures === 1 ? "try" : "tries"} so far`], | |
| 60 | ["First failed", html`${time(n.since)}, ${String(n.failures)} failed ${n.failures === 1 ? "attempt" : "attempts"} so far${tries === undefined ? "" : `; the last one tried ${tries} ${tries === 1 ? "time" : "times"}`}`], | |
| 60 | 61 | ["Next try", n.nextAt === null ? "Soon." : html`${time(new Date(n.nextAt).toISOString())} (${new Date(n.nextAt).toISOString().slice(11, 16)} UTC)`], |
| 61 | 62 | ]; |
| 62 | 63 | return html`<section class="settings-block" id="proof" aria-label="The release's proof"> |
| @@ -133,7 +134,7 @@ export async function releasesPage(ctx: Ctx): Promise<Response> { | ||
| 133 | 134 | const response = page({ |
| 134 | 135 | ctx, |
| 135 | 136 | title: "Releases", |
| 136 | body: html`<div data-releases><div class="page-head"><h1>Releases</h1></div>${proofCard(ctx, user.kind !== "agent")}${keeperPanel(ctx.app.db, writer)}${journeyCard(ctx)}${body}</div>`, | |
| 137 | body: html`<div data-releases><div class="page-head"><h1>Releases</h1></div>${proofCard(ctx.app.config.dataDir, user.kind !== "agent")}${keeperPanel(ctx.app.db, writer)}${journeyCard(ctx)}${body}</div>`, | |
| 137 | 138 | styleCss: RELEASES_CSS, |
| 138 | 139 | }); |
| 139 | 140 | notePhase(ctx.request, "render", performance.now() - rendering); |
src/web/studio/asks_page.ts +1 −1
| @@ -410,7 +410,7 @@ export const forgeCard = (d: ForgeNeed): Html => | ||
| 410 | 410 | html`<section class="stack"> |
| 411 | 411 | <h2>The forge</h2> |
| 412 | 412 | <p role="alert"><span class="dot" data-tone="danger" aria-hidden="true"></span> <a href="${d.href}">${d.line}</a></p> |
| 413 | <div class="bar">${btn("Health", { href: "/api/v1/health", size: "sm" })}</div> | |
| 413 | <div class="bar">${d.cause.startsWith("probe:") ? btn("Remove", { href: d.href, size: "sm", variant: "primary" }) : null}${btn("Health", { href: "/api/v1/health", size: "sm" })}</div> | |
| 414 | 414 | </section>`; |
| 415 | 415 | |
| 416 | 416 | /** What sits above the asks after a bulk step: a preview to confirm, a receipt, or why nothing was done. */ |
test/asks-forge-alert.test.ts +7
| @@ -14,3 +14,10 @@ test("a degraded forge on /asks is an alert line in the danger tone, with Health | ||
| 14 | 14 | expect(out).not.toContain("<now>"); |
| 15 | 15 | expect(out).not.toContain('class="card'); |
| 16 | 16 | }); |
| 17 | ||
| 18 | test("a probe forge the release proof left behind is a card with a Remove button to its remove page; other causes have none", () => { | |
| 19 | const d = { line: "Left." }; | |
| 20 | const probe = String(forgeCard({ ...d, cause: "probe:probe-abcd1234", href: "/forges/probe-abcd1234/remove" })); | |
| 21 | expect(probe).toMatch(/data-variant="primary"[\s\S]*?>Remove</); | |
| 22 | expect(String(forgeCard({ ...d, cause: "errors", href: "/x" }))).not.toContain("Remove"); | |
| 23 | }); |
test/release-journey.test.ts +2
| @@ -267,6 +267,8 @@ test("a removal that never succeeds waits between its attempts and says it is no | ||
| 267 | 267 | let t = 0; |
| 268 | 268 | const result = await walkJourney({ base: parentServers.url, fetch: stuck, adminToken, pollMs: 0, now: () => (t += 5_000), sleep: async (ms) => void pauses.push(ms) }); |
| 269 | 269 | expect(result.removed).toBe(false); |
| 270 | expect(result.ok).toBe(false); | |
| 271 | expect(result.failure).toContain(`the probe forge ${result.forge} could not be removed`); | |
| 270 | 272 | expect(pauses).toEqual([...REMOVE_BACKOFF_MS]); |
| 271 | 273 | await fetch(`${parentServers.url}/forges/${result.forge}/remove`, { method: "POST", headers: { origin: parentServers.url, authorization: `Bearer ${adminToken}` } }); |
| 272 | 274 | }, 60_000); |
test/release-now.test.ts +16
| @@ -1,6 +1,9 @@ | ||
| 1 | 1 | // A hotfix does not wait (#11839): `release now` cuts at the next tick past the spacing, the merge queue and a |
| 2 | 2 | // snapshot, never past a hold or a full disk; each release says where its time went; the keeper says what holds it. |
| 3 | 3 | import { describe, expect, test } from "bun:test"; |
| 4 | import { mkdirSync, writeFileSync } from "node:fs"; | |
| 5 | import { join } from "node:path"; | |
| 6 | import { lastProof } from "../src/release/prove.ts"; | |
| 4 | 7 | import type { Ctx } from "../src/http/context.ts"; |
| 5 | 8 | import { tool as keeperTool } from "../src/http/mcp/tools/release_keeper.ts"; |
| 6 | 9 | import { tool as nowTool } from "../src/http/mcp/tools/release_now.ts"; |
| @@ -53,6 +56,19 @@ describe("release now", () => { | ||
| 53 | 56 | await nowTool.run(ctx, f.user, { reason: "later" }); |
| 54 | 57 | expect(pendingNow(f.app.db, Date.now() + 2 * 3_600_000)).toBeUndefined(); |
| 55 | 58 | }); |
| 59 | test("prove: true asks for the failed proof again; an agent and a passing proof are refused", async () => { | |
| 60 | const f = await keeperForge(); | |
| 61 | const ctx = { app: f.app } as unknown as Ctx; | |
| 62 | const dir = f.app.config.dataDir; | |
| 63 | expect((await nowTool.run(ctx, f.user, { prove: true })).ok).toBe(false); | |
| 64 | mkdirSync(join(dir, "release"), { recursive: true }); | |
| 65 | writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ version: "1.0.0", at: "2026-11-14T02:00:00.000Z", ok: false, failure: "x" })); | |
| 66 | expect((await nowTool.run(ctx, { ...f.user, kind: "agent" as const }, { prove: true })).ok).toBe(false); | |
| 67 | expect(lastProof(dir)?.retryNow).toBeUndefined(); | |
| 68 | expect((await nowTool.run(ctx, f.user, { prove: true })).ok).toBe(true); | |
| 69 | expect(lastProof(dir)?.retryNow).toBe(true); | |
| 70 | expect(pendingNow(f.app.db, f.world.now)).toBeUndefined(); | |
| 71 | }); | |
| 56 | 72 | test("a hold still stops it", async () => { |
| 57 | 73 | const f = await keeperForge(); |
| 58 | 74 | setHold(f.app.db, { kind: "owner", version: "0.7.283", reason: "launch freeze", at: f.world.now }); |
test/release-proof-words.test.ts added +42
| @@ -0,0 +1,42 @@ | ||
| 1 | /** What a person reads of a proof: the home page's figure and the /releases card (#12367). */ | |
| 2 | import { afterEach, beforeEach, expect, test } from "bun:test"; | |
| 3 | import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; | |
| 4 | import { tmpdir } from "node:os"; | |
| 5 | import { join } from "node:path"; | |
| 6 | import { proofWords } from "../src/web/pages/home_flow.ts"; | |
| 7 | import { proofCard } from "../src/web/pages/releases.ts"; | |
| 8 | import type { Proof } from "../src/release/prove.ts"; | |
| 9 | ||
| 10 | let dir = ""; | |
| 11 | const was = process.env["FORGE_VERSION"]; | |
| 12 | beforeEach(() => { | |
| 13 | dir = mkdtempSync(join(tmpdir(), "forge-proof-words-")); | |
| 14 | process.env["FORGE_VERSION"] = "9.9.9"; | |
| 15 | }); | |
| 16 | afterEach(() => { | |
| 17 | rmSync(dir, { recursive: true, force: true }); | |
| 18 | if (was === undefined) delete process.env["FORGE_VERSION"]; | |
| 19 | else process.env["FORGE_VERSION"] = was; | |
| 20 | }); | |
| 21 | ||
| 22 | const base: Proof = { version: "9.9.9", at: "2026-11-14T06:00:00.000Z", ok: true }; | |
| 23 | ||
| 24 | test("the home figure says a proof is running and when it began, until a pass replaces it", () => { | |
| 25 | const since = "2026-11-14T06:01:30.000Z"; | |
| 26 | expect(proofWords(undefined, "9.9.9", since)).toEqual({ text: "still proving, started 06:01 UTC", tone: null }); | |
| 27 | expect(proofWords({ ...base, ok: false }, "9.9.9", since).text).toBe("still proving, started 06:01 UTC"); | |
| 28 | expect(proofWords({ ...base, version: "9.9.8" }, "9.9.9", since).text).toBe("still proving, started 06:01 UTC"); | |
| 29 | expect(proofWords(base, "9.9.9", since).text).toBe("journey proof passed"); | |
| 30 | expect(proofWords({ ...base, ok: false }, "9.9.9").tone).toBe("attn"); | |
| 31 | expect(proofWords(undefined, "9.9.9").text).toBe("not proven yet"); | |
| 32 | }); | |
| 33 | ||
| 34 | test("the /releases card counts failed attempts and the last one's tries in one plain word", () => { | |
| 35 | mkdirSync(join(dir, "release"), { recursive: true }); | |
| 36 | writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ ...base, ok: false, failure: "a stranger's journey failed: x", failures: 2, tries: 3 })); | |
| 37 | const card = String(proofCard(dir, false)); | |
| 38 | expect(card).toContain("2 failed attempts so far; the last one tried 3 times"); | |
| 39 | expect(card).not.toMatch(/proofs so far|walks/); | |
| 40 | writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ ...base, ok: false, failure: "a stranger's journey failed: x", failures: 1, tries: 1 })); | |
| 41 | expect(String(proofCard(dir, false))).toContain("1 failed attempt so far; the last one tried 1 time<"); | |
| 42 | }); |
test/release-prove.test.ts +91 −1
| @@ -9,7 +9,7 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; | ||
| 9 | 9 | import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; |
| 10 | 10 | import { tmpdir } from "node:os"; |
| 11 | 11 | import { join } from "node:path"; |
| 12 | import { GIVE_UP_MS, isNewer, lastProof, proofDue, proofNeed, proveRelease, proveRunning, QUICK_RETRIES, retryAfter, RETRY_MS, startProving, type Proof, type ProveOptions } from "../src/release/prove.ts"; | |
| 12 | import { GIVE_UP_MS, HANG_MS, JOURNEY_TRIES, JOURNEY_RETRY_MS, SETTLE_MS, markProving, isNewer, lastProof, leftoverProbes, proofDue, proofNeed, proofNotice, provingSince, proveRelease, proveRunning, QUICK_RETRIES, retryAfter, RETRY_MS, startProving, type Proof, type ProveOptions } from "../src/release/prove.ts"; | |
| 13 | 13 | import { journeyLine, plain } from "../src/release/words.ts"; |
| 14 | 14 | import type { JourneyResult } from "../src/release/journey.ts"; |
| 15 | 15 | import { owned } from "../src/crew/incidents.ts"; |
| @@ -23,6 +23,7 @@ function site(version: string, over: Record<string, () => Response> = {}) { | ||
| 23 | 23 | hits.push(path); |
| 24 | 24 | const made = over[path]?.(); |
| 25 | 25 | if (made) return made; |
| 26 | if (path === "/readyz") return new Response("ok"); | |
| 26 | 27 | if (path === "/healthz") return new Response("ok", { headers: { "x-forge-version": version } }); |
| 27 | 28 | if (path === "/") return new Response(CSS, { headers: { "content-type": "text/html" } }); |
| 28 | 29 | if (path === "/a.css") return new Response("a{}", { headers: { "content-type": "text/css" } }); |
| @@ -104,6 +105,54 @@ describe("proving a release", () => { | ||
| 104 | 105 | expect(later?.failures).toBe(2); |
| 105 | 106 | }); |
| 106 | 107 | |
| 108 | test("a failed walk is tried twice more, and the proof records how many tries it took", async () => { | |
| 109 | let walks = 0; | |
| 110 | const flaky = async (): Promise<JourneyResult> => (++walks < 3 ? { ...(await walk({})()), ok: false, failure: "the forge was not ready in 20 s" } : walk({})()); | |
| 111 | const p = await run("4.1.0", site("4.1.0").fake, { journey: flaky }); | |
| 112 | expect(p?.ok).toBe(true); | |
| 113 | expect(p?.tries).toBe(3); | |
| 114 | walks = -10; | |
| 115 | const q = await run("4.1.1", site("4.1.1").fake, { journey: flaky }); | |
| 116 | expect(q?.ok).toBe(false); | |
| 117 | expect(q?.tries).toBe(3); | |
| 118 | expect(walks).toBe(-7); | |
| 119 | }); | |
| 120 | ||
| 121 | test("the first walk waits until the forge is ready and its loop is calm", async () => { | |
| 122 | let ready = 0; | |
| 123 | let walked = 0; | |
| 124 | const { fake: base } = site("4.2.0"); | |
| 125 | const fake = (async (input: string | URL, init?: RequestInit) => { | |
| 126 | const path = new URL(String(input)).pathname; | |
| 127 | if (path === "/readyz") return ++ready < 3 ? new Response("starting 2 apps", { status: 503 }) : new Response("ok"); | |
| 128 | return base(input, init); | |
| 129 | }) as unknown as typeof fetch; | |
| 130 | const p = await run("4.2.0", fake, { journey: async () => (walked++, walk({})()) }); | |
| 131 | expect(ready).toBe(3); | |
| 132 | expect(walked).toBe(1); | |
| 133 | expect(p?.ok).toBe(true); | |
| 134 | }); | |
| 135 | ||
| 136 | test("a forge that never settles fails the try without a walk", async () => { | |
| 137 | let t = 0; | |
| 138 | const fake = (async (input: string | URL, init?: RequestInit) => | |
| 139 | new URL(String(input)).pathname === "/readyz" ? new Response("starting", { status: 503 }) : site("4.3.0").fake(input, init)) as unknown as typeof fetch; | |
| 140 | let walked = 0; | |
| 141 | const p = await run("4.3.0", fake, { now: () => (t += 20_000), journey: async () => (walked++, walk({})()) }); | |
| 142 | expect(walked).toBe(0); | |
| 143 | expect(p?.ok).toBe(false); | |
| 144 | expect(p?.failure).toContain("had not settled"); | |
| 145 | }); | |
| 146 | ||
| 147 | test("a probe forge that could not be removed is named, ends the tries, and is kept as leftover", async () => { | |
| 148 | let walks = 0; | |
| 149 | const p = await run("4.4.0", site("4.4.0").fake, { journey: async () => (walks++, walk({ ok: false, failure: "the crew was not at work in 60 s; the probe forge probe-abcd1234 could not be removed", forge: "probe-abcd1234", removed: false })()) }); | |
| 150 | expect(walks).toBe(1); | |
| 151 | expect(p?.failure).toContain("probe-abcd1234 could not be removed"); | |
| 152 | expect(p?.leftover).toEqual(["probe-abcd1234"]); | |
| 153 | expect(leftoverProbes(dir)).toEqual(["probe-abcd1234"]); | |
| 154 | }); | |
| 155 | ||
| 107 | 156 | test("a passing walk keeps its timings, and they reach the card", async () => { |
| 108 | 157 | const p = await run("4.0.1", site("4.0.1").fake, { journey: walk({}) }); |
| 109 | 158 | expect(p?.ok).toBe(true); |
| @@ -373,6 +422,12 @@ describe("proving a release", () => { | ||
| 373 | 422 | await Bun.sleep(60); |
| 374 | 423 | expect(signals[0]!.aborted).toBe(true); |
| 375 | 424 | expect(warned).toEqual(["release proof hung"]); |
| 425 | // The hang is a verdict, not silence: a failed proof the home page and the card show (#12367). | |
| 426 | const kept = lastProof(dir); | |
| 427 | expect(kept).toMatchObject({ version: "4.2.0", ok: false, hung: true, slow: true }); | |
| 428 | expect(kept?.failure).toMatch(/did not finish in/); | |
| 429 | expect(proofNotice(dir, "4.2.0")?.sentence).toMatch(/ran out of time/); | |
| 430 | expect(provingSince(dir, "4.2.0")).toBeUndefined(); | |
| 376 | 431 | expect(queued.map((q) => q.ms)).toEqual([60_000]); |
| 377 | 432 | queued.shift()!.fn(); // the hung one has not settled: no second proof |
| 378 | 433 | await Bun.sleep(5); |
| @@ -388,6 +443,41 @@ describe("proving a release", () => { | ||
| 388 | 443 | stop(); |
| 389 | 444 | }); |
| 390 | 445 | |
| 446 | test("while a proof runs the start time is kept for the home page, and cleared when it ends (#12367)", async () => { | |
| 447 | let release!: () => void; | |
| 448 | const prove = () => new Promise<Proof | undefined>((resolve) => (release = () => resolve(undefined))); | |
| 449 | const stop = startProving({ dataDir: dir, version: "4.2.0", url: "https://forge.test" }, () => {}, 60_000, () => () => {}, prove); | |
| 450 | await Bun.sleep(5); // real-timer: lets the proof promise settle | |
| 451 | expect(provingSince(dir, "4.2.0")).toMatch(/^\d{4}-\d\d-\d\dT/); | |
| 452 | expect(provingSince(dir, "4.2.1")).toBeUndefined(); | |
| 453 | release(); | |
| 454 | await Bun.sleep(5); // real-timer: lets the proof promise settle | |
| 455 | expect(provingSince(dir, "4.2.0")).toBeUndefined(); | |
| 456 | stop(); | |
| 457 | }); | |
| 458 | ||
| 459 | test("the ceiling covers the worst healthy proof: give-up polling, smoke, every walk with its settle, the waits between (#12367)", () => { | |
| 460 | const walks = JOURNEY_TRIES * (SETTLE_MS + 20_000) + (JOURNEY_TRIES - 1) * JOURNEY_RETRY_MS; | |
| 461 | expect(HANG_MS).toBeGreaterThanOrEqual(GIVE_UP_MS + walks + 3 * 60_000); | |
| 462 | }); | |
| 463 | ||
| 464 | test("a marker left by a process that died mid-proof is cleared at boot, and one older than the ceiling is not believed", async () => { | |
| 465 | markProving(dir, "4.2.0", Date.now()); | |
| 466 | expect(provingSince(dir, "4.2.0")).toBeDefined(); | |
| 467 | markProving(dir, "4.2.0", Date.now() - HANG_MS - 1000); | |
| 468 | expect(provingSince(dir, "4.2.0")).toBeUndefined(); | |
| 469 | markProving(dir, "4.2.0", Date.now()); | |
| 470 | // a failed version whose retry is not yet due: the boot finds nothing to prove and says nothing | |
| 471 | writeFileSync(join(dir, "release", "proof.json"), JSON.stringify({ version: "4.2.0", at: new Date().toISOString(), ok: false, failure: "x", failures: 1 })); | |
| 472 | let release!: () => void; | |
| 473 | const prove = () => new Promise<Proof | undefined>((resolve) => (release = () => resolve(undefined))); | |
| 474 | const stop = startProving({ dataDir: dir, version: "4.2.0", url: "https://forge.test" }, () => {}, 60_000, () => () => {}, prove); | |
| 475 | await Bun.sleep(5); // real-timer: lets the boot tick settle | |
| 476 | expect(provingSince(dir, "4.2.0")).toBeUndefined(); | |
| 477 | release(); | |
| 478 | stop(); | |
| 479 | }); | |
| 480 | ||
| 391 | 481 | test("proveRelease aborted mid-wait stops asking and keeps no verdict (#10313)", async () => { |
| 392 | 482 | const abort = new AbortController(); |
| 393 | 483 | const fetcher = ((_u: unknown, init?: RequestInit) => |
test/tui-releases.test.ts +20 −1
| @@ -5,7 +5,7 @@ | ||
| 5 | 5 | * repository, and says so when the forge does not swap itself. |
| 6 | 6 | */ |
| 7 | 7 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; |
| 8 | import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; | |
| 8 | import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; | |
| 9 | 9 | import { tmpdir } from "node:os"; |
| 10 | 10 | import { join } from "node:path"; |
| 11 | 11 | import { createRepo } from "../src/actions/repos.ts"; |
| @@ -159,6 +159,25 @@ describe("the releases screen", () => { | ||
| 159 | 159 | } |
| 160 | 160 | }); |
| 161 | 161 | |
| 162 | test("p asks a failed proof to be tried again, for a reader who is a person; an agent or a stranger changes nothing", async () => { | |
| 163 | const file = join(root, "release", "proof.json"); | |
| 164 | const failed = () => { | |
| 165 | mkdirSync(join(root, "release"), { recursive: true }); | |
| 166 | writeFileSync(file, JSON.stringify({ version: "9.9.9", at: new Date().toISOString(), ok: false, failure: "x", failures: 1 })); | |
| 167 | }; | |
| 168 | try { | |
| 169 | expect(await open(alice, "p")).toContain("There is no failed proof to try again"); | |
| 170 | failed(); | |
| 171 | expect(await open(alice, "p")).toContain("proves this release again"); | |
| 172 | expect(JSON.parse(readFileSync(file, "utf8")).retryNow).toBe(true); | |
| 173 | failed(); | |
| 174 | await open({ ...alice, kind: "agent" } as User, "p"); | |
| 175 | expect(JSON.parse(readFileSync(file, "utf8")).retryNow).toBeUndefined(); | |
| 176 | } finally { | |
| 177 | rmSync(file, { force: true }); | |
| 178 | } | |
| 179 | }); | |
| 180 | ||
| 162 | 181 | test("the stranger's journey line: present with a passing proof of the serving version, absent without, nothing for one who may not read", () => { |
| 163 | 182 | const was = process.env["FORGE_VERSION"]; |
| 164 | 183 | process.env["FORGE_VERSION"] = "9.9.9"; |
test/vitals-release.test.ts +9
| @@ -81,3 +81,12 @@ test("a failed proof for the running version is still a card", () => { | ||
| 81 | 81 | ); |
| 82 | 82 | runningAs("v0.7.479", () => expect(card()?.sentence).toStartWith("Release 0.7.479 is unproven")); |
| 83 | 83 | }); |
| 84 | ||
| 85 | test("a probe forge the proof could not remove is a card with its Remove page; none once a proof leaves none", () => { | |
| 86 | keep({ version: "9.9.9", at: "2026-10-18T10:00:00.000Z", ok: false, failure: "a stranger's journey failed: x; the probe forge probe-abcd1234 could not be removed", leftover: ["probe-abcd1234"] }); | |
| 87 | const left = vitals({ db: app.db, signals: app.signals, dataDir }).degraded.find((d) => d.cause === "probe:probe-abcd1234"); | |
| 88 | expect(left).toMatchObject({ href: "/forges/probe-abcd1234/remove" }); | |
| 89 | expect(left?.sentence).toContain("probe-abcd1234"); | |
| 90 | keep({ version: "9.9.9", at: "2026-10-18T10:10:00.000Z", ok: true, tries: 2 }); | |
| 91 | expect(vitals({ db: app.db, signals: app.signals, dataDir }).degraded.some((d) => d.cause.startsWith("probe:"))).toBe(false); | |
| 92 | }); |
new above
offline
High: 15 min total does not cover smoke polling (up to 10 min) plus three settle+walk tries and two 2-min waits. A slow-but-healthy release is recorded as hung. Size it to the worst case.