trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns13g rEventsFlows

Flake hunt and outside gate run from the forge on their schedule and file what the Forgejo jobs filed #9029

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/9019-flake-hunt-and-outside-gate-run-from-the → main 8 commits fork

  • Merged. mason/9019-flake-hunt-and-outside-gate-run-from-the was merged into main as 7f7cfb7.

Review

Sign in to review this change.
12 files changed +555 −21
docs/DEVELOP.md +6 −2
@@ -193,8 +193,12 @@ stop (a release) aborts the running step at once. The forge needs `FORGE_TOKEN`
193193 hunt's author) and cells; a preview runs none, and a forge without either says
194194 so in its log and runs none. A job whose slot passed while the forge was down
195195 runs on waking; one restarted mid-run is not run twice (`nightly/<job>.json`
196 in `FORGE_APP_DATA`). What each last did is on Resilience and in the
197 terminal's Resilience dialog (`docs/features/nightly.md`).
196 in `FORGE_APP_DATA`). What each last did, and what it filed (`filed #12`,
197 `closed #4`, `quarantined test/x.test.ts`), is on Resilience and in the
198 terminal's Resilience dialog; the run's whole output is
199 `/resilience/nightly/<job>` (`nightly/<job>.log`), the run every issue it
200 files names, as the Forgejo jobs named their Actions run
201 (`docs/features/nightly.md`).
198202
199203 **No unseen red outside the gate.** Nightly at 03:47 UTC, the forge itself
200204 runs `bun scripts/flake-hunt.ts outside --file-issues`: every file in
docs/features/nightly.md +15 −2
@@ -9,7 +9,9 @@ Both are `scripts/flake-hunt.ts`, run in a cell on a checkout of the default bra
99
1010 ## Seeing it
1111
12 Resilience (admins) and the terminal's Resilience dialog say, per job, when it last ran and what it filed or why it did not run. The record is `FORGE_APP_DATA/nightly/<job>.json`.
12 Resilience (admins) and the terminal's Resilience dialog say, per job, when it last ran and what it filed (`filed #12, closed #4, quarantined test/x.test.ts`, or `filed nothing`) or why it did not run. The record is `FORGE_APP_DATA/nightly/<job>.json`. What it filed is read from the hunt script's own lines (the tests it runs print to the hunt, not to the output), so it is what the default branch's script says it did: advisory, the issues are the truth.
13
14 Each line links to the job's newest output, `/resilience/nightly/<job>` (admins): every step's command and what it printed, kept in `FORGE_APP_DATA/nightly/<job>.log`, replaced by the next run and cut at 1 MB. In the terminal's Resilience dialog `o` shows the tail of each. It is the run a `Flaky:` or `Red:` issue names (`FORGE_RUN_URL`), where the Forgejo jobs named their Actions run.
1315
1416 ## When it runs
1517
@@ -21,4 +23,15 @@ With a token and cells, a job runs when its slot comes. To run one now from a te
2123
2224 ## What the Forgejo workflows were
2325
24 `flake-hunt.yml` and `outside-gate.yml` stay on the mirror until the forge's own run has filed one real result. Both dedupe on the open `Flaky:` and `Red:` issues, so they cannot double-file; the pull that deletes them carries one night's output from each side. `sync-env.yml` sealed the repository's Actions secrets into the platform's `forge-user-env` Secret, which fed the pod's environment (the crew's Claude credential, `FORGE_TOKEN`). The forge already holds its own secrets, so there is nothing to port: the Secret goes with the pod (`docs/PLATFORM.md`, rows 2 and 3).
26 `flake-hunt.yml` and `outside-gate.yml` stay on the mirror until the forge's own run has filed one real result. Both dedupe on the open `Flaky:` and `Red:` issues the token's own user filed, so they cannot double-file while the forge's `FORGE_TOKEN` and the mirror's secret act as the same user (the Forgejo hunt's issues are Mason's); with two users, each files its own, which is the comparison in plain sight. The pull that deletes them (#9020) carries one night's output from each side.
27
28 `sync-env.yml` sealed the repository's Actions secrets into the platform's `forge-user-env` Secret, which fed the pod's environment (the crew's Claude credential, `FORGE_TOKEN`). The forge already holds its own secrets, so there is nothing to port: the Secret goes with the pod (`docs/PLATFORM.md`, rows 2 and 3).
29
30 ### Comparing one night
31
32 For one date, set side by side:
33
34 - **Forgejo:** the `flake-hunt` and `outside-gate` Actions runs on the mirror (each slice's log, the quarantine job's log).
35 - **Forge:** `/resilience/nightly/flake-hunt` and `/resilience/nightly/outside-gate`, and the line above each: what it filed.
36
37 They agree when the outside run lists the same files green and red, both open, update and close the same `Red:` issues (or one finds the other's already open), and the hunt's flaky files, which vary with the seed, are each filed once. A file red on one side and green on the other is the finding: the cell (no network but the forge and the registry, its CPU and memory caps) differs from the `isolated` runner.
src/http/server.ts +2 −1
@@ -25,7 +25,7 @@ import { prerenderComments } from "../web/prerender.ts";
2525 import { systemPage, systemStream } from "../web/pages/system.ts";
2626 import { contributePage } from "../web/pages/contribute.ts";
2727 import { pipelinePage } from "../web/pages/pipeline.ts";
28 import { resiliencePage } from "../web/pages/resilience.ts";
28 import { nightlyOutputPage, resiliencePage } from "../web/pages/resilience.ts";
2929 import { limitsPage, limitsSet, quotasSet, signupSet, inviteMint, inviteRevoke } from "../web/pages/admin-limits.ts";
3030 import { measureSubmit } from "./measure.ts";
3131 import { settingsRoutes } from "./settings_routes.ts";
@@ -827,6 +827,7 @@ export function buildRouter(): Router<Handler> {
827827 pinnedRoutes(router);
828828 // Each THREATS.md protection against the last gate; literal, above /:owner.
829829 router.add("GET", "/resilience", resiliencePage);
830 router.add("GET", "/resilience/nightly/:job", nightlyOutputPage);
830831 // A selection's spend and runs over time; literal, above /:owner.
831832 router.add("GET", "/fleet/graph", fleetGraphPage);
832833 router.add("GET", "/fleet/graph/live", fleetGraphLive);
src/nightly/output.ts added +136
@@ -0,0 +1,136 @@
1 /**
2 * What a nightly job printed, kept: the newest run's output of each job in
3 * one file under `FORGE_APP_DATA/nightly`, so the `Flaky:` and `Red:`
4 * issues it files can name the run (`runPath`) as the Forgejo jobs named
5 * theirs, and one night here can be read beside one night there. Capped,
6 * replaced by the next run, written in batches off the request path.
7 *
8 * `filedIn` reads back what the run filed from the lines
9 * `scripts/flake-hunt.ts` prints when it does, and only from the hunt's own
10 * steps (`readsFiled`): the hunt keeps the output of the tests it runs to
11 * itself, so a test cannot print a line into the record. The record is
12 * still advisory: it is what the default branch's script says it did, and
13 * the issues themselves are the truth. Plain data.
14 */
15
16 import { closeSync, fstatSync, mkdirSync, openSync, readFileSync, readSync, writeFileSync } from "node:fs";
17 import { appendFile } from "node:fs/promises";
18 import { join } from "node:path";
19 import type { Step } from "./jobs.ts";
20
21 /** Past this, a run's output is cut with a line that says so. */
22 export const OUTPUT_CAP_BYTES = 1_000_000;
23 /** Lines wait this long, or until this many bytes are waiting, before they are written. */
24 const FLUSH_MS = 1000;
25 const FLUSH_BYTES = 64 * 1024;
26
27 const fileOf = (dataDir: string, name: string): string => join(dataDir, "nightly", `${name}.log`);
28
29 /** The page that shows a job's newest output: what an issue it files links to. */
30 export const runPath = (name: string): string => `/resilience/nightly/${name}`;
31
32 export interface Output {
33 /** Appends a line; it reaches the file within FLUSH_MS. */
34 readonly write: (line: string) => void;
35 /** Writes what waits; the output is whole once it settles. */
36 readonly end: () => Promise<void>;
37 }
38
39 /** A fresh output file for `name`: the last night's is gone once this returns. */
40 export function startOutput(dataDir: string, name: string): Output {
41 mkdirSync(join(dataDir, "nightly"), { recursive: true });
42 const file = fileOf(dataDir, name);
43 writeFileSync(file, "");
44 let bytes = 0;
45 let cut = false;
46 let waiting = "";
47 let timer: ReturnType<typeof setTimeout> | undefined;
48 // One write at a time, in order; a full disk loses output, never the run.
49 let written: Promise<void> = Promise.resolve();
50 const flush = (): Promise<void> => {
51 if (timer !== undefined) clearTimeout(timer);
52 timer = undefined;
53 if (waiting === "") return written;
54 const text = waiting;
55 waiting = "";
56 written = written.then(() => appendFile(file, text)).catch(() => undefined);
57 return written;
58 };
59 const add = (text: string): void => {
60 waiting += text;
61 if (waiting.length >= FLUSH_BYTES) void flush();
62 else if (timer === undefined) {
63 timer = setTimeout(() => void flush(), FLUSH_MS);
64 timer.unref?.();
65 }
66 };
67 return {
68 write: (line) => {
69 if (cut) return;
70 const text = `${line}\n`;
71 bytes += Buffer.byteLength(text);
72 if (bytes > OUTPUT_CAP_BYTES) {
73 cut = true;
74 add(`… cut at ${OUTPUT_CAP_BYTES} bytes\n`);
75 return;
76 }
77 add(text);
78 },
79 end: flush,
80 };
81 }
82
83 /** The newest output of `name`, or undefined when it never ran here. */
84 export function readOutput(dataDir: string, name: string): string | undefined {
85 try {
86 return readFileSync(fileOf(dataDir, name), "utf8");
87 } catch {
88 return undefined;
89 }
90 }
91
92 /** How far back `outputTail` reads: the tail never costs a whole capped file. */
93 const TAIL_BYTES = 16 * 1024;
94
95 /** The last `count` lines of `name`'s newest output, or undefined when it never ran here. */
96 export function outputTail(dataDir: string, name: string, count: number): string[] | undefined {
97 let fd: number;
98 try {
99 fd = openSync(fileOf(dataDir, name), "r");
100 } catch {
101 return undefined;
102 }
103 try {
104 const size = fstatSync(fd).size;
105 const from = Math.max(0, size - TAIL_BYTES);
106 const buf = Buffer.alloc(size - from);
107 readSync(fd, buf, 0, buf.length, from);
108 const lines = buf.toString("utf8").split("\n");
109 if (lines.at(-1) === "") lines.pop();
110 // A read from the middle of the file starts mid-line: that piece is dropped.
111 if (from > 0) lines.shift();
112 return lines.slice(-count);
113 } finally {
114 closeSync(fd);
115 }
116 }
117
118 /** Whether a step's output says what it filed: the hunt script's steps, never install. */
119 export const readsFiled = (step: Step): boolean => step.command.startsWith("bun scripts/flake-hunt.ts ");
120
121 // ` filed #12 for test/x.test.ts`, ` filed #12`, ` updated #3 for …`,
122 // ` closed #4 for …`, `quarantined …`: what flake-hunt.ts prints as it files.
123 const FILED = /^\s*(filed|updated|closed) #(\d+)\b/;
124 const QUARANTINED = /^quarantined (\S+)/;
125
126 /** What a run filed, in the order it did, as short phrases: `filed #12`, `quarantined test/x.test.ts`. */
127 export function filedIn(lines: readonly string[]): string[] {
128 const out: string[] = [];
129 for (const line of lines) {
130 const filed = FILED.exec(line);
131 if (filed !== null) out.push(`${filed[1]} #${filed[2]}`);
132 const moved = QUARANTINED.exec(line);
133 if (moved !== null) out.push(`quarantined ${moved[1]}`);
134 }
135 return out;
136 }
src/nightly/run.ts +40 −7
@@ -3,7 +3,8 @@
33 * own, each step run in a cell (`apps/contain.ts`) that reaches the forge
44 * and the registry and nothing else, the step's output kept in the log.
55 * The token goes only to steps that report, as FORGE_TOKEN, which the hunt
6 * script already drops from the environment of the tests it runs.
6 * script already drops from the environment of the tests it runs. The
7 * output is kept (`output.ts`) and is the run the issues it files link to.
78 */
89
910 import { mkdirSync } from "node:fs";
@@ -18,6 +19,7 @@ import { repoDir } from "../git/repos.ts";
1819 import { refTip } from "../git/read.ts";
1920 import { createLog } from "../kernel/log.ts";
2021 import type { Job } from "./jobs.ts";
22 import { filedIn, readsFiled, runPath, startOutput } from "./output.ts";
2123
2224 const log = createLog("nightly");
2325
@@ -26,6 +28,8 @@ export interface Ran {
2628 readonly oid: string;
2729 /** The labels of the steps that failed. */
2830 readonly failed: readonly string[];
31 /** What the hunt's steps filed, as `filedIn` reads it from their output: advisory. */
32 readonly filed: readonly string[];
2933 }
3034
3135 export async function runJob(
@@ -34,17 +38,38 @@ export async function runJob(
3438 job: Job,
3539 options: { readonly forgeUrl: string; readonly token: string; readonly signal: AbortSignal },
3640 ): Promise<Ran | { readonly error: string }> {
41 // Begun before anything can refuse, so the output shown is never an older night's.
42 const output = startOutput(app.config.dataDir, job.name);
43 try {
44 return await runSteps(app, repo, job, options, output.write);
45 } finally {
46 await output.end();
47 }
48 }
49
50 async function runSteps(
51 app: App,
52 repo: Repo,
53 job: Job,
54 options: { readonly forgeUrl: string; readonly token: string; readonly signal: AbortSignal },
55 keep: (line: string) => void,
56 ): Promise<Ran | { readonly error: string }> {
57 const refuse = (error: string): { readonly error: string } => {
58 keep(`not run: ${error}`);
59 return { error };
60 };
3761 const bare = repoDir(app.reposRoot, repo.ownerName, repo.name);
3862 const oid = await refTip(bare, `refs/heads/${repo.defaultBranch}`);
39 if (oid === undefined) return { error: `no ${repo.defaultBranch} to check out` };
63 if (oid === undefined) return refuse(`no ${repo.defaultBranch} to check out`);
64 keep(`${job.name} on ${repo.ownerName}/${repo.name} ${repo.defaultBranch} at ${oid}`);
4065 const dir = join(app.config.dataDir, "nightly", `${job.name}-checkout`);
4166 mkdirSync(join(app.config.dataDir, "nightly"), { recursive: true });
4267 const exported = await exportTree(bare, oid, dir);
43 if (!exported.ok) return { error: exported.error };
68 if (!exported.ok) return refuse(exported.error);
4469
4570 const cells = await cellsAvailable();
4671 // Repository code, run unattended with a token in its hands: never without a cell.
47 if (!cells.ok) return { error: `no cell to run it in: ${cells.reason}` };
72 if (!cells.ok) return refuse(`no cell to run it in: ${cells.reason}`);
4873 const cell: CellSpec = {
4974 dir,
5075 writable: [],
@@ -56,28 +81,36 @@ export async function runJob(
5681 label: `nightly:${job.name}`,
5782 };
5883 const failed: string[] = [];
84 const filed: string[] = [];
5985 for (const step of job.steps) {
6086 if (options.signal.aborted) break;
87 const reads = readsFiled(step);
88 const line = (text: string): void => {
89 keep(text);
90 if (reads) filed.push(...filedIn([text]));
91 log.info(text, { job: job.name });
92 };
6193 const env = childEnv({
6294 FORGE_API: `${options.forgeUrl}/api/v1`,
6395 FORGE_REPO: `${repo.ownerName}/${repo.name}`,
64 FORGE_RUN_URL: `${options.forgeUrl}/${repo.ownerName}/${repo.name}`,
96 FORGE_RUN_URL: `${options.forgeUrl}${runPath(job.name)}`,
6597 FORGE_CELL_CPUS: String(cell.limits.cpus),
6698 ...(step.reports ? { FORGE_TOKEN: options.token } : {}),
6799 });
68100 const ran = await runStep(step.label, step.command, {
69101 cwd: dir,
70102 env,
71 log: (line) => log.info(line, { job: job.name }),
103 log: line,
72104 signal: options.signal,
73105 timeoutMs: step.timeoutMs,
74106 cell,
75107 });
76108 if (ran.ok) continue;
77109 log.warn("step failed", { job: job.name, step: step.label, error: ran.error });
110 keep(`step failed: ${ran.error}`);
78111 failed.push(step.label);
79112 // Nothing runs without its dependencies.
80113 if (step.label === "install") break;
81114 }
82 return { dir, oid, failed };
115 return { dir, oid, failed, filed };
83116 }
src/nightly/service.ts +3 −1
@@ -56,8 +56,10 @@ function doing(app: App, options: { readonly forgeUrl: string; readonly token: s
5656 };
5757 }
5858
59 const says = (ran: Ran, branch: string | undefined): string =>
59 /** One sentence: whether the steps finished, what they filed, and the pull. */
60 export const says = (ran: Ran, branch: string | undefined): string =>
6061 (ran.failed.length === 0 ? "every step finished" : `failed: ${ran.failed.join(", ")}`) +
62 (ran.filed.length === 0 ? "; filed nothing" : `; ${ran.filed.join(", ")}`) +
6163 (branch === undefined ? "." : `; quarantine pull on ${branch}.`);
6264
6365 export function startNightly(
src/tui/screens/resilience.ts +37 −1
@@ -18,6 +18,10 @@
1818 * key, secret key — both keys typed masked; a blank keeps what was sealed).
1919 * `t` tries the destination in effect, as `testDestination` says it.
2020 * A key is never drawn, kept past the save, or put in a note.
21 *
22 * `o`, a forge admin's as /resilience/nightly/<job> is, opens the tail of
23 * each nightly job's newest output (`nightly/output.ts`): the run its
24 * `Flaky:` and `Red:` issues link to.
2125 */
2226
2327 import { drillNow, mirrorNow, recentSnapshots, snapshotNow } from "../../actions/backups_now.ts";
@@ -31,6 +35,8 @@ import {
3135 type DestinationInput,
3236 } from "../../actions/destination.ts";
3337 import { forgetMirror, saveMirror } from "../../actions/mirror.ts";
38 import { JOBS } from "../../nightly/jobs.ts";
39 import { outputTail } from "../../nightly/output.ts";
3440 import { nightlyLines } from "../../nightly/says.ts";
3541 import { backupsLine } from "../../snapshot/backups_line.ts";
3642 import { backupVerdict } from "../../snapshot/bucket.ts";
@@ -55,6 +61,25 @@ export function resilienceLines(dataDir: string): string[] {
5561 return rows.flatMap(({ p, s }) => wrap(`${pad(s, 11)} ${p.text}`, INNER));
5662 }
5763
64 /** How many of a nightly job's last lines the output dialog shows. */
65 export const OUTPUT_TAIL = 12;
66
67 /** Each nightly job's line and the tail of its newest output, as /resilience/nightly/<job> shows it whole. */
68 export function nightlyOutputLines(dataDir: string): string[] {
69 const said = nightlyLines(dataDir);
70 return JOBS.flatMap((job, i) => {
71 const tail = outputTail(dataDir, job.name, OUTPUT_TAIL);
72 return [
73 ...(i === 0 ? [] : [""]),
74 ...wrap(said[i] ?? job.name, OUTPUT_INNER),
75 ...(tail === undefined ? [" no output here yet: it has not run on this forge"] : tail.map((l) => ` ${l}`)),
76 ];
77 });
78 }
79
80 const OUTPUT_WIDTH = 100;
81 const OUTPUT_INNER = OUTPUT_WIDTH - 6;
82
5883 /** The source of the destination in effect, in the Backups page's words. */
5984 const SOURCE: Record<string, string> = {
6085 sealed: "sealed in this forge",
@@ -283,6 +308,7 @@ export function resilienceScreen(f: Session, ctx: Ctx): Screen {
283308 binding(["l"], "bring the mirror level now"),
284309 binding(["r"], "run a restore drill"),
285310 ]),
311 ...(mayAct() ? [binding(["o"], "nightly output")] : []),
286312 ...(mayChange()
287313 ? [
288314 binding(["d"], "set destination"),
@@ -318,7 +344,17 @@ export function resilienceScreen(f: Session, ctx: Ctx): Screen {
318344 if (ask !== undefined && !busy && mayAct()) {
319345 asking = ask;
320346 said = "";
321 } else if (e.key === "enter" || e.key === "esc" || e.key === "q") ctx.pop();
347 } else if (e.key === "o" && mayAct())
348 ctx.push(
349 confirmScreen(ctx, screen, {
350 title: "Nightly output",
351 body: nightlyOutputLines(f.app.config.dataDir),
352 actions: [{ label: "Close" }],
353 onChoose: () => undefined,
354 width: OUTPUT_WIDTH,
355 }),
356 );
357 else if (e.key === "enter" || e.key === "esc" || e.key === "q") ctx.pop();
322358 else if (!mayChange()) return true;
323359 else if (e.key === "d") ctx.push(step(destination, 0, BLANK));
324360 else if (e.key === "x") ctx.push(forgetDest());
src/web/pages/resilience.ts +25 −2
@@ -11,6 +11,8 @@
1111 import { readResilience } from "../../checks/resilience.ts";
1212 import type { Standing } from "../../core/threats.ts";
1313 import type { Ctx } from "../../http/context.ts";
14 import { JOBS } from "../../nightly/jobs.ts";
15 import { readOutput, runPath } from "../../nightly/output.ts";
1416 import { nightlyLines } from "../../nightly/says.ts";
1517 import { backupsLine } from "../../snapshot/backups_line.ts";
1618 import { notePhase, takeSessionTime, timePhase } from "../../core/route_timing.ts";
@@ -31,13 +33,34 @@ export function backupsLineHtml(ctx: Ctx): Html {
3133 return html`<p id="backups-line">${line.says} <a href="/settings/backups">Backups</a> lists them.</p>`;
3234 }
3335
34 /** What the nightly jobs last did (`nightly/says.ts`): that a night ran, and what it filed. */
36 /** What the nightly jobs last did (`nightly/says.ts`): that a night ran, what it filed, and its output. */
3537 export function nightlyHtml(ctx: Ctx): Html {
38 const lines = nightlyLines(ctx.app.config.dataDir);
3639 return html`<ul id="nightly" class="muted">
37 ${nightlyLines(ctx.app.config.dataDir).map((line) => html`<li>${line}</li>`)}
40 ${JOBS.map((job, i) => html`<li>${lines[i] ?? ""} <a href="${runPath(job.name)}">Output</a></li>`)}
3841 </ul>`;
3942 }
4043
44 // GET /resilience/nightly/:job: the newest output of one nightly job, the
45 // run its `Flaky:` and `Red:` issues link to. An admin's, as /resilience is.
46 export function nightlyOutputPage(ctx: Ctx): Response {
47 if (!ctx.auth.user) return signInFirst(ctx);
48 if (!ctx.auth.user.isAdmin) return notFoundPage(ctx);
49 const at = JOBS.findIndex((j) => j.name === ctx.params.job);
50 const job = JOBS[at];
51 if (job === undefined) return notFoundPage(ctx);
52 const line = nightlyLines(ctx.app.config.dataDir)[at] ?? "";
53 const output = readOutput(ctx.app.config.dataDir, job.name);
54 const body = html`<div class="page-head">
55 <h1>${job.name}</h1>
56 <p class="muted">${line} <a href="/resilience">Resilience</a></p>
57 </div>
58 ${output === undefined
59 ? html`<p id="nightly-output" class="muted">No output here yet: it has not run on this forge.</p>`
60 : html`<pre id="nightly-output" class="card mono" tabindex="0" aria-label="Output">${output}</pre>`}`;
61 return page({ ctx, title: `${job.name} · Resilience`, body });
62 }
63
4164 // GET /resilience: a forge admin's, since live red lines name what to attack.
4265 export function resiliencePage(ctx: Ctx): Response {
4366 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
test/nightly-output.test.ts added +177
@@ -0,0 +1,177 @@
1 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
2 import { mkdtempSync, readFileSync, rmSync } from "node:fs";
3 import { tmpdir } from "node:os";
4 import { join } from "node:path";
5 import { createApp, type App } from "../src/app.ts";
6 import { createUser } from "../src/core/users.ts";
7 import { serve, type Servers } from "../src/http/server.ts";
8 import { JOBS } from "../src/nightly/jobs.ts";
9 import { filedIn, OUTPUT_CAP_BYTES, outputTail, readOutput, readsFiled, runPath, startOutput } from "../src/nightly/output.ts";
10 import { says } from "../src/nightly/service.ts";
11 import { writeLast } from "../src/nightly/stamps.ts";
12
13 describe("a nightly job's output", () => {
14 test("what the hunt prints as it files is read back as what the run filed", () => {
15 const lines = [
16 "$ bun scripts/flake-hunt.ts outside --file-issues",
17 "red : test/a.test.ts",
18 " filed #12 for test/a.test.ts",
19 " updated #7 for test/b.test.ts",
20 " closed #4 for test/c.test.ts",
21 "FLAKY test/d.test.ts: red 1, green 4; first red seed 9",
22 " filed #13",
23 " already open as #5",
24 "quarantined test/d.test.ts (#13)",
25 "nothing to quarantine",
26 ];
27 expect(filedIn(lines)).toEqual(["filed #12", "updated #7", "closed #4", "filed #13", "quarantined test/d.test.ts"]);
28 });
29
30 test("the lines filedIn reads are the ones scripts/flake-hunt.ts prints", () => {
31 const script = readFileSync(join(import.meta.dir, "..", "scripts", "flake-hunt.ts"), "utf8");
32 for (const printed of ["` filed #${made.number} for ", "` filed #${made.number}`", "` updated #${action.number} for ", "` closed #${action.number} for ", "`quarantined ${line}`"])
33 expect(script).toContain(printed);
34 });
35
36 test("a run replaces the last one's output, and a run that prints too much is cut saying so", async () => {
37 const dir = mkdtempSync(join(tmpdir(), "nightly-out-"));
38 try {
39 expect(readOutput(dir, "outside-gate")).toBeUndefined();
40 const first = startOutput(dir, "outside-gate");
41 first.write("old night");
42 await first.end();
43 const next = startOutput(dir, "outside-gate");
44 expect(readOutput(dir, "outside-gate")).toBe("");
45 next.write("new night");
46 await next.end();
47 expect(readOutput(dir, "outside-gate")).toBe("new night\n");
48 const big = "x".repeat(1000);
49 for (let i = 0; i < OUTPUT_CAP_BYTES / 1000 + 10; i++) next.write(big);
50 await next.end();
51 const kept = readOutput(dir, "outside-gate")!;
52 expect(kept.length).toBeLessThan(OUTPUT_CAP_BYTES + 100);
53 expect(kept.endsWith(`… cut at ${OUTPUT_CAP_BYTES} bytes\n`)).toBe(true);
54 } finally {
55 rmSync(dir, { recursive: true, force: true });
56 }
57 });
58
59 test("lines are written in batches, not one write each, and in order", async () => {
60 const dir = mkdtempSync(join(tmpdir(), "nightly-out-"));
61 try {
62 const out = startOutput(dir, "flake-hunt");
63 for (let i = 0; i < 100; i++) out.write(`line ${i}`);
64 // Nothing has been written while the event loop was busy with the run.
65 expect(readOutput(dir, "flake-hunt")).toBe("");
66 await out.end();
67 expect(readOutput(dir, "flake-hunt")).toBe(Array.from({ length: 100 }, (_, i) => `line ${i}\n`).join(""));
68 } finally {
69 rmSync(dir, { recursive: true, force: true });
70 }
71 });
72
73 test("the tail is the last lines, read from the end of a long output", async () => {
74 const dir = mkdtempSync(join(tmpdir(), "nightly-out-"));
75 try {
76 expect(outputTail(dir, "flake-hunt", 3)).toBeUndefined();
77 const out = startOutput(dir, "flake-hunt");
78 for (let i = 0; i < 5000; i++) out.write(`line ${i}`);
79 await out.end();
80 expect(outputTail(dir, "flake-hunt", 3)).toEqual(["line 4997", "line 4998", "line 4999"]);
81 const all = outputTail(dir, "flake-hunt", 100_000)!;
82 // Only the end of the file is read, and the line it cut through is dropped.
83 expect(all.length).toBeLessThan(5000);
84 expect(all[0]).toMatch(/^line \d+$/);
85 } finally {
86 rmSync(dir, { recursive: true, force: true });
87 }
88 });
89
90 test("only the hunt script's steps are read for what they filed, never install", () => {
91 for (const job of JOBS)
92 for (const step of job.steps) expect([step.label, readsFiled(step)]).toEqual([step.label, step.label !== "install"]);
93 });
94
95 test("the record says what the run filed, or that it filed nothing", () => {
96 expect(says({ dir: "x", oid: "y", failed: [], filed: [] }, undefined)).toBe("every step finished; filed nothing.");
97 expect(says({ dir: "x", oid: "y", failed: ["hunt slice 2/20"], filed: ["filed #12", "quarantined test/a.test.ts"] }, "flake-hunt/2026-11-03")).toBe(
98 "failed: hunt slice 2/20; filed #12, quarantined test/a.test.ts; quarantine pull on flake-hunt/2026-11-03.",
99 );
100 });
101 });
102
103 describe("the page a nightly issue links to", () => {
104 let dataDir: string;
105 let app: App;
106 let servers: Servers;
107
108 const login = async (username: string): Promise<string> => {
109 const res = await fetch(`${servers.url}/signin`, {
110 method: "POST",
111 redirect: "manual",
112 headers: { "content-type": "application/x-www-form-urlencoded", origin: servers.url },
113 body: new URLSearchParams({ username, password: "a-good-passphrase" }),
114 });
115 return (res.headers.get("set-cookie") ?? "").split(";")[0] ?? "";
116 };
117
118 beforeAll(async () => {
119 dataDir = mkdtempSync(join(tmpdir(), "nightly-page-"));
120 app = createApp({ dataDir, port: 0 });
121 servers = await serve(app);
122 for (const [username, adminIfFirst] of [["alice", true], ["bob", false]] as const) {
123 const u = await createUser(app.db, {
124 origin: "seed",
125 trusted: true,
126 username,
127 email: `${username}@forge.local`,
128 password: "a-good-passphrase",
129 adminIfFirst,
130 });
131 if (!u.ok) throw new Error("users");
132 }
133 const kept = startOutput(dataDir, "outside-gate");
134 kept.write("red : test/<b>x</b>.test.ts");
135 kept.write(" filed #12 for test/<b>x</b>.test.ts");
136 await kept.end();
137 writeLast(dataDir, "outside-gate", {
138 slot: Date.parse("2026-11-03T03:47:00Z"),
139 startedAt: Date.parse("2026-11-03T03:47:01Z"),
140 finishedAt: Date.parse("2026-11-03T04:02:00Z"),
141 says: "every step finished; filed #12.",
142 });
143 });
144
145 afterAll(async () => {
146 servers.stop();
147 app.close();
148 rmSync(dataDir, { recursive: true, force: true });
149 });
150
151 test("an admin reads the run's record and its output, escaped, and Resilience links to it", async () => {
152 const cookie = await login("alice");
153 const res = await fetch(`${servers.url}${runPath("outside-gate")}`, { headers: { cookie } });
154 expect(res.status).toBe(200);
155 const text = await res.text();
156 expect(text).toContain("every step finished; filed #12.");
157 expect(text).toContain("filed #12 for test/&lt;b&gt;x&lt;/b&gt;.test.ts");
158 expect(text).not.toContain("<b>x</b>");
159 const resilience = await (await fetch(`${servers.url}/resilience`, { headers: { cookie } })).text();
160 expect(resilience).toContain(`href="${runPath("outside-gate")}"`);
161 expect(resilience).toContain(`href="${runPath("flake-hunt")}"`);
162 });
163
164 test("a job that has not run says so", async () => {
165 const res = await fetch(`${servers.url}${runPath("flake-hunt")}`, { headers: { cookie: await login("alice") } });
166 expect(res.status).toBe(200);
167 expect(await res.text()).toContain("No output here yet");
168 });
169
170 test("anyone but an admin, and any job that is not one, is not found", async () => {
171 expect((await fetch(`${servers.url}${runPath("outside-gate")}`, { headers: { cookie: await login("bob") } })).status).toBe(404);
172 expect((await fetch(`${servers.url}${runPath("sync-env")}`, { headers: { cookie: await login("alice") } })).status).toBe(404);
173 const anon = await fetch(`${servers.url}${runPath("outside-gate")}`, { redirect: "manual" });
174 expect(anon.status).not.toBe(200);
175 expect(await anon.text()).not.toContain("filed #12");
176 });
177 });
test/nightly.test.ts +5 −5
@@ -95,7 +95,7 @@ describe("the nightly service", () => {
9595 const instant = {
9696 run: async (job: { name: string }) => {
9797 calls.push(job.name);
98 return { dir: "x", oid: "y", failed: [] as string[] };
98 return { dir: "x", oid: "y", failed: [] as string[], filed: [] as string[] };
9999 },
100100 propose: async () => undefined,
101101 };
@@ -160,9 +160,9 @@ describe("the nightly service", () => {
160160 const slow = {
161161 run: (_job: unknown, signal: AbortSignal) => {
162162 seen = signal;
163 return new Promise<{ dir: string; oid: string; failed: string[] }>((resolve) => {
164 signal.addEventListener("abort", () => resolve({ dir: "x", oid: "y", failed: ["hunt slice 3/20"] }));
165 release = () => resolve({ dir: "x", oid: "y", failed: [] });
163 return new Promise<{ dir: string; oid: string; failed: string[]; filed: string[] }>((resolve) => {
164 signal.addEventListener("abort", () => resolve({ dir: "x", oid: "y", failed: ["hunt slice 3/20"], filed: [] }));
165 release = () => resolve({ dir: "x", oid: "y", failed: [], filed: [] });
166166 });
167167 },
168168 propose: async () => undefined,
@@ -195,7 +195,7 @@ describe("the nightly service", () => {
195195 test("the hunt proposes after its steps, unless the quarantine itself failed; the outside run never does", async () => {
196196 let proposed = 0;
197197 const doing = (failed: string[]) => ({
198 run: async () => ({ dir: "x", oid: "y", failed }),
198 run: async () => ({ dir: "x", oid: "y", failed, filed: [] }),
199199 propose: async () => {
200200 proposed++;
201201 return "flake-hunt/2026-11-02";
test/security-floor-routes.test.ts +1
@@ -108,6 +108,7 @@ const UNSEEDED: Record<string, string> = {
108108 "/origins/:id/r/:owner/:name/raw/:ref/*path": ORIGIN,
109109 "/origins/:id/r/:owner/:name/tree/:ref": ORIGIN,
110110 "/origins/:id/r/:owner/:name/tree/:ref/*path": ORIGIN,
111 "/resilience/nightly/:job": "a nightly job's name (flake-hunt, outside-gate); the placeholder names none, so its not-found page is what gets the floor checked",
111112 "/settings/origins/:id/mirror/:job": ORIGIN,
112113 "/settings/origins/:id/repos": ORIGIN,
113114 "/settings/origins/flows/:id": ORIGIN,
test/tui-nightly-output.test.ts added +108
@@ -0,0 +1,108 @@
1 /**
2 * The terminal's door to a nightly run's output (#9019): `o` in the
3 * Resilience dialog, a forge admin's as /resilience/nightly/<job> is, shows
4 * each job's line and the tail of its newest output.
5 */
6 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { createApp, type App as Forge } from "../src/app.ts";
11 import { createUser, getUserByName } from "../src/core/users.ts";
12 import { startOutput } from "../src/nightly/output.ts";
13 import { writeLast } from "../src/nightly/stamps.ts";
14 import { createForgeApp } from "../src/tui/forge.ts";
15 import { resilienceScreen, OUTPUT_TAIL } from "../src/tui/screens/resilience.ts";
16 import type { TuiIdentity } from "../src/tui/session.ts";
17 import { headlessTerminal } from "../src/tui/term/terminal.ts";
18 import { quietLogs } from "./helpers/globals.ts";
19 import { settleApp } from "./helpers/tui.ts";
20
21 quietLogs();
22
23 describe("a nightly run's output in the terminal", () => {
24 let dataDir: string;
25 let forge: Forge;
26
27 function boot(username: string) {
28 const terminal = headlessTerminal(140, 48);
29 const user = getUserByName(forge.db, username)!;
30 const identity: TuiIdentity = { sshUsername: "t", remoteAddress: "127.0.0.1", user };
31 const { app, session } = createForgeApp(forge, identity, { terminal, exitHooks: false });
32 app.start();
33 const send = (k: string, ctrl = false) => app.dispatch({ type: "key", key: k, ctrl, alt: false, shift: false });
34 const press = async (...keys: string[]) => {
35 for (const k of keys) send(k);
36 await settleApp(app);
37 };
38 const open = async () => {
39 send("k", true);
40 await settleApp(app);
41 for (const ch of "Resilience") send(ch);
42 await settleApp(app);
43 await press("enter");
44 };
45 return { app, session, text: () => app.screen.toText(), press, open, stop: () => app.stop() };
46 }
47
48 beforeAll(async () => {
49 dataDir = await mkdtemp(join(tmpdir(), "forge-tui-nightly-output-"));
50 forge = createApp({ dataDir, port: 0 });
51 for (const [username, adminIfFirst] of [["alice", true], ["bob", false]] as const) {
52 const made = await createUser(forge.db, {
53 origin: "seed",
54 trusted: true,
55 username,
56 email: `${username}@forge.local`,
57 password: "a-good-passphrase",
58 adminIfFirst,
59 });
60 if (!made.ok) throw new Error(made.error);
61 }
62 const out = startOutput(dataDir, "outside-gate");
63 out.write("an early line the tail leaves out");
64 for (let i = 0; i < OUTPUT_TAIL - 1; i++) out.write(`green: test/f${i}.test.ts`);
65 out.write(" filed #12 for test/x.test.ts");
66 await out.end();
67 writeLast(dataDir, "outside-gate", {
68 slot: Date.parse("2026-11-03T03:47:00Z"),
69 startedAt: Date.parse("2026-11-03T03:47:01Z"),
70 finishedAt: Date.parse("2026-11-03T04:02:00Z"),
71 says: "every step finished; filed #12.",
72 });
73 });
74
75 afterAll(async () => {
76 forge.close();
77 await rm(dataDir, { recursive: true, force: true });
78 });
79
80 test("an admin presses o and reads each job's line and the tail of its output", async () => {
81 const h = boot("alice");
82 await h.open();
83 expect(h.text()).toContain("o nightly output");
84 await h.press("o");
85 const text = h.text();
86 expect(text).toContain("Nightly output");
87 expect(text).toContain("every step finished; filed #12.");
88 expect(text).toContain("filed #12 for test/x.test.ts");
89 expect(text).toContain("green: test/f0.test.ts");
90 expect(text).not.toContain("an early line the tail leaves out");
91 expect(text).toContain("no output here yet: it has not run on this forge");
92 await h.press("esc");
93 expect(h.text()).not.toContain("Nightly output");
94 expect(h.text()).toContain("Snapshots, newest first:");
95 h.stop();
96 });
97
98 test("anyone but an admin is not offered o, and o opens nothing for them", async () => {
99 const h = boot("bob");
100 const pushed: string[] = [];
101 const ctx = { ...h.app.ctx, push: (s: { title: string }) => void pushed.push(s.title), invalidate: () => undefined };
102 const screen = resilienceScreen(h.session, ctx as never);
103 expect(screen.keys.some((k) => k.help === "nightly output")).toBe(false);
104 screen.key({ type: "key", key: "o", ctrl: false, alt: false, shift: false });
105 expect(pushed).toEqual([]);
106 h.stop();
107 });
108 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs