trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

Move the Forgejo Actions jobs that are not releases into the forge: flake hunt, outside gate, sync-env #8957

Merged masoncrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/8943-move-the-forgejo-actions-jobs-that-are-n → main 3 commits fork 3 threads · 3 unresolved

  • Merged. mason/8943-move-the-forgejo-actions-jobs-that-are-n was merged into main as 17ba943.

Review

Sign in to review this change.
15 files changed +752 −10
docs/DEVELOP.md +20 −7
@@ -164,8 +164,8 @@ own green check, not ten runs. That is accepted on purpose — a flaky new
164164 test can turn the gate red once before anyone sees it — and the nightly
165165 flake hunt below is the backstop that files it and quarantines it.
166166
167 **No flaky tests.** Nightly, `.forgejo/workflows/flake-hunt.yml` runs every
168 gate file five times on a loaded box, each run a fresh seed that orders both
167 **No flaky tests.** Nightly at 02:17 UTC the forge itself (`src/nightly/`,
junipercrew outdated

This line runs far past the file's wrap width; rewrap it.
168 a cell on a checkout of main) runs every gate file five times on a loaded box, each run a fresh seed that orders both
169169 the files and (through `--randomize`) the tests inside them, nothing stopped
170170 early. A file red in some runs and green in others is flaky: the hunt files
171171 a `Flaky: test/x.test.ts` issue carrying the failure text, the seed and the
@@ -182,18 +182,31 @@ a retry: `test/flake.test.ts` fails on `--retry`, `--rerun-each`, a bunfig
182182 `retry` or a `{ retry: N }` anywhere the gate runs, and on any test that
183183 bounds a measured time under 200 ms (a loopback fetch on a loaded runner
184184 takes tens of ms): assert relative to a baseline from the same run, or bound
185 only the failure the test names. The workflow needs a
186 `FORGE_TOKEN` secret and a `FORGE_URL` variable on the mirror.
185 only the failure the test names. The hunt takes the gate in
186 20 slices, one after another, each cut at 30 minutes, and ends by proposing the
187 quarantine as a pull on `flake-hunt/<date>`. Worst case, all 20 slices hit
188 their cut: 20 x 30 minutes, plus the install and the quarantine (10 each), so
189 about 10 h 20 min, ending near 12:40 UTC. Jobs run one at a time, so the
190 outside run (03:47) queues behind a long hunt and starts late; it still keeps
191 its own slot, which is what the record and the owed-on-waking rule read. A
192 stop (a release) aborts the running step at once. The forge needs `FORGE_TOKEN` (the
193 hunt's author) and cells; a preview runs none, and a forge without either says
194 so in its log and runs none. A job whose slot passed while the forge was down
195 runs on waking; one restarted mid-run is not run twice (`nightly/<job>.json`
196 in `FORGE_APP_DATA`). What each last did is on Resilience and in the
197 terminal's Resilience dialog (`docs/features/nightly.md`).
187198
188 **No unseen red outside the gate.** Nightly, `.forgejo/workflows/outside-gate.yml`
199 **No unseen red outside the gate.** Nightly at 03:47 UTC, the forge itself
189200 runs `bun scripts/flake-hunt.ts outside --file-issues`: every file in
190201 `EXCLUDED` (the complement of the gate, from the same list, so a file moved
191202 into the gate is no longer run there), once, one at a time, with the gate's
192203 60 s timeout. A file that newly fails opens one `Red: test/x.test.ts` issue
193204 naming its first failing test; still red on another test, it updates that
194205 issue; still red on the same test, it says nothing, so the owner's asks show
195 it once; passing again, it closes the issue. Same `FORGE_TOKEN` and
196 `FORGE_URL` as the hunt.
206 it once; passing again, it closes the issue. Same `FORGE_TOKEN` as the hunt.
207
208 The Forgejo workflows these replace, and `sync-env`, are covered in
209 `docs/features/nightly.md`.
197210
198211 Bun's own `--randomize --seed=N` drives every hunt and admission run. Not
199212 used: `--rerun-each` (one seed for every rerun, so ten reruns explore one
docs/PLATFORM.md +1 −1
@@ -55,7 +55,7 @@ What it did **not** retire, because production does not run it yet:
5555 | 7 | **Forgejo, where outside people send pulls.** grokbot's #42 to #67, `jmoran/forge-jay` (#8935), tdobbertin-espo's six (#8938) on git.open-platform.sh. | The forge: a real merge for a kept fork (#8935), a URL-to-landed-pull path for an agent (#6359), forge-to-forge origin (`docs/features/forge-origin.md`). | Origins and Bring exist (`docs/ORIGINS.md`). | Tell each person, then disable (not delete) new pulls on the Forgejo repository (#8950; waits for #6359, #8935). |
5656 | 8 | **Hosted apps.** Apps the platform runs that are not on the forge. The list is in the cluster: **unread**. | Bring (paste a link) or Migrate; GitHub and Vercel go through #6007, not here. | Bring an app whole (#8919). | List with the owner's call per app (#8951). |
5757 | 9 | **The droplet release tick.** `forge-release-tick` on pg-grand and the Mac's `tick release`. | The keeper (#1). | The keeper's code. | Switched off by the owner on the day #8942 records. |
58 | 10 | **Forgejo Actions jobs that are not releases.** `flake-hunt.yml`, `outside-gate.yml`, `sync-env.yml` on platform runners. | Scheduled jobs the forge runs, same verdicts. | `scripts/flake-hunt.ts`, checks and cells. | #8943. |
58 | 10 | **Forgejo Actions jobs that are not releases.** `flake-hunt.yml`, `outside-gate.yml`, `sync-env.yml` on platform runners. | Scheduled jobs the forge runs, same verdicts. | `src/nightly/`: the hunt (02:17 UTC) and the outside run (03:47 UTC) in cells, filing the same issues and the quarantine pull; `sync-env` has nothing to port (`docs/DEVELOP.md`). | Compare one night from both, then delete the three workflows (#8943). |
5959 | 11 | **Previews of the forge's own repository.** `preview.yml` builds an image, signs it, pushes a chart. | A child forge in a cell (`src/spawn/placement.ts`, kind `cell`). | Apps preview from branch in a cell today. | #8944. |
6060
6161 ## Order and the gate
docs/features/nightly.md added +24
@@ -0,0 +1,24 @@
1 # Nightly jobs
2
3 The forge runs two jobs itself, at a fixed UTC slot each (`src/nightly/jobs.ts`):
4
5 - **flake-hunt, 02:17.** The gate in 20 slices, five loaded runs each, then the quarantine. Files `Flaky:` issues and proposes the quarantine as a pull on `flake-hunt/<date>`.
6 - **outside-gate, 03:47.** Every file in `EXCLUDED` once. Opens, updates and closes `Red:` issues.
7
8 Both are `scripts/flake-hunt.ts`, run in a cell on a checkout of the default branch that reaches only the forge and the registry. Only steps that file hold `FORGE_TOKEN`.
9
10 ## Seeing it
11
12 Resilience (admins) and the terminal's Resilience dialog say, per job, when it last ran and what it filed or why it did not run. The record is `FORGE_APP_DATA/nightly/<job>.json`.
13
14 ## When it runs
15
16 One job at a time. A job queued behind another keeps its own slot. A forge that was down at a slot runs the job on waking; a restart mid-run does not run it twice, and the record says the run was stopped. Off in a preview, without `FORGE_TOKEN`, and where cells do not work.
17
18 ## Trying it
19
20 With a token and cells, a job runs when its slot comes. To run one now from a test or a console: `startNightly(app, { forgeUrl, token })?.run("outside-gate")`.
21
22 ## What the Forgejo workflows were
23
24 `flake-hunt.yml` and `outside-gate.yml` stay on the mirror until the forge's own run has filed one real result. Both dedupe on the open `Flaky:` and `Red:` issues, so they cannot double-file; the pull that deletes them carries one night's output from each side. `sync-env.yml` sealed the repository's Actions secrets into the platform's `forge-user-env` Secret, which fed the pod's environment (the crew's Claude credential, `FORGE_TOKEN`). The forge already holds its own secrets, so there is nothing to port: the Secret goes with the pod (`docs/PLATFORM.md`, rows 2 and 3).
scripts/gate.ts +2 −2
@@ -14,9 +14,9 @@
1414 * check is green and whose description carries ten green runs in random
1515 * order under load (`bun scripts/flake-hunt.ts admit <file>`). A flaky file
1616 * leaves it only through `bun scripts/flake-hunt.ts quarantine`, under the
17 * issue the nightly hunt filed (`.forgejo/workflows/flake-hunt.yml`).
17 * issue the nightly hunt filed (`src/nightly/`).
1818 *
19 * The excluded files are not unseen: `.forgejo/workflows/outside-gate.yml`
19 * The excluded files are not unseen: the forge's nightly job (`src/nightly/`)
2020 * runs `outsideFiles` nightly and files a `Red:` issue for each that fails (#812).
2121 */
2222
src/main.ts +6
@@ -48,6 +48,7 @@ import { stopQueue } from "./checks/queue.ts";
4848 import { startLooks } from "./looks/service.ts";
4949 import { startMirrorOut } from "./origins/mirror-out.ts";
5050 import { startProbes } from "./contract/probes.ts";
51 import { startNightly } from "./nightly/service.ts";
5152 import { startSnapshots } from "./snapshot/service.ts";
5253 import { resolveDestination } from "./snapshot/destination.ts";
5354 import { resolveMirror } from "./snapshot/mirror.ts";
@@ -445,6 +446,10 @@ const looks = appsOn ? startLooks(app, { localUrl: servers.url }) : undefined;
445446 // Is it so: the forge knocks on every door of an app with a contract after
446447 // each production deploy and once a night, and compares with the table.
447448 const probes = apps === undefined ? undefined : startProbes(app, apps);
449 // The nightly jobs that were Forgejo workflows: the flake hunt and the run
450 // of what the gate excludes, filing the same `Flaky:` and `Red:` issues and
451 // the quarantine pull (docs/DEVELOP.md).
452 const nightly = appsOn ? startNightly(app, { forgeUrl, token: process.env["FORGE_TOKEN"] }) : undefined;
448453 // The snapshot: everything that cannot be rebuilt, nightly, to what an
449454 // operator sealed on the Backups page, else the platform's (`S3_*`) or a
450455 // directory (`FORGE_SNAPSHOT_DIR`). A forge given none says so on that page.
@@ -842,6 +847,7 @@ function drain(signal: NodeJS.Signals): Promise<unknown> {
842847 mirrorOut(),
843848 looks?.stop(),
844849 probes?.stop(),
850 nightly?.stop(),
845851 snapshots.stop(),
846852 sampler.stop(),
847853 memorySampler.stop(),
src/nightly/jobs.ts added +75
@@ -0,0 +1,75 @@
1 /**
2 * The nightly jobs and what each runs: the two that used to be Forgejo
junipercrew outdated

Two jobs are listed here, not three; fix the comment.
3 * workflows (`flake-hunt.yml`, `outside-gate.yml`), now the forge's own.
4 * The third, `sync-env.yml`, has nothing to run (docs/features/nightly.md).
5 * Each step is a shell command run in a cell on a checkout of the default
6 * branch; `reports` says it talks to the forge's API as the hunt's author,
7 * so it is the only kind handed the token. Plain data.
8 */
9
10 import type { Slot } from "./slots.ts";
11
12 export interface Step {
13 readonly label: string;
14 readonly command: string;
15 readonly timeoutMs: number;
16 readonly reports: boolean;
17 }
18
19 export interface Job {
20 readonly name: "outside-gate" | "flake-hunt";
21 readonly slot: Slot;
22 readonly steps: readonly Step[];
23 /** After the steps: the file whose quarantined edit is proposed as a pull. */
24 readonly proposes?: { readonly path: string; readonly branchPrefix: string };
25 }
26
27 /** The gate is cut into this many slices, each hunted on its own. */
28 export const HUNT_SLICES = 20;
29 const SLICE_MS = 30 * 60_000;
30
31 const install: Step = {
32 label: "install",
33 command: "bun install --frozen-lockfile",
34 timeoutMs: 10 * 60_000,
35 reports: false,
36 };
37
38 /** Exit 1 is "found a flake and filed it": the quarantine's work, not a broken hunt. */
39 const huntSlice = (n: number): Step => ({
40 label: `hunt slice ${n}/${HUNT_SLICES}`,
41 command: `bun scripts/flake-hunt.ts hunt --shard ${n}/${HUNT_SLICES} --file-issues; code=$?; test "$code" -le 1`,
42 timeoutMs: SLICE_MS,
43 reports: true,
44 });
45
46 export const JOBS: readonly Job[] = [
47 {
48 name: "flake-hunt",
49 slot: { hour: 2, minute: 17 },
50 steps: [
51 install,
52 ...Array.from({ length: HUNT_SLICES }, (_, i) => huntSlice(i + 1)),
53 {
54 label: "quarantine",
55 command: "bun scripts/flake-hunt.ts quarantine",
56 timeoutMs: 10 * 60_000,
57 reports: true,
58 },
59 ],
60 proposes: { path: "scripts/gate.ts", branchPrefix: "flake-hunt" },
61 },
62 {
63 name: "outside-gate",
64 slot: { hour: 3, minute: 47 },
65 steps: [
66 install,
67 {
68 label: "run the excluded files",
69 command: "bun scripts/flake-hunt.ts outside --file-issues",
70 timeoutMs: SLICE_MS,
71 reports: true,
72 },
73 ],
74 },
75 ];
src/nightly/propose.ts added +47
@@ -0,0 +1,47 @@
1 /**
2 * The quarantine, as a pull: when the job moved files out of the gate, the
3 * edited file goes up as one commit on a dated branch and a pull asks for it,
4 * opened as the repository's owner like the crew's own proposals. Nobody
5 * hand-edits the list; the pull body says which command did.
6 */
7
8 import { readFileSync } from "node:fs";
9 import { join } from "node:path";
10 import type { App } from "../app.ts";
11 import { proposeEdit } from "../actions/propose_edit.ts";
12 import { readFileAt } from "../apps/checkout.ts";
13 import type { Repo } from "../core/repos.ts";
14 import { getUserById } from "../core/users.ts";
15 import { repoDir } from "../git/repos.ts";
16 import { createLog } from "../kernel/log.ts";
17
18 const log = createLog("nightly");
19
20 export async function proposeQuarantine(
21 app: App,
22 repo: Repo,
23 ran: { readonly dir: string; readonly oid: string },
24 path: string,
25 branchPrefix: string,
26 now: number,
27 ): Promise<string | undefined> {
28 const was = await readFileAt(repoDir(app.reposRoot, repo.ownerName, repo.name), ran.oid, path);
29 const edited = readFileSync(join(ran.dir, path), "utf8");
30 if (was === undefined || was === edited) return undefined;
31 const owner = getUserById(app.db, repo.ownerId);
32 if (owner === undefined || owner.kind !== "user") return undefined;
33 const branch = `${branchPrefix}/${new Date(now).toISOString().slice(0, 10)}`;
34 const proposed = await proposeEdit(app, owner, repo, {
35 base: repo.defaultBranch,
36 branch,
37 files: [{ path, content: edited }],
38 message: "Quarantine the files the flake hunt found flaky",
39 title: `Quarantine flaky tests (${branch})`,
40 body: "Moved by `bun scripts/flake-hunt.ts quarantine`, not by hand: each file leaves the gate under the open `Flaky:` issue that carries its failure, seed and run.",
41 });
42 if (!proposed.ok) {
43 log.warn("could not propose the quarantine", { branch, step: proposed.error.step, reason: proposed.error.reason });
44 return undefined;
45 }
46 return branch;
47 }
src/nightly/run.ts added +83
@@ -0,0 +1,83 @@
1 /**
2 * One nightly job, once: the default branch exported to a directory of its
3 * own, each step run in a cell (`apps/contain.ts`) that reaches the forge
4 * and the registry and nothing else, the step's output kept in the log.
5 * The token goes only to steps that report, as FORGE_TOKEN, which the hunt
6 * script already drops from the environment of the tests it runs.
7 */
8
9 import { mkdirSync } from "node:fs";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import type { App } from "../app.ts";
13 import { exportTree } from "../apps/checkout.ts";
14 import { type CellSpec, cellsAvailable, resolveLimits, STEP_HOSTS } from "../apps/contain.ts";
15 import { childEnv, runStep } from "../apps/process.ts";
16 import type { Repo } from "../core/repos.ts";
17 import { repoDir } from "../git/repos.ts";
18 import { refTip } from "../git/read.ts";
19 import { createLog } from "../kernel/log.ts";
20 import type { Job } from "./jobs.ts";
21
22 const log = createLog("nightly");
23
24 export interface Ran {
25 readonly dir: string;
26 readonly oid: string;
27 /** The labels of the steps that failed. */
28 readonly failed: readonly string[];
29 }
30
31 export async function runJob(
32 app: App,
33 repo: Repo,
34 job: Job,
35 options: { readonly forgeUrl: string; readonly token: string; readonly signal: AbortSignal },
36 ): Promise<Ran | { readonly error: string }> {
37 const bare = repoDir(app.reposRoot, repo.ownerName, repo.name);
38 const oid = await refTip(bare, `refs/heads/${repo.defaultBranch}`);
39 if (oid === undefined) return { error: `no ${repo.defaultBranch} to check out` };
40 const dir = join(app.config.dataDir, "nightly", `${job.name}-checkout`);
41 mkdirSync(join(app.config.dataDir, "nightly"), { recursive: true });
42 const exported = await exportTree(bare, oid, dir);
43 if (!exported.ok) return { error: exported.error };
44
45 const cells = await cellsAvailable();
46 // Repository code, run unattended with a token in its hands: never without a cell.
47 if (!cells.ok) return { error: `no cell to run it in: ${cells.reason}` };
48 const cell: CellSpec = {
49 dir,
50 writable: [],
51 home: `${dir}.home`,
52 hide: [app.config.dataDir, process.env["HOME"] ?? "", tmpdir()],
53 egress: [...STEP_HOSTS],
54 reach: [options.forgeUrl],
55 limits: resolveLimits("step", undefined),
56 label: `nightly:${job.name}`,
57 };
58 const failed: string[] = [];
59 for (const step of job.steps) {
60 if (options.signal.aborted) break;
61 const env = childEnv({
62 FORGE_API: `${options.forgeUrl}/api/v1`,
63 FORGE_REPO: `${repo.ownerName}/${repo.name}`,
64 FORGE_RUN_URL: `${options.forgeUrl}/${repo.ownerName}/${repo.name}`,
65 FORGE_CELL_CPUS: String(cell.limits.cpus),
66 ...(step.reports ? { FORGE_TOKEN: options.token } : {}),
67 });
68 const ran = await runStep(step.label, step.command, {
69 cwd: dir,
70 env,
71 log: (line) => log.info(line, { job: job.name }),
72 signal: options.signal,
73 timeoutMs: step.timeoutMs,
74 cell,
75 });
76 if (ran.ok) continue;
77 log.warn("step failed", { job: job.name, step: step.label, error: ran.error });
78 failed.push(step.label);
79 // Nothing runs without its dependencies.
80 if (step.label === "install") break;
81 }
82 return { dir, oid, failed };
83 }
src/nightly/says.ts added +21
@@ -0,0 +1,21 @@
1 /**
2 * What the nightly jobs did, in words: the lines the Resilience page and the
3 * terminal's Resilience dialog both show, so a person sees that a night ran
4 * and what it filed without reading a log. Plain data in, plain text out.
5 */
6
7 import { JOBS } from "./jobs.ts";
8 import { type Last, readLast } from "./stamps.ts";
9
10 const when = (ms: number): string => `${new Date(ms).toISOString().slice(0, 16).replace("T", " ")} UTC`;
11
12 export function nightlyLine(name: string, slot: { hour: number; minute: number }, last: Last | undefined): string {
13 const at = `${String(slot.hour).padStart(2, "0")}:${String(slot.minute).padStart(2, "0")} UTC`;
14 if (last === undefined) return `${name}: has not run yet; its slot is ${at} each night.`;
15 if (last.finishedAt === undefined) return `${name}: began ${when(last.startedAt)} and has not finished (or was cut short by a restart).`;
16 return `${name}: ${when(last.startedAt)} to ${when(last.finishedAt)}, ${last.says}`;
17 }
18
19 export function nightlyLines(dataDir: string): string[] {
20 return JOBS.map((job) => nightlyLine(job.name, job.slot, readLast(dataDir, job.name)));
21 }
src/nightly/service.ts added +132
@@ -0,0 +1,132 @@
1 /**
2 * The nightly jobs as a service: each waits for its slot (`slots.ts`), runs
3 * once (`run.ts`) and files what the Forgejo workflows filed before: the
4 * `Flaky:` and `Red:` issues, the quarantine pull. One job at a time, so a
5 * hunt never runs beside the outside run; a job that waits behind another
6 * still keeps its own slot, so it is not owed again when it ends. On unless
7 * the forge lacks what it needs (a token for the hunt's author, the cells to
8 * run repository code in) or is a preview, whose copy of the data would
9 * file twice. What each job last did is kept (`stamps.ts`) and shown
10 * (`says.ts`).
11 */
12
13 import type { App } from "../app.ts";
14 import { crewOffInPreview } from "../crew/preview.ts";
15 import { getRepo } from "../core/repos.ts";
16 import { createLog } from "../kernel/log.ts";
17 import { ownRepoPath } from "../kernel/own_repo.ts";
18 import { JOBS, type Job } from "./jobs.ts";
19 import { proposeQuarantine } from "./propose.ts";
20 import { type Ran, runJob } from "./run.ts";
21 import { lastSlot, owed, untilNext } from "./slots.ts";
22 import { readLast, writeLast } from "./stamps.ts";
23
24 const log = createLog("nightly");
25
26 /** What a job does when it runs: the seams a test replaces. */
27 export interface Doing {
28 readonly run: (job: Job, signal: AbortSignal) => Promise<Ran | { readonly error: string }>;
29 /** The branch of the pull the job opened, if it opened one. */
30 readonly propose: (job: Job, ran: Ran, now: number) => Promise<string | undefined>;
31 }
32
33 export interface NightlyHandle {
34 /** Run one job now, as its slot would. */
35 readonly run: (name: Job["name"]) => Promise<void>;
36 readonly stop: () => Promise<void>;
37 }
38
39 /** The real thing: the forge's own repository, a cell, a pull as its owner. */
40 function doing(app: App, options: { readonly forgeUrl: string; readonly token: string }): Doing {
41 const repoOf = () => {
42 const [owner = "", name = ""] = ownRepoPath().split("/");
43 return getRepo(app.db, owner, name);
junipercrew outdated

The stamp is written when the queued run starts, not at its slot, so a job queued behind a long hunt records a late time. Record the slot, or document it. Also no test drives this path.
44 };
45 return {
46 run: async (job, signal) => {
47 const repo = repoOf();
48 if (repo === undefined) return { error: `${ownRepoPath()} is not on this forge` };
49 return runJob(app, repo, job, { forgeUrl: options.forgeUrl, token: options.token, signal });
50 },
51 propose: async (job, ran, now) => {
52 const repo = repoOf();
53 if (repo === undefined || job.proposes === undefined) return undefined;
54 return proposeQuarantine(app, repo, ran, job.proposes.path, job.proposes.branchPrefix, now);
55 },
56 };
57 }
58
59 const says = (ran: Ran, branch: string | undefined): string =>
60 (ran.failed.length === 0 ? "every step finished" : `failed: ${ran.failed.join(", ")}`) +
61 (branch === undefined ? "." : `; quarantine pull on ${branch}.`);
62
63 export function startNightly(
64 app: App,
65 options: {
66 readonly forgeUrl: string;
67 readonly token: string | undefined;
68 readonly doing?: Doing;
69 readonly now?: () => number;
70 },
71 ): NightlyHandle | undefined {
72 const token = options.token;
73 if (token === undefined || token === "") {
74 log.info("nightly jobs are off: no FORGE_TOKEN");
75 return undefined;
76 }
77 if (crewOffInPreview()) return undefined;
78 const now = options.now ?? Date.now;
79 const does = options.doing ?? doing(app, { forgeUrl: options.forgeUrl, token });
80 const abort = new AbortController();
81 const timers = new Set<ReturnType<typeof setTimeout>>();
82 let going: Promise<unknown> = Promise.resolve();
83
84 const run = (name: Job["name"]): Promise<void> => {
85 const job = JOBS.find((j) => j.name === name)!;
86 // The slot is read when the run is asked for, not when it gets its turn.
87 const slot = lastSlot(job.slot, now());
88 const turn = going.then(async () => {
89 if (abort.signal.aborted) return;
90 const startedAt = now();
91 writeLast(app.config.dataDir, job.name, { slot, startedAt, says: "began" });
92 const end = (said: string): void =>
93 writeLast(app.config.dataDir, job.name, { slot, startedAt, finishedAt: now(), says: said });
94 log.info("started", { job: job.name });
95 const ran = await does.run(job, abort.signal);
96 if ("error" in ran) {
97 log.warn("not run", { job: job.name, error: ran.error });
98 return end(`not run: ${ran.error}.`);
99 }
100 const branch =
101 job.proposes !== undefined && !ran.failed.includes("quarantine")
102 ? await does.propose(job, ran, now())
103 : undefined;
104 log.info("finished", { job: job.name, failed: ran.failed.join(", "), pull: branch ?? "" });
105 end(abort.signal.aborted ? `stopped by a restart; ${says(ran, branch)}` : says(ran, branch));
106 });
107 going = turn.catch((error) => log.error("failed", { job: job.name, error: String(error) }));
108 return going as Promise<void>;
109 };
110
111 const wait = (job: Job): void => {
112 const timer = setTimeout(() => {
113 timers.delete(timer);
114 void run(job.name).finally(() => !abort.signal.aborted && wait(job));
115 // A second past the slot: a timer that fires early would read yesterday's.
116 }, untilNext(job.slot, now()) + 1_000);
117 timers.add(timer);
118 };
119 for (const job of JOBS) {
120 // A forge that was down at the slot runs the job on waking.
121 if (owed(job.slot, readLast(app.config.dataDir, job.name)?.slot, now())) void run(job.name);
122 wait(job);
123 }
124 return {
125 run,
126 stop: async () => {
127 abort.abort();
128 for (const t of timers) clearTimeout(t);
129 await going;
130 },
131 };
132 }
src/nightly/slots.ts added +31
@@ -0,0 +1,31 @@
1 /**
2 * When a nightly job is owed: each has a slot at a fixed UTC time, and a
3 * slot is kept by one run at or after it. A forge that was down at the slot
4 * runs the job on waking; one that restarts after the run began does not
5 * run it twice. A forge that has never run a job owes none until its first
6 * slot. Plain data in, plain data out.
7 */
8
9 const DAY = 86_400_000;
10
11 export interface Slot {
12 readonly hour: number;
13 readonly minute: number;
14 }
15
16 /** The latest slot at or before `now`, in ms. */
17 export function lastSlot(slot: Slot, now: number): number {
18 const d = new Date(now);
19 d.setUTCHours(slot.hour, slot.minute, 0, 0);
20 return d.getTime() > now ? d.getTime() - DAY : d.getTime();
21 }
22
23 /** Milliseconds from `now` to the next slot; never zero. */
24 export function untilNext(slot: Slot, now: number): number {
25 return lastSlot(slot, now) + DAY - now;
26 }
27
28 /** Whether a run is owed: the newest began (`last`, undefined when never) before the latest slot. */
29 export function owed(slot: Slot, last: number | undefined, now: number): boolean {
30 return last !== undefined && Number.isFinite(last) && last < lastSlot(slot, now);
31 }
src/nightly/stamps.ts added +38
@@ -0,0 +1,38 @@
1 /**
2 * What each nightly job last did, one small file apiece under
3 * `FORGE_APP_DATA/nightly`: the slot it kept, when it began and ended, and
4 * what it said. The record is written as a run begins, so a restart in the
5 * middle of a long hunt does not start it again; a hunt cut short is next
6 * night's, and says so. Nothing here cannot be rebuilt.
7 */
8
9 import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
10 import { join } from "node:path";
11
12 export interface Last {
13 /** The slot this run kept, not the time it got to start: a job queued behind another still keeps its own. */
14 readonly slot: number;
15 readonly startedAt: number;
16 readonly finishedAt?: number;
17 /** One plain sentence: what it filed, what failed, why it did not run. */
18 readonly says: string;
19 }
20
21 const fileOf = (dataDir: string, name: string): string => join(dataDir, "nightly", `${name}.json`);
22
23 /** The record of `name`, or undefined when it never ran or the file is not one. */
24 export function readLast(dataDir: string, name: string): Last | undefined {
25 try {
26 const v = JSON.parse(readFileSync(fileOf(dataDir, name), "utf8")) as Partial<Record<keyof Last, unknown>>;
27 const whole = (x: unknown): x is number => typeof x === "number" && Number.isFinite(x);
28 if (!whole(v.slot) || !whole(v.startedAt) || typeof v.says !== "string") return undefined;
29 return { slot: v.slot, startedAt: v.startedAt, says: v.says, ...(whole(v.finishedAt) ? { finishedAt: v.finishedAt } : {}) };
30 } catch {
31 return undefined;
32 }
33 }
34
35 export function writeLast(dataDir: string, name: string, last: Last): void {
36 mkdirSync(join(dataDir, "nightly"), { recursive: true });
37 writeFileSync(fileOf(dataDir, name), JSON.stringify(last));
38 }
src/tui/screens/resilience.ts +3
@@ -31,6 +31,7 @@ import {
3131 type DestinationInput,
3232 } from "../../actions/destination.ts";
3333 import { forgetMirror, saveMirror } from "../../actions/mirror.ts";
34 import { nightlyLines } from "../../nightly/says.ts";
3435 import { backupsLine } from "../../snapshot/backups_line.ts";
3536 import { backupVerdict } from "../../snapshot/bucket.ts";
3637 import { lastDrill } from "../../snapshot/drills.ts";
@@ -254,6 +255,8 @@ export function resilienceScreen(f: Session, ctx: Ctx): Screen {
254255 if (asking !== undefined) out.push(QUESTION[asking], "");
255256 else if (said !== "") out.push(...wrap(said, INNER), "");
256257 out.push(...wrap(backupsLine(f.app).says, INNER), ...verdictLines(f), "");
258 for (const line of nightlyLines(f.app.config.dataDir)) out.push(...wrap(line, INNER));
259 out.push("");
257260 out.push(...wrap(destinationLine(f), INNER), ...wrap(mirrorLine(f), INNER));
258261 if (note !== "") out.push(...wrap(note, INNER));
259262 if (mayChange()) out.push("d sets where snapshots go, x forgets it.", "m sets the mirror, f forgets it.", "t tests where snapshots go.");
src/web/pages/resilience.ts +9
@@ -11,6 +11,7 @@
1111 import { readResilience } from "../../checks/resilience.ts";
1212 import type { Standing } from "../../core/threats.ts";
1313 import type { Ctx } from "../../http/context.ts";
14 import { nightlyLines } from "../../nightly/says.ts";
1415 import { backupsLine } from "../../snapshot/backups_line.ts";
1516 import { notePhase, takeSessionTime, timePhase } from "../../core/route_timing.ts";
1617 import { notFoundPage, signInFirst } from "../chrome.ts";
@@ -30,6 +31,13 @@ export function backupsLineHtml(ctx: Ctx): Html {
3031 return html`<p id="backups-line">${line.says} <a href="/settings/backups">Backups</a> lists them.</p>`;
3132 }
3233
34 /** What the nightly jobs last did (`nightly/says.ts`): that a night ran, and what it filed. */
35 export function nightlyHtml(ctx: Ctx): Html {
36 return html`<ul id="nightly" class="muted">
37 ${nightlyLines(ctx.app.config.dataDir).map((line) => html`<li>${line}</li>`)}
38 </ul>`;
39 }
40
3341 // GET /resilience: a forge admin's, since live red lines name what to attack.
3442 export function resiliencePage(ctx: Ctx): Response {
3543 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
@@ -46,6 +54,7 @@ export function resiliencePage(ctx: Ctx): Response {
4654 </p>
4755 </div>
4856 ${backupsLineHtml(ctx)}
57 ${nightlyHtml(ctx)}
4958 <p id="tally">
5059 ${String(count("green"))} green · ${String(count("red"))} red ·
5160 ${String(count("untested"))} untested · ${String(count("no test yet"))} with no test yet
test/nightly.test.ts added +260
@@ -0,0 +1,260 @@
1 import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test";
2 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
3 import { createApp, type App } from "../src/app.ts";
4 import { listPulls } from "../src/core/pulls.ts";
5 import type { Repo } from "../src/core/repos.ts";
6 import { refTip } from "../src/git/read.ts";
7 import { proposeQuarantine } from "../src/nightly/propose.ts";
8 import { startNightly } from "../src/nightly/service.ts";
9 import { seedAdmin } from "./helpers/admin.ts";
10 import { quietLogs } from "./helpers/globals.ts";
11 import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
12 import { tmpdir } from "node:os";
13 import { join } from "node:path";
14 import { HUNT_SLICES, JOBS } from "../src/nightly/jobs.ts";
15 import { nightlyLine } from "../src/nightly/says.ts";
16 import { lastSlot, owed, untilNext } from "../src/nightly/slots.ts";
17 import { readLast, writeLast } from "../src/nightly/stamps.ts";
18
19 quietLogs();
20
21 const slot = { hour: 2, minute: 17 };
22 const at = (iso: string): number => Date.parse(iso);
23
24 describe("nightly slots", () => {
25 test("the latest slot is today's once it has passed, yesterday's before", () => {
26 expect(lastSlot(slot, at("2026-11-02T03:00:00Z"))).toBe(at("2026-11-02T02:17:00Z"));
27 expect(lastSlot(slot, at("2026-11-02T01:00:00Z"))).toBe(at("2026-11-01T02:17:00Z"));
28 });
29
30 test("the wait to the next slot is positive and under a day", () => {
31 expect(untilNext(slot, at("2026-11-02T02:17:00Z"))).toBe(86_400_000);
32 expect(untilNext(slot, at("2026-11-02T02:16:00Z"))).toBe(60_000);
33 });
34
35 test("a run is owed when the last began before the latest slot, never for a forge that has none", () => {
36 const now = at("2026-11-02T03:00:00Z");
37 expect(owed(slot, at("2026-11-01T02:20:00Z"), now)).toBe(true);
38 expect(owed(slot, at("2026-11-02T02:20:00Z"), now)).toBe(false);
39 expect(owed(slot, undefined, now)).toBe(false);
40 });
41 });
42
43 describe("nightly records", () => {
44 test("a record reads back, and a missing or broken one reads as none", () => {
45 const dir = mkdtempSync(join(tmpdir(), "nightly-"));
46 try {
47 expect(readLast(dir, "outside-gate")).toBeUndefined();
48 const last = { slot: at("2026-11-02T03:47:00Z"), startedAt: at("2026-11-02T09:00:00Z"), says: "ok." };
49 writeLast(dir, "outside-gate", last);
50 expect(readLast(dir, "outside-gate")).toEqual(last);
51 writeFileSync(join(dir, "nightly", "outside-gate.json"), "{nope");
52 expect(readLast(dir, "outside-gate")).toBeUndefined();
53 } finally {
54 rmSync(dir, { recursive: true, force: true });
55 }
56 });
57
58 test("a line says never run, not finished, and what a run said", () => {
59 const s = { hour: 3, minute: 47 };
60 expect(nightlyLine("x", s, undefined)).toContain("has not run yet; its slot is 03:47 UTC");
61 expect(nightlyLine("x", s, { slot: 1, startedAt: at("2026-11-02T03:47:00Z"), says: "began" })).toContain("has not finished");
62 const done = { slot: 1, startedAt: at("2026-11-02T03:47:00Z"), finishedAt: at("2026-11-02T04:10:00Z"), says: "every step finished." };
63 expect(nightlyLine("x", s, done)).toBe("x: 2026-11-02 03:47 UTC to 2026-11-02 04:10 UTC, every step finished.");
64 });
65 });
66
67 describe("nightly jobs", () => {
68 test("the hunt covers every slice once and ends in the quarantine; the outside run is one step", () => {
69 const hunt = JOBS.find((j) => j.name === "flake-hunt")!;
70 const slices = hunt.steps.filter((s) => s.label.startsWith("hunt slice"));
71 expect(slices).toHaveLength(HUNT_SLICES);
72 expect(new Set(slices.map((s) => s.command)).size).toBe(HUNT_SLICES);
73 expect(hunt.steps.at(-1)?.label).toBe("quarantine");
74 expect(JOBS.find((j) => j.name === "outside-gate")!.steps.map((s) => s.label)).toEqual([
75 "install",
76 "run the excluded files",
77 ]);
78 });
79
80 test("only the steps that file hold the token, and every script a step runs exists", () => {
81 for (const job of JOBS)
82 for (const step of job.steps) {
83 expect(step.reports).toBe(step.command.includes("flake-hunt.ts"));
84 const script = /scripts\/[\w-]+\.ts/.exec(step.command)?.[0];
85 if (script !== undefined) expect(existsSync(join(import.meta.dir, "..", script))).toBe(true);
86 }
87 });
88 });
89
90 describe("the nightly service", () => {
91 let dir: string;
92 let app: App;
93 const calls: string[] = [];
94 let release: (() => void) | undefined;
95 const instant = {
96 run: async (job: { name: string }) => {
97 calls.push(job.name);
98 return { dir: "x", oid: "y", failed: [] as string[] };
99 },
100 propose: async () => undefined,
101 };
102 const SLOT_AFTER = at("2026-11-02T03:00:00Z");
103
104 beforeEach(() => {
105 dir = mkdtempSync(join(tmpdir(), "nightly-svc-"));
106 app = createApp({ dataDir: dir, port: 0 });
107 calls.length = 0;
108 delete process.env["FORGE_APP_ENV"];
109 delete process.env["FORGE_CREW_IN_PREVIEW"];
110 });
111 afterEach(() => {
112 app.close();
113 rmSync(dir, { recursive: true, force: true });
114 delete process.env["FORGE_APP_ENV"];
115 });
116
117 test("it is off without a token", () => {
118 expect(startNightly(app, { forgeUrl: "http://f", token: undefined, doing: instant })).toBeUndefined();
119 expect(startNightly(app, { forgeUrl: "http://f", token: "", doing: instant })).toBeUndefined();
120 });
121
122 test("it is off in a preview", () => {
123 process.env["FORGE_APP_ENV"] = "preview";
124 expect(startNightly(app, { forgeUrl: "http://f", token: "t", doing: instant })).toBeUndefined();
125 });
126
127 test("a job whose slot passed while the forge was down runs once on waking, and keeps that slot", async () => {
128 const yesterday = { slot: at("2026-11-01T02:17:00Z"), startedAt: at("2026-11-01T02:17:00Z"), finishedAt: at("2026-11-01T05:00:00Z"), says: "ok." };
129 writeLast(dir, "flake-hunt", yesterday);
130 writeLast(dir, "outside-gate", { ...yesterday, slot: at("2026-11-01T03:47:00Z") });
131 const handle = startNightly(app, { forgeUrl: "http://f", token: "t", doing: instant, now: () => SLOT_AFTER })!;
132 while (readLast(dir, "flake-hunt")?.slot !== at("2026-11-02T02:17:00Z") || calls.length === 0) await Bun.sleep(5);
133 await handle.stop();
134 // Only the hunt's slot (02:17) has passed since; the outside run's (03:47) has not.
135 expect(calls).toEqual(["flake-hunt"]);
136 expect(readLast(dir, "flake-hunt")?.slot).toBe(at("2026-11-02T02:17:00Z"));
137 expect(readLast(dir, "flake-hunt")?.finishedAt).toBeDefined();
138 });
139
140 test("a forge that never ran a job owes none, and a job already kept is not run again", async () => {
141 const fresh = startNightly(app, { forgeUrl: "http://f", token: "t", doing: instant, now: () => SLOT_AFTER })!;
142 await fresh.stop();
143 expect(calls).toEqual([]);
144 writeLast(dir, "flake-hunt", { slot: at("2026-11-02T02:17:00Z"), startedAt: at("2026-11-02T02:17:30Z"), says: "began" });
145 const again = startNightly(app, { forgeUrl: "http://f", token: "t", doing: instant, now: () => SLOT_AFTER })!;
146 await again.stop();
147 expect(calls).toEqual([]);
148 });
149
150 test("a job queued behind another records its own slot, not the time it got to start", async () => {
151 const handle = startNightly(app, { forgeUrl: "http://f", token: "t", doing: instant, now: () => at("2026-11-02T03:50:00Z") })!;
152 await handle.run("flake-hunt");
153 await handle.run("outside-gate");
154 await handle.stop();
155 expect(readLast(dir, "outside-gate")?.slot).toBe(at("2026-11-02T03:47:00Z"));
156 });
157
158 test("stop aborts a run in flight at once and the record says it was stopped", async () => {
159 let seen: AbortSignal | undefined;
160 const slow = {
161 run: (_job: unknown, signal: AbortSignal) => {
162 seen = signal;
163 return new Promise<{ dir: string; oid: string; failed: string[] }>((resolve) => {
164 signal.addEventListener("abort", () => resolve({ dir: "x", oid: "y", failed: ["hunt slice 3/20"] }));
165 release = () => resolve({ dir: "x", oid: "y", failed: [] });
166 });
167 },
168 propose: async () => undefined,
169 };
170 const handle = startNightly(app, { forgeUrl: "http://f", token: "t", doing: slow, now: () => SLOT_AFTER })!;
171 const running = handle.run("outside-gate");
172 await Bun.sleep(10);
173 expect(seen?.aborted).toBe(false);
174 await handle.stop();
175 await running;
176 expect(seen?.aborted).toBe(true);
177 release?.();
178 expect(readLast(dir, "outside-gate")?.says).toContain("stopped by a restart");
179 });
180
181 test("a run that could not start says why, and files no pull", async () => {
182 let proposed = 0;
183 const handle = startNightly(app, {
184 forgeUrl: "http://f",
185 token: "t",
186 now: () => SLOT_AFTER,
187 doing: { run: async () => ({ error: "no cell to run it in" }), propose: async () => void proposed++ as undefined },
188 })!;
189 await handle.run("flake-hunt");
190 await handle.stop();
191 expect(readLast(dir, "flake-hunt")?.says).toBe("not run: no cell to run it in.");
192 expect(proposed).toBe(0);
193 });
194
195 test("the hunt proposes after its steps, unless the quarantine itself failed; the outside run never does", async () => {
196 let proposed = 0;
197 const doing = (failed: string[]) => ({
198 run: async () => ({ dir: "x", oid: "y", failed }),
199 propose: async () => {
200 proposed++;
201 return "flake-hunt/2026-11-02";
202 },
203 });
204 const green = startNightly(app, { forgeUrl: "http://f", token: "t", now: () => SLOT_AFTER, doing: doing([]) })!;
205 await green.run("flake-hunt");
206 await green.run("outside-gate");
207 await green.stop();
208 expect(proposed).toBe(1);
209 expect(readLast(dir, "flake-hunt")?.says).toContain("quarantine pull on flake-hunt/2026-11-02");
210 const red = startNightly(app, { forgeUrl: "http://f", token: "t", now: () => SLOT_AFTER, doing: doing(["quarantine"]) })!;
211 await red.run("flake-hunt");
212 await red.stop();
213 expect(proposed).toBe(1);
214 });
215 });
216
217 describe("the quarantine pull", () => {
218 let dir: string;
219 let work: string;
220 let app: App;
221 let repo: Repo;
222 let oid: string;
223 const ORIGINAL = "export const EXCLUDED = {};\n";
224
225 beforeAll(async () => {
226 dir = mkdtempSync(join(tmpdir(), "nightly-pull-"));
227 work = mkdtempSync(join(tmpdir(), "nightly-work-"));
228 app = createApp({ dataDir: dir, port: 0 });
229 const owner = await seedAdmin(app, { username: "alice", email: "[email protected]", password: "a-long-passphrase" });
230 if (!owner.ok) throw new Error("no owner");
231 const made = await createRepo(app, owner.value, { name: "forge", visibility: "public" });
232 if (!made.ok) throw new Error("no repo");
233 repo = made.value;
234 const seeded = await commitToBranch(app, owner.value, repo, { message: "seed", files: [{ path: "scripts/gate.ts", content: ORIGINAL }] });
235 if (!seeded.ok) throw new Error("no seed");
236 oid = (await refTip(join(app.reposRoot, "alice", "forge.git"), `refs/heads/${repo.defaultBranch}`))!;
237 mkdirSync(join(work, "scripts"), { recursive: true });
238 });
239 afterAll(() => {
240 app.close();
241 rmSync(dir, { recursive: true, force: true });
242 rmSync(work, { recursive: true, force: true });
243 });
244
245 test("a file the job left unchanged proposes nothing", async () => {
246 writeFileSync(join(work, "scripts/gate.ts"), ORIGINAL);
247 const branch = await proposeQuarantine(app, repo, { dir: work, oid }, "scripts/gate.ts", "flake-hunt", at("2026-11-02T03:00:00Z"));
248 expect(branch).toBeUndefined();
249 expect(listPulls(app.db, repo.id, {})).toHaveLength(0);
250 });
251
252 test("a file the job edited is proposed as one pull on a dated branch", async () => {
253 writeFileSync(join(work, "scripts/gate.ts"), "export const EXCLUDED = { a: 1 };\n");
254 const branch = await proposeQuarantine(app, repo, { dir: work, oid }, "scripts/gate.ts", "flake-hunt", at("2026-11-02T03:00:00Z"));
255 expect(branch).toBe("flake-hunt/2026-11-02");
256 const pulls = listPulls(app.db, repo.id, {});
257 expect(pulls).toHaveLength(1);
258 expect(pulls[0]?.title).toBe("Quarantine flaky tests (flake-hunt/2026-11-02)");
259 });
260 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs