trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns16g rEventsFlows

One night of the forge's flake hunt and outside gate is compared with Forgejo's, and the result is written down #9056

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/9045-one-night-of-the-forge-s-flake-hunt-and → main 1 commit fork

  • Merged. scout/9045-one-night-of-the-forge-s-flake-hunt-and was merged into main as d2c6a75.

Review

Sign in to review this change.
4 files changed +254 −1
docs/DEVELOP.md +16
@@ -243,6 +243,22 @@ it once; passing again, it closes the issue. Same `FORGE_TOKEN` as the hunt.
243243 The Forgejo workflows these replace, and `sync-env`, are covered in
244244 `docs/features/nightly.md`.
245245
246 **Can the Forgejo jobs be switched off? Not yet.** They go (#9020) once one
247 night of each side has been set side by side and agrees:
248 `bun scripts/flake-compare.ts --forge hunt.log --forge outside.log --forgejo
249 slice-1.log … --forgejo outside.log` reads what each side's run printed (the
250 forge's `/resilience/nightly/<job>`, with the job's Resilience line pasted in
251 so its `quarantine pull on flake-hunt/<date>` counts; each Forgejo step log)
252 and prints what each filed — `Red:` issues opened, updated or closed, `Flaky:`
253 issues, broken and quarantined files, the quarantine pull — then every
254 difference, naming the file; it exits 1 when there is one
255 (`test/flake-compare.test.ts`). One side silent where the other filed is the
256 dedupe, not a difference. A flaky file only one side found can be the seed;
257 a file red outside the gate on one side and green on the other is the cell
258 differing from the `isolated` runner, and blocks the switch until explained.
259 No such night has been compared yet: the logs are admin-only on both sides,
260 so an admin runs it and posts the output on #9045.
261
246262 Bun's own `--randomize --seed=N` drives every hunt and admission run. Not
247263 used: `--rerun-each` (one seed for every rerun, so ten reruns explore one
248264 order) and `--parallel` (it changes the load the gate is judged under).
docs/features/nightly.md +1 −1
@@ -34,4 +34,4 @@ For one date, set side by side:
3434 - **Forgejo:** the `flake-hunt` and `outside-gate` Actions runs on the mirror (each slice's log, the quarantine job's log).
3535 - **Forge:** `/resilience/nightly/flake-hunt` and `/resilience/nightly/outside-gate`, and the line above each: what it filed.
3636
37 They agree when the outside run lists the same files green and red, both open, update and close the same `Red:` issues (or one finds the other's already open), and the hunt's flaky files, which vary with the seed, are each filed once. A file red on one side and green on the other is the finding: the cell (no network but the forge and the registry, its CPU and memory caps) differs from the `isolated` runner.
37 They agree when the outside run lists the same files green and red, both open, update and close the same `Red:` issues (or one finds the other's already open), and the hunt's flaky files, which vary with the seed, are each filed once. `bun scripts/flake-compare.ts --forge <log>… --forgejo <log>…` does this from the logs and names each difference (`docs/DEVELOP.md`). A file red on one side and green on the other is the finding: the cell (no network but the forge and the registry, its CPU and memory caps) differs from the `isolated` runner.
scripts/flake-compare.ts added +163
@@ -0,0 +1,163 @@
1 /**
2 * One night of the flake hunt and the outside gate, the forge's run beside
3 * the Forgejo job's (#9045). Both sides run `scripts/flake-hunt.ts`, so
4 * both logs carry the same lines; this reads what each filed out of them
5 * and names where they differ.
6 *
7 * bun scripts/flake-compare.ts --forge a.log [--forge b.log]
8 * --forgejo c.log [--forgejo d.log]
9 *
10 * Each log is a run's whole output (the forge's `/resilience/nightly/<job>`,
11 * a Forgejo Actions step log); give each side its hunt and its outside run,
12 * in any order. A Forgejo slice log may be passed once per slice. Prints
13 * both sides and the differences; exits 1 when there are any.
14 *
15 * They agree when the outside run sees the same files red and green, the
16 * `Red:` issues move the same way (one side finding the other's already
17 * open is agreement: it says nothing), and the same files are flaky,
18 * broken and quarantined. Plain text in, plain data out.
19 */
20
21 import { readFileSync } from "node:fs";
22
23 export type RedAction = { kind: "filed" | "updated" | "closed"; issue: number };
24
25 export type Night = {
26 /** Outside the gate: file → red or green. */
27 outside: Map<string, "red" | "green">;
28 /** `Red:` issues the outside run opened, updated or closed: file → action. */
29 red: Map<string, RedAction>;
30 /** Hunt: file → the `Flaky:` issue filed or found open (null: not filed). */
31 flaky: Map<string, number | null>;
32 broken: Set<string>;
33 /** Quarantine: file → the issue it was moved under. */
34 quarantined: Map<string, number>;
35 /** The quarantine pull's branch, when the log names one. */
36 pull: string | null;
37 };
38
39 const TEST = String.raw`(test/[\w./-]+\.test\.ts)`;
40 const OUTSIDE = new RegExp(String.raw`(?:^|\s)(red |green): ${TEST}\s*$`);
41 const RED = new RegExp(String.raw`(?:^|\s)(filed|updated|closed) #(\d+) for ${TEST}`);
42 const FLAKY = new RegExp(String.raw`(?:^|\s)FLAKY ${TEST}:`);
43 const FILED = /(?:^|\s)(?:filed|already open as) #(\d+)\s*$/;
44 const BROKEN = new RegExp(String.raw`red in every run \(broken, not flaky\): ${TEST}`);
45 const QUARANTINED = new RegExp(String.raw`(?:^|\s)quarantined ${TEST} \(#(\d+)\)`);
46 const PULL = /flake-hunt\/\d{4}-\d{2}-\d{2}/;
47
48 /** What one side's logs say it did. */
49 export function readNight(logs: string[]): Night {
50 const night: Night = {
51 outside: new Map(),
52 red: new Map(),
53 flaky: new Map(),
54 broken: new Set(),
55 quarantined: new Map(),
56 pull: null,
57 };
58 for (const log of logs) {
59 let lastFlaky: string | null = null;
60 for (const line of log.split("\n")) {
61 let m: RegExpExecArray | null;
62 if ((m = OUTSIDE.exec(line))) night.outside.set(m[2]!, m[1] === "green" ? "green" : "red");
63 else if ((m = RED.exec(line)))
64 night.red.set(m[3]!, { kind: m[1] as RedAction["kind"], issue: Number(m[2]) });
65 else if ((m = FLAKY.exec(line))) {
66 lastFlaky = m[1]!;
67 if (!night.flaky.has(lastFlaky)) night.flaky.set(lastFlaky, null);
68 } else if ((m = FILED.exec(line)) && lastFlaky) night.flaky.set(lastFlaky, Number(m[1]));
69 else if ((m = BROKEN.exec(line))) night.broken.add(m[1]!);
70 else if ((m = QUARANTINED.exec(line))) night.quarantined.set(m[1]!, Number(m[2]));
71 if ((m = PULL.exec(line))) night.pull = m[0];
72 }
73 }
74 return night;
75 }
76
77 const issue = (n: number | null) => (n === null ? "not filed" : `#${n}`);
78
79 /** Every way the two nights disagree, one line each; empty when they agree. */
80 export function compareNights(forge: Night, forgejo: Night): string[] {
81 const out: string[] = [];
82 const sides = [
83 ["forge", forge],
84 ["forgejo", forgejo],
85 ] as const;
86 const union = <T>(a: Iterable<T>, b: Iterable<T>) => [...new Set([...a, ...b])].sort();
87
88 for (const file of union(forge.outside.keys(), forgejo.outside.keys())) {
89 const a = forge.outside.get(file);
90 const b = forgejo.outside.get(file);
91 if (a === b) continue;
92 out.push(`outside ${file}: ${a ?? "not run"} on forge, ${b ?? "not run"} on forgejo`);
93 }
94 for (const file of union(forge.red.keys(), forgejo.red.keys())) {
95 const a = forge.red.get(file);
96 const b = forgejo.red.get(file);
97 // One side acting and the other silent is the dedupe at work, unless the
98 // outside runs already disagree on the file (named above).
99 if (!a || !b || a.kind === b.kind) continue;
100 out.push(`Red: ${file}: forge ${a.kind} #${a.issue}, forgejo ${b.kind} #${b.issue}`);
101 }
102 for (const file of union(forge.flaky.keys(), forgejo.flaky.keys())) {
103 for (const [name, night] of sides) {
104 const other = night === forge ? forgejo : forge;
105 if (night.flaky.has(file) && !other.flaky.has(file))
106 out.push(`Flaky: ${file}: only ${name} found it (${issue(night.flaky.get(file)!)})`);
107 }
108 }
109 for (const file of union(forge.broken, forgejo.broken)) {
110 if (forge.broken.has(file) !== forgejo.broken.has(file))
111 out.push(`broken ${file}: only ${forge.broken.has(file) ? "forge" : "forgejo"} found it`);
112 }
113 for (const file of union(forge.quarantined.keys(), forgejo.quarantined.keys())) {
114 if (forge.quarantined.has(file) !== forgejo.quarantined.has(file))
115 out.push(`quarantined ${file}: only ${forge.quarantined.has(file) ? "forge" : "forgejo"} moved it`);
116 }
117 if ((forge.pull === null) !== (forgejo.pull === null))
118 out.push(`quarantine pull: only ${forge.pull ? "forge" : "forgejo"} proposed one`);
119 return out;
120 }
121
122 /** One side, as lines a person reads. */
123 export function describeNight(name: string, night: Night): string[] {
124 const reds = [...night.outside].filter(([, s]) => s === "red").map(([f]) => f);
125 const lines = [`${name}: outside ${night.outside.size} files, ${reds.length} red`];
126 for (const [file, a] of night.red) lines.push(` Red: ${a.kind} #${a.issue} for ${file}`);
127 for (const [file, n] of night.flaky) lines.push(` Flaky: ${file} (${issue(n)})`);
128 for (const file of night.broken) lines.push(` broken: ${file}`);
129 for (const [file, n] of night.quarantined) lines.push(` quarantined ${file} (#${n})`);
130 lines.push(` quarantine pull: ${night.pull ?? "none"}`);
131 return lines;
132 }
133
134 /** The whole report and whether the two sides agree. */
135 export function report(forge: Night, forgejo: Night): { text: string; agree: boolean } {
136 const diffs = compareNights(forge, forgejo);
137 const lines = [...describeNight("forge", forge), ...describeNight("forgejo", forgejo), ""];
138 if (diffs.length === 0) lines.push("no differences");
139 else lines.push(`${diffs.length} differences:`, ...diffs.map((d) => ` ${d}`));
140 return { text: lines.join("\n"), agree: diffs.length === 0 };
141 }
142
143 if (import.meta.main) {
144 const args = process.argv.slice(2);
145 const paths: Record<"--forge" | "--forgejo", string[]> = { "--forge": [], "--forgejo": [] };
146 for (let i = 0; i < args.length; i += 2) {
147 const flag = args[i];
148 const path = args[i + 1];
149 if ((flag !== "--forge" && flag !== "--forgejo") || !path) {
150 console.error("usage: bun scripts/flake-compare.ts --forge <log>... --forgejo <log>...");
151 process.exit(2);
152 }
153 paths[flag].push(path);
154 }
155 if (paths["--forge"].length === 0 || paths["--forgejo"].length === 0) {
156 console.error("give at least one log for each side: --forge <log> --forgejo <log>");
157 process.exit(2);
158 }
159 const read = (list: string[]) => readNight(list.map((p) => readFileSync(p, "utf8")));
160 const { text, agree } = report(read(paths["--forge"]), read(paths["--forgejo"]));
161 console.log(text);
162 process.exit(agree ? 0 : 1);
163 }
test/flake-compare.test.ts added +74
@@ -0,0 +1,74 @@
1 /**
2 * `scripts/flake-compare.ts` (#9045): one night of the forge's flake hunt
3 * and outside gate beside the Forgejo job's, read from their logs.
4 */
5
6 import { describe, expect, test } from "bun:test";
7 import { compareNights, readNight, report } from "../scripts/flake-compare.ts";
8
9 const outside = `$ bun scripts/flake-hunt.ts outside --file-issues
10 green: test/a.test.ts
11 red : test/b.test.ts
12 filed #12 for test/b.test.ts
13 green: test/c.test.ts
14 closed #7 for test/c.test.ts
15 3 files outside the gate, 1 red`;
16
17 const hunt = `hunt run 1/5: seed 42, 3 files
18 red: test/d.test.ts (seed 42)
19 red in every run (broken, not flaky): test/e.test.ts
20 FLAKY test/d.test.ts: red 1, green 4; first red seed 42
21 filed #30
22 quarantined test/d.test.ts (#30)
23 pushed flake-hunt/2026-11-03`;
24
25 /** A Forgejo step log: each line behind a timestamp. */
26 const stamped = (log: string) =>
27 log
28 .split("\n")
29 .map((line) => `2026-11-03T02:17:00.0000000Z ${line}`)
30 .join("\n");
31
32 describe("flake-compare", () => {
33 test("reads what a night filed", () => {
34 const night = readNight([outside, hunt]);
35 expect([...night.outside]).toEqual([
36 ["test/a.test.ts", "green"],
37 ["test/b.test.ts", "red"],
38 ["test/c.test.ts", "green"],
39 ]);
40 expect(night.red.get("test/b.test.ts")).toEqual({ kind: "filed", issue: 12 });
41 expect(night.red.get("test/c.test.ts")).toEqual({ kind: "closed", issue: 7 });
42 expect([...night.flaky]).toEqual([["test/d.test.ts", 30]]);
43 expect([...night.broken]).toEqual(["test/e.test.ts"]);
44 expect([...night.quarantined]).toEqual([["test/d.test.ts", 30]]);
45 expect(night.pull).toBe("flake-hunt/2026-11-03");
46 });
47
48 test("the same night on both sides has no differences, timestamps or not", () => {
49 const { text, agree } = report(readNight([outside, hunt]), readNight([stamped(hunt), stamped(outside)]));
50 expect(agree).toBe(true);
51 expect(text).toContain("no differences");
52 });
53
54 test("one side finding the other's Red: issue already open is agreement", () => {
55 const silent = outside.replace(" filed #12 for test/b.test.ts\n", "");
56 expect(compareNights(readNight([outside]), readNight([silent]))).toEqual([]);
57 });
58
59 test("a test filed by only one side is named", () => {
60 const forgeHunt = hunt.replace(" filed #30", " filed #30\nFLAKY test/f.test.ts: red 2, green 3; first red seed 9\n filed #31");
61 const diffs = compareNights(readNight([outside, forgeHunt]), readNight([outside, hunt]));
62 expect(diffs).toEqual(["Flaky: test/f.test.ts: only forge found it (#31)"]);
63 });
64
65 test("a file red on one side and green on the other is named, as is a missing pull", () => {
66 const forgejoOutside = outside.replace("red : test/b.test.ts\n filed #12 for test/b.test.ts", "green: test/b.test.ts");
67 const forgejoHunt = hunt.replace("quarantined test/d.test.ts (#30)\npushed flake-hunt/2026-11-03", "nothing to quarantine");
68 const { text, agree } = report(readNight([outside, hunt]), readNight([forgejoOutside, forgejoHunt]));
69 expect(agree).toBe(false);
70 expect(text).toContain("outside test/b.test.ts: red on forge, green on forgejo");
71 expect(text).toContain("quarantined test/d.test.ts: only forge moved it");
72 expect(text).toContain("quarantine pull: only forge proposed one");
73 });
74 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs