trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns15g rEventsFlows

A pull is not red for main's red, and the fast tier finds every tree scan, so main does not land red through one #11669

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/10304-a-red-main-fails-every-pull-s-check-tell → main 10 commits fork 1 thread · 1 unresolved

  • Merged. mason/10304-a-red-main-fails-every-pull-s-check-tell was merged into main as 1063e47.

Review

Sign in to review this change.
38 files changed +824 −26
docs/DEVELOP.md +28 −6
@@ -105,12 +105,34 @@ map has never seen fall back to the import-hop guess, and a change to
105105 everything. With no map yet the line says `by imports (no impact map)`.
106106
107107 A test that reads the whole tree — every test file, every migration, the
108 README — is reached by no diff, yet any file can break it. Those are
109 `RULE_TESTS` in `src/checks/tiers.ts`, and every fast tier runs them
110 whatever else the diff holds (#7138) — a diff of only the timing or impact table breaks no rule and runs none; a new rule of that kind goes on the list. Write
111 the rule with `breaches()` from `test/helpers/rule.ts`: one red test per
112 breach, named `path:line: …`, so the check's summary names the rule and
113 the line.
108 README — is reached by no diff, yet any file can break it, so every fast
109 tier runs it whatever else the diff holds (#7138) — a diff of only the
110 timing or impact table breaks no rule and runs none. Such a test says
111 `@tree-scan` in its header comment (`src/checks/scans.ts`, #10304); that
112 mark is the contract. The plain cases are found without it: a test whose
113 source, or a `scripts/` module it imports, lists the repository (a `Glob`,
114 `git ls-files`, a `readdir` handed the root or a name bound to it). A
115 walk's parameter is not inferred, so a test that walks through one
116 declares itself. One the reading mistakes for a scan (it lists only its
117 own temp directories) says `@not-a-tree-scan` and why. `RULE_TESTS` in
118 `src/checks/tiers.ts` is the floor the forge reads without a tree; a new
sentinelcrew outdated

Rewrap: this line runs past the paragraph's width.
119 scan needs no line there. Together the scans may cost half the fast
120 tier's budget (`test/tree-scans.test.ts`).
121
122 Write the rule with `breaches()` from `test/helpers/rule.ts`: one red test
123 per breach, named `path:line: …`, so the check's summary names the rule
124 and the line.
125
126 A fast tier red only on test files the pull did not change is asked of
127 main: each is run once more on the base's tree, and when it is red there
128 too with the same tests the log says `… is red on main at <base> too: not
129 counted against this pull` and the check is green
130 (`src/checks/base_red.ts`). The pull's check summary keeps it — `check
131 passed in 40s; red on main at abcdef0, not this pull's: test/a.test.ts` —
132 so the masked red is seen on the pull's page after the log is gone. Main's
133 red itself is recorded by main's own checks: the red-main hold
134 (`src/checks/red_main.ts`) and its vitals card, and the revert the forge
135 proposes (`src/checks/revert_red.ts`).
114136
115137 Before you push, ask what the fast tier would run for your change:
116138 `files=$(bun scripts/fast.ts) && bun test $files`. `bun scripts/fast.ts
docs/OPERATING.md +1 −1
@@ -218,7 +218,7 @@ A check, test or app cell asks for memory in `forge.json` (`check.limits.memory`
218218
219219 ## Check tiers
220220
221 `forge.json`'s `check` may declare `fast` beside `run` (the full gate; `full` names it explicitly). `bun scripts/test-shards.ts --tier fast --diff <base>..<head>` runs the tests that touch the diff (changed, named for a changed source file, or importing one within two hops), every repo-wide rule test (`RULE_TESTS`, #7138), every test the times table records under 1 s, and any named by `--learned a,b`. `src/checks/tiers.ts` decides; it adds no dependency.
221 `forge.json`'s `check` may declare `fast` beside `run` (the full gate; `full` names it explicitly). `bun scripts/test-shards.ts --tier fast --diff <base>..<head>` runs the tests that touch the diff (changed, named for a changed source file, or importing one within two hops), every repo-wide rule test (`RULE_TESTS`, #7138) and every test that reads the tree (`src/checks/scans.ts`, #10304), every test the times table records under 1 s, and any named by `--learned a,b`; a red only on files the pull did not change is run again on the base, and red there too it is main's and not counted (`src/checks/base_red.ts`). `src/checks/tiers.ts` decides; it adds no dependency.
222222
223223 To see what the fast tier would run for your own change before you push: `files=$(bun scripts/fast.ts) && bun test $files`. `bun scripts/fast.ts [--base origin/main] [--why | --null | --json]` lists, one per line on stdout and nothing else there, the test files `fastSet` picks for the files changed between the base's merge-base and the working tree (committed, uncommitted and untracked), by the same times table and `scripts/test-impact.json` as the check; `--why` adds a tab and the reason per file, `--null` ends each path with NUL (`| xargs -0 -r bun test`), `--json` prints `{full, files:[{file, why}]}`. Exit 0 lists files; exit 3 means the change reaches no test (stdout empty); exit 2 means the diff is one the map cannot bound (`package.json`, `forge.json`, a gate change that is logic…; stderr says `full tier: … changed`) or the arguments or git failed. The `&&` guard matters: `bun test $(…)` alone ignores the exit code and, with nothing listed, runs the whole suite. It reads the diff with `git` called by argument list, never a shell; `scripts/tier-input.ts` is the reading it shares with `test-shards.ts`.
224224
scripts/fast.ts +2 −1
@@ -20,7 +20,7 @@
2020 import { readFileSync } from "node:fs";
2121 import { GATE_FILE, gateListAdds, handedImpact, needsFull } from "../src/checks/impact.ts";
2222 import { fastSet, duration } from "../src/checks/tiers.ts";
23 import { importsReader, readImpact } from "./tier-input.ts";
23 import { importsReader, readImpact, scansIn } from "./tier-input.ts";
2424 import { CHECK_CACHE } from "../src/checks/cache.ts";
2525 import { gateFiles } from "./gate.ts";
2626 import { overlayTimes, parseTimes } from "./shards.ts";
@@ -174,6 +174,7 @@ function main(): number {
174174 ...(gateAdds === undefined ? {} : { gateAdds }),
175175 tests,
176176 imports,
177 scans: scansIn(tests, imports),
177178 times: loadTimes(),
178179 });
179180 const out = report(plan, needsFull(changed, gateAdds), args.format);
scripts/gate.ts +3
@@ -18,6 +18,9 @@
1818 *
1919 * The excluded files are not unseen: the forge's nightly job (`src/nightly/`)
2020 * runs `outsideFiles` nightly and files a `Red:` issue for each that fails (#812).
21 *
22 * @not-a-tree-scan: most of its importers judge its rules on fixtures; a
23 * test that lists the gate's files through it says `@tree-scan` itself.
2124 */
2225
2326 import { existsSync, readdirSync, readFileSync } from "node:fs";
scripts/mcp-tools.ts +2
@@ -11,6 +11,8 @@
1111 *
1212 * `test/mcp-tools.test.ts` runs the check, so a pull that adds a tool file
1313 * without running this goes red with the command to run.
14 *
15 * @tree-scan: lists src/http/mcp/tools/ through a parameter's default.
1416 */
1517 import { readdirSync, readFileSync, writeFileSync } from "node:fs";
1618 import { join } from "node:path";
scripts/openapi-routes.ts +2
@@ -8,6 +8,8 @@
88 *
99 * `test/openapi-routes.test.ts` runs the same comparison, so a pull that adds
1010 * a route file without running this goes red with the command to run.
11 *
12 * @tree-scan: lists src/contract/openapi/routes/ through a parameter's default.
1113 */
1214 import { readdirSync, readFileSync, writeFileSync } from "node:fs";
1315 import { join } from "node:path";
scripts/shards.ts +19 −3
@@ -559,14 +559,30 @@ const FAILED = /^\(fail\) /;
559559 * would not load) yields none.
560560 */
561561 export function failingFiles(output: string): string[] {
562 const found: string[] = [];
562 return [...failingTests(output).keys()];
563 }
564
565 /** `(fail) a > b [1.20ms]` → `a > b`. */
566 const testName = (line: string): string => line.replace(FAILED, "").replace(/\s+\[[\d.]+m?s\]\s*$/, "").trim();
567
568 /**
569 * The failing tests of one `bun test` report, by file, each file's names in
570 * report order: what {@link failingFiles} names, with which of its tests
571 * failed. The closing summary repeats them under no file and is not read.
572 */
573 export function failingTests(output: string): Map<string, string[]> {
574 const found = new Map<string, string[]>();
563575 let current: string | undefined;
564576 for (const line of output.split("\n")) {
565577 const header = HEADER.exec(line);
566578 if (header) current = header[1]!;
567579 else if (isSummaryLine(line)) current = undefined;
568 else if (FAILED.test(line) && current !== undefined && !found.includes(current))
569 found.push(current);
580 else if (FAILED.test(line) && current !== undefined) {
581 const names = found.get(current) ?? [];
582 const name = testName(line);
583 if (!names.includes(name)) names.push(name);
584 found.set(current, names);
585 }
570586 }
571587 return found;
572588 }
scripts/test-shards.ts +95 −5
@@ -53,8 +53,8 @@
5353 */
5454
5555 import { availableParallelism, setPriority, tmpdir, totalmem } from "node:os";
56 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
57 import { join } from "node:path";
56 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs";
57 import { join, resolve } from "node:path";
5858 import { resolveLimits } from "../src/apps/contain.ts";
5959 import {
6060 formatTimes,
@@ -75,6 +75,7 @@ import {
7575 weigher,
7676 besideOthers,
7777 failingFiles,
78 failingTests,
7879 onlyTestFailures,
7980 crashedOutside,
8081 quoteCrash,
@@ -84,13 +85,14 @@ import {
8485 import { gateFiles, gateRefusal, presentTests } from "./gate.ts";
8586 import { cgroupMemory, summaryLines } from "./gate-summary.ts";
8687 import { GATE_FILE, executedFiles, formatImpact, gateListAdds, handedImpact, mergeImpact, parseImpact } from "../src/checks/impact.ts";
87 import { importsReader, readImpact } from "./tier-input.ts";
88 import { importsReader, readImpact, scansIn } from "./tier-input.ts";
8889 import { shardHome, spaceLine } from "../src/checks/disk_full.ts";
8990 import { fastSet, duration } from "../src/checks/tiers.ts";
9091 import { CHECK_CACHE } from "../src/checks/cache.ts";
9192 import { parseCheck } from "../src/apps/manifest.ts";
9293 import { FLAKY_ALONE, onlyFiles } from "../src/checks/verify.ts";
9394 import { crashFlakeLine } from "../src/checks/flake.ts";
95 import { askBase, baseVerdict, type OnBase } from "../src/checks/base_red.ts";
9496
9597 const TIMES = new URL("./test-times.json", import.meta.url).pathname;
9698 // The forge hands the map over in the checkout (`src/checks/impact-store.ts`):
@@ -270,6 +272,8 @@ if (gate || tier === "fast") {
270272 }
271273 const GATE = gateFiles();
272274 let files = [...(gate || tier === "fast" ? GATE : []), ...split.files];
275 // The fast tier's diff, for asking its base about a red (#10304).
276 let changed: string[] = [];
273277 if (tier === "fast") {
274278 const diff = Bun.spawnSync(
275279 ["git", "diff", "--name-only", "--no-renames", diffArg.value!],
@@ -279,7 +283,7 @@ if (tier === "fast") {
279283 console.error(`test-shards: git diff failed: ${diff.stderr.toString()}`);
280284 process.exit(2);
281285 }
282 const changed = diff.stdout.toString().split("\n").filter((f) => f !== "");
286 changed = diff.stdout.toString().split("\n").filter((f) => f !== "");
283287 let gateAdds: string[] | undefined;
284288 if (changed.includes(GATE_FILE)) {
285289 const gateDiff = Bun.spawnSync(
@@ -296,6 +300,7 @@ if (tier === "fast") {
296300 ...(gateAdds === undefined ? {} : { gateAdds }),
297301 tests: files,
298302 imports,
303 scans: scansIn(files, imports),
299304 times: loadTimes(),
300305 learned: (learnedArg.value ?? "").split(",").filter((f) => f !== ""),
301306 });
@@ -759,4 +764,89 @@ if (timed)
759764 times: results.map((result) => ({ file: result.files[0]!, seconds: result.seconds })),
760765 }).join("\n")}`,
761766 );
762 process.exit(ending.code);
767
768 /**
769 * The red test files of this run, each with the tests in it that failed;
770 * `undefined` when a red names no file or no test (a crash, a load error).
771 */
772 function redTests(results: readonly Timed[]): Map<string, string[]> | undefined {
773 const red = new Map<string, string[]>();
774 const plain = (file: string): string => file.replace(/^\.\//, "");
775 for (const result of results) {
776 if (result.code === 0) continue;
777 const named = onlyTestFailures(result.output) ? failingTests(result.output) : new Map<string, string[]>();
778 if (named.size === 0) return undefined;
779 for (const [file, tests] of named) red.set(plain(file), [...(red.get(plain(file)) ?? []), ...tests]);
780 }
781 return red;
782 }
783
784 /**
785 * Each file run alone on the tree of `base`, in a worktree on the shard
786 * disk that shares this checkout's node_modules; removed however it ends.
787 */
788 async function runOnBase(asked: readonly string[], base: string): Promise<Map<string, OnBase>> {
789 const did = new Map<string, OnBase>();
790 const home = shardHome(process.env, tmpdir());
791 const tree = join(home, "base");
792 const git = (args: string[]): boolean =>
793 Bun.spawnSync(["git", ...args], { stdin: "ignore", stdout: "ignore", stderr: "pipe" }).exitCode === 0;
794 try {
795 if (!git(["worktree", "add", "--detach", tree, base])) {
796 console.log(`test-shards: could not check out ${base} to ask it about the red`);
797 return did;
798 }
799 if (existsSync("node_modules")) symlinkSync(resolve("node_modules"), join(tree, "node_modules"));
800 const env: Record<string, string | undefined> = { ...process.env };
801 const scratch = { tmp: join(home, "tmp"), data: join(home, "data") };
802 env["TMPDIR"] = scratch.tmp;
803 env["FORGE_APP_DATA"] = scratch.data;
804 delete env["NODE_OPTIONS"];
805 for (const file of asked) {
806 if (!existsSync(join(tree, file))) {
807 did.set(file, { did: "absent" });
808 continue;
809 }
810 mkdirSync(scratch.tmp, { recursive: true });
811 mkdirSync(scratch.data, { recursive: true });
812 const child = Bun.spawn(["bun", "test", ...flags, `./${file}`], {
813 cwd: tree,
814 env,
815 stdin: "ignore",
816 stdout: "pipe",
817 stderr: "pipe",
818 });
819 children.add(child);
820 const output: string[] = [];
821 await Promise.all([drain(child.stdout, output), drain(child.stderr, output)]);
822 const report = output.join("");
823 if ((await child.exited) === 0) did.set(file, { did: "green" });
824 else {
825 // A crash or load error names no test: nothing to compare the pull's red with.
826 const named = onlyTestFailures(report) ? failingTests(report) : new Map<string, string[]>();
827 did.set(file, { did: "red", failing: [...named.values()].flat() });
828 }
829 children.delete(child);
830 rmSync(scratch.data, { recursive: true, force: true });
831 }
832 } finally {
833 git(["worktree", "remove", "--force", tree]);
834 rmSync(home, { recursive: true, force: true });
835 }
836 return did;
837 }
838
839 // A fast tier red only on tests main's own tree fails too is main's red,
840 // not this pull's (#10304): said, and not counted against it.
841 let code = ending.code;
842 if (code !== 0 && tier === "fast" && range !== undefined && only === undefined) {
843 const red = redTests(results);
844 const asked = askBase({ red, changed });
845 if (red !== undefined && asked !== undefined) {
846 console.log(`\ntest-shards: asking main at ${range[0]} about ${asked.join(", ")}`);
847 const verdict = baseVerdict(red, await runOnBase(asked, range[0]!), range[0]!);
848 for (const line of verdict.lines) console.log(`test-shards: ${line}`);
849 if (verdict.cleared) code = 0;
850 }
851 }
852 process.exit(code);
scripts/tier-input.ts +19
@@ -7,6 +7,7 @@
77 import { readFileSync } from "node:fs";
88 import { importsOf } from "../src/checks/tiers.ts";
99 import { parseImpact } from "../src/checks/impact.ts";
10 import { treeScans } from "../src/checks/scans.ts";
1011
1112 /** A file's relative imports as candidate paths (an extensionless one also as `.ts`); `[]` when unreadable. Memoised. */
1213 export function importsReader(): (file: string) => string[] {
@@ -45,3 +46,21 @@ export function readImpact(
4546 }
4647 return undefined;
4748 }
49
50 /** The gate's tests that read the tree (`src/checks/scans.ts`), read from the working tree. */
51 export function scansIn(
52 tests: readonly string[],
53 imports: (file: string) => readonly string[],
54 ): Map<string, string> {
55 return treeScans(
56 tests,
57 (file) => {
58 try {
59 return readFileSync(file, "utf8");
60 } catch {
61 return undefined;
62 }
63 },
64 imports,
65 );
66 }
src/checks/base_red.ts added +123
@@ -0,0 +1,123 @@
1 /**
2 * A pull's red that is main's (#10304). A test that landed red on main
3 * fails every pull's check after it, and each pull's builder reworks a
4 * change that was fine. So when a pull's fast tier is red, each failing
5 * test file is run once more on the tree of the pull's base. Red there with
6 * the same tests, it is main's: the log says so, and it does not count
7 * against the pull.
8 *
9 * What stays the pull's, always: a file the pull changed (its red may be
10 * the change's), a file the base does not have, one green on the base, a
11 * test that fails on the pull and passes on the base (a second break in a
12 * file main already has red), and a red no file or test is named for (a
13 * crash, a type error). Over
14 * {@link BASE_RERUN_CAP} red files the base is not asked: that many is
15 * this pull's, or main's red is wide enough for the red-main hold
16 * (`red_main.ts`) to say.
17 *
18 * Plain data in, plain data out: `scripts/test-shards.ts` runs the files
19 * on the base and hands over what each did.
20 */
21
22 /** The most red files the base is asked about. */
23 export const BASE_RERUN_CAP = 5;
24
25 /**
26 * What one red file did on the base's tree: red, with the tests that
27 * failed (none named when it crashed or would not load), green, or not
28 * there to run.
29 */
30 export type OnBase =
31 | { readonly did: "red"; readonly failing: readonly string[] }
32 | { readonly did: "green" }
33 | { readonly did: "absent" };
34
35 /** Each red file of the pull's run, with the tests in it that failed. */
36 export type RedTests = ReadonlyMap<string, readonly string[]>;
37
38 export interface BaseInput {
39 /** The pull's red files and their failing tests; `undefined` when some red names no file or no test. */
40 readonly red: RedTests | undefined;
41 /** Files the pull changed. */
42 readonly changed: readonly string[];
43 }
44
45 /** The red files worth asking the base about, or `undefined` when none could clear the run. */
46 export function askBase(input: BaseInput): string[] | undefined {
47 const red = input.red;
48 if (red === undefined || red.size === 0 || red.size > BASE_RERUN_CAP) return undefined;
49 const files = [...red.keys()];
50 if (files.some((file) => input.changed.includes(file) || red.get(file)!.length === 0)) return undefined;
51 return files;
52 }
53
54 export interface BaseVerdict {
55 /** Every red was main's: the pull's run is green. */
56 readonly cleared: boolean;
57 /** What the log says, one line each. */
58 readonly lines: string[];
59 }
60
61 /** `abcdef0123…` → `abcdef0`. */
62 const short = (rev: string): string => (/^[0-9a-f]{40,64}$/.test(rev) ? rev.slice(0, 7) : rev);
63
64 /** At most three names, quoted, and how many more. */
65 const some = (names: readonly string[]): string =>
66 names.slice(0, 3).map((name) => JSON.stringify(name)).join(", ") + (names.length > 3 ? ` and ${names.length - 3} more` : "");
67
68 /**
69 * Whether the pull's red is main's, from what each red file did on `base`.
70 * A file is main's only when every test of it that fails on the pull fails
71 * on the base too: red on the base is not the same red, and a second break
72 * in a file main already has red is the pull's.
73 */
74 export function baseVerdict(red: RedTests, onBase: ReadonlyMap<string, OnBase>, base: string): BaseVerdict {
75 const at = `main at ${short(base)}`;
76 const mains: string[] = [];
77 const lines: string[] = [];
78 const theirs: string[] = [];
79 for (const [file, failing] of red) {
80 const there = onBase.get(file);
81 if (there === undefined) theirs.push(`${file} did not run on ${at}: this pull's red`);
82 else if (there.did === "green") theirs.push(`${file} is green on ${at}: this pull's red`);
83 else if (there.did === "absent") theirs.push(`${file} is not on ${at}: this pull's red`);
84 else {
85 const own = failing.filter((name) => !there.failing.includes(name));
86 if (failing.length > 0 && own.length === 0) {
87 mains.push(file);
88 lines.push(`${file} is red on ${at} too, the same tests: not counted against this pull`);
89 } else if (there.failing.length === 0)
90 theirs.push(`${file} is red on ${at}, but names no failing test there to compare: this pull's red`);
91 else theirs.push(`${file} is red on ${at} too, but ${some(own)} pass${own.length === 1 ? "es" : ""} there: this pull's red`);
92 }
93 }
94 lines.push(...theirs);
95 const cleared = red.size > 0 && mains.length === red.size;
96 if (cleared) lines.push(`${CLEARED} ${at} too, so it is main's, not this pull's (${mains.join(", ")})`);
97 return { cleared, lines };
98 }
99
100 /** How the verdict that cleared a run opens. */
101 const CLEARED = "green: every red test is red on";
102
103 /** The cleared verdict as `scripts/test-shards.ts` prints it, its base and files captured. */
104 const CLEARED_LINE = new RegExp(`^test-shards: ${CLEARED} (main at \\S+) too, so it is main's, not this pull's \\((.+)\\)$`);
105
106 /** The longest a cleared verdict runs in a check's summary. */
107 const SUMMARY_MAX = 300;
108
109 /**
110 * What a green check's summary keeps of a red it cleared as main's, from the
111 * check's own output: `red on main at abcdef0, not this pull's: test/a.test.ts`,
112 * or `undefined` when the run cleared none. The run's summary is what the
113 * pull's page and its builder read, so a masked red stays in sight there
114 * after the log is gone; main's own red is the red-main hold's
115 * (`red_main.ts`), recorded by main's checks.
116 */
117 export function clearedIn(output: readonly string[]): string | undefined {
118 for (let i = output.length - 1; i >= 0; i--) {
119 const m = CLEARED_LINE.exec(output[i]!.trimEnd());
120 if (m !== null) return `red on ${m[1]}, not this pull's: ${m[2]}`.slice(0, SUMMARY_MAX);
121 }
122 return undefined;
123 }
src/checks/scans.ts added +168
@@ -0,0 +1,168 @@
1 /**
2 * Tree scans (#10304): tests that read the repository's tree — every file
3 * under src/, every test, the README — rather than import what they judge.
4 * No import map and no coverage ties one to a diff (a new module, a long
5 * comment line), so a scan the fast tier did not know of landed green and
6 * turned every later pull red. A hand list of them (`RULE_TESTS`) misses
7 * the next one written.
8 *
9 * So a scan declares itself with a comment line holding {@link SCAN_MARK}:
10 * that is the contract. The plain cases are found without it, by what the
11 * test reads: its own source, or a `scripts/` module it imports, lists a
12 * directory of the repository (a `Glob`, `git ls-files`, or a `readdir`
13 * handed the root or a name bound to it). Nothing subtler is guessed — a
14 * walk's parameter, a fixed file such as the README — so such a scan
15 * carries the mark; a test the
16 * reading takes for a scan that only lists its own temporary directories
17 * says so with {@link NOT_SCAN_MARK} and the reason, and is not one.
18 *
19 * Plain data in, plain data out: sources come through `read`; the scripts
20 * read the files.
21 */
22
23 import { FAST_BUDGET_SECONDS, QUICK_SECONDS } from "./tiers.ts";
24
25 /**
26 * What the tree scans may cost together: half the fast tier's budget, so
27 * a reading that comes to over-match as the tree grows goes red in
28 * `test/tree-scans.test.ts` before it slows every pull. On 2026-11-12: 67
29 * files, 1210 tests, 80 s run serially in one process.
30 */
31 export const SCAN_BUDGET_SECONDS = FAST_BUDGET_SECONDS / 2;
32
33 /**
34 * The summed seconds of `scans` by the times table; a test it has no time
35 * for weighs {@link QUICK_SECONDS}, what the scans measured on average.
36 */
37 export function scanSeconds(scans: Iterable<string>, times: ReadonlyMap<string, number>): number {
38 let sum = 0;
39 for (const test of scans) sum += times.get(test) ?? QUICK_SECONDS;
40 return sum;
41 }
42
43 /** The declaration a test makes when it scans the tree in a way {@link walksTree} cannot see. */
44 export const SCAN_MARK = "@tree-scan";
45
46 /** The declaration a test makes when it lists only what it made itself. */
47 export const NOT_SCAN_MARK = "@not-a-tree-scan";
48
49 /** Lists a directory: a glob or `git ls-files`. */
50 const GLOB = /\bnew\s+(?:Bun\.)?Glob\s*\(|["'`]ls-files["'`]/;
51 /** A `readdir` call, up to its opening parenthesis. */
52 const READDIR = /\breaddir(?:Sync)?\s*\(/g;
53 /** A path expression's head: what a `join(`, `resolve(` or template opens with. */
54 const HEAD = String.raw`^\s*(?:(?:path\.)?(?:join|resolve)\s*\(\s*)?`;
55 /** Names the repository's root: through the module's own place, or the working directory. */
56 const ROOT = /import\.meta\.(?:dir|dirname|url|path)\b|process\.cwd\s*\(/;
57 /** A path that opens with a top-level directory of the repository, or `.`/`..`, by name. */
58 const TOP = new RegExp(`${HEAD}["'\`](?:\\.{1,2}|(?:\\./)?(?:src|test|docs|scripts))(?:/|["'\`])`);
59
60 /** Whether `text` is a path into the repository, given the names already known to hold one. */
61 function intoRepo(text: string, rooted: ReadonlySet<string>): boolean {
62 if (ROOT.test(text) || TOP.test(text)) return true;
63 if (rooted.size === 0) return false;
64 const names = [...rooted].map((name) => name.replace(/\$/g, "\\$")).join("|");
65 return new RegExp(`${HEAD}(?:\\$\\{\\s*)?(?:${names})(?![\\w$])`).test(text.replace(/^\s*`/, ""));
66 }
67
68 /** The text between the parenthesis at `open` and the one that closes it. */
69 function argumentAt(source: string, open: number): string {
70 let depth = 0;
71 for (let i = open; i < source.length; i++) {
72 const c = source[i];
73 if (c === "(") depth++;
74 else if (c === ")" && --depth === 0) return source.slice(open + 1, i);
75 }
76 return source.slice(open + 1);
77 }
78
79 /**
80 * The names that hold a path into the repository: bound (`const`, `let`,
81 * `var`) to an expression that names the root or to a path built from such
82 * a name. Nothing subtler is inferred (not a walk's parameter, not a
83 * default); a scan this misses declares itself with {@link SCAN_MARK},
84 * which is the contract — this reading only spares the plain cases a line.
85 */
86 function rootedNames(source: string): Set<string> {
87 const bindings = [...source.matchAll(/\b(?:const|let|var)\s+([\w$]+)\s*(?::[^=\n]+)?=\s*([^;\n]+)/g)].map(
88 (m) => ({ name: m[1]!, value: m[2]! }),
89 );
90 const rooted = new Set<string>();
91 for (let grew = true; grew; ) {
92 grew = false;
93 for (const { name, value } of bindings)
94 if (!rooted.has(name) && intoRepo(value, rooted)) {
95 rooted.add(name);
96 grew = true;
97 }
98 }
99 return rooted;
100 }
101
102 /** Whether a `readdir` in the source is handed a path into the repository. */
103 function readsRepoDir(source: string): boolean {
104 let rooted: Set<string> | undefined;
105 for (const call of source.matchAll(READDIR)) {
106 const arg = argumentAt(source, call.index! + call[0].length - 1);
107 rooted ??= rootedNames(source);
108 if (intoRepo(arg, rooted)) return true;
109 }
110 return false;
111 }
112
113 /** Whether a comment line of the source holds `mark` (as a word: `@tree-scan` is not in `@not-a-tree-scan`). */
114 const declares = (source: string, mark: string): boolean =>
115 source
116 .split("\n")
117 .some((line) => /^\s*(?:\/\/|\/?\*)/.test(line) && new RegExp(`(?:^|[\\s*/])${mark}\\b`).test(line));
118
119 /** Whether a source declares itself a scan: {@link SCAN_MARK} on a comment line. */
120 export const declaresScan = (source: string): boolean => declares(source, SCAN_MARK);
121
122 /** Whether a source declares it is no scan: {@link NOT_SCAN_MARK} on a comment line. */
123 export const declaresNoScan = (source: string): boolean => declares(source, NOT_SCAN_MARK);
124
125 /** Whether a source lists the repository's tree. */
126 export const walksTree = (source: string): boolean => GLOB.test(source) || readsRepoDir(source);
127
128 /** Why a source is a scan, or `undefined` when it is not one. */
129 export function scanOf(source: string): "declared" | "lists the tree" | undefined {
130 if (declaresScan(source)) return "declared";
131 if (declaresNoScan(source)) return undefined;
132 return walksTree(source) ? "lists the tree" : undefined;
133 }
134
135 /**
136 * The tests that scan the tree, each with why: by its own source, or by a
137 * `scripts/` module it imports directly (a scan's logic usually lives in
138 * the script the check also runs). `read` answers `undefined` for a file it
139 * cannot read; `imports` is a file's repository-relative imports.
140 */
141 export function treeScans(
142 tests: readonly string[],
143 read: (file: string) => string | undefined,
144 imports: (file: string) => readonly string[],
145 ): Map<string, string> {
146 const scripts = new Map<string, ReturnType<typeof scanOf>>();
147 const scriptScan = (file: string): ReturnType<typeof scanOf> => {
148 if (!scripts.has(file)) {
149 const source = read(file);
150 scripts.set(file, source === undefined ? undefined : scanOf(source));
151 }
152 return scripts.get(file);
153 };
154 const found = new Map<string, string>();
155 for (const test of tests) {
156 const source = read(test);
157 if (source === undefined) continue;
158 const own = scanOf(source);
159 if (own !== undefined) {
160 found.set(test, own === "declared" ? "declares a tree scan" : "lists the tree");
161 continue;
162 }
163 if (declaresNoScan(source)) continue;
164 const via = imports(test).find((file) => file.startsWith("scripts/") && scriptScan(file) !== undefined);
165 if (via !== undefined) found.set(test, `lists the tree through ${via}`);
166 }
167 return found;
168 }
src/checks/service.ts +22 −4
@@ -61,6 +61,7 @@ import { readLearned } from "./learned.ts";
6161 import { keepCache, restoreCache } from "./cache.ts";
6262 import { MAX_FILED, clearedFiles, fileOrderDependent } from "./order_dependent.ts";
6363 import { parseBunTestFailure } from "./bun-test.ts";
64 import { clearedIn } from "./base_red.ts";
6465 import { recordCrashFlakes, passedAlone, recordFlakes, rerunCommand, rerunnable, pullOwnsRed } from "./flake.ts";
6566 import { diskRecovered, withDisk } from "./disk_red.ts";
6667 import { scratchEnv } from "./disk_full.ts";
@@ -867,7 +868,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
867868 state: outcome.ok ? "ok" : "failed",
868869 summary: outcome.ok
869870 ? outcome.proven === undefined
870 ? `check passed in ${seconds}s`
871 ? `check passed in ${seconds}s${outcome.cleared === undefined ? "" : `; ${outcome.cleared}`}`
871872 : `already proven green by ${outcome.proven}`
872873 : `check failed: ${outcome.error}`,
873874 ...(outcome.ok
@@ -904,7 +905,13 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
904905 write: (line: string) => void,
905906 signal: AbortSignal,
906907 ): Promise<
907 | { ok: true; proven?: string; passed?: readonly string[] }
908 | {
909 ok: true;
910 proven?: string;
911 passed?: readonly string[];
912 /** A red it cleared as main's (#10304): `base_red.ts`'s words, kept in the summary. */
913 cleared?: string;
914 }
908915 | { ok: false; error: string; detail?: CheckDetail | undefined; infrastructure?: boolean; passed?: readonly string[] }
909916 > {
910917 const bare = await runDir(app.db, app.reposRoot, repo, run.issueId);
@@ -983,7 +990,13 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
983990 async function runTier(
984991 x: Execution,
985992 ): Promise<
986 | { ok: true; proven?: string; passed?: readonly string[] }
993 | {
994 ok: true;
995 proven?: string;
996 passed?: readonly string[];
997 /** A red it cleared as main's (#10304): `base_red.ts`'s words, kept in the summary. */
998 cleared?: string;
999 }
9871000 | {
9881001 ok: false;
9891002 error: string;
@@ -1480,7 +1493,12 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
14801493 }
14811494 }
14821495 if (key !== undefined) recordProven(app.db, key, x.id);
1483 return wholeVerify ? { ok: true, passed: passedFiles(checkLines) } : { ok: true };
1496 const cleared = clearedIn(checkLines);
1497 return {
1498 ok: true,
1499 ...(wholeVerify ? { passed: passedFiles(checkLines) } : {}),
1500 ...(cleared === undefined ? {} : { cleared }),
1501 };
14841502 } finally {
14851503 if (existsSync(dir)) await removeRunDir(dir);
14861504 await removeRunDir(home);
src/checks/tiers.ts +11 −3
@@ -20,7 +20,10 @@ export const QUICK_SECONDS = 1;
2020 export const FAST_BUDGET_SECONDS = 5 * 60;
2121
2222 /**
23 * The repo-wide rule tests (#7138): each reads every test file, every
23 * The repo-wide rule tests (#7138), the floor that needs no tree to read
24 * (`outside_diff.ts` judges by it from a run's log alone). Every other test
25 * that reads the tree is found by what it reads (`scans.ts`, #10304), so a
26 * new scan needs no line here. Each reads every test file, every
2427 * migration or the whole README, so no diff "touches" them by coverage,
2528 * import or name, yet any file can break them. They run in every fast
2629 * tier, whatever else the diff holds (a diff of only the check's own
@@ -111,6 +114,8 @@ export interface FastInput {
111114 readonly learned?: readonly string[];
112115 /** Measured: the `src/**` files each test executed (`scripts/test-impact.json`). */
113116 readonly impact?: Impact;
117 /** The tests that read the tree, with why (`treeScans`, #10304): every diff can break one. */
118 readonly scans?: ReadonlyMap<string, string>;
114119 }
115120
116121 export interface FastPlan {
@@ -127,7 +132,8 @@ export interface FastPlan {
127132 /**
128133 * The fast tier for a diff: a changed test; a test importing a changed file
129134 * directly or through up to two hops of source; a test whose name matches a
130 * changed source path; every repo-wide rule test; whatever the pull
135 * changed source path; every repo-wide rule test and every tree scan
136 * (`scans.ts`); whatever the pull
131137 * learned; and every test known to take under a second. A test with no recorded time is not known to be quick, so it runs only when
132138 * something above picks it.
133139 */
@@ -176,8 +182,10 @@ export function fastSet(input: FastInput): FastPlan {
176182 const changed = new Set(diffed);
177183 for (const file of changed) take(file, "changed");
178184 // The check's own tables can break no rule: a diff of only those runs none (#3579).
179 if (diffed.some((f) => !f.endsWith("test-times.json") && !f.endsWith("test-impact.json")))
185 if (diffed.some((f) => !f.endsWith("test-times.json") && !f.endsWith("test-impact.json"))) {
180186 for (const test of RULE_TESTS) take(test, "repo-wide rule");
187 for (const [test, why] of input.scans ?? []) take(test, `tree scan: ${why}`);
188 }
181189 // What a guess looks for. A test the map has measured already answered
182190 // for every change the map can see, by what it executes; it is guessed
183191 // only against the changes the map cannot see, so one unreached file
src/spawn/host.ts +7 −3
@@ -233,13 +233,17 @@ function wakeWhenDue(parent: App, ledger: ChildLedger, at: number, now = Date.no
233233 const was = wakes.get(ledger.bodyId);
234234 if (was !== undefined) clearTimeout(was);
235235 const mine = wakes;
236 const timer = setTimeout(() => {
236 const due = async (): Promise<void> => {
237237 if (mine.get(ledger.bodyId) === timer) mine.delete(ledger.bodyId);
238238 if (Date.now() < at) return wakeWhenDue(parent, ledger, at);
239 void wakeDue(parent, ledger.name).catch((error: unknown) =>
239 await wakeDue(parent, ledger.name).catch((error: unknown) =>
240240 parent.log.warn("child forge not woken for its run", { name: ledger.name, error: String(error) }),
241241 );
242 }, Math.min(TIMER_MAX_MS, Math.max(0, at - now)));
242 };
243 const timer = setTimeout(
244 () => void during("spawn-due-wake", due).catch(() => {}),
245 Math.min(TIMER_MAX_MS, Math.max(0, at - now)),
246 );
243247 timer.unref?.();
244248 wakes.set(ledger.bodyId, timer);
245249 }
test/danger-buttons.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 // Every destructive button asks first. A `btn(..., { variant: "danger" })`
23 // inside a plain `<form method="post">` deletes on one click; new ones should
34 // go through `confirmAction` from src/web/ui.ts, which links to /-/confirm.
test/design-audit.test.ts +2
@@ -3,6 +3,8 @@
33 * route the router does not have (#2709). The router is read as text: the
44 * static GET paths in src/http/server.ts and settings_routes.ts, less the
55 * ones that are not pages.
6 *
7 * @tree-scan: reads docs/design/audit.md and the router as text.
68 */
79 import { expect, test } from "bun:test";
810 import { readFileSync } from "node:fs";
test/gate-stray-test.test.ts +1
@@ -1,4 +1,5 @@
11 // A `*.test.ts` under `src/` is outside every gate plan, so the gate refuses it by name (#11222).
2 // @tree-scan: asks the gate about this repository's own tree.
23 import { expect, test } from "bun:test";
34 import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
45 import { tmpdir } from "node:os";
test/hollow.test.ts +2
@@ -3,6 +3,8 @@
33 * nothing that runs reaches, or a screen no page links to, is named by
44 * `scripts/hollow.ts` unless `scripts/hollow-allow.ts` lists it under an
55 * owning issue.
6 *
7 * @tree-scan: `scripts/hollow.ts` reads the whole of src/ through the gate.
68 */
79
810 import { describe, expect, test } from "bun:test";
test/job-slices.test.ts +2
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * How long each background job holds the loop (#11118): one fixture with a busy forge's
34 * volume (thousands of issues and pulls, tens of thousands of runs, comments and events)
@@ -246,6 +247,7 @@ const UNTIMED: Record<string, string> = {
246247 "release-behind": "reads the forge's own release tags with git; needs FORGE_REPO and its repository",
247248 "release-keeper": "cluster only (FORGE_RELEASE_KEEPER=on): talks to the cluster's API",
248249 "roll-probe": "cluster only (FORGE_RELEASE_KEEPER=on): one HTTP probe",
250 "spawn-due-wake": "one child's wake when its paused run falls due; the fixture has no children",
249251 "apps-sweep": "AppsRuntime's sweep over running app processes; the fixture has no apps",
250252 "cell-watch": "watches running app cells' cgroups; needs cells, skipped in this sandbox mode",
251253 "cell-probe-deadline": "a cell's start probe running out of time: needs cells, skipped in this sandbox mode",
test/launch-no-tla.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * 0.7.283 did not boot on real data: a lazy require() of the crew's flow-label dispatcher reached a module
34 * graph that awaits at its top, and Bun refuses to require() that. Lazy loads in src/ use import(), never require().
test/readme-cap.test.ts +2
@@ -1,6 +1,8 @@
11 /**
22 * README.md stops growing (#6380): a feature's how-to goes in its own file
33 * under docs/features/, so pulls stop colliding on one append-only page.
4 *
5 * @tree-scan: reads README.md and docs/features/README.md.
46 */
57
68 import { expect, test } from "bun:test";
test/red-on-main.test.ts added +152
@@ -0,0 +1,152 @@
1 /**
2 * A pull is not red for main's red (#10304): when a fast tier's red test is
3 * red on the pull's base too, with the same tests, the check says so and is
4 * green; any other red is the pull's. Once end to end through
5 * `scripts/test-shards.ts`, in a scratch repository of trivial tests.
6 */
7 import { describe, expect, test } from "bun:test";
8 import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { EXCLUDED } from "../scripts/gate.ts";
12 import { askBase, baseVerdict, BASE_RERUN_CAP, clearedIn, type OnBase } from "../src/checks/base_red.ts";
13
14 const BASE = "0123456789abcdef0123456789abcdef01234567";
15 const red = (...failing: string[]) => ({ did: "red" as const, failing });
16
17 describe("whose red it is", () => {
18 test("a file red on the base with the same tests is main's, and clears the run", () => {
19 const got = baseVerdict(
20 new Map([["test/a.test.ts", ["x"]]]),
21 new Map([["test/a.test.ts", red("x", "y")]]),
22 BASE,
23 );
24 expect(got.cleared).toBe(true);
25 expect(got.lines).toEqual([
26 "test/a.test.ts is red on main at 0123456 too, the same tests: not counted against this pull",
27 "green: every red test is red on main at 0123456 too, so it is main's, not this pull's (test/a.test.ts)",
28 ]);
29 });
30
31 test("a second break in a file main already has red is the pull's", () => {
32 const got = baseVerdict(
33 new Map([["test/a.test.ts", ["x", "new break"]]]),
34 new Map([["test/a.test.ts", red("x")]]),
35 BASE,
36 );
37 expect(got.cleared).toBe(false);
38 expect(got.lines).toEqual([
39 `test/a.test.ts is red on main at 0123456 too, but "new break" passes there: this pull's red`,
40 ]);
41 });
42
43 test("a base red that names no test cannot be compared, and keeps the run red", () => {
44 const got = baseVerdict(new Map([["test/a.test.ts", ["x"]]]), new Map([["test/a.test.ts", red()]]), BASE);
45 expect(got.cleared).toBe(false);
46 expect(got.lines[0]).toContain("names no failing test there to compare: this pull's red");
47 });
48
49 test("one red file green on the base, absent from it or not run keeps the run red", () => {
50 for (const did of [{ did: "green" as const }, { did: "absent" as const }, undefined]) {
51 const onBase = new Map<string, OnBase>([["test/a.test.ts", red("x")]]);
52 if (did !== undefined) onBase.set("test/b.test.ts", did);
53 const got = baseVerdict(new Map([["test/a.test.ts", ["x"]], ["test/b.test.ts", ["z"]]]), onBase, BASE);
54 expect(got.cleared).toBe(false);
55 expect(got.lines[0]).toContain("test/a.test.ts is red on main");
56 expect(got.lines[1]).toMatch(/^test\/b\.test\.ts .*: this pull's red$/);
57 }
58 });
59
60 test("a cleared run's summary keeps main's red; a run that cleared none keeps nothing", () => {
61 const { lines } = baseVerdict(new Map([["test/a.test.ts", ["x"]]]), new Map([["test/a.test.ts", red("x")]]), BASE);
62 const output = ["(pass) b", ...lines.map((line) => `test-shards: ${line}`), "exit 0"];
63 expect(clearedIn(output)).toBe("red on main at 0123456, not this pull's: test/a.test.ts");
64 const kept = baseVerdict(new Map([["test/a.test.ts", ["x", "y"]]]), new Map([["test/a.test.ts", red("x")]]), BASE);
65 expect(clearedIn(kept.lines.map((line) => `test-shards: ${line}`))).toBeUndefined();
66 // A test printing the words is not the script's verdict.
67 expect(clearedIn([` ${lines.at(-1)}`])).toBeUndefined();
68 });
69
70 test("the base is not asked about a red with no file or test, a changed file, or too many", () => {
71 const one = (file: string, ...tests: string[]) => new Map([[file, tests]]);
72 expect(askBase({ red: undefined, changed: [] })).toBeUndefined();
73 expect(askBase({ red: new Map(), changed: [] })).toBeUndefined();
74 expect(askBase({ red: one("test/a.test.ts"), changed: ["README.md"] })).toBeUndefined();
75 expect(askBase({ red: one("test/a.test.ts", "x"), changed: ["test/a.test.ts"] })).toBeUndefined();
76 const many = new Map(Array.from({ length: BASE_RERUN_CAP + 1 }, (_, i) => [`test/t${i}.test.ts`, ["x"]]));
77 expect(askBase({ red: many, changed: ["README.md"] })).toBeUndefined();
78 expect(askBase({ red: one("test/a.test.ts", "x"), changed: ["README.md"] })).toEqual(["test/a.test.ts"]);
79 });
80 });
81
82 describe("the fast tier asks main", () => {
83 const script = new URL("../scripts/test-shards.ts", import.meta.url).pathname;
84 const RED = "test/unreached.test.ts";
85 const pass = `import { test } from "bun:test";\ntest("ok", () => {});\n`;
86 const fail = `import { expect, test } from "bun:test";\ntest("red", () => { expect(1).toBe(2); });\n`;
87 const git = (dir: string, ...args: string[]): number =>
88 Bun.spawnSync(["git", "-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", ...args], {
89 cwd: dir,
90 stdout: "ignore",
91 stderr: "ignore",
92 }).exitCode;
93 const env: Record<string, string | undefined> = { ...process.env, FORGE_CHECK_SHARDS: "2", FORGE_CELL_CPUS: "2" };
94 for (const name of Object.keys(env)) if (name.startsWith("FORGE_CHECK_") && name !== "FORGE_CHECK_SHARDS") delete env[name];
95
96 /** Main with RED as `onMain`, a pull that changes `pull`; the fast check of the pull. */
97 const check = async (onMain: string, pull: Record<string, string>, alsoOnMain: Record<string, string> = {}) => {
98 const dir = mkdtempSync(join(tmpdir(), "red-on-main-"));
99 try {
100 mkdirSync(join(dir, "test"));
101 // The gate refuses a tree its EXCLUDED list names files missing from; they never run.
102 for (const { file } of EXCLUDED) writeFileSync(join(dir, file), pass);
103 writeFileSync(join(dir, RED), onMain);
104 writeFileSync(join(dir, "test/plain.test.ts"), pass);
105 writeFileSync(join(dir, "README.md"), "# scratch\n");
106 for (const [file, text] of Object.entries(alsoOnMain)) {
107 mkdirSync(join(dir, file, ".."), { recursive: true });
108 writeFileSync(join(dir, file), text);
109 }
110 expect(git(dir, "init", "-q")).toBe(0);
111 expect(git(dir, "add", "-A")).toBe(0);
112 expect(git(dir, "commit", "-qm", "main")).toBe(0);
113 for (const [file, text] of Object.entries(pull)) writeFileSync(join(dir, file), text);
114 expect(git(dir, "add", "-A")).toBe(0);
115 expect(git(dir, "commit", "-qm", "the pull")).toBe(0);
116 const child = Bun.spawn(["bun", script, "--tier", "fast", "--diff", "HEAD~1..HEAD"], {
117 cwd: dir,
118 env,
119 stdin: "ignore",
120 stdout: "pipe",
121 stderr: "pipe",
122 });
123 const [out] = await Promise.all([new Response(child.stdout).text(), new Response(child.stderr).text()]);
124 return { code: await child.exited, out };
125 } finally {
126 rmSync(dir, { recursive: true, force: true });
127 }
128 };
129
130 test("a README-only pull opened while main fails a test is green, and names main's red", async () => {
131 const got = await check(fail, { "README.md": "# scratch, changed\n" });
132 expect(got.out).toContain(`${RED} is red on main at HEAD~1 too, the same tests: not counted against this pull`);
133 expect(got.out).toContain("green: every red test is red on main at HEAD~1 too");
134 expect(clearedIn(got.out.split("\n"))).toBe(`red on main at HEAD~1, not this pull's: ${RED}`);
135 expect(got.code).toBe(0);
136 }, 60_000);
137
138 test("a pull that changes main's red file is red: its red is its own to answer", async () => {
139 const got = await check(fail, { [RED]: `${fail}test("also red", () => { expect(2).toBe(3); });\n` });
140 expect(got.out).not.toContain("green: every red test is red on main");
141 expect(got.code).not.toBe(0);
142 }, 60_000);
143
144 test("a pull that adds a second break to main's red file without touching it is red", async () => {
145 const two = `import { expect, test } from "bun:test";\nimport { two } from "../src/two.ts";\n` +
146 `test("red", () => { expect(1).toBe(2); });\ntest("second", () => { expect(two).toBe(2); });\n`;
147 const got = await check(two, { "src/two.ts": "export const two = 3;\n" }, { "src/two.ts": "export const two = 2;\n" });
148 expect(got.out).toContain(`"second" passes there: this pull's red`);
149 expect(got.out).not.toContain("green: every red test is red on main");
150 expect(got.code).not.toBe(0);
151 }, 60_000);
152 });
test/release-boot.test.ts +2
@@ -4,6 +4,8 @@
44 * starts the tree on a scratch data directory and wants `/api/v1/health` to
55 * answer as the stamped version. 0.7.166's image crashed at import; a tree
66 * that does the same is refused here, naming the error.
7 *
8 * @tree-scan: copies all of src/ and boots the copy.
79 */
810
911 import { afterAll, beforeAll, expect, test } from "bun:test";
test/retry-after.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * Every 429 and 503 the HTTP layer answers says for how long (#10762).
34 * Part one reads src/http per site: a status literal must sit in a `tryAgain`
test/roll.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * The roll recipe's surge check (#1588): can a surge pod schedule beside the
34 * running forge, and if not, how much headroom it lacks. Every number below
test/rule-tests-always.test.ts +2
@@ -2,6 +2,8 @@
22 * The repo-wide rule tests run in every pull's fast tier (#7138): a pull
33 * that only adds a test file holding a raw ms budget is red on its own
44 * check, and the check's summary names the rule and the line.
5 *
6 * @tree-scan: lists the gate through `gateFiles()`.
57 */
68
79 import { afterAll, describe, expect, test } from "bun:test";
test/schema-migrations.test.ts +2
@@ -7,6 +7,8 @@
77 * twice, a branch whose migration was renumbered by its rebase, and a branch
88 * that adds a table the data already has. A failure names its statement in
99 * one line, which is what the deploy's summary on the pull shows.
10 *
11 * @tree-scan: reads every migration and its pin.
1012 */
1113
1214 import { afterAll, describe, expect, test } from "bun:test";
test/security-floor-doors.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * Every response leaves through a listener, so the floor is proven at the
34 * listeners (#10949). A `new Response(` anywhere in src/ — html, text, a
test/shards-run.test.ts +3
@@ -3,6 +3,9 @@
33 * it, and it spawns `bun test`, so each costs a few hundred milliseconds.
44 * The pure parts it is built from (`scripts/shards.ts`) are proved in
55 * test/shards.test.ts, which stays fast because it spawns nothing.
6 *
7 * @not-a-tree-scan: it lists only its own temporary directories, and what
8 * it takes from scripts/gate.ts does not read the tree.
69 */
710 import { afterAll, describe, expect, test } from "bun:test";
811 import { mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";
test/shards.test.ts +27
@@ -4,6 +4,8 @@
44 * red run whose tail names every failing test from every shard. Pure: it
55 * spawns nothing, so it stays fast; the runner end to end is
66 * test/shards-run.test.ts.
7 *
8 * @tree-scan: splits the real gate (`gateFiles()`) once.
79 */
810 import { describe, expect, test } from "bun:test";
911 import { readFileSync } from "node:fs";
@@ -16,6 +18,7 @@ import {
1618 CRASH_RERUNS,
1719 quoteCrash,
1820 failingFiles,
21 failingTests,
1922 onlyTestFailures,
2023 WORKER_MEMORY_BYTES,
2124 CELL_CPU_SHARE,
@@ -729,6 +732,30 @@ describe("failingFiles", () => {
729732 });
730733 });
731734
735 describe("failingTests", () => {
736 test("names each file's failing tests without their timing, not the summary's repeats", () => {
737 const report = [
738 "test/b.test.ts:",
739 "(pass) fine [0.1ms]",
740 "(fail) outer > one [0.12ms]",
741 "(fail) two [1.50s]",
742 "test/c.test.ts:",
743 "(fail) three",
744 "2 tests failed:",
745 "(fail) outer > one [0.12ms]",
746 " 1 pass",
747 " 3 fail",
748 "Ran 4 tests across 2 files. [9.00ms]",
749 ].join("\n");
750 expect(failingTests(report)).toEqual(
751 new Map([
752 ["test/b.test.ts", ["outer > one", "two"]],
753 ["test/c.test.ts", ["three"]],
754 ]),
755 );
756 });
757 });
758
732759 describe("onlyTestFailures", () => {
733760 const ran = "Ran 3 tests across 2 files. [9ms]";
734761 test("failed tests and a summary: true", () => {
test/signin-password-doors.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 import { afterEach, describe, expect, test } from "bun:test";
23 import { mkdtemp, readdir, readFile, rm } from "node:fs/promises";
34 import { tmpdir } from "node:os";
test/spawn-pipes.test.ts +2
@@ -3,6 +3,8 @@
33 * stdio nobody drains leaks native memory in a long-lived Bun server. Every
44 * Bun.spawn / spawnSync in src/ either ignores its unused streams or is
55 * listed here with the reason it reads each pipe to its end.
6 *
7 * @tree-scan: reads every file under src/.
68 */
79
810 import { expect, test } from "bun:test";
test/src-imports.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * The image ships `src/` and nothing else (Dockerfile): a module under src/
34 * that imports from scripts/ or test/ boots from the repository and crashes
test/thoughts.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
23 import { mkdtemp, readdir, readFile, rm } from "node:fs/promises";
34 import { tmpdir } from "node:os";
test/timers-named.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * Every server-side background timer names its job (#11117): a loop stall is
34 * blamed only on work run inside `during` (`src/perf/jobs.ts`), so a timer
test/tree-scans.test.ts added +112
@@ -0,0 +1,112 @@
1 /**
2 * A test that reads the tree is found by what it reads, not by a hand list
3 * (#10304): a new scan runs in every pull's fast tier the day it is
4 * written, so the pull that breaks it is red on its own check, not main.
5 */
6 import { describe, expect, test } from "bun:test";
7 import { readFileSync } from "node:fs";
8 import { gateFiles } from "../scripts/gate.ts";
9 import { parseTimes } from "../scripts/shards.ts";
10 import { importsReader, scansIn } from "../scripts/tier-input.ts";
11 import { declaresNoScan, declaresScan, SCAN_BUDGET_SECONDS, scanSeconds, treeScans, walksTree } from "../src/checks/scans.ts";
12 import { fastSet, RULE_TESTS } from "../src/checks/tiers.ts";
13
14 describe("what a scan reads", () => {
15 test("a glob or git ls-files lists the tree", () => {
16 expect(walksTree(`for (const f of new Glob("src/**/*.ts").scanSync(".")) {}`)).toBe(true);
17 expect(walksTree(`Bun.spawnSync(["git", "ls-files"])`)).toBe(true);
18 });
19
20 test("a readdir lists the tree only when it is rooted at the repository", () => {
21 expect(walksTree(`readdirSync(join(import.meta.dir, "..", "src"))`)).toBe(true);
22 expect(walksTree(`readdirSync("test").filter(f => f.endsWith(".ts"))`)).toBe(true);
23 expect(walksTree(`readdirSync(process.cwd())`)).toBe(true);
24 expect(walksTree(`const d = mkdtempSync(join(tmpdir(), "x")); readdirSync(d)`)).toBe(false);
25 });
26
27 test("the root counts only when it is what the readdir is handed", () => {
28 // The module's own place, used for a fixture, beside a readdir of a temp dir: no scan.
29 const fixture = `const jpeg = readFileSync(\`\${import.meta.dir}/fixtures/a.jpg\`);\nconst dir = join(dataDir, "looks");\nreaddirSync(dir);`;
30 expect(walksTree(fixture)).toBe(false);
31 expect(walksTree(`const repo = join(import.meta.dir, "..");\nreaddirSync(join(data, ".taking"));`)).toBe(false);
32 // Through a bound name: a scan.
33 expect(walksTree(`const ROOT = join(import.meta.dir, "..");\nconst DIR = join(ROOT, "src");\nreaddirSync(DIR);`)).toBe(true);
34 const walk = `const walk = (d: string): string[] => readdirSync(d).flatMap((n) => walk(join(d, n)));\nwalk(join(import.meta.dir, "..", "src"));`;
35 // A walk's parameter and a parameter's default are not inferred: such a scan declares itself.
36 expect(walksTree(walk)).toBe(false);
37 expect(declaresScan(`// @tree-scan: walks src/.\n${walk}`)).toBe(true);
38 expect(walksTree(`export function files(root = ".") { return readdirSync(\`\${root}/test\`); }`)).toBe(false);
39 });
40
41 test("a scan the reading cannot see declares itself; a false one says it is not", () => {
42 expect(declaresScan(`/**\n * @tree-scan: reads README.md.\n */`)).toBe(true);
43 expect(declaresScan(`// @tree-scan`)).toBe(true);
44 expect(declaresScan(`const s = "@tree-scan";`)).toBe(false);
45 expect(declaresScan(` * @not-a-tree-scan: its own temp dirs.`)).toBe(false);
46 expect(declaresNoScan(` * @not-a-tree-scan: its own temp dirs.`)).toBe(true);
47 });
48 });
49
50 describe("the tests that scan", () => {
51 const files: Record<string, string> = {
52 "test/new-scan.test.ts": `readdirSync(join(import.meta.dir, "../src"));`,
53 "test/via.test.ts": `import { audit } from "../scripts/audit.ts";`,
54 "scripts/audit.ts": `export const audit = () => [...new Glob("**/*.ts").scanSync(".")];`,
55 "test/readme.test.ts": `// @tree-scan: reads README.md\nreadFileSync("README.md");`,
56 "test/temp.test.ts": `// @not-a-tree-scan: lists its own temp dir\nreaddirSync(join(import.meta.dir, "tmp"));`,
57 "test/plain.test.ts": `import { a } from "../src/a.ts";`,
58 };
59 const imports = (file: string): string[] =>
60 file === "test/via.test.ts" ? ["scripts/audit.ts"] : file === "test/plain.test.ts" ? ["src/a.ts"] : [];
61 const tests = Object.keys(files).filter((f) => f.startsWith("test/"));
62 const scans = treeScans(tests, (f) => files[f], imports);
63
64 test("are found by their own source, a script they import, or their declaration", () => {
65 expect(Object.fromEntries(scans)).toEqual({
66 "test/new-scan.test.ts": "lists the tree",
67 "test/via.test.ts": "lists the tree through scripts/audit.ts",
68 "test/readme.test.ts": "declares a tree scan",
69 });
70 });
71
72 test("run in the fast tier of a diff no import or coverage ties to them", () => {
73 const plan = fastSet({
74 changed: ["src/brand-new.ts"],
75 tests,
76 imports,
77 times: new Map(),
78 impact: new Map([["test/plain.test.ts", ["src/a.ts"]]]),
79 scans,
80 });
81 for (const scan of scans.keys()) expect(plan.files).toContain(scan);
82 expect(plan.why.get("test/new-scan.test.ts")).toBe("tree scan: lists the tree");
83 expect(plan.files).not.toContain("test/temp.test.ts");
84 });
85
86 test("do not run for a diff of only the check's own tables", () => {
87 const plan = fastSet({
88 changed: ["scripts/test-times.json"],
89 tests,
90 imports,
91 times: new Map(),
92 scans,
93 });
94 expect(plan.files).toEqual([]);
95 });
96 });
97
98 describe("this repository's scans", () => {
99 const gate = gateFiles();
100 const scans = scansIn(gate, importsReader());
101
102 test("include every repo-wide rule test, so the hand list is no longer what finds them", () => {
103 for (const rule of RULE_TESTS) expect([rule, scans.has(rule)]).toEqual([rule, true]);
104 });
105
106 test("cost together no more than their share of the fast tier's budget", () => {
107 const all = new Set([...scans.keys(), ...RULE_TESTS]);
108 const times = parseTimes(readFileSync(new URL("../scripts/test-times.json", import.meta.url), "utf8"));
109 expect(times.ok).toBe(true);
110 expect(scanSeconds(all, times.ok ? times.value : new Map())).toBeLessThanOrEqual(SCAN_BUDGET_SECONDS);
111 });
112 });
test/tui-asks-first.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 // Every terminal key that deletes, removes, revokes, stops or detaches asks
23 // first, as test/danger-buttons.test.ts holds the web to. A call to one of the
34 // destructive actions below passes when it sits inside an `onChoose` or
test/version.test.ts +1
@@ -1,3 +1,4 @@
1 // @tree-scan: lists the repository through a walk it hands a rooted path.
12 /**
23 * The forge reports the version it is running (#756).
34 *

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs