Public: anyone can read this; the crew can change it.

A red main fails every pull's check: tell a pull's own red from main's, and keep main from landing red #10309

opened this in answer to

Check · Review · Land · Live — Changes were asked on this head. Next: builder, changes were asked on this head.

A reviewer asked for changes Review

mason/forge:mason/10304-a-red-main-fails-every-pull-s-check-tell → main 4 commits fork 3 threads · 3 unresolved

  • Waiting on review. juniper asked for changes. Push a fix, and ask for another look.

    changes requested by juniper

    Checked against an older mainCheck of 4b75a7echecked against main at 3dd8c7bfast tier: green in 13m12s · full gate runs on land

    Website spec: 1 finding (report only)

Review

Sign in to review this change.
18 files changed +684 −17
docs/DEVELOP.md +15 −4
@@ -105,13 +105,24 @@ map has never seen fall back to the import-hop guess, and a change to
105105 everything. With no map yet the line says `by imports (no impact map)`.
106106
107107 A test that reads the whole tree — every test file, every migration, the
108 README — is reached by no diff, yet any file can break it. Those are
109 `RULE_TESTS` in `src/checks/tiers.ts`, and every fast tier runs them
110 whatever else the diff holds (#7138) — a diff of only the timing or impact table breaks no rule and runs none; a new rule of that kind goes on the list. Write
111 the rule with `breaches()` from `test/helpers/rule.ts`: one red test per
108 README — is reached by no diff, yet any file can break it, so every fast
109 tier runs it whatever else the diff holds (#7138) — a diff of only the
110 timing or impact table breaks no rule and runs none. Such a test is found
111 by what it reads (`src/checks/scans.ts`, #10304): its source, or a
112 `scripts/` module it imports, lists the repository (a `Glob`, `git
113 ls-files`, a `readdir` rooted at the repo). One that reads fixed files
114 says `@tree-scan` in its header comment; one the reading mistakes for a
115 scan (it lists only its own temp directories) says `@not-a-tree-scan` and
116 why. `RULE_TESTS` in `src/checks/tiers.ts` is the floor the forge reads
117 without a tree; a new scan needs no line there. Write the rule with `breaches()` from `test/helpers/rule.ts`: one red test per
112118 breach, named `path:line: …`, so the check's summary names the rule and
113119 the line.
114120
121 A fast tier red only on test files the pull did not change is asked of
122 main: each is run once more on the base's tree, and when it is red there
123 too the log says `… is red on main at <base> too: not counted against
124 this pull` and the check is green (`src/checks/base_red.ts`).
125
115126 Before you push, ask what the fast tier would run for your change:
116127 `files=$(bun scripts/fast.ts) && bun test $files`. `bun scripts/fast.ts
117128 [--base origin/main] [--why | --null | --json]` lists the picked test files
docs/OPERATING.md +1 −1
@@ -218,7 +218,7 @@ A check, test or app cell asks for memory in `forge.json` (`check.limits.memory`
218218
219219 ## Check tiers
220220
221 `forge.json`'s `check` may declare `fast` beside `run` (the full gate; `full` names it explicitly). `bun scripts/test-shards.ts --tier fast --diff <base>..<head>` runs the tests that touch the diff (changed, named for a changed source file, or importing one within two hops), every repo-wide rule test (`RULE_TESTS`, #7138), every test the times table records under 1 s, and any named by `--learned a,b`. `src/checks/tiers.ts` decides; it adds no dependency.
221 `forge.json`'s `check` may declare `fast` beside `run` (the full gate; `full` names it explicitly). `bun scripts/test-shards.ts --tier fast --diff <base>..<head>` runs the tests that touch the diff (changed, named for a changed source file, or importing one within two hops), every repo-wide rule test (`RULE_TESTS`, #7138) and every test that reads the tree (`src/checks/scans.ts`, #10304), every test the times table records under 1 s, and any named by `--learned a,b`; a red only on files the pull did not change is run again on the base, and red there too it is main's and not counted (`src/checks/base_red.ts`). `src/checks/tiers.ts` decides; it adds no dependency.
222222
223223 To see what the fast tier would run for your own change before you push: `files=$(bun scripts/fast.ts) && bun test $files`. `bun scripts/fast.ts [--base origin/main] [--why | --null | --json]` lists, one per line on stdout and nothing else there, the test files `fastSet` picks for the files changed between the base's merge-base and the working tree (committed, uncommitted and untracked), by the same times table and `scripts/test-impact.json` as the check; `--why` adds a tab and the reason per file, `--null` ends each path with NUL (`| xargs -0 -r bun test`), `--json` prints `{full, files:[{file, why}]}`. Exit 0 lists files; exit 3 means the change reaches no test (stdout empty); exit 2 means the diff is one the map cannot bound (`package.json`, `forge.json`, a gate change that is logic…; stderr says `full tier: … changed`) or the arguments or git failed. The `&&` guard matters: `bun test $(…)` alone ignores the exit code and, with nothing listed, runs the whole suite. It reads the diff with `git` called by argument list, never a shell; `scripts/tier-input.ts` is the reading it shares with `test-shards.ts`.
224224
scripts/fast.ts +2 −1
@@ -20,7 +20,7 @@
2020 import { readFileSync } from "node:fs";
2121 import { GATE_FILE, gateListAdds, handedImpact, needsFull } from "../src/checks/impact.ts";
2222 import { fastSet, duration } from "../src/checks/tiers.ts";
23 import { importsReader, readImpact } from "./tier-input.ts";
23 import { importsReader, readImpact, scansIn } from "./tier-input.ts";
2424 import { CHECK_CACHE } from "../src/checks/cache.ts";
2525 import { gateFiles } from "./gate.ts";
2626 import { overlayTimes, parseTimes } from "./shards.ts";
@@ -174,6 +174,7 @@ function main(): number {
174174 ...(gateAdds === undefined ? {} : { gateAdds }),
175175 tests,
176176 imports,
177 scans: scansIn(tests, imports),
177178 times: loadTimes(),
178179 });
179180 const out = report(plan, needsFull(changed, gateAdds), args.format);
scripts/shards.ts +19 −3
@@ -559,14 +559,30 @@ const FAILED = /^\(fail\) /;
559559 * would not load) yields none.
560560 */
561561 export function failingFiles(output: string): string[] {
562 const found: string[] = [];
562 return [...failingTests(output).keys()];
563 }
564
565 /** `(fail) a > b [1.20ms]` → `a > b`. */
566 const testName = (line: string): string => line.replace(FAILED, "").replace(/\s+\[[\d.]+m?s\]\s*$/, "").trim();
567
568 /**
569 * The failing tests of one `bun test` report, by file, each file's names in
570 * report order: what {@link failingFiles} names, with which of its tests
571 * failed. The closing summary repeats them under no file and is not read.
572 */
573 export function failingTests(output: string): Map<string, string[]> {
574 const found = new Map<string, string[]>();
563575 let current: string | undefined;
564576 for (const line of output.split("\n")) {
565577 const header = HEADER.exec(line);
566578 if (header) current = header[1]!;
567579 else if (isSummaryLine(line)) current = undefined;
568 else if (FAILED.test(line) && current !== undefined && !found.includes(current))
569 found.push(current);
580 else if (FAILED.test(line) && current !== undefined) {
581 const names = found.get(current) ?? [];
582 const name = testName(line);
583 if (!names.includes(name)) names.push(name);
584 found.set(current, names);
585 }
570586 }
571587 return found;
572588 }
scripts/test-shards.ts +95 −5
@@ -53,8 +53,8 @@
5353 */
5454
5555 import { availableParallelism, setPriority, tmpdir, totalmem } from "node:os";
56 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
57 import { join } from "node:path";
56 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs";
57 import { join, resolve } from "node:path";
5858 import { resolveLimits } from "../src/apps/contain.ts";
5959 import {
6060 formatTimes,
@@ -75,6 +75,7 @@ import {
7575 weigher,
7676 besideOthers,
7777 failingFiles,
78 failingTests,
7879 onlyTestFailures,
7980 crashedOutside,
8081 quoteCrash,
@@ -84,13 +85,14 @@ import {
8485 import { gateFiles, gateRefusal, presentTests } from "./gate.ts";
8586 import { cgroupMemory, summaryLines } from "./gate-summary.ts";
8687 import { GATE_FILE, executedFiles, formatImpact, gateListAdds, handedImpact, mergeImpact, parseImpact } from "../src/checks/impact.ts";
87 import { importsReader, readImpact } from "./tier-input.ts";
88 import { importsReader, readImpact, scansIn } from "./tier-input.ts";
8889 import { shardHome, spaceLine } from "../src/checks/disk_full.ts";
8990 import { fastSet, duration } from "../src/checks/tiers.ts";
9091 import { CHECK_CACHE } from "../src/checks/cache.ts";
9192 import { parseCheck } from "../src/apps/manifest.ts";
9293 import { FLAKY_ALONE, onlyFiles } from "../src/checks/verify.ts";
9394 import { crashFlakeLine } from "../src/checks/flake.ts";
95 import { askBase, baseVerdict, type OnBase } from "../src/checks/base_red.ts";
9496
9597 const TIMES = new URL("./test-times.json", import.meta.url).pathname;
9698 // The forge hands the map over in the checkout (`src/checks/impact-store.ts`):
@@ -270,6 +272,8 @@ if (gate || tier === "fast") {
270272 }
271273 const GATE = gateFiles();
272274 let files = [...(gate || tier === "fast" ? GATE : []), ...split.files];
275 // The fast tier's diff, for asking its base about a red (#10304).
276 let changed: string[] = [];
273277 if (tier === "fast") {
274278 const diff = Bun.spawnSync(
275279 ["git", "diff", "--name-only", "--no-renames", diffArg.value!],
@@ -279,7 +283,7 @@ if (tier === "fast") {
279283 console.error(`test-shards: git diff failed: ${diff.stderr.toString()}`);
280284 process.exit(2);
281285 }
282 const changed = diff.stdout.toString().split("\n").filter((f) => f !== "");
286 changed = diff.stdout.toString().split("\n").filter((f) => f !== "");
283287 let gateAdds: string[] | undefined;
284288 if (changed.includes(GATE_FILE)) {
285289 const gateDiff = Bun.spawnSync(
@@ -296,6 +300,7 @@ if (tier === "fast") {
296300 ...(gateAdds === undefined ? {} : { gateAdds }),
297301 tests: files,
298302 imports,
303 scans: scansIn(files, imports),
299304 times: loadTimes(),
300305 learned: (learnedArg.value ?? "").split(",").filter((f) => f !== ""),
301306 });
@@ -759,4 +764,89 @@ if (timed)
759764 times: results.map((result) => ({ file: result.files[0]!, seconds: result.seconds })),
760765 }).join("\n")}`,
761766 );
762 process.exit(ending.code);
767
768 /**
769 * The red test files of this run, each with the tests in it that failed;
770 * `undefined` when a red names no file or no test (a crash, a load error).
771 */
772 function redTests(results: readonly Timed[]): Map<string, string[]> | undefined {
773 const red = new Map<string, string[]>();
774 const plain = (file: string): string => file.replace(/^\.\//, "");
775 for (const result of results) {
776 if (result.code === 0) continue;
777 const named = onlyTestFailures(result.output) ? failingTests(result.output) : new Map<string, string[]>();
778 if (named.size === 0) return undefined;
779 for (const [file, tests] of named) red.set(plain(file), [...(red.get(plain(file)) ?? []), ...tests]);
780 }
781 return red;
782 }
783
784 /**
785 * Each file run alone on the tree of `base`, in a worktree on the shard
786 * disk that shares this checkout's node_modules; removed however it ends.
787 */
788 async function runOnBase(asked: readonly string[], base: string): Promise<Map<string, OnBase>> {
789 const did = new Map<string, OnBase>();
790 const home = shardHome(process.env, tmpdir());
791 const tree = join(home, "base");
792 const git = (args: string[]): boolean =>
793 Bun.spawnSync(["git", ...args], { stdin: "ignore", stdout: "ignore", stderr: "pipe" }).exitCode === 0;
794 try {
795 if (!git(["worktree", "add", "--detach", tree, base])) {
796 console.log(`test-shards: could not check out ${base} to ask it about the red`);
797 return did;
798 }
799 if (existsSync("node_modules")) symlinkSync(resolve("node_modules"), join(tree, "node_modules"));
800 const env: Record<string, string | undefined> = { ...process.env };
801 const scratch = { tmp: join(home, "tmp"), data: join(home, "data") };
802 env["TMPDIR"] = scratch.tmp;
803 env["FORGE_APP_DATA"] = scratch.data;
804 delete env["NODE_OPTIONS"];
805 for (const file of asked) {
806 if (!existsSync(join(tree, file))) {
807 did.set(file, { did: "absent" });
808 continue;
809 }
810 mkdirSync(scratch.tmp, { recursive: true });
811 mkdirSync(scratch.data, { recursive: true });
812 const child = Bun.spawn(["bun", "test", ...flags, `./${file}`], {
813 cwd: tree,
814 env,
815 stdin: "ignore",
816 stdout: "pipe",
817 stderr: "pipe",
818 });
819 children.add(child);
820 const output: string[] = [];
821 await Promise.all([drain(child.stdout, output), drain(child.stderr, output)]);
822 const report = output.join("");
823 if ((await child.exited) === 0) did.set(file, { did: "green" });
824 else {
825 // A crash or load error names no test: nothing to compare the pull's red with.
826 const named = onlyTestFailures(report) ? failingTests(report) : new Map<string, string[]>();
827 did.set(file, { did: "red", failing: [...named.values()].flat() });
828 }
829 children.delete(child);
830 rmSync(scratch.data, { recursive: true, force: true });
831 }
832 } finally {
833 git(["worktree", "remove", "--force", tree]);
834 rmSync(home, { recursive: true, force: true });
835 }
836 return did;
837 }
838
839 // A fast tier red only on tests main's own tree fails too is main's red,
840 // not this pull's (#10304): said, and not counted against it.
841 let code = ending.code;
842 if (code !== 0 && tier === "fast" && range !== undefined && only === undefined) {
843 const red = redTests(results);
844 const asked = askBase({ red, changed });
845 if (red !== undefined && asked !== undefined) {
846 console.log(`\ntest-shards: asking main at ${range[0]} about ${asked.join(", ")}`);
847 const verdict = baseVerdict(red, await runOnBase(asked, range[0]!), range[0]!);
848 for (const line of verdict.lines) console.log(`test-shards: ${line}`);
849 if (verdict.cleared) code = 0;
850 }
851 }
852 process.exit(code);
scripts/tier-input.ts +19
@@ -7,6 +7,7 @@
77 import { readFileSync } from "node:fs";
88 import { importsOf } from "../src/checks/tiers.ts";
99 import { parseImpact } from "../src/checks/impact.ts";
10 import { treeScans } from "../src/checks/scans.ts";
1011
1112 /** A file's relative imports as candidate paths (an extensionless one also as `.ts`); `[]` when unreadable. Memoised. */
1213 export function importsReader(): (file: string) => string[] {
@@ -45,3 +46,21 @@ export function readImpact(
4546 }
4647 return undefined;
4748 }
49
50 /** The gate's tests that read the tree (`src/checks/scans.ts`), read from the working tree. */
51 export function scansIn(
52 tests: readonly string[],
53 imports: (file: string) => readonly string[],
54 ): Map<string, string> {
55 return treeScans(
56 tests,
57 (file) => {
58 try {
59 return readFileSync(file, "utf8");
60 } catch {
61 return undefined;
62 }
63 },
64 imports,
65 );
66 }
src/checks/base_red.ts added +101
@@ -0,0 +1,101 @@
1 /**
2 * A pull's red that is main's (#10304). A test that landed red on main
3 * fails every pull's check after it, and each pull's builder reworks a
4 * change that was fine. So when a pull's fast tier is red, each failing
5 * test file is run once more on the tree of the pull's base. Red there with
6 * the same tests, it is main's: the log says so, and it does not count
7 * against the pull.
8 *
9 * What stays the pull's, always: a file the pull changed (its red may be
10 * the change's), a file the base does not have, one green on the base, a
11 * test that fails on the pull and passes on the base (a second break in a
12 * file main already has red), and a red no file or test is named for (a
13 * crash, a type error). Over
14 * {@link BASE_RERUN_CAP} red files the base is not asked: that many is
15 * this pull's, or main's red is wide enough for the red-main hold
16 * (`red_main.ts`) to say.
17 *
18 * Plain data in, plain data out: `scripts/test-shards.ts` runs the files
19 * on the base and hands over what each did.
20 */
21
22 /** The most red files the base is asked about. */
23 export const BASE_RERUN_CAP = 5;
24
25 /**
26 * What one red file did on the base's tree: red, with the tests that
27 * failed (none named when it crashed or would not load), green, or not
28 * there to run.
29 */
30 export type OnBase =
31 | { readonly did: "red"; readonly failing: readonly string[] }
32 | { readonly did: "green" }
33 | { readonly did: "absent" };
34
35 /** Each red file of the pull's run, with the tests in it that failed. */
36 export type RedTests = ReadonlyMap<string, readonly string[]>;
37
38 export interface BaseInput {
39 /** The pull's red files and their failing tests; `undefined` when some red names no file or no test. */
40 readonly red: RedTests | undefined;
41 /** Files the pull changed. */
42 readonly changed: readonly string[];
43 }
44
45 /** The red files worth asking the base about, or `undefined` when none could clear the run. */
46 export function askBase(input: BaseInput): string[] | undefined {
47 const red = input.red;
48 if (red === undefined || red.size === 0 || red.size > BASE_RERUN_CAP) return undefined;
49 const files = [...red.keys()];
50 if (files.some((file) => input.changed.includes(file) || red.get(file)!.length === 0)) return undefined;
51 return files;
52 }
53
54 export interface BaseVerdict {
55 /** Every red was main's: the pull's run is green. */
56 readonly cleared: boolean;
57 /** What the log says, one line each. */
58 readonly lines: string[];
outdated

Red on the base is not the same red. Clear only when the pull's failing test names are a subset of the base's; otherwise a second break in an already-red file is hidden.

59 }
60
61 /** `abcdef0123…` → `abcdef0`. */
62 const short = (rev: string): string => (/^[0-9a-f]{40,64}$/.test(rev) ? rev.slice(0, 7) : rev);
63
64 /** At most three names, quoted, and how many more. */
65 const some = (names: readonly string[]): string =>
66 names.slice(0, 3).map((name) => JSON.stringify(name)).join(", ") + (names.length > 3 ? ` and ${names.length - 3} more` : "");
67
68 /**
69 * Whether the pull's red is main's, from what each red file did on `base`.
70 * A file is main's only when every test of it that fails on the pull fails
71 * on the base too: red on the base is not the same red, and a second break
72 * in a file main already has red is the pull's.
73 */
74 export function baseVerdict(red: RedTests, onBase: ReadonlyMap<string, OnBase>, base: string): BaseVerdict {
75 const at = `main at ${short(base)}`;
76 const mains: string[] = [];
77 const lines: string[] = [];
78 const theirs: string[] = [];
79 for (const [file, failing] of red) {
80 const there = onBase.get(file);
81 if (there === undefined) theirs.push(`${file} did not run on ${at}: this pull's red`);
82 else if (there.did === "green") theirs.push(`${file} is green on ${at}: this pull's red`);
83 else if (there.did === "absent") theirs.push(`${file} is not on ${at}: this pull's red`);
84 else {
85 const own = failing.filter((name) => !there.failing.includes(name));
86 if (failing.length > 0 && own.length === 0) {
87 mains.push(file);
88 lines.push(`${file} is red on ${at} too, the same tests: not counted against this pull`);
89 } else if (there.failing.length === 0)
90 theirs.push(`${file} is red on ${at}, but names no failing test there to compare: this pull's red`);
91 else theirs.push(`${file} is red on ${at} too, but ${some(own)} pass${own.length === 1 ? "es" : ""} there: this pull's red`);
92 }
93 }
94 lines.push(...theirs);
95 const cleared = red.size > 0 && mains.length === red.size;
96 if (cleared)
97 lines.push(
98 `green: every red test is red on ${at} too, so it is main's, not this pull's (${mains.join(", ")})`,
99 );
100 return { cleared, lines };
101 }
src/checks/scans.ts added +155
@@ -0,0 +1,155 @@
1 /**
2 * Tree scans (#10304): tests that read the repository's tree — every file
3 * under src/, every test, the README — rather than import what they judge.
4 * No import map and no coverage ties one to a diff (a new module, a long
5 * comment line), so a scan the fast tier did not know of landed green and
6 * turned every later pull red. A hand list of them (`RULE_TESTS`) misses
7 * the next one written.
8 *
9 * So a scan is found by what it reads: a test whose own source, or a
10 * `scripts/` module it imports, lists a directory of the repository (a
11 * `Glob`, `git ls-files`, or a `readdir` handed a path rooted at the repo). A
12 * scan the reading cannot see (one that reads a fixed file such as the
13 * README) declares itself with a line holding {@link SCAN_MARK}; a test the
14 * reading takes for a scan that only lists its own temporary directories
15 * says so with {@link NOT_SCAN_MARK} and the reason, and is not one.
16 *
17 * Plain data in, plain data out: sources come through `read`; the scripts
18 * read the files.
19 */
20
21 /** The declaration a test makes when it scans the tree in a way {@link walksTree} cannot see. */
22 export const SCAN_MARK = "@tree-scan";
23
24 /** The declaration a test makes when it lists only what it made itself. */
25 export const NOT_SCAN_MARK = "@not-a-tree-scan";
26
27 /** Lists a directory: a glob or `git ls-files`. */
28 const GLOB = /\bnew\s+(?:Bun\.)?Glob\s*\(|["'`]ls-files["'`]/;

ROOT is tested against the whole source, not against the readdir argument, so any file that uses import.meta.dir and has a readdir is a scan. Tie the root to the readdir call.

29 /** A `readdir` call, up to its opening parenthesis. */
30 const READDIR = /\breaddir(?:Sync)?\s*\(/g;
31 /** A path expression's head: what a `join(`, `resolve(` or template opens with. */
32 const HEAD = String.raw`^\s*(?:(?:path\.)?(?:join|resolve)\s*\(\s*)?`;
33 /** Names the repository's root: through the module's own place, or the working directory. */
34 const ROOT = /import\.meta\.(?:dir|dirname|url|path)\b|process\.cwd\s*\(/;
35 /** A path that opens with a top-level directory of the repository, or `.`/`..`, by name. */
36 const TOP = new RegExp(`${HEAD}["'\`](?:\\.{1,2}|(?:\\./)?(?:src|test|docs|scripts))(?:/|["'\`])`);
37
38 /** Whether `text` is a path into the repository, given the names already known to hold one. */
39 function intoRepo(text: string, rooted: ReadonlySet<string>): boolean {
40 if (ROOT.test(text) || TOP.test(text)) return true;
41 if (rooted.size === 0) return false;
42 const names = [...rooted].map((name) => name.replace(/\$/g, "\\$")).join("|");
43 return new RegExp(`${HEAD}(?:\\$\\{\\s*)?(?:${names})(?![\\w$])`).test(text.replace(/^\s*`/, ""));
44 }
45
46 /** The text between the parenthesis at `open` and the one that closes it. */
47 function argumentAt(source: string, open: number): string {
48 let depth = 0;
49 for (let i = open; i < source.length; i++) {
50 const c = source[i];
51 if (c === "(") depth++;
52 else if (c === ")" && --depth === 0) return source.slice(open + 1, i);
53 }
54 return source.slice(open + 1);
55 }
56
57 /**
58 * The names that hold a path into the repository: bound (or defaulted, as
59 * a parameter) to an expression that names the root, or to a path built
60 * from such a name. A parameter a
61 * walk takes (a `function` or an arrow bound to a name) counts when a call
62 * of it hands it one.
63 */
64 function rootedNames(source: string): Set<string> {
65 const bindings: { name: string; value: string }[] = [];
66 for (const m of source.matchAll(/\b(?:const|let|var)\s+([\w$]+)\s*(?::[^=\n]+)?=\s*([^;\n]+)/g))
67 bindings.push({ name: m[1]!, value: m[2]! });
68 // A parameter's default: `function files(root = ".")`, `(dir: string = DIR) =>`.
69 for (const m of source.matchAll(/[(,]\s*([\w$]+)\s*(?::\s*[\w$.<>[\]| ]+)?=(?![=>])\s*([^,)\n]+)/g))
70 bindings.push({ name: m[1]!, value: m[2]! });
71 const walkers = /\bfunction\s+([\w$]+)\s*\(\s*([\w$]+)|\b(?:const|let|var)\s+([\w$]+)\s*=\s*(?:async\s*)?\(\s*([\w$]+)/g;
72 for (const w of source.matchAll(walkers)) {
73 const [walker, param] = [(w[1] ?? w[3])!, (w[2] ?? w[4])!];
74 const calls = source.matchAll(new RegExp(`(?<![\\w$.])${walker.replace(/\$/g, "\\$")}\\s*\\(`, "g"));
75 for (const call of calls) bindings.push({ name: param, value: argumentAt(source, call.index! + call[0].length - 1) });
76 }
77 const rooted = new Set<string>();
78 for (let grew = true; grew; ) {
79 grew = false;
80 for (const { name, value } of bindings)
81 if (!rooted.has(name) && intoRepo(value, rooted)) {
82 rooted.add(name);
83 grew = true;
84 }
85 }
86 return rooted;
87 }
88
89 /** Whether a `readdir` in the source is handed a path into the repository. */
90 function readsRepoDir(source: string): boolean {
91 let rooted: Set<string> | undefined;
92 for (const call of source.matchAll(READDIR)) {
93 const arg = argumentAt(source, call.index! + call[0].length - 1);
94 rooted ??= rootedNames(source);
95 if (intoRepo(arg, rooted)) return true;
96 }
97 return false;
98 }
99
100 /** Whether a comment line of the source holds `mark` (as a word: `@tree-scan` is not in `@not-a-tree-scan`). */
101 const declares = (source: string, mark: string): boolean =>
102 source
103 .split("\n")
104 .some((line) => /^\s*(?:\/\/|\/?\*)/.test(line) && new RegExp(`(?:^|[\\s*/])${mark}\\b`).test(line));
105
106 /** Whether a source declares itself a scan: {@link SCAN_MARK} on a comment line. */
107 export const declaresScan = (source: string): boolean => declares(source, SCAN_MARK);
108
109 /** Whether a source declares it is no scan: {@link NOT_SCAN_MARK} on a comment line. */
110 export const declaresNoScan = (source: string): boolean => declares(source, NOT_SCAN_MARK);
111
112 /** Whether a source lists the repository's tree. */
113 export const walksTree = (source: string): boolean => GLOB.test(source) || readsRepoDir(source);
114
115 /** Why a source is a scan, or `undefined` when it is not one. */
116 export function scanOf(source: string): "declared" | "lists the tree" | undefined {
117 if (declaresScan(source)) return "declared";
118 if (declaresNoScan(source)) return undefined;
119 return walksTree(source) ? "lists the tree" : undefined;
120 }
121
122 /**
123 * The tests that scan the tree, each with why: by its own source, or by a
124 * `scripts/` module it imports directly (a scan's logic usually lives in
125 * the script the check also runs). `read` answers `undefined` for a file it
126 * cannot read; `imports` is a file's repository-relative imports.
127 */
128 export function treeScans(
129 tests: readonly string[],
130 read: (file: string) => string | undefined,
131 imports: (file: string) => readonly string[],
132 ): Map<string, string> {
133 const scripts = new Map<string, ReturnType<typeof scanOf>>();
134 const scriptScan = (file: string): ReturnType<typeof scanOf> => {
135 if (!scripts.has(file)) {
136 const source = read(file);
137 scripts.set(file, source === undefined ? undefined : scanOf(source));
138 }
139 return scripts.get(file);
140 };
141 const found = new Map<string, string>();
142 for (const test of tests) {
143 const source = read(test);
144 if (source === undefined) continue;
145 const own = scanOf(source);
146 if (own !== undefined) {
147 found.set(test, own === "declared" ? "declares a tree scan" : "lists the tree");
148 continue;
149 }
150 if (declaresNoScan(source)) continue;
151 const via = imports(test).find((file) => file.startsWith("scripts/") && scriptScan(file) !== undefined);
152 if (via !== undefined) found.set(test, `lists the tree through ${via}`);
153 }
154 return found;
155 }
src/checks/tiers.ts +11 −3
@@ -20,7 +20,10 @@ export const QUICK_SECONDS = 1;
2020 export const FAST_BUDGET_SECONDS = 5 * 60;
2121
2222 /**
23 * The repo-wide rule tests (#7138): each reads every test file, every
23 * The repo-wide rule tests (#7138), the floor that needs no tree to read
24 * (`outside_diff.ts` judges by it from a run's log alone). Every other test
25 * that reads the tree is found by what it reads (`scans.ts`, #10304), so a
26 * new scan needs no line here. Each reads every test file, every
2427 * migration or the whole README, so no diff "touches" them by coverage,
2528 * import or name, yet any file can break them. They run in every fast
2629 * tier, whatever else the diff holds (a diff of only the check's own
@@ -111,6 +114,8 @@ export interface FastInput {
111114 readonly learned?: readonly string[];
112115 /** Measured: the `src/**` files each test executed (`scripts/test-impact.json`). */
113116 readonly impact?: Impact;
117 /** The tests that read the tree, with why (`treeScans`, #10304): every diff can break one. */
118 readonly scans?: ReadonlyMap<string, string>;
114119 }
115120
116121 export interface FastPlan {
@@ -127,7 +132,8 @@ export interface FastPlan {
127132 /**
128133 * The fast tier for a diff: a changed test; a test importing a changed file
129134 * directly or through up to two hops of source; a test whose name matches a
130 * changed source path; every repo-wide rule test; whatever the pull
135 * changed source path; every repo-wide rule test and every tree scan
136 * (`scans.ts`); whatever the pull
131137 * learned; and every test known to take under a second. A test with no recorded time is not known to be quick, so it runs only when
132138 * something above picks it.
133139 */
@@ -176,8 +182,10 @@ export function fastSet(input: FastInput): FastPlan {
176182 const changed = new Set(diffed);
177183 for (const file of changed) take(file, "changed");
178184 // The check's own tables can break no rule: a diff of only those runs none (#3579).
179 if (diffed.some((f) => !f.endsWith("test-times.json") && !f.endsWith("test-impact.json")))
185 if (diffed.some((f) => !f.endsWith("test-times.json") && !f.endsWith("test-impact.json"))) {
180186 for (const test of RULE_TESTS) take(test, "repo-wide rule");
187 for (const [test, why] of input.scans ?? []) take(test, `tree scan: ${why}`);
188 }
181189 // What a guess looks for. A test the map has measured already answered
182190 // for every change the map can see, by what it executes; it is guessed
183191 // only against the changes the map cannot see, so one unreached file
test/design-audit.test.ts +2
@@ -3,6 +3,8 @@
33 * route the router does not have (#2709). The router is read as text: the
44 * static GET paths in src/http/server.ts and settings_routes.ts, less the
55 * ones that are not pages.
6 *
7 * @tree-scan: reads docs/design/audit.md and the router as text.
68 */
79 import { expect, test } from "bun:test";
810 import { readFileSync } from "node:fs";
test/readme-cap.test.ts +2
@@ -1,6 +1,8 @@
11 /**
22 * README.md stops growing (#6380): a feature's how-to goes in its own file
33 * under docs/features/, so pulls stop colliding on one append-only page.
4 *
5 * @tree-scan: reads README.md and docs/features/README.md.
46 */
57
68 import { expect, test } from "bun:test";
test/red-on-main.test.ts added +122
@@ -0,0 +1,122 @@
1 /**
2 * A pull is not red for main's red (#10304): when a fast tier's red test is
3 * red on the pull's base too, with the same tests, the check says so and is
4 * green; any other red is the pull's. Once end to end through
5 * `scripts/test-shards.ts`, in a scratch repository of trivial tests.
6 */
7 import { describe, expect, test } from "bun:test";
8 import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { EXCLUDED } from "../scripts/gate.ts";
12 import { askBase, baseVerdict, BASE_RERUN_CAP, type OnBase } from "../src/checks/base_red.ts";
13
14 const BASE = "0123456789abcdef0123456789abcdef01234567";
15 const red = (...failing: string[]) => ({ did: "red" as const, failing });
16
17 describe("whose red it is", () => {
18 test("a file red on the base with the same tests is main's, and clears the run", () => {
19 const got = baseVerdict(
20 new Map([["test/a.test.ts", ["x"]]]),
21 new Map([["test/a.test.ts", red("x", "y")]]),
22 BASE,
23 );
24 expect(got.cleared).toBe(true);
25 expect(got.lines).toEqual([
26 "test/a.test.ts is red on main at 0123456 too, the same tests: not counted against this pull",
27 "green: every red test is red on main at 0123456 too, so it is main's, not this pull's (test/a.test.ts)",
28 ]);
29 });
30
31 test("a second break in a file main already has red is the pull's", () => {
32 const got = baseVerdict(
33 new Map([["test/a.test.ts", ["x", "new break"]]]),
34 new Map([["test/a.test.ts", red("x")]]),
35 BASE,
36 );
37 expect(got.cleared).toBe(false);
38 expect(got.lines).toEqual([
39 `test/a.test.ts is red on main at 0123456 too, but "new break" passes there: this pull's red`,
40 ]);
41 });
42
43 test("a base red that names no test cannot be compared, and keeps the run red", () => {
44 const got = baseVerdict(new Map([["test/a.test.ts", ["x"]]]), new Map([["test/a.test.ts", red()]]), BASE);
45 expect(got.cleared).toBe(false);
46 expect(got.lines[0]).toContain("names no failing test there to compare: this pull's red");
47 });
48
49 test("one red file green on the base, absent from it or not run keeps the run red", () => {
50 for (const did of [{ did: "green" as const }, { did: "absent" as const }, undefined]) {
outdated

Each case writes every present test file into a fresh repo and spawns bun test-shards. Slow and flaky-prone; keep the minimum of e2e cases.

51 const onBase = new Map<string, OnBase>([["test/a.test.ts", red("x")]]);
52 if (did !== undefined) onBase.set("test/b.test.ts", did);
53 const got = baseVerdict(new Map([["test/a.test.ts", ["x"]], ["test/b.test.ts", ["z"]]]), onBase, BASE);
54 expect(got.cleared).toBe(false);
55 expect(got.lines[0]).toContain("test/a.test.ts is red on main");
56 expect(got.lines[1]).toMatch(/^test\/b\.test\.ts .*: this pull's red$/);
57 }
58 });
59
60 test("the base is not asked about a red with no file or test, a changed file, or too many", () => {
61 const one = (file: string, ...tests: string[]) => new Map([[file, tests]]);
62 expect(askBase({ red: undefined, changed: [] })).toBeUndefined();
63 expect(askBase({ red: new Map(), changed: [] })).toBeUndefined();
64 expect(askBase({ red: one("test/a.test.ts"), changed: ["README.md"] })).toBeUndefined();
65 expect(askBase({ red: one("test/a.test.ts", "x"), changed: ["test/a.test.ts"] })).toBeUndefined();
66 const many = new Map(Array.from({ length: BASE_RERUN_CAP + 1 }, (_, i) => [`test/t${i}.test.ts`, ["x"]]));
67 expect(askBase({ red: many, changed: ["README.md"] })).toBeUndefined();
68 expect(askBase({ red: one("test/a.test.ts", "x"), changed: ["README.md"] })).toEqual(["test/a.test.ts"]);
69 });
70 });
71
72 describe("the fast tier asks main", () => {
73 const script = new URL("../scripts/test-shards.ts", import.meta.url).pathname;
74 const RED = "test/unreached.test.ts";
75 const pass = `import { test } from "bun:test";\ntest("ok", () => {});\n`;
76 const fail = `import { expect, test } from "bun:test";\ntest("red", () => { expect(1).toBe(2); });\n`;
77 const git = (dir: string, ...args: string[]): number =>
78 Bun.spawnSync(["git", "-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", ...args], {
79 cwd: dir,
80 stdout: "ignore",
81 stderr: "ignore",
82 }).exitCode;
83 const env: Record<string, string | undefined> = { ...process.env, FORGE_CHECK_SHARDS: "2", FORGE_CELL_CPUS: "2" };
84 for (const name of Object.keys(env)) if (name.startsWith("FORGE_CHECK_") && name !== "FORGE_CHECK_SHARDS") delete env[name];
85
86 /** Main with RED as `onMain`, a pull that changes `pull`; the fast check of the pull. */
87 const check = async (onMain: string, pull: Record<string, string>) => {
88 const dir = mkdtempSync(join(tmpdir(), "red-on-main-"));
89 try {
90 mkdirSync(join(dir, "test"));
91 // The gate refuses a tree its EXCLUDED list names files missing from; they never run.
92 for (const { file } of EXCLUDED) writeFileSync(join(dir, file), pass);
93 writeFileSync(join(dir, RED), onMain);
94 writeFileSync(join(dir, "test/plain.test.ts"), pass);
95 writeFileSync(join(dir, "README.md"), "# scratch\n");
96 expect(git(dir, "init", "-q")).toBe(0);
97 expect(git(dir, "add", "-A")).toBe(0);
98 expect(git(dir, "commit", "-qm", "main")).toBe(0);
99 for (const [file, text] of Object.entries(pull)) writeFileSync(join(dir, file), text);
100 expect(git(dir, "add", "-A")).toBe(0);
101 expect(git(dir, "commit", "-qm", "the pull")).toBe(0);
102 const child = Bun.spawn(["bun", script, "--tier", "fast", "--diff", "HEAD~1..HEAD"], {
103 cwd: dir,
104 env,
105 stdin: "ignore",
106 stdout: "pipe",
107 stderr: "pipe",
108 });
109 const [out] = await Promise.all([new Response(child.stdout).text(), new Response(child.stderr).text()]);
110 return { code: await child.exited, out };
111 } finally {
112 rmSync(dir, { recursive: true, force: true });
113 }
114 };
115
116 test("a README-only pull opened while main fails a test is green, and names main's red", async () => {
117 const got = await check(fail, { "README.md": "# scratch, changed\n" });
118 expect(got.out).toContain(`${RED} is red on main at HEAD~1 too, the same tests: not counted against this pull`);
119 expect(got.out).toContain("green: every red test is red on main at HEAD~1 too");
120 expect(got.code).toBe(0);
121 }, 60_000);
122 });
test/release-boot.test.ts +2
@@ -4,6 +4,8 @@
44 * starts the tree on a scratch data directory and wants `/api/v1/health` to
55 * answer as the stamped version. 0.7.166's image crashed at import; a tree
66 * that does the same is refused here, naming the error.
7 *
8 * @tree-scan: copies all of src/ and boots the copy.
79 */
810
911 import { afterAll, beforeAll, expect, test } from "bun:test";
test/schema-migrations.test.ts +2
@@ -7,6 +7,8 @@
77 * twice, a branch whose migration was renumbered by its rebase, and a branch
88 * that adds a table the data already has. A failure names its statement in
99 * one line, which is what the deploy's summary on the pull shows.
10 *
11 * @tree-scan: reads every migration and its pin.
1012 */
1113
1214 import { afterAll, describe, expect, test } from "bun:test";
test/shards-run.test.ts +3
@@ -3,6 +3,9 @@
33 * it, and it spawns `bun test`, so each costs a few hundred milliseconds.
44 * The pure parts it is built from (`scripts/shards.ts`) are proved in
55 * test/shards.test.ts, which stays fast because it spawns nothing.
6 *
7 * @not-a-tree-scan: it lists only its own temporary directories, and what
8 * it takes from scripts/gate.ts does not read the tree.
69 */
710 import { afterAll, describe, expect, test } from "bun:test";
811 import { mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";
test/shards.test.ts +25
@@ -16,6 +16,7 @@ import {
1616 CRASH_RERUNS,
1717 quoteCrash,
1818 failingFiles,
19 failingTests,
1920 onlyTestFailures,
2021 WORKER_MEMORY_BYTES,
2122 CELL_CPU_SHARE,
@@ -729,6 +730,30 @@ describe("failingFiles", () => {
729730 });
730731 });
731732
733 describe("failingTests", () => {
734 test("names each file's failing tests without their timing, not the summary's repeats", () => {
735 const report = [
736 "test/b.test.ts:",
737 "(pass) fine [0.1ms]",
738 "(fail) outer > one [0.12ms]",
739 "(fail) two [1.50s]",
740 "test/c.test.ts:",
741 "(fail) three",
742 "2 tests failed:",
743 "(fail) outer > one [0.12ms]",
744 " 1 pass",
745 " 3 fail",
746 "Ran 4 tests across 2 files. [9.00ms]",
747 ].join("\n");
748 expect(failingTests(report)).toEqual(
749 new Map([
750 ["test/b.test.ts", ["outer > one", "two"]],
751 ["test/c.test.ts", ["three"]],
752 ]),
753 );
754 });
755 });
756
732757 describe("onlyTestFailures", () => {
733758 const ran = "Ran 3 tests across 2 files. [9ms]";
734759 test("failed tests and a summary: true", () => {
test/spawn-pipes.test.ts +2
@@ -3,6 +3,8 @@
33 * stdio nobody drains leaks native memory in a long-lived Bun server. Every
44 * Bun.spawn / spawnSync in src/ either ignores its unused streams or is
55 * listed here with the reason it reads each pipe to its end.
6 *
7 * @tree-scan: reads every file under src/.
68 */
79
810 import { expect, test } from "bun:test";
test/tree-scans.test.ts added +106
@@ -0,0 +1,106 @@
1 /**
2 * A test that reads the tree is found by what it reads, not by a hand list
3 * (#10304): a new scan runs in every pull's fast tier the day it is
4 * written, so the pull that breaks it is red on its own check, not main.
5 */
6 import { describe, expect, test } from "bun:test";
7 import { gateFiles } from "../scripts/gate.ts";
8 import { importsReader, scansIn } from "../scripts/tier-input.ts";
9 import { declaresNoScan, declaresScan, treeScans, walksTree } from "../src/checks/scans.ts";
10 import { fastSet, RULE_TESTS } from "../src/checks/tiers.ts";
11
12 describe("what a scan reads", () => {
13 test("a glob or git ls-files lists the tree", () => {
14 expect(walksTree(`for (const f of new Glob("src/**/*.ts").scanSync(".")) {}`)).toBe(true);
15 expect(walksTree(`Bun.spawnSync(["git", "ls-files"])`)).toBe(true);
16 });
17
18 test("a readdir lists the tree only when it is rooted at the repository", () => {
19 expect(walksTree(`readdirSync(join(import.meta.dir, "..", "src"))`)).toBe(true);
20 expect(walksTree(`readdirSync("test").filter(f => f.endsWith(".ts"))`)).toBe(true);
21 expect(walksTree(`readdirSync(process.cwd())`)).toBe(true);
22 expect(walksTree(`const d = mkdtempSync(join(tmpdir(), "x")); readdirSync(d)`)).toBe(false);
23 });
24
25 test("the root counts only when it is what the readdir is handed", () => {
26 // The module's own place, used for a fixture, beside a readdir of a temp dir: no scan.
27 const fixture = `const jpeg = readFileSync(\`\${import.meta.dir}/fixtures/a.jpg\`);\nconst dir = join(dataDir, "looks");\nreaddirSync(dir);`;
28 expect(walksTree(fixture)).toBe(false);
29 expect(walksTree(`const repo = join(import.meta.dir, "..");\nreaddirSync(join(data, ".taking"));`)).toBe(false);
30 // Through a name, a walk's parameter, or a parameter's default: a scan.
31 expect(walksTree(`const ROOT = join(import.meta.dir, "..");\nconst DIR = join(ROOT, "src");\nreaddirSync(DIR);`)).toBe(true);
32 const walk = `const walk = (d: string): string[] => readdirSync(d).flatMap((n) => walk(join(d, n)));\nwalk(join(import.meta.dir, "..", "src"));`;
33 expect(walksTree(walk)).toBe(true);
34 expect(walksTree(`const walk = (d: string) => readdirSync(d);\nwalk(mkdtempSync("x"));`)).toBe(false);
35 expect(walksTree(`export function files(root = ".") { return readdirSync(\`\${root}/test\`); }`)).toBe(true);
36 });
37
38 test("a scan the reading cannot see declares itself; a false one says it is not", () => {
39 expect(declaresScan(`/**\n * @tree-scan: reads README.md.\n */`)).toBe(true);
40 expect(declaresScan(`// @tree-scan`)).toBe(true);
41 expect(declaresScan(`const s = "@tree-scan";`)).toBe(false);
42 expect(declaresScan(` * @not-a-tree-scan: its own temp dirs.`)).toBe(false);
43 expect(declaresNoScan(` * @not-a-tree-scan: its own temp dirs.`)).toBe(true);
44 });
45 });
46
47 describe("the tests that scan", () => {
48 const files: Record<string, string> = {
49 "test/new-scan.test.ts": `readdirSync(join(import.meta.dir, "../src"));`,
50 "test/via.test.ts": `import { audit } from "../scripts/audit.ts";`,
51 "scripts/audit.ts": `export const audit = () => [...new Glob("**/*.ts").scanSync(".")];`,
52 "test/readme.test.ts": `// @tree-scan: reads README.md\nreadFileSync("README.md");`,
53 "test/temp.test.ts": `// @not-a-tree-scan: lists its own temp dir\nreaddirSync(join(import.meta.dir, "tmp"));`,
54 "test/plain.test.ts": `import { a } from "../src/a.ts";`,
55 };
56 const imports = (file: string): string[] =>
57 file === "test/via.test.ts" ? ["scripts/audit.ts"] : file === "test/plain.test.ts" ? ["src/a.ts"] : [];
58 const tests = Object.keys(files).filter((f) => f.startsWith("test/"));
59 const scans = treeScans(tests, (f) => files[f], imports);
60
61 test("are found by their own source, a script they import, or their declaration", () => {
62 expect(Object.fromEntries(scans)).toEqual({
63 "test/new-scan.test.ts": "lists the tree",
64 "test/via.test.ts": "lists the tree through scripts/audit.ts",
65 "test/readme.test.ts": "declares a tree scan",
66 });
67 });
68
69 test("run in the fast tier of a diff no import or coverage ties to them", () => {
70 const plan = fastSet({
71 changed: ["src/brand-new.ts"],
72 tests,
73 imports,
74 times: new Map(),
75 impact: new Map([["test/plain.test.ts", ["src/a.ts"]]]),
76 scans,
77 });
78 for (const scan of scans.keys()) expect(plan.files).toContain(scan);
79 expect(plan.why.get("test/new-scan.test.ts")).toBe("tree scan: lists the tree");
80 expect(plan.files).not.toContain("test/temp.test.ts");
81 });
82
83 test("do not run for a diff of only the check's own tables", () => {
84 const plan = fastSet({
85 changed: ["scripts/test-times.json"],
86 tests,
87 imports,
88 times: new Map(),
89 scans,
90 });
91 expect(plan.files).toEqual([]);
92 });
93 });
94
95 describe("this repository's scans", () => {
96 const gate = gateFiles();
97 const scans = scansIn(gate, importsReader());
98
99 test("include every repo-wide rule test, so the hand list is no longer what finds them", () => {
100 for (const rule of RULE_TESTS) expect([rule, scans.has(rule)]).toEqual([rule, true]);
101 });
102
103 test("an unimported new module reaches test/unreached.test.ts as a scan", () => {
104 expect(scans.get("test/unreached.test.ts")).toBe("lists the tree through scripts/unreached.ts");
105 });
106 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs