Public: anyone can read this; the crew can change it.
A red main fails every pull's check: tell a pull's own red from main's, and keep main from landing red #10309
Check · Review · Land · Live — Changes were asked on this head.
mason/forge:mason/10304-a-red-main-fails-every-pull-s-check-tell → main 4 commits fork 3 threads · 3 unresolved
18 files
changed +684
−17
docs/DEVELOP.md +15 −4
| @@ -105,13 +105,24 @@ map has never seen fall back to the import-hop guess, and a change to | ||
| 105 | 105 | everything. With no map yet the line says `by imports (no impact map)`. |
| 106 | 106 | |
| 107 | 107 | A test that reads the whole tree — every test file, every migration, the |
| 108 | README — is reached by no diff, yet any file can break it. Those are | |
| 109 | `RULE_TESTS` in `src/checks/tiers.ts`, and every fast tier runs them | |
| 110 | whatever else the diff holds (#7138) — a diff of only the timing or impact table breaks no rule and runs none; a new rule of that kind goes on the list. Write | |
| 111 | the rule with `breaches()` from `test/helpers/rule.ts`: one red test per | |
| 108 | README — is reached by no diff, yet any file can break it, so every fast | |
| 109 | tier runs it whatever else the diff holds (#7138) — a diff of only the | |
| 110 | timing or impact table breaks no rule and runs none. Such a test is found | |
| 111 | by what it reads (`src/checks/scans.ts`, #10304): its source, or a | |
| 112 | `scripts/` module it imports, lists the repository (a `Glob`, `git | |
| 113 | ls-files`, a `readdir` rooted at the repo). One that reads fixed files | |
| 114 | says `@tree-scan` in its header comment; one the reading mistakes for a | |
| 115 | scan (it lists only its own temp directories) says `@not-a-tree-scan` and | |
| 116 | why. `RULE_TESTS` in `src/checks/tiers.ts` is the floor the forge reads | |
| 117 | without a tree; a new scan needs no line there. Write the rule with `breaches()` from `test/helpers/rule.ts`: one red test per | |
| 112 | 118 | breach, named `path:line: …`, so the check's summary names the rule and |
| 113 | 119 | the line. |
| 114 | 120 | |
| 121 | A fast tier red only on test files the pull did not change is asked of | |
| 122 | main: each is run once more on the base's tree, and when it is red there | |
| 123 | too the log says `… is red on main at <base> too: not counted against | |
| 124 | this pull` and the check is green (`src/checks/base_red.ts`). | |
| 125 | ||
| 115 | 126 | Before you push, ask what the fast tier would run for your change: |
| 116 | 127 | `files=$(bun scripts/fast.ts) && bun test $files`. `bun scripts/fast.ts |
| 117 | 128 | [--base origin/main] [--why | --null | --json]` lists the picked test files |
docs/OPERATING.md +1 −1
| @@ -218,7 +218,7 @@ A check, test or app cell asks for memory in `forge.json` (`check.limits.memory` | ||
| 218 | 218 | |
| 219 | 219 | ## Check tiers |
| 220 | 220 | |
| 221 | `forge.json`'s `check` may declare `fast` beside `run` (the full gate; `full` names it explicitly). `bun scripts/test-shards.ts --tier fast --diff <base>..<head>` runs the tests that touch the diff (changed, named for a changed source file, or importing one within two hops), every repo-wide rule test (`RULE_TESTS`, #7138), every test the times table records under 1 s, and any named by `--learned a,b`. `src/checks/tiers.ts` decides; it adds no dependency. | |
| 221 | `forge.json`'s `check` may declare `fast` beside `run` (the full gate; `full` names it explicitly). `bun scripts/test-shards.ts --tier fast --diff <base>..<head>` runs the tests that touch the diff (changed, named for a changed source file, or importing one within two hops), every repo-wide rule test (`RULE_TESTS`, #7138) and every test that reads the tree (`src/checks/scans.ts`, #10304), every test the times table records under 1 s, and any named by `--learned a,b`; a red only on files the pull did not change is run again on the base, and red there too it is main's and not counted (`src/checks/base_red.ts`). `src/checks/tiers.ts` decides; it adds no dependency. | |
| 222 | 222 | |
| 223 | 223 | To see what the fast tier would run for your own change before you push: `files=$(bun scripts/fast.ts) && bun test $files`. `bun scripts/fast.ts [--base origin/main] [--why | --null | --json]` lists, one per line on stdout and nothing else there, the test files `fastSet` picks for the files changed between the base's merge-base and the working tree (committed, uncommitted and untracked), by the same times table and `scripts/test-impact.json` as the check; `--why` adds a tab and the reason per file, `--null` ends each path with NUL (`| xargs -0 -r bun test`), `--json` prints `{full, files:[{file, why}]}`. Exit 0 lists files; exit 3 means the change reaches no test (stdout empty); exit 2 means the diff is one the map cannot bound (`package.json`, `forge.json`, a gate change that is logic…; stderr says `full tier: … changed`) or the arguments or git failed. The `&&` guard matters: `bun test $(…)` alone ignores the exit code and, with nothing listed, runs the whole suite. It reads the diff with `git` called by argument list, never a shell; `scripts/tier-input.ts` is the reading it shares with `test-shards.ts`. |
| 224 | 224 |
scripts/fast.ts +2 −1
| @@ -20,7 +20,7 @@ | ||
| 20 | 20 | import { readFileSync } from "node:fs"; |
| 21 | 21 | import { GATE_FILE, gateListAdds, handedImpact, needsFull } from "../src/checks/impact.ts"; |
| 22 | 22 | import { fastSet, duration } from "../src/checks/tiers.ts"; |
| 23 | import { importsReader, readImpact } from "./tier-input.ts"; | |
| 23 | import { importsReader, readImpact, scansIn } from "./tier-input.ts"; | |
| 24 | 24 | import { CHECK_CACHE } from "../src/checks/cache.ts"; |
| 25 | 25 | import { gateFiles } from "./gate.ts"; |
| 26 | 26 | import { overlayTimes, parseTimes } from "./shards.ts"; |
| @@ -174,6 +174,7 @@ function main(): number { | ||
| 174 | 174 | ...(gateAdds === undefined ? {} : { gateAdds }), |
| 175 | 175 | tests, |
| 176 | 176 | imports, |
| 177 | scans: scansIn(tests, imports), | |
| 177 | 178 | times: loadTimes(), |
| 178 | 179 | }); |
| 179 | 180 | const out = report(plan, needsFull(changed, gateAdds), args.format); |
scripts/shards.ts +19 −3
| @@ -559,14 +559,30 @@ const FAILED = /^\(fail\) /; | ||
| 559 | 559 | * would not load) yields none. |
| 560 | 560 | */ |
| 561 | 561 | export function failingFiles(output: string): string[] { |
| 562 | const found: string[] = []; | |
| 562 | return [...failingTests(output).keys()]; | |
| 563 | } | |
| 564 | ||
| 565 | /** `(fail) a > b [1.20ms]` → `a > b`. */ | |
| 566 | const testName = (line: string): string => line.replace(FAILED, "").replace(/\s+\[[\d.]+m?s\]\s*$/, "").trim(); | |
| 567 | ||
| 568 | /** | |
| 569 | * The failing tests of one `bun test` report, by file, each file's names in | |
| 570 | * report order: what {@link failingFiles} names, with which of its tests | |
| 571 | * failed. The closing summary repeats them under no file and is not read. | |
| 572 | */ | |
| 573 | export function failingTests(output: string): Map<string, string[]> { | |
| 574 | const found = new Map<string, string[]>(); | |
| 563 | 575 | let current: string | undefined; |
| 564 | 576 | for (const line of output.split("\n")) { |
| 565 | 577 | const header = HEADER.exec(line); |
| 566 | 578 | if (header) current = header[1]!; |
| 567 | 579 | else if (isSummaryLine(line)) current = undefined; |
| 568 | else if (FAILED.test(line) && current !== undefined && !found.includes(current)) | |
| 569 | found.push(current); | |
| 580 | else if (FAILED.test(line) && current !== undefined) { | |
| 581 | const names = found.get(current) ?? []; | |
| 582 | const name = testName(line); | |
| 583 | if (!names.includes(name)) names.push(name); | |
| 584 | found.set(current, names); | |
| 585 | } | |
| 570 | 586 | } |
| 571 | 587 | return found; |
| 572 | 588 | } |
scripts/test-shards.ts +95 −5
| @@ -53,8 +53,8 @@ | ||
| 53 | 53 | */ |
| 54 | 54 | |
| 55 | 55 | import { availableParallelism, setPriority, tmpdir, totalmem } from "node:os"; |
| 56 | import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; | |
| 57 | import { join } from "node:path"; | |
| 56 | import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; | |
| 57 | import { join, resolve } from "node:path"; | |
| 58 | 58 | import { resolveLimits } from "../src/apps/contain.ts"; |
| 59 | 59 | import { |
| 60 | 60 | formatTimes, |
| @@ -75,6 +75,7 @@ import { | ||
| 75 | 75 | weigher, |
| 76 | 76 | besideOthers, |
| 77 | 77 | failingFiles, |
| 78 | failingTests, | |
| 78 | 79 | onlyTestFailures, |
| 79 | 80 | crashedOutside, |
| 80 | 81 | quoteCrash, |
| @@ -84,13 +85,14 @@ import { | ||
| 84 | 85 | import { gateFiles, gateRefusal, presentTests } from "./gate.ts"; |
| 85 | 86 | import { cgroupMemory, summaryLines } from "./gate-summary.ts"; |
| 86 | 87 | import { GATE_FILE, executedFiles, formatImpact, gateListAdds, handedImpact, mergeImpact, parseImpact } from "../src/checks/impact.ts"; |
| 87 | import { importsReader, readImpact } from "./tier-input.ts"; | |
| 88 | import { importsReader, readImpact, scansIn } from "./tier-input.ts"; | |
| 88 | 89 | import { shardHome, spaceLine } from "../src/checks/disk_full.ts"; |
| 89 | 90 | import { fastSet, duration } from "../src/checks/tiers.ts"; |
| 90 | 91 | import { CHECK_CACHE } from "../src/checks/cache.ts"; |
| 91 | 92 | import { parseCheck } from "../src/apps/manifest.ts"; |
| 92 | 93 | import { FLAKY_ALONE, onlyFiles } from "../src/checks/verify.ts"; |
| 93 | 94 | import { crashFlakeLine } from "../src/checks/flake.ts"; |
| 95 | import { askBase, baseVerdict, type OnBase } from "../src/checks/base_red.ts"; | |
| 94 | 96 | |
| 95 | 97 | const TIMES = new URL("./test-times.json", import.meta.url).pathname; |
| 96 | 98 | // The forge hands the map over in the checkout (`src/checks/impact-store.ts`): |
| @@ -270,6 +272,8 @@ if (gate || tier === "fast") { | ||
| 270 | 272 | } |
| 271 | 273 | const GATE = gateFiles(); |
| 272 | 274 | let files = [...(gate || tier === "fast" ? GATE : []), ...split.files]; |
| 275 | // The fast tier's diff, for asking its base about a red (#10304). | |
| 276 | let changed: string[] = []; | |
| 273 | 277 | if (tier === "fast") { |
| 274 | 278 | const diff = Bun.spawnSync( |
| 275 | 279 | ["git", "diff", "--name-only", "--no-renames", diffArg.value!], |
| @@ -279,7 +283,7 @@ if (tier === "fast") { | ||
| 279 | 283 | console.error(`test-shards: git diff failed: ${diff.stderr.toString()}`); |
| 280 | 284 | process.exit(2); |
| 281 | 285 | } |
| 282 | const changed = diff.stdout.toString().split("\n").filter((f) => f !== ""); | |
| 286 | changed = diff.stdout.toString().split("\n").filter((f) => f !== ""); | |
| 283 | 287 | let gateAdds: string[] | undefined; |
| 284 | 288 | if (changed.includes(GATE_FILE)) { |
| 285 | 289 | const gateDiff = Bun.spawnSync( |
| @@ -296,6 +300,7 @@ if (tier === "fast") { | ||
| 296 | 300 | ...(gateAdds === undefined ? {} : { gateAdds }), |
| 297 | 301 | tests: files, |
| 298 | 302 | imports, |
| 303 | scans: scansIn(files, imports), | |
| 299 | 304 | times: loadTimes(), |
| 300 | 305 | learned: (learnedArg.value ?? "").split(",").filter((f) => f !== ""), |
| 301 | 306 | }); |
| @@ -759,4 +764,89 @@ if (timed) | ||
| 759 | 764 | times: results.map((result) => ({ file: result.files[0]!, seconds: result.seconds })), |
| 760 | 765 | }).join("\n")}`, |
| 761 | 766 | ); |
| 762 | process.exit(ending.code); | |
| 767 | ||
| 768 | /** | |
| 769 | * The red test files of this run, each with the tests in it that failed; | |
| 770 | * `undefined` when a red names no file or no test (a crash, a load error). | |
| 771 | */ | |
| 772 | function redTests(results: readonly Timed[]): Map<string, string[]> | undefined { | |
| 773 | const red = new Map<string, string[]>(); | |
| 774 | const plain = (file: string): string => file.replace(/^\.\//, ""); | |
| 775 | for (const result of results) { | |
| 776 | if (result.code === 0) continue; | |
| 777 | const named = onlyTestFailures(result.output) ? failingTests(result.output) : new Map<string, string[]>(); | |
| 778 | if (named.size === 0) return undefined; | |
| 779 | for (const [file, tests] of named) red.set(plain(file), [...(red.get(plain(file)) ?? []), ...tests]); | |
| 780 | } | |
| 781 | return red; | |
| 782 | } | |
| 783 | ||
| 784 | /** | |
| 785 | * Each file run alone on the tree of `base`, in a worktree on the shard | |
| 786 | * disk that shares this checkout's node_modules; removed however it ends. | |
| 787 | */ | |
| 788 | async function runOnBase(asked: readonly string[], base: string): Promise<Map<string, OnBase>> { | |
| 789 | const did = new Map<string, OnBase>(); | |
| 790 | const home = shardHome(process.env, tmpdir()); | |
| 791 | const tree = join(home, "base"); | |
| 792 | const git = (args: string[]): boolean => | |
| 793 | Bun.spawnSync(["git", ...args], { stdin: "ignore", stdout: "ignore", stderr: "pipe" }).exitCode === 0; | |
| 794 | try { | |
| 795 | if (!git(["worktree", "add", "--detach", tree, base])) { | |
| 796 | console.log(`test-shards: could not check out ${base} to ask it about the red`); | |
| 797 | return did; | |
| 798 | } | |
| 799 | if (existsSync("node_modules")) symlinkSync(resolve("node_modules"), join(tree, "node_modules")); | |
| 800 | const env: Record<string, string | undefined> = { ...process.env }; | |
| 801 | const scratch = { tmp: join(home, "tmp"), data: join(home, "data") }; | |
| 802 | env["TMPDIR"] = scratch.tmp; | |
| 803 | env["FORGE_APP_DATA"] = scratch.data; | |
| 804 | delete env["NODE_OPTIONS"]; | |
| 805 | for (const file of asked) { | |
| 806 | if (!existsSync(join(tree, file))) { | |
| 807 | did.set(file, { did: "absent" }); | |
| 808 | continue; | |
| 809 | } | |
| 810 | mkdirSync(scratch.tmp, { recursive: true }); | |
| 811 | mkdirSync(scratch.data, { recursive: true }); | |
| 812 | const child = Bun.spawn(["bun", "test", ...flags, `./${file}`], { | |
| 813 | cwd: tree, | |
| 814 | env, | |
| 815 | stdin: "ignore", | |
| 816 | stdout: "pipe", | |
| 817 | stderr: "pipe", | |
| 818 | }); | |
| 819 | children.add(child); | |
| 820 | const output: string[] = []; | |
| 821 | await Promise.all([drain(child.stdout, output), drain(child.stderr, output)]); | |
| 822 | const report = output.join(""); | |
| 823 | if ((await child.exited) === 0) did.set(file, { did: "green" }); | |
| 824 | else { | |
| 825 | // A crash or load error names no test: nothing to compare the pull's red with. | |
| 826 | const named = onlyTestFailures(report) ? failingTests(report) : new Map<string, string[]>(); | |
| 827 | did.set(file, { did: "red", failing: [...named.values()].flat() }); | |
| 828 | } | |
| 829 | children.delete(child); | |
| 830 | rmSync(scratch.data, { recursive: true, force: true }); | |
| 831 | } | |
| 832 | } finally { | |
| 833 | git(["worktree", "remove", "--force", tree]); | |
| 834 | rmSync(home, { recursive: true, force: true }); | |
| 835 | } | |
| 836 | return did; | |
| 837 | } | |
| 838 | ||
| 839 | // A fast tier red only on tests main's own tree fails too is main's red, | |
| 840 | // not this pull's (#10304): said, and not counted against it. | |
| 841 | let code = ending.code; | |
| 842 | if (code !== 0 && tier === "fast" && range !== undefined && only === undefined) { | |
| 843 | const red = redTests(results); | |
| 844 | const asked = askBase({ red, changed }); | |
| 845 | if (red !== undefined && asked !== undefined) { | |
| 846 | console.log(`\ntest-shards: asking main at ${range[0]} about ${asked.join(", ")}`); | |
| 847 | const verdict = baseVerdict(red, await runOnBase(asked, range[0]!), range[0]!); | |
| 848 | for (const line of verdict.lines) console.log(`test-shards: ${line}`); | |
| 849 | if (verdict.cleared) code = 0; | |
| 850 | } | |
| 851 | } | |
| 852 | process.exit(code); |
scripts/tier-input.ts +19
| @@ -7,6 +7,7 @@ | ||
| 7 | 7 | import { readFileSync } from "node:fs"; |
| 8 | 8 | import { importsOf } from "../src/checks/tiers.ts"; |
| 9 | 9 | import { parseImpact } from "../src/checks/impact.ts"; |
| 10 | import { treeScans } from "../src/checks/scans.ts"; | |
| 10 | 11 | |
| 11 | 12 | /** A file's relative imports as candidate paths (an extensionless one also as `.ts`); `[]` when unreadable. Memoised. */ |
| 12 | 13 | export function importsReader(): (file: string) => string[] { |
| @@ -45,3 +46,21 @@ export function readImpact( | ||
| 45 | 46 | } |
| 46 | 47 | return undefined; |
| 47 | 48 | } |
| 49 | ||
| 50 | /** The gate's tests that read the tree (`src/checks/scans.ts`), read from the working tree. */ | |
| 51 | export function scansIn( | |
| 52 | tests: readonly string[], | |
| 53 | imports: (file: string) => readonly string[], | |
| 54 | ): Map<string, string> { | |
| 55 | return treeScans( | |
| 56 | tests, | |
| 57 | (file) => { | |
| 58 | try { | |
| 59 | return readFileSync(file, "utf8"); | |
| 60 | } catch { | |
| 61 | return undefined; | |
| 62 | } | |
| 63 | }, | |
| 64 | imports, | |
| 65 | ); | |
| 66 | } |
src/checks/base_red.ts added +101
| @@ -0,0 +1,101 @@ | ||
| 1 | /** | |
| 2 | * A pull's red that is main's (#10304). A test that landed red on main | |
| 3 | * fails every pull's check after it, and each pull's builder reworks a | |
| 4 | * change that was fine. So when a pull's fast tier is red, each failing | |
| 5 | * test file is run once more on the tree of the pull's base. Red there with | |
| 6 | * the same tests, it is main's: the log says so, and it does not count | |
| 7 | * against the pull. | |
| 8 | * | |
| 9 | * What stays the pull's, always: a file the pull changed (its red may be | |
| 10 | * the change's), a file the base does not have, one green on the base, a | |
| 11 | * test that fails on the pull and passes on the base (a second break in a | |
| 12 | * file main already has red), and a red no file or test is named for (a | |
| 13 | * crash, a type error). Over | |
| 14 | * {@link BASE_RERUN_CAP} red files the base is not asked: that many is | |
| 15 | * this pull's, or main's red is wide enough for the red-main hold | |
| 16 | * (`red_main.ts`) to say. | |
| 17 | * | |
| 18 | * Plain data in, plain data out: `scripts/test-shards.ts` runs the files | |
| 19 | * on the base and hands over what each did. | |
| 20 | */ | |
| 21 | ||
| 22 | /** The most red files the base is asked about. */ | |
| 23 | export const BASE_RERUN_CAP = 5; | |
| 24 | ||
| 25 | /** | |
| 26 | * What one red file did on the base's tree: red, with the tests that | |
| 27 | * failed (none named when it crashed or would not load), green, or not | |
| 28 | * there to run. | |
| 29 | */ | |
| 30 | export type OnBase = | |
| 31 | | { readonly did: "red"; readonly failing: readonly string[] } | |
| 32 | | { readonly did: "green" } | |
| 33 | | { readonly did: "absent" }; | |
| 34 | ||
| 35 | /** Each red file of the pull's run, with the tests in it that failed. */ | |
| 36 | export type RedTests = ReadonlyMap<string, readonly string[]>; | |
| 37 | ||
| 38 | export interface BaseInput { | |
| 39 | /** The pull's red files and their failing tests; `undefined` when some red names no file or no test. */ | |
| 40 | readonly red: RedTests | undefined; | |
| 41 | /** Files the pull changed. */ | |
| 42 | readonly changed: readonly string[]; | |
| 43 | } | |
| 44 | ||
| 45 | /** The red files worth asking the base about, or `undefined` when none could clear the run. */ | |
| 46 | export function askBase(input: BaseInput): string[] | undefined { | |
| 47 | const red = input.red; | |
| 48 | if (red === undefined || red.size === 0 || red.size > BASE_RERUN_CAP) return undefined; | |
| 49 | const files = [...red.keys()]; | |
| 50 | if (files.some((file) => input.changed.includes(file) || red.get(file)!.length === 0)) return undefined; | |
| 51 | return files; | |
| 52 | } | |
| 53 | ||
| 54 | export interface BaseVerdict { | |
| 55 | /** Every red was main's: the pull's run is green. */ | |
| 56 | readonly cleared: boolean; | |
| 57 | /** What the log says, one line each. */ | |
| 58 | readonly lines: string[]; | |
| 59 | } | |
| 60 | ||
| 61 | /** `abcdef0123…` → `abcdef0`. */ | |
| 62 | const short = (rev: string): string => (/^[0-9a-f]{40,64}$/.test(rev) ? rev.slice(0, 7) : rev); | |
| 63 | ||
| 64 | /** At most three names, quoted, and how many more. */ | |
| 65 | const some = (names: readonly string[]): string => | |
| 66 | names.slice(0, 3).map((name) => JSON.stringify(name)).join(", ") + (names.length > 3 ? ` and ${names.length - 3} more` : ""); | |
| 67 | ||
| 68 | /** | |
| 69 | * Whether the pull's red is main's, from what each red file did on `base`. | |
| 70 | * A file is main's only when every test of it that fails on the pull fails | |
| 71 | * on the base too: red on the base is not the same red, and a second break | |
| 72 | * in a file main already has red is the pull's. | |
| 73 | */ | |
| 74 | export function baseVerdict(red: RedTests, onBase: ReadonlyMap<string, OnBase>, base: string): BaseVerdict { | |
| 75 | const at = `main at ${short(base)}`; | |
| 76 | const mains: string[] = []; | |
| 77 | const lines: string[] = []; | |
| 78 | const theirs: string[] = []; | |
| 79 | for (const [file, failing] of red) { | |
| 80 | const there = onBase.get(file); | |
| 81 | if (there === undefined) theirs.push(`${file} did not run on ${at}: this pull's red`); | |
| 82 | else if (there.did === "green") theirs.push(`${file} is green on ${at}: this pull's red`); | |
| 83 | else if (there.did === "absent") theirs.push(`${file} is not on ${at}: this pull's red`); | |
| 84 | else { | |
| 85 | const own = failing.filter((name) => !there.failing.includes(name)); | |
| 86 | if (failing.length > 0 && own.length === 0) { | |
| 87 | mains.push(file); | |
| 88 | lines.push(`${file} is red on ${at} too, the same tests: not counted against this pull`); | |
| 89 | } else if (there.failing.length === 0) | |
| 90 | theirs.push(`${file} is red on ${at}, but names no failing test there to compare: this pull's red`); | |
| 91 | else theirs.push(`${file} is red on ${at} too, but ${some(own)} pass${own.length === 1 ? "es" : ""} there: this pull's red`); | |
| 92 | } | |
| 93 | } | |
| 94 | lines.push(...theirs); | |
| 95 | const cleared = red.size > 0 && mains.length === red.size; | |
| 96 | if (cleared) | |
| 97 | lines.push( | |
| 98 | `green: every red test is red on ${at} too, so it is main's, not this pull's (${mains.join(", ")})`, | |
| 99 | ); | |
| 100 | return { cleared, lines }; | |
| 101 | } |
src/checks/scans.ts added +155
| @@ -0,0 +1,155 @@ | ||
| 1 | /** | |
| 2 | * Tree scans (#10304): tests that read the repository's tree — every file | |
| 3 | * under src/, every test, the README — rather than import what they judge. | |
| 4 | * No import map and no coverage ties one to a diff (a new module, a long | |
| 5 | * comment line), so a scan the fast tier did not know of landed green and | |
| 6 | * turned every later pull red. A hand list of them (`RULE_TESTS`) misses | |
| 7 | * the next one written. | |
| 8 | * | |
| 9 | * So a scan is found by what it reads: a test whose own source, or a | |
| 10 | * `scripts/` module it imports, lists a directory of the repository (a | |
| 11 | * `Glob`, `git ls-files`, or a `readdir` handed a path rooted at the repo). A | |
| 12 | * scan the reading cannot see (one that reads a fixed file such as the | |
| 13 | * README) declares itself with a line holding {@link SCAN_MARK}; a test the | |
| 14 | * reading takes for a scan that only lists its own temporary directories | |
| 15 | * says so with {@link NOT_SCAN_MARK} and the reason, and is not one. | |
| 16 | * | |
| 17 | * Plain data in, plain data out: sources come through `read`; the scripts | |
| 18 | * read the files. | |
| 19 | */ | |
| 20 | ||
| 21 | /** The declaration a test makes when it scans the tree in a way {@link walksTree} cannot see. */ | |
| 22 | export const SCAN_MARK = "@tree-scan"; | |
| 23 | ||
| 24 | /** The declaration a test makes when it lists only what it made itself. */ | |
| 25 | export const NOT_SCAN_MARK = "@not-a-tree-scan"; | |
| 26 | ||
| 27 | /** Lists a directory: a glob or `git ls-files`. */ | |
| 28 | const GLOB = /\bnew\s+(?:Bun\.)?Glob\s*\(|["'`]ls-files["'`]/; | |
ROOT is tested against the whole source, not against the readdir argument, so any file that uses import.meta.dir and has a readdir is a scan. Tie the root to the readdir call. |
||
| 29 | /** A `readdir` call, up to its opening parenthesis. */ | |
| 30 | const READDIR = /\breaddir(?:Sync)?\s*\(/g; | |
| 31 | /** A path expression's head: what a `join(`, `resolve(` or template opens with. */ | |
| 32 | const HEAD = String.raw`^\s*(?:(?:path\.)?(?:join|resolve)\s*\(\s*)?`; | |
| 33 | /** Names the repository's root: through the module's own place, or the working directory. */ | |
| 34 | const ROOT = /import\.meta\.(?:dir|dirname|url|path)\b|process\.cwd\s*\(/; | |
| 35 | /** A path that opens with a top-level directory of the repository, or `.`/`..`, by name. */ | |
| 36 | const TOP = new RegExp(`${HEAD}["'\`](?:\\.{1,2}|(?:\\./)?(?:src|test|docs|scripts))(?:/|["'\`])`); | |
| 37 | ||
| 38 | /** Whether `text` is a path into the repository, given the names already known to hold one. */ | |
| 39 | function intoRepo(text: string, rooted: ReadonlySet<string>): boolean { | |
| 40 | if (ROOT.test(text) || TOP.test(text)) return true; | |
| 41 | if (rooted.size === 0) return false; | |
| 42 | const names = [...rooted].map((name) => name.replace(/\$/g, "\\$")).join("|"); | |
| 43 | return new RegExp(`${HEAD}(?:\\$\\{\\s*)?(?:${names})(?![\\w$])`).test(text.replace(/^\s*`/, "")); | |
| 44 | } | |
| 45 | ||
| 46 | /** The text between the parenthesis at `open` and the one that closes it. */ | |
| 47 | function argumentAt(source: string, open: number): string { | |
| 48 | let depth = 0; | |
| 49 | for (let i = open; i < source.length; i++) { | |
| 50 | const c = source[i]; | |
| 51 | if (c === "(") depth++; | |
| 52 | else if (c === ")" && --depth === 0) return source.slice(open + 1, i); | |
| 53 | } | |
| 54 | return source.slice(open + 1); | |
| 55 | } | |
| 56 | ||
| 57 | /** | |
| 58 | * The names that hold a path into the repository: bound (or defaulted, as | |
| 59 | * a parameter) to an expression that names the root, or to a path built | |
| 60 | * from such a name. A parameter a | |
| 61 | * walk takes (a `function` or an arrow bound to a name) counts when a call | |
| 62 | * of it hands it one. | |
| 63 | */ | |
| 64 | function rootedNames(source: string): Set<string> { | |
| 65 | const bindings: { name: string; value: string }[] = []; | |
| 66 | for (const m of source.matchAll(/\b(?:const|let|var)\s+([\w$]+)\s*(?::[^=\n]+)?=\s*([^;\n]+)/g)) | |
| 67 | bindings.push({ name: m[1]!, value: m[2]! }); | |
| 68 | // A parameter's default: `function files(root = ".")`, `(dir: string = DIR) =>`. | |
| 69 | for (const m of source.matchAll(/[(,]\s*([\w$]+)\s*(?::\s*[\w$.<>[\]| ]+)?=(?![=>])\s*([^,)\n]+)/g)) | |
| 70 | bindings.push({ name: m[1]!, value: m[2]! }); | |
| 71 | const walkers = /\bfunction\s+([\w$]+)\s*\(\s*([\w$]+)|\b(?:const|let|var)\s+([\w$]+)\s*=\s*(?:async\s*)?\(\s*([\w$]+)/g; | |
| 72 | for (const w of source.matchAll(walkers)) { | |
| 73 | const [walker, param] = [(w[1] ?? w[3])!, (w[2] ?? w[4])!]; | |
| 74 | const calls = source.matchAll(new RegExp(`(?<![\\w$.])${walker.replace(/\$/g, "\\$")}\\s*\\(`, "g")); | |
| 75 | for (const call of calls) bindings.push({ name: param, value: argumentAt(source, call.index! + call[0].length - 1) }); | |
| 76 | } | |
| 77 | const rooted = new Set<string>(); | |
| 78 | for (let grew = true; grew; ) { | |
| 79 | grew = false; | |
| 80 | for (const { name, value } of bindings) | |
| 81 | if (!rooted.has(name) && intoRepo(value, rooted)) { | |
| 82 | rooted.add(name); | |
| 83 | grew = true; | |
| 84 | } | |
| 85 | } | |
| 86 | return rooted; | |
| 87 | } | |
| 88 | ||
| 89 | /** Whether a `readdir` in the source is handed a path into the repository. */ | |
| 90 | function readsRepoDir(source: string): boolean { | |
| 91 | let rooted: Set<string> | undefined; | |
| 92 | for (const call of source.matchAll(READDIR)) { | |
| 93 | const arg = argumentAt(source, call.index! + call[0].length - 1); | |
| 94 | rooted ??= rootedNames(source); | |
| 95 | if (intoRepo(arg, rooted)) return true; | |
| 96 | } | |
| 97 | return false; | |
| 98 | } | |
| 99 | ||
| 100 | /** Whether a comment line of the source holds `mark` (as a word: `@tree-scan` is not in `@not-a-tree-scan`). */ | |
| 101 | const declares = (source: string, mark: string): boolean => | |
| 102 | source | |
| 103 | .split("\n") | |
| 104 | .some((line) => /^\s*(?:\/\/|\/?\*)/.test(line) && new RegExp(`(?:^|[\\s*/])${mark}\\b`).test(line)); | |
| 105 | ||
| 106 | /** Whether a source declares itself a scan: {@link SCAN_MARK} on a comment line. */ | |
| 107 | export const declaresScan = (source: string): boolean => declares(source, SCAN_MARK); | |
| 108 | ||
| 109 | /** Whether a source declares it is no scan: {@link NOT_SCAN_MARK} on a comment line. */ | |
| 110 | export const declaresNoScan = (source: string): boolean => declares(source, NOT_SCAN_MARK); | |
| 111 | ||
| 112 | /** Whether a source lists the repository's tree. */ | |
| 113 | export const walksTree = (source: string): boolean => GLOB.test(source) || readsRepoDir(source); | |
| 114 | ||
| 115 | /** Why a source is a scan, or `undefined` when it is not one. */ | |
| 116 | export function scanOf(source: string): "declared" | "lists the tree" | undefined { | |
| 117 | if (declaresScan(source)) return "declared"; | |
| 118 | if (declaresNoScan(source)) return undefined; | |
| 119 | return walksTree(source) ? "lists the tree" : undefined; | |
| 120 | } | |
| 121 | ||
| 122 | /** | |
| 123 | * The tests that scan the tree, each with why: by its own source, or by a | |
| 124 | * `scripts/` module it imports directly (a scan's logic usually lives in | |
| 125 | * the script the check also runs). `read` answers `undefined` for a file it | |
| 126 | * cannot read; `imports` is a file's repository-relative imports. | |
| 127 | */ | |
| 128 | export function treeScans( | |
| 129 | tests: readonly string[], | |
| 130 | read: (file: string) => string | undefined, | |
| 131 | imports: (file: string) => readonly string[], | |
| 132 | ): Map<string, string> { | |
| 133 | const scripts = new Map<string, ReturnType<typeof scanOf>>(); | |
| 134 | const scriptScan = (file: string): ReturnType<typeof scanOf> => { | |
| 135 | if (!scripts.has(file)) { | |
| 136 | const source = read(file); | |
| 137 | scripts.set(file, source === undefined ? undefined : scanOf(source)); | |
| 138 | } | |
| 139 | return scripts.get(file); | |
| 140 | }; | |
| 141 | const found = new Map<string, string>(); | |
| 142 | for (const test of tests) { | |
| 143 | const source = read(test); | |
| 144 | if (source === undefined) continue; | |
| 145 | const own = scanOf(source); | |
| 146 | if (own !== undefined) { | |
| 147 | found.set(test, own === "declared" ? "declares a tree scan" : "lists the tree"); | |
| 148 | continue; | |
| 149 | } | |
| 150 | if (declaresNoScan(source)) continue; | |
| 151 | const via = imports(test).find((file) => file.startsWith("scripts/") && scriptScan(file) !== undefined); | |
| 152 | if (via !== undefined) found.set(test, `lists the tree through ${via}`); | |
| 153 | } | |
| 154 | return found; | |
| 155 | } |
src/checks/tiers.ts +11 −3
| @@ -20,7 +20,10 @@ export const QUICK_SECONDS = 1; | ||
| 20 | 20 | export const FAST_BUDGET_SECONDS = 5 * 60; |
| 21 | 21 | |
| 22 | 22 | /** |
| 23 | * The repo-wide rule tests (#7138): each reads every test file, every | |
| 23 | * The repo-wide rule tests (#7138), the floor that needs no tree to read | |
| 24 | * (`outside_diff.ts` judges by it from a run's log alone). Every other test | |
| 25 | * that reads the tree is found by what it reads (`scans.ts`, #10304), so a | |
| 26 | * new scan needs no line here. Each reads every test file, every | |
| 24 | 27 | * migration or the whole README, so no diff "touches" them by coverage, |
| 25 | 28 | * import or name, yet any file can break them. They run in every fast |
| 26 | 29 | * tier, whatever else the diff holds (a diff of only the check's own |
| @@ -111,6 +114,8 @@ export interface FastInput { | ||
| 111 | 114 | readonly learned?: readonly string[]; |
| 112 | 115 | /** Measured: the `src/**` files each test executed (`scripts/test-impact.json`). */ |
| 113 | 116 | readonly impact?: Impact; |
| 117 | /** The tests that read the tree, with why (`treeScans`, #10304): every diff can break one. */ | |
| 118 | readonly scans?: ReadonlyMap<string, string>; | |
| 114 | 119 | } |
| 115 | 120 | |
| 116 | 121 | export interface FastPlan { |
| @@ -127,7 +132,8 @@ export interface FastPlan { | ||
| 127 | 132 | /** |
| 128 | 133 | * The fast tier for a diff: a changed test; a test importing a changed file |
| 129 | 134 | * directly or through up to two hops of source; a test whose name matches a |
| 130 | * changed source path; every repo-wide rule test; whatever the pull | |
| 135 | * changed source path; every repo-wide rule test and every tree scan | |
| 136 | * (`scans.ts`); whatever the pull | |
| 131 | 137 | * learned; and every test known to take under a second. A test with no recorded time is not known to be quick, so it runs only when |
| 132 | 138 | * something above picks it. |
| 133 | 139 | */ |
| @@ -176,8 +182,10 @@ export function fastSet(input: FastInput): FastPlan { | ||
| 176 | 182 | const changed = new Set(diffed); |
| 177 | 183 | for (const file of changed) take(file, "changed"); |
| 178 | 184 | // The check's own tables can break no rule: a diff of only those runs none (#3579). |
| 179 | if (diffed.some((f) => !f.endsWith("test-times.json") && !f.endsWith("test-impact.json"))) | |
| 185 | if (diffed.some((f) => !f.endsWith("test-times.json") && !f.endsWith("test-impact.json"))) { | |
| 180 | 186 | for (const test of RULE_TESTS) take(test, "repo-wide rule"); |
| 187 | for (const [test, why] of input.scans ?? []) take(test, `tree scan: ${why}`); | |
| 188 | } | |
| 181 | 189 | // What a guess looks for. A test the map has measured already answered |
| 182 | 190 | // for every change the map can see, by what it executes; it is guessed |
| 183 | 191 | // only against the changes the map cannot see, so one unreached file |
test/design-audit.test.ts +2
| @@ -3,6 +3,8 @@ | ||
| 3 | 3 | * route the router does not have (#2709). The router is read as text: the |
| 4 | 4 | * static GET paths in src/http/server.ts and settings_routes.ts, less the |
| 5 | 5 | * ones that are not pages. |
| 6 | * | |
| 7 | * @tree-scan: reads docs/design/audit.md and the router as text. | |
| 6 | 8 | */ |
| 7 | 9 | import { expect, test } from "bun:test"; |
| 8 | 10 | import { readFileSync } from "node:fs"; |
test/readme-cap.test.ts +2
| @@ -1,6 +1,8 @@ | ||
| 1 | 1 | /** |
| 2 | 2 | * README.md stops growing (#6380): a feature's how-to goes in its own file |
| 3 | 3 | * under docs/features/, so pulls stop colliding on one append-only page. |
| 4 | * | |
| 5 | * @tree-scan: reads README.md and docs/features/README.md. | |
| 4 | 6 | */ |
| 5 | 7 | |
| 6 | 8 | import { expect, test } from "bun:test"; |
test/red-on-main.test.ts added +122
| @@ -0,0 +1,122 @@ | ||
| 1 | /** | |
| 2 | * A pull is not red for main's red (#10304): when a fast tier's red test is | |
| 3 | * red on the pull's base too, with the same tests, the check says so and is | |
| 4 | * green; any other red is the pull's. Once end to end through | |
| 5 | * `scripts/test-shards.ts`, in a scratch repository of trivial tests. | |
| 6 | */ | |
| 7 | import { describe, expect, test } from "bun:test"; | |
| 8 | import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; | |
| 9 | import { tmpdir } from "node:os"; | |
| 10 | import { join } from "node:path"; | |
| 11 | import { EXCLUDED } from "../scripts/gate.ts"; | |
| 12 | import { askBase, baseVerdict, BASE_RERUN_CAP, type OnBase } from "../src/checks/base_red.ts"; | |
| 13 | ||
| 14 | const BASE = "0123456789abcdef0123456789abcdef01234567"; | |
| 15 | const red = (...failing: string[]) => ({ did: "red" as const, failing }); | |
| 16 | ||
| 17 | describe("whose red it is", () => { | |
| 18 | test("a file red on the base with the same tests is main's, and clears the run", () => { | |
| 19 | const got = baseVerdict( | |
| 20 | new Map([["test/a.test.ts", ["x"]]]), | |
| 21 | new Map([["test/a.test.ts", red("x", "y")]]), | |
| 22 | BASE, | |
| 23 | ); | |
| 24 | expect(got.cleared).toBe(true); | |
| 25 | expect(got.lines).toEqual([ | |
| 26 | "test/a.test.ts is red on main at 0123456 too, the same tests: not counted against this pull", | |
| 27 | "green: every red test is red on main at 0123456 too, so it is main's, not this pull's (test/a.test.ts)", | |
| 28 | ]); | |
| 29 | }); | |
| 30 | ||
| 31 | test("a second break in a file main already has red is the pull's", () => { | |
| 32 | const got = baseVerdict( | |
| 33 | new Map([["test/a.test.ts", ["x", "new break"]]]), | |
| 34 | new Map([["test/a.test.ts", red("x")]]), | |
| 35 | BASE, | |
| 36 | ); | |
| 37 | expect(got.cleared).toBe(false); | |
| 38 | expect(got.lines).toEqual([ | |
| 39 | `test/a.test.ts is red on main at 0123456 too, but "new break" passes there: this pull's red`, | |
| 40 | ]); | |
| 41 | }); | |
| 42 | ||
| 43 | test("a base red that names no test cannot be compared, and keeps the run red", () => { | |
| 44 | const got = baseVerdict(new Map([["test/a.test.ts", ["x"]]]), new Map([["test/a.test.ts", red()]]), BASE); | |
| 45 | expect(got.cleared).toBe(false); | |
| 46 | expect(got.lines[0]).toContain("names no failing test there to compare: this pull's red"); | |
| 47 | }); | |
| 48 | ||
| 49 | test("one red file green on the base, absent from it or not run keeps the run red", () => { | |
| 50 | for (const did of [{ did: "green" as const }, { did: "absent" as const }, undefined]) { | |
outdated
Each case writes every present test file into a fresh repo and spawns bun test-shards. Slow and flaky-prone; keep the minimum of e2e cases. |
||
| 51 | const onBase = new Map<string, OnBase>([["test/a.test.ts", red("x")]]); | |
| 52 | if (did !== undefined) onBase.set("test/b.test.ts", did); | |
| 53 | const got = baseVerdict(new Map([["test/a.test.ts", ["x"]], ["test/b.test.ts", ["z"]]]), onBase, BASE); | |
| 54 | expect(got.cleared).toBe(false); | |
| 55 | expect(got.lines[0]).toContain("test/a.test.ts is red on main"); | |
| 56 | expect(got.lines[1]).toMatch(/^test\/b\.test\.ts .*: this pull's red$/); | |
| 57 | } | |
| 58 | }); | |
| 59 | ||
| 60 | test("the base is not asked about a red with no file or test, a changed file, or too many", () => { | |
| 61 | const one = (file: string, ...tests: string[]) => new Map([[file, tests]]); | |
| 62 | expect(askBase({ red: undefined, changed: [] })).toBeUndefined(); | |
| 63 | expect(askBase({ red: new Map(), changed: [] })).toBeUndefined(); | |
| 64 | expect(askBase({ red: one("test/a.test.ts"), changed: ["README.md"] })).toBeUndefined(); | |
| 65 | expect(askBase({ red: one("test/a.test.ts", "x"), changed: ["test/a.test.ts"] })).toBeUndefined(); | |
| 66 | const many = new Map(Array.from({ length: BASE_RERUN_CAP + 1 }, (_, i) => [`test/t${i}.test.ts`, ["x"]])); | |
| 67 | expect(askBase({ red: many, changed: ["README.md"] })).toBeUndefined(); | |
| 68 | expect(askBase({ red: one("test/a.test.ts", "x"), changed: ["README.md"] })).toEqual(["test/a.test.ts"]); | |
| 69 | }); | |
| 70 | }); | |
| 71 | ||
| 72 | describe("the fast tier asks main", () => { | |
| 73 | const script = new URL("../scripts/test-shards.ts", import.meta.url).pathname; | |
| 74 | const RED = "test/unreached.test.ts"; | |
| 75 | const pass = `import { test } from "bun:test";\ntest("ok", () => {});\n`; | |
| 76 | const fail = `import { expect, test } from "bun:test";\ntest("red", () => { expect(1).toBe(2); });\n`; | |
| 77 | const git = (dir: string, ...args: string[]): number => | |
| 78 | Bun.spawnSync(["git", "-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", ...args], { | |
| 79 | cwd: dir, | |
| 80 | stdout: "ignore", | |
| 81 | stderr: "ignore", | |
| 82 | }).exitCode; | |
| 83 | const env: Record<string, string | undefined> = { ...process.env, FORGE_CHECK_SHARDS: "2", FORGE_CELL_CPUS: "2" }; | |
| 84 | for (const name of Object.keys(env)) if (name.startsWith("FORGE_CHECK_") && name !== "FORGE_CHECK_SHARDS") delete env[name]; | |
| 85 | ||
| 86 | /** Main with RED as `onMain`, a pull that changes `pull`; the fast check of the pull. */ | |
| 87 | const check = async (onMain: string, pull: Record<string, string>) => { | |
| 88 | const dir = mkdtempSync(join(tmpdir(), "red-on-main-")); | |
| 89 | try { | |
| 90 | mkdirSync(join(dir, "test")); | |
| 91 | // The gate refuses a tree its EXCLUDED list names files missing from; they never run. | |
| 92 | for (const { file } of EXCLUDED) writeFileSync(join(dir, file), pass); | |
| 93 | writeFileSync(join(dir, RED), onMain); | |
| 94 | writeFileSync(join(dir, "test/plain.test.ts"), pass); | |
| 95 | writeFileSync(join(dir, "README.md"), "# scratch\n"); | |
| 96 | expect(git(dir, "init", "-q")).toBe(0); | |
| 97 | expect(git(dir, "add", "-A")).toBe(0); | |
| 98 | expect(git(dir, "commit", "-qm", "main")).toBe(0); | |
| 99 | for (const [file, text] of Object.entries(pull)) writeFileSync(join(dir, file), text); | |
| 100 | expect(git(dir, "add", "-A")).toBe(0); | |
| 101 | expect(git(dir, "commit", "-qm", "the pull")).toBe(0); | |
| 102 | const child = Bun.spawn(["bun", script, "--tier", "fast", "--diff", "HEAD~1..HEAD"], { | |
| 103 | cwd: dir, | |
| 104 | env, | |
| 105 | stdin: "ignore", | |
| 106 | stdout: "pipe", | |
| 107 | stderr: "pipe", | |
| 108 | }); | |
| 109 | const [out] = await Promise.all([new Response(child.stdout).text(), new Response(child.stderr).text()]); | |
| 110 | return { code: await child.exited, out }; | |
| 111 | } finally { | |
| 112 | rmSync(dir, { recursive: true, force: true }); | |
| 113 | } | |
| 114 | }; | |
| 115 | ||
| 116 | test("a README-only pull opened while main fails a test is green, and names main's red", async () => { | |
| 117 | const got = await check(fail, { "README.md": "# scratch, changed\n" }); | |
| 118 | expect(got.out).toContain(`${RED} is red on main at HEAD~1 too, the same tests: not counted against this pull`); | |
| 119 | expect(got.out).toContain("green: every red test is red on main at HEAD~1 too"); | |
| 120 | expect(got.code).toBe(0); | |
| 121 | }, 60_000); | |
| 122 | }); |
test/release-boot.test.ts +2
| @@ -4,6 +4,8 @@ | ||
| 4 | 4 | * starts the tree on a scratch data directory and wants `/api/v1/health` to |
| 5 | 5 | * answer as the stamped version. 0.7.166's image crashed at import; a tree |
| 6 | 6 | * that does the same is refused here, naming the error. |
| 7 | * | |
| 8 | * @tree-scan: copies all of src/ and boots the copy. | |
| 7 | 9 | */ |
| 8 | 10 | |
| 9 | 11 | import { afterAll, beforeAll, expect, test } from "bun:test"; |
test/schema-migrations.test.ts +2
| @@ -7,6 +7,8 @@ | ||
| 7 | 7 | * twice, a branch whose migration was renumbered by its rebase, and a branch |
| 8 | 8 | * that adds a table the data already has. A failure names its statement in |
| 9 | 9 | * one line, which is what the deploy's summary on the pull shows. |
| 10 | * | |
| 11 | * @tree-scan: reads every migration and its pin. | |
| 10 | 12 | */ |
| 11 | 13 | |
| 12 | 14 | import { afterAll, describe, expect, test } from "bun:test"; |
test/shards-run.test.ts +3
| @@ -3,6 +3,9 @@ | ||
| 3 | 3 | * it, and it spawns `bun test`, so each costs a few hundred milliseconds. |
| 4 | 4 | * The pure parts it is built from (`scripts/shards.ts`) are proved in |
| 5 | 5 | * test/shards.test.ts, which stays fast because it spawns nothing. |
| 6 | * | |
| 7 | * @not-a-tree-scan: it lists only its own temporary directories, and what | |
| 8 | * it takes from scripts/gate.ts does not read the tree. | |
| 6 | 9 | */ |
| 7 | 10 | import { afterAll, describe, expect, test } from "bun:test"; |
| 8 | 11 | import { mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs"; |
test/shards.test.ts +25
| @@ -16,6 +16,7 @@ import { | ||
| 16 | 16 | CRASH_RERUNS, |
| 17 | 17 | quoteCrash, |
| 18 | 18 | failingFiles, |
| 19 | failingTests, | |
| 19 | 20 | onlyTestFailures, |
| 20 | 21 | WORKER_MEMORY_BYTES, |
| 21 | 22 | CELL_CPU_SHARE, |
| @@ -729,6 +730,30 @@ describe("failingFiles", () => { | ||
| 729 | 730 | }); |
| 730 | 731 | }); |
| 731 | 732 | |
| 733 | describe("failingTests", () => { | |
| 734 | test("names each file's failing tests without their timing, not the summary's repeats", () => { | |
| 735 | const report = [ | |
| 736 | "test/b.test.ts:", | |
| 737 | "(pass) fine [0.1ms]", | |
| 738 | "(fail) outer > one [0.12ms]", | |
| 739 | "(fail) two [1.50s]", | |
| 740 | "test/c.test.ts:", | |
| 741 | "(fail) three", | |
| 742 | "2 tests failed:", | |
| 743 | "(fail) outer > one [0.12ms]", | |
| 744 | " 1 pass", | |
| 745 | " 3 fail", | |
| 746 | "Ran 4 tests across 2 files. [9.00ms]", | |
| 747 | ].join("\n"); | |
| 748 | expect(failingTests(report)).toEqual( | |
| 749 | new Map([ | |
| 750 | ["test/b.test.ts", ["outer > one", "two"]], | |
| 751 | ["test/c.test.ts", ["three"]], | |
| 752 | ]), | |
| 753 | ); | |
| 754 | }); | |
| 755 | }); | |
| 756 | ||
| 732 | 757 | describe("onlyTestFailures", () => { |
| 733 | 758 | const ran = "Ran 3 tests across 2 files. [9ms]"; |
| 734 | 759 | test("failed tests and a summary: true", () => { |
test/spawn-pipes.test.ts +2
| @@ -3,6 +3,8 @@ | ||
| 3 | 3 | * stdio nobody drains leaks native memory in a long-lived Bun server. Every |
| 4 | 4 | * Bun.spawn / spawnSync in src/ either ignores its unused streams or is |
| 5 | 5 | * listed here with the reason it reads each pipe to its end. |
| 6 | * | |
| 7 | * @tree-scan: reads every file under src/. | |
| 6 | 8 | */ |
| 7 | 9 | |
| 8 | 10 | import { expect, test } from "bun:test"; |
test/tree-scans.test.ts added +106
| @@ -0,0 +1,106 @@ | ||
| 1 | /** | |
| 2 | * A test that reads the tree is found by what it reads, not by a hand list | |
| 3 | * (#10304): a new scan runs in every pull's fast tier the day it is | |
| 4 | * written, so the pull that breaks it is red on its own check, not main. | |
| 5 | */ | |
| 6 | import { describe, expect, test } from "bun:test"; | |
| 7 | import { gateFiles } from "../scripts/gate.ts"; | |
| 8 | import { importsReader, scansIn } from "../scripts/tier-input.ts"; | |
| 9 | import { declaresNoScan, declaresScan, treeScans, walksTree } from "../src/checks/scans.ts"; | |
| 10 | import { fastSet, RULE_TESTS } from "../src/checks/tiers.ts"; | |
| 11 | ||
| 12 | describe("what a scan reads", () => { | |
| 13 | test("a glob or git ls-files lists the tree", () => { | |
| 14 | expect(walksTree(`for (const f of new Glob("src/**/*.ts").scanSync(".")) {}`)).toBe(true); | |
| 15 | expect(walksTree(`Bun.spawnSync(["git", "ls-files"])`)).toBe(true); | |
| 16 | }); | |
| 17 | ||
| 18 | test("a readdir lists the tree only when it is rooted at the repository", () => { | |
| 19 | expect(walksTree(`readdirSync(join(import.meta.dir, "..", "src"))`)).toBe(true); | |
| 20 | expect(walksTree(`readdirSync("test").filter(f => f.endsWith(".ts"))`)).toBe(true); | |
| 21 | expect(walksTree(`readdirSync(process.cwd())`)).toBe(true); | |
| 22 | expect(walksTree(`const d = mkdtempSync(join(tmpdir(), "x")); readdirSync(d)`)).toBe(false); | |
| 23 | }); | |
| 24 | ||
| 25 | test("the root counts only when it is what the readdir is handed", () => { | |
| 26 | // The module's own place, used for a fixture, beside a readdir of a temp dir: no scan. | |
| 27 | const fixture = `const jpeg = readFileSync(\`\${import.meta.dir}/fixtures/a.jpg\`);\nconst dir = join(dataDir, "looks");\nreaddirSync(dir);`; | |
| 28 | expect(walksTree(fixture)).toBe(false); | |
| 29 | expect(walksTree(`const repo = join(import.meta.dir, "..");\nreaddirSync(join(data, ".taking"));`)).toBe(false); | |
| 30 | // Through a name, a walk's parameter, or a parameter's default: a scan. | |
| 31 | expect(walksTree(`const ROOT = join(import.meta.dir, "..");\nconst DIR = join(ROOT, "src");\nreaddirSync(DIR);`)).toBe(true); | |
| 32 | const walk = `const walk = (d: string): string[] => readdirSync(d).flatMap((n) => walk(join(d, n)));\nwalk(join(import.meta.dir, "..", "src"));`; | |
| 33 | expect(walksTree(walk)).toBe(true); | |
| 34 | expect(walksTree(`const walk = (d: string) => readdirSync(d);\nwalk(mkdtempSync("x"));`)).toBe(false); | |
| 35 | expect(walksTree(`export function files(root = ".") { return readdirSync(\`\${root}/test\`); }`)).toBe(true); | |
| 36 | }); | |
| 37 | ||
| 38 | test("a scan the reading cannot see declares itself; a false one says it is not", () => { | |
| 39 | expect(declaresScan(`/**\n * @tree-scan: reads README.md.\n */`)).toBe(true); | |
| 40 | expect(declaresScan(`// @tree-scan`)).toBe(true); | |
| 41 | expect(declaresScan(`const s = "@tree-scan";`)).toBe(false); | |
| 42 | expect(declaresScan(` * @not-a-tree-scan: its own temp dirs.`)).toBe(false); | |
| 43 | expect(declaresNoScan(` * @not-a-tree-scan: its own temp dirs.`)).toBe(true); | |
| 44 | }); | |
| 45 | }); | |
| 46 | ||
| 47 | describe("the tests that scan", () => { | |
| 48 | const files: Record<string, string> = { | |
| 49 | "test/new-scan.test.ts": `readdirSync(join(import.meta.dir, "../src"));`, | |
| 50 | "test/via.test.ts": `import { audit } from "../scripts/audit.ts";`, | |
| 51 | "scripts/audit.ts": `export const audit = () => [...new Glob("**/*.ts").scanSync(".")];`, | |
| 52 | "test/readme.test.ts": `// @tree-scan: reads README.md\nreadFileSync("README.md");`, | |
| 53 | "test/temp.test.ts": `// @not-a-tree-scan: lists its own temp dir\nreaddirSync(join(import.meta.dir, "tmp"));`, | |
| 54 | "test/plain.test.ts": `import { a } from "../src/a.ts";`, | |
| 55 | }; | |
| 56 | const imports = (file: string): string[] => | |
| 57 | file === "test/via.test.ts" ? ["scripts/audit.ts"] : file === "test/plain.test.ts" ? ["src/a.ts"] : []; | |
| 58 | const tests = Object.keys(files).filter((f) => f.startsWith("test/")); | |
| 59 | const scans = treeScans(tests, (f) => files[f], imports); | |
| 60 | ||
| 61 | test("are found by their own source, a script they import, or their declaration", () => { | |
| 62 | expect(Object.fromEntries(scans)).toEqual({ | |
| 63 | "test/new-scan.test.ts": "lists the tree", | |
| 64 | "test/via.test.ts": "lists the tree through scripts/audit.ts", | |
| 65 | "test/readme.test.ts": "declares a tree scan", | |
| 66 | }); | |
| 67 | }); | |
| 68 | ||
| 69 | test("run in the fast tier of a diff no import or coverage ties to them", () => { | |
| 70 | const plan = fastSet({ | |
| 71 | changed: ["src/brand-new.ts"], | |
| 72 | tests, | |
| 73 | imports, | |
| 74 | times: new Map(), | |
| 75 | impact: new Map([["test/plain.test.ts", ["src/a.ts"]]]), | |
| 76 | scans, | |
| 77 | }); | |
| 78 | for (const scan of scans.keys()) expect(plan.files).toContain(scan); | |
| 79 | expect(plan.why.get("test/new-scan.test.ts")).toBe("tree scan: lists the tree"); | |
| 80 | expect(plan.files).not.toContain("test/temp.test.ts"); | |
| 81 | }); | |
| 82 | ||
| 83 | test("do not run for a diff of only the check's own tables", () => { | |
| 84 | const plan = fastSet({ | |
| 85 | changed: ["scripts/test-times.json"], | |
| 86 | tests, | |
| 87 | imports, | |
| 88 | times: new Map(), | |
| 89 | scans, | |
| 90 | }); | |
| 91 | expect(plan.files).toEqual([]); | |
| 92 | }); | |
| 93 | }); | |
| 94 | ||
| 95 | describe("this repository's scans", () => { | |
| 96 | const gate = gateFiles(); | |
| 97 | const scans = scansIn(gate, importsReader()); | |
| 98 | ||
| 99 | test("include every repo-wide rule test, so the hand list is no longer what finds them", () => { | |
| 100 | for (const rule of RULE_TESTS) expect([rule, scans.has(rule)]).toEqual([rule, true]); | |
| 101 | }); | |
| 102 | ||
| 103 | test("an unimported new module reaches test/unreached.test.ts as a scan", () => { | |
| 104 | expect(scans.get("test/unreached.test.ts")).toBe("lists the tree through scripts/unreached.ts"); | |
| 105 | }); | |
| 106 | }); |
Red on the base is not the same red. Clear only when the pull's failing test names are a subset of the base's; otherwise a second break in an already-red file is hidden.