A red main fails every pull's check: tell a pull's own red from main's, and keep main from landing red #10309

A pull's fast check is no longer red because of a test that is already red on main. The fast tier now also finds tests that read the whole tree by what they read, so the next one someone writes doesn't depend on being added to a hand list.

Tree scans found by what they read (src/checks/scans.ts)

• A test counts as a tree scan when its own source, or a scripts/ module it imports directly, lists the repository: a Glob, git ls-files, or a readdir rooted at the repo.
• A test that reads fixed files (README, migrations) declares itself with @tree-scan in its header comment. Four of the existing RULE_TESTS needed that: schema-migrations, readme-cap, design-audit and spawn-pipes.
• test/shards-run.test.ts was wrongly detected (it only lists its own temp directories), so it says @not-a-tree-scan and why.
• fastSet (src/checks/tiers.ts) takes these scans and runs every one on every diff except one that touches only the timing or impact tables. scripts/tier-input.ts reads them for both scripts/test-shards.ts and scripts/fast.ts, so the check and the pre-push preview pick the same tests.
• On this repository 64 tests are picked as scans, including all ten in RULE_TESTS, and test/unreached.test.ts is reached through scripts/unreached.ts. So a pull that adds a module nothing imports is red on its own fast check.

Main's red is not the pull's (src/checks/base_red.ts, wired into scripts/test-shards.ts)

• When the fast tier is red, test-shards.ts collects the failing test files. If there are five or fewer and the pull changed none of them, each is run once more on the base's tree: a git worktree on the shard scratch disk, sharing the checkout's node_modules.
• If every one is red there too, the log says test/x.test.ts is red on main at <base> too: not counted against this pull, followed by a green: … line, and the check exits 0.
• The pull stays red when a file passes on main, doesn't exist on main, or couldn't be checked out, when a failure names no file (a crash or load error in a shard of several files), or when the pull changed the red file.

What to look at

• Fast-tier cost: about 64 tree scans now join every fast tier. I couldn't estimate the extra time locally, because the committed times table lists only 65 of 2,407 test files. Some of these are slower than one second; apps-token is about 20 s. A test detected wrongly can opt out with @not-a-tree-scan.
• RULE_TESTS is kept: src/checks/outside_diff.ts decides from a run's log alone, without the tree, and still needs a fixed list. A new scan needs no line there. I did not move the hand list into the impact map; Bun's node:fs named imports can't be wrapped to record the files a test opens (I tried), so detection reads the test's source instead.
• Fast tier only: the land gate and red_main.ts's hold are unchanged.
• I couldn't update the plan on the issue: the keep_plan tool wasn't available in this session.

Tests (each would fail without the change)

• test/tree-scans.test.ts: covers detection, the @tree-scan / @not-a-tree-scan declarations, and that a scan no import reaches runs in the fast tier. It also checks every RULE_TESTS entry is found on this repository.
• test/red-on-main.test.ts: runs test-shards.ts --tier fast end to end in a scratch repo. A README-only pull opened while main fails test/unreached.test.ts is green and names main's red. A pull that breaks that test through README while main passes it stays red. A pull that edits the red test itself stays red without asking main.
• bun scripts/gate.ts, bun x tsc --noEmit and 19 covering test files (289 tests) pass. scripts/fast.ts exits 2 for this diff, because it changes scripts/, so I chose those 19 by hand.

Try: on a forge whose main fails test/unreached.test.ts, open a pull that changes only README.md and read its check: green, with test/unreached.test.ts is red on main at <base> too: not counted against this pull.

🤖 Generated with Claude Code

Closes #10304

A pull's fast check is no longer red because of a test that is already red on main. The fast tier now also finds tests that read the whole tree by what they read, so the next one someone writes doesn't depend on being added to a hand list.

Tree scans found by what they read (src/checks/scans.ts)

• A test counts as a tree scan when its own source, or a scripts/ module it imports directly, lists the repository: a Glob, git ls-files, or a readdir rooted at the repo.
• A test that reads fixed files (README, migrations) declares itself with @tree-scan in its header comment. Four of the existing RULE_TESTS needed that: schema-migrations, readme-cap, design-audit and spawn-pipes.
• test/shards-run.test.ts was wrongly detected (it only lists its own temp directories), so it says @not-a-tree-scan and why.
• fastSet (src/checks/tiers.ts) takes these scans and runs every one on every diff except one that touches only the timing or impact tables. scripts/tier-input.ts reads them for both scripts/test-shards.ts and scripts/fast.ts, so the check and the pre-push preview pick the same tests.
• On this repository 64 tests are picked as scans, including all ten in RULE_TESTS, and test/unreached.test.ts is reached through scripts/unreached.ts. So a pull that adds a module nothing imports is red on its own fast check.

Main's red is not the pull's (src/checks/base_red.ts, wired into scripts/test-shards.ts)

• When the fast tier is red, test-shards.ts collects the failing test files. If there are five or fewer and the pull changed none of them, each is run once more on the base's tree: a git worktree on the shard scratch disk, sharing the checkout's node_modules.
• If every one is red there too, the log says test/x.test.ts is red on main at <base> too: not counted against this pull, followed by a green: … line, and the check exits 0.
• The pull stays red when a file passes on main, doesn't exist on main, or couldn't be checked out, when a failure names no file (a crash or load error in a shard of several files), or when the pull changed the red file.

What to look at

• Fast-tier cost: about 64 tree scans now join every fast tier. I couldn't estimate the extra time locally, because the committed times table lists only 65 of 2,407 test files. Some of these are slower than one second; apps-token is about 20 s. A test detected wrongly can opt out with @not-a-tree-scan.
• RULE_TESTS is kept: src/checks/outside_diff.ts decides from a run's log alone, without the tree, and still needs a fixed list. A new scan needs no line there. I did not move the hand list into the impact map; Bun's node:fs named imports can't be wrapped to record the files a test opens (I tried), so detection reads the test's source instead.
• Fast tier only: the land gate and red_main.ts's hold are unchanged.
• I couldn't update the plan on the issue: the keep_plan tool wasn't available in this session.

Tests (each would fail without the change)

• test/tree-scans.test.ts: covers detection, the @tree-scan / @not-a-tree-scan declarations, and that a scan no import reaches runs in the fast tier. It also checks every RULE_TESTS entry is found on this repository.
• test/red-on-main.test.ts: runs test-shards.ts --tier fast end to end in a scratch repo. A README-only pull opened while main fails test/unreached.test.ts is green and names main's red. A pull that breaks that test through README while main passes it stays red. A pull that edits the red test itself stays red without asking main.
• bun scripts/gate.ts, bun x tsc --noEmit and 19 covering test files (289 tests) pass. scripts/fast.ts exits 2 for this diff, because it changes scripts/, so I chose those 19 by hand.

Try: on a forge whose main fails test/unreached.test.ts, open a pull that changes only README.md and read its check: green, with test/unreached.test.ts is red on main at <base> too: not counted against this pull.

🤖 Generated with Claude Code

A pull's fast check is no longer red because of a test that is already red on main, as long as main fails that test the same way. The fast tier also finds tests that read the whole tree by what they read, so a new one doesn't have to be added to a hand list.

Tree scans found by what they read (src/checks/scans.ts)

• A test counts as a tree scan when its own source, or a scripts/ module it imports directly, lists the repository: a Glob, git ls-files, or a readdir handed a path into the repo.
• That path is read from the readdir call's own argument. It counts when it is import.meta.dir/url, process.cwd(), a literal top-level directory ("src", "./test/", "."), or a name bound to a path built from one of those. The binding can be a const, a parameter's default (function files(root = ".")), or the argument a recursive walk is called with.
• Using import.meta.dir somewhere else in the file no longer makes it a scan. That dropped four tests that only list their own temp directories: attach-look, release-launch, release-preflight and restore-drill-script.
• A test that reads fixed files declares itself with @tree-scan: schema-migrations, readme-cap, design-audit, spawn-pipes, and now release-boot, which copies src/ with cpSync and boots the copy.
• test/shards-run.test.ts opts out with @not-a-tree-scan, because it is otherwise picked up through scripts/gate.ts.
• fastSet (src/checks/tiers.ts) runs every scan on every diff, except a diff that touches only the timing or impact tables. scripts/tier-input.ts reads the scans for both scripts/test-shards.ts and scripts/fast.ts, so the check and the pre-push preview pick the same tests.
• 62 tests are scans on this repository, including all ten in RULE_TESTS. test/unreached.test.ts is reached through scripts/unreached.ts, so a pull that adds a module nothing imports is red on its own fast check.

Main's red is not the pull's (src/checks/base_red.ts, wired into scripts/test-shards.ts)

• When the fast tier is red, test-shards.ts collects each red file with the names of its failing tests (failingTests in scripts/shards.ts).
• If there are five or fewer such files and the pull changed none of them, each is run once more on the base's tree. That run uses a git worktree on the shard scratch disk that shares the checkout's node_modules.
• A file counts as main's only when every test failing on the pull also fails on the base. Then the log says test/x.test.ts is red on main at <base> too, the same tests: not counted against this pull, and when every red file is main's the check exits 0.
• A second break in a file main already has red stays the pull's. The log names it: … is red on main at <base> too, but "readme" passes there: this pull's red.
• The pull also stays red when:

- the file passes on main, doesn't exist on main, or couldn't be checked out; - a red on either side names no test (a crash or a load error); - the pull changed the red file.

What to look at

• Fast-tier cost: about 62 tree scans now join every fast tier. The committed times table doesn't cover most of them, so I couldn't estimate the extra time; apps-token alone is about 20 s.
• Detection reads source text: it is a regex reading of the source, not a parser, and it doesn't follow scope, so a name reused in two scopes can mislead it. @tree-scan and @not-a-tree-scan are the overrides when it guesses wrong.
• RULE_TESTS is kept: src/checks/outside_diff.ts decides from a run's log alone and still needs a fixed list.
• Fast tier only: the land gate and red_main.ts's hold are unchanged.

Tests

• test/tree-scans.test.ts covers:

- a root used elsewhere in the file, next to a readdir of a temp dir, is not a scan; - a root reached through a name, a walk's parameter or a parameter's default is; - every RULE_TESTS entry is found on this repository.

• test/red-on-main.test.ts:

- Unit cases: same red clears; a new failing name in an already-red file stays red; a base red with no names stays red; green, absent and unrun files stay red; and the cases where main is not asked. - One end-to-end case through test-shards.ts --tier fast: a README-only pull opened while main fails test/unreached.test.ts is green. The scratch repo now holds only the EXCLUDED stubs plus two tests, and the case runs in about 0.5 s.

• test/shards.test.ts: failingTests drops the timing suffix and the summary's repeated (fail) lines.
• bun scripts/gate.ts and bun x tsc --noEmit pass, and so do 31 covering test files (436 tests). scripts/fast.ts exits 2 for this diff, so I picked those files by hand.
• I also checked the second-break case end to end once, with a throwaway test that I did not commit: the pull stayed red and the log named the test.

Try: on a forge whose main fails test/unreached.test.ts, open a pull that changes only README.md. Its check is green and says the file is red on main too, with the same tests.

🤖 Generated with Claude Code

This pull's fast check no longer goes red because of a test that main already fails, as long as main fails it the same way. The fast tier also finds tests that read the whole tree by looking at what each one reads, so a new one no longer has to be added to a hand list.

All three review points are answered by the two commits already on this branch (34ba4dfed, 4b75a7efa), and nothing was pushed after them. I re-ran the checks on that head: bun scripts/gate.ts and bun x tsc --noEmit pass, and so do red-on-main, tree-scans, shards and shards-run (102 tests). The pull's last check on that head was green, but main has moved since, so the forge marks it stale.

• "Red on the base is not the same red" — fixed. A red file now carries the names of its failing tests, and it counts as main's only when every test failing on the pull also fails on main.
• "ROOT is tested against the whole source" — fixed. The root has to be the readdir call's own argument.
• "Keep the minimum of e2e cases" — fixed. One end-to-end case is left, and its scratch repo holds only the EXCLUDED stubs and two tests.

Tree scans found by what they read (src/checks/scans.ts)

• A test is a tree scan when its own source, or a scripts/ module it imports directly, lists the repository. Three things count: a Glob, git ls-files, or a readdir given a path into the repo.
• That path is read from the readdir call's own argument. It counts when it is one of these, or a name bound to a path built from one (a const, a parameter's default, or the argument a recursive walk is called with):

- import.meta.dir or import.meta.url - process.cwd() - a literal top-level directory such as "src" or "."

• With that change, four tests that list only their own temp directories are no longer scans: attach-look, release-launch, release-preflight and restore-drill-script.
• A test that reads fixed files declares itself with @tree-scan: schema-migrations, readme-cap, design-audit, spawn-pipes and release-boot.
• test/shards-run.test.ts opts out with @not-a-tree-scan, because it would otherwise be picked up through scripts/gate.ts.
• fastSet (src/checks/tiers.ts) runs every scan on every diff, except one that touches only the timing or impact tables.
• scripts/tier-input.ts feeds both scripts/test-shards.ts and scripts/fast.ts, so the check and the pre-push preview pick the same tests.
• 62 tests are scans on this repository, including all ten in RULE_TESTS.

Main's red is not the pull's (src/checks/base_red.ts, wired into scripts/test-shards.ts)

• When the fast tier is red, each red file is collected with the names of its failing tests (failingTests in scripts/shards.ts).
• If there are five or fewer such files and the pull changed none of them, each is run again on the base's tree. That run uses a git worktree on the shard scratch disk, sharing the checkout's node_modules.
• A file is main's only when every test failing on the pull also fails on the base. Then the log says … is red on main at <base> too, the same tests: not counted against this pull, and when every red file is main's the check exits 0.
• A second break in a file main already has red stays the pull's, and the log names the test (… but "readme" passes there: this pull's red).
• The pull also stays red when:

- the file passes on main, doesn't exist there, or couldn't be checked out; - a red on either side names no test (a crash or a load error); - the pull changed the red file.

What to look at

• Fast-tier cost: about 62 tree scans now join every fast tier. Most aren't in the committed times table, so I couldn't estimate the extra time; apps-token alone takes about 20 s.
• Detection reads source text with regexes, not a parser, and doesn't follow scope. If it guesses wrong, @tree-scan and @not-a-tree-scan override it.
• RULE_TESTS is kept: src/checks/outside_diff.ts decides from a run's log alone, so it still needs a fixed list.
• Fast tier only: the land gate and the hold in red_main.ts are unchanged.

Tests

• test/tree-scans.test.ts:

- a root used elsewhere next to a readdir of a temp dir is not a scan; - a root reached through a name, a walk's parameter or a parameter's default is; - every RULE_TESTS entry is found.

• test/red-on-main.test.ts:

- unit cases: the same red clears; a new failing name in an already-red file stays red; a base red with no names stays red; green, absent and unrun files stay red; and the cases where main isn't asked; - one end-to-end case through test-shards.ts --tier fast, taking about 0.5 s.

• test/shards.test.ts: failingTests drops the timing suffix and the summary's repeated (fail) lines.

Try: on a forge whose main fails test/unreached.test.ts, open a pull that changes only README.md. Its check is green and says the file is red on main too, with the same tests.

Closes #10304

🤖 Generated with Claude Code

Handed to a person by the crew: two follow-up runs on the review's changes ended without a push, so no more start here until you say what to try. (The forge posts this under the reviewer's name; the reviewer did not write it.)