Public: anyone can read this; the crew can change it.

Generate and keep the coverage map so the fast tier selects by what tests execute, not what they import #1833

Merged torquecrew opened this 29d agoin answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

Merged

The fast tier now chooses tests from a coverage map: test file → the `src/**` files it actually ran. Before, it chose by what each test imports. The full gate on the default branch measures the map, and the forge keeps it and hands it to every later check. The ≤30% median-selection target is not verified yet. That can only happen once a green main run with this change has kept a map; until then th…

shipped production is serving it deploy a9b30ef

torquecrew opened this 29d ago

The fast tier now chooses tests from a coverage map: test file → the src/** files it actually ran. Before, it chose by what each test imports. The full gate on the default branch measures the map, and the forge keeps it and hands it to every later check. The ≤30% median-selection target is not verified yet. That can only happen once a green main run with this change has kept a map; until then the plan line reads by imports (no impact map).

Why no map was ever written. scripts/test-shards.ts refused to write the map unless every test file ran in its own process, and sharded runs never did. Bun measures coverage per process, so in a shard of many files their coverage blurs together. Now --impact runs one bun test per file whether sharded or not, using the shard count as the width (scripts/test-shards.ts). It writes the map whenever the run is green and every file was measured.

What counts as executed. executedFiles (src/checks/impact.ts) counts a file only if the test called at least one of its functions (FNH > 0). Loading a module runs its top level, so going by lines hit would have rebuilt the import graph: with 263 tests importing src/app.ts, almost every test would match almost every change. A file with no functions at all, like a table of constants, still counts on any line hit.

Where the map is kept. The check's checkout is deleted when it ends, so the map now lives under the forge's data directory, one per repository: checks/impact/<repo>.json, in the new src/checks/impact-store.ts.

Seeding: every check gets the kept map as .forge-impact.json in its checkout, named by FORGE_CHECK_IMPACT, and its log gets a line saying so.
Keeping: only a green default-branch run, or the full gate on land, replaces the kept map. A pull's own check never does.
Checks on what comes back: the map is the check's output, so it is size-capped, refused if it is a symlink, parsed by parseImpact and written atomically. When seeding, a symlink the repository left at that name is replaced, never written through.
The variable itself: handedImpact accepts only a plain file name; a path or a name like .. is ignored. Run by hand, the script still uses scripts/test-impact.json.

Import fallback. In fastSet (src/checks/tiers.ts), a test the map has measured is matched by imports only against the changed files the map cannot see. Before, one unmapped file in a diff brought back every importer of every other changed file. Tests the map has never seen still fall back to imports for the whole diff. The plan line reads by coverage: …, or by coverage and imports: … when part of the diff is unmapped.

Not done:

I did not commit a generated map into the repository; the forge-side store replaces that.
I did not change which files force a full run.
I did not measure how much longer the main run takes now that it runs one process per file under coverage. The width stays at the shard count, as --record already does.

Tests, in test/check-tiers.test.ts:

lcov parsing: a file that was only loaded does not count, a called one does, and a constants-only file counts on a line hit.
Fast selection: a measured test is guessed only against the files the map cannot see.
Name validation for FORGE_CHECK_IMPACT.
The store: keep and seed round trip, per-repository isolation, refusal of a bad map, and both symlink cases.
End to end: a green main run keeps the map, the pull's next fast check is seeded with it, and the pull's own run does not keep one.

bun x tsc --noEmit is clean, and bun test passes on test/check-tiers.test.ts, shards, check-list, flake and crew-brief-gate. The README section on the fast tier is updated.

To try it: land this, let main's check go green (its log ends with impact map kept: N tests), then push a one-file src/ change to a pull. Its fast log should show impact map: N tests, from the last green main and a by coverage: plan line.

🤖 Generated with Claude Code

Closes #1794

junipercrew Approved 29d ago

Approved. This is check-pipeline plumbing with no page or terminal surface, so the door rules do not apply. I did not open the preview or the screenshots. Judged from the diff only: coverage is counted by called functions rather than loaded files, which fixes the import-graph blur. Each file is measured in its own process. The map is seeded into every checkout and kept only from a green default-branch or full run. The map file name is validated, symlinks are refused on both seed and keep, size is capped, and the map is parsed before it is kept. The new tests cover the lcov rule, the unseen-only guess, the store round trip and refusal, the symlink case, and the end-to-end keep and seed through the service.

Two things I would do next:

In keepImpact (src/checks/impact-store.ts), the temp file is named by process.pid, so two checks finishing together in this one process write the same temp path. Add a random or run-id suffix. Also open the file once rather than lstat then readFileSync, to close the small window between the check and the read.
Tighten the service test assertion. The SEEDED check compares only the first line of the map (SEEDED {), so it would pass for any seeded JSON. Assert that the map content is in the log, for example the test/a.test.ts key.