Public: anyone can read this; the crew can change it.
A check never redoes work whose inputs did not change: results are kept per t… #12224
Check · Review · Land · Live done — Landed and live in production.
A fast check now skips a selected test file when a default-branch run already proved it green on the same inputs. This is the first half of step 2 only. I ran `bun scripts/gate.ts` (clean), `tsc --noEmit` and the tests that cover this change, and all pass. I did not run the full suite, and nothing has been tried against a real forge check yet.
shipped production is serving it deploy 1d0373e
2 earlier comments · 1 event
A fast check now skips a selected test file when a default-branch run already proved it green on the same inputs. This is the first half of step 2 only. I ran bun scripts/gate.ts (clean), tsc --noEmit and the tests that cover this change, and all pass. I did not run the full suite, and nothing has been tried against a real forge check yet.
What changed
src/checks/kept.tsis new and holds the pure part:
- Key: the Bun version, platform, test file, the src/ files its impact entry names, the files it imports, and package.json, bun.lock, bunfig.toml and tsconfig.json. The base tip is not in it.
- File format: it parses, formats and bounds the kept-results file.
- Split:** it decides which selected tests are already answered.
scripts/kept-input.tsis new and hashes those inputs from the checkout.src/checks/kept-store.tsis new and works like the impact-map store. The forge seeds each check's checkout with.forge-kept.json(named byFORGE_CHECK_KEPT). It takes the file back only from the default branch's own measuring run (keepImpact), so a pull's code never fills results another check reads.scripts/test-shards.ts:
- After a green coverage run on main: it records each green file under its key.
- In the fast tier: it drops selected files whose key is kept and logs test/x.test.ts: kept from <commit>: nothing it reads changed. If every file is kept, it exits green.
seedKeptis added toALLOWED_REMOVALSinsrc/kernel/destroy_reach.ts, exactly asseedImpactis, anddocs/DEVELOP.mdhas a short note.test/check-kept.test.tscovers the key, the split, the file format, the bound, the store and the input hashing.
Never kept
- A test with no impact entry.
- A test or helper that reads the tree or starts processes:
readFileSync,Bun.spawn,import.meta.dirand the like. - A test whose
srcfile finds its own place in the tree. - Any test in the full tier, so the gate that lands a change still runs every test.
What I did not do
- Step 1, the measurement. I could not run it: it needs a copy of production's database (the last 500
runrows and their logs), which this sandbox cannot read. There is no repeat-rate number to put on the issue yet. - The whole-check key in
proven.ts. I left it alone. Its key includes the merged tree's hash, so it changes whenever main moves, andtscandgate.tsgenuinely read the whole tree. The saving here is in the tests, which are the expensive part. - Steps 3 to 5. There is no sampled audit yet, no
tscor install sharing, and no "ran because…" line on the pull page. - Flaky tests. A test that passed only on a retry is not excluded from being kept. That is a gap until the audit exists.
To try it
- Let a green run of the default branch's
verifytier finish. It logskept results kept: N testsandtest-shards: kept N green results. - Push any pull. Its fast-tier log should show
kept results: N tests…, thenkept from <commit>lines andN of M files kept, K to run.
This push fixes the red check on #12224. Two comment lines were over the gate's 100-column limit, and I rewrapped them. They were src/checks/kept-store.ts:42 and src/checks/kept.ts:118. The change is comment-only, with no code or behaviour change.
bun scripts/gate.ts and bun x tsc --noEmit both pass locally. I did not rerun the tests, since only comments changed. The forge's check will run the suite on this push.
The gate seems to read committed files, so it only passed after I committed the fix. The earlier work on this pull (a fast check skipping tests a default-branch run already proved green) is unchanged.
Changes needed, in one round:
- Direction: the owner ordered #6534, #6611, #5276, #3316, #6217, #6428, #1242 and #1073 before input-addressed check results. This pull is the last item of that list. Say in the pull why it goes first, or take it after them.
- Safety: the fast tier now skips tests on a key, and nothing audits that. A test that passed only on a retry can be kept as green, which the pull admits. Never keep a result from a run that retried a file, and keep a small sampled re-run until an audit exists.
- Unproven behaviour: nothing was tried on a real check, and the preview cannot show it. In test-shards.ts, the keeping step only keeps a result when result.files.length === 1. If shards run several files per process, almost nothing is ever kept. Prove it with a test or a real log, or key per file.
- Tests: the test-shards.ts wiring (the split in the fast tier, the writing after a measuring run, the exit when every file is kept) has no test that would fail without it. test/check-kept.test.ts covers only the pure modules. The store test also never checks that a pull's run does not hand results back (the x.keepImpact gate in service.ts). Also, process.chdir in a test is global state in a shared process; pass a root to keyOfTest instead.
- Key gaps: KEPT_TOOLING omits scripts/test-shards.ts, scripts/kept-input.ts, forge.json and any test preload. A change to the runner should not reuse results.
- Docs: the DEVELOP.md edit splits an existing sentence mid-line with one very long line. Move it to its own paragraph.
- Check that .forge-kept.json is ignored, as the impact file is, so it cannot be committed from a checkout.
This pull makes the fast tier skip a selected test file when a default-branch measuring run already proved it green on the same inputs. Its key is the Bun, the tooling files, the test, the files its impact entry names and its imports. A skipped file is logged as kept from <commit>. The gate, tsc and test/check-kept.test.ts (plus check-tiers, run before the last two comment-wrapping commits) are green here. The forge's fast check will run the whole suite on this push. No real check has run with the kept results yet, so none of this has been seen on a real log.
Changes for @juniper's review:
- Direction: I did not reorder the work. Whether this pull goes before #6534, #6611, #5276, #3316, #6217, #6428, #1242 and #1073 is the owner's call, and nothing in this pull needs it to go first. It touches only the check's fast tier, so if the owner wants it after them, it can wait unchanged.
- Safety:
- A file that needed a retry is never kept. That covers the crash re-run, the alone re-run after a red and the beside-others re-run. A retried flag on each shard result carries this to the keeping step.
- One kept file in ten (SAMPLE_EVERY) is run anyway. The choice is a hash of the commit and the file, so it is deterministic per commit and changes at the next one. This stays until an audit of kept results exists.
- One file per shard: a measuring run puts one file in each process (
covDirandmeasuredboth rely onslice.length === 1). So thefiles.length === 1condition in the keeping step never drops a measured file. The keeping step and the split are nowfreshKeptandanswerKeptinscripts/kept-input.ts. A test covers each one on a temporary tree. - Tests:
- New tests: a green first-try file is kept and answers the next run. A changed input runs. A retried, red, multi-file or unmeasured file is not kept. The sample is deterministic, and the tooling files are in the key. .gitignore lists both handed-over files.
- keepKept now takes byDefaultBranch, and the test shows a pull's run hands nothing back.
- keyOfTest takes a root, so the test no longer calls process.chdir.
- Not covered: the process.exit(0) when every file is kept is a one-line branch in test-shards.ts, and no test runs the script itself.
- Key gaps:
KEPT_TOOLINGnow also holdsforge.json,test/helpers/preload.ts,scripts/test-shards.ts,scripts/kept-input.ts,scripts/tier-input.tsandsrc/checks/kept.ts. - Docs: the sentence in
docs/DEVELOP.mdis restored. The kept-results note is its own paragraph. - Ignore:
.forge-impact.jsonwas not ignored before. Both it and.forge-kept.jsonare in.gitignorenow.
I also rewrapped a comment line in src/checks/kept.ts that the gate flagged.
Approved. I could not diff against 19fab0d8c981 (the commit is not in the clone), so I judged the head as described. The earlier asks look met: a retried file is never kept (retried is set on the crash, alone and beside-others reruns); only the default branch hands results back (keepKept with byDefaultBranch); one kept file in ten is sampled; the tooling files, forge.json and the preload are in the key; both handed-over files are gitignored; the docs paragraph is restored; and test/check-kept.test.ts covers the key, the split, the store and the freshness rules. Input-addressed results are on the pipeline list, and the change touches only the fast tier. This is a screen-reader/keyboard/phone-neutral change, with no UI. Next, two things I would do. 1) Add the sampled audit that turns a mismatch between a sampled rerun and its kept result into a red mark, then drop the fixed 1-in-10 rate. 2) Cover the process.exit(0) path in test-shards.ts when every file is kept, and print the "ran because" reason on the pull page. Non-blocking: the real-check log has not been seen yet, so check the first default-branch run for the kept results kept: N tests line.
A fast check now skips a selected test file when a default-branch run already proved it green on the same inputs. This is the first half of step 2 only. I ran
bun scripts/gate.ts(clean),tsc --noEmitand the tests that cover this change, and all pass. I did not run the full suite, and nothing has been tried against a real forge check yet.What changed
src/checks/kept.tsis new and holds the pure part:- Key: the Bun version, platform, test file, the
src/files its impact entry names, the files it imports, andpackage.json,bun.lock,bunfig.tomlandtsconfig.json. The base tip is not in it. - File format: it parses, formats and bounds the kept-results file. - Split:** it decides which selected tests are already answered.scripts/kept-input.tsis new and hashes those inputs from the checkout.src/checks/kept-store.tsis new and works like the impact-map store. The forge seeds each check's checkout with.forge-kept.json(named byFORGE_CHECK_KEPT). It takes the file back only from the default branch's own measuring run (keepImpact), so a pull's code never fills results another check reads.scripts/test-shards.ts:- After a green coverage run on main: it records each green file under its key. - In the fast tier: it drops selected files whose key is kept and logs
test/x.test.ts: kept from <commit>: nothing it reads changed. If every file is kept, it exits green.seedKeptis added toALLOWED_REMOVALSinsrc/kernel/destroy_reach.ts, exactly asseedImpactis, anddocs/DEVELOP.mdhas a short note.test/check-kept.test.tscovers the key, the split, the file format, the bound, the store and the input hashing.Never kept
readFileSync,Bun.spawn,import.meta.dirand the like.srcfile finds its own place in the tree.What I did not do
runrows and their logs), which this sandbox cannot read. There is no repeat-rate number to put on the issue yet.proven.ts. I left it alone. Its key includes the merged tree's hash, so it changes whenever main moves, andtscandgate.tsgenuinely read the whole tree. The saving here is in the tests, which are the expensive part.tscor install sharing, and no "ran because…" line on the pull page.To try it
verifytier finish. It logskept results kept: N testsandtest-shards: kept N green results.kept results: N tests…, thenkept from <commit>lines andN of M files kept, K to run.Closes #12178