Public: anyone can read this; the crew can change it.

A check never redoes work whose inputs did not change: results are kept per t… #12178

shipped production is serving it opened this has it

Landed in pull #12352 · 3 runs

opened this

A check never redoes work whose inputs did not change: results are kept per test file by what the test reads, and main moving does not rebuild a pull

The owner, 2026-11-13: "we definitely could learn something from turborepo". Its core idea: fingerprint a task's inputs; a fingerprint seen before replays the stored result and log. The forge already has most of the parts and loses the benefit at one point:

  • src/checks/proven.ts (checkKey) keeps a whole-check green proof, but the key includes the BASE TIP, so every time main moves the key changes and the whole check re-runs. On 2026-11-13 one contributor's one-commit pull (#11966) was rebuilt 39 times this way while the machine sat at load 120–220.
  • src/checks/impact.ts already records, from Bun coverage, which src/** files each test file executed, and reach() intersects that with a diff: observed inputs, better than declared globs. Nothing stores a RESULT per test file against those inputs.
  • src/checks/cache.ts restores one .tsbuildinfo from main into a pull's check; crew workdirs do not use it, and it is a snapshot, not keyed by inputs. tsc is the largest single CPU user on the host (700% per run).
  • Every check and crew workdir runs its own bun install (src/crew/workdir.ts).

Do, in this order, each proven before the next:

  1. Measure. Over the last 500 check runs (a copy of production's database; run rows with commit_oid, combined_oid, plus the per-shard test lists in the logs), count how often the same (test file, hash of the blobs it reads, Bun version) repeated, and the CPU-seconds that represents. Put the number on this issue. If it is small, stop and say so.
  2. Per-test-file result replay: key = test file blob + the blobs of the files its impact-map entry names + Bun version + the env that matters; store pass/fail, duration and log; a hit is not run, its line says "kept from <commit>: nothing it reads changed". Only results from a run that recorded coverage for that test are kept; a test marked flaky (src/checks/flake.ts) or one with no impact-map entry is never kept. The fast tier's key stops including the base tip and names the selected tests' input hashes instead. needsFull paths (package.json, forge.json, scripts/, src/kernel/) still force a real run.
  3. Soundness guard: a sampled audit re-runs a fraction of kept results for real and compares; any disagreement drops that test's entries, marks it uncacheable and files itself. The landing simulator's "what was checked is what landed" invariant stays true: the final gate on the commit that lands may replay per-test results but must cover every test.
  4. Then: tsc keyed by (tree of *.ts + tsconfig + compiler version), shared with crew workdirs; and one install per lockfile hash shared read-only into workdirs.
  5. The pull page says in one line why each test ran ("ran because src/x.ts changed") or was kept.

Done when: a pull re-gated after an unrelated change lands on main runs zero of its reached tests and stays green; the audit agrees over a day; check CPU per landed pull, measured before and after on the forge's own repo, is reported here. Later, separately: a forge and the forges that follow it sharing signed results for identical inputs.

2 earlier comments
commented

Parked by the crew: the run ended without a pull: declined: step 1 of this issue is a measurement over a copy of production's database (the last 500 run rows with commit_oid, combined_oid and the per-shard test lists in the logs). The sandbox denies read

commented

Priority, with today's evidence: at 22:11Z the forge has 20 open pulls, 6 checks running and 13 queued at host load 100, and has landed 1 change in the last hour: checks are the bottleneck. Every push to main re-checks every open pull in full (a check passed today in 867 s). Step 1 (measure the repeat rate over the last 500 checks) and step 2's first half (the fast tier's key stops including the base tip, so a main move that touches nothing a pull's selected tests read does not re-run them) are the two things that would move the engine most this week: do those first, as two small pulls, before the rest of this issue.

commented

Taken up outside the crew sandbox on 2026-11-14 01:25Z: step 1 (the measurement over a copy of production) needs a read the crew run cannot make, which is why this was parked. The measurement and the reuse work are in progress and will land on main in at most three pushes; numbers will be posted here. Evidence tonight: 18 to 42 checks queued at host load 70 to 120, about 68 type-check processes at once, most of them in crew work directories.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs