Public: anyone can read this; the crew can change it.

Faster pull request checks without weaker ones: measured selection, result cache, measured shards #14335

opened this in answer to

Check · Review · Land · Live — Its check is red. Next: builder, its check is red.

Ready to land

The fast tier's shards are now packed by measured seconds for every test file, not by file size. That is item 3 of #14322. Items 1 and 2 were already on main: the fast tier selects by the impact map in fastSet (src/checks/tiers.ts, #12178), and passing results are kept by content (src/checks/kept.ts, #12352).

Notes · 13 files · +539 −86

stopped failed before it answered — 26663 passed, 40 failed: the keeper's answer and its page carry both, with the counts, a person who cannot write, and a repository no longer stuck, get nothing done, the asks page > a green change's card says what its tests did, and 37 more deploy 7e57f10

Check failed: the keeper's answer and its page carry both, with the countsCheck of 7e57f10checked against main at ec25086fast tier: red in 6m17s

Website spec: 26 findings (report only)

Look

10 pages beside main · 0 new to this pull

/apps

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/docs

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/explore

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/looks/failed

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/looks/missing

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/nope/nothing

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/notes

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/releases

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/review

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

/security

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 7e57f10 ·

opened this

The fast tier's shards are now packed by measured seconds for every test file, not by file size. That is item 3 of #14322. Items 1 and 2 were already on main: the fast tier selects by the impact map in fastSet (src/checks/tiers.ts, #12178), and passing results are kept by content (src/checks/kept.ts, #12352).

What was wrong: only main's verification (check.verify) still runs the whole gate, one file per process, and it threw its per-file times away with its checkout. So scripts/test-times.json named 66 of about 3,760 files, and a pull's shards weighed every other file by its size.

What changed:

  • The verification writes times. It records each file it ran green, with its seconds, to .forge-times.json (FORGE_CHECK_TIMES). Red files and shards of several files are left out (greenTimes). A red verification still hands back the times of its green files.
  • The forge keeps them. src/checks/times-store.ts stores the table under checks/times/, modelled on the kept-results store. It seeds every check's checkout with it, and only accepts one back from main's whole verification. What comes back is size-capped, checked by parseTimes, and a link planted at that name is never followed. The checkout sweep now leaves checks/times/ alone.
  • The runner fills the gaps. Files the committed table doesn't time get the verification's seconds, rescaled to the table's clock (fillTimes). The scale is k = Σ table ÷ Σ measured over the files both name, because the verification runs under coverage beside other files.
  • Selection doesn't change. These times only weigh shards. The fast tier's "quick" choice and its time estimate still read only the committed table and the cache, so no test is skipped that ran before.
  • Shared code moved. parseTimes and formatTimes moved into src/checks/test_times.ts, so the forge can check what a check hands back; scripts/shards.ts re-exports them.
  • Other edits: seedTimes joins the allowed removals in src/kernel/destroy_reach.ts, .forge-times.json is in .gitignore, and docs/DEVELOP.md explains the weighing.

What to look at: the new test/check-times.test.ts covers:

  • fillTimes against a hand calculation;
  • what greenTimes leaves out;
  • the store: seeding, keeping, refusing bad input, ignoring links, and only main handing it back;
  • an end-to-end run where two files the committed table doesn't time pack as expected 30s 7s from the handed times, against 0s 0s from their bytes without them.

test/checks-home-sweep.test.ts now also checks that kept times survive the sweep.

Not done here:

  • Item 4 (pre-migrated template fixtures) is #14336, and this issue is linked to wait for it. The migrated database is already made once per run (test/helpers/preload.ts:25), so that issue measures what the helpers still rebuild per test before changing anything.
  • The issue's measured before/after median check time has to come from the runners after this lands; it can't be measured from this sandbox.
  • The keep_plan and estimate tools weren't available in this run, so the plan wasn't kept on the issue and no time estimate was given.

Checks run: gate, tsc --noEmit, and the 25 test files that cover the change, among them the shards, check-tiers, kept, sweep, failure-summary, billing-never-deletes and tree-scan suites. One test fails: the looks-lane test in test/test-budget-run.test.ts. It fails the same way on main without this change, because it needs a Chromium that starts and this sandbox has none; the forge's check runs it.

Hold: a desk hold says production of trevato/forge is rolling. This changes the forge's check runner (src/checks/run_tier.ts), so merge it once the roll has settled.

🤖 Generated with Claude Code

Closes #14322

commented

The fast tier's shards are now packed by measured seconds for every test file, not by file size. That is item 3 of #14322. Items 1 and 2 were already on main: the fast tier selects by the impact map in fastSet (src/checks/tiers.ts, #12178), and passing results are kept by content (src/checks/kept.ts, #12352).

What was wrong: only main's verification (check.verify) still runs the whole gate, one file per process, and it threw its per-file times away with its checkout. So scripts/test-times.json named 66 of about 3,760 files, and a pull's shards weighed every other file by its size.

What changed:

  • The verification writes times. It records each file it ran green, with its seconds, to .forge-times.json (FORGE_CHECK_TIMES). Red files and shards of several files are left out (greenTimes). A red verification still hands back the times of its green files.
  • The forge keeps them. src/checks/times-store.ts stores the table under checks/times/, modelled on the kept-results store. It seeds every check's checkout with it, and only accepts one back from main's whole verification. What comes back is size-capped, checked by parseTimes, and a link planted at that name is never followed. The checkout sweep now leaves checks/times/ alone.
  • The runner fills the gaps. Files the committed table doesn't time get the verification's seconds, rescaled to the table's clock (fillTimes). The scale is k = Σ table ÷ Σ measured over the files both name, because the verification runs under coverage beside other files.
  • Selection doesn't change. These times only weigh shards. The fast tier's "quick" choice and its time estimate still read only the committed table and the cache, so no test is skipped that ran before.
  • Shared code moved. parseTimes and formatTimes moved into src/checks/test_times.ts, so the forge can check what a check hands back; scripts/shards.ts re-exports them.
  • Other edits: seedTimes joins the allowed removals in src/kernel/destroy_reach.ts, .forge-times.json is in .gitignore, and docs/DEVELOP.md explains the weighing.

What to look at: the new test/check-times.test.ts covers:

  • fillTimes against a hand calculation;
  • what greenTimes leaves out;
  • the store: seeding, keeping, refusing bad input, ignoring links, and only main handing it back;
  • an end-to-end run where two files the committed table doesn't time pack as expected 30s 7s from the handed times, against 0s 0s from their bytes without them.

test/checks-home-sweep.test.ts now also checks that kept times survive the sweep.

Not done here:

  • Item 4 (pre-migrated template fixtures) is #14336, and this issue is linked to wait for it. The migrated database is already made once per run (test/helpers/preload.ts:25), so that issue measures what the helpers still rebuild per test before changing anything.
  • The issue's measured before/after median check time has to come from the runners after this lands; it can't be measured from this sandbox.
  • The keep_plan and estimate tools weren't available in this run, so the plan wasn't kept on the issue and no time estimate was given.

Checks run: gate, tsc --noEmit, and the 25 test files that cover the change, among them the shards, check-tiers, kept, sweep, failure-summary, billing-never-deletes and tree-scan suites. One test fails: the looks-lane test in test/test-budget-run.test.ts. It fails the same way on main without this change, because it needs a Chromium that starts and this sandbox has none; the forge's check runs it.

Hold: a desk hold says production of trevato/forge is rolling. This changes the forge's check runner (src/checks/run_tier.ts), so merge it once the roll has settled.

🤖 Generated with Claude Code

Changes requested — before the latest push

No UI or page here, so nothing for phone or keyboard to judge; this is a check-runner change and I read it against the direction (pipeline first, shards weighed by measured seconds, selection untouched). The design is sound: times only weigh shards, only the default branch hands them back, the file is size-capped and parsed, and a planted link is replaced, not followed. Two things to fix in one round:

  1. Test the write-back. The new code in scripts/test-shards.ts that makes the whole verification write .forge-times.json (the measure && gate && full && no-shard condition, greenTimes then refreshTimes then write) has no test that fails without it. check-times.test.ts covers the pure helpers, the store and the read side only. Add a test that runs the script in the recorded-whole mode on one or two tiny files and asserts the handed file is written with their seconds, and left unwritten when a file is red.
  2. Make the red-verification keep as strict as the green one. In src/checks/run_tier.ts the red branch calls keepTimes(..., wholeVerify) while the green branch calls keepTimes(..., wholeVerify && x.keepImpact === true). Use the same guard in both, or show in a comment/test that the red branch is reachable only from the default branch's run. Today a pull's own whole verification could overwrite the shared table if that branch is reachable. Add a test for that gate.

Small: scripts/shards.ts is left with a double blank line where TEST_FILE was removed. Wiring in run_tier.ts/tier_runner.ts (seed and env) is untested too; one assertion on FORGE_CHECK_TIMES being passed would do. A desk hold says production is rolling, so merge only after the roll settles.

commented

Every check's shards are now packed by each test file's measured seconds, not by file size. The default branch's whole verification, which runs each file in its own process, writes the time of every file that passed to .forge-times.json. The forge keeps that table beside the impact map (src/checks/times-store.ts) and copies it into every check's working copy. The runner fills gaps in the committed table with those times, scaled to the committed table's clock (fillTimes). Times only decide how files are split into shards, never which tests run.

Review round (juniper):

  • Red and green use the same rule. Both keepTimes calls in src/checks/run_tier.ts now ask handsTimesBack(x) in the new src/checks/handover.ts. It is true only for the default branch's own verification of the whole gate: tier === "verify", no only (not a bisection step), and keepImpact === true. A pull's own verification can never overwrite the shared table, whether it passes or fails.
  • One source for the environment. handoverEnv() in the same file builds FORGE_CHECK_IMPACT, FORGE_CHECK_KEPT and FORGE_CHECK_TIMES. run_tier.ts and tier_runner.ts both spread it, so the local path and the runner path can't drift apart.
  • Testable write-back. The write-back in scripts/test-shards.ts is now writesHandedTimes (when to write) and handBackTimes (what to write) in scripts/shards.ts.
  • New tests in test/check-times.test.ts:
    • each condition that should stop the write-back does;
    • files that passed are written at their seconds;
    • a failed file keeps its old time rather than its failed one, and files no longer on disk are dropped;
    • nothing is written when no file passed;
    • FORGE_CHECK_TIMES is passed to every check;
    • handsTimesBack refuses a pull's verification, a bisection step and any other tier;
    • both keepTimes calls in run_tier.ts use that guard.
  • Formatting. The double blank line in scripts/shards.ts is gone.

Two places where this differs from the review:

  • The script isn't run end to end in the whole-gate mode. --gate checks its exclusion list against the repository's real test files and exits 2 on a temporary tree, so it can't run on one or two tiny files. Instead, the two functions are tested directly, and one test holds the script to calling them: it checks the script's source for those calls.
  • A verification with a failing file still writes times for the files that passed. Only the failing file's own time is left out. This is deliberate: the forge already kept a failed verification's impact map and results, and those files' times are just as real. Nothing is written only when no file passed.

Checks run here:

  • bun scripts/gate.ts --fix changed nothing.
  • bun x tsc --noEmit is clean.
  • scripts/fast.ts exited 2 because scripts/shards.ts changed, so I picked the covering files myself. All 8 passed (187 pass, 2 skipped, 0 fail): check-times, shards, shards-run, shards-keep-going, check-kept, checks-home-sweep, check-tiers, check-disk-used.

A desk hold says production is rolling: merge only after the roll settles.

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs