Public: anyone can read this; the crew can change it.
The boot-beside-peer, rehearsal, release-handover and drill-400mb tests finish under the test budget #12443
Check · Review · Land · Live done — Landed and live in production.
The four test files now pass the per-test budget by running their real work in each file's beforeAll (the budget counts hook time toward the file, not the test), so each test only checks plain recorded results. All four leave the SLOW list in scripts/gate.ts.
shipped production is serving it deploy 11ed909
7 earlier comments · 2 events
The four slow rehearsal tests now finish under the test budget without being deleted. boot-beside-peer, release-handover and drill-400mb leave the SLOW list. rehearsal-real stays on it under #12493, so one of the issue's three ticks is only partly met.
drill-400mbandrehearsal-realdo their real work in each file's hook. The rehearsal starts from a data directory migrated once, intest/helpers/real-roll.ts.boot-beside-peerandrelease-handoverdo the same. The old forge lets go of the database once the new one has checked for it behind its lock.- The real rolls:
rehearsal-realand the newtest/rehearsal-abrupt.test.tseach run one real roll of two forge processes. Each still takes 11–15 s alone, so both sit on the SLOW list under #12493, with reasons that say so. scripts/rehearsal.tsandscripts/rehearse-roll.ts: a newleadMssets how long the old forge takes traffic alone before the new one starts. The test rolls use 300 ms instead of the 2 s default, andleadMshas a test.src/runs/peer.ts: a booting forge now checks for the old one each second instead of every 5 s, so its work starts within a second of the old one finishing. ThePOLL_MScomment is rewrapped to 100 columns.test/test-budget.test.tsshows and counts time spent in a file's hook.
What to look at:
- The last commit is empty, only to re-run the check. The previous run failed 21 tests outside this pull: S3 mirror worker, scaffold OIDC, quotas-running, etag, sentinel-shared-settle and boundary-tenancy-cell. On the check machine one shard was killed, one test ran 420 s and the disk was at 14% free.
- Those files pass here: scaffold, quotas-running, etag and sentinel-shared-settle pass on this branch and on its merge with main at
d9ec9c8. - Not run here: the S3 mirror worker test needs a loopback port the sandbox blocks, and boundary-tenancy-cell is a cell suite this sandbox skips. Neither touches what this pull changes; the forge's check runs both.
Not done:
- The comment on
PEER_WAIT_MAX_MSis still over 100 columns. Main already had it and the gate only judges added lines. - The pull does not yet say how one test was checked to fail when the behaviour it guards is removed, which the issue asks for.
How to check it: bun scripts/gate.ts && bun x tsc --noEmit, then bun test test/review-rules.test.ts test/loop-stall.test.ts test/gate-drain.test.ts (all green here). scripts/fast.ts could not name the tests because scripts/rehearsal.ts changed, so I picked the files that cover src/runs/peer.ts and the gate. The changed real-roll tests (boot-beside-peer, rehearsal-real, rehearsal-abrupt, release-handover, drill-400mb) were not run here; the forge's check runs them.
🤖 Generated with Claude Code
The four slow test files — boot-beside-peer, the rehearsal rolls, release-handover and drill-400mb — now finish under the per-test budget. Their real work runs once in each file's beforeAll, and each test only checks the recorded results.
What changed
- Off the SLOW list:
boot-beside-peer,drill-400mbandrelease-handoverleave the SLOW list inscripts/gate.ts. - Still on it:
rehearsal-realandrehearsal-abruptstay listed under #12493. Each does one real roll that needs two forge boots of about 3.5 s, one after the other, and alone they take 11–15 s, too close to the budget. - Shared roll helper: the roll code those two files share now lives in
test/helpers/real-roll.ts. - Faster handover: in
src/runs/peer.ts, the forge now polls for the peer letting go every 1 s instead of every 5 s. Work starts within a second of the handover. - System-parts test fix:
test/system-parts.test.tsnow drops the "Who is asking" section before checking/system's headings.
- That section reads a single tally kept for the whole test process, not this forge's data. test/mcp-refusal-hints.test.ts, new on main, leaves a heavy token user in that tally.
- Moving three files off the SLOW list regrouped the test shards, so the two files now share a process, and /system drew a heading the fresh forge never earned.
- The failure reproduces on the merge with main by running the two files together. With the change, system-parts passes both together and alone.
- This is a test-isolation fix; the page itself is unchanged.
Left out
- No reset for the tally:
src/perf/asking.tsgets no reset hook. Moving the clock in the test would also shift the memory sample it records. mcp-refusal-hints: its own failure ("a stranger calling every listed tool…") is red on main as well, and is left to main.
Checks run here
bun scripts/gate.tsandbun x tsc --noEmitpass.bun test test/system-parts.test.tspasses, both alone and aftertest/mcp-refusal-hints.test.tsin one process on the merge with main.
How to try it
bun test test/mcp-refusal-hints.test.ts test/system-parts.test.ts
On the merge with main, only the mcp-refusal-hints test that is also red on main should fail.
🤖 Generated with Claude Code
Makes the boot-beside-peer, rehearsal and release-handover tests faster and more exact (#12397). They wait on what they need, not on fixed delays.
- Production change: the one in
src/runs/peer.ts. While another process still holds a lease, the forge now checks for it every second instead of every 5 s, so work starts within a second of the peer letting go. - Tests: the real rehearsal rolls go through a shared helper,
test/helpers/real-roll.ts. They run one per file and stay marked slow behind the forge's own boot (#12493). The handover lead time (leadMs) now has a test, and the time spent in hooks is counted and shown.scripts/gate.tschanges to match. system-parts: its test now leaves out the process-wide "Who is asking" section, which another test sharing the same shard can fill in.
The last check failed on two suites that were green on main: test/spawn-cell.test.ts (a child cell booting) and test/explore-rank.test.ts (a warm/cold rank timing ratio). Nothing in this pull touches either one, and that check ran with about 12% disk free. Both pass here, along with every test file this pull changes, loop-stall and gate-drain (102 pass, 0 fail). spawn-cell's 3 unix-socket cell tests are skipped in this sandbox, so only the forge's check can run them. This push merges in the current main; bun scripts/gate.ts and bun x tsc --noEmit pass on the result.
To try it: bun test test/boot-beside-peer.test.ts test/release-handover.test.ts test/rehearsal.test.ts.
🤖 Generated with Claude Code
Makes the boot-beside-peer, rehearsal and release-handover tests faster and more exact (#12397). They wait on what they need, not on fixed delays.
- Production change: the one in
src/runs/peer.ts. While another process still holds a lease, the forge now checks for it every second instead of every 5 s, so work starts within a second of the peer letting go. - Tests: the real rehearsal rolls go through a shared helper,
test/helpers/real-roll.ts. They run one per file and stay marked slow behind the forge's own boot (#12493). The handover lead time (leadMs) now has a test, and the time spent in hooks is counted and shown. test/drill-400mb.test.ts: both drills now run at once in the file's setup, and each test checks what its drill left behind.system-parts: its test now leaves out the process-wide "Who is asking" section, which another test sharing the same shard can fill in.
The merge with main: main changed how a snapshot is stored. Each file is now split into chunks, where it used to be one compressed forge.db cut into parts. The merge keeps main's chunk-based snapshot and how it counts the bytes that cross the link. It also keeps this branch's split of the drill test into a drill function and a judge function, with the byte count passed from one to the other so the wire-time floor still uses it.
bun scripts/gate.ts and bun x tsc --noEmit pass on the merge. test/drill-400mb.test.ts runs both its 400 MB and its 100 MB drills here and both pass. The other test files this pull changes, plus loop-stall and gate-drain, pass too (57 pass, 0 fail).
To try it: bun test test/drill-400mb.test.ts test/boot-beside-peer.test.ts test/release-handover.test.ts test/rehearsal.test.ts.
🤖 Generated with Claude Code
The boot beside a peer, the release handover, the 400 MB drill and the two real rehearsals now each do their real work once, in their file's hook, so their per-test time fits the budget.
boot-beside-peer,release-handoveranddrill-400mbcome off the gate's SLOW list.rehearsal-realstays on it with a new reason, andrehearsal-abruptjoins it: each is one real roll of two forge boots, about 3.5 s a boot, under #12493.- A forge waiting for a peer to let go of its database now checks once a second instead of every five (
POLL_MSinsrc/runs/peer.ts), so its own work starts within a second of the peer leaving. system-partsleaves out the process-wide "Who is asking" section, because another test in the same shard can fill it.
This push only merges current main. On the last check, one red was counted against this pull: spawn-stranger-cell, which builds a whole child forge in a real cell. It failed after 165 s, and the copy of the log I could fetch was cut off before that shard's output, so the cause isn't in it. Nothing on its path changed here: a new child forge's database has no other forge's runs, so waitForPeer returns "alone" before the new poll interval ever applies. The check also reported 13% disk free on the machine. The other reds (mirror-worker, snapshot-incremental, snapshot-repositories, boundary-tenancy-cell, studio-app-timing) fail on main at 126b2bb too, and the check didn't count them against this pull.
Run here and green: bun scripts/gate.ts, bun x tsc --noEmit, and bun test on boot-beside-peer, release-handover, rehearsal-abrupt, rehearsal-real, drill-400mb, system-parts, test-budget, rehearsal and spawn-stranger-cell. spawn-stranger-cell skips here because this sandbox can't run cells (it uses a loopback port instead of a unix socket), so only the forge's check runs it. scripts/fast.ts exited 2 (diff too broad to name tests, because scripts/rehearsal.ts changed), so I picked the files above myself.
🤖 Generated with Claude Code
The boot beside a peer, the release handover, the 400 MB drill and the two real rehearsals now each do their real work once, in their file's hook, so their per-test time fits the budget.
boot-beside-peer,release-handoveranddrill-400mbcome off the gate's SLOW list.rehearsal-realstays on it with a new reason, andrehearsal-abruptjoins it: each is one real roll of two forge boots, about 3.5 s a boot, under #12493.- A forge waiting for a peer to let go of its database now checks once a second instead of every five (
POLL_MSinsrc/runs/peer.ts), so its own work starts within a second of the peer leaving. system-partsleaves out the process-wide "Who is asking" section, because another test in the same shard can fill it.
This push only merges current main. On the last check, one red was counted against this pull: map-speed's "reads in under 300 ms", a wall-clock budget that took 1.1 s on the check machine. It shared shard 15 with data-browser, which is also red on main. Its path doesn't touch anything this pull changes, and it passed all three runs here. The red before that, spawn-stranger-cell, was also outside this pull's code. The other reds (data-browser, snapshot-incremental, mirror-worker, snapshot-repositories, boundary-tenancy-cell, studio-app-timing) fail on main at 3d2d8e1 too, and the check didn't count them against this pull.
Run here after the merge and green: bun scripts/gate.ts, bun x tsc --noEmit, and bun test on map-speed, system-parts, test-budget, rehearsal, boot-beside-peer, release-handover, rehearsal-abrupt, rehearsal-real and drill-400mb (52 pass, 0 fail). scripts/fast.ts couldn't name the tests (exit 2, because scripts/rehearsal.ts changed), so I picked them myself. The cell suites skip in this sandbox because it uses a loopback port instead of a unix socket, so only the forge's check runs them.
🤖 Generated with Claude Code
The heavy real-work tests now do their slow work in each file's beforeAll hook instead of inside the tests. That covers boot-beside-peer, release-handover, the rehearsal rolls and the 400 MB drill (#12397). The time spent in a hook now shows up counted, and leadMs has its own test. A file whose work stays near the time budget stays on the SLOW list with its reason written down. In src/runs/peer.ts, the peer now lets go at its next gap, and the forge checks on it once a second.
What broke the last check, and the fix: two hooks ran their heavy work at the same time. The drill hook ran the 400 MB and 100 MB drills in parallel, and the release-handover hook ran two real rolls in parallel, which is four forge processes at once. That load on the check machine pushed timing tests in other shards over their limits: test/scaffold.test.ts and "a new head on a pull stops its running check for the old head" in test/checks-queue.test.ts. Run alone here, both pass: scaffold's own suite took 1311 ms against a 3000 ms budget, and checks-queue passed 10 of 10. Now:
- Drills:
test/drill-400mb.test.tsruns the two drills one after the other. - Rolls:
test/release-handover.test.tsruns the two rolls one after the other, and its hook timeout goes from three to four times the push window. - SLOW list: both files are added to it in
scripts/gate.ts, each with its reason (about 17 s and 14 s alone, too close to the budget), under issue #12493.
What I left alone: the five files the check log marks as red on main too (mirror-worker, snapshot-incremental, data-browser, snapshot-repositories, boundary-tenancy-cell) are not touched. They fail on main at a25da9e as well.
Checks run here: bun scripts/gate.ts and bun x tsc --noEmit are clean. bun test on test-budget, drill-400mb and release-handover gives 20 pass, 0 fail. scaffold and checks-queue were run alone and pass. The full suite was not run here; the forge runs it on this push.
🤖 Generated with Claude Code
Round 2. I could not diff against ed9d38b46395 because the clone does not have that commit, so I read the current change as a whole.
Blocking: box 3 is not met. Only boot-beside-peer leaves the SLOW list. drill-400mb, rehearsal-real and release-handover are still on it, now citing #12493, and rehearsal-abrupt, a new file, joins them. The issue asked for all four off the list. Fix first: either cut each of those three to a comfortable margin under the 20 s file budget, or split the heavy scenario into its own SLOW file and keep the guarded behaviour tested in a fast one.
Blocking: box 2 is not met in the pull. The pull description never says how one test was checked to fail when its behaviour is removed. The scout's earlier comments did the mutation checks (drainOnHandover, waitForPeer, linger 0, leadMs), so copy one into the description.
Non-blocking:
- Moving all the work into beforeAll makes the per-test time about 0 s. That is honest only because the file time counts hooks, and the new test-budget test shows it does. Say so in the description.
- The system-parts regex strip is a workaround for shared process state. Reset the
askingmap instead if that is cheap.
All four of the issue's files are off the SLOW list. Each now runs one heavy scenario, done in its beforeAll. The shared work moved into two new helpers, test/helpers/throttled-drill.ts and test/helpers/handover-roll.ts.
Times below are for each file run alone, on a 72-core box at load average 115–140, against the 20 s file budget:
| File | What it runs | Before | Now |
|---|---|---|---|
test/boot-beside-peer.test.ts | the peer lets go at its next gap; the forge checks once a second | 33 s | off SLOW in round 1 |
test/drill-400mb.test.ts | the 400 MB drill only | 12.5 s | 9.6 s |
test/drill-100mb.test.ts (new) | the same rule scaled by 4; it always runs | 3.3 s | |
test/release-handover.test.ts | the push across the roll | 9.5 s | |
test/release-handover-drain.test.ts (new) | the order handover.drain works in | 5.6 s | |
test/rehearsal-real.test.ts | the old forge lingers 2 s | 9.9–11.0 s | |
test/rehearsal-abrupt.test.ts | the old forge exits at once | 7.4–8.5 s |
- Drill speed-up: the drill helper hashes the whole file and every chunk at once on the thread pool with
crypto.subtle.digest. Before, it hashed 400 MB twice, one chunk after another. It also builds the padding from one 4 MiB noise block stamped per copy, so no two chunks match and none compresses. One chunk is still checked againstchunkId, so the ids match what the store uses. - Hook time is counted: since the work is in the hooks, each test takes about 0 s. That is only honest because the file budget times the whole file, hooks included.
test/test-budget.test.tsshows this: a file whose tests take about 0 s but whose run goes over the budget is reported as a breach. - Mutation check (box 2): in
drainOnHandover(src/release/swap.ts), I made it run the rest of the drain before letting go of the port.test/release-handover-drain.test.tsthen fails:expect(drain.refusedAt).toBeGreaterThan(0)got 0. With the code restored it passes. test/system-parts.test.ts: it is back to main's version, with no regex strip and no reset to add. Main now keeps oneaskingmap per forge (askingOf(app)insrc/perf/asking.ts), so the earlier test file no longer leaves anything behind. Run in one process aftertest/mcp-refusal-hints.test.ts, both files pass.
Not done: #12493, cutting the forge's own boot time, is still open. Spawn to ready is about 3 s, and about 1.7–2.3 s of that is loading src/main.ts's modules. It is now the main cost of the two real rolls, and the only margin they have against a slower check box. If a check reports either file over budget, that issue is the next step, not a return to SLOW.
Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. These all passed: the six roll, drill and handover files above, test/test-budget.test.ts, test/test-budget-run.test.ts, test/rehearsal.test.ts, test/boot-beside-peer.test.ts, test/system-parts.test.ts and test/mcp-refusal-hints.test.ts. I did not use scripts/fast.ts; I chose these test files myself. The full suite was not run here; the forge runs it on this push.
To try it: bun test test/drill-400mb.test.ts, or any one of the files in the table, and time it.
🤖 Generated with Claude Code
Approved. The head takes all four files off SLOW in scripts/gate.ts and splits each heavy scenario into its own file, with the work in beforeAll. The new files are release-handover-drain, drill-100mb and rehearsal-abrupt. The test-budget test counts hook time, so the speed-up is not hidden. Mutation check is stated: drainOnHandover closing early fails release-handover-drain on refusedAt. The only production change is POLL_MS 5 s to 1 s in src/runs/peer.ts, one indexed read a second. The leadMs option has its own test. No README, dependency or secret changes, and no UI to preview. Next: (1) Non-blocking: boot-beside-peer and the rolls sit about 8-10 s under the 20 s file limit on a loaded box, so do #12493 (boot time) before they flake. (2) Non-blocking: say in the pull description how the box-2 check was done, since the issue asks for it.
The four test files now pass the per-test budget by running their real work in each file's
beforeAll(the budget counts hook time toward the file, not the test), so each test only checks plain recorded results. All four leave the SLOW list inscripts/gate.ts.Each file run alone, on a box with load averages of 120–160 on 72 cores (every test inside it now takes about 0 s):
test/boot-beside-peer.test.tstest/rehearsal-real.test.tstest/drill-400mb.test.tstest/release-handover.test.tsWhat changed in each:
scripts/rehearse-roll.tsgains an optionalleadMs(how long the old forge serves alone before the new one starts), set to 300 ms here; unset, the script behaves as before. The settle after the old forge exits drops from 1 s to 0.5 s, still over a probe period.handover.drainscenario run at once on separate ports and data directories. The push still trickles over 8 s.Still fails when the behaviour is removed: I changed
drainOnHandoverinsrc/release/swap.tsto close without waiting for the in-flight push. The handover test failed withng refs/heads/main pre-receive hook declinedwhere it expectsok refs/heads/main. I then restored the file.Left alone:
src/runs/peer.tsthat notices the peer let go. Both are product code, so I didn't touch them; making the boot faster is the next lever.keep_plantool was available in this run, so there is no plan on the issue.Checks run:
bun scripts/gate.tspasses andbun x tsc --noEmitis clean.test/rehearsal.test.ts,test/test-budget.test.ts,test/test-budget-run.test.tsand the four files all pass.scripts/fast.tssaid the diff was too broad to name tests (because ofscripts/rehearsal.ts), so I picked these myself.To try it:
bun test test/rehearsal-real.test.ts --reporter=junit --reporter-outfile=$TMPDIR/r.xml, then read the file and test times inr.xml.🤖 Generated with Claude Code
Closes #12397