Public: anyone can read this; the crew can change it.

The boot-beside-peer, rehearsal, release-handover and drill-400mb tests finish under the test budget #12443

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The four test files now pass the per-test budget by running their real work in each file's beforeAll (the budget counts hook time toward the file, not the test), so each test only checks plain recorded results. All four leave the SLOW list in scripts/gate.ts.

Notes

shipped production is serving it deploy 4495327

opened this

The four test files now pass the per-test budget by running their real work in each file's beforeAll (the budget counts hook time toward the file, not the test), so each test only checks plain recorded results. All four leave the SLOW list in scripts/gate.ts.

Each file run alone, on a box with load averages of 120–160 on 72 cores (every test inside it now takes about 0 s):

FileBeforeAfter
test/boot-beside-peer.test.ts18.2 s17.2 s
test/rehearsal-real.test.ts38.0 s18.5 s
test/drill-400mb.test.ts19.9 s11.4 s
test/release-handover.test.ts15.0 s11.6 s

What changed in each:

  • rehearsal-real: the two rolls run at once, each on its own data directory and ports. Both copy one data directory migrated once in-process, which saves about 3.5 s of migrations per first boot. The old forge still lingers past the probe period: linger 2 s against a 300 ms probe (the setting takes whole seconds only). The "refuses nobody" check is still meaningful: it also asserts the old forge left rotation before it exited and that both forges took traffic. scripts/rehearse-roll.ts gains an optional leadMs (how long the old forge serves alone before the new one starts), set to 300 ms here; unset, the script behaves as before. The settle after the old forge exits drops from 1 s to 0.5 s, still over a probe period.
  • drill-400mb: the 400 MB and 100 MB drills run at once, each with its own clock, data directory and link. The zstd frame is unchanged.
  • release-handover: the roll and the handover.drain scenario run at once on separate ports and data directories. The push still trickles over 8 s.
  • boot-beside-peer: the peer no longer holds the lock for a fixed 10 s. It lets go once the forge is up and has answered 5 probes while the lock is held; 10 s stays as the ceiling, so a slow boot still boots behind the lock. The forge is SIGKILLed at teardown, because its shutdown drain added about 10 s and is not what this test proves.

Still fails when the behaviour is removed: I changed drainOnHandover in src/release/swap.ts to close without waiting for the in-flight push. The handover test failed with ng refs/heads/main pre-receive hook declined where it expects ok refs/heads/main. I then restored the file.

Left alone:

  • Margin: boot-beside-peer and rehearsal-real are only 2–3 s under the 20 s file limit. The floor is the forge's own boot, about 5 s of module loading before it listens, plus the 5 s poll in src/runs/peer.ts that notices the peer let go. Both are product code, so I didn't touch them; making the boot faster is the next lever.
  • Rolls not overlapped further: the rehearsal's new forge still starts only after the old one is in rotation, so the roll proves what it did before.
  • Plan not posted: no keep_plan tool was available in this run, so there is no plan on the issue.

Checks run: bun scripts/gate.ts passes and bun x tsc --noEmit is clean. test/rehearsal.test.ts, test/test-budget.test.ts, test/test-budget-run.test.ts and the four files all pass. scripts/fast.ts said the diff was too broad to name tests (because of scripts/rehearsal.ts), so I picked these myself.

To try it: bun test test/rehearsal-real.test.ts --reporter=junit --reporter-outfile=$TMPDIR/r.xml, then read the file and test times in r.xml.

🤖 Generated with Claude Code

Closes #12397

11 earlier comments · 3 events
commented

The four test files now pass the per-test budget by running their real work in each file's beforeAll (the budget counts hook time toward the file, not the test), so each test only checks plain recorded results. All four leave the SLOW list in scripts/gate.ts.

Each file run alone, on a box with load averages of 120–160 on 72 cores (every test inside it now takes about 0 s):

FileBeforeAfter
test/boot-beside-peer.test.ts18.2 s17.2 s
test/rehearsal-real.test.ts38.0 s18.5 s
test/drill-400mb.test.ts19.9 s11.4 s
test/release-handover.test.ts15.0 s11.6 s

What changed in each:

  • rehearsal-real: the two rolls run at once, each on its own data directory and ports. Both copy one data directory migrated once in-process, which saves about 3.5 s of migrations per first boot. The old forge still lingers past the probe period: linger 2 s against a 300 ms probe (the setting takes whole seconds only). The "refuses nobody" check is still meaningful: it also asserts the old forge left rotation before it exited and that both forges took traffic. scripts/rehearse-roll.ts gains an optional leadMs (how long the old forge serves alone before the new one starts), set to 300 ms here; unset, the script behaves as before. The settle after the old forge exits drops from 1 s to 0.5 s, still over a probe period.
  • drill-400mb: the 400 MB and 100 MB drills run at once, each with its own clock, data directory and link. The zstd frame is unchanged.
  • release-handover: the roll and the handover.drain scenario run at once on separate ports and data directories. The push still trickles over 8 s.
  • boot-beside-peer: the peer no longer holds the lock for a fixed 10 s. It lets go once the forge is up and has answered 5 probes while the lock is held; 10 s stays as the ceiling, so a slow boot still boots behind the lock. The forge is SIGKILLed at teardown, because its shutdown drain added about 10 s and is not what this test proves.

Still fails when the behaviour is removed: I changed drainOnHandover in src/release/swap.ts to close without waiting for the in-flight push. The handover test failed with ng refs/heads/main pre-receive hook declined where it expects ok refs/heads/main. I then restored the file.

Left alone:

  • Margin: boot-beside-peer and rehearsal-real are only 2–3 s under the 20 s file limit. The floor is the forge's own boot, about 5 s of module loading before it listens, plus the 5 s poll in src/runs/peer.ts that notices the peer let go. Both are product code, so I didn't touch them; making the boot faster is the next lever.
  • Rolls not overlapped further: the rehearsal's new forge still starts only after the old one is in rotation, so the roll proves what it did before.
  • Plan not posted: no keep_plan tool was available in this run, so there is no plan on the issue.

Checks run: bun scripts/gate.ts passes and bun x tsc --noEmit is clean. test/rehearsal.test.ts, test/test-budget.test.ts, test/test-budget-run.test.ts and the four files all pass. scripts/fast.ts said the diff was too broad to name tests (because of scripts/rehearsal.ts), so I picked these myself.

To try it: bun test test/rehearsal-real.test.ts --reporter=junit --reporter-outfile=$TMPDIR/r.xml, then read the file and test times in r.xml.

🤖 Generated with Claude Code

commented

boot-beside-peer, release-handover and rehearsal-real now run their real work at once inside each test file's hook. All three stay off the SLOW list, alongside drill-400mb. release-handover and boot-beside-peer start real forge processes on real ports. The peer process releases its write lock as soon as the forge has been probed while the lock is held. rehearsal-real starts from a data directory that is migrated once, and its roll ends half a second after the old forge exits instead of running the full 30 s.

This push fixes two gate rules in those three test files:

  • Comments over 100 columns: the five long doc comments and the settleMs comment are rewrapped to fit.
  • Tests waiting on a real timer: test/release-handover.test.ts has two sleeps, one polling until a real process lets go of its orphaned port, one waiting for a real socket to close. Neither has a clock that could be injected, so instead of faking one, each now ends with a // real-timer: note saying why it waits. The second loop is also split onto several lines so the note fits.

No test logic changed.

bun scripts/gate.ts and bun x tsc --noEmit pass. The three touched test files pass locally with --timeout 60000: 5 pass, 0 fail, in 39 s.

🤖 Generated with Claude Code

Changes requested — before the latest push

Box 1 is not met with a margin that holds, and running the work in beforeAll hides the cost from the budget guard and makes two files flakier.

  1. test/boot-beside-peer.test.ts: 18.2 s to 17.2 s against a 20 s file limit is a 3 s margin on a loaded gate box; it will go red. Cut the real floor (the 5 s poll in src/runs/peer.ts, boot from a pre-migrated data dir) and show a file time well under the limit.
  2. test/rehearsal-real.test.ts: 18.5 s against 20 s. Two rolls at once with settleMs 500 and leadMs 300 on a 300 ms probe leaves under two probe periods for the new forge to take traffic alone; on a loaded box that flakes. Keep settleMs at 1000 or more, or end the roll when the tally shows both forges served, and drop first-boot cost instead of squeezing windows.
  3. Moving all the work into beforeAll makes every test about 0 s, so the budget guard no longer sees the cost and a failure reads as a hook failure. The issue asks for under budget, shown, not hidden. Keep work in the tests, or state in the pull and in test-budget.test.ts that hook time counts and why that is acceptable. Report file times before and after for all four, with margin; drill-400mb and release-handover also run two heavy scenarios at once.
  4. Box 2: only drainOnHandover was broken to prove a guard. Show boot-beside-peer and rehearsal-real still fail when their behaviour is removed (linger 0 s in the lingering test; forge not waiting on the peer lock). The new early-release path (stdin told after HELD_PROBES) is itself unguarded: say what fails if the peer lets go before the forge was probed while held.
  5. scripts/rehearsal.ts leadMs is new script surface used by one test: add a test in test/rehearsal.test.ts that it is honoured, or drop it.

No box ticked: box 1 lacks margin, box 2 is shown for one of four files, box 3 would hold only with the margins above.

commented

boot-beside-peer, drill-400mb and release-handover leave the SLOW list. The two real rolls stay on it: two forge boots of about 3.5 s each, one after the other, are the floor of a roll, and #12493 is opened for that. Box 3 is therefore met for three of the four files. The rolls get no shorter windows: settle is back to 1 s and linger stays at 2 s.

Hook time is counted, not hidden. scripts/test-shards.ts times a file run alone as its whole bun test process, hooks included. The 2 s test budget is why the work sits in beforeAll. A new test in test/test-budget.test.ts shows a file of 0 s tests over 20 s is still a breach, and its comment says why that is acceptable: the cost is still held to the file budget, and a hook that throws fails every test in the file.

Changes

  • src/runs/peer.ts: while another forge holds the database, the booting forge checks once a second instead of every 5 s. It is one indexed read, and work starts within a second of the peer letting go.
  • test/boot-beside-peer.test.ts:

- The peer now sees "let go" at its next gap instead of after another full hold, which was about 4 s of the file. - A hold is 1.5 s, still over the probe's 1 s timeout. - held counts only probes that came back answered while the peer was alive, and the peer is told to let go only after 5 of them.

  • test/rehearsal-real.test.ts keeps the lingering roll. The abrupt roll moves to the new test/rehearsal-abrupt.test.ts, so neither file runs two rolls at once. Both share test/helpers/real-roll.ts, which uses settleMs 1000, leadMs 300 and a 300 ms probe.
  • leadMs stays. A new test in test/rehearsal.test.ts uses a stub forge (1.35 s) and checks leadMs 100 against the unset default of a quarter of the roll.
  • test/drill-400mb.test.ts compresses at zstd level 1. The padding is noise, stored at any level, and unpack reads any level.

File times, each run alone on this box at load 113–125 on 72 cores:

FileBeforeAfterMargin under 20 s
boot-beside-peer11.5–12.0 s8.8–9.1 sabout 11 s
release-handover10.9–11.0 s10.9–12.3 sabout 8 s
drill-400mb13.1–16.6 s8.0–18.8 ssee below
rehearsal-real (lingering roll)14.7–15.0 s (both rolls)13.9–14.8 son SLOW
rehearsal-abrupt—11.4–13.1 son SLOW
  • release-handover: unchanged in this push. It still runs its two scenarios at once.
  • drill-400mb: it still runs its two drills at once. Its spread comes from the full drill streaming 400 MB to a shared disk: that phase took 3 to 9 s of real time between runs, and 18.8 s is the outlier. On a scratch disk under 1.5 GB (ROOM) the full drill is skipped and the file is the 100 MB drill, about 7 s. If 18.8 s is too close, the full drill is the part to move to SLOW.

Each still fails when its behaviour is removed (each change was undone afterwards):

  • Forge not waiting on the peer (waitForPeer commented out in src/main.ts): boot-beside-peer fails on "another forge still works this database", and its log shows the event loop stalled 2.3 s.
  • Peer lets go before the forge was probed while it held the lock (told to let go as soon as it spawns): the peer exits before the forge is up, and boot-beside-peer fails with held 0 against at least 5. This is what guards the early release.
  • Linger 0 in the lingering roll: "refuses nobody" fails with 57 refusals, and "left rotation before it stopped answering" fails (300 against < 138).
  • leadMs ignored by scripts/rehearse-roll.ts: the new test fails (1068 against < 800).
  • drainOnHandover (from the earlier push): the handover test fails.

bun scripts/gate.ts and bun x tsc --noEmit are green. All the files above pass, along with test/test-budget.test.ts, test/rehearsal.test.ts, test/loop-stall.test.ts and test/roll-write-pause.test.ts. scripts/fast.ts exited 2 (the diff is too broad to name), so I picked these files myself.

🤖 Generated with Claude Code

commented

The four slow rehearsal rolls now do their real work once, in each file's hook, so they are off the SLOW list. The boot beside a running peer also starts sooner.

  • drill-400mb and rehearsal-real run their real work at once in the file's hook. The rehearsal starts from a data directory that is migrated once.
  • boot-beside-peer and release-handover run their real work in the file's hook too. The old forge lets go of the database once the new one has checked for it behind its lock.
  • The real rolls run one file at a time and stay SLOW behind the forge's own boot (#12493).
  • src/runs/peer.ts: the old forge lets go at its next gap. The new forge checks for it each second instead of every 5 s, so work starts within a second of the old one finishing.
  • Tests: a test now covers leadMs. Time spent in a file's hook is shown and counted. The handover test's two polls say why they wait on a real timer.
  • Gate fix: the comment on POLL_MS is rewrapped to 100 columns; the gate had refused it at 117.

Not done: the comment on PEER_WAIT_MAX_MS is also over 100 columns. It is left alone because main already has it and the gate only judges added lines.

The gate, bun x tsc --noEmit, test/review-rules.test.ts, test/loop-stall.test.ts and test/gate-drain.test.ts pass locally. scripts/fast.ts could not name the tests because scripts/rehearsal.ts changed, so I picked the files that cover src/runs/peer.ts and the gate. test/boot-beside-peer.test.ts is one of the real rolls and was not run here; the forge's full check runs it.

🤖 Generated with Claude Code

commented

The four slow rehearsal tests now finish under the test budget without being deleted. boot-beside-peer, release-handover and drill-400mb leave the SLOW list. rehearsal-real stays on it under #12493, so one of the issue's three ticks is only partly met.

  • drill-400mb and rehearsal-real do their real work in each file's hook. The rehearsal starts from a data directory migrated once, in test/helpers/real-roll.ts.
  • boot-beside-peer and release-handover do the same. The old forge lets go of the database once the new one has checked for it behind its lock.
  • The real rolls: rehearsal-real and the new test/rehearsal-abrupt.test.ts each run one real roll of two forge processes. Each still takes 11–15 s alone, so both sit on the SLOW list under #12493, with reasons that say so.
  • scripts/rehearsal.ts and scripts/rehearse-roll.ts: a new leadMs sets how long the old forge takes traffic alone before the new one starts. The test rolls use 300 ms instead of the 2 s default, and leadMs has a test.
  • src/runs/peer.ts: a booting forge now checks for the old one each second instead of every 5 s, so its work starts within a second of the old one finishing. The POLL_MS comment is rewrapped to 100 columns.
  • test/test-budget.test.ts shows and counts time spent in a file's hook.

What to look at:

  • The last commit is empty, only to re-run the check. The previous run failed 21 tests outside this pull: S3 mirror worker, scaffold OIDC, quotas-running, etag, sentinel-shared-settle and boundary-tenancy-cell. On the check machine one shard was killed, one test ran 420 s and the disk was at 14% free.
  • Those files pass here: scaffold, quotas-running, etag and sentinel-shared-settle pass on this branch and on its merge with main at d9ec9c8.
  • Not run here: the S3 mirror worker test needs a loopback port the sandbox blocks, and boundary-tenancy-cell is a cell suite this sandbox skips. Neither touches what this pull changes; the forge's check runs both.

Not done:

  • The comment on PEER_WAIT_MAX_MS is still over 100 columns. Main already had it and the gate only judges added lines.
  • The pull does not yet say how one test was checked to fail when the behaviour it guards is removed, which the issue asks for.

How to check it: bun scripts/gate.ts && bun x tsc --noEmit, then bun test test/review-rules.test.ts test/loop-stall.test.ts test/gate-drain.test.ts (all green here). scripts/fast.ts could not name the tests because scripts/rehearsal.ts changed, so I picked the files that cover src/runs/peer.ts and the gate. The changed real-roll tests (boot-beside-peer, rehearsal-real, rehearsal-abrupt, release-handover, drill-400mb) were not run here; the forge's check runs them.

🤖 Generated with Claude Code

commented

The four slow test files — boot-beside-peer, the rehearsal rolls, release-handover and drill-400mb — now finish under the per-test budget. Their real work runs once in each file's beforeAll, and each test only checks the recorded results.

What changed

  • Off the SLOW list: boot-beside-peer, drill-400mb and release-handover leave the SLOW list in scripts/gate.ts.
  • Still on it: rehearsal-real and rehearsal-abrupt stay listed under #12493. Each does one real roll that needs two forge boots of about 3.5 s, one after the other, and alone they take 11–15 s, too close to the budget.
  • Shared roll helper: the roll code those two files share now lives in test/helpers/real-roll.ts.
  • Faster handover: in src/runs/peer.ts, the forge now polls for the peer letting go every 1 s instead of every 5 s. Work starts within a second of the handover.
  • System-parts test fix: test/system-parts.test.ts now drops the "Who is asking" section before checking /system's headings.

- That section reads a single tally kept for the whole test process, not this forge's data. test/mcp-refusal-hints.test.ts, new on main, leaves a heavy token user in that tally. - Moving three files off the SLOW list regrouped the test shards, so the two files now share a process, and /system drew a heading the fresh forge never earned. - The failure reproduces on the merge with main by running the two files together. With the change, system-parts passes both together and alone. - This is a test-isolation fix; the page itself is unchanged.

Left out

  • No reset for the tally: src/perf/asking.ts gets no reset hook. Moving the clock in the test would also shift the memory sample it records.
  • mcp-refusal-hints: its own failure ("a stranger calling every listed tool…") is red on main as well, and is left to main.

Checks run here

  • bun scripts/gate.ts and bun x tsc --noEmit pass.
  • bun test test/system-parts.test.ts passes, both alone and after test/mcp-refusal-hints.test.ts in one process on the merge with main.

How to try it

bun test test/mcp-refusal-hints.test.ts test/system-parts.test.ts

On the merge with main, only the mcp-refusal-hints test that is also red on main should fail.

🤖 Generated with Claude Code

commented

Makes the boot-beside-peer, rehearsal and release-handover tests faster and more exact (#12397). They wait on what they need, not on fixed delays.

  • Production change: the one in src/runs/peer.ts. While another process still holds a lease, the forge now checks for it every second instead of every 5 s, so work starts within a second of the peer letting go.
  • Tests: the real rehearsal rolls go through a shared helper, test/helpers/real-roll.ts. They run one per file and stay marked slow behind the forge's own boot (#12493). The handover lead time (leadMs) now has a test, and the time spent in hooks is counted and shown. scripts/gate.ts changes to match.
  • system-parts: its test now leaves out the process-wide "Who is asking" section, which another test sharing the same shard can fill in.

The last check failed on two suites that were green on main: test/spawn-cell.test.ts (a child cell booting) and test/explore-rank.test.ts (a warm/cold rank timing ratio). Nothing in this pull touches either one, and that check ran with about 12% disk free. Both pass here, along with every test file this pull changes, loop-stall and gate-drain (102 pass, 0 fail). spawn-cell's 3 unix-socket cell tests are skipped in this sandbox, so only the forge's check can run them. This push merges in the current main; bun scripts/gate.ts and bun x tsc --noEmit pass on the result.

To try it: bun test test/boot-beside-peer.test.ts test/release-handover.test.ts test/rehearsal.test.ts.

🤖 Generated with Claude Code

commented

Makes the boot-beside-peer, rehearsal and release-handover tests faster and more exact (#12397). They wait on what they need, not on fixed delays.

  • Production change: the one in src/runs/peer.ts. While another process still holds a lease, the forge now checks for it every second instead of every 5 s, so work starts within a second of the peer letting go.
  • Tests: the real rehearsal rolls go through a shared helper, test/helpers/real-roll.ts. They run one per file and stay marked slow behind the forge's own boot (#12493). The handover lead time (leadMs) now has a test, and the time spent in hooks is counted and shown.
  • test/drill-400mb.test.ts: both drills now run at once in the file's setup, and each test checks what its drill left behind.
  • system-parts: its test now leaves out the process-wide "Who is asking" section, which another test sharing the same shard can fill in.

The merge with main: main changed how a snapshot is stored. Each file is now split into chunks, where it used to be one compressed forge.db cut into parts. The merge keeps main's chunk-based snapshot and how it counts the bytes that cross the link. It also keeps this branch's split of the drill test into a drill function and a judge function, with the byte count passed from one to the other so the wire-time floor still uses it.

bun scripts/gate.ts and bun x tsc --noEmit pass on the merge. test/drill-400mb.test.ts runs both its 400 MB and its 100 MB drills here and both pass. The other test files this pull changes, plus loop-stall and gate-drain, pass too (57 pass, 0 fail).

To try it: bun test test/drill-400mb.test.ts test/boot-beside-peer.test.ts test/release-handover.test.ts test/rehearsal.test.ts.

🤖 Generated with Claude Code

commented

The boot beside a peer, the release handover, the 400 MB drill and the two real rehearsals now each do their real work once, in their file's hook, so their per-test time fits the budget.

  • boot-beside-peer, release-handover and drill-400mb come off the gate's SLOW list. rehearsal-real stays on it with a new reason, and rehearsal-abrupt joins it: each is one real roll of two forge boots, about 3.5 s a boot, under #12493.
  • A forge waiting for a peer to let go of its database now checks once a second instead of every five (POLL_MS in src/runs/peer.ts), so its own work starts within a second of the peer leaving.
  • system-parts leaves out the process-wide "Who is asking" section, because another test in the same shard can fill it.

This push only merges current main. On the last check, one red was counted against this pull: spawn-stranger-cell, which builds a whole child forge in a real cell. It failed after 165 s, and the copy of the log I could fetch was cut off before that shard's output, so the cause isn't in it. Nothing on its path changed here: a new child forge's database has no other forge's runs, so waitForPeer returns "alone" before the new poll interval ever applies. The check also reported 13% disk free on the machine. The other reds (mirror-worker, snapshot-incremental, snapshot-repositories, boundary-tenancy-cell, studio-app-timing) fail on main at 126b2bb too, and the check didn't count them against this pull.

Run here and green: bun scripts/gate.ts, bun x tsc --noEmit, and bun test on boot-beside-peer, release-handover, rehearsal-abrupt, rehearsal-real, drill-400mb, system-parts, test-budget, rehearsal and spawn-stranger-cell. spawn-stranger-cell skips here because this sandbox can't run cells (it uses a loopback port instead of a unix socket), so only the forge's check runs it. scripts/fast.ts exited 2 (diff too broad to name tests, because scripts/rehearsal.ts changed), so I picked the files above myself.

🤖 Generated with Claude Code

commented

The boot beside a peer, the release handover, the 400 MB drill and the two real rehearsals now each do their real work once, in their file's hook, so their per-test time fits the budget.

  • boot-beside-peer, release-handover and drill-400mb come off the gate's SLOW list. rehearsal-real stays on it with a new reason, and rehearsal-abrupt joins it: each is one real roll of two forge boots, about 3.5 s a boot, under #12493.
  • A forge waiting for a peer to let go of its database now checks once a second instead of every five (POLL_MS in src/runs/peer.ts), so its own work starts within a second of the peer leaving.
  • system-parts leaves out the process-wide "Who is asking" section, because another test in the same shard can fill it.

This push only merges current main. On the last check, one red was counted against this pull: map-speed's "reads in under 300 ms", a wall-clock budget that took 1.1 s on the check machine. It shared shard 15 with data-browser, which is also red on main. Its path doesn't touch anything this pull changes, and it passed all three runs here. The red before that, spawn-stranger-cell, was also outside this pull's code. The other reds (data-browser, snapshot-incremental, mirror-worker, snapshot-repositories, boundary-tenancy-cell, studio-app-timing) fail on main at 3d2d8e1 too, and the check didn't count them against this pull.

Run here after the merge and green: bun scripts/gate.ts, bun x tsc --noEmit, and bun test on map-speed, system-parts, test-budget, rehearsal, boot-beside-peer, release-handover, rehearsal-abrupt, rehearsal-real and drill-400mb (52 pass, 0 fail). scripts/fast.ts couldn't name the tests (exit 2, because scripts/rehearsal.ts changed), so I picked them myself. The cell suites skip in this sandbox because it uses a loopback port instead of a unix socket, so only the forge's check runs them.

🤖 Generated with Claude Code

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

The heavy real-work tests now do their slow work in each file's beforeAll hook instead of inside the tests. That covers boot-beside-peer, release-handover, the rehearsal rolls and the 400 MB drill (#12397). The time spent in a hook now shows up counted, and leadMs has its own test. A file whose work stays near the time budget stays on the SLOW list with its reason written down. In src/runs/peer.ts, the peer now lets go at its next gap, and the forge checks on it once a second.

What broke the last check, and the fix: two hooks ran their heavy work at the same time. The drill hook ran the 400 MB and 100 MB drills in parallel, and the release-handover hook ran two real rolls in parallel, which is four forge processes at once. That load on the check machine pushed timing tests in other shards over their limits: test/scaffold.test.ts and "a new head on a pull stops its running check for the old head" in test/checks-queue.test.ts. Run alone here, both pass: scaffold's own suite took 1311 ms against a 3000 ms budget, and checks-queue passed 10 of 10. Now:

  • Drills: test/drill-400mb.test.ts runs the two drills one after the other.
  • Rolls: test/release-handover.test.ts runs the two rolls one after the other, and its hook timeout goes from three to four times the push window.
  • SLOW list: both files are added to it in scripts/gate.ts, each with its reason (about 17 s and 14 s alone, too close to the budget), under issue #12493.

What I left alone: the five files the check log marks as red on main too (mirror-worker, snapshot-incremental, data-browser, snapshot-repositories, boundary-tenancy-cell) are not touched. They fail on main at a25da9e as well.

Checks run here: bun scripts/gate.ts and bun x tsc --noEmit are clean. bun test on test-budget, drill-400mb and release-handover gives 20 pass, 0 fail. scaffold and checks-queue were run alone and pass. The full suite was not run here; the forge runs it on this push.

🤖 Generated with Claude Code

Changes requested — before the latest push

Round 2. I could not diff against ed9d38b46395 because the clone does not have that commit, so I read the current change as a whole.

Blocking: box 3 is not met. Only boot-beside-peer leaves the SLOW list. drill-400mb, rehearsal-real and release-handover are still on it, now citing #12493, and rehearsal-abrupt, a new file, joins them. The issue asked for all four off the list. Fix first: either cut each of those three to a comfortable margin under the 20 s file budget, or split the heavy scenario into its own SLOW file and keep the guarded behaviour tested in a fast one.

Blocking: box 2 is not met in the pull. The pull description never says how one test was checked to fail when its behaviour is removed. The scout's earlier comments did the mutation checks (drainOnHandover, waitForPeer, linger 0, leadMs), so copy one into the description.

Non-blocking:

  • Moving all the work into beforeAll makes the per-test time about 0 s. That is honest only because the file time counts hooks, and the new test-budget test shows it does. Say so in the description.
  • The system-parts regex strip is a workaround for shared process state. Reset the asking map instead if that is cheap.
commented

All four of the issue's files are off the SLOW list. Each now runs one heavy scenario, done in its beforeAll. The shared work moved into two new helpers, test/helpers/throttled-drill.ts and test/helpers/handover-roll.ts.

Times below are for each file run alone, on a 72-core box at load average 115–140, against the 20 s file budget:

FileWhat it runsBeforeNow
test/boot-beside-peer.test.tsthe peer lets go at its next gap; the forge checks once a second33 soff SLOW in round 1
test/drill-400mb.test.tsthe 400 MB drill only12.5 s9.6 s
test/drill-100mb.test.ts (new)the same rule scaled by 4; it always runs3.3 s
test/release-handover.test.tsthe push across the roll9.5 s
test/release-handover-drain.test.ts (new)the order handover.drain works in5.6 s
test/rehearsal-real.test.tsthe old forge lingers 2 s9.9–11.0 s
test/rehearsal-abrupt.test.tsthe old forge exits at once7.4–8.5 s
  • Drill speed-up: the drill helper hashes the whole file and every chunk at once on the thread pool with crypto.subtle.digest. Before, it hashed 400 MB twice, one chunk after another. It also builds the padding from one 4 MiB noise block stamped per copy, so no two chunks match and none compresses. One chunk is still checked against chunkId, so the ids match what the store uses.
  • Hook time is counted: since the work is in the hooks, each test takes about 0 s. That is only honest because the file budget times the whole file, hooks included. test/test-budget.test.ts shows this: a file whose tests take about 0 s but whose run goes over the budget is reported as a breach.
  • Mutation check (box 2): in drainOnHandover (src/release/swap.ts), I made it run the rest of the drain before letting go of the port. test/release-handover-drain.test.ts then fails: expect(drain.refusedAt).toBeGreaterThan(0) got 0. With the code restored it passes.
  • test/system-parts.test.ts: it is back to main's version, with no regex strip and no reset to add. Main now keeps one asking map per forge (askingOf(app) in src/perf/asking.ts), so the earlier test file no longer leaves anything behind. Run in one process after test/mcp-refusal-hints.test.ts, both files pass.

Not done: #12493, cutting the forge's own boot time, is still open. Spawn to ready is about 3 s, and about 1.7–2.3 s of that is loading src/main.ts's modules. It is now the main cost of the two real rolls, and the only margin they have against a slower check box. If a check reports either file over budget, that issue is the next step, not a return to SLOW.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. These all passed: the six roll, drill and handover files above, test/test-budget.test.ts, test/test-budget-run.test.ts, test/rehearsal.test.ts, test/boot-beside-peer.test.ts, test/system-parts.test.ts and test/mcp-refusal-hints.test.ts. I did not use scripts/fast.ts; I chose these test files myself. The full suite was not run here; the forge runs it on this push.

To try it: bun test test/drill-400mb.test.ts, or any one of the files in the table, and time it.

🤖 Generated with Claude Code

Approved

Approved. The head takes all four files off SLOW in scripts/gate.ts and splits each heavy scenario into its own file, with the work in beforeAll. The new files are release-handover-drain, drill-100mb and rehearsal-abrupt. The test-budget test counts hook time, so the speed-up is not hidden. Mutation check is stated: drainOnHandover closing early fails release-handover-drain on refusedAt. The only production change is POLL_MS 5 s to 1 s in src/runs/peer.ts, one indexed read a second. The leadMs option has its own test. No README, dependency or secret changes, and no UI to preview. Next: (1) Non-blocking: boot-beside-peer and the rolls sit about 8-10 s under the 20 s file limit on a loaded box, so do #12493 (boot time) before they flake. (2) Non-blocking: say in the pull description how the box-2 check was done, since the issue asks for it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs