Public: anyone can read this; the crew can change it.

The gate answers in a minute, not four: the check runs its test files in parallel shards #306

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`bun run check` now type-checks once, then runs the gate's test files as parallel `bun test` shards. On this machine it went from **266 s to 102 s and 109 s**, green on two back-to-back runs from a fresh clone; the old serial command was timed on the same machine. It is not yet a minute: `test/crew.test.ts` alone takes about 50 s and shares a shard with other files.

shipped production is serving it deploy 674862e

opened this

bun run check now type-checks once, then runs the gate's test files as parallel bun test shards. On this machine it went from 266 s to 102 s and 109 s, green on two back-to-back runs from a fresh clone; the old serial command was timed on the same machine. It is not yet a minute: test/crew.test.ts alone takes about 50 s and shares a shard with other files.

What moved where

  • forge.json check.run is now bun x tsc --noEmit && bun scripts/test-shards.ts --timeout 60000 <files…>. The file list lives only there, and scripts/check.ts still just runs that string.
  • scripts/shards.ts holds the logic and spawns nothing:

- shardCount: one shard per CPU the process may use (a cgroup quota counts), at most 8, never more than there are files. FORGE_CHECK_SHARDS overrides it, and anything that isn't a whole number from 1 is refused. - splitFiles: the largest test file (by bytes) goes to the shard with the least so far. The slices never overlap and together hold every file, and the same list always gives the same split. - tally: reads every shard's output. Green ends with one line (Ran 435 tests across 23 files in 86.0s (4 shards).). Red lists every (fail) from every shard, names any shard that exited non-zero without naming a test by its files, and exits 1.

  • scripts/test-shards.ts spawns one bun test per slice. It passes exact ./ paths, not name filters, so a slice can't match another shard's files. Each shard's output is written out whole when that shard finishes. It refuses a named file that doesn't exist, and passes SIGINT/SIGTERM on to its children.
  • test/shards.test.ts checks that the slices don't overlap and cover the list for shard counts 1–40, the CPU count and override rules, and that one red shard makes the whole run red with every failure named. It is part of the gate now.
  • tsconfig.json now includes scripts/, so the runner is type-checked. README.md describes the sharded check.

Measured here (a 4-CPU cgroup on an 80-core host):

CommandWall time
Old serial check.run266 s
Default (4 shards), run 1102 s
Default (4 shards), run 2109 s
FORGE_CHECK_SHARDS=896 s

About 15 s of each run is tsc.

Things this found

  • test/steer.test.ts failed when running alongside other files. It slept a fixed 300 ms before posting its comment, but a run is marked running before the fake agent starts, so on a loaded machine the comment arrived too early and a second run queued. The test was wrong, not the code: it now waits for the session s2 line in the run log.
  • test/repos.test.ts was in the gate list but doesn't exist; bun test treated it as a filter that matched nothing, which is why logs said 22 files. I removed it, and the runner now fails on any missing file.
  • The SSH port was not a problem. The sshPort = 2222 lines in apps, studio and ssh-site only set a config value for display text; tests never bind that port, so I left them alone.

Not done: better balancing. Byte size is a weak guide to runtime (crew.test.ts is small but slow). A pool that hands out files one at a time would get closer to a minute, but it gives up fixed slices. The release workflow and chart are untouched.

To try it: bun run check, or FORGE_CHECK_SHARDS=2 bun run check to turn it down.

🤖 Generated with Claude Code

Closes #277

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs