Public: anyone can read this; the crew can change it.
The test runner stops killing healthy shards, and each test file is isolated #7879
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/7788-the-test-runner-stops-killing-healthy-sh → main 2 commits fork 5 threads · 5 unresolved
docs/research/test-runner.md added +63
| @@ -0,0 +1,63 @@ | ||
| 1 | # Test runner spike: `--isolate`, `--parallel`, `--changed` (#7788) | |
| 2 | ||
| 3 | Bun 1.3.14, the real suite (1738 files), measured 2026-11-02 on an 80-core box. | |
| 4 | The first half of #7788 (every shard runs to the end, a crash outside any test | |
| 5 | re-runs its shard once) landed in #7817; this records the numbers for the rest. | |
| 6 | ||
| 7 | ## `--isolate` and `--parallel` | |
| 8 | ||
| 9 | A sample of 59 test files (every 29th), 357 tests, all green in every mode: | |
| 10 | ||
| 11 | | mode | wall time | crashes ("closed database", unhandled error) | | |
| 12 | |---|---|---| | |
| 13 | | one `bun test`, no flags | 85 s | 0 | | |
| 14 | | one `bun test --isolate` | 155 s | 0 | | |
| 15 | | one `bun test --parallel=8` | 36 s | 0 | | |
| 16 | | `scripts/test-shards.ts`, 8 shards (one `bun test` per file) | 34 s | 0 | | |
| 17 | ||
| 18 | - `--isolate` costs 1.8x the time of a shared global and, run alone, adds | |
The crash column is 0 everywhere, so the sample cannot show whether |
||
| 19 | nothing the runner lacks: it already spawns one `bun test` per file, so a | |
| 20 | leaked timer or closed socket dies with its process. | |
| 21 | - `--parallel=8` is as fast as the runner (36 s against 34 s) and no safer: it | |
| 22 | is the same one-file-per-worker isolation, but loses the runner's weighing | |
| 23 | by recorded time, the crash re-run, the per-shard log and the impact map. | |
| 24 | - Each mode ran once: there is no spread. 34 s against 36 s is a tie, not a | |
| 25 | win; only the 85 s / 155 s gap to the others is large enough to read. | |
| 26 | - No run crashed on this sample, so it cannot show whether `--isolate` removes | |
| 27 | the "closed database" class. That question is unanswered: it was not | |
| 28 | reproduced under load. The sample is 59 of 1738 files (3%), so a slow or | |
| 29 | crash-prone file outside it is unmeasured. What is known is only that the | |
| 30 | runner's re-run absorbs a crash outside any test, whatever its cause. | |
| 31 | ||
| 32 | Adopted: nothing. Replaces nothing, so nothing is deleted. | |
| 33 | ||
| 34 | ## `--changed` against the current selection | |
| 35 | ||
| 36 | `scripts/compare-changed.ts --pulls 15` (the last 15 merged pulls; the fast | |
outdated
Only 15 pulls, and the difference is 3 vs 2 missed reds. Run the default 50 and paste the script's verdict output; otherwise the 'neither' verdict rests on one red. |
||
| 37 | tier's pick against `bun test --changed=<parent>`, against the files red in | |
| 38 | the pulls' logs): | |
| 39 | ||
| 40 | | | files selected (sum) | missed reds | | |
| 41 | |---|---|---| | |
| 42 | | fast tier (`fastSet`) | 15 721 | 3 | | |
outdated
|
||
| 43 | | `--changed` | 19 769 | 2 | | |
| 44 | ||
outdated
'the impact map covers by recording what each file reads' is asserted without evidence. Point to the code (src/checks/impact.ts) or add a probe showing a changed .sql selects its tests. |
||
| 45 | `--changed` picks more files and still misses reds, and the script's verdict | |
| 46 | is `neither`. The window is 15 pulls, not the script's default 50, and the gap | |
| 47 | is 3 against 2 missed reds, so the verdict rests on very few reds; the full 50 | |
| 48 | was not run. | |
| 49 | ||
| 50 | Probes in a scratch repo (Bun 1.3.14): | |
| 51 | - an edit to a statically imported `.ts` selects the test; | |
| 52 | - a constant expression, `import("../src/" + "b.ts")`, is followed (it is not | |
| 53 | really dynamic); | |
| 54 | - a computed path, `import("../src/" + name + ".ts")`, is not followed: editing | |
| 55 | `src/b.ts` and `src/c.ts` alike printed "no test files are affected"; | |
| 56 | - an edit to a `.sql` file read with `readFileSync` selects nothing (0 files). | |
| 57 | ||
| 58 | The impact map (`src/checks/impact.ts`) is built differently: it records the | |
| 59 | `src/**` files each test executed, from coverage of the full run, so a computed | |
| 60 | import that ran is in it. It reads lcov, so a data file such as `.sql` is | |
| 61 | covered only if something else records it; that was not probed here. | |
| 62 | ||
| 63 | Adopted: nothing; the fast tier stays. |
One run per mode, no spread. Repeat each (3 runs) and give min/median, or call 34 vs 36 s a tie explicitly.