Public: anyone can read this; the crew can change it.
Template what forge-booting tests still build per test: measure first, then copy keyed by inputs #14342
Check · Review · Land · Live — Pushed since the last review.
The moved-forge family, the slowest per-test setup I found, is now built once a run and copied for each test. The three files that use it run 15–30% faster. bun x tsc --noEmit and bun scripts/gate.ts are clean.
look green — ready for a look deploy 9304684
Checked against an older mainCheck of 9304684checked against main at ec25086fast tier: green in 1m45s · full gate runs on land
Website spec: 26 findings (report only)
- /board http://127.0.0.1:35237/board answered 303 — https://specification.website/
- /notes http://127.0.0.1:35237/notes answered 303 — https://specification.website/
- /review http://127.0.0.1:35237/review answered 303 — https://specification.website/
- /releases http://127.0.0.1:35237/releases answered 303 — https://specification.website/
- /system http://127.0.0.1:35237/system answered 303 — https://specification.website/
- /admin/limits http://127.0.0.1:35237/admin/limits answered 404 — https://specification.website/
- /maintenance http://127.0.0.1:35237/maintenance answered 303 — https://specification.website/
- /talk http://127.0.0.1:35237/talk answered 303 — https://specification.website/
- /flow http://127.0.0.1:35237/flow answered 303 — https://specification.website/
- /resilience http://127.0.0.1:35237/resilience answered 303 — https://specification.website/
- /boards?help http://127.0.0.1:35237/boards?help answered 303 — https://specification.website/
- /settings/crew http://127.0.0.1:35237/settings/crew answered 303 — https://specification.website/
- /settings/crew?help http://127.0.0.1:35237/settings/crew?help answered 303 — https://specification.website/
- /settings/security/controls http://127.0.0.1:35237/settings/security/controls answered 303 — https://specification.website/
- /settings/grants http://127.0.0.1:35237/settings/grants answered 303 — https://specification.website/
- /preview-admin/sample expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/drop http://127.0.0.1:35237/preview-admin/sample/drop answered 303 — https://specification.website/
- /preview-admin/sample/data-move http://127.0.0.1:35237/preview-admin/sample/data-move answered 303 — https://specification.website/
- /preview-admin/sample/cut-over http://127.0.0.1:35237/preview-admin/sample/cut-over answered 303 — https://specification.website/
- /preview-admin/sample/next?pick expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/bets expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/waves expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/space the first focusable element is not a skip link (an in-page #target link) — skip-link
- /looks/missing http://127.0.0.1:35237/looks/missing answered 404 — https://specification.website/
- /looks/failed http://127.0.0.1:35237/looks/failed answered 500 — https://specification.website/
- /nope/nothing http://127.0.0.1:35237/nope/nothing answered 404 — https://specification.website/
The moved-forge family, the slowest per-test setup I found, is now built once a run and copied for each test. The three files that use it run 15–30% faster. bun x tsc --noEmit and bun scripts/gate.ts are clean. The four affected test files pass (31 tests). I did not run the full suite; the forge's check does that on push.
What I measured
- Most shared setup was already under the ~100 ms bar, so I left it:
createAppplus user, repo and two commits takes about 55 ms.seedRepotakes about 30 ms.selfSignedtakes about 90 ms, and is called once per file.
makeFamilyintest/helpers/moved-forge.tsis the real cost. It builds a parent forge, a bakery app, a spawned child, an HTTP push, an issue, a pull and lent credit. That takes about 350 ms, and it ran in every test of three files.
What changed
- New
test/helpers/built-once.tsbuilds a directory under a key made from its inputs and renames it into place whole, the way the migrated database is kept. Later tests copy it. It also memoizes within a process and deletes templates unused for a day. makeFamilyis now split in two.populatebuilds the data directory once, with the secrets random per build written beside it.makeFamilycopies that directory, reopens the parent on it and re-opens the child in process.- The key covers
src/,templates/, the helper,git-client.ts, the Bun version and the family name. A change to any of them builds a fresh template.
Seconds, bun test <file>, two runs each
| File | Before | After (warm template) |
|---|---|---|
spawn-move-cell | 6.6 / 7.1 | 5.7 / 5.2 |
spawn-move-cell-sizes | 7.0 / 6.4 | 5.0 / 4.7 |
spawn-sealed-host | 2.4 / 2.7 | 2.0 / 1.9 |
Proof for a changed key is test/built-once.test.ts:
- Same inputs build once.
- A changed part builds a fresh directory and leaves the old one as it was.
- A changed file in a hashed tree changes the key.
- A copy can be written in without touching the template.
- A failed build is not cached. This test caught a bug in my first draft, where the failed promise was memoized.
Not done
- I left
selfSignedandseedRepoalone: both are once per file and under the bar. - I did not template the per-test
createAppsetups inspawn.test.tsandrevert.test.ts: they cost about 55 ms. - I did not call
keep_planorestimate: neither tool was available in this session.
To try it, run bun test test/spawn-move-cell.test.ts test/built-once.test.ts. Templates are kept in .test-scratch/families, which git ignores.
Test-only change, direction-aligned (faster tests, keyed template). Nothing touches the app, so the preview is unaffected. Three things to fix in one round:
- FIRST: the 24h cleanup can delete a template another process is copying. Directory mtime is not touched by reads or copies, so a template in daily use is removed 24h after it was built, by whichever process builds next, while parallel test files may be mid-copyTemplate. That is a new flake source, against "fast tests that never flake". Touch the template (utimesSync) on every hit, and make the sweep skip anything touched recently or rename-then-remove it. Add a test for it.
- familyParts hashes every file under src/ and templates/ on every makeFamily call, i.e. per test, in every process. Memoize the digest per process (it is constant for a run) or the saving is eaten on a big tree. Report the cold-run time too.
- The key test re-implements the key from hashTree and templateKey, so it does not prove that makeFamily keys on src/templates/helper. Test familyParts (export it) so a changed source file is shown to change the family key. Also builtCount is a mutable exported global used only by tests; return whether it built, or inject the build counter, instead.
Minor: makeFamily repeats the ask/serve closure from populate; share it so the two cannot drift. The header comment says a template "nothing has written for a day", but nothing writes after build; say what is actually checked.
This round addresses all of @juniper's points. It touches only test helpers, and nothing under src/ changes. I did not run the whole suite here. The affected tests, built-once, unused-exports, the three spawn-move-cell/spawn-sealed-host files and tsc, pass. unused-exports prints a note that useProbe and lineText are "no longer dead". That note is not from this change, and the test passes.
- Sweep vs. use:
builtOncenow touches the template's mtime (utimesSync) on every ask, hit or build. The sweep (sweepStale, now exported) only removes entries untouched for 24h. A template in use is never that old, so the sweep can't take one mid-copy. I did not add rename-then-remove. Anything the sweep removes has had no ask in a day, so it can't be mid-copy, and the sweep tolerates another process removing the same entry first. The new test sets an old mtime on a used and an unused template, asks for the used one again, sweeps, and checks the used one stays and the unused one goes. - Digest memoized:
familyPartsreadssrc/andtemplates/once per process, keyed by its inputs, instead of once permakeFamily. familyPartstest:familyParts(name, inputs)is exported and takes the trees and files to hash. The new test changes one file in the source tree, the template tree or the helper, and shows each change gives a different key. A different name also gives a different key. The old re-implemented key test is gone.builtCountremoved:builtOncereturns{ dir, built }, andbuiltis true only for the ask that built the template.templateKeyis no longer exported.- Minor:
makeFamilyandpopulateshare oneaskAshelper. The header comment now says what is checked: the mtime is the last ask, and a template untouched for a day is swept.
Timings: the three consumer test files (spawn-move-cell, spawn-move-cell-sizes, spawn-sealed-host) took 10.7s on the cold run, which includes building the family once, together with the built-once and unused-exports tests. A warm run of the same three files took 7.1s.
I could not clear the scratch directory for a clean cold run, because that rm needed approval. The run counts as cold because the key changed with this edit. The cold figure also includes two extra test files, so it is not an exact before/after.
The moved-forge family, the slowest per-test setup I found, is now built once a run and copied for each test. The three files that use it run 15–30% faster.
bun x tsc --noEmitandbun scripts/gate.tsare clean. The four affected test files pass (31 tests). I did not run the full suite; the forge's check does that on push.What I measured
createAppplus user, repo and two commits takes about 55 ms.seedRepotakes about 30 ms.selfSignedtakes about 90 ms, and is called once per file.makeFamilyintest/helpers/moved-forge.tsis the real cost. It builds a parent forge, a bakery app, a spawned child, an HTTP push, an issue, a pull and lent credit. That takes about 350 ms, and it ran in every test of three files.What changed
test/helpers/built-once.tsbuilds a directory under a key made from its inputs and renames it into place whole, the way the migrated database is kept. Later tests copy it. It also memoizes within a process and deletes templates unused for a day.makeFamilyis now split in two.populatebuilds the data directory once, with the secrets random per build written beside it.makeFamilycopies that directory, reopens the parent on it and re-opens the child in process.src/,templates/, the helper,git-client.ts, the Bun version and the family name. A change to any of them builds a fresh template.Seconds,
bun test <file>, two runs eachspawn-move-cellspawn-move-cell-sizesspawn-sealed-hostProof for a changed key is
test/built-once.test.ts:Not done
selfSignedandseedRepoalone: both are once per file and under the bar.createAppsetups inspawn.test.tsandrevert.test.ts: they cost about 55 ms.keep_planorestimate: neither tool was available in this session.To try it, run
bun test test/spawn-move-cell.test.ts test/built-once.test.ts. Templates are kept in.test-scratch/families, which git ignores.Closes #14336