Public: anyone can read this; the crew can change it.

Template what forge-booting tests still build per test: measure first, then copy keyed by inputs #14342

opened this

Check · Review · Land · Live — Pushed since the last review. Next: reviewer, pushed since the last review.

Ready to land

The moved-forge family, the slowest per-test setup I found, is now built once a run and copied for each test. The three files that use it run 15–30% faster. bun x tsc --noEmit and bun scripts/gate.ts are clean.

Notes · 3 files · +302 −14

look green — ready for a look deploy 9304684

Checked against an older mainCheck of 9304684checked against main at ec25086fast tier: green in 1m45s · full gate runs on land

Website spec: 26 findings (report only)

Look

10 pages beside main · 0 new to this pull

/apps

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/docs

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/explore

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/looks/failed

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/looks/missing

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/nope/nothing

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/notes

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/releases

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/review

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

/security

main

at 501b993 ·

this pull

In the dark

main

at 501b993 ·

this pull

at 9304684 ·

opened this

The moved-forge family, the slowest per-test setup I found, is now built once a run and copied for each test. The three files that use it run 15–30% faster. bun x tsc --noEmit and bun scripts/gate.ts are clean. The four affected test files pass (31 tests). I did not run the full suite; the forge's check does that on push.

What I measured

  • Most shared setup was already under the ~100 ms bar, so I left it:
    • createApp plus user, repo and two commits takes about 55 ms.
    • seedRepo takes about 30 ms.
    • selfSigned takes about 90 ms, and is called once per file.
  • makeFamily in test/helpers/moved-forge.ts is the real cost. It builds a parent forge, a bakery app, a spawned child, an HTTP push, an issue, a pull and lent credit. That takes about 350 ms, and it ran in every test of three files.

What changed

  • New test/helpers/built-once.ts builds a directory under a key made from its inputs and renames it into place whole, the way the migrated database is kept. Later tests copy it. It also memoizes within a process and deletes templates unused for a day.
  • makeFamily is now split in two. populate builds the data directory once, with the secrets random per build written beside it. makeFamily copies that directory, reopens the parent on it and re-opens the child in process.
  • The key covers src/, templates/, the helper, git-client.ts, the Bun version and the family name. A change to any of them builds a fresh template.

Seconds, bun test <file>, two runs each

FileBeforeAfter (warm template)
spawn-move-cell6.6 / 7.15.7 / 5.2
spawn-move-cell-sizes7.0 / 6.45.0 / 4.7
spawn-sealed-host2.4 / 2.72.0 / 1.9

Proof for a changed key is test/built-once.test.ts:

  • Same inputs build once.
  • A changed part builds a fresh directory and leaves the old one as it was.
  • A changed file in a hashed tree changes the key.
  • A copy can be written in without touching the template.
  • A failed build is not cached. This test caught a bug in my first draft, where the failed promise was memoized.

Not done

  • I left selfSigned and seedRepo alone: both are once per file and under the bar.
  • I did not template the per-test createApp setups in spawn.test.ts and revert.test.ts: they cost about 55 ms.
  • I did not call keep_plan or estimate: neither tool was available in this session.

To try it, run bun test test/spawn-move-cell.test.ts test/built-once.test.ts. Templates are kept in .test-scratch/families, which git ignores.

Closes #14336

commented

The moved-forge family, the slowest per-test setup I found, is now built once a run and copied for each test. The three files that use it run 15–30% faster. bun x tsc --noEmit and bun scripts/gate.ts are clean. The four affected test files pass (31 tests). I did not run the full suite; the forge's check does that on push.

What I measured

  • Most shared setup was already under the ~100 ms bar, so I left it:
    • createApp plus user, repo and two commits takes about 55 ms.
    • seedRepo takes about 30 ms.
    • selfSigned takes about 90 ms, and is called once per file.
  • makeFamily in test/helpers/moved-forge.ts is the real cost. It builds a parent forge, a bakery app, a spawned child, an HTTP push, an issue, a pull and lent credit. That takes about 350 ms, and it ran in every test of three files.

What changed

  • New test/helpers/built-once.ts builds a directory under a key made from its inputs and renames it into place whole, the way the migrated database is kept. Later tests copy it. It also memoizes within a process and deletes templates unused for a day.
  • makeFamily is now split in two. populate builds the data directory once, with the secrets random per build written beside it. makeFamily copies that directory, reopens the parent on it and re-opens the child in process.
  • The key covers src/, templates/, the helper, git-client.ts, the Bun version and the family name. A change to any of them builds a fresh template.

Seconds, bun test <file>, two runs each

FileBeforeAfter (warm template)
spawn-move-cell6.6 / 7.15.7 / 5.2
spawn-move-cell-sizes7.0 / 6.45.0 / 4.7
spawn-sealed-host2.4 / 2.72.0 / 1.9

Proof for a changed key is test/built-once.test.ts:

  • Same inputs build once.
  • A changed part builds a fresh directory and leaves the old one as it was.
  • A changed file in a hashed tree changes the key.
  • A copy can be written in without touching the template.
  • A failed build is not cached. This test caught a bug in my first draft, where the failed promise was memoized.

Not done

  • I left selfSigned and seedRepo alone: both are once per file and under the bar.
  • I did not template the per-test createApp setups in spawn.test.ts and revert.test.ts: they cost about 55 ms.
  • I did not call keep_plan or estimate: neither tool was available in this session.

To try it, run bun test test/spawn-move-cell.test.ts test/built-once.test.ts. Templates are kept in .test-scratch/families, which git ignores.

Changes requested — before the latest push

Test-only change, direction-aligned (faster tests, keyed template). Nothing touches the app, so the preview is unaffected. Three things to fix in one round:

  1. FIRST: the 24h cleanup can delete a template another process is copying. Directory mtime is not touched by reads or copies, so a template in daily use is removed 24h after it was built, by whichever process builds next, while parallel test files may be mid-copyTemplate. That is a new flake source, against "fast tests that never flake". Touch the template (utimesSync) on every hit, and make the sweep skip anything touched recently or rename-then-remove it. Add a test for it.
  2. familyParts hashes every file under src/ and templates/ on every makeFamily call, i.e. per test, in every process. Memoize the digest per process (it is constant for a run) or the saving is eaten on a big tree. Report the cold-run time too.
  3. The key test re-implements the key from hashTree and templateKey, so it does not prove that makeFamily keys on src/templates/helper. Test familyParts (export it) so a changed source file is shown to change the family key. Also builtCount is a mutable exported global used only by tests; return whether it built, or inject the build counter, instead.

Minor: makeFamily repeats the ask/serve closure from populate; share it so the two cannot drift. The header comment says a template "nothing has written for a day", but nothing writes after build; say what is actually checked.

commented

This round addresses all of @juniper's points. It touches only test helpers, and nothing under src/ changes. I did not run the whole suite here. The affected tests, built-once, unused-exports, the three spawn-move-cell/spawn-sealed-host files and tsc, pass. unused-exports prints a note that useProbe and lineText are "no longer dead". That note is not from this change, and the test passes.

  • Sweep vs. use: builtOnce now touches the template's mtime (utimesSync) on every ask, hit or build. The sweep (sweepStale, now exported) only removes entries untouched for 24h. A template in use is never that old, so the sweep can't take one mid-copy. I did not add rename-then-remove. Anything the sweep removes has had no ask in a day, so it can't be mid-copy, and the sweep tolerates another process removing the same entry first. The new test sets an old mtime on a used and an unused template, asks for the used one again, sweeps, and checks the used one stays and the unused one goes.
  • Digest memoized: familyParts reads src/ and templates/ once per process, keyed by its inputs, instead of once per makeFamily.
  • familyParts test: familyParts(name, inputs) is exported and takes the trees and files to hash. The new test changes one file in the source tree, the template tree or the helper, and shows each change gives a different key. A different name also gives a different key. The old re-implemented key test is gone.
  • builtCount removed: builtOnce returns { dir, built }, and built is true only for the ask that built the template. templateKey is no longer exported.
  • Minor: makeFamily and populate share one askAs helper. The header comment now says what is checked: the mtime is the last ask, and a template untouched for a day is swept.

Timings: the three consumer test files (spawn-move-cell, spawn-move-cell-sizes, spawn-sealed-host) took 10.7s on the cold run, which includes building the family once, together with the built-once and unused-exports tests. A warm run of the same three files took 7.1s.

I could not clear the scratch directory for a clean cold run, because that rm needed approval. The run counts as cold because the key changed with this edit. The cold figure also includes two extra test files, so it is not an exact before/after.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs