Public: anyone can read this; the crew can change it.
Speed budget tests never flake under check load: /map on the forge-sized fixture, warm #3070
Check · Review · Land · Live done — Landed and live in production.
The warm `/map` test in `test/map-cache.test.ts` no longer compares wall-clock times. It passed 5 runs in a row here, but I did not run the 20-run test beside 5 busy check cells. `bun x tsc --noEmit` couldn't write to its temp dir in the sandbox, so the type check did not run.
shipped production is serving it deploy 15c0f93
The warm /map test in test/map-cache.test.ts no longer compares wall-clock times. It passed 5 runs in a row here, but I did not run the 20-run test beside 5 busy check cells. bun x tsc --noEmit couldn't write to its temp dir in the sandbox, so the type check did not run.
- What flaked:
median * 3 < coldcompared the warm median with a single cold request. A loaded check cell can slow either side, so the ratio failed even though the cache worked. - New assertion: the test now counts SQL statements during the cold request and requires
warm statements × 5 < cold statements. The cold count was about 350 against 27–35 warm. - Why 5×, not 10×: my first try at 10× failed once on the stranger case, where cold was ~350 and warm 35. 5× leaves room.
- Unchanged: the existing bound of
CHROME + 2 × lanesstatements per warm hit stays. The p95 check is now a 30 s hang guard instead of 5 s. - Left out: I didn't add a quiet lane or touch
bench/map-home.ts. The real 50 ms budget is still read off that bench and the RUM numbers (#2674/#2675).
The idea is right: comparing statement counts instead of wall-clock removes the flake and still fails if the cache goes missing. Two small things to fix before it lands, both in test/map-cache.test.ts.
- The line
// The steady state: a lane...lost its indentation (12 spaces instead of 6). Restore it. - The comment "Cold, it was ~2,700" now contradicts the new cold count, which the pull says is about 350. Update it or drop the number.
The cold count is taken from a single request. That is fine because it is a statement count, not a time.
This changes only a test, so there is no page, terminal or README surface to check.
I fixed both comments in test/map-cache.test.ts and pushed to the same branch. test/map-cache.test.ts passes (6 pass, 0 fail).
- Indentation: the "The steady state" comment is back at 6 spaces.
- Stale figure: the comment now says "Cold, it was ~350 here (coldStatements)". The old "~2,700" figure is gone.
The pull's earlier points still hold: only the test changed, and the 20-run check beside 5 busy check cells was not run here.
Test-only change, and the right fix. The flaky median * 3 < cold compared two wall-clock timings, and a loaded check cell can skew either one. The test now compares work done: statements during the cold build against the warm median, which a missing cache would put level. The CHROME + 2 * lanes bound stays. countStatements is wrapped in try/finally, so the counter is always stopped. There is no product, security, or dependency surface to judge, and the preview is unaffected.
Two things I would do next:
- Run the 20-iteration loop beside busy check cells once, as scout said they had not. The 5x margin came from one observed cold/warm of 350/35, which is only 10x.
- Make the
~350in the comment a loose bound or drop the number, since it drifts as the fixture grows.coldStatementsis already logged, so the comment can just point to that.
I merged main into the branch and pushed. test/map-cache.test.ts passes (6 pass, 0 fail).
main had already landed the same fix: it counts cold and warm statements instead of comparing wall-clock times. I kept main's version of both conflicting hunks, which means a 3× margin (statements * 3 < cold) and the 5 s hang guard. This replaces my 5× margin and 30 s guard, so the 5× the earlier pushes used is gone.
What this pull now changes against main is two edits to the test:
- a reworded test title;
- the stale "Cold, it was ~2,700" comment, now "~350 here (cold)".
I did not re-run the 20-run test beside 5 busy check cells after the merge.
Comment-and-title only change; the flake itself was already fixed on main (statement counts, not wall-clock), and the merge kept main's version. Nothing here touches security, the page or the terminal, so the preview is unaffected. Two things I would do next: (1) reword the new comment, which reads "~350 here (cold)" — the parenthetical repeats itself; say "about 350 statements cold on this fixture"; (2) run the 20-iteration loop beside busy check cells once, as the thread says was not done, to confirm the 3x margin holds for the stranger case (scout saw cold ~350 vs warm ~35 earlier).
The full gate on land was red and named no test; the pull is open again.
This pull changes one comment in test/map-cache.test.ts. It does not fix the red gate on land, and I couldn't find what failed.
- Comment: I took juniper's suggestion and reworded it to "Cold, it was about 350 statements on this fixture."
- Red gate: The gate named no test, so I had nothing to fix. The flake was already fixed on main, where the test reads statement counts instead of wall-clock. I merged the current main into the branch, so the pull now differs from main only by the comment. The red gate may have come from the branch being behind main. I can't confirm that from here.
- Test run: I ran only
test/map-cache.test.ts, and all 6 tests pass. The owner ran 27 statements warm against 386 cold. The stranger ran 35 warm against 350 cold, so the 3x margin holds. I did not run the full gate. - Not done: juniper's second suggestion, running the 20-iteration loop beside busy check cells. The sandbox can't reproduce check-cell load. The statement counts don't depend on load, so I don't expect it to change the result.
Test-only change: the title now says what is measured (a fraction of a cold build's work, statements), and the stale ~2,700 figure in the comment is corrected to about 350. It matches main's statement-counting assertions, adds no behaviour and touches no page, so there is no experience to regress. The preview is unaffected. Two things I would do next: (1) the pull does not address the red gate on land. Re-run the full check at this head and, if it is still red, open an issue that names the failing suite. (2) Put the 350 figure in a named constant or derive it from the measured cold count, so the comment cannot go stale again. The comment line is also long; wrap it to match the neighbouring lines.
The full gate on land was red and named no test; the pull is open again.
The warm
/maptest intest/map-cache.test.tsno longer compares wall-clock times. It passed 5 runs in a row here, but I did not run the 20-run test beside 5 busy check cells.bun x tsc --noEmitcouldn't write to its temp dir in the sandbox, so the type check did not run.median * 3 < coldcompared the warm median with a single cold request. A loaded check cell can slow either side, so the ratio failed even though the cache worked.warm statements × 5 < cold statements. The cold count was about 350 against 27–35 warm.CHROME + 2 × lanesstatements per warm hit stays. The p95 check is now a 30 s hang guard instead of 5 s.bench/map-home.ts. The real 50 ms budget is still read off that bench and the RUM numbers (#2674/#2675).Closes #3069