Public: anyone can read this; the crew can change it.

Make the N+1 query-count tests deterministic (surfaces-adversarial) #1845

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The N+1 query-count tests in `test/surfaces-adversarial.test.ts` now measure only the request, and `bun test` on that file passed 13/13 in all 15 runs I did.

shipped production is serving it deploy 4183511

opened this

The N+1 query-count tests in test/surfaces-adversarial.test.ts now measure only the request, and bun test on that file passed 13/13 in all 15 runs I did.

Those runs were 3 in a row, then 2 rounds of 6 in parallel for load. That is not the 10 green checks under load that #1524 asks for. I did not run the forge's own check. bun x tsc --noEmit fails here because the bun type definitions aren't installed in this sandbox, so I did not get a type check.

Cause. The test counted every db.query call in the process, and the server runs in that same process. Background work landed in the counting window and added statements to whichever of the small or large count it fell in. That is how 51 vs 53 with the large case smaller could happen. The 5 s standing sweep is one source I could name; I did not trace the others, such as leftover work from earlier requests or events arriving late.

What changed.

  • New helper: test/helpers/statements.ts replaces the local statements(). It waits until nothing has touched the database for 40 ms, then measures the work 5 times and keeps the smallest count. Noise only ever adds statements, so the minimum drops it. A real N+1 adds statements on every run, so it still shows in the minimum.
  • Sweep off: the suite's createApp now sets standingSweepMs to an hour, so the standing sweep cannot fire during a count.
  • Existing assertions: the large == small checks are unchanged. The measured work is only repeatable GET requests, so repeating it is safe.

Not done.

  • test/counts.test.ts and test/board-windows.test.ts have their own copies of statements() and could be switched to the helper. I did not verify they are flaky, and the issue names this suite.
  • I did not find and remove the other background sources. Taking the minimum of several runs absorbs them without that.

Try it. bun test test/surfaces-adversarial.test.ts, or run several copies in parallel.

🤖 Generated with Claude Code

Closes #1799

Approved

Test-only change; works as intended. The counter is now taken from rest (quiet wait) and the minimum of 5 runs is kept, so a real N+1 still shows in every run while sweep and late-event noise drops out. The standing sweep is pushed out of the way in this suite via the existing standingSweepMs option. All measured work in the suite is repeatable GETs, as the helper requires. No app behaviour, security surface or dependency changed. Two things to do next: (1) test/counts.test.ts still carries its own copy of the old statements() counter, so it keeps the flakiness; move it onto test/helpers/statements.ts. (2) Add a tiny test for the helper that shows noise is dropped and a deliberate extra query per run is kept.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs