Public: anyone can read this; the crew can change it.

Tests wait for a signal, not a clock: a ratchet on fixed sleeps in test/ #14358

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

bun scripts/gate.ts now refuses any test file whose count of fixed real-time waits goes up or down from what is recorded in a committed allowance file, so new sleeps stop at merge and the list can only shrink.

Notes

shipped production is serving it deploy 404563e

opened this

bun scripts/gate.ts now refuses any test file whose count of fixed real-time waits goes up or down from what is recorded in a committed allowance file, so new sleeps stop at merge and the list can only shrink. Adding one await Bun.sleep(50) to a file fails the gate with "await the operation or an event it emits instead of a fixed delay".

What it counts. scripts/test_honesty.ts reads every .ts file under test/ (helpers and fixtures included) and counts three kinds per file, skipping lines that are only a comment:

  • wait: Bun.sleep(<number>), Bun.sleepSync(<number>) and setTimeout(<name>, <number>).
  • mock: mock.module(.
  • timeout: a test's { timeout: N } or { retry: N } option, and setDefaultTimeout(.

The allowance. Each count is compared with scripts/test-honesty-allow.json, one file per line so pulls that touch different files don't conflict.

  • A file over its allowance is refused, with advice for that kind.
  • A file under its allowance (including a deleted file still listed) is refused until its line is lowered.
  • A file not listed is allowed none.
  • FORGE_SLEEP_WRITE=1 bun scripts/test_honesty.ts (or the same variable on bun scripts/gate.ts) rewrites the file from today's counts.
  • gateVerdict runs the check as testHonestyRefusal, so it runs on every check that starts with the gate: fast, run, full and verify. That covers pulls and main. A tree that has no scripts/test_honesty.ts (the gate's own test fixtures) is skipped.

Today's allowance: 904 waits in 465 files, 3 mock.module calls in 2 files, and 65 timeouts/retries in 64 files (almost all setDefaultTimeout), across 518 files. The issue said 90 setTimeout waits; counting every callback name, not just resolve, gives 100 in 69 files.

Choices worth reviewing:

  • The positional timeout }, 30_000); is not counted. About 2,160 tests that boot a forge use it. Holding them would put 1,352 files in the allowance and refuse every new forge-booting test, which is a much larger change than the issue's "5 today". This leaves a gap: someone can still raise a timeout with the positional form. It deserves its own issue.
  • Matching is line by line, not parsed. A counted pattern inside a string still counts. test/test-honesty.test.ts therefore builds its samples from pieces, which also keeps test/flake.test.ts's retry detector quiet.
  • Speed. One scan takes about 0.4 s; a cheap text check skips lines that contain none of the keywords.
  • Docs. docs/PRINCIPLES.md now names the gate as one enforcer of "Speed is counted in work", and the flaky-test rule in CLAUDE.md says how to lower an allowance.
  • Out of scope, as the issue asks: landmines' no-comments and no-explicit-return-types.

Checks run here: bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. I ran test/test-honesty.test.ts and the 38 test files that import scripts/gate.ts or related scripts, plus the 7 that read the edited docs. Two failed:

  • test/flake.test.ts flagged a retry sample in the new test file; that sample is now split up and the file passes.
  • test/test-budget-run.test.ts ("the looks lane in the runner") fails the same way on main in this sandbox, because no Chromium starts here.

The keep_plan and estimate tools the issue mentions were not available in this run, so no plan or estimate was recorded.

To try it: add await Bun.sleep(50); to any test file and run bun scripts/gate.ts; it exits 2 naming that file. Then remove an existing sleep and run it again: the gate asks for the allowance to be lowered.

🤖 Generated with Claude Code

Closes #14321

commented

bun scripts/gate.ts now refuses any test file whose count of fixed real-time waits goes up or down from what is recorded in a committed allowance file, so new sleeps stop at merge and the list can only shrink. Adding one await Bun.sleep(50) to a file fails the gate with "await the operation or an event it emits instead of a fixed delay".

What it counts. scripts/test_honesty.ts reads every .ts file under test/ (helpers and fixtures included) and counts three kinds per file, skipping lines that are only a comment:

  • wait: Bun.sleep(<number>), Bun.sleepSync(<number>) and setTimeout(<name>, <number>).
  • mock: mock.module(.
  • timeout: a test's { timeout: N } or { retry: N } option, and setDefaultTimeout(.

The allowance. Each count is compared with scripts/test-honesty-allow.json, one file per line so pulls that touch different files don't conflict.

  • A file over its allowance is refused, with advice for that kind.
  • A file under its allowance (including a deleted file still listed) is refused until its line is lowered.
  • A file not listed is allowed none.
  • FORGE_SLEEP_WRITE=1 bun scripts/test_honesty.ts (or the same variable on bun scripts/gate.ts) rewrites the file from today's counts.
  • gateVerdict runs the check as testHonestyRefusal, so it runs on every check that starts with the gate: fast, run, full and verify. That covers pulls and main. A tree that has no scripts/test_honesty.ts (the gate's own test fixtures) is skipped.

Today's allowance: 904 waits in 465 files, 3 mock.module calls in 2 files, and 65 timeouts/retries in 64 files (almost all setDefaultTimeout), across 518 files. The issue said 90 setTimeout waits; counting every callback name, not just resolve, gives 100 in 69 files.

Choices worth reviewing:

  • The positional timeout }, 30_000); is not counted. About 2,160 tests that boot a forge use it. Holding them would put 1,352 files in the allowance and refuse every new forge-booting test, which is a much larger change than the issue's "5 today". This leaves a gap: someone can still raise a timeout with the positional form. It deserves its own issue.
  • Matching is line by line, not parsed. A counted pattern inside a string still counts. test/test-honesty.test.ts therefore builds its samples from pieces, which also keeps test/flake.test.ts's retry detector quiet.
  • Speed. One scan takes about 0.4 s; a cheap text check skips lines that contain none of the keywords.
  • Docs. docs/PRINCIPLES.md now names the gate as one enforcer of "Speed is counted in work", and the flaky-test rule in CLAUDE.md says how to lower an allowance.
  • Out of scope, as the issue asks: landmines' no-comments and no-explicit-return-types.

Checks run here: bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. I ran test/test-honesty.test.ts and the 38 test files that import scripts/gate.ts or related scripts, plus the 7 that read the edited docs. Two failed:

  • test/flake.test.ts flagged a retry sample in the new test file; that sample is now split up and the file passes.
  • test/test-budget-run.test.ts ("the looks lane in the runner") fails the same way on main in this sandbox, because no Chromium starts here.

The keep_plan and estimate tools the issue mentions were not available in this run, so no plan or estimate was recorded.

To try it: add await Bun.sleep(50); to any test file and run bun scripts/gate.ts; it exits 2 naming that file. Then remove an existing sleep and run it again: the gate asks for the allowance to be lowered.

🤖 Generated with Claude Code

Approved

Approved. It adds a ratchet on fixed sleeps, mock.module and raised timeouts, held per file against a committed one-line-per-file allowance. It runs in gateVerdict, is tested in both directions (over and under the allowance, new file, deleted file), and is documented in CLAUDE.md and PRINCIPLES.md. It adds no dependency and does not touch README. It is a test-honesty gate, so it serves the pipeline direction (#1242, #1073) and there is no page or terminal door to review.

Two things I would do next:

  1. Close the bypasses. The regexes only see Bun.sleep(<literal>) and setTimeout(name, <literal>) on one line. setTimeout(() => r(), 50), Bun.sleep(50 * 2), a delay held in a const, and a call split over two lines all pass. The positional }, 30_000) timeout is not counted either, as the pull says. Open an issue for each gap, or move to an AST scan.
  2. Watch the cost of the under-allowance refusal. Any pull that removes a sleep, or deletes a test file, must also edit the JSON, and a stacked or batched landing can go red on a count another pull lowered. Consider having the gate --fix path rewrite the allowance, so a builder does not hand-run FORGE_SLEEP_WRITE.

Small nit: the sample line in test/test-honesty.test.ts that begins C + ", { timeout: 5000 }" is over-indented. It is harmless.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs