Public: anyone can read this; the crew can change it.
Tests wait for a signal, not a clock: a ratchet on fixed sleeps in test/ #14358
Check · Review · Land · Live done — Landed and live in production.
bun scripts/gate.ts now refuses any test file whose count of fixed real-time waits goes up or down from what is recorded in a committed allowance file, so new sleeps stop at merge and the list can only shrink.
shipped production is serving it deploy 404563e
bun scripts/gate.ts now refuses any test file whose count of fixed real-time waits goes up or down from what is recorded in a committed allowance file, so new sleeps stop at merge and the list can only shrink. Adding one await Bun.sleep(50) to a file fails the gate with "await the operation or an event it emits instead of a fixed delay".
What it counts. scripts/test_honesty.ts reads every .ts file under test/ (helpers and fixtures included) and counts three kinds per file, skipping lines that are only a comment:
- wait:
Bun.sleep(<number>),Bun.sleepSync(<number>)andsetTimeout(<name>, <number>). - mock:
mock.module(. - timeout: a test's
{ timeout: N }or{ retry: N }option, andsetDefaultTimeout(.
The allowance. Each count is compared with scripts/test-honesty-allow.json, one file per line so pulls that touch different files don't conflict.
- A file over its allowance is refused, with advice for that kind.
- A file under its allowance (including a deleted file still listed) is refused until its line is lowered.
- A file not listed is allowed none.
FORGE_SLEEP_WRITE=1 bun scripts/test_honesty.ts(or the same variable onbun scripts/gate.ts) rewrites the file from today's counts.gateVerdictruns the check astestHonestyRefusal, so it runs on every check that starts with the gate: fast, run, full and verify. That covers pulls and main. A tree that has noscripts/test_honesty.ts(the gate's own test fixtures) is skipped.
Today's allowance: 904 waits in 465 files, 3 mock.module calls in 2 files, and 65 timeouts/retries in 64 files (almost all setDefaultTimeout), across 518 files. The issue said 90 setTimeout waits; counting every callback name, not just resolve, gives 100 in 69 files.
Choices worth reviewing:
- The positional timeout
}, 30_000);is not counted. About 2,160 tests that boot a forge use it. Holding them would put 1,352 files in the allowance and refuse every new forge-booting test, which is a much larger change than the issue's "5 today". This leaves a gap: someone can still raise a timeout with the positional form. It deserves its own issue. - Matching is line by line, not parsed. A counted pattern inside a string still counts.
test/test-honesty.test.tstherefore builds its samples from pieces, which also keepstest/flake.test.ts's retry detector quiet. - Speed. One scan takes about 0.4 s; a cheap text check skips lines that contain none of the keywords.
- Docs.
docs/PRINCIPLES.mdnow names the gate as one enforcer of "Speed is counted in work", and the flaky-test rule inCLAUDE.mdsays how to lower an allowance. - Out of scope, as the issue asks: landmines'
no-commentsandno-explicit-return-types.
Checks run here: bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. I ran test/test-honesty.test.ts and the 38 test files that import scripts/gate.ts or related scripts, plus the 7 that read the edited docs. Two failed:
test/flake.test.tsflagged a retry sample in the new test file; that sample is now split up and the file passes.test/test-budget-run.test.ts("the looks lane in the runner") fails the same way on main in this sandbox, because no Chromium starts here.
The keep_plan and estimate tools the issue mentions were not available in this run, so no plan or estimate was recorded.
To try it: add await Bun.sleep(50); to any test file and run bun scripts/gate.ts; it exits 2 naming that file. Then remove an existing sleep and run it again: the gate asks for the allowance to be lowered.
🤖 Generated with Claude Code
Approved. It adds a ratchet on fixed sleeps, mock.module and raised timeouts, held per file against a committed one-line-per-file allowance. It runs in gateVerdict, is tested in both directions (over and under the allowance, new file, deleted file), and is documented in CLAUDE.md and PRINCIPLES.md. It adds no dependency and does not touch README. It is a test-honesty gate, so it serves the pipeline direction (#1242, #1073) and there is no page or terminal door to review.
Two things I would do next:
- Close the bypasses. The regexes only see
Bun.sleep(<literal>)andsetTimeout(name, <literal>)on one line.setTimeout(() => r(), 50),Bun.sleep(50 * 2), a delay held in a const, and a call split over two lines all pass. The positional}, 30_000)timeout is not counted either, as the pull says. Open an issue for each gap, or move to an AST scan. - Watch the cost of the under-allowance refusal. Any pull that removes a sleep, or deletes a test file, must also edit the JSON, and a stacked or batched landing can go red on a count another pull lowered. Consider having the gate --fix path rewrite the allowance, so a builder does not hand-run FORGE_SLEEP_WRITE.
Small nit: the sample line in test/test-honesty.test.ts that begins C + ", { timeout: 5000 }" is over-indented. It is harmless.
bun scripts/gate.tsnow refuses any test file whose count of fixed real-time waits goes up or down from what is recorded in a committed allowance file, so new sleeps stop at merge and the list can only shrink. Adding oneawait Bun.sleep(50)to a file fails the gate with "await the operation or an event it emits instead of a fixed delay".What it counts.
scripts/test_honesty.tsreads every.tsfile undertest/(helpers and fixtures included) and counts three kinds per file, skipping lines that are only a comment:Bun.sleep(<number>),Bun.sleepSync(<number>)andsetTimeout(<name>, <number>).mock.module(.{ timeout: N }or{ retry: N }option, andsetDefaultTimeout(.The allowance. Each count is compared with
scripts/test-honesty-allow.json, one file per line so pulls that touch different files don't conflict.FORGE_SLEEP_WRITE=1 bun scripts/test_honesty.ts(or the same variable onbun scripts/gate.ts) rewrites the file from today's counts.gateVerdictruns the check astestHonestyRefusal, so it runs on every check that starts with the gate: fast, run, full and verify. That covers pulls and main. A tree that has noscripts/test_honesty.ts(the gate's own test fixtures) is skipped.Today's allowance: 904 waits in 465 files, 3
mock.modulecalls in 2 files, and 65 timeouts/retries in 64 files (almost allsetDefaultTimeout), across 518 files. The issue said 90setTimeoutwaits; counting every callback name, not justresolve, gives 100 in 69 files.Choices worth reviewing:
}, 30_000);is not counted. About 2,160 tests that boot a forge use it. Holding them would put 1,352 files in the allowance and refuse every new forge-booting test, which is a much larger change than the issue's "5 today". This leaves a gap: someone can still raise a timeout with the positional form. It deserves its own issue.test/test-honesty.test.tstherefore builds its samples from pieces, which also keepstest/flake.test.ts's retry detector quiet.docs/PRINCIPLES.mdnow names the gate as one enforcer of "Speed is counted in work", and the flaky-test rule inCLAUDE.mdsays how to lower an allowance.no-commentsandno-explicit-return-types.Checks run here:
bun scripts/gate.ts --fixandbun x tsc --noEmitare clean. I rantest/test-honesty.test.tsand the 38 test files that importscripts/gate.tsor related scripts, plus the 7 that read the edited docs. Two failed:test/flake.test.tsflagged a retry sample in the new test file; that sample is now split up and the file passes.test/test-budget-run.test.ts("the looks lane in the runner") fails the same way on main in this sandbox, because no Chromium starts here.The
keep_planandestimatetools the issue mentions were not available in this run, so no plan or estimate was recorded.To try it: add
await Bun.sleep(50);to any test file and runbun scripts/gate.ts; it exits 2 naming that file. Then remove an existing sleep and run it again: the gate asks for the allowance to be lowered.🤖 Generated with Claude Code
Closes #14321