Public: anyone can read this; the crew can change it.

Tests wait for a signal, not a clock: a ratchet on fixed sleeps in test/ #14333

opened this in answer to

Check · Review · Land · Live — No review yet. Next: reviewer, no review yet.

Ready to land

The gate now checks each test file for fixed waits and compares the count to an allowance file. A file with more than its allowance fails with "await the operation or an event it emits instead of a fixed delay". A file with fewer also fails until its allowance is lowered, so the list only shrinks.

Notes · 4 files · +1594 −0

look green — it answered, no tests were run, ready for a look deploy 0bc0c37

Check passeddeploy 0bc0c37

Website spec: 26 findings (report only)
opened this

The gate now checks each test file for fixed waits and compares the count to an allowance file. A file with more than its allowance fails with "await the operation or an event it emits instead of a fixed delay". A file with fewer also fails until its allowance is lowered, so the list only shrinks. Because forge.json's run, fast and full checks all start with bun scripts/gate.ts, this runs on pull checks and on main.

  • Counting: src/kernel/test_honesty.ts takes test sources and returns counts. Per file, it counts four things in every .ts under test/:
    • await Bun.sleep(<number>)
    • setTimeout(<name>, <number>)
    • mock.module(
    • test retry: / timeout: options, on their own line or as }, { retry: N }

    A wait with a name or computed value, like Bun.sleep(ms), is not counted. The allowance file is checked when read: a malformed one is rejected with a message, not trusted.

  • Gate: scripts/gate.ts gains testHonestyRefusal, which is added to gateVerdict's refusals, and writeTestHonesty. FORGE_SLEEP_WRITE=1 bun scripts/gate.ts rewrites the allowance to today's counts. A tree without the allowance file, such as the gate's own test fixtures, is not checked.
  • Allowance: docs/test-honesty.json holds today's counts: 764 sleeps, 100 setTimeout waits, 3 mock.module and 7 retry/timeout options, across 461 files. The sleep and mock.module counts match the issue. The other two came out higher than the issue's 90 and 5 because these patterns are a bit wider; the allowance covers the difference.
  • Tests: test/test-honesty.test.ts covers each kind of wait, a malformed allowance, and the repository's own tree matching its allowance. It also shows one new await Bun.sleep(50) failing with the advice and a drop being refused until the allowance is lowered.

Choices to look at:

  • The counting code is in src/kernel/test_honesty.ts, not the scripts/test_honesty.ts the issue named. The gate's other rules already live in src/kernel/ and are loaded by require, so treeGate can still load gate.ts on its own.
  • Raised timeout: lines are matched by text, so the count includes some flagged lines that are not test timeouts (for example timeout: 600 in check config fixtures). Counting every trailing }, 30_000) timeout argument would have matched about 2,100 lines, so I left that pattern out.
  • landmines' no-comments and no-explicit-return-types are out of scope, as the issue says.
  • I did not call keep_plan or estimate; neither tool was available in this run.

To try it: add an await Bun.sleep(50); to any test file and run bun scripts/gate.ts (it exits 2 with the message), then remove it.

Checks: bun scripts/gate.ts --fix changed nothing and passed, and bun x tsc --noEmit passed. scripts/fast.ts exited 2 because scripts/gate.ts changed, so I picked the tests by hand: test/test-honesty.test.ts, test/gate-*.test.ts, test/check-list.test.ts and test/look-count.test.ts all pass (105 tests).

🤖 Generated with Claude Code

Closes #14321

commented

The gate now checks each test file for fixed waits and compares the count to an allowance file. A file with more than its allowance fails with "await the operation or an event it emits instead of a fixed delay". A file with fewer also fails until its allowance is lowered, so the list only shrinks. Because forge.json's run, fast and full checks all start with bun scripts/gate.ts, this runs on pull checks and on main.

  • Counting: src/kernel/test_honesty.ts takes test sources and returns counts. Per file, it counts four things in every .ts under test/:
    • await Bun.sleep(<number>)
    • setTimeout(<name>, <number>)
    • mock.module(
    • test retry: / timeout: options, on their own line or as }, { retry: N }

    A wait with a name or computed value, like Bun.sleep(ms), is not counted. The allowance file is checked when read: a malformed one is rejected with a message, not trusted.

  • Gate: scripts/gate.ts gains testHonestyRefusal, which is added to gateVerdict's refusals, and writeTestHonesty. FORGE_SLEEP_WRITE=1 bun scripts/gate.ts rewrites the allowance to today's counts. A tree without the allowance file, such as the gate's own test fixtures, is not checked.
  • Allowance: docs/test-honesty.json holds today's counts: 764 sleeps, 100 setTimeout waits, 3 mock.module and 7 retry/timeout options, across 461 files. The sleep and mock.module counts match the issue. The other two came out higher than the issue's 90 and 5 because these patterns are a bit wider; the allowance covers the difference.
  • Tests: test/test-honesty.test.ts covers each kind of wait, a malformed allowance, and the repository's own tree matching its allowance. It also shows one new await Bun.sleep(50) failing with the advice and a drop being refused until the allowance is lowered.

Choices to look at:

  • The counting code is in src/kernel/test_honesty.ts, not the scripts/test_honesty.ts the issue named. The gate's other rules already live in src/kernel/ and are loaded by require, so treeGate can still load gate.ts on its own.
  • Raised timeout: lines are matched by text, so the count includes some flagged lines that are not test timeouts (for example timeout: 600 in check config fixtures). Counting every trailing }, 30_000) timeout argument would have matched about 2,100 lines, so I left that pattern out.
  • landmines' no-comments and no-explicit-return-types are out of scope, as the issue says.
  • I did not call keep_plan or estimate; neither tool was available in this run.

To try it: add an await Bun.sleep(50); to any test file and run bun scripts/gate.ts (it exits 2 with the message), then remove it.

Checks: bun scripts/gate.ts --fix changed nothing and passed, and bun x tsc --noEmit passed. scripts/fast.ts exited 2 because scripts/gate.ts changed, so I picked the tests by hand: test/test-honesty.test.ts, test/gate-*.test.ts, test/check-list.test.ts and test/look-count.test.ts all pass (105 tests).

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs