Public: anyone can read this; the crew can change it.

scripts/test-budget.ts names each test file over 20 s and each test over 2 s from plain times, with no test run #12142

Parked opened this
Closed without merging

This adds scripts/test-budget.ts and its test, the first half of the per-test budget (#1242). The module only decides what is over budget; it runs no tests and reads no files. Nothing calls it yet: scripts/test-shards.ts and scripts/gate.ts are untouched, so the gate's output is unchanged.

Notes

stopped parked by the crew

opened this

This adds scripts/test-budget.ts and its test, the first half of the per-test budget (#1242). The module only decides what is over budget; it runs no tests and reads no files. Nothing calls it yet: scripts/test-shards.ts and scripts/gate.ts are untouched, so the gate's output is unchanged. The diff is these two files, on top of main.

The module:

  • breaches({files, tests}, slow = SLOW): takes file times as {path: seconds}, the same shape as scripts/test-times.json, and test times as {file, name, seconds}[]. It names each file over 20 s and each test over 2 s. A time exactly at the limit is within budget. It returns:
    • over: the breaches that count, files first, slowest first.
    • aside: breaches in a slow file, each carrying its {issue, reason}.
    • slowest: the five slowest tests, each with its share of the run.
    • totalSeconds: the sum of the file times, or of the test times when no file is timed.
  • Bad times are refused: a time that is NaN, infinite, negative or not a number throws a RangeError, so a broken table can never pass as under budget.
  • SLOW is empty: a file should join the list with the issue that rewrites it, and no such issues exist yet, so I didn't invent any.
  • budgetBinds(tier, changed, file): true only on the fast tier, and only when file is a .test.ts among the pull's changed files.
  • summaryLines(result): returns the five slowest (seconds, share in %, file > test), then over the budget: N, SLOW aside: M, then each breach as file > test: 3.1 s over 2 s. Set-aside breaches come last and are tagged (SLOW, #issue: reason).

Tests: test/test-budget-run.test.ts uses fixture times only and passes (11 tests). It covers:

  • a run under budget passes, including the exact 20 s and 2 s limits;
  • a file over 20 s and a test over 2 s are named in the result and in the summary lines;
  • a SLOW file is set aside with its issue;
  • the slowest five and their shares match a hand calculation (4/20 = 20%, 3/20 = 15%, …), with a separate case for when no file is timed;
  • bad times are refused;
  • budgetBinds: a full-tier breach does not bind, and neither does a fast-tier breach in an unchanged file or in a changed file that isn't a test.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts exits 2 because a script changed, so I chose the covering files myself: the new test, test/unreached.test.ts and test/unused-exports.test.ts. One test in test/unreached.test.ts fails because nothing under src/ imports src/crew/week_burn.ts; it fails the same way on main without this change.

Not done here: wiring the budget into scripts/test-shards.ts and the check summary, and writing per-test times into test-times.json. Those are the second half of #1242.

To try it: bun test test/test-budget-run.test.ts

🤖 Generated with Claude Code

Closes #12136

3 earlier comments · 1 event
commented

This adds scripts/test-budget.ts and its test, the first half of the per-test budget (#1242). The module only decides what is over budget; it runs no tests and reads no files. Nothing calls it yet: scripts/test-shards.ts and scripts/gate.ts are untouched, so the gate's output is unchanged. The diff is these two files, on top of main.

The module:

  • breaches({files, tests}, slow = SLOW): takes file times as {path: seconds}, the same shape as scripts/test-times.json, and test times as {file, name, seconds}[]. It names each file over 20 s and each test over 2 s. A time exactly at the limit is within budget. It returns:
    • over: the breaches that count, files first, slowest first.
    • aside: breaches in a slow file, each carrying its {issue, reason}.
    • slowest: the five slowest tests, each with its share of the run.
    • totalSeconds: the sum of the file times, or of the test times when no file is timed.
  • Bad times are refused: a time that is NaN, infinite, negative or not a number throws a RangeError, so a broken table can never pass as under budget.
  • SLOW is empty: a file should join the list with the issue that rewrites it, and no such issues exist yet, so I didn't invent any.
  • budgetBinds(tier, changed, file): true only on the fast tier, and only when file is a .test.ts among the pull's changed files.
  • summaryLines(result): returns the five slowest (seconds, share in %, file > test), then over the budget: N, SLOW aside: M, then each breach as file > test: 3.1 s over 2 s. Set-aside breaches come last and are tagged (SLOW, #issue: reason).

Tests: test/test-budget-run.test.ts uses fixture times only and passes (11 tests). It covers:

  • a run under budget passes, including the exact 20 s and 2 s limits;
  • a file over 20 s and a test over 2 s are named in the result and in the summary lines;
  • a SLOW file is set aside with its issue;
  • the slowest five and their shares match a hand calculation (4/20 = 20%, 3/20 = 15%, …), with a separate case for when no file is timed;
  • bad times are refused;
  • budgetBinds: a full-tier breach does not bind, and neither does a fast-tier breach in an unchanged file or in a changed file that isn't a test.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts exits 2 because a script changed, so I chose the covering files myself: the new test, test/unreached.test.ts and test/unused-exports.test.ts. One test in test/unreached.test.ts fails because nothing under src/ imports src/crew/week_burn.ts; it fails the same way on main without this change.

Not done here: wiring the budget into scripts/test-shards.ts and the check summary, and writing per-test times into test-times.json. Those are the second half of #1242.

To try it: bun test test/test-budget-run.test.ts

🤖 Generated with Claude Code

commented

This adds scripts/test-budget.ts and its test, the first half of the per-test budget (#1242). The module only decides what is over budget; it runs no tests and reads no files. Nothing calls it yet: scripts/test-shards.ts and scripts/gate.ts are untouched, so the gate's output is unchanged. The diff is these two files, on top of main.

The module:

  • breaches({files, tests}, slow = SLOW): takes file times as {path: seconds}, the same shape as scripts/test-times.json, and test times as {file, name, seconds}[]. It names each file over 20 s and each test over 2 s. A time exactly at the limit is within budget. It returns:
    • over: the breaches that count, files first, slowest first.
    • aside: breaches in a slow file, each carrying its {issue, reason}.
    • slowest: the five slowest tests, each with its share of the run.
    • totalSeconds: the sum of the file times, or of the test times when no file is timed.
  • Bad times are refused: a time that is NaN, infinite, negative or not a number throws a RangeError, so a broken table can never pass as under budget.
  • SLOW is empty: a file should join the list with the issue that rewrites it, and no such issues exist yet, so I didn't invent any.
  • budgetBinds(tier, changed, file): true only on the fast tier, and only when file is a .test.ts among the pull's changed files.
  • summaryLines(result): returns the five slowest (seconds, share in %, file > test), then over the budget: N, SLOW aside: M, then each breach as file > test: 3.1 s over 2 s. Set-aside breaches come last and are tagged (SLOW, #issue: reason).

Tests: test/test-budget-run.test.ts uses fixture times only and passes (11 tests). It covers:

  • a run under budget passes, including the exact 20 s and 2 s limits;
  • a file over 20 s and a test over 2 s are named in the result and in the summary lines;
  • a SLOW file is set aside with its issue;
  • the slowest five and their shares match a hand calculation (4/20 = 20%, 3/20 = 15%, …), with a separate case for when no file is timed;
  • bad times are refused;
  • budgetBinds: a full-tier breach does not bind, and neither does a fast-tier breach in an unchanged file or in a changed file that isn't a test.

Fix for the failed check: a comment in the test was 103 columns and broke the gate's 100-column rule; it is now two lines. The rule only checks lines a pull adds, so the gate passes locally only when run with FORGE_CHECK_BASE set to the merge base. With that set, bun scripts/gate.ts passes now, and so do bun x tsc --noEmit and the new test.

Pre-existing failure: test/unreached.test.ts fails because nothing under src/ imports src/crew/week_burn.ts. It fails the same way on main without this change.

Not done here: wiring the budget into scripts/test-shards.ts and the check summary, and writing per-test times into test-times.json. Those are the second half of #1242.

To try it: bun test test/test-budget-run.test.ts

🤖 Generated with Claude Code

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

This pull adds scripts/test-budget.ts, which reads plain test times and lists each test file over 20 s and each test over 2 s. Files that are already known to be slow go in a separate SLOW list. The check only fails on the fast tier, and only when the change touches a test file. It also writes out the five slowest tests and their share of the total. test/test-budget-run.test.ts covers this against times that can be checked by hand.

The 50 failures in the last check don't come from this branch. They are already on main at e56c7c8a3, which this branch has merged in. The diff against main is only the two files above: neither touches src/, and the new test only works on in-memory data. To confirm, I ran a sample of the failing tests on a separate checkout of origin/main, and they fail there the same way:

  • test/unreached.test.ts: src/crew/week_burn.ts (from ce7846af2) is not imported by anything.
  • test/reserved-names.test.ts: broker is a top-level page that is not in the list of reserved names.
  • test/board-grid-style.test.ts: the .bgrid/.bp-* rules and .chart-box.is-table are no longer in the stylesheet that gets served.

These come from different commits that landed on main today, not one shared regression. I left them out of this pull: they cover unrelated parts of the app, and their fixes belong in their own pulls. Nothing new is pushed. Once main is green again, re-running the check should pass with no change here.

On this branch, bun scripts/gate.ts, bun x tsc --noEmit and bun test test/test-budget-run.test.ts all pass (11 tests).

To try it: bun scripts/test-budget.ts, giving it a file of per-test times; the header comment in the script describes the input format.

🤖 Generated with Claude Code

commented

Parked by the crew: the check stayed red through 2 automatic tries

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs