Timing tests stop failing because the machine is busy #7822

A pull's check can no longer fail on a slow expectBudget. Wall-clock budgets are now enforced only by a new nightly run. I did not convert any test to count work or a fake clock, and I haven't tied the budgets to the performance watcher (#7559).

What changed

• Pull checks: expectBudget in test/helpers/budget.ts now warns on stderr when a time is over its limit and passes. It enforces the limit only when FORGE_TEST_TIMING=1. A missing or NaN phase still fails, because that is a broken measurement, not a slow one. This covers all 71 calls in 30 files without editing them.
• Nightly run: .forgejo/workflows/timing.yml runs every test file that calls expectBudget, one at a time on an idle runner, with FORGE_TEST_TIMING=1 and FORGE_TEST_QUIET=1. Each budget is then held to its absolute ms.
• Tests: test/budget.test.ts has a new case showing a 5000 ms time does not throw when timing is off, and a NaN still does. The old "throws on regression" case now sets FORGE_TEST_TIMING=1. test/budget-retry.test.ts sets it so its real-expectBudget subprocess still produces a budget red. I ran those two files, test/flake.test.ts, test/merge-queue-budget-retry.test.ts, and six other budget-using files; bun x tsc --noEmit and bun scripts/gate.ts were clean. I did not run all 41 files scripts/fast.ts named.
• Docs: the speed-budget paragraph in CLAUDE.md describes the new rule.

What I chose not to do

• Count work: the 71 call sites are still wall-clock measurements. Replacing each with a statement, byte or phase count, or a fake clock, is a per-test job, and a test that must catch a regression in a pull needs that kind of check. That is the follow-up the first checkbox still wants.
• Merge queue: I left the re-run's FORGE_TEST_QUIET=1 in src/checks/service.ts. With timing off it never fires, because no budget red can happen.
• Watcher: I didn't wire anything to #7559. The second checkbox is only half done: budgets moved to a scheduled run, but nothing reports them from real traffic yet.

To try it: run bun test test/budget.test.ts, which prints the warning. To see enforcement, run FORGE_TEST_TIMING=1 bun test test/budget.test.ts.

Closes #7789

A pull's check can no longer fail on a slow expectBudget. Wall-clock budgets are now enforced only by a new nightly run. I did not convert any test to count work or a fake clock, and I haven't tied the budgets to the performance watcher (#7559).

What changed

• Pull checks: expectBudget in test/helpers/budget.ts now warns on stderr when a time is over its limit and passes. It enforces the limit only when FORGE_TEST_TIMING=1. A missing or NaN phase still fails, because that is a broken measurement, not a slow one. This covers all 71 calls in 30 files without editing them.
• Nightly run: .forgejo/workflows/timing.yml runs every test file that calls expectBudget, one at a time on an idle runner, with FORGE_TEST_TIMING=1 and FORGE_TEST_QUIET=1. Each budget is then held to its absolute ms.
• Tests: test/budget.test.ts has a new case showing a 5000 ms time does not throw when timing is off, and a NaN still does. The old "throws on regression" case now sets FORGE_TEST_TIMING=1. test/budget-retry.test.ts sets it so its real-expectBudget subprocess still produces a budget red. I ran those two files, test/flake.test.ts, test/merge-queue-budget-retry.test.ts, and six other budget-using files; bun x tsc --noEmit and bun scripts/gate.ts were clean. I did not run all 41 files scripts/fast.ts named.
• Docs: the speed-budget paragraph in CLAUDE.md describes the new rule.

What I chose not to do

• Count work: the 71 call sites are still wall-clock measurements. Replacing each with a statement, byte or phase count, or a fake clock, is a per-test job, and a test that must catch a regression in a pull needs that kind of check. That is the follow-up the first checkbox still wants.
• Merge queue: I left the re-run's FORGE_TEST_QUIET=1 in src/checks/service.ts. With timing off it never fires, because no budget red can happen.
• Watcher: I didn't wire anything to #7559. The second checkbox is only half done: budgets moved to a scheduled run, but nothing reports them from real traffic yet.

To try it: run bun test test/budget.test.ts, which prints the warning. To see enforcement, run FORGE_TEST_TIMING=1 bun test test/budget.test.ts.

I've pushed fixes for all three points and opened the follow-up issue, #7824. The 209 tests in the twelve files that touch workflows, flake-hunt or checks/service pass. tsc and scripts/gate.ts are clean. The queue-side change in src/checks/service.ts has no test of its own, and I haven't run the full suite.

1. Nightly was invisible: scripts/flake-hunt.ts has a new timing [--file-issues] mode. It runs each file that calls expectBudget on its own, and the workflow now passes FORGE_TOKEN, FORGE_URL and the run URL, as outside-gate does. It files Red: issues and closes them when the file is green again, reusing the outside logic. I ran it locally with timing on: 27 files, 0 red.
2. Dead retry path: I kept it and made it reachable on purpose. The merge queue's gate now sets FORGE_TEST_TIMING=1. A budget red there is re-run serially with FORGE_TEST_QUIET=1, as before (src/checks/service.ts). A pull's own check still only warns. I chose this over deleting the path because the retry code is also read by src/crew/autonomy.ts and scripts/shards.ts. It also means a wall-clock regression is still caught before it lands, which covers your minor point. CLAUDE.md and docs/features/gate-budget-retry.md say so. If you'd rather the queue not enforce wall-clock, say so and I'll remove the retry path in a separate round.
3. Cron: timing.yml moved from 47 3 to 23 4, so it no longer shares a slot with outside-gate or flake-hunt.

Issue #7824, "Convert the 71 expectBudget call sites to counted work", covers the conversion to counted work or a fake clock.

No check the forge runs can fail on a wall-clock budget any more, whether it is a pull's own check or the merge queue's gate. A time over budget only prints a warning. The nightly timing workflow is the one place budgets are enforced: it runs each file alone with FORGE_TEST_TIMING=1 FORGE_TEST_QUIET=1 at 23 4 * * *, and files a Red: test/….test.ts issue when a file goes red and closes it when the file is green again. Converting the 71 expectBudget calls to counted work (statements, bytes, phases) or a fake clock is tracked in #7824.

This round, answering @sentinel:

• Gate (point 3): I took the second option: the gate no longer enforces, and the budget-retry path is deleted. The issue asks that a load-only red never sink a landing and that wall-clock budgets move to the scheduled run. Removed:

- src/checks/budget_retry.ts - the serial re-run in src/checks/service.ts, along with the now-unused gate flag on runTier - the budgetRetries field on the tick line in src/crew/autonomy.ts - the fail-fast exception in scripts/shards.ts and scripts/test-shards.ts, so any red shard now stops the rest - test/budget-retry.test.ts, test/merge-queue-budget-retry.test.ts and docs/features/gate-budget-retry.md

The error mark now lives in test/helpers/budget.ts. The wording in CLAUDE.md, the helper's header and timing.yml now says the same thing.

• Env test (point 1): a new test in test/check-tiers.test.ts sets FORGE_TEST_TIMING=1 on the forge process itself. It then checks that a pull's check and a checks.gate run both see the flag unset. Re-adding the flag to the check environment makes it fail; I tried that.
• File selection (points 2 and 4): what the nightly runs is now timingFiles() in scripts/flake.ts. It picks files that import ./helpers/budget.ts and call expectBudget, less an explicit TIMING_EXCLUDED list, which holds only test/budget.test.ts. A file that merely mentions the name, such as test/flake.test.ts, is no longer picked; the earlier name match picked it. A timing Red: issue says it is a budget red and gives the command to reproduce it with timing on. Tests in test/flake.test.ts cover the selection, the real repository's selection, the issue body and the usage line.

tsc and scripts/gate.ts pass. These test files pass: flake, budget, check-tiers, shards, checks, check-steps, crew-brief-gate, tests-row, autonomy, autonomy_sweep and all twelve merge-queue-*.

🤖 Generated with Claude Code

No check the forge runs (a pull's own check or the merge queue's gate) fails on a wall-clock budget any more; a time over budget only prints a warning. The nightly timing workflow is the one place budgets are enforced. It runs at 23 4 * * *, each budget file on its own with FORGE_TEST_TIMING=1 FORGE_TEST_QUIET=1. A newly red file opens a Red: test/….test.ts issue, which closes when the file is green again.

• Gate: it no longer sets FORGE_TEST_TIMING. The budget-retry path that existed only to rescue load-only budget reds is deleted:

- src/checks/budget_retry.ts and its re-run in src/checks/service.ts - the budgetRetries field on the tick line in src/crew/autonomy.ts - the budget exception to fail-fast in scripts/shards.ts and scripts/test-shards.ts, so any red shard now stops the rest - test/budget-retry.test.ts, test/merge-queue-budget-retry.test.ts and docs/features/gate-budget-retry.md

• Proof: a test in test/check-tiers.test.ts sets the flag on the forge process itself and checks that a pull's check and a gate run both see it unset. Putting the flag back into the check's environment makes that test fail; this was tried.
• What the nightly runs: timingFiles() in scripts/flake.ts picks files that import ./helpers/budget.ts and call expectBudget, minus an explicit TIMING_EXCLUDED list. A timing Red: issue gives the command to reproduce it with timing on. Tests are in test/flake.test.ts.
• Merge with main: scripts/test-shards.ts conflicted with #7786. The resolution keeps its keepGoing guard (a verification after landing runs every shard) and drops the old budget exception.
• Docs: CLAUDE.md, the header of test/helpers/budget.ts and timing.yml say the same thing.

The 71 expectBudget calls are still wall-clock measurements, so a slow regression is now caught only by the nightly run, not by a pull. Converting them to counted work (statements, bytes, phases) or a fake clock is #7824.

tsc and scripts/gate.ts pass, and these test files pass after the merge: shards, flake, budget, check-tiers, checks, check-steps, plus the tests that cover #7786.

🤖 Generated with Claude Code