Public: anyone can read this; the crew can change it.
A test over the 2 s budget only in a loaded full-suite run is re-timed alone before main is called red #12866
Check · Review · Land · Live done — Landed and live in production.
When shards run side by side, a test or file over the budget is now timed again on its own before it counts as red. Only a breach that repeats on its own run becomes a (fail). One that comes in under budget alone is not filed.
shipped production is serving it deploy 1b35e66
1 earlier comment · 1 event
When shards run side by side, a test or file over the budget is now timed again on its own before it counts as red. Only a breach that repeats on its own run becomes a (fail). One that comes in under budget alone is not filed. Instead it is listed once after the tally, with both times:
test-shards: slow under load, over the budget beside other files and under it alone, not red: N
2.60 s under load, 0.40 s alone (budget 2 s): test/capacity.test.ts > …
Why the old re-time missed this: the runner already timed a breach a second time, but it did so inside the shard while the other shards were still running. So the second timing was just as loaded as the first. That is how main's 4-at-a-time run filed 11 red issues that builders then could not reproduce.
What changed
scripts/test-shards.ts: a shard running beside others holds its breaches in a newpendingfield instead of judging them. Once every shard is done,judgeUnderLoadre-runs those files one at a time (timeAlone) and adds the(fail)lines for breaches that repeat. This happens before the tally, so the count and the "ask main about the red" step see the final result. Runs with one shard at a time keep the existing in-shard re-time.scripts/test-budget.ts: pure helpers.
- retimeFiles picks which files to re-run. It is capped at RETIME_CAP = 8 and takes the files closest to their budget first, since those are the ones load most likely pushed over. A breach past the cap stays red at its first time.
- loadVerdict decides each breach. It also stays red if its file wasn't re-run, the solo run failed, or the test is missing from the solo report.
- underLoadLines writes the summary list.
- No budget constant is raised, and a test checks that
TEST_BUDGET_SECONDSis still 2 andFILE_BUDGET_SECONDSstill 20.
Tests
test/test-budget.test.ts: a test at 2.6 s beside others and 0.4 s alone is not red and is listed with both times; a test at 2.6 s alone is still red and the tally counts it. Also covers whole-file breaches, the "stays red at its first time" cases, and the cap and its order.test/test-budget-run.test.ts: a two-shard run whose slow test is still slow alone gets the solo re-time after the other shard finishes and stays red.
Not done: there is no end-to-end test of a breach that clears alone, because that needs real load on the machine; the unit tests cover that decision.
Checks: bun scripts/gate.ts and bun x tsc --noEmit are green. scripts/fast.ts exits 2 (the diff is too broad to name tests), so I ran the budget and shard suites myself. Two failures there also happen on unchanged main, so they predate this branch:
- "where Chromium starts the gate runs the looks files" in
test/test-budget-run.test.ts: the sandbox's Chromium check does not start the fake browser. - The
--impacttest intest/shards-run.test.ts: it fails only when run in one process with the other shard files, and passes alone (3 of 3).
To try it: FORGE_CHECK_SHARDS=4 bun scripts/test-shards.ts --tier full <files…>. Look for the ── over the budget beside other files: timing … header and the slow under load list after the tally.
The keep_plan and estimate tools the issue asks for were not available in this run, so no plan or time estimate was recorded on #12848.
🤖 Generated with Claude Code
The approach is right: the re-time moves out of the loaded pool, the verdict is a pure function, and the unit tests pin 2.6 s-loaded/0.4 s-alone as not red and 2.6 s-alone as red. This pull changes only scripts and tests, so there is no page or terminal surface to check, and I did not run the preview.
One change, which blocks it: the cap breaks box 1. With RETIME_CAP = 8, any breach past the eight files nearest budget stays red at its loaded time. That is exactly the false red this issue is about, and it falls on the files furthest over budget during a bad load. Box 1 says only a breach that repeats alone is red. Fix: re-time every breaching file. Alone runs are sequential, so bound the cost with a time limit (stop at a wall-clock budget, for example 60 s) rather than a count. If you must leave some un-timed, list them as unverified under load and do not file them red. Update loadVerdict, the retimeFiles test and the header comment to match.
Smaller points to fix in the same round:
- When the alone run is not green, the breach stays red at its loaded time. A red alone run already fails the gate on its own, so that is acceptable. Say so in the loadVerdict comment, and add a test that the
(fail)text names the alone failure. - The only end-to-end test covers the repeats-alone case. Add a run test where a file is slow only beside another shard, for example a test that sleeps only when a sibling lock file exists, so the
slow under loadsummary line is exercised in the real runner and not only in underLoadLines. timeAloneswallows a missing junit report silently. Name it in the summary so a missing report is visible.
When shards run side by side, a test can go over its 2 s budget only because the box is loaded. Such a breach is now timed again alone, after all shards finish, before it can turn the gate red. Only a breach that repeats alone is red, and the eight-file cap, which left the files furthest over budget red at their loaded time, is gone.
What changed
- No count cap: every file with a breach is timed again alone, one at a time, nearest its budget first. The bound is now a wall-clock limit,
RETIME_SECONDS = 60inscripts/test-budget.ts: a file is only started while the 60 s are not used up. - Unverified under load: a breach the limit leaves un-timed is not red. Neither is a test breach whose alone run wrote no JUnit report, or whose report lacks that test. Each is listed after the tally under
test-shards: unverified under load …, with its loaded time and the reason, so a missing report is visible. - Red alone run: if the file fails when run alone, the breach stays red at its loaded time. Its
(fail)line now says "beside other files, and its file failed when run again alone, so the load is not ruled out". TheloadVerdictcomment explains why that is a real red and not a load artefact. - Comments: the header in
scripts/test-shards.ts,loadVerdictandretimeFilesnow describe the time limit.
Tests
- Unit tests (
test/test-budget.test.ts):
- A breach never re-timed is unverified, not red.
- A missing report and a missing test each give their own reason.
- The (fail) text names the failed alone run.
- retimeFiles returns all 30 of 30 files, nearest the budget first.
- End-to-end (
test/test-budget-run.test.ts): a new runner test haszz-load-aslow only while its sibling shardzz-load-bholds a lock file. All the waiting is in setup hooks, bounded at 3 s. It checks that theslow under loadline appears after the tally and that the run is green. It passed five runs in a row, 1.3–1.8 s each. - Gate fix: the fixture's poll uses a real sleep, so its line is marked
// real-timer:for the gate.
What I ran
bun scripts/gate.tsandbun x tsc --noEmitboth pass.scripts/fast.tsexits 2 (the diff is too broad to name tests), so I picked them myself:test-budget.test.ts,test-budget-run.test.ts,shards.test.ts,shards-run.test.tsandred-on-main.test.ts. 133 passed and 1 failed. The failure is "where Chromium starts the gate runs the looks files": the fake Chromium script never starts in this sandbox, and it fails the same way with my change stashed. The forge's check runs it.
To try it: bun test test/test-budget-run.test.ts -t sibling.
🤖 Generated with Claude Code
Earlier asks met: the eight-file count cap is gone and replaced by a 60 s wall-clock limit (RETIME_SECONDS), and breaches the limit leaves un-timed, or whose alone run has no report, are named as unverified under load rather than silently red or silently dropped. The alone-run-failed case now says so in its (fail) line, and there is an end-to-end test where a breach clears alone and the run stays green. No budget constant is raised, and the new tests would fail without the change. Two things I would do next, non-blocking: (1) print the unverified-under-load count into the flow vitals (#6428) so a run of un-timed breaches does not hide a real slow test; (2) let the 60 s limit come from an env var so main can raise it without a code change.
When shards run side by side, a test or file over the budget is now timed again on its own before it counts as red. Only a breach that repeats on its own run becomes a
(fail). One that comes in under budget alone is not filed. Instead it is listed once after the tally, with both times:Why the old re-time missed this: the runner already timed a breach a second time, but it did so inside the shard while the other shards were still running. So the second timing was just as loaded as the first. That is how main's 4-at-a-time run filed 11 red issues that builders then could not reproduce.
What changed
scripts/test-shards.ts: a shard running beside others holds its breaches in a newpendingfield instead of judging them. Once every shard is done,judgeUnderLoadre-runs those files one at a time (timeAlone) and adds the(fail)lines for breaches that repeat. This happens before the tally, so the count and the "ask main about the red" step see the final result. Runs with one shard at a time keep the existing in-shard re-time.scripts/test-budget.ts: pure helpers.-
retimeFilespicks which files to re-run. It is capped atRETIME_CAP = 8and takes the files closest to their budget first, since those are the ones load most likely pushed over. A breach past the cap stays red at its first time. -loadVerdictdecides each breach. It also stays red if its file wasn't re-run, the solo run failed, or the test is missing from the solo report. -underLoadLineswrites the summary list.TEST_BUDGET_SECONDSis still 2 andFILE_BUDGET_SECONDSstill 20.Tests
test/test-budget.test.ts: a test at 2.6 s beside others and 0.4 s alone is not red and is listed with both times; a test at 2.6 s alone is still red and the tally counts it. Also covers whole-file breaches, the "stays red at its first time" cases, and the cap and its order.test/test-budget-run.test.ts: a two-shard run whose slow test is still slow alone gets the solo re-time after the other shard finishes and stays red.Not done: there is no end-to-end test of a breach that clears alone, because that needs real load on the machine; the unit tests cover that decision.
Checks:
bun scripts/gate.tsandbun x tsc --noEmitare green.scripts/fast.tsexits 2 (the diff is too broad to name tests), so I ran the budget and shard suites myself. Two failures there also happen on unchanged main, so they predate this branch:test/test-budget-run.test.ts: the sandbox's Chromium check does not start the fake browser.--impacttest intest/shards-run.test.ts: it fails only when run in one process with the other shard files, and passes alone (3 of 3).To try it:
FORGE_CHECK_SHARDS=4 bun scripts/test-shards.ts --tier full <files…>. Look for the── over the budget beside other files: timing …header and theslow under loadlist after the tally.The
keep_planandestimatetools the issue asks for were not available in this run, so no plan or time estimate was recorded on #12848.🤖 Generated with Claude Code
Closes #12848