A file that is red in the full suite and green when run alone is filed as flaky under load, not as red #12302

After a full-suite verification fails, the service now runs each failing file once more on its own. A file that passes alone is marked flakyUnderLoad on the run's detail; a file that fails alone stays a plain red.

• src/checks/rerun.ts (new): rerunAlone(files, runner, { budgetMs, now }) runs the files one at a time and returns, for each, whether it was green alone, red alone, or skipped, plus how long it took. All re-runs share one 5-minute budget, and each run is only given what is left of it. A pass that finishes over the budget counts as red. A file the budget doesn't reach, any file past the tenth, and any path that isn't a plain test path are skipped and stay plain red. The module also builds the log lines (rerun: test/x.test.ts green alone in 3.1 s), the quoted bun test './file' command, and withFlaky, which only names files that were actually in the failing list.
• src/core/runs.ts: CheckDetail gets an optional flakyUnderLoad?: string[]. It is plain data; report.ts is untouched, as the issue asked, so reports don't use the flag until the issue on quoting the assertion lands.
• src/checks/service.ts: on a red verification, the check runs the failing files through rerunAlone, writes one log line per file and puts the flaky ones on the detail. It skips the re-runs when the step was killed, the run was aborted, or the check runs as a cluster Job. The Job exclusion matches the existing single-test flake re-run.
• test/checks-rerun.test.ts checks the green/red split and its log lines, and that files past the budget are skipped. It also checks that the runner gets what is left of the budget, that a runner that throws counts as red, and that unsafe paths and the file-count limit are skipped. It fails without the split or without the budget.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2517 files because runs.ts changed, so I ran the eight verification, flake and report test files myself instead: 109 pass, 0 fail. keep_plan and estimate were not available in this session, so I recorded no plan or estimate on the issue.

To try it: bun test test/checks-rerun.test.ts. On a real red verification, the run log lists one rerun: line per failing file, and the run's detail has flakyUnderLoad.

🤖 Generated with Claude Code

When a full-suite verification fails, the service now runs each failing file once more on its own. A file that passes alone is marked flaky under load instead of plain red.

• src/checks/rerun.ts (new): rerunAlone(files, runner, { budgetMs, now }) runs the files one at a time and returns, for each, whether it passed alone, failed alone, or was skipped, plus how long it took. All re-runs share one 5-minute budget, and each run is only given what is left of it. A pass that finishes over the budget counts as red. A file the budget doesn't reach, any file past the tenth, and any path that isn't a plain test path are skipped and stay plain red. The module also builds:

- the log lines, e.g. rerun: test/x.test.ts green alone in 3.1 s; - the quoted command, bun test './file'; - withFlaky, which only names files that were in the failing list.

• src/core/runs.ts: the run's check detail gets an optional flakyUnderLoad?: string[]. report.ts is unchanged, because the issue on quoting the assertion owns it. The report starts using the flag once both have landed.
• src/checks/service.ts: when a verification fails, the failing files go through rerunAlone and each gets a line in the log. The files that passed alone go on the detail as flakyUnderLoad. The re-runs are skipped when the step was killed, the run was aborted, or the check ran as a cluster Job. The Job case is skipped for the same reason the existing single-test flake re-run skips it.
• test/checks-rerun.test.ts fails without the green/red split or without the budget. It also covers a runner that throws, unsafe paths, the file-count limit, and withFlaky.

The first preview deploy failed on the gate, which runs first in the check. One comment line in rerun.ts was 103 columns, over the 100 limit; it is now wrapped. My first local gate run had printed nothing, and I took that as a pass. The branch also has an empty commit I pushed to redeploy before I found the real cause.

Checks: bun scripts/gate.ts exits 0 and bun x tsc --noEmit is clean. scripts/fast.ts named 2,517 files because runs.ts changed, so I ran the covering tests myself instead (new, verify, report and flake tests): 61 pass, 0 fail. The keep_plan and estimate tools were not available in this session, so I recorded no plan or estimate on the issue.

To try it: run bun test test/checks-rerun.test.ts. On a real failing verification, the run log shows one rerun: line per failing file, and the run's detail has flakyUnderLoad.

🤖 Generated with Claude Code

A red full-suite verification now runs each failing file once more on its own. A file that passes alone is recorded as flakyUnderLoad on the run's detail instead of as a plain red. A file that fails alone stays red. Each re-run writes a rerun: line to the verification's log.

Changes from review

1. The service wiring is now tested. The re-run logic that sat inline in src/checks/service.ts (the guard, the loop and the log lines) is now one function, rerunRed, in src/checks/rerun.ts. The service calls it with its own step runner. The new test test/check-verify-rerun.test.ts drives a real red verification through startChecks:

- With one file that passes alone and one that fails alone, the run detail's flakyUnderLoad holds only the passing file, and both rerun: lines are in the log. - A verification killed by SIGKILL writes no rerun: lines and has no flakyUnderLoad. - With the service call replaced by an empty list, the first test fails on the flakyUnderLoad assertion. - The cluster Job and cancelled-run cases are unit tests on rerunRed: they re-run nothing and write nothing. Covering a Job end to end would need the stand-in kube client from checks-job-vanish, and the guard is the same code either way.

1. Over-budget passes have their own outcome, "over". The log line is rerun: x passed alone in N s, past the budget: not counted. It is not counted as flaky, and a test checks the exact line.
2. RERUN_BUDGET_MS is cut from 5 minutes to 60 s. I timed single files alone here: a unit file takes about 1 s, and files that boot the checks service take 2–6 s (check-flake 2.1 s, check-verify 3.1 s, check-tiers 6.1 s). A minute covers the few files a flake under load leaves red, about 10 even at the slow end. A suite that needs more than that is broken, not loaded, and the most it can add to the gate is one minute.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 2518 files, because service.ts reaches most of the suite, so I chose the covering tests myself: checks-rerun, check-verify-rerun, check-verify, check-flake, check-tiers and check-order-dependent. All 75 tests pass.

Not done: there is no UI or terminal surface for flakyUnderLoad; it lives on the run detail and in the log.

🤖 Generated with Claude Code