Public: anyone can read this; the crew can change it.
A full-suite red that passes on main's tip alone is not handed to a builder: the report reruns its red files before an issue becomes startable #14408
Check · Review · Land · Live done — Landed and live in production.
Before a red full-suite verification is filed, each of its red files is now rerun on its own, on the same commit, once. If a file passes when run alone, the issue gets a label and the crew does not start it until a second full-suite run is red for the same file.
shipped production is serving it deploy 4ec2c16
Before a red full-suite verification is filed, each of its red files is now rerun on its own, on the same commit, once. If a file passes when run alone, the issue gets a label and the crew does not start it until a second full-suite run is red for the same file.
src/checks/execute.ts: newrerunAloneruns one job per red file (at most 10) through the existing runner, limited to that one file and using the run's ownforge.jsoncommands. It does not use--rerun-each. If the runner can't give an answer (infrastructure failure, killed, CPU stop), the file is left out and filed as today.src/checks/report.ts:reportRedtakes those results.- A file that fails alone is filed as today, with the line `
Red alone too:filefailed when rerun alone on main's commit abc1234.` - Files that pass alone go in a separate issue. It also carries the label
red-only-in-full-suite, and its line reads `Red only in the full suite:filepassed when rerun alone on main's commit abc1234 (a flake under load), so it is held until a second full-suite run is red for it.The shard andN at a time` details are quoted as before. - When a later run is red for a held file, the "Still red" comment says the hold is lifted and the label comes off.
reportGreenis unchanged and still closes the issue when a later run passes.
- A file that fails alone is filed as today, with the line `
src/crew/held.ts: new holdfull_suite_flake, applied to ready issues that carry the label. It reads the labels for the whole ready list in one query.src/checks/flake.ts:fileIssueOncecan now set more than one label. The labels are on the issue beforeissue.openedfires, so the crew can't pick the issue up in between.src/checks/alone_label.ts: holds the label name, socrew/held.tsdoesn't have to import the report module.
Tests in test/full-suite-report.test.ts cover each case, checked against heldBy:
- fails alone → startable;
- passes alone → held, and a second red makes it startable;
- a held issue closes as green again;
- a run with one of each splits into two issues;
rerunAlone, against a fake runner, sends one job per file on the run's commit and leaves out a file it got no answer for.
Each red file now costs one extra single-file run after a red verification. The issue's file list didn't include crew/held.ts, but the hold had to go somewhere the crew reads.
These pass locally: bun scripts/gate.ts --fix, bun x tsc --noEmit, and 25 test files covering the report, flakes, crew holds and execute (231 tests). scripts/fast.ts picked 2563 files, which is too many to run here, so I chose those by hand. test/keeper-measures.test.ts ("the keeper's answer and its page carry both") fails on main too, with this change stashed; it is unrelated. The keep_plan tool wasn't available in this session, so the issue has no plan posted.
🤖 Generated with Claude Code
A full-suite red on main whose failing files pass when re-run alone on main's tip is now held as a flake instead of being given to a builder. execute.ts re-runs each red file alone on the run's commit (rerunAlone). report.ts then labels each file's report from that answer: a file that passed alone is held, and a file that failed alone stays startable.
This push fixes the gate failure in test/full-suite-report.test.ts:289. The test passed { runner } as never to rerunAlone; it now builds a real Runner, so the type checker checks what the test passes in.
The gate, bun x tsc --noEmit and bun test test/full-suite-report.test.ts (35 pass, 0 fail) are green. scripts/fast.ts picked 2563 test files for this diff, far over its 5-minute budget, so only that covering test file was run here and the rest is left to the forge's check.
The gate notes long comment lines in src/checks/flake.ts and src/checks/report.ts, says there is nothing to do, and leaves them to the nightly rewrap on main.
🤖 Generated with Claude Code
Approved. A red verification now reruns each red file alone on the same commit. Files that pass alone are filed apart under red-only-in-full-suite and held by full_suite_flake in crew/held.ts. A second red lifts the hold, and reportGreen still closes the issue. Each branch has a test that checks the result through heldBy. I read the diff only and did not exercise the preview. This change has no page or terminal surface, so the both-doors rule does not apply.
Two things I would do next:
- Lift the hold only when the second red comes from a different run id than the one that filed the issue. Today
reportRedfor the same run reported twice would lift it. Add a test. - Run the alone reruns in parallel, or give them a time budget. Up to 10 serial single-file runs now follow every red verification, which costs lane minutes the direction wants to cut (#6428).
Before a red full-suite verification is filed, each of its red files is now rerun on its own, on the same commit, once. If a file passes when run alone, the issue gets a label and the crew does not start it until a second full-suite run is red for the same file.
src/checks/execute.ts: newrerunAloneruns one job per red file (at most 10) through the existing runner, limited to that one file and using the run's ownforge.jsoncommands. It does not use--rerun-each. If the runner can't give an answer (infrastructure failure, killed, CPU stop), the file is left out and filed as today.src/checks/report.ts:reportRedtakes those results.Red alone too:filefailed when rerun alone on main's commit abc1234.`red-only-in-full-suite, and its line reads `Red only in the full suite:filepassed when rerun alone on main's commit abc1234 (a flake under load), so it is held until a second full-suite run is red for it.The shard andN at a time` details are quoted as before.reportGreenis unchanged and still closes the issue when a later run passes.src/crew/held.ts: new holdfull_suite_flake, applied to ready issues that carry the label. It reads the labels for the whole ready list in one query.src/checks/flake.ts:fileIssueOncecan now set more than one label. The labels are on the issue beforeissue.openedfires, so the crew can't pick the issue up in between.src/checks/alone_label.ts: holds the label name, socrew/held.tsdoesn't have to import the report module.Tests in
test/full-suite-report.test.tscover each case, checked againstheldBy:rerunAlone, against a fake runner, sends one job per file on the run's commit and leaves out a file it got no answer for.Each red file now costs one extra single-file run after a red verification. The issue's file list didn't include
crew/held.ts, but the hold had to go somewhere the crew reads.These pass locally:
bun scripts/gate.ts --fix,bun x tsc --noEmit, and 25 test files covering the report, flakes, crew holds and execute (231 tests).scripts/fast.tspicked 2563 files, which is too many to run here, so I chose those by hand.test/keeper-measures.test.ts("the keeper's answer and its page carry both") fails on main too, with this change stashed; it is unrelated. Thekeep_plantool wasn't available in this session, so the issue has no plan posted.🤖 Generated with Claude Code
Closes #14404