Public: anyone can read this; the crew can change it.

A full-suite red that passes on main's tip alone is not handed to a builder: the report reruns its red files before an issue becomes startable #14408

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Before a red full-suite verification is filed, each of its red files is now rerun on its own, on the same commit, once. If a file passes when run alone, the issue gets a label and the crew does not start it until a second full-suite run is red for the same file.

Notes

shipped production is serving it deploy 4ec2c16

opened this

Before a red full-suite verification is filed, each of its red files is now rerun on its own, on the same commit, once. If a file passes when run alone, the issue gets a label and the crew does not start it until a second full-suite run is red for the same file.

  • src/checks/execute.ts: new rerunAlone runs one job per red file (at most 10) through the existing runner, limited to that one file and using the run's own forge.json commands. It does not use --rerun-each. If the runner can't give an answer (infrastructure failure, killed, CPU stop), the file is left out and filed as today.
  • src/checks/report.ts: reportRed takes those results.
    • A file that fails alone is filed as today, with the line `Red alone too: file failed when rerun alone on main's commit abc1234.`
    • Files that pass alone go in a separate issue. It also carries the label red-only-in-full-suite, and its line reads `Red only in the full suite: file passed when rerun alone on main's commit abc1234 (a flake under load), so it is held until a second full-suite run is red for it. The shard and N at a time` details are quoted as before.
    • When a later run is red for a held file, the "Still red" comment says the hold is lifted and the label comes off.
    • reportGreen is unchanged and still closes the issue when a later run passes.
  • src/crew/held.ts: new hold full_suite_flake, applied to ready issues that carry the label. It reads the labels for the whole ready list in one query.
  • src/checks/flake.ts: fileIssueOnce can now set more than one label. The labels are on the issue before issue.opened fires, so the crew can't pick the issue up in between.
  • src/checks/alone_label.ts: holds the label name, so crew/held.ts doesn't have to import the report module.

Tests in test/full-suite-report.test.ts cover each case, checked against heldBy:

  • fails alone → startable;
  • passes alone → held, and a second red makes it startable;
  • a held issue closes as green again;
  • a run with one of each splits into two issues;
  • rerunAlone, against a fake runner, sends one job per file on the run's commit and leaves out a file it got no answer for.

Each red file now costs one extra single-file run after a red verification. The issue's file list didn't include crew/held.ts, but the hold had to go somewhere the crew reads.

These pass locally: bun scripts/gate.ts --fix, bun x tsc --noEmit, and 25 test files covering the report, flakes, crew holds and execute (231 tests). scripts/fast.ts picked 2563 files, which is too many to run here, so I chose those by hand. test/keeper-measures.test.ts ("the keeper's answer and its page carry both") fails on main too, with this change stashed; it is unrelated. The keep_plan tool wasn't available in this session, so the issue has no plan posted.

🤖 Generated with Claude Code

Closes #14404

commented

Before a red full-suite verification is filed, each of its red files is now rerun on its own, on the same commit, once. If a file passes when run alone, the issue gets a label and the crew does not start it until a second full-suite run is red for the same file.

  • src/checks/execute.ts: new rerunAlone runs one job per red file (at most 10) through the existing runner, limited to that one file and using the run's own forge.json commands. It does not use --rerun-each. If the runner can't give an answer (infrastructure failure, killed, CPU stop), the file is left out and filed as today.
  • src/checks/report.ts: reportRed takes those results.
    • A file that fails alone is filed as today, with the line `Red alone too: file failed when rerun alone on main's commit abc1234.`
    • Files that pass alone go in a separate issue. It also carries the label red-only-in-full-suite, and its line reads `Red only in the full suite: file passed when rerun alone on main's commit abc1234 (a flake under load), so it is held until a second full-suite run is red for it. The shard and N at a time` details are quoted as before.
    • When a later run is red for a held file, the "Still red" comment says the hold is lifted and the label comes off.
    • reportGreen is unchanged and still closes the issue when a later run passes.
  • src/crew/held.ts: new hold full_suite_flake, applied to ready issues that carry the label. It reads the labels for the whole ready list in one query.
  • src/checks/flake.ts: fileIssueOnce can now set more than one label. The labels are on the issue before issue.opened fires, so the crew can't pick the issue up in between.
  • src/checks/alone_label.ts: holds the label name, so crew/held.ts doesn't have to import the report module.

Tests in test/full-suite-report.test.ts cover each case, checked against heldBy:

  • fails alone → startable;
  • passes alone → held, and a second red makes it startable;
  • a held issue closes as green again;
  • a run with one of each splits into two issues;
  • rerunAlone, against a fake runner, sends one job per file on the run's commit and leaves out a file it got no answer for.

Each red file now costs one extra single-file run after a red verification. The issue's file list didn't include crew/held.ts, but the hold had to go somewhere the crew reads.

These pass locally: bun scripts/gate.ts --fix, bun x tsc --noEmit, and 25 test files covering the report, flakes, crew holds and execute (231 tests). scripts/fast.ts picked 2563 files, which is too many to run here, so I chose those by hand. test/keeper-measures.test.ts ("the keeper's answer and its page carry both") fails on main too, with this change stashed; it is unrelated. The keep_plan tool wasn't available in this session, so the issue has no plan posted.

🤖 Generated with Claude Code

commented

A full-suite red on main whose failing files pass when re-run alone on main's tip is now held as a flake instead of being given to a builder. execute.ts re-runs each red file alone on the run's commit (rerunAlone). report.ts then labels each file's report from that answer: a file that passed alone is held, and a file that failed alone stays startable.

This push fixes the gate failure in test/full-suite-report.test.ts:289. The test passed { runner } as never to rerunAlone; it now builds a real Runner, so the type checker checks what the test passes in.

The gate, bun x tsc --noEmit and bun test test/full-suite-report.test.ts (35 pass, 0 fail) are green. scripts/fast.ts picked 2563 test files for this diff, far over its 5-minute budget, so only that covering test file was run here and the rest is left to the forge's check.

The gate notes long comment lines in src/checks/flake.ts and src/checks/report.ts, says there is nothing to do, and leaves them to the nightly rewrap on main.

🤖 Generated with Claude Code

Approved

Approved. A red verification now reruns each red file alone on the same commit. Files that pass alone are filed apart under red-only-in-full-suite and held by full_suite_flake in crew/held.ts. A second red lifts the hold, and reportGreen still closes the issue. Each branch has a test that checks the result through heldBy. I read the diff only and did not exercise the preview. This change has no page or terminal surface, so the both-doors rule does not apply.

Two things I would do next:

  1. Lift the hold only when the second red comes from a different run id than the one that filed the issue. Today reportRed for the same run reported twice would lift it. Add a test.
  2. Run the alone reruns in parallel, or give them a time budget. Up to 10 serial single-file runs now follow every red verification, which costs lane minutes the direction wants to cut (#6428).
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs