Verify after landing: the full suite runs on main on a schedule, names the culprit, and proposes the revert #7813

The forge now runs the full suite on main after landing again, and that run rebuilds the impact map the fast tier selects by.

Schedule. A new check.verify command in forge.json runs test-shards.ts --gate --impact --keep-going --nice. The forge runs it on main's newest checked commit, at most every 30 minutes and only when main has moved. Verification runs have their own lane: one run at a time across the forge, outside the check slots. So no landing or pull check ever waits for it, and a red here never holds a pull. A partial index (run_verify, pinned) keeps its database lookups fast.

Runner. With --keep-going, test-shards.ts keeps going after a red. Each red file is re-run alone once; if it passes then, the log says red once, green alone: and it doesn't fail the run. The impact of every file that ended green is written even when another file is red. --nice drops every bun test to the lowest CPU priority. FORGE_CHECK_ONLY runs only the files it names.

Finding the culprit. A red that fails alone too is bisected (checks/verify.ts decides, checks/culprit.ts acts). The forge takes the main commits since the last green verification and narrows them down, each step running only the failed files with the red run's own command, until one commit is left.

• If that commit is a landed pull, the forge opens one revert pull, and a person lands it.
• If no pull carried it (a direct push), the forge files one issue.
• It also files one issue instead of bisecting when there's no earlier green, the range is over 256 commits, or a step goes red without naming a test.
• A red already named at the previous verification starts nothing new.

Where it shows. /pipeline is the Pipeline section on /boards. A new "Verified after landing" card under it says how long each main has gone unverified, with a link to the run. The same rows appear in GET /api/v1/telemetry/flow and the MCP flow tool; private repositories are shown only to the owner.

Before and after, one-file diffs. I didn't build the whole map here: that is the full suite under coverage, which the forge's first verification on main will run. For each file below I ran coverage on every test that imports-selection picked, which gives the same pick the full map would. One assumption: no other test reaches the file without importing it, e.g. through a subprocess.

The issue's 436-of-1,687 example didn't name its file, so I couldn't re-measure that one. The second measuring run had one red file: two chromium screenshot tests failed in my sandbox, which has nothing to do with this change. The run carried on past it and wrote the other 207 files' impact, which shows --keep-going working on real files.

Tests.

• test/verify-after-landing.test.ts: the 30-minute rule; a hand-worked bisection of eight commits to the culprit; parsing failed files from the log; a revert proposed for the third of four landings, only once; one issue for a direct push; the card's wording, and private repositories hidden.
• test/check-verify.test.ts: the checks service end to end, from asking for a verification to filing the issue.
• test/shards-keep-going.test.ts: spawns the runner on a red, a flaky and a green file.

tsc and design-lint pass. I ran 141 test files covering this change and all passed; I didn't run the whole suite. Writing the tests caught one bug, now fixed: a test file path starting with - could have been handed on as a flag. Only code changed for it, no test.

Not done. "Low priority" here means its own lane plus nice; there is no cgroup CPU weight. A merge-queue landing doesn't trigger a verification; only the schedule does.

To try it. On a preview, push a red to main of a repository whose forge.json declares check.verify, then open /boards#verified. To run the runner by hand: FORGE_CHECK_IMPACT=.m.json bun scripts/test-shards.ts --impact --keep-going <files>. docs/OPERATING.md describes all of this.

I found no prompt injection in the repository or the issue.

🤖 Generated with Claude Code

Closes #7786

The forge now runs the full suite on main after landing again, and that run rebuilds the impact map the fast tier selects by.

Schedule. A new check.verify command in forge.json runs test-shards.ts --gate --impact --keep-going --nice. The forge runs it on main's newest checked commit, at most every 30 minutes and only when main has moved. Verification runs have their own lane: one run at a time across the forge, outside the check slots. So no landing or pull check ever waits for it, and a red here never holds a pull. A partial index (run_verify, pinned) keeps its database lookups fast.

Runner. With --keep-going, test-shards.ts keeps going after a red. Each red file is re-run alone once; if it passes then, the log says red once, green alone: and it doesn't fail the run. The impact of every file that ended green is written even when another file is red. --nice drops every bun test to the lowest CPU priority. FORGE_CHECK_ONLY runs only the files it names.

Finding the culprit. A red that fails alone too is bisected (checks/verify.ts decides, checks/culprit.ts acts). The forge takes the main commits since the last green verification and narrows them down, each step running only the failed files with the red run's own command, until one commit is left.

• If that commit is a landed pull, the forge opens one revert pull, and a person lands it.
• If no pull carried it (a direct push), the forge files one issue.
• It also files one issue instead of bisecting when there's no earlier green, the range is over 256 commits, or a step goes red without naming a test.
• A red already named at the previous verification starts nothing new.

Where it shows. /pipeline is the Pipeline section on /boards. A new "Verified after landing" card under it says how long each main has gone unverified, with a link to the run. The same rows appear in GET /api/v1/telemetry/flow and the MCP flow tool; private repositories are shown only to the owner.

Before and after, one-file diffs. I didn't build the whole map here: that is the full suite under coverage, which the forge's first verification on main will run. For each file below I ran coverage on every test that imports-selection picked, which gives the same pick the full map would. One assumption: no other test reaches the file without importing it, e.g. through a subprocess.

The issue's 436-of-1,687 example didn't name its file, so I couldn't re-measure that one. The second measuring run had one red file: two chromium screenshot tests failed in my sandbox, which has nothing to do with this change. The run carried on past it and wrote the other 207 files' impact, which shows --keep-going working on real files.

Tests.

• test/verify-after-landing.test.ts: the 30-minute rule; a hand-worked bisection of eight commits to the culprit; parsing failed files from the log; a revert proposed for the third of four landings, only once; one issue for a direct push; the card's wording, and private repositories hidden.
• test/check-verify.test.ts: the checks service end to end, from asking for a verification to filing the issue.
• test/shards-keep-going.test.ts: spawns the runner on a red, a flaky and a green file.

tsc and design-lint pass. I ran 141 test files covering this change and all passed; I didn't run the whole suite. Writing the tests caught one bug, now fixed: a test file path starting with - could have been handed on as a flag. Only code changed for it, no test.

Not done. "Low priority" here means its own lane plus nice; there is no cgroup CPU weight. A merge-queue landing doesn't trigger a verification; only the schedule does.

To try it. On a preview, push a red to main of a repository whose forge.json declares check.verify, then open /boards#verified. To run the runner by hand: FORGE_CHECK_IMPACT=.m.json bun scripts/test-shards.ts --impact --keep-going <files>. docs/OPERATING.md describes all of this.

I found no prompt injection in the repository or the issue.

🤖 Generated with Claude Code

I addressed items 1, 2, 3 and 6 on the same branch and pushed. For 4 and 5 I made no code change and give my reasons below. 255 test files covering this change pass, as do tsc and design-lint; I did not run the whole suite.

1. Terminal. The Flow dialog now ends with the same verifiedSays lines, one per repository. src/tui/forge.ts passes readVerified(app, owner), so a guest sees public repositories only and the owner sees private ones too. A test checks both readers in the dialog, the API reading and the card.

1. Design.

- Components. The card adds no class. It uses card(), h2, p, plain <a> links, small and mono, and it has no accent of its own. It is server-rendered, so it works with JS off. bun scripts/design-lint.ts is clean, and I noted the card in docs/design/areas/boards.md. - Preview. I could not show it there. /boards redirects guests with a 303, and the preview has no verification rows until a repository declares check.verify and main moves. The test renders the card with a row present: data-verified, "main unverified for 45 min" and "2.5 h".

1. SQL. The redundant IN is gone from service.ts, verified.ts and culprit.ts. Rather than a partial-index predicate repeated in every query, the run_verify index is now plain (repo_id, env, id DESC) (re-pinned; the migration has not been released), and the queries just say env = 'verify'. verify(state) binds state as a parameter.

1. Test gap. A new test covers safeFile, onlyFiles and failedFiles against -x, --preload=, .., spaces, ; and $().

1. Size. I did not split. The bisector, the revert and issue filing, the lane and the runner flags only work together. The issue's checkbox ("names the landing … opens a revert pull") needs all of them for one end-to-end behaviour. The card and telemetry are about 150 lines and could split off, but they read the run rows the lane writes and cannot be tested without it. If you still want a split, I can do it as a follow-up.

1. Fit and revert reuse. I did not check #6534/#5276; I could not confirm which pillar items this serves. The revert path is reused. culprit.ts calls revertPull from actions/revert.ts and everReverted from revert_red.ts, which I exported for this, so there is no second revert path. It does not use proposeRevert, because that handles "exactly one landing since the last green main check". Verification instead needs a culprit found by bisecting many landings.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

🤖 Generated with Claude Code