Sentinel: red:01m23vygyffx5hg2f5vm8r74sr was degraded 96 minutes today: trevato/forge: main is red at 931899b — "a private repository is linked only to who may read it" fails; 3 pull checks w… #6643

I raised the timeout on the flaky migrate test from 30 s to 90 s. I never saw the red failure, so I haven't confirmed that the timeout was the cause.

What I found

• Both failing tests pass when run alone. test/migrate.test.ts and test/apps-door.test.ts together came out 37 for 37, and migrate.test.ts on its own 25 for 25 after the change.
• The latest commit on main already says b83077d's check failed migrate.test.ts under load and that it passes alone.
• The test "the app's branch: the person's newest beats the source's default…" runs a git setup, three migrations and a fake GitHub round trip. That makes a timeout the likely failure under load.
• The earlier red, "a private repository is linked only to who may read it" in apps-door.test.ts, no longer fails. Its test already pins a running run to stop the preview deploy racing it, and I made no change there.

What changed

• One line in test/migrate.test.ts (line 1061): the timeout on the "the app's branch…" test goes from 30 s to 90 s. Nothing else changed.

What I chose not to do

• I left the test's body alone, so it still runs three migrations. Splitting it would cut the load, but that is a bigger change than this issue needs.
• If main goes red on this test again with a message other than a timeout, the cause is something else, and the log from that check is the place to start.

How to try it

• Run bun test test/migrate.test.ts.
• The sentinel's red:01m23vygyffx5hg2f5vm8r74sr should stay out of the degraded list once main's check is green again.

Closes #6614

I raised the timeout on the flaky migrate test from 30 s to 90 s. I never saw the red failure, so I haven't confirmed that the timeout was the cause.

What I found

• Both failing tests pass when run alone. test/migrate.test.ts and test/apps-door.test.ts together came out 37 for 37, and migrate.test.ts on its own 25 for 25 after the change.
• The latest commit on main already says b83077d's check failed migrate.test.ts under load and that it passes alone.
• The test "the app's branch: the person's newest beats the source's default…" runs a git setup, three migrations and a fake GitHub round trip. That makes a timeout the likely failure under load.
• The earlier red, "a private repository is linked only to who may read it" in apps-door.test.ts, no longer fails. Its test already pins a running run to stop the preview deploy racing it, and I made no change there.

What changed

• One line in test/migrate.test.ts (line 1061): the timeout on the "the app's branch…" test goes from 30 s to 90 s. Nothing else changed.

What I chose not to do

• I left the test's body alone, so it still runs three migrations. Splitting it would cut the load, but that is a bigger change than this issue needs.
• If main goes red on this test again with a message other than a timeout, the cause is something else, and the log from that check is the place to start.

How to try it

• Run bun test test/migrate.test.ts.
• The sentinel's red:01m23vygyffx5hg2f5vm8r74sr should stay out of the degraded list once main's check is green again.

Steward, 23:15Z: here is the fix, and it's pipeline priority. Main went red twice in an hour on commits that change no code, each on a different single test:

• b83077d (a release-record commit only): migrate > the app's branch: the person's newest beats…
• 88e4bed (an empty commit): reviews, threads and the merge gate > require-review refuses the merge until the head is… (test/tui-workflows.test.ts)

Both pass alone, 3 of 3 (25/25 and 8/8). Each red main holds every pull check and gate up to 20 min (checks/red_main.ts, RED_HOLD_MS), so one flake costs the whole forge about 20 min, and the steward had to push empty commits to re-check.

Done when:

1. A main check that fails ≤2 tests re-runs only those tests once, alone (same commit, same cell) before main counts as red. If they pass alone, main is green-with-flake: no hold, and the test names go on the flake list with the commit and the count.
2. A flake list (a db table plus a vitals card): a test that flakes twice in 7 days gets an issue auto-filed for its file, de-duplicated by test name (not one per red, cf. #6614/#6643).
3. A red main from a real failure still holds, unchanged.
4. Test: a fake check run failing one test that passes alone leaves main green and the flake recorded. A test failing both times leaves main red.

Taken over by nobody; stopped @scout.

Nothing new to push. #6643 is the pull the forge opened from this branch, so the duplicate note points at my own change.

The branch already holds the fix: the timeout on the "the app's branch…" test in test/migrate.test.ts goes from 30 s to 90 s. I didn't read a separate brief on #6643. I can't confirm that the timeout caused the red, because I never saw the failure itself.

Please close #6614 as a duplicate of #6643, and let the pull's check decide whether it lands.

A default-branch check that fails one or two named tests now re-runs only those tests once, alone, before main counts as red. I ran type-check, check-flake.test.ts and five related suites (116 pass, 0 fail), plus design-lint, but not the full suite. I did not try it on a live preview.

What changed

• Re-run: the re-run is bun test <file> -t <name> on the same commit, in the same cell. It happens in src/checks/service.ts right after the check step fails, and only for the default branch's own check.
• Passes alone: the check finishes green. The flake is a row in the new check_flake table (migration 20261026190000-check-flake.ts), and the run log says flaky: … recorded, main stays green. No red-main hold starts.
• Still red: a test that fails alone too, more than two failures, a build error, a timeout, or a hook failure all stay red as before. A re-run that matches no test also counts as a failure, since bun exits 0 on a pattern that matches nothing.
• Names and quoting: names come from the repo's own output, so each is validated and shell-quoted before it reaches the re-run command.
• Flake list: a test that flakes twice in 7 days gets one issue titled Flaky test: <name>, with a Files: line naming its test file. An open issue with that title blocks a second one. src/checks/flake.ts holds the logic.
• Vitals: the vitals gain a flakes field with the week's list (repo, test, file, count, last commit). It shows up in the health answer for an admin. It is not a degraded cause, so a flake never makes the forge look degraded.
• Docs and tests: docs/features/check-flake.md describes the behaviour. test/check-flake.test.ts covers these cases:

- one test fails and passes alone, so main is green and the flake recorded; - a test that fails alone too leaves main red (redMain still reports it); - two flakes in a week file one issue, and a third adds none; - the week window and the command quoting.

What I chose not to do

• No page card: the flake list is in the vitals data only. There is no page card for it, which I'd add as a follow-up.
• No pull checks: pull checks keep their current behaviour, since only main's check holds the forge.
• No cluster Jobs: the re-run needs the check's own cell.
• No named hook failures: hook and (unnamed) failures have no test name to select by, so they stay red.
• No migration pin: I did not add a pin for the new migration; per scripts/pin-migrations.ts that is the release agent's job.

How to try it

• Run bun test test/check-flake.test.ts.
• On a preview, push a commit to main whose check fails one test that passes alone. The run log shows re-running 1 failed test(s) alone, and main stays green.

Closes #6643

The two failing tests in test/vitals.test.ts were my own change, not flakes. The tests were out of date, so I updated them rather than the code. The last push fixes that. I re-ran vitals.test.ts and four related suites (health, guard-verdict, rehearsal, notify-drain); all pass, and tsc is clean. I did not re-run the full check.

• Shape test: the vitals answer now has a flakes key, so the expected key list in "a site admin with a read-only token gets the shape" gains flakes.
• Query-count test: "the number of queries does not grow with runs" expected 9 queries. The flake list adds one read, so it now expects 10. That read is a single fixed query, so the count still does not grow with runs.

The check log also listed shards 24, 26 and 27 as killed after a fail-fast skip, with no test named. That looks like fallout from the red shard rather than a separate problem, so I did nothing about it.

The pull description from my first message still stands. The only addition is these two test updates, committed as fba91691.

All four review points are addressed and pushed in 0bc7e3a3. I ran the flake, board, terminal-screen and vitals suites (72 pass), tsc and design-lint. The forge's full check hasn't run yet, and I haven't looked at the preview.

• Flake list reachable: a site admin now sees a "Flaky this week: <test> ×N (<repo>)" line under the forge notes on the home board. It lists the top three flaky tests and then "and N more". The terminal dashboard's forge strip shows the same line. Both are notes, not needs, so a flake never marks the forge degraded. I read "the health page" as the home board's forge notes, since no other page renders the vitals; correct me if you meant a different page. I added one test for the board, which also checks that a non-admin doesn't see it, and one for the terminal dashboard. The terminal test also checks that a non-admin sees no line.
• Misplaced constant: RERUN_TIMEOUT_S now sits above the TICK_MS doc comment, so that comment describes TICK_MS again.
• migrate.test.ts timeout: reverted to 30 s. That bump came in on this branch's starting commit, and main still has 30 s.
• Indentation: the mainCheck helper in test/check-flake.test.ts is fixed.
• Sibling tests:

- Anchored pattern: the re-run pattern is now ^…$, so a sibling whose name contains the flaky one's is not selected. - Exact count: passedAlone now requires exactly one passing test. The command test pins the anchored form, and a new assertion rejects a re-run where two tests passed.

The docs page docs/features/check-flake.md now names the board and terminal lines. To see it on the preview, sign in as the site admin once main has recorded a flake.

One more rule this needs, seen 2026-10-27 03:15–03:31Z: the prefix #6582,#6590 came back red (4 failed, 369 s), then the longer prefix #6582,#6590,#6592 came back green (696 s). A longer prefix green means the shorter red was a flake. The queue didn't land the green prefix; it gated #6582 alone as a suspect (625 s, green) and landed only that. About 20 gate-minutes were lost, and 1 landing in that hour instead of 3.

Add:

• When a longer stacked prefix is green, land it, including every pull in it, even if a shorter prefix was red. Record the shorter red as a flake (test names onto the flake list).
• A suspect that has ridden in a green prefix is cleared.
• Test: prefixes [a,b] red and [a,b,c] green give a, b and c landing in one step, with no solo gate.

This push covers @juniper's first point and @trevato's longer-prefix rule. I did not do the 3+ flakes alert or the migration pin. The type check and the merge-queue, flake, vitals, health and dashboard tests pass here. I did not run the full suite, and the forge's check runs on this push.

What changed

• Terminal refresh (@juniper's line comment, src/actions/health.ts:113): forgeFlakes now calls flakeList directly, one query, instead of running all of forgeVitals.
• Longer green prefix lands (src/checks/queue.ts, judged):

- In a red candidate's prefix search, the longest green prefix now lands whole, every pull in it, even if a shorter prefix was red. - Every pull in it is cleared as a suspect. - The shorter red goes to a new onFlake hook. src/crew/autonomy.ts wires it to recordFlakes, so the test names reach the flake list and the 2-in-7-days issue rule applies. - Red verdicts now carry the failed test names (tests, from landVerdict). - Test in test/merge-queue-search.test.ts: prefixes [a,b] red and [a,b,c] green land a, b and c in one step. The gates are exactly the whole, [a,b] and [a,b,c], with no solo gate, and the red is reported as a flake.

• Left for the next candidate: in that test the whole was already known red, so d is handed back as the culprit, with no extra gate.

Not done

• Flakes from the queue: these rows have no file, since the queue only sees test names. They show on the list and board, but a filed issue's "Files:" line is empty.
• Stacked flights: the rule covers prefixes inside one candidate's search. A red head flight with a green flight stacked behind it is still discarded and re-formed. Landing that green flight needs settleHead changes, so I'd do it as its own pull.
• 3+ flakes alert (@juniper's second point): not added, so a green main can still hide a repeated flake until someone reads the list.
• Migration pin for the release agent: not added. I couldn't find what it expects, and I won't guess a schema.

The merge conflict is fixed and pushed. Merging main into the branch conflicted in src/checks/queue.ts, src/checks/tiers.ts and src/crew/autonomy.ts. Main had added a persistent gate store, a gate reason on red verdicts and adopt. I kept both sides in each file and changed none of main's code.

The type check is clean and 160 tests pass across the merge-queue, flake, vitals, health and autonomy suites. I did not run the full suite, and the forge's check runs on this push.

This is still the earlier pull description, plus the merge:

• Longer green prefix lands whole: a longer green prefix now lands whole over a shorter red one. Every pull in it is cleared as a suspect. The red is recorded as a flake through onFlake, so the test names reach the flake list.
• Terminal refresh: forgeFlakes now reads the flake list in one query. It no longer recomputes all the vitals.
• Test: [a,b] red and [a,b,c] green land a, b and c in one step with no solo gate.
• Not done: the stacked-flight case, where a green flight stacked behind a red head is still discarded and re-formed. The 3+ flakes alert and the release agent's migration pin are also not done.

All of @juniper's requested changes are in and pushed. The type check, the design lint and 328 tests across the check-flake, merge-queue, vitals, health and board suites pass here. I did not run the full suite, and the forge's check runs on this push.

• Queue flake rows (src/crew/autonomy.ts, src/checks/flake.ts):

- The onFlake callback now calls a new recordQueueFlakes. - The row's commit is the green prefix's tip, and its run id is gate-<tip12>. - The file comes from the red verdicts' learn, but only when they name exactly one file. The verdict names tests and files separately, so with several files or none I can't tell which test came from which. - With no known file, the flake is still counted and listed but files no issue, so there is no empty Files: line. recordFlakes now skips filing whenever the file is empty. - A new test checks what lands in check_flake and the issue body, including the no-issue path.

• Re-run flag (src/checks/service.ts): the main-check re-run now depends on its own isMain flag instead of fileOrderDependent.
• Line width: I wrapped the long lines I had added in src/checks/queue.ts and src/checks/flake.ts. In src/crew/autonomy.ts the lines I added are under the width; the long lines already there I left alone.
• Attention need: a test that flakes 3 or more times in a week now shows in the admin needs list, through flakeNeeds in forgeNeeds. This is not an alert or page card, only a line in that list. A test covers the 2-versus-3 threshold.

Still not done: the stacked-flight case (a green flight stacked behind a red head is still discarded and re-formed) and the release agent's migration pin. I couldn't find what the pin expects and won't guess a schema.

Handed to a person by the crew: this pull is approved, but 2 runs to bring it up to date with main ended without a push, so no more start here. Conflicting files: src/checks/queue.ts. Merge main into it, or say what to try and assign it back to the crew. (The forge posts this under your name; you did not write it.)

This pull now re-runs a failing main check's one or two named tests alone before main counts as red. It is merged up to date with main, and the only conflict was in src/checks/queue.ts. I ran tsc and the flake, merge-queue, TUI-screens and vitals tests, and they pass. I did not run the whole suite, which the forge's check does on push.

• What the pull does: A main check that fails one or two named tests re-runs them alone before main counts as red. Passing alone keeps main green and records the flake. Failing alone, a build error, a hook or more than two failures stay red.
• Queue rule: A longer green prefix lands whole over a shorter red one, and the red is recorded as a flake.
• Where it shows: The flake line is on the home board for an admin only, and in the terminal dashboard.
• Merge: Main added a tests: GateTests field to the gate verdict, which clashed with the tests: string[] field this pull had added. I dropped mine and read the failing tests from main's field.
• Follow-up from the review: Each failing test now carries its own file from the verdict. A queue-recorded flake therefore files an issue even when the verdicts name several files. A test covers it in test/check-flake.test.ts. Flakes with no file anywhere are still listed and file no issue.
• Left out: The stacked-flight case, where a green flight behind a red head is still discarded and re-formed, is for its own pull.
• Try it: bun test test/check-flake.test.ts test/merge-queue-search.test.ts.

I merged main into the branch and pushed it. The only conflict was in src/checks/queue.ts, in the doc comment above judged. I kept both sides' text: the flake rule (a longer green prefix lands whole over a shorter red one) and main's √n − 1 prefix search with leave-one-out. No code needed changing.

tsc is clean, and the flake and merge-queue tests pass (80 tests). I did not run the whole suite; the forge's check does that on the push.

The rest of the pull is as described before:

• A main check that fails one or two named tests re-runs them alone before main counts as red.
• Passing alone keeps main green and records the flake.
• Failing alone, a build error, a hook or more than two failures stay red.
• The queue records a flake when a longer green prefix lands over a shorter red one.
• Each failing test carries its own file, so queue flakes can file issues.
• The flake line is on the home board for an admin only, and in the terminal dashboard.

Left out: the stacked-flight case, where a green flight behind a red head is still discarded and re-formed, is for its own pull.

I changed nothing and pushed nothing, because the diff this review describes isn't the one on the branch. git diff origin/main...HEAD touches 15 files, and none of them is a flow-panel, idle-pull or CSS file.

• Flake work is in the diff:

- The re-run of one or two failed tests alone is in src/checks/flake.ts and src/checks/service.ts. - The recorded flakes are in the check_flake migration, with a flake line on the home board and in the terminal dashboard. - The queue rule is in src/checks/queue.ts (a longer green prefix lands whole over a shorter red one). - The tests are test/check-flake.test.ts, test/merge-queue-search.test.ts, test/board.test.ts, test/vitals.test.ts and test/tui-screens.test.ts.

• Flow-panel work is not here:

- test/flow-panel-idle-parked.test.ts doesn't exist in this tree. - .flow-row appears nowhere in src, and the branch has no flow-panel.ts, flow.ts or components.ts change. - src/flow/idle_pulls.ts is already on main. It is not from this branch (the diff touches no flow file).

• The three comments: the duplicate import in src/tui/screens/flow.ts, the one-off .flow-row class and the idle-pull lines on /boards all belong to the #6480 idle-pull change, not to this branch. They look like they were written against another pull's diff, so I couldn't act on them.

If a review still sees those files, re-diff it against origin/main at 62ca365c.

On the third point: the flake line is on the home board for an admin only, and in the terminal dashboard. Unlike the idle-pull lines, it is not drawn only on /boards.