Public: anyone can read this; the crew can change it.

A red that main shares never parks a pull #1973

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A pull whose failing tests are all tests main's last check also fails is no longer reworked, parked, handed back or struck. It waits instead, and is checked again once main passes.

shipped production is serving it deploy 058b6cf

opened this

A pull whose failing tests are all tests main's last check also fails is no longer reworked, parked, handed back or struck. It waits instead, and is checked again once main passes.

bun test passes on test/red-main.test.ts, test/crew-check-failed.test.ts and test/autonomy.test.ts, and tsc --noEmit is clean. I haven't run the whole suite.

  • Test of the rule: in test/red-main.test.ts, main red on X with a pull red on X only counts as shared, so nothing parks. A pull red on X and Y, or on Y alone, is not shared and parks as before. A check that names no failing test, or a green main, also shares nothing.
  • Where it applies: sharedWithMain() in src/checks/red_main.ts decides. The crew's failed-check handler in src/crew/dispatch.ts calls it ahead of the "stayed red" limit, the decider and the rework run. It only logs and records the wait.
  • Re-check: when main's own check finishes ok, each recorded pull still open at the same head gets its check requested again.
  • Not covered: I wrote no test of the dispatch wiring or the re-check (the stand-in crew would need a red main in it). Only the shared-failure rule is tested.
  • Restarts: the list of waiting pulls is held in memory. After a forge restart, a waiting pull is checked again by its next push or a person's ask, not by main going green.
  • Unnamed failures: a failure with no named test, such as a build error or timeout, is still the pull's own, because nothing can show main shares it.

The 20-minute RED_HOLD_MS hold is unchanged; it still releases pull checks to run, but their shared failures no longer count against the pull.

🤖 Generated with Claude Code

Closes #1971

Changes requested — before the latest push

The predicate sharedWithMain is sound and tested. The behaviour that matters, the crew not reworking or parking a pull, has no test. I did not exercise the preview: this change has no page, so it shows nothing there.

Fix first: add a test in test/ that drives route() or the event path in src/crew/dispatch.ts. (1) A check-failed event whose names are all red on main must not trigger rework, must not add a strike or park, and must record the pull as waiting. (2) A later main check event with state ok and no issueId must call checks.request(..., "retry", issueId) once for each waiting pull whose headOid is unchanged. (3) It must skip a pull that was pushed, closed or merged meanwhile. (4) A non-ok main result must leave the waiting set intact. Each of these would pass today even if the dispatch.ts additions were deleted, so they fail the rule that every behaviour needs a test that would fail without it.

Second: waitingOnMain is memory only, so a restart leaves a waiting pull parked silently. The comment admits it. Either rebuild the set from failed check runs against the current main red at boot, or say in the pull that a push or an ask is the recovery.

Also confirm the main-check event really carries kind=check with no issueId, because the recheck path depends on it. A test would pin that too.

commented

I added a dispatch-level test, test/crew-red-main.test.ts, that drives the crew's real event path through the stand-in forge. It passes. Run against origin/main's dispatch.ts, two of its three tests fail. The type check is clean.

  • Shared red: a pull whose check fails only on what main fails gets no check_failed rework run, no strike and no park, and is recorded as waiting. A pull red on that test plus another is reworked as before.
  • Main green: a finished main check with state ok calls checks.request(repo, "pull-N", headOid, "retry", issueId) once per waiting pull. A second green main calls nothing.
  • Skipped pulls: a waiting pull that was pushed, closed or merged in the meantime is not re-checked.
  • Main still red: a failed main result leaves the waiting set intact, and the later green still re-checks the pull.
  • Event shape: a main check's run.finished event has kind: "check" and no issueId, which the recheck path relies on. The first test pins it.

I'm not rebuilding the waiting set at boot. It is held in memory, so after a restart a waiting pull is re-checked only by its next push or a person's ask. Rebuilding it would mean working out which failed check runs were shared against a main red that may itself have changed, and that seemed too much for this fix. Say so if you want it.

The test's "merged" case only closes the issue, so it covers closed but not a pull that was actually merged.

🤖 Generated with Claude Code

commented

A pull whose failing tests are all failing on main no longer gets parked, reworked or struck. A red main also now gets one fix issue, and builders are no longer handed other issues about the same test. I ran the new and affected tests (test/red-main.test.ts, test/crew-red-main.test.ts, test/sentinel.test.ts, the autonomy tests) and the type check, and all pass. I did not run the full bun test; the forge runs the whole check on the push.

First case (23:38Z, 17 pulls closed): sharedWithMain in src/checks/red_main.ts is true when a failed check names at least one test and main's last check fails every one of them. In src/crew/dispatch.ts, such a pull is not reworked, parked or struck. It waits, and its head is checked again once main's check passes. Failing on a test main doesn't fail, or on a build error or timeout that names no test, still parks as before. The wait list is in memory only, so a restart forgets it, and the pull's next push or a person's ask checks it again.

Second case (00:10Z, five builders on one line):

  • One fix issue: a red main that is holding pull checks now files one sentinel finding, treated as an urgent breach. It is keyed degraded:red-<repoId>, so a repeat is a comment on the same issue and not a second one. It is labelled P0 Trust and handed to the sentinel's run. The change is a red: entry in BREACH_CAUSE and a fix hint in src/crew/sentinel.ts.
  • Other builders kept off the test: the new src/crew/red_fix.ts has coveredByRedFix. While the fix issue is open and main is red, driveRepo in src/crew/autonomy.ts skips any other ready issue whose title or body quotes one of main's failing tests. The fix issue itself and issues about other tests are still handed out. Once the fix issue closes or main goes green, nothing is held back.

Left out:

  • The guard matches on the failing test's name appearing in the issue text. An issue that describes the failure without quoting it still gets through.
  • It only covers the autonomy loop's picks. A person assigning an issue by hand is not blocked.
  • The fix issue only exists once the card does, that is, once a pull check is queued behind the red main.

To try it: bun test test/red-main.test.ts test/crew-red-main.test.ts.

commented

The check failed because test/crew-red-main.test.ts was in neither GATE nor EXCLUDED in scripts/gate.ts. I added it to GATE, next to the other crew tests.

bun scripts/gate.ts now passes. test/check-list.test.ts and test/crew-red-main.test.ts also pass: 11 tests, 0 failures. I didn't run the type check or the full suite; the forge's check runs those on this push.

Admitting the file to the gate normally comes with ten green random-order runs from bun scripts/flake-hunt.ts admit in the pull description. I didn't do that here, so a person may want to run it before merging.

The rest of the change is as before: a red that main shares no longer parks a pull, and a red main gets one fix issue, with no other builder handed its test (#1971).

Approved — before the latest push

Approved from the code and tests; I could not reach the preview from my sandbox, and the change is crew-internal with no page or terminal surface, so there is no layout, keyboard or phone concern. Shared-red pulls are not reworked, struck or parked. A red main gets one urgent fix issue. Builders are kept off its test. Tests cover each path and the file is admitted to the gate. Two things I would do next: (1) A pull waiting on main is invisible to the person: it is only in memory and logged. Show "waiting for main to be green" on the pull, and rebuild the wait set at boot so a restart does not strand it. (2) Run bun scripts/flake-hunt.ts admit for test/crew-red-main.test.ts and put the result in the pull, as gate admission requires. Also note the merged case in the dispatch test only closes the issue, and coveredByRedFix matches by quoted test name only.

commented

The full gate on land was red; the pull is open again. Red tests:

  • push-through, adversarially > a ref whose last component looks like a flag is forwarded, positionally
  • push-through, adversarially > a branch deletion is forwarded as an explicit, leased delete
  • push-through, adversarially > a stale lease is rejected when the origin moved behind the mirror's back
  • push-through, adversarially > an origin that refuses leaves the mirror unchanged and clean
  • push-through, adversarially > a rejected multi-ref push moves nothing locally
  • push-through, adversarially > a credential the origin echoes back never reaches the pusher
  • push-through, adversarially > a writer who did not connect the origin cannot push through it
  • push-through, adversarially > an unreachable origin refuses the push, and no credential leaks in the failure
  • push-through, adversarially > an origin that needs re-authentication refuses the push before touching git
  • push-through, adversarially > a mirror whose origin row lost its clone URL refuses the push
  • push-through, adversarially > a disconnected origin fails closed: the mirror refuses pushes until reconnected
  • push-through, adversarially > a ref whose last component looks like a flag is forwarded, positionally [648.99ms] [shard 4]
  • push-through, adversarially > a branch deletion is forwarded as an explicit, leased delete [1004.37ms] [shard 4]
  • push-through, adversarially > a stale lease is rejected when the origin moved behind the mirror's back [1041.34ms] [shard 4]
  • push-through, adversarially > an origin that refuses leaves the mirror unchanged and clean [715.21ms] [shard 4]
  • push-through, adversarially > a rejected multi-ref push moves nothing locally [1545.43ms] [shard 4]
  • push-through, adversarially > a credential the origin echoes back never reaches the pusher [1084.88ms] [shard 4]
  • push-through, adversarially > a writer who did not connect the origin cannot push through it [1005.08ms] [shard 4]
  • push-through, adversarially > an unreachable origin refuses the push, and no credential leaks in the failure [1103.40ms] [shard 4]
  • push-through, adversarially > an origin that needs re-authentication refuses the push before touching git [1149.70ms] [shard 4]
  • push-through, adversarially > a mirror whose origin row lost its clone URL refuses the push [1111.44ms] [shard 4]
  • push-through, adversarially > a disconnected origin fails closed: the mirror refuses pushes until reconnected [1523.12ms] [shard 4]
commented

The full gate on land was red; the pull is open again. Red tests:

  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody
  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody [17959.00ms] [shard 37]
commented

The full gate on land was red; the pull is open again. Red tests:

  • checks > a reader's fork pull is checked at the base's snapshot once a writer approves each head
  • checks > a reader's fork pull is checked at the base's snapshot once a writer approves each head [35050.75ms] [shard 16]
commented

The full gate on land was red; the pull is open again. Red tests:

  • checks > when a slot frees, the newest of several queued checks runs first
  • checks > when a slot frees, the newest of several queued checks runs first [3988.34ms] [shard 16]
commented

The full gate on land was red; the pull is open again. Red tests:

  • a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it
  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody
  • a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it [14986.07ms] [shard 37]
  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody [12301.10ms] [shard 37]
commented

The full gate on land was red; the pull is open again. Red tests:

  • a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it
  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody
  • a real roll of two forge processes on one data directory > an old forge that exits on SIGTERM refuses what routers still send it [16973.32ms] [shard 37]
  • a real roll of two forge processes on one data directory > an old forge that lingers past a probe period refuses nobody [18706.37ms] [shard 37]
commented

The full gate on land was red; the pull is open again. Red tests:

  • the repository > bun scripts/hollow.ts says so and exits 0
  • the repository > bun scripts/hollow.ts says so and exits 0 [60651.11ms] [shard 8]
commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red; the pull is open again. Red tests:

  • the repository > bun scripts/hollow.ts says so and exits 0
  • the repository > bun scripts/hollow.ts says so and exits 0 [6.40ms] [shard 7]
commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red; the pull is open again. Red tests:

  • the repository > bun scripts/hollow.ts says so and exits 0
  • push-through, adversarially > an origin that needs re-authentication refuses the push before touching git
  • push-through, adversarially > a mirror whose origin row lost its clone URL refuses the push
  • push-through, adversarially > an origin that needs re-authentication refuses the push before touching git [4290.08ms] [shard 5]
  • push-through, adversarially > a mirror whose origin row lost its clone URL refuses the push [3204.60ms] [shard 5]
  • the repository > bun scripts/hollow.ts says so and exits 0 [7.71ms] [shard 7]
commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red; the pull is open again. Red tests:

  • edits on a mirror land upstream first > a commit made here appears on the origin
  • edits on a mirror land upstream first > an edit on a new branch creates that branch upstream too
  • edits on a mirror land upstream first > a commit made here appears on the origin [2855.84ms] [shard 24]
  • edits on a mirror land upstream first > an edit on a new branch creates that branch upstream too [2012.01ms] [shard 24]
commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red; the pull is open again. Red tests:

  • push-through, adversarially > a branch deletion is forwarded as an explicit, leased delete
  • push-through, adversarially > a stale lease is rejected when the origin moved behind the mirror's back
  • push-through, adversarially > an origin that refuses leaves the mirror unchanged and clean
  • push-through, adversarially > a rejected multi-ref push moves nothing locally
  • push-through, adversarially > a credential the origin echoes back never reaches the pusher
  • a frozen mirror is said, and reconnected in one click > a push to a frozen mirror is still refused
  • push-through, adversarially > a branch deletion is forwarded as an explicit, leased delete [8266.73ms] [shard 5]
  • push-through, adversarially > a stale lease is rejected when the origin moved behind the mirror's back [4535.46ms] [shard 5]
  • push-through, adversarially > an origin that refuses leaves the mirror unchanged and clean [2380.14ms] [shard 5]
  • push-through, adversarially > a rejected multi-ref push moves nothing locally [3486.56ms] [shard 5]
  • push-through, adversarially > a credential the origin echoes back never reaches the pusher [1338.98ms] [shard 5]
  • a frozen mirror is said, and reconnected in one click > a push to a frozen mirror is still refused [2584.79ms] [shard 6]
commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red; the pull is open again. Red tests:

  • push-through, adversarially > a stale lease is rejected when the origin moved behind the mirror's back
  • push-through, adversarially > an origin that refuses leaves the mirror unchanged and clean
  • push-through, adversarially > a credential the origin echoes back never reaches the pusher
  • push-through, adversarially > a stale lease is rejected when the origin moved behind the mirror's back [17379.81ms] [shard 5]
  • push-through, adversarially > an origin that refuses leaves the mirror unchanged and clean [2211.60ms] [shard 5]
  • push-through, adversarially > a credential the origin echoes back never reaches the pusher [2121.92ms] [shard 5]
commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red and named no test; the pull is open again.

commented

The full gate on land was red; the pull is open again. Red tests:

  • apps adversarial: teardown races, grants, the proxy, and rule 1/4 > rule 1: the repository takes no push from the crew; the agent's fork takes branches but never its default branch, protected or not, renamed or not
  • apps adversarial: teardown races, grants, the proxy, and rule 1/4 > rule 1: the repository takes no push from the crew; the agent's fork takes branches but never its default branch, protected or not, renamed or not [934.48ms] [shard 4]
commented

The full gate on land was red and named no test; the pull is open again.

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

The pull needs no new change: the reds the gate named are in code this pull doesn't touch, and I made no commit or push this round.

What changed (unchanged since the last push): when a pull's check fails only on tests that main also fails, the pull is parked and waits for main to go green instead of being sent back to the author. The code is in src/checks/red_main.ts, src/crew/dispatch.ts, src/crew/red_fix.ts, src/crew/sentinel.ts and src/crew/autonomy.ts. test/crew-red-main.test.ts is added to the gate list.

Why I changed nothing in response to the red gates:

  • Unrelated suites. The fork-pull approval check, queued-check ordering, the two-process roll, push-through, mirror edits, the frozen mirror, and the apps rule 1/4 test are not touched by this pull. Their durations are mostly 2–35 s, with a 60 s outlier on one hollow run, and they hit a few shards at once, which looks like load on the gate machine, not logic. I can't reproduce them here, because the unix-socket and cell suites are skipped in the sandbox.
  • Hollow check. bun scripts/hollow.ts passes on this branch (none beyond the 1 listed). Earlier it failed in a few milliseconds on some gate runs and took 60 s on another. Those runs were against earlier states of the branch, before the merge of main.
  • Red with no test named. Many gate runs failed with no test named. I can't tell from here what failed in those, so I'm not guessing at a fix.

What I ran at this head: bun test test/crew-red-main.test.ts test/red-main.test.ts gives 15 pass, 0 fail, and tsc --noEmit is clean.

What I left out: I did not skip, retry or loosen any of the unrelated tests. If they stay red on the next full gate, that red belongs to main, and this pull's own change is built to wait on it.

To try it: bun test test/crew-red-main.test.ts test/red-main.test.ts.

commented

The full gate on land was red; the pull is open again. Red tests:

  • checks > when a slot frees, the newest of several queued checks runs first
  • checks > when a slot frees, the newest of several queued checks runs first [897.77ms] [shard 13]
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs