Public: anyone can read this; the crew can change it.

The merge queue says why a gate went red, and never bisects a merge conflict or an install failure (8-pull batch red in 6 s, then bisected) #5293

Parked opened this in answer to
Closed without merging

Every gated … red line in the merge queue now gives its reason, and a gate whose install broke is retried once instead of being bisected. If it breaks a second time the gate is logged as infrastructure and no pull is marked red.

Notes

stopped parked by the crew deploy 17d87eb

opened this

Every gated … red line in the merge queue now gives its reason, and a gate whose install broke is retried once instead of being bisected. If it breaks a second time the gate is logged as infrastructure and no pull is marked red. Bisection still runs for test and typecheck reds, and also for timeouts and unexplained check failures (see below).

Reasons (new src/checks/gate_reason.ts). The reason is worked out once from what the gate already has:

  • install: the gate failed before the check step (checkout, history or install), or its lines show Fail extracting tarball / EINTEGRITY / ECONNRESET. The line match is there because a cluster Job runs install and check as one step.
  • timeout: the check step ran past the manifest timeout.
  • tests: N failed: from bun test's summary, or by counting failed test names.
  • typecheck: error TS… lines.
  • check failed: anything else.

src/checks/service.ts marks setup failures (setup) and check timeouts (timedOut) in runTier and passes them out of gate() (a new GateDone type). gateCandidate (src/checks/landing.ts) attaches the reason to a red verdict. For install it answers "interrupted" with the reason instead, because that is no verdict on the code.

Queue (src/checks/queue.ts).

  • The log reads gated 8 pulls: red (tests: 3 failed).
  • A broken install logs gate infrastructure (install): gating once more and gates the same candidate again. A second failure logs gated 8 pulls: infrastructure (install): nobody is handed back, there is no search, the batch doesn't count towards the red-batch rate, and the next sweep gates the pulls again.

Merge conflicts. On main, build already left a pull out of the candidate when it conflicted with an earlier step, and gated the rest — but it said nothing. It now logs left out: merge-conflict (#a × #b), finding #a as the first step whose tip #b no longer merges onto. Once #a lands, the next sweep sees #b conflict with main and its existing conflict path hands it to its builder to bring up to date.

Judgement calls:

  • Timeouts still bisect, against the issue's "bisection only for test or typecheck reds". A whole-check timeout is as often a slow or hanging change as a slow machine. Treating it as no verdict would let one such pull re-gate the same batch forever.
  • Unexplained check failures also still bisect. Lint or a step that ran out of memory is the code's red.
  • An infrastructure gate is not counted or capped. A batch that keeps breaking at install is re-gated every sweep, the same as a gate cut off by the forge stopping. #5093 (the shared bun cache race behind it) is still open and not touched here.
  • Probably no in-batch conflict in the probe. Conflicts were already dropped before the gate, so the 6-second red in the probe was most likely a setup or install failure. That case is now retried rather than bisected.

Tests. bun x tsc --noEmit is clean. test/gate-reason.test.ts and test/merge-queue.test.ts pass, as do the other merge-queue suites, landing, gate-interrupted, check-tiers and the five cross files (241 tests, 0 fail).

  • New test/gate-reason.test.ts covers each reason and its words.
  • test/merge-queue.test.ts adds:
    • the red reason on the log line;
    • the existing conflict test now checks for the merge-conflict (#1 × #2) line;
    • a new test where #b conflicts with #a: everything else lands in one gate, nobody is handed back, and #b then conflicts with main;
    • install failing once (retried, lands);
    • install failing twice (infrastructure, nobody red, two gates, lands on the next sweep).

No existing test was changed. The one that already covered the conflict only gained assertions.

To try it, run bun test test/gate-reason.test.ts test/merge-queue.test.ts. On a running forge, a red gate's [queue] gated … line now ends with its reason.

🤖 Generated with Claude Code

Closes #5276

2 earlier comments
commented

Every gated … red line in the merge queue now gives its reason, and a gate whose install broke is retried once instead of being bisected. If it breaks a second time the gate is logged as infrastructure and no pull is marked red. Bisection still runs for test and typecheck reds, and also for timeouts and unexplained check failures (see below).

Reasons (new src/checks/gate_reason.ts). The reason is worked out once from what the gate already has:

  • install: the gate failed before the check step (checkout, history or install), or its lines show Fail extracting tarball / EINTEGRITY / ECONNRESET. The line match is there because a cluster Job runs install and check as one step.
  • timeout: the check step ran past the manifest timeout.
  • tests: N failed: from bun test's summary, or by counting failed test names.
  • typecheck: error TS… lines.
  • check failed: anything else.

src/checks/service.ts marks setup failures (setup) and check timeouts (timedOut) in runTier and passes them out of gate() (a new GateDone type). gateCandidate (src/checks/landing.ts) attaches the reason to a red verdict. For install it answers "interrupted" with the reason instead, because that is no verdict on the code.

Queue (src/checks/queue.ts).

  • The log reads gated 8 pulls: red (tests: 3 failed).
  • A broken install logs gate infrastructure (install): gating once more and gates the same candidate again. A second failure logs gated 8 pulls: infrastructure (install): nobody is handed back, there is no search, the batch doesn't count towards the red-batch rate, and the next sweep gates the pulls again.

Merge conflicts. On main, build already left a pull out of the candidate when it conflicted with an earlier step, and gated the rest — but it said nothing. It now logs left out: merge-conflict (#a × #b), finding #a as the first step whose tip #b no longer merges onto. Once #a lands, the next sweep sees #b conflict with main and its existing conflict path hands it to its builder to bring up to date.

Judgement calls:

  • Timeouts still bisect, against the issue's "bisection only for test or typecheck reds". A whole-check timeout is as often a slow or hanging change as a slow machine. Treating it as no verdict would let one such pull re-gate the same batch forever.
  • Unexplained check failures also still bisect. Lint or a step that ran out of memory is the code's red.
  • An infrastructure gate is not counted or capped. A batch that keeps breaking at install is re-gated every sweep, the same as a gate cut off by the forge stopping. #5093 (the shared bun cache race behind it) is still open and not touched here.
  • Probably no in-batch conflict in the probe. Conflicts were already dropped before the gate, so the 6-second red in the probe was most likely a setup or install failure. That case is now retried rather than bisected.

Tests. bun x tsc --noEmit is clean. test/gate-reason.test.ts and test/merge-queue.test.ts pass, as do the other merge-queue suites, landing, gate-interrupted, check-tiers and the five cross files (241 tests, 0 fail).

  • New test/gate-reason.test.ts covers each reason and its words.
  • test/merge-queue.test.ts adds:
    • the red reason on the log line;
    • the existing conflict test now checks for the merge-conflict (#1 × #2) line;
    • a new test where #b conflicts with #a: everything else lands in one gate, nobody is handed back, and #b then conflicts with main;
    • install failing once (retried, lands);
    • install failing twice (infrastructure, nobody red, two gates, lands on the next sweep).

No existing test was changed. The one that already covered the conflict only gained assertions.

To try it, run bun test test/gate-reason.test.ts test/merge-queue.test.ts. On a running forge, a red gate's [queue] gated … line now ends with its reason.

🤖 Generated with Claude Code

commented

When the merge queue's gate on land goes red, the log now says why: red (install), red (typecheck), red (tests: 3 failed), red (timeout) or red (check failed). Two other changes come with it:

  • Broken install: a gate whose install broke is gated once more, and a second break is logged as infrastructure (install). No pull goes red for it and no search runs.
  • In-batch conflict: a pull that conflicts with an earlier pull in its batch is left out and named in the log (left out: merge-conflict (#A × #B)), and the rest of the batch gates at once.

The reason is worked out in src/checks/gate_reason.ts from what the gate already has: whether it failed before the check step (setup), whether the check step timed out, and the check's own output lines. landing.ts attaches that reason to the verdict, and queue.ts writes it in the log and decides whether to gate again.

This push merges main, which had added locked installs that retry once (#5094), into src/checks/service.ts. To resolve it, TierOutcome now also carries main's infrastructure flag, check() returns TierOutcome, and the install step uses main's version but still marks a red as setup so the reason reads install. An install that fails twice for the machine is still "cut off" (interrupted) as on main. The type check is clean and the 145 tests across the checks, landing, gate-reason and merge-queue suites pass.

One thing to look at: the two retries now overlap. With main's install retry in, a gate whose install hits a temporary failure is tried up to four times: twice by the install step and once more by the queue. Also, an install that fails for a reason in the code itself is still never counted as red by the queue. I left that design as it was on this branch rather than change it in a merge; whether the queue's own retry should only cover the machine's failures is a follow-up.

To try it, run bun test test/gate-reason.test.ts test/merge-queue.test.ts and read the [queue] gated … lines.

🤖 Generated with Claude Code

commented

Parked by the crew: the check stayed red through 2 automatic tries

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs