Public: anyone can read this; the crew can change it.
The merge queue never re-forms a batch around the same failing head: suspects are judged alone while the rest lands (one red pull stalled 70+ for ~45 min) #5022
Check · Review · Land · Live done — Landed and live in production.
After a red batch, the merge queue now gates the head alone while the rest of the batch lands without it. It also never puts a pull at the head of a third batch once it has been in two red ones. All of this is in `src/checks/queue.ts`, and the type check and the queue tests (`test/merge-queue*`, `test/autonomy*`, 51 + 2 new) pass. I could not read the forge's `[queue]` log, so the root cause of th…
shipped production is serving it deploy d480d55
After a red batch, the merge queue now gates the head alone while the rest of the batch lands without it. It also never puts a pull at the head of a third batch once it has been in two red ones. All of this is in src/checks/queue.ts, and the type check and the queue tests (test/merge-queue*, test/autonomy*, 51 + 2 new) pass. I could not read the forge's [queue] log, so the root cause of the 02:45–03:27 loop isn't confirmed. The likeliest cause is that the search kept getting cut off, which hands every pull back with no verdict.
What changed
- The head of a red batch is gated alone. This applies when the batch has three or more pulls and a check slot is free. The head is the pull every prefix holds, and it gets a gate of its own beside the line (logged as
red batch: its head is gated alone). The rest go back to the front of the line and re-form without it, stacked up toqueue.depth. - A pull in two red batches is never batched again. The line counts red batches per pull and head (
suspects,suspected). Such a pull goes into no batch and waits for a gate of its own; suspects are gated before the line's next batch. - What a solo verdict does:
- Red alone: the pull is handed back with no other pulls named (logged as red alone). It is remembered as a red, and the tick holds it as for any red.
- Green alone: it rides at the tail of a later batch, never at its head. If it only fails on top of other pulls, the existing search finds it there.
- The count resets. When the search finds the culprit, the other pulls in that batch get one red forgiven (
explained), so innocent pulls don't become suspects. - Free slots: the existing stacking up to
queue.depthgates disjoint batches in parallel. The fix keeps it running while the suspect is judged instead of stopping it.
Tests
- New
test/merge-queue-suspect.test.ts:
- One bad pull at the head and 30 good ones behind, with depth 4. Gate-time one is four candidates; gate-time two is the bad pull alone beside the 30 in four candidates. All 30 land, and the bad pull is handed back after its solo red. - A batch goes red twice with its search cut off both times. On the third try every pull is gated alone, the bad one never heads a batch, and the two good ones land.
- I changed the test "ten pulls, one red" in
test/merge-queue-parallel.test.ts. It now expects four gate-times instead of three, because the new rule gates the innocent head alone first. That test encoded the old search order this issue replaces; the code was not wrong.
Choices to look at
- When the culprit is not the head, a red batch now costs one extra gate-time. That is the price of never re-forming around the head.
- With no free slot, or a red batch of two, the queue still halves the batch as before; the existing suites without spare slots are unchanged.
- A solo green does not land on its own, because that would move main under the batches already in flight.
README.mddescribes the new behaviour in the merge-queue section.
To try it: bun test test/merge-queue-suspect.test.ts test/merge-queue-parallel.test.ts, then look in the log for [queue] red batch: its head is gated alone followed by [queue] red alone.
🤖 Generated with Claude Code
The change does what #4999 asked and stays inside src/checks/queue.ts. A red batch of three or more now gates its head alone while the rest re-forms without it. A pull in two red batches never heads a third. The suspects map is keyed by pull id and head, so a new head clears it. A culprit found elsewhere forgives one red for the others in its batch (explained). The two new tests would fail without the change. The one changed test, ten pulls with one red, is rightly adjusted and the PR says so. There is no new input, write or dependency, so the security and both-doors items do not apply. The README describes the behaviour. I read the diff and tests only; I did not run the tests or exercise the preview.
Two things I would do next:
- Measure the cost on the live queue. When the culprit is not the head, a red batch now takes one more gate-time (3 to 4 in the ten-pull test). That is a trade against #4053/#4073 (the queue lands fast). Watch the
[queue] red batch: its head is gated alonelines. If the culprit is usually not the head, consider gating a middle prefix alone instead. - Tidy
drainin queue.ts. The solo-start loop and thehead === undefinedwait branch now carry a lot of state (solos,line.wake). Pull the solo bookkeeping into its own small function. Add a test where a suspect's solo gate is still running when the line empties, so that wake path is covered.
When a batch of three or more fails its check and a check slot is free, the merge queue now takes the batch's head (the pull that every smaller sub-batch also contains) and checks it on its own, off to the side. The rest of the batch goes back to the front of the queue and is checked without it, stacking batches up to queue.depth. A pull that has been in two failing batches is never put in a third. It waits for its solo verdict: if it fails alone it goes back to its author, and if it passes alone it joins the end of the next batch. One failing pull can no longer hold up the line again and again while the queue rebuilds batches around it, which is how the 70+ pulls behind it stalled for about 45 minutes.
What to look at:
src/checks/queue.ts: where a failing batch picks its suspect, the solo check beside the line, and the "two failing batches" rule.test/merge-queue-suspect.test.tscovers the new behaviour.test/merge-queue-parallel.test.tswas changed: some of its expectations described batches re-forming around the same failing head, which is the behaviour this pull removes.README.mdexplains how the queue treats a suspect.
All 28 merge-queue tests pass. The latest message on #4999 is the automatic triage note saying the issue has no pillar and so waits behind the Later gate. That needs no code; a person has to name a pillar for the issue to move.
To try it: queue three or more pulls where one fails its check. The log shows that pull checked alone while the others are checked together without it, and land.
Closes #4999
🤖 Generated with Claude Code
After a red batch, the merge queue now gates the head alone while the rest of the batch lands without it. It also never puts a pull at the head of a third batch once it has been in two red ones. All of this is in
src/checks/queue.ts, and the type check and the queue tests (test/merge-queue*,test/autonomy*, 51 + 2 new) pass. I could not read the forge's[queue]log, so the root cause of the 02:45–03:27 loop isn't confirmed. The likeliest cause is that the search kept getting cut off, which hands every pull back with no verdict.What changed
red batch: its head is gated alone). The rest go back to the front of the line and re-form without it, stacked up toqueue.depth.suspects,suspected). Such a pull goes into no batch and waits for a gate of its own; suspects are gated before the line's next batch.- Red alone: the pull is handed back with no other pulls named (logged as
red alone). It is remembered as a red, and the tick holds it as for any red. - Green alone: it rides at the tail of a later batch, never at its head. If it only fails on top of other pulls, the existing search finds it there.explained), so innocent pulls don't become suspects.queue.depthgates disjoint batches in parallel. The fix keeps it running while the suspect is judged instead of stopping it.Tests
test/merge-queue-suspect.test.ts:- One bad pull at the head and 30 good ones behind, with depth 4. Gate-time one is four candidates; gate-time two is the bad pull alone beside the 30 in four candidates. All 30 land, and the bad pull is handed back after its solo red. - A batch goes red twice with its search cut off both times. On the third try every pull is gated alone, the bad one never heads a batch, and the two good ones land.
test/merge-queue-parallel.test.ts. It now expects four gate-times instead of three, because the new rule gates the innocent head alone first. That test encoded the old search order this issue replaces; the code was not wrong.Choices to look at
README.mddescribes the new behaviour in the merge-queue section.To try it:
bun test test/merge-queue-suspect.test.ts test/merge-queue-parallel.test.ts, then look in the log for[queue] red batch: its head is gated alonefollowed by[queue] red alone.🤖 Generated with Claude Code
Closes #4999