URGENT: the merge queue has landed nothing since the 18:54 boot - 33 approved pulls 'queued', no gate running #3436

A candidate gate that has not answered in 15 minutes is now abandoned, so one stuck gate can no longer stop the queue landing anything (#3430).

What the first post-boot sweep waits on: sweepGreen → enqueue → drain → round → gateCandidate → checks.gate. That gate creates no row in /runs and takes no slot in the check lane, which is why the probe saw nothing gating. Nothing bounded it except the manifest's own timeout: for this repo that is 7200 s per step, for install and again for check, and the kube Job calls have no timeout at all. A gate that was slow or hung after boot kept drain busy and the sweep slot taken, so every later tick reused that sweep. Meanwhile lastWaits kept showing the first sweep's queued=33. I can't read production data from here, so I haven't confirmed what this particular gate was stuck on, only that nothing would ever end it.

Changes:

• Deadline (src/checks/queue.ts): each candidate's gate races GATE_DEADLINE_MS (15 min). At the deadline:

- the check it started is aborted; - the log says [queue] candidate abandoned, with the pull numbers, the start time and the deadline; - it counts as interrupted, like a gate cut off by a roll (#3155): no red recorded, no comment, nothing added to the learned tests. Its pulls settle as not landed and the next sweep admits them again.

• Abort reaches the check (src/checks/landing.ts, src/checks/service.ts): gateCandidate and ChecksHandle.gate take an optional signal, and aborting it stops the gate's own check.
• Tick line (src/crew/autonomy.ts): a new queue= field says what the queue is doing now: gating N since HH:MM (UTC), N waiting, both, or idle. The approved= tally is still there.
• Test (test/merge-queue.test.ts): a gate that never answers, with the deadline cut to 100 ms. While it runs, queueWords reads gating 1 since HH:MM. At the deadline its signal is aborted, the abandon line is logged, nothing red is recorded, and the next sweep lands the pull. Without the fix the sweep never returns and the test times out.
• The README's merge-queue section describes the deadline and the new tick field.

The type check passes, and so do the merge-queue, sweep, interrupted-gate, resilience, TUI and check-tier suites.

Not done:

• No deadline on the sweep itself. Once its gates are bounded the sweep is too, unless admit hangs, and I saw nothing pointing to that.
• No timeouts on the kube fetch calls. Racing the deadline covers them, though a hung request stays open in the background.
• The gate still doesn't appear in /runs or take a lane slot.

Watch for: a repo whose full tier really takes longer than 15 minutes, for example because it shares CPUs with other checks, will have every gate abandoned and nothing will land. The [queue] candidate abandoned line repeating with the same pulls is the sign; the fix then is to raise GATE_DEADLINE_MS.

To try it: bun test test/merge-queue.test.ts. On the preview, the tick line should show queue=idle, or gating N since HH:MM while a candidate is gating.

🤖 Generated with Claude Code

Closes #3430

A candidate gate that has not answered in 15 minutes is now abandoned, so one stuck gate can no longer stop the queue landing anything (#3430).

What the first post-boot sweep waits on: sweepGreen → enqueue → drain → round → gateCandidate → checks.gate. That gate creates no row in /runs and takes no slot in the check lane, which is why the probe saw nothing gating. Nothing bounded it except the manifest's own timeout: for this repo that is 7200 s per step, for install and again for check, and the kube Job calls have no timeout at all. A gate that was slow or hung after boot kept drain busy and the sweep slot taken, so every later tick reused that sweep. Meanwhile lastWaits kept showing the first sweep's queued=33. I can't read production data from here, so I haven't confirmed what this particular gate was stuck on, only that nothing would ever end it.

Changes:

• Deadline (src/checks/queue.ts): each candidate's gate races GATE_DEADLINE_MS (15 min). At the deadline:

- the check it started is aborted; - the log says [queue] candidate abandoned, with the pull numbers, the start time and the deadline; - it counts as interrupted, like a gate cut off by a roll (#3155): no red recorded, no comment, nothing added to the learned tests. Its pulls settle as not landed and the next sweep admits them again.

• Abort reaches the check (src/checks/landing.ts, src/checks/service.ts): gateCandidate and ChecksHandle.gate take an optional signal, and aborting it stops the gate's own check.
• Tick line (src/crew/autonomy.ts): a new queue= field says what the queue is doing now: gating N since HH:MM (UTC), N waiting, both, or idle. The approved= tally is still there.
• Test (test/merge-queue.test.ts): a gate that never answers, with the deadline cut to 100 ms. While it runs, queueWords reads gating 1 since HH:MM. At the deadline its signal is aborted, the abandon line is logged, nothing red is recorded, and the next sweep lands the pull. Without the fix the sweep never returns and the test times out.
• The README's merge-queue section describes the deadline and the new tick field.

The type check passes, and so do the merge-queue, sweep, interrupted-gate, resilience, TUI and check-tier suites.

Not done:

• No deadline on the sweep itself. Once its gates are bounded the sweep is too, unless admit hangs, and I saw nothing pointing to that.
• No timeouts on the kube fetch calls. Racing the deadline covers them, though a hung request stays open in the background.
• The gate still doesn't appear in /runs or take a lane slot.

Watch for: a repo whose full tier really takes longer than 15 minutes, for example because it shares CPUs with other checks, will have every gate abandoned and nothing will land. The [queue] candidate abandoned line repeating with the same pulls is the sign; the fix then is to raise GATE_DEADLINE_MS.

To try it: bun test test/merge-queue.test.ts. On the preview, the tick line should show queue=idle, or gating N since HH:MM while a candidate is gating.

🤖 Generated with Claude Code

A merge-queue candidate now carries at most eight pulls, and the log says when each gate starts and how it ends, so a big candidate no longer gates for 20 minutes without a word. This follows the 19:40Z correction: the queue was slow, not hung.

Cap (src/checks/queue.ts): MAX_CANDIDATE = 8. drain takes at most the first eight waiting pulls that share a place and a base branch. Pulls past the cap wait for the next candidate, in order. The 33 pulls after the boot all went into one candidate, which gated and bisected for about 21 minutes. A batch of eight should keep one gate close to a single full tier, and a red needs about three more gates to find the culprit.

Logging: each gate logs [queue] gating N pulls since HH:MM (UTC) when it starts, with the repo, the pull numbers and how many are waiting. When it ends it logs [queue] gated N pulls: green|red|cut off, with how long it took.

Tick line: the queue= field reads like gating 8 pulls since 19:15, 18 waiting, N waiting or idle.

Deadline: kept as the safety net from the first commit. A candidate gate that has not answered in 15 minutes (GATE_DEADLINE_MS) is abandoned:

• the check it started is aborted;
• the log says [queue] candidate abandoned;
• no red is recorded;
• the next sweep admits the pulls again.

The 21 minutes was several gates in a row (a red bisecting), so a single gate should finish well inside the deadline. If one regularly takes longer than 15 minutes, every gate will be abandoned and nothing will land. The repeated [queue] candidate abandoned line is the sign, and the fix is to raise GATE_DEADLINE_MS.

Tests (test/merge-queue.test.ts):

• Ten green pulls make two gates: the first candidate is exactly eight commits on main, all ten land, and the start and end lines appear for each gate.
• A gate that never answers is abandoned at a shortened deadline with its signal aborted, nothing red is recorded, and the next sweep lands the pull.

The type check and the merge-queue, sweep, interrupted-gate, resilience, TUI and check-tier suites pass.

Not done: the cap is a constant, not a setting. Each sweep still waits until its whole line has been gated. The gate still doesn't appear in /runs.

To try it: bun test test/merge-queue.test.ts. On the preview, the tick line shows queue=idle, or queue="gating N pulls since HH:MM, M waiting" while a candidate is gating.

🤖 Generated with Claude Code

The merge queue now gives up on a candidate gate that hasn't answered in 15 minutes, takes at most eight pulls per candidate, and logs each gate as it starts and ends. The autonomy tick line gets a queue= field that says what the queue is doing (queue="gating 2 pulls since 18:54", queue="1 waiting", queue=idle, in UTC).

This update merges main into the branch. Both conflicts came from main's change in #3316/#3329: an approved pull that is green only against an older main now goes straight into the candidate instead of getting its own check re-run first.

• src/crew/autonomy.ts: keeps main's imports, which no longer use checksFor or checkStale, and adds back this branch's queueWords, which the tick line uses for queue=.
• README.md: keeps main's paragraph on stale greens joining the candidate as they are, with its approved="3 approved: queued=2 review=1" example. It adds this branch's sentence about the queue= field. The branch's old text about re-queuing a stale check is dropped, since main replaced that behaviour.

The type check passes. The autonomy, merge-queue, stale-green, check-stale, pull-mergeable and docs suites pass (69 tests).

🤖 Generated with Claude Code

The merge queue now gates at most eight pulls per candidate, gives up on a gate that hasn't answered in 15 minutes, and says in the log and on the tick line what it is doing. With the steward's 33 approved pulls, the line goes through in batches of eight. Each gate is about 7 minutes instead of one silent ~21-minute gate, and a red result is cheaper to bisect.

What changed

• Size cap (src/checks/queue.ts): drain takes at most MAX_CANDIDATE (8) pulls from the front of the line. The rest wait for the next candidate.
• Logging: each gate logs when it starts and when it ends: [queue] gating N pulls since HH:MM with the pull numbers and how many are waiting, then [queue] gated N pulls: <outcome> with how long it took.
• Deadline: a gate that hasn't answered in 15 minutes is abandoned. Its abort signal is fired (src/checks/service.ts passes it on to the check) and [queue] candidate abandoned is logged. No verdict is recorded, so nobody is told their pull went red, and the line forms again on the next sweep.
• Tick line (src/crew/autonomy.ts): it now reads queue: gating N pulls since HH:MM, N waiting, both, or idle, instead of a bare queued= count. queueWords in queue.ts builds this text.
• Tests (test/merge-queue.test.ts): ten green pulls make two candidates (8 then 2) and the log shows both. A gate that never answers is abandoned at a shortened deadline, and the next sweep lands the pull.
• Test fix in the last commit: the abandoned-gate test failed once in my first dozen runs. It checked the status after a fixed 30 ms, but building the candidate (git work) can take longer than that on a cold run. It now waits until the gate has actually started. The test was wrong, not the code. The type check is clean, and the queue suite passed 13 of 13 on every run since the fix.

Left out on purpose

• The issue is still labelled "Later" with no pillar named; whoever owns the plan needs to name one.
• The cap and the deadline are constants, not settings.

To try it: run bun test test/merge-queue.test.ts. On a preview with several approved green pulls, the log shows one [queue] gating 8 pulls since … line per candidate, and the tick line shows queue: ….

🤖 Generated with Claude Code