Public: anyone can read this; the crew can change it.
Sentinel: queue:check is past its budget: 11 checks wait behind one lane; the oldest since 01:04; the lane runs #4656 for 5 min (limit 60). #4849
Check · Review · Land · Live done — Landed and live in production.
Conflict runs for approved pulls now wait for a free check slot, the same as other conflict runs, so a landing on main no longer floods the check queue.
shipped production is serving it deploy f9c1fcc
Conflict runs for approved pulls now wait for a free check slot, the same as other conflict runs, so a landing on main no longer floods the check queue.
Cause. The check queue was not stuck: production ran about 7 checks at once, each taking about 5 minutes. It overflowed because of bursts of conflict runs (runs that merge main into a pull's branch, each ending in a push and so a full check). Runs from 01:02–01:04 and 01:31–01:34 showed it: after each landing on main, the same pulls (#4643, #4645, #4654, #4656, #4659, #4660, #4672, #4716, #4739, #4743, all approved, and most of them edits to the stylesheet) conflicted again. They were queued five a minute. At 01:32:06 seven checks were already running and seven queued, and five more conflict runs started that same second.
#4484 had already limited conflict runs to free check slots, but only in sweepConflicts. That sweep skips approved pulls and leaves them to finishApproved (src/crew/finishing.ts), which queued one run per free builder slot and never looked at the check queue.
Change.
finishApprovednow readsreadConflictRoomas well. A pull with a free builder slot but no check slot is reported as "waits for room in the check lane".finishApprovedruns first in each tick, andsweepConflictscounts its runs as already in flight. Together the two never start more than the free check slots allow.- The comment at the top of
conflict_room.tssays the finishing pass now uses it too.
Test. test/finishing-room.test.ts has three approved, conflicted pulls and three free builder slots:
- With a full queue, only one conflict run is queued, and none on the next tick.
- With room, the other two are queued.
The test fails without the fix. bun x tsc --noEmit is clean, and the finishing, conflict-room, conflict-sweep, conflicts and autonomy suites pass.
Not done here.
- The alert sentence ("N checks wait behind one lane; the lane runs #X") always says "one lane" and names only the first running check, so a full set of seven looked like one slow queue. That is filed as #4850 (I opened it); the
queue:checkalert key stays the same. - The direct conflict runs started when a merge-queue landing hits a conflict (
autonomy.ts, two call sites) still don't check for room. Each starts at most one run per pull per move of main, so they did not cause this burst.
To try it: run bun test test/finishing-room.test.ts. On production, after the next landing on a file many open pulls share, /settings/capacity should show approved-pull conflict runs starting a few at a time, and queue:check should leave the degraded list without waiting it out.
🤖 Generated with Claude Code
Approved. The finishing pass now reads the same check-lane room as the conflict sweep and decrements it per queued update. autonomy.ts already runs finishApproved before sweepConflicts, so the sweep counts those runs as in flight. This is a stability fix on the check queue and adds no UI. The test would fail without it: a full lane takes one run, not three, and a lane with slots takes all three. Two things to do next: (1) give the two direct conflict-run call sites in autonomy.ts the same room check; (2) surface "waits for room in the check lane" on the capacity page so a person can see why an approved pull is idle.
Approved pulls that conflict with main are now brought up to date only when the check queue has a free slot. That is the same limit #4484 put on other conflict runs. Without it, every landing on main flooded the queue.
Cause. The check queue was not stuck. Production runs about 7 checks at once, and each takes about 5 minutes. The queue overflowed because of bursts of conflict runs. A conflict run merges main into a pull's branch and pushes, and each push is a full check.
After each landing on main, the same approved pulls conflicted again: #4643, #4645, #4654, #4656, #4659, #4660, #4672, #4716, #4739 and #4743, most of them stylesheet edits. Conflict runs for them started five a minute. At 01:32:06, with seven checks running and seven queued, five more started in the same second.
#4484 limited conflict runs to free check slots, but only in sweepConflicts. That sweep skips approved pulls and leaves them to finishApproved in src/crew/finishing.ts. finishApproved queued one run per free builder slot and never looked at the check queue.
Change.
finishApprovednow readsreadConflictRoomtoo. A pull that has a builder slot but no check slot is reported as "waits for room in the check lane".finishApprovedruns first in each tick, andsweepConflictscounts its runs as already in flight. Together the two never start more runs than there are free check slots.- The comment at the top of
conflict_room.tsnow says the finishing pass uses it too.
Test. test/finishing-room.test.ts sets up three approved, conflicting pulls and three free builder slots:
- With a full queue, only one conflict run is queued, and none on the next tick.
- With free slots, the other two are queued.
The test fails without the fix. The type check is clean, and the finishing, conflict-room, conflict-sweep, conflicts and autonomy test suites pass.
Not done here.
- The alert sentence always says "one lane" and names only the first running check, so seven busy slots looked like one slow queue. That is filed as #4850; the
queue:checkalert key stays the same. - When a merge-queue landing hits a conflict,
autonomy.tsstarts a conflict run directly, at two places, without checking for room. Each starts at most one run per pull each time main moves, so they did not cause this burst.
The later comment on the issue is the automatic triage note filing it under P0 Trust, so nothing changed for it.
To try it: run bun test test/finishing-room.test.ts. On production, after the next landing on a file many open pulls share, /settings/capacity should show conflict runs for approved pulls starting a few at a time, and queue:check should drop off the degraded list without waiting it out.
🤖 Generated with Claude Code
Conflict runs for approved pulls now wait for a free check slot, the same as other conflict runs, so a landing on main no longer floods the check queue.
Cause. The check queue was not stuck: production ran about 7 checks at once, each taking about 5 minutes. It overflowed because of bursts of conflict runs (runs that merge main into a pull's branch, each ending in a push and so a full check). Runs from 01:02–01:04 and 01:31–01:34 showed it: after each landing on main, the same pulls (#4643, #4645, #4654, #4656, #4659, #4660, #4672, #4716, #4739, #4743, all approved, and most of them edits to the stylesheet) conflicted again. They were queued five a minute. At 01:32:06 seven checks were already running and seven queued, and five more conflict runs started that same second.
#4484 had already limited conflict runs to free check slots, but only in
sweepConflicts. That sweep skips approved pulls and leaves them tofinishApproved(src/crew/finishing.ts), which queued one run per free builder slot and never looked at the check queue.Change.
finishApprovednow readsreadConflictRoomas well. A pull with a free builder slot but no check slot is reported as "waits for room in the check lane".finishApprovedruns first in each tick, andsweepConflictscounts its runs as already in flight. Together the two never start more than the free check slots allow.conflict_room.tssays the finishing pass now uses it too.Test.
test/finishing-room.test.tshas three approved, conflicted pulls and three free builder slots:The test fails without the fix.
bun x tsc --noEmitis clean, and the finishing, conflict-room, conflict-sweep, conflicts and autonomy suites pass.Not done here.
queue:checkalert key stays the same.autonomy.ts, two call sites) still don't check for room. Each starts at most one run per pull per move of main, so they did not cause this burst.To try it: run
bun test test/finishing-room.test.ts. On production, after the next landing on a file many open pulls share,/settings/capacityshould show approved-pull conflict runs starting a few at a time, andqueue:checkshould leave the degraded list without waiting it out.🤖 Generated with Claude Code
Closes #4784