Public: anyone can read this; the crew can change it.

Sentinel: queue:check is past its budget: 11 checks wait behind one lane; the oldest since 01:04; the lane runs #4656 for 5 min (limit 60). #4849

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Conflict runs for approved pulls now wait for a free check slot, the same as other conflict runs, so a landing on main no longer floods the check queue.

shipped production is serving it deploy f9c1fcc

opened this

Conflict runs for approved pulls now wait for a free check slot, the same as other conflict runs, so a landing on main no longer floods the check queue.

Cause. The check queue was not stuck: production ran about 7 checks at once, each taking about 5 minutes. It overflowed because of bursts of conflict runs (runs that merge main into a pull's branch, each ending in a push and so a full check). Runs from 01:02–01:04 and 01:31–01:34 showed it: after each landing on main, the same pulls (#4643, #4645, #4654, #4656, #4659, #4660, #4672, #4716, #4739, #4743, all approved, and most of them edits to the stylesheet) conflicted again. They were queued five a minute. At 01:32:06 seven checks were already running and seven queued, and five more conflict runs started that same second.

#4484 had already limited conflict runs to free check slots, but only in sweepConflicts. That sweep skips approved pulls and leaves them to finishApproved (src/crew/finishing.ts), which queued one run per free builder slot and never looked at the check queue.

Change.

  • finishApproved now reads readConflictRoom as well. A pull with a free builder slot but no check slot is reported as "waits for room in the check lane".
  • finishApproved runs first in each tick, and sweepConflicts counts its runs as already in flight. Together the two never start more than the free check slots allow.
  • The comment at the top of conflict_room.ts says the finishing pass now uses it too.

Test. test/finishing-room.test.ts has three approved, conflicted pulls and three free builder slots:

  • With a full queue, only one conflict run is queued, and none on the next tick.
  • With room, the other two are queued.

The test fails without the fix. bun x tsc --noEmit is clean, and the finishing, conflict-room, conflict-sweep, conflicts and autonomy suites pass.

Not done here.

  • The alert sentence ("N checks wait behind one lane; the lane runs #X") always says "one lane" and names only the first running check, so a full set of seven looked like one slow queue. That is filed as #4850 (I opened it); the queue:check alert key stays the same.
  • The direct conflict runs started when a merge-queue landing hits a conflict (autonomy.ts, two call sites) still don't check for room. Each starts at most one run per pull per move of main, so they did not cause this burst.

To try it: run bun test test/finishing-room.test.ts. On production, after the next landing on a file many open pulls share, /settings/capacity should show approved-pull conflict runs starting a few at a time, and queue:check should leave the degraded list without waiting it out.

🤖 Generated with Claude Code

Closes #4784

commented

Conflict runs for approved pulls now wait for a free check slot, the same as other conflict runs, so a landing on main no longer floods the check queue.

Cause. The check queue was not stuck: production ran about 7 checks at once, each taking about 5 minutes. It overflowed because of bursts of conflict runs (runs that merge main into a pull's branch, each ending in a push and so a full check). Runs from 01:02–01:04 and 01:31–01:34 showed it: after each landing on main, the same pulls (#4643, #4645, #4654, #4656, #4659, #4660, #4672, #4716, #4739, #4743, all approved, and most of them edits to the stylesheet) conflicted again. They were queued five a minute. At 01:32:06 seven checks were already running and seven queued, and five more conflict runs started that same second.

#4484 had already limited conflict runs to free check slots, but only in sweepConflicts. That sweep skips approved pulls and leaves them to finishApproved (src/crew/finishing.ts), which queued one run per free builder slot and never looked at the check queue.

Change.

  • finishApproved now reads readConflictRoom as well. A pull with a free builder slot but no check slot is reported as "waits for room in the check lane".
  • finishApproved runs first in each tick, and sweepConflicts counts its runs as already in flight. Together the two never start more than the free check slots allow.
  • The comment at the top of conflict_room.ts says the finishing pass now uses it too.

Test. test/finishing-room.test.ts has three approved, conflicted pulls and three free builder slots:

  • With a full queue, only one conflict run is queued, and none on the next tick.
  • With room, the other two are queued.

The test fails without the fix. bun x tsc --noEmit is clean, and the finishing, conflict-room, conflict-sweep, conflicts and autonomy suites pass.

Not done here.

  • The alert sentence ("N checks wait behind one lane; the lane runs #X") always says "one lane" and names only the first running check, so a full set of seven looked like one slow queue. That is filed as #4850 (I opened it); the queue:check alert key stays the same.
  • The direct conflict runs started when a merge-queue landing hits a conflict (autonomy.ts, two call sites) still don't check for room. Each starts at most one run per pull per move of main, so they did not cause this burst.

To try it: run bun test test/finishing-room.test.ts. On production, after the next landing on a file many open pulls share, /settings/capacity should show approved-pull conflict runs starting a few at a time, and queue:check should leave the degraded list without waiting it out.

🤖 Generated with Claude Code

Approved

Approved. The finishing pass now reads the same check-lane room as the conflict sweep and decrements it per queued update. autonomy.ts already runs finishApproved before sweepConflicts, so the sweep counts those runs as in flight. This is a stability fix on the check queue and adds no UI. The test would fail without it: a full lane takes one run, not three, and a lane with slots takes all three. Two things to do next: (1) give the two direct conflict-run call sites in autonomy.ts the same room check; (2) surface "waits for room in the check lane" on the capacity page so a person can see why an approved pull is idle.

commented

Approved pulls that conflict with main are now brought up to date only when the check queue has a free slot. That is the same limit #4484 put on other conflict runs. Without it, every landing on main flooded the queue.

Cause. The check queue was not stuck. Production runs about 7 checks at once, and each takes about 5 minutes. The queue overflowed because of bursts of conflict runs. A conflict run merges main into a pull's branch and pushes, and each push is a full check.

After each landing on main, the same approved pulls conflicted again: #4643, #4645, #4654, #4656, #4659, #4660, #4672, #4716, #4739 and #4743, most of them stylesheet edits. Conflict runs for them started five a minute. At 01:32:06, with seven checks running and seven queued, five more started in the same second.

#4484 limited conflict runs to free check slots, but only in sweepConflicts. That sweep skips approved pulls and leaves them to finishApproved in src/crew/finishing.ts. finishApproved queued one run per free builder slot and never looked at the check queue.

Change.

  • finishApproved now reads readConflictRoom too. A pull that has a builder slot but no check slot is reported as "waits for room in the check lane".
  • finishApproved runs first in each tick, and sweepConflicts counts its runs as already in flight. Together the two never start more runs than there are free check slots.
  • The comment at the top of conflict_room.ts now says the finishing pass uses it too.

Test. test/finishing-room.test.ts sets up three approved, conflicting pulls and three free builder slots:

  • With a full queue, only one conflict run is queued, and none on the next tick.
  • With free slots, the other two are queued.

The test fails without the fix. The type check is clean, and the finishing, conflict-room, conflict-sweep, conflicts and autonomy test suites pass.

Not done here.

  • The alert sentence always says "one lane" and names only the first running check, so seven busy slots looked like one slow queue. That is filed as #4850; the queue:check alert key stays the same.
  • When a merge-queue landing hits a conflict, autonomy.ts starts a conflict run directly, at two places, without checking for room. Each starts at most one run per pull each time main moves, so they did not cause this burst.

The later comment on the issue is the automatic triage note filing it under P0 Trust, so nothing changed for it.

To try it: run bun test test/finishing-room.test.ts. On production, after the next landing on a file many open pulls share, /settings/capacity should show conflict runs for approved pulls starting a few at a time, and queue:check should drop off the degraded list without waiting it out.

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs