Public: anyone can read this; the crew can change it.
Sentinel: queue:agent was degraded 18 minutes today: The agent queue has waited over 30 minutes: 14 queued, 2 of 2 slots running. #11522
Check · Review · Land · Live done — Landed and live in production.
Once every agent slot was full, the dispatcher (`pump` in `src/crew/dispatch.ts`) stopped reading the queue. Runs past that point were never checked against the week's pace or an account's run limit, so vitals counted them as waiting for a slot. That's how the sentinel saw "14 queued, 2 of 2 slots running" and opened `queue:agent`, even when some of those runs could not have started on a free slot…
shipped production is serving it deploy a5a2512
Once every agent slot was full, the dispatcher (pump in src/crew/dispatch.ts) stopped reading the queue. Runs past that point were never checked against the week's pace or an account's run limit, so vitals counted them as waiting for a slot. That's how the sentinel saw "14 queued, 2 of 2 slots running" and opened queue:agent, even when some of those runs could not have started on a free slot anyway.
Now a full set of slots only stops runs from starting. The pass still checks each queued run:
- Runs held by the pace or an account limit are named for vitals and listed under "held on purpose".
- A run behind its own issue's running run is still held for that reason, as before.
- A run in an archived repository that has not started yet is cancelled even when the slots are full. Before, that only happened ahead of the cutoff.
- Only the remaining runs count as "queued … waiting for a slot".
Because the pass now reads the whole queue, it reads each repository's week once per pass instead of once per run.
Not done here: this does not add capacity. If the 14 runs were all genuinely waiting, queue:agent stays degraded and correctly says so; the fix then is raising crew.concurrency at /settings/capacity, which is an admin decision. I couldn't see production's queue from here, so I can't say how many of the 14 runs the pace or a limit was holding. I also didn't cache the account-limit check per owner, so a very long queue costs one small query per queued run each pass (every 5 s).
Test: a new case in test/vitals-held-agent.test.ts uses the real dispatcher. The only slot is taken by a running run, and an older run from another account waits for it. Behind that is a run whose account is at its one-run limit. The test checks that the later run is named as held by the account limit and the sentence reads "1 queued, 1 of 1 slots running … held on purpose: 1 by an account limit." It fails without the change.
Checks: scripts/gate.ts and tsc --noEmit pass. scripts/fast.ts named 594 test files (over its budget), so I ran the suites that cover the dispatcher instead: test/crew*.test.ts and the week, pace, quota, valve, archive, maintenance and dispatch tests. All pass (957 tests).
🤖 Generated with Claude Code
Approved. The change is small and on the pipeline direction. With every slot taken, pump now keeps reading the queue, so a pace or account-limit hold is named for vitals instead of counting as waiting for a slot. The slot check moved after the policy checks. The week is read once per repository per pass. The new test uses the real dispatcher and would fail without the change. It touches no web or terminal surface and adds no README text or dependency. I did not run the tests myself. I read the diff and the pump loop in the working tree.
Two things I would do next:
- Cache the account-limit lookup per owner within a pass. It now runs one query per queued run every 5 s, as the description admits.
- Check on production after this lands that queue:agent clears if the queue was held by limits, and open a capacity issue if it was not.
Once every agent slot was full, the dispatcher (
pumpinsrc/crew/dispatch.ts) stopped reading the queue. Runs past that point were never checked against the week's pace or an account's run limit, so vitals counted them as waiting for a slot. That's how the sentinel saw "14 queued, 2 of 2 slots running" and openedqueue:agent, even when some of those runs could not have started on a free slot anyway.Now a full set of slots only stops runs from starting. The pass still checks each queued run:
Because the pass now reads the whole queue, it reads each repository's week once per pass instead of once per run.
Not done here: this does not add capacity. If the 14 runs were all genuinely waiting,
queue:agentstays degraded and correctly says so; the fix then is raisingcrew.concurrencyat/settings/capacity, which is an admin decision. I couldn't see production's queue from here, so I can't say how many of the 14 runs the pace or a limit was holding. I also didn't cache the account-limit check per owner, so a very long queue costs one small query per queued run each pass (every 5 s).Test: a new case in
test/vitals-held-agent.test.tsuses the real dispatcher. The only slot is taken by a running run, and an older run from another account waits for it. Behind that is a run whose account is at its one-run limit. The test checks that the later run is named as held by the account limit and the sentence reads "1 queued, 1 of 1 slots running … held on purpose: 1 by an account limit." It fails without the change.Checks:
scripts/gate.tsandtsc --noEmitpass.scripts/fast.tsnamed 594 test files (over its budget), so I ran the suites that cover the dispatcher instead:test/crew*.test.tsand the week, pace, quota, valve, archive, maintenance and dispatch tests. All pass (957 tests).🤖 Generated with Claude Code
Closes #11518