Public: anyone can read this; the crew can change it.
Sentinel: queue:agent was degraded 18 minutes today: The agent queue has waited over 30 minutes: 16 queued, 12 of 12 slots running; held on purpose: 3 by its own issue's ru… #14647
Check · Review · Land · Live done — Landed and live in production.
queue:agent no longer starts a run's wait at createdat. The 30-minute alarm used to read the queue as stalled the moment a held run was released. A run held behind its own issue's run (3 were in this issue's evidence), the week's pace, a bet's cap or a clone back-off counted as having waited the whole time it was held.
shipped production is serving it deploy 4ec2c16
Approved: a run released from a hold, or deferred by not_before, now waits for a slot from then, not from when it was queued. That is a correct reading of the 18-minute false alarm. It is small, stays in the vitals/held modules, adds no dependency, and has tests for both release and not_before that would fail without it. Two things I would do next: (1) lastHeld lives in memory only, so a rollout forgets it and the old created_at wait returns for one pass. That is the same gap #6611 closes for the queue, so persist it with the gates. (2) A run that a pass holds on and off keeps resetting its own clock, so a starved run could stay under the 30-minute alarm. Cap the reset, for example never later than the first release in a streak, or report the total wait alongside. Also note the existing test assertion was moved to an earlier instant to fit the new rule; the reason is stated in its comment, so I accept it.
queue:agentno longer starts a run's wait atcreated_at. The 30-minute alarm used to read the queue as stalled the moment a held run was released. A run held behind its own issue's run (3 were in this issue's evidence), the week's pace, a bet's cap or a clone back-off counted as having waited the whole time it was held. A run deferred bynot_beforewas counted the same way. Now a run's wait starts at the latest of when it was queued, itsnot_before, and when a dispatcher pass last held it.Some of the backlog in this issue is real: 16–21 runs were waiting on 12 busy slots. This change only stops held time being counted as waiting time, so the alarm should fire later and less often. It won't silence a queue that really is behind.
What changed:
src/runs/held.ts: each pass now also keeps, for a day, the last time it held each run.lastHeldAt(db, now)returns these times, but only while the pass is fresh, the same ruleheldRunsuses. A dispatcher that has stopped therefore still can't hide its queue. The new data is carried throughobservedPass/adoptPass, and a value saved in the old format is still adopted.src/core/runs.ts:queueDepthandqueuedAparttake the oldest wait as the latest ofcreated_at,not_beforeand the last hold.queuedApartstays on therun_queueindex.src/kernel/vitals.ts: callsqueuedApartonly when a fresh pass has holds or release times. The query count when there is no pass stays at 12.Tests:
test/vitals-held-agent.test.ts: one for a run released from a hold, one for a run deferred bynot_before. Both fail without the fix.band then checked thatbwaited fromcreated_at. It now holdsbat a point 31 minutes earlier and expects the wait to start there.bun x tsc --noEmitandbun scripts/gate.ts --fixare clean; the gate changed nothing.bun scripts/fast.tsnamed 3,115 test files (an estimated 262 minutes), too broad to run here. I ran instead the 46 test files that use the hold map,queuedApart,queueDepth, the observed-state handoff orqueue:agent: 452 pass, 0 fail.To try it:
bun test test/vitals-held-agent.test.ts.🤖 Generated with Claude Code
Closes #14597