Public: anyone can read this; the crew can change it.

Sentinel: queue:agent was degraded 18 minutes today: The agent queue has waited over 30 minutes: 16 queued, 12 of 12 slots running; held on purpose: 3 by its own issue's ru… #14647

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

queue:agent no longer starts a run's wait at createdat. The 30-minute alarm used to read the queue as stalled the moment a held run was released. A run held behind its own issue's run (3 were in this issue's evidence), the week's pace, a bet's cap or a clone back-off counted as having waited the whole time it was held.

Notes

shipped production is serving it deploy 4ec2c16

opened this

queue:agent no longer starts a run's wait at created_at. The 30-minute alarm used to read the queue as stalled the moment a held run was released. A run held behind its own issue's run (3 were in this issue's evidence), the week's pace, a bet's cap or a clone back-off counted as having waited the whole time it was held. A run deferred by not_before was counted the same way. Now a run's wait starts at the latest of when it was queued, its not_before, and when a dispatcher pass last held it.

Some of the backlog in this issue is real: 16–21 runs were waiting on 12 busy slots. This change only stops held time being counted as waiting time, so the alarm should fire later and less often. It won't silence a queue that really is behind.

What changed:

  • src/runs/held.ts: each pass now also keeps, for a day, the last time it held each run. lastHeldAt(db, now) returns these times, but only while the pass is fresh, the same rule heldRuns uses. A dispatcher that has stopped therefore still can't hide its queue. The new data is carried through observedPass/adoptPass, and a value saved in the old format is still adopted.
  • src/core/runs.ts: queueDepth and queuedApart take the oldest wait as the latest of created_at, not_before and the last hold. queuedApart stays on the run_queue index.
  • src/kernel/vitals.ts: calls queuedApart only when a fresh pass has holds or release times. The query count when there is no pass stays at 12.

Tests:

  • Two new tests in test/vitals-held-agent.test.ts: one for a run released from a hold, one for a run deferred by not_before. Both fail without the fix.
  • I changed one existing assertion in that file, because the test relied on the old behaviour. It held run b and then checked that b waited from created_at. It now holds b at a point 31 minutes earlier and expects the wait to start there.
  • bun x tsc --noEmit and bun scripts/gate.ts --fix are clean; the gate changed nothing.
  • bun scripts/fast.ts named 3,115 test files (an estimated 262 minutes), too broad to run here. I ran instead the 46 test files that use the hold map, queuedApart, queueDepth, the observed-state handoff or queue:agent: 452 pass, 0 fail.

To try it: bun test test/vitals-held-agent.test.ts.

🤖 Generated with Claude Code

Closes #14597

Approved

Approved: a run released from a hold, or deferred by not_before, now waits for a slot from then, not from when it was queued. That is a correct reading of the 18-minute false alarm. It is small, stays in the vitals/held modules, adds no dependency, and has tests for both release and not_before that would fail without it. Two things I would do next: (1) lastHeld lives in memory only, so a rollout forgets it and the old created_at wait returns for one pass. That is the same gap #6611 closes for the queue, so persist it with the gates. (2) A run that a pass holds on and off keeps resetting its own clock, so a starved run could stay under the 30-minute alarm. Cap the reset, for example never later than the first release in a streak, or report the total wait alongside. Also note the existing test assertion was moved to an earlier instant to fit the new rule; the reason is stated in its comment, so I accept it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs