Sentinel: queue:agent was degraded 50 minutes today: The agent queue has waited over 30 minutes: 2 queued, 2 of 2 slots running. #10209

queue:agent no longer counts a run that is waiting on its own issue's earlier run as a stalled queue. I haven't confirmed that this is what caused the 50 minutes in #10192. The vitals test file passes, tsc is clean, and the gate script printed no output, which I took as passing. The change is pushed, but I haven't seen the forge's check result yet.

Why I think it's the cause. The dispatcher starts only one agent run per issue at a time (pump in src/crew/dispatch.ts). A queued run whose issue already had a run going was skipped, but vitals still counted it as waiting for a slot and aged it toward the 30-minute alarm. That fits "2 queued, 2 of 2 slots running". It also fits a real backlog behind two long runs, and the evidence in the issue can't tell the two apart.

What changed.

• src/runs/held.ts has a new hold cause, issue, worded "its own issue's run still going".
• pump marks those skipped runs as held with that cause. The existing held-run logic then leaves them out of the queue's age and "since", and names them in the sentence (for example "held on purpose: 1 by its own issue's run still going").
• A new test in test/vitals-held-agent.test.ts covers it. Without the change, the run counts as waiting.

What I didn't do. I left the 30-minute limit alone. If both slots are held by long runs for over 30 minutes while other issues queue, the alarm still fires, because that is a real saturation signal.

How to check. The fix worked if queue:agent stays out of the degraded list for a day, or holds under 10 minutes. If it fires again with distinct issues queued, the cause was slot saturation, and the next step is raising crew.concurrency at /settings/capacity.

🤖 Generated with Claude Code

Closes #10192

queue:agent no longer counts a run that is waiting on its own issue's earlier run as a stalled queue. I haven't confirmed that this is what caused the 50 minutes in #10192. The vitals test file passes, tsc is clean, and the gate script printed no output, which I took as passing. The change is pushed, but I haven't seen the forge's check result yet.

Why I think it's the cause. The dispatcher starts only one agent run per issue at a time (pump in src/crew/dispatch.ts). A queued run whose issue already had a run going was skipped, but vitals still counted it as waiting for a slot and aged it toward the 30-minute alarm. That fits "2 queued, 2 of 2 slots running". It also fits a real backlog behind two long runs, and the evidence in the issue can't tell the two apart.

What changed.

• src/runs/held.ts has a new hold cause, issue, worded "its own issue's run still going".
• pump marks those skipped runs as held with that cause. The existing held-run logic then leaves them out of the queue's age and "since", and names them in the sentence (for example "held on purpose: 1 by its own issue's run still going").
• A new test in test/vitals-held-agent.test.ts covers it. Without the change, the run counts as waiting.

What I didn't do. I left the 30-minute limit alone. If both slots are held by long runs for over 30 minutes while other issues queue, the alarm still fires, because that is a real saturation signal.

How to check. The fix worked if queue:agent stays out of the degraded list for a day, or holds under 10 minutes. If it fires again with distinct issues queued, the cause was slot saturation, and the next step is raising crew.concurrency at /settings/capacity.

🤖 Generated with Claude Code

I've addressed all three of juniper's points and pushed. bun x tsc --noEmit is clean and test/vitals-held-agent.test.ts passes (6 tests). I ran only that file, plus the gate script, which printed nothing; I'm leaving the full check to the forge.

The cause is still unconfirmed: runs queued behind their own issue's running run are no longer counted as a stalled queue. That fits "2 queued, 2 of 2 slots running", but a plain backlog behind two long runs fits it too.

What changed in src/crew/dispatch.ts, src/runs/held.ts, test/vitals-held-agent.test.ts

• Test now uses the real dispatcher. It replaces the noteHeld-only test. One run is running on an issue, a sibling run for the same issue is queued, and an unrelated older run sits ahead of the sibling, with a single slot. It starts the crew and waits for the sibling to be marked issue. It then asserts the vitals sentence reads "1 queued, 1 of 1 slots running … held on purpose: 1 by its own issue's run still going." The sibling is named but not counted.
• Test checked against reverts. It fails with dispatch.ts at the base version, and also with only the new slot-limit block removed (point 1 below). It passes with both in place.
• Point 1 (the slots <= 0 break). When slots run out, the pass now marks the remaining runs that wait on a running sibling before it breaks. Nothing else starts differently. The unrelated run ahead of the sibling in the test covers this.
• Point 2 (a hung sibling). The alarm is not silenced for a hung sibling. The running run is caught by the stuck: vitals card, which fires on a lapsed heartbeat or after 6 hours (NO_PROGRESS_MS). I added a comment in pump saying so. I did not add a separate test for that card.

🤖 Generated with Claude Code

queue:agent no longer counts a queued agent run that is waiting behind a run on its own issue as a stalled queue. The dispatcher starts only one run per issue at a time, so that run is not waiting for a slot. Before this change it still aged toward the 30-minute alarm and counted toward the "N queued" in the alarm's message.

It is not confirmed that this caused #10192's 50 minutes. "2 queued, 2 of 2 slots running" fits this case, but it also fits a real backlog behind two long runs.

What changed

• src/runs/held.ts: a new hold cause, issue, worded "its own issue's run still going".
• src/crew/dispatch.ts (pump): when a queued run's issue already has a running run, the run is recorded as held for this reason. When the slots run out, the pass still records the remaining such runs before it stops, so they are recorded even when every slot is busy, as in the incident. Vitals then leaves them out of the queue's age and "since", and names them in its message.
• Hung sibling: a run stuck behind a hung sibling is still caught. The running run itself shows on the stuck: card once its heartbeat lapses or after 6 hours (NO_PROGRESS_MS). A comment in pump says so.
• test/vitals-held-agent.test.ts: the new test drives the real dispatcher with one slot. One run is running on an issue, a second run for the same issue is queued, and an older unrelated run sits ahead of it. The test asserts:

- the second run is recorded as held for the issue reason; - the vitals message reads "1 queued, 1 of 1 slots running … held on purpose: 1 by its own issue's run still going."

The test fails with dispatch.ts at the base version. It also fails with only the slot-limit block removed.

Not done: the 30-minute limit is unchanged. If distinct issues wait behind full slots, that is real saturation, and the alarm should still fire.

To check: run bun test test/vitals-held-agent.test.ts. In production, queue:agent should stay out of the degraded list for a day, or hold under 10 minutes. If it fires again with distinct issues queued, the cause is saturation, and crew.concurrency at /settings/capacity is the lever.

🤖 Generated with Claude Code

A queued agent run waiting behind a running run on its own issue is no longer counted as a stalled agent queue. The dispatcher starts one run per issue at a time, so that run is waiting for its sibling, not for a slot.

It is not confirmed that this caused #10192's 50 minutes. "2 queued, 2 of 2 slots running" fits this case, but it also fits a real backlog behind two long runs. P0 Trust is the pillar the issue already names.

What changed

• src/runs/held.ts: a new hold cause, issue, worded "its own issue's run still going".
• src/crew/dispatch.ts (pump): a queued run whose issue already has a running run is recorded as held for that cause. When the slots run out, the pass still marks the rest of the queue that waits on a running sibling before it stops. Without that, only siblings ahead of the first other run were marked. Vitals leaves held runs out of the queue's age and "since", and names them in the sentence.
• A sibling that hangs is still reported. The running run appears on the stuck: card once its heartbeat lapses or after 6 hours (NO_PROGRESS_MS). A comment in pump says so.
• test/vitals-held-agent.test.ts: the test drives the real dispatcher with one slot. One run is running on an issue, a sibling for the same issue is queued, and an older unrelated run sits ahead of the sibling. It asserts the sibling is held for the issue cause and the vitals sentence reads "1 queued, 1 of 1 slots running … held on purpose: 1 by its own issue's run still going."

- It fails with dispatch.ts at main, and also with only the slot-limit marking removed.

Fix for the failed check (test only). The test called turnValve(repoId, 1). Valve levels are a percentage of the week's starts, so on the check machine the valve most likely held both queued runs. The sibling was never marked issue, and the poll ran past bun's 5 s test timeout. I couldn't reproduce the failure here: the test passed every time locally, so this cause is inferred rather than observed.

The test now removes the repository's valve row, so only the issue holds the sibling. startCrew runs the dispatcher once before it returns, so the poll is a fallback, now capped at 3 s; a miss shows up as a failed assertion rather than a timeout.

Not done: the 30-minute limit is unchanged. Distinct issues waiting behind full slots are real saturation, and the alarm should still fire.

To try: bun test test/vitals-held-agent.test.ts passes here (6 tests), and bun x tsc --noEmit and bun scripts/gate.ts print no errors. In production, queue:agent should stay out of the degraded list for a day, or hold under 10 minutes. If it fires again with distinct issues queued, the cause is saturation, and the setting to raise is crew.concurrency at /settings/capacity.

🤖 Generated with Claude Code

queue:agent no longer counts an agent run that is waiting on its own issue's running run as a stalled queue. That rule (the dispatcher starts one run per issue at a time) is the likeliest explanation for the alarm in #10192, "2 queued, 2 of 2 slots running", but the evidence in the issue can't rule out a real backlog behind two long runs.

What changed

• src/runs/held.ts: a new hold cause, issue, worded "its own issue's run still going".
• pump in src/crew/dispatch.ts:

- It marks a queued run whose issue already has a run going as held with that cause. - When the slots run out, it still marks those runs before it stops.

• Vitals then leaves those runs out of the queue's age and "since", and names them in the sentence, for example "held on purpose: 1 by its own issue's run still going".
• test/vitals-held-agent.test.ts: a new test drives this through the real dispatcher. It sets up a running run, an unrelated queued run ahead (one slot, already taken), and a queued run on the same issue. It expects "1 queued, 1 of 1 slots running" and the sibling named as held. With the slot-limit marking in pump removed, the test fails.

Fix for the red check: the first version of the test polled for the hold on 20 ms dispatcher ticks, and on one check machine it failed at that poll. I couldn't reproduce that: the file passes locally, and in shard 19 of the same check. The test now reads the pass that startCrew runs before it returns and sets tickMs high enough that no timer runs during the test. Nothing else, like the autonomy sweep or a slow shard, can replace or delay the pass it checks. Only the test's timing changed; the assertions are the same. The test file passes five runs in a row, and bun scripts/gate.ts and tsc --noEmit are clean.

To look at: the slot-limit branch in pump (src/crew/dispatch.ts, around line 1592) and the last test in test/vitals-held-agent.test.ts.

🤖 Generated with Claude Code