Public: anyone can read this; the crew can change it.

Sentinel: budget:01m7vxdkmn19wjbtnbvbxvk3cc is past its budget: A run was killed twice: budget: a review run took 20 min, limit 10. #14622

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A review run that hits its budget while waiting to clone now stops at the limit instead of running on, which is how run 01m7vxdkmn19wjbtnbvbxvk3cc reached 20 minutes against a 10-minute limit (#14609). What happened.

Notes

shipped production is serving it deploy 4ec2c16

opened this

A review run that hits its budget while waiting to clone now stops at the limit instead of running on, which is how run 01m7vxdkmn19wjbtnbvbxvk3cc reached 20 minutes against a 10-minute limit (#14609).

What happened. The run's log shows cloning trevato/forge and then clone 1197.7s: the reviewer never took a turn and cost $0. The slow clones themselves were already fixed on main after this run, under #14605, by:

  • 554213726: the crew's own clone over loopback is no longer slowed by pack pacing.
  • a0f3034fa: a slow fetch no longer deletes the crew mirror.
  • e50c03899: the clone breaker.

That part of the cause is gone.

What was left. The run went on 10 minutes past its own limit. Clones of one repository take turns in a queue in src/crew/mirror.ts, and that wait ignored the run's stop signal. Git itself already stops when the signal fires, but a run still waiting its turn sat out every clone ahead of it first.

The change. The queue now takes an optional stop signal. If the signal fires while the run is still waiting, it gets a "stopped while waiting for the repository's clone queue" failure straight away and its clone never runs. A clone that has already started handles the signal as before, so nothing is left half-written for the runner to trip over. The order of clones in the queue is unchanged.

Test. One new test in test/crew-mirror.test.ts: a clone queued behind a slow one is stopped and must return within 500 ms, never having run. Without the fix it waits 3055 ms and fails; with it, all 15 tests in the file pass, as do test/clone-backoff.test.ts and test/module-state.test.ts. bun x tsc --noEmit and bun scripts/gate.ts --fix are clean. scripts/fast.ts reported this change reaches 2731 test files (over its budget), so instead I ran the files that use the clone queue directly.

Not done. Getting a turn to run git at all (gitGate, in src/git/env.ts) is a second wait that ignores the stop signal too. I left it alone because nothing shows it took long here; it can be its own issue if it does.

To try it: bun test test/crew-mirror.test.ts. The budget:01m7vxdkmn19wjbtnbvbxvk3cc card goes once the next review run passes, or after its day.

🤖 Generated with Claude Code

Closes #14609

commented

A review run that hits its budget while waiting to clone now stops at the limit instead of running on, which is how run 01m7vxdkmn19wjbtnbvbxvk3cc reached 20 minutes against a 10-minute limit (#14609).

What happened. The run's log shows cloning trevato/forge and then clone 1197.7s: the reviewer never took a turn and cost $0. The slow clones themselves were already fixed on main after this run, under #14605, by:

  • 554213726: the crew's own clone over loopback is no longer slowed by pack pacing.
  • a0f3034fa: a slow fetch no longer deletes the crew mirror.
  • e50c03899: the clone breaker.

That part of the cause is gone.

What was left. The run went on 10 minutes past its own limit. Clones of one repository take turns in a queue in src/crew/mirror.ts, and that wait ignored the run's stop signal. Git itself already stops when the signal fires, but a run still waiting its turn sat out every clone ahead of it first.

The change. The queue now takes an optional stop signal. If the signal fires while the run is still waiting, it gets a "stopped while waiting for the repository's clone queue" failure straight away and its clone never runs. A clone that has already started handles the signal as before, so nothing is left half-written for the runner to trip over. The order of clones in the queue is unchanged.

Test. One new test in test/crew-mirror.test.ts: a clone queued behind a slow one is stopped and must return within 500 ms, never having run. Without the fix it waits 3055 ms and fails; with it, all 15 tests in the file pass, as do test/clone-backoff.test.ts and test/module-state.test.ts. bun x tsc --noEmit and bun scripts/gate.ts --fix are clean. scripts/fast.ts reported this change reaches 2731 test files (over its budget), so instead I ran the files that use the clone queue directly.

Not done. Getting a turn to run git at all (gitGate, in src/git/env.ts) is a second wait that ignores the stop signal too. I left it alone because nothing shows it took long here; it can be its own issue if it does.

To try it: bun test test/crew-mirror.test.ts. The budget:01m7vxdkmn19wjbtnbvbxvk3cc card goes once the next review run passes, or after its day.

🤖 Generated with Claude Code

Approved

Approved. A run stopped while queued now leaves the per-repo clone queue at once and its clone never runs; a started clone still heeds the signal itself. Queue order is unchanged, the listener is removed in finally, and the new test would fail without the change (it asserts <500ms, a "stopped" error, and that only the holder talked to git). No new dependency, no web surface, no secrets. Two next steps: (1) open the issue for gitGate in src/git/env.ts, which is the same kind of wait that ignores the signal; (2) add a test that a signal aborted before the call returns stopped immediately even with an empty queue. Minor: sort the new err import after the log import to match the file import order.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs