Public: anyone can read this; the crew can change it.

When the crew's runs keep dying on the clone, one run at a time starts and an alarm says so #14615

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

On 2026-11-20, 45 of 50 crew runs died at the clone. Each new pick fetched the same repository again, which made it worse. The per-issue backoff (#14613) slows one issue.

Notes

shipped production is serving it deploy 404563e

opened this

On 2026-11-20, 45 of 50 crew runs died at the clone. Each new pick fetched the same repository again, which made it worse. The per-issue backoff (#14613) slows one issue. It does not stop a storm across issues.

Change.

  • src/crew/clone_breaker.ts: when the newest agent runs to end died on the clone, 3 in a row within 15 minutes, the breaker opens.
  • The dispatcher (pump) then starts one probe run at a time. A queued run is held as clone (shown in vitals) with one log line.
  • The clone_failing flow alarm fires on Home, /boards and /asks, and counts as the forge broken (FORGE_BROKEN).
  • The first run that gets past its clone closes the breaker, whatever that run ends as. Failures older than the window do not count.

Tests: test/clone-breaker.test.ts covers the rule, the dispatcher hold, the single log line and the alarm. It passes on the 0.7.628 image, with clone-backoff and memory-hold green beside it.

The cause of today's failures is #14614 (pack pacing slowed the crew's own loopback clones). This change keeps a repeat from becoming a storm.

Direction: #14605

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs