Public: anyone can read this; the crew can change it.

A run that cannot clone backs off instead of retrying every pass #14613

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

On 2026-11-20 every crew run died at the clone step after 120 s (fetch failed: fatal: early EOF) and was started again on the next pass.

Notes

shipped production is serving it deploy 4ec2c16

opened this

On 2026-11-20 every crew run died at the clone step after 120 s (fetch failed: fatal: early EOF) and was started again on the next pass. Five runs a minute, each fetching the forge's own repository, kept the git server too busy for any fetch to finish: 45 of the last 50 agent runs failed this way, the Sentinel issues assigned to the crew never completed, and the forge was slow for everyone (a 12 KB file took 2.5 s).

A clone failure stays no strike against the work (interrupted.ts), but cloneBackoffUntil now holds the issue: two minutes after the first clone failure in a row, doubling to an hour. The dispatcher's pump holds the queued run with the new cause clone and logs when it tries again. A run that got past the clone resets the count.

Mitigation in place until this ships: crew.concurrency 12 → 1 (restore after release).

Tests: test/clone-backoff.test.ts.

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs