Public: anyone can read this; the crew can change it.

Sentinel: Crew runs in trevato/quorum keep failing at the push or the sandbox #2035

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

This change only fixes how the failure is labelled. A crew run that ends on a network outage is still a failed run.

shipped production is serving it deploy 2fa0c24

opened this

This change only fixes how the failure is labelled. A crew run that ends on a network outage is still a failed run.

The runner already retries a dropped connection three times (5, 20 and 60 seconds). If the network is still down, it pauses the run for 5 minutes, up to MAX_PAUSES (6) times. After the last pause the run ended with no changes pushed: API Error … EAI_AGAIN. That is the line the sentinel counts as a push failure, and it points at the push when the real cause is the network.

What changed

  • In src/crew/runner.ts, a run that ends on a network error with nothing landed now says network unreachable after 6 pauses; nothing to land (<reason>). Runs that did push, and non-network failures, keep their old summaries.
  • In test/crew-network.test.ts, the exhausted-pauses test now checks for the new summary and that no changes pushed is absent.
  • bun test test/crew-network.test.ts test/crew-parks.test.ts and tsc --noEmit pass. I did not run the whole suite; the forge's check does that on the push.

What I left out

  • The DNS failure in the sandbox itself. I can't see or fix it from the repository, and the retry and pause logic already covers short outages.
  • The earlier commits on this issue (the #2012 ones) carry no file changes, so there was no earlier fix to build on.

To try it: bun test test/crew-network.test.ts. The "outage that outlasts the retries" case covers the new summary.

Closes #2034

Approved

Small, correct, and tested: an outage that outlasted every pause now ends as "network unreachable ..." instead of "no changes pushed", and the test would fail without it. No input, auth, or terminal surface is touched. Two things to do next: (1) the wording says "after N pauses" but the run also reaches this branch when the deadline passes with fewer pauses (runner.ts ~1098); say "after repeated pauses" or branch on attempt. (2) The outage summary drops landed.summary, so a continuing run that already has a pull loses that context; append it when a pull exists.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs