Sentinel: Crew runs in trevato/relay keep failing at the push or the sandbox #3113

I changed the sentinel so a run that ended on a dropped connection no longer counts as a failed push. That should stop this finding refiling, but I haven't touched the sandbox or the runner, and I couldn't look at the five runs themselves. The runs API has no route for them. The sentinel tests pass. I did not run tsc here: bun couldn't write its tempdir in the sandbox.

Cause. The example summary, "no changes pushed: API Error: Can't reach the API server … (EAI_AGAIN)", is a network outage, not a push problem. The sentinel's push-or-sandbox filter in src/crew/sentinel.ts matches the word "pushed" in "no changes pushed", so these runs counted as failed pushes.

The runner already handles outages: it retries, then pauses and resumes (#1895, #1982, for trevato/quorum). Runs that exhaust the pauses are summarised "network unreachable after N pauses", which the filter doesn't match.

Change.

• isNetworkSummary in src/crew/network.ts reuses the runner's network pattern on a run summary.
• runStreaks in src/crew/sentinel.ts skips failed agent runs whose summary names a dropped connection when it looks for push or sandbox failures.
• test/sentinel.test.ts has a new case: five EAI_AGAIN runs raise no push: fault. The existing case where a real "push failed: operation not permitted" still raises the fault passes unchanged.

Left out. I did not change how the runner handles an outage. Earlier runs that ended with a plain "no changes pushed: …" summary stay failed in the history. They will drop out of the sentinel's window as newer runs replace them.

Try it. bun test test/sentinel.test.ts

Closes #3106

I changed the sentinel so a run that ended on a dropped connection no longer counts as a failed push. That should stop this finding refiling, but I haven't touched the sandbox or the runner, and I couldn't look at the five runs themselves. The runs API has no route for them. The sentinel tests pass. I did not run tsc here: bun couldn't write its tempdir in the sandbox.

Cause. The example summary, "no changes pushed: API Error: Can't reach the API server … (EAI_AGAIN)", is a network outage, not a push problem. The sentinel's push-or-sandbox filter in src/crew/sentinel.ts matches the word "pushed" in "no changes pushed", so these runs counted as failed pushes.

The runner already handles outages: it retries, then pauses and resumes (#1895, #1982, for trevato/quorum). Runs that exhaust the pauses are summarised "network unreachable after N pauses", which the filter doesn't match.

Change.

• isNetworkSummary in src/crew/network.ts reuses the runner's network pattern on a run summary.
• runStreaks in src/crew/sentinel.ts skips failed agent runs whose summary names a dropped connection when it looks for push or sandbox failures.
• test/sentinel.test.ts has a new case: five EAI_AGAIN runs raise no push: fault. The existing case where a real "push failed: operation not permitted" still raises the fault passes unchanged.

Left out. I did not change how the runner handles an outage. Earlier runs that ended with a plain "no changes pushed: …" summary stay failed in the history. They will drop out of the sentinel's window as newer runs replace them.

Try it. bun test test/sentinel.test.ts

I narrowed the exemption as you asked. The sentinel no longer counts a "no changes pushed: …" run that names the network as a failed push, but a bare "push failed: …" still raises push:<id>. test/sentinel.test.ts passes (36 tests). I did not run tsc, because bun can't write its tempdir in this sandbox.

• src/crew/network.ts: I replaced isNetworkSummary with isOutageSummary. It matches only a summary that starts with "no changes pushed:" and then names the network, which is the runner's own wording for an outage it retried. The doc comment now says that.
• src/crew/sentinel.ts: The push-fault filter calls isOutageSummary.
• test/sentinel.test.ts: The new test uses five "push failed: unable to connect to forge" and "push failed: ECONNREFUSED" runs. They still raise push:<id>. The existing EAI_AGAIN test still passes.

My first push of this round went up with two failures. The new test left its runs behind, which broke the later suites that count the day's faults. The test now deletes its runs when it finishes, and the second push fixed this.

I agree with all three comments and have nothing to dispute.

The sentinel no longer files a push-or-sandbox fault for runs that ended on a dropped connection. I don't know why the preview failed. I couldn't read its log, so the merge below is my guess at a fix, not a diagnosed one. The new preview run from this push will show whether it worked.

What changed

• isNetworkSummary in src/crew/network.ts reuses the runner's network-outage pattern on a run summary.
• runStreaks in src/crew/sentinel.ts skips failed agent runs whose summary names a dropped connection when it looks for push or sandbox failures.
• test/sentinel.test.ts has a new case: five EAI_AGAIN runs raise no push: fault. The existing case where a real "push" failure still raises the fault is unchanged.

Cause. Summaries like "no changes pushed: API Error: Can't reach the API server … (EAI_AGAIN)" are network outages. The old filter matched the word "pushed" in them and counted the runs as failed pushes.

Testing. I merged origin/main into the branch, since it had moved ahead, and pushed. After the merge tsc --noEmit is clean and test/sentinel.test.ts passes (36 tests). Before the merge, test/crew-network.test.ts also passed (9 tests). I did not run the full suite; the forge runs it on the push.

Not done

• I didn't touch the sandbox or the runner. The five runs behind the original finding can't be read through the runs API, so I didn't check them directly.

Try it. Run bun test test/sentinel.test.ts.