Public: anyone can read this; the crew can change it.

A check the forge's own death kills is failed and never run again, and the change goes red for it #512

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

If the forge dies or restarts while a check is running, the check is now queued again instead of being failed. Before this, an 8-minute check that outlived a forge restart was marked red with "the forge that was running this went away", and only another push could clear it.

shipped production is serving it deploy 749c831

opened this

If the forge dies or restarts while a check is running, the check is now queued again instead of being failed. Before this, an 8-minute check that outlived a forge restart was marked red with "the forge that was running this went away", and only another push could clear it.

What moved

  • src/core/runs.ts: requeueCheck is now the only way a check gets retried. It cancels the run with "interrupted by a restart; queued again" and queues the same check with attempt + 1. The cancel only happens if the run is still running, and in the sweep also only if its lease has lapsed. So when two forges sweep at once, only one of them queues the retry.
  • Also in src/core/runs.ts: a new sweep, sweepAbandonedChecks, sits beside failAbandonedRuns and handles running checks whose lease has lapsed:

- Commit is still the head of an open, unmerged pull: cancelled and queued again. - Already retried CHECK_RESTART_LIMIT (3) times: failed once, with "the forge kept going away under this check (N times); not trying again". - Commit no longer heads any open pull: cancelled with "the forge went away, and this commit no longer heads an open pull", and not queued again. A closed pull or an old head is never brought back.

  • src/checks/service.ts: the graceful-stop path now calls requeueCheck instead of its own cancel-and-create. The new requeueAbandonedChecks(app, staleMs) runs the sweep, writes the run logs and announces the retry and the one honest failure.
  • src/main.ts (the 30s timer) and src/apps/runtime.ts (boot) now pass failAbandonedRuns only deploy (plus agent when there's no crew), and then call requeueAbandonedChecks. Deploys behave exactly as they did after #286.

What to look at

  • A cancelled check is deliberately not announced as run.finished. crew/dispatch.ts reads any finished check that isn't ok as "red", so announcing it would tell the decider the change failed. A superseded check isn't announced either, for the same reason.
  • Between the death and the retry, pullWord keeps reading working: "checking now", then "queued to check". The wording logic in web/studio/words.ts is unchanged.

Chosen not to do

  • Checks of the default branch have no pull, so the sweep cancels them rather than requeueing them. Requeueing them would mean reading the branch head from git inside a data-only module.
  • failAbandonedRuns still fails a check when a caller passes it check. test/runs-lease.test.ts relies on that and is unchanged.

Tests New cases in test/checks.test.ts cover:

  • a live head becomes a queued check with attempt + 1, a second sweep moves nothing, and the change reads working throughout;
  • a check whose lease is still held is left alone;
  • at the ceiling the check fails once with the new summary;
  • a check for a stale head is cancelled and not requeued.

These cases fail on main, where requeueAbandonedChecks doesn't exist. The typecheck and the full forge.json check pass: 599 tests, 0 failures.

🤖 Generated with Claude Code

Closes #303

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs