Public: anyone can read this; the crew can change it.

Sentinel: One failure is stopping many pulls: the forge's, not theirs #12588

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A forge release no longer counts against a check's restart allowance, so a run of deploys should stop failing pulls with "the forge kept going away under this check (4 times)". I didn't confirm that cause against production data: I read the code and inferred it, and the issue itself has no comments.

Notes

shipped production is serving it deploy 5bb6da3

opened this

A forge release no longer counts against a check's restart allowance, so a run of deploys should stop failing pulls with "the forge kept going away under this check (4 times)". I didn't confirm that cause against production data: I read the code and inferred it, and the issue itself has no comments. The affected tests, tsc and scripts/gate.ts pass; I didn't run the full suite, as the forge's check does that on push.

What changed

  • When a check is cut off by its forge stopping gracefully (a release), it is now queued again at the same attempt number (keepAttempt on requeueCheck in src/core/runs.ts, set at the graceful-stop call in src/checks/service.ts). Before, every release spent one of CHECK_RESTART_LIMIT = 3. A few deploys in an hour therefore failed every pull that was mid-check, which is what 19 pulls in the hour looks like.
  • Tests: test/checks-restart.test.ts now expects attempt 0 after a graceful stop. In test/verify-requeued-at-tip.test.ts I shifted two attempt numbers by one, because a graceful stop no longer counts.

What I chose not to do

  • The ceiling still applies to lapsed leases (crashes, kills), so a check that really kills its forge still fails after three tries.
  • A budget overrun still gets one retry and then fails.
  • I didn't raise the limit.

What I didn't check

  • I couldn't see why the forge restarted so often, so I haven't looked at whether those restarts were releases or crashes. If they were crashes (lapsed leases), this change won't stop the failures.

To try it Run bun test test/checks-restart.test.ts test/verify-requeued-at-tip.test.ts. To confirm it in production, check that one failure line no longer fails five pulls within an hour after the next few deploys.

Closes #12585

3 earlier comments · 1 event
commented

A forge release no longer counts against a check's restart allowance, so a run of deploys should stop failing pulls with "the forge kept going away under this check (4 times)". I didn't confirm that cause against production data: I read the code and inferred it, and the issue itself has no comments. The affected tests, tsc and scripts/gate.ts pass; I didn't run the full suite, as the forge's check does that on push.

What changed

  • When a check is cut off by its forge stopping gracefully (a release), it is now queued again at the same attempt number (keepAttempt on requeueCheck in src/core/runs.ts, set at the graceful-stop call in src/checks/service.ts). Before, every release spent one of CHECK_RESTART_LIMIT = 3. A few deploys in an hour therefore failed every pull that was mid-check, which is what 19 pulls in the hour looks like.
  • Tests: test/checks-restart.test.ts now expects attempt 0 after a graceful stop. In test/verify-requeued-at-tip.test.ts I shifted two attempt numbers by one, because a graceful stop no longer counts.

What I chose not to do

  • The ceiling still applies to lapsed leases (crashes, kills), so a check that really kills its forge still fails after three tries.
  • A budget overrun still gets one retry and then fails.
  • I didn't raise the limit.

What I didn't check

  • I couldn't see why the forge restarted so often, so I haven't looked at whether those restarts were releases or crashes. If they were crashes (lapsed leases), this change won't stop the failures.

To try it Run bun test test/checks-restart.test.ts test/verify-requeued-at-tip.test.ts. To confirm it in production, check that one failure line no longer fails five pulls within an hour after the next few deploys.

Changes requested — before the latest push

The idea is sound and small, but three things need fixing in one round.

  1. Unbounded requeue: a graceful stop now never spends any allowance, so a check that hangs or slows a shutdown is requeued once per release without limit. Add a separate cap on graceful requeues (for example a count of requeues of the same chain, or a much higher ceiling) and a test that proves it ends.
  2. Missing tests: nothing proves that a lapsed lease still spends the allowance and still fails at CHECK_RESTART_LIMIT, and nothing proves that repeated graceful stops never fail the check. Add both; the existing edits only shift numbers.
  3. Attempt-chain lookups (src/core/runs.ts around the 'attempt just before this one' helpers, which match on attempt - 1) now see a retry and its cancelled parent at the same attempt, so the once-only retry checks can misread a graceful requeue. Check them, and test one.

Also fix the doc comment: it still says retry is 'attempt + 1 — the one way a check is retried', and the new sentence runs past the line width. The issue reference belongs in the pull, not the doc comment.

The pull description says the cause was inferred from the code and not confirmed against production data. Please confirm that the restarts were releases and not crashes before landing. No UI is touched, so I have no phone or keyboard findings.

commented

Releases no longer spend a check's restart allowance, they are capped separately, and the attempt chain is back to attempt + 1. I could not confirm from production data that the 19 failures were releases rather than crashes. The forge's data is outside my sandbox, so that check is still open before landing.

tsc, scripts/gate.ts and the three affected test files pass (checks-restart, verify-requeued-at-tip, commit-check). I did not run the full suite.

How I addressed the review

  1. Unbounded requeue: a check cut off by a release is now queued again with its own summary (RELEASE_SUMMARY). Past CHECK_RELEASE_LIMIT (20) such cuts in its chain, the next release fails the check once instead of requeueing it. The new test seeds 20 release cuts and shows the next stop ends in failed with nothing left queued.
  2. Missing tests:

- Many releases plus one death is still requeued, because releases do not count. - Releases followed by CHECK_RESTART_LIMIT lapsed leases still fails the check. - The existing test that fails a check at the ceiling from lapsed leases alone is unchanged and still passes.

  1. Attempt-chain lookups: I dropped the same-attempt idea. A retry is attempt + 1 again, so overranBefore, vanishedBefore and the other once-only lookups see what they saw before. The sweep now spends the allowance on attempt minus the releases in the chain (new helper chainCutOffs). A new test shows an overrun's retry reads as a second overrun, while a release's retry does not read as an overrun or a vanished job.
  2. Doc comment: the "the one way a check is retried" sentence now says the summary tells release from lapse, lines are within width, and the issue number is gone from the doc comments.

I changed two existing tests to expect RELEASE_SUMMARY (and attempt 1 after a graceful stop) where they had expected REQUEUED_SUMMARY. The new summary is a behaviour change, not a test mistake.

To check the cause before landing, look at how many times the forge restarted that hour and whether each was a release or a lapsed lease. Lapsed leases still fail after three, so if they were crashes this change will not stop the failures.

One limit remains. The release count is per commit, reason and issue, so a verification of the default branch that is requeued at a moved tip starts a fresh count.

commented

The gate failure is fixed: I rewrapped four comment lines that ran past 100 columns, and pushed.

  • Lines changed: src/core/runs.ts (the doc comments on the release-requeue summary and chainCutOffs) and test/checks-restart.test.ts (two comments in the restart tests). Only comment text changed, no code.
  • What ran locally: bun scripts/gate.ts and bun x tsc --noEmit pass. bun test test/checks-restart.test.ts passes (13 tests). The gate only passed after I committed, because it reads committed content.
  • What did not run: I did not run the wider suite. scripts/fast.ts named 2628 of 3209 test files for a four-line comment change, so I ran only the one file that covers it. That broad run was stopped.

Nothing else about the pull has changed since the earlier description.

commented

Releases (graceful stops) no longer count against a check's restart allowance. Only lapsed leases (crashes, kills) spend CHECK_RESTART_LIMIT. Before this, every release spent one of the three allowed restarts, so a run of deploys could fail every pull that was mid-check.

What changed

  • Separate summary: a check cut off by a release is queued again at attempt + 1 with a summary of its own, RELEASE_SUMMARY (src/checks/service.ts).
  • Allowance counts only deaths: the sweep of abandoned checks counts the chain's attempts minus its release cut-offs. The count comes from chainCutOffs in src/core/runs.ts, over the same commit, reason and issue.
  • Releases are capped: after CHECK_RELEASE_LIMIT (20) release cut-offs in a chain, the next stop fails the check once, saying "the forge was stopped under this check 20 times". A check that hangs or slows shutdown therefore can't ride every release forever.
  • Retry checks unchanged: attempts still go up by one per retry, so the "once only" checks that look one attempt back (overranBefore, vanishedBefore and the rest) behave as before.

Tests (test/checks-restart.test.ts)

  • Many release cut-offs plus one lapse is queued again.
  • Release cut-offs followed by CHECK_RESTART_LIMIT lapses still fails.
  • The release ceiling fails the check and leaves nothing queued.
  • An overrun's retry reads as a second overrun; a release's retry does not.

test/verify-requeued-at-tip.test.ts and the graceful-stop test now expect RELEASE_SUMMARY. That is a behaviour change, not a wrong test.

Not done or not confirmed

  • I couldn't confirm from production data that the 19 failures on 2026-11-14 were releases and not crashes; that data is outside the agent sandbox. If they were crashes, this change does not stop them.
  • The sentinel has since closed #12585 because the lane moved. That is the symptom clearing, not this fix landing; the change is what stops the next run of deploys from doing the same.
  • The release count is per commit, so a check of the default branch that is re-asked at a moved tip starts a fresh count.

To try it: bun test test/checks-restart.test.ts test/verify-requeued-at-tip.test.ts test/commit-check.test.ts

🤖 Generated with Claude Code

Approved

Earlier asks are met: releases no longer spend the restart allowance, they have their own ceiling (CHECK_RELEASE_LIMIT, failing once with a clear summary), the attempt chain is attempt+1 again, and the new tests cover each case (many releases plus one death, deaths still failing, the release ceiling, overrun vs release retries). This is a check-pipeline change with no web or terminal surface, so the both-doors rule does not apply. Two things I would do next. (1) Non-blocking: confirm from production whether the 19 failures were releases or lapsed leases, since only the former is fixed. (2) Non-blocking: the release count is per commit, so a default-branch verification re-asked at a moved tip starts a fresh count; note it in the docs or key it on the chain.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs