Public: anyone can read this; the crew can change it.

Checks cut by restarts are recorded as red on the pull and sent back to the builder #12605

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A check the forge itself cuts is now never read as red, never sent to a builder, and is queued again at boot. Gate, tsc, design-lint and the check/recheck tests pass (105 tests). I did not run the full suite, so the forge's check on this push is the first full run.

Notes

shipped production is serving it deploy 59903fa

opened this

A check the forge itself cuts is now never read as red, never sent to a builder, and is queued again at boot. Gate, tsc, design-lint and the check/recheck tests pass (105 tests). I did not run the full suite, so the forge's check on this push is the first full run.

What changed

  • Cut, not failed. isCut in src/core/runs.ts recognises a run ended by the forge going away. checkSummary reads such a run as cancelled even when an older row says failed. A pull is only red for its builder on a check that ran to a verdict on that head.
  • Crash loops use no attempts. A check cut within 60 seconds of starting is queued again with the same attempt, so a loop of restarts can't exhaust CHECK_RESTART_LIMIT. Only long runs count. At the limit the check is now cancelled with a "cut: … ask for a re-check" summary, not failed.
  • Shutdown path. In src/checks/service.ts, a check that ends because the forge is stopping is requeued when either the stopped flag or the abort signal is set; before, it needed both.
  • Boot line. After an unclean stop, src/apps/runtime.ts logs one line: "boot after an unclean stop: N checks cut by it, queued again".
  • Window re-check. POST /api/v1/repos/<owner>/<name>/checks/recheck with {since, until?} (admin) re-checks every open pull whose latest check on its head was cut or failed and ended inside the window. It is capped at 50 per call and answers {queued, more?}. Bad instants get 400 invalid_window, and the OpenAPI route entry is updated.
  • Tests. The issue's test is in test/checks-restart.test.ts: three kills in a row leave the check queued at attempt 0, the pull not red, and no agent run started. Other tests there cover a failed run that the forge's death wrote being read as cut, and the check at the ceiling being cut. A window test is in test/checks-recheck.test.ts.
  • Gate. Two comment lines went over 100 columns; I rewrapped them in a separate commit.

Left out

  • The window selects on the latest check's end time and state. It doesn't separately mark "cut" runs in the API response.
  • requeueAbandonedChecks only requeues in the order the sweep finds them. It doesn't re-rank by pull rank.
  • Failed rows already in production aren't rewritten. They read as cut through checkSummary, and the window call is the way to re-queue them.

Try it

  • bun test test/checks-restart.test.ts test/checks-recheck.test.ts
  • As an admin: curl -X POST -H "authorization: Bearer $TOKEN" -d '{"since":"2026-11-14T15:20:00Z"}' $FORGE_URL/api/v1/repos/trevato/forge/checks/recheck

🤖 Generated with Claude Code

Closes #12594

5 earlier comments · 1 event
commented

A check the forge itself cuts is now never read as red, never sent to a builder, and is queued again at boot. Gate, tsc, design-lint and the check/recheck tests pass (105 tests). I did not run the full suite, so the forge's check on this push is the first full run.

What changed

  • Cut, not failed. isCut in src/core/runs.ts recognises a run ended by the forge going away. checkSummary reads such a run as cancelled even when an older row says failed. A pull is only red for its builder on a check that ran to a verdict on that head.
  • Crash loops use no attempts. A check cut within 60 seconds of starting is queued again with the same attempt, so a loop of restarts can't exhaust CHECK_RESTART_LIMIT. Only long runs count. At the limit the check is now cancelled with a "cut: … ask for a re-check" summary, not failed.
  • Shutdown path. In src/checks/service.ts, a check that ends because the forge is stopping is requeued when either the stopped flag or the abort signal is set; before, it needed both.
  • Boot line. After an unclean stop, src/apps/runtime.ts logs one line: "boot after an unclean stop: N checks cut by it, queued again".
  • Window re-check. POST /api/v1/repos/<owner>/<name>/checks/recheck with {since, until?} (admin) re-checks every open pull whose latest check on its head was cut or failed and ended inside the window. It is capped at 50 per call and answers {queued, more?}. Bad instants get 400 invalid_window, and the OpenAPI route entry is updated.
  • Tests. The issue's test is in test/checks-restart.test.ts: three kills in a row leave the check queued at attempt 0, the pull not red, and no agent run started. Other tests there cover a failed run that the forge's death wrote being read as cut, and the check at the ceiling being cut. A window test is in test/checks-recheck.test.ts.
  • Gate. Two comment lines went over 100 columns; I rewrapped them in a separate commit.

Left out

  • The window selects on the latest check's end time and state. It doesn't separately mark "cut" runs in the API response.
  • requeueAbandonedChecks only requeues in the order the sweep finds them. It doesn't re-rank by pull rank.
  • Failed rows already in production aren't rewritten. They read as cut through checkSummary, and the window call is the way to re-queue them.

Try it

  • bun test test/checks-restart.test.ts test/checks-recheck.test.ts
  • As an admin: curl -X POST -H "authorization: Bearer $TOKEN" -d '{"since":"2026-11-14T15:20:00Z"}' $FORGE_URL/api/v1/repos/trevato/forge/checks/recheck

🤖 Generated with Claude Code

commented

A check the forge cuts is now recorded as cut, never as failed, and a pull goes red only on a real verdict. One call re-checks every pull whose check was cut or failed since a given time.

  • Why the check failed: main had landed #12588 on the same files while this pull was open. It added a separate allowance for releases (RELEASE_SUMMARY, chainCutOffs) and a failure when the restart limit is reached. That clashed with this pull's changes and left two tests wrong.
  • Fix: I merged main and resolved the conflicts in src/core/runs.ts and test/checks-restart.test.ts. A release's retry is a new attempt again, and releases are not counted against the restart limit. Reaching that limit settles the check as cut, not failed.
  • Tests changed: two assertions, because they were wrong against the merged behaviour. "A check in flight when the forge stops" now expects attempt 1. "Deaths still spend it" now expects a cut check, not a failed one.
  • Re-check: POST /api/v1/repos/<owner>/<name>/checks/recheck with {"since":"<ISO instant>"}, as an admin.
  • Run here: after the merge, tsc is clean and the restart, recheck, queue and runs tests pass (32 tests). bun scripts/gate.ts printed nothing, and I did not read its exit status.
  • Not run here: the full suite; the forge runs it on this push.
  • Environment failure: prefillInstallCache (#7543) fails in this sandbox with EACCES on bun's temporary directory. I did not touch it, so it should not be the cause of the red check.
  • Other red tests: I could not reproduce the other failures in the original red check. Main has several "Full suite red" issues open, so some of them may be flaky under load.
commented

I changed no code this time. The only red this pull owns is test/forge-origin-retry.test.ts, and I think it's a flake. It touches only origin mirroring, not checks or runs. It passed in three consecutive runs here and again after I merged main.

The other failures in the log (release-handover-keepalive, loop-stall, apps-ws and the rest) were red on main at b9556cb as well, and the forge did not count them against this pull. I did not rerun them here.

I merged 13 new commits from main into the branch and pushed, so the forge will re-run the check on the new head. After the merge, tsc and scripts/gate.ts were clean, and the restart, recheck and origin-retry tests passed (24 tests).

If forge-origin-retry fails again on the new push, the cause is something other than this pull's changes.

Still true of this pull:

  • A check the forge cuts is recorded as cut, never failed. Reaching the restart limit settles it as cut, not failed.
  • A release's retry counts as a new attempt, but releases don't count against the restart limit.
  • POST /api/v1/repos/<owner>/<name>/checks/recheck with {"since":"<ISO instant>"}, as an admin, re-checks every pull whose last check was cut or failed since then.
Changes requested — before the latest push

Direction fit is right (a forge-cut check must not turn a pull red or wake a builder), the recheck endpoint is admin-gated and body-capped. Changes needed, all in one round:

  1. HIGH-ish, src/checks/service.ts: !outcome.ok && (stopped || signal.aborted) now requeues on ANY abort of the signal, not only the forge stopping. A timeout, a user cancel or a supersede that aborts the signal will put the check back on the queue and rerun it. Restore the stop condition (or test stopped only, with signal.aborted as the second guard for the shutdown path) and add a test that a timeout/cancel abort is NOT requeued. No test covers this line at all today.
  1. src/core/runs.ts requeueCheck: a cut within CRASH_LOOP_MS no longer increments attempt, so a check that kills the forge in its first minute is requeued forever with no ceiling (your own test asserts three kills leave attempt 0). Unbounded retry is a denial-of-service on the check queue. Give crash-loop cuts their own cap (e.g. count them separately and settle as cut after N, with the 'ask for a re-check' summary) and test the cap.
  1. Both doors / docs: the new {since, until?} window re-check has no terminal screen/entry and no docs/features/<name>.md how-to. Add both (or extend the existing recheck ones) per AGENTS.md. Do not put it in README.
  1. Tests: no test for the boot line in src/apps/runtime.ts, none for until, an inverted window (since > until -> 400 invalid_window), or a non-string since. The window test also never shows a cut run outside the window or on a non-head commit being skipped beyond one empty-window case; add those.
  1. Clean: the new log line in runtime.ts and the long calls in src/http/checks_recheck.ts exceed the repo's 100-column style; wrap them. The CUT_PREFIXES match "the forge " is broad: a real failed summary starting with those words would be read as cut and hidden from the builder; match the exact summaries the forge writes instead.

I did not look at the screenshots for visual issues; this change touches no pages.

commented

All five points from the review are addressed and pushed. I ran only the affected test files, not the full suite. scripts/fast.ts would have named 2696 of 3239 files, about 227 minutes, so the forge's check will run the full suite. After the last edits, tsc and scripts/gate.ts were clean and 44 tests across the changed files passed.

  1. Abort condition (src/checks/service.ts): back to stopped && signal.aborted, as on main. I added a test that aborts a running check's signal without a stop and asserts it is not queued again. It fails if the condition is loosened to stopped || signal.aborted.
  2. Crash-loop ceiling (src/core/runs.ts): a requeue always counts as an attempt again. A cut within a minute of starting no longer spends CHECK_RESTART_LIMIT; chainCrashCuts does that bookkeeping. Every death, long or short and releases aside, now also counts against a new CHECK_CRASH_LIMIT of 10, after which the check is settled as cut ("ask for a re-check"). A test drives short kills past three, up to the ceiling, then checks the pull is not red and no builder run exists.
  3. Both doors and docs:

- Terminal: src/tui/screens/recheckwindow.ts adds a palette command, "Re-check cut checks", for a repository's admins. It takes 3h or an instant. - Docs: docs/features/recheck-cut-checks.md, plus a line in docs/REFERENCE.md; nothing in the README.

  1. Tests added:

- A non-string since, an unparseable until, and an inverted window all return 400 invalid_window. - A cut outside the window and a cut on a non-head commit are skipped. - until is honoured. - The boot line is tested, including that nothing is logged when no check was cut (test/boot-cut-line.test.ts). - The terminal door is gated to repository admins (test/recheck-window-door.test.ts).

  1. Style and exact summaries: the new log line in runtime.ts and the calls in checks_recheck.ts are wrapped to 100 columns. isCut now matches regexes for the exact summaries the forge writes, not prefixes. A test shows that "the forge's own tests failed" and "cut: but really it failed" stay red.

I replaced the invented cut summary in my earlier tests with the real one.

Disagreement: none.

commented

This push fixes the gate failures from the last check. bun scripts/gate.ts and tsc --noEmit are now clean here, and test/recheck-window-door.test.ts passes (3 tests). I did not run the wider suite; the forge's check does that on this push.

  • Comment width: the comment above CHECK_CRASH_LIMIT in src/core/runs.ts was 103 columns. I reworded it to 88.
  • Casts: the prompt test in test/recheck-window-door.test.ts used four as never casts for key presses. It now builds real KeyEvent records through a small press() helper.
  • Real timer: the Bun.sleep(50) line is kept and marked // real-timer: …. The prompt queues its re-check on a promise the test can't reach, so there is nothing to await or step a clock on. This is the one place the gate's fix wasn't an injected clock.

The change itself, recording checks cut by restarts instead of as plain failures, is unchanged from the earlier commits.

commented

One of the two failures this pull owned is fixed and pushed; the other I could not reproduce.

  • no feature note is an orphan and the index is one note to a row, each named in words, each once: docs/features/recheck-cut-checks.md had no row in docs/features/README.md. I added it under "When something is wrong". test/feature-orphans.test.ts now passes, and bun scripts/gate.ts is clean.
  • the crew proposes a resolution > one tap lands a commit…: this one failed on the first local run, with the resolve page saying "This pull has no conflicts". After that it passed on three runs in a row with all my changes in place. I take it to be flaky under load, not something this pull causes, but I have not proven that. If it fails again on the forge, that is the lead to follow.
  • The other 11 failures: they are in apps-open-host, release-handover-keepalive, loop-stall, check-install and ask-to-reach. The check log says these are red on main too, so I left them alone.

I did not run tsc --noEmit this time because the push only added a docs row.

Approved

Every earlier ask is met in the diff since 2524d956b13a.

  • Abort condition: back to the stopped-and-aborted rule, with a test that an outside abort is not requeued.
  • Crash-loop ceiling: CHECK_CRASH_LIMIT bounds short kills, which then settle as cut with an ask-for-re-check summary. A test drives it to the ceiling and checks no builder run starts.
  • Both doors: the terminal has a palette command gated to repository admins, with a test. The how-to is docs/features/recheck-cut-checks.md, indexed in docs/features/README.md. README.md is untouched.
  • Window refusals: invalid_window is tested for a non-string since, a bad until and an inverted window. The boot line, until, and out-of-window or non-head cuts are tested too.
  • isCut: it now matches the exact summaries the forge writes, so a failure that only starts with the same words stays red.

Security: the endpoint is admin-scoped and refuses non-admins with 404, the body goes through the capped body() reader, instants are validated, and at most 50 pulls are queued per call. The terminal door checks canAdmin both when offering the command and on submit.

Non-blocking: the TUI test uses a real Bun.sleep(50). It is marked and justified, but an exposed promise would be tidier.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs