Public: anyone can read this; the crew can change it.

The engine went idle for an hour with 17 red pulls: builders sent back to a red are shown a healthy preview log, find nothing, and the pull stays red for good #12969

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A builder who changes nothing on a red pull now gets that head checked again, and a hand-back never quotes a deploy's success as a failure. The last ask, re-queueing checks after the limit resets, is not done.

Notes

shipped production is serving it deploy 1b35e66

opened this

A builder who changes nothing on a red pull now gets that head checked again, and a hand-back never quotes a deploy's success as a failure. The last ask, re-queueing checks after the limit resets, is not done.

I ran bun x tsc --noEmit and the new test plus the related suites (stale reds, brief, autonomy, crew loop and parks): 99 and 89 passing, none failing. bun scripts/gate.ts printed nothing, so I'm treating it as clean. I did not run the full suite, so the forge's check on this push is the first full run.

What changed

  • Wrong evidence. failedRunOf (src/core/runs.ts) picks the newest check or deploy of the head that actually failed and was not cut by the forge. The runner uses it for check_failed hand-backs.

- The brief now quotes the failing test, its assertion and its output (src/crew/failure_quote.ts). - When no check of the head is recorded as failed, the brief says so and tells the builder to change nothing.

  • "Nothing to fix" dead end. src/crew/no_change.ts, called from settleAgentRun, handles a builder run that ends with no push on a pull whose check is red or cut.

- First time, it queues a re-check of that head with reason no_change. If that comes back red on a named test, the existing path sends the builder that test's failing output. If it comes back green, nothing more is queued. - A second "no change" on a head whose re-check was red posts one comment quoting the failure, and the pull stays open for a person. It is never closed for a count of tries. - A re-check the forge cut short is queued again.

  • Idle with red pulls. idleReason now says "stalled: N open pulls are red with nobody working them" instead of "nothing ready". It only applies when nothing else could start. A pull a person holds is not counted.
  • Test. test/no_change_recheck.test.ts covers a cut check followed by "no change" (re-check queued), a second "no change" quoting the failure, a green re-check moving on, the quoted assertion, and the stall count.

What I chose not to do

  • Re-queue after the limit resets or an unclean stop. Restart cuts already count as cut, not red, and are re-queued. A check that failed inside the limit window with an ordinary-looking summary can't be told from a real red, and the cases I could identify need a separate change in the checks service.
  • Idle line wiring. I changed the idle reason string but not the did=idle field the engine prints, and I haven't checked that the dashboard shows the new line. I haven't checked either that production now goes a tick without a pull sitting red with did=idle.

I added no Try: commit line, which CLAUDE.md says a pull touching src/actions, src/web and the like needs. This change is in src/core and src/crew, which that rule doesn't list.

How to try it

  • Run bun test test/no_change_recheck.test.ts.
  • In production, a builder return of "no change" on a red pull shows a no_change check in that pull's runs.

🤖 Generated with Claude Code

Closes #12968

4 earlier comments · 2 events
commented

A builder who changes nothing on a red pull now gets that head checked again, and a hand-back never quotes a deploy's success as a failure. The last ask, re-queueing checks after the limit resets, is not done.

I ran bun x tsc --noEmit and the new test plus the related suites (stale reds, brief, autonomy, crew loop and parks): 99 and 89 passing, none failing. bun scripts/gate.ts printed nothing, so I'm treating it as clean. I did not run the full suite, so the forge's check on this push is the first full run.

What changed

  • Wrong evidence. failedRunOf (src/core/runs.ts) picks the newest check or deploy of the head that actually failed and was not cut by the forge. The runner uses it for check_failed hand-backs.

- The brief now quotes the failing test, its assertion and its output (src/crew/failure_quote.ts). - When no check of the head is recorded as failed, the brief says so and tells the builder to change nothing.

  • "Nothing to fix" dead end. src/crew/no_change.ts, called from settleAgentRun, handles a builder run that ends with no push on a pull whose check is red or cut.

- First time, it queues a re-check of that head with reason no_change. If that comes back red on a named test, the existing path sends the builder that test's failing output. If it comes back green, nothing more is queued. - A second "no change" on a head whose re-check was red posts one comment quoting the failure, and the pull stays open for a person. It is never closed for a count of tries. - A re-check the forge cut short is queued again.

  • Idle with red pulls. idleReason now says "stalled: N open pulls are red with nobody working them" instead of "nothing ready". It only applies when nothing else could start. A pull a person holds is not counted.
  • Test. test/no_change_recheck.test.ts covers a cut check followed by "no change" (re-check queued), a second "no change" quoting the failure, a green re-check moving on, the quoted assertion, and the stall count.

What I chose not to do

  • Re-queue after the limit resets or an unclean stop. Restart cuts already count as cut, not red, and are re-queued. A check that failed inside the limit window with an ordinary-looking summary can't be told from a real red, and the cases I could identify need a separate change in the checks service.
  • Idle line wiring. I changed the idle reason string but not the did=idle field the engine prints, and I haven't checked that the dashboard shows the new line. I haven't checked either that production now goes a tick without a pull sitting red with did=idle.

I added no Try: commit line, which CLAUDE.md says a pull touching src/actions, src/web and the like needs. This change is in src/core and src/crew, which that rule doesn't list.

How to try it

  • Run bun test test/no_change_recheck.test.ts.
  • In production, a builder return of "no change" on a red pull shows a no_change check in that pull's runs.

🤖 Generated with Claude Code

commented

The gate failed on the as never casts in test/no_change_recheck.test.ts, and I replaced them with as unknown as TaskInput, the form the other brief tests use. bun scripts/gate.ts now prints nothing against the committed head, and bun x tsc --noEmit and the 3 tests in that file pass. I have not run the forge's own check, so I don't know yet whether it goes green on this push.

The rest of the pull is as I described it before:

  • Hand-backs. A hand-back to a builder quotes the failing test, its assertion and its output from a check that actually failed, never a deploy's success summary.
  • "No change" returns. A builder's "no change" on a red pull queues a re-check of that head. A second one on a red re-check quotes the failure in a comment and leaves the pull open for a person.
  • Idle reading. The idle reason now reads "stalled" when a pull is red and nobody is working it.

Still not done: re-queueing every check and run that ended inside the provider's limit window.

🤖 Generated with Claude Code

commented

This push covers asks 1 and 2 of your correction and half of ask 3. bun x tsc --noEmit, bun scripts/gate.ts and the targeted tests pass (23 tests across the formatter, review rules and re-check suites); the forge's full check has not run on this push yet.

What changed

  • Hand-backs quote the check's own lines. A builder sent back to a red check now gets the check's gate: rule …, (fail) …, failed: … and error: lines plus the failing test's assertion (src/crew/failure_quote.ts, read from the failed check's log). The second "no change" comment quotes the same lines. I could not reproduce from the code why the deploy log was being attached. What I changed is the selection: only a check or deploy that itself failed is quoted, so a deploy that came up never is. When nothing of the head failed, the brief tells the builder to change nothing and wait for the re-check.
  • Comment width is no longer a gate. The "wrapped at 100 columns" rule is gone from src/kernel/review_rules.ts. src/kernel/rewrap.ts is the formatter. bun scripts/rewrap.ts [files] rewrites long // and block-comment prose lines in src/ and test/ files changed against origin/main. It leaves code and lines with URLs alone. test/rewrap.test.ts holds that it is idempotent.
  • The looks rule names candidates. When a pull changes a page with no Looks: line, the gate failure lists the app.looks pages that best match the touched files, as ready-to-paste Looks: <path> lines.
  • "No change" and idle. The re-check after "no change" and the "stalled" idle reading are unchanged from my earlier push.

What I chose not to do

  • Formatter wiring. Nothing in the check or the builder's push step runs the rewrap script yet. Long comments just no longer fail anything, and nothing wraps them for you until the script is called.
  • Looks taken by the forge. The forge does not take the looks itself or add the Looks: line; the failure only offers candidates.
  • Re-queue after a limit reset. Re-queueing checks that ended inside a provider-limit window is still not done.

To try it: run bun test test/rewrap.test.ts test/no_change_recheck.test.ts test/review-rules.test.ts.

🤖 Generated with Claude Code

Changes requested — before the latest push

The core fix is sound: failedRunOf never quotes a deploy success or a cut check, and the no_change re-check is tested. Changes needed, all in one round:

  1. FIRST: the issue is that the engine reads idle beside red pulls. You changed the idle reason string but did not wire the did=idle field the engine prints, and did not verify the dashboard shows it. As shipped the stated bug is not shown fixed in the preview. Wire it and show it.
  2. Scope: the rewrap formatter, scripts/rewrap.ts, the removal of the 100-column rule, and the looks candidates are unrelated to this issue. Direction says small single-purpose pulls. Split them out. At minimum, do not delete a gate rule while the formatter that replaces it is wired to nothing (you said so yourself): scripts/rewrap.ts is dead code until something calls it.
  3. test/rewrap.test.ts has mid-file imports and holds tests for failure_quote.ts and look_rule.ts, which do not belong in a rewrap test. Move them to tests named for their module (or drop with the split).
  4. settleAgentRun: afterNoChange has side effects (queues a check, posts a comment) inside a boolean-returning chain. Make the call explicit, and say why the comment is posted as the autonomy owner rather than as the crew.
  5. Add a test that idleReason itself returns the stalled word; today only redUnattended is tested.
commented

All five points are addressed and pushed. The gate and tsc --noEmit are clean. I ran the tests for what I touched (36 pass across 6 files) and did not run the full suite. I did not check the dashboard on the preview.

What changed

  • did= field (1):

- driveRepo now returns a new "stalled" value instead of "idle" when no builder can start anything and some open crew pull is red with nobody working it. - The tick log line prints that value as did, so it no longer reads idle beside a red pull. - The idle line on the dashboard already used idleReason's stalled word. I did not open the dashboard to confirm it. - I moved redUnattended into src/crew/red_unattended.ts. The tick and idleReason both use it, and importing idle_reason.ts from autonomy.ts would have been circular. idle_reason.ts still re-exports it.

  • Scope (2, 3):

- Removed scripts/rewrap.ts, src/kernel/rewrap.ts and test/rewrap.test.ts. - Restored the 100-column rule in review_rules.ts and its test. - Restored look_rule.ts without the candidate pages. - I'm not opening the formatter or the looks candidates as separate pulls from here. They have to be re-proposed on their own if wanted. - The failingLines test moved to test/failure-quote.test.ts, since failing-line quoting is part of this fix.

  • settleAgentRun (4): the afterNoChange call is now its own step with a comment saying it queues a re-check or posts a comment. The comment is posted as the autonomy owner for two reasons. The loop acts as that person everywhere else (assigning, parking). A builder's account should also not answer its own hand-back. A comment in no_change.ts says so.
  • idleReason test (5): test/no_change_recheck.test.ts now has a test with a red pull and no active runs. It asserts that idleReason returns the stalled: 1 open pull is red… word and that driveRepo returns "stalled".

To try it: bun test test/no_change_recheck.test.ts test/failure-quote.test.ts test/idle-reason.test.ts.

Changes requested — before the latest push

The core idea is sound: failedRunOf never quotes a deploy success or a cut check, and the stall is named. I read the diff only; I did not run the preview. Changes needed, all in one round:

  1. FIRST: afterNoChange posts a new comment on every no-change settle once a head has a red non-cut re-check. Nothing records that the failure was already quoted. The pull stays red, unheld and unattended, so the loop sends the builder back again and each no-change run adds another identical comment. The docstring promises quoting once, and the code does not do that. Fix: check whether the quote comment already exists for this head, or record it (a hold or a marker), and return undefined when it does. Add a test that a third no-change posts no second comment.
  2. afterNoChange runs for every non-feedback run end on a red pull, including runs that errored or were interrupted without saying anything. Those get a re-check queued as if the builder had answered no change. Gate it on run.state === ok, or on a summary that actually says no change. Test the errored case.
  3. The runner wiring is untested: failedRunOf feeding checkFailing and noFailure into the brief is only exercised by calling taskPrompt directly. Add a test through the runner path, or at least one that asserts the brief for a head whose only run is a green deploy.
  4. Name the new Drove value where other code branches on it, and confirm nothing treats an unknown value as an error. Add a test for that consumer if there is one.

No docs/features page is needed, since this is engine behaviour with no new door.

commented

Builders sent back to a red pull no longer get a healthy preview's log as "the failure". A builder who finds nothing to change gets the head checked again. If that re-check is red too, the failure is quoted on the pull once. And red pulls with nobody working them now read as stalled, not idle. This round fixes the four points from review.

1. The failure is quoted once per head (src/crew/no_change.ts). Nothing recorded the quote before, so every later no-change on a red pull posted the same comment again. Now the comment opens with the head's short id, and alreadyQuoted looks for a comment from the autonomy owner on this pull that starts that way. If one exists, afterNoChange returns undefined and normal settling carries on. The comment itself is the record, so there is no new table or hold. I chose not to use an "until green" hold: it would hide the pull from the stall count and change what the review loop does with it. The test runs two more no-change settles after the quote and asserts no new comment and no new re-check.

2. Only a run that ended ok counts as "no change" (settleAgentRun in src/crew/autonomy.ts). Before, a builder run that errored or was cut off still queued a re-check. A new test settles a failed builder run on a red pull and asserts nothing is queued.

3. The runner's wiring is tested through the real path. The five failure fields of the brief (checkSummary, checkFailing, noFailure, checkKind, checkLog) moved out of runner.ts into one function, failureBrief in src/crew/failure_brief.ts. The runner calls it, and it decides what a builder is told. The test feeds failureBrief's output for a head whose only run is a green deploy into taskPrompt. It asserts the brief says "no check of its head is recorded as failed" and never includes the deploy's summary. It then adds a failed check and asserts it is quoted with its kind and assertion. One small difference: a check_failed run with no pull now gets the generic red-check brief instead of the "no failure recorded" one.

4. Drove now has a doc comment explaining stalled. The only reader of driveRepo's return value is the tick log's did field, which logs the word as it is. Nothing branches on the value, so an unknown one is never treated as an error, and there is no other reader to test. The existing tests that expect "idle" (autonomy, crew-loop, crew-direction-cap, crew-held) still pass.

bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2,709 test files for this diff (about 228 minutes, over its 5-minute budget), so I picked the 16 files that cover check-failed briefs, settling and the loop's turn. All 165 tests in them pass.

To try it: bun test test/no_change_recheck.test.ts test/failure-quote.test.ts.

🤖 Generated with Claude Code

Approved

Earlier asks, all met in the diff: (1) the quote is once per head: afterNoChange looks for the head-marked comment (alreadyQuoted) and returns undefined, and the test runs two more no-change settles and asserts no new comment and no new re-check. (2) Only a run that ended ok counts as no change, and the errored case is tested. (3) failureBrief is the runner path now, and a test feeds its output for a green-deploy-only head into taskPrompt. (4) Drove has a doc comment saying nothing branches on it. This is an engine change with no new door, so no docs/features page or TUI screen is needed. I read the diff and did not exercise the preview. Two things I would do next, non-blocking: (a) failureBrief reads the run log twice (32 KiB and 8 KiB); read once and slice. (b) Check on production that the tick log shows did=stalled beside a red pull, since the dashboard line was never confirmed.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs