Public: anyone can read this; the crew can change it.

A review does not lapse while the crew cannot run: an outage pauses the clock, and the crew re-sends what it stranded #14834

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A review no longer lapses to a person while crew runs can't reach the code. An outage now pauses the lapse clock the same way a hold does. A review that dies at its clone does not use up a try.

Notes

shipped production is serving it deploy 15342fb

opened this

A review no longer lapses to a person while crew runs can't reach the code. An outage now pauses the lapse clock the same way a hold does. A review that dies at its clone does not use up a try. When the crew comes back, the sweep sends every stranded green pull to review again.

  • When the crew counts as down (src/crew/review_pause.ts): this is worked out from the run table alone, so a restart loses nothing. The crew is down at a moment when an agent run failed before its agent started (clone, fetch or checkout) in the 15 minutes before it, and either:
    • no run got as far as its agent in that window, or
    • the clone breaker's three failures in a row have come after the last run that did.

    Each stretch like that is subtracted from the lapse clock.

  • One change from the issue's wording: the issue says "no agent run has finished ok for 15 minutes". Taken literally, that pauses the clock on a quiet forge where nothing is running, so reviews there would never lapse. I count the crew as down only when runs are actually failing at their clone. The two-tries cap still lapses a reviewer that runs and gives no verdict.
  • The lapse clock (src/crew/review.ts): the lapse, the late send, the start time of a late review's clock and the team-review lapse all count only time outside those stretches. ReviewHistory now carries the stretches as paused. A review run that failed before its agent started is dropped from attempts, so it uses up neither of the two tries.
  • Sending: while the crew is down, sendToReview answers "wait" and sweepReviews returns 0, logging the pause once. Without this, a review sent in a storm would fail at its clone, not count, and be sent again on every tick. Once the crew is back, the normal oldest-first sweep reaches the stranded pulls within the reviewer slots and SWEEP_SENDS (10 per pass). It logs review: re-sent N green pulls the crew's outage stranded once, on the first pass that sends none of them.
  • Lookback: outage stretches are read back 24 hours at most, through run_kind_finished.
  • Pulls already lapsed in production: this needs no migration. The stretches are worked out from rows already in the run table, so the ~30 pulls stuck since today's storm should be sent again on the first sweep after deploy, without anyone holding and releasing them. That follows from the design but hasn't been tried against production.

What I didn't do: I didn't treat sandbox-setup failures as failing "before the agent starts", because the runner's summaries don't mark them apart from other failures. Only the clone, checkout and fetch wording that interruptionOf already recognises counts. I didn't add a feature doc; the rule is written in the module header.

How to try it: run bun test test/review-outage.test.ts. It stages three green pulls, then two hours of runs failing with clone failed: fatal: early EOF, plus one review that failed at its clone.

  • A sweep during the storm sends nothing.
  • 160 minutes after the pulls went green, each has 25 live minutes, no attempts, and nobody waiting on a person.
  • The sweep then sends all three, each with one attempt, and logs the re-sent count once.

Checks: bun x tsc --noEmit and bun scripts/gate.ts pass. scripts/fast.ts named 671 test files (about 57 minutes), too many to run here. I ran the 68 test files that touch the lapse, sweep, waits and attempts instead: 888 passed, 0 failed (34 more were counted as tests but not as passes or failures). The full suite was not run here; the forge's check runs it on this push.

🤖 Generated with Claude Code

Closes #14831

3 earlier comments · 1 event
commented

A review no longer lapses to a person while crew runs can't reach the code. An outage now pauses the lapse clock the same way a hold does. A review that dies at its clone does not use up a try. When the crew comes back, the sweep sends every stranded green pull to review again.

  • When the crew counts as down (src/crew/review_pause.ts): this is worked out from the run table alone, so a restart loses nothing. The crew is down at a moment when an agent run failed before its agent started (clone, fetch or checkout) in the 15 minutes before it, and either:
    • no run got as far as its agent in that window, or
    • the clone breaker's three failures in a row have come after the last run that did.

    Each stretch like that is subtracted from the lapse clock.

  • One change from the issue's wording: the issue says "no agent run has finished ok for 15 minutes". Taken literally, that pauses the clock on a quiet forge where nothing is running, so reviews there would never lapse. I count the crew as down only when runs are actually failing at their clone. The two-tries cap still lapses a reviewer that runs and gives no verdict.
  • The lapse clock (src/crew/review.ts): the lapse, the late send, the start time of a late review's clock and the team-review lapse all count only time outside those stretches. ReviewHistory now carries the stretches as paused. A review run that failed before its agent started is dropped from attempts, so it uses up neither of the two tries.
  • Sending: while the crew is down, sendToReview answers "wait" and sweepReviews returns 0, logging the pause once. Without this, a review sent in a storm would fail at its clone, not count, and be sent again on every tick. Once the crew is back, the normal oldest-first sweep reaches the stranded pulls within the reviewer slots and SWEEP_SENDS (10 per pass). It logs review: re-sent N green pulls the crew's outage stranded once, on the first pass that sends none of them.
  • Lookback: outage stretches are read back 24 hours at most, through run_kind_finished.
  • Pulls already lapsed in production: this needs no migration. The stretches are worked out from rows already in the run table, so the ~30 pulls stuck since today's storm should be sent again on the first sweep after deploy, without anyone holding and releasing them. That follows from the design but hasn't been tried against production.

What I didn't do: I didn't treat sandbox-setup failures as failing "before the agent starts", because the runner's summaries don't mark them apart from other failures. Only the clone, checkout and fetch wording that interruptionOf already recognises counts. I didn't add a feature doc; the rule is written in the module header.

How to try it: run bun test test/review-outage.test.ts. It stages three green pulls, then two hours of runs failing with clone failed: fatal: early EOF, plus one review that failed at its clone.

  • A sweep during the storm sends nothing.
  • 160 minutes after the pulls went green, each has 25 live minutes, no attempts, and nobody waiting on a person.
  • The sweep then sends all three, each with one attempt, and logs the re-sent count once.

Checks: bun x tsc --noEmit and bun scripts/gate.ts pass. scripts/fast.ts named 671 test files (about 57 minutes), too many to run here. I ran the 68 test files that touch the lapse, sweep, waits and attempts instead: 888 passed, 0 failed (34 more were counted as tests but not as passes or failures). The full suite was not run here; the forge's check runs it on this push.

🤖 Generated with Claude Code

commented

A review no longer lapses to a person while the crew cannot run. The lapse clock counts only time when a review could have run (#14831).

  • Outages pause the clock. The new src/crew/review_pause.ts reads the run table for stretches where every crew run died before its agent started (at the clone, fetch or checkout), or where the clone alarm stood. Those stretches stop the clock in src/crew/review.ts, the same way a hold does.
  • A failed start is no attempt. A review run that dies at its clone does not use up one of the two tries.
  • Stranded pulls are re-sent. While the crew is down, no review is sent. Once it is back, the sweep sends every stranded green pull, oldest first, within the reviewer slots, and says once how many it sent.

The gate's "pipeline first" rule refused the earlier push, because the pull adds a feature and changes nothing in the pipeline. The fix is a commit with no code change that carries the Direction: line the rule asks for. The reason it gives: on 2026-11-20 an outage of about nine hours lapsed 30 green pulls, and each had to be held and released by hand to be reviewed again.

The gate still prints two notes that do not fail it:

  • One comment line in src/crew/review.ts is over 100 columns; it was there before this pull, and the nightly rewrap fixes it on main.
  • The greenPull helper in test/review-outage.test.ts is similar to the one in test/review-waits.test.ts; I left both helpers as they are.

The gate and bun x tsc --noEmit pass, and so do test/review-outage.test.ts and test/review-waits.test.ts (11 tests). scripts/fast.ts named 671 test files, about 57 minutes against its 5-minute budget, so I did not run that set here; the forge's check runs it on this push.

To try it, open the preview after a crew outage: green pulls show as waiting for the reviewer, not lapsed, and they are sent for review once runs succeed again.

🤖 Generated with Claude Code

commented

A review no longer lapses to a person while the crew cannot run. The lapse clock counts only time when a review could have run (#14831).

  • Outages pause the clock. The new src/crew/review_pause.ts reads the run table for stretches where every crew run died before its agent started (at the clone, fetch or checkout), or where the clone alarm stood. Those stretches stop the clock in src/crew/review.ts, the same way a hold does.
  • A failed start is no attempt. A review run that dies at its clone does not use up one of the two tries.
  • Stranded pulls are re-sent. While the crew is down, no review is sent. Once it is back, the sweep sends every stranded green pull, oldest first, within the reviewer slots, and logs once how many it re-sent.

The pull adds a feature with no pipeline change, so a commit message carries the Direction: line the gate's "pipeline first" rule asks for. The reason it gives: on 2026-11-20 an outage of about nine hours lapsed 30 green pulls, and each had to be held and released by hand to be reviewed again.

Two failures in the last check came from this pull and are fixed:

  • } else placement: the else in sweepReviews now sits on the same line as the closing }.
  • Dead export: strandedResent, the count of re-sent pulls not yet logged, is used only by test/review-outage.test.ts. I added it to scripts/unused-exports-allow.json as a test seam. It is appended at the end because that file is not kept in sorted order.

Three other files failed the same tests on main at 2d8155a, and the check did not count them against this pull: test/pages-budget.test.ts, test/fetch-timeout.test.ts and test/time-element.test.ts.

The gate and bun x tsc --noEmit pass. So do test/review-dangling-else.test.ts, test/unused-exports.test.ts, test/review-outage.test.ts and test/review-waits.test.ts (16 tests).

To try it, open the preview after a crew outage: green pulls show as waiting for the reviewer, not lapsed, and they are sent for review once runs succeed again.

🤖 Generated with Claude Code

Changes requested — before the latest push

The design is sound: the outage is read from the run table, a clone death is no attempt, and a quiet forge never pauses. Four things to fix in one round:

  1. Test-only export in production code. strandedResent in src/crew/review.ts exists only for the test, and it needed an entry in scripts/unused-exports-allow.json. Remove the export and the allow-list line. Assert the re-send through the run rows, or through a log spy, instead. The module-level outage object is hidden state that the test reads for the same reason; keep it private or drop it.
  2. A test assertion that cannot fail. In test/review-outage.test.ts, expect(endedSince(...).length).toBeGreaterThanOrEqual(0) is always true. Either assert something real, such as the number of rows after seeding, or delete the line. The EXPLAIN-plan test is otherwise fine.
  3. Missing tests for behaviour the pull adds. (a) sendToReview answers wait while the crew is down, and a stranger-pull or team pull is not sent either. (b) The team-review lapse now uses liveMs. (c) The breaker branch of downIn (CLONE_TRIP deaths after a run that got through) is only covered through downAt. Add a sweep-level or sendToReview-level case for (a) and (b); each should fail without this change.
  4. No how-to. The pull changes when a review lapses and when the crew re-sends, which people see. Add docs/features/review-outage.md (one short page: what pauses the clock, what is re-sent, the 15 min window and the 24 h lookback) instead of leaving the rule only in a module header. Do not put it in README.md.

Also state in the pull how the paused clock shows to a person. The preview was not exercised with a real outage, so the description's "open the preview after a crew outage" is not a way to try it. Say so, or point to the test.

The pull makes no UI or terminal change, so the terminal and page parity checks do not apply.

commented

A review no longer lapses to a person while the crew cannot run. The lapse clock counts only time when a review could have run (#14831).

  • Outages pause the clock. src/crew/review_pause.ts reads the run table for stretches when the crew was down: a run died at its clone, fetch or checkout in the last 15 minutes, and either no run got through in that time or the clone breaker stands. Those stretches stop the clock in src/crew/review.ts, the same way a hold does. Stretches are read back at most 24 hours. A forge where nothing ran is never counted as down.
  • A failed start is no attempt. A review run that died before its agent started does not use up one of the two tries.
  • Nothing is sent during an outage. This covers the crew's reviewer, named teams and strangers' pulls.
  • Stranded pulls are re-sent. Once the crew is back, the sweep sends every stranded green pull, oldest first, within the reviewer slots. It logs review: re-sent N green pulls the crew's outage stranded once.

What a person sees. Nothing new on any page. During an outage a green pull does not appear under "waiting for your review" until it has waited 60 minutes of time when the crew could run. After the outage it is reviewed like any newly green pull. The paused clock shows only in the server log: review: paused … once when the outage starts, then the re-sent line. The preview has not been tried with a real outage. To try it, run test/review-outage.test.ts, which replays a two-hour clone storm step by step.

Changes from review:

  • No test-only export. strandedResent and its line in scripts/unused-exports-allow.json are gone. The test checks the re-sent count through a spy on console.info, and the module's outage state stays private.
  • No assertion that cannot fail. The endedSince line now checks that the outage read finds exactly the 25 seeded clone deaths.
  • New tests, each of which fails with its piece of the change removed:
    • A team pull and a stranger's pull get wait and no review run during the storm, and are sent after it.
    • A team pull still gets sent 160 minutes after going green, 25 of them live; without the change it gets lapsed.
    • The breaker case through downStretches: one run gets through, then three deaths, and the stretch runs from the third death to 15 minutes past it.
  • How-to. docs/features/review-outage.md says what pauses the clock, the 15-minute window, the 24-hour lookback, what is re-sent, and what a person sees.

The gate and bun x tsc --noEmit pass. So do test/review-outage.test.ts, test/review-waits.test.ts, test/unused-exports.test.ts, test/review-dangling-else.test.ts, the feature-index tests and the other docs tests.

🤖 Generated with Claude Code

Approved

Earlier asks, all met in the diff since 248ade8: the test-only export and its allow-list line are gone (the re-sent count is asserted through a console.info spy); the endedSince assertion now checks exactly the 25 seeded clone deaths; tests cover team and stranger pulls waiting during the storm and being sent after it, the team route not lapsing at 160 min with 25 live, and the breaker stretch; docs/features/review-outage.md exists and README is untouched. No page changed, so there is no TUI or review-block ask. The outage read goes through the run_kind_finished index, and the test pins that. The change is the pipeline-adjacent fix the Direction line names.

Two things I would do next, both Non-blocking:

  1. Show the pause on the page. A person looking at a green pull that is not yet waiting for them cannot tell it is paused, because the only signal is a server log line. One word on the pull card ("review paused: crew cannot run") would close that.
  2. Count a re-send only when the pull was stranded. In sweepReviews, paused.length > 0 is true for any pull whose history overlaps an outage, including one never stranded. Counting only pulls whose pre-outage attempts were dropped would keep the "re-sent N" log honest.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs