A crew run interrupted by a forge stop twice in a row is not queued a third time, and the forge's home says why #11576

A run that a forge restart has stopped twice in a row is now marked failed with the reason, instead of being queued a third time. The owner's home shows it with a "Try again" button.

The work was already on this branch from the earlier tries. I re-checked it here and changed no code. The previous check was stopped for CPU rather than failing a test, so I pushed an empty commit to run the check again. The forge's check has not reported yet.

What the change does

• Second interruption: the first restart still queues the run again. The second one fails it with "stopped twice by its forge restarting; not tried again" (src/core/runs.ts, src/crew/dispatch.ts).
• Credit: the interrupted run is recorded once in the ledger, so the second try is not charged for minutes the first already paid for.
• Home: the forge owner sees one line naming the run and why it stopped. "Try again" starts it afresh (src/web/studio/home.ts, src/web/studio/held.ts, src/web/pages/runs.ts, src/actions/runs.ts, src/http/server.ts).
• Tests: test/crew-interrupted.test.ts covers one interruption queuing again, two in a row failing with the reason and not queuing, and credit charged once.

Checked here

• bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean.
• The interrupted tests pass, along with 93 held, continues, home and runs tests (94 in total).
• I did not run the full suite, because scripts/fast.ts named 2342 of the 2859 test files, a diff too broad to select from. The forge's check runs it.

Left out: no feature doc. The earlier try dropped it because a feature doc needs a pipeline change, and this description carries the feature instead.

Closes #11560

A run that a forge restart has stopped twice in a row is now marked failed with the reason, instead of being queued a third time. The owner's home shows it with a "Try again" button.

The work was already on this branch from the earlier tries. I re-checked it here and changed no code. The previous check was stopped for CPU rather than failing a test, so I pushed an empty commit to run the check again. The forge's check has not reported yet.

What the change does

• Second interruption: the first restart still queues the run again. The second one fails it with "stopped twice by its forge restarting; not tried again" (src/core/runs.ts, src/crew/dispatch.ts).
• Credit: the interrupted run is recorded once in the ledger, so the second try is not charged for minutes the first already paid for.
• Home: the forge owner sees one line naming the run and why it stopped. "Try again" starts it afresh (src/web/studio/home.ts, src/web/studio/held.ts, src/web/pages/runs.ts, src/actions/runs.ts, src/http/server.ts).
• Tests: test/crew-interrupted.test.ts covers one interruption queuing again, two in a row failing with the reason and not queuing, and credit charged once.

Checked here

• bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean.
• The interrupted tests pass, along with 93 held, continues, home and runs tests (94 in total).
• I did not run the full suite, because scripts/fast.ts named 2342 of the 2859 test files, a diff too broad to select from. The forge's check runs it.

Left out: no feature doc. The earlier try dropped it because a feature doc needs a pipeline change, and this description carries the feature instead.

A run that a forge restart has stopped twice in a row is now marked failed with the reason, instead of being queued a third time. The owner's home shows it with a "Try again" button. I pushed one new change, and the forge's check on it has not reported yet.

Why the check failed. The code was never red. Because the change touches src/core/runs.ts and src/http/server.ts, the fast tier selected 2616 test files in 96 shards. The CPU guard stopped that check at 19.9 cores against a 16-core limit. The log also shows Chrome-style egress refusals, so some tests spawn a browser. That is the second CPU stop on this change.

What I changed. In the check.fast command in forge.json, I set FORGE_CHECK_SHARDS=8 before scripts/test-shards.ts. That runs the fast tier on 8 workers instead of one per core, so it uses fewer cores and takes longer. It is a pipeline change, and the explicit setting overrides the cell's own CPU cap, so please look at it. If you'd rather not touch forge.json, drop that commit and re-run, but a change this broad is likely to hit the CPU guard again.

Checked here. scripts/gate.ts is clean, forge.json still parses, and test/crew-interrupted.test.ts passes (15 tests). I did not run the full suite.

The feature (unchanged since the earlier push):

• The first restart still queues the run again. The second one fails it with "stopped twice by its forge restarting; not tried again".
• The credit ledger records the interrupted run once.
• The forge owner's home shows one line naming the run and why it stopped, with "Try again".
• test/crew-interrupted.test.ts covers one interruption queuing again, two in a row failing with the reason, and credit charged once.

There is no feature doc, because it would need a pipeline change and this description carries the feature instead.

Runs that a forge restart has stopped twice in a row are now marked failed with the reason, instead of being queued a third time. The check has failed again on CPU, not on a test, and I've pushed a further cap. That push has not been checked yet.

The feature. The first restart still queues the run again. The second one fails it with "stopped twice by its forge restarting; not tried again".

• Credit: the ledger records the interrupted run once, so the second try isn't charged again.
• Home: the forge owner sees one line naming the run and why it stopped, with "Try again".
• Tests: test/crew-interrupted.test.ts covers one interruption queuing again, two in a row failing with the reason, and credit charged once.

Why the check keeps failing. The change touches src/core/runs.ts and src/http/server.ts, which most of the suite imports. The fast tier therefore selects about 2,600 test files, many of which launch headless Chromium. Last time I capped it at 8 workers; it still hit 16.1 cores against a 16-core limit, so I think Chromium's bursts, not the number of workers, set the peak.

What I changed. In forge.json, check.fast now sets FORGE_CHECK_SHARDS=4, which overrides the cell's own cap. This is a pipeline change, so please look at it. The fast tier will take a good deal longer, though it should stay well inside the 7200 s timeout. If the check is stopped for CPU again, the next step is to limit how many Chromium tests run at once, or to split this change so it reaches fewer files. I did not do either.

Checked here. I did not re-run the tests this time, since only the forge.json setting changed. The gate and test/crew-interrupted.test.ts passed in the previous round.

Parked by the crew: the check stayed red through 2 automatic tries