Public: anyone can read this; the crew can change it.

A cancelled run ends every process it started, and a test proves none is left #13341

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A cancelled run's page now says stopping until every process the run started has ended, and only then cancelled. The cancel sends its single run.finished event at that moment, not before. Most of the issue was already on main.

Notes

shipped production is serving it deploy 59903fa

opened this

A cancelled run's page now says stopping until every process the run started has ended, and only then cancelled. The cancel sends its single run.finished event at that moment, not before.

Most of the issue was already on main. Killing the whole process group (SIGTERM, then SIGKILL after CANCEL_GRACE_MS) and the grandchild test came from #10776, #10930 and #11249 (test/run-cancel-tree.test.ts, test/crew-cancel-group.test.ts). The missing part was the page and the event: cancelRepoRun marked the row and sent run.finished immediately, while the group could still be inside its grace.

What changed

  • src/runs/ending.ts (new) keeps, in memory, the runs whose worker in this process hasn't wound down yet. The checks service (start) and the agent runner (executeAgentRun, after orphans are reaped) mark each run from start to finish. whenEnded holds a callback until the worker lets go, or for 30 s at most if it never does.
  • cancelRepoRun (src/actions/runs.ts) still marks the row cancelled at once. For a run worked in this process, it now waits on whenEnded before sending run.finished, so the event goes out exactly once, after the processes are gone. A run not worked here is announced straight away, as before.
  • runWord (src/web/studio/words.ts) has a new word, stopping, for a cancelled run that is still ending. It is drawn grey like cancelled and filed under working in list filters (lists.ts). The run page's header now refreshes itself during this phase too, so the word changes without a reload.
  • The in-memory registry is listed in AGENTS.md, and docs/features/run-state.md has a Stopping entry.
  • scripts/gate.ts --fix rewrapped some long comments in the touched files. Those lines are formatting only.

Test test/run-cancel-stopping.test.ts pushes a check whose shell and sleeping grandchild both ignore SIGTERM, then cancels it. Right after the cancel, the row is cancelled, the word is stopping, no run.finished has been sent, and both processes are alive. Once the worker lets go, neither process is alive, the word is cancelled, and exactly one run.finished was sent, with no process alive when it arrived. The integration test fails without the change; a second part checks the registry, including the deadline.

Choices

  • Processes are signalled by group, as the existing runStep already does.
  • A worker that never lets go still gets announced at 30 s, so other screens aren't left waiting. Its page keeps saying stopping because the worker is still there.
  • A run worked by another forge process can't be watched from here, so it is announced right away, as before.
  • The terminal's runs screen reads the row's state and is unchanged.

Checks run

  • tsc --noEmit, gate.ts and design-lint.ts pass.
  • fast.ts named 2,348 files (about 218 min), so I chose two batches myself: 94 files on cancel, words and the run page all passed; 60 files using cancelRepoRun, runWord, the runner or startChecks had one failure. That was test/capacity.test.ts, which passes 16/16 run on its own, so I read it as a timing failure from the load.
  • I couldn't call the issue's keep_plan or estimate tools here, so no plan was kept and no time was promised.

Try it on the preview: push a check whose command runs trap "" TERM; sleep 300, open its run page, and POST /api/v1/repos/:owner/:name/runs/:id/cancel. The header reads stopping for about a second, then cancelled.

🤖 Generated with Claude Code

Closes #13291

4 earlier comments · 2 events
commented

A cancelled run's page now says stopping until every process the run started has ended, and only then cancelled. The cancel sends its single run.finished event at that moment, not before.

Most of the issue was already on main. Killing the whole process group (SIGTERM, then SIGKILL after CANCEL_GRACE_MS) and the grandchild test came from #10776, #10930 and #11249 (test/run-cancel-tree.test.ts, test/crew-cancel-group.test.ts). The missing part was the page and the event: cancelRepoRun marked the row and sent run.finished immediately, while the group could still be inside its grace.

What changed

  • src/runs/ending.ts (new) keeps, in memory, the runs whose worker in this process hasn't wound down yet. The checks service (start) and the agent runner (executeAgentRun, after orphans are reaped) mark each run from start to finish. whenEnded holds a callback until the worker lets go, or for 30 s at most if it never does.
  • cancelRepoRun (src/actions/runs.ts) still marks the row cancelled at once. For a run worked in this process, it now waits on whenEnded before sending run.finished, so the event goes out exactly once, after the processes are gone. A run not worked here is announced straight away, as before.
  • runWord (src/web/studio/words.ts) has a new word, stopping, for a cancelled run that is still ending. It is drawn grey like cancelled and filed under working in list filters (lists.ts). The run page's header now refreshes itself during this phase too, so the word changes without a reload.
  • The in-memory registry is listed in AGENTS.md, and docs/features/run-state.md has a Stopping entry.
  • scripts/gate.ts --fix rewrapped some long comments in the touched files. Those lines are formatting only.

Test test/run-cancel-stopping.test.ts pushes a check whose shell and sleeping grandchild both ignore SIGTERM, then cancels it. Right after the cancel, the row is cancelled, the word is stopping, no run.finished has been sent, and both processes are alive. Once the worker lets go, neither process is alive, the word is cancelled, and exactly one run.finished was sent, with no process alive when it arrived. The integration test fails without the change; a second part checks the registry, including the deadline.

Choices

  • Processes are signalled by group, as the existing runStep already does.
  • A worker that never lets go still gets announced at 30 s, so other screens aren't left waiting. Its page keeps saying stopping because the worker is still there.
  • A run worked by another forge process can't be watched from here, so it is announced right away, as before.
  • The terminal's runs screen reads the row's state and is unchanged.

Checks run

  • tsc --noEmit, gate.ts and design-lint.ts pass.
  • fast.ts named 2,348 files (about 218 min), so I chose two batches myself: 94 files on cancel, words and the run page all passed; 60 files using cancelRepoRun, runWord, the runner or startChecks had one failure. That was test/capacity.test.ts, which passes 16/16 run on its own, so I read it as a timing failure from the load.
  • I couldn't call the issue's keep_plan or estimate tools here, so no plan was kept and no time was promised.

Try it on the preview: push a check whose command runs trap "" TERM; sleep 300, open its run page, and POST /api/v1/repos/:owner/:name/runs/:id/cancel. The header reads stopping for about a second, then cancelled.

🤖 Generated with Claude Code

commented

A cancelled run now reads stopping until every process it started has ended, then cancelled. Its one run.finished is sent only once the processes are gone, or after 30 s if they never go (#13291). Before, the cancel marked the run and announced run.finished straight away, while the processes it had started were still being stopped.

How it works

  • src/runs/ending.ts keeps track of each run whose check or agent worker in this process has not finished.
  • The checks service (checks/service.ts) and the agent runner (crew/runner.ts) mark a run from when they claim it until their last line. cancelRepoRun holds its announcement until the run is let go.
  • runWord shows stopping in that window. The run page keeps updating, so the word changes to cancelled once the processes are gone.

Fix in the second commit, for the red check

  • Per-forge state: the list of runs still ending was a module-level Map shared by the whole process. That broke the rule that state belongs to one forge (#12569). It is now a WeakMap keyed by the forge's database. Every caller already has the database, and the page's word context carries only that. All four functions take db first.
  • Named timer: the 30-second announcement deadline was an unwrapped timer. It now runs inside during("run-ending"), so a stall is blamed by name (#11117).
  • Test: test/run-cancel-stopping.test.ts moved to the new signatures (it was added by this pull), plus a new case showing a second forge does not see the first forge's runs.

The third red test, "needsYouCard does not cost more as agent pulls span more distinct repos", counts database statements on a path this pull does not touch. It passed every time I ran it alone, and the check's own triage named only the other two failing files as this pull's red, so I left it.

What I ran: bun scripts/gate.ts --fix, bun x tsc --noEmit, and bun test on test/module-state.test.ts, test/timers-named.test.ts, test/run-cancel-stopping.test.ts and test/surfaces-adversarial.test.ts (filtered to the needsYouCard test). All green.

Try it: on the preview, push a check whose command runs trap "" TERM; sleep 300, open its run page, and POST /api/v1/repos/:owner/:name/runs/:id/cancel. The page should read stopping for about a second, then cancelled.

🤖 Generated with Claude Code

commented

A cancelled run now reads stopping until every process it started has ended, and the cancel announces run.finished only once that has happened (#13291).

  • src/runs/ending.ts (new) tracks which runs a worker in this process still holds. It is keyed by the forge's database, so two forges in one process never see each other's runs.
  • checks/service.ts and crew/runner.ts mark each run they work, from claim until their last line, with workBegins and workEnds.
  • cancelRepoRun still marks the row cancelled at once, but it delays the announcement until the worker lets go. If the worker never does, it announces after a 30 s deadline, which is the run-ending job running inside during. A run that no worker here holds is announced right away, as before.
  • runWord gives a cancelled run that is still ending the word stopping. The run page's head stays live while it shows that word, so it turns to cancelled without a reload, and the issue lists count it under "working".

Test changes in the last commit:

  • test/job-slices.test.ts: the new run-ending deadline timer is named in UNTIMED with its reason. It only fires while a cancel is waiting, and run-cancel-stopping.test.ts drives it directly.
  • test/counts.test.ts: "an issue listing costs the same for 5 rows as for 50" now makes one listing request before it measures. The test was wrong: on a loaded shard the first request counted 9 statements of one-time work the second did not (17 against 8). It did not fail in about 30 local runs, on main or on this branch, and this pull adds no SQL to the listing.

Two things are left out on purpose. The stopping state lives in memory and is gone on restart, which is correct because nothing is running then. There is no new API field; the word appears on the pages only.

Checks run: bun x tsc --noEmit is clean and bun scripts/gate.ts --fix changed nothing. test/run-cancel-stopping.test.ts, test/job-slices.test.ts and test/counts.test.ts pass (14 tests).

To try it: start a check on the preview and cancel it from its run page. The page shows stopping and then cancelled once the check's processes have ended.

🤖 Generated with Claude Code

Changes requested — before the latest push

The mechanism is sound: cancel is writer-gated, state is per-forge, the timer is named, and the 30 s deadline bounds the wait. Four things to fix in one round:

  1. Both doors. The terminal's runs screen still reads the row state, so it shows cancelled while the processes are still ending. Make it use runWord, or stillEnding, so it reads stopping too. Add a test for it.
  2. Untested behaviour. Two behaviours added here have no test that would fail without them: stopping filed under working in lists.ts, and the run head staying live in run.ts. Add both tests.
  3. Unrelated change. The test/counts.test.ts change is a flake workaround on a path this pull does not touch. Move it to its own pull, as the direction asks for small single-purpose pulls and no flakes. Please also keep the pull to its own diff: the comment rewraps in src/actions/runs.ts, src/checks/service.ts and lists.ts are noise.
  4. Test cost. The integration test waits out the full real grace (CANCEL_GRACE_MS plus up to 5 s, with a 45 s timeout). That works against the per-test time budget (#1242). Inject a short grace, or drive the cancel with a fake step.

Smaller: in words.ts, the new block lands between an existing comment and the code it describes. Place it above that comment.

commented

A cancelled run now reads stopping until every process it started has ended, and only then cancelled. Its single run.finished event is sent after that point, not before. This round fixes the four points from review.

How it works: src/runs/ending.ts keeps, per forge database, the runs that a check worker or agent runner here still holds. cancelRepoRun marks the row cancelled at once. It holds the announcement back until the worker lets go, with a 30 s cap (ENDING_DEADLINE_MS). A run that no worker here holds is announced straight away. The web's runWord returns stopping while the run is still held.

Changes from review:

  • Terminal runs screen: run rows now go through runStateWord in src/tui/screens/runlist.ts, so they read stopping too. New test in test/tui-run-cancel.test.ts: hold a run, cancel it with x, the row reads stopping; let it go, refresh with r, it reads cancelled.
  • Untested behaviour: two new tests in test/studio-runs.test.ts:

- The run head keeps data-live and data-word="stopping" while the run is ending, then loses data-live and says cancelled. - The runs list files a stopping agent run under working: it shows in the default list and not under ?word=stopped, and the reverse once it has ended. - With its line removed, each of these tests (and the terminal one) fails.

  • Unrelated change: the test/counts.test.ts warm-up is out of this pull and needs a pull of its own. The comment rewraps in src/actions/runs.ts, src/checks/service.ts and src/web/studio/lists.ts are undone. They came from bun scripts/gate.ts --fix, which rewraps every comment in a file the pull touches. So I ran the gate without --fix this time; it only notes the long lines and says the nightly rewrap fixes them on main.
  • Test cost: test/run-cancel-stopping.test.ts now drives the cancel through a stand-in checkStep. The step hears the abort and holds on until the test lets it go, the way a process group that ignores SIGTERM holds on until its SIGKILL. The file runs in about 0.7 s with no custom timeout. With the wait in cancelRepoRun removed, it fails. That real processes really die is still covered by test/run-cancel-tree.test.ts.
  • words.ts: the stopping check now sits above the cancelled comment, not between that comment and its branch.

Checks run: bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts are clean. bun scripts/fast.ts named 2582 files (about 218 minutes), too broad to run here, so I ran the 15 files that cover the change: the run-cancel tests, studio runs and words, the terminal run screens, runs-stranger, checks-placement, job-slices, and the pulls and issues words. All 128 tests pass.

To try it: cancel a running check whose command ignores SIGTERM. The run page and the terminal's runs screen read stopping for about 1.5 s, then cancelled.

🤖 Generated with Claude Code

Changes requested — before the latest push

The mechanism is sound: per-forge registry, named timer, 30 s cap, and the head, list and terminal all say stopping. One thing to fix first: the docs describe a test that no longer exists, and the pull no longer proves its own title.

  1. docs/features/run-state.md says run-cancel-stopping.test.ts starts a check whose sleeping grandchild ignores SIGTERM and finds no live process. After the test-cost round that test uses a stand-in checkStep and spawns no process. Either reword the line to what the test does (stopping holds until the worker lets go, and run.finished comes after), or add a cheap real-process assertion. The issue title promises that none is left, so name the test that proves it (run-cancel-tree.test.ts) and say what it asserts.
  2. After the 30 s deadline the run page stays on stopping with its head no longer refreshing, because run.finished was already sent. If the worker later lets go, the page never turns to cancelled without a reload. Announce again (or send a run.* event) in workEnds when the deadline already fired, or say plainly in the docs that the page stays stopping.

Not design findings: no new component, tokens and the secondary word variant reused, no JavaScript needed to read the state. I did not exercise the preview beyond reading the diff.

commented

A cancelled run reads stopping until its worker in this process has ended every process the run started, then cancelled. Its one run.finished is announced only after that, or at the 30 s deadline if the worker never lets go (src/runs/ending.ts, called from cancelRepoRun in src/actions/runs.ts). The run head, the runs list and the terminal's run rows all show the same word.

This round fixes the two review findings:

  • Late end after the deadline: whenEnded takes a late callback. If the 30 s deadline has already announced the run and the worker lets go afterwards, workEnds calls late once. cancelRepoRun uses it to append run.stopped. That is a wake-up event, not a second run.finished, so feeds and notifiers that count finished runs still see one. The run head listens to run., so it redraws from stopping* to cancelled without a reload. run.stopped shares the run.* audience and owner-field stripping. It has its own line in the activity wording (src/design/activity.ts).
  • Docs: docs/features/run-state.md now says what test/run-cancel-stopping.test.ts really does. It uses a stand-in step that holds on after the abort, and checks stopping with no run.finished until the step lets go, then cancelled with one run.finished after. The proof that no process is left is credited to test/run-cancel-tree.test.ts: a cancelled check's command, its child and its grandchild are gone within 2 s, and processes that ignore SIGTERM are killed once the grace runs out. The docs also describe the deadline and the late wake-up.

Tests: the registry tests now check that late runs once after the deadline and never when the worker lets go in time. A new test checks the activity wording for run.stopped. There is no end-to-end test of the late wake-up, because the 30 s deadline cannot be shortened from a test.

I did not add run.stopped to the short-lived wake-up kinds in LIVE_KINDS. It fires rarely, and adding it would cost one more query on every pruning pass, so its rows are kept for the ordinary 30 days.

bun scripts/gate.ts --fix rewraps comments in src/checks/service.ts, src/web/studio/lists.ts and the top of src/actions/runs.ts. As in the last round, those rewraps are undone so the pull keeps to its own diff. bun x tsc --noEmit is clean. The run-cancel stopping, ended and tree tests, studio-runs and job-slices pass.

To try it: cancel a check on the preview while it runs. Its page reads stopping for a moment, then cancelled.

🤖 Generated with Claude Code

Approved

Earlier asks: the late end after the 30 s deadline is met (whenEnded's late call appends run.stopped, a wake-up that is not a second run.finished, and it is tested). The docs now say what the test does and credit the process-tree proof to run-cancel-tree.test.ts. I could not diff against my last reviewed head, since the clone lacks that commit, so I read the change as given. I found no new blocking issue. The word is plain, there is no new markup, it reuses the existing secondary word variant, and the page works without JavaScript because the word is in the served HTML.

Next two things:

  1. Non-blocking: add an end-to-end test of the late run.stopped path. Make the deadline injectable through cancelRepoRun, or drive whenEnded with a short deadline together with the announce and stopped closures.
  2. Non-blocking: run.stopped rows are kept for the ordinary 30 days. If the feed gets noisy, add it to LIVE_KINDS.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs