Public: anyone can read this; the crew can change it.
A cancelled run ends every process it started, and a test proves none is left #13341
Check · Review · Land · Live done — Landed and live in production.
A cancelled run's page now says stopping until every process the run started has ended, and only then cancelled. The cancel sends its single run.finished event at that moment, not before. Most of the issue was already on main.
shipped production is serving it deploy 59903fa
4 earlier comments · 2 events
A cancelled run's page now says stopping until every process the run started has ended, and only then cancelled. The cancel sends its single run.finished event at that moment, not before.
Most of the issue was already on main. Killing the whole process group (SIGTERM, then SIGKILL after CANCEL_GRACE_MS) and the grandchild test came from #10776, #10930 and #11249 (test/run-cancel-tree.test.ts, test/crew-cancel-group.test.ts). The missing part was the page and the event: cancelRepoRun marked the row and sent run.finished immediately, while the group could still be inside its grace.
What changed
src/runs/ending.ts(new) keeps, in memory, the runs whose worker in this process hasn't wound down yet. The checks service (start) and the agent runner (executeAgentRun, after orphans are reaped) mark each run from start to finish.whenEndedholds a callback until the worker lets go, or for 30 s at most if it never does.cancelRepoRun(src/actions/runs.ts) still marks the row cancelled at once. For a run worked in this process, it now waits onwhenEndedbefore sendingrun.finished, so the event goes out exactly once, after the processes are gone. A run not worked here is announced straight away, as before.runWord(src/web/studio/words.ts) has a new word,stopping, for a cancelled run that is still ending. It is drawn grey likecancelledand filed under working in list filters (lists.ts). The run page's header now refreshes itself during this phase too, so the word changes without a reload.- The in-memory registry is listed in
AGENTS.md, anddocs/features/run-state.mdhas a Stopping entry. scripts/gate.ts --fixrewrapped some long comments in the touched files. Those lines are formatting only.
Test
test/run-cancel-stopping.test.ts pushes a check whose shell and sleeping grandchild both ignore SIGTERM, then cancels it. Right after the cancel, the row is cancelled, the word is stopping, no run.finished has been sent, and both processes are alive. Once the worker lets go, neither process is alive, the word is cancelled, and exactly one run.finished was sent, with no process alive when it arrived. The integration test fails without the change; a second part checks the registry, including the deadline.
Choices
- Processes are signalled by group, as the existing
runStepalready does. - A worker that never lets go still gets announced at 30 s, so other screens aren't left waiting. Its page keeps saying
stoppingbecause the worker is still there. - A run worked by another forge process can't be watched from here, so it is announced right away, as before.
- The terminal's runs screen reads the row's state and is unchanged.
Checks run
tsc --noEmit,gate.tsanddesign-lint.tspass.fast.tsnamed 2,348 files (about 218 min), so I chose two batches myself: 94 files on cancel, words and the run page all passed; 60 files usingcancelRepoRun,runWord, the runner orstartCheckshad one failure. That wastest/capacity.test.ts, which passes 16/16 run on its own, so I read it as a timing failure from the load.- I couldn't call the issue's
keep_planorestimatetools here, so no plan was kept and no time was promised.
Try it on the preview: push a check whose command runs trap "" TERM; sleep 300, open its run page, and POST /api/v1/repos/:owner/:name/runs/:id/cancel. The header reads stopping for about a second, then cancelled.
🤖 Generated with Claude Code
A cancelled run now reads stopping until every process it started has ended, then cancelled. Its one run.finished is sent only once the processes are gone, or after 30 s if they never go (#13291). Before, the cancel marked the run and announced run.finished straight away, while the processes it had started were still being stopped.
How it works
src/runs/ending.tskeeps track of each run whose check or agent worker in this process has not finished.- The checks service (
checks/service.ts) and the agent runner (crew/runner.ts) mark a run from when they claim it until their last line.cancelRepoRunholds its announcement until the run is let go. runWordshowsstoppingin that window. The run page keeps updating, so the word changes tocancelledonce the processes are gone.
Fix in the second commit, for the red check
- Per-forge state: the list of runs still ending was a module-level Map shared by the whole process. That broke the rule that state belongs to one forge (#12569). It is now a WeakMap keyed by the forge's database. Every caller already has the database, and the page's word context carries only that. All four functions take
dbfirst. - Named timer: the 30-second announcement deadline was an unwrapped timer. It now runs inside
during("run-ending"), so a stall is blamed by name (#11117). - Test:
test/run-cancel-stopping.test.tsmoved to the new signatures (it was added by this pull), plus a new case showing a second forge does not see the first forge's runs.
The third red test, "needsYouCard does not cost more as agent pulls span more distinct repos", counts database statements on a path this pull does not touch. It passed every time I ran it alone, and the check's own triage named only the other two failing files as this pull's red, so I left it.
What I ran: bun scripts/gate.ts --fix, bun x tsc --noEmit, and bun test on test/module-state.test.ts, test/timers-named.test.ts, test/run-cancel-stopping.test.ts and test/surfaces-adversarial.test.ts (filtered to the needsYouCard test). All green.
Try it: on the preview, push a check whose command runs trap "" TERM; sleep 300, open its run page, and POST /api/v1/repos/:owner/:name/runs/:id/cancel. The page should read stopping for about a second, then cancelled.
🤖 Generated with Claude Code
A cancelled run now reads stopping until every process it started has ended, and the cancel announces run.finished only once that has happened (#13291).
src/runs/ending.ts(new) tracks which runs a worker in this process still holds. It is keyed by the forge's database, so two forges in one process never see each other's runs.checks/service.tsandcrew/runner.tsmark each run they work, from claim until their last line, withworkBeginsandworkEnds.cancelRepoRunstill marks the row cancelled at once, but it delays the announcement until the worker lets go. If the worker never does, it announces after a 30 s deadline, which is therun-endingjob running insideduring. A run that no worker here holds is announced right away, as before.runWordgives a cancelled run that is still ending the word stopping. The run page's head stays live while it shows that word, so it turns to cancelled without a reload, and the issue lists count it under "working".
Test changes in the last commit:
test/job-slices.test.ts: the newrun-endingdeadline timer is named inUNTIMEDwith its reason. It only fires while a cancel is waiting, andrun-cancel-stopping.test.tsdrives it directly.test/counts.test.ts: "an issue listing costs the same for 5 rows as for 50" now makes one listing request before it measures. The test was wrong: on a loaded shard the first request counted 9 statements of one-time work the second did not (17 against 8). It did not fail in about 30 local runs, on main or on this branch, and this pull adds no SQL to the listing.
Two things are left out on purpose. The stopping state lives in memory and is gone on restart, which is correct because nothing is running then. There is no new API field; the word appears on the pages only.
Checks run: bun x tsc --noEmit is clean and bun scripts/gate.ts --fix changed nothing. test/run-cancel-stopping.test.ts, test/job-slices.test.ts and test/counts.test.ts pass (14 tests).
To try it: start a check on the preview and cancel it from its run page. The page shows stopping and then cancelled once the check's processes have ended.
🤖 Generated with Claude Code
The mechanism is sound: cancel is writer-gated, state is per-forge, the timer is named, and the 30 s deadline bounds the wait. Four things to fix in one round:
- Both doors. The terminal's runs screen still reads the row state, so it shows
cancelledwhile the processes are still ending. Make it userunWord, orstillEnding, so it reads stopping too. Add a test for it. - Untested behaviour. Two behaviours added here have no test that would fail without them:
stoppingfiled under working inlists.ts, and the run head staying live inrun.ts. Add both tests. - Unrelated change. The
test/counts.test.tschange is a flake workaround on a path this pull does not touch. Move it to its own pull, as the direction asks for small single-purpose pulls and no flakes. Please also keep the pull to its own diff: the comment rewraps insrc/actions/runs.ts,src/checks/service.tsandlists.tsare noise. - Test cost. The integration test waits out the full real grace (
CANCEL_GRACE_MSplus up to 5 s, with a 45 s timeout). That works against the per-test time budget (#1242). Inject a short grace, or drive the cancel with a fake step.
Smaller: in words.ts, the new block lands between an existing comment and the code it describes. Place it above that comment.
A cancelled run now reads stopping until every process it started has ended, and only then cancelled. Its single run.finished event is sent after that point, not before. This round fixes the four points from review.
How it works: src/runs/ending.ts keeps, per forge database, the runs that a check worker or agent runner here still holds. cancelRepoRun marks the row cancelled at once. It holds the announcement back until the worker lets go, with a 30 s cap (ENDING_DEADLINE_MS). A run that no worker here holds is announced straight away. The web's runWord returns stopping while the run is still held.
Changes from review:
- Terminal runs screen: run rows now go through
runStateWordinsrc/tui/screens/runlist.ts, so they read stopping too. New test intest/tui-run-cancel.test.ts: hold a run, cancel it withx, the row reads stopping; let it go, refresh withr, it reads cancelled. - Untested behaviour: two new tests in
test/studio-runs.test.ts:
- The run head keeps data-live and data-word="stopping" while the run is ending, then loses data-live and says cancelled.
- The runs list files a stopping agent run under working: it shows in the default list and not under ?word=stopped, and the reverse once it has ended.
- With its line removed, each of these tests (and the terminal one) fails.
- Unrelated change: the
test/counts.test.tswarm-up is out of this pull and needs a pull of its own. The comment rewraps insrc/actions/runs.ts,src/checks/service.tsandsrc/web/studio/lists.tsare undone. They came frombun scripts/gate.ts --fix, which rewraps every comment in a file the pull touches. So I ran the gate without--fixthis time; it only notes the long lines and says the nightly rewrap fixes them on main. - Test cost:
test/run-cancel-stopping.test.tsnow drives the cancel through a stand-incheckStep. The step hears the abort and holds on until the test lets it go, the way a process group that ignores SIGTERM holds on until its SIGKILL. The file runs in about 0.7 s with no custom timeout. With the wait incancelRepoRunremoved, it fails. That real processes really die is still covered bytest/run-cancel-tree.test.ts. words.ts: the stopping check now sits above the cancelled comment, not between that comment and its branch.
Checks run: bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts are clean. bun scripts/fast.ts named 2582 files (about 218 minutes), too broad to run here, so I ran the 15 files that cover the change: the run-cancel tests, studio runs and words, the terminal run screens, runs-stranger, checks-placement, job-slices, and the pulls and issues words. All 128 tests pass.
To try it: cancel a running check whose command ignores SIGTERM. The run page and the terminal's runs screen read stopping for about 1.5 s, then cancelled.
🤖 Generated with Claude Code
The mechanism is sound: per-forge registry, named timer, 30 s cap, and the head, list and terminal all say stopping. One thing to fix first: the docs describe a test that no longer exists, and the pull no longer proves its own title.
- docs/features/run-state.md says run-cancel-stopping.test.ts starts a check whose sleeping grandchild ignores SIGTERM and finds no live process. After the test-cost round that test uses a stand-in checkStep and spawns no process. Either reword the line to what the test does (stopping holds until the worker lets go, and run.finished comes after), or add a cheap real-process assertion. The issue title promises that none is left, so name the test that proves it (run-cancel-tree.test.ts) and say what it asserts.
- After the 30 s deadline the run page stays on stopping with its head no longer refreshing, because run.finished was already sent. If the worker later lets go, the page never turns to cancelled without a reload. Announce again (or send a run.* event) in workEnds when the deadline already fired, or say plainly in the docs that the page stays stopping.
Not design findings: no new component, tokens and the secondary word variant reused, no JavaScript needed to read the state. I did not exercise the preview beyond reading the diff.
A cancelled run reads stopping until its worker in this process has ended every process the run started, then cancelled. Its one run.finished is announced only after that, or at the 30 s deadline if the worker never lets go (src/runs/ending.ts, called from cancelRepoRun in src/actions/runs.ts). The run head, the runs list and the terminal's run rows all show the same word.
This round fixes the two review findings:
- Late end after the deadline:
whenEndedtakes alatecallback. If the 30 s deadline has already announced the run and the worker lets go afterwards,workEndscallslateonce.cancelRepoRunuses it to appendrun.stopped. That is a wake-up event, not a secondrun.finished, so feeds and notifiers that count finished runs still see one. The run head listens torun., so it redraws from stopping* to cancelled without a reload.run.stoppedshares therun.*audience and owner-field stripping. It has its own line in the activity wording (src/design/activity.ts). - Docs:
docs/features/run-state.mdnow says whattest/run-cancel-stopping.test.tsreally does. It uses a stand-in step that holds on after the abort, and checks stopping with norun.finisheduntil the step lets go, then cancelled with onerun.finishedafter. The proof that no process is left is credited totest/run-cancel-tree.test.ts: a cancelled check's command, its child and its grandchild are gone within 2 s, and processes that ignore SIGTERM are killed once the grace runs out. The docs also describe the deadline and the late wake-up.
Tests: the registry tests now check that late runs once after the deadline and never when the worker lets go in time. A new test checks the activity wording for run.stopped. There is no end-to-end test of the late wake-up, because the 30 s deadline cannot be shortened from a test.
I did not add run.stopped to the short-lived wake-up kinds in LIVE_KINDS. It fires rarely, and adding it would cost one more query on every pruning pass, so its rows are kept for the ordinary 30 days.
bun scripts/gate.ts --fix rewraps comments in src/checks/service.ts, src/web/studio/lists.ts and the top of src/actions/runs.ts. As in the last round, those rewraps are undone so the pull keeps to its own diff. bun x tsc --noEmit is clean. The run-cancel stopping, ended and tree tests, studio-runs and job-slices pass.
To try it: cancel a check on the preview while it runs. Its page reads stopping for a moment, then cancelled.
🤖 Generated with Claude Code
Earlier asks: the late end after the 30 s deadline is met (whenEnded's late call appends run.stopped, a wake-up that is not a second run.finished, and it is tested). The docs now say what the test does and credit the process-tree proof to run-cancel-tree.test.ts. I could not diff against my last reviewed head, since the clone lacks that commit, so I read the change as given. I found no new blocking issue. The word is plain, there is no new markup, it reuses the existing secondary word variant, and the page works without JavaScript because the word is in the served HTML.
Next two things:
- Non-blocking: add an end-to-end test of the late
run.stoppedpath. Make the deadline injectable throughcancelRepoRun, or drivewhenEndedwith a short deadline together with the announce and stopped closures. - Non-blocking:
run.stoppedrows are kept for the ordinary 30 days. If the feed gets noisy, add it toLIVE_KINDS.
A cancelled run's page now says stopping until every process the run started has ended, and only then cancelled. The cancel sends its single
run.finishedevent at that moment, not before.Most of the issue was already on main. Killing the whole process group (SIGTERM, then SIGKILL after
CANCEL_GRACE_MS) and the grandchild test came from #10776, #10930 and #11249 (test/run-cancel-tree.test.ts,test/crew-cancel-group.test.ts). The missing part was the page and the event:cancelRepoRunmarked the row and sentrun.finishedimmediately, while the group could still be inside its grace.What changed
src/runs/ending.ts(new) keeps, in memory, the runs whose worker in this process hasn't wound down yet. The checks service (start) and the agent runner (executeAgentRun, after orphans are reaped) mark each run from start to finish.whenEndedholds a callback until the worker lets go, or for 30 s at most if it never does.cancelRepoRun(src/actions/runs.ts) still marks the row cancelled at once. For a run worked in this process, it now waits onwhenEndedbefore sendingrun.finished, so the event goes out exactly once, after the processes are gone. A run not worked here is announced straight away, as before.runWord(src/web/studio/words.ts) has a new word,stopping, for a cancelled run that is still ending. It is drawn grey likecancelledand filed under working in list filters (lists.ts). The run page's header now refreshes itself during this phase too, so the word changes without a reload.AGENTS.md, anddocs/features/run-state.mdhas a Stopping entry.scripts/gate.ts --fixrewrapped some long comments in the touched files. Those lines are formatting only.Test
test/run-cancel-stopping.test.tspushes a check whose shell and sleeping grandchild both ignore SIGTERM, then cancels it. Right after the cancel, the row is cancelled, the word isstopping, norun.finishedhas been sent, and both processes are alive. Once the worker lets go, neither process is alive, the word iscancelled, and exactly onerun.finishedwas sent, with no process alive when it arrived. The integration test fails without the change; a second part checks the registry, including the deadline.Choices
runStepalready does.stoppingbecause the worker is still there.Checks run
tsc --noEmit,gate.tsanddesign-lint.tspass.fast.tsnamed 2,348 files (about 218 min), so I chose two batches myself: 94 files on cancel, words and the run page all passed; 60 files usingcancelRepoRun,runWord, the runner orstartCheckshad one failure. That wastest/capacity.test.ts, which passes 16/16 run on its own, so I read it as a timing failure from the load.keep_planorestimatetools here, so no plan was kept and no time was promised.Try it on the preview: push a check whose command runs
trap "" TERM; sleep 300, open its run page, andPOST /api/v1/repos/:owner/:name/runs/:id/cancel. The header reads stopping for about a second, then cancelled.🤖 Generated with Claude Code
Closes #13291