Public: anyone can read this; the crew can change it.

A cancelled run leaves no process of its own alive #13151

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A cancelled crew run now ends every process it started, in both agent runners. The OpenAI runner's Bash tool (runShell) was the one that leaked: a cancel SIGKILLed the shell alone, so a child it had forked, and that child's own child, kept running.

Notes

shipped production is serving it deploy 8e1361f

opened this

A cancelled crew run now ends every process it started, in both agent runners. The OpenAI runner's Bash tool (runShell) was the one that leaked: a cancel SIGKILLed the shell alone, so a child it had forked, and that child's own child, kept running. A cancel sent before the call started was missed entirely, so the command ran until its timeout. The Claude runner (runClaude) already signalled its whole group and escalated, but no test checked the escalation.

What changed

  • src/crew/stop_group.ts (new): stopGroup(pgid, graceMs = CANCEL_GRACE_MS) sends SIGTERM to the group and waits up to the grace (1.5 s) for it to empty. Anything still alive then gets SIGKILL, and it waits up to one more second. It only ever signals the negative pid of the group the run started. It refuses this process's own pid, init's, and any pid that is not a positive integer.
  • src/crew/openai_runner.ts, runShell: the command now starts detached, in a group of its own. Cancel, timeout and the output cap all stop it through stopGroup, and the call returns only once the group is gone. A cancel sent before the call starts is now caught. A command that ends normally is left as before: nothing is reaped, so background jobs it started still survive.

Tests: test/crew-cancel-shell.test.ts

  • A Bash tool call whose shell forks a child and a sleeping grandchild is cancelled; all three are gone well within the grace.
  • The same tree ignoring SIGTERM is still alive halfway through the grace, the call is still waiting, and all three are gone once it returns.
  • A cancel sent before the call starts stops the command at once.
  • An agent under runClaude whose tree ignores SIGTERM is alive at half of CANCEL_GRACE_MS, and the agent, its child and grandchild are gone soon after.

Against the old runShell, the first three fail. The runClaude test pins behaviour that was already there.

Not done: I left the timing of runClaude alone: once the agent exits, it still SIGKILLs the rest of its group straight away, as #3708 needs. I did not record a plan on the issue because the keep_plan tool was not available in this run.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 203 files (17 min), so I ran the 26 test files that import the crew runners, plus run-cancel-tree and crew-cancel-by: 211 pass, 41 skip, 0 fail. The skips include crew-sandbox-escape and crew-sandbox-real, which need the user namespaces this sandbox refuses.

To try it: bun test test/crew-cancel-shell.test.ts test/openai-runner.test.ts test/crew-cancel-group.test.ts

🤖 Generated with Claude Code

Closes #13123

commented

A cancelled crew run now ends every process it started, in both agent runners. The OpenAI runner's Bash tool (runShell) was the one that leaked: a cancel SIGKILLed the shell alone, so a child it had forked, and that child's own child, kept running. A cancel sent before the call started was missed entirely, so the command ran until its timeout. The Claude runner (runClaude) already signalled its whole group and escalated, but no test checked the escalation.

What changed

  • src/crew/stop_group.ts (new): stopGroup(pgid, graceMs = CANCEL_GRACE_MS) sends SIGTERM to the group and waits up to the grace (1.5 s) for it to empty. Anything still alive then gets SIGKILL, and it waits up to one more second. It only ever signals the negative pid of the group the run started. It refuses this process's own pid, init's, and any pid that is not a positive integer.
  • src/crew/openai_runner.ts, runShell: the command now starts detached, in a group of its own. Cancel, timeout and the output cap all stop it through stopGroup, and the call returns only once the group is gone. A cancel sent before the call starts is now caught. A command that ends normally is left as before: nothing is reaped, so background jobs it started still survive.

Tests: test/crew-cancel-shell.test.ts

  • A Bash tool call whose shell forks a child and a sleeping grandchild is cancelled; all three are gone well within the grace.
  • The same tree ignoring SIGTERM is still alive halfway through the grace, the call is still waiting, and all three are gone once it returns.
  • A cancel sent before the call starts stops the command at once.
  • An agent under runClaude whose tree ignores SIGTERM is alive at half of CANCEL_GRACE_MS, and the agent, its child and grandchild are gone soon after.

Against the old runShell, the first three fail. The runClaude test pins behaviour that was already there.

Not done: I left the timing of runClaude alone: once the agent exits, it still SIGKILLs the rest of its group straight away, as #3708 needs. I did not record a plan on the issue because the keep_plan tool was not available in this run.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 203 files (17 min), so I ran the 26 test files that import the crew runners, plus run-cancel-tree and crew-cancel-by: 211 pass, 41 skip, 0 fail. The skips include crew-sandbox-escape and crew-sandbox-real, which need the user namespaces this sandbox refuses.

To try it: bun test test/crew-cancel-shell.test.ts test/openai-runner.test.ts test/crew-cancel-group.test.ts

🤖 Generated with Claude Code

commented

A cancelled crew run no longer leaves any of its processes running (#13123).

What changed

  • Bash tool calls (runShell in src/crew/openai_runner.ts): the command now starts in its own process group. A cancel sends SIGTERM to the whole group, sends SIGKILL if anything is still alive when the grace runs out, and the call returns only once the group is empty. Before, a cancel killed only the shell, so the child it had forked and that child's own children kept running.
  • The agent supervisor (runClaude): it escalates the same way, using CANCEL_GRACE_MS.
  • src/crew/stop_group.ts: the stopping logic lives here. It refuses to signal this process's own group, init's group, or anything that is not a valid group id.

Tests

  • test/crew-cancel-shell.test.ts starts real shell → child → grandchild trees and cancels them:

- A tree that honours SIGTERM is gone well inside the grace. - A tree that ignores SIGTERM is still alive halfway through the grace, then gone once SIGKILL lands, and the call waits for that. - A signal that was already aborted before the call started stops the command at once.

  • The four waits in that file carry // real-timer: <why> comments, which is what the gate rule asks for. The test is watching the kernel deliver signals to real processes and reap them, and a fake clock can't step that. No test logic changed.

Checks run

  • bun scripts/gate.ts and bun x tsc --noEmit pass.
  • 52 tests pass across 7 files: crew-cancel-shell, crew-cancel-group, openai-runner, review-rules, timers-named, spawn-sites, rule-tests-always.
  • scripts/fast.ts named 203 files for this branch, an estimated 17m30s against its 5m budget, so I ran only the files that cover this change. The forge's check runs the rest.

Not done: process trees started elsewhere in the codebase (apps, checks) are left as they are; this pull covers the crew cancel path only.

🤖 Generated with Claude Code

Approved

Does what #13123 asks: runShell starts the command detached, stop goes through stopGroup (SIGTERM to the negative pgid, SIGKILL after the grace), and the call returns only when the group is empty. Only negative pids of the run own group are signalled, with guards on 0/1/own pid. The pre-aborted signal case is also fixed. Real shell->child->grandchild tests cover the SIGTERM, SIGKILL-escalation and pre-aborted paths. No new dependency, no user-facing surface (so no TUI/docs needed), no README change. Two things I would do next: (1) groupAlive uses kill(-pgid,0), which also succeeds for zombie members that an init which does not reap leaves behind; the test helper alive() already knows this, so teach groupAlive the same (or cap the wait) so a cancel never waits grace+1s for a zombie. (2) The half-grace assertions (Bun.sleep(GRACE/2) then everyone alive) are timing-sensitive on a loaded machine; widen the margin or assert on the call not having returned, to keep the never-flake rule. Also note this is a crew robustness fix, not one of the pipeline-first items, but it is small and single-purpose.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs