Public: anyone can read this; the crew can change it.
A cancelled run's child processes are proven gone: a test starts a run that spawns a grandchild, cancels it, and finds no process left #14134
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/14119-a-cancelled-run-s-child-processes-are-pr → main 3 commits fork
2 files
changed +150
−1
docs/features/run-state.md +1 −1
| @@ -9,4 +9,4 @@ | ||
| 9 | 9 | - `GET …/runs?state=queued|running|ok|failed|cancelled` filters; an unknown state is a 422. |
| 10 | 10 | - **In the terminal**: the issue screen (`src/tui/screens/issue.ts`, rows from `src/tui/screens/runplan.ts`) shows a *Plan* card above the thread — `[x]`/`[ ]` items and the question as `needs input: …` — read through the same `readPlan` (so gated as the page is), and its title says *silent* or *needs input* when the issue's runs read that way, from `shownStates`. |
| 11 | 11 | - The runs API works out `shown` for a page of runs with one plan read (`shownStates`), not one per row. |
| 12 | - **Stopping**: a cancel marks the row at once, but the run's page says *stopping* (a moving word, filed under *working*) until its worker here has ended every process the run started — SIGTERM to the group, SIGKILL once the grace runs out, an agent's orphans reaped — and only then *cancelled*; the cancel's one `run.finished` is announced at that moment, not before (`src/runs/ending.ts`, #13291; `test/run-cancel-stopping.test.ts` cancels a check whose stand-in step holds on after the abort, and finds the word *stopping* and no `run.finished` until the step lets go, then *cancelled* and the one `run.finished` after). That none of the processes is left is `test/run-cancel-tree.test.ts`'s: a cancelled check's command, the child it forked and that child's child are all gone within 2 s, and a step and the sleeper it forked that both ignore SIGTERM are killed once the grace runs out. A worker that never lets go is announced at the 30 s deadline (`ENDING_DEADLINE_MS`) while the page still says *stopping*; when it lets go later, a `run.stopped` wake-up (not a second `run.finished`) turns the live head to *cancelled*. | |
| 12 | - **Stopping**: a cancel marks the row at once, but the run's page says *stopping* (a moving word, filed under *working*) until its worker here has ended every process the run started — SIGTERM to the group, SIGKILL once the grace runs out, an agent's orphans reaped — and only then *cancelled*; the cancel's one `run.finished` is announced at that moment, not before (`src/runs/ending.ts`, #13291; `test/run-cancel-stopping.test.ts` cancels a check whose stand-in step holds on after the abort, and finds the word *stopping* and no `run.finished` until the step lets go, then *cancelled* and the one `run.finished` after). That none of the processes is left is `test/run-cancel-tree.test.ts`'s: a cancelled check's command, the child it forked and that child's child are all gone within 2 s, and a step and the sleeper it forked that both ignore SIGTERM are killed once the grace runs out. That the whole process group is empty, asked of the kernel and not pid by pid, is `test/run-cancel.test.ts`'s (#14119): a cancelled step whose child forked a sleeper leaves no member of its group within 2 s, a sleeper deaf to SIGTERM included, and where the sandbox cannot signal a group the suite is skipped under a name that says why. A worker that never lets go is announced at the 30 s deadline (`ENDING_DEADLINE_MS`) while the page still says *stopping*; when it lets go later, a `run.stopped` wake-up (not a second `run.finished`) turns the live head to *cancelled*. |
test/run-cancel.test.ts added +149
| @@ -0,0 +1,149 @@ | ||
| 1 | /** | |
| 2 | * A cancelled run's process group is proven gone (#14119). A run's step | |
| 3 | * leads a process group of its own (`apps/process.ts` `runStep`, | |
| 4 | * `detached`); its command forks a child, and that child forks a sleeper. | |
| 5 | * A cancel goes the way every surface's does past the row | |
| 6 | * (`runs/stoppable.ts` `stopRun` aborts the run's controller), and within | |
| 7 | * 2 s not one member of the group is left: `kill(-pgid, 0)` answers ESRCH. | |
| 8 | * | |
| 9 | * `test/run-cancel-tree.test.ts` follows the three pids it was told of; | |
| 10 | * this asks the kernel about the group itself, so a member the test never | |
| 11 | * heard of cannot hide. The second case is a sleeper that ignores SIGTERM | |
| 12 | * under shells that honour it: the leader ends at once, and the group's | |
| 13 | * SIGKILL on its exit takes the sleeper, well inside the 2 s and long | |
| 14 | * before any grace. | |
| 15 | * | |
| 16 | * Only the group the test created is ever signalled. Where this machine | |
| 17 | * cannot signal a process group (no /proc, or a sandbox that refuses it), | |
| 18 | * the suite is skipped and its name says why. | |
| 19 | */ | |
| 20 | import { afterAll, describe, expect, test } from "bun:test"; | |
| 21 | import { existsSync, readFileSync } from "node:fs"; | |
| 22 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 23 | import { tmpdir } from "node:os"; | |
| 24 | import { join } from "node:path"; | |
| 25 | import { childEnv, runStep } from "../src/apps/process.ts"; | |
| 26 | import { runningRun, stopRun, trackRun, untrackRun } from "../src/runs/stoppable.ts"; | |
| 27 | ||
| 28 | /** Why this machine cannot run the proof, or undefined when it can. */ | |
| 29 | async function cannotSignal(): Promise<string | undefined> { | |
| 30 | if (process.platform !== "linux") return `no /proc to read on ${process.platform}`; | |
| 31 | if (!existsSync("/proc/self/stat")) return "no /proc mounted"; | |
| 32 | const probe = Bun.spawn(["/bin/sh", "-c", "sleep 30"], { detached: true, stdout: "ignore", stderr: "ignore" }); | |
| 33 | try { | |
| 34 | process.kill(-probe.pid, 0); | |
| 35 | process.kill(-probe.pid, "SIGKILL"); | |
| 36 | } catch (cause) { | |
| 37 | probe.kill("SIGKILL"); | |
| 38 | const code = (cause as { code?: string }).code ?? String(cause); | |
| 39 | return `the sandbox cannot signal a process group (${code})`; | |
| 40 | } | |
| 41 | await probe.exited; | |
| 42 | return undefined; | |
| 43 | } | |
| 44 | ||
| 45 | const reason = await cannotSignal(); | |
| 46 | const dir = await mkdtemp(join(tmpdir(), "forge-run-cancel-")); | |
| 47 | const groups: number[] = []; | |
| 48 | ||
| 49 | afterAll(async () => { | |
| 50 | // A red run must not leave its sleepers behind: only the groups this test made. | |
| 51 | for (const pgid of groups) | |
| 52 | try { | |
| 53 | process.kill(-pgid, "SIGKILL"); | |
| 54 | } catch { | |
| 55 | // gone, as it should be | |
| 56 | } | |
| 57 | await rm(dir, { recursive: true, force: true }); | |
| 58 | }); | |
| 59 | ||
| 60 | /** True while any process is in group `pgid`. */ | |
| 61 | function groupAlive(pgid: number): boolean { | |
| 62 | try { | |
| 63 | process.kill(-pgid, 0); | |
| 64 | return true; | |
| 65 | } catch { | |
| 66 | return false; | |
| 67 | } | |
| 68 | } | |
| 69 | ||
| 70 | /** Polls, never a fixed sleep: true once `done` holds, false at the deadline. */ | |
| 71 | async function within(ms: number, done: () => boolean): Promise<boolean> { | |
| 72 | const deadline = Date.now() + ms; | |
| 73 | while (!done() && Date.now() < deadline) { | |
| 74 | await Bun.sleep(20); // real-timer: polls real processes the kernel ends | |
| 75 | } | |
| 76 | return done(); | |
| 77 | } | |
| 78 | ||
| 79 | async function pidIn(file: string, ms: number): Promise<number> { | |
| 80 | const named = (): number => (existsSync(file) ? Number(readFileSync(file, "utf8").trim()) : 0); | |
| 81 | await within(ms, () => Number.isInteger(named()) && named() > 1); | |
| 82 | const pid = named(); | |
| 83 | if (!Number.isInteger(pid) || pid <= 1) throw new Error(`${file} never named a pid`); | |
| 84 | return pid; | |
| 85 | } | |
| 86 | ||
| 87 | /** The group a pid is in, read from /proc; undefined once it is gone. */ | |
| 88 | function groupOf(pid: number): number | undefined { | |
| 89 | try { | |
| 90 | const stat = readFileSync(`/proc/${pid}/stat`, "utf8"); | |
| 91 | return Number(stat.slice(stat.lastIndexOf(")") + 2).split(" ")[2]); | |
| 92 | } catch { | |
| 93 | return undefined; | |
| 94 | } | |
| 95 | } | |
| 96 | ||
| 97 | /** | |
| 98 | * Start a tracked run whose step writes its own pid (the group's id, since | |
| 99 | * it leads the group), forks a shell, and that shell forks `sleeper`; once | |
| 100 | * the sleeper's pid is on disk, all three exist. Cancel it and say whether | |
| 101 | * the group emptied within 2 s. | |
| 102 | */ | |
| 103 | async function cancelTree(name: string, sleeper: string, trap = ""): Promise<{ gone: boolean; error?: string }> { | |
| 104 | const lead = join(dir, `${name}.lead`); | |
| 105 | const grand = join(dir, `${name}.grand`); | |
| 106 | const command = | |
| 107 | `${trap}echo $$ > ${lead}; ` + | |
| 108 | `sh -c '${sleeper} & echo $! > ${grand}.tmp; mv ${grand}.tmp ${grand}; wait' & ` + | |
| 109 | `wait`; | |
| 110 | const runId = `run-${name}`; | |
| 111 | const controller = new AbortController(); | |
| 112 | trackRun(runId, controller); | |
| 113 | const step = runStep("check", command, { cwd: dir, env: childEnv({}), log: () => {}, signal: controller.signal }); | |
| 114 | try { | |
| 115 | const sleeperPid = await pidIn(grand, 10_000); | |
| 116 | const pgid = await pidIn(lead, 1_000); | |
| 117 | groups.push(pgid); | |
| 118 | // The sleeper is in the group the step leads, and that group is not ours. | |
| 119 | expect(groupOf(sleeperPid)).toBe(pgid); | |
| 120 | expect(pgid).not.toBe(process.pid); | |
| 121 | expect(groupAlive(pgid)).toBe(true); | |
| 122 | ||
| 123 | expect(stopRun(runId)).toBe(true); | |
| 124 | const gone = await within(2_000, () => !groupAlive(pgid)); | |
| 125 | const result = await step; | |
| 126 | return { gone, ...(result.ok ? {} : { error: result.error }) }; | |
| 127 | } finally { | |
| 128 | untrackRun(runId, controller); | |
| 129 | } | |
| 130 | } | |
| 131 | ||
| 132 | describe.skipIf(reason !== undefined)( | |
| 133 | `a cancelled run${reason === undefined ? "" : ` (skipped: ${reason})`}`, | |
| 134 | () => { | |
| 135 | test("leaves no member of its process group within 2 s: the command, its child, and the sleeper", async () => { | |
| 136 | const ended = await cancelTree("plain", "sleep 300"); | |
| 137 | expect(ended.gone).toBe(true); | |
| 138 | expect(ended.error).toBe("check cancelled"); | |
| 139 | expect(runningRun("run-plain")).toBe(false); | |
| 140 | }, 20_000); | |
| 141 | ||
| 142 | test("takes a sleeper that ignores SIGTERM when its shells end on it, within 2 s", async () => { | |
| 143 | // The sleeper sets its own `trap '' TERM`; the shells above it keep the default. | |
| 144 | const ended = await cancelTree("deaf", "sh -c \"trap \\\"\\\" TERM; sleep 300 & wait\""); | |
| 145 | expect(ended.gone).toBe(true); | |
| 146 | expect(ended.error).toBe("check cancelled"); | |
| 147 | }, 20_000); | |
| 148 | }, | |
| 149 | ); |
new above
offline