Public: anyone can read this; the crew can change it.

A cancelled run's child processes are proven gone: a test starts a run that spawns a grandchild, cancels it, and finds no process left #14134

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

mason/forge:mason/14119-a-cancelled-run-s-child-processes-are-pr → main 3 commits fork

  • Merged. mason/14119-a-cancelled-run-s-child-processes-are-pr was merged into main as 11b52b9.

Review

Sign in to review this change.
2 files changed +150 −1
docs/features/run-state.md +1 −1
@@ -9,4 +9,4 @@
99 - `GET …/runs?state=queued|running|ok|failed|cancelled` filters; an unknown state is a 422.
1010 - **In the terminal**: the issue screen (`src/tui/screens/issue.ts`, rows from `src/tui/screens/runplan.ts`) shows a *Plan* card above the thread — `[x]`/`[ ]` items and the question as `needs input: …` — read through the same `readPlan` (so gated as the page is), and its title says *silent* or *needs input* when the issue's runs read that way, from `shownStates`.
1111 - The runs API works out `shown` for a page of runs with one plan read (`shownStates`), not one per row.
12 - **Stopping**: a cancel marks the row at once, but the run's page says *stopping* (a moving word, filed under *working*) until its worker here has ended every process the run started — SIGTERM to the group, SIGKILL once the grace runs out, an agent's orphans reaped — and only then *cancelled*; the cancel's one `run.finished` is announced at that moment, not before (`src/runs/ending.ts`, #13291; `test/run-cancel-stopping.test.ts` cancels a check whose stand-in step holds on after the abort, and finds the word *stopping* and no `run.finished` until the step lets go, then *cancelled* and the one `run.finished` after). That none of the processes is left is `test/run-cancel-tree.test.ts`'s: a cancelled check's command, the child it forked and that child's child are all gone within 2 s, and a step and the sleeper it forked that both ignore SIGTERM are killed once the grace runs out. A worker that never lets go is announced at the 30 s deadline (`ENDING_DEADLINE_MS`) while the page still says *stopping*; when it lets go later, a `run.stopped` wake-up (not a second `run.finished`) turns the live head to *cancelled*.
12 - **Stopping**: a cancel marks the row at once, but the run's page says *stopping* (a moving word, filed under *working*) until its worker here has ended every process the run started — SIGTERM to the group, SIGKILL once the grace runs out, an agent's orphans reaped — and only then *cancelled*; the cancel's one `run.finished` is announced at that moment, not before (`src/runs/ending.ts`, #13291; `test/run-cancel-stopping.test.ts` cancels a check whose stand-in step holds on after the abort, and finds the word *stopping* and no `run.finished` until the step lets go, then *cancelled* and the one `run.finished` after). That none of the processes is left is `test/run-cancel-tree.test.ts`'s: a cancelled check's command, the child it forked and that child's child are all gone within 2 s, and a step and the sleeper it forked that both ignore SIGTERM are killed once the grace runs out. That the whole process group is empty, asked of the kernel and not pid by pid, is `test/run-cancel.test.ts`'s (#14119): a cancelled step whose child forked a sleeper leaves no member of its group within 2 s, a sleeper deaf to SIGTERM included, and where the sandbox cannot signal a group the suite is skipped under a name that says why. A worker that never lets go is announced at the 30 s deadline (`ENDING_DEADLINE_MS`) while the page still says *stopping*; when it lets go later, a `run.stopped` wake-up (not a second `run.finished`) turns the live head to *cancelled*.
test/run-cancel.test.ts added +149
@@ -0,0 +1,149 @@
1 /**
2 * A cancelled run's process group is proven gone (#14119). A run's step
3 * leads a process group of its own (`apps/process.ts` `runStep`,
4 * `detached`); its command forks a child, and that child forks a sleeper.
5 * A cancel goes the way every surface's does past the row
6 * (`runs/stoppable.ts` `stopRun` aborts the run's controller), and within
7 * 2 s not one member of the group is left: `kill(-pgid, 0)` answers ESRCH.
8 *
9 * `test/run-cancel-tree.test.ts` follows the three pids it was told of;
10 * this asks the kernel about the group itself, so a member the test never
11 * heard of cannot hide. The second case is a sleeper that ignores SIGTERM
12 * under shells that honour it: the leader ends at once, and the group's
13 * SIGKILL on its exit takes the sleeper, well inside the 2 s and long
14 * before any grace.
15 *
16 * Only the group the test created is ever signalled. Where this machine
17 * cannot signal a process group (no /proc, or a sandbox that refuses it),
18 * the suite is skipped and its name says why.
19 */
20 import { afterAll, describe, expect, test } from "bun:test";
21 import { existsSync, readFileSync } from "node:fs";
22 import { mkdtemp, rm } from "node:fs/promises";
23 import { tmpdir } from "node:os";
24 import { join } from "node:path";
25 import { childEnv, runStep } from "../src/apps/process.ts";
26 import { runningRun, stopRun, trackRun, untrackRun } from "../src/runs/stoppable.ts";
27
28 /** Why this machine cannot run the proof, or undefined when it can. */
29 async function cannotSignal(): Promise<string | undefined> {
30 if (process.platform !== "linux") return `no /proc to read on ${process.platform}`;
31 if (!existsSync("/proc/self/stat")) return "no /proc mounted";
32 const probe = Bun.spawn(["/bin/sh", "-c", "sleep 30"], { detached: true, stdout: "ignore", stderr: "ignore" });
33 try {
34 process.kill(-probe.pid, 0);
35 process.kill(-probe.pid, "SIGKILL");
36 } catch (cause) {
37 probe.kill("SIGKILL");
38 const code = (cause as { code?: string }).code ?? String(cause);
39 return `the sandbox cannot signal a process group (${code})`;
40 }
41 await probe.exited;
42 return undefined;
43 }
44
45 const reason = await cannotSignal();
46 const dir = await mkdtemp(join(tmpdir(), "forge-run-cancel-"));
47 const groups: number[] = [];
48
49 afterAll(async () => {
50 // A red run must not leave its sleepers behind: only the groups this test made.
51 for (const pgid of groups)
52 try {
53 process.kill(-pgid, "SIGKILL");
54 } catch {
55 // gone, as it should be
56 }
57 await rm(dir, { recursive: true, force: true });
58 });
59
60 /** True while any process is in group `pgid`. */
61 function groupAlive(pgid: number): boolean {
62 try {
63 process.kill(-pgid, 0);
64 return true;
65 } catch {
66 return false;
67 }
68 }
69
70 /** Polls, never a fixed sleep: true once `done` holds, false at the deadline. */
71 async function within(ms: number, done: () => boolean): Promise<boolean> {
72 const deadline = Date.now() + ms;
73 while (!done() && Date.now() < deadline) {
74 await Bun.sleep(20); // real-timer: polls real processes the kernel ends
75 }
76 return done();
77 }
78
79 async function pidIn(file: string, ms: number): Promise<number> {
80 const named = (): number => (existsSync(file) ? Number(readFileSync(file, "utf8").trim()) : 0);
81 await within(ms, () => Number.isInteger(named()) && named() > 1);
82 const pid = named();
83 if (!Number.isInteger(pid) || pid <= 1) throw new Error(`${file} never named a pid`);
84 return pid;
85 }
86
87 /** The group a pid is in, read from /proc; undefined once it is gone. */
88 function groupOf(pid: number): number | undefined {
89 try {
90 const stat = readFileSync(`/proc/${pid}/stat`, "utf8");
91 return Number(stat.slice(stat.lastIndexOf(")") + 2).split(" ")[2]);
92 } catch {
93 return undefined;
94 }
95 }
96
97 /**
98 * Start a tracked run whose step writes its own pid (the group's id, since
99 * it leads the group), forks a shell, and that shell forks `sleeper`; once
100 * the sleeper's pid is on disk, all three exist. Cancel it and say whether
101 * the group emptied within 2 s.
102 */
103 async function cancelTree(name: string, sleeper: string, trap = ""): Promise<{ gone: boolean; error?: string }> {
104 const lead = join(dir, `${name}.lead`);
105 const grand = join(dir, `${name}.grand`);
106 const command =
107 `${trap}echo $$ > ${lead}; ` +
108 `sh -c '${sleeper} & echo $! > ${grand}.tmp; mv ${grand}.tmp ${grand}; wait' & ` +
109 `wait`;
110 const runId = `run-${name}`;
111 const controller = new AbortController();
112 trackRun(runId, controller);
113 const step = runStep("check", command, { cwd: dir, env: childEnv({}), log: () => {}, signal: controller.signal });
114 try {
115 const sleeperPid = await pidIn(grand, 10_000);
116 const pgid = await pidIn(lead, 1_000);
117 groups.push(pgid);
118 // The sleeper is in the group the step leads, and that group is not ours.
119 expect(groupOf(sleeperPid)).toBe(pgid);
120 expect(pgid).not.toBe(process.pid);
121 expect(groupAlive(pgid)).toBe(true);
122
123 expect(stopRun(runId)).toBe(true);
124 const gone = await within(2_000, () => !groupAlive(pgid));
125 const result = await step;
126 return { gone, ...(result.ok ? {} : { error: result.error }) };
127 } finally {
128 untrackRun(runId, controller);
129 }
130 }
131
132 describe.skipIf(reason !== undefined)(
133 `a cancelled run${reason === undefined ? "" : ` (skipped: ${reason})`}`,
134 () => {
135 test("leaves no member of its process group within 2 s: the command, its child, and the sleeper", async () => {
136 const ended = await cancelTree("plain", "sleep 300");
137 expect(ended.gone).toBe(true);
138 expect(ended.error).toBe("check cancelled");
139 expect(runningRun("run-plain")).toBe(false);
140 }, 20_000);
141
142 test("takes a sleeper that ignores SIGTERM when its shells end on it, within 2 s", async () => {
143 // The sleeper sets its own `trap '' TERM`; the shells above it keep the default.
144 const ended = await cancelTree("deaf", "sh -c \"trap \\\"\\\" TERM; sleep 300 & wait\"");
145 expect(ended.gone).toBe(true);
146 expect(ended.error).toBe("check cancelled");
147 }, 20_000);
148 },
149 );

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs