Public: anyone can read this; the crew can change it.

A cell child cannot steer its adoption: cell records live outside the child's directory and adoption checks pid start time and the supervisor's port #15237

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

mason/forge:mason/15235-a-cell-child-cannot-steer-its-adoption-c → main 2 commits fork

  • Merged. mason/15235-a-cell-child-cannot-steer-its-adoption-c was merged into main as e92b8be.

Review

Sign in to review this change.
8 files changed +284 −103
docs/features/cells-outlive.md +5 −2
@@ -6,12 +6,15 @@ row: Cells that outlive the host, and are taken back on start
66
77 On when the host boots (#13750), so a production release leaves a hosted forge's checks and runs running; set `FORGE_CELLS_OUTLIVE=off` to tie cells to the host again. With it on, a child's cell runs its proxy and relays in a supervisor of its own and is not tied to this process: stopping the host leaves the cell running (#13755).
88
9 **On start the host takes its cells back (#13756).** Each supervised cell writes `children/<body>/cell.json` (pid, port, supervisor control directory, start time, name, size) as soon as it is spawned, and clears it when it exits. At boot the host reads every record and, for each:
9 **On start the host takes its cells back (#13756).** For each supervised cell the host writes a record, `<data>/cell-records/<body>.json` (pid, port, supervisor control directory, start time, name, size; `spawn/cell_record.ts`), as soon as it is spawned, and clears it when it exits. That directory is the parent's and is bound into no cell, and a record path takes only a safe body id, so a child cannot write or name a record (#15235). A `cell.json` an older host left in a child's own directory is removed at boot without being read (`dropLegacyRecords`), and the log says so for each. At boot the host reads every record and, for each:
1010
11 - the pid must still be the cell its supervisor names (`cell.pid` in the control directory, pid and start time), so a reused pid is never adopted or signalled;
11 - the pid must still be the cell its supervisor names: `cell.pid` in the control directory must carry the pid and its start time, and that start time must be the running process's (`readStartedPid`). A bare pid, or a reused one, is never adopted or signalled;
12 - the port must be the one the supervisor relays the cell's way in on (`config.json` `listen`, `readListen`). A record naming any other port is dropped without that port being asked anything;
1213 - the port must answer. A cell that is its supervisor's but silent is stopped and its record cleared, since nothing else would reap it;
1314 - otherwise the cell is held in the registry as if this host had started it. A start for the same body waits for adoption to finish and finds the adopted cell, never a second one. The record is cleared when the cell goes.
1415
1516 Not yet: the beat door still runs in the host, egress bytes of an outliving cell are not in the host's gauges (#13921).
1617
1718 **What the host did is shown.** For this host's life it keeps each cell it adopted or dropped at boot (`spawn/adopted.ts`: name, pid, when, why). /system lists them under "Child forges across the restart" ("kept running across the restart", "stopped: did not answer"), and nothing when there were none; the MCP `system` tool (`kind: adopted`) and the terminal's System screen read the same rows.
19
20 **Upgrading from a host that kept records in the child's directory.** The first boot after #15235 finds no record in `cell-records/`, so cells the older host left running are neither adopted nor signalled: each is started afresh when next asked, and the old cell ends with its supervisor. Every restart after that adopts the cells it recorded.
src/kernel/destroy_reach.ts +5 −1
@@ -95,7 +95,11 @@ export const ALLOWED_REMOVALS: readonly { unit: string; why: string }[] = [
9595 { unit: "src/apps/contain.ts:openCell", why: "clears the cell's socket directory before the cell opens" },
9696 { unit: "src/spawn/supervisor.ts:startSupervisor", why: "removes the cell's own control directory (a fresh temp directory it made) when the host closes the cell" },
9797 { unit: "src/spawn/supervisor.ts:runSupervisor", why: "removes the cell's own socket directory and, when no host is left, its control directory: both fresh temp directories made for that cell" },
98 { unit: "src/spawn/cell_record.ts:clearRecord", why: "removes the one cell.json a running cell left in its own child directory, never the directory or anything else in it" },
98 { unit: "src/spawn/cell_record.ts:clearRecord", why: "removes the one record file of a cell, `<data>/cell-records/<body>.json` for a checked body id, never the directory or anything else in it" },
99 {
100 unit: "src/spawn/cell_record.ts:dropLegacyRecords",
101 why: "removes the one cell.json an older host left in each child's directory, a file the child could write and so never read, never the directory or anything else in it",
102 },
99103 { unit: "src/apps/cgroup.ts:removeDir", why: "removes an empty cgroup directory" },
100104 { unit: "src/actions/spawn.ts:unmake", why: "removes the directory and ledger of a child whose first attempt failed before it was ever finished, and only that call's own (never a finished or resumed one): nothing of anyone's work is in it (#13135)" },
101105 { unit: "src/spawn/own_code.ts:checkout", why: "removes a half-made export of the forge's own code, keyed by commit" },
src/spawn/cell.ts +19 −12
@@ -27,9 +27,9 @@ import { childModelHosts, childPaymentHosts, childRegistryHosts } from "../core/
2727 import { liveChildDisk, type ChildDisk } from "./disk_mount.ts";
2828 import { oneAtATime, sourcesFor, startFailure, startFirstThatWorks, useTree, type CodeSource } from "./own_code.ts";
2929 import { childGate, type ChildGate } from "./child_reach.ts";
30 import { files, readPid } from "./supervisor.ts";
30 import { files, readListen, readStartedPid } from "./supervisor.ts";
3131 import { adoptionRows, type Adoption, type AdoptionRow } from "./adopted.ts";
32 import { clearRecord, leaveRecord, readRecords, type CellRecord } from "./cell_record.ts";
32 import { clearRecord, dropLegacyRecords, leaveRecord, readRecords, type CellRecord } from "./cell_record.ts";
3333
3434 /**
3535 * Whether a child's cell outlives this process (`FORGE_CELLS_OUTLIVE=on`; #13755): its proxy and
@@ -462,13 +462,18 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
462462 * A cell a previous host left running: its pid is watched, since this process did not start it.
463463 */
464464 const boot: Adoption[] = [];
465 const adopt = async (record: CellRecord, dir: string): Promise<void> => {
465 const dataDir = parent.config.dataDir;
466 const adopt = async (record: CellRecord): Promise<void> => {
467 const drop = (): void => clearRecord(dataDir, record.bodyId);
466468 // The pid alone can be another process's by now: it is the cell only when its supervisor
467 // still names it (pid and start time, `readPid`). A pid that fails this is never signalled.
468 const isCell = (): boolean => alive(record.pid) && readPid(files(record.supervisor).cell) === record.pid;
469 if (!isCell()) {
469 // still names it with the start time that process has (`readStartedPid`). A pid that fails
470 // this is never signalled.
471 const isCell = (): boolean => alive(record.pid) && readStartedPid(files(record.supervisor).cell) === record.pid;
472 // The port is the supervisor's word too (`config.json`), never the record's alone: a record
473 // naming another port is dropped before that port is asked anything.
474 if (!isCell() || readListen(record.supervisor) !== record.port) {
470475 boot.push({ name: record.name, pid: record.pid, at: Date.now(), outcome: "dropped", why: "not-its-cell" });
471 clearRecord(dir);
476 drop();
472477 return;
473478 }
474479 if (!(await answers(record.port, Promise.resolve(), 3))) {
@@ -476,7 +481,7 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
476481 if (isCell()) killTree(record.pid);
477482 boot.push({ name: record.name, pid: record.pid, at: Date.now(), outcome: "dropped", why: "silent" });
478483 parent.log.warn("child forge left by a previous host did not answer; stopped", { name: record.name, pid: record.pid });
479 clearRecord(dir);
484 drop();
480485 return;
481486 }
482487 // Watched only now it is held: the timer ends when the cell is no longer the one recorded.
@@ -495,14 +500,16 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
495500 boot.push({ name: record.name, pid: record.pid, at: Date.now(), outcome: "kept", why: null });
496501 parent.log.info("child forge adopted", { name: record.name, pid: record.pid });
497502 void gone.then(() => {
498 clearRecord(dir);
503 drop();
499504 if (held.get(record.bodyId) === child) {
500505 held.delete(record.bodyId);
501506 startedAt.delete(record.bodyId);
502507 }
503508 });
504509 };
505 const adopted = Promise.all(readRecords(join(parent.config.dataDir, "children")).map(({ record, dir }) => adopt(record, dir))).then(() => {});
510 // What an older host left in a child's own directory is the child's to write: never read.
511 dropLegacyRecords(dataDir, (bodyId) => parent.log.warn("child forge's own cell.json removed unread", { bodyId }));
512 const adopted = Promise.all(readRecords(dataDir).map(adopt)).then(() => {});
506513 let adoptionOver = false;
507514 void adopted.finally(() => (adoptionOver = true)).catch(() => {});
508515
@@ -547,7 +554,7 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
547554 noteStop(parent, input.bodyId, input.name, why);
548555 }
549556 cell.close();
550 clearRecord(dir);
557 clearRecord(dataDir, input.bodyId);
551558 door.close();
552559 release();
553560 if (why !== undefined) recent.push(why);
@@ -556,7 +563,7 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
556563 cell.watch(proc.pid, () => killTree(proc.pid));
557564 // Written before anything awaits, so the exit hook above (which clears it) can only come
558565 // after; a host that dies before the cell answers leaves a record the next one reaps.
559 leaveRecord(dir, cell.supervised, { bodyId: input.bodyId, pid: proc.pid, port, startedAt: Date.now(), name: input.name, size: input.size }, (why) =>
566 leaveRecord(dataDir, cell.supervised, { bodyId: input.bodyId, pid: proc.pid, port, startedAt: Date.now(), name: input.name, size: input.size }, (why) =>
560567 parent.log.warn("child forge record not written", { name: input.name, why }),
561568 );
562569 const child: Held = { ...input, port, limits: spec.limits, pid: proc.pid, exited, cell, proc, door };
src/spawn/cell_record.ts +77 −22
@@ -1,15 +1,25 @@
11 /**
22 * What a cell that outlives its host leaves for the next host to find (#13756): one small file
3 * under the child's directory saying which process is the cell, where it answers and which
4 * supervisor holds its proxy. Plain data in, plain data out; `cell.ts` writes it when a cell
5 * answers and reads it at boot to adopt the live ones.
3 * saying which process is the cell, where it answers and which supervisor holds its proxy. Plain
4 * data in, plain data out; `cell.ts` writes it when a cell answers and reads it at boot to adopt
5 * the live ones.
6 *
7 * The records live in `<data>/cell-records/<body>.json` (#15235): a directory the parent owns and
8 * binds into no cell. They used to sit in the child's own directory, which its cell writes, so a
9 * child could name any port and pid for its parent to adopt or signal; `dropLegacyRecords`
10 * removes those, unread.
611 */
712
8 import { mkdirSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
13 import { lstatSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
914 import { join } from "node:path";
1015 import { SIZES, type Size } from "./sizes.ts";
16 import { validBodyId } from "./disk_mount.ts";
1117
12 export const RECORD_FILE = "cell.json";
18 /** The directory under the parent's data the records live in, one file a body. */
19 export const RECORDS_DIR = "cell-records";
20
21 /** The file an older host left in a child's own directory, which the child could write. */
22 export const LEGACY_RECORD_FILE = "cell.json";
1323
1424 export interface CellRecord {
1525 readonly bodyId: string;
@@ -36,12 +46,18 @@ export function parseRecord(raw: unknown): CellRecord | undefined {
3646 return { bodyId, pid, port, supervisor, startedAt, name, size: size as Size };
3747 }
3848
39 export const recordPath = (childDir: string): string => join(childDir, RECORD_FILE);
49 export const recordsDir = (dataDir: string): string => join(dataDir, RECORDS_DIR);
50
51 /** The record's path for `bodyId`; a body id that could leave the directory is refused (throws). */
52 export function recordPath(dataDir: string, bodyId: string): string {
53 if (!validBodyId(bodyId)) throw new Error(`not a body id: ${JSON.stringify(bodyId.slice(0, 80))}`);
54 return join(recordsDir(dataDir), `${bodyId}.json`);
55 }
4056
4157 /** Write the record whole, through a rename, so a reader never sees half of one. */
42 export function writeRecord(childDir: string, record: CellRecord): void {
43 mkdirSync(childDir, { recursive: true });
44 const to = recordPath(childDir);
58 export function writeRecord(dataDir: string, record: CellRecord): void {
59 const to = recordPath(dataDir, record.bodyId);
60 mkdirSync(recordsDir(dataDir), { recursive: true, mode: 0o700 });
4561 const tmp = `${to}.tmp`;
4662 writeFileSync(tmp, JSON.stringify(record));
4763 renameSync(tmp, to);
@@ -51,36 +67,75 @@ export function writeRecord(childDir: string, record: CellRecord): void {
5167 * Leave the record for a cell, only when it is supervised (so it can outlive its host). A write
5268 * that fails (disk full) is told to `warn`, never thrown: the cell is up either way.
5369 */
54 export function leaveRecord(childDir: string, supervised: { readonly ctl: string } | undefined, record: Omit<CellRecord, "supervisor">, warn: (why: string) => void): void {
70 export function leaveRecord(dataDir: string, supervised: { readonly ctl: string } | undefined, record: Omit<CellRecord, "supervisor">, warn: (why: string) => void): void {
5571 if (supervised === undefined) return;
5672 try {
57 writeRecord(childDir, { ...record, supervisor: supervised.ctl });
73 writeRecord(dataDir, { ...record, supervisor: supervised.ctl });
5874 } catch (error) {
5975 warn(String(error).slice(0, 200));
6076 }
6177 }
6278
63 export function clearRecord(childDir: string): void {
64 rmSync(recordPath(childDir), { force: true });
79 /** Remove the one record of `bodyId`; nothing for a body id that is not one. */
80 export function clearRecord(dataDir: string, bodyId: string): void {
81 if (!validBodyId(bodyId)) return;
82 rmSync(recordPath(dataDir, bodyId), { force: true });
6583 }
6684
67 /** Every record under `childrenDir` (one directory a body); unreadable ones are skipped. */
68 export function readRecords(childrenDir: string): { record: CellRecord; dir: string }[] {
85 /** The record of `bodyId`, when there is one that parses and names that body. */
86 export function readRecord(dataDir: string, bodyId: string): CellRecord | undefined {
87 if (!validBodyId(bodyId)) return undefined;
88 try {
89 const record = parseRecord(JSON.parse(readFileSync(recordPath(dataDir, bodyId), "utf8")));
90 return record?.bodyId === bodyId ? record : undefined;
91 } catch {
92 return undefined; // no record, or not JSON
93 }
94 }
95
96 /** Every record in the parent's records directory; unreadable ones are skipped. */
97 export function readRecords(dataDir: string): CellRecord[] {
98 let names: string[];
99 try {
100 names = readdirSync(recordsDir(dataDir));
101 } catch {
102 return [];
103 }
104 const found: CellRecord[] = [];
105 for (const name of names) {
106 if (!name.endsWith(".json")) continue;
107 const record = readRecord(dataDir, name.slice(0, -".json".length));
108 if (record !== undefined) found.push(record);
109 }
110 return found;
111 }
112
113 /**
114 * Remove every `cell.json` an older host left in a child's own directory
115 * (`<data>/children/<body>`), without reading it: the child could have written it. Each removal is
116 * told to `removed`. A path that is a directory (or anything rm without recursion refuses) is left
117 * alone.
118 */
119 export function dropLegacyRecords(dataDir: string, removed: (bodyId: string) => void): void {
120 const childrenDir = join(dataDir, "children");
69121 let bodies: string[];
70122 try {
71123 bodies = readdirSync(childrenDir);
72124 } catch {
73 return [];
125 return;
74126 }
75 const found: { record: CellRecord; dir: string }[] = [];
76127 for (const body of bodies) {
77 const dir = join(childrenDir, body);
128 const path = join(childrenDir, body, LEGACY_RECORD_FILE);
78129 try {
79 const record = parseRecord(JSON.parse(readFileSync(recordPath(dir), "utf8")));
80 if (record !== undefined && record.bodyId === body) found.push({ record, dir });
130 lstatSync(path);
81131 } catch {
82 // no record, or not JSON
132 continue; // none here
133 }
134 try {
135 rmSync(path);
136 removed(body);
137 } catch {
138 // not a file this can remove
83139 }
84140 }
85 return found;
86141 }
src/spawn/reach.ts +4 −10
@@ -28,7 +28,7 @@ import type { FiredAlarm } from "../flow/alarms.ts";
2828 import { childPublicUrl, parentOrigin } from "./placement.ts";
2929 import { uncleanInFirstMinutes } from "../kernel/boots.ts";
3030 import { lastStop, stoppedMark } from "./stopped.ts";
31 import { parseRecord, recordPath } from "./cell_record.ts";
31 import { readRecord } from "./cell_record.ts";
3232 import { during } from "../perf/jobs.ts";
3333
3434 export const APP_HEADER = "x-forge-app";
@@ -321,14 +321,8 @@ export async function settleChild(app: App, name: string): Promise<void> {
321321 }
322322 }
323323
324 /** When the cell record in a child's directory says its cell started, if it has one. */
325 function startedOf(dir: string): number | undefined {
326 try {
327 return parseRecord(JSON.parse(readFileSync(recordPath(dir), "utf8")))?.startedAt;
328 } catch {
329 return undefined;
330 }
331 }
324 /** When the child's cell record (`cell_record.ts`) says its cell started, if it has one. */
325 const startedOf = (app: App, bodyId: string): number | undefined => readRecord(app.config.dataDir, bodyId)?.startedAt;
332326
333327 /**
334328 * Whether the host stopped or (re)started this child's cell within the settle window: its cell
@@ -338,7 +332,7 @@ function startedOf(dir: string): number | undefined {
338332 export function restartedRecently(app: App, child: Pick<Child, "bodyId">, now = Date.now()): boolean {
339333 const window = optionsOf(app).settleMs ?? SETTLE_MS;
340334 const dir = childDirOf(app, child.bodyId);
341 const times = [lastStop(app, child.bodyId, now)?.at, stoppedMark(dir)?.at, startedOf(dir)];
335 const times = [lastStop(app, child.bodyId, now)?.at, stoppedMark(dir)?.at, startedOf(app, child.bodyId)];
342336 return times.some((at) => at !== undefined && now - at >= 0 && now - at < window);
343337 }
344338
src/spawn/supervisor.ts +26
@@ -191,6 +191,32 @@ export function readPid(path: string): number | undefined {
191191 }
192192 }
193193
194 /**
195 * The pid in a pid file only when it carries its start time and that is still the process's
196 * (#15235): stricter than `readPid`, for adopting a cell, where a bare pid could be anyone's.
197 */
198 export function readStartedPid(path: string): number | undefined {
199 try {
200 const [a, start, more] = readFileSync(path, "utf8").trim().split(":");
201 const n = Number(a);
202 if (!Number.isInteger(n) || n <= 0 || start === undefined || start === "" || more !== undefined) return undefined;
203 return startTime(n) === start ? n : undefined;
204 } catch {
205 return undefined;
206 }
207 }
208
209 /**
210 * The port the supervisor in `ctl` relays the cell's way in on (`config.json` `listen`), if any.
211 */
212 export function readListen(ctl: string): number | undefined {
213 try {
214 return parseConfig(readFileSync(files(ctl).config, "utf8"))?.listen;
215 } catch {
216 return undefined;
217 }
218 }
219
194220 /** The entries of `allowed.json`, only ever strings; what is not is ignored. */
195221 export function readAllowed(path: string): string[] {
196222 try {
test/cell-adopted-on-start.test.ts +145 −50
@@ -1,12 +1,12 @@
11 import { expect, test } from "bun:test";
22 import type { Ctx } from "../src/http/context.ts";
3 import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
3 import { existsSync, mkdirSync, mkdtempSync, renameSync, writeFileSync } from "node:fs";
44 import { tmpdir } from "node:os";
55 import { join } from "node:path";
66 import { createApp } from "../src/app.ts";
77 import { makeRegistry } from "../src/spawn/cell.ts";
8 import { files } from "../src/spawn/supervisor.ts";
9 import { clearRecord, leaveRecord, parseRecord, readRecords, recordPath, writeRecord, type CellRecord } from "../src/spawn/cell_record.ts";
8 import { files, pidRecord } from "../src/spawn/supervisor.ts";
9 import { clearRecord, leaveRecord, parseRecord, readRecord, readRecords, recordPath, writeRecord, type CellRecord } from "../src/spawn/cell_record.ts";
1010
1111 const recordFor = (over: Partial<CellRecord>): CellRecord => ({ bodyId: "cell-a", pid: process.pid, port: 1, supervisor: "/x", startedAt: 1, name: "a", size: "small", ...over });
1212
@@ -15,14 +15,24 @@ const setup = (): { app: ReturnType<typeof createApp>; dir: string; childDir: (i
1515 return { app: createApp({ dataDir: dir, port: 0 }), dir, childDir: (id) => join(dir, "children", id) };
1616 };
1717
18 /**
19 * A supervisor's control directory as `startSupervisor` leaves it: `cell.pid` with the start time
20 * (`pidRecord`), and the port it relays the cell's way in on in `config.json`.
21 */
22 const supervised = (ctl: string, pid: number, listen?: number): void => {
23 mkdirSync(ctl, { recursive: true });
24 writeFileSync(files(ctl).cell, pidRecord(pid));
25 const config = { sockets: join(ctl, "s"), hosts: [], report: false, loops: [], origins: [], ...(listen === undefined ? {} : { listen }) };
26 writeFileSync(files(ctl).config, JSON.stringify(config));
27 };
28
1829 test("a registry built over a live cell adopts it, and get answers", async () => {
19 const { app, childDir, dir } = setup();
30 const { app, dir } = setup();
2031 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
2132 try {
2233 const ctl = join(dir, "ctl");
23 mkdirSync(ctl, { recursive: true });
24 writeFileSync(files(ctl).cell, String(process.pid));
25 writeRecord(childDir("cell-a"), recordFor({ port: server.port as number, supervisor: ctl }));
34 supervised(ctl, process.pid, server.port as number);
35 writeRecord(dir, recordFor({ port: server.port as number, supervisor: ctl }));
2636 const cells = makeRegistry(app);
2737 await cells.adopted;
2838 const child = cells.get("cell-a");
@@ -36,23 +46,23 @@ test("a registry built over a live cell adopts it, and get answers", async () =>
3646 });
3747
3848 test("a record whose cell is gone is dropped and nothing is adopted", async () => {
39 const { app, childDir } = setup();
49 const { app, dir } = setup();
4050 const dead = Bun.spawn(["true"]);
4151 await dead.exited;
4252 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
4353 const port = server.port as number;
4454 try {
45 writeRecord(childDir("cell-dead"), recordFor({ bodyId: "cell-dead", pid: dead.pid, port }));
55 writeRecord(dir, recordFor({ bodyId: "cell-dead", pid: dead.pid, port }));
4656 // Alive pid, but nothing answers on the port.
4757 const closed = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("") });
4858 const closedPort = closed.port as number;
4959 closed.stop(true);
50 writeRecord(childDir("cell-quiet"), recordFor({ bodyId: "cell-quiet", port: closedPort }));
60 writeRecord(dir, recordFor({ bodyId: "cell-quiet", port: closedPort }));
5161 const cells = makeRegistry(app);
5262 await cells.adopted;
5363 expect(cells.list()).toEqual([]);
54 expect(existsSync(recordPath(childDir("cell-dead")))).toBe(false);
55 expect(existsSync(recordPath(childDir("cell-quiet")))).toBe(false);
64 expect(existsSync(recordPath(dir, "cell-dead"))).toBe(false);
65 expect(existsSync(recordPath(dir, "cell-quiet"))).toBe(false);
5666 } finally {
5767 server.stop(true);
5868 app.close();
@@ -65,45 +75,55 @@ test("a record that is not one is never trusted", () => {
6575 expect(parseRecord({ ...recordFor({}), size: "huge" })).toBeUndefined();
6676 expect(parseRecord({ ...recordFor({}), pid: "1" })).toBeUndefined();
6777 expect(parseRecord(null)).toBeUndefined();
68 const { dir, childDir } = setup();
69 mkdirSync(childDir("cell-x"), { recursive: true });
70 writeRecord(childDir("cell-x"), recordFor({ bodyId: "cell-other" }));
71 expect(readRecords(join(dir, "children"))).toEqual([]);
78 const { dir } = setup();
79 // A record filed under another body's name is not that body's.
80 writeRecord(dir, recordFor({ bodyId: "cell-other" }));
81 renameSync(recordPath(dir, "cell-other"), recordPath(dir, "cell-x"));
82 expect(readRecords(dir)).toEqual([]);
83 expect(readRecord(dir, "cell-x")).toBeUndefined();
84 });
85
86 test("a record path takes only a safe body id, and the records sit outside every child's directory", () => {
87 const { dir } = setup();
88 expect(recordPath(dir, "cell-a")).toBe(join(dir, "cell-records", "cell-a.json"));
89 for (const bad of ["../children/cell-a/x", "..", ".hidden", "a/b", "", "cell a"]) {
90 expect(() => recordPath(dir, bad)).toThrow();
91 expect(readRecord(dir, bad)).toBeUndefined();
92 expect(() => clearRecord(dir, bad)).not.toThrow();
93 }
94 expect(() => writeRecord(dir, recordFor({ bodyId: "../../escape" }))).toThrow();
95 expect(existsSync(join(dir, "..", "escape.json"))).toBe(false);
7296 });
7397
7498 test("leaveRecord writes only for a supervised cell, and a failed write is told, not thrown", () => {
75 const { childDir } = setup();
99 const { dir } = setup();
76100 const base = { bodyId: "cell-w", pid: 5, port: 6, startedAt: 7, name: "w", size: "small" as const };
77 leaveRecord(childDir("cell-w"), undefined, base, () => {});
78 expect(existsSync(recordPath(childDir("cell-w")))).toBe(false);
79 leaveRecord(childDir("cell-w"), { ctl: "/ctl" }, base, () => {});
80 expect(readRecords(join(childDir("cell-w"), ".."))[0]?.record.supervisor).toBe("/ctl");
81 clearRecord(childDir("cell-w"));
82 expect(existsSync(recordPath(childDir("cell-w")))).toBe(false);
101 leaveRecord(dir, undefined, base, () => {});
102 expect(existsSync(recordPath(dir, "cell-w"))).toBe(false);
103 leaveRecord(dir, { ctl: "/ctl" }, base, () => {});
104 expect(readRecords(dir)[0]?.supervisor).toBe("/ctl");
105 clearRecord(dir, "cell-w");
106 expect(existsSync(recordPath(dir, "cell-w"))).toBe(false);
83107 const told: string[] = [];
84108 // a file where the directory should be: the write fails
85 writeFileSync(join(childDir("cell-w"), "..", "blocked"), "x");
86 leaveRecord(join(childDir("cell-w"), "..", "blocked", "x"), { ctl: "/ctl" }, base, (why) => told.push(why));
109 const blocked = mkdtempSync(join(tmpdir(), "adopt-blocked-"));
110 writeFileSync(join(blocked, "cell-records"), "x");
111 leaveRecord(blocked, { ctl: "/ctl" }, base, (why) => told.push(why));
87112 expect(told.length).toBe(1);
88113 });
89114
90 const supervised = (ctl: string, pid: number): void => {
91 mkdirSync(ctl, { recursive: true });
92 writeFileSync(files(ctl).cell, String(pid));
93 };
94
95115 test("an alive pid its supervisor does not name is not adopted, and not signalled", async () => {
96 const { app, childDir, dir } = setup();
116 const { app, dir } = setup();
97117 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
98118 const other = Bun.spawn(["sleep", "30"]);
99119 try {
100120 const ctl = join(dir, "ctl");
101 supervised(ctl, other.pid + 1);
102 writeRecord(childDir("cell-a"), recordFor({ pid: other.pid, port: server.port as number, supervisor: ctl }));
121 supervised(ctl, other.pid + 1, server.port as number);
122 writeRecord(dir, recordFor({ pid: other.pid, port: server.port as number, supervisor: ctl }));
103123 const cells = makeRegistry(app);
104124 await cells.adopted;
105125 expect(cells.list()).toEqual([]);
106 expect(existsSync(recordPath(childDir("cell-a")))).toBe(false);
126 expect(existsSync(recordPath(dir, "cell-a"))).toBe(false);
107127 expect(other.killed).toBe(false);
108128 expect(process.kill(other.pid, 0)).toBe(true);
109129 } finally {
@@ -114,19 +134,19 @@ test("an alive pid its supervisor does not name is not adopted, and not signalle
114134 });
115135
116136 test("its supervisor's cell that does not answer is stopped, and the record cleared", async () => {
117 const { app, childDir, dir } = setup();
137 const { app, dir } = setup();
118138 const quiet = Bun.spawn(["sleep", "30"]);
119139 const closed = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("") });
120140 const port = closed.port as number;
121141 closed.stop(true);
122142 try {
123143 const ctl = join(dir, "ctl");
124 supervised(ctl, quiet.pid);
125 writeRecord(childDir("cell-a"), recordFor({ pid: quiet.pid, port, supervisor: ctl }));
144 supervised(ctl, quiet.pid, port);
145 writeRecord(dir, recordFor({ pid: quiet.pid, port, supervisor: ctl }));
126146 const cells = makeRegistry(app);
127147 await cells.adopted;
128148 expect(cells.list()).toEqual([]);
129 expect(existsSync(recordPath(childDir("cell-a")))).toBe(false);
149 expect(existsSync(recordPath(dir, "cell-a"))).toBe(false);
130150 await quiet.exited;
131151 expect(quiet.signalCode !== null || quiet.exitCode !== null).toBe(true);
132152 } finally {
@@ -136,13 +156,13 @@ test("its supervisor's cell that does not answer is stopped, and the record clea
136156 });
137157
138158 test("an adopted cell dropped on its death clears its record, and start waits for adoption", async () => {
139 const { app, childDir, dir } = setup();
159 const { app, dir } = setup();
140160 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
141161 const proc = Bun.spawn(["sleep", "30"]);
142162 try {
143163 const ctl = join(dir, "ctl");
144 supervised(ctl, proc.pid);
145 writeRecord(childDir("cell-a"), recordFor({ pid: proc.pid, port: server.port as number, supervisor: ctl }));
164 supervised(ctl, proc.pid, server.port as number);
165 writeRecord(dir, recordFor({ pid: proc.pid, port: server.port as number, supervisor: ctl }));
146166 let launched = 0;
147167 const cells = makeRegistry(app, async () => {
148168 launched++;
@@ -154,7 +174,7 @@ test("an adopted cell dropped on its death clears its record, and start waits fo
154174 expect(launched).toBe(0);
155175 proc.kill();
156176 await started?.exited;
157 expect(existsSync(recordPath(childDir("cell-a")))).toBe(false);
177 expect(existsSync(recordPath(dir, "cell-a"))).toBe(false);
158178 expect(cells.get("cell-a")).toBeUndefined();
159179 } finally {
160180 proc.kill();
@@ -167,7 +187,7 @@ test("the rows say which cells were kept and which stopped, the page and the ter
167187 const { adoptedLines } = await import("../src/tui/screens/system.ts");
168188 const { adoptedSection } = await import("../src/web/pages/system.ts");
169189 const { adoptionsOf } = await import("../src/spawn/cell.ts");
170 const { app, childDir, dir } = setup();
190 const { app, dir } = setup();
171191 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
172192 const quiet = Bun.spawn(["sleep", "30"]);
173193 const closed = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("") });
@@ -175,11 +195,11 @@ test("the rows say which cells were kept and which stopped, the page and the ter
175195 closed.stop(true);
176196 try {
177197 const ctl = join(dir, "ctl");
178 supervised(ctl, process.pid);
179 writeRecord(childDir("cell-a"), recordFor({ port: server.port as number, supervisor: ctl }));
198 supervised(ctl, process.pid, server.port as number);
199 writeRecord(dir, recordFor({ port: server.port as number, supervisor: ctl }));
180200 const ctlQuiet = join(dir, "ctl-quiet");
181 supervised(ctlQuiet, quiet.pid);
182 writeRecord(childDir("cell-q"), recordFor({ bodyId: "cell-q", name: "q", pid: quiet.pid, port: closedPort, supervisor: ctlQuiet }));
201 supervised(ctlQuiet, quiet.pid, closedPort);
202 writeRecord(dir, recordFor({ bodyId: "cell-q", name: "q", pid: quiet.pid, port: closedPort, supervisor: ctlQuiet }));
183203 const cells = makeRegistry(app);
184204 await cells.adopted;
185205 const rows = cells.adoptions();
@@ -206,12 +226,12 @@ test("the MCP system tool's adopted read gives an admin the rows and refuses any
206226 const { tool } = await import("../src/http/mcp/tools/system.ts");
207227 const { cellChildren } = await import("../src/spawn/cell.ts");
208228 const { createUser } = await import("../src/core/users.ts");
209 const { app, childDir, dir } = setup();
229 const { app, dir } = setup();
210230 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
211231 try {
212232 const ctl = join(dir, "ctl");
213 supervised(ctl, process.pid);
214 writeRecord(childDir("cell-a"), recordFor({ port: server.port as number, supervisor: ctl }));
233 supervised(ctl, process.pid, server.port as number);
234 writeRecord(dir, recordFor({ port: server.port as number, supervisor: ctl }));
215235 await cellChildren(app).adopted;
216236 const person = async (username: string, isAdmin: boolean) => {
217237 const r = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", isAdmin });
@@ -230,3 +250,78 @@ test("the MCP system tool's adopted read gives an admin the rows and refuses any
230250 app.close();
231251 }
232252 });
253
254 /** A server standing in for something else on this host: it counts what it is asked. */
255 const victimServer = (): { port: number; asked: () => number; stop: () => void } => {
256 let n = 0;
257 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => (n++, new Response("victim")) });
258 return { port: server.port as number, asked: () => n, stop: () => server.stop(true) };
259 };
260
261 test("a cell.json a child forged in its own directory is removed unread, and the server it names is never asked (#15235)", async () => {
262 const { app, childDir } = setup();
263 const victim = victimServer();
264 try {
265 // Everything here is what a child's cell can write: its own directory, a control directory
266 // inside it naming a live pid with its true start time, and the victim's port.
267 const ctl = join(childDir("cell-a"), "fake-ctl");
268 supervised(ctl, process.pid, victim.port);
269 writeFileSync(join(childDir("cell-a"), "cell.json"), JSON.stringify(recordFor({ port: victim.port, supervisor: ctl })));
270 const cells = makeRegistry(app);
271 await cells.adopted;
272 expect(cells.list()).toEqual([]);
273 expect(existsSync(join(childDir("cell-a"), "cell.json"))).toBe(false);
274 expect(victim.asked()).toBe(0);
275 } finally {
276 victim.stop();
277 app.close();
278 }
279 });
280
281 test("a record whose port is not its supervisor's is dropped, and that port is never asked (#15235)", async () => {
282 const { app, dir } = setup();
283 const victim = victimServer();
284 const cellSide = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
285 try {
286 const ctl = join(dir, "ctl");
287 supervised(ctl, process.pid, cellSide.port as number);
288 writeRecord(dir, recordFor({ port: victim.port, supervisor: ctl }));
289 const cells = makeRegistry(app);
290 await cells.adopted;
291 expect(cells.list()).toEqual([]);
292 expect(cells.adoptions().map((r) => r.outcome)).toEqual(["dropped"]);
293 expect(existsSync(recordPath(dir, "cell-a"))).toBe(false);
294 expect(victim.asked()).toBe(0);
295 } finally {
296 cellSide.stop(true);
297 victim.stop();
298 app.close();
299 }
300 });
301
302 test("a cell.pid with no start time is no cell's: the process it names is never signalled (#15235)", async () => {
303 const { app, dir } = setup();
304 // `sleep` runs under `sh`, which says its pid and exits with its status: 143 for the SIGTERM
305 // this test sends, 137 had adoption killed the tree (`killTree`, SIGKILL).
306 const victim = Bun.spawn(["sh", "-c", "sleep 30 & echo $!; wait $!; exit $?"], { stdout: "pipe" });
307 const said = await victim.stdout.getReader().read();
308 const sleeper = Number(new TextDecoder().decode(said.value).trim());
309 const closed = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("") });
310 const silentPort = closed.port as number;
311 closed.stop(true);
312 try {
313 const ctl = join(dir, "ctl");
314 supervised(ctl, victim.pid, silentPort);
315 // A bare pid: what an older host wrote, or anyone who knows a pid and not its start.
316 writeFileSync(files(ctl).cell, String(victim.pid));
317 writeRecord(dir, recordFor({ pid: victim.pid, port: silentPort, supervisor: ctl }));
318 const cells = makeRegistry(app);
319 await cells.adopted;
320 expect(cells.adoptions().map((r) => r.outcome)).toEqual(["dropped"]);
321 process.kill(sleeper, "SIGTERM");
322 expect(await victim.exited).toBe(143);
323 } finally {
324 victim.kill();
325 app.close();
326 }
327 });
test/spawn-reach.test.ts +3 −6
@@ -13,6 +13,7 @@ import { listEvents } from "../src/core/events.ts";
1313 import { clientRows, runningWords, useChildFetch } from "../src/spawn/clients.ts";
1414 import { Database } from "bun:sqlite";
1515 import { childDirOf } from "../src/actions/spawn.ts";
16 import { clearRecord, writeRecord } from "../src/spawn/cell_record.ts";
1617 import { appUrlOf, slugsOf, proveApp, proveHome, reachAlarms, reachOf, reachWords, sweepReach, useReach } from "../src/spawn/reach.ts";
1718 import { noteStop, cancelRestarts, lastStop, markStopped } from "../src/spawn/stopped.ts";
1819 import { quietLogs } from "./helpers/globals.ts";
@@ -171,17 +172,13 @@ describe("a forge watches its children's addresses", () => {
171172 const reset = () => {
172173 world.set("https://ok-app--kid--forge.test/", () => new Response("ok", { headers: { "x-forge-app": "ok-app" } }));
173174 rmSync(join(dir, "reach"), { recursive: true, force: true });
174 rmSync(childDirOf(parent, "cell-k") + "/cell.json", { force: true });
175 clearRecord(dir, "cell-k");
175176 rmSync(childDirOf(parent, "cell-k") + "/stopped.json", { force: true });
176177 cancelRestarts(parent);
177178 };
178179 const raised = () => events("forge.child_unreachable").length;
179180
180 const startCell = (at: number) =>
181 writeFileSync(
182 join(childDirOf(parent, "cell-k"), "cell.json"),
183 JSON.stringify({ bodyId: "cell-k", pid: 1, port: 1, supervisor: "x", startedAt: at, name: "kid", size: "small" }),
184 );
181 const startCell = (at: number) => writeRecord(dir, { bodyId: "cell-k", pid: 1, port: 1, supervisor: "x", startedAt: at, name: "kid", size: "small" });
185182 const answers = () => world.set("https://deal-clock--kid--forge.test/", () => new Response("deal", { headers: { "x-forge-app": "deal-clock" } }));
186183 const keptQuietly = async () => {
187184 const before = raised();

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs