Public: anyone can read this; the crew can change it.

On start the host re-adopts the cells it left running (record, probe, take over the registry entry) #13927

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

mason/forge:mason/13756-on-start-the-host-re-adopts-the-cells-it → main 7 commits fork 2 threads · 2 unresolved

  • Merged. mason/13756-on-start-the-host-re-adopts-the-cells-it was merged into main as 762c115.

Review

Sign in to review this change.
8 files changed +330 −5
docs/features/README.md +1
@@ -505,6 +505,7 @@ Read with the guide page [When something is wrong](../guide-when-wrong.md).
505505 - [The machine's real size on /system and in health](system.md)
506506 - [Where the bottleneck is: the line, CPU, checks, crew, memory and disk, each beside its lever](bottleneck.md)
507507 - [A child forge the host stopped](child-stopped.md)
508 - [Cells that outlive the host, and are taken back on start](cells-outlive.md)
508509 - [A failed adapter pull speaks once in the writers' feed](sync-failed-feed.md)
509510 - [A failing webhook is said once](webhook-failing.md)
510511 - [A full disk](full-disk.md)
docs/features/cells-outlive.md added +11
@@ -0,0 +1,11 @@
1 # Cells outlive the host (`FORGE_CELLS_OUTLIVE=on`)
2
3 Off by default. With it on, a child's cell runs its proxy and relays in a supervisor of its own and is not tied to this process: stopping the host leaves the cell running (#13755).
4
5 **On start the host takes its cells back (#13756).** Each supervised cell writes `children/<body>/cell.json` (pid, port, supervisor control directory, start time, name, size) as soon as it is spawned, and clears it when it exits. At boot the host reads every record and, for each:
6
7 - the pid must still be the cell its supervisor names (`cell.pid` in the control directory, pid and start time), so a reused pid is never adopted or signalled;
8 - the port must answer. A cell that is its supervisor's but silent is stopped and its record cleared, since nothing else would reap it;
9 - otherwise the cell is held in the registry as if this host had started it. A start for the same body waits for adoption to finish and finds the adopted cell, never a second one. The record is cleared when the cell goes.
10
11 Not yet: the beat door still runs in the host, egress bytes of an outliving cell are not in the host's gauges (#13921), and there is no terminal view of adopted cells; they appear in the registry and the log (`child forge adopted`).
src/kernel/destroy_reach.ts +1
@@ -88,6 +88,7 @@ export const ALLOWED_REMOVALS: readonly { unit: string; why: string }[] = [
8888 { unit: "src/apps/contain.ts:openCell", why: "clears the cell's socket directory before the cell opens" },
8989 { unit: "src/spawn/supervisor.ts:startSupervisor", why: "removes the cell's own control directory (a fresh temp directory it made) when the host closes the cell" },
9090 { unit: "src/spawn/supervisor.ts:runSupervisor", why: "removes the cell's own socket directory and, when no host is left, its control directory: both fresh temp directories made for that cell" },
91 { unit: "src/spawn/cell_record.ts:clearRecord", why: "removes the one cell.json a running cell left in its own child directory, never the directory or anything else in it" },
9192 { unit: "src/apps/cgroup.ts:removeDir", why: "removes an empty cgroup directory" },
9293 { unit: "src/actions/spawn.ts:unmake", why: "removes the directory and ledger of a child whose first attempt failed before it was ever finished, and only that call's own (never a finished or resumed one): nothing of anyone's work is in it (#13135)" },
9394 { unit: "src/spawn/own_code.ts:checkout", why: "removes a half-made export of the forge's own code, keyed by commit" },
src/spawn/cell.ts +63 −5
@@ -27,15 +27,18 @@ import { childModelHosts, childPaymentHosts, childRegistryHosts } from "../core/
2727 import { liveChildDisk, type ChildDisk } from "./disk_mount.ts";
2828 import { oneAtATime, sourcesFor, startFailure, startFirstThatWorks, useTree, type CodeSource } from "./own_code.ts";
2929 import { childGate, type ChildGate } from "./child_reach.ts";
30 import { files, readPid } from "./supervisor.ts";
31 import { clearRecord, leaveRecord, readRecords, type CellRecord } from "./cell_record.ts";
3032
3133 /**
3234 * Whether a child's cell outlives this process (`FORGE_CELLS_OUTLIVE=on`; #13755): its proxy and
3335 * relays run in a supervisor of their own, the cell is not tied to its parent, and a stop of this
34 * process leaves it running. Off by default and undocumented: nothing yet re-adopts or reaps a cell
35 * on start (#13750 step 2), its beat door (`beat-door.ts`) is this process's, reached through the
36 * supervisor's relay while this process lives and answered by the supervisor's held door after
37 * (`held_door.ts`, #13906), and its egress bytes are counted in the supervisor, out of this
38 * process's gauges (#13921).
36 * process leaves it running. Off by default (docs/features/cells-outlive.md): each cell leaves a
37 * record (`cell_record.ts`) that the next host adopts if the cell is still its supervisor's and
38 * answers, and stops if it does not answer (#13756); its beat door (`beat-door.ts`) is this
39 * process's, reached through the supervisor's relay while this process lives and answered by the
40 * supervisor's held door after (`held_door.ts`, #13906), and its egress bytes are counted in the
41 * supervisor, out of this process's gauges (#13921).
3942 */
4043 export const cellsOutlive = (env: Record<string, string | undefined> = process.env): boolean => env["FORGE_CELLS_OUTLIVE"] === "on";
4144
@@ -263,6 +266,8 @@ export interface CellChildren {
263266 */
264267 forgetFailure(bodyId: string): void;
265268 list(): readonly CellChild[];
269 /** Settles once the cells a previous host left running have been probed, adopted or dropped. */
270 readonly adopted: Promise<void>;
266271 /**
267272 * Mark a request under way for this child; the returned function ends it. A child with one is not
268273 * put to sleep.
@@ -427,6 +432,50 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
427432 /** Requests under way, by body id. */
428433 const busy = new Map<string, number>();
429434
435 /**
436 * A cell a previous host left running: its pid is watched, since this process did not start it.
437 */
438 const adopt = async (record: CellRecord, dir: string): Promise<void> => {
outdated

Alive pid + silent port: the record is cleared and the process is left running, unowned. Also no identity check beyond the port, so a reused pid is adopted and later signalled.

439 // The pid alone can be another process's by now: it is the cell only when its supervisor
440 // still names it (pid and start time, `readPid`). A pid that fails this is never signalled.
441 const isCell = (): boolean => alive(record.pid) && readPid(files(record.supervisor).cell) === record.pid;
442 if (!isCell()) {
443 clearRecord(dir);
444 return;
445 }
446 if (!(await answers(record.port, Promise.resolve(), 3))) {
447 // Ours and silent: nobody would reap it, so it goes now, as its supervisor's cell.
448 if (isCell()) killTree(record.pid);
449 parent.log.warn("child forge left by a previous host did not answer; stopped", { name: record.name, pid: record.pid });
450 clearRecord(dir);
451 return;
452 }
453 // Watched only now it is held: the timer ends when the cell is no longer the one recorded.
454 const gone = new Promise<number>((resolve) => {
455 const timer = setInterval(() => {
456 if (isCell()) return;
457 clearInterval(timer);
458 resolve(0);
459 }, 1000);
460 timer.unref();
461 });
462 const child: CellChild = { bodyId: record.bodyId, name: record.name, size: record.size, port: record.port, limits: cellLimits(record.size), pid: record.pid, exited: gone };
463 held.set(record.bodyId, child);
464 seen.set(record.bodyId, Date.now());
465 startedAt.set(record.bodyId, record.startedAt);
466 parent.log.info("child forge adopted", { name: record.name, pid: record.pid });
467 void gone.then(() => {
468 clearRecord(dir);
469 if (held.get(record.bodyId) === child) {
470 held.delete(record.bodyId);
471 startedAt.delete(record.bodyId);
472 }
473 });
474 };
475 const adopted = Promise.all(readRecords(join(parent.config.dataDir, "children")).map(({ record, dir }) => adopt(record, dir))).then(() => {});
476 let adoptionOver = false;
477 void adopted.finally(() => (adoptionOver = true)).catch(() => {});
478
430479 const failOf = (bodyId: string) => (why: string): void => void failed.set(bodyId, { at: Date.now(), why });
431480 const startOn = async (input: { bodyId: string; name: string; size: Size }, source: CodeSource): Promise<{ child: CellChild } | { why: string }> => {
432481 const recent: string[] = [];
@@ -468,12 +517,18 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
468517 noteStop(parent, input.bodyId, input.name, why);
469518 }
470519 cell.close();
520 clearRecord(dir);
471521 door.close();
472522 release();
473523 if (why !== undefined) recent.push(why);
474524 return code;
475525 });
476526 cell.watch(proc.pid, () => killTree(proc.pid));
527 // Written before anything awaits, so the exit hook above (which clears it) can only come
528 // after; a host that dies before the cell answers leaves a record the next one reaps.
529 leaveRecord(dir, cell.supervised, { bodyId: input.bodyId, pid: proc.pid, port, startedAt: Date.now(), name: input.name, size: input.size }, (why) =>
530 parent.log.warn("child forge record not written", { name: input.name, why }),
531 );
477532 const child: Held = { ...input, port, limits: spec.limits, pid: proc.pid, exited, cell, proc, door };
478533 let gone = false;
479534 void exited.then(() => (gone = true));
@@ -527,7 +582,10 @@ export function makeRegistry(parent: App, launcher?: Launcher): CellChildren {
527582 };
528583
529584 return {
585 adopted,
530586 start(input) {
587 // A cell a previous host left running is found before a second is started beside it.
588 if (!adoptionOver) return adopted.then(() => this.start(input));
531589 const leaving = stopping.get(input.bodyId);
532590 if (leaving !== undefined) return leaving.then(() => this.start(input));
533591 const there = held.get(input.bodyId);
src/spawn/cell_record.ts added +86
@@ -0,0 +1,86 @@
1 /**
2 * What a cell that outlives its host leaves for the next host to find (#13756): one small file
3 * under the child's directory saying which process is the cell, where it answers and which
4 * supervisor holds its proxy. Plain data in, plain data out; `cell.ts` writes it when a cell
5 * answers and reads it at boot to adopt the live ones.
6 */
7
8 import { mkdirSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
9 import { join } from "node:path";
10 import { SIZES, type Size } from "./sizes.ts";
11
12 export const RECORD_FILE = "cell.json";
13
14 export interface CellRecord {
15 readonly bodyId: string;
16 readonly pid: number;
17 readonly port: number;
18 /** The supervisor's control directory. */
19 readonly supervisor: string;
20 readonly startedAt: number;
21 /** The child's name and size, so a host that did not start it can answer for it. */
22 readonly name: string;
23 readonly size: Size;
24 }
25
26 const int = (v: unknown): v is number => typeof v === "number" && Number.isInteger(v) && v > 0;
27
28 /** The record in `raw`, or undefined when it is not one: a file on disk is never trusted. */
29 export function parseRecord(raw: unknown): CellRecord | undefined {
30 if (typeof raw !== "object" || raw === null) return undefined;
31 const r = raw as Record<string, unknown>;
32 const { bodyId, pid, port, supervisor, startedAt, name, size } = r;
33 if (typeof bodyId !== "string" || bodyId === "" || typeof supervisor !== "string" || typeof name !== "string" || name === "") return undefined;
34 if (!int(pid) || !int(port) || port > 65535 || !int(startedAt)) return undefined;
35 if (typeof size !== "string" || !Object.hasOwn(SIZES, size)) return undefined;
36 return { bodyId, pid, port, supervisor, startedAt, name, size: size as Size };
37 }
38
39 export const recordPath = (childDir: string): string => join(childDir, RECORD_FILE);
40
41 /** Write the record whole, through a rename, so a reader never sees half of one. */
42 export function writeRecord(childDir: string, record: CellRecord): void {
43 mkdirSync(childDir, { recursive: true });
44 const to = recordPath(childDir);
45 const tmp = `${to}.tmp`;
46 writeFileSync(tmp, JSON.stringify(record));
47 renameSync(tmp, to);
48 }
49
50 /**
51 * Leave the record for a cell, only when it is supervised (so it can outlive its host). A write
52 * that fails (disk full) is told to `warn`, never thrown: the cell is up either way.
53 */
54 export function leaveRecord(childDir: string, supervised: { readonly ctl: string } | undefined, record: Omit<CellRecord, "supervisor">, warn: (why: string) => void): void {
55 if (supervised === undefined) return;
56 try {
57 writeRecord(childDir, { ...record, supervisor: supervised.ctl });
58 } catch (error) {
59 warn(String(error).slice(0, 200));
60 }
61 }
62
63 export function clearRecord(childDir: string): void {
64 rmSync(recordPath(childDir), { force: true });
65 }
66
67 /** Every record under `childrenDir` (one directory a body); unreadable ones are skipped. */
68 export function readRecords(childrenDir: string): { record: CellRecord; dir: string }[] {
69 let bodies: string[];
70 try {
71 bodies = readdirSync(childrenDir);
72 } catch {
73 return [];
74 }
75 const found: { record: CellRecord; dir: string }[] = [];
76 for (const body of bodies) {
77 const dir = join(childrenDir, body);
78 try {
79 const record = parseRecord(JSON.parse(readFileSync(recordPath(dir), "utf8")));
80 if (record !== undefined && record.bodyId === body) found.push({ record, dir });
81 } catch {
82 // no record, or not JSON
83 }
84 }
85 return found;
86 }
src/spawn/host.ts +2
@@ -977,6 +977,8 @@ export async function bootHostedChild(parent: App, name: string, evict = true):
977977 export async function bootHostedChildren(parent: App, wasAwake: ReadonlySet<string> = takeAwake(parent.config.dataDir)): Promise<void> {
978978 keepWatchingFloor(parent);
979979 keepSweeping(parent);
980 // Cells the last host left running are adopted before any is woken or started again (#13756).
981 await cellChildren(parent).adopted;
980982 let files: string[] = [];
981983 try {
982984 files = readdirSync(join(parent.config.dataDir, "spawn"));
test/cell-adopted-on-start.test.ts added +163
@@ -0,0 +1,163 @@
1 import { expect, test } from "bun:test";
2 import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
3 import { tmpdir } from "node:os";
4 import { join } from "node:path";
5 import { createApp } from "../src/app.ts";
6 import { makeRegistry } from "../src/spawn/cell.ts";
7 import { files } from "../src/spawn/supervisor.ts";
8 import { clearRecord, leaveRecord, parseRecord, readRecords, recordPath, writeRecord, type CellRecord } from "../src/spawn/cell_record.ts";
9
10 const recordFor = (over: Partial<CellRecord>): CellRecord => ({ bodyId: "cell-a", pid: process.pid, port: 1, supervisor: "/x", startedAt: 1, name: "a", size: "small", ...over });
11
12 const setup = (): { app: ReturnType<typeof createApp>; dir: string; childDir: (id: string) => string } => {
13 const dir = mkdtempSync(join(tmpdir(), "adopt-"));
14 return { app: createApp({ dataDir: dir, port: 0 }), dir, childDir: (id) => join(dir, "children", id) };
15 };
16
17 test("a registry built over a live cell adopts it, and get answers", async () => {
18 const { app, childDir, dir } = setup();
19 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
20 try {
21 const ctl = join(dir, "ctl");
22 mkdirSync(ctl, { recursive: true });
23 writeFileSync(files(ctl).cell, String(process.pid));
24 writeRecord(childDir("cell-a"), recordFor({ port: server.port as number, supervisor: ctl }));
25 const cells = makeRegistry(app);
26 await cells.adopted;
27 const child = cells.get("cell-a");
28 expect(child?.port).toBe(server.port as number);
29 expect(child?.name).toBe("a");
30 expect(cells.list().length).toBe(1);
31 } finally {
32 server.stop(true);
33 app.close();
34 }
35 });
36
37 test("a record whose cell is gone is dropped and nothing is adopted", async () => {
38 const { app, childDir } = setup();
39 const dead = Bun.spawn(["true"]);
40 await dead.exited;
41 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
42 const port = server.port as number;
43 try {
44 writeRecord(childDir("cell-dead"), recordFor({ bodyId: "cell-dead", pid: dead.pid, port }));
45 // Alive pid, but nothing answers on the port.
46 const closed = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("") });
47 const closedPort = closed.port as number;
48 closed.stop(true);
49 writeRecord(childDir("cell-quiet"), recordFor({ bodyId: "cell-quiet", port: closedPort }));
50 const cells = makeRegistry(app);
51 await cells.adopted;
52 expect(cells.list()).toEqual([]);
53 expect(existsSync(recordPath(childDir("cell-dead")))).toBe(false);
54 expect(existsSync(recordPath(childDir("cell-quiet")))).toBe(false);
55 } finally {
56 server.stop(true);
57 app.close();
58 }
59 });
60
61 test("a record that is not one is never trusted", () => {
62 expect(parseRecord(recordFor({}))).toBeDefined();
63 expect(parseRecord({ ...recordFor({}), port: 70000 })).toBeUndefined();
64 expect(parseRecord({ ...recordFor({}), size: "huge" })).toBeUndefined();
65 expect(parseRecord({ ...recordFor({}), pid: "1" })).toBeUndefined();
66 expect(parseRecord(null)).toBeUndefined();
67 const { dir, childDir } = setup();
68 mkdirSync(childDir("cell-x"), { recursive: true });
69 writeRecord(childDir("cell-x"), recordFor({ bodyId: "cell-other" }));
70 expect(readRecords(join(dir, "children"))).toEqual([]);
71 });
72
73 test("leaveRecord writes only for a supervised cell, and a failed write is told, not thrown", () => {
74 const { childDir } = setup();
75 const base = { bodyId: "cell-w", pid: 5, port: 6, startedAt: 7, name: "w", size: "small" as const };
76 leaveRecord(childDir("cell-w"), undefined, base, () => {});
77 expect(existsSync(recordPath(childDir("cell-w")))).toBe(false);
78 leaveRecord(childDir("cell-w"), { ctl: "/ctl" }, base, () => {});
79 expect(readRecords(join(childDir("cell-w"), ".."))[0]?.record.supervisor).toBe("/ctl");
80 clearRecord(childDir("cell-w"));
81 expect(existsSync(recordPath(childDir("cell-w")))).toBe(false);
82 const told: string[] = [];
83 // a file where the directory should be: the write fails
84 writeFileSync(join(childDir("cell-w"), "..", "blocked"), "x");
85 leaveRecord(join(childDir("cell-w"), "..", "blocked", "x"), { ctl: "/ctl" }, base, (why) => told.push(why));
86 expect(told.length).toBe(1);
87 });
88
89 const supervised = (ctl: string, pid: number): void => {
90 mkdirSync(ctl, { recursive: true });
91 writeFileSync(files(ctl).cell, String(pid));
92 };
93
94 test("an alive pid its supervisor does not name is not adopted, and not signalled", async () => {
95 const { app, childDir, dir } = setup();
96 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
97 const other = Bun.spawn(["sleep", "30"]);
98 try {
99 const ctl = join(dir, "ctl");
100 supervised(ctl, other.pid + 1);
101 writeRecord(childDir("cell-a"), recordFor({ pid: other.pid, port: server.port as number, supervisor: ctl }));
102 const cells = makeRegistry(app);
103 await cells.adopted;
104 expect(cells.list()).toEqual([]);
105 expect(existsSync(recordPath(childDir("cell-a")))).toBe(false);
106 expect(other.killed).toBe(false);
107 expect(process.kill(other.pid, 0)).toBe(true);
108 } finally {
109 other.kill();
110 server.stop(true);
111 app.close();
112 }
113 });
114
115 test("its supervisor's cell that does not answer is stopped, and the record cleared", async () => {
116 const { app, childDir, dir } = setup();
117 const quiet = Bun.spawn(["sleep", "30"]);
118 const closed = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("") });
119 const port = closed.port as number;
120 closed.stop(true);
121 try {
122 const ctl = join(dir, "ctl");
123 supervised(ctl, quiet.pid);
124 writeRecord(childDir("cell-a"), recordFor({ pid: quiet.pid, port, supervisor: ctl }));
125 const cells = makeRegistry(app);
126 await cells.adopted;
127 expect(cells.list()).toEqual([]);
128 expect(existsSync(recordPath(childDir("cell-a")))).toBe(false);
129 await quiet.exited;
130 expect(quiet.signalCode !== null || quiet.exitCode !== null).toBe(true);
131 } finally {
132 quiet.kill();
133 app.close();
134 }
135 });
136
137 test("an adopted cell dropped on its death clears its record, and start waits for adoption", async () => {
138 const { app, childDir, dir } = setup();
139 const server = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("ok") });
140 const proc = Bun.spawn(["sleep", "30"]);
141 try {
142 const ctl = join(dir, "ctl");
143 supervised(ctl, proc.pid);
144 writeRecord(childDir("cell-a"), recordFor({ pid: proc.pid, port: server.port as number, supervisor: ctl }));
145 let launched = 0;
146 const cells = makeRegistry(app, async () => {
147 launched++;
148 return undefined;
149 });
150 // Asked for before adoption is over: it gets the adopted cell, and no second is launched.
151 const started = await cells.start({ bodyId: "cell-a", name: "a", size: "small" });
152 expect(started?.pid).toBe(proc.pid);
153 expect(launched).toBe(0);
154 proc.kill();
155 await started?.exited;
156 expect(existsSync(recordPath(childDir("cell-a")))).toBe(false);
157 expect(cells.get("cell-a")).toBeUndefined();
158 } finally {
159 proc.kill();
160 server.stop(true);
161 app.close();
162 }
163 });
test/timers-named.test.ts +3
@@ -67,6 +67,9 @@ const ALLOWED: readonly Allowed[] = [
6767 ...allow("setInterval", "the cell supervisor's own tick: a separate process with no job ring, its one timer", [
6868 ["src/spawn/supervisor.ts", "() => { capFile(f.log"],
6969 ]),
70 ...allow("setInterval", "an adopted cell's identity check: one kill(0) and a pid-file read a second per cell, ended when the cell is gone", [
71 ["src/spawn/cell.ts", "() => { if (isCell()) return;"],
72 ]),
7073 ...allow("setInterval", `SSE heartbeat, ${PER_CONNECTION}`, [["src/http/sse.ts", "() => write("]]),
7174 ...allow("setInterval", `a talk stream's poll, ${PER_CONNECTION}`, [["src/web/pages/talk.ts", "tick"]]),
7275 ...allow("setInterval", `a shell session's idle check, ${PER_CONNECTION}`, [["src/machines/shell.ts", "() => { if (idle(session"]]),

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs