trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns6g rEventsFlows

A child forge's disk, memory, CPU and process limits are enforced by the kernel; a 50G fallocate on a small child must fail #10855

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/10850-a-child-forge-s-disk-memory-cpu-and-proc → main 9 commits fork

  • Merged. torque/10850-a-child-forge-s-disk-memory-cpu-and-proc was merged into main as cc5614b.

Review

Sign in to review this change.
24 files changed +468 −18
docs/APPS.md +18
@@ -623,6 +623,24 @@ never exhausted (#10518). Crew runs get no process cap, as before.
623623 /system says beside the memory line who holds the process count, and why
624624 when it is the watch alone.
625625
626 Every cell also starts under kernel rlimits, with or without a cgroup: its
627 first shell sets, soft and hard, `RLIMIT_FSIZE` to the cell's disk (a small
628 child forge: 2048 MiB, so `fallocate -l 50G` or a runaway `dd` fails at
629 once with SIGXFSZ), `RLIMIT_NOFILE` 8192 and `RLIMIT_NPROC` 4096 tasks,
630 counted per user namespace (`cellRlimits`, `RLIMIT_LINE` in
631 `apps/contain.ts`). The file cap bounds one file, not their sum: the total
632 is the watch's disk walk until a child's writable space is a filesystem of
633 its own (#10850). A cell's hostname is its own: a child forge's cell
634 answers `uname -n` with the child's name, a machine shell with the
635 machine's, anything else with `cell`, never the pod's — where the host
636 lets bwrap set it: the cell probe asks once (`hostnameArgs`), and a host
637 that refuses `sethostname` runs its cells under the pod's name rather
638 than not at all.
639
640 The parent keeps 15% of its own volume (`spawn/disk_floor.ts`): below it
641 no child forge is made or started (a sleeping one stays asleep), the New
642 forge page and menu say why, and vitals carries the `disk-floor` line.
643
626644 On a Kubernetes pod the forge is usually not at a cgroup namespace root
627645 (`/proc/self/cgroup` is not `0::/`) and `/sys/fs/cgroup` is read-only, so
628646 cells are watched only. The operator delegates one: run the container in
src/apps/contain.ts +83 −2
@@ -238,6 +238,8 @@ export interface CellSpec {
238238 readonly limits: Limits;
239239 /** Read-only paths to keep visible when they sit under a hidden directory (a forge's own code). */
240240 readonly reveal?: readonly string[] | undefined;
241 /** The name the cell's processes see as the host's (`cellHostname`); "cell" when unset. */
242 readonly hostname?: string | undefined;
241243 /** The port an app listens on; the forge relays `127.0.0.1:<port>` in. */
242244 readonly listen?: number | undefined;
243245 /** Keep the controlling terminal (an app's `tty`). */
@@ -332,7 +334,24 @@ async function probe(): Promise<{ ok: true } | { ok: false; reason: string }> {
332334 for (const tool of ["bwrap", "socat"])
333335 if (Bun.which(tool) === null)
334336 return { ok: false, reason: `${tool} is not installed` };
335 return askUntilBelieved(tryOnce);
337 const verdict = await askUntilBelieved(() => tryOnce());
338 // Some hosts hold a cell but refuse its sethostname (EPERM from bwrap's
339 // `--hostname`, seen on the check machine): asked once, after the cell is
340 // believed, so a refused name costs the cell nothing (#10850).
341 if (verdict.ok) hostnameAllowed = (await tryOnce(["--hostname", "cell"])).ok;
342 return verdict;
343 }
344
345 /** Whether this host lets a cell set its own hostname; false until `cellsAvailable` has asked. */
346 let hostnameAllowed = false;
347
348 /**
349 * The `--hostname` bwrap gets: the cell's own name (`cellHostname`) where
350 * the host allows it, nothing where it does not — the cell then keeps the
351 * name it was born with, the pod's, rather than not starting at all.
352 */
353 export function hostnameArgs(name: string | undefined, allowed: boolean = hostnameAllowed): string[] {
354 return allowed ? ["--hostname", cellHostname(name)] : [];
336355 }
337356
338357 type Verdict = { ok: true } | { ok: false; reason: string };
@@ -359,13 +378,14 @@ const PROBE_TRIES = 3;
359378 const PROBE_PAUSE_MS = 500;
360379 const PROBE_TIMEOUT_MS = 10_000;
361380
362 async function tryOnce(): Promise<Verdict> {
381 async function tryOnce(extra: readonly string[] = []): Promise<Verdict> {
363382 let timedOut = false;
364383 try {
365384 const proc = Bun.spawn(
366385 [
367386 "bwrap",
368387 ...NAMESPACES,
388 ...extra,
369389 "--ro-bind",
370390 "/",
371391 "/",
@@ -422,6 +442,61 @@ const NAMESPACES = [
422442 /** The size of a cell's own `/tmp`, MiB. */
423443 export const TMP_MIB = 256;
424444
445 /** Open files a process in a cell may hold (`RLIMIT_NOFILE`): sockets count. */
446 export const FILES_MAX = 8192;
447 /**
448 * Tasks the cell's user may run (`RLIMIT_NPROC`), threads included: the
449 * kernel counts them per user namespace (since 5.14, ucounts), so this is
450 * the cell's own count. Above `PROCESSES_MAX`, which counts processes, since
451 * a Bun process runs a dozen threads; where pids is delegated `pids.max` is
452 * the tighter one.
453 */
454 export const TASKS_MAX = 4096;
455
456 /**
457 * The kernel's limits every process in a cell starts under, set by the
458 * cell's first shell with `ulimit` (both soft and hard, so nothing inside
459 * raises them; a hard limit already lower stays lower):
460 *
461 * - `fileBlocks`, `RLIMIT_FSIZE` in 512-byte blocks (dash's `ulimit -f`
462 * unit): no file grows past the cell's disk, so one `fallocate -l 50G`
463 * or `dd` on a small cell fails with SIGXFSZ / EFBIG at once, not at the
464 * watch's next disk walk. It caps a file, not the sum of them: the total
465 * is still the watch's (#10850).
466 * - `files`, `RLIMIT_NOFILE`, and `tasks`, `RLIMIT_NPROC`.
467 */
468 export function cellRlimits(limits: Pick<Limits, "diskMiB">): { fileBlocks: number; files: number; tasks: number } {
469 const blocks = Number.isFinite(limits.diskMiB) && limits.diskMiB > 0 ? Math.ceil(limits.diskMiB * 2048) : undefined;
470 return { fileBlocks: blocks ?? Number.MAX_SAFE_INTEGER, files: FILES_MAX, tasks: TASKS_MAX };
471 }
472
473 /**
474 * The name a cell answers `uname -n` and a prompt's `\h` with: its own,
475 * never the pod's (the UTS namespace is unshared). Lowercased to a DNS
476 * label (RFC 1123: letters, digits, hyphens, at most 63); "cell" when
477 * nothing usable is left.
478 */
479 export function cellHostname(name: string | undefined): string {
480 const label = (name ?? "")
481 .toLowerCase()
482 .replace(/[^a-z0-9-]+/g, "-")
483 .replace(/^-+|-+$/g, "")
484 .slice(0, 63)
485 .replace(/-+$/, "");
486 return label === "" ? "cell" : label;
487 }
488
489 /**
490 * The line the cell's first shell runs with `cellRlimits`' three figures
491 * as `$1 $2 $3`, then drops them. Arguments, never text spliced in.
492 * Processes are dash's `-p` and bash's `-u` (bash's `-p` is the pipe size,
493 * read-only, so its refusal is silent); `-f` counts 512-byte blocks in dash,
494 * the image's `/bin/sh`, and 1024-byte ones in bash, where a file's cap is
495 * then twice the disk: still finite.
496 */
497 export const RLIMIT_LINE =
498 "ulimit -f $1 2>/dev/null; ulimit -n $2 2>/dev/null; ulimit -u $3 2>/dev/null || ulimit -p $3 2>/dev/null; shift 3";
499
425500 /** The proxy's port inside every cell. */
426501 const PROXY_PORT = 3128;
427502 const INSIDE = "/run/forge";
@@ -435,6 +510,7 @@ const INSIDE = "/run/forge";
435510 * forge tells a busy app from an idle one (`apps/busy.ts`).
436511 */
437512 const WRAPPER = `s=${INSIDE}; listen=$1; shift; loops=$1; shift
513 ${RLIMIT_LINE}
438514 (socat TCP-LISTEN:${PROXY_PORT},bind=127.0.0.1,fork,reuseaddr UNIX-CONNECT:$s/out.sock >/dev/null 2>&1 &)
439515 for p in $loops; do (socat TCP-LISTEN:$p,bind=127.0.0.1,fork,reuseaddr UNIX-CONNECT:$s/loop-$p.sock >/dev/null 2>&1 &); done
440516 [ "$listen" = - ] || (socat UNIX-LISTEN:$s/in.sock,fork,unlink-early TCP:127.0.0.1:$listen >/dev/null 2>&1 &)
@@ -567,9 +643,11 @@ export function openCell(
567643 const inner = bwrapArgv(command);
568644 return cgroup === undefined ? inner : enterArgv(join(cgroup.dir, "cgroup.procs"), inner);
569645 };
646 const rlimits = cellRlimits(spec.limits);
570647 const bwrapArgv = (command: string): string[] => [
571648 "bwrap",
572649 ...NAMESPACES,
650 ...hostnameArgs(spec.hostname),
573651 ...(spec.terminal === true ? [] : ["--new-session"]),
574652 "--ro-bind", "/", "/",
575653 "--size", String(TMP_MIB * 1024 * 1024), "--tmpfs", "/tmp",
@@ -594,6 +672,9 @@ export function openCell(
594672 "cell",
595673 spec.listen === undefined ? "-" : String(spec.listen),
596674 [...new Set(loops)].join(" "),
675 String(rlimits.fileBlocks),
676 String(rlimits.files),
677 String(rlimits.tasks),
597678 "/bin/sh",
598679 "-c",
599680 command,
src/kernel/disk_reading.ts added +37
@@ -0,0 +1,37 @@
1 /**
2 * The volume under a directory, as statfs(2) reports it: free is `bavail`,
3 * what an unprivileged writer can still use, total is `blocks`, both times
4 * `bsize`. One reader every default reading goes through (vitals, the disk
5 * floor of `spawn/disk_floor.ts`), so the suite can give it a disk.
6 */
7
8 import { statfsSync } from "node:fs";
9
10 /** What a volume holds and has free. */
11 export interface DiskReading {
12 readonly freeBytes: number;
13 readonly totalBytes: number;
14 }
15
16 /** The volume under `dir`, as the operating system reports it. */
17 export function statDisk(dir: string): DiskReading {
18 const fs = statfsSync(dir);
19 return {
20 freeBytes: Number(fs.bavail) * Number(fs.bsize),
21 totalBytes: Number(fs.blocks) * Number(fs.bsize),
22 };
23 }
24
25 let reader: (dir: string) => DiskReading = statDisk;
26
27 /**
28 * The suite runs on machines whose disks are whatever they are; its preload
29 * gives a roomy one, so no test is refused by the machine it ran on, and a
30 * test of a full disk gives its own. `undefined` puts statfs back.
31 */
32 export function provideDiskForTest(r: ((dir: string) => DiskReading) | undefined): void {
33 reader = r ?? statDisk;
34 }
35
36 /** The volume under `dir` as the forge reads it by default. */
37 export const readDisk = (dir: string): DiskReading => reader(dir);
src/kernel/vitals.ts +12 −16
@@ -9,7 +9,9 @@
99 */
1010
1111 import type { Database } from "bun:sqlite";
12 import { statfsSync, statSync } from "node:fs";
12 import { statSync } from "node:fs";
13 import { readDisk, type DiskReading } from "./disk_reading.ts";
14 import { DISK_FLOOR } from "../spawn/disk_floor.ts";
1315 import { dirname } from "node:path";
1416 import { freemem, totalmem } from "node:os";
1517 import { refusedInternal, type RefusedLedger } from "./internal_refused.ts";
@@ -120,20 +122,7 @@ export interface Vitals {
120122 readonly at: string;
121123 }
122124
123 /** What a volume holds and has free. */
124 export interface DiskReading {
125 readonly freeBytes: number;
126 readonly totalBytes: number;
127 }
128
129 /** The volume under `dir`, as the operating system reports it. */
130 export function statDisk(dir: string): DiskReading {
131 const fs = statfsSync(dir);
132 return {
133 freeBytes: Number(fs.bavail) * Number(fs.bsize),
134 totalBytes: Number(fs.blocks) * Number(fs.bsize),
135 };
136 }
125 export { statDisk, readDisk, type DiskReading } from "./disk_reading.ts";
137126
138127 export interface VitalsInput {
139128 readonly db: Database;
@@ -343,7 +332,7 @@ export function vitals(input: VitalsInput): Vitals {
343332 degraded.push({ cause, sentence, since, href });
344333 };
345334
346 const disk = (input.disk ?? statDisk)(input.dataDir);
335 const disk = (input.disk ?? readDisk)(input.dataDir);
347336 if (disk.totalBytes > 0 && disk.freeBytes / disk.totalBytes < (input.diskFreeMin ?? DISK_FREE_MIN))
348337 need(
349338 "disk",
@@ -351,6 +340,13 @@ export function vitals(input: VitalsInput): Vitals {
351340 undefined,
352341 "/settings/capacity",
353342 );
343 else if (disk.totalBytes > 0 && disk.freeBytes / disk.totalBytes < DISK_FLOOR)
344 need(
345 "disk-floor",
346 `Disk is under the ${DISK_FLOOR * 100}% the forge keeps for itself: ${Math.round((disk.freeBytes / disk.totalBytes) * 100)}% free. No child forge is made or started until room is made.`,
347 undefined,
348 "/settings/capacity",
349 );
354350
355351 for (const file of openDbFiles(input.dataDir))
356352 need(`file:${file.name}`, file.sentence, undefined, "/settings/capacity");
src/machines/shell.ts +2
@@ -425,6 +425,8 @@ export function shellCell(
425425 },
426426 terminal: true,
427427 label,
428 // The prompt names the machine, never the pod it runs on (#10850).
429 hostname: machine.name,
428430 };
429431 }
430432
src/spawn/cell.ts +13
@@ -21,6 +21,7 @@ import { childDirOf } from "../actions/spawn.ts";
2121 import { openBeatDoor, parentLocal, type BeatDoor } from "./beat-door.ts";
2222 import { hostedPlacement, parentOrigin, childPublicUrl, type BodyStore, type Placement } from "./placement.ts";
2323 import { SIZE_IO, SIZES, type Size } from "./sizes.ts";
24 import { floorRefusal } from "./disk_floor.ts";
2425 import { CGROUP_DIR_ENV, SIZE_ENV } from "./usage.ts";
2526 import { childModelHosts, childRegistryHosts } from "../core/capacity.ts";
2627 import { liveChildDisk, type ChildDisk } from "./disk_mount.ts";
@@ -113,6 +114,8 @@ export function cellSpecFor(input: {
113114 size: Size;
114115 port: number;
115116 label: string;
117 /** The child's name: its cell's hostname. */
118 name?: string;
116119 /** The model provider hosts an admin let children reach (`children.modelEgress`); none by default. */
117120 modelHosts?: readonly string[];
118121 /** The package registry an admin let children reach (`children.registryEgress`); none by default. */
@@ -155,6 +158,8 @@ export function cellSpecFor(input: {
155158 io: SIZE_IO[input.size],
156159 listen: input.port,
157160 label: input.label,
161 // Its shell's prompt names the child, never the parent's pod (#10850).
162 hostname: input.name,
158163 };
159164 }
160165
@@ -254,6 +259,7 @@ export function launchSpecFor(
254259 size: input.size,
255260 port: at.port,
256261 label: `forge:${input.name}`,
262 name: input.name,
257263 modelHosts: childModelHosts(parent.db),
258264 registryHosts: childRegistryHosts(parent.db),
259265 disk: liveChildDisk(parent.config.dataDir, input.bodyId, mountinfo),
@@ -272,6 +278,13 @@ function makeRegistry(parent: App): CellChildren {
272278 const busy = new Map<string, number>();
273279
274280 const launch = async (input: { bodyId: string; name: string; size: Size }): Promise<CellChild | undefined> => {
281 // Below the disk floor no child cell starts, a sleeping one included (#10850).
282 const floor = floorRefusal(parent.config.dataDir);
283 if (floor !== undefined) {
284 parent.log.warn("child cell not started", { name: input.name, reason: floor });
285 failed.set(input.bodyId, { at: Date.now(), why: floor });
286 return undefined;
287 }
275288 const cells = await cellsAvailable();
276289 if (!cells.ok) {
277290 parent.log.warn("child cell not started", { name: input.name, reason: cells.reason });
src/spawn/disk_floor.ts added +37
@@ -0,0 +1,37 @@
1 /**
2 * The disk floor: the share of the forge's own volume it keeps for itself
3 * (#10850, item 5). Child forges write to the one disk under the parent's
4 * data; below the floor no new child forge is made (`spawn/quota.ts`
5 * `quotaRefusal`, which the New forge page and menu also show) and no child
6 * cell starts (`spawn/cell.ts`; a sleeping one stays asleep), so what room
7 * is left is the parent's to recover with. The owner's line is the vitals
8 * need `disk-floor` (`kernel/vitals.ts`).
9 */
10
11 import { readDisk, type DiskReading } from "../kernel/disk_reading.ts";
12
13 /** Free disk under this share of the volume and nothing new starts. */
14 export const DISK_FLOOR = 0.15;
15
16 /** The volume under `dir` (`readDisk`, which the suite's preload makes roomy), or undefined when it cannot be read. */
17 export function diskUnder(dir: string): DiskReading | undefined {
18 try {
19 return readDisk(dir);
20 } catch {
21 return undefined;
22 }
23 }
24
25 const gib = (bytes: number): string => (bytes / 1024 ** 3).toFixed(1);
26
27 /**
28 * One sentence when the volume under `dir` has less than `DISK_FLOOR` free,
29 * else undefined. A volume that cannot be read, or reads as empty, is not
30 * refused: the floor protects a disk it can see, it does not guess.
31 */
32 export function floorRefusal(dir: string, disk: DiskReading | undefined = diskUnder(dir)): string | undefined {
33 if (disk === undefined || !(disk.totalBytes > 0)) return undefined;
34 const share = disk.freeBytes / disk.totalBytes;
35 if (share >= DISK_FLOOR) return undefined;
36 return `this host's disk has ${(share * 100).toFixed(1)}% free (${gib(disk.freeBytes)} of ${gib(disk.totalBytes)} GiB), under the ${DISK_FLOOR * 100}% it keeps for itself: nothing new starts until room is made`;
37 }
src/spawn/quota.ts +4
@@ -8,6 +8,7 @@
88
99 import { readdirSync, readFileSync } from "node:fs";
1010 import { join } from "node:path";
11 import { floorRefusal } from "./disk_floor.ts";
1112 import { DEFAULT_SIZE, isSize, SIZES, type Size, type SizeLimits } from "./sizes.ts";
1213
1314 interface Axis {
@@ -61,6 +62,9 @@ export function quotaRefusal(
6162 size: Size,
6263 env: Record<string, string | undefined> = process.env,
6364 ): string | undefined {
65 // The host's own floor comes first, declared limits or not (#10850).
66 const floor = floorRefusal(dataDir);
67 if (floor !== undefined) return floor;
6468 const limits = AXES.map((a) => ({ a, max: limitOf(env, a.env) })).filter((l) => l.max !== undefined);
6569 if (limits.length === 0) return undefined;
6670 const held = heldByChildren(dataDir);
test/cell-kernel-limits.test.ts added +148
@@ -0,0 +1,148 @@
1 /**
2 * #10850: what the kernel holds a cell to, tried. A cell's first shell runs
3 * `RLIMIT_LINE` with `cellRlimits`; here the same line runs under the same
4 * `/bin/sh` and then tries to break each limit: a file past the disk, a
5 * file descriptor past the open-file limit, and raising a limit back.
6 * No bubblewrap is needed for that (rlimits are the kernel's per process),
7 * so these run in every sandbox; the cell's argv itself is the check's.
8 */
9
10 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
11 import { mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
12 import { tmpdir } from "node:os";
13 import { join } from "node:path";
14 import { cellCgroup, cellHostname, cellRlimits, hostnameArgs, FILES_MAX, PROCESSES_MAX, RLIMIT_LINE, TASKS_MAX } from "../src/apps/contain.ts";
15 import { cellLimits, cellSpecFor } from "../src/spawn/cell.ts";
16 import { SIZES } from "../src/spawn/sizes.ts";
17 import { missing } from "./helpers/tools.ts";
18
19 let dir: string;
20 beforeAll(() => {
21 dir = mkdtempSync(join(tmpdir(), "cell-rlimits-"));
22 });
23 afterAll(() => rmSync(dir, { recursive: true, force: true }));
24
25 /** `script` run by `/bin/sh` after the cell's own limit line, with these figures. */
26 async function underLimits(figures: { fileBlocks: number; files: number; tasks: number }, script: string) {
27 const proc = Bun.spawn(
28 ["/bin/sh", "-c", `${RLIMIT_LINE}\n${script}`, "cell", String(figures.fileBlocks), String(figures.files), String(figures.tasks)],
29 { cwd: dir, stdout: "pipe", stderr: "pipe", env: { PATH: process.env["PATH"] ?? "/usr/bin:/bin" } },
30 );
31 const [out, code] = await Promise.all([new Response(proc.stdout).text(), proc.exited]);
32 return { out: out.trim(), code };
33 }
34
35 describe("a cell's rlimits, figured from its size", () => {
36 test("a small child's file cap is its 2048 MiB in 512-byte blocks: 2048 × 2048 = 4194304", () => {
37 expect(SIZES.small.diskMiB).toBe(2048);
38 expect(cellRlimits(cellLimits("small"))).toEqual({ fileBlocks: 4_194_304, files: FILES_MAX, tasks: TASKS_MAX });
39 // large: 40960 MiB × 2048 = 83886080 blocks = 40 GiB, never the 50G of the report.
40 expect(cellRlimits(cellLimits("large")).fileBlocks).toBe(83_886_080);
41 });
42
43 test("an unbounded disk is no file cap, never a zero one", () => {
44 expect(cellRlimits({ diskMiB: Infinity }).fileBlocks).toBe(Number.MAX_SAFE_INTEGER);
45 expect(cellRlimits({ diskMiB: 0 }).fileBlocks).toBe(Number.MAX_SAFE_INTEGER);
46 });
47 });
48
49 describe.skipIf(missing("/usr/bin/fallocate"))("the kernel refuses what passes them", () => {
50 // 2048 blocks of 512 bytes: a 1 MiB cell.
51 const oneMiB = { fileBlocks: 2048, files: 64, tasks: TASKS_MAX };
52
53 test("fallocate past the cell's disk fails, and nothing was allocated", async () => {
54 const r = await underLimits(oneMiB, "fallocate -l 50G big; echo $?");
55 // SIGXFSZ: 128 + 25.
56 expect(Number(r.out)).not.toBe(0);
57 expect(statSync(join(dir, "big")).size).toBe(0);
58 expect(statSync(join(dir, "big")).blocks).toBe(0);
59 });
60
61 test("dd past the cell's disk stops at the cap", async () => {
62 const r = await underLimits(oneMiB, "dd if=/dev/zero of=dd.out bs=1M count=5 2>/dev/null; echo $?");
63 expect(Number(r.out)).not.toBe(0);
64 expect(statSync(join(dir, "dd.out")).size).toBeLessThanOrEqual(1024 * 1024);
65 });
66
67 test("a file under the cap is written whole", async () => {
68 const r = await underLimits(oneMiB, "dd if=/dev/zero of=ok.out bs=1k count=512 2>/dev/null; echo $?");
69 expect(r.out).toBe("0");
70 expect(statSync(join(dir, "ok.out")).size).toBe(512 * 1024);
71 });
72
73 test("open files past the limit are refused (EMFILE)", async () => {
74 const script = `exec ${JSON.stringify(process.execPath)} -e 'const fs=require("fs");let n=0;try{for(;n<200;n++)fs.openSync("/dev/null","r")}catch(e){console.log(e.code)}'`;
75 const r = await underLimits(oneMiB, script);
76 expect(r.out).toBe("EMFILE");
77 });
78
79 test("nothing inside raises a limit back: soft and hard were both set", async () => {
80 const r = await underLimits(oneMiB, "ulimit -f unlimited 2>/dev/null; ulimit -f; ulimit -n 100000 2>/dev/null; ulimit -n");
81 expect(r.out.split("\n")).toEqual(["2048", "64"]);
82 });
83
84 test("the task limit is set", async () => {
85 const r = await underLimits({ ...oneMiB, tasks: 300 }, "ulimit -u 2>/dev/null || ulimit -p");
86 expect(r.out).toBe("300");
87 });
88 });
89
90 describe("a cell's hostname is its own", () => {
91 test("a child's cell is named for the child, never the parent's pod", () => {
92 const spec = cellSpecFor({
93 dir: join(dir, "kid"),
94 parentDataDir: dir,
95 parentHome: join(dir, "home"),
96 parentUrl: "https://f.test",
97 size: "small",
98 port: 4000,
99 label: "forge:kid",
100 name: "kid",
101 });
102 expect(spec.hostname).toBe("kid");
103 expect(cellHostname(spec.hostname)).toBe("kid");
104 });
105
106 test("a name becomes a DNS label, and nothing becomes 'cell'", () => {
107 expect(cellHostname("Scottish Bob's forge")).toBe("scottish-bob-s-forge");
108 expect(cellHostname("x".repeat(80))).toHaveLength(63);
109 expect(cellHostname("--")).toBe("cell");
110 expect(cellHostname(undefined)).toBe("cell");
111 expect(cellHostname("a;$(reboot)")).toBe("a-reboot");
112 });
113
114 test("a host that refuses sethostname gets no --hostname, so the cell still starts", () => {
115 // The check machine's bwrap answers `--hostname` with EPERM; passing it
116 // anyway kept every child cell from coming up.
117 expect(hostnameArgs("kid", true)).toEqual(["--hostname", "kid"]);
118 expect(hostnameArgs("kid", false)).toEqual([]);
119 });
120 });
121
122 describe("a small child's cgroup, as written", () => {
123 // A stand-in for the delegated cgroup: cgroupfs files are plain files to write.
124 test("memory.max 2 GiB, pids.max, and cpu.max one core whatever the host's count", () => {
125 const root = join(dir, "cg");
126 mkdirSync(join(root, "cells"), { recursive: true });
127 writeFileSync(join(root, "cells", "cgroup.subtree_control"), "memory pids cpu io");
128 const spec = cellSpecFor({
129 dir: join(dir, "kid2"),
130 parentDataDir: dir,
131 parentHome: join(dir, "home"),
132 parentUrl: "https://f.test",
133 size: "small",
134 port: 4001,
135 label: "forge:kid2",
136 });
137 const made = cellCgroup(spec, { ok: true, root });
138 expect(made).toBeDefined();
139 const at = (file: string) => readFileSync(join(made!.dir, file), "utf8");
140 // 2048 MiB × 1048576 = 2147483648 bytes.
141 expect(at("memory.max")).toBe("2147483648");
142 expect(at("pids.max")).toBe(String(PROCESSES_MAX));
143 // One core is 100000 µs of every 100000 µs period: an absolute quota, so 64 cores on the host change nothing.
144 expect(at("cpu.max")).toBe("100000 100000");
145 expect(at("io.weight")).toBe("default 100");
146 expect(made!.cpuCapped).toBe(true);
147 });
148 });
test/disk-floor.test.ts added +88
@@ -0,0 +1,88 @@
1 /**
2 * #10850 item 5: the parent protects itself. Below 15% free on its own
3 * volume no child forge is made, no child cell starts, and the owner reads
4 * one line in vitals. Each is tried against a disk this test hands in.
5 */
6
7 import { afterAll, afterEach, describe, expect, test } from "bun:test";
8 import { mkdtempSync, rmSync } from "node:fs";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { createApp } from "../src/app.ts";
12 import { provideDiskForTest, type DiskReading } from "../src/kernel/disk_reading.ts";
13 import { vitals } from "../src/kernel/vitals.ts";
14 import { cellChildren, stopCellChildren } from "../src/spawn/cell.ts";
15 import { DISK_FLOOR, floorRefusal } from "../src/spawn/disk_floor.ts";
16 import { quotaFull, quotaRefusal } from "../src/spawn/quota.ts";
17
18 const tmp = mkdtempSync(join(tmpdir(), "disk-floor-"));
19 const GiB = 1024 ** 3;
20 const reading = (freeGiB: number, totalGiB = 100): DiskReading => ({ freeBytes: freeGiB * GiB, totalBytes: totalGiB * GiB });
21 const roomy = (): DiskReading => reading(900, 1000);
22
23 afterEach(() => provideDiskForTest(roomy));
24 afterAll(() => rmSync(tmp, { recursive: true, force: true }));
25
26 describe("floorRefusal", () => {
27 test("the floor is 15%", () => expect(DISK_FLOOR).toBe(0.15));
28
29 test("14.9 GiB free of 100 is under it, and the sentence says the figures", () => {
30 expect(floorRefusal(tmp, reading(14.9))).toBe(
31 "this host's disk has 14.9% free (14.9 of 100.0 GiB), under the 15% it keeps for itself: nothing new starts until room is made",
32 );
33 });
34
35 test("exactly 15% free is not under it", () => expect(floorRefusal(tmp, reading(15))).toBeUndefined());
36
37 test("the reported pod (314 GiB free of 669) is above it; at 90 GiB free it is not", () => {
38 expect(floorRefusal(tmp, reading(314, 669))).toBeUndefined();
39 expect(floorRefusal(tmp, reading(90, 669))).toContain("13.5% free");
40 });
41
42 test("a disk that cannot be read or reads as empty is not refused", () => {
43 expect(floorRefusal(tmp, undefined)).toBeUndefined();
44 expect(floorRefusal(tmp, { freeBytes: 0, totalBytes: 0 })).toBeUndefined();
45 });
46
47 test("with no reading handed in, it reads the forge's disk reader", () => {
48 provideDiskForTest(() => reading(10));
49 expect(floorRefusal(tmp)).toContain("10.0% free");
50 });
51 });
52
53 describe("below the floor", () => {
54 test("no child forge is made, whatever limits are declared, and the New forge page says why", () => {
55 provideDiskForTest(() => reading(5));
56 expect(quotaRefusal(tmp, "small", {})).toContain("under the 15% it keeps for itself");
57 expect(quotaFull(tmp, {})).toContain("under the 15% it keeps for itself");
58 provideDiskForTest(roomy);
59 expect(quotaRefusal(tmp, "small", {})).toBeUndefined();
60 });
61
62 test("a child cell does not start, and its failure names the floor", async () => {
63 const app = createApp({ dataDir: join(tmp, "parent"), url: "https://f.test" });
64 try {
65 provideDiskForTest(() => reading(12));
66 const children = cellChildren(app);
67 expect(await children.start({ bodyId: "cell-floor", name: "kid", size: "small" })).toBeUndefined();
68 expect(children.failure("cell-floor")).toContain("12.0% free");
69 expect(children.list()).toEqual([]);
70 } finally {
71 await stopCellChildren(app);
72 app.close();
73 }
74 });
75
76 test("the owner reads one line in vitals between the floor and nearly full", () => {
77 const app = createApp({ dataDir: join(tmp, "vitals"), url: "https://f.test" });
78 try {
79 const v = vitals({ db: app.db, signals: app.signals, dataDir: app.config.dataDir, disk: () => ({ freeBytes: 12, totalBytes: 100 }) });
80 expect(v.degraded.map((d) => d.cause)).toEqual(["disk-floor"]);
81 expect(v.needs).toEqual(["Disk is under the 15% the forge keeps for itself: 12% free. No child forge is made or started until room is made."]);
82 const at15 = vitals({ db: app.db, signals: app.signals, dataDir: app.config.dataDir, disk: () => ({ freeBytes: 15, totalBytes: 100 }) });
83 expect(at15.degraded.map((d) => d.cause)).toEqual([]);
84 } finally {
85 app.close();
86 }
87 });
88 });
test/disk-mount.test.ts +2
@@ -237,6 +237,8 @@ describe("what the forge launches and boots", () => {
237237 const own = launchSpecFor(parent, input, at, `${ROOT}\n731 22 7:3 / ${real} rw - ext4 /dev/loop3 rw\n`);
238238 expect(own.writable).toEqual([real]);
239239 expect(launchSpecFor(parent, input, at, `${ROOT}\n`).writable).toEqual([]);
240 // The launch path names the cell after the child, not the parent's pod (#10850).
241 expect(own.hostname).toBe("kid");
240242 });
241243
242244 test("the boot does not open a child on the shared disk when one of its own is required, and opens it when not", async () => {
test/helpers/apps-forge.ts +1
@@ -16,6 +16,7 @@ import type { Repo } from "../../src/core/repos.ts";
1616 import { listRuns, type Run } from "../../src/core/runs.ts";
1717 import { createUser, type User } from "../../src/core/users.ts";
1818 import { serve, type Servers } from "../../src/http/server.ts";
19 import "./roomy-disk.ts";
1920
2021 export const APPS_DOMAIN = "apps.test";
2122
test/helpers/big-forge.ts +1
@@ -8,6 +8,7 @@ import { mkdtempSync } from "node:fs";
88 import { tmpdir } from "node:os";
99 import { join } from "node:path";
1010 import { createApp, type App } from "../../src/app.ts";
11 import "./roomy-disk.ts";
1112
1213 export interface BigForgeSize {
1314 readonly issues: number;
test/helpers/checks-forge.ts +1
@@ -25,6 +25,7 @@ import type { Repo } from "../../src/core/repos.ts";
2525 import { serve, type Servers } from "../../src/http/server.ts";
2626 import { listEvents } from "../../src/core/events.ts";
2727 import { promote } from "./members.ts";
28 import "./roomy-disk.ts";
2829
2930 /** A forge.json declaring `run` as the check, with no install step. */
3031 export const checkJson = (run: string, extra = ""): string =>
test/helpers/claimed-forge.ts +1
@@ -19,6 +19,7 @@ import { firstBoot } from "../../src/spawn/first-boot.ts";
1919 import { hostedPlacement } from "../../src/spawn/placement.ts";
2020 import { passkeyFields } from "./passkey.ts";
2121 import { seatAs } from "./seat.ts";
22 import "./roomy-disk.ts";
2223
2324 export const PASSWORD = "a-long-next-steps-card-passphrase";
2425 const FORGE_JSON = JSON.stringify({
test/helpers/crew-standin.ts +1
@@ -24,6 +24,7 @@ import { listRuns, type Run } from "../../src/core/runs.ts";
2424 import { createUser, type User } from "../../src/core/users.ts";
2525 import type { InstallRunner } from "../../src/crew/workdir.ts";
2626 import { serve, type Servers } from "../../src/http/server.ts";
27 import "./roomy-disk.ts";
2728
2829 export interface StandInCrew {
2930 readonly dataDir: string;
test/helpers/keeper.ts +1
@@ -9,6 +9,7 @@ import { id } from "../../src/kernel/id.ts";
99 import type { Deployer } from "../../src/keeper/deploy.ts";
1010 import type { KeeperPorts } from "../../src/keeper/keeper.ts";
1111 import { verifyUnderLoad, type Probe } from "../../src/keeper/verify.ts";
12 import "./roomy-disk.ts";
1213
1314 export const A = "a".repeat(40);
1415 export const B = "b".repeat(40);
test/helpers/listed-forge.ts +1
@@ -9,6 +9,7 @@ import { createApp, type App } from "../../src/app.ts";
99 import { createSession } from "../../src/core/sessions.ts";
1010 import { forgeHttp, type ForgeHttp } from "../../src/http/server.ts";
1111 import { seedAdmin } from "./admin.ts";
12 import "./roomy-disk.ts";
1213
1314 export interface ListedForge {
1415 app: App;
test/helpers/overlap-worker.ts +1
@@ -16,6 +16,7 @@ import { getRepo } from "../../src/core/repos.ts";
1616 import { deliverDue } from "../../src/core/webhooks.ts";
1717 import { LOOPBACK } from "./loopback-hook.ts";
1818 import { queueAgentRun } from "../../src/crew/dispatch.ts";
19 import "./roomy-disk.ts";
1920
2021 const [mode, dataDir, startAt, count] = process.argv.slice(2);
2122 const app = createApp({ dataDir: dataDir!, port: 0 });
test/helpers/parks-engine.ts +1
@@ -15,6 +15,7 @@ import { createUser, getUserByName } from "../../src/core/users.ts";
1515 import { serve } from "../../src/http/server.ts";
1616 import { settleAgentRun } from "../../src/crew/autonomy.ts";
1717 import { interruptedSummary } from "../../src/crew/interrupted.ts";
18 import "./roomy-disk.ts";
1819
1920 /**
2021 * One forge with alice's `engine` on its own: atlas plans, mason builds.
test/helpers/preload.ts +1
@@ -7,6 +7,7 @@ import { provideCpusForTest } from "../../src/core/capacity.ts";
77 import { provideMirrorOutboundForTest } from "../../src/origins/git-mirror.ts";
88 import { installCheapHash } from "./cheap-hash.ts";
99 import { useReachByDefault } from "../../src/spawn/reach.ts";
10 import "./roomy-disk.ts";
1011
1112 installCheapHash();
1213 // Defaults sized from CPUs (#2549) read as on a 4-CPU pod — 4 shards, one
test/helpers/real-child-forge.ts +1
@@ -15,6 +15,7 @@ import { createApp } from "../../src/app.ts";
1515 import { cellCounters, cellsAvailable } from "../../src/apps/contain.ts";
1616 import { cellChildren, stopCellChildren } from "../../src/spawn/cell.ts";
1717 import { answerFromCell } from "../../src/spawn/host.ts";
18 import "./roomy-disk.ts";
1819
1920 const cells = await cellsAvailable();
2021 if (!cells.ok) {
test/helpers/real-spawn-e2e.ts +1
@@ -22,6 +22,7 @@ import { serve } from "../../src/http/server.ts";
2222 import { cellPlacement, stopCellChildren } from "../../src/spawn/cell.ts";
2323 import { bootHostedChild } from "../../src/spawn/host.ts";
2424 import { memoryBodies } from "../../src/spawn/placement.ts";
25 import "./roomy-disk.ts";
2526
2627 // Two budgets, each its own: spawn+boot may take SPAWN_MS, then the poll gets
2728 // POLL_MS counted from when boot ended. A slow spawn never shortens the poll,
test/helpers/roomy-disk.ts added +12
@@ -0,0 +1,12 @@
1 /**
2 * The suite runs on machines whose disks are whatever they are: every
3 * default disk reading (vitals, the disk floor of #10850) sees a roomy one,
4 * so no test is refused by the machine it ran on, and a test of a full disk
5 * gives its own reading. The preload imports it; so does every helper the
6 * suite starts as a process of its own (`real-child-forge.ts` and the like),
7 * which no preload reaches: a check host near 15% free must not refuse the
8 * child a cell test boots.
9 */
10 import { provideDiskForTest } from "../../src/kernel/disk_reading.ts";
11
12 provideDiskForTest(() => ({ freeBytes: 900 * 1024 ** 3, totalBytes: 1000 * 1024 ** 3 }));

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs