trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

A check never takes the forge down: each check cell has a real memory cap, and test processes stay small #2426

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/2410-a-check-never-takes-the-forge-down-each → main 2 commits fork 1 thread · 1 unresolved

  • Merged. torque/2410-a-check-never-takes-the-forge-down-each was merged into main as 5b73d91.

Review

Sign in to review this change.
14 files changed +646 −21
docs/APPS.md +17
@@ -261,6 +261,22 @@ are capped by the machine (`FORGE_CELL_MEMORY_MAX_MIB`, default 4096; the
261261 number of cores; `FORGE_CELL_DISK_MAX_MIB`, default 16384). A cell's `/tmp`
262262 holds at most 256 MiB.
263263
264 Memory is held twice. The watch reads every cell's resident memory once a
265 second and stops one over its limit. Where the forge holds a delegated
266 cgroup v2 (`apps/cgroup.ts`), the kernel holds it as well: each cell runs
267 in `<R>/cells/c<n>-<pid>` with `memory.max` set to its limit,
268 `memory.swap.max` 0 and `memory.oom.group` 1, and the forge moves itself
269 to `<R>/forge`. A check that allocates past its cap faster than a tick is
270 then OOM-killed alone (`check stopped: it reached its 8192 MiB memory cap,
271 and the kernel killed it`), never the forge or the other checks (#2410).
272 `R` is `/sys/fs/cgroup` when the forge is at its container's cgroup
273 namespace root, or `FORGE_CELL_CGROUP` (a path, or `off`). The cgroup has to
274 be writable by the forge's user. A container's `/sys/fs/cgroup` is
275 read-only by default, so the operator delegates it, for example as a
276 writable cgroup mount owned by the forge's uid. Without that, cells are
277 watched only. /system says which holds the cap, and why, and shows each
278 cell's memory beside its cap.
279
264280 Where the host cannot hold cells (or the crew's shell sandbox), the forge
265281 runs uncontained and says so in one line: a boot warning, and, in the foot of
266282 every page a site admin reads, `Reduced protection: apps (<reason>) runs
@@ -1103,6 +1119,7 @@ a link to the milestone (`docs/PLANNING.md`); its progress never does.
11031119 | `FORGE_CHECKS_DRAIN_S` | `300` | seconds a roll lets running checks finish first |
11041120 | `FORGE_CHECK_SHARDS` | the machine's cores | test shards per check (live: Settings → Capacity) |
11051121 | `FORGE_CELLS_EGRESS` | `enforce` | `report` lets undeclared egress through on 80/443 (live: Settings → Capacity) |
1122 | `FORGE_CELL_CGROUP` | `/sys/fs/cgroup` at a namespace root | the delegated cgroup v2 whose children cap each cell's memory; `off` watches only |
11061123 | `FORGE_CREW_BUDGET_USD` | `5` | spend per run unless the agent says otherwise |
11071124 | `FORGE_CREW_TIMEOUT_MIN` | `45` | wall-clock limit of one agent run |
11081125 | `FORGE_CREW_ALLOW_HOSTS` | — | hosts an agent's shell may reach, comma-separated |
scripts/shards.ts +30
@@ -157,6 +157,36 @@ export function splitFiles(
157157 );
158158 }
159159
160 /**
161 * The most test files one `bun test` process runs. A process keeps what
162 * each file leaves behind (one global per process), and the forge was
163 * OOM-killed with 29 of them at 230–380 MiB each, ~110 files apiece
164 * (#2410). Measured with VmHWM on Bun 1.3.14, light gate files in name
165 * order: 40 files peaked at 548 MiB and 120 at 549 MiB — a process's peak
166 * is its heaviest file's plus what the files before it left, so 40 keeps
167 * that under 1 GiB with room, and costs a spawn (~0.1 s) per 40 files.
168 */
169 export const FILES_PER_PROCESS = 40;
170
171 /**
172 * {@link splitFiles}, with no slice over `max` files: enough slices that
173 * every one fits, balanced by weight, and any still over (many light files
174 * on one slice) cut in order into runs of `max`. A runner runs at most
175 * `n` of them at once; this decides only what each process holds.
176 */
177 export function splitCapped(
178 files: readonly string[],
179 n: number,
180 weight: (file: string) => number = () => 1,
181 max = FILES_PER_PROCESS,
182 ): string[][] {
183 const per = Math.max(1, Math.floor(max));
184 const slices = splitFiles(files, Math.max(n, Math.ceil(files.length / per)), weight);
185 return slices.flatMap((slice) =>
186 Array.from({ length: Math.ceil(slice.length / per) }, (_, i) => slice.slice(i * per, (i + 1) * per)),
187 );
188 }
189
160190 /** How a test file is named: the same rule `scripts/test-shards.ts` uses. */
161191 export const TEST_FILE = /\.test\.[cm]?[jt]sx?$/;
162192
scripts/test-shards.ts +4 −3
@@ -55,7 +55,7 @@ import {
5555 shardCountFromEnv,
5656 sliceFiles,
5757 splitArgs,
58 splitFiles,
58 splitCapped,
5959 tally,
6060 weigher,
6161 type ShardResult,
@@ -381,7 +381,7 @@ const perFile = record || measure || serialFiles || timed;
381381 const slices =
382382 perFile
383383 ? heaviestFirst(files, weight).map((file) => [file])
384 : splitFiles(files, count.value, weight);
384 : splitCapped(files, count.value, weight);
385385 const work = files.reduce((sum, file) => sum + weight(file), 0);
386386
387387 const banner = record
@@ -537,7 +537,8 @@ const started = performance.now();
537537 // Serial one-file processes when SHARDS=1; record keeps its worker width;
538538 // multi-shard still runs every slice side by side.
539539 const results = await pool(
540 serialFiles ? 1 : record || measure || timed ? count.value : slices.length,
540 // Never more side by side than the count: capped slices outnumber it.
541 serialFiles ? 1 : Math.min(count.value, slices.length),
541542 );
542543 if (results.length < slices.length) {
543544 const skipped = slices.length - results.length;
src/apps/cgroup.ts added +231
@@ -0,0 +1,231 @@
1 /**
2 * A cell's memory cap, held by the kernel: each cell gets a cgroup v2 of
3 * its own under the forge's, with `memory.max` set to the cell's memory
4 * limit, so a cell that allocates past it is OOM-killed alone — never the
5 * forge, never its neighbours (#2410). The watch in `apps/contain.ts`
6 * still reads every cell once a second; this is the floor under it, for
7 * a cell that allocates faster than a second's tick.
8 *
9 * It needs the forge's own cgroup delegated to it: cgroup v2, mounted
10 * read-write, owned by the forge's user, with the memory controller
11 * available. A container gets that from its runtime (a delegated
12 * subtree, or a writable `/sys/fs/cgroup`); without it, cells are
13 * watched only, and `cgroupHost()` says why (shown on /system).
14 *
15 * Layout under the forge's cgroup `R` (cgroup v2's "no internal
16 * processes" rule: a cgroup that hands memory to children holds no
17 * process of its own):
18 *
19 * R/forge/ the forge and everything it starts outside a cell
20 * R/cells/c<n>/ one per cell: memory.max, memory.swap.max = 0,
21 * memory.oom.group = 1 (the whole cell dies together)
22 *
23 * `FORGE_CELL_CGROUP` names `R` (an absolute path) or turns this `off`;
24 * unset, `R` is `/sys/fs/cgroup` when the forge is at the root of its
25 * cgroup namespace (`0::/` in `/proc/self/cgroup`: a container's own).
26 */
27
28 import {
29 accessSync,
30 constants,
31 existsSync,
32 mkdirSync,
33 readdirSync,
34 readFileSync,
35 rmdirSync,
36 writeFileSync,
37 } from "node:fs";
38 import { join } from "node:path";
39
40 export type CgroupHost =
41 | { readonly ok: true; readonly root: string }
42 | { readonly ok: false; readonly reason: string };
43
44 /** The cgroup a cell's process is in, open for the life of the cell. */
45 export interface CellCgroup {
46 readonly dir: string;
47 /** The cap, bytes (`memory.max`). */
48 readonly maxBytes: number;
49 /** How many times the kernel OOM-killed in it (`memory.events` `oom_kill`). */
50 readonly oomKills: () => number;
51 /** Kill whatever is left and remove the cgroup, retrying while it empties. */
52 readonly remove: () => void;
53 }
54
55 let host: CgroupHost | undefined;
56 let nextCell = 1;
57
58 /** Whether cells get a kernel memory cap here, decided once (and the layout prepared). */
59 export function cgroupHost(): CgroupHost {
60 host ??= prepare(cgroupRoot());
61 return host;
62 }
63
64 /** The forge's own cgroup directory, or why there is none to use. */
65 export function cgroupRoot(
66 env: string | undefined = process.env["FORGE_CELL_CGROUP"],
67 self: () => string = () => readFileSync("/proc/self/cgroup", "utf8"),
68 ): { ok: true; root: string } | { ok: false; reason: string } {
69 if (env === "off") return { ok: false, reason: "FORGE_CELL_CGROUP=off" };
70 if (env !== undefined && env !== "") {
71 if (!env.startsWith("/") || env.split("/").includes(".."))
72 return { ok: false, reason: `FORGE_CELL_CGROUP must be an absolute path, not ${JSON.stringify(env)}` };
73 return { ok: true, root: env };
74 }
75 if (process.platform !== "linux") return { ok: false, reason: `no cgroups on ${process.platform}` };
76 let text: string;
77 try {
78 text = self();
79 } catch {
80 return { ok: false, reason: "/proc/self/cgroup is unreadable" };
81 }
82 // cgroup v2 is the one line `0::<path>`; v1 has a line per controller.
83 const line = text.split("\n").find((l) => l.startsWith("0::"));
84 if (line === undefined) return { ok: false, reason: "this host has no cgroup v2" };
85 // At the root of its own cgroup namespace (a container), the cgroup is
86 // the container's to divide. Anywhere else (a developer's login session)
87 // it is shared with processes that are not the forge's: only when named.
88 if (line.slice(3).trim() !== "/")
89 return { ok: false, reason: "not at a cgroup namespace root; FORGE_CELL_CGROUP names a delegated cgroup" };
90 return { ok: true, root: "/sys/fs/cgroup" };
91 }
92
93 const read = (path: string): string | undefined => {
94 try {
95 return readFileSync(path, "utf8");
96 } catch {
97 return undefined;
98 }
99 };
100
101 const writable = (path: string): boolean => {
102 try {
103 accessSync(path, constants.W_OK);
104 return true;
105 } catch {
106 return false;
107 }
108 };
109
110 const words = (text: string | undefined): string[] => (text ?? "").trim().split(/\s+/).filter((w) => w !== "");
111
112 /**
113 * Make `R/forge` and `R/cells` and give `cells` the memory controller,
114 * moving the forge's processes out of `R` first. Idempotent: a forge that
115 * restarts finds the layout standing, and removes the cells of a forge
116 * that is gone — never those of one still draining its checks in a roll.
117 */
118 export function prepare(
119 found: { ok: true; root: string } | { ok: false; reason: string },
120 ): CgroupHost {
121 if (!found.ok) return found;
122 const { root } = found;
123 if (!words(read(join(root, "cgroup.controllers"))).includes("memory"))
124 return { ok: false, reason: `${root} has no memory controller (cgroup v2 not delegated here)` };
125 for (const file of ["", "cgroup.procs", "cgroup.subtree_control"])
126 if (!writable(join(root, file)))
127 return { ok: false, reason: `${root} is read-only to the forge (no delegated cgroup)` };
128 try {
129 if (!words(read(join(root, "cgroup.subtree_control"))).includes("memory")) {
130 const leaf = join(root, "forge");
131 mkdirSync(leaf, { recursive: true });
132 // One pid per write: cgroupfs takes them one at a time.
133 for (const pid of words(read(join(root, "cgroup.procs")))) {
134 try {
135 writeFileSync(join(leaf, "cgroup.procs"), pid);
136 } catch {
137 // gone between the read and the write, or a kernel thread
138 }
139 }
140 writeFileSync(join(root, "cgroup.subtree_control"), "+memory");
141 }
142 const cells = join(root, "cells");
143 mkdirSync(cells, { recursive: true });
144 writeFileSync(join(cells, "cgroup.subtree_control"), "+memory");
145 for (const name of readdirSync(cells)) {
146 const owner = /^c\d+-(\d+)$/.exec(name)?.[1];
147 if (owner !== undefined && !alive(Number(owner))) removeDir(join(cells, name));
148 }
149 } catch (error) {
150 return { ok: false, reason: `cannot lay out ${root}: ${error instanceof Error ? error.message : String(error)}` };
151 }
152 return { ok: true, root };
153 }
154
155 function alive(pid: number): boolean {
156 try {
157 process.kill(pid, 0);
158 return true;
159 } catch (error) {
160 return (error as NodeJS.ErrnoException).code === "EPERM";
161 }
162 }
163
164 /** Kill what is in a cell's cgroup and remove it; a few tries while the processes go. */
165 function removeDir(dir: string, tries = 10): void {
166 try {
167 if (existsSync(join(dir, "cgroup.kill"))) writeFileSync(join(dir, "cgroup.kill"), "1");
168 } catch {
169 // already empty, or the kernel has no cgroup.kill (before 5.14)
170 }
171 try {
172 rmdirSync(dir);
173 } catch (error) {
174 const code = (error as NodeJS.ErrnoException).code;
175 if (code === "ENOENT" || tries <= 1) return;
176 // EBUSY: a killed process is not reaped yet.
177 setTimeout(() => removeDir(dir, tries - 1), 200).unref();
178 }
179 }
180
181 /** A cgroup of its own for one cell, capped at `memoryMiB`; `undefined` where the host has none. */
182 export function openCellCgroup(memoryMiB: number, at: CgroupHost = cgroupHost()): CellCgroup | undefined {
junipercrew outdated

current() has no caller in src. Remove it, or surface it on /system.
183 if (!at.ok) return undefined;
184 const dir = join(at.root, "cells", `c${nextCell++}-${process.pid}`);
185 const maxBytes = Math.floor(memoryMiB * 1024 * 1024);
186 try {
187 mkdirSync(dir);
188 writeFileSync(join(dir, "memory.max"), String(maxBytes));
189 } catch {
190 removeDir(dir, 1);
191 return undefined;
192 }
193 // Swap would only let a cell outlive its cap slowly; one OOM kills the cell, not a process of it.
194 for (const [file, value] of [["memory.swap.max", "0"], ["memory.oom.group", "1"]] as const) {
195 try {
196 if (existsSync(join(dir, file))) writeFileSync(join(dir, file), value);
197 } catch {
198 // an older kernel: the cap still holds
199 }
200 }
201 return {
202 dir,
203 maxBytes,
204 oomKills: () => oomKills(read(join(dir, "memory.events"))),
205 remove: () => removeDir(dir),
206 };
207 }
208
209 /** `oom_kill` from a `memory.events` file; 0 when it is absent. */
210 export function oomKills(events: string | undefined): number {
211 const match = /^oom_kill (\d+)$/m.exec(events ?? "");
212 return match ? Number(match[1]) : 0;
213 }
214
215 /**
216 * The argv that moves itself into the cgroup whose `cgroup.procs` is
217 * `procs`, then becomes `argv` — so the cell and every process it starts
218 * are under the cap from their first instruction. If it cannot enter, it
219 * refuses to run rather than run uncapped. No caller text reaches the
220 * shell: the path and the argv are arguments, never script.
221 */
222 export function enterArgv(procs: string, argv: readonly string[]): string[] {
223 return [
224 "/bin/sh",
225 "-c",
226 'echo $$ > "$1" || { echo "cell: cannot enter its memory cgroup" >&2; exit 125; }; shift; exec "$@"',
227 "cgroup-enter",
228 procs,
229 ...argv,
230 ];
231 }
src/apps/contain.ts +45 −9
@@ -12,9 +12,11 @@
1212 * else. The way out is the forge's proxy (`apps/egress.ts`), which reaches
1313 * the forge and the hosts the repository declared, and refuses the rest.
1414 * The way in, for an app, is a relay from the port the forge proxies to.
15 * - **Limits.** Memory, CPU and disk are watched from outside (a container
16 * cannot create cgroups), and a cell over any of them is stopped with the
17 * reason, which becomes the step's error or the app's last log line.
15 * - **Limits.** Memory, CPU and disk are watched from outside, and a cell
16 * over any of them is stopped with the reason, which becomes the step's
17 * error or the app's last log line. Where the forge holds a delegated
18 * cgroup (`apps/cgroup.ts`), memory is also capped by the kernel: a cell
19 * that allocates past it between two ticks is OOM-killed alone.
1820 *
1921 * Where the host cannot hold a cell (no bubblewrap, no user namespaces —
2022 * macOS, or a locked-down container), the code runs as before and every
@@ -35,6 +37,7 @@ import { availableParallelism, tmpdir } from "node:os";
3537 import { dirname, join } from "node:path";
3638 import { portInsteadOfSocket } from "../git/hook-address.ts";
3739 import { relay, startEgress, type Egress } from "./egress.ts";
40 import { cgroupHost, enterArgv, openCellCgroup } from "./cgroup.ts";
3841
3942 export interface Limits {
4043 /** Resident memory of every process in the cell together, MiB. */
@@ -316,7 +319,12 @@ export function openCell(
316319 // small and bounded, and the ones that only exist to hide are read-only.
317320 const SMALL = String(1 << 20);
318321
319 const argv = (command: string): string[] => [
322 const cgroup = openCellCgroup(spec.limits.memoryMiB);
323 const argv = (command: string): string[] => {
324 const inner = bwrapArgv(command);
325 return cgroup === undefined ? inner : enterArgv(join(cgroup.dir, "cgroup.procs"), inner);
326 };
327 const bwrapArgv = (command: string): string[] => [
320328 "bwrap",
321329 ...NAMESPACES,
322330 ...(spec.terminal === true ? [] : ["--new-session"]),
@@ -365,8 +373,17 @@ export function openCell(
365373
366374 let breach: string | undefined;
367375 let unwatch: (() => void) | undefined;
376 // The kernel's kill leaves no watch tick behind: its count is the reason.
377 const killed = (): string | undefined =>
378 cgroup !== undefined && cgroup.oomKills() > 0
379 ? `reached its ${spec.limits.memoryMiB} MiB memory cap, and the kernel killed it`
380 : undefined;
368381 const watch = (pid: number, stop: (why: string) => void): void => {
369 unwatch = watchTree(pid, spec.limits, writable, spec.label, (why) => {
382 const cap: MemoryCap =
383 cgroup === undefined
384 ? { bytes: Math.floor(spec.limits.memoryMiB * 1024 * 1024), by: "watch" }
385 : { bytes: cgroup.maxBytes, by: "kernel" };
386 unwatch = watchTree(pid, spec.limits, cap, writable, spec.label, (why) => {
370387 if (breach !== undefined) return;
371388 breach = why;
372389 log(`stopped: ${why}`);
@@ -377,8 +394,11 @@ export function openCell(
377394 argv,
378395 env,
379396 watch,
380 breach: () => breach,
397 breach: () => breach ?? killed(),
381398 close: () => {
399 const kernel = breach === undefined ? killed() : undefined;
400 if (kernel !== undefined) log(`stopped: ${kernel}`);
401 cgroup?.remove();
382402 unwatch?.();
383403 for (const o of opened) o.close();
384404 rmSync(sockets, { recursive: true, force: true });
@@ -514,6 +534,8 @@ interface Watched {
514534 ticks: number;
515535 readonly key: number;
516536 readonly label: string | undefined;
537 /** The memory cap, and who holds it: the kernel (a cgroup) or this watch alone. */
538 readonly cap: MemoryCap;
517539 /** The tree's CPU ticks so far and resident KiB, at the last tick. */
518540 last?: { ticks: number; kib: number };
519541 }
@@ -522,6 +544,18 @@ const watched = new Set<Watched>();
522544 let timer: ReturnType<typeof setInterval> | undefined;
523545 let nextKey = 1;
524546
547 /** A cell's memory cap, bytes, and who holds it: the kernel (`apps/cgroup.ts`) or the watch alone. */
548 export interface MemoryCap {
549 readonly bytes: number;
550 readonly by: "kernel" | "watch";
551 }
552
553 /** Whether cells' memory caps are the kernel's here, or why they are watched only. */
554 export function memoryCapHolder(): { by: "kernel" } | { by: "watch"; reason: string } {
555 const host = cgroupHost();
556 return host.ok ? { by: "kernel" } : { by: "watch", reason: host.reason };
557 }
558
525559 /** One labelled cell as its watch last read it: cumulative CPU and resident memory. */
526560 export interface CellCounter {
527561 /** Unique per watched tree, so a rate never spans two processes. */
@@ -641,6 +675,7 @@ function descendants(root: number): number[] {
641675 function watchTree(
642676 root: number,
643677 limits: Limits,
678 cap: MemoryCap,
644679 writable: readonly string[],
645680 label: string | undefined,
646681 breach: (why: string) => void,
@@ -654,6 +689,7 @@ function watchTree(
654689 ticks: 0,
655690 key: nextKey++,
656691 label,
692 cap,
657693 };
658694 watched.add(entry);
659695 timer ??= setInterval(tick, WATCH.tickMs);
@@ -808,8 +844,8 @@ function tick(): void {
808844 }
809845
810846 /** The root process of every labelled cell being watched, for grouping the process table by what launched it (`system/view.ts`). */
811 export function cellRoots(): { label: string; root: number }[] {
812 const out: { label: string; root: number }[] = [];
813 for (const e of watched) if (e.label !== undefined) out.push({ label: e.label, root: e.root });
847 export function cellRoots(): { label: string; root: number; cap: MemoryCap }[] {
848 const out: { label: string; root: number; cap: MemoryCap }[] = [];
849 for (const e of watched) if (e.label !== undefined) out.push({ label: e.label, root: e.root, cap: e.cap });
814850 return out;
815851 }
src/apps/process.ts +3
@@ -294,6 +294,9 @@ export async function runStep(
294294 proc.exited,
295295 ]);
296296 if (why !== undefined) return err(why);
297 // The kernel's OOM kill at the cell's cap is a breach no watch tick saw.
298 const kernel = code === 0 ? undefined : cell?.breach();
299 if (kernel !== undefined) return err(`${name} stopped: it ${kernel}`);
297300 if (code !== 0) {
298301 const reason = recent.length > 0 ? ` — ${failureReason(recent)}` : "";
299302 return err(`${name} exited with code ${code}${reason}`);
src/system/live.ts +2 −2
@@ -8,7 +8,7 @@
88 import type { App } from "../app.ts";
99 import { cellRoots } from "../apps/contain.ts";
1010 import { HOST, readRaw, type Paths, type Raw } from "./proc.ts";
11 import { buildSystem, forAgent, type SystemView } from "./view.ts";
11 import { buildSystem, forAgent, type Roots, type SystemView } from "./view.ts";
1212
1313 const FRESH_MS = 1_000;
1414
@@ -16,7 +16,7 @@ export type Live = {
1616 read(now?: number): SystemView;
1717 };
1818
19 export function createLive(options: { paths?: Paths; roots?: typeof cellRoots; self?: number } = {}): Live {
19 export function createLive(options: { paths?: Paths; roots?: () => Roots; self?: number } = {}): Live {
2020 const paths = options.paths ?? HOST;
2121 const roots = options.roots ?? cellRoots;
2222 const self = options.self ?? process.pid;
src/system/view.ts +17 −2
@@ -15,6 +15,9 @@ export type Group = {
1515 procs: number;
1616 cpuCores: number | null;
1717 rssBytes: number;
18 /** A cell's memory cap, bytes, and who holds it (`apps/contain.ts`); null for the forge and the host. */
19 capBytes: number | null;
20 capBy: "kernel" | "watch" | null;
1821 oldestSeconds: number;
1922 top: { pid: number; comm: string; cpuCores: number | null; rssBytes: number; ageSeconds: number }[];
2023 tests: string[];
@@ -38,7 +41,7 @@ export type SystemView = {
3841 duplicates: Duplicate[];
3942 };
4043
41 export type Roots = readonly { label: string; root: number }[];
44 export type Roots = readonly { label: string; root: number; cap?: { bytes: number; by: "kernel" | "watch" } }[];
4245
4346 const kindOf = (label: string): Group["kind"] =>
4447 label.startsWith("check:") ? "check" : label.startsWith("app:") ? "app" : label === "forge" ? "forge" : "host";
@@ -88,12 +91,24 @@ export function buildSystem(prev: Raw | undefined, next: Raw, roots: Roots, self
8891 const before = new Map((prev?.procs ?? []).map((p) => [p.pid, p]));
8992 const where = groupOf(next.procs, roots, self);
9093 const groups = new Map<string, Group>();
94 const caps = new Map(roots.map((r) => [r.label, r.cap]));
9195 const files = new Map<string, Set<string>>();
9296 for (const p of next.procs) {
9397 const label = where.get(p.pid) ?? "host";
9498 const g =
9599 groups.get(label) ??
96 { label, kind: kindOf(label), procs: 0, cpuCores: null, rssBytes: 0, oldestSeconds: 0, top: [], tests: [] };
100 {
101 label,
102 kind: kindOf(label),
103 procs: 0,
104 cpuCores: null,
105 rssBytes: 0,
106 capBytes: caps.get(label)?.bytes ?? null,
107 capBy: caps.get(label)?.by ?? null,
108 oldestSeconds: 0,
109 top: [],
110 tests: [],
111 };
97112 const old = before.get(p.pid);
98113 const cpu = old && old.age <= p.age ? rate(old.ticks, p.ticks, seconds * TICKS_PER_SECOND) : null;
99114 g.procs++;
src/tui/screens/system.ts +5 −1
@@ -40,6 +40,10 @@ export function duplicateLines(dups: SystemView["duplicates"], inner: number): s
4040 );
4141 }
4242
43 /** A cell's memory beside its cap and who holds it, as /system says it (#2410); the forge and the host have none. */
44 const capOf = (g: SystemView["groups"][number]): string =>
45 g.capBytes === null ? "" : `${bytes(g.rssBytes)} of ${bytes(g.capBytes)} ${g.capBy === "kernel" ? "kernel" : "watched"}`;
46
4347 export function systemLines(v: SystemView): string[] {
4448 const inner = WIDTH - 6;
4549 return [
@@ -47,7 +51,7 @@ export function systemLines(v: SystemView): string[] {
4751 `hottest ${v.hottestC === null ? "—" : `${v.hottestC.toFixed(0)}°C`} · memory ${v.memory ? `${bytes(v.memory.usedBytes)} of ${bytes(v.memory.totalBytes)}` : "—"} · ${v.processes} processes`,
4852 ...v.disks.map((d) => `${pad(d.mount, 20)} ${Math.round((d.usedBytes / Math.max(1, d.totalBytes)) * 100)}% of ${bytes(d.totalBytes)}`),
4953 "",
50 ...v.groups.slice(0, GROUPS).map((g) => `${pad(g.label, 28)} ${pad(String(g.procs), 4, "right")} procs`),
54 ...v.groups.slice(0, GROUPS).map((g) => `${pad(g.label, 24)} ${pad(String(g.procs), 4, "right")} procs ${capOf(g)}`.trimEnd()),
5155 "",
5256 ...duplicateLines(v.duplicates, inner),
5357 ];
src/web/pages/system.ts +14 −1
@@ -13,6 +13,7 @@ import { drawChart } from "../../boards/chart.ts";
1313 import { canSeeMemory, memoryHours } from "../../boards/memory.ts";
1414 import { telemetryReader } from "../../actions/telemetry.ts";
1515 import type { Ctx } from "../../http/context.ts";
16 import { memoryCapHolder } from "../../apps/contain.ts";
1617 import { bytes, span } from "../../system/format.ts";
1718 import { live } from "../../system/live.ts";
1819 import { notFoundPage, signInFirst } from "../chrome.ts";
@@ -20,6 +21,12 @@ import { html, raw, type Html } from "../html.ts";
2021 import { page } from "../layout.ts";
2122 import { card, empty } from "../ui.ts";
2223
24 /** A group's memory, and a cell's beside its cap and who holds the cap. */
25 const memoryOf = (g: { rssBytes: number; capBytes: number | null; capBy: "kernel" | "watch" | null }): Html =>
26 g.capBytes === null
27 ? html`${bytes(g.rssBytes)}`
28 : html`${bytes(g.rssBytes)} <span class="muted small" data-cap="${String(g.capBytes)}" data-cap-by="${g.capBy ?? ""}">of ${bytes(g.capBytes)} (${g.capBy === "kernel" ? "kernel cap" : "watched"})</span>`;
29
2330 const pct = (n: number | null): string => (n === null ? "—" : `${n.toFixed(1)}%`);
2431 const coresOf = (n: number | null): string => (n === null ? "—" : n.toFixed(2));
2532
@@ -62,6 +69,11 @@ export function systemPage(ctx: Ctx): Response {
6269 <ul class="list small mono">
6370 ${v.duplicates.map((d) => html`<li>${d.file} · ${String(d.checks.length)} checks: ${d.checks.join(", ")}</li>`)}
6471 </ul>`;
72 const holder = memoryCapHolder();
73 const caps: Html =
74 holder.by === "kernel"
75 ? html`<p class="small" id="cell-caps" data-by="kernel">Each cell's memory is capped by the kernel (its own cgroup's <code>memory.max</code>): a cell over it is killed alone.</p>`
76 : html`<p class="small" id="cell-caps" data-by="watch">Each cell's memory is watched once a second and the cell stopped over its cap; the kernel does not cap it (${holder.reason}).</p>`;
6577 const cores = v.cpu.cores.map(
6678 (c, i) => html`<span class="mono small" title="cpu${String(i)}">${c === null ? "—" : String(Math.round(c))}</span> `,
6779 );
@@ -85,6 +97,7 @@ export function systemPage(ctx: Ctx): Response {
8597 ${String(v.processes)} processes
8698 </p>
8799 <p class="small" id="cores" aria-label="Per-core CPU, percent">${cores}</p>
100 ${caps}
88101 <p class="small mono" id="temps">${v.temps.map((t) => `${t.chip} ${t.label} ${t.celsius.toFixed(0)}°C`).join(" · ")}</p>
89102 </div>`,
90103 { head: html`<h2>Machine</h2>` },
@@ -111,7 +124,7 @@ export function systemPage(ctx: Ctx): Response {
111124 <tbody>
112125 ${v.groups.map(
113126 (g) => html`<tr data-kind="${g.kind}"><td class="mono">${g.label}</td><td>${String(g.procs)}</td>
114 <td>${coresOf(g.cpuCores)}</td><td>${bytes(g.rssBytes)}</td><td>${span(g.oldestSeconds)}</td>
127 <td>${coresOf(g.cpuCores)}</td><td>${memoryOf(g)}</td><td>${span(g.oldestSeconds)}</td>
115128 <td class="small mono">${g.top.slice(0, 3).map((p) => `${p.comm} ${coresOf(p.cpuCores)}`).join(" · ")}</td></tr>`,
116129 )}
117130 </tbody></table></div>`,
test/cgroup.test.ts added +229
@@ -0,0 +1,229 @@
1 /**
2 * A check never takes the forge down (#2410): each cell runs in a cgroup of
3 * its own whose `memory.max` is the cell's memory limit, so a check that
4 * allocates past it is killed by the kernel, alone, between two ticks of
5 * the watch. The layout is proven on a directory shaped like cgroupfs; the
6 * kernel's kill end to end where this host delegates a cgroup to the forge
7 * (it skips, and says why, where it does not — a crew sandbox's is
8 * read-only).
9 */
10
11 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
12 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
13 import { tmpdir } from "node:os";
14 import { join } from "node:path";
15 import {
16 cgroupHost,
17 cgroupRoot,
18 enterArgv,
19 oomKills,
20 openCellCgroup,
21 prepare,
22 } from "../src/apps/cgroup.ts";
23 import { cellsAvailable, type CellSpec } from "../src/apps/contain.ts";
24 import { childEnv, runStep } from "../src/apps/process.ts";
25 import type { Proc, Raw } from "../src/system/proc.ts";
26 import { buildSystem } from "../src/system/view.ts";
27
28 describe("which cgroup is the forge's to divide", () => {
29 test("off, a relative or climbing path, cgroup v1 and a shared login session are refused", () => {
30 expect(cgroupRoot("off").ok).toBe(false);
31 expect(cgroupRoot("sys/fs/cgroup").ok).toBe(false);
32 expect(cgroupRoot("/sys/fs/cgroup/../../etc").ok).toBe(false);
33 expect(cgroupRoot("", () => "12:memory:/x\n11:cpu:/x\n").ok).toBe(false);
34 const session = cgroupRoot("", () => "0::/user.slice/user-1000.slice/session-2.scope\n");
35 expect(session.ok).toBe(false);
36 if (!session.ok) expect(session.reason).toContain("FORGE_CELL_CGROUP");
37 });
38
39 test("a container's own cgroup namespace root, or the one named", () => {
40 if (process.platform === "linux")
41 expect(cgroupRoot("", () => "0::/\n")).toEqual({ ok: true, root: "/sys/fs/cgroup" });
42 expect(cgroupRoot("/sys/fs/cgroup/forge.slice")).toEqual({ ok: true, root: "/sys/fs/cgroup/forge.slice" });
43 });
44 });
45
46 describe("the layout, on a directory shaped like cgroupfs", () => {
47 let root: string;
48 beforeAll(() => {
49 root = mkdtempSync(join(tmpdir(), "forge-cgroup-"));
50 });
51 afterAll(() => rmSync(root, { recursive: true, force: true }));
52
53 const fake = (name: string, controllers: string): string => {
54 const dir = join(root, name);
55 mkdirSync(dir);
56 writeFileSync(join(dir, "cgroup.controllers"), controllers);
57 writeFileSync(join(dir, "cgroup.procs"), "11\n12\n");
58 writeFileSync(join(dir, "cgroup.subtree_control"), "");
59 return dir;
60 };
61
62 test("no memory controller: watched only, and why", () => {
63 const dir = fake("nomem", "cpu io\n");
64 const host = prepare({ ok: true, root: dir });
65 expect(host.ok).toBe(false);
66 if (!host.ok) expect(host.reason).toContain("no memory controller");
67 expect(openCellCgroup(64, host)).toBeUndefined();
68 });
69
70 test.skipIf(process.getuid?.() === 0)("read-only to the forge: watched only, and why", () => {
71 const dir = fake("ro", "cpu memory\n");
72 for (const f of ["cgroup.procs", "cgroup.subtree_control"]) Bun.spawnSync(["chmod", "0444", join(dir, f)]);
73 const host = prepare({ ok: true, root: dir });
74 expect(host.ok).toBe(false);
75 if (!host.ok) expect(host.reason).toContain("read-only");
76 });
77
78 test("the forge's processes move to a leaf, cells get the memory controller, and a crash's leftovers go", () => {
79 const dir = fake("rw", "cpuset cpu memory pids\n");
80 // A forge that is gone left one; a forge still draining a roll (this process) keeps its own.
81 const gone = Bun.spawnSync(["/bin/true"]).pid;
82 mkdirSync(join(dir, "cells", `c7-${gone}`), { recursive: true });
83 mkdirSync(join(dir, "cells", `c8-${process.pid}`), { recursive: true });
84 const host = prepare({ ok: true, root: dir });
85 expect(host).toEqual({ ok: true, root: dir });
86 // One pid a write: the last one written is what the plain file holds.
87 expect(readFileSync(join(dir, "forge", "cgroup.procs"), "utf8")).toBe("12");
88 expect(readFileSync(join(dir, "cgroup.subtree_control"), "utf8")).toBe("+memory");
89 expect(readFileSync(join(dir, "cells", "cgroup.subtree_control"), "utf8")).toBe("+memory");
90 expect(existsSync(join(dir, "cells", `c7-${gone}`))).toBe(false);
91 expect(existsSync(join(dir, "cells", `c8-${process.pid}`))).toBe(true);
92 // Already laid out (cgroupfs reads back `memory`): nothing moves again.
93 writeFileSync(join(dir, "cgroup.subtree_control"), "memory\n");
94 writeFileSync(join(dir, "forge", "cgroup.procs"), "");
95 expect(prepare({ ok: true, root: dir }).ok).toBe(true);
96 expect(readFileSync(join(dir, "forge", "cgroup.procs"), "utf8")).toBe("");
97 });
98
99 test("a cell's cgroup is capped at its limit, swapless, killed whole; its OOM kills read back", () => {
100 const dir = fake("cell", "memory\n");
101 const host = prepare({ ok: true, root: dir });
102 const cell = openCellCgroup(8192, host);
103 expect(cell).toBeDefined();
104 expect(cell!.dir.startsWith(join(dir, "cells", "c"))).toBe(true);
105 expect(cell!.maxBytes).toBe(8192 * 1024 * 1024);
106 expect(readFileSync(join(cell!.dir, "memory.max"), "utf8")).toBe("8589934592");
107 // The kernel makes these files; here, a kernel that has them.
108 writeFileSync(join(cell!.dir, "memory.events"), "low 0\nhigh 0\nmax 9\noom 1\noom_kill 1\noom_group_kill 1\n");
109 expect(cell!.oomKills()).toBe(1);
110 expect(openCellCgroup(64, host)!.dir).not.toBe(cell!.dir);
111 });
112
113 test("oom_kill is read from memory.events, and absent is none", () => {
114 expect(oomKills("oom 0\noom_kill 0\n")).toBe(0);
115 expect(oomKills("oom 3\noom_kill 2\n")).toBe(2);
116 expect(oomKills(undefined)).toBe(0);
117 expect(oomKills("oom_kill_x 5\n")).toBe(0);
118 });
119 });
120
121 describe("entering the cgroup before the cell starts", () => {
122 test("the process writes its own pid, then becomes the command — same pid, so all it starts is inside", async () => {
123 const dir = mkdtempSync(join(tmpdir(), "forge-cgroup-enter-"));
124 try {
125 const procs = join(dir, "cgroup.procs");
126 writeFileSync(procs, "");
127 const proc = Bun.spawn(enterArgv(procs, ["/bin/sh", "-c", "echo $$"]), { stdout: "pipe" });
128 const out = (await new Response(proc.stdout).text()).trim();
129 expect(await proc.exited).toBe(0);
130 expect(out).toBe(String(proc.pid));
131 expect(readFileSync(procs, "utf8").trim()).toBe(String(proc.pid));
132 } finally {
133 rmSync(dir, { recursive: true, force: true });
134 }
135 });
136
137 test("a cgroup it cannot enter: it refuses to run uncapped, and the path is never script", async () => {
138 const dir = mkdtempSync(join(tmpdir(), "forge-cgroup-enter-"));
139 try {
140 const marker = join(dir, "ran");
141 const procs = join(dir, "gone", `$(touch ${join(dir, "injected")})`, "cgroup.procs");
142 const proc = Bun.spawn(enterArgv(procs, ["/bin/sh", "-c", `touch ${marker}`]), { stderr: "pipe" });
143 const text = await new Response(proc.stderr).text();
144 expect(await proc.exited).toBe(125);
145 expect(text).toContain("cannot enter its memory cgroup");
146 expect(existsSync(marker)).toBe(false);
147 expect(existsSync(join(dir, "injected"))).toBe(false);
148 } finally {
149 rmSync(dir, { recursive: true, force: true });
150 }
151 });
152 });
153
154 describe("/system: a check cell's memory beside its cap", () => {
155 test("a cell's group carries its cap and who holds it; the forge's carries none", () => {
156 const proc = (pid: number, ppid: number, rss: number): Proc => ({ pid, ppid, comm: "bun", ticks: 0, rss, age: 5, tests: [] });
157 const raw: Raw = {
158 at: 1_000,
159 cores: [],
160 temps: [],
161 load: null,
162 memory: null,
163 disks: [],
164 net: [],
165 procs: [proc(1, 0, 100), proc(200, 1, 1 << 20), proc(201, 200, 3 << 20)],
166 };
167 const v = buildSystem(undefined, raw, [
168 { label: "check:RUN1", root: 200, cap: { bytes: 8192 * 2 ** 20, by: "kernel" } },
169 ], 1);
170 const check = v.groups.find((g) => g.label === "check:RUN1")!;
171 expect(check.rssBytes).toBe(4 << 20);
172 expect(check.capBytes).toBe(8589934592);
173 expect(check.capBy).toBe("kernel");
174 const forge = v.groups.find((g) => g.label === "forge")!;
175 expect(forge.capBytes).toBeNull();
176 expect(forge.capBy).toBeNull();
177 });
178 });
179
180 // The kernel's own kill, on a host that delegates a cgroup to the forge and holds cells.
181 const host = cgroupHost();
182 const cells = await cellsAvailable();
183 const why = !host.ok ? host.reason : !cells.ok ? cells.reason : undefined;
184 if (why !== undefined) console.log(`cgroup.test.ts: the kernel's kill is skipped here: ${why}`);
185
186 describe.skipIf(why !== undefined)("a check over its memory cap fails alone; the forge stays up", () => {
187 let dir: string;
188 beforeAll(() => {
189 dir = mkdtempSync(join(tmpdir(), "forge-cgroup-e2e-"));
190 });
191 afterAll(() => rmSync(dir, { recursive: true, force: true }));
192
193 const cell = (name: string, memoryMiB: number): CellSpec => ({
194 dir: join(dir, name),
195 writable: [],
196 home: join(dir, name, ".home"),
197 hide: [],
198 egress: [],
199 reach: [],
200 // CPU and disk out of the way: only memory decides.
201 limits: { memoryMiB, cpus: 64, diskMiB: 4096 },
202 });
203 const step = (command: string, spec: CellSpec) =>
204 runStep("check", command, {
205 cwd: spec.dir,
206 env: childEnv({}),
207 log: () => {},
208 signal: new AbortController().signal,
209 timeoutMs: 60_000,
210 cell: spec,
211 });
212
213 test("1 GiB allocated in a burst under a 96 MiB cap: the kernel kills that check, its neighbour finishes", async () => {
214 mkdirSync(join(dir, "hog"), { recursive: true });
215 mkdirSync(join(dir, "calm"), { recursive: true });
216 const before = process.memoryUsage().rss;
217 const [hog, calm] = await Promise.all([
218 // Faster than the watch's one-second tick: only the kernel can stop it in time.
219 step(`bun -e "const a = []; for (let i = 0; i < 64; i++) a.push(new Uint8Array(16 << 20).fill(1)); console.log(a.length)"`, cell("hog", 96)),
220 step(`bun -e "const a = new Uint8Array(16 << 20).fill(1); await Bun.sleep(1500); console.log(a[0])"`, cell("calm", 256)),
221 ]);
222 expect(hog.ok).toBe(false);
223 if (!hog.ok)
224 expect(hog.error).toBe("check stopped: it reached its 96 MiB memory cap, and the kernel killed it");
225 expect(calm.ok).toBe(true);
226 // The forge (this process) is alive and did not pay for the hog.
227 expect(process.memoryUsage().rss - before).toBeLessThan(256 * 2 ** 20);
228 }, 60_000);
229 });
test/contain.test.ts +2 −1
@@ -888,7 +888,8 @@ describe.skipIf(noCells)("cells: limits stop a step, and say which", () => {
888888 expect(result.ok).toBe(false);
889889 if (!result.ok)
890890 expect(result.error).toMatch(
891 /^check stopped: it used \d+ MiB of memory, over its 128 MiB limit$/,
891 // The watch's tick, or — where the forge holds a delegated cgroup — the kernel's cap first.
892 /^check stopped: it (used \d+ MiB of memory, over its 128 MiB limit|reached its 128 MiB memory cap, and the kernel killed it)$/,
892893 );
893894 }, 30_000);
894895
test/shards.test.ts +38
@@ -12,6 +12,7 @@ import {
1212 SHARDS_MAX,
1313 WORKER_MEMORY_BYTES,
1414 CELL_CPU_SHARE,
15 FILES_PER_PROCESS,
1516 formatTimes,
1617 heaviestFirst,
1718 memoryLimit,
@@ -23,6 +24,7 @@ import {
2324 shardCountFromEnv,
2425 sliceFiles,
2526 splitArgs,
27 splitCapped,
2628 splitFiles,
2729 tally,
2830 weigher,
@@ -67,6 +69,42 @@ describe("splitFiles", () => {
6769 });
6870 });
6971
72 describe("splitCapped: no bun test holds more than FILES_PER_PROCESS files (#2410)", () => {
73 const many = Array.from({ length: 330 }, (_, i) => `test/m${String(i).padStart(3, "0")}.test.ts`);
74
75 test("the fast tier's ~110 files a process, at 3 shards, become slices of at most 40", () => {
76 expect(FILES_PER_PROCESS).toBe(40);
77 const slices = splitCapped(many, 3);
78 // 330 / 40 = 8.25, so at least 9 slices; the old split made 3 of 110.
79 expect(splitFiles(many, 3).map((s) => s.length)).toEqual([110, 110, 110]);
80 expect(slices.length).toBeGreaterThanOrEqual(9);
81 for (const slice of slices) {
82 expect(slice.length).toBeGreaterThan(0);
83 expect(slice.length).toBeLessThanOrEqual(40);
84 }
85 const all = slices.flat();
86 expect(all.length).toBe(many.length);
87 expect([...all].sort()).toEqual(many);
88 });
89
90 test("many light files on one heavy-balanced slice are cut, in order", () => {
91 // One heavy file takes a slice alone; the other 99 share the other and are cut in runs of 40.
92 const weight = (file: string) => (file === "test/m000.test.ts" ? 1_000 : 1);
93 const list = many.slice(0, 100);
94 const slices = splitCapped(list, 2, weight, 40);
95 for (const slice of slices) expect(slice.length).toBeLessThanOrEqual(40);
96 expect(slices.find((s) => s.includes("test/m000.test.ts"))!.length).toBeLessThanOrEqual(40);
97 expect(slices.flat().sort()).toEqual(list);
98 for (const slice of slices) expect(slice).toEqual(list.filter((f) => slice.includes(f)));
99 });
100
101 test("under the cap it is splitFiles exactly", () => {
102 const weight = (file: string) => (Number(file.match(/\d+/)![0]) * 37) % 11;
103 expect(splitCapped(files, 3, weight)).toEqual(splitFiles(files, 3, weight));
104 expect(splitCapped(["only"], 8)).toEqual([["only"]]);
105 });
106 });
107
70108 describe("weighing by measured time", () => {
71109 // Four files of one size: bytes cannot tell them apart, the table can.
72110 const list = [
test/system-telemetry.test.ts +9 −2
@@ -193,7 +193,7 @@ describe("telemetry:read: the owner grants it, an agent reads with it, nobody el
193193 createLive({
194194 paths: { proc: join(host, "proc"), sys: join(host, "sys") },
195195 roots: () => [
196 { label: "check:RUN1", root: 200 },
196 { label: "check:RUN1", root: 200, cap: { bytes: 8192 * 2 ** 20, by: "kernel" } },
197197 { label: "check:RUN2", root: 300 },
198198 { label: "check:RUN3", root: 400 },
199199 ],
@@ -299,6 +299,9 @@ describe("telemetry:read: the owner grants it, an agent reads with it, nobody el
299299 const html = await page.text();
300300 expect(html).toContain('id="duplicates" data-count="1"');
301301 expect(html).toContain("test/shared.test.ts");
302 // Each check cell's memory beside its cap, and who holds the cap (#2410).
303 expect(html).toContain('data-cap="8589934592" data-cap-by="kernel">of 8.0 GiB (kernel cap)');
304 expect(html).toMatch(/id="cell-caps" data-by="(kernel|watch)"/);
302305 const flowPage = await fetch(`${servers.url}/flow`, { headers: { cookie: await login("alice") } });
303306 expect(await flowPage.text()).toContain('id="duplicate-work" data-count="1"');
304307
@@ -409,7 +412,11 @@ describe("telemetry:read: the owner grants it, an agent reads with it, nobody el
409412 const sys = run(alice, "system");
410413 sys.cmd!.run();
411414 expect(sys.pushed.map((x) => x.title)).toEqual(["System"]);
412 expect(systemLines(JSON.parse(JSON.stringify(live().read()))).join("\n")).toContain("1 test file is running in more than one check");
415 const lines = systemLines(JSON.parse(JSON.stringify(live().read())));
416 expect(lines.join("\n")).toContain("1 test file is running in more than one check");
417 // A check cell's memory beside its cap and who holds it, as /system says it (#2410).
418 expect(lines.find((l) => l.startsWith("check:RUN1"))).toMatch(/procs {2}\S+ \S+ of 8\.0 GiB kernel$/);
419 expect(lines.every((l) => l.length <= 66)).toBe(true);
413420
414421 expect(flowLines(app.db, new Date(), JSON.parse(JSON.stringify(live().read().duplicates))).join("\n")).toContain(
415422 "1 test file is running in more than one check",

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs