trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns3g rEventsFlows

Each cell's disk bandwidth is capped with io.max on the data volume's device #10867

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/10854-each-cell-s-disk-bandwidth-is-capped-wit → main 4 commits fork 1 thread · 1 unresolved

  • Merged. scout/10854-each-cell-s-disk-bandwidth-is-capped-wit was merged into main as dde86ca.

Review

Sign in to review this change.
6 files changed +232 −7
docs/features/child-disk-bandwidth.md added +30
@@ -0,0 +1,30 @@
1 # A child forge's disk bandwidth is capped by its size
2
3 A child forge's cell shares the parent's disk. `io.weight` only orders the
4 queue, so one busy child could still starve the parent's database and its
5 neighbours. Each child's cell now gets `io.max` on the disk under the forge's
6 data, sized by the child's size (#10854).
7
8 | Size | Read | Write | Read ops/s | Write ops/s |
9 | ------ | --------- | --------- | ---------- | ----------- |
10 | small | 50 MiB/s | 25 MiB/s | 1000 | 500 |
11 | medium | 100 MiB/s | 50 MiB/s | 2000 | 1000 |
12 | large | 200 MiB/s | 100 MiB/s | 4000 | 2000 |
13
14 - **Where.** The cell's cgroup, `cells/<child>/io.max`, one line:
15 `MAJ:MIN rbps=… wbps=… riops=… wiops=…`. The device is the whole disk under
16 the child's directory in the forge's data (a partition is walked up to its disk through `/sys/dev/block`).
17 - **When it is not written.** Where the io controller is not delegated to
18 `cells/`, where the data sits on no block device (tmpfs, overlay, a network
19 mount), or where the kernel refuses the device. The cell still starts; only
20 `io.weight` orders it then. The cell's cgroup result says which
21 (`ioCapped`).
22 - **Tuning.** The numbers are a starting point, not a measurement. They live in
23 `SIZE_IO` in `src/spawn/sizes.ts`. To tune: run `fio` (or `dd oflag=direct`)
24 against the data volume on the host to learn what the disk sustains, then
25 keep the largest size's cap well under it so the parent always has headroom.
26 A new cap applies when a child's cell next starts. Watch
27 `cells/<child>/io.stat` against `io.max` to see whether a child is pinned at
28 its cap.
29 - Tests: `test/cgroup.test.ts` (the line, the device, and a child cell that
30 gets `io.max` only where io is delegated).
src/apps/cgroup.ts +82 −2
@@ -23,6 +23,8 @@
2323 * pids.max the watch's process limit (#10518);
2424 * a step's cpu.weight and io.weight 10, below an app's
2525 * 100, and cpu.max its declared cores (`apps/background.ts`)
26 * a child forge's io.max on the device under the
27 * forge's data, sized by its size (`spawn/sizes.ts`)
2628 * R/cells/c<n>/ one per crew run, too: no memory cap, weight 10 and
2729 * cpu.max of `FORGE_CREW_CPUS` cores — out of the
2830 * forge's own leaf, where only nice told them apart
@@ -42,10 +44,12 @@ import {
4244 mkdirSync,
4345 readdirSync,
4446 readFileSync,
47 realpathSync,
4548 rmdirSync,
49 statSync,
4650 writeFileSync,
4751 } from "node:fs";
48 import { join } from "node:path";
52 import { dirname, join } from "node:path";
4953
5054 export type CgroupHost =
5155 | { readonly ok: true; readonly root: string }
@@ -60,6 +64,8 @@ export interface CellCgroup {
6064 readonly oomKills: () => number;
6165 /** Whether the kernel holds its cores (`cpu.max` written): then a busy cell is throttled, never stopped. */
6266 readonly cpuCapped: boolean;
67 /** Whether the kernel holds its disk bandwidth (`io.max` written). */
68 readonly ioCapped: boolean;
6369 /** Kill whatever is left and remove the cgroup, retrying while it empties. */
6470 readonly remove: () => void;
6571 }
@@ -85,6 +91,67 @@ export function cpuMax(cpus: number): string {
8591 return `${Math.max(1000, Math.round(cpus * CPU_PERIOD_US))} ${CPU_PERIOD_US}`;
8692 }
8793
94 /** A cell's disk bandwidth on one device: `io.max`'s keys, per second. */
95 export interface IoMax {
96 readonly rbps: number;
97 readonly wbps: number;
98 readonly riops: number;
99 readonly wiops: number;
100 }
101
102 /** Where and how much: the device's `MAJ:MIN` and the cell's `IoMax` on it. */
103 export interface IoCap {
104 readonly device: string;
105 readonly max: IoMax;
106 }
107
108 /**
109 * The `io.max` line for `max` on `device`: "MAJ:MIN rbps=… wbps=… riops=…
110 * wiops=…" (cgroup-v2.rst, "io.max"). A key that is not a positive integer
111 * is `max` (no limit); a device that is not `MAJ:MIN` is no line at all.
112 */
113 export function ioMaxLine(cap: IoCap): string | undefined {
114 if (!/^\d+:\d+$/.test(cap.device)) return undefined;
115 const v = (n: number): string => (Number.isFinite(n) && n >= 1 ? String(Math.floor(n)) : "max");
116 const { rbps, wbps, riops, wiops } = cap.max;
117 return `${cap.device} rbps=${v(rbps)} wbps=${v(wbps)} riops=${v(riops)} wiops=${v(wiops)}`;
118 }
119
120 /** Linux's `dev_t` split into major and minor (glibc's gnu_dev_major/minor). */
121 export function majorMinor(dev: number | bigint): string {
122 const d = BigInt(dev);
123 const major = ((d >> 8n) & 0xfffn) | ((d >> 32n) & ~0xfffn);
124 const minor = (d & 0xffn) | ((d >> 12n) & ~0xffn);
125 return `${major}:${minor}`;
126 }
127
128 /**
129 * The whole disk under `path`, as `MAJ:MIN`, or `undefined` where there is
130 * no block device to throttle (tmpfs, overlay, a network mount). io.max
131 * takes a disk, not a partition: a partition's `/sys/dev/block/MAJ:MIN`
132 * has a `partition` file, and its parent directory is the disk's.
133 */
134 export function diskDevice(path: string, sysBlock = "/sys/dev/block"): string | undefined {
135 let dev: string;
136 try {
137 dev = majorMinor(statSync(path, { bigint: true }).dev);
138 } catch {
139 return undefined;
140 }
141 const at = join(sysBlock, dev);
142 if (!existsSync(at)) return undefined;
143 if (!existsSync(join(at, "partition"))) return dev;
144 // The link's target, not the link: `..` of /sys/dev/block/8:1 is /sys/dev.
145 let parent: string;
146 try {
147 parent = dirname(realpathSync(at));
148 } catch {
149 return undefined;
150 }
151 const disk = words(read(join(parent, "dev")))[0];
152 return disk !== undefined && /^\d+:\d+$/.test(disk) ? disk : undefined;
153 }
154
88155 let host: CgroupHost | undefined;
89156 let nextCell = 1;
90157
@@ -242,13 +309,15 @@ export function pidsDelegated(at: CgroupHost): boolean {
242309 * `cpu.weight`, `io.weight` and `cpu.max` where those controllers reach it;
243310 * unset leaves the defaults, 100 and no quota. `processes` is its
244311 * `pids.max` where the pids controller reaches it (`pidsDelegated`); unset
245 * or not a positive integer, no kernel cap.
312 * or not a positive integer, no kernel cap. `io` is its `io.max` where the
313 * io controller reaches it; unset, no bandwidth cap.
246314 */
247315 export function openCellCgroup(
248316 memoryMiB: number | undefined,
249317 at: CgroupHost = cgroupHost(),
250318 share?: CpuShare,
251319 processes?: number,
320 io?: IoCap,
252321 ): CellCgroup | undefined {
253322 if (!at.ok) return undefined;
254323 const dir = join(at.root, "cells", `c${nextCell++}-${process.pid}`);
@@ -294,11 +363,22 @@ export function openCellCgroup(
294363 // no pids controller here: the watch still counts the cell's processes
295364 }
296365 }
366 let ioCapped = false;
367 const ioLine = io === undefined ? undefined : ioMaxLine(io);
368 if (ioLine !== undefined && words(read(join(at.root, "cells", "cgroup.subtree_control"))).includes("io")) {
369 try {
370 writeFileSync(join(dir, "io.max"), ioLine);
371 ioCapped = true;
372 } catch {
373 // the device is not one the io controller knows: io.weight still orders the queue
374 }
375 }
297376 return {
298377 dir,
299378 maxBytes,
300379 oomKills: () => oomKills(read(join(dir, "memory.events"))),
301380 cpuCapped,
381 ioCapped,
302382 remove: () => removeDir(dir),
303383 };
304384 }
src/apps/contain.ts +25 −3
@@ -38,7 +38,16 @@ import { dirname, join } from "node:path";
3838 import { portInsteadOfSocket } from "../git/hook-address.ts";
3939 import { relay, startEgress, upstreamProxy, type Egress } from "./egress.ts";
4040 import { BACKGROUND_CPU_WEIGHT } from "./background.ts";
41 import { cgroupHost, enterArgv, openCellCgroup, pidsDelegated, type CellCgroup, type CgroupHost } from "./cgroup.ts";
41 import {
42 cgroupHost,
43 diskDevice,
44 enterArgv,
45 openCellCgroup,
46 pidsDelegated,
47 type CellCgroup,
48 type CgroupHost,
49 type IoMax,
50 } from "./cgroup.ts";
4251
4352 /** `ceiling`: a declaration was cut to the machine's maximum; `declared`: forge.json's own; `default`: neither. */
4453 export type CeilingFrom = "env" | "pod" | "default";
@@ -263,6 +272,12 @@ export interface CellSpec {
263272 * it is busy rather than stopped.
264273 */
265274 readonly cpuQuota?: boolean | undefined;
275 /**
276 * Disk bandwidth the kernel holds the cell to (`io.max`) on the disk
277 * under `dir` — the forge's data volume — where io is delegated. A child
278 * forge's comes from its size (`spawn/sizes.ts`); unset, no cap.
279 */
280 readonly io?: IoMax | undefined;
266281 }
267282
268283 /**
@@ -475,9 +490,15 @@ export function cellMounts(
475490
476491 /**
477492 * A cell's own cgroup on `at`: its memory cap, its share of the cores, and
478 * `pids.max` at the watch's process limit where pids is delegated (#10518).
493 * `pids.max` at the watch's process limit where pids is delegated (#10518),
494 * and `io.max` on the disk under its directory when it declares `io`.
479495 */
480 export function cellCgroup(spec: CellSpec, at: CgroupHost = cgroupHost()): CellCgroup | undefined {
496 export function cellCgroup(
497 spec: CellSpec,
498 at: CgroupHost = cgroupHost(),
499 device: (path: string) => string | undefined = diskDevice,
500 ): CellCgroup | undefined {
501 const disk = spec.io === undefined || !at.ok ? undefined : device(spec.dir);
481502 return openCellCgroup(
482503 spec.limits.memoryMiB,
483504 at,
@@ -487,6 +508,7 @@ export function cellCgroup(spec: CellSpec, at: CgroupHost = cgroupHost()): CellC
487508 ? { weight: 100, cpus: spec.limits.cpus }
488509 : undefined,
489510 PROCESSES_MAX,
511 spec.io === undefined || disk === undefined ? undefined : { device: disk, max: spec.io },
490512 );
491513 }
492514
src/spawn/cell.ts +4 −1
@@ -20,7 +20,7 @@ import { childEnv, freePort } from "../apps/process.ts";
2020 import { childDirOf } from "../actions/spawn.ts";
2121 import { openBeatDoor, parentLocal, type BeatDoor } from "./beat-door.ts";
2222 import { hostedPlacement, parentOrigin, childPublicUrl, type BodyStore, type Placement } from "./placement.ts";
23 import { SIZES, type Size } from "./sizes.ts";
23 import { SIZE_IO, SIZES, type Size } from "./sizes.ts";
2424 import { childModelHosts, childRegistryHosts } from "../core/capacity.ts";
2525
2626 export const CELL_PREFIX = "cell-";
@@ -140,6 +140,9 @@ export function cellSpecFor(input: {
140140 // A forge is busy in bursts (a check's shards, an install): the kernel
141141 // holds it to its cores; the watch stopping it killed the check with it.
142142 cpuQuota: true,
143 // Its disk bandwidth too, on the data volume: one child's install
144 // cannot starve the forge's database of the disk (#10854).
145 io: SIZE_IO[input.size],
143146 listen: input.port,
144147 label: input.label,
145148 };
src/spawn/sizes.ts +21
@@ -22,6 +22,27 @@ export const SIZES: Readonly<Record<Size, SizeLimits>> = {
2222 large: { cpu: 4, memoryMiB: 8192, diskMiB: 40960 },
2323 };
2424
25 /**
26 * A child's disk bandwidth, held by the kernel (`io.max` on the device under
27 * the forge's data, `apps/cgroup.ts`): bytes and operations a second, read
28 * and write. One child's install or checkout cannot starve the forge's own
29 * database or its neighbours' of the disk; io.weight only orders a queue.
30 */
31 export interface SizeIo {
32 readonly rbps: number;
33 readonly wbps: number;
34 readonly riops: number;
35 readonly wiops: number;
36 }
37
38 const MiB = 1024 * 1024;
39
40 export const SIZE_IO: Readonly<Record<Size, SizeIo>> = {
41 small: { rbps: 50 * MiB, wbps: 25 * MiB, riops: 1000, wiops: 500 },
42 medium: { rbps: 100 * MiB, wbps: 50 * MiB, riops: 2000, wiops: 1000 },
43 large: { rbps: 200 * MiB, wbps: 100 * MiB, riops: 4000, wiops: 2000 },
44 };
45
2546 /** A size, or one sentence saying which are allowed. */
2647 export function parseSize(v: unknown): Result<Size, string> {
2748 if (typeof v === "string" && Object.hasOwn(SIZES, v)) return ok(v as Size);
test/cgroup.test.ts +70 −1
@@ -9,13 +9,16 @@
99 */
1010
1111 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
12 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
12 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs";
1313 import { tmpdir } from "node:os";
1414 import { join } from "node:path";
1515 import {
1616 cgroupHost,
1717 cgroupRoot,
1818 cpuMax,
19 diskDevice,
20 ioMaxLine,
21 majorMinor,
1922 enterArgv,
2023 oomKills,
2124 tryEnterArgv,
@@ -24,6 +27,8 @@ import {
2427 } from "../src/apps/cgroup.ts";
2528 import { cellCgroup, cellsAvailable, processCapHolder, PROCESSES_MAX, type CellSpec } from "../src/apps/contain.ts";
2629 import { childEnv, runStep } from "../src/apps/process.ts";
30 import { cellSpecFor } from "../src/spawn/cell.ts";
31 import { SIZE_IO } from "../src/spawn/sizes.ts";
2732 import type { Proc, Raw } from "../src/system/proc.ts";
2833 import { buildSystem } from "../src/system/view.ts";
2934
@@ -183,6 +188,70 @@ describe("the layout, on a directory shaped like cgroupfs", () => {
183188 expect(run.dir).toMatch(/\/cells\/c\d+-\d+$/);
184189 });
185190
191 test("a child forge's cell gets io.max of its size on the data disk, where io is delegated", () => {
192 const dir = fake("io", "cpu io memory\n");
193 const host = prepare({ ok: true, root: dir });
194 // cgroupfs reads back every controller enabled, without the `+`.
195 writeFileSync(join(dir, "cells", "cgroup.subtree_control"), "memory cpu io\n");
196 const child = cellSpecFor({
197 dir: join(root, "child"),
198 parentDataDir: root,
199 parentUrl: "http://127.0.0.1:1",
200 size: "medium",
201 port: 4100,
202 label: "child:x",
203 });
204 expect(child.io).toEqual(SIZE_IO.medium);
205 const seen: string[] = [];
206 const cell = cellCgroup(child, host, (path) => (seen.push(path), "259:0"))!;
207 expect(seen).toEqual([child.dir]);
208 const { rbps, wbps, riops, wiops } = SIZE_IO.medium;
209 expect(readFileSync(join(cell.dir, "io.max"), "utf8")).toBe(`259:0 rbps=${rbps} wbps=${wbps} riops=${riops} wiops=${wiops}`);
210 expect(cell.ioCapped).toBe(true);
211 // An app declares no io: no io.max. No disk under the data (tmpfs): none either.
212 expect(existsSync(join(cellCgroup(spec(join(root, "app")), host)!.dir, "io.max"))).toBe(false);
213 const none = cellCgroup(child, host, () => undefined)!;
214 expect(existsSync(join(none.dir, "io.max"))).toBe(false);
215 expect(none.ioCapped).toBe(false);
216 // The io controller not delegated to the cells: no io.max, and the cell still opens.
217 const memOnly = prepare({ ok: true, root: fake("io-none", "memory\n") });
218 const bare = cellCgroup(child, memOnly, () => "259:0")!;
219 expect(existsSync(join(bare.dir, "io.max"))).toBe(false);
220 expect(bare.ioCapped).toBe(false);
221 });
222
223 test("io.max by hand: each key a whole number or max, and a device that is not MAJ:MIN writes nothing", () => {
224 expect(ioMaxLine({ device: "8:0", max: { rbps: 1048576.7, wbps: 0, riops: Infinity, wiops: 500 } })).toBe(
225 "8:0 rbps=1048576 wbps=max riops=max wiops=500",
226 );
227 expect(ioMaxLine({ device: "8:0\n9:0", max: SIZE_IO.small })).toBeUndefined();
228 expect(ioMaxLine({ device: "sda", max: SIZE_IO.small })).toBeUndefined();
229 });
230
231 test("the disk under a path: dev_t split as Linux does, a partition climbs to its disk, no block device is none", () => {
232 expect(majorMinor(0x0801)).toBe("8:1");
233 expect(majorMinor(259n << 8n)).toBe("259:0");
234 // A large minor spills into the high bits: 8:300.
235 expect(majorMinor((8n << 8n) | 44n | (1n << 20n))).toBe("8:300");
236 const sys = mkdtempSync(join(tmpdir(), "forge-sysblock-"));
237 try {
238 const here = majorMinor(statSync(root, { bigint: true }).dev);
239 const disk = join(sys, "devices", "sda");
240 mkdirSync(join(disk, "sda1"), { recursive: true });
241 writeFileSync(join(disk, "dev"), "8:0\n");
242 writeFileSync(join(disk, "sda1", "partition"), "1\n");
243 symlinkSync(join(disk, "sda1"), join(sys, here));
244 expect(diskDevice(root, sys)).toBe("8:0");
245 rmSync(join(disk, "sda1", "partition"));
246 expect(diskDevice(root, sys)).toBe(here);
247 rmSync(join(sys, here));
248 expect(diskDevice(root, sys)).toBeUndefined();
249 expect(diskDevice(join(root, "absent"), sys)).toBeUndefined();
250 } finally {
251 rmSync(sys, { recursive: true, force: true });
252 }
253 });
254
186255 test("cpu.max by hand: 2 cores, half a core, and the kernel's 1 ms floor", () => {
187256 expect(cpuMax(2)).toBe("200000 100000");
188257 expect(cpuMax(0.5)).toBe("50000 100000");

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs