trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns11g rEventsFlows

The forge's own web process always answers: check cells and crew runs never starve it (readiness flapped with 9 checks) #4477

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/3319-the-forge-s-own-web-process-always-answe → main 5 commits fork

  • Merged. torque/3319-the-forge-s-own-web-process-always-answe was merged into main as 7d6d9fe.

Review

Sign in to review this change.
18 files changed +673 −38
README.md +1 −1
@@ -1808,7 +1808,7 @@ work here.
18081808 - the terminal: on a board's screen `[` `]` mark a panel, `p` pins or unpins it, `<` `>` move a pinned one; the dashboard lists the pinned panels by title;
18091809 - `GET /api/v1/boards/pins` (the list, in order) and `POST /api/v1/boards/pins` with `{"op":"pin|unpin|move","slug":"…","title":"…","by":-1|1}` (`by` for move only);
18101810 - the `board_pins` MCP tool (no `op` lists; `op` pin, unpin or move changes). `pins` is therefore not a board slug.
1811 The first boot also writes seven starting boards, **Usage**, **Crew**, **Releases**, **Apps**, **Security**, **Throughput** and **Forges**. Apps' per-app panels show repo ids until a source exposes a safe name; Security (sign-ins, failed sign-ins, tokens by scope, admin actions, from the admin-only `signins`, `tokens` and `adminActions` sources and `usage`) shows no token text, hash, payload or IP; Throughput is items entering each stage from issue to live, per hour, from the `stages` source. Each is written once: `.offered` in that repo records them, so an edit stays and a deletion sticks.
1811 The first boot also writes the starting boards, **Usage**, **Serving**, **Crew**, **Releases**, **Apps**, **Security**, **Throughput**, **Reach** and **Forges**; Serving is the forge's event-loop lag p99 beside the checks and crew runs running, so checks' concurrency is raised on numbers. Apps' per-app panels show repo ids until a source exposes a safe name; Security (sign-ins, failed sign-ins, tokens by scope, admin actions, from the admin-only `signins`, `tokens` and `adminActions` sources and `usage`) shows no token text, hash, payload or IP; Throughput is items entering each stage from issue to live, per hour, from the `stages` source. Each is written once: `.offered` in that repo records them, so an edit stays and a deletion sticks.
18121812 A live board (any `refresh` but `off`; the form offers off or on, and `10s` and `1m` are still accepted and mean the same) redraws its grid when a run, pull, check, issue or usage event arrives, through the live-region swap
18131813 (`/boards/:slug/live`, etag'd, one fetch per burst). The same doors are in the API: `GET /api/v1/boards`, `GET|PUT|DELETE /api/v1/boards/:slug`, and each panel's data as JSON or CSV at `GET /api/v1/boards/:slug/panels/:id?from=&to=&step=5m&format=csv`, bucketed in SQL to the step and never cut short (`docs/APPS.md`); each panel's card links `json` and `csv` (a plain download link) beside `explore`
18141814 (an agent or anyone but an admin is 403; a panel whose query cannot run is refused with 422). The same four doors are MCP tools (`boards`, `board_write`, `board_delete`; `boards` with a `slug` reads one, and with a `panel` too that panel's data, with the same `from`, `to` and `step`) and, for the same person, a **Boards** screen in the terminal that lists boards and shows a board's panels as text, each with what its data holds now (a time panel's last bucket and step, another's row count). No dependency was added.
docs/APPS.md +22
@@ -277,6 +277,22 @@ writable cgroup mount owned by the forge's uid. Without that, cells are
277277 watched only. /system says which holds the cap, and why, and shows each
278278 cell's memory beside its cap.
279279
280 The forge's own process outranks what it starts (#3319, `apps/background.ts`).
281 Every check step (install, test, check) and every crew run starts in the
282 background class: `SCHED_BATCH`, nice 10 (a CFS weight of 110 against the
283 forge's 1024) and best-effort IO at level 7, inherited by everything it
284 starts. Where the cpu and io controllers are delegated with memory, the
285 forge's leaf `<R>/forge` gets `cpu.weight` and `io.weight` 1000 against the
286 cells' 100, and a step's cell 10 against an app's 100. Each crew run gets a
287 cell cgroup of its own at weight 10 (no memory cap; it ran in the forge's
288 leaf, where only nice told it apart). The kernel holds each step's cell to
289 its declared cores (`check.limits.cpus`, by `cpu.max`) and each crew run to
290 `FORGE_CREW_CPUS` cores (default 4, at most 1024), so one busy check slows
291 itself instead of taking the pod's quota. Each is asked for and
292 none is required: a host without `chrt` or `ionice`, or one that refuses
293 them, runs the work as before. A full lane slows checks, not pages; an app's
294 own process is not lowered.
295
280296 Where the host cannot hold cells (or the crew's shell sandbox), the forge
281297 runs uncontained and says so in one line: a boot warning, and, in the foot of
282298 every page a site admin reads, `Reduced protection: apps (<reason>) runs
@@ -1156,6 +1172,7 @@ a link to the milestone (`docs/PLANNING.md`); its progress never does.
11561172 | `FORGE_CHECK_SHARDS` | the machine's cores | test shards per check (live: Settings → Capacity) |
11571173 | `FORGE_CELLS_EGRESS` | `enforce` | `report` lets undeclared egress through on 80/443 (live: Settings → Capacity) |
11581174 | `FORGE_CELL_CGROUP` | `/sys/fs/cgroup` at a namespace root | the delegated cgroup v2 whose children cap each cell's memory; `off` watches only |
1175 | `FORGE_CREW_CPUS` | `4` | cores one crew run may use (`cpu.max` on its own cell cgroup, where the cpu controller is delegated) |
11591176 | `FORGE_CREW_BUDGET_USD` | `5` | spend per run unless the agent says otherwise |
11601177 | `FORGE_CREW_TIMEOUT_MIN` | `45` | wall-clock limit of one agent run |
11611178 | `FORGE_CREW_ALLOW_HOSTS` | — | hosts an agent's shell may reach, comma-separated |
@@ -1251,6 +1268,11 @@ CPU's `topology/physical_package_id` and `core_id`. Every other thermometer
12511268 `target starts cpu:`; the machine card on `/boards/data?source=machine`
12521269 links a per-core CPU and a per-core temperature query. On an 80-CPU node
12531270 that is 80 more targets, about 1.2 M rows at most (~80 MB) in the ring.
1271 The `forge` target also carries whether the forge still serves (#3319):
1272 `lag_ms`, its event loop's p99 lateness over the step in ms (a timer asks to
1273 wake every 50 ms; `kernel/loop_lag.ts`), and `checks` and `agents`, the
1274 checks and crew runs running at the instant. The **Serving** starting board
1275 draws the lag beside them.
12541276 `boards/ring.ts` keeps it in
12551277 `machine_sample`: 15-s points for 24 h and 5-min mean and max for 30 days,
12561278 dropping what leaves its window on every write. A file this machine lacks
src/apps/background.ts added +64
@@ -0,0 +1,64 @@
1 /**
2 * The forge's serving process outranks the work it starts (#3319). A full
3 * lane of checks and crew runs once took 30 cores beside the forge, its
4 * `/healthz` timed out, and the pod left the endpoints: the children were
5 * scheduled as equals of the process that answers every page. Every check
6 * step and every crew run now starts in the background class:
7 *
8 * - **CPU, by scheduler.** `SCHED_BATCH` (sched(7): "the scheduler will
9 * apply a small scheduling penalty with respect to wakeup behavior, so
10 * that this thread is mildly disfavored in scheduling decisions") and
11 * nice 10 — a weight of 110 against the forge's 1024 at nice 0
12 * (`sched_prio_to_weight`, kernel/sched/core.c): on a core it shares
13 * with one such thread, the forge gets 1024 / 1134 ≈ 90% of the time.
14 * - **IO.** Best-effort class, level 7 — the lowest that is never starved
15 * (ioprio_set(2); "idle" would wait forever behind a forge that is
16 * always writing a little).
17 * - **CPU and IO, by cgroup.** Where the forge holds a delegated cgroup
18 * (`apps/cgroup.ts`), nice orders threads only within one cgroup, so the
19 * forge's own leaf weighs 1000 against the cells' 100, and a step's cell
20 * weighs 10 against an app's 100 among them. Each crew run gets a cell
21 * cgroup of its own at the same weight (it ran in the forge's leaf,
22 * where nice was the only difference).
23 * - **CPU, by quota.** Each step's cell is held to its declared cores
24 * (`check.limits.cpus`, `cpu.max`) and each crew run to
25 * `FORGE_CREW_CPUS` (default 4): one busy check slows itself, and cannot
26 * take the pod's whole quota while the forge waits for its turn.
27 *
28 * Each is asked for and none is required: a host without `chrt` or
29 * `ionice`, or one that refuses the call, runs the work at the priority it
30 * had. An app's own process is not lowered — it serves people too.
31 */
32
33 /** The niceness every background child runs at (nice(1)): 0 is the forge's, 19 the lowest. */
34 export const BACKGROUND_NICE = 10;
35 /** Best-effort IO class (2) at its lowest level (7), ionice(1). */
36 export const BACKGROUND_IO = { class: 2, level: 7 } as const;
37 /** A step cell's `cpu.weight` (cgroup v2, 1–10000; the default is 100). */
38 export const BACKGROUND_CPU_WEIGHT = 10;
39 /** Cores one crew run may use where `FORGE_CREW_CPUS` does not say. */
40 export const CREW_CPUS = 4;
41
42 /** `FORGE_CREW_CPUS`: a number of cores above 0 and at most 1024, or the default. */
43 export function crewCpus(raw: string | undefined = process.env["FORGE_CREW_CPUS"]): number {
44 const n = Number(raw);
45 return raw !== undefined && raw.trim() !== "" && Number.isFinite(n) && n > 0 && n <= 1024 ? n : CREW_CPUS;
46 }
47
48 /**
49 * Lower the shell's own priority, then become `argv`, so the child and
50 * everything it starts inherit the background class from their first
51 * instruction. Each step's failure is silenced and ignored; `argv` always
52 * runs. No caller text reaches the script: `argv` is arguments, never code.
53 */
54 export function backgroundArgv(argv: readonly string[]): string[] {
55 return [
56 "/bin/sh",
57 "-c",
58 "chrt -b -p 0 $$ >/dev/null 2>&1; " +
59 `ionice -c ${BACKGROUND_IO.class} -n ${BACKGROUND_IO.level} -p $$ >/dev/null 2>&1; ` +
60 `command -v nice >/dev/null 2>&1 && exec nice -n ${BACKGROUND_NICE} "$@"; exec "$@"`,
61 "background",
62 ...argv,
63 ];
64 }
src/apps/cgroup.ts +89 −11
@@ -16,9 +16,18 @@
1616 * processes" rule: a cgroup that hands memory to children holds no
1717 * process of its own):
1818 *
19 * R/forge/ the forge and everything it starts outside a cell
19 * R/forge/ the forge and everything it starts outside a cell;
20 * cpu.weight and io.weight 1000 against the cells' 100
2021 * R/cells/c<n>/ one per cell: memory.max, memory.swap.max = 0,
21 * memory.oom.group = 1 (the whole cell dies together)
22 * memory.oom.group = 1 (the whole cell dies together);
23 * a step's cpu.weight and io.weight 10, below an app's
24 * 100, and cpu.max its declared cores (`apps/background.ts`)
25 * R/cells/c<n>/ one per crew run, too: no memory cap, weight 10 and
26 * cpu.max of `FORGE_CREW_CPUS` cores — out of the
27 * forge's own leaf, where only nice told them apart
28 *
29 * The weights are asked for where the cpu and io controllers are
30 * delegated too, and never required: memory is what a cell must have.
2231 *
2332 * `FORGE_CELL_CGROUP` names `R` (an absolute path) or turns this `off`;
2433 * unset, `R` is `/sys/fs/cgroup` when the forge is at the root of its
@@ -52,6 +61,27 @@ export interface CellCgroup {
5261 readonly remove: () => void;
5362 }
5463
64 /** The forge's own leaf's `cpu.weight` and `io.weight`: ten times a cell's default 100. */
65 export const FORGE_WEIGHT = 1000;
66
67 /** A cell's place in the CPU and IO queue: its `cpu.weight` and `io.weight` (1–10000), and at most `cpus` cores (`cpu.max`). */
68 export interface CpuShare {
69 readonly weight: number;
70 readonly cpus?: number | undefined;
71 }
72
73 /** `cpu.max`'s period, µs: the kernel's default (cgroup-v2.rst, "cpu.max ... max 100000"). */
74 export const CPU_PERIOD_US = 100_000;
75
76 /**
77 * `cpu.max` for `cpus` cores: "$MAX $PERIOD — the group may consume up to
78 * $MAX in each $PERIOD duration" (cgroup-v2.rst), so 2 cores is
79 * `200000 100000`. The kernel refuses a quota under 1 ms; that is the floor.
80 */
81 export function cpuMax(cpus: number): string {
82 return `${Math.max(1000, Math.round(cpus * CPU_PERIOD_US))} ${CPU_PERIOD_US}`;
83 }
84
5585 let host: CgroupHost | undefined;
5686 let nextCell = 1;
5787
@@ -142,6 +172,20 @@ export function prepare(
142172 const cells = join(root, "cells");
143173 mkdirSync(cells, { recursive: true });
144174 writeFileSync(join(cells, "cgroup.subtree_control"), "+memory");
175 // The forge outweighs every cell, and a step weighs less than an app
176 // (#3319). Asked for, never required: without the controller, nice and
177 // SCHED_BATCH still hold.
178 for (const controller of ["cpu", "io"]) {
179 if (!words(read(join(root, "cgroup.controllers"))).includes(controller)) continue;
180 try {
181 for (const dir of [root, cells])
182 if (!words(read(join(dir, "cgroup.subtree_control"))).includes(controller))
183 writeFileSync(join(dir, "cgroup.subtree_control"), `+${controller}`);
184 writeFileSync(join(root, "forge", `${controller}.weight`), `${controller === "io" ? "default " : ""}${FORGE_WEIGHT}`);
185 } catch {
186 // not delegated here, or the forge's leaf is elsewhere: the memory cap still holds
187 }
188 }
145189 for (const name of readdirSync(cells)) {
146190 const owner = /^c\d+-(\d+)$/.exec(name)?.[1];
147191 if (owner !== undefined && !alive(Number(owner))) removeDir(join(cells, name));
@@ -178,24 +222,50 @@ function removeDir(dir: string, tries = 10): void {
178222 }
179223 }
180224
181 /** A cgroup of its own for one cell, capped at `memoryMiB`; `undefined` where the host has none. */
182 export function openCellCgroup(memoryMiB: number, at: CgroupHost = cgroupHost()): CellCgroup | undefined {
225 /**
226 * A cgroup of its own for one cell, capped at `memoryMiB` (`undefined`: no
227 * cap, for a crew run); `undefined` where the host has none. `share` is its
228 * `cpu.weight`, `io.weight` and `cpu.max` where those controllers reach it;
229 * unset leaves the defaults, 100 and no quota.
230 */
231 export function openCellCgroup(
232 memoryMiB: number | undefined,
233 at: CgroupHost = cgroupHost(),
234 share?: CpuShare,
235 ): CellCgroup | undefined {
183236 if (!at.ok) return undefined;
184237 const dir = join(at.root, "cells", `c${nextCell++}-${process.pid}`);
185 const maxBytes = Math.floor(memoryMiB * 1024 * 1024);
238 const maxBytes = memoryMiB === undefined ? Infinity : Math.floor(memoryMiB * 1024 * 1024);
186239 try {
187240 mkdirSync(dir);
188 writeFileSync(join(dir, "memory.max"), String(maxBytes));
241 if (memoryMiB !== undefined) writeFileSync(join(dir, "memory.max"), String(maxBytes));
189242 } catch {
190243 removeDir(dir, 1);
191244 return undefined;
192245 }
193246 // Swap would only let a cell outlive its cap slowly; one OOM kills the cell, not a process of it.
194 for (const [file, value] of [["memory.swap.max", "0"], ["memory.oom.group", "1"]] as const) {
195 try {
196 if (existsSync(join(dir, file))) writeFileSync(join(dir, file), value);
197 } catch {
198 // an older kernel: the cap still holds
247 if (memoryMiB !== undefined)
248 for (const [file, value] of [["memory.swap.max", "0"], ["memory.oom.group", "1"]] as const) {
249 try {
250 if (existsSync(join(dir, file))) writeFileSync(join(dir, file), value);
251 } catch {
252 // an older kernel: the cap still holds
253 }
254 }
255 // cgroupfs makes no file it does not have: a controller not delegated is a write refused.
256 if (share !== undefined) {
257 const files: [string, string][] = [
258 ["cpu.weight", String(share.weight)],
259 ["io.weight", `default ${share.weight}`],
260 ];
261 if (share.cpus !== undefined && Number.isFinite(share.cpus) && share.cpus > 0)
262 files.push(["cpu.max", cpuMax(share.cpus)]);
263 for (const [file, value] of files) {
264 try {
265 writeFileSync(join(dir, file), value);
266 } catch {
267 // no cpu or io controller here: nice and SCHED_BATCH still hold
268 }
199269 }
200270 }
201271 return {
@@ -229,3 +299,11 @@ export function enterArgv(procs: string, argv: readonly string[]): string[] {
229299 ...argv,
230300 ];
231301 }
302
303 /**
304 * Like `enterArgv`, for work that runs either way (a crew run): it enters
305 * the cgroup if it can, and runs as before if it cannot. Arguments only.
306 */
307 export function tryEnterArgv(procs: string, argv: readonly string[]): string[] {
308 return ["/bin/sh", "-c", '{ echo $$ > "$1"; } 2>/dev/null; shift; exec "$@"', "cgroup-enter", procs, ...argv];
309 }
src/apps/contain.ts +12 −1
@@ -37,6 +37,7 @@ import { availableParallelism, tmpdir } from "node:os";
3737 import { dirname, join } from "node:path";
3838 import { portInsteadOfSocket } from "../git/hook-address.ts";
3939 import { relay, startEgress, type Egress } from "./egress.ts";
40 import { BACKGROUND_CPU_WEIGHT } from "./background.ts";
4041 import { cgroupHost, enterArgv, openCellCgroup } from "./cgroup.ts";
4142
4243 /** `ceiling`: a declaration was cut to the machine's maximum; `declared`: forge.json's own; `default`: neither. */
@@ -206,6 +207,12 @@ export interface CellSpec {
206207 * the same, and never sampled.
207208 */
208209 readonly label?: string | undefined;
210 /**
211 * A step, not an app: its cgroup weighs less than an app's and the
212 * forge's, and the kernel holds it to `limits.cpus` cores
213 * (`apps/background.ts`). The step runner sets it.
214 */
215 readonly background?: boolean | undefined;
209216 }
210217
211218 /**
@@ -407,7 +414,11 @@ export function openCell(
407414 // small and bounded, and the ones that only exist to hide are read-only.
408415 const SMALL = String(1 << 20);
409416
410 const cgroup = openCellCgroup(spec.limits.memoryMiB);
417 const cgroup = openCellCgroup(
418 spec.limits.memoryMiB,
419 cgroupHost(),
420 spec.background === true ? { weight: BACKGROUND_CPU_WEIGHT, cpus: spec.limits.cpus } : undefined,
421 );
411422 const argv = (command: string): string[] => {
412423 const inner = bwrapArgv(command);
413424 return cgroup === undefined ? inner : enterArgv(join(cgroup.dir, "cgroup.procs"), inner);
src/apps/process.ts +6 −2
@@ -14,6 +14,7 @@ import type { AppManifest } from "./manifest.ts";
1414 import type { EnvName } from "./data.ts";
1515 import { envPull } from "./data.ts";
1616 import { forgeApiUrl } from "./token.ts";
17 import { backgroundArgv } from "./background.ts";
1718 import { isReservedAppEnvName } from "../core/app_env.ts";
1819 import { err, ok, type Result } from "../kernel/result.ts";
1920 import { spellings, stripForgeDirs } from "../runs/paths.ts";
@@ -261,9 +262,12 @@ export async function runStep(
261262 },
262263 ): Promise<Result<void, string>> {
263264 options.log(`$ ${command}`);
265 // A step is background work: the forge's own pages outrank it (#3319).
264266 const cell: Cell | undefined =
265 options.cell === undefined ? undefined : openCell(options.cell, options.log);
266 const proc = Bun.spawn(cell?.argv(command) ?? ["/bin/sh", "-c", command], {
267 options.cell === undefined
268 ? undefined
269 : openCell({ ...options.cell, background: true }, options.log);
270 const proc = Bun.spawn(backgroundArgv(cell?.argv(command) ?? ["/bin/sh", "-c", command]), {
267271 cwd: options.cwd,
268272 env: cell?.env(options.env) ?? options.env,
269273 stdin: "ignore",
src/boards/ring.ts +28 −5
@@ -24,6 +24,12 @@ export type Point = {
2424 load: number | null;
2525 /** °C, for a CPU or a sensor with a thermometer (`boards/thermal.ts`); absent for every other target. */
2626 celsius?: number;
27 /** `forge` only: the event loop's p99 lateness over the step, ms (`kernel/loop_lag.ts`). */
28 lagMs?: number;
29 /** `forge` only: checks running at the instant. */
30 checks?: number;
31 /** `forge` only: crew runs running at the instant. */
32 agents?: number;
2733 };
2834
2935 /** One target over one 5-min window, `at` its start. */
@@ -37,6 +43,10 @@ export type Rolled = {
3743 load: number | null;
3844 celsius: number | null;
3945 celsiusMax: number | null;
46 lagMs: number | null;
47 lagMsMax: number | null;
48 checks: number | null;
49 agents: number | null;
4050 };
4151
4252 const iso = (ms: number) => new Date(ms).toISOString();
@@ -63,31 +73,41 @@ export function rollUp(points: readonly Point[]): Rolled[] {
6373 const cpu = stats(ps.map((p) => p.cpuCores));
6474 const mem = stats(ps.map((p) => p.memoryBytes));
6575 const temp = stats(ps.map((p) => p.celsius ?? null));
76 const lag = stats(ps.map((p) => p.lagMs ?? null));
6677 return {
6778 at, target,
6879 cpuCores: cpu.mean, cpuCoresMax: cpu.max,
6980 memoryBytes: mem.mean, memoryBytesMax: mem.max,
7081 load: stats(ps.map((p) => p.load)).mean,
7182 celsius: temp.mean, celsiusMax: temp.max,
83 lagMs: lag.mean, lagMsMax: lag.max,
84 checks: stats(ps.map((p) => p.checks ?? null)).mean,
85 agents: stats(ps.map((p) => p.agents ?? null)).mean,
7286 };
7387 });
7488 }
7589
7690 type FineRow = {
7791 at: string; target: string; cpu_cores: number | null; memory_bytes: number | null; load: number | null; celsius: number | null;
92 lag_ms: number | null; checks: number | null; agents: number | null;
7893 };
7994
8095 /** Keep `points` (taken at `now`), roll up the windows that closed, drop what is past its window. One transaction. */
8196 export function record(db: Database, points: readonly Point[], now: number): void {
8297 const put = db.query(
8398 `INSERT OR REPLACE INTO machine_sample
84 (at, step, target, cpu_cores, cpu_cores_max, memory_bytes, memory_bytes_max, load, celsius, celsius_max)
85 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
99 (at, step, target, cpu_cores, cpu_cores_max, memory_bytes, memory_bytes_max, load, celsius, celsius_max,
100 lag_ms, lag_ms_max, checks, agents)
101 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
86102 );
87103 db.transaction(() => {
88104 for (const p of points) {
89105 const c = p.celsius ?? null;
90 put.run(iso(p.at), 15, p.target, p.cpuCores, p.cpuCores, p.memoryBytes, p.memoryBytes, p.load, c, c);
106 const lag = p.lagMs ?? null;
107 put.run(
108 iso(p.at), 15, p.target, p.cpuCores, p.cpuCores, p.memoryBytes, p.memoryBytes, p.load, c, c,
109 lag, lag, p.checks ?? null, p.agents ?? null,
110 );
91111 }
92112 // Every window that ended by now and is not rolled yet: after the last rolled one.
93113 const last = db.query<{ at: string | null }, []>(
@@ -96,16 +116,19 @@ export function record(db: Database, points: readonly Point[], now: number): voi
96116 const from = last == null ? "" : iso(Date.parse(last) + COARSE_MS);
97117 const to = iso(Math.floor(now / COARSE_MS) * COARSE_MS);
98118 const fine = db.query<FineRow, [string, string]>(
99 `SELECT at, target, cpu_cores, memory_bytes, load, celsius FROM machine_sample
119 `SELECT at, target, cpu_cores, memory_bytes, load, celsius, lag_ms, checks, agents FROM machine_sample
100120 WHERE step = 15 AND at >= ? AND at < ?`,
101121 ).all(from, to);
102122 for (const r of rollUp(fine.map((f) => ({
103123 at: Date.parse(f.at), target: f.target, cpuCores: f.cpu_cores, memoryBytes: f.memory_bytes, load: f.load,
104124 ...(f.celsius === null ? {} : { celsius: f.celsius }),
125 ...(f.lag_ms === null ? {} : { lagMs: f.lag_ms }),
126 ...(f.checks === null ? {} : { checks: f.checks }),
127 ...(f.agents === null ? {} : { agents: f.agents }),
105128 })))) {
106129 put.run(
107130 iso(r.at), 300, r.target, r.cpuCores, r.cpuCoresMax, r.memoryBytes, r.memoryBytesMax, r.load,
108 r.celsius, r.celsiusMax,
131 r.celsius, r.celsiusMax, r.lagMs, r.lagMsMax, r.checks, r.agents,
109132 );
110133 }
111134 db.query("DELETE FROM machine_sample WHERE step = 15 AND at < ?").run(iso(now - FINE_KEEP_MS));
src/boards/sampler.ts +35 −8
@@ -3,7 +3,9 @@
33 * for the `machine` source a board queries:
44 *
55 * - `forge`: the pod, from its cgroup v2 — `cpu.stat` `usage_usec` as a
6 * rate, and `memory.current`.
6 * rate, and `memory.current` — and whether it still serves (#3319): its
7 * event loop's p99 lateness over the step (`kernel/loop_lag.ts`), beside
8 * the checks and crew runs running at the instant (`run` rows).
79 * - `check:<run id>` and `app:<slug>/<env>`: each labelled cell, from the
810 * watch that holds it to its limits (`apps/contain.ts`). A container
911 * cannot make a cgroup per cell, so this is the cell's process tree read
@@ -23,6 +25,7 @@ import { readFileSync } from "node:fs";
2325 import { join } from "node:path";
2426 import type { App } from "../app.ts";
2527 import { cellCounters, type CellCounter } from "../apps/contain.ts";
28 import { startLoopLag, type LoopLag } from "../kernel/loop_lag.ts";
2629 import { FINE_MS, record, type Point } from "./ring.ts";
2730 import { readHwmon, readTopology, temperatures } from "./thermal.ts";
2831
@@ -131,23 +134,40 @@ function cores(a: number | null | undefined, b: number | null | undefined, secon
131134 return (b - a) / 1_000_000 / seconds;
132135 }
133136
137 /** What the forge's own serving looked like over one step, for its `forge` point. */
138 export type Serving = { lagMs: number | null; checks: number | null; agents: number | null };
139
140 /** Checks and crew runs running now: the children that compete with the forge for its CPU. */
141 export function runningChildren(db: App["db"]): { checks: number; agents: number } {
142 const rows = db.query<{ kind: string; n: number }, []>(
143 "SELECT kind, COUNT(*) AS n FROM run WHERE state = 'running' AND kind IN ('check', 'agent') GROUP BY kind",
144 ).all();
145 const n = (kind: string) => rows.find((r) => r.kind === kind)?.n ?? 0;
146 return { checks: n("check"), agents: n("agent") };
147 }
148
134149 /**
135150 * The points between two readings. Without a reading before, CPU is null
136151 * (a rate needs two). A target with nothing readable has no point at all.
137152 * A label's cells are summed; a cell new since `prev` adds its memory but
138153 * not yet its CPU.
139154 */
140 export function pointsOf(prev: Counters | undefined, next: Counters): Point[] {
155 export function pointsOf(prev: Counters | undefined, next: Counters, serving?: Serving): Point[] {
141156 const seconds = prev === undefined ? 0 : (next.at - prev.at) / 1000;
142157 const out: Point[] = [];
143158 const push = (p: Point) => {
144 if (p.cpuCores !== null || p.memoryBytes !== null || p.load !== null || p.celsius != null) out.push(p);
159 if (p.cpuCores !== null || p.memoryBytes !== null || p.load !== null || p.celsius != null || p.lagMs != null)
160 out.push(p);
145161 };
146 push({
162 const forge: Point = {
147163 at: next.at, target: "forge",
148164 cpuCores: cores(prev?.forge.cpuUsec, next.forge.cpuUsec, seconds),
149165 memoryBytes: next.forge.memoryBytes, load: null,
150 });
166 };
167 if (serving?.lagMs != null) forge.lagMs = serving.lagMs;
168 if (serving?.checks != null) forge.checks = serving.checks;
169 if (serving?.agents != null) forge.agents = serving.agents;
170 push(forge);
151171 const before = new Map((prev?.cells ?? []).map((c) => [c.key, c]));
152172 const byLabel = new Map<string, Point>();
153173 for (const c of next.cells) {
@@ -205,15 +225,16 @@ export type Sampler = { tick(now?: number): void; stop(): void };
205225 /** Sample every 15 s into `app.db` until stopped. A failed tick is logged and the next one tries again. */
206226 export function startSampler(
207227 app: App,
208 options: { paths?: Paths; cells?: () => readonly CellCounter[]; everyMs?: number } = {},
228 options: { paths?: Paths; cells?: () => readonly CellCounter[]; everyMs?: number; lag?: LoopLag } = {},
209229 ): Sampler {
210230 const paths = options.paths ?? SYSTEM;
211231 const cells = options.cells ?? cellCounters;
232 const lag = options.lag ?? startLoopLag();
212233 let prev: Counters | undefined;
213234 const tick = (now = Date.now()) => {
214235 try {
215236 const next = readCounters(paths, cells(), now);
216 record(app.db, pointsOf(prev, next), now);
237 record(app.db, pointsOf(prev, next, { lagMs: lag.take(), ...runningChildren(app.db) }), now);
217238 prev = next;
218239 } catch (error) {
219240 app.log.warn("machine sample failed", { error: String(error) });
@@ -222,5 +243,11 @@ export function startSampler(
222243 tick();
223244 const timer = setInterval(tick, options.everyMs ?? FINE_MS);
224245 timer.unref?.();
225 return { tick, stop: () => clearInterval(timer) };
246 return {
247 tick,
248 stop: () => {
249 clearInterval(timer);
250 lag.stop();
251 },
252 };
226253 }
src/boards/sources.ts +6
@@ -206,6 +206,7 @@ export const SOURCES: Readonly<Record<string, Source>> = {
206206 cpu_cores: "number", cpu_cores_max: "number",
207207 memory_bytes: "number", memory_bytes_max: "number", load: "number",
208208 celsius: "number", celsius_max: "number",
209 lag_ms: "number", lag_ms_max: "number", checks: "number", agents: "number",
209210 },
210211 docs: {
211212 target:
@@ -220,6 +221,11 @@ export const SOURCES: Readonly<Record<string, Source>> = {
220221 load: "node only: the 1-min load average, runnable tasks",
221222 celsius: "°C, mean over the step: a cpu:<nnn>'s core, a package or a sensor; null for the rest",
222223 celsius_max: "°C, the highest 15-s point in the step",
224 lag_ms:
225 "forge only: how late its event loop woke, ms, 99th percentile over the 15 s; a 5-min point is the mean of its 15-s points",
226 lag_ms_max: "forge only: ms, the highest 15-s point in the step",
227 checks: "forge only: checks running at the instant; a 5-min point is the mean",
228 agents: "forge only: crew runs running at the instant; a 5-min point is the mean",
223229 },
224230 },
225231 // Who visits and how fast it is (#2712): one row per metric (and route, and
src/boards/starting.ts +15 −1
@@ -1,5 +1,5 @@
11 /**
2 * The boards the forge offers a new install: Usage, Crew, Releases, Apps, Security, Throughput, Forges and Reach. They are plain
2 * The boards the forge offers a new install: Usage, Serving, Crew, Releases, Apps, Security, Throughput, Forges and Reach. They are plain
33 * boards in `forge/boards` once written, so a person edits or deletes them
44 * like any other; `.offered` (a JSON list of slugs, committed with the
55 * boards) records what was offered so a deletion sticks and an edit is
@@ -55,6 +55,20 @@ export const STARTING_BOARDS: Readonly<Record<string, Board>> = {
5555 panel("Requests per hour", "bar", { source: "usage", aggregate: "sum", column: "n", groupBy: "hour", series: "status" }, 2),
5656 ],
5757 },
58 // Whether the forge still answers while its children work (#3319): the
59 // event loop's lateness beside what competes with it for CPU.
60 serving: {
61 title: "Serving",
62 range: "24h",
63 refresh: "1m",
64 panels: [
65 panel("Event-loop lag p99 (ms)", "line", { ...machine("eq", "forge", "lag_ms_max"), aggregate: "max" }, 2),
66 panel("Checks running", "line", { ...machine("eq", "forge", "checks"), aggregate: "max" }),
67 panel("Crew runs running", "line", { ...machine("eq", "forge", "agents"), aggregate: "max" }),
68 panel("Pod CPU (cores)", "line", machine("eq", "forge", "cpu_cores"), 2),
69 panel("Checks CPU (cores)", "area", { ...machine("starts", "check:", "cpu_cores"), series: "target" }, 2),
70 ],
71 },
5872 releases: {
5973 title: "Releases",
6074 range: "30d",
src/core/migrations/20261020130000-machine-serving.ts added +10
@@ -0,0 +1,10 @@
1 // Whether the forge still serves while its children work (#3319): on the
2 // `forge` target, the event loop's p99 lateness in ms over the step (and
3 // the worst 15-s point in a 5-min roll-up), and how many checks and crew
4 // runs were running then. Telemetry, not state: a lost row is a gap.
5 export default `
6 ALTER TABLE machine_sample ADD COLUMN lag_ms REAL;
7 ALTER TABLE machine_sample ADD COLUMN lag_ms_max REAL;
8 ALTER TABLE machine_sample ADD COLUMN checks REAL;
9 ALTER TABLE machine_sample ADD COLUMN agents REAL;
10 `;
src/crew/claude.ts +26 −1
@@ -12,6 +12,9 @@
1212 * settings files of the operator's, a spend budget, and a deadline.
1313 */
1414
15 import { BACKGROUND_CPU_WEIGHT, backgroundArgv, crewCpus } from "../apps/background.ts";
16 import { cgroupHost, openCellCgroup, tryEnterArgv } from "../apps/cgroup.ts";
17 import { join } from "node:path";
1518 import { watchLabelled } from "../apps/contain.ts";
1619 import { childEnv, pump } from "../apps/process.ts";
1720 import { spellings, stripForgeDirs } from "../runs/paths.ts";
@@ -175,7 +178,20 @@ export function claudeArgs(input: ClaudeInput): string[] {
175178
176179 /** Run Claude Code to completion. Never throws; every ending is an outcome. */
177180 export async function runClaude(input: ClaudeInput): Promise<ClaudeOutcome> {
178 const proc = Bun.spawn([input.binary, ...claudeArgs(input)], {
181 // Background work, with everything its shell starts: the forge's own
182 // pages outrank it (#3319, `apps/background.ts`). A binary that is not
183 // there is spawned bare, so it fails as itself (ENOENT, caught by the
184 // caller as a failed run) and not as `nice`'s exit 127. Where the forge
185 // holds a cgroup, the run gets a cell of its own, weighed and held to
186 // `FORGE_CREW_CPUS` cores, so it no longer shares the forge's leaf.
187 const argv = [input.binary, ...claudeArgs(input)];
188 const present = Bun.which(input.binary) !== null;
189 const cgroup = present
190 ? openCellCgroup(undefined, cgroupHost(), { weight: BACKGROUND_CPU_WEIGHT, cpus: crewCpus() })
191 : undefined;
192 const contained = cgroup === undefined ? argv : tryEnterArgv(join(cgroup.dir, "cgroup.procs"), argv);
193 const spawn = (): ReturnType<typeof Bun.spawn<"pipe", "pipe", "pipe">> =>
194 Bun.spawn(present ? backgroundArgv(contained) : argv, {
179195 cwd: input.cwd,
180196 env: childEnv({
181197 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: "1",
@@ -190,6 +206,15 @@ export async function runClaude(input: ClaudeInput): Promise<ClaudeOutcome> {
190206 // to one pid never touches a grandchild holding stdout open.
191207 detached: true,
192208 });
209 let proc: ReturnType<typeof spawn>;
210 try {
211 proc = spawn();
212 } catch (error) {
213 cgroup?.remove();
214 throw error;
215 }
216 // The cell goes with the run, and whatever it left running with it.
217 if (cgroup !== undefined) void proc.exited.finally(() => cgroup.remove());
193218 if (input.label !== undefined) {
194219 const unwatch = watchLabelled(proc.pid, input.label);
195220 void proc.exited.finally(unwatch);
src/kernel/loop_lag.ts added +54
@@ -0,0 +1,54 @@
1 /**
2 * How late the forge's event loop runs (#3319): a timer asks to wake every
3 * `everyMs`, and how much later than that it actually wakes is the time a
4 * request arriving then would have waited before the forge could even read
5 * it. A loop starved of CPU (a full lane of checks beside it) shows here
6 * before a probe times out.
7 *
8 * `take()` hands back the 99th percentile of the lateness since the last
9 * take, in ms, and starts a new window; null when no timer fired in it.
10 */
11
12 export type LoopLag = { take(): number | null; stop(): void };
13
14 /** How often the timer asks to wake, ms: 20 wakes a second, no measurable cost. */
15 export const LAG_EVERY_MS = 50;
16
17 /** The most samples one window keeps: 15 s at 50 ms is 300; a stalled sampler never grows it past this. */
18 const MAX_SAMPLES = 4096;
19
20 /**
21 * Nearest-rank percentile `p` (0 < p ≤ 100) of `xs`: the value at rank
22 * ⌈p/100 · n⌉ of the sorted list (NIST/Hyndman–Fan definition 1). Null
23 * for an empty list.
24 */
25 export function nearestRank(xs: readonly number[], p: number): number | null {
26 if (xs.length === 0) return null;
27 const sorted = [...xs].sort((a, b) => a - b);
28 const rank = Math.max(1, Math.ceil((p / 100) * sorted.length));
29 return sorted[rank - 1]!;
30 }
31
32 /** Start measuring; `now` is a clock in ms (`performance.now` in production). */
33 export function startLoopLag(everyMs = LAG_EVERY_MS, now: () => number = () => performance.now()): LoopLag {
34 let samples: number[] = [];
35 let timer: ReturnType<typeof setTimeout> | undefined;
36 const arm = (): void => {
37 const due = now() + everyMs;
38 timer = setTimeout(() => {
39 // Early by a fraction of a ms is timer rounding, not a fast loop.
40 if (samples.length < MAX_SAMPLES) samples.push(Math.max(0, now() - due));
41 arm();
42 }, everyMs);
43 timer.unref?.();
44 };
45 arm();
46 return {
47 take: () => {
48 const p99 = nearestRank(samples, 99);
49 samples = [];
50 return p99;
51 },
52 stop: () => clearTimeout(timer),
53 };
54 }
src/main.ts +1 −1
@@ -380,7 +380,7 @@ const snapshots = startSnapshots(
380380 // node, every 15 s into a ring the `machine` board source reads.
381381 const sampler = startSampler(app);
382382 const memorySampler = startMemorySampler(app);
383 // The Usage, Crew, Security and Throughput starting boards, once each (actions/boards.ts).
383 // The starting boards, once each (actions/boards.ts).
384384 void offerStartingBoards(app)
385385 .then((slugs) => slugs.length > 0 && app.log.info("starting boards offered", { slugs: slugs.join(", ") }))
386386 .catch((error) => app.log.warn("starting boards not offered", { error: String(error) }));
test/boards-sampler.test.ts +43 −2
@@ -123,11 +123,21 @@ describe("the 5-min roll-up", () => {
123123 pts.push({ at: T0 + 315_000, target: "forge", cpuCores: null, memoryBytes: 42, load: null });
124124 expect(rollUp(pts)).toEqual([
125125 // (0 + 1 + … + 19) / 4 / 20 = 190 / 80 = 2.375; max 19 / 4 = 4.75
126 { at: T0, target: "forge", cpuCores: 2.375, cpuCoresMax: 4.75, memoryBytes: 9500, memoryBytesMax: 19000, load: null, celsius: null, celsiusMax: null },
127 { at: T0 + 300_000, target: "forge", cpuCores: 7, cpuCoresMax: 7, memoryBytes: 42, memoryBytesMax: 42, load: null, celsius: null, celsiusMax: null },
126 { at: T0, target: "forge", cpuCores: 2.375, cpuCoresMax: 4.75, memoryBytes: 9500, memoryBytesMax: 19000, load: null, celsius: null, celsiusMax: null, lagMs: null, lagMsMax: null, checks: null, agents: null },
127 { at: T0 + 300_000, target: "forge", cpuCores: 7, cpuCoresMax: 7, memoryBytes: 42, memoryBytesMax: 42, load: null, celsius: null, celsiusMax: null, lagMs: null, lagMsMax: null, checks: null, agents: null },
128128 ]);
129129 });
130130
131 test("the forge's event-loop lag rolls up as mean and max, its checks and crew runs as the mean", () => {
132 const pts: Point[] = [];
133 for (let i = 0; i < 20; i++) {
134 pts.push({ at: T0 + i * 15_000, target: "forge", cpuCores: null, memoryBytes: 1, load: null, lagMs: i, checks: i < 10 ? 9 : 12, agents: 3 });
135 }
136 const [r] = rollUp(pts);
137 // (0 + … + 19) / 20 = 9.5 ms, worst 19 ms; checks (10 × 9 + 10 × 12) / 20 = 10.5
138 expect([r!.lagMs, r!.lagMsMax, r!.checks, r!.agents]).toEqual([9.5, 19, 10.5, 3]);
139 });
140
131141 test("record rolls a window up once it has closed, and only then", () => {
132142 const db = fresh();
133143 const coarse = () => db.query("SELECT at, cpu_cores, cpu_cores_max FROM machine_sample WHERE step = 300").all();
@@ -280,4 +290,35 @@ describe("startSampler", () => {
280290 ]);
281291 expect(warnings).toEqual([]);
282292 });
293
294 test("the forge's point carries the event loop's p99 lag and the checks and crew runs running", () => {
295 const db = fresh();
296 db.run("PRAGMA foreign_keys = OFF");
297 const run = db.query("INSERT INTO run (id, repo_id, kind, state) VALUES (?, 'r', ?, ?)");
298 for (const [id, kind, state] of [
299 ["a", "check", "running"], ["b", "check", "running"], ["c", "check", "ok"], ["d", "check", "queued"],
300 ["e", "agent", "running"], ["f", "deploy", "running"],
301 ]) run.run(id!, kind!, state!);
302 const app = { db, log: { warn: () => {} } } as unknown as App;
303 let stopped = false;
304 const lags = [7.5, null];
305 const s = startSampler(app, {
306 paths, everyMs: 3_600_000, cells: () => [],
307 lag: { take: () => lags.shift() ?? null, stop: () => void (stopped = true) },
308 });
309 s.tick(T0 + 30_000);
310 s.stop();
311 expect(stopped).toBe(true);
312 const rows = db.query(
313 "SELECT at = ? AS later, lag_ms, lag_ms_max, checks, agents FROM machine_sample WHERE target = 'forge' AND step = 15 ORDER BY later",
314 ).all(new Date(T0 + 30_000).toISOString());
315 // The first tick (at start) took 7.5 ms; the second, a window with no wake, has no lag and still counts.
316 expect(rows).toEqual([
317 { later: 0, lag_ms: 7.5, lag_ms_max: 7.5, checks: 2, agents: 1 },
318 { later: 1, lag_ms: null, lag_ms_max: null, checks: 2, agents: 1 },
319 ]);
320 expect(runQuery(db, {
321 source: "machine", aggregate: "max", column: "lag_ms_max", filter: [{ column: "target", op: "eq", value: "forge" }],
322 }, { repoIds: "all" })).toEqual({ ok: true, rows: [{ group: null, value: 7.5 }] });
323 });
283324 });
test/boards-starting.test.ts +4 −4
@@ -1,5 +1,5 @@
11 /**
2 * The forge offers the eight starting boards on boot: every panel runs, an
2 * The forge offers the nine starting boards on boot: every panel runs, an
33 * edit survives the next boot, and a deletion is not undone.
44 */
55 import { afterAll, beforeAll, expect, test } from "bun:test";
@@ -36,10 +36,10 @@ test("a forge with no person to found the repo writes nothing", async () => {
3636 expect(await offerStartingBoards(bare)).toEqual([]);
3737 });
3838
39 test("a fresh boot writes the eight starting boards, and every panel runs", async () => {
40 expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "reach", "releases", "security", "throughput", "usage"]);
39 test("a fresh boot writes the nine starting boards, and every panel runs", async () => {
40 expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "reach", "releases", "security", "serving", "throughput", "usage"]);
4141 const boards = await listBoards(app.reposRoot);
42 expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "reach", "releases", "security", "throughput", "usage"]);
42 expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "reach", "releases", "security", "serving", "throughput", "usage"]);
4343 let panels = 0;
4444 for (const { board } of boards) {
4545 for (const p of board.panels) {
test/cgroup.test.ts +63 −1
@@ -15,8 +15,10 @@ import { join } from "node:path";
1515 import {
1616 cgroupHost,
1717 cgroupRoot,
18 cpuMax,
1819 enterArgv,
1920 oomKills,
21 tryEnterArgv,
2022 openCellCgroup,
2123 prepare,
2224 } from "../src/apps/cgroup.ts";
@@ -76,7 +78,8 @@ describe("the layout, on a directory shaped like cgroupfs", () => {
7678 });
7779
7880 test("the forge's processes move to a leaf, cells get the memory controller, and a crash's leftovers go", () => {
79 const dir = fake("rw", "cpuset cpu memory pids\n");
81 // No cpu or io controller: the fake's files hold the last write, and this is about memory.
82 const dir = fake("rw", "cpuset memory pids\n");
8083 // A forge that is gone left one; a forge still draining a roll (this process) keeps its own.
8184 const gone = Bun.spawnSync(["/bin/true"]).pid;
8285 mkdirSync(join(dir, "cells", `c7-${gone}`), { recursive: true });
@@ -96,6 +99,45 @@ describe("the layout, on a directory shaped like cgroupfs", () => {
9699 expect(readFileSync(join(dir, "forge", "cgroup.procs"), "utf8")).toBe("");
97100 });
98101
102 test("where cpu and io are delegated, the forge weighs 1000, a step's cell 10 and an app's the default", () => {
103 const dir = fake("weights", "cpu io memory\n");
104 const host = prepare({ ok: true, root: dir });
105 expect(host.ok).toBe(true);
106 // The fake holds the last write; cgroupfs adds each: memory, then cpu, then io.
107 expect(readFileSync(join(dir, "cgroup.subtree_control"), "utf8")).toBe("+io");
108 expect(readFileSync(join(dir, "cells", "cgroup.subtree_control"), "utf8")).toBe("+io");
109 expect(readFileSync(join(dir, "forge", "cpu.weight"), "utf8")).toBe("1000");
110 expect(readFileSync(join(dir, "forge", "io.weight"), "utf8")).toBe("default 1000");
111 const step = openCellCgroup(64, host, { weight: 10, cpus: 16 })!;
112 expect(readFileSync(join(step.dir, "memory.max"), "utf8")).toBe(String(64 * 1024 * 1024));
113 expect(readFileSync(join(step.dir, "cpu.weight"), "utf8")).toBe("10");
114 expect(readFileSync(join(step.dir, "io.weight"), "utf8")).toBe("default 10");
115 // 16 cores: 16 × 100 000 µs of CPU in each 100 000 µs period.
116 expect(readFileSync(join(step.dir, "cpu.max"), "utf8")).toBe("1600000 100000");
117 const app = openCellCgroup(64, host)!;
118 expect(existsSync(join(app.dir, "cpu.weight"))).toBe(false);
119 expect(existsSync(join(app.dir, "cpu.max"))).toBe(false);
120 });
121
122 test("a crew run's cell has no memory cap, weighs 10 and is held to its cores", () => {
123 const dir = fake("crew", "cpu io memory\n");
124 const host = prepare({ ok: true, root: dir });
125 const run = openCellCgroup(undefined, host, { weight: 10, cpus: 4 })!;
126 expect(run.maxBytes).toBe(Infinity);
127 for (const file of ["memory.max", "memory.swap.max", "memory.oom.group"])
128 expect(existsSync(join(run.dir, file))).toBe(false);
129 expect(readFileSync(join(run.dir, "cpu.weight"), "utf8")).toBe("10");
130 expect(readFileSync(join(run.dir, "cpu.max"), "utf8")).toBe("400000 100000");
131 // Named like any cell, so a restart sweeps it once this forge is gone.
132 expect(run.dir).toMatch(/\/cells\/c\d+-\d+$/);
133 });
134
135 test("cpu.max by hand: 2 cores, half a core, and the kernel's 1 ms floor", () => {
136 expect(cpuMax(2)).toBe("200000 100000");
137 expect(cpuMax(0.5)).toBe("50000 100000");
138 expect(cpuMax(0.001)).toBe("1000 100000");
139 });
140
99141 test("a cell's cgroup is capped at its limit, swapless, killed whole; its OOM kills read back", () => {
100142 const dir = fake("cell", "memory\n");
101143 const host = prepare({ ok: true, root: dir });
@@ -151,6 +193,26 @@ describe("entering the cgroup before the cell starts", () => {
151193 });
152194 });
153195
196 describe("a crew run enters its cell if it can", () => {
197 test("it enters, then becomes the command; a cell it cannot enter runs it anyway, and the path is never script", async () => {
198 const dir = mkdtempSync(join(tmpdir(), "forge-cgroup-try-"));
199 try {
200 const procs = join(dir, "cgroup.procs");
201 writeFileSync(procs, "");
202 const inside = Bun.spawn(tryEnterArgv(procs, ["/bin/sh", "-c", "echo $$"]), { stdout: "pipe" });
203 expect((await new Response(inside.stdout).text()).trim()).toBe(String(inside.pid));
204 expect(readFileSync(procs, "utf8").trim()).toBe(String(inside.pid));
205 const gone = join(dir, "gone", `$(touch ${join(dir, "injected")})`, "cgroup.procs");
206 const outside = Bun.spawn(tryEnterArgv(gone, ["/bin/sh", "-c", "exit 3"]), { stderr: "pipe" });
207 expect(await outside.exited).toBe(3);
208 expect(await new Response(outside.stderr).text()).toBe("");
209 expect(existsSync(join(dir, "injected"))).toBe(false);
210 } finally {
211 rmSync(dir, { recursive: true, force: true });
212 }
213 });
214 });
215
154216 describe("/system: a check cell's memory beside its cap", () => {
155217 test("a cell's group carries its cap and who holds it; the forge's carries none", () => {
156218 const proc = (pid: number, ppid: number, rss: number): Proc => ({ pid, ppid, comm: "bun", ticks: 0, rss, age: 5, tests: [] });
test/serving-priority.test.ts added +194
@@ -0,0 +1,194 @@
1 /**
2 * The forge's own web process always answers (#3319): its event loop's
3 * lateness is measured as a p99 a board can chart, and the work it starts
4 * — check steps and crew runs — runs in the background class (nice 10,
5 * SCHED_BATCH, best-effort IO level 7), so a full lane slows checks, not
6 * pages. The load test burns 12 cells' worth of CPU beside a server and
7 * holds its /healthz p99 under 200 ms.
8 */
9
10 import { describe, expect, test } from "bun:test";
11 import { backgroundArgv, BACKGROUND_NICE, CREW_CPUS, crewCpus } from "../src/apps/background.ts";
12 import { nearestRank, startLoopLag } from "../src/kernel/loop_lag.ts";
13 import { runClaude } from "../src/crew/claude.ts";
14 import { sandboxPolicy } from "../src/crew/sandbox.ts";
15
16 /** Fields of `/proc/<pid>/stat` after the command name: index 0 is field 3 (state), so field n is n − 3. */
17 const statFields = (text: string): string[] => text.slice(text.lastIndexOf(")") + 2).trim().split(" ");
18 const NICE = 19 - 3;
19 const POLICY = 41 - 3;
20 const SCHED_BATCH = 3;
21
22 describe("nearest-rank percentile", () => {
23 test("rank ⌈p/100 · n⌉ of the sorted list, hand-counted", () => {
24 const hundred = Array.from({ length: 100 }, (_, i) => 100 - i); // 100 … 1
25 expect(nearestRank(hundred, 99)).toBe(99); // rank 99
26 expect(nearestRank(hundred, 50)).toBe(50);
27 expect(nearestRank([3, 1, 2], 99)).toBe(3); // ⌈2.97⌉ = 3
28 expect(nearestRank([5], 99)).toBe(5);
29 expect(nearestRank([], 99)).toBeNull();
30 });
31 });
32
33 describe("event-loop lag", () => {
34 test("a loop blocked for 120 ms shows it in the window's p99; the next window starts empty", async () => {
35 const lag = startLoopLag(10);
36 try {
37 await Bun.sleep(40);
38 const until = performance.now() + 120;
39 while (performance.now() < until) {
40 // the loop is held: no timer can fire
41 }
42 await Bun.sleep(30);
43 expect(lag.take()!).toBeGreaterThanOrEqual(100);
44 await Bun.sleep(60);
45 expect(lag.take()!).toBeLessThan(100);
46 } finally {
47 lag.stop();
48 }
49 });
50
51 test("no wake in the window is no number, never a zero", () => {
52 const lag = startLoopLag(60_000);
53 expect(lag.take()).toBeNull();
54 lag.stop();
55 });
56 });
57
58 describe("FORGE_CREW_CPUS", () => {
59 test("a number of cores above 0 and at most 1024; anything else is the default", () => {
60 expect(crewCpus("6")).toBe(6);
61 expect(crewCpus("0.5")).toBe(0.5);
62 for (const raw of [undefined, "", " ", "0", "-2", "1025", "four", "Infinity"]) expect(crewCpus(raw)).toBe(CREW_CPUS);
63 });
64 });
65
66 const hasProc = await Bun.file("/proc/self/stat").exists();
67
68 describe.skipIf(!hasProc)("the background class", () => {
69 test("the child and what it starts run at nice 10, SCHED_BATCH where chrt is, same argv", async () => {
70 const proc = Bun.spawn(backgroundArgv(["/bin/sh", "-c", 'cat /proc/self/stat; echo "$1|$2"', "x", "a b", "$(c)"]), {
71 stdout: "pipe",
72 });
73 const [stat, args] = (await new Response(proc.stdout).text()).trim().split("\n");
74 expect(await proc.exited).toBe(0);
75 const f = statFields(stat!);
76 expect(Number(f[NICE])).toBe(BACKGROUND_NICE);
77 if (Bun.which("chrt") !== null) expect(Number(f[POLICY])).toBe(SCHED_BATCH);
78 // Arguments pass as arguments: a space and a `$(…)` are never script.
79 expect(args).toBe("a b|$(c)");
80 });
81
82 test.skipIf(Bun.which("ionice") === null)("its IO is best-effort at level 7", async () => {
83 const proc = Bun.spawn(backgroundArgv(["/bin/sh", "-c", "ionice -p $$"]), { stdout: "pipe" });
84 expect((await new Response(proc.stdout).text()).trim()).toBe("best-effort: prio 7");
85 });
86
87 test("a crew run whose binary is not there fails at spawn, as it did before the background class", async () => {
88 const dir = process.env.TMPDIR ?? "/tmp";
89 const run = runClaude({
90 binary: `${dir}/no-such-claude-${process.pid}`,
91 cwd: dir,
92 prompt: "",
93 appendSystemPrompt: "",
94 budgetUsd: 0,
95 env: {},
96 sandbox: sandboxPolicy({ dataDir: dir, agentName: "bot", urls: [] }),
97 timeoutMs: 5_000,
98 log: () => {},
99 signal: new AbortController().signal,
100 });
101 // ENOENT is what the caller turns into a failed run; nice's exit 127 read as "no changes pushed".
102 await expect(run).rejects.toThrow(/ENOENT/);
103 });
104
105 test("a command that is not there fails as itself, and its exit status comes through", async () => {
106 const missing = Bun.spawn(backgroundArgv(["/no/such/binary"]), { stdout: "ignore", stderr: "ignore" });
107 expect(await missing.exited).toBe(127);
108 const three = Bun.spawn(backgroundArgv(["/bin/sh", "-c", "exit 3"]));
109 expect(await three.exited).toBe(3);
110 });
111 });
112
113 /**
114 * A busy loop that is the spawned process itself (backgroundArgv execs
115 * into it), so `kill(9)` stops it, and that ends itself once the process
116 * named as its owner is gone, so none outlives a test cut off mid-run. A
117 * wrapper such as timeout(1) would fork the loop as its child: killing the
118 * wrapper orphans the loop, and twelve of them burned a check's 16 cores.
119 */
120 const BURN_WHILE = 'while kill -0 "$1" 2>/dev/null; do :; done';
121 const burner = (owner: number) => ["/bin/sh", "-c", BURN_WHILE, "burner", String(owner)];
122
123 describe.skipIf(!hasProc)("a burner", () => {
124 test("ends itself when its owner is gone", async () => {
125 // The outer shell starts a burner it owns and exits at once.
126 const outer = Bun.spawn(["/bin/sh", "-c", 'sh -c "$1" burner $$ >/dev/null 2>&1 & echo $!', "outer", BURN_WHILE], {
127 stdout: "pipe",
128 });
129 const pid = Number((await new Response(outer.stdout).text()).trim());
130 await outer.exited;
131 expect(pid).toBeGreaterThan(0);
132 const gone = async () => {
133 const stat = Bun.file(`/proc/${pid}/stat`);
134 if (!(await stat.exists())) return true;
135 try {
136 return statFields(await stat.text())[0] === "Z";
137 } catch {
138 return true;
139 }
140 };
141 const until = performance.now() + 2000;
142 while (!(await gone()) && performance.now() < until) await Bun.sleep(20);
143 const ended = await gone();
144 if (!ended) process.kill(pid, 9);
145 expect(ended).toBe(true);
146 });
147 });
148
149 /**
150 * 12 cells' worth of CPU (each a busy loop, in the background class, as a
151 * check step is) beside a server whose /healthz runs one `select 1`. On a
152 * machine with fewer cores than burners the server competes for every
153 * core, which is the case the background class is for.
154 */
155 describe.skipIf(!hasProc)("12 CPU-burning cells beside the server", () => {
156 test("/healthz p99 stays under 200 ms and no probe fails", async () => {
157 const { Database } = await import("bun:sqlite");
158 const db = new Database(":memory:");
159 const server = Bun.serve({
160 port: 0,
161 hostname: "127.0.0.1",
162 fetch: (req) =>
163 new URL(req.url).pathname === "/healthz"
164 ? (db.query("select 1").get(), new Response("ok"))
165 : new Response("no", { status: 404 }),
166 });
167 const burners = Array.from({ length: 12 }, () =>
168 Bun.spawn(backgroundArgv(burner(process.pid)), { stdout: "ignore", stderr: "ignore" }),
169 );
170 try {
171 await Bun.sleep(300);
172 const times: number[] = [];
173 let failed = 0;
174 for (let i = 0; i < 60; i++) {
175 const start = performance.now();
176 try {
177 const res = await fetch(`http://127.0.0.1:${server.port}/healthz`, { signal: AbortSignal.timeout(1000) });
178 if (res.status !== 200 || (await res.text()) !== "ok") failed++;
179 } catch {
180 failed++;
181 }
182 times.push(performance.now() - start);
183 await Bun.sleep(10);
184 }
185 expect(failed).toBe(0);
186 expect(nearestRank(times, 99)!).toBeLessThan(200);
187 } finally {
188 for (const b of burners) b.kill(9);
189 await Promise.all(burners.map((b) => b.exited));
190 server.stop(true);
191 db.close();
192 }
193 }, 30_000);
194 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs