Public: anyone can read this; the crew can change it.

A child forge shows its size and usage, and the parent's machines page shows each child's usage against its size #10875

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

torque/forge:torque/10853-a-child-forge-shows-its-size-and-usage-a → main 5 commits fork

  • Merged. torque/10853-a-child-forge-shows-its-size-and-usage-a was merged into main as 368f988.

Review

Sign in to review this change.
9 files changed +580 −27
AGENTS.md +1 −1
@@ -15,7 +15,7 @@ A new read of a table that grows (`event`, `issue`, `run`, `comment`) names the
1515
1616 A new `/settings…` page route goes in `src/http/settings_routes.ts` (`settingsRoutes`), not `src/http/server.ts`.
1717
18 State shared between modules through process memory, not the database, is named here. `src/perf/views_report.ts`: `readVisits` holds each scope and window's visits for 15 seconds (dropped by `flushViews`), so a page and its probe share one read; `src/perf/views_store.ts`: the view buffer, flushed every 10 seconds. The traffic sources view's terminal door is `src/tui/screens/trafficview.ts` (palette entry for admins, per-app command for a repository's admins). `src/runs/held.ts`: each dispatcher pass (`src/crew/dispatch.ts`, `pump`) records which queued agent runs it held and why (valve, pace, limit), and vitals (`src/kernel/vitals.ts`) leaves those out of `queue:agent`. A record older than `HELD_FRESH_MS` is ignored, so a stopped dispatcher cannot hide its queue. `src/runs/stoppable.ts`: the checks in flight in this process by run id (`src/checks/service.ts` `start` tracks each), so a person's cancel (`cancelRepoRun`) aborts the check and its process group ends with it; agent runs keep `src/crew/running.ts`. `src/apps/egress_refused.ts`: refused-host counts per repository, flushed to disk every 5 seconds (`flushRefused` forces it).
18 State shared between modules through process memory, not the database, is named here. `src/perf/views_report.ts`: `readVisits` holds each scope and window's visits for 15 seconds (dropped by `flushViews`), so a page and its probe share one read; `src/perf/views_store.ts`: the view buffer, flushed every 10 seconds. The traffic sources view's terminal door is `src/tui/screens/trafficview.ts` (palette entry for admins, per-app command for a repository's admins). `src/runs/held.ts`: each dispatcher pass (`src/crew/dispatch.ts`, `pump`) records which queued agent runs it held and why (valve, pace, limit), and vitals (`src/kernel/vitals.ts`) leaves those out of `queue:agent`. A record older than `HELD_FRESH_MS` is ignored, so a stopped dispatcher cannot hide its queue. `src/runs/stoppable.ts`: the checks in flight in this process by run id (`src/checks/service.ts` `start` tracks each), so a person's cancel (`cancelRepoRun`) aborts the check and its process group ends with it; agent runs keep `src/crew/running.ts`. `src/apps/egress_refused.ts`: refused-host counts per repository, flushed to disk every 5 seconds (`flushRefused` forces it). `src/spawn/usage.ts`: the last disk reading of each child's data directory (30 s), and a cell child's last reading of its own cgroup CPU (`selfUse`, #10853), from which the next reading's cores are taken; `forgetSelfCpu` drops it.
1919
2020 `src/perf/loop_worst.ts`: `loopWorst`, the worst event-loop lag of the last 10 s in ten one-second slots, fed by one unref'd 50 ms timer that the first `serve` starts (`startLoopWorst`), and read by every response's `lag;dur` and by `/healthz` asked for JSON (`loopLagMs`). Tests listing a request's phases read `requestTiming` (`test/helpers/request-timing.ts`), which leaves `lag` out.
2121
docs/features/forge-size-use.md added +13
@@ -0,0 +1,13 @@
1 # A forge's size and use on /system (#10853)
2
3 **A child forge** in a cell of its own opens /system (or the terminal's System command) to a "This forge's size: small" block (or medium, large), then memory, CPU and disk in use against it, one line each. The forge that made it sets the size, and it cannot be changed from inside.
4
5 - **Memory**: the whole cell's `memory.current`, from its cgroup.
6 - **CPU**: cores = Δ`cpu.stat` `usage_usec` / Δt between two readings at least a second apart. The first reading averages over the time since the process started.
7 - **Disk**: a bounded walk of the data directory, taken in the background and kept 30 s. The first view says "measuring".
8
9 The parent passes the size and the cgroup directory in the child's environment (`FORGE_CELL_SIZE`, `FORGE_CELL_CGROUP_DIR`; `cellEnv` in `src/spawn/cell.ts`). Where the kernel does not hold the cell's caps, there is no cgroup to read. Memory and CPU are then this process's alone, and the page and terminal say so. A forge that is no cell child shows no such block.
10
11 **A parent** lists every finished child under "Child forges", each name a link to the child, with its size and its use against it, read the same way as on `/forges`. A hosted child reads "shared with this forge" and a stopped cell reads "not running".
12
13 Both are visible only to the forge's owner (`canSeeMemory`, the same check as the memory block and the terminal entry). Anyone else who opens /system gets neither.
src/actions/spawn.ts +54 −15
@@ -12,6 +12,7 @@
1212 * idempotent — and a finished one is answered, not made twice.
1313 */
1414
15 import { readdirSync, readFileSync } from "node:fs";
1516 import { mkdir, readdir, rename } from "node:fs/promises";
1617 import { join } from "node:path";
1718 import { createApp, type App } from "../app.ts";
@@ -159,11 +160,25 @@ const ledgerPath = (app: App, name: string): string =>
159160 export const liveChildUrl = (app: App, name: string): string =>
160161 childPublicUrl(name, parentOrigin(app.config));
161162
163 /** A ledger file's text as a `Ledger`, or undefined when it is not JSON or lacks name, bodyId and done: the one check every reader shares. */
164 export function parseLedger(text: string): Ledger | undefined {
165 let v: unknown;
166 try {
167 v = JSON.parse(text);
168 } catch {
169 return undefined;
170 }
171 if (typeof v !== "object" || v === null) return undefined;
172 const o = v as Record<string, unknown>;
173 if (typeof o.name !== "string" || typeof o.bodyId !== "string" || typeof o.done !== "boolean") return undefined;
174 return v as Ledger;
175 }
176
162177 async function readLedger(app: App, name: string): Promise<Ledger | undefined> {
163178 const file = Bun.file(ledgerPath(app, name));
164179 if (!(await file.exists())) return undefined;
165180 try {
166 return (await file.json()) as Ledger;
181 return parseLedger(await file.text());
167182 } catch {
168183 return undefined;
169184 }
@@ -710,6 +725,21 @@ export interface Child {
710725 readonly placementNote: string | null;
711726 }
712727
728 /** One ledger as a `Child`. */
729 const childOf = (app: App, ledger: Ledger): Child => ({
730 name: ledger.name,
731 url: liveChildUrl(app, ledger.name),
732 bodyId: ledger.bodyId,
733 done: ledger.done,
734 for: ledger.forWhom ?? null,
735 apps: ledger.apps ?? [],
736 createdAt: ledger.createdAt ?? null,
737 size: isSize(ledger.size) ? ledger.size : DEFAULT_SIZE,
738 ...(typeof ledger.template === "string" ? { template: ledger.template } : {}),
739 placement: ledger.placement ?? "hosted",
740 placementNote: ledger.placementNote ?? null,
741 });
742
713743 /** The children this forge has made or begun, from its ledger, by name. */
714744 export async function listChildren(app: App): Promise<Child[]> {
715745 const dir = join(app.config.dataDir, "spawn");
@@ -717,20 +747,29 @@ export async function listChildren(app: App): Promise<Child[]> {
717747 const out: Child[] = [];
718748 for (const f of files.filter((f) => f.endsWith(".json")).sort()) {
719749 const ledger = await readLedger(app, f.slice(0, -".json".length));
720 if (ledger)
721 out.push({
722 name: ledger.name,
723 url: liveChildUrl(app, ledger.name),
724 bodyId: ledger.bodyId,
725 done: ledger.done,
726 for: ledger.forWhom ?? null,
727 apps: ledger.apps ?? [],
728 createdAt: ledger.createdAt ?? null,
729 size: isSize(ledger.size) ? ledger.size : DEFAULT_SIZE,
730 ...(typeof ledger.template === "string" ? { template: ledger.template } : {}),
731 placement: ledger.placement ?? "hosted",
732 placementNote: ledger.placementNote ?? null,
733 });
750 if (ledger) out.push(childOf(app, ledger));
751 }
752 return out;
753 }
754
755 /** `listChildren`, read at once: for a terminal command that draws in the same turn (one small file per child). */
756 export function listChildrenNow(app: App): Child[] {
757 const dir = join(app.config.dataDir, "spawn");
758 let files: string[];
759 try {
760 files = readdirSync(dir);
761 } catch {
762 return [];
763 }
764 const out: Child[] = [];
765 for (const f of files.filter((f) => f.endsWith(".json")).sort()) {
766 let ledger: Ledger | undefined;
767 try {
768 ledger = parseLedger(readFileSync(join(dir, f), "utf8"));
769 } catch {
770 ledger = undefined; // unreadable: skipped, as listChildren skips it
771 }
772 if (ledger) out.push(childOf(app, ledger));
734773 }
735774 return out;
736775 }
src/apps/contain.ts +3
@@ -300,6 +300,8 @@ export interface Cell {
300300 readonly breach: () => string | undefined;
301301 /** Stop watching and close the cell's sockets. */
302302 readonly close: () => void;
303 /** The cell's own cgroup directory, where the kernel holds its caps; absent when only the watch does. */
304 readonly cgroupDir?: string;
303305 }
304306
305307 // ---- can this host hold a cell? ----
@@ -635,6 +637,7 @@ export function openCell(
635637 argv,
636638 env,
637639 watch,
640 ...(cgroup === undefined ? {} : { cgroupDir: cgroup.dir }),
638641 breach: () => breach ?? killed(),
639642 close: () => {
640643 const kernel = breach === undefined ? killed() : undefined;
src/spawn/cell.ts +10 −3
@@ -21,6 +21,7 @@ import { childDirOf } from "../actions/spawn.ts";
2121 import { openBeatDoor, parentLocal, type BeatDoor } from "./beat-door.ts";
2222 import { hostedPlacement, parentOrigin, childPublicUrl, type BodyStore, type Placement } from "./placement.ts";
2323 import { SIZE_IO, SIZES, type Size } from "./sizes.ts";
24 import { CGROUP_DIR_ENV, SIZE_ENV } from "./usage.ts";
2425 import { childModelHosts, childRegistryHosts } from "../core/capacity.ts";
2526
2627 export const CELL_PREFIX = "cell-";
@@ -148,14 +149,20 @@ export function cellSpecFor(input: {
148149 };
149150 }
150151
151 /** The child forge's environment: short, explicit, and its own. */
152 export function cellEnv(input: { port: number; dataDir: string; url: string; parentPort?: number }): Record<string, string> {
152 /**
153 * The child forge's environment: short, explicit, and its own. Its size and
154 * its cgroup's directory (when the kernel holds the caps) let it read its own
155 * use against its size (`selfUse`, `spawn/usage.ts`; #10853).
156 */
157 export function cellEnv(input: { port: number; dataDir: string; url: string; parentPort?: number; size?: Size; cgroupDir?: string }): Record<string, string> {
153158 const env = childEnv({
154159 PORT: String(input.port),
155160 FORGE_HOST: "127.0.0.1",
156161 FORGE_APP_URL: input.url,
157162 FORGE_APP_DATA: input.dataDir,
158163 NODE_ENV: "production",
164 ...(input.size === undefined ? {} : { [SIZE_ENV]: input.size }),
165 ...(input.cgroupDir === undefined ? {} : { [CGROUP_DIR_ENV]: input.cgroupDir }),
159166 ...(input.parentPort === undefined ? {} : { FORGE_PARENT_LOCAL: parentLocal(input.parentPort) }),
160167 });
161168 delete env["USER"];
@@ -270,7 +277,7 @@ function makeRegistry(parent: App): CellChildren {
270277 const command = `exec ${JSON.stringify(process.execPath)} ${JSON.stringify(MAIN)}`;
271278 const proc = Bun.spawn(cell.argv(command), {
272279 cwd: dirname(MAIN),
273 env: cell.env(cellEnv({ port, dataDir: dir, url, parentPort })),
280 env: cell.env(cellEnv({ port, dataDir: dir, url, parentPort, size: input.size, ...(cell.cgroupDir === undefined ? {} : { cgroupDir: cell.cgroupDir }) })),
274281 stdin: "ignore",
275282 stdout: "pipe",
276283 stderr: "pipe",
src/spawn/usage.ts +146 −1
@@ -5,9 +5,11 @@
55 * figures of its own and reads "shared". Plain data in, plain data out.
66 */
77
8 import { readFileSync } from "node:fs";
9 import { join } from "node:path";
810 import { diskBytes, cellCounters } from "../apps/contain.ts";
911 import type { Child } from "../actions/spawn.ts";
10 import { SIZES, type Size, type SizeLimits } from "./sizes.ts";
12 import { isSize, SIZES, type Size, type SizeLimits } from "./sizes.ts";
1113
1214 export type ForgeUse =
1315 | { readonly kind: "shared" }
@@ -95,3 +97,146 @@ export function useLine(size: Size, u: ForgeUse): string {
9597 const w = useWords(u);
9698 return u.kind === "shared" ? `${size} · shared` : `${sizeWords(size)} · memory ${w.memory} · cpu ${w.cpu} · disk ${w.disk}`;
9799 }
100
101 // ---- a child reading itself (#10853) ----
102
103 /** The child's size, as its parent set it (`cellEnv`, `spawn/cell.ts`); unset on a forge that is no cell child. */
104 export const SIZE_ENV = "FORGE_CELL_SIZE";
105 /** The child's own cgroup directory, absolute, as its parent made it; unset where only the watch holds the caps. */
106 export const CGROUP_DIR_ENV = "FORGE_CELL_CGROUP_DIR";
107
108 /** A forge in a cell, reading its own use against its size. */
109 export interface SelfUse {
110 readonly size: Size;
111 readonly use: Extract<ForgeUse, { kind: "cell" }>;
112 /** Where memory and CPU were read: the whole cell's cgroup, or this process alone (no cgroup to read). */
113 readonly from: "cgroup" | "process";
114 }
115
116 /** What `selfUse` reads, injectable so a test hands it a cgroup of its own. */
117 export interface SelfSource {
118 readonly env: Readonly<Record<string, string | undefined>>;
119 readonly dataDir: string;
120 readonly now: number;
121 /** When this process started, ms; the CPU before the first reading is averaged over it. */
122 readonly startedAt: number;
123 readonly read: (path: string) => string | undefined;
124 /** This process's own figures, for a cell whose cgroup is not readable. */
125 readonly rssBytes: () => number;
126 readonly cpuUsec: () => number;
127 /** The data directory's bytes, or undefined before the first reading (`diskOf`). */
128 readonly disk: (dir: string, now: number) => number | undefined;
129 }
130
131 const readOr = (path: string): string | undefined => {
132 try {
133 return readFileSync(path, "utf8");
134 } catch {
135 return undefined;
136 }
137 };
138
139 const STARTED_AT = Date.now() - Math.round(process.uptime() * 1000);
140
141 export const realSelf = (dataDir: string, now = Date.now()): SelfSource => ({
142 env: process.env,
143 dataDir,
144 now,
145 startedAt: STARTED_AT,
146 read: readOr,
147 rssBytes: () => process.memoryUsage.rss(),
148 cpuUsec: () => {
149 const c = process.cpuUsage();
150 return c.user + c.system;
151 },
152 disk: diskOf,
153 });
154
155 /** `memory.current`: one integer, bytes. */
156 export function parseMemoryCurrent(text: string | undefined): number | undefined {
157 const n = text === undefined ? NaN : Number(text.trim());
158 return Number.isSafeInteger(n) && n >= 0 ? n : undefined;
159 }
160
161 /** `cpu.stat`'s `usage_usec`: user plus system CPU of every process in the cgroup, µs. */
162 export function parseUsageUsec(text: string | undefined): number | undefined {
163 const m = text === undefined ? null : /^usage_usec (\d+)$/m.exec(text);
164 const n = m === null ? NaN : Number(m[1]);
165 return Number.isSafeInteger(n) ? n : undefined;
166 }
167
168 /**
169 * The last CPU reading and the cores it gave, kept in process memory (named
170 * in AGENTS.md): cores are the CPU µs between two readings over the
171 * wall-clock µs between them (cores = Δusage_usec / Δt_µs). A reading under
172 * a second after the last gives the last figure again; the first gives the
173 * average since the process started.
174 */
175 let lastCpu: { at: number; usec: number; cores: number | null; from: "cgroup" | "process" } | undefined;
176 const CPU_MIN_SPAN_MS = 1000;
177
178 /** Forget the kept CPU reading; for tests. */
179 export function forgetSelfCpu(): void {
180 lastCpu = undefined;
181 }
182
183 /** Cores from two cumulative readings `ms` apart, to two places; null over a span too short to say. */
184 export function coresBetween(usecBefore: number, usecAfter: number, ms: number): number | null {
185 if (ms < CPU_MIN_SPAN_MS || usecAfter < usecBefore) return null;
186 return Math.round(((usecAfter - usecBefore) / (ms * 1000)) * 100) / 100;
187 }
188
189 /** This forge's own use against its size, or null when it is no cell child (no size in its environment). */
190 export function selfUse(s: SelfSource): SelfUse | null {
191 const raw = s.env[SIZE_ENV];
192 if (!isSize(raw)) return null;
193 const dir = s.env[CGROUP_DIR_ENV];
194 const cg = dir === undefined || !dir.startsWith("/") ? undefined : dir;
195 const mem = cg === undefined ? undefined : parseMemoryCurrent(s.read(join(cg, "memory.current")));
196 const usage = cg === undefined ? undefined : parseUsageUsec(s.read(join(cg, "cpu.stat")));
197 const from = mem !== undefined && usage !== undefined ? "cgroup" : "process";
198 const memoryBytes = from === "cgroup" ? mem! : s.rssBytes();
199 const usec = from === "cgroup" ? usage! : s.cpuUsec();
200 let cores: number | null;
201 if (lastCpu === undefined || lastCpu.from !== from || usec < lastCpu.usec) {
202 // The cgroup was made as this process started, so its count runs from about then.
203 cores = coresBetween(0, usec, s.now - s.startedAt);
204 lastCpu = { at: s.now, usec, cores, from };
205 } else if (s.now - lastCpu.at < CPU_MIN_SPAN_MS) {
206 cores = lastCpu.cores;
207 } else {
208 cores = coresBetween(lastCpu.usec, usec, s.now - lastCpu.at);
209 lastCpu = { at: s.now, usec, cores, from };
210 }
211 const disk = s.disk(s.dataDir, s.now);
212 return {
213 size: raw,
214 from,
215 use: {
216 kind: "cell",
217 limits: SIZES[raw],
218 memoryMiB: Math.round(memoryBytes / 1024 / 1024),
219 cores,
220 diskMiB: disk === undefined ? null : Math.round(disk / 1024 / 1024),
221 },
222 };
223 }
224
225 /** One child as the parent's machines page lists it: its name, its size and its use against it. */
226 export interface ChildUseRow {
227 readonly name: string;
228 /** Where the child answers, for the row's link. */
229 readonly url: string;
230 readonly size: Size;
231 readonly use: ForgeUse;
232 }
233
234 /** Every finished child's use against its size, by name; `dirOf` is where each one's data lives. */
235 export function childrenUse(children: readonly Child[], dirOf: (bodyId: string) => string, now = Date.now()): ChildUseRow[] {
236 return children
237 .filter((c) => c.done)
238 .map((c) => ({ name: c.name, url: c.url, size: c.size, use: forgeUse(c, dirOf(c.bodyId), now) }));
239 }
240
241 /** The share of a limit used, percent, for a meter; 0 when unknown. */
242 export const shareOf = (used: number | null, limit: number): number => (used === null || limit <= 0 ? 0 : (used / limit) * 100);
src/tui/screens/system.ts +51 −6
@@ -15,6 +15,10 @@
1515 * its JS heap, an hour a row for the last 24 h, read through `memoryHours`
1616 * like the page. The gate is the page's (`canSeeMemory`): anyone but a forge
1717 * admin who is a person gets no rows. Read-only.
18 *
19 * Under the forge line, as /system says it (#10853): a child forge in a cell
20 * its own size and use against it (`selfLines`), and a parent each child's
21 * use against its size (`childLines`), both read through `spawn/usage.ts`.
1822 */
1923
2024 import type { Ctx, Screen } from "../app/app.ts";
@@ -38,6 +42,8 @@ import { egressReportedLine } from "../../system/egress_line.ts";
3842 import { featureHealth } from "../../system/feature_health.ts";
3943 import { readerPaces } from "../../system/telemetry_reads.ts";
4044 import { live as liveMachine } from "../../system/live.ts";
45 import { childDirOf, listChildrenNow } from "../../actions/spawn.ts";
46 import { childrenUse, realSelf, selfUse, sizeWords, useWords, type ChildUseRow, type SelfUse } from "../../spawn/usage.ts";
4147
4248 const WIDTH = 72;
4349 const GROUPS = 6;
@@ -309,6 +315,39 @@ export function memoryScreen(session: Session, ctx: Ctx): Screen {
309315 };
310316 }
311317
318 /** This forge's size and its use against it, a line each, as /system's "This forge's size" says it; none on a forge that is no cell child. */
319 export function selfLines(self: SelfUse | null): string[] {
320 if (self === null) return [];
321 const w = useWords(self.use);
322 return [
323 `size ${sizeWords(self.size)}`,
324 ` memory ${w.memory}${self.from === "process" ? " (this process alone)" : ""}`,
325 ` cpu ${self.use.cores === null ? `measuring · ${w.cpu}` : w.cpu}`,
326 ` disk ${w.disk}`,
327 ];
328 }
329
330 /** Each child forge's use against its size, a line each, as /system's "Child forges" says it. */
331 export function childLines(rows: readonly ChildUseRow[], inner = WIDTH - 6): string[] {
332 if (rows.length === 0) return [];
333 return [
334 "Child forges",
335 ...rows.flatMap((r) => {
336 if (r.use.kind === "shared") return [truncate(`${pad(r.name, 16)} ${r.size} · shared with this forge`, inner)];
337 const w = useWords(r.use);
338 const parts = r.use.kind === "idle" ? ["not running"] : [`memory ${w.memory}`, `cpu ${w.cpu}`, `disk ${w.disk}`];
339 // Figures go on whole, never split across lines: a part that does not fit starts the next line.
340 const lines: string[] = [];
341 for (const part of parts) {
342 const last = lines.length - 1;
343 if (last >= 0 && lines[last]!.length + 3 + part.length <= inner - 2) lines[last] += ` · ${part}`;
344 else lines.push(part);
345 }
346 return [truncate(`${pad(r.name, 16)} ${sizeWords(r.size)}`, inner), ...lines.map((l) => truncate(` ${l}`, inner))];
347 }),
348 ];
349 }
350
312351 /** The egress fault, one line, as /system says it; nothing when egress is enforced. */
313352 export function egressLines(session: Session): string[] {
314353 const said = egressReportedLine(session.db);
@@ -322,10 +361,16 @@ export const entry: PaletteEntry = {
322361 hint: "the machine now: CPU, memory, disks, processes",
323362 after: ["flow", "system-memory", "boards"],
324363 offered: (session) => session.user?.isAdmin === true && session.user.kind !== "agent",
325 run: (session, ctx) => ctx.push(systemScreen(ctx, [
326 ...forgeLines(readHealth(session.app, session.user)),
327 ...egressLines(session),
328 "",
329 ...systemLines(liveMachine().read()),
330 ], session)),
364 run: (session, ctx) => {
365 const self = selfLines(selfUse(realSelf(session.app.config.dataDir)));
366 const kids = childLines(childrenUse(listChildrenNow(session.app), (bodyId) => childDirOf(session.app, bodyId)));
367 ctx.push(systemScreen(ctx, [
368 ...forgeLines(readHealth(session.app, session.user)),
369 ...egressLines(session),
370 ...(self.length === 0 ? [] : ["", ...self]),
371 ...(kids.length === 0 ? [] : ["", ...kids]),
372 "",
373 ...systemLines(liveMachine().read()),
374 ], session));
375 },
331376 };
src/web/pages/system.ts +51 −1
@@ -15,6 +15,9 @@
1515 * Then the in-process caches that answer strangers' code pages (#9514), each
1616 * with its entries, bytes held beside its cap, and hits and misses: together
1717 * they may hold ~128 MB, so what they hold is shown where memory is.
18 * A child forge in a cell shows its own size and its use against it, one
19 * line each for memory, CPU and disk; a parent lists each child's use
20 * against its size (#10853, `spawn/usage.ts`).
1821 */
1922 import { canSeeMemory, memoryHours } from "../../boards/memory.ts";
2023 import { telemetryReader } from "../../actions/telemetry.ts";
@@ -40,6 +43,8 @@ import { COLUMN_BYTES, PARTS_BYTES, keptPartsStats, treeCacheStats } from "./tre
4043 import { featureHealth, type FeatureHealth } from "../../system/feature_health.ts";
4144 import { egressReportedLine } from "../../system/egress_line.ts";
4245 import { badge, chartBox, meter, pill, row } from "../ui.ts";
46 import { childDirOf, listChildren } from "../../actions/spawn.ts";
47 import { childrenUse, realSelf, selfUse, shareOf, sizeWords, useWords, type ChildUseRow, type SelfUse } from "../../spawn/usage.ts";
4348
4449 /** A group's memory, and a cell's beside its cap and who holds the cap. */
4550 const memoryOf = (g: { rssBytes: number; capBytes: number | null; capBy: "kernel" | "watch" | null }): Html =>
@@ -67,6 +72,45 @@ const meterRow = (name: string, value: number | null, max: number, figure: strin
6772 const figureRow = (name: string, figure: string, data: Record<string, string> = {}): Html =>
6873 row({ title: name, trail: html`<span class="mono">${figure}</span>`, data });
6974
75 /** This forge's own size and use when it is a child in a cell (#10853); nothing on any other forge. */
76 export function selfSection(self: SelfUse | null): Html {
77 if (self === null) return html``;
78 const { use, size } = self;
79 const w = useWords(use);
80 return html`<section class="settings-block" id="sys-self" data-size="${size}" data-from="${self.from}"><h2>This forge's size: ${size}</h2>
81 <ul class="list" aria-label="This forge's use against its size">
82 ${meterRow("memory", use.memoryMiB, use.limits.memoryMiB, w.memory)}
83 ${meterRow("cpu", use.cores, use.limits.cpu, use.cores === null ? `measuring · ${w.cpu}` : w.cpu)}
84 ${meterRow("disk", use.diskMiB, use.limits.diskMiB, w.disk)}
85 </ul>
86 <p class="small muted">Set by the forge that made this one. ${self.from === "process" ? "Memory and CPU count this process alone. " : ""}Disk is measured every 30 seconds.</p></section>`;
87 }
88
89 /** What a child row says beside its meter: a cell's three figures, or why there are none. */
90 const childSaid = (r: ChildUseRow): string => {
91 if (r.use.kind === "shared") return "shared with this forge";
92 if (r.use.kind === "idle") return "not running";
93 const w = useWords(r.use);
94 return `memory ${w.memory} · cpu ${w.cpu} · disk ${w.disk}`;
95 };
96
97 /** Each child this forge made, its use against its size (#10853); nothing when it made none. */
98 export function childrenSection(rows: readonly ChildUseRow[]): Html {
99 if (rows.length === 0) return html``;
100 return html`<section class="settings-block" id="sys-children"><h2>Child forges</h2>
101 <ul class="list" aria-label="Each child forge's use against its size">${rows.map((r) => {
102 const said = childSaid(r);
103 const share = r.use.kind === "cell" ? shareOf(r.use.memoryMiB, r.use.limits.memoryMiB) : null;
104 return row({
105 title: /^https?:\/\//.test(r.url) ? html`<a href="${r.url}">${r.name}</a>` : r.name,
106 titleAttr: r.name,
107 meta: html`${share === null ? null : html`${meter(share, `${r.name} memory`, share >= 85 ? "attn" : undefined)} `}<span class="mono small">${said}</span>`,
108 trail: html`<span class="mono small">${sizeWords(r.size)}</span>`,
109 data: { child: r.name, use: r.use.kind },
110 });
111 })}</ul></section>`;
112 }
113
70114 /** Processes grouped forge, crew, check, app, host; the client re-sorts them by CPU or memory (`system-live.ts`). */
71115 const KIND_ORDER = ["forge", "crew", "check", "app", "host"] as const;
72116 function procTable(v: SystemView): Html {
@@ -252,7 +296,7 @@ export function systemStream(ctx: Ctx): Response {
252296 }
253297
254298 // GET /system
255 export function systemPage(ctx: Ctx): Response {
299 export async function systemPage(ctx: Ctx): Promise<Response> {
256300 if (!ctx.auth.user) return signInFirst(ctx);
257301 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
258302 const reader = timePhase(ctx.request, "reader", () => telemetryReader(ctx.app, ctx.auth.user!));
@@ -271,6 +315,10 @@ export function systemPage(ctx: Ctx): Response {
271315 const egressLine: Html | null =
272316 egressSaid === null ? null : html`<p class="small" id="egress-reported">${egressSaid} <a href="/settings/capacity">Capacity</a></p>`;
273317 const memory = memorySection(ctx);
318 // Size and children are the owner's to see, as the terminal's System command is (`canSeeMemory`).
319 const owner = canSeeMemory(ctx.auth.user);
320 const self = owner ? timePhase(ctx.request, "self", () => selfUse(realSelf(ctx.app.config.dataDir))) : null;
321 const children = owner ? childrenUse(await listChildren(ctx.app), (bodyId) => childDirOf(ctx.app, bodyId)) : [];
274322 const stalls = timePhase(ctx.request, "stalls", () => stallHour(loopWatch.tally(performance.now())));
275323 const rendering = performance.now();
276324 const r = regions(v);
@@ -293,6 +341,8 @@ export function systemPage(ctx: Ctx): Response {
293341 ${processCaps}
294342 ${egressLine}
295343 <p class="small mono" id="temps">${at("temps")}</p></section>
344 ${selfSection(self)}
345 ${childrenSection(children)}
296346 <section class="settings-block"><h2>Disks and network</h2>
297347 <ul class="list" id="sys-disks">${at("sys-disks")}</ul></section>
298348 <section class="settings-block"><h2>Processes</h2>
test/spawn-self-use.test.ts added +251
@@ -0,0 +1,251 @@
1 /**
2 * A child forge shows its size and use, and its parent's /system shows each
3 * child's use against its size (#10853). The figures are checked against a
4 * hand calculation: cores = Δusage_usec / Δt_µs, MiB = bytes / 2^20.
5 */
6 import { afterAll, beforeAll, beforeEach, describe, expect, test } from "bun:test";
7 import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { createApp } from "../src/app.ts";
11 import { listChildren, listChildrenNow, parseLedger, type Child } from "../src/actions/spawn.ts";
12 import { createToken } from "../src/core/tokens.ts";
13 import { createUser } from "../src/core/users.ts";
14 import { serve } from "../src/http/server.ts";
15 import { cellEnv } from "../src/spawn/cell.ts";
16 import {
17 CGROUP_DIR_ENV,
18 childrenUse,
19 coresBetween,
20 forgetSelfCpu,
21 parseMemoryCurrent,
22 parseUsageUsec,
23 selfUse,
24 SIZE_ENV,
25 type SelfSource,
26 } from "../src/spawn/usage.ts";
27 import { childLines, selfLines } from "../src/tui/screens/system.ts";
28 import { childrenSection, selfSection } from "../src/web/pages/system.ts";
29 import { quietLogs } from "./helpers/globals.ts";
30 import { settleApp } from "./helpers/tui.ts";
31 import { createForgeApp, forgeCommands } from "../src/tui/forge.ts";
32 import { headlessTerminal } from "../src/tui/term/terminal.ts";
33 import { getUserByName } from "../src/core/users.ts";
34
35 quietLogs();
36
37 const MiB = 1_048_576;
38 const CG = "/sys/fs/cgroup/forge/cells/f1";
39
40 /** A cell child of `size` whose cgroup holds `files`; the clock and the disk are the test's. */
41 const source = (o: { size?: string; files?: Record<string, string>; now: number; diskBytes?: number; cg?: string }): SelfSource => ({
42 env: { ...(o.size === undefined ? {} : { [SIZE_ENV]: o.size }), [CGROUP_DIR_ENV]: o.cg ?? CG },
43 dataDir: "/data",
44 now: o.now,
45 startedAt: 0,
46 read: (path) => o.files?.[path],
47 rssBytes: () => 300 * MiB,
48 cpuUsec: () => 2_000_000,
49 disk: () => o.diskBytes,
50 });
51
52 const cgFiles = (memBytes: number, usec: number): Record<string, string> => ({
53 [`${CG}/memory.current`]: `${memBytes}\n`,
54 [`${CG}/cpu.stat`]: `usage_usec ${usec}\nuser_usec ${usec - 1}\nsystem_usec 1\n`,
55 });
56
57 beforeEach(() => forgetSelfCpu());
58
59 describe("the cell tells the child its size", () => {
60 test("cellEnv carries the size and the cgroup directory", () => {
61 const env = cellEnv({ port: 4000, dataDir: "/d", url: "https://x.test", size: "medium", cgroupDir: CG });
62 expect(env[SIZE_ENV]).toBe("medium");
63 expect(env[CGROUP_DIR_ENV]).toBe(CG);
64 const bare = cellEnv({ port: 4000, dataDir: "/d", url: "https://x.test" });
65 expect(bare[SIZE_ENV]).toBeUndefined();
66 expect(bare[CGROUP_DIR_ENV]).toBeUndefined();
67 });
68 });
69
70 describe("reading the cgroup", () => {
71 test("memory.current and cpu.stat parse; junk does not", () => {
72 expect(parseMemoryCurrent("1073741824\n")).toBe(1024 * MiB);
73 expect(parseMemoryCurrent("max")).toBeUndefined();
74 expect(parseMemoryCurrent(undefined)).toBeUndefined();
75 expect(parseUsageUsec("usage_usec 123\nuser_usec 100\n")).toBe(123);
76 expect(parseUsageUsec("user_usec 100\n")).toBeUndefined();
77 });
78
79 test("cores between two readings: 3 s of CPU over 2 s is 1.5 cores; under a second says nothing", () => {
80 expect(coresBetween(5_000_000, 8_000_000, 2000)).toBe(1.5);
81 expect(coresBetween(0, 1_000_000, 999)).toBeNull();
82 expect(coresBetween(9, 1, 5000)).toBeNull();
83 });
84 });
85
86 describe("selfUse", () => {
87 test("no size in the environment: not a cell child, nothing to show", () => {
88 expect(selfUse(source({ now: 10_000 }))).toBeNull();
89 expect(selfUse(source({ size: "huge", now: 10_000 }))).toBeNull();
90 expect(selfSection(null).text).toBe("");
91 expect(selfLines(null)).toEqual([]);
92 });
93
94 test("a medium child reads its cgroup: 1 GiB of 4 GiB, cores by hand, disk measuring then 512 MiB", () => {
95 // First reading: 5 s of CPU over the 10 s since the start = 0.5 cores.
96 const first = selfUse(source({ size: "medium", files: cgFiles(1024 * MiB, 5_000_000), now: 10_000 }))!;
97 expect(first.from).toBe("cgroup");
98 expect(first.use).toEqual({ kind: "cell", limits: { cpu: 2, memoryMiB: 4096, diskMiB: 10240 }, memoryMiB: 1024, cores: 0.5, diskMiB: null });
99 // 2 s later, 3 s more CPU: 1.5 cores.
100 const second = selfUse(source({ size: "medium", files: cgFiles(1536 * MiB, 8_000_000), now: 12_000, diskBytes: 512 * MiB }))!;
101 expect(second.use.cores).toBe(1.5);
102 expect(second.use.memoryMiB).toBe(1536);
103 expect(second.use.diskMiB).toBe(512);
104 // Under a second later: the last figure again.
105 expect(selfUse(source({ size: "medium", files: cgFiles(1536 * MiB, 9_000_000), now: 12_500 }))!.use.cores).toBe(1.5);
106
107 expect(selfLines(second)).toEqual([
108 "size medium · 4 GiB, 2 cores, 10 GiB",
109 " memory 1.5 GiB of 4 GiB",
110 " cpu 1.5 of 2 cores",
111 " disk 512 MiB of 10 GiB",
112 ]);
113 const page = selfSection(second).text;
114 expect(page).toContain('data-size="medium"');
115 expect(page).toContain('data-from="cgroup"');
116 expect(page).toContain("1.5 GiB of 4 GiB");
117 expect(page).toContain("1.5 of 2 cores");
118 expect(page).toContain("512 MiB of 10 GiB");
119 expect(page).toContain("This forge's size: medium</h2>");
120 expect(page).toContain('<p class="small muted">Set by the forge that made this one. Disk is measured every 30 seconds.</p>');
121 expect(page).not.toContain("memory.current");
122 });
123
124 test("an unreadable cgroup falls back to this process, and says so", () => {
125 const s = selfUse(source({ size: "small", now: 4000, cg: "relative/path" }))!;
126 expect(s.from).toBe("process");
127 expect(s.use.memoryMiB).toBe(300);
128 // 2 s of CPU over 4 s = 0.5 cores.
129 expect(s.use.cores).toBe(0.5);
130 expect(selfLines(s)[1]).toBe(" memory 300 MiB of 2 GiB (this process alone)");
131 expect(selfSection(s).text).toContain("Memory and CPU count this process alone.");
132 });
133 });
134
135 const child = (name: string, over: Partial<Child>): Child => ({
136 name, url: `https://${name}.test`, bodyId: `b-${name}`, done: true, for: null, apps: [], createdAt: null,
137 size: "small", placement: "hosted", placementNote: null, ...over,
138 });
139
140 describe("each child against its size", () => {
141 test("finished children only; hosted reads shared, a stopped cell reads not running", () => {
142 const rows = childrenUse(
143 [child("ada", { size: "large" }), child("bea", { placement: "cell", size: "medium" }), child("cy", { done: false })],
144 (id) => `/nowhere/${id}`,
145 );
146 expect(rows.map((r) => [r.name, r.size, r.use.kind])).toEqual([["ada", "large", "shared"], ["bea", "medium", "idle"]]);
147 expect(childLines(rows)).toEqual([
148 "Child forges",
149 "ada large · shared with this forge",
150 "bea medium · 4 GiB, 2 cores, 10 GiB",
151 " not running",
152 ]);
153 // A running cell's three figures are never cut off: they wrap under the size.
154 const running = childLines([{ name: "cy", url: "https://cy.test", size: "large", use: { kind: "cell", limits: { cpu: 4, memoryMiB: 8192, diskMiB: 40960 }, memoryMiB: 1536, cores: 0.25, diskMiB: 12595 } }]);
155 expect(running.slice(2)).toEqual([" memory 1.5 GiB of 8 GiB · cpu 0.25 of 4 cores", " disk 12.3 GiB of 40 GiB"]);
156 const html = childrenSection(rows).text;
157 expect(html).toContain('data-child="ada"');
158 expect(html).toContain("shared with this forge");
159 expect(html).toContain('<a href="https://ada.test">ada</a>');
160 expect(html).toContain(">not running</span>");
161 expect(html).not.toContain("memory not running");
162 expect(html).not.toContain('href="/forges"');
163 expect(childrenSection([]).text).toBe("");
164 expect(childLines([])).toEqual([]);
165 });
166 });
167
168 describe("GET /system, both sides", () => {
169 let dir: string;
170 let app: ReturnType<typeof createApp>;
171 let servers: Awaited<ReturnType<typeof serve>>;
172 let tok: string;
173 let guestTok: string;
174 const saved = process.env[SIZE_ENV];
175
176 beforeAll(async () => {
177 dir = await mkdtemp(join(tmpdir(), "forge-self-use-"));
178 app = createApp({ dataDir: dir, port: 0 });
179 servers = await serve(app);
180 const r = await createUser(app.db, {
181 origin: "seed", trusted: true, username: "alice", email: "[email protected]",
182 password: "a-good-passphrase", adminIfFirst: true,
183 });
184 if (!r.ok) throw new Error(r.error);
185 tok = createToken(app.db, r.value.id, "t", "admin").token;
186 const g = await createUser(app.db, {
187 origin: "seed", trusted: true, username: "bob", email: "[email protected]",
188 password: "a-good-passphrase", adminIfFirst: true,
189 });
190 if (!g.ok) throw new Error(g.error);
191 guestTok = createToken(app.db, g.value.id, "t", "admin").token;
192 await mkdir(join(dir, "spawn"), { recursive: true });
193 await writeFile(join(dir, "spawn", "ada.json"), JSON.stringify({ name: "ada", bodyId: "b-ada", done: true, size: "large", placement: "hosted" }));
194 // Not a ledger: no bodyId. Both readers skip it the same way.
195 await writeFile(join(dir, "spawn", "bad.json"), JSON.stringify({ name: "bad", done: true }));
196 });
197 afterAll(async () => {
198 saved === undefined ? delete process.env[SIZE_ENV] : (process.env[SIZE_ENV] = saved);
199 await servers.stop();
200 await rm(dir, { recursive: true, force: true });
201 });
202
203 const page = async (t = tok) => (await fetch(servers.url + "/system", { headers: { authorization: `Bearer ${t}` } })).text();
204
205 test("an invalid ledger is skipped alike by the page's reader and the terminal's", async () => {
206 expect(parseLedger('{"name":"bad","done":true}')).toBeUndefined();
207 expect(parseLedger("not json")).toBeUndefined();
208 expect(parseLedger("null")).toBeUndefined();
209 expect(parseLedger('{"name":"ok","bodyId":"b","done":false}')?.name).toBe("ok");
210 expect((await listChildren(app)).map((c) => c.name)).toEqual(["ada"]);
211 expect(listChildrenNow(app).map((c) => c.name)).toEqual(["ada"]);
212 });
213
214 test("anyone but the owner gets neither section", async () => {
215 process.env[SIZE_ENV] = "small";
216 const text = await page(guestTok);
217 expect(text).toContain("<h1");
218 expect(text).not.toContain('id="sys-children"');
219 expect(text).not.toContain('id="sys-self"');
220 });
221
222 test("the parent lists its child against its size, and shows no size of its own", async () => {
223 delete process.env[SIZE_ENV];
224 const text = await page();
225 expect(text).toContain('id="sys-children"');
226 expect(text).toContain('data-child="ada"');
227 expect(text).toContain("large · 8 GiB, 4 cores, 40 GiB");
228 expect(text).not.toContain('id="sys-self"');
229 });
230
231 test("a child in a cell shows its own size", async () => {
232 process.env[SIZE_ENV] = "small";
233 const text = await page();
234 expect(text).toContain('id="sys-self"');
235 expect(text).toContain("This forge's size: small</h2>");
236 });
237
238 test("the terminal's System command says both, the same way", async () => {
239 process.env[SIZE_ENV] = "small";
240 const alice = getUserByName(app.db, "alice")!;
241 const { app: tui, session } = createForgeApp(app, { sshUsername: "alice", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user: alice }, { terminal: headlessTerminal(120, 60), exitHooks: false });
242 tui.start();
243 forgeCommands(session)(tui.ctx).find((c) => c.id === "system")!.run();
244 await settleApp(tui);
245 const text = tui.screen.toText();
246 tui.stop?.();
247 expect(text).toContain("size small · 2 GiB, 1 core, 2 GiB");
248 expect(text).toContain("Child forges");
249 expect(text).toContain("large · shared with this forge");
250 });
251 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs