trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

The owner's phone hears at 70% and 80% disk on each machine the forge uses, and /system names the five largest directories #11812

masoncrew opened this

Check · Review · Land · Live — Its check is queued. Next: reviewer, pushed since the last review.

ConversationFiles
Ready to land

mason/forge:mason/11800-the-owner-s-phone-hears-at-70-and-80-dis → main 7 commits fork 1 thread · 1 unresolved

Review

Sign in to review this change.
13 files changed +444 −12
AGENTS.md +1
@@ -26,6 +26,7 @@ State shared between modules through process memory, not the database, is named
2626 `src/core/transfer.ts`: the old owner/name pairs a transfer left behind (`repo_redirect`), held per database after the first request reads them and dropped by `transferRepo`, so a request under a path nothing moved from costs no statement (#569).
2727
2828 `src/system/footing.ts`: `footing`, the last reading of what the forge stands on from the kube API (#11685), held 60 s so /system, its terminal screen (`src/tui/screens/footing.ts`, palette entry for admins) and the MCP `system` tool (`kind: footing`) share one read; a probe that throws is not retried for 60 s; one reading, gone on restart.
29 `src/system/largest_dirs.ts`: `largestDirs`, the last measurement of the five largest directories under the forge's data (#11800), held 60 s so /system, its terminal screen (`src/tui/screens/system.ts`, `largestLines`) and the MCP `system` tool share one asynchronous walk; a walk that fails is not retried for 60 s; one reading, gone on restart.
2930
3031 `src/apps/timer_fire.ts`: `inFlight`, the ids of timers being fired in this process, so a slow app's timer is never fired twice at once; at most the timers due, gone when each settles. The timers themselves are rows (`timer`), read at every boot, never memory.
3132
docs/features/disk-marks.md added +43
@@ -0,0 +1,43 @@
1 # Disk marks and the largest directories (#11800)
2
3 The owner hears about the disk that holds the forge's data well before it
4 fills, and can see what fills it.
5
6 ## The marks
7
8 The forge's vitals (`src/kernel/vitals.ts`) read the disk under the forge's
9 data. Past 70% used they say "Disk under the forge's data is N% used, past
10 the 70% mark."; past 80% the same sentence names the 80% mark. Further up,
11 the free-space floor and "disk nearly full" take over, as before.
12
13 The pager (`src/crew/incidents.ts`) sends each mark to the owner's phone once,
14 when the disk first passes it. Each mark is one rung: 70%, 80%, the floor,
15 nearly full. When the disk climbs to a higher rung, the lower rungs still
16 count as open. They are not paged again, and they do not recover until the
17 disk drops back under them. A disk that goes from 72% to 85% therefore pages
18 twice (70%, then 80%). It does not page again on the way down until it
19 crosses back under 70%.
20
21 The marks cover the forge's own data disk only. A machine the forge uses that
22 does not hold its data is not measured here.
23
24 ## The largest directories
25
26 Each directory directly under the forge's data is sized as `du -x` would size
27 it: blocks allocated on disk, symlinks not followed, other volumes not
28 entered. The five largest are shown, largest first. The walk never blocks a
29 request. A reading is at most a minute old, and a new one starts in the
30 background when it is older.
31
32 Who: a forge admin, or an agent granted `telemetry:read`. Everyone else sees
33 nothing.
34
35 - **/system**: the "Largest under the forge's data" section, below "Disks
36 and network", with when it was measured. While the first reading runs, it
37 says "Being measured; reload in a moment".
38 - **The terminal**: `:` → "System". The same five sit below the
39 telemetry history, with "measured … ago".
40 - **MCP**: the `system` tool returns `largestDirs` as
41 `{ at, dirs: [{ name, bytes }] }`. It is `null` while the first reading
42 runs. The tool waits at most 0.8 s for a new reading, then answers with the
43 last one.
src/crew/incidents.ts +16 −6
@@ -48,9 +48,16 @@ export const HOLD_MS = 5 * 60_000;
4848
4949 /** The vitals causes this pager owns; the rest are paged elsewhere or are settings. */
5050 export function owned(cause: string): boolean {
51 return cause === "errors" || cause === "internal" || cause === "disk" || cause === "disk-floor" || cause === "release" || cause.startsWith("queue:");
51 return cause === "errors" || cause === "internal" || DISK_LADDER.includes(cause) || cause === "release" || cause.startsWith("queue:");
5252 }
5353
54 /**
55 * The disk's rungs, lowest first. `kernel/vitals.ts` names only the highest
56 * rung the disk is on, so a rung present holds every lower rung already
57 * watched: unpaged anew and not recovered while the disk stays above it.
58 */
59 export const DISK_LADDER: readonly string[] = ["disk-70", "disk-80", "disk-floor", "disk"];
60
5461 /** One owned cause being watched. */
5562 export interface Watched {
5663 readonly since: number;
@@ -78,11 +85,14 @@ export function observe(
7885 for (const p of present) {
7986 if (!owned(p.cause)) continue;
8087 here.add(p.cause);
81 // A disk nearly full is still under the floor (`kernel/vitals.ts` names only the
82 // worse): a floor already watched holds, unpaged anew and not recovered (#10873).
83 if (p.cause === "disk" && watch.has("disk-floor")) {
84 here.add("disk-floor");
85 delete watch.get("disk-floor")!.absentSince;
88 // A disk nearly full is still under the floor, and past 80% still past 70%
89 // (`kernel/vitals.ts` names only the worse): a lower rung already watched
90 // holds, unpaged anew and not recovered (#10873, #11800).
91 for (const lower of DISK_LADDER.slice(0, Math.max(0, DISK_LADDER.indexOf(p.cause)))) {
92 const held = watch.get(lower);
93 if (held === undefined) continue;
94 here.add(lower);
95 delete held.absentSince;
8696 }
8797 const w = watch.get(p.cause);
8898 if (w === undefined) watch.set(p.cause, { since: now, sentence: p.sentence, paged: seed });
src/http/mcp/tools/system.ts +5 −2
@@ -8,11 +8,12 @@ import { lagMinutes, loopWatch, perfRecord } from "../../../perf/live.ts";
88 import { LAG_BAR_MS, overBar } from "../../../perf/lag_minutes.ts";
99 import { slowestRoutes } from "../../../perf/watch.ts";
1010 import { footing, riskLines } from "../../../system/footing.ts";
11 import { largestDirs } from "../../../system/largest_dirs.ts";
1112 import { good, bad, type Tool } from "../shared.ts";
1213
1314 export const tool: Tool = {
1415 name: "system",
15 doc: "the machine the forge runs on, now: CPU, load, memory, disk, network, processes, `stalls`: the last hour's main-thread stalls by route (`<route> (in flight)` when still running at the wake), or `job:<name>`, as {cause, ms, count}, `readings`: the last 20 minutes' worst loop lag as {at, ms, what} beside `barMs` (100) and `over`, and `readers`: who polls the telemetry and how fast, `{ kind: \"routes\" }`: the 10 slowest budgeted routes as {route, p95Ms, budgetMs, n, slowestPhase}, over budget first; a forge admin, or an agent granted `telemetry:read`. `{ kind: \"csp\" }`: the last 24 h of Content-Security-Policy violations browsers reported, `{ directive, n, newest }` per directive (`newest` the blocked host last counted), most first, counted per UTC hour from `since`, and `scriptErrors` `{ page, where, message, n, last }`, as the security page reads them; a forge admin only. `{ kind: \"footing\" }`: what the forge stands on (#11685), as /system's \"What the forge stands on\": `forgeNode`, and per piece `{ id, need, what, machine, seen, risk, ifDies, issues }` read from the live cluster, with `risks`, one line per piece not on the forge's machine; a forge admin only",
16 doc: "the machine the forge runs on, now: CPU, load, memory, disk, network, processes, `largestDirs`: the five largest directories under the forge's data as {at, dirs: [{name, bytes}]} (null while first measured; at most once a minute), `stalls`: the last hour's main-thread stalls by route (`<route> (in flight)` when still running at the wake), or `job:<name>`, as {cause, ms, count}, `readings`: the last 20 minutes' worst loop lag as {at, ms, what} beside `barMs` (100) and `over`, and `readers`: who polls the telemetry and how fast, `{ kind: \"routes\" }`: the 10 slowest budgeted routes as {route, p95Ms, budgetMs, n, slowestPhase}, over budget first; a forge admin, or an agent granted `telemetry:read`. `{ kind: \"csp\" }`: the last 24 h of Content-Security-Policy violations browsers reported, `{ directive, n, newest }` per directive (`newest` the blocked host last counted), most first, counted per UTC hour from `since`, and `scriptErrors` `{ page, where, message, n, last }`, as the security page reads them; a forge admin only. `{ kind: \"footing\" }`: what the forge stands on (#11685), as /system's \"What the forge stands on\": `forgeNode`, and per piece `{ id, need, what, machine, seen, risk, ifDies, issues }` read from the live cluster, with `risks`, one line per piece not on the forge's machine; a forge admin only",
1617 description:
1718 "The forge machine now: CPU, memory, disk, network. Admin or telemetry:read; `kind: csp`: 24 h of CSP violations (admin).",
1819 inputSchema: { type: "object", properties: { kind: { type: "string", enum: ["csp", "routes", "footing"] } } },
@@ -47,6 +48,8 @@ export const tool: Tool = {
4748 // The same 20 minutes /system draws (#11220); `what` is null when no stall past 200 ms was named.
4849 const rows = lagMinutes.readings();
4950 const readings = rows.map((r) => ({ at: new Date(r.at).toISOString(), ms: Math.round(r.ms * 10) / 10, what: r.what }));
50 return good({ ...out, stalls, barMs: LAG_BAR_MS, over: overBar(rows), readings });
51 // At most 0.8 s on the walk (#11800); the last measurement, or null, when it is slower.
52 const largest = await largestDirs.read(ctx.app.config.dataDir);
53 return good({ ...out, largestDirs: largest, stalls, barMs: LAG_BAR_MS, over: overBar(rows), readings });
5154 },
5255 };
src/kernel/vitals.ts +15 −1
@@ -54,6 +54,8 @@ function foreignNote(repo: string): string {
5454
5555 /** Free disk under this share of the volume is something to say. */
5656 export const DISK_FREE_MIN = 0.1;
57 /** Used shares of the disk under the forge's data the owner hears of, highest first (#11800). */
58 export const DISK_USED_MARKS = [0.8, 0.7] as const;
5759 /** This many error lines in an hour is a burst worth saying. */
5860 export const ERROR_BURST = 20;
5961 /** A run still running this long after it started is stuck even if it breathes. */
@@ -71,7 +73,7 @@ export interface EnvCount {
7173
7274 /**
7375 * One cause of the forge being off: the `needs` sentence, a `cause` key that
74 * holds for as long as its condition does (`disk`, `errors`, `internal`,
76 * holds for as long as its condition does (`disk`, `disk-floor`, `disk-80`, `disk-70`, `errors`, `internal`,
7577 * `queue:<kind>`, `release`, `stuck:<runId>`, `red:<repoId>`, `shared:<repoId>:<test>`), when it began by the forge's own evidence
7678 * (the oldest evidence the forge holds, else the first time this process saw
7779 * it, held until the cause clears; `at` is never the answer to a repeat
@@ -347,6 +349,18 @@ export function vitals(input: VitalsInput): Vitals {
347349 undefined,
348350 "/settings/capacity",
349351 );
352 else if (disk.totalBytes > 0) {
353 // The owner hears well before the floor (#11800): one need for the highest mark passed.
354 const used = 1 - disk.freeBytes / disk.totalBytes;
355 const mark = DISK_USED_MARKS.find((m) => used >= m);
356 if (mark !== undefined)
357 need(
358 `disk-${Math.round(mark * 100)}`,
359 `Disk under the forge's data is ${Math.floor(used * 100)}% used, past the ${Math.round(mark * 100)}% mark.`,
360 undefined,
361 "/system",
362 );
363 }
350364
351365 for (const file of openDbFiles(input.dataDir))
352366 need(`file:${file.name}`, file.sentence, undefined, "/settings/capacity");
src/system/largest_dirs.ts added +136
@@ -0,0 +1,136 @@
1 /**
2 * The five largest directories under the forge's data (#11800): each
3 * directory directly under `FORGE_DATA`, sized as `du -x` would (allocated
4 * blocks, symlinks not followed, other volumes not entered). The walk is
5 * asynchronous, so it never holds the event loop; one reading is shared by
6 * /system and the MCP `system` tool, measured at most once a minute, and
7 * no request waits on it longer than 0.8 s.
8 */
9
10 import { lstat, readdir } from "node:fs/promises";
11 import { join } from "node:path";
12
13 /** One directory under the forge's data and what it holds on disk. */
14 export interface DirSize {
15 readonly name: string;
16 readonly bytes: number;
17 }
18
19 /** What one measurement found. */
20 export interface LargestDirs {
21 /** When the measurement finished, ISO. */
22 readonly at: string;
23 readonly dirs: readonly DirSize[];
24 }
25
26 /** How many directories /system names. */
27 export const LARGEST_DIRS = 5;
28
29 /** The `n` largest, largest first; equal sizes by name, so the order is stable. */
30 export function largest(sizes: readonly DirSize[], n = LARGEST_DIRS): DirSize[] {
31 return [...sizes].sort((a, b) => b.bytes - a.bytes || a.name.localeCompare(b.name)).slice(0, n);
32 }
33
34 /** Bytes a file or directory entry takes on disk: its blocks where the system reports them. */
35 const onDisk = (s: { blocks: number; size: number }): number =>
36 Number.isFinite(s.blocks) && s.blocks > 0 ? s.blocks * 512 : s.size;
37
38 /** Everything under `path` on the device `dev`, in bytes; what cannot be read counts as nothing. */
39 async function sizeOf(path: string, dev: number): Promise<number> {
40 let total = 0;
41 const stack = [path];
42 while (stack.length > 0) {
43 const dir = stack.pop()!;
44 let names: string[];
45 try {
46 names = await readdir(dir);
47 } catch {
48 continue;
49 }
50 for (const name of names) {
51 const full = join(dir, name);
52 try {
53 const s = await lstat(full);
54 if (s.dev !== dev) continue;
55 total += onDisk(s);
56 if (s.isDirectory()) stack.push(full);
57 } catch {
58 // Gone between the listing and the stat, or not ours to read.
59 }
60 }
61 }
62 return total;
63 }
64
65 /** Measures each directory directly under `dataDir` and keeps the five largest. */
66 export async function measureLargestDirs(dataDir: string, clock: () => number = Date.now): Promise<LargestDirs> {
67 const root = await lstat(dataDir);
68 const sizes: DirSize[] = [];
69 for (const entry of await readdir(dataDir, { withFileTypes: true })) {
70 if (!entry.isDirectory()) continue;
71 const full = join(dataDir, entry.name);
72 try {
73 const s = await lstat(full);
74 if (s.dev !== root.dev) continue;
75 sizes.push({ name: entry.name, bytes: onDisk(s) + (await sizeOf(full, root.dev)) });
76 } catch {
77 // A directory removed mid-walk is not one of the largest.
78 }
79 }
80 return { at: new Date(clock()).toISOString(), dirs: largest(sizes) };
81 }
82
83 const FRESH_MS = 60_000;
84 const WAIT_MS = 800;
85
86 /** The last measurement, shared by /system and the MCP `system` tool; one per process, gone on restart. */
87 export function largestDirsCache(
88 measure: (dataDir: string) => Promise<LargestDirs> = (d) => measureLargestDirs(d),
89 clock: () => number = Date.now,
90 ) {
91 let last: { at: number; dataDir: string; value: LargestDirs } | null = null;
92 // A walk that failed waits a minute too, so a missing directory costs one try a minute.
93 let triedAt: number | null = null;
94 let pending: Promise<LargestDirs | null> | null = null;
95 const due = (dataDir: string): boolean =>
96 pending === null &&
97 (last === null || last.dataDir !== dataDir || clock() - last.at > FRESH_MS) &&
98 (triedAt === null || clock() - triedAt > FRESH_MS || last?.dataDir !== dataDir);
99 const refresh = (dataDir: string): Promise<LargestDirs | null> => {
100 triedAt = clock();
101 pending = measure(dataDir)
102 .then((value) => {
103 last = { at: clock(), dataDir, value };
104 return value;
105 })
106 .catch(() => null)
107 .finally(() => {
108 pending = null;
109 });
110 return pending;
111 };
112 const current = (dataDir: string): LargestDirs | null => (last?.dataDir === dataDir ? last.value : null);
113 return {
114 /** The last measurement, a new one started behind it when it is over a minute old; null before the first. */
115 peek(dataDir: string): LargestDirs | null {
116 if (due(dataDir)) void refresh(dataDir);
117 return current(dataDir);
118 },
119 /** A measurement no more than a minute old when one finishes within `waitMs`, else the last one, else null. */
120 async read(dataDir: string, waitMs = WAIT_MS): Promise<LargestDirs | null> {
121 const running = due(dataDir) ? refresh(dataDir) : pending;
122 if (running === null) return current(dataDir);
123 let deadline: ReturnType<typeof setTimeout> | undefined;
124 const timer = new Promise<null>((r) => {
125 deadline = setTimeout(() => r(null), waitMs);
126 });
127 try {
128 return (await Promise.race([running, timer])) ?? current(dataDir);
129 } finally {
130 clearTimeout(deadline);
131 }
132 },
133 };
134 }
135
136 export const largestDirs = largestDirsCache();
src/tui/screens/system.ts +26 −1
@@ -19,6 +19,10 @@
1919 * Under the forge line, as /system says it (#10853): a child forge in a cell
2020 * its own size and use against it (`selfLines`), and a parent each child's
2121 * use against its size (`childLines`), both read through `spawn/usage.ts`.
22 *
23 * Below the telemetry history, the five largest directories under the
24 * forge's data (#11800, `largestLines`), read through the cache /system and
25 * the MCP `system` tool share and gated as they are (`telemetryReader`).
2226 */
2327
2428 import type { Ctx, Screen } from "../app/app.ts";
@@ -35,7 +39,8 @@ import { bytes, spark } from "../../system/format.ts";
3539 import { style } from "../term/style.ts";
3640 import { pad, truncate, wrap } from "../term/width.ts";
3741 import { dialog } from "../ui/widgets.ts";
38 import { Scroll, confirmScreen, drawLines, hitScroll, later, pickerScreen } from "./common.ts";
42 import { Scroll, ago, confirmScreen, drawLines, hitScroll, later, pickerScreen } from "./common.ts";
43 import { largestDirs, type LargestDirs } from "../../system/largest_dirs.ts";
3944 import type { PaletteEntry } from "./entries.ts";
4045 import { readHealth } from "../../actions/health.ts";
4146 import { egressReportedLine } from "../../system/egress_line.ts";
@@ -154,6 +159,23 @@ export function historyLines(session: Session, hours: number): string[] {
154159 return [head, ...rows];
155160 }
156161
162 /**
163 * The five largest directories under the forge's data, as /system's
164 * "Largest under the forge's data" says them (#11800): a name and its size a
165 * line, then when it was measured. Empty for anyone the page refuses.
166 */
167 export function largestLines(session: Session, found: LargestDirs | null, inner = WIDTH - 6): string[] {
168 if (!telemetryReader(session.app, session.user)) return [];
169 const head = "Largest under the forge's data";
170 if (found === null) return [head, "being measured; open again in a moment"];
171 if (found.dirs.length === 0) return [head, "no directories under the forge's data"];
172 return [
173 head,
174 ...found.dirs.map((d) => truncate(`${pad(truncate(d.name, inner - 12), inner - 12)} ${pad(bytes(d.bytes), 11, "right")}`, inner)),
175 `measured ${ago(found.at)}`,
176 ];
177 }
178
157179 /** Who may grant telemetry:read: the MCP tool's gate (`mayBoards`), a forge admin who is a person. */
158180 const mayGrant = (session: Session | undefined): boolean => session?.user?.isAdmin === true && session.user.kind !== "agent";
159181
@@ -247,12 +269,15 @@ export function systemScreen(ctx: Ctx, lines: readonly string[], session: Sessio
247269 keys: [binding(["h"], "window"), ...(offered ? [binding(["g"], "grant or take back telemetry")] : []), binding(["enter", "esc", "q"], "close")],
248270 render(area, s) {
249271 const features = featuresLines(session);
272 // Never waits on the walk: the last measurement, a new one started behind it when stale.
273 const largest = largestLines(session, largestDirs.peek(session.app.config.dataDir));
250274 const buttons = dialog(s, area, ctx.theme.c, {
251275 title: "System",
252276 body: [
253277 ...lines,
254278 "",
255279 ...history,
280 ...(largest.length === 0 ? [] : ["", ...largest]),
256281 ...(offered ? ["", ...readerLines(session)] : []),
257282 ...(features.length === 0 ? [] : ["", ...features]),
258283 ...(offered ? ["", grantLine(session), "g: grant or take back telemetry:read"] : []),
src/web/pages/system.ts +15
@@ -49,6 +49,7 @@ import { featureHealth, type FeatureHealth } from "../../system/feature_health.t
4949 import { egressReportedLine } from "../../system/egress_line.ts";
5050 import { BREAK_STEPS, MACHINES, declaredFooting, footing, offMachine, whereWords, type Footing, type Machine } from "../../system/footing.ts";
5151 import { badge, chartBox, disclosure, meter, pill, row, time } from "../ui.ts";
52 import { LARGEST_DIRS, largestDirs, type LargestDirs } from "../../system/largest_dirs.ts";
5253 import { childDirOf, listChildren } from "../../actions/spawn.ts";
5354 import { childrenUse, realSelf, selfUse, shareOf, sizeWords, useWords, type ChildUseRow, type SelfUse } from "../../spawn/usage.ts";
5455
@@ -78,6 +79,17 @@ const meterRow = (name: string, value: number | null, max: number, figure: strin
7879 const figureRow = (name: string, figure: string, data: Record<string, string> = {}): Html =>
7980 row({ title: name, trail: html`<span class="mono">${figure}</span>`, data });
8081
82 /** The five largest directories under the forge's data (#11800), measured at most once a minute, never waited on. */
83 export function largestDirsSection(found: LargestDirs | null): Html {
84 return html`<section class="settings-block" id="sys-largest"><h2>Largest under the forge's data</h2>
85 ${found === null
86 ? html`<p class="muted small">Being measured; reload in a moment.</p>`
87 : found.dirs.length === 0
88 ? html`<p class="muted small">No directories under the forge's data.</p>`
89 : html`<ul class="list" aria-label="The ${String(LARGEST_DIRS)} largest directories under the forge's data">${found.dirs.map((d) => figureRow(d.name, bytes(d.bytes), { dir: d.name, bytes: String(d.bytes) }))}</ul>
90 <p class="muted small">Measured ${time(found.at)}</p>`}</section>`;
sentinelcrew outdated

Principle 1 / 8: "Measured …; again at most once a minute" explains the cache mechanism, which the reader does not need. Keep the time ("Measured 3 min ago") and drop the clause.
91 }
92
8193 /** This forge's own size and use when it is a child in a cell (#10853); nothing on any other forge. */
8294 export function selfSection(self: SelfUse | null): Html {
8395 if (self === null) return html``;
@@ -405,6 +417,8 @@ export async function systemPage(ctx: Ctx): Promise<Response> {
405417 const worst = stallBlame.worst(nowMono, HOUR_MS, WORST_STALLS);
406418 // Never waits on the cluster: the last reading, or the declared rows while the first read runs behind.
407419 const standsOn = footing.peek() ?? declaredFooting("the cluster is being read; reload in a moment");
420 // Never waits on the walk: the last measurement, a new one started behind it when stale.
421 const largestUnder = largestDirs.peek(ctx.app.config.dataDir);
408422 const rendering = performance.now();
409423 const r = regions(v);
410424 const at = (id: string): Html => raw(r[id]!);
@@ -434,6 +448,7 @@ export async function systemPage(ctx: Ctx): Promise<Response> {
434448 ${childrenSection(children)}
435449 <section class="settings-block"><h2>Disks and network</h2>
436450 <ul class="list" id="sys-disks">${at("sys-disks")}</ul></section>
451 ${largestDirsSection(largestUnder)}
437452 <section class="settings-block"><h2>Processes</h2>
438453 <div class="js-only small muted">Sort by <button class="btn" data-size="sm" type="button" data-sort="cpu" aria-pressed="true"><kbd>c</kbd> CPU</button>
439454 <button class="btn" data-size="sm" type="button" data-sort="mem" aria-pressed="false"><kbd>m</kbd> memory</button></div>
test/disk-floor.test.ts +2 −1
@@ -94,7 +94,8 @@ describe("below the floor", () => {
9494 expect(v.degraded.map((d) => d.cause)).toEqual(["disk-floor"]);
9595 expect(v.needs).toEqual(["Disk is under the 15% the forge keeps for itself: 12% free. Child forges sleep, and no child forge, app or check starts, until room is made."]);
9696 const at15 = vitals({ db: app.db, signals: app.signals, dataDir: app.config.dataDir, disk: () => ({ freeBytes: 15, totalBytes: 100 }) });
97 expect(at15.degraded.map((d) => d.cause)).toEqual([]);
97 // At the floor is not under it; 85% used is past the 80% mark (#11800), which says only that.
98 expect(at15.degraded.map((d) => d.cause)).toEqual(["disk-80"]);
9899 } finally {
99100 app.close();
100101 }
test/disk-marks.test.ts added +148
@@ -0,0 +1,148 @@
1 /**
2 * #11800: the owner's phone hears at 70% and 80% of the disk under the
3 * forge's data used, once per crossing and Recovered once, and /system
4 * names the five largest directories under it.
5 */
6 import { afterAll, describe, expect, test } from "bun:test";
7 import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { createApp } from "../src/app.ts";
11 import { observe, owned, type Watch } from "../src/crew/incidents.ts";
12 import { vitals } from "../src/kernel/vitals.ts";
13 import { largest, largestDirsCache, measureLargestDirs, type LargestDirs } from "../src/system/largest_dirs.ts";
14 import { largestDirsSection } from "../src/web/pages/system.ts";
15
16 const tmp = mkdtempSync(join(tmpdir(), "disk-marks-"));
17 afterAll(() => rmSync(tmp, { recursive: true, force: true }));
18
19 describe("vitals names the highest mark passed", () => {
20 const app = createApp({ dataDir: join(tmp, "vitals"), url: "https://f.test" });
21 afterAll(() => app.close());
22 const at = (usedPct: number) =>
23 vitals({ db: app.db, signals: app.signals, dataDir: app.config.dataDir, disk: () => ({ freeBytes: 100 - usedPct, totalBytes: 100 }) });
24
25 test("69% says nothing, 70% and 80% say their mark, the floor says only the floor", () => {
26 expect(at(69).degraded.map((d) => d.cause)).toEqual([]);
27 expect(at(70).degraded.map((d) => d.cause)).toEqual(["disk-70"]);
28 expect(at(70).needs).toEqual(["Disk under the forge's data is 70% used, past the 70% mark."]);
29 expect(at(79).degraded.map((d) => d.cause)).toEqual(["disk-70"]);
30 expect(at(80).degraded.map((d) => d.cause)).toEqual(["disk-80"]);
31 expect(at(88).degraded.map((d) => d.cause)).toEqual(["disk-floor"]);
32 });
33
34 test("the pager owns both marks", () => {
35 expect(owned("disk-70")).toBe(true);
36 expect(owned("disk-80")).toBe(true);
37 });
38 });
39
40 describe("the owner hears once per crossing", () => {
41 const MIN = 60_000;
42 const mark = (pct: 70 | 80) => ({ cause: `disk-${pct}`, sentence: `Disk under the forge's data is ${pct}% used, past the ${pct}% mark.` });
43
44 test("70% pages once, 80% once more, back under 80% and under 70% each say Recovered once", () => {
45 const watch: Watch = new Map();
46 let t = 0;
47 const pages: string[] = [];
48 const recovered: string[] = [];
49 const step = (minutes: number, present: { cause: string; sentence: string }[]) => {
50 t += minutes * MIN;
51 const due = observe(watch, present, t);
52 for (const k of due.pages) watch.get(k)!.paged = true;
53 for (const k of due.recovered) watch.delete(k);
54 pages.push(...due.pages);
55 recovered.push(...due.recovered);
56 };
57 step(0, []);
58 step(6, []);
59 expect(pages).toEqual([]);
60 step(1, [mark(70)]);
61 step(6, [mark(70)]);
62 for (let i = 0; i < 6; i++) step(5, [mark(70)]);
63 expect(pages).toEqual(["disk-70"]);
64 // Past 80% vitals names only disk-80: the 70% mark holds, not paged anew nor recovered.
65 step(1, [mark(80)]);
66 step(6, [mark(80)]);
67 for (let i = 0; i < 6; i++) step(5, [mark(80)]);
68 expect(pages).toEqual(["disk-70", "disk-80"]);
69 expect(recovered).toEqual([]);
70 step(1, [mark(70)]);
71 step(6, [mark(70)]);
72 expect(recovered).toEqual(["disk-80"]);
73 step(1, []);
74 step(6, []);
75 step(30, []);
76 expect(recovered).toEqual(["disk-80", "disk-70"]);
77 expect(pages).toEqual(["disk-70", "disk-80"]);
78 });
79
80 test("a disk that climbs past the floor keeps the marks under it", () => {
81 const watch: Watch = new Map();
82 const floor = { cause: "disk-floor", sentence: "floor" };
83 observe(watch, [mark(80)], 0);
84 expect(observe(watch, [mark(80)], 6 * MIN).pages).toEqual(["disk-80"]);
85 watch.get("disk-80")!.paged = true;
86 observe(watch, [floor], 7 * MIN);
87 const due = observe(watch, [floor], 20 * MIN);
88 expect(due.pages).toEqual(["disk-floor"]);
89 expect(due.recovered).toEqual([]);
90 });
91 });
92
93 describe("the five largest directories under the forge's data", () => {
94 test("ordered by size, largest first, five at most", () => {
95 const sizes = [
96 { name: "a", bytes: 3 },
97 { name: "b", bytes: 10 },
98 { name: "c", bytes: 1 },
99 { name: "d", bytes: 7 },
100 { name: "e", bytes: 7 },
101 { name: "f", bytes: 2 },
102 ];
103 expect(largest(sizes).map((d) => d.name)).toEqual(["b", "d", "e", "a", "f"]);
104 });
105
106 test("measured from the disk: a directory's files and its nested ones count, top-level files do not", async () => {
107 const data = join(tmp, "measure");
108 mkdirSync(join(data, "big", "deep"), { recursive: true });
109 mkdirSync(join(data, "small"), { recursive: true });
110 writeFileSync(join(data, "big", "deep", "blob"), Buffer.alloc(256 * 1024, 1));
111 writeFileSync(join(data, "small", "note"), "x");
112 writeFileSync(join(data, "loose.db"), Buffer.alloc(1024 * 1024, 1));
113 const found = await measureLargestDirs(data);
114 expect(found.dirs.map((d) => d.name)).toEqual(["big", "small"]);
115 expect(found.dirs[0]!.bytes).toBeGreaterThanOrEqual(256 * 1024);
116 });
117
118 test("measured at most once a minute, and a slow walk is not waited on past the wait", async () => {
119 let now = 0;
120 let walks = 0;
121 let release: (() => void) | undefined;
122 const value: LargestDirs = { at: "2026-11-12T00:00:00.000Z", dirs: [{ name: "repos", bytes: 5 }] };
123 const cache = largestDirsCache(async () => {
124 walks++;
125 if (walks === 2) await new Promise<void>((r) => (release = r));
126 return value;
127 }, () => now);
128 expect(await cache.read("/d")).toEqual(value);
129 now += 30_000;
130 expect(await cache.read("/d")).toEqual(value);
131 expect(cache.peek("/d")).toEqual(value);
132 expect(walks).toBe(1);
133 now += 31_000;
134 const started = performance.now();
135 expect(await cache.read("/d", 20)).toEqual(value);
136 expect(performance.now() - started).toBeLessThan(500);
137 expect(walks).toBe(2);
138 release!();
139 });
140
141 test("/system's section names each with its size, escaped", () => {
142 const out = String(largestDirsSection({ at: "2026-11-12T00:00:00.000Z", dirs: [{ name: "<repos>", bytes: 2 * 1024 ** 3 }] }));
143 expect(out).toContain("&lt;repos&gt;");
144 expect(out).not.toContain("<repos>");
145 expect(out).toContain('data-bytes="2147483648"');
146 expect(String(largestDirsSection(null))).toContain("Being measured");
147 });
148 });
test/system-parts.test.ts +1
@@ -51,6 +51,7 @@ test("one <h2> per part, each in a settings-block, and no card", async () => {
5151 expect(heads).toEqual([
5252 "Machine",
5353 "Disks and network",
54 "Largest under the forge's data",
5455 "Processes",
5556 "What the forge stands on",
5657 "Loop lag, last 20 minutes",
test/timers-named.test.ts +1
@@ -125,6 +125,7 @@ const ALLOWED: readonly Allowed[] = [
125125 ["src/snapshot/store.ts", "() => reject("],
126126 ["src/snapshot/lanes.ts", "() => reject("],
127127 ["src/system/footing.ts", "() => r(null)"],
128 ["src/system/largest_dirs.ts", "() => r(null)"],
128129 ["src/release/devtools.ts", "() => { if (pending.delete(n))"],
129130 ["src/release/devtools.ts", "() => { waiting.set(event"],
130131 ["src/release/prove.ts", "() => resolve(\"hung\")"],
test/tui-system.test.ts +35 −1
@@ -16,7 +16,7 @@ import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
1616 import { setTelemetryGrant } from "../src/actions/telemetry.ts";
1717 import { hasGrant } from "../src/core/telemetry.ts";
1818 import { countRead, resetTelemetryReads } from "../src/system/telemetry_reads.ts";
19 import { forgeLines, historyLines, readerLines, memoryLines, memoryScreen, systemScreen } from "../src/tui/screens/system.ts";
19 import { forgeLines, historyLines, largestLines, readerLines, memoryLines, memoryScreen, systemScreen } from "../src/tui/screens/system.ts";
2020 import { changeCapacity } from "../src/actions/capacity.ts";
2121 import { EGRESS_REPORTED } from "../src/system/egress_line.ts";
2222 import { headlessTerminal } from "../src/tui/term/terminal.ts";
@@ -106,6 +106,40 @@ describe("tui system memory", () => {
106106 });
107107 });
108108
109 describe("tui system largest directories (#11800)", () => {
110 const found = { at: new Date(Date.now() - 3 * 60_000).toISOString(), dirs: [{ name: "repos", bytes: 2 * 1024 ** 3 }, { name: "apps", bytes: 5 * MiB }] };
111
112 test("an admin and a granted agent read the five largest with when they were measured", () => {
113 const lines = largestLines(new Session(app, identity(carol)), found);
114 expect(lines[0]).toBe("Largest under the forge's data");
115 expect(lines[1]).toContain("repos");
116 expect(lines[1]).toContain("2.0 GiB");
117 expect(lines[2]).toContain("apps");
118 expect(lines[3]).toBe("measured 3m ago");
119 expect(largestLines(new Session(app, identity(carol)), null)[1]).toContain("being measured");
120 setTelemetryGrant(app, carol, "sysbot", true);
121 try {
122 expect(largestLines(new Session(app, identity(bot)), found)[1]).toContain("repos");
123 } finally {
124 setTelemetryGrant(app, carol, "sysbot", false);
125 }
126 });
127
128 test("a member, an ungranted agent and a guest read nothing", () => {
129 for (const who of [bob, bot, undefined]) expect(largestLines(new Session(app, identity(who)), found)).toEqual([]);
130 });
131
132 test("the System screen shows the section from the shared cache", async () => {
133 const terminal = headlessTerminal(120, 60);
134 const { app: tui, session } = createForgeApp(app, identity(carol), { terminal, exitHooks: false });
135 tui.start();
136 tui.ctx.push(systemScreen(tui.ctx, ["CPU"], session));
137 await settleApp(tui);
138 expect(tui.screen.toText()).toContain("Largest under the forge's data");
139 tui.stop?.();
140 });
141 });
142
109143 describe("tui system history", () => {
110144 const put = (at: string, cpu: number, mem: number, load: number) =>
111145 app.db

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs