Public: anyone can read this; the crew can change it.

The forge knows whether each child forge's writable space is a filesystem of its own, says so, and can refuse to start one that is not #10881

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

torque/forge:torque/10856-the-forge-knows-whether-each-child-forge → main 4 commits fork

  • Merged. torque/10856-the-forge-knows-whether-each-child-forge was merged into main as 92aa5d6.

Review

Sign in to review this change.
17 files changed +580 −22
docs/APPS.md +25
@@ -1240,6 +1240,31 @@ memory only: a process is not state, and a restart rebuilds every
12401240 environment from the default branch and the open pulls, exactly as
12411241 `core/reconcile.ts` rebuilds the index from git.
12421242
1243 ### A child forge's own filesystem
1244
1245 How to read the line and turn the guard on: [docs/features/child-filesystem.md](features/child-filesystem.md).
1246
1247 A child forge (`/forges`) keeps everything it writes under
1248 `<data>/children/<body id>`. The forge reads `/proc/self/mountinfo` to tell
1249 whether that directory is a filesystem of its own, a mount whose mount point
1250 is exactly that path, and reads its size with `statfs`: blocks × block size
1251 in bytes, and its inodes (`src/spawn/disk_mount.ts`). The forge mounts
1252 nothing itself. Making the filesystem is the platform's job (#10851).
1253
1254 - **On its own filesystem**, writing past its size fails with ENOSPC inside
1255 the child, and the parent's free space does not change. Its cell binds that
1256 filesystem by its real path (`cellSpecFor`). On the shared disk, only the
1257 cell's disk watch stops a child, and only after it has written too much.
1258 - **The child's line** on `/forges` and on the terminal's forges screen says
1259 `own filesystem, N GiB` (GiB = 2^30 bytes, to one place) or `shared disk`.
1260 - **`children.ownFilesystem`** (Settings → Capacity, or the `capacity` MCP
1261 tool with `key: "children.ownFilesystem"`) is `off` by default. When it is
1262 `on`, the forge refuses to start a child with no filesystem of its own,
1263 whether that child runs in a cell or in this forge's process. The refusal is
1264 a 503 with one plain line naming the child and saying why. Its pages and its
1265 apps answer with that line, and the boot skips the child and logs the same
1266 line. A child that is already running is not stopped.
1267
12431268 ## Runs
12441269
12451270 A run row is created `queued`, claimed with a compare-and-swap into
docs/MCP.md +1 −1
@@ -77,7 +77,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
7777 | `boards` | — | The forge's boards for person admins: slug, title, range, refresh and panels | 425 |
7878 | `branches` | `create`, `delete`, `branches`, `tags` | A repo's branches; `name`, `action` create (from `from`) or delete one, not the default | 407 |
7979 | `builds` | — | A repo's build pins (`refs/builds/`): label, commit, pinned by, note | 231 |
80 | `capacity` | — | Forge capacity and guards, live | 390 |
80 | `capacity` | — | Forge capacity and guards, live | 415 |
8181 | `claim_link` | `revoke`, `undo` | A child's claim link: `reveal` a new one, or `action` revoke/undo | 304 |
8282 | `close` | — | Close or reopen (`state` open) an issue or pull | 463 |
8383 | `cloud_account` | — | Change a cloud account by `verb` (admin) | 549 |
docs/features/child-filesystem.md added +48
@@ -0,0 +1,48 @@
1 # A child forge's own filesystem (#10856)
2
3 Everything a child forge writes is kept under `<data>/children/<body id>`. The
4 forge checks whether that directory is a **filesystem of its own**: a mount
5 whose mount point is exactly that path in `/proc/self/mountinfo`. The forge
6 mounts nothing itself. Making that filesystem is the platform's job (#10851).
7
8 ## What the line means
9
10 Each child's line on **`/forges`** and on the terminal's **Forges you sent**
11 screen (ctrl-k) ends with one of:
12
13 - **`own filesystem, N GiB`**: the child's data has a filesystem of its own.
14 N is `statfs` blocks × block size, in GiB (2^30 bytes), to one place.
15 Writing past that size fails with ENOSPC inside the child, and the parent's
16 free space stays the same. The child's cell binds that filesystem by its
17 real path.
18 - **`shared disk`**: the child's data sits on the disk this forge and its
19 other children share. Only the cell's disk watch stops it, and only after it
20 has written too much.
21
22 ## Requiring one: `children.ownFilesystem`
23
24 The setting is `off` by default. To turn it on, use one of:
25
26 - **Settings → Capacity**: set "Child forges need a filesystem of their own"
27 to `on`.
28 - **The terminal's Capacity screen**: press `e` on `children.ownFilesystem`.
29 - **The `capacity` MCP tool**:
30 `{ "key": "children.ownFilesystem", "value": "on", "reason": "…", "dryRun": false }`.
31 The value `""` puts back the default.
32
33 While it is on, the forge does not start a child that has no filesystem of its
34 own. This applies whether the child runs in a cell or in this forge's process.
35 The forge reads the setting on every start, and a child that is already
36 running is not stopped.
37
38 ## What the refusal says
39
40 The child's pages and its apps answer **503**, `text/plain`, with one line:
41
42 > kid was not started: its data is on the shared disk, not a filesystem of its own, and this forge requires one (children.ownFilesystem is on).
43
44 The boot skips that child and logs the same line as `child forge not started`.
45
46 Code: `src/spawn/disk_mount.ts` (reading the mounts), `startRefused` in
47 `src/spawn/host.ts`, and `launchSpecFor` in `src/spawn/cell.ts`. Tests:
48 `test/disk-mount.test.ts`.
src/actions/capacity.ts +2 −1
@@ -42,7 +42,7 @@ import { err, ok, type Result } from "../kernel/result.ts";
4242 /** What a refused change to a dial that follows Speed says, on every door. */
4343 export const DERIVED_WORDS = "The pace floor and the queue depth follow Speed: change Speed (crew.concurrency) and they move with it.";
4444
45 /** The six guards a caller may set by `key`: the MCP tool's enum and the terminal's `e`. */
45 /** The guards a caller may set by `key`: the MCP tool's enum and the terminal's `e`. */
4646 export const SETTABLE_CAPACITY = [
4747 "crew.concurrency",
4848 "checks.concurrency",
@@ -50,6 +50,7 @@ export const SETTABLE_CAPACITY = [
5050 "cells.egress",
5151 "guard_new_accounts",
5252 "signin.edge",
53 "children.ownFilesystem",
5354 ] as const satisfies readonly CapacityKey[];
5455
5556 export interface CapacityRow {
src/core/capacity.ts +14
@@ -26,6 +26,7 @@ export type CapacityKey =
2626 | "cells.egress"
2727 | "children.modelEgress"
2828 | "children.registryEgress"
29 | "children.ownFilesystem"
2930 | "guard_new_accounts"
3031 | "signin.edge"
3132 | "pace.floor";
@@ -121,6 +122,14 @@ export const CAPACITY: Record<CapacityKey, Definition> = {
121122 fallback: "off",
122123 help: "off keeps a child forge's cell from the package registries, so its checks cannot fill their install cache; npm also lets it reach registry.npmjs.org on 443; build opens the hosts ordinary app builds fetch from, on 443 alone: npm and yarn, PyPI, GitHub (git dependencies and release tarballs), crates.io and the Go module proxy. Nothing else opens, and the cell enforces this even when cells.egress is report. Takes effect on the next child cell started.",
123124 },
125 "children.ownFilesystem": {
126 label: "Child forges need a filesystem of their own",
127 // No environment default: refusing children is an admin's decision.
128 env: "",
129 kind: { type: "choice", values: ["off", "on"] },
130 fallback: "off",
131 help: "on refuses to start a child forge whose data directory (<data>/children/<id>) is not a filesystem of its own, so no child can fill the disk this forge and its other children share; the refusal names the child and says why. off starts it on the shared disk, held only by its cell's disk watch. Off by default, because a forge mounts nothing itself: turn it on once the platform mounts one per child. A child already running is not stopped. Read on every start.",
132 },
124133 "guard_new_accounts": {
125134 label: "New-account repository guard",
126135 env: "FORGE_GUARD_NEW_ACCOUNTS",
@@ -434,6 +443,11 @@ const MODEL_HOSTS: Readonly<Record<string, readonly string[]>> = {
434443 both: ["openrouter.ai:443", "api.anthropic.com:443"],
435444 };
436445
446 /** Whether a child forge must have a filesystem of its own to start (`children.ownFilesystem`); off by default. */
447 export function childNeedsOwnFilesystem(db: Database): boolean {
448 return capacityValue(db, "children.ownFilesystem") === "on";
449 }
450
437451 /** The package registry a child forge cell may reach (`children.registryEgress`); none by default. */
438452 export function childRegistryHosts(db: Database): readonly string[] {
439453 const value = capacityValue(db, "children.registryEgress");
src/http/mcp/tools/capacity.ts +1 −1
@@ -5,7 +5,7 @@ import { good, bad, type Tool } from "../shared.ts";
55
66 export const tool: Tool = {
77 name: "capacity",
8 doc: "`{ key?, value?, reason?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges, `egress.npmBytesLastHour` and `npmTopRepoLastHour`, and `selfSwap: { enabled, migrateReq, blocked }`: whether `FORGE_SELF_SWAP=1`, the kernel's `tcp_migrate_req` (`1`, `0`, `unreadable`), and why this forge cannot swap itself, null when it can; only the first six are settable by `key`); with `key` and `value` (`\"\"` restores the default) and `reason` (kept with who; required for `cells.egress`), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token",
8 doc: "`{ key?, value?, reason?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, whether a child forge needs its own filesystem, queue depth, child forges, `egress.npmBytesLastHour` and `npmTopRepoLastHour`, and `selfSwap: { enabled, migrateReq, blocked }`: whether `FORGE_SELF_SWAP=1`, the kernel's `tcp_migrate_req` (`1`, `0`, `unreadable`), and why this forge cannot swap itself, null when it can; only the first seven are settable by `key`); with `key` and `value` (`\"\"` restores the default) and `reason` (kept with who; required for `cells.egress`), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token",
99 description:
1010 "Forge capacity and guards, live. With `key` and `value` (\"\" restores default), plans.",
1111 inputSchema: {
src/http/server.ts +2
@@ -578,6 +578,7 @@ import {
578578 hostedChildOpen,
579579 resolveHostedChild,
580580 resolveHostedChildApp,
581 refusedResponse,
581582 socketToCell,
582583 wakeGate,
583584 watchHostedChildClose,
@@ -2571,6 +2572,7 @@ export async function serve(app: App): Promise<Servers> {
25712572 if (hit === "unknown")
25722573 return securityFloor(new Response("not found", { status: 404 }));
25732574 if (hit !== undefined) {
2575 if ("refused" in hit) return securityFloor(refusedResponse(hit.refused));
25742576 if ("cell" in hit) return fromCell(hit.cell);
25752577 notePeer(request, server.requestIP(request)?.address, trust);
25762578 return proxyApp(hit.runtime, request, hit.target, server);
src/spawn/cell.ts +38 −13
@@ -23,6 +23,7 @@ import { hostedPlacement, parentOrigin, childPublicUrl, type BodyStore, type Pla
2323 import { SIZE_IO, SIZES, type Size } from "./sizes.ts";
2424 import { CGROUP_DIR_ENV, SIZE_ENV } from "./usage.ts";
2525 import { childModelHosts, childRegistryHosts } from "../core/capacity.ts";
26 import { liveChildDisk, type ChildDisk } from "./disk_mount.ts";
2627
2728 export const CELL_PREFIX = "cell-";
2829 export const ownsBody = (bodyId: string): boolean => bodyId.startsWith(CELL_PREFIX);
@@ -116,10 +117,18 @@ export function cellSpecFor(input: {
116117 modelHosts?: readonly string[];
117118 /** The package registry an admin let children reach (`children.registryEgress`); none by default. */
118119 registryHosts?: readonly string[];
120 /**
121 * The child's writable space as found (`spawn/disk_mount.ts`). A filesystem
122 * of its own is named among the binds by its real path (bwrap binds
123 * recursively, and `openCell` binds a path once), so a data directory
124 * reached through a symlink still writes to it; with none, the cell is
125 * what it always was.
126 */
127 disk?: ChildDisk;
119128 }): CellSpec {
120129 return {
121130 dir: input.dir,
122 writable: [],
131 writable: input.disk?.own === true ? [input.disk.path] : [],
123132 home: join(input.dir, ".home"),
124133 // The parent's data and home are what a child must not see: the home holds
125134 // the parent crew's sessions and model login, and the child runs as the
@@ -224,6 +233,33 @@ async function answers(port: number, gone: Promise<unknown>): Promise<boolean> {
224233 return false;
225234 }
226235
236 /**
237 * The cell a child forge is launched in, from what the parent holds now: its
238 * egress settings and its data directory as found (`liveChildDisk`), so a
239 * child with a filesystem of its own has it bound. `mountinfo` is this
240 * process's unless a test passes its own.
241 */
242 export function launchSpecFor(
243 parent: App,
244 input: { readonly bodyId: string; readonly name: string; readonly size: Size },
245 at: { readonly port: number; readonly parentPort: number },
246 mountinfo?: string,
247 ): CellSpec {
248 return cellSpecFor({
249 dir: childDirOf(parent, input.bodyId),
250 parentDataDir: parent.config.dataDir,
251 parentUrl: parentOrigin(parent.config),
252 url: childPublicUrl(input.name, parentOrigin(parent.config)),
253 parentPort: at.parentPort,
254 size: input.size,
255 port: at.port,
256 label: `forge:${input.name}`,
257 modelHosts: childModelHosts(parent.db),
258 registryHosts: childRegistryHosts(parent.db),
259 disk: liveChildDisk(parent.config.dataDir, input.bodyId, mountinfo),
260 });
261 }
262
227263 function makeRegistry(parent: App): CellChildren {
228264 const held = new Map<string, Held>();
229265 const starting = new Map<string, Promise<CellChild | undefined>>();
@@ -248,18 +284,7 @@ function makeRegistry(parent: App): CellChildren {
248284 const url = childPublicUrl(input.name, parentOrigin(parent.config));
249285 const door = openBeatDoor(parent);
250286 const parentPort = door.port;
251 const spec = cellSpecFor({
252 dir,
253 parentDataDir: parent.config.dataDir,
254 parentUrl: parentOrigin(parent.config),
255 url,
256 parentPort,
257 size: input.size,
258 port,
259 label: `forge:${input.name}`,
260 modelHosts: childModelHosts(parent.db),
261 registryHosts: childRegistryHosts(parent.db),
262 });
287 const spec = launchSpecFor(parent, input, { port, parentPort });
263288 const heard = childLog((line) => parent.log.info("child forge", { name: input.name, line }));
264289 const log = (raw: string): void => {
265290 const line = plainLine(raw);
src/spawn/clients.ts +6
@@ -9,6 +9,7 @@
99 * list never waits on, or fails for, one forge that is down.
1010 */
1111
12 import { liveChildDisk, readMountinfo, type ChildDisk } from "./disk_mount.ts";
1213 import type { App } from "../app.ts";
1314 import { childDirOf, listChildren, statusTokenOf, type Child } from "../actions/spawn.ts";
1415 import { timingSafeEqual } from "node:crypto";
@@ -62,6 +63,8 @@ export interface ClientRow {
6263 readonly size: Size;
6364 /** What it uses against that size: its cell's reading, or "shared" when it is hosted here. */
6465 readonly use: ForgeUse;
66 /** Whether its data is a filesystem of its own, and that filesystem's size (`spawn/disk_mount.ts`). */
67 readonly disk: ChildDisk;
6568 /** When its last heartbeat reached this forge, ISO; null when none has. */
6669 readonly lastHeartbeatAt: string | null;
6770 /** The first step its person has not taken, read from the child's heartbeat. */
@@ -220,6 +223,8 @@ export async function clientRows(app: App): Promise<ClientRow[]> {
220223 }),
221224 );
222225 const beats = await Promise.all(children.map((c) => lastHeard(app, c.name)));
226 // One read of the mount table for every row.
227 const mountinfo = children.length === 0 ? "" : readMountinfo();
223228 return orderRows(
224229 children.map((c, i) => ({
225230 name: c.name,
@@ -238,6 +243,7 @@ export async function clientRows(app: App): Promise<ClientRow[]> {
238243 ...creditView(app, c.name),
239244 size: c.size,
240245 use: forgeUse(c, childDirOf(app, c.bodyId)),
246 disk: liveChildDisk(app.config.dataDir, c.bodyId, mountinfo),
241247 lastHeartbeatAt: beats[i] === undefined ? null : new Date(beats[i]!.receivedAt).toISOString(),
242248 link: states[i]?.link ?? null,
243249 nextStep: nextStepOf(states[i]?.claimed ?? null, beats[i]?.beat),
src/spawn/disk_mount.ts added +143
@@ -0,0 +1,143 @@
1 /**
2 * Whether a child forge's writable space, `<data>/children/<id>`, is a
3 * filesystem of its own: a mount whose mount point is exactly that
4 * directory, read from `/proc/self/mountinfo` (proc(5)). On one, writing
5 * past its size is ENOSPC inside the child and the parent's free space does
6 * not move; on the shared disk only the cell's disk watch stops it, after
7 * the fact (#10851). This module reads what is mounted; it mounts nothing.
8 *
9 * Plain data in, plain data out: the mountinfo is passed in as text and the
10 * size read through `stat`, so a test feeds both.
11 */
12
13 import { readFileSync, realpathSync, statfsSync } from "node:fs";
14 import { join } from "node:path";
15
16 /** One line of mountinfo, the fields this module reads. */
17 export interface Mount {
18 /** Field 5: where it is mounted, its octal escapes (`\040` is a space) undone. */
19 readonly mountPoint: string;
20 /** The first field after ` - `: ext4, xfs, overlay, tmpfs… */
21 readonly fsType: string;
22 }
23
24 /** A child's writable space as the forge finds it. */
25 export type ChildDisk =
26 | { readonly own: true; readonly path: string; readonly fsType: string; readonly bytes: number; readonly inodes: number }
27 | { readonly own: false; readonly path: string };
28
29 /** What `statfs(2)` says of a filesystem, the fields read here. */
30 export interface FsSize {
31 /** Block size in bytes (`f_bsize`). */
32 readonly bsize: number;
33 /** Total blocks (`f_blocks`). */
34 readonly blocks: number;
35 /** Total inodes (`f_files`). */
36 readonly files: number;
37 }
38
39 /** A body id is one path segment: letters, digits, `.`, `_`, `-`; a first character other than `.` rules out `.` and `..`. */
40 const BODY_ID = /^[A-Za-z0-9_-][A-Za-z0-9._-]{0,127}$/;
41 export const validBodyId = (id: string): boolean => BODY_ID.test(id);
42
43 /** Undo mountinfo's octal escapes: space, tab, newline and backslash are written `\ooo`. */
44 const unescape = (field: string): string => field.replace(/\\([0-7]{3})/g, (_, o: string) => String.fromCharCode(parseInt(o, 8)));
45
46 /**
47 * Every mount in a mountinfo text. A line reads
48 * `36 35 98:0 /mnt1 /mnt/parent rw,noatime master:1 - ext3 /dev/root rw`:
49 * mount point fifth, then optional fields up to a lone `-`, then the type. A line that does not have that shape is skipped.
50 */
51 export function parseMountinfo(text: string): Mount[] {
52 const mounts: Mount[] = [];
53 for (const line of text.split("\n")) {
54 const fields = line.split(" ");
55 const dash = fields.indexOf("-", 6);
56 if (fields.length < 7 || dash < 0) continue;
57 const mountPoint = fields[4];
58 const fsType = fields[dash + 1];
59 if (mountPoint === undefined || fsType === undefined || !mountPoint.startsWith("/")) continue;
60 mounts.push({ mountPoint: unescape(mountPoint), fsType });
61 }
62 return mounts;
63 }
64
65 /** The mount whose mount point is exactly `path` (the last one, as the kernel stacks them); undefined when none. */
66 export function mountAt(mounts: readonly Mount[], path: string): Mount | undefined {
67 const want = path.length > 1 ? path.replace(/\/+$/, "") : path;
68 let found: Mount | undefined;
69 for (const m of mounts) if (m.mountPoint === want) found = m;
70 return found;
71 }
72
73 /**
74 * The child's writable space: its own filesystem with its size in bytes
75 * (blocks × block size) and inodes, or the shared disk. `path` is where the
76 * mount must sit, already resolved (`childDiskPath`). A filesystem whose size
77 * cannot be read is not counted as the child's own.
78 */
79 export function childDisk(input: { path: string; mountinfo: string; size: (path: string) => FsSize | undefined }): ChildDisk {
80 const mount = mountAt(parseMountinfo(input.mountinfo), input.path);
81 if (mount === undefined) return { own: false, path: input.path };
82 const fs = input.size(input.path);
83 if (fs === undefined || !(fs.bsize > 0) || !(fs.blocks > 0)) return { own: false, path: input.path };
84 return { own: true, path: input.path, fsType: mount.fsType, bytes: fs.blocks * fs.bsize, inodes: fs.files };
85 }
86
87 /** `<data>/children/<id>`, symlinks resolved (mountinfo names real paths); the joined path when it does not exist yet. */
88 export function childDiskPath(dataDir: string, bodyId: string): string | undefined {
89 if (!validBodyId(bodyId)) return undefined;
90 const path = join(dataDir, "children", bodyId);
91 try {
92 return realpathSync(path);
93 } catch {
94 return path;
95 }
96 }
97
98 /** This process's mountinfo; "" where there is none (not Linux), which reads as no mount at all. */
99 export function readMountinfo(): string {
100 try {
101 return readFileSync("/proc/self/mountinfo", "utf8");
102 } catch {
103 return "";
104 }
105 }
106
107 /** `statfs` of `path`; undefined when it cannot be read. */
108 export function statSize(path: string): FsSize | undefined {
109 try {
110 const s = statfsSync(path);
111 return { bsize: Number(s.bsize), blocks: Number(s.blocks), files: Number(s.files) };
112 } catch {
113 return undefined;
114 }
115 }
116
117 /** The live answer for one child: this process's mounts, the real `statfs`. */
118 export function liveChildDisk(dataDir: string, bodyId: string, mountinfo: string = readMountinfo()): ChildDisk {
119 const path = childDiskPath(dataDir, bodyId);
120 return path === undefined ? { own: false, path: join(dataDir, "children") } : childDisk({ path, mountinfo, size: statSize });
121 }
122
123 const GIB = 1024 ** 3;
124
125 /** GiB to one place, the `.0` dropped: 10 GiB is 10 737 418 240 bytes. */
126 export function gib(bytes: number): string {
127 const n = Math.round((bytes / GIB) * 10) / 10;
128 return Number.isInteger(n) ? String(n) : n.toFixed(1);
129 }
130
131 /** The words a child's line carries: `own filesystem, N GiB` or `shared disk`. */
132 export function diskWords(disk: ChildDisk): string {
133 return disk.own ? `own filesystem, ${gib(disk.bytes)} GiB` : "shared disk";
134 }
135
136 /**
137 * Why a child may not start, in one plain line; undefined when it may. Only
138 * when the forge requires a filesystem of its own (`children.ownFilesystem`).
139 */
140 export function startRefusal(name: string, disk: ChildDisk, required: boolean): string | undefined {
141 if (!required || disk.own) return undefined;
142 return `${name} was not started: its data is on the shared disk, not a filesystem of its own, and this forge requires one (children.ownFilesystem is on).`;
143 }
src/spawn/host.ts +37 −1
@@ -25,6 +25,8 @@ import {
2525 import { DEFAULT_SIZE, isSize, type Size } from "./sizes.ts";
2626 import { cellChildren, forward, forwardSocket, type CellChild } from "./cell.ts";
2727 import { cellsAvailable } from "../apps/contain.ts";
28 import { childNeedsOwnFilesystem } from "../core/capacity.ts";
29 import { liveChildDisk, startRefusal } from "./disk_mount.ts";
2830 import { limitValue } from "../settings/limits.ts";
2931 import { awakeRefusal, idleAfterMs, WakeRate } from "./sleep.ts";
3032 import { wakingResponse } from "../web/pages/waking.ts";
@@ -252,6 +254,27 @@ function keepSweeping(parent: App): void {
252254 const carriesCredentials = (parent: App, bodyId: string, request: Request): boolean =>
253255 vouched(childDirOf(parent, bodyId), request, caches.get(parent)?.get(bodyId)?.db);
254256
257 /**
258 * Why the child `ledger` names may not start now, in one plain line;
259 * undefined when it may. Only `children.ownFilesystem` refuses: a child whose
260 * data is not a filesystem of its own. The setting is read first, so with it
261 * off a start costs no mountinfo read.
262 */
263 export function startRefused(parent: App, ledger: { readonly name: string; readonly bodyId: string }): string | undefined {
264 if (!childNeedsOwnFilesystem(parent.db)) return undefined;
265 const why = startRefusal(ledger.name, liveChildDisk(parent.config.dataDir, ledger.bodyId), true);
266 if (why !== undefined) parent.log.warn("child forge not started", { name: ledger.name, why });
267 return why;
268 }
269
270 /** The answer to a request for a child that may not start: the line, as text, 503. */
271 export function refusedResponse(why: string): Response {
272 return new Response(`${why}\n`, {
273 status: 503,
274 headers: { "content-type": "text/plain; charset=utf-8", "cache-control": "no-store" },
275 });
276 }
277
255278 /**
256279 * Before a child's Host is answered: a child already awake is marked busy (by a
257280 * request whose session or token the child verifies, so a stranger cannot keep it awake); one
@@ -274,6 +297,9 @@ export function wakeGate(parent: App, request: Request): Response | undefined {
274297 }
275298 return undefined;
276299 }
300 // Asleep, so answering would start it: not where it has no filesystem of its own and one is required.
301 const refused = startRefused(parent, ledger);
302 if (refused !== undefined) return refusedResponse(refused);
277303 // A wake already granted (a cell takes a moment to answer) is one wake, not one per request.
278304 let reserved = waking.get(parent);
279305 if (!reserved) waking.set(parent, (reserved = new Map()));
@@ -444,6 +470,7 @@ function openOnce(parent: App, ledger: ChildLedger): Promise<App> | App {
444470 export async function bootHostedChild(parent: App, name: string): Promise<void> {
445471 let ledger = readLedgerSync(parent, name);
446472 if (ledger === undefined || !ledger.done) return;
473 if (!isAwakeBody(parent, ledger.bodyId) && startRefused(parent, ledger) !== undefined) return;
447474 if (ledger.placement === "cell") {
448475 if ((await settleCell(parent, ledger)) !== undefined) return;
449476 ledger = { ...ledger, placement: "hosted" };
@@ -484,6 +511,11 @@ export interface ChildAppHit {
484511 readonly target: AppTarget;
485512 }
486513
514 /** A child app whose forge may not start (`startRefused`): the line it is answered with. */
515 export interface RefusedHit {
516 readonly refused: string;
517 }
518
487519 /** A child app in a child's own cell: its runtime is in that process, so the request is carried there. */
488520 export interface CellAppHit {
489521 readonly cell: string;
@@ -497,7 +529,7 @@ export interface CellAppHit {
497529 export async function resolveHostedChildApp(
498530 parent: App,
499531 request: Request,
500 ): Promise<ChildAppHit | CellAppHit | "unknown" | undefined> {
532 ): Promise<ChildAppHit | CellAppHit | RefusedHit | "unknown" | undefined> {
501533 const host = request.headers.get("host");
502534 if (host === null) return undefined;
503535 const label = parseHostedChildAppLabel(host, childHost(parent));
@@ -513,6 +545,10 @@ export async function resolveHostedChildApp(
513545 if (!CHILD_NAME.test(name)) continue;
514546 const ledger = readLedgerSync(parent, name);
515547 if (ledger === undefined || !ledger.done) continue;
548 if (!isAwakeBody(parent, ledger.bodyId)) {
549 const refused = startRefused(parent, ledger);
550 if (refused !== undefined) return { refused };
551 }
516552 if (ledger.placement === "cell") return { cell: name };
517553 const child = await openOnce(parent, ledger);
518554 const runtime = appsRuntime(child);
src/tui/screens/forges.ts +2 −1
@@ -13,6 +13,7 @@ import { card } from "../ui/widgets.ts";
1313 import { drawFacts, runAsync } from "./common.ts";
1414 import type { PaletteEntry } from "./entries.ts";
1515 import { clientRows } from "../../spawn/clients.ts";
16 import { diskWords } from "../../spawn/disk_mount.ts";
1617 import { changeLink, removeForge, stopForge } from "../../spawn/manage.ts";
1718 import { undoable, type LinkInfo } from "../../spawn/claim.ts";
1819 import { confirmScreen, pickerScreen } from "./common.ts";
@@ -23,7 +24,7 @@ export function forgeLines(rows: readonly ClientRow[]): [string, string][] {
2324 if (rows.length === 0) return [["", "You have not sent a forge yet"]];
2425 return rows.flatMap((r): [string, string][] => [[
2526 r.name,
26 [r.made ? (r.awake === true ? "awake" : "asleep") : null, r.attention === null ? null : ATTENTION_WORDS[r.attention], r.for === null ? null : `for ${r.for}`, CLAIM_WORDS[claimState(r)], linkWords(r.link), runningWords(r), r.nextStep.words, spendWords(r)?.words ?? null, creditWords(r), useLine(r.size, r.use), r.lastHeartbeatAt === null ? "no heartbeat yet" : `heartbeat ${timeAgo(r.lastHeartbeatAt)}`]
27 [r.made ? (r.awake === true ? "awake" : "asleep") : null, r.attention === null ? null : ATTENTION_WORDS[r.attention], r.for === null ? null : `for ${r.for}`, CLAIM_WORDS[claimState(r)], linkWords(r.link), runningWords(r), r.nextStep.words, spendWords(r)?.words ?? null, creditWords(r), useLine(r.size, r.use), r.made ? diskWords(r.disk) : null, r.lastHeartbeatAt === null ? "no heartbeat yet" : `heartbeat ${timeAgo(r.lastHeartbeatAt)}`]
2728 .filter((m): m is string => m !== null)
2829 .join(" · "),
2930 ], ...addressLines(r)]);
src/web/pages/forges.ts +3 −1
@@ -18,6 +18,7 @@ import { readForm } from "../form.ts";
1818 import { badge, btn, row, time } from "../ui.ts";
1919 import { noRoom } from "../../spawn/door.ts";
2020 import { sizeWords, useWords } from "../../spawn/usage.ts";
21 import { diskWords } from "../../spawn/disk_mount.ts";
2122 import { listChildren } from "../../actions/spawn.ts";
2223 import { removeForge, stopForge, type Managed } from "../../spawn/manage.ts";
2324 import { claimLinkBody } from "./new-forge.ts";
@@ -42,7 +43,7 @@ function spendNote(r: ClientRow): Html | null {
4243 : html`<span data-spend="ok">${s.words}</span>`;
4344 }
4445
45 function forgeRow(r: ClientRow, now: number): Html {
46 export function forgeRow(r: ClientRow, now: number): Html {
4647 const state = claimState(r);
4748 const meta = [
4849 r.for === null ? null : html`for ${r.for}`,
@@ -50,6 +51,7 @@ function forgeRow(r: ClientRow, now: number): Html {
5051 state === "unknown" ? html`it did not answer just now` : null,
5152 r.made ? badge(r.awake === true ? "awake" : "asleep", r.awake === true ? { tone: "success" } : { variant: "outline" }) : null,
5253 r.made ? html`${sizeWords(r.size)}` : null,
54 r.made ? html`<span data-disk="${r.disk.own ? "own" : "shared"}">${diskWords(r.disk)}</span>` : null,
5355 runningNote(r),
5456 spendNote(r),
5557 creditWords(r) === null ? null : html`<span data-credit>${creditWords(r)}</span>`,
test/disk-mount.test.ts added +252
@@ -0,0 +1,252 @@
1 /**
2 * #10856: the forge knows whether a child forge's writable space is a
3 * filesystem of its own, binds it into the child's cell, says so on the
4 * child's line, and refuses to start a child without one when an admin asks.
5 */
6
7 import { afterAll, describe, expect, test } from "bun:test";
8 import { mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { createApp, type App } from "../src/app.ts";
12 import { createUser } from "../src/core/users.ts";
13 import { capacityValue, childNeedsOwnFilesystem, setCapacity } from "../src/core/capacity.ts";
14 import { childDisk, diskWords, gib, mountAt, parseMountinfo, startRefusal, validBodyId, type FsSize } from "../src/spawn/disk_mount.ts";
15 import { cellSpecFor, launchSpecFor } from "../src/spawn/cell.ts";
16 import { bootHostedChild, childAwake, resolveHostedChildApp, sleepChild, startRefused, wakeGate } from "../src/spawn/host.ts";
17 import { childDirOf, spawnForge } from "../src/actions/spawn.ts";
18 import { hostedPlacement } from "../src/spawn/placement.ts";
19 import { forgeRow } from "../src/web/pages/forges.ts";
20 import { forgeLines } from "../src/tui/screens/forges.ts";
21 import { tool as capacityTool } from "../src/http/mcp/tools/capacity.ts";
22 import type { ClientRow } from "../src/spawn/clients.ts";
23 import { quietLogs } from "./helpers/globals.ts";
24
25 quietLogs();
26
27 const tmp = mkdtempSync(join(tmpdir(), "disk-mount-"));
28 afterAll(() => rmSync(tmp, { recursive: true, force: true }));
29
30 // Two real lines in proc(5)'s shape: the root, and a child's own ext4 at /d/children/c1.
31 const ROOT = "22 1 259:2 / / rw,relatime shared:1 - ext4 /dev/nvme0n1p2 rw";
32 const OWN = "731 22 7:3 / /d/children/c1 rw,nosuid,nodev,relatime shared:401 - ext4 /dev/loop3 rw";
33 // 10 GiB of 4 KiB blocks: 2 621 440 × 4096 = 10 737 418 240 bytes; 655 360 inodes.
34 const TEN_GIB: FsSize = { bsize: 4096, blocks: 2_621_440, files: 655_360 };
35 const sized = (fs: FsSize | undefined) => () => fs;
36
37 describe("a child's writable space, read from mountinfo", () => {
38 test("a mount exactly at <data>/children/<id> is its own filesystem, with its size in bytes and inodes", () => {
39 const disk = childDisk({ path: "/d/children/c1", mountinfo: `${ROOT}\n${OWN}\n`, size: sized(TEN_GIB) });
40 expect(disk).toEqual({ own: true, path: "/d/children/c1", fsType: "ext4", bytes: 10_737_418_240, inodes: 655_360 });
41 expect(diskWords(disk)).toBe("own filesystem, 10 GiB");
42 });
43
44 test("without the mount it is the shared disk", () => {
45 const disk = childDisk({ path: "/d/children/c1", mountinfo: `${ROOT}\n`, size: sized(TEN_GIB) });
46 expect(disk).toEqual({ own: false, path: "/d/children/c1" });
47 expect(diskWords(disk)).toBe("shared disk");
48 });
49
50 test("a mount above it, beside it or below it is not its own", () => {
51 const near = [
52 "40 22 7:4 / /d/children rw - ext4 /dev/loop4 rw",
53 "41 22 7:5 / /d/children/c10 rw - ext4 /dev/loop5 rw",
54 "42 22 7:6 / /d/children/c1/sub rw - ext4 /dev/loop6 rw",
55 ].join("\n");
56 expect(childDisk({ path: "/d/children/c1", mountinfo: `${ROOT}\n${near}`, size: sized(TEN_GIB) }).own).toBe(false);
57 });
58
59 test("a mount whose size cannot be read is not counted as its own", () => {
60 expect(childDisk({ path: "/d/children/c1", mountinfo: OWN, size: sized(undefined) }).own).toBe(false);
61 expect(childDisk({ path: "/d/children/c1", mountinfo: OWN, size: sized({ bsize: 4096, blocks: 0, files: 0 }) }).own).toBe(false);
62 });
63
64 test("octal escapes in a mount point are undone, optional fields skipped, junk lines ignored", () => {
65 const text = "50 22 7:7 / /d/my\\040data/children/c1 rw shared:9 master:2 - xfs /dev/loop7 rw\nnot a mount line\n\n";
66 const mounts = parseMountinfo(text);
67 expect(mounts).toEqual([{ mountPoint: "/d/my data/children/c1", fsType: "xfs" }]);
68 expect(mountAt(mounts, "/d/my data/children/c1/")?.fsType).toBe("xfs");
69 });
70
71 test("GiB to one place: 1.5 GiB is 1 610 612 736 bytes", () => {
72 expect(gib(1_610_612_736)).toBe("1.5");
73 expect(gib(10 * 1024 ** 3)).toBe("10");
74 });
75
76 test("a body id is one path segment", () => {
77 expect(validBodyId("cell-01abc")).toBe(true);
78 for (const bad of ["..", ".", "a/b", "", "../x", "a\nb"]) expect(validBodyId(bad)).toBe(false);
79 });
80 });
81
82 describe("the child's cell", () => {
83 const base = { dir: "/d/children/c1", parentDataDir: "/d", parentHome: "/home/f", parentUrl: "https://f.test", size: "small", port: 4100, label: "forge:kid" } as const;
84
85 test("binds its own filesystem when it has one", () => {
86 const spec = cellSpecFor({ ...base, disk: { own: true, path: "/mnt/kids/c1", fsType: "ext4", bytes: 1, inodes: 1 } });
87 expect(spec.writable).toEqual(["/mnt/kids/c1"]);
88 });
89
90 test("with no mount is exactly what it was", () => {
91 expect(cellSpecFor({ ...base, disk: { own: false, path: "/d/children/c1" } })).toEqual(cellSpecFor(base));
92 expect(cellSpecFor(base).writable).toEqual([]);
93 });
94 });
95
96 describe("children.ownFilesystem", () => {
97 const made: App[] = [];
98 afterAll(() => {
99 for (const app of made) app.close();
100 });
101 const parentWith = (name: string): App => {
102 const dataDir = join(tmp, name);
103 const app = createApp({ dataDir, url: "https://forge.test" });
104 made.push(app);
105 mkdirSync(join(dataDir, "spawn"), { recursive: true });
106 mkdirSync(join(dataDir, "children", "body-kid"), { recursive: true });
107 writeFileSync(join(dataDir, "spawn", "kid.json"), JSON.stringify({ name: "kid", bodyId: "body-kid", done: true, placement: "hosted" }));
108 return app;
109 };
110
111 test("the line names the child and says why, and is nothing when one is not required or there is one", () => {
112 const own = { own: true, path: "/d/children/c1", fsType: "ext4", bytes: 1, inodes: 1 } as const;
113 const shared = { own: false, path: "/d/children/c1" } as const;
114 const line = startRefusal("kid", shared, true)!;
115 expect(line).toContain("kid was not started");
116 expect(line).toContain("not a filesystem of its own");
117 expect(line.includes("\n")).toBe(false);
118 expect(startRefusal("kid", shared, false)).toBeUndefined();
119 expect(startRefusal("kid", own, true)).toBeUndefined();
120 });
121
122 test("off by default: a child on the shared disk starts", () => {
123 const parent = parentWith("off");
124 expect(capacityValue(parent.db, "children.ownFilesystem")).toBe("off");
125 expect(startRefused(parent, { name: "kid", bodyId: "body-kid" })).toBeUndefined();
126 expect(wakeGate(parent, new Request("https://forge.test/", { headers: { host: "kid--forge.test" } }))).toBeUndefined();
127 });
128
129 test("on: a child on the shared disk is refused, its page and its apps answer the line", async () => {
130 const parent = parentWith("on");
131 setCapacity(parent.db, "children.ownFilesystem", "on", "system");
132 expect(childNeedsOwnFilesystem(parent.db)).toBe(true);
133 const page = wakeGate(parent, new Request("https://forge.test/", { headers: { host: "kid--forge.test" } }));
134 expect(page?.status).toBe(503);
135 expect(page?.headers.get("content-type")).toContain("text/plain");
136 const said = await page!.text();
137 expect(said).toContain("kid was not started");
138 expect(said.trim().split("\n")).toHaveLength(1);
139 const app = await resolveHostedChildApp(parent, new Request("https://forge.test/", { headers: { host: "shop--kid--forge.test" } }));
140 expect(app !== undefined && typeof app === "object" && "refused" in app ? app.refused : "").toContain("kid was not started");
141 });
142
143 test("the capacity MCP tool reads and writes it", async () => {
144 const parent = parentWith("mcp");
145 const made = await createUser(parent.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", isAdmin: true });
146 if (!made.ok) throw new Error("no admin");
147 const admin = made.value;
148 const ctx = { app: parent, auth: { user: admin, scope: "admin" }, request: new Request("https://forge.test/mcp") } as never;
149 const listed = await capacityTool.run(ctx, admin, {});
150 expect(JSON.stringify(listed)).toContain("children.ownFilesystem");
151 // Read, never matched with an asymmetric matcher: bun's toMatchObject writes the matcher into the
152 // object it is given, and this schema is the tool's own, shared with test/mcp-table.test.ts.
153 const keys = (capacityTool.inputSchema as { properties: { key: { enum: readonly string[] } } }).properties.key.enum;
154 expect([...keys]).toContain("children.ownFilesystem");
155 const set = await capacityTool.run(ctx, admin, { key: "children.ownFilesystem", value: "on", dryRun: false });
156 expect(set.ok).toBe(true);
157 expect(childNeedsOwnFilesystem(parent.db)).toBe(true);
158 await capacityTool.run(ctx, admin, { key: "children.ownFilesystem", value: "", dryRun: false });
159 expect(childNeedsOwnFilesystem(parent.db)).toBe(false);
160 });
161 });
162
163 describe("the child's line", () => {
164 const row = (over: Partial<ClientRow>): ClientRow => ({
165 name: "kid",
166 url: "https://kid--forge.test",
167 for: null,
168 apps: [],
169 sentAt: null,
170 made: true,
171 claimed: true,
172 lastActivityAt: null,
173 running: null,
174 reach: null,
175 link: null,
176 attention: null,
177 spendTodayUsd: null,
178 ceilingUsd: null,
179 size: "small",
180 use: { kind: "shared" },
181 disk: { own: false, path: "/d/children/c1" },
182 lastHeartbeatAt: null,
183 nextStep: { key: "unreported", words: "has not reported yet", lastTellAt: null },
184 ...over,
185 });
186
187 test("reads own filesystem, N GiB or shared disk on /forges and in the terminal", () => {
188 const own = row({ disk: { own: true, path: "/d/children/c1", fsType: "ext4", bytes: 10 * 1024 ** 3, inodes: 1 } });
189 expect(String(forgeRow(own, Date.now()))).toContain(`<span data-disk="own">own filesystem, 10 GiB</span>`);
190 expect(String(forgeRow(row({}), Date.now()))).toContain(`<span data-disk="shared">shared disk</span>`);
191 expect(forgeLines([own])[0]![1]).toContain("own filesystem, 10 GiB");
192 expect(forgeLines([row({})])[0]![1]).toContain("shared disk");
193 });
194
195 test("is escaped", () => {
196 const html = String(forgeRow(row({ name: `<img src=x onerror=alert(1)>`, for: `"><script>` }), Date.now()));
197 expect(html).not.toContain("<img src=x");
198 expect(html).not.toContain("<script>");
199 expect(html).toContain("shared disk");
200 });
201 });
202
203 describe("what the forge launches and boots", () => {
204 const made: App[] = [];
205 afterAll(() => {
206 for (const app of made) app.close();
207 });
208
209 const spawned = async (name: string): Promise<App> => {
210 const parent = createApp({ dataDir: join(tmp, name), url: "https://forge.test", port: 0 });
211 made.push(parent);
212 const admin = await createUser(parent.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", isAdmin: true });
213 if (!admin.ok) throw new Error("user");
214 const out = await spawnForge(
215 parent,
216 admin.value,
217 { name: "kid", org: "kid", apps: [], carryBriefs: false, carrySubscriptions: false },
218 { catalog: [], briefs: [], subscriptions: [] },
219 {
220 placement: hostedPlacement({ host: "forge.test" }),
221 request: new Request("https://forge.test/spawn", { method: "POST", headers: { origin: "https://forge.test" } }),
222 url: new URL("https://forge.test/spawn"),
223 },
224 );
225 if (!out.ok) throw new Error(JSON.stringify(out.error));
226 await sleepChild(parent, "kid");
227 return parent;
228 };
229
230 test("the registry's cell spec binds the child's own filesystem, and binds nothing on the shared disk", async () => {
231 const parent = createApp({ dataDir: join(tmp, "launch"), url: "https://forge.test", port: 0 });
232 made.push(parent);
233 mkdirSync(childDirOf(parent, "cell-kid1"), { recursive: true });
234 const real = realpathSync(childDirOf(parent, "cell-kid1"));
235 const input = { bodyId: "cell-kid1", name: "kid", size: "small" } as const;
236 const at = { port: 4100, parentPort: 4101 };
237 const own = launchSpecFor(parent, input, at, `${ROOT}\n731 22 7:3 / ${real} rw - ext4 /dev/loop3 rw\n`);
238 expect(own.writable).toEqual([real]);
239 expect(launchSpecFor(parent, input, at, `${ROOT}\n`).writable).toEqual([]);
240 });
241
242 test("the boot does not open a child on the shared disk when one of its own is required, and opens it when not", async () => {
243 const parent = await spawned("boot");
244 expect(childAwake(parent, "kid")).toBe(false);
245 setCapacity(parent.db, "children.ownFilesystem", "on", "system");
246 await bootHostedChild(parent, "kid");
247 expect(childAwake(parent, "kid")).toBe(false);
248 setCapacity(parent.db, "children.ownFilesystem", "off", "system");
249 await bootHostedChild(parent, "kid");
250 expect(childAwake(parent, "kid")).toBe(true);
251 });
252 });
test/forges-attention.test.ts +1
@@ -24,6 +24,7 @@ const row = (name: string, o: Partial<ClientRow> = {}): Omit<ClientRow, "attenti
2424 link: null,
2525 size: "small",
2626 use: { kind: "shared" },
27 disk: { own: false, path: "/d/children/c" },
2728 lastHeartbeatAt: null,
2829 nextStep: { key: "unreported", words: "has not reported yet", lastTellAt: null },
2930 spendTodayUsd: null,
test/spawn-child-runs-apps.test.ts +1
@@ -44,6 +44,7 @@ describe("a hosted child's apps run in the parent", () => {
4444 if (hit === undefined) return new Response("not an app host", { status: 418 });
4545 if (hit === "unknown") return new Response("not found", { status: 404 });
4646 if ("cell" in hit) throw new Error("this child is hosted");
47 if ("refused" in hit) return new Response(hit.refused, { status: 503 });
4748 // The server is only for a websocket upgrade, which these requests are not.
4849 return (await proxyApp(hit.runtime, request, hit.target, {} as never))!;
4950 };
test/tui-forges.test.ts +4 −3
@@ -32,6 +32,7 @@ const row = (over: Partial<ClientRow>): ClientRow => ({
3232 ceilingUsd: null,
3333 size: "small",
3434 use: { kind: "shared" },
35 disk: { own: false, path: "/d/children/c" },
3536 lastHeartbeatAt: null,
3637 nextStep: { key: "unreported", words: "has not reported yet", lastTellAt: null },
3738 ...over,
@@ -172,15 +173,15 @@ describe("forges you sent in the terminal", () => {
172173 const up = { total: 3, running: 3, notRunning: [] };
173174 const down = { total: 2, running: 1, notRunning: [{ slug: "notes", why: "build failed" }] };
174175 expect(forgeLines([row({ running: up })])).toEqual([
175 ["maria", "asleep · for Maria's bakery · Not claimed yet · 3 apps running · has not reported yet · small · shared · no heartbeat yet"],
176 ["maria", "asleep · for Maria's bakery · Not claimed yet · 3 apps running · has not reported yet · small · shared · shared disk · no heartbeat yet"],
176177 ]);
177178 expect(forgeLines([row({ claimed: true, running: down, for: null })])).toEqual([
178 ["maria", "asleep · Claimed · 1 app running · notes: build failed · has not reported yet · small · shared · no heartbeat yet"],
179 ["maria", "asleep · Claimed · 1 app running · notes: build failed · has not reported yet · small · shared · shared disk · no heartbeat yet"],
179180 ]);
180181 expect(forgeLines([row({ claimed: null })])[0]![1]).toContain("Unknown");
181182 expect(forgeLines([row({ made: false })])[0]![1]).toContain("Still being made");
182183 expect(forgeLines([row({ attention: "down", claimed: true, for: null })])).toEqual([
183 ["maria", "asleep · App down · Claimed · has not reported yet · small · shared · no heartbeat yet"],
184 ["maria", "asleep · App down · Claimed · has not reported yet · small · shared · shared disk · no heartbeat yet"],
184185 ]);
185186 expect(forgeLines([row({})])[0]![1]).not.toMatch(/Not reporting|App down|Unclaimed/);
186187 expect(forgeLines([row({ nextStep: { key: "tell", words: "no tell yet", lastTellAt: null } })])[0]![1]).toContain("no tell yet");

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs