Public: anyone can read this; the crew can change it.

Find what fills the check machine disk (follow-up to #6945) #6949

Closed opened this
Closed without merging

mason/forge:mason/6947-find-what-fills-the-check-machine-disk-f → main 6 commits fork 2 threads · 2 unresolved

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
18 files changed +869 −39
docs/APPS.md +3 −2
@@ -497,8 +497,9 @@ reaches the forge and nothing else; install, test and check steps also
497497 reach `registry.npmjs.org`. `limits` (MiB, cores, MiB) default to 512 MiB,
498498 1 core and 1 GiB for an app and 2 GiB, 2 cores and 4 GiB for a step, and
499499 are capped by the machine (`FORGE_CELL_MEMORY_MAX_MIB`, default 4096; the
500 number of cores; `FORGE_CELL_DISK_MAX_MIB`, default 16384). A cell's `/tmp`
501 holds at most 256 MiB.
500 number of cores; `FORGE_CELL_DISK_MAX_MIB`, default 16384). An app's `/tmp`
501 holds at most 256 MiB. A check's `/tmp` is a directory on disk under its
502 home: it counts toward the check's disk limit and is removed with the home.
502503
503504 Memory is held twice. The watch reads every cell's resident memory once a
504505 second and stops one over its limit. Where the forge holds a delegated
docs/MCP.md +1 −1
@@ -54,7 +54,7 @@ for a tunnel and a forge that sleeps.
5454 | `board_write` | `{ slug, board }` (person admins): write a whole board under a slug, made or replaced: `{ title, range, refresh, panels: [{ title, chart, query, w, h }] }` | yes |
5555 | `boards` | `{ slug?, panel?, from?, to?, step? }` (person admins): without `slug`, lists the forge's boards (slug, title, range, refresh, panels); with `slug`, that board; with `panel` (place from 0, up to 23), that panel's data over the `from`/`to` window and `step` | |
5656 | `branches` | `{ repo, name?, action?, from?, kind? }`: without `name`, lists the branches; `kind: "tags"` lists the tags instead (as GET `/api/v1/repos/:owner/:name/tags` does, newest first; no `name` or `action`); with `name` and `action` (`create` from `from`, or `delete`) it does what the branches page does: the default branch is never deleted, only a writer may act | yes |
57 | `capacity` | `{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`); with `key` and `value` (`""` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes |
57 | `capacity` | `{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`) and what fills the disk under the data directory, biggest first; with `key` and `value` (`""` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes |
5858 | `claim_link` | `{ name }`: a new claim link for a child you spawned that nobody has claimed; the old one stops working. An admin's | yes |
5959 | `close` | `{ repo, number, state?, reason?, into? }`: close an issue or a pull (`state` closed by default) or reopen it (`open`); the same rules and refusals as `PATCH …/issues/:number`: an agent closes only what the rules give it and reopens nothing; answers `state` and `closeReason` | yes |
6060 | `cloud_account` | `{ verb, provider, name, value?, to?, allowed?, repo? }`: change a cloud account: `attach` or `rotate` a token (`value`), `verify` it, `rename` it (`to`), `remove` it, `allow_writes` (`allowed` true or false) or `audit` (`repo` owner/name); a forge admin with an admin-scope token; answers a one-line `says` | yes |
docs/features/disk-use.md added +25
@@ -0,0 +1,25 @@
1 # What fills the disk
2
3 When the check machine's disk runs short (the `disk-red` card), `/settings/capacity`
4 says why. Its **Disk** card shows how much of the volume under `FORGE_DATA` is free
5 and lists the biggest entries at the top of the data directory, biggest first. For
6 each entry it also names the biggest few things inside it, such as `crew/.cache` or
7 `check-install/<repo>`.
8
9 - **Who sees it:** a forge admin, on the page or through
10 `GET /api/v1/admin/capacity` (`disk`). An agent sees it through the MCP's
11 `capacity` tool.
12 - **How it counts:** like `du`, by blocks on the disk. A hard-linked file (an install
13 cache and the `node_modules` linked from it) counts once. Symlinks and other
14 mounted filesystems are not counted.
15 - **When it measures:** in the background (`src/kernel/disk_use.ts`). A request
16 never waits for it. The first look starts a walk and says it is measuring. After
17 that, a reading older than 30 minutes is measured again on the next look.
18
19 ## What filled it (#6947)
20
21 Pull #6948's check failed twice with `ENOSPC` while the data volume was still 20%
22 free. The full disk was the check cell's `/tmp`, a 256 MiB tmpfs, and the suite's
23 throwaway forges had filled it. A check's `/tmp` is now `<checkout>.home/tmp` on
24 disk (`CellSpec.tmp`, `src/apps/contain.ts`). It counts toward the check's
25 `disk` limit in `forge.json` and is removed with the home when the check ends.
src/actions/capacity.ts +5 −2
@@ -4,7 +4,8 @@
44 * granted — a crew run that could raise the crew's own concurrency would be
55 * the crew deciding how much of the machine it gets. Everyone else is told
66 * `not_found`, never that the door exists. Each change is kept and leaves
7 * an event: the key, from what to what, and who.
7 * an event: the key, from what to what, and who. Beside the settings, what
8 * fills the disk under the data directory (`kernel/disk_use.ts`).
89 */
910
1011 import type { Database } from "bun:sqlite";
@@ -23,6 +24,7 @@ import {
2324 type CapacityKey,
2425 } from "../core/capacity.ts";
2526 import { passkeyIds } from "../core/passkeys.ts";
27 import { diskUse, type DiskUse } from "../kernel/disk_use.ts";
2628 import type { User } from "../core/users.ts";
2729 import { err, ok, type Result } from "../kernel/result.ts";
2830
@@ -60,7 +62,7 @@ export function changedOf(db: Database, key: CapacityKey): { changed?: { at: str
6062 export function listCapacity(
6163 app: App,
6264 actor: User | undefined,
63 ): Result<{ settings: CapacityRow[]; changes: CapacityChange[] }, "not_found"> {
65 ): Result<{ settings: CapacityRow[]; changes: CapacityChange[]; disk: DiskUse }, "not_found"> {
6466 const who = admin(actor);
6567 if (!who.ok) return who;
6668 return ok({
@@ -75,6 +77,7 @@ export function listCapacity(
7577 ...changedOf(app.db, key),
7678 })),
7779 changes: capacityChanges(app.db),
80 disk: diskUse(app.config.dataDir),
7881 });
7982 }
8083
src/apps/contain.ts +17 −2
@@ -215,6 +215,14 @@ export interface CellSpec {
215215 * (`apps/background.ts`). The step runner sets it.
216216 */
217217 readonly background?: boolean | undefined;
218 /**
219 * A directory on disk mounted as the cell's `/tmp`, in place of the small
220 * tmpfs: a check's tests write thousands of throwaway forges there, which
221 * filled {@link TMP_MIB} and failed checks for the machine, not the code
222 * (#6947). Kept under `home`, so the disk watch counts it and the home's
223 * removal takes it.
224 */
225 readonly tmp?: string | undefined;
218226 }
219227
220228 /**
@@ -355,9 +363,15 @@ const NAMESPACES = [
355363
356364 // ---- the cell ----
357365
358 /** The size of a cell's own `/tmp`, MiB. */
366 /** The size of a cell's own `/tmp`, MiB, unless it brings its own (`CellSpec.tmp`). */
359367 export const TMP_MIB = 256;
360368
369 /** How a cell's `/tmp` is mounted: its own directory on disk, else a bounded tmpfs. */
370 export const tmpMount = (spec: Pick<CellSpec, "tmp">): string[] =>
371 spec.tmp === undefined
372 ? ["--size", String(TMP_MIB * 1024 * 1024), "--tmpfs", "/tmp"]
373 : ["--bind", spec.tmp, "/tmp"];
374
361375 /** The proxy's port inside every cell. */
362376 const PROXY_PORT = 3128;
363377 const INSIDE = "/run/forge";
@@ -463,6 +477,7 @@ export function openCell(
463477 const { hide, blank, reveal } = cellMounts(spec, hiddenMounts());
464478 const writable = [...new Set([spec.dir, spec.home, ...spec.writable])];
465479 for (const path of writable) mkdirSync(path, { recursive: true });
480 if (spec.tmp !== undefined) mkdirSync(spec.tmp, { recursive: true });
466481 // tmpfs is memory the watch cannot see from outside, so every one is
467482 // small and bounded, and the ones that only exist to hide are read-only.
468483 const SMALL = String(1 << 20);
@@ -481,7 +496,7 @@ export function openCell(
481496 ...NAMESPACES,
482497 ...(spec.terminal === true ? [] : ["--new-session"]),
483498 "--ro-bind", "/", "/",
484 "--size", String(TMP_MIB * 1024 * 1024), "--tmpfs", "/tmp",
499 ...tmpMount(spec),
485500 "--size", SMALL, "--tmpfs", "/run",
486501 ...hide.flatMap((path) => ["--size", SMALL, "--tmpfs", path]),
487502 ...blank.flatMap((path) => ["--ro-bind", "/dev/null", path]),
src/checks/check_cell.ts added +44
@@ -0,0 +1,44 @@
1 /**
2 * The cell a check runs in: plain data in, the `CellSpec` out, so what a
3 * check may see, write and reach is read in one place and tested without a
4 * cell. Its HOME sits beside the checkout, and its /tmp inside that HOME on
5 * disk, not the cell's small tmpfs (#6947).
6 */
7
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { resolveLimits, STEP_HOSTS, type CellSpec } from "../apps/contain.ts";
11 import type { CheckManifest } from "../apps/manifest.ts";
12
13 export interface CheckCellInput {
14 /** The checkout the check runs in. */
15 readonly dir: string;
16 /** The forge's data directory, hidden inside. */
17 readonly dataDir: string;
18 readonly manifest: Pick<CheckManifest, "egress" | "limits">;
19 readonly forgeUrl: string;
20 readonly egressMode: "enforce" | "report";
21 readonly onUndeclared: (host: string, port: number) => void;
22 readonly runId: string;
23 }
24
25 /** A check's HOME: beside its checkout, removed with it. */
26 export const checkHome = (dir: string): string => `${dir}.home`;
27
28 export function checkCell(x: CheckCellInput): CellSpec {
29 const home = checkHome(x.dir);
30 return {
31 dir: x.dir,
32 writable: [],
33 home,
34 // The suite's temp on disk, not the cell's small tmpfs (#6947).
35 tmp: join(home, "tmp"),
36 hide: [x.dataDir, process.env["HOME"] ?? "", tmpdir()],
37 egress: [...STEP_HOSTS, ...x.manifest.egress],
38 reach: [x.forgeUrl],
39 limits: resolveLimits("step", x.manifest.limits),
40 onUndeclared: x.onUndeclared,
41 egressMode: x.egressMode,
42 label: `check:${x.runId}`,
43 };
44 }
src/checks/service.ts +38 −21
@@ -13,7 +13,6 @@
1313
1414 import { pullDir, runDir } from "../core/pulls.ts";
1515 import { existsSync, mkdirSync, readdirSync, rmSync } from "node:fs";
16 import { tmpdir } from "node:os";
1716 import { join } from "node:path";
1817 import type { App } from "../app.ts";
1918 import { releaseOnGreen } from "../actions/holds.ts";
@@ -87,14 +86,15 @@ import {
8786 cellsAvailable,
8887 limitWarnings,
8988 resolveLimits,
90 STEP_HOSTS,
91 type CellSpec,
9289 } from "../apps/contain.ts";
90 import { checkCell, checkHome } from "./check_cell.ts";
9391 import { summariseCheckFailure } from "./summary.ts";
9492 import type { Result } from "../kernel/result.ts";
9593 import { placementFromEnv, type Placement } from "./placement.ts";
9694 import { JOB_VANISHED, runJobStep } from "./job_step.ts";
9795 import { checkInstallCache, installWithRetry, withLock } from "./install.ts";
96 import { boundCache } from "../kernel/cache_bound.ts";
97 import { bytes } from "../system/format.ts";
9898 import { checkStage, type GateStage } from "./gate_reason.ts";
9999 import { extraGates, gateQueued, gateRunId, gatesInFlight, watchGates } from "./queue.ts";
100100 import type { JobOptions } from "../cluster/jobs.ts";
@@ -290,17 +290,27 @@ export function requeueAbandonedChecks(app: App, staleMs: number): number {
290290 /** Where a check's checkout lives while it runs. */
291291 const checksRoot = (dataDir: string): string => join(dataDir, "checks");
292292
293 /**
294 * What the checks keep beside their checkouts, never swept: the gate logs
295 * (`gate_log.ts`), learned files (`learned.ts`), impact maps
296 * (`impact-store.ts`) and the last gate (`gate_last.ts`).
297 */
298 const KEPT = new Set(["gates", "learned", "impact", "gate-last.json"]);
299
293300 /**
294301 * Remove the checkouts nobody is using: one whose run is not running with a
295 * live lease, or has no run at all. A forge booting during a roll shares this
296 * directory and database with the old one, whose checks are still running in
297 * theirs; those are left alone.
302 * live lease, or has no run at all, and its cell's home with it. A forge
303 * booting during a roll shares this directory and database with the old one,
304 * whose checks are still running in theirs; those are left alone, and so is
305 * what the checks keep here ({@link KEPT}).
298306 */
299307 export function sweepCheckouts(app: App, now: number = Date.now()): void {
300308 const root = checksRoot(app.config.dataDir);
301309 if (!existsSync(root)) return;
302310 for (const name of readdirSync(root)) {
303 const run = getRun(app.db, name);
311 if (KEPT.has(name)) continue;
312 // A cell's home (`<run id>.home`) lives and goes with its run's checkout.
313 const run = getRun(app.db, name.endsWith(".home") ? name.slice(0, -".home".length) : name);
304314 const live =
305315 run !== undefined &&
306316 run.state === "running" &&
@@ -912,21 +922,18 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
912922 });
913923 write("placement: job");
914924 }
915 const home = `${dir}.home`;
916 const cell: CellSpec | undefined = cells?.ok
917 ? {
925 const home = checkHome(dir);
926 const cell = cells?.ok
927 ? checkCell({
918928 dir,
919 writable: [],
920 home,
921 hide: [app.config.dataDir, process.env["HOME"] ?? "", tmpdir()],
922 egress: [...STEP_HOSTS, ...manifest.egress],
923 reach: [options.forgeUrl],
924 limits: resolveLimits("step", manifest.limits),
929 dataDir: app.config.dataDir,
930 manifest,
931 forgeUrl: options.forgeUrl,
932 egressMode: egressSetting(app.db),
925933 onUndeclared: (host, port) =>
926934 noteReached(app.config.dataDir, repo.id, host, port),
outdated

This line is the whole fix for #6947, and no test fails if it is removed: tmpMount is tested on its own, and the real-cell test skips under FORGE_CELLS=off. Move this spec into a named function, for example checkCell(...), and assert that its tmp sits under home.

927 egressMode: egressSetting(app.db),
928 label: `check:${run.id}`,
929 }
935 runId: run.id,
936 })
930937 : undefined;
931938 try {
932939 for (const line of limitWarnings(manifest.limits)) write(line);
@@ -992,8 +999,18 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
992999 const installed = await installWithRetry(
9931000 async (log) => {
9941001 if (shared === undefined) return once(log);
995 mkdirSync(shared, { recursive: true });
996 const held = await withLock(`${shared}.lock`, () => once(log), { signal });
1002 // Emptied under the lock, so no other install is reading it.
1003 const held = await withLock(
1004 `${shared}.lock`,
1005 async () => {
1006 const freed = await boundCache(shared);
1007 if (freed !== undefined)
1008 log(`install cache past its ceiling: emptied ${bytes(freed)} first`);
1009 mkdirSync(shared, { recursive: true });
1010 return once(log);
1011 },
1012 { signal },
1013 );
9971014 return held.ok ? held.value : held;
9981015 },
9991016 { log: write, signal, retryMs: options.installRetryMs },
src/contract/openapi/routes/getApiV1AdminCapacity.ts +1 −1
@@ -4,7 +4,7 @@ export const route: Route = {
44 method: "get",
55 path: "/api/v1/admin/capacity",
66 summary:
7 "Read the forge's capacity settings (a person admin's token, any scope); answers { items: [{ key, label, help, value, source, default, env, changed? }], changes, next_cursor }",
7 "Read the forge's capacity settings (a person admin's token, any scope); answers { items: [{ key, label, help, value, source, default, env, changed? }], changes, disk: { disk: { freeBytes, totalBytes }, measuredAt?, measuring, consumers: [{ path, bytes, parts: [{ path, bytes }] }] }, next_cursor }",
88 auth: true,
99 refusals: { not_found: "the token is not a person who administers the forge" },
1010 };
src/crew/workdir.ts +28 −6
@@ -9,6 +9,9 @@ import { rm } from "node:fs/promises";
99 import { dirname, join } from "node:path";
1010 import type { Agent } from "../core/agents.ts";
1111 import type { Repo } from "../core/repos.ts";
12 import { withLock } from "../checks/install.ts";
13 import { boundCache, INSTALL_CACHE_MAX_BYTES } from "../kernel/cache_bound.ts";
14 import { bytes } from "../system/format.ts";
1215
1316 export const crewRoot = (dataDir: string): string => join(dataDir, "crew");
1417
@@ -146,10 +149,14 @@ const bunInstall: InstallRunner = (dir, env) =>
146149 export function installNote(result: {
147150 installed: boolean;
148151 error?: string;
152 emptied?: string;
149153 }): string | undefined {
150154 if (result.error !== undefined)
151155 return `dependency install failed: ${result.error}`;
152 return result.installed ? "dependencies installed" : undefined;
156 if (!result.installed) return undefined;
157 return result.emptied === undefined
158 ? "dependencies installed"
159 : `dependencies installed; the install cache was past its ceiling, so its ${result.emptied} were emptied first`;
153160 }
154161
155162 /**
@@ -158,12 +165,17 @@ export function installNote(result: {
158165 * package.json and a lockfile is installed; a workdir whose node_modules was
159166 * installed for the same lockfile hash is left alone. Never throws: a failure
160167 * comes back as a message for the log, and the run starts regardless.
168 *
169 * The cache is shared by every crew run on the repository, so it is bounded
170 * and installed into under the lock the checks hold on theirs
171 * (`checks/install.ts`): one run never empties it under another's install.
161172 */
162173 export async function installDependencies(
163174 dir: string,
164175 cacheDir: string,
165176 run: InstallRunner = bunInstall,
166 ): Promise<{ installed: boolean; error?: string }> {
177 bound: { readonly max?: number; readonly now?: number } = {},
178 ): Promise<{ installed: boolean; error?: string; emptied?: string }> {
167179 try {
168180 const lock = LOCKFILES.map((name) => join(dir, name)).find((p) =>
169181 existsSync(p),
@@ -177,9 +189,19 @@ export async function installDependencies(
177189 if (existsSync(mark) && readFileSync(mark, "utf8") === hash)
178190 return { installed: false };
179191 const env = installEnv(cacheDir);
180 mkdirSync(cacheDir, { recursive: true });
181 mkdirSync(env["HOME"]!, { recursive: true });
182 const result = await run(dir, env);
192 const held = await withLock(
193 `${cacheDir}.lock`,
194 async () => {
195 // A cache past its ceiling starts over (`kernel/cache_bound.ts`).
196 const freed = await boundCache(cacheDir, bound.max ?? INSTALL_CACHE_MAX_BYTES, bound.now);
197 mkdirSync(cacheDir, { recursive: true });
198 mkdirSync(env["HOME"]!, { recursive: true });
199 return { freed, result: await run(dir, env) };
200 },
201 { signal: new AbortController().signal },
202 );
203 if (!held.ok) return { installed: false, error: held.error };
204 const { freed, result } = held.value;
183205 if (result.code !== 0)
184206 return {
185207 installed: false,
@@ -187,7 +209,7 @@ export async function installDependencies(
187209 };
188210 mkdirSync(join(dir, "node_modules"), { recursive: true });
189211 writeFileSync(mark, hash);
190 return { installed: true };
212 return freed === undefined ? { installed: true } : { installed: true, emptied: bytes(freed) };
191213 } catch (error) {
192214 return { installed: false, error: String(error) };
193215 }
src/http/api.ts +1 −1
@@ -1679,7 +1679,7 @@ export function apiRoutes(router: Router<Handler>): void {
16791679 const denied = accountAdmin(ctx, "ro");
16801680 if (denied) return denied;
16811681 const listed = timePhase(ctx.request, "capacity", () => listCapacity(ctx.app, ctx.auth.user));
1682 return timePhase(ctx.request, "render", () => (listed.ok ? json({ items: listed.value.settings, changes: listed.value.changes, next_cursor: null }) : notFound()));
1682 return timePhase(ctx.request, "render", () => (listed.ok ? json({ items: listed.value.settings, changes: listed.value.changes, disk: listed.value.disk, next_cursor: null }) : notFound()));
16831683 });
16841684
16851685 router.add("PUT", "/api/v1/admin/capacity/:key", async (ctx) => {
src/http/mcp/tools/capacity.ts +1 −1
@@ -5,7 +5,7 @@ import { good, bad, type Tool } from "../shared.ts";
55
66 export const tool: Tool = {
77 name: "capacity",
8 doc: "`{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`); with `key` and `value` (`\"\"` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token",
8 doc: "`{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`) and what fills the disk under the data directory, biggest first; with `key` and `value` (`\"\"` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token",
99 description:
1010 "Forge capacity and guards, live. With `key` and `value` (\"\" restores default), plans.",
1111 inputSchema: {
src/kernel/cache_bound.ts added +63
@@ -0,0 +1,63 @@
1 /**
2 * An install cache with a ceiling (#6947). Bun's cache keeps every version
3 * of every package it ever fetched and never forgets one, so a repository's
4 * cache — `crew/.cache/<repo>/bun` for the crew, `check-install/<repo>/bun`
5 * for uncontained checks — only grows. Before an install the cache is
6 * measured, at most once per {@link BOUND_EVERY_MS}, and one over
7 * {@link INSTALL_CACHE_MAX_BYTES} is emptied whole: the install fetches what
8 * it needs again, and the cache starts over holding only what is in use.
9 *
10 * The cache is renamed aside before it is removed, so a reader that holds
11 * no lock — an agent's own `bun install` in its sandbox — finds either the
12 * whole cache or none, never one half-deleted. Callers that install empty
13 * it under the install lock as well (`checks/install.ts` `withLock`).
14 */
15
16 import { readdirSync } from "node:fs";
17 import { rename, rm } from "node:fs/promises";
18 import { basename, dirname, join } from "node:path";
19 import { diskBytes } from "./disk_use.ts";
20
21 /** A repository's install cache over this is emptied before the next install. */
22 export const INSTALL_CACHE_MAX_BYTES = 4 * 1024 ** 3;
23 /** A cache is measured no oftener than this; the walk is not free. */
24 export const BOUND_EVERY_MS = 3_600_000;
25
26 const measuredAt = new Map<string, number>();
27
28 /**
29 * Empty `dir` when it holds more than `max` bytes. Answers the bytes freed,
30 * or undefined when it was under, or measured too lately to look again.
31 */
32 export async function boundCache(
33 dir: string,
34 max: number = INSTALL_CACHE_MAX_BYTES,
35 now: number = Date.now(),
36 ): Promise<number | undefined> {
37 const last = measuredAt.get(dir);
38 if (last !== undefined && now - last < BOUND_EVERY_MS) return undefined;
39 measuredAt.set(dir, now);
40 const bytes = await diskBytes(dir);
41 if (bytes <= max) return undefined;
42 const aside = `${dir}${ASIDE}${process.pid}-${now}`;
43 await rename(dir, aside);
44 await rm(aside, { recursive: true, force: true });
45 await removeLeftAside(dir);
46 return bytes;
47 }
48
49 const ASIDE = ".emptied-";
50
51 /** Removes what an earlier emptying renamed aside and did not live to remove. */
52 async function removeLeftAside(dir: string): Promise<void> {
53 const prefix = `${basename(dir)}${ASIDE}`;
54 let names: string[];
55 try {
56 names = readdirSync(dirname(dir));
57 } catch {
58 return;
59 }
60 for (const name of names)
61 if (name.startsWith(prefix))
62 await rm(join(dirname(dir), name), { recursive: true, force: true });
63 }
src/kernel/disk_use.ts added +213
@@ -0,0 +1,213 @@
1 /**
2 * What fills the disk under `FORGE_DATA` (#6947): each entry at its top,
3 * biggest first, with the biggest few inside each — `crew 40 GiB, of which
4 * crew/.cache 31 GiB`. The free figure (`statDisk`) says the disk is full;
5 * this says by what.
6 *
7 * Counted as `du` counts: blocks on the disk, not apparent sizes, a file
8 * linked in twice (an install cache and the `node_modules` linked from it)
9 * once, symlinks never followed, and another filesystem mounted inside not
10 * at all — none of those fill this disk.
11 *
12 * Walking a large data directory takes long, so the reading is kept in
13 * memory and a request never waits for it: the first ask starts a walk in
14 * the background and answers what it has, and a reading older than
15 * {@link DISK_USE_FRESH_MS} starts the next. The disk is the truth; a
16 * restart measures again.
17 */
18
19 import { lstat, readdir } from "node:fs/promises";
20 import { join } from "node:path";
21 import { createLog } from "./log.ts";
22 import { statDisk, type DiskReading } from "./vitals.ts";
23
24 const log = createLog("disk-use");
25
26 /** A reading this old is measured again on the next ask. */
27 export const DISK_USE_FRESH_MS = 30 * 60_000;
28 /** How many top-level entries are named, and how many parts inside each. */
29 export const TOP_ENTRIES = 8;
30 export const TOP_PARTS = 3;
31
32 export interface DiskPart {
33 /** Relative to the data directory, `/`-separated: `crew/.cache`. */
34 readonly path: string;
35 readonly bytes: number;
36 }
37
38 export interface DiskConsumer extends DiskPart {
39 /** The biggest entries inside it, biggest first; none for a file. */
40 readonly parts: readonly DiskPart[];
41 }
42
43 export interface DiskUse {
44 readonly disk: DiskReading;
45 /** When the consumers were measured; absent until the first walk ends. */
46 readonly measuredAt?: string;
47 /** Whether a walk is under way now. */
48 readonly measuring: boolean;
49 /** The biggest entries under the data directory, biggest first. */
50 readonly consumers: readonly DiskConsumer[];
51 }
52
53 interface Walk {
54 readonly dev: number;
55 /** Inodes already counted, for files with more than one link. */
56 readonly seen: Set<string>;
57 }
58
59 /** Blocks a stat says the entry holds, counting a multiply-linked file once. */
60 function held(walk: Walk, info: Awaited<ReturnType<typeof lstat>>): number {
61 if (info.isDirectory() || info.nlink <= 1) return Number(info.blocks) * 512;
62 const key = `${info.dev}:${info.ino}`;
63 if (walk.seen.has(key)) return 0;
64 walk.seen.add(key);
65 return Number(info.blocks) * 512;
66 }
67
68 /** The bytes under `path`, on the walk's own device only. */
69 async function treeBytes(walk: Walk, path: string): Promise<number> {
70 let total = 0;
71 const pending = [path];
72 while (pending.length > 0) {
73 const dir = pending.pop()!;
74 let names: string[];
75 try {
76 names = await readdir(dir);
77 } catch {
78 continue; // gone, or not ours to read: nothing to count
79 }
80 for (const name of names) {
81 const child = join(dir, name);
82 try {
83 const info = await lstat(child);
84 if (Number(info.dev) !== walk.dev) continue;
85 total += held(walk, info);
86 if (info.isDirectory()) pending.push(child);
87 } catch {
88 // removed while walking
89 }
90 }
91 }
92 return total;
93 }
94
95 /** The bytes `path` holds on its own disk, counted as the walk counts them. */
96 export async function diskBytes(path: string): Promise<number> {
97 let info;
98 try {
99 info = await lstat(path);
100 } catch {
101 return 0;
102 }
103 const walk: Walk = { dev: Number(info.dev), seen: new Set() };
104 return held(walk, info) + (info.isDirectory() ? await treeBytes(walk, path) : 0);
105 }
106
107 /** One entry and the parts inside it, each measured whole. */
108 async function consumer(walk: Walk, root: string, name: string): Promise<DiskConsumer | undefined> {
109 const path = join(root, name);
110 let info;
111 try {
112 info = await lstat(path);
113 } catch {
114 return undefined;
115 }
116 if (Number(info.dev) !== walk.dev) return undefined;
117 const own = held(walk, info);
118 if (!info.isDirectory()) return { path: name, bytes: own, parts: [] };
119 let names: string[];
120 try {
121 names = await readdir(path);
122 } catch {
123 return { path: name, bytes: own, parts: [] };
124 }
125 const parts: DiskPart[] = [];
126 let bytes = own;
127 for (const inner of names.sort()) {
128 const at = join(path, inner);
129 let part;
130 try {
131 part = await lstat(at);
132 } catch {
133 continue;
134 }
135 if (Number(part.dev) !== walk.dev) continue;
136 const size = held(walk, part) + (part.isDirectory() ? await treeBytes(walk, at) : 0);
137 bytes += size;
138 parts.push({ path: `${name}/${inner}`, bytes: size });
139 }
140 return { path: name, bytes, parts: biggest(parts).slice(0, TOP_PARTS) };
141 }
142
143 const biggest = <T extends DiskPart>(list: T[]): T[] =>
144 list.sort((a, b) => b.bytes - a.bytes || (a.path < b.path ? -1 : 1));
145
146 /** Walk `dataDir` now: its biggest entries, biggest first. */
147 export async function measureDiskUse(dataDir: string): Promise<DiskConsumer[]> {
148 let names: string[];
149 let dev: number;
150 try {
151 dev = Number((await lstat(dataDir)).dev);
152 names = await readdir(dataDir);
153 } catch {
154 return [];
155 }
156 const walk: Walk = { dev, seen: new Set() };
157 const found: DiskConsumer[] = [];
158 for (const name of names.sort()) {
159 const entry = await consumer(walk, dataDir, name);
160 if (entry !== undefined) found.push(entry);
161 }
162 return biggest(found).slice(0, TOP_ENTRIES);
163 }
164
165 interface Kept {
166 consumers: DiskConsumer[];
167 measuredAt?: number;
168 walking?: Promise<DiskConsumer[]>;
169 }
170
171 const kept = new Map<string, Kept>();
172
173 /** Start a walk of `dataDir` unless one is under way; resolves when it ends. */
174 export function refreshDiskUse(dataDir: string): Promise<DiskConsumer[]> {
175 let entry = kept.get(dataDir);
176 if (entry === undefined) kept.set(dataDir, (entry = { consumers: [] }));
177 const into = entry;
178 into.walking ??= measureDiskUse(dataDir)
179 .then((consumers) => {
180 into.consumers = consumers;
181 into.measuredAt = Date.now();
182 return consumers;
183 })
184 .catch((error: unknown) => {
185 log.warn("disk use not measured", { error: String(error) });
186 return into.consumers;
187 })
188 .finally(() => {
189 delete into.walking;
190 });
191 return into.walking;
192 }
193
194 /**
195 * What fills the disk under `dataDir`, as last measured, never waiting: a
196 * missing or stale reading starts a walk and answers what it has.
197 */
198 export function diskUse(
199 dataDir: string,
200 now: number = Date.now(),
201 read: (dir: string) => DiskReading = statDisk,
202 ): DiskUse {
203 const entry = kept.get(dataDir);
204 if (entry?.measuredAt === undefined || now - entry.measuredAt >= DISK_USE_FRESH_MS)
205 void refreshDiskUse(dataDir);
206 const after = kept.get(dataDir)!;
207 return {
208 disk: read(dataDir),
209 ...(after.measuredAt === undefined ? {} : { measuredAt: new Date(after.measuredAt).toISOString() }),
210 measuring: after.walking !== undefined,
211 consumers: after.consumers,
212 };
213 }
src/web/pages/capacity.ts +50 −2
@@ -4,10 +4,13 @@
44 * refused or reported (`core/capacity.ts`). A change here is honoured by
55 * the next run, check or cell — never by a restart, which is the point —
66 * and each one is listed below the settings: who, when, from what to what.
7 * Above them, what fills the disk (`kernel/disk_use.ts`): the page the
8 * disk-red card links to names the biggest consumers.
79 */
810
911 import { changeCapacity, listCapacity, type CapacityRow } from "../../actions/capacity.ts";
1012 import { CAPACITY, type CapacityKey } from "../../core/capacity.ts";
13 import type { DiskUse } from "../../kernel/disk_use.ts";
1114 import { bytes } from "../../system/format.ts";
1215 import { egressSnapshot } from "../../apps/egress.ts";
1316 import type { Ctx } from "../../http/context.ts";
@@ -67,6 +70,51 @@ function input(key: CapacityKey, value: string) {
6770 />`;
6871 }
6972
73 /** "12% of the disk" for a share of the volume; "" when it reports no size. */
74 const share = (bytesOf: number, total: number): string =>
75 total > 0 ? `${Math.round((bytesOf / total) * 100)}% of the disk` : "";
76
77 /** The disk's free figure and its biggest consumers, biggest first. */
78 function diskCard(use: DiskUse) {
79 const { freeBytes, totalBytes } = use.disk;
80 const free =
81 totalBytes > 0
82 ? `${bytes(freeBytes)} free of ${bytes(totalBytes)} (${Math.round((freeBytes / totalBytes) * 100)}%).`
83 : "The volume reports no size.";
84 const measured =
85 use.measuredAt === undefined
86 ? html`Measuring what fills it now; reload in a minute.`
87 : html`What fills it, biggest first, measured ${time(use.measuredAt)}${use.measuring ? "; measuring again now" : ""}.`;
88 return card(
89 html`<div class="card-body">
90 <p class="muted small" id="disk-free">${free} ${measured}</p>
91 </div>
92 ${when(
93 use.consumers.length > 0,
94 () => html`<ul class="list" id="disk-use">
95 ${use.consumers.map((c) =>
96 row({
97 lead: icon("activity"),
98 title: html`<span class="mono">${c.path}</span>`,
99 meta: html`${share(c.bytes, totalBytes)}${c.parts.length === 0
100 ? ""
101 : html`${totalBytes > 0 ? "; " : ""}biggest inside:
102 ${c.parts.map(
103 (p, i) => html`${i === 0 ? "" : ", "}<span class="mono">${p.path}</span> ${bytes(p.bytes)}`,
104 )}`}`,
105 trail: html`<span class="mono">${bytes(c.bytes)}</span>`,
106 }),
107 )}
108 </ul>`,
109 )}`,
110 {
111 head: html`${icon("activity")}
112 <h2>Disk</h2>`,
113 pad: false,
114 },
115 );
116 }
117
70118 // GET /settings/capacity
71119 export function capacityPage(ctx: Ctx): Response {
72120 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
@@ -76,7 +124,7 @@ export function capacityPage(ctx: Ctx): Response {
76124 listCapacity(ctx.app, ctx.auth.user),
77125 );
78126 if (!listed.ok) return notFoundPage(ctx);
79 const { settings, changes } = listed.value;
127 const { settings, changes, disk } = listed.value;
80128 const errs = timePhase(ctx.request, "errors", () =>
81129 ctx.app.signals.errors.recent(10),
82130 );
@@ -182,7 +230,7 @@ export function capacityPage(ctx: Ctx): Response {
182230 pad: false,
183231 },
184232 );
185 const body = html`${happened}${connections}${when(
233 const body = html`${diskCard(disk)}${happened}${connections}${when(
186234 reported,
187235 () => html`<p class="muted small mb" id="egress-reported">
188236 ${icon("activity")} Egress is reported, not enforced: every new app,
test/cache-bound.test.ts added +118
@@ -0,0 +1,118 @@
1 /**
2 * An install cache has a ceiling (#6947): bun never forgets a package, so a
3 * repository's cache only grew. One over its ceiling is emptied before the
4 * next install, one under it is left alone, and it is measured at most once
5 * an hour; the crew's install says in the run log when it emptied one.
6 */
7
8 import { afterAll, beforeAll, expect, test } from "bun:test";
9 import { existsSync, mkdirSync, readdirSync, writeFileSync } from "node:fs";
10 import { mkdtemp, rm } from "node:fs/promises";
11 import { tmpdir } from "node:os";
12 import { join } from "node:path";
13 import { BOUND_EVERY_MS, boundCache } from "../src/kernel/cache_bound.ts";
14 import { installDependencies, installNote } from "../src/crew/workdir.ts";
15
16 let root = "";
17
18 beforeAll(async () => {
19 root = await mkdtemp(join(tmpdir(), "forge-cache-bound-"));
20 });
21
22 afterAll(async () => {
23 await rm(root, { recursive: true, force: true });
24 });
25
26 const cacheWith = (name: string, n: number): string => {
27 const dir = join(root, name, "bun");
28 mkdirSync(dir, { recursive: true });
29 writeFileSync(join(dir, "pkg.tgz"), new Uint8Array(n).fill(7));
30 return dir;
31 };
32
33 test("a cache over its ceiling is emptied, one under it kept", async () => {
34 const over = cacheWith("over", 256 * 1024);
35 const under = cacheWith("under", 1024);
36 expect(await boundCache(over, 64 * 1024)).toBeGreaterThanOrEqual(256 * 1024);
37 expect(existsSync(over)).toBe(false);
38 expect(await boundCache(under, 64 * 1024)).toBeUndefined();
39 expect(existsSync(join(under, "pkg.tgz"))).toBe(true);
40 });
41
42 test("a cache is measured at most once an hour", async () => {
43 const now = Date.now();
44 const dir = cacheWith("hourly", 1024);
45 expect(await boundCache(dir, 64 * 1024, now)).toBeUndefined();
46 writeFileSync(join(dir, "more.tgz"), new Uint8Array(256 * 1024).fill(7));
47 expect(await boundCache(dir, 64 * 1024, now + 60_000)).toBeUndefined();
48 expect(existsSync(dir)).toBe(true);
49 expect(await boundCache(dir, 64 * 1024, now + BOUND_EVERY_MS)).toBeGreaterThan(0);
50 expect(existsSync(dir)).toBe(false);
51 });
52
53 test("the crew's install keeps a cache under the ceiling, and its note names one emptied", async () => {
54 const work = join(root, "work");
55 mkdirSync(work, { recursive: true });
56 writeFileSync(join(work, "package.json"), "{}");
57 writeFileSync(join(work, "bun.lock"), "{}");
58 const cache = cacheWith("crew", 1024);
59 const result = await installDependencies(work, cache, async () => ({ code: 0, stderr: "" }));
60 expect(result).toEqual({ installed: true });
61 expect(existsSync(join(cache, "pkg.tgz"))).toBe(true);
62 expect(installNote({ installed: true, emptied: "4.2 GiB" })).toContain("emptied");
63 expect(installNote({ installed: true })).toBe("dependencies installed");
64 });
65
66 test("an emptied cache leaves nothing aside: renamed first, then removed", async () => {
67 const over = cacheWith("aside", 256 * 1024);
68 const parent = join(over, "..");
69 mkdirSync(join(parent, "bun.emptied-1-1"), { recursive: true });
70 expect(await boundCache(over, 64 * 1024)).toBeGreaterThan(0);
71 expect(readdirSync(parent).filter((n) => n.startsWith("bun."))).toEqual([]);
72 });
73
74 test("two crew installs on one cache: the one that empties it waits for the other's install to finish", async () => {
75 const workdir = (name: string): string => {
76 const work = join(root, name);
77 mkdirSync(work, { recursive: true });
78 writeFileSync(join(work, "package.json"), "{}");
79 writeFileSync(join(work, "bun.lock"), name);
80 return work;
81 };
82 const cache = cacheWith("shared", 1024);
83 const max = 64 * 1024;
84 const now = Date.now();
85 let started = (): void => {};
86 const first = new Promise<void>((resolve) => (started = resolve));
87 let release = (): void => {};
88 const gate = new Promise<void>((resolve) => (release = resolve));
89 // The first install fills the cache past its ceiling, then reads back
90 // what it fetched while the second run is asking to empty it.
91 const reading = installDependencies(
92 workdir("reading"),
93 cache,
94 async () => {
95 writeFileSync(join(cache, "fetched.tgz"), new Uint8Array(256 * 1024).fill(7));
96 started();
97 await gate;
98 return existsSync(join(cache, "fetched.tgz"))
99 ? { code: 0, stderr: "" }
100 : { code: 1, stderr: "the cache was emptied under this install" };
101 },
102 { max, now },
103 );
104 await first;
105 const emptying = installDependencies(
106 workdir("emptying"),
107 cache,
108 async () => ({ code: 0, stderr: "" }),
109 { max, now: now + BOUND_EVERY_MS },
110 );
111 await Bun.sleep(50);
112 release();
113 expect(await reading).toEqual({ installed: true });
114 const second = await emptying;
115 expect(second.installed).toBe(true);
116 expect(second.emptied).toBeDefined();
117 expect(existsSync(join(cache, "fetched.tgz"))).toBe(false);
118 });
test/cell-tmp.test.ts added +85
@@ -0,0 +1,85 @@
1 /**
2 * A check's /tmp is on disk (#6947). A cell's /tmp was a 256 MiB tmpfs, and
3 * the suite's throwaway forges filled it: checks failed with ENOSPC for the
4 * machine, not the code. A cell given `tmp` mounts that directory as /tmp —
5 * the paths inside stay short enough for a unix socket, the disk watch
6 * counts it under the home, and the home's removal takes it.
7 */
8
9 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
10 import { existsSync, readFileSync } from "node:fs";
11 import { mkdtemp, rm } from "node:fs/promises";
12 import { tmpdir } from "node:os";
13 import { join } from "node:path";
14 import { TMP_MIB, tmpMount, type CellSpec } from "../src/apps/contain.ts";
15 import { childEnv, runStep } from "../src/apps/process.ts";
16 import { checkCell, checkHome } from "../src/checks/check_cell.ts";
17 import { portInsteadOfSocket } from "../src/git/hook-address.ts";
18 import { missing } from "./helpers/tools.ts";
19
20 const noCells =
21 process.env["FORGE_CELLS"] === "off" ||
22 process.platform !== "linux" ||
23 portInsteadOfSocket() ||
24 missing("bwrap", "socat");
25
26 test("a cell with its own tmp binds it at /tmp; one without gets the bounded tmpfs", () => {
27 expect(tmpMount({ tmp: "/data/checks/run.home/tmp" })).toEqual(["--bind", "/data/checks/run.home/tmp", "/tmp"]);
28 expect(tmpMount({})).toEqual(["--size", String(TMP_MIB * 1024 * 1024), "--tmpfs", "/tmp"]);
29 });
30
31 test("a check's cell puts its /tmp on disk inside its home, never the tmpfs", () => {
32 const dir = "/data/checks/01run";
33 const cell = checkCell({
34 dir,
35 dataDir: "/data",
36 manifest: { egress: ["example.com"], limits: {} },
37 forgeUrl: "https://forge.test",
38 egressMode: "enforce",
39 onUndeclared: () => {},
40 runId: "01run",
41 });
42 expect(cell.home).toBe(checkHome(dir));
43 expect(cell.tmp).toBe(join(checkHome(dir), "tmp"));
44 expect(tmpMount(cell)).toEqual(["--bind", join(checkHome(dir), "tmp"), "/tmp"]);
45 expect(cell.hide).toContain("/data");
46 expect(cell.egress).toContain("example.com");
47 expect(cell.label).toBe("check:01run");
48 });
49
50 describe.skipIf(noCells)("a cell's /tmp on disk", () => {
51 let dir = "";
52
53 beforeAll(async () => {
54 dir = await mkdtemp(join(tmpdir(), "forge-cell-tmp-"));
55 });
56
57 afterAll(async () => {
58 await rm(dir, { recursive: true, force: true });
59 });
60
61 test("what the step writes to /tmp lands in the cell's tmp, past the tmpfs's size", async () => {
62 const home = join(dir, ".home");
63 const spec: CellSpec = {
64 dir,
65 writable: [],
66 home,
67 tmp: join(home, "tmp"),
68 hide: [],
69 egress: [],
70 reach: [],
71 limits: { memoryMiB: 2048, cpus: 2, diskMiB: 4096 },
72 };
73 const ran = await runStep("check", `head -c ${(TMP_MIB + 16) * 1024 * 1024} /dev/zero > /tmp/big && echo kept > /tmp/mark`, {
74 cwd: dir,
75 env: childEnv({}),
76 log: () => {},
77 signal: new AbortController().signal,
78 timeoutMs: 60_000,
79 cell: spec,
80 });
81 expect(ran.ok).toBe(true);
82 expect(readFileSync(join(home, "tmp", "mark"), "utf8")).toBe("kept\n");
83 expect(existsSync(join(home, "tmp", "big"))).toBe(true);
84 }, 60_000);
85 });
test/checks-sweep-kept.test.ts added +51
@@ -0,0 +1,51 @@
1 /**
2 * The checkout sweep removes checkouts, nothing else (#6947): the checks
3 * keep their gate logs, learned files and impact maps beside the checkouts,
4 * and a live check's cell home is as much in use as its checkout.
5 */
6
7 import { afterAll, beforeAll, expect, test } from "bun:test";
8 import { existsSync, mkdirSync, writeFileSync } from "node:fs";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { createApp, type App } from "../src/app.ts";
13 import { createRepo } from "../src/actions/repos.ts";
14 import { sweepCheckouts } from "../src/checks/service.ts";
15 import { claimRun, createRun } from "../src/core/runs.ts";
16 import { id } from "../src/kernel/id.ts";
17 import { seedAdmin } from "./helpers/admin.ts";
18
19 let dataDir = "";
20 let app: App;
21
22 beforeAll(async () => {
23 dataDir = await mkdtemp(join(tmpdir(), "forge-sweep-kept-"));
24 app = createApp({ dataDir, port: 0 });
25 });
26
27 afterAll(async () => {
28 app.close();
29 await rm(dataDir, { recursive: true, force: true });
30 });
31
32 test("the sweep keeps what the checks keep and a live check's home, and removes the rest", async () => {
33 const admin = await seedAdmin(app, { username: "alice", email: "[email protected]", password: "correct-horse-battery" });
34 if (!admin.ok) throw new Error(admin.error);
35 const repo = await createRepo(app, admin.value, { name: "notes", visibility: "public", description: "" });
36 if (!repo.ok) throw new Error(repo.error);
37 const live = createRun(app.db, { repoId: repo.value.id, kind: "check" });
38 claimRun(app.db, live.id);
39 const gone = id();
40 const root = join(dataDir, "checks");
41 for (const dir of ["gates/alice/notes", "learned", "impact", live.id, `${live.id}.home`, gone, `${gone}.home`])
42 mkdirSync(join(root, dir), { recursive: true });
43 writeFileSync(join(root, "gate-last.json"), "{}");
44
45 sweepCheckouts(app);
46
47 for (const kept of ["gates/alice/notes", "learned", "impact", "gate-last.json", live.id, `${live.id}.home`])
48 expect(existsSync(join(root, kept))).toBe(true);
49 expect(existsSync(join(root, gone))).toBe(false);
50 expect(existsSync(join(root, `${gone}.home`))).toBe(false);
51 });
test/disk-use.test.ts added +125
@@ -0,0 +1,125 @@
1 /**
2 * What fills the disk (#6947): /settings/capacity, its API and the MCP's
3 * capacity tool name the biggest entries under the data directory, biggest
4 * first, counted as `du` counts — a hard-linked file once, a symlink never
5 * followed — and a request never waits for the walk.
6 */
7
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { linkSync, mkdirSync, symlinkSync, writeFileSync } from "node:fs";
10 import { mkdtemp, rm } from "node:fs/promises";
11 import { tmpdir } from "node:os";
12 import { join } from "node:path";
13 import { createApp, type App } from "../src/app.ts";
14 import { listCapacity } from "../src/actions/capacity.ts";
15 import { diskUse, measureDiskUse, refreshDiskUse } from "../src/kernel/disk_use.ts";
16 import type { User } from "../src/core/users.ts";
17 import { serve, type Servers } from "../src/http/server.ts";
18 import { seedAdmin } from "./helpers/admin.ts";
19
20 const MiB = 1024 * 1024;
21 const filled = (n: number): Uint8Array => new Uint8Array(n).fill(7);
22
23 describe("measuring the data directory", () => {
24 let dir = "";
25 let outside = "";
26
27 beforeAll(async () => {
28 dir = await mkdtemp(join(tmpdir(), "forge-disk-use-"));
29 outside = await mkdtemp(join(tmpdir(), "forge-disk-outside-"));
30 mkdirSync(join(dir, "crew", ".cache", "repo"), { recursive: true });
31 mkdirSync(join(dir, "crew", "mason"), { recursive: true });
32 mkdirSync(join(dir, "checks"), { recursive: true });
33 writeFileSync(join(dir, "crew", ".cache", "repo", "pkg.tgz"), filled(2 * MiB));
34 // node_modules linked from the cache holds no more of the disk.
35 linkSync(join(dir, "crew", ".cache", "repo", "pkg.tgz"), join(dir, "crew", "mason", "pkg.tgz"));
36 writeFileSync(join(dir, "checks", "gate-last.json"), filled(64 * 1024));
37 writeFileSync(join(dir, "forge.db"), filled(512 * 1024));
38 writeFileSync(join(outside, "elsewhere.bin"), filled(4 * MiB));
39 symlinkSync(outside, join(dir, "linked"));
40 });
41
42 afterAll(async () => {
43 await rm(dir, { recursive: true, force: true });
44 await rm(outside, { recursive: true, force: true });
45 });
46
47 test("the biggest entry comes first, with the biggest part inside it", async () => {
48 const found = await measureDiskUse(dir);
49 expect(found.map((c) => c.path).slice(0, 3)).toEqual(["crew", "forge.db", "checks"]);
50 const crew = found[0]!;
51 expect(crew.parts[0]?.path).toBe("crew/.cache");
52 expect(crew.parts[0]!.bytes).toBeGreaterThanOrEqual(2 * MiB);
53 expect(found.find((c) => c.path === "forge.db")?.parts).toEqual([]);
54 });
55
56 test("a hard-linked file counts once and a symlink is not followed", async () => {
57 const found = await measureDiskUse(dir);
58 const crew = found.find((c) => c.path === "crew")!;
59 expect(crew.bytes).toBeLessThan(3 * MiB);
60 expect(found.find((c) => c.path === "linked")!.bytes).toBeLessThan(64 * 1024);
61 });
62
63 test("the first ask answers at once and starts the walk; the next has the reading", async () => {
64 const first = diskUse(dir);
65 expect(first.measuring).toBe(true);
66 expect(first.measuredAt).toBeUndefined();
67 expect(first.disk.totalBytes).toBeGreaterThan(0);
68 await refreshDiskUse(dir);
69 const next = diskUse(dir);
70 expect(next.measuring).toBe(false);
71 expect(next.measuredAt).toBeDefined();
72 expect(next.consumers[0]?.path).toBe("crew");
73 });
74 });
75
76 describe("the capacity page names the biggest consumers", () => {
77 let dataDir = "";
78 let app: App;
79 let servers: Servers;
80 let bearer = "";
81 let alice: User;
82
83 beforeAll(async () => {
84 dataDir = await mkdtemp(join(tmpdir(), "forge-disk-page-"));
85 app = createApp({ dataDir, port: 0 });
86 servers = await serve(app);
87 const admin = await seedAdmin(app, { username: "alice", email: "[email protected]", password: "correct-horse-battery" });
88 if (!admin.ok) throw new Error(admin.error);
89 alice = admin.value;
90 const minted = await fetch(`${servers.url}/api/v1/tokens`, {
91 method: "POST",
92 headers: { "content-type": "application/json" },
93 body: JSON.stringify({ adminConfirm: true, username: "alice", password: "correct-horse-battery", name: "cli", scope: "admin" }),
94 });
95 bearer = ((await minted.json()) as { token: string }).token;
96 mkdirSync(join(dataDir, "check-install", "repo", "bun"), { recursive: true });
97 writeFileSync(join(dataDir, "check-install", "repo", "bun", "pkg.tgz"), filled(8 * MiB));
98 await refreshDiskUse(dataDir);
99 });
100
101 afterAll(async () => {
102 servers.stop();
103 app.close();
104 await rm(dataDir, { recursive: true, force: true });
105 });
106
107 test("the page names the biggest entry and what is inside it", async () => {
108 const res = await fetch(`${servers.url}/settings/capacity`, { headers: { authorization: `Bearer ${bearer}` } });
109 const page = await res.text();
110 expect(res.status).toBe(200);
111 expect(page).toContain("<h2>Disk</h2>");
112 expect(page).toMatch(/id="disk-free">[^<]*free of/);
113 const listed = /<ul class="list" id="disk-use">([\s\S]*?)<\/ul>/.exec(page)?.[1] ?? "";
114 expect(listed.indexOf("check-install")).toBeGreaterThan(-1);
115 expect(listed).toContain("check-install/repo");
116 });
117
118 test("the API and the MCP's capacity listing carry the same reading", async () => {
119 const res = await fetch(`${servers.url}/api/v1/admin/capacity`, { headers: { authorization: `Bearer ${bearer}` } });
120 const body = (await res.json()) as { disk: { consumers: { path: string }[] } };
121 expect(body.disk.consumers[0]?.path).toBe("check-install");
122 const listed = listCapacity(app, alice);
123 expect(listed.ok && listed.value.disk.consumers[0]?.path).toBe("check-install");
124 });
125 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs