Public: anyone can read this; the crew can change it.
Find what fills the check machine disk (follow-up to #6945) #6949
mason/forge:mason/6947-find-what-fills-the-check-machine-disk-f → main 6 commits fork 2 threads · 2 unresolved
18 files
changed +869
−39
docs/APPS.md +3 −2
| @@ -497,8 +497,9 @@ reaches the forge and nothing else; install, test and check steps also | ||
| 497 | 497 | reach `registry.npmjs.org`. `limits` (MiB, cores, MiB) default to 512 MiB, |
| 498 | 498 | 1 core and 1 GiB for an app and 2 GiB, 2 cores and 4 GiB for a step, and |
| 499 | 499 | are capped by the machine (`FORGE_CELL_MEMORY_MAX_MIB`, default 4096; the |
| 500 | number of cores; `FORGE_CELL_DISK_MAX_MIB`, default 16384). A cell's `/tmp` | |
| 501 | holds at most 256 MiB. | |
| 500 | number of cores; `FORGE_CELL_DISK_MAX_MIB`, default 16384). An app's `/tmp` | |
| 501 | holds at most 256 MiB. A check's `/tmp` is a directory on disk under its | |
| 502 | home: it counts toward the check's disk limit and is removed with the home. | |
| 502 | 503 | |
| 503 | 504 | Memory is held twice. The watch reads every cell's resident memory once a |
| 504 | 505 | second and stops one over its limit. Where the forge holds a delegated |
docs/MCP.md +1 −1
| @@ -54,7 +54,7 @@ for a tunnel and a forge that sleeps. | ||
| 54 | 54 | | `board_write` | `{ slug, board }` (person admins): write a whole board under a slug, made or replaced: `{ title, range, refresh, panels: [{ title, chart, query, w, h }] }` | yes | |
| 55 | 55 | | `boards` | `{ slug?, panel?, from?, to?, step? }` (person admins): without `slug`, lists the forge's boards (slug, title, range, refresh, panels); with `slug`, that board; with `panel` (place from 0, up to 23), that panel's data over the `from`/`to` window and `step` | | |
| 56 | 56 | | `branches` | `{ repo, name?, action?, from?, kind? }`: without `name`, lists the branches; `kind: "tags"` lists the tags instead (as GET `/api/v1/repos/:owner/:name/tags` does, newest first; no `name` or `action`); with `name` and `action` (`create` from `from`, or `delete`) it does what the branches page does: the default branch is never deleted, only a writer may act | yes | |
| 57 | | `capacity` | `{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`); with `key` and `value` (`""` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes | | |
| 57 | | `capacity` | `{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`) and what fills the disk under the data directory, biggest first; with `key` and `value` (`""` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes | | |
| 58 | 58 | | `claim_link` | `{ name }`: a new claim link for a child you spawned that nobody has claimed; the old one stops working. An admin's | yes | |
| 59 | 59 | | `close` | `{ repo, number, state?, reason?, into? }`: close an issue or a pull (`state` closed by default) or reopen it (`open`); the same rules and refusals as `PATCH …/issues/:number`: an agent closes only what the rules give it and reopens nothing; answers `state` and `closeReason` | yes | |
| 60 | 60 | | `cloud_account` | `{ verb, provider, name, value?, to?, allowed?, repo? }`: change a cloud account: `attach` or `rotate` a token (`value`), `verify` it, `rename` it (`to`), `remove` it, `allow_writes` (`allowed` true or false) or `audit` (`repo` owner/name); a forge admin with an admin-scope token; answers a one-line `says` | yes | |
docs/features/disk-use.md added +25
| @@ -0,0 +1,25 @@ | ||
| 1 | # What fills the disk | |
| 2 | ||
| 3 | When the check machine's disk runs short (the `disk-red` card), `/settings/capacity` | |
| 4 | says why. Its **Disk** card shows how much of the volume under `FORGE_DATA` is free | |
| 5 | and lists the biggest entries at the top of the data directory, biggest first. For | |
| 6 | each entry it also names the biggest few things inside it, such as `crew/.cache` or | |
| 7 | `check-install/<repo>`. | |
| 8 | ||
| 9 | - **Who sees it:** a forge admin, on the page or through | |
| 10 | `GET /api/v1/admin/capacity` (`disk`). An agent sees it through the MCP's | |
| 11 | `capacity` tool. | |
| 12 | - **How it counts:** like `du`, by blocks on the disk. A hard-linked file (an install | |
| 13 | cache and the `node_modules` linked from it) counts once. Symlinks and other | |
| 14 | mounted filesystems are not counted. | |
| 15 | - **When it measures:** in the background (`src/kernel/disk_use.ts`). A request | |
| 16 | never waits for it. The first look starts a walk and says it is measuring. After | |
| 17 | that, a reading older than 30 minutes is measured again on the next look. | |
| 18 | ||
| 19 | ## What filled it (#6947) | |
| 20 | ||
| 21 | Pull #6948's check failed twice with `ENOSPC` while the data volume was still 20% | |
| 22 | free. The full disk was the check cell's `/tmp`, a 256 MiB tmpfs, and the suite's | |
| 23 | throwaway forges had filled it. A check's `/tmp` is now `<checkout>.home/tmp` on | |
| 24 | disk (`CellSpec.tmp`, `src/apps/contain.ts`). It counts toward the check's | |
| 25 | `disk` limit in `forge.json` and is removed with the home when the check ends. |
src/actions/capacity.ts +5 −2
| @@ -4,7 +4,8 @@ | ||
| 4 | 4 | * granted — a crew run that could raise the crew's own concurrency would be |
| 5 | 5 | * the crew deciding how much of the machine it gets. Everyone else is told |
| 6 | 6 | * `not_found`, never that the door exists. Each change is kept and leaves |
| 7 | * an event: the key, from what to what, and who. | |
| 7 | * an event: the key, from what to what, and who. Beside the settings, what | |
| 8 | * fills the disk under the data directory (`kernel/disk_use.ts`). | |
| 8 | 9 | */ |
| 9 | 10 | |
| 10 | 11 | import type { Database } from "bun:sqlite"; |
| @@ -23,6 +24,7 @@ import { | ||
| 23 | 24 | type CapacityKey, |
| 24 | 25 | } from "../core/capacity.ts"; |
| 25 | 26 | import { passkeyIds } from "../core/passkeys.ts"; |
| 27 | import { diskUse, type DiskUse } from "../kernel/disk_use.ts"; | |
| 26 | 28 | import type { User } from "../core/users.ts"; |
| 27 | 29 | import { err, ok, type Result } from "../kernel/result.ts"; |
| 28 | 30 | |
| @@ -60,7 +62,7 @@ export function changedOf(db: Database, key: CapacityKey): { changed?: { at: str | ||
| 60 | 62 | export function listCapacity( |
| 61 | 63 | app: App, |
| 62 | 64 | actor: User | undefined, |
| 63 | ): Result<{ settings: CapacityRow[]; changes: CapacityChange[] }, "not_found"> { | |
| 65 | ): Result<{ settings: CapacityRow[]; changes: CapacityChange[]; disk: DiskUse }, "not_found"> { | |
| 64 | 66 | const who = admin(actor); |
| 65 | 67 | if (!who.ok) return who; |
| 66 | 68 | return ok({ |
| @@ -75,6 +77,7 @@ export function listCapacity( | ||
| 75 | 77 | ...changedOf(app.db, key), |
| 76 | 78 | })), |
| 77 | 79 | changes: capacityChanges(app.db), |
| 80 | disk: diskUse(app.config.dataDir), | |
| 78 | 81 | }); |
| 79 | 82 | } |
| 80 | 83 |
src/apps/contain.ts +17 −2
| @@ -215,6 +215,14 @@ export interface CellSpec { | ||
| 215 | 215 | * (`apps/background.ts`). The step runner sets it. |
| 216 | 216 | */ |
| 217 | 217 | readonly background?: boolean | undefined; |
| 218 | /** | |
| 219 | * A directory on disk mounted as the cell's `/tmp`, in place of the small | |
| 220 | * tmpfs: a check's tests write thousands of throwaway forges there, which | |
| 221 | * filled {@link TMP_MIB} and failed checks for the machine, not the code | |
| 222 | * (#6947). Kept under `home`, so the disk watch counts it and the home's | |
| 223 | * removal takes it. | |
| 224 | */ | |
| 225 | readonly tmp?: string | undefined; | |
| 218 | 226 | } |
| 219 | 227 | |
| 220 | 228 | /** |
| @@ -355,9 +363,15 @@ const NAMESPACES = [ | ||
| 355 | 363 | |
| 356 | 364 | // ---- the cell ---- |
| 357 | 365 | |
| 358 | /** The size of a cell's own `/tmp`, MiB. */ | |
| 366 | /** The size of a cell's own `/tmp`, MiB, unless it brings its own (`CellSpec.tmp`). */ | |
| 359 | 367 | export const TMP_MIB = 256; |
| 360 | 368 | |
| 369 | /** How a cell's `/tmp` is mounted: its own directory on disk, else a bounded tmpfs. */ | |
| 370 | export const tmpMount = (spec: Pick<CellSpec, "tmp">): string[] => | |
| 371 | spec.tmp === undefined | |
| 372 | ? ["--size", String(TMP_MIB * 1024 * 1024), "--tmpfs", "/tmp"] | |
| 373 | : ["--bind", spec.tmp, "/tmp"]; | |
| 374 | ||
| 361 | 375 | /** The proxy's port inside every cell. */ |
| 362 | 376 | const PROXY_PORT = 3128; |
| 363 | 377 | const INSIDE = "/run/forge"; |
| @@ -463,6 +477,7 @@ export function openCell( | ||
| 463 | 477 | const { hide, blank, reveal } = cellMounts(spec, hiddenMounts()); |
| 464 | 478 | const writable = [...new Set([spec.dir, spec.home, ...spec.writable])]; |
| 465 | 479 | for (const path of writable) mkdirSync(path, { recursive: true }); |
| 480 | if (spec.tmp !== undefined) mkdirSync(spec.tmp, { recursive: true }); | |
| 466 | 481 | // tmpfs is memory the watch cannot see from outside, so every one is |
| 467 | 482 | // small and bounded, and the ones that only exist to hide are read-only. |
| 468 | 483 | const SMALL = String(1 << 20); |
| @@ -481,7 +496,7 @@ export function openCell( | ||
| 481 | 496 | ...NAMESPACES, |
| 482 | 497 | ...(spec.terminal === true ? [] : ["--new-session"]), |
| 483 | 498 | "--ro-bind", "/", "/", |
| 484 | "--size", String(TMP_MIB * 1024 * 1024), "--tmpfs", "/tmp", | |
| 499 | ...tmpMount(spec), | |
| 485 | 500 | "--size", SMALL, "--tmpfs", "/run", |
| 486 | 501 | ...hide.flatMap((path) => ["--size", SMALL, "--tmpfs", path]), |
| 487 | 502 | ...blank.flatMap((path) => ["--ro-bind", "/dev/null", path]), |
src/checks/check_cell.ts added +44
| @@ -0,0 +1,44 @@ | ||
| 1 | /** | |
| 2 | * The cell a check runs in: plain data in, the `CellSpec` out, so what a | |
| 3 | * check may see, write and reach is read in one place and tested without a | |
| 4 | * cell. Its HOME sits beside the checkout, and its /tmp inside that HOME on | |
| 5 | * disk, not the cell's small tmpfs (#6947). | |
| 6 | */ | |
| 7 | ||
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { resolveLimits, STEP_HOSTS, type CellSpec } from "../apps/contain.ts"; | |
| 11 | import type { CheckManifest } from "../apps/manifest.ts"; | |
| 12 | ||
| 13 | export interface CheckCellInput { | |
| 14 | /** The checkout the check runs in. */ | |
| 15 | readonly dir: string; | |
| 16 | /** The forge's data directory, hidden inside. */ | |
| 17 | readonly dataDir: string; | |
| 18 | readonly manifest: Pick<CheckManifest, "egress" | "limits">; | |
| 19 | readonly forgeUrl: string; | |
| 20 | readonly egressMode: "enforce" | "report"; | |
| 21 | readonly onUndeclared: (host: string, port: number) => void; | |
| 22 | readonly runId: string; | |
| 23 | } | |
| 24 | ||
| 25 | /** A check's HOME: beside its checkout, removed with it. */ | |
| 26 | export const checkHome = (dir: string): string => `${dir}.home`; | |
| 27 | ||
| 28 | export function checkCell(x: CheckCellInput): CellSpec { | |
| 29 | const home = checkHome(x.dir); | |
| 30 | return { | |
| 31 | dir: x.dir, | |
| 32 | writable: [], | |
| 33 | home, | |
| 34 | // The suite's temp on disk, not the cell's small tmpfs (#6947). | |
| 35 | tmp: join(home, "tmp"), | |
| 36 | hide: [x.dataDir, process.env["HOME"] ?? "", tmpdir()], | |
| 37 | egress: [...STEP_HOSTS, ...x.manifest.egress], | |
| 38 | reach: [x.forgeUrl], | |
| 39 | limits: resolveLimits("step", x.manifest.limits), | |
| 40 | onUndeclared: x.onUndeclared, | |
| 41 | egressMode: x.egressMode, | |
| 42 | label: `check:${x.runId}`, | |
| 43 | }; | |
| 44 | } |
src/checks/service.ts +38 −21
| @@ -13,7 +13,6 @@ | ||
| 13 | 13 | |
| 14 | 14 | import { pullDir, runDir } from "../core/pulls.ts"; |
| 15 | 15 | import { existsSync, mkdirSync, readdirSync, rmSync } from "node:fs"; |
| 16 | import { tmpdir } from "node:os"; | |
| 17 | 16 | import { join } from "node:path"; |
| 18 | 17 | import type { App } from "../app.ts"; |
| 19 | 18 | import { releaseOnGreen } from "../actions/holds.ts"; |
| @@ -87,14 +86,15 @@ import { | ||
| 87 | 86 | cellsAvailable, |
| 88 | 87 | limitWarnings, |
| 89 | 88 | resolveLimits, |
| 90 | STEP_HOSTS, | |
| 91 | type CellSpec, | |
| 92 | 89 | } from "../apps/contain.ts"; |
| 90 | import { checkCell, checkHome } from "./check_cell.ts"; | |
| 93 | 91 | import { summariseCheckFailure } from "./summary.ts"; |
| 94 | 92 | import type { Result } from "../kernel/result.ts"; |
| 95 | 93 | import { placementFromEnv, type Placement } from "./placement.ts"; |
| 96 | 94 | import { JOB_VANISHED, runJobStep } from "./job_step.ts"; |
| 97 | 95 | import { checkInstallCache, installWithRetry, withLock } from "./install.ts"; |
| 96 | import { boundCache } from "../kernel/cache_bound.ts"; | |
| 97 | import { bytes } from "../system/format.ts"; | |
| 98 | 98 | import { checkStage, type GateStage } from "./gate_reason.ts"; |
| 99 | 99 | import { extraGates, gateQueued, gateRunId, gatesInFlight, watchGates } from "./queue.ts"; |
| 100 | 100 | import type { JobOptions } from "../cluster/jobs.ts"; |
| @@ -290,17 +290,27 @@ export function requeueAbandonedChecks(app: App, staleMs: number): number { | ||
| 290 | 290 | /** Where a check's checkout lives while it runs. */ |
| 291 | 291 | const checksRoot = (dataDir: string): string => join(dataDir, "checks"); |
| 292 | 292 | |
| 293 | /** | |
| 294 | * What the checks keep beside their checkouts, never swept: the gate logs | |
| 295 | * (`gate_log.ts`), learned files (`learned.ts`), impact maps | |
| 296 | * (`impact-store.ts`) and the last gate (`gate_last.ts`). | |
| 297 | */ | |
| 298 | const KEPT = new Set(["gates", "learned", "impact", "gate-last.json"]); | |
| 299 | ||
| 293 | 300 | /** |
| 294 | 301 | * Remove the checkouts nobody is using: one whose run is not running with a |
| 295 | * live lease, or has no run at all. A forge booting during a roll shares this | |
| 296 | * directory and database with the old one, whose checks are still running in | |
| 297 | * theirs; those are left alone. | |
| 302 | * live lease, or has no run at all, and its cell's home with it. A forge | |
| 303 | * booting during a roll shares this directory and database with the old one, | |
| 304 | * whose checks are still running in theirs; those are left alone, and so is | |
| 305 | * what the checks keep here ({@link KEPT}). | |
| 298 | 306 | */ |
| 299 | 307 | export function sweepCheckouts(app: App, now: number = Date.now()): void { |
| 300 | 308 | const root = checksRoot(app.config.dataDir); |
| 301 | 309 | if (!existsSync(root)) return; |
| 302 | 310 | for (const name of readdirSync(root)) { |
| 303 | const run = getRun(app.db, name); | |
| 311 | if (KEPT.has(name)) continue; | |
| 312 | // A cell's home (`<run id>.home`) lives and goes with its run's checkout. | |
| 313 | const run = getRun(app.db, name.endsWith(".home") ? name.slice(0, -".home".length) : name); | |
| 304 | 314 | const live = |
| 305 | 315 | run !== undefined && |
| 306 | 316 | run.state === "running" && |
| @@ -912,21 +922,18 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 912 | 922 | }); |
| 913 | 923 | write("placement: job"); |
| 914 | 924 | } |
| 915 | const home = `${dir}.home`; | |
| 916 | const cell: CellSpec | undefined = cells?.ok | |
| 917 | ? { | |
| 925 | const home = checkHome(dir); | |
| 926 | const cell = cells?.ok | |
| 927 | ? checkCell({ | |
| 918 | 928 | dir, |
| 919 | writable: [], | |
| 920 | home, | |
| 921 | hide: [app.config.dataDir, process.env["HOME"] ?? "", tmpdir()], | |
| 922 | egress: [...STEP_HOSTS, ...manifest.egress], | |
| 923 | reach: [options.forgeUrl], | |
| 924 | limits: resolveLimits("step", manifest.limits), | |
| 929 | dataDir: app.config.dataDir, | |
| 930 | manifest, | |
| 931 | forgeUrl: options.forgeUrl, | |
| 932 | egressMode: egressSetting(app.db), | |
| 925 | 933 | onUndeclared: (host, port) => |
| 926 | 934 | noteReached(app.config.dataDir, repo.id, host, port), |
| 927 | egressMode: egressSetting(app.db), | |
| 928 | label: `check:${run.id}`, | |
| 929 | } | |
| 935 | runId: run.id, | |
| 936 | }) | |
| 930 | 937 | : undefined; |
| 931 | 938 | try { |
| 932 | 939 | for (const line of limitWarnings(manifest.limits)) write(line); |
| @@ -992,8 +999,18 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 992 | 999 | const installed = await installWithRetry( |
| 993 | 1000 | async (log) => { |
| 994 | 1001 | if (shared === undefined) return once(log); |
| 995 | mkdirSync(shared, { recursive: true }); | |
| 996 | const held = await withLock(`${shared}.lock`, () => once(log), { signal }); | |
| 1002 | // Emptied under the lock, so no other install is reading it. | |
| 1003 | const held = await withLock( | |
| 1004 | `${shared}.lock`, | |
| 1005 | async () => { | |
| 1006 | const freed = await boundCache(shared); | |
| 1007 | if (freed !== undefined) | |
| 1008 | log(`install cache past its ceiling: emptied ${bytes(freed)} first`); | |
| 1009 | mkdirSync(shared, { recursive: true }); | |
| 1010 | return once(log); | |
| 1011 | }, | |
| 1012 | { signal }, | |
| 1013 | ); | |
| 997 | 1014 | return held.ok ? held.value : held; |
| 998 | 1015 | }, |
| 999 | 1016 | { log: write, signal, retryMs: options.installRetryMs }, |
src/contract/openapi/routes/getApiV1AdminCapacity.ts +1 −1
| @@ -4,7 +4,7 @@ export const route: Route = { | ||
| 4 | 4 | method: "get", |
| 5 | 5 | path: "/api/v1/admin/capacity", |
| 6 | 6 | summary: |
| 7 | "Read the forge's capacity settings (a person admin's token, any scope); answers { items: [{ key, label, help, value, source, default, env, changed? }], changes, next_cursor }", | |
| 7 | "Read the forge's capacity settings (a person admin's token, any scope); answers { items: [{ key, label, help, value, source, default, env, changed? }], changes, disk: { disk: { freeBytes, totalBytes }, measuredAt?, measuring, consumers: [{ path, bytes, parts: [{ path, bytes }] }] }, next_cursor }", | |
| 8 | 8 | auth: true, |
| 9 | 9 | refusals: { not_found: "the token is not a person who administers the forge" }, |
| 10 | 10 | }; |
src/crew/workdir.ts +28 −6
| @@ -9,6 +9,9 @@ import { rm } from "node:fs/promises"; | ||
| 9 | 9 | import { dirname, join } from "node:path"; |
| 10 | 10 | import type { Agent } from "../core/agents.ts"; |
| 11 | 11 | import type { Repo } from "../core/repos.ts"; |
| 12 | import { withLock } from "../checks/install.ts"; | |
| 13 | import { boundCache, INSTALL_CACHE_MAX_BYTES } from "../kernel/cache_bound.ts"; | |
| 14 | import { bytes } from "../system/format.ts"; | |
| 12 | 15 | |
| 13 | 16 | export const crewRoot = (dataDir: string): string => join(dataDir, "crew"); |
| 14 | 17 | |
| @@ -146,10 +149,14 @@ const bunInstall: InstallRunner = (dir, env) => | ||
| 146 | 149 | export function installNote(result: { |
| 147 | 150 | installed: boolean; |
| 148 | 151 | error?: string; |
| 152 | emptied?: string; | |
| 149 | 153 | }): string | undefined { |
| 150 | 154 | if (result.error !== undefined) |
| 151 | 155 | return `dependency install failed: ${result.error}`; |
| 152 | return result.installed ? "dependencies installed" : undefined; | |
| 156 | if (!result.installed) return undefined; | |
| 157 | return result.emptied === undefined | |
| 158 | ? "dependencies installed" | |
| 159 | : `dependencies installed; the install cache was past its ceiling, so its ${result.emptied} were emptied first`; | |
| 153 | 160 | } |
| 154 | 161 | |
| 155 | 162 | /** |
| @@ -158,12 +165,17 @@ export function installNote(result: { | ||
| 158 | 165 | * package.json and a lockfile is installed; a workdir whose node_modules was |
| 159 | 166 | * installed for the same lockfile hash is left alone. Never throws: a failure |
| 160 | 167 | * comes back as a message for the log, and the run starts regardless. |
| 168 | * | |
| 169 | * The cache is shared by every crew run on the repository, so it is bounded | |
| 170 | * and installed into under the lock the checks hold on theirs | |
| 171 | * (`checks/install.ts`): one run never empties it under another's install. | |
| 161 | 172 | */ |
| 162 | 173 | export async function installDependencies( |
| 163 | 174 | dir: string, |
| 164 | 175 | cacheDir: string, |
| 165 | 176 | run: InstallRunner = bunInstall, |
| 166 | ): Promise<{ installed: boolean; error?: string }> { | |
| 177 | bound: { readonly max?: number; readonly now?: number } = {}, | |
| 178 | ): Promise<{ installed: boolean; error?: string; emptied?: string }> { | |
| 167 | 179 | try { |
| 168 | 180 | const lock = LOCKFILES.map((name) => join(dir, name)).find((p) => |
| 169 | 181 | existsSync(p), |
| @@ -177,9 +189,19 @@ export async function installDependencies( | ||
| 177 | 189 | if (existsSync(mark) && readFileSync(mark, "utf8") === hash) |
| 178 | 190 | return { installed: false }; |
| 179 | 191 | const env = installEnv(cacheDir); |
| 180 | mkdirSync(cacheDir, { recursive: true }); | |
| 181 | mkdirSync(env["HOME"]!, { recursive: true }); | |
| 182 | const result = await run(dir, env); | |
| 192 | const held = await withLock( | |
| 193 | `${cacheDir}.lock`, | |
| 194 | async () => { | |
| 195 | // A cache past its ceiling starts over (`kernel/cache_bound.ts`). | |
| 196 | const freed = await boundCache(cacheDir, bound.max ?? INSTALL_CACHE_MAX_BYTES, bound.now); | |
| 197 | mkdirSync(cacheDir, { recursive: true }); | |
| 198 | mkdirSync(env["HOME"]!, { recursive: true }); | |
| 199 | return { freed, result: await run(dir, env) }; | |
| 200 | }, | |
| 201 | { signal: new AbortController().signal }, | |
| 202 | ); | |
| 203 | if (!held.ok) return { installed: false, error: held.error }; | |
| 204 | const { freed, result } = held.value; | |
| 183 | 205 | if (result.code !== 0) |
| 184 | 206 | return { |
| 185 | 207 | installed: false, |
| @@ -187,7 +209,7 @@ export async function installDependencies( | ||
| 187 | 209 | }; |
| 188 | 210 | mkdirSync(join(dir, "node_modules"), { recursive: true }); |
| 189 | 211 | writeFileSync(mark, hash); |
| 190 | return { installed: true }; | |
| 212 | return freed === undefined ? { installed: true } : { installed: true, emptied: bytes(freed) }; | |
| 191 | 213 | } catch (error) { |
| 192 | 214 | return { installed: false, error: String(error) }; |
| 193 | 215 | } |
src/http/api.ts +1 −1
| @@ -1679,7 +1679,7 @@ export function apiRoutes(router: Router<Handler>): void { | ||
| 1679 | 1679 | const denied = accountAdmin(ctx, "ro"); |
| 1680 | 1680 | if (denied) return denied; |
| 1681 | 1681 | const listed = timePhase(ctx.request, "capacity", () => listCapacity(ctx.app, ctx.auth.user)); |
| 1682 | return timePhase(ctx.request, "render", () => (listed.ok ? json({ items: listed.value.settings, changes: listed.value.changes, next_cursor: null }) : notFound())); | |
| 1682 | return timePhase(ctx.request, "render", () => (listed.ok ? json({ items: listed.value.settings, changes: listed.value.changes, disk: listed.value.disk, next_cursor: null }) : notFound())); | |
| 1683 | 1683 | }); |
| 1684 | 1684 | |
| 1685 | 1685 | router.add("PUT", "/api/v1/admin/capacity/:key", async (ctx) => { |
src/http/mcp/tools/capacity.ts +1 −1
| @@ -5,7 +5,7 @@ import { good, bad, type Tool } from "../shared.ts"; | ||
| 5 | 5 | |
| 6 | 6 | export const tool: Tool = { |
| 7 | 7 | name: "capacity", |
| 8 | doc: "`{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`); with `key` and `value` (`\"\"` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token", | |
| 8 | doc: "`{ key?, value?, dryRun? }` (forge admin): the forge's capacity and guards, live (crew and checks concurrency, check shards, cell egress, the new-account guard, the sign-in edge, queue depth, child forges; only the first six are settable by `key`) and what fills the disk under the data directory, biggest first; with `key` and `value` (`\"\"` restores the default), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token", | |
| 9 | 9 | description: |
| 10 | 10 | "Forge capacity and guards, live. With `key` and `value` (\"\" restores default), plans.", |
| 11 | 11 | inputSchema: { |
src/kernel/cache_bound.ts added +63
| @@ -0,0 +1,63 @@ | ||
| 1 | /** | |
| 2 | * An install cache with a ceiling (#6947). Bun's cache keeps every version | |
| 3 | * of every package it ever fetched and never forgets one, so a repository's | |
| 4 | * cache — `crew/.cache/<repo>/bun` for the crew, `check-install/<repo>/bun` | |
| 5 | * for uncontained checks — only grows. Before an install the cache is | |
| 6 | * measured, at most once per {@link BOUND_EVERY_MS}, and one over | |
| 7 | * {@link INSTALL_CACHE_MAX_BYTES} is emptied whole: the install fetches what | |
| 8 | * it needs again, and the cache starts over holding only what is in use. | |
| 9 | * | |
| 10 | * The cache is renamed aside before it is removed, so a reader that holds | |
| 11 | * no lock — an agent's own `bun install` in its sandbox — finds either the | |
| 12 | * whole cache or none, never one half-deleted. Callers that install empty | |
| 13 | * it under the install lock as well (`checks/install.ts` `withLock`). | |
| 14 | */ | |
| 15 | ||
| 16 | import { readdirSync } from "node:fs"; | |
| 17 | import { rename, rm } from "node:fs/promises"; | |
| 18 | import { basename, dirname, join } from "node:path"; | |
| 19 | import { diskBytes } from "./disk_use.ts"; | |
| 20 | ||
| 21 | /** A repository's install cache over this is emptied before the next install. */ | |
| 22 | export const INSTALL_CACHE_MAX_BYTES = 4 * 1024 ** 3; | |
| 23 | /** A cache is measured no oftener than this; the walk is not free. */ | |
| 24 | export const BOUND_EVERY_MS = 3_600_000; | |
| 25 | ||
| 26 | const measuredAt = new Map<string, number>(); | |
| 27 | ||
| 28 | /** | |
| 29 | * Empty `dir` when it holds more than `max` bytes. Answers the bytes freed, | |
| 30 | * or undefined when it was under, or measured too lately to look again. | |
| 31 | */ | |
| 32 | export async function boundCache( | |
| 33 | dir: string, | |
| 34 | max: number = INSTALL_CACHE_MAX_BYTES, | |
| 35 | now: number = Date.now(), | |
| 36 | ): Promise<number | undefined> { | |
| 37 | const last = measuredAt.get(dir); | |
| 38 | if (last !== undefined && now - last < BOUND_EVERY_MS) return undefined; | |
| 39 | measuredAt.set(dir, now); | |
| 40 | const bytes = await diskBytes(dir); | |
| 41 | if (bytes <= max) return undefined; | |
| 42 | const aside = `${dir}${ASIDE}${process.pid}-${now}`; | |
| 43 | await rename(dir, aside); | |
| 44 | await rm(aside, { recursive: true, force: true }); | |
| 45 | await removeLeftAside(dir); | |
| 46 | return bytes; | |
| 47 | } | |
| 48 | ||
| 49 | const ASIDE = ".emptied-"; | |
| 50 | ||
| 51 | /** Removes what an earlier emptying renamed aside and did not live to remove. */ | |
| 52 | async function removeLeftAside(dir: string): Promise<void> { | |
| 53 | const prefix = `${basename(dir)}${ASIDE}`; | |
| 54 | let names: string[]; | |
| 55 | try { | |
| 56 | names = readdirSync(dirname(dir)); | |
| 57 | } catch { | |
| 58 | return; | |
| 59 | } | |
| 60 | for (const name of names) | |
| 61 | if (name.startsWith(prefix)) | |
| 62 | await rm(join(dirname(dir), name), { recursive: true, force: true }); | |
| 63 | } |
src/kernel/disk_use.ts added +213
| @@ -0,0 +1,213 @@ | ||
| 1 | /** | |
| 2 | * What fills the disk under `FORGE_DATA` (#6947): each entry at its top, | |
| 3 | * biggest first, with the biggest few inside each — `crew 40 GiB, of which | |
| 4 | * crew/.cache 31 GiB`. The free figure (`statDisk`) says the disk is full; | |
| 5 | * this says by what. | |
| 6 | * | |
| 7 | * Counted as `du` counts: blocks on the disk, not apparent sizes, a file | |
| 8 | * linked in twice (an install cache and the `node_modules` linked from it) | |
| 9 | * once, symlinks never followed, and another filesystem mounted inside not | |
| 10 | * at all — none of those fill this disk. | |
| 11 | * | |
| 12 | * Walking a large data directory takes long, so the reading is kept in | |
| 13 | * memory and a request never waits for it: the first ask starts a walk in | |
| 14 | * the background and answers what it has, and a reading older than | |
| 15 | * {@link DISK_USE_FRESH_MS} starts the next. The disk is the truth; a | |
| 16 | * restart measures again. | |
| 17 | */ | |
| 18 | ||
| 19 | import { lstat, readdir } from "node:fs/promises"; | |
| 20 | import { join } from "node:path"; | |
| 21 | import { createLog } from "./log.ts"; | |
| 22 | import { statDisk, type DiskReading } from "./vitals.ts"; | |
| 23 | ||
| 24 | const log = createLog("disk-use"); | |
| 25 | ||
| 26 | /** A reading this old is measured again on the next ask. */ | |
| 27 | export const DISK_USE_FRESH_MS = 30 * 60_000; | |
| 28 | /** How many top-level entries are named, and how many parts inside each. */ | |
| 29 | export const TOP_ENTRIES = 8; | |
| 30 | export const TOP_PARTS = 3; | |
| 31 | ||
| 32 | export interface DiskPart { | |
| 33 | /** Relative to the data directory, `/`-separated: `crew/.cache`. */ | |
| 34 | readonly path: string; | |
| 35 | readonly bytes: number; | |
| 36 | } | |
| 37 | ||
| 38 | export interface DiskConsumer extends DiskPart { | |
| 39 | /** The biggest entries inside it, biggest first; none for a file. */ | |
| 40 | readonly parts: readonly DiskPart[]; | |
| 41 | } | |
| 42 | ||
| 43 | export interface DiskUse { | |
| 44 | readonly disk: DiskReading; | |
| 45 | /** When the consumers were measured; absent until the first walk ends. */ | |
| 46 | readonly measuredAt?: string; | |
| 47 | /** Whether a walk is under way now. */ | |
| 48 | readonly measuring: boolean; | |
| 49 | /** The biggest entries under the data directory, biggest first. */ | |
| 50 | readonly consumers: readonly DiskConsumer[]; | |
| 51 | } | |
| 52 | ||
| 53 | interface Walk { | |
| 54 | readonly dev: number; | |
| 55 | /** Inodes already counted, for files with more than one link. */ | |
| 56 | readonly seen: Set<string>; | |
| 57 | } | |
| 58 | ||
| 59 | /** Blocks a stat says the entry holds, counting a multiply-linked file once. */ | |
| 60 | function held(walk: Walk, info: Awaited<ReturnType<typeof lstat>>): number { | |
| 61 | if (info.isDirectory() || info.nlink <= 1) return Number(info.blocks) * 512; | |
| 62 | const key = `${info.dev}:${info.ino}`; | |
| 63 | if (walk.seen.has(key)) return 0; | |
| 64 | walk.seen.add(key); | |
| 65 | return Number(info.blocks) * 512; | |
| 66 | } | |
| 67 | ||
| 68 | /** The bytes under `path`, on the walk's own device only. */ | |
| 69 | async function treeBytes(walk: Walk, path: string): Promise<number> { | |
| 70 | let total = 0; | |
| 71 | const pending = [path]; | |
| 72 | while (pending.length > 0) { | |
| 73 | const dir = pending.pop()!; | |
| 74 | let names: string[]; | |
| 75 | try { | |
| 76 | names = await readdir(dir); | |
| 77 | } catch { | |
| 78 | continue; // gone, or not ours to read: nothing to count | |
| 79 | } | |
| 80 | for (const name of names) { | |
| 81 | const child = join(dir, name); | |
| 82 | try { | |
| 83 | const info = await lstat(child); | |
| 84 | if (Number(info.dev) !== walk.dev) continue; | |
| 85 | total += held(walk, info); | |
| 86 | if (info.isDirectory()) pending.push(child); | |
| 87 | } catch { | |
| 88 | // removed while walking | |
| 89 | } | |
| 90 | } | |
| 91 | } | |
| 92 | return total; | |
| 93 | } | |
| 94 | ||
| 95 | /** The bytes `path` holds on its own disk, counted as the walk counts them. */ | |
| 96 | export async function diskBytes(path: string): Promise<number> { | |
| 97 | let info; | |
| 98 | try { | |
| 99 | info = await lstat(path); | |
| 100 | } catch { | |
| 101 | return 0; | |
| 102 | } | |
| 103 | const walk: Walk = { dev: Number(info.dev), seen: new Set() }; | |
| 104 | return held(walk, info) + (info.isDirectory() ? await treeBytes(walk, path) : 0); | |
| 105 | } | |
| 106 | ||
| 107 | /** One entry and the parts inside it, each measured whole. */ | |
| 108 | async function consumer(walk: Walk, root: string, name: string): Promise<DiskConsumer | undefined> { | |
| 109 | const path = join(root, name); | |
| 110 | let info; | |
| 111 | try { | |
| 112 | info = await lstat(path); | |
| 113 | } catch { | |
| 114 | return undefined; | |
| 115 | } | |
| 116 | if (Number(info.dev) !== walk.dev) return undefined; | |
| 117 | const own = held(walk, info); | |
| 118 | if (!info.isDirectory()) return { path: name, bytes: own, parts: [] }; | |
| 119 | let names: string[]; | |
| 120 | try { | |
| 121 | names = await readdir(path); | |
| 122 | } catch { | |
| 123 | return { path: name, bytes: own, parts: [] }; | |
| 124 | } | |
| 125 | const parts: DiskPart[] = []; | |
| 126 | let bytes = own; | |
| 127 | for (const inner of names.sort()) { | |
| 128 | const at = join(path, inner); | |
| 129 | let part; | |
| 130 | try { | |
| 131 | part = await lstat(at); | |
| 132 | } catch { | |
| 133 | continue; | |
| 134 | } | |
| 135 | if (Number(part.dev) !== walk.dev) continue; | |
| 136 | const size = held(walk, part) + (part.isDirectory() ? await treeBytes(walk, at) : 0); | |
| 137 | bytes += size; | |
| 138 | parts.push({ path: `${name}/${inner}`, bytes: size }); | |
| 139 | } | |
| 140 | return { path: name, bytes, parts: biggest(parts).slice(0, TOP_PARTS) }; | |
| 141 | } | |
| 142 | ||
| 143 | const biggest = <T extends DiskPart>(list: T[]): T[] => | |
| 144 | list.sort((a, b) => b.bytes - a.bytes || (a.path < b.path ? -1 : 1)); | |
| 145 | ||
| 146 | /** Walk `dataDir` now: its biggest entries, biggest first. */ | |
| 147 | export async function measureDiskUse(dataDir: string): Promise<DiskConsumer[]> { | |
| 148 | let names: string[]; | |
| 149 | let dev: number; | |
| 150 | try { | |
| 151 | dev = Number((await lstat(dataDir)).dev); | |
| 152 | names = await readdir(dataDir); | |
| 153 | } catch { | |
| 154 | return []; | |
| 155 | } | |
| 156 | const walk: Walk = { dev, seen: new Set() }; | |
| 157 | const found: DiskConsumer[] = []; | |
| 158 | for (const name of names.sort()) { | |
| 159 | const entry = await consumer(walk, dataDir, name); | |
| 160 | if (entry !== undefined) found.push(entry); | |
| 161 | } | |
| 162 | return biggest(found).slice(0, TOP_ENTRIES); | |
| 163 | } | |
| 164 | ||
| 165 | interface Kept { | |
| 166 | consumers: DiskConsumer[]; | |
| 167 | measuredAt?: number; | |
| 168 | walking?: Promise<DiskConsumer[]>; | |
| 169 | } | |
| 170 | ||
| 171 | const kept = new Map<string, Kept>(); | |
| 172 | ||
| 173 | /** Start a walk of `dataDir` unless one is under way; resolves when it ends. */ | |
| 174 | export function refreshDiskUse(dataDir: string): Promise<DiskConsumer[]> { | |
| 175 | let entry = kept.get(dataDir); | |
| 176 | if (entry === undefined) kept.set(dataDir, (entry = { consumers: [] })); | |
| 177 | const into = entry; | |
| 178 | into.walking ??= measureDiskUse(dataDir) | |
| 179 | .then((consumers) => { | |
| 180 | into.consumers = consumers; | |
| 181 | into.measuredAt = Date.now(); | |
| 182 | return consumers; | |
| 183 | }) | |
| 184 | .catch((error: unknown) => { | |
| 185 | log.warn("disk use not measured", { error: String(error) }); | |
| 186 | return into.consumers; | |
| 187 | }) | |
| 188 | .finally(() => { | |
| 189 | delete into.walking; | |
| 190 | }); | |
| 191 | return into.walking; | |
| 192 | } | |
| 193 | ||
| 194 | /** | |
| 195 | * What fills the disk under `dataDir`, as last measured, never waiting: a | |
| 196 | * missing or stale reading starts a walk and answers what it has. | |
| 197 | */ | |
| 198 | export function diskUse( | |
| 199 | dataDir: string, | |
| 200 | now: number = Date.now(), | |
| 201 | read: (dir: string) => DiskReading = statDisk, | |
| 202 | ): DiskUse { | |
| 203 | const entry = kept.get(dataDir); | |
| 204 | if (entry?.measuredAt === undefined || now - entry.measuredAt >= DISK_USE_FRESH_MS) | |
| 205 | void refreshDiskUse(dataDir); | |
| 206 | const after = kept.get(dataDir)!; | |
| 207 | return { | |
| 208 | disk: read(dataDir), | |
| 209 | ...(after.measuredAt === undefined ? {} : { measuredAt: new Date(after.measuredAt).toISOString() }), | |
| 210 | measuring: after.walking !== undefined, | |
| 211 | consumers: after.consumers, | |
| 212 | }; | |
| 213 | } |
src/web/pages/capacity.ts +50 −2
| @@ -4,10 +4,13 @@ | ||
| 4 | 4 | * refused or reported (`core/capacity.ts`). A change here is honoured by |
| 5 | 5 | * the next run, check or cell — never by a restart, which is the point — |
| 6 | 6 | * and each one is listed below the settings: who, when, from what to what. |
| 7 | * Above them, what fills the disk (`kernel/disk_use.ts`): the page the | |
| 8 | * disk-red card links to names the biggest consumers. | |
| 7 | 9 | */ |
| 8 | 10 | |
| 9 | 11 | import { changeCapacity, listCapacity, type CapacityRow } from "../../actions/capacity.ts"; |
| 10 | 12 | import { CAPACITY, type CapacityKey } from "../../core/capacity.ts"; |
| 13 | import type { DiskUse } from "../../kernel/disk_use.ts"; | |
| 11 | 14 | import { bytes } from "../../system/format.ts"; |
| 12 | 15 | import { egressSnapshot } from "../../apps/egress.ts"; |
| 13 | 16 | import type { Ctx } from "../../http/context.ts"; |
| @@ -67,6 +70,51 @@ function input(key: CapacityKey, value: string) { | ||
| 67 | 70 | />`; |
| 68 | 71 | } |
| 69 | 72 | |
| 73 | /** "12% of the disk" for a share of the volume; "" when it reports no size. */ | |
| 74 | const share = (bytesOf: number, total: number): string => | |
| 75 | total > 0 ? `${Math.round((bytesOf / total) * 100)}% of the disk` : ""; | |
| 76 | ||
| 77 | /** The disk's free figure and its biggest consumers, biggest first. */ | |
| 78 | function diskCard(use: DiskUse) { | |
| 79 | const { freeBytes, totalBytes } = use.disk; | |
| 80 | const free = | |
| 81 | totalBytes > 0 | |
| 82 | ? `${bytes(freeBytes)} free of ${bytes(totalBytes)} (${Math.round((freeBytes / totalBytes) * 100)}%).` | |
| 83 | : "The volume reports no size."; | |
| 84 | const measured = | |
| 85 | use.measuredAt === undefined | |
| 86 | ? html`Measuring what fills it now; reload in a minute.` | |
| 87 | : html`What fills it, biggest first, measured ${time(use.measuredAt)}${use.measuring ? "; measuring again now" : ""}.`; | |
| 88 | return card( | |
| 89 | html`<div class="card-body"> | |
| 90 | <p class="muted small" id="disk-free">${free} ${measured}</p> | |
| 91 | </div> | |
| 92 | ${when( | |
| 93 | use.consumers.length > 0, | |
| 94 | () => html`<ul class="list" id="disk-use"> | |
| 95 | ${use.consumers.map((c) => | |
| 96 | row({ | |
| 97 | lead: icon("activity"), | |
| 98 | title: html`<span class="mono">${c.path}</span>`, | |
| 99 | meta: html`${share(c.bytes, totalBytes)}${c.parts.length === 0 | |
| 100 | ? "" | |
| 101 | : html`${totalBytes > 0 ? "; " : ""}biggest inside: | |
| 102 | ${c.parts.map( | |
| 103 | (p, i) => html`${i === 0 ? "" : ", "}<span class="mono">${p.path}</span> ${bytes(p.bytes)}`, | |
| 104 | )}`}`, | |
| 105 | trail: html`<span class="mono">${bytes(c.bytes)}</span>`, | |
| 106 | }), | |
| 107 | )} | |
| 108 | </ul>`, | |
| 109 | )}`, | |
| 110 | { | |
| 111 | head: html`${icon("activity")} | |
| 112 | <h2>Disk</h2>`, | |
| 113 | pad: false, | |
| 114 | }, | |
| 115 | ); | |
| 116 | } | |
| 117 | ||
| 70 | 118 | // GET /settings/capacity |
| 71 | 119 | export function capacityPage(ctx: Ctx): Response { |
| 72 | 120 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); |
| @@ -76,7 +124,7 @@ export function capacityPage(ctx: Ctx): Response { | ||
| 76 | 124 | listCapacity(ctx.app, ctx.auth.user), |
| 77 | 125 | ); |
| 78 | 126 | if (!listed.ok) return notFoundPage(ctx); |
| 79 | const { settings, changes } = listed.value; | |
| 127 | const { settings, changes, disk } = listed.value; | |
| 80 | 128 | const errs = timePhase(ctx.request, "errors", () => |
| 81 | 129 | ctx.app.signals.errors.recent(10), |
| 82 | 130 | ); |
| @@ -182,7 +230,7 @@ export function capacityPage(ctx: Ctx): Response { | ||
| 182 | 230 | pad: false, |
| 183 | 231 | }, |
| 184 | 232 | ); |
| 185 | const body = html`${happened}${connections}${when( | |
| 233 | const body = html`${diskCard(disk)}${happened}${connections}${when( | |
| 186 | 234 | reported, |
| 187 | 235 | () => html`<p class="muted small mb" id="egress-reported"> |
| 188 | 236 | ${icon("activity")} Egress is reported, not enforced: every new app, |
test/cache-bound.test.ts added +118
| @@ -0,0 +1,118 @@ | ||
| 1 | /** | |
| 2 | * An install cache has a ceiling (#6947): bun never forgets a package, so a | |
| 3 | * repository's cache only grew. One over its ceiling is emptied before the | |
| 4 | * next install, one under it is left alone, and it is measured at most once | |
| 5 | * an hour; the crew's install says in the run log when it emptied one. | |
| 6 | */ | |
| 7 | ||
| 8 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 9 | import { existsSync, mkdirSync, readdirSync, writeFileSync } from "node:fs"; | |
| 10 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 11 | import { tmpdir } from "node:os"; | |
| 12 | import { join } from "node:path"; | |
| 13 | import { BOUND_EVERY_MS, boundCache } from "../src/kernel/cache_bound.ts"; | |
| 14 | import { installDependencies, installNote } from "../src/crew/workdir.ts"; | |
| 15 | ||
| 16 | let root = ""; | |
| 17 | ||
| 18 | beforeAll(async () => { | |
| 19 | root = await mkdtemp(join(tmpdir(), "forge-cache-bound-")); | |
| 20 | }); | |
| 21 | ||
| 22 | afterAll(async () => { | |
| 23 | await rm(root, { recursive: true, force: true }); | |
| 24 | }); | |
| 25 | ||
| 26 | const cacheWith = (name: string, n: number): string => { | |
| 27 | const dir = join(root, name, "bun"); | |
| 28 | mkdirSync(dir, { recursive: true }); | |
| 29 | writeFileSync(join(dir, "pkg.tgz"), new Uint8Array(n).fill(7)); | |
| 30 | return dir; | |
| 31 | }; | |
| 32 | ||
| 33 | test("a cache over its ceiling is emptied, one under it kept", async () => { | |
| 34 | const over = cacheWith("over", 256 * 1024); | |
| 35 | const under = cacheWith("under", 1024); | |
| 36 | expect(await boundCache(over, 64 * 1024)).toBeGreaterThanOrEqual(256 * 1024); | |
| 37 | expect(existsSync(over)).toBe(false); | |
| 38 | expect(await boundCache(under, 64 * 1024)).toBeUndefined(); | |
| 39 | expect(existsSync(join(under, "pkg.tgz"))).toBe(true); | |
| 40 | }); | |
| 41 | ||
| 42 | test("a cache is measured at most once an hour", async () => { | |
| 43 | const now = Date.now(); | |
| 44 | const dir = cacheWith("hourly", 1024); | |
| 45 | expect(await boundCache(dir, 64 * 1024, now)).toBeUndefined(); | |
| 46 | writeFileSync(join(dir, "more.tgz"), new Uint8Array(256 * 1024).fill(7)); | |
| 47 | expect(await boundCache(dir, 64 * 1024, now + 60_000)).toBeUndefined(); | |
| 48 | expect(existsSync(dir)).toBe(true); | |
| 49 | expect(await boundCache(dir, 64 * 1024, now + BOUND_EVERY_MS)).toBeGreaterThan(0); | |
| 50 | expect(existsSync(dir)).toBe(false); | |
| 51 | }); | |
| 52 | ||
| 53 | test("the crew's install keeps a cache under the ceiling, and its note names one emptied", async () => { | |
| 54 | const work = join(root, "work"); | |
| 55 | mkdirSync(work, { recursive: true }); | |
| 56 | writeFileSync(join(work, "package.json"), "{}"); | |
| 57 | writeFileSync(join(work, "bun.lock"), "{}"); | |
| 58 | const cache = cacheWith("crew", 1024); | |
| 59 | const result = await installDependencies(work, cache, async () => ({ code: 0, stderr: "" })); | |
| 60 | expect(result).toEqual({ installed: true }); | |
| 61 | expect(existsSync(join(cache, "pkg.tgz"))).toBe(true); | |
| 62 | expect(installNote({ installed: true, emptied: "4.2 GiB" })).toContain("emptied"); | |
| 63 | expect(installNote({ installed: true })).toBe("dependencies installed"); | |
| 64 | }); | |
| 65 | ||
| 66 | test("an emptied cache leaves nothing aside: renamed first, then removed", async () => { | |
| 67 | const over = cacheWith("aside", 256 * 1024); | |
| 68 | const parent = join(over, ".."); | |
| 69 | mkdirSync(join(parent, "bun.emptied-1-1"), { recursive: true }); | |
| 70 | expect(await boundCache(over, 64 * 1024)).toBeGreaterThan(0); | |
| 71 | expect(readdirSync(parent).filter((n) => n.startsWith("bun."))).toEqual([]); | |
| 72 | }); | |
| 73 | ||
| 74 | test("two crew installs on one cache: the one that empties it waits for the other's install to finish", async () => { | |
| 75 | const workdir = (name: string): string => { | |
| 76 | const work = join(root, name); | |
| 77 | mkdirSync(work, { recursive: true }); | |
| 78 | writeFileSync(join(work, "package.json"), "{}"); | |
| 79 | writeFileSync(join(work, "bun.lock"), name); | |
| 80 | return work; | |
| 81 | }; | |
| 82 | const cache = cacheWith("shared", 1024); | |
| 83 | const max = 64 * 1024; | |
| 84 | const now = Date.now(); | |
| 85 | let started = (): void => {}; | |
| 86 | const first = new Promise<void>((resolve) => (started = resolve)); | |
| 87 | let release = (): void => {}; | |
| 88 | const gate = new Promise<void>((resolve) => (release = resolve)); | |
| 89 | // The first install fills the cache past its ceiling, then reads back | |
| 90 | // what it fetched while the second run is asking to empty it. | |
| 91 | const reading = installDependencies( | |
| 92 | workdir("reading"), | |
| 93 | cache, | |
| 94 | async () => { | |
| 95 | writeFileSync(join(cache, "fetched.tgz"), new Uint8Array(256 * 1024).fill(7)); | |
| 96 | started(); | |
| 97 | await gate; | |
| 98 | return existsSync(join(cache, "fetched.tgz")) | |
| 99 | ? { code: 0, stderr: "" } | |
| 100 | : { code: 1, stderr: "the cache was emptied under this install" }; | |
| 101 | }, | |
| 102 | { max, now }, | |
| 103 | ); | |
| 104 | await first; | |
| 105 | const emptying = installDependencies( | |
| 106 | workdir("emptying"), | |
| 107 | cache, | |
| 108 | async () => ({ code: 0, stderr: "" }), | |
| 109 | { max, now: now + BOUND_EVERY_MS }, | |
| 110 | ); | |
| 111 | await Bun.sleep(50); | |
| 112 | release(); | |
| 113 | expect(await reading).toEqual({ installed: true }); | |
| 114 | const second = await emptying; | |
| 115 | expect(second.installed).toBe(true); | |
| 116 | expect(second.emptied).toBeDefined(); | |
| 117 | expect(existsSync(join(cache, "fetched.tgz"))).toBe(false); | |
| 118 | }); |
test/cell-tmp.test.ts added +85
| @@ -0,0 +1,85 @@ | ||
| 1 | /** | |
| 2 | * A check's /tmp is on disk (#6947). A cell's /tmp was a 256 MiB tmpfs, and | |
| 3 | * the suite's throwaway forges filled it: checks failed with ENOSPC for the | |
| 4 | * machine, not the code. A cell given `tmp` mounts that directory as /tmp — | |
| 5 | * the paths inside stay short enough for a unix socket, the disk watch | |
| 6 | * counts it under the home, and the home's removal takes it. | |
| 7 | */ | |
| 8 | ||
| 9 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 10 | import { existsSync, readFileSync } from "node:fs"; | |
| 11 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 12 | import { tmpdir } from "node:os"; | |
| 13 | import { join } from "node:path"; | |
| 14 | import { TMP_MIB, tmpMount, type CellSpec } from "../src/apps/contain.ts"; | |
| 15 | import { childEnv, runStep } from "../src/apps/process.ts"; | |
| 16 | import { checkCell, checkHome } from "../src/checks/check_cell.ts"; | |
| 17 | import { portInsteadOfSocket } from "../src/git/hook-address.ts"; | |
| 18 | import { missing } from "./helpers/tools.ts"; | |
| 19 | ||
| 20 | const noCells = | |
| 21 | process.env["FORGE_CELLS"] === "off" || | |
| 22 | process.platform !== "linux" || | |
| 23 | portInsteadOfSocket() || | |
| 24 | missing("bwrap", "socat"); | |
| 25 | ||
| 26 | test("a cell with its own tmp binds it at /tmp; one without gets the bounded tmpfs", () => { | |
| 27 | expect(tmpMount({ tmp: "/data/checks/run.home/tmp" })).toEqual(["--bind", "/data/checks/run.home/tmp", "/tmp"]); | |
| 28 | expect(tmpMount({})).toEqual(["--size", String(TMP_MIB * 1024 * 1024), "--tmpfs", "/tmp"]); | |
| 29 | }); | |
| 30 | ||
| 31 | test("a check's cell puts its /tmp on disk inside its home, never the tmpfs", () => { | |
| 32 | const dir = "/data/checks/01run"; | |
| 33 | const cell = checkCell({ | |
| 34 | dir, | |
| 35 | dataDir: "/data", | |
| 36 | manifest: { egress: ["example.com"], limits: {} }, | |
| 37 | forgeUrl: "https://forge.test", | |
| 38 | egressMode: "enforce", | |
| 39 | onUndeclared: () => {}, | |
| 40 | runId: "01run", | |
| 41 | }); | |
| 42 | expect(cell.home).toBe(checkHome(dir)); | |
| 43 | expect(cell.tmp).toBe(join(checkHome(dir), "tmp")); | |
| 44 | expect(tmpMount(cell)).toEqual(["--bind", join(checkHome(dir), "tmp"), "/tmp"]); | |
| 45 | expect(cell.hide).toContain("/data"); | |
| 46 | expect(cell.egress).toContain("example.com"); | |
| 47 | expect(cell.label).toBe("check:01run"); | |
| 48 | }); | |
| 49 | ||
| 50 | describe.skipIf(noCells)("a cell's /tmp on disk", () => { | |
| 51 | let dir = ""; | |
| 52 | ||
| 53 | beforeAll(async () => { | |
| 54 | dir = await mkdtemp(join(tmpdir(), "forge-cell-tmp-")); | |
| 55 | }); | |
| 56 | ||
| 57 | afterAll(async () => { | |
| 58 | await rm(dir, { recursive: true, force: true }); | |
| 59 | }); | |
| 60 | ||
| 61 | test("what the step writes to /tmp lands in the cell's tmp, past the tmpfs's size", async () => { | |
| 62 | const home = join(dir, ".home"); | |
| 63 | const spec: CellSpec = { | |
| 64 | dir, | |
| 65 | writable: [], | |
| 66 | home, | |
| 67 | tmp: join(home, "tmp"), | |
| 68 | hide: [], | |
| 69 | egress: [], | |
| 70 | reach: [], | |
| 71 | limits: { memoryMiB: 2048, cpus: 2, diskMiB: 4096 }, | |
| 72 | }; | |
| 73 | const ran = await runStep("check", `head -c ${(TMP_MIB + 16) * 1024 * 1024} /dev/zero > /tmp/big && echo kept > /tmp/mark`, { | |
| 74 | cwd: dir, | |
| 75 | env: childEnv({}), | |
| 76 | log: () => {}, | |
| 77 | signal: new AbortController().signal, | |
| 78 | timeoutMs: 60_000, | |
| 79 | cell: spec, | |
| 80 | }); | |
| 81 | expect(ran.ok).toBe(true); | |
| 82 | expect(readFileSync(join(home, "tmp", "mark"), "utf8")).toBe("kept\n"); | |
| 83 | expect(existsSync(join(home, "tmp", "big"))).toBe(true); | |
| 84 | }, 60_000); | |
| 85 | }); |
test/checks-sweep-kept.test.ts added +51
| @@ -0,0 +1,51 @@ | ||
| 1 | /** | |
| 2 | * The checkout sweep removes checkouts, nothing else (#6947): the checks | |
| 3 | * keep their gate logs, learned files and impact maps beside the checkouts, | |
| 4 | * and a live check's cell home is as much in use as its checkout. | |
| 5 | */ | |
| 6 | ||
| 7 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 8 | import { existsSync, mkdirSync, writeFileSync } from "node:fs"; | |
| 9 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 10 | import { tmpdir } from "node:os"; | |
| 11 | import { join } from "node:path"; | |
| 12 | import { createApp, type App } from "../src/app.ts"; | |
| 13 | import { createRepo } from "../src/actions/repos.ts"; | |
| 14 | import { sweepCheckouts } from "../src/checks/service.ts"; | |
| 15 | import { claimRun, createRun } from "../src/core/runs.ts"; | |
| 16 | import { id } from "../src/kernel/id.ts"; | |
| 17 | import { seedAdmin } from "./helpers/admin.ts"; | |
| 18 | ||
| 19 | let dataDir = ""; | |
| 20 | let app: App; | |
| 21 | ||
| 22 | beforeAll(async () => { | |
| 23 | dataDir = await mkdtemp(join(tmpdir(), "forge-sweep-kept-")); | |
| 24 | app = createApp({ dataDir, port: 0 }); | |
| 25 | }); | |
| 26 | ||
| 27 | afterAll(async () => { | |
| 28 | app.close(); | |
| 29 | await rm(dataDir, { recursive: true, force: true }); | |
| 30 | }); | |
| 31 | ||
| 32 | test("the sweep keeps what the checks keep and a live check's home, and removes the rest", async () => { | |
| 33 | const admin = await seedAdmin(app, { username: "alice", email: "[email protected]", password: "correct-horse-battery" }); | |
| 34 | if (!admin.ok) throw new Error(admin.error); | |
| 35 | const repo = await createRepo(app, admin.value, { name: "notes", visibility: "public", description: "" }); | |
| 36 | if (!repo.ok) throw new Error(repo.error); | |
| 37 | const live = createRun(app.db, { repoId: repo.value.id, kind: "check" }); | |
| 38 | claimRun(app.db, live.id); | |
| 39 | const gone = id(); | |
| 40 | const root = join(dataDir, "checks"); | |
| 41 | for (const dir of ["gates/alice/notes", "learned", "impact", live.id, `${live.id}.home`, gone, `${gone}.home`]) | |
| 42 | mkdirSync(join(root, dir), { recursive: true }); | |
| 43 | writeFileSync(join(root, "gate-last.json"), "{}"); | |
| 44 | ||
| 45 | sweepCheckouts(app); | |
| 46 | ||
| 47 | for (const kept of ["gates/alice/notes", "learned", "impact", "gate-last.json", live.id, `${live.id}.home`]) | |
| 48 | expect(existsSync(join(root, kept))).toBe(true); | |
| 49 | expect(existsSync(join(root, gone))).toBe(false); | |
| 50 | expect(existsSync(join(root, `${gone}.home`))).toBe(false); | |
| 51 | }); |
test/disk-use.test.ts added +125
| @@ -0,0 +1,125 @@ | ||
| 1 | /** | |
| 2 | * What fills the disk (#6947): /settings/capacity, its API and the MCP's | |
| 3 | * capacity tool name the biggest entries under the data directory, biggest | |
| 4 | * first, counted as `du` counts — a hard-linked file once, a symlink never | |
| 5 | * followed — and a request never waits for the walk. | |
| 6 | */ | |
| 7 | ||
| 8 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 9 | import { linkSync, mkdirSync, symlinkSync, writeFileSync } from "node:fs"; | |
| 10 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 11 | import { tmpdir } from "node:os"; | |
| 12 | import { join } from "node:path"; | |
| 13 | import { createApp, type App } from "../src/app.ts"; | |
| 14 | import { listCapacity } from "../src/actions/capacity.ts"; | |
| 15 | import { diskUse, measureDiskUse, refreshDiskUse } from "../src/kernel/disk_use.ts"; | |
| 16 | import type { User } from "../src/core/users.ts"; | |
| 17 | import { serve, type Servers } from "../src/http/server.ts"; | |
| 18 | import { seedAdmin } from "./helpers/admin.ts"; | |
| 19 | ||
| 20 | const MiB = 1024 * 1024; | |
| 21 | const filled = (n: number): Uint8Array => new Uint8Array(n).fill(7); | |
| 22 | ||
| 23 | describe("measuring the data directory", () => { | |
| 24 | let dir = ""; | |
| 25 | let outside = ""; | |
| 26 | ||
| 27 | beforeAll(async () => { | |
| 28 | dir = await mkdtemp(join(tmpdir(), "forge-disk-use-")); | |
| 29 | outside = await mkdtemp(join(tmpdir(), "forge-disk-outside-")); | |
| 30 | mkdirSync(join(dir, "crew", ".cache", "repo"), { recursive: true }); | |
| 31 | mkdirSync(join(dir, "crew", "mason"), { recursive: true }); | |
| 32 | mkdirSync(join(dir, "checks"), { recursive: true }); | |
| 33 | writeFileSync(join(dir, "crew", ".cache", "repo", "pkg.tgz"), filled(2 * MiB)); | |
| 34 | // node_modules linked from the cache holds no more of the disk. | |
| 35 | linkSync(join(dir, "crew", ".cache", "repo", "pkg.tgz"), join(dir, "crew", "mason", "pkg.tgz")); | |
| 36 | writeFileSync(join(dir, "checks", "gate-last.json"), filled(64 * 1024)); | |
| 37 | writeFileSync(join(dir, "forge.db"), filled(512 * 1024)); | |
| 38 | writeFileSync(join(outside, "elsewhere.bin"), filled(4 * MiB)); | |
| 39 | symlinkSync(outside, join(dir, "linked")); | |
| 40 | }); | |
| 41 | ||
| 42 | afterAll(async () => { | |
| 43 | await rm(dir, { recursive: true, force: true }); | |
| 44 | await rm(outside, { recursive: true, force: true }); | |
| 45 | }); | |
| 46 | ||
| 47 | test("the biggest entry comes first, with the biggest part inside it", async () => { | |
| 48 | const found = await measureDiskUse(dir); | |
| 49 | expect(found.map((c) => c.path).slice(0, 3)).toEqual(["crew", "forge.db", "checks"]); | |
| 50 | const crew = found[0]!; | |
| 51 | expect(crew.parts[0]?.path).toBe("crew/.cache"); | |
| 52 | expect(crew.parts[0]!.bytes).toBeGreaterThanOrEqual(2 * MiB); | |
| 53 | expect(found.find((c) => c.path === "forge.db")?.parts).toEqual([]); | |
| 54 | }); | |
| 55 | ||
| 56 | test("a hard-linked file counts once and a symlink is not followed", async () => { | |
| 57 | const found = await measureDiskUse(dir); | |
| 58 | const crew = found.find((c) => c.path === "crew")!; | |
| 59 | expect(crew.bytes).toBeLessThan(3 * MiB); | |
| 60 | expect(found.find((c) => c.path === "linked")!.bytes).toBeLessThan(64 * 1024); | |
| 61 | }); | |
| 62 | ||
| 63 | test("the first ask answers at once and starts the walk; the next has the reading", async () => { | |
| 64 | const first = diskUse(dir); | |
| 65 | expect(first.measuring).toBe(true); | |
| 66 | expect(first.measuredAt).toBeUndefined(); | |
| 67 | expect(first.disk.totalBytes).toBeGreaterThan(0); | |
| 68 | await refreshDiskUse(dir); | |
| 69 | const next = diskUse(dir); | |
| 70 | expect(next.measuring).toBe(false); | |
| 71 | expect(next.measuredAt).toBeDefined(); | |
| 72 | expect(next.consumers[0]?.path).toBe("crew"); | |
| 73 | }); | |
| 74 | }); | |
| 75 | ||
| 76 | describe("the capacity page names the biggest consumers", () => { | |
| 77 | let dataDir = ""; | |
| 78 | let app: App; | |
| 79 | let servers: Servers; | |
| 80 | let bearer = ""; | |
| 81 | let alice: User; | |
| 82 | ||
| 83 | beforeAll(async () => { | |
| 84 | dataDir = await mkdtemp(join(tmpdir(), "forge-disk-page-")); | |
| 85 | app = createApp({ dataDir, port: 0 }); | |
| 86 | servers = await serve(app); | |
| 87 | const admin = await seedAdmin(app, { username: "alice", email: "[email protected]", password: "correct-horse-battery" }); | |
| 88 | if (!admin.ok) throw new Error(admin.error); | |
| 89 | alice = admin.value; | |
| 90 | const minted = await fetch(`${servers.url}/api/v1/tokens`, { | |
| 91 | method: "POST", | |
| 92 | headers: { "content-type": "application/json" }, | |
| 93 | body: JSON.stringify({ adminConfirm: true, username: "alice", password: "correct-horse-battery", name: "cli", scope: "admin" }), | |
| 94 | }); | |
| 95 | bearer = ((await minted.json()) as { token: string }).token; | |
| 96 | mkdirSync(join(dataDir, "check-install", "repo", "bun"), { recursive: true }); | |
| 97 | writeFileSync(join(dataDir, "check-install", "repo", "bun", "pkg.tgz"), filled(8 * MiB)); | |
| 98 | await refreshDiskUse(dataDir); | |
| 99 | }); | |
| 100 | ||
| 101 | afterAll(async () => { | |
| 102 | servers.stop(); | |
| 103 | app.close(); | |
| 104 | await rm(dataDir, { recursive: true, force: true }); | |
| 105 | }); | |
| 106 | ||
| 107 | test("the page names the biggest entry and what is inside it", async () => { | |
| 108 | const res = await fetch(`${servers.url}/settings/capacity`, { headers: { authorization: `Bearer ${bearer}` } }); | |
| 109 | const page = await res.text(); | |
| 110 | expect(res.status).toBe(200); | |
| 111 | expect(page).toContain("<h2>Disk</h2>"); | |
| 112 | expect(page).toMatch(/id="disk-free">[^<]*free of/); | |
| 113 | const listed = /<ul class="list" id="disk-use">([\s\S]*?)<\/ul>/.exec(page)?.[1] ?? ""; | |
| 114 | expect(listed.indexOf("check-install")).toBeGreaterThan(-1); | |
| 115 | expect(listed).toContain("check-install/repo"); | |
| 116 | }); | |
| 117 | ||
| 118 | test("the API and the MCP's capacity listing carry the same reading", async () => { | |
| 119 | const res = await fetch(`${servers.url}/api/v1/admin/capacity`, { headers: { authorization: `Bearer ${bearer}` } }); | |
| 120 | const body = (await res.json()) as { disk: { consumers: { path: string }[] } }; | |
| 121 | expect(body.disk.consumers[0]?.path).toBe("check-install"); | |
| 122 | const listed = listCapacity(app, alice); | |
| 123 | expect(listed.ok && listed.value.disk.consumers[0]?.path).toBe("check-install"); | |
| 124 | }); | |
| 125 | }); |
new above
offline
This line is the whole fix for #6947, and no test fails if it is removed:
tmpMountis tested on its own, and the real-cell test skips underFORGE_CELLS=off. Move this spec into a named function, for examplecheckCell(...), and assert that itstmpsits underhome.