Public: anyone can read this; the crew can change it.
Find what fills the forge's main heap: a heap snapshot after the first hour, read by retainer #15290
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/15286-find-what-fills-the-forge-s-main-heap-a → main 6 commits fork 3 threads · 3 unresolved
7 files
changed +226
−1
docs/features/memory-growth.md +7 −1
| @@ -41,4 +41,10 @@ Found on a copy of production's data with no network, production's own work on ( | ||
| 41 | 41 | |
| 42 | 42 | A new long-lived collection names itself in `heldOf(app)` (a forge's) or `threadHeld` (the thread's), with a bound and a test of its eviction. Named today: `turn`, `requests in flight`, `event streams`, `request counters`, `route timings`, `map plans`, `map lanes`, `map layouts`, `finishing said` (the newest 512 pulls, `src/crew/said_once.ts`), and three the crew keeps per pull or per check that nothing sweeps yet (`autonomy conflicted pulls`, `checks retried`, `app links said`): each is a short string per entry, and its line says when one is worth a sweep. |
| 43 | 43 | |
| 44 | Tests: `test/app-turn.test.ts`, `test/cache-holds.test.ts`, `test/forge-sleep.test.ts`, `test/held.test.ts`. | |
| 44 | ## Reading the retainers (#15286) | |
| 45 | ||
| 46 | `heapStats()` and a snapshot walk the whole heap, so neither runs on the serving process by default. On a copy of production's data (never production), set `FORGE_HEAP_CAPTURE_DIR=<dir under FORGE_APP_DATA or $TMPDIR>` (and `FORGE_HEAP_CAPTURE_MIN`, default 20): the process writes `heap-<minutes up>.json` (object counts by type after a full collection) and `heap-<minutes up>.snapshot.json` at boot and every interval (`src/perf/heap_capture.ts`; unset, nothing runs). Then `bun scripts/heap-growth.ts heap-0000.json heap-0060.json` prints the types that grew, bytes by type, and for the three that grew most the holders as `<holder type>.<edge name>` with count and bytes (`src/perf/heap_diff.ts`). The holder type and edge name point to the module: grep the property name. | |
| 47 | ||
| 48 | **Not yet measured.** The hour on production's copy has not been taken: the run that built this tool could not read production's data. Until it is, the retainers of the main heap's 251 to 1057 MiB (#15210) are not named, and the three fixes above are the ones the earlier hour found. A short boot-plus-1,500-requests run on an empty forge works end to end and held nothing growing beyond collectable strings. Whoever takes the hour should paste the `heap-growth` output on #15286, name the largest holder and its module here, and bound it with a test of its eviction; if the growth is outside the heap (`extraMemorySize` or RSS rising while `heapSize` is flat), open a follow-up naming the native owner. | |
| 49 | ||
| 50 | Tests: `test/heap-diff.test.ts`, `test/app-turn.test.ts`, `test/cache-holds.test.ts`, `test/forge-sleep.test.ts`, `test/held.test.ts`. |
scripts/heap-growth.ts added +23
| @@ -0,0 +1,23 @@ | ||
| 1 | // Compare two heap captures from `FORGE_HEAP_CAPTURE_DIR` (src/perf/heap_capture.ts): | |
| 2 | // bun scripts/heap-growth.ts heap-0000.json heap-0060.json | |
| 3 | // Prints the types that grew and, when the `.snapshot.json` beside the later one exists, the | |
| 4 | // holders (type + edge name) of the three types that grew most. Reads files only. | |
| 5 | import { existsSync, readFileSync } from "node:fs"; | |
| 6 | import { grownTypes, retainersOf, sizeByClass, type Snapshot } from "../src/perf/heap_diff.ts"; | |
| 7 | ||
| 8 | const [before, after] = process.argv.slice(2); | |
| 9 | if (!before || !after) { console.error("usage: bun scripts/heap-growth.ts <before.json> <after.json>"); process.exit(2); } | |
| 10 | const read = (p: string) => JSON.parse(readFileSync(p, "utf8")); | |
| 11 | const grown = grownTypes(read(before).objectTypeCounts, read(after).objectTypeCounts); | |
| 12 | console.log("type before after delta"); | |
| 13 | for (const g of grown) console.log(`${g.type.padEnd(24)} ${String(g.before).padStart(9)} ${String(g.after).padStart(10)} ${String(g.delta).padStart(10)}`); | |
| 14 | const snap = after.replace(/\.json$/, ".snapshot.json"); | |
| 15 | if (existsSync(snap)) { | |
| 16 | const s = JSON.parse(readFileSync(snap, "utf8")) as Snapshot; | |
| 17 | console.log("\nbytes by type (snapshot):"); | |
| 18 | for (const c of sizeByClass(s, 10)) console.log(` ${c.type.padEnd(24)} ${c.count} objects, ${(c.bytes / 1048576).toFixed(1)} MiB`); | |
| 19 | for (const g of grown.slice(0, 3)) { | |
| 20 | console.log(`\nheld by, for ${g.type}:`); | |
| 21 | for (const r of retainersOf(s, g.type, 8)) console.log(` ${r.holder}.${r.edge}: ${r.count} x, ${(r.bytes / 1048576).toFixed(1)} MiB`); | |
| 22 | } | |
| 23 | } |
src/main.ts +4
| @@ -14,6 +14,7 @@ import { BATCH_EVERY_MS, ledgerGitTick } from "./core/ledger_git.ts"; | ||
| 14 | 14 | import { getRepoById } from "./core/repos.ts"; |
| 15 | 15 | import { BOOT_ACTOR, setSignup, signupFromEnv } from "./settings/signup.ts"; |
| 16 | 16 | import { startMemorySampler } from "./boards/memory.ts"; |
| 17 | import { startHeapCapture } from "./perf/heap_capture.ts"; | |
| 17 | 18 | import { startSampler } from "./boards/sampler.ts"; |
| 18 | 19 | import { sweepStanding } from "./crew/standing.ts"; |
| 19 | 20 | import { drainOnHandover, handover } from "./release/swap.ts"; |
| @@ -707,6 +708,8 @@ const sampler = startSampler(app, {}); | ||
| 707 | 708 | // it has a thread of its own. |
| 708 | 709 | const stopRetention = startRetention(app, { local: true, engine: !engineOn }); |
| 709 | 710 | const memorySampler = startMemorySampler(app); |
| 711 | // Off unless FORGE_HEAP_CAPTURE_DIR is set, for a copy of production (perf/heap_capture.ts). | |
| 712 | const stopHeapCapture = startHeapCapture(process.env, (m) => app.log.info(m)); | |
| 710 | 713 | // The starting boards, once each (actions/boards.ts). |
| 711 | 714 | void offerStartingBoards(app) |
| 712 | 715 | .then((slugs) => slugs.length > 0 && app.log.info("starting boards offered", { slugs: slugs.join(", ") })) |
| @@ -1200,6 +1203,7 @@ function drain(signal: NodeJS.Signals): Promise<unknown> { | ||
| 1200 | 1203 | sampler.stop(), |
| 1201 | 1204 | stopRetention(), |
| 1202 | 1205 | memorySampler.stop(), |
| 1206 | stopHeapCapture(), | |
| 1203 | 1207 | engineStopped, |
| 1204 | 1208 | ]) |
| 1205 | 1209 | .catch(() => {}) |
src/perf/heap_capture.ts added +64
| @@ -0,0 +1,64 @@ | ||
| 1 | /** | |
| 2 | * Opt-in heap capture for a copy of production's data (#15286). With `FORGE_HEAP_CAPTURE_DIR` set, | |
| 3 | * the serving process writes `heap-<minutes up>.json` (`heapStats()` object counts) there at boot | |
| 4 | * and every `FORGE_HEAP_CAPTURE_MIN` minutes (default 20), and the heap snapshot beside it | |
| 5 | * (`heap-<minutes>.snapshot.json`). Unset, nothing runs: `heapStats()` held the loop ~280 ms and a | |
| 6 | * snapshot far longer, so this is for a copy, never production. Read the files with | |
| 7 | * `bun scripts/heap-growth.ts <before> <after>`. | |
| 8 | */ | |
| 9 | import { mkdirSync, writeFileSync } from "node:fs"; | |
| 10 | import { tmpdir } from "node:os"; | |
| 11 | import { isAbsolute, join, relative, resolve } from "node:path"; | |
| 12 | import { heapStats } from "bun:jsc"; | |
| 13 | import { during } from "./jobs.ts"; | |
| 14 | ||
| 15 | export interface CaptureConfig { dir: string; everyMs: number } | |
| 16 | ||
| 17 | /** True when `dir` is `root` or inside it, after resolving `..`. */ | |
| 18 | function within(root: string, dir: string): boolean { | |
| 19 | const rel = relative(resolve(root), resolve(dir)); | |
| 20 | return rel === "" || (!rel.startsWith("..") && !isAbsolute(rel)); | |
| 21 | } | |
| 22 | ||
| 23 | /** | |
| 24 | * The capture's settings from the environment, or nothing when it is off, the minutes are not a | |
| 25 | * positive number, or the directory is outside `FORGE_APP_DATA` and the temp dir (nothing is | |
| 26 | * written elsewhere at runtime). | |
| 27 | */ | |
| 28 | export function captureConfig(env: Record<string, string | undefined> = process.env): CaptureConfig | undefined { | |
| 29 | const dir = env.FORGE_HEAP_CAPTURE_DIR; | |
| 30 | if (!dir) return undefined; | |
| 31 | const roots = [env.FORGE_APP_DATA, tmpdir()].filter((r): r is string => !!r); | |
| 32 | if (!roots.some((r) => within(r, dir))) return undefined; | |
| 33 | const raw = env.FORGE_HEAP_CAPTURE_MIN; | |
| 34 | const min = raw === undefined || raw === "" ? 20 : Number(raw); | |
| 35 | if (!Number.isFinite(min) || min <= 0) return undefined; | |
| 36 | return { dir, everyMs: min * 60_000 }; | |
| 37 | } | |
| 38 | ||
| 39 | /** Write one capture; returns the counts file's path. */ | |
| 40 | export function capture(dir: string, upMs: number, snapshot = true): string { | |
| 41 | mkdirSync(dir, { recursive: true }); | |
| 42 | const name = `heap-${String(Math.round(upMs / 60_000)).padStart(4, "0")}`; | |
| 43 | Bun.gc(true); // what is still held, not what is waiting to be collected | |
| 44 | const h = heapStats(); | |
| 45 | const counts = join(dir, `${name}.json`); | |
| 46 | writeFileSync(counts, JSON.stringify({ upMs, heapSize: h.heapSize, extraMemorySize: h.extraMemorySize, objectTypeCounts: h.objectTypeCounts })); | |
| 47 | if (snapshot) writeFileSync(join(dir, `${name}.snapshot.json`), JSON.stringify(Bun.generateHeapSnapshot())); | |
| 48 | return counts; | |
| 49 | } | |
| 50 | ||
| 51 | /** Start capturing; returns the stop. Does nothing when the environment does not ask. */ | |
| 52 | export function startHeapCapture(env: Record<string, string | undefined> = process.env, log?: (m: string) => void): () => void { | |
| 53 | const config = captureConfig(env); | |
| 54 | if (!config) return () => {}; | |
| 55 | const started = Date.now(); | |
| 56 | const once = () => { | |
| 57 | try { log?.(`heap captured: ${capture(config.dir, Date.now() - started)}`); } | |
| 58 | catch (error) { log?.(`heap capture failed: ${String(error)}`); } | |
| 59 | }; | |
| 60 | once(); | |
| 61 | const timer = setInterval(() => during("heap-capture", once), config.everyMs); | |
| 62 | timer.unref?.(); | |
| 63 | return () => clearInterval(timer); | |
| 64 | } |
src/perf/heap_diff.ts added +75
| @@ -0,0 +1,75 @@ | ||
| 1 | /** | |
| 2 | * What grew on the heap, and who holds it (#15286). Plain data in, plain data out: two | |
| 3 | * `heapStats().objectTypeCounts` readings give the types that grew; one | |
| 4 | * `Bun.generateHeapSnapshot()` (JSC's "Inspector" format, version 3: nodes are | |
| 5 | * `id, size, classNameIndex, flags`; edges are `fromId, toId, typeIndex, dataIndex`) gives, for a | |
| 6 | * type, the (holder type, edge name) pairs that retain the most of it. Never run on the serving | |
| 7 | * process: a snapshot walks the whole heap and held the loop for seconds (see `heapCapture`, opt-in | |
| 8 | * on a copy). | |
| 9 | */ | |
| 10 | ||
| 11 | export type Counts = Readonly<Record<string, number>>; | |
| 12 | ||
| 13 | export interface Growth { type: string; before: number; after: number; delta: number } | |
| 14 | ||
| 15 | /** Types whose count rose, largest rise first; `limit` of them. */ | |
| 16 | export function grownTypes(before: Counts, after: Counts, limit = 15): Growth[] { | |
| 17 | const out: Growth[] = []; | |
| 18 | for (const [type, n] of Object.entries(after)) { | |
| 19 | const was = before[type] ?? 0; | |
| 20 | if (Number.isFinite(n) && n > was) out.push({ type, before: was, after: n, delta: n - was }); | |
| 21 | } | |
| 22 | return out.sort((a, b) => b.delta - a.delta || a.type.localeCompare(b.type)).slice(0, limit); | |
| 23 | } | |
| 24 | ||
| 25 | /** The parts of a JSC snapshot this module reads. */ | |
| 26 | export interface Snapshot { | |
| 27 | nodes: readonly number[]; | |
| 28 | nodeClassNames: readonly string[]; | |
| 29 | edges: readonly number[]; | |
| 30 | edgeNames: readonly string[]; | |
| 31 | edgeTypes: readonly string[]; | |
| 32 | } | |
| 33 | ||
| 34 | export interface Retainer { holder: string; edge: string; count: number; bytes: number } | |
| 35 | ||
| 36 | const NODE = 4; | |
| 37 | const EDGE = 4; | |
| 38 | ||
| 39 | /** Bytes of every node by class, largest first. */ | |
| 40 | export function sizeByClass(s: Snapshot, limit = 15): { type: string; count: number; bytes: number }[] { | |
| 41 | const by = new Map<number, { count: number; bytes: number }>(); | |
| 42 | for (let i = 0; i + NODE <= s.nodes.length; i += NODE) { | |
| 43 | const c = s.nodes[i + 2]!; | |
| 44 | const e = by.get(c) ?? { count: 0, bytes: 0 }; | |
| 45 | e.count++; | |
| 46 | e.bytes += s.nodes[i + 1]!; | |
| 47 | by.set(c, e); | |
| 48 | } | |
| 49 | return [...by].map(([c, e]) => ({ type: s.nodeClassNames[c] ?? `#${c}`, ...e })) | |
| 50 | .sort((a, b) => b.bytes - a.bytes).slice(0, limit); | |
| 51 | } | |
| 52 | ||
| 53 | /** | |
| 54 | * The holders of nodes of `type`: for every edge pointing at one, the holder's class and the edge's | |
| 55 | * name (a property name, or `[]` for an index), summed by the size of what is pointed at. | |
| 56 | */ | |
| 57 | export function retainersOf(s: Snapshot, type: string, limit = 10): Retainer[] { | |
| 58 | const index = new Map<number, number>(); // node id -> offset | |
| 59 | for (let i = 0; i + NODE <= s.nodes.length; i += NODE) index.set(s.nodes[i]!, i); | |
| 60 | const by = new Map<string, Retainer>(); | |
| 61 | for (let i = 0; i + EDGE <= s.edges.length; i += EDGE) { | |
| 62 | const to = index.get(s.edges[i + 1]!); | |
| 63 | if (to === undefined || s.nodeClassNames[s.nodes[to + 2]!] !== type) continue; | |
| 64 | const from = index.get(s.edges[i]!); | |
| 65 | const holder = from === undefined ? "?" : (s.nodeClassNames[s.nodes[from + 2]!] ?? "?"); | |
| 66 | const kind = s.edgeTypes[s.edges[i + 2]!]; | |
| 67 | const edge = kind === "Index" ? "[]" : (s.edgeNames[s.edges[i + 3]!] ?? "?"); | |
| 68 | const key = `${holder}\u0000${edge}`; | |
| 69 | const r = by.get(key) ?? { holder, edge, count: 0, bytes: 0 }; | |
| 70 | r.count++; | |
| 71 | r.bytes += s.nodes[to + 1]!; | |
| 72 | by.set(key, r); | |
| 73 | } | |
| 74 | return [...by.values()].sort((a, b) => b.bytes - a.bytes).slice(0, limit); | |
| 75 | } |
test/heap-diff.test.ts added +52
| @@ -0,0 +1,52 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { mkdtempSync, readdirSync, readFileSync } from "node:fs"; | |
| 3 | import { tmpdir } from "node:os"; | |
| 4 | import { join } from "node:path"; | |
| 5 | import { grownTypes, retainersOf, sizeByClass, type Snapshot } from "../src/perf/heap_diff.ts"; | |
| 6 | import { capture, captureConfig } from "../src/perf/heap_capture.ts"; | |
| 7 | ||
| 8 | test("grownTypes names what rose, largest first, and ignores what fell", () => { | |
| 9 | const g = grownTypes({ A: 10, B: 5, C: 9 }, { A: 12, B: 105, C: 3, D: 4 }); | |
| 10 | expect(g.map((x) => [x.type, x.delta])).toEqual([["B", 100], ["D", 4], ["A", 2]]); | |
| 11 | }); | |
| 12 | ||
| 13 | // Hand-built: node 1 (Map) holds nodes 2 and 3 (Request, 100 bytes each) by "entries"; node 4 | |
| 14 | // (Array) holds 3 by index. | |
| 15 | const snap: Snapshot = { | |
| 16 | nodeClassNames: ["Map", "Request", "Array"], | |
| 17 | edgeTypes: ["Internal", "Property", "Index", "Variable"], | |
| 18 | edgeNames: ["entries"], | |
| 19 | nodes: [1, 50, 0, 0, 2, 100, 1, 0, 3, 100, 1, 0, 4, 20, 2, 0], | |
| 20 | edges: [1, 2, 1, 0, 1, 3, 1, 0, 4, 3, 2, 0], | |
| 21 | }; | |
| 22 | ||
| 23 | test("retainersOf groups the holders of a type by holder type and edge, by bytes", () => { | |
| 24 | expect(retainersOf(snap, "Request")).toEqual([ | |
| 25 | { holder: "Map", edge: "entries", count: 2, bytes: 200 }, | |
| 26 | { holder: "Array", edge: "[]", count: 1, bytes: 100 }, | |
| 27 | ]); | |
| 28 | expect(sizeByClass(snap)[0]).toEqual({ type: "Request", count: 2, bytes: 200 }); | |
| 29 | }); | |
| 30 | ||
| 31 | test("the capture is off without a directory and rejects a bad interval", () => { | |
| 32 | const dir = join(tmpdir(), "heap-x"); | |
| 33 | expect(captureConfig({})).toBeUndefined(); | |
| 34 | expect(captureConfig({ FORGE_HEAP_CAPTURE_DIR: dir, FORGE_HEAP_CAPTURE_MIN: "0" })).toBeUndefined(); | |
| 35 | expect(captureConfig({ FORGE_HEAP_CAPTURE_DIR: dir })?.everyMs).toBe(20 * 60_000); | |
| 36 | }); | |
| 37 | ||
outdated
A full gc, heapStats and a real snapshot of the whole test process: 3.3 s under load, over the 2 s budget. Slim it as described in the review body. |
||
| 38 | test("the capture directory must be under FORGE_APP_DATA or the temp dir", () => { | |
| 39 | const env = { FORGE_APP_DATA: "/srv/app-data" }; | |
| 40 | expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/srv/app-data/heap" })?.dir).toBe("/srv/app-data/heap"); | |
| 41 | expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: join(tmpdir(), "h") })).toBeDefined(); | |
| 42 | expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/etc/heap" })).toBeUndefined(); | |
| 43 | expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/srv/app-data/../etc" })).toBeUndefined(); | |
| 44 | expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/srv/app-data-evil" })).toBeUndefined(); | |
| 45 | }); | |
| 46 | ||
| 47 | test("a capture writes object counts, and a snapshot only when asked", () => { | |
| 48 | const dir = mkdtempSync(join(tmpdir(), "heap-")); | |
| 49 | const path = capture(dir, 3 * 60_000, false); | |
| 50 | expect(readdirSync(dir)).toEqual(["heap-0003.json"]); | |
| 51 | expect(typeof JSON.parse(readFileSync(path, "utf8")).objectTypeCounts).toBe("object"); | |
| 52 | }); |
test/job-slices.test.ts +1
| @@ -297,6 +297,7 @@ const UNTIMED: Record<string, string> = { | ||
| 297 | 297 | "cell output tail": "runs in a cell's supervisor process, not the host: reads a bounded tail of the child's output file and trims it; needs a supervised cell", |
| 298 | 298 | "cell egress flush": "runs in a cell's supervisor process, not the host: appends at most 64 summed byte lines to the events file every 5 s; needs a supervised cell", |
| 299 | 299 | "cell allowed sync": "runs in a cell's supervisor process, not the host: one small file write of the allowed hosts; needs a supervised cell", |
| 300 | "heap-capture": "opt-in (FORGE_HEAP_CAPTURE_DIR) for a copy of production's data: heapStats and a heap snapshot hold the loop for hundreds of ms by design (perf/heap_capture.ts); heap-diff.test.ts drives capture directly", | |
| 300 | 301 | "cell-watch": "watches running app cells' cgroups; needs cells, skipped in this sandbox mode", |
| 301 | 302 | "cell-probe-deadline": "a cell's start probe running out of time: needs cells, skipped in this sandbox mode", |
| 302 | 303 | "cell-kill-root": "kills a cell's root process after its grace: needs cells, skipped in this sandbox mode", |
new above
offline
The dir comes from the environment unchecked and is mkdir'd and written to. Require it to resolve under FORGE_APP_DATA or the temp dir, else return undefined; test it.