Public: anyone can read this; the crew can change it.

Find what fills the forge's main heap: a heap snapshot after the first hour, read by retainer #15290

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

torque/forge:torque/15286-find-what-fills-the-forge-s-main-heap-a → main 6 commits fork 3 threads · 3 unresolved

  • Merged. torque/15286-find-what-fills-the-forge-s-main-heap-a was merged into main as e2ade16.

Review

Sign in to review this change.
7 files changed +226 −1
docs/features/memory-growth.md +7 −1
@@ -41,4 +41,10 @@ Found on a copy of production's data with no network, production's own work on (
4141
4242 A new long-lived collection names itself in `heldOf(app)` (a forge's) or `threadHeld` (the thread's), with a bound and a test of its eviction. Named today: `turn`, `requests in flight`, `event streams`, `request counters`, `route timings`, `map plans`, `map lanes`, `map layouts`, `finishing said` (the newest 512 pulls, `src/crew/said_once.ts`), and three the crew keeps per pull or per check that nothing sweeps yet (`autonomy conflicted pulls`, `checks retried`, `app links said`): each is a short string per entry, and its line says when one is worth a sweep.
4343
44 Tests: `test/app-turn.test.ts`, `test/cache-holds.test.ts`, `test/forge-sleep.test.ts`, `test/held.test.ts`.
44 ## Reading the retainers (#15286)
45
46 `heapStats()` and a snapshot walk the whole heap, so neither runs on the serving process by default. On a copy of production's data (never production), set `FORGE_HEAP_CAPTURE_DIR=<dir under FORGE_APP_DATA or $TMPDIR>` (and `FORGE_HEAP_CAPTURE_MIN`, default 20): the process writes `heap-<minutes up>.json` (object counts by type after a full collection) and `heap-<minutes up>.snapshot.json` at boot and every interval (`src/perf/heap_capture.ts`; unset, nothing runs). Then `bun scripts/heap-growth.ts heap-0000.json heap-0060.json` prints the types that grew, bytes by type, and for the three that grew most the holders as `<holder type>.<edge name>` with count and bytes (`src/perf/heap_diff.ts`). The holder type and edge name point to the module: grep the property name.
47
48 **Not yet measured.** The hour on production's copy has not been taken: the run that built this tool could not read production's data. Until it is, the retainers of the main heap's 251 to 1057 MiB (#15210) are not named, and the three fixes above are the ones the earlier hour found. A short boot-plus-1,500-requests run on an empty forge works end to end and held nothing growing beyond collectable strings. Whoever takes the hour should paste the `heap-growth` output on #15286, name the largest holder and its module here, and bound it with a test of its eviction; if the growth is outside the heap (`extraMemorySize` or RSS rising while `heapSize` is flat), open a follow-up naming the native owner.
49
50 Tests: `test/heap-diff.test.ts`, `test/app-turn.test.ts`, `test/cache-holds.test.ts`, `test/forge-sleep.test.ts`, `test/held.test.ts`.
scripts/heap-growth.ts added +23
@@ -0,0 +1,23 @@
1 // Compare two heap captures from `FORGE_HEAP_CAPTURE_DIR` (src/perf/heap_capture.ts):
2 // bun scripts/heap-growth.ts heap-0000.json heap-0060.json
3 // Prints the types that grew and, when the `.snapshot.json` beside the later one exists, the
4 // holders (type + edge name) of the three types that grew most. Reads files only.
5 import { existsSync, readFileSync } from "node:fs";
6 import { grownTypes, retainersOf, sizeByClass, type Snapshot } from "../src/perf/heap_diff.ts";
7
8 const [before, after] = process.argv.slice(2);
9 if (!before || !after) { console.error("usage: bun scripts/heap-growth.ts <before.json> <after.json>"); process.exit(2); }
10 const read = (p: string) => JSON.parse(readFileSync(p, "utf8"));
11 const grown = grownTypes(read(before).objectTypeCounts, read(after).objectTypeCounts);
12 console.log("type before after delta");
13 for (const g of grown) console.log(`${g.type.padEnd(24)} ${String(g.before).padStart(9)} ${String(g.after).padStart(10)} ${String(g.delta).padStart(10)}`);
14 const snap = after.replace(/\.json$/, ".snapshot.json");
15 if (existsSync(snap)) {
16 const s = JSON.parse(readFileSync(snap, "utf8")) as Snapshot;
17 console.log("\nbytes by type (snapshot):");
18 for (const c of sizeByClass(s, 10)) console.log(` ${c.type.padEnd(24)} ${c.count} objects, ${(c.bytes / 1048576).toFixed(1)} MiB`);
19 for (const g of grown.slice(0, 3)) {
20 console.log(`\nheld by, for ${g.type}:`);
21 for (const r of retainersOf(s, g.type, 8)) console.log(` ${r.holder}.${r.edge}: ${r.count} x, ${(r.bytes / 1048576).toFixed(1)} MiB`);
22 }
23 }
src/main.ts +4
@@ -14,6 +14,7 @@ import { BATCH_EVERY_MS, ledgerGitTick } from "./core/ledger_git.ts";
1414 import { getRepoById } from "./core/repos.ts";
1515 import { BOOT_ACTOR, setSignup, signupFromEnv } from "./settings/signup.ts";
1616 import { startMemorySampler } from "./boards/memory.ts";
17 import { startHeapCapture } from "./perf/heap_capture.ts";
1718 import { startSampler } from "./boards/sampler.ts";
1819 import { sweepStanding } from "./crew/standing.ts";
1920 import { drainOnHandover, handover } from "./release/swap.ts";
@@ -707,6 +708,8 @@ const sampler = startSampler(app, {});
707708 // it has a thread of its own.
708709 const stopRetention = startRetention(app, { local: true, engine: !engineOn });
709710 const memorySampler = startMemorySampler(app);
711 // Off unless FORGE_HEAP_CAPTURE_DIR is set, for a copy of production (perf/heap_capture.ts).
712 const stopHeapCapture = startHeapCapture(process.env, (m) => app.log.info(m));
710713 // The starting boards, once each (actions/boards.ts).
711714 void offerStartingBoards(app)
712715 .then((slugs) => slugs.length > 0 && app.log.info("starting boards offered", { slugs: slugs.join(", ") }))
@@ -1200,6 +1203,7 @@ function drain(signal: NodeJS.Signals): Promise<unknown> {
12001203 sampler.stop(),
12011204 stopRetention(),
12021205 memorySampler.stop(),
1206 stopHeapCapture(),
12031207 engineStopped,
12041208 ])
12051209 .catch(() => {})
src/perf/heap_capture.ts added +64
@@ -0,0 +1,64 @@
1 /**
2 * Opt-in heap capture for a copy of production's data (#15286). With `FORGE_HEAP_CAPTURE_DIR` set,
3 * the serving process writes `heap-<minutes up>.json` (`heapStats()` object counts) there at boot
4 * and every `FORGE_HEAP_CAPTURE_MIN` minutes (default 20), and the heap snapshot beside it
5 * (`heap-<minutes>.snapshot.json`). Unset, nothing runs: `heapStats()` held the loop ~280 ms and a
6 * snapshot far longer, so this is for a copy, never production. Read the files with
7 * `bun scripts/heap-growth.ts <before> <after>`.
8 */
9 import { mkdirSync, writeFileSync } from "node:fs";
10 import { tmpdir } from "node:os";
11 import { isAbsolute, join, relative, resolve } from "node:path";
12 import { heapStats } from "bun:jsc";
13 import { during } from "./jobs.ts";
14
15 export interface CaptureConfig { dir: string; everyMs: number }
16
17 /** True when `dir` is `root` or inside it, after resolving `..`. */
18 function within(root: string, dir: string): boolean {
19 const rel = relative(resolve(root), resolve(dir));
20 return rel === "" || (!rel.startsWith("..") && !isAbsolute(rel));
21 }
outdated

The dir comes from the environment unchecked and is mkdir'd and written to. Require it to resolve under FORGE_APP_DATA or the temp dir, else return undefined; test it.

22
23 /**
24 * The capture's settings from the environment, or nothing when it is off, the minutes are not a
25 * positive number, or the directory is outside `FORGE_APP_DATA` and the temp dir (nothing is
26 * written elsewhere at runtime).
27 */
28 export function captureConfig(env: Record<string, string | undefined> = process.env): CaptureConfig | undefined {
29 const dir = env.FORGE_HEAP_CAPTURE_DIR;
30 if (!dir) return undefined;
31 const roots = [env.FORGE_APP_DATA, tmpdir()].filter((r): r is string => !!r);
32 if (!roots.some((r) => within(r, dir))) return undefined;
33 const raw = env.FORGE_HEAP_CAPTURE_MIN;
34 const min = raw === undefined || raw === "" ? 20 : Number(raw);
35 if (!Number.isFinite(min) || min <= 0) return undefined;
36 return { dir, everyMs: min * 60_000 };
37 }
38
39 /** Write one capture; returns the counts file's path. */
40 export function capture(dir: string, upMs: number, snapshot = true): string {
41 mkdirSync(dir, { recursive: true });
42 const name = `heap-${String(Math.round(upMs / 60_000)).padStart(4, "0")}`;
43 Bun.gc(true); // what is still held, not what is waiting to be collected
44 const h = heapStats();
45 const counts = join(dir, `${name}.json`);
46 writeFileSync(counts, JSON.stringify({ upMs, heapSize: h.heapSize, extraMemorySize: h.extraMemorySize, objectTypeCounts: h.objectTypeCounts }));
47 if (snapshot) writeFileSync(join(dir, `${name}.snapshot.json`), JSON.stringify(Bun.generateHeapSnapshot()));
48 return counts;
49 }
50
51 /** Start capturing; returns the stop. Does nothing when the environment does not ask. */
52 export function startHeapCapture(env: Record<string, string | undefined> = process.env, log?: (m: string) => void): () => void {
53 const config = captureConfig(env);
54 if (!config) return () => {};
55 const started = Date.now();
56 const once = () => {
57 try { log?.(`heap captured: ${capture(config.dir, Date.now() - started)}`); }
58 catch (error) { log?.(`heap capture failed: ${String(error)}`); }
59 };
60 once();
61 const timer = setInterval(() => during("heap-capture", once), config.everyMs);
62 timer.unref?.();
63 return () => clearInterval(timer);
64 }
src/perf/heap_diff.ts added +75
@@ -0,0 +1,75 @@
1 /**
2 * What grew on the heap, and who holds it (#15286). Plain data in, plain data out: two
3 * `heapStats().objectTypeCounts` readings give the types that grew; one
4 * `Bun.generateHeapSnapshot()` (JSC's "Inspector" format, version 3: nodes are
5 * `id, size, classNameIndex, flags`; edges are `fromId, toId, typeIndex, dataIndex`) gives, for a
6 * type, the (holder type, edge name) pairs that retain the most of it. Never run on the serving
7 * process: a snapshot walks the whole heap and held the loop for seconds (see `heapCapture`, opt-in
8 * on a copy).
9 */
10
11 export type Counts = Readonly<Record<string, number>>;
12
13 export interface Growth { type: string; before: number; after: number; delta: number }
14
15 /** Types whose count rose, largest rise first; `limit` of them. */
16 export function grownTypes(before: Counts, after: Counts, limit = 15): Growth[] {
17 const out: Growth[] = [];
18 for (const [type, n] of Object.entries(after)) {
19 const was = before[type] ?? 0;
20 if (Number.isFinite(n) && n > was) out.push({ type, before: was, after: n, delta: n - was });
21 }
22 return out.sort((a, b) => b.delta - a.delta || a.type.localeCompare(b.type)).slice(0, limit);
23 }
24
25 /** The parts of a JSC snapshot this module reads. */
26 export interface Snapshot {
27 nodes: readonly number[];
28 nodeClassNames: readonly string[];
29 edges: readonly number[];
30 edgeNames: readonly string[];
31 edgeTypes: readonly string[];
32 }
33
34 export interface Retainer { holder: string; edge: string; count: number; bytes: number }
35
36 const NODE = 4;
37 const EDGE = 4;
38
39 /** Bytes of every node by class, largest first. */
40 export function sizeByClass(s: Snapshot, limit = 15): { type: string; count: number; bytes: number }[] {
41 const by = new Map<number, { count: number; bytes: number }>();
42 for (let i = 0; i + NODE <= s.nodes.length; i += NODE) {
43 const c = s.nodes[i + 2]!;
44 const e = by.get(c) ?? { count: 0, bytes: 0 };
45 e.count++;
46 e.bytes += s.nodes[i + 1]!;
47 by.set(c, e);
48 }
49 return [...by].map(([c, e]) => ({ type: s.nodeClassNames[c] ?? `#${c}`, ...e }))
50 .sort((a, b) => b.bytes - a.bytes).slice(0, limit);
51 }
52
53 /**
54 * The holders of nodes of `type`: for every edge pointing at one, the holder's class and the edge's
55 * name (a property name, or `[]` for an index), summed by the size of what is pointed at.
56 */
57 export function retainersOf(s: Snapshot, type: string, limit = 10): Retainer[] {
58 const index = new Map<number, number>(); // node id -> offset
59 for (let i = 0; i + NODE <= s.nodes.length; i += NODE) index.set(s.nodes[i]!, i);
60 const by = new Map<string, Retainer>();
61 for (let i = 0; i + EDGE <= s.edges.length; i += EDGE) {
62 const to = index.get(s.edges[i + 1]!);
63 if (to === undefined || s.nodeClassNames[s.nodes[to + 2]!] !== type) continue;
64 const from = index.get(s.edges[i]!);
65 const holder = from === undefined ? "?" : (s.nodeClassNames[s.nodes[from + 2]!] ?? "?");
66 const kind = s.edgeTypes[s.edges[i + 2]!];
67 const edge = kind === "Index" ? "[]" : (s.edgeNames[s.edges[i + 3]!] ?? "?");
68 const key = `${holder}\u0000${edge}`;
69 const r = by.get(key) ?? { holder, edge, count: 0, bytes: 0 };
70 r.count++;
71 r.bytes += s.nodes[to + 1]!;
72 by.set(key, r);
73 }
74 return [...by.values()].sort((a, b) => b.bytes - a.bytes).slice(0, limit);
75 }
test/heap-diff.test.ts added +52
@@ -0,0 +1,52 @@
1 import { expect, test } from "bun:test";
2 import { mkdtempSync, readdirSync, readFileSync } from "node:fs";
3 import { tmpdir } from "node:os";
4 import { join } from "node:path";
5 import { grownTypes, retainersOf, sizeByClass, type Snapshot } from "../src/perf/heap_diff.ts";
6 import { capture, captureConfig } from "../src/perf/heap_capture.ts";
7
8 test("grownTypes names what rose, largest first, and ignores what fell", () => {
9 const g = grownTypes({ A: 10, B: 5, C: 9 }, { A: 12, B: 105, C: 3, D: 4 });
10 expect(g.map((x) => [x.type, x.delta])).toEqual([["B", 100], ["D", 4], ["A", 2]]);
11 });
12
13 // Hand-built: node 1 (Map) holds nodes 2 and 3 (Request, 100 bytes each) by "entries"; node 4
14 // (Array) holds 3 by index.
15 const snap: Snapshot = {
16 nodeClassNames: ["Map", "Request", "Array"],
17 edgeTypes: ["Internal", "Property", "Index", "Variable"],
18 edgeNames: ["entries"],
19 nodes: [1, 50, 0, 0, 2, 100, 1, 0, 3, 100, 1, 0, 4, 20, 2, 0],
20 edges: [1, 2, 1, 0, 1, 3, 1, 0, 4, 3, 2, 0],
21 };
22
23 test("retainersOf groups the holders of a type by holder type and edge, by bytes", () => {
24 expect(retainersOf(snap, "Request")).toEqual([
25 { holder: "Map", edge: "entries", count: 2, bytes: 200 },
26 { holder: "Array", edge: "[]", count: 1, bytes: 100 },
27 ]);
28 expect(sizeByClass(snap)[0]).toEqual({ type: "Request", count: 2, bytes: 200 });
29 });
30
31 test("the capture is off without a directory and rejects a bad interval", () => {
32 const dir = join(tmpdir(), "heap-x");
33 expect(captureConfig({})).toBeUndefined();
34 expect(captureConfig({ FORGE_HEAP_CAPTURE_DIR: dir, FORGE_HEAP_CAPTURE_MIN: "0" })).toBeUndefined();
35 expect(captureConfig({ FORGE_HEAP_CAPTURE_DIR: dir })?.everyMs).toBe(20 * 60_000);
36 });
37
outdated

A full gc, heapStats and a real snapshot of the whole test process: 3.3 s under load, over the 2 s budget. Slim it as described in the review body.

38 test("the capture directory must be under FORGE_APP_DATA or the temp dir", () => {
39 const env = { FORGE_APP_DATA: "/srv/app-data" };
40 expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/srv/app-data/heap" })?.dir).toBe("/srv/app-data/heap");
41 expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: join(tmpdir(), "h") })).toBeDefined();
42 expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/etc/heap" })).toBeUndefined();
43 expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/srv/app-data/../etc" })).toBeUndefined();
44 expect(captureConfig({ ...env, FORGE_HEAP_CAPTURE_DIR: "/srv/app-data-evil" })).toBeUndefined();
45 });
46
47 test("a capture writes object counts, and a snapshot only when asked", () => {
48 const dir = mkdtempSync(join(tmpdir(), "heap-"));
49 const path = capture(dir, 3 * 60_000, false);
50 expect(readdirSync(dir)).toEqual(["heap-0003.json"]);
51 expect(typeof JSON.parse(readFileSync(path, "utf8")).objectTypeCounts).toBe("object");
52 });
test/job-slices.test.ts +1
@@ -297,6 +297,7 @@ const UNTIMED: Record<string, string> = {
297297 "cell output tail": "runs in a cell's supervisor process, not the host: reads a bounded tail of the child's output file and trims it; needs a supervised cell",
298298 "cell egress flush": "runs in a cell's supervisor process, not the host: appends at most 64 summed byte lines to the events file every 5 s; needs a supervised cell",
299299 "cell allowed sync": "runs in a cell's supervisor process, not the host: one small file write of the allowed hosts; needs a supervised cell",
300 "heap-capture": "opt-in (FORGE_HEAP_CAPTURE_DIR) for a copy of production's data: heapStats and a heap snapshot hold the loop for hundreds of ms by design (perf/heap_capture.ts); heap-diff.test.ts drives capture directly",
300301 "cell-watch": "watches running app cells' cgroups; needs cells, skipped in this sandbox mode",
301302 "cell-probe-deadline": "a cell's start probe running out of time: needs cells, skipped in this sandbox mode",
302303 "cell-kill-root": "kills a cell's root process after its grace: needs cells, skipped in this sandbox mode",

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs