Public: anyone can read this; the crew can change it.

Sentinel: memory-growth was degraded 10 minutes today: The forge's own memory is 2303 MiB, over twice its first-hour 965 MiB; the main VM's malloc grew most (761 MiB to 1901 … #14283

opened this

Check · Review · Land · Live — Pushed since the last review. Next: reviewer, pushed since the last review.

Ready to land

mason/forge:mason/14264-sentinel-memory-growth-was-degraded-10-m → main 2 commits fork 1 thread · 1 unresolved

Review

Sign in to review this change.
6 files changed +141 −112
docs/features/memory-growth.md +3 −3
@@ -4,13 +4,13 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang
44
55 ## Where it shows
66
7 - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. Every worker VM writes `memory worker <name>: heap …, malloc …, process rss …` on its own minute (#12925). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log.
7 - **The minute line.** Each minute the log carries `memory main: rss …, heap …, process mimalloc committed … (peak …), up … min`. Every worker VM writes `memory worker <name>: heap …, process rss …` on its own minute (#12925). Only the heap is one VM's own: Bun's `memoryUsage()` is the whole process's, and its `current` is the RSS itself, not malloc (#14264). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log.
88 - **What is held.** Beside the minute line the log carries `memory holds: <name> <count>, …`: the forge's long-lived Maps, Sets and lists by name with how many entries each holds now, largest first, twelve at most (`src/perf/held.ts`). A worker whose thread holds anything writes `memory worker <name> holds: …`. A name whose count only rises from line to line is the holder to read.
99 - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample.
1010 - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured.
1111 - `did not fall: no collection seen` means the stall was not the collector's.
12 - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z."
13 - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of each part: `JS heap` (`heapSize()`), `malloc` beyond the heap (`memoryUsage().current - heap`, floored at 0) and, for main only, `native memory outside every VM's malloc` (RSS less main's malloc and every reporting worker's, floored at 0). A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its malloc stops counting against native. When no part is above its own hour the sentence says so.
12 - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's JS heap grew most (100 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z."
13 - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of its `JS heap` (`heapSize()`); main also keeps the process's `memory outside every JS heap` (RSS less main's heap and every reporting worker's, floored at 0: malloc, SQLite's pages, native buffers). That part is the process's, not one VM's, and the sentence says so; mimalloc's commit is printed on the line but is never a part, since it overlaps the heaps. A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its heap stops counting against the outside. When no part is above its own hour the sentence says so.
1414 - **The worker names.** `engine` (src/engine/worker.ts); `disk`, `tree` (src/apps); `lanes`, `pages`, `pipeline` (src/flow); `scan`, `du`, `leak` (the `answerAsks` workers of src/kernel/worker_ask.ts); `thumb` (src/looks); `log-writer` (src/runs); `mirror`, `vacuum` (src/snapshot); `code-rows`, `map`, `raster` (src/web). A one-shot worker (`mirror`, `vacuum`) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive.
1515
1616 ## What to do when the need shows
src/boards/memory.ts +1 −1
@@ -76,7 +76,7 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS
7676 try {
7777 const reading = readNow();
7878 recordMemory(app.db, reading.rss, reading.heap, now);
79 memoryWatch.note(reading.rss, reading.heap, now, reading.current);
79 memoryWatch.note(reading.rss, reading.heap, now);
8080 app.log.info(memoryLine(reading, process.uptime() * 1000));
8181 app.log.info(heldLine(heldNow(app)));
8282 const allocated = allocationLine(allocationsOf(app));
src/perf/memory_watch.ts +52 −54
@@ -6,11 +6,18 @@
66 * RSS passes twice it (`grown`), which the needs-you line reads (`kernel/vitals.ts`).
77 *
88 * It names what grew (#12951): each owner, the main VM and every worker by name, keeps its
9 * own first-hour mean of each part (JS heap, malloc beyond the heap, and for main the RSS
10 * outside every VM's malloc), and `grown` names the part that grew the most bytes past its
11 * baseline. A worker's reading reaches the serving VM on a `BroadcastChannel` the worker
12 * posts to from the same minute timer that writes its line (`tellReading`, `hearWorkers`),
13 * so no ask and no hot-path message carries it; a worker silent for five minutes is not blamed.
9 * own first-hour mean of its JS heap, main also of the RSS outside every VM's heap, and
10 * `grown` names the part that grew the most bytes past its baseline.
11 *
12 * What Bun can tell apart (#14264): `heapSize()` is one VM's own; `jsc.memoryUsage()` is the
13 * whole process's, and its `current` is the RSS itself (not malloc: reading `current - heap`
14 * as a VM's malloc blamed "the main VM's malloc" for every worker's heap and every native
15 * byte). Its `currentCommit` is mimalloc's commit for every thread at once, and overlaps the
16 * heaps (`heapSize()` counts the buffers its objects own), so it is printed on the line and
17 * is never a part. What is outside every heap belongs to the process, not to one VM. A worker's
18 * reading reaches the serving VM on a `BroadcastChannel` the worker posts to from the same minute
19 * timer that writes its line (`tellReading`, `hearWorkers`), so no ask and no hot-path message
20 * carries it; a worker silent for five minutes is not blamed.
1421 *
1522 * Beside each line, what is held by name (`perf/held.ts`): the serving forge's holders
1623 * and the thread's on `memory holds:`, a worker's own thread's on its second line.
@@ -39,27 +46,27 @@ const mib = (bytes: number): string => `${(bytes / MIB).toFixed(bytes < 100 * MI
3946
4047 export interface Reading {
4148 readonly rss: number;
49 /** This VM's own JS heap (`heapSize()`). */
4250 readonly heap: number;
43 /** `jsc.memoryUsage()`: the allocator's current and peak bytes, and peak committed. */
44 readonly current: number;
45 readonly peak: number;
51 /** `jsc.memoryUsage().currentCommit` and its peak: mimalloc's, for the whole process. */
52 readonly commit: number;
4653 readonly peakCommit: number;
4754 }
4855
4956 /** One reading of this VM: constant time, never `heapStats()`. */
5057 export function readNow(rss: number = process.memoryUsage().rss): Reading {
5158 const m = jscMemory();
52 return { rss, heap: heapSize(), current: m.current, peak: m.peak, peakCommit: m.peakCommit };
59 return { rss, heap: heapSize(), commit: m.currentCommit, peakCommit: m.peakCommit };
5360 }
5461
5562 /** The serving VM's minute line: `memory main: rss 5.1 GiB, heap …`. */
5663 export function memoryLine(r: Reading, uptimeMs: number): string {
57 return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}, committed peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`;
64 return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, process mimalloc committed ${mib(r.commit)} (peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`;
5865 }
5966
60 /** A worker VM's line: its heap and malloc are its own; RSS is the process's. */
67 /** A worker VM's line: its heap is its own; RSS is the process's. */
6168 export function workerMemoryLine(name: string, r: Reading): string {
62 return `memory worker ${name}: heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}), process rss ${mib(r.rss)}`;
69 return `memory worker ${name}: heap ${mib(r.heap)}, process rss ${mib(r.rss)}`;
6370 }
6471
6572 export interface Grown {
@@ -72,12 +79,12 @@ export interface Grown {
7279 }
7380
7481 /**
75 * The three parts of a VM's reading (#12951): `heap` is the JS heap (`heapSize()`),
76 * `malloc` the allocator's bytes beyond it (`memoryUsage().current - heap`, floored at 0),
77 * `native` the process's RSS outside every VM's malloc (main only: RSS is the process's).
82 * The parts a reading splits into (#12951, #14264): `heap` is one VM's JS heap
83 * (`heapSize()`); `outside` is the process's RSS less every fresh VM's heap, floored at 0
84 * (malloc, SQLite's pages, native buffers, code), kept on main since RSS is the process's.
7885 */
79 export type Part = "heap" | "malloc" | "native";
80 const PARTS: readonly Part[] = ["heap", "malloc", "native"];
86 export type Part = "heap" | "outside";
87 const PARTS: readonly Part[] = ["heap", "outside"];
8188
8289 export interface Growth {
8390 /** `main`, or the worker's name as its `memory worker <name>` line gives it. */
@@ -87,11 +94,10 @@ export interface Growth {
8794 readonly toBytes: number;
8895 }
8996
90 /** A worker's minute reading as it crosses to the serving VM. */
97 /** A worker's minute reading as it crosses to the serving VM: its own heap. */
9198 export interface WorkerReading {
9299 readonly name: string;
93100 readonly heap: number;
94 readonly current: number;
95101 }
96102
97103 /** A worker silent this long is not blamed: it ended, or it stopped reporting. */
@@ -105,8 +111,8 @@ const bytes = (n: unknown): n is number => typeof n === "number" && Number.isFin
105111 export function workerReadingOf(data: unknown): WorkerReading | undefined {
106112 if (typeof data !== "object" || data === null) return undefined;
107113 const d = data as Record<string, unknown>;
108 if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap) || !bytes(d.current)) return undefined;
109 return { name: d.name, heap: d.heap, current: d.current };
114 if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap)) return undefined;
115 return { name: d.name, heap: d.heap };
110116 }
111117
112118 /** One owner's first-hour sums per part, its baseline once that hour is over, its latest parts. */
@@ -116,18 +122,18 @@ interface Owner {
116122 readonly count: Record<Part, number>;
117123 base: Partial<Record<Part, number>> | undefined;
118124 latest: Partial<Record<Part, number>>;
119 /** The latest reading's malloc, which main's native excludes. */
120 malloc: number;
125 /** The latest reading's heap, which main's `outside` excludes. */
126 heap: number;
121127 at: number;
122128 }
123129
124130 const owner = (startedAt: number): Owner => ({
125131 startedAt,
126 sum: { heap: 0, malloc: 0, native: 0 },
127 count: { heap: 0, malloc: 0, native: 0 },
132 sum: { heap: 0, outside: 0 },
133 count: { heap: 0, outside: 0 },
128134 base: undefined,
129135 latest: {},
130 malloc: 0,
136 heap: 0,
131137 at: startedAt,
132138 });
133139
@@ -167,11 +173,8 @@ function mostGrown(owners: Iterable<readonly [string, Owner]>, now: number): Gro
167173 }
168174
169175 export interface MemoryWatch {
170 /**
171 * The serving VM's minute reading. `malloc` (`memoryUsage().current`) splits it into
172 * heap, malloc beyond the heap and native; without it only the heap is kept per part.
173 */
174 note(rss: number, heap: number, now?: number, malloc?: number): void;
176 /** The serving VM's minute reading: the process's RSS and main's own heap. */
177 note(rss: number, heap: number, now?: number): void;
175178 /** A worker's minute reading (`hearWorkers` carries it here), against its own first hour. */
176179 noteWorker(r: WorkerReading, now?: number): void;
177180 /** RSS at least twice its first-hour mean, with the part that grew most, or undefined. */
@@ -193,20 +196,15 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
193196 const main = owner(startedAt);
194197 const workers = new Map<string, Owner>();
195198 return {
196 note(rss, heap, now = Date.now(), malloc) {
199 note(rss, heap, now = Date.now()) {
197200 if (!Number.isFinite(rss) || rss <= 0) return;
198201 last = { heap, rss, at: now };
199202 latest = rss;
200 const parts: Partial<Record<Part, number>> = { heap };
201 if (bytes(malloc)) {
202 // Every fresh worker's malloc is its own VM's, inside the process's RSS: not native.
203 let others = 0;
204 for (const w of workers.values()) if (now - w.at <= WORKER_STALE_MS) others += w.malloc;
205 parts.malloc = Math.max(0, malloc - heap);
206 parts.native = Math.max(0, rss - malloc - others);
207 main.malloc = malloc;
208 }
209 take(main, parts, now);
203 // Every fresh worker's heap is its own VM's, inside the process's RSS: not outside.
204 let others = 0;
205 for (const w of workers.values()) if (now - w.at <= WORKER_STALE_MS) others += w.heap;
206 main.heap = heap;
207 take(main, { heap, outside: Math.max(0, rss - heap - others) }, now);
210208 if (base === undefined) {
211209 if (now - startedAt < FIRST_HOUR_MS) {
212210 sum += rss;
@@ -226,8 +224,8 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
226224 o = owner(now);
227225 workers.set(r.name, o);
228226 }
229 o.malloc = r.current;
230 take(o, { heap: r.heap, malloc: Math.max(0, r.current - r.heap) }, now);
227 o.heap = r.heap;
228 take(o, { heap: r.heap }, now);
231229 },
232230 grown() {
233231 if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined;
@@ -246,21 +244,21 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
246244 /** This process's watch, started with it. */
247245 export const memoryWatch: MemoryWatch = createMemoryWatch();
248246
249 const PART_WORDS: Record<Part, string> = {
250 heap: "JS heap",
251 malloc: "malloc",
252 native: "native memory outside every VM's malloc",
253 };
254
255 /** Who grew, in words: `the main VM's JS heap`, `worker pages's malloc`. */
256 const grewWords = (g: Growth): string => `${g.owner === "main" ? "the main VM" : `worker ${g.owner}`}'s ${PART_WORDS[g.part]}`;
247 /**
248 * Who grew, in words: `the main VM's JS heap`, `worker pages's JS heap`, or the process's
249 * memory outside every JS heap, which no one VM owns.
250 */
251 const grewWords = (g: Growth): string =>
252 g.part === "outside"
253 ? "the process's memory outside every JS heap (malloc, SQLite, native buffers)"
254 : `${g.owner === "main" ? "the main VM" : `worker ${g.owner}`}'s JS heap`;
257255
258256 /** The needs-you sentence for a grown process: the figure, and whose part grew most (#12951). */
259257 export function grownSentence(g: Grown): string {
260258 const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}`;
261259 const since = new Date(g.since).toISOString();
262260 if (g.grew === undefined)
263 return `${head}; it began growing at ${since}, and no VM's heap or malloc is above its own first hour.`;
261 return `${head}; it began growing at ${since}, and no VM's heap, nor the memory outside them, is above its own first hour.`;
264262 return `${head}; ${grewWords(g.grew)} grew most (${mib(g.grew.fromBytes)} to ${mib(g.grew.toBytes)}) since ${since}.`;
265263 }
266264
@@ -280,7 +278,7 @@ export function tellReading(name: string, r: Reading): Reading {
280278 told = new BroadcastChannel(MEMORY_CHANNEL);
281279 (told as { unref?: () => void }).unref?.();
282280 }
283 told.postMessage({ name, heap: r.heap, current: r.current } satisfies WorkerReading);
281 told.postMessage({ name, heap: r.heap } satisfies WorkerReading);
284282 } catch {
285283 // A closed channel loses one minute's reading; the line is still written.
286284 }
test/memory-watch.test.ts +79 −48
@@ -29,7 +29,11 @@ test("raises once RSS passes twice the first-hour mean, with the figure and when
2929 w.note(3900 * MIB, 1, began + 60_000); // 1.95x
3030 expect(w.grown()).toBeUndefined();
3131 w.note(4100 * MIB, 1, began + 120_000); // 2.05x
32 expect(w.grown()).toEqual({ baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: began });
32 // Main's heap fell to 1 byte, so every RSS byte past it is outside: that part grew most.
33 expect(w.grown()).toEqual({
34 baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: began,
35 grew: { owner: "main", part: "outside", fromBytes: 1000 * MIB, toBytes: 4100 * MIB - 1 },
36 });
3337 expect(grownSentence(w.grown()!)).toContain("4100 MiB, over twice its first-hour 2000 MiB");
3438 expect(grownSentence(w.grown()!)).toContain(new Date(began).toISOString());
3539 });
@@ -62,30 +66,36 @@ test("a stall's heap with no argument is read from the VM", () => {
6266 });
6367
6468 test("the minute lines name the owner and the figures", () => {
65 const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, current: 512 * MIB, peak: 600 * MIB, peakCommit: 700 * MIB };
66 expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, malloc 512 MiB (peak 600 MiB, committed peak 700 MiB), up 60 min");
67 expect(workerMemoryLine("engine", r)).toStartWith("memory worker engine: heap 1024 MiB");
69 const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, commit: 512 * MIB, peakCommit: 700 * MIB };
70 expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, process mimalloc committed 512 MiB (peak 700 MiB), up 60 min");
71 expect(workerMemoryLine("engine", r)).toBe("memory worker engine: heap 1024 MiB, process rss 5120 MiB");
6872 expect(readNow().heap).toBeGreaterThan(0);
outdated

Compares a live jsc reading with a RSS taken at a different instant. This can flake, and it tests Bun, not this code. Remove the RSS comparison and keep the key-shape assertion.

6973 });
7074
71 // #12951: which VM grew, and which part. Main's first hour: rss 2000, heap 500, malloc 800
72 // (so 300 beyond the heap); worker pages: heap 100, malloc 410 (310 beyond); worker lanes:
73 // heap 50, malloc 90; main's native is rss less every VM's malloc: 2000 - 800 - 410 - 90 = 700 MiB.
75 // #14264: `jsc.memoryUsage().current` is the RSS, not malloc. Read as "malloc", RSS less
76 // main's heap blamed the main VM for every worker's heap and every native byte.
77 test("no part is read from Bun's memoryUsage().current", () => {
78 expect(Object.keys(readNow()).sort()).toEqual(["commit", "heap", "peakCommit", "rss"]);
79 });
80
81 // #12951: which VM grew, and which part. First hour: rss 2000, main heap 500, worker pages
82 // heap 100, worker lanes heap 50; outside every heap: 2000 - 500 - 100 - 50 = 1350 MiB.
7483 const HOUR = T0 + FIRST_HOUR_MS;
7584 function owners(w: ReturnType<typeof createMemoryWatch>) {
7685 for (let i = 0; i < 12; i++) {
7786 const at = T0 + i * 300_000;
78 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, at);
79 w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at);
80 w.note(2000 * MIB, 500 * MIB, at, 800 * MIB);
87 w.noteWorker({ name: "pages", heap: 100 * MIB }, at);
88 w.noteWorker({ name: "lanes", heap: 50 * MIB }, at);
89 w.note(2000 * MIB, 500 * MIB, at);
8190 }
8291 }
8392
8493 test("main heap growth is named as the main VM's JS heap", () => {
8594 const w = createMemoryWatch(T0);
8695 owners(w);
87 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, HOUR + 60_000);
88 w.note(4700 * MIB, 3100 * MIB, HOUR + 60_000, 3400 * MIB);
96 w.noteWorker({ name: "pages", heap: 100 * MIB }, HOUR + 60_000);
97 w.noteWorker({ name: "lanes", heap: 50 * MIB }, HOUR + 60_000);
98 w.note(4700 * MIB, 3100 * MIB, HOUR + 60_000); // outside: 4700 - 3100 - 150 = 1450, +100
8999 const g = w.grown()!;
90100 expect(g.grew).toEqual({ owner: "main", part: "heap", fromBytes: 500 * MIB, toBytes: 3100 * MIB });
91101 expect(grownSentence(g)).toBe(
@@ -93,72 +103,93 @@ test("main heap growth is named as the main VM's JS heap", () => {
93103 );
94104 });
95105
96 test("one worker's malloc growth is named as that worker, not as main's native memory", () => {
106 test("one worker's heap growth is named as that worker, not as the main VM's", () => {
97107 const w = createMemoryWatch(T0);
98108 owners(w);
99109 const at = HOUR + 120_000;
100 w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at);
101 w.noteWorker({ name: "pages", heap: 110 * MIB, current: 2510 * MIB }, at); // 2400 beyond its heap
102 w.note(4200 * MIB, 520 * MIB, at, 800 * MIB); // native: 4200 - 800 - 2510 - 90 = 800, +100 over its hour
110 w.noteWorker({ name: "lanes", heap: 50 * MIB }, at);
111 w.noteWorker({ name: "pages", heap: 2400 * MIB }, at);
112 w.note(4200 * MIB, 520 * MIB, at); // outside: 4200 - 520 - 2450 = 1230, under its hour
103113 const g = w.grown()!;
104 expect(g.grew).toEqual({ owner: "pages", part: "malloc", fromBytes: 310 * MIB, toBytes: 2400 * MIB });
105 expect(grownSentence(g)).toContain("2000 MiB; worker pages's malloc grew most (310 MiB to 2400 MiB) since ");
114 expect(g.grew).toEqual({ owner: "pages", part: "heap", fromBytes: 100 * MIB, toBytes: 2400 * MIB });
115 expect(grownSentence(g)).toContain("2000 MiB; worker pages's JS heap grew most (100 MiB to 2400 MiB) since ");
116 });
117
118 test("growth outside every heap is the process's, never one VM's malloc", () => {
119 const w = createMemoryWatch(T0);
120 owners(w);
121 const at = HOUR + 60_000;
122 w.noteWorker({ name: "pages", heap: 110 * MIB }, at);
123 w.noteWorker({ name: "lanes", heap: 50 * MIB }, at);
124 w.note(4100 * MIB, 540 * MIB, at); // outside: 4100 - 540 - 160 = 3400
125 const g = w.grown()!;
126 expect(g.grew).toEqual({ owner: "main", part: "outside", fromBytes: 1350 * MIB, toBytes: 3400 * MIB });
127 const said = grownSentence(g);
128 expect(said).toContain("the process's memory outside every JS heap (malloc, SQLite, native buffers) grew most (1350 MiB to 3400 MiB)");
129 expect(said).not.toContain("main VM's malloc");
106130 });
107131
108132 test("growth under twice the first hour names nothing", () => {
109133 const w = createMemoryWatch(T0);
110134 owners(w);
111 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2000 * MIB }, HOUR + 60_000);
112 w.note(3900 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB); // 1.95x
135 w.noteWorker({ name: "pages", heap: 1700 * MIB }, HOUR + 60_000);
136 w.note(3900 * MIB, 500 * MIB, HOUR + 60_000); // 1.95x
113137 expect(w.grown()).toBeUndefined();
114138 });
115139
116140 test("a worker that stopped reporting is not blamed", () => {
117141 const w = createMemoryWatch(T0);
118142 owners(w);
119 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000); // its last word
120 w.note(4200 * MIB, 700 * MIB, HOUR + 60_000, 1000 * MIB);
143 w.noteWorker({ name: "pages", heap: 3000 * MIB }, HOUR + 60_000); // its last word
144 w.note(4200 * MIB, 700 * MIB, HOUR + 60_000);
121145 expect(w.grown()!.grew?.owner).toBe("pages"); // still reporting: named
122146 const later = HOUR + 60_000 + WORKER_STALE_MS + 60_000;
123 w.note(4200 * MIB, 700 * MIB, later, 1000 * MIB);
124 // Silent five minutes (lanes too): their malloc is no longer subtracted from native either,
125 // so main's own native (4200 - 1000 = 3200, against 2000 - 800 - 410 - 90 = 700) grew most.
126 expect(w.grown()!.grew).toEqual({ owner: "main", part: "native", fromBytes: 700 * MIB, toBytes: 3200 * MIB });
127 expect(w.grown()!.grew?.owner).not.toBe("pages");
147 w.note(4200 * MIB, 700 * MIB, later);
148 // Silent five minutes (lanes too): their heaps are no longer subtracted from outside either,
149 // so outside (4200 - 700 = 3500, against 1350) grew most.
150 expect(w.grown()!.grew).toEqual({ owner: "main", part: "outside", fromBytes: 1350 * MIB, toBytes: 3500 * MIB });
128151 });
129152
130 test("with nothing above its own hour the sentence says so; an old caller without malloc still names the heap", () => {
153 test("with nothing above its own hour the sentence says so", () => {
131154 const w = createMemoryWatch(T0);
132 firstHour(w, 2000 * MIB); // heap 1000, no malloc
133 w.note(4100 * MIB, 900 * MIB, HOUR + 60_000);
155 owners(w);
156 const at = HOUR + 60_000;
157 w.noteWorker({ name: "pages", heap: 100 * MIB }, at);
158 w.noteWorker({ name: "lanes", heap: 50 * MIB }, at);
159 // A worker started after main's hour has no first hour yet: its heap is never "outside",
160 // and it cannot be named either. Outside: 4100 - 500 - 150 - 2100 = 1350, its own hour.
161 w.noteWorker({ name: "map", heap: 2100 * MIB }, at);
162 w.note(4100 * MIB, 500 * MIB, at);
134163 const g = w.grown()!;
164 expect(g).toEqual({ baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: at });
135165 expect(g.grew).toBeUndefined();
136 expect(grownSentence(g)).toContain("no VM's heap or malloc is above its own first hour");
137 w.note(4100 * MIB, 1500 * MIB, HOUR + 120_000);
138 expect(w.grown()!.grew).toEqual({ owner: "main", part: "heap", fromBytes: 1000 * MIB, toBytes: 1500 * MIB });
166 expect(grownSentence(g)).toContain("no VM's heap, nor the memory outside them, is above its own first hour");
139167 });
140168
141169 test("a worker's first hour is its own, counted from its first reading", () => {
142170 const w = createMemoryWatch(T0);
143171 owners(w);
144172 const late = HOUR + 600_000; // a worker started on its first ask, after main's hour
145 for (let i = 0; i < 12; i++) w.noteWorker({ name: "map", heap: 40 * MIB, current: 60 * MIB }, late + i * 300_000);
173 for (let i = 0; i < 12; i++) w.noteWorker({ name: "map", heap: 40 * MIB }, late + i * 300_000);
146174 const at = late + FIRST_HOUR_MS + 60_000;
147 w.noteWorker({ name: "map", heap: 40 * MIB, current: 2060 * MIB }, at);
148 w.note(4500 * MIB, 500 * MIB, at, 800 * MIB);
149 expect(w.grown()!.grew).toEqual({ owner: "map", part: "malloc", fromBytes: 20 * MIB, toBytes: 2020 * MIB });
175 w.noteWorker({ name: "pages", heap: 100 * MIB }, at);
176 w.noteWorker({ name: "lanes", heap: 50 * MIB }, at);
177 w.noteWorker({ name: "map", heap: 2040 * MIB }, at);
178 w.note(4500 * MIB, 500 * MIB, at); // outside: 4500 - 500 - 2190 = 1810, +460
179 expect(w.grown()!.grew).toEqual({ owner: "map", part: "heap", fromBytes: 40 * MIB, toBytes: 2040 * MIB });
150180 });
151181
152182 test("a worker's reading is validated at the boundary, and the owners are bounded", () => {
153 expect(workerReadingOf({ name: "pages", heap: 1, current: 2 })).toEqual({ name: "pages", heap: 1, current: 2 });
154 for (const bad of [null, 5, {}, { name: "Pages", heap: 1, current: 1 }, { name: "main", heap: 1, current: 1 }, { name: "a b", heap: 1, current: 1 },
155 { name: "x".repeat(41), heap: 1, current: 1 }, { name: "p", heap: -1, current: 1 }, { name: "p", heap: 1, current: Infinity }, { name: "p", heap: "1", current: 1 }])
183 expect(workerReadingOf({ name: "pages", heap: 1 })).toEqual({ name: "pages", heap: 1 });
184 expect(workerReadingOf({ name: "pages", heap: 1, current: 2 })).toEqual({ name: "pages", heap: 1 });
185 for (const bad of [null, 5, {}, { name: "Pages", heap: 1 }, { name: "main", heap: 1 }, { name: "a b", heap: 1 },
186 { name: "x".repeat(41), heap: 1 }, { name: "p", heap: -1 }, { name: "p", heap: Infinity }, { name: "p", heap: "1" }])
156187 expect(workerReadingOf(bad)).toBeUndefined();
157188 const w = createMemoryWatch(T0);
158 for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1, current: 1 }, T0);
189 for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1 }, T0);
159190 owners(w); // pages and lanes arrive past the bound: never kept, never named
160 w.note(4200 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB);
161 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000);
191 w.noteWorker({ name: "pages", heap: 3000 * MIB }, HOUR + 60_000);
192 w.note(4200 * MIB, 500 * MIB, HOUR + 60_000);
162193 expect(w.grown()!.grew?.owner).toBe("main");
163194 });
164195
@@ -168,16 +199,16 @@ test("a worker's reading crosses on the memory channel to the watch that hears i
168199 const { promise: two, resolve } = Promise.withResolvers<void>();
169200 const stop = hearWorkers({ noteWorker: (r) => { got.push(r); if (got.length === 2) resolve(); } });
170201 try {
171 const r = { rss: 9, heap: 3 * MIB, current: 7 * MIB, peak: 8, peakCommit: 9 };
202 const r = { rss: 9, heap: 3 * MIB, commit: 7 * MIB, peakCommit: 9 };
172203 expect(tellReading("pages", r)).toBe(r);
173204 const other = new BroadcastChannel(MEMORY_CHANNEL);
174 other.postMessage({ name: "../etc", heap: 1, current: 1 }); // malformed: dropped
175 other.postMessage({ name: "lanes", heap: 1, current: 2 }); // after it on one channel: it was heard first
205 other.postMessage({ name: "../etc", heap: 1 }); // malformed: dropped
206 other.postMessage({ name: "lanes", heap: 1 }); // after it on one channel: it was heard first
176207 await two;
177208 other.close();
178209 expect(got.sort((a, b) => a.name.localeCompare(b.name))).toEqual([
179 { name: "lanes", heap: 1, current: 2 },
180 { name: "pages", heap: 3 * MIB, current: 7 * MIB },
210 { name: "lanes", heap: 1 },
211 { name: "pages", heap: 3 * MIB },
181212 ]);
182213 } finally {
183214 stop();
test/vitals.test.ts +5 −5
@@ -587,13 +587,13 @@ test("the needs-you line names which VM grew and which part, in the watch's own
587587 const start = Date.now() - 2 * FIRST_HOUR_MS;
588588 const w = createMemoryWatch(start);
589589 for (let i = 0; i < 12; i++) {
590 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, start + i * 300_000);
591 w.note(2000 * MIB, 500 * MIB, start + i * 300_000, 800 * MIB);
590 w.noteWorker({ name: "pages", heap: 100 * MIB }, start + i * 300_000);
591 w.note(2000 * MIB, 500 * MIB, start + i * 300_000);
592592 }
593593 const at = start + FIRST_HOUR_MS + 60_000;
594 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2510 * MIB }, at);
595 w.note(4200 * MIB, 500 * MIB, at, 800 * MIB);
594 w.noteWorker({ name: "pages", heap: 2410 * MIB }, at);
595 w.note(4200 * MIB, 500 * MIB, at);
596596 const d = vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w }).degraded.find((x) => x.cause === "memory-growth")!;
597597 expect(d.sentence).toBe(grownSentence(w.grown()!));
598 expect(d.sentence).toContain("worker pages's malloc grew most (310 MiB to 2410 MiB)");
598 expect(d.sentence).toContain("worker pages's JS heap grew most (100 MiB to 2410 MiB)");
599599 });
test/worker-memory-lines.test.ts +1 −1
@@ -54,7 +54,7 @@ test("two workers each write their own memory line, named for themselves", async
5454 test("a worker's minute reading reaches the serving VM's watch by name (#12951)", async () => {
5555 const script = `
5656 import { hearWorkers } from ${JSON.stringify(join(ROOT, "src/perf/memory_watch.ts"))};
57 hearWorkers({ noteWorker: (r) => console.log("heard " + r.name + " " + (r.current >= r.heap)) });
57 hearWorkers({ noteWorker: (r) => console.log("heard " + r.name + " " + (r.heap > 0)) });
5858 new Worker(${JSON.stringify(join(ROOT, "src/flow/pipeline_worker.ts"))});
5959 `;
6060 const { out } = await runChild(script, "50", (o) => o.includes("heard pipeline"));

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs