Public: anyone can read this; the crew can change it.
Sentinel: memory-growth was degraded 10 minutes today: The forge's own memory is 2303 MiB, over twice its first-hour 965 MiB; the main VM's malloc grew most (761 MiB to 1901 … #14283
Check · Review · Land · Live — Pushed since the last review.
mason/forge:mason/14264-sentinel-memory-growth-was-degraded-10-m → main 2 commits fork 1 thread · 1 unresolved
6 files
changed +141
−112
docs/features/memory-growth.md +3 −3
| @@ -4,13 +4,13 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang | ||
| 4 | 4 | |
| 5 | 5 | ## Where it shows |
| 6 | 6 | |
| 7 | - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. Every worker VM writes `memory worker <name>: heap …, malloc …, process rss …` on its own minute (#12925). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log. | |
| 7 | - **The minute line.** Each minute the log carries `memory main: rss …, heap …, process mimalloc committed … (peak …), up … min`. Every worker VM writes `memory worker <name>: heap …, process rss …` on its own minute (#12925). Only the heap is one VM's own: Bun's `memoryUsage()` is the whole process's, and its `current` is the RSS itself, not malloc (#14264). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log. | |
| 8 | 8 | - **What is held.** Beside the minute line the log carries `memory holds: <name> <count>, …`: the forge's long-lived Maps, Sets and lists by name with how many entries each holds now, largest first, twelve at most (`src/perf/held.ts`). A worker whose thread holds anything writes `memory worker <name> holds: …`. A name whose count only rises from line to line is the holder to read. |
| 9 | 9 | - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample. |
| 10 | 10 | - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured. |
| 11 | 11 | - `did not fall: no collection seen` means the stall was not the collector's. |
| 12 | - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z." | |
| 13 | - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of each part: `JS heap` (`heapSize()`), `malloc` beyond the heap (`memoryUsage().current - heap`, floored at 0) and, for main only, `native memory outside every VM's malloc` (RSS less main's malloc and every reporting worker's, floored at 0). A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its malloc stops counting against native. When no part is above its own hour the sentence says so. | |
| 12 | - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's JS heap grew most (100 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z." | |
| 13 | - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of its `JS heap` (`heapSize()`); main also keeps the process's `memory outside every JS heap` (RSS less main's heap and every reporting worker's, floored at 0: malloc, SQLite's pages, native buffers). That part is the process's, not one VM's, and the sentence says so; mimalloc's commit is printed on the line but is never a part, since it overlaps the heaps. A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its heap stops counting against the outside. When no part is above its own hour the sentence says so. | |
| 14 | 14 | - **The worker names.** `engine` (src/engine/worker.ts); `disk`, `tree` (src/apps); `lanes`, `pages`, `pipeline` (src/flow); `scan`, `du`, `leak` (the `answerAsks` workers of src/kernel/worker_ask.ts); `thumb` (src/looks); `log-writer` (src/runs); `mirror`, `vacuum` (src/snapshot); `code-rows`, `map`, `raster` (src/web). A one-shot worker (`mirror`, `vacuum`) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive. |
| 15 | 15 | |
| 16 | 16 | ## What to do when the need shows |
src/boards/memory.ts +1 −1
| @@ -76,7 +76,7 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS | ||
| 76 | 76 | try { |
| 77 | 77 | const reading = readNow(); |
| 78 | 78 | recordMemory(app.db, reading.rss, reading.heap, now); |
| 79 | memoryWatch.note(reading.rss, reading.heap, now, reading.current); | |
| 79 | memoryWatch.note(reading.rss, reading.heap, now); | |
| 80 | 80 | app.log.info(memoryLine(reading, process.uptime() * 1000)); |
| 81 | 81 | app.log.info(heldLine(heldNow(app))); |
| 82 | 82 | const allocated = allocationLine(allocationsOf(app)); |
src/perf/memory_watch.ts +52 −54
| @@ -6,11 +6,18 @@ | ||
| 6 | 6 | * RSS passes twice it (`grown`), which the needs-you line reads (`kernel/vitals.ts`). |
| 7 | 7 | * |
| 8 | 8 | * It names what grew (#12951): each owner, the main VM and every worker by name, keeps its |
| 9 | * own first-hour mean of each part (JS heap, malloc beyond the heap, and for main the RSS | |
| 10 | * outside every VM's malloc), and `grown` names the part that grew the most bytes past its | |
| 11 | * baseline. A worker's reading reaches the serving VM on a `BroadcastChannel` the worker | |
| 12 | * posts to from the same minute timer that writes its line (`tellReading`, `hearWorkers`), | |
| 13 | * so no ask and no hot-path message carries it; a worker silent for five minutes is not blamed. | |
| 9 | * own first-hour mean of its JS heap, main also of the RSS outside every VM's heap, and | |
| 10 | * `grown` names the part that grew the most bytes past its baseline. | |
| 11 | * | |
| 12 | * What Bun can tell apart (#14264): `heapSize()` is one VM's own; `jsc.memoryUsage()` is the | |
| 13 | * whole process's, and its `current` is the RSS itself (not malloc: reading `current - heap` | |
| 14 | * as a VM's malloc blamed "the main VM's malloc" for every worker's heap and every native | |
| 15 | * byte). Its `currentCommit` is mimalloc's commit for every thread at once, and overlaps the | |
| 16 | * heaps (`heapSize()` counts the buffers its objects own), so it is printed on the line and | |
| 17 | * is never a part. What is outside every heap belongs to the process, not to one VM. A worker's | |
| 18 | * reading reaches the serving VM on a `BroadcastChannel` the worker posts to from the same minute | |
| 19 | * timer that writes its line (`tellReading`, `hearWorkers`), so no ask and no hot-path message | |
| 20 | * carries it; a worker silent for five minutes is not blamed. | |
| 14 | 21 | * |
| 15 | 22 | * Beside each line, what is held by name (`perf/held.ts`): the serving forge's holders |
| 16 | 23 | * and the thread's on `memory holds:`, a worker's own thread's on its second line. |
| @@ -39,27 +46,27 @@ const mib = (bytes: number): string => `${(bytes / MIB).toFixed(bytes < 100 * MI | ||
| 39 | 46 | |
| 40 | 47 | export interface Reading { |
| 41 | 48 | readonly rss: number; |
| 49 | /** This VM's own JS heap (`heapSize()`). */ | |
| 42 | 50 | readonly heap: number; |
| 43 | /** `jsc.memoryUsage()`: the allocator's current and peak bytes, and peak committed. */ | |
| 44 | readonly current: number; | |
| 45 | readonly peak: number; | |
| 51 | /** `jsc.memoryUsage().currentCommit` and its peak: mimalloc's, for the whole process. */ | |
| 52 | readonly commit: number; | |
| 46 | 53 | readonly peakCommit: number; |
| 47 | 54 | } |
| 48 | 55 | |
| 49 | 56 | /** One reading of this VM: constant time, never `heapStats()`. */ |
| 50 | 57 | export function readNow(rss: number = process.memoryUsage().rss): Reading { |
| 51 | 58 | const m = jscMemory(); |
| 52 | return { rss, heap: heapSize(), current: m.current, peak: m.peak, peakCommit: m.peakCommit }; | |
| 59 | return { rss, heap: heapSize(), commit: m.currentCommit, peakCommit: m.peakCommit }; | |
| 53 | 60 | } |
| 54 | 61 | |
| 55 | 62 | /** The serving VM's minute line: `memory main: rss 5.1 GiB, heap …`. */ |
| 56 | 63 | export function memoryLine(r: Reading, uptimeMs: number): string { |
| 57 | return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}, committed peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`; | |
| 64 | return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, process mimalloc committed ${mib(r.commit)} (peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`; | |
| 58 | 65 | } |
| 59 | 66 | |
| 60 | /** A worker VM's line: its heap and malloc are its own; RSS is the process's. */ | |
| 67 | /** A worker VM's line: its heap is its own; RSS is the process's. */ | |
| 61 | 68 | export function workerMemoryLine(name: string, r: Reading): string { |
| 62 | return `memory worker ${name}: heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}), process rss ${mib(r.rss)}`; | |
| 69 | return `memory worker ${name}: heap ${mib(r.heap)}, process rss ${mib(r.rss)}`; | |
| 63 | 70 | } |
| 64 | 71 | |
| 65 | 72 | export interface Grown { |
| @@ -72,12 +79,12 @@ export interface Grown { | ||
| 72 | 79 | } |
| 73 | 80 | |
| 74 | 81 | /** |
| 75 | * The three parts of a VM's reading (#12951): `heap` is the JS heap (`heapSize()`), | |
| 76 | * `malloc` the allocator's bytes beyond it (`memoryUsage().current - heap`, floored at 0), | |
| 77 | * `native` the process's RSS outside every VM's malloc (main only: RSS is the process's). | |
| 82 | * The parts a reading splits into (#12951, #14264): `heap` is one VM's JS heap | |
| 83 | * (`heapSize()`); `outside` is the process's RSS less every fresh VM's heap, floored at 0 | |
| 84 | * (malloc, SQLite's pages, native buffers, code), kept on main since RSS is the process's. | |
| 78 | 85 | */ |
| 79 | export type Part = "heap" | "malloc" | "native"; | |
| 80 | const PARTS: readonly Part[] = ["heap", "malloc", "native"]; | |
| 86 | export type Part = "heap" | "outside"; | |
| 87 | const PARTS: readonly Part[] = ["heap", "outside"]; | |
| 81 | 88 | |
| 82 | 89 | export interface Growth { |
| 83 | 90 | /** `main`, or the worker's name as its `memory worker <name>` line gives it. */ |
| @@ -87,11 +94,10 @@ export interface Growth { | ||
| 87 | 94 | readonly toBytes: number; |
| 88 | 95 | } |
| 89 | 96 | |
| 90 | /** A worker's minute reading as it crosses to the serving VM. */ | |
| 97 | /** A worker's minute reading as it crosses to the serving VM: its own heap. */ | |
| 91 | 98 | export interface WorkerReading { |
| 92 | 99 | readonly name: string; |
| 93 | 100 | readonly heap: number; |
| 94 | readonly current: number; | |
| 95 | 101 | } |
| 96 | 102 | |
| 97 | 103 | /** A worker silent this long is not blamed: it ended, or it stopped reporting. */ |
| @@ -105,8 +111,8 @@ const bytes = (n: unknown): n is number => typeof n === "number" && Number.isFin | ||
| 105 | 111 | export function workerReadingOf(data: unknown): WorkerReading | undefined { |
| 106 | 112 | if (typeof data !== "object" || data === null) return undefined; |
| 107 | 113 | const d = data as Record<string, unknown>; |
| 108 | if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap) || !bytes(d.current)) return undefined; | |
| 109 | return { name: d.name, heap: d.heap, current: d.current }; | |
| 114 | if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap)) return undefined; | |
| 115 | return { name: d.name, heap: d.heap }; | |
| 110 | 116 | } |
| 111 | 117 | |
| 112 | 118 | /** One owner's first-hour sums per part, its baseline once that hour is over, its latest parts. */ |
| @@ -116,18 +122,18 @@ interface Owner { | ||
| 116 | 122 | readonly count: Record<Part, number>; |
| 117 | 123 | base: Partial<Record<Part, number>> | undefined; |
| 118 | 124 | latest: Partial<Record<Part, number>>; |
| 119 | /** The latest reading's malloc, which main's native excludes. */ | |
| 120 | malloc: number; | |
| 125 | /** The latest reading's heap, which main's `outside` excludes. */ | |
| 126 | heap: number; | |
| 121 | 127 | at: number; |
| 122 | 128 | } |
| 123 | 129 | |
| 124 | 130 | const owner = (startedAt: number): Owner => ({ |
| 125 | 131 | startedAt, |
| 126 | sum: { heap: 0, malloc: 0, native: 0 }, | |
| 127 | count: { heap: 0, malloc: 0, native: 0 }, | |
| 132 | sum: { heap: 0, outside: 0 }, | |
| 133 | count: { heap: 0, outside: 0 }, | |
| 128 | 134 | base: undefined, |
| 129 | 135 | latest: {}, |
| 130 | malloc: 0, | |
| 136 | heap: 0, | |
| 131 | 137 | at: startedAt, |
| 132 | 138 | }); |
| 133 | 139 | |
| @@ -167,11 +173,8 @@ function mostGrown(owners: Iterable<readonly [string, Owner]>, now: number): Gro | ||
| 167 | 173 | } |
| 168 | 174 | |
| 169 | 175 | export interface MemoryWatch { |
| 170 | /** | |
| 171 | * The serving VM's minute reading. `malloc` (`memoryUsage().current`) splits it into | |
| 172 | * heap, malloc beyond the heap and native; without it only the heap is kept per part. | |
| 173 | */ | |
| 174 | note(rss: number, heap: number, now?: number, malloc?: number): void; | |
| 176 | /** The serving VM's minute reading: the process's RSS and main's own heap. */ | |
| 177 | note(rss: number, heap: number, now?: number): void; | |
| 175 | 178 | /** A worker's minute reading (`hearWorkers` carries it here), against its own first hour. */ |
| 176 | 179 | noteWorker(r: WorkerReading, now?: number): void; |
| 177 | 180 | /** RSS at least twice its first-hour mean, with the part that grew most, or undefined. */ |
| @@ -193,20 +196,15 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 193 | 196 | const main = owner(startedAt); |
| 194 | 197 | const workers = new Map<string, Owner>(); |
| 195 | 198 | return { |
| 196 | note(rss, heap, now = Date.now(), malloc) { | |
| 199 | note(rss, heap, now = Date.now()) { | |
| 197 | 200 | if (!Number.isFinite(rss) || rss <= 0) return; |
| 198 | 201 | last = { heap, rss, at: now }; |
| 199 | 202 | latest = rss; |
| 200 | const parts: Partial<Record<Part, number>> = { heap }; | |
| 201 | if (bytes(malloc)) { | |
| 202 | // Every fresh worker's malloc is its own VM's, inside the process's RSS: not native. | |
| 203 | let others = 0; | |
| 204 | for (const w of workers.values()) if (now - w.at <= WORKER_STALE_MS) others += w.malloc; | |
| 205 | parts.malloc = Math.max(0, malloc - heap); | |
| 206 | parts.native = Math.max(0, rss - malloc - others); | |
| 207 | main.malloc = malloc; | |
| 208 | } | |
| 209 | take(main, parts, now); | |
| 203 | // Every fresh worker's heap is its own VM's, inside the process's RSS: not outside. | |
| 204 | let others = 0; | |
| 205 | for (const w of workers.values()) if (now - w.at <= WORKER_STALE_MS) others += w.heap; | |
| 206 | main.heap = heap; | |
| 207 | take(main, { heap, outside: Math.max(0, rss - heap - others) }, now); | |
| 210 | 208 | if (base === undefined) { |
| 211 | 209 | if (now - startedAt < FIRST_HOUR_MS) { |
| 212 | 210 | sum += rss; |
| @@ -226,8 +224,8 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 226 | 224 | o = owner(now); |
| 227 | 225 | workers.set(r.name, o); |
| 228 | 226 | } |
| 229 | o.malloc = r.current; | |
| 230 | take(o, { heap: r.heap, malloc: Math.max(0, r.current - r.heap) }, now); | |
| 227 | o.heap = r.heap; | |
| 228 | take(o, { heap: r.heap }, now); | |
| 231 | 229 | }, |
| 232 | 230 | grown() { |
| 233 | 231 | if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined; |
| @@ -246,21 +244,21 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 246 | 244 | /** This process's watch, started with it. */ |
| 247 | 245 | export const memoryWatch: MemoryWatch = createMemoryWatch(); |
| 248 | 246 | |
| 249 | const PART_WORDS: Record<Part, string> = { | |
| 250 | heap: "JS heap", | |
| 251 | malloc: "malloc", | |
| 252 | native: "native memory outside every VM's malloc", | |
| 253 | }; | |
| 254 | ||
| 255 | /** Who grew, in words: `the main VM's JS heap`, `worker pages's malloc`. */ | |
| 256 | const grewWords = (g: Growth): string => `${g.owner === "main" ? "the main VM" : `worker ${g.owner}`}'s ${PART_WORDS[g.part]}`; | |
| 247 | /** | |
| 248 | * Who grew, in words: `the main VM's JS heap`, `worker pages's JS heap`, or the process's | |
| 249 | * memory outside every JS heap, which no one VM owns. | |
| 250 | */ | |
| 251 | const grewWords = (g: Growth): string => | |
| 252 | g.part === "outside" | |
| 253 | ? "the process's memory outside every JS heap (malloc, SQLite, native buffers)" | |
| 254 | : `${g.owner === "main" ? "the main VM" : `worker ${g.owner}`}'s JS heap`; | |
| 257 | 255 | |
| 258 | 256 | /** The needs-you sentence for a grown process: the figure, and whose part grew most (#12951). */ |
| 259 | 257 | export function grownSentence(g: Grown): string { |
| 260 | 258 | const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}`; |
| 261 | 259 | const since = new Date(g.since).toISOString(); |
| 262 | 260 | if (g.grew === undefined) |
| 263 | return `${head}; it began growing at ${since}, and no VM's heap or malloc is above its own first hour.`; | |
| 261 | return `${head}; it began growing at ${since}, and no VM's heap, nor the memory outside them, is above its own first hour.`; | |
| 264 | 262 | return `${head}; ${grewWords(g.grew)} grew most (${mib(g.grew.fromBytes)} to ${mib(g.grew.toBytes)}) since ${since}.`; |
| 265 | 263 | } |
| 266 | 264 | |
| @@ -280,7 +278,7 @@ export function tellReading(name: string, r: Reading): Reading { | ||
| 280 | 278 | told = new BroadcastChannel(MEMORY_CHANNEL); |
| 281 | 279 | (told as { unref?: () => void }).unref?.(); |
| 282 | 280 | } |
| 283 | told.postMessage({ name, heap: r.heap, current: r.current } satisfies WorkerReading); | |
| 281 | told.postMessage({ name, heap: r.heap } satisfies WorkerReading); | |
| 284 | 282 | } catch { |
| 285 | 283 | // A closed channel loses one minute's reading; the line is still written. |
| 286 | 284 | } |
test/memory-watch.test.ts +79 −48
| @@ -29,7 +29,11 @@ test("raises once RSS passes twice the first-hour mean, with the figure and when | ||
| 29 | 29 | w.note(3900 * MIB, 1, began + 60_000); // 1.95x |
| 30 | 30 | expect(w.grown()).toBeUndefined(); |
| 31 | 31 | w.note(4100 * MIB, 1, began + 120_000); // 2.05x |
| 32 | expect(w.grown()).toEqual({ baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: began }); | |
| 32 | // Main's heap fell to 1 byte, so every RSS byte past it is outside: that part grew most. | |
| 33 | expect(w.grown()).toEqual({ | |
| 34 | baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: began, | |
| 35 | grew: { owner: "main", part: "outside", fromBytes: 1000 * MIB, toBytes: 4100 * MIB - 1 }, | |
| 36 | }); | |
| 33 | 37 | expect(grownSentence(w.grown()!)).toContain("4100 MiB, over twice its first-hour 2000 MiB"); |
| 34 | 38 | expect(grownSentence(w.grown()!)).toContain(new Date(began).toISOString()); |
| 35 | 39 | }); |
| @@ -62,30 +66,36 @@ test("a stall's heap with no argument is read from the VM", () => { | ||
| 62 | 66 | }); |
| 63 | 67 | |
| 64 | 68 | test("the minute lines name the owner and the figures", () => { |
| 65 | const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, current: 512 * MIB, peak: 600 * MIB, peakCommit: 700 * MIB }; | |
| 66 | expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, malloc 512 MiB (peak 600 MiB, committed peak 700 MiB), up 60 min"); | |
| 67 | expect(workerMemoryLine("engine", r)).toStartWith("memory worker engine: heap 1024 MiB"); | |
| 69 | const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, commit: 512 * MIB, peakCommit: 700 * MIB }; | |
| 70 | expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, process mimalloc committed 512 MiB (peak 700 MiB), up 60 min"); | |
| 71 | expect(workerMemoryLine("engine", r)).toBe("memory worker engine: heap 1024 MiB, process rss 5120 MiB"); | |
| 68 | 72 | expect(readNow().heap).toBeGreaterThan(0); |
| 69 | 73 | }); |
| 70 | 74 | |
| 71 | // #12951: which VM grew, and which part. Main's first hour: rss 2000, heap 500, malloc 800 | |
| 72 | // (so 300 beyond the heap); worker pages: heap 100, malloc 410 (310 beyond); worker lanes: | |
| 73 | // heap 50, malloc 90; main's native is rss less every VM's malloc: 2000 - 800 - 410 - 90 = 700 MiB. | |
| 75 | // #14264: `jsc.memoryUsage().current` is the RSS, not malloc. Read as "malloc", RSS less | |
| 76 | // main's heap blamed the main VM for every worker's heap and every native byte. | |
| 77 | test("no part is read from Bun's memoryUsage().current", () => { | |
| 78 | expect(Object.keys(readNow()).sort()).toEqual(["commit", "heap", "peakCommit", "rss"]); | |
| 79 | }); | |
| 80 | ||
| 81 | // #12951: which VM grew, and which part. First hour: rss 2000, main heap 500, worker pages | |
| 82 | // heap 100, worker lanes heap 50; outside every heap: 2000 - 500 - 100 - 50 = 1350 MiB. | |
| 74 | 83 | const HOUR = T0 + FIRST_HOUR_MS; |
| 75 | 84 | function owners(w: ReturnType<typeof createMemoryWatch>) { |
| 76 | 85 | for (let i = 0; i < 12; i++) { |
| 77 | 86 | const at = T0 + i * 300_000; |
| 78 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, at); | |
| 79 | w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at); | |
| 80 | w.note(2000 * MIB, 500 * MIB, at, 800 * MIB); | |
| 87 | w.noteWorker({ name: "pages", heap: 100 * MIB }, at); | |
| 88 | w.noteWorker({ name: "lanes", heap: 50 * MIB }, at); | |
| 89 | w.note(2000 * MIB, 500 * MIB, at); | |
| 81 | 90 | } |
| 82 | 91 | } |
| 83 | 92 | |
| 84 | 93 | test("main heap growth is named as the main VM's JS heap", () => { |
| 85 | 94 | const w = createMemoryWatch(T0); |
| 86 | 95 | owners(w); |
| 87 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, HOUR + 60_000); | |
| 88 | w.note(4700 * MIB, 3100 * MIB, HOUR + 60_000, 3400 * MIB); | |
| 96 | w.noteWorker({ name: "pages", heap: 100 * MIB }, HOUR + 60_000); | |
| 97 | w.noteWorker({ name: "lanes", heap: 50 * MIB }, HOUR + 60_000); | |
| 98 | w.note(4700 * MIB, 3100 * MIB, HOUR + 60_000); // outside: 4700 - 3100 - 150 = 1450, +100 | |
| 89 | 99 | const g = w.grown()!; |
| 90 | 100 | expect(g.grew).toEqual({ owner: "main", part: "heap", fromBytes: 500 * MIB, toBytes: 3100 * MIB }); |
| 91 | 101 | expect(grownSentence(g)).toBe( |
| @@ -93,72 +103,93 @@ test("main heap growth is named as the main VM's JS heap", () => { | ||
| 93 | 103 | ); |
| 94 | 104 | }); |
| 95 | 105 | |
| 96 | test("one worker's malloc growth is named as that worker, not as main's native memory", () => { | |
| 106 | test("one worker's heap growth is named as that worker, not as the main VM's", () => { | |
| 97 | 107 | const w = createMemoryWatch(T0); |
| 98 | 108 | owners(w); |
| 99 | 109 | const at = HOUR + 120_000; |
| 100 | w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at); | |
| 101 | w.noteWorker({ name: "pages", heap: 110 * MIB, current: 2510 * MIB }, at); // 2400 beyond its heap | |
| 102 | w.note(4200 * MIB, 520 * MIB, at, 800 * MIB); // native: 4200 - 800 - 2510 - 90 = 800, +100 over its hour | |
| 110 | w.noteWorker({ name: "lanes", heap: 50 * MIB }, at); | |
| 111 | w.noteWorker({ name: "pages", heap: 2400 * MIB }, at); | |
| 112 | w.note(4200 * MIB, 520 * MIB, at); // outside: 4200 - 520 - 2450 = 1230, under its hour | |
| 103 | 113 | const g = w.grown()!; |
| 104 | expect(g.grew).toEqual({ owner: "pages", part: "malloc", fromBytes: 310 * MIB, toBytes: 2400 * MIB }); | |
| 105 | expect(grownSentence(g)).toContain("2000 MiB; worker pages's malloc grew most (310 MiB to 2400 MiB) since "); | |
| 114 | expect(g.grew).toEqual({ owner: "pages", part: "heap", fromBytes: 100 * MIB, toBytes: 2400 * MIB }); | |
| 115 | expect(grownSentence(g)).toContain("2000 MiB; worker pages's JS heap grew most (100 MiB to 2400 MiB) since "); | |
| 116 | }); | |
| 117 | ||
| 118 | test("growth outside every heap is the process's, never one VM's malloc", () => { | |
| 119 | const w = createMemoryWatch(T0); | |
| 120 | owners(w); | |
| 121 | const at = HOUR + 60_000; | |
| 122 | w.noteWorker({ name: "pages", heap: 110 * MIB }, at); | |
| 123 | w.noteWorker({ name: "lanes", heap: 50 * MIB }, at); | |
| 124 | w.note(4100 * MIB, 540 * MIB, at); // outside: 4100 - 540 - 160 = 3400 | |
| 125 | const g = w.grown()!; | |
| 126 | expect(g.grew).toEqual({ owner: "main", part: "outside", fromBytes: 1350 * MIB, toBytes: 3400 * MIB }); | |
| 127 | const said = grownSentence(g); | |
| 128 | expect(said).toContain("the process's memory outside every JS heap (malloc, SQLite, native buffers) grew most (1350 MiB to 3400 MiB)"); | |
| 129 | expect(said).not.toContain("main VM's malloc"); | |
| 106 | 130 | }); |
| 107 | 131 | |
| 108 | 132 | test("growth under twice the first hour names nothing", () => { |
| 109 | 133 | const w = createMemoryWatch(T0); |
| 110 | 134 | owners(w); |
| 111 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2000 * MIB }, HOUR + 60_000); | |
| 112 | w.note(3900 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB); // 1.95x | |
| 135 | w.noteWorker({ name: "pages", heap: 1700 * MIB }, HOUR + 60_000); | |
| 136 | w.note(3900 * MIB, 500 * MIB, HOUR + 60_000); // 1.95x | |
| 113 | 137 | expect(w.grown()).toBeUndefined(); |
| 114 | 138 | }); |
| 115 | 139 | |
| 116 | 140 | test("a worker that stopped reporting is not blamed", () => { |
| 117 | 141 | const w = createMemoryWatch(T0); |
| 118 | 142 | owners(w); |
| 119 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000); // its last word | |
| 120 | w.note(4200 * MIB, 700 * MIB, HOUR + 60_000, 1000 * MIB); | |
| 143 | w.noteWorker({ name: "pages", heap: 3000 * MIB }, HOUR + 60_000); // its last word | |
| 144 | w.note(4200 * MIB, 700 * MIB, HOUR + 60_000); | |
| 121 | 145 | expect(w.grown()!.grew?.owner).toBe("pages"); // still reporting: named |
| 122 | 146 | const later = HOUR + 60_000 + WORKER_STALE_MS + 60_000; |
| 123 | w.note(4200 * MIB, 700 * MIB, later, 1000 * MIB); | |
| 124 | // Silent five minutes (lanes too): their malloc is no longer subtracted from native either, | |
| 125 | // so main's own native (4200 - 1000 = 3200, against 2000 - 800 - 410 - 90 = 700) grew most. | |
| 126 | expect(w.grown()!.grew).toEqual({ owner: "main", part: "native", fromBytes: 700 * MIB, toBytes: 3200 * MIB }); | |
| 127 | expect(w.grown()!.grew?.owner).not.toBe("pages"); | |
| 147 | w.note(4200 * MIB, 700 * MIB, later); | |
| 148 | // Silent five minutes (lanes too): their heaps are no longer subtracted from outside either, | |
| 149 | // so outside (4200 - 700 = 3500, against 1350) grew most. | |
| 150 | expect(w.grown()!.grew).toEqual({ owner: "main", part: "outside", fromBytes: 1350 * MIB, toBytes: 3500 * MIB }); | |
| 128 | 151 | }); |
| 129 | 152 | |
| 130 | test("with nothing above its own hour the sentence says so; an old caller without malloc still names the heap", () => { | |
| 153 | test("with nothing above its own hour the sentence says so", () => { | |
| 131 | 154 | const w = createMemoryWatch(T0); |
| 132 | firstHour(w, 2000 * MIB); // heap 1000, no malloc | |
| 133 | w.note(4100 * MIB, 900 * MIB, HOUR + 60_000); | |
| 155 | owners(w); | |
| 156 | const at = HOUR + 60_000; | |
| 157 | w.noteWorker({ name: "pages", heap: 100 * MIB }, at); | |
| 158 | w.noteWorker({ name: "lanes", heap: 50 * MIB }, at); | |
| 159 | // A worker started after main's hour has no first hour yet: its heap is never "outside", | |
| 160 | // and it cannot be named either. Outside: 4100 - 500 - 150 - 2100 = 1350, its own hour. | |
| 161 | w.noteWorker({ name: "map", heap: 2100 * MIB }, at); | |
| 162 | w.note(4100 * MIB, 500 * MIB, at); | |
| 134 | 163 | const g = w.grown()!; |
| 164 | expect(g).toEqual({ baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: at }); | |
| 135 | 165 | expect(g.grew).toBeUndefined(); |
| 136 | expect(grownSentence(g)).toContain("no VM's heap or malloc is above its own first hour"); | |
| 137 | w.note(4100 * MIB, 1500 * MIB, HOUR + 120_000); | |
| 138 | expect(w.grown()!.grew).toEqual({ owner: "main", part: "heap", fromBytes: 1000 * MIB, toBytes: 1500 * MIB }); | |
| 166 | expect(grownSentence(g)).toContain("no VM's heap, nor the memory outside them, is above its own first hour"); | |
| 139 | 167 | }); |
| 140 | 168 | |
| 141 | 169 | test("a worker's first hour is its own, counted from its first reading", () => { |
| 142 | 170 | const w = createMemoryWatch(T0); |
| 143 | 171 | owners(w); |
| 144 | 172 | const late = HOUR + 600_000; // a worker started on its first ask, after main's hour |
| 145 | for (let i = 0; i < 12; i++) w.noteWorker({ name: "map", heap: 40 * MIB, current: 60 * MIB }, late + i * 300_000); | |
| 173 | for (let i = 0; i < 12; i++) w.noteWorker({ name: "map", heap: 40 * MIB }, late + i * 300_000); | |
| 146 | 174 | const at = late + FIRST_HOUR_MS + 60_000; |
| 147 | w.noteWorker({ name: "map", heap: 40 * MIB, current: 2060 * MIB }, at); | |
| 148 | w.note(4500 * MIB, 500 * MIB, at, 800 * MIB); | |
| 149 | expect(w.grown()!.grew).toEqual({ owner: "map", part: "malloc", fromBytes: 20 * MIB, toBytes: 2020 * MIB }); | |
| 175 | w.noteWorker({ name: "pages", heap: 100 * MIB }, at); | |
| 176 | w.noteWorker({ name: "lanes", heap: 50 * MIB }, at); | |
| 177 | w.noteWorker({ name: "map", heap: 2040 * MIB }, at); | |
| 178 | w.note(4500 * MIB, 500 * MIB, at); // outside: 4500 - 500 - 2190 = 1810, +460 | |
| 179 | expect(w.grown()!.grew).toEqual({ owner: "map", part: "heap", fromBytes: 40 * MIB, toBytes: 2040 * MIB }); | |
| 150 | 180 | }); |
| 151 | 181 | |
| 152 | 182 | test("a worker's reading is validated at the boundary, and the owners are bounded", () => { |
| 153 | expect(workerReadingOf({ name: "pages", heap: 1, current: 2 })).toEqual({ name: "pages", heap: 1, current: 2 }); | |
| 154 | for (const bad of [null, 5, {}, { name: "Pages", heap: 1, current: 1 }, { name: "main", heap: 1, current: 1 }, { name: "a b", heap: 1, current: 1 }, | |
| 155 | { name: "x".repeat(41), heap: 1, current: 1 }, { name: "p", heap: -1, current: 1 }, { name: "p", heap: 1, current: Infinity }, { name: "p", heap: "1", current: 1 }]) | |
| 183 | expect(workerReadingOf({ name: "pages", heap: 1 })).toEqual({ name: "pages", heap: 1 }); | |
| 184 | expect(workerReadingOf({ name: "pages", heap: 1, current: 2 })).toEqual({ name: "pages", heap: 1 }); | |
| 185 | for (const bad of [null, 5, {}, { name: "Pages", heap: 1 }, { name: "main", heap: 1 }, { name: "a b", heap: 1 }, | |
| 186 | { name: "x".repeat(41), heap: 1 }, { name: "p", heap: -1 }, { name: "p", heap: Infinity }, { name: "p", heap: "1" }]) | |
| 156 | 187 | expect(workerReadingOf(bad)).toBeUndefined(); |
| 157 | 188 | const w = createMemoryWatch(T0); |
| 158 | for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1, current: 1 }, T0); | |
| 189 | for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1 }, T0); | |
| 159 | 190 | owners(w); // pages and lanes arrive past the bound: never kept, never named |
| 160 | w.note(4200 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB); | |
| 161 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000); | |
| 191 | w.noteWorker({ name: "pages", heap: 3000 * MIB }, HOUR + 60_000); | |
| 192 | w.note(4200 * MIB, 500 * MIB, HOUR + 60_000); | |
| 162 | 193 | expect(w.grown()!.grew?.owner).toBe("main"); |
| 163 | 194 | }); |
| 164 | 195 | |
| @@ -168,16 +199,16 @@ test("a worker's reading crosses on the memory channel to the watch that hears i | ||
| 168 | 199 | const { promise: two, resolve } = Promise.withResolvers<void>(); |
| 169 | 200 | const stop = hearWorkers({ noteWorker: (r) => { got.push(r); if (got.length === 2) resolve(); } }); |
| 170 | 201 | try { |
| 171 | const r = { rss: 9, heap: 3 * MIB, current: 7 * MIB, peak: 8, peakCommit: 9 }; | |
| 202 | const r = { rss: 9, heap: 3 * MIB, commit: 7 * MIB, peakCommit: 9 }; | |
| 172 | 203 | expect(tellReading("pages", r)).toBe(r); |
| 173 | 204 | const other = new BroadcastChannel(MEMORY_CHANNEL); |
| 174 | other.postMessage({ name: "../etc", heap: 1, current: 1 }); // malformed: dropped | |
| 175 | other.postMessage({ name: "lanes", heap: 1, current: 2 }); // after it on one channel: it was heard first | |
| 205 | other.postMessage({ name: "../etc", heap: 1 }); // malformed: dropped | |
| 206 | other.postMessage({ name: "lanes", heap: 1 }); // after it on one channel: it was heard first | |
| 176 | 207 | await two; |
| 177 | 208 | other.close(); |
| 178 | 209 | expect(got.sort((a, b) => a.name.localeCompare(b.name))).toEqual([ |
| 179 | { name: "lanes", heap: 1, current: 2 }, | |
| 180 | { name: "pages", heap: 3 * MIB, current: 7 * MIB }, | |
| 210 | { name: "lanes", heap: 1 }, | |
| 211 | { name: "pages", heap: 3 * MIB }, | |
| 181 | 212 | ]); |
| 182 | 213 | } finally { |
| 183 | 214 | stop(); |
test/vitals.test.ts +5 −5
| @@ -587,13 +587,13 @@ test("the needs-you line names which VM grew and which part, in the watch's own | ||
| 587 | 587 | const start = Date.now() - 2 * FIRST_HOUR_MS; |
| 588 | 588 | const w = createMemoryWatch(start); |
| 589 | 589 | for (let i = 0; i < 12; i++) { |
| 590 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, start + i * 300_000); | |
| 591 | w.note(2000 * MIB, 500 * MIB, start + i * 300_000, 800 * MIB); | |
| 590 | w.noteWorker({ name: "pages", heap: 100 * MIB }, start + i * 300_000); | |
| 591 | w.note(2000 * MIB, 500 * MIB, start + i * 300_000); | |
| 592 | 592 | } |
| 593 | 593 | const at = start + FIRST_HOUR_MS + 60_000; |
| 594 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2510 * MIB }, at); | |
| 595 | w.note(4200 * MIB, 500 * MIB, at, 800 * MIB); | |
| 594 | w.noteWorker({ name: "pages", heap: 2410 * MIB }, at); | |
| 595 | w.note(4200 * MIB, 500 * MIB, at); | |
| 596 | 596 | const d = vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w }).degraded.find((x) => x.cause === "memory-growth")!; |
| 597 | 597 | expect(d.sentence).toBe(grownSentence(w.grown()!)); |
| 598 | expect(d.sentence).toContain("worker pages's malloc grew most (310 MiB to 2410 MiB)"); | |
| 598 | expect(d.sentence).toContain("worker pages's JS heap grew most (100 MiB to 2410 MiB)"); | |
| 599 | 599 | }); |
test/worker-memory-lines.test.ts +1 −1
| @@ -54,7 +54,7 @@ test("two workers each write their own memory line, named for themselves", async | ||
| 54 | 54 | test("a worker's minute reading reaches the serving VM's watch by name (#12951)", async () => { |
| 55 | 55 | const script = ` |
| 56 | 56 | import { hearWorkers } from ${JSON.stringify(join(ROOT, "src/perf/memory_watch.ts"))}; |
| 57 | hearWorkers({ noteWorker: (r) => console.log("heard " + r.name + " " + (r.current >= r.heap)) }); | |
| 57 | hearWorkers({ noteWorker: (r) => console.log("heard " + r.name + " " + (r.heap > 0)) }); | |
| 58 | 58 | new Worker(${JSON.stringify(join(ROOT, "src/flow/pipeline_worker.ts"))}); |
| 59 | 59 | `; |
| 60 | 60 | const { out } = await runChild(script, "50", (o) => o.includes("heard pipeline")); |
new above
offline
Compares a live jsc reading with a RSS taken at a different instant. This can flake, and it tests Bun, not this code. Remove the RSS comparison and keep the key-shape assertion.