Public: anyone can read this; the crew can change it.
When the forge's memory has grown, the needs-you line names which VM grew and whether it is JS heap, malloc or native #12952
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/12951-when-the-forge-s-memory-has-grown-the-ne → main 5 commits fork 1 thread · 1 unresolved
7 files
changed +359
−13
AGENTS.md +2
| @@ -35,6 +35,8 @@ State shared between modules through process memory, not the database, is named | ||
| 35 | 35 | `src/system/largest_dirs.ts`: `largestDirs`, the last measurement of the five largest directories under the forge's data (#11800), held 60 s so /system, its terminal screen (`src/tui/screens/system.ts`, `largestLines`) and the MCP `system` tool share one asynchronous walk; a walk that fails is not retried for 60 s; one reading, gone on restart. |
| 36 | 36 | `src/system/db_size.ts`: `dbSize`, the last walk of the database file's pages (#12010: the ten largest tables and indexes), held an hour so /system and the MCP `system` tool (`kind: size`) share one asynchronous walk; a walk that fails is not retried for an hour; one reading, gone on restart. The file's bytes and its growth are read fresh each time; growth comes from `db_size_day`, one row a UTC day, 60 kept. |
| 37 | 37 | |
| 38 | `src/perf/memory_watch.ts`: the memory watch's first-hour sums and baseline per owner (the main VM and each worker by name, at most 64), and each owner's latest reading, fed each minute by the sampler and the workers' `forge-memory` channel (#12951); gone with the process, which is when the baseline should be. | |
| 39 | ||
| 38 | 40 | `src/apps/timer_fire.ts`: `inFlight`, the ids of timers being fired in this process, so a slow app's timer is never fired twice at once; at most the timers due, gone when each settles. The timers themselves are rows (`timer`), read at every boot, never memory. |
| 39 | 41 | |
| 40 | 42 | `src/actions/jobs.ts`: `jobReach.override`, a test seam only (a test hands in its own way to reach an app); the real one is `runtimeReach`. A durable job's runs are the app's own rows (`src/jobs.ts`, carried in the app template bundle `templates/app-template.json`; the template repository `trevato/app-template` must take the same files before a forge that reads that repository hands them out); the template's `Jobs` keeps in process memory only the ids of runs being advanced now (`active`), so a restart forgets nothing it needs. |
docs/features/memory-growth.md +4 −3
| @@ -8,12 +8,13 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang | ||
| 8 | 8 | - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample. |
| 9 | 9 | - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured. |
| 10 | 10 | - `did not fall: no collection seen` means the stall was not the collector's. |
| 11 | - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure and when it began growing. | |
| 11 | - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z." | |
| 12 | - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of each part: `JS heap` (`heapSize()`), `malloc` beyond the heap (`memoryUsage().current - heap`, floored at 0) and, for main only, `native memory outside every VM's malloc` (RSS less main's malloc and every reporting worker's, floored at 0). A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its malloc stops counting against native. When no part is above its own hour the sentence says so. | |
| 12 | 13 | - **The worker names.** `engine` (src/engine/worker.ts); `disk`, `tree` (src/apps); `lanes`, `pages`, `pipeline` (src/flow); `scan`, `du`, `leak` (the `answerAsks` workers of src/kernel/worker_ask.ts); `thumb` (src/looks); `log-writer` (src/runs); `mirror`, `vacuum` (src/snapshot); `code-rows`, `map`, `raster` (src/web). A one-shot worker (`mirror`, `vacuum`) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive. |
| 13 | 14 | |
| 14 | 15 | ## What to do when the need shows |
| 15 | 16 | |
| 16 | 1. Read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker. | |
| 17 | 1. The need names the VM and the part. To see the curve, read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker. | |
| 17 | 18 | 2. Check the stall lines for `fell: a collection ran`: stalls that are collections point at the heap. |
| 18 | 19 | 3. Restart the forge to release it, and report the lines with the issue. |
| 19 | 20 | |
| @@ -21,7 +22,7 @@ The need clears by itself once RSS falls under twice the baseline. A process tha | ||
| 21 | 22 | |
| 22 | 23 | ## How it is built |
| 23 | 24 | |
| 24 | `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`. | |
| 25 | `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, current }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`. | |
| 25 | 26 | |
| 26 | 27 | ## Not done |
| 27 | 28 |
src/boards/memory.ts +3 −2
| @@ -15,7 +15,7 @@ import type { Database } from "bun:sqlite"; | ||
| 15 | 15 | import type { App } from "../app.ts"; |
| 16 | 16 | import { quietly } from "../core/write_clock.ts"; |
| 17 | 17 | import { during } from "../perf/jobs.ts"; |
| 18 | import { memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts"; | |
| 18 | import { hearWorkers, memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts"; | |
| 19 | 19 | |
| 20 | 20 | export const MEMORY_EVERY_MS = 60_000; |
| 21 | 21 | export const HOUR_MS = 3_600_000; |
| @@ -72,12 +72,13 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS | ||
| 72 | 72 | try { |
| 73 | 73 | const reading = readNow(); |
| 74 | 74 | recordMemory(app.db, reading.rss, reading.heap, now); |
| 75 | memoryWatch.note(reading.rss, reading.heap, now); | |
| 75 | memoryWatch.note(reading.rss, reading.heap, now, reading.current); | |
| 76 | 76 | app.log.info(memoryLine(reading, process.uptime() * 1000)); |
| 77 | 77 | } catch (error) { |
| 78 | 78 | app.log.warn("memory sample failed", { error: String(error) }); |
| 79 | 79 | } |
| 80 | 80 | }; |
| 81 | hearWorkers(); | |
| 81 | 82 | during("memory-sampler", () => tick(), undefined, app); |
| 82 | 83 | const timer = setInterval(() => during("memory-sampler", () => tick(), undefined, app), everyMs); |
| 83 | 84 | timer.unref?.(); |
src/perf/memory_watch.ts +202 −7
| @@ -5,6 +5,13 @@ | ||
| 5 | 5 | * in the log by its owner. The watch also keeps the first-hour RSS figure and says when |
| 6 | 6 | * RSS passes twice it (`grown`), which the needs-you line reads (`kernel/vitals.ts`). |
| 7 | 7 | * |
| 8 | * It names what grew (#12951): each owner, the main VM and every worker by name, keeps its | |
| 9 | * own first-hour mean of each part (JS heap, malloc beyond the heap, and for main the RSS | |
| 10 | * outside every VM's malloc), and `grown` names the part that grew the most bytes past its | |
| 11 | * baseline. A worker's reading reaches the serving VM on a `BroadcastChannel` the worker | |
| 12 | * posts to from the same minute timer that writes its line (`tellReading`, `hearWorkers`), | |
| 13 | * so no ask and no hot-path message carries it; a worker silent for five minutes is not blamed. | |
| 14 | * | |
| 8 | 15 | * `heapStats()` counts every object on the heap by type and held the loop ~280 ms a |
| 9 | 16 | * minute on a 2M-object heap (#11118), so the minute line never calls it: `heapSize()` |
| 10 | 17 | * and `memoryUsage()` are constant-time. Bun 1.4.4 exposes no collector time (a |
| @@ -56,11 +63,114 @@ export interface Grown { | ||
| 56 | 63 | readonly rssBytes: number; |
| 57 | 64 | /** Epoch ms of the first sample of the run above 1.25x the baseline. */ |
| 58 | 65 | readonly since: number; |
| 66 | /** The owner's part that grew most against its own first hour, when one did (#12951). */ | |
| 67 | readonly grew?: Growth; | |
| 68 | } | |
| 69 | ||
| 70 | /** | |
| 71 | * The three parts of a VM's reading (#12951): `heap` is the JS heap (`heapSize()`), | |
| 72 | * `malloc` the allocator's bytes beyond it (`memoryUsage().current - heap`, floored at 0), | |
| 73 | * `native` the process's RSS outside every VM's malloc (main only: RSS is the process's). | |
| 74 | */ | |
| 75 | export type Part = "heap" | "malloc" | "native"; | |
| 76 | const PARTS: readonly Part[] = ["heap", "malloc", "native"]; | |
| 77 | ||
| 78 | export interface Growth { | |
| 79 | /** `main`, or the worker's name as its `memory worker <name>` line gives it. */ | |
| 80 | readonly owner: string; | |
| 81 | readonly part: Part; | |
| 82 | readonly fromBytes: number; | |
| 83 | readonly toBytes: number; | |
| 84 | } | |
| 85 | ||
| 86 | /** A worker's minute reading as it crosses to the serving VM. */ | |
| 87 | export interface WorkerReading { | |
| 88 | readonly name: string; | |
| 89 | readonly heap: number; | |
| 90 | readonly current: number; | |
| 91 | } | |
| 92 | ||
| 93 | /** A worker silent this long is not blamed: it ended, or it stopped reporting. */ | |
| 94 | export const WORKER_STALE_MS = 5 * 60_000; | |
| 95 | /** The most workers the watch keeps a baseline for; a name past it is not kept. */ | |
| 96 | export const MAX_OWNERS = 64; | |
| 97 | const WORKER_NAME = /^[a-z][a-z0-9-]{0,39}$/; | |
| 98 | const bytes = (n: unknown): n is number => typeof n === "number" && Number.isFinite(n) && n >= 0; | |
| 99 | ||
| 100 | /** The boundary for a message on the memory channel: a well-formed reading, or undefined. */ | |
| 101 | export function workerReadingOf(data: unknown): WorkerReading | undefined { | |
| 102 | if (typeof data !== "object" || data === null) return undefined; | |
| 103 | const d = data as Record<string, unknown>; | |
| 104 | if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap) || !bytes(d.current)) return undefined; | |
| 105 | return { name: d.name, heap: d.heap, current: d.current }; | |
| 106 | } | |
| 107 | ||
| 108 | /** One owner's first-hour sums per part, its baseline once that hour is over, its latest parts. */ | |
| 109 | interface Owner { | |
| 110 | readonly startedAt: number; | |
| 111 | readonly sum: Record<Part, number>; | |
| 112 | readonly count: Record<Part, number>; | |
| 113 | base: Partial<Record<Part, number>> | undefined; | |
| 114 | latest: Partial<Record<Part, number>>; | |
| 115 | /** The latest reading's malloc, which main's native excludes. */ | |
| 116 | malloc: number; | |
| 117 | at: number; | |
| 118 | } | |
| 119 | ||
| 120 | const owner = (startedAt: number): Owner => ({ | |
| 121 | startedAt, | |
| 122 | sum: { heap: 0, malloc: 0, native: 0 }, | |
| 123 | count: { heap: 0, malloc: 0, native: 0 }, | |
| 124 | base: undefined, | |
| 125 | latest: {}, | |
| 126 | malloc: 0, | |
| 127 | at: startedAt, | |
| 128 | }); | |
| 129 | ||
| 130 | /** Add one reading's parts to an owner: into the first-hour sums, or against its baseline. */ | |
| 131 | function take(o: Owner, parts: Partial<Record<Part, number>>, now: number): void { | |
| 132 | o.latest = parts; | |
| 133 | o.at = now; | |
| 134 | if (o.base !== undefined) return; | |
| 135 | if (now - o.startedAt < FIRST_HOUR_MS) { | |
| 136 | for (const p of PARTS) { | |
| 137 | const v = parts[p]; | |
| 138 | if (v === undefined) continue; | |
| 139 | o.sum[p] += v; | |
| 140 | o.count[p]++; | |
| 141 | } | |
| 142 | return; | |
| 143 | } | |
| 144 | // The hour is over: a part with too few samples has no figure, ever (an idle hour is not one). | |
| 145 | const base: Partial<Record<Part, number>> = {}; | |
| 146 | for (const p of PARTS) if (o.count[p] >= MIN_SAMPLES) base[p] = o.sum[p] / o.count[p]; | |
| 147 | o.base = base; | |
| 148 | } | |
| 149 | ||
| 150 | /** The fresh owner's part that grew the most bytes past its own baseline, or undefined. */ | |
| 151 | function mostGrown(owners: Iterable<readonly [string, Owner]>, now: number): Growth | undefined { | |
| 152 | let best: Growth | undefined; | |
| 153 | for (const [name, o] of owners) { | |
| 154 | if (o.base === undefined || now - o.at > WORKER_STALE_MS) continue; | |
| 155 | for (const p of PARTS) { | |
| 156 | const from = o.base[p]; | |
| 157 | const to = o.latest[p]; | |
| 158 | if (from === undefined || to === undefined || to <= from) continue; | |
| 159 | if (best === undefined || to - from > best.toBytes - best.fromBytes) best = { owner: name, part: p, fromBytes: from, toBytes: to }; | |
| 160 | } | |
| 161 | } | |
| 162 | return best; | |
| 59 | 163 | } |
| 60 | 164 | |
| 61 | 165 | export interface MemoryWatch { |
| 62 | note(rss: number, heap: number, now?: number): void; | |
| 63 | /** RSS at least twice its first-hour mean, or undefined. */ | |
| 166 | /** | |
| 167 | * The serving VM's minute reading. `malloc` (`memoryUsage().current`) splits it into | |
| 168 | * heap, malloc beyond the heap and native; without it only the heap is kept per part. | |
| 169 | */ | |
| 170 | note(rss: number, heap: number, now?: number, malloc?: number): void; | |
| 171 | /** A worker's minute reading (`hearWorkers` carries it here), against its own first hour. */ | |
| 172 | noteWorker(r: WorkerReading, now?: number): void; | |
| 173 | /** RSS at least twice its first-hour mean, with the part that grew most, or undefined. */ | |
| 64 | 174 | grown(): Grown | undefined; |
| 65 | 175 | /** |
| 66 | 176 | * The heap now (read here, or `heapNow` in a test) against the last minute's sample, for a |
| @@ -76,11 +186,23 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 76 | 186 | let last: { heap: number; rss: number; at: number } | undefined; |
| 77 | 187 | let above: number | undefined; |
| 78 | 188 | let latest: number | undefined; |
| 189 | const main = owner(startedAt); | |
| 190 | const workers = new Map<string, Owner>(); | |
| 79 | 191 | return { |
| 80 | note(rss, heap, now = Date.now()) { | |
| 192 | note(rss, heap, now = Date.now(), malloc) { | |
| 81 | 193 | if (!Number.isFinite(rss) || rss <= 0) return; |
| 82 | 194 | last = { heap, rss, at: now }; |
| 83 | 195 | latest = rss; |
| 196 | const parts: Partial<Record<Part, number>> = { heap }; | |
| 197 | if (bytes(malloc)) { | |
| 198 | // Every fresh worker's malloc is its own VM's, inside the process's RSS: not native. | |
| 199 | let others = 0; | |
| 200 | for (const w of workers.values()) if (now - w.at <= WORKER_STALE_MS) others += w.malloc; | |
| 201 | parts.malloc = Math.max(0, malloc - heap); | |
| 202 | parts.native = Math.max(0, rss - malloc - others); | |
| 203 | main.malloc = malloc; | |
| 204 | } | |
| 205 | take(main, parts, now); | |
| 84 | 206 | if (base === undefined) { |
| 85 | 207 | if (now - startedAt < FIRST_HOUR_MS) { |
| 86 | 208 | sum += rss; |
| @@ -93,9 +215,20 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 93 | 215 | if (rss > base * GROWING_RATIO) above ??= now; |
| 94 | 216 | else above = undefined; |
| 95 | 217 | }, |
| 218 | noteWorker(r, now = Date.now()) { | |
| 219 | let o = workers.get(r.name); | |
| 220 | if (o === undefined) { | |
| 221 | if (workers.size >= MAX_OWNERS) return; | |
| 222 | o = owner(now); | |
| 223 | workers.set(r.name, o); | |
| 224 | } | |
| 225 | o.malloc = r.current; | |
| 226 | take(o, { heap: r.heap, malloc: Math.max(0, r.current - r.heap) }, now); | |
| 227 | }, | |
| 96 | 228 | grown() { |
| 97 | 229 | if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined; |
| 98 | return { baselineBytes: base, rssBytes: latest, since: above }; | |
| 230 | const grew = mostGrown([["main", main], ...workers], main.at); | |
| 231 | return { baselineBytes: base, rssBytes: latest, since: above, ...(grew === undefined ? {} : { grew }) }; | |
| 99 | 232 | }, |
| 100 | 233 | heapAcross(now = Date.now(), heapNow = heapSize()) { |
| 101 | 234 | if (last === undefined) return `heap ${mib(heapNow)}`; |
| @@ -109,14 +242,76 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 109 | 242 | /** This process's watch, started with it. */ |
| 110 | 243 | export const memoryWatch: MemoryWatch = createMemoryWatch(); |
| 111 | 244 | |
| 112 | /** The needs-you sentence for a grown process. */ | |
| 245 | const PART_WORDS: Record<Part, string> = { | |
| 246 | heap: "JS heap", | |
| 247 | malloc: "malloc", | |
| 248 | native: "native memory outside every VM's malloc", | |
| 249 | }; | |
| 250 | ||
| 251 | /** Who grew, in words: `the main VM's JS heap`, `worker pages's malloc`. */ | |
| 252 | const grewWords = (g: Growth): string => `${g.owner === "main" ? "the main VM" : `worker ${g.owner}`}'s ${PART_WORDS[g.part]}`; | |
| 253 | ||
| 254 | /** The needs-you sentence for a grown process: the figure, and whose part grew most (#12951). */ | |
| 113 | 255 | export function grownSentence(g: Grown): string { |
| 114 | return `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}; it began growing at ${new Date(g.since).toISOString()}. The log's "memory main" and "memory worker" lines say which VM holds it.`; | |
| 256 | const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}`; | |
| 257 | const since = new Date(g.since).toISOString(); | |
| 258 | if (g.grew === undefined) | |
| 259 | return `${head}; it began growing at ${since}, and no VM's heap or malloc is above its own first hour.`; | |
| 260 | return `${head}; ${grewWords(g.grew)} grew most (${mib(g.grew.fromBytes)} to ${mib(g.grew.toBytes)}) since ${since}.`; | |
| 261 | } | |
| 262 | ||
| 263 | /** | |
| 264 | * The channel a worker's minute reading crosses to the serving VM on: one message beside | |
| 265 | * the line the worker already writes, never one per ask. Every worker of the process | |
| 266 | * shares it; only the serving VM listens (`hearWorkers`). | |
| 267 | */ | |
| 268 | export const MEMORY_CHANNEL = "forge-memory"; | |
| 269 | ||
| 270 | let told: BroadcastChannel | undefined; | |
| 271 | ||
| 272 | /** Post this worker's reading on the memory channel (unref'd: it never holds a worker open). */ | |
| 273 | export function tellReading(name: string, r: Reading): Reading { | |
| 274 | try { | |
| 275 | if (told === undefined) { | |
| 276 | told = new BroadcastChannel(MEMORY_CHANNEL); | |
| 277 | (told as { unref?: () => void }).unref?.(); | |
| 278 | } | |
| 279 | told.postMessage({ name, heap: r.heap, current: r.current } satisfies WorkerReading); | |
| 280 | } catch { | |
| 281 | // A closed channel loses one minute's reading; the line is still written. | |
| 282 | } | |
| 283 | return r; | |
| 284 | } | |
| 285 | ||
| 286 | const heard = new WeakMap<Pick<MemoryWatch, "noteWorker">, BroadcastChannel>(); | |
| 287 | ||
| 288 | /** | |
| 289 | * Listen, once per watch, for the workers' readings and note each into `watch`. Called by | |
| 290 | * the serving VM's memory sampler (`boards/memory.ts`); a malformed message is dropped. | |
| 291 | * Returns the stop. | |
| 292 | */ | |
| 293 | export function hearWorkers(watch: Pick<MemoryWatch, "noteWorker"> = memoryWatch): () => void { | |
| 294 | let channel = heard.get(watch); | |
| 295 | if (channel === undefined) { | |
| 296 | const c = new BroadcastChannel(MEMORY_CHANNEL); | |
| 297 | (c as { unref?: () => void }).unref?.(); | |
| 298 | c.onmessage = (e: { data: unknown }) => { | |
| 299 | const r = workerReadingOf(e.data); | |
| 300 | if (r !== undefined) watch.noteWorker(r); | |
| 301 | }; | |
| 302 | heard.set(watch, c); | |
| 303 | channel = c; | |
| 304 | } | |
| 305 | const c = channel; | |
| 306 | return () => { | |
| 307 | c.close(); | |
| 308 | if (heard.get(watch) === c) heard.delete(watch); | |
| 309 | }; | |
| 115 | 310 | } |
| 116 | 311 | |
| 117 | 312 | /** Start a worker's own minute line (an unref'd timer inside the worker). Returns the stop. */ |
| 118 | 313 | export function startWorkerMemory(name: string, write: (line: string) => void, everyMs = 60_000): () => void { |
| 119 | const timer = setInterval(() => write(workerMemoryLine(name, readNow())), everyMs); | |
| 314 | const timer = setInterval(() => write(workerMemoryLine(name, tellReading(name, readNow()))), everyMs); | |
| 120 | 315 | (timer as { unref?: () => void }).unref?.(); |
| 121 | 316 | return () => clearInterval(timer); |
| 122 | 317 | } |
test/memory-watch.test.ts +121 −1
| @@ -1,5 +1,9 @@ | ||
| 1 | 1 | import { expect, test } from "bun:test"; |
| 2 | import { createMemoryWatch, grownSentence, memoryLine, workerMemoryLine, FIRST_HOUR_MS, readNow } from "../src/perf/memory_watch.ts"; | |
| 2 | import { | |
| 3 | createMemoryWatch, grownSentence, hearWorkers, memoryLine, tellReading, workerMemoryLine, workerReadingOf, | |
| 4 | FIRST_HOUR_MS, MAX_OWNERS, MEMORY_CHANNEL, WORKER_STALE_MS, type WorkerReading, | |
| 5 | } from "../src/perf/memory_watch.ts"; | |
| 6 | import { readNow } from "../src/perf/memory_watch.ts"; | |
| 3 | 7 | |
| 4 | 8 | const MIB = 1048576; |
| 5 | 9 | const T0 = 1_000_000_000_000; |
| @@ -63,3 +67,119 @@ test("the minute lines name the owner and the figures", () => { | ||
| 63 | 67 | expect(workerMemoryLine("engine", r)).toStartWith("memory worker engine: heap 1024 MiB"); |
| 64 | 68 | expect(readNow().heap).toBeGreaterThan(0); |
| 65 | 69 | }); |
| 70 | ||
| 71 | // #12951: which VM grew, and which part. Main's first hour: rss 2000, heap 500, malloc 800 | |
| 72 | // (so 300 beyond the heap); worker pages: heap 100, malloc 410 (310 beyond); worker lanes: | |
| 73 | // heap 50, malloc 90; main's native is rss less every VM's malloc: 2000 - 800 - 410 - 90 = 700 MiB. | |
| 74 | const HOUR = T0 + FIRST_HOUR_MS; | |
| 75 | function owners(w: ReturnType<typeof createMemoryWatch>) { | |
| 76 | for (let i = 0; i < 12; i++) { | |
| 77 | const at = T0 + i * 300_000; | |
| 78 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, at); | |
| 79 | w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at); | |
| 80 | w.note(2000 * MIB, 500 * MIB, at, 800 * MIB); | |
| 81 | } | |
| 82 | } | |
| 83 | ||
| 84 | test("main heap growth is named as the main VM's JS heap", () => { | |
| 85 | const w = createMemoryWatch(T0); | |
| 86 | owners(w); | |
| 87 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, HOUR + 60_000); | |
| 88 | w.note(4700 * MIB, 3100 * MIB, HOUR + 60_000, 3400 * MIB); | |
| 89 | const g = w.grown()!; | |
| 90 | expect(g.grew).toEqual({ owner: "main", part: "heap", fromBytes: 500 * MIB, toBytes: 3100 * MIB }); | |
| 91 | expect(grownSentence(g)).toBe( | |
| 92 | `The forge's own memory is 4700 MiB, over twice its first-hour 2000 MiB; the main VM's JS heap grew most (500 MiB to 3100 MiB) since ${new Date(HOUR + 60_000).toISOString()}.`, | |
| 93 | ); | |
| 94 | }); | |
| 95 | ||
| 96 | test("one worker's malloc growth is named as that worker, not as main's native memory", () => { | |
| 97 | const w = createMemoryWatch(T0); | |
| 98 | owners(w); | |
| 99 | const at = HOUR + 120_000; | |
| 100 | w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at); | |
| 101 | w.noteWorker({ name: "pages", heap: 110 * MIB, current: 2510 * MIB }, at); // 2400 beyond its heap | |
| 102 | w.note(4200 * MIB, 520 * MIB, at, 800 * MIB); // native: 4200 - 800 - 2510 - 90 = 800, +100 over its hour | |
| 103 | const g = w.grown()!; | |
| 104 | expect(g.grew).toEqual({ owner: "pages", part: "malloc", fromBytes: 310 * MIB, toBytes: 2400 * MIB }); | |
| 105 | expect(grownSentence(g)).toContain("2000 MiB; worker pages's malloc grew most (310 MiB to 2400 MiB) since "); | |
| 106 | }); | |
| 107 | ||
| 108 | test("growth under twice the first hour names nothing", () => { | |
| 109 | const w = createMemoryWatch(T0); | |
| 110 | owners(w); | |
| 111 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2000 * MIB }, HOUR + 60_000); | |
| 112 | w.note(3900 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB); // 1.95x | |
| 113 | expect(w.grown()).toBeUndefined(); | |
| 114 | }); | |
| 115 | ||
| 116 | test("a worker that stopped reporting is not blamed", () => { | |
| 117 | const w = createMemoryWatch(T0); | |
| 118 | owners(w); | |
| 119 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000); // its last word | |
| 120 | w.note(4200 * MIB, 700 * MIB, HOUR + 60_000, 1000 * MIB); | |
| 121 | expect(w.grown()!.grew?.owner).toBe("pages"); // still reporting: named | |
| 122 | const later = HOUR + 60_000 + WORKER_STALE_MS + 60_000; | |
| 123 | w.note(4200 * MIB, 700 * MIB, later, 1000 * MIB); | |
| 124 | // Silent five minutes (lanes too): their malloc is no longer subtracted from native either, | |
| 125 | // so main's own native (4200 - 1000 = 3200, against 2000 - 800 - 410 - 90 = 700) grew most. | |
| 126 | expect(w.grown()!.grew).toEqual({ owner: "main", part: "native", fromBytes: 700 * MIB, toBytes: 3200 * MIB }); | |
| 127 | expect(w.grown()!.grew?.owner).not.toBe("pages"); | |
| 128 | }); | |
| 129 | ||
| 130 | test("with nothing above its own hour the sentence says so; an old caller without malloc still names the heap", () => { | |
| 131 | const w = createMemoryWatch(T0); | |
| 132 | firstHour(w, 2000 * MIB); // heap 1000, no malloc | |
| 133 | w.note(4100 * MIB, 900 * MIB, HOUR + 60_000); | |
| 134 | const g = w.grown()!; | |
| 135 | expect(g.grew).toBeUndefined(); | |
| 136 | expect(grownSentence(g)).toContain("no VM's heap or malloc is above its own first hour"); | |
| 137 | w.note(4100 * MIB, 1500 * MIB, HOUR + 120_000); | |
| 138 | expect(w.grown()!.grew).toEqual({ owner: "main", part: "heap", fromBytes: 1000 * MIB, toBytes: 1500 * MIB }); | |
| 139 | }); | |
| 140 | ||
| 141 | test("a worker's first hour is its own, counted from its first reading", () => { | |
| 142 | const w = createMemoryWatch(T0); | |
| 143 | owners(w); | |
| 144 | const late = HOUR + 600_000; // a worker started on its first ask, after main's hour | |
| 145 | for (let i = 0; i < 12; i++) w.noteWorker({ name: "map", heap: 40 * MIB, current: 60 * MIB }, late + i * 300_000); | |
| 146 | const at = late + FIRST_HOUR_MS + 60_000; | |
| 147 | w.noteWorker({ name: "map", heap: 40 * MIB, current: 2060 * MIB }, at); | |
| 148 | w.note(4500 * MIB, 500 * MIB, at, 800 * MIB); | |
| 149 | expect(w.grown()!.grew).toEqual({ owner: "map", part: "malloc", fromBytes: 20 * MIB, toBytes: 2020 * MIB }); | |
| 150 | }); | |
| 151 | ||
| 152 | test("a worker's reading is validated at the boundary, and the owners are bounded", () => { | |
| 153 | expect(workerReadingOf({ name: "pages", heap: 1, current: 2 })).toEqual({ name: "pages", heap: 1, current: 2 }); | |
| 154 | for (const bad of [null, 5, {}, { name: "Pages", heap: 1, current: 1 }, { name: "main", heap: 1, current: 1 }, { name: "a b", heap: 1, current: 1 }, | |
| 155 | { name: "x".repeat(41), heap: 1, current: 1 }, { name: "p", heap: -1, current: 1 }, { name: "p", heap: 1, current: Infinity }, { name: "p", heap: "1", current: 1 }]) | |
| 156 | expect(workerReadingOf(bad)).toBeUndefined(); | |
| 157 | const w = createMemoryWatch(T0); | |
| 158 | for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1, current: 1 }, T0); | |
| 159 | owners(w); // pages and lanes arrive past the bound: never kept, never named | |
| 160 | w.note(4200 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB); | |
| 161 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000); | |
| 162 | expect(w.grown()!.grew?.owner).toBe("main"); | |
| 163 | }); | |
| 164 | ||
| 165 | test("a worker's reading crosses on the memory channel to the watch that hears it", async () => { | |
| 166 | const got: WorkerReading[] = []; | |
| 167 | // Resolved by the second reading heard: the delivery itself, not a clock. | |
| 168 | const { promise: two, resolve } = Promise.withResolvers<void>(); | |
| 169 | const stop = hearWorkers({ noteWorker: (r) => { got.push(r); if (got.length === 2) resolve(); } }); | |
| 170 | try { | |
| 171 | const r = { rss: 9, heap: 3 * MIB, current: 7 * MIB, peak: 8, peakCommit: 9 }; | |
| 172 | expect(tellReading("pages", r)).toBe(r); | |
| 173 | const other = new BroadcastChannel(MEMORY_CHANNEL); | |
| 174 | other.postMessage({ name: "../etc", heap: 1, current: 1 }); // malformed: dropped | |
| 175 | other.postMessage({ name: "lanes", heap: 1, current: 2 }); // after it on one channel: it was heard first | |
| 176 | await two; | |
| 177 | other.close(); | |
| 178 | expect(got.sort((a, b) => a.name.localeCompare(b.name))).toEqual([ | |
| 179 | { name: "lanes", heap: 1, current: 2 }, | |
| 180 | { name: "pages", heap: 3 * MIB, current: 7 * MIB }, | |
| 181 | ]); | |
| 182 | } finally { | |
| 183 | stop(); | |
| 184 | } | |
| 185 | }); |
test/vitals.test.ts +17
| @@ -580,3 +580,20 @@ test("memory twice its first hour is one todo need with the figure and when it b | ||
| 580 | 580 | expect(d.sentence).toContain("2500 MiB, over twice its first-hour 1000 MiB"); |
| 581 | 581 | expect(d.since).toBe(new Date(start + FIRST_HOUR_MS + 1000).toISOString()); |
| 582 | 582 | }); |
| 583 | ||
| 584 | test("the needs-you line names which VM grew and which part, in the watch's own sentence (#12951)", async () => { | |
| 585 | const { FIRST_HOUR_MS, createMemoryWatch, grownSentence } = await import("../src/perf/memory_watch.ts"); | |
| 586 | const MIB = 1048576; | |
| 587 | const start = Date.now() - 2 * FIRST_HOUR_MS; | |
| 588 | const w = createMemoryWatch(start); | |
| 589 | for (let i = 0; i < 12; i++) { | |
| 590 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, start + i * 300_000); | |
| 591 | w.note(2000 * MIB, 500 * MIB, start + i * 300_000, 800 * MIB); | |
| 592 | } | |
| 593 | const at = start + FIRST_HOUR_MS + 60_000; | |
| 594 | w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2510 * MIB }, at); | |
| 595 | w.note(4200 * MIB, 500 * MIB, at, 800 * MIB); | |
| 596 | const d = vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w }).degraded.find((x) => x.cause === "memory-growth")!; | |
| 597 | expect(d.sentence).toBe(grownSentence(w.grown()!)); | |
| 598 | expect(d.sentence).toContain("worker pages's malloc grew most (310 MiB to 2410 MiB)"); | |
| 599 | }); |
test/worker-memory-lines.test.ts +10
| @@ -51,6 +51,16 @@ test("two workers each write their own memory line, named for themselves", async | ||
| 51 | 51 | expect([...named(out)].sort()).toEqual(["code-rows", "pipeline"]); |
| 52 | 52 | }, 20_000); |
| 53 | 53 | |
| 54 | test("a worker's minute reading reaches the serving VM's watch by name (#12951)", async () => { | |
| 55 | const script = ` | |
| 56 | import { hearWorkers } from ${JSON.stringify(join(ROOT, "src/perf/memory_watch.ts"))}; | |
| 57 | hearWorkers({ noteWorker: (r) => console.log("heard " + r.name + " " + (r.current >= r.heap)) }); | |
| 58 | new Worker(${JSON.stringify(join(ROOT, "src/flow/pipeline_worker.ts"))}); | |
| 59 | `; | |
| 60 | const { out } = await runChild(script, "50", (o) => o.includes("heard pipeline")); | |
| 61 | expect(out).toContain("heard pipeline true"); | |
| 62 | }, 20_000); | |
| 63 | ||
| 54 | 64 | test("a worker that ends by itself inside its first interval writes no line and is not held open", async () => { |
| 55 | 65 | // The worker starts its line and then has nothing left to do: with a ref'd timer |
| 56 | 66 | // its loop would never empty, and the process would stay up until the 15 s kill. |
new above
offline
Nit: readNow is imported from the same module on the line above; fold it into that import.