Public: anyone can read this; the crew can change it.

When the forge's memory has grown, the needs-you line names which VM grew and whether it is JS heap, malloc or native #12952

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

torque/forge:torque/12951-when-the-forge-s-memory-has-grown-the-ne → main 5 commits fork 1 thread · 1 unresolved

  • Merged. torque/12951-when-the-forge-s-memory-has-grown-the-ne was merged into main as dafbf1a.

Review

Sign in to review this change.
7 files changed +359 −13
AGENTS.md +2
@@ -35,6 +35,8 @@ State shared between modules through process memory, not the database, is named
3535 `src/system/largest_dirs.ts`: `largestDirs`, the last measurement of the five largest directories under the forge's data (#11800), held 60 s so /system, its terminal screen (`src/tui/screens/system.ts`, `largestLines`) and the MCP `system` tool share one asynchronous walk; a walk that fails is not retried for 60 s; one reading, gone on restart.
3636 `src/system/db_size.ts`: `dbSize`, the last walk of the database file's pages (#12010: the ten largest tables and indexes), held an hour so /system and the MCP `system` tool (`kind: size`) share one asynchronous walk; a walk that fails is not retried for an hour; one reading, gone on restart. The file's bytes and its growth are read fresh each time; growth comes from `db_size_day`, one row a UTC day, 60 kept.
3737
38 `src/perf/memory_watch.ts`: the memory watch's first-hour sums and baseline per owner (the main VM and each worker by name, at most 64), and each owner's latest reading, fed each minute by the sampler and the workers' `forge-memory` channel (#12951); gone with the process, which is when the baseline should be.
39
3840 `src/apps/timer_fire.ts`: `inFlight`, the ids of timers being fired in this process, so a slow app's timer is never fired twice at once; at most the timers due, gone when each settles. The timers themselves are rows (`timer`), read at every boot, never memory.
3941
4042 `src/actions/jobs.ts`: `jobReach.override`, a test seam only (a test hands in its own way to reach an app); the real one is `runtimeReach`. A durable job's runs are the app's own rows (`src/jobs.ts`, carried in the app template bundle `templates/app-template.json`; the template repository `trevato/app-template` must take the same files before a forge that reads that repository hands them out); the template's `Jobs` keeps in process memory only the ids of runs being advanced now (`active`), so a restart forgets nothing it needs.
docs/features/memory-growth.md +4 −3
@@ -8,12 +8,13 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang
88 - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample.
99 - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured.
1010 - `did not fall: no collection seen` means the stall was not the collector's.
11 - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure and when it began growing.
11 - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z."
12 - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of each part: `JS heap` (`heapSize()`), `malloc` beyond the heap (`memoryUsage().current - heap`, floored at 0) and, for main only, `native memory outside every VM's malloc` (RSS less main's malloc and every reporting worker's, floored at 0). A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its malloc stops counting against native. When no part is above its own hour the sentence says so.
1213 - **The worker names.** `engine` (src/engine/worker.ts); `disk`, `tree` (src/apps); `lanes`, `pages`, `pipeline` (src/flow); `scan`, `du`, `leak` (the `answerAsks` workers of src/kernel/worker_ask.ts); `thumb` (src/looks); `log-writer` (src/runs); `mirror`, `vacuum` (src/snapshot); `code-rows`, `map`, `raster` (src/web). A one-shot worker (`mirror`, `vacuum`) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive.
1314
1415 ## What to do when the need shows
1516
16 1. Read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker.
17 1. The need names the VM and the part. To see the curve, read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker.
1718 2. Check the stall lines for `fell: a collection ran`: stalls that are collections point at the heap.
1819 3. Restart the forge to release it, and report the lines with the issue.
1920
@@ -21,7 +22,7 @@ The need clears by itself once RSS falls under twice the baseline. A process tha
2122
2223 ## How it is built
2324
24 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
25 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, current }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
2526
2627 ## Not done
2728
src/boards/memory.ts +3 −2
@@ -15,7 +15,7 @@ import type { Database } from "bun:sqlite";
1515 import type { App } from "../app.ts";
1616 import { quietly } from "../core/write_clock.ts";
1717 import { during } from "../perf/jobs.ts";
18 import { memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts";
18 import { hearWorkers, memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts";
1919
2020 export const MEMORY_EVERY_MS = 60_000;
2121 export const HOUR_MS = 3_600_000;
@@ -72,12 +72,13 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS
7272 try {
7373 const reading = readNow();
7474 recordMemory(app.db, reading.rss, reading.heap, now);
75 memoryWatch.note(reading.rss, reading.heap, now);
75 memoryWatch.note(reading.rss, reading.heap, now, reading.current);
7676 app.log.info(memoryLine(reading, process.uptime() * 1000));
7777 } catch (error) {
7878 app.log.warn("memory sample failed", { error: String(error) });
7979 }
8080 };
81 hearWorkers();
8182 during("memory-sampler", () => tick(), undefined, app);
8283 const timer = setInterval(() => during("memory-sampler", () => tick(), undefined, app), everyMs);
8384 timer.unref?.();
src/perf/memory_watch.ts +202 −7
@@ -5,6 +5,13 @@
55 * in the log by its owner. The watch also keeps the first-hour RSS figure and says when
66 * RSS passes twice it (`grown`), which the needs-you line reads (`kernel/vitals.ts`).
77 *
8 * It names what grew (#12951): each owner, the main VM and every worker by name, keeps its
9 * own first-hour mean of each part (JS heap, malloc beyond the heap, and for main the RSS
10 * outside every VM's malloc), and `grown` names the part that grew the most bytes past its
11 * baseline. A worker's reading reaches the serving VM on a `BroadcastChannel` the worker
12 * posts to from the same minute timer that writes its line (`tellReading`, `hearWorkers`),
13 * so no ask and no hot-path message carries it; a worker silent for five minutes is not blamed.
14 *
815 * `heapStats()` counts every object on the heap by type and held the loop ~280 ms a
916 * minute on a 2M-object heap (#11118), so the minute line never calls it: `heapSize()`
1017 * and `memoryUsage()` are constant-time. Bun 1.4.4 exposes no collector time (a
@@ -56,11 +63,114 @@ export interface Grown {
5663 readonly rssBytes: number;
5764 /** Epoch ms of the first sample of the run above 1.25x the baseline. */
5865 readonly since: number;
66 /** The owner's part that grew most against its own first hour, when one did (#12951). */
67 readonly grew?: Growth;
68 }
69
70 /**
71 * The three parts of a VM's reading (#12951): `heap` is the JS heap (`heapSize()`),
72 * `malloc` the allocator's bytes beyond it (`memoryUsage().current - heap`, floored at 0),
73 * `native` the process's RSS outside every VM's malloc (main only: RSS is the process's).
74 */
75 export type Part = "heap" | "malloc" | "native";
76 const PARTS: readonly Part[] = ["heap", "malloc", "native"];
77
78 export interface Growth {
79 /** `main`, or the worker's name as its `memory worker <name>` line gives it. */
80 readonly owner: string;
81 readonly part: Part;
82 readonly fromBytes: number;
83 readonly toBytes: number;
84 }
85
86 /** A worker's minute reading as it crosses to the serving VM. */
87 export interface WorkerReading {
88 readonly name: string;
89 readonly heap: number;
90 readonly current: number;
91 }
92
93 /** A worker silent this long is not blamed: it ended, or it stopped reporting. */
94 export const WORKER_STALE_MS = 5 * 60_000;
95 /** The most workers the watch keeps a baseline for; a name past it is not kept. */
96 export const MAX_OWNERS = 64;
97 const WORKER_NAME = /^[a-z][a-z0-9-]{0,39}$/;
98 const bytes = (n: unknown): n is number => typeof n === "number" && Number.isFinite(n) && n >= 0;
99
100 /** The boundary for a message on the memory channel: a well-formed reading, or undefined. */
101 export function workerReadingOf(data: unknown): WorkerReading | undefined {
102 if (typeof data !== "object" || data === null) return undefined;
103 const d = data as Record<string, unknown>;
104 if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap) || !bytes(d.current)) return undefined;
105 return { name: d.name, heap: d.heap, current: d.current };
106 }
107
108 /** One owner's first-hour sums per part, its baseline once that hour is over, its latest parts. */
109 interface Owner {
110 readonly startedAt: number;
111 readonly sum: Record<Part, number>;
112 readonly count: Record<Part, number>;
113 base: Partial<Record<Part, number>> | undefined;
114 latest: Partial<Record<Part, number>>;
115 /** The latest reading's malloc, which main's native excludes. */
116 malloc: number;
117 at: number;
118 }
119
120 const owner = (startedAt: number): Owner => ({
121 startedAt,
122 sum: { heap: 0, malloc: 0, native: 0 },
123 count: { heap: 0, malloc: 0, native: 0 },
124 base: undefined,
125 latest: {},
126 malloc: 0,
127 at: startedAt,
128 });
129
130 /** Add one reading's parts to an owner: into the first-hour sums, or against its baseline. */
131 function take(o: Owner, parts: Partial<Record<Part, number>>, now: number): void {
132 o.latest = parts;
133 o.at = now;
134 if (o.base !== undefined) return;
135 if (now - o.startedAt < FIRST_HOUR_MS) {
136 for (const p of PARTS) {
137 const v = parts[p];
138 if (v === undefined) continue;
139 o.sum[p] += v;
140 o.count[p]++;
141 }
142 return;
143 }
144 // The hour is over: a part with too few samples has no figure, ever (an idle hour is not one).
145 const base: Partial<Record<Part, number>> = {};
146 for (const p of PARTS) if (o.count[p] >= MIN_SAMPLES) base[p] = o.sum[p] / o.count[p];
147 o.base = base;
148 }
149
150 /** The fresh owner's part that grew the most bytes past its own baseline, or undefined. */
151 function mostGrown(owners: Iterable<readonly [string, Owner]>, now: number): Growth | undefined {
152 let best: Growth | undefined;
153 for (const [name, o] of owners) {
154 if (o.base === undefined || now - o.at > WORKER_STALE_MS) continue;
155 for (const p of PARTS) {
156 const from = o.base[p];
157 const to = o.latest[p];
158 if (from === undefined || to === undefined || to <= from) continue;
159 if (best === undefined || to - from > best.toBytes - best.fromBytes) best = { owner: name, part: p, fromBytes: from, toBytes: to };
160 }
161 }
162 return best;
59163 }
60164
61165 export interface MemoryWatch {
62 note(rss: number, heap: number, now?: number): void;
63 /** RSS at least twice its first-hour mean, or undefined. */
166 /**
167 * The serving VM's minute reading. `malloc` (`memoryUsage().current`) splits it into
168 * heap, malloc beyond the heap and native; without it only the heap is kept per part.
169 */
170 note(rss: number, heap: number, now?: number, malloc?: number): void;
171 /** A worker's minute reading (`hearWorkers` carries it here), against its own first hour. */
172 noteWorker(r: WorkerReading, now?: number): void;
173 /** RSS at least twice its first-hour mean, with the part that grew most, or undefined. */
64174 grown(): Grown | undefined;
65175 /**
66176 * The heap now (read here, or `heapNow` in a test) against the last minute's sample, for a
@@ -76,11 +186,23 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
76186 let last: { heap: number; rss: number; at: number } | undefined;
77187 let above: number | undefined;
78188 let latest: number | undefined;
189 const main = owner(startedAt);
190 const workers = new Map<string, Owner>();
79191 return {
80 note(rss, heap, now = Date.now()) {
192 note(rss, heap, now = Date.now(), malloc) {
81193 if (!Number.isFinite(rss) || rss <= 0) return;
82194 last = { heap, rss, at: now };
83195 latest = rss;
196 const parts: Partial<Record<Part, number>> = { heap };
197 if (bytes(malloc)) {
198 // Every fresh worker's malloc is its own VM's, inside the process's RSS: not native.
199 let others = 0;
200 for (const w of workers.values()) if (now - w.at <= WORKER_STALE_MS) others += w.malloc;
201 parts.malloc = Math.max(0, malloc - heap);
202 parts.native = Math.max(0, rss - malloc - others);
203 main.malloc = malloc;
204 }
205 take(main, parts, now);
84206 if (base === undefined) {
85207 if (now - startedAt < FIRST_HOUR_MS) {
86208 sum += rss;
@@ -93,9 +215,20 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
93215 if (rss > base * GROWING_RATIO) above ??= now;
94216 else above = undefined;
95217 },
218 noteWorker(r, now = Date.now()) {
219 let o = workers.get(r.name);
220 if (o === undefined) {
221 if (workers.size >= MAX_OWNERS) return;
222 o = owner(now);
223 workers.set(r.name, o);
224 }
225 o.malloc = r.current;
226 take(o, { heap: r.heap, malloc: Math.max(0, r.current - r.heap) }, now);
227 },
96228 grown() {
97229 if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined;
98 return { baselineBytes: base, rssBytes: latest, since: above };
230 const grew = mostGrown([["main", main], ...workers], main.at);
231 return { baselineBytes: base, rssBytes: latest, since: above, ...(grew === undefined ? {} : { grew }) };
99232 },
100233 heapAcross(now = Date.now(), heapNow = heapSize()) {
101234 if (last === undefined) return `heap ${mib(heapNow)}`;
@@ -109,14 +242,76 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
109242 /** This process's watch, started with it. */
110243 export const memoryWatch: MemoryWatch = createMemoryWatch();
111244
112 /** The needs-you sentence for a grown process. */
245 const PART_WORDS: Record<Part, string> = {
246 heap: "JS heap",
247 malloc: "malloc",
248 native: "native memory outside every VM's malloc",
249 };
250
251 /** Who grew, in words: `the main VM's JS heap`, `worker pages's malloc`. */
252 const grewWords = (g: Growth): string => `${g.owner === "main" ? "the main VM" : `worker ${g.owner}`}'s ${PART_WORDS[g.part]}`;
253
254 /** The needs-you sentence for a grown process: the figure, and whose part grew most (#12951). */
113255 export function grownSentence(g: Grown): string {
114 return `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}; it began growing at ${new Date(g.since).toISOString()}. The log's "memory main" and "memory worker" lines say which VM holds it.`;
256 const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}`;
257 const since = new Date(g.since).toISOString();
258 if (g.grew === undefined)
259 return `${head}; it began growing at ${since}, and no VM's heap or malloc is above its own first hour.`;
260 return `${head}; ${grewWords(g.grew)} grew most (${mib(g.grew.fromBytes)} to ${mib(g.grew.toBytes)}) since ${since}.`;
261 }
262
263 /**
264 * The channel a worker's minute reading crosses to the serving VM on: one message beside
265 * the line the worker already writes, never one per ask. Every worker of the process
266 * shares it; only the serving VM listens (`hearWorkers`).
267 */
268 export const MEMORY_CHANNEL = "forge-memory";
269
270 let told: BroadcastChannel | undefined;
271
272 /** Post this worker's reading on the memory channel (unref'd: it never holds a worker open). */
273 export function tellReading(name: string, r: Reading): Reading {
274 try {
275 if (told === undefined) {
276 told = new BroadcastChannel(MEMORY_CHANNEL);
277 (told as { unref?: () => void }).unref?.();
278 }
279 told.postMessage({ name, heap: r.heap, current: r.current } satisfies WorkerReading);
280 } catch {
281 // A closed channel loses one minute's reading; the line is still written.
282 }
283 return r;
284 }
285
286 const heard = new WeakMap<Pick<MemoryWatch, "noteWorker">, BroadcastChannel>();
287
288 /**
289 * Listen, once per watch, for the workers' readings and note each into `watch`. Called by
290 * the serving VM's memory sampler (`boards/memory.ts`); a malformed message is dropped.
291 * Returns the stop.
292 */
293 export function hearWorkers(watch: Pick<MemoryWatch, "noteWorker"> = memoryWatch): () => void {
294 let channel = heard.get(watch);
295 if (channel === undefined) {
296 const c = new BroadcastChannel(MEMORY_CHANNEL);
297 (c as { unref?: () => void }).unref?.();
298 c.onmessage = (e: { data: unknown }) => {
299 const r = workerReadingOf(e.data);
300 if (r !== undefined) watch.noteWorker(r);
301 };
302 heard.set(watch, c);
303 channel = c;
304 }
305 const c = channel;
306 return () => {
307 c.close();
308 if (heard.get(watch) === c) heard.delete(watch);
309 };
115310 }
116311
117312 /** Start a worker's own minute line (an unref'd timer inside the worker). Returns the stop. */
118313 export function startWorkerMemory(name: string, write: (line: string) => void, everyMs = 60_000): () => void {
119 const timer = setInterval(() => write(workerMemoryLine(name, readNow())), everyMs);
314 const timer = setInterval(() => write(workerMemoryLine(name, tellReading(name, readNow()))), everyMs);
120315 (timer as { unref?: () => void }).unref?.();
121316 return () => clearInterval(timer);
122317 }
test/memory-watch.test.ts +121 −1
@@ -1,5 +1,9 @@
11 import { expect, test } from "bun:test";
2 import { createMemoryWatch, grownSentence, memoryLine, workerMemoryLine, FIRST_HOUR_MS, readNow } from "../src/perf/memory_watch.ts";
2 import {
3 createMemoryWatch, grownSentence, hearWorkers, memoryLine, tellReading, workerMemoryLine, workerReadingOf,
4 FIRST_HOUR_MS, MAX_OWNERS, MEMORY_CHANNEL, WORKER_STALE_MS, type WorkerReading,
5 } from "../src/perf/memory_watch.ts";
6 import { readNow } from "../src/perf/memory_watch.ts";

Nit: readNow is imported from the same module on the line above; fold it into that import.

37
48 const MIB = 1048576;
59 const T0 = 1_000_000_000_000;
@@ -63,3 +67,119 @@ test("the minute lines name the owner and the figures", () => {
6367 expect(workerMemoryLine("engine", r)).toStartWith("memory worker engine: heap 1024 MiB");
6468 expect(readNow().heap).toBeGreaterThan(0);
6569 });
70
71 // #12951: which VM grew, and which part. Main's first hour: rss 2000, heap 500, malloc 800
72 // (so 300 beyond the heap); worker pages: heap 100, malloc 410 (310 beyond); worker lanes:
73 // heap 50, malloc 90; main's native is rss less every VM's malloc: 2000 - 800 - 410 - 90 = 700 MiB.
74 const HOUR = T0 + FIRST_HOUR_MS;
75 function owners(w: ReturnType<typeof createMemoryWatch>) {
76 for (let i = 0; i < 12; i++) {
77 const at = T0 + i * 300_000;
78 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, at);
79 w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at);
80 w.note(2000 * MIB, 500 * MIB, at, 800 * MIB);
81 }
82 }
83
84 test("main heap growth is named as the main VM's JS heap", () => {
85 const w = createMemoryWatch(T0);
86 owners(w);
87 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, HOUR + 60_000);
88 w.note(4700 * MIB, 3100 * MIB, HOUR + 60_000, 3400 * MIB);
89 const g = w.grown()!;
90 expect(g.grew).toEqual({ owner: "main", part: "heap", fromBytes: 500 * MIB, toBytes: 3100 * MIB });
91 expect(grownSentence(g)).toBe(
92 `The forge's own memory is 4700 MiB, over twice its first-hour 2000 MiB; the main VM's JS heap grew most (500 MiB to 3100 MiB) since ${new Date(HOUR + 60_000).toISOString()}.`,
93 );
94 });
95
96 test("one worker's malloc growth is named as that worker, not as main's native memory", () => {
97 const w = createMemoryWatch(T0);
98 owners(w);
99 const at = HOUR + 120_000;
100 w.noteWorker({ name: "lanes", heap: 50 * MIB, current: 90 * MIB }, at);
101 w.noteWorker({ name: "pages", heap: 110 * MIB, current: 2510 * MIB }, at); // 2400 beyond its heap
102 w.note(4200 * MIB, 520 * MIB, at, 800 * MIB); // native: 4200 - 800 - 2510 - 90 = 800, +100 over its hour
103 const g = w.grown()!;
104 expect(g.grew).toEqual({ owner: "pages", part: "malloc", fromBytes: 310 * MIB, toBytes: 2400 * MIB });
105 expect(grownSentence(g)).toContain("2000 MiB; worker pages's malloc grew most (310 MiB to 2400 MiB) since ");
106 });
107
108 test("growth under twice the first hour names nothing", () => {
109 const w = createMemoryWatch(T0);
110 owners(w);
111 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2000 * MIB }, HOUR + 60_000);
112 w.note(3900 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB); // 1.95x
113 expect(w.grown()).toBeUndefined();
114 });
115
116 test("a worker that stopped reporting is not blamed", () => {
117 const w = createMemoryWatch(T0);
118 owners(w);
119 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000); // its last word
120 w.note(4200 * MIB, 700 * MIB, HOUR + 60_000, 1000 * MIB);
121 expect(w.grown()!.grew?.owner).toBe("pages"); // still reporting: named
122 const later = HOUR + 60_000 + WORKER_STALE_MS + 60_000;
123 w.note(4200 * MIB, 700 * MIB, later, 1000 * MIB);
124 // Silent five minutes (lanes too): their malloc is no longer subtracted from native either,
125 // so main's own native (4200 - 1000 = 3200, against 2000 - 800 - 410 - 90 = 700) grew most.
126 expect(w.grown()!.grew).toEqual({ owner: "main", part: "native", fromBytes: 700 * MIB, toBytes: 3200 * MIB });
127 expect(w.grown()!.grew?.owner).not.toBe("pages");
128 });
129
130 test("with nothing above its own hour the sentence says so; an old caller without malloc still names the heap", () => {
131 const w = createMemoryWatch(T0);
132 firstHour(w, 2000 * MIB); // heap 1000, no malloc
133 w.note(4100 * MIB, 900 * MIB, HOUR + 60_000);
134 const g = w.grown()!;
135 expect(g.grew).toBeUndefined();
136 expect(grownSentence(g)).toContain("no VM's heap or malloc is above its own first hour");
137 w.note(4100 * MIB, 1500 * MIB, HOUR + 120_000);
138 expect(w.grown()!.grew).toEqual({ owner: "main", part: "heap", fromBytes: 1000 * MIB, toBytes: 1500 * MIB });
139 });
140
141 test("a worker's first hour is its own, counted from its first reading", () => {
142 const w = createMemoryWatch(T0);
143 owners(w);
144 const late = HOUR + 600_000; // a worker started on its first ask, after main's hour
145 for (let i = 0; i < 12; i++) w.noteWorker({ name: "map", heap: 40 * MIB, current: 60 * MIB }, late + i * 300_000);
146 const at = late + FIRST_HOUR_MS + 60_000;
147 w.noteWorker({ name: "map", heap: 40 * MIB, current: 2060 * MIB }, at);
148 w.note(4500 * MIB, 500 * MIB, at, 800 * MIB);
149 expect(w.grown()!.grew).toEqual({ owner: "map", part: "malloc", fromBytes: 20 * MIB, toBytes: 2020 * MIB });
150 });
151
152 test("a worker's reading is validated at the boundary, and the owners are bounded", () => {
153 expect(workerReadingOf({ name: "pages", heap: 1, current: 2 })).toEqual({ name: "pages", heap: 1, current: 2 });
154 for (const bad of [null, 5, {}, { name: "Pages", heap: 1, current: 1 }, { name: "main", heap: 1, current: 1 }, { name: "a b", heap: 1, current: 1 },
155 { name: "x".repeat(41), heap: 1, current: 1 }, { name: "p", heap: -1, current: 1 }, { name: "p", heap: 1, current: Infinity }, { name: "p", heap: "1", current: 1 }])
156 expect(workerReadingOf(bad)).toBeUndefined();
157 const w = createMemoryWatch(T0);
158 for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1, current: 1 }, T0);
159 owners(w); // pages and lanes arrive past the bound: never kept, never named
160 w.note(4200 * MIB, 500 * MIB, HOUR + 60_000, 800 * MIB);
161 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 3000 * MIB }, HOUR + 60_000);
162 expect(w.grown()!.grew?.owner).toBe("main");
163 });
164
165 test("a worker's reading crosses on the memory channel to the watch that hears it", async () => {
166 const got: WorkerReading[] = [];
167 // Resolved by the second reading heard: the delivery itself, not a clock.
168 const { promise: two, resolve } = Promise.withResolvers<void>();
169 const stop = hearWorkers({ noteWorker: (r) => { got.push(r); if (got.length === 2) resolve(); } });
170 try {
171 const r = { rss: 9, heap: 3 * MIB, current: 7 * MIB, peak: 8, peakCommit: 9 };
172 expect(tellReading("pages", r)).toBe(r);
173 const other = new BroadcastChannel(MEMORY_CHANNEL);
174 other.postMessage({ name: "../etc", heap: 1, current: 1 }); // malformed: dropped
175 other.postMessage({ name: "lanes", heap: 1, current: 2 }); // after it on one channel: it was heard first
176 await two;
177 other.close();
178 expect(got.sort((a, b) => a.name.localeCompare(b.name))).toEqual([
179 { name: "lanes", heap: 1, current: 2 },
180 { name: "pages", heap: 3 * MIB, current: 7 * MIB },
181 ]);
182 } finally {
183 stop();
184 }
185 });
test/vitals.test.ts +17
@@ -580,3 +580,20 @@ test("memory twice its first hour is one todo need with the figure and when it b
580580 expect(d.sentence).toContain("2500 MiB, over twice its first-hour 1000 MiB");
581581 expect(d.since).toBe(new Date(start + FIRST_HOUR_MS + 1000).toISOString());
582582 });
583
584 test("the needs-you line names which VM grew and which part, in the watch's own sentence (#12951)", async () => {
585 const { FIRST_HOUR_MS, createMemoryWatch, grownSentence } = await import("../src/perf/memory_watch.ts");
586 const MIB = 1048576;
587 const start = Date.now() - 2 * FIRST_HOUR_MS;
588 const w = createMemoryWatch(start);
589 for (let i = 0; i < 12; i++) {
590 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 410 * MIB }, start + i * 300_000);
591 w.note(2000 * MIB, 500 * MIB, start + i * 300_000, 800 * MIB);
592 }
593 const at = start + FIRST_HOUR_MS + 60_000;
594 w.noteWorker({ name: "pages", heap: 100 * MIB, current: 2510 * MIB }, at);
595 w.note(4200 * MIB, 500 * MIB, at, 800 * MIB);
596 const d = vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w }).degraded.find((x) => x.cause === "memory-growth")!;
597 expect(d.sentence).toBe(grownSentence(w.grown()!));
598 expect(d.sentence).toContain("worker pages's malloc grew most (310 MiB to 2410 MiB)");
599 });
test/worker-memory-lines.test.ts +10
@@ -51,6 +51,16 @@ test("two workers each write their own memory line, named for themselves", async
5151 expect([...named(out)].sort()).toEqual(["code-rows", "pipeline"]);
5252 }, 20_000);
5353
54 test("a worker's minute reading reaches the serving VM's watch by name (#12951)", async () => {
55 const script = `
56 import { hearWorkers } from ${JSON.stringify(join(ROOT, "src/perf/memory_watch.ts"))};
57 hearWorkers({ noteWorker: (r) => console.log("heard " + r.name + " " + (r.current >= r.heap)) });
58 new Worker(${JSON.stringify(join(ROOT, "src/flow/pipeline_worker.ts"))});
59 `;
60 const { out } = await runChild(script, "50", (o) => o.includes("heard pipeline"));
61 expect(out).toContain("heard pipeline true");
62 }, 20_000);
63
5464 test("a worker that ends by itself inside its first interval writes no line and is not held open", async () => {
5565 // The worker starts its line and then has nothing left to do: with a ref'd timer
5666 // its loop would never empty, and the process would stay up until the 15 s kill.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs