Public: anyone can read this; the crew can change it.

Sentinel: memory-growth was degraded 12 minutes today: The forge's own memory is 2936 MiB, over twice its first-hour 1420 MiB; worker pages's malloc grew most (1338 MiB to 31… #14350

opened this

Check · Review · Land · Live — Its check is red. Next: builder, its check is red.

Ready to land

torque/forge:torque/14343-sentinel-memory-growth-was-degraded-12-m → main 1 commit fork

  • Ready to merge — no conflicts with main. 1 commit will land on main.

    Check failed: the keeper's answer and its page carry both, with the countsCheck of e08fb51checked against main at 501b993fast tier: red in 5m09s

    You need write access to merge.

Review

Sign in to review this change.
7 files changed +229 −19
docs/features/memory-growth.md +2 −1
@@ -10,6 +10,7 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang
1010 - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured.
1111 - `did not fall: no collection seen` means the stall was not the collector's.
1212 - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's JS heap grew most (100 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z."
13 - **What fills by design is not growth (#14343).** Each database handle `openDb` opens may fill its page cache (`FORGE_DB_CACHE_MIB`, 1 GiB on the serving handle, 128 MiB on the engine's) and its 512 MiB map as pages are read, each capped by the file's size (`src/kernel/db_ceiling.ts`). On production's 4.5 GB database that is 2176 MiB, most of it filled after the first hour. Each VM reads its own handles' ceiling with its minute reading (the line says `database caches up to …`; a worker's crosses on the channel as `ceiling`), and the need speaks only when RSS is past twice the first hour *and* past the first hour plus every fresh VM's ceiling; the sentence then names that ceiling. Production's evidence on #14343 held 3.0 to 3.3 GiB for twelve hours against a 1420 MiB first hour: a plateau under 1420 + 2176 = 3596 MiB, the shape of a warm cache, where a leak keeps climbing. A handle counts from open until its `close()`; the ceiling is an upper bound, so a leak smaller than the part of the caches still cold is not said until it passes it.
1314 - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of its `JS heap` (`heapSize()`); main also keeps the process's `memory outside every JS heap` (RSS less main's heap and every reporting worker's, floored at 0: malloc, SQLite's pages, native buffers). That part is the process's, not one VM's, and the sentence says so; mimalloc's commit is printed on the line but is never a part, since it overlaps the heaps. A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its heap stops counting against the outside. When no part is above its own hour the sentence says so.
1415 - **The worker names.** `engine` (src/engine/worker.ts); `disk`, `tree` (src/apps); `lanes`, `pages`, `pipeline` (src/flow); `scan`, `du`, `leak` (the `answerAsks` workers of src/kernel/worker_ask.ts); `thumb` (src/looks); `log-writer` (src/runs); `mirror`, `vacuum` (src/snapshot); `code-rows`, `map`, `raster` (src/web). A one-shot worker (`mirror`, `vacuum`) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive.
1516
@@ -24,7 +25,7 @@ The need clears by itself once RSS falls under twice the baseline. A process tha
2425
2526 ## How it is built
2627
27 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, current }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
28 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, ceiling? }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/db-ceiling.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
2829
2930 ## What grew, and what holds it now
3031
src/boards/memory.ts +1 −1
@@ -76,7 +76,7 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS
7676 try {
7777 const reading = readNow();
7878 recordMemory(app.db, reading.rss, reading.heap, now);
79 memoryWatch.note(reading.rss, reading.heap, now);
79 memoryWatch.note(reading.rss, reading.heap, now, reading.dbCeiling);
8080 app.log.info(memoryLine(reading, process.uptime() * 1000));
8181 app.log.info(heldLine(heldNow(app)));
8282 const allocated = allocationLine(allocationsOf(app));
src/kernel/db.ts +6 −1
@@ -9,6 +9,7 @@ import { Database } from "bun:sqlite";
99 import { chmodSync, existsSync, readFileSync, writeFileSync } from "node:fs";
1010 import { dirname, join } from "node:path";
1111 import { createLog } from "./log.ts";
12 import { noteHandle } from "./db_ceiling.ts";
1213 import { tightenDbFiles } from "./file_modes.ts";
1314 import { type Migration, migrate } from "./migrate.ts";
1415
@@ -75,7 +76,11 @@ export function openDb(path: string, migrations: readonly Migration[]): Database
7576 if (image !== undefined) writeFileSync(path, image, { flag: "wx", mode: 0o600 });
7677 db = new Database(path, { create: true, strict: true });
7778 }
78 applyPragmas(db, memory ? CACHE_KIB : mainCacheKiB());
79 const cacheKiB = memory ? CACHE_KIB : mainCacheKiB();
80 applyPragmas(db, cacheKiB);
81 // What this handle may fill as it is read: counted so a warm cache is not read as a leak
82 // (#14343).
83 if (!memory) noteHandle(db, path, cacheKiB * 1024, MMAP_BYTES);
7984 // Only files this call made: one that already exists is tightened below if
8085 // too open (`file_modes.ts`; vitals names any it could not). The umask can widen
8186 // the mode at creation, so it is set again.
src/kernel/db_ceiling.ts added +68
@@ -0,0 +1,68 @@
1 /**
2 * How much memory this VM's open databases may fill by design (#14343). A handle `openDb` opens
3 * keeps up to its `cache_size` of pages in malloc and maps up to its `mmap_size` of the file:
4 * both are taken only as pages are read, so a cold boot holds little and a warm forge holds up
5 * to the ceiling, hours later. On the forge's 4.5 GB database that is 1 GiB of cache and 512 MiB
6 * of map on the serving handle, 128 MiB and 512 MiB on the engine's: about 2.1 GiB that fills
7 * after the first hour, and that the memory watch (`perf/memory_watch.ts`) must not read as a
8 * leak.
9 *
10 * A handle's ceiling is each of the two figures capped by its file's size now (neither can hold
11 * more pages than the file has): `min(cache, size) + min(map, size)`. A handle is counted from
12 * `noteHandle` until its `close()`; `:memory:` handles are not counted (their pages are the
13 * database, not a cache of it).
14 */
15 import { statSync } from "node:fs";
16
17 interface Handle {
18 readonly path: string;
19 readonly cacheBytes: number;
20 readonly mapBytes: number;
21 }
22
23 /** The most handles counted; one past it is not (an over-count would only hide growth). */
24 export const MAX_HANDLES = 256;
25
26 const open = new Set<Handle>();
27
28 const fileSize = (path: string): number => {
29 try {
30 return statSync(path).size;
31 } catch {
32 return 0; // gone: nothing left to cache
33 }
34 };
35
36 /**
37 * Count a handle's cache and map until it closes: wraps its `close` so the count leaves with
38 * it. Returns the handle.
39 */
40 export function noteHandle<D extends { close(...args: never[]): unknown }>(
41 db: D,
42 path: string,
43 cacheBytes: number,
44 mapBytes: number,
45 ): D {
46 if (open.size >= MAX_HANDLES) return db;
47 const h: Handle = { path, cacheBytes: Math.max(0, cacheBytes), mapBytes: Math.max(0, mapBytes) };
48 open.add(h);
49 const close = db.close.bind(db) as (...args: never[]) => unknown;
50 (db as { close: (...args: never[]) => unknown }).close = (...args: never[]) => {
51 open.delete(h);
52 return close(...args);
53 };
54 return db;
55 }
56
57 /** The ceiling of every open handle's cache and map, in bytes, against each file's size now. */
58 export function dbCeilingBytes(sizeOf: (path: string) => number = fileSize): number {
59 let total = 0;
60 for (const h of open) {
61 const size = sizeOf(h.path);
62 total += Math.min(h.cacheBytes, size) + Math.min(h.mapBytes, size);
63 }
64 return total;
65 }
66
67 /** How many handles are counted now (for a test). */
68 export const countedHandles = (): number => open.size;
src/perf/memory_watch.ts +47 −11
@@ -28,8 +28,16 @@
2828 * "gc" PerformanceObserver gets no entries), so a stall is attributed to the collector
2929 * by what the heap did across it (`heapAcross`): a heap that fell is a collection that
3030 * ran inside the stall.
31 *
32 * What fills by design is not growth (#14343): each open database handle may fill its page cache
33 * and its map as it is read (`kernel/db_ceiling.ts`), about 2.1 GiB on production's database,
34 * most of it after the first hour. Each VM reads its own handles' ceiling with its reading (a
35 * worker's crosses on the channel), and `grown` speaks only past twice the first hour *and*
36 * past the first hour plus every fresh VM's ceiling. A leak still raises the need; a warm cache
37 * at its ceiling, which levels off where a leak keeps climbing, does not.
3138 */
3239 import { heapSize, memoryUsage as jscMemory } from "bun:jsc";
40 import { dbCeilingBytes } from "../kernel/db_ceiling.ts";
3341 import { createLog } from "../kernel/log.ts";
3442 import { heldLine, threadHeld, type HeldCount } from "./held.ts";
3543
@@ -51,17 +59,19 @@ export interface Reading {
5159 /** `jsc.memoryUsage().currentCommit` and its peak: mimalloc's, for the whole process. */
5260 readonly commit: number;
5361 readonly peakCommit: number;
62 /** What this VM's open databases may fill by design (`kernel/db_ceiling.ts`). */
63 readonly dbCeiling: number;
5464 }
5565
5666 /** One reading of this VM: constant time, never `heapStats()`. */
5767 export function readNow(rss: number = process.memoryUsage().rss): Reading {
5868 const m = jscMemory();
59 return { rss, heap: heapSize(), commit: m.currentCommit, peakCommit: m.peakCommit };
69 return { rss, heap: heapSize(), commit: m.currentCommit, peakCommit: m.peakCommit, dbCeiling: dbCeilingBytes() };
6070 }
6171
6272 /** The serving VM's minute line: `memory main: rss 5.1 GiB, heap …`. */
6373 export function memoryLine(r: Reading, uptimeMs: number): string {
64 return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, process mimalloc committed ${mib(r.commit)} (peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`;
74 return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, process mimalloc committed ${mib(r.commit)} (peak ${mib(r.peakCommit)}), database caches up to ${mib(r.dbCeiling)}, up ${Math.round(uptimeMs / 60000)} min`;
6575 }
6676
6777 /** A worker VM's line: its heap is its own; RSS is the process's. */
@@ -76,6 +86,8 @@ export interface Grown {
7686 readonly since: number;
7787 /** The owner's part that grew most against its own first hour, when one did (#12951). */
7888 readonly grew?: Growth;
89 /** Every fresh VM's database ceiling, when one is open: RSS was past the first hour plus it. */
90 readonly ceilingBytes?: number;
7991 }
8092
8193 /**
@@ -94,10 +106,14 @@ export interface Growth {
94106 readonly toBytes: number;
95107 }
96108
97 /** A worker's minute reading as it crosses to the serving VM: its own heap. */
109 /**
110 * A worker's minute reading as it crosses to the serving VM: its own heap, its databases' ceiling.
111 */
98112 export interface WorkerReading {
99113 readonly name: string;
100114 readonly heap: number;
115 /** Present when the worker has a database open (`kernel/db_ceiling.ts`). */
116 readonly ceiling?: number;
101117 }
102118
103119 /** A worker silent this long is not blamed: it ended, or it stopped reporting. */
@@ -112,7 +128,8 @@ export function workerReadingOf(data: unknown): WorkerReading | undefined {
112128 if (typeof data !== "object" || data === null) return undefined;
113129 const d = data as Record<string, unknown>;
114130 if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap)) return undefined;
115 return { name: d.name, heap: d.heap };
131 if (d.ceiling !== undefined && !bytes(d.ceiling)) return undefined;
132 return d.ceiling === undefined || d.ceiling === 0 ? { name: d.name, heap: d.heap } : { name: d.name, heap: d.heap, ceiling: d.ceiling };
116133 }
117134
118135 /** One owner's first-hour sums per part, its baseline once that hour is over, its latest parts. */
@@ -124,6 +141,8 @@ interface Owner {
124141 latest: Partial<Record<Part, number>>;
125142 /** The latest reading's heap, which main's `outside` excludes. */
126143 heap: number;
144 /** The latest reading's database ceiling. */
145 ceiling: number;
127146 at: number;
128147 }
129148
@@ -134,6 +153,7 @@ const owner = (startedAt: number): Owner => ({
134153 base: undefined,
135154 latest: {},
136155 heap: 0,
156 ceiling: 0,
137157 at: startedAt,
138158 });
139159
@@ -173,11 +193,16 @@ function mostGrown(owners: Iterable<readonly [string, Owner]>, now: number): Gro
173193 }
174194
175195 export interface MemoryWatch {
176 /** The serving VM's minute reading: the process's RSS and main's own heap. */
177 note(rss: number, heap: number, now?: number): void;
196 /**
197 * The serving VM's minute reading: the process's RSS, main's own heap, its databases' ceiling.
198 */
199 note(rss: number, heap: number, now?: number, ceiling?: number): void;
178200 /** A worker's minute reading (`hearWorkers` carries it here), against its own first hour. */
179201 noteWorker(r: WorkerReading, now?: number): void;
180 /** RSS at least twice its first-hour mean, with the part that grew most, or undefined. */
202 /**
203 * RSS at least twice its first-hour mean and past that mean plus every fresh VM's database
204 * ceiling, with the part that grew most, or undefined.
205 */
181206 grown(): Grown | undefined;
182207 /**
183208 * The heap now (read here, or `heapNow` in a test) against the last minute's sample, for a
@@ -196,8 +221,9 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
196221 const main = owner(startedAt);
197222 const workers = new Map<string, Owner>();
198223 return {
199 note(rss, heap, now = Date.now()) {
224 note(rss, heap, now = Date.now(), ceiling = 0) {
200225 if (!Number.isFinite(rss) || rss <= 0) return;
226 main.ceiling = Number.isFinite(ceiling) && ceiling > 0 ? ceiling : 0;
201227 last = { heap, rss, at: now };
202228 latest = rss;
203229 // Every fresh worker's heap is its own VM's, inside the process's RSS: not outside.
@@ -225,12 +251,21 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
225251 workers.set(r.name, o);
226252 }
227253 o.heap = r.heap;
254 o.ceiling = r.ceiling ?? 0;
228255 take(o, { heap: r.heap }, now);
229256 },
230257 grown() {
231258 if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined;
259 let ceiling = main.ceiling;
260 for (const w of workers.values()) if (main.at - w.at <= WORKER_STALE_MS) ceiling += w.ceiling;
261 // A cache filling to its ceiling is the design, not a leak: only what is past it is growth.
262 if (latest <= base + ceiling) return undefined;
232263 const grew = mostGrown([["main", main], ...workers], main.at);
233 return { baselineBytes: base, rssBytes: latest, since: above, ...(grew === undefined ? {} : { grew }) };
264 return {
265 baselineBytes: base, rssBytes: latest, since: above,
266 ...(grew === undefined ? {} : { grew }),
267 ...(ceiling > 0 ? { ceilingBytes: ceiling } : {}),
268 };
234269 },
235270 heapAcross(now = Date.now(), heapNow = heapSize()) {
236271 if (last === undefined) return `heap ${mib(heapNow)}`;
@@ -255,7 +290,8 @@ const grewWords = (g: Growth): string =>
255290
256291 /** The needs-you sentence for a grown process: the figure, and whose part grew most (#12951). */
257292 export function grownSentence(g: Grown): string {
258 const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}`;
293 const past = g.ceilingBytes === undefined ? "" : ` and past the ${mib(g.ceilingBytes)} more its database caches may fill`;
294 const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}${past}`;
259295 const since = new Date(g.since).toISOString();
260296 if (g.grew === undefined)
261297 return `${head}; it began growing at ${since}, and no VM's heap, nor the memory outside them, is above its own first hour.`;
@@ -278,7 +314,7 @@ export function tellReading(name: string, r: Reading): Reading {
278314 told = new BroadcastChannel(MEMORY_CHANNEL);
279315 (told as { unref?: () => void }).unref?.();
280316 }
281 told.postMessage({ name, heap: r.heap } satisfies WorkerReading);
317 told.postMessage((r.dbCeiling > 0 ? { name, heap: r.heap, ceiling: r.dbCeiling } : { name, heap: r.heap }) satisfies WorkerReading);
282318 } catch {
283319 // A closed channel loses one minute's reading; the line is still written.
284320 }
test/db-ceiling.test.ts added +51
@@ -0,0 +1,51 @@
1 /**
2 * What the open databases may fill by design (#14343): counted from open to close, capped by each
3 * file.
4 */
5 import { expect, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { statSync } from "node:fs";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { openDb } from "../src/kernel/db.ts";
11 import { countedHandles, dbCeilingBytes, noteHandle } from "../src/kernel/db_ceiling.ts";
12
13 const MIB = 1048576;
14
15 test("each handle counts min(cache, size) + min(map, size) until it closes", () => {
16 const before = countedHandles();
17 const closed: string[] = [];
18 const a = noteHandle({ close: () => closed.push("a") }, "/a", 1024 * MIB, 512 * MIB);
19 const b = noteHandle({ close: () => closed.push("b") }, "/b", 128 * MIB, 512 * MIB);
20 const sizes: Record<string, number> = { "/a": 4500 * MIB, "/b": 300 * MIB };
21 const others = dbCeilingBytes(() => 0);
22 expect(others).toBe(0);
23 // a: 1024 + 512 (a 4.5 GB file holds both); b: 128 + 300 (its map stops at the file).
24 expect(dbCeilingBytes((p) => sizes[p] ?? 0)).toBe((1024 + 512 + 128 + 300) * MIB);
25 a.close();
26 expect(closed).toEqual(["a"]);
27 expect(countedHandles()).toBe(before + 1);
28 expect(dbCeilingBytes((p) => sizes[p] ?? 0)).toBe((128 + 300) * MIB);
29 b.close();
30 expect(countedHandles()).toBe(before);
31 });
32
33 test("openDb counts a file's handle and forgets it on close; :memory: is not counted", async () => {
34 const dir = await mkdtemp(join(tmpdir(), "forge-ceiling-"));
35 const before = countedHandles();
36 try {
37 const path = join(dir, "f.db");
38 const db = openDb(path, []);
39 expect(countedHandles()).toBe(before + 1);
40 // A small file: its cache and map can each hold the file at most.
41 const size = statSync(path).size;
42 expect(dbCeilingBytes((p) => (p === path ? size : 0))).toBe(2 * size);
43 db.close();
44 expect(countedHandles()).toBe(before);
45 const mem = openDb(":memory:", []);
46 expect(countedHandles()).toBe(before);
47 mem.close();
48 } finally {
49 await rm(dir, { recursive: true, force: true });
50 }
51 });
test/memory-watch.test.ts +54 −5
@@ -66,8 +66,8 @@ test("a stall's heap with no argument is read from the VM", () => {
6666 });
6767
6868 test("the minute lines name the owner and the figures", () => {
69 const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, commit: 512 * MIB, peakCommit: 700 * MIB };
70 expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, process mimalloc committed 512 MiB (peak 700 MiB), up 60 min");
69 const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, commit: 512 * MIB, peakCommit: 700 * MIB, dbCeiling: 1536 * MIB };
70 expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, process mimalloc committed 512 MiB (peak 700 MiB), database caches up to 1536 MiB, up 60 min");
7171 expect(workerMemoryLine("engine", r)).toBe("memory worker engine: heap 1024 MiB, process rss 5120 MiB");
7272 expect(readNow().heap).toBeGreaterThan(0);
7373 });
@@ -75,7 +75,7 @@ test("the minute lines name the owner and the figures", () => {
7575 // #14264: `jsc.memoryUsage().current` is the RSS, not malloc. Read as "malloc", RSS less
7676 // main's heap blamed the main VM for every worker's heap and every native byte.
7777 test("no part is read from Bun's memoryUsage().current", () => {
78 expect(Object.keys(readNow()).sort()).toEqual(["commit", "heap", "peakCommit", "rss"]);
78 expect(Object.keys(readNow()).sort()).toEqual(["commit", "dbCeiling", "heap", "peakCommit", "rss"]);
7979 });
8080
8181 // #12951: which VM grew, and which part. First hour: rss 2000, main heap 500, worker pages
@@ -185,6 +185,9 @@ test("a worker's reading is validated at the boundary, and the owners are bounde
185185 for (const bad of [null, 5, {}, { name: "Pages", heap: 1 }, { name: "main", heap: 1 }, { name: "a b", heap: 1 },
186186 { name: "x".repeat(41), heap: 1 }, { name: "p", heap: -1 }, { name: "p", heap: Infinity }, { name: "p", heap: "1" }])
187187 expect(workerReadingOf(bad)).toBeUndefined();
188 expect(workerReadingOf({ name: "engine", heap: 1, ceiling: 2 })).toEqual({ name: "engine", heap: 1, ceiling: 2 });
189 expect(workerReadingOf({ name: "engine", heap: 1, ceiling: 0 })).toEqual({ name: "engine", heap: 1 });
190 for (const ceiling of [-1, NaN, Infinity, "2", null]) expect(workerReadingOf({ name: "engine", heap: 1, ceiling })).toBeUndefined();
188191 const w = createMemoryWatch(T0);
189192 for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1 }, T0);
190193 owners(w); // pages and lanes arrive past the bound: never kept, never named
@@ -199,7 +202,7 @@ test("a worker's reading crosses on the memory channel to the watch that hears i
199202 const { promise: two, resolve } = Promise.withResolvers<void>();
200203 const stop = hearWorkers({ noteWorker: (r) => { got.push(r); if (got.length === 2) resolve(); } });
201204 try {
202 const r = { rss: 9, heap: 3 * MIB, commit: 7 * MIB, peakCommit: 9 };
205 const r = { rss: 9, heap: 3 * MIB, commit: 7 * MIB, peakCommit: 9, dbCeiling: 640 * MIB };
203206 expect(tellReading("pages", r)).toBe(r);
204207 const other = new BroadcastChannel(MEMORY_CHANNEL);
205208 other.postMessage({ name: "../etc", heap: 1 }); // malformed: dropped
@@ -208,9 +211,55 @@ test("a worker's reading crosses on the memory channel to the watch that hears i
208211 other.close();
209212 expect(got.sort((a, b) => a.name.localeCompare(b.name))).toEqual([
210213 { name: "lanes", heap: 1 },
211 { name: "pages", heap: 3 * MIB },
214 { name: "pages", heap: 3 * MIB, ceiling: 640 * MIB },
212215 ]);
213216 } finally {
214217 stop();
215218 }
216219 });
220
221 // #14343: production's evidence held 3.0 to 3.3 GiB for twelve hours against a 1420 MiB first
222 // hour: a plateau, the shape of a cache at its ceiling, not of a leak. The serving handle may
223 // fill 1024 MiB of cache and 512 MiB of map; the engine's 128 and 512: 2176 MiB in all.
224 const SERVING = 1536 * MIB;
225 const ENGINE = 640 * MIB;
226 function warmHour(w: ReturnType<typeof createMemoryWatch>) {
227 for (let i = 0; i < 12; i++) {
228 const at = T0 + i * 300_000;
229 w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, at);
230 w.note(1420 * MIB, 400 * MIB, at, SERVING);
231 }
232 }
233
234 test("a database cache filling to its ceiling is not growth: production's 3290 MiB raises nothing", () => {
235 const w = createMemoryWatch(T0);
236 warmHour(w);
237 w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, HOUR + 60_000);
238 w.note(3290 * MIB, 400 * MIB, HOUR + 60_000, SERVING); // 2.3x, but 1420 + 2176 = 3596 is the line
239 expect(w.grown()).toBeUndefined();
240 w.note(3596 * MIB, 400 * MIB, HOUR + 120_000, SERVING); // at the line, not past it
241 expect(w.grown()).toBeUndefined();
242 });
243
244 test("past the first hour plus every database's ceiling, the need speaks and names the ceiling", () => {
245 const w = createMemoryWatch(T0);
246 warmHour(w);
247 const at = HOUR + 60_000;
248 w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, at);
249 w.note(3700 * MIB, 400 * MIB, at, SERVING);
250 const g = w.grown()!;
251 expect(g.ceilingBytes).toBe(2176 * MIB);
252 expect(grownSentence(g)).toStartWith(
253 "The forge's own memory is 3700 MiB, over twice its first-hour 1420 MiB and past the 2176 MiB more its database caches may fill; ",
254 );
255 });
256
257 test("a worker silent five minutes no longer lends its ceiling", () => {
258 const w = createMemoryWatch(T0);
259 warmHour(w);
260 w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, HOUR);
261 w.note(3290 * MIB, 400 * MIB, HOUR + 60_000, SERVING);
262 expect(w.grown()).toBeUndefined(); // 1420 + 1536 + 640 = 3596
263 w.note(3290 * MIB, 400 * MIB, HOUR + WORKER_STALE_MS + 60_000, SERVING);
264 expect(w.grown()?.ceilingBytes).toBe(SERVING); // 1420 + 1536 = 2956 < 3290
265 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs