Public: anyone can read this; the crew can change it.
Sentinel: memory-growth was degraded 12 minutes today: The forge's own memory is 2936 MiB, over twice its first-hour 1420 MiB; worker pages's malloc grew most (1338 MiB to 31… #14350
Check · Review · Land · Live — Its check is red.
torque/forge:torque/14343-sentinel-memory-growth-was-degraded-12-m → main 1 commit fork
7 files
changed +229
−19
docs/features/memory-growth.md +2 −1
| @@ -10,6 +10,7 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang | ||
| 10 | 10 | - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured. |
| 11 | 11 | - `did not fall: no collection seen` means the stall was not the collector's. |
| 12 | 12 | - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's JS heap grew most (100 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z." |
| 13 | - **What fills by design is not growth (#14343).** Each database handle `openDb` opens may fill its page cache (`FORGE_DB_CACHE_MIB`, 1 GiB on the serving handle, 128 MiB on the engine's) and its 512 MiB map as pages are read, each capped by the file's size (`src/kernel/db_ceiling.ts`). On production's 4.5 GB database that is 2176 MiB, most of it filled after the first hour. Each VM reads its own handles' ceiling with its minute reading (the line says `database caches up to …`; a worker's crosses on the channel as `ceiling`), and the need speaks only when RSS is past twice the first hour *and* past the first hour plus every fresh VM's ceiling; the sentence then names that ceiling. Production's evidence on #14343 held 3.0 to 3.3 GiB for twelve hours against a 1420 MiB first hour: a plateau under 1420 + 2176 = 3596 MiB, the shape of a warm cache, where a leak keeps climbing. A handle counts from open until its `close()`; the ceiling is an upper bound, so a leak smaller than the part of the caches still cold is not said until it passes it. | |
| 13 | 14 | - **The parts.** Each owner (the main VM, and each worker by name) keeps its own first-hour mean of its `JS heap` (`heapSize()`); main also keeps the process's `memory outside every JS heap` (RSS less main's heap and every reporting worker's, floored at 0: malloc, SQLite's pages, native buffers). That part is the process's, not one VM's, and the sentence says so; mimalloc's commit is printed on the line but is never a part, since it overlaps the heaps. A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its heap stops counting against the outside. When no part is above its own hour the sentence says so. |
| 14 | 15 | - **The worker names.** `engine` (src/engine/worker.ts); `disk`, `tree` (src/apps); `lanes`, `pages`, `pipeline` (src/flow); `scan`, `du`, `leak` (the `answerAsks` workers of src/kernel/worker_ask.ts); `thumb` (src/looks); `log-writer` (src/runs); `mirror`, `vacuum` (src/snapshot); `code-rows`, `map`, `raster` (src/web). A one-shot worker (`mirror`, `vacuum`) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive. |
| 15 | 16 | |
| @@ -24,7 +25,7 @@ The need clears by itself once RSS falls under twice the baseline. A process tha | ||
| 24 | 25 | |
| 25 | 26 | ## How it is built |
| 26 | 27 | |
| 27 | `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, current }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`. | |
| 28 | `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. The same minute timer that writes a worker's line posts its reading (`{ name, heap, ceiling? }`) on the `forge-memory` `BroadcastChannel` (`tellReading`); the serving VM's memory sampler listens (`hearWorkers`), validates each message (`workerReadingOf`: a lower-case name, not `main`, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: `test/memory-watch.test.ts`, `test/db-ceiling.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`. | |
| 28 | 29 | |
| 29 | 30 | ## What grew, and what holds it now |
| 30 | 31 |
src/boards/memory.ts +1 −1
| @@ -76,7 +76,7 @@ export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemoryS | ||
| 76 | 76 | try { |
| 77 | 77 | const reading = readNow(); |
| 78 | 78 | recordMemory(app.db, reading.rss, reading.heap, now); |
| 79 | memoryWatch.note(reading.rss, reading.heap, now); | |
| 79 | memoryWatch.note(reading.rss, reading.heap, now, reading.dbCeiling); | |
| 80 | 80 | app.log.info(memoryLine(reading, process.uptime() * 1000)); |
| 81 | 81 | app.log.info(heldLine(heldNow(app))); |
| 82 | 82 | const allocated = allocationLine(allocationsOf(app)); |
src/kernel/db.ts +6 −1
| @@ -9,6 +9,7 @@ import { Database } from "bun:sqlite"; | ||
| 9 | 9 | import { chmodSync, existsSync, readFileSync, writeFileSync } from "node:fs"; |
| 10 | 10 | import { dirname, join } from "node:path"; |
| 11 | 11 | import { createLog } from "./log.ts"; |
| 12 | import { noteHandle } from "./db_ceiling.ts"; | |
| 12 | 13 | import { tightenDbFiles } from "./file_modes.ts"; |
| 13 | 14 | import { type Migration, migrate } from "./migrate.ts"; |
| 14 | 15 | |
| @@ -75,7 +76,11 @@ export function openDb(path: string, migrations: readonly Migration[]): Database | ||
| 75 | 76 | if (image !== undefined) writeFileSync(path, image, { flag: "wx", mode: 0o600 }); |
| 76 | 77 | db = new Database(path, { create: true, strict: true }); |
| 77 | 78 | } |
| 78 | applyPragmas(db, memory ? CACHE_KIB : mainCacheKiB()); | |
| 79 | const cacheKiB = memory ? CACHE_KIB : mainCacheKiB(); | |
| 80 | applyPragmas(db, cacheKiB); | |
| 81 | // What this handle may fill as it is read: counted so a warm cache is not read as a leak | |
| 82 | // (#14343). | |
| 83 | if (!memory) noteHandle(db, path, cacheKiB * 1024, MMAP_BYTES); | |
| 79 | 84 | // Only files this call made: one that already exists is tightened below if |
| 80 | 85 | // too open (`file_modes.ts`; vitals names any it could not). The umask can widen |
| 81 | 86 | // the mode at creation, so it is set again. |
src/kernel/db_ceiling.ts added +68
| @@ -0,0 +1,68 @@ | ||
| 1 | /** | |
| 2 | * How much memory this VM's open databases may fill by design (#14343). A handle `openDb` opens | |
| 3 | * keeps up to its `cache_size` of pages in malloc and maps up to its `mmap_size` of the file: | |
| 4 | * both are taken only as pages are read, so a cold boot holds little and a warm forge holds up | |
| 5 | * to the ceiling, hours later. On the forge's 4.5 GB database that is 1 GiB of cache and 512 MiB | |
| 6 | * of map on the serving handle, 128 MiB and 512 MiB on the engine's: about 2.1 GiB that fills | |
| 7 | * after the first hour, and that the memory watch (`perf/memory_watch.ts`) must not read as a | |
| 8 | * leak. | |
| 9 | * | |
| 10 | * A handle's ceiling is each of the two figures capped by its file's size now (neither can hold | |
| 11 | * more pages than the file has): `min(cache, size) + min(map, size)`. A handle is counted from | |
| 12 | * `noteHandle` until its `close()`; `:memory:` handles are not counted (their pages are the | |
| 13 | * database, not a cache of it). | |
| 14 | */ | |
| 15 | import { statSync } from "node:fs"; | |
| 16 | ||
| 17 | interface Handle { | |
| 18 | readonly path: string; | |
| 19 | readonly cacheBytes: number; | |
| 20 | readonly mapBytes: number; | |
| 21 | } | |
| 22 | ||
| 23 | /** The most handles counted; one past it is not (an over-count would only hide growth). */ | |
| 24 | export const MAX_HANDLES = 256; | |
| 25 | ||
| 26 | const open = new Set<Handle>(); | |
| 27 | ||
| 28 | const fileSize = (path: string): number => { | |
| 29 | try { | |
| 30 | return statSync(path).size; | |
| 31 | } catch { | |
| 32 | return 0; // gone: nothing left to cache | |
| 33 | } | |
| 34 | }; | |
| 35 | ||
| 36 | /** | |
| 37 | * Count a handle's cache and map until it closes: wraps its `close` so the count leaves with | |
| 38 | * it. Returns the handle. | |
| 39 | */ | |
| 40 | export function noteHandle<D extends { close(...args: never[]): unknown }>( | |
| 41 | db: D, | |
| 42 | path: string, | |
| 43 | cacheBytes: number, | |
| 44 | mapBytes: number, | |
| 45 | ): D { | |
| 46 | if (open.size >= MAX_HANDLES) return db; | |
| 47 | const h: Handle = { path, cacheBytes: Math.max(0, cacheBytes), mapBytes: Math.max(0, mapBytes) }; | |
| 48 | open.add(h); | |
| 49 | const close = db.close.bind(db) as (...args: never[]) => unknown; | |
| 50 | (db as { close: (...args: never[]) => unknown }).close = (...args: never[]) => { | |
| 51 | open.delete(h); | |
| 52 | return close(...args); | |
| 53 | }; | |
| 54 | return db; | |
| 55 | } | |
| 56 | ||
| 57 | /** The ceiling of every open handle's cache and map, in bytes, against each file's size now. */ | |
| 58 | export function dbCeilingBytes(sizeOf: (path: string) => number = fileSize): number { | |
| 59 | let total = 0; | |
| 60 | for (const h of open) { | |
| 61 | const size = sizeOf(h.path); | |
| 62 | total += Math.min(h.cacheBytes, size) + Math.min(h.mapBytes, size); | |
| 63 | } | |
| 64 | return total; | |
| 65 | } | |
| 66 | ||
| 67 | /** How many handles are counted now (for a test). */ | |
| 68 | export const countedHandles = (): number => open.size; |
src/perf/memory_watch.ts +47 −11
| @@ -28,8 +28,16 @@ | ||
| 28 | 28 | * "gc" PerformanceObserver gets no entries), so a stall is attributed to the collector |
| 29 | 29 | * by what the heap did across it (`heapAcross`): a heap that fell is a collection that |
| 30 | 30 | * ran inside the stall. |
| 31 | * | |
| 32 | * What fills by design is not growth (#14343): each open database handle may fill its page cache | |
| 33 | * and its map as it is read (`kernel/db_ceiling.ts`), about 2.1 GiB on production's database, | |
| 34 | * most of it after the first hour. Each VM reads its own handles' ceiling with its reading (a | |
| 35 | * worker's crosses on the channel), and `grown` speaks only past twice the first hour *and* | |
| 36 | * past the first hour plus every fresh VM's ceiling. A leak still raises the need; a warm cache | |
| 37 | * at its ceiling, which levels off where a leak keeps climbing, does not. | |
| 31 | 38 | */ |
| 32 | 39 | import { heapSize, memoryUsage as jscMemory } from "bun:jsc"; |
| 40 | import { dbCeilingBytes } from "../kernel/db_ceiling.ts"; | |
| 33 | 41 | import { createLog } from "../kernel/log.ts"; |
| 34 | 42 | import { heldLine, threadHeld, type HeldCount } from "./held.ts"; |
| 35 | 43 | |
| @@ -51,17 +59,19 @@ export interface Reading { | ||
| 51 | 59 | /** `jsc.memoryUsage().currentCommit` and its peak: mimalloc's, for the whole process. */ |
| 52 | 60 | readonly commit: number; |
| 53 | 61 | readonly peakCommit: number; |
| 62 | /** What this VM's open databases may fill by design (`kernel/db_ceiling.ts`). */ | |
| 63 | readonly dbCeiling: number; | |
| 54 | 64 | } |
| 55 | 65 | |
| 56 | 66 | /** One reading of this VM: constant time, never `heapStats()`. */ |
| 57 | 67 | export function readNow(rss: number = process.memoryUsage().rss): Reading { |
| 58 | 68 | const m = jscMemory(); |
| 59 | return { rss, heap: heapSize(), commit: m.currentCommit, peakCommit: m.peakCommit }; | |
| 69 | return { rss, heap: heapSize(), commit: m.currentCommit, peakCommit: m.peakCommit, dbCeiling: dbCeilingBytes() }; | |
| 60 | 70 | } |
| 61 | 71 | |
| 62 | 72 | /** The serving VM's minute line: `memory main: rss 5.1 GiB, heap …`. */ |
| 63 | 73 | export function memoryLine(r: Reading, uptimeMs: number): string { |
| 64 | return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, process mimalloc committed ${mib(r.commit)} (peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`; | |
| 74 | return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, process mimalloc committed ${mib(r.commit)} (peak ${mib(r.peakCommit)}), database caches up to ${mib(r.dbCeiling)}, up ${Math.round(uptimeMs / 60000)} min`; | |
| 65 | 75 | } |
| 66 | 76 | |
| 67 | 77 | /** A worker VM's line: its heap is its own; RSS is the process's. */ |
| @@ -76,6 +86,8 @@ export interface Grown { | ||
| 76 | 86 | readonly since: number; |
| 77 | 87 | /** The owner's part that grew most against its own first hour, when one did (#12951). */ |
| 78 | 88 | readonly grew?: Growth; |
| 89 | /** Every fresh VM's database ceiling, when one is open: RSS was past the first hour plus it. */ | |
| 90 | readonly ceilingBytes?: number; | |
| 79 | 91 | } |
| 80 | 92 | |
| 81 | 93 | /** |
| @@ -94,10 +106,14 @@ export interface Growth { | ||
| 94 | 106 | readonly toBytes: number; |
| 95 | 107 | } |
| 96 | 108 | |
| 97 | /** A worker's minute reading as it crosses to the serving VM: its own heap. */ | |
| 109 | /** | |
| 110 | * A worker's minute reading as it crosses to the serving VM: its own heap, its databases' ceiling. | |
| 111 | */ | |
| 98 | 112 | export interface WorkerReading { |
| 99 | 113 | readonly name: string; |
| 100 | 114 | readonly heap: number; |
| 115 | /** Present when the worker has a database open (`kernel/db_ceiling.ts`). */ | |
| 116 | readonly ceiling?: number; | |
| 101 | 117 | } |
| 102 | 118 | |
| 103 | 119 | /** A worker silent this long is not blamed: it ended, or it stopped reporting. */ |
| @@ -112,7 +128,8 @@ export function workerReadingOf(data: unknown): WorkerReading | undefined { | ||
| 112 | 128 | if (typeof data !== "object" || data === null) return undefined; |
| 113 | 129 | const d = data as Record<string, unknown>; |
| 114 | 130 | if (typeof d.name !== "string" || !WORKER_NAME.test(d.name) || d.name === "main" || !bytes(d.heap)) return undefined; |
| 115 | return { name: d.name, heap: d.heap }; | |
| 131 | if (d.ceiling !== undefined && !bytes(d.ceiling)) return undefined; | |
| 132 | return d.ceiling === undefined || d.ceiling === 0 ? { name: d.name, heap: d.heap } : { name: d.name, heap: d.heap, ceiling: d.ceiling }; | |
| 116 | 133 | } |
| 117 | 134 | |
| 118 | 135 | /** One owner's first-hour sums per part, its baseline once that hour is over, its latest parts. */ |
| @@ -124,6 +141,8 @@ interface Owner { | ||
| 124 | 141 | latest: Partial<Record<Part, number>>; |
| 125 | 142 | /** The latest reading's heap, which main's `outside` excludes. */ |
| 126 | 143 | heap: number; |
| 144 | /** The latest reading's database ceiling. */ | |
| 145 | ceiling: number; | |
| 127 | 146 | at: number; |
| 128 | 147 | } |
| 129 | 148 | |
| @@ -134,6 +153,7 @@ const owner = (startedAt: number): Owner => ({ | ||
| 134 | 153 | base: undefined, |
| 135 | 154 | latest: {}, |
| 136 | 155 | heap: 0, |
| 156 | ceiling: 0, | |
| 137 | 157 | at: startedAt, |
| 138 | 158 | }); |
| 139 | 159 | |
| @@ -173,11 +193,16 @@ function mostGrown(owners: Iterable<readonly [string, Owner]>, now: number): Gro | ||
| 173 | 193 | } |
| 174 | 194 | |
| 175 | 195 | export interface MemoryWatch { |
| 176 | /** The serving VM's minute reading: the process's RSS and main's own heap. */ | |
| 177 | note(rss: number, heap: number, now?: number): void; | |
| 196 | /** | |
| 197 | * The serving VM's minute reading: the process's RSS, main's own heap, its databases' ceiling. | |
| 198 | */ | |
| 199 | note(rss: number, heap: number, now?: number, ceiling?: number): void; | |
| 178 | 200 | /** A worker's minute reading (`hearWorkers` carries it here), against its own first hour. */ |
| 179 | 201 | noteWorker(r: WorkerReading, now?: number): void; |
| 180 | /** RSS at least twice its first-hour mean, with the part that grew most, or undefined. */ | |
| 202 | /** | |
| 203 | * RSS at least twice its first-hour mean and past that mean plus every fresh VM's database | |
| 204 | * ceiling, with the part that grew most, or undefined. | |
| 205 | */ | |
| 181 | 206 | grown(): Grown | undefined; |
| 182 | 207 | /** |
| 183 | 208 | * The heap now (read here, or `heapNow` in a test) against the last minute's sample, for a |
| @@ -196,8 +221,9 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 196 | 221 | const main = owner(startedAt); |
| 197 | 222 | const workers = new Map<string, Owner>(); |
| 198 | 223 | return { |
| 199 | note(rss, heap, now = Date.now()) { | |
| 224 | note(rss, heap, now = Date.now(), ceiling = 0) { | |
| 200 | 225 | if (!Number.isFinite(rss) || rss <= 0) return; |
| 226 | main.ceiling = Number.isFinite(ceiling) && ceiling > 0 ? ceiling : 0; | |
| 201 | 227 | last = { heap, rss, at: now }; |
| 202 | 228 | latest = rss; |
| 203 | 229 | // Every fresh worker's heap is its own VM's, inside the process's RSS: not outside. |
| @@ -225,12 +251,21 @@ export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | ||
| 225 | 251 | workers.set(r.name, o); |
| 226 | 252 | } |
| 227 | 253 | o.heap = r.heap; |
| 254 | o.ceiling = r.ceiling ?? 0; | |
| 228 | 255 | take(o, { heap: r.heap }, now); |
| 229 | 256 | }, |
| 230 | 257 | grown() { |
| 231 | 258 | if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined; |
| 259 | let ceiling = main.ceiling; | |
| 260 | for (const w of workers.values()) if (main.at - w.at <= WORKER_STALE_MS) ceiling += w.ceiling; | |
| 261 | // A cache filling to its ceiling is the design, not a leak: only what is past it is growth. | |
| 262 | if (latest <= base + ceiling) return undefined; | |
| 232 | 263 | const grew = mostGrown([["main", main], ...workers], main.at); |
| 233 | return { baselineBytes: base, rssBytes: latest, since: above, ...(grew === undefined ? {} : { grew }) }; | |
| 264 | return { | |
| 265 | baselineBytes: base, rssBytes: latest, since: above, | |
| 266 | ...(grew === undefined ? {} : { grew }), | |
| 267 | ...(ceiling > 0 ? { ceilingBytes: ceiling } : {}), | |
| 268 | }; | |
| 234 | 269 | }, |
| 235 | 270 | heapAcross(now = Date.now(), heapNow = heapSize()) { |
| 236 | 271 | if (last === undefined) return `heap ${mib(heapNow)}`; |
| @@ -255,7 +290,8 @@ const grewWords = (g: Growth): string => | ||
| 255 | 290 | |
| 256 | 291 | /** The needs-you sentence for a grown process: the figure, and whose part grew most (#12951). */ |
| 257 | 292 | export function grownSentence(g: Grown): string { |
| 258 | const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}`; | |
| 293 | const past = g.ceilingBytes === undefined ? "" : ` and past the ${mib(g.ceilingBytes)} more its database caches may fill`; | |
| 294 | const head = `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}${past}`; | |
| 259 | 295 | const since = new Date(g.since).toISOString(); |
| 260 | 296 | if (g.grew === undefined) |
| 261 | 297 | return `${head}; it began growing at ${since}, and no VM's heap, nor the memory outside them, is above its own first hour.`; |
| @@ -278,7 +314,7 @@ export function tellReading(name: string, r: Reading): Reading { | ||
| 278 | 314 | told = new BroadcastChannel(MEMORY_CHANNEL); |
| 279 | 315 | (told as { unref?: () => void }).unref?.(); |
| 280 | 316 | } |
| 281 | told.postMessage({ name, heap: r.heap } satisfies WorkerReading); | |
| 317 | told.postMessage((r.dbCeiling > 0 ? { name, heap: r.heap, ceiling: r.dbCeiling } : { name, heap: r.heap }) satisfies WorkerReading); | |
| 282 | 318 | } catch { |
| 283 | 319 | // A closed channel loses one minute's reading; the line is still written. |
| 284 | 320 | } |
test/db-ceiling.test.ts added +51
| @@ -0,0 +1,51 @@ | ||
| 1 | /** | |
| 2 | * What the open databases may fill by design (#14343): counted from open to close, capped by each | |
| 3 | * file. | |
| 4 | */ | |
| 5 | import { expect, test } from "bun:test"; | |
| 6 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 7 | import { statSync } from "node:fs"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { openDb } from "../src/kernel/db.ts"; | |
| 11 | import { countedHandles, dbCeilingBytes, noteHandle } from "../src/kernel/db_ceiling.ts"; | |
| 12 | ||
| 13 | const MIB = 1048576; | |
| 14 | ||
| 15 | test("each handle counts min(cache, size) + min(map, size) until it closes", () => { | |
| 16 | const before = countedHandles(); | |
| 17 | const closed: string[] = []; | |
| 18 | const a = noteHandle({ close: () => closed.push("a") }, "/a", 1024 * MIB, 512 * MIB); | |
| 19 | const b = noteHandle({ close: () => closed.push("b") }, "/b", 128 * MIB, 512 * MIB); | |
| 20 | const sizes: Record<string, number> = { "/a": 4500 * MIB, "/b": 300 * MIB }; | |
| 21 | const others = dbCeilingBytes(() => 0); | |
| 22 | expect(others).toBe(0); | |
| 23 | // a: 1024 + 512 (a 4.5 GB file holds both); b: 128 + 300 (its map stops at the file). | |
| 24 | expect(dbCeilingBytes((p) => sizes[p] ?? 0)).toBe((1024 + 512 + 128 + 300) * MIB); | |
| 25 | a.close(); | |
| 26 | expect(closed).toEqual(["a"]); | |
| 27 | expect(countedHandles()).toBe(before + 1); | |
| 28 | expect(dbCeilingBytes((p) => sizes[p] ?? 0)).toBe((128 + 300) * MIB); | |
| 29 | b.close(); | |
| 30 | expect(countedHandles()).toBe(before); | |
| 31 | }); | |
| 32 | ||
| 33 | test("openDb counts a file's handle and forgets it on close; :memory: is not counted", async () => { | |
| 34 | const dir = await mkdtemp(join(tmpdir(), "forge-ceiling-")); | |
| 35 | const before = countedHandles(); | |
| 36 | try { | |
| 37 | const path = join(dir, "f.db"); | |
| 38 | const db = openDb(path, []); | |
| 39 | expect(countedHandles()).toBe(before + 1); | |
| 40 | // A small file: its cache and map can each hold the file at most. | |
| 41 | const size = statSync(path).size; | |
| 42 | expect(dbCeilingBytes((p) => (p === path ? size : 0))).toBe(2 * size); | |
| 43 | db.close(); | |
| 44 | expect(countedHandles()).toBe(before); | |
| 45 | const mem = openDb(":memory:", []); | |
| 46 | expect(countedHandles()).toBe(before); | |
| 47 | mem.close(); | |
| 48 | } finally { | |
| 49 | await rm(dir, { recursive: true, force: true }); | |
| 50 | } | |
| 51 | }); |
test/memory-watch.test.ts +54 −5
| @@ -66,8 +66,8 @@ test("a stall's heap with no argument is read from the VM", () => { | ||
| 66 | 66 | }); |
| 67 | 67 | |
| 68 | 68 | test("the minute lines name the owner and the figures", () => { |
| 69 | const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, commit: 512 * MIB, peakCommit: 700 * MIB }; | |
| 70 | expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, process mimalloc committed 512 MiB (peak 700 MiB), up 60 min"); | |
| 69 | const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, commit: 512 * MIB, peakCommit: 700 * MIB, dbCeiling: 1536 * MIB }; | |
| 70 | expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, process mimalloc committed 512 MiB (peak 700 MiB), database caches up to 1536 MiB, up 60 min"); | |
| 71 | 71 | expect(workerMemoryLine("engine", r)).toBe("memory worker engine: heap 1024 MiB, process rss 5120 MiB"); |
| 72 | 72 | expect(readNow().heap).toBeGreaterThan(0); |
| 73 | 73 | }); |
| @@ -75,7 +75,7 @@ test("the minute lines name the owner and the figures", () => { | ||
| 75 | 75 | // #14264: `jsc.memoryUsage().current` is the RSS, not malloc. Read as "malloc", RSS less |
| 76 | 76 | // main's heap blamed the main VM for every worker's heap and every native byte. |
| 77 | 77 | test("no part is read from Bun's memoryUsage().current", () => { |
| 78 | expect(Object.keys(readNow()).sort()).toEqual(["commit", "heap", "peakCommit", "rss"]); | |
| 78 | expect(Object.keys(readNow()).sort()).toEqual(["commit", "dbCeiling", "heap", "peakCommit", "rss"]); | |
| 79 | 79 | }); |
| 80 | 80 | |
| 81 | 81 | // #12951: which VM grew, and which part. First hour: rss 2000, main heap 500, worker pages |
| @@ -185,6 +185,9 @@ test("a worker's reading is validated at the boundary, and the owners are bounde | ||
| 185 | 185 | for (const bad of [null, 5, {}, { name: "Pages", heap: 1 }, { name: "main", heap: 1 }, { name: "a b", heap: 1 }, |
| 186 | 186 | { name: "x".repeat(41), heap: 1 }, { name: "p", heap: -1 }, { name: "p", heap: Infinity }, { name: "p", heap: "1" }]) |
| 187 | 187 | expect(workerReadingOf(bad)).toBeUndefined(); |
| 188 | expect(workerReadingOf({ name: "engine", heap: 1, ceiling: 2 })).toEqual({ name: "engine", heap: 1, ceiling: 2 }); | |
| 189 | expect(workerReadingOf({ name: "engine", heap: 1, ceiling: 0 })).toEqual({ name: "engine", heap: 1 }); | |
| 190 | for (const ceiling of [-1, NaN, Infinity, "2", null]) expect(workerReadingOf({ name: "engine", heap: 1, ceiling })).toBeUndefined(); | |
| 188 | 191 | const w = createMemoryWatch(T0); |
| 189 | 192 | for (let i = 0; i < MAX_OWNERS + 10; i++) w.noteWorker({ name: `w${i}`, heap: 1 }, T0); |
| 190 | 193 | owners(w); // pages and lanes arrive past the bound: never kept, never named |
| @@ -199,7 +202,7 @@ test("a worker's reading crosses on the memory channel to the watch that hears i | ||
| 199 | 202 | const { promise: two, resolve } = Promise.withResolvers<void>(); |
| 200 | 203 | const stop = hearWorkers({ noteWorker: (r) => { got.push(r); if (got.length === 2) resolve(); } }); |
| 201 | 204 | try { |
| 202 | const r = { rss: 9, heap: 3 * MIB, commit: 7 * MIB, peakCommit: 9 }; | |
| 205 | const r = { rss: 9, heap: 3 * MIB, commit: 7 * MIB, peakCommit: 9, dbCeiling: 640 * MIB }; | |
| 203 | 206 | expect(tellReading("pages", r)).toBe(r); |
| 204 | 207 | const other = new BroadcastChannel(MEMORY_CHANNEL); |
| 205 | 208 | other.postMessage({ name: "../etc", heap: 1 }); // malformed: dropped |
| @@ -208,9 +211,55 @@ test("a worker's reading crosses on the memory channel to the watch that hears i | ||
| 208 | 211 | other.close(); |
| 209 | 212 | expect(got.sort((a, b) => a.name.localeCompare(b.name))).toEqual([ |
| 210 | 213 | { name: "lanes", heap: 1 }, |
| 211 | { name: "pages", heap: 3 * MIB }, | |
| 214 | { name: "pages", heap: 3 * MIB, ceiling: 640 * MIB }, | |
| 212 | 215 | ]); |
| 213 | 216 | } finally { |
| 214 | 217 | stop(); |
| 215 | 218 | } |
| 216 | 219 | }); |
| 220 | ||
| 221 | // #14343: production's evidence held 3.0 to 3.3 GiB for twelve hours against a 1420 MiB first | |
| 222 | // hour: a plateau, the shape of a cache at its ceiling, not of a leak. The serving handle may | |
| 223 | // fill 1024 MiB of cache and 512 MiB of map; the engine's 128 and 512: 2176 MiB in all. | |
| 224 | const SERVING = 1536 * MIB; | |
| 225 | const ENGINE = 640 * MIB; | |
| 226 | function warmHour(w: ReturnType<typeof createMemoryWatch>) { | |
| 227 | for (let i = 0; i < 12; i++) { | |
| 228 | const at = T0 + i * 300_000; | |
| 229 | w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, at); | |
| 230 | w.note(1420 * MIB, 400 * MIB, at, SERVING); | |
| 231 | } | |
| 232 | } | |
| 233 | ||
| 234 | test("a database cache filling to its ceiling is not growth: production's 3290 MiB raises nothing", () => { | |
| 235 | const w = createMemoryWatch(T0); | |
| 236 | warmHour(w); | |
| 237 | w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, HOUR + 60_000); | |
| 238 | w.note(3290 * MIB, 400 * MIB, HOUR + 60_000, SERVING); // 2.3x, but 1420 + 2176 = 3596 is the line | |
| 239 | expect(w.grown()).toBeUndefined(); | |
| 240 | w.note(3596 * MIB, 400 * MIB, HOUR + 120_000, SERVING); // at the line, not past it | |
| 241 | expect(w.grown()).toBeUndefined(); | |
| 242 | }); | |
| 243 | ||
| 244 | test("past the first hour plus every database's ceiling, the need speaks and names the ceiling", () => { | |
| 245 | const w = createMemoryWatch(T0); | |
| 246 | warmHour(w); | |
| 247 | const at = HOUR + 60_000; | |
| 248 | w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, at); | |
| 249 | w.note(3700 * MIB, 400 * MIB, at, SERVING); | |
| 250 | const g = w.grown()!; | |
| 251 | expect(g.ceilingBytes).toBe(2176 * MIB); | |
| 252 | expect(grownSentence(g)).toStartWith( | |
| 253 | "The forge's own memory is 3700 MiB, over twice its first-hour 1420 MiB and past the 2176 MiB more its database caches may fill; ", | |
| 254 | ); | |
| 255 | }); | |
| 256 | ||
| 257 | test("a worker silent five minutes no longer lends its ceiling", () => { | |
| 258 | const w = createMemoryWatch(T0); | |
| 259 | warmHour(w); | |
| 260 | w.noteWorker({ name: "engine", heap: 200 * MIB, ceiling: ENGINE }, HOUR); | |
| 261 | w.note(3290 * MIB, 400 * MIB, HOUR + 60_000, SERVING); | |
| 262 | expect(w.grown()).toBeUndefined(); // 1420 + 1536 + 640 = 3596 | |
| 263 | w.note(3290 * MIB, 400 * MIB, HOUR + WORKER_STALE_MS + 60_000, SERVING); | |
| 264 | expect(w.grown()?.ceilingBytes).toBe(SERVING); // 1420 + 1536 = 2956 < 3290 | |
| 265 | }); |
new above
offline