Public: anyone can read this; the crew can change it.
Since Bun 1.4.4 the serving process holds 4.5 to 7 GiB after four hours (was 1.8 to 2.4) and stalls three times as often as it ages #12902
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/12901-since-bun-1-4-4-the-serving-process-hold → main 5 commits fork 4 threads · 4 unresolved
15 files
changed +311
−3
docs/features/README.md +1
| @@ -417,6 +417,7 @@ Read with the guide page [When something is wrong](../guide-when-wrong.md). | ||
| 417 | 417 | - [](stalls-by-route.md) |
| 418 | 418 | - [](maintenance-week.md) |
| 419 | 419 | - [](memory-held.md) |
| 420 | - [](memory-growth.md) | |
| 420 | 421 | - [](offline.md) |
| 421 | 422 | - [](slow-pages.md) |
| 422 | 423 | - [](webhook-pause.md) |
docs/features/memory-growth.md added +27
| @@ -0,0 +1,27 @@ | ||
| 1 | # Memory by owner, the heap across a stall, memory growth (#12901) | |
| 2 | ||
| 3 | Since Bun 1.4.4 the serving process has held several GiB after hours. This change makes the growth readable; it does not stop it (see "Not done"). | |
| 4 | ||
| 5 | ## Where it shows | |
| 6 | ||
| 7 | - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. A worker VM writes `memory worker engine: heap …, malloc …, process rss …` on its own minute. RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log. | |
| 8 | - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample. | |
| 9 | - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured. | |
| 10 | - `did not fall: no collection seen` means the stall was not the collector's. | |
| 11 | - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure and when it began growing. | |
| 12 | ||
| 13 | ## What to do when the need shows | |
| 14 | ||
| 15 | 1. Read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker. | |
| 16 | 2. Check the stall lines for `fell: a collection ran`: stalls that are collections point at the heap. | |
| 17 | 3. Restart the forge to release it, and report the lines with the issue. | |
| 18 | ||
| 19 | The need clears by itself once RSS falls under twice the baseline. A process that slept through its first hour has no baseline and raises none. | |
| 20 | ||
| 21 | ## How it is built | |
| 22 | ||
| 23 | `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Tests: `test/memory-watch.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`. | |
| 24 | ||
| 25 | ## Not done | |
| 26 | ||
| 27 | Finding what grows and fixing it, and wiring the other workers to write their own line. |
src/actions/health.ts +1
| @@ -81,6 +81,7 @@ function gatherVitals(app: App): Vitals { | ||
| 81 | 81 | dataDir: app.config.dataDir, |
| 82 | 82 | ...(app.config.disk ? { disk: app.config.disk } : {}), |
| 83 | 83 | ...(app.config.hostFree ? { hostFree: app.config.hostFree } : {}), |
| 84 | ...(app.config.memoryWatch ? { watch: app.config.memoryWatch } : {}), | |
| 84 | 85 | envs, |
| 85 | 86 | signals: app.signals, |
| 86 | 87 | forgeUrl: app.config.url ?? `http://localhost:${app.config.port}`, |
src/app.ts +2
| @@ -53,6 +53,8 @@ export interface ForgeConfig { | ||
| 53 | 53 | * when unset. Tests give it a reading. |
| 54 | 54 | */ |
| 55 | 55 | hostFree?: () => number; |
| 56 | /** The memory watch vitals reads for growth; the process's own when unset. Tests give it one. */ | |
| 57 | memoryWatch?: { grown(): { baselineBytes: number; rssBytes: number; since: number } | undefined }; | |
| 56 | 58 | /** |
| 57 | 59 | * The machine's memory in bytes, which the forges' sizes are summed against |
| 58 | 60 | * (`spawn/memory_fit.ts`); its pod's limit or its own when unset. Tests give it a figure. |
src/boards/memory.ts +7 −2
| @@ -5,15 +5,17 @@ | ||
| 5 | 5 | * directory), as a count, a sum and a max. RSS rising while the |
| 6 | 6 | * heap stays flat is native memory growth. A week of hours is kept. |
| 7 | 7 | * |
| 8 | * Each tick is also one log line (`perf/memory_watch.ts`, #12901). | |
| 9 | * | |
| 8 | 10 | * `heapSize()`, never `heapStats().heapSize`: the same byte count, but |
| 9 | 11 | * `heapStats()` first counts every object on the heap by type, which held the |
| 10 | 12 | * loop ~280 ms a minute on a 2M-object heap and grows with it (#11118). |
| 11 | 13 | */ |
| 12 | 14 | import type { Database } from "bun:sqlite"; |
| 13 | import { heapSize } from "bun:jsc"; | |
| 14 | 15 | import type { App } from "../app.ts"; |
| 15 | 16 | import { quietly } from "../core/write_clock.ts"; |
| 16 | 17 | import { during } from "../perf/jobs.ts"; |
| 18 | import { memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts"; | |
| 17 | 19 | |
| 18 | 20 | export const MEMORY_EVERY_MS = 60_000; |
| 19 | 21 | export const HOUR_MS = 3_600_000; |
| @@ -68,7 +70,10 @@ export type MemorySampler = { tick(now?: number): void; stop(): void }; | ||
| 68 | 70 | export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemorySampler { |
| 69 | 71 | const tick = (now = Date.now()) => { |
| 70 | 72 | try { |
| 71 | recordMemory(app.db, process.memoryUsage().rss, heapSize(), now); | |
| 73 | const reading = readNow(); | |
| 74 | recordMemory(app.db, reading.rss, reading.heap, now); | |
| 75 | memoryWatch.note(reading.rss, reading.heap, now); | |
| 76 | app.log.info(memoryLine(reading, process.uptime() * 1000)); | |
| 72 | 77 | } catch (error) { |
| 73 | 78 | app.log.warn("memory sample failed", { error: String(error) }); |
| 74 | 79 | } |
src/engine/crossing.ts +2
| @@ -63,6 +63,8 @@ export const STAYS = { | ||
| 63 | 63 | "a function: a test's stand-in for the free-memory probe; the engine's thread reads the real figure", |
| 64 | 64 | memory: |
| 65 | 65 | "a function: a test's stand-in for the machine's memory; the engine's thread reads the real figure", |
| 66 | memoryWatch: | |
| 67 | "an object with a method: a test's stand-in for the serving VM's memory watch; a worker writes its own minute line", | |
| 66 | 68 | decider: |
| 67 | 69 | "an object with a method; no duty of the engine asks it, and one that comes to must make its own on that thread", |
| 68 | 70 | mailer: |
src/engine/worker.ts +4
| @@ -14,6 +14,7 @@ import { startDuties } from "./duties.ts"; | ||
| 14 | 14 | import { timeStatements } from "../kernel/slow_sql.ts"; |
| 15 | 15 | import { createLog } from "../kernel/log.ts"; |
| 16 | 16 | import { during } from "../perf/jobs.ts"; |
| 17 | import { startWorkerMemory } from "../perf/memory_watch.ts"; | |
| 17 | 18 | import { BEAT_MS, FAILED } from "./beat.ts"; |
| 18 | 19 | |
| 19 | 20 | declare const self: Worker; |
| @@ -65,9 +66,12 @@ async function boot(init: EngineInit): Promise<void> { | ||
| 65 | 66 | }; |
| 66 | 67 | // Alive: a thread stuck in one statement stops saying so, and the serving thread ends it (`host.ts`). |
| 67 | 68 | const beat = setInterval(() => during("engine-beat", () => postMessage("h"), undefined, app), BEAT_MS); |
| 69 | const memoryLog = createLog("memory"); | |
| 70 | const stopMemory = startWorkerMemory("engine", (line) => memoryLog.info(line)); | |
| 68 | 71 | const was = stop; |
| 69 | 72 | stop = () => { |
| 70 | 73 | clearInterval(beat); |
| 74 | stopMemory(); | |
| 71 | 75 | was?.(); |
| 72 | 76 | }; |
| 73 | 77 | postMessage("ready"); |
src/kernel/vitals.ts +8
| @@ -11,6 +11,7 @@ | ||
| 11 | 11 | import type { Database } from "bun:sqlite"; |
| 12 | 12 | import { statSync } from "node:fs"; |
| 13 | 13 | import { fullSentence, fullSince } from "./full_disk.ts"; |
| 14 | import { grownSentence, memoryWatch, type MemoryWatch } from "../perf/memory_watch.ts"; | |
| 14 | 15 | import { readDisk, type DiskReading } from "./disk_reading.ts"; |
| 15 | 16 | import { DISK_FLOOR, roomy } from "../spawn/disk_floor.ts"; |
| 16 | 17 | import { dirname } from "node:path"; |
| @@ -185,6 +186,8 @@ export interface VitalsInput { | ||
| 185 | 186 | readonly forgeUrl?: string; |
| 186 | 187 | /** Reads the disk under `dataDir`; `statDisk` unless a test gives it a reading. */ |
| 187 | 188 | readonly disk?: (dir: string) => DiskReading; |
| 189 | /** The memory watch whose growth is a need; the process's own unless a test gives one. */ | |
| 190 | readonly watch?: Pick<MemoryWatch, "grown">; | |
| 188 | 191 | /** The free-disk share below which the disk is a need; `DISK_FREE_MIN` unless a test says. */ |
| 189 | 192 | readonly diskFreeMin?: number; |
| 190 | 193 | /** The pod's memory limit and usage; read from the cgroup unless a test gives them. */ |
| @@ -445,6 +448,11 @@ export function vitals(input: VitalsInput): Vitals { | ||
| 445 | 448 | if (hostProbe !== undefined && memoryFloorMiB > 0 && hostFreeBytes < memoryFloorMiB * MIB) |
| 446 | 449 | need("broken", "memory", memorySentence(hostFreeBytes, memoryFloorMiB), undefined, "/settings/capacity"); |
| 447 | 450 | |
| 451 | // Twice its first hour (#12901, #12455): the figure and when it began growing. | |
| 452 | const grown = (input.watch ?? memoryWatch).grown(); | |
| 453 | if (grown !== undefined) | |
| 454 | need("todo", "memory-growth", grownSentence(grown), new Date(grown.since).toISOString(), "/system"); | |
| 455 | ||
| 448 | 456 | const fullAt = fullSince(input.dataDir); |
| 449 | 457 | if (fullAt !== undefined) need("broken", "disk-full", fullSentence(input.dataDir, true), fullAt, "/system"); |
| 450 | 458 | const disk = (input.disk ?? readDisk)(input.dataDir); |
src/main.ts +2 −1
| @@ -135,6 +135,7 @@ import { createStallRelay } from "./kernel/stall_relay.ts"; | ||
| 135 | 135 | import { startPerfWatcher, TICK_MS } from "./perf/duty.ts"; |
| 136 | 136 | import { during } from "./perf/jobs.ts"; |
| 137 | 137 | import { stallLine } from "./perf/loop.ts"; |
| 138 | import { memoryWatch } from "./perf/memory_watch.ts"; | |
| 138 | 139 | import { bootStall, gitRecord, liveOf } from "./perf/live.ts"; |
| 139 | 140 | import { keepFlowReadings, KEPT_MS } from "./flow/kept.ts"; |
| 140 | 141 | import { startKeeping } from "./flow/keep.ts"; |
| @@ -278,7 +279,7 @@ const stallRelay = createStallRelay((live: ReturnType<typeof liveOf>, late, at) | ||
| 278 | 279 | live.lagMinutes.note(late, at); |
| 279 | 280 | const stall = live.loopWatch.stalled(late, at); |
| 280 | 281 | if (stall === null) return; |
| 281 | loopLog.warn(stallLine(stall)); | |
| 282 | loopLog.warn(`${stallLine(stall)}; ${memoryWatch.heapAcross()}`); | |
| 282 | 283 | live.stallBlame.note(stall); |
| 283 | 284 | }); |
| 284 | 285 | watchStalls(loopLog, undefined, undefined, undefined, (late, at) => { |
src/perf/memory_watch.ts added +121
| @@ -0,0 +1,121 @@ | ||
| 1 | /** | |
| 2 | * What holds the forge's memory, by owner (#12901). Once a minute the serving VM's | |
| 3 | * reading (`bun:jsc` `heapSize()` and `memoryUsage()`) and the process's RSS are one log line | |
| 4 | * (`memoryLine`), and a worker VM writes its own (`workerMemoryLine`), so growth reads | |
outdated
Stale claim: no code reads heapStats().extraMemorySize or objectCount. Remove it from the header. |
||
| 5 | * in the log by its owner. The watch also keeps the first-hour RSS figure and says when | |
| 6 | * RSS passes twice it (`grown`), which the needs-you line reads (`kernel/vitals.ts`). | |
| 7 | * | |
| 8 | * `heapStats()` counts every object on the heap by type and held the loop ~280 ms a | |
| 9 | * minute on a 2M-object heap (#11118), so the minute line never calls it: `heapSize()` | |
| 10 | * and `memoryUsage()` are constant-time. Bun 1.4.4 exposes no collector time (a | |
| 11 | * "gc" PerformanceObserver gets no entries), so a stall is attributed to the collector | |
| 12 | * by what the heap did across it (`heapAcross`): a heap that fell is a collection that | |
| 13 | * ran inside the stall. | |
| 14 | */ | |
| 15 | import { heapSize, memoryUsage as jscMemory } from "bun:jsc"; | |
| 16 | ||
| 17 | const MIB = 1048576; | |
| 18 | export const FIRST_HOUR_MS = 3_600_000; | |
| 19 | /** Samples the first hour must hold before its mean is a figure (an idle boot is not one). */ | |
| 20 | export const MIN_SAMPLES = 10; | |
| 21 | /** The ratio at which the needs-you line speaks (#12455). */ | |
| 22 | export const GROWN_RATIO = 2; | |
| 23 | /** RSS above this share of the baseline counts as growing, for the time it started. */ | |
| 24 | const GROWING_RATIO = 1.25; | |
| 25 | ||
| 26 | const mib = (bytes: number): string => `${(bytes / MIB).toFixed(bytes < 100 * MIB ? 1 : 0)} MiB`; | |
| 27 | ||
| 28 | export interface Reading { | |
| 29 | readonly rss: number; | |
| 30 | readonly heap: number; | |
| 31 | /** `jsc.memoryUsage()`: the allocator's current and peak bytes, and peak committed. */ | |
| 32 | readonly current: number; | |
| 33 | readonly peak: number; | |
| 34 | readonly peakCommit: number; | |
| 35 | } | |
| 36 | ||
| 37 | /** One reading of this VM: constant time, never `heapStats()`. */ | |
| 38 | export function readNow(rss: number = process.memoryUsage().rss): Reading { | |
| 39 | const m = jscMemory(); | |
| 40 | return { rss, heap: heapSize(), current: m.current, peak: m.peak, peakCommit: m.peakCommit }; | |
| 41 | } | |
| 42 | ||
| 43 | /** The serving VM's minute line: `memory main: rss 5.1 GiB, heap …`. */ | |
| 44 | export function memoryLine(r: Reading, uptimeMs: number): string { | |
| 45 | return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}, committed peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`; | |
| 46 | } | |
| 47 | ||
| 48 | /** A worker VM's line: its heap and malloc are its own; RSS is the process's. */ | |
| 49 | export function workerMemoryLine(name: string, r: Reading): string { | |
| 50 | return `memory worker ${name}: heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}), process rss ${mib(r.rss)}`; | |
| 51 | } | |
| 52 | ||
| 53 | export interface Grown { | |
| 54 | readonly baselineBytes: number; | |
| 55 | readonly rssBytes: number; | |
| 56 | /** Epoch ms of the first sample of the run above 1.25x the baseline. */ | |
| 57 | readonly since: number; | |
| 58 | } | |
| 59 | ||
| 60 | export interface MemoryWatch { | |
| 61 | note(rss: number, heap: number, now?: number): void; | |
| 62 | /** RSS at least twice its first-hour mean, or undefined. */ | |
| 63 | grown(): Grown | undefined; | |
| 64 | /** | |
| 65 | * The heap now (read here, or `heapNow` in a test) against the last minute's sample, for a | |
| 66 | * stall line. | |
| 67 | */ | |
| 68 | heapAcross(now?: number, heapNow?: number): string; | |
| 69 | } | |
| 70 | ||
| 71 | export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch { | |
| 72 | let sum = 0; | |
| 73 | let count = 0; | |
| 74 | let base: number | undefined; | |
| 75 | let last: { heap: number; rss: number; at: number } | undefined; | |
| 76 | let above: number | undefined; | |
| 77 | let latest: number | undefined; | |
| 78 | return { | |
| 79 | note(rss, heap, now = Date.now()) { | |
| 80 | if (!Number.isFinite(rss) || rss <= 0) return; | |
| 81 | last = { heap, rss, at: now }; | |
| 82 | latest = rss; | |
baseline() is used only by tests. Remove it or give it a caller. |
||
| 83 | if (base === undefined) { | |
| 84 | if (now - startedAt < FIRST_HOUR_MS) { | |
| 85 | sum += rss; | |
| 86 | count++; | |
| 87 | return; | |
| 88 | } | |
| 89 | if (count < MIN_SAMPLES) return; // the process slept through its first hour: no figure | |
| 90 | base = sum / count; | |
| 91 | } | |
| 92 | if (rss > base * GROWING_RATIO) above ??= now; | |
| 93 | else above = undefined; | |
| 94 | }, | |
| 95 | grown() { | |
| 96 | if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined; | |
| 97 | return { baselineBytes: base, rssBytes: latest, since: above }; | |
| 98 | }, | |
| 99 | heapAcross(now = Date.now(), heapNow = heapSize()) { | |
| 100 | if (last === undefined) return `heap ${mib(heapNow)}`; | |
| 101 | const was = `${mib(last.heap)} ${Math.max(0, Math.round((now - last.at) / 1000))} s ago`; | |
| 102 | const verdict = heapNow < last.heap * 0.9 ? "fell: a collection ran" : "did not fall: no collection seen"; | |
| 103 | return `heap ${mib(heapNow)} (was ${was}; ${verdict})`; | |
| 104 | }, | |
| 105 | }; | |
| 106 | } | |
| 107 | ||
| 108 | /** This process's watch, started with it. */ | |
| 109 | export const memoryWatch: MemoryWatch = createMemoryWatch(); | |
| 110 | ||
| 111 | /** The needs-you sentence for a grown process. */ | |
| 112 | export function grownSentence(g: Grown): string { | |
| 113 | return `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}; it began growing at ${new Date(g.since).toISOString()}. The log's "memory main" and "memory worker" lines say which VM holds it.`; | |
| 114 | } | |
| 115 | ||
| 116 | /** Start a worker's own minute line (an unref'd timer inside the worker). Returns the stop. */ | |
| 117 | export function startWorkerMemory(name: string, write: (line: string) => void, everyMs = 60_000): () => void { | |
| 118 | const timer = setInterval(() => write(workerMemoryLine(name, readNow())), everyMs); | |
| 119 | (timer as { unref?: () => void }).unref?.(); | |
| 120 | return () => clearInterval(timer); | |
| 121 | } |
test/engine-crossing.test.ts +2
| @@ -177,6 +177,7 @@ describe("every setting is on one list, and what crosses is data", () => { | ||
| 177 | 177 | "hostFree", |
| 178 | 178 | "mailer", |
| 179 | 179 | "memory", |
| 180 | "memoryWatch", | |
| 180 | 181 | "verifyHuman", |
| 181 | 182 | ]); |
| 182 | 183 | expect(Object.keys(STAYS).sort()).toEqual( |
| @@ -203,6 +204,7 @@ const whole = (dataDir: string): Required<ForgeConfig> => ({ | ||
| 203 | 204 | seedSignup: true, |
| 204 | 205 | disk: () => ({ freeBytes: 1, totalBytes: 2 }), |
| 205 | 206 | hostFree: () => 1, |
| 207 | memoryWatch: { grown: () => undefined }, | |
| 206 | 208 | memory: () => 1, |
| 207 | 209 | codeBudgetMs: 30, |
| 208 | 210 | port: 8080, |
test/memory-watch.test.ts added +65
| @@ -0,0 +1,65 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { createMemoryWatch, grownSentence, memoryLine, workerMemoryLine, FIRST_HOUR_MS, readNow } from "../src/perf/memory_watch.ts"; | |
| 3 | ||
| 4 | const MIB = 1048576; | |
| 5 | const T0 = 1_000_000_000_000; | |
| 6 | ||
| 7 | function firstHour(w: ReturnType<typeof createMemoryWatch>, rss: number) { | |
| 8 | for (let i = 0; i < 12; i++) w.note(rss, rss / 2, T0 + i * 300_000); | |
| 9 | } | |
| 10 | ||
| 11 | test("no figure, and no growth, before the first hour has passed", () => { | |
| 12 | const w = createMemoryWatch(T0); | |
| 13 | w.note(2000 * MIB, 1, T0 + 1000); | |
| 14 | w.note(9000 * MIB, 1, T0 + 2000); | |
| 15 | expect(w.grown()).toBeUndefined(); | |
| 16 | }); | |
| 17 | ||
| 18 | test("raises once RSS passes twice the first-hour mean, with the figure and when it began", () => { | |
| 19 | const w = createMemoryWatch(T0); | |
| 20 | firstHour(w, 2000 * MIB); // mean 2000 MiB | |
| 21 | w.note(2200 * MIB, 1, T0 + FIRST_HOUR_MS + 60_000); | |
| 22 | expect(w.grown()).toBeUndefined(); // 1.1x | |
| 23 | const began = T0 + FIRST_HOUR_MS + 120_000; | |
| 24 | w.note(2600 * MIB, 1, began); // 1.3x: growing, not yet 2x | |
| 25 | w.note(3900 * MIB, 1, began + 60_000); // 1.95x | |
| 26 | expect(w.grown()).toBeUndefined(); | |
| 27 | w.note(4100 * MIB, 1, began + 120_000); // 2.05x | |
| 28 | expect(w.grown()).toEqual({ baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: began }); | |
| 29 | expect(grownSentence(w.grown()!)).toContain("4100 MiB, over twice its first-hour 2000 MiB"); | |
| 30 | expect(grownSentence(w.grown()!)).toContain(new Date(began).toISOString()); | |
| 31 | }); | |
| 32 | ||
| 33 | test("a drop back under 1.25x forgets when it began; an idle first hour gives no figure", () => { | |
| 34 | const w = createMemoryWatch(T0); | |
| 35 | firstHour(w, 1000 * MIB); | |
| 36 | const t = T0 + FIRST_HOUR_MS; | |
| 37 | w.note(1500 * MIB, 1, t + 1000); | |
| 38 | w.note(1000 * MIB, 1, t + 2000); | |
| 39 | w.note(2500 * MIB, 1, t + 3000); | |
| 40 | expect(w.grown()?.since).toBe(t + 3000); | |
| 41 | const idle = createMemoryWatch(T0); | |
| 42 | idle.note(1000 * MIB, 1, T0 + 1000); | |
| 43 | idle.note(9000 * MIB, 1, T0 + FIRST_HOUR_MS + 1000); | |
| 44 | expect(idle.grown()).toBeUndefined(); | |
| 45 | }); | |
| 46 | ||
| 47 | test("a stall's heap is read against the last minute: a fall is a collection", () => { | |
| 48 | const w = createMemoryWatch(T0); | |
| 49 | expect(w.heapAcross(T0, 500 * MIB)).toBe("heap 500 MiB"); | |
| 50 | w.note(3000 * MIB, 1000 * MIB, T0); | |
| 51 | expect(w.heapAcross(T0 + 41_000, 400 * MIB)).toBe("heap 400 MiB (was 1000 MiB 41 s ago; fell: a collection ran)"); | |
| 52 | expect(w.heapAcross(T0 + 5000, 1000 * MIB)).toContain("did not fall"); | |
| 53 | }); | |
| 54 | ||
| 55 | test("a stall's heap with no argument is read from the VM", () => { | |
| 56 | const w = createMemoryWatch(T0); | |
| 57 | expect(w.heapAcross()).toMatch(/^heap \d/); | |
| 58 | }); | |
| 59 | ||
| 60 | test("the minute lines name the owner and the figures", () => { | |
| 61 | const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, current: 512 * MIB, peak: 600 * MIB, peakCommit: 700 * MIB }; | |
| 62 | expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, malloc 512 MiB (peak 600 MiB, committed peak 700 MiB), up 60 min"); | |
| 63 | expect(workerMemoryLine("engine", r)).toStartWith("memory worker engine: heap 1024 MiB"); | |
| 64 | expect(readNow().heap).toBeGreaterThan(0); | |
| 65 | }); |
test/timers-named.test.ts +1
| @@ -81,6 +81,7 @@ const ALLOWED: readonly Allowed[] = [ | ||
| 81 | 81 | ...allow("setTimeout", "an app request's look-again sleep in the ladder: it holds one request, does no work on the loop, and ends within the bound", [["src/http/app_turn.ts", "r"]]), |
| 82 | 82 | ...allow("setTimeout", `a media stream's idle deadline, ${PER_CONNECTION}`, [["src/git/read.ts", "() => { finish(); controller.error("]]), |
| 83 | 83 | ...allow("setTimeout", `a git upload's idle deadline, ${PER_CONNECTION}`, [["src/git/smart-http.ts", "stop"]]), |
| 84 | ...allow("setInterval", "a worker VM's minute memory line: it runs in the worker, where the main process's span ring is not reachable, and reads two counters", [["src/perf/memory_watch.ts", "() => write(workerMemoryLine("]]), | |
| 84 | 85 | ...allow("setTimeout", "the loop-lag meter's own tick: it measures how late the loop runs, and a span around it would count that lateness as its own work", [["src/kernel/loop_lag.ts", "() => { // Early"]]), |
| 85 | 86 | ...allow("setTimeout", "the next slice of a sliced flow computation: one span per slice would fill the 256-span ring in one refresh", [["src/system/flow_cache.ts", "fn"]]), |
| 86 | 87 | ...allow("setTimeout", "the release prover's scheduler seam: its caller passes a callback already inside during(\"release-prove\")", [["src/release/prove.ts", "fn"]]), |
test/tui-memory-growth.test.ts added +54
| @@ -0,0 +1,54 @@ | ||
| 1 | /** | |
| 2 | * The memory-growth need reaches the terminal's home as it reaches /system: both read | |
| 3 | * `forgeNeedList` (#12901). | |
| 4 | */ | |
| 5 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 6 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 7 | import { tmpdir } from "node:os"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import { createApp, type App as Forge } from "../src/app.ts"; | |
| 10 | import { forgeNeedList } from "../src/actions/health.ts"; | |
| 11 | import { createForgeApp } from "../src/tui/forge.ts"; | |
| 12 | import { headlessTerminal } from "../src/tui/term/terminal.ts"; | |
| 13 | import { createMemoryWatch, FIRST_HOUR_MS } from "../src/perf/memory_watch.ts"; | |
| 14 | import { quietLogs } from "./helpers/globals.ts"; | |
| 15 | import { settleApp } from "./helpers/tui.ts"; | |
| 16 | import { seedAdmin } from "./helpers/admin.ts"; | |
| 17 | ||
| 18 | quietLogs(); | |
| 19 | const MIB = 1048576; | |
| 20 | let dir: string; | |
| 21 | let forge: Forge; | |
| 22 | ||
| 23 | beforeAll(async () => { | |
| 24 | dir = await mkdtemp(join(tmpdir(), "forge-tui-growth-")); | |
| 25 | const start = Date.now() - 2 * FIRST_HOUR_MS; | |
| 26 | const w = createMemoryWatch(start); | |
| 27 | for (let i = 0; i < 12; i++) w.note(1000 * MIB, 1, start + i * 300_000); | |
| 28 | w.note(2500 * MIB, 1, start + FIRST_HOUR_MS + 1000); | |
| 29 | forge = createApp({ dataDir: dir, port: 0, memoryWatch: w }); | |
| 30 | }); | |
| 31 | ||
| 32 | afterAll(async () => { | |
| 33 | forge.close(); | |
| 34 | await rm(dir, { recursive: true, force: true }); | |
| 35 | }); | |
| 36 | ||
| 37 | test("an admin's terminal home lists the memory-growth need beside the same need in the shared list", async () => { | |
| 38 | const a = await seedAdmin(forge, { username: "alice", email: "[email protected]", password: "a-long-passphrase" }); | |
| 39 | if (!a.ok) throw new Error("signup failed"); | |
| 40 | const needs = forgeNeedList(forge, a.value)!; | |
| 41 | expect(needs.find((n) => n.cause === "memory-growth")?.href).toBe("/system"); | |
| 42 | const terminal = headlessTerminal(200, 40); | |
| 43 | const { app } = createForgeApp( | |
| 44 | forge, | |
| 45 | { sshUsername: "alice", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user: a.value }, | |
| 46 | { terminal, exitHooks: false }, | |
| 47 | ); | |
| 48 | app.start(); | |
| 49 | await settleApp(app); | |
| 50 | const text = app.screen.toText(); | |
| 51 | app.stop(); | |
| 52 | expect(text).toContain("2500 MiB"); | |
| 53 | expect(text).toContain("twice its first-hour"); | |
| 54 | }); |
test/vitals.test.ts +14
| @@ -566,3 +566,17 @@ test("vitals memory follows the pod's injected limit and usage (#12347)", async | ||
| 566 | 566 | await rm(dir, { recursive: true, force: true }); |
| 567 | 567 | } |
| 568 | 568 | }); |
| 569 | ||
This monkey-patches the process-wide memoryWatch and asserts on its pristine state. Inject the watch into vitals() instead, so the test exercises the real path and cannot depend on test order. |
||
| 570 | test("memory twice its first hour is one todo need with the figure and when it began (#12901)", async () => { | |
| 571 | const { FIRST_HOUR_MS, createMemoryWatch } = await import("../src/perf/memory_watch.ts"); | |
| 572 | const MIB = 1048576; | |
| 573 | const start = Date.now() - 2 * FIRST_HOUR_MS; | |
| 574 | const w = createMemoryWatch(start); | |
| 575 | for (let i = 0; i < 12; i++) w.note(1000 * MIB, 1, start + i * 300_000); | |
| 576 | expect(vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w }).degraded.some((x) => x.cause === "memory-growth")).toBe(false); | |
| 577 | w.note(2500 * MIB, 1, start + FIRST_HOUR_MS + 1000); | |
| 578 | const v = vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w }); | |
| 579 | const d = v.degraded.find((x) => x.cause === "memory-growth")!; | |
| 580 | expect(d.sentence).toContain("2500 MiB, over twice its first-hour 1000 MiB"); | |
| 581 | expect(d.since).toBe(new Date(start + FIRST_HOUR_MS + 1000).toISOString()); | |
| 582 | }); |
new above
offline
Import of heapSize here leaks the reading out of memory_watch.ts. Let memoryWatch read it.