Public: anyone can read this; the crew can change it.

Since Bun 1.4.4 the serving process holds 4.5 to 7 GiB after four hours (was 1.8 to 2.4) and stalls three times as often as it ages #12902

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

torque/forge:torque/12901-since-bun-1-4-4-the-serving-process-hold → main 5 commits fork 4 threads · 4 unresolved

  • Merged. torque/12901-since-bun-1-4-4-the-serving-process-hold was merged into main as 9187455.

Review

Sign in to review this change.
15 files changed +311 −3
docs/features/README.md +1
@@ -417,6 +417,7 @@ Read with the guide page [When something is wrong](../guide-when-wrong.md).
417417 - [Main-thread stalls by route](stalls-by-route.md)
418418 - [Maintenance, last 7 days](maintenance-week.md)
419419 - [Memory held, not hit](memory-held.md)
420 - [Memory by owner and growth across a stall](memory-growth.md)
420421 - [Offline page](offline.md)
421422 - [Pages over budget (real visitors)](slow-pages.md)
422423 - [Pausing a webhook](webhook-pause.md)
docs/features/memory-growth.md added +27
@@ -0,0 +1,27 @@
1 # Memory by owner, the heap across a stall, memory growth (#12901)
2
3 Since Bun 1.4.4 the serving process has held several GiB after hours. This change makes the growth readable; it does not stop it (see "Not done").
4
5 ## Where it shows
6
7 - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. A worker VM writes `memory worker engine: heap …, malloc …, process rss …` on its own minute. RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log.
8 - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample.
9 - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured.
10 - `did not fall: no collection seen` means the stall was not the collector's.
11 - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure and when it began growing.
12
13 ## What to do when the need shows
14
15 1. Read the `memory main` and `memory worker` lines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker.
16 2. Check the stall lines for `fell: a collection ran`: stalls that are collections point at the heap.
17 3. Restart the forge to release it, and report the lines with the issue.
18
19 The need clears by itself once RSS falls under twice the baseline. A process that slept through its first hour has no baseline and raises none.
20
21 ## How it is built
22
23 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Tests: `test/memory-watch.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
24
25 ## Not done
26
27 Finding what grows and fixing it, and wiring the other workers to write their own line.
src/actions/health.ts +1
@@ -81,6 +81,7 @@ function gatherVitals(app: App): Vitals {
8181 dataDir: app.config.dataDir,
8282 ...(app.config.disk ? { disk: app.config.disk } : {}),
8383 ...(app.config.hostFree ? { hostFree: app.config.hostFree } : {}),
84 ...(app.config.memoryWatch ? { watch: app.config.memoryWatch } : {}),
8485 envs,
8586 signals: app.signals,
8687 forgeUrl: app.config.url ?? `http://localhost:${app.config.port}`,
src/app.ts +2
@@ -53,6 +53,8 @@ export interface ForgeConfig {
5353 * when unset. Tests give it a reading.
5454 */
5555 hostFree?: () => number;
56 /** The memory watch vitals reads for growth; the process's own when unset. Tests give it one. */
57 memoryWatch?: { grown(): { baselineBytes: number; rssBytes: number; since: number } | undefined };
5658 /**
5759 * The machine's memory in bytes, which the forges' sizes are summed against
5860 * (`spawn/memory_fit.ts`); its pod's limit or its own when unset. Tests give it a figure.
src/boards/memory.ts +7 −2
@@ -5,15 +5,17 @@
55 * directory), as a count, a sum and a max. RSS rising while the
66 * heap stays flat is native memory growth. A week of hours is kept.
77 *
8 * Each tick is also one log line (`perf/memory_watch.ts`, #12901).
9 *
810 * `heapSize()`, never `heapStats().heapSize`: the same byte count, but
911 * `heapStats()` first counts every object on the heap by type, which held the
1012 * loop ~280 ms a minute on a 2M-object heap and grows with it (#11118).
1113 */
1214 import type { Database } from "bun:sqlite";
13 import { heapSize } from "bun:jsc";
1415 import type { App } from "../app.ts";
1516 import { quietly } from "../core/write_clock.ts";
1617 import { during } from "../perf/jobs.ts";
18 import { memoryLine, memoryWatch, readNow } from "../perf/memory_watch.ts";
1719
1820 export const MEMORY_EVERY_MS = 60_000;
1921 export const HOUR_MS = 3_600_000;
@@ -68,7 +70,10 @@ export type MemorySampler = { tick(now?: number): void; stop(): void };
6870 export function startMemorySampler(app: App, everyMs = MEMORY_EVERY_MS): MemorySampler {
6971 const tick = (now = Date.now()) => {
7072 try {
71 recordMemory(app.db, process.memoryUsage().rss, heapSize(), now);
73 const reading = readNow();
74 recordMemory(app.db, reading.rss, reading.heap, now);
75 memoryWatch.note(reading.rss, reading.heap, now);
76 app.log.info(memoryLine(reading, process.uptime() * 1000));
7277 } catch (error) {
7378 app.log.warn("memory sample failed", { error: String(error) });
7479 }
src/engine/crossing.ts +2
@@ -63,6 +63,8 @@ export const STAYS = {
6363 "a function: a test's stand-in for the free-memory probe; the engine's thread reads the real figure",
6464 memory:
6565 "a function: a test's stand-in for the machine's memory; the engine's thread reads the real figure",
66 memoryWatch:
67 "an object with a method: a test's stand-in for the serving VM's memory watch; a worker writes its own minute line",
6668 decider:
6769 "an object with a method; no duty of the engine asks it, and one that comes to must make its own on that thread",
6870 mailer:
src/engine/worker.ts +4
@@ -14,6 +14,7 @@ import { startDuties } from "./duties.ts";
1414 import { timeStatements } from "../kernel/slow_sql.ts";
1515 import { createLog } from "../kernel/log.ts";
1616 import { during } from "../perf/jobs.ts";
17 import { startWorkerMemory } from "../perf/memory_watch.ts";
1718 import { BEAT_MS, FAILED } from "./beat.ts";
1819
1920 declare const self: Worker;
@@ -65,9 +66,12 @@ async function boot(init: EngineInit): Promise<void> {
6566 };
6667 // Alive: a thread stuck in one statement stops saying so, and the serving thread ends it (`host.ts`).
6768 const beat = setInterval(() => during("engine-beat", () => postMessage("h"), undefined, app), BEAT_MS);
69 const memoryLog = createLog("memory");
70 const stopMemory = startWorkerMemory("engine", (line) => memoryLog.info(line));
6871 const was = stop;
6972 stop = () => {
7073 clearInterval(beat);
74 stopMemory();
7175 was?.();
7276 };
7377 postMessage("ready");
src/kernel/vitals.ts +8
@@ -11,6 +11,7 @@
1111 import type { Database } from "bun:sqlite";
1212 import { statSync } from "node:fs";
1313 import { fullSentence, fullSince } from "./full_disk.ts";
14 import { grownSentence, memoryWatch, type MemoryWatch } from "../perf/memory_watch.ts";
1415 import { readDisk, type DiskReading } from "./disk_reading.ts";
1516 import { DISK_FLOOR, roomy } from "../spawn/disk_floor.ts";
1617 import { dirname } from "node:path";
@@ -185,6 +186,8 @@ export interface VitalsInput {
185186 readonly forgeUrl?: string;
186187 /** Reads the disk under `dataDir`; `statDisk` unless a test gives it a reading. */
187188 readonly disk?: (dir: string) => DiskReading;
189 /** The memory watch whose growth is a need; the process's own unless a test gives one. */
190 readonly watch?: Pick<MemoryWatch, "grown">;
188191 /** The free-disk share below which the disk is a need; `DISK_FREE_MIN` unless a test says. */
189192 readonly diskFreeMin?: number;
190193 /** The pod's memory limit and usage; read from the cgroup unless a test gives them. */
@@ -445,6 +448,11 @@ export function vitals(input: VitalsInput): Vitals {
445448 if (hostProbe !== undefined && memoryFloorMiB > 0 && hostFreeBytes < memoryFloorMiB * MIB)
446449 need("broken", "memory", memorySentence(hostFreeBytes, memoryFloorMiB), undefined, "/settings/capacity");
447450
451 // Twice its first hour (#12901, #12455): the figure and when it began growing.
452 const grown = (input.watch ?? memoryWatch).grown();
453 if (grown !== undefined)
454 need("todo", "memory-growth", grownSentence(grown), new Date(grown.since).toISOString(), "/system");
455
448456 const fullAt = fullSince(input.dataDir);
449457 if (fullAt !== undefined) need("broken", "disk-full", fullSentence(input.dataDir, true), fullAt, "/system");
450458 const disk = (input.disk ?? readDisk)(input.dataDir);
src/main.ts +2 −1
@@ -135,6 +135,7 @@ import { createStallRelay } from "./kernel/stall_relay.ts";
135135 import { startPerfWatcher, TICK_MS } from "./perf/duty.ts";
136136 import { during } from "./perf/jobs.ts";
137137 import { stallLine } from "./perf/loop.ts";
138 import { memoryWatch } from "./perf/memory_watch.ts";
138139 import { bootStall, gitRecord, liveOf } from "./perf/live.ts";
outdated

Import of heapSize here leaks the reading out of memory_watch.ts. Let memoryWatch read it.

139140 import { keepFlowReadings, KEPT_MS } from "./flow/kept.ts";
140141 import { startKeeping } from "./flow/keep.ts";
@@ -278,7 +279,7 @@ const stallRelay = createStallRelay((live: ReturnType<typeof liveOf>, late, at)
278279 live.lagMinutes.note(late, at);
279280 const stall = live.loopWatch.stalled(late, at);
280281 if (stall === null) return;
281 loopLog.warn(stallLine(stall));
282 loopLog.warn(`${stallLine(stall)}; ${memoryWatch.heapAcross()}`);
282283 live.stallBlame.note(stall);
283284 });
284285 watchStalls(loopLog, undefined, undefined, undefined, (late, at) => {
src/perf/memory_watch.ts added +121
@@ -0,0 +1,121 @@
1 /**
2 * What holds the forge's memory, by owner (#12901). Once a minute the serving VM's
3 * reading (`bun:jsc` `heapSize()` and `memoryUsage()`) and the process's RSS are one log line
4 * (`memoryLine`), and a worker VM writes its own (`workerMemoryLine`), so growth reads
outdated

Stale claim: no code reads heapStats().extraMemorySize or objectCount. Remove it from the header.

5 * in the log by its owner. The watch also keeps the first-hour RSS figure and says when
6 * RSS passes twice it (`grown`), which the needs-you line reads (`kernel/vitals.ts`).
7 *
8 * `heapStats()` counts every object on the heap by type and held the loop ~280 ms a
9 * minute on a 2M-object heap (#11118), so the minute line never calls it: `heapSize()`
10 * and `memoryUsage()` are constant-time. Bun 1.4.4 exposes no collector time (a
11 * "gc" PerformanceObserver gets no entries), so a stall is attributed to the collector
12 * by what the heap did across it (`heapAcross`): a heap that fell is a collection that
13 * ran inside the stall.
14 */
15 import { heapSize, memoryUsage as jscMemory } from "bun:jsc";
16
17 const MIB = 1048576;
18 export const FIRST_HOUR_MS = 3_600_000;
19 /** Samples the first hour must hold before its mean is a figure (an idle boot is not one). */
20 export const MIN_SAMPLES = 10;
21 /** The ratio at which the needs-you line speaks (#12455). */
22 export const GROWN_RATIO = 2;
23 /** RSS above this share of the baseline counts as growing, for the time it started. */
24 const GROWING_RATIO = 1.25;
25
26 const mib = (bytes: number): string => `${(bytes / MIB).toFixed(bytes < 100 * MIB ? 1 : 0)} MiB`;
27
28 export interface Reading {
29 readonly rss: number;
30 readonly heap: number;
31 /** `jsc.memoryUsage()`: the allocator's current and peak bytes, and peak committed. */
32 readonly current: number;
33 readonly peak: number;
34 readonly peakCommit: number;
35 }
36
37 /** One reading of this VM: constant time, never `heapStats()`. */
38 export function readNow(rss: number = process.memoryUsage().rss): Reading {
39 const m = jscMemory();
40 return { rss, heap: heapSize(), current: m.current, peak: m.peak, peakCommit: m.peakCommit };
41 }
42
43 /** The serving VM's minute line: `memory main: rss 5.1 GiB, heap …`. */
44 export function memoryLine(r: Reading, uptimeMs: number): string {
45 return `memory main: rss ${mib(r.rss)}, heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}, committed peak ${mib(r.peakCommit)}), up ${Math.round(uptimeMs / 60000)} min`;
46 }
47
48 /** A worker VM's line: its heap and malloc are its own; RSS is the process's. */
49 export function workerMemoryLine(name: string, r: Reading): string {
50 return `memory worker ${name}: heap ${mib(r.heap)}, malloc ${mib(r.current)} (peak ${mib(r.peak)}), process rss ${mib(r.rss)}`;
51 }
52
53 export interface Grown {
54 readonly baselineBytes: number;
55 readonly rssBytes: number;
56 /** Epoch ms of the first sample of the run above 1.25x the baseline. */
57 readonly since: number;
58 }
59
60 export interface MemoryWatch {
61 note(rss: number, heap: number, now?: number): void;
62 /** RSS at least twice its first-hour mean, or undefined. */
63 grown(): Grown | undefined;
64 /**
65 * The heap now (read here, or `heapNow` in a test) against the last minute's sample, for a
66 * stall line.
67 */
68 heapAcross(now?: number, heapNow?: number): string;
69 }
70
71 export function createMemoryWatch(startedAt: number = Date.now()): MemoryWatch {
72 let sum = 0;
73 let count = 0;
74 let base: number | undefined;
75 let last: { heap: number; rss: number; at: number } | undefined;
76 let above: number | undefined;
77 let latest: number | undefined;
78 return {
79 note(rss, heap, now = Date.now()) {
80 if (!Number.isFinite(rss) || rss <= 0) return;
81 last = { heap, rss, at: now };
82 latest = rss;

baseline() is used only by tests. Remove it or give it a caller.

83 if (base === undefined) {
84 if (now - startedAt < FIRST_HOUR_MS) {
85 sum += rss;
86 count++;
87 return;
88 }
89 if (count < MIN_SAMPLES) return; // the process slept through its first hour: no figure
90 base = sum / count;
91 }
92 if (rss > base * GROWING_RATIO) above ??= now;
93 else above = undefined;
94 },
95 grown() {
96 if (base === undefined || latest === undefined || above === undefined || latest < base * GROWN_RATIO) return undefined;
97 return { baselineBytes: base, rssBytes: latest, since: above };
98 },
99 heapAcross(now = Date.now(), heapNow = heapSize()) {
100 if (last === undefined) return `heap ${mib(heapNow)}`;
101 const was = `${mib(last.heap)} ${Math.max(0, Math.round((now - last.at) / 1000))} s ago`;
102 const verdict = heapNow < last.heap * 0.9 ? "fell: a collection ran" : "did not fall: no collection seen";
103 return `heap ${mib(heapNow)} (was ${was}; ${verdict})`;
104 },
105 };
106 }
107
108 /** This process's watch, started with it. */
109 export const memoryWatch: MemoryWatch = createMemoryWatch();
110
111 /** The needs-you sentence for a grown process. */
112 export function grownSentence(g: Grown): string {
113 return `The forge's own memory is ${mib(g.rssBytes)}, over twice its first-hour ${mib(g.baselineBytes)}; it began growing at ${new Date(g.since).toISOString()}. The log's "memory main" and "memory worker" lines say which VM holds it.`;
114 }
115
116 /** Start a worker's own minute line (an unref'd timer inside the worker). Returns the stop. */
117 export function startWorkerMemory(name: string, write: (line: string) => void, everyMs = 60_000): () => void {
118 const timer = setInterval(() => write(workerMemoryLine(name, readNow())), everyMs);
119 (timer as { unref?: () => void }).unref?.();
120 return () => clearInterval(timer);
121 }
test/engine-crossing.test.ts +2
@@ -177,6 +177,7 @@ describe("every setting is on one list, and what crosses is data", () => {
177177 "hostFree",
178178 "mailer",
179179 "memory",
180 "memoryWatch",
180181 "verifyHuman",
181182 ]);
182183 expect(Object.keys(STAYS).sort()).toEqual(
@@ -203,6 +204,7 @@ const whole = (dataDir: string): Required<ForgeConfig> => ({
203204 seedSignup: true,
204205 disk: () => ({ freeBytes: 1, totalBytes: 2 }),
205206 hostFree: () => 1,
207 memoryWatch: { grown: () => undefined },
206208 memory: () => 1,
207209 codeBudgetMs: 30,
208210 port: 8080,
test/memory-watch.test.ts added +65
@@ -0,0 +1,65 @@
1 import { expect, test } from "bun:test";
2 import { createMemoryWatch, grownSentence, memoryLine, workerMemoryLine, FIRST_HOUR_MS, readNow } from "../src/perf/memory_watch.ts";
3
4 const MIB = 1048576;
5 const T0 = 1_000_000_000_000;
6
7 function firstHour(w: ReturnType<typeof createMemoryWatch>, rss: number) {
8 for (let i = 0; i < 12; i++) w.note(rss, rss / 2, T0 + i * 300_000);
9 }
10
11 test("no figure, and no growth, before the first hour has passed", () => {
12 const w = createMemoryWatch(T0);
13 w.note(2000 * MIB, 1, T0 + 1000);
14 w.note(9000 * MIB, 1, T0 + 2000);
15 expect(w.grown()).toBeUndefined();
16 });
17
18 test("raises once RSS passes twice the first-hour mean, with the figure and when it began", () => {
19 const w = createMemoryWatch(T0);
20 firstHour(w, 2000 * MIB); // mean 2000 MiB
21 w.note(2200 * MIB, 1, T0 + FIRST_HOUR_MS + 60_000);
22 expect(w.grown()).toBeUndefined(); // 1.1x
23 const began = T0 + FIRST_HOUR_MS + 120_000;
24 w.note(2600 * MIB, 1, began); // 1.3x: growing, not yet 2x
25 w.note(3900 * MIB, 1, began + 60_000); // 1.95x
26 expect(w.grown()).toBeUndefined();
27 w.note(4100 * MIB, 1, began + 120_000); // 2.05x
28 expect(w.grown()).toEqual({ baselineBytes: 2000 * MIB, rssBytes: 4100 * MIB, since: began });
29 expect(grownSentence(w.grown()!)).toContain("4100 MiB, over twice its first-hour 2000 MiB");
30 expect(grownSentence(w.grown()!)).toContain(new Date(began).toISOString());
31 });
32
33 test("a drop back under 1.25x forgets when it began; an idle first hour gives no figure", () => {
34 const w = createMemoryWatch(T0);
35 firstHour(w, 1000 * MIB);
36 const t = T0 + FIRST_HOUR_MS;
37 w.note(1500 * MIB, 1, t + 1000);
38 w.note(1000 * MIB, 1, t + 2000);
39 w.note(2500 * MIB, 1, t + 3000);
40 expect(w.grown()?.since).toBe(t + 3000);
41 const idle = createMemoryWatch(T0);
42 idle.note(1000 * MIB, 1, T0 + 1000);
43 idle.note(9000 * MIB, 1, T0 + FIRST_HOUR_MS + 1000);
44 expect(idle.grown()).toBeUndefined();
45 });
46
47 test("a stall's heap is read against the last minute: a fall is a collection", () => {
48 const w = createMemoryWatch(T0);
49 expect(w.heapAcross(T0, 500 * MIB)).toBe("heap 500 MiB");
50 w.note(3000 * MIB, 1000 * MIB, T0);
51 expect(w.heapAcross(T0 + 41_000, 400 * MIB)).toBe("heap 400 MiB (was 1000 MiB 41 s ago; fell: a collection ran)");
52 expect(w.heapAcross(T0 + 5000, 1000 * MIB)).toContain("did not fall");
53 });
54
55 test("a stall's heap with no argument is read from the VM", () => {
56 const w = createMemoryWatch(T0);
57 expect(w.heapAcross()).toMatch(/^heap \d/);
58 });
59
60 test("the minute lines name the owner and the figures", () => {
61 const r = { rss: 5 * 1024 * MIB, heap: 1024 * MIB, current: 512 * MIB, peak: 600 * MIB, peakCommit: 700 * MIB };
62 expect(memoryLine(r, 3_600_000)).toBe("memory main: rss 5120 MiB, heap 1024 MiB, malloc 512 MiB (peak 600 MiB, committed peak 700 MiB), up 60 min");
63 expect(workerMemoryLine("engine", r)).toStartWith("memory worker engine: heap 1024 MiB");
64 expect(readNow().heap).toBeGreaterThan(0);
65 });
test/timers-named.test.ts +1
@@ -81,6 +81,7 @@ const ALLOWED: readonly Allowed[] = [
8181 ...allow("setTimeout", "an app request's look-again sleep in the ladder: it holds one request, does no work on the loop, and ends within the bound", [["src/http/app_turn.ts", "r"]]),
8282 ...allow("setTimeout", `a media stream's idle deadline, ${PER_CONNECTION}`, [["src/git/read.ts", "() => { finish(); controller.error("]]),
8383 ...allow("setTimeout", `a git upload's idle deadline, ${PER_CONNECTION}`, [["src/git/smart-http.ts", "stop"]]),
84 ...allow("setInterval", "a worker VM's minute memory line: it runs in the worker, where the main process's span ring is not reachable, and reads two counters", [["src/perf/memory_watch.ts", "() => write(workerMemoryLine("]]),
8485 ...allow("setTimeout", "the loop-lag meter's own tick: it measures how late the loop runs, and a span around it would count that lateness as its own work", [["src/kernel/loop_lag.ts", "() => { // Early"]]),
8586 ...allow("setTimeout", "the next slice of a sliced flow computation: one span per slice would fill the 256-span ring in one refresh", [["src/system/flow_cache.ts", "fn"]]),
8687 ...allow("setTimeout", "the release prover's scheduler seam: its caller passes a callback already inside during(\"release-prove\")", [["src/release/prove.ts", "fn"]]),
test/tui-memory-growth.test.ts added +54
@@ -0,0 +1,54 @@
1 /**
2 * The memory-growth need reaches the terminal's home as it reaches /system: both read
3 * `forgeNeedList` (#12901).
4 */
5 import { afterAll, beforeAll, expect, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import { createApp, type App as Forge } from "../src/app.ts";
10 import { forgeNeedList } from "../src/actions/health.ts";
11 import { createForgeApp } from "../src/tui/forge.ts";
12 import { headlessTerminal } from "../src/tui/term/terminal.ts";
13 import { createMemoryWatch, FIRST_HOUR_MS } from "../src/perf/memory_watch.ts";
14 import { quietLogs } from "./helpers/globals.ts";
15 import { settleApp } from "./helpers/tui.ts";
16 import { seedAdmin } from "./helpers/admin.ts";
17
18 quietLogs();
19 const MIB = 1048576;
20 let dir: string;
21 let forge: Forge;
22
23 beforeAll(async () => {
24 dir = await mkdtemp(join(tmpdir(), "forge-tui-growth-"));
25 const start = Date.now() - 2 * FIRST_HOUR_MS;
26 const w = createMemoryWatch(start);
27 for (let i = 0; i < 12; i++) w.note(1000 * MIB, 1, start + i * 300_000);
28 w.note(2500 * MIB, 1, start + FIRST_HOUR_MS + 1000);
29 forge = createApp({ dataDir: dir, port: 0, memoryWatch: w });
30 });
31
32 afterAll(async () => {
33 forge.close();
34 await rm(dir, { recursive: true, force: true });
35 });
36
37 test("an admin's terminal home lists the memory-growth need beside the same need in the shared list", async () => {
38 const a = await seedAdmin(forge, { username: "alice", email: "[email protected]", password: "a-long-passphrase" });
39 if (!a.ok) throw new Error("signup failed");
40 const needs = forgeNeedList(forge, a.value)!;
41 expect(needs.find((n) => n.cause === "memory-growth")?.href).toBe("/system");
42 const terminal = headlessTerminal(200, 40);
43 const { app } = createForgeApp(
44 forge,
45 { sshUsername: "alice", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user: a.value },
46 { terminal, exitHooks: false },
47 );
48 app.start();
49 await settleApp(app);
50 const text = app.screen.toText();
51 app.stop();
52 expect(text).toContain("2500 MiB");
53 expect(text).toContain("twice its first-hour");
54 });
test/vitals.test.ts +14
@@ -566,3 +566,17 @@ test("vitals memory follows the pod's injected limit and usage (#12347)", async
566566 await rm(dir, { recursive: true, force: true });
567567 }
568568 });
569

This monkey-patches the process-wide memoryWatch and asserts on its pristine state. Inject the watch into vitals() instead, so the test exercises the real path and cannot depend on test order.

570 test("memory twice its first hour is one todo need with the figure and when it began (#12901)", async () => {
571 const { FIRST_HOUR_MS, createMemoryWatch } = await import("../src/perf/memory_watch.ts");
572 const MIB = 1048576;
573 const start = Date.now() - 2 * FIRST_HOUR_MS;
574 const w = createMemoryWatch(start);
575 for (let i = 0; i < 12; i++) w.note(1000 * MIB, 1, start + i * 300_000);
576 expect(vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w }).degraded.some((x) => x.cause === "memory-growth")).toBe(false);
577 w.note(2500 * MIB, 1, start + FIRST_HOUR_MS + 1000);
578 const v = vitals({ db: app.db, signals: app.signals, dataDir, disk: roomy, watch: w });
579 const d = v.degraded.find((x) => x.cause === "memory-growth")!;
580 expect(d.sentence).toContain("2500 MiB, over twice its first-hour 1000 MiB");
581 expect(d.since).toBe(new Date(start + FIRST_HOUR_MS + 1000).toISOString());
582 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs