Public: anyone can read this; the crew can change it.

Every worker VM writes its own `memory worker <name>` line, not only the engine worker #12938

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

mason/forge:mason/12925-every-worker-vm-writes-its-own-memory-wo → main 4 commits fork 2 threads · 2 unresolved

  • Merged. mason/12925-every-worker-vm-writes-its-own-memory-wo was merged into main as 0b4b4ed.

Review

Sign in to review this change.
20 files changed +153 −11
docs/features/memory-growth.md +4 −3
@@ -4,11 +4,12 @@ Since Bun 1.4.4 the serving process has held several GiB after hours. This chang
44
55 ## Where it shows
66
7 - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. A worker VM writes `memory worker engine: heap …, malloc …, process rss …` on its own minute. RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log.
7 - **The minute line.** Each minute the log carries `memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min`. Every worker VM writes `memory worker <name>: heap …, malloc …, process rss …` on its own minute (#12925). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in `/system`'s log or the process log.
88 - **The stall line.** Each loop-stall line ends with `heap X (was Y N s ago; …)`, the heap now against the last minute's sample.
99 - `fell: a collection ran` means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured.
1010 - `did not fall: no collection seen` means the stall was not the collector's.
1111 - **The need.** When RSS is at least twice the mean of the process's first hour (at least ten samples), `/system`, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure and when it began growing.
12 - **The worker names.** `engine` (src/engine/worker.ts); `disk`, `tree` (src/apps); `lanes`, `pages`, `pipeline` (src/flow); `scan`, `du`, `leak` (the `answerAsks` workers of src/kernel/worker_ask.ts); `thumb` (src/looks); `log-writer` (src/runs); `mirror`, `vacuum` (src/snapshot); `code-rows`, `map`, `raster` (src/web). A one-shot worker (`mirror`, `vacuum`) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive.
1213
1314 ## What to do when the need shows
1415
@@ -20,8 +21,8 @@ The need clears by itself once RSS falls under twice the baseline. A process tha
2021
2122 ## How it is built
2223
23 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Tests: `test/memory-watch.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
24 `src/perf/memory_watch.ts` owns the readings (`heapSize()` and `memoryUsage()`, constant time; never `heapStats()`, which held the loop ~280 ms a minute, #11118). `vitals()` takes the watch as `watch`, as it takes `disk`. Each worker entry calls `workerMemory(<name>)` once at start; `FORGE_WORKER_MEMORY_MS` shortens the minute for tests. Tests: `test/memory-watch.test.ts`, `test/worker-memory-lines.test.ts`, `test/vitals.test.ts`, `test/tui-memory-growth.test.ts`.
2425
2526 ## Not done
2627
27 Finding what grows and fixing it, and wiring the other workers to write their own line.
28 Finding what grows and fixing it.
src/apps/disk_worker.ts +3
@@ -1,9 +1,12 @@
11 /** The worker side of `diskWalkOff` (`contain.ts`): walk the directories and say what they hold. */
22
33 import { diskWalk } from "./disk_walk.ts";
4 import { workerMemory } from "../perf/memory_watch.ts";
45
56 declare const self: Worker;
67
8 workerMemory("disk");
9
710 self.onmessage = (e: MessageEvent<{ id: number; dirs: string[]; max: number }>) => {
811 try {
912 postMessage({ id: e.data.id, walk: diskWalk(e.data.dirs, e.data.max) });
src/apps/tree_worker.ts +3
@@ -1,9 +1,12 @@
11 /** The worker side of `readTreesOff` (`contain.ts`): read every watched tree, as plain data. */
22
33 import { readTrees, type TreeAsk } from "./tree_read.ts";
4 import { workerMemory } from "../perf/memory_watch.ts";
45
56 declare const self: Worker;
67
8 workerMemory("tree");
9
710 self.onmessage = (e: MessageEvent<{ id: number; asks: TreeAsk[] }>) => {
811 try {
912 postMessage({ id: e.data.id, trees: readTrees(e.data.asks) });
src/checks/scan_worker.ts +1 −1
@@ -3,4 +3,4 @@
33 import { answerAsks } from "../kernel/worker_ask.ts";
44 import { findingsIn, type Finding } from "./scan_rules.ts";
55
6 answerAsks<{ files: [string, string][]; withStatic: boolean }, Finding[]>((ask) => findingsIn(new Map(ask.files), ask.withStatic));
6 answerAsks<{ files: [string, string][]; withStatic: boolean }, Finding[]>("scan", (ask) => findingsIn(new Map(ask.files), ask.withStatic));
src/crew/leak_worker.ts +1 −1
@@ -3,4 +3,4 @@
33 import { answerAsks } from "../kernel/worker_ask.ts";
44 import { type ScrubAsk, scrubFiles } from "./leak_files.ts";
55
6 answerAsks<ScrubAsk, string[]>(scrubFiles);
6 answerAsks<ScrubAsk, string[]>("leak", scrubFiles);
src/engine/worker.ts +2 −3
@@ -14,7 +14,7 @@ import { startDuties } from "./duties.ts";
1414 import { timeStatements } from "../kernel/slow_sql.ts";
1515 import { createLog } from "../kernel/log.ts";
1616 import { during } from "../perf/jobs.ts";
17 import { startWorkerMemory } from "../perf/memory_watch.ts";
17 import { workerMemory } from "../perf/memory_watch.ts";
1818 import { BEAT_MS, FAILED } from "./beat.ts";
1919
2020 declare const self: Worker;
@@ -66,8 +66,7 @@ async function boot(init: EngineInit): Promise<void> {
6666 };
6767 // Alive: a thread stuck in one statement stops saying so, and the serving thread ends it (`host.ts`).
6868 const beat = setInterval(() => during("engine-beat", () => postMessage("h"), undefined, app), BEAT_MS);
69 const memoryLog = createLog("memory");
70 const stopMemory = startWorkerMemory("engine", (line) => memoryLog.info(line));
69 const stopMemory = workerMemory("engine");
7170 const was = stop;
7271 stop = () => {
7372 clearInterval(beat);
src/flow/lanes_worker.ts +3
@@ -11,6 +11,7 @@
1111
1212 import { Database } from "bun:sqlite";
1313 import { compose, readFast, readSlow, SLOW_MS, type LanePicture, type Slow } from "./lanes_read.ts";
14 import { workerMemory } from "../perf/memory_watch.ts";
1415
1516 /** What the worker is asked. */
1617 export interface LaneJob {
@@ -37,6 +38,8 @@ export interface LaneAnswer {
3738
3839 declare const self: Worker;
3940
41 workerMemory("lanes");
42
4043 const handles = new Map<string, Database>();
4144 const kept = new Map<string, { at: number; slow: Slow }>();
4245
src/flow/pages_worker.ts +3
@@ -8,6 +8,7 @@
88
99 import { Database } from "bun:sqlite";
1010 import { readPagePicture, type PageJob, type PagesConfig, type PagesEnv, type Picture } from "./pages_read.ts";
11 import { workerMemory } from "../perf/memory_watch.ts";
1112
1213 /** What the worker is asked. */
1314 export interface PagesAsk {
@@ -29,6 +30,8 @@ export interface PagesAnswer {
2930
3031 declare const self: Worker;
3132
33 workerMemory("pages");
34
3235 const envs = new Map<string, PagesEnv>();
3336
3437 self.onmessage = (e: MessageEvent<PagesAsk>) => {
src/flow/pipeline_worker.ts +3
@@ -6,6 +6,7 @@
66 */
77
88 import { Database } from "bun:sqlite";
9 import { workerMemory } from "../perf/memory_watch.ts";
910
1011 /** What the worker is asked: the database file, the stage's query and its arguments. */
1112 export interface WaitingJob {
@@ -25,6 +26,8 @@ export interface WaitingAnswer {
2526
2627 declare const self: Worker;
2728
29 workerMemory("pipeline");
30
2831 const handles = new Map<string, Database>();
2932
3033 self.onmessage = (e: MessageEvent<WaitingJob>) => {
src/kernel/worker_ask.ts +8 −2
@@ -9,6 +9,8 @@
99 * to decide: most do the work on their own thread, so nothing is lost.
1010 */
1111
12 import { workerMemory } from "../perf/memory_watch.ts";
13
1214 interface Asked<Ask> {
1315 readonly id: number;
1416 readonly ask: Ask;
@@ -58,8 +60,12 @@ export function workerAsker<Ask, Answer>(url: string): (ask: Ask) => Promise<Ans
5860
5961 declare const self: Worker;
6062
61 /** The worker's side: answer each ask with `work`'s result, or with why it threw. */
62 export function answerAsks<Ask, Answer>(work: (ask: Ask) => Answer): void {
63 /**
64 * The worker's side: write its `memory worker <name>` line, and answer each ask with
65 * `work`'s result, or with why it threw.
66 */
67 export function answerAsks<Ask, Answer>(name: string, work: (ask: Ask) => Answer): void {
68 workerMemory(name);
6369 self.onmessage = (e: MessageEvent<Asked<Ask>>) => {
6470 const { id, ask } = e.data;
6571 try {
src/looks/thumb_worker.ts +3
@@ -4,9 +4,12 @@
44 */
55
66 import { reduceHere, type ThumbAnswer, type ThumbAsk } from "./thumb.ts";
7 import { workerMemory } from "../perf/memory_watch.ts";
78
89 declare const self: Worker;
910
11 workerMemory("thumb");
12
1013 self.onmessage = (e: MessageEvent<ThumbAsk>) => {
1114 const { id, path, size } = e.data;
1215 try {
src/perf/memory_watch.ts +21
@@ -13,6 +13,7 @@
1313 * ran inside the stall.
1414 */
1515 import { heapSize, memoryUsage as jscMemory } from "bun:jsc";
16 import { createLog } from "../kernel/log.ts";
1617
1718 const MIB = 1048576;
1819 export const FIRST_HOUR_MS = 3_600_000;
@@ -119,3 +120,23 @@ export function startWorkerMemory(name: string, write: (line: string) => void, e
119120 (timer as { unref?: () => void }).unref?.();
120121 return () => clearInterval(timer);
121122 }
123
124 /**
125 * How often a worker writes its line: a minute, or `FORGE_WORKER_MEMORY_MS` (a test seam,
126 * a positive integer).
127 */
128 export function workerMemoryEvery(env: Record<string, string | undefined> = process.env): number {
129 const raw = env.FORGE_WORKER_MEMORY_MS;
130 const n = raw !== undefined && /^\d{1,9}$/.test(raw) ? Number(raw) : 0;
131 return n > 0 ? n : 60_000;
132 }
133
134 /**
135 * The call each worker entry makes once at start: its `memory worker <name>` line on the
136 * `memory` log, each minute. The timer is unref'd, so a one-shot worker is never kept
137 * alive by it, and one that ends inside its first minute writes no line.
138 */
139 export function workerMemory(name: string): () => void {
140 const log = createLog("memory");
141 return startWorkerMemory(name, (line) => log.info(line), workerMemoryEvery());
142 }
src/runs/log_writer.ts +3
@@ -5,6 +5,7 @@
55 */
66
77 import { appendTo } from "./log_append.ts";
8 import { workerMemory } from "../perf/memory_watch.ts";
89
910 /** What the writer is given: a batch number, the shared counter, and `[runs root, file, text]` per file. */
1011 export interface LogBatch {
@@ -15,6 +16,8 @@ export interface LogBatch {
1516
1617 declare const self: Worker;
1718
19 workerMemory("log-writer");
20
1821 self.onmessage = (e: MessageEvent<LogBatch>) => {
1922 for (const [root, path, text] of e.data.items) {
2023 try {
src/snapshot/mirror_worker.ts +3
@@ -14,6 +14,7 @@
1414 import { mirrorPass, type MirrorState } from "./mirror.ts";
1515 import { storeFrom } from "./remake.ts";
1616 import type { StoreSpec } from "./store.ts";
17 import { workerMemory } from "../perf/memory_watch.ts";
1718
1819 export interface MirrorJob {
1920 readonly dataDir: string;
@@ -25,6 +26,8 @@ export type MirrorAnswer = { readonly state: MirrorState } | { readonly error: s
2526
2627 declare const self: Worker;
2728
29 workerMemory("mirror");
30
2831 self.onmessage = async (event: MessageEvent<MirrorJob>): Promise<void> => {
2932 const job = event.data;
3033 try {
src/snapshot/vacuum_worker.ts +3
@@ -1,6 +1,7 @@
11 /** The worker side of `vacuum.ts`: copy `from` into `to` and say so. */
22
33 import { Database } from "bun:sqlite";
4 import { workerMemory } from "../perf/memory_watch.ts";
45
56 /** What the worker is asked: the file to read and the copy to make. */
67 export interface VacuumJob {
@@ -10,6 +11,8 @@ export interface VacuumJob {
1011
1112 declare const self: Worker;
1213
14 workerMemory("vacuum");
15
1316 self.onmessage = (e: MessageEvent<VacuumJob>) => {
1417 try {
1518 const db = new Database(e.data.from, { readonly: true });
src/system/du_worker.ts +1 −1
@@ -3,4 +3,4 @@
33 import { answerAsks } from "../kernel/worker_ask.ts";
44 import { dirSizes, type DirSize, type DuAsk, treeSize } from "./du.ts";
55
6 answerAsks<DuAsk, DirSize[] | number>((ask) => ("dirs" in ask ? dirSizes(ask.dirs) : treeSize(ask.tree)));
6 answerAsks<DuAsk, DirSize[] | number>("du", (ask) => ("dirs" in ask ? dirSizes(ask.dirs) : treeSize(ask.tree)));
src/web/code_rows_worker.ts +3
@@ -2,9 +2,12 @@
22 import { compressBytes, type Encoding } from "../http/compress.ts";
33 import { parseGcode } from "./render/gcode.ts";
44 import { codeRows } from "./code_rows.ts";
5 import { workerMemory } from "../perf/memory_watch.ts";
56
67 declare const self: Worker;
78
9 workerMemory("code-rows");
10
811 export type Job =
912 | { id: number; kind: "rows"; body: Uint8Array; path: string; limit: number }
1013 | { id: number; kind: "gcode"; body: Uint8Array }
src/web/map_worker.ts +3
@@ -12,6 +12,7 @@ import { Database } from "bun:sqlite";
1212 import type { App } from "../app.ts";
1313 import type { PagesConfig } from "../flow/pages_read.ts";
1414 import { readMap, type Learned, type MapJob, type MapReading } from "./map_read.ts";
15 import { workerMemory } from "../perf/memory_watch.ts";
1516
1617 /** What the worker is asked: plain data (never the app's config, which holds objects with methods). */
1718 export interface MapAsk {
@@ -33,6 +34,8 @@ export interface MapAnswer {
3334
3435 declare const self: Worker;
3536
37 workerMemory("map");
38
3639 /** The app the readers that take one are given: its database, its repositories and its settings, nothing that serves. */
3740 const apps = new Map<string, App>();
3841
src/web/og/raster_worker.ts +3
@@ -4,9 +4,12 @@
44 */
55
66 import { rasterizer, type RasterAnswer, type RasterAsk } from "./raster.ts";
7 import { workerMemory } from "../../perf/memory_watch.ts";
78
89 declare const self: Worker;
910
11 workerMemory("raster");
12
1013 self.onmessage = async (e: MessageEvent<RasterAsk>) => {
1114 const { id, svg } = e.data;
1215 try {
test/worker-memory-lines.test.ts added +79
@@ -0,0 +1,79 @@
1 /**
2 * Every worker VM writes its own `memory worker <name>` line (#12925), not only the
3 * engine's. A child process starts two of the forge's workers with the minute shortened
4 * through `FORGE_WORKER_MEMORY_MS`, and its output is read until each name appears.
5 */
6 import { expect, test } from "bun:test";
7 import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { workerMemoryEvery } from "../src/perf/memory_watch.ts";
11
12 const ROOT = join(import.meta.dir, "..");
13
14 /** Run `script` in a child forge process; stop it once `done(out)` holds, or at the deadline. */
15 async function runChild(
16 script: string,
17 everyMs: string,
18 done: (out: string) => boolean = () => false,
19 ): Promise<{ out: string; code: number; ms: number }> {
20 const t0 = performance.now();
21 const child = Bun.spawn([process.execPath, "-e", script], {
22 cwd: ROOT,
23 env: { ...process.env, FORGE_WORKER_MEMORY_MS: everyMs },
24 stdout: "pipe",
25 stderr: "pipe",
26 });
27 const killer = setTimeout(() => child.kill(), 15_000);
28 let out = "";
29 const read = async (stream: ReadableStream<Uint8Array>) => {
30 const decoder = new TextDecoder();
31 for await (const chunk of stream) {
32 out += decoder.decode(chunk, { stream: true });
outdated

Fixed 1.5 s sleep: poll the output for both names against a deadline so the test is neither slow nor flaky on a loaded machine.

33 if (done(out)) child.kill();
34 }
35 };
36 const [, , code] = await Promise.all([read(child.stdout), read(child.stderr), child.exited]);
37 clearTimeout(killer);
38 return { out, code, ms: performance.now() - t0 };
39 }
40
41 const named = (out: string) =>
42 new Set([...out.matchAll(/\[memory\] memory worker ([\w-]+): heap /g)].map((m) => m[1]));
outdated

This test passes with a ref d timer too: vacuumFile terminates the worker, so exit is fast either way. Make it fail without unref (a worker that ends by itself, so a held timer would keep the process up until the kill), or assert unref directly.

43
44 test("two workers each write their own memory line, named for themselves", async () => {
45 // The child keeps both workers up until the test has read a line from each.
46 const script = `
47 new Worker(${JSON.stringify(join(ROOT, "src/flow/pipeline_worker.ts"))});
48 new Worker(${JSON.stringify(join(ROOT, "src/web/code_rows_worker.ts"))});
49 `;
50 const { out } = await runChild(script, "50", (o) => named(o).size >= 2);
51 expect([...named(out)].sort()).toEqual(["code-rows", "pipeline"]);
52 }, 20_000);
53
54 test("a worker that ends by itself inside its first interval writes no line and is not held open", async () => {
55 // The worker starts its line and then has nothing left to do: with a ref'd timer
56 // its loop would never empty, and the process would stay up until the 15 s kill.
57 const dir = mkdtempSync(join(tmpdir(), "worker-memory-"));
58 try {
59 const entry = join(dir, "idle_worker.ts");
60 writeFileSync(
61 entry,
62 `import { workerMemory } from ${JSON.stringify(join(ROOT, "src/perf/memory_watch.ts"))};\n` +
63 `workerMemory("idle");\n`,
64 );
65 const script = `new Worker(${JSON.stringify(entry)});`;
66 const { out, code, ms } = await runChild(script, "60000");
67 expect(code).toBe(0);
68 expect(out).not.toContain("memory worker");
69 expect(ms).toBeLessThan(10_000);
70 } finally {
71 rmSync(dir, { recursive: true, force: true });
72 }
73 }, 20_000);
74
75 test("the interval is a minute unless the seam names a positive whole number", () => {
76 expect(workerMemoryEvery({})).toBe(60_000);
77 expect(workerMemoryEvery({ FORGE_WORKER_MEMORY_MS: "250" })).toBe(250);
78 for (const bad of ["0", "-5", "1e3", "abc", ""]) expect(workerMemoryEvery({ FORGE_WORKER_MEMORY_MS: bad })).toBe(60_000);
79 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs