Public: anyone can read this; the crew can change it.

When the forge's memory has grown, the needs-you line names which VM grew and whether it is JS heap, malloc or native #12952

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

When memory has grown, the needs-you line now says which VM grew and which part of its memory, for example: "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z." How worker readings reach main.

Notes

shipped production is serving it deploy 5a4f606

opened this

When memory has grown, the needs-you line now says which VM grew and which part of its memory, for example: "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z."

How worker readings reach main. I used the worker's own line, not worker_ask.ts. The minute timer that already writes each worker's memory worker <name> line also sends { name, heap, current } on a forge-memory BroadcastChannel (tellReading). The main VM's memory sampler in src/boards/memory.ts listens with hearWorkers and checks each message in workerReadingOf: a lower-case name that is not main, finite non-negative byte counts, and at most 64 workers kept. This covers every worker, including the ones with their own message formats that answerAsks never sees. Asks carry nothing extra. The channel is unref'd, so a worker that is done is still not held open.

What src/perf/memory_watch.ts keeps. Each owner (main, and each worker by name) has its own first-hour mean for each part:

  • JS heap: heap.
  • malloc: malloc − heap, floored at 0.
  • native memory, main only: RSS minus main's malloc and every reporting worker's malloc, floored at 0. Without taking the workers' malloc out, a growing worker would also show up as main's native growth. This assumes each VM's memoryUsage() covers only that VM, as the worker line already does.

A worker's hour starts at its first reading, so a worker started on its first ask gets its own baseline. grown() still triggers on main RSS at 2x, unchanged, and adds grew: the part that grew the most bytes past its own baseline. A worker silent for five minutes is not blamed, and its malloc stops being subtracted from native. When no part is above its own first hour, the sentence says so. The minute stays constant-time, and there is no heapStats() call.

src/kernel/vitals.ts itself is unchanged: it passes the new sentence through as it is, and a test in test/vitals.test.ts checks that. I kept sizes in MiB, matching the log lines, rather than the issue's GiB example.

Tests.

  • test/memory-watch.test.ts:

- main heap growth is named as main's JS heap; - one worker's malloc growth is named as that worker, not as main's native; - growth under 2x names nothing; - a worker that stopped reporting is not blamed; - a worker started late gets its own hour; - bad messages are refused and the worker cap holds; - a reading crosses the channel.

  • test/worker-memory-lines.test.ts: a real pipeline_worker reaches hearWorkers in a child process.
  • test/vitals.test.ts: the needs-you sentence is exactly grownSentence.

docs/features/memory-growth.md and AGENTS.md (the state kept in process memory) are updated.

Not done: fixing the growth itself. Once this line points at a cause, that is a new issue with the evidence.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff reaches 2120 test files, too many to run here, so I picked 15 files covering the change: the four test files above plus tui-memory-growth, timers-named, engine-crossing, system-memory, system-parts, tui-system, agents-md, feature-orphans, features-empty, no-polling and perf-jobs-per-forge. All 135 tests pass. The full suite runs on the push.

To try it: bun test test/memory-watch.test.ts test/worker-memory-lines.test.ts.

🤖 Generated with Claude Code

Closes #12951

3 earlier comments
commented

When memory has grown, the needs-you line now says which VM grew and which part of its memory, for example: "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z."

How worker readings reach main. I used the worker's own line, not worker_ask.ts. The minute timer that already writes each worker's memory worker <name> line also sends { name, heap, current } on a forge-memory BroadcastChannel (tellReading). The main VM's memory sampler in src/boards/memory.ts listens with hearWorkers and checks each message in workerReadingOf: a lower-case name that is not main, finite non-negative byte counts, and at most 64 workers kept. This covers every worker, including the ones with their own message formats that answerAsks never sees. Asks carry nothing extra. The channel is unref'd, so a worker that is done is still not held open.

What src/perf/memory_watch.ts keeps. Each owner (main, and each worker by name) has its own first-hour mean for each part:

  • JS heap: heap.
  • malloc: malloc − heap, floored at 0.
  • native memory, main only: RSS minus main's malloc and every reporting worker's malloc, floored at 0. Without taking the workers' malloc out, a growing worker would also show up as main's native growth. This assumes each VM's memoryUsage() covers only that VM, as the worker line already does.

A worker's hour starts at its first reading, so a worker started on its first ask gets its own baseline. grown() still triggers on main RSS at 2x, unchanged, and adds grew: the part that grew the most bytes past its own baseline. A worker silent for five minutes is not blamed, and its malloc stops being subtracted from native. When no part is above its own first hour, the sentence says so. The minute stays constant-time, and there is no heapStats() call.

src/kernel/vitals.ts itself is unchanged: it passes the new sentence through as it is, and a test in test/vitals.test.ts checks that. I kept sizes in MiB, matching the log lines, rather than the issue's GiB example.

Tests.

  • test/memory-watch.test.ts:

- main heap growth is named as main's JS heap; - one worker's malloc growth is named as that worker, not as main's native; - growth under 2x names nothing; - a worker that stopped reporting is not blamed; - a worker started late gets its own hour; - bad messages are refused and the worker cap holds; - a reading crosses the channel.

  • test/worker-memory-lines.test.ts: a real pipeline_worker reaches hearWorkers in a child process.
  • test/vitals.test.ts: the needs-you sentence is exactly grownSentence.

docs/features/memory-growth.md and AGENTS.md (the state kept in process memory) are updated.

Not done: fixing the growth itself. Once this line points at a cause, that is a new issue with the evidence.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff reaches 2120 test files, too many to run here, so I picked 15 files covering the change: the four test files above plus tui-memory-growth, timers-named, engine-crossing, system-memory, system-parts, tui-system, agents-md, feature-orphans, features-empty, no-polling and perf-jobs-per-forge. All 135 tests pass. The full suite runs on the push.

To try it: bun test test/memory-watch.test.ts test/worker-memory-lines.test.ts.

🤖 Generated with Claude Code

commented

When memory has grown, the needs-you line now says which VM grew and which part of its memory, for example: "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z." I kept sizes in MiB, the same as the log lines, rather than the GiB in the issue's example.

How worker readings reach main. They go with the worker's own line, not through worker_ask.ts. The timer that already writes each worker's memory worker <name> line also posts { name, heap, current } on a forge-memory BroadcastChannel (tellReading). This covers every worker, including those with their own message shapes that answerAsks never sees, and asks carry nothing extra. The channel is unref'd, so an idle worker can still exit.

The main VM's sampler (src/boards/memory.ts) listens with hearWorkers and checks each message in workerReadingOf. A valid message has a lower-case name that is not main and finite byte counts. At most 64 workers are kept.

What src/perf/memory_watch.ts keeps. Each owner (main, and each worker by name) has its own first-hour average of each part:

  • JS heap: heap.
  • malloc: malloc − heap, floored at 0.
  • native (main only): RSS minus main's malloc and every reporting worker's malloc, floored at 0. Without taking the workers' malloc out, one growing worker would also show up as main's native growth. This assumes each VM's memoryUsage() counts only that VM, as the worker line already does.

A worker's first hour starts at its first reading, so a worker started late still gets a baseline of its own.

grown() still fires when main's RSS reaches twice its first hour, as before. It now adds grew: the part that grew the most bytes past its own baseline. A worker silent for five minutes is not blamed, and its malloc stops being subtracted from native. When no part is above its own first hour, the sentence says so. There is no heapStats() call, so the minute stays constant-time.

src/kernel/vitals.ts is unchanged: it shows the new sentence exactly as grownSentence returns it, and a test checks that.

Not done: fixing the growth itself. Once this line names a cause, that becomes a new issue with the evidence.

Fixes after the first check: it failed on the gate. The channel test now waits for the second message to arrive instead of sleeping. Two comments are rewrapped under 100 columns, and the new commit has a Try: line.

Tests:

  • test/memory-watch.test.ts proves that:

- main heap growth is named as main's JS heap; - one worker's malloc growth is named as that worker, not as main's native; - growth under 2x names nothing; - a worker that stopped reporting is not blamed; - a worker started late gets its own first hour; - bad messages are refused and the 64-worker limit holds; - a reading crosses the channel.

  • test/worker-memory-lines.test.ts: a real pipeline_worker reaches hearWorkers in a child process.
  • test/vitals.test.ts: the needs-you line is exactly grownSentence.

scripts/fast.ts said the diff reaches 2120 test files, too many to run here, so I chose the files covering the change. The gate (with FORGE_CHECK_BASE=main) and bun x tsc --noEmit pass, and 56 tests pass across memory-watch, worker-memory-lines, vitals, tui-memory-growth, timers-named and system-memory. The full suite runs on the push.

To try it: run bun test test/memory-watch.test.ts test/worker-memory-lines.test.ts. On the preview, /system shows the memory-growth need only after RSS has doubled its first hour.

docs/features/memory-growth.md and AGENTS.md (state kept in process memory) are updated.

🤖 Generated with Claude Code

commented

Two separate causes broke the check, and I fixed both.

  • Module state (src/perf/memory_watch.ts): hearWorkers kept its channel for each watch in a module-level Map. That broke the #12569 rule against process-wide state that functions write to. The map is keyed by the watch object and exists only to make the listener once per watch, so it is now a WeakMap, the form the rule asks for. Nothing else changes.
  • Contribute-token test (test/contribute-token.test.ts): this one had nothing to do with the memory change. It fails on main too. The test made its tokens at a fixed 2026-11-14T00:00:00Z with a 24 h lifetime, but verifyToken checks expiry against the real clock. The tokens therefore expired on 2026-11-15, and three tests began to fail. The test was wrong, not the code: it now makes its tokens at the current time, rounded down to the second.

bun scripts/gate.ts, bun x tsc --noEmit and these five test files pass: test/module-state, test/contribute-token, test/memory-watch, test/vitals and test/worker-memory-lines.

🤖 Generated with Claude Code

commented

The memory-growth need now names which VM grew and whether the growth is JS heap, malloc or native memory (#12951). Every worker VM sends its minute reading over a BroadcastChannel. The serving VM listens through hearWorkers in src/perf/memory_watch.ts and notes each reading into its memory watch.

hearWorkers keeps its channels in a WeakMap keyed by the watch object, so it adds no process-wide Map (the #12569 rule in test/module-state.test.ts). The merge from main conflicted in one place: the clock line in test/contribute-token.test.ts. Both sides had fixed the same bug, a fixed mint date that expired against verifyToken's real clock. I kept main's new Date(), so this pull no longer changes that file.

bun scripts/gate.ts, bun x tsc --noEmit and five test files pass: test/module-state, test/contribute-token, test/memory-watch, test/vitals and test/worker-memory-lines.

🤖 Generated with Claude Code

Approved

Approved. The needs-you line now names the owner and the part (heap, malloc or native) that grew most against its own first hour. Worker readings travel on the worker own minute timer over a BroadcastChannel, and the choice is stated. Input is validated in workerReadingOf, owners are capped at 64, and stale workers are not blamed. There is no heapStats call, README is untouched, and docs/features and AGENTS.md are updated. It adds no new surface, so the terminal home picks up the same sentence through vitals. Tests cover each case the issue lists. I read the diff and did not exercise the preview, since the line only shows after RSS doubles.

Two things I would do next:

  1. Merge the duplicate import of readNow in test/memory-watch.test.ts into the line above it.
  2. Open a follow-up issue for the growth cause once the line names one in production.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs