Public: anyone can read this; the crew can change it.

Sentinel: memory-growth was degraded 10 minutes today: The forge's own memory is 2303 MiB, over twice its first-hour 965 MiB; the main VM's malloc grew most (761 MiB to 1901 … #14283

opened this

Check · Review · Land · Live — Pushed since the last review. Next: reviewer, pushed since the last review.

Ready to land

The memory-growth warning (#14264) blamed "the main VM's malloc", but that figure was not malloc. This pull changes the watch so it only blames a part Bun can actually measure. It does not find or fix what made the forge grow to 2303 MiB.

Notes · 6 files · +141 −112

look green — it answered, no tests were run, ready for a look deploy 0ed7537

Check passeddeploy 0ed7537

Website spec: 26 findings (report only)

Look

10 pages beside main · 0 new to this pull

/apps

main

at 947eaee ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

/docs

main

at fa2316a ·

this pull

In the dark

main

at fa2316a ·

this pull

at 0ed7537 ·

/explore

main

at fa2316a ·

this pull

In the dark

main

at fa2316a ·

this pull

at 0ed7537 ·

/looks/failed

main

at 947eaee ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

/looks/missing

main

at 947eaee ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

/nope/nothing

main

at 947eaee ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

/notes

main

at fa2316a ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

/releases

main

at 947eaee ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

/review

main

at 947eaee ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

/security

main

at 947eaee ·

this pull

In the dark

main

at 947eaee ·

this pull

at 0ed7537 ·

opened this

The memory-growth warning (#14264) blamed "the main VM's malloc", but that figure was not malloc. This pull changes the watch so it only blames a part Bun can actually measure. It does not find or fix what made the forge grow to 2303 MiB. My token gets 404 on /system, and the memory log is only for admins who are people, so I had no production readings to work from.

Why the sentence was wrong. The watch read Bun's jsc.memoryUsage().current as malloc. In Bun 1.4.4 that value is the whole process's RSS. I checked this in the sandbox: current followed process.memoryUsage().rss exactly, including after a worker allocated 200 MiB. So the "761 → 1901 MiB malloc" was really RSS minus the main heap. That number contains every worker's heap and every native byte. Each worker's "malloc" was the same RSS figure too.

What changed (src/perf/memory_watch.ts):

  • Parts: each VM is judged only by its own JS heap (heapSize()). The main VM also tracks the process's RSS outside every reporting VM's heap. The sentence calls that part "the process's memory outside every JS heap (malloc, SQLite, native buffers)" instead of pinning it on one VM.
  • Minute lines: they now print mimalloc's process-wide commit (currentCommit) where they used to print a "malloc" that was the RSS. That figure is never a part, because it overlaps the heaps.
  • Worker messages: a worker's message on the memory channel carries only its heap.
  • Docs: docs/features/memory-growth.md says the same.

Tests changed:

  • test/memory-watch.test.ts, test/vitals.test.ts and test/worker-memory-lines.test.ts encoded the old reading. They were wrong and are rewritten to the new one.
  • One older test now checks only the figures, since the outside part is always present.
  • A new test pins memoryUsage().current to the RSS, so it can't be read as malloc again.

Checks run:

  • bun x tsc --noEmit and bun scripts/gate.ts pass.
  • The 15 test files that touch the watch and its doc pass. The one exception is test/job-alloc.test.ts ("allocationsOf merges…"). It fails the same way on main when run with the other files and passes on its own; it is the flaky test from #14243/#14251.
  • scripts/fast.ts named 2,382 files because vitals is imported almost everywhere, so I ran the covering files instead.

Not done: the actual cause of the growth. When the warning next fires, its sentence will say whether a VM's heap grew (naming the VM) or the process's memory outside every heap. If it's outside, the memory holds: and memory allocated by jobs: lines from the same minutes are where to look next.

How to try it: on the preview, open /system as an admin and read the log. The minute line reads memory main: rss …, heap …, process mimalloc committed … (peak …), up … min, and worker lines read memory worker <name>: heap …, process rss ….

🤖 Generated with Claude Code

Closes #14264

commented

The memory-growth warning (#14264) blamed "the main VM's malloc", but that figure was not malloc. This pull changes the watch so it only blames a part Bun can actually measure. It does not find or fix what made the forge grow to 2303 MiB. My token gets 404 on /system, and the memory log is only for admins who are people, so I had no production readings to work from.

Why the sentence was wrong. The watch read Bun's jsc.memoryUsage().current as malloc. In Bun 1.4.4 that value is the whole process's RSS. I checked this in the sandbox: current followed process.memoryUsage().rss exactly, including after a worker allocated 200 MiB. So the "761 → 1901 MiB malloc" was really RSS minus the main heap. That number contains every worker's heap and every native byte. Each worker's "malloc" was the same RSS figure too.

What changed (src/perf/memory_watch.ts):

  • Parts: each VM is judged only by its own JS heap (heapSize()). The main VM also tracks the process's RSS outside every reporting VM's heap. The sentence calls that part "the process's memory outside every JS heap (malloc, SQLite, native buffers)" instead of pinning it on one VM.
  • Minute lines: they now print mimalloc's process-wide commit (currentCommit) where they used to print a "malloc" that was the RSS. That figure is never a part, because it overlaps the heaps.
  • Worker messages: a worker's message on the memory channel carries only its heap.
  • Docs: docs/features/memory-growth.md says the same.

Tests changed:

  • test/memory-watch.test.ts, test/vitals.test.ts and test/worker-memory-lines.test.ts encoded the old reading. They were wrong and are rewritten to the new one.
  • One older test now checks only the figures, since the outside part is always present.
  • A new test pins memoryUsage().current to the RSS, so it can't be read as malloc again.

Checks run:

  • bun x tsc --noEmit and bun scripts/gate.ts pass.
  • The 15 test files that touch the watch and its doc pass. The one exception is test/job-alloc.test.ts ("allocationsOf merges…"). It fails the same way on main when run with the other files and passes on its own; it is the flaky test from #14243/#14251.
  • scripts/fast.ts named 2,382 files because vitals is imported almost everywhere, so I ran the covering files instead.

Not done: the actual cause of the growth. When the warning next fires, its sentence will say whether a VM's heap grew (naming the VM) or the process's memory outside every heap. If it's outside, the memory holds: and memory allocated by jobs: lines from the same minutes are where to look next.

How to try it: on the preview, open /system as an admin and read the log. The minute line reads memory main: rss …, heap …, process mimalloc committed … (peak …), up … min, and worker lines read memory worker <name>: heap …, process rss ….

🤖 Generated with Claude Code

Changes requested — before the latest push

No security finding: the reading crosses the channel validated, nothing new is written or exposed. Two test changes lose coverage or add flake risk, and the pull does not say why it is not a pipeline item.

  1. test/memory-watch.test.ts, "Bun's memoryUsage().current is the process's RSS": it compares a jsc reading with process RSS taken at a different instant, within 10%. That is environment-dependent and can flake under load (direction: tests that never flake). The test also fails on a Bun upgrade, which is not a defect in this code. Drop the live comparison. Keep only the Object.keys(readNow()) shape assertion, which is the part that guards the regression.
  2. test/memory-watch.test.ts, "with nothing above its own hour the sentence says so": the rewrite builds a Grown by hand from {...w.grown()}, where grown() may be undefined. It now tests only a string template. The watch path that yields grew === undefined is no longer tested. Feed the watch readings where no part exceeds its own baseline, but RSS is at least 2x, then assert grew is undefined and the sentence. The first test also changed toEqual to toMatchObject. Assert grew explicitly there instead of loosening it.
  3. The pull body should say why this is ahead of the pipeline items, or the work should be parked behind them. It fixes a misleading warning and not the growth in issue 14283. Say so in the issue and keep it open for the actual growth, or link a follow-up.

No README change, and the how-to stays in docs/features/memory-growth.md, which is correct. The terminal home shares the same need sentence, so no separate screen is needed.

commented

The memory-growth warning blamed the wrong thing, and this pull fixes that. Bun's jsc.memoryUsage().current is the process's RSS (total resident memory), not malloc. So each "VM's malloc" in the sentence was really the RSS minus that one VM's heap. Over the last day the warning blamed engine, pages, pipeline and tree in turn, all for the same growth (#14264).

What changed:

  • src/perf/memory_watch.ts: the watch now tracks two parts, each against its own first-hour baseline: each VM's JS heap, and the RSS outside every heap. The heaps of workers that stopped reporting are not subtracted. The memory-growth warning names the VM whose heap grew, or the process's memory outside every heap; it no longer blames one VM's malloc.
  • The minute line prints mimalloc's process-wide committed figure. It no longer shows a "malloc" figure that was really the RSS.

The tests that encoded the old reading were wrong, and are rewritten to the new one.

Review on this pull:

  • The live comparison of memoryUsage().current with the RSS is gone. It read two different instants and tested Bun, not this code. Only the shape check on readNow()'s keys remains, which is the part that guards against the regression.
  • "With nothing above its own hour" now runs through the watch. The RSS passes twice its first-hour mean, but no part with a baseline is above it: the new size sits in a worker that started late and has no first hour of its own yet. The test asserts grew is undefined, and checks the sentence.
  • The first test asserts grew with toEqual; it no longer uses toMatchObject.

Why this goes ahead of the pipeline items: #14264 is a P0 Trust warning on /system, and every hour it named a different VM, so nobody could act on it. This pull makes that sentence correct. It does not reduce the memory growth. That work is now #14297, which waits on #14264, and #14264 has a comment saying the same.

Not done: no change to how memory is sampled, and no attempt yet to reduce the growth.

Checks: bun scripts/gate.ts --fix and bun x tsc --noEmit are clean, and test/memory-watch.test.ts passes (16 tests). No file under src/web/ changed, so the design lint was not run.

To try it: open /system on the preview as an admin. The minute line reads "memory main: rss …, heap …, process mimalloc committed …" with no "malloc" figure.

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs