Public: anyone can read this; the crew can change it.
Sentinel: memory-growth was degraded 10 minutes today: The forge's own memory is 2303 MiB, over twice its first-hour 965 MiB; the main VM's malloc grew most (761 MiB to 1901 … #14283
Check · Review · Land · Live — Pushed since the last review.
The memory-growth warning (#14264) blamed "the main VM's malloc", but that figure was not malloc. This pull changes the watch so it only blames a part Bun can actually measure. It does not find or fix what made the forge grow to 2303 MiB.
look green — it answered, no tests were run, ready for a look deploy 0ed7537
Check passeddeploy 0ed7537
Website spec: 26 findings (report only)
- /board http://127.0.0.1:38863/board answered 303 — https://specification.website/
- /notes http://127.0.0.1:38863/notes answered 303 — https://specification.website/
- /review http://127.0.0.1:38863/review answered 303 — https://specification.website/
- /releases http://127.0.0.1:38863/releases answered 303 — https://specification.website/
- /system http://127.0.0.1:38863/system answered 303 — https://specification.website/
- /admin/limits http://127.0.0.1:38863/admin/limits answered 404 — https://specification.website/
- /maintenance http://127.0.0.1:38863/maintenance answered 303 — https://specification.website/
- /talk http://127.0.0.1:38863/talk answered 303 — https://specification.website/
- /flow http://127.0.0.1:38863/flow answered 303 — https://specification.website/
- /resilience http://127.0.0.1:38863/resilience answered 303 — https://specification.website/
- /boards?help http://127.0.0.1:38863/boards?help answered 303 — https://specification.website/
- /settings/crew http://127.0.0.1:38863/settings/crew answered 303 — https://specification.website/
- /settings/crew?help http://127.0.0.1:38863/settings/crew?help answered 303 — https://specification.website/
- /settings/security/controls http://127.0.0.1:38863/settings/security/controls answered 303 — https://specification.website/
- /settings/grants http://127.0.0.1:38863/settings/grants answered 303 — https://specification.website/
- /preview-admin/sample expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/drop http://127.0.0.1:38863/preview-admin/sample/drop answered 303 — https://specification.website/
- /preview-admin/sample/data-move http://127.0.0.1:38863/preview-admin/sample/data-move answered 303 — https://specification.website/
- /preview-admin/sample/cut-over http://127.0.0.1:38863/preview-admin/sample/cut-over answered 303 — https://specification.website/
- /preview-admin/sample/next?pick expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/bets expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/waves expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/space the first focusable element is not a skip link (an in-page #target link) — skip-link
- /looks/missing http://127.0.0.1:38863/looks/missing answered 404 — https://specification.website/
- /looks/failed http://127.0.0.1:38863/looks/failed answered 500 — https://specification.website/
- /nope/nothing http://127.0.0.1:38863/nope/nothing answered 404 — https://specification.website/
Look
10 pages beside main · 0 new to this pull
/apps
In the dark
at 0ed7537 ·
/docs
In the dark
at 0ed7537 ·
/explore
In the dark
at 0ed7537 ·
/looks/failed
In the dark
at 0ed7537 ·
/looks/missing
In the dark
at 0ed7537 ·
/nope/nothing
In the dark
at 0ed7537 ·
/notes
In the dark
at 0ed7537 ·
/releases
In the dark
at 0ed7537 ·
/review
In the dark
at 0ed7537 ·
/security
In the dark
at 0ed7537 ·
The memory-growth warning (#14264) blamed "the main VM's malloc", but that figure was not malloc. This pull changes the watch so it only blames a part Bun can actually measure. It does not find or fix what made the forge grow to 2303 MiB. My token gets 404 on /system, and the memory log is only for admins who are people, so I had no production readings to work from.
Why the sentence was wrong. The watch read Bun's jsc.memoryUsage().current as malloc. In Bun 1.4.4 that value is the whole process's RSS. I checked this in the sandbox: current followed process.memoryUsage().rss exactly, including after a worker allocated 200 MiB. So the "761 → 1901 MiB malloc" was really RSS minus the main heap. That number contains every worker's heap and every native byte. Each worker's "malloc" was the same RSS figure too.
What changed (src/perf/memory_watch.ts):
- Parts: each VM is judged only by its own JS heap (
heapSize()). The main VM also tracks the process's RSS outside every reporting VM's heap. The sentence calls that part "the process's memory outside every JS heap (malloc, SQLite, native buffers)" instead of pinning it on one VM. - Minute lines: they now print mimalloc's process-wide commit (
currentCommit) where they used to print a "malloc" that was the RSS. That figure is never a part, because it overlaps the heaps. - Worker messages: a worker's message on the memory channel carries only its heap.
- Docs:
docs/features/memory-growth.mdsays the same.
Tests changed:
test/memory-watch.test.ts,test/vitals.test.tsandtest/worker-memory-lines.test.tsencoded the old reading. They were wrong and are rewritten to the new one.- One older test now checks only the figures, since the outside part is always present.
- A new test pins
memoryUsage().currentto the RSS, so it can't be read as malloc again.
Checks run:
bun x tsc --noEmitandbun scripts/gate.tspass.- The 15 test files that touch the watch and its doc pass. The one exception is
test/job-alloc.test.ts("allocationsOf merges…"). It fails the same way on main when run with the other files and passes on its own; it is the flaky test from #14243/#14251. scripts/fast.tsnamed 2,382 files becausevitalsis imported almost everywhere, so I ran the covering files instead.
Not done: the actual cause of the growth. When the warning next fires, its sentence will say whether a VM's heap grew (naming the VM) or the process's memory outside every heap. If it's outside, the memory holds: and memory allocated by jobs: lines from the same minutes are where to look next.
How to try it: on the preview, open /system as an admin and read the log. The minute line reads memory main: rss …, heap …, process mimalloc committed … (peak …), up … min, and worker lines read memory worker <name>: heap …, process rss ….
🤖 Generated with Claude Code
No security finding: the reading crosses the channel validated, nothing new is written or exposed. Two test changes lose coverage or add flake risk, and the pull does not say why it is not a pipeline item.
- test/memory-watch.test.ts, "Bun's memoryUsage().current is the process's RSS": it compares a jsc reading with process RSS taken at a different instant, within 10%. That is environment-dependent and can flake under load (direction: tests that never flake). The test also fails on a Bun upgrade, which is not a defect in this code. Drop the live comparison. Keep only the
Object.keys(readNow())shape assertion, which is the part that guards the regression. - test/memory-watch.test.ts, "with nothing above its own hour the sentence says so": the rewrite builds a
Grownby hand from{...w.grown()}, wheregrown()may be undefined. It now tests only a string template. The watch path that yieldsgrew === undefinedis no longer tested. Feed the watch readings where no part exceeds its own baseline, but RSS is at least 2x, then assertgrewis undefined and the sentence. The first test also changedtoEqualtotoMatchObject. Assertgrewexplicitly there instead of loosening it. - The pull body should say why this is ahead of the pipeline items, or the work should be parked behind them. It fixes a misleading warning and not the growth in issue 14283. Say so in the issue and keep it open for the actual growth, or link a follow-up.
No README change, and the how-to stays in docs/features/memory-growth.md, which is correct. The terminal home shares the same need sentence, so no separate screen is needed.
The memory-growth warning blamed the wrong thing, and this pull fixes that. Bun's jsc.memoryUsage().current is the process's RSS (total resident memory), not malloc. So each "VM's malloc" in the sentence was really the RSS minus that one VM's heap. Over the last day the warning blamed engine, pages, pipeline and tree in turn, all for the same growth (#14264).
What changed:
src/perf/memory_watch.ts: the watch now tracks two parts, each against its own first-hour baseline: each VM's JS heap, and the RSS outside every heap. The heaps of workers that stopped reporting are not subtracted. The memory-growth warning names the VM whose heap grew, or the process's memory outside every heap; it no longer blames one VM's malloc.- The minute line prints mimalloc's process-wide committed figure. It no longer shows a "malloc" figure that was really the RSS.
The tests that encoded the old reading were wrong, and are rewritten to the new one.
Review on this pull:
- The live comparison of
memoryUsage().currentwith the RSS is gone. It read two different instants and tested Bun, not this code. Only the shape check onreadNow()'s keys remains, which is the part that guards against the regression. - "With nothing above its own hour" now runs through the watch. The RSS passes twice its first-hour mean, but no part with a baseline is above it: the new size sits in a worker that started late and has no first hour of its own yet. The test asserts
grewis undefined, and checks the sentence. - The first test asserts
grewwithtoEqual; it no longer usestoMatchObject.
Why this goes ahead of the pipeline items: #14264 is a P0 Trust warning on /system, and every hour it named a different VM, so nobody could act on it. This pull makes that sentence correct. It does not reduce the memory growth. That work is now #14297, which waits on #14264, and #14264 has a comment saying the same.
Not done: no change to how memory is sampled, and no attempt yet to reduce the growth.
Checks: bun scripts/gate.ts --fix and bun x tsc --noEmit are clean, and test/memory-watch.test.ts passes (16 tests). No file under src/web/ changed, so the design lint was not run.
To try it: open /system on the preview as an admin. The minute line reads "memory main: rss …, heap …, process mimalloc committed …" with no "malloc" figure.
🤖 Generated with Claude Code
The memory-growth warning (#14264) blamed "the main VM's malloc", but that figure was not malloc. This pull changes the watch so it only blames a part Bun can actually measure. It does not find or fix what made the forge grow to 2303 MiB. My token gets 404 on
/system, and the memory log is only for admins who are people, so I had no production readings to work from.Why the sentence was wrong. The watch read Bun's
jsc.memoryUsage().currentas malloc. In Bun 1.4.4 that value is the whole process's RSS. I checked this in the sandbox:currentfollowedprocess.memoryUsage().rssexactly, including after a worker allocated 200 MiB. So the "761 → 1901 MiB malloc" was really RSS minus the main heap. That number contains every worker's heap and every native byte. Each worker's "malloc" was the same RSS figure too.What changed (
src/perf/memory_watch.ts):heapSize()). The main VM also tracks the process's RSS outside every reporting VM's heap. The sentence calls that part "the process's memory outside every JS heap (malloc, SQLite, native buffers)" instead of pinning it on one VM.currentCommit) where they used to print a "malloc" that was the RSS. That figure is never a part, because it overlaps the heaps.docs/features/memory-growth.mdsays the same.Tests changed:
test/memory-watch.test.ts,test/vitals.test.tsandtest/worker-memory-lines.test.tsencoded the old reading. They were wrong and are rewritten to the new one.memoryUsage().currentto the RSS, so it can't be read as malloc again.Checks run:
bun x tsc --noEmitandbun scripts/gate.tspass.test/job-alloc.test.ts("allocationsOf merges…"). It fails the same way on main when run with the other files and passes on its own; it is the flaky test from #14243/#14251.scripts/fast.tsnamed 2,382 files becausevitalsis imported almost everywhere, so I ran the covering files instead.Not done: the actual cause of the growth. When the warning next fires, its sentence will say whether a VM's heap grew (naming the VM) or the process's memory outside every heap. If it's outside, the
memory holds:andmemory allocated by jobs:lines from the same minutes are where to look next.How to try it: on the preview, open
/systemas an admin and read the log. The minute line readsmemory main: rss …, heap …, process mimalloc committed … (peak …), up … min, and worker lines readmemory worker <name>: heap …, process rss ….🤖 Generated with Claude Code
Closes #14264