Public: anyone can read this; the crew can change it.
Since Bun 1.4.4 the serving process holds 4.5 to 7 GiB after four hours (was 1.8 to 2.4) and stalls three times as often as it ages #12902
Check · Review · Land · Live done — Landed and live in production.
This pull covers asks 1, 2 and 4 in part. It does not find what holds the memory and fixes nothing that grows, so it does not meet the "done when" at four hours. What changed Per-minute memory line (ask 1, main VM).
shipped production is serving it deploy 1b35e66
4 earlier comments · 1 event
This pull covers asks 1, 2 and 4 in part. It does not find what holds the memory and fixes nothing that grows, so it does not meet the "done when" at four hours.
What changed
- Per-minute memory line (ask 1, main VM). The log now carries
memory main: rss …, heap …, malloc … (peak …, committed peak …), up N minevery minute. It readsheapSize()andmemoryUsage()frombun:jscand is written from the existing memory sampler. It never callsheapStats(), because that held the loop about 280 ms a minute on a 2M-object heap (#11118). - Engine worker line (ask 1, part). The engine worker logs its own
memory worker engine: …line each minute. This is the one Worker wired so far. A Worker's heap can only be read from inside it. The helperstartWorkerMemoryis ready for picture, scan and watch workers, but I did not wire those. - Stall line (ask 2). Bun 1.4.4 exposes no collector time: a
gcPerformanceObserver is accepted but delivers no entries. So each stall line now ends with the heap across it, for exampleheap 400 MiB (was 1000 MiB 41 s ago; fell: a collection ran). A heap that fell means a collection ran inside the stall. This is an inference from the heap, not a measured GC time. - Needs-you line (ask 4).
src/perf/memory_watch.tskeeps the mean RSS of the first hour. When RSS reaches twice that figure, onememory-growthneed is raised invitals. It states both figures and when the current run above 1.25× the baseline began. A process that slept through its first hour (under 10 samples) gets no figure.
What I did not do
- Heap snapshots (ask 1) and the replay. I could not take them at 30 minutes and 3 hours, because the sandbox cannot reach production or the stage rig.
- The fix (ask 3). I have no heap-by-owner data, so I did not bound or change anything on a guess. The lines above will show which VM grows once 0.7.60x runs for a few hours. The
FORGE_DB_CACHE_MIBdefault of 1024 is a candidate to check against that data. - GC settings. I changed no collector setting.
Tests and checks
- Tests:
test/memory-watch.test.tshas hand-computed cases for the baseline, the 2× trigger, the start time and the heap across a stall. A new case intest/vitals.test.tscovers the need. - Checks run:
bun scripts/gate.tsandtscare clean. - Not run:
scripts/fast.tsnamed 2205 files, too broad to run here. I ran the 16 test files around memory, stalls, vitals and needs instead. - One failure is not from this change.
test/loop-stall.test.ts(slow statements) fails identically with my changes stashed. - Hold: the "Releasing trevato/forge" note did not touch this change.
To try
Run it, then check the log for the memory main line after a minute. The needs-you line will show on /system and the home needs stack once RSS passes twice its first-hour mean.
This pull adds memory-by-owner lines to the log, the heap reading across a stall, and a needs-you sentence when the process passes twice its first-hour memory. This push fixes the timers test that failed on it.
- Fixed:
test/timers-named.test.tsrejected the worker's new minute memory timer insrc/perf/memory_watch.tsbecause it wasn't wrapped induring. I added it to the test's allow-list instead. It runs inside the worker VM, where the main process's span ring isn't reachable, and it only reads two counters.test/timers-named.test.tsandtest/memory-watch.test.tspass, andbun x tsc --noEmitis clean. - Not fixed:
test/looks-owner.test.tsalso failed, taking 90 s against a 2 s budget. The check log says it fails only beside other tests, and this change touches no page or look, so I think it is load-related. I did not run or change it, so it may fail again on this push. - Not run:
bun scripts/gate.tsandbun scripts/fast.ts. I didn't check whether this clone has them. - Try it: open
/systemas an admin. The log carries amemory mainline each minute, and each stall line ends with the heap across it.
🤖 Generated with Claude Code
Changes needed, all in one round:
- FIRST: no docs/features/<name>.md for the new memory log lines, the stall heap reading and the memory-growth need. The how-to belongs there. Say where the lines appear, what 'fell: a collection ran' means and what to do when the need shows.
- Both doors: the memory-growth need is only shown to be raised in vitals(). Add a test that it reaches the terminal needs view in src/tui/ as it reaches /system. Or show that the TUI already renders every vitals need, and say so.
- The vitals test is not a real test of the wiring. It replaces memoryWatch.grown by hand and has a leftover
realcheck that depends on process-wide state. Give vitals an injected watch (likedisk) and use it. - Dead and false claims in memory_watch.ts. The header says it reads heapStats().extraMemorySize and objectCount 'only when asked for', but nothing does. Remove that.
baseline()is only used by tests. Drop it or use it. - main.ts imports heapSize from bun:jsc just for the stall line. Move that into memoryWatch (a heapAcross() with no argument) so the module owns the reading.
Also, this does not meet the issue's done-when (it finds and fixes nothing that grows). The pull says so honestly. Open a follow-up issue for the heap-by-owner analysis and the fix, and for wiring the other workers.
This pull makes the memory growth readable. It does not find or fix what grows, so it does not meet the issue's done-when. The pull adds memory main and memory worker engine log lines each minute, the heap reading at the end of each stall line, and a memory-growth need. The need appears when RSS reaches twice its first-hour mean. The analysis and the fix are follow-up #12924, which also covers wiring the other workers; only the engine worker writes a line today.
Changed in response to review:
- Feature doc:
docs/features/memory-growth.mdsays where the lines appear, whatfell: a collection ranmeans (the heap shrank by more than a tenth since the last sample) and what to do when the need shows. - Terminal needs view: the terminal home already renders every vitals need through
forgeNeedList, the same list/systemreads.test/tui-memory-growth.test.tsnow shows the memory-growth need in the shared list and on an admin's terminal home. - Injected watch:
vitals()takes awatchinput, likedisk, andapp.config.memoryWatchpasses it throughreadHealth. The vitals test uses it, so the patching and the leftoverrealcheck are gone. Because the new setting is a method-holding object, I listed it underSTAYSinsrc/engine/crossing.tsand updatedtest/engine-crossing.test.tsto match. memory_watch.ts:
- I removed the stale header claim about extraMemorySize and objectCount.
- I dropped baseline(), which only tests used.
- heapAcross(now?, heapNow?) now reads heapSize() itself, so main.ts no longer imports bun:jsc.
bun x tsc --noEmit and bun scripts/gate.ts are clean, and the tests covering the change pass (46 tests across 5 files). I did not run scripts/fast.ts. The earlier test/looks-owner.test.ts failure was not re-run or changed, and the forge's check will show whether it recurs.
🤖 Generated with Claude Code
This push clears the three gate failures from the last check and changes no behaviour. bun scripts/gate.ts and bun x tsc --noEmit are clean locally. I did not re-run the tests, since only comments changed.
- Comment wraps: the long comment in
src/perf/memory_watch.tsand the header oftest/tui-memory-growth.test.tsare rewrapped to 100 columns. - Pipeline-first rule: the gate refused a feature doc with no pipeline change. I put a
Direction:line in the commit message instead of changing the pipeline. It says the memory growth is the issue's own ask and the doc is its how-to. Whether that justification is accepted is the reviewer's call.
The pull otherwise stands as described before: it makes the memory growth readable (log lines, stall heap reading, memory-growth need) but does not find or fix what grows. The analysis, the fix and the other workers' memory lines are follow-up #12924.
To try it, open /system as an admin. The log carries a memory main line each minute, and each stall line ends with the heap across it.
🤖 Generated with Claude Code
The feature-note index was the cause of two of the three failures; I added the missing row and pushed. The third failure, "live previews > a fresh account's second preview is refused…", looks like a hung shard rather than a code problem.
- Index failures:
docs/features/memory-growth.mdhad no row indocs/features/README.md. I added one ("Memory by owner and growth across a stall").test/feature-orphans.test.tsnow passes, 4 of 4, which covers both "no feature note is an orphan" and "the index is one note to a row". - Live previews failure: the check log shows it ran 420 s before failing, which points to a hang rather than an assertion. The test lives in
test/quotas-running.test.ts; that file passes locally, 7 of 7. I made no code change for it, so if it fails again on this push it needs a closer look. - Not run: I ran only these two test files, not
bun scripts/gate.tsorbun x tsc --noEmit. The change is one added line in a markdown file.
Earlier asks are met. The how-to is in docs/features/memory-growth.md and indexed, not in README. The terminal home shows the need through the shared forgeNeedList, and test/tui-memory-growth.test.ts covers it. The watch is injected like disk, and its STAYS entry is listed. The stale header claims are gone, and the timers allow-list entry is justified. The pull makes growth readable and does not fix it; that is stated honestly and tracked in #12924. Nothing here touches the page, so there is no layout, keyboard or phone concern.
Next two things, both non-blocking:
- Non-blocking: wire the picture, scan and watch workers to startWorkerMemory in #12924 so every owner has a line.
- Non-blocking: the "fell: a collection ran" verdict is an inference from a 10% heap drop. Keep the doc wording that says so, and replace it with a measured collector time if Bun exposes one.
This pull covers asks 1, 2 and 4 in part. It does not find what holds the memory and fixes nothing that grows, so it does not meet the "done when" at four hours.
What changed
memory main: rss …, heap …, malloc … (peak …, committed peak …), up N minevery minute. It readsheapSize()andmemoryUsage()frombun:jscand is written from the existing memory sampler. It never callsheapStats(), because that held the loop about 280 ms a minute on a 2M-object heap (#11118).memory worker engine: …line each minute. This is the one Worker wired so far. A Worker's heap can only be read from inside it. The helperstartWorkerMemoryis ready for picture, scan and watch workers, but I did not wire those.gcPerformanceObserver is accepted but delivers no entries. So each stall line now ends with the heap across it, for exampleheap 400 MiB (was 1000 MiB 41 s ago; fell: a collection ran). A heap that fell means a collection ran inside the stall. This is an inference from the heap, not a measured GC time.src/perf/memory_watch.tskeeps the mean RSS of the first hour. When RSS reaches twice that figure, onememory-growthneed is raised invitals. It states both figures and when the current run above 1.25× the baseline began. A process that slept through its first hour (under 10 samples) gets no figure.What I did not do
FORGE_DB_CACHE_MIBdefault of 1024 is a candidate to check against that data.Tests and checks
test/memory-watch.test.tshas hand-computed cases for the baseline, the 2× trigger, the start time and the heap across a stall. A new case intest/vitals.test.tscovers the need.bun scripts/gate.tsandtscare clean.scripts/fast.tsnamed 2205 files, too broad to run here. I ran the 16 test files around memory, stalls, vitals and needs instead.test/loop-stall.test.ts(slow statements) fails identically with my changes stashed.To try Run it, then check the log for the
memory mainline after a minute. The needs-you line will show on/systemand the home needs stack once RSS passes twice its first-hour mean.Closes #12901