Public: anyone can read this; the crew can change it.

Since Bun 1.4.4 the serving process holds 4.5 to 7 GiB after four hours (was 1.8 to 2.4) and stalls three times as often as it ages #12902

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull covers asks 1, 2 and 4 in part. It does not find what holds the memory and fixes nothing that grows, so it does not meet the "done when" at four hours. What changed Per-minute memory line (ask 1, main VM).

Notes

shipped production is serving it deploy 1b35e66

opened this

This pull covers asks 1, 2 and 4 in part. It does not find what holds the memory and fixes nothing that grows, so it does not meet the "done when" at four hours.

What changed

  • Per-minute memory line (ask 1, main VM). The log now carries memory main: rss …, heap …, malloc … (peak …, committed peak …), up N min every minute. It reads heapSize() and memoryUsage() from bun:jsc and is written from the existing memory sampler. It never calls heapStats(), because that held the loop about 280 ms a minute on a 2M-object heap (#11118).
  • Engine worker line (ask 1, part). The engine worker logs its own memory worker engine: … line each minute. This is the one Worker wired so far. A Worker's heap can only be read from inside it. The helper startWorkerMemory is ready for picture, scan and watch workers, but I did not wire those.
  • Stall line (ask 2). Bun 1.4.4 exposes no collector time: a gc PerformanceObserver is accepted but delivers no entries. So each stall line now ends with the heap across it, for example heap 400 MiB (was 1000 MiB 41 s ago; fell: a collection ran). A heap that fell means a collection ran inside the stall. This is an inference from the heap, not a measured GC time.
  • Needs-you line (ask 4). src/perf/memory_watch.ts keeps the mean RSS of the first hour. When RSS reaches twice that figure, one memory-growth need is raised in vitals. It states both figures and when the current run above 1.25× the baseline began. A process that slept through its first hour (under 10 samples) gets no figure.

What I did not do

  • Heap snapshots (ask 1) and the replay. I could not take them at 30 minutes and 3 hours, because the sandbox cannot reach production or the stage rig.
  • The fix (ask 3). I have no heap-by-owner data, so I did not bound or change anything on a guess. The lines above will show which VM grows once 0.7.60x runs for a few hours. The FORGE_DB_CACHE_MIB default of 1024 is a candidate to check against that data.
  • GC settings. I changed no collector setting.

Tests and checks

  • Tests: test/memory-watch.test.ts has hand-computed cases for the baseline, the 2× trigger, the start time and the heap across a stall. A new case in test/vitals.test.ts covers the need.
  • Checks run: bun scripts/gate.ts and tsc are clean.
  • Not run: scripts/fast.ts named 2205 files, too broad to run here. I ran the 16 test files around memory, stalls, vitals and needs instead.
  • One failure is not from this change. test/loop-stall.test.ts (slow statements) fails identically with my changes stashed.
  • Hold: the "Releasing trevato/forge" note did not touch this change.

To try Run it, then check the log for the memory main line after a minute. The needs-you line will show on /system and the home needs stack once RSS passes twice its first-hour mean.

Closes #12901

4 earlier comments · 1 event
commented

This pull covers asks 1, 2 and 4 in part. It does not find what holds the memory and fixes nothing that grows, so it does not meet the "done when" at four hours.

What changed

  • Per-minute memory line (ask 1, main VM). The log now carries memory main: rss …, heap …, malloc … (peak …, committed peak …), up N min every minute. It reads heapSize() and memoryUsage() from bun:jsc and is written from the existing memory sampler. It never calls heapStats(), because that held the loop about 280 ms a minute on a 2M-object heap (#11118).
  • Engine worker line (ask 1, part). The engine worker logs its own memory worker engine: … line each minute. This is the one Worker wired so far. A Worker's heap can only be read from inside it. The helper startWorkerMemory is ready for picture, scan and watch workers, but I did not wire those.
  • Stall line (ask 2). Bun 1.4.4 exposes no collector time: a gc PerformanceObserver is accepted but delivers no entries. So each stall line now ends with the heap across it, for example heap 400 MiB (was 1000 MiB 41 s ago; fell: a collection ran). A heap that fell means a collection ran inside the stall. This is an inference from the heap, not a measured GC time.
  • Needs-you line (ask 4). src/perf/memory_watch.ts keeps the mean RSS of the first hour. When RSS reaches twice that figure, one memory-growth need is raised in vitals. It states both figures and when the current run above 1.25× the baseline began. A process that slept through its first hour (under 10 samples) gets no figure.

What I did not do

  • Heap snapshots (ask 1) and the replay. I could not take them at 30 minutes and 3 hours, because the sandbox cannot reach production or the stage rig.
  • The fix (ask 3). I have no heap-by-owner data, so I did not bound or change anything on a guess. The lines above will show which VM grows once 0.7.60x runs for a few hours. The FORGE_DB_CACHE_MIB default of 1024 is a candidate to check against that data.
  • GC settings. I changed no collector setting.

Tests and checks

  • Tests: test/memory-watch.test.ts has hand-computed cases for the baseline, the 2× trigger, the start time and the heap across a stall. A new case in test/vitals.test.ts covers the need.
  • Checks run: bun scripts/gate.ts and tsc are clean.
  • Not run: scripts/fast.ts named 2205 files, too broad to run here. I ran the 16 test files around memory, stalls, vitals and needs instead.
  • One failure is not from this change. test/loop-stall.test.ts (slow statements) fails identically with my changes stashed.
  • Hold: the "Releasing trevato/forge" note did not touch this change.

To try Run it, then check the log for the memory main line after a minute. The needs-you line will show on /system and the home needs stack once RSS passes twice its first-hour mean.

commented

This pull adds memory-by-owner lines to the log, the heap reading across a stall, and a needs-you sentence when the process passes twice its first-hour memory. This push fixes the timers test that failed on it.

  • Fixed: test/timers-named.test.ts rejected the worker's new minute memory timer in src/perf/memory_watch.ts because it wasn't wrapped in during. I added it to the test's allow-list instead. It runs inside the worker VM, where the main process's span ring isn't reachable, and it only reads two counters. test/timers-named.test.ts and test/memory-watch.test.ts pass, and bun x tsc --noEmit is clean.
  • Not fixed: test/looks-owner.test.ts also failed, taking 90 s against a 2 s budget. The check log says it fails only beside other tests, and this change touches no page or look, so I think it is load-related. I did not run or change it, so it may fail again on this push.
  • Not run: bun scripts/gate.ts and bun scripts/fast.ts. I didn't check whether this clone has them.
  • Try it: open /system as an admin. The log carries a memory main line each minute, and each stall line ends with the heap across it.

🤖 Generated with Claude Code

Changes requested — before the latest push

Changes needed, all in one round:

  1. FIRST: no docs/features/<name>.md for the new memory log lines, the stall heap reading and the memory-growth need. The how-to belongs there. Say where the lines appear, what 'fell: a collection ran' means and what to do when the need shows.
  2. Both doors: the memory-growth need is only shown to be raised in vitals(). Add a test that it reaches the terminal needs view in src/tui/ as it reaches /system. Or show that the TUI already renders every vitals need, and say so.
  3. The vitals test is not a real test of the wiring. It replaces memoryWatch.grown by hand and has a leftover real check that depends on process-wide state. Give vitals an injected watch (like disk) and use it.
  4. Dead and false claims in memory_watch.ts. The header says it reads heapStats().extraMemorySize and objectCount 'only when asked for', but nothing does. Remove that. baseline() is only used by tests. Drop it or use it.
  5. main.ts imports heapSize from bun:jsc just for the stall line. Move that into memoryWatch (a heapAcross() with no argument) so the module owns the reading.

Also, this does not meet the issue's done-when (it finds and fixes nothing that grows). The pull says so honestly. Open a follow-up issue for the heap-by-owner analysis and the fix, and for wiring the other workers.

commented

This pull makes the memory growth readable. It does not find or fix what grows, so it does not meet the issue's done-when. The pull adds memory main and memory worker engine log lines each minute, the heap reading at the end of each stall line, and a memory-growth need. The need appears when RSS reaches twice its first-hour mean. The analysis and the fix are follow-up #12924, which also covers wiring the other workers; only the engine worker writes a line today.

Changed in response to review:

  • Feature doc: docs/features/memory-growth.md says where the lines appear, what fell: a collection ran means (the heap shrank by more than a tenth since the last sample) and what to do when the need shows.
  • Terminal needs view: the terminal home already renders every vitals need through forgeNeedList, the same list /system reads. test/tui-memory-growth.test.ts now shows the memory-growth need in the shared list and on an admin's terminal home.
  • Injected watch: vitals() takes a watch input, like disk, and app.config.memoryWatch passes it through readHealth. The vitals test uses it, so the patching and the leftover real check are gone. Because the new setting is a method-holding object, I listed it under STAYS in src/engine/crossing.ts and updated test/engine-crossing.test.ts to match.
  • memory_watch.ts:

- I removed the stale header claim about extraMemorySize and objectCount. - I dropped baseline(), which only tests used. - heapAcross(now?, heapNow?) now reads heapSize() itself, so main.ts no longer imports bun:jsc.

bun x tsc --noEmit and bun scripts/gate.ts are clean, and the tests covering the change pass (46 tests across 5 files). I did not run scripts/fast.ts. The earlier test/looks-owner.test.ts failure was not re-run or changed, and the forge's check will show whether it recurs.

🤖 Generated with Claude Code

commented

This push clears the three gate failures from the last check and changes no behaviour. bun scripts/gate.ts and bun x tsc --noEmit are clean locally. I did not re-run the tests, since only comments changed.

  • Comment wraps: the long comment in src/perf/memory_watch.ts and the header of test/tui-memory-growth.test.ts are rewrapped to 100 columns.
  • Pipeline-first rule: the gate refused a feature doc with no pipeline change. I put a Direction: line in the commit message instead of changing the pipeline. It says the memory growth is the issue's own ask and the doc is its how-to. Whether that justification is accepted is the reviewer's call.

The pull otherwise stands as described before: it makes the memory growth readable (log lines, stall heap reading, memory-growth need) but does not find or fix what grows. The analysis, the fix and the other workers' memory lines are follow-up #12924.

To try it, open /system as an admin. The log carries a memory main line each minute, and each stall line ends with the heap across it.

🤖 Generated with Claude Code

commented

The feature-note index was the cause of two of the three failures; I added the missing row and pushed. The third failure, "live previews > a fresh account's second preview is refused…", looks like a hung shard rather than a code problem.

  • Index failures: docs/features/memory-growth.md had no row in docs/features/README.md. I added one ("Memory by owner and growth across a stall"). test/feature-orphans.test.ts now passes, 4 of 4, which covers both "no feature note is an orphan" and "the index is one note to a row".
  • Live previews failure: the check log shows it ran 420 s before failing, which points to a hang rather than an assertion. The test lives in test/quotas-running.test.ts; that file passes locally, 7 of 7. I made no code change for it, so if it fails again on this push it needs a closer look.
  • Not run: I ran only these two test files, not bun scripts/gate.ts or bun x tsc --noEmit. The change is one added line in a markdown file.
Approved

Earlier asks are met. The how-to is in docs/features/memory-growth.md and indexed, not in README. The terminal home shows the need through the shared forgeNeedList, and test/tui-memory-growth.test.ts covers it. The watch is injected like disk, and its STAYS entry is listed. The stale header claims are gone, and the timers allow-list entry is justified. The pull makes growth readable and does not fix it; that is stated honestly and tracked in #12924. Nothing here touches the page, so there is no layout, keyboard or phone concern.

Next two things, both non-blocking:

  1. Non-blocking: wire the picture, scan and watch workers to startWorkerMemory in #12924 so every owner has a line.
  2. Non-blocking: the "fell: a collection ran" verdict is an inference from a 10% heap drop. Keep the doc wording that says so, and replace it with a measured collector time if Bun exposes one.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs