Memory by owner, the heap across a stall, memory growth (#12901)
Since Bun 1.4.4 the serving process has held several GiB after hours. This change makes the growth readable; it does not stop it (see "Not done").
Where it shows
- The minute line. Each minute the log carries
memory main: rss …, heap …, malloc … (peak …, committed peak …), up … min. Every worker VM writesmemory worker <name>: heap …, malloc …, process rss …on its own minute (#12925). RSS rising while the main heap stays flat is native memory; a worker line rising is that worker's. Read them in/system's log or the process log. - What is held. Beside the minute line the log carries
memory holds: <name> <count>, …: the forge's long-lived Maps, Sets and lists by name with how many entries each holds now, largest first, twelve at most (src/perf/held.ts). A worker whose thread holds anything writesmemory worker <name> holds: …. A name whose count only rises from line to line is the holder to read. - The stall line. Each loop-stall line ends with
heap X (was Y N s ago; …), the heap now against the last minute's sample.
- fell: a collection ran means the heap shrank by more than a tenth since the sample, so the collector ran inside that stall. Bun 1.4.4 reports no collector time, so this is inferred, not measured.
- did not fall: no collection seen means the stall was not the collector's.
- The need. When RSS is at least twice the mean of the process's first hour (at least ten samples),
/system, the board's forge strip and the terminal's home each list one need: the figure, the first-hour figure, which VM's part grew most and since when (#12951), for example "The forge's own memory is 4200 MiB, over twice its first-hour 2000 MiB; worker pages's malloc grew most (310 MiB to 2410 MiB) since 2026-11-15T04:00:00.000Z." - The parts. Each owner (the main VM, and each worker by name) keeps its own first-hour mean of each part:
JS heap(heapSize()),mallocbeyond the heap (memoryUsage().current - heap, floored at 0) and, for main only,native memory outside every VM's malloc(RSS less main's malloc and every reporting worker's, floored at 0). A worker's hour starts at its first reading, so one started on its first ask has its own. The part named is the one that grew the most bytes past its baseline. A worker silent five minutes is not named, and its malloc stops counting against native. When no part is above its own hour the sentence says so. - The worker names.
engine(src/engine/worker.ts);disk,tree(src/apps);lanes,pages,pipeline(src/flow);scan,du,leak(theanswerAsksworkers of src/kernel/worker_ask.ts);thumb(src/looks);log-writer(src/runs);mirror,vacuum(src/snapshot);code-rows,map,raster(src/web). A one-shot worker (mirror,vacuum) that ends inside its first minute writes no line; the timer is unref'd, so it never keeps a worker alive.
What to do when the need shows
- The need names the VM and the part. To see the curve, read the
memory mainandmemory workerlines around the time named: heap up means JS objects, heap flat with RSS up means native memory, a worker line up means that worker.
Then read the memory holds: lines across the same minutes: a holder that grew with the heap is named there with its count.
- Check the stall lines for
fell: a collection ran: stalls that are collections point at the heap. - Restart the forge to release it, and report the lines with the issue.
The need clears by itself once RSS falls under twice the baseline. A process that slept through its first hour has no baseline and raises none.
How it is built
src/perf/memory_watch.ts owns the readings (heapSize() and memoryUsage(), constant time; never heapStats(), which held the loop ~280 ms a minute, #11118). vitals() takes the watch as watch, as it takes disk. Each worker entry calls workerMemory(<name>) once at start; FORGE_WORKER_MEMORY_MS shortens the minute for tests. The same minute timer that writes a worker's line posts its reading ({ name, heap, current }) on the forge-memory BroadcastChannel (tellReading); the serving VM's memory sampler listens (hearWorkers), validates each message (workerReadingOf: a lower-case name, not main, finite byte counts) and notes it. Asks carry nothing extra, and at most 64 workers are kept. Tests: test/memory-watch.test.ts, test/worker-memory-lines.test.ts, test/vitals.test.ts, test/tui-memory-growth.test.ts.
What grew, and what holds it now
Found on a copy of production's data with no network, production's own work on (apps, crew, checks, the picture keeper) and a replay of pages, API and MCP calls, event streams and pushes; object counts by type from bun:jsc heapStats() after a full collection, and heap snapshots, at 5, 20 and 60 minutes.
- The turn gate kept every request an app's host answered (
src/http/app_turn.ts, in since 0.7.598, one release before Bun 1.4.4). The listener began a turn for every request and only the forge's own routes ended it; a request answered ahead of the routes (an app's host through the relay, a refusal) stayed in the gate's Map, with itsRequest,HeadersandAbortSignal, until an app's own token next waited on a quiet thread, which on production is never. On the copy, with nobody visiting an app, the looks' Chromium alone asked previews for about 450 requests a minute. The listener now leaves the gate in itsfinally, whatever door answered: the gate holds what is in flight. Bun holds nothing here: a servedRequestis collected on 1.3.14 and 1.4.4 alike once nothing of ours keeps it. - The plan and lane caches kept answers nobody could ask for again (
src/web/map_cache.ts,src/view/cache.ts). A plan was keyed by its stamp, which moves with every run that starts or spends, and a lane by its repository's latest event; an entry past its minute was only dropped when the same key was read again or the count (64 plans, 128 lanes) pushed it out. A plan holds its repository's whole graph: the copy held ten graphs of the busiest repository (4,593 issues each) after an hour of page views, and the cache had room for 64. Now there is one plan per repository, spend visibility and day, and one lane per repository, scope and day, each replaced when its stamp moves; and every view cache drops what is past its time on the first write afterSWEEP_MS(30 s). - A hosted forge that slept left its request counter and route timing behind (
src/http/server.ts), one more of each with every wake, each keeping the closed database and a flush that retried against it. They now go when the forge closes, written out first.
A new long-lived collection names itself in heldOf(app) (a forge's) or threadHeld (the thread's), with a bound and a test of its eviction. Named today: turn, requests in flight, event streams, request counters, route timings, map plans, map lanes, map layouts, and four the crew keeps per pull or per check that nothing sweeps yet (autonomy conflicted pulls, finishing said, checks retried, app links said): each is a short string per entry, and its line says when one is worth a sweep.
Tests: test/app-turn.test.ts, test/cache-holds.test.ts, test/forge-sleep.test.ts, test/held.test.ts.