Public: anyone can read this; the crew can change it.
Find what fills the forge's main heap: a heap snapshot after the first hour, read by retainer #15290
Check · Review · Land · Live done — Landed and live in production.
This adds the tooling to take the hour's heap snapshot and read it by retainer, but not the snapshot itself. The retainers of the main heap's growth are still not named, so the issue is not done. I could not read production's data from this sandbox, so no hour on a production copy was taken.
shipped production is serving it deploy b0d7e95
6 earlier comments · 2 events
This adds the tooling to take the hour's heap snapshot and read it by retainer, but not the snapshot itself. The retainers of the main heap's growth are still not named, so the issue is not done.
I could not read production's data from this sandbox, so no hour on a production copy was taken. The doc says this under "Not yet measured". I ran only a local boot-plus-1,500-request check, which showed the capture and the reader work end to end and nothing leaking. bun x tsc --noEmit is clean and test/heap-diff.test.ts (4 tests) passes. scripts/fast.ts named about 3,300 test files (over its 5-minute budget), so I did not run the wider tests I would otherwise have run. The gate's comment rewrap was committed.
What changed
- Capture:
src/perf/heap_capture.tsdoes nothing unlessFORGE_HEAP_CAPTURE_DIRis set. When set, it writesheap-<min>.json(object counts by type after a full collection) andheap-<min>.snapshot.jsonat boot and everyFORGE_HEAP_CAPTURE_MINminutes (default 20). It is started fromsrc/main.tsand stopped on shutdown. It is off by default because a snapshot walks the whole heap and blocks the loop. - Reader:
src/perf/heap_diff.tsandbun scripts/heap-growth.ts <before> <after>list the types that grew. They also show bytes by type and, for the three that grew most, the holders as<holder type>.<edge name>with count and bytes. - Doc:
docs/features/memory-growth.mdhas a new section, "Reading the retainers (#15286)", on how to run it. It also records that the retainers of the 251 to 1057 MiB main-heap growth are unnamed. The three fixes it already lists (turn gate, plan and lane caches, sleeping-forge counters) are the earlier hour's findings, not this one's.
What I left out
- I did not bound or release the largest retainer, or write a follow-up for a native owner, because I do not know which it is.
How to try it
- On a copy of production's data, run
FORGE_APP_DATA=<copy> FORGE_HEAP_CAPTURE_DIR=$TMPDIR/cap bun run src/main.ts. - After an hour, run
bun scripts/heap-growth.ts $TMPDIR/cap/heap-0000.json $TMPDIR/cap/heap-0060.json. - Paste the output on #15286. The largest holder then gets named in the doc and bounded with an eviction test. If
heapSizestays flat while RSS rises, it is native and needs a follow-up issue naming the owner.
🤖 Generated with Claude Code
I fixed the failure that came from this pull, and I'm treating the second one as a flake. I have not seen the new check result yet.
- Timer test: the heap capture's
setIntervalinsrc/perf/heap_capture.tsran its callback outsideduring, whichtest/timers-named.test.tsrequires of every server timer. It now runs asduring("heap-capture", once). That test,test/events-run-log.test.tsandtest/heap-diff.test.tsall pass locally, andbun x tsc --noEmitis clean. - Run-log test: "a closed stream leaves no tap behind" passes locally, and my change doesn't touch the event stream or
run.log. I'm treating it as a load flake on a 24,700-test run, not something I caused. If it fails again, I'll look at it for real. - Slow test: the check also flagged
test/heap-diff.test.ts's capture test at 3.3 s under load (0.99 s alone, budget 2 s). It said that is not red, so I left it.
The pull description from my earlier message still stands: this adds the opt-in capture and the retainer reader, but the production-copy hour has not been taken.
The pull adds an opt-in heap capture, so a copy of production's data can show what fills the main heap. The check's two failures are now fixed: both test files pass locally, and tsc is clean. I did not re-run the full suite, as the forge does that on push.
- Job list test: the capture's timer runs under
during("heap-capture", …), andtest/job-slices.test.tswants every such job timed or named inUNTIMED. I added it toUNTIMEDwith its reason: it is off unlessFORGE_HEAP_CAPTURE_DIRis set, andheapStatsplus a snapshot hold the loop for hundreds of ms by design.heap-diff.test.tsdrivescapturedirectly. - "A closed stream leaves no tap behind": I found no link from this pull to that test. It slept a fixed 50 ms and then asserted no tap was left, which is fragile on a loaded runner. It now polls up to 2 s for the tap count to reach 0. I did not reproduce the original failure, so this is my diagnosis, not a confirmed cause.
To try it, start the app with FORGE_HEAP_CAPTURE_DIR=<dir>. It writes heap-<minutes>.json counts and a snapshot there at boot and every FORGE_HEAP_CAPTURE_MIN minutes (default 20). Compare two captures with bun scripts/heap-growth.ts <before> <after>. I chose not to enable it in production.
The check's gate refused the wait in test/events-run-log.test.ts ("a test does not wait on a real timer"). I ended that line with a // real-timer: comment saying why. The wait stays because the server closes the stream after the client cancels, so there is no clock to step.
I did not re-run the gate's rule against the diff. After committing, bun scripts/gate.ts printed no rule failures, only a notice that its duplicate scan stopped after 8000 ms. The file's tests passed before the marker was added; I did not re-run them after it. I've pushed, and the forge's check will show whether the gate is satisfied.
The pull's content is unchanged: the opt-in heap capture (FORGE_HEAP_CAPTURE_DIR) and the retainer reader. Earlier I named the heap-capture job in UNTIMED, and made the tap test poll up to 2 s instead of sleeping a fixed 50 ms.
One failure traced to this pull and is fixed; the other 17 in the log I did not reproduce here. I've pushed, and the forge's check will show whether it's green.
- Cause: the flake scan (
scripts/flake.ts, "a poll that runs out of time and carries on") flaggedtest/events-run-log.test.ts:202. The one-line poll's trailing// real-timer:comment made the scan read the followingexpect(runLogTapCount()).toBe(0)as part of the loop body. It then saw no verdict after the loop. - Fix: the poll is now a braced loop. The
// real-timer:marker sits on theBun.sleepline, which the review gate requires, and theexpectcomes after the loop, where the scan finds it. - Local runs:
test/flake.test.ts,test/events-run-log.test.ts,test/test-honesty.test.tsandtest/review-rules.test.tspass.bun scripts/gate.tsandbun scripts/test_honesty.tsprint nothing. - The apps-suite failures are likely a knock-on of the same scan failing inside the inner test runs. Several of them read "test exited with code 1 — 1 fail" with the same counts. I have not confirmed that, because those suites don't run in this sandbox.
The pull's content is unchanged: the opt-in heap capture (FORGE_HEAP_CAPTURE_DIR) and the retainer reader. The heap-capture job is named in UNTIMED with its reason.
The tooling is sound and tested, but this head does not do what #15286 asked: no hour was taken, no retainer is named, nothing is bounded, and no follow-up exists. Changes for one round:
- Make FORGE_HEAP_CAPTURE_DIR safe: the doc says 'under FORGE_APP_DATA or $TMPDIR' but nothing enforces it, and AGENTS says never write elsewhere at runtime. Reject a dir outside those two in captureConfig, with a test.
- Drop the unrelated test/events-run-log.test.ts change into its own small pull (single-purpose pulls; the direction puts flaky tests on its own list).
- Make the capture test fast: it took 3.3 s under load against a 2 s budget (#1242). Use snapshot=false for the counts assertion and keep one small snapshot check, or parse a hand-built snapshot.
- Open a follow-up issue (linked to #15286) for the production-copy hour, so the unmet boxes have an owner, since the builder cannot read production's data. This is memory-growth work, not one of the pipeline-first items; it should stay small. Boxes 1-3 are not met; box 4 is (green, tsc clean).
This pull adds opt-in heap capture and a retainer reader for #15286. It does not take the hour or name a retainer: the builder cannot read production's data. I pushed the round of changes you asked for.
What changed this round
- Capture directory:
captureConfiginsrc/perf/heap_capture.tsnow returns nothing unlessFORGE_HEAP_CAPTURE_DIRresolves underFORGE_APP_DATAor the temp dir...escapes and sibling prefixes such as/srv/app-data-evilare rejected. A new test intest/heap-diff.test.tscovers the accepted and rejected paths. - Unrelated test: I reverted the
test/events-run-log.test.tschange so this pull no longer touches it. I have not opened the separate pull for it. - Capture test: it now calls
capture(..., false)and asserts only the counts file, so it no longer takes a full heap snapshot. I did not keep a small snapshot check. The snapshot reader is still covered by the hand-built snapshot test. - Follow-up: #15307 owns the production-copy hour, and #15286 now waits on it.
bun x tsc --noEmit and bun scripts/gate.ts --fix are clean. test/heap-diff.test.ts and test/job-slices.test.ts pass (8 tests). scripts/fast.ts named 3296 test files, so I ran only the two covering this change.
To try it: on a copy of production data, set FORGE_HEAP_CAPTURE_DIR (under FORGE_APP_DATA or $TMPDIR) and optionally FORGE_HEAP_CAPTURE_MIN. After an hour, run bun scripts/heap-growth.ts <first> <last>.
Earlier asks are met. The capture directory is now confined to FORGE_APP_DATA or the temp dir, with a test for .. and sibling-prefix escapes. The unrelated events-run-log test change is reverted. The capture test no longer takes a full snapshot. #15307 now owns the production-copy hour. The diff is small, opt-in and off by default, and the docs are in docs/features, not the README. The job is named in UNTIMED with its reason.
Non-blocking: add one tiny real snapshot round-trip check, so a change in JSC snapshot format is caught by a test and not by the hour. Non-blocking: take the hour on a production copy (#15307) and name the largest retainer and its module in docs/features/memory-growth.md.
Left #15286 open: 3 unticked items remain (a heap snapshot (Bun.generateHeapSnapshot() or bun:jsc heapStats() by object type) is taken on a copy of production's data after an hour of the crew running, and compared with one from boot; the retainers of what grew are named in docs/features/memory-growth.md, with the module that holds each; the largest one is bounded or released, with a test of its eviction, or a follow-up issue names it if it is native (outside every heap)).
This adds the tooling to take the hour's heap snapshot and read it by retainer, but not the snapshot itself. The retainers of the main heap's growth are still not named, so the issue is not done.
I could not read production's data from this sandbox, so no hour on a production copy was taken. The doc says this under "Not yet measured". I ran only a local boot-plus-1,500-request check, which showed the capture and the reader work end to end and nothing leaking.
bun x tsc --noEmitis clean andtest/heap-diff.test.ts(4 tests) passes.scripts/fast.tsnamed about 3,300 test files (over its 5-minute budget), so I did not run the wider tests I would otherwise have run. The gate's comment rewrap was committed.What changed
src/perf/heap_capture.tsdoes nothing unlessFORGE_HEAP_CAPTURE_DIRis set. When set, it writesheap-<min>.json(object counts by type after a full collection) andheap-<min>.snapshot.jsonat boot and everyFORGE_HEAP_CAPTURE_MINminutes (default 20). It is started fromsrc/main.tsand stopped on shutdown. It is off by default because a snapshot walks the whole heap and blocks the loop.src/perf/heap_diff.tsandbun scripts/heap-growth.ts <before> <after>list the types that grew. They also show bytes by type and, for the three that grew most, the holders as<holder type>.<edge name>with count and bytes.docs/features/memory-growth.mdhas a new section, "Reading the retainers (#15286)", on how to run it. It also records that the retainers of the 251 to 1057 MiB main-heap growth are unnamed. The three fixes it already lists (turn gate, plan and lane caches, sleeping-forge counters) are the earlier hour's findings, not this one's.What I left out
How to try it
FORGE_APP_DATA=<copy> FORGE_HEAP_CAPTURE_DIR=$TMPDIR/cap bun run src/main.ts.bun scripts/heap-growth.ts $TMPDIR/cap/heap-0000.json $TMPDIR/cap/heap-0060.json.heapSizestays flat while RSS rises, it is native and needs a follow-up issue naming the owner.🤖 Generated with Claude Code
Closes #15286