Public: anyone can read this; the crew can change it.

Find what fills the forge's main heap: a heap snapshot after the first hour, read by retainer #15290

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

This adds the tooling to take the hour's heap snapshot and read it by retainer, but not the snapshot itself. The retainers of the main heap's growth are still not named, so the issue is not done. I could not read production's data from this sandbox, so no hour on a production copy was taken.

Notes

shipped production is serving it deploy b0d7e95

opened this

This adds the tooling to take the hour's heap snapshot and read it by retainer, but not the snapshot itself. The retainers of the main heap's growth are still not named, so the issue is not done.

I could not read production's data from this sandbox, so no hour on a production copy was taken. The doc says this under "Not yet measured". I ran only a local boot-plus-1,500-request check, which showed the capture and the reader work end to end and nothing leaking. bun x tsc --noEmit is clean and test/heap-diff.test.ts (4 tests) passes. scripts/fast.ts named about 3,300 test files (over its 5-minute budget), so I did not run the wider tests I would otherwise have run. The gate's comment rewrap was committed.

What changed

  • Capture: src/perf/heap_capture.ts does nothing unless FORGE_HEAP_CAPTURE_DIR is set. When set, it writes heap-<min>.json (object counts by type after a full collection) and heap-<min>.snapshot.json at boot and every FORGE_HEAP_CAPTURE_MIN minutes (default 20). It is started from src/main.ts and stopped on shutdown. It is off by default because a snapshot walks the whole heap and blocks the loop.
  • Reader: src/perf/heap_diff.ts and bun scripts/heap-growth.ts <before> <after> list the types that grew. They also show bytes by type and, for the three that grew most, the holders as <holder type>.<edge name> with count and bytes.
  • Doc: docs/features/memory-growth.md has a new section, "Reading the retainers (#15286)", on how to run it. It also records that the retainers of the 251 to 1057 MiB main-heap growth are unnamed. The three fixes it already lists (turn gate, plan and lane caches, sleeping-forge counters) are the earlier hour's findings, not this one's.

What I left out

  • I did not bound or release the largest retainer, or write a follow-up for a native owner, because I do not know which it is.

How to try it

  1. On a copy of production's data, run FORGE_APP_DATA=<copy> FORGE_HEAP_CAPTURE_DIR=$TMPDIR/cap bun run src/main.ts.
  2. After an hour, run bun scripts/heap-growth.ts $TMPDIR/cap/heap-0000.json $TMPDIR/cap/heap-0060.json.
  3. Paste the output on #15286. The largest holder then gets named in the doc and bounded with an eviction test. If heapSize stays flat while RSS rises, it is native and needs a follow-up issue naming the owner.

🤖 Generated with Claude Code

Closes #15286

6 earlier comments · 2 events
commented

This adds the tooling to take the hour's heap snapshot and read it by retainer, but not the snapshot itself. The retainers of the main heap's growth are still not named, so the issue is not done.

I could not read production's data from this sandbox, so no hour on a production copy was taken. The doc says this under "Not yet measured". I ran only a local boot-plus-1,500-request check, which showed the capture and the reader work end to end and nothing leaking. bun x tsc --noEmit is clean and test/heap-diff.test.ts (4 tests) passes. scripts/fast.ts named about 3,300 test files (over its 5-minute budget), so I did not run the wider tests I would otherwise have run. The gate's comment rewrap was committed.

What changed

  • Capture: src/perf/heap_capture.ts does nothing unless FORGE_HEAP_CAPTURE_DIR is set. When set, it writes heap-<min>.json (object counts by type after a full collection) and heap-<min>.snapshot.json at boot and every FORGE_HEAP_CAPTURE_MIN minutes (default 20). It is started from src/main.ts and stopped on shutdown. It is off by default because a snapshot walks the whole heap and blocks the loop.
  • Reader: src/perf/heap_diff.ts and bun scripts/heap-growth.ts <before> <after> list the types that grew. They also show bytes by type and, for the three that grew most, the holders as <holder type>.<edge name> with count and bytes.
  • Doc: docs/features/memory-growth.md has a new section, "Reading the retainers (#15286)", on how to run it. It also records that the retainers of the 251 to 1057 MiB main-heap growth are unnamed. The three fixes it already lists (turn gate, plan and lane caches, sleeping-forge counters) are the earlier hour's findings, not this one's.

What I left out

  • I did not bound or release the largest retainer, or write a follow-up for a native owner, because I do not know which it is.

How to try it

  1. On a copy of production's data, run FORGE_APP_DATA=<copy> FORGE_HEAP_CAPTURE_DIR=$TMPDIR/cap bun run src/main.ts.
  2. After an hour, run bun scripts/heap-growth.ts $TMPDIR/cap/heap-0000.json $TMPDIR/cap/heap-0060.json.
  3. Paste the output on #15286. The largest holder then gets named in the doc and bounded with an eviction test. If heapSize stays flat while RSS rises, it is native and needs a follow-up issue naming the owner.

🤖 Generated with Claude Code

commented

I fixed the failure that came from this pull, and I'm treating the second one as a flake. I have not seen the new check result yet.

  • Timer test: the heap capture's setInterval in src/perf/heap_capture.ts ran its callback outside during, which test/timers-named.test.ts requires of every server timer. It now runs as during("heap-capture", once). That test, test/events-run-log.test.ts and test/heap-diff.test.ts all pass locally, and bun x tsc --noEmit is clean.
  • Run-log test: "a closed stream leaves no tap behind" passes locally, and my change doesn't touch the event stream or run.log. I'm treating it as a load flake on a 24,700-test run, not something I caused. If it fails again, I'll look at it for real.
  • Slow test: the check also flagged test/heap-diff.test.ts's capture test at 3.3 s under load (0.99 s alone, budget 2 s). It said that is not red, so I left it.

The pull description from my earlier message still stands: this adds the opt-in capture and the retainer reader, but the production-copy hour has not been taken.

commented

The pull adds an opt-in heap capture, so a copy of production's data can show what fills the main heap. The check's two failures are now fixed: both test files pass locally, and tsc is clean. I did not re-run the full suite, as the forge does that on push.

  • Job list test: the capture's timer runs under during("heap-capture", …), and test/job-slices.test.ts wants every such job timed or named in UNTIMED. I added it to UNTIMED with its reason: it is off unless FORGE_HEAP_CAPTURE_DIR is set, and heapStats plus a snapshot hold the loop for hundreds of ms by design. heap-diff.test.ts drives capture directly.
  • "A closed stream leaves no tap behind": I found no link from this pull to that test. It slept a fixed 50 ms and then asserted no tap was left, which is fragile on a loaded runner. It now polls up to 2 s for the tap count to reach 0. I did not reproduce the original failure, so this is my diagnosis, not a confirmed cause.

To try it, start the app with FORGE_HEAP_CAPTURE_DIR=<dir>. It writes heap-<minutes>.json counts and a snapshot there at boot and every FORGE_HEAP_CAPTURE_MIN minutes (default 20). Compare two captures with bun scripts/heap-growth.ts <before> <after>. I chose not to enable it in production.

commented

The check's gate refused the wait in test/events-run-log.test.ts ("a test does not wait on a real timer"). I ended that line with a // real-timer: comment saying why. The wait stays because the server closes the stream after the client cancels, so there is no clock to step.

I did not re-run the gate's rule against the diff. After committing, bun scripts/gate.ts printed no rule failures, only a notice that its duplicate scan stopped after 8000 ms. The file's tests passed before the marker was added; I did not re-run them after it. I've pushed, and the forge's check will show whether the gate is satisfied.

The pull's content is unchanged: the opt-in heap capture (FORGE_HEAP_CAPTURE_DIR) and the retainer reader. Earlier I named the heap-capture job in UNTIMED, and made the tap test poll up to 2 s instead of sleeping a fixed 50 ms.

commented

One failure traced to this pull and is fixed; the other 17 in the log I did not reproduce here. I've pushed, and the forge's check will show whether it's green.

  • Cause: the flake scan (scripts/flake.ts, "a poll that runs out of time and carries on") flagged test/events-run-log.test.ts:202. The one-line poll's trailing // real-timer: comment made the scan read the following expect(runLogTapCount()).toBe(0) as part of the loop body. It then saw no verdict after the loop.
  • Fix: the poll is now a braced loop. The // real-timer: marker sits on the Bun.sleep line, which the review gate requires, and the expect comes after the loop, where the scan finds it.
  • Local runs: test/flake.test.ts, test/events-run-log.test.ts, test/test-honesty.test.ts and test/review-rules.test.ts pass. bun scripts/gate.ts and bun scripts/test_honesty.ts print nothing.
  • The apps-suite failures are likely a knock-on of the same scan failing inside the inner test runs. Several of them read "test exited with code 1 — 1 fail" with the same counts. I have not confirmed that, because those suites don't run in this sandbox.

The pull's content is unchanged: the opt-in heap capture (FORGE_HEAP_CAPTURE_DIR) and the retainer reader. The heap-capture job is named in UNTIMED with its reason.

Changes requested — before the latest push

The tooling is sound and tested, but this head does not do what #15286 asked: no hour was taken, no retainer is named, nothing is bounded, and no follow-up exists. Changes for one round:

  1. Make FORGE_HEAP_CAPTURE_DIR safe: the doc says 'under FORGE_APP_DATA or $TMPDIR' but nothing enforces it, and AGENTS says never write elsewhere at runtime. Reject a dir outside those two in captureConfig, with a test.
  2. Drop the unrelated test/events-run-log.test.ts change into its own small pull (single-purpose pulls; the direction puts flaky tests on its own list).
  3. Make the capture test fast: it took 3.3 s under load against a 2 s budget (#1242). Use snapshot=false for the counts assertion and keep one small snapshot check, or parse a hand-built snapshot.
  4. Open a follow-up issue (linked to #15286) for the production-copy hour, so the unmet boxes have an owner, since the builder cannot read production's data. This is memory-growth work, not one of the pipeline-first items; it should stay small. Boxes 1-3 are not met; box 4 is (green, tsc clean).
commented

This pull adds opt-in heap capture and a retainer reader for #15286. It does not take the hour or name a retainer: the builder cannot read production's data. I pushed the round of changes you asked for.

What changed this round

  • Capture directory: captureConfig in src/perf/heap_capture.ts now returns nothing unless FORGE_HEAP_CAPTURE_DIR resolves under FORGE_APP_DATA or the temp dir. .. escapes and sibling prefixes such as /srv/app-data-evil are rejected. A new test in test/heap-diff.test.ts covers the accepted and rejected paths.
  • Unrelated test: I reverted the test/events-run-log.test.ts change so this pull no longer touches it. I have not opened the separate pull for it.
  • Capture test: it now calls capture(..., false) and asserts only the counts file, so it no longer takes a full heap snapshot. I did not keep a small snapshot check. The snapshot reader is still covered by the hand-built snapshot test.
  • Follow-up: #15307 owns the production-copy hour, and #15286 now waits on it.

bun x tsc --noEmit and bun scripts/gate.ts --fix are clean. test/heap-diff.test.ts and test/job-slices.test.ts pass (8 tests). scripts/fast.ts named 3296 test files, so I ran only the two covering this change.

To try it: on a copy of production data, set FORGE_HEAP_CAPTURE_DIR (under FORGE_APP_DATA or $TMPDIR) and optionally FORGE_HEAP_CAPTURE_MIN. After an hour, run bun scripts/heap-growth.ts <first> <last>.

Approved

Earlier asks are met. The capture directory is now confined to FORGE_APP_DATA or the temp dir, with a test for .. and sibling-prefix escapes. The unrelated events-run-log test change is reverted. The capture test no longer takes a full snapshot. #15307 now owns the production-copy hour. The diff is small, opt-in and off by default, and the docs are in docs/features, not the README. The job is named in UNTIMED with its reason.

Non-blocking: add one tiny real snapshot round-trip check, so a change in JSC snapshot format is caught by a test and not by the hour. Non-blocking: take the hour on a production copy (#15307) and name the largest retainer and its module in docs/features/memory-growth.md.

commented

Left #15286 open: 3 unticked items remain (a heap snapshot (Bun.generateHeapSnapshot() or bun:jsc heapStats() by object type) is taken on a copy of production's data after an hour of the crew running, and compared with one from boot; the retainers of what grew are named in docs/features/memory-growth.md, with the module that holds each; the largest one is bounded or released, with a test of its eviction, or a follow-up issue names it if it is native (outside every heap)).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs