Public: anyone can read this; the crew can change it.

The admin profiler leaks memory until the forge crashes: it brought production down for 2.5 minutes #12456

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The profiler's traces are now drained while it runs, the endpoint is refused on a serving forge, and the forge watches its own memory. I did not re-run the three-minute table or write the ten-minute flat-memory test, so "flat in both rows" is not shown yet. I also did not call keepplan or estimate.

Notes

shipped production is serving it deploy e5f3b40

opened this

The profiler's traces are now drained while it runs, the endpoint is refused on a serving forge, and the forge watches its own memory. I did not re-run the three-minute table or write the ten-minute flat-memory test, so "flat in both rows" is not shown yet. I also did not call keep_plan or estimate.

What changed

  • Profile window: Bun 1.3.14's bun:jsc has no stop for the sampler, so it can't be stopped. src/perf/sample.ts now discards the sampler's traces every 2 s on a named, unref'd timer for as long as it stays attached. The note in the response says so.
  • Guard: GET /api/v1/admin/profile answers 403 with one sentence unless FORGE_PROFILE=1. The sentence says the sampler can't be stopped and once brought production down, and that the stage rig allows it with FORGE_PROFILE=1. The guard stays until the table is re-run flat.
  • Memory watch: new src/perf/memory_growth.ts. main.ts records the process's resident memory every 30 s and keeps three hours. "Usual" is the median of readings older than 10 minutes, so a young process says nothing. Past 1.5× usual, vitals raises a memory-growth needs-you line with the current size, the multiple, usual, and when growth started (the last time it was within 1.15× of usual).

Tests

  • Added test/memory-growth.test.ts: steady memory raises nothing, growth names its size and start time, a young process raises nothing, and FORGE_PROFILE gates profiling.
  • Added a case in test/sample.test.ts run in a child process: after a sample, a 1.5 s busy loop leaves few traces, so the drain is working.
  • Passing: those two files plus the vitals, needs and home tests I ran, and bun x tsc --noEmit.
  • Not run: bun scripts/gate.ts printed nothing I could use, and scripts/fast.ts said the diff was too broad to name tests. I picked the covering tests by hand.
  • One stray run: because of that fast.ts result, I accidentally started a bare bun test and killed it partway. Don't read anything into it.
  • I added no test for the 403 through the HTTP route.

To try it: call GET /api/v1/admin/profile?seconds=1 as an admin. You get the 403 reason. On a stage rig started with FORGE_PROFILE=1 it samples, and process memory should stay flat afterwards under load.

Left out

  • The ten-minute Linux flat-memory test.
  • Moving the profile into a Worker, which can't sample the serving thread.

🤖 Generated with Claude Code

Closes #12455

6 earlier comments · 1 event
commented

The profiler's traces are now drained while it runs, the endpoint is refused on a serving forge, and the forge watches its own memory. I did not re-run the three-minute table or write the ten-minute flat-memory test, so "flat in both rows" is not shown yet. I also did not call keep_plan or estimate.

What changed

  • Profile window: Bun 1.3.14's bun:jsc has no stop for the sampler, so it can't be stopped. src/perf/sample.ts now discards the sampler's traces every 2 s on a named, unref'd timer for as long as it stays attached. The note in the response says so.
  • Guard: GET /api/v1/admin/profile answers 403 with one sentence unless FORGE_PROFILE=1. The sentence says the sampler can't be stopped and once brought production down, and that the stage rig allows it with FORGE_PROFILE=1. The guard stays until the table is re-run flat.
  • Memory watch: new src/perf/memory_growth.ts. main.ts records the process's resident memory every 30 s and keeps three hours. "Usual" is the median of readings older than 10 minutes, so a young process says nothing. Past 1.5× usual, vitals raises a memory-growth needs-you line with the current size, the multiple, usual, and when growth started (the last time it was within 1.15× of usual).

Tests

  • Added test/memory-growth.test.ts: steady memory raises nothing, growth names its size and start time, a young process raises nothing, and FORGE_PROFILE gates profiling.
  • Added a case in test/sample.test.ts run in a child process: after a sample, a 1.5 s busy loop leaves few traces, so the drain is working.
  • Passing: those two files plus the vitals, needs and home tests I ran, and bun x tsc --noEmit.
  • Not run: bun scripts/gate.ts printed nothing I could use, and scripts/fast.ts said the diff was too broad to name tests. I picked the covering tests by hand.
  • One stray run: because of that fast.ts result, I accidentally started a bare bun test and killed it partway. Don't read anything into it.
  • I added no test for the 403 through the HTTP route.

To try it: call GET /api/v1/admin/profile?seconds=1 as an admin. You get the 403 reason. On a stage rig started with FORGE_PROFILE=1 it samples, and process memory should stay flat afterwards under load.

Left out

  • The ten-minute Linux flat-memory test.
  • Moving the profile into a Worker, which can't sample the serving thread.

🤖 Generated with Claude Code

commented

The check on this pull's head fails, and it is not the preview: the deploy log you were shown is a success. These are the failing lines from check run 01m7eq1wx0080j40swyg9e5bmc (ended 2026-11-15T04:22), quoted from its log:

gate: rule "a comment line is wrapped at 100 columns": src/perf/sample.ts:50 is 118 long; rewrap it
gate: rule "a comment line is wrapped at 100 columns": src/perf/sample.ts:56 is 116 long; rewrap it
gate: rule "a test does not wait on a real timer": test/sample.test.ts:45 sleeps; inject a clock and step it, or end the line with `// real-timer: <why>`

Fix exactly these, then push. Nothing else is being asked.

commented

I fixed the three gate failures and pushed. bun scripts/gate.ts and bun x tsc --noEmit now pass, and test/sample.test.ts passed (3 tests) before the commit. I did not re-run it after the commit, and the forge's check has not run on the new head yet.

  • Comment lines: the two long doc comments in src/perf/sample.ts (lines 50 and 56) are rewrapped under 100 columns.
  • Real timer: the 100 ms wait in test/sample.test.ts now ends with // real-timer: the child process must let its real drain timer fire. I used the marker rather than a fake clock because the wait runs inside a child bun -e process, so the test has no clock to inject. The wait is there so the real 2 s drain timer fires.

The gate only sees committed changes. It kept reporting the same lines until I committed.

Nothing else changed.

commented

The pushed head merges main into the branch and fixes the three check failures. They were the new module-state rule, the timer-naming rule, and the dead-code rule. bun scripts/gate.ts passes and timers-named, unused-exports, module-state, sample and vitals pass (36 tests). The forge's full check hasn't run on this head yet.

bun x tsc --noEmit is not clean here: it reports 'latin1' encoding errors in unrelated src/web/render files and test/render-xlsx.test.ts. It reports nothing in the files I touched, and I did not fix the others.

  • Duplicate removed: main already has src/perf/memory_watch.ts, which raises the forge's own memory-growth need through vitals.ts. My memory_growth.ts did the same job, so I deleted it, its test and its main.ts hooks, and took main's vitals.ts. That also clears the dead-code failure (resetGrowthForTest) and the module-state failure (its history array).
  • Timer rule: the profiler's drain timer in src/perf/sample.ts now runs inside during("profiler-drain", …). The check's other flagged timer was the one in main.ts that I deleted.
  • Bug I found: the drain also ran during a sampling window. A 10 s sample would have read back only the last 2 s of traces. The drain now skips while a window is being sampled, so a window keeps its traces. The cost is that traces pile up for the length of a window, at most 30 s.
  • Slow test: the drain test took 2.61 s against the 2 s budget. sampleStacks takes an optional drain interval, and the test uses 200 ms with a shorter spin, so it now runs in about 1.7 s. Its wait still carries the // real-timer: marker.

I did not fix the unrelated "no longer dead" exports (forgetFrozenItems, useRunner and others) that unused-exports.test.ts prints; they come from main.

commented

The check failed because test/job-slices.test.ts requires every job run under during to be either timed or listed in UNTIMED, and the new profiler-drain job was in neither. I added it to UNTIMED in that file with a reason: it drains the sampling profiler's stack traces on a timer in src/perf/sample.ts, and the test fixture doesn't sample anything.

bun test test/job-slices.test.ts now passes (3 tests) and bun x tsc --noEmit is clean. I didn't run the rest of the suite or bun scripts/gate.ts. The fix is pushed as 4d8a2f611, so the forge will run the full check again.

This is the pull description: the pull makes the admin profiler's drain timer a named job and lets a sampling window keep its traces. The change since the last push only registers that job in the job-slices test.

Changes requested — before the latest push

Security-wise the guard is sound (it sits after the admin check and fails closed), but changes are needed:

  1. FIRST: the new behaviour has no test. The 403 from GET /api/v1/admin/profile when FORGE_PROFILE is not 1, and profilingAllowed()/REFUSED, are untested (the earlier test file for them was deleted in the merge). Add a route test: an admin without FORGE_PROFILE gets 403 code refused with the REFUSED sentence, a guest still gets the admin denial, and profilingAllowed({FORGE_PROFILE:"1"}) is true while {} and "0" are false. It must fail without the guard.
  2. src/perf/sample.ts: there is a stray double blank line after profilingAllowed; remove it. Also the profilingAllowed env default reads process.env at call time; fine, but keep it a single place the route uses.
  3. The drain test depends on a real-timer wait of 100 ms against a 200 ms interval plus a 600 ms spin in a child process, which can flake under load; assert on draining() plus a bounded poll, not one fixed sleep.
  4. Document the FORGE_PROFILE switch in docs/features/ (not README) so an admin who gets the 403 can find how to enable it on the stage rig.
  5. The description says the memory watch was removed in favour of main's memory_watch; make sure the pull text matches the diff (it is now only drain + guard).
commented

The admin profiler can no longer take the forge down: Bun can't stop the sampling profiler once started, so it is now refused on a forge that serves people, and its traces are drained while it's attached.

What changed

  • Guard: GET /api/v1/admin/profile returns 403 with code refused and the REFUSED sentence unless FORGE_PROFILE=1. The check sits after the admin check, via profilingAllowed() in src/perf/sample.ts, the only place the route reads the switch.
  • Drain: once a sample has run, a named profiler-drain job throws the traces away every 2 s. A window that is being sampled keeps its traces until it ends. The job is listed in UNTIMED in test/job-slices.test.ts.
  • Memory watch: the pull no longer touches it. The earlier duplicate was dropped in the merge with main, so the pull is only the guard and the drain.

Review fixes

  • Guard test: new test/profile-guard.test.ts covers an admin without the switch getting 403 refused with the sentence, a guest getting the plain admin denial, and profilingAllowed for "1", {} and "0". I checked it fails with the guard disabled.
  • Drain test: test/sample.test.ts now polls up to 5 s for the traces to drain instead of one 100 ms sleep.
  • Blank line: the stray double blank line in src/perf/sample.ts is gone.
  • Docs: the FORGE_PROFILE=1 switch is documented in docs/features/performance-watcher.md, not the README.

Checks: test/sample.test.ts, test/profile-guard.test.ts and test/job-slices.test.ts pass (9 tests), bun x tsc --noEmit is clean, and bun scripts/gate.ts exits 0. I didn't run the full suite; the forge's check does that on push.

Left out: stopping the profiler. Bun has no API for that, so after a sample it stays attached until the process restarts.

To try it: run the forge with FORGE_PROFILE=1, then call /api/v1/admin/profile?seconds=3 with an admin token. Without the variable it answers 403.

commented

The check failed on two gate rules, and both are fixed and pushed as 529334f2b. bun scripts/gate.ts now exits 0, and test/sample.test.ts and test/profile-guard.test.ts pass (6 tests). I haven't re-run tsc or the full check; the forge does that on this push.

  • Real-timer rule: the 50 ms poll sleep in the drain test (test/sample.test.ts:49) now ends with // real-timer: the child process must let its real drain timer fire. The sleep is unchanged: the test needs the child's real drain timer to fire, so a stepped clock wouldn't exercise it.
  • Comment-width rule: the header comment in test/profile-guard.test.ts was 132 columns and is now rewrapped under 100.

The pull still does what the earlier description said:

  • Guard: the admin profile route answers 403 refused unless FORGE_PROFILE=1.
  • Drain: the profiler's traces are drained every 2 s by a named profiler-drain job, and a window being sampled keeps its traces until it ends.
  • Tests and docs: the new tests cover the 403, the guest denial and profilingAllowed, and docs/features/performance-watcher.md documents FORGE_PROFILE=1.
  • Not done: stopping the profiler, because Bun has no API for it. It stays attached until the process restarts.

To try it, start the forge with FORGE_PROFILE=1 and call /api/v1/admin/profile?seconds=3 with an admin token. Without the variable it answers 403.

Approved

Approved. Since my last review (4d8a2f611b74): the admin profile route now answers 403 refused with the REFUSED sentence unless FORGE_PROFILE=1, checked after the admin check so a guest still gets the plain denial. test/profile-guard.test.ts covers the admin 403, the guest denial and profilingAllowed. The drain test polls with a bound instead of one fixed sleep, and the feature doc (not README) documents the switch. Every earlier ask is met, and the new lines raise no security issues. Non-blocking: traces still pile up for up to 30 s while a window is sampled; the doc could state that bound.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs