Public: anyone can read this; the crew can change it.
The admin profiler leaks memory until the forge crashes: it brought production down for 2.5 minutes #12456
Check · Review · Land · Live done — Landed and live in production.
The profiler's traces are now drained while it runs, the endpoint is refused on a serving forge, and the forge watches its own memory. I did not re-run the three-minute table or write the ten-minute flat-memory test, so "flat in both rows" is not shown yet. I also did not call keepplan or estimate.
shipped production is serving it deploy e5f3b40
6 earlier comments · 1 event
The profiler's traces are now drained while it runs, the endpoint is refused on a serving forge, and the forge watches its own memory. I did not re-run the three-minute table or write the ten-minute flat-memory test, so "flat in both rows" is not shown yet. I also did not call keep_plan or estimate.
What changed
- Profile window: Bun 1.3.14's
bun:jschas no stop for the sampler, so it can't be stopped.src/perf/sample.tsnow discards the sampler's traces every 2 s on a named, unref'd timer for as long as it stays attached. The note in the response says so. - Guard:
GET /api/v1/admin/profileanswers 403 with one sentence unlessFORGE_PROFILE=1. The sentence says the sampler can't be stopped and once brought production down, and that the stage rig allows it withFORGE_PROFILE=1. The guard stays until the table is re-run flat. - Memory watch: new
src/perf/memory_growth.ts.main.tsrecords the process's resident memory every 30 s and keeps three hours. "Usual" is the median of readings older than 10 minutes, so a young process says nothing. Past 1.5× usual,vitalsraises amemory-growthneeds-you line with the current size, the multiple, usual, and when growth started (the last time it was within 1.15× of usual).
Tests
- Added
test/memory-growth.test.ts: steady memory raises nothing, growth names its size and start time, a young process raises nothing, andFORGE_PROFILEgates profiling. - Added a case in
test/sample.test.tsrun in a child process: after a sample, a 1.5 s busy loop leaves few traces, so the drain is working. - Passing: those two files plus the vitals, needs and home tests I ran, and
bun x tsc --noEmit. - Not run:
bun scripts/gate.tsprinted nothing I could use, andscripts/fast.tssaid the diff was too broad to name tests. I picked the covering tests by hand. - One stray run: because of that
fast.tsresult, I accidentally started a barebun testand killed it partway. Don't read anything into it. - I added no test for the 403 through the HTTP route.
To try it: call GET /api/v1/admin/profile?seconds=1 as an admin. You get the 403 reason. On a stage rig started with FORGE_PROFILE=1 it samples, and process memory should stay flat afterwards under load.
Left out
- The ten-minute Linux flat-memory test.
- Moving the profile into a Worker, which can't sample the serving thread.
🤖 Generated with Claude Code
The check on this pull's head fails, and it is not the preview: the deploy log you were shown is a success. These are the failing lines from check run 01m7eq1wx0080j40swyg9e5bmc (ended 2026-11-15T04:22), quoted from its log:
gate: rule "a comment line is wrapped at 100 columns": src/perf/sample.ts:50 is 118 long; rewrap it
gate: rule "a comment line is wrapped at 100 columns": src/perf/sample.ts:56 is 116 long; rewrap it
gate: rule "a test does not wait on a real timer": test/sample.test.ts:45 sleeps; inject a clock and step it, or end the line with `// real-timer: <why>`
Fix exactly these, then push. Nothing else is being asked.
I fixed the three gate failures and pushed. bun scripts/gate.ts and bun x tsc --noEmit now pass, and test/sample.test.ts passed (3 tests) before the commit. I did not re-run it after the commit, and the forge's check has not run on the new head yet.
- Comment lines: the two long doc comments in
src/perf/sample.ts(lines 50 and 56) are rewrapped under 100 columns. - Real timer: the 100 ms wait in
test/sample.test.tsnow ends with// real-timer: the child process must let its real drain timer fire. I used the marker rather than a fake clock because the wait runs inside a childbun -eprocess, so the test has no clock to inject. The wait is there so the real 2 s drain timer fires.
The gate only sees committed changes. It kept reporting the same lines until I committed.
Nothing else changed.
The pushed head merges main into the branch and fixes the three check failures. They were the new module-state rule, the timer-naming rule, and the dead-code rule. bun scripts/gate.ts passes and timers-named, unused-exports, module-state, sample and vitals pass (36 tests). The forge's full check hasn't run on this head yet.
bun x tsc --noEmit is not clean here: it reports 'latin1' encoding errors in unrelated src/web/render files and test/render-xlsx.test.ts. It reports nothing in the files I touched, and I did not fix the others.
- Duplicate removed: main already has
src/perf/memory_watch.ts, which raises the forge's own memory-growth need throughvitals.ts. Mymemory_growth.tsdid the same job, so I deleted it, its test and itsmain.tshooks, and took main'svitals.ts. That also clears the dead-code failure (resetGrowthForTest) and the module-state failure (itshistoryarray). - Timer rule: the profiler's drain timer in
src/perf/sample.tsnow runs insideduring("profiler-drain", …). The check's other flagged timer was the one inmain.tsthat I deleted. - Bug I found: the drain also ran during a sampling window. A 10 s sample would have read back only the last 2 s of traces. The drain now skips while a window is being sampled, so a window keeps its traces. The cost is that traces pile up for the length of a window, at most 30 s.
- Slow test: the drain test took 2.61 s against the 2 s budget.
sampleStackstakes an optional drain interval, and the test uses 200 ms with a shorter spin, so it now runs in about 1.7 s. Its wait still carries the// real-timer:marker.
I did not fix the unrelated "no longer dead" exports (forgetFrozenItems, useRunner and others) that unused-exports.test.ts prints; they come from main.
The check failed because test/job-slices.test.ts requires every job run under during to be either timed or listed in UNTIMED, and the new profiler-drain job was in neither. I added it to UNTIMED in that file with a reason: it drains the sampling profiler's stack traces on a timer in src/perf/sample.ts, and the test fixture doesn't sample anything.
bun test test/job-slices.test.ts now passes (3 tests) and bun x tsc --noEmit is clean. I didn't run the rest of the suite or bun scripts/gate.ts. The fix is pushed as 4d8a2f611, so the forge will run the full check again.
This is the pull description: the pull makes the admin profiler's drain timer a named job and lets a sampling window keep its traces. The change since the last push only registers that job in the job-slices test.
Security-wise the guard is sound (it sits after the admin check and fails closed), but changes are needed:
- FIRST: the new behaviour has no test. The 403 from GET /api/v1/admin/profile when FORGE_PROFILE is not 1, and profilingAllowed()/REFUSED, are untested (the earlier test file for them was deleted in the merge). Add a route test: an admin without FORGE_PROFILE gets 403 code refused with the REFUSED sentence, a guest still gets the admin denial, and profilingAllowed({FORGE_PROFILE:"1"}) is true while {} and "0" are false. It must fail without the guard.
- src/perf/sample.ts: there is a stray double blank line after profilingAllowed; remove it. Also the profilingAllowed env default reads process.env at call time; fine, but keep it a single place the route uses.
- The drain test depends on a real-timer wait of 100 ms against a 200 ms interval plus a 600 ms spin in a child process, which can flake under load; assert on draining() plus a bounded poll, not one fixed sleep.
- Document the FORGE_PROFILE switch in docs/features/ (not README) so an admin who gets the 403 can find how to enable it on the stage rig.
- The description says the memory watch was removed in favour of main's memory_watch; make sure the pull text matches the diff (it is now only drain + guard).
The admin profiler can no longer take the forge down: Bun can't stop the sampling profiler once started, so it is now refused on a forge that serves people, and its traces are drained while it's attached.
What changed
- Guard:
GET /api/v1/admin/profilereturns 403 with coderefusedand theREFUSEDsentence unlessFORGE_PROFILE=1. The check sits after the admin check, viaprofilingAllowed()insrc/perf/sample.ts, the only place the route reads the switch. - Drain: once a sample has run, a named
profiler-drainjob throws the traces away every 2 s. A window that is being sampled keeps its traces until it ends. The job is listed inUNTIMEDintest/job-slices.test.ts. - Memory watch: the pull no longer touches it. The earlier duplicate was dropped in the merge with main, so the pull is only the guard and the drain.
Review fixes
- Guard test: new
test/profile-guard.test.tscovers an admin without the switch getting 403refusedwith the sentence, a guest getting the plain admin denial, andprofilingAllowedfor"1",{}and"0". I checked it fails with the guard disabled. - Drain test:
test/sample.test.tsnow polls up to 5 s for the traces to drain instead of one 100 ms sleep. - Blank line: the stray double blank line in
src/perf/sample.tsis gone. - Docs: the
FORGE_PROFILE=1switch is documented indocs/features/performance-watcher.md, not the README.
Checks: test/sample.test.ts, test/profile-guard.test.ts and test/job-slices.test.ts pass (9 tests), bun x tsc --noEmit is clean, and bun scripts/gate.ts exits 0. I didn't run the full suite; the forge's check does that on push.
Left out: stopping the profiler. Bun has no API for that, so after a sample it stays attached until the process restarts.
To try it: run the forge with FORGE_PROFILE=1, then call /api/v1/admin/profile?seconds=3 with an admin token. Without the variable it answers 403.
The check failed on two gate rules, and both are fixed and pushed as 529334f2b. bun scripts/gate.ts now exits 0, and test/sample.test.ts and test/profile-guard.test.ts pass (6 tests). I haven't re-run tsc or the full check; the forge does that on this push.
- Real-timer rule: the 50 ms poll sleep in the drain test (
test/sample.test.ts:49) now ends with// real-timer: the child process must let its real drain timer fire. The sleep is unchanged: the test needs the child's real drain timer to fire, so a stepped clock wouldn't exercise it. - Comment-width rule: the header comment in
test/profile-guard.test.tswas 132 columns and is now rewrapped under 100.
The pull still does what the earlier description said:
- Guard: the admin profile route answers 403
refusedunlessFORGE_PROFILE=1. - Drain: the profiler's traces are drained every 2 s by a named
profiler-drainjob, and a window being sampled keeps its traces until it ends. - Tests and docs: the new tests cover the 403, the guest denial and
profilingAllowed, anddocs/features/performance-watcher.mddocumentsFORGE_PROFILE=1. - Not done: stopping the profiler, because Bun has no API for it. It stays attached until the process restarts.
To try it, start the forge with FORGE_PROFILE=1 and call /api/v1/admin/profile?seconds=3 with an admin token. Without the variable it answers 403.
Approved. Since my last review (4d8a2f611b74): the admin profile route now answers 403 refused with the REFUSED sentence unless FORGE_PROFILE=1, checked after the admin check so a guest still gets the plain denial. test/profile-guard.test.ts covers the admin 403, the guest denial and profilingAllowed. The drain test polls with a bound instead of one fixed sleep, and the feature doc (not README) documents the switch. Every earlier ask is met, and the new lines raise no security issues. Non-blocking: traces still pile up for up to 30 s while a window is sampled; the doc could state that bound.
The profiler's traces are now drained while it runs, the endpoint is refused on a serving forge, and the forge watches its own memory. I did not re-run the three-minute table or write the ten-minute flat-memory test, so "flat in both rows" is not shown yet. I also did not call
keep_planorestimate.What changed
bun:jschas no stop for the sampler, so it can't be stopped.src/perf/sample.tsnow discards the sampler's traces every 2 s on a named, unref'd timer for as long as it stays attached. The note in the response says so.GET /api/v1/admin/profileanswers 403 with one sentence unlessFORGE_PROFILE=1. The sentence says the sampler can't be stopped and once brought production down, and that the stage rig allows it withFORGE_PROFILE=1. The guard stays until the table is re-run flat.src/perf/memory_growth.ts.main.tsrecords the process's resident memory every 30 s and keeps three hours. "Usual" is the median of readings older than 10 minutes, so a young process says nothing. Past 1.5× usual,vitalsraises amemory-growthneeds-you line with the current size, the multiple, usual, and when growth started (the last time it was within 1.15× of usual).Tests
test/memory-growth.test.ts: steady memory raises nothing, growth names its size and start time, a young process raises nothing, andFORGE_PROFILEgates profiling.test/sample.test.tsrun in a child process: after a sample, a 1.5 s busy loop leaves few traces, so the drain is working.bun x tsc --noEmit.bun scripts/gate.tsprinted nothing I could use, andscripts/fast.tssaid the diff was too broad to name tests. I picked the covering tests by hand.fast.tsresult, I accidentally started a barebun testand killed it partway. Don't read anything into it.To try it: call
GET /api/v1/admin/profile?seconds=1as an admin. You get the 403 reason. On a stage rig started withFORGE_PROFILE=1it samples, and process memory should stay flat afterwards under load.Left out
🤖 Generated with Claude Code
Closes #12455