Public: anyone can read this; the crew can change it.
The page thread stops for half a second about five times a minute with no request running: its heap grows 5 MB a second and is collected in long pauses #13925
Check · Review · Land · Live done — Landed and live in production.
This pull does not fix the stalls. It adds the measurement that names the allocator, and I could not reproduce the growth here. What I found.
shipped production is serving it deploy 4adf612
7 earlier comments · 1 event
This pull does not fix the stalls. It adds the measurement that names the allocator, and I could not reproduce the growth here.
What I found. I seeded a forge (300 repos, 5,000 issues, 20,000 comments, 25,000 events), booted src/main.ts over it with FORGE_ENGINE=worker, and asked it nothing for about two minutes. The serving thread's heap stayed flat at about 27 MiB after boot. No job grew it by even 1 MiB a minute, and I saw no stall after boot. So the 5 MiB/s climb needs something this rig lacks: hosted forges, a running crew, or open live-update streams. I could not seed those, and the stage rig isn't reachable from my sandbox. Reading the code, I found no timer that allocates visibly. The per-second cell watch already reads its process trees off-thread.
What changed:
src/perf/job_alloc.ts(new):during(src/perf/jobs.ts) now reads the JS heap size on either side of each job's synchronous slice and credits any growth to the job's name, up to 128 names.- Minute memory line:
src/boards/memory.tslogs the result asmemory allocated by jobs: timers 84.0 MiB in 60 slices, …when a job grew the heap by at least 1 MiB, then empties the count. One production minute with this deployed should name the culprit. - Limits of the numbers: they are a floor. A collection inside a slice hides growth, and allocation after an
awaitis not seen. bench/idle.ts(new): the idle budget as a script. It boots the forge over a seeded directory, waits (--secs, default 120), parses the forge's own log, and exits 1 on heap growth over 20 MiB a minute or any stall over 100 ms (the first 20 s of boot are ignored). Runbun run bench/seed.ts --data Dfirst, thenbun run bench/idle.ts --data D. On my seeded rig it passes.test/job-alloc.test.ts(new): covers the attribution.AGENTS.md: one line for the new in-memory state.
Not done:
- Ask 2 (move the allocator to the Worker or stream it). There is nothing named to move yet.
- The "fewer than five stalls in ten minutes" done-condition. Nothing here claims to meet it.
- The 2-minute budget as a
bun test. It is a script, not a suite test, because the run is too long.
To try it: deploy, read the memory allocated by jobs: lines in the production log, and open a follow-up on whichever timer leads. bun test test/job-alloc.test.ts runs the unit test.
I ran bun scripts/gate.ts --fix (it rewrapped comments in 2 files, which are committed), bun x tsc --noEmit and the tests that cover the change. I did not run the whole suite; the forge's check does that on the push.
This push fixes the two gate failures from the check. I did not run the tests, since this change only restores comments.
- Comment rewraps: I restored the three comments in
src/boards/memory.tsto their original one-line form, so the diff there is now only the newjob_allocimport and the allocation log line in the sampler. The gate no longer reports the "diff is the change" rule. It still prints a note about comment lines over 100 columns in that file; the note says nothing needs doing and the nightly rewrap handles it on main. - Try line: the new commit message has a
Try:line: open the preview and watch its log for the once-a-minute allocation line after the memory line.
The gate and tsc --noEmit came back clean before I pushed.
The check's red came from module-level state in the new allocation counter. I fixed that and pushed, but I haven't seen the forge's full check run on it yet.
What changed
- The jobs that allocate heap used to be counted in two module-level
Maps insrc/perf/job_alloc.ts.test/module-state.test.tsrefuses that, because a forge's counts must not leak into another forge in the same process. - The counter is now
createTally(). Each job watch insrc/perf/jobs.tsowns one, so a forge's jobs are counted on that forge. The thread's unowned jobs are counted on the shared unowned watch. JobWatch.allocated()reads and empties a watch's tally, andallocationsOf(app)merges the unowned and per-forge tallies.src/boards/memory.tsnow calls it for the once-a-minute allocation line.test/job-alloc.test.tsuses a tally of its own.
What I ran
bun testontest/job-alloc.test.ts,test/module-state.test.tsand the suites that hold the memory-sampler, fast-forward and upstream tests: 30 pass, 0 fail.tsc --noEmitis clean.scripts/gate.ts --fixrewrapped comments in two files, and that is in the commit.
Not investigated The fast-forward, memory-sampler and "looking upstream" failures did not reproduce in the suites I ran. I did not find a separate cause for them. They may have been fallout from the module-state red, or flaky in the shard.
Try it Open the preview and watch its log for the allocation line after the memory line.
🤖 Generated with Claude Code
The forge's gate refused src/boards/memory.ts because my earlier gate.ts --fix had rewrapped three comments I didn't write. I put those three comments back on their original single lines and pushed. I haven't seen the forge's check run on this push yet.
What I ran here:
bun scripts/gate.tsnow passes. It still prints a note that those restored lines are over 100 columns. The note says nothing is required of this pull and the nightly rewrap handles them on main.bun x tsc --noEmitis clean.- I did not re-run any tests, since this push only changes comment lines.
The diff to memory.ts is now just the import changes and the two-line allocation log in the sampler tick. The rest of the change is as described in the last pull description: each job watch keeps its own allocation tally instead of module-level maps, and the once-a-minute allocation line is read from it.
To try it, open the preview and watch its log for the allocation line after the memory line.
🤖 Generated with Claude Code
The forge's gate refused src/boards/memory.ts again, this time for the comment above startMemorySampler. I had missed it when I restored the others. It is back on its original single line, and I pushed. I haven't seen the forge's check run on this push.
bun scripts/gate.ts passes here and still prints the over-100-columns note. As before, the note says nothing is required of this pull. I did not re-run tsc or any tests, since only a comment changed.
memory.ts now differs from main in the two imports and two lines in the sampler tick that log the allocation line. That diff is the whole change to the file. The rest of the pull is unchanged: each job watch keeps its own allocation tally instead of module-level maps, and the once-a-minute allocation line is read from it.
To try it, open the preview and watch its log for the allocation line after the memory line.
🤖 Generated with Claude Code
The measurement idea is sound and honestly scoped, and I found nothing unsafe in it. Three things need fixing in one round.
- Tests (first fix). Not every behaviour added has a test that would fail without it. (a) The wiring is untested:
allocationsOf(merging the unowned and per-forge tallies, top 5) and thememory allocated by jobs:line in thesrc/boards/memory.tssampler. Delete either and everything stays green. Add a test that runsduring(...)through a watch, ticks the sampler, and asserts the log line. (b) The second test is titled 'a throw is still counted and rethrown' but only asserts the rethrow. Assert the job appears intake(). (c) The 128-name cap has no test. (d) The first test relies onheapSize()growing by more than 8 MiB with no GC and may flake in the shard. Retain more, and assert> 0or retry. bench/idle.tsmeasures the wrong thing. 'Growth' is max minus min of heap readings, divided by minutes. A normal GC sawtooth, or one reading taken just after a collection, reads as growth or hides it, and over a short--secsthe divisor is under 2. Fit the slope, or compare the last reading's floor to the first's, and say so. It is also untested and nothing runs it. Name where the budget is enforced, or keep it out of the pull.- Direction and the issue. The issue asked for the stalls to stop. This pull adds instrumentation and says so, which is honest. But it is not pipeline work, and the preview cannot show the line: the repro needs hosted forges and a crew. Open a follow-up issue for the fix and link it in the description. Then this lands as an explicit measurement step.
Smaller: the doc comment in job_alloc.ts shows memory allocated: timers 84.0 MiB (60 runs), but the real format is memory allocated by jobs: ... in N slices. Nested during calls count the same growth for both the outer and the inner name. Note it or guard it.
Checked from the diff: no input, security or terminal surface is touched, no dependency is added, and AGENTS.md is the right place for the state note.
This pull measures; it does not stop the stalls. It adds a per-job heap-allocation tally and a minute log line, memory allocated by jobs: <name> <MiB> in N slices, that names the background job growing the JS heap while no request runs. The fix is split off to #14028, which waits on what that line names on a hosted forge with a crew. The preview can't show the line, because the repro needs hosted forges and a crew.
What changed since review
- Wiring tests (
test/job-alloc.test.ts):allocationsOfis tested through realduring(...)calls on the unowned watch and the forge's watch. It checks that the two tallies merge, that the result is sorted most first, that it is capped at five, and that reading empties it.- A test runs
duringthrough a watch, ticksstartMemorySampler, and asserts thememory allocated by jobs:log line. - The throw test now asserts that
boomappears intake(). - The 128-name cap is tested, including that a name already kept still counts after the cap is reached.
- Flake fix:
createTallytakes an optional heap-size reader that defaults toheapSize. The throw and cap tests use a fake reader, so they don't depend on the GC. The two integration tests that need the real heap retry until growth is seen and release what they hold afterwards. I found thatheapSize()often shows no growth across a slice, even when the slice allocates tens of MiB, and typed arrays don't count at all, so the old Uint8Array test was fragile. bench/idle.ts: it now fits a least-squares slope of heap against log time. With fewer than three readings it reports "unmeasured (run longer)" rather than a number. Nothing runs it on a push, and the header says so: the budget is enforced only by whoever runs it by hand. I kept it in the pull, but it still has no test, since it's a top-level script.job_alloc.tscomment: the example now matches the real output, and the comment notes that a nestedduringcounts the same growth under both names, so totals aren't additive.
What I chose not to do
- I did not guard against nested
duringdouble-counting. It would need an owner stack in the hot path for a diagnostic that is only a floor anyway. - I did not change the heap reading into anything heavier than
heapSize(), which is constant time.
Checks
bun test test/job-alloc.test.tspasses (6 tests, run three times), andbun x tsc --noEmitis clean.scripts/gate.ts --fixwants to rewrap two comments insrc/boards/memory.tsthat this pull didn't write. I left them as the earlier commits did, so the gate may still flag them.- I did not run the full suite; the forge runs it on push.
To try it: run bun test test/job-alloc.test.ts. On a real forge, watch the log for the memory allocated by jobs: line.
🤖 Generated with Claude Code
The rewrap was already folded into the commit (-a), so the tree is clean.
The failing test, allocationsOf merges the unowned and this forge's jobs, most first, five at most, ran over the 2 s test budget (2.06 s in the check). I made it cheaper and pushed.
- Change: the test now runs 5 own jobs and 2 shared runs per side instead of 7 and 3, and each job grows the heap by 100k objects instead of 150k. That still gives six distinct names, so the five-at-most cap is still tested.
- What I tried first: cutting the allocation to 30k objects made the test flaky, because the heap sometimes never registered growth. I went back to a larger size and cut the number of jobs instead.
- Result: the test took about 1 s when I timed it. All 6 tests in
test/job-alloc.test.tspass, andtscis clean. I did not run the full suite or rerun the check. The other failure in the log, the?pageattacks test in shard 43, I did not touch or investigate. It looks like the same slow-machine noise, but I haven't confirmed that. - Gate:
scripts/gate.ts --fixrewrapped one file's comments, and that went into the same commit.
The gate failed because my earlier bun scripts/gate.ts --fix had rewrapped three comments in src/boards/memory.ts that this pull did not write. I put them back to their original single-line form and pushed.
The diff to memory.ts is now only the allocation line the pull adds. bun scripts/gate.ts no longer reports a diff-is-the-change violation. It prints a note that those three comments run over 100 columns, which it says the nightly rewrap handles on main. tsc is clean and the 6 tests in test/job-alloc.test.ts pass. I did not run any other tests, and I have not seen the forge's check on this push yet.
The earlier fix for the slow allocationsOf test is still in place: it uses fewer, smaller jobs and finished in about 1 s.
Approved. The earlier asks are met: the tally is per job watch, not module state. The wiring from during to allocationsOf to the minute log line is tested. The throw and 128-name cap tests use a fake heap reader, so they do not depend on the GC. bench/idle.ts fits a slope and says so when it has fewer than three readings. The memory.ts diff is now only the import and the two sampler lines. There is no page or terminal surface here, so the both-doors and README checks do not apply. The pull says it measures and does not fix, and the fix is split to #14028.
Two things I would do next:
- Non-blocking: read the first production
memory allocated by jobs:line and open the fix pull for whichever timer leads. Nothing runsbench/idle.tson a push, so until it is wired into a weekly duty the budget is enforced by hand only. - Non-blocking:
test/job-alloc.test.tsretries up to 40 times against the real heap, which is the one place it could flake or run slow on a loaded runner. If it flakes again, move those two integration tests to the fake reader by injecting the tally intocreateJobWatch.
This pull does not fix the stalls. It adds the measurement that names the allocator, and I could not reproduce the growth here.
What I found. I seeded a forge (300 repos, 5,000 issues, 20,000 comments, 25,000 events), booted
src/main.tsover it withFORGE_ENGINE=worker, and asked it nothing for about two minutes. The serving thread's heap stayed flat at about 27 MiB after boot. No job grew it by even 1 MiB a minute, and I saw no stall after boot. So the 5 MiB/s climb needs something this rig lacks: hosted forges, a running crew, or open live-update streams. I could not seed those, and the stage rig isn't reachable from my sandbox. Reading the code, I found no timer that allocates visibly. The per-second cell watch already reads its process trees off-thread.What changed:
src/perf/job_alloc.ts(new):during(src/perf/jobs.ts) now reads the JS heap size on either side of each job's synchronous slice and credits any growth to the job's name, up to 128 names.src/boards/memory.tslogs the result asmemory allocated by jobs: timers 84.0 MiB in 60 slices, …when a job grew the heap by at least 1 MiB, then empties the count. One production minute with this deployed should name the culprit.awaitis not seen.bench/idle.ts(new): the idle budget as a script. It boots the forge over a seeded directory, waits (--secs, default 120), parses the forge's own log, and exits 1 on heap growth over 20 MiB a minute or any stall over 100 ms (the first 20 s of boot are ignored). Runbun run bench/seed.ts --data Dfirst, thenbun run bench/idle.ts --data D. On my seeded rig it passes.test/job-alloc.test.ts(new): covers the attribution.AGENTS.md: one line for the new in-memory state.Not done:
bun test. It is a script, not a suite test, because the run is too long.To try it: deploy, read the
memory allocated by jobs:lines in the production log, and open a follow-up on whichever timer leads.bun test test/job-alloc.test.tsruns the unit test.I ran
bun scripts/gate.ts --fix(it rewrapped comments in 2 files, which are committed),bun x tsc --noEmitand the tests that cover the change. I did not run the whole suite; the forge's check does that on the push.Closes #13924