Public: anyone can read this; the crew can change it.

The page thread stops for half a second about five times a minute with no request running: its heap grows 5 MB a second and is collected in long pauses #13925

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull does not fix the stalls. It adds the measurement that names the allocator, and I could not reproduce the growth here. What I found.

Notes

shipped production is serving it deploy 4adf612

opened this

This pull does not fix the stalls. It adds the measurement that names the allocator, and I could not reproduce the growth here.

What I found. I seeded a forge (300 repos, 5,000 issues, 20,000 comments, 25,000 events), booted src/main.ts over it with FORGE_ENGINE=worker, and asked it nothing for about two minutes. The serving thread's heap stayed flat at about 27 MiB after boot. No job grew it by even 1 MiB a minute, and I saw no stall after boot. So the 5 MiB/s climb needs something this rig lacks: hosted forges, a running crew, or open live-update streams. I could not seed those, and the stage rig isn't reachable from my sandbox. Reading the code, I found no timer that allocates visibly. The per-second cell watch already reads its process trees off-thread.

What changed:

  • src/perf/job_alloc.ts (new): during (src/perf/jobs.ts) now reads the JS heap size on either side of each job's synchronous slice and credits any growth to the job's name, up to 128 names.
  • Minute memory line: src/boards/memory.ts logs the result as memory allocated by jobs: timers 84.0 MiB in 60 slices, … when a job grew the heap by at least 1 MiB, then empties the count. One production minute with this deployed should name the culprit.
  • Limits of the numbers: they are a floor. A collection inside a slice hides growth, and allocation after an await is not seen.
  • bench/idle.ts (new): the idle budget as a script. It boots the forge over a seeded directory, waits (--secs, default 120), parses the forge's own log, and exits 1 on heap growth over 20 MiB a minute or any stall over 100 ms (the first 20 s of boot are ignored). Run bun run bench/seed.ts --data D first, then bun run bench/idle.ts --data D. On my seeded rig it passes.
  • test/job-alloc.test.ts (new): covers the attribution.
  • AGENTS.md: one line for the new in-memory state.

Not done:

  • Ask 2 (move the allocator to the Worker or stream it). There is nothing named to move yet.
  • The "fewer than five stalls in ten minutes" done-condition. Nothing here claims to meet it.
  • The 2-minute budget as a bun test. It is a script, not a suite test, because the run is too long.

To try it: deploy, read the memory allocated by jobs: lines in the production log, and open a follow-up on whichever timer leads. bun test test/job-alloc.test.ts runs the unit test.

I ran bun scripts/gate.ts --fix (it rewrapped comments in 2 files, which are committed), bun x tsc --noEmit and the tests that cover the change. I did not run the whole suite; the forge's check does that on the push.

Closes #13924

7 earlier comments · 1 event
commented

This pull does not fix the stalls. It adds the measurement that names the allocator, and I could not reproduce the growth here.

What I found. I seeded a forge (300 repos, 5,000 issues, 20,000 comments, 25,000 events), booted src/main.ts over it with FORGE_ENGINE=worker, and asked it nothing for about two minutes. The serving thread's heap stayed flat at about 27 MiB after boot. No job grew it by even 1 MiB a minute, and I saw no stall after boot. So the 5 MiB/s climb needs something this rig lacks: hosted forges, a running crew, or open live-update streams. I could not seed those, and the stage rig isn't reachable from my sandbox. Reading the code, I found no timer that allocates visibly. The per-second cell watch already reads its process trees off-thread.

What changed:

  • src/perf/job_alloc.ts (new): during (src/perf/jobs.ts) now reads the JS heap size on either side of each job's synchronous slice and credits any growth to the job's name, up to 128 names.
  • Minute memory line: src/boards/memory.ts logs the result as memory allocated by jobs: timers 84.0 MiB in 60 slices, … when a job grew the heap by at least 1 MiB, then empties the count. One production minute with this deployed should name the culprit.
  • Limits of the numbers: they are a floor. A collection inside a slice hides growth, and allocation after an await is not seen.
  • bench/idle.ts (new): the idle budget as a script. It boots the forge over a seeded directory, waits (--secs, default 120), parses the forge's own log, and exits 1 on heap growth over 20 MiB a minute or any stall over 100 ms (the first 20 s of boot are ignored). Run bun run bench/seed.ts --data D first, then bun run bench/idle.ts --data D. On my seeded rig it passes.
  • test/job-alloc.test.ts (new): covers the attribution.
  • AGENTS.md: one line for the new in-memory state.

Not done:

  • Ask 2 (move the allocator to the Worker or stream it). There is nothing named to move yet.
  • The "fewer than five stalls in ten minutes" done-condition. Nothing here claims to meet it.
  • The 2-minute budget as a bun test. It is a script, not a suite test, because the run is too long.

To try it: deploy, read the memory allocated by jobs: lines in the production log, and open a follow-up on whichever timer leads. bun test test/job-alloc.test.ts runs the unit test.

I ran bun scripts/gate.ts --fix (it rewrapped comments in 2 files, which are committed), bun x tsc --noEmit and the tests that cover the change. I did not run the whole suite; the forge's check does that on the push.

commented

This push fixes the two gate failures from the check. I did not run the tests, since this change only restores comments.

  • Comment rewraps: I restored the three comments in src/boards/memory.ts to their original one-line form, so the diff there is now only the new job_alloc import and the allocation log line in the sampler. The gate no longer reports the "diff is the change" rule. It still prints a note about comment lines over 100 columns in that file; the note says nothing needs doing and the nightly rewrap handles it on main.
  • Try line: the new commit message has a Try: line: open the preview and watch its log for the once-a-minute allocation line after the memory line.

The gate and tsc --noEmit came back clean before I pushed.

commented

The check's red came from module-level state in the new allocation counter. I fixed that and pushed, but I haven't seen the forge's full check run on it yet.

What changed

  • The jobs that allocate heap used to be counted in two module-level Maps in src/perf/job_alloc.ts. test/module-state.test.ts refuses that, because a forge's counts must not leak into another forge in the same process.
  • The counter is now createTally(). Each job watch in src/perf/jobs.ts owns one, so a forge's jobs are counted on that forge. The thread's unowned jobs are counted on the shared unowned watch.
  • JobWatch.allocated() reads and empties a watch's tally, and allocationsOf(app) merges the unowned and per-forge tallies. src/boards/memory.ts now calls it for the once-a-minute allocation line.
  • test/job-alloc.test.ts uses a tally of its own.

What I ran

  • bun test on test/job-alloc.test.ts, test/module-state.test.ts and the suites that hold the memory-sampler, fast-forward and upstream tests: 30 pass, 0 fail.
  • tsc --noEmit is clean.
  • scripts/gate.ts --fix rewrapped comments in two files, and that is in the commit.

Not investigated The fast-forward, memory-sampler and "looking upstream" failures did not reproduce in the suites I ran. I did not find a separate cause for them. They may have been fallout from the module-state red, or flaky in the shard.

Try it Open the preview and watch its log for the allocation line after the memory line.

🤖 Generated with Claude Code

commented

The forge's gate refused src/boards/memory.ts because my earlier gate.ts --fix had rewrapped three comments I didn't write. I put those three comments back on their original single lines and pushed. I haven't seen the forge's check run on this push yet.

What I ran here:

  • bun scripts/gate.ts now passes. It still prints a note that those restored lines are over 100 columns. The note says nothing is required of this pull and the nightly rewrap handles them on main.
  • bun x tsc --noEmit is clean.
  • I did not re-run any tests, since this push only changes comment lines.

The diff to memory.ts is now just the import changes and the two-line allocation log in the sampler tick. The rest of the change is as described in the last pull description: each job watch keeps its own allocation tally instead of module-level maps, and the once-a-minute allocation line is read from it.

To try it, open the preview and watch its log for the allocation line after the memory line.

🤖 Generated with Claude Code

commented

The forge's gate refused src/boards/memory.ts again, this time for the comment above startMemorySampler. I had missed it when I restored the others. It is back on its original single line, and I pushed. I haven't seen the forge's check run on this push.

bun scripts/gate.ts passes here and still prints the over-100-columns note. As before, the note says nothing is required of this pull. I did not re-run tsc or any tests, since only a comment changed.

memory.ts now differs from main in the two imports and two lines in the sampler tick that log the allocation line. That diff is the whole change to the file. The rest of the pull is unchanged: each job watch keeps its own allocation tally instead of module-level maps, and the once-a-minute allocation line is read from it.

To try it, open the preview and watch its log for the allocation line after the memory line.

🤖 Generated with Claude Code

Changes requested — before the latest push

The measurement idea is sound and honestly scoped, and I found nothing unsafe in it. Three things need fixing in one round.

  1. Tests (first fix). Not every behaviour added has a test that would fail without it. (a) The wiring is untested: allocationsOf (merging the unowned and per-forge tallies, top 5) and the memory allocated by jobs: line in the src/boards/memory.ts sampler. Delete either and everything stays green. Add a test that runs during(...) through a watch, ticks the sampler, and asserts the log line. (b) The second test is titled 'a throw is still counted and rethrown' but only asserts the rethrow. Assert the job appears in take(). (c) The 128-name cap has no test. (d) The first test relies on heapSize() growing by more than 8 MiB with no GC and may flake in the shard. Retain more, and assert > 0 or retry.
  2. bench/idle.ts measures the wrong thing. 'Growth' is max minus min of heap readings, divided by minutes. A normal GC sawtooth, or one reading taken just after a collection, reads as growth or hides it, and over a short --secs the divisor is under 2. Fit the slope, or compare the last reading's floor to the first's, and say so. It is also untested and nothing runs it. Name where the budget is enforced, or keep it out of the pull.
  3. Direction and the issue. The issue asked for the stalls to stop. This pull adds instrumentation and says so, which is honest. But it is not pipeline work, and the preview cannot show the line: the repro needs hosted forges and a crew. Open a follow-up issue for the fix and link it in the description. Then this lands as an explicit measurement step.

Smaller: the doc comment in job_alloc.ts shows memory allocated: timers 84.0 MiB (60 runs), but the real format is memory allocated by jobs: ... in N slices. Nested during calls count the same growth for both the outer and the inner name. Note it or guard it.

Checked from the diff: no input, security or terminal surface is touched, no dependency is added, and AGENTS.md is the right place for the state note.

commented

This pull measures; it does not stop the stalls. It adds a per-job heap-allocation tally and a minute log line, memory allocated by jobs: <name> <MiB> in N slices, that names the background job growing the JS heap while no request runs. The fix is split off to #14028, which waits on what that line names on a hosted forge with a crew. The preview can't show the line, because the repro needs hosted forges and a crew.

What changed since review

  • Wiring tests (test/job-alloc.test.ts):
    • allocationsOf is tested through real during(...) calls on the unowned watch and the forge's watch. It checks that the two tallies merge, that the result is sorted most first, that it is capped at five, and that reading empties it.
    • A test runs during through a watch, ticks startMemorySampler, and asserts the memory allocated by jobs: log line.
    • The throw test now asserts that boom appears in take().
    • The 128-name cap is tested, including that a name already kept still counts after the cap is reached.
  • Flake fix: createTally takes an optional heap-size reader that defaults to heapSize. The throw and cap tests use a fake reader, so they don't depend on the GC. The two integration tests that need the real heap retry until growth is seen and release what they hold afterwards. I found that heapSize() often shows no growth across a slice, even when the slice allocates tens of MiB, and typed arrays don't count at all, so the old Uint8Array test was fragile.
  • bench/idle.ts: it now fits a least-squares slope of heap against log time. With fewer than three readings it reports "unmeasured (run longer)" rather than a number. Nothing runs it on a push, and the header says so: the budget is enforced only by whoever runs it by hand. I kept it in the pull, but it still has no test, since it's a top-level script.
  • job_alloc.ts comment: the example now matches the real output, and the comment notes that a nested during counts the same growth under both names, so totals aren't additive.

What I chose not to do

  • I did not guard against nested during double-counting. It would need an owner stack in the hot path for a diagnostic that is only a floor anyway.
  • I did not change the heap reading into anything heavier than heapSize(), which is constant time.

Checks

  • bun test test/job-alloc.test.ts passes (6 tests, run three times), and bun x tsc --noEmit is clean.
  • scripts/gate.ts --fix wants to rewrap two comments in src/boards/memory.ts that this pull didn't write. I left them as the earlier commits did, so the gate may still flag them.
  • I did not run the full suite; the forge runs it on push.

To try it: run bun test test/job-alloc.test.ts. On a real forge, watch the log for the memory allocated by jobs: line.

🤖 Generated with Claude Code

commented

The rewrap was already folded into the commit (-a), so the tree is clean.

The failing test, allocationsOf merges the unowned and this forge's jobs, most first, five at most, ran over the 2 s test budget (2.06 s in the check). I made it cheaper and pushed.

  • Change: the test now runs 5 own jobs and 2 shared runs per side instead of 7 and 3, and each job grows the heap by 100k objects instead of 150k. That still gives six distinct names, so the five-at-most cap is still tested.
  • What I tried first: cutting the allocation to 30k objects made the test flaky, because the heap sometimes never registered growth. I went back to a larger size and cut the number of jobs instead.
  • Result: the test took about 1 s when I timed it. All 6 tests in test/job-alloc.test.ts pass, and tsc is clean. I did not run the full suite or rerun the check. The other failure in the log, the ?page attacks test in shard 43, I did not touch or investigate. It looks like the same slow-machine noise, but I haven't confirmed that.
  • Gate: scripts/gate.ts --fix rewrapped one file's comments, and that went into the same commit.
commented

The gate failed because my earlier bun scripts/gate.ts --fix had rewrapped three comments in src/boards/memory.ts that this pull did not write. I put them back to their original single-line form and pushed.

The diff to memory.ts is now only the allocation line the pull adds. bun scripts/gate.ts no longer reports a diff-is-the-change violation. It prints a note that those three comments run over 100 columns, which it says the nightly rewrap handles on main. tsc is clean and the 6 tests in test/job-alloc.test.ts pass. I did not run any other tests, and I have not seen the forge's check on this push yet.

The earlier fix for the slow allocationsOf test is still in place: it uses fewer, smaller jobs and finished in about 1 s.

Approved

Approved. The earlier asks are met: the tally is per job watch, not module state. The wiring from during to allocationsOf to the minute log line is tested. The throw and 128-name cap tests use a fake heap reader, so they do not depend on the GC. bench/idle.ts fits a slope and says so when it has fewer than three readings. The memory.ts diff is now only the import and the two sampler lines. There is no page or terminal surface here, so the both-doors and README checks do not apply. The pull says it measures and does not fix, and the fix is split to #14028.

Two things I would do next:

  1. Non-blocking: read the first production memory allocated by jobs: line and open the fix pull for whichever timer leads. Nothing runs bench/idle.ts on a push, so until it is wired into a weekly duty the budget is enforced by hand only.
  2. Non-blocking: test/job-alloc.test.ts retries up to 40 times against the real heap, which is the one place it could flake or run slow on a loaded runner. If it flakes again, move those two integration tests to the fake reader by injecting the tally into createJobWatch.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs