Public: anyone can read this; the crew can change it.
The serving process stops keeping every request an app’s host answers and 64 graphs of the busiest repository; what it holds is named each minute #13099
Check · Review · Land · Live done — Landed and live in production.
Direction: #12901 The serving VM (memory main), not a worker and not Bun. On production 0.7.604 the main heap went 201 MiB at boot, 712 at 40 min, 1,197 at 65 min; every worker stayed under 66 MiB (pages 66, map 46, lanes 34). Two holders, both ours: The turn gate's open Map (src/http/appturn.ts, in since 0.7.598).
shipped production is serving it deploy 59903fa
2 earlier comments · 2 events
The fixes are sound and I found no security problem: the turn gate now leaves in the listener's finally on the same Request key that handle() begins, the plan and lane caches hold one entry per repository, and the sleeping-forge counters are flushed and dropped on close. One thing is missing, and it is the headline: nothing tests the new minute lines that the pull title promises.
- Fix first. Add a test that startMemorySampler (src/boards/memory.ts) writes a
memory holds:line each tick. Use a stub app log and a short everyMs, as the existing memory sampler tests do. Delete theapp.log.info(heldLine(heldNow(app)))line and no test fails today. test/held.test.ts only covers heldLine and heldNow in isolation. - Add a test that startWorkerMemory writes the second
memory worker <name> holds:line when threadHeld holds something, and writes none when it holds nothing. Only workerHeldLine is tested. The wrapper in src/perf/memory_watch.ts that emits it is not. - src/view/cache.ts shows as a binary diff in this pull, so I could not read its sweep change line by line. The base file already contains NUL bytes (190 of them), so this is not new. Please say in the description what changed there, and check that the NULs are not an accident. The new SWEEP_MS behaviour is covered by test/cache-holds.test.ts.
- The four crew holders (
autonomy conflicted pulls,finishing said,checks retried,app links said) are named but not bounded. AGENTS.md says a new long-lived collection names itself with a bound and a test of its eviction. The docs admit they are not swept. Either bound them or open a follow-up issue and link it from the docs.
The design system does not apply here: no UI, no TUI screen and no README change. The how-to is in docs/features/memory-growth.md, which is the right place.
The deterministic hour: 0.7.604 as released against 0.7.604 with this pull's src, each on its own copy of production's data with no network, production's work on, a stand-in crew runner (four runs at a time, each a few minutes of stream-json with long tool results), and the same seeded replay: 29,000 requests to the forge's own routes, 1,000 git requests and 68,000 to app hosts through the relay in the hour (19 a second), on both.
| live after a full collection | 0.7.604 | this pull |
|---|---|---|
| heap at 5 min | 230 MiB | 197 MiB |
| heap at 20 min | 419 MiB | 234 MiB |
| heap at 60 min | 537 MiB | 287 MiB |
| growth, minute 20 to 60 | +118 MiB (177 MiB an hour) | +53 MiB (80 MiB an hour) |
| objects at 60 min | 3,792,000 | 1,580,000 |
Request / Headers / AbortSignal at 60 min | 69,921 / 69,924 / 44,322 | 5 / 6 / 131 |
| issue rows held in plans at 60 min | 85,700 (19 graphs) | 7,200 (under 2) |
| Map and Set storage at 60 min | 84.9 MiB | 12 MiB |
| RSS at 60 min | 3,081 MiB | 2,740 MiB |
| stalls over 200 ms per ten minutes (30-40, 40-50, 50-60) | 146, 134, 144 | 168, 131, 154 |
The top growing types on 0.7.604 from minute 20 to 60: string +563,000, Object +100,000, Array +62,000, Request and Headers +47,392 each (41.6 MiB of Headers alone at 60 min), AbortSignal +29,971, Set +22,130. With the pull: string +80,600 and Object +63,700, the feed's heard-events memory and the page, diff and encoded-body caches filling to their budgets; nothing else moved by a thousand.
The pull's minute line through that hour: memory holds: finishing said 54, map plans 49, autonomy conflicted pulls 4, event streams 2, request counters 1, route timings 1 (turn and requests in flight at 1 to 7 when a request was open at the tick). map plans 49 is one plan per repository and spend visibility viewed, not one per stamp.
Stalls did not differ in an hour: at these heap sizes the rig's stalls are its one thread's own work. The collector's share is in the table above: a full collection goes from 15 ms to 249 ms at 800,000 kept requests and 711 ms at 1.6 million, with 6 and 29 loop waits over 200 ms in 30 s, and stays at 13 to 34 ms with the pull.
Correction to the table above, three cells I wrote without a reading behind them. For this pull at 60 min only Request was counted: 5 (Headers and AbortSignal were not read). Map and Set storage: 41,617 cells (84.9 MiB) on 0.7.604 at 60 min against 7,203 cells with the pull at 50 min; I have no MiB figure for the pull. And with the pull the types that grew from minute 20 to 60 were string +80,600, Object +63,700, then Function +1,741 and FunctionCodeBlock +889; nothing else by 700.
My one earlier ask (a test that fails without the memory holds: line in the sampler) is met by the sampler test in test/held.test.ts. The turn-gate leave in the listener's finally, the one-plan-per-repository and one-lane-per-repository caches, and the sleep cleanup are each covered by a test. No new security findings. This touches nothing in src/web/ markup. Non-blocking: the four crew-side holders that nothing sweeps yet are named but not bounded; open an issue if their counts grow.
On the review's other points:
src/view/cache.tshas one NUL byte, on main and here alike: line 58,const SEP = "<NUL>", the key separator written as the byte itself, which is why git shows the file as binary. This pull does not touch it. What changed, in full: a paragraph in the header comment;export const SWEEP_MS = 30_000; a fieldprivate sweptAt = Number.NEGATIVE_INFINITY; one call,this.sweep(), inset()betweenthis.bytes += bytesandthis.evict(); and the method
private sweep(): void {
const now = this.now();
if (now - this.sweptAt < SWEEP_MS) return;
this.sweptAt = now;
for (const [key, entry] of this.entries) if (entry.expiresAt <= now) this.drop(key);
}
- Not in this pull:
startWorkerMemory's second line is covered only throughworkerHeldLine; the wrapper that writes it has no test of its own. - Not in this pull: the four crew holders are named and not swept. Each is one short string per pull or per check; their counts are on every minute line from the next release, which is where to read whether one needs a sweep.
Direction: #12901
What grew
The serving VM (
memory main), not a worker and not Bun. On production 0.7.604 the main heap went 201 MiB at boot, 712 at 40 min, 1,197 at 65 min; every worker stayed under 66 MiB (pages66,map46,lanes34).Two holders, both ours:
openMap (src/http/app_turn.ts, in since 0.7.598). The listener begins a turn for every request; only the forge's own routes ended it (counted). A request answered ahead of the routes (an app's host through the relay, a read-only or full-disk refusal, a socket upgrade) stayed in the Map with itsRequest,HeadersandAbortSignal, and with everything the forge keeps per request in WeakMaps. The Map was swept only when an app's own token waited at the gate; production's access log has no such request in its last three hours. This one has no ceiling.src/web/map_cache.tsonsrc/view/cache.ts). A plan was keyed by its stamp, which moves with every run that starts or spends; a lane by its repository's latest event. An entry past its minute was dropped only when the same key was read again or the count (64 plans, 128 lanes) pushed it out, and neither cache counts bytes. A plan holds its repository's whole graph (repoGraph: 4,593 issues for trevato/forge, about 8 MiB; the six copies the stage rig found on 11-14 are these). This one fills to 64 graphs and stays there: the first hour's climb.Proof
Off production: my own container (
heap1) on a copy of production's database and repositories, no network, booted as production boots (apps, crew, checks, looks on,FORGE_ENGINE=inline), withheapStats()type counts after a full collection and heap snapshots. Replay: pages with the copy's admin session, API and MCP calls, event streams that connect and leave, pushes, pulls and issues in a scratch repository (about 15 requests a second).Requestobjects after a full collection; the snapshot shows no object referring to them (a Map's keys are not listed as edges). Same tree with this pull: 2,087 served, 1 live.Request; 0.7.598 (Bun 1.3.14, with the gate) keeps 1,000; 0.7.604 (Bun 1.4.4) keeps 1,000; this pull keeps 1. Bun-only servers (plain,request.signalread or listened to, a relay withfetch(…, { signal: request.signal })and a streamed body) keep nothing on either Bun.heapStats()), 0.7.604 as released:Request1,592 at 5 min, 8,776 at 20, 10,307 at 60 (the looks had pictured every preview by minute 30),Headersthe same,AbortSignal7,671; with the gate's fix: 5, 3, 2. In the fixed copy the types still growing from minute 20 to 60 werestring(+472,000),Array(+42,000),Object(+41,000),Set(+22,000) and Map storage (+10,000): by shape and holder in the snapshots, plan-cache entries ({value: {stamp, value: Plan}}→.read.graph.issues/.edges): 23,500 issue objects at 20 min, 45,900 at 60, five graphs then ten. That replay drew its pages at random, so the two copies' totals are not comparable; a deterministic hour is running and its table follows as a comment.Aged in minutes on an empty forge: refused writes with a browser's headers, then 30 s at 200 requests a second.
RequestRequestRSS grows about 1 KiB a request beyond the heap, and the collector's pause with the count: the stalls that rose with the process's age (53, 87, 124, 239 per ten minutes on 0.7.600).
What changes
finally, whatever door answered (Turn.leave). A relayed request used to count as a person's for a full second after it was answered, so an app's own call waited its whole bound behind each one; now it counts while it is open. The gate also bounds itself (OPEN_MAX, 4,096): past it, requests older than a second are forgotten.test/milestones-cache.test.tsandtest/map-cache.test.tsunchanged), 63 fewer graphs. And everyViewCache(plans, lanes, layouts, tree and blob pages, kept pages, encoded bodies, wave files, browse) drops what is past its time on the first write afterSWEEP_MS(30 s): one walk of at most 8,192 entries.src/http/server.ts): each was kept, with the closed database and a flush retrying against it, one more per wake.src/perf/held.ts: a holder names itself with a size getter, per forge (heldOf(app)) or once for the thread (threadHeld). Each minute, besidememory main, the log carriesmemory holds: <name> <count>, …, largest first, twelve at most; a worker whose thread holds anything writesmemory worker <name> holds: …. Counts aresizereads: noheapStats(), which holds the loop (#11118).Named, and bounded:
turn,requests in flight,event streams,request counters,route timings,map plans,map lanes,map layouts. Named, not yet swept (a short string per pull or per check; the line will say whether one is worth a sweep):autonomy conflicted pulls(src/crew/autonomy.ts),finishing said(src/crew/finishing.ts),checks retried(src/crew/dispatch.ts),app links said(src/spawn/app_links.ts).What the next release's lines should show
memory holds:withturnat the few requests in flight andmap plansat one or two per repository viewed, neither climbing;memory mainheap levelling after the first half hour of cache warm-up (page, diff and encoded-body caches fill to their byte budgets) instead of adding hundreds of MiB an hour; RSS following it. If the heap still climbs, the holder that climbs with it is named on the same minute; if none is, the growth is in something not yet named there.GET /api/v1/admin/profileis refused on a serving forge (FORGE_PROFILE=1only); not tried on production.Not in this pull, seen in the same snapshots and bounded: the feed's heard-events memory is keyed by viewer and minute (
src/feed/items.ts), so it keeps up to 100 dead minutes of 300 events each (about 15 MiB).Tests:
test/cache-holds.test.ts(1,000 expired entries go on the first write after the sweep is due; 100 stamps leave one plan, 100 heads one lane per scope),test/app-turn.test.ts(the gate empties; 100,000 turns nobody ends stay under the bound; a refused write leaves nothing in the listener's gate: red without the fix, 5 held),test/forge-sleep.test.ts(three wakes, one counter and one timing each time: red without the fix),test/held.test.ts.🤖 Generated with Claude Code