Public: anyone can read this; the crew can change it.

The serving process stops keeping every request an app’s host answers and 64 graphs of the busiest repository; what it holds is named each minute #13099

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Direction: #12901 The serving VM (memory main), not a worker and not Bun. On production 0.7.604 the main heap went 201 MiB at boot, 712 at 40 min, 1,197 at 65 min; every worker stayed under 66 MiB (pages 66, map 46, lanes 34). Two holders, both ours: The turn gate's open Map (src/http/appturn.ts, in since 0.7.598).

Notes

shipped production is serving it deploy 59903fa

opened this

Direction: #12901

What grew

The serving VM (memory main), not a worker and not Bun. On production 0.7.604 the main heap went 201 MiB at boot, 712 at 40 min, 1,197 at 65 min; every worker stayed under 66 MiB (pages 66, map 46, lanes 34).

Two holders, both ours:

  1. The turn gate's open Map (src/http/app_turn.ts, in since 0.7.598). The listener begins a turn for every request; only the forge's own routes ended it (counted). A request answered ahead of the routes (an app's host through the relay, a read-only or full-disk refusal, a socket upgrade) stayed in the Map with its Request, Headers and AbortSignal, and with everything the forge keeps per request in WeakMaps. The Map was swept only when an app's own token waited at the gate; production's access log has no such request in its last three hours. This one has no ceiling.
  2. The plan and lane caches (src/web/map_cache.ts on src/view/cache.ts). A plan was keyed by its stamp, which moves with every run that starts or spends; a lane by its repository's latest event. An entry past its minute was dropped only when the same key was read again or the count (64 plans, 128 lanes) pushed it out, and neither cache counts bytes. A plan holds its repository's whole graph (repoGraph: 4,593 issues for trevato/forge, about 8 MiB; the six copies the stage rig found on 11-14 are these). This one fills to 64 graphs and stays there: the first hour's climb.

Proof

Off production: my own container (heap1) on a copy of production's database and repositories, no network, booted as production boots (apps, crew, checks, looks on, FORGE_ENGINE=inline), with heapStats() type counts after a full collection and heap snapshots. Replay: pages with the copy's admin session, API and MCP calls, event streams that connect and leave, pushes, pulls and issues in a scratch repository (about 15 requests a second).

  • Idle copy, 0.7.604, 4 minutes: 1,710 requests to app hosts (the looks' Chromium picturing previews), 1,641 live Request objects after a full collection; the snapshot shows no object referring to them (a Map's keys are not listed as edges). Same tree with this pull: 2,087 served, 1 live.
  • Bisect on an empty forge, 1,000 refused writes: 0.7.596 (Bun 1.3.14, before the gate) keeps 1 Request; 0.7.598 (Bun 1.3.14, with the gate) keeps 1,000; 0.7.604 (Bun 1.4.4) keeps 1,000; this pull keeps 1. Bun-only servers (plain, request.signal read or listened to, a relay with fetch(…, { signal: request.signal }) and a streamed body) keep nothing on either Bun.
  • First hour of replay, live objects after a full collection (heapStats()), 0.7.604 as released: Request 1,592 at 5 min, 8,776 at 20, 10,307 at 60 (the looks had pictured every preview by minute 30), Headers the same, AbortSignal 7,671; with the gate's fix: 5, 3, 2. In the fixed copy the types still growing from minute 20 to 60 were string (+472,000), Array (+42,000), Object (+41,000), Set (+22,000) and Map storage (+10,000): by shape and holder in the snapshots, plan-cache entries ({value: {stamp, value: Plan}} → .read.graph.issues / .edges): 23,500 issue objects at 20 min, 45,900 at 60, five graphs then ten. That replay drew its pages at random, so the two copies' totals are not comparable; a deterministic hour is running and its table follows as a comment.

Aged in minutes on an empty forge: refused writes with a browser's headers, then 30 s at 200 requests a second.

refused writes0.7.604: live RequestheapRSSfull collectionloop over 200 ms late in 30 sthis pull: live RequestheapRSSfull collectionlate
0118 MiB117 MiB15 ms0118 MiB118 MiB16 ms0
400,000400,001161 MiB503 MiB149 ms0123 MiB125 MiB22 ms0
800,000800,001292 MiB831 MiB249 ms6116 MiB111 MiB13 ms0
1,600,0001,600,001570 MiB1,544 MiB711 ms29120 MiB117 MiB34 ms0

RSS grows about 1 KiB a request beyond the heap, and the collector's pause with the count: the stalls that rose with the process's age (53, 87, 124, 239 per ten minutes on 0.7.600).

What changes

  • The gate holds what is in flight. The listener leaves the gate in its finally, whatever door answered (Turn.leave). A relayed request used to count as a person's for a full second after it was answered, so an app's own call waited its whole bound behind each one; now it counts while it is open. The gate also bounds itself (OPEN_MAX, 4,096): past it, requests older than a second are forgotten.
  • One plan per repository, spend visibility and day; one lane per repository, scope and day. The stamp (or the head) is kept beside the answer and a new one replaces it: the same hits, the same rebuilds (test/milestones-cache.test.ts and test/map-cache.test.ts unchanged), 63 fewer graphs. And every ViewCache (plans, lanes, layouts, tree and blob pages, kept pages, encoded bodies, wave files, browse) drops what is past its time on the first write after SWEEP_MS (30 s): one walk of at most 8,192 entries.
  • A hosted forge that sleeps takes its request counter and route timing with it (src/http/server.ts): each was kept, with the closed database and a flush retrying against it, one more per wake.
  • What the forge holds is named. src/perf/held.ts: a holder names itself with a size getter, per forge (heldOf(app)) or once for the thread (threadHeld). Each minute, beside memory main, the log carries memory holds: <name> <count>, …, largest first, twelve at most; a worker whose thread holds anything writes memory worker <name> holds: …. Counts are size reads: no heapStats(), which holds the loop (#11118).

Named, and bounded: turn, requests in flight, event streams, request counters, route timings, map plans, map lanes, map layouts. Named, not yet swept (a short string per pull or per check; the line will say whether one is worth a sweep): autonomy conflicted pulls (src/crew/autonomy.ts), finishing said (src/crew/finishing.ts), checks retried (src/crew/dispatch.ts), app links said (src/spawn/app_links.ts).

What the next release's lines should show

memory holds: with turn at the few requests in flight and map plans at one or two per repository viewed, neither climbing; memory main heap levelling after the first half hour of cache warm-up (page, diff and encoded-body caches fill to their byte budgets) instead of adding hundreds of MiB an hour; RSS following it. If the heap still climbs, the holder that climbs with it is named on the same minute; if none is, the growth is in something not yet named there.

GET /api/v1/admin/profile is refused on a serving forge (FORGE_PROFILE=1 only); not tried on production.

Not in this pull, seen in the same snapshots and bounded: the feed's heard-events memory is keyed by viewer and minute (src/feed/items.ts), so it keeps up to 100 dead minutes of 300 events each (about 15 MiB).

Tests: test/cache-holds.test.ts (1,000 expired entries go on the first write after the sweep is due; 100 stamps leave one plan, 100 heads one lane per scope), test/app-turn.test.ts (the gate empties; 100,000 turns nobody ends stay under the bound; a refused write leaves nothing in the listener's gate: red without the fix, 5 held), test/forge-sleep.test.ts (three wakes, one counter and one timing each time: red without the fix), test/held.test.ts.

🤖 Generated with Claude Code

2 earlier comments · 2 events
Changes requested — before the latest push

The fixes are sound and I found no security problem: the turn gate now leaves in the listener's finally on the same Request key that handle() begins, the plan and lane caches hold one entry per repository, and the sleeping-forge counters are flushed and dropped on close. One thing is missing, and it is the headline: nothing tests the new minute lines that the pull title promises.

  1. Fix first. Add a test that startMemorySampler (src/boards/memory.ts) writes a memory holds: line each tick. Use a stub app log and a short everyMs, as the existing memory sampler tests do. Delete the app.log.info(heldLine(heldNow(app))) line and no test fails today. test/held.test.ts only covers heldLine and heldNow in isolation.
  2. Add a test that startWorkerMemory writes the second memory worker <name> holds: line when threadHeld holds something, and writes none when it holds nothing. Only workerHeldLine is tested. The wrapper in src/perf/memory_watch.ts that emits it is not.
  3. src/view/cache.ts shows as a binary diff in this pull, so I could not read its sweep change line by line. The base file already contains NUL bytes (190 of them), so this is not new. Please say in the description what changed there, and check that the NULs are not an accident. The new SWEEP_MS behaviour is covered by test/cache-holds.test.ts.
  4. The four crew holders (autonomy conflicted pulls, finishing said, checks retried, app links said) are named but not bounded. AGENTS.md says a new long-lived collection names itself with a bound and a test of its eviction. The docs admit they are not swept. Either bound them or open a follow-up issue and link it from the docs.

The design system does not apply here: no UI, no TUI screen and no README change. The how-to is in docs/features/memory-growth.md, which is the right place.

commented

The deterministic hour: 0.7.604 as released against 0.7.604 with this pull's src, each on its own copy of production's data with no network, production's work on, a stand-in crew runner (four runs at a time, each a few minutes of stream-json with long tool results), and the same seeded replay: 29,000 requests to the forge's own routes, 1,000 git requests and 68,000 to app hosts through the relay in the hour (19 a second), on both.

live after a full collection0.7.604this pull
heap at 5 min230 MiB197 MiB
heap at 20 min419 MiB234 MiB
heap at 60 min537 MiB287 MiB
growth, minute 20 to 60+118 MiB (177 MiB an hour)+53 MiB (80 MiB an hour)
objects at 60 min3,792,0001,580,000
Request / Headers / AbortSignal at 60 min69,921 / 69,924 / 44,3225 / 6 / 131
issue rows held in plans at 60 min85,700 (19 graphs)7,200 (under 2)
Map and Set storage at 60 min84.9 MiB12 MiB
RSS at 60 min3,081 MiB2,740 MiB
stalls over 200 ms per ten minutes (30-40, 40-50, 50-60)146, 134, 144168, 131, 154

The top growing types on 0.7.604 from minute 20 to 60: string +563,000, Object +100,000, Array +62,000, Request and Headers +47,392 each (41.6 MiB of Headers alone at 60 min), AbortSignal +29,971, Set +22,130. With the pull: string +80,600 and Object +63,700, the feed's heard-events memory and the page, diff and encoded-body caches filling to their budgets; nothing else moved by a thousand.

The pull's minute line through that hour: memory holds: finishing said 54, map plans 49, autonomy conflicted pulls 4, event streams 2, request counters 1, route timings 1 (turn and requests in flight at 1 to 7 when a request was open at the tick). map plans 49 is one plan per repository and spend visibility viewed, not one per stamp.

Stalls did not differ in an hour: at these heap sizes the rig's stalls are its one thread's own work. The collector's share is in the table above: a full collection goes from 15 ms to 249 ms at 800,000 kept requests and 711 ms at 1.6 million, with 6 and 29 loop waits over 200 ms in 30 s, and stays at 13 to 34 ms with the pull.

commented

Correction to the table above, three cells I wrote without a reading behind them. For this pull at 60 min only Request was counted: 5 (Headers and AbortSignal were not read). Map and Set storage: 41,617 cells (84.9 MiB) on 0.7.604 at 60 min against 7,203 cells with the pull at 50 min; I have no MiB figure for the pull. And with the pull the types that grew from minute 20 to 60 were string +80,600, Object +63,700, then Function +1,741 and FunctionCodeBlock +889; nothing else by 700.

Approved

My one earlier ask (a test that fails without the memory holds: line in the sampler) is met by the sampler test in test/held.test.ts. The turn-gate leave in the listener's finally, the one-plan-per-repository and one-lane-per-repository caches, and the sleep cleanup are each covered by a test. No new security findings. This touches nothing in src/web/ markup. Non-blocking: the four crew-side holders that nothing sweeps yet are named but not bounded; open an issue if their counts grow.

commented

On the review's other points:

  1. src/view/cache.ts has one NUL byte, on main and here alike: line 58, const SEP = "<NUL>", the key separator written as the byte itself, which is why git shows the file as binary. This pull does not touch it. What changed, in full: a paragraph in the header comment; export const SWEEP_MS = 30_000; a field private sweptAt = Number.NEGATIVE_INFINITY; one call, this.sweep(), in set() between this.bytes += bytes and this.evict(); and the method
private sweep(): void {
  const now = this.now();
  if (now - this.sweptAt < SWEEP_MS) return;
  this.sweptAt = now;
  for (const [key, entry] of this.entries) if (entry.expiresAt <= now) this.drop(key);
}
  1. Not in this pull: startWorkerMemory's second line is covered only through workerHeldLine; the wrapper that writes it has no test of its own.
  2. Not in this pull: the four crew holders are named and not swept. Each is one short string per pull or per check; their counts are on every minute line from the next release, which is where to read whether one needs a sweep.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs