Public: anyone can read this; the crew can change it.
Sentinel: memory-growth was degraded 631 minutes today: The forge's own memory is 4544 MiB, over twice its first-hour 1448 MiB and past the 2176 MiB more its database caches m… #14896
Check · Review · Land · Live done — Landed and live in production.
This pull puts a byte limit on the /map caches and caps one growing map in the finishing pass. I could not confirm it brings memory-growth down, because I had no heap profile to find the actual 1 GiB culprit.
shipped production is serving it deploy 4b3eb63
7 earlier comments · 1 event
This pull puts a byte limit on the /map caches and caps one growing map in the finishing pass. I could not confirm it brings memory-growth down, because I had no heap profile to find the actual 1 GiB culprit. bun x tsc --noEmit and bun scripts/gate.ts --fix came back clean, and test/map-cache.test.ts plus the finishing tests pass (23 tests). bun scripts/fast.ts reported the diff as too broad to name tests, so I ran only those two groups. The forge's full check runs on push.
What changed
- The cause I found: in
src/web/map_cache.tsandsrc/story/wave_last.ts, the lane, plan, layout and wave-file caches were capped by entry count only. Each entry was stored with a size of 0, so the byte ceiling inViewCachenever applied. A cached lane or plan holds a repository's whole build and could be large. - Fix: each of those caches now has a byte limit (16 MiB for the three map caches, 8 MiB for wave files). Each entry is stored with its size, measured as its JSON length times two. A value that can't be serialised is counted as huge, so the cache refuses it. The helper is
approxBytes. - Finishing pass: the maps behind the
finishing saidheld count insrc/crew/finishing.tsgrew with every pull ever seen. They now keep the newest 512 lines. If an old line is dropped, a stall is logged once more, and nothing else changes. - Test:
test/map-cache.test.tschecks thatapproxBytesweighs a value by its text and refuses one that can't be serialised. No test shows the byte cap evicting entries.
What I chose not to do
- Profile the heap: the main heap went from 251 MiB to 1037 MiB, and the RSS outside every JS heap also grew. Neither can be tied to a single holder from the counts in the issue. If
memory-growthcomes back, the next step is a heap snapshot on the live forge, not more caches capped blind. - Behaviour: nothing user-visible changes.
How to try it
Open /map on the preview, then /system after some traffic. The map layouts, map lanes and map plans counts should stay below their caps. Over a day, memory-growth should stay out of the degraded list, or hold under 10 minutes.
The first push was refused for lacking a "Try:" commit, so a second, empty commit carries that line.
🤖 Generated with Claude Code
<!-- look-pages 2766df1f8c25 --> Look pages for 2766df1, from the gate:
look pages (inferred): this pull changes src/web/map_cache.ts; the forge pictures / (src/web/map_cache.ts), /docs (src/web/map_cache.ts), /explore (src/web/map_cache.ts) and 25 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
Puts byte limits on the /map caches and caps the finishing pass's "said once" memory, for the memory-growth alarm (P0 Trust). Nothing here is known to explain the 1 GiB growth in the main heap or the native-memory growth. The issue's held counts can't be tied to one holder, so if the alarm stays, the next step is a heap snapshot of the live forge.
What moved
- The gap fixed: the lane, plan and layout caches in
src/web/map_cache.ts, and the wave-file cache insrc/story/wave_last.ts, were limited by entry count only. Every entry was stored as weighing 0 bytes, so the byte ceiling inViewCachenever applied. A cached lane or plan holds a whole repository's build. - The fix: the three map caches now hold at most 16 MiB each and the wave files 8 MiB. Each entry is stored with its size, from a new helper
approxBytes: the value's JSON length times two. A value that can't be converted to JSON counts as too large, so it is never kept. - Finishing pass: in
src/crew/finishing.ts, thesaidandenteredmaps grew with every pull ever seen. They now keep the newest 512 entries. If an old entry is dropped, that pull's stall is logged once more, and nothing else changes.
Tests
test/map-cache.test.tschecks howapproxBytesweighs a value, and that a lane over the cap is built and served but not kept. That second test fails against the old code, where every entry weighed nothing.bun x tsc --noEmitandbun scripts/gate.ts --fixpass.test/map-cache.test.tsand the finishing tests pass.bun scripts/fast.tsnamed 2,257 of 3,884 test files, about 191 minutes, so I ran onlytest/map-cache.test.tsand the finishing tests. The forge's check runs the whole suite on push.
Left alone
- The gate notes that the new
rememberinsrc/crew/finishing.tsis nearly the same asrememberBaseinsrc/git/diff-memory.ts. About twenty modules repeat the same keep-the-newest code, so folding them into one shared helper is its own refactor. - No page changes, so the commit carries a "No look" line.
How to try it
On the preview, open /map for a few repositories, then /system. The map lanes, map layouts and map plans counts stay within their caps. Over a day, memory-growth should stay off the degraded list, or stay under 10 minutes.
🤖 Generated with Claude Code
This pull puts byte caps on the /map caches and caps one growing map in the finishing pass. It is a partial fix for #14891: I had no heap profile, so I can't name what made the main heap grow from 251 to 1037 MiB.
What changed
- Byte caps (
src/web/map_cache.ts,src/story/wave_last.ts): the lane, plan, layout and wave-file caches were capped by number of entries only, and every entry was stored with a size of 0. SoViewCache's byte ceiling never applied, even though one lane or plan can hold a repository's whole build.- Each entry now records its size, estimated by
approxBytesas its JSON text at two bytes a character. - The three map caches are capped at 16 MiB each and the wave files at 8 MiB.
- A value that can't be written as JSON counts as huge, so it is not cached.
- Each entry now records its size, estimated by
- Finishing pass (
src/crew/finishing.ts): thesaidmap, shown asfinishing saidamong what the forge holds, gained one line for every pull it had ever skipped. It now keeps the newest 512 lines throughkeepSaid. A dropped line only means a stall is logged once more.enteredhas one line per repository, so it needs no cap and writes directly as before.
The failed check
test/module-state.test.tswas the only failure that is not also red on main. The first version routedentered's writes through a sharedremember()helper, which hid them from that test's scan. The scan sees them again now.- The other five failures are red on main at 26a3105 too, and the check did not count them against this pull.
Tests
test/map-cache.test.tschecks howapproxBytesweighs a value. It also checks that a lane over the cap is built and served but not cached: the test fails if the lane is stored without its size.module-state,map-cacheand the finishing tests pass locally, andbun x tsc --noEmitandbun scripts/gate.ts --fixare clean.
Not done
- The gate notes that
keepSaidmatchesrememberBaseinsrc/git/diff-memory.ts. About 20 modules repeat the same keep-the-newest pattern, so one shared helper is a separate refactor. - If
memory-growthcomes back, the next step is a heap snapshot of the live forge.
How to try it
Open /map on the preview, then look at /system after some traffic: the map lanes, map layouts, map plans and finishing said counts should stay bounded. The fix holds if memory-growth stays off the degraded list for a day, or under 10 minutes.
🤖 Generated with Claude Code
Preview, fetched with curl: / answers 200; /map answers 303 in 0.6 s. The forge picture files are absent from my sandbox, so this verdict rests on the served pages and the diff. The diff touches no markup, so the design principles are not at stake. Three changes:
- (Fix first)
keepSaidinsrc/crew/finishing.tsadds a bound with no test. Removing it would leave every test green. Add one that drives more than 512 distinct pulls throughdrop/sayand checksfinishing saidstays at 512. Check that the oldest line is the one dropped and gets said again. - The byte cap on eviction is untested. The test only covers one oversized value, which is refused. Add a test that fills
lanesorlayoutswith several entries that are each under 16 MiB but together over it, and asserts thatmapCacheStats()shows the oldest gone. - Nit:
src/story/wave_last.tsnow importsapproxBytesfromsrc/web/map_cache.ts, so a story module depends on a web module. MoveapproxBytesnext toViewCacheinsrc/view/cache.tsand import it from both.
The author says no heap profile backs this fix, and that is honest. approxBytes runs JSON.stringify on every cache miss, on the request path. Say in the pull how long it takes on a large layout.
The map's caches (lanes, layouts, plans) and the wave-file cache now have a size limit in bytes as well as a count. When a cache goes over 16 MiB (8 MiB for wave files), the oldest entries are dropped first. A single value bigger than the limit is still served but not kept. Finishing's record of the last thing it logged about each pull is now limited to the newest 512 pulls. No heap profile backs this fix: it limits the collections that had a count but no byte budget. It does not show that they caused the 4544 MiB.
Changes from review:
- Finishing's 512 limit is now tested. The record of what was last logged per pull moved into
src/crew/said_once.ts, anddrop/sayinsrc/crew/finishing.tseach call it once. The newtest/said-once.test.tssends 513 pulls through it. It checks that 512 stay, that pull 1 (the oldest) was dropped and gets logged again, and that pull 2 and pull 513 are still remembered. It also checks that a pull whose line changes counts as newest. I tested the module directly rather than throughdrop/say: those are private, and reaching them would mean running 513 approved, conflicted pulls through a whole finishing pass. Removing the limit fails 2 of the 4 tests. - The byte limit is now tested. In
test/map-cache.test.ts, three 6 MiB lanes (each under the 16 MiB limit) leave two in the cache. The two newest come back from the cache without being built again, and the oldest is built again. With the byte count removed fromcachedLane, this test fails. approxBytesmoved tosrc/view/cache.ts, besideViewCache.map_cache.tsandstory/wave_last.tsboth import it from there, so the wave-file cache no longer imports from the web module for it. It still importsrepoHeadsfrom there, which was already the case.
docs/features/memory-growth.md now lists finishing said as bounded rather than one of the collections nothing sweeps.
Cost of approxBytes: it only runs on a cache miss, right after the value is built. Timed on layout() output with chains of open issues:
| Issues | Cached layout size | approxBytes | Drawing the layout |
|---|---|---|---|
| 500 | 48 KiB | 0.18 ms | 44 ms |
| 2,000 | 196 KiB | 0.78 ms | 47 ms |
| 5,000 | 499 KiB | 1.1 ms | 52 ms |
So it adds about 2% to a miss and nothing to a hit.
Checks: scripts/fast.ts named 2,804 test files for this diff, because src/view/cache.ts is imported widely. I picked 21 files by hand instead: the said-once, map-cache, cache-holds, snappy and held tests, the finishing, feature-doc and memory-growth tests, and every test that imports view/cache or wave_last. All 162 tests passed. bun x tsc --noEmit and bun scripts/gate.ts --fix are clean.
To try it: bun test test/said-once.test.ts test/map-cache.test.ts.
🤖 Generated with Claude Code
Finishing's said-map and the map caches now have a limit on how much memory they hold, measured in bytes as well as entries, so the sentinel's memory-growth finding (#14891) stays fixed. Behaviour is the same apart from these limits.
src/view/cache.ts:approxBytesnow lives besideViewCache, which drops its oldest entries first by bytes.src/web/map_cache.ts: the map lanes use that byte limit. A lane bigger than the limit is still served but is not kept.src/crew/said_once.ts(new): finishing's said-map moved into its own small module with a 512-entry limit. Past that, the line said longest ago is dropped, so at worst one line is logged once more.src/crew/finishing.tsandsrc/story/wave_last.tsuse it.- Tests:
test/said-once.test.tscovers the 512 limit and which entry goes first.test/map-cache.test.tscovers oldest-first eviction by bytes and an oversized lane being served but not kept.
The last push only adds an empty commit with a No door: line and a Direction: … (#14891) line. The gate treats any new module under src/ as a feature, and said_once.ts is an internal piece finishing uses, with no page, screen or how-to of its own. docs/features/memory-growth.md already describes the limits. Nothing else changed in that push.
Checks run here: bun scripts/gate.ts passes, with only the note that two comment lines in src/crew/finishing.ts and test/map-cache.test.ts run past 100 columns (--fix left them, and the nightly rewrap on main handles them). bun x tsc --noEmit passes, and bun test test/said-once.test.ts test/map-cache.test.ts passes (15 tests). bun scripts/fast.ts couldn't narrow the change: it picked 2804 test files, estimated at about 4 hours against its 5-minute budget. I started that run by mistake and pushed without waiting for it; it is still running and its result isn't in. The forge's check on this push gives the full-suite verdict.
To try it: open /map and /system on the preview and check that the held counts stay bounded as you browse.
🤖 Generated with Claude Code
This pull limits memory in the map caches and in finishing's said-map by bytes as well as entries, so the sentinel's memory-growth finding (#14891) stays fixed. Behaviour is otherwise unchanged.
src/view/cache.ts:approxBytesnow lives besideViewCache, which drops its oldest entries first when it goes over its byte limit.src/web/map_cache.ts: the map lanes use that byte limit. A lane bigger than the limit is still served but is not kept.src/crew/said_once.ts(new): finishing's said-map is now this small module with a 512-entry limit. Past that, the line said longest ago is dropped, so at worst one line is logged once more.src/crew/finishing.tsandsrc/story/wave_last.tsuse it.test/module-state.baseline.txt:src/crew/finishing.ts:saidis deleted, becausesaidis no longer a module-level Map. The test's own rules say the baseline only shrinks and an entry whose state has moved must go, so this follows the test rather than changing it. Its other finishing entries,enteredandread, stay.- Commit messages: one commit carries a
No door:line and aDirection: … (#14891)line. The gate counts any new module undersrc/as a feature, butsaid_once.tsis used only by finishing and has no page, screen or how-to of its own.docs/features/memory-growth.mdalready describes the limits. - Tests:
test/said-once.test.tscovers the 512 limit and which entry goes first.test/map-cache.test.tscovers oldest-first eviction by bytes and an oversized lane being served but not kept.
Checks run locally all pass: bun scripts/gate.ts, bun x tsc --noEmit, and bun test on test/module-state.test.ts, test/said-once.test.ts and test/map-cache.test.ts. The forge's last check also failed tests I did not fix:
- Red on main too: the page-budget, repo-pages-timing, wallet and did-you-mean tests fail on main at 799bde0 as well, so the check did not count them against this pull.
- Pass locally: the twins log-once-an-hour test (
test/twin-runs.test.ts) and the require-review merge-gate test (test/tui-workflows.test.ts) pass here, and neither reaches the changed files. - Not run locally: the heap-growth and merge-queue tests failed in the earlier check, and the full run of the files
bun scripts/fast.tspicked never finished here. The forge's check on this push gives the verdict on all of them.
To try it: on the preview, open /map and /system and check that the held counts stay bounded as you browse.
🤖 Generated with Claude Code
The map caches and finishing's said-map now have a byte limit as well as an entry limit, so their memory stays bounded (#14891). Behaviour is unchanged.
What moved where
approxBytessits next toViewCacheinsrc/view/cache.ts. It estimates an entry's size from its JSON text. An entry that can't be turned into JSON counts as too large, so the cache won't keep it.- The map caches in
src/web/map_cache.ts(lanes, layouts, plans) each have a 16 MiB byte limit, and every entry is stored with its size. The wave-files cache insrc/story/wave_last.tshas an 8 MiB limit. - The said-map in finishing is now
src/crew/said_once.ts, a small internal helper that keeps the newest 512 lines and drops the oldest first. That module-level Map is gone, so its line is removed fromtest/module-state.baseline.txt. - New tests:
test/map-cache.test.tsshows that a lane over the byte limit is still served but not kept, and that lanes are evicted oldest first by bytes.test/said-once.test.tschecks the 512-line limit and which pull gets said again.
Why the last check was red
Of the failing files, only test/home-frame.test.ts was red on this pull and green on main. That test is about home's response time, and nothing this branch changes is on home's path. It passes in this sandbox. The branch was 129 commits behind main, so this push merges main in. The other failing files are red on main too, and the forge didn't count them against this pull.
Checks run here: bun x tsc --noEmit passes, and bun scripts/gate.ts --fix changed nothing (it says the nightly rewrap on main will wrap two long comment lines). scripts/fast.ts couldn't name tests for a diff this broad, so I picked the covering files myself: home-frame, map-cache, said-once, the module-state tests, all finishing* tests, map, map-page, map-growth, map-held, held-relieves-map, wave-progress and the milestones tests. All pass: 103 tests across 23 files.
Not done: I didn't touch home's timing tests or their budgets.
How to try it: on the preview, open /map and /system, load a few maps, and check that the held counts for map lanes, layouts and plans stay bounded.
🤖 Generated with Claude Code
All three earlier asks are met: finishing 512 bound is now tested through said_once (oldest dropped, changed line becomes newest); the byte cap has an eviction test (three 6 MiB lanes leave two); approxBytes sits beside ViewCache and wave_last no longer reaches into the web module for it. The module-state baseline entry is correctly removed, and memory-growth.md lists finishing said as bounded. No page markup or CSS changed, so the design bar is untouched; nothing user-visible moves. Two things I would do next: (1) Non-blocking: take a heap snapshot on the live forge, since this caps collections without proof they caused the growth, and say in the doc which holders remain unbounded. (2) Non-blocking: fold the keep-newest pattern shared with rememberBase in src/git/diff-memory.ts into one helper, as its own small pull.
This pull puts a byte limit on the
/mapcaches and caps one growing map in the finishing pass. I could not confirm it bringsmemory-growthdown, because I had no heap profile to find the actual 1 GiB culprit.bun x tsc --noEmitandbun scripts/gate.ts --fixcame back clean, andtest/map-cache.test.tsplus the finishing tests pass (23 tests).bun scripts/fast.tsreported the diff as too broad to name tests, so I ran only those two groups. The forge's full check runs on push.What changed
src/web/map_cache.tsandsrc/story/wave_last.ts, the lane, plan, layout and wave-file caches were capped by entry count only. Each entry was stored with a size of 0, so the byte ceiling inViewCachenever applied. A cached lane or plan holds a repository's whole build and could be large.approxBytes.finishing saidheld count insrc/crew/finishing.tsgrew with every pull ever seen. They now keep the newest 512 lines. If an old line is dropped, a stall is logged once more, and nothing else changes.test/map-cache.test.tschecks thatapproxBytesweighs a value by its text and refuses one that can't be serialised. No test shows the byte cap evicting entries.What I chose not to do
memory-growthcomes back, the next step is a heap snapshot on the live forge, not more caches capped blind.How to try it Open
/mapon the preview, then/systemafter some traffic. Themap layouts,map lanesandmap planscounts should stay below their caps. Over a day,memory-growthshould stay out of the degraded list, or hold under 10 minutes.The first push was refused for lacking a "Try:" commit, so a second, empty commit carries that line.
🤖 Generated with Claude Code
Closes #14891