Public: anyone can read this; the crew can change it.
Sentinel: memory-growth was degraded 12 minutes today: The forge's own memory is 2936 MiB, over twice its first-hour 1420 MiB; worker pages's malloc grew most (1338 MiB to 31… #14350
Check · Review · Land · Live — Its check is red.
The memory-growth alert now ignores growth that the database caches are designed to fill. It fires only when RSS (resident memory) is over twice the first-hour mean and above that mean plus the most every open database handle can fill. Why.
stopped failed before it answered — 26655 passed, 42 failed: the keeper's answer and its page carry both, with the counts, the findings are the known ones, and every known one still happens, a person who cannot write, and a repository no longer stuck, get nothing done, and 39 more deploy e08fb51
Check failed: the keeper's answer and its page carry both, with the countsCheck of e08fb51checked against main at 501b993fast tier: red in 5m09s
The memory-growth alert now ignores growth that the database caches are designed to fill. It fires only when RSS (resident memory) is over twice the first-hour mean and above that mean plus the most every open database handle can fill.
Why. On #14343 the forge stayed at 2907–3290 MiB for twelve hours against a 1420 MiB first hour. Memory that stops at a level like that is what a full cache looks like; a leak keeps climbing. Each handle openDb opens can fill its SQLite page cache and its 512 MiB file map as pages are read. On the 4.5 GB production database that is 1024 + 512 MiB on the serving handle and 128 + 512 MiB on the engine's, 2176 MiB in all. 1420 + 2176 = 3596 MiB, which is above every reading in the issue. This is reasoned from the plateau and the configured sizes, not measured on production: I could not read production's memory log lines, and Bun gives no access to SQLite's own memory counters.
What changed
src/kernel/db_ceiling.ts(new) records each handleopenDbopens on a file, with its cache size and map size, until itsclose(). A handle's limit ismin(cache, file size) + min(map, file size);:memory:handles are not counted, and at most 256 handles are counted.src/kernel/db.tsregisters each handle when it opens.src/perf/memory_watch.ts:readNow()reads the database cache limit of the VM it runs in.- A worker's reading carries its limit on the memory channel as
ceiling, checked where it arrives (workerReadingOf). grown()adds the limits of every VM that reported in the last five minutes and says nothing until RSS is above first hour + that total.- When the alert does fire, the sentence names the total ("…and past the 2176 MiB more its database caches may fill").
- The once-a-minute memory log line prints
database caches up to ….
src/boards/memory.tspasses the limit through, anddocs/features/memory-growth.mddescribes the rule and the numbers.
Things to look at
- The limit is an upper bound, so a real leak is reported later: not until it is bigger than whatever part of the caches is still empty. A leak that keeps climbing past that still raises the alert.
- The "worker raster's malloc" wording in the issue comes from the running release (947eaeebc), which is older than #14283. That fix already stopped blaming one worker for the whole process's memory, but it is not deployed yet.
What I chose not to do
- Shrinking the 1 GiB cache: it was set there because of measured read-call savings.
- Reading
RssFile/RssAnonfrom/procto split mapped file pages from malloc: possible as a follow-up.
Tests
test/db-ceiling.test.ts(new) checks the hand-calculated limit, the count until close, and that:memory:is skipped.test/memory-watch.test.tsadds:- production's 3290 MiB raises nothing, and exactly 3596 MiB still raises nothing;
- 3700 MiB raises the alert and the sentence names the 2176 MiB;
- a worker silent for five minutes no longer adds its limit;
ceilingis checked where it arrives.
bun x tsc --noEmitpasses and the gate ran.scripts/fast.tscalled this diff too broad to name tests (it touchessrc/kernel/db.ts), so I ran 22 files by hand, including the memory, vitals, database, migration and forge-sleep suites: 176 pass, 0 fail.
Try it: bun test test/memory-watch.test.ts test/db-ceiling.test.ts, or on a running forge read the memory main: … database caches up to … line in /system's log.
🤖 Generated with Claude Code
The memory-growth alert now ignores growth that the database caches are designed to fill. It fires only when RSS (resident memory) is over twice the first-hour mean and above that mean plus the most every open database handle can fill.
Why. On #14343 the forge stayed at 2907–3290 MiB for twelve hours against a 1420 MiB first hour. Memory that stops at a level like that is what a full cache looks like; a leak keeps climbing. Each handle
openDbopens can fill its SQLite page cache and its 512 MiB file map as pages are read. On the 4.5 GB production database that is 1024 + 512 MiB on the serving handle and 128 + 512 MiB on the engine's, 2176 MiB in all. 1420 + 2176 = 3596 MiB, which is above every reading in the issue. This is reasoned from the plateau and the configured sizes, not measured on production: I could not read production's memory log lines, and Bun gives no access to SQLite's own memory counters.What changed
src/kernel/db_ceiling.ts(new) records each handleopenDbopens on a file, with its cache size and map size, until itsclose(). A handle's limit ismin(cache, file size) + min(map, file size);:memory:handles are not counted, and at most 256 handles are counted.src/kernel/db.tsregisters each handle when it opens.src/perf/memory_watch.ts:readNow()reads the database cache limit of the VM it runs in.ceiling, checked where it arrives (workerReadingOf).grown()adds the limits of every VM that reported in the last five minutes and says nothing until RSS is above first hour + that total.database caches up to ….src/boards/memory.tspasses the limit through, anddocs/features/memory-growth.mddescribes the rule and the numbers.Things to look at
What I chose not to do
RssFile/RssAnonfrom/procto split mapped file pages from malloc: possible as a follow-up.Tests
test/db-ceiling.test.ts(new) checks the hand-calculated limit, the count until close, and that:memory:is skipped.test/memory-watch.test.tsadds:ceilingis checked where it arrives.bun x tsc --noEmitpasses and the gate ran.scripts/fast.tscalled this diff too broad to name tests (it touchessrc/kernel/db.ts), so I ran 22 files by hand, including the memory, vitals, database, migration and forge-sleep suites: 176 pass, 0 fail.Try it:
bun test test/memory-watch.test.ts test/db-ceiling.test.ts, or on a running forge read thememory main: … database caches up to …line in/system's log.🤖 Generated with Claude Code
Closes #14343