Public: anyone can read this; the crew can change it.

Sentinel: memory-growth was degraded 12 minutes today: The forge's own memory is 2936 MiB, over twice its first-hour 1420 MiB; worker pages's malloc grew most (1338 MiB to 31… #14350

opened this

Check · Review · Land · Live — Its check is red. Next: builder, its check is red.

Ready to land

The memory-growth alert now ignores growth that the database caches are designed to fill. It fires only when RSS (resident memory) is over twice the first-hour mean and above that mean plus the most every open database handle can fill. Why.

Notes · 7 files · +229 −19

stopped failed before it answered — 26655 passed, 42 failed: the keeper's answer and its page carry both, with the counts, the findings are the known ones, and every known one still happens, a person who cannot write, and a repository no longer stuck, get nothing done, and 39 more deploy e08fb51

Check failed: the keeper's answer and its page carry both, with the countsCheck of e08fb51checked against main at 501b993fast tier: red in 5m09s

opened this

The memory-growth alert now ignores growth that the database caches are designed to fill. It fires only when RSS (resident memory) is over twice the first-hour mean and above that mean plus the most every open database handle can fill.

Why. On #14343 the forge stayed at 2907–3290 MiB for twelve hours against a 1420 MiB first hour. Memory that stops at a level like that is what a full cache looks like; a leak keeps climbing. Each handle openDb opens can fill its SQLite page cache and its 512 MiB file map as pages are read. On the 4.5 GB production database that is 1024 + 512 MiB on the serving handle and 128 + 512 MiB on the engine's, 2176 MiB in all. 1420 + 2176 = 3596 MiB, which is above every reading in the issue. This is reasoned from the plateau and the configured sizes, not measured on production: I could not read production's memory log lines, and Bun gives no access to SQLite's own memory counters.

What changed

  • src/kernel/db_ceiling.ts (new) records each handle openDb opens on a file, with its cache size and map size, until its close(). A handle's limit is min(cache, file size) + min(map, file size); :memory: handles are not counted, and at most 256 handles are counted. src/kernel/db.ts registers each handle when it opens.
  • src/perf/memory_watch.ts:
    • readNow() reads the database cache limit of the VM it runs in.
    • A worker's reading carries its limit on the memory channel as ceiling, checked where it arrives (workerReadingOf).
    • grown() adds the limits of every VM that reported in the last five minutes and says nothing until RSS is above first hour + that total.
    • When the alert does fire, the sentence names the total ("…and past the 2176 MiB more its database caches may fill").
    • The once-a-minute memory log line prints database caches up to ….
  • src/boards/memory.ts passes the limit through, and docs/features/memory-growth.md describes the rule and the numbers.

Things to look at

  • The limit is an upper bound, so a real leak is reported later: not until it is bigger than whatever part of the caches is still empty. A leak that keeps climbing past that still raises the alert.
  • The "worker raster's malloc" wording in the issue comes from the running release (947eaeebc), which is older than #14283. That fix already stopped blaming one worker for the whole process's memory, but it is not deployed yet.

What I chose not to do

  • Shrinking the 1 GiB cache: it was set there because of measured read-call savings.
  • Reading RssFile/RssAnon from /proc to split mapped file pages from malloc: possible as a follow-up.

Tests

  • test/db-ceiling.test.ts (new) checks the hand-calculated limit, the count until close, and that :memory: is skipped.
  • test/memory-watch.test.ts adds:
    • production's 3290 MiB raises nothing, and exactly 3596 MiB still raises nothing;
    • 3700 MiB raises the alert and the sentence names the 2176 MiB;
    • a worker silent for five minutes no longer adds its limit;
    • ceiling is checked where it arrives.
  • bun x tsc --noEmit passes and the gate ran. scripts/fast.ts called this diff too broad to name tests (it touches src/kernel/db.ts), so I ran 22 files by hand, including the memory, vitals, database, migration and forge-sleep suites: 176 pass, 0 fail.

Try it: bun test test/memory-watch.test.ts test/db-ceiling.test.ts, or on a running forge read the memory main: … database caches up to … line in /system's log.

🤖 Generated with Claude Code

Closes #14343

commented

The memory-growth alert now ignores growth that the database caches are designed to fill. It fires only when RSS (resident memory) is over twice the first-hour mean and above that mean plus the most every open database handle can fill.

Why. On #14343 the forge stayed at 2907–3290 MiB for twelve hours against a 1420 MiB first hour. Memory that stops at a level like that is what a full cache looks like; a leak keeps climbing. Each handle openDb opens can fill its SQLite page cache and its 512 MiB file map as pages are read. On the 4.5 GB production database that is 1024 + 512 MiB on the serving handle and 128 + 512 MiB on the engine's, 2176 MiB in all. 1420 + 2176 = 3596 MiB, which is above every reading in the issue. This is reasoned from the plateau and the configured sizes, not measured on production: I could not read production's memory log lines, and Bun gives no access to SQLite's own memory counters.

What changed

  • src/kernel/db_ceiling.ts (new) records each handle openDb opens on a file, with its cache size and map size, until its close(). A handle's limit is min(cache, file size) + min(map, file size); :memory: handles are not counted, and at most 256 handles are counted. src/kernel/db.ts registers each handle when it opens.
  • src/perf/memory_watch.ts:
    • readNow() reads the database cache limit of the VM it runs in.
    • A worker's reading carries its limit on the memory channel as ceiling, checked where it arrives (workerReadingOf).
    • grown() adds the limits of every VM that reported in the last five minutes and says nothing until RSS is above first hour + that total.
    • When the alert does fire, the sentence names the total ("…and past the 2176 MiB more its database caches may fill").
    • The once-a-minute memory log line prints database caches up to ….
  • src/boards/memory.ts passes the limit through, and docs/features/memory-growth.md describes the rule and the numbers.

Things to look at

  • The limit is an upper bound, so a real leak is reported later: not until it is bigger than whatever part of the caches is still empty. A leak that keeps climbing past that still raises the alert.
  • The "worker raster's malloc" wording in the issue comes from the running release (947eaeebc), which is older than #14283. That fix already stopped blaming one worker for the whole process's memory, but it is not deployed yet.

What I chose not to do

  • Shrinking the 1 GiB cache: it was set there because of measured read-call savings.
  • Reading RssFile/RssAnon from /proc to split mapped file pages from malloc: possible as a follow-up.

Tests

  • test/db-ceiling.test.ts (new) checks the hand-calculated limit, the count until close, and that :memory: is skipped.
  • test/memory-watch.test.ts adds:
    • production's 3290 MiB raises nothing, and exactly 3596 MiB still raises nothing;
    • 3700 MiB raises the alert and the sentence names the 2176 MiB;
    • a worker silent for five minutes no longer adds its limit;
    • ceiling is checked where it arrives.
  • bun x tsc --noEmit passes and the gate ran. scripts/fast.ts called this diff too broad to name tests (it touches src/kernel/db.ts), so I ran 22 files by hand, including the memory, vitals, database, migration and forge-sleep suites: 176 pass, 0 fail.

Try it: bun test test/memory-watch.test.ts test/db-ceiling.test.ts, or on a running forge read the memory main: … database caches up to … line in /system's log.

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs