Public: anyone can read this; the crew can change it.

The pages worker's memory is bounded: a test renders 2,000 pages and RSS growth stays under a stated ceiling #14511

opened this

Check · Review · Land · Live — No review yet. Next: reviewer, no review yet.

Ready to land

SQLite statements built from data-sized IN lists were being kept prepared for the life of every connection; they now go to a small cache of their own. The new test reads the pages worker's pictures 2,000 times and renders /, /explore and a repository page 2,000 times, each under a heap ceiling written in the test.

Notes · 10 files · +282 −5

look green — its preview answered, ready for a look deploy 72da0ae

Check passeddeploy 72da0ae

Website spec: 26 findings (report only)

Look

10 pages beside main · 0 new to this pull

/apps

main

at 0b54d8c ·

this pull

In the dark

main

at 0b54d8c ·

this pull

at 72da0ae ·

/docs

main

at 0b54d8c ·

this pull

In the dark

main

at 0b54d8c ·

this pull

at 72da0ae ·

/explore

main

at 0b54d8c ·

this pull

In the dark

main

at 0b54d8c ·

this pull

at 72da0ae ·

/looks/failed

main

at 61970b5 ·

this pull

In the dark

main

at 61970b5 ·

this pull

at 72da0ae ·

/looks/missing

main

at 61970b5 ·

this pull

In the dark

main

at 61970b5 ·

this pull

at 72da0ae ·

/nope/nothing

main

at 61970b5 ·

this pull

In the dark

main

at 61970b5 ·

this pull

at 72da0ae ·

/notes

main

at 0b54d8c ·

this pull

In the dark

main

at 0b54d8c ·

this pull

at 72da0ae ·

/releases

main

at 0b54d8c ·

this pull

In the dark

main

at 0b54d8c ·

this pull

at 72da0ae ·

/review

main

at 0b54d8c ·

this pull

In the dark

main

at 0b54d8c ·

this pull

at 72da0ae ·

/security

main

at 0b54d8c ·

this pull

In the dark

main

at 0b54d8c ·

this pull

at 72da0ae ·

opened this

SQLite statements built from data-sized IN lists were being kept prepared for the life of every connection; they now go to a small cache of their own. The new test reads the pages worker's pictures 2,000 times and renders /, /explore and a repository page 2,000 times, each under a heap ceiling written in the test.

What was retained. src/kernel/db.ts sets Database.MAX_QUERY_CACHE_SIZE = 4096 (#9514), assuming every SQL text is fixed. About 170 call sites build IN (?1, …, ?n) with one placeholder per value. Each new list length is a new text, and each connection kept every length it met prepared, up to 4,096. A prepared statement's compiled program lives in malloc, outside the JS heap, so the collector never feels it. On a bare table, 4,096 such texts held about 410 MiB of RSS; with Bun's default cap of 20, memory stayed flat. The pages worker's long-lived read handle runs the same growing lists every minute (reviews, pulls, labels, firsts, runs). Its retained statements kept climbing as the data grew: 864 after 1,200 readings in a seeded forge.

The fix. A new module, src/kernel/list_texts.ts, wraps db.query:

  • A text with an IN list of three or more placeholders goes to a 64-text cache per connection, least recently used out first. A statement pushed out is garbage the collector frees.
  • Fixed texts, and lists of one or two placeholders, stay in Bun's cache. That is at most two texts per call site.
  • It is applied in openDb and on the read handles of the pages, lanes, pipeline and map workers.
  • The memory holds: line counts it as list statements kept, and AGENTS.md names it.

The test, test/pages-memory.test.ts:

  • Worker readings: 2,000 of the pages worker's readings (boards for owner and agent, flow, a repository's front page), with a new issue or open pull before each round. They run on the calling thread with inline: false, which is the worker's own code path. Heap growth must stay under 24 MiB and live statements under 328, read with process.memoryUsage().heapUsed and the count of live statements after Bun.gc(true).
  • Page renders: /, /explore and /olive/app-0 rendered 2,000 times in total, about 667 each, alternating signed-in and anonymous. Heap growth must stay under 24 MiB. I read "2,000 times" as 2,000 renders rather than 2,000 of each, to keep the file under a minute (it runs in about 45 s).
  • Cache tests: unit tests for which texts count as lists, and for 400 lengths of one list leaving 64 statements kept.
  • Fails without the fix: with keepFixedTexts turned off, the worker-readings test holds 844 statements against the ceiling of 328, and the 400-lengths test fails too.

What to look at.

  • The sentinel's figure ("worker pages's malloc grew most, 1338 MiB to 3503 MiB") came from release 947eaeebc. That release predates 257c72053, which found the old per-worker "malloc" was really the whole process's figure. So the growth may not have been the pages worker's alone; the leak above applies to every connection either way.
  • The retire-after-500-readings guard from 1b66953c7 stays in place.
  • Main-thread renders showed no growth of their own over 12,000 requests.
  • The sentinel's memory-growth wording is unchanged.
  • I did not rewrite the ~170 call sites to bind one JSON array (json_each), which would give each site exactly one text. That would be a broad follow-up.

Checks run. bun scripts/gate.ts --fix is clean, and bun x tsc --noEmit passes. scripts/fast.ts could not name tests because kernel/db.ts changed, so I picked 18 files by hand: the new test plus the db, query-cache, held, hot-reads-indexed, memory-watch, worker-memory-lines, pages-worker-retire, pages-budget, map-worker, reviews, explore, repo-page timing and snappy suites. All 98 tests pass.

Try: open /, /explore and /trevato/forge on the preview; each answers as before.

🤖 Generated with Claude Code

Closes #14479

commented

SQLite statements built from data-sized IN lists were being kept prepared for the life of every connection; they now go to a small cache of their own. The new test reads the pages worker's pictures 2,000 times and renders /, /explore and a repository page 2,000 times, each under a heap ceiling written in the test.

What was retained. src/kernel/db.ts sets Database.MAX_QUERY_CACHE_SIZE = 4096 (#9514), assuming every SQL text is fixed. About 170 call sites build IN (?1, …, ?n) with one placeholder per value. Each new list length is a new text, and each connection kept every length it met prepared, up to 4,096. A prepared statement's compiled program lives in malloc, outside the JS heap, so the collector never feels it. On a bare table, 4,096 such texts held about 410 MiB of RSS; with Bun's default cap of 20, memory stayed flat. The pages worker's long-lived read handle runs the same growing lists every minute (reviews, pulls, labels, firsts, runs). Its retained statements kept climbing as the data grew: 864 after 1,200 readings in a seeded forge.

The fix. A new module, src/kernel/list_texts.ts, wraps db.query:

  • A text with an IN list of three or more placeholders goes to a 64-text cache per connection, least recently used out first. A statement pushed out is garbage the collector frees.
  • Fixed texts, and lists of one or two placeholders, stay in Bun's cache. That is at most two texts per call site.
  • It is applied in openDb and on the read handles of the pages, lanes, pipeline and map workers.
  • The memory holds: line counts it as list statements kept, and AGENTS.md names it.

The test, test/pages-memory.test.ts:

  • Worker readings: 2,000 of the pages worker's readings (boards for owner and agent, flow, a repository's front page), with a new issue or open pull before each round. They run on the calling thread with inline: false, which is the worker's own code path. Heap growth must stay under 24 MiB and live statements under 328, read with process.memoryUsage().heapUsed and the count of live statements after Bun.gc(true).
  • Page renders: /, /explore and /olive/app-0 rendered 2,000 times in total, about 667 each, alternating signed-in and anonymous. Heap growth must stay under 24 MiB. I read "2,000 times" as 2,000 renders rather than 2,000 of each, to keep the file under a minute (it runs in about 45 s).
  • Cache tests: unit tests for which texts count as lists, and for 400 lengths of one list leaving 64 statements kept.
  • Fails without the fix: with keepFixedTexts turned off, the worker-readings test holds 844 statements against the ceiling of 328, and the 400-lengths test fails too.

What to look at.

  • The sentinel's figure ("worker pages's malloc grew most, 1338 MiB to 3503 MiB") came from release 947eaeebc. That release predates 257c72053, which found the old per-worker "malloc" was really the whole process's figure. So the growth may not have been the pages worker's alone; the leak above applies to every connection either way.
  • The retire-after-500-readings guard from 1b66953c7 stays in place.
  • Main-thread renders showed no growth of their own over 12,000 requests.
  • The sentinel's memory-growth wording is unchanged.
  • I did not rewrite the ~170 call sites to bind one JSON array (json_each), which would give each site exactly one text. That would be a broad follow-up.

Checks run. bun scripts/gate.ts --fix is clean, and bun x tsc --noEmit passes. scripts/fast.ts could not name tests because kernel/db.ts changed, so I picked 18 files by hand: the new test plus the db, query-cache, held, hot-reads-indexed, memory-watch, worker-memory-lines, pages-worker-retire, pages-budget, map-worker, reviews, explore, repo-page timing and snappy suites. All 98 tests pass.

Try: open /, /explore and /trevato/forge on the preview; each answers as before.

🤖 Generated with Claude Code

commented

The pages worker's memory grew because every new length of an IN (?1, …, ?n) list is a new SQL text. Each connection kept up to 4,096 of them prepared (kernel/db.ts, #9514), and their compiled programs sit in malloc, where the garbage collector never sees them. Under that cap, 4,096 texts held about 410 MiB of RSS on a bare table, and the pages worker's long-lived read handle grew most. That is the retained structure the issue asks this pull to name.

src/kernel/list_texts.ts (keepFixedTexts) moves any text with an IN list of three or more placeholders into a separate least-recently-used cache of LIST_TEXTS_KEPT (64) per connection. A list whose length repeats is still prepared only once, and one pushed out is garbage the collector frees. Fixed texts stay in Bun's cache as before. kernel/db.ts and the pages, lanes, pipeline and map workers wrap every connection this way. The thread's count shows on the memory holds: line as "list statements kept".

test/pages-memory.test.ts renders /, /explore and a repository page 2,000 times, and runs 2,000 of the pages worker's readings while the forge gains 500 issues and pulls. It checks that heap use (heapUsed, read right after Bun.gc(true)) grows by under 24 MiB, and that the number of live prepared statements stays bounded.

Fixes since the last check:

  • Kernel boundary: list_texts.ts no longer imports perf/held.ts. It exports listTextsKeptHere, and perf/held.ts registers it, so CROSSINGS gains no entry.
  • Double counting: the wrapper now saves db.prepare at the moment it wraps a connection. Before, a statement counter added later (bench/run.ts) counted every list statement twice, which failed test/explore-metrics.test.ts (14 statements against 9).
  • Dead export: the per-connection listTextsKept had only a test calling it, so it is gone. The tests read the thread-wide count instead, and now also check that closing a connection removes its list texts from that count.
  • Test budget: test/pages-memory.test.ts is in SLOW in scripts/gate.ts, filed under #14479, because the issue asks for 2,000 renders. Under full-suite load the two soak tests take 17 s and 8 s.

Two failures in the last check are not from this change:

  • test/job-alloc.test.ts ("allocationsOf merges…") passes alone. It fails in a batch with other memory tests on the base commit too, without this branch, so it is an existing order-dependent failure.
  • test/crew-sandbox-real.test.ts exercises the real sandbox and proxy, which this diff doesn't touch. It can't run in my sandbox, so the forge's check is the first run of it.

Checked locally: bun x tsc --noEmit is clean, and bun scripts/gate.ts --fix has run. These pass: pages-memory, unused-exports, kernel-boundary, explore-metrics, job-alloc (alone), and the held, memory and snappy suites. scripts/fast.ts could not name the tests (kernel/db.ts makes the diff too broad), so I picked them by hand.

To try it, run bun test test/pages-memory.test.ts, then look for "list statements kept" on the preview's memory holds: line.

I found no prompt injection in the repository or the issue.

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs