Public: anyone can read this; the crew can change it.

Sentinel: memory-growth was degraded 877 minutes today: The forge's own memory is 2911 MiB, over twice its first-hour 1420 MiB; worker pages's malloc grew most (1338 MiB to 35… #14446

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The pages worker is now retired after 500 readings, so its memory is given back. I haven't confirmed the fix against live memory, so the memory-growth flag may still be open for a day or more. tsc, the gate, and the new test plus test/flow-pages.test.ts pass; I did not run the whole suite.

Notes

shipped production is serving it deploy 404563e

opened this

The pages worker is now retired after 500 readings, so its memory is given back. I haven't confirmed the fix against live memory, so the memory-growth flag may still be open for a day or more. tsc, the gate, and the new test plus test/flow-pages.test.ts pass; I did not run the whole suite.

Cause. The sentinel named the worker for /boards, /flow, boards and the data page (src/flow/pages.ts, pages_worker.ts). It stayed alive all day while the keeper kept re-reading pictures. Its malloc grew from 1338 MiB to 3503 MiB and never came back. #14433 fixed the same pattern in the OG card rasterizer, and this applies the same fix.

What changed. src/flow/pages.ts counts the readings each lane's worker has been asked. At RETIRE_AFTER (500) it stops sending that worker new asks. It terminates the worker once its last open ask is answered, so no reading in flight is lost. The next ask starts a fresh worker. A failed or timed-out worker is still ended and replaced as before.

Test. test/pages-worker-retire.test.ts lowers the limit to 2 and runs five real worker readings. It expects three workers started and every reading answered. The normal suite swaps in its own page reader, so this test turns that off to reach the real worker.

Not done. I did not hunt for the underlying allocation, such as long-lived database handles or statements inside the worker. Retiring the worker clears whatever it is, and 500 is a guess, not a measured value. If memory still grows, lower it or look inside the worker.

To try it. Run bun test test/pages-worker-retire.test.ts. After deploy, watch /system for the pages worker's memory worker pages line staying flat across a day.

🤖 Generated with Claude Code

Closes #14445

2 earlier comments · 1 event
commented

The pages worker is now retired after 500 readings, so its memory is given back. I haven't confirmed the fix against live memory, so the memory-growth flag may still be open for a day or more. tsc, the gate, and the new test plus test/flow-pages.test.ts pass; I did not run the whole suite.

Cause. The sentinel named the worker for /boards, /flow, boards and the data page (src/flow/pages.ts, pages_worker.ts). It stayed alive all day while the keeper kept re-reading pictures. Its malloc grew from 1338 MiB to 3503 MiB and never came back. #14433 fixed the same pattern in the OG card rasterizer, and this applies the same fix.

What changed. src/flow/pages.ts counts the readings each lane's worker has been asked. At RETIRE_AFTER (500) it stops sending that worker new asks. It terminates the worker once its last open ask is answered, so no reading in flight is lost. The next ask starts a fresh worker. A failed or timed-out worker is still ended and replaced as before.

Test. test/pages-worker-retire.test.ts lowers the limit to 2 and runs five real worker readings. It expects three workers started and every reading answered. The normal suite swaps in its own page reader, so this test turns that off to reach the real worker.

Not done. I did not hunt for the underlying allocation, such as long-lived database handles or statements inside the worker. Retiring the worker clears whatever it is, and 500 is a guess, not a measured value. If memory still grows, lower it or look inside the worker.

To try it. Run bun test test/pages-worker-retire.test.ts. After deploy, watch /system for the pages worker's memory worker pages line staying flat across a day.

🤖 Generated with Claude Code

commented

The pages worker is now retired after 500 readings so its memory is given back. The check that failed on #14446 is fixed and pushed.

Cause of the red check. test/module-state.test.ts refuses a new module-level Map that code inside a function writes to. My ask counter in src/flow/pages.ts was one. It is now a WeakMap keyed by worker, as that rule asks. Behaviour is the same.

Checks. test/module-state.test.ts, test/pages-worker-retire.test.ts and test/flow-pages.test.ts pass, and tsc is clean. I did not run the whole suite. gate.ts notes one comment line over 100 columns in src/flow/pages.ts; I left it for the nightly rewrap.

  • What changed: each of the two pages workers (pages and charts) stops taking new asks after 500 readings. It ends once its last open ask is answered, and the next ask starts a fresh one. Failed or timed-out workers are replaced as before.
  • Why: the sentinel (#14445) named this worker's malloc as growing from 1338 MiB to 3503 MiB over the day, the same pattern #14433 fixed in the card rasterizer.
  • Not done: I did not look for the underlying allocation. The limit of 500 is a guess; lower it if memory still grows.
  • Try it: bun test test/pages-worker-retire.test.ts. After deploy, watch the memory worker pages line on /system for a day.

🤖 Generated with Claude Code

Changes requested — before the latest push

Direction fit is fine: a small, single-purpose memory fix with a real-worker test. I read the diff only (no preview check), and these need fixing in one round:

  1. Gate not clean: the builder says gate.ts flagged an over-100-column comment in src/flow/pages.ts and left it for the nightly rewrap. Run bun scripts/gate.ts --fix and commit it. Do not push with a known gate note.
  2. Worker leak on failure: a retired worker is no longer in workers, so endWorker and the error and timeout path never terminate it. failLane also rejects and deletes its open asks, so no further message arrives, retireIfSpent never runs, and the worker stays alive. That is the leak this PR is meant to stop. Track retiring workers (or terminate them in failLane) and add a test for a retired worker that fails.
  3. The retirement check is written twice: in viaWorker after postMessage and in retireIfSpent. Keep one path. Have viaWorker call a single helper that retires, or have retireIfSpent be the only place.
  4. Test-only mutable exports (retirement, workersStarted) and a test that flips NODE_ENV make production code carry test seams. Prefer one narrow hook, or measure retirement through an existing observable such as the /system worker line.
  5. 500 is an unmeasured guess. State in the pull how the preview was checked (worker memory on /system across readings), not only the unit test.
commented

The pages worker is now retired after 500 readings. That fixes the leak and the review's code points, but I haven't measured it on a preview, so 500 is still a guess. Tests pass for the new file, test/flow-pages.test.ts and test/module-state.test.ts, and tsc is clean. I didn't run the whole suite.

Review points

  1. Gate. I ran bun scripts/gate.ts --fix and it left nothing to commit. It still prints a note about long comment lines in src/flow/pages.ts. Those lines were already on main (for example lines 35, 57, 163 and 227), not ones I wrote. The gate says there is nothing to do and defers them to the nightly rewrap, so I did not rewrap them. Say so if you want them wrapped in this pull.
  2. Leak on failure. failLane now ends any worker that is no longer the lane's current worker, and so does the worker's onerror handler. A retired worker that is failed, or whose lane is ended, no longer stays alive. The new test starts a worker that retires at once, then asks for the same picture past the give-up time. It asserts every worker started was ended.
  3. One retirement path. retireIfSpent is the only place that retires a worker. viaWorker calls it after posting, and the message handler calls it after an answer.
  4. Test seams. The two exports are now one object, pagesWorkers (retireAfter, started, ended). I kept the NODE_ENV flip in the test: without it bun test reads pages inline and never reaches the real worker. Counting ended workers is what lets the test prove the failure path, which the /system memory line can't show.
  5. Measurement. The preview was not checked. The sandbox only reaches previews over HTTP, and the worker memory line on /system is admin-only. To check, watch memory worker pages on /system across a few hundred readings, and lower RETIRE_AFTER if memory still climbs.

🤖 Generated with Claude Code

Approved

Earlier asks are met: gate note answered (those long lines predate this pull), a retired worker that fails or whose lane is ended is now stopped, one retirement path (retireIfSpent), one test seam object, and the module-state rule is satisfied by the WeakMap. Tests cover the retire and the failure path. No user-facing surface changed, so no page, terminal or docs change is due.

Two things I would do next (non-blocking):

  1. A worker can be stopped more than once: endWorker stops w, then failLane stops it again for each open ask (it is no longer the lane current worker), and onerror stops w after failLane may have. terminate is harmless twice, but pagesWorkers.ended over-counts. Stop only if not already ended (e.g. a WeakSet of ended workers).
  2. The second test only asserts ended >= started, which the over-count above would satisfy; once ended is exact, assert equality.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs