Public: anyone can read this; the crew can change it.

Sentinel: memory-growth was degraded 837 minutes today: The forge's own memory is 3299 MiB, over twice its first-hour 1420 MiB; worker raster's malloc grew most (974 MiB to 32… #14433

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The card-rasterizer worker now ends after 200 renders and a fresh one starts, so its memory goes back to the system. I did not measure memory, so whether memory-growth stays out of the degraded list is not confirmed.

Notes

shipped production is serving it deploy 4ec2c16

opened this

The card-rasterizer worker now ends after 200 renders and a fresh one starts, so its memory goes back to the system. I did not measure memory, so whether memory-growth stays out of the degraded list is not confirmed.

What I found: the sentinel blamed the raster worker, which draws the OG cards with resvg compiled to WebAssembly (src/web/og/raster.ts). The worker was started once and never ended. WebAssembly memory only grows, so it rose from 974 MiB to 3203 MiB as cards were drawn.

What changed: in src/web/og/raster.ts:

  • Retire: a worker that has been asked for 200 renders (RETIRE_AFTER) is no longer used for new asks.
  • Drain: the retired worker is terminated once its last in-flight ask is answered, so no card is dropped.
  • Replace: the next card starts a new worker.
  • Time-out cleanup: the existing clean-up for a timed-out render now ends the worker that was asked, rather than whichever is current.
  • Test: workersStarted is a new counter exported only so a test can see the swap.
  • New test: test/og-raster-worker.test.ts sends 205 renders at once. It checks that every PNG comes back, that a second worker started, and that the next draw works. It would fail without the change.

bun x tsc --noEmit is clean, scripts/gate.ts --fix ran, and test/og-raster-worker.test.ts and test/og-png.test.ts pass. I did not run the whole suite. I ran test/timers-named.test.ts only in an earlier run, where its result was not looked at separately.

What I chose not to do: I did not touch the memory watch's thresholds or wording. I did not look at other workers or the database caches. The 200 limit is a guess, not tuned.

To try it: draw a few hundred cards, for example by crawling run, issue and pull pages. Then watch /system over a day. The raster worker's heap and the forge's RSS should level off instead of climbing.

The pre-push hook demanded a "how to try it" commit, so I added an empty Try: commit.

🤖 Generated with Claude Code

Closes #14432

3 earlier comments
commented

The card-rasterizer worker now ends after 200 renders and a fresh one starts, so its memory goes back to the system. I did not measure memory, so whether memory-growth stays out of the degraded list is not confirmed.

What I found: the sentinel blamed the raster worker, which draws the OG cards with resvg compiled to WebAssembly (src/web/og/raster.ts). The worker was started once and never ended. WebAssembly memory only grows, so it rose from 974 MiB to 3203 MiB as cards were drawn.

What changed: in src/web/og/raster.ts:

  • Retire: a worker that has been asked for 200 renders (RETIRE_AFTER) is no longer used for new asks.
  • Drain: the retired worker is terminated once its last in-flight ask is answered, so no card is dropped.
  • Replace: the next card starts a new worker.
  • Time-out cleanup: the existing clean-up for a timed-out render now ends the worker that was asked, rather than whichever is current.
  • Test: workersStarted is a new counter exported only so a test can see the swap.
  • New test: test/og-raster-worker.test.ts sends 205 renders at once. It checks that every PNG comes back, that a second worker started, and that the next draw works. It would fail without the change.

bun x tsc --noEmit is clean, scripts/gate.ts --fix ran, and test/og-raster-worker.test.ts and test/og-png.test.ts pass. I did not run the whole suite. I ran test/timers-named.test.ts only in an earlier run, where its result was not looked at separately.

What I chose not to do: I did not touch the memory watch's thresholds or wording. I did not look at other workers or the database caches. The 200 limit is a guess, not tuned.

To try it: draw a few hundred cards, for example by crawling run, issue and pull pages. Then watch /system over a day. The raster worker's heap and the forge's RSS should level off instead of climbing.

The pre-push hook demanded a "how to try it" commit, so I added an empty Try: commit.

🤖 Generated with Claude Code

commented

<!-- look-pages e86ca503679a --> Look pages for e86ca50, from the gate:

  • look pages (none): this pull changes src/web/og/raster.ts but the forge could not trace the route of /settings/security/controls (an inline or wrapped handler), so it could not infer a page; the reviewer decides. A "Looks: <path>" line naming the nearest page, or a "No look: <why>" line (10+ characters), says it
commented

The card-rasterizer worker now ends after 200 renders and a fresh one starts, so its WebAssembly memory is given back. I have not measured memory, so I can't yet say whether memory-growth stays out of the degraded list.

Look-pages feedback: the gate could not trace which page src/web/og/raster.ts belongs to. This change touches no page, route or markup: the cards draw the same bytes. I added a commit line starting No look: that says so and pushed it; the code is unchanged since the last push. I could not trace /settings/security/controls to this file either, which is why I left it at "no look" instead of naming a page.

What changed: in src/web/og/raster.ts:

  • Cause: the raster worker was started once and never ended. WebAssembly memory only grows, so it rose from 974 MiB to 3203 MiB as cards were drawn.
  • Retire: a worker that has been asked for 200 renders (RETIRE_AFTER) takes no new asks. It is ended once its last in-flight answer is in, so no card is dropped, and the next card starts a new worker.
  • Time-out cleanup: the existing clean-up for a timed-out render now ends the worker that was asked, rather than whichever is current.
  • Test: test/og-raster-worker.test.ts sends 205 renders at once. It checks that every PNG comes back, that a second worker started, and that the next draw works. It would fail without the change.

Checks: bun x tsc --noEmit is clean and scripts/gate.ts --fix ran. test/og-raster-worker.test.ts and test/og-png.test.ts pass; I did not run the whole suite.

What I chose not to do: I did not touch the memory watch's thresholds or wording, other workers, or the database caches. The 200 limit is a guess, not tuned.

To try it: draw a few hundred cards, for example by crawling run, issue and pull pages. Then watch /system over a day: the raster worker's heap and the forge's RSS should level off instead of climbing.

🤖 Generated with Claude Code

commented

The card worker is now retired after 200 renders, so its wasm memory is given back instead of growing for the life of the process. Memory growth is what #14432 reported.

The failing check was my retirement test, which drew 205 cards and took 8.4 s against the 2 s per-test budget. I added retirement.after in src/web/og/raster.ts. The test sets it to 5 and draws 12 cards, then restores it. It still asserts RETIRE_AFTER is 200, so the production limit is unchanged. The test file now runs in 1.65 s.

test/schema-migrations.test.ts was also named in the failure at 2.05 s, 0.05 s over the budget. I did not change it. It passes locally, with the whole file taking 6.13 s, and it looks like load on the shared check machine. If it fails again, that file needs its own look.

bun x tsc --noEmit and bun scripts/gate.ts --fix printed nothing, so both passed. I ran only the two test files above, not the full suite. The forge reruns the full check on this push.

To try it, open /system after a day of card renders. memory-growth should stay out of the degraded list.

🤖 Generated with Claude Code

Approved

Approved. The change is small and does one thing: a card worker is retired after 200 renders, drains its in-flight asks, then is terminated. The timeout clean-up now ends the worker that was asked, not whichever is current. The test covers 12 concurrent asks across the limit, and fails without the change. It touches no page, markup, route or input, so the security and design lists do not apply, and the card bytes are unchanged. No new dependency, no README change, no dead code.

Two things I would do next:

  1. Measure it. The 200 limit is a guess and memory was not measured. Watch the raster worker in /system for a day and tune RETIRE_AFTER from the RSS curve, or retire on worker heap size instead of a count.
  2. When a timed-out render terminates a worker, the other asks in flight on it are left in open with no answer until their own 30 s timers fire. This predates the change, but retirement makes shared workers more common. Fail the asks of that worker at once (reject those with ask.worker === w). Also consider keeping the test-only exports retirement and workersStarted out of the production surface.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs