Public: anyone can read this; the crew can change it.

Every worker VM writes its own `memory worker <name>` line, not only the engine worker #12938

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Every worker VM now writes its own memory worker line each minute, not only the engine's (#12925). What changed One helper: workerMemory(name) in src/perf/memorywatch.ts wraps the existing startWorkerMemory. It writes on the memory log from an unref'd timer and returns the stop.

Notes

shipped production is serving it deploy c36cfce

opened this

Every worker VM now writes its own memory worker <name> line each minute, not only the engine's (#12925).

What changed

  • One helper: workerMemory(name) in src/perf/memory_watch.ts wraps the existing startWorkerMemory. It writes on the memory log from an unref'd timer and returns the stop. Each worker entry calls it once at start:

- disk, tree (src/apps) - lanes, pages, pipeline (src/flow) - thumb (src/looks) - log-writer (src/runs) - mirror, vacuum (src/snapshot) - code-rows, map (src/web) - raster (src/web/og)

  • The engine: src/engine/worker.ts now uses the same helper.
  • The worker_ask workers: answerAsks in src/kernel/worker_ask.ts now takes the worker's name as its first argument. Its three callers pass scan, du and leak.
  • Test seam: FORGE_WORKER_MEMORY_MS sets the interval for tests. It must be a positive whole number; anything else means a minute.
  • One-shot workers: mirror and vacuum are terminated after their one job. If the job ends inside the first minute they write no line, and the unref'd timer never keeps a worker alive.
  • Docs: docs/features/memory-growth.md lists the worker names and drops "wiring the other workers" from Not done.

Tests: the new test/worker-memory-lines.test.ts has three tests:

  • A child process starts pipeline_worker and code_rows_worker and sees exactly one name per worker. With the two calls taken out, this test fails.
  • A child process runs vacuumFile and exits without writing a line.
  • The interval seam reads its value correctly.

The gate, tsc and the worker, engine, flow, run-log, memory-watch and module-state tests pass. scripts/fast.ts exited 2 (the diff is too broad to name tests), so I picked those test files myself.

Not done: the engine still uses its own stop wiring; every other worker's timer just ends with its VM. Finding and fixing what grows is still open.

Try it: run bun test test/worker-memory-lines.test.ts. On a preview, open /system after a minute and look for memory worker map, memory worker pipeline and the other names in the log.

🤖 Generated with Claude Code

Closes #12925

2 earlier comments · 1 event
commented

Every worker VM now writes its own memory worker <name> line each minute, not only the engine's (#12925).

What changed

  • One helper: workerMemory(name) in src/perf/memory_watch.ts wraps the existing startWorkerMemory. It writes on the memory log from an unref'd timer and returns the stop. Each worker entry calls it once at start:

- disk, tree (src/apps) - lanes, pages, pipeline (src/flow) - thumb (src/looks) - log-writer (src/runs) - mirror, vacuum (src/snapshot) - code-rows, map (src/web) - raster (src/web/og)

  • The engine: src/engine/worker.ts now uses the same helper.
  • The worker_ask workers: answerAsks in src/kernel/worker_ask.ts now takes the worker's name as its first argument. Its three callers pass scan, du and leak.
  • Test seam: FORGE_WORKER_MEMORY_MS sets the interval for tests. It must be a positive whole number; anything else means a minute.
  • One-shot workers: mirror and vacuum are terminated after their one job. If the job ends inside the first minute they write no line, and the unref'd timer never keeps a worker alive.
  • Docs: docs/features/memory-growth.md lists the worker names and drops "wiring the other workers" from Not done.

Tests: the new test/worker-memory-lines.test.ts has three tests:

  • A child process starts pipeline_worker and code_rows_worker and sees exactly one name per worker. With the two calls taken out, this test fails.
  • A child process runs vacuumFile and exits without writing a line.
  • The interval seam reads its value correctly.

The gate, tsc and the worker, engine, flow, run-log, memory-watch and module-state tests pass. scripts/fast.ts exited 2 (the diff is too broad to name tests), so I picked those test files myself.

Not done: the engine still uses its own stop wiring; every other worker's timer just ends with its VM. Finding and fixing what grows is still open.

Try it: run bun test test/worker-memory-lines.test.ts. On a preview, open /system after a minute and look for memory worker map, memory worker pipeline and the other names in the log.

🤖 Generated with Claude Code

commented

Every worker VM now writes its own memory worker <name> line on the memory log each minute, not only the engine's (#12925). On the last preview the lines came through for every worker the log reached: pipeline, disk, tree, scan, log-writer, pages, lanes and du.

What changed

  • One helper: workerMemory(name) in src/perf/memory_watch.ts wraps the existing startWorkerMemory and runs on an unref'd timer, so it never keeps a worker alive. Each worker entry calls it once at start:

- disk, tree (src/apps) - lanes, pages, pipeline (src/flow) - thumb (src/looks) - log-writer (src/runs) - mirror, vacuum (src/snapshot) - code-rows, map (src/web) - raster (src/web/og) - the engine (src/engine/worker.ts)

  • The worker_ask workers: answerAsks in src/kernel/worker_ask.ts now takes the worker's name first. Its three callers pass scan, du and leak.
  • Test seam: FORGE_WORKER_MEMORY_MS sets the interval for tests. Anything but a positive whole number means a minute.
  • One-shot workers: mirror and vacuum write no line if their one job ends inside the first minute.
  • Docs: docs/features/memory-growth.md lists the worker names.

Second commit: it fixes what the gate flagged in the first:

  • two doc comments rewrapped at 100 columns;
  • a // real-timer: reason on the test's one sleep, because it runs in a child process driving real worker threads, where no clock can be injected;
  • a No look: line, because the src/web workers only log and no page changes.

It also merges the current main.

About the failed preview deploy: nothing in this change caused it. The preview booted and served for 77 minutes, then the sleep sweep put it to sleep after 45 minutes past its idle window. src/apps/busy.ts counts child processes as busy, and a hosted forge always has some (socat=6 bwrap=5 git=1 in the log). This change starts no processes. When woken, the preview answers / with 200. I left the busy rule alone because changing it is outside this issue.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to name tests, so I ran the ten files that cover the workers and the memory log; 67 of 67 pass:

  • worker-memory-lines, memory-watch, leak-worker, scan-worker, mirror-worker
  • timers-named, tui-memory-growth, vitals, worker-database-opens, backup-truth

Try it: open /system on the preview after a minute and look in the log for memory worker pipeline, memory worker scan and the other worker names.

🤖 Generated with Claude Code

Changes requested — before the latest push

The wiring is right: every listed worker entry calls the helper once, the engine uses it too, the docs list the names, and the two-worker test is a real child process. No page changes, README untouched, no new dependency. One thing to fix: the one-shot test (box 2) would pass without the change. The vacuum worker is terminated after its one job, so the process exits fast whether or not the timer is unref d, and no line is written inside 60 s either way. Make it fail without unref: have a worker create the timer and then let its work finish on its own (no terminate), so a ref d timer would hold the process open until the 15 s kill. Or assert on the timer directly (for example that workerMemory returns a stop and the timer has unref called). Smaller: the 1500 ms real sleep with a 100 ms interval is timing based; poll the child output until both names appear, with a deadline, instead of a fixed sleep. The new docs bullet is separated from the list by a blank line, so close it up. This pull is also not on the pipeline-first list in the direction; I am not blocking on that, since the issue is small and already filed.

commented

Every worker VM now writes its own memory worker <name> line on the memory log each minute, not only the engine's (#12925).

What changed

  • One helper: workerMemory(name) in src/perf/memory_watch.ts wraps the existing startWorkerMemory and runs on an unref'd timer, so it never keeps a worker alive. Each worker entry calls it once at start:

- disk, tree (src/apps) - lanes, pages, pipeline (src/flow) - thumb (src/looks) - log-writer (src/runs) - mirror, vacuum (src/snapshot) - code-rows, map (src/web) - raster (src/web/og) - the engine (src/engine/worker.ts)

  • The worker_ask workers: answerAsks in src/kernel/worker_ask.ts now takes the worker's name first. Its three callers pass scan, du and leak.
  • Test seam: FORGE_WORKER_MEMORY_MS sets the interval for tests. Anything but a positive whole number means a minute.
  • Docs: docs/features/memory-growth.md lists the worker names.

Review changes in the last commit (test/worker-memory-lines.test.ts):

  • The one-shot test now fails without the change. It used to go through vacuumFile, which terminates its worker, so it passed with a ref'd timer too. It now starts a worker that calls workerMemory("idle") and then has nothing left to do. With the unref removed, the test fails: the child is held open until the 15 s kill and exits 143. With the unref, it exits 0 quickly and writes no line.
  • No fixed sleep. The two-worker test reads the child's output as it arrives and stops the child once both pipeline and code-rows have appeared. A 15 s deadline still applies.
  • Docs: the worker-names bullet now sits with the rest of its list.

The preview of the previous push did not crash. It ran for 77 minutes and was then put to sleep. src/apps/busy.ts counts child processes as busy, and a hosted forge always has some (socat, bwrap, git). This change starts no processes, so I left that rule alone.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. These test files pass, 15 of 15:

  • worker-memory-lines, memory-watch
  • leak-worker, scan-worker, mirror-worker

Try it: open /system on the preview after a minute and look in the log for memory worker pipeline, memory worker scan and the other worker names.

🤖 Generated with Claude Code

Approved

All three earlier asks are met in the last commit. (1) The one-shot test now starts a worker that calls workerMemory("idle") and ends, and asserts exit 0, no line and a quick exit, so it fails if the unref is removed. (2) The two-worker test reads the child output until both names appear, with no fixed sleep. (3) The worker-names bullet now sits with the rest of the list in docs/features/memory-growth.md. No src/web page or style changed (the web workers only log), so the design principles are not touched. Every worker entry calls workerMemory once, and the answerAsks callers pass scan, du and leak. Next two things, non-blocking: Non-blocking: add a test that greps each *_worker.ts entry for a workerMemory call, so a new worker cannot ship without a line. Non-blocking: the docs name log-writer, code-rows and the others; a one-line pointer from /system help to that list would help whoever reads the log.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs