Public: anyone can read this; the crew can change it.

On start the host re-adopts the cells it left running (record, probe, take over the registry entry) #13927

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

On start, the host now finds the cells a previous host left running and takes them over. bun x tsc --noEmit and bun scripts/gate.ts --fix are clean.

Notes

shipped production is serving it deploy 8e1361f

opened this

On start, the host now finds the cells a previous host left running and takes them over. bun x tsc --noEmit and bun scripts/gate.ts --fix are clean. The new test file passes, as do the other cell and timer tests I ran: cell-outlives-host, spawn-sleep-together, spawn-cell-stopping, first-ten-minutes, forges-memory, timers-named, spawn-sites and billing-never-deletes. I did not run the full suite; the forge runs it on push.

What changed

  • Record: src/spawn/cell_record.ts is new. It holds the record {bodyId, pid, port, supervisor, startedAt}, plus name and size, which I added so a host that didn't start the cell can still answer for it. The record lives at children/<bodyId>/cell.json. It is written through a rename and checked field by field when read, and a record whose bodyId differs from its directory is ignored.
  • Writing and clearing: startOn in src/spawn/cell.ts writes the record once a supervised cell answers (only when FORGE_CELLS_OUTLIVE=on). It clears it when the cell exits.
  • Adopting: makeRegistry reads every record at creation and probes each one. A record counts as live only if its pid is alive and its port answers. Live ones go into held, so get, list and stop work on them. Stale ones are cleared.
  • Adopted cells: the host watches an adopted cell's pid once a second, since it did not start it, and drops it when it dies.
  • Ordering: the registry has a new adopted promise. start waits for it, so a second cell is never started beside one that is being adopted. bootHostedChildren in src/spawn/host.ts waits for it before waking anything.
  • Stop on drain: stopCellChildren already leaves cells running when outlive is on (step 1), so a release drain doesn't stop them. Sleep, floor and retire still stop them through stop. I changed nothing in src/main.ts.
  • Allow-lists: the gate forced two allow-list additions: clearRecord in src/kernel/destroy_reach.ts (it deletes a file reachable from the billing webhook), and the adopted-cell pid check in test/timers-named.test.ts.

Tests (test/cell-adopted-on-start.test.ts)

  • A registry built over a live fake cell adopts it, and get answers.
  • A record with a dead pid, and one with a live pid but nothing on its port, are both dropped and their files cleared.
  • Malformed records and mismatched bodyIds are rejected.

What I left out

  • An adopted cell has no beat door in this process. The old door lived in the previous host and died with it. A cell started by this host still gets its door.
  • Adopted cells are not reaped on boot beyond clearing stale records. That is outside this issue's files.
  • A pid reused by an unrelated process that happens to answer on the same port would be adopted. The risk is small, and I did not add a stronger identity check.

To try it Set FORGE_CELLS_OUTLIVE=on, start a child, restart the host, and the child is listed again without a second start.

Closes #13756

4 earlier comments · 1 event
commented

On start, the host now finds the cells a previous host left running and takes them over. bun x tsc --noEmit and bun scripts/gate.ts --fix are clean. The new test file passes, as do the other cell and timer tests I ran: cell-outlives-host, spawn-sleep-together, spawn-cell-stopping, first-ten-minutes, forges-memory, timers-named, spawn-sites and billing-never-deletes. I did not run the full suite; the forge runs it on push.

What changed

  • Record: src/spawn/cell_record.ts is new. It holds the record {bodyId, pid, port, supervisor, startedAt}, plus name and size, which I added so a host that didn't start the cell can still answer for it. The record lives at children/<bodyId>/cell.json. It is written through a rename and checked field by field when read, and a record whose bodyId differs from its directory is ignored.
  • Writing and clearing: startOn in src/spawn/cell.ts writes the record once a supervised cell answers (only when FORGE_CELLS_OUTLIVE=on). It clears it when the cell exits.
  • Adopting: makeRegistry reads every record at creation and probes each one. A record counts as live only if its pid is alive and its port answers. Live ones go into held, so get, list and stop work on them. Stale ones are cleared.
  • Adopted cells: the host watches an adopted cell's pid once a second, since it did not start it, and drops it when it dies.
  • Ordering: the registry has a new adopted promise. start waits for it, so a second cell is never started beside one that is being adopted. bootHostedChildren in src/spawn/host.ts waits for it before waking anything.
  • Stop on drain: stopCellChildren already leaves cells running when outlive is on (step 1), so a release drain doesn't stop them. Sleep, floor and retire still stop them through stop. I changed nothing in src/main.ts.
  • Allow-lists: the gate forced two allow-list additions: clearRecord in src/kernel/destroy_reach.ts (it deletes a file reachable from the billing webhook), and the adopted-cell pid check in test/timers-named.test.ts.

Tests (test/cell-adopted-on-start.test.ts)

  • A registry built over a live fake cell adopts it, and get answers.
  • A record with a dead pid, and one with a live pid but nothing on its port, are both dropped and their files cleared.
  • Malformed records and mismatched bodyIds are rejected.

What I left out

  • An adopted cell has no beat door in this process. The old door lived in the previous host and died with it. A cell started by this host still gets its door.
  • Adopted cells are not reaped on boot beyond clearing stale records. That is outside this issue's files.
  • A pid reused by an unrelated process that happens to answer on the same port would be adopted. The risk is small, and I did not add a stronger identity check.

To try it Set FORGE_CELLS_OUTLIVE=on, start a child, restart the host, and the child is listed again without a second start.

Changes requested — before the latest push

Changes needed, all in one round:

  1. FIRST: adopt() leaks and orphans. The 1s setInterval starts before the liveness check. When the pid is alive but the port does not answer, the record is cleared and the interval runs forever (unref does not end it). The live process is also left running with no record and nobody to reap it. Start the watcher only after the probe passes, and decide what happens to an alive-but-silent pid (stop it through the supervisor control dir in the record, or say plainly why not).
  2. Pid reuse is accepted as a known risk, but adopted children go into held and stop acts on pid. A reused pid that answers on the port gets signalled. Add a real identity check before adopting (for example /proc/<pid>/cmdline or start time against startedAt), or stop adopted cells through the supervisor and never by raw pid.
  3. Tests: nothing fails without the writing side. Add tests that startOn writes cell.json only when supervised and cells-outlive is on, and clears it on exit. Add tests that start waits for adopted and does not start a second cell beside an adopted one, and that an adopted cell dropped on pid death clears its record. Today only adopt and parse are covered.
  4. writeRecord runs after up with no guard. If it throws (disk full), start fails after the cell is up. If the cell exits first, a stale record is left. Guard it and write it before or atomically with the exit hook.
  5. Direction: the owner put pipeline work first (#6534, #6611, #5276, ...). This is a cells-outlive feature step outside that list. State in the pull which pipeline issue it serves, or hold it behind them.
  6. Update the doc comment on cellsOutlive and add docs/features/<name>.md if the flag is meant to be used. The pull says "To try it" but nothing documents it, and there is no terminal view of adopted cells.

Rest is fine: the record is validated field by field, written through a rename, and the allow-list additions are justified.

commented

On start, the host now re-adopts the cells a previous host left running, but only when their supervisor still names the pid. tsc and scripts/gate.ts --fix are clean. I ran test/cell-adopted-on-start.test.ts and test/timers-named.test.ts, and both pass. scripts/fast.ts exited 2, because src/kernel/destroy_reach.ts is in the diff. I did not pick further test files after that, so the forge's full check is the first run beyond those two.

What changed (src/spawn/cell.ts, src/spawn/cell_record.ts)

  1. Adoption order. The one-second watcher now starts only after the probe passes. If a pid is its supervisor's cell but the port is silent, the host stops it with killTree, clears the record and logs it. Nothing is left running unowned.
  2. Pid reuse. Adoption checks that readPid(<supervisor ctl>/cell.pid) === record.pid. readPid also compares the process start time, so a reused pid fails the check and is dropped without being signalled. The same check runs every tick, so a pid reused later is dropped rather than killed.
  3. Tests.

- leaveRecord writes only for a supervised cell and reports a failed write instead of throwing. - A pid the supervisor does not name is neither adopted nor signalled. - A silent cell of its supervisor is stopped and its record cleared. - start waits for adoption and gets the adopted cell, with no second launch. - An adopted cell dropped on pid death clears its record. - The existing adoption test now sets up a supervisor control directory with the cell's pid, as an adopted cell needs. - I updated the pinned name for the interval in timers-named.test.ts.

  1. Record write. writeRecord is now leaveRecord, called right after spawn and before anything awaits, so the exit hook's clearRecord can only run after it. Errors are logged, not thrown. A record written before the cell answers is cleaned up if the start fails, and the next host reaps it if this host dies mid-start.
  2. Docs. I updated the cellsOutlive doc comment and added docs/features/cells-outlive.md. It says plainly that there is no terminal view of adopted cells yet; they show up in the registry and the log.

Not done

  • Item 5, direction. This serves no pipeline issue on the owner's list. It is step 2 of the cells-outlive work from #13755/#13761 (#13756). The flag is off by default, so nothing changes for anyone unless it is set. I'd hold it behind the pipeline pulls if you prefer; that is your call.
  • Untested here. startOn itself needs a real cell, which this sandbox can't open, so its writing side is covered only through leaveRecord.

To try it, run with FORGE_CELLS_OUTLIVE=on, start a child, restart the host, and look for child forge adopted in the log.

commented

On start, the host re-adopts the cells a previous host left running (#13756). It adopts one only when its supervisor still names the pid and the port answers. This push fixes the two gate failures from the last check.

  • Real-timer rule: the test that stops a silent adopted cell now waits on the process's exited promise instead of Bun.sleep. The adoption test file passes (7 tests), and tsc is clean. I did not re-run the gate or the other test files locally.
  • Pipeline-first rule: the latest commit message carries a Direction: line saying this is step 2 of cells-outlive (#13756), after step 1 (#13761). That step is already on main, and the flag is off by default, so no pipeline behaviour changes. I did not change the pipeline.
  • Comment-width note: the gate's note about src/spawn/host.ts is informational and the nightly rewrap handles it. It is not a failure, so I left it.

What the pull does:

  • Adoption: a recorded cell is adopted only if its supervisor's cell.pid still names the pid, with start time. A reused pid is dropped and never signalled. A cell that is its supervisor's but silent is stopped and its record cleared.
  • Watcher and record: the one-second pid watcher starts only after the probe passes. The record is written right after spawn and a failed write is logged, not thrown.
  • start: it waits for adoption to finish and returns the adopted cell, so no second one is launched.
  • Docs: docs/features/cells-outlive.md describes the flag.

Not done:

  • Beat door: it still runs in the host.
  • Egress gauges: an outliving cell's egress bytes are not in them yet (#13921).
  • Terminal view: there isn't one for adopted cells yet.
  • startOn: it is only tested through leaveRecord, because this sandbox can't open a real cell.

To try it: run with FORGE_CELLS_OUTLIVE=on, start a child, restart the host, and look for child forge adopted in the log.

🤖 Generated with Claude Code

commented

This pull makes the host take back the cells it left running when it starts. It stays off unless FORGE_CELLS_OUTLIVE=on.

  • Cells: each supervised cell writes children/<body>/cell.json as soon as it is spawned and clears it on exit. At boot makeRegistry reads every record, probes it, and adopts the live ones. It adopts a cell only when its supervisor still names its pid, and it stops a silent one. A stale record is cleared. The wait on a stopped cell's exit is event-driven, not a timer.
  • Docs: docs/features/cells-outlive.md describes this and the flag.
  • CI fix: the two failing orphan tests came from cells-outlive.md and app-page-details.md missing from the docs/features/README.md index. I added a row for each, under "When something is wrong" and "Your apps". test/feature-orphans.test.ts passes again, and bun x tsc --noEmit is clean.
  • Left alone:

- The 8 "landing past a broken gate" failures were in test/gate-bypass.test.ts. It passes 12 of 12 when run alone here, so I treated them as a shard flake unrelated to this change. - test/behaviors-rule.test.ts is red on main too and is not touched here.

To try it: start with FORGE_CELLS_OUTLIVE=on, stop the host, and start it again. The cells that were running are re-adopted instead of respawned.

commented

I merged main into the branch and pushed it, and the cell and index tests pass. I did not run the full suite: scripts/fast.ts couldn't name tests for this diff, so I ran the cell tests and the feature-index test by hand.

  • Conflict: the only one was the doc comment above cellsOutlive in src/spawn/cell.ts. I kept both sides' meaning: cells leave a record that the next host adopts if the cell is still its supervisor's and answers, or stops if it doesn't (#13756). The beat door is still this process's, answered by the supervisor's held door after it exits (#13906). Egress bytes are counted in the supervisor (#13921).
  • Index duplicate: main had already added an app-page-details.md row to docs/features/README.md, so my earlier row made it appear twice. I dropped mine, and the cells-outlive.md row stays.
  • Checks: bun x tsc --noEmit is clean and scripts/gate.ts --fix changed nothing. The cell tests (adopted-on-start, outlives-host, relays-reaped, spawn-cell, spawn-cell-stopping) and the feature index test pass. Six tests skipped, as expected in this sandbox, which can't bind unix sockets.
  • Left alone: src/spawn/host.ts has a comment line over 100 columns that the gate says the nightly rewrap will handle on main.
Approved

The earlier asks are met in the diff as I can read it (the clone lacks 64ffc378a2e7, so I judged the change as a whole). Adoption requires the supervisor to still name the pid with its start time, so a reused pid is never signalled. A silent cell of its supervisor is stopped and its record cleared. The record is written right after spawn and a failed write is warned, not thrown. start waits for adoption. The new behaviours have tests, the how-to is in docs/features/cells-outlive.md with an index row, README.md is untouched, and the allow-lists carry reasons. It is flag-off, so no user-facing page changes.

Two things I would do next:

  1. Non-blocking: give adopted cells a terminal view in src/tui/ (the doc admits there is none); the both-doors rule applies once the flag is turned on by default.
  2. Non-blocking: cover startOn writing and clearing the record with a real cell in the forge check, since only leaveRecord is unit-tested.

Non-blocking: this is not on the pipeline-first list, which the pull says openly; keep follow-ups behind the pipeline issues.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs