A child forge runs as its own process in its own cell, held to its size #6019

A child forge now runs as its own bun src/main.ts process inside a cell held to its size, wherever the host can hold a cell. I couldn't run a real cell in this sandbox, so the cell path has not run end to end.

What changed

• Placement. spawn/cell.ts adds the cell placement. It starts the child from childEnv with FORGE_APP_URL, FORGE_APP_DATA and PORT. The cell's memory, CPU and disk come from the child's size in SIZES (small is 512 MiB, 1 CPU and 2 GiB). The parent's data directory is hidden inside the cell, and only the child's own directory is writable.
• Fallback. fallbackPlacement in spawn/placement.ts picks a cell when cellsAvailable() says yes and hosted otherwise. Both share one body store, so the child limit and name checks are unchanged.
• Which one was used. The ledger records placement, and the spawn answer and the children list report it. An old ledger reads as hosted.
• Proxy. For a cell child, resolveHostedChild returns "cell" and answerFromCell starts the child if it isn't up and forwards the request to its loopback port. The response is streamed, and the security headers still apply.
• Lifecycle. bootHostedChild starts a cell child, so a new spawn and a parent restart both bring it up. Retiring a body stops its process, and so does shutting the parent down.
• Docs. docs/SPAWN.md has a line for this.

Tests

• test/spawn-cell.test.ts checks that the limits come from the size and that the parent's secrets don't reach the child's environment.
• It also checks that the placement picks a cell or hosted correctly, that retiring frees the slot, and that forwarding keeps method, path, query, body and cookies and answers 502 for a child that isn't listening.
• One further test runs a real cell and checks that the parent's data is hidden. It is skipped in this sandbox with a reason. It has never run, so check its result when the forge's check runs it.
• bun test test/spawn, which includes the new test, and the type check pass here. The design lint passes.

Left out

• Retire: nothing in the forge calls whole-forge retire yet, so retiring a cell child works at the placement seam but has no page or tool to reach it.
• Websockets: these are not proxied to cell children.
• Crash recovery: a crashed cell child is restarted on the next request, not by a supervisor.
• Outbound access: the cell allows no outbound hosts beyond the parent's origin, so child features that call out, such as a model endpoint, need an egress grant.
• Apps inside a child: a cell child's own apps have not been tried inside the outer cell.
• Splitting: I did not split the proxy and lifecycle into separate issues.

To try Spawn with size: small on a Linux host that has bwrap and socat. The response's placement reads cell, and the forge:<name> cell shows up in the cell sampler. Elsewhere it reads hosted.

Closes #2926

A child forge now runs as its own bun src/main.ts process inside a cell held to its size, wherever the host can hold a cell. I couldn't run a real cell in this sandbox, so the cell path has not run end to end.

What changed

• Placement. spawn/cell.ts adds the cell placement. It starts the child from childEnv with FORGE_APP_URL, FORGE_APP_DATA and PORT. The cell's memory, CPU and disk come from the child's size in SIZES (small is 512 MiB, 1 CPU and 2 GiB). The parent's data directory is hidden inside the cell, and only the child's own directory is writable.
• Fallback. fallbackPlacement in spawn/placement.ts picks a cell when cellsAvailable() says yes and hosted otherwise. Both share one body store, so the child limit and name checks are unchanged.
• Which one was used. The ledger records placement, and the spawn answer and the children list report it. An old ledger reads as hosted.
• Proxy. For a cell child, resolveHostedChild returns "cell" and answerFromCell starts the child if it isn't up and forwards the request to its loopback port. The response is streamed, and the security headers still apply.
• Lifecycle. bootHostedChild starts a cell child, so a new spawn and a parent restart both bring it up. Retiring a body stops its process, and so does shutting the parent down.
• Docs. docs/SPAWN.md has a line for this.

Tests

• test/spawn-cell.test.ts checks that the limits come from the size and that the parent's secrets don't reach the child's environment.
• It also checks that the placement picks a cell or hosted correctly, that retiring frees the slot, and that forwarding keeps method, path, query, body and cookies and answers 502 for a child that isn't listening.
• One further test runs a real cell and checks that the parent's data is hidden. It is skipped in this sandbox with a reason. It has never run, so check its result when the forge's check runs it.
• bun test test/spawn, which includes the new test, and the type check pass here. The design lint passes.

Left out

• Retire: nothing in the forge calls whole-forge retire yet, so retiring a cell child works at the placement seam but has no page or tool to reach it.
• Websockets: these are not proxied to cell children.
• Crash recovery: a crashed cell child is restarted on the next request, not by a supervisor.
• Outbound access: the cell allows no outbound hosts beyond the parent's origin, so child features that call out, such as a model endpoint, need an egress grant.
• Apps inside a child: a cell child's own apps have not been tried inside the outer cell.
• Splitting: I did not split the proxy and lifecycle into separate issues.

To try Spawn with size: small on a Linux host that has bwrap and socat. The response's placement reads cell, and the forge:<name> cell shows up in the cell sampler. Elsewhere it reads hosted.

I've fixed all three of @sentinel's findings and pushed them to the same branch. bun test test/spawn (174 tests) and the type check pass here. The real-cell test is still skipped in this sandbox, so none of this has run against a live cell.

• Spoofed client address (high). The cell branch in server.ts now calls notePeer before it forwards. forward drops the inbound x-forwarded-for, forwarded, x-real-ip, cf-connecting-ip, true-client-ip, x-forwarded-host, x-forwarded-proto and the configured client-IP header. It then sets x-forwarded-for to clientAddress(request), so the child's limits key on the address the parent resolved. The new test sends a request with those headers all set to 6.6.6.6 from a noted peer 203.0.113.9. It asserts the child receives exactly 203.0.113.9 and none of the others.
• Hosted wrapper skipped (medium). The cell branch now runs the same armClose, answered, counted, logged, settleBody and mustClose steps as the hosted branch. A cell child has no App in the parent, so its requests are counted and logged under the parent's app and data directory, not under their own. I chose that over adding a separate metrics key.
• Relaunch loop (medium). A start that fails is remembered for 30 seconds (START_BACKOFF_MS). During that time answerFromCell answers 503 straight away. A test checks that a second start inside the window does not relaunch. It only runs where cells are unavailable, because it relies on the launch failing.

I did not add websocket proxying.

I've pushed fixes for the failures on the preview, but I haven't found the cause of the child process exiting, and I couldn't run a cell here to find it. bun test (spawn, family, web), the type check and the design lint pass here.

Cause: unconfirmed. Your guess is the most likely cause I could find. I added a reveal option to CellSpec, and the forge's own code root is now bound back read-only inside the cell. The old code only did that for the directory holding the bun binary. If the preview runs the forge from data/release/…, hiding the parent's data directory also hid src/main.ts, which would explain a child that dies within a second. I couldn't confirm that. If the child still fails on the preview, it no longer stays dead (see the first fix below), and the reason is recorded.

Fixes

• No more dead-on-arrival children. A cell that doesn't answer now makes resolveHostedChild and bootHostedChild rewrite the ledger to hosted, with a placementNote of hosted: <why>, and open the child in the parent. The reason is the last stderr lines of the child, the cell's memory breach if there was one, or the error from making the cell. listChildren returns placementNote, and each /forges row shows own cell, hosted, or the note.
• Launch that throws. .catch runs before the .then, so the failure is recorded and the backoff applies.
• Early 502. A child is held only once it answers. A concurrent start waits on the in-flight launch.
• New tests.

- test/spawn-cell.test.ts has a test, skipped where cells aren't available, that boots a real child through cellChildren(app).start({ …, size: "small" }). It asserts the limits and that forward(child, GET /healthz) answers 200. It has never run, so it will run for the first time in the forge's check. - A second test checks the fallback: a ledger saying cell where cells can't start comes out hosted with a note. It runs here.

Disagreement: the first failed start demotes the child permanently, with no retry. If your cell host is merely slow, the child stays hosted until someone changes its ledger. A cell that can't boot within 30 s is not one to keep promising, and the note makes the reason visible.

I did not check by hand that a spawned child's / answers 200 on the preview, because my shell can't spawn there. Please spawn a new child once this push has deployed. The old sentinelkid will be demoted to hosted the first time it is requested.