Public: anyone can read this; the crew can change it.
docs/research/celld.md: the celld brief from #483, its local references checked against main #495
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/489-docs-research-celld-md-the-celld-brief-f → main 1 commit fork
3 files
changed +77
−0
docs/SPAWN.md +1
| @@ -28,6 +28,7 @@ and user accounts. The child mints its own. A test in #420 and #422 proves it. | ||
| 28 | 28 | |
| 29 | 29 | External bodies (Cloudflare, Fly, metal) are decided by spike #339 and arrive |
| 30 | 30 | as a second implementation of the placement seam (#427), in their own issue. |
| 31 | A self-hosted front door on celld is weighed in `docs/research/celld.md`. | |
| 31 | 32 | |
| 32 | 33 | ## Choices |
| 33 | 34 |
docs/SPIKE-CLOUDFLARE.md +3
| @@ -45,6 +45,9 @@ forge" therefore needs two tools for that body, one for Fly (`fly deploy`). | ||
| 45 | 45 | Both bodies sit behind the same Worker front door, so choosing later costs a |
| 46 | 46 | variable, not a rewrite. |
| 47 | 47 | |
| 48 | celld could run that front door on our own machines; see | |
| 49 | `docs/research/celld.md`. | |
| 50 | ||
| 48 | 51 | ## To finish the spike |
| 49 | 52 | |
| 50 | 53 | 1. Declare the Cloudflare token and bucket key as run secrets; install `tofu`. |
docs/research/celld.md added +73
| @@ -0,0 +1,73 @@ | ||
| 1 | # Brief: celld (research for #483) | |
| 2 | ||
| 3 | Everything below was read on 2026-10-02, from the celld repository at `v0.6.1` and from this repository at `922e84b`. Each claim cites its source. Nothing was run. Where a sentence is an inference, it says **inference**. The paths into this repository were checked against `main` when the brief landed (#489); the places where the first draft differed are listed at the end. | |
| 4 | ||
| 5 | ## 0. Which "celld" | |
| 6 | Only one project of that name exists. The candidates are below, and the brief covers the first: | |
| 7 | - **`denoland/celld`**, "self-hosted, distributed Durable Objects" [[repo]](https://github.com/denoland/celld). It matches "cells/sandboxes for compute". | |
| 8 | - Packages *built on* it, not separate projects: `@ewhauser/world-celld` (a Vercel Workflow backend) [[npm]](https://www.npmjs.com/package/@ewhauser/world-celld), `effect-durable-agent-celld`, `@criblio/cell-workspace` ("cells (server-side agents on celld)"), `@agentback/actors-do`, `@softwaiz/recell` [[npm search "celld"]](https://www.npmjs.com/search?q=celld). | |
| 9 | - `@distilled.cloud/celld` is a name placeholder ("Placeholder package for initial npm registration") [[npm]](https://www.npmjs.com/package/@distilled.cloud/celld). | |
| 10 | ||
| 11 | ## 1. What it is | |
| 12 | - **Source and owner:** github.com/denoland/celld, written in Rust. It was created on 2025-04-25 and last pushed on 2026-10-01. It has about 4.9k stars and 203 forks [[GitHub API]](https://api.github.com/repos/denoland/celld). Patches are emailed to [email protected] and assigned to Deno Land Inc. Pull requests are disabled: "Coding agents make it too easy to send a large, low-context change" [[README §Contributions]](https://github.com/denoland/celld#contributions). | |
| 13 | - **License:** Apache-2.0 [[LICENSE]](https://github.com/denoland/celld/blob/main/LICENSE). | |
| 14 | - **Maturity:** "celld v0.6.1 is a beta release" [[limitations]](https://github.com/denoland/celld/blob/main/docs/limitations.md). Releases ship roughly weekly: v0.2.1 on 2026-08-14 through v0.6.1 on 2026-10-01 [[releases]](https://github.com/denoland/celld/releases). Upgrades are not rolling: "Mixed versions cannot share a serving fleet" [[README]](https://github.com/denoland/celld#run-it). Security fixes go to the latest release only [[security]](https://github.com/denoland/celld/blob/main/docs/security.md). | |
| 15 | ||
| 16 | ## 2. Architecture and primitives | |
| 17 | - **What it runs:** "an open-source daemon that runs a Cloudflare Workers application on your own machines: Workers, Durable Objects, KV, Queues, D1, R2, Workflows, Cron Triggers, and static assets, deployed from the `wrangler.json` that you already have" [[README]](https://github.com/denoland/celld#celld). | |
| 18 | - **The cell:** "Each object is a cell: a named server with its own SQLite database." KV namespaces, queues, D1 databases and Workflows are each a cell too, with the same lease, replication and failover [[README §What runs]](https://github.com/denoland/celld#what-runs-on-celld). | |
| 19 | - **Node and fleet:** a node is one `celld` process per machine, and it embeds V8. The nodes that share one bucket (S3, GCS or Azure) form a fleet. The bucket holds the deployments, the cell state and the ownership records. There is "no membership protocol, failure detector, or consensus service" [[README §How it works]](https://github.com/denoland/celld#how-it-works). | |
| 20 | - **Ownership:** "A conditional bucket write gives a node the ownership of a cell, so exactly one node owns a cell at a time", fenced by an epoch [[README]](https://github.com/denoland/celld#how-it-works), [[security §ownership]](https://github.com/denoland/celld/blob/main/docs/security.md). | |
| 21 | - **Durability:** each committed SQLite write is captured as LTX. On a single node, the write is durable once it is uploaded to the bucket. In a fleet of two or more, it is durable once one or two peers hold it on disk, and the bucket upload follows [[README]](https://github.com/denoland/celld#how-it-works), [[guarantees]](https://github.com/denoland/celld/blob/main/docs/guarantees.md). | |
| 22 | - **One app per fleet:** "A fleet runs one application. celld has no account service, multi-tenant scheduler, or managed ingress" [[limitations]](https://github.com/denoland/celld/blob/main/docs/limitations.md). | |
| 23 | - **Local mode:** `celld dev` runs one node with a local SQLite object store. It needs no Docker or bucket, serves on `127.0.0.1:9876`, and keeps its state in `.celld/dev` [[README §Run it]](https://github.com/denoland/celld#run-it). | |
| 24 | - **Distribution:** a static binary for Linux x86-64/ARM64 and Apple Silicon, plus the image `ghcr.io/denoland/celld`. Windows is not supported [[README §Install]](https://github.com/denoland/celld#install), [[limitations]](https://github.com/denoland/celld/blob/main/docs/limitations.md). `celld deploy` needs `esbuild` on `PATH` [[README]](https://github.com/denoland/celld#install). | |
| 25 | ||
| 26 | ## 3. Isolation and scheduling | |
| 27 | - **JavaScript isolation:** each cell runs in a V8 isolate with a 128 MB heap by default, matching Cloudflare's Durable Object limit (`CELLD_V8_HEAP_LIMIT_MB`) [[README §Operate]](https://github.com/denoland/celld#operate-a-fleet). Host functions cannot be reached from the bundle [[security]](https://github.com/denoland/celld/blob/main/docs/security.md). | |
| 28 | - **Trust boundary:** "It is not safe for hostile multi-tenant use … Do not run code from mutually distrusting tenants in one fleet" [[security]](https://github.com/denoland/celld/blob/main/docs/security.md). | |
| 29 | - **Containers (experimental):** a Durable Object can supervise one container through `ctx.container`, driven by a Docker or Podman daemon on the node. All capabilities are dropped, `no-new-privileges` is set, and an nftables fence sits on the bridge. `CELLD_CONTAINER_RUNTIME=runsc` (gVisor) or `kata` gives each container its own kernel. "Under the default runtime, a container is a kernel namespace boundary and not a virtual machine. For code that you did not write, name a runtime that gives each container its own kernel" [[containers]](https://github.com/denoland/celld/blob/main/docs/services/containers.md). `@cloudflare/sandbox` runs "on a container per sandbox" [[README table]](https://github.com/denoland/celld#what-runs-on-celld). | |
| 30 | - **Scheduling and placement:** no central scheduler. A request goes to the cell's owner. Balancing reads a shared fleet sample every 5 s and moves at most 32 *hibernated* cells per sample, weighted by CPU count (`CELLD_PLACEMENT_WEIGHT`). It "does not measure the CPU or memory that one cell uses" [[README §Operate]](https://github.com/denoland/celld#operate-a-fleet), [[limitations]](https://github.com/denoland/celld/blob/main/docs/limitations.md). | |
| 31 | - **Pressure:** memory shedding starts at 80% of available memory, with a hard cap at 95% of the cgroup charge. Idle least-recently-used cells are replicated, fenced and released, and a cell with active work is never shed. `CELLD_MAX_RESIDENT_CELLS` caps how many cells stay resident [[README §Operate]](https://github.com/denoland/celld#operate-a-fleet). | |
| 32 | - **Network:** celld terminates no TLS. Peer traffic is plaintext HTTP inside an HMAC-established tunnel, so the internal listener must sit on a private network or a WireGuard/Tailscale overlay. Anyone who holds the bucket credentials controls the fleet [[security]](https://github.com/denoland/celld/blob/main/docs/security.md). | |
| 33 | ||
| 34 | ## 4. Performance and cost | |
| 35 | - **Write latency:** "A write to a region-local store takes about 90 ms." A loaded lab fleet measured about 600 ms against a store in another region, and about 25 ms once a second node held the write. A single node pays the full bucket round trip on every write, so "Run two or more nodes if write latency matters" [[README §Run it]](https://github.com/denoland/celld#run-it). These are celld's own figures; nobody here measured them. | |
| 36 | - **Idle cost:** "An inactive cell costs almost nothing" [[README]](https://github.com/denoland/celld#celld). Hibernated cells release their storage cache from RSS [[README §Operate]](https://github.com/denoland/celld#operate-a-fleet). | |
| 37 | - **Density:** about 50,000 hibernatable WebSockets per cell at the default heap [[README §Operate]](https://github.com/denoland/celld#operate-a-fleet). | |
| 38 | - **Cost (inference):** the bill is your own machines plus bucket requests and storage. Each durable write on a single node is at least one PUT, so a write-heavy app on one node pays per write. Nothing published measures requests per second or cold-start time. The experiments below should measure those. | |
| 39 | ||
| 40 | ## 5. Compared with what the forge uses today | |
| 41 | | | The forge today | celld | | |
| 42 | |---|---|---| | |
| 43 | | Hosted apps | One Bun child process per environment, started with `/bin/sh -c "exec …"` and an explicit env (`src/apps/process.ts`). State in `FORGE_APP_DATA` with bun:sqlite (`docs/APPS.md`). | One Workers bundle per fleet. State in Durable Object SQLite, replicated to a bucket. Not Bun, and no `bun:sqlite` [[compat]](https://github.com/denoland/celld/blob/main/docs/cloudflare-compat.md). | | |
| 44 | | Crew run sandbox | Claude Code's bubblewrap policy on Linux (seatbelt on macOS); the network is the forge and the registry (`src/crew/sandbox.ts`). When `bwrap --unshare-user` fails its probe the run is unsandboxed. | A container per sandbox; a separate kernel only with gVisor or Kata [[containers]](https://github.com/denoland/celld/blob/main/docs/services/containers.md). Needs a Docker or Podman daemon. | | |
| 45 | | Cluster | A Helm chart, `charts/app` (`Chart.yaml`, templates for a deployment, a volume claim, a service, ssh and apps). It deploys the forge itself; the repository does not name k3s. | No orchestrator. The bucket is the control plane. Runs as a container or a bare binary [[README §Container]](https://github.com/denoland/celld#container). | | |
| 46 | | Spawned bodies (#339) | A Cloudflare Worker + Durable Object front door (`infra/forge-cloudflare/worker.js`), written and tested but not applied, with a Container or Fly body behind it. Firecracker on metal is the third option, not built (`docs/SPIKE-CLOUDFLARE.md`). | Runs that same Worker + Durable Object shape on our own machines. It cannot run the forge itself, which is a Bun process with git and ssh. | | |
| 47 | | Isolation | bubblewrap namespaces, one machine. | V8 isolates (trusted code only) and optional gVisor/Kata containers. | | |
| 48 | ||
| 49 | The forge is a Bun process with SQLite on local disk. celld is a Workers runtime with SQLite in a bucket. The two are complementary, not interchangeable (**inference**). | |
| 50 | ||
| 51 | ## 6. Three ways the forge could use it | |
| 52 | ### A. A self-hostable front door for spawned forges (#389, #427) | |
| 53 | The spike's front door is already a Worker with a Durable Object (`infra/forge-cloudflare/worker.js`). celld runs that shape from `wrangler.json` on our own machines, so a child forge could get the same front door without a Cloudflare account. | |
| 54 | - **Risks:** beta and weekly breaking upgrades. One fleet per app, so one fleet per front door unless a single front door routes every child. The front door's DNS and TLS still need an ingress, because celld does no TLS. | |
| 55 | - **Smallest experiment:** put a `wrangler.json` next to `worker.js`, start it under `celld dev`, and run a test that goes through it to a stub body. Record the latency and the cold start. | |
| 56 | ||
| 57 | ### B. A "worker" app kind for hosted apps | |
| 58 | An app that ships a `wrangler.json` would run under `celld dev --port $PORT` in place of `bun run`. It would get alarms, queues, cron and hibernating WebSockets, and it would scale to near zero when idle. | |
| 59 | - **Risks:** a second app runtime to maintain beside Bun. The app's state moves out of `FORGE_APP_DATA` into `.celld/dev`, so the root must be pointed inside it. celld has no multi-tenant boundary, but a celld per app keeps today's per-process boundary. The binary is about 25 MB per host [[release v0.6.1]](https://github.com/denoland/celld/releases/tag/v0.6.1). | |
| 60 | - **Smallest experiment:** add an example app with a `wrangler.json` and a counter Durable Object. A script fetches the pinned release, starts `celld dev` on a port and checks that `/` answers and the counter survives a restart. Measure first-request and p50 latency against a Bun hello app. | |
| 61 | ||
| 62 | ### C. Crew run sandboxes with a kernel boundary | |
| 63 | celld's `@cloudflare/sandbox` support with `CELLD_CONTAINER_RUNTIME=runsc` would give each crew run its own gVisor kernel. A Durable Object would supervise the run, and the forge would talk to it over HTTP. That is stronger than bubblewrap's shared kernel. | |
| 64 | - **Risks:** Containers are "experimental … the security boundary can change without notice" [[containers]](https://github.com/denoland/celld/blob/main/docs/services/containers.md). Every node needs Docker or Podman plus gVisor. `hardTimeout` is ignored, so the forge must enforce the run's time limit itself [[containers]](https://github.com/denoland/celld/blob/main/docs/services/containers.md). This also overlaps with the microVM sandbox `docs/EXTRACT.md` (_What is given up_) names for apps, and with the Firecracker option in `docs/SPIKE-CLOUDFLARE.md`. | |
| 65 | - **Smallest experiment:** on a host with Docker and runsc (not the crew sandbox, which has neither), run celld's `examples/sandbox`, `exec` a `git clone` of a forge repo and time the container start. This needs a person and a machine, so no issue is opened for it yet. | |
| 66 | ||
| 67 | **Recommendation:** try A and B. They run with `celld dev` alone, no Docker and no bucket, and either can be dropped without touching the forge's core. Leave C until someone decides on hardware for #339's Firecracker option. | |
| 68 | ||
| 69 | ## Checked against `main` (#489) | |
| 70 | - `src/apps/process.ts`, `src/crew/sandbox.ts`, `infra/forge-cloudflare/worker.js`, `docs/APPS.md` match the table, with two additions made above: the sandbox policy is Linux bubblewrap or macOS seatbelt and falls back to unsandboxed, and the app command runs as `/bin/sh -c "exec …"`. | |
| 71 | - `charts/app` is a Helm chart that deploys the forge. The first draft called it a "k3s chart"; nothing in the repository names k3s, so the row says Helm. | |
| 72 | - `docs/SPIKE-CLOUDFLARE.md` says the front door is tested and nothing is applied, and that the bodies behind it are a Container or Fly, with Firecracker only a verdict. The first draft listed all three as behind it. | |
| 73 | - `docs/EXTRACT.md` does not mention Firecracker. It says apps need a bubblewrap or microVM sandbox before friends share a box. The first draft cited it for what Firecracker would give crew runs; the sentence now says what it does say. |
new above
offline