trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

Spike: spawn a forge on Cloudflare with OpenTofu from a crew run, state in the forge's own bucket #377

Merged scoutcrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/339-spike-spawn-a-forge-on-cloudflare-with-o → main 1 commit fork

  • Merged. scout/339-spike-spawn-a-forge-on-cloudflare-with-o was merged into main as 3e4679a.

Review

Sign in to review this change.
10 files changed +286 −2
docs/EXTRACT.md +3 −1
@@ -8,7 +8,9 @@ turned off, with what each move costs and what it gives up.
88
99 ## What a host owes the forge
1010
11 - **Binaries**: `bun`, `git`, `claude` (the crew), `chromium` (the look).
11 - **Binaries**: `bun`, `git`, `claude` (the crew), `chromium` (the look), and
12 `tofu` where a run spawns another forge (`infra/forge-cloudflare`,
13 `docs/SPIKE-CLOUDFLARE.md`).
1214 - **A name and a wildcard**: `forge.example` for the forge, `*.forge.example`
1315 for the apps it serves (`FORGE_APP_DOMAIN`). One wildcard certificate
1416 covers them; the forge speaks plain HTTP on `:3000` and expects a proxy in
docs/SPIKE-CLOUDFLARE.md added +53
@@ -0,0 +1,53 @@
1 # Spike: a forge on Cloudflare, spawned with OpenTofu (#339)
2
3 **Status: the module is written and its front door is tested; nothing has been
4 applied.** This run had no `tofu` binary, no Cloudflare token and no bucket
5 credential (the run-secret injection it depends on was not declared for it), and
6 no `bwrap`. So `tofu plan`, `apply` and `destroy` are unproven, and each open
7 question below says whether it is *evidence* or *reading*. Nothing here claims
8 a measurement that was not made.
9
10 ## What exists
11
12 `infra/forge-cloudflare/`: Worker + Durable Object front door (`worker.js`,
13 tested in `test/forge-cloudflare.test.ts`), proxied DNS record, optional tunnel
14 for a body with no public address, S3 backend with `use_lockfile` and a
15 per-forge key. Names must match `^spike-…`, so the module cannot address the
16 production tunnel or zone records. Every record carries a comment naming the
17 spike. The provider (v5) syntax was written from memory and is unchecked by
18 `tofu validate`.
19
20 Gap found: the Cloudflare provider has no Containers resource, so a Container
21 body is deployed with `wrangler` and passed in as `body_url`. "Apply produces a
22 forge" therefore needs two tools for that body, one for Fly (`fly deploy`).
23
24 ## The five questions
25
26 | Question | Answer | Basis |
27 | --- | --- | --- |
28 | bubblewrap inside the container | **Unknown; probably no.** `src/crew/sandbox.ts` probes `bwrap --unshare-user` and falls back to unsandboxed. Containers run in a VM without user-namespace privileges we control; expect the probe to fail. Check: run the probe in a Container. | reading |
29 | SQLite WAL across snapshot/restore | **Unknown.** A disk snapshot taken mid-write holds a consistent main file plus `-wal`, which SQLite recovers on open, provided both are captured together. The forge's own snapshot (`docs/SNAPSHOT.md`) does not depend on disk snapshots, so restore via S3 sidesteps the question. | reading |
30 | HTTPS git instead of SSH | **Yes for the forge's clients.** Inbound is HTTP only on Containers, so `FORGE_SSH_PORT=0` and clone/push over HTTPS with tokens. The app terminal (`FORGE_APP_SSH`) is off there. Fly keeps SSH. | reading |
31 | Does a crew run keep the container awake past 10 min idle | **Unknown.** Idleness is measured on inbound requests; a run is outbound work inside the container, so assume it does *not* count. Mitigation to test: the Durable Object pings the body every few minutes while a run is open, or `sleepAfter` is raised. | reading |
32 | 30-day snapshot retention | **Fine for a cache, wrong for a record.** The nightly S3 snapshot is the record; the platform snapshot is a warm-start only. | reading |
33
34 ## Verdict (provisional)
35
36 - **Container body:** right for a forge that is mostly idle, small, and used
37 from a browser: zero idle cost. Wrong for a forge whose crew runs long, wants
38 bwrap, or wants SSH.
39 - **Fly:** the default for a forge people work in daily: a volume, raw TCP for
40 SSH, a full VM (bwrap likely works), scale-to-zero still available.
41 - **Firecracker on metal:** only when many forges share hardware and need
42 kernel isolation between them (`docs/EXTRACT.md`, _What is given up_); more
43 to run than a spike warrants.
44
45 Both bodies sit behind the same Worker front door, so choosing later costs a
46 variable, not a rewrite.
47
48 ## To finish the spike
49
50 1. Declare the Cloudflare token and bucket key as run secrets; install `tofu`.
51 2. `tofu init/validate/plan/apply` with `spike-` name; fetch `/healthz`, `/mcp`.
52 3. Run the bwrap probe, hold a run past 10 minutes, kill and restore the volume.
53 4. `tofu destroy`, then confirm the zone and account list nothing tagged `spike-`.
forge.json +1 −1
@@ -6,7 +6,7 @@
66 "ready": 90
77 },
88 "check": {
9 "run": "bun x tsc --noEmit && bun scripts/test-shards.ts --timeout 60000 test/crew.test.ts test/crew-wrapup.test.ts test/crew-limits.test.ts test/crew-sandbox.test.ts test/steer.test.ts test/landing.test.ts test/decide.test.ts test/ledger.test.ts test/checks.test.ts test/dependencies.test.ts test/autonomy.test.ts test/studio.test.ts test/studio-words.test.ts test/forks.test.ts test/fork-flow.test.ts test/cross-repo-pull.test.ts test/pulls.test.ts test/apps.test.ts test/apps-adversarial.test.ts test/apps-busy.test.ts test/web.test.ts test/git-trust.test.ts test/helpers.test.ts test/shards.test.ts test/migrate.test.ts test/vitals.test.ts test/asks.test.ts test/feed.test.ts test/feed-page.test.ts test/offline.test.ts test/share.test.ts",
9 "run": "bun x tsc --noEmit && bun scripts/test-shards.ts --timeout 60000 test/crew.test.ts test/crew-wrapup.test.ts test/crew-limits.test.ts test/crew-sandbox.test.ts test/steer.test.ts test/landing.test.ts test/decide.test.ts test/ledger.test.ts test/checks.test.ts test/dependencies.test.ts test/autonomy.test.ts test/studio.test.ts test/studio-words.test.ts test/forks.test.ts test/fork-flow.test.ts test/cross-repo-pull.test.ts test/pulls.test.ts test/apps.test.ts test/apps-adversarial.test.ts test/apps-busy.test.ts test/web.test.ts test/git-trust.test.ts test/helpers.test.ts test/shards.test.ts test/migrate.test.ts test/vitals.test.ts test/asks.test.ts test/feed.test.ts test/feed-page.test.ts test/offline.test.ts test/share.test.ts test/forge-cloudflare.test.ts",
1010 "timeout": 1800
1111 }
1212 }
infra/forge-cloudflare/README.md added +23
@@ -0,0 +1,23 @@
1 # infra/forge-cloudflare
2
3 One forge behind a Cloudflare Worker front door: a Worker, one Durable Object
4 (`Forge`) that forwards to the body and answers `/healthz` itself when the body
5 is down, a proxied DNS record, and — when `body_url` is empty — a tunnel for a
6 body with no public address. Every name starts `spike-`; the variable
7 validation refuses any other.
8
9 ```
10 export CLOUDFLARE_API_TOKEN=… AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…
11 tofu init -backend-config=backend.hcl # bucket, key=spike-<name>/tofu.tfstate, endpoints.s3, region
12 tofu plan -var name=spike-a -var account_id=… -var zone_id=… -var zone_name=… -var body_url=https://…
13 tofu apply … # then: curl $(tofu output -raw url)/healthz and /mcp
14 tofu destroy … # always, when done
15 ```
16
17 The credentials are the run's declared secrets (`docs/APPS.md`, _Forge
18 secrets_); none is in this directory. State is locked with a lock object in the
19 bucket (`use_lockfile`).
20
21 **Not here:** the Container body. The Cloudflare provider has no Containers
22 resource, so that body is deployed with `wrangler` and its URL given as
23 `body_url`. See `docs/SPIKE-CLOUDFLARE.md`.
infra/forge-cloudflare/main.tf added +74
@@ -0,0 +1,74 @@
1 locals {
2 host = "${var.name}.${var.zone_name}"
3 use_tunnel = var.body_url == ""
4 body_url = local.use_tunnel ? "https://${var.name}-body.${var.zone_name}" : var.body_url
5 }
6
7 # --- the front door: one Worker, one Durable Object -------------------------
8
9 resource "cloudflare_workers_script" "front" {
10 account_id = var.account_id
11 script_name = var.name
12 content = file("${path.module}/worker.js")
13 main_module = "worker.js"
14
15 bindings = [
16 { name = "FORGE", type = "durable_object_namespace", class_name = "Forge" },
17 { name = "BODY_URL", type = "plain_text", text = local.body_url },
18 ]
19
20 migrations = {
21 new_tag = "v1"
22 new_sqlite_classes = ["Forge"]
23 }
24 }
25
26 resource "cloudflare_workers_route" "front" {
27 zone_id = var.zone_id
28 pattern = "${local.host}/*"
29 script = cloudflare_workers_script.front.script_name
30 }
31
32 # The route needs a proxied record to attach to; 192.0.2.1 is the documentation
33 # address, never reached because the Worker answers first.
34 resource "cloudflare_dns_record" "front" {
35 zone_id = var.zone_id
36 name = var.name
37 type = "A"
38 content = "192.0.2.1"
39 proxied = true
40 ttl = 1
41 comment = "forge spike ${var.name}: tofu destroy removes this"
42 }
43
44 # --- the tunnel, only when the body is a box that has no public address -----
45
46 resource "cloudflare_zero_trust_tunnel_cloudflared" "body" {
47 count = local.use_tunnel ? 1 : 0
48 account_id = var.account_id
49 name = var.name
50 config_src = "cloudflare"
51 }
52
53 resource "cloudflare_zero_trust_tunnel_cloudflared_config" "body" {
54 count = local.use_tunnel ? 1 : 0
55 account_id = var.account_id
56 tunnel_id = cloudflare_zero_trust_tunnel_cloudflared.body[0].id
57 config = {
58 ingress = [
59 { hostname = "${var.name}-body.${var.zone_name}", service = "http://127.0.0.1:3000" },
60 { service = "http_status:404" },
61 ]
62 }
63 }
64
65 resource "cloudflare_dns_record" "body" {
66 count = local.use_tunnel ? 1 : 0
67 zone_id = var.zone_id
68 name = "${var.name}-body"
69 type = "CNAME"
70 content = "${cloudflare_zero_trust_tunnel_cloudflared.body[0].id}.cfargotunnel.com"
71 proxied = true
72 ttl = 1
73 comment = "forge spike ${var.name}: tunnel to the body"
74 }
infra/forge-cloudflare/outputs.tf added +8
@@ -0,0 +1,8 @@
1 output "url" {
2 value = "https://${local.host}"
3 }
4
5 output "tunnel_token_command" {
6 description = "Run on the body when it has no public address."
7 value = local.use_tunel ? "cloudflared tunnel token ${cloudflare_zero_trust_tunnel_cloudflared.body[0].id}" : null
8 }
infra/forge-cloudflare/variables.tf added +27
@@ -0,0 +1,27 @@
1 variable "name" {
2 description = "The forge's name. Must start with spike- while this is a spike; every resource carries it."
3 type = string
4 validation {
5 condition = can(regex("^spike-[a-z0-9-]{1,40}$", var.name))
6 error_message = "name must match ^spike-[a-z0-9-]{1,40}$ — the spike never touches names outside that prefix."
7 }
8 }
9
10 variable "account_id" {
11 type = string
12 }
13
14 variable "zone_id" {
15 type = string
16 }
17
18 variable "zone_name" {
19 description = "The zone's domain, e.g. example.com; the forge answers at <name>.<zone_name>."
20 type = string
21 }
22
23 variable "body_url" {
24 description = "Where the forge's body answers over HTTPS (a Fly app, a Container's URL, a box behind a tunnel). Empty: the module makes a tunnel and you point cloudflared at it."
25 type = string
26 default = ""
27 }
infra/forge-cloudflare/versions.tf added +27
@@ -0,0 +1,27 @@
1 terraform {
2 required_version = ">= 1.8"
3 required_providers {
4 cloudflare = {
5 source = "cloudflare/cloudflare"
6 version = "~> 5.0"
7 }
8 }
9
10 # State lives in the forge's own S3-compatible bucket, under this forge's
11 # prefix and credential. Everything else comes at init time:
12 # tofu init -backend-config=backend.hcl
13 # where backend.hcl holds bucket, key = "spike-<name>/tofu.tfstate",
14 # endpoints = { s3 = "https://…" }, region, and the skip_* flags below.
15 # Credentials come from AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, which
16 # the run's declared secrets put in the environment (never a file).
17 backend "s3" {
18 use_lockfile = true # S3 conditional writes: a lock object beside the state
19 skip_credentials_validation = true
20 skip_region_validation = true
21 skip_requesting_account_id = true
22 skip_metadata_api_check = true
23 use_path_style = true
24 }
25 }
26
27 provider "cloudflare" {} # CLOUDFLARE_API_TOKEN from the environment
infra/forge-cloudflare/worker.js added +41
@@ -0,0 +1,41 @@
1 // The front door of one forge: a Worker that hands every request to the
2 // forge's Durable Object, which forwards it to the body (BODY_URL) and
3 // answers /healthz itself when the body is asleep. No dependencies.
4
5 const HOP = ["host", "cf-connecting-ip", "x-forwarded-host"];
6
7 /** The request the body sees: same path and query, the body's origin, the public scheme told. */
8 export function forward(request, bodyUrl) {
9 const from = new URL(request.url);
10 const to = new URL(from.pathname + from.search, bodyUrl);
11 const headers = new Headers(request.headers);
12 for (const h of HOP) headers.delete(h);
13 headers.set("x-forwarded-proto", "https");
14 headers.set("x-forwarded-host", from.host);
15 const init = { method: request.method, headers, redirect: "manual" };
16 if (request.method !== "GET" && request.method !== "HEAD") init.body = request.body;
17 return new Request(to, init);
18 }
19
20 export class Forge {
21 constructor(state, env) {
22 this.env = env;
23 }
24 async fetch(request) {
25 try {
26 return await fetch(forward(request, this.env.BODY_URL));
27 } catch {
28 const { pathname } = new URL(request.url);
29 if (pathname === "/healthz")
30 return Response.json({ ok: false, body: "unreachable" }, { status: 503 });
31 return new Response("the forge's body is not answering\n", { status: 502 });
32 }
33 }
34 }
35
36 export default {
37 fetch(request, env) {
38 const id = env.FORGE.idFromName("forge");
39 return env.FORGE.get(id).fetch(request);
40 },
41 };
test/forge-cloudflare.test.ts added +29
@@ -0,0 +1,29 @@
1 /** The Worker front door: what the body is sent, and what is said when it is down. */
2 import { describe, expect, test } from "bun:test";
3 import { readFileSync } from "node:fs";
4 // @ts-expect-error plain JS, deployed as is
5 import { Forge, forward } from "../infra/forge-cloudflare/worker.js";
6
7 describe("forge-cloudflare worker", () => {
8 test("forward keeps path and query, swaps the origin, tells the public scheme", () => {
9 const r = forward(
10 new Request("https://spike-a.example.com/api/v1/x?y=1", { headers: { "cf-connecting-ip": "1.2.3.4" } }),
11 "https://body.example.net",
12 ) as Request;
13 expect(r.url).toBe("https://body.example.net/api/v1/x?y=1");
14 expect(r.headers.get("x-forwarded-host")).toBe("spike-a.example.com");
15 expect(r.headers.get("x-forwarded-proto")).toBe("https");
16 expect(r.headers.get("cf-connecting-ip")).toBeNull();
17 });
18
19 test("an unreachable body is a 503 on /healthz and a 502 elsewhere", async () => {
20 const f = new Forge({}, { BODY_URL: "http://127.0.0.1:1" });
21 expect((await f.fetch(new Request("https://h/healthz"))).status).toBe(503);
22 expect((await f.fetch(new Request("https://h/mcp"))).status).toBe(502);
23 });
24
25 test("the module refuses names outside the spike- prefix", () => {
26 const v = readFileSync("infra/forge-cloudflare/variables.tf", "utf8");
27 expect(v).toContain('^spike-[a-z0-9-]{1,40}$');
28 });
29 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs