Public: anyone can read this; the crew can change it.

Hard to bring down, not an Achilles heel: one place to manage, many places to fail; the forge's failure domains named and each one tested #1148

shipped production is serving it opened this has it
opened this

From the owner, 2026-10-12: "I want everything in one place but I don't want everything able to fail in one place. This needs to make the kingdom hard to bring down, not an Achilles heel. Simple and in a single place so I can very easily manage backups and trust them."

Today one droplet runs the forge, its database, its snapshot store, its runner, its edge tunnel and its DNS automation. One place to manage, and one place to fail.

Done looks like:

  • [ ] A one-page map of the forge's failure domains: compute (the pod), state (the volume), the snapshot store, the edge (tunnel and DNS), the identity (the origin's OAuth, the owner's passkeys), the model credential, the runner. For each: what breaks when it fails, how the forge says so, how long until it is back, and what the owner does. Kept in docs/DOMAINS.md, read by the sentinel (#1131).
  • [ ] No two domains share a single point by accident: the snapshot store is not on the state volume; the edge keeps serving a static "the forge is restarting" page from the edge itself when compute is gone; the owner can sign in with a passkey when the origin is down; the runner's loss slows the gate and never blocks a release of a fix.
  • [ ] Each domain has a drill, run on a schedule like the restore drill: kill the pod, cut the tunnel, lose the runner, expire a token; the forge's own words during each are recorded and compared to the map.
  • [ ] Management stays in one place: the owner reads the map, the drills and the snapshots on one page (Settings → Resilience), and that page is itself available from the edge when the forge is not.

Files I expect: docs/DOMAINS.md, src/kernel/vitals.ts, src/web/pages/resilience.ts (new), src/cloud/edge.ts (the static page), the operator's standby scripts.

commented

Parked by the crew: the check failed ("the gate's list > every test/*.test.ts is in exactly one list, and every entry exists") and the decider saw no point in another try (pull #1202 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs