Failure domains

The seven ways the forge stops, and what it says when each does. src/kernel/domains.ts is the data; test/domains.test.ts keeps this table equal to it and proves each quoted phrase is what the forge's own code emits under that condition. The words are quoted from kernel/vitals.ts, never drafted. "Nothing yet" means the forge has no words for that domain today. Minutes back are estimates, not measurements. A drill named here is the name issue #1195 will build. Today the restore drill exists, and POST /api/v1/drills keeps a drill's verdict for any domain here (see the README).

DomainWhat breaksThe forge saysMinutes backThe ownerDrill
Compute (the pod)Nothing answers until the pod is rescheduled.nothing yet2Wait for the platform to restart it; read the pod's events if it does not.kill-pod
State (the volume)Writes fail once the volume fills; the data is lost with the volume."Disk is nearly full:"60Free or grow the volume; if it is gone, restore the newest snapshot (docs/RECOVERY.md).none: Filling or detaching the live volume would lose writes.
The snapshot storeBackups stop landing or cannot be read back; nothing live is affected."The restore drill failed at"30Fix the bucket or its keys, then run the restore drill again.restore-drill
The edge (tunnel and DNS)The forge runs but nobody outside can reach it.nothing yet10Restart the tunnel; check the DNS record.none: Cutting the tunnel cuts every person and agent off the forge.
IdentitySign-in fails; sessions and tokens already issued may still work.nothing yet15Restore the sign-in provider's settings; sign in with the admin password.none: Breaking sign-in locks the owner out of the forge that would fix it.
The model credentialAgents cannot start work; everything else runs.nothing yet5Set a fresh credential in the settings.expire-token
The runnerChecks and runs queue behind a lane that is not moving."the oldest since"; "is stuck: no heartbeat for"10Cancel the stuck run; restart the runner if the lane stays empty.lose-runner

Shared by accident

Two domains that fail together are one domain. The forge checks the one pairing it can see from inside: snapshots written to a directory on the volume they copy, with no off-box mirror, are lost with that volume. Vitals then says "Snapshots are kept on the forge's own volume and mirrored nowhere" on the board, the asks and /api/v1/health, and links Settings → Backups, until the destination moves or a mirror is set. "One volume" means the two paths report the same device (st_dev): a snapshot directory on a different filesystem bind-mounted into the data directory is told apart, but two directories on one disk under different mounts that share a device, or two devices backed by the same physical disk, are not.

Running a drill

scripts/drills/run.ts walks you through one. It refuses a domain whose drill is none, prints the fault to cause, and waits for Enter (a line on stdin, so piping one works; closing stdin aborts). It then polls /api/v1/health and / once a second, keeping each distinct line the forge says while unhealthy; when the forge is healthy again it posts what it saw and the seconds it took to POST /api/v1/drills and prints the verdict. If the forge is still down after an hour it posts nothing, says so, and exits 0. A refused post exits non-zero with the forge's message. FORGE_TOKEN must be a site admin's token; without it the script still watches, prints the result as JSON and says that nothing was posted.

FORGE_TOKEN=… bun scripts/drills/run.ts runner --forge https://forge.open-platform.sh
↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs