Recovery — read this at 3 a.m.

Print it. Keep it somewhere that is not the forge. test/recovery-doc.test.ts keeps it true to src/snapshot/.

Where the copies are

Snapshots are objects in the forge's bucket (S3_BUCKET, at S3_ENDPOINT, region S3_REGION): one manifest each under snapshots/<name>/, named like 2026-09-22T08-00-00Z, and the chunks they name under chunks/. The restore command accepts latest for the newest; no object is called that.

Where the verdicts are posted

Sundays 09:00 UTC the forge drills a restore of the newest snapshot, boots it on a side port and reads /api/v1/health, all within 10 min plus 1 s per MB (10^6 bytes) of the snapshot's bytes, at most 45 min — a 400 MB snapshot has 16 min 40 s. Every get and its retries keep to what is left of that budget, so a stalled store fails the drill within it, naming the object. One comment lands on the recovery issue (FORGE_RECOVERY_ISSUE in FORGE_RECOVERY_REPO): snapshot, restore time, health, counts, pass or fail. A failed drill is an inbox card naming the step (fetch, restore, open, count, unseal, health) and its budget in minutes, run again at boot and every three hours until one passes. No comment this week: backups unproven.

What a restore needs

Export these, with the values the forge ran under:

S3_ENDPOINT S3_BUCKET S3_REGION S3_ACCESS_KEY S3_SECRET_KEY

and FORGE_SNAPSHOT_PASSPHRASE, which is in the owner's safe and nowhere else. Without it the restore still works, but the secret key stays behind and every sealed value is set again by hand. A wrong one refuses before writing. A machine with git, bun and a clone of the forge (bun install).

Route 1 — the restore command

bun run src/snapshot/restore.ts <snapshot-name|latest> <data-dir> [--clones N] [--offsite] [--forge NAME]

<data-dir> must be empty or new. --clones N (1–32): clones at once (default: cores less one, 2–16). --offsite: features/offsite-lane.md. --forge NAME: one hosted forge, features/hosted-backup.md.

Then boot the forge on that directory, on a side port, off the public one:

FORGE_DATA=<data-dir> PORT=3999 FORGE_SSH_PORT=0 bun run src/main.ts

Look at, in order:

  1. curl localhost:3999/api/v1/health answers ok.
  2. The home page, signed in as the owner: the repositories and the board.
  3. One repository's issues: the latest ones you remember are there.

If all three hold, point the real service at <data-dir>.

Route 2 — the objects by hand

Use it when the restore command fails or only one file is wanted. The manifest gives each file's sha256 and its chunks to join in order; a chunk is the object chunks/<id>.chunk: one byte (1 = zstd), then bytes. No format: plain objects.

forge.db the database secret.key.wrapped the key, under the passphrase ssh/... the SSH host key repos/<owner>/<repo>.bundle one git bundle per repository apps/<id>/production/data/... each app's data runs/... run logs

Put forge.db and ssh/ in a new data directory; turn each bundle back into a repository with git clone --mirror <file> <data-dir>/repos/<owner>/<repo>.git (the layout src/git/repos.ts reads). Boot as above. The key is wrapped with scrypt and AES-256-GCM (src/snapshot/wrap.ts): only restore.ts opens it, so by hand the sealed values are set again.

When the drill is red

The weekly drill restores the newest snapshot into scratch and names where it stopped: a step, and for restore a phase inside it. Each entry says what it was doing, the likely cause, and the first thing to check. Whatever the step, the next move is the same: run Route 1 by hand on a new directory and watch which line it stops at.

Steps

unreachable or its keys changed, or no snapshot exists. Check the S3_* variables and that the store lists the snapshot objects.

Likely a bad snapshot or a slow store. Check the phase, then Route 1.

corrupt. Check its size against the snapshot's manifest.

Likely the snapshot was cut mid-write or rows were lost. Check which table the message names and its count in the snapshot.

is missing or from another forge, or the passphrase is wrong. Check FORGE_SNAPSHOT_PASSPHRASE and that the snapshot holds the key.

migration or config problem, or a boot slower than the limit. Boot it by hand as in Route 1 and read its log.

Restore phases

manifest object is missing or damaged. Check the object exists in the store.

how far it got against the manifest's file count, then retry.

disk is full or the archive is damaged. Check free space.

or many repositories. Compare its counts with the snapshot's and retry the restore by hand with Route 1.

passphrase is missing or wrong. Check FORGE_SNAPSHOT_PASSPHRASE.

Replacing a leaked secret.key: docs/features/rotate-secret-key.md.

What a snapshot does NOT hold

again by hand (the restore lists them by name).

repositories.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs