The off-site backup lane

The forge's own backup lane (#11805) is the mirror pass that runs in the forge's process every hour and after every snapshot. It copies the newest snapshots to a bucket on neither machine (FORGE_MIRROR_S3_*, or the mirror sealed on the Backups page), and every object it writes is sealed first: AES-256-GCM under a fresh IV, the key derived from FORGE_SNAPSHOT_PASSPHRASE by scrypt (src/snapshot/sealed_store.ts). Paths inside a snapshot are hidden (an HMAC of each path), so the bucket does not even name a repository.

sealed/lane.json: a random 16-byte salt and a check value (an HMAC under the derived key, never the key). A guess at the passphrase costs one scrypt for that lane alone, and a wrong passphrase is said as such before any object is read.

bucket. Keep it with the operator, apart from the bucket's keys.

FORGE_SNAPSHOT_PASSPHRASE is not set, and /system shows the lane red; not one unsealed byte leaves the forge.

For the lane to stand without pg-grand, the nightly snapshot itself must land on the forge's own disk (FORGE_SNAPSHOT_DIR on its volume) rather than the platform's MinIO: the lane copies from there.

Restore from it

On a fresh machine, follow RESTORE.md, exporting the lane's bucket in place of line 3, keeping line 4 (the passphrase), and adding --offsite to line 5:

export FORGE_MIRROR_S3_ENDPOINT=… FORGE_MIRROR_S3_BUCKET=… FORGE_MIRROR_S3_ACCESS_KEY=… FORGE_MIRROR_S3_SECRET_KEY=… bun run src/snapshot/restore.ts latest <data-dir> --offsite

bun scripts/restore-drill.ts <empty-data-dir> --offsite drills it the same way. Without the passphrase both refuse before writing anything; a restore never starts a lane on an empty bucket.

The drill

The weekly drill, the Backups page's button and the MCP restore_drill restore the newest snapshot from the off-site lane when the forge has one (the destination otherwise) into a scratch forge, boot it, and check that /api/v1/health answers and / answers below 400. A drill waits for a mirror pass in flight, at most the drill's floor budget (10 min), then reads the lane as it stands. Each drill is kept with the store it restored from.

Where to see it

where the lane writes, whether it is encrypted, its last success, and the last drill with where it restored from.

{ health, says, where, sealed, lastSuccess, newest, error, drill }.

All three are a forge admin's alone.

When the nightly fails (#13046)

A failed nightly backup says why, in one line: timeout after 42 s, 1.5 MiB sent. The word is the cause (timeout, auth, disk, refused, or unknown when the error names none), then the seconds the attempt ran and the bytes that landed before it failed. It is read once from the error (src/snapshot/cause.ts) and kept with the snapshot outcome, so a landed snapshot clears it.

backup" (the palette's Off-site entry).

backupFailure: { cause, bytes, seconds, says }, or null while the last snapshot landed.

All three are a forge admin's, a person's, as the rest of the lane is. Only the cause word and two numbers are shown, never the error text.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs