Snapshots — a forge back from a copy

Every night at 08:00 UTC a forge writes what cannot be rebuilt to a store, keeps the newest fourteen, and an admin can take one any time from Settings → Backups. Restoring is a command a person runs against an empty data directory. src/snapshot/.

A snapshot sends what changed

Whole, a snapshot was 8.9 GB over a home uplink every time: two hours in which no release could restart the forge (2026-11-14). Now every file of a snapshot is cut into chunks (src/snapshot/chunks.ts), each one object under chunks/, named by the SHA-256 of its bytes and compressed when that is smaller. A chunk the store already lists is not sent again.

object of its own: format: 2, and for every file its path, size, SHA-256 and chunks in order. It is written last, after a fresh listing shows every chunk it names. So each snapshot restores from its manifest alone, whatever became of the others, and one cut short (a release, a dropped line) is not a snapshot: the next sends only the chunks still missing.

and every file against its size and hash, and refuses otherwise. bun run src/snapshot/verify.ts <name|latest> [--deep] [--offsite] asks the same of a snapshot without restoring it: every chunk listed, or with --deep every chunk and file read and proved.

chunks of 256 KiB to 4 MiB), so a pack whose bytes only moved is the same chunks; the rule depends on a seed the forge keeps in snapshot/chunk.seed, so chunk sizes do not say which file they are of. A database copy is cut every 4 MiB: VACUUM INTO writes it anew each time, so one that changed goes whole (measured: two copies of the forge's own five hours apart shared almost nothing, however cut) and one that did not is not sent.

fourteen are deleted, every chunk no remaining manifest names is deleted, and nothing else. Nothing is counted, so nothing drifts. The sweep deletes nothing when a manifest does not read, is of a format it does not know, or appeared while it was looking.

name under the lane's key, and its body is sealed under a fresh IV once, when first sent. The bucket holds no hash of any plaintext and never two ciphertexts of one; it sees object sizes and which chunks each night added.

files are objects under its own prefix, the database as zstd parts (forge.db.zst.0000, …). Restore and verify still read those; nothing writes them, and retention lets them go.

The log line snapshot taken says what a night cost: chunks named, sent and sentMb put, heldBefore already there.

What a snapshot holds

never a raw copy of a file in WAL mode). It is read as a stream, never held whole.

secret.key.wrapped, AES-256-GCM under a key scrypt-derived from the operator's passphrase. The raw secret.key is never written to the store, so whoever reads the bucket holds the data without the key. See The key.

back with an owner-only mode; the forge refuses a key anyone else can read.

and none left out without the snapshot saying so (src/snapshot/bundle.ts). The manifest lists every repository the database held (repositories), and for each without a bundle its reason (absent; an empty repository has nothing to bundle and is marked empty). A ref whose commits are missing fails git's whole bundle, so the refs are sorted and every one that resolves is bundled; the others are in brokenRefs by name and are one line a repository for the forge's admin ("2 branches of alice/notes cannot be backed up: their commits are missing (…)"), linked to its Branches page; how to clear it is docs/features/backup-refs.md. A repository with no bundle at all turns the backups verdict red, naming it: a snapshot that lacks a repository is not green however fresh. A bundle's deadline follows the repository's size (2 min, and 2 s for each MB of its objects and those it borrows, at most an hour), and one that runs out says so with the time it had. verify.ts refuses a manifest in which a repository has neither a bundle nor a reason.

inside are copied the consistent way; their -wal/-shm side-files are the live process's and stay out. Previews are rebuilt from production and stay out.

dropped for the crew, kept nowhere else. Uploads still in flight (dot-files) stay out. The Drop lives beside apps/, never inside an environment's FORGE_APP_DATA, so a preview's copy of production's data never carries it. A restore returns each file byte for byte (test/drop-backup.test.ts). A large Drop makes the night longer by the time it takes to read and hash it; only chunks the store lacks are sent.

(runs/<day>[-n].ndjson, cut at 4 MB of logs), in the order the logs were last written: a bundle once full is the same bytes in every later snapshot, so it goes up once. Restore unpacks them to the logs they were.

with its size, hash and chunks, sent (what this snapshot itself put), and key: wrapped or none (older snapshots: raw).

What stays out is what a boot or a deploy rebuilds: clones, checks, pictures, previews.

Is a snapshot encrypted before it leaves?

Not unless you say so. By default the destination holds what the forge wrote: the database, bundles, app data and run logs as they are (compressed), readable by whoever can read the bucket. Two things nearby sound like encryption and are not: the destination's keys are sealed in the forge (the Backups page's "keys sealed in this forge"), and the forge's secret.key travels only wrapped (key=wrapped). Neither encrypts a snapshot's data. The Backups page, the terminal's Resilience screen, MCP backups (destination.bodies, and bodies on each lane) and the snapshot taken log line (bodies=) say which it is, in the same words: "encrypted before upload", or "not encrypted before upload; the bucket's own protection only".

FORGE_SNAPSHOT_SEAL=1 seals the destination: it becomes the same lane the off-box mirror is (src/snapshot/sealed_store.ts), every chunk named by an HMAC and every body and manifest AES-256-GCM under a key derived from FORGE_SNAPSHOT_PASSPHRASE. Before turning it on, know that every restore then needs the passphrase: keep it somewhere that is not the forge. Set without a passphrase, the forge writes no snapshot at all, never a plain one, and says why. Snapshots already in the bucket unsealed stay where they are; the sealed lane does not list them (they restore with the setting off, and retention no longer reaches them: delete them by hand once the sealed ones have been drilled). The restore, verify and drill commands read the same setting from the environment.

The key

The wrapping secret is FORGE_SNAPSHOT_PASSPHRASE, in the forge's environment. It is not in the bucket, the repository or the database; the operator holds it elsewhere too (a password manager, a sealed envelope).

says the last one did.

restore.

Repositories, issues, app data and everything unsealed restore; every sealed value (forge, org and repo secrets, app variables, origin tokens, subscription tokens) is lost and is set again by hand. Choose the passphrase knowing that. A passphrase changed on the forge applies to the next snapshot; older snapshots stay under the one they were taken with.

Snapshots from before this change hold the raw secret.key; they still restore with the key, and the Backups page tells you to take a new one. Delete the old ones from the bucket if the bucket is not trusted.

Where

in the pod (S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY, S3_SECRET_KEY, S3_REGION). The platform's own lanes mirror the bucket off the machine.

access key, secret key): sealed as forge-level secrets (SNAPSHOT_S3_*), write-only, with a test button that lists the bucket. Precedence: sealed, then FORGE_SNAPSHOT_DIR, then S3_*; the page says which is in effect. A restore into an empty directory has nothing sealed yet, so it reads the environment: keep S3_* or FORGE_SNAPSHOT_DIR at hand for it.

instead of showing an empty list that looks like a clean bill.

Off the machine

Home's Last off-box backup is the newest time a snapshot was known to be off the machine (src/snapshot/off_box.ts): the newest the mirror holds, or the newest that landed in a destination that is a bucket, whichever is later. Each landing records whether its store was one (offBox in snapshot/state.json): a directory never is, whatever volume it is on, nor is a bucket served from loopback. Before #3701 the tile read the mirror alone, so a forge whose nightly landed in the platform's bucket every morning, with no second bucket, read "never". An outcome kept before stores were recorded is annotated from the store in effect at the first mirror pass after boot.

The off-box mirror

A second S3-compatible bucket (Backblaze B2, say) that every snapshot is copied to, so the snapshots do not die with the machine. Seal it on the Backups page (SNAPSHOT_MIRROR_S3_*, write-only like the destination) or give FORGE_MIRROR_S3_ENDPOINT, _BUCKET, _ACCESS_KEY, _SECRET_KEY and optionally _REGION. A pass runs right after each snapshot, thirty seconds after boot and every hour: it copies the newest fourteen snapshots the mirror lacks (the chunks its manifest names that the mirror does not list, then the manifest, so a half copy is never listed and a night costs the mirror what it cost the destination), then prunes and sweeps the mirror to fourteen. What it found is kept in mirror/state.json under the data directory. The Backups page shows the mirror's reachability, its newest snapshot and when it was last confirmed level. A pass that fails (store unreachable, or a snapshot that did not land) or a mirror not confirmed level for a day is one card on the board (mirror) and one push to the admin, the path the drill uses. Not done: the mirror is copied from the primary store, not written in parallel, so a snapshot that never landed on the primary is not seen by it.

The store is Bun.S3Client or a directory behind one small interface (src/snapshot/store.ts): put, get, list, delete. Nothing else. Every call is tried five times; an attempt that has not answered by its deadline counts as failed. A put's deadline follows its size: its bytes at a floor of 5 Mbit/s, plus 60 s. A fixed 180 s once failed every attempt at a 334 MB database behind a 15 Mbit/s uplink (#2536). An attempt past its deadline is aborted, not just abandoned: a put goes as a presigned PUT through fetch with the attempt's signal, because client.write takes none, and an abandoned upload went on sending beside the tries after it (#13733).

Backup lanes

The platform's backup lanes (FORGE_BACKUP_LANES) and how each is judged: docs/features/backup-truth.md, "Platform backup lanes".

Progress

A snapshot uploads eight chunks at a time, and only those the store lacks. While one runs, progress (chunks and bytes sent) shows on Settings → Backups and as a line in the log every 500 objects or minute. The terminal has no backups view, so it has no readout.

When a snapshot fails

Its outcome is kept in snapshot/state.json under the data directory, so a restart forgets nothing. A failure shows straight away as a card on the board (snapshot-failed, pointing at Backups), in the terminal's dashboard among the forge's needs, and as a red line on the Backups page with the reason and since when. The forge tries again after ten minutes (SNAPSHOT_RETRY_MS, capped at the next nightly), not the next night: a store that is down gets six tries an hour, each of five attempts per put. The next snapshot that lands, nightly or on demand, clears both.

A forge given nowhere to put a snapshot has not failed: nothing is kept, no card is raised, and nothing is retried sooner than the nightly. The Backups page says there is no destination.

The restore drill

A backup nobody has restored is a hope. Settings → Backups → Run a drill (admins only; anyone else gets 403) restores the newest snapshot into a scratch directory under the data directory, opens the database read-only, counts repositories and issues, opens one sealed value with the restored key, and removes the scratch whatever happened. The page waits until it finishes, for up to 10 minutes; a step that hangs past that is red and names the step. The restore reads several chunks at a time (RESTORE_IN_FLIGHT), each written in its file's order: read one after another, a snapshot of thousands of objects spent the drill's whole 10 minutes in the restore step (#2539). Repositories come back four clones at a time, not one after another (#2543). One drill runs at a time: a second click, or a retry after a proxy timeout, joins the drill already running rather than starting another restore. Every run is a row in the drill table, and the last one shows as a line on the Backups page: when, which snapshot, how long, the counts, and green or the step that failed.

The drill runs on its own every Sunday 09:00 UTC, and a forge that was down then catches up five minutes after boot. A red drill does not keep the week's slot: it is owed again RED_RETRY_MS (three hours) after it ran, and at a boot once it is BOOT_RED_FLOOR_MS (30 minutes) old, until one passes (src/snapshot/due.ts). The floor keeps a crash loop with a broken restore from running a restore on every boot. Kept for the week, a drill red at 09:11 stood as a card long after the fix deployed at 10:18 (#2541).

Run a drill on the same page (admins only) restores the newest snapshot into scratch, opens it, counts what is there and unseals a value; the page waits for it, up to 10 minutes, and the last drill shows there as one line, ending with the drill's time against its budget and each part's time, longest first ("6 min 12 s of its 14 min budget: clone 4 min, download 58 s, …"), or only the time against the budget when no part was timed; drills from before this show none. The budget is 10 min plus 1 s per MB of snapshot, at most 45 min (docs/RECOVERY.md). The parts are download, decompress, clone, open, and start and migrate (the whole side-boot of the restored forge, process start included). The same words, slowest named, are in the drill's log line and in a red drill's card ("… (download about 8 min, decompress 40 s, …; download was the slowest)").

Each snapshot's manifest keeps the five counts of its database; a restore that counts differently is red at count and, like any red drill, one board line and one push. Snapshots from before the counts were kept are not compared.

Failure domains, and what the owner is told

Failure domains (a card at the foot of /settings/backups; the old /settings/resilience sends there, and /resilience is the THREATS.md page): one row per failure domain from docs/DOMAINS.md — what breaks, the words the forge says, and the domain's last kept drill (when, green or red with its reason, seconds until back) or the sentence on why it is not drilled. In the terminal it is the Failure domains command (ctrl+k; the Resilience command is the THREATS.md one), shown to an admin only and read through the same listResilience action. Only the latest drill per domain is shown. An admin presses d on Failure domains to run the restore drill, behind a confirm, with the same words as the Backups button and MCP restore_drill; it keeps the snapshot drill record only and leaves the domain drills as they are. The Owner's needs command (admin only, ownerView) is the owner's one screen: the forge's needs as the board strip says them, where snapshots go (and any mirror) with the newest snapshot, and every domain with what the owner does.

A red drill, or none in eight days, is one line for the forge's admin on the board's forge strip, in the terminal's dashboard (same forgeNeeds), and in MCP health and asks; it names the failed step and the next green drill clears it. An admin with a phone URL gets one push per cause (a new red drill, or the silence), also when eight days pass with no event: the notifier wakes every 10 minutes for it. The time the forge first owed a drill is kept at boot in drill/first-seen; if the disk will not take it, a warning is logged and a forge with no drill ever is not flagged.

Snapshots written to a directory on the forge's own volume, with no off-box mirror set, are lost with that volume. That is one line for the forge's admin, "Snapshots are kept on the forge's own volume and mirrored nowhere: losing the volume loses them too.", on the board's forge strip (linking Settings → Backups), in the terminal's dashboard (same forgeNeeds), in the asks, in GET /api/v1/health and MCP health. Anyone else sees nothing. It clears when the destination moves to another volume or a bucket, or a mirror is set. Every one of those reads the vitals through forgeVitals in src/actions/health.ts.

Restore

The printable card, seven lines, is RESTORE.md; a test runs it.

# same S3_* as the forge runs under, or FORGE_SNAPSHOT_DIR
FORGE_SNAPSHOT_PASSPHRASE=… \
  bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z /path/to/empty/data

With the passphrase the key is unwrapped into secret.key (mode 0600) and sealed values open. A wrong passphrase refuses the restore before anything is written. With none, the key is left out, repositories and the rest restore, and the command lists every sealed value by name (NAME (app slug), NAME (scope secret)) that must be set again; the restored forge makes a new key on first boot.

It refuses a directory that is not empty, a snapshot it cannot find, one that is missing a chunk, and any chunk or file that is not the bytes its manifest names. Bundles become bare repositories with the forge's hooks installed; the mirror's memory of the bundle as an origin is removed. Then start the forge on that directory: it migrates the database forward if it has moved on, reconciles the index and redeploys what runs.

test/snapshot-incremental.test.ts proves the chunks: the second snapshot puts only what changed; the first, the second and those left after a sweep restore byte for byte; a missing chunk fails the verify; the sweep never takes a chunk a kept manifest names. test/snapshot.test.ts is the proof of the whole: a running forge with history, a sealed variable and an app's SQLite is snapshotted, restored into an empty directory with the passphrase and booted there as the same forge; restored without it, the repositories come back and the sealed variable is reported by name. It also checks that the raw key is in no file of the store.

The dump, a line per lane and the platform-free drill: backup-truth.md, _Leaving the platform_.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs