Snapshots — a forge back from a copy
Every night at 08:00 UTC a forge writes what cannot be rebuilt to a store,
keeps the newest fourteen, and an admin can take one any time from
Settings → Backups. Restoring is a command a person runs against an
empty data directory. src/snapshot/.
A snapshot sends what changed
Whole, a snapshot was 8.9 GB over a home uplink every time: two hours in
which no release could restart the forge (2026-11-14). Now every file of a
snapshot is cut into chunks (src/snapshot/chunks.ts), each one object
under chunks/, named by the SHA-256 of its bytes and compressed when that
is smaller. A chunk the store already lists is not sent again.
- The manifest,
snapshots/<name>/manifest.json, is a snapshot's only
object of its own: format: 2, and for every file its path, size, SHA-256
and chunks in order. It is written last, after a fresh listing shows every
chunk it names. So each snapshot restores from its manifest alone, whatever
became of the others, and one cut short (a release, a dropped line) is not
a snapshot: the next sends only the chunks still missing.
- Proved on the way back. A restore checks every chunk against its name
and every file against its size and hash, and refuses otherwise.
bun run src/snapshot/verify.ts <name|latest> [--deep] [--offsite] asks
the same of a snapshot without restoring it: every chunk listed, or with
--deep every chunk and file read and proved.
- Cutting. Most files are cut where their content says (a rolling hash,
chunks of 256 KiB to 4 MiB), so a pack whose bytes only moved is the same
chunks; the rule depends on a seed the forge keeps in
snapshot/chunk.seed, so chunk sizes do not say which file they are of. A
database copy is cut every 4 MiB: VACUUM INTO writes it anew each time,
so one that changed goes whole (measured: two copies of the forge's own
five hours apart shared almost nothing, however cut) and one that did not
is not sent.
- Retention is mark and sweep. After the manifests past the newest
fourteen are deleted, every chunk no remaining manifest names is deleted, and nothing else. Nothing is counted, so nothing drifts. The sweep deletes nothing when a manifest does not read, is of a format it does not know, or appeared while it was looking.
- Sealed lanes. On the off-site lane a chunk's name is an HMAC of its
name under the lane's key, and its body is sealed under a fresh IV once, when first sent. The bucket holds no hash of any plaintext and never two ciphertexts of one; it sees object sizes and which chunks each night added.
- Before chunks. A manifest with no
formatis an older snapshot: its
files are objects under its own prefix, the database as zstd parts
(forge.db.zst.0000, …). Restore and verify still read those; nothing
writes them, and retention lets them go.
The log line snapshot taken says what a night cost: chunks named, sent
and sentMb put, heldBefore already there.
What a snapshot holds
forge.db, copied consistently while the forge runs (VACUUM INTO,
never a raw copy of a file in WAL mode). It is read as a stream, never held whole.
- The key the database's sealed values need, only wrapped:
secret.key.wrapped, AES-256-GCM under a key scrypt-derived from the
operator's passphrase. The raw secret.key is never written to the store,
so whoever reads the bucket holds the data without the key. See
The key.
ssh/, the host key that is the forge's identity to every clone. It comes
back with an owner-only mode; the forge refuses a key anyone else can read.
- Every repository as one git bundle of every ref (
repos/<owner>/<name>.bundle),
and none left out without the snapshot saying so (src/snapshot/bundle.ts).
The manifest lists every repository the database held (repositories), and
for each without a bundle its reason (absent; an empty repository has
nothing to bundle and is marked empty). A ref whose commits are missing
fails git's whole bundle, so the refs are sorted and every one that resolves
is bundled; the others are in brokenRefs by name and are one line a
repository for the forge's admin ("2 branches of alice/notes cannot be
backed up: their commits are missing (…)"), linked to its Branches page;
how to clear it is docs/features/backup-refs.md. A repository with no bundle at
all turns the backups verdict red, naming it: a snapshot that lacks a
repository is not green however fresh. A bundle's deadline follows the
repository's size (2 min, and 2 s for each MB of its objects and those it
borrows, at most an hour), and one that runs out says so with the time it
had. verify.ts refuses a manifest in which a repository has neither a
bundle nor a reason.
- Every app's production data (
apps/<id>/production/data/…). SQLite files
inside are copied the consistent way; their -wal/-shm side-files are
the live process's and stay out. Previews are rebuilt from production and
stay out.
- Every app's Drop (
drop/<repo id>/<name>): the reference files its owner
dropped for the crew, kept nowhere else. Uploads still in flight (dot-files)
stay out. The Drop lives beside apps/, never inside an environment's
FORGE_APP_DATA, so a preview's copy of production's data never carries it.
A restore returns each file byte for byte (test/drop-backup.test.ts). A
large Drop makes the night longer by the time it takes to read and hash it;
only chunks the store lacks are sent.
- The crew's run logs, a day's worth to a few bundles
(runs/<day>[-n].ndjson, cut at 4 MB of logs), in the order the logs were
last written: a bundle once full is the same bytes in every later
snapshot, so it goes up once. Restore unpacks them to the logs they were.
manifest.json: the name, when, the forge's version, counts, every file
with its size, hash and chunks, sent (what this snapshot itself put),
and key: wrapped or none (older snapshots: raw).
What stays out is what a boot or a deploy rebuilds: clones, checks, pictures, previews.
Is a snapshot encrypted before it leaves?
Not unless you say so. By default the destination holds what the forge
wrote: the database, bundles, app data and run logs as they are (compressed),
readable by whoever can read the bucket. Two things nearby sound like
encryption and are not: the destination's keys are sealed in the forge
(the Backups page's "keys sealed in this forge"), and the forge's
secret.key travels only wrapped (key=wrapped). Neither encrypts a
snapshot's data. The Backups page, the terminal's Resilience screen, MCP
backups (destination.bodies, and bodies on each lane) and the
snapshot taken log line (bodies=) say which it is, in the same words:
"encrypted before upload", or "not encrypted before upload; the bucket's own
protection only".
FORGE_SNAPSHOT_SEAL=1 seals the destination: it becomes the same lane the
off-box mirror is (src/snapshot/sealed_store.ts), every chunk named by an
HMAC and every body and manifest AES-256-GCM under a key derived from
FORGE_SNAPSHOT_PASSPHRASE. Before turning it on, know that every restore
then needs the passphrase: keep it somewhere that is not the forge. Set
without a passphrase, the forge writes no snapshot at all, never a plain one,
and says why. Snapshots already in the bucket unsealed stay where they are;
the sealed lane does not list them (they restore with the setting off, and
retention no longer reaches them: delete them by hand once the sealed ones
have been drilled). The restore, verify and drill commands read the same
setting from the environment.
The key
The wrapping secret is FORGE_SNAPSHOT_PASSPHRASE, in the forge's
environment. It is not in the bucket, the repository or the database; the
operator holds it elsewhere too (a password manager, a sealed envelope).
- Set: every snapshot carries
secret.key.wrapped, and the Backups page
says the last one did.
- Unset: no key travels, and the Backups page says so. Snapshots still
restore.
- Lost: the wrapped key cannot be opened by anyone, including us.
Repositories, issues, app data and everything unsealed restore; every sealed value (forge, org and repo secrets, app variables, origin tokens, subscription tokens) is lost and is set again by hand. Choose the passphrase knowing that. A passphrase changed on the forge applies to the next snapshot; older snapshots stay under the one they were taken with.
Snapshots from before this change hold the raw secret.key; they still
restore with the key, and the Backups page tells you to take a new one.
Delete the old ones from the bucket if the bucket is not trusted.
Where
- On the platform, the app's bucket: the
S3_*variables the platform puts
in the pod (S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY, S3_SECRET_KEY,
S3_REGION). The platform's own lanes mirror the bucket off the machine.
- Anywhere else,
FORGE_SNAPSHOT_DIR=/some/dir. - Or your own bucket, set on the Backups page (endpoint, bucket, region,
access key, secret key): sealed as forge-level secrets
(SNAPSHOT_S3_*), write-only, with a test button that lists the bucket.
Precedence: sealed, then FORGE_SNAPSHOT_DIR, then S3_*; the page says
which is in effect. A restore into an empty directory has nothing sealed
yet, so it reads the environment: keep S3_* or FORGE_SNAPSHOT_DIR
at hand for it.
- Given neither, no snapshots are taken, and the Backups page says so
instead of showing an empty list that looks like a clean bill.
Off the machine
Home's Last off-box backup is the newest time a snapshot was known to be
off the machine (src/snapshot/off_box.ts): the newest the mirror holds, or
the newest that landed in a destination that is a bucket, whichever is later.
Each landing records whether its store was one (offBox in
snapshot/state.json): a directory never is, whatever volume it is on, nor
is a bucket served from loopback. Before #3701 the tile read the mirror
alone, so a forge whose nightly landed in the platform's bucket every morning,
with no second bucket, read "never". An outcome kept before stores were
recorded is annotated from the store in effect at the first mirror pass after
boot.
The off-box mirror
A second S3-compatible bucket (Backblaze B2, say) that every snapshot is
copied to, so the snapshots do not die with the machine. Seal it on the
Backups page (SNAPSHOT_MIRROR_S3_*, write-only like the destination) or
give FORGE_MIRROR_S3_ENDPOINT, _BUCKET, _ACCESS_KEY, _SECRET_KEY and
optionally _REGION. A pass runs right after each snapshot, thirty seconds
after boot and every hour: it copies the newest fourteen snapshots the mirror
lacks (the chunks its manifest names that the mirror does not list, then the
manifest, so a half copy is never listed and a night costs the mirror what
it cost the destination), then prunes and sweeps the mirror to fourteen.
What it found is kept in
mirror/state.json under the data directory. The Backups page shows the
mirror's reachability, its newest snapshot and when it was last confirmed
level. A pass that fails (store unreachable, or a snapshot that did not land)
or a mirror not confirmed level for a day is one card on the board
(mirror) and one push to the admin, the path the drill uses. Not done: the
mirror is copied from the primary store, not written in parallel, so a snapshot
that never landed on the primary is not seen by it.
The store is Bun.S3Client or a directory behind one small interface
(src/snapshot/store.ts): put, get, list, delete. Nothing else. Every call
is tried five times; an attempt that has not answered by its deadline counts
as failed. A put's deadline follows its size: its bytes at a floor of
5 Mbit/s, plus 60 s. A fixed 180 s once failed every attempt at a 334 MB
database behind a 15 Mbit/s uplink (#2536). An attempt past its deadline is
aborted, not just abandoned: a put goes as a presigned PUT through fetch
with the attempt's signal, because client.write takes none, and an
abandoned upload went on sending beside the tries after it (#13733).
Backup lanes
The platform's backup lanes (FORGE_BACKUP_LANES) and how each is judged: docs/features/backup-truth.md, "Platform backup lanes".
Progress
A snapshot uploads eight chunks at a time, and only those the store lacks. While one runs, progress (chunks and bytes sent) shows on Settings → Backups and as a line in the log every 500 objects or minute. The terminal has no backups view, so it has no readout.
When a snapshot fails
Its outcome is kept in snapshot/state.json under the data directory, so a
restart forgets nothing. A failure shows straight away as a card on the board
(snapshot-failed, pointing at Backups), in the terminal's dashboard among
the forge's needs, and as a red line on the Backups page with the reason and
since when. The forge tries again after ten minutes (SNAPSHOT_RETRY_MS, capped at the
next nightly), not the next night: a store that is down gets six tries an hour, each of five
attempts per put. The
next snapshot that lands, nightly or on demand, clears both.
A forge given nowhere to put a snapshot has not failed: nothing is kept, no card is raised, and nothing is retried sooner than the nightly. The Backups page says there is no destination.
The restore drill
A backup nobody has restored is a hope. Settings → Backups → Run a drill
(admins only; anyone else gets 403) restores the newest snapshot into a
scratch directory under the data directory, opens the database read-only,
counts repositories and issues, opens one sealed value with the restored key,
and removes the scratch whatever happened. The page waits until it finishes,
for up to 10 minutes; a step that hangs past that is red and names the step.
The restore reads several chunks at a time (RESTORE_IN_FLIGHT), each
written in its file's order: read one after another, a
snapshot of thousands of objects spent the drill's whole 10 minutes in the
restore step (#2539).
Repositories come back four clones at a time, not one after another (#2543).
One drill runs at a time: a second click, or a retry after a proxy timeout,
joins the drill already running rather than starting another restore. Every
run is a row in the drill table, and the last one shows as a line on the
Backups page: when, which snapshot, how long, the counts, and green or the
step that failed.
The drill runs on its own every Sunday 09:00 UTC, and a forge that was down
then catches up five minutes after boot. A red drill does not keep the
week's slot: it is owed again RED_RETRY_MS (three hours) after it ran,
and at a boot once it is BOOT_RED_FLOOR_MS (30 minutes) old, until one
passes (src/snapshot/due.ts). The floor keeps a crash loop with a broken
restore from running a restore on every boot. Kept for the
week, a drill red at 09:11 stood as a card long after the fix deployed at
10:18 (#2541).
Run a drill on the same page (admins only) restores the newest snapshot
into scratch, opens it, counts what is there and unseals a value; the page
waits for it, up to 10 minutes, and the last drill shows there as one line,
ending with the drill's time against its budget and each part's time, longest first ("6 min
12 s of its 14 min budget: clone 4 min, download 58 s, …"), or only the time against the
budget when no part was timed; drills from before this show none. The budget is 10 min
plus 1 s per MB of snapshot, at most 45 min (docs/RECOVERY.md).
The parts are download, decompress, clone, open, and start and migrate (the
whole side-boot of the restored forge, process start included). The same words,
slowest named, are in the drill's log line and in a red drill's card
("… (download about 8 min, decompress 40 s, …; download was the slowest)").
Each snapshot's manifest keeps
the five counts of its database; a restore that counts differently is red at
count and, like any red drill, one board line and one push. Snapshots from
before the counts were kept are not compared.
Failure domains, and what the owner is told
Failure domains (a card at the foot of /settings/backups; the old
/settings/resilience sends there, and /resilience is the THREATS.md page): one row per failure domain from
docs/DOMAINS.md — what breaks, the words the forge says, and the domain's
last kept drill (when, green or red with its reason, seconds until back) or
the sentence on why it is not drilled. In the terminal it is the Failure domains
command (ctrl+k; the Resilience command is the THREATS.md one), shown to an admin only and read through the same
listResilience action. Only the latest drill per domain is shown.
An admin presses d on Failure domains to run the restore drill, behind a confirm, with the same words as the Backups button and MCP restore_drill; it keeps the snapshot drill record only and leaves the domain drills as they are.
The Owner's needs command (admin only, ownerView) is the owner's one screen: the forge's needs as the board strip says them, where snapshots go (and any mirror) with the newest snapshot, and every domain with what the owner does.
A red drill, or none in eight days, is one line for the forge's admin on the
board's forge strip, in the terminal's dashboard (same forgeNeeds), and in
MCP health and asks; it names the failed step and the next green drill
clears it. An admin with a phone URL gets one push per cause (a new red
drill, or the silence), also when eight days pass with no event: the notifier
wakes every 10 minutes for it. The time the forge first owed a drill is kept
at boot in drill/first-seen; if the disk will not take it, a warning is
logged and a forge with no drill ever is not flagged.
Snapshots written to a directory on the forge's own volume, with no off-box
mirror set, are lost with that volume. That is one line for the forge's admin,
"Snapshots are kept on the forge's own volume and mirrored nowhere: losing the
volume loses them too.", on the board's forge strip (linking Settings →
Backups), in the terminal's dashboard (same forgeNeeds), in the asks, in
GET /api/v1/health and MCP health. Anyone else sees nothing. It clears when
the destination moves to another volume or a bucket, or a mirror is set. Every
one of those reads the vitals through forgeVitals in src/actions/health.ts.
Restore
The printable card, seven lines, is RESTORE.md; a test runs it.
# same S3_* as the forge runs under, or FORGE_SNAPSHOT_DIR
FORGE_SNAPSHOT_PASSPHRASE=… \
bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z /path/to/empty/data
With the passphrase the key is unwrapped into secret.key (mode 0600) and
sealed values open. A wrong passphrase refuses the restore before anything is
written. With none, the key is left out, repositories and the rest restore,
and the command lists every sealed value by name (NAME (app slug),
NAME (scope secret)) that must be set again; the restored forge makes a new
key on first boot.
It refuses a directory that is not empty, a snapshot it cannot find, one that is missing a chunk, and any chunk or file that is not the bytes its manifest names. Bundles become bare repositories with the forge's hooks installed; the mirror's memory of the bundle as an origin is removed. Then start the forge on that directory: it migrates the database forward if it has moved on, reconciles the index and redeploys what runs.
test/snapshot-incremental.test.ts proves the chunks: the second snapshot
puts only what changed; the first, the second and those left after a sweep
restore byte for byte; a missing chunk fails the verify; the sweep never
takes a chunk a kept manifest names. test/snapshot.test.ts is the proof
of the whole: a running forge with history, a
sealed variable and an app's SQLite is snapshotted, restored into an empty
directory with the passphrase and booted there as the same forge; restored
without it, the repositories come back and the sealed variable is reported by
name. It also checks that the raw key is in no file of the store.
The dump, a line per lane and the platform-free drill: backup-truth.md, _Leaving the platform_.