Backups that cannot lie (#8742)
- Read from the bucket.
src/snapshot/bucket.tslists the primary and mirror stores every 5 minutes, in the serving process (no CronJob to die). Home's backup tile, vitals (onebackupcard) and the MCPbackupstool'sverdictall callbackupVerdict. - Three states.
green,red, andunknown: nothing read yet since boot, or no restore drill yet. Unknown is never a calm tile and never an alarm; only red raises the onebackupcard. A store error is scrubbed of keys and presigned queries before it is kept. - Red when: the newest snapshot is over 26 h old; a lane cannot be read or holds nothing; the newest restore drill failed, did not run, or brought back no rows; or the check itself has not run for 20 min (dead man's switch). The reason says when it last ran:
The backup check has not run for 25 min (last ran 2026-11-02T08:00:00Z); it is the alarm, and it is down. - The run record. Every run of the check appends one JSON line (
at, and each lane'sokandnewestAt) tobackup-checks.jsonlunder the data directory. The count is kept in memory; once the file holds 400 runs it is cut back to the last 200. A disk that refuses the write never stops the check. The verdict carriesrunsandcheckLine(The backup check last ran <ISO> (3 min ago); 42 runs on record.), shown in the Resilience dialog, the home tile's title and thebackupcard, so a person can see the check is alive. - Catch-up. A boot with the newest snapshot older than the last 08:00 UTC slot takes one two minutes later (
snapshotDue). A snapshot cut short by a release has no manifest, so it counts as lost. - In the terminal. The Resilience dialog (palette: Resilience, admins) opens with the same verdict, each lane's newest, and every reason. On the web it is home's backup tile and
/settings/backups; for an agent,verdictin the MCPbackupstool. - Off the serving thread.
VACUUM INTOruns in a worker (vacuum.ts). - Not done, tracked: snapshots outside the serving process (#8828); resuming across a release is below, and so is the off-process probe.
Platform backup lanes (#8811)
Every other place a backup lands — one per CronJob in the backup namespace —
is a lane the platform declares in FORGE_BACKUP_LANES, a JSON list kept in
the forge's configuration, never in this repository:
[{ "name": "backup/postgres", "endpoint": "https://s3.example.com", "bucket": "platform-backups",
"prefix": "postgres/", "region": "us-east-1", "accessKey": "…", "secretKey": "…",
"backsUp": "the platform's Postgres dump, nightly",
"afterSpinDown": "ops reads it from the bucket with the restore runbook" }]
- List-only keys.
src/snapshot/lanes.tsonly lists; give each lane keys
that can do nothing else.
- Read on the watch's clock. Each pass reads every lane beside
primaryand
mirror, and the verdict names each one's newest object, size and age.
- Bounded, never a full listing. One delimited listing per folder level (at
most three pages of 1,000), walking only into the folder whose name sorts last, since backup names carry their date. A level wider than that is read in part: a fresh object found still proves the lane fresh; a stale one, or none, is red and says to name a narrower prefix.
- Red when: the store does not answer within 10 s, nothing is under the
prefix, or the newest object is over 26 h old. A lane the configuration names but that does not read whole is red with what is wrong, never with its values.
- Nothing unsafe printed. Errors are scrubbed of keys, and every key or
error the store wrote has its control bytes replaced with ? before it
reaches a terminal or an MCP client.
- A line per lane (#9067).
backsUp(one sentence) andafterSpinDown(who
reads it once the platform is gone, or retired) are optional strings of at
most 200 characters, control bytes replaced. A lane missing either shows
No line yet: say what it backs up and who reads it after spin-down beside
its newest object: information, never red. A lane marked retired is listed,
never read, and never turns the tile red.
- Where to see it. MCP
backups→verdict.lanes; the terminal's
Resilience dialog; home's backup tile reads the same verdict.
The probe outside the forge (#8806, part 1)
The in-process bucket check dies with the forge, so a dead forge looked like
a quiet one. scripts/backup-probe.ts is a separate process (run it from cron
or another host) that reads the forge and the bucket and goes red when either
is silent.
S3_ENDPOINT=… S3_BUCKET=… S3_ACCESS_KEY=… S3_SECRET_KEY=… \
bun scripts/backup-probe.ts --forge https://your.forge [--max-age-hours 36]
bun scripts/backup-probe.ts --forge http://127.0.0.1:3000 --dir /path/to/backups
Exit 0 when green; exit 1 and a RED line per failing read otherwise. Red is:
/healthz unanswered or not 200 within 10 s; no store configured; the store
unreadable or empty; or the newest manifest's finishedAt older than the
limit (default 36 h: one nightly plus grace). Use read-only credentials.
Logic lives in src/snapshot/probe.ts; tests in test/backup-probe.test.ts.
Resuming across a release (#8828)
A snapshot cut short has no manifest, so it is not a snapshot; but what it
sent is not sent again. Every file is chunks under chunks/, named by the
hash of their bytes (docs/SNAPSHOT.md, "A snapshot sends what changed"),
and takeSnapshot begins by listing them: the catch-up after boot
(takeIfDue) takes the night's snapshot anew, under a new name, and puts
only the chunks the store lacks. There is no journal to write on a clock
and none to trust: the store's own listing says what landed. The manifest
goes last, after a fresh listing shows every chunk it names. pruneSnapshots
removes a prefix with no manifest (one left by a snapshot from before chunks)
that is older than the newest whole snapshot or more than 24 h old, and
sweeps the chunks no manifest names when retention lets a snapshot go.
Not done: the snapshot still runs in the serving process (the database copy is
in a worker). Tests: test/snapshot-incremental.test.ts.
Leaving the platform (#8949)
Nothing here turns anything off. Each step below waits for the owner present; the forge and the backup tile stay up and green throughout.
The Forgejo dump (forgejo-dump, suspended, 36 GB)
Recommendation, for the owner to confirm in the issue: keep one last copy, and let the migrate flow make it redundant for what it covers.
- Every Forgejo repository the forge should keep comes in through Migrate or
Bring (docs/ORIGINS.md); once in, the forge's snapshot carries its bundle.
- The dump is also the only copy of what Migrate does not read (issues, pull
history, users, LFS, packages, Actions secrets on Forgejo). So it is not deleted on the strength of step 1.
- One copy goes to B2 (a prefix of its own, outside the nightly's
retention). The B2 dump prefix is not a lane, so it is not in
FORGE_BACKUP_LANES and the tile does not read it. A second copy goes
to the owner's disk; its SHA-256 is written in this issue.
- Only after the owner has opened the copy (restored a repository from it) is
the CronJob and its volume retired.
Owner's answer: _pending_. Until it is written, the dump stays where it is.
A line per lane
Each lane's backsUp and afterSpinDown now live beside it in
FORGE_BACKUP_LANES (the forge's own primary and mirror say theirs in
src/snapshot/bucket.ts) and show next to its newest object in MCP backups →
verdict.lanes, the Resilience dialog and /settings/backups. A lane with no
reader after spin-down is marked retired there, so the tile does not go red
for a lane nobody keeps.
Restore drill with the platform unreachable
On the box (scripts/box-install.sh, docs/EXTRACT.md), with the platform's
network blocked (egress allowed to B2 only, e.g. a firewall rule on the box):
- Point
S3_*at the B2 mirror credentials, not the platform bucket. - With the B2
S3_*values and the passphrase exported, as theforge
user, run bun scripts/restore-drill.ts <empty-data-dir>
(RESTORE.md, _The drill on the box_). Exit 0 is the pass
condition: it restored, booted, answered / and the counts matched. Exit
1 is a failed drill, whatever the step it names.
- Check that the backup tile reads green once the check has run.
- Record the date, snapshot name and counts in this issue.
Not done here: the drill itself, the dump decision, and the lane rows. Each needs the box or the owner; this section is what they follow.
An app's own data has its own encrypted backup and restore proof: app-backup.md (#11701).
The drill is a job (#12592)
A restore drill is asked for and answers at once with an id (MCP restore_drill, action: start; the Backups page's button; in the terminal, d on the Failure domains screen and r on the Resilience screen, which start the same job and follow it, saying "A drill is running: <phase>, X of Y." until the verdict). One runs at a time; a second ask says one is running and where. It runs in its own process (src/snapshot/drill_child.ts, started by drill_job.ts with --smol), so the restore is never held by the serving process; that process's resident memory is read every 3 s and it is killed, the drill red, above its limit: 60% of the machine's memory, between 3 and 16 GiB, or FORGE_DRILL_MEMORY_MIB (at least 512). Every drill run this way keeps its cost in its verdict (#13253): the process's elapsed time (spawn to exit) and peak resident memory (the child reads process.memoryUsage.rss() once per progress report and getrusage at the end; the parent takes the larger of those, its own 3 s readings and the reaped child's getrusage, so a peak between readings or at a kill counts), and the limit it ran under. The Backups page shows them as rows (Took, Peak memory with its share of the limit, Limit with how it is chosen, said once); MCP restore_drill action: status (cost, elapsedMs, peakMiB, limitMiB) and the terminal's drill verdict (d on Failure domains, r on Resilience) say one line: "took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%)" (src/snapshot/drill_cost.ts). Each real drill so proves how near the limit it came, and the number is tuned from those lines. A child killed over its memory or time, or one that dies without a result, is a red drill at restore whose reason names the cause (memory: the drill process held …, time: …, the drill process exited with …) and keeps its cost; it never ends as a log line alone. The restore itself already streams chunks to disk with at most 24 reads in flight; what the serving process held on 2026-11-14 is read memory the collector had not yet given back, so the child collects after each progress report. Where it is (phase, done of total, bytes, memory) is on the Backups page, in MCP action: status, and in the log (restore drill working, every 15 s). action: status answers the verdict once it ends.
A marker (drill/running.json) is left while a drill runs. A boot that finds it records a red drill "cut by a restart", removes every drill/restore-* directory (none runs at boot), and drills again. The status line says the proof's age ("restore drill 2026-11-08 passed (6 d ago)") and goes amber when the newest snapshot is more than 3 days newer than the drill. A snapshot whose format differs from the last drilled one is drilled by the forge on its own.