Backups that cannot lie (#8742)

Platform backup lanes (#8811)

Every other place a backup lands — one per CronJob in the backup namespace — is a lane the platform declares in FORGE_BACKUP_LANES, a JSON list kept in the forge's configuration, never in this repository:

[{ "name": "backup/postgres", "endpoint": "https://s3.example.com", "bucket": "platform-backups",
   "prefix": "postgres/", "region": "us-east-1", "accessKey": "…", "secretKey": "…",
   "backsUp": "the platform's Postgres dump, nightly",
   "afterSpinDown": "ops reads it from the bucket with the restore runbook" }]

that can do nothing else.

mirror, and the verdict names each one's newest object, size and age.

most three pages of 1,000), walking only into the folder whose name sorts last, since backup names carry their date. A level wider than that is read in part: a fresh object found still proves the lane fresh; a stale one, or none, is red and says to name a narrower prefix.

prefix, or the newest object is over 26 h old. A lane the configuration names but that does not read whole is red with what is wrong, never with its values.

error the store wrote has its control bytes replaced with ? before it reaches a terminal or an MCP client.

reads it once the platform is gone, or retired) are optional strings of at most 200 characters, control bytes replaced. A lane missing either shows No line yet: say what it backs up and who reads it after spin-down beside its newest object: information, never red. A lane marked retired is listed, never read, and never turns the tile red.

Resilience dialog; home's backup tile reads the same verdict.

The probe outside the forge (#8806, part 1)

The in-process bucket check dies with the forge, so a dead forge looked like a quiet one. scripts/backup-probe.ts is a separate process (run it from cron or another host) that reads the forge and the bucket and goes red when either is silent.

S3_ENDPOINT=… S3_BUCKET=… S3_ACCESS_KEY=… S3_SECRET_KEY=… \
  bun scripts/backup-probe.ts --forge https://your.forge [--max-age-hours 36]
bun scripts/backup-probe.ts --forge http://127.0.0.1:3000 --dir /path/to/backups

Exit 0 when green; exit 1 and a RED line per failing read otherwise. Red is: /healthz unanswered or not 200 within 10 s; no store configured; the store unreadable or empty; or the newest manifest's finishedAt older than the limit (default 36 h: one nightly plus grace). Use read-only credentials. Logic lives in src/snapshot/probe.ts; tests in test/backup-probe.test.ts.

Resuming across a release (#8828)

A snapshot cut short has no manifest, so it is not a snapshot; but what it sent is not sent again. Every file is chunks under chunks/, named by the hash of their bytes (docs/SNAPSHOT.md, "A snapshot sends what changed"), and takeSnapshot begins by listing them: the catch-up after boot (takeIfDue) takes the night's snapshot anew, under a new name, and puts only the chunks the store lacks. There is no journal to write on a clock and none to trust: the store's own listing says what landed. The manifest goes last, after a fresh listing shows every chunk it names. pruneSnapshots removes a prefix with no manifest (one left by a snapshot from before chunks) that is older than the newest whole snapshot or more than 24 h old, and sweeps the chunks no manifest names when retention lets a snapshot go.

Not done: the snapshot still runs in the serving process (the database copy is in a worker). Tests: test/snapshot-incremental.test.ts.

Leaving the platform (#8949)

Nothing here turns anything off. Each step below waits for the owner present; the forge and the backup tile stay up and green throughout.

The Forgejo dump (forgejo-dump, suspended, 36 GB)

Recommendation, for the owner to confirm in the issue: keep one last copy, and let the migrate flow make it redundant for what it covers.

  1. Every Forgejo repository the forge should keep comes in through Migrate or

Bring (docs/ORIGINS.md); once in, the forge's snapshot carries its bundle.

  1. The dump is also the only copy of what Migrate does not read (issues, pull

history, users, LFS, packages, Actions secrets on Forgejo). So it is not deleted on the strength of step 1.

  1. One copy goes to B2 (a prefix of its own, outside the nightly's

retention). The B2 dump prefix is not a lane, so it is not in FORGE_BACKUP_LANES and the tile does not read it. A second copy goes to the owner's disk; its SHA-256 is written in this issue.

  1. Only after the owner has opened the copy (restored a repository from it) is

the CronJob and its volume retired.

Owner's answer: _pending_. Until it is written, the dump stays where it is.

A line per lane

Each lane's backsUp and afterSpinDown now live beside it in FORGE_BACKUP_LANES (the forge's own primary and mirror say theirs in src/snapshot/bucket.ts) and show next to its newest object in MCP backups → verdict.lanes, the Resilience dialog and /settings/backups. A lane with no reader after spin-down is marked retired there, so the tile does not go red for a lane nobody keeps.

Restore drill with the platform unreachable

On the box (scripts/box-install.sh, docs/EXTRACT.md), with the platform's network blocked (egress allowed to B2 only, e.g. a firewall rule on the box):

  1. Point S3_* at the B2 mirror credentials, not the platform bucket.
  2. With the B2 S3_* values and the passphrase exported, as the forge

user, run bun scripts/restore-drill.ts <empty-data-dir> (RESTORE.md, _The drill on the box_). Exit 0 is the pass condition: it restored, booted, answered / and the counts matched. Exit 1 is a failed drill, whatever the step it names.

  1. Check that the backup tile reads green once the check has run.
  2. Record the date, snapshot name and counts in this issue.

Not done here: the drill itself, the dump decision, and the lane rows. Each needs the box or the owner; this section is what they follow.

An app's own data has its own encrypted backup and restore proof: app-backup.md (#11701).

The drill is a job (#12592)

A restore drill is asked for and answers at once with an id (MCP restore_drill, action: start; the Backups page's button; in the terminal, d on the Failure domains screen and r on the Resilience screen, which start the same job and follow it, saying "A drill is running: <phase>, X of Y." until the verdict). One runs at a time; a second ask says one is running and where. It runs in its own process (src/snapshot/drill_child.ts, started by drill_job.ts with --smol), so the restore is never held by the serving process; that process's resident memory is read every 3 s and it is killed, the drill red, above its limit: 60% of the machine's memory, between 3 and 16 GiB, or FORGE_DRILL_MEMORY_MIB (at least 512). Every drill run this way keeps its cost in its verdict (#13253): the process's elapsed time (spawn to exit) and peak resident memory (the child reads process.memoryUsage.rss() once per progress report and getrusage at the end; the parent takes the larger of those, its own 3 s readings and the reaped child's getrusage, so a peak between readings or at a kill counts), and the limit it ran under. The Backups page shows them as rows (Took, Peak memory with its share of the limit, Limit with how it is chosen, said once); MCP restore_drill action: status (cost, elapsedMs, peakMiB, limitMiB) and the terminal's drill verdict (d on Failure domains, r on Resilience) say one line: "took 1 min 2 s, peak 412 MiB of its 3 GiB limit (14%)" (src/snapshot/drill_cost.ts). Each real drill so proves how near the limit it came, and the number is tuned from those lines. A child killed over its memory or time, or one that dies without a result, is a red drill at restore whose reason names the cause (memory: the drill process held …, time: …, the drill process exited with …) and keeps its cost; it never ends as a log line alone. The restore itself already streams chunks to disk with at most 24 reads in flight; what the serving process held on 2026-11-14 is read memory the collector had not yet given back, so the child collects after each progress report. Where it is (phase, done of total, bytes, memory) is on the Backups page, in MCP action: status, and in the log (restore drill working, every 15 s). action: status answers the verdict once it ends.

A marker (drill/running.json) is left while a drill runs. A boot that finds it records a red drill "cut by a restart", removes every drill/restore-* directory (none runs at boot), and drills again. The status line says the proof's age ("restore drill 2026-11-08 passed (6 d ago)") and goes amber when the newest snapshot is more than 3 days newer than the drill. A snapshot whose format differs from the last drilled one is drilled by the forge on its own.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs