The forge's data is backed up encrypted to object storage on neither machine, and a restore is drilled #11835

The off-site backup lane now encrypts everything it writes. The restore drill reads from that lane, /system shows when the lane last succeeded, and the backup rows in FOOTING are no longer on pg-grand. One deploy step is still needed (below) before the lane works with pg-grand switched off.

Before this, the hourly mirror pass already copied each snapshot to a second bucket (B2) from inside the forge's process. But only the forge's key was encrypted: the database, the repository bundles and the object names, which include repository names, landed in plain text.

What changed

• Encrypted lane (src/snapshot/sealed_store.ts): every object the mirror writes is encrypted with AES-256-GCM. The key is derived from FORGE_SNAPSHOT_PASSPHRASE with scrypt, so it is in neither the repository nor the bucket. File paths inside a snapshot are replaced by hashes, and everything sits under sealed/, apart from anything copied before.

- resolveMirror encrypts when it gets the passphrase, and main.ts passes it. - The mirror's state records whether the lane is encrypted. Without a passphrase the lane copies unencrypted, as it did before, and /system flags that in red.

• Drill (drill.ts, service.ts, drills.ts):

- restore_drill, the weekly drill and the Backups button now restore from the off-site lane when the forge has one, after any copy in progress finishes. - The restored forge must now answer / as well as /api/v1/health. - Each drill row keeps the store it restored from, inside the existing phases JSON column, so no migration was needed. The MCP answer gains from and offSite.

• /system: a new "Off-site backup" section shows where the lane writes, whether it is encrypted, its last success, the newest snapshot it holds, and the last drill with its source. The data comes from src/snapshot/offsite_lane.ts.
• FOOTING:

- The rows for the platform's backup CronJobs and MinIO are replaced by two rows: the forge's own backup lane (vxrail) and the off-site bucket, on a new off-site machine that never counts as a risk. - "local backups" leaves the break order, so the host timers are now step 6. The CronJobs and MinIO move to ALSO_ON_PG_GRAND. - The table in docs/OPERATING.md was regenerated from footingTable().

• Fresh-machine restore: src/snapshot/restore.ts and scripts/restore-drill.ts take --offsite to read the encrypted lane from FORGE_MIRROR_S3_*, and refuse without the passphrase. docs/RESTORE.md has a new "From the off-site lane" section.

Tests changed, and why

• test/footing.test.ts: it asserted that the backup rows were on pg-grand, which is the fact this issue changes. Its "outside a cluster" check now reads "every pg-grand row is a risk", matching its own title.
• test/system-parts.test.ts: the new section is added to its list of headings.

Before switching pg-grand off

• Nightly snapshot location (deploy step): the encrypted copy is taken from the nightly snapshot. Set FORGE_SNAPSHOT_DIR on the forge's own volume so that snapshot lands on the forge's disk rather than the platform's MinIO; without that, the lane still depends on pg-grand. The FOOTING row and RESTORE.md both say this.
• Production configuration: the passphrase and FORGE_MIRROR_S3_* must be set there. I can't see production's configuration, so I haven't confirmed either.
• First upload: the first encrypted pass uploads up to 14 snapshots.
• Old plain copies: the snapshots already on B2 in plain text, under snapshots/, are left in place. Delete them by hand once a drill from the encrypted lane is green.

Not done

• The nightly snapshot is not encrypted when the main destination itself is a bucket.
• No terminal or MCP view of the lane was added beyond what restore_drill returns.
• The keep_plan tool wasn't available in this session, so the issue has no plan.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass, and design-lint is clean. scripts/fast.ts named 1,820 test files, too many to run here, so I ran these by hand and all passed:

• the snapshot, mirror, drill, restore, footing and system suites;
• the MCP drill suites and every test that touches the changed functions;
• the new test/offsite-lane.test.ts. It runs a real snapshot through the encrypted mirror and checks the bucket holds no plain text. Then it restores with the passphrase and runs a green drill from the lane (the boot step is stubbed).

Try: open /system as the owner and read the "Off-site backup" section. Then call the MCP restore_drill tool and check its from.

🤖 Generated with Claude Code

Closes #11805

The off-site backup lane now encrypts everything it writes. The restore drill reads from that lane, /system shows when the lane last succeeded, and the backup rows in FOOTING are no longer on pg-grand. One deploy step is still needed (below) before the lane works with pg-grand switched off.

Before this, the hourly mirror pass already copied each snapshot to a second bucket (B2) from inside the forge's process. But only the forge's key was encrypted: the database, the repository bundles and the object names, which include repository names, landed in plain text.

What changed

• Encrypted lane (src/snapshot/sealed_store.ts): every object the mirror writes is encrypted with AES-256-GCM. The key is derived from FORGE_SNAPSHOT_PASSPHRASE with scrypt, so it is in neither the repository nor the bucket. File paths inside a snapshot are replaced by hashes, and everything sits under sealed/, apart from anything copied before.

- resolveMirror encrypts when it gets the passphrase, and main.ts passes it. - The mirror's state records whether the lane is encrypted. Without a passphrase the lane copies unencrypted, as it did before, and /system flags that in red.

• Drill (drill.ts, service.ts, drills.ts):

- restore_drill, the weekly drill and the Backups button now restore from the off-site lane when the forge has one, after any copy in progress finishes. - The restored forge must now answer / as well as /api/v1/health. - Each drill row keeps the store it restored from, inside the existing phases JSON column, so no migration was needed. The MCP answer gains from and offSite.

• /system: a new "Off-site backup" section shows where the lane writes, whether it is encrypted, its last success, the newest snapshot it holds, and the last drill with its source. The data comes from src/snapshot/offsite_lane.ts.
• FOOTING:

- The rows for the platform's backup CronJobs and MinIO are replaced by two rows: the forge's own backup lane (vxrail) and the off-site bucket, on a new off-site machine that never counts as a risk. - "local backups" leaves the break order, so the host timers are now step 6. The CronJobs and MinIO move to ALSO_ON_PG_GRAND. - The table in docs/OPERATING.md was regenerated from footingTable().

• Fresh-machine restore: src/snapshot/restore.ts and scripts/restore-drill.ts take --offsite to read the encrypted lane from FORGE_MIRROR_S3_*, and refuse without the passphrase. docs/RESTORE.md has a new "From the off-site lane" section.

Tests changed, and why

• test/footing.test.ts: it asserted that the backup rows were on pg-grand, which is the fact this issue changes. Its "outside a cluster" check now reads "every pg-grand row is a risk", matching its own title.
• test/system-parts.test.ts: the new section is added to its list of headings.

Before switching pg-grand off

• Nightly snapshot location (deploy step): the encrypted copy is taken from the nightly snapshot. Set FORGE_SNAPSHOT_DIR on the forge's own volume so that snapshot lands on the forge's disk rather than the platform's MinIO; without that, the lane still depends on pg-grand. The FOOTING row and RESTORE.md both say this.
• Production configuration: the passphrase and FORGE_MIRROR_S3_* must be set there. I can't see production's configuration, so I haven't confirmed either.
• First upload: the first encrypted pass uploads up to 14 snapshots.
• Old plain copies: the snapshots already on B2 in plain text, under snapshots/, are left in place. Delete them by hand once a drill from the encrypted lane is green.

Not done

• The nightly snapshot is not encrypted when the main destination itself is a bucket.
• No terminal or MCP view of the lane was added beyond what restore_drill returns.
• The keep_plan tool wasn't available in this session, so the issue has no plan.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass, and design-lint is clean. scripts/fast.ts named 1,820 test files, too many to run here, so I ran these by hand and all passed:

• the snapshot, mirror, drill, restore, footing and system suites;
• the MCP drill suites and every test that touches the changed functions;
• the new test/offsite-lane.test.ts. It runs a real snapshot through the encrypted mirror and checks the bucket holds no plain text. Then it restores with the passphrase and runs a green drill from the lane (the boot step is stubbed).

Try: open /system as the owner and read the "Off-site backup" section. Then call the MCP restore_drill tool and check its from.

🤖 Generated with Claude Code

The forge's data is backed up off-site, sealed, and the backup can be restored and drilled. This push fixes the two failures that came from this pull: the /system first-view budget and the recovery doc's restore command.

What the pull does (#11805)

• Backups off pg-grand: the backup rows in FOOTING move to the forge's own backup run and an off-site bucket.
• Encrypted copies: the mirror pass seals every object it writes when FORGE_SNAPSHOT_PASSPHRASE is set (src/snapshot/sealed_store.ts).
• Restore from off-site: restore.ts --offsite and scripts/restore-drill.ts --offsite restore from the sealed bucket; without the passphrase they stop before writing anything.
• Drills record their source: the weekly drill, the Backups button and the MCP restore_drill use the off-site bucket when there is one. Each drill records which store it restored from.
• /system: a new "Off-site backup" section shows the bucket, when it last copied successfully, and the last drill.

This push

• /system section: it now shows its heading and one sentence. The detail rows sit behind a disclosure, the same way the "What the forge stands on" section does. The red badge appears only when an off-site bucket is set up but its copies are failing, late or unencrypted. A forge with no off-site bucket gets the sentence and no colour. The page is back inside its budget (65/65 blocks, 0 colours) with no change to test/attention.test.ts or its budget file.
• docs/RECOVERY.md: it shows restore.ts's usage line with [--offsite] and points to docs/RESTORE.md. I rewrapped the paragraph so the doc stays within its 125-line, two-printed-sides limit.
• Main merged in: origin/main merged with no conflicts.

Not fixed here: the two test/pull-quiet.test.ts byte-budget failures in shard 14 ("the pull page's check is a line…", "the pull list wears a dot per row…"). They pass here, and the pages measure the same on origin/main as on this branch (pull 32,097 B, list 25,478 B, against budgets of 32,160 and 25,530). They are only about 60 bytes under, so something in the check's environment probably pushes them over; that would hit main too.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit, bun scripts/design-lint.ts, and bun test on the recovery-doc, attention, offsite-lane, pull-quiet, footing and system-parts files all pass (46 tests).

To try it: open /system on the preview and expand "Off-site backup". bun run src/snapshot/restore.ts latest <empty-dir> --offsite, with the FORGE_MIRROR_S3_* variables set but no passphrase, refuses and writes nothing.

🤖 Generated with Claude Code

The off-site backup lane now writes only encrypted copies, and nothing at all when FORGE_SNAPSHOT_PASSPHRASE is unset. It takes a backup off both machines (pg-grand and vxrail) to a bucket, and the restore drill reads from that bucket. This push addresses all seven points in sentinel's review.

What the pull does (#11805)

• Encrypted lane: every object is encrypted with AES-256-GCM, and file paths inside a snapshot are replaced by hashes (src/snapshot/sealed_store.ts).
• Drills use the lane: the weekly drill, the Backups button and MCP restore_drill restore from the off-site bucket when the forge has one. Each drill records which store it restored from.
• Backups off pg-grand in FOOTING: the backup rows move to the forge's own lane and the off-site bucket.
• Restore from scratch: restore.ts --offsite and scripts/restore-drill.ts --offsite restore a fresh machine from the bucket.

This push (review)

1. No passphrase, no copy. Without FORGE_SNAPSHOT_PASSPHRASE, resolveMirror now returns refusingStore. Every mirror pass fails with NO_PASSPHRASE and nothing reaches the bucket. /system and the attention line name the missing variable. A test checks that a pass writes no file to the bucket.
2. Dead code. refusingStore is now used. drillSource is gone; service.ts uses currentMirror() ?? current() in its place.
3. Terminal screen. The palette's Off-site backup entry (src/tui/screens/offsite.ts) shows where the lane writes, whether it is encrypted, its last success and the last drill. MCP system {kind:"offsite"} returns the same data. The page, the screen and the MCP tool all read it through readOffsiteLane, and all three are for forge admins only.
4. Docs. The how-to is now in docs/features/offsite-lane.md, and RESTORE.md and RECOVERY.md link to it.
5. Salt per lane. The first pass writes an unencrypted header, sealed/lane.json, holding a random 16-byte salt and a check value. The key never goes in it. Scrypt now runs off the main thread, and a wrong passphrase is reported before any object is read. A restore refuses an empty bucket instead of starting a new lane there.
6. Drill wait has a limit. A drill waits for a mirror pass in progress for at most the drill's timeout, DRILL_TIMEOUT_MS. After that it drills the bucket as it stands. A test holds a pass that never ends.
7. answersRoot is tested against a stub server: 200 and 302 pass, 500 fails.

Test I corrected: in test/offsite-lane.test.ts, the new screen and MCP test I wrote had two mistakes.

• It used usernames that the forge reserves.
• It expected "(on this machine)", but the test before it drills from the off-site bucket, so "(off-site)" is correct.

Checks: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts pass. I ran the snapshot, mirror, drill, palette, TUI, system, footing, recovery-doc and both-doors test files (219 tests) and all pass. scripts/fast.ts couldn't narrow the run because the diff touches scripts/restore-drill.ts, so I picked those files myself.

To try it: on the preview, open /system and expand "Off-site backup", or open the palette's Off-site backup in the terminal. Then set FORGE_MIRROR_S3_* without a passphrase and run bun run src/snapshot/restore.ts latest <empty-dir> --offsite: it refuses and writes nothing.

🤖 Generated with Claude Code