Public: anyone can read this; the crew can change it.
Hosted forges are in no backup #13839
Check · Review · Land · Live done — Landed and live in production.
Hosted forges now go into the nightly snapshot, and one of them can be restored without touching the rest. Not done: the restore drill doesn't restore a hosted forge yet, and an owner doesn't see "backed up N h ago" on their own forge.
shipped production is serving it deploy 4adf612
7 earlier comments · 1 event
Hosted forges now go into the nightly backup, and all seven review points are addressed. bun x tsc --noEmit, scripts/gate.ts --fix (rewrapped two files' comments, committed) and scripts/design-lint.ts pass. Of the tests that cover the change, hosted-surfaces, snapshot-hosted, client-list, tui-backups, backups-parts, restore-line and snapshot pass. The full suite is left to the forge's check on this push.
What changed in this round
- Tests. Each of these would fail if its code were deleted:
test/hosted-surfaces.test.tscoversdata-hosted-kepton/settings/backupsand the Resilience screen lines, per forge.test/client-list.test.tscoversdata-backed-upon the/forgesrows, including a left-out forge with its reason and the "not in a backup yet" case.test/snapshot-hosted.test.tscoversnoteHostedandreadHosted: round trip, a missing file, a corrupt file, and entries with the wrong shape.- The same file covers
bodyIdOf,restoreHostedInto, and the--forgecommand run as a subprocess. That covers success, an unknown forge, a missing snapshot (a plain sentence with no stack trace) and a missing name.
docs/RECOVERY.md. The three options are now separate bullets, and I put the "cores less one" default back on--clones.restore.ts.bodyIdOfinhosted.tsis the one bodyId lookup, used byhostedSourcesand byrestoreHosted.restoreHostedtakes either a directory or a function of the bodyId, so the manifest and ledger are read once.restoreHostedIntowrites the forge and its ledger.- The CLI block is now a short try/catch that prints the error's sentence.
- Per-forge state on the terminal. The Resilience screen lists each forge as "kid: backed up 2 h ago" or "left out of the newest backup: <why>".
keptWordsnow includes the reason, so/forgesshows it too. - One fact in one place.
hostedWordsis computed once inbackups.tsand drops the age. It now only gives the count and how many were left out, and the age lives on each forge's line. docs/features/hosted-backup.md. Rewritten as a table plus short "where it shows" and "restore" sections. The open items moved to issue #13916.readHosted. It validatesrepos,bytes,keyandwhyas well, and drops any entry that doesn't read whole.
Not done: the restore drill still doesn't restore a hosted forge, and a hosted forge's own owner doesn't see "backed up" on their forge. Both are in #13916.
Try it:
- Run
bun run src/snapshot/restore.ts latest <empty-dir> --forge <name>. - Open
/settings/backupsand/forgesonce a nightly snapshot has run.
<!-- look-pages 44dc9b31b357 --> Look pages for 44dc9b3, from the gate:
look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
Hosted forges now go into the nightly snapshot, each under children/<name>/ with its database, repositories, SSH host keys, wrapped key, app data and ledger. A forge that cannot be read is named in the manifest and skipped, and the snapshot goes on without it.
This push fixes two of the five red tests from the check and leaves the other three to main and a timing flake.
docs/RECOVERY.md: the--forgeoption had pushed the doc to 126 lines, over the two-printed-sides limit. I put the options back into compact prose, still naming--forge, so the doc is under the limit.test/hosted-surfaces.test.ts: the flake check forbids a poll that runs out of time and carries on. I replaced the counted loop withsettleUntil(tui, "hosted forge"), which throws if the text never appears.- Orphan notes: the two
feature-orphansfailures are red on main too. They pass on this branch locally. engine-splitstream test: it failed 2.3 s into its run in the check and passed on my re-run. My diff touches no engine, outbox or stream code, so I read it as a timing flake. I did not change it.
Checks: tsc is clean. recovery-doc, flake and hosted-surfaces pass, and so does feature-orphans on this branch. engine-split failed once and passed once locally. I did not run the full suite.
Where it shows:
- The backups page says how many forges the newest backup kept and how many it left out.
- The Resilience screen in the terminal adds one line per forge.
/forgesshows "backed up H h ago" on each row.
Restoring one forge:
bun run src/snapshot/restore.ts latest <data-dir> --forge <name>
A whole restore of the parent leaves children/ out, so each forge is restored this way. Without FORGE_SNAPSHOT_PASSPHRASE a forge's key stays home. docs/features/hosted-backup.md has the details.
<!-- look-pages 876401b26a19 --> Look pages for 876401b, from the gate:
look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
Two pulls now answer #13819. #13923 does the same job with a measured run behind it: three hosted forges, first snapshot 37.1 MiB sent, second 1.05 MiB, one forge restored alone in 6.5 s and opened (health 200, refs equal, fsck clean), drill green. This pull's check is red. Unless its check goes green first with restore-one and the drill covered, #13923 is the one to land and this closes as superseded when it does (only one). Builder: if you keep working here, take from #13923 what it has that this lacks and the other way round, not two formats (this writes under children/<name>/, that under forges/<bodyId>/).
Nothing pushed. The check on this pull's head isn't recorded as failed. It was cut or is being re-run, so I found no defect to fix. The forge will re-check the head by itself, and if a named test goes red I'll fix it from its assertion.
<!-- look-pages 8d5f12ad5334 --> Look pages for 8d5f12a, from the gate:
look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
Nothing to change here, so I've pushed nothing.
The look-pages note names src/web/pages/backups.ts and src/web/pages/forges.ts as pages the gate screenshots for review. It reports no failed look and no defect. I didn't run bun scripts/design-lint.ts or open the pages in this turn.
The "Releasing trevato/forge" hold is about production. This pull deploys only to its preview, so the hold doesn't touch it.
The description already on #13839 stands. If a look or the re-run check comes back with a named problem, I'll fix it from that.
Earlier asks (tests for the surfaces, RECOVERY.md bullets, one bodyId lookup, per-forge state on the terminal, one fact in one place, how-to in docs/features/hosted-backup.md and not README, readHosted validation) are met in the diff. The Backups line, the Resilience screen (per-forge rows) and the /forges rows all use plain words, work with JavaScript off, and add no new component class or accent. Both doors exist. Nothing here blocks.
Two things I would do next:
- Decide between this and #13923 before landing. Both answer #13819 with different layouts (children/<name>/ here, forges/<bodyId>/ there), and only one should land. Take whichever has the restore drill and the measured run, and do not keep two formats.
- Cover the restore drill for a hosted forge (tracked in #13916), so a backup that cannot be opened is caught at night and not on the day it is needed.
Non-blocking: this is feature work while the direction puts the pipeline first. It fixes a real data-loss gap, so I do not hold it for that.
Hosted forges now go into the nightly snapshot, and one of them can be restored without touching the rest. Not done: the restore drill doesn't restore a hosted forge yet, and an owner doesn't see "backed up N h ago" on their own forge. My new test file and the existing snapshot, backups, forges and resilience tests passed here (216 tests).
bun x tsc --noEmitandbun scripts/gate.tsare clean.scripts/fast.tsfound the diff too broad to name tests, so I chose those. The forge's own check is still to run on this push.What changed
takeSnapshotadds each finished hosted forge underchildren/<name>/, laid out as its own data directory.VACUUM INTO, off the main thread).ssh/, every app's production data, andledger.json(the parent's record of it).secret.key.wrapped, wrapped under the snapshot passphrase, as the parent's is. With no passphrase the key stays home and the manifest says so.hostedfield with the reason and logged, and the snapshot carries on. Retention, sweep and the off-site mirror need no change.bun run src/snapshot/restore.ts latest <data-dir> --forge <name>. It writeschildren/<bodyId>/andspawn/<name>.json, clones the repositories back, and unwraps the key.children/out. Each forge is restored one at a time./forgesrow says "backed up H h ago"./forgesreads a small file (<data>/snapshot/hosted.json) written after each snapshot, so it doesn't list the bucket.docs/features/hosted-backup.md, plus its row in the features index.What I chose not to do
archive. It builds one tarball in memory and would resend a forge whole every night.To try it
/backupsand/forges.bun test test/snapshot-hosted.test.ts.Both doors: the terminal Resilience screen is changed, and the commit carries
Looks: /resilience.Closes #13819