Public: anyone can read this; the crew can change it.

Hosted forges are in no backup #13839

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

Hosted forges now go into the nightly snapshot, and one of them can be restored without touching the rest. Not done: the restore drill doesn't restore a hosted forge yet, and an owner doesn't see "backed up N h ago" on their own forge.

Notes

shipped production is serving it deploy 4adf612

opened this

Hosted forges now go into the nightly snapshot, and one of them can be restored without touching the rest. Not done: the restore drill doesn't restore a hosted forge yet, and an owner doesn't see "backed up N h ago" on their own forge. My new test file and the existing snapshot, backups, forges and resilience tests passed here (216 tests). bun x tsc --noEmit and bun scripts/gate.ts are clean. scripts/fast.ts found the diff too broad to name tests, so I chose those. The forge's own check is still to run on this push.

What changed

  • What gets kept: takeSnapshot adds each finished hosted forge under children/<name>/, laid out as its own data directory.
    • Database: a consistent copy (VACUUM INTO, off the main thread).
    • Repositories: one bundle per repository.
    • Other state: ssh/, every app's production data, and ledger.json (the parent's record of it).
    • Key: secret.key.wrapped, wrapped under the snapshot passphrase, as the parent's is. With no passphrase the key stays home and the manifest says so.
  • Slow uplink (#13743): the files go through the same chunk path as the rest of the snapshot, so only changed chunks are sent and puts get the same retries. A test shows a second snapshot of an unchanged forge sending under 3 chunks.
  • A forge that can't be read: it is listed in the manifest's new hosted field with the reason and logged, and the snapshot carries on. Retention, sweep and the off-site mirror need no change.
  • Restore one forge:
    • Command: bun run src/snapshot/restore.ts latest <data-dir> --forge <name>. It writes children/<bodyId>/ and spawn/<name>.json, clones the repositories back, and unwraps the key.
    • Whole-parent restore: it now leaves children/ out. Each forge is restored one at a time.
    • Test: restores one forge alone and boots it, with its history and key.
  • Where the news shows: the Backups page says "N hosted forges, newest copy H h ago". The Resilience screen in the terminal says the same. Each /forges row says "backed up H h ago". /forges reads a small file (<data>/snapshot/hosted.json) written after each snapshot, so it doesn't list the bucket.
  • Docs: docs/features/hosted-backup.md, plus its row in the features index.

What I chose not to do

  • Restore drill: the nightly drill doesn't yet restore a hosted forge and open it. It is the next piece.
  • Owner's view: a hosted forge's own owner doesn't see "backed up N h ago" on their forge, because that needs a signed call from the child to the parent.
  • Archive reuse: I didn't reuse archive. It builds one tarball in memory and would resend a forge whole every night.
  • Snapshot time: the first snapshot after deploy will take a while. It sends 7.3 GiB at about 3.5 Mbit/s, which is roughly 4.8 hours; later ones send only what changed.

To try it

  • Snapshot: after the next snapshot, open /backups and /forges.
  • Restore: run the restore command above into an empty directory.
  • Tests: bun test test/snapshot-hosted.test.ts.

Both doors: the terminal Resilience screen is changed, and the commit carries Looks: /resilience.

Closes #13819

14 earlier comments · 2 events
commented

Hosted forges now go into the nightly snapshot, and one of them can be restored without touching the rest. Not done: the restore drill doesn't restore a hosted forge yet, and an owner doesn't see "backed up N h ago" on their own forge. My new test file and the existing snapshot, backups, forges and resilience tests passed here (216 tests). bun x tsc --noEmit and bun scripts/gate.ts are clean. scripts/fast.ts found the diff too broad to name tests, so I chose those. The forge's own check is still to run on this push.

What changed

  • What gets kept: takeSnapshot adds each finished hosted forge under children/<name>/, laid out as its own data directory.
    • Database: a consistent copy (VACUUM INTO, off the main thread).
    • Repositories: one bundle per repository.
    • Other state: ssh/, every app's production data, and ledger.json (the parent's record of it).
    • Key: secret.key.wrapped, wrapped under the snapshot passphrase, as the parent's is. With no passphrase the key stays home and the manifest says so.
  • Slow uplink (#13743): the files go through the same chunk path as the rest of the snapshot, so only changed chunks are sent and puts get the same retries. A test shows a second snapshot of an unchanged forge sending under 3 chunks.
  • A forge that can't be read: it is listed in the manifest's new hosted field with the reason and logged, and the snapshot carries on. Retention, sweep and the off-site mirror need no change.
  • Restore one forge:
    • Command: bun run src/snapshot/restore.ts latest <data-dir> --forge <name>. It writes children/<bodyId>/ and spawn/<name>.json, clones the repositories back, and unwraps the key.
    • Whole-parent restore: it now leaves children/ out. Each forge is restored one at a time.
    • Test: restores one forge alone and boots it, with its history and key.
  • Where the news shows: the Backups page says "N hosted forges, newest copy H h ago". The Resilience screen in the terminal says the same. Each /forges row says "backed up H h ago". /forges reads a small file (<data>/snapshot/hosted.json) written after each snapshot, so it doesn't list the bucket.
  • Docs: docs/features/hosted-backup.md, plus its row in the features index.

What I chose not to do

  • Restore drill: the nightly drill doesn't yet restore a hosted forge and open it. It is the next piece.
  • Owner's view: a hosted forge's own owner doesn't see "backed up N h ago" on their forge, because that needs a signed call from the child to the parent.
  • Archive reuse: I didn't reuse archive. It builds one tarball in memory and would resend a forge whole every night.
  • Snapshot time: the first snapshot after deploy will take a while. It sends 7.3 GiB at about 3.5 Mbit/s, which is roughly 4.8 hours; later ones send only what changed.

To try it

  • Snapshot: after the next snapshot, open /backups and /forges.
  • Restore: run the restore command above into an empty directory.
  • Tests: bun test test/snapshot-hosted.test.ts.

Both doors: the terminal Resilience screen is changed, and the commit carries Looks: /resilience.

commented

<!-- look-pages 0102495573a9 --> Look pages for 0102495, from the gate:

  • look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
commented

I fixed two of the three failing tests and pushed. The third passes when I run it alone here, so I left it alone.

Fixed

  • test/recovery-doc.test.ts: docs/RECOVERY.md still showed the old restore usage line. It now matches what restore.ts prints, including [--forge NAME], and has one sentence saying --forge NAME restores a single hosted forge from the snapshot.
  • test/worker-database-opens.test.ts: snapshot.ts now opens a second database, a read-only handle on each hosted forge's copy, for its repository and app counts. I raised its entry in the audit list from 1 to 2 opens and said why in the entry.

Not fixed

  • "the screen shows the share, the budgets that refused and the agent refused" failed in the check but passes when I run its file alone. I didn't find a cause and changed nothing for it. If it fails again on this push, it's a real problem.

tsc --noEmit and both fixed tests pass here. gate.ts --fix printed only a note that this pull touches look pages, and I didn't act on it.

commented

The look-pages note is information, not a failure, so I changed nothing and pushed nothing.

The gate printed look pages (named). That means it took /resilience, the page my commit names with Looks: /resilience, as the page to picture. It does not mean the gate wants more.

This pull changes src/web/pages/backups.ts and src/web/pages/forges.ts. Neither /backups nor /forges is in app.looks in forge.json, so I can't name them as pages to picture. /resilience is the closest entry in that list. I haven't checked whether it renders the new backed-up wording.

I haven't added a look entry for /backups or /forges, because that would mean extending app.looks and looksAsOwner, a change this issue doesn't call for. If you want those pages pictured, say so and I'll add them.

commented

I merged main into the branch and pushed it. The merge had one conflict, in src/web/pages/forges.ts, and I resolved it by keeping both sides.

  • Conflict: main added the XLARGE_SAYS constant above forgeRow, and my change added the kept?: string parameter to the same function. I kept the constant and the parameter.
  • Other files: snapshot.ts and the resilience screen merged without conflict.
  • Checks: tsc --noEmit is clean. The gate printed only its look-pages note. I ran 10 test files covering the hosted snapshot, the forges and resilience pages, the recovery doc, the Worker database audit and the TUI resilience screen. All 59 tests pass, with 2 skipped.
  • Not run: scripts/fast.ts couldn't narrow the run, since it named 2424 of 3573 files. I ran the covering files above and left the full suite to the forge's check on this push.
commented

<!-- look-pages b1d4383e31d1 --> Look pages for b1d4383, from the gate:

  • look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
commented

I've changed nothing and pushed nothing for this. It's the same look-pages note as before, now on commit b1d4383, and it's informational, not a failure.

The gate printed look pages (named) because my commit has a Looks: /resilience line. /resilience is the only page in app.looks that is close to what this pull changes. The pull changes src/web/pages/backups.ts and src/web/pages/forges.ts, and neither /backups nor /forges is listed in app.looks or looksAsOwner in forge.json. I haven't checked whether /resilience shows the new backed-up wording.

Pictured pages are listed in forge.json, so adding /backups and /forges there is the way to get them pictured. That would widen this pull beyond hosted-forge backup. If you want it, say so and I'll add them.

Changes requested — before the latest push

Hosted forges in the nightly backup is the right piece of work, and the snapshot and restore halves are tested well. I did not exercise the preview beyond reading the diff. Changes needed in one round:

  1. (First) Tests: the web and terminal surfaces and the restore CLI have no test. Nothing renders data-hosted-kept on /backups or data-backed-up on /forges. Nothing covers the Resilience screen line, noteHosted/readHosted (including a missing or corrupt hosted.json), or the --forge CLI path in restore.ts. Each would pass unchanged if the code were deleted. Add tests that fail without them.
  2. docs/RECOVERY.md: the --forge NAME sentence is glued onto the end of the --clones N line, which breaks that sentence. Give it its own line or bullet.
  3. restore.ts CLI --forge block: it re-parses the manifest and the ledger by hand, duplicating what restoreHosted does. A missing manifest becomes JSON.parse of an empty buffer, which throws a stack trace instead of a plain sentence. Move the bodyId lookup into one function (hosted.ts) that both paths use and say plainly when there is no manifest. That block is also a large inline addition to import.meta.main.
  4. Both doors: the terminal only gets a read-only line; the page and terminal show the same words, but there is no terminal path to say which forge is out of the newest backup. Show per-forge state (kept, left out and why) on the Resilience screen, or say in the docs why it is not needed.
  5. backups.ts:361: hostedWords(...) is computed twice in one line. Compute it once and render the <p> only when it is defined. Principle 1 (nothing says itself twice): the Backups line and the /forges row both say the same age. Keep one place per fact.
  6. docs/features/hosted-backup.md: a single dense paragraph, and it ends with a "Not done" list that reads as left-over work. Break it into short rows, and put the open items in issues instead of the how-to.
  7. hosted.ts readHosted: as HostedEntry[] after a name/at check trusts the rest of the file. Validate why, key and repos too, or type them as optional.
commented

Hosted forges now go into the nightly backup, and all seven review points are addressed. bun x tsc --noEmit, scripts/gate.ts --fix (rewrapped two files' comments, committed) and scripts/design-lint.ts pass. Of the tests that cover the change, hosted-surfaces, snapshot-hosted, client-list, tui-backups, backups-parts, restore-line and snapshot pass. The full suite is left to the forge's check on this push.

What changed in this round

  1. Tests. Each of these would fail if its code were deleted:
    • test/hosted-surfaces.test.ts covers data-hosted-kept on /settings/backups and the Resilience screen lines, per forge.
    • test/client-list.test.ts covers data-backed-up on the /forges rows, including a left-out forge with its reason and the "not in a backup yet" case.
    • test/snapshot-hosted.test.ts covers noteHosted and readHosted: round trip, a missing file, a corrupt file, and entries with the wrong shape.
    • The same file covers bodyIdOf, restoreHostedInto, and the --forge command run as a subprocess. That covers success, an unknown forge, a missing snapshot (a plain sentence with no stack trace) and a missing name.
  2. docs/RECOVERY.md. The three options are now separate bullets, and I put the "cores less one" default back on --clones.
  3. restore.ts.
    • bodyIdOf in hosted.ts is the one bodyId lookup, used by hostedSources and by restoreHosted.
    • restoreHosted takes either a directory or a function of the bodyId, so the manifest and ledger are read once.
    • restoreHostedInto writes the forge and its ledger.
    • The CLI block is now a short try/catch that prints the error's sentence.
  4. Per-forge state on the terminal. The Resilience screen lists each forge as "kid: backed up 2 h ago" or "left out of the newest backup: <why>". keptWords now includes the reason, so /forges shows it too.
  5. One fact in one place. hostedWords is computed once in backups.ts and drops the age. It now only gives the count and how many were left out, and the age lives on each forge's line.
  6. docs/features/hosted-backup.md. Rewritten as a table plus short "where it shows" and "restore" sections. The open items moved to issue #13916.
  7. readHosted. It validates repos, bytes, key and why as well, and drops any entry that doesn't read whole.

Not done: the restore drill still doesn't restore a hosted forge, and a hosted forge's own owner doesn't see "backed up" on their forge. Both are in #13916.

Try it:

  • Run bun run src/snapshot/restore.ts latest <empty-dir> --forge <name>.
  • Open /settings/backups and /forges once a nightly snapshot has run.
commented

<!-- look-pages 44dc9b31b357 --> Look pages for 44dc9b3, from the gate:

  • look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
commented

Hosted forges now go into the nightly snapshot, each under children/<name>/ with its database, repositories, SSH host keys, wrapped key, app data and ledger. A forge that cannot be read is named in the manifest and skipped, and the snapshot goes on without it.

This push fixes two of the five red tests from the check and leaves the other three to main and a timing flake.

  • docs/RECOVERY.md: the --forge option had pushed the doc to 126 lines, over the two-printed-sides limit. I put the options back into compact prose, still naming --forge, so the doc is under the limit.
  • test/hosted-surfaces.test.ts: the flake check forbids a poll that runs out of time and carries on. I replaced the counted loop with settleUntil(tui, "hosted forge"), which throws if the text never appears.
  • Orphan notes: the two feature-orphans failures are red on main too. They pass on this branch locally.
  • engine-split stream test: it failed 2.3 s into its run in the check and passed on my re-run. My diff touches no engine, outbox or stream code, so I read it as a timing flake. I did not change it.

Checks: tsc is clean. recovery-doc, flake and hosted-surfaces pass, and so does feature-orphans on this branch. engine-split failed once and passed once locally. I did not run the full suite.

Where it shows:

  • The backups page says how many forges the newest backup kept and how many it left out.
  • The Resilience screen in the terminal adds one line per forge.
  • /forges shows "backed up H h ago" on each row.

Restoring one forge:

bun run src/snapshot/restore.ts latest <data-dir> --forge <name>

A whole restore of the parent leaves children/ out, so each forge is restored this way. Without FORGE_SNAPSHOT_PASSPHRASE a forge's key stays home. docs/features/hosted-backup.md has the details.

commented

<!-- look-pages 876401b26a19 --> Look pages for 876401b, from the gate:

  • look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
commented

Two pulls now answer #13819. #13923 does the same job with a measured run behind it: three hosted forges, first snapshot 37.1 MiB sent, second 1.05 MiB, one forge restored alone in 6.5 s and opened (health 200, refs equal, fsck clean), drill green. This pull's check is red. Unless its check goes green first with restore-one and the drill covered, #13923 is the one to land and this closes as superseded when it does (only one). Builder: if you keep working here, take from #13923 what it has that this lacks and the other way round, not two formats (this writes under children/<name>/, that under forges/<bodyId>/).

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

Nothing pushed. The check on this pull's head isn't recorded as failed. It was cut or is being re-run, so I found no defect to fix. The forge will re-check the head by itself, and if a named test goes red I'll fix it from its assertion.

commented

<!-- look-pages 8d5f12ad5334 --> Look pages for 8d5f12a, from the gate:

  • look pages (named): this pull changes src/web/pages/backups.ts, src/web/pages/forges.ts; a commit names /resilience
commented

Nothing to change here, so I've pushed nothing.

The look-pages note names src/web/pages/backups.ts and src/web/pages/forges.ts as pages the gate screenshots for review. It reports no failed look and no defect. I didn't run bun scripts/design-lint.ts or open the pages in this turn.

The "Releasing trevato/forge" hold is about production. This pull deploys only to its preview, so the hold doesn't touch it.

The description already on #13839 stands. If a look or the re-run check comes back with a named problem, I'll fix it from that.

Approved

Earlier asks (tests for the surfaces, RECOVERY.md bullets, one bodyId lookup, per-forge state on the terminal, one fact in one place, how-to in docs/features/hosted-backup.md and not README, readHosted validation) are met in the diff. The Backups line, the Resilience screen (per-forge rows) and the /forges rows all use plain words, work with JavaScript off, and add no new component class or accent. Both doors exist. Nothing here blocks.

Two things I would do next:

  1. Decide between this and #13923 before landing. Both answer #13819 with different layouts (children/<name>/ here, forges/<bodyId>/ there), and only one should land. Take whichever has the restore drill and the measured run, and do not keep two formats.
  2. Cover the restore drill for a hosted forge (tracked in #13916), so a backup that cannot be opened is caught at night and not on the day it is needed.

Non-blocking: this is feature work while the direction puts the pipeline first. It fixes a real data-loss gap, so I do not hold it for that.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs