Public: anyone can read this; the crew can change it.

The forge can be restored when the forge is gone: the bucket's credentials are not only inside the database being backed up #14246

opened this in answer to

Check · Review · Land · Live — Changes were asked on this head. Next: builder, changes were asked on this head.

A reviewer asked for changes

The forge now produces a recovery kit, restore.ts --kit reads it, and the drill has an outside mode that the Backups verdict waits on. I ran the kit tests, the route list, the restore-card and recovery-doc tests and the existing backup, drill and verdict tests; one unrelated look test needing chromium failed here.

Notes · 18 files · +825 −12

look green — it answered, no tests were run, ready for a look deploy 423bdf6

Check passeddeploy 423bdf6

Website spec: 26 findings (report only)
opened this

The forge now produces a recovery kit, restore.ts --kit reads it, and the drill has an outside mode that the Backups verdict waits on. I ran the kit tests, the route list, the restore-card and recovery-doc tests and the existing backup, drill and verdict tests; one unrelated look test needing chromium failed here. The full suite was not run. I did not boot a real forge from a kit against a real bucket: the test restores through a stub S3 server and stubs the boot step.

What changed

  • Kit (src/snapshot/kit.ts): a JSON file with the bucket's endpoint, name and region, a read-only key, whether snapshots are sealed, the passphrase's fingerprint and the command. The fingerprint is a scrypt hash, never the passphrase.
    • The kit is made only from a read-only key sealed on the Backups page (SNAPSHOT_S3_READ_*). The forge cannot mint one on a generic S3, so the owner makes it in the bucket's console. Until then the page says no kit can be made.
    • New "Recovery kit" card on /settings/backups with a download button and a form for the read-only key. The download is a same-origin-checked POST, admin-only, no-store, and recorded.
    • The card says when the kit was last downloaded and whether it still matches the destination, key and passphrase. It also tells the owner to keep the kit off the machine.
  • restore.ts --kit <file>: reads the bucket from the kit and unseals with FORGE_SNAPSHOT_PASSPHRASE. A wrong passphrase is refused before the bucket is read. test/recovery-kit.test.ts restores the newest snapshot this way in a process with no S3_*, no FORGE_* and an empty home.
  • Outside drill (drill_outside.ts): after each green inside drill, a fresh process restores from the kit and passphrase alone into an empty directory, then the forge is booted and asked for /. The result is kept under drill/outside.json.
  • Verdict (bucket.ts): a forge with a destination is not green while only the inside drill has passed. It reads amber with "inside only" and the reason. A failed outside drill is red. The drill line says "inside and outside" once both have passed. A forge with no destination is judged as before.
  • Docs: docs/RESTORE.md has a new section and docs/RECOVERY.md lists the new flag.

Left out

  • The outside drill runs in a fresh process on the release checkout, not in a container from the release image. The sandbox has no container runtime. It shares the machine's disk and network, but none of the forge's environment or data.
  • The kit covers the primary destination only, not the off-site mirror.
  • The drill is chained after the inside drill and has no button of its own.
  • The kit's read-only key is only as read-only as the one the owner supplies. The forge checks that it can list the bucket, not that it cannot write.

Try it

  1. Open /settings/backups, seal a read-only key under "Recovery kit", and download the kit.
  2. On a clean machine with the checkout: FORGE_SNAPSHOT_PASSPHRASE=… bun run src/snapshot/restore.ts latest <empty-dir> --kit <file> then FORGE_DATA=<empty-dir> bun run src/main.ts.

🤖 Generated with Claude Code

Closes #14242

6 earlier comments
commented

The forge now produces a recovery kit, restore.ts --kit reads it, and the drill has an outside mode that the Backups verdict waits on. I ran the kit tests, the route list, the restore-card and recovery-doc tests and the existing backup, drill and verdict tests; one unrelated look test needing chromium failed here. The full suite was not run. I did not boot a real forge from a kit against a real bucket: the test restores through a stub S3 server and stubs the boot step.

What changed

  • Kit (src/snapshot/kit.ts): a JSON file with the bucket's endpoint, name and region, a read-only key, whether snapshots are sealed, the passphrase's fingerprint and the command. The fingerprint is a scrypt hash, never the passphrase.
    • The kit is made only from a read-only key sealed on the Backups page (SNAPSHOT_S3_READ_*). The forge cannot mint one on a generic S3, so the owner makes it in the bucket's console. Until then the page says no kit can be made.
    • New "Recovery kit" card on /settings/backups with a download button and a form for the read-only key. The download is a same-origin-checked POST, admin-only, no-store, and recorded.
    • The card says when the kit was last downloaded and whether it still matches the destination, key and passphrase. It also tells the owner to keep the kit off the machine.
  • restore.ts --kit <file>: reads the bucket from the kit and unseals with FORGE_SNAPSHOT_PASSPHRASE. A wrong passphrase is refused before the bucket is read. test/recovery-kit.test.ts restores the newest snapshot this way in a process with no S3_*, no FORGE_* and an empty home.
  • Outside drill (drill_outside.ts): after each green inside drill, a fresh process restores from the kit and passphrase alone into an empty directory, then the forge is booted and asked for /. The result is kept under drill/outside.json.
  • Verdict (bucket.ts): a forge with a destination is not green while only the inside drill has passed. It reads amber with "inside only" and the reason. A failed outside drill is red. The drill line says "inside and outside" once both have passed. A forge with no destination is judged as before.
  • Docs: docs/RESTORE.md has a new section and docs/RECOVERY.md lists the new flag.

Left out

  • The outside drill runs in a fresh process on the release checkout, not in a container from the release image. The sandbox has no container runtime. It shares the machine's disk and network, but none of the forge's environment or data.
  • The kit covers the primary destination only, not the off-site mirror.
  • The drill is chained after the inside drill and has no button of its own.
  • The kit's read-only key is only as read-only as the one the owner supplies. The forge checks that it can list the bucket, not that it cannot write.

Try it

  1. Open /settings/backups, seal a read-only key under "Recovery kit", and download the kit.
  2. On a clean machine with the checkout: FORGE_SNAPSHOT_PASSPHRASE=… bun run src/snapshot/restore.ts latest <empty-dir> --kit <file> then FORGE_DATA=<empty-dir> bun run src/main.ts.

🤖 Generated with Claude Code

commented

<!-- look-pages 5f154b96232b --> Look pages for 5f154b9, from the gate:

  • look pages (none): this pull changes src/web/pages/backups.ts, src/web/pages/backups_kit.ts but the forge could not trace the route of /settings/security/controls (an inline or wrapped handler), so it could not infer a page; the reviewer decides. A "Looks: <path>" line naming the nearest page, or a "No look: <why>" line (10+ characters), says it
commented

Pushed to the pull: the failing check was the gate's commit-message rules, so I added one commit carrying the lines it asks for. I did not run the gate again after committing, so those lines are untested.

  • Direction: the kit closes a trust gap (order 0 in docs/VISION.md): a forge that can't be restored once its machine is lost. That is why it goes ahead of pipeline items.
  • No door: there is no new terminal screen or docs/features how-to. The owner downloads the kit from the Backups page, and the terminal side is restore.ts --kit, documented in docs/RESTORE.md.
  • Looks: /settings/backups.

I also ran bun scripts/gate.ts --fix; it had no comment rewraps left to make, so nothing else changed.

No code changed, so the earlier test results stand.

commented

<!-- look-pages 78730313213c --> Look pages for 7873031, from the gate:

  • look pages (none): "Looks: /settings/backups" is not a page of app.looks in forge.json; this pull changes src/web/pages/backups.ts, src/web/pages/backups_kit.ts but the forge could not trace the route of /settings/security/controls (an inline or wrapped handler), so it could not infer a page; the reviewer decides. A "Looks: <path>" line naming the nearest page, or a "No look: <why>" line (10+ characters), says it
commented

Still red on current main: every spawn that pipes is listed with the reason it reads the pipe. Fix it and push.

commented

The three failing audits tripped on one file: src/snapshot/drill_outside.ts, which this pull adds. It starts a process, pipes its output, and sets a kill timer, and none of the three audit lists knew about it. I added it to each list, and the audits pass again:

  • test/spawn-sites.test.ts: it now lists the file as starting bun on the forge's own restore script with a fixed argv. The kit goes in a 0600 file in a fresh scratch directory. The passphrase goes in the environment only, and nothing comes from a request.
  • test/spawn-pipes.test.ts: it now lists the stdout+stderr pipes. Stdout is read for the restored snapshot id and stderr for the failure text, alongside the exit.
  • test/timers-named.test.ts: the SIGKILL timeout is now on the allowed list.

I ran only those three test files here, not the whole suite. bun x tsc --noEmit is clean.

bun scripts/gate.ts --fix left no changes to commit. Two notes from it:

  • Long comments: it reports comment lines over 100 columns in bucket.ts, destination.ts and restore.ts, and says the nightly rewrap on main handles them.
  • Looks: it can't tell which page /settings/backups belongs to. The pull description needs a Looks: line naming the nearest page, or a No look: line with the reason.

This pull adds a recovery kit and an outside drill, so the forge can be restored from the kit and the passphrase alone. To try it, open /settings/backups, save a read-only key under Recovery kit, and download the kit.

commented

<!-- look-pages 423bdf655566 --> Look pages for 423bdf6, from the gate:

  • look pages (none): "Looks: /settings/backups" is not a page of app.looks in forge.json; this pull changes src/web/pages/backups.ts, src/web/pages/backups_kit.ts but the forge could not trace the route of /settings/security/controls (an inline or wrapped handler), so it could not infer a page; the reviewer decides. A "Looks: <path>" line naming the nearest page, or a "No look: <why>" line (10+ characters), says it
Changes requested

The kit design is sound (read-only key only, POST + admin + same-origin, no-store, passphrase never stored), but it misses the bar in these ways. Fix in one round:

  1. Both doors (first): the kit is a page-only feature. There is no src/tui/ screen (download or save the read-only key, see the kit and outside-drill state) with its entry where AGENTS.md says, and no docs/features/<name>.md how-to. The restore.ts --kit CLI is not the terminal door for the Backups card. Add both; keep RESTORE.md as the recovery text. A terminal write must reach web viewers live.
  2. Tests missing: no test for the two POST handlers (non-admin gets 404, guest signs in, cross-origin is refused, download sends attachment + no-store), none that clearSealed also removes the read-only key, none for kitCard output, none for restore.ts --kit with a sealed kit (sealed: true; only the unsealed path is tested).
  3. Performance: kitCard calls makeKit on every render of /settings/backups, which runs scrypt (N=2^15, ~100ms) and decrypts the sealed secrets just to compute a status line. Compute the identity from stored values cheaply or cache it; the page must answer in milliseconds.
  4. The passphrase fingerprint is an offline guessing oracle for whoever holds the kit (8 bytes of scrypt, fixed salt). Document it in RESTORE.md, or derive the check from a sealed-snapshot header (the sealed store already fails on a wrong passphrase) rather than a value in the kit.
  5. Verdict regression: every existing forge with a destination goes from green to amber ("inside drill only") until an owner makes a read-only key in the bucket console. Say this in the pull, and make sure nothing that gates on green (card, recovery issue, release) is tripped by it.
  6. The outside drill is awaited inside the inside drill's chain (if (result.verdict === "green") await outside() before reportVerdict), so a slow or hung kit restore (budget 3x DRILL_TIMEOUT_MS) delays the drill's own report and drill.finished event. Run it after reporting, or detached.
  7. Clean: outsideNote/outsideUnproved are a convoluted pair of flags; the service.ts watchBucket callback has a mis-indented return array and a side effect (owesOutsideDrill) hidden in a getter; Bun.file(...).text().catch(() => "not readable") in restore.ts parses an error string as a kit. Tidy these.
  8. Design (see line comments).
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs