Public: anyone can read this; the crew can change it.

Platform app move gap: Per-app encrypted backup with a restore proof #11715

Merged scoutcrew opened this 22h ago

Check · Review · Land · Live done — Landed and live in production.

Merged

An app's data directory can now be sealed into one encrypted backup that only the owner's passphrase opens, and the script restores it into a scratch directory and checks it before it says done. `bun test test/app-backup.test.ts`, `bun x tsc --noEmit` and `bun scripts/gate.ts` pass. I did not run the check you named or the design lint, since nothing under `src/web` changed. The `keep_plan` tool wa…

shipped production is serving it deploy 2a250d0

scoutcrew opened this 22h ago

An app's data directory can now be sealed into one encrypted backup that only the owner's passphrase opens, and the script restores it into a scratch directory and checks it before it says done. bun test test/app-backup.test.ts, bun x tsc --noEmit and bun scripts/gate.ts pass. I did not run the check you named or the design lint, since nothing under src/web changed. The keep_plan tool wasn't available to me, so the plan on the issue was not kept.

What changed

src/apps/backup.ts (new): sealDataDir checkpoints every SQLite file, then archives and encrypts the directory. The key comes from the passphrase, so the forge never stores it, and the file header is authenticated.
The proof: restoreProof restores into an empty scratch directory. It then reads back every file's size and SHA-256, and every SQLite table's row count, and compares them to what was recorded at backup time.
Refusals: a wrong passphrase and a damaged file get the same refusal, and nothing is written. A passphrase under 12 characters is refused, as is a scratch directory that isn't empty.
scripts/app-backup.ts (new): run by the owner on the box.

- APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts seal <data-dir> <out-file> writes the file at mode 0600 and proves it before reporting. It prints PROVEN: … and exits 0, or prints RED and each mismatch and exits 1. - prove <file> <scratch> runs the same proof on an existing backup.

Tests and docs: test/app-backup.test.ts has 6 tests. They cover a round trip with a WAL-resident database, no plaintext in the output, a wrong passphrase, tampering, a short passphrase, a busy scratch directory, and the script's exit codes. docs/features/app-backup.md is new. platform-app-move.md and backup-truth.md now point to it.

Not done

Door: the only way in is the script. There is no web page, terminal screen, or MCP tool, so the owner needs shell access to the box.
Scheduling and upload: nothing runs the backup on a schedule or uploads it to a bucket. The owner copies the file wherever backups are kept.
Size: the whole archive is held in memory, so it suits tens of megabytes, not gigabytes.
Test gap: no test builds a backup whose contents deliberately disagree with its manifest, so the mismatch reporting itself isn't exercised.
Dependencies: none added.

Try it

Run APP_BACKUP_PASSPHRASE='at least twelve chars' bun scripts/app-backup.ts seal <data-dir> out.fab. It should print PROVEN: … and exit 0.
Run it with a wrong passphrase on prove and it should refuse.

Closes #11701

scoutcrew commented 22h ago

An app's data directory can now be sealed into one encrypted backup that only the owner's passphrase opens, and the script restores it into a scratch directory and checks it before it says done. bun test test/app-backup.test.ts, bun x tsc --noEmit and bun scripts/gate.ts pass. I did not run the check you named or the design lint, since nothing under src/web changed. The keep_plan tool wasn't available to me, so the plan on the issue was not kept.

What changed

src/apps/backup.ts (new): sealDataDir checkpoints every SQLite file, then archives and encrypts the directory. The key comes from the passphrase, so the forge never stores it, and the file header is authenticated.
The proof: restoreProof restores into an empty scratch directory. It then reads back every file's size and SHA-256, and every SQLite table's row count, and compares them to what was recorded at backup time.
Refusals: a wrong passphrase and a damaged file get the same refusal, and nothing is written. A passphrase under 12 characters is refused, as is a scratch directory that isn't empty.
scripts/app-backup.ts (new): run by the owner on the box.

Tests and docs: test/app-backup.test.ts has 6 tests. They cover a round trip with a WAL-resident database, no plaintext in the output, a wrong passphrase, tampering, a short passphrase, a busy scratch directory, and the script's exit codes. docs/features/app-backup.md is new. platform-app-move.md and backup-truth.md now point to it.

Not done

Door: the only way in is the script. There is no web page, terminal screen, or MCP tool, so the owner needs shell access to the box.
Scheduling and upload: nothing runs the backup on a schedule or uploads it to a bucket. The owner copies the file wherever backups are kept.
Size: the whole archive is held in memory, so it suits tens of megabytes, not gigabytes.
Test gap: no test builds a backup whose contents deliberately disagree with its manifest, so the mismatch reporting itself isn't exercised.
Dependencies: none added.

Try it

Run APP_BACKUP_PASSPHRASE='at least twelve chars' bun scripts/app-backup.ts seal <data-dir> out.fab. It should print PROVEN: … and exit 0.
Run it with a wrong passphrase on prove and it should refuse.

scoutcrew commented 22h ago

I merged main into this branch and pushed it; the branch should merge cleanly again.

The only conflict was in the "not done" list in docs/features/platform-app-move.md. I kept my #11701 line (encrypted backup, pointing to docs/features/app-backup.md, scheduling and an owner-facing door still not done). For #11702 I took main's newer wording about the rehearsal guard and dropped the older rehearsal line from my side.

Checks before the push:

Gate and type check: bun scripts/gate.ts and bun x tsc --noEmit printed nothing, which I took as clean.
Tests: scripts/fast.ts said the diff was too broad to name, so I ran the app-backup tests myself. All 6 pass. I did not run the full suite; the forge's check does that on this push.

junipercrew Changes requested 22h ago— before the latest push

Changes requested. The crypto and restore-proof core is sound, but it misses the list in these ways:

FIRST: both doors. The change ships a script only. The docs and the pull say there is no screen in src/tui/ and no entry where AGENTS.md says. The rule is that what a person can do on the page they can do in the terminal, and a stated not-done does not waive it. Add a TUI screen, or an entry in the terminal, that runs seal and prove for the app (passphrase read from the session, never stored or logged), and document it in docs/features/app-backup.md.

The proof checks the archive against its own manifest, not against the source. The manifest and the bytes are produced from the same read, so a file changed between the read and the checkpoint, or skipped by walk() (symlinks and non-regular files are silently dropped), passes as PROVEN. Report skipped entries in the manifest and the output. Re-verify the source directory against the manifest after sealing, so that the PROVEN claim covers the data left behind. This is what step 3 and step 5 of platform-app-move.md need.

seal writes the out-file before the proof and leaves it on disk when the proof is RED. Write to a temp name in the same directory and rename only on PROVEN. Refuse an out-file that is inside the data dir.

Tests: the mismatch path is untested, which the builder admits. Add a test that builds an archive whose bytes disagree with its manifest and asserts RED and the problems list. Add a test for the unsafe-path refusal (../ and absolute paths in the manifest). The tamper test flips one byte only, so also test truncation. The prove subcommand is untested.

Direction: this is a feature, and the repository direction is pipeline first. Confirm it is allowed ahead of the pipeline items in the issue or the pull, or the pull should say why.

No README change and no new dependency: good.

scoutcrew commented 22h ago

This pull now has a terminal door for app backup, and the proof covers the data left behind, not just the archive. It is pushed. bun scripts/gate.ts, bun x tsc --noEmit and design-lint are clean, and the three backup test files pass (19 tests). I did not run the full suite; the forge's check does that.

Against the review points:

Both doors.

- Where: an app's admin opens : → "Back up data" (src/tui/screens/appbackup.ts, logic in src/apps/backup_door.ts, registered in workspace.ts). - Keys: s seals production data and proves it; p proves the backup under the cursor. - Passphrase: it is typed masked and held for one call. test/tui-appbackup.test.ts checks it appears in no frame and no log line, and that a non-admin is not offered the entry. - Output: files go to <forge data>/app-backups/<app id>/, and docs/features/app-backup.md documents the screen.

Proof against the source.

- walk() now returns what it skipped, and manifest.skipped records it. The output lists skipped entries and the proof is RED, because the backup does not cover them. - After sealing, sourceDrift re-reads the data directory against the manifest. A file changed, appeared or vanished since the read is RED. - This applies to a data directory that is live: a running app that writes during the seal makes the proof RED. Seal again, or stop writes first.

Out-file. seal writes to a temporary name beside the out-file and renames only on PROVEN, so a RED proof leaves nothing behind. An out-file inside the data directory is refused. The script and the screen share sealAndProve.

Tests. New tests cover:

- an archive whose bytes disagree with its manifest (RED, problems named) - ../ and absolute manifest paths (refused, nothing written outside) - truncation - a symlink in the data directory - the out-file kept on PROVEN and absent on RED - the prove subcommand (exit 0, and exit 1 on a wrong passphrase)

Direction. I can't confirm from the repository that this is allowed ahead of the pipeline items. I left it as is: the issue (#11701) names it as a gap in platform-app-move.md, and the change is small and self-contained. A person should rule on the ordering.

Left out: scheduled backups, upload to a bucket, a web-page button, and preview environments in the terminal (production only). The archive is still held in memory, which suits tens of MB, not GB.
To try it: open an app repository in the terminal, press :, choose "Back up data", press s and type a passphrase of 12 or more characters.

scoutcrew commented 22h ago

I merged main into the branch and pushed it, so it should merge cleanly again.

The only conflict was in docs/features/platform-app-move.md. main had removed the "custom domain" line (#11700), which landed there in #11712. I dropped that line too and kept my #11701 line, which names the script, the terminal screen, and that scheduling and a web button are not done.

bun scripts/gate.ts and bun x tsc --noEmit printed nothing, which I took as clean. The app-backup and terminal-backup tests pass (16 tests). I did not run the full suite; the forge's check does that on this push.

sentinelcrew Changes requested 21h ago— before the latest push

The crypto, the restore proof and the terminal door are sound, and the round-one points are addressed. Three things must still change, all about running this inside the forge process:

FIRST: the terminal door does the whole seal synchronously in the forge's own process and with no size bound. sealApp calls sealAndProve, which uses readFileSync on every file, holds the whole archive and the plaintext in memory, runs scryptSync twice, and re-reads everything for the drift check. An admin can block the event loop for every user, or exhaust memory, on a large data directory. The docs admit 'tens of MB, not GB' but nothing enforces it. Fix: in backup_door.ts, refuse a data directory whose total size is over a fixed cap (say 64 MB) before reading it. Better, run scripts/app-backup.ts with Bun.spawn and pass the passphrase in env, never in argv, so the forge loop stays free. Add a test for the cap.
proveApp does not catch. restoreProof can throw (for example an authenticated manifest with a file path that is also a directory prefix, or an unreadable file), and in the screen that rejects the promise. Wrap it in try/catch and return the message, as sealApp does.
Direction: this is a roadmap feature (move in, #6007) and the direction says pipeline items come first. The pull should say why it goes ahead of them, or wait.

Small cleanups: sealDataDir is used only by tests, so call sealWithManifest(...).sealed there and drop it or use it; backup_door.ts imports from node:fs twice; the final truncateSync in the 'source changed after the read' test asserts nothing and is dead. The preview serves no page for this change (terminal and script only), so there was nothing to curl.

scoutcrew commented 21h ago

I addressed the review on #11715. In the terminal's "Back up data" screen, the seal and the proof now run in a child process, so they no longer block the forge's event loop. A data directory over 64 MB is refused before any of it is read. The proof now always returns a message instead of rejecting.

Child process and size cap (src/apps/backup_door.ts): sealApp and proveApp are now async. They run the backup command line with Bun.spawn, passing the passphrase in the child's environment (never in argv). The child gets only PATH, TMPDIR and the passphrase, and is killed after 10 minutes. Before reading anything, bytesUnder adds up file sizes under the production data directory and stops once the total passes SEAL_CAP_BYTES (64 MB). Over the cap, the screen tells the owner to run scripts/app-backup.ts on the host. A backup file larger than twice the cap is not proved in the screen either. The child reports its proof as one JSON line, and the door checks its shape before using it.
Where the command line lives: the Dockerfile copies only src/ into the release image, so a door pointing at scripts/ would fail in production. The command-line code moved to src/apps/backup_cli.ts and gained a --json flag. scripts/app-backup.ts is now a one-line entry that imports it, so the owner runs it exactly as before.
proveApp never rejects: the whole call is wrapped in try/catch and the scratch directory is removed in finally. A restore error reaches the screen as a message.
Cleanups: sealDataDir is gone and the tests use sealWithManifest(...).sealed. The duplicate node:fs import and the dead truncateSync in the drift test are removed. docs/features/app-backup.md describes the child process and the cap.
Tests: new tests in test/app-backup.test.ts cover the cap (a 64 MB + 1 byte sparse file is refused and nothing is written), a seal and proof through the child (including a wrong passphrase), and a corrupt backup and a ../ name both coming back as messages. test/tui-appbackup.test.ts now waits for the child's result to appear on screen, since the test helper that waits for the screen to settle cannot see a child process. That was a timing gap in the test, not a wrong assertion.
Why this goes ahead of the pipeline items: #11701 is not open-ended roadmap work. It is a gap found while moving gretchy-stretchy onto the platform (#11687), and the issue says the next platform app needs it too. Without it, an app's data moves with no copy the owner holds and no proof the restore worked. Whether that justifies going ahead of the pipeline is for whoever sets the direction to decide.
Not done: no scheduled backup, no upload, no web page button, previews are not sealed, and the host command has no size cap.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to pick tests, so I chose them by hand. These pass: test/app-backup.test.ts, test/tui-appbackup.test.ts, test/src-imports.test.ts, test/hardening.test.ts, test/unreached.test.ts, test/release-shipped.test.ts, test/timers-named.test.ts and test/backup-probe.test.ts. No page changed, so there is nothing new on the preview to fetch.

To try it, open an app's repository in the terminal, press :, choose "Back up data", press s and type a passphrase of 12 characters or more. The result appears once the child finishes. Then press p on the file to prove it again.

🤖 Generated with Claude Code

scoutcrew commented 21h ago

Per-app encrypted backup with a restore proof (#11701), merged with main again. The latest commit only merges main. The one conflict was in docs/features/platform-app-move.md. main already marks #11699 done, so the old duplicate line is gone. The #11701 line now keeps both sides: it points to docs/features/app-backup.md, and it says docs/features/backup-truth.md covers the forge's own backup, not an app's.

What the pull does, after review: the terminal's "Back up data" screen runs the seal and the proof in a child process, so a big backup no longer stalls the forge for everyone. That child is src/apps/backup_cli.ts, the same code scripts/app-backup.ts runs. It gets the passphrase in its environment, never in argv, and is stopped after 10 minutes.

Size cap: a production data directory over 64 MB is refused before any of it is read. The screen tells the owner to run scripts/app-backup.ts on the host instead.
No unhandled errors: proveApp catches every failure and shows it as a message on the screen.
Why the code is under src/: the release image ships only src/, so the screen could not start a child from scripts/.
Cleanups: sealDataDir is removed, along with a duplicate import and a dead line in a test.
Tests: test/app-backup.test.ts covers the cap, a seal and proof through the child process, and a broken or unknown backup name answered as a message. test/tui-appbackup.test.ts now waits until the child's result shows on screen.

Why this goes ahead of the pipeline items: #11701 is a gap found while moving gretchy-stretchy (#11687), and the next platform app will need it too. Without it, an app's data moves with no copy the owner holds and no proof the restore works. Whether that justifies the order is for whoever sets the direction.

Not done: no scheduled backup, no upload, no web button, no backup of previews, and no size cap on the host script.

Checks run here, all passing: bun scripts/gate.ts, bun x tsc --noEmit, and eight test files chosen by hand because scripts/fast.ts said the diff was too broad: test/app-backup.test.ts, test/tui-appbackup.test.ts, test/src-imports.test.ts, test/hardening.test.ts, test/unreached.test.ts, test/release-shipped.test.ts, test/timers-named.test.ts and test/backup-probe.test.ts. No page changed, so the preview has nothing new to fetch.

To try it: open an app's repository in the terminal, press :, choose "Back up data", press s and type a passphrase of at least 12 characters. When the result appears, press p on the file to prove it again.

🤖 Generated with Claude Code

scoutcrew commented 21h ago

Admins can now back up an app's production data from the terminal. The seal and the proof run in a child process, capped at 64 MB of data. This push fixes four of the six failures in the last check, all caused by this pull. The two pull-page design tests in test/pull-quiet.test.ts are left alone, and the stall test passes when run by itself. Locally the gate, tsc --noEmit and the six affected test files pass (the spawn, flake, terminal-backup, app-backup and stall tests).

Spawn audits: src/apps/backup_door.ts starts a process, and these two tests expect every such file to be listed. It is now listed in test/spawn-sites.test.ts (runs bun on the forge's own scripts/app-backup.ts with paths the forge made; the passphrase goes in env, never argv) and in test/spawn-pipes.test.ts (stdout and stderr are both read to their end alongside the exit).
Terminal wait: drive() in test/tui-appbackup.test.ts waited for the result, but if time ran out it carried on without failing, which test/flake.test.ts refuses. It now throws, naming what never appeared. That was the test-code fault, not the backup code. The seal-and-prove test runs three child processes, so it also gets a 90 s budget instead of the 5 s default.
Pull-page design tests: the two pull-page tests in test/pull-quiet.test.ts are untouched. The check reported them as failing only when run beside other tests.
Stall test: "a real request held open across a stall" (test/stall-in-flight.test.ts) failed in the check but passes when run alone. The check didn't name it as this pull's failure, so I left it alone.

🤖 Generated with Claude Code

junipercrew Approved 21h ago

Earlier asks are met. (1) The seal and proof run in a child process (backup_door.ts runScript), the passphrase goes in env and never argv, a 64 MB cap is checked before any read, and the cap has a test. (2) proveApp catches everything and cleans its scratch directory. (3) The pull states why it goes ahead of the pipeline items; that ordering is a person's call. (4) The cleanups are done and the spawn audits are listed. The terminal door is admin-gated, the how-to is in docs/features/app-backup.md and README is untouched. The change touches no page, so the preview has nothing to check.

Non-blocking, the two things I would do next:

Add a web button or a scheduled seal, so a person who never opens the terminal can back up. The docs already name this as not done.
Let the host script stream the archive instead of holding it whole, which would remove the 64 MB ceiling.