Public: anyone can read this; the crew can change it.
Platform app move gap: Per-app encrypted backup with a restore proof #11715
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/11701-platform-app-move-gap-per-app-encrypted → main 8 commits fork 2 threads · 2 unresolved
14 files
changed +954
−1
docs/features/app-backup.md added +53
| @@ -0,0 +1,53 @@ | ||
| 1 | # An app's encrypted backup, with a restore proof (#11701) | |
| 2 | ||
| 3 | The forge's backup (`backup-truth.md`) covers the forge. An app's data | |
| 4 | (`FORGE_APP_DATA`: its SQLite files and uploads) gets a backup the **owner** | |
| 5 | holds the key to, and a restore that proves itself. | |
| 6 | ||
| 7 | ||
| 8 | export APP_BACKUP_PASSPHRASE='…at least 12 characters, kept off the forge…' | |
| 9 | bun scripts/app-backup.ts seal <data-dir> <out-file> | |
| 10 | bun scripts/app-backup.ts prove <out-file> <empty-scratch-dir> | |
| 11 | ||
| 12 | ||
| 13 | - `seal` checkpoints every SQLite file, archives the directory, encrypts it | |
| 14 | (AES-256-GCM, key = scrypt of the passphrase, header authenticated) and | |
| 15 | writes `<out-file>` with mode 0600. It then restores the file into a scratch | |
| 16 | directory of its own and prints `PROVEN: <files>, <bytes>, <tables>, <rows>`, | |
| 17 | exit 0, or `RED` and each mismatch, exit 1. Delete nothing on the old side | |
| 18 | until it says PROVEN. | |
| 19 | - The proof reads back from disk every file's size and SHA-256, and every | |
| 20 | SQLite table's row count, against the manifest taken at backup time | |
| 21 | (the step-3 and step-5 comparison in `platform-app-move.md`). | |
| 22 | - The seal is only kept when it is clean. It is written under a temporary | |
| 23 | name beside `<out-file>` and renamed on PROVEN; a RED proof leaves nothing. | |
| 24 | An out-file inside the data directory is refused. | |
| 25 | - The proof covers the data left behind too: after sealing, the data | |
| 26 | directory is read again against the manifest, and a file that changed, | |
| 27 | appeared or vanished since the read is RED. Entries the walk cannot | |
| 28 | archive (symlinks, sockets, devices) are listed in the manifest (`skipped`) | |
| 29 | and in the output, and make the proof RED: the backup does not cover them. | |
| 30 | `-wal` and `-shm` files are not archived; the checkpoint empties them. | |
| 31 | - A wrong passphrase and a damaged or truncated file are refused alike, before | |
| 32 | anything is written. The passphrase is only ever read from the environment | |
| 33 | (script) or typed masked (terminal); the forge never stores or logs it. | |
| 34 | ||
| 35 | ## In the terminal | |
| 36 | ||
| 37 | An admin of an app opens `:` → "Back up data" (`src/tui/screens/appbackup.ts`). | |
| 38 | `s` asks for the passphrase (masked) and seals the app's production data into | |
| 39 | `<forge data>/app-backups/<app id>/production-<time>.fab`, proving it and the | |
| 40 | data left behind as above; `p` proves the file under the cursor. The screen | |
| 41 | shows the same PROVEN or RED lines as the script. The file is encrypted under | |
| 42 | a key only the owner holds; copy it off the forge to keep it. The seal and | |
| 43 | the proof run `src/apps/backup_cli.ts` (the code behind `scripts/app-backup.ts`) in a child process (the passphrase in | |
| 44 | its environment, never in argv), so the forge's event loop stays free; a | |
| 45 | production data directory over 64 MB (`SEAL_CAP_BYTES`) is refused before it | |
| 46 | is read, and the screen says to run the script on the host instead. Logic: | |
| 47 | `src/apps/backup_door.ts`; tests: `test/tui-appbackup.test.ts`. | |
| 48 | ||
| 49 | - Code: `src/apps/backup.ts`; tests: `test/app-backup.test.ts`. | |
| 50 | ||
| 51 | **Not done:** the archive is held in memory (the terminal caps it at 64 MB; | |
| 52 | the script on the host has no cap); no scheduled backup, no upload to a bucket, and no web page button; the | |
| 53 | terminal seals production data only, not previews. |
docs/features/backup-truth.md +2
| @@ -134,3 +134,5 @@ network blocked (egress allowed to B2 only, e.g. a firewall rule on the box): | ||
| 134 | 134 | |
| 135 | 135 | Not done here: the drill itself, the dump decision, and the lane rows. Each |
| 136 | 136 | needs the box or the owner; this section is what they follow. |
| 137 | ||
| 138 | An app's own data has its own encrypted backup and restore proof: `app-backup.md` (#11701). |
docs/features/platform-app-move.md +1 −1
| @@ -150,7 +150,7 @@ reminds anyone when the week ends; both are a person's. | ||
| 150 | 150 | - A database import for an app (`migrate` has no data path): upload a dump, load it, report counts and checksums. (#11697) |
| 151 | 151 | - ~~A file and upload copy with a manifest compared on both ends.~~ Done as a script a person runs (#11699); no page or upload from a browser yet. |
| 152 | 152 | - ~~A Postgres app has no home here.~~ Decided, not built: no managed Postgres; port to `bun:sqlite` or keep an external one by `egress` (docs/APPS.md). (#11698) |
| 153 | - Per-app encrypted backup with a restore proof (`docs/features/backup-truth.md` covers the forge's own, not an app's). (#11701) | |
| 153 | - Per-app encrypted backup with a restore proof: `scripts/app-backup.ts` and the terminal, see `docs/features/app-backup.md` (`docs/features/backup-truth.md` covers the forge's own, not an app's). Scheduling and a web button are not done. (#11701) | |
| 154 | 154 | - A guard for rehearsal: data copy and `preview`-scoped sandbox keys exist and are the procedure in step 4; nothing yet refuses a payment name left in scope `all`. (#11702) |
| 155 | 155 | - ~~Inbound payment webhooks that survive a preview's changing address.~~ Built: `/__forge/hook/<pull>/…` on production's host. (#11703) |
| 156 | 156 | - ~~A read-only switch for the old side of a migration.~~ A procedure at the old proxy and database, section 7; the forge cannot reach the old node, so no feature is built. (#11704) |
scripts/app-backup.ts added +7
| @@ -0,0 +1,7 @@ | ||
| 1 | /** | |
| 2 | * An app's encrypted backup and its restore proof, run by the owner: | |
| 3 | * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts seal <data-dir> <out-file> | |
| 4 | * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts prove <backup-file> <empty-scratch-dir> | |
| 5 | * The work is src/apps/backup_cli.ts. See docs/features/app-backup.md. | |
| 6 | */ | |
| 7 | import "../src/apps/backup_cli.ts"; |
src/apps/backup.ts added +281
| @@ -0,0 +1,281 @@ | ||
| 1 | /** | |
| 2 | * An app's own backup: its data directory sealed under a passphrase the | |
| 3 | * owner holds, and a restore that proves itself. The forge's snapshot | |
| 4 | * (`src/snapshot`) covers the forge; this covers one environment's | |
| 5 | * `FORGE_APP_DATA`, so an owner can keep a copy the forge's operator cannot | |
| 6 | * open and a person who moves an app off the forge can prove it came back. | |
| 7 | * | |
| 8 | * File format: `FAB1\n`, one JSON header line (`v`, `kdf`, `salt`, `iv`, | |
| 9 | * base64url), then AES-256-GCM ciphertext ending in its 16-byte tag. The | |
| 10 | * key is scrypt of the passphrase; the header is authenticated. Inside is a | |
| 11 | * manifest (every file's path, size, SHA-256, and every SQLite table's row | |
| 12 | * count) followed by the file bytes. Whole-archive in memory: meant for | |
| 13 | * app data of tens of megabytes, not gigabytes (`docs/features/app-backup.md`). | |
| 14 | */ | |
| 15 | ||
| 16 | import { Database } from "bun:sqlite"; | |
| 17 | import { | |
| 18 | createCipheriv, | |
| 19 | createDecipheriv, | |
| 20 | createHash, | |
| 21 | randomBytes, | |
| 22 | scryptSync, | |
| 23 | } from "node:crypto"; | |
| 24 | import { | |
| 25 | existsSync, | |
| 26 | lstatSync, | |
| 27 | mkdirSync, | |
| 28 | readdirSync, | |
| 29 | mkdtempSync, | |
| 30 | readFileSync, | |
| 31 | realpathSync, | |
| 32 | renameSync, | |
| 33 | rmSync, | |
| 34 | writeFileSync, | |
| 35 | } from "node:fs"; | |
| 36 | import { tmpdir } from "node:os"; | |
| 37 | import { basename, dirname, join, resolve, sep } from "node:path"; | |
| 38 | import { quiesceSqlite } from "./data.ts"; | |
| 39 | ||
| 40 | const MAGIC = "FAB1\n"; | |
| 41 | const SCRYPT = { N: 1 << 15, r: 8, p: 1, maxmem: 64 * 1024 * 1024 } as const; | |
| 42 | const MIN_PASSPHRASE = 12; | |
| 43 | ||
| 44 | export interface FileEntry { | |
| 45 | path: string; | |
| 46 | size: number; | |
| 47 | sha256: string; | |
| 48 | } | |
| 49 | export interface Manifest { | |
| 50 | files: FileEntry[]; | |
| 51 | /** SQLite file path → table → row count, at the moment of the backup. */ | |
| 52 | tables: Record<string, Record<string, number>>; | |
| 53 | /** Entries the walk could not archive (symlinks, devices, sockets): never silently dropped. */ | |
| 54 | skipped?: string[]; | |
| 55 | } | |
| 56 | export interface Proof { | |
| 57 | ok: boolean; | |
| 58 | files: number; | |
| 59 | bytes: number; | |
| 60 | tables: number; | |
| 61 | rows: number; | |
| 62 | /** What did not match; empty when ok. */ | |
| 63 | problems: string[]; | |
| 64 | } | |
| 65 | ||
| 66 | const sha = (b: Uint8Array): string => createHash("sha256").update(b).digest("hex"); | |
| 67 | ||
| 68 | export function passphraseProblem(p: string | undefined): string | undefined { | |
| 69 | if (p === undefined || p === "") return "no passphrase: set APP_BACKUP_PASSPHRASE"; | |
| 70 | if (p.length < MIN_PASSPHRASE) return `the passphrase is under ${MIN_PASSPHRASE} characters`; | |
| 71 | return undefined; | |
| 72 | } | |
| 73 | ||
| 74 | function walk(root: string, rel = "", out: string[] = [], skipped: string[] = []): { files: string[]; skipped: string[] } { | |
| 75 | for (const e of readdirSync(join(root, rel), { withFileTypes: true })) { | |
| 76 | const r = rel === "" ? e.name : `${rel}/${e.name}`; | |
| 77 | if (e.isDirectory()) walk(root, r, out, skipped); | |
| 78 | else if (/-(wal|shm)$/.test(e.name)) continue; // emptied by the checkpoint | |
| 79 | else if (e.isFile()) out.push(r); | |
| 80 | else skipped.push(r); | |
| 81 | } | |
| 82 | return { files: out.sort(), skipped: skipped.sort() }; | |
| 83 | } | |
| 84 | ||
| 85 | const isSqlite = (bytes: Uint8Array): boolean => | |
| 86 | bytes.length >= 16 && Buffer.from(bytes.subarray(0, 16)).toString("latin1") === "SQLite format 3\0"; | |
| 87 | ||
| 88 | /** Row counts for every table of a SQLite file, read-only; `undefined` if it will not open. */ | |
| 89 | export function tableCounts(path: string): Record<string, number> | undefined { | |
| 90 | let db: Database | undefined; | |
| 91 | try { | |
| 92 | db = new Database(path, { readonly: true }); | |
| 93 | const names = db | |
| 94 | .query("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name") | |
| 95 | .all() as { name: string }[]; | |
| 96 | const counts: Record<string, number> = {}; | |
| 97 | for (const { name } of names) { | |
| 98 | const row = db.query(`SELECT count(*) AS n FROM "${name.replaceAll('"', '""')}"`).get() as { n: number }; | |
| 99 | counts[name] = row.n; | |
| 100 | } | |
| 101 | return counts; | |
| 102 | } catch { | |
| 103 | return undefined; | |
| 104 | } finally { | |
| 105 | db?.close(); | |
| 106 | } | |
| 107 | } | |
| 108 | ||
| 109 | /** Seal a data directory. Throws a message (not a bug) for an unusable passphrase or directory. */ | |
| 110 | export function sealWithManifest(dir: string, passphrase: string): { sealed: Buffer; manifest: Manifest } { | |
| 111 | const bad = passphraseProblem(passphrase); | |
| 112 | if (bad !== undefined) throw new Error(bad); | |
| 113 | if (!existsSync(dir) || !lstatSync(dir).isDirectory()) throw new Error("the data directory does not exist"); | |
| 114 | quiesceSqlite(dir); | |
| 115 | const found = walk(dir); | |
| 116 | const manifest: Manifest = { files: [], tables: {}, skipped: found.skipped }; | |
| 117 | const bodies: Buffer[] = []; | |
| 118 | for (const path of found.files) { | |
| 119 | const bytes = readFileSync(join(dir, path)); | |
| 120 | manifest.files.push({ path, size: bytes.length, sha256: sha(bytes) }); | |
| 121 | bodies.push(bytes); | |
| 122 | if (isSqlite(bytes)) { | |
| 123 | const counts = tableCounts(join(dir, path)); | |
| 124 | if (counts !== undefined) manifest.tables[path] = counts; | |
| 125 | } | |
| 126 | } | |
| 127 | return { sealed: sealParts(manifest, bodies, passphrase), manifest }; | |
| 128 | } | |
| 129 | ||
| 130 | /** Encrypt a manifest and its file bodies as given; `sealWithManifest` is the honest caller. */ | |
| 131 | export function sealParts(manifest: Manifest, bodies: Buffer[], passphrase: string): Buffer { | |
| 132 | const m = Buffer.from(JSON.stringify(manifest)); | |
| 133 | const len = Buffer.alloc(4); | |
| 134 | len.writeUInt32BE(m.length); | |
| 135 | const plain = Buffer.concat([len, m, ...bodies]); | |
| 136 | ||
| 137 | const salt = randomBytes(16); | |
| 138 | const iv = randomBytes(12); | |
| 139 | const header = Buffer.from( | |
| 140 | `${JSON.stringify({ v: 1, kdf: "scrypt", salt: salt.toString("base64url"), iv: iv.toString("base64url") })}\n`, | |
| 141 | ); | |
| 142 | const cipher = createCipheriv("aes-256-gcm", scryptSync(passphrase, salt, 32, SCRYPT), iv, { authTagLength: 16 }); | |
| 143 | cipher.setAAD(header); | |
| 144 | return Buffer.concat([Buffer.from(MAGIC), header, cipher.update(plain), cipher.final(), cipher.getAuthTag()]); | |
| 145 | } | |
| 146 | ||
| 147 | /** The manifest and file bytes, or a message: a wrong passphrase and a damaged file say the same thing. */ | |
| 148 | export function openBackup(sealed: Uint8Array, passphrase: string): { manifest: Manifest; bodies: Buffer[] } | string { | |
| 149 | const refuse = "cannot open this backup: wrong passphrase, or the file is damaged"; | |
| 150 | try { | |
| 151 | const buf = Buffer.from(sealed); | |
| 152 | if (buf.subarray(0, MAGIC.length).toString("latin1") !== MAGIC) return "not an app backup file"; | |
| 153 | const nl = buf.indexOf(10, MAGIC.length); | |
| 154 | if (nl < 0 || nl > 512 || buf.length < nl + 1 + 16) return refuse; | |
| 155 | const header = buf.subarray(MAGIC.length, nl + 1); | |
| 156 | const h = JSON.parse(header.toString()) as Record<string, unknown>; | |
| 157 | if (h["v"] !== 1 || h["kdf"] !== "scrypt") return refuse; | |
| 158 | const salt = Buffer.from(String(h["salt"]), "base64url"); | |
| 159 | const iv = Buffer.from(String(h["iv"]), "base64url"); | |
| 160 | if (salt.length !== 16 || iv.length !== 12) return refuse; | |
| 161 | const body = buf.subarray(nl + 1); | |
| 162 | const d = createDecipheriv("aes-256-gcm", scryptSync(passphrase, salt, 32, SCRYPT), iv, { authTagLength: 16 }); | |
| 163 | d.setAAD(header); | |
| 164 | d.setAuthTag(body.subarray(body.length - 16)); | |
| 165 | const plain = Buffer.concat([d.update(body.subarray(0, body.length - 16)), d.final()]); | |
| 166 | const mlen = plain.readUInt32BE(0); | |
| 167 | const manifest = JSON.parse(plain.subarray(4, 4 + mlen).toString()) as Manifest; | |
| 168 | let at = 4 + mlen; | |
| 169 | const bodies: Buffer[] = []; | |
| 170 | for (const f of manifest.files) { | |
| 171 | bodies.push(plain.subarray(at, at + f.size)); | |
| 172 | at += f.size; | |
| 173 | } | |
| 174 | return { manifest, bodies }; | |
| 175 | } catch { | |
| 176 | return refuse; | |
| 177 | } | |
| 178 | } | |
| 179 | ||
| 180 | const safeRel = (p: string): boolean => | |
| 181 | p !== "" && !p.startsWith("/") && !p.includes("\0") && !p.split("/").some((s) => s === ".." || s === "." || s === ""); | |
| 182 | ||
| 183 | /** | |
| 184 | * Restore a backup into `scratch` (must not exist or be empty) and prove it: | |
| 185 | * every file's SHA-256 and size read back from disk, every SQLite table's row | |
| 186 | * count read back from the restored database. Returns a message when it cannot | |
| 187 | * even try (wrong passphrase, scratch in use). | |
| 188 | */ | |
| 189 | export function restoreProof(sealed: Uint8Array, passphrase: string, scratch: string): Proof | string { | |
| 190 | const bad = passphraseProblem(passphrase); | |
| 191 | if (bad !== undefined) return bad; | |
| 192 | if (existsSync(scratch) && readdirSync(scratch).length > 0) return "the scratch directory is not empty"; | |
| 193 | const opened = openBackup(sealed, passphrase); | |
| 194 | if (typeof opened === "string") return opened; | |
| 195 | const { manifest, bodies } = opened; | |
| 196 | const root = resolve(scratch); | |
| 197 | mkdirSync(root, { recursive: true }); | |
| 198 | const problems: string[] = []; | |
| 199 | for (const k of manifest.skipped ?? []) | |
| 200 | problems.push(`${k}: not archived (a symlink or special file); the backup does not cover it`); | |
| 201 | let bytes = 0; | |
| 202 | manifest.files.forEach((f, i) => { | |
| 203 | const target = resolve(root, f.path); | |
| 204 | if (!safeRel(f.path) || !target.startsWith(root + sep)) { | |
| 205 | problems.push(`refused an unsafe path: ${JSON.stringify(f.path)}`); | |
| 206 | return; | |
| 207 | } | |
| 208 | mkdirSync(dirname(target), { recursive: true }); | |
| 209 | writeFileSync(target, bodies[i]!); | |
| 210 | }); | |
| 211 | for (const f of manifest.files) { | |
| 212 | const target = resolve(root, f.path); | |
| 213 | if (!safeRel(f.path) || !target.startsWith(root + sep)) continue; | |
| 214 | const back = readFileSync(target); | |
| 215 | bytes += back.length; | |
| 216 | if (back.length !== f.size || sha(back) !== f.sha256) problems.push(`${f.path}: bytes differ from the backup's manifest`); | |
| 217 | } | |
| 218 | let tables = 0; | |
| 219 | let rows = 0; | |
| 220 | for (const [file, expected] of Object.entries(manifest.tables)) { | |
| 221 | if (!safeRel(file)) continue; | |
| 222 | const got = tableCounts(join(root, file)); | |
| 223 | for (const [t, n] of Object.entries(expected)) { | |
| 224 | tables++; | |
| 225 | rows += n; | |
| 226 | if (got === undefined) problems.push(`${file}: restored database does not open`); | |
| 227 | else if (got[t] !== n) problems.push(`${file} ${t}: ${n} rows backed up, ${got[t] ?? "no table"} restored`); | |
| 228 | } | |
| 229 | } | |
| 230 | return { ok: problems.length === 0, files: manifest.files.length, bytes, tables, rows, problems }; | |
| 231 | } | |
| 232 | ||
| 233 | /** The data directory against the manifest taken from it: what changed since, and what was left out. */ | |
| 234 | export function sourceDrift(dir: string, manifest: Manifest): string[] { | |
| 235 | const problems: string[] = []; | |
| 236 | const now = walk(dir); | |
| 237 | const had = new Set(manifest.files.map((f) => f.path)); | |
| 238 | for (const p of now.files) if (!had.has(p)) problems.push(`${p}: appeared in the data directory after the backup was read`); | |
| 239 | for (const f of manifest.files) { | |
| 240 | let back: Buffer; | |
| 241 | try { | |
| 242 | back = readFileSync(join(dir, f.path)); | |
| 243 | } catch { | |
| 244 | problems.push(`${f.path}: gone from the data directory since the backup was read`); | |
| 245 | continue; | |
| 246 | } | |
| 247 | if (back.length !== f.size || sha(back) !== f.sha256) problems.push(`${f.path}: changed in the data directory since the backup was read`); | |
| 248 | } | |
| 249 | return problems; | |
| 250 | } | |
| 251 | ||
| 252 | /** | |
| 253 | * Seal `dir` into `out`, prove the result and the data left behind, and keep | |
| 254 | * the file only when both are clean: it is written under a temporary name in | |
| 255 | * the same directory and renamed on PROVEN, so a RED proof leaves no backup | |
| 256 | * that looks like one. Throws a message for unusable input. | |
| 257 | */ | |
| 258 | export function sealAndProve(dir: string, out: string, passphrase: string): Proof { | |
| 259 | const bad = passphraseProblem(passphrase); | |
| 260 | if (bad !== undefined) throw new Error(bad); | |
| 261 | if (!existsSync(dir) || !lstatSync(dir).isDirectory()) throw new Error("the data directory does not exist"); | |
| 262 | const outDir = realpathSync(dirname(resolve(out))); | |
| 263 | const target = join(outDir, basename(out)); | |
| 264 | const root = realpathSync(dir); | |
| 265 | if (target === root || target.startsWith(root + sep)) throw new Error("the out-file is inside the data directory"); | |
| 266 | const { sealed, manifest } = sealWithManifest(dir, passphrase); | |
| 267 | const part = join(outDir, `.${basename(out)}.${randomBytes(6).toString("hex")}.part`); | |
| 268 | const scratch = mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "app-backup-")); | |
| 269 | try { | |
| 270 | writeFileSync(part, sealed, { mode: 0o600 }); | |
| 271 | const p = restoreProof(readFileSync(part), passphrase, join(scratch, "restore")); | |
| 272 | if (typeof p === "string") throw new Error(p); | |
| 273 | p.problems.push(...sourceDrift(dir, manifest)); | |
| 274 | p.ok = p.problems.length === 0; | |
| 275 | if (p.ok) renameSync(part, target); | |
| 276 | return p; | |
| 277 | } finally { | |
| 278 | rmSync(part, { force: true }); | |
| 279 | rmSync(scratch, { recursive: true, force: true }); | |
| 280 | } | |
| 281 | } |
src/apps/backup_cli.ts added +43
| @@ -0,0 +1,43 @@ | ||
| 1 | /** | |
| 2 | * An app's encrypted backup and its restore proof, as a command line: the owner runs it, and so does the terminal's door. | |
| 3 | * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts seal <data-dir> <out-file> | |
| 4 | * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts prove <backup-file> <empty-scratch-dir> | |
| 5 | * A trailing `--json` prints the proof as one JSON line in place of the | |
| 6 | * words (the terminal's door, src/apps/backup_door.ts, reads it). | |
| 7 | * `seal` proves the sealed bytes and the data directory left behind, and | |
| 8 | * keeps <out-file> only when the proof passed. Exit 0 only then. | |
| 9 | * This lives under src/ because the terminal's door (backup_door.ts) runs it as | |
| 10 | * a child process and the release image ships src/ alone; scripts/app-backup.ts | |
| 11 | * is the owner's entry to it. See docs/features/app-backup.md. | |
| 12 | */ | |
| 13 | import { readFileSync } from "node:fs"; | |
| 14 | import { restoreProof, sealAndProve, type Proof } from "./backup.ts"; | |
| 15 | import { proofLines } from "./backup_words.ts"; | |
| 16 | ||
| 17 | const argv = process.argv.slice(2); | |
| 18 | const json = argv.at(-1) === "--json"; | |
| 19 | const [cmd, a, b] = json ? argv.slice(0, -1) : argv; | |
| 20 | const pass = process.env["APP_BACKUP_PASSPHRASE"] ?? ""; | |
| 21 | const show = (p: Proof): void => { | |
| 22 | if (json) console.log(JSON.stringify(p)); | |
| 23 | else for (const l of proofLines(p)) console.log(l); | |
| 24 | }; | |
| 25 | ||
| 26 | try { | |
| 27 | if (cmd === "seal" && a && b) { | |
| 28 | const p = sealAndProve(a, b, pass); | |
| 29 | show(p); | |
| 30 | process.exit(p.ok ? 0 : 1); | |
| 31 | } else if (cmd === "prove" && a && b) { | |
| 32 | const p = restoreProof(readFileSync(a), pass, b); | |
| 33 | if (typeof p === "string") throw new Error(p); | |
| 34 | show(p); | |
| 35 | process.exit(p.ok ? 0 : 1); | |
| 36 | } else { | |
| 37 | console.error("usage: app-backup.ts seal <data-dir> <out-file> | prove <backup-file> <empty-scratch-dir>"); | |
| 38 | process.exit(2); | |
| 39 | } | |
| 40 | } catch (e) { | |
| 41 | console.error(`app-backup: ${(e as Error).message}`); | |
| 42 | process.exit(1); | |
| 43 | } |
src/apps/backup_door.ts added +122
| @@ -0,0 +1,122 @@ | ||
| 1 | /** | |
| 2 | * What the terminal's "Back up data" screen does for one app: seal its | |
| 3 | * production data under the owner's passphrase into `<data>/app-backups/<appId>/`, | |
| 4 | * list what is there, and prove one file. The work runs in a child process | |
| 5 | * (`src/apps/backup_cli.ts`, the code behind `scripts/app-backup.ts`), so the forge's event loop stays free; the | |
| 6 | * passphrase goes to it in the environment, never in argv, and is never | |
| 7 | * stored, logged or returned. A data directory over `SEAL_CAP_BYTES` is | |
| 8 | * refused before a byte of it is read: the archive is held whole in memory. | |
| 9 | */ | |
| 10 | import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statSync } from "node:fs"; | |
| 11 | import { tmpdir } from "node:os"; | |
| 12 | import { join } from "node:path"; | |
| 13 | import type { Proof } from "./backup.ts"; | |
| 14 | import { envDataDir } from "./data.ts"; | |
| 15 | ||
| 16 | export const SEAL_CAP_BYTES = 64 * 1024 * 1024; | |
| 17 | const SCRIPT = new URL("./backup_cli.ts", import.meta.url).pathname; | |
| 18 | const CHILD_TIMEOUT_MS = 10 * 60_000; | |
| 19 | ||
| 20 | const dirFor = (dataDir: string, appId: string): string => join(dataDir, "app-backups", appId); | |
| 21 | ||
| 22 | export interface BackupFile { | |
| 23 | name: string; | |
| 24 | bytes: number; | |
| 25 | at: string; | |
| 26 | } | |
| 27 | ||
| 28 | export function listBackups(dataDir: string, appId: string): BackupFile[] { | |
| 29 | const d = dirFor(dataDir, appId); | |
| 30 | if (!existsSync(d)) return []; | |
| 31 | return readdirSync(d) | |
| 32 | .filter((n) => /^production-[0-9TZ-]+\.fab$/.test(n)) | |
| 33 | .sort() | |
| 34 | .reverse() | |
| 35 | .map((name) => { | |
| 36 | const st = statSync(join(d, name)); | |
| 37 | return { name, bytes: st.size, at: st.mtime.toISOString() }; | |
| 38 | }); | |
| 39 | } | |
| 40 | ||
| 41 | /** Total bytes of the regular files under `dir`, stopping as soon as it passes `cap`. Sizes only: nothing is read. */ | |
| 42 | export function bytesUnder(dir: string, cap: number): number { | |
| 43 | let total = 0; | |
| 44 | const stack = [dir]; | |
| 45 | while (stack.length > 0 && total <= cap) { | |
| 46 | const d = stack.pop()!; | |
| 47 | for (const e of readdirSync(d, { withFileTypes: true })) { | |
| 48 | const p = join(d, e.name); | |
| 49 | if (e.isDirectory()) stack.push(p); | |
| 50 | else if (e.isFile()) total += lstatSync(p).size; | |
| 51 | if (total > cap) break; | |
| 52 | } | |
outdated
restoreProof can throw; catch and return the message like sealApp, or the TUI promise rejects. |
||
| 53 | } | |
| 54 | return total; | |
| 55 | } | |
| 56 | ||
| 57 | const isProof = (v: unknown): v is Proof => { | |
| 58 | if (typeof v !== "object" || v === null) return false; | |
| 59 | const p = v as Record<string, unknown>; | |
| 60 | return ( | |
| 61 | typeof p["ok"] === "boolean" && | |
| 62 | ["files", "bytes", "tables", "rows"].every((k) => typeof p[k] === "number") && | |
| 63 | Array.isArray(p["problems"]) && | |
| 64 | (p["problems"] as unknown[]).every((x) => typeof x === "string") | |
| 65 | ); | |
| 66 | }; | |
| 67 | ||
| 68 | /** Run the script with the passphrase in env; its last stdout line is the proof as JSON, its stderr the refusal. */ | |
| 69 | async function runScript(args: string[], passphrase: string): Promise<Proof | string> { | |
| 70 | try { | |
| 71 | const child = Bun.spawn([process.execPath, SCRIPT, ...args, "--json"], { | |
| 72 | env: { PATH: process.env["PATH"] ?? "", TMPDIR: process.env["TMPDIR"] ?? tmpdir(), APP_BACKUP_PASSPHRASE: passphrase }, | |
| 73 | stdin: "ignore", | |
| 74 | stdout: "pipe", | |
| 75 | stderr: "pipe", | |
| 76 | timeout: CHILD_TIMEOUT_MS, | |
| 77 | }); | |
| 78 | const [out, err] = await Promise.all([new Response(child.stdout).text(), new Response(child.stderr).text(), child.exited]); | |
| 79 | const last = out.trim().split("\n").pop() ?? ""; | |
| 80 | try { | |
| 81 | const parsed: unknown = JSON.parse(last); | |
| 82 | if (isProof(parsed)) return parsed; | |
| 83 | } catch { | |
| 84 | // not a proof line: the refusal is on stderr | |
| 85 | } | |
| 86 | const why = err.trim().split("\n").pop()?.replace(/^app-backup: /, "") ?? ""; | |
| 87 | return why === "" ? "the backup did not finish" : why; | |
| 88 | } catch (e) { | |
| 89 | return (e as Error).message; | |
| 90 | } | |
| 91 | } | |
| 92 | ||
| 93 | export async function sealApp(dataDir: string, appId: string, passphrase: string, now = new Date()): Promise<Proof | string> { | |
| 94 | try { | |
| 95 | const src = envDataDir(dataDir, appId, "production"); | |
| 96 | if (!existsSync(src)) return "this app has no production data yet"; | |
| 97 | if (bytesUnder(src, SEAL_CAP_BYTES) > SEAL_CAP_BYTES) { | |
| 98 | return `the data directory is over ${SEAL_CAP_BYTES / (1024 * 1024)} MB: back it up with scripts/app-backup.ts on the host`; | |
| 99 | } | |
| 100 | const d = dirFor(dataDir, appId); | |
| 101 | mkdirSync(d, { recursive: true, mode: 0o700 }); | |
| 102 | const stamp = now.toISOString().replace(/[:.]/g, "-"); | |
| 103 | return await runScript(["seal", src, join(d, `production-${stamp}.fab`)], passphrase); | |
| 104 | } catch (e) { | |
| 105 | return (e as Error).message; | |
| 106 | } | |
| 107 | } | |
| 108 | ||
| 109 | export async function proveApp(dataDir: string, appId: string, name: string, passphrase: string): Promise<Proof | string> { | |
| 110 | let scratch: string | undefined; | |
| 111 | try { | |
| 112 | const at = listBackups(dataDir, appId).find((b) => b.name === name); | |
| 113 | if (at === undefined) return "no such backup"; | |
| 114 | if (at.bytes > SEAL_CAP_BYTES * 2) return "this backup is too large to prove here: use scripts/app-backup.ts on the host"; | |
| 115 | scratch = mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "app-prove-")); | |
| 116 | return await runScript(["prove", join(dirFor(dataDir, appId), name), join(scratch, "restore")], passphrase); | |
| 117 | } catch (e) { | |
| 118 | return (e as Error).message; | |
| 119 | } finally { | |
| 120 | if (scratch !== undefined) rmSync(scratch, { recursive: true, force: true }); | |
| 121 | } | |
| 122 | } |
src/apps/backup_words.ts added +9
| @@ -0,0 +1,9 @@ | ||
| 1 | /** A proof as printed lines: the script and the terminal screen say the same thing. */ | |
| 2 | import type { Proof } from "./backup.ts"; | |
| 3 | ||
| 4 | export function proofLines(p: Proof): string[] { | |
| 5 | return [ | |
| 6 | `${p.ok ? "PROVEN" : "RED"}: ${p.files} files, ${p.bytes} bytes, ${p.tables} tables, ${p.rows} rows`, | |
| 7 | ...p.problems.map((m) => ` ${m.replace(/[\x00-\x1f]/g, "?")}`), | |
| 8 | ]; | |
| 9 | } |
src/tui/screens/appbackup.ts added +109
| @@ -0,0 +1,109 @@ | ||
| 1 | /** | |
| 2 | * An app's encrypted backup in the terminal, an admin's (#11701): the same | |
| 3 | * seal and restore proof as `scripts/app-backup.ts` (`apps/backup.ts`), from a | |
| 4 | * repository's `:` palette. `s` seals the app's production data and proves it | |
| 5 | * (and the data left behind) before keeping the file; `p` proves the backup | |
| 6 | * under the cursor. The passphrase is typed masked, held in a closure for one | |
| 7 | * call, and never stored, drawn, toasted or logged. | |
| 8 | */ | |
| 9 | ||
| 10 | import { listBackups, proveApp, sealApp, type BackupFile } from "../../apps/backup_door.ts"; | |
| 11 | import { proofLines } from "../../apps/backup_words.ts"; | |
| 12 | import { getAppByRepo } from "../../core/apps.ts"; | |
| 13 | import { canAdmin, type Repo } from "../../core/repos.ts"; | |
| 14 | import type { Ctx, Screen } from "../app/app.ts"; | |
| 15 | import { binding } from "../app/keys.ts"; | |
| 16 | import type { Command } from "../app/palette.ts"; | |
| 17 | import { style } from "../term/style.ts"; | |
| 18 | import { truncate } from "../term/width.ts"; | |
| 19 | import { fill, len, split } from "../ui/layout.ts"; | |
| 20 | import type { Session } from "../session.ts"; | |
| 21 | import { ListState, drawList, promptScreen, type ListRow } from "./common.ts"; | |
| 22 | ||
| 23 | /** The palette's "Back up data", for an admin of a repository that is an app. */ | |
| 24 | export function appBackupCommands(f: Session, ctx: Ctx, repo: Repo | undefined): Command[] { | |
| 25 | if (repo === undefined || f.user === undefined) return []; | |
| 26 | if (getAppByRepo(f.app.db, repo.id) === undefined || !canAdmin(f.app.db, repo, f.user)) return []; | |
| 27 | return [ | |
| 28 | { | |
| 29 | id: "app-backup", | |
| 30 | title: `${repo.ownerName}/${repo.name} — Back up data`, | |
| 31 | hint: "seal the app's data under your passphrase and prove it restores", | |
| 32 | run: () => ctx.push(appBackupScreen(f, ctx, repo)), | |
| 33 | }, | |
| 34 | ]; | |
| 35 | } | |
| 36 | ||
| 37 | export function appBackupScreen(f: Session, ctx: Ctx, repo: Repo): Screen { | |
| 38 | const state = new ListState(); | |
| 39 | const app = getAppByRepo(f.app.db, repo.id); | |
| 40 | const dataDir = f.app.config.dataDir; | |
| 41 | let rows: BackupFile[] = []; | |
| 42 | let note: string[] = []; | |
| 43 | const reload = (): void => { | |
| 44 | rows = app === undefined ? [] : listBackups(dataDir, app.id); | |
| 45 | state.clamp(rows.length); | |
| 46 | ctx.invalidate(); | |
| 47 | }; | |
| 48 | reload(); | |
| 49 | const allowed = (): boolean => f.user !== undefined && canAdmin(f.app.db, repo, f.user); | |
| 50 | const ask = (title: string, go: (pass: string) => Promise<void>): Screen => | |
| 51 | promptScreen(ctx, screen, { | |
| 52 | title, | |
| 53 | label: "your passphrase (12 characters at least; the forge never keeps it)", | |
| 54 | mask: true, | |
| 55 | onSubmit: (pass) => { | |
| 56 | if (pass.length < 12) return "The passphrase is under 12 characters."; | |
| 57 | return go(pass); | |
| 58 | }, | |
| 59 | }); | |
| 60 | const report = (r: Awaited<ReturnType<typeof sealApp>>): void => { | |
| 61 | note = typeof r === "string" ? [r] : proofLines(r); | |
| 62 | reload(); | |
| 63 | }; | |
| 64 | ||
| 65 | const screen: Screen = { | |
| 66 | title: `${repo.ownerName}/${repo.name} — Back up data`, | |
| 67 | keys: [binding(["s"], "seal and prove"), binding(["p"], "prove this one"), binding(["j", "k"], "move cursor")], | |
| 68 | render(area, s) { | |
| 69 | const c = ctx.theme.c; | |
| 70 | const shown = note.slice(0, 8); | |
| 71 | const [head, list, foot] = split(area, "v", [len(1), fill(1), len(Math.max(1, shown.length))]); | |
| 72 | s.text( | |
| 73 | head!.x, | |
| 74 | head!.y, | |
| 75 | truncate(rows.length === 0 ? "No backup yet. s seals production data." : "s seals again, p proves the one under the cursor.", head!.w), | |
| 76 | style(c.mutedFg, c.bg), | |
| 77 | head!.x + head!.w, | |
| 78 | ); | |
| 79 | const lines: ListRow[] = rows.map((r) => ({ title: r.name, meta: `${r.bytes} bytes · ${r.at.slice(0, 16).replace("T", " ")} UTC` })); | |
| 80 | drawList(s, list!, c, state, lines); | |
| 81 | shown.forEach((l, i) => s.text(foot!.x, foot!.y + i, truncate(l, foot!.w), style(c.fg, c.bg), foot!.x + foot!.w)); | |
| 82 | }, | |
| 83 | key(e) { | |
| 84 | if (state.key(e, rows.length)) return true; | |
| 85 | if (e.ctrl || e.alt || (e.key !== "s" && e.key !== "p")) return false; | |
| 86 | if (app === undefined || !allowed()) { | |
| 87 | ctx.toast("An admin of the app backs it up.", "danger"); | |
| 88 | return true; | |
| 89 | } | |
| 90 | if (e.key === "s") { | |
| 91 | ctx.push( | |
| 92 | ask("Seal this app's data", (pass) => | |
| 93 | sealApp(dataDir, app.id, pass).then(report), | |
| 94 | ), | |
| 95 | ); | |
| 96 | return true; | |
| 97 | } | |
| 98 | const at = rows[state.index]; | |
| 99 | if (at === undefined) return true; | |
| 100 | ctx.push( | |
| 101 | ask(`Prove ${at.name}`, (pass) => | |
| 102 | proveApp(dataDir, app.id, at.name, pass).then(report), | |
| 103 | ), | |
| 104 | ); | |
| 105 | return true; | |
| 106 | }, | |
| 107 | }; | |
| 108 | return screen; | |
| 109 | } |
src/tui/screens/workspace.ts +2
| @@ -166,6 +166,7 @@ import { displayCommands } from "./display.ts"; | ||
| 166 | 166 | import { iterationsCommands } from "./iterations.ts"; |
| 167 | 167 | import { labelsCommands } from "./labels.ts"; |
| 168 | 168 | import { appEnvCommands } from "./appenv.ts"; |
| 169 | import { appBackupCommands } from "./appbackup.ts"; | |
| 169 | 170 | import { variantsCommands } from "./variants.ts"; |
| 170 | 171 | import { autonomyCommands } from "./autonomy.ts"; |
| 171 | 172 | import { trafficCommands } from "./trafficview.ts"; |
| @@ -1992,6 +1993,7 @@ export function workspaceScreen( | ||
| 1992 | 1993 | ...iterationsCommands(f, ctx, repo), |
| 1993 | 1994 | ...displayCommands(f, ctx, repo), |
| 1994 | 1995 | ...appEnvCommands(f, ctx, repo), |
| 1996 | ...appBackupCommands(f, ctx, repo), | |
| 1995 | 1997 | ...variantsCommands(f, ctx, repo), |
| 1996 | 1998 | ...autonomyCommands(f, ctx, repo), |
| 1997 | 1999 | ...trafficCommands(f, ctx, repo), |
test/app-backup.test.ts added +195
| @@ -0,0 +1,195 @@ | ||
| 1 | import { Database } from "bun:sqlite"; | |
| 2 | import { describe, expect, test } from "bun:test"; | |
| 3 | import { existsSync, mkdirSync, mkdtempSync, readdirSync, symlinkSync, truncateSync, writeFileSync } from "node:fs"; | |
| 4 | import { tmpdir } from "node:os"; | |
| 5 | import { join } from "node:path"; | |
| 6 | import { openBackup, restoreProof, sealAndProve, sealParts, sealWithManifest, sourceDrift as sourceDriftOf } from "../src/apps/backup.ts"; | |
| 7 | import { bytesUnder, listBackups, proveApp, SEAL_CAP_BYTES, sealApp } from "../src/apps/backup_door.ts"; | |
| 8 | import { envDataDir } from "../src/apps/data.ts"; | |
| 9 | ||
| 10 | const PASS = "correct horse battery"; | |
| 11 | const tmp = () => mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "ab-")); | |
| 12 | ||
| 13 | function app(): string { | |
| 14 | const d = tmp(); | |
| 15 | const db = new Database(join(d, "app.db")); | |
| 16 | db.exec("PRAGMA journal_mode=WAL; CREATE TABLE orders(id INTEGER PRIMARY KEY, n TEXT)"); | |
| 17 | for (let i = 0; i < 7; i++) db.run("INSERT INTO orders(n) VALUES (?)", [`o${i}`]); | |
| 18 | mkdirSync(join(d, "uploads/a"), { recursive: true }); | |
| 19 | writeFileSync(join(d, "uploads/a/x.txt"), "hello"); | |
| 20 | return d; // db left open: the WAL holds writes, as in a running app | |
| 21 | } | |
| 22 | ||
| 23 | describe("app backup", () => { | |
| 24 | test("seals, restores and proves files and row counts", () => { | |
| 25 | const sealed = sealWithManifest(app(), PASS).sealed; | |
| 26 | const scratch = join(tmp(), "out"); | |
| 27 | const p = restoreProof(sealed, PASS, scratch); | |
| 28 | expect(p).toMatchObject({ ok: true, files: 2, tables: 1, rows: 7 }); | |
| 29 | }); | |
| 30 | test("holds no plaintext", () => { | |
| 31 | const sealed = sealWithManifest(app(), PASS).sealed; | |
| 32 | expect(sealed.includes("hello")).toBe(false); | |
| 33 | expect(sealed.includes("SQLite format")).toBe(false); | |
| 34 | }); | |
| 35 | test("wrong passphrase and tampering are refused alike, writing nothing", () => { | |
| 36 | const sealed = sealWithManifest(app(), PASS).sealed; | |
| 37 | const scratch = join(tmp(), "out"); | |
| 38 | expect(restoreProof(sealed, "another passphrase!", scratch)).toMatch(/wrong passphrase/); | |
| 39 | const bent = Buffer.from(sealed); | |
| 40 | bent[bent.length - 20] = bent[bent.length - 20]! ^ 1; | |
| 41 | expect(restoreProof(bent, PASS, scratch)).toMatch(/wrong passphrase/); | |
| 42 | expect(openBackup(Buffer.from("junk"), PASS)).toBe("not an app backup file"); | |
| 43 | }); | |
| 44 | test("a short passphrase is refused up front", () => { | |
| 45 | expect(() => sealWithManifest(app(), "short")).toThrow(/under 12/); | |
| 46 | }); | |
| 47 | test("refuses a scratch directory in use", () => { | |
| 48 | const sealed = sealWithManifest(app(), PASS).sealed; | |
| 49 | const busy = tmp(); | |
| 50 | writeFileSync(join(busy, "keep"), "x"); | |
| 51 | expect(restoreProof(sealed, PASS, busy)).toMatch(/not empty/); | |
| 52 | expect(readdirSync(busy)).toEqual(["keep"]); | |
| 53 | }); | |
| 54 | }); | |
| 55 | ||
| 56 | describe("the proof is against the manifest and the source", () => { | |
| 57 | test("bytes that disagree with the manifest are RED, with the problem named", () => { | |
| 58 | const d = app(); | |
| 59 | const { manifest } = sealWithManifest(d, PASS); | |
| 60 | const bodies = manifest.files.map((f) => Buffer.alloc(f.size, 65)); | |
| 61 | const p = restoreProof(sealParts(manifest, bodies, PASS), PASS, join(tmp(), "out")); | |
| 62 | expect(typeof p).toBe("object"); | |
| 63 | if (typeof p === "string") return; | |
| 64 | expect(p.ok).toBe(false); | |
| 65 | expect(p.problems.some((m) => /bytes differ from the backup's manifest/.test(m))).toBe(true); | |
| 66 | expect(p.problems.some((m) => /app.db: restored database does not open/.test(m))).toBe(true); | |
| 67 | }); | |
| 68 | test("an unsafe path in the manifest is refused and nothing is written outside", () => { | |
| 69 | const base = tmp(); | |
| 70 | const m = { | |
| 71 | files: [ | |
| 72 | { path: "../escape.txt", size: 1, sha256: "x" }, | |
| 73 | { path: "/abs.txt", size: 1, sha256: "x" }, | |
| 74 | ], | |
| 75 | tables: {}, | |
| 76 | }; | |
| 77 | const p = restoreProof(sealParts(m, [Buffer.from("a"), Buffer.from("b")], PASS), PASS, join(base, "out")); | |
| 78 | if (typeof p === "string") throw new Error(p); | |
| 79 | expect(p.ok).toBe(false); | |
| 80 | expect(p.problems.filter((x) => /unsafe path/.test(x)).length).toBe(2); | |
| 81 | expect(existsSync(join(base, "escape.txt"))).toBe(false); | |
| 82 | expect(existsSync("/abs.txt")).toBe(false); | |
| 83 | }); | |
| 84 | test("a truncated file is refused", () => { | |
| 85 | const sealed = sealWithManifest(app(), PASS).sealed; | |
| 86 | const scratch = join(tmp(), "out"); | |
| 87 | expect(restoreProof(sealed.subarray(0, sealed.length - 30), PASS, scratch)).toMatch(/wrong passphrase/); | |
| 88 | expect(restoreProof(sealed.subarray(0, 40), PASS, scratch)).toMatch(/wrong passphrase|not an app/); | |
| 89 | expect(existsSync(scratch)).toBe(false); | |
| 90 | }); | |
| 91 | test("a symlink is reported, not silently dropped, and the proof is RED", () => { | |
| 92 | const d = app(); | |
| 93 | symlinkSync("/etc/hostname", join(d, "link")); | |
| 94 | const { manifest } = sealWithManifest(d, PASS); | |
| 95 | expect(manifest.skipped).toEqual(["link"]); | |
| 96 | const p = restoreProof(sealWithManifest(d, PASS).sealed, PASS, join(tmp(), "out")); | |
| 97 | if (typeof p === "string") throw new Error(p); | |
| 98 | expect(p.ok).toBe(false); | |
| 99 | expect(p.problems[0]).toMatch(/^link: not archived/); | |
| 100 | }); | |
| 101 | test("sealAndProve: out-file kept on PROVEN, absent on RED, refused inside the data dir", () => { | |
| 102 | const d = app(); | |
| 103 | const out = join(tmp(), "ok.fab"); | |
| 104 | expect(sealAndProve(d, out, PASS).ok).toBe(true); | |
| 105 | expect(existsSync(out)).toBe(true); | |
| 106 | const outDir = tmp(); | |
| 107 | symlinkSync("/etc/hostname", join(d, "link")); | |
| 108 | const red = sealAndProve(d, join(outDir, "red.fab"), PASS); | |
| 109 | expect(red.ok).toBe(false); | |
| 110 | expect(readdirSync(outDir)).toEqual([]); | |
| 111 | expect(() => sealAndProve(d, join(d, "uploads", "in.fab"), PASS)).toThrow(/inside the data directory/); | |
| 112 | }); | |
| 113 | test("a source changed after the read is RED", () => { | |
| 114 | const d = app(); | |
| 115 | const m = sealWithManifest(d, PASS).manifest; | |
| 116 | writeFileSync(join(d, "uploads/a/x.txt"), "changed!"); | |
| 117 | expect(sourceDriftOf(d, m)).toEqual(["uploads/a/x.txt: changed in the data directory since the backup was read"]); | |
| 118 | }); | |
| 119 | }); | |
| 120 | ||
| 121 | describe("scripts/app-backup.ts", () => { | |
| 122 | test("prove restores a sealed file and exits 0; a wrong passphrase exits 1", async () => { | |
| 123 | const out = join(tmp(), "b.fab"); | |
| 124 | writeFileSync(out, sealWithManifest(app(), PASS).sealed); | |
| 125 | const run = async (pass: string, scratch: string) => { | |
| 126 | const p = Bun.spawn(["bun", "scripts/app-backup.ts", "prove", out, scratch], { | |
| 127 | env: { PATH: process.env["PATH"] ?? "", TMPDIR: process.env["TMPDIR"] ?? tmpdir(), APP_BACKUP_PASSPHRASE: pass }, | |
| 128 | stdout: "pipe", | |
| 129 | stderr: "pipe", | |
| 130 | }); | |
| 131 | return { code: await p.exited, out: await new Response(p.stdout).text(), err: await new Response(p.stderr).text() }; | |
| 132 | }; | |
| 133 | const ok = await run(PASS, join(tmp(), "r")); | |
| 134 | expect(ok.code).toBe(0); | |
| 135 | expect(ok.out).toMatch(/^PROVEN: 2 files/); | |
| 136 | const bad = await run("not the passphrase!", join(tmp(), "r2")); | |
| 137 | expect(bad.code).toBe(1); | |
| 138 | expect(bad.err).toMatch(/wrong passphrase/); | |
| 139 | }); | |
| 140 | test("seal writes the file and exits 0 with PROVEN; no passphrase exits 1", async () => { | |
| 141 | const d = app(); | |
| 142 | const out = join(tmp(), "b.fab"); | |
| 143 | const run = async (env: Record<string, string>) => { | |
| 144 | const p = Bun.spawn(["bun", "scripts/app-backup.ts", "seal", d, out], { | |
| 145 | env: { PATH: process.env["PATH"] ?? "", TMPDIR: process.env["TMPDIR"] ?? tmpdir(), ...env }, | |
| 146 | stdout: "pipe", | |
| 147 | stderr: "pipe", | |
| 148 | }); | |
| 149 | return { code: await p.exited, out: await new Response(p.stdout).text(), err: await new Response(p.stderr).text() }; | |
| 150 | }; | |
| 151 | const ok = await run({ APP_BACKUP_PASSPHRASE: PASS }); | |
| 152 | expect(ok.code).toBe(0); | |
| 153 | expect(ok.out).toMatch(/^PROVEN: 2 files/); | |
| 154 | const none = await run({}); | |
| 155 | expect(none.code).toBe(1); | |
| 156 | expect(none.err).toMatch(/no passphrase/); | |
| 157 | }); | |
| 158 | }); | |
| 159 | ||
| 160 | describe("src/apps/backup_door.ts", () => { | |
| 161 | const appData = (forge: string, appId: string): string => { | |
| 162 | const d = envDataDir(forge, appId, "production"); | |
| 163 | mkdirSync(d, { recursive: true }); | |
| 164 | return d; | |
| 165 | }; | |
| 166 | test("a data directory over the cap is refused before it is read, and nothing is written", async () => { | |
| 167 | const forge = tmp(); | |
| 168 | const d = appData(forge, "big"); | |
| 169 | mkdirSync(join(d, "deep"), { recursive: true }); | |
| 170 | writeFileSync(join(d, "deep", "huge.bin"), ""); | |
| 171 | truncateSync(join(d, "deep", "huge.bin"), SEAL_CAP_BYTES + 1); // sparse: no bytes on disk | |
| 172 | expect(bytesUnder(d, SEAL_CAP_BYTES)).toBeGreaterThan(SEAL_CAP_BYTES); | |
| 173 | const r = await sealApp(forge, "big", PASS); | |
| 174 | expect(r).toMatch(/over 64 MB/); | |
| 175 | expect(listBackups(forge, "big")).toEqual([]); | |
| 176 | }); | |
| 177 | test("under the cap it seals in a child process and proves", async () => { | |
| 178 | const forge = tmp(); | |
| 179 | writeFileSync(join(appData(forge, "small"), "x.txt"), "hello"); | |
| 180 | const r = await sealApp(forge, "small", PASS); | |
| 181 | expect(typeof r === "string" ? r : r.ok).toBe(true); | |
| 182 | const [b] = listBackups(forge, "small"); | |
| 183 | const p = await proveApp(forge, "small", b!.name, PASS); | |
| 184 | expect(typeof p === "string" ? p : p.files).toBe(1); | |
| 185 | expect(await proveApp(forge, "small", b!.name, "the wrong passphrase")).toMatch(/wrong passphrase/); | |
| 186 | }); | |
| 187 | test("prove answers a message, never a rejection, for a backup that will not open", async () => { | |
| 188 | const forge = tmp(); | |
| 189 | const d = join(forge, "app-backups", "bad"); | |
| 190 | mkdirSync(d, { recursive: true }); | |
| 191 | writeFileSync(join(d, "production-2026-01-01T00-00-00-000Z.fab"), "FAB1\nnot json\n"); | |
| 192 | expect(typeof (await proveApp(forge, "bad", "production-2026-01-01T00-00-00-000Z.fab", PASS))).toBe("string"); | |
| 193 | expect(await proveApp(forge, "bad", "../../etc/passwd", PASS)).toBe("no such backup"); | |
| 194 | }); | |
| 195 | }); |
test/spawn-pipes.test.ts +1
| @@ -16,6 +16,7 @@ import { join } from "node:path"; | ||
| 16 | 16 | * drained. Re-piping a stream that was ignored changes a site's entry. |
| 17 | 17 | */ |
| 18 | 18 | const PIPING: Record<string, { sites: string[]; reason: string }> = { |
| 19 | "src/apps/backup_door.ts": { sites: ["stdout+stderr"], reason: "both read to their end alongside the exit: stdout's last line is the proof, stderr the refusal" }, | |
| 19 | 20 | "src/apps/contain.ts": { sites: ["stderr"], reason: "stderr awaited with the exit; stdout ignored" }, |
| 20 | 21 | "src/apps/process.ts": { sites: ["stdout+stderr", "stdout+stderr"], reason: "pump() reads stdout and stderr of the app until exit" }, |
| 21 | 22 | "src/spawn/cell.ts": { sites: ["stdout+stderr"], reason: "drain() reads the child forge's stdout and stderr to its log until exit" }, |
test/spawn-sites.test.ts +2
| @@ -11,6 +11,8 @@ import { readFileSync } from "node:fs"; | ||
| 11 | 11 | import { join } from "node:path"; |
| 12 | 12 | |
| 13 | 13 | const SITES: Record<string, string> = { |
| 14 | "src/apps/backup_door.ts": | |
| 15 | "runs bun on the forge's own scripts/app-backup.ts with paths the forge made; the passphrase goes in env, never argv, and nothing in argv is text from a request", | |
| 14 | 16 | "src/apps/contain.ts": |
| 15 | 17 | "probes whether bwrap works: a fixed argv of namespace flags and /bin/true, nothing from a request", |
| 16 | 18 | "src/apps/process.ts": |
test/tui-appbackup.test.ts added +127
| @@ -0,0 +1,127 @@ | ||
| 1 | /** | |
| 2 | * The terminal's "Back up data" screen (src/tui/screens/appbackup.ts, #11701): | |
| 3 | * an admin seals and proves an app's production data; the passphrase is in no | |
| 4 | * frame and no log line; a non-admin is not offered it. | |
| 5 | */ | |
| 6 | import { afterAll, beforeAll, describe, expect, spyOn, test } from "bun:test"; | |
| 7 | import { mkdirSync, readdirSync, writeFileSync } from "node:fs"; | |
| 8 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 9 | import { tmpdir } from "node:os"; | |
| 10 | import { join } from "node:path"; | |
| 11 | import { newApp } from "../src/actions/apps.ts"; | |
| 12 | import { createApp, type App } from "../src/app.ts"; | |
| 13 | import { envDataDir } from "../src/apps/data.ts"; | |
| 14 | import { getAppByRepo } from "../src/core/apps.ts"; | |
| 15 | import { getRepo, type Repo } from "../src/core/repos.ts"; | |
| 16 | import { createUser, type User } from "../src/core/users.ts"; | |
| 17 | import { logLevel, setLogLevel, type LogLevel } from "../src/kernel/log.ts"; | |
| 18 | import { createForgeApp } from "../src/tui/forge.ts"; | |
| 19 | import { repoScreen } from "../src/tui/screens/repo.ts"; | |
| 20 | import { local } from "../src/tui/screens/where.ts"; | |
| 21 | import { headlessTerminal } from "../src/tui/term/terminal.ts"; | |
| 22 | import { settleApp } from "./helpers/tui.ts"; | |
| 23 | import { proveAll } from "./helpers/verified.ts"; | |
| 24 | ||
| 25 | const SECRET = "tr0ub4dor-and-three"; | |
| 26 | const open = [":", "Back up data", "\r"]; | |
| 27 | const DONE = /PROVEN|RED|wrong passphrase/; | |
| 28 | let dir: string; | |
| 29 | let app: App; | |
| 30 | let admin: User; | |
| 31 | let writer: User; | |
| 32 | let shop: Repo; | |
| 33 | let found: LogLevel; | |
| 34 | const frames: string[] = []; | |
| 35 | const logged: string[] = []; | |
| 36 | const spies: { mockRestore(): void }[] = []; | |
| 37 | const backups = () => { | |
| 38 | const d = join(dir, "app-backups", getAppByRepo(app.db, shop.id)!.id); | |
| 39 | try { return readdirSync(d); } catch { return []; } | |
| 40 | }; | |
| 41 | ||
| 42 | /** `until`: the seal and the proof run in a child process, so wait for its answer to be painted. */ | |
| 43 | async function drive(user: User, keys: string[], until?: RegExp): Promise<string> { | |
| 44 | const terminal = headlessTerminal(120, 40); | |
| 45 | const { app: tui, session } = createForgeApp( | |
| 46 | app, | |
| 47 | { sshUsername: user.username, remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user }, | |
| 48 | { terminal, exitHooks: false }, | |
| 49 | ); | |
| 50 | tui.start(); | |
| 51 | tui.ctx.push(repoScreen(session, tui.ctx, local(session, shop))); | |
| 52 | await settleApp(tui); | |
| 53 | for (const k of keys) { | |
| 54 | terminal.feed(k); | |
| 55 | await settleApp(tui); | |
| 56 | frames.push(tui.screen.toText()); | |
| 57 | } | |
| 58 | const deadline = Date.now() + 20_000; | |
| 59 | while (until !== undefined && !until.test(tui.screen.toText()) && Date.now() < deadline) { | |
| 60 | await Bun.sleep(25); | |
| 61 | await settleApp(tui); | |
| 62 | frames.push(tui.screen.toText()); | |
| 63 | } | |
| 64 | if (until !== undefined && !until.test(tui.screen.toText())) throw new Error(`timed out waiting for ${until}`); | |
| 65 | const text = tui.screen.toText(); | |
| 66 | tui.stop?.(); | |
| 67 | return text; | |
| 68 | } | |
| 69 | ||
| 70 | beforeAll(async () => { | |
| 71 | found = logLevel(); | |
| 72 | setLogLevel("debug"); | |
| 73 | for (const m of ["log", "info", "warn", "error", "debug"] as const) { | |
| 74 | spies.push(spyOn(console, m).mockImplementation((...a: unknown[]) => void logged.push(a.join(" ")))); | |
| 75 | } | |
| 76 | dir = await mkdtemp(join(tmpdir(), "forge-tui-appbackup-")); | |
| 77 | app = createApp({ dataDir: dir, port: 0 }); | |
| 78 | const mk = async (username: string, first = false) => { | |
| 79 | const r = await createUser(app.db, { | |
| 80 | origin: "seed", trusted: true, username, email: `${username}@forge.local`, | |
| 81 | password: "a-good-passphrase", adminIfFirst: first, | |
| 82 | }); | |
| 83 | if (!r.ok) throw new Error(r.error); | |
| 84 | return r.value; | |
| 85 | }; | |
| 86 | admin = await mk("alice", true); | |
| 87 | writer = await mk("walt"); | |
| 88 | proveAll(app); | |
| 89 | const made = await newApp(app, admin, { name: "shop", visibility: "public" }); | |
| 90 | if (!made.ok) throw new Error(String(made.error)); | |
| 91 | shop = getRepo(app.db, admin.username, "shop")!; | |
| 92 | const data = envDataDir(dir, getAppByRepo(app.db, shop.id)!.id, "production"); | |
| 93 | mkdirSync(join(data, "uploads"), { recursive: true }); | |
| 94 | writeFileSync(join(data, "uploads/a.txt"), "hello"); | |
| 95 | }); | |
| 96 | afterAll(async () => { | |
| 97 | for (const s of spies) s.mockRestore(); | |
| 98 | setLogLevel(found); | |
| 99 | app.db.close(); | |
| 100 | await rm(dir, { recursive: true, force: true }); | |
| 101 | }); | |
| 102 | ||
| 103 | describe("the terminal's back up screen", () => { | |
| 104 | test("a non-admin is not offered it", async () => { | |
| 105 | expect(await drive(writer, [":", "Back up data"])).not.toContain("— Back up data"); | |
| 106 | expect(await drive(admin, [":", "Back up data"])).toContain("— Back up data"); | |
| 107 | }); | |
| 108 | ||
| 109 | test("a short passphrase is refused and nothing is written", async () => { | |
| 110 | const text = await drive(admin, [...open, "s", "short", "\r"]); | |
| 111 | expect(text).toContain("under 12"); | |
| 112 | expect(backups()).toEqual([]); | |
| 113 | }); | |
| 114 | ||
| 115 | test("s seals and proves; p proves it again; the passphrase is nowhere", async () => { | |
| 116 | const sealed = await drive(admin, [...open, "s", SECRET, "\r"], DONE); | |
| 117 | expect(sealed).toContain("PROVEN: 1 files"); | |
| 118 | expect(backups().length).toBe(1); | |
| 119 | expect(frames.some((f) => f.includes("•".repeat(SECRET.length)))).toBe(true); | |
| 120 | const proved = await drive(admin, [...open, "p", SECRET, "\r"], DONE); | |
| 121 | expect(proved).toContain("PROVEN: 1 files"); | |
| 122 | const wrong = await drive(admin, [...open, "p", "another passphrase", "\r"], DONE); | |
| 123 | expect(wrong).toContain("wrong passphrase"); | |
| 124 | for (const f of frames) expect(f).not.toContain(SECRET); | |
| 125 | expect(logged.join("\n")).not.toContain(SECRET); | |
| 126 | }, 90_000); // three child processes seal and prove: past the 5 s default on a busy shard | |
| 127 | }); |
new above
offline
Synchronous, unbounded seal inside the forge process: cap the data directory's total bytes before sealing, or run the script as a child process with the passphrase in env.