Public: anyone can read this; the crew can change it.
An app's backup streams to disk and is not held in memory #13182
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/13111-an-app-s-backup-streams-to-disk-and-is-n → main 6 commits fork
6 files
changed +469
−147
docs/features/app-backup.md +24 −11
| @@ -12,15 +12,24 @@ bun scripts/app-backup.ts prove <out-file> <empty-scratch-dir> | ||
| 12 | 12 | |
| 13 | 13 | - `seal` checkpoints every SQLite file, archives the directory, encrypts it |
| 14 | 14 | (AES-256-GCM, key = scrypt of the passphrase, header authenticated) and |
| 15 | writes `<out-file>` with mode 0600. It then restores the file into a scratch | |
| 16 | directory of its own and prints `PROVEN: <files>, <bytes>, <tables>, <rows>`, | |
| 17 | exit 0, or `RED` and each mismatch, exit 1. Delete nothing on the old side | |
| 18 | until it says PROVEN. | |
| 15 | writes `<out-file>` with mode 0600. It then restores the file into a | |
| 16 | scratch directory of its own and prints `PROVEN: <files>, <bytes>, <tables>, | |
| 17 | <rows>`, exit 0, or `RED` and each mismatch, exit 1. Delete nothing on the | |
| 18 | old side until it says PROVEN. | |
| 19 | - The archive streams to disk as it is built (#13111): each file is hashed | |
| 20 | for the manifest, then passed through the cipher into the file 256 KB at a | |
| 21 | time. The proof checks the whole file's tag in one pass that keeps nothing, | |
| 22 | then decrypts into the scratch directory in a second. Memory stays flat | |
| 23 | whatever the data's size: a synthetic 500 MB app seals and proves in about | |
| 24 | 70 MB resident, most of it the runtime and scrypt | |
| 25 | (`test/app-backup.test.ts`). The format is unchanged, so older backups | |
| 26 | still open. | |
| 19 | 27 | - The proof reads back from disk every file's size and SHA-256, and every |
| 20 | 28 | SQLite table's row count, against the manifest taken at backup time |
| 21 | 29 | (the step-3 and step-5 comparison in `platform-app-move.md`). |
| 22 | - The seal is only kept when it is clean. It is written under a temporary | |
| 23 | name beside `<out-file>` and renamed on PROVEN; a RED proof leaves nothing. | |
| 30 | - The seal is only kept when it is clean. It is streamed to a temporary name | |
| 31 | beside `<out-file>` and renamed on PROVEN; a half-written archive or a RED | |
| 32 | proof leaves nothing that looks like a backup. | |
| 24 | 33 | An out-file inside the data directory is refused. |
| 25 | 34 | - The proof covers the data left behind too: after sealing, the data |
| 26 | 35 | directory is read again against the manifest, and a file that changed, |
| @@ -41,13 +50,17 @@ data left behind as above; `p` proves the file under the cursor. The screen | ||
| 41 | 50 | shows the same PROVEN or RED lines as the script. The file is encrypted under |
| 42 | 51 | a key only the owner holds; copy it off the forge to keep it. The seal and |
| 43 | 52 | the proof run `src/apps/backup_cli.ts` (the code behind `scripts/app-backup.ts`) in a child process (the passphrase in |
| 44 | its environment, never in argv), so the forge's event loop stays free; a | |
| 45 | production data directory over 64 MB (`SEAL_CAP_BYTES`) is refused before it | |
| 46 | is read, and the screen says to run the script on the host instead. Logic: | |
| 53 | its environment, never in argv), so the forge's event loop stays free. There | |
| 54 | is no size cap: before a byte is read, the screen checks the disk has room | |
| 55 | for the archive and its proof restore (the data's size each, added up when | |
| 56 | they share a disk, plus 64 MB to spare; a proof needs the backup's size in | |
| 57 | the temp directory) and otherwise refuses in one sentence: | |
| 58 | `not enough disk space: this needs about N MB and M MB are free`. Logic: | |
| 47 | 59 | `src/apps/backup_door.ts`; tests: `test/tui-appbackup.test.ts`. |
| 48 | 60 | |
| 49 | 61 | - Code: `src/apps/backup.ts`; tests: `test/app-backup.test.ts`. |
| 50 | 62 | |
| 51 | **Not done:** the archive is held in memory (the terminal caps it at 64 MB; | |
| 52 | the script on the host has no cap); no scheduled backup, no upload to a bucket, and no web page button; the | |
| 63 | **Not done:** the seal still reads each file twice (hash, then encrypt) and | |
| 64 | the proof decrypts twice (tag, then restore), trading time for flat memory; | |
| 65 | no scheduled backup, no upload to a bucket, and no web page button; the | |
| 53 | 66 | terminal seals production data only, not previews. |
scripts/gate.ts +1
| @@ -72,6 +72,7 @@ export const SLOW: readonly Exclusion[] = [ | ||
| 72 | 72 | { file: "test/release-journey.test.ts", reason: "walks the journey on a real spawned child forge with a stand-in crew, polling in real time; the passing walk takes 10 s under load", issue: 12876 }, |
| 73 | 73 | { file: "test/guide-journey.test.ts", reason: "runs the first-app guide end to end: signup, clone, push and a real deploy; 5.7 s a test on the gate box", issue: 12069 }, |
| 74 | 74 | { file: "test/apps-crew.test.ts", reason: "runs the crew against a real app, a restart and a push that cannot boot; 3.0 s a test on the gate box", issue: 1242 }, |
| 75 | { file: "test/app-backup.test.ts", reason: "seals and proves a 500 MB app to show memory stays under 100 MB while it streams; 11.3 s a test on the gate box", issue: 13111 }, | |
| 75 | 76 | { file: "test/apps-measure.test.ts", reason: "sends 601 reports through the app proxy to reach the rate limit; 3.7 s a test on the gate box", issue: 1242 }, |
| 76 | 77 | { file: "test/release-keeper-swap.test.ts", reason: "swaps two real stand-in processes in and one back under a hammering client, polling each to answer; 2.7 s a test on the gate box", issue: 12790 }, |
| 77 | 78 | { file: "test/apps.test.ts", reason: "deploys real apps to production and a preview, copying data; 2.0 s to 3.3 s a test on the gate box", issue: 12874 }, |
src/apps/backup.ts +273 −107
| @@ -9,8 +9,9 @@ | ||
| 9 | 9 | * base64url), then AES-256-GCM ciphertext ending in its 16-byte tag. The |
| 10 | 10 | * key is scrypt of the passphrase; the header is authenticated. Inside is a |
| 11 | 11 | * manifest (every file's path, size, SHA-256, and every SQLite table's row |
| 12 | * count) followed by the file bytes. Whole-archive in memory: meant for | |
| 13 | * app data of tens of megabytes, not gigabytes (`docs/features/app-backup.md`). | |
| 12 | * count) followed by the file bytes. Sealing streams each file through the | |
| 13 | * cipher into the out-file and opening streams it back out, 256 KB at a | |
| 14 | * time: memory stays flat whatever the data's size (`docs/features/app-backup.md`). | |
| 14 | 15 | */ |
| 15 | 16 | |
| 16 | 17 | import { Database } from "bun:sqlite"; |
| @@ -22,16 +23,20 @@ import { | ||
| 22 | 23 | scryptSync, |
| 23 | 24 | } from "node:crypto"; |
| 24 | 25 | import { |
| 26 | closeSync, | |
| 25 | 27 | existsSync, |
| 28 | fstatSync, | |
| 29 | fsyncSync, | |
| 26 | 30 | lstatSync, |
| 27 | 31 | mkdirSync, |
| 28 | readdirSync, | |
| 29 | 32 | mkdtempSync, |
| 30 | readFileSync, | |
| 33 | openSync, | |
| 34 | readdirSync, | |
| 35 | readSync, | |
| 31 | 36 | realpathSync, |
| 32 | 37 | renameSync, |
| 33 | 38 | rmSync, |
| 34 | writeFileSync, | |
| 39 | writeSync, | |
| 35 | 40 | } from "node:fs"; |
| 36 | 41 | import { tmpdir } from "node:os"; |
| 37 | 42 | import { basename, dirname, join, resolve, sep } from "node:path"; |
| @@ -63,7 +68,53 @@ export interface Proof { | ||
| 63 | 68 | problems: string[]; |
| 64 | 69 | } |
| 65 | 70 | |
| 66 | const sha = (b: Uint8Array): string => createHash("sha256").update(b).digest("hex"); | |
| 71 | const CHUNK = 1 << 18; | |
| 72 | ||
| 73 | /** Size and SHA-256 of a file, read a chunk at a time: the file is never whole in memory. */ | |
| 74 | export function hashFile(path: string): { size: number; sha256: string } { | |
| 75 | const fd = openSync(path, "r"); | |
| 76 | try { | |
| 77 | const h = createHash("sha256"); | |
| 78 | const buf = Buffer.allocUnsafe(CHUNK); | |
| 79 | let size = 0; | |
| 80 | for (let n; (n = readSync(fd, buf, 0, CHUNK, null)) > 0; size += n) h.update(buf.subarray(0, n)); | |
| 81 | return { size, sha256: h.digest("hex") }; | |
| 82 | } finally { | |
| 83 | closeSync(fd); | |
| 84 | } | |
| 85 | } | |
| 86 | ||
| 87 | /** | |
| 88 | * Random-access bytes of a sealed archive: a file on disk, or a buffer (tests, small callers). | |
| 89 | * A read is valid until the next. | |
| 90 | */ | |
| 91 | interface Source { | |
| 92 | size: number; | |
| 93 | read(pos: number, len: number): Buffer; | |
| 94 | close(): void; | |
| 95 | } | |
| 96 | function sourceOf(sealed: Uint8Array | string): Source { | |
| 97 | if (typeof sealed !== "string") { | |
| 98 | const b = Buffer.from(sealed.buffer, sealed.byteOffset, sealed.byteLength); | |
| 99 | return { size: b.length, read: (pos, len) => b.subarray(pos, pos + len), close: () => {} }; | |
| 100 | } | |
| 101 | const fd = openSync(sealed, "r"); | |
| 102 | const reused = Buffer.allocUnsafe(CHUNK); | |
| 103 | return { | |
| 104 | size: fstatSync(fd).size, | |
| 105 | read: (pos, len) => { | |
| 106 | const out = len <= CHUNK ? reused.subarray(0, len) : Buffer.allocUnsafe(len); | |
| 107 | let got = 0; | |
| 108 | while (got < len) { | |
| 109 | const n = readSync(fd, out, got, len - got, pos + got); | |
| 110 | if (n === 0) break; | |
| 111 | got += n; | |
| 112 | } | |
| 113 | return out.subarray(0, got); | |
| 114 | }, | |
| 115 | close: () => closeSync(fd), | |
| 116 | }; | |
| 117 | } | |
| 67 | 118 | |
| 68 | 119 | export function passphraseProblem(p: string | undefined): string | undefined { |
| 69 | 120 | if (p === undefined || p === "") return "no passphrase: set APP_BACKUP_PASSPHRASE"; |
| @@ -106,154 +157,270 @@ export function tableCounts(path: string): Record<string, number> | undefined { | ||
| 106 | 157 | } |
| 107 | 158 | } |
| 108 | 159 | |
| 109 | /** Seal a data directory. Throws a message (not a bug) for an unusable passphrase or directory. */ | |
| 110 | export function sealWithManifest(dir: string, passphrase: string): { sealed: Buffer; manifest: Manifest } { | |
| 160 | const headerFor = (salt: Buffer, iv: Buffer): Buffer => | |
| 161 | Buffer.from(`${JSON.stringify({ v: 1, kdf: "scrypt", salt: salt.toString("base64url"), iv: iv.toString("base64url") })}\n`); | |
| 162 | ||
| 163 | const startsSqlite = (path: string): boolean => { | |
| 164 | const fd = openSync(path, "r"); | |
| 165 | try { | |
| 166 | const b = Buffer.alloc(16); | |
| 167 | return readSync(fd, b, 0, 16, 0) === 16 && isSqlite(b); | |
| 168 | } finally { | |
| 169 | closeSync(fd); | |
| 170 | } | |
| 171 | }; | |
| 172 | ||
| 173 | /** | |
| 174 | * Seal a data directory into `out`, written as it is built: each file is | |
| 175 | * hashed for the manifest, then streamed a chunk at a time through the cipher | |
| 176 | * into `out`, so memory stays flat whatever the data's size. A file that | |
| 177 | * changes between the two reads is written at its manifest size and the | |
| 178 | * proof calls it out. `out` is written as given; `sealAndProve` names it | |
| 179 | * `.part` and renames it only on PROVEN. Throws a message for unusable input. | |
| 180 | */ | |
| 181 | export function sealToFile(dir: string, out: string, passphrase: string): Manifest { | |
| 111 | 182 | const bad = passphraseProblem(passphrase); |
| 112 | 183 | if (bad !== undefined) throw new Error(bad); |
| 113 | 184 | if (!existsSync(dir) || !lstatSync(dir).isDirectory()) throw new Error("the data directory does not exist"); |
| 114 | 185 | quiesceSqlite(dir); |
| 115 | 186 | const found = walk(dir); |
| 116 | 187 | const manifest: Manifest = { files: [], tables: {}, skipped: found.skipped }; |
| 117 | const bodies: Buffer[] = []; | |
| 118 | 188 | for (const path of found.files) { |
| 119 | const bytes = readFileSync(join(dir, path)); | |
| 120 | manifest.files.push({ path, size: bytes.length, sha256: sha(bytes) }); | |
| 121 | bodies.push(bytes); | |
| 122 | if (isSqlite(bytes)) { | |
| 123 | const counts = tableCounts(join(dir, path)); | |
| 189 | const full = join(dir, path); | |
| 190 | manifest.files.push({ path, ...hashFile(full) }); | |
| 191 | if (startsSqlite(full)) { | |
| 192 | const counts = tableCounts(full); | |
| 124 | 193 | if (counts !== undefined) manifest.tables[path] = counts; |
| 125 | 194 | } |
| 126 | 195 | } |
| 127 | return { sealed: sealParts(manifest, bodies, passphrase), manifest }; | |
| 128 | } | |
| 129 | ||
| 130 | /** Encrypt a manifest and its file bodies as given; `sealWithManifest` is the honest caller. */ | |
| 131 | export function sealParts(manifest: Manifest, bodies: Buffer[], passphrase: string): Buffer { | |
| 132 | const m = Buffer.from(JSON.stringify(manifest)); | |
| 133 | const len = Buffer.alloc(4); | |
| 134 | len.writeUInt32BE(m.length); | |
| 135 | const plain = Buffer.concat([len, m, ...bodies]); | |
| 136 | ||
| 137 | 196 | const salt = randomBytes(16); |
| 138 | 197 | const iv = randomBytes(12); |
| 139 | const header = Buffer.from( | |
| 140 | `${JSON.stringify({ v: 1, kdf: "scrypt", salt: salt.toString("base64url"), iv: iv.toString("base64url") })}\n`, | |
| 141 | ); | |
| 198 | const header = headerFor(salt, iv); | |
| 142 | 199 | const cipher = createCipheriv("aes-256-gcm", scryptSync(passphrase, salt, 32, SCRYPT), iv, { authTagLength: 16 }); |
| 143 | 200 | cipher.setAAD(header); |
| 144 | return Buffer.concat([Buffer.from(MAGIC), header, cipher.update(plain), cipher.final(), cipher.getAuthTag()]); | |
| 201 | const fd = openSync(out, "w", 0o600); | |
| 202 | try { | |
| 203 | const put = (b: Buffer): void => { | |
| 204 | for (let at = 0; at < b.length; ) at += writeSync(fd, b, at); | |
| 205 | }; | |
| 206 | put(Buffer.concat([Buffer.from(MAGIC), header])); | |
| 207 | const m = Buffer.from(JSON.stringify(manifest)); | |
| 208 | const len = Buffer.alloc(4); | |
| 209 | len.writeUInt32BE(m.length); | |
| 210 | put(cipher.update(Buffer.concat([len, m]))); | |
| 211 | const buf = Buffer.allocUnsafe(CHUNK); | |
| 212 | for (const f of manifest.files) { | |
| 213 | const src = openSync(join(dir, f.path), "r"); | |
| 214 | try { | |
| 215 | let left = f.size; | |
| 216 | while (left > 0) { | |
| 217 | const n = readSync(src, buf, 0, Math.min(CHUNK, left), null); | |
| 218 | if (n === 0) { | |
| 219 | put(cipher.update(Buffer.alloc(left))); // shrank since hashed: keep the layout, the proof says so | |
| 220 | break; | |
| 221 | } | |
| 222 | put(cipher.update(buf.subarray(0, n))); | |
| 223 | left -= n; | |
| 224 | } | |
| 225 | } finally { | |
| 226 | closeSync(src); | |
| 227 | } | |
| 228 | } | |
| 229 | put(cipher.final()); | |
| 230 | put(cipher.getAuthTag()); | |
| 231 | fsyncSync(fd); | |
| 232 | } finally { | |
| 233 | closeSync(fd); | |
| 234 | } | |
| 235 | return manifest; | |
| 145 | 236 | } |
| 146 | 237 | |
| 147 | /** The manifest and file bytes, or a message: a wrong passphrase and a damaged file say the same thing. */ | |
| 148 | export function openBackup(sealed: Uint8Array, passphrase: string): { manifest: Manifest; bodies: Buffer[] } | string { | |
| 149 | const refuse = "cannot open this backup: wrong passphrase, or the file is damaged"; | |
| 150 | try { | |
| 151 | const buf = Buffer.from(sealed); | |
| 152 | if (buf.subarray(0, MAGIC.length).toString("latin1") !== MAGIC) return "not an app backup file"; | |
| 153 | const nl = buf.indexOf(10, MAGIC.length); | |
| 154 | if (nl < 0 || nl > 512 || buf.length < nl + 1 + 16) return refuse; | |
| 155 | const header = buf.subarray(MAGIC.length, nl + 1); | |
| 156 | const h = JSON.parse(header.toString()) as Record<string, unknown>; | |
| 157 | if (h["v"] !== 1 || h["kdf"] !== "scrypt") return refuse; | |
| 158 | const salt = Buffer.from(String(h["salt"]), "base64url"); | |
| 159 | const iv = Buffer.from(String(h["iv"]), "base64url"); | |
| 160 | if (salt.length !== 16 || iv.length !== 12) return refuse; | |
| 161 | const body = buf.subarray(nl + 1); | |
| 162 | const d = createDecipheriv("aes-256-gcm", scryptSync(passphrase, salt, 32, SCRYPT), iv, { authTagLength: 16 }); | |
| 238 | const REFUSE = "cannot open this backup: wrong passphrase, or the file is damaged"; | |
| 239 | ||
| 240 | /** | |
| 241 | * The archive's plaintext, decrypted a chunk at a time after the whole file's tag has been | |
| 242 | * checked. | |
| 243 | */ | |
| 244 | interface Plain { | |
| 245 | /** Up to `n` bytes, fewer only at the end. */ | |
| 246 | take(n: number): Buffer; | |
| 247 | /** Pass the next `n` bytes to `sink` in chunks; returns how many there were. */ | |
| 248 | pipe(n: number, sink: (b: Buffer) => void): number; | |
| 249 | } | |
| 250 | ||
| 251 | /** | |
| 252 | * Check the archive's header and its GCM tag over every byte (a first pass | |
| 253 | * that keeps nothing), then return a reader of the plaintext. Nothing is | |
| 254 | * handed out before the whole file is authenticated. | |
| 255 | */ | |
| 256 | function openPlain(src: Source, passphrase: string): Plain | string { | |
| 257 | if (src.read(0, MAGIC.length).toString("latin1") !== MAGIC) return "not an app backup file"; | |
| 258 | const head = src.read(MAGIC.length, 512); | |
| 259 | const nl = head.indexOf(10); | |
| 260 | if (nl < 0) return REFUSE; | |
| 261 | const header = Buffer.from(head.subarray(0, nl + 1)); | |
| 262 | const bodyAt = MAGIC.length + nl + 1; | |
| 263 | if (src.size < bodyAt + 16) return REFUSE; | |
| 264 | const h = JSON.parse(header.toString()) as Record<string, unknown>; | |
| 265 | if (h["v"] !== 1 || h["kdf"] !== "scrypt") return REFUSE; | |
| 266 | const salt = Buffer.from(String(h["salt"]), "base64url"); | |
| 267 | const iv = Buffer.from(String(h["iv"]), "base64url"); | |
| 268 | if (salt.length !== 16 || iv.length !== 12) return REFUSE; | |
| 269 | const key = scryptSync(passphrase, salt, 32, SCRYPT); | |
| 270 | const end = src.size - 16; | |
| 271 | const tag = Buffer.from(src.read(end, 16)); | |
| 272 | const decipher = () => { | |
| 273 | const d = createDecipheriv("aes-256-gcm", key, iv, { authTagLength: 16 }); | |
| 163 | 274 | d.setAAD(header); |
| 164 | d.setAuthTag(body.subarray(body.length - 16)); | |
| 165 | const plain = Buffer.concat([d.update(body.subarray(0, body.length - 16)), d.final()]); | |
| 166 | const mlen = plain.readUInt32BE(0); | |
| 167 | const manifest = JSON.parse(plain.subarray(4, 4 + mlen).toString()) as Manifest; | |
| 168 | let at = 4 + mlen; | |
| 169 | const bodies: Buffer[] = []; | |
| 170 | for (const f of manifest.files) { | |
| 171 | bodies.push(plain.subarray(at, at + f.size)); | |
| 172 | at += f.size; | |
| 275 | d.setAuthTag(tag); | |
| 276 | return d; | |
| 277 | }; | |
| 278 | const check = decipher(); | |
| 279 | for (let at = bodyAt; at < end; at += CHUNK) check.update(src.read(at, Math.min(CHUNK, end - at))); | |
| 280 | check.final(); // throws on a wrong passphrase or a damaged file | |
| 281 | const d = decipher(); | |
| 282 | let at = bodyAt; | |
| 283 | let held = Buffer.alloc(0); | |
| 284 | const fill = (): boolean => { | |
| 285 | if (held.length > 0) return true; | |
| 286 | if (at >= end) return false; | |
| 287 | held = d.update(src.read(at, Math.min(CHUNK, end - at))); | |
| 288 | at += CHUNK; | |
| 289 | return true; | |
| 290 | }; | |
| 291 | const pipe = (n: number, sink: (b: Buffer) => void): number => { | |
| 292 | let got = 0; | |
| 293 | while (got < n && fill()) { | |
| 294 | const part = held.subarray(0, n - got); | |
| 295 | held = held.subarray(part.length); | |
| 296 | got += part.length; | |
| 297 | sink(part); | |
| 173 | 298 | } |
| 174 | return { manifest, bodies }; | |
| 299 | return got; | |
| 300 | }; | |
| 301 | return { | |
| 302 | pipe, | |
| 303 | take: (n) => { | |
| 304 | const parts: Buffer[] = []; | |
| 305 | pipe(n, (b) => parts.push(Buffer.from(b))); | |
| 306 | return Buffer.concat(parts); | |
| 307 | }, | |
| 308 | }; | |
| 309 | } | |
| 310 | ||
| 311 | function withPlain<T>(sealed: Uint8Array | string, passphrase: string, use: (manifest: Manifest, plain: Plain) => T): T | string { | |
| 312 | let src: Source; | |
| 313 | try { | |
| 314 | src = sourceOf(sealed); | |
| 175 | 315 | } catch { |
| 176 | return refuse; | |
| 316 | return "cannot read this backup file"; | |
| 317 | } | |
| 318 | try { | |
| 319 | let plain: Plain | string; | |
| 320 | let manifest: Manifest; | |
| 321 | try { | |
| 322 | plain = openPlain(src, passphrase); | |
| 323 | if (typeof plain === "string") return plain; | |
| 324 | const len = plain.take(4); | |
| 325 | if (len.length !== 4) return REFUSE; | |
| 326 | manifest = JSON.parse(plain.take(len.readUInt32BE(0)).toString()) as Manifest; | |
| 327 | if (!Array.isArray(manifest.files) || typeof manifest.tables !== "object" || manifest.tables === null) return REFUSE; | |
| 328 | } catch { | |
| 329 | return REFUSE; | |
| 330 | } | |
| 331 | return use(manifest, plain); | |
| 332 | } finally { | |
| 333 | src.close(); | |
| 177 | 334 | } |
| 178 | 335 | } |
| 179 | 336 | |
| 180 | 337 | const safeRel = (p: string): boolean => |
| 181 | p !== "" && !p.startsWith("/") && !p.includes("\0") && !p.split("/").some((s) => s === ".." || s === "." || s === ""); | |
| 338 | typeof p === "string" && p !== "" && !p.startsWith("/") && !p.includes("\0") && !p.split("/").some((s) => s === ".." || s === "." || s === ""); | |
| 182 | 339 | |
| 183 | 340 | /** |
| 184 | * Restore a backup into `scratch` (must not exist or be empty) and prove it: | |
| 185 | * every file's SHA-256 and size read back from disk, every SQLite table's row | |
| 186 | * count read back from the restored database. Returns a message when it cannot | |
| 187 | * even try (wrong passphrase, scratch in use). | |
| 341 | * Restore a backup (a file path, streamed; or bytes) into `scratch` (must not | |
| 342 | * exist or be empty) and prove it: every file's SHA-256 and size read back | |
| 343 | * from disk, every SQLite table's row count read back from the restored | |
| 344 | * database. Returns a message when it cannot even try (wrong passphrase, | |
| 345 | * damaged file, scratch in use); then nothing has been written. | |
| 188 | 346 | */ |
| 189 | export function restoreProof(sealed: Uint8Array, passphrase: string, scratch: string): Proof | string { | |
| 347 | export function restoreProof(sealed: Uint8Array | string, passphrase: string, scratch: string): Proof | string { | |
| 190 | 348 | const bad = passphraseProblem(passphrase); |
| 191 | 349 | if (bad !== undefined) return bad; |
| 192 | 350 | if (existsSync(scratch) && readdirSync(scratch).length > 0) return "the scratch directory is not empty"; |
| 193 | const opened = openBackup(sealed, passphrase); | |
| 194 | if (typeof opened === "string") return opened; | |
| 195 | const { manifest, bodies } = opened; | |
| 196 | const root = resolve(scratch); | |
| 197 | mkdirSync(root, { recursive: true }); | |
| 198 | const problems: string[] = []; | |
| 199 | for (const k of manifest.skipped ?? []) | |
| 200 | problems.push(`${k}: not archived (a symlink or special file); the backup does not cover it`); | |
| 201 | let bytes = 0; | |
| 202 | manifest.files.forEach((f, i) => { | |
| 203 | const target = resolve(root, f.path); | |
| 204 | if (!safeRel(f.path) || !target.startsWith(root + sep)) { | |
| 205 | problems.push(`refused an unsafe path: ${JSON.stringify(f.path)}`); | |
| 206 | return; | |
| 351 | return withPlain(sealed, passphrase, (manifest, plain) => { | |
| 352 | const root = resolve(scratch); | |
| 353 | mkdirSync(root, { recursive: true }); | |
| 354 | const problems: string[] = []; | |
| 355 | for (const k of manifest.skipped ?? []) | |
| 356 | problems.push(`${k}: not archived (a symlink or special file); the backup does not cover it`); | |
| 357 | const inside = (p: string): boolean => safeRel(p) && resolve(root, p).startsWith(root + sep); | |
| 358 | for (const f of manifest.files) { | |
| 359 | if (!inside(f.path)) { | |
| 360 | problems.push(`refused an unsafe path: ${JSON.stringify(f.path)}`); | |
| 361 | plain.pipe(f.size, () => {}); | |
| 362 | continue; | |
| 363 | } | |
| 364 | const target = resolve(root, f.path); | |
| 365 | mkdirSync(dirname(target), { recursive: true }); | |
| 366 | const fd = openSync(target, "w", 0o600); | |
| 367 | try { | |
| 368 | plain.pipe(f.size, (b) => { | |
| 369 | for (let at = 0; at < b.length; ) at += writeSync(fd, b, at); | |
| 370 | }); | |
| 371 | } finally { | |
| 372 | closeSync(fd); | |
| 373 | } | |
| 207 | 374 | } |
| 208 | mkdirSync(dirname(target), { recursive: true }); | |
| 209 | writeFileSync(target, bodies[i]!); | |
| 375 | let bytes = 0; | |
| 376 | for (const f of manifest.files) { | |
| 377 | if (!inside(f.path)) continue; | |
| 378 | const back = hashFile(resolve(root, f.path)); | |
| 379 | bytes += back.size; | |
| 380 | if (back.size !== f.size || back.sha256 !== f.sha256) problems.push(`${f.path}: bytes differ from the backup's manifest`); | |
| 381 | } | |
| 382 | let tables = 0; | |
| 383 | let rows = 0; | |
| 384 | for (const [file, expected] of Object.entries(manifest.tables)) { | |
| 385 | if (!inside(file)) continue; | |
| 386 | const got = tableCounts(join(root, file)); | |
| 387 | for (const [t, n] of Object.entries(expected)) { | |
| 388 | tables++; | |
| 389 | rows += n; | |
| 390 | if (got === undefined) problems.push(`${file}: restored database does not open`); | |
| 391 | else if (got[t] !== n) problems.push(`${file} ${t}: ${n} rows backed up, ${got[t] ?? "no table"} restored`); | |
| 392 | } | |
| 393 | } | |
| 394 | return { ok: problems.length === 0, files: manifest.files.length, bytes, tables, rows, problems }; | |
| 210 | 395 | }); |
| 211 | for (const f of manifest.files) { | |
| 212 | const target = resolve(root, f.path); | |
| 213 | if (!safeRel(f.path) || !target.startsWith(root + sep)) continue; | |
| 214 | const back = readFileSync(target); | |
| 215 | bytes += back.length; | |
| 216 | if (back.length !== f.size || sha(back) !== f.sha256) problems.push(`${f.path}: bytes differ from the backup's manifest`); | |
| 217 | } | |
| 218 | let tables = 0; | |
| 219 | let rows = 0; | |
| 220 | for (const [file, expected] of Object.entries(manifest.tables)) { | |
| 221 | if (!safeRel(file)) continue; | |
| 222 | const got = tableCounts(join(root, file)); | |
| 223 | for (const [t, n] of Object.entries(expected)) { | |
| 224 | tables++; | |
| 225 | rows += n; | |
| 226 | if (got === undefined) problems.push(`${file}: restored database does not open`); | |
| 227 | else if (got[t] !== n) problems.push(`${file} ${t}: ${n} rows backed up, ${got[t] ?? "no table"} restored`); | |
| 228 | } | |
| 229 | } | |
| 230 | return { ok: problems.length === 0, files: manifest.files.length, bytes, tables, rows, problems }; | |
| 231 | 396 | } |
| 232 | 397 | |
| 233 | /** The data directory against the manifest taken from it: what changed since, and what was left out. */ | |
| 398 | /** | |
| 399 | * The data directory against the manifest taken from it: what changed since, and what was left out. | |
| 400 | */ | |
| 234 | 401 | export function sourceDrift(dir: string, manifest: Manifest): string[] { |
| 235 | 402 | const problems: string[] = []; |
| 236 | 403 | const now = walk(dir); |
| 237 | 404 | const had = new Set(manifest.files.map((f) => f.path)); |
| 238 | 405 | for (const p of now.files) if (!had.has(p)) problems.push(`${p}: appeared in the data directory after the backup was read`); |
| 239 | 406 | for (const f of manifest.files) { |
| 240 | let back: Buffer; | |
| 407 | let back: { size: number; sha256: string }; | |
| 241 | 408 | try { |
| 242 | back = readFileSync(join(dir, f.path)); | |
| 409 | back = hashFile(join(dir, f.path)); | |
| 243 | 410 | } catch { |
| 244 | 411 | problems.push(`${f.path}: gone from the data directory since the backup was read`); |
| 245 | 412 | continue; |
| 246 | 413 | } |
| 247 | if (back.length !== f.size || sha(back) !== f.sha256) problems.push(`${f.path}: changed in the data directory since the backup was read`); | |
| 414 | if (back.size !== f.size || back.sha256 !== f.sha256) problems.push(`${f.path}: changed in the data directory since the backup was read`); | |
| 248 | 415 | } |
| 249 | 416 | return problems; |
| 250 | 417 | } |
| 251 | 418 | |
| 252 | 419 | /** |
| 253 | 420 | * Seal `dir` into `out`, prove the result and the data left behind, and keep |
| 254 | * the file only when both are clean: it is written under a temporary name in | |
| 255 | * the same directory and renamed on PROVEN, so a RED proof leaves no backup | |
| 256 | * that looks like one. Throws a message for unusable input. | |
| 421 | * the file only when both are clean: it is streamed to a temporary name in | |
| 422 | * the same directory and renamed on PROVEN, so a half-written archive or a RED | |
| 423 | * proof leaves no backup that looks like one. Throws a message for unusable input. | |
| 257 | 424 | */ |
| 258 | 425 | export function sealAndProve(dir: string, out: string, passphrase: string): Proof { |
| 259 | 426 | const bad = passphraseProblem(passphrase); |
| @@ -263,12 +430,11 @@ export function sealAndProve(dir: string, out: string, passphrase: string): Proo | ||
| 263 | 430 | const target = join(outDir, basename(out)); |
| 264 | 431 | const root = realpathSync(dir); |
| 265 | 432 | if (target === root || target.startsWith(root + sep)) throw new Error("the out-file is inside the data directory"); |
| 266 | const { sealed, manifest } = sealWithManifest(dir, passphrase); | |
| 267 | 433 | const part = join(outDir, `.${basename(out)}.${randomBytes(6).toString("hex")}.part`); |
| 268 | 434 | const scratch = mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "app-backup-")); |
| 269 | 435 | try { |
| 270 | writeFileSync(part, sealed, { mode: 0o600 }); | |
| 271 | const p = restoreProof(readFileSync(part), passphrase, join(scratch, "restore")); | |
| 436 | const manifest = sealToFile(dir, part, passphrase); | |
| 437 | const p = restoreProof(part, passphrase, join(scratch, "restore")); | |
| 272 | 438 | if (typeof p === "string") throw new Error(p); |
| 273 | 439 | p.problems.push(...sourceDrift(dir, manifest)); |
| 274 | 440 | p.ok = p.problems.length === 0; |
src/apps/backup_cli.ts +3 −3
| @@ -1,5 +1,6 @@ | ||
| 1 | 1 | /** |
| 2 | * An app's encrypted backup and its restore proof, as a command line: the owner runs it, and so does the terminal's door. | |
| 2 | * An app's encrypted backup and its restore proof, as a command line: the owner runs it, and so | |
| 3 | * does the terminal's door. | |
| 3 | 4 | * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts seal <data-dir> <out-file> |
| 4 | 5 | * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts prove <backup-file> <empty-scratch-dir> |
| 5 | 6 | * A trailing `--json` prints the proof as one JSON line in place of the |
| @@ -10,7 +11,6 @@ | ||
| 10 | 11 | * a child process and the release image ships src/ alone; scripts/app-backup.ts |
| 11 | 12 | * is the owner's entry to it. See docs/features/app-backup.md. |
| 12 | 13 | */ |
| 13 | import { readFileSync } from "node:fs"; | |
| 14 | 14 | import { restoreProof, sealAndProve, type Proof } from "./backup.ts"; |
| 15 | 15 | import { proofLines } from "./backup_words.ts"; |
| 16 | 16 | |
| @@ -29,7 +29,7 @@ try { | ||
| 29 | 29 | show(p); |
| 30 | 30 | process.exit(p.ok ? 0 : 1); |
| 31 | 31 | } else if (cmd === "prove" && a && b) { |
| 32 | const p = restoreProof(readFileSync(a), pass, b); | |
| 32 | const p = restoreProof(a, pass, b); | |
| 33 | 33 | if (typeof p === "string") throw new Error(p); |
| 34 | 34 | show(p); |
| 35 | 35 | process.exit(p.ok ? 0 : 1); |
src/apps/backup_door.ts +71 −16
| @@ -2,18 +2,23 @@ | ||
| 2 | 2 | * What the terminal's "Back up data" screen does for one app: seal its |
| 3 | 3 | * production data under the owner's passphrase into `<data>/app-backups/<appId>/`, |
| 4 | 4 | * list what is there, and prove one file. The work runs in a child process |
| 5 | * (`src/apps/backup_cli.ts`, the code behind `scripts/app-backup.ts`), so the forge's event loop stays free; the | |
| 6 | * passphrase goes to it in the environment, never in argv, and is never | |
| 7 | * stored, logged or returned. A data directory over `SEAL_CAP_BYTES` is | |
| 8 | * refused before a byte of it is read: the archive is held whole in memory. | |
| 5 | * (`src/apps/backup_cli.ts`, the code behind `scripts/app-backup.ts`), so the forge's event loop | |
| 6 | * stays free; the passphrase goes to it in the environment, never in argv, and is never stored, | |
| 7 | * logged or returned. The archive streams to disk, so the only limit is room for it: a seal or a | |
| 8 | * proof that would not fit on the disk is refused in one sentence before a byte is read | |
| 9 | * (`spaceProblem`). | |
| 9 | 10 | */ |
| 10 | import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statSync } from "node:fs"; | |
| 11 | import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statfsSync, statSync } from "node:fs"; | |
| 11 | 12 | import { tmpdir } from "node:os"; |
| 12 | 13 | import { join } from "node:path"; |
| 13 | 14 | import type { Proof } from "./backup.ts"; |
| 14 | 15 | import { envDataDir } from "./data.ts"; |
| 15 | 16 | |
| 16 | export const SEAL_CAP_BYTES = 64 * 1024 * 1024; | |
| 17 | /** | |
| 18 | * Room kept free beyond the archive and its proof, so a backup never fills the disk to the last | |
| 19 | * byte. | |
| 20 | */ | |
| 21 | export const SPACE_MARGIN_BYTES = 64 * 1024 * 1024; | |
| 17 | 22 | const SCRIPT = new URL("./backup_cli.ts", import.meta.url).pathname; |
| 18 | 23 | const CHILD_TIMEOUT_MS = 10 * 60_000; |
| 19 | 24 | |
| @@ -38,8 +43,39 @@ export function listBackups(dataDir: string, appId: string): BackupFile[] { | ||
| 38 | 43 | }); |
| 39 | 44 | } |
| 40 | 45 | |
| 41 | /** Total bytes of the regular files under `dir`, stopping as soon as it passes `cap`. Sizes only: nothing is read. */ | |
| 42 | export function bytesUnder(dir: string, cap: number): number { | |
| 46 | /** Bytes free to an unprivileged writer on the disk holding `path`. */ | |
| 47 | export const freeBytes = (path: string): number => { | |
| 48 | const s = statfsSync(path); | |
| 49 | return Number(s.bavail) * Number(s.bsize); | |
| 50 | }; | |
| 51 | ||
| 52 | const mb = (n: number): number => Math.ceil(n / (1024 * 1024)); | |
| 53 | ||
| 54 | /** | |
| 55 | * Whether each `[directory, bytes]` the work will write fits on its disk with | |
| 56 | * `SPACE_MARGIN_BYTES` to spare; two directories on one disk add up. One | |
| 57 | * sentence when one does not, else `undefined`. | |
| 58 | */ | |
| 59 | export function spaceProblem(needs: [string, number][], free: (path: string) => number = freeBytes): string | undefined { | |
| 60 | const byDisk = new Map<number, { path: string; bytes: number }>(); | |
| 61 | for (const [path, bytes] of needs) { | |
| 62 | const dev = statSync(path).dev; | |
| 63 | const had = byDisk.get(dev); | |
| 64 | byDisk.set(dev, { path, bytes: (had?.bytes ?? 0) + bytes }); | |
| 65 | } | |
| 66 | for (const { path, bytes } of byDisk.values()) { | |
| 67 | const room = free(path); | |
| 68 | if (room < bytes + SPACE_MARGIN_BYTES) | |
| 69 | return `not enough disk space: this needs about ${mb(bytes + SPACE_MARGIN_BYTES)} MB and ${mb(Math.max(0, room))} MB are free`; | |
| 70 | } | |
| 71 | return undefined; | |
| 72 | } | |
| 73 | ||
| 74 | /** | |
| 75 | * Total bytes of the regular files under `dir`, stopping as soon as it passes `cap`. Sizes only: | |
| 76 | * nothing is read. | |
| 77 | */ | |
| 78 | export function bytesUnder(dir: string, cap = Number.POSITIVE_INFINITY): number { | |
| 43 | 79 | let total = 0; |
| 44 | 80 | const stack = [dir]; |
| 45 | 81 | while (stack.length > 0 && total <= cap) { |
| @@ -65,7 +101,10 @@ const isProof = (v: unknown): v is Proof => { | ||
| 65 | 101 | ); |
| 66 | 102 | }; |
| 67 | 103 | |
| 68 | /** Run the script with the passphrase in env; its last stdout line is the proof as JSON, its stderr the refusal. */ | |
| 104 | /** | |
| 105 | * Run the script with the passphrase in env; its last stdout line is the proof as JSON, its stderr | |
| 106 | * the refusal. | |
| 107 | */ | |
| 69 | 108 | async function runScript(args: string[], passphrase: string): Promise<Proof | string> { |
| 70 | 109 | try { |
| 71 | 110 | const child = Bun.spawn([process.execPath, SCRIPT, ...args, "--json"], { |
| @@ -90,15 +129,24 @@ async function runScript(args: string[], passphrase: string): Promise<Proof | st | ||
| 90 | 129 | } |
| 91 | 130 | } |
| 92 | 131 | |
| 93 | export async function sealApp(dataDir: string, appId: string, passphrase: string, now = new Date()): Promise<Proof | string> { | |
| 132 | const tempDir = (): string => process.env["TMPDIR"] ?? tmpdir(); | |
| 133 | ||
| 134 | export async function sealApp( | |
| 135 | dataDir: string, | |
| 136 | appId: string, | |
| 137 | passphrase: string, | |
| 138 | now = new Date(), | |
| 139 | free: (path: string) => number = freeBytes, | |
| 140 | ): Promise<Proof | string> { | |
| 94 | 141 | try { |
| 95 | 142 | const src = envDataDir(dataDir, appId, "production"); |
| 96 | 143 | if (!existsSync(src)) return "this app has no production data yet"; |
| 97 | if (bytesUnder(src, SEAL_CAP_BYTES) > SEAL_CAP_BYTES) { | |
| 98 | return `the data directory is over ${SEAL_CAP_BYTES / (1024 * 1024)} MB: back it up with scripts/app-backup.ts on the host`; | |
| 99 | } | |
| 100 | 144 | const d = dirFor(dataDir, appId); |
| 101 | 145 | mkdirSync(d, { recursive: true, mode: 0o700 }); |
| 146 | const bytes = bytesUnder(src); | |
| 147 | // the archive beside the others, and its proof restored in the temp dir | |
| 148 | const short = spaceProblem([[d, bytes], [tempDir(), bytes]], free); | |
| 149 | if (short !== undefined) return short; | |
| 102 | 150 | const stamp = now.toISOString().replace(/[:.]/g, "-"); |
| 103 | 151 | return await runScript(["seal", src, join(d, `production-${stamp}.fab`)], passphrase); |
| 104 | 152 | } catch (e) { |
| @@ -106,13 +154,20 @@ export async function sealApp(dataDir: string, appId: string, passphrase: string | ||
| 106 | 154 | } |
| 107 | 155 | } |
| 108 | 156 | |
| 109 | export async function proveApp(dataDir: string, appId: string, name: string, passphrase: string): Promise<Proof | string> { | |
| 157 | export async function proveApp( | |
| 158 | dataDir: string, | |
| 159 | appId: string, | |
| 160 | name: string, | |
| 161 | passphrase: string, | |
| 162 | free: (path: string) => number = freeBytes, | |
| 163 | ): Promise<Proof | string> { | |
| 110 | 164 | let scratch: string | undefined; |
| 111 | 165 | try { |
| 112 | 166 | const at = listBackups(dataDir, appId).find((b) => b.name === name); |
| 113 | 167 | if (at === undefined) return "no such backup"; |
| 114 | if (at.bytes > SEAL_CAP_BYTES * 2) return "this backup is too large to prove here: use scripts/app-backup.ts on the host"; | |
| 115 | scratch = mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "app-prove-")); | |
| 168 | const short = spaceProblem([[tempDir(), at.bytes]], free); | |
| 169 | if (short !== undefined) return short; | |
| 170 | scratch = mkdtempSync(join(tempDir(), "app-prove-")); | |
| 116 | 171 | return await runScript(["prove", join(dirFor(dataDir, appId), name), join(scratch, "restore")], passphrase); |
| 117 | 172 | } catch (e) { |
| 118 | 173 | return (e as Error).message; |
test/app-backup.test.ts +97 −10
| @@ -1,15 +1,41 @@ | ||
| 1 | 1 | import { Database } from "bun:sqlite"; |
| 2 | 2 | import { describe, expect, test } from "bun:test"; |
| 3 | import { existsSync, mkdirSync, mkdtempSync, readdirSync, symlinkSync, truncateSync, writeFileSync } from "node:fs"; | |
| 3 | import { createCipheriv, randomBytes, scryptSync } from "node:crypto"; | |
| 4 | import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, symlinkSync, truncateSync, writeFileSync } from "node:fs"; | |
| 4 | 5 | import { tmpdir } from "node:os"; |
| 5 | 6 | import { join } from "node:path"; |
| 6 | import { openBackup, restoreProof, sealAndProve, sealParts, sealWithManifest, sourceDrift as sourceDriftOf } from "../src/apps/backup.ts"; | |
| 7 | import { bytesUnder, listBackups, proveApp, SEAL_CAP_BYTES, sealApp } from "../src/apps/backup_door.ts"; | |
| 7 | import { type Manifest, restoreProof, sealAndProve, sealToFile, sourceDrift as sourceDriftOf } from "../src/apps/backup.ts"; | |
| 8 | import { bytesUnder, listBackups, proveApp, sealApp, spaceProblem } from "../src/apps/backup_door.ts"; | |
| 8 | 9 | import { envDataDir } from "../src/apps/data.ts"; |
| 9 | 10 | |
| 10 | 11 | const PASS = "correct horse battery"; |
| 11 | 12 | const tmp = () => mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "ab-")); |
| 12 | 13 | |
| 14 | /** | |
| 15 | * Encrypt a manifest and file bodies as given, in memory, in the archive's format | |
| 16 | * (`src/apps/backup.ts`): the seam for forging a dishonest archive the honest writer never makes. | |
| 17 | */ | |
| 18 | function sealParts(manifest: Manifest, bodies: Buffer[], passphrase: string): Buffer { | |
| 19 | const m = Buffer.from(JSON.stringify(manifest)); | |
| 20 | const len = Buffer.alloc(4); | |
| 21 | len.writeUInt32BE(m.length); | |
| 22 | const salt = randomBytes(16); | |
| 23 | const iv = randomBytes(12); | |
| 24 | const header = Buffer.from(`${JSON.stringify({ v: 1, kdf: "scrypt", salt: salt.toString("base64url"), iv: iv.toString("base64url") })}\n`); | |
| 25 | const key = scryptSync(passphrase, salt, 32, { N: 1 << 15, r: 8, p: 1, maxmem: 64 * 1024 * 1024 }); | |
| 26 | const cipher = createCipheriv("aes-256-gcm", key, iv, { authTagLength: 16 }); | |
| 27 | cipher.setAAD(header); | |
| 28 | const body = Buffer.concat([cipher.update(Buffer.concat([len, m, ...bodies])), cipher.final(), cipher.getAuthTag()]); | |
| 29 | return Buffer.concat([Buffer.from("FAB1\n"), header, body]); | |
| 30 | } | |
| 31 | ||
| 32 | /** Seal to a file and read it back: the bytes, for tests that bend them. */ | |
| 33 | function sealWithManifest(dir: string, pass: string): { sealed: Buffer; manifest: Manifest } { | |
| 34 | const out = join(tmp(), "b.fab"); | |
| 35 | const manifest = sealToFile(dir, out, pass); | |
| 36 | return { sealed: readFileSync(out), manifest }; | |
| 37 | } | |
| 38 | ||
| 13 | 39 | function app(): string { |
| 14 | 40 | const d = tmp(); |
| 15 | 41 | const db = new Database(join(d, "app.db")); |
| @@ -39,7 +65,7 @@ describe("app backup", () => { | ||
| 39 | 65 | const bent = Buffer.from(sealed); |
| 40 | 66 | bent[bent.length - 20] = bent[bent.length - 20]! ^ 1; |
| 41 | 67 | expect(restoreProof(bent, PASS, scratch)).toMatch(/wrong passphrase/); |
| 42 | expect(openBackup(Buffer.from("junk"), PASS)).toBe("not an app backup file"); | |
| 68 | expect(restoreProof(Buffer.from("junk"), PASS, join(tmp(), "j"))).toBe("not an app backup file"); | |
| 43 | 69 | }); |
| 44 | 70 | test("a short passphrase is refused up front", () => { |
| 45 | 71 | expect(() => sealWithManifest(app(), "short")).toThrow(/under 12/); |
| @@ -163,18 +189,24 @@ describe("src/apps/backup_door.ts", () => { | ||
| 163 | 189 | mkdirSync(d, { recursive: true }); |
| 164 | 190 | return d; |
| 165 | 191 | }; |
| 166 | test("a data directory over the cap is refused before it is read, and nothing is written", async () => { | |
| 192 | test("a seal that would not fit on the disk is refused in one sentence, and nothing is written", async () => { | |
| 167 | 193 | const forge = tmp(); |
| 168 | 194 | const d = appData(forge, "big"); |
| 169 | 195 | mkdirSync(join(d, "deep"), { recursive: true }); |
| 170 | 196 | writeFileSync(join(d, "deep", "huge.bin"), ""); |
| 171 | truncateSync(join(d, "deep", "huge.bin"), SEAL_CAP_BYTES + 1); // sparse: no bytes on disk | |
| 172 | expect(bytesUnder(d, SEAL_CAP_BYTES)).toBeGreaterThan(SEAL_CAP_BYTES); | |
| 173 | const r = await sealApp(forge, "big", PASS); | |
| 174 | expect(r).toMatch(/over 64 MB/); | |
| 197 | truncateSync(join(d, "deep", "huge.bin"), 300 * 1024 * 1024); // sparse: no bytes on disk | |
| 198 | expect(bytesUnder(d)).toBe(300 * 1024 * 1024); | |
| 199 | const r = await sealApp(forge, "big", PASS, new Date(), () => 400 * 1024 * 1024); | |
| 200 | // archive and proof share the disk: 600 MB plus the margin | |
| 201 | expect(r).toBe("not enough disk space: this needs about 664 MB and 400 MB are free"); | |
| 175 | 202 | expect(listBackups(forge, "big")).toEqual([]); |
| 176 | 203 | }); |
| 177 | test("under the cap it seals in a child process and proves", async () => { | |
| 204 | test("space on two disks is counted per disk; room enough answers nothing", () => { | |
| 205 | const a = tmp(); | |
| 206 | expect(spaceProblem([[a, 10], [a, 10]], () => 1 << 30)).toBeUndefined(); | |
| 207 | expect(spaceProblem([[a, 10]], () => 0)).toMatch(/^not enough disk space: .* 0 MB are free$/); | |
| 208 | }); | |
| 209 | test("with room it seals in a child process and proves", async () => { | |
| 178 | 210 | const forge = tmp(); |
| 179 | 211 | writeFileSync(join(appData(forge, "small"), "x.txt"), "hello"); |
| 180 | 212 | const r = await sealApp(forge, "small", PASS); |
| @@ -193,3 +225,58 @@ describe("src/apps/backup_door.ts", () => { | ||
| 193 | 225 | expect(await proveApp(forge, "bad", "../../etc/passwd", PASS)).toBe("no such backup"); |
| 194 | 226 | }); |
| 195 | 227 | }); |
| 228 | ||
| 229 | describe("streamed archive", () => { | |
| 230 | test("a restore of the streamed archive gives the same files, byte for byte", () => { | |
| 231 | const d = app(); | |
| 232 | mkdirSync(join(d, "big/nested"), { recursive: true }); | |
| 233 | writeFileSync(join(d, "big/nested/blob.bin"), randomBytes(3 * 1024 * 1024 + 17)); // spans chunks | |
| 234 | writeFileSync(join(d, "empty"), ""); | |
| 235 | const out = join(tmp(), "b.fab"); | |
| 236 | const m = sealToFile(d, out, PASS); | |
| 237 | const scratch = join(tmp(), "r"); | |
| 238 | const p = restoreProof(out, PASS, scratch); | |
| 239 | expect(p).toMatchObject({ ok: true, files: 4, rows: 7, problems: [] }); | |
| 240 | for (const f of m.files) expect(readFileSync(join(scratch, f.path)).equals(readFileSync(join(d, f.path)))).toBe(true); | |
| 241 | }); | |
| 242 | test("a truncated archive file is refused and writes nothing", () => { | |
| 243 | const out = join(tmp(), "b.fab"); | |
| 244 | writeFileSync(out, sealWithManifest(app(), PASS).sealed.subarray(0, -5)); | |
| 245 | const scratch = join(tmp(), "r"); | |
| 246 | expect(restoreProof(out, PASS, scratch)).toMatch(/wrong passphrase, or the file is damaged/); | |
| 247 | expect(existsSync(scratch)).toBe(false); | |
| 248 | }); | |
| 249 | // VmHWM in /proc is the only honest peak: getrusage's maxRSS survives exec and would count the | |
| 250 | // test runner. Without /proc (not Linux) there is no true number to check, so the test is | |
| 251 | // skipped. | |
| 252 | test.skipIf(!existsSync("/proc/self/status"))("a 500 MB app seals and proves with resident memory under 100 MB", async () => { | |
| 253 | const d = app(); | |
| 254 | for (let i = 0; i < 5; i++) { | |
| 255 | writeFileSync(join(d, `part${i}.bin`), randomBytes(4096)); | |
| 256 | truncateSync(join(d, `part${i}.bin`), 100 * 1024 * 1024); // sparse: the test costs no disk to set up | |
| 257 | } | |
| 258 | const base = tmp(); | |
| 259 | const out = join(base, "b.fab"); | |
| 260 | // The child reports its own peak after sealing and proving: VmHWM, the high-water mark of this | |
| 261 | // process image, or null when it cannot read it, which fails the test. | |
| 262 | const script = `import { readFileSync } from "node:fs"; | |
| 263 | import { sealAndProve } from ${JSON.stringify(join(import.meta.dir, "../src/apps/backup.ts"))}; | |
| 264 | const p = sealAndProve(process.argv[1], process.argv[2], process.env.APP_BACKUP_PASSPHRASE); | |
| 265 | const hwm = /VmHWM:\\s+(\\d+) kB/.exec(readFileSync("/proc/self/status", "utf8")); | |
| 266 | console.log(JSON.stringify({ ok: p.ok, files: p.files, bytes: p.bytes, maxKB: hwm ? Number(hwm[1]) : null }));`; | |
| 267 | const child = Bun.spawn(["bun", "-e", script, d, out], { | |
| 268 | env: { PATH: process.env["PATH"] ?? "", TMPDIR: base, APP_BACKUP_PASSPHRASE: PASS }, | |
| 269 | stdout: "pipe", | |
| 270 | stderr: "inherit", | |
| 271 | }); | |
| 272 | const text = await new Response(child.stdout).text(); | |
| 273 | expect(await child.exited).toBe(0); | |
| 274 | const got = JSON.parse(text.trim()) as { ok: boolean; files: number; bytes: number; maxKB: number | null }; | |
| 275 | rmSync(base, { recursive: true, force: true }); | |
| 276 | rmSync(d, { recursive: true, force: true }); | |
| 277 | expect(got).toMatchObject({ ok: true, files: 7 }); | |
| 278 | expect(got.bytes).toBeGreaterThan(500 * 1024 * 1024); | |
| 279 | expect(got.maxKB).not.toBeNull(); | |
| 280 | expect((got.maxKB ?? Infinity) * 1024).toBeLessThan(100 * 1024 * 1024); | |
| 281 | }, 120_000); | |
| 282 | }); |
new above
offline