Public: anyone can read this; the crew can change it.

An app's backup streams to disk and is not held in memory #13182

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13111-an-app-s-backup-streams-to-disk-and-is-n → main 6 commits fork

  • Merged. scout/13111-an-app-s-backup-streams-to-disk-and-is-n was merged into main as 6f34b06.

Review

Sign in to review this change.
6 files changed +469 −147
docs/features/app-backup.md +24 −11
@@ -12,15 +12,24 @@ bun scripts/app-backup.ts prove <out-file> <empty-scratch-dir>
1212
1313 - `seal` checkpoints every SQLite file, archives the directory, encrypts it
1414 (AES-256-GCM, key = scrypt of the passphrase, header authenticated) and
15 writes `<out-file>` with mode 0600. It then restores the file into a scratch
16 directory of its own and prints `PROVEN: <files>, <bytes>, <tables>, <rows>`,
17 exit 0, or `RED` and each mismatch, exit 1. Delete nothing on the old side
18 until it says PROVEN.
15 writes `<out-file>` with mode 0600. It then restores the file into a
16 scratch directory of its own and prints `PROVEN: <files>, <bytes>, <tables>,
17 <rows>`, exit 0, or `RED` and each mismatch, exit 1. Delete nothing on the
18 old side until it says PROVEN.
19 - The archive streams to disk as it is built (#13111): each file is hashed
20 for the manifest, then passed through the cipher into the file 256 KB at a
21 time. The proof checks the whole file's tag in one pass that keeps nothing,
22 then decrypts into the scratch directory in a second. Memory stays flat
23 whatever the data's size: a synthetic 500 MB app seals and proves in about
24 70 MB resident, most of it the runtime and scrypt
25 (`test/app-backup.test.ts`). The format is unchanged, so older backups
26 still open.
1927 - The proof reads back from disk every file's size and SHA-256, and every
2028 SQLite table's row count, against the manifest taken at backup time
2129 (the step-3 and step-5 comparison in `platform-app-move.md`).
22 - The seal is only kept when it is clean. It is written under a temporary
23 name beside `<out-file>` and renamed on PROVEN; a RED proof leaves nothing.
30 - The seal is only kept when it is clean. It is streamed to a temporary name
31 beside `<out-file>` and renamed on PROVEN; a half-written archive or a RED
32 proof leaves nothing that looks like a backup.
2433 An out-file inside the data directory is refused.
2534 - The proof covers the data left behind too: after sealing, the data
2635 directory is read again against the manifest, and a file that changed,
@@ -41,13 +50,17 @@ data left behind as above; `p` proves the file under the cursor. The screen
4150 shows the same PROVEN or RED lines as the script. The file is encrypted under
4251 a key only the owner holds; copy it off the forge to keep it. The seal and
4352 the proof run `src/apps/backup_cli.ts` (the code behind `scripts/app-backup.ts`) in a child process (the passphrase in
44 its environment, never in argv), so the forge's event loop stays free; a
45 production data directory over 64 MB (`SEAL_CAP_BYTES`) is refused before it
46 is read, and the screen says to run the script on the host instead. Logic:
53 its environment, never in argv), so the forge's event loop stays free. There
54 is no size cap: before a byte is read, the screen checks the disk has room
55 for the archive and its proof restore (the data's size each, added up when
56 they share a disk, plus 64 MB to spare; a proof needs the backup's size in
57 the temp directory) and otherwise refuses in one sentence:
58 `not enough disk space: this needs about N MB and M MB are free`. Logic:
4759 `src/apps/backup_door.ts`; tests: `test/tui-appbackup.test.ts`.
4860
4961 - Code: `src/apps/backup.ts`; tests: `test/app-backup.test.ts`.
5062
51 **Not done:** the archive is held in memory (the terminal caps it at 64 MB;
52 the script on the host has no cap); no scheduled backup, no upload to a bucket, and no web page button; the
63 **Not done:** the seal still reads each file twice (hash, then encrypt) and
64 the proof decrypts twice (tag, then restore), trading time for flat memory;
65 no scheduled backup, no upload to a bucket, and no web page button; the
5366 terminal seals production data only, not previews.
scripts/gate.ts +1
@@ -72,6 +72,7 @@ export const SLOW: readonly Exclusion[] = [
7272 { file: "test/release-journey.test.ts", reason: "walks the journey on a real spawned child forge with a stand-in crew, polling in real time; the passing walk takes 10 s under load", issue: 12876 },
7373 { file: "test/guide-journey.test.ts", reason: "runs the first-app guide end to end: signup, clone, push and a real deploy; 5.7 s a test on the gate box", issue: 12069 },
7474 { file: "test/apps-crew.test.ts", reason: "runs the crew against a real app, a restart and a push that cannot boot; 3.0 s a test on the gate box", issue: 1242 },
75 { file: "test/app-backup.test.ts", reason: "seals and proves a 500 MB app to show memory stays under 100 MB while it streams; 11.3 s a test on the gate box", issue: 13111 },
7576 { file: "test/apps-measure.test.ts", reason: "sends 601 reports through the app proxy to reach the rate limit; 3.7 s a test on the gate box", issue: 1242 },
7677 { file: "test/release-keeper-swap.test.ts", reason: "swaps two real stand-in processes in and one back under a hammering client, polling each to answer; 2.7 s a test on the gate box", issue: 12790 },
7778 { file: "test/apps.test.ts", reason: "deploys real apps to production and a preview, copying data; 2.0 s to 3.3 s a test on the gate box", issue: 12874 },
src/apps/backup.ts +273 −107
@@ -9,8 +9,9 @@
99 * base64url), then AES-256-GCM ciphertext ending in its 16-byte tag. The
1010 * key is scrypt of the passphrase; the header is authenticated. Inside is a
1111 * manifest (every file's path, size, SHA-256, and every SQLite table's row
12 * count) followed by the file bytes. Whole-archive in memory: meant for
13 * app data of tens of megabytes, not gigabytes (`docs/features/app-backup.md`).
12 * count) followed by the file bytes. Sealing streams each file through the
13 * cipher into the out-file and opening streams it back out, 256 KB at a
14 * time: memory stays flat whatever the data's size (`docs/features/app-backup.md`).
1415 */
1516
1617 import { Database } from "bun:sqlite";
@@ -22,16 +23,20 @@ import {
2223 scryptSync,
2324 } from "node:crypto";
2425 import {
26 closeSync,
2527 existsSync,
28 fstatSync,
29 fsyncSync,
2630 lstatSync,
2731 mkdirSync,
28 readdirSync,
2932 mkdtempSync,
30 readFileSync,
33 openSync,
34 readdirSync,
35 readSync,
3136 realpathSync,
3237 renameSync,
3338 rmSync,
34 writeFileSync,
39 writeSync,
3540 } from "node:fs";
3641 import { tmpdir } from "node:os";
3742 import { basename, dirname, join, resolve, sep } from "node:path";
@@ -63,7 +68,53 @@ export interface Proof {
6368 problems: string[];
6469 }
6570
66 const sha = (b: Uint8Array): string => createHash("sha256").update(b).digest("hex");
71 const CHUNK = 1 << 18;
72
73 /** Size and SHA-256 of a file, read a chunk at a time: the file is never whole in memory. */
74 export function hashFile(path: string): { size: number; sha256: string } {
75 const fd = openSync(path, "r");
76 try {
77 const h = createHash("sha256");
78 const buf = Buffer.allocUnsafe(CHUNK);
79 let size = 0;
80 for (let n; (n = readSync(fd, buf, 0, CHUNK, null)) > 0; size += n) h.update(buf.subarray(0, n));
81 return { size, sha256: h.digest("hex") };
82 } finally {
83 closeSync(fd);
84 }
85 }
86
87 /**
88 * Random-access bytes of a sealed archive: a file on disk, or a buffer (tests, small callers).
89 * A read is valid until the next.
90 */
91 interface Source {
92 size: number;
93 read(pos: number, len: number): Buffer;
94 close(): void;
95 }
96 function sourceOf(sealed: Uint8Array | string): Source {
97 if (typeof sealed !== "string") {
98 const b = Buffer.from(sealed.buffer, sealed.byteOffset, sealed.byteLength);
99 return { size: b.length, read: (pos, len) => b.subarray(pos, pos + len), close: () => {} };
100 }
101 const fd = openSync(sealed, "r");
102 const reused = Buffer.allocUnsafe(CHUNK);
103 return {
104 size: fstatSync(fd).size,
105 read: (pos, len) => {
106 const out = len <= CHUNK ? reused.subarray(0, len) : Buffer.allocUnsafe(len);
107 let got = 0;
108 while (got < len) {
109 const n = readSync(fd, out, got, len - got, pos + got);
110 if (n === 0) break;
111 got += n;
112 }
113 return out.subarray(0, got);
114 },
115 close: () => closeSync(fd),
116 };
117 }
67118
68119 export function passphraseProblem(p: string | undefined): string | undefined {
69120 if (p === undefined || p === "") return "no passphrase: set APP_BACKUP_PASSPHRASE";
@@ -106,154 +157,270 @@ export function tableCounts(path: string): Record<string, number> | undefined {
106157 }
107158 }
108159
109 /** Seal a data directory. Throws a message (not a bug) for an unusable passphrase or directory. */
110 export function sealWithManifest(dir: string, passphrase: string): { sealed: Buffer; manifest: Manifest } {
160 const headerFor = (salt: Buffer, iv: Buffer): Buffer =>
161 Buffer.from(`${JSON.stringify({ v: 1, kdf: "scrypt", salt: salt.toString("base64url"), iv: iv.toString("base64url") })}\n`);
162
163 const startsSqlite = (path: string): boolean => {
164 const fd = openSync(path, "r");
165 try {
166 const b = Buffer.alloc(16);
167 return readSync(fd, b, 0, 16, 0) === 16 && isSqlite(b);
168 } finally {
169 closeSync(fd);
170 }
171 };
172
173 /**
174 * Seal a data directory into `out`, written as it is built: each file is
175 * hashed for the manifest, then streamed a chunk at a time through the cipher
176 * into `out`, so memory stays flat whatever the data's size. A file that
177 * changes between the two reads is written at its manifest size and the
178 * proof calls it out. `out` is written as given; `sealAndProve` names it
179 * `.part` and renames it only on PROVEN. Throws a message for unusable input.
180 */
181 export function sealToFile(dir: string, out: string, passphrase: string): Manifest {
111182 const bad = passphraseProblem(passphrase);
112183 if (bad !== undefined) throw new Error(bad);
113184 if (!existsSync(dir) || !lstatSync(dir).isDirectory()) throw new Error("the data directory does not exist");
114185 quiesceSqlite(dir);
115186 const found = walk(dir);
116187 const manifest: Manifest = { files: [], tables: {}, skipped: found.skipped };
117 const bodies: Buffer[] = [];
118188 for (const path of found.files) {
119 const bytes = readFileSync(join(dir, path));
120 manifest.files.push({ path, size: bytes.length, sha256: sha(bytes) });
121 bodies.push(bytes);
122 if (isSqlite(bytes)) {
123 const counts = tableCounts(join(dir, path));
189 const full = join(dir, path);
190 manifest.files.push({ path, ...hashFile(full) });
191 if (startsSqlite(full)) {
192 const counts = tableCounts(full);
124193 if (counts !== undefined) manifest.tables[path] = counts;
125194 }
126195 }
127 return { sealed: sealParts(manifest, bodies, passphrase), manifest };
128 }
129
130 /** Encrypt a manifest and its file bodies as given; `sealWithManifest` is the honest caller. */
131 export function sealParts(manifest: Manifest, bodies: Buffer[], passphrase: string): Buffer {
132 const m = Buffer.from(JSON.stringify(manifest));
133 const len = Buffer.alloc(4);
134 len.writeUInt32BE(m.length);
135 const plain = Buffer.concat([len, m, ...bodies]);
136
137196 const salt = randomBytes(16);
138197 const iv = randomBytes(12);
139 const header = Buffer.from(
140 `${JSON.stringify({ v: 1, kdf: "scrypt", salt: salt.toString("base64url"), iv: iv.toString("base64url") })}\n`,
141 );
198 const header = headerFor(salt, iv);
142199 const cipher = createCipheriv("aes-256-gcm", scryptSync(passphrase, salt, 32, SCRYPT), iv, { authTagLength: 16 });
143200 cipher.setAAD(header);
144 return Buffer.concat([Buffer.from(MAGIC), header, cipher.update(plain), cipher.final(), cipher.getAuthTag()]);
201 const fd = openSync(out, "w", 0o600);
202 try {
203 const put = (b: Buffer): void => {
204 for (let at = 0; at < b.length; ) at += writeSync(fd, b, at);
205 };
206 put(Buffer.concat([Buffer.from(MAGIC), header]));
207 const m = Buffer.from(JSON.stringify(manifest));
208 const len = Buffer.alloc(4);
209 len.writeUInt32BE(m.length);
210 put(cipher.update(Buffer.concat([len, m])));
211 const buf = Buffer.allocUnsafe(CHUNK);
212 for (const f of manifest.files) {
213 const src = openSync(join(dir, f.path), "r");
214 try {
215 let left = f.size;
216 while (left > 0) {
217 const n = readSync(src, buf, 0, Math.min(CHUNK, left), null);
218 if (n === 0) {
219 put(cipher.update(Buffer.alloc(left))); // shrank since hashed: keep the layout, the proof says so
220 break;
221 }
222 put(cipher.update(buf.subarray(0, n)));
223 left -= n;
224 }
225 } finally {
226 closeSync(src);
227 }
228 }
229 put(cipher.final());
230 put(cipher.getAuthTag());
231 fsyncSync(fd);
232 } finally {
233 closeSync(fd);
234 }
235 return manifest;
145236 }
146237
147 /** The manifest and file bytes, or a message: a wrong passphrase and a damaged file say the same thing. */
148 export function openBackup(sealed: Uint8Array, passphrase: string): { manifest: Manifest; bodies: Buffer[] } | string {
149 const refuse = "cannot open this backup: wrong passphrase, or the file is damaged";
150 try {
151 const buf = Buffer.from(sealed);
152 if (buf.subarray(0, MAGIC.length).toString("latin1") !== MAGIC) return "not an app backup file";
153 const nl = buf.indexOf(10, MAGIC.length);
154 if (nl < 0 || nl > 512 || buf.length < nl + 1 + 16) return refuse;
155 const header = buf.subarray(MAGIC.length, nl + 1);
156 const h = JSON.parse(header.toString()) as Record<string, unknown>;
157 if (h["v"] !== 1 || h["kdf"] !== "scrypt") return refuse;
158 const salt = Buffer.from(String(h["salt"]), "base64url");
159 const iv = Buffer.from(String(h["iv"]), "base64url");
160 if (salt.length !== 16 || iv.length !== 12) return refuse;
161 const body = buf.subarray(nl + 1);
162 const d = createDecipheriv("aes-256-gcm", scryptSync(passphrase, salt, 32, SCRYPT), iv, { authTagLength: 16 });
238 const REFUSE = "cannot open this backup: wrong passphrase, or the file is damaged";
239
240 /**
241 * The archive's plaintext, decrypted a chunk at a time after the whole file's tag has been
242 * checked.
243 */
244 interface Plain {
245 /** Up to `n` bytes, fewer only at the end. */
246 take(n: number): Buffer;
247 /** Pass the next `n` bytes to `sink` in chunks; returns how many there were. */
248 pipe(n: number, sink: (b: Buffer) => void): number;
249 }
250
251 /**
252 * Check the archive's header and its GCM tag over every byte (a first pass
253 * that keeps nothing), then return a reader of the plaintext. Nothing is
254 * handed out before the whole file is authenticated.
255 */
256 function openPlain(src: Source, passphrase: string): Plain | string {
257 if (src.read(0, MAGIC.length).toString("latin1") !== MAGIC) return "not an app backup file";
258 const head = src.read(MAGIC.length, 512);
259 const nl = head.indexOf(10);
260 if (nl < 0) return REFUSE;
261 const header = Buffer.from(head.subarray(0, nl + 1));
262 const bodyAt = MAGIC.length + nl + 1;
263 if (src.size < bodyAt + 16) return REFUSE;
264 const h = JSON.parse(header.toString()) as Record<string, unknown>;
265 if (h["v"] !== 1 || h["kdf"] !== "scrypt") return REFUSE;
266 const salt = Buffer.from(String(h["salt"]), "base64url");
267 const iv = Buffer.from(String(h["iv"]), "base64url");
268 if (salt.length !== 16 || iv.length !== 12) return REFUSE;
269 const key = scryptSync(passphrase, salt, 32, SCRYPT);
270 const end = src.size - 16;
271 const tag = Buffer.from(src.read(end, 16));
272 const decipher = () => {
273 const d = createDecipheriv("aes-256-gcm", key, iv, { authTagLength: 16 });
163274 d.setAAD(header);
164 d.setAuthTag(body.subarray(body.length - 16));
165 const plain = Buffer.concat([d.update(body.subarray(0, body.length - 16)), d.final()]);
166 const mlen = plain.readUInt32BE(0);
167 const manifest = JSON.parse(plain.subarray(4, 4 + mlen).toString()) as Manifest;
168 let at = 4 + mlen;
169 const bodies: Buffer[] = [];
170 for (const f of manifest.files) {
171 bodies.push(plain.subarray(at, at + f.size));
172 at += f.size;
275 d.setAuthTag(tag);
276 return d;
277 };
278 const check = decipher();
279 for (let at = bodyAt; at < end; at += CHUNK) check.update(src.read(at, Math.min(CHUNK, end - at)));
280 check.final(); // throws on a wrong passphrase or a damaged file
281 const d = decipher();
282 let at = bodyAt;
283 let held = Buffer.alloc(0);
284 const fill = (): boolean => {
285 if (held.length > 0) return true;
286 if (at >= end) return false;
287 held = d.update(src.read(at, Math.min(CHUNK, end - at)));
288 at += CHUNK;
289 return true;
290 };
291 const pipe = (n: number, sink: (b: Buffer) => void): number => {
292 let got = 0;
293 while (got < n && fill()) {
294 const part = held.subarray(0, n - got);
295 held = held.subarray(part.length);
296 got += part.length;
297 sink(part);
173298 }
174 return { manifest, bodies };
299 return got;
300 };
301 return {
302 pipe,
303 take: (n) => {
304 const parts: Buffer[] = [];
305 pipe(n, (b) => parts.push(Buffer.from(b)));
306 return Buffer.concat(parts);
307 },
308 };
309 }
310
311 function withPlain<T>(sealed: Uint8Array | string, passphrase: string, use: (manifest: Manifest, plain: Plain) => T): T | string {
312 let src: Source;
313 try {
314 src = sourceOf(sealed);
175315 } catch {
176 return refuse;
316 return "cannot read this backup file";
317 }
318 try {
319 let plain: Plain | string;
320 let manifest: Manifest;
321 try {
322 plain = openPlain(src, passphrase);
323 if (typeof plain === "string") return plain;
324 const len = plain.take(4);
325 if (len.length !== 4) return REFUSE;
326 manifest = JSON.parse(plain.take(len.readUInt32BE(0)).toString()) as Manifest;
327 if (!Array.isArray(manifest.files) || typeof manifest.tables !== "object" || manifest.tables === null) return REFUSE;
328 } catch {
329 return REFUSE;
330 }
331 return use(manifest, plain);
332 } finally {
333 src.close();
177334 }
178335 }
179336
180337 const safeRel = (p: string): boolean =>
181 p !== "" && !p.startsWith("/") && !p.includes("\0") && !p.split("/").some((s) => s === ".." || s === "." || s === "");
338 typeof p === "string" && p !== "" && !p.startsWith("/") && !p.includes("\0") && !p.split("/").some((s) => s === ".." || s === "." || s === "");
182339
183340 /**
184 * Restore a backup into `scratch` (must not exist or be empty) and prove it:
185 * every file's SHA-256 and size read back from disk, every SQLite table's row
186 * count read back from the restored database. Returns a message when it cannot
187 * even try (wrong passphrase, scratch in use).
341 * Restore a backup (a file path, streamed; or bytes) into `scratch` (must not
342 * exist or be empty) and prove it: every file's SHA-256 and size read back
343 * from disk, every SQLite table's row count read back from the restored
344 * database. Returns a message when it cannot even try (wrong passphrase,
345 * damaged file, scratch in use); then nothing has been written.
188346 */
189 export function restoreProof(sealed: Uint8Array, passphrase: string, scratch: string): Proof | string {
347 export function restoreProof(sealed: Uint8Array | string, passphrase: string, scratch: string): Proof | string {
190348 const bad = passphraseProblem(passphrase);
191349 if (bad !== undefined) return bad;
192350 if (existsSync(scratch) && readdirSync(scratch).length > 0) return "the scratch directory is not empty";
193 const opened = openBackup(sealed, passphrase);
194 if (typeof opened === "string") return opened;
195 const { manifest, bodies } = opened;
196 const root = resolve(scratch);
197 mkdirSync(root, { recursive: true });
198 const problems: string[] = [];
199 for (const k of manifest.skipped ?? [])
200 problems.push(`${k}: not archived (a symlink or special file); the backup does not cover it`);
201 let bytes = 0;
202 manifest.files.forEach((f, i) => {
203 const target = resolve(root, f.path);
204 if (!safeRel(f.path) || !target.startsWith(root + sep)) {
205 problems.push(`refused an unsafe path: ${JSON.stringify(f.path)}`);
206 return;
351 return withPlain(sealed, passphrase, (manifest, plain) => {
352 const root = resolve(scratch);
353 mkdirSync(root, { recursive: true });
354 const problems: string[] = [];
355 for (const k of manifest.skipped ?? [])
356 problems.push(`${k}: not archived (a symlink or special file); the backup does not cover it`);
357 const inside = (p: string): boolean => safeRel(p) && resolve(root, p).startsWith(root + sep);
358 for (const f of manifest.files) {
359 if (!inside(f.path)) {
360 problems.push(`refused an unsafe path: ${JSON.stringify(f.path)}`);
361 plain.pipe(f.size, () => {});
362 continue;
363 }
364 const target = resolve(root, f.path);
365 mkdirSync(dirname(target), { recursive: true });
366 const fd = openSync(target, "w", 0o600);
367 try {
368 plain.pipe(f.size, (b) => {
369 for (let at = 0; at < b.length; ) at += writeSync(fd, b, at);
370 });
371 } finally {
372 closeSync(fd);
373 }
207374 }
208 mkdirSync(dirname(target), { recursive: true });
209 writeFileSync(target, bodies[i]!);
375 let bytes = 0;
376 for (const f of manifest.files) {
377 if (!inside(f.path)) continue;
378 const back = hashFile(resolve(root, f.path));
379 bytes += back.size;
380 if (back.size !== f.size || back.sha256 !== f.sha256) problems.push(`${f.path}: bytes differ from the backup's manifest`);
381 }
382 let tables = 0;
383 let rows = 0;
384 for (const [file, expected] of Object.entries(manifest.tables)) {
385 if (!inside(file)) continue;
386 const got = tableCounts(join(root, file));
387 for (const [t, n] of Object.entries(expected)) {
388 tables++;
389 rows += n;
390 if (got === undefined) problems.push(`${file}: restored database does not open`);
391 else if (got[t] !== n) problems.push(`${file} ${t}: ${n} rows backed up, ${got[t] ?? "no table"} restored`);
392 }
393 }
394 return { ok: problems.length === 0, files: manifest.files.length, bytes, tables, rows, problems };
210395 });
211 for (const f of manifest.files) {
212 const target = resolve(root, f.path);
213 if (!safeRel(f.path) || !target.startsWith(root + sep)) continue;
214 const back = readFileSync(target);
215 bytes += back.length;
216 if (back.length !== f.size || sha(back) !== f.sha256) problems.push(`${f.path}: bytes differ from the backup's manifest`);
217 }
218 let tables = 0;
219 let rows = 0;
220 for (const [file, expected] of Object.entries(manifest.tables)) {
221 if (!safeRel(file)) continue;
222 const got = tableCounts(join(root, file));
223 for (const [t, n] of Object.entries(expected)) {
224 tables++;
225 rows += n;
226 if (got === undefined) problems.push(`${file}: restored database does not open`);
227 else if (got[t] !== n) problems.push(`${file} ${t}: ${n} rows backed up, ${got[t] ?? "no table"} restored`);
228 }
229 }
230 return { ok: problems.length === 0, files: manifest.files.length, bytes, tables, rows, problems };
231396 }
232397
233 /** The data directory against the manifest taken from it: what changed since, and what was left out. */
398 /**
399 * The data directory against the manifest taken from it: what changed since, and what was left out.
400 */
234401 export function sourceDrift(dir: string, manifest: Manifest): string[] {
235402 const problems: string[] = [];
236403 const now = walk(dir);
237404 const had = new Set(manifest.files.map((f) => f.path));
238405 for (const p of now.files) if (!had.has(p)) problems.push(`${p}: appeared in the data directory after the backup was read`);
239406 for (const f of manifest.files) {
240 let back: Buffer;
407 let back: { size: number; sha256: string };
241408 try {
242 back = readFileSync(join(dir, f.path));
409 back = hashFile(join(dir, f.path));
243410 } catch {
244411 problems.push(`${f.path}: gone from the data directory since the backup was read`);
245412 continue;
246413 }
247 if (back.length !== f.size || sha(back) !== f.sha256) problems.push(`${f.path}: changed in the data directory since the backup was read`);
414 if (back.size !== f.size || back.sha256 !== f.sha256) problems.push(`${f.path}: changed in the data directory since the backup was read`);
248415 }
249416 return problems;
250417 }
251418
252419 /**
253420 * Seal `dir` into `out`, prove the result and the data left behind, and keep
254 * the file only when both are clean: it is written under a temporary name in
255 * the same directory and renamed on PROVEN, so a RED proof leaves no backup
256 * that looks like one. Throws a message for unusable input.
421 * the file only when both are clean: it is streamed to a temporary name in
422 * the same directory and renamed on PROVEN, so a half-written archive or a RED
423 * proof leaves no backup that looks like one. Throws a message for unusable input.
257424 */
258425 export function sealAndProve(dir: string, out: string, passphrase: string): Proof {
259426 const bad = passphraseProblem(passphrase);
@@ -263,12 +430,11 @@ export function sealAndProve(dir: string, out: string, passphrase: string): Proo
263430 const target = join(outDir, basename(out));
264431 const root = realpathSync(dir);
265432 if (target === root || target.startsWith(root + sep)) throw new Error("the out-file is inside the data directory");
266 const { sealed, manifest } = sealWithManifest(dir, passphrase);
267433 const part = join(outDir, `.${basename(out)}.${randomBytes(6).toString("hex")}.part`);
268434 const scratch = mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "app-backup-"));
269435 try {
270 writeFileSync(part, sealed, { mode: 0o600 });
271 const p = restoreProof(readFileSync(part), passphrase, join(scratch, "restore"));
436 const manifest = sealToFile(dir, part, passphrase);
437 const p = restoreProof(part, passphrase, join(scratch, "restore"));
272438 if (typeof p === "string") throw new Error(p);
273439 p.problems.push(...sourceDrift(dir, manifest));
274440 p.ok = p.problems.length === 0;
src/apps/backup_cli.ts +3 −3
@@ -1,5 +1,6 @@
11 /**
2 * An app's encrypted backup and its restore proof, as a command line: the owner runs it, and so does the terminal's door.
2 * An app's encrypted backup and its restore proof, as a command line: the owner runs it, and so
3 * does the terminal's door.
34 * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts seal <data-dir> <out-file>
45 * APP_BACKUP_PASSPHRASE=… bun scripts/app-backup.ts prove <backup-file> <empty-scratch-dir>
56 * A trailing `--json` prints the proof as one JSON line in place of the
@@ -10,7 +11,6 @@
1011 * a child process and the release image ships src/ alone; scripts/app-backup.ts
1112 * is the owner's entry to it. See docs/features/app-backup.md.
1213 */
13 import { readFileSync } from "node:fs";
1414 import { restoreProof, sealAndProve, type Proof } from "./backup.ts";
1515 import { proofLines } from "./backup_words.ts";
1616
@@ -29,7 +29,7 @@ try {
2929 show(p);
3030 process.exit(p.ok ? 0 : 1);
3131 } else if (cmd === "prove" && a && b) {
32 const p = restoreProof(readFileSync(a), pass, b);
32 const p = restoreProof(a, pass, b);
3333 if (typeof p === "string") throw new Error(p);
3434 show(p);
3535 process.exit(p.ok ? 0 : 1);
src/apps/backup_door.ts +71 −16
@@ -2,18 +2,23 @@
22 * What the terminal's "Back up data" screen does for one app: seal its
33 * production data under the owner's passphrase into `<data>/app-backups/<appId>/`,
44 * list what is there, and prove one file. The work runs in a child process
5 * (`src/apps/backup_cli.ts`, the code behind `scripts/app-backup.ts`), so the forge's event loop stays free; the
6 * passphrase goes to it in the environment, never in argv, and is never
7 * stored, logged or returned. A data directory over `SEAL_CAP_BYTES` is
8 * refused before a byte of it is read: the archive is held whole in memory.
5 * (`src/apps/backup_cli.ts`, the code behind `scripts/app-backup.ts`), so the forge's event loop
6 * stays free; the passphrase goes to it in the environment, never in argv, and is never stored,
7 * logged or returned. The archive streams to disk, so the only limit is room for it: a seal or a
8 * proof that would not fit on the disk is refused in one sentence before a byte is read
9 * (`spaceProblem`).
910 */
10 import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statSync } from "node:fs";
11 import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statfsSync, statSync } from "node:fs";
1112 import { tmpdir } from "node:os";
1213 import { join } from "node:path";
1314 import type { Proof } from "./backup.ts";
1415 import { envDataDir } from "./data.ts";
1516
16 export const SEAL_CAP_BYTES = 64 * 1024 * 1024;
17 /**
18 * Room kept free beyond the archive and its proof, so a backup never fills the disk to the last
19 * byte.
20 */
21 export const SPACE_MARGIN_BYTES = 64 * 1024 * 1024;
1722 const SCRIPT = new URL("./backup_cli.ts", import.meta.url).pathname;
1823 const CHILD_TIMEOUT_MS = 10 * 60_000;
1924
@@ -38,8 +43,39 @@ export function listBackups(dataDir: string, appId: string): BackupFile[] {
3843 });
3944 }
4045
41 /** Total bytes of the regular files under `dir`, stopping as soon as it passes `cap`. Sizes only: nothing is read. */
42 export function bytesUnder(dir: string, cap: number): number {
46 /** Bytes free to an unprivileged writer on the disk holding `path`. */
47 export const freeBytes = (path: string): number => {
48 const s = statfsSync(path);
49 return Number(s.bavail) * Number(s.bsize);
50 };
51
52 const mb = (n: number): number => Math.ceil(n / (1024 * 1024));
53
54 /**
55 * Whether each `[directory, bytes]` the work will write fits on its disk with
56 * `SPACE_MARGIN_BYTES` to spare; two directories on one disk add up. One
57 * sentence when one does not, else `undefined`.
58 */
59 export function spaceProblem(needs: [string, number][], free: (path: string) => number = freeBytes): string | undefined {
60 const byDisk = new Map<number, { path: string; bytes: number }>();
61 for (const [path, bytes] of needs) {
62 const dev = statSync(path).dev;
63 const had = byDisk.get(dev);
64 byDisk.set(dev, { path, bytes: (had?.bytes ?? 0) + bytes });
65 }
66 for (const { path, bytes } of byDisk.values()) {
67 const room = free(path);
68 if (room < bytes + SPACE_MARGIN_BYTES)
69 return `not enough disk space: this needs about ${mb(bytes + SPACE_MARGIN_BYTES)} MB and ${mb(Math.max(0, room))} MB are free`;
70 }
71 return undefined;
72 }
73
74 /**
75 * Total bytes of the regular files under `dir`, stopping as soon as it passes `cap`. Sizes only:
76 * nothing is read.
77 */
78 export function bytesUnder(dir: string, cap = Number.POSITIVE_INFINITY): number {
4379 let total = 0;
4480 const stack = [dir];
4581 while (stack.length > 0 && total <= cap) {
@@ -65,7 +101,10 @@ const isProof = (v: unknown): v is Proof => {
65101 );
66102 };
67103
68 /** Run the script with the passphrase in env; its last stdout line is the proof as JSON, its stderr the refusal. */
104 /**
105 * Run the script with the passphrase in env; its last stdout line is the proof as JSON, its stderr
106 * the refusal.
107 */
69108 async function runScript(args: string[], passphrase: string): Promise<Proof | string> {
70109 try {
71110 const child = Bun.spawn([process.execPath, SCRIPT, ...args, "--json"], {
@@ -90,15 +129,24 @@ async function runScript(args: string[], passphrase: string): Promise<Proof | st
90129 }
91130 }
92131
93 export async function sealApp(dataDir: string, appId: string, passphrase: string, now = new Date()): Promise<Proof | string> {
132 const tempDir = (): string => process.env["TMPDIR"] ?? tmpdir();
133
134 export async function sealApp(
135 dataDir: string,
136 appId: string,
137 passphrase: string,
138 now = new Date(),
139 free: (path: string) => number = freeBytes,
140 ): Promise<Proof | string> {
94141 try {
95142 const src = envDataDir(dataDir, appId, "production");
96143 if (!existsSync(src)) return "this app has no production data yet";
97 if (bytesUnder(src, SEAL_CAP_BYTES) > SEAL_CAP_BYTES) {
98 return `the data directory is over ${SEAL_CAP_BYTES / (1024 * 1024)} MB: back it up with scripts/app-backup.ts on the host`;
99 }
100144 const d = dirFor(dataDir, appId);
101145 mkdirSync(d, { recursive: true, mode: 0o700 });
146 const bytes = bytesUnder(src);
147 // the archive beside the others, and its proof restored in the temp dir
148 const short = spaceProblem([[d, bytes], [tempDir(), bytes]], free);
149 if (short !== undefined) return short;
102150 const stamp = now.toISOString().replace(/[:.]/g, "-");
103151 return await runScript(["seal", src, join(d, `production-${stamp}.fab`)], passphrase);
104152 } catch (e) {
@@ -106,13 +154,20 @@ export async function sealApp(dataDir: string, appId: string, passphrase: string
106154 }
107155 }
108156
109 export async function proveApp(dataDir: string, appId: string, name: string, passphrase: string): Promise<Proof | string> {
157 export async function proveApp(
158 dataDir: string,
159 appId: string,
160 name: string,
161 passphrase: string,
162 free: (path: string) => number = freeBytes,
163 ): Promise<Proof | string> {
110164 let scratch: string | undefined;
111165 try {
112166 const at = listBackups(dataDir, appId).find((b) => b.name === name);
113167 if (at === undefined) return "no such backup";
114 if (at.bytes > SEAL_CAP_BYTES * 2) return "this backup is too large to prove here: use scripts/app-backup.ts on the host";
115 scratch = mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "app-prove-"));
168 const short = spaceProblem([[tempDir(), at.bytes]], free);
169 if (short !== undefined) return short;
170 scratch = mkdtempSync(join(tempDir(), "app-prove-"));
116171 return await runScript(["prove", join(dirFor(dataDir, appId), name), join(scratch, "restore")], passphrase);
117172 } catch (e) {
118173 return (e as Error).message;
test/app-backup.test.ts +97 −10
@@ -1,15 +1,41 @@
11 import { Database } from "bun:sqlite";
22 import { describe, expect, test } from "bun:test";
3 import { existsSync, mkdirSync, mkdtempSync, readdirSync, symlinkSync, truncateSync, writeFileSync } from "node:fs";
3 import { createCipheriv, randomBytes, scryptSync } from "node:crypto";
4 import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, symlinkSync, truncateSync, writeFileSync } from "node:fs";
45 import { tmpdir } from "node:os";
56 import { join } from "node:path";
6 import { openBackup, restoreProof, sealAndProve, sealParts, sealWithManifest, sourceDrift as sourceDriftOf } from "../src/apps/backup.ts";
7 import { bytesUnder, listBackups, proveApp, SEAL_CAP_BYTES, sealApp } from "../src/apps/backup_door.ts";
7 import { type Manifest, restoreProof, sealAndProve, sealToFile, sourceDrift as sourceDriftOf } from "../src/apps/backup.ts";
8 import { bytesUnder, listBackups, proveApp, sealApp, spaceProblem } from "../src/apps/backup_door.ts";
89 import { envDataDir } from "../src/apps/data.ts";
910
1011 const PASS = "correct horse battery";
1112 const tmp = () => mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "ab-"));
1213
14 /**
15 * Encrypt a manifest and file bodies as given, in memory, in the archive's format
16 * (`src/apps/backup.ts`): the seam for forging a dishonest archive the honest writer never makes.
17 */
18 function sealParts(manifest: Manifest, bodies: Buffer[], passphrase: string): Buffer {
19 const m = Buffer.from(JSON.stringify(manifest));
20 const len = Buffer.alloc(4);
21 len.writeUInt32BE(m.length);
22 const salt = randomBytes(16);
23 const iv = randomBytes(12);
24 const header = Buffer.from(`${JSON.stringify({ v: 1, kdf: "scrypt", salt: salt.toString("base64url"), iv: iv.toString("base64url") })}\n`);
25 const key = scryptSync(passphrase, salt, 32, { N: 1 << 15, r: 8, p: 1, maxmem: 64 * 1024 * 1024 });
26 const cipher = createCipheriv("aes-256-gcm", key, iv, { authTagLength: 16 });
27 cipher.setAAD(header);
28 const body = Buffer.concat([cipher.update(Buffer.concat([len, m, ...bodies])), cipher.final(), cipher.getAuthTag()]);
29 return Buffer.concat([Buffer.from("FAB1\n"), header, body]);
30 }
31
32 /** Seal to a file and read it back: the bytes, for tests that bend them. */
33 function sealWithManifest(dir: string, pass: string): { sealed: Buffer; manifest: Manifest } {
34 const out = join(tmp(), "b.fab");
35 const manifest = sealToFile(dir, out, pass);
36 return { sealed: readFileSync(out), manifest };
37 }
38
1339 function app(): string {
1440 const d = tmp();
1541 const db = new Database(join(d, "app.db"));
@@ -39,7 +65,7 @@ describe("app backup", () => {
3965 const bent = Buffer.from(sealed);
4066 bent[bent.length - 20] = bent[bent.length - 20]! ^ 1;
4167 expect(restoreProof(bent, PASS, scratch)).toMatch(/wrong passphrase/);
42 expect(openBackup(Buffer.from("junk"), PASS)).toBe("not an app backup file");
68 expect(restoreProof(Buffer.from("junk"), PASS, join(tmp(), "j"))).toBe("not an app backup file");
4369 });
4470 test("a short passphrase is refused up front", () => {
4571 expect(() => sealWithManifest(app(), "short")).toThrow(/under 12/);
@@ -163,18 +189,24 @@ describe("src/apps/backup_door.ts", () => {
163189 mkdirSync(d, { recursive: true });
164190 return d;
165191 };
166 test("a data directory over the cap is refused before it is read, and nothing is written", async () => {
192 test("a seal that would not fit on the disk is refused in one sentence, and nothing is written", async () => {
167193 const forge = tmp();
168194 const d = appData(forge, "big");
169195 mkdirSync(join(d, "deep"), { recursive: true });
170196 writeFileSync(join(d, "deep", "huge.bin"), "");
171 truncateSync(join(d, "deep", "huge.bin"), SEAL_CAP_BYTES + 1); // sparse: no bytes on disk
172 expect(bytesUnder(d, SEAL_CAP_BYTES)).toBeGreaterThan(SEAL_CAP_BYTES);
173 const r = await sealApp(forge, "big", PASS);
174 expect(r).toMatch(/over 64 MB/);
197 truncateSync(join(d, "deep", "huge.bin"), 300 * 1024 * 1024); // sparse: no bytes on disk
198 expect(bytesUnder(d)).toBe(300 * 1024 * 1024);
199 const r = await sealApp(forge, "big", PASS, new Date(), () => 400 * 1024 * 1024);
200 // archive and proof share the disk: 600 MB plus the margin
201 expect(r).toBe("not enough disk space: this needs about 664 MB and 400 MB are free");
175202 expect(listBackups(forge, "big")).toEqual([]);
176203 });
177 test("under the cap it seals in a child process and proves", async () => {
204 test("space on two disks is counted per disk; room enough answers nothing", () => {
205 const a = tmp();
206 expect(spaceProblem([[a, 10], [a, 10]], () => 1 << 30)).toBeUndefined();
207 expect(spaceProblem([[a, 10]], () => 0)).toMatch(/^not enough disk space: .* 0 MB are free$/);
208 });
209 test("with room it seals in a child process and proves", async () => {
178210 const forge = tmp();
179211 writeFileSync(join(appData(forge, "small"), "x.txt"), "hello");
180212 const r = await sealApp(forge, "small", PASS);
@@ -193,3 +225,58 @@ describe("src/apps/backup_door.ts", () => {
193225 expect(await proveApp(forge, "bad", "../../etc/passwd", PASS)).toBe("no such backup");
194226 });
195227 });
228
229 describe("streamed archive", () => {
230 test("a restore of the streamed archive gives the same files, byte for byte", () => {
231 const d = app();
232 mkdirSync(join(d, "big/nested"), { recursive: true });
233 writeFileSync(join(d, "big/nested/blob.bin"), randomBytes(3 * 1024 * 1024 + 17)); // spans chunks
234 writeFileSync(join(d, "empty"), "");
235 const out = join(tmp(), "b.fab");
236 const m = sealToFile(d, out, PASS);
237 const scratch = join(tmp(), "r");
238 const p = restoreProof(out, PASS, scratch);
239 expect(p).toMatchObject({ ok: true, files: 4, rows: 7, problems: [] });
240 for (const f of m.files) expect(readFileSync(join(scratch, f.path)).equals(readFileSync(join(d, f.path)))).toBe(true);
241 });
242 test("a truncated archive file is refused and writes nothing", () => {
243 const out = join(tmp(), "b.fab");
244 writeFileSync(out, sealWithManifest(app(), PASS).sealed.subarray(0, -5));
245 const scratch = join(tmp(), "r");
246 expect(restoreProof(out, PASS, scratch)).toMatch(/wrong passphrase, or the file is damaged/);
247 expect(existsSync(scratch)).toBe(false);
248 });
249 // VmHWM in /proc is the only honest peak: getrusage's maxRSS survives exec and would count the
250 // test runner. Without /proc (not Linux) there is no true number to check, so the test is
251 // skipped.
252 test.skipIf(!existsSync("/proc/self/status"))("a 500 MB app seals and proves with resident memory under 100 MB", async () => {
253 const d = app();
254 for (let i = 0; i < 5; i++) {
255 writeFileSync(join(d, `part${i}.bin`), randomBytes(4096));
256 truncateSync(join(d, `part${i}.bin`), 100 * 1024 * 1024); // sparse: the test costs no disk to set up
257 }
258 const base = tmp();
259 const out = join(base, "b.fab");
260 // The child reports its own peak after sealing and proving: VmHWM, the high-water mark of this
261 // process image, or null when it cannot read it, which fails the test.
262 const script = `import { readFileSync } from "node:fs";
263 import { sealAndProve } from ${JSON.stringify(join(import.meta.dir, "../src/apps/backup.ts"))};
264 const p = sealAndProve(process.argv[1], process.argv[2], process.env.APP_BACKUP_PASSPHRASE);
265 const hwm = /VmHWM:\\s+(\\d+) kB/.exec(readFileSync("/proc/self/status", "utf8"));
266 console.log(JSON.stringify({ ok: p.ok, files: p.files, bytes: p.bytes, maxKB: hwm ? Number(hwm[1]) : null }));`;
267 const child = Bun.spawn(["bun", "-e", script, d, out], {
268 env: { PATH: process.env["PATH"] ?? "", TMPDIR: base, APP_BACKUP_PASSPHRASE: PASS },
269 stdout: "pipe",
270 stderr: "inherit",
271 });
272 const text = await new Response(child.stdout).text();
273 expect(await child.exited).toBe(0);
274 const got = JSON.parse(text.trim()) as { ok: boolean; files: number; bytes: number; maxKB: number | null };
275 rmSync(base, { recursive: true, force: true });
276 rmSync(d, { recursive: true, force: true });
277 expect(got).toMatchObject({ ok: true, files: 7 });
278 expect(got.bytes).toBeGreaterThan(500 * 1024 * 1024);
279 expect(got.maxKB).not.toBeNull();
280 expect((got.maxKB ?? Infinity) * 1024).toBeLessThan(100 * 1024 * 1024);
281 }, 120_000);
282 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs