trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

Platform app move gap: File and upload copy with a manifest #11711

Merged masoncrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/11699-platform-app-move-gap-file-and-upload-co → main 5 commits fork

  • Merged. mason/11699-platform-app-move-gap-file-and-upload-co was merged into main as 90e8569.

Review

Sign in to review this change.
6 files changed +407 −3
docs/features/platform-app-move.md +11 −2
@@ -28,7 +28,16 @@ The decision and both paths are in docs/APPS.md, "What is deliberately not here"
2828 ## 3. Copy the data (person runs, crew writes the script)
2929 Dump with `pg_dump`, load, then compare per-table row counts and a per-table
3030 checksum (`md5` of ordered rows). Counts and checksums go in the issue. Files:
31 copy, then compare the SHA-256 manifest. Secrets: the owner sets each by name
31 copy, then compare the SHA-256 manifest:
32 `bun scripts/copy-app-files.ts <source-dir> [target-dir]` (target defaults to
33 `$FORGE_APP_DATA`) copies regular files, hashes both ends
34 (`src/apps/file_manifest.ts`), prints MISSING / CHANGED / EXTRA lines and exits
35 0 only when the manifests match and nothing was left behind (a target that
36 already holds other files is EXTRA, exit 1). `--manifest <dir>` prints a
37 manifest alone, for the old node. Symlinks and devices are listed as SKIPPED,
38 never followed, and exit 1 unless `--allow-skips` says so. A target holding a
39 symlink is refused.
40 Secrets: the owner sets each by name
3241 in the app's settings; they are never pasted into an issue, a pull or a run.
3342
3443 ## 4. Rehearse on a preview (person)
@@ -71,8 +80,8 @@ counts. The old copy stays until that has passed.
7180
7281 ## What the forge lacks (each is an issue)
7382 - A database import for an app (`migrate` has no data path): upload a dump, load it, report counts and checksums. (#11697)
83 - ~~A file and upload copy with a manifest compared on both ends.~~ Done as a script a person runs (#11699); no page or upload from a browser yet.
7484 - ~~A Postgres app has no home here.~~ Decided, not built: no managed Postgres; port to `bun:sqlite` or keep an external one by `egress` (docs/APPS.md). (#11698)
75 - A file and upload copy with a manifest compared on both ends. (#11699)
7685 - Per-app encrypted backup with a restore proof (`docs/features/backup-truth.md` covers the forge's own, not an app's). (#11701)
7786 - A guard for rehearsal: data copy and `preview`-scoped sandbox keys exist and are the procedure in step 4; nothing yet refuses a payment name left in scope `all`. (#11702)
7887 - Inbound payment webhooks that survive a preview's changing address. (#11703)
scripts/copy-app-files.ts added +44
@@ -0,0 +1,44 @@
1 /**
2 * Copy an app's files and uploads into its FORGE_APP_DATA and prove it:
3 *
4 * bun scripts/copy-app-files.ts [--allow-skips] <source-dir> [target-dir] # target defaults to $FORGE_APP_DATA
5 * bun scripts/copy-app-files.ts --manifest <dir> # print a manifest, e.g. on the old node
6 *
7 * Exit 0: the manifests at both ends match and nothing was left behind.
8 * Exit 1: they differ, or symlinks/devices were skipped (unless --allow-skips).
9 * Exit 2: usage or an unreadable path. Prints counts and the target's
10 * manifest, ready to paste into the migration issue.
11 * See docs/features/platform-app-move.md.
12 */
13
14 import { buildManifest, copyWithManifest, manifestText } from "../src/apps/file_manifest.ts";
15
16 const argv = process.argv.slice(2);
17 const allowSkips = argv.includes("--allow-skips");
18 const args = argv.filter((a) => a !== "--allow-skips");
19 try {
20 if (args[0] === "--manifest" && args[1]) {
21 const m = buildManifest(args[1]);
22 process.stdout.write(manifestText(m));
23 for (const s of m.skipped) console.error(`skipped (not a regular file): ${s}`);
24 process.exit(0);
25 }
26 const from = args[0];
27 const to = args[1] ?? process.env.FORGE_APP_DATA;
28 if (!from || !to) {
29 console.error("usage: bun scripts/copy-app-files.ts [--allow-skips] <source-dir> [target-dir] (target defaults to $FORGE_APP_DATA)");
30 process.exit(2);
31 }
32 const r = copyWithManifest(from, to, { allowSkips });
33 console.log(`copied ${r.files} files, ${r.bytes} bytes`);
34 for (const s of r.skipped) console.log(`SKIPPED (not a regular file, not copied): ${s}`);
35 for (const p of r.diff.missing) console.log(`MISSING ${p}`);
36 for (const p of r.diff.changed) console.log(`CHANGED ${p}`);
37 for (const p of r.diff.extra) console.log(`EXTRA ${p} (already in the target, not in the source)`);
38 console.log(r.ok ? "manifests match" : "NOT PROVEN: manifests differ or files were left behind");
39 if (r.ok) process.stdout.write(manifestText(r.target));
40 process.exit(r.ok ? 0 : 1);
41 } catch (e) {
42 console.error(e instanceof Error ? e.message : String(e));
43 process.exit(2);
44 }
src/actions/migrate.ts +1 −1
@@ -893,7 +893,7 @@ Adapt, don't rewrite:
893893 - Listen on the port in \`PORT\`, on \`HOST\`. A hard-coded port becomes the fallback, not the rule.
894894 - Keep every piece of persistent state — a SQLite file, uploads, a data folder — under \`FORGE_APP_DATA\`. It must start on an empty one and restart on a full one.
895895 - The forge runs processes, not containers: no Dockerfile, no compose file. Read those only to learn what the app needs. A second service it cannot do without (a database server, a queue) becomes the app's embedded or file mode where it has one.
896 - Its data: if it used a database server elsewhere (Postgres, MySQL), say in the pull where the rows are and what it uses now. Leave the old database untouched. Move the app's state under \`FORGE_APP_DATA\` (bun:sqlite, or the app's embedded mode). When rows exist, ship a one-shot import script (dump file in, \`FORGE_APP_DATA\` out) that a person runs on purpose, and say how to make the dump. Never commit or read at boot a connection string to the old database.
896 - Its data: if it used a database server elsewhere (Postgres, MySQL), say in the pull where the rows are and what it uses now. Leave the old database untouched. Move the app's state under \`FORGE_APP_DATA\` (bun:sqlite, or the app's embedded mode). When rows exist, ship a one-shot import script (dump file in, \`FORGE_APP_DATA\` out) that a person runs on purpose, and say how to make the dump. Files and uploads are copied by a person with \`bun scripts/copy-app-files.ts <source-dir>\`, which compares a SHA-256 manifest at both ends; the app only reads them from \`FORGE_APP_DATA\`. Never commit or read at boot a connection string to the old database.
897897 - Keep its routes, pages and behaviour as they were. The smallest diff a person can review quickly — no refactors, no restyling, no dependency churn beyond what running here needs.
898898 - Write down what could not fit — a service the forge cannot run, a secret you do not have — plainly, in the pull, so a person can decide. An honest blocker beats a broken deploy.
899899
src/apps/file_manifest.ts added +183
@@ -0,0 +1,183 @@
1 /**
2 * Files moved with a proof: a SHA-256 manifest of a tree, a copy of it into
3 * `FORGE_APP_DATA`, and a comparison of the manifests at both ends
4 * (docs/features/platform-app-move.md, step 3). Plain data in, plain data out.
5 *
6 * Only regular files and directories move. A symlink or device is never
7 * followed or copied: it is listed as `skipped`, so the person sees what was
8 * left behind instead of what a link happened to point at.
9 */
10
11 import { createHash } from "node:crypto";
12 import {
13 closeSync,
14 copyFileSync,
15 lstatSync,
16 mkdirSync,
17 openSync,
18 readdirSync,
19 readSync,
20 realpathSync,
21 } from "node:fs";
22 import { dirname, join, resolve, sep } from "node:path";
23
24 export interface ManifestEntry {
25 /** Path relative to the tree's root, `/`-separated. */
26 path: string;
27 bytes: number;
28 sha256: string;
29 }
30
31 export interface Manifest {
32 entries: ManifestEntry[];
33 /** Relative paths of anything that is not a regular file or directory. */
34 skipped: string[];
35 }
36
37 export interface ManifestDiff {
38 missing: string[];
39 extra: string[];
40 changed: string[];
41 }
42
43 /** SHA-256 of one file, read in chunks so a large upload never sits in memory. */
44 export function hashFile(path: string): string {
45 const hash = createHash("sha256");
46 const buf = Buffer.allocUnsafe(1 << 16);
47 const fd = openSync(path, "r");
48 try {
49 for (;;) {
50 const n = readSync(fd, buf, 0, buf.length, null);
51 if (n === 0) break;
52 hash.update(buf.subarray(0, n));
53 }
54 } finally {
55 closeSync(fd);
56 }
57 return hash.digest("hex");
58 }
59
60 /** The manifest of `root`: every regular file, sorted by path. */
61 export function buildManifest(root: string): Manifest {
62 const entries: ManifestEntry[] = [];
63 const skipped: string[] = [];
64 const walk = (rel: string): void => {
65 const dir = rel === "" ? root : join(root, rel);
66 for (const name of readdirSync(dir).sort()) {
67 const childRel = rel === "" ? name : `${rel}/${name}`;
68 const stat = lstatSync(join(root, childRel));
69 if (stat.isDirectory()) walk(childRel);
70 else if (stat.isFile()) {
71 entries.push({ path: childRel, bytes: stat.size, sha256: hashFile(join(root, childRel)) });
72 } else skipped.push(childRel);
73 }
74 };
75 walk("");
76 entries.sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0));
77 return { entries, skipped };
78 }
79
80 /** The manifest as text, `sha256 bytes path` a line: what goes in the issue, and what `sha256sum` style tools can check. */
81 export function manifestText(manifest: Manifest): string {
82 return manifest.entries.map((e) => `${e.sha256} ${e.bytes} ${e.path}`).join("\n") + (manifest.entries.length ? "\n" : "");
83 }
84
85 /** What differs between the source's manifest and the target's. Empty on all three means the copy is proven. */
86 export function compareManifests(source: Manifest, target: Manifest): ManifestDiff {
87 const have = new Map(target.entries.map((e) => [e.path, e]));
88 const want = new Set(source.entries.map((e) => e.path));
89 const missing: string[] = [];
90 const changed: string[] = [];
91 for (const e of source.entries) {
92 const there = have.get(e.path);
93 if (!there) missing.push(e.path);
94 else if (there.sha256 !== e.sha256 || there.bytes !== e.bytes) changed.push(e.path);
95 }
96 const extra = target.entries.filter((e) => !want.has(e.path)).map((e) => e.path);
97 return { missing, extra, changed };
98 }
99
100 export const manifestsMatch = (d: ManifestDiff): boolean =>
101 d.missing.length === 0 && d.extra.length === 0 && d.changed.length === 0;
102
103 export interface CopyReport {
104 files: number;
105 bytes: number;
106 skipped: string[];
107 source: Manifest;
108 target: Manifest;
109 diff: ManifestDiff;
110 /** True only when both manifests match and nothing was left behind (or `allowSkips` said that is fine). */
111 ok: boolean;
112 }
113
114 /** `path` with its deepest existing ancestor resolved through symlinks. */
115 function realish(path: string): string {
116 const abs = resolve(path);
117 const rest: string[] = [];
118 let at = abs;
119 for (;;) {
120 try {
121 return join(realpathSync(at), ...rest.reverse());
122 } catch {
123 const up = dirname(at);
124 if (up === at) return abs;
125 rest.push(at.slice(up.length + 1));
126 at = up;
127 }
128 }
129 }
130
131 /** Throw when a path that already exists under `dst` for any entry is a symlink: a write would go through it. */
132 function refuseTargetLinks(dst: string, paths: string[]): void {
133 const isLink = (p: string): boolean => {
134 try {
135 return lstatSync(p).isSymbolicLink();
136 } catch {
137 return false;
138 }
139 };
140 if (isLink(dst)) throw new Error("the target is a symlink");
141 for (const rel of paths) {
142 let at = dst;
143 for (const part of rel.split("/")) {
144 at = join(at, part);
145 if (isLink(at)) throw new Error(`the target holds a symlink at ${rel}`);
146 }
147 }
148 }
149
150 /**
151 * Copy `from`'s regular files into `to`, then hash both ends and compare.
152 * Refuses, before writing anything, a target nested in the source (or the
153 * reverse) once symlinks are resolved, and a target that holds a symlink on
154 * any path it would write. Files left behind (symlinks, devices) make `ok`
155 * false unless `allowSkips` is set.
156 */
157 export function copyWithManifest(from: string, to: string, opts: { allowSkips?: boolean } = {}): CopyReport {
158 if (!lstatSync(from).isDirectory()) throw new Error("the source is not a directory");
159 const src = realpathSync(from);
160 const dst = realish(to);
161 if (src === dst || dst.startsWith(src + sep) || src.startsWith(dst + sep)) {
162 throw new Error("the source and the target must be separate directories");
163 }
164 const source = buildManifest(src);
165 refuseTargetLinks(resolve(to), source.entries.map((e) => e.path));
166 mkdirSync(dst, { recursive: true });
167 for (const e of source.entries) {
168 const out = join(dst, e.path);
169 mkdirSync(dirname(out), { recursive: true });
170 copyFileSync(join(src, e.path), out);
171 }
172 const target = buildManifest(dst);
173 const diff = compareManifests(source, target);
174 return {
175 files: source.entries.length,
176 bytes: source.entries.reduce((n, e) => n + e.bytes, 0),
177 skipped: source.skipped,
178 source,
179 target,
180 diff,
181 ok: manifestsMatch(diff) && (opts.allowSkips === true || source.skipped.length === 0),
182 };
183 }
test/copy-app-files-script.test.ts added +82
@@ -0,0 +1,82 @@
1 import { afterEach, expect, test } from "bun:test";
2 import { mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
3 import { tmpdir } from "node:os";
4 import { join } from "node:path";
5
6 const made: string[] = [];
7 const tmp = (): string => {
8 const d = mkdtempSync(join(tmpdir(), "copyfiles-"));
9 made.push(d);
10 return d;
11 };
12 afterEach(() => {
13 for (const d of made.splice(0)) rmSync(d, { recursive: true, force: true });
14 });
15
16 const script = join(import.meta.dir, "..", "scripts", "copy-app-files.ts");
17 async function run(args: string[], env: Record<string, string> = {}) {
18 const p = Bun.spawn(["bun", script, ...args], {
19 stdout: "pipe",
20 stderr: "pipe",
21 env: { ...process.env, FORGE_APP_DATA: "", ...env },
22 });
23 const [out, err] = await Promise.all([new Response(p.stdout).text(), new Response(p.stderr).text()]);
24 return { code: await p.exited, out, err };
25 }
26
27 function source(): string {
28 const d = tmp();
29 mkdirSync(join(d, "up"));
30 writeFileSync(join(d, "up/a.txt"), "hello");
31 return d;
32 }
33
34 test("exit 0 and the manifest when the copy matches", async () => {
35 const to = join(tmp(), "data");
36 const r = await run([source(), to]);
37 expect(r.code).toBe(0);
38 expect(r.out).toContain("manifests match");
39 expect(r.out).toContain("2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 5 up/a.txt");
40 });
41
42 test("the target defaults to FORGE_APP_DATA", async () => {
43 const to = join(tmp(), "data");
44 const r = await run([source()], { FORGE_APP_DATA: to });
45 expect(r.code).toBe(0);
46 });
47
48 test("exit 1 and an EXTRA line when the target holds other files", async () => {
49 const to = tmp();
50 writeFileSync(join(to, "other.db"), "x");
51 const r = await run([source(), to]);
52 expect(r.code).toBe(1);
53 expect(r.out).toContain("EXTRA other.db");
54 expect(r.out).toContain("NOT PROVEN");
55 });
56
57 test("exit 1 when a symlink was left behind, 0 with --allow-skips", async () => {
58 const from = source();
59 symlinkSync("/etc/passwd", join(from, "link"));
60 const r = await run([from, join(tmp(), "data")]);
61 expect(r.code).toBe(1);
62 expect(r.out).toContain("SKIPPED (not a regular file, not copied): link");
63 expect((await run(["--allow-skips", from, join(tmp(), "data")])).code).toBe(0);
64 });
65
66 test("exit 2 on usage and on a missing directory, with a message and no stack", async () => {
67 const usage = await run([]);
68 expect(usage.code).toBe(2);
69 expect(usage.err).toContain("usage:");
70 for (const args of [["--manifest", join(tmp(), "nope")], [join(tmp(), "nope"), tmp()]]) {
71 const r = await run(args);
72 expect(r.code).toBe(2);
73 expect(r.err).not.toContain(" at ");
74 expect(r.err.length).toBeGreaterThan(0);
75 }
76 });
77
78 test("--manifest prints the manifest alone", async () => {
79 const r = await run(["--manifest", source()]);
80 expect(r.code).toBe(0);
81 expect(r.out).toBe("2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 5 up/a.txt\n");
82 });
test/file-manifest.test.ts added +86
@@ -0,0 +1,86 @@
1 import { afterEach, expect, test } from "bun:test";
2 import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
3 import { tmpdir } from "node:os";
4 import { join } from "node:path";
5 import { buildManifest, compareManifests, copyWithManifest, manifestText } from "../src/apps/file_manifest.ts";
6
7 const made: string[] = [];
8 const tmp = (): string => {
9 const d = mkdtempSync(join(tmpdir(), "manifest-"));
10 made.push(d);
11 return d;
12 };
13 afterEach(() => {
14 for (const d of made.splice(0)) rmSync(d, { recursive: true, force: true });
15 });
16
17 function tree(): string {
18 const d = tmp();
19 mkdirSync(join(d, "up/2024"), { recursive: true });
20 writeFileSync(join(d, "a.txt"), "hello");
21 writeFileSync(join(d, "up/2024/b.bin"), Buffer.from([0, 1, 2, 255]));
22 return d;
23 }
24
25 test("the manifest lists regular files sorted with their sha-256", () => {
26 const m = buildManifest(tree());
27 expect(m.entries.map((e) => e.path)).toEqual(["a.txt", "up/2024/b.bin"]);
28 expect(m.entries[0]!.sha256).toBe("2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824");
29 expect(manifestText(m).split("\n")[0]).toBe(`${m.entries[0]!.sha256} 5 a.txt`);
30 });
31
32 test("a copy is proven by both manifests", () => {
33 const r = copyWithManifest(tree(), join(tmp(), "data"));
34 expect(r.ok).toBe(true);
35 expect(r.files).toBe(2);
36 expect(r.bytes).toBe(9);
37 });
38
39 test("a symlink is skipped, never followed, and the copy is not proven", () => {
40 const from = tree();
41 symlinkSync("/etc/passwd", join(from, "link"));
42 const r = copyWithManifest(from, join(tmp(), "data"));
43 expect(r.skipped).toEqual(["link"]);
44 expect(r.diff).toEqual({ missing: [], extra: [], changed: [] });
45 expect(r.ok).toBe(false);
46 expect(r.target.entries.map((e) => e.path)).not.toContain("link");
47 expect(copyWithManifest(from, join(tmp(), "data"), { allowSkips: true }).ok).toBe(true);
48 });
49
50 test("the comparison names missing, changed and extra files", () => {
51 const a = buildManifest(tree());
52 const bDir = tree();
53 writeFileSync(join(bDir, "a.txt"), "HELLO");
54 writeFileSync(join(bDir, "new.txt"), "x");
55 const b = buildManifest(bDir);
56 b.entries = b.entries.filter((e) => e.path !== "up/2024/b.bin");
57 expect(compareManifests(a, b)).toEqual({ missing: ["up/2024/b.bin"], extra: ["new.txt"], changed: ["a.txt"] });
58 });
59
60 test("a target inside the source is refused", () => {
61 const from = tree();
62 expect(() => copyWithManifest(from, join(from, "inside"))).toThrow();
63 });
64
65 test("a target that is a symlink into the source is refused", () => {
66 const from = tree();
67 const link = join(tmp(), "alias");
68 symlinkSync(from, link);
69 expect(() => copyWithManifest(from, link)).toThrow();
70 expect(() => copyWithManifest(from, join(link, "deeper"))).toThrow();
71 });
72
73 test("a target holding a symlinked directory or file is refused before anything is written", () => {
74 const outside = tmp();
75 const to = tmp();
76 symlinkSync(outside, join(to, "up"));
77 expect(() => copyWithManifest(tree(), to)).toThrow(/symlink/);
78 expect(existsSync(join(outside, "2024"))).toBe(false);
79 expect(existsSync(join(to, "a.txt"))).toBe(false);
80
81 const to2 = tmp();
82 writeFileSync(join(outside, "victim"), "keep");
83 symlinkSync(join(outside, "victim"), join(to2, "a.txt"));
84 expect(() => copyWithManifest(tree(), to2)).toThrow(/symlink/);
85 expect(readFileSync(join(outside, "victim"), "utf8")).toBe("keep");
86 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs