Public: anyone can read this; the crew can change it.

Data migration: a platform app's Postgres goes into the drop and then into the forge app's store, with a receipt #13473

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

torque/forge:torque/647-data-migration-a-platform-app-s-postgres → main 5 commits fork 3 threads · 3 unresolved

  • Merged. torque/647-data-migration-a-platform-app-s-postgres was merged into main as 065f7a1.

Review

Sign in to review this change.
32 files changed +1220 −11
docs/MCP.md +1
@@ -154,6 +154,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
154154 | `map` | — | An app's plan: ready or waiting, critical path, crew, parkedTwice; `done` adds closed; `first: N` ranks N first | 521 |
155155 | `merge` | `land` | Land a pull, as its page's Merge button does; a person's only | 362 |
156156 | `migrate` | — | Bring a repo in by public `url` (`name`, `crew`), history and all | 315 |
157 | `migrate_data` | — | Move a platform app's Postgres dump from its drop into its store; dry-run unless `apply` | 299 |
157158 | `milestones` | — | How far each ask is | 439 |
158159 | `mint_invite` | — | Mint a one-use invite link, shown once, valid `days` (1–30, default 7) | 225 |
159160 | `mirror_now` | — | Admin: bring the snapshot mirror level now | 129 |
docs/features/README.md +1
@@ -35,6 +35,7 @@ Read with the guide page [What a forge is](../guide-what-a-forge-is.md).
3535 - [Looks of `/` signed out](home-looks.md)
3636 - [Milestones at the front door](front-milestones.md)
3737 - [Migration mode: a crew whose standing job is parity](migration-mode.md)
38 - [Data move: a platform app's Postgres through the drop into the app's store, with a receipt and an undo](data-move.md)
3839 - [Now pages and Repositories](now-pages.md)
3940 - [Pixel art: the crew's faces](pixel.md)
4041 - [Plans on the home](home-plans.md)
docs/features/data-move.md added +45
@@ -0,0 +1,45 @@
1 # Moving a platform app's data (#647)
2
3 A platform app's Postgres goes through the app's drop into the forge app's own store, with a
4 receipt, and can be taken back until a domain is cut over. It is the data line of the migration
5 checklist (`docs/features/migration-mode.md`); the engine under it is the database import in
6 `docs/features/platform-app-move.md`.
7
8 ## The path
9 1. **Make the dump** where the old database is reachable. The page shows the one command
10 (`EXPORT_COMMAND`, `src/apps/pg_dump.ts`): `pg_dump --inserts --no-owner --no-privileges
11 --no-comments "$DATABASE_URL" > dump.sql`. The connection string stays on that machine; the forge
12 never receives it. (The forge running `pg_dump` itself with a connection string given once was
13 left out: it needs a Postgres client and an egress entry to a host the owner names.)
14 2. **Put it in the drop** (`/<owner>/<name>/drop`, up to 16 MB for an import).
15 3. **Check, then import**, at `/<owner>/<name>/data-move`, reached from **Import data** on the app
16 page's migration card. Check does everything but put the file in place; Import first keeps the
17 store it replaces as `app.db.before-import-<ms>` (the snapshot), then replaces it.
18 4. **The receipt**: per table the rows and SHA-256 on the dump's side and on the store's, and the
19 rows counted from the dump's own text. Import happens only when all agree. What a Postgres dump
20 has that SQLite has no meaning for (session `SET`s, `set_config`, `setval`, ownership,
21 column defaults, `ALTER TABLE` constraints, sequences, comments, extensions, schemas) is left out, counted, and
22 shown on the receipt. **Primary, unique and foreign keys are not kept**: the tables load without them, and the receipt says so; `public.` is dropped from names. `COPY` is refused: dump with `--inserts`.
23 Anything else the import's vetting refuses by statement number (`SELECT`, functions, `UPDATE`).
24 5. **Undo** puts the snapshot back and sets the imported store aside as
25 `app.db.undone-import-<ms>`; nothing is deleted. When there was no store before, the imported one
26 is set aside and the app starts empty as it was. It is open until a domain of the app is verified
27 and has its old DNS recorded (the cut-over, `app_domain.old_records`); then an import and an undo
28 are both refused, since the old data is no longer the way back.
29
30 The record is the `app.data_imported` and `app.data_undone` events (the first carries the snapshot's
31 name and each table's rows and checksum): no state of its own.
32
33 ## Doors (all a repository admin's; anyone else gets not found)
34 - Page `/<owner>/<name>/data-move` (GET, POST `do=check|import|undo`, same-origin checked).
35 - API `GET` and `POST /api/v1/repos/<owner>/<name>/data-move`: `{ dump }` checks, `{ dump, apply: true }`
36 imports, `{ undo: true }` undoes; an admin-scope token for POST.
37 - MCP `migrate_data { repo, dump?, apply?, undo? }`: only `repo` answers the command, the drop's
38 dumps and the last receipt; `dump` is a dry run; `apply`/`undo` need an admin-scope token.
39 - Terminal: the "Move the data" screen (`src/tui/screens/datamove.ts`): `m` checks a drop file,
40 `a` imports it, `u` undoes after a y/n.
41
42 ## Not done
43 - The running app keeps its old file open until it restarts: restart it after an import or an undo.
44 - Only the store `app.db` in production; files and uploads are `scripts/copy-app-files.ts`.
45 - Types are not translated: SQLite keeps the Postgres type names as declared and is loose about them.
docs/features/platform-app-move.md +2 −1
@@ -75,7 +75,8 @@ Not done: the load runs on the request, not in a worker. With only literal value
7575 its time is linear in the dump's bytes (about a second near 16 MB), which is why a worker with a
7676 timeout was left out; the running app keeps its old file open until it restarts, so import with the app
7777 stopped or restart it after; the proof shows the load agrees with itself and with the dump's row count,
78 not that the old node's dump was complete: compare its row counts too. No web page yet.
78 not that the old node's dump was complete: compare its row counts too. A Postgres dump read from
79 the app's drop, with a page, a receipt and an undo: `docs/features/data-move.md` (#647).
7980
8081 ## 4. Rehearse on a preview (person)
8182 Preview address, a copy of the data, the payment **sandbox** keys. The owner,
forge.json +2 −2
@@ -4,8 +4,8 @@
44 "api.github.com"
55 ],
66 "app": {
7 "looks": ["/", "/docs", "/explore", "/board", "/notes", "/review", "/releases", "/apps", "/security", "/system", "/maintenance", "/talk", "/flow", "/resilience", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"],
8 "looksAsOwner": ["/", "/board", "/flow", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/talk", "/system", "/maintenance", "/resilience", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"],
7 "looks": ["/", "/docs", "/explore", "/board", "/notes", "/review", "/releases", "/apps", "/security", "/system", "/maintenance", "/talk", "/flow", "/resilience", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"],
8 "looksAsOwner": ["/", "/board", "/flow", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/talk", "/system", "/maintenance", "/resilience", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"],
99 "run": "bun run src/main.ts",
1010 "install": "",
1111 "health": "/healthz",
src/actions/migrate.ts +8 −1
@@ -953,6 +953,8 @@ export function importAppData(
953953 readonly env?: string;
954954 readonly file?: string;
955955 readonly replace?: boolean;
956 /** Everything but putting the file in place; no event, nothing changed (#647). */
957 readonly dryRun?: boolean;
956958 },
957959 ): Result<ImportReport, ImportDataError> {
958960 const user = requireUser(actor);
@@ -969,8 +971,12 @@ export function importAppData(
969971 file,
970972 input.dump,
971973 input.replace === true,
974 undefined,
975 Date.now(),
976 input.dryRun === true,
972977 );
973978 if (!done.ok) return err(done.error);
979 if (input.dryRun === true) return ok(done.value);
974980 storageStale(app, repo.ownerId);
975981 app.bus.append({
976982 kind: "app.data_imported",
@@ -979,7 +985,8 @@ export function importAppData(
979985 payload: {
980986 env,
981987 file,
982 tables: done.value.tables.map((t) => ({ table: t.table, rows: t.rows, sha256: t.sha256 })),
988 kept: done.value.kept ?? null,
989 tables: done.value.tables.map((t) => ({ table: t.table, rows: t.rows, sha256: t.sha256, store: t.store?.sha256 ?? null })),
983990 },
984991 });
985992 return ok(done.value);
src/actions/migrate_data.ts added +237
@@ -0,0 +1,237 @@
1 /**
2 * Moving a platform app's data into the forge app's own store (#647), one flow for the page, the
3 * API, the MCP `migrate_data` tool and the terminal: the owner runs one shown command on the old
4 * database (`apps/pg_dump.ts`) and puts the dump in the app's drop; `moveData` reads it from there
5 * (dry-run unless `apply`), loads it into the app's production SQLite and answers the receipt —
6 * per table, rows and a SHA-256 on the dump's side and on the store's — and `undoMove` puts the
7 * store back as the snapshot taken before the import, until a domain has been cut over. Admins of
8 * the repository only; anyone else is told `not_found`. The record of what was imported is the
9 * `app.data_imported` / `app.data_undone` events, so there is no state of its own.
10 */
11
12 import { readFileSync, statSync } from "node:fs";
13 import type { App } from "../app.ts";
14 import { envDataDir } from "../apps/data.ts";
15 import { DUMP_MAX } from "../apps/dump_file.ts";
16 import type { ImportReport } from "../apps/dump_import.ts";
17 import { undoImport } from "../apps/dump_undo.ts";
18 import { EXPORT_COMMAND, fromPostgres, skippedWords, type Skipped } from "../apps/pg_dump.ts";
19 import { getAppByRepo } from "../core/apps.ts";
20 import { listDomains } from "../core/domains.ts";
21 import { listEvents } from "../core/events.ts";
22 import { canAdmin, type Repo } from "../core/repos.ts";
23 import type { User } from "../core/users.ts";
24 import { filePath, listFiles, validName } from "../drop/store.ts";
25 import { err, ok, type Result } from "../kernel/result.ts";
26 import { requireUser } from "./access.ts";
27 import { importAppData, importDataMessage, type ImportDataError } from "./migrate.ts";
28
29 export type MoveError =
30 | ImportDataError
31 | "invalid_dump"
32 | "missing"
33 | "too_large_dump"
34 | "copy"
35 | "cut_over"
36 | "nothing_to_undo"
37 | "snapshot_gone"
38 | "store_gone";
39
40 /** One table as the receipt shows it. */
41 export interface ReceiptRow {
42 readonly table: string;
43 readonly rows: number;
44 /** SHA-256, hex, of the rows as the dump holds them. */
45 readonly dump: string;
46 /** The same as the store holds them; equal to `dump` or the move did not happen. */
47 readonly store: string;
48 /** Rows the dump's own text says it inserts, counted apart from SQLite. */
49 readonly counted: number;
50 }
51
52 export interface MoveReceipt {
53 readonly applied: boolean;
54 /** The drop file the dump was read from. */
55 readonly dump: string;
56 readonly file: string;
57 readonly statements: number;
58 readonly tables: readonly ReceiptRow[];
59 /** What a Postgres dump had that SQLite has no meaning for, left out and counted. */
60 readonly skipped: readonly Skipped[];
61 /** The snapshot the import took of the store it replaced; null when there was none. */
62 readonly snapshot: string | null;
63 }
64
65 const rowsOf = (report: ImportReport): ReceiptRow[] =>
66 report.tables.map((t) => ({
67 table: t.table,
68 rows: t.rows,
69 dump: t.sha256,
70 store: t.store?.sha256 ?? "",
71 counted: t.inserted,
72 }));
73
74 /** A domain whose old DNS was written down is the cut-over: the old data's way back is closed. */
75 export function cutOverOf(app: App, appId: string): boolean {
76 return listDomains(app.db, appId).some((d) => d.verifiedAt !== null && d.oldRecords.length > 0);
77 }
78
79 /**
80 * Read `dump` (a name in the app's drop) and load it into the production store. Without `apply`
81 * it goes as far as the proof and puts nothing in place: the receipt is the one apply would give.
82 * With it, the store being replaced is kept first, and `undoMove` can restore it.
83 */
84 export function moveData(
85 app: App,
86 actor: User | undefined,
87 repo: Repo,
88 input: { readonly dump: string; readonly apply?: boolean },
89 ): Result<MoveReceipt, MoveError> {
90 const user = requireUser(actor);
91 if (!user.ok) return user;
92 if (!canAdmin(app.db, repo, user.value)) return err("not_found");
93 const record = getAppByRepo(app.db, repo.id);
94 if (!record) return err("no_app");
95 if (!validName(input.dump)) return err("invalid_dump");
96 const path = filePath(app.config.dataDir, repo.id, input.dump);
97 if (path === undefined) return err("missing");
98 // The file may be removed between the listing and this read: that is `missing`, not a throw.
99 let text: string;
100 try {
101 if (statSync(path).size > DUMP_MAX) return err("too_large_dump");
102 text = readFileSync(path, "utf8");
103 } catch {
104 return err("missing");
105 }
106 const read = fromPostgres(text);
107 if (!read.ok) return err(read.error);
108 const apply = input.apply === true;
109 if (apply && cutOverOf(app, record.id)) return err("cut_over");
110 const done = importAppData(app, user.value, repo, { dump: read.sql, replace: true, dryRun: !apply });
111 if (!done.ok) return done;
112 return ok({
113 applied: apply,
114 dump: input.dump,
115 file: done.value.file,
116 statements: done.value.statements,
117 tables: rowsOf(done.value),
118 skipped: read.skipped,
119 snapshot: done.value.kept ?? null,
120 });
121 }
122
123 export interface LastImport {
124 readonly at: string;
125 readonly file: string;
126 readonly snapshot: string | null;
127 /** `store` is the store's own hash when the import recorded it; null on an older record. */
128 readonly tables: readonly { readonly table: string; readonly rows: number; readonly sha256: string; readonly store?: string | null }[];
129 /** Taken back since. */
130 readonly undone: boolean;
131 }
132
133 export interface DataMoveState {
134 /** The one command that makes the dump. */
135 readonly command: string;
136 /** `.sql` files in the drop, newest first: what `moveData` may read. */
137 readonly files: readonly { readonly name: string; readonly bytes: number }[];
138 readonly last: LastImport | undefined;
139 readonly cutOver: boolean;
140 /** An import is there to take back, and nothing has been cut over. */
141 readonly canUndo: boolean;
142 }
143
144 interface Imported {
145 readonly id: string;
146 readonly at: string;
147 readonly file: string;
148 readonly snapshot: string | null;
149 readonly tables: LastImport["tables"];
150 }
151
152 /** The latest production import event of this repository, with the rows it recorded. */
153 function lastImported(app: App, repo: Repo): Imported | undefined {
154 for (const e of listEvents(app.db, { repoId: repo.id, kind: "app.data_imported", limit: 20 })) {
155 const p = e.payload;
156 if (p["env"] !== "production" || typeof p["file"] !== "string") continue;
157 return {
158 id: e.id,
159 at: e.createdAt,
160 file: p["file"],
161 snapshot: typeof p["kept"] === "string" ? p["kept"] : null,
162 tables: Array.isArray(p["tables"]) ? (p["tables"] as LastImport["tables"]) : [],
163 };
164 }
165 return undefined;
166 }
167
168 export function dataMoveState(app: App, actor: User | undefined, repo: Repo): Result<DataMoveState, "not_found"> {
169 if (actor === undefined || !canAdmin(app.db, repo, actor)) return err("not_found");
170 const record = getAppByRepo(app.db, repo.id);
171 if (!record) return err("not_found");
172 const imported = lastImported(app, repo);
173 const undoneAt = listEvents(app.db, { repoId: repo.id, kind: "app.data_undone", limit: 1 })[0]?.id;
174 const undone = imported !== undefined && undoneAt !== undefined && undoneAt > imported.id;
175 const cutOver = cutOverOf(app, record.id);
176 return ok({
177 command: EXPORT_COMMAND,
178 files: listFiles(app.config.dataDir, repo.id)
179 .filter((f) => /\.sql$/i.test(f.name))
180 .map((f) => ({ name: f.name, bytes: f.bytes })),
181 last: imported === undefined ? undefined : { ...imported, undone },
182 cutOver,
183 canUndo: imported !== undefined && !undone && !cutOver,
184 });
185 }
186
187 /** Put the store back as it was before the last import, until a domain has been cut over. */
188 export function undoMove(
189 app: App,
190 actor: User | undefined,
191 repo: Repo,
192 ): Result<{ readonly file: string; readonly restored: boolean; readonly setAside: string }, MoveError> {
193 const user = requireUser(actor);
194 if (!user.ok) return user;
195 const state = dataMoveState(app, user.value, repo);
196 if (!state.ok) return state;
197 const record = getAppByRepo(app.db, repo.id)!;
198 if (state.value.cutOver) return err("cut_over");
199 const last = state.value.last;
200 if (last === undefined || last.undone) return err("nothing_to_undo");
201 const done = undoImport(envDataDir(app.config.dataDir, record.id, "production"), last.file, last.snapshot);
202 if (!done.ok) return err(done.error === "missing_snapshot" ? "snapshot_gone" : done.error === "missing_store" ? "store_gone" : "invalid_dump");
203 app.bus.append({
204 kind: "app.data_undone",
205 repoId: repo.id,
206 actorId: user.value.id,
207 payload: { env: "production", file: last.file, restored: done.value.restored, setAside: done.value.setAside },
208 });
209 return ok({ file: last.file, restored: done.value.restored, setAside: done.value.setAside });
210 }
211
212 /** A refusal in words a person can act on. */
213 export function moveMessage(error: MoveError): string {
214 if (typeof error !== "string" || ["unauthorized", "not_found", "no_app", "invalid_env", "invalid_file"].includes(error))
215 return importDataMessage(error as ImportDataError);
216 switch (error) {
217 case "invalid_dump":
218 return "dump is the name of a file in the app's drop.";
219 case "missing":
220 return "No such file in the app's drop: put the dump there first.";
221 case "too_large_dump":
222 return "The dump is over 16 MB.";
223 case "copy":
224 return "This dump uses COPY. Dump again with --inserts: the shown command does.";
225 case "cut_over":
226 return "A domain has been cut over, so the old data is no longer the way back: nothing was changed.";
227 case "nothing_to_undo":
228 return "There is no import to take back.";
229 case "snapshot_gone":
230 return "The snapshot taken before the import is gone from the app's data, so it cannot be restored.";
231 case "store_gone":
232 return "The store the import made is gone from the app's data: nothing to put back.";
233 }
234 return importDataMessage(error);
235 }
236
237 export { skippedWords };
src/apps/dump_import.ts +7 −1
@@ -204,7 +204,9 @@ const tableKey = (raw: string): string => {
204204 return inner.toLowerCase();
205205 };
206206
207 /** Rows each table should hold, counted from the dump's own text: the independent side of the proof. */
207 /**
208 * Rows each table should hold, counted from the dump's own text: the independent side of the proof.
209 */
208210 export function insertedRows(statements: readonly string[]): Map<string, number> {
209211 const counts = new Map<string, number>();
210212 for (const s of statements) {
@@ -318,6 +320,7 @@ function backupName(dir: string, file: string, now: number): string {
318320 * `replace` is set, else refused. The target is never absent: the old file
319321 * is hard-linked to its backup, then the new one is renamed over it.
320322 * `afterLoad` is a seam for tests: it may touch the store before the proof.
323 * `dryRun` does all of it but the last step: the receipt is the real one, the target untouched.
321324 */
322325 export function importDump(
323326 dir: string,
@@ -326,6 +329,7 @@ export function importDump(
326329 replace: boolean,
327330 afterLoad?: (store: Database) => void,
328331 now: number = Date.now(),
332 dryRun = false,
329333 ): { ok: true; value: ImportReport } | { ok: false; error: ImportError } {
330334 const statements = splitSql(sql);
331335 if (statements.length === 0) return { ok: false, error: { error: "empty" } };
@@ -352,6 +356,8 @@ export function importDump(
352356 const tables = compareTables(fingerprint(scratch), fingerprint(store), insertedRows(statements));
353357 if (tables.some((t) => !t.match) || fingerprint(store).length !== tables.length)
354358 return { ok: false, error: { error: "mismatch", tables } };
359 // A dry run stops here: everything above ran, nothing is put in place (#647).
360 if (dryRun) return { ok: true, value: { file, statements: statements.length, tables } };
355361 store.exec("PRAGMA wal_checkpoint(TRUNCATE)");
356362 store.close();
357363 let kept: string | undefined;
src/apps/dump_undo.ts added +42
@@ -0,0 +1,42 @@
1 /**
2 * Taking an import back (#647). An import with `replace` keeps the store it replaced as
3 * `<file>.before-import-<ms>` (`dump_import.ts`): that is the snapshot. Undo puts it back as the
4 * store and sets the imported store aside as `<file>.undone-import-<ms>`, so nothing is deleted by
5 * either step. When the import found no store, there is no snapshot and the imported one is set
6 * aside, which leaves the app as empty as it was. Names are checked against what an import writes,
7 * so a name from a record is never a path out of the directory.
8 */
9
10 import { Database } from "bun:sqlite";
11 import { existsSync, renameSync, rmSync } from "node:fs";
12 import { join } from "node:path";
13 import { IMPORT_FILE } from "./dump_import.ts";
14
15 export type UndoError = "invalid" | "missing_snapshot" | "missing_store";
16
17 const keptName = (file: string, kept: string): boolean =>
18 kept.startsWith(`${file}.before-import-`) && /^[\w.-]+$/.test(kept) && !kept.includes("..");
19
20 export function undoImport(
21 dir: string,
22 file: string,
23 kept: string | null,
24 now: number = Date.now(),
25 ): { ok: true; value: { restored: boolean; setAside: string } } | { ok: false; error: UndoError } {
26 if (!IMPORT_FILE.test(file) || (kept !== null && !keptName(file, kept))) return { ok: false, error: "invalid" };
27 const target = join(dir, file);
28 if (!existsSync(target)) return { ok: false, error: "missing_store" };
29 if (kept !== null && !existsSync(join(dir, kept))) return { ok: false, error: "missing_snapshot" };
30 try {
31 const live = new Database(target, { readwrite: true });
32 live.exec("PRAGMA wal_checkpoint(TRUNCATE)");
33 live.close();
34 } catch {
35 // Not SQLite after all: it is set aside as it is.
36 }
37 const setAside = `${file}.undone-import-${now}`;
38 renameSync(target, join(dir, setAside));
39 for (const suffix of ["-wal", "-shm"]) rmSync(`${target}${suffix}`, { force: true });
40 if (kept !== null) renameSync(join(dir, kept), target);
41 return { ok: true, value: { restored: kept !== null, setAside } };
42 }
src/apps/pg_dump.ts added +79
@@ -0,0 +1,79 @@
1 /**
2 * A Postgres dump made loadable by the SQLite import (`dump_import.ts`), and the one command
3 * that makes it (#647). The import runs only `CREATE TABLE`, `CREATE INDEX`, `INSERT ... VALUES`
4 * and transaction words, so a `pg_dump --inserts` file is first stripped of what SQLite has no
5 * meaning for: session settings, `SELECT pg_catalog.set_config/setval`, ownership, comments,
6 * extensions, schemas, sequences and `ALTER TABLE` (defaults, keys, constraints). Primary, unique
7 * and foreign keys are therefore NOT kept: the tables load without them, and the receipt says so.
8 * Nothing is dropped silently:
9 * each kind skipped is counted and shown on the receipt, and anything else stays in the text for
10 * the import's own vetting to refuse by statement number. `COPY ... FROM stdin` is refused with
11 * the fix (dump again with `--inserts`). Plain data in, plain data out; the text is only read.
12 */
13
14 import { splitSql } from "./dump_import.ts";
15
16 /**
17 * What the owner runs on a machine that can reach the old database; the connection string never
18 * leaves it.
19 */
20 export const EXPORT_COMMAND =
21 'pg_dump --inserts --no-owner --no-privileges --no-comments "$DATABASE_URL" > dump.sql';
22
23 export interface Skipped {
24 readonly kind: string;
25 readonly count: number;
26 }
27
28 export type PgRead =
29 | { readonly ok: true; readonly sql: string; readonly skipped: readonly Skipped[]; readonly postgres: boolean }
30 | { readonly ok: false; readonly error: "copy" };
31
32 /** Whether the text is the kind `pg_dump` writes. */
33 export const looksLikePostgres = (text: string): boolean =>
34 /^--\s*PostgreSQL database dump/m.test(text.slice(0, 2000)) || /^SET\s+statement_timeout\b/im.test(text.slice(0, 4000));
35
36 const SKIPPABLE: readonly [string, RegExp][] = [
37 ["SET", /^SET\s/i],
38 ["set_config", /^SELECT\s+pg_catalog\.set_config\s*\(/i],
39 ["setval", /^SELECT\s+pg_catalog\.setval\s*\(/i],
40 ["OWNER", /^ALTER\s+\S+(\s+ONLY)?\s+\S+\s+OWNER\s+TO\s/i],
41 // Column defaults come before the general ALTER TABLE, which would match them first.
42 ["default", /^ALTER\s+TABLE\s+\S+(\s+ONLY)?\s+\S*\s*ALTER\s+COLUMN\s/i],
43 ["key", /^ALTER\s+TABLE\s.*\b(PRIMARY\s+KEY|UNIQUE|FOREIGN\s+KEY)\b/is],
44 ["constraint", /^ALTER\s+TABLE\s/i],
45 ["sequence", /^(CREATE|ALTER)\s+SEQUENCE\s/i],
46 ["COMMENT", /^COMMENT\s+ON\s/i],
47 ["EXTENSION", /^CREATE\s+EXTENSION\s/i],
48 ["SCHEMA", /^CREATE\s+SCHEMA\s/i],
49 ];
50
51 /** `public.name` and `"public"."name"` become `name`: SQLite has no schemas. */
52 const unqualify = (s: string): string => {
53 // An INSERT's rows are data: only what comes before VALUES is a name.
54 const at = /^INSERT\s/i.test(s) ? s.search(/\sVALUES\s*\(/i) : s.length;
55 const head = at < 0 ? s : s.slice(0, at);
56 return head.replace(/(?<![\w"])"?public"?\.(?=["\w])/gi, "") + (at < 0 ? "" : s.slice(at));
57 };
58
59 export function fromPostgres(text: string): PgRead {
60 if (!looksLikePostgres(text)) return { ok: true, sql: text, skipped: [], postgres: false };
61 const kept: string[] = [];
62 const counts = new Map<string, number>();
63 for (const statement of splitSql(text)) {
64 if (/^COPY\s/i.test(statement)) return { ok: false, error: "copy" };
65 const hit = SKIPPABLE.find(([, re]) => re.test(statement));
66 if (hit) counts.set(hit[0], (counts.get(hit[0]) ?? 0) + 1);
67 else kept.push(unqualify(statement));
68 }
69 const skipped = [...counts].map(([kind, count]): Skipped => ({ kind, count }));
70 return { ok: true, sql: kept.map((s) => `${s};`).join("\n"), skipped, postgres: true };
71 }
72
73 /** Whether the dump had keys: the tables load without them, and the receipt must say so. */
74 export const lostKeys = (skipped: readonly Skipped[]): boolean => skipped.some((s) => s.kind === "key");
75
76 /** One line for the receipt: what was left out of a Postgres dump, or "" when nothing was. */
77 export function skippedWords(skipped: readonly Skipped[]): string {
78 return skipped.map((s) => `${s.kind} ×${s.count}`).join(", ");
79 }
src/checks/looks.ts +1 −1
@@ -31,7 +31,7 @@ const SAFE = /^[0-9A-Za-z]{1,64}$/;
3131
3232 const MAX_BYTES = 1024 * 1024;
3333 // The forge's own preview takes ~55 looks (`PREVIEW_OWNER_LOOKS`); every one must fit.
34 const MAX_LOOKS = 76;
34 const MAX_LOOKS = 80;
3535 const MAX_WORDS = 80;
3636
3737 const words = (v: unknown): string | null =>
src/contract/openapi/routes/getApiV1ReposOwnerNameDataMove.ts added +10
@@ -0,0 +1,10 @@
1 import { REPO, type Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "get",
5 path: `${REPO}/data-move`,
6 summary:
7 "Where a platform app's data move stands: { command, files: [{ name, bytes }], last?: { at, file, snapshot, tables: [{ table, rows, sha256 }], undone }, cutOver, canUndo }; the command makes the dump (pg_dump --inserts), files are the .sql dumps in the app's drop; a repository admin's, anyone else a 404; needs a token",
8 auth: true,
9 role: "member",
10 };
src/contract/openapi/routes/index.ts +2
@@ -153,6 +153,7 @@ export { route as getApiV1ReposOwnerNameCrew } from "./getApiV1ReposOwnerNameCre
153153 export { route as getApiV1ReposOwnerNameData } from "./getApiV1ReposOwnerNameData.ts";
154154 export { route as getApiV1ReposOwnerNameDataEnvFileRows } from "./getApiV1ReposOwnerNameDataEnvFileRows.ts";
155155 export { route as getApiV1ReposOwnerNameDataEnvFileTables } from "./getApiV1ReposOwnerNameDataEnvFileTables.ts";
156 export { route as getApiV1ReposOwnerNameDataMove } from "./getApiV1ReposOwnerNameDataMove.ts";
156157 export { route as getApiV1ReposOwnerNameDecisions } from "./getApiV1ReposOwnerNameDecisions.ts";
157158 export { route as getApiV1ReposOwnerNameDesk } from "./getApiV1ReposOwnerNameDesk.ts";
158159 export { route as getApiV1ReposOwnerNameDrop } from "./getApiV1ReposOwnerNameDrop.ts";
@@ -351,6 +352,7 @@ export { route as postApiV1ReposOwnerNameContents } from "./postApiV1ReposOwnerN
351352 export { route as postApiV1ReposOwnerNameContribute } from "./postApiV1ReposOwnerNameContribute.ts";
352353 export { route as postApiV1ReposOwnerNameDataEnvFileQuery } from "./postApiV1ReposOwnerNameDataEnvFileQuery.ts";
353354 export { route as postApiV1ReposOwnerNameDataImport } from "./postApiV1ReposOwnerNameDataImport.ts";
355 export { route as postApiV1ReposOwnerNameDataMove } from "./postApiV1ReposOwnerNameDataMove.ts";
354356 export { route as postApiV1ReposOwnerNameDesk } from "./postApiV1ReposOwnerNameDesk.ts";
355357 export { route as postApiV1ReposOwnerNameDrop } from "./postApiV1ReposOwnerNameDrop.ts";
356358 export { route as postApiV1ReposOwnerNameEdits } from "./postApiV1ReposOwnerNameEdits.ts";
src/contract/openapi/routes/postApiV1ReposOwnerNameDataMove.ts added +22
@@ -0,0 +1,22 @@
1 import { REPO, type Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: `${REPO}/data-move`,
6 summary:
7 "Move a platform app's Postgres dump from its drop into its SQLite store: { dump } checks (nothing changes) and { dump, apply: true } imports, keeping the store it replaces as the snapshot; { undo: true } puts the snapshot back until a domain has been cut over. Answers the receipt { applied, dump, file, statements, tables: [{ table, rows, dump, store, counted }], skipped, snapshot }; needs an admin-scope token and the right to administer the repository",
8 auth: true,
9 role: "member",
10 body: { dump: "string?", apply: "boolean?", undo: "boolean?" },
11 refusals: {
12 invalid_dump: "dump is not the name of a file (422)",
13 missing: "no such file in the app's drop (404)",
14 no_app: "the repository is not an app (422)",
15 copy: "the dump uses COPY; dump again with --inserts (422)",
16 refused: "a statement is of a kind a dump may not run; its number is named (422)",
17 mismatch: "the store does not match the dump in some table; nothing was changed (409)",
18 cut_over: "a domain has been cut over, so neither an import nor an undo is allowed (409)",
19 nothing_to_undo: "there is no import to take back (409)",
20 snapshot_gone: "the snapshot is no longer in the app's data (409)",
21 },
22 };
src/contract/tags.ts +1
@@ -93,6 +93,7 @@ const ALIAS: Readonly<Record<string, string>> = {
9393 boards: "forges",
9494 data: "apps",
9595 "data-import": "apps",
96 "data-move": "apps",
9697 drop: "apps",
9798 };
9899
src/crew/conversation_confirm.ts +1
@@ -119,6 +119,7 @@ export const ASKING: Readonly<Record<string, Asking>> = {
119119 quotas: { does: "change an account's quotas" },
120120 spawn: { does: "spawn a child forge" },
121121 app_data_import: { does: "load a SQL dump into an app's database" },
122 migrate_data: { does: "load a dump from an app's drop into its database, or take that import back", asks: (a) => a["apply"] === true || a["undo"] === true },
122123 app_deploy: { does: "redeploy or stop an app's production or preview" },
123124 stop: { does: "stop a crew run" },
124125 converge: { does: "have a fork's crew merge the parent's updates" },
src/design/activity.ts +1
@@ -207,6 +207,7 @@ const WORDS: Record<string, string> = {
207207 "repo.mirrored": "mirrored a repository",
208208 "repo.imported": "imported issues and pulls",
209209 "app.data_imported": "loaded a database dump into an app",
210 "app.data_undone": "put an app's data back as it was before the import",
210211 "repo.pinned": "pinned a repository",
211212 "repo.unpinned": "unpinned a repository",
212213 "board.pins": "changed their pinned panels",
src/http/api/data.ts +32 −1
@@ -16,6 +16,7 @@ import {
1616 tableList,
1717 } from "../../data/repo_scope.ts";
1818 import { importAppData, importDataMessage } from "../../actions/migrate.ts";
19 import { dataMoveState, moveData, moveMessage, undoMove } from "../../actions/migrate_data.ts";
1920 import { loadRepo } from "../api.ts";
2021 import type { Ctx, Handler } from "../context.ts";
2122 import { QUERY_TIMEOUT_MS } from "../../data/query.ts";
@@ -64,7 +65,8 @@ export function dataRoutes(router: Router<Handler>): void {
6465 return result ? reply(result) : notFound();
6566 });
6667
67 // A dump in, loaded and reported per table on both sides (#11697). Body: { dump, env?, file?, replace? }.
68 // A dump in, loaded and reported per table on both sides (#11697). Body: { dump, env?, file?,
69 // replace? }.
6870 router.add("POST", "/api/v1/repos/:owner/:name/data-import", async (ctx) => {
6971 const denied = requireScope(ctx, "admin");
7072 if (denied) return denied;
@@ -86,6 +88,35 @@ export function dataRoutes(router: Router<Handler>): void {
8688 return json(done.value, 200, PRIVATE);
8789 });
8890
91 // A platform app's data (#647): where it stands (the command, the drop's dumps, the last receipt)
92 // …
93 router.add("GET", "/api/v1/repos/:owner/:name/data-move", (ctx) => {
94 const repo = loadRepo(ctx);
95 if (!repo) return notFound();
96 const state = dataMoveState(ctx.app, ctx.auth.user, repo);
97 return state.ok ? json(state.value, 200, PRIVATE) : notFound();
98 });
99
100 // … and a move: { dump } checks, { dump, apply: true } imports, { undo: true } takes it back.
101 router.add("POST", "/api/v1/repos/:owner/:name/data-move", async (ctx) => {
102 const denied = requireScope(ctx, "admin");
103 if (denied) return denied;
104 const repo = loadRepo(ctx);
105 if (!repo) return notFound();
106 const input = await body(ctx.request);
107 const done = input["undo"] === true
108 ? undoMove(ctx.app, ctx.auth.user, repo)
109 : typeof input["dump"] === "string"
110 ? moveData(ctx.app, ctx.auth.user, repo, { dump: input["dump"], apply: input["apply"] === true })
111 : undefined;
112 if (done === undefined) return apiError("invalid_dump", 422, "dump is the name of a file in the app's drop");
113 if (done.ok) return json(done.value, 200, PRIVATE);
114 if (done.error === "not_found") return notFound();
115 const code = typeof done.error === "string" ? done.error : done.error.error;
116 const conflict = ["exists", "mismatch", "cut_over", "nothing_to_undo", "snapshot_gone", "store_gone"].includes(code);
117 return apiError(code, conflict ? 409 : code === "too_large" || code === "too_large_dump" ? 413 : code === "missing" ? 404 : code === "unauthorized" ? 401 : 422, moveMessage(done.error));
118 });
119
89120 router.add("POST", "/api/v1/repos/:owner/:name/data/:env/:file/query", async (ctx) => {
90121 const denied = requireScope(ctx, "ro");
91122 if (denied) return denied;
src/http/mcp/tools/index.ts +1
@@ -105,6 +105,7 @@ export { tool as make_it_yours } from "./make_it_yours.ts";
105105 export { tool as map } from "./map.ts";
106106 export { tool as merge } from "./merge.ts";
107107 export { tool as migrate } from "./migrate.ts";
108 export { tool as migrate_data } from "./migrate_data.ts";
108109 export { tool as milestones } from "./milestones.ts";
109110 export { tool as mint_invite } from "./mint_invite.ts";
110111 export { tool as mirror_now } from "./mirror_now.ts";
src/http/mcp/tools/migrate_data.ts added +41
@@ -0,0 +1,41 @@
1 import { dataMoveState, moveData, moveMessage, undoMove } from "../../../actions/migrate_data.ts";
2 import { scopeAllows } from "../../../core/tokens.ts";
3 import { bad, good, isAnswer, repoArg, str, type Tool } from "../shared.ts";
4 import { REPO_ARG } from "../shared.ts";
5
6 export const tool: Tool = {
7 name: "migrate_data",
8 doc: "`{ repo, dump?, apply?, undo? }`: a platform app's Postgres into the app's own SQLite store (#647). The owner makes the dump with the `command` this answers when given only `repo` (`pg_dump --inserts`, run where the old database is reachable; the connection string never reaches the forge) and puts it in the app's drop. With `dump` (its name there) it loads it in a scratch copy and answers the receipt: per table the rows and SHA-256 on the dump's side and on the store's, `counted` from the dump's own text, and what a Postgres dump had that SQLite cannot hold (`skipped`); nothing is changed unless `apply: true`, which first keeps the store it replaces as the snapshot. `undo: true` puts that snapshot back, until a domain has been cut over. A repository admin's; applying and undoing need an admin-scope token",
9 description: "Move a platform app's Postgres dump from its drop into its store; dry-run unless `apply`.",
10 inputSchema: {
11 type: "object",
12 properties: {
13 repo: REPO_ARG,
14 dump: { type: "string" },
15 apply: { type: "boolean" },
16 undo: { type: "boolean" },
17 },
18 required: ["repo"],
19 },
20 writes: true,
21 readsWhen: (args) => args["undo"] !== true && args["apply"] !== true,
22 run(ctx, user, args) {
23 const repo = repoArg(ctx, user, args);
24 if (isAnswer(repo)) return repo;
25 if ((args["undo"] === true || args["apply"] === true) && !scopeAllows(ctx.auth.scope ?? "ro", "admin"))
26 return bad("forbidden", "applying or undoing a data move needs an admin-scope token");
27 if (args["undo"] === true) {
28 const undone = undoMove(ctx.app, user, repo);
29 return undone.ok ? good(undone.value) : bad(String(undone.error), moveMessage(undone.error));
30 }
31 const dump = str(args["dump"]);
32 if (dump === "") {
33 const state = dataMoveState(ctx.app, user, repo);
34 return state.ok ? good(state.value) : bad("not_found", "no such app, or you are not its admin");
35 }
36 const done = moveData(ctx.app, user, repo, { dump, apply: args["apply"] === true });
37 if (done.ok) return good(done.value);
38 const code = typeof done.error === "string" ? done.error : done.error.error;
39 return bad(code, moveMessage(done.error));
40 },
41 };
src/http/server.ts +3
@@ -270,6 +270,7 @@ import {
270270 import { adapterPull } from "../web/studio/adapter.ts";
271271 import { jobsPage, steerJobPage } from "../web/studio/jobs.ts";
272272 import { steerSchedulePage } from "../web/studio/schedules.ts";
273 import { dataMoveAct, dataMovePage } from "../web/studio/data_move.ts";
273274 import { dropPageGet, removeDropPage, uploadDropPage } from "../web/studio/drop.ts";
274275 import { noteModerate } from "../web/studio/notes.ts";
275276 import { tellPage } from "../web/studio/tell.ts";
@@ -1545,6 +1546,8 @@ export function buildRouter(): Router<Handler> {
15451546 router.add("GET", "/:owner/:name/app/jobs", jobsPage);
15461547 router.add("POST", "/:owner/:name/app/jobs/:id/:action", steerJobPage);
15471548 router.add("POST", "/:owner/:name/app/schedules/:schedule/:action", steerSchedulePage);
1549 router.add("GET", "/:owner/:name/data-move", dataMovePage);
1550 router.add("POST", "/:owner/:name/data-move", dataMoveAct);
15481551 router.add("GET", "/:owner/:name/drop", dropPageGet);
15491552 router.add("POST", "/:owner/:name/drop", uploadDropPage);
15501553 router.add("POST", "/:owner/:name/drop/remove", removeDropPage);
src/spawn/preview-looks.ts +3
@@ -38,6 +38,8 @@ export const PREVIEW_APP = "hello";
3838 export const PREVIEW_APP_PAGE = `/${PREVIEW_ADMIN}/${PREVIEW_APP}`;
3939 /** The seeded app's Drop: the page the sample files are listed on (#645). */
4040 export const PREVIEW_DROP_PAGE = `${PREVIEW_APP_PAGE}/drop`;
41 /** The seeded app's data move (#647): a Postgres dump from its drop, imported, with its receipt. */
42 export const PREVIEW_DATA_MOVE_PAGE = `${PREVIEW_APP_PAGE}/data-move`;
4143
4244 /**
4345 * The recently moved repository (#10195): made by the admin as `moved`, then
@@ -208,6 +210,7 @@ export const PREVIEW_OWNER_LOOKS: readonly OwnerLook[] = [
208210 PREVIEW_REPO_HOME,
209211 PREVIEW_APP_PAGE,
210212 PREVIEW_DROP_PAGE,
213 PREVIEW_DATA_MOVE_PAGE,
211214 PREVIEW_MOVED_PAGE,
212215 PREVIEW_MOVE_LOOK,
213216 PREVIEW_PULL_PAGE,
src/spawn/preview-seed.ts +15
@@ -15,6 +15,9 @@
1515 */
1616
1717 import { putStream } from "../drop/store.ts";
18 import { moveData } from "../actions/migrate_data.ts";
19 import { setMigration } from "../apps/migration_mode.ts";
20 import { dataMovePage } from "../web/studio/data_move.ts";
1821 import { dropPageGet } from "../web/studio/drop.ts";
1922 import { existsSync, mkdirSync, writeFileSync } from "node:fs";
2023 import { keepNotice, noticeOf } from "../billing/notice.ts";
@@ -64,6 +67,7 @@ import {
6467 PREVIEW_APP,
6568 PREVIEW_APP_PAGE,
6669 PREVIEW_DROP_PAGE,
70 PREVIEW_DATA_MOVE_PAGE,
6771 PREVIEW_DISK_ALERT,
6872 PREVIEW_FIXTURES,
6973 PREVIEW_FORKS,
@@ -889,6 +893,14 @@ async function seedApp(app: App, admin: User): Promise<Result<void, string>> {
889893 const put = await putStream(app.config.dataDir, repo.value.id, name, new Response(text).body);
890894 if (!put.ok) return err(`the sample app's drop: ${put.error}`);
891895 }
896 // Its data move (#647): the app is in migration mode, a Postgres dump sits in the drop and has
897 // been imported, so the data line and the receipt have something to show.
898 const pg = "-- PostgreSQL database dump\nSET statement_timeout = 0;\nCREATE TABLE public.parts (id integer NOT NULL, name text, torque_nm real);\nINSERT INTO public.parts VALUES (1, 'M6 bolt', 9.5);\nINSERT INTO public.parts VALUES (2, 'M8 bolt', 22);\nINSERT INTO public.parts VALUES (3, 'M10 nut', 45);\nCREATE TABLE public.orders (id integer NOT NULL, part_id integer, qty integer);\nINSERT INTO public.orders VALUES (1, 1, 40), (2, 3, 120);\nALTER TABLE ONLY public.parts ADD CONSTRAINT parts_pkey PRIMARY KEY (id);\n";
899 const dumped = await putStream(app.config.dataDir, repo.value.id, "dump.sql", new Response(pg).body);
900 if (!dumped.ok) return err(`the sample app's dump: ${dumped.error}`);
901 setMigration(app.db, repo.value.id, "https://platform.example/hello", true);
902 const moved = moveData(app, admin, repo.value, { dump: "dump.sql", apply: true });
903 if (!moved.ok) return err(`the sample app's data move: ${JSON.stringify(moved.error)}`);
892904 const url = new URL(`https://preview.invalid${PREVIEW_APP_PAGE}`);
893905 const answer = await studioApp(
894906 { app, request: new Request(url.href), url, params: {}, auth: { user: admin }, theme: undefined } as Ctx,
@@ -896,6 +908,9 @@ async function seedApp(app: App, admin: User): Promise<Result<void, string>> {
896908 record.value,
897909 );
898910 if (answer.status !== 200) return err(`${PREVIEW_APP_PAGE} would answer ${String(answer.status)}, not 200`);
911 const moveUrl = new URL(`https://preview.invalid${PREVIEW_DATA_MOVE_PAGE}`);
912 const move = dataMovePage({ app, request: new Request(moveUrl.href), url: moveUrl, params: { owner: PREVIEW_ADMIN, name: PREVIEW_APP }, auth: { user: admin }, theme: undefined } as unknown as Ctx);
913 if (move.status !== 200 || !(await move.text()).includes("parts")) return err(`${PREVIEW_DATA_MOVE_PAGE} does not show the sample receipt`);
899914 const dropUrl = new URL(`https://preview.invalid${PREVIEW_DROP_PAGE}`);
900915 const drop = dropPageGet({ app, request: new Request(dropUrl.href), url: dropUrl, params: { owner: PREVIEW_ADMIN, name: PREVIEW_APP }, auth: { user: admin }, theme: undefined } as unknown as Ctx);
901916 return drop.status === 200 && (await drop.text()).includes("torque-spec.pdf") ? ok() : err(`${PREVIEW_DROP_PAGE} does not show the sample drop`);
src/tui/screens/dataimport.ts +1 −1
@@ -89,7 +89,7 @@ export function dataImportScreen(f: Session, ctx: Ctx, repo: Repo): Screen {
8989 },
9090 key(e) {
9191 if (state.key(e, rows.length)) return true;
92 if (e.ctrl || e.alt || (e.key !== "i" && e.key !== "r")) return false;
92 if (e.ctrl || e.alt || !["i", "r"].includes(e.key)) return false;
9393 if (!mayImport(f, repo)) {
9494 ctx.toast("Only the app's admin imports a database.", "danger");
9595 return true;
src/tui/screens/datamove.ts added +119
@@ -0,0 +1,119 @@
1 /**
2 * Moving a platform app's data in the terminal (#647): the same `moveData` and `undoMove` as the
3 * page `…/data-move`, the API route and the MCP `migrate_data` tool, an admin's only. `m` asks for
4 * the name of a `.sql` file in the app's drop and checks it (nothing is changed), `a` imports it,
5 * keeping the store it replaces, and `u` puts that store back after a y/n, until a domain has been
6 * cut over. The rows are the receipt's: per table, the dump's rows and checksum, the store's, and
7 * whether they match. A running app still holds the old file: restart it after an import or undo.
8 */
9
10 import { dataMoveState, moveData, moveMessage, undoMove, type MoveReceipt } from "../../actions/migrate_data.ts";
11 import { lostKeys, skippedWords } from "../../apps/pg_dump.ts";
12 import { canAdmin, type Repo } from "../../core/repos.ts";
13 import type { Ctx, Screen } from "../app/app.ts";
14 import { binding } from "../app/keys.ts";
15 import type { Command } from "../app/palette.ts";
16 import type { Session } from "../session.ts";
17 import { style } from "../term/style.ts";
18 import { truncate } from "../term/width.ts";
19 import { fill, len, split } from "../ui/layout.ts";
20 import { ListState, confirmScreen, drawList, promptScreen, type ListRow } from "./common.ts";
21 import { mayImport } from "./dataimport.ts";
22
23 /** A move's receipt, a row per table. */
24 export function moveRows(receipt: MoveReceipt): ListRow[] {
25 return receipt.tables.map((t) => {
26 const match = t.dump === t.store && t.rows === t.counted;
27 return {
28 title: `${t.table}: ${t.rows} rows`,
29 tone: match ? "success" : "danger",
30 note: `dump ${t.dump.slice(0, 16)} store ${t.store.slice(0, 16)} counted in dump ${t.counted}`,
31 ...(match ? {} : { meta: "mismatch" }),
32 };
33 });
34 }
35
36 /** What was left out of the dump, in a line; keys are named plainly. */
37 export function leftOut(receipt: MoveReceipt): string {
38 if (receipt.skipped.length === 0) return "";
39 return `; left out: ${skippedWords(receipt.skipped)}${lostKeys(receipt.skipped) ? " (keys are not kept: the tables load without them)" : ""}`;
40 }
41
42 export function dataMoveScreen(f: Session, ctx: Ctx, repo: Repo): Screen {
43 const state = new ListState();
44 let rows: ListRow[] = [];
45 let note = "m checks a dump in the app's drop, a imports it, u undoes it. Restart the app after an import or undo.";
46 const move = (apply: boolean): Screen =>
47 promptScreen(ctx, screen, {
48 title: apply ? "Import from the drop" : "Check a dump in the drop",
49 label: "file in the app's drop, like dump.sql",
50 placeholder: "dump.sql",
51 onSubmit: (name) => {
52 if (name === "") return "Name a file.";
53 const done = moveData(f.app, f.user, repo, { dump: name, apply });
54 if (!done.ok) return moveMessage(done.error);
55 rows = moveRows(done.value);
56 note = `${apply ? "Imported" : "Checked, nothing changed:"} ${done.value.tables.length} tables${leftOut(done.value)}.`;
57 ctx.invalidate();
58 return undefined;
59 },
60 });
61 const undo = (): Screen =>
62 confirmScreen(ctx, screen, {
63 title: "Undo the import?",
64 body: ["The store from before the import is put back.", "Writes the app made since the import are not in it."],
65 actions: [{ label: "Undo", variant: "danger" }, { label: "Cancel" }],
66 yesNo: true,
67 selected: 1,
68 onChoose: (i) => {
69 if (i !== 0) return;
70 const undone = undoMove(f.app, f.user, repo);
71 if (!undone.ok) ctx.toast(moveMessage(undone.error), "danger");
72 else {
73 rows = [];
74 note = `${undone.value.file} is back as it was; writes since the import are not in it. The imported store is kept as ${undone.value.setAside}. Restart the app.`;
75 ctx.invalidate();
76 }
77 },
78 });
79 const screen: Screen = {
80 title: `${repo.ownerName}/${repo.name} — Move the data`,
81 keys: [binding(["m"], "check a drop dump"), binding(["a"], "import a drop dump"), binding(["u"], "undo the import"), binding(["j", "k"], "move cursor")],
82 render(area, s) {
83 const c = ctx.theme.c;
84 const [head, list] = split(area, "v", [len(1), fill(1)]);
85 s.text(head!.x, head!.y, truncate(mayImport(f, repo) ? note : "Only the app's admin moves the data.", head!.w), style(c.mutedFg, c.bg), head!.x + head!.w);
86 drawList(s, list!, c, state, rows);
87 },
88 key(e) {
89 if (state.key(e, rows.length)) return true;
90 if (e.ctrl || e.alt || !["m", "a", "u"].includes(e.key)) return false;
91 if (!mayImport(f, repo)) {
92 ctx.toast("Only the app's admin moves the data.", "danger");
93 return true;
94 }
95 if (e.key === "u") {
96 const can = dataMoveState(f.app, f.user, repo);
97 if (!can.ok || !can.value.canUndo) ctx.toast(moveMessage("nothing_to_undo"), "danger");
98 else ctx.push(undo());
99 return true;
100 }
101 ctx.push(move(e.key === "a"));
102 return true;
103 },
104 };
105 return screen;
106 }
107
108 /** The palette's command: an admin of a repository that is an app. */
109 export function dataMoveCommands(f: Session, ctx: Ctx, repo: Repo | undefined): Command[] {
110 if (!mayImport(f, repo) || f.user === undefined || !canAdmin(f.app.db, repo, f.user)) return [];
111 return [
112 {
113 id: "data-move",
114 title: `${repo.ownerName}/${repo.name} — Move the data`,
115 hint: "a platform app's Postgres dump from the drop, receipt and undo",
116 run: () => ctx.push(dataMoveScreen(f, ctx, repo)),
117 },
118 ];
119 }
src/tui/screens/workspace.ts +2
@@ -193,6 +193,7 @@ import { deskCommands } from "./desk.ts";
193193 import { landCommands } from "./landasis.ts";
194194 import { shiftCommands } from "./subtitle_shift.ts";
195195 import { dataImportCommands } from "./dataimport.ts";
196 import { dataMoveCommands } from "./datamove.ts";
196197 import { jobsCommands } from "./jobs.ts";
197198 import { agentsMdCommands } from "./agents_md.ts";
198199 import { queueCommands } from "./queuegrowth.ts";
@@ -2141,6 +2142,7 @@ export function workspaceScreen(
21412142 ...landCommands(f, ctx, repo),
21422143 ...shiftCommands(f, ctx, repo, () => screen),
21432144 ...dataImportCommands(f, ctx, repo),
2145 ...dataMoveCommands(f, ctx, repo),
21442146 ...jobsCommands(f, ctx, repo),
21452147 ...agentsMdCommands(f, ctx, repo),
21462148 ...chaptersCommands(f, ctx, repo),
src/web/studio/data_move.ts added +178
@@ -0,0 +1,178 @@
1 /**
2 * Moving a platform app's data, as a page (#647): `/:owner/:name/data-move`, reached from the data
3 * line of the migration checklist. In order: the one command that makes the dump, the `.sql`
4 * files in the app's drop with **Check** (everything but putting it in place) and **Import**, then
5 * the receipt — per table rows and a SHA-256 on the dump's side and the store's — and **Undo**
6 * while no domain has been cut over. Server-rendered, no script; every button is a same-origin
7 * form POST (`readForm`), an admin's only: to anyone else the page is not found. The words and
8 * the gate are `actions/migrate_data.ts`'s, the same the API, the MCP tool and the terminal use.
9 */
10
11 import { dataMoveState, moveData, moveMessage, undoMove, type MoveReceipt, type DataMoveState } from "../../actions/migrate_data.ts";
12 import { lostKeys, skippedWords } from "../../apps/pg_dump.ts";
13 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
14 import type { Ctx } from "../../http/context.ts";
15 import { adminForm, loadAdminRepo } from "../admin_form.ts";
16 import { crumbsFor, notFoundPage, pageHead, repoBase, signInFirst } from "../chrome.ts";
17 import { field } from "../form.ts";
18 import { html, type Html } from "../html.ts";
19 import { flashTo, page } from "../layout.ts";
20 import { badge, btn, fileSize } from "../ui.ts";
21
22 const short = (sha: string): string => sha.slice(0, 12);
23
24 const KEYS_WORDS = "Primary, unique and foreign keys are not kept: the tables load without them. Add them again in the app's own schema.";
25
26 /**
27 * The receipt's rows: what the dump says and what the store holds. A row whose store hash was
28 * never computed (`store` null) shows no verdict: a match is only printed when someone compared.
29 */
30 export function receiptTable(
31 rows: readonly { table: string; rows: number; dump: string; store: string | null; counted: number | null }[],
32 ): Html {
33 const proven = rows.every((r) => r.store !== null);
34 return html`<div class="table-scroll">
35 <table class="table" id="receipt" data-dense>
36 <caption class="sr-only">Per table, on the dump's side${proven ? " and the store's" : ""}</caption>
37 <thead><tr><th scope="col">Table</th><th scope="col" class="num">Rows</th><th scope="col">Dump SHA-256</th>${proven ? html`<th scope="col">Store SHA-256</th><th scope="col"></th>` : null}</tr></thead>
38 <tbody>
39 ${rows.map((r) => {
40 const match = r.store !== null && r.dump === r.store && (r.counted === null || r.rows === r.counted);
41 return html`<tr>
42 <td class="break">${r.table}</td>
43 <td class="num">${r.rows}</td>
44 <td class="mono">${short(r.dump)}</td>
45 ${proven ? html`<td class="mono">${short(r.store ?? "")}</td><td>${badge(match ? "match" : "differs", { tone: match ? "success" : "danger" })}</td>` : null}
46 </tr>`;
47 })}
48 </tbody>
49 </table>
50 </div>`;
51 }
52
53 function skippedNote(skipped: MoveReceipt["skipped"]): Html | null {
54 if (skipped.length === 0) return null;
55 return html`<p class="muted small">Left out of the Postgres dump, SQLite has no meaning for it: ${skippedWords(skipped)}.${lostKeys(skipped) ? ` ${KEYS_WORDS}` : ""}</p>`;
56 }
57
58 function receiptCard(receipt: MoveReceipt): Html {
59 const total = receipt.tables.reduce((n, t) => n + t.rows, 0);
60 return html`<section id="receipt-block">
61 <h2>${receipt.applied ? "Receipt" : "Check"}</h2>
62 <p>${receipt.applied ? "Loaded" : "Would load"} ${receipt.tables.length} tables, ${total} rows from <span class="mono break">${receipt.dump}</span> into <span class="mono">${receipt.file}</span>. ${receipt.applied ? "" : "Nothing was changed."}</p>
outdated

store: t.sha256 and counted: t.rows make this receipt always say match. Record the store hash in the event or do not show it.

63 ${receiptTable(receipt.tables)}
64 ${skippedNote(receipt.skipped)}
65 </section>`;
66 }
67
68 /**
69 * What the import recorded, and only that: rows and the dump's checksum, the store's when it was
70 * kept.
71 */
72 function lastCard(state: DataMoveState): Html | null {
73 const last = state.last;
74 if (last === undefined) return null;
75 const rows = last.tables.map((t) => ({ table: t.table, rows: t.rows, dump: t.sha256, store: t.store ?? null, counted: null }));
76 return html`<section id="last-import">
77 <h2>${last.undone ? "Last import, taken back" : "Last import"}</h2>
78 <dl class="status-rows">
79 <div class="status-row"><dt>When</dt><dd>${last.at}</dd></div>
80 <div class="status-row"><dt>Dump</dt><dd class="mono">${last.file}</dd></div>
81 <div class="status-row"><dt>Store before</dt><dd>${last.snapshot === null ? "none" : html`kept as <span class="mono break">${last.snapshot}</span>`}</dd></div>
82 </dl>
83 ${last.undone ? null : receiptTable(rows)}
84 </section>`;
85 }
86
87 function body(base: string, state: DataMoveState, shown: MoveReceipt | undefined): Html {
88 const form = state.files.length === 0
89 ? html`<p class="muted">No <span class="mono">.sql</span> file in the drop yet. Put <span class="mono">dump.sql</span> in the app's <a href="${base}/drop">drop</a> (up to 16 MB).</p>`
outdated

Import has no name/value, so it posts do empty and only imports because dataMoveAct treats empty as apply. Set name="do" value="import" and require it.

90 : html`<form method="post" action="${base}/data-move" class="row">
91 <select class="input" name="dump" aria-label="Dump in the drop">
92 ${state.files.map((f) => html`<option value="${f.name}">${f.name} (${fileSize(f.bytes)})</option>`)}
93 </select>
94 ${state.cutOver
95 ? null
96 : html`${btn("Check", { name: "do", value: "check", variant: "outline" })}${btn("Import", { name: "do", value: "import", variant: "primary" })}`}
97 </form>`;
98 return html`${pageHead("Move the data", { sub: "A platform app's Postgres into this app's own store." })}
99 <section id="dump">
outdated

Principle 5: this is primary when imported, next to the Import primary. One accent per page.

100 <h2>Dump</h2>
101 <p>Run this where the old database is reachable; its connection string stays there.</p>
102 <pre class="mono break" id="export-command">${state.command}</pre>
103 </section>
104 <section id="load">
105 <h2>Import</h2>
106 ${state.cutOver ? html`<p class="muted">A domain has been cut over, so the old data is no longer the way back: importing is closed.</p>` : null}
107 ${form}
108 ${state.cutOver || state.files.length === 0
109 ? null
110 : html`<p class="muted small">Check changes nothing. Import keeps the store it replaces for Undo. Restart the app afterwards: it still holds the old file open until it does, and anything it writes in between is lost.</p>`}
111 </section>
112 ${shown === undefined ? null : receiptCard(shown)}
113 ${lastCard(state)}
114 <div class="row">
115 ${state.canUndo
116 ? html`<form method="post" action="${base}/data-move"><input type="hidden" name="do" value="undo" />${btn("Undo the import", { variant: "outline" })}</form>
117 <p class="muted small">Undo puts the earlier store back; writes the app made since the import are not in it (they stay in the set-aside file). Restart the app after.</p>`
118 : null}
119 ${btn("Back to the checklist", { href: `${base}#app-migration`, variant: "outline" })}
120 </div>`;
121 }
122
123 function draw(ctx: Ctx, shown: MoveReceipt | undefined, status = 200, flash?: { tone: "danger"; message: string }): Response {
124 const rendering = performance.now();
125 const repo = loadAdminRepo(ctx);
126 const gone = (): Response => {
127 const out = notFoundPage(ctx);
128 notePhase(ctx.request, "render", performance.now() - rendering);
129 return out;
130 };
131 if (!repo) return gone();
132 const state = dataMoveState(ctx.app, ctx.auth.user, repo);
133 if (!state.ok) return gone();
134 const out = page(
135 { ctx, title: `Move the data · ${repo.name}`, crumbs: crumbsFor(repo), body: body(repoBase(repo), state.value, shown), ...(flash ? { flash } : {}) },
136 status,
137 );
138 notePhase(ctx.request, "render", performance.now() - rendering);
139 return out;
140 }
141
142 /** GET /:owner/:name/data-move */
143 export function dataMovePage(ctx: Ctx): Response {
144 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
145 if (ctx.auth.user === undefined) {
146 const rendering = performance.now();
147 const out = signInFirst(ctx);
148 notePhase(ctx.request, "render", performance.now() - rendering);
149 return out;
150 }
151 return draw(ctx, undefined);
152 }
153
154 /** POST /:owner/:name/data-move: `do` is check, import or undo. */
155 export async function dataMoveAct(ctx: Ctx): Promise<Response> {
156 const gate = await adminForm(ctx);
157 if (gate instanceof Response) return gate;
158 const { repo, form } = gate;
159 const to = `${repoBase(repo)}/data-move`;
160 const what = field(form, "do");
161 if (what === "undo") {
162 const done = undoMove(ctx.app, ctx.auth.user, repo);
163 return flashTo(
164 ctx,
165 to,
166 done.ok
167 ? { message: done.value.restored ? `${done.value.file} is back as it was before the import; writes the app made since are not in it. Restart the app. The imported store is kept as ${done.value.setAside}.` : `The imported ${done.value.file} is set aside as ${done.value.setAside}; the app starts empty, as it was.` }
168 : { tone: "danger", message: moveMessage(done.error) },
169 );
170 }
171 // Only an explicit `do=import` writes; anything else, including nothing, is refused.
172 if (what !== "import" && what !== "check") return draw(ctx, undefined, 422, { tone: "danger", message: "do is check, import or undo: nothing was changed." });
173 const apply = what === "import";
174 const done = moveData(ctx.app, ctx.auth.user, repo, { dump: field(form, "dump"), apply });
175 if (!done.ok) return flashTo(ctx, to, { tone: "danger", message: moveMessage(done.error) });
176 if (apply) return flashTo(ctx, to, { message: `Imported ${done.value.tables.length} tables from ${done.value.dump}: every table's rows and checksum match.` });
177 return draw(ctx, done.value);
178 }
src/web/studio/migration.ts +31 −3
@@ -6,6 +6,7 @@
66 * text and goes through `html`, which escapes it.
77 */
88 import { migrationView } from "../../actions/migration.ts";
9 import { dataMoveState } from "../../actions/migrate_data.ts";
910 import { progressWords, type Progress } from "../../apps/migration_mode.ts";
1011 import type { Repo } from "../../core/repos.ts";
1112 import type { Ctx } from "../../http/context.ts";
@@ -13,8 +14,15 @@ import { html, type Html } from "../html.ts";
1314 import { icon } from "../icons.ts";
1415 import { badge, btn, card, row } from "../ui.ts";
1516
17 /** The data line (#647): where the app's rows stand, and the way to the page that moves them. */
18 export interface DataLine {
19 readonly href: string;
20 /** Rows and tables of the import still in place, when there is one. */
21 readonly imported: { readonly tables: number; readonly rows: number } | undefined;
22 }
23
1624 /** The card's markup for a progress; the page and the tests share it. */
17 export function migrationCard(origin: string | undefined, progress: Progress, cutOver: string | null): Html {
25 export function migrationCard(origin: string | undefined, progress: Progress, cutOver: string | null, data: DataLine | null = null): Html {
1826 const areas = new Map<string, Progress["items"][number][]>();
1927 for (const item of progress.items) areas.set(item.area, [...(areas.get(item.area) ?? []), item]);
2028 const lists = [...areas].map(
@@ -35,7 +43,17 @@ export function migrationCard(origin: string | undefined, progress: Progress, cu
3543 </div>
3644 ${origin === undefined ? null : html`<p class="muted small">Parity with ${origin}</p>`}
3745 ${progress.total === 0 ? html`<p class="muted">Waiting for the crew's checklist.</p>` : lists}
38 ${progress.atParity && cutOver !== null ? btn("Cut over", { href: cutOver, variant: "primary" }) : null}`,
46 ${data === null
47 ? null
48 : html`<div class="bar" id="app-migration-data">
49 <span><strong>Data</strong>
50 <span class="muted small">${data.imported === undefined ? "not moved yet" : `${data.imported.tables} tables, ${data.imported.rows} rows imported and proven`}</span></span>
51 <span class="spacer"></span>
52 ${data.imported === undefined
53 ? btn("Import data", { href: data.href, size: "sm", variant: "primary" })
54 : btn("Receipt", { href: data.href, size: "sm" })}
55 </div>`}
56 ${progress.atParity && cutOver !== null ? btn("Cut over", { href: cutOver, variant: data !== null && data.imported === undefined ? "outline" : "primary" }) : null}`,
3957 { id: "app-migration" },
4058 );
4159 }
@@ -44,5 +62,15 @@ export function migrationCard(origin: string | undefined, progress: Progress, cu
4462 export async function migrationRegion(ctx: Ctx, repo: Repo, owner: boolean): Promise<Html | null> {
4563 const view = await migrationView(ctx.app, ctx.auth.user, repo);
4664 if (view === undefined) return null;
47 return migrationCard(view.origin, view.progress, owner ? `/${repo.ownerName}/${repo.name}/settings/app` : null);
65 const base = `/${repo.ownerName}/${repo.name}`;
66 const state = owner ? dataMoveState(ctx.app, ctx.auth.user, repo) : undefined;
67 const last = state?.ok === true && state.value.last !== undefined && !state.value.last.undone ? state.value.last : undefined;
68 const data: DataLine | null =
69 state?.ok === true
70 ? {
71 href: `${base}/data-move`,
72 imported: last === undefined ? undefined : { tables: last.tables.length, rows: last.tables.reduce((n, t) => n + t.rows, 0) },
73 }
74 : null;
75 return migrationCard(view.origin, view.progress, owner ? `${base}/settings/app` : null, data);
4876 }
test/data-move.test.ts added +329
@@ -0,0 +1,329 @@
1 /**
2 * A platform app's data into the forge app's store (#647): the receipt matches a hand
3 * calculation, undo restores the snapshot, a cut-over closes the way back, and every door
4 * (page, API, MCP, terminal) gives the same gate and words.
5 */
6 import { Database } from "bun:sqlite";
7 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
8 import { existsSync, mkdirSync, readdirSync, rmSync, writeFileSync } from "node:fs";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { createApp, type App } from "../src/app.ts";
13 import { createRepo } from "../src/actions/repos.ts";
14 import { dataMoveState, moveData, undoMove } from "../src/actions/migrate_data.ts";
15 import { envDataDir } from "../src/apps/data.ts";
16 import { EXPORT_COMMAND, fromPostgres, lostKeys } from "../src/apps/pg_dump.ts";
17 import { createApp as insertApp } from "../src/core/apps.ts";
18 import { addDomain, recordCheck, setOldRecords } from "../src/core/domains.ts";
19 import type { Repo } from "../src/core/repos.ts";
20 import { createUser, type User } from "../src/core/users.ts";
21 import { receiptTable } from "../src/web/studio/data_move.ts";
22 import { moveRows } from "../src/tui/screens/datamove.ts";
23 import { dropDir } from "../src/drop/store.ts";
24 import type { Ctx } from "../src/http/context.ts";
25 import { serve, type Servers } from "../src/http/server.ts";
26 import { TOOLS } from "../src/http/mcp/tools.ts";
27 import { quietLogs } from "./helpers/globals.ts";
28 import { promote } from "./helpers/members.ts";
29
30 quietLogs();
31
32 const PG = `--
33 -- PostgreSQL database dump
34 --
35 SET statement_timeout = 0;
36 SET client_encoding = 'UTF8';
37 SELECT pg_catalog.set_config('search_path', '', false);
38 CREATE TABLE public.items (
39 id integer NOT NULL,
40 name character varying(40),
41 note text
42 );
43 ALTER TABLE public.items OWNER TO shop;
44 INSERT INTO public.items (id, name, note) VALUES (1, 'bolt', 'see public.items; ok');
45 INSERT INTO public.items (id, name, note) VALUES (2, 'nut', NULL);
46 INSERT INTO public.items (id, name, note) VALUES (3, 'it''s', 'x');
47 ALTER TABLE ONLY public.items ADD CONSTRAINT items_pkey PRIMARY KEY (id);
48 SELECT pg_catalog.setval('public.items_id_seq', 3, true);
49 `;
50
51 /**
52 * The receipt's checksum by hand: each row's values as SQLite quote() literals, \x1f between, \n
53 * after.
54 */
55 function handSha(): string {
56 const rows = [
57 ["1", "'bolt'", "'see public.items; ok'"],
58 ["2", "'nut'", "NULL"],
59 ["3", "'it''s'", "'x'"],
60 ];
61 const h = new Bun.CryptoHasher("sha256");
62 for (const r of rows) h.update(`${r.join("\x1f")}\n`);
63 return h.digest("hex");
64 }
65
66 describe("a Postgres dump read for SQLite", () => {
67 test("session settings, ownership, constraints and sequences are left out and counted; data is untouched", () => {
68 const read = fromPostgres(PG);
69 expect(read.ok && read.postgres).toBe(true);
70 if (!read.ok) return;
71 expect(read.skipped).toEqual([
72 { kind: "SET", count: 2 },
73 { kind: "set_config", count: 1 },
74 { kind: "OWNER", count: 1 },
75 { kind: "key", count: 1 },
76 { kind: "setval", count: 1 },
77 ]);
78 expect(read.sql).toContain("CREATE TABLE items");
79 expect(read.sql).toContain("'see public.items; ok'");
80 expect(read.sql).not.toContain("OWNER");
81 });
82 test("keys are counted as keys, defaults before the general ALTER TABLE, and the words say keys are not kept", () => {
83 const read = fromPostgres(`-- PostgreSQL database dump
84 CREATE TABLE a (id integer NOT NULL, b integer);
85 ALTER TABLE ONLY public.a ALTER COLUMN id SET DEFAULT nextval('public.a_id_seq'::regclass);
86 ALTER TABLE ONLY public.a ADD CONSTRAINT a_pkey PRIMARY KEY (id);
87 ALTER TABLE ONLY public.a ADD CONSTRAINT a_b_key UNIQUE (b);
88 ALTER TABLE ONLY public.a ADD CONSTRAINT a_b_fk FOREIGN KEY (b) REFERENCES public.a(id);
89 ALTER TABLE ONLY public.a ADD CONSTRAINT a_chk CHECK (id > 0);
90 INSERT INTO public.a VALUES (1, 1);
91 `);
92 expect(read.ok && read.skipped).toEqual([
93 { kind: "default", count: 1 },
94 { kind: "key", count: 3 },
95 { kind: "constraint", count: 1 },
96 ]);
97 expect(read.ok && lostKeys(read.skipped)).toBe(true);
98 expect(read.ok && read.sql).not.toContain("PRIMARY KEY");
99 expect(lostKeys([])).toBe(false);
100 });
101 test("COPY is refused with the fix; plain SQL goes through as it is", () => {
102 expect(fromPostgres("-- PostgreSQL database dump\nCOPY t (a) FROM stdin;\n1\n\\.\n")).toEqual({ ok: false, error: "copy" });
103 expect(fromPostgres("CREATE TABLE t(a);")).toEqual({ ok: true, sql: "CREATE TABLE t(a);", skipped: [], postgres: false });
104 });
105 });
106
107 describe("moving the data", () => {
108 let dataDir: string;
109 let app: App;
110 let servers: Servers;
111 let cy: User;
112 let dee: User;
113 let repo: Repo;
114 let appId: string;
115 let cookie: string;
116 let deeCookie: string;
117
118 const prod = () => envDataDir(dataDir, appId, "production");
119 const drop = (name: string, text: string) => {
120 mkdirSync(dropDir(dataDir, repo.id), { recursive: true });
121 writeFileSync(join(dropDir(dataDir, repo.id), name), text);
122 };
123 const signin = async (username: string) => {
124 const r = await fetch(`${servers.url}/signin`, { method: "POST", body: new URLSearchParams({ username, password: "a-good-passphrase" }), redirect: "manual" });
125 return `${(r.headers.get("set-cookie") ?? "").split(";")[0]!}; forge_mode=dev`;
126 };
127 const items = (file = "app.db") => {
128 const db = new Database(join(prod(), file), { readonly: true });
129 try {
130 return db.query("SELECT id FROM items ORDER BY id").values().map((r) => r[0]);
131 } finally {
132 db.close();
133 }
134 };
135
136 beforeAll(async () => {
137 dataDir = await mkdtemp(join(tmpdir(), "forge-data-move-"));
138 app = createApp({ dataDir, port: 0, host: "forge.test" });
139 servers = await serve(app);
140 const make = async (username: string, adminIfFirst = false) => {
141 const made = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst });
142 if (!made.ok) throw new Error(made.error);
143 return made.value;
144 };
145 await make("ada", true);
146 cy = await make("cy");
147 promote(app.db, cy);
148 dee = await make("dee");
149 const r = await createRepo(app, cy, { name: "shop", visibility: "public", description: "" });
150 if (!r.ok) throw new Error(String(r.error));
151 repo = r.value;
152 const row = insertApp(app.db, { repoId: repo.id, slug: "shop", access: "public" });
153 if (!row.ok) throw new Error(row.error);
154 appId = row.value.id;
155 cookie = await signin("cy");
156 deeCookie = await signin("dee");
157 drop("dump.sql", PG);
158 }, 60_000);
159
160 afterAll(async () => {
161 servers?.stop();
162 app.db.close();
163 await rm(dataDir, { recursive: true, force: true });
164 });
165
166 test("the first state names the command and the dump in the drop", () => {
167 const state = dataMoveState(app, cy, repo);
168 expect(state.ok && state.value.command).toBe(EXPORT_COMMAND);
169 expect(state.ok && state.value.files.map((f) => f.name)).toEqual(["dump.sql"]);
170 expect(state.ok && state.value.last).toBeUndefined();
171 expect(dataMoveState(app, dee, repo)).toEqual({ ok: false, error: "not_found" });
172 });
173
174 test("a check gives the real receipt and changes nothing", () => {
175 const done = moveData(app, cy, repo, { dump: "dump.sql" });
176 expect(done.ok).toBe(true);
177 if (!done.ok) return;
178 expect(done.value.applied).toBe(false);
179 expect(done.value.tables).toEqual([{ table: "items", rows: 3, dump: handSha(), store: handSha(), counted: 3 }]);
180 expect(done.value.skipped.length).toBeGreaterThan(0);
181 expect(existsSync(join(prod(), "app.db"))).toBe(false);
182 expect(readdirSync(prod()).filter((n) => n.includes("importing"))).toEqual([]);
183 expect(dataMoveState(app, cy, repo)).toMatchObject({ ok: true, value: { last: undefined } });
184 });
185
186 test("a reader's and a stranger's move is not found and writes nothing", () => {
187 expect(moveData(app, dee, repo, { dump: "dump.sql", apply: true })).toEqual({ ok: false, error: "not_found" });
188 expect(moveData(app, undefined, repo, { dump: "dump.sql", apply: true })).toEqual({ ok: false, error: "unauthorized" });
189 expect(existsSync(join(prod(), "app.db"))).toBe(false);
190 });
191
192 test("names are the drop's: a path or a missing file is refused", () => {
193 expect(moveData(app, cy, repo, { dump: "../app.db" })).toEqual({ ok: false, error: "invalid_dump" });
194 expect(moveData(app, cy, repo, { dump: "nope.sql" })).toEqual({ ok: false, error: "missing" });
195 });
196
197 test("with no store before, apply loads it, the receipt matches, and undo leaves the app empty", () => {
198 const done = moveData(app, cy, repo, { dump: "dump.sql", apply: true });
199 expect(done.ok && done.value.tables[0]).toEqual({ table: "items", rows: 3, dump: handSha(), store: handSha(), counted: 3 });
200 expect(done.ok && done.value.snapshot).toBeNull();
201 expect(items()).toEqual([1, 2, 3]);
202 const state = dataMoveState(app, cy, repo);
203 expect(state.ok && state.value.canUndo).toBe(true);
204 expect(state.ok && state.value.last?.tables).toEqual([{ table: "items", rows: 3, sha256: handSha(), store: handSha() }]);
205 const undone = undoMove(app, cy, repo);
206 expect(undone.ok && undone.value.restored).toBe(false);
207 expect(existsSync(join(prod(), "app.db"))).toBe(false);
208 expect(readdirSync(prod()).some((n) => n.startsWith("app.db.undone-import-"))).toBe(true);
209 expect(undoMove(app, cy, repo)).toEqual({ ok: false, error: "nothing_to_undo" });
210 });
211
212 test("with a store before, apply keeps it as the snapshot and undo restores exactly it", () => {
213 const old = new Database(join(prod(), "app.db"), { create: true });
214 old.exec("CREATE TABLE items(id INTEGER, name TEXT); INSERT INTO items VALUES (10,'old'),(11,'older');");
215 old.close();
216 const done = moveData(app, cy, repo, { dump: "dump.sql", apply: true });
217 expect(done.ok && done.value.snapshot).toMatch(/^app\.db\.before-import-\d+$/);
218 expect(items()).toEqual([1, 2, 3]);
219 const undone = undoMove(app, cy, repo);
220 expect(undone.ok && undone.value.restored).toBe(true);
221 expect(items()).toEqual([10, 11]);
222 expect(dataMoveState(app, cy, repo)).toMatchObject({ ok: true, value: { canUndo: false, last: { undone: true } } });
223 });
224
225 test("a POST without do, or with an unknown do, changes nothing; the buttons carry their do", async () => {
226 const before = readdirSync(prod()).sort();
227 for (const form of [{ dump: "dump.sql" }, { dump: "dump.sql", do: "" }, { dump: "dump.sql", do: "drop" }]) {
228 const r = await fetch(`${servers.url}/cy/shop/data-move`, { method: "POST", headers: { cookie }, body: new URLSearchParams(form) });
229 expect(r.status).toBe(422);
230 expect(await r.text()).toContain("nothing was changed");
231 }
232 expect(readdirSync(prod()).sort()).toEqual(before);
233 const text = await (await fetch(`${servers.url}/cy/shop/data-move`, { headers: { cookie } })).text();
234 expect(text).toMatch(/name="do"\s+value="check"/);
235 });
236
237 test("a cut-over closes the way back: no undo, no new import", () => {
238 expect(moveData(app, cy, repo, { dump: "dump.sql", apply: true }).ok).toBe(true);
239 const added = addDomain(app.db, { host: "shop.example.org", appId }, { forgeHost: "forge.test" });
240 if (!added.ok) throw new Error(added.error);
241 recordCheck(app.db, "shop.example.org", { verified: true, said: "ok" });
242 setOldRecords(app.db, "shop.example.org", [{ type: "A", name: "@", value: "192.0.2.1" }]);
243 expect(undoMove(app, cy, repo)).toEqual({ ok: false, error: "cut_over" });
244 expect(moveData(app, cy, repo, { dump: "dump.sql", apply: true })).toEqual({ ok: false, error: "cut_over" });
245 expect(moveData(app, cy, repo, { dump: "dump.sql" }).ok).toBe(true);
246 expect(items()).toEqual([1, 2, 3]);
247 });
248
249 test("the page: an admin sees the command and files, a reader finds nothing, a foreign post is refused", async () => {
250 const seen = await fetch(`${servers.url}/cy/shop/data-move`, { headers: { cookie } });
251 expect(seen.status).toBe(200);
252 const text = await seen.text();
253 expect(text).toContain("pg_dump --inserts");
254 expect(text).toContain("dump.sql");
255 expect(text).toContain("Import");
256 expect((await fetch(`${servers.url}/cy/shop/data-move`, { headers: { cookie: deeCookie } })).status).toBe(404);
257 const foreign = await fetch(`${servers.url}/cy/shop/data-move`, { method: "POST", headers: { cookie, origin: "https://evil.example" }, body: new URLSearchParams({ dump: "dump.sql", do: "import" }) });
258 expect(foreign.status).toBe(403);
259 const check = await fetch(`${servers.url}/cy/shop/data-move`, { method: "POST", headers: { cookie }, body: new URLSearchParams({ dump: "dump.sql", do: "check" }) });
260 expect(check.status).toBe(200);
261 const receipt = await check.text();
262 expect(receipt).toContain("Would load 1 tables, 3 rows");
263 expect(receipt).toContain(handSha().slice(0, 12));
264 });
265
266 test("the last import's receipt shows only what was recorded: no store hash, no match", () => {
267 const state = dataMoveState(app, cy, repo);
268 expect(state.ok && state.value.last?.tables[0]?.store).toBe(handSha());
269 const bare = String(receiptTable([{ table: "items", rows: 3, dump: handSha(), store: null, counted: null }]));
270 expect(bare).not.toContain("match");
271 expect(bare).not.toContain("Store SHA-256");
272 expect(String(receiptTable([{ table: "items", rows: 3, dump: handSha(), store: handSha(), counted: 3 }]))).toContain("match");
273 });
274
275 test("a file removed after listing is missing, not a throw; unauthorized is 401 on the API", async () => {
276 drop("gone.sql", "CREATE TABLE t(a);");
277 expect(dataMoveState(app, cy, repo).ok).toBe(true);
278 rmSync(join(dropDir(dataDir, repo.id), "gone.sql"));
279 expect(moveData(app, cy, repo, { dump: "gone.sql" })).toEqual({ ok: false, error: "missing" });
280 const r = await fetch(`${servers.url}/api/v1/repos/cy/shop/data-move`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ dump: "dump.sql" }) });
281 expect([401, 403]).toContain(r.status);
282 });
283
284 test("the app page's data line points owner at the page", async () => {
285 const { migrationCard } = await import("../src/web/studio/migration.ts");
286 const markup = String(migrationCard(undefined, { items: [], done: 0, total: 0, atParity: false }, null, { href: "/cy/shop/data-move", imported: undefined }));
287 expect(markup).toContain('href="/cy/shop/data-move"');
288 expect(markup).toContain("Import data");
289 const after = String(migrationCard(undefined, { items: [], done: 0, total: 0, atParity: false }, null, { href: "/cy/shop/data-move", imported: { tables: 2, rows: 9 } }));
290 expect(after).toContain("2 tables, 9 rows imported and proven");
291 expect(after).toContain("Receipt");
292 });
293
294 test("the API: state, check, and an admin scope for the move", async () => {
295 const api = `${servers.url}/api/v1/repos/cy/shop/data-move`;
296 const state = await fetch(api, { headers: { cookie } });
297 expect(state.status).toBe(200);
298 expect(((await state.json()) as { command: string }).command).toBe(EXPORT_COMMAND);
299 expect((await fetch(api, { headers: { cookie: deeCookie } })).status).toBe(404);
300 const bad = await fetch(api, { method: "POST", headers: { cookie, "content-type": "application/json" }, body: JSON.stringify({}) });
301 expect(bad.status).toBe(422);
302 const missing = await fetch(api, { method: "POST", headers: { cookie, "content-type": "application/json" }, body: JSON.stringify({ dump: "nope.sql" }) });
303 expect(missing.status).toBe(404);
304 });
305
306 test("the terminal draws the same receipt: a row per table, success when it matches", () => {
307 const done = moveData(app, cy, repo, { dump: "dump.sql" });
308 expect(done.ok && moveRows(done.value)).toEqual([
309 { title: "items: 3 rows", tone: "success", note: `dump ${handSha().slice(0, 16)} store ${handSha().slice(0, 16)} counted in dump 3` },
310 ]);
311 });
312
313 test("the MCP tool: state, a dry-run receipt, and an admin scope to apply or undo", async () => {
314 const tool = TOOLS.find((t) => t.name === "migrate_data")!;
315 const as = (user: User, scope: string, args: Record<string, unknown>) =>
316 tool.run({ app, url: new URL("http://forge.test/mcp"), auth: { scope } } as unknown as Ctx, user, args);
317 expect(tool.readsWhen?.({ repo: "cy/shop", dump: "dump.sql" })).toBe(true);
318 expect(tool.readsWhen?.({ repo: "cy/shop", dump: "dump.sql", apply: true })).toBe(false);
319 expect(await as(cy, "ro", { repo: "cy/shop" })).toMatchObject({ ok: true });
320 const plan = await as(cy, "ro", { repo: "cy/shop", dump: "dump.sql" });
321 expect(plan).toMatchObject({ ok: true });
322 expect(JSON.stringify(plan)).toContain(handSha());
323 expect(await as(cy, "rw", { repo: "cy/shop", dump: "dump.sql", apply: true })).toMatchObject({ ok: false, error: "forbidden" });
324 expect(await as(cy, "rw", { repo: "cy/shop", undo: true })).toMatchObject({ ok: false, error: "forbidden" });
325 expect(await as(dee, "admin", { repo: "cy/shop", dump: "dump.sql" })).toMatchObject({ ok: false });
326 // The cut-over is still in place from the test above: the move is refused in words.
327 expect(await as(cy, "admin", { repo: "cy/shop", dump: "dump.sql", apply: true })).toMatchObject({ ok: false, error: "cut_over" });
328 });
329 });
test/mcp-write-scope.test.ts +1
@@ -51,6 +51,7 @@ const ADMIN_ARGS: Record<string, Record<string, unknown>> = {
5151 const WRITE_ARGS: Record<string, Record<string, unknown>> = {
5252 flow: { kind: "drift", action: "update", repo: "alice/game", pull: 1 },
5353 repo: { repo: "alice/game", action: "open", chapter: "v1", confirm: true },
54 migrate_data: { repo: "alice/game", dump: "dump.sql", apply: true },
5455 system: { kind: "growth", repo: "alice/game", action: "pause", confirm: true },
5556 };
5657
test/mcp.test.ts +1
@@ -286,6 +286,7 @@ describe("the forge over MCP", () => {
286286 "detach",
287287 "trail",
288288 "migrate",
289 "migrate_data",
289290 "events",
290291 "reports",
291292 "branches",
test/worker-database-opens.test.ts +1
@@ -58,6 +58,7 @@ const ONE_JOB: Record<string, string> = {
5858 const REACHED: Record<string, { opens: number; why: string }> = {
5959 "src/apps/data.ts": { opens: 1, why: "quiesceSqlite folds each app database's log before a copy: called by the serving thread's clone and backup paths, in reach of Workers only through the page layer's imports" },
6060 "src/apps/dump_import.ts": { opens: 3, why: "a person's import of a dump, on the serving thread behind a request; no Worker calls it" },
61 "src/apps/dump_undo.ts": { opens: 1, why: "an undo folds the live store's log before it is set aside, on the serving thread behind an admin's request; no Worker calls it" },
6162 "src/crew/sentinel.ts": { opens: 1, why: "one scratch database in memory, made once for the process and remembered; the sentinel is a duty of the serving thread" },
6263 "src/data/browse.ts": { opens: 1, why: "openReadOnly for a data page, on the serving thread behind a request; no Worker calls it" },
6364 "src/kernel/db.ts": { opens: 2, why: "the forge's own database, opened once by createApp on the thread that boots; the Workers that read it open their own handle and keep it (KEEPS)" },

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs