Public: anyone can read this; the crew can change it.
Data migration: a platform app's Postgres goes into the drop and then into the forge app's store, with a receipt #13473
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/647-data-migration-a-platform-app-s-postgres → main 5 commits fork 3 threads · 3 unresolved
32 files
changed +1220
−11
docs/MCP.md +1
| @@ -154,6 +154,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`; | ||
| 154 | 154 | | `map` | — | An app's plan: ready or waiting, critical path, crew, parkedTwice; `done` adds closed; `first: N` ranks N first | 521 | |
| 155 | 155 | | `merge` | `land` | Land a pull, as its page's Merge button does; a person's only | 362 | |
| 156 | 156 | | `migrate` | — | Bring a repo in by public `url` (`name`, `crew`), history and all | 315 | |
| 157 | | `migrate_data` | — | Move a platform app's Postgres dump from its drop into its store; dry-run unless `apply` | 299 | | |
| 157 | 158 | | `milestones` | — | How far each ask is | 439 | |
| 158 | 159 | | `mint_invite` | — | Mint a one-use invite link, shown once, valid `days` (1–30, default 7) | 225 | |
| 159 | 160 | | `mirror_now` | — | Admin: bring the snapshot mirror level now | 129 | |
docs/features/README.md +1
| @@ -35,6 +35,7 @@ Read with the guide page [What a forge is](../guide-what-a-forge-is.md). | ||
| 35 | 35 | - [](home-looks.md) |
| 36 | 36 | - [](front-milestones.md) |
| 37 | 37 | - [](migration-mode.md) |
| 38 | - [](data-move.md) | |
| 38 | 39 | - [](now-pages.md) |
| 39 | 40 | - [](pixel.md) |
| 40 | 41 | - [](home-plans.md) |
docs/features/data-move.md added +45
| @@ -0,0 +1,45 @@ | ||
| 1 | # Moving a platform app's data (#647) | |
| 2 | ||
| 3 | A platform app's Postgres goes through the app's drop into the forge app's own store, with a | |
| 4 | receipt, and can be taken back until a domain is cut over. It is the data line of the migration | |
| 5 | checklist (`docs/features/migration-mode.md`); the engine under it is the database import in | |
| 6 | `docs/features/platform-app-move.md`. | |
| 7 | ||
| 8 | ## The path | |
| 9 | 1. **Make the dump** where the old database is reachable. The page shows the one command | |
| 10 | (`EXPORT_COMMAND`, `src/apps/pg_dump.ts`): `pg_dump --inserts --no-owner --no-privileges | |
| 11 | --no-comments "$DATABASE_URL" > dump.sql`. The connection string stays on that machine; the forge | |
| 12 | never receives it. (The forge running `pg_dump` itself with a connection string given once was | |
| 13 | left out: it needs a Postgres client and an egress entry to a host the owner names.) | |
| 14 | 2. **Put it in the drop** (`/<owner>/<name>/drop`, up to 16 MB for an import). | |
| 15 | 3. **Check, then import**, at `/<owner>/<name>/data-move`, reached from **Import data** on the app | |
| 16 | page's migration card. Check does everything but put the file in place; Import first keeps the | |
| 17 | store it replaces as `app.db.before-import-<ms>` (the snapshot), then replaces it. | |
| 18 | 4. **The receipt**: per table the rows and SHA-256 on the dump's side and on the store's, and the | |
| 19 | rows counted from the dump's own text. Import happens only when all agree. What a Postgres dump | |
| 20 | has that SQLite has no meaning for (session `SET`s, `set_config`, `setval`, ownership, | |
| 21 | column defaults, `ALTER TABLE` constraints, sequences, comments, extensions, schemas) is left out, counted, and | |
| 22 | shown on the receipt. **Primary, unique and foreign keys are not kept**: the tables load without them, and the receipt says so; `public.` is dropped from names. `COPY` is refused: dump with `--inserts`. | |
| 23 | Anything else the import's vetting refuses by statement number (`SELECT`, functions, `UPDATE`). | |
| 24 | 5. **Undo** puts the snapshot back and sets the imported store aside as | |
| 25 | `app.db.undone-import-<ms>`; nothing is deleted. When there was no store before, the imported one | |
| 26 | is set aside and the app starts empty as it was. It is open until a domain of the app is verified | |
| 27 | and has its old DNS recorded (the cut-over, `app_domain.old_records`); then an import and an undo | |
| 28 | are both refused, since the old data is no longer the way back. | |
| 29 | ||
| 30 | The record is the `app.data_imported` and `app.data_undone` events (the first carries the snapshot's | |
| 31 | name and each table's rows and checksum): no state of its own. | |
| 32 | ||
| 33 | ## Doors (all a repository admin's; anyone else gets not found) | |
| 34 | - Page `/<owner>/<name>/data-move` (GET, POST `do=check|import|undo`, same-origin checked). | |
| 35 | - API `GET` and `POST /api/v1/repos/<owner>/<name>/data-move`: `{ dump }` checks, `{ dump, apply: true }` | |
| 36 | imports, `{ undo: true }` undoes; an admin-scope token for POST. | |
| 37 | - MCP `migrate_data { repo, dump?, apply?, undo? }`: only `repo` answers the command, the drop's | |
| 38 | dumps and the last receipt; `dump` is a dry run; `apply`/`undo` need an admin-scope token. | |
| 39 | - Terminal: the "Move the data" screen (`src/tui/screens/datamove.ts`): `m` checks a drop file, | |
| 40 | `a` imports it, `u` undoes after a y/n. | |
| 41 | ||
| 42 | ## Not done | |
| 43 | - The running app keeps its old file open until it restarts: restart it after an import or an undo. | |
| 44 | - Only the store `app.db` in production; files and uploads are `scripts/copy-app-files.ts`. | |
| 45 | - Types are not translated: SQLite keeps the Postgres type names as declared and is loose about them. |
docs/features/platform-app-move.md +2 −1
| @@ -75,7 +75,8 @@ Not done: the load runs on the request, not in a worker. With only literal value | ||
| 75 | 75 | its time is linear in the dump's bytes (about a second near 16 MB), which is why a worker with a |
| 76 | 76 | timeout was left out; the running app keeps its old file open until it restarts, so import with the app |
| 77 | 77 | stopped or restart it after; the proof shows the load agrees with itself and with the dump's row count, |
| 78 | not that the old node's dump was complete: compare its row counts too. No web page yet. | |
| 78 | not that the old node's dump was complete: compare its row counts too. A Postgres dump read from | |
| 79 | the app's drop, with a page, a receipt and an undo: `docs/features/data-move.md` (#647). | |
| 79 | 80 | |
| 80 | 81 | ## 4. Rehearse on a preview (person) |
| 81 | 82 | Preview address, a copy of the data, the payment **sandbox** keys. The owner, |
forge.json +2 −2
| @@ -4,8 +4,8 @@ | ||
| 4 | 4 | "api.github.com" |
| 5 | 5 | ], |
| 6 | 6 | "app": { |
| 7 | "looks": ["/", "/docs", "/explore", "/board", "/notes", "/review", "/releases", "/apps", "/security", "/system", "/maintenance", "/talk", "/flow", "/resilience", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"], | |
| 8 | "looksAsOwner": ["/", "/board", "/flow", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/talk", "/system", "/maintenance", "/resilience", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"], | |
| 7 | "looks": ["/", "/docs", "/explore", "/board", "/notes", "/review", "/releases", "/apps", "/security", "/system", "/maintenance", "/talk", "/flow", "/resilience", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"], | |
| 8 | "looksAsOwner": ["/", "/board", "/flow", "/boards?help", "/settings/crew", "/settings/crew?help", "/settings/security/controls", "/settings/grants", "/talk", "/system", "/maintenance", "/resilience", "/preview-admin/sample", "/preview-admin/sample/drop", "/preview-admin/sample/data-move", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next?pick", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/space"], | |
| 9 | 9 | "run": "bun run src/main.ts", |
| 10 | 10 | "install": "", |
| 11 | 11 | "health": "/healthz", |
src/actions/migrate.ts +8 −1
| @@ -953,6 +953,8 @@ export function importAppData( | ||
| 953 | 953 | readonly env?: string; |
| 954 | 954 | readonly file?: string; |
| 955 | 955 | readonly replace?: boolean; |
| 956 | /** Everything but putting the file in place; no event, nothing changed (#647). */ | |
| 957 | readonly dryRun?: boolean; | |
| 956 | 958 | }, |
| 957 | 959 | ): Result<ImportReport, ImportDataError> { |
| 958 | 960 | const user = requireUser(actor); |
| @@ -969,8 +971,12 @@ export function importAppData( | ||
| 969 | 971 | file, |
| 970 | 972 | input.dump, |
| 971 | 973 | input.replace === true, |
| 974 | undefined, | |
| 975 | Date.now(), | |
| 976 | input.dryRun === true, | |
| 972 | 977 | ); |
| 973 | 978 | if (!done.ok) return err(done.error); |
| 979 | if (input.dryRun === true) return ok(done.value); | |
| 974 | 980 | storageStale(app, repo.ownerId); |
| 975 | 981 | app.bus.append({ |
| 976 | 982 | kind: "app.data_imported", |
| @@ -979,7 +985,8 @@ export function importAppData( | ||
| 979 | 985 | payload: { |
| 980 | 986 | env, |
| 981 | 987 | file, |
| 982 | tables: done.value.tables.map((t) => ({ table: t.table, rows: t.rows, sha256: t.sha256 })), | |
| 988 | kept: done.value.kept ?? null, | |
| 989 | tables: done.value.tables.map((t) => ({ table: t.table, rows: t.rows, sha256: t.sha256, store: t.store?.sha256 ?? null })), | |
| 983 | 990 | }, |
| 984 | 991 | }); |
| 985 | 992 | return ok(done.value); |
src/actions/migrate_data.ts added +237
| @@ -0,0 +1,237 @@ | ||
| 1 | /** | |
| 2 | * Moving a platform app's data into the forge app's own store (#647), one flow for the page, the | |
| 3 | * API, the MCP `migrate_data` tool and the terminal: the owner runs one shown command on the old | |
| 4 | * database (`apps/pg_dump.ts`) and puts the dump in the app's drop; `moveData` reads it from there | |
| 5 | * (dry-run unless `apply`), loads it into the app's production SQLite and answers the receipt — | |
| 6 | * per table, rows and a SHA-256 on the dump's side and on the store's — and `undoMove` puts the | |
| 7 | * store back as the snapshot taken before the import, until a domain has been cut over. Admins of | |
| 8 | * the repository only; anyone else is told `not_found`. The record of what was imported is the | |
| 9 | * `app.data_imported` / `app.data_undone` events, so there is no state of its own. | |
| 10 | */ | |
| 11 | ||
| 12 | import { readFileSync, statSync } from "node:fs"; | |
| 13 | import type { App } from "../app.ts"; | |
| 14 | import { envDataDir } from "../apps/data.ts"; | |
| 15 | import { DUMP_MAX } from "../apps/dump_file.ts"; | |
| 16 | import type { ImportReport } from "../apps/dump_import.ts"; | |
| 17 | import { undoImport } from "../apps/dump_undo.ts"; | |
| 18 | import { EXPORT_COMMAND, fromPostgres, skippedWords, type Skipped } from "../apps/pg_dump.ts"; | |
| 19 | import { getAppByRepo } from "../core/apps.ts"; | |
| 20 | import { listDomains } from "../core/domains.ts"; | |
| 21 | import { listEvents } from "../core/events.ts"; | |
| 22 | import { canAdmin, type Repo } from "../core/repos.ts"; | |
| 23 | import type { User } from "../core/users.ts"; | |
| 24 | import { filePath, listFiles, validName } from "../drop/store.ts"; | |
| 25 | import { err, ok, type Result } from "../kernel/result.ts"; | |
| 26 | import { requireUser } from "./access.ts"; | |
| 27 | import { importAppData, importDataMessage, type ImportDataError } from "./migrate.ts"; | |
| 28 | ||
| 29 | export type MoveError = | |
| 30 | | ImportDataError | |
| 31 | | "invalid_dump" | |
| 32 | | "missing" | |
| 33 | | "too_large_dump" | |
| 34 | | "copy" | |
| 35 | | "cut_over" | |
| 36 | | "nothing_to_undo" | |
| 37 | | "snapshot_gone" | |
| 38 | | "store_gone"; | |
| 39 | ||
| 40 | /** One table as the receipt shows it. */ | |
| 41 | export interface ReceiptRow { | |
| 42 | readonly table: string; | |
| 43 | readonly rows: number; | |
| 44 | /** SHA-256, hex, of the rows as the dump holds them. */ | |
| 45 | readonly dump: string; | |
| 46 | /** The same as the store holds them; equal to `dump` or the move did not happen. */ | |
| 47 | readonly store: string; | |
| 48 | /** Rows the dump's own text says it inserts, counted apart from SQLite. */ | |
| 49 | readonly counted: number; | |
| 50 | } | |
| 51 | ||
| 52 | export interface MoveReceipt { | |
| 53 | readonly applied: boolean; | |
| 54 | /** The drop file the dump was read from. */ | |
| 55 | readonly dump: string; | |
| 56 | readonly file: string; | |
| 57 | readonly statements: number; | |
| 58 | readonly tables: readonly ReceiptRow[]; | |
| 59 | /** What a Postgres dump had that SQLite has no meaning for, left out and counted. */ | |
| 60 | readonly skipped: readonly Skipped[]; | |
| 61 | /** The snapshot the import took of the store it replaced; null when there was none. */ | |
| 62 | readonly snapshot: string | null; | |
| 63 | } | |
| 64 | ||
| 65 | const rowsOf = (report: ImportReport): ReceiptRow[] => | |
| 66 | report.tables.map((t) => ({ | |
| 67 | table: t.table, | |
| 68 | rows: t.rows, | |
| 69 | dump: t.sha256, | |
| 70 | store: t.store?.sha256 ?? "", | |
| 71 | counted: t.inserted, | |
| 72 | })); | |
| 73 | ||
| 74 | /** A domain whose old DNS was written down is the cut-over: the old data's way back is closed. */ | |
| 75 | export function cutOverOf(app: App, appId: string): boolean { | |
| 76 | return listDomains(app.db, appId).some((d) => d.verifiedAt !== null && d.oldRecords.length > 0); | |
| 77 | } | |
| 78 | ||
| 79 | /** | |
| 80 | * Read `dump` (a name in the app's drop) and load it into the production store. Without `apply` | |
| 81 | * it goes as far as the proof and puts nothing in place: the receipt is the one apply would give. | |
| 82 | * With it, the store being replaced is kept first, and `undoMove` can restore it. | |
| 83 | */ | |
| 84 | export function moveData( | |
| 85 | app: App, | |
| 86 | actor: User | undefined, | |
| 87 | repo: Repo, | |
| 88 | input: { readonly dump: string; readonly apply?: boolean }, | |
| 89 | ): Result<MoveReceipt, MoveError> { | |
| 90 | const user = requireUser(actor); | |
| 91 | if (!user.ok) return user; | |
| 92 | if (!canAdmin(app.db, repo, user.value)) return err("not_found"); | |
| 93 | const record = getAppByRepo(app.db, repo.id); | |
| 94 | if (!record) return err("no_app"); | |
| 95 | if (!validName(input.dump)) return err("invalid_dump"); | |
| 96 | const path = filePath(app.config.dataDir, repo.id, input.dump); | |
| 97 | if (path === undefined) return err("missing"); | |
| 98 | // The file may be removed between the listing and this read: that is `missing`, not a throw. | |
| 99 | let text: string; | |
| 100 | try { | |
| 101 | if (statSync(path).size > DUMP_MAX) return err("too_large_dump"); | |
| 102 | text = readFileSync(path, "utf8"); | |
| 103 | } catch { | |
| 104 | return err("missing"); | |
| 105 | } | |
| 106 | const read = fromPostgres(text); | |
| 107 | if (!read.ok) return err(read.error); | |
| 108 | const apply = input.apply === true; | |
| 109 | if (apply && cutOverOf(app, record.id)) return err("cut_over"); | |
| 110 | const done = importAppData(app, user.value, repo, { dump: read.sql, replace: true, dryRun: !apply }); | |
| 111 | if (!done.ok) return done; | |
| 112 | return ok({ | |
| 113 | applied: apply, | |
| 114 | dump: input.dump, | |
| 115 | file: done.value.file, | |
| 116 | statements: done.value.statements, | |
| 117 | tables: rowsOf(done.value), | |
| 118 | skipped: read.skipped, | |
| 119 | snapshot: done.value.kept ?? null, | |
| 120 | }); | |
| 121 | } | |
| 122 | ||
| 123 | export interface LastImport { | |
| 124 | readonly at: string; | |
| 125 | readonly file: string; | |
| 126 | readonly snapshot: string | null; | |
| 127 | /** `store` is the store's own hash when the import recorded it; null on an older record. */ | |
| 128 | readonly tables: readonly { readonly table: string; readonly rows: number; readonly sha256: string; readonly store?: string | null }[]; | |
| 129 | /** Taken back since. */ | |
| 130 | readonly undone: boolean; | |
| 131 | } | |
| 132 | ||
| 133 | export interface DataMoveState { | |
| 134 | /** The one command that makes the dump. */ | |
| 135 | readonly command: string; | |
| 136 | /** `.sql` files in the drop, newest first: what `moveData` may read. */ | |
| 137 | readonly files: readonly { readonly name: string; readonly bytes: number }[]; | |
| 138 | readonly last: LastImport | undefined; | |
| 139 | readonly cutOver: boolean; | |
| 140 | /** An import is there to take back, and nothing has been cut over. */ | |
| 141 | readonly canUndo: boolean; | |
| 142 | } | |
| 143 | ||
| 144 | interface Imported { | |
| 145 | readonly id: string; | |
| 146 | readonly at: string; | |
| 147 | readonly file: string; | |
| 148 | readonly snapshot: string | null; | |
| 149 | readonly tables: LastImport["tables"]; | |
| 150 | } | |
| 151 | ||
| 152 | /** The latest production import event of this repository, with the rows it recorded. */ | |
| 153 | function lastImported(app: App, repo: Repo): Imported | undefined { | |
| 154 | for (const e of listEvents(app.db, { repoId: repo.id, kind: "app.data_imported", limit: 20 })) { | |
| 155 | const p = e.payload; | |
| 156 | if (p["env"] !== "production" || typeof p["file"] !== "string") continue; | |
| 157 | return { | |
| 158 | id: e.id, | |
| 159 | at: e.createdAt, | |
| 160 | file: p["file"], | |
| 161 | snapshot: typeof p["kept"] === "string" ? p["kept"] : null, | |
| 162 | tables: Array.isArray(p["tables"]) ? (p["tables"] as LastImport["tables"]) : [], | |
| 163 | }; | |
| 164 | } | |
| 165 | return undefined; | |
| 166 | } | |
| 167 | ||
| 168 | export function dataMoveState(app: App, actor: User | undefined, repo: Repo): Result<DataMoveState, "not_found"> { | |
| 169 | if (actor === undefined || !canAdmin(app.db, repo, actor)) return err("not_found"); | |
| 170 | const record = getAppByRepo(app.db, repo.id); | |
| 171 | if (!record) return err("not_found"); | |
| 172 | const imported = lastImported(app, repo); | |
| 173 | const undoneAt = listEvents(app.db, { repoId: repo.id, kind: "app.data_undone", limit: 1 })[0]?.id; | |
| 174 | const undone = imported !== undefined && undoneAt !== undefined && undoneAt > imported.id; | |
| 175 | const cutOver = cutOverOf(app, record.id); | |
| 176 | return ok({ | |
| 177 | command: EXPORT_COMMAND, | |
| 178 | files: listFiles(app.config.dataDir, repo.id) | |
| 179 | .filter((f) => /\.sql$/i.test(f.name)) | |
| 180 | .map((f) => ({ name: f.name, bytes: f.bytes })), | |
| 181 | last: imported === undefined ? undefined : { ...imported, undone }, | |
| 182 | cutOver, | |
| 183 | canUndo: imported !== undefined && !undone && !cutOver, | |
| 184 | }); | |
| 185 | } | |
| 186 | ||
| 187 | /** Put the store back as it was before the last import, until a domain has been cut over. */ | |
| 188 | export function undoMove( | |
| 189 | app: App, | |
| 190 | actor: User | undefined, | |
| 191 | repo: Repo, | |
| 192 | ): Result<{ readonly file: string; readonly restored: boolean; readonly setAside: string }, MoveError> { | |
| 193 | const user = requireUser(actor); | |
| 194 | if (!user.ok) return user; | |
| 195 | const state = dataMoveState(app, user.value, repo); | |
| 196 | if (!state.ok) return state; | |
| 197 | const record = getAppByRepo(app.db, repo.id)!; | |
| 198 | if (state.value.cutOver) return err("cut_over"); | |
| 199 | const last = state.value.last; | |
| 200 | if (last === undefined || last.undone) return err("nothing_to_undo"); | |
| 201 | const done = undoImport(envDataDir(app.config.dataDir, record.id, "production"), last.file, last.snapshot); | |
| 202 | if (!done.ok) return err(done.error === "missing_snapshot" ? "snapshot_gone" : done.error === "missing_store" ? "store_gone" : "invalid_dump"); | |
| 203 | app.bus.append({ | |
| 204 | kind: "app.data_undone", | |
| 205 | repoId: repo.id, | |
| 206 | actorId: user.value.id, | |
| 207 | payload: { env: "production", file: last.file, restored: done.value.restored, setAside: done.value.setAside }, | |
| 208 | }); | |
| 209 | return ok({ file: last.file, restored: done.value.restored, setAside: done.value.setAside }); | |
| 210 | } | |
| 211 | ||
| 212 | /** A refusal in words a person can act on. */ | |
| 213 | export function moveMessage(error: MoveError): string { | |
| 214 | if (typeof error !== "string" || ["unauthorized", "not_found", "no_app", "invalid_env", "invalid_file"].includes(error)) | |
| 215 | return importDataMessage(error as ImportDataError); | |
| 216 | switch (error) { | |
| 217 | case "invalid_dump": | |
| 218 | return "dump is the name of a file in the app's drop."; | |
| 219 | case "missing": | |
| 220 | return "No such file in the app's drop: put the dump there first."; | |
| 221 | case "too_large_dump": | |
| 222 | return "The dump is over 16 MB."; | |
| 223 | case "copy": | |
| 224 | return "This dump uses COPY. Dump again with --inserts: the shown command does."; | |
| 225 | case "cut_over": | |
| 226 | return "A domain has been cut over, so the old data is no longer the way back: nothing was changed."; | |
| 227 | case "nothing_to_undo": | |
| 228 | return "There is no import to take back."; | |
| 229 | case "snapshot_gone": | |
| 230 | return "The snapshot taken before the import is gone from the app's data, so it cannot be restored."; | |
| 231 | case "store_gone": | |
| 232 | return "The store the import made is gone from the app's data: nothing to put back."; | |
| 233 | } | |
| 234 | return importDataMessage(error); | |
| 235 | } | |
| 236 | ||
| 237 | export { skippedWords }; |
src/apps/dump_import.ts +7 −1
| @@ -204,7 +204,9 @@ const tableKey = (raw: string): string => { | ||
| 204 | 204 | return inner.toLowerCase(); |
| 205 | 205 | }; |
| 206 | 206 | |
| 207 | /** Rows each table should hold, counted from the dump's own text: the independent side of the proof. */ | |
| 207 | /** | |
| 208 | * Rows each table should hold, counted from the dump's own text: the independent side of the proof. | |
| 209 | */ | |
| 208 | 210 | export function insertedRows(statements: readonly string[]): Map<string, number> { |
| 209 | 211 | const counts = new Map<string, number>(); |
| 210 | 212 | for (const s of statements) { |
| @@ -318,6 +320,7 @@ function backupName(dir: string, file: string, now: number): string { | ||
| 318 | 320 | * `replace` is set, else refused. The target is never absent: the old file |
| 319 | 321 | * is hard-linked to its backup, then the new one is renamed over it. |
| 320 | 322 | * `afterLoad` is a seam for tests: it may touch the store before the proof. |
| 323 | * `dryRun` does all of it but the last step: the receipt is the real one, the target untouched. | |
| 321 | 324 | */ |
| 322 | 325 | export function importDump( |
| 323 | 326 | dir: string, |
| @@ -326,6 +329,7 @@ export function importDump( | ||
| 326 | 329 | replace: boolean, |
| 327 | 330 | afterLoad?: (store: Database) => void, |
| 328 | 331 | now: number = Date.now(), |
| 332 | dryRun = false, | |
| 329 | 333 | ): { ok: true; value: ImportReport } | { ok: false; error: ImportError } { |
| 330 | 334 | const statements = splitSql(sql); |
| 331 | 335 | if (statements.length === 0) return { ok: false, error: { error: "empty" } }; |
| @@ -352,6 +356,8 @@ export function importDump( | ||
| 352 | 356 | const tables = compareTables(fingerprint(scratch), fingerprint(store), insertedRows(statements)); |
| 353 | 357 | if (tables.some((t) => !t.match) || fingerprint(store).length !== tables.length) |
| 354 | 358 | return { ok: false, error: { error: "mismatch", tables } }; |
| 359 | // A dry run stops here: everything above ran, nothing is put in place (#647). | |
| 360 | if (dryRun) return { ok: true, value: { file, statements: statements.length, tables } }; | |
| 355 | 361 | store.exec("PRAGMA wal_checkpoint(TRUNCATE)"); |
| 356 | 362 | store.close(); |
| 357 | 363 | let kept: string | undefined; |
src/apps/dump_undo.ts added +42
| @@ -0,0 +1,42 @@ | ||
| 1 | /** | |
| 2 | * Taking an import back (#647). An import with `replace` keeps the store it replaced as | |
| 3 | * `<file>.before-import-<ms>` (`dump_import.ts`): that is the snapshot. Undo puts it back as the | |
| 4 | * store and sets the imported store aside as `<file>.undone-import-<ms>`, so nothing is deleted by | |
| 5 | * either step. When the import found no store, there is no snapshot and the imported one is set | |
| 6 | * aside, which leaves the app as empty as it was. Names are checked against what an import writes, | |
| 7 | * so a name from a record is never a path out of the directory. | |
| 8 | */ | |
| 9 | ||
| 10 | import { Database } from "bun:sqlite"; | |
| 11 | import { existsSync, renameSync, rmSync } from "node:fs"; | |
| 12 | import { join } from "node:path"; | |
| 13 | import { IMPORT_FILE } from "./dump_import.ts"; | |
| 14 | ||
| 15 | export type UndoError = "invalid" | "missing_snapshot" | "missing_store"; | |
| 16 | ||
| 17 | const keptName = (file: string, kept: string): boolean => | |
| 18 | kept.startsWith(`${file}.before-import-`) && /^[\w.-]+$/.test(kept) && !kept.includes(".."); | |
| 19 | ||
| 20 | export function undoImport( | |
| 21 | dir: string, | |
| 22 | file: string, | |
| 23 | kept: string | null, | |
| 24 | now: number = Date.now(), | |
| 25 | ): { ok: true; value: { restored: boolean; setAside: string } } | { ok: false; error: UndoError } { | |
| 26 | if (!IMPORT_FILE.test(file) || (kept !== null && !keptName(file, kept))) return { ok: false, error: "invalid" }; | |
| 27 | const target = join(dir, file); | |
| 28 | if (!existsSync(target)) return { ok: false, error: "missing_store" }; | |
| 29 | if (kept !== null && !existsSync(join(dir, kept))) return { ok: false, error: "missing_snapshot" }; | |
| 30 | try { | |
| 31 | const live = new Database(target, { readwrite: true }); | |
| 32 | live.exec("PRAGMA wal_checkpoint(TRUNCATE)"); | |
| 33 | live.close(); | |
| 34 | } catch { | |
| 35 | // Not SQLite after all: it is set aside as it is. | |
| 36 | } | |
| 37 | const setAside = `${file}.undone-import-${now}`; | |
| 38 | renameSync(target, join(dir, setAside)); | |
| 39 | for (const suffix of ["-wal", "-shm"]) rmSync(`${target}${suffix}`, { force: true }); | |
| 40 | if (kept !== null) renameSync(join(dir, kept), target); | |
| 41 | return { ok: true, value: { restored: kept !== null, setAside } }; | |
| 42 | } |
src/apps/pg_dump.ts added +79
| @@ -0,0 +1,79 @@ | ||
| 1 | /** | |
| 2 | * A Postgres dump made loadable by the SQLite import (`dump_import.ts`), and the one command | |
| 3 | * that makes it (#647). The import runs only `CREATE TABLE`, `CREATE INDEX`, `INSERT ... VALUES` | |
| 4 | * and transaction words, so a `pg_dump --inserts` file is first stripped of what SQLite has no | |
| 5 | * meaning for: session settings, `SELECT pg_catalog.set_config/setval`, ownership, comments, | |
| 6 | * extensions, schemas, sequences and `ALTER TABLE` (defaults, keys, constraints). Primary, unique | |
| 7 | * and foreign keys are therefore NOT kept: the tables load without them, and the receipt says so. | |
| 8 | * Nothing is dropped silently: | |
| 9 | * each kind skipped is counted and shown on the receipt, and anything else stays in the text for | |
| 10 | * the import's own vetting to refuse by statement number. `COPY ... FROM stdin` is refused with | |
| 11 | * the fix (dump again with `--inserts`). Plain data in, plain data out; the text is only read. | |
| 12 | */ | |
| 13 | ||
| 14 | import { splitSql } from "./dump_import.ts"; | |
| 15 | ||
| 16 | /** | |
| 17 | * What the owner runs on a machine that can reach the old database; the connection string never | |
| 18 | * leaves it. | |
| 19 | */ | |
| 20 | export const EXPORT_COMMAND = | |
| 21 | 'pg_dump --inserts --no-owner --no-privileges --no-comments "$DATABASE_URL" > dump.sql'; | |
| 22 | ||
| 23 | export interface Skipped { | |
| 24 | readonly kind: string; | |
| 25 | readonly count: number; | |
| 26 | } | |
| 27 | ||
| 28 | export type PgRead = | |
| 29 | | { readonly ok: true; readonly sql: string; readonly skipped: readonly Skipped[]; readonly postgres: boolean } | |
| 30 | | { readonly ok: false; readonly error: "copy" }; | |
| 31 | ||
| 32 | /** Whether the text is the kind `pg_dump` writes. */ | |
| 33 | export const looksLikePostgres = (text: string): boolean => | |
| 34 | /^--\s*PostgreSQL database dump/m.test(text.slice(0, 2000)) || /^SET\s+statement_timeout\b/im.test(text.slice(0, 4000)); | |
| 35 | ||
| 36 | const SKIPPABLE: readonly [string, RegExp][] = [ | |
| 37 | ["SET", /^SET\s/i], | |
| 38 | ["set_config", /^SELECT\s+pg_catalog\.set_config\s*\(/i], | |
| 39 | ["setval", /^SELECT\s+pg_catalog\.setval\s*\(/i], | |
| 40 | ["OWNER", /^ALTER\s+\S+(\s+ONLY)?\s+\S+\s+OWNER\s+TO\s/i], | |
| 41 | // Column defaults come before the general ALTER TABLE, which would match them first. | |
| 42 | ["default", /^ALTER\s+TABLE\s+\S+(\s+ONLY)?\s+\S*\s*ALTER\s+COLUMN\s/i], | |
| 43 | ["key", /^ALTER\s+TABLE\s.*\b(PRIMARY\s+KEY|UNIQUE|FOREIGN\s+KEY)\b/is], | |
| 44 | ["constraint", /^ALTER\s+TABLE\s/i], | |
| 45 | ["sequence", /^(CREATE|ALTER)\s+SEQUENCE\s/i], | |
| 46 | ["COMMENT", /^COMMENT\s+ON\s/i], | |
| 47 | ["EXTENSION", /^CREATE\s+EXTENSION\s/i], | |
| 48 | ["SCHEMA", /^CREATE\s+SCHEMA\s/i], | |
| 49 | ]; | |
| 50 | ||
| 51 | /** `public.name` and `"public"."name"` become `name`: SQLite has no schemas. */ | |
| 52 | const unqualify = (s: string): string => { | |
| 53 | // An INSERT's rows are data: only what comes before VALUES is a name. | |
| 54 | const at = /^INSERT\s/i.test(s) ? s.search(/\sVALUES\s*\(/i) : s.length; | |
| 55 | const head = at < 0 ? s : s.slice(0, at); | |
| 56 | return head.replace(/(?<![\w"])"?public"?\.(?=["\w])/gi, "") + (at < 0 ? "" : s.slice(at)); | |
| 57 | }; | |
| 58 | ||
| 59 | export function fromPostgres(text: string): PgRead { | |
| 60 | if (!looksLikePostgres(text)) return { ok: true, sql: text, skipped: [], postgres: false }; | |
| 61 | const kept: string[] = []; | |
| 62 | const counts = new Map<string, number>(); | |
| 63 | for (const statement of splitSql(text)) { | |
| 64 | if (/^COPY\s/i.test(statement)) return { ok: false, error: "copy" }; | |
| 65 | const hit = SKIPPABLE.find(([, re]) => re.test(statement)); | |
| 66 | if (hit) counts.set(hit[0], (counts.get(hit[0]) ?? 0) + 1); | |
| 67 | else kept.push(unqualify(statement)); | |
| 68 | } | |
| 69 | const skipped = [...counts].map(([kind, count]): Skipped => ({ kind, count })); | |
| 70 | return { ok: true, sql: kept.map((s) => `${s};`).join("\n"), skipped, postgres: true }; | |
| 71 | } | |
| 72 | ||
| 73 | /** Whether the dump had keys: the tables load without them, and the receipt must say so. */ | |
| 74 | export const lostKeys = (skipped: readonly Skipped[]): boolean => skipped.some((s) => s.kind === "key"); | |
| 75 | ||
| 76 | /** One line for the receipt: what was left out of a Postgres dump, or "" when nothing was. */ | |
| 77 | export function skippedWords(skipped: readonly Skipped[]): string { | |
| 78 | return skipped.map((s) => `${s.kind} ×${s.count}`).join(", "); | |
| 79 | } |
src/checks/looks.ts +1 −1
| @@ -31,7 +31,7 @@ const SAFE = /^[0-9A-Za-z]{1,64}$/; | ||
| 31 | 31 | |
| 32 | 32 | const MAX_BYTES = 1024 * 1024; |
| 33 | 33 | // The forge's own preview takes ~55 looks (`PREVIEW_OWNER_LOOKS`); every one must fit. |
| 34 | const MAX_LOOKS = 76; | |
| 34 | const MAX_LOOKS = 80; | |
| 35 | 35 | const MAX_WORDS = 80; |
| 36 | 36 | |
| 37 | 37 | const words = (v: unknown): string | null => |
src/contract/openapi/routes/getApiV1ReposOwnerNameDataMove.ts added +10
| @@ -0,0 +1,10 @@ | ||
| 1 | import { REPO, type Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "get", | |
| 5 | path: `${REPO}/data-move`, | |
| 6 | summary: | |
| 7 | "Where a platform app's data move stands: { command, files: [{ name, bytes }], last?: { at, file, snapshot, tables: [{ table, rows, sha256 }], undone }, cutOver, canUndo }; the command makes the dump (pg_dump --inserts), files are the .sql dumps in the app's drop; a repository admin's, anyone else a 404; needs a token", | |
| 8 | auth: true, | |
| 9 | role: "member", | |
| 10 | }; |
src/contract/openapi/routes/index.ts +2
| @@ -153,6 +153,7 @@ export { route as getApiV1ReposOwnerNameCrew } from "./getApiV1ReposOwnerNameCre | ||
| 153 | 153 | export { route as getApiV1ReposOwnerNameData } from "./getApiV1ReposOwnerNameData.ts"; |
| 154 | 154 | export { route as getApiV1ReposOwnerNameDataEnvFileRows } from "./getApiV1ReposOwnerNameDataEnvFileRows.ts"; |
| 155 | 155 | export { route as getApiV1ReposOwnerNameDataEnvFileTables } from "./getApiV1ReposOwnerNameDataEnvFileTables.ts"; |
| 156 | export { route as getApiV1ReposOwnerNameDataMove } from "./getApiV1ReposOwnerNameDataMove.ts"; | |
| 156 | 157 | export { route as getApiV1ReposOwnerNameDecisions } from "./getApiV1ReposOwnerNameDecisions.ts"; |
| 157 | 158 | export { route as getApiV1ReposOwnerNameDesk } from "./getApiV1ReposOwnerNameDesk.ts"; |
| 158 | 159 | export { route as getApiV1ReposOwnerNameDrop } from "./getApiV1ReposOwnerNameDrop.ts"; |
| @@ -351,6 +352,7 @@ export { route as postApiV1ReposOwnerNameContents } from "./postApiV1ReposOwnerN | ||
| 351 | 352 | export { route as postApiV1ReposOwnerNameContribute } from "./postApiV1ReposOwnerNameContribute.ts"; |
| 352 | 353 | export { route as postApiV1ReposOwnerNameDataEnvFileQuery } from "./postApiV1ReposOwnerNameDataEnvFileQuery.ts"; |
| 353 | 354 | export { route as postApiV1ReposOwnerNameDataImport } from "./postApiV1ReposOwnerNameDataImport.ts"; |
| 355 | export { route as postApiV1ReposOwnerNameDataMove } from "./postApiV1ReposOwnerNameDataMove.ts"; | |
| 354 | 356 | export { route as postApiV1ReposOwnerNameDesk } from "./postApiV1ReposOwnerNameDesk.ts"; |
| 355 | 357 | export { route as postApiV1ReposOwnerNameDrop } from "./postApiV1ReposOwnerNameDrop.ts"; |
| 356 | 358 | export { route as postApiV1ReposOwnerNameEdits } from "./postApiV1ReposOwnerNameEdits.ts"; |
src/contract/openapi/routes/postApiV1ReposOwnerNameDataMove.ts added +22
| @@ -0,0 +1,22 @@ | ||
| 1 | import { REPO, type Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "post", | |
| 5 | path: `${REPO}/data-move`, | |
| 6 | summary: | |
| 7 | "Move a platform app's Postgres dump from its drop into its SQLite store: { dump } checks (nothing changes) and { dump, apply: true } imports, keeping the store it replaces as the snapshot; { undo: true } puts the snapshot back until a domain has been cut over. Answers the receipt { applied, dump, file, statements, tables: [{ table, rows, dump, store, counted }], skipped, snapshot }; needs an admin-scope token and the right to administer the repository", | |
| 8 | auth: true, | |
| 9 | role: "member", | |
| 10 | body: { dump: "string?", apply: "boolean?", undo: "boolean?" }, | |
| 11 | refusals: { | |
| 12 | invalid_dump: "dump is not the name of a file (422)", | |
| 13 | missing: "no such file in the app's drop (404)", | |
| 14 | no_app: "the repository is not an app (422)", | |
| 15 | copy: "the dump uses COPY; dump again with --inserts (422)", | |
| 16 | refused: "a statement is of a kind a dump may not run; its number is named (422)", | |
| 17 | mismatch: "the store does not match the dump in some table; nothing was changed (409)", | |
| 18 | cut_over: "a domain has been cut over, so neither an import nor an undo is allowed (409)", | |
| 19 | nothing_to_undo: "there is no import to take back (409)", | |
| 20 | snapshot_gone: "the snapshot is no longer in the app's data (409)", | |
| 21 | }, | |
| 22 | }; |
src/contract/tags.ts +1
| @@ -93,6 +93,7 @@ const ALIAS: Readonly<Record<string, string>> = { | ||
| 93 | 93 | boards: "forges", |
| 94 | 94 | data: "apps", |
| 95 | 95 | "data-import": "apps", |
| 96 | "data-move": "apps", | |
| 96 | 97 | drop: "apps", |
| 97 | 98 | }; |
| 98 | 99 |
src/crew/conversation_confirm.ts +1
| @@ -119,6 +119,7 @@ export const ASKING: Readonly<Record<string, Asking>> = { | ||
| 119 | 119 | quotas: { does: "change an account's quotas" }, |
| 120 | 120 | spawn: { does: "spawn a child forge" }, |
| 121 | 121 | app_data_import: { does: "load a SQL dump into an app's database" }, |
| 122 | migrate_data: { does: "load a dump from an app's drop into its database, or take that import back", asks: (a) => a["apply"] === true || a["undo"] === true }, | |
| 122 | 123 | app_deploy: { does: "redeploy or stop an app's production or preview" }, |
| 123 | 124 | stop: { does: "stop a crew run" }, |
| 124 | 125 | converge: { does: "have a fork's crew merge the parent's updates" }, |
src/design/activity.ts +1
| @@ -207,6 +207,7 @@ const WORDS: Record<string, string> = { | ||
| 207 | 207 | "repo.mirrored": "mirrored a repository", |
| 208 | 208 | "repo.imported": "imported issues and pulls", |
| 209 | 209 | "app.data_imported": "loaded a database dump into an app", |
| 210 | "app.data_undone": "put an app's data back as it was before the import", | |
| 210 | 211 | "repo.pinned": "pinned a repository", |
| 211 | 212 | "repo.unpinned": "unpinned a repository", |
| 212 | 213 | "board.pins": "changed their pinned panels", |
src/http/api/data.ts +32 −1
| @@ -16,6 +16,7 @@ import { | ||
| 16 | 16 | tableList, |
| 17 | 17 | } from "../../data/repo_scope.ts"; |
| 18 | 18 | import { importAppData, importDataMessage } from "../../actions/migrate.ts"; |
| 19 | import { dataMoveState, moveData, moveMessage, undoMove } from "../../actions/migrate_data.ts"; | |
| 19 | 20 | import { loadRepo } from "../api.ts"; |
| 20 | 21 | import type { Ctx, Handler } from "../context.ts"; |
| 21 | 22 | import { QUERY_TIMEOUT_MS } from "../../data/query.ts"; |
| @@ -64,7 +65,8 @@ export function dataRoutes(router: Router<Handler>): void { | ||
| 64 | 65 | return result ? reply(result) : notFound(); |
| 65 | 66 | }); |
| 66 | 67 | |
| 67 | // A dump in, loaded and reported per table on both sides (#11697). Body: { dump, env?, file?, replace? }. | |
| 68 | // A dump in, loaded and reported per table on both sides (#11697). Body: { dump, env?, file?, | |
| 69 | // replace? }. | |
| 68 | 70 | router.add("POST", "/api/v1/repos/:owner/:name/data-import", async (ctx) => { |
| 69 | 71 | const denied = requireScope(ctx, "admin"); |
| 70 | 72 | if (denied) return denied; |
| @@ -86,6 +88,35 @@ export function dataRoutes(router: Router<Handler>): void { | ||
| 86 | 88 | return json(done.value, 200, PRIVATE); |
| 87 | 89 | }); |
| 88 | 90 | |
| 91 | // A platform app's data (#647): where it stands (the command, the drop's dumps, the last receipt) | |
| 92 | // … | |
| 93 | router.add("GET", "/api/v1/repos/:owner/:name/data-move", (ctx) => { | |
| 94 | const repo = loadRepo(ctx); | |
| 95 | if (!repo) return notFound(); | |
| 96 | const state = dataMoveState(ctx.app, ctx.auth.user, repo); | |
| 97 | return state.ok ? json(state.value, 200, PRIVATE) : notFound(); | |
| 98 | }); | |
| 99 | ||
| 100 | // … and a move: { dump } checks, { dump, apply: true } imports, { undo: true } takes it back. | |
| 101 | router.add("POST", "/api/v1/repos/:owner/:name/data-move", async (ctx) => { | |
| 102 | const denied = requireScope(ctx, "admin"); | |
| 103 | if (denied) return denied; | |
| 104 | const repo = loadRepo(ctx); | |
| 105 | if (!repo) return notFound(); | |
| 106 | const input = await body(ctx.request); | |
| 107 | const done = input["undo"] === true | |
| 108 | ? undoMove(ctx.app, ctx.auth.user, repo) | |
| 109 | : typeof input["dump"] === "string" | |
| 110 | ? moveData(ctx.app, ctx.auth.user, repo, { dump: input["dump"], apply: input["apply"] === true }) | |
| 111 | : undefined; | |
| 112 | if (done === undefined) return apiError("invalid_dump", 422, "dump is the name of a file in the app's drop"); | |
| 113 | if (done.ok) return json(done.value, 200, PRIVATE); | |
| 114 | if (done.error === "not_found") return notFound(); | |
| 115 | const code = typeof done.error === "string" ? done.error : done.error.error; | |
| 116 | const conflict = ["exists", "mismatch", "cut_over", "nothing_to_undo", "snapshot_gone", "store_gone"].includes(code); | |
| 117 | return apiError(code, conflict ? 409 : code === "too_large" || code === "too_large_dump" ? 413 : code === "missing" ? 404 : code === "unauthorized" ? 401 : 422, moveMessage(done.error)); | |
| 118 | }); | |
| 119 | ||
| 89 | 120 | router.add("POST", "/api/v1/repos/:owner/:name/data/:env/:file/query", async (ctx) => { |
| 90 | 121 | const denied = requireScope(ctx, "ro"); |
| 91 | 122 | if (denied) return denied; |
src/http/mcp/tools/index.ts +1
| @@ -105,6 +105,7 @@ export { tool as make_it_yours } from "./make_it_yours.ts"; | ||
| 105 | 105 | export { tool as map } from "./map.ts"; |
| 106 | 106 | export { tool as merge } from "./merge.ts"; |
| 107 | 107 | export { tool as migrate } from "./migrate.ts"; |
| 108 | export { tool as migrate_data } from "./migrate_data.ts"; | |
| 108 | 109 | export { tool as milestones } from "./milestones.ts"; |
| 109 | 110 | export { tool as mint_invite } from "./mint_invite.ts"; |
| 110 | 111 | export { tool as mirror_now } from "./mirror_now.ts"; |
src/http/mcp/tools/migrate_data.ts added +41
| @@ -0,0 +1,41 @@ | ||
| 1 | import { dataMoveState, moveData, moveMessage, undoMove } from "../../../actions/migrate_data.ts"; | |
| 2 | import { scopeAllows } from "../../../core/tokens.ts"; | |
| 3 | import { bad, good, isAnswer, repoArg, str, type Tool } from "../shared.ts"; | |
| 4 | import { REPO_ARG } from "../shared.ts"; | |
| 5 | ||
| 6 | export const tool: Tool = { | |
| 7 | name: "migrate_data", | |
| 8 | doc: "`{ repo, dump?, apply?, undo? }`: a platform app's Postgres into the app's own SQLite store (#647). The owner makes the dump with the `command` this answers when given only `repo` (`pg_dump --inserts`, run where the old database is reachable; the connection string never reaches the forge) and puts it in the app's drop. With `dump` (its name there) it loads it in a scratch copy and answers the receipt: per table the rows and SHA-256 on the dump's side and on the store's, `counted` from the dump's own text, and what a Postgres dump had that SQLite cannot hold (`skipped`); nothing is changed unless `apply: true`, which first keeps the store it replaces as the snapshot. `undo: true` puts that snapshot back, until a domain has been cut over. A repository admin's; applying and undoing need an admin-scope token", | |
| 9 | description: "Move a platform app's Postgres dump from its drop into its store; dry-run unless `apply`.", | |
| 10 | inputSchema: { | |
| 11 | type: "object", | |
| 12 | properties: { | |
| 13 | repo: REPO_ARG, | |
| 14 | dump: { type: "string" }, | |
| 15 | apply: { type: "boolean" }, | |
| 16 | undo: { type: "boolean" }, | |
| 17 | }, | |
| 18 | required: ["repo"], | |
| 19 | }, | |
| 20 | writes: true, | |
| 21 | readsWhen: (args) => args["undo"] !== true && args["apply"] !== true, | |
| 22 | run(ctx, user, args) { | |
| 23 | const repo = repoArg(ctx, user, args); | |
| 24 | if (isAnswer(repo)) return repo; | |
| 25 | if ((args["undo"] === true || args["apply"] === true) && !scopeAllows(ctx.auth.scope ?? "ro", "admin")) | |
| 26 | return bad("forbidden", "applying or undoing a data move needs an admin-scope token"); | |
| 27 | if (args["undo"] === true) { | |
| 28 | const undone = undoMove(ctx.app, user, repo); | |
| 29 | return undone.ok ? good(undone.value) : bad(String(undone.error), moveMessage(undone.error)); | |
| 30 | } | |
| 31 | const dump = str(args["dump"]); | |
| 32 | if (dump === "") { | |
| 33 | const state = dataMoveState(ctx.app, user, repo); | |
| 34 | return state.ok ? good(state.value) : bad("not_found", "no such app, or you are not its admin"); | |
| 35 | } | |
| 36 | const done = moveData(ctx.app, user, repo, { dump, apply: args["apply"] === true }); | |
| 37 | if (done.ok) return good(done.value); | |
| 38 | const code = typeof done.error === "string" ? done.error : done.error.error; | |
| 39 | return bad(code, moveMessage(done.error)); | |
| 40 | }, | |
| 41 | }; |
src/http/server.ts +3
| @@ -270,6 +270,7 @@ import { | ||
| 270 | 270 | import { adapterPull } from "../web/studio/adapter.ts"; |
| 271 | 271 | import { jobsPage, steerJobPage } from "../web/studio/jobs.ts"; |
| 272 | 272 | import { steerSchedulePage } from "../web/studio/schedules.ts"; |
| 273 | import { dataMoveAct, dataMovePage } from "../web/studio/data_move.ts"; | |
| 273 | 274 | import { dropPageGet, removeDropPage, uploadDropPage } from "../web/studio/drop.ts"; |
| 274 | 275 | import { noteModerate } from "../web/studio/notes.ts"; |
| 275 | 276 | import { tellPage } from "../web/studio/tell.ts"; |
| @@ -1545,6 +1546,8 @@ export function buildRouter(): Router<Handler> { | ||
| 1545 | 1546 | router.add("GET", "/:owner/:name/app/jobs", jobsPage); |
| 1546 | 1547 | router.add("POST", "/:owner/:name/app/jobs/:id/:action", steerJobPage); |
| 1547 | 1548 | router.add("POST", "/:owner/:name/app/schedules/:schedule/:action", steerSchedulePage); |
| 1549 | router.add("GET", "/:owner/:name/data-move", dataMovePage); | |
| 1550 | router.add("POST", "/:owner/:name/data-move", dataMoveAct); | |
| 1548 | 1551 | router.add("GET", "/:owner/:name/drop", dropPageGet); |
| 1549 | 1552 | router.add("POST", "/:owner/:name/drop", uploadDropPage); |
| 1550 | 1553 | router.add("POST", "/:owner/:name/drop/remove", removeDropPage); |
src/spawn/preview-looks.ts +3
| @@ -38,6 +38,8 @@ export const PREVIEW_APP = "hello"; | ||
| 38 | 38 | export const PREVIEW_APP_PAGE = `/${PREVIEW_ADMIN}/${PREVIEW_APP}`; |
| 39 | 39 | /** The seeded app's Drop: the page the sample files are listed on (#645). */ |
| 40 | 40 | export const PREVIEW_DROP_PAGE = `${PREVIEW_APP_PAGE}/drop`; |
| 41 | /** The seeded app's data move (#647): a Postgres dump from its drop, imported, with its receipt. */ | |
| 42 | export const PREVIEW_DATA_MOVE_PAGE = `${PREVIEW_APP_PAGE}/data-move`; | |
| 41 | 43 | |
| 42 | 44 | /** |
| 43 | 45 | * The recently moved repository (#10195): made by the admin as `moved`, then |
| @@ -208,6 +210,7 @@ export const PREVIEW_OWNER_LOOKS: readonly OwnerLook[] = [ | ||
| 208 | 210 | PREVIEW_REPO_HOME, |
| 209 | 211 | PREVIEW_APP_PAGE, |
| 210 | 212 | PREVIEW_DROP_PAGE, |
| 213 | PREVIEW_DATA_MOVE_PAGE, | |
| 211 | 214 | PREVIEW_MOVED_PAGE, |
| 212 | 215 | PREVIEW_MOVE_LOOK, |
| 213 | 216 | PREVIEW_PULL_PAGE, |
src/spawn/preview-seed.ts +15
| @@ -15,6 +15,9 @@ | ||
| 15 | 15 | */ |
| 16 | 16 | |
| 17 | 17 | import { putStream } from "../drop/store.ts"; |
| 18 | import { moveData } from "../actions/migrate_data.ts"; | |
| 19 | import { setMigration } from "../apps/migration_mode.ts"; | |
| 20 | import { dataMovePage } from "../web/studio/data_move.ts"; | |
| 18 | 21 | import { dropPageGet } from "../web/studio/drop.ts"; |
| 19 | 22 | import { existsSync, mkdirSync, writeFileSync } from "node:fs"; |
| 20 | 23 | import { keepNotice, noticeOf } from "../billing/notice.ts"; |
| @@ -64,6 +67,7 @@ import { | ||
| 64 | 67 | PREVIEW_APP, |
| 65 | 68 | PREVIEW_APP_PAGE, |
| 66 | 69 | PREVIEW_DROP_PAGE, |
| 70 | PREVIEW_DATA_MOVE_PAGE, | |
| 67 | 71 | PREVIEW_DISK_ALERT, |
| 68 | 72 | PREVIEW_FIXTURES, |
| 69 | 73 | PREVIEW_FORKS, |
| @@ -889,6 +893,14 @@ async function seedApp(app: App, admin: User): Promise<Result<void, string>> { | ||
| 889 | 893 | const put = await putStream(app.config.dataDir, repo.value.id, name, new Response(text).body); |
| 890 | 894 | if (!put.ok) return err(`the sample app's drop: ${put.error}`); |
| 891 | 895 | } |
| 896 | // Its data move (#647): the app is in migration mode, a Postgres dump sits in the drop and has | |
| 897 | // been imported, so the data line and the receipt have something to show. | |
| 898 | const pg = "-- PostgreSQL database dump\nSET statement_timeout = 0;\nCREATE TABLE public.parts (id integer NOT NULL, name text, torque_nm real);\nINSERT INTO public.parts VALUES (1, 'M6 bolt', 9.5);\nINSERT INTO public.parts VALUES (2, 'M8 bolt', 22);\nINSERT INTO public.parts VALUES (3, 'M10 nut', 45);\nCREATE TABLE public.orders (id integer NOT NULL, part_id integer, qty integer);\nINSERT INTO public.orders VALUES (1, 1, 40), (2, 3, 120);\nALTER TABLE ONLY public.parts ADD CONSTRAINT parts_pkey PRIMARY KEY (id);\n"; | |
| 899 | const dumped = await putStream(app.config.dataDir, repo.value.id, "dump.sql", new Response(pg).body); | |
| 900 | if (!dumped.ok) return err(`the sample app's dump: ${dumped.error}`); | |
| 901 | setMigration(app.db, repo.value.id, "https://platform.example/hello", true); | |
| 902 | const moved = moveData(app, admin, repo.value, { dump: "dump.sql", apply: true }); | |
| 903 | if (!moved.ok) return err(`the sample app's data move: ${JSON.stringify(moved.error)}`); | |
| 892 | 904 | const url = new URL(`https://preview.invalid${PREVIEW_APP_PAGE}`); |
| 893 | 905 | const answer = await studioApp( |
| 894 | 906 | { app, request: new Request(url.href), url, params: {}, auth: { user: admin }, theme: undefined } as Ctx, |
| @@ -896,6 +908,9 @@ async function seedApp(app: App, admin: User): Promise<Result<void, string>> { | ||
| 896 | 908 | record.value, |
| 897 | 909 | ); |
| 898 | 910 | if (answer.status !== 200) return err(`${PREVIEW_APP_PAGE} would answer ${String(answer.status)}, not 200`); |
| 911 | const moveUrl = new URL(`https://preview.invalid${PREVIEW_DATA_MOVE_PAGE}`); | |
| 912 | const move = dataMovePage({ app, request: new Request(moveUrl.href), url: moveUrl, params: { owner: PREVIEW_ADMIN, name: PREVIEW_APP }, auth: { user: admin }, theme: undefined } as unknown as Ctx); | |
| 913 | if (move.status !== 200 || !(await move.text()).includes("parts")) return err(`${PREVIEW_DATA_MOVE_PAGE} does not show the sample receipt`); | |
| 899 | 914 | const dropUrl = new URL(`https://preview.invalid${PREVIEW_DROP_PAGE}`); |
| 900 | 915 | const drop = dropPageGet({ app, request: new Request(dropUrl.href), url: dropUrl, params: { owner: PREVIEW_ADMIN, name: PREVIEW_APP }, auth: { user: admin }, theme: undefined } as unknown as Ctx); |
| 901 | 916 | return drop.status === 200 && (await drop.text()).includes("torque-spec.pdf") ? ok() : err(`${PREVIEW_DROP_PAGE} does not show the sample drop`); |
src/tui/screens/dataimport.ts +1 −1
| @@ -89,7 +89,7 @@ export function dataImportScreen(f: Session, ctx: Ctx, repo: Repo): Screen { | ||
| 89 | 89 | }, |
| 90 | 90 | key(e) { |
| 91 | 91 | if (state.key(e, rows.length)) return true; |
| 92 | if (e.ctrl || e.alt || (e.key !== "i" && e.key !== "r")) return false; | |
| 92 | if (e.ctrl || e.alt || !["i", "r"].includes(e.key)) return false; | |
| 93 | 93 | if (!mayImport(f, repo)) { |
| 94 | 94 | ctx.toast("Only the app's admin imports a database.", "danger"); |
| 95 | 95 | return true; |
src/tui/screens/datamove.ts added +119
| @@ -0,0 +1,119 @@ | ||
| 1 | /** | |
| 2 | * Moving a platform app's data in the terminal (#647): the same `moveData` and `undoMove` as the | |
| 3 | * page `…/data-move`, the API route and the MCP `migrate_data` tool, an admin's only. `m` asks for | |
| 4 | * the name of a `.sql` file in the app's drop and checks it (nothing is changed), `a` imports it, | |
| 5 | * keeping the store it replaces, and `u` puts that store back after a y/n, until a domain has been | |
| 6 | * cut over. The rows are the receipt's: per table, the dump's rows and checksum, the store's, and | |
| 7 | * whether they match. A running app still holds the old file: restart it after an import or undo. | |
| 8 | */ | |
| 9 | ||
| 10 | import { dataMoveState, moveData, moveMessage, undoMove, type MoveReceipt } from "../../actions/migrate_data.ts"; | |
| 11 | import { lostKeys, skippedWords } from "../../apps/pg_dump.ts"; | |
| 12 | import { canAdmin, type Repo } from "../../core/repos.ts"; | |
| 13 | import type { Ctx, Screen } from "../app/app.ts"; | |
| 14 | import { binding } from "../app/keys.ts"; | |
| 15 | import type { Command } from "../app/palette.ts"; | |
| 16 | import type { Session } from "../session.ts"; | |
| 17 | import { style } from "../term/style.ts"; | |
| 18 | import { truncate } from "../term/width.ts"; | |
| 19 | import { fill, len, split } from "../ui/layout.ts"; | |
| 20 | import { ListState, confirmScreen, drawList, promptScreen, type ListRow } from "./common.ts"; | |
| 21 | import { mayImport } from "./dataimport.ts"; | |
| 22 | ||
| 23 | /** A move's receipt, a row per table. */ | |
| 24 | export function moveRows(receipt: MoveReceipt): ListRow[] { | |
| 25 | return receipt.tables.map((t) => { | |
| 26 | const match = t.dump === t.store && t.rows === t.counted; | |
| 27 | return { | |
| 28 | title: `${t.table}: ${t.rows} rows`, | |
| 29 | tone: match ? "success" : "danger", | |
| 30 | note: `dump ${t.dump.slice(0, 16)} store ${t.store.slice(0, 16)} counted in dump ${t.counted}`, | |
| 31 | ...(match ? {} : { meta: "mismatch" }), | |
| 32 | }; | |
| 33 | }); | |
| 34 | } | |
| 35 | ||
| 36 | /** What was left out of the dump, in a line; keys are named plainly. */ | |
| 37 | export function leftOut(receipt: MoveReceipt): string { | |
| 38 | if (receipt.skipped.length === 0) return ""; | |
| 39 | return `; left out: ${skippedWords(receipt.skipped)}${lostKeys(receipt.skipped) ? " (keys are not kept: the tables load without them)" : ""}`; | |
| 40 | } | |
| 41 | ||
| 42 | export function dataMoveScreen(f: Session, ctx: Ctx, repo: Repo): Screen { | |
| 43 | const state = new ListState(); | |
| 44 | let rows: ListRow[] = []; | |
| 45 | let note = "m checks a dump in the app's drop, a imports it, u undoes it. Restart the app after an import or undo."; | |
| 46 | const move = (apply: boolean): Screen => | |
| 47 | promptScreen(ctx, screen, { | |
| 48 | title: apply ? "Import from the drop" : "Check a dump in the drop", | |
| 49 | label: "file in the app's drop, like dump.sql", | |
| 50 | placeholder: "dump.sql", | |
| 51 | onSubmit: (name) => { | |
| 52 | if (name === "") return "Name a file."; | |
| 53 | const done = moveData(f.app, f.user, repo, { dump: name, apply }); | |
| 54 | if (!done.ok) return moveMessage(done.error); | |
| 55 | rows = moveRows(done.value); | |
| 56 | note = `${apply ? "Imported" : "Checked, nothing changed:"} ${done.value.tables.length} tables${leftOut(done.value)}.`; | |
| 57 | ctx.invalidate(); | |
| 58 | return undefined; | |
| 59 | }, | |
| 60 | }); | |
| 61 | const undo = (): Screen => | |
| 62 | confirmScreen(ctx, screen, { | |
| 63 | title: "Undo the import?", | |
| 64 | body: ["The store from before the import is put back.", "Writes the app made since the import are not in it."], | |
| 65 | actions: [{ label: "Undo", variant: "danger" }, { label: "Cancel" }], | |
| 66 | yesNo: true, | |
| 67 | selected: 1, | |
| 68 | onChoose: (i) => { | |
| 69 | if (i !== 0) return; | |
| 70 | const undone = undoMove(f.app, f.user, repo); | |
| 71 | if (!undone.ok) ctx.toast(moveMessage(undone.error), "danger"); | |
| 72 | else { | |
| 73 | rows = []; | |
| 74 | note = `${undone.value.file} is back as it was; writes since the import are not in it. The imported store is kept as ${undone.value.setAside}. Restart the app.`; | |
| 75 | ctx.invalidate(); | |
| 76 | } | |
| 77 | }, | |
| 78 | }); | |
| 79 | const screen: Screen = { | |
| 80 | title: `${repo.ownerName}/${repo.name} — Move the data`, | |
| 81 | keys: [binding(["m"], "check a drop dump"), binding(["a"], "import a drop dump"), binding(["u"], "undo the import"), binding(["j", "k"], "move cursor")], | |
| 82 | render(area, s) { | |
| 83 | const c = ctx.theme.c; | |
| 84 | const [head, list] = split(area, "v", [len(1), fill(1)]); | |
| 85 | s.text(head!.x, head!.y, truncate(mayImport(f, repo) ? note : "Only the app's admin moves the data.", head!.w), style(c.mutedFg, c.bg), head!.x + head!.w); | |
| 86 | drawList(s, list!, c, state, rows); | |
| 87 | }, | |
| 88 | key(e) { | |
| 89 | if (state.key(e, rows.length)) return true; | |
| 90 | if (e.ctrl || e.alt || !["m", "a", "u"].includes(e.key)) return false; | |
| 91 | if (!mayImport(f, repo)) { | |
| 92 | ctx.toast("Only the app's admin moves the data.", "danger"); | |
| 93 | return true; | |
| 94 | } | |
| 95 | if (e.key === "u") { | |
| 96 | const can = dataMoveState(f.app, f.user, repo); | |
| 97 | if (!can.ok || !can.value.canUndo) ctx.toast(moveMessage("nothing_to_undo"), "danger"); | |
| 98 | else ctx.push(undo()); | |
| 99 | return true; | |
| 100 | } | |
| 101 | ctx.push(move(e.key === "a")); | |
| 102 | return true; | |
| 103 | }, | |
| 104 | }; | |
| 105 | return screen; | |
| 106 | } | |
| 107 | ||
| 108 | /** The palette's command: an admin of a repository that is an app. */ | |
| 109 | export function dataMoveCommands(f: Session, ctx: Ctx, repo: Repo | undefined): Command[] { | |
| 110 | if (!mayImport(f, repo) || f.user === undefined || !canAdmin(f.app.db, repo, f.user)) return []; | |
| 111 | return [ | |
| 112 | { | |
| 113 | id: "data-move", | |
| 114 | title: `${repo.ownerName}/${repo.name} — Move the data`, | |
| 115 | hint: "a platform app's Postgres dump from the drop, receipt and undo", | |
| 116 | run: () => ctx.push(dataMoveScreen(f, ctx, repo)), | |
| 117 | }, | |
| 118 | ]; | |
| 119 | } |
src/tui/screens/workspace.ts +2
| @@ -193,6 +193,7 @@ import { deskCommands } from "./desk.ts"; | ||
| 193 | 193 | import { landCommands } from "./landasis.ts"; |
| 194 | 194 | import { shiftCommands } from "./subtitle_shift.ts"; |
| 195 | 195 | import { dataImportCommands } from "./dataimport.ts"; |
| 196 | import { dataMoveCommands } from "./datamove.ts"; | |
| 196 | 197 | import { jobsCommands } from "./jobs.ts"; |
| 197 | 198 | import { agentsMdCommands } from "./agents_md.ts"; |
| 198 | 199 | import { queueCommands } from "./queuegrowth.ts"; |
| @@ -2141,6 +2142,7 @@ export function workspaceScreen( | ||
| 2141 | 2142 | ...landCommands(f, ctx, repo), |
| 2142 | 2143 | ...shiftCommands(f, ctx, repo, () => screen), |
| 2143 | 2144 | ...dataImportCommands(f, ctx, repo), |
| 2145 | ...dataMoveCommands(f, ctx, repo), | |
| 2144 | 2146 | ...jobsCommands(f, ctx, repo), |
| 2145 | 2147 | ...agentsMdCommands(f, ctx, repo), |
| 2146 | 2148 | ...chaptersCommands(f, ctx, repo), |
src/web/studio/data_move.ts added +178
| @@ -0,0 +1,178 @@ | ||
| 1 | /** | |
| 2 | * Moving a platform app's data, as a page (#647): `/:owner/:name/data-move`, reached from the data | |
| 3 | * line of the migration checklist. In order: the one command that makes the dump, the `.sql` | |
| 4 | * files in the app's drop with **Check** (everything but putting it in place) and **Import**, then | |
| 5 | * the receipt — per table rows and a SHA-256 on the dump's side and the store's — and **Undo** | |
| 6 | * while no domain has been cut over. Server-rendered, no script; every button is a same-origin | |
| 7 | * form POST (`readForm`), an admin's only: to anyone else the page is not found. The words and | |
| 8 | * the gate are `actions/migrate_data.ts`'s, the same the API, the MCP tool and the terminal use. | |
| 9 | */ | |
| 10 | ||
| 11 | import { dataMoveState, moveData, moveMessage, undoMove, type MoveReceipt, type DataMoveState } from "../../actions/migrate_data.ts"; | |
| 12 | import { lostKeys, skippedWords } from "../../apps/pg_dump.ts"; | |
| 13 | import { notePhase, takeSessionTime } from "../../core/route_timing.ts"; | |
| 14 | import type { Ctx } from "../../http/context.ts"; | |
| 15 | import { adminForm, loadAdminRepo } from "../admin_form.ts"; | |
| 16 | import { crumbsFor, notFoundPage, pageHead, repoBase, signInFirst } from "../chrome.ts"; | |
| 17 | import { field } from "../form.ts"; | |
| 18 | import { html, type Html } from "../html.ts"; | |
| 19 | import { flashTo, page } from "../layout.ts"; | |
| 20 | import { badge, btn, fileSize } from "../ui.ts"; | |
| 21 | ||
| 22 | const short = (sha: string): string => sha.slice(0, 12); | |
| 23 | ||
| 24 | const KEYS_WORDS = "Primary, unique and foreign keys are not kept: the tables load without them. Add them again in the app's own schema."; | |
| 25 | ||
| 26 | /** | |
| 27 | * The receipt's rows: what the dump says and what the store holds. A row whose store hash was | |
| 28 | * never computed (`store` null) shows no verdict: a match is only printed when someone compared. | |
| 29 | */ | |
| 30 | export function receiptTable( | |
| 31 | rows: readonly { table: string; rows: number; dump: string; store: string | null; counted: number | null }[], | |
| 32 | ): Html { | |
| 33 | const proven = rows.every((r) => r.store !== null); | |
| 34 | return html`<div class="table-scroll"> | |
| 35 | <table class="table" id="receipt" data-dense> | |
| 36 | <caption class="sr-only">Per table, on the dump's side${proven ? " and the store's" : ""}</caption> | |
| 37 | <thead><tr><th scope="col">Table</th><th scope="col" class="num">Rows</th><th scope="col">Dump SHA-256</th>${proven ? html`<th scope="col">Store SHA-256</th><th scope="col"></th>` : null}</tr></thead> | |
| 38 | <tbody> | |
| 39 | ${rows.map((r) => { | |
| 40 | const match = r.store !== null && r.dump === r.store && (r.counted === null || r.rows === r.counted); | |
| 41 | return html`<tr> | |
| 42 | <td class="break">${r.table}</td> | |
| 43 | <td class="num">${r.rows}</td> | |
| 44 | <td class="mono">${short(r.dump)}</td> | |
| 45 | ${proven ? html`<td class="mono">${short(r.store ?? "")}</td><td>${badge(match ? "match" : "differs", { tone: match ? "success" : "danger" })}</td>` : null} | |
| 46 | </tr>`; | |
| 47 | })} | |
| 48 | </tbody> | |
| 49 | </table> | |
| 50 | </div>`; | |
| 51 | } | |
| 52 | ||
| 53 | function skippedNote(skipped: MoveReceipt["skipped"]): Html | null { | |
| 54 | if (skipped.length === 0) return null; | |
| 55 | return html`<p class="muted small">Left out of the Postgres dump, SQLite has no meaning for it: ${skippedWords(skipped)}.${lostKeys(skipped) ? ` ${KEYS_WORDS}` : ""}</p>`; | |
| 56 | } | |
| 57 | ||
| 58 | function receiptCard(receipt: MoveReceipt): Html { | |
| 59 | const total = receipt.tables.reduce((n, t) => n + t.rows, 0); | |
| 60 | return html`<section id="receipt-block"> | |
| 61 | <h2>${receipt.applied ? "Receipt" : "Check"}</h2> | |
| 62 | <p>${receipt.applied ? "Loaded" : "Would load"} ${receipt.tables.length} tables, ${total} rows from <span class="mono break">${receipt.dump}</span> into <span class="mono">${receipt.file}</span>. ${receipt.applied ? "" : "Nothing was changed."}</p> | |
| 63 | ${receiptTable(receipt.tables)} | |
| 64 | ${skippedNote(receipt.skipped)} | |
| 65 | </section>`; | |
| 66 | } | |
| 67 | ||
| 68 | /** | |
| 69 | * What the import recorded, and only that: rows and the dump's checksum, the store's when it was | |
| 70 | * kept. | |
| 71 | */ | |
| 72 | function lastCard(state: DataMoveState): Html | null { | |
| 73 | const last = state.last; | |
| 74 | if (last === undefined) return null; | |
| 75 | const rows = last.tables.map((t) => ({ table: t.table, rows: t.rows, dump: t.sha256, store: t.store ?? null, counted: null })); | |
| 76 | return html`<section id="last-import"> | |
| 77 | <h2>${last.undone ? "Last import, taken back" : "Last import"}</h2> | |
| 78 | <dl class="status-rows"> | |
| 79 | <div class="status-row"><dt>When</dt><dd>${last.at}</dd></div> | |
| 80 | <div class="status-row"><dt>Dump</dt><dd class="mono">${last.file}</dd></div> | |
| 81 | <div class="status-row"><dt>Store before</dt><dd>${last.snapshot === null ? "none" : html`kept as <span class="mono break">${last.snapshot}</span>`}</dd></div> | |
| 82 | </dl> | |
| 83 | ${last.undone ? null : receiptTable(rows)} | |
| 84 | </section>`; | |
| 85 | } | |
| 86 | ||
| 87 | function body(base: string, state: DataMoveState, shown: MoveReceipt | undefined): Html { | |
| 88 | const form = state.files.length === 0 | |
| 89 | ? html`<p class="muted">No <span class="mono">.sql</span> file in the drop yet. Put <span class="mono">dump.sql</span> in the app's <a href="${base}/drop">drop</a> (up to 16 MB).</p>` | |
outdated
Import has no name/value, so it posts do empty and only imports because dataMoveAct treats empty as apply. Set name="do" value="import" and require it. |
||
| 90 | : html`<form method="post" action="${base}/data-move" class="row"> | |
| 91 | <select class="input" name="dump" aria-label="Dump in the drop"> | |
| 92 | ${state.files.map((f) => html`<option value="${f.name}">${f.name} (${fileSize(f.bytes)})</option>`)} | |
| 93 | </select> | |
| 94 | ${state.cutOver | |
| 95 | ? null | |
| 96 | : html`${btn("Check", { name: "do", value: "check", variant: "outline" })}${btn("Import", { name: "do", value: "import", variant: "primary" })}`} | |
| 97 | </form>`; | |
| 98 | return html`${pageHead("Move the data", { sub: "A platform app's Postgres into this app's own store." })} | |
| 99 | <section id="dump"> | |
outdated
Principle 5: this is primary when imported, next to the Import primary. One accent per page. |
||
| 100 | <h2>Dump</h2> | |
| 101 | <p>Run this where the old database is reachable; its connection string stays there.</p> | |
| 102 | <pre class="mono break" id="export-command">${state.command}</pre> | |
| 103 | </section> | |
| 104 | <section id="load"> | |
| 105 | <h2>Import</h2> | |
| 106 | ${state.cutOver ? html`<p class="muted">A domain has been cut over, so the old data is no longer the way back: importing is closed.</p>` : null} | |
| 107 | ${form} | |
| 108 | ${state.cutOver || state.files.length === 0 | |
| 109 | ? null | |
| 110 | : html`<p class="muted small">Check changes nothing. Import keeps the store it replaces for Undo. Restart the app afterwards: it still holds the old file open until it does, and anything it writes in between is lost.</p>`} | |
| 111 | </section> | |
| 112 | ${shown === undefined ? null : receiptCard(shown)} | |
| 113 | ${lastCard(state)} | |
| 114 | <div class="row"> | |
| 115 | ${state.canUndo | |
| 116 | ? html`<form method="post" action="${base}/data-move"><input type="hidden" name="do" value="undo" />${btn("Undo the import", { variant: "outline" })}</form> | |
| 117 | <p class="muted small">Undo puts the earlier store back; writes the app made since the import are not in it (they stay in the set-aside file). Restart the app after.</p>` | |
| 118 | : null} | |
| 119 | ${btn("Back to the checklist", { href: `${base}#app-migration`, variant: "outline" })} | |
| 120 | </div>`; | |
| 121 | } | |
| 122 | ||
| 123 | function draw(ctx: Ctx, shown: MoveReceipt | undefined, status = 200, flash?: { tone: "danger"; message: string }): Response { | |
| 124 | const rendering = performance.now(); | |
| 125 | const repo = loadAdminRepo(ctx); | |
| 126 | const gone = (): Response => { | |
| 127 | const out = notFoundPage(ctx); | |
| 128 | notePhase(ctx.request, "render", performance.now() - rendering); | |
| 129 | return out; | |
| 130 | }; | |
| 131 | if (!repo) return gone(); | |
| 132 | const state = dataMoveState(ctx.app, ctx.auth.user, repo); | |
| 133 | if (!state.ok) return gone(); | |
| 134 | const out = page( | |
| 135 | { ctx, title: `Move the data · ${repo.name}`, crumbs: crumbsFor(repo), body: body(repoBase(repo), state.value, shown), ...(flash ? { flash } : {}) }, | |
| 136 | status, | |
| 137 | ); | |
| 138 | notePhase(ctx.request, "render", performance.now() - rendering); | |
| 139 | return out; | |
| 140 | } | |
| 141 | ||
| 142 | /** GET /:owner/:name/data-move */ | |
| 143 | export function dataMovePage(ctx: Ctx): Response { | |
| 144 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); | |
| 145 | if (ctx.auth.user === undefined) { | |
| 146 | const rendering = performance.now(); | |
| 147 | const out = signInFirst(ctx); | |
| 148 | notePhase(ctx.request, "render", performance.now() - rendering); | |
| 149 | return out; | |
| 150 | } | |
| 151 | return draw(ctx, undefined); | |
| 152 | } | |
| 153 | ||
| 154 | /** POST /:owner/:name/data-move: `do` is check, import or undo. */ | |
| 155 | export async function dataMoveAct(ctx: Ctx): Promise<Response> { | |
| 156 | const gate = await adminForm(ctx); | |
| 157 | if (gate instanceof Response) return gate; | |
| 158 | const { repo, form } = gate; | |
| 159 | const to = `${repoBase(repo)}/data-move`; | |
| 160 | const what = field(form, "do"); | |
| 161 | if (what === "undo") { | |
| 162 | const done = undoMove(ctx.app, ctx.auth.user, repo); | |
| 163 | return flashTo( | |
| 164 | ctx, | |
| 165 | to, | |
| 166 | done.ok | |
| 167 | ? { message: done.value.restored ? `${done.value.file} is back as it was before the import; writes the app made since are not in it. Restart the app. The imported store is kept as ${done.value.setAside}.` : `The imported ${done.value.file} is set aside as ${done.value.setAside}; the app starts empty, as it was.` } | |
| 168 | : { tone: "danger", message: moveMessage(done.error) }, | |
| 169 | ); | |
| 170 | } | |
| 171 | // Only an explicit `do=import` writes; anything else, including nothing, is refused. | |
| 172 | if (what !== "import" && what !== "check") return draw(ctx, undefined, 422, { tone: "danger", message: "do is check, import or undo: nothing was changed." }); | |
| 173 | const apply = what === "import"; | |
| 174 | const done = moveData(ctx.app, ctx.auth.user, repo, { dump: field(form, "dump"), apply }); | |
| 175 | if (!done.ok) return flashTo(ctx, to, { tone: "danger", message: moveMessage(done.error) }); | |
| 176 | if (apply) return flashTo(ctx, to, { message: `Imported ${done.value.tables.length} tables from ${done.value.dump}: every table's rows and checksum match.` }); | |
| 177 | return draw(ctx, done.value); | |
| 178 | } |
src/web/studio/migration.ts +31 −3
| @@ -6,6 +6,7 @@ | ||
| 6 | 6 | * text and goes through `html`, which escapes it. |
| 7 | 7 | */ |
| 8 | 8 | import { migrationView } from "../../actions/migration.ts"; |
| 9 | import { dataMoveState } from "../../actions/migrate_data.ts"; | |
| 9 | 10 | import { progressWords, type Progress } from "../../apps/migration_mode.ts"; |
| 10 | 11 | import type { Repo } from "../../core/repos.ts"; |
| 11 | 12 | import type { Ctx } from "../../http/context.ts"; |
| @@ -13,8 +14,15 @@ import { html, type Html } from "../html.ts"; | ||
| 13 | 14 | import { icon } from "../icons.ts"; |
| 14 | 15 | import { badge, btn, card, row } from "../ui.ts"; |
| 15 | 16 | |
| 17 | /** The data line (#647): where the app's rows stand, and the way to the page that moves them. */ | |
| 18 | export interface DataLine { | |
| 19 | readonly href: string; | |
| 20 | /** Rows and tables of the import still in place, when there is one. */ | |
| 21 | readonly imported: { readonly tables: number; readonly rows: number } | undefined; | |
| 22 | } | |
| 23 | ||
| 16 | 24 | /** The card's markup for a progress; the page and the tests share it. */ |
| 17 | export function migrationCard(origin: string | undefined, progress: Progress, cutOver: string | null): Html { | |
| 25 | export function migrationCard(origin: string | undefined, progress: Progress, cutOver: string | null, data: DataLine | null = null): Html { | |
| 18 | 26 | const areas = new Map<string, Progress["items"][number][]>(); |
| 19 | 27 | for (const item of progress.items) areas.set(item.area, [...(areas.get(item.area) ?? []), item]); |
| 20 | 28 | const lists = [...areas].map( |
| @@ -35,7 +43,17 @@ export function migrationCard(origin: string | undefined, progress: Progress, cu | ||
| 35 | 43 | </div> |
| 36 | 44 | ${origin === undefined ? null : html`<p class="muted small">Parity with ${origin}</p>`} |
| 37 | 45 | ${progress.total === 0 ? html`<p class="muted">Waiting for the crew's checklist.</p>` : lists} |
| 38 | ${progress.atParity && cutOver !== null ? btn("Cut over", { href: cutOver, variant: "primary" }) : null}`, | |
| 46 | ${data === null | |
| 47 | ? null | |
| 48 | : html`<div class="bar" id="app-migration-data"> | |
| 49 | <span><strong>Data</strong> | |
| 50 | <span class="muted small">${data.imported === undefined ? "not moved yet" : `${data.imported.tables} tables, ${data.imported.rows} rows imported and proven`}</span></span> | |
| 51 | <span class="spacer"></span> | |
| 52 | ${data.imported === undefined | |
| 53 | ? btn("Import data", { href: data.href, size: "sm", variant: "primary" }) | |
| 54 | : btn("Receipt", { href: data.href, size: "sm" })} | |
| 55 | </div>`} | |
| 56 | ${progress.atParity && cutOver !== null ? btn("Cut over", { href: cutOver, variant: data !== null && data.imported === undefined ? "outline" : "primary" }) : null}`, | |
| 39 | 57 | { id: "app-migration" }, |
| 40 | 58 | ); |
| 41 | 59 | } |
| @@ -44,5 +62,15 @@ export function migrationCard(origin: string | undefined, progress: Progress, cu | ||
| 44 | 62 | export async function migrationRegion(ctx: Ctx, repo: Repo, owner: boolean): Promise<Html | null> { |
| 45 | 63 | const view = await migrationView(ctx.app, ctx.auth.user, repo); |
| 46 | 64 | if (view === undefined) return null; |
| 47 | return migrationCard(view.origin, view.progress, owner ? `/${repo.ownerName}/${repo.name}/settings/app` : null); | |
| 65 | const base = `/${repo.ownerName}/${repo.name}`; | |
| 66 | const state = owner ? dataMoveState(ctx.app, ctx.auth.user, repo) : undefined; | |
| 67 | const last = state?.ok === true && state.value.last !== undefined && !state.value.last.undone ? state.value.last : undefined; | |
| 68 | const data: DataLine | null = | |
| 69 | state?.ok === true | |
| 70 | ? { | |
| 71 | href: `${base}/data-move`, | |
| 72 | imported: last === undefined ? undefined : { tables: last.tables.length, rows: last.tables.reduce((n, t) => n + t.rows, 0) }, | |
| 73 | } | |
| 74 | : null; | |
| 75 | return migrationCard(view.origin, view.progress, owner ? `${base}/settings/app` : null, data); | |
| 48 | 76 | } |
test/data-move.test.ts added +329
| @@ -0,0 +1,329 @@ | ||
| 1 | /** | |
| 2 | * A platform app's data into the forge app's store (#647): the receipt matches a hand | |
| 3 | * calculation, undo restores the snapshot, a cut-over closes the way back, and every door | |
| 4 | * (page, API, MCP, terminal) gives the same gate and words. | |
| 5 | */ | |
| 6 | import { Database } from "bun:sqlite"; | |
| 7 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 8 | import { existsSync, mkdirSync, readdirSync, rmSync, writeFileSync } from "node:fs"; | |
| 9 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 10 | import { tmpdir } from "node:os"; | |
| 11 | import { join } from "node:path"; | |
| 12 | import { createApp, type App } from "../src/app.ts"; | |
| 13 | import { createRepo } from "../src/actions/repos.ts"; | |
| 14 | import { dataMoveState, moveData, undoMove } from "../src/actions/migrate_data.ts"; | |
| 15 | import { envDataDir } from "../src/apps/data.ts"; | |
| 16 | import { EXPORT_COMMAND, fromPostgres, lostKeys } from "../src/apps/pg_dump.ts"; | |
| 17 | import { createApp as insertApp } from "../src/core/apps.ts"; | |
| 18 | import { addDomain, recordCheck, setOldRecords } from "../src/core/domains.ts"; | |
| 19 | import type { Repo } from "../src/core/repos.ts"; | |
| 20 | import { createUser, type User } from "../src/core/users.ts"; | |
| 21 | import { receiptTable } from "../src/web/studio/data_move.ts"; | |
| 22 | import { moveRows } from "../src/tui/screens/datamove.ts"; | |
| 23 | import { dropDir } from "../src/drop/store.ts"; | |
| 24 | import type { Ctx } from "../src/http/context.ts"; | |
| 25 | import { serve, type Servers } from "../src/http/server.ts"; | |
| 26 | import { TOOLS } from "../src/http/mcp/tools.ts"; | |
| 27 | import { quietLogs } from "./helpers/globals.ts"; | |
| 28 | import { promote } from "./helpers/members.ts"; | |
| 29 | ||
| 30 | quietLogs(); | |
| 31 | ||
| 32 | const PG = `-- | |
| 33 | -- PostgreSQL database dump | |
| 34 | -- | |
| 35 | SET statement_timeout = 0; | |
| 36 | SET client_encoding = 'UTF8'; | |
| 37 | SELECT pg_catalog.set_config('search_path', '', false); | |
| 38 | CREATE TABLE public.items ( | |
| 39 | id integer NOT NULL, | |
| 40 | name character varying(40), | |
| 41 | note text | |
| 42 | ); | |
| 43 | ALTER TABLE public.items OWNER TO shop; | |
| 44 | INSERT INTO public.items (id, name, note) VALUES (1, 'bolt', 'see public.items; ok'); | |
| 45 | INSERT INTO public.items (id, name, note) VALUES (2, 'nut', NULL); | |
| 46 | INSERT INTO public.items (id, name, note) VALUES (3, 'it''s', 'x'); | |
| 47 | ALTER TABLE ONLY public.items ADD CONSTRAINT items_pkey PRIMARY KEY (id); | |
| 48 | SELECT pg_catalog.setval('public.items_id_seq', 3, true); | |
| 49 | `; | |
| 50 | ||
| 51 | /** | |
| 52 | * The receipt's checksum by hand: each row's values as SQLite quote() literals, \x1f between, \n | |
| 53 | * after. | |
| 54 | */ | |
| 55 | function handSha(): string { | |
| 56 | const rows = [ | |
| 57 | ["1", "'bolt'", "'see public.items; ok'"], | |
| 58 | ["2", "'nut'", "NULL"], | |
| 59 | ["3", "'it''s'", "'x'"], | |
| 60 | ]; | |
| 61 | const h = new Bun.CryptoHasher("sha256"); | |
| 62 | for (const r of rows) h.update(`${r.join("\x1f")}\n`); | |
| 63 | return h.digest("hex"); | |
| 64 | } | |
| 65 | ||
| 66 | describe("a Postgres dump read for SQLite", () => { | |
| 67 | test("session settings, ownership, constraints and sequences are left out and counted; data is untouched", () => { | |
| 68 | const read = fromPostgres(PG); | |
| 69 | expect(read.ok && read.postgres).toBe(true); | |
| 70 | if (!read.ok) return; | |
| 71 | expect(read.skipped).toEqual([ | |
| 72 | { kind: "SET", count: 2 }, | |
| 73 | { kind: "set_config", count: 1 }, | |
| 74 | { kind: "OWNER", count: 1 }, | |
| 75 | { kind: "key", count: 1 }, | |
| 76 | { kind: "setval", count: 1 }, | |
| 77 | ]); | |
| 78 | expect(read.sql).toContain("CREATE TABLE items"); | |
| 79 | expect(read.sql).toContain("'see public.items; ok'"); | |
| 80 | expect(read.sql).not.toContain("OWNER"); | |
| 81 | }); | |
| 82 | test("keys are counted as keys, defaults before the general ALTER TABLE, and the words say keys are not kept", () => { | |
| 83 | const read = fromPostgres(`-- PostgreSQL database dump | |
| 84 | CREATE TABLE a (id integer NOT NULL, b integer); | |
| 85 | ALTER TABLE ONLY public.a ALTER COLUMN id SET DEFAULT nextval('public.a_id_seq'::regclass); | |
| 86 | ALTER TABLE ONLY public.a ADD CONSTRAINT a_pkey PRIMARY KEY (id); | |
| 87 | ALTER TABLE ONLY public.a ADD CONSTRAINT a_b_key UNIQUE (b); | |
| 88 | ALTER TABLE ONLY public.a ADD CONSTRAINT a_b_fk FOREIGN KEY (b) REFERENCES public.a(id); | |
| 89 | ALTER TABLE ONLY public.a ADD CONSTRAINT a_chk CHECK (id > 0); | |
| 90 | INSERT INTO public.a VALUES (1, 1); | |
| 91 | `); | |
| 92 | expect(read.ok && read.skipped).toEqual([ | |
| 93 | { kind: "default", count: 1 }, | |
| 94 | { kind: "key", count: 3 }, | |
| 95 | { kind: "constraint", count: 1 }, | |
| 96 | ]); | |
| 97 | expect(read.ok && lostKeys(read.skipped)).toBe(true); | |
| 98 | expect(read.ok && read.sql).not.toContain("PRIMARY KEY"); | |
| 99 | expect(lostKeys([])).toBe(false); | |
| 100 | }); | |
| 101 | test("COPY is refused with the fix; plain SQL goes through as it is", () => { | |
| 102 | expect(fromPostgres("-- PostgreSQL database dump\nCOPY t (a) FROM stdin;\n1\n\\.\n")).toEqual({ ok: false, error: "copy" }); | |
| 103 | expect(fromPostgres("CREATE TABLE t(a);")).toEqual({ ok: true, sql: "CREATE TABLE t(a);", skipped: [], postgres: false }); | |
| 104 | }); | |
| 105 | }); | |
| 106 | ||
| 107 | describe("moving the data", () => { | |
| 108 | let dataDir: string; | |
| 109 | let app: App; | |
| 110 | let servers: Servers; | |
| 111 | let cy: User; | |
| 112 | let dee: User; | |
| 113 | let repo: Repo; | |
| 114 | let appId: string; | |
| 115 | let cookie: string; | |
| 116 | let deeCookie: string; | |
| 117 | ||
| 118 | const prod = () => envDataDir(dataDir, appId, "production"); | |
| 119 | const drop = (name: string, text: string) => { | |
| 120 | mkdirSync(dropDir(dataDir, repo.id), { recursive: true }); | |
| 121 | writeFileSync(join(dropDir(dataDir, repo.id), name), text); | |
| 122 | }; | |
| 123 | const signin = async (username: string) => { | |
| 124 | const r = await fetch(`${servers.url}/signin`, { method: "POST", body: new URLSearchParams({ username, password: "a-good-passphrase" }), redirect: "manual" }); | |
| 125 | return `${(r.headers.get("set-cookie") ?? "").split(";")[0]!}; forge_mode=dev`; | |
| 126 | }; | |
| 127 | const items = (file = "app.db") => { | |
| 128 | const db = new Database(join(prod(), file), { readonly: true }); | |
| 129 | try { | |
| 130 | return db.query("SELECT id FROM items ORDER BY id").values().map((r) => r[0]); | |
| 131 | } finally { | |
| 132 | db.close(); | |
| 133 | } | |
| 134 | }; | |
| 135 | ||
| 136 | beforeAll(async () => { | |
| 137 | dataDir = await mkdtemp(join(tmpdir(), "forge-data-move-")); | |
| 138 | app = createApp({ dataDir, port: 0, host: "forge.test" }); | |
| 139 | servers = await serve(app); | |
| 140 | const make = async (username: string, adminIfFirst = false) => { | |
| 141 | const made = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst }); | |
| 142 | if (!made.ok) throw new Error(made.error); | |
| 143 | return made.value; | |
| 144 | }; | |
| 145 | await make("ada", true); | |
| 146 | cy = await make("cy"); | |
| 147 | promote(app.db, cy); | |
| 148 | dee = await make("dee"); | |
| 149 | const r = await createRepo(app, cy, { name: "shop", visibility: "public", description: "" }); | |
| 150 | if (!r.ok) throw new Error(String(r.error)); | |
| 151 | repo = r.value; | |
| 152 | const row = insertApp(app.db, { repoId: repo.id, slug: "shop", access: "public" }); | |
| 153 | if (!row.ok) throw new Error(row.error); | |
| 154 | appId = row.value.id; | |
| 155 | cookie = await signin("cy"); | |
| 156 | deeCookie = await signin("dee"); | |
| 157 | drop("dump.sql", PG); | |
| 158 | }, 60_000); | |
| 159 | ||
| 160 | afterAll(async () => { | |
| 161 | servers?.stop(); | |
| 162 | app.db.close(); | |
| 163 | await rm(dataDir, { recursive: true, force: true }); | |
| 164 | }); | |
| 165 | ||
| 166 | test("the first state names the command and the dump in the drop", () => { | |
| 167 | const state = dataMoveState(app, cy, repo); | |
| 168 | expect(state.ok && state.value.command).toBe(EXPORT_COMMAND); | |
| 169 | expect(state.ok && state.value.files.map((f) => f.name)).toEqual(["dump.sql"]); | |
| 170 | expect(state.ok && state.value.last).toBeUndefined(); | |
| 171 | expect(dataMoveState(app, dee, repo)).toEqual({ ok: false, error: "not_found" }); | |
| 172 | }); | |
| 173 | ||
| 174 | test("a check gives the real receipt and changes nothing", () => { | |
| 175 | const done = moveData(app, cy, repo, { dump: "dump.sql" }); | |
| 176 | expect(done.ok).toBe(true); | |
| 177 | if (!done.ok) return; | |
| 178 | expect(done.value.applied).toBe(false); | |
| 179 | expect(done.value.tables).toEqual([{ table: "items", rows: 3, dump: handSha(), store: handSha(), counted: 3 }]); | |
| 180 | expect(done.value.skipped.length).toBeGreaterThan(0); | |
| 181 | expect(existsSync(join(prod(), "app.db"))).toBe(false); | |
| 182 | expect(readdirSync(prod()).filter((n) => n.includes("importing"))).toEqual([]); | |
| 183 | expect(dataMoveState(app, cy, repo)).toMatchObject({ ok: true, value: { last: undefined } }); | |
| 184 | }); | |
| 185 | ||
| 186 | test("a reader's and a stranger's move is not found and writes nothing", () => { | |
| 187 | expect(moveData(app, dee, repo, { dump: "dump.sql", apply: true })).toEqual({ ok: false, error: "not_found" }); | |
| 188 | expect(moveData(app, undefined, repo, { dump: "dump.sql", apply: true })).toEqual({ ok: false, error: "unauthorized" }); | |
| 189 | expect(existsSync(join(prod(), "app.db"))).toBe(false); | |
| 190 | }); | |
| 191 | ||
| 192 | test("names are the drop's: a path or a missing file is refused", () => { | |
| 193 | expect(moveData(app, cy, repo, { dump: "../app.db" })).toEqual({ ok: false, error: "invalid_dump" }); | |
| 194 | expect(moveData(app, cy, repo, { dump: "nope.sql" })).toEqual({ ok: false, error: "missing" }); | |
| 195 | }); | |
| 196 | ||
| 197 | test("with no store before, apply loads it, the receipt matches, and undo leaves the app empty", () => { | |
| 198 | const done = moveData(app, cy, repo, { dump: "dump.sql", apply: true }); | |
| 199 | expect(done.ok && done.value.tables[0]).toEqual({ table: "items", rows: 3, dump: handSha(), store: handSha(), counted: 3 }); | |
| 200 | expect(done.ok && done.value.snapshot).toBeNull(); | |
| 201 | expect(items()).toEqual([1, 2, 3]); | |
| 202 | const state = dataMoveState(app, cy, repo); | |
| 203 | expect(state.ok && state.value.canUndo).toBe(true); | |
| 204 | expect(state.ok && state.value.last?.tables).toEqual([{ table: "items", rows: 3, sha256: handSha(), store: handSha() }]); | |
| 205 | const undone = undoMove(app, cy, repo); | |
| 206 | expect(undone.ok && undone.value.restored).toBe(false); | |
| 207 | expect(existsSync(join(prod(), "app.db"))).toBe(false); | |
| 208 | expect(readdirSync(prod()).some((n) => n.startsWith("app.db.undone-import-"))).toBe(true); | |
| 209 | expect(undoMove(app, cy, repo)).toEqual({ ok: false, error: "nothing_to_undo" }); | |
| 210 | }); | |
| 211 | ||
| 212 | test("with a store before, apply keeps it as the snapshot and undo restores exactly it", () => { | |
| 213 | const old = new Database(join(prod(), "app.db"), { create: true }); | |
| 214 | old.exec("CREATE TABLE items(id INTEGER, name TEXT); INSERT INTO items VALUES (10,'old'),(11,'older');"); | |
| 215 | old.close(); | |
| 216 | const done = moveData(app, cy, repo, { dump: "dump.sql", apply: true }); | |
| 217 | expect(done.ok && done.value.snapshot).toMatch(/^app\.db\.before-import-\d+$/); | |
| 218 | expect(items()).toEqual([1, 2, 3]); | |
| 219 | const undone = undoMove(app, cy, repo); | |
| 220 | expect(undone.ok && undone.value.restored).toBe(true); | |
| 221 | expect(items()).toEqual([10, 11]); | |
| 222 | expect(dataMoveState(app, cy, repo)).toMatchObject({ ok: true, value: { canUndo: false, last: { undone: true } } }); | |
| 223 | }); | |
| 224 | ||
| 225 | test("a POST without do, or with an unknown do, changes nothing; the buttons carry their do", async () => { | |
| 226 | const before = readdirSync(prod()).sort(); | |
| 227 | for (const form of [{ dump: "dump.sql" }, { dump: "dump.sql", do: "" }, { dump: "dump.sql", do: "drop" }]) { | |
| 228 | const r = await fetch(`${servers.url}/cy/shop/data-move`, { method: "POST", headers: { cookie }, body: new URLSearchParams(form) }); | |
| 229 | expect(r.status).toBe(422); | |
| 230 | expect(await r.text()).toContain("nothing was changed"); | |
| 231 | } | |
| 232 | expect(readdirSync(prod()).sort()).toEqual(before); | |
| 233 | const text = await (await fetch(`${servers.url}/cy/shop/data-move`, { headers: { cookie } })).text(); | |
| 234 | expect(text).toMatch(/name="do"\s+value="check"/); | |
| 235 | }); | |
| 236 | ||
| 237 | test("a cut-over closes the way back: no undo, no new import", () => { | |
| 238 | expect(moveData(app, cy, repo, { dump: "dump.sql", apply: true }).ok).toBe(true); | |
| 239 | const added = addDomain(app.db, { host: "shop.example.org", appId }, { forgeHost: "forge.test" }); | |
| 240 | if (!added.ok) throw new Error(added.error); | |
| 241 | recordCheck(app.db, "shop.example.org", { verified: true, said: "ok" }); | |
| 242 | setOldRecords(app.db, "shop.example.org", [{ type: "A", name: "@", value: "192.0.2.1" }]); | |
| 243 | expect(undoMove(app, cy, repo)).toEqual({ ok: false, error: "cut_over" }); | |
| 244 | expect(moveData(app, cy, repo, { dump: "dump.sql", apply: true })).toEqual({ ok: false, error: "cut_over" }); | |
| 245 | expect(moveData(app, cy, repo, { dump: "dump.sql" }).ok).toBe(true); | |
| 246 | expect(items()).toEqual([1, 2, 3]); | |
| 247 | }); | |
| 248 | ||
| 249 | test("the page: an admin sees the command and files, a reader finds nothing, a foreign post is refused", async () => { | |
| 250 | const seen = await fetch(`${servers.url}/cy/shop/data-move`, { headers: { cookie } }); | |
| 251 | expect(seen.status).toBe(200); | |
| 252 | const text = await seen.text(); | |
| 253 | expect(text).toContain("pg_dump --inserts"); | |
| 254 | expect(text).toContain("dump.sql"); | |
| 255 | expect(text).toContain("Import"); | |
| 256 | expect((await fetch(`${servers.url}/cy/shop/data-move`, { headers: { cookie: deeCookie } })).status).toBe(404); | |
| 257 | const foreign = await fetch(`${servers.url}/cy/shop/data-move`, { method: "POST", headers: { cookie, origin: "https://evil.example" }, body: new URLSearchParams({ dump: "dump.sql", do: "import" }) }); | |
| 258 | expect(foreign.status).toBe(403); | |
| 259 | const check = await fetch(`${servers.url}/cy/shop/data-move`, { method: "POST", headers: { cookie }, body: new URLSearchParams({ dump: "dump.sql", do: "check" }) }); | |
| 260 | expect(check.status).toBe(200); | |
| 261 | const receipt = await check.text(); | |
| 262 | expect(receipt).toContain("Would load 1 tables, 3 rows"); | |
| 263 | expect(receipt).toContain(handSha().slice(0, 12)); | |
| 264 | }); | |
| 265 | ||
| 266 | test("the last import's receipt shows only what was recorded: no store hash, no match", () => { | |
| 267 | const state = dataMoveState(app, cy, repo); | |
| 268 | expect(state.ok && state.value.last?.tables[0]?.store).toBe(handSha()); | |
| 269 | const bare = String(receiptTable([{ table: "items", rows: 3, dump: handSha(), store: null, counted: null }])); | |
| 270 | expect(bare).not.toContain("match"); | |
| 271 | expect(bare).not.toContain("Store SHA-256"); | |
| 272 | expect(String(receiptTable([{ table: "items", rows: 3, dump: handSha(), store: handSha(), counted: 3 }]))).toContain("match"); | |
| 273 | }); | |
| 274 | ||
| 275 | test("a file removed after listing is missing, not a throw; unauthorized is 401 on the API", async () => { | |
| 276 | drop("gone.sql", "CREATE TABLE t(a);"); | |
| 277 | expect(dataMoveState(app, cy, repo).ok).toBe(true); | |
| 278 | rmSync(join(dropDir(dataDir, repo.id), "gone.sql")); | |
| 279 | expect(moveData(app, cy, repo, { dump: "gone.sql" })).toEqual({ ok: false, error: "missing" }); | |
| 280 | const r = await fetch(`${servers.url}/api/v1/repos/cy/shop/data-move`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ dump: "dump.sql" }) }); | |
| 281 | expect([401, 403]).toContain(r.status); | |
| 282 | }); | |
| 283 | ||
| 284 | test("the app page's data line points owner at the page", async () => { | |
| 285 | const { migrationCard } = await import("../src/web/studio/migration.ts"); | |
| 286 | const markup = String(migrationCard(undefined, { items: [], done: 0, total: 0, atParity: false }, null, { href: "/cy/shop/data-move", imported: undefined })); | |
| 287 | expect(markup).toContain('href="/cy/shop/data-move"'); | |
| 288 | expect(markup).toContain("Import data"); | |
| 289 | const after = String(migrationCard(undefined, { items: [], done: 0, total: 0, atParity: false }, null, { href: "/cy/shop/data-move", imported: { tables: 2, rows: 9 } })); | |
| 290 | expect(after).toContain("2 tables, 9 rows imported and proven"); | |
| 291 | expect(after).toContain("Receipt"); | |
| 292 | }); | |
| 293 | ||
| 294 | test("the API: state, check, and an admin scope for the move", async () => { | |
| 295 | const api = `${servers.url}/api/v1/repos/cy/shop/data-move`; | |
| 296 | const state = await fetch(api, { headers: { cookie } }); | |
| 297 | expect(state.status).toBe(200); | |
| 298 | expect(((await state.json()) as { command: string }).command).toBe(EXPORT_COMMAND); | |
| 299 | expect((await fetch(api, { headers: { cookie: deeCookie } })).status).toBe(404); | |
| 300 | const bad = await fetch(api, { method: "POST", headers: { cookie, "content-type": "application/json" }, body: JSON.stringify({}) }); | |
| 301 | expect(bad.status).toBe(422); | |
| 302 | const missing = await fetch(api, { method: "POST", headers: { cookie, "content-type": "application/json" }, body: JSON.stringify({ dump: "nope.sql" }) }); | |
| 303 | expect(missing.status).toBe(404); | |
| 304 | }); | |
| 305 | ||
| 306 | test("the terminal draws the same receipt: a row per table, success when it matches", () => { | |
| 307 | const done = moveData(app, cy, repo, { dump: "dump.sql" }); | |
| 308 | expect(done.ok && moveRows(done.value)).toEqual([ | |
| 309 | { title: "items: 3 rows", tone: "success", note: `dump ${handSha().slice(0, 16)} store ${handSha().slice(0, 16)} counted in dump 3` }, | |
| 310 | ]); | |
| 311 | }); | |
| 312 | ||
| 313 | test("the MCP tool: state, a dry-run receipt, and an admin scope to apply or undo", async () => { | |
| 314 | const tool = TOOLS.find((t) => t.name === "migrate_data")!; | |
| 315 | const as = (user: User, scope: string, args: Record<string, unknown>) => | |
| 316 | tool.run({ app, url: new URL("http://forge.test/mcp"), auth: { scope } } as unknown as Ctx, user, args); | |
| 317 | expect(tool.readsWhen?.({ repo: "cy/shop", dump: "dump.sql" })).toBe(true); | |
| 318 | expect(tool.readsWhen?.({ repo: "cy/shop", dump: "dump.sql", apply: true })).toBe(false); | |
| 319 | expect(await as(cy, "ro", { repo: "cy/shop" })).toMatchObject({ ok: true }); | |
| 320 | const plan = await as(cy, "ro", { repo: "cy/shop", dump: "dump.sql" }); | |
| 321 | expect(plan).toMatchObject({ ok: true }); | |
| 322 | expect(JSON.stringify(plan)).toContain(handSha()); | |
| 323 | expect(await as(cy, "rw", { repo: "cy/shop", dump: "dump.sql", apply: true })).toMatchObject({ ok: false, error: "forbidden" }); | |
| 324 | expect(await as(cy, "rw", { repo: "cy/shop", undo: true })).toMatchObject({ ok: false, error: "forbidden" }); | |
| 325 | expect(await as(dee, "admin", { repo: "cy/shop", dump: "dump.sql" })).toMatchObject({ ok: false }); | |
| 326 | // The cut-over is still in place from the test above: the move is refused in words. | |
| 327 | expect(await as(cy, "admin", { repo: "cy/shop", dump: "dump.sql", apply: true })).toMatchObject({ ok: false, error: "cut_over" }); | |
| 328 | }); | |
| 329 | }); |
test/mcp-write-scope.test.ts +1
| @@ -51,6 +51,7 @@ const ADMIN_ARGS: Record<string, Record<string, unknown>> = { | ||
| 51 | 51 | const WRITE_ARGS: Record<string, Record<string, unknown>> = { |
| 52 | 52 | flow: { kind: "drift", action: "update", repo: "alice/game", pull: 1 }, |
| 53 | 53 | repo: { repo: "alice/game", action: "open", chapter: "v1", confirm: true }, |
| 54 | migrate_data: { repo: "alice/game", dump: "dump.sql", apply: true }, | |
| 54 | 55 | system: { kind: "growth", repo: "alice/game", action: "pause", confirm: true }, |
| 55 | 56 | }; |
| 56 | 57 |
test/mcp.test.ts +1
| @@ -286,6 +286,7 @@ describe("the forge over MCP", () => { | ||
| 286 | 286 | "detach", |
| 287 | 287 | "trail", |
| 288 | 288 | "migrate", |
| 289 | "migrate_data", | |
| 289 | 290 | "events", |
| 290 | 291 | "reports", |
| 291 | 292 | "branches", |
test/worker-database-opens.test.ts +1
| @@ -58,6 +58,7 @@ const ONE_JOB: Record<string, string> = { | ||
| 58 | 58 | const REACHED: Record<string, { opens: number; why: string }> = { |
| 59 | 59 | "src/apps/data.ts": { opens: 1, why: "quiesceSqlite folds each app database's log before a copy: called by the serving thread's clone and backup paths, in reach of Workers only through the page layer's imports" }, |
| 60 | 60 | "src/apps/dump_import.ts": { opens: 3, why: "a person's import of a dump, on the serving thread behind a request; no Worker calls it" }, |
| 61 | "src/apps/dump_undo.ts": { opens: 1, why: "an undo folds the live store's log before it is set aside, on the serving thread behind an admin's request; no Worker calls it" }, | |
| 61 | 62 | "src/crew/sentinel.ts": { opens: 1, why: "one scratch database in memory, made once for the process and remembered; the sentinel is a duty of the serving thread" }, |
| 62 | 63 | "src/data/browse.ts": { opens: 1, why: "openReadOnly for a data page, on the serving thread behind a request; no Worker calls it" }, |
| 63 | 64 | "src/kernel/db.ts": { opens: 2, why: "the forge's own database, opened once by createApp on the thread that boots; the Workers that read it open their own handle and keep it (KEEPS)" }, |
new above
offline
store: t.sha256 and counted: t.rows make this receipt always say match. Record the store hash in the event or do not show it.