trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

Platform app move gap: A database import for an app #11713

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/11697-platform-app-move-gap-a-database-import → main 7 commits fork

  • Merged. torque/11697-platform-app-move-gap-a-database-import was merged into main as 61c5ce9.

Review

Sign in to review this change.
18 files changed +1102 −2
docs/MCP.md +1
@@ -49,6 +49,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
4949 | `accounts` | — | Accounts, newest first: role, verified, suspended; `cursor` pages | 264 |
5050 | `agent` | — | Admin: create\|update\|remove an agent | 371 |
5151 | `agents` | — | Agents you may see, as GET /api/v1/agents | 160 |
52 | `app_data_import` | — | Load a SQL dump into an app's database and report per-table counts and checksums; admins only | 339 |
5253 | `app_deploy` | `redeploy`, `stop`, `pull`, `schedules`, `run_schedule`, `pause_schedule`, `resume_schedule`, `jobs`, `cancel_job`, `retry_job` | Redeploy or stop an env, pull, list and steer schedules and jobs | 408 |
5354 | `app_domains` | — | App domains: list, add, check, remove | 201 |
5455 | `app_env` | `list`, `set`, `unset` | An app's environment variables: list names, set or unset by scope; write-only | 393 |
docs/features/platform-app-move.md +38 −1
@@ -40,6 +40,43 @@ symlink is refused.
4040 Secrets: the owner sets each by name
4141 in the app's settings; they are never pasted into an issue, a pull or a run.
4242
43 ### The database import (built, #11697)
44 `POST /api/v1/repos/<owner>/<name>/data-import` with an admin-scope token (or a same-origin
45 admin session) and `{ "dump": "<sql text>", "env": "production", "file": "app.db", "replace": false }`
46 (`importAppData`, `src/actions/migrate.ts`; engine `src/apps/dump_import.ts`). Only a repository's
47 admins; the repository must be an app. The dump is SQL text up to 16 MB (413 past it, counted by
48 the bytes read), the kind `sqlite3 .dump` writes. A Postgres dump is not run as it is:
49 `pg_dump --inserts --no-owner` output that SQLite accepts works, nothing else (#11698).
50
51 What runs is narrower than a statement's first words. `CREATE TABLE` (never `AS`), `CREATE [UNIQUE]
52 INDEX`, `BEGIN`, `COMMIT`, `PRAGMA foreign_keys`, and `INSERT INTO <table> [(columns)] VALUES (...)[, (...)]`
53 whose values are literals: numbers, strings, `X'..'`, NULL, TRUE, FALSE. `SELECT` and `WITH` outside
54 quotes, a function call in a `VALUES` list, and the functions that make a big value from a small
55 statement (`randomblob`, `zeroblob`, `printf`, `replace`, `hex`...) are refused with the statement's
56 number before anything is written. At most 1,000,000 statements, and each database is capped at
57 16,384 pages (64 MiB).
58
59 The dump is loaded into a scratch in-memory database (the dump side) and into a new file beside the
60 target (the store side). The answer lists each table with its row count and a SHA-256 of its rows in
61 rowid order, each value written as SQLite's `quote()` literal, for both sides, and `inserted`: the
62 rows the dump's own text declares, counted from its `VALUES` tuples without SQLite. The file replaces
63 the target only when the two loads agree on the checksum and each table's rows equal `inserted`;
64 otherwise the answer is 409 `mismatch` and nothing changed. An existing file is refused (409
65 `exists`) unless `replace` is true, which keeps it as `<file>.before-import-<ms>` (a name no earlier
66 backup has; a hard link, so the target is never absent) and renames the new file over it. Put the
67 counts and checksums in the migration issue, beside the old node's `SELECT count(*)` per table.
68
69 Doors: the terminal's palette entry "Database import" on a repository's admin (`src/tui/screens/dataimport.ts`;
70 `i` imports, `r` imports and replaces) reads a `.sql` file already in the app's `imports/` directory
71 under the environment's data directory, since a terminal cannot paste 16 MB and must not read a path
72 it is given; MCP `app_data_import` takes the text as the API does. All three are the admin's alone.
73
74 Not done: the load runs on the request, not in a worker. With only literal values and the caps above
75 its time is linear in the dump's bytes (about a second near 16 MB), which is why a worker with a
76 timeout was left out; the running app keeps its old file open until it restarts, so import with the app
77 stopped or restart it after; the proof shows the load agrees with itself and with the dump's row count,
78 not that the old node's dump was complete: compare its row counts too. No web page yet.
79
4380 ## 4. Rehearse on a preview (person)
4481 Preview address, a copy of the data, the payment **sandbox** keys. The owner,
4582 or the shop's owner, places an order and the webhook lands. No cut-over before
@@ -199,7 +236,7 @@ carry a provider id, not a hash, and the new app needs the same client
199236 credentials set by name or those people sign in as new accounts.
200237
201238 ## What the forge lacks (each is an issue)
202 - A database import for an app (`migrate` has no data path): upload a dump, load it, report counts and checksums. (#11697)
239 - ~~A database import for an app.~~ Built: see "The database import" above. (#11697)
203240 - ~~A file and upload copy with a manifest compared on both ends.~~ Done as a script a person runs (#11699); no page or upload from a browser yet.
204241 - ~~A Postgres app has no home here.~~ Decided, not built: no managed Postgres; port to `bun:sqlite` or keep an external one by `egress` (docs/APPS.md). (#11698)
205242 - Per-app encrypted backup with a restore proof (`docs/features/backup-truth.md` covers the forge's own, not an app's). (#11701)
src/actions/migrate.ts +95
@@ -26,6 +26,10 @@
2626 * page can watch it. With a crew asked for, the same job then puts the
2727 * agent on the crew, makes the repository an app, and tells the crew one
2828 * thing: make it run here, adapting rather than rewriting.
29 *
30 * The data is a second step (`importAppData`, below): a dump a person
31 * uploads is loaded into the app's store and reported per table, counts and
32 * checksums, on both sides (`apps/dump_import.ts`).
2933 */
3034
3135 import { NOT_A_MEMBER } from "../core/roles.ts";
@@ -77,6 +81,10 @@ import { tell, toldWords } from "./tell.ts";
7781 import { putFollow, upstreamUrl } from "../core/upstream_follow.ts";
7882 import { pinFor, remoteGitEnv, type Pin } from "../git/remote-env.ts";
7983 import { AT_REPOSITORY_LIMIT, NO_SUCH_AGENT } from "./crew_limit_words.ts";
84 import { envDataDir, isEnvName } from "../apps/data.ts";
85 import { IMPORT_FILE, MAX_STATEMENTS, importDump, type ImportError, type ImportReport } from "../apps/dump_import.ts";
86 import { getAppByRepo } from "../core/apps.ts";
87 import { canAdmin } from "../core/repos.ts";
8088
8189 /** The whole clone — the HEAD probe and the fetch — gets this long. */
8290 export const MIGRATE_TIMEOUT_MS = 120_000;
@@ -900,3 +908,90 @@ Adapt, don't rewrite:
900908 Done means the pull's preview starts, answers its \`health\` path, and does what the original did.`;
901909 }
902910
911
912 export type ImportDataError =
913 | "unauthorized"
914 | "not_found"
915 | "no_app"
916 | "invalid_env"
917 | "invalid_file"
918 | ImportError;
919
920 /**
921 * Load a SQL dump into an app's store (#11697). Admins of the repository
922 * only; anyone else is told `not_found`. `migrate` brings code and this
923 * brings rows: the dump is vetted, loaded into a scratch database and a new
924 * file, compared table by table, and only then put in place. A refusal
925 * leaves the app's data as it was.
926 */
927 export function importAppData(
928 app: App,
929 actor: User | undefined,
930 repo: Repo,
931 input: {
932 readonly dump: string;
933 readonly env?: string;
934 readonly file?: string;
935 readonly replace?: boolean;
936 },
937 ): Result<ImportReport, ImportDataError> {
938 const user = requireUser(actor);
939 if (!user.ok) return user;
940 if (!canAdmin(app.db, repo, user.value)) return err("not_found");
941 const record = getAppByRepo(app.db, repo.id);
942 if (!record) return err("no_app");
943 const env = input.env ?? "production";
944 if (!isEnvName(env)) return err("invalid_env");
945 const file = input.file ?? "app.db";
946 if (!IMPORT_FILE.test(file)) return err("invalid_file");
947 const done = importDump(
948 envDataDir(app.config.dataDir, record.id, env),
949 file,
950 input.dump,
951 input.replace === true,
952 );
953 if (!done.ok) return err(done.error);
954 storageStale(app, repo.ownerId);
955 app.bus.append({
956 kind: "app.data_imported",
957 repoId: repo.id,
958 actorId: user.value.id,
959 payload: {
960 env,
961 file,
962 tables: done.value.tables.map((t) => ({ table: t.table, rows: t.rows, sha256: t.sha256 })),
963 },
964 });
965 return ok(done.value);
966 }
967
968 /** An import's refusal in words a person can act on. */
969 export function importDataMessage(error: ImportDataError): string {
970 if (typeof error === "string")
971 switch (error) {
972 case "unauthorized":
973 return "Sign in first.";
974 case "not_found":
975 return "No such repository, or you are not its admin.";
976 case "no_app":
977 return "This repository is not an app yet: turn it on under Settings first.";
978 case "invalid_env":
979 return "env is production, or pull-<number> for a preview.";
980 case "invalid_file":
981 return "file is a name ending in .db, with letters, digits, dots, dashes and underscores only.";
982 }
983 switch (error.error) {
984 case "empty":
985 return "The dump has no statements.";
986 case "refused":
987 return `Statement ${error.statement} (${error.kind}) is not allowed: a dump may only CREATE TABLE (not AS), CREATE INDEX, INSERT INTO ... VALUES of plain values, BEGIN, COMMIT and PRAGMA foreign_keys; no SELECT, WITH or function that builds a large value.`;
988 case "invalid":
989 return `Statement ${error.statement} failed: ${error.message}`;
990 case "mismatch":
991 return `The store does not match the dump in ${error.tables.filter((t) => !t.match).map((t) => t.table).join(", ")} (rows or checksum, or not the rows the dump's INSERTs count). Nothing was changed.`;
992 case "too_large":
993 return `The dump has ${error.statements} statements; at most ${MAX_STATEMENTS} are loaded.`;
994 case "exists":
995 return `${error.file} already exists in that environment. Send replace to keep it as ${error.file}.before-import-<time> and load the dump in its place.`;
996 }
997 }
src/apps/dump_file.ts added +32
@@ -0,0 +1,32 @@
1 /**
2 * A dump already in an app's data directory, for the terminal (#11697). A
3 * terminal cannot paste 16 MB, and it must not read a path a person types,
4 * so it reads one named file from `imports/` under the environment's data
5 * directory: a plain name ending `.sql`, no separator, at most `DUMP_MAX` bytes.
6 */
7
8 import { existsSync, readFileSync, statSync } from "node:fs";
9 import { join } from "node:path";
10 import { BODY_MAX } from "../http/limits.ts";
11
12 export const DUMP_MAX = BODY_MAX;
13 export const DUMP_FILE = /^[\w][\w.-]*\.sql$/;
14 export const IMPORTS_DIR = "imports";
15
16 export type DumpRead =
17 | { readonly ok: true; readonly sql: string }
18 | { readonly ok: false; readonly error: "invalid_name" | "missing" | "too_large" };
19
20 export function readStagedDump(envDir: string, name: string): DumpRead {
21 if (!DUMP_FILE.test(name) || name.includes("..")) return { ok: false, error: "invalid_name" };
22 const path = join(envDir, IMPORTS_DIR, name);
23 if (!existsSync(path) || !statSync(path).isFile()) return { ok: false, error: "missing" };
24 if (statSync(path).size > DUMP_MAX) return { ok: false, error: "too_large" };
25 return { ok: true, sql: readFileSync(path, "utf8") };
26 }
27
28 export const DUMP_FILE_WORDS = {
29 invalid_name: "A dump file is a name ending .sql: letters, digits, dots, dashes and underscores.",
30 missing: `No such file in the app's ${IMPORTS_DIR}/ directory.`,
31 too_large: "The dump is over 16 MB.",
32 } as const;
src/apps/dump_import.ts added +371
@@ -0,0 +1,371 @@
1 /**
2 * Loading a SQL dump into an app's SQLite store, and proving it arrived.
3 *
4 * A dump is text a person uploaded, so it is never run as given. It is split
5 * into statements (quotes and comments respected) and only a short list of
6 * kinds may run: `CREATE TABLE`, `CREATE [UNIQUE] INDEX`, `INSERT INTO`,
7 * `BEGIN`/`COMMIT`, and `PRAGMA foreign_keys`. `ATTACH`, other `PRAGMA`s,
8 * triggers, views, `DROP`, `UPDATE` and `DELETE` are refused with the
9 * statement's number, before anything is written.
10 *
11 * The kinds are narrower than their first words. An `INSERT` is only
12 * `INSERT INTO <table> [(columns)] VALUES (...)[, (...)]`, its values
13 * literals (numbers, strings, `X'..'` blobs, NULL, TRUE, FALSE): no function
14 * call, no `SELECT`, no `WITH`. A `CREATE TABLE` is never `AS`. `SELECT` and
15 * `WITH` are refused anywhere outside quotes, and so are the functions that
16 * make a big value out of a small statement (`randomblob`, `zeroblob`,
17 * `printf`, `replace`...). What is left costs time and disk in proportion to
18 * the bytes of the dump: it is capped at 16 MB by the route, at
19 * `MAX_STATEMENTS` statements here, and each database at `MAX_PAGES` pages.
20 *
21 * It is loaded twice, on purpose: into a scratch in-memory database (the
22 * dump side) and into a new file beside the target (the store side). Each
23 * side reports, per table, its row count and a SHA-256 of its rows in rowid
24 * order, every value written as SQLite's own `quote()` literal so integers,
25 * reals, text and blobs each hash by type. The two loads agree on the hash,
26 * and each table's row count agrees with the number of `VALUES` tuples
27 * counted in the dump's text, which SQLite did not produce. The file
28 * replaces the target only when all of that holds.
29 *
30 * The input is the dump `sqlite3 .dump` writes, or `pg_dump --inserts
31 * --no-owner` made SQLite-compatible; a Postgres app's home is #11698.
32 */
33
34 import { Database } from "bun:sqlite";
35 import { existsSync, linkSync, mkdirSync, renameSync, rmSync } from "node:fs";
36 import { join } from "node:path";
37
38 /** A table's rows as one side holds them. */
39 export interface TableFingerprint {
40 readonly table: string;
41 readonly rows: number;
42 /** SHA-256, hex, of the rows in rowid order. */
43 readonly sha256: string;
44 }
45
46 export interface TableCompare extends TableFingerprint {
47 /** The same table as the store holds it. */
48 readonly store: TableFingerprint | undefined;
49 /** Rows the dump's own text says were inserted, counted from its `VALUES` lists. */
50 readonly inserted: number;
51 readonly match: boolean;
52 }
53
54 export type ImportError =
55 | { readonly error: "empty" }
56 | { readonly error: "refused"; readonly statement: number; readonly kind: string }
57 | { readonly error: "invalid"; readonly statement: number; readonly message: string }
58 | { readonly error: "mismatch"; readonly tables: readonly TableCompare[] }
59 | { readonly error: "too_large"; readonly statements: number }
60 | { readonly error: "exists"; readonly file: string };
61
62 export interface ImportReport {
63 readonly file: string;
64 readonly statements: number;
65 /** One per table: the dump side, then what the store holds. */
66 readonly tables: readonly TableCompare[];
67 /** The file the replaced database was kept as, when there was one. */
68 readonly kept?: string;
69 }
70
71 /**
72 * Split SQL at top-level semicolons. Single, double and backtick quotes
73 * (doubled to escape), `--` and block comments are understood; comments are
74 * dropped. Empty statements are not returned.
75 */
76 export function splitSql(sql: string): string[] {
77 const out: string[] = [];
78 let cur = "";
79 let i = 0;
80 const n = sql.length;
81 while (i < n) {
82 const c = sql[i]!;
83 if (c === "'" || c === '"' || c === "`") {
84 let j = i + 1;
85 while (j < n) {
86 if (sql[j] === c) {
87 if (sql[j + 1] === c) j += 2;
88 else break;
89 } else j++;
90 }
91 cur += sql.slice(i, j + 1);
92 i = j + 1;
93 } else if (c === "-" && sql[i + 1] === "-") {
94 const end = sql.indexOf("\n", i);
95 i = end === -1 ? n : end;
96 } else if (c === "/" && sql[i + 1] === "*") {
97 const end = sql.indexOf("*/", i + 2);
98 i = end === -1 ? n : end + 2;
99 cur += " ";
100 } else if (c === ";") {
101 if (cur.trim() !== "") out.push(cur.trim());
102 cur = "";
103 i++;
104 } else {
105 cur += c;
106 i++;
107 }
108 }
109 if (cur.trim() !== "") out.push(cur.trim());
110 return out;
111 }
112
113 /** Statements one dump may hold: a 16 MB dump of `INSERT INTO t VALUES(1);` is about 700,000. */
114 export const MAX_STATEMENTS = 1_000_000;
115 /** Pages either database may grow to: 16,384 x 4 KiB = 64 MiB. */
116 export const MAX_PAGES = 16_384;
117
118 const NAME = String.raw`(?:"(?:[^"]|"")+"|` + "`(?:[^`]|``)+`" + String.raw`|\[[^\]]+\]|\w+)`;
119 const ALLOWED: readonly RegExp[] = [
120 /^BEGIN(\s+(DEFERRED\s+|IMMEDIATE\s+|EXCLUSIVE\s+)?TRANSACTION)?$/i,
121 /^(COMMIT|END)(\s+TRANSACTION)?$/i,
122 /^PRAGMA\s+foreign_keys\s*=\s*(ON|OFF|0|1)$/i,
123 /^CREATE\s+TABLE\s/i,
124 /^CREATE\s+(UNIQUE\s+)?INDEX\s/i,
125 ];
126 const INSERT = new RegExp(String.raw`^INSERT\s+INTO\s+(${NAME})\s*(\([^()]*\))?\s*VALUES\s*(\(.*)$`, "is");
127 const CREATE_TABLE_AS = new RegExp(String.raw`^CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?${NAME}\s+AS\b`, "i");
128 /** Words that make a lot out of a little, or read the machine; none belongs in a dump. */
129 const HEAVY = /\b(select|with)\b|\b(randomblob|zeroblob|printf|format|replace|hex|unhex|load_extension|readfile|writefile|generate_series|json_each|json_tree|group_concat)\s*\(/i;
130 const LITERAL_WORD = /^(null|true|false|x)$/i;
131
132 /** `sql` with every quoted piece emptied, so what is left is syntax. */
133 function bare(sql: string): string {
134 let out = "";
135 let i = 0;
136 while (i < sql.length) {
137 const c = sql[i]!;
138 if (c === "'" || c === '"' || c === "`") {
139 let j = i + 1;
140 while (j < sql.length) {
141 if (sql[j] === c) {
142 if (sql[j + 1] === c) j += 2;
143 else break;
144 } else j++;
145 }
146 out += c + c;
147 i = j + 1;
148 } else if (c === "[") {
149 const end = sql.indexOf("]", i);
150 out += "[]";
151 i = end === -1 ? sql.length : end + 1;
152 } else {
153 out += c;
154 i++;
155 }
156 }
157 return out;
158 }
159
160 /** The tuples of a VALUES list: how many, or undefined when it is not only literals. */
161 function valueTuples(tail: string): number | undefined {
162 const syntax = bare(tail);
163 if (/[A-Za-z_]\w*\s*\(/.test(syntax.replace(/(?<![\w.])[xX](?=')/g, ""))) return undefined; // a call
164 for (const w of syntax.match(/(?<![\w.])[A-Za-z_]\w*/g) ?? []) if (!LITERAL_WORD.test(w)) return undefined;
165 if (!/^[\s\d.,()+\-'"xXeE\w]*$/.test(syntax)) return undefined;
166 let depth = 0;
167 let tuples = 0;
168 let expectComma = false;
169 for (const ch of syntax) {
170 if (ch === "(") {
171 if (depth === 0) {
172 if (expectComma) return undefined;
173 tuples++;
174 }
175 if (++depth > 1) return undefined;
176 } else if (ch === ")") {
177 if (--depth < 0) return undefined;
178 if (depth === 0) expectComma = true;
179 } else if (depth === 0 && ch === ",") expectComma = false;
180 else if (depth === 0 && !/\s/.test(ch)) return undefined;
181 }
182 return depth === 0 && tuples > 0 ? tuples : undefined;
183 }
184
185 const tableKey = (raw: string): string => {
186 const t = raw.trim();
187 const inner = /^"(.*)"$/s.test(t) ? t.slice(1, -1).replaceAll('""', '"')
188 : /^`(.*)`$/s.test(t) ? t.slice(1, -1).replaceAll("``", "`")
189 : /^\[(.*)\]$/s.test(t) ? t.slice(1, -1)
190 : t;
191 return inner.toLowerCase();
192 };
193
194 /** Rows each table should hold, counted from the dump's own text: the independent side of the proof. */
195 export function insertedRows(statements: readonly string[]): Map<string, number> {
196 const counts = new Map<string, number>();
197 for (const s of statements) {
198 const m = INSERT.exec(s);
199 if (!m) continue;
200 const n = valueTuples(m[3]!) ?? 0;
201 const key = tableKey(m[1]!);
202 counts.set(key, (counts.get(key) ?? 0) + n);
203 }
204 return counts;
205 }
206
207 const kindOf = (s: string): string => s.split(/\s+/).slice(0, 2).join(" ").slice(0, 40);
208
209 /** The first statement that may not run, as its 1-based number and first words. */
210 export function firstRefused(
211 statements: readonly string[],
212 ): { statement: number; kind: string } | undefined {
213 for (const [i, s] of statements.entries()) {
214 const refuse = { statement: i + 1, kind: kindOf(s) };
215 if (/^INSERT\s/i.test(s)) {
216 const m = INSERT.exec(s);
217 if (!m || valueTuples(m[3]!) === undefined) return refuse;
218 continue;
219 }
220 if (!ALLOWED.some((re) => re.test(s)) || CREATE_TABLE_AS.test(s) || HEAVY.test(bare(s))) return refuse;
221 }
222 return undefined;
223 }
224
225 const ident = (name: string): string => `"${name.replaceAll('"', '""')}"`;
226
227 /** Row count and checksum of every user table, by name. */
228 export function fingerprint(db: Database): TableFingerprint[] {
229 const names = db
230 .query<{ name: string }, []>(
231 "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name",
232 )
233 .all();
234 return names.map(({ name }) => {
235 const columns = db
236 .query<{ name: string }, []>(`PRAGMA table_info(${ident(name)})`)
237 .all();
238 const select = columns.map((c) => `quote(${ident(c.name)})`).join(", ");
239 const hasher = new Bun.CryptoHasher("sha256");
240 let rows = 0;
241 const read = (order: string) => {
242 const query = db.query(`SELECT ${select} FROM ${ident(name)}${order}`).values();
243 for (const row of query) {
244 hasher.update(`${row.join("\x1f")}\n`);
245 rows++;
246 }
247 };
248 try {
249 read(" ORDER BY rowid");
250 } catch {
251 // WITHOUT ROWID: a scan walks the primary key, which is its order.
252 read("");
253 }
254 return { table: name, rows, sha256: hasher.digest("hex") };
255 });
256 }
257
258 function run(
259 db: Database,
260 statements: readonly string[],
261 ): { statement: number; message: string } | undefined {
262 for (const [i, s] of statements.entries()) {
263 try {
264 db.exec(s);
265 } catch (e) {
266 return { statement: i + 1, message: e instanceof Error ? e.message : String(e) };
267 }
268 }
269 return undefined;
270 }
271
272 export const IMPORT_FILE = /^[\w.-]+\.db$/;
273
274 /** The row counts of a store against the dump's: pure, so the proof can be tried on its own. */
275 export function compareTables(
276 dumpSide: readonly TableFingerprint[],
277 storeSide: readonly TableFingerprint[],
278 inserted: ReadonlyMap<string, number>,
279 ): TableCompare[] {
280 const store = new Map(storeSide.map((t) => [t.table, t]));
281 return dumpSide.map((t): TableCompare => {
282 const s = store.get(t.table);
283 const counted = inserted.get(t.table.toLowerCase()) ?? 0;
284 return {
285 ...t,
286 store: s,
287 inserted: counted,
288 match: s?.rows === t.rows && s.sha256 === t.sha256 && s.rows === counted,
289 };
290 });
291 }
292
293 /** A name for the replaced file that no earlier import's backup holds. */
294 function backupName(dir: string, file: string, now: number): string {
295 for (let n = 0; ; n++) {
296 const name = `${file}.before-import-${now}${n === 0 ? "" : `-${n}`}`;
297 if (!existsSync(join(dir, name))) return name;
298 }
299 }
300
301 /**
302 * Load `sql` as `<dir>/<file>`. `dir` is an environment's data directory,
303 * built by the caller from ids, never from input. An existing file is kept
304 * (`<file>.before-import-<ms>`, a name no earlier backup has) only when
305 * `replace` is set, else refused. The target is never absent: the old file
306 * is hard-linked to its backup, then the new one is renamed over it.
307 * `afterLoad` is a seam for tests: it may touch the store before the proof.
308 */
309 export function importDump(
310 dir: string,
311 file: string,
312 sql: string,
313 replace: boolean,
314 afterLoad?: (store: Database) => void,
315 now: number = Date.now(),
316 ): { ok: true; value: ImportReport } | { ok: false; error: ImportError } {
317 const statements = splitSql(sql);
318 if (statements.length === 0) return { ok: false, error: { error: "empty" } };
319 if (statements.length > MAX_STATEMENTS)
320 return { ok: false, error: { error: "too_large", statements: statements.length } };
321 const refused = firstRefused(statements);
322 if (refused) return { ok: false, error: { error: "refused", ...refused } };
323 const target = join(dir, file);
324 if (existsSync(target) && !replace)
325 return { ok: false, error: { error: "exists", file } };
326
327 const scratch = new Database(":memory:");
328 const staged = join(dir, `${file}.importing`);
329 mkdirSync(dir, { recursive: true });
330 for (const p of [staged, `${staged}-wal`, `${staged}-shm`]) rmSync(p, { force: true });
331 const store = new Database(staged, { create: true });
332 try {
333 for (const side of [scratch, store]) {
334 side.exec(`PRAGMA max_page_count = ${MAX_PAGES}`);
335 const failed = run(side, statements);
336 if (failed) return { ok: false, error: { error: "invalid", ...failed } };
337 }
338 afterLoad?.(store);
339 const tables = compareTables(fingerprint(scratch), fingerprint(store), insertedRows(statements));
340 if (tables.some((t) => !t.match) || fingerprint(store).length !== tables.length)
341 return { ok: false, error: { error: "mismatch", tables } };
342 store.exec("PRAGMA wal_checkpoint(TRUNCATE)");
343 store.close();
344 let kept: string | undefined;
345 if (existsSync(target)) {
346 try {
347 const old = new Database(target, { readwrite: true });
348 old.exec("PRAGMA wal_checkpoint(TRUNCATE)");
349 old.close();
350 } catch {
351 // Not SQLite after all: it is kept as it is.
352 }
353 kept = backupName(dir, file, now);
354 linkSync(target, join(dir, kept));
355 }
356 for (const suffix of ["-wal", "-shm"]) rmSync(`${target}${suffix}`, { force: true });
357 renameSync(staged, target);
358 return {
359 ok: true,
360 value: { file, statements: statements.length, tables, ...(kept ? { kept } : {}) },
361 };
362 } finally {
363 scratch.close();
364 try {
365 store.close();
366 } catch {
367 // Already closed.
368 }
369 for (const p of [staged, `${staged}-wal`, `${staged}-shm`]) rmSync(p, { force: true });
370 }
371 }
src/contract/openapi/routes/index.ts +1
@@ -326,6 +326,7 @@ export { route as postApiV1ReposOwnerNameChecksRecheck } from "./postApiV1ReposO
326326 export { route as postApiV1ReposOwnerNameContents } from "./postApiV1ReposOwnerNameContents.ts";
327327 export { route as postApiV1ReposOwnerNameContribute } from "./postApiV1ReposOwnerNameContribute.ts";
328328 export { route as postApiV1ReposOwnerNameDataEnvFileQuery } from "./postApiV1ReposOwnerNameDataEnvFileQuery.ts";
329 export { route as postApiV1ReposOwnerNameDataImport } from "./postApiV1ReposOwnerNameDataImport.ts";
329330 export { route as postApiV1ReposOwnerNameEdits } from "./postApiV1ReposOwnerNameEdits.ts";
330331 export { route as postApiV1ReposOwnerNameEgressGrants } from "./postApiV1ReposOwnerNameEgressGrants.ts";
331332 export { route as postApiV1ReposOwnerNameFollowingConverge } from "./postApiV1ReposOwnerNameFollowingConverge.ts";
src/contract/openapi/routes/postApiV1ReposOwnerNameDataImport.ts added +22
@@ -0,0 +1,22 @@
1 import { REPO, type Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: `${REPO}/data-import`,
6 summary:
7 "Load a SQL dump (CREATE TABLE, CREATE INDEX, INSERT INTO only; up to 16 MB) into the app's SQLite store and report each table's rows and SHA-256 on the dump side and the store side: { file, statements, tables: [{ table, rows, sha256, store, match }], kept? }; nothing changes unless every table matches; needs an admin token",
8 auth: true,
9 role: "admin",
10 body: { dump: "string", env: "string?", file: "string?", replace: "boolean?" },
11 refusals: {
12 invalid_dump: "dump is not text (422)",
13 no_app: "the repository is not an app (422)",
14 invalid_env: "env is production or pull-<number> (422)",
15 invalid_file: "file is a name ending in .db (422)",
16 empty: "the dump has no statements (422)",
17 refused: "a statement is of a kind a dump may not run; its number is named (422)",
18 invalid: "SQLite refused a statement; its number is named (422)",
19 exists: "the file exists; send replace to keep it as <file>.before-import (409)",
20 mismatch: "the store does not match the dump in some table; nothing was changed (409)",
21 },
22 };
src/contract/tags.ts +1
@@ -88,6 +88,7 @@ const ALIAS: Readonly<Record<string, string>> = {
8888 orgs: "forges",
8989 boards: "forges",
9090 data: "apps",
91 "data-import": "apps",
9192 };
9293
9394 /** One line on each tag the document lists. */
src/crew/conversation_confirm.ts +1
@@ -106,6 +106,7 @@ export const ASKING: Readonly<Record<string, Asking>> = {
106106 parent_unlink: { does: "unlink parent-forge sign-in" },
107107 quotas: { does: "change an account's quotas" },
108108 spawn: { does: "spawn a child forge" },
109 app_data_import: { does: "load a SQL dump into an app's database" },
109110 app_deploy: { does: "redeploy or stop an app's production or preview" },
110111 stop: { does: "stop a crew run" },
111112 converge: { does: "have a fork's crew merge the parent's updates" },
src/design/activity.ts +1
@@ -180,6 +180,7 @@ const WORDS: Record<string, string> = {
180180 "repo.crew_outsiders": "changed who may trigger the crew",
181181 "repo.mirrored": "mirrored a repository",
182182 "repo.imported": "imported issues and pulls",
183 "app.data_imported": "loaded a database dump into an app",
183184 "repo.pinned": "pinned a repository",
184185 "repo.unpinned": "unpinned a repository",
185186 "board.pins": "changed their pinned panels",
src/http/api/data.ts +23
@@ -15,6 +15,7 @@ import {
1515 sourceList,
1616 tableList,
1717 } from "../../data/repo_scope.ts";
18 import { importAppData, importDataMessage } from "../../actions/migrate.ts";
1819 import { loadRepo } from "../api.ts";
1920 import type { Ctx, Handler } from "../context.ts";
2021 import { QUERY_TIMEOUT_MS } from "../../data/query.ts";
@@ -63,6 +64,28 @@ export function dataRoutes(router: Router<Handler>): void {
6364 return result ? reply(result) : notFound();
6465 });
6566
67 // A dump in, loaded and reported per table on both sides (#11697). Body: { dump, env?, file?, replace? }.
68 router.add("POST", "/api/v1/repos/:owner/:name/data-import", async (ctx) => {
69 const denied = requireScope(ctx, "admin");
70 if (denied) return denied;
71 const repo = loadRepo(ctx);
72 if (!repo) return notFound();
73 const input = await body(ctx.request);
74 if (typeof input["dump"] !== "string") return apiError("invalid_dump", 422, "dump is the SQL text");
75 const done = importAppData(ctx.app, ctx.auth.user, repo, {
76 dump: input["dump"],
77 ...(typeof input["env"] === "string" ? { env: input["env"] } : {}),
78 ...(typeof input["file"] === "string" ? { file: input["file"] } : {}),
79 replace: input["replace"] === true,
80 });
81 if (!done.ok) {
82 if (done.error === "not_found") return notFound();
83 const code = typeof done.error === "string" ? done.error : done.error.error;
84 return apiError(code, code === "exists" || code === "mismatch" ? 409 : code === "too_large" ? 413 : 422, importDataMessage(done.error));
85 }
86 return json(done.value, 200, PRIVATE);
87 });
88
6689 router.add("POST", "/api/v1/repos/:owner/:name/data/:env/:file/query", async (ctx) => {
6790 const denied = requireScope(ctx, "ro");
6891 if (denied) return denied;
src/http/mcp/tools/app_data_import.ts added +39
@@ -0,0 +1,39 @@
1 import { importAppData, importDataMessage } from "../../../actions/migrate.ts";
2 import { bad, good, isAnswer, repoArg, str, type Tool } from "../shared.ts";
3 import { REPO_ARG } from "../shared.ts";
4
5 export const tool: Tool = {
6 name: "app_data_import",
7 doc: "`{ repo, dump, env?, file?, replace? }` as `POST …/data-import`: load SQL text (`sqlite3 .dump` kind, up to 16 MB: CREATE TABLE, CREATE INDEX, INSERT … VALUES, BEGIN, COMMIT) into an app's SQLite store; a repository admin's. Answers per table the dump's rows and checksum, the store's, and `match`; 409 `exists` unless `replace` (the old file is kept as `<file>.before-import-<time>`), 409 `mismatch` changes nothing",
8 description: "Load a SQL dump into an app's database and report per-table counts and checksums; admins only.",
9 inputSchema: {
10 type: "object",
11 properties: {
12 repo: REPO_ARG,
13 dump: { type: "string" },
14 env: { type: "string" },
15 file: { type: "string" },
16 replace: { type: "boolean" },
17 },
18 required: ["repo", "dump"],
19 },
20 writes: true,
21 async run(ctx, user, args) {
22 const repo = repoArg(ctx, user, args);
23 if (isAnswer(repo)) return repo;
24 const dump = args["dump"];
25 if (typeof dump !== "string") return bad("invalid_dump", "dump is the SQL text");
26 if (Buffer.byteLength(dump) > 16 * 1024 * 1024) return bad("too_large", "the dump is over 16 MB");
27 const env = str(args["env"]);
28 const file = str(args["file"]);
29 const done = importAppData(ctx.app, user, repo, {
30 dump,
31 ...(env !== "" ? { env } : {}),
32 ...(file !== "" ? { file } : {}),
33 replace: args["replace"] === true,
34 });
35 if (done.ok) return good(done.value);
36 const code = typeof done.error === "string" ? done.error : done.error.error;
37 return bad(code, importDataMessage(done.error));
38 },
39 };
src/http/mcp/tools/index.ts +1
@@ -8,6 +8,7 @@ export { tool as account_verify } from "./account_verify.ts";
88 export { tool as accounts } from "./accounts.ts";
99 export { tool as agent } from "./agent.ts";
1010 export { tool as agents } from "./agents.ts";
11 export { tool as app_data_import } from "./app_data_import.ts";
1112 export { tool as app_deploy } from "./app_deploy.ts";
1213 export { tool as app_domains } from "./app_domains.ts";
1314 export { tool as app_env } from "./app_env.ts";
src/tui/screens/dataimport.ts added +115
@@ -0,0 +1,115 @@
1 /**
2 * An app's database import in the terminal (#11697): the same
3 * `importAppData` as `POST …/data-import` and MCP `app_data_import`, an
4 * admin's only. A person who is not the repository's admin is offered
5 * nothing. A terminal cannot paste a dump, so `i` asks for the name of a
6 * `.sql` file already in the app's `imports/` directory (`apps/dump_file.ts`);
7 * `r` does the same and replaces an existing store, keeping the old file as
8 * `<file>.before-import-<time>`. The answer is the API's: per table, the
9 * dump's rows and checksum, the store's, and whether they match.
10 */
11
12 import { importAppData, importDataMessage } from "../../actions/migrate.ts";
13 import { envDataDir } from "../../apps/data.ts";
14 import { DUMP_FILE_WORDS, readStagedDump } from "../../apps/dump_file.ts";
15 import type { ImportReport } from "../../apps/dump_import.ts";
16 import { getAppByRepo } from "../../core/apps.ts";
17 import { canAdmin, type Repo } from "../../core/repos.ts";
18 import type { Ctx, Screen } from "../app/app.ts";
19 import { binding } from "../app/keys.ts";
20 import type { Command } from "../app/palette.ts";
21 import type { Session } from "../session.ts";
22 import { style } from "../term/style.ts";
23 import { truncate } from "../term/width.ts";
24 import { fill, len, split } from "../ui/layout.ts";
25 import { ListState, drawList, promptScreen, type ListRow } from "./common.ts";
26
27 /** Whether this viewer may import: the repository's admin, and it is an app. */
28 export function mayImport(f: Session, repo: Repo | undefined): repo is Repo {
29 return (
30 repo !== undefined &&
31 f.user !== undefined &&
32 getAppByRepo(f.app.db, repo.id) !== undefined &&
33 canAdmin(f.app.db, repo, f.user)
34 );
35 }
36
37 /** What an import's report says, a row per table. */
38 export function reportRows(report: ImportReport): ListRow[] {
39 return report.tables.map((t) => ({
40 title: `${t.table}: ${t.rows} rows`,
41 tone: t.match ? "success" : "danger",
42 note: `dump ${t.sha256.slice(0, 16)} store ${t.store?.sha256.slice(0, 16) ?? "missing"} counted in dump ${t.inserted}`,
43 ...(t.match ? {} : { meta: "mismatch" }),
44 }));
45 }
46
47 /** Run an import from a staged file for this viewer: the report, or the words to say. */
48 export function importStaged(
49 f: Session,
50 repo: Repo,
51 name: string,
52 replace: boolean,
53 ): { ok: true; report: ImportReport } | { ok: false; words: string } {
54 const record = getAppByRepo(f.app.db, repo.id);
55 if (record === undefined || !canAdmin(f.app.db, repo, f.user)) return { ok: false, words: "No such repository, or you are not its admin." };
56 const read = readStagedDump(envDataDir(f.app.config.dataDir, record.id, "production"), name);
57 if (!read.ok) return { ok: false, words: DUMP_FILE_WORDS[read.error] };
58 const done = importAppData(f.app, f.user, repo, { dump: read.sql, replace });
59 return done.ok ? { ok: true, report: done.value } : { ok: false, words: importDataMessage(done.error) };
60 }
61
62 export function dataImportScreen(f: Session, ctx: Ctx, repo: Repo): Screen {
63 const state = new ListState();
64 let rows: ListRow[] = [];
65 let note = "i loads a .sql file from the app's imports/ directory; r loads it over an existing store.";
66 const ask = (replace: boolean): Screen =>
67 promptScreen(ctx, screen, {
68 title: replace ? "Import and replace" : "Import a dump",
69 label: "file in imports/, like dump.sql",
70 placeholder: "dump.sql",
71 onSubmit: (name) => {
72 if (name === "") return "Name a file.";
73 const done = importStaged(f, repo, name, replace);
74 if (!done.ok) return done.words;
75 rows = reportRows(done.report);
76 note = `Loaded ${done.report.statements} statements into ${done.report.file}${done.report.kept ? `; the old file is ${done.report.kept}` : ""}.`;
77 ctx.invalidate();
78 return undefined;
79 },
80 });
81 const screen: Screen = {
82 title: `${repo.ownerName}/${repo.name} — Database import`,
83 keys: [binding(["i"], "import"), binding(["r"], "import, replace"), binding(["j", "k"], "move cursor")],
84 render(area, s) {
85 const c = ctx.theme.c;
86 const [head, list] = split(area, "v", [len(1), fill(1)]);
87 s.text(head!.x, head!.y, truncate(mayImport(f, repo) ? note : "Only the app's admin imports a database.", head!.w), style(c.mutedFg, c.bg), head!.x + head!.w);
88 drawList(s, list!, c, state, rows);
89 },
90 key(e) {
91 if (state.key(e, rows.length)) return true;
92 if (e.ctrl || e.alt || (e.key !== "i" && e.key !== "r")) return false;
93 if (!mayImport(f, repo)) {
94 ctx.toast("Only the app's admin imports a database.", "danger");
95 return true;
96 }
97 ctx.push(ask(e.key === "r"));
98 return true;
99 },
100 };
101 return screen;
102 }
103
104 /** The palette's command: an admin of a repository that is an app. */
105 export function dataImportCommands(f: Session, ctx: Ctx, repo: Repo | undefined): Command[] {
106 if (!mayImport(f, repo)) return [];
107 return [
108 {
109 id: "data-import",
110 title: `${repo.ownerName}/${repo.name} — Database import`,
111 hint: "load a SQL dump into the app's store, counts and checksums on both sides",
112 run: () => ctx.push(dataImportScreen(f, ctx, repo)),
113 },
114 ];
115 }
src/tui/screens/workspace.ts +2
@@ -171,6 +171,7 @@ import { autonomyCommands } from "./autonomy.ts";
171171 import { trafficCommands } from "./trafficview.ts";
172172 import { chaptersCommands } from "./chapters.ts";
173173 import { schedulesCommands } from "./schedules.ts";
174 import { dataImportCommands } from "./dataimport.ts";
174175 import { jobsCommands } from "./jobs.ts";
175176 import { queueCommands } from "./queuegrowth.ts";
176177 import { blockedCommands } from "./blocked.ts";
@@ -1997,6 +1998,7 @@ export function workspaceScreen(
19971998 ...autonomyCommands(f, ctx, repo),
19981999 ...trafficCommands(f, ctx, repo),
19992000 ...schedulesCommands(f, ctx, repo),
2001 ...dataImportCommands(f, ctx, repo),
20002002 ...jobsCommands(f, ctx, repo),
20012003 ...chaptersCommands(f, ctx, repo),
20022004 ...queueCommands(f, ctx, repo),
test/app-dump-import.test.ts added +254
@@ -0,0 +1,254 @@
1 /** A dump in, loaded and proved per table on both sides (#11697). */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { Database } from "bun:sqlite";
4 import { existsSync, readdirSync, mkdirSync, writeFileSync } from "node:fs";
5 import { mkdtemp, rm } from "node:fs/promises";
6 import { tmpdir } from "node:os";
7 import { join } from "node:path";
8 import { createHash } from "node:crypto";
9 import { createApp, type App } from "../src/app.ts";
10 import { importAppData } from "../src/actions/migrate.ts";
11 import { envDataDir } from "../src/apps/data.ts";
12 import { compareTables, firstRefused, importDump, insertedRows, splitSql } from "../src/apps/dump_import.ts";
13 import { createApp as createAppRecord } from "../src/core/apps.ts";
14 import { createRepo, getRepoById, type Repo } from "../src/core/repos.ts";
15 import { createUser, type User } from "../src/core/users.ts";
16 import { serve } from "../src/http/server.ts";
17 import { createToken } from "../src/core/tokens.ts";
18 import { quietLogs } from "./helpers/globals.ts";
19
20 quietLogs();
21
22 const DUMP = `-- a dump; with; semicolons in comments
23 BEGIN TRANSACTION;
24 CREATE TABLE orders (id INTEGER PRIMARY KEY, who TEXT, total REAL, note BLOB);
25 INSERT INTO orders VALUES (1,'ann; "the first"',9.5,NULL);
26 INSERT INTO orders VALUES (2,'it''s /* not a comment */',12,X'00ff');
27 CREATE TABLE empty_t (a TEXT);
28 CREATE UNIQUE INDEX orders_who ON orders(who);
29 COMMIT;`;
30
31 describe("splitting and vetting", () => {
32 test("quotes and comments hide semicolons", () => {
33 expect(splitSql(DUMP)).toHaveLength(7);
34 expect(splitSql("INSERT INTO t VALUES ('a;b'); /* ; */ ")).toEqual(["INSERT INTO t VALUES ('a;b')"]);
35 });
36 test("only the allowed kinds pass, and the refusal names its statement", () => {
37 expect(firstRefused(splitSql(DUMP))).toBeUndefined();
38 expect(firstRefused(splitSql("CREATE TABLE t(a); ATTACH DATABASE '/x' AS y"))).toEqual({ statement: 2, kind: "ATTACH DATABASE" });
39 expect(firstRefused(splitSql("PRAGMA writable_schema=1"))?.statement).toBe(1);
40 expect(firstRefused(splitSql("DROP TABLE t"))?.statement).toBe(1);
41 });
42 });
43
44 describe("what a dump may do (#11697 review)", () => {
45 const refused = (sql: string) => firstRefused(splitSql(sql));
46 test("a recursive CTE, INSERT ... SELECT and CREATE TABLE AS are refused", () => {
47 expect(refused("CREATE TABLE t(x); INSERT INTO t WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM c) SELECT x FROM c")?.statement).toBe(2);
48 expect(refused("INSERT INTO t SELECT randomblob(1000000000)")?.statement).toBe(1);
49 expect(refused("INSERT INTO t VALUES ((SELECT 1))")?.statement).toBe(1);
50 expect(refused("INSERT INTO t VALUES (randomblob(1000000000))")?.statement).toBe(1);
51 expect(refused("INSERT INTO t VALUES (1); INSERT INTO t VALUES (zeroblob(5))")?.statement).toBe(2);
52 expect(refused("CREATE TABLE x AS SELECT 1")?.statement).toBe(1);
53 expect(refused("CREATE TABLE x AS VALUES (1)")?.statement).toBe(1);
54 expect(refused("CREATE TABLE x (a TEXT DEFAULT (hex(randomblob(500000000))))")?.statement).toBe(1);
55 expect(refused("CREATE INDEX i ON t(a) WHERE a IN (SELECT 1)")?.statement).toBe(1);
56 });
57 test("plain dumps still pass, quoted words and column names that look like functions included", () => {
58 expect(refused(`CREATE TABLE "t" (format TEXT, "select" TEXT, price DECIMAL(10,2) DEFAULT (datetime('now')));
59 INSERT INTO "t" VALUES('select * from x; replace(1)', 'with', 1.5e3), ('b', NULL, -2);
60 INSERT INTO "t"(format) VALUES (X'00ff'), (TRUE);`)).toBeUndefined();
61 });
62 test("an unbounded or mixed VALUES list is refused", () => {
63 expect(refused("INSERT INTO t VALUES (1) (2)")?.statement).toBe(1);
64 expect(refused("INSERT INTO t VALUES (1), (2) ON CONFLICT DO NOTHING")?.statement).toBe(1);
65 expect(refused("INSERT INTO t VALUES (1); DELETE FROM t")?.statement).toBe(2);
66 });
67 test("a recursive CTE loaded through importDump writes nothing", () => {
68 const dir = join(tmpdir(), `forge-dump-cte-${process.pid}`);
69 const done = importDump(dir, "x.db", "CREATE TABLE t(x INT); INSERT INTO t WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM c) SELECT x FROM c", false);
70 expect(done).toMatchObject({ ok: false, error: { error: "refused", statement: 2 } });
71 expect(existsSync(join(dir, "x.db"))).toBe(false);
72 });
73 test("rows are counted from the dump's text, per table, apart from SQLite", () => {
74 const counts = insertedRows(splitSql(`CREATE TABLE a(x); INSERT INTO a VALUES (1),(2),('3);(4)'); INSERT INTO "A" VALUES (9); INSERT INTO b(x) VALUES (1)`));
75 expect([...counts]).toEqual([["a", 4], ["b", 1]]);
76 });
77 test("mismatch: a store that lost or gained a row, or changed one, is not accepted and nothing is put in place", () => {
78 const fp = (rows: number, sha: string) => ({ table: "t", rows, sha256: sha });
79 const counted = new Map([["t", 3]]);
80 expect(compareTables([fp(3, "a")], [fp(3, "a")], counted)[0]!.match).toBe(true);
81 expect(compareTables([fp(3, "a")], [fp(3, "b")], counted)[0]!.match).toBe(false);
82 expect(compareTables([fp(3, "a")], [fp(2, "a")], counted)[0]!.match).toBe(false);
83 // both loads agree with each other but not with the dump's own text
84 expect(compareTables([fp(2, "a")], [fp(2, "a")], counted)[0]!.match).toBe(false);
85 expect(compareTables([fp(3, "a")], [], counted)[0]!.match).toBe(false);
86 const dir = join(tmpdir(), `forge-dump-mm-${process.pid}`);
87 const done = importDump(dir, "m.db", "CREATE TABLE t(a); INSERT INTO t VALUES (1),(2);", false, (store) => store.exec("INSERT INTO t VALUES (3)"));
88 expect(done).toMatchObject({ ok: false, error: { error: "mismatch", tables: [{ table: "t", rows: 2, inserted: 2, match: false, store: { rows: 3 } }] } });
89 expect(existsSync(join(dir, "m.db"))).toBe(false);
90 expect(existsSync(join(dir, "m.db.importing"))).toBe(false);
91 const kept = join(dir, "n.db");
92 mkdirSync(dir, { recursive: true });
93 writeFileSync(kept, "old");
94 importDump(dir, "n.db", "CREATE TABLE t(a); INSERT INTO t VALUES (1);", true, (store) => store.exec("DELETE FROM t"));
95 expect(Bun.file(kept).size).toBe(3);
96 });
97 test("a dump of more statements than the ceiling is refused before it runs", () => {
98 const dir = join(tmpdir(), `forge-dump-big-${process.pid}`);
99 const sql = "BEGIN;" + ";".repeat(0) + "INSERT INTO t VALUES (1);".repeat(1_000_001);
100 expect(importDump(dir, "b.db", sql, false)).toMatchObject({ ok: false, error: { error: "too_large" } });
101 });
102 });
103
104 describe("importAppData", () => {
105 let dataDir: string;
106 let app: App;
107 let owner: User;
108 let stranger: User;
109 let repo: Repo;
110 let bare: Repo;
111 const mk = async (name: string, admin = false): Promise<User> => {
112 const made = await createUser(app.db, { origin: "seed", trusted: true, username: name, email: `${name}@forge.local`, password: "a-good-passphrase", adminIfFirst: admin });
113 if (!made.ok) throw new Error(made.error);
114 return made.value;
115 };
116 beforeAll(async () => {
117 dataDir = await mkdtemp(join(tmpdir(), "forge-dump-"));
118 app = createApp({ dataDir, port: 0, host: "forge.test" });
119 await mk("owner1", true);
120 owner = await mk("alice");
121 stranger = await mk("bob");
122 const r = await createRepo(app.db, app.reposRoot, { ownerId: owner.id, name: "shop", visibility: "public" });
123 if (!r.ok) throw new Error(r.error);
124 repo = getRepoById(app.db, r.value.id)!;
125 const rec = createAppRecord(app.db, { repoId: repo.id, slug: "shop", access: "public" });
126 if (!rec.ok) throw new Error(rec.error);
127 const b = await createRepo(app.db, app.reposRoot, { ownerId: owner.id, name: "plain", visibility: "public" });
128 if (!b.ok) throw new Error(b.error);
129 bare = b.value;
130 });
131 afterAll(async () => {
132 app.close();
133 await rm(dataDir, { recursive: true, force: true });
134 });
135 const dirOf = () => envDataDir(dataDir, (app.db.query("SELECT id FROM app").get() as { id: string }).id, "production");
136
137 test("loads, and the counts and checksums match a hand calculation", () => {
138 const done = importAppData(app, owner, repo, { dump: DUMP });
139 expect(done.ok).toBe(true);
140 if (!done.ok) return;
141 const orders = done.value.tables.find((t) => t.table === "orders")!;
142 expect(orders.rows).toBe(2);
143 // rows in rowid order, each value as SQLite's quote() writes it, joined by US, ended by newline
144 const by = createHash("sha256")
145 .update("1\x1f'ann; \"the first\"'\x1f9.5\x1fNULL\n2\x1f'it''s /* not a comment */'\x1f12.0\x1fX'00FF'\n")
146 .digest("hex");
147 expect(orders.sha256).toBe(by);
148 expect(orders.store).toEqual({ table: "orders", rows: 2, sha256: by });
149 expect(orders.inserted).toBe(2);
150 expect(orders.match).toBe(true);
151 expect(done.value.tables.find((t) => t.table === "empty_t")?.rows).toBe(0);
152 const stored = new Database(join(dirOf(), "app.db"), { readonly: true });
153 expect(stored.query("SELECT count(*) AS n FROM orders").get()).toEqual({ n: 2 });
154 stored.close();
155 });
156
157 test("an existing file is refused unless replace is said, then kept", () => {
158 const again = importAppData(app, owner, repo, { dump: DUMP });
159 expect(again).toEqual({ ok: false, error: { error: "exists", file: "app.db" } });
160 const replaced = importAppData(app, owner, repo, { dump: "CREATE TABLE t(a); INSERT INTO t VALUES (1);", replace: true });
161 const kept = replaced.ok ? replaced.value.kept : undefined;
162 expect(kept).toMatch(/^app\.db\.before-import-\d+$/);
163 expect(existsSync(join(dirOf(), kept!))).toBe(true);
164 const d = new Database(join(dirOf(), "app.db"), { readonly: true });
165 expect(d.query("SELECT name FROM sqlite_master").all()).toEqual([{ name: "t" }]);
166 d.close();
167 });
168
169 test("a refused or broken dump changes nothing", () => {
170 const before = Bun.file(join(dirOf(), "app.db")).size;
171 expect(importAppData(app, owner, repo, { dump: "ATTACH DATABASE '/etc/x' AS y", replace: true })).toMatchObject({ error: { error: "refused", statement: 1 } });
172 expect(importAppData(app, owner, repo, { dump: "CREATE TABLE u(a); INSERT INTO nope VALUES (1)", replace: true })).toMatchObject({ error: { error: "invalid", statement: 2 } });
173 expect(importAppData(app, owner, repo, { dump: " -- nothing ", replace: true })).toMatchObject({ error: { error: "empty" } });
174 expect(Bun.file(join(dirOf(), "app.db")).size).toBe(before);
175 expect(existsSync(join(dirOf(), "app.db.importing"))).toBe(false);
176 });
177
178 test("a second replace keeps the first backup too", () => {
179 const backups = () => readdirSync(dirOf()).filter((n) => n.startsWith("app.db.before-import"));
180 const before = backups().length;
181 const again = importAppData(app, owner, repo, { dump: "CREATE TABLE t2(a); INSERT INTO t2 VALUES (7);", replace: true });
182 expect(again.ok).toBe(true);
183 expect(backups().length).toBe(before + 1);
184 // the same millisecond still gets its own name
185 const dir = join(dataDir, "twice");
186 const one = importDump(dir, "a.db", "CREATE TABLE t(a);", false, undefined, 5);
187 const two = importDump(dir, "a.db", "CREATE TABLE u(a);", true, undefined, 5);
188 const three = importDump(dir, "a.db", "CREATE TABLE v(a);", true, undefined, 5);
189 expect([one.ok, two.ok && two.value.kept, three.ok && three.value.kept]).toEqual([true, "a.db.before-import-5", "a.db.before-import-5-1"]);
190 const old = new Database(join(dir, "a.db.before-import-5"), { readonly: true });
191 expect(old.query("SELECT name FROM sqlite_master").all()).toEqual([{ name: "t" }]);
192 old.close();
193 });
194
195 test("who and where are checked at the boundary", () => {
196 expect(importAppData(app, stranger, repo, { dump: DUMP, file: "x.db" })).toEqual({ ok: false, error: "not_found" });
197 expect(importAppData(app, undefined, repo, { dump: DUMP })).toEqual({ ok: false, error: "unauthorized" });
198 expect(importAppData(app, owner, bare, { dump: DUMP })).toEqual({ ok: false, error: "no_app" });
199 expect(importAppData(app, owner, repo, { dump: DUMP, env: "../x" })).toEqual({ ok: false, error: "invalid_env" });
200 expect(importAppData(app, owner, repo, { dump: DUMP, file: "../x.db" })).toEqual({ ok: false, error: "invalid_file" });
201 });
202
203 test("the API door: POST data-import, admin token, 404 for a stranger", async () => {
204 const servers = await serve(app);
205 try {
206 const post = (who: User, payload: unknown) =>
207 fetch(`${servers.url}/api/v1/repos/alice/shop/data-import`, {
208 method: "POST",
209 headers: { authorization: `Bearer ${createToken(app.db, who.id, "t", "admin").token}`, "content-type": "application/json" },
210 body: JSON.stringify(payload),
211 });
212 const ok = await post(owner, { dump: DUMP, replace: true, file: "api.db" });
213 expect(ok.status).toBe(200);
214 const report = (await ok.json()) as { tables: { table: string; rows: number; match: boolean }[] };
215 expect(report.tables.map((t) => [t.table, t.rows, t.match])).toEqual([["empty_t", 0, true], ["orders", 2, true]]);
216 expect((await post(owner, { dump: DUMP, file: "api.db" })).status).toBe(409);
217 expect((await post(owner, { dump: "DROP TABLE t" })).status).toBe(422);
218 expect((await post(stranger, { dump: DUMP })).status).toBe(404);
219 // a recursive CTE is refused at the door, not run
220 const cte = await post(owner, { dump: "CREATE TABLE t(x); INSERT INTO t WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM c) SELECT x FROM c", file: "cte.db" });
221 expect(cte.status).toBe(422);
222 expect(((await cte.json()) as { error: string }).error).toBe("refused");
223 expect(existsSync(join(dirOf(), "cte.db"))).toBe(false);
224 // a token that may only read cannot import, even for its own repository
225 const ro = await fetch(`${servers.url}/api/v1/repos/alice/shop/data-import`, {
226 method: "POST",
227 headers: { authorization: `Bearer ${createToken(app.db, owner.id, "ro", "ro").token}`, "content-type": "application/json" },
228 body: JSON.stringify({ dump: DUMP, file: "ro.db" }),
229 });
230 expect(ro.status).toBeGreaterThanOrEqual(400);
231 expect(ro.status).toBeLessThan(500);
232 expect(existsSync(join(dirOf(), "ro.db"))).toBe(false);
233 // a body past 16 MB is stopped by the bytes read, with no length declared
234 const chunk = new TextEncoder().encode("x".repeat(1024 * 1024));
235 let sent = 0;
236 const stream = new ReadableStream<Uint8Array>({
237 pull(c) {
238 if (sent++ < 18) c.enqueue(chunk);
239 else c.close();
240 },
241 });
242 const huge = await fetch(`${servers.url}/api/v1/repos/alice/shop/data-import`, {
243 method: "POST",
244 headers: { authorization: `Bearer ${createToken(app.db, owner.id, "t", "admin").token}`, "content-type": "application/json" },
245 body: stream,
246 duplex: "half",
247 });
248 expect(huge.status).toBe(413);
249 expect(existsSync(join(dirOf(), "app.db.importing"))).toBe(false);
250 } finally {
251 servers.stop();
252 }
253 });
254 });
test/dataimport-doors.test.ts added +104
@@ -0,0 +1,104 @@
1 /**
2 * The database import through the terminal and MCP doors (#11697): the same
3 * gate as the API (the repository's admin; anyone else is offered nothing and
4 * changes nothing), the same engine, the same words.
5 */
6 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
7 import { mkdirSync, writeFileSync, existsSync } from "node:fs";
8 import { mkdtemp, rm } from "node:fs/promises";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { createApp, type App } from "../src/app.ts";
12 import { createRepo } from "../src/actions/repos.ts";
13 import { envDataDir } from "../src/apps/data.ts";
14 import { readStagedDump } from "../src/apps/dump_file.ts";
15 import { createApp as insertApp } from "../src/core/apps.ts";
16 import type { Repo } from "../src/core/repos.ts";
17 import { createUser, type User } from "../src/core/users.ts";
18 import { TOOLS } from "../src/http/mcp/tools.ts";
19 import { dataImportCommands, importStaged } from "../src/tui/screens/dataimport.ts";
20 import type { Ctx } from "../src/tui/app/app.ts";
21 import type { Session } from "../src/tui/session.ts";
22 import { quietLogs } from "./helpers/globals.ts";
23 import { promote } from "./helpers/members.ts";
24
25 quietLogs();
26
27 let dataDir: string;
28 let app: App;
29 let cy: User; // admin of the repository
30 let dee: User; // a reader
31 let repo: Repo;
32 let appId: string;
33
34 beforeAll(async () => {
35 dataDir = await mkdtemp(join(tmpdir(), "forge-import-doors-"));
36 app = createApp({ dataDir, port: 0, host: "forge.test" });
37 const make = async (username: string, adminIfFirst = false) => {
38 const made = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst });
39 if (!made.ok) throw new Error(made.error);
40 return made.value;
41 };
42 await make("ada", true);
43 cy = await make("cy");
44 promote(app.db, cy);
45 dee = await make("dee");
46 const made = await createRepo(app, cy, { name: "shop", visibility: "public", description: "" });
47 if (!made.ok) throw new Error(String(made.error));
48 repo = made.value;
49 const row = insertApp(app.db, { repoId: repo.id, slug: "shop", access: "public" });
50 if (!row.ok) throw new Error(row.error);
51 appId = row.value.id;
52 const dir = join(envDataDir(dataDir, appId, "production"), "imports");
53 mkdirSync(dir, { recursive: true });
54 writeFileSync(join(dir, "dump.sql"), "CREATE TABLE t(a); INSERT INTO t VALUES (1),(2);");
55 }, 60_000);
56
57 afterAll(async () => {
58 app.db.close();
59 await rm(dataDir, { recursive: true, force: true });
60 });
61
62 const session = (user: User | undefined) => ({ user, db: app.db, app }) as unknown as Session;
63 const ctx = { toast() {}, invalidate() {}, push() {} } as unknown as Ctx;
64
65 describe("terminal", () => {
66 test("only the repository's admin is offered the screen", () => {
67 expect(dataImportCommands(session(cy), ctx, repo).map((c) => c.id)).toEqual(["data-import"]);
68 expect(dataImportCommands(session(dee), ctx, repo)).toEqual([]);
69 expect(dataImportCommands(session(undefined), ctx, repo)).toEqual([]);
70 expect(dataImportCommands(session(cy), ctx, undefined)).toEqual([]);
71 });
72 test("a non-admin's import is refused and writes nothing", () => {
73 const refused = importStaged(session(dee), repo, "dump.sql", false);
74 expect(refused.ok).toBe(false);
75 expect(existsSync(join(envDataDir(dataDir, appId, "production"), "app.db"))).toBe(false);
76 });
77 test("the admin's import loads a staged file and reports per table", () => {
78 const done = importStaged(session(cy), repo, "dump.sql", false);
79 expect(done.ok && done.report.tables.map((t) => [t.table, t.rows, t.inserted, t.match])).toEqual([["t", 2, 2, true]]);
80 const again = importStaged(session(cy), repo, "dump.sql", false);
81 expect(again).toMatchObject({ ok: false });
82 });
83 test("a file name is a plain .sql name that exists", () => {
84 const dir = envDataDir(dataDir, appId, "production");
85 for (const name of ["../app.db", "a/b.sql", "dump.txt", ".sql", ""]) expect(readStagedDump(dir, name)).toEqual({ ok: false, error: "invalid_name" });
86 expect(readStagedDump(dir, "nope.sql")).toEqual({ ok: false, error: "missing" });
87 });
88 });
89
90 describe("MCP app_data_import", () => {
91 const tool = TOOLS.find((t) => t.name === "app_data_import")!;
92 const call = (user: User, args: Record<string, unknown>) => tool.run({ app } as never, user, args);
93 test("is a write tool, registered", () => {
94 expect(tool.writes).toBe(true);
95 });
96 test("a non-admin is told not_found; an admin loads; a refused dump says why", async () => {
97 const dump = "CREATE TABLE m(a); INSERT INTO m VALUES (1);";
98 expect(await call(dee, { repo: "cy/shop", dump, file: "m.db" })).toMatchObject({ ok: false, error: "not_found" });
99 const done = await call(cy, { repo: "cy/shop", dump, file: "m.db" });
100 expect(done).toMatchObject({ ok: true });
101 expect(await call(cy, { repo: "cy/shop", dump: "CREATE TABLE x AS SELECT 1", file: "x.db" })).toMatchObject({ ok: false, error: "refused" });
102 expect(await call(cy, { repo: "cy/shop", dump, file: "m.db" })).toMatchObject({ ok: false, error: "exists" });
103 });
104 });
test/mcp.test.ts +1 −1
@@ -371,7 +371,7 @@ describe("the forge over MCP", () => {
371371 "contribute",
372372 "limits",
373373 "app_env",
374 "agents",
374 "agents", "app_data_import",
375375 "teams",
376376 "cloud_accounts",
377377 "cloud_account",

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs