trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

MCP and API for data: a `data` tool and /api/v1/…/data endpoints, so agents and editors browse what a person can #10347

Merged scoutcrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/7582-mcp-and-api-for-data-a-data-tool-and-api → main 10 commits fork

  • Merged. scout/7582-mcp-and-api-for-data-a-data-tool-and-api was merged into main as 9112d0c.

Review

Sign in to review this change.
12 files changed +454 −13
docs/features/data-browser.md +4
@@ -14,6 +14,10 @@ Limits on work: `query` runs at most 2 workers at once (a third is told it is bu
1414
1515 Terminal: `src/tui/screens/data.ts` (files → tables → rows, `n`/`p` for pages, `s` sort, `f` filter, `c` clear, `x` read-only SQL), offered by `maySeeData`; the forge's own database is among an admin's files. SQL on either door is 20 runs a minute per person (`src/data/sql_window.ts`); beyond that it says when to retry.
1616
17 API and MCP by repository (`data/repo_scope.ts`, `http/api/data.ts`): `GET /api/v1/repos/:o/:n/data` (databases as `{ env, file }`), `…/data/:env/:file/tables`, `…/rows?table=&page=`, `POST …/query {sql}`; MCP `data` with `{ repo, env?, file?, table?, sql? }` answers the same. Only writers of the repository (and admins) see anything; everyone else gets a 404 (an error from the tool). Same masking, read-only SQL and 1,000-row cap as the web; SQL shares the per-person 20 a minute (`sql_window.ts`), and past it the API answers 429 with `retry-after`, the tool an error. `page` is a whole number from 0 to 1000.
18
19 Not done: SQL in the terminal (SQL is on the web page and MCP).
20
1721 ## On the web (#7580)
1822
1923 Three doors, one body (`src/web/pages/data_view.ts`): `/data` (every file you may see), `/admin/data` (the forge's own database, admins; the Forge data button on `/data`) and `/:owner/:name/data` (an app's Data tab for its writers: environments → files → tables → rows; a reader gets the 404). Rows sort by header (`sort`, `dir`), filter by text on one column or all (`q`, `col`), JSON cells are pretty-printed, and a copy button per row puts it on the clipboard as JSON. Sealed columns show `••••••` and cannot be sorted or searched. The SQL box is a GET form (`sql`, capped at 4,000 characters), so a query is an address to share; it shows the rows and the elapsed ms. The sorted column carries `aria-sort` and a muted arrow from CSS; a wide grid scrolls sideways only. Not done: saved queries, export, editing rows.
src/contract/openapi/routes/getApiV1ReposOwnerNameData.ts added +9
@@ -0,0 +1,9 @@
1 import { REPO, type Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "get",
5 path: `${REPO}/data`,
6 summary:
7 "The SQLite databases of the repository's app that you may read, as the MCP data tool with repo lists them: { items: [{ env, file, label }] }; only people who can write the repository (and admins) see them, anyone else a 404; needs a token",
8 auth: true,
9 };
src/contract/openapi/routes/getApiV1ReposOwnerNameDataEnvFileRows.ts added +14
@@ -0,0 +1,14 @@
1 import { REPO, type Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "get",
5 path: `${REPO}/data/{env}/{file}/rows`,
6 summary:
7 "One page of 50 rows of a table: { columns, page, total, rows }; secret columns (token, secret, password, hash, key, cookie, …) show ••••••; no such table is a 404; needs a token",
8 auth: true,
9 query: { table: "string", page: { type: "integer", min: 0, max: 1000 } },
10 refusals: {
11 invalid_table: "table is missing (422)",
12 invalid_page: "page is not a whole number from 0 (422)",
13 },
14 };
src/contract/openapi/routes/getApiV1ReposOwnerNameDataEnvFileTables.ts added +8
@@ -0,0 +1,8 @@
1 import { REPO, type Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "get",
5 path: `${REPO}/data/{env}/{file}/tables`,
6 summary: "The tables of one database with their row counts: { items: [{ name, rows }] }; a database you may not read is a 404; needs a token",
7 auth: true,
8 };
src/contract/openapi/routes/index.ts +4
@@ -121,6 +121,9 @@ export { route as getApiV1ReposOwnerNameCommitsOidCheck } from "./getApiV1ReposO
121121 export { route as getApiV1ReposOwnerNameContents } from "./getApiV1ReposOwnerNameContents.ts";
122122 export { route as getApiV1ReposOwnerNameContentsPath } from "./getApiV1ReposOwnerNameContentsPath.ts";
123123 export { route as getApiV1ReposOwnerNameCrew } from "./getApiV1ReposOwnerNameCrew.ts";
124 export { route as getApiV1ReposOwnerNameData } from "./getApiV1ReposOwnerNameData.ts";
125 export { route as getApiV1ReposOwnerNameDataEnvFileRows } from "./getApiV1ReposOwnerNameDataEnvFileRows.ts";
126 export { route as getApiV1ReposOwnerNameDataEnvFileTables } from "./getApiV1ReposOwnerNameDataEnvFileTables.ts";
124127 export { route as getApiV1ReposOwnerNameDecisions } from "./getApiV1ReposOwnerNameDecisions.ts";
125128 export { route as getApiV1ReposOwnerNameEgressGrants } from "./getApiV1ReposOwnerNameEgressGrants.ts";
126129 export { route as getApiV1ReposOwnerNameEvents } from "./getApiV1ReposOwnerNameEvents.ts";
@@ -284,6 +287,7 @@ export { route as postApiV1ReposOwnerNameBuildsPreview } from "./postApiV1ReposO
284287 export { route as postApiV1ReposOwnerNameChecksRecheck } from "./postApiV1ReposOwnerNameChecksRecheck.ts";
285288 export { route as postApiV1ReposOwnerNameContents } from "./postApiV1ReposOwnerNameContents.ts";
286289 export { route as postApiV1ReposOwnerNameContribute } from "./postApiV1ReposOwnerNameContribute.ts";
290 export { route as postApiV1ReposOwnerNameDataEnvFileQuery } from "./postApiV1ReposOwnerNameDataEnvFileQuery.ts";
287291 export { route as postApiV1ReposOwnerNameEdits } from "./postApiV1ReposOwnerNameEdits.ts";
288292 export { route as postApiV1ReposOwnerNameEgressGrants } from "./postApiV1ReposOwnerNameEgressGrants.ts";
289293 export { route as postApiV1ReposOwnerNameFollowingConverge } from "./postApiV1ReposOwnerNameFollowingConverge.ts";
src/contract/openapi/routes/postApiV1ReposOwnerNameDataEnvFileQuery.ts added +15
@@ -0,0 +1,15 @@
1 import { REPO, type Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: `${REPO}/data/{env}/{file}/query`,
6 summary:
7 "Run one read-only SELECT or WITH on a database (read-only connection, 5 s, at most 1,000 rows, secret columns masked): { columns, rows, truncated }; reads only, so a read-scope token suffices; needs a token",
8 auth: true,
9 body: { sql: "string" },
10 refusals: {
11 invalid_sql: "sql is not one SELECT or WITH statement, or SQLite refused it (422)",
12 timeout: "the query ran past 5 seconds (422)",
13 busy: "other queries are running; try again (429)",
14 },
15 };
src/data/repo_scope.ts added +78
@@ -0,0 +1,78 @@
1 /**
2 * The data browser addressed by repository, env and file: what `/api/v1/repos/:o/:n/data…` and the
3 * MCP `data` tool (with `repo`) answer. Visibility is `listSources` alone, so a reader, a guest and
4 * a stranger see exactly what the web shows them: nothing, as a 404.
5 */
6
7 import type { Database } from "bun:sqlite";
8 import type { App } from "../app.ts";
9 import { getAppByRepo } from "../core/apps.ts";
10 import { canWrite, type Repo } from "../core/repos.ts";
11 import type { User } from "../core/users.ts";
12 import { isEnvName } from "../apps/data.ts";
13 import { rows, tables } from "./browse.ts";
14 import { query } from "./query.ts";
15 import { sqlWindow } from "./sql_window.ts";
16 import { listSources, type Source } from "./sources.ts";
17
18 /**
19 * The one guard of every door: the sources of the repository's app this viewer may read, or
20 * undefined (a 404 to the caller) when the viewer cannot write to the repository: a reader, a
21 * guest or a stranger has no door to the data. An empty list means a writer, and no database yet.
22 */
23 export function repoSources(db: Database, dataDir: string, viewer: User | undefined, repo: Repo): Source[] | undefined {
24 if (viewer === undefined || !canWrite(db, repo, viewer)) return undefined;
25 const app = getAppByRepo(db, repo.id);
26 if (app === undefined) return [];
27 const prefix = `app:${app.id}:`;
28 return listSources(db, dataDir, viewer).filter((s) => s.kind === "app" && s.id.startsWith(prefix));
29 }
30
31 /** A page number as REST and MCP both take it: a whole number from 0 to 1000, else undefined. */
32 export function parsePage(raw: unknown): number | undefined {
33 if (typeof raw === "string") return /^[0-9]{1,4}$/.test(raw) && Number(raw) <= MAX_PAGE ? Number(raw) : undefined;
34 return typeof raw === "number" && Number.isInteger(raw) && raw >= 0 && raw <= MAX_PAGE ? raw : undefined;
35 }
36 const MAX_PAGE = 1000;
37
38 export interface Pick {
39 readonly env?: string | undefined;
40 readonly file?: string | undefined;
41 }
42
43 export type Failure = { readonly error: "not_found" | "invalid"; readonly message: string };
44
45 /** `{ env, file }` to one source; both are checked, then looked up in the allowed list. */
46 export function pickSource(sources: readonly Source[], pick: Pick): Source | Failure {
47 const { env, file } = pick;
48 if (env === undefined || file === undefined) return { error: "invalid", message: "`env` and `file` name a database" };
49 if (!isEnvName(env)) return { error: "invalid", message: "`env` is an environment name" };
50 const found = sources.find((s) => {
51 const part = s.id.split(":");
52 return part.length === 4 && part[2] === env && part[3] === file;
53 });
54 return found ?? { error: "not_found", message: "no such database" };
55 }
56
57 export const isFailure = (v: Source | Failure): v is Failure => "error" in v;
58
59 export const sourceList = (sources: readonly Source[]) =>
60 sources.map((s) => {
61 const [, , env, file] = s.id.split(":");
62 return { env, file, label: s.label };
63 });
64
65 export const tableList = (source: Source) => tables(source);
66
67 export function rowPage(source: Source, table: string, page: number | undefined) {
68 return rows(source, table, page === undefined ? {} : { page });
69 }
70
71 export type SqlOutcome = Awaited<ReturnType<typeof query>> | { readonly ok: false; readonly error: "rate"; readonly message: string; readonly retryAfter: number };
72
73 /** Read-only SQL: the same checks, caps and masking as the web (`query.ts`), and its per-person budget (`sql_window.ts`). */
74 export async function runSql(app: App, user: User, source: Source, sql: string): Promise<SqlOutcome> {
75 const wait = sqlWindow(app, user.id);
76 if (wait !== undefined) return { ok: false, error: "rate", message: `too many queries: try again in ${wait} s`, retryAfter: wait };
77 return query(source, sql);
78 }
src/http/api.ts +2
@@ -352,6 +352,7 @@ import { boardRoutes } from "./api/boards.ts";
352352 import { displayRoutes } from "./api/displays.ts";
353353 import { followingRoutes } from "./api/following.ts";
354354 import { labelsRoutes } from "./api/labels.ts";
355 import { dataRoutes } from "./api/data.ts";
355356 import { visionRoutes } from "./api/vision.ts";
356357 import { businessPlanRoutes } from "./api/business_plan.ts";
357358 import { tokenRoutes } from "./api/tokens.ts";
@@ -2089,6 +2090,7 @@ export function apiRoutes(router: Router<Handler>): void {
20892090
20902091 labelsRoutes(router);
20912092 visionRoutes(router);
2093 dataRoutes(router);
20922094 businessPlanRoutes(router);
20932095 router.add(
20942096 "PUT",
src/http/api/data.ts added +82
@@ -0,0 +1,82 @@
1 /**
2 * A repository's databases over the API, as the web `/data` and MCP `data` show them: `GET …/data`,
3 * `…/data/:env/:file/tables`, `…/rows?table=&page=`, and `POST …/query` (one read-only SELECT). Who
4 * may see what is `data/sources.ts`; to anyone else every one of these is a 404.
5 */
6
7 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
8 import {
9 isFailure,
10 parsePage,
11 pickSource,
12 repoSources,
13 rowPage,
14 runSql,
15 sourceList,
16 tableList,
17 } from "../../data/repo_scope.ts";
18 import { loadRepo } from "../api.ts";
19 import type { Ctx, Handler } from "../context.ts";
20 import { apiError, body, json, notFound, requireScope } from "../respond.ts";
21 import type { Router } from "../router.ts";
22
23 const PRIVATE = { "cache-control": "private, no-store", vary: "authorization, cookie" };
24 const reply = (value: unknown): Response => json(value, 200, PRIVATE);
25
26 function find(ctx: Ctx) {
27 const repo = loadRepo(ctx);
28 if (!repo) return undefined;
29 const sources = repoSources(ctx.app.db, ctx.app.config.dataDir, ctx.auth.user, repo);
30 if (sources === undefined) return undefined;
31 const picked = pickSource(sources, { env: ctx.params.env, file: ctx.params.file });
32 return isFailure(picked) ? undefined : { sources, source: picked };
33 }
34
35 export function dataRoutes(router: Router<Handler>): void {
36 router.add("GET", "/api/v1/repos/:owner/:name/data", (ctx) => {
37 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
38 const repo = loadRepo(ctx);
39 if (!repo) return notFound();
40 const sources = repoSources(ctx.app.db, ctx.app.config.dataDir, ctx.auth.user, repo);
41 if (sources === undefined) return notFound();
42 return reply({ items: sourceList(sources) });
43 });
44
45 router.add("GET", "/api/v1/repos/:owner/:name/data/:env/:file/tables", (ctx) => {
46 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
47 const found = find(ctx);
48 if (!found?.source) return notFound();
49 return reply({ items: tableList(found.source) });
50 });
51
52 router.add("GET", "/api/v1/repos/:owner/:name/data/:env/:file/rows", (ctx) => {
53 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
54 const found = find(ctx);
55 if (!found?.source) return notFound();
56 const table = ctx.url.searchParams.get("table");
57 if (table === null || table === "") return apiError("invalid_table", 422, "table names a table");
58 const raw = ctx.url.searchParams.get("page");
59 const page = raw === null ? undefined : parsePage(raw);
60 if (raw !== null && page === undefined) return apiError("invalid_page", 422, "page is a whole number from 0 to 1000");
61 const result = rowPage(found.source, table, page);
62 return result ? reply(result) : notFound();
63 });
64
65 router.add("POST", "/api/v1/repos/:owner/:name/data/:env/:file/query", async (ctx) => {
66 const denied = requireScope(ctx, "ro");
67 if (denied) return denied;
68 const found = find(ctx);
69 if (!found?.source) return notFound();
70 const input = await body(ctx.request);
71 if (typeof input["sql"] !== "string") return apiError("invalid_sql", 422, "sql is text");
72 const user = ctx.auth.user;
73 if (!user) return notFound();
74 const result = await runSql(ctx.app, user, found.source, input["sql"]);
75 if (!result.ok) {
76 if (result.error === "rate")
77 return json({ error: "rate_limited", message: result.message, retryAfter: result.retryAfter }, 429, { ...PRIVATE, "retry-after": String(result.retryAfter) });
78 return apiError(result.error === "busy" ? "busy" : result.error === "timeout" ? "timeout" : "invalid_sql", result.error === "busy" ? 429 : 422, result.message);
79 }
80 return reply(result);
81 });
82 }
src/http/mcp/tools/data.ts +37 −12
@@ -1,15 +1,18 @@
11 import { rows, tables } from "../../../data/browse.ts";
2 import { query } from "../../../data/query.ts";
32 import { findSource, listSources } from "../../../data/sources.ts";
4 import { bad, good, type Tool } from "../shared.ts";
3 import { isFailure, parsePage, pickSource, repoSources, rowPage, runSql, sourceList, tableList } from "../../../data/repo_scope.ts";
4 import { bad, good, isAnswer, repoArg, type Tool } from "../shared.ts";
55
66 export const tool: Tool = {
77 name: "data",
8 doc: "the SQLite files you may read (the forge's own for an admin; each app's per environment for its writers), with `source`: its tables and row counts, with `table` (and `page`): a page of rows, with `sql`: a read-only SELECT (5 s, 1,000 rows); secret columns are masked",
8 doc: "`{ repo, env?, file?, table?, sql? }` as `GET …/data`: an app's databases, then `env`+`file`: its tables, `table`: rows, `sql`: one read-only SELECT; its writers only. Without `repo`: the SQLite files you may read (the forge's own for an admin; each app's per environment for its writers), with `source`: its tables and row counts, with `table` (and `page`): a page of rows, with `sql`: a read-only SELECT (5 s, 1,000 rows); secret columns are masked",
99 description: "Read SQLite files you may see: tables, rows, read-only SQL; secrets masked.",
1010 inputSchema: {
1111 type: "object",
1212 properties: {
13 repo: { type: "string" },
14 env: { type: "string" },
15 file: { type: "string" },
1316 source: { type: "string" },
1417 table: { type: "string" },
1518 page: { type: "integer" },
@@ -19,21 +22,43 @@ export const tool: Tool = {
1922 writes: false,
2023 async run(ctx, user, args) {
2124 const { db, config } = ctx.app;
22 const { source: id, table, page, sql } = args;
23 for (const [k, v] of Object.entries({ source: id, table, sql }))
24 if (v !== undefined && typeof v !== "string") return bad("invalid", `\`${k}\` is text`);
25 if (page !== undefined && typeof page !== "number") return bad("invalid", "`page` is a number");
25 for (const k of ["source", "table", "sql", "env", "file"])
26 if (args[k] !== undefined && typeof args[k] !== "string") return bad("invalid", `\`${k}\` is text`);
27 const str = (k: string): string | undefined => {
28 const v = args[k];
29 return typeof v === "string" ? v : undefined;
30 };
31 const [id, table, sql, env, file] = ["source", "table", "sql", "env", "file"].map(str);
32 const page = args["page"];
33 const pageNo = page === undefined || typeof page !== "number" ? undefined : parsePage(page);
34 if (page !== undefined && pageNo === undefined) return bad("invalid", "`page` is a whole number from 0 to 1000");
35 if (args["repo"] !== undefined) {
36 const repo = repoArg(ctx, user, args);
37 if (isAnswer(repo)) return repo;
38 const sources = repoSources(db, config.dataDir, user, repo);
39 if (sources === undefined) return bad("not_found", "no such app or repository (owner/name)");
40 if (env === undefined && file === undefined) return good({ items: sourceList(sources) });
41 const picked = pickSource(sources, { env, file });
42 if (isFailure(picked)) return bad(picked.error, picked.message);
43 if (typeof sql === "string") {
44 const result = await runSql(ctx.app, user, picked, sql);
45 return result.ok ? good(result) : bad(result.error === "rate" ? "rate_limited" : "invalid", result.message);
46 }
47 if (table === undefined) return good({ items: tableList(picked) });
48 const found = rowPage(picked, table, pageNo);
49 return found ? good(found) : bad("not_found", "no such table");
50 }
2651 if (id === undefined) {
2752 return good(listSources(db, config.dataDir, user).map((s) => ({ source: s.id, label: s.label })));
2853 }
29 const source = findSource(db, config.dataDir, user, id as string);
54 const source = findSource(db, config.dataDir, user, id);
3055 if (!source) return bad("not_found", "no such source");
31 if (sql !== undefined) {
32 const result = await query(source, sql as string);
33 return result.ok ? good(result) : bad("invalid", result.message);
56 if (typeof sql === "string") {
57 const result = await runSql(ctx.app, user, source, sql);
58 return result.ok ? good(result) : bad(result.error === "rate" ? "rate_limited" : "invalid", result.message);
3459 }
3560 if (table === undefined) return good(tables(source));
36 const found = rows(source, table as string, page === undefined ? {} : { page: page as number });
61 const found = rows(source, table, pageNo === undefined ? {} : { page: pageNo });
3762 return found ? good(found) : bad("not_found", "no such table");
3863 },
3964 };
test/api-data.test.ts added +188
@@ -0,0 +1,188 @@
1 /** `/api/v1/repos/:o/:n/data…` and MCP `data` with `repo`: the web's permissions, read-only SQL, masking, the 1,000-row cap. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { Database } from "bun:sqlite";
4 import { mkdirSync } from "node:fs";
5 import { mkdtemp, rm } from "node:fs/promises";
6 import { tmpdir } from "node:os";
7 import { join } from "node:path";
8 import { createApp, type App } from "../src/app.ts";
9 import { envDataDir } from "../src/apps/data.ts";
10 import { createApp as createAppRecord } from "../src/core/apps.ts";
11 import { createRepo } from "../src/core/repos.ts";
12 import { createSession } from "../src/core/sessions.ts";
13 import { createToken } from "../src/core/tokens.ts";
14 import { createUser } from "../src/core/users.ts";
15 import { listSources } from "../src/data/sources.ts";
16 import { serve, type Servers } from "../src/http/server.ts";
17 import { quietLogs } from "./helpers/globals.ts";
18
19 quietLogs();
20
21 let dataDir: string;
22 let app: App;
23 let aliceId = "";
24 let servers: Servers;
25 const tok: Record<string, string> = {};
26 const base = "/api/v1/repos/alice/notes/data";
27
28 const ok = <T>(r: { ok: true; value: T } | { ok: false; error: unknown }): T => {
29 if (!r.ok) throw new Error(JSON.stringify(r.error));
30 return r.value;
31 };
32 const get = (token: string | null, path: string) =>
33 fetch(`${servers.url}${base}${path}`, { headers: token ? { authorization: `Bearer ${token}` } : {} });
34 const post = (token: string, path: string, sql: unknown) =>
35 fetch(`${servers.url}${base}${path}`, {
36 method: "POST",
37 headers: { authorization: `Bearer ${token}`, "content-type": "application/json" },
38 body: JSON.stringify({ sql }),
39 });
40 async function mcp(token: string, args: Record<string, unknown>): Promise<any> {
41 const res = await fetch(`${servers.url}/mcp`, {
42 method: "POST",
43 headers: { "content-type": "application/json", accept: "application/json, text/event-stream", authorization: `Bearer ${token}` },
44 body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "data", arguments: args } }),
45 });
46 const result = ((await res.json()) as any).result;
47 const text = result.content[0].text as string;
48 return { isError: result.isError === true, body: result.isError === true ? text : JSON.parse(text) };
49 }
50
51 beforeAll(async () => {
52 dataDir = await mkdtemp(join(tmpdir(), "forge-api-data-"));
53 app = createApp({ dataDir, port: 0 });
54 servers = await serve(app);
55 const make = async (username: string, adminIfFirst = false) =>
56 ok(await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst }));
57 const admin = await make("boss", true);
58 const alice = await make("alice");
59 const bob = await make("bob");
60 aliceId = alice.id;
61 const repo = ok(await createRepo(app.db, app.reposRoot, { ownerId: alice.id, name: "notes", visibility: "public" }));
62 const record = ok(createAppRecord(app.db, { repoId: repo.id, slug: "notes", access: "public" }));
63 mkdirSync(envDataDir(dataDir, record.id, "production"), { recursive: true });
64 const d = new Database(join(envDataDir(dataDir, record.id, "production"), "app.db"));
65 d.exec("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, api_token TEXT)");
66 d.exec("CREATE TABLE big (n INTEGER)");
67 d.exec("INSERT INTO users VALUES (1, 'ann', 'tok-secret-1')");
68 d.exec("WITH RECURSIVE c(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM c WHERE n < 1500) INSERT INTO big SELECT n FROM c");
69 d.close();
70 tok["alice"] = createToken(app.db, alice.id, "rw", "rw").token;
71 tok["alice-ro"] = createToken(app.db, alice.id, "ro", "ro").token;
72 tok["bob"] = createToken(app.db, bob.id, "rw", "rw").token;
73 tok["admin"] = createToken(app.db, admin.id, "rw", "rw").token;
74 }, 30_000);
75
76 afterAll(async () => {
77 servers?.stop();
78 app?.close();
79 await rm(dataDir, { recursive: true, force: true });
80 });
81
82 test("a writer lists, browses and gets the same from MCP", async () => {
83 const list = (await (await get(tok["alice"]!, "")).json()) as any;
84 expect(list.items).toEqual([{ env: "production", file: "app.db", label: "notes · production · app.db" }]);
85 expect((await mcp(tok["alice"]!, { repo: "alice/notes" })).body).toEqual(list);
86 const tables = (await (await get(tok["alice"]!, "/production/app.db/tables")).json()) as any;
87 expect(tables.items).toEqual([{ name: "big", rows: 1500 }, { name: "users", rows: 1 }]);
88 expect((await mcp(tok["alice"]!, { repo: "alice/notes", env: "production", file: "app.db" })).body).toEqual(tables);
89 const rows = (await (await get(tok["alice"]!, "/production/app.db/rows?table=big&page=1")).json()) as any;
90 expect(rows.rows.length).toBe(50);
91 expect((await mcp(tok["alice"]!, { repo: "alice/notes", env: "production", file: "app.db", table: "big", page: 1 })).body).toEqual(rows);
92 });
93
94 test("permission parity with the web: a reader, a stranger and a guest get 404s", () =>
95 Promise.all(
96 [null, tok["bob"]!].map(async (t) => {
97 for (const path of ["", "/production/app.db/tables", "/production/app.db/rows?table=users"])
98 expect((await get(t, path)).status).toBe(404);
99 }),
100 ).then(async () => {
101 expect(listSources(app.db, dataDir, undefined)).toEqual([]);
102 // A reader's token and a signed-out caller meet the same refusal: 404 for the token, 401 for no identity.
103 expect((await post(tok["bob"]!, "/production/app.db/query", "SELECT 1")).status).toBe(404);
104 expect((await post("x", "/production/app.db/query", "SELECT 1")).status).toBe(401);
105 const m = await mcp(tok["bob"]!, { repo: "alice/notes" });
106 expect(m.isError).toBe(true);
107 expect((await get(tok["admin"]!, "")).status).toBe(200);
108 }));
109
110 test("unknown env, file or table is a 404; odd names resolve to nothing", async () => {
111 for (const path of ["/staging/app.db/tables", "/production/other.db/tables", "/production/..%2Fforge.db/tables", "/production/app.db/rows?table=nope"])
112 expect((await get(tok["alice"]!, path)).status).toBe(404);
113 expect((await get(tok["alice"]!, "/production/app.db/rows")).status).toBe(422);
114 expect((await get(tok["alice"]!, "/production/app.db/rows?table=big&page=x")).status).toBe(422);
115 });
116
117 test("masking: secret columns show dots in rows and queries, over REST and MCP", async () => {
118 const rows = await (await get(tok["alice"]!, "/production/app.db/rows?table=users")).text();
119 expect(rows).not.toContain("tok-secret-1");
120 expect(rows).toContain("ann");
121 const q = await (await post(tok["alice"]!, "/production/app.db/query", "SELECT api_token AS t FROM users")).text();
122 expect(q).not.toContain("tok-secret-1");
123 const m = await mcp(tok["alice"]!, { repo: "alice/notes", env: "production", file: "app.db", sql: "SELECT * FROM users" });
124 expect(JSON.stringify(m.body)).not.toContain("tok-secret-1");
125 });
126
127 test("read-only: writes are refused and change nothing; a read token may query", async () => {
128 for (const sql of ["DELETE FROM users", "DROP TABLE users", "SELECT 1; DELETE FROM users", "PRAGMA writable_schema = 1"]) {
129 const res = await post(tok["alice"]!, "/production/app.db/query", sql);
130 expect(res.status).toBe(422);
131 expect(((await res.json()) as any).error).toBe("invalid_sql");
132 expect((await mcp(tok["alice"]!, { repo: "alice/notes", env: "production", file: "app.db", sql })).isError).toBe(true);
133 }
134 expect((await post(tok["alice"]!, "/production/app.db/query", 5)).status).toBe(422);
135 const count = await post(tok["alice-ro"]!, "/production/app.db/query", "SELECT count(*) AS n FROM users");
136 expect(((await count.json()) as any).rows).toEqual([[1]]);
137 });
138
139 test("a query returns at most 1,000 rows and says it was cut", async () => {
140 const res = (await (await post(tok["alice"]!, "/production/app.db/query", "SELECT n FROM big")).json()) as any;
141 expect(res.rows.length).toBe(1000);
142 expect(res.truncated).toBe(true);
143 const m = await mcp(tok["alice"]!, { repo: "alice/notes", env: "production", file: "app.db", sql: "SELECT n FROM big" });
144 expect(m.body.rows.length).toBe(1000);
145 });
146
147 test("a cross-origin session post is refused with a 403; the same-origin one runs", async () => {
148 const cookie = `forge_session=${createSession(app.db, aliceId).token}`;
149 const send = (origin: string) =>
150 fetch(`${servers.url}${base}/production/app.db/query`, {
151 method: "POST",
152 headers: { cookie, origin, "content-type": "application/json" },
153 body: JSON.stringify({ sql: "SELECT 1 AS one" }),
154 });
155 expect((await send("https://evil.example")).status).toBe(403);
156 expect((await send(servers.url)).status).toBe(200);
157 });
158
159 test("the spec lists all four routes", async () => {
160 const spec = (await (await fetch(`${servers.url}/api/v1/openapi.json`, { redirect: "follow" })).json()) as any;
161 const paths = Object.keys(spec.paths);
162 for (const p of ["/data", "/data/{env}/{file}/tables", "/data/{env}/{file}/rows", "/data/{env}/{file}/query"])
163 expect(paths).toContain(`/api/v1/repos/{owner}/{name}${p}`);
164 });
165
166 test("page is a whole number from 0 to 1000 on both doors", async () => {
167 for (const page of ["-1", "1.5", "1001", "99999", "abc"])
168 expect((await get(tok["alice"]!, `/production/app.db/rows?table=big&page=${page}`)).status).toBe(422);
169 expect((await get(tok["alice"]!, "/production/app.db/rows?table=big&page=1000")).status).toBe(200);
170 const args = { repo: "alice/notes", env: "production", file: "app.db", table: "big" };
171 for (const page of [-1, 1.5, 1001, Number.MAX_SAFE_INTEGER, "2", null])
172 expect((await mcp(tok["alice"]!, { ...args, page })).isError).toBe(true);
173 expect((await mcp(tok["alice"]!, { ...args, page: 0 })).isError).toBe(false);
174 });
175
176 test("SQL is 20 runs a minute per person on both doors, then 429 with the wait", async () => {
177 const run = () => post(tok["admin"]!, "/production/app.db/query", "SELECT 1 AS one");
178 for (let i = 0; i < 20; i++) expect((await run()).status).toBe(200);
179 const refused = await run();
180 expect(refused.status).toBe(429);
181 expect(Number(refused.headers.get("retry-after"))).toBeGreaterThanOrEqual(1);
182 expect(((await refused.json()) as any).error).toBe("rate_limited");
183 const m = await mcp(tok["admin"]!, { repo: "alice/notes", env: "production", file: "app.db", sql: "SELECT 1" });
184 expect(m.isError).toBe(true);
185 expect(m.body).toContain("too many queries");
186 // Another person has their own budget.
187 expect((await post(tok["alice"]!, "/production/app.db/query", "SELECT 1")).status).toBe(200);
188 });
test/security-floor-routes.test.ts +13 −1
@@ -9,6 +9,9 @@
99 * miss answers carry the floor too — and the list is checked both ways.
1010 */
1111 import { afterAll, beforeAll, expect, test } from "bun:test";
12 import { Database } from "bun:sqlite";
13 import { mkdirSync } from "node:fs";
14 import { envDataDir } from "../src/apps/data.ts";
1215 import { mkdtemp, rm } from "node:fs/promises";
1316 import { tmpdir } from "node:os";
1417 import { join } from "node:path";
@@ -155,6 +158,11 @@ beforeAll(async () => {
155158 const run = createRun(app.db, { repoId: repo.id, kind: "check", issueId: pull.value.id, commitOid: pull.value.headOid });
156159 const inserted = insertApp(app.db, { repoId: repo.id, slug: "notes", access: "public" });
157160 if (!inserted.ok) throw new Error(inserted.error);
161 // An app database for the data routes: data routes name the production env.
162 mkdirSync(envDataDir(dataDir, inserted.value.id, "production"), { recursive: true });
163 const appDb = new Database(join(envDataDir(dataDir, inserted.value.id, "production"), "app.db"));
164 appDb.exec("CREATE TABLE t (n INTEGER); INSERT INTO t VALUES (1)");
165 appDb.close();
158166 Object.assign(fill, {
159167 owner: "ada", name: "notes", username: "ada", ref: "main", oid: pull.value.headOid,
160168 path: "a.txt", env: "prod", slug: "notes", spec: "192", machine: "standard",
@@ -182,12 +190,16 @@ function pathFor(pattern: string): string {
182190 .replace(/[:*]([a-z_]+)/g, (_, key: string) => {
183191 if (key === "number") return number!;
184192 if (key === "id") return runId!;
193 if (key === "env" && pattern.includes("/data/:env/")) return "production";
194 if (key === "file" && pattern.includes("/data/:env/:file/")) return "app.db";
185195 return SEEDED.has(key) ? fill[key]! : "x";
186196 });
187197 }
188198
189199 const unseeded = (pattern: string): boolean =>
190 paramsOf(pattern).some((key) => !SEEDED.has(key) && !(key === "id" && pattern.includes("/runs/:id")));
200 paramsOf(pattern).some(
201 (key) => !SEEDED.has(key) && !(key === "id" && pattern.includes("/runs/:id")) && !(key === "file" && pattern.includes("/data/:env/:file/")),
202 );
191203
192204 /**
193205 * `accept-encoding: identity` skips gzip, which only costs time here: the floor

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs