Public: anyone can read this; the crew can change it.

A Data tab: every app's databases in the web, and the forge's own at /admin/data — table list, sortable paged rows, filters, a read-only SQL box #10314

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/7580-a-data-tab-every-app-s-databases-in-the → main 5 commits fork 3 threads · 3 unresolved

  • Merged. scout/7580-a-data-tab-every-app-s-databases-in-the was merged into main as 71d797a.

Review

Sign in to review this change.
12 files changed +597 −86
docs/design/areas/table.md +2
@@ -3,3 +3,5 @@
33 A §5 `.table` too wide for a narrow screen sits inside a horizontal scroller, so the page itself never scrolls sideways and only the table's figures move.
44
55 - `.table-scroll` — the wrapper `<div>` around a wide `.table`: `overflow-x: auto` with a thin scrollbar, styled in `src/web/style/components.ts`. Pair it with `.table[data-sticky-lead]` to keep each row's name in view while the figures scroll. Written by `src/web/pages/account.ts`, `src/web/pages/audit.ts`, `src/web/pages/boards-held.ts`, `src/web/pages/crew.ts`, `src/web/pages/flow-panel.ts`, `src/web/pages/keeper-panel.ts`, `src/web/pages/passkeys.ts`, `src/web/pages/proposal-tokens.ts`, `src/web/pages/settings.ts`, `src/web/pages/spend.ts`, `src/web/pages/subscriptions.ts`, `src/web/pages/system.ts`, `fleetGraphSection`'s figures table in `src/web/fleet-graph.ts`, and `vitalsTable` in `src/web/studio/vitals.ts`.
6
7 - The data browser (#7580, `src/web/pages/data_view.ts`) adds no class. A sortable `.table` carries `data-sortable`; its sorted header has `aria-sort`, and `style/components.ts` draws a muted arrow after the link from that attribute. A cell's JSON is a `<pre>` inside the `td`. The grid scrolls sideways only; 50 rows a page keep it in the page's flow, so there is no max-height box and no sticky header.
docs/design/audit.md +1
@@ -104,6 +104,7 @@ Machine routes are not pages and have no entry: `/audit.csv`,
104104 - `/maintenance` — `pages/maintenance.ts` · prepare the forge for a power-off and end it · `/resilience`, `/settings/backups` · none seen.
105105 - `/admin/limits` — `pages/admin-limits.ts` · every limit with value, default and reason · `/settings/capacity` (also a numbers-you-can-change page) · none seen.
106106 - `/admin/flows` — `pages/flows-all.ts` · every repository's flow lines with state and last moved, by kind and stale · `/:owner/:name/flows` · none seen.
107 - `/admin/data` — `pages/data.ts` · the forge's own database: tables, rows, read-only SQL, secrets masked · `/data` (the Forge data button) · none seen; shares its body (`pages/data_view.ts`) with `/data` and `/:owner/:name/data`.
107108 - `/settings/economics` — `pages/economics.ts` · the profit slider's position and the margin, with the policy in words · `/admin/limits`, `/settings/capacity` · none seen.
108109 - `/settings/capacity` — `pages/capacity.ts` · how many runs and checks go at once · `/admin/limits`, `/settings/usage` · none seen.
109110 - `/settings/usage` — `pages/usage.ts` · route patterns by use, unused first · `/settings/telemetry`, `/board/returns` · none seen. Not about cost, despite the name.
docs/features/data-browser.md +4 −2
@@ -12,6 +12,8 @@ Views are masked whole when their text mentions a masked column (an alias cannot
1212
1313 Limits on work: `query` runs at most 2 workers at once (a third is told it is busy), each with a 128 MB heap cap (`hard_heap_limit`), a 5 s timer, 1,000 rows and about 250 KB of result; every cell is cut to 1,000 characters inside SQLite and a blob shows as its size. `rows()` cuts cells the same way.
1414
15 Terminal: `src/tui/screens/data.ts` (files → tables → rows, `n`/`p` for pages), offered by `maySeeData`.
15 Terminal: `src/tui/screens/data.ts` (files → tables → rows, `n`/`p` for pages, `s` sort, `f` filter, `c` clear, `x` read-only SQL), offered by `maySeeData`; the forge's own database is among an admin's files. SQL on either door is 20 runs a minute per person (`src/data/sql_window.ts`); beyond that it says when to retry.
1616
17 Not done: a SQL editor on the page or in the terminal (SQL is MCP only).
17 ## On the web (#7580)
18
19 Three doors, one body (`src/web/pages/data_view.ts`): `/data` (every file you may see), `/admin/data` (the forge's own database, admins; the Forge data button on `/data`) and `/:owner/:name/data` (an app's Data tab for its writers: environments → files → tables → rows; a reader gets the 404). Rows sort by header (`sort`, `dir`), filter by text on one column or all (`q`, `col`), JSON cells are pretty-printed, and a copy button per row puts it on the clipboard as JSON. Sealed columns show `••••••` and cannot be sorted or searched. The SQL box is a GET form (`sql`, capped at 4,000 characters), so a query is an address to share; it shows the rows and the elapsed ms. The sorted column carries `aria-sort` and a muted arrow from CSS; a wide grid scrolls sideways only. Not done: saved queries, export, editing rows.
src/data/sql_window.ts added +25
@@ -0,0 +1,25 @@
1 /**
2 * How often one signed-in person may run SQL in the data browser, on either
3 * door. A query is an address (`?sql=`), so a reload, a link prefetcher or a
4 * crawler with a session would re-run it; the budget is per identity, in
5 * memory, and a refused run costs nothing. The work of one run is bounded
6 * separately by `query` (5 s, 1,000 rows, 2 workers).
7 */
8 import type { App } from "../app.ts";
9 import { RateLimiter } from "../core/ratelimit.ts";
10
11 /** Runs a minute, per person. */
12 export const SQL_PER_MINUTE = 20;
13
14 const windows = new WeakMap<App, RateLimiter>();
15
16 /** Seconds to wait when refused, or undefined when this run may go on. */
17 export function sqlWindow(app: App, userId: string): number | undefined {
18 let limiter = windows.get(app);
19 if (limiter === undefined) {
20 limiter = new RateLimiter({ perMinute: SQL_PER_MINUTE, adminPerMinute: SQL_PER_MINUTE, enabled: true });
21 windows.set(app, limiter);
22 }
23 const v = limiter.take(`sql:${userId}`, "ro");
24 return v.allowed ? undefined : Math.max(1, v.retryAfter);
25 }
src/http/server.ts +3 −1
@@ -27,7 +27,7 @@ import { prerenderComments } from "../web/prerender.ts";
2727 import { systemPage, systemStream } from "../web/pages/system.ts";
2828 import { contributePage } from "../web/pages/contribute.ts";
2929 import { pipelinePage } from "../web/pages/pipeline.ts";
30 import { dataPage } from "../web/pages/data.ts";
30 import { adminDataPage, appDataPage, dataPage } from "../web/pages/data.ts";
3131 import { fleetAct, fleetReceipt, fleetUndo } from "../web/pages/fleet-ops.ts";
3232 import { maintenancePage, maintenancePrepare, maintenanceResume } from "../web/pages/maintenance.ts";
3333 import { nightlyOutputPage, resiliencePage } from "../web/pages/resilience.ts";
@@ -997,6 +997,7 @@ export function buildRouter(): Router<Handler> {
997997 router.add("GET", "/boards/:slug", boardShow);
998998 router.add("GET", "/boards/:slug/live", boardLive);
999999 router.add("POST", "/boards/:slug/:op", boardPost);
1000 router.add("GET", "/admin/data", adminDataPage);
10001001 router.add("GET", "/admin/limits", limitsPage);
10011002 router.add("GET", "/admin/flows", allFlowsPage);
10021003 router.add("POST", "/admin/limits", limitsSet);
@@ -1305,6 +1306,7 @@ export function buildRouter(): Router<Handler> {
13051306 router.add("POST", "/:owner/:name/app/:env/redeploy", appRedeploy);
13061307 router.add("POST", "/:owner/:name/app/:env/stop", appStop);
13071308 router.add("GET", "/:owner/:name/events", eventsList);
1309 router.add("GET", "/:owner/:name/data", appDataPage);
13081310 router.add("GET", "/:owner/:name/runs", runsList);
13091311 router.add("GET", "/:owner/:name/runs/:id", runPage);
13101312 router.add("POST", "/:owner/:name/runs/:id/cancel", runCancel);
src/tui/screens/data.ts +96 −9
@@ -2,15 +2,21 @@
22 * `/data` in the terminal: the SQLite files the viewer may see, a file's
33 * tables, a table's rows 50 a page, secrets masked. The same reads, and the
44 * same gate (`maySeeData`, `listSources`), as the web page and MCP `data`.
5 * `s` sorts a table by a column (`-name` for descending), `f` filters it
6 * (`text`, or `column=text`), `c` clears both; `x` on a file's tables runs one
7 * read-only SELECT, on the page's per-person budget (`sqlWindow`).
58 */
69
7 import { PAGE_SIZE, rows, tables } from "../../data/browse.ts";
10 import { PAGE_SIZE, rows, tables, type RowsOptions } from "../../data/browse.ts";
11 import { query } from "../../data/query.ts";
812 import { listSources, maySeeData, type Source } from "../../data/sources.ts";
13 import { sqlWindow } from "../../data/sql_window.ts";
914 import type { Ctx, Screen } from "../app/app.ts";
1015 import { binding } from "../app/keys.ts";
1116 import type { Session } from "../session.ts";
1217 import { style } from "../term/style.ts";
1318 import { truncate } from "../term/width.ts";
19 import { later, promptScreen } from "./common.ts";
1420
1521 /** A list of rows to move through; `open` is what Enter does on one. */
1622 function listScreen(
@@ -54,18 +60,45 @@ function listScreen(
5460
5561 const cellText = (v: string | number | null): string => (v === null ? "" : String(v).replace(/[\r\n\t]+/g, " ").replace(/[\u0000-\u001f\u007f]/g, " "));
5662
63 /** Splits `-name` / `name` into a sort; `undefined` for nothing. */
64 const parseSort = (text: string): RowsOptions["order"] | undefined => {
65 const t = text.trim();
66 if (t === "") return undefined;
67 return t.startsWith("-") ? { column: t.slice(1), dir: "desc" } : { column: t, dir: "asc" };
68 };
69
70 /** `text` or `column=text`; an empty string is no filter. */
71 const parseFilter = (text: string, columns: readonly string[]): RowsOptions["filter"] | undefined => {
72 const eq = text.indexOf("=");
73 const head = eq > 0 ? text.slice(0, eq).trim() : "";
74 if (head !== "" && columns.includes(head)) return { column: head, text: text.slice(eq + 1) };
75 return text === "" ? undefined : { text };
76 };
77
5778 function rowsScreen(ctx: Ctx, source: Source, table: string): Screen {
5879 let page = 0;
59 const read = () => rows(source, table, { page });
60 return listScreen(
80 let order: RowsOptions["order"] | undefined;
81 let filter: RowsOptions["filter"] | undefined;
82 let note = "";
83 const read = () => {
84 const wanted = { page, ...(order ? { order } : {}), ...(filter ? { filter } : {}) };
85 const r = rows(source, table, wanted);
86 if (r || (!order && !filter)) return r;
87 // An unknown or sealed column cannot be sorted or searched: show the table, say so.
88 note = " · that column cannot be sorted or searched";
89 return rows(source, table, { page });
90 };
91 const screen: Screen = listScreen(
6192 ctx,
6293 `${table} · ${source.label}`,
6394 "",
6495 () => {
96 note = "";
6597 const r = read();
6698 if (!r) return [];
99 const how = `${order ? ` · by ${order.column} ${order.dir}` : ""}${filter ? ` · filter ${filter.column ? `${filter.column}=` : ""}${filter.text}` : ""}`;
67100 return [
68 `${r.total} rows, page ${r.page + 1} · ${r.columns.join(" | ")}`,
101 `${r.total} rows, page ${r.page + 1}${how}${note} · ${r.columns.join(" | ")}`,
69102 ...r.rows.map((row) => row.map(cellText).join(" | ")),
70103 ];
71104 },
@@ -75,16 +108,48 @@ function rowsScreen(ctx: Ctx, source: Source, table: string): Screen {
75108 if (!r) return false;
76109 if (key === "n" && (r.page + 1) * PAGE_SIZE < r.total) page = r.page + 1;
77110 else if (key === "p" && r.page > 0) page = r.page - 1;
78 else return false;
111 else if (key === "c") {
112 order = undefined;
113 filter = undefined;
114 page = 0;
115 } else if (key === "s" || key === "f") {
116 const sorting = key === "s";
117 ctx.push(
118 promptScreen(ctx, screen, {
119 title: sorting ? "Sort by" : "Filter",
120 label: sorting ? "Column" : "Text",
121 placeholder: sorting ? "name, or -name for descending" : "text, or column=text",
122 onSubmit: (v) => {
123 if (sorting) order = parseSort(v);
124 else filter = parseFilter(v.trim().slice(0, 200), r.columns);
125 page = 0;
126 return undefined;
127 },
128 }),
129 );
130 } else return false;
79131 return true;
80132 },
81 [binding(["n", "p"], "next / previous page")],
133 [binding(["n", "p"], "next / previous page"), binding(["s", "f", "c"], "sort / filter / clear")],
82134 );
135 return screen;
83136 }
84137
85 function tablesScreen(ctx: Ctx, source: Source): Screen {
138 /** One SELECT's rows as lines, or the reason it was refused. */
139 async function sqlLines(source: Source, sql: string): Promise<string[]> {
140 const t0 = performance.now();
141 const r = await query(source, sql);
142 const ms = Math.max(1, Math.round(performance.now() - t0));
143 if (!r.ok) return [`${r.message} · ${ms} ms`];
144 return [
145 `${r.rows.length} rows${r.truncated ? " (cut)" : ""} · ${ms} ms · ${r.columns.join(" | ")}`,
146 ...r.rows.map((row) => row.map(cellText).join(" | ")),
147 ];
148 }
149
150 function tablesScreen(ctx: Ctx, session: Session, source: Source): Screen {
86151 const list = tables(source);
87 return listScreen(
152 const screen: Screen = listScreen(
88153 ctx,
89154 source.label,
90155 "Tables; secrets are masked.",
@@ -93,7 +158,29 @@ function tablesScreen(ctx: Ctx, source: Source): Screen {
93158 const t = list[i];
94159 if (t) ctx.push(rowsScreen(ctx, source, t.name));
95160 },
161 (key) => {
162 if (key !== "x") return false;
163 ctx.push(
164 promptScreen(ctx, screen, {
165 title: "SQL",
166 label: "SELECT",
167 placeholder: "SELECT * FROM … (read-only, one statement)",
168 onSubmit: (sql) => {
169 const text = sql.trim().slice(0, 4000);
170 if (text === "") return "Type one SELECT.";
171 const wait = session.user ? sqlWindow(session.app, session.user.id) : 1;
172 if (wait !== undefined) return `Too many queries; try again in ${wait} s.`;
173 return sqlLines(source, text).then((lines) => {
174 void later(ctx, () => listScreen(ctx, "SQL", "", () => lines, () => undefined));
175 });
176 },
177 }),
178 );
179 return true;
180 },
181 [binding(["x"], "run SQL")],
96182 );
183 return screen;
97184 }
98185
99186 export function dataScreen(session: Session, ctx: Ctx): Screen {
@@ -105,7 +192,7 @@ export function dataScreen(session: Session, ctx: Ctx): Screen {
105192 () => sources().map((s) => s.label),
106193 (i) => {
107194 const s = sources()[i];
108 if (s) ctx.push(tablesScreen(ctx, s));
195 if (s) ctx.push(tablesScreen(ctx, session, s));
109196 },
110197 );
111198 }
src/web/pages/data.ts +67 −73
@@ -1,87 +1,81 @@
11 /**

Both doors: no src/tui/ screen change accompanies this. The terminal needs the app-scoped tab, /admin/data for admins, sort/filter and the read-only SQL box, gated identically.

2 * /data: the SQLite files the viewer may see (`data/sources.ts`), the
3 * tables in one, and a page of a table's rows, secrets masked. Anyone with
4 * no door to the data (a guest, a reader) gets the not-found page; an admin
5 * or an app's writer gets the page, with an empty state until a file exists.
2 * The data browser's three doors (body in `data_view.ts`):
3 * - `/data`: every SQLite file the viewer may see (an admin, or an app's writer).
4 * - `/admin/data`: the forge's own database, admins only.
5 * - `/:owner/:name/data`: one app's databases, environment by environment, for
6 * the people who can write the app. Anyone else gets the one 404.
67 */
78
8 import { PAGE_SIZE, rows, tables } from "../../data/browse.ts";
9 import { findSource, listSources, maySeeData } from "../../data/sources.ts";
9 import { getAppByRepo } from "../../core/apps.ts";
10 import { canWrite } from "../../core/repos.ts";
11 import { sqlWindow } from "../../data/sql_window.ts";
12 import { listSources, maySeeData } from "../../data/sources.ts";
1013 import type { Ctx } from "../../http/context.ts";
1114 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
12 import { notFoundPage, pageHead, signInFirst } from "../chrome.ts";
13 import { html } from "../html.ts";
15 import { loadRepo, notFoundPage, pageHead, scopeHeader, signInFirst } from "../chrome.ts";
16 import { html, type Html } from "../html.ts";
1417 import { page } from "../layout.ts";
15 import { empty, pager, row } from "../ui.ts";
18 import { btn } from "../ui.ts";
19 import { localScope } from "../scope.ts";
20 import { dataBody, type DataView } from "./data_view.ts";
1621
17 const PURPOSE = "The SQLite files you may read, their tables and rows; secrets are masked.";
22 const PURPOSE = "Every SQLite file you may read; secrets are masked.";
1823
19 const href = (params: Record<string, string | number>): string =>
20 `/data?${new URLSearchParams(Object.entries(params).map(([k, v]): [string, string] => [k, String(v)]))}`;
24 /** What `page()` takes besides the body, for the doors that draw their own frame. */
25 type Frame = Partial<Pick<Parameters<typeof page>[0], "crumbs" | "palette">>;
2126
22 // GET /data[?source=<id>[&table=<name>[&page=<n>]]]
23 export function dataPage(ctx: Ctx): Response {
27 async function draw(ctx: Ctx, view: DataView, title: string, head: Html, extra: Frame = {}): Promise<Response> {
28 const rendering = performance.now();
29 const inner = await dataBody(view, ctx.url.searchParams);
30 if (inner === undefined) return refuse(ctx);
31 const response = page({ ctx, title, body: html`${head}${inner}`, ...extra });
32 notePhase(ctx.request, "render", performance.now() - rendering);
33 return response;
34 }
35
36 /** A refusal still names `render`, so every answer carries both phases. */
37 function refuse(ctx: Ctx): Response {
38 const rendering = performance.now();
39 const response = notFoundPage(ctx);
40 notePhase(ctx.request, "render", performance.now() - rendering);
41 return response;
42 }
43
44 // GET /data[?source=<id>[&table=<name>…][&sql=…]]
45 export async function dataPage(ctx: Ctx): Promise<Response> {
2446 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
2547 const viewer = ctx.auth.user;
2648 if (!viewer) return signInFirst(ctx);
2749 const { db, config } = ctx.app;
28 if (!maySeeData(db, viewer)) return notFoundPage(ctx);
29 const q = ctx.url.searchParams;
30 const all = listSources(db, config.dataDir, viewer);
31 const chosen = q.get("source");
32 const source = chosen === null ? undefined : findSource(db, config.dataDir, viewer, chosen);
33 if (chosen !== null && !source) return notFoundPage(ctx);
34
35 let body = html`${pageHead("Data", { sub: PURPOSE })}
36 ${all.length === 0
37 ? empty({
38 title: "No database yet",
39 note: "An app's SQLite files appear here once it has written one under its data directory.",
40 })
41 : html`<ul class="list">
42 ${all.map((s) => row({ href: href({ source: s.id }), title: s.label, current: s.id === source?.id }))}
43 </ul>`}`;
44
45 if (source) {
46 const list = tables(source);
47 const name = q.get("table");
48 const shown = name === null ? undefined : rows(source, name, { page: Number(q.get("page") ?? 0) });
49 if (name !== null && !shown) return notFoundPage(ctx);
50 body = html`${body}
51 <h2>${source.label}</h2>
52 <ul class="list">
53 ${list.map((t) =>
54 row({
55 href: href({ source: source.id, table: t.name }),
56 title: t.name,
57 trail: html`${String(t.rows)} rows`,
58 current: t.name === name,
59 }),
60 )}
61 </ul>`;
62 if (shown && name !== null) {
63 const more = (shown.page + 1) * PAGE_SIZE < shown.total;
64 body = html`${body}
65 <h2>${name}</h2>
66 <div class="table-scroll">
67 <table class="table" data-dense>
68 <caption class="sr-only">${name} in ${source.label}</caption>
69 <thead><tr>${shown.columns.map((c) => html`<th scope="col">${c}</th>`)}</tr></thead>
70 <tbody>
71 ${shown.rows.map((r) => html`<tr>${r.map((v) => html`<td>${v === null ? "" : String(v)}</td>`)}</tr>`)}
72 </tbody>
73 </table>
74 </div>
75 ${pager({
76 note: `${shown.total} rows`,
77 words: { prev: "Previous page", next: "Next page" },
78 ...(shown.page > 0 ? { prev: href({ source: source.id, table: name, page: shown.page - 1 }) } : {}),
79 ...(more ? { next: href({ source: source.id, table: name, page: shown.page + 1 }) } : {}),
80 })}`;
81 }
82 }
83 const rendering = performance.now();
84 const response = page({ ctx, title: "Data", body });
85 notePhase(ctx.request, "render", performance.now() - rendering);
86 return response;
50 if (!maySeeData(db, viewer)) return refuse(ctx);
51 const sources = listSources(db, config.dataDir, viewer);
52 return draw(ctx, { path: "/data", sources, sqlWait: () => sqlWindow(ctx.app, viewer.id) }, "Data", pageHead("Data", { sub: PURPOSE, ...(viewer.isAdmin ? { actions: btn("Forge data", { href: "/admin/data", size: "sm" }) } : {}) }));
53 }
54
55 // GET /admin/data — the forge's own database (the admin gate has already refused non-admins).
56 export async function adminDataPage(ctx: Ctx): Promise<Response> {
57 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
58 const viewer = ctx.auth.user;
59 if (!viewer) return signInFirst(ctx);
60 if (!viewer.isAdmin) return refuse(ctx);
61 const sources = listSources(ctx.app.db, ctx.app.config.dataDir, viewer).filter((s) => s.kind === "forge");
62 return draw(ctx, { path: "/admin/data", sources, sqlWait: () => sqlWindow(ctx.app, viewer.id) }, "Data · forge", pageHead("Forge data", { sub: "Secrets are masked; queries are read-only." }));
63 }
64
65 // GET /:owner/:name/data — an app's databases, for its writers.
66 export async function appDataPage(ctx: Ctx): Promise<Response> {
67 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
68 const repo = loadRepo(ctx);
69 const record = repo && getAppByRepo(ctx.app.db, repo.id);
70 if (!repo || !record || !ctx.auth.user || !canWrite(ctx.app.db, repo, ctx.auth.user)) return refuse(ctx);
71 const viewer = ctx.auth.user;
72 const sources = listSources(ctx.app.db, ctx.app.config.dataDir, viewer).filter((s) => s.id.startsWith(`app:${record.id}:`));
73 const scope = localScope(ctx, repo);
74 return draw(
75 ctx,
76 { path: `/${repo.ownerName}/${repo.name}/data`, sources, byEnv: true, sqlWait: () => sqlWindow(ctx.app, viewer.id) },
77 `Data · ${repo.name}`,
78 scopeHeader(ctx, scope, "data"),
79 { crumbs: scope.crumbs, palette: scope.palette },
80 );
8781 }
src/web/pages/data_view.ts added +218
@@ -0,0 +1,218 @@
1 /**

Tests: ?sql= re-runs a query on every GET with no per-identity rate limit (DoS). Add a limit keyed on the signed-in user, plus a test. Also add tests for forge-db masking via SQL and rows, cellView JSON, and the SQL cap. Hand-rolled <select> and glyph sort marks: use the named components (DESIGN principle 8/7).

2 * The data browser's body, shared by its three doors: `/data` (every file the
3 * viewer may see), `/admin/data` (the forge's own) and `/:owner/:name/data`
4 * (one app's, environment by environment). A door picks the sources and the
5 * address; this draws files → tables → rows (sort, one-column filter, JSON
6 * cells, copy row as JSON) and the read-only SQL box. Everything is a GET, so
7 * every view, SQL included, is an address that can be shared.
8 */
9
10 import { PAGE_SIZE, rows, tables, type RowsResult } from "../../data/browse.ts";
11 import { query } from "../../data/query.ts";
12 import type { Source } from "../../data/sources.ts";
13 import { html, type Html } from "../html.ts";
14 import { btn, empty, field, iconBtn, pager, row } from "../ui.ts";
15
16 /** Longest SQL an address may carry. */
17 export const SQL_MAX = 4000;
18
19 type Cell = string | number | null;
20
21 export interface DataView {
22 /** The path every link points at, without a query. */
23 readonly path: string;
24 readonly sources: readonly Source[];
25 /** Group file rows under their environment (an app's own tab). */
26 readonly byEnv?: boolean;
27 /** Asked before a SQL run: seconds to wait when this person has run too many, else undefined. */
28 readonly sqlWait?: () => number | undefined;
29 }
30
31 const url = (path: string, params: Record<string, string | number | undefined>): string => {
32 const q = new URLSearchParams();
33 for (const [k, v] of Object.entries(params)) if (v !== undefined && v !== "") q.set(k, String(v));
34 const s = q.toString();
35 return s === "" ? path : `${path}?${s}`;
36 };
37
38 /** A cell: JSON text pretty-printed, null left empty, anything else as text. */
39 export function cellView(v: Cell): Html {
40 if (v === null) return html``;
41 const text = String(v);
42 if (typeof v === "string" && (text.startsWith("{") || text.startsWith("["))) {
43 try {
44 const parsed: unknown = JSON.parse(text);
45 if (typeof parsed === "object" && parsed !== null)
46 return html`<pre class="mono">${JSON.stringify(parsed, null, 2)}</pre>`;
47 } catch {
48 // not JSON after all: plain text below
49 }
50 }
51 return html`${text}`;
52 }
53
54 const asObject = (columns: readonly string[], r: readonly Cell[]): Record<string, Cell> =>
55 Object.fromEntries(columns.map((c, i): [string, Cell] => [c, r[i] ?? null]));
56
57 function grid(
58 columns: readonly string[],
59 data: readonly (readonly Cell[])[],
60 caption: string,
61 head: (c: string) => Html,
62 copyIds: boolean,
63 ): Html {
64 return html`<div class="table-scroll">
65 <table class="table" data-dense data-sortable>
66 <caption class="sr-only">${caption}</caption>
67 <thead><tr>${copyIds ? html`<th scope="col"><span class="sr-only">Copy</span></th>` : null}${columns.map(head)}</tr></thead>
68 <tbody>
69 ${data.map(
70 (r, i) => html`<tr>
71 ${copyIds
72 ? html`<td>${iconBtn("copy", "Copy row as JSON", { copy: `row-${i}`, size: "sm" })}<pre id="row-${i}" hidden>${JSON.stringify(asObject(columns, r), null, 2)}</pre></td>`
73 : null}
74 ${r.map((v) => html`<td>${cellView(v)}</td>`)}
75 </tr>`,
76 )}
77 </tbody>
78 </table>
79 </div>`;
80 }
81
82 const intOr0 = (s: string | null): number => {
83 const n = Number(s ?? 0);
84 return Number.isInteger(n) && n > 0 ? n : 0;
85 };
86
87 /** The body for these sources and this query string, or `undefined` for a 404. */
88 export async function dataBody(view: DataView, q: URLSearchParams): Promise<Html | undefined> {
89 const { path, sources } = view;
90 const chosen = q.get("source") ?? (sources.length === 1 ? sources[0]!.id : null);
91 const source = chosen === null ? undefined : sources.find((s) => s.id === chosen);
92 if (chosen !== null && !source) return undefined;
93
94 const fileRow = (s: Source) =>
95 row({ href: url(path, { source: s.id }), title: view.byEnv ? (s.id.split(":").slice(3).join(":") || s.label) : s.label, current: s.id === source?.id });
96 const envOf = (s: Source) => s.id.split(":")[2] ?? "";
97 const files =
98 sources.length === 0
99 ? empty({
100 title: "No database yet",
101 note: "An app's SQLite files appear here once it has written one under its data directory.",
102 })
103 : view.byEnv
104 ? html`${[...new Set(sources.map(envOf))].map(
105 (env) => html`<h2>${env}</h2><ul class="list">${sources.filter((s) => envOf(s) === env).map(fileRow)}</ul>`,
106 )}`
107 : sources.length === 1 && source
108 ? html``
109 : html`<ul class="list">${sources.map(fileRow)}</ul>`;
110 if (!source) return files;
111
112 const tableName = q.get("table");
113 const sql = (q.get("sql") ?? "").slice(0, SQL_MAX);
114 const list = tables(source);
115 let out = html`${files}
116 <h2>${source.label}</h2>
117 <ul class="list">
118 ${list.map((t) =>
119 row({
120 href: url(path, { source: source.id, table: t.name }),
121 title: t.name,
122 trail: html`${String(t.rows)} rows`,
123 current: t.name === tableName,
124 }),
125 )}
126 </ul>`;
127
128 if (tableName !== null) {
129 const sort = q.get("sort") ?? undefined;
130 const dir = q.get("dir") === "desc" ? "desc" : "asc";
131 const text = (q.get("q") ?? "").slice(0, 200);
132 const col = q.get("col") || undefined;
133 const pageNo = intOr0(q.get("page"));
134 const base = { source: source.id, table: tableName, sort, dir: sort ? dir : undefined, q: text, col };
135 let shown: RowsResult | undefined = rows(source, tableName, {
136 page: pageNo,
137 ...(sort ? { order: { column: sort, dir } } : {}),
138 ...(text !== "" ? { filter: { text, ...(col ? { column: col } : {}) } } : {}),
139 });
140 let note: string | null = null;
141 if (!shown && (sort || text !== "")) {
142 // An unknown or sealed column cannot be sorted or searched: show the table, say so.
143 shown = rows(source, tableName, { page: pageNo });
144 note = "That column cannot be sorted or searched here.";
145 }
146 if (!shown) return undefined;
147 const cols = shown.columns;
148 const more = (shown.page + 1) * PAGE_SIZE < shown.total;
149 const header = (c: string): Html => {
150 const next = sort === c && dir === "asc" ? "desc" : "asc";
151 return html`<th scope="col" ${sort === c ? html`aria-sort="${dir === "asc" ? "ascending" : "descending"}"` : null}><a href="${url(path, { ...base, sort: c, dir: next })}">${c}</a></th>`;
152 };
153 out = html`${out}
154 <h2>${tableName}</h2>
155 <form method="get" action="${path}" role="search">
156 <input type="hidden" name="source" value="${source.id}" />
157 <input type="hidden" name="table" value="${tableName}" />
158 ${sort ? html`<input type="hidden" name="sort" value="${sort}" /><input type="hidden" name="dir" value="${dir}" />` : null}
159 ${field({
160 name: "q",
161 value: text,
162 label: "Filter",
163 labelHidden: true,
164 placeholder: "Filter rows",
165 type: "search",
166 })}
167 ${field({
168 name: "col",
169 type: "select",
170 label: "Column",
171 labelHidden: true,
172 options: [{ value: "", label: "Any column" }, ...cols.map((c) => ({ value: c, label: c, selected: c === col }))],
173 })}
174 <div class="form-actions">
175 ${btn("Filter", { size: "sm", type: "submit" })}
176 ${text !== "" ? btn("Clear", { href: url(path, { source: source.id, table: tableName, sort, dir: sort ? dir : undefined }), size: "sm", variant: "outline" }) : null}
177 </div>
178 </form>
179 ${note ? html`<p class="muted small">${note}</p>` : null}
180 ${shown.rows.length === 0 ? empty({ title: "No rows", note: text !== "" ? "Nothing matches that filter." : "This table is empty." }) : grid(cols, shown.rows, `${tableName} in ${source.label}`, header, true)}
181 ${pager({
182 note: `${shown.total} rows`,
183 words: { prev: "Previous page", next: "Next page" },
184 ...(shown.page > 0 ? { prev: url(path, { ...base, page: shown.page - 1 }) } : {}),
185 ...(more ? { next: url(path, { ...base, page: shown.page + 1 }) } : {}),
186 })}`;
187 }
188
189 out = html`${out}
190 <form method="get" action="${path}">
191 <input type="hidden" name="source" value="${source.id}" />
192 ${field({
193 name: "sql",
194 type: "textarea",
195 value: sql,
196 mono: true,
197 rows: 4,
198 placeholder: "SELECT * FROM … (read-only, one statement)",
199 spellcheck: false,
200 maxlength: SQL_MAX,
201 label: "SQL",
202 })}
203 <div class="form-actions">${btn("Run", { size: "sm", type: "submit" })}</div>
204 </form>`;
205 if (sql.trim() !== "") {
206 const wait = view.sqlWait?.();
207 if (wait !== undefined) return html`${out}<p role="alert" class="muted">Too many queries; try again in ${String(wait)} s.</p>`;
208 const t0 = performance.now();
209 const result = await query(source, sql);
210 const ms = Math.max(1, Math.round(performance.now() - t0));
211 out = result.ok
212 ? html`${out}
213 <p class="muted small">${String(result.rows.length)} rows${result.truncated ? " (cut)" : ""} · ${String(ms)} ms</p>
214 ${result.rows.length === 0 ? empty({ title: "No rows" }) : grid(result.columns, result.rows, "Query result", (c) => html`<th scope="col">${c}</th>`, false)}`
215 : html`${out}<p role="alert" class="muted">${result.message} · ${String(ms)} ms</p>`;
216 }
217 return out;
218 }
src/web/repo_steps.ts +5 −1
@@ -11,7 +11,7 @@ import { appsRuntime } from "../apps/runtime.ts";
1111 import { knownSpec, warmSpec } from "../apps/api_spec.ts";
1212 import { runningCount } from "../core/runs.ts";
1313 import { getAppByRepo } from "../core/apps.ts";
14 import { canAdmin, canEnterApp, visibleForks, type Repo } from "../core/repos.ts";
14 import { canAdmin, canEnterApp, canWrite, visibleForks, type Repo } from "../core/repos.ts";
1515 import type { Ctx } from "../http/context.ts";
1616 import { sshCommand } from "../ssh/command.ts";
1717 import { changesHref, tellsHref } from "./studio/nouns.ts";
@@ -27,6 +27,7 @@ export type RepoTab =
2727 | "app"
2828 | "runs"
2929 | "events"
30 | "data"
3031 | "flows"
3132 | "api"
3233 | "settings"
@@ -114,6 +115,9 @@ function work(ctx: Ctx, base: string, repo: Repo | undefined): Steps {
114115 ...(running > 0 ? { count: running } : {}),
115116 });
116117 }
118 // An app's databases, for the people who can write the app.
119 if (record !== undefined && canWrite(ctx.app.db, repo, ctx.auth.user))
120 steps.push({ key: "data", label: "Data", href: `${base}/data` });
117121 steps.push({ key: "events", label: "Events", href: `${base}/events` });
118122 // What flows in and out: origin, sources, pins, forks (#9434). The page gates itself like the repo.
119123 steps.push({ key: "flows", label: "Flows", href: `${base}/flows` });
src/web/style/components.ts +5
@@ -583,6 +583,11 @@ details[open] > summary.row > .row-lead { transform: rotate(90deg); }
583583 .table[data-dense] td, .table[data-dense] th { padding: 5px 10px; }
584584 .table .num { text-align: right; }
585585 .table-scroll { overflow-x: auto; scrollbar-width: thin; }
586 /* The data browser: a cell's JSON keeps its shape, and the sorted column is marked by its aria-sort. */
587 .table td pre { margin: 0; white-space: pre-wrap; overflow-wrap: anywhere; font-size: inherit; }
588 .table[data-sortable] th[aria-sort] a { color: var(--fg); }
outdated

DESIGN principle 4: one background, --bg. The sticky header uses --card; use var(--bg). Principle 6: a max-height scrolling box around the grid is a box around a thing; prefer page flow.

589 .table[data-sortable] th[aria-sort="ascending"] a::after { content: " \\2191"; color: var(--muted-fg); }
590 .table[data-sortable] th[aria-sort="descending"] a::after { content: " \\2193"; color: var(--muted-fg); }
586591 /* The row's name stays in view while a narrow screen scrolls the figures. */
587592 .table[data-sticky-lead] th:first-child,
588593 .table[data-sticky-lead] td:first-child {
test/data-door.test.ts +108
@@ -9,6 +9,7 @@ import { createApp, type App } from "../src/app.ts";
99 import { envDataDir } from "../src/apps/data.ts";
1010 import { createApp as createAppRecord } from "../src/core/apps.ts";
1111 import { createRepo } from "../src/core/repos.ts";
12 import { SQL_PER_MINUTE } from "../src/data/sql_window.ts";
1213 import { createSession } from "../src/core/sessions.ts";
1314 import { createToken } from "../src/core/tokens.ts";
1415 import { createUser } from "../src/core/users.ts";
@@ -123,4 +124,111 @@ describe("the data door", () => {
123124 expect((await mcpTool(servers.url, token["bob"]!, "data", { source: id })).error).toBeDefined();
124125 expect((await mcpTool(servers.url, token["bob"]!, "data")).value).toEqual([]);
125126 });
127
128 test("an app's Data tab: a writer sees it, a reader gets a 404, secrets stay masked", async () => {
129 const tab = await get("/alice/notes/data", "alice");
130 expect(tab.status).toBe(200);
131 const text = await tab.text();
132 expect(text).toContain('href="/alice/notes/data"');
133 expect(text).toContain("production");
134 expect(text).toContain("app.db");
135 expect((await get("/alice/notes/data", "bob")).status).toBe(404);
136 expect((await get("/alice/notes/data")).status).not.toBe(200);
137 const rows = await (await get(`/alice/notes/data?source=${src()}&table=users`, "alice")).text();
138 expect(rows).toContain("ann");
139 expect(rows).toContain("••••••");
140 expect(rows).not.toContain("tok-ann-1");
141 expect(rows).toContain('data-copy="row-0"');
142 expect(rows).toContain("&quot;name&quot;: &quot;ann&quot;");
143 });
144
145 test("another app's source is not reachable through this app's tab", async () => {
146 expect((await get(`/alice/notes/data?source=forge`, "owner1")).status).toBe(404);
147 expect((await get(`/dana/bare/data`, "alice")).status).toBe(404);
148 });
149
150 test("sort and filter narrow the rows; a sealed column cannot be searched", async () => {
151 const d = new Database(join(envDataDir(dataDir, appId, "production"), "app.db"));
152 d.exec("INSERT INTO users VALUES (2, 'zed', 'tok-zed-2')");
153 d.close();
154 const base = `/alice/notes/data?source=${src()}&table=users`;
155 const sorted = await (await get(`${base}&sort=name&dir=desc`, "alice")).text();
156 expect(sorted.indexOf("<td>zed</td>")).toBeLessThan(sorted.indexOf("<td>ann</td>"));
157 expect(sorted).toContain('aria-sort="descending"');
158 const filtered = await (await get(`${base}&q=zed&col=name`, "alice")).text();
159 expect(filtered).toContain("zed");
160 expect(filtered).not.toContain("<td>ann</td>");
161 const sealed = await (await get(`${base}&q=tok-zed&col=api_token`, "alice")).text();
162 expect(sealed).toContain("Nothing matches that filter");
163 expect(sealed).not.toContain("tok-zed-2");
164 });
165
166 test("the SQL box runs a read-only query from the address and shows rows and ms", async () => {
167 const ok = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent("SELECT name FROM users ORDER BY id")}`, "alice")).text();
168 expect(ok).toContain("ann");
169 expect(ok).toMatch(/\d+ ms/);
170 const sealed = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent("SELECT api_token FROM users")}`, "alice")).text();
171 expect(sealed).not.toContain("tok-ann-1");
172 const bad = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent("DELETE FROM users")}`, "alice")).text();
173 expect(bad).toContain("Only SELECT");
174 const d = new Database(join(envDataDir(dataDir, appId, "production"), "app.db"), { readonly: true });
175 expect(d.query("SELECT count(*) AS n FROM users").get()).toEqual({ n: 2 });
176 d.close();
177 });
178
179 test("/admin/data is the forge's own database, for admins only", async () => {
180 const res = await get("/admin/data", "owner1");
181 expect(res.status).toBe(200);
182 expect(await res.text()).toContain("forge_setting");
183 expect((await get("/admin/data", "alice")).status).toBeGreaterThanOrEqual(403);
184 expect((await get("/admin/data")).status).not.toBe(200);
185 });
186
187 test("the Forge data button is for admins alone", async () => {
188 // owner1 is the admin and can see /data (forge.db); alice is a writer; bob a reader.
189 expect(await (await get("/data", "owner1")).text()).toContain('href="/admin/data"');
190 expect(await (await get("/data", "alice")).text()).not.toContain('href="/admin/data"');
191 });
192
193 test("a JSON cell is pretty-printed and a plain one is not", async () => {
194 const d = new Database(join(envDataDir(dataDir, appId, "production"), "app.db"));
195 d.exec("CREATE TABLE IF NOT EXISTS docs (id INTEGER PRIMARY KEY, body TEXT)");
196 d.exec(`INSERT INTO docs VALUES (1, '{"a":1,"b":[2]}'), (2, '{not json'), (3, 'plain')`);
197 d.close();
198 const text = await (await get(`/alice/notes/data?source=${src()}&table=docs`, "alice")).text();
199 expect(text).toContain('<pre class="mono">{\n &quot;a&quot;: 1,');
200 expect(text).toContain("<td>{not json</td>");
201 expect(text).toContain("<td>plain</td>");
202 });
203
204 test("the SQL in an address is cut at 4,000 characters", async () => {
205 const long = `SELECT 1 AS a ${"y".repeat(5000)} tail-marker`;
206 const text = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent(long)}`, "alice")).text();
207 expect(text).not.toContain("tail-marker");
208 const inBox = text.match(/<textarea[^>]*>([\s\S]*?)<\/textarea\s*>/)![1]!;
209 expect(inBox.trim().length).toBe(4000);
210 });
211
212 test("the forge database's secrets are masked in rows and in SQL on /admin/data", async () => {
213 const hash = app.db.query<{ h: string }, []>("SELECT password_hash AS h FROM user LIMIT 1").get()!.h;
214 const rowsHtml = await (await get("/admin/data?table=user", "owner1")).text();
215 expect(rowsHtml).toContain("alice");
216 expect(rowsHtml).toContain("••••••");
217 expect(rowsHtml).not.toContain(hash);
218 const sqlHtml = await (await get(`/admin/data?sql=${encodeURIComponent("SELECT username, password_hash AS p FROM user")}`, "owner1")).text();
219 expect(sqlHtml).toContain("alice");
220 expect(sqlHtml).not.toContain(hash);
221 const aliased = await (await get(`/admin/data?sql=${encodeURIComponent("SELECT * FROM (SELECT password_hash AS q FROM user)")}`, "owner1")).text();
222 expect(aliased).not.toContain(hash);
223 });
224
225 test("SQL runs are limited per person; another person is not held back", async () => {
226 const q = `/alice/notes/data?source=${src()}&sql=${encodeURIComponent("SELECT 1 AS one")}`;
227 let refused = 0;
228 for (let i = 0; i < SQL_PER_MINUTE + 3; i++) if ((await (await get(q, "alice")).text()).includes("Too many queries")) refused++;
229 expect(refused).toBeGreaterThanOrEqual(1);
230 expect(await (await get("/admin/data?sql=SELECT%201", "owner1")).text()).not.toContain("Too many queries");
231 // Browsing a table is not SQL and is never held.
232 expect(await (await get(`/alice/notes/data?source=${src()}&table=users`, "alice")).text()).not.toContain("Too many queries");
233 });
126234 });
test/tui-data.test.ts +63
@@ -12,6 +12,8 @@ import { createRepo } from "../src/core/repos.ts";
1212 import { createUser, getUserByName } from "../src/core/users.ts";
1313 import { forgeCommands } from "../src/tui/forge.ts";
1414 import { createForgeApp } from "../src/tui/forge.ts";
15 import { query } from "../src/data/query.ts";
16 import { listSources } from "../src/data/sources.ts";
1517 import { dataScreen } from "../src/tui/screens/data.ts";
1618 import { headlessTerminal } from "../src/tui/term/terminal.ts";
1719 import { quietLogs } from "./helpers/globals.ts";
@@ -21,6 +23,8 @@ quietLogs();
2123
2224 let dir: string;
2325 let app: App;
26 let appRecordId = "";
27 const appIdOf = () => appRecordId;
2428
2529 beforeAll(async () => {
2630 dir = await mkdtemp(join(tmpdir(), "forge-tui-data-"));
@@ -35,6 +39,7 @@ beforeAll(async () => {
3539 if (!repo.ok) throw new Error(repo.error);
3640 const rec = createAppRecord(app.db, { repoId: repo.value.id, slug: "notes", access: "public" });
3741 if (!rec.ok) throw new Error(rec.error);
42 appRecordId = rec.value.id;
3843 const where = envDataDir(dir, rec.value.id, "production");
3944 mkdirSync(where, { recursive: true });
4045 const d = new Database(join(where, "app.db"));
@@ -92,3 +97,61 @@ test("an admin sees forge.db among the files", async () => {
9297 expect(tui.screen.toText()).toContain("forge");
9398 tui.stop?.();
9499 });
100
101 const press = async (tui: ReturnType<typeof open>["app"], ...keys: string[]) => {
102 for (const key of keys) {
103 tui.dispatch({ type: "key", key, ctrl: false, alt: false, shift: false });
104 await settleApp(tui);
105 }
106 };
107 const type = (tui: ReturnType<typeof open>["app"], text: string) => press(tui, ...text.split(""));
108
109 test("a writer sorts and filters a table, and runs read-only SQL with secrets masked", async () => {
110 const d = new Database(join(envDataDir(dir, appIdOf(), "production"), "app.db"));
111 d.exec("INSERT INTO users VALUES (2, 'zed-visible', 'tok-secret-2')");
112 d.close();
113 const { app: tui, session } = open("alice");
114 tui.start();
115 tui.ctx.push(dataScreen(session, tui.ctx));
116 await settleApp(tui);
117 await press(tui, "enter");
118 await press(tui, "x");
119 await type(tui, "SELECT api_token, name FROM users");
120 await press(tui, "enter");
121 await new Promise((r) => setTimeout(r, 300));
122 await settleApp(tui);
123 let text = tui.screen.toText();
124 expect(text).toContain("zed-visible");
125 expect(text).not.toContain("tok-secret");
126 await press(tui, "esc", "enter");
127 await press(tui, "s");
128 await type(tui, "-name");
129 await press(tui, "enter");
130 text = tui.screen.toText();
131 expect(text.indexOf("zed-visible")).toBeLessThan(text.indexOf("ann-visible"));
132 await press(tui, "f");
133 await type(tui, "name=ann");
134 await press(tui, "enter");
135 text = tui.screen.toText();
136 expect(text).toContain("ann-visible");
137 expect(text).not.toContain("zed-visible");
138 tui.stop?.();
139 });
140
141 test("a reader and a guest get no files, so no SQL either", () => {
142 for (const who of ["bob", undefined]) {
143 const { app: tui, session } = open(who);
144 expect(listSources(app.db, dir, session.user)).toEqual([]);
145 tui.stop?.();
146 }
147 });
148
149 test("the forge database is among an admin's files, and SQL on it is masked", async () => {
150 const forge = listSources(app.db, dir, getUserByName(app.db, "owner1")!).find((s) => s.kind === "forge")!;
151 const r = await query(forge, "SELECT password_hash FROM user");
152 expect(r.ok).toBe(true);
153 const hash = app.db.query<{ h: string }, []>("SELECT password_hash AS h FROM user LIMIT 1").get()!.h;
154 expect(hash.length).toBeGreaterThan(10);
155 expect(JSON.stringify(r)).not.toContain(hash);
156 expect(JSON.stringify(r)).toContain("••••••");
157 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs