Public: anyone can read this; the crew can change it.
A Data tab: every app's databases in the web, and the forge's own at /admin/data — table list, sortable paged rows, filters, a read-only SQL box #10314
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/7580-a-data-tab-every-app-s-databases-in-the → main 5 commits fork 3 threads · 3 unresolved
docs/design/areas/table.md +2
| @@ -3,3 +3,5 @@ | ||
| 3 | 3 | A §5 `.table` too wide for a narrow screen sits inside a horizontal scroller, so the page itself never scrolls sideways and only the table's figures move. |
| 4 | 4 | |
| 5 | 5 | - `.table-scroll` — the wrapper `<div>` around a wide `.table`: `overflow-x: auto` with a thin scrollbar, styled in `src/web/style/components.ts`. Pair it with `.table[data-sticky-lead]` to keep each row's name in view while the figures scroll. Written by `src/web/pages/account.ts`, `src/web/pages/audit.ts`, `src/web/pages/boards-held.ts`, `src/web/pages/crew.ts`, `src/web/pages/flow-panel.ts`, `src/web/pages/keeper-panel.ts`, `src/web/pages/passkeys.ts`, `src/web/pages/proposal-tokens.ts`, `src/web/pages/settings.ts`, `src/web/pages/spend.ts`, `src/web/pages/subscriptions.ts`, `src/web/pages/system.ts`, `fleetGraphSection`'s figures table in `src/web/fleet-graph.ts`, and `vitalsTable` in `src/web/studio/vitals.ts`. |
| 6 | ||
| 7 | - The data browser (#7580, `src/web/pages/data_view.ts`) adds no class. A sortable `.table` carries `data-sortable`; its sorted header has `aria-sort`, and `style/components.ts` draws a muted arrow after the link from that attribute. A cell's JSON is a `<pre>` inside the `td`. The grid scrolls sideways only; 50 rows a page keep it in the page's flow, so there is no max-height box and no sticky header. |
docs/design/audit.md +1
| @@ -104,6 +104,7 @@ Machine routes are not pages and have no entry: `/audit.csv`, | ||
| 104 | 104 | - `/maintenance` — `pages/maintenance.ts` · prepare the forge for a power-off and end it · `/resilience`, `/settings/backups` · none seen. |
| 105 | 105 | - `/admin/limits` — `pages/admin-limits.ts` · every limit with value, default and reason · `/settings/capacity` (also a numbers-you-can-change page) · none seen. |
| 106 | 106 | - `/admin/flows` — `pages/flows-all.ts` · every repository's flow lines with state and last moved, by kind and stale · `/:owner/:name/flows` · none seen. |
| 107 | - `/admin/data` — `pages/data.ts` · the forge's own database: tables, rows, read-only SQL, secrets masked · `/data` (the Forge data button) · none seen; shares its body (`pages/data_view.ts`) with `/data` and `/:owner/:name/data`. | |
| 107 | 108 | - `/settings/economics` — `pages/economics.ts` · the profit slider's position and the margin, with the policy in words · `/admin/limits`, `/settings/capacity` · none seen. |
| 108 | 109 | - `/settings/capacity` — `pages/capacity.ts` · how many runs and checks go at once · `/admin/limits`, `/settings/usage` · none seen. |
| 109 | 110 | - `/settings/usage` — `pages/usage.ts` · route patterns by use, unused first · `/settings/telemetry`, `/board/returns` · none seen. Not about cost, despite the name. |
docs/features/data-browser.md +4 −2
| @@ -12,6 +12,8 @@ Views are masked whole when their text mentions a masked column (an alias cannot | ||
| 12 | 12 | |
| 13 | 13 | Limits on work: `query` runs at most 2 workers at once (a third is told it is busy), each with a 128 MB heap cap (`hard_heap_limit`), a 5 s timer, 1,000 rows and about 250 KB of result; every cell is cut to 1,000 characters inside SQLite and a blob shows as its size. `rows()` cuts cells the same way. |
| 14 | 14 | |
| 15 | Terminal: `src/tui/screens/data.ts` (files → tables → rows, `n`/`p` for pages), offered by `maySeeData`. | |
| 15 | Terminal: `src/tui/screens/data.ts` (files → tables → rows, `n`/`p` for pages, `s` sort, `f` filter, `c` clear, `x` read-only SQL), offered by `maySeeData`; the forge's own database is among an admin's files. SQL on either door is 20 runs a minute per person (`src/data/sql_window.ts`); beyond that it says when to retry. | |
| 16 | 16 | |
| 17 | Not done: a SQL editor on the page or in the terminal (SQL is MCP only). | |
| 17 | ## On the web (#7580) | |
| 18 | ||
| 19 | Three doors, one body (`src/web/pages/data_view.ts`): `/data` (every file you may see), `/admin/data` (the forge's own database, admins; the Forge data button on `/data`) and `/:owner/:name/data` (an app's Data tab for its writers: environments → files → tables → rows; a reader gets the 404). Rows sort by header (`sort`, `dir`), filter by text on one column or all (`q`, `col`), JSON cells are pretty-printed, and a copy button per row puts it on the clipboard as JSON. Sealed columns show `••••••` and cannot be sorted or searched. The SQL box is a GET form (`sql`, capped at 4,000 characters), so a query is an address to share; it shows the rows and the elapsed ms. The sorted column carries `aria-sort` and a muted arrow from CSS; a wide grid scrolls sideways only. Not done: saved queries, export, editing rows. |
src/data/sql_window.ts added +25
| @@ -0,0 +1,25 @@ | ||
| 1 | /** | |
| 2 | * How often one signed-in person may run SQL in the data browser, on either | |
| 3 | * door. A query is an address (`?sql=`), so a reload, a link prefetcher or a | |
| 4 | * crawler with a session would re-run it; the budget is per identity, in | |
| 5 | * memory, and a refused run costs nothing. The work of one run is bounded | |
| 6 | * separately by `query` (5 s, 1,000 rows, 2 workers). | |
| 7 | */ | |
| 8 | import type { App } from "../app.ts"; | |
| 9 | import { RateLimiter } from "../core/ratelimit.ts"; | |
| 10 | ||
| 11 | /** Runs a minute, per person. */ | |
| 12 | export const SQL_PER_MINUTE = 20; | |
| 13 | ||
| 14 | const windows = new WeakMap<App, RateLimiter>(); | |
| 15 | ||
| 16 | /** Seconds to wait when refused, or undefined when this run may go on. */ | |
| 17 | export function sqlWindow(app: App, userId: string): number | undefined { | |
| 18 | let limiter = windows.get(app); | |
| 19 | if (limiter === undefined) { | |
| 20 | limiter = new RateLimiter({ perMinute: SQL_PER_MINUTE, adminPerMinute: SQL_PER_MINUTE, enabled: true }); | |
| 21 | windows.set(app, limiter); | |
| 22 | } | |
| 23 | const v = limiter.take(`sql:${userId}`, "ro"); | |
| 24 | return v.allowed ? undefined : Math.max(1, v.retryAfter); | |
| 25 | } |
src/http/server.ts +3 −1
| @@ -27,7 +27,7 @@ import { prerenderComments } from "../web/prerender.ts"; | ||
| 27 | 27 | import { systemPage, systemStream } from "../web/pages/system.ts"; |
| 28 | 28 | import { contributePage } from "../web/pages/contribute.ts"; |
| 29 | 29 | import { pipelinePage } from "../web/pages/pipeline.ts"; |
| 30 | import { dataPage } from "../web/pages/data.ts"; | |
| 30 | import { adminDataPage, appDataPage, dataPage } from "../web/pages/data.ts"; | |
| 31 | 31 | import { fleetAct, fleetReceipt, fleetUndo } from "../web/pages/fleet-ops.ts"; |
| 32 | 32 | import { maintenancePage, maintenancePrepare, maintenanceResume } from "../web/pages/maintenance.ts"; |
| 33 | 33 | import { nightlyOutputPage, resiliencePage } from "../web/pages/resilience.ts"; |
| @@ -997,6 +997,7 @@ export function buildRouter(): Router<Handler> { | ||
| 997 | 997 | router.add("GET", "/boards/:slug", boardShow); |
| 998 | 998 | router.add("GET", "/boards/:slug/live", boardLive); |
| 999 | 999 | router.add("POST", "/boards/:slug/:op", boardPost); |
| 1000 | router.add("GET", "/admin/data", adminDataPage); | |
| 1000 | 1001 | router.add("GET", "/admin/limits", limitsPage); |
| 1001 | 1002 | router.add("GET", "/admin/flows", allFlowsPage); |
| 1002 | 1003 | router.add("POST", "/admin/limits", limitsSet); |
| @@ -1305,6 +1306,7 @@ export function buildRouter(): Router<Handler> { | ||
| 1305 | 1306 | router.add("POST", "/:owner/:name/app/:env/redeploy", appRedeploy); |
| 1306 | 1307 | router.add("POST", "/:owner/:name/app/:env/stop", appStop); |
| 1307 | 1308 | router.add("GET", "/:owner/:name/events", eventsList); |
| 1309 | router.add("GET", "/:owner/:name/data", appDataPage); | |
| 1308 | 1310 | router.add("GET", "/:owner/:name/runs", runsList); |
| 1309 | 1311 | router.add("GET", "/:owner/:name/runs/:id", runPage); |
| 1310 | 1312 | router.add("POST", "/:owner/:name/runs/:id/cancel", runCancel); |
src/tui/screens/data.ts +96 −9
| @@ -2,15 +2,21 @@ | ||
| 2 | 2 | * `/data` in the terminal: the SQLite files the viewer may see, a file's |
| 3 | 3 | * tables, a table's rows 50 a page, secrets masked. The same reads, and the |
| 4 | 4 | * same gate (`maySeeData`, `listSources`), as the web page and MCP `data`. |
| 5 | * `s` sorts a table by a column (`-name` for descending), `f` filters it | |
| 6 | * (`text`, or `column=text`), `c` clears both; `x` on a file's tables runs one | |
| 7 | * read-only SELECT, on the page's per-person budget (`sqlWindow`). | |
| 5 | 8 | */ |
| 6 | 9 | |
| 7 | import { PAGE_SIZE, rows, tables } from "../../data/browse.ts"; | |
| 10 | import { PAGE_SIZE, rows, tables, type RowsOptions } from "../../data/browse.ts"; | |
| 11 | import { query } from "../../data/query.ts"; | |
| 8 | 12 | import { listSources, maySeeData, type Source } from "../../data/sources.ts"; |
| 13 | import { sqlWindow } from "../../data/sql_window.ts"; | |
| 9 | 14 | import type { Ctx, Screen } from "../app/app.ts"; |
| 10 | 15 | import { binding } from "../app/keys.ts"; |
| 11 | 16 | import type { Session } from "../session.ts"; |
| 12 | 17 | import { style } from "../term/style.ts"; |
| 13 | 18 | import { truncate } from "../term/width.ts"; |
| 19 | import { later, promptScreen } from "./common.ts"; | |
| 14 | 20 | |
| 15 | 21 | /** A list of rows to move through; `open` is what Enter does on one. */ |
| 16 | 22 | function listScreen( |
| @@ -54,18 +60,45 @@ function listScreen( | ||
| 54 | 60 | |
| 55 | 61 | const cellText = (v: string | number | null): string => (v === null ? "" : String(v).replace(/[\r\n\t]+/g, " ").replace(/[\u0000-\u001f\u007f]/g, " ")); |
| 56 | 62 | |
| 63 | /** Splits `-name` / `name` into a sort; `undefined` for nothing. */ | |
| 64 | const parseSort = (text: string): RowsOptions["order"] | undefined => { | |
| 65 | const t = text.trim(); | |
| 66 | if (t === "") return undefined; | |
| 67 | return t.startsWith("-") ? { column: t.slice(1), dir: "desc" } : { column: t, dir: "asc" }; | |
| 68 | }; | |
| 69 | ||
| 70 | /** `text` or `column=text`; an empty string is no filter. */ | |
| 71 | const parseFilter = (text: string, columns: readonly string[]): RowsOptions["filter"] | undefined => { | |
| 72 | const eq = text.indexOf("="); | |
| 73 | const head = eq > 0 ? text.slice(0, eq).trim() : ""; | |
| 74 | if (head !== "" && columns.includes(head)) return { column: head, text: text.slice(eq + 1) }; | |
| 75 | return text === "" ? undefined : { text }; | |
| 76 | }; | |
| 77 | ||
| 57 | 78 | function rowsScreen(ctx: Ctx, source: Source, table: string): Screen { |
| 58 | 79 | let page = 0; |
| 59 | const read = () => rows(source, table, { page }); | |
| 60 | return listScreen( | |
| 80 | let order: RowsOptions["order"] | undefined; | |
| 81 | let filter: RowsOptions["filter"] | undefined; | |
| 82 | let note = ""; | |
| 83 | const read = () => { | |
| 84 | const wanted = { page, ...(order ? { order } : {}), ...(filter ? { filter } : {}) }; | |
| 85 | const r = rows(source, table, wanted); | |
| 86 | if (r || (!order && !filter)) return r; | |
| 87 | // An unknown or sealed column cannot be sorted or searched: show the table, say so. | |
| 88 | note = " · that column cannot be sorted or searched"; | |
| 89 | return rows(source, table, { page }); | |
| 90 | }; | |
| 91 | const screen: Screen = listScreen( | |
| 61 | 92 | ctx, |
| 62 | 93 | `${table} · ${source.label}`, |
| 63 | 94 | "", |
| 64 | 95 | () => { |
| 96 | note = ""; | |
| 65 | 97 | const r = read(); |
| 66 | 98 | if (!r) return []; |
| 99 | const how = `${order ? ` · by ${order.column} ${order.dir}` : ""}${filter ? ` · filter ${filter.column ? `${filter.column}=` : ""}${filter.text}` : ""}`; | |
| 67 | 100 | return [ |
| 68 | `${r.total} rows, page ${r.page + 1} · ${r.columns.join(" | ")}`, | |
| 101 | `${r.total} rows, page ${r.page + 1}${how}${note} · ${r.columns.join(" | ")}`, | |
| 69 | 102 | ...r.rows.map((row) => row.map(cellText).join(" | ")), |
| 70 | 103 | ]; |
| 71 | 104 | }, |
| @@ -75,16 +108,48 @@ function rowsScreen(ctx: Ctx, source: Source, table: string): Screen { | ||
| 75 | 108 | if (!r) return false; |
| 76 | 109 | if (key === "n" && (r.page + 1) * PAGE_SIZE < r.total) page = r.page + 1; |
| 77 | 110 | else if (key === "p" && r.page > 0) page = r.page - 1; |
| 78 | else return false; | |
| 111 | else if (key === "c") { | |
| 112 | order = undefined; | |
| 113 | filter = undefined; | |
| 114 | page = 0; | |
| 115 | } else if (key === "s" || key === "f") { | |
| 116 | const sorting = key === "s"; | |
| 117 | ctx.push( | |
| 118 | promptScreen(ctx, screen, { | |
| 119 | title: sorting ? "Sort by" : "Filter", | |
| 120 | label: sorting ? "Column" : "Text", | |
| 121 | placeholder: sorting ? "name, or -name for descending" : "text, or column=text", | |
| 122 | onSubmit: (v) => { | |
| 123 | if (sorting) order = parseSort(v); | |
| 124 | else filter = parseFilter(v.trim().slice(0, 200), r.columns); | |
| 125 | page = 0; | |
| 126 | return undefined; | |
| 127 | }, | |
| 128 | }), | |
| 129 | ); | |
| 130 | } else return false; | |
| 79 | 131 | return true; |
| 80 | 132 | }, |
| 81 | [binding(["n", "p"], "next / previous page")], | |
| 133 | [binding(["n", "p"], "next / previous page"), binding(["s", "f", "c"], "sort / filter / clear")], | |
| 82 | 134 | ); |
| 135 | return screen; | |
| 83 | 136 | } |
| 84 | 137 | |
| 85 | function tablesScreen(ctx: Ctx, source: Source): Screen { | |
| 138 | /** One SELECT's rows as lines, or the reason it was refused. */ | |
| 139 | async function sqlLines(source: Source, sql: string): Promise<string[]> { | |
| 140 | const t0 = performance.now(); | |
| 141 | const r = await query(source, sql); | |
| 142 | const ms = Math.max(1, Math.round(performance.now() - t0)); | |
| 143 | if (!r.ok) return [`${r.message} · ${ms} ms`]; | |
| 144 | return [ | |
| 145 | `${r.rows.length} rows${r.truncated ? " (cut)" : ""} · ${ms} ms · ${r.columns.join(" | ")}`, | |
| 146 | ...r.rows.map((row) => row.map(cellText).join(" | ")), | |
| 147 | ]; | |
| 148 | } | |
| 149 | ||
| 150 | function tablesScreen(ctx: Ctx, session: Session, source: Source): Screen { | |
| 86 | 151 | const list = tables(source); |
| 87 | return listScreen( | |
| 152 | const screen: Screen = listScreen( | |
| 88 | 153 | ctx, |
| 89 | 154 | source.label, |
| 90 | 155 | "Tables; secrets are masked.", |
| @@ -93,7 +158,29 @@ function tablesScreen(ctx: Ctx, source: Source): Screen { | ||
| 93 | 158 | const t = list[i]; |
| 94 | 159 | if (t) ctx.push(rowsScreen(ctx, source, t.name)); |
| 95 | 160 | }, |
| 161 | (key) => { | |
| 162 | if (key !== "x") return false; | |
| 163 | ctx.push( | |
| 164 | promptScreen(ctx, screen, { | |
| 165 | title: "SQL", | |
| 166 | label: "SELECT", | |
| 167 | placeholder: "SELECT * FROM … (read-only, one statement)", | |
| 168 | onSubmit: (sql) => { | |
| 169 | const text = sql.trim().slice(0, 4000); | |
| 170 | if (text === "") return "Type one SELECT."; | |
| 171 | const wait = session.user ? sqlWindow(session.app, session.user.id) : 1; | |
| 172 | if (wait !== undefined) return `Too many queries; try again in ${wait} s.`; | |
| 173 | return sqlLines(source, text).then((lines) => { | |
| 174 | void later(ctx, () => listScreen(ctx, "SQL", "", () => lines, () => undefined)); | |
| 175 | }); | |
| 176 | }, | |
| 177 | }), | |
| 178 | ); | |
| 179 | return true; | |
| 180 | }, | |
| 181 | [binding(["x"], "run SQL")], | |
| 96 | 182 | ); |
| 183 | return screen; | |
| 97 | 184 | } |
| 98 | 185 | |
| 99 | 186 | export function dataScreen(session: Session, ctx: Ctx): Screen { |
| @@ -105,7 +192,7 @@ export function dataScreen(session: Session, ctx: Ctx): Screen { | ||
| 105 | 192 | () => sources().map((s) => s.label), |
| 106 | 193 | (i) => { |
| 107 | 194 | const s = sources()[i]; |
| 108 | if (s) ctx.push(tablesScreen(ctx, s)); | |
| 195 | if (s) ctx.push(tablesScreen(ctx, session, s)); | |
| 109 | 196 | }, |
| 110 | 197 | ); |
| 111 | 198 | } |
src/web/pages/data.ts +67 −73
| @@ -1,87 +1,81 @@ | ||
| 1 | 1 | /** |
| 2 | * /data: the SQLite files the viewer may see (`data/sources.ts`), the | |
| 3 | * tables in one, and a page of a table's rows, secrets masked. Anyone with | |
| 4 | * no door to the data (a guest, a reader) gets the not-found page; an admin | |
| 5 | * or an app's writer gets the page, with an empty state until a file exists. | |
| 2 | * The data browser's three doors (body in `data_view.ts`): | |
| 3 | * - `/data`: every SQLite file the viewer may see (an admin, or an app's writer). | |
| 4 | * - `/admin/data`: the forge's own database, admins only. | |
| 5 | * - `/:owner/:name/data`: one app's databases, environment by environment, for | |
| 6 | * the people who can write the app. Anyone else gets the one 404. | |
| 6 | 7 | */ |
| 7 | 8 | |
| 8 | import { PAGE_SIZE, rows, tables } from "../../data/browse.ts"; | |
| 9 | import { findSource, listSources, maySeeData } from "../../data/sources.ts"; | |
| 9 | import { getAppByRepo } from "../../core/apps.ts"; | |
| 10 | import { canWrite } from "../../core/repos.ts"; | |
| 11 | import { sqlWindow } from "../../data/sql_window.ts"; | |
| 12 | import { listSources, maySeeData } from "../../data/sources.ts"; | |
| 10 | 13 | import type { Ctx } from "../../http/context.ts"; |
| 11 | 14 | import { notePhase, takeSessionTime } from "../../core/route_timing.ts"; |
| 12 | import { notFoundPage, pageHead, signInFirst } from "../chrome.ts"; | |
| 13 | import { html } from "../html.ts"; | |
| 15 | import { loadRepo, notFoundPage, pageHead, scopeHeader, signInFirst } from "../chrome.ts"; | |
| 16 | import { html, type Html } from "../html.ts"; | |
| 14 | 17 | import { page } from "../layout.ts"; |
| 15 | import { empty, pager, row } from "../ui.ts"; | |
| 18 | import { btn } from "../ui.ts"; | |
| 19 | import { localScope } from "../scope.ts"; | |
| 20 | import { dataBody, type DataView } from "./data_view.ts"; | |
| 16 | 21 | |
| 17 | const PURPOSE = "The SQLite files you may read, their tables and rows; secrets are masked."; | |
| 22 | const PURPOSE = "Every SQLite file you may read; secrets are masked."; | |
| 18 | 23 | |
| 19 | const href = (params: Record<string, string | number>): string => | |
| 20 | `/data?${new URLSearchParams(Object.entries(params).map(([k, v]): [string, string] => [k, String(v)]))}`; | |
| 24 | /** What `page()` takes besides the body, for the doors that draw their own frame. */ | |
| 25 | type Frame = Partial<Pick<Parameters<typeof page>[0], "crumbs" | "palette">>; | |
| 21 | 26 | |
| 22 | // GET /data[?source=<id>[&table=<name>[&page=<n>]]] | |
| 23 | export function dataPage(ctx: Ctx): Response { | |
| 27 | async function draw(ctx: Ctx, view: DataView, title: string, head: Html, extra: Frame = {}): Promise<Response> { | |
| 28 | const rendering = performance.now(); | |
| 29 | const inner = await dataBody(view, ctx.url.searchParams); | |
| 30 | if (inner === undefined) return refuse(ctx); | |
| 31 | const response = page({ ctx, title, body: html`${head}${inner}`, ...extra }); | |
| 32 | notePhase(ctx.request, "render", performance.now() - rendering); | |
| 33 | return response; | |
| 34 | } | |
| 35 | ||
| 36 | /** A refusal still names `render`, so every answer carries both phases. */ | |
| 37 | function refuse(ctx: Ctx): Response { | |
| 38 | const rendering = performance.now(); | |
| 39 | const response = notFoundPage(ctx); | |
| 40 | notePhase(ctx.request, "render", performance.now() - rendering); | |
| 41 | return response; | |
| 42 | } | |
| 43 | ||
| 44 | // GET /data[?source=<id>[&table=<name>…][&sql=…]] | |
| 45 | export async function dataPage(ctx: Ctx): Promise<Response> { | |
| 24 | 46 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); |
| 25 | 47 | const viewer = ctx.auth.user; |
| 26 | 48 | if (!viewer) return signInFirst(ctx); |
| 27 | 49 | const { db, config } = ctx.app; |
| 28 | if (!maySeeData(db, viewer)) return notFoundPage(ctx); | |
| 29 | const q = ctx.url.searchParams; | |
| 30 | const all = listSources(db, config.dataDir, viewer); | |
| 31 | const chosen = q.get("source"); | |
| 32 | const source = chosen === null ? undefined : findSource(db, config.dataDir, viewer, chosen); | |
| 33 | if (chosen !== null && !source) return notFoundPage(ctx); | |
| 34 | ||
| 35 | let body = html`${pageHead("Data", { sub: PURPOSE })} | |
| 36 | ${all.length === 0 | |
| 37 | ? empty({ | |
| 38 | title: "No database yet", | |
| 39 | note: "An app's SQLite files appear here once it has written one under its data directory.", | |
| 40 | }) | |
| 41 | : html`<ul class="list"> | |
| 42 | ${all.map((s) => row({ href: href({ source: s.id }), title: s.label, current: s.id === source?.id }))} | |
| 43 | </ul>`}`; | |
| 44 | ||
| 45 | if (source) { | |
| 46 | const list = tables(source); | |
| 47 | const name = q.get("table"); | |
| 48 | const shown = name === null ? undefined : rows(source, name, { page: Number(q.get("page") ?? 0) }); | |
| 49 | if (name !== null && !shown) return notFoundPage(ctx); | |
| 50 | body = html`${body} | |
| 51 | <h2>${source.label}</h2> | |
| 52 | <ul class="list"> | |
| 53 | ${list.map((t) => | |
| 54 | row({ | |
| 55 | href: href({ source: source.id, table: t.name }), | |
| 56 | title: t.name, | |
| 57 | trail: html`${String(t.rows)} rows`, | |
| 58 | current: t.name === name, | |
| 59 | }), | |
| 60 | )} | |
| 61 | </ul>`; | |
| 62 | if (shown && name !== null) { | |
| 63 | const more = (shown.page + 1) * PAGE_SIZE < shown.total; | |
| 64 | body = html`${body} | |
| 65 | <h2>${name}</h2> | |
| 66 | <div class="table-scroll"> | |
| 67 | <table class="table" data-dense> | |
| 68 | <caption class="sr-only">${name} in ${source.label}</caption> | |
| 69 | <thead><tr>${shown.columns.map((c) => html`<th scope="col">${c}</th>`)}</tr></thead> | |
| 70 | <tbody> | |
| 71 | ${shown.rows.map((r) => html`<tr>${r.map((v) => html`<td>${v === null ? "" : String(v)}</td>`)}</tr>`)} | |
| 72 | </tbody> | |
| 73 | </table> | |
| 74 | </div> | |
| 75 | ${pager({ | |
| 76 | note: `${shown.total} rows`, | |
| 77 | words: { prev: "Previous page", next: "Next page" }, | |
| 78 | ...(shown.page > 0 ? { prev: href({ source: source.id, table: name, page: shown.page - 1 }) } : {}), | |
| 79 | ...(more ? { next: href({ source: source.id, table: name, page: shown.page + 1 }) } : {}), | |
| 80 | })}`; | |
| 81 | } | |
| 82 | } | |
| 83 | const rendering = performance.now(); | |
| 84 | const response = page({ ctx, title: "Data", body }); | |
| 85 | notePhase(ctx.request, "render", performance.now() - rendering); | |
| 86 | return response; | |
| 50 | if (!maySeeData(db, viewer)) return refuse(ctx); | |
| 51 | const sources = listSources(db, config.dataDir, viewer); | |
| 52 | return draw(ctx, { path: "/data", sources, sqlWait: () => sqlWindow(ctx.app, viewer.id) }, "Data", pageHead("Data", { sub: PURPOSE, ...(viewer.isAdmin ? { actions: btn("Forge data", { href: "/admin/data", size: "sm" }) } : {}) })); | |
| 53 | } | |
| 54 | ||
| 55 | // GET /admin/data — the forge's own database (the admin gate has already refused non-admins). | |
| 56 | export async function adminDataPage(ctx: Ctx): Promise<Response> { | |
| 57 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); | |
| 58 | const viewer = ctx.auth.user; | |
| 59 | if (!viewer) return signInFirst(ctx); | |
| 60 | if (!viewer.isAdmin) return refuse(ctx); | |
| 61 | const sources = listSources(ctx.app.db, ctx.app.config.dataDir, viewer).filter((s) => s.kind === "forge"); | |
| 62 | return draw(ctx, { path: "/admin/data", sources, sqlWait: () => sqlWindow(ctx.app, viewer.id) }, "Data · forge", pageHead("Forge data", { sub: "Secrets are masked; queries are read-only." })); | |
| 63 | } | |
| 64 | ||
| 65 | // GET /:owner/:name/data — an app's databases, for its writers. | |
| 66 | export async function appDataPage(ctx: Ctx): Promise<Response> { | |
| 67 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); | |
| 68 | const repo = loadRepo(ctx); | |
| 69 | const record = repo && getAppByRepo(ctx.app.db, repo.id); | |
| 70 | if (!repo || !record || !ctx.auth.user || !canWrite(ctx.app.db, repo, ctx.auth.user)) return refuse(ctx); | |
| 71 | const viewer = ctx.auth.user; | |
| 72 | const sources = listSources(ctx.app.db, ctx.app.config.dataDir, viewer).filter((s) => s.id.startsWith(`app:${record.id}:`)); | |
| 73 | const scope = localScope(ctx, repo); | |
| 74 | return draw( | |
| 75 | ctx, | |
| 76 | { path: `/${repo.ownerName}/${repo.name}/data`, sources, byEnv: true, sqlWait: () => sqlWindow(ctx.app, viewer.id) }, | |
| 77 | `Data · ${repo.name}`, | |
| 78 | scopeHeader(ctx, scope, "data"), | |
| 79 | { crumbs: scope.crumbs, palette: scope.palette }, | |
| 80 | ); | |
| 87 | 81 | } |
src/web/pages/data_view.ts added +218
| @@ -0,0 +1,218 @@ | ||
| 1 | /** | |
Tests: ?sql= re-runs a query on every GET with no per-identity rate limit (DoS). Add a limit keyed on the signed-in user, plus a test. Also add tests for forge-db masking via SQL and rows, cellView JSON, and the SQL cap. Hand-rolled <select> and glyph sort marks: use the named components (DESIGN principle 8/7). |
||
| 2 | * The data browser's body, shared by its three doors: `/data` (every file the | |
| 3 | * viewer may see), `/admin/data` (the forge's own) and `/:owner/:name/data` | |
| 4 | * (one app's, environment by environment). A door picks the sources and the | |
| 5 | * address; this draws files → tables → rows (sort, one-column filter, JSON | |
| 6 | * cells, copy row as JSON) and the read-only SQL box. Everything is a GET, so | |
| 7 | * every view, SQL included, is an address that can be shared. | |
| 8 | */ | |
| 9 | ||
| 10 | import { PAGE_SIZE, rows, tables, type RowsResult } from "../../data/browse.ts"; | |
| 11 | import { query } from "../../data/query.ts"; | |
| 12 | import type { Source } from "../../data/sources.ts"; | |
| 13 | import { html, type Html } from "../html.ts"; | |
| 14 | import { btn, empty, field, iconBtn, pager, row } from "../ui.ts"; | |
| 15 | ||
| 16 | /** Longest SQL an address may carry. */ | |
| 17 | export const SQL_MAX = 4000; | |
| 18 | ||
| 19 | type Cell = string | number | null; | |
| 20 | ||
| 21 | export interface DataView { | |
| 22 | /** The path every link points at, without a query. */ | |
| 23 | readonly path: string; | |
| 24 | readonly sources: readonly Source[]; | |
| 25 | /** Group file rows under their environment (an app's own tab). */ | |
| 26 | readonly byEnv?: boolean; | |
| 27 | /** Asked before a SQL run: seconds to wait when this person has run too many, else undefined. */ | |
| 28 | readonly sqlWait?: () => number | undefined; | |
| 29 | } | |
| 30 | ||
| 31 | const url = (path: string, params: Record<string, string | number | undefined>): string => { | |
| 32 | const q = new URLSearchParams(); | |
| 33 | for (const [k, v] of Object.entries(params)) if (v !== undefined && v !== "") q.set(k, String(v)); | |
| 34 | const s = q.toString(); | |
| 35 | return s === "" ? path : `${path}?${s}`; | |
| 36 | }; | |
| 37 | ||
| 38 | /** A cell: JSON text pretty-printed, null left empty, anything else as text. */ | |
| 39 | export function cellView(v: Cell): Html { | |
| 40 | if (v === null) return html``; | |
| 41 | const text = String(v); | |
| 42 | if (typeof v === "string" && (text.startsWith("{") || text.startsWith("["))) { | |
| 43 | try { | |
| 44 | const parsed: unknown = JSON.parse(text); | |
| 45 | if (typeof parsed === "object" && parsed !== null) | |
| 46 | return html`<pre class="mono">${JSON.stringify(parsed, null, 2)}</pre>`; | |
| 47 | } catch { | |
| 48 | // not JSON after all: plain text below | |
| 49 | } | |
| 50 | } | |
| 51 | return html`${text}`; | |
| 52 | } | |
| 53 | ||
| 54 | const asObject = (columns: readonly string[], r: readonly Cell[]): Record<string, Cell> => | |
| 55 | Object.fromEntries(columns.map((c, i): [string, Cell] => [c, r[i] ?? null])); | |
| 56 | ||
| 57 | function grid( | |
| 58 | columns: readonly string[], | |
| 59 | data: readonly (readonly Cell[])[], | |
| 60 | caption: string, | |
| 61 | head: (c: string) => Html, | |
| 62 | copyIds: boolean, | |
| 63 | ): Html { | |
| 64 | return html`<div class="table-scroll"> | |
| 65 | <table class="table" data-dense data-sortable> | |
| 66 | <caption class="sr-only">${caption}</caption> | |
| 67 | <thead><tr>${copyIds ? html`<th scope="col"><span class="sr-only">Copy</span></th>` : null}${columns.map(head)}</tr></thead> | |
| 68 | <tbody> | |
| 69 | ${data.map( | |
| 70 | (r, i) => html`<tr> | |
| 71 | ${copyIds | |
| 72 | ? html`<td>${iconBtn("copy", "Copy row as JSON", { copy: `row-${i}`, size: "sm" })}<pre id="row-${i}" hidden>${JSON.stringify(asObject(columns, r), null, 2)}</pre></td>` | |
| 73 | : null} | |
| 74 | ${r.map((v) => html`<td>${cellView(v)}</td>`)} | |
| 75 | </tr>`, | |
| 76 | )} | |
| 77 | </tbody> | |
| 78 | </table> | |
| 79 | </div>`; | |
| 80 | } | |
| 81 | ||
| 82 | const intOr0 = (s: string | null): number => { | |
| 83 | const n = Number(s ?? 0); | |
| 84 | return Number.isInteger(n) && n > 0 ? n : 0; | |
| 85 | }; | |
| 86 | ||
| 87 | /** The body for these sources and this query string, or `undefined` for a 404. */ | |
| 88 | export async function dataBody(view: DataView, q: URLSearchParams): Promise<Html | undefined> { | |
| 89 | const { path, sources } = view; | |
| 90 | const chosen = q.get("source") ?? (sources.length === 1 ? sources[0]!.id : null); | |
| 91 | const source = chosen === null ? undefined : sources.find((s) => s.id === chosen); | |
| 92 | if (chosen !== null && !source) return undefined; | |
| 93 | ||
| 94 | const fileRow = (s: Source) => | |
| 95 | row({ href: url(path, { source: s.id }), title: view.byEnv ? (s.id.split(":").slice(3).join(":") || s.label) : s.label, current: s.id === source?.id }); | |
| 96 | const envOf = (s: Source) => s.id.split(":")[2] ?? ""; | |
| 97 | const files = | |
| 98 | sources.length === 0 | |
| 99 | ? empty({ | |
| 100 | title: "No database yet", | |
| 101 | note: "An app's SQLite files appear here once it has written one under its data directory.", | |
| 102 | }) | |
| 103 | : view.byEnv | |
| 104 | ? html`${[...new Set(sources.map(envOf))].map( | |
| 105 | (env) => html`<h2>${env}</h2><ul class="list">${sources.filter((s) => envOf(s) === env).map(fileRow)}</ul>`, | |
| 106 | )}` | |
| 107 | : sources.length === 1 && source | |
| 108 | ? html`` | |
| 109 | : html`<ul class="list">${sources.map(fileRow)}</ul>`; | |
| 110 | if (!source) return files; | |
| 111 | ||
| 112 | const tableName = q.get("table"); | |
| 113 | const sql = (q.get("sql") ?? "").slice(0, SQL_MAX); | |
| 114 | const list = tables(source); | |
| 115 | let out = html`${files} | |
| 116 | <h2>${source.label}</h2> | |
| 117 | <ul class="list"> | |
| 118 | ${list.map((t) => | |
| 119 | row({ | |
| 120 | href: url(path, { source: source.id, table: t.name }), | |
| 121 | title: t.name, | |
| 122 | trail: html`${String(t.rows)} rows`, | |
| 123 | current: t.name === tableName, | |
| 124 | }), | |
| 125 | )} | |
| 126 | </ul>`; | |
| 127 | ||
| 128 | if (tableName !== null) { | |
| 129 | const sort = q.get("sort") ?? undefined; | |
| 130 | const dir = q.get("dir") === "desc" ? "desc" : "asc"; | |
| 131 | const text = (q.get("q") ?? "").slice(0, 200); | |
| 132 | const col = q.get("col") || undefined; | |
| 133 | const pageNo = intOr0(q.get("page")); | |
| 134 | const base = { source: source.id, table: tableName, sort, dir: sort ? dir : undefined, q: text, col }; | |
| 135 | let shown: RowsResult | undefined = rows(source, tableName, { | |
| 136 | page: pageNo, | |
| 137 | ...(sort ? { order: { column: sort, dir } } : {}), | |
| 138 | ...(text !== "" ? { filter: { text, ...(col ? { column: col } : {}) } } : {}), | |
| 139 | }); | |
| 140 | let note: string | null = null; | |
| 141 | if (!shown && (sort || text !== "")) { | |
| 142 | // An unknown or sealed column cannot be sorted or searched: show the table, say so. | |
| 143 | shown = rows(source, tableName, { page: pageNo }); | |
| 144 | note = "That column cannot be sorted or searched here."; | |
| 145 | } | |
| 146 | if (!shown) return undefined; | |
| 147 | const cols = shown.columns; | |
| 148 | const more = (shown.page + 1) * PAGE_SIZE < shown.total; | |
| 149 | const header = (c: string): Html => { | |
| 150 | const next = sort === c && dir === "asc" ? "desc" : "asc"; | |
| 151 | return html`<th scope="col" ${sort === c ? html`aria-sort="${dir === "asc" ? "ascending" : "descending"}"` : null}><a href="${url(path, { ...base, sort: c, dir: next })}">${c}</a></th>`; | |
| 152 | }; | |
| 153 | out = html`${out} | |
| 154 | <h2>${tableName}</h2> | |
| 155 | <form method="get" action="${path}" role="search"> | |
| 156 | <input type="hidden" name="source" value="${source.id}" /> | |
| 157 | <input type="hidden" name="table" value="${tableName}" /> | |
| 158 | ${sort ? html`<input type="hidden" name="sort" value="${sort}" /><input type="hidden" name="dir" value="${dir}" />` : null} | |
| 159 | ${field({ | |
| 160 | name: "q", | |
| 161 | value: text, | |
| 162 | label: "Filter", | |
| 163 | labelHidden: true, | |
| 164 | placeholder: "Filter rows", | |
| 165 | type: "search", | |
| 166 | })} | |
| 167 | ${field({ | |
| 168 | name: "col", | |
| 169 | type: "select", | |
| 170 | label: "Column", | |
| 171 | labelHidden: true, | |
| 172 | options: [{ value: "", label: "Any column" }, ...cols.map((c) => ({ value: c, label: c, selected: c === col }))], | |
| 173 | })} | |
| 174 | <div class="form-actions"> | |
| 175 | ${btn("Filter", { size: "sm", type: "submit" })} | |
| 176 | ${text !== "" ? btn("Clear", { href: url(path, { source: source.id, table: tableName, sort, dir: sort ? dir : undefined }), size: "sm", variant: "outline" }) : null} | |
| 177 | </div> | |
| 178 | </form> | |
| 179 | ${note ? html`<p class="muted small">${note}</p>` : null} | |
| 180 | ${shown.rows.length === 0 ? empty({ title: "No rows", note: text !== "" ? "Nothing matches that filter." : "This table is empty." }) : grid(cols, shown.rows, `${tableName} in ${source.label}`, header, true)} | |
| 181 | ${pager({ | |
| 182 | note: `${shown.total} rows`, | |
| 183 | words: { prev: "Previous page", next: "Next page" }, | |
| 184 | ...(shown.page > 0 ? { prev: url(path, { ...base, page: shown.page - 1 }) } : {}), | |
| 185 | ...(more ? { next: url(path, { ...base, page: shown.page + 1 }) } : {}), | |
| 186 | })}`; | |
| 187 | } | |
| 188 | ||
| 189 | out = html`${out} | |
| 190 | <form method="get" action="${path}"> | |
| 191 | <input type="hidden" name="source" value="${source.id}" /> | |
| 192 | ${field({ | |
| 193 | name: "sql", | |
| 194 | type: "textarea", | |
| 195 | value: sql, | |
| 196 | mono: true, | |
| 197 | rows: 4, | |
| 198 | placeholder: "SELECT * FROM … (read-only, one statement)", | |
| 199 | spellcheck: false, | |
| 200 | maxlength: SQL_MAX, | |
| 201 | label: "SQL", | |
| 202 | })} | |
| 203 | <div class="form-actions">${btn("Run", { size: "sm", type: "submit" })}</div> | |
| 204 | </form>`; | |
| 205 | if (sql.trim() !== "") { | |
| 206 | const wait = view.sqlWait?.(); | |
| 207 | if (wait !== undefined) return html`${out}<p role="alert" class="muted">Too many queries; try again in ${String(wait)} s.</p>`; | |
| 208 | const t0 = performance.now(); | |
| 209 | const result = await query(source, sql); | |
| 210 | const ms = Math.max(1, Math.round(performance.now() - t0)); | |
| 211 | out = result.ok | |
| 212 | ? html`${out} | |
| 213 | <p class="muted small">${String(result.rows.length)} rows${result.truncated ? " (cut)" : ""} · ${String(ms)} ms</p> | |
| 214 | ${result.rows.length === 0 ? empty({ title: "No rows" }) : grid(result.columns, result.rows, "Query result", (c) => html`<th scope="col">${c}</th>`, false)}` | |
| 215 | : html`${out}<p role="alert" class="muted">${result.message} · ${String(ms)} ms</p>`; | |
| 216 | } | |
| 217 | return out; | |
| 218 | } |
src/web/repo_steps.ts +5 −1
| @@ -11,7 +11,7 @@ import { appsRuntime } from "../apps/runtime.ts"; | ||
| 11 | 11 | import { knownSpec, warmSpec } from "../apps/api_spec.ts"; |
| 12 | 12 | import { runningCount } from "../core/runs.ts"; |
| 13 | 13 | import { getAppByRepo } from "../core/apps.ts"; |
| 14 | import { canAdmin, canEnterApp, visibleForks, type Repo } from "../core/repos.ts"; | |
| 14 | import { canAdmin, canEnterApp, canWrite, visibleForks, type Repo } from "../core/repos.ts"; | |
| 15 | 15 | import type { Ctx } from "../http/context.ts"; |
| 16 | 16 | import { sshCommand } from "../ssh/command.ts"; |
| 17 | 17 | import { changesHref, tellsHref } from "./studio/nouns.ts"; |
| @@ -27,6 +27,7 @@ export type RepoTab = | ||
| 27 | 27 | | "app" |
| 28 | 28 | | "runs" |
| 29 | 29 | | "events" |
| 30 | | "data" | |
| 30 | 31 | | "flows" |
| 31 | 32 | | "api" |
| 32 | 33 | | "settings" |
| @@ -114,6 +115,9 @@ function work(ctx: Ctx, base: string, repo: Repo | undefined): Steps { | ||
| 114 | 115 | ...(running > 0 ? { count: running } : {}), |
| 115 | 116 | }); |
| 116 | 117 | } |
| 118 | // An app's databases, for the people who can write the app. | |
| 119 | if (record !== undefined && canWrite(ctx.app.db, repo, ctx.auth.user)) | |
| 120 | steps.push({ key: "data", label: "Data", href: `${base}/data` }); | |
| 117 | 121 | steps.push({ key: "events", label: "Events", href: `${base}/events` }); |
| 118 | 122 | // What flows in and out: origin, sources, pins, forks (#9434). The page gates itself like the repo. |
| 119 | 123 | steps.push({ key: "flows", label: "Flows", href: `${base}/flows` }); |
src/web/style/components.ts +5
| @@ -583,6 +583,11 @@ details[open] > summary.row > .row-lead { transform: rotate(90deg); } | ||
| 583 | 583 | .table[data-dense] td, .table[data-dense] th { padding: 5px 10px; } |
| 584 | 584 | .table .num { text-align: right; } |
| 585 | 585 | .table-scroll { overflow-x: auto; scrollbar-width: thin; } |
| 586 | /* The data browser: a cell's JSON keeps its shape, and the sorted column is marked by its aria-sort. */ | |
| 587 | .table td pre { margin: 0; white-space: pre-wrap; overflow-wrap: anywhere; font-size: inherit; } | |
| 588 | .table[data-sortable] th[aria-sort] a { color: var(--fg); } | |
outdated
DESIGN principle 4: one background, --bg. The sticky header uses --card; use var(--bg). Principle 6: a max-height scrolling box around the grid is a box around a thing; prefer page flow. |
||
| 589 | .table[data-sortable] th[aria-sort="ascending"] a::after { content: " \\2191"; color: var(--muted-fg); } | |
| 590 | .table[data-sortable] th[aria-sort="descending"] a::after { content: " \\2193"; color: var(--muted-fg); } | |
| 586 | 591 | /* The row's name stays in view while a narrow screen scrolls the figures. */ |
| 587 | 592 | .table[data-sticky-lead] th:first-child, |
| 588 | 593 | .table[data-sticky-lead] td:first-child { |
test/data-door.test.ts +108
| @@ -9,6 +9,7 @@ import { createApp, type App } from "../src/app.ts"; | ||
| 9 | 9 | import { envDataDir } from "../src/apps/data.ts"; |
| 10 | 10 | import { createApp as createAppRecord } from "../src/core/apps.ts"; |
| 11 | 11 | import { createRepo } from "../src/core/repos.ts"; |
| 12 | import { SQL_PER_MINUTE } from "../src/data/sql_window.ts"; | |
| 12 | 13 | import { createSession } from "../src/core/sessions.ts"; |
| 13 | 14 | import { createToken } from "../src/core/tokens.ts"; |
| 14 | 15 | import { createUser } from "../src/core/users.ts"; |
| @@ -123,4 +124,111 @@ describe("the data door", () => { | ||
| 123 | 124 | expect((await mcpTool(servers.url, token["bob"]!, "data", { source: id })).error).toBeDefined(); |
| 124 | 125 | expect((await mcpTool(servers.url, token["bob"]!, "data")).value).toEqual([]); |
| 125 | 126 | }); |
| 127 | ||
| 128 | test("an app's Data tab: a writer sees it, a reader gets a 404, secrets stay masked", async () => { | |
| 129 | const tab = await get("/alice/notes/data", "alice"); | |
| 130 | expect(tab.status).toBe(200); | |
| 131 | const text = await tab.text(); | |
| 132 | expect(text).toContain('href="/alice/notes/data"'); | |
| 133 | expect(text).toContain("production"); | |
| 134 | expect(text).toContain("app.db"); | |
| 135 | expect((await get("/alice/notes/data", "bob")).status).toBe(404); | |
| 136 | expect((await get("/alice/notes/data")).status).not.toBe(200); | |
| 137 | const rows = await (await get(`/alice/notes/data?source=${src()}&table=users`, "alice")).text(); | |
| 138 | expect(rows).toContain("ann"); | |
| 139 | expect(rows).toContain("••••••"); | |
| 140 | expect(rows).not.toContain("tok-ann-1"); | |
| 141 | expect(rows).toContain('data-copy="row-0"'); | |
| 142 | expect(rows).toContain(""name": "ann""); | |
| 143 | }); | |
| 144 | ||
| 145 | test("another app's source is not reachable through this app's tab", async () => { | |
| 146 | expect((await get(`/alice/notes/data?source=forge`, "owner1")).status).toBe(404); | |
| 147 | expect((await get(`/dana/bare/data`, "alice")).status).toBe(404); | |
| 148 | }); | |
| 149 | ||
| 150 | test("sort and filter narrow the rows; a sealed column cannot be searched", async () => { | |
| 151 | const d = new Database(join(envDataDir(dataDir, appId, "production"), "app.db")); | |
| 152 | d.exec("INSERT INTO users VALUES (2, 'zed', 'tok-zed-2')"); | |
| 153 | d.close(); | |
| 154 | const base = `/alice/notes/data?source=${src()}&table=users`; | |
| 155 | const sorted = await (await get(`${base}&sort=name&dir=desc`, "alice")).text(); | |
| 156 | expect(sorted.indexOf("<td>zed</td>")).toBeLessThan(sorted.indexOf("<td>ann</td>")); | |
| 157 | expect(sorted).toContain('aria-sort="descending"'); | |
| 158 | const filtered = await (await get(`${base}&q=zed&col=name`, "alice")).text(); | |
| 159 | expect(filtered).toContain("zed"); | |
| 160 | expect(filtered).not.toContain("<td>ann</td>"); | |
| 161 | const sealed = await (await get(`${base}&q=tok-zed&col=api_token`, "alice")).text(); | |
| 162 | expect(sealed).toContain("Nothing matches that filter"); | |
| 163 | expect(sealed).not.toContain("tok-zed-2"); | |
| 164 | }); | |
| 165 | ||
| 166 | test("the SQL box runs a read-only query from the address and shows rows and ms", async () => { | |
| 167 | const ok = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent("SELECT name FROM users ORDER BY id")}`, "alice")).text(); | |
| 168 | expect(ok).toContain("ann"); | |
| 169 | expect(ok).toMatch(/\d+ ms/); | |
| 170 | const sealed = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent("SELECT api_token FROM users")}`, "alice")).text(); | |
| 171 | expect(sealed).not.toContain("tok-ann-1"); | |
| 172 | const bad = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent("DELETE FROM users")}`, "alice")).text(); | |
| 173 | expect(bad).toContain("Only SELECT"); | |
| 174 | const d = new Database(join(envDataDir(dataDir, appId, "production"), "app.db"), { readonly: true }); | |
| 175 | expect(d.query("SELECT count(*) AS n FROM users").get()).toEqual({ n: 2 }); | |
| 176 | d.close(); | |
| 177 | }); | |
| 178 | ||
| 179 | test("/admin/data is the forge's own database, for admins only", async () => { | |
| 180 | const res = await get("/admin/data", "owner1"); | |
| 181 | expect(res.status).toBe(200); | |
| 182 | expect(await res.text()).toContain("forge_setting"); | |
| 183 | expect((await get("/admin/data", "alice")).status).toBeGreaterThanOrEqual(403); | |
| 184 | expect((await get("/admin/data")).status).not.toBe(200); | |
| 185 | }); | |
| 186 | ||
| 187 | test("the Forge data button is for admins alone", async () => { | |
| 188 | // owner1 is the admin and can see /data (forge.db); alice is a writer; bob a reader. | |
| 189 | expect(await (await get("/data", "owner1")).text()).toContain('href="/admin/data"'); | |
| 190 | expect(await (await get("/data", "alice")).text()).not.toContain('href="/admin/data"'); | |
| 191 | }); | |
| 192 | ||
| 193 | test("a JSON cell is pretty-printed and a plain one is not", async () => { | |
| 194 | const d = new Database(join(envDataDir(dataDir, appId, "production"), "app.db")); | |
| 195 | d.exec("CREATE TABLE IF NOT EXISTS docs (id INTEGER PRIMARY KEY, body TEXT)"); | |
| 196 | d.exec(`INSERT INTO docs VALUES (1, '{"a":1,"b":[2]}'), (2, '{not json'), (3, 'plain')`); | |
| 197 | d.close(); | |
| 198 | const text = await (await get(`/alice/notes/data?source=${src()}&table=docs`, "alice")).text(); | |
| 199 | expect(text).toContain('<pre class="mono">{\n "a": 1,'); | |
| 200 | expect(text).toContain("<td>{not json</td>"); | |
| 201 | expect(text).toContain("<td>plain</td>"); | |
| 202 | }); | |
| 203 | ||
| 204 | test("the SQL in an address is cut at 4,000 characters", async () => { | |
| 205 | const long = `SELECT 1 AS a ${"y".repeat(5000)} tail-marker`; | |
| 206 | const text = await (await get(`/alice/notes/data?source=${src()}&sql=${encodeURIComponent(long)}`, "alice")).text(); | |
| 207 | expect(text).not.toContain("tail-marker"); | |
| 208 | const inBox = text.match(/<textarea[^>]*>([\s\S]*?)<\/textarea\s*>/)![1]!; | |
| 209 | expect(inBox.trim().length).toBe(4000); | |
| 210 | }); | |
| 211 | ||
| 212 | test("the forge database's secrets are masked in rows and in SQL on /admin/data", async () => { | |
| 213 | const hash = app.db.query<{ h: string }, []>("SELECT password_hash AS h FROM user LIMIT 1").get()!.h; | |
| 214 | const rowsHtml = await (await get("/admin/data?table=user", "owner1")).text(); | |
| 215 | expect(rowsHtml).toContain("alice"); | |
| 216 | expect(rowsHtml).toContain("••••••"); | |
| 217 | expect(rowsHtml).not.toContain(hash); | |
| 218 | const sqlHtml = await (await get(`/admin/data?sql=${encodeURIComponent("SELECT username, password_hash AS p FROM user")}`, "owner1")).text(); | |
| 219 | expect(sqlHtml).toContain("alice"); | |
| 220 | expect(sqlHtml).not.toContain(hash); | |
| 221 | const aliased = await (await get(`/admin/data?sql=${encodeURIComponent("SELECT * FROM (SELECT password_hash AS q FROM user)")}`, "owner1")).text(); | |
| 222 | expect(aliased).not.toContain(hash); | |
| 223 | }); | |
| 224 | ||
| 225 | test("SQL runs are limited per person; another person is not held back", async () => { | |
| 226 | const q = `/alice/notes/data?source=${src()}&sql=${encodeURIComponent("SELECT 1 AS one")}`; | |
| 227 | let refused = 0; | |
| 228 | for (let i = 0; i < SQL_PER_MINUTE + 3; i++) if ((await (await get(q, "alice")).text()).includes("Too many queries")) refused++; | |
| 229 | expect(refused).toBeGreaterThanOrEqual(1); | |
| 230 | expect(await (await get("/admin/data?sql=SELECT%201", "owner1")).text()).not.toContain("Too many queries"); | |
| 231 | // Browsing a table is not SQL and is never held. | |
| 232 | expect(await (await get(`/alice/notes/data?source=${src()}&table=users`, "alice")).text()).not.toContain("Too many queries"); | |
| 233 | }); | |
| 126 | 234 | }); |
test/tui-data.test.ts +63
| @@ -12,6 +12,8 @@ import { createRepo } from "../src/core/repos.ts"; | ||
| 12 | 12 | import { createUser, getUserByName } from "../src/core/users.ts"; |
| 13 | 13 | import { forgeCommands } from "../src/tui/forge.ts"; |
| 14 | 14 | import { createForgeApp } from "../src/tui/forge.ts"; |
| 15 | import { query } from "../src/data/query.ts"; | |
| 16 | import { listSources } from "../src/data/sources.ts"; | |
| 15 | 17 | import { dataScreen } from "../src/tui/screens/data.ts"; |
| 16 | 18 | import { headlessTerminal } from "../src/tui/term/terminal.ts"; |
| 17 | 19 | import { quietLogs } from "./helpers/globals.ts"; |
| @@ -21,6 +23,8 @@ quietLogs(); | ||
| 21 | 23 | |
| 22 | 24 | let dir: string; |
| 23 | 25 | let app: App; |
| 26 | let appRecordId = ""; | |
| 27 | const appIdOf = () => appRecordId; | |
| 24 | 28 | |
| 25 | 29 | beforeAll(async () => { |
| 26 | 30 | dir = await mkdtemp(join(tmpdir(), "forge-tui-data-")); |
| @@ -35,6 +39,7 @@ beforeAll(async () => { | ||
| 35 | 39 | if (!repo.ok) throw new Error(repo.error); |
| 36 | 40 | const rec = createAppRecord(app.db, { repoId: repo.value.id, slug: "notes", access: "public" }); |
| 37 | 41 | if (!rec.ok) throw new Error(rec.error); |
| 42 | appRecordId = rec.value.id; | |
| 38 | 43 | const where = envDataDir(dir, rec.value.id, "production"); |
| 39 | 44 | mkdirSync(where, { recursive: true }); |
| 40 | 45 | const d = new Database(join(where, "app.db")); |
| @@ -92,3 +97,61 @@ test("an admin sees forge.db among the files", async () => { | ||
| 92 | 97 | expect(tui.screen.toText()).toContain("forge"); |
| 93 | 98 | tui.stop?.(); |
| 94 | 99 | }); |
| 100 | ||
| 101 | const press = async (tui: ReturnType<typeof open>["app"], ...keys: string[]) => { | |
| 102 | for (const key of keys) { | |
| 103 | tui.dispatch({ type: "key", key, ctrl: false, alt: false, shift: false }); | |
| 104 | await settleApp(tui); | |
| 105 | } | |
| 106 | }; | |
| 107 | const type = (tui: ReturnType<typeof open>["app"], text: string) => press(tui, ...text.split("")); | |
| 108 | ||
| 109 | test("a writer sorts and filters a table, and runs read-only SQL with secrets masked", async () => { | |
| 110 | const d = new Database(join(envDataDir(dir, appIdOf(), "production"), "app.db")); | |
| 111 | d.exec("INSERT INTO users VALUES (2, 'zed-visible', 'tok-secret-2')"); | |
| 112 | d.close(); | |
| 113 | const { app: tui, session } = open("alice"); | |
| 114 | tui.start(); | |
| 115 | tui.ctx.push(dataScreen(session, tui.ctx)); | |
| 116 | await settleApp(tui); | |
| 117 | await press(tui, "enter"); | |
| 118 | await press(tui, "x"); | |
| 119 | await type(tui, "SELECT api_token, name FROM users"); | |
| 120 | await press(tui, "enter"); | |
| 121 | await new Promise((r) => setTimeout(r, 300)); | |
| 122 | await settleApp(tui); | |
| 123 | let text = tui.screen.toText(); | |
| 124 | expect(text).toContain("zed-visible"); | |
| 125 | expect(text).not.toContain("tok-secret"); | |
| 126 | await press(tui, "esc", "enter"); | |
| 127 | await press(tui, "s"); | |
| 128 | await type(tui, "-name"); | |
| 129 | await press(tui, "enter"); | |
| 130 | text = tui.screen.toText(); | |
| 131 | expect(text.indexOf("zed-visible")).toBeLessThan(text.indexOf("ann-visible")); | |
| 132 | await press(tui, "f"); | |
| 133 | await type(tui, "name=ann"); | |
| 134 | await press(tui, "enter"); | |
| 135 | text = tui.screen.toText(); | |
| 136 | expect(text).toContain("ann-visible"); | |
| 137 | expect(text).not.toContain("zed-visible"); | |
| 138 | tui.stop?.(); | |
| 139 | }); | |
| 140 | ||
| 141 | test("a reader and a guest get no files, so no SQL either", () => { | |
| 142 | for (const who of ["bob", undefined]) { | |
| 143 | const { app: tui, session } = open(who); | |
| 144 | expect(listSources(app.db, dir, session.user)).toEqual([]); | |
| 145 | tui.stop?.(); | |
| 146 | } | |
| 147 | }); | |
| 148 | ||
| 149 | test("the forge database is among an admin's files, and SQL on it is masked", async () => { | |
| 150 | const forge = listSources(app.db, dir, getUserByName(app.db, "owner1")!).find((s) => s.kind === "forge")!; | |
| 151 | const r = await query(forge, "SELECT password_hash FROM user"); | |
| 152 | expect(r.ok).toBe(true); | |
| 153 | const hash = app.db.query<{ h: string }, []>("SELECT password_hash AS h FROM user LIMIT 1").get()!.h; | |
| 154 | expect(hash.length).toBeGreaterThan(10); | |
| 155 | expect(JSON.stringify(r)).not.toContain(hash); | |
| 156 | expect(JSON.stringify(r)).toContain("••••••"); | |
| 157 | }); |
Both doors: no src/tui/ screen change accompanies this. The terminal needs the app-scoped tab, /admin/data for admins, sort/filter and the read-only SQL box, gated identically.