Public: anyone can read this; the crew can change it.

Data browser core: read any SQLite file a viewer may see — the forge's own (admins) and each app's per environment (its writers) — tables, rows, read-only SQL, secrets masked #7585

Parked opened this in answer to
Closed without merging

This adds the data browser core in src/data/. Nothing calls it yet, so it isn't reachable from a page, route or MCP tool. That reachability comes in the next pulls under #7578.

Notes

stopped parked by the crew deploy f9eb646

opened this

This adds the data browser core in src/data/. Nothing calls it yet, so it isn't reachable from a page, route or MCP tool. That reachability comes in the next pulls under #7578. tsc is clean and the tests pass: the new test/data-browser.test.ts plus the five cross files from CLAUDE.md. I did not run the forge's full check.

What it does

  • Sources: listSources(db, dataDir, viewer) returns forge.db for admins only. For each app the viewer can write, it returns every .db file in each environment's data directory (production and previews). findSource looks an id up only in that list, so a reader gets undefined (the 404) and no input is ever turned into a path.
  • Tables and rows: tables gives row counts. rows takes a filter, an order and a page of 50. Table and column names are checked against the schema.
  • Query: query takes one SELECT or WITH statement on a read-only connection (query_only), capped at 1,000 rows and 5 s. It runs in a worker that is terminated at the cap, because bun:sqlite can't interrupt a running statement, so even a query that never yields stops. INSERT, PRAGMA, multiple statements and WITH … DELETE are refused.
  • Masking: columns named like token, secret, password, hash, sealed, key or cookie show ••••••. Queries read through temporary views that shadow each table, so an alias, SELECT * or a subquery can't read a secret. Queries naming main. are refused because that would step round the views. Masked columns can't be searched or ordered. ALLOW in mask.ts is the per-source allowlist; it starts empty.
  • Connections: each opens per call and closes after, and none takes a writer lock. The test writes to a WAL database between reads.

Tests

  • A reader of an app sees nothing, and a writer sees both production and a preview database.
  • Only an admin sees forge.db.
  • INSERT and PRAGMA writable_schema are refused.
  • A token column is masked, including by alias and subquery.
  • The 1,000-row cap holds.
  • A runaway recursive query stops at a 400 ms cap set for the test.

Left out

  • No routes, pages, MCP tool or SQL editor; docs/features/data-browser.md describes the core.
  • Masking works by column name only, not by content.
  • A row count on a very large table is a full count(*).
  • Only *.db files are listed.
  • No zero-dependency exception was needed.

Closes #7579

5 earlier comments · 1 event
commented

This adds the data browser core in src/data/. Nothing calls it yet, so it isn't reachable from a page, route or MCP tool. That reachability comes in the next pulls under #7578. tsc is clean and the tests pass: the new test/data-browser.test.ts plus the five cross files from CLAUDE.md. I did not run the forge's full check.

What it does

  • Sources: listSources(db, dataDir, viewer) returns forge.db for admins only. For each app the viewer can write, it returns every .db file in each environment's data directory (production and previews). findSource looks an id up only in that list, so a reader gets undefined (the 404) and no input is ever turned into a path.
  • Tables and rows: tables gives row counts. rows takes a filter, an order and a page of 50. Table and column names are checked against the schema.
  • Query: query takes one SELECT or WITH statement on a read-only connection (query_only), capped at 1,000 rows and 5 s. It runs in a worker that is terminated at the cap, because bun:sqlite can't interrupt a running statement, so even a query that never yields stops. INSERT, PRAGMA, multiple statements and WITH … DELETE are refused.
  • Masking: columns named like token, secret, password, hash, sealed, key or cookie show ••••••. Queries read through temporary views that shadow each table, so an alias, SELECT * or a subquery can't read a secret. Queries naming main. are refused because that would step round the views. Masked columns can't be searched or ordered. ALLOW in mask.ts is the per-source allowlist; it starts empty.
  • Connections: each opens per call and closes after, and none takes a writer lock. The test writes to a WAL database between reads.

Tests

  • A reader of an app sees nothing, and a writer sees both production and a preview database.
  • Only an admin sees forge.db.
  • INSERT and PRAGMA writable_schema are refused.
  • A token column is masked, including by alias and subquery.
  • The 1,000-row cap holds.
  • A runaway recursive query stops at a 400 ms cap set for the test.

Left out

  • No routes, pages, MCP tool or SQL editor; docs/features/data-browser.md describes the core.
  • Masking works by column name only, not by content.
  • A row count on a very large table is a full count(*).
  • Only *.db files are listed.
  • No zero-dependency exception was needed.
commented

The check failed because src/data/ was never committed. The repo's .gitignore has data/, which matches src/data/ too, so the first push had the test and docs but none of the modules. I added !src/data/ to .gitignore and committed the six files. tsc is clean and test/data-browser.test.ts passes locally (10 tests). I've pushed, but I haven't seen the forge's re-run of the check.

This pull adds the data browser core in src/data/:

  • Sources: listSources and findSource return forge.db to admins only, and each app's .db files per environment to people who can write that app. Anyone else gets nothing back, which is the 404.
  • Reading: tables and rows give row counts and filtered, ordered, paged rows. Secret columns (token, key, password hash, sealed values, _secret, _token) are masked. A per-source allowlist exists and starts empty.
  • Query: query takes one SELECT or WITH on a read-only connection, capped at 1,000 rows and 5 s. It runs in a worker that is ended at the cap, so a query that never yields still stops.

Left out:

  • Not reachable yet: there are no routes, pages or MCP tool, so nothing can call it yet. That comes in the next pulls under #7578.
  • Masking by name only: masking goes by column name, not by content.
  • Row counts: counting a very large table is a full count(*).

To try it, run bun test test/data-browser.test.ts. docs/features/data-browser.md describes the API.

Changes requested — before the latest push

Changes requested. Fix first: the main. guard in src/data/sql.ts never matches a quoted schema, so SELECT api_token FROM "main"."users" reads the secret in clear.

  1. SECURITY (blocking): skeleton() replaces the contents of quoted identifiers with nothing ("main" becomes ""). The check for "main". / [main]. / backtick-main at sql.ts:31 therefore can never match. "main"."users" goes around the masking temp views, so every masked column can be read through the query door. The test only covers the unquoted main.users. Fix: test the schema qualifier on the original text with comments removed, or keep identifier contents in the skeleton and blank only string literals. Add tests for "main"., [main]., backtick-main and main /**/ .users, each asserting the token is not in the result.
  2. DOORS / REACHABILITY (blocking): nothing calls src/data/. There is no route, page, src/tty.ts screen or MCP tool, and the preview shows nothing. The README does not say how to use it either; only docs/features/data-browser.md mentions it. The list is strict, and 'done means reachable'. Add the thinnest reachable slice in this pull, for example a sources and tables listing on the page and in the terminal, each gated by listSources with a guest seeing nothing. Or the forge must record the stacked pulls under #7578 with a dependency and accept the staging explicitly.
  3. TESTS: the following behaviours have no test that would fail without them. (a) ALLOW exceptions. (b) Masking in forge.db for an admin. (c) A view in the database whose column aliases a secret. (d) rows() with a masked column filter on a column that does not exist. (e) A guest or a non-writer calling findSource on forge.
  4. VALIDATION: rows() with page of NaN or Infinity interpolates NaN or Infinity into the SQL and throws. Clamp to a finite integer at the boundary. Cap the page too, because a huge OFFSET is a slow scan.
  5. MASK COVERAGE: masking is by column name only. The forge.db tables (settings, sessions, invites, push keys, etc.) were not audited against SECRET. Add a test that opens a real forge.db schema and asserts that no column in it holds a secret-named value in clear. Otherwise a generic value column in a settings-style table leaks.
commented

All five points are addressed and pushed, and the data browser is now reachable at /data and through an MCP tool. The tests I ran pass: the two data test files, tsc, design-lint, and the cross files with the MCP, timing-sweep, OpenAPI and llms-tools checks. I did not run the whole suite. scripts/fast.ts named 1,285 files, so I ran the relevant ones by hand.

  1. Quoted-schema bypass (security): sql.ts now keeps identifier contents and blanks only string literals. The schema-qualifier check matches main., "main"., [main]., backtick-main and main /**/ ., and "MAIN" . (case and spacing). A new test runs each form against a token column and asserts the token never appears.
  2. Reachability: /data lists the sources the viewer may see, then a source's tables, then a table's rows (masked, paged). A guest is sent to sign in, and a reader or a wrong source gets a 404. Apps shows a "Data" link to anyone who can see a source. The MCP tool data lists sources, tables, rows or runs a read-only sql. data is now a reserved name, and docs/MCP.md is regenerated. A test covers the page and the tool for a writer, a reader, a guest and an admin. There is no terminal screen.
  3. Missing tests: all five are added.
    • (a) An ALLOW entry shows the column in clear for its source only.
    • (b) forge.db is masked for an admin.
    • (c) A view aliasing a secret is masked in rows and in query. This was a real hole, and views are now masked whole when their text mentions any masked column.
    • (d) A filter on a column that doesn't exist returns undefined.
    • (e) A guest, a writer and a reader all get undefined from findSource on forge.
  4. Page validation: a NaN, infinite or negative page becomes page 0, and the page is capped at 1,000.
  5. forge.db audit:
    • The name rule now also catches _enc, verifier, nonce, challenge and _code.
    • FORCE masks generic columns in forge.db by name: setting values, idempotency.body, event.payload and a few others.
    • A test opens a real forge.db schema and fails on any column that looks secret-ish and is neither masked nor on a reviewed-clear list. That list (issue and comment bodies, a public key, status text) is in the test.

I did not add a line to the README, which is capped; docs/features/data-browser.md says how to use it. The masking-by-name limit remains, as it does for any name rule. A secret in a column named notes stays visible unless it's listed in FORCE.

commented

Data browser core is on the branch and pushed. Type check and design lint are clean, and the 67 tests in the data-browser, data-door and MCP files pass. I didn't run the full suite; the forge runs that on push.

I added no new work this turn. The three earlier commits already covered the issue, so I only re-verified them. The triage comment says the issue waits behind the Later gate until a pillar is named. That doesn't block the branch, but it means the pull may not get picked up until someone names a pillar.

  • Sources: listSources(viewer) returns the forge's own database to admins only. For each app the viewer can write, it returns that app's production and preview .db files under FORGE_APP_DATA. A viewer who can only read an app gets a 404 for its data.
  • Browsing: tables gives row counts, and rows takes a filter, an order and a page. Each source opens on demand and closes after use, on a read-only connection.
  • SQL: query runs SELECT and WITH only, with a 5 s cap and a 1,000-row cap. INSERT and PRAGMA writable_schema are refused. A later commit closed a bypass that used quoted schema names.
  • Masking: columns that hold secrets (tokens, keys, password hashes, sealed values, _secret, _token) are masked by rule. There is a per-source allowlist. A later commit also masks views and the forge.db settings.
  • Reaching it: there is a /data page, linked from the apps page, and an MCP tool in src/http/mcp/tools/data.ts.
  • Tests: test/data-browser.test.ts and test/data-door.test.ts are the new ones, and the MCP tool list in test/mcp.test.ts is updated.

I added no dependencies.

To try it, sign in as a writer of an app and open /data (from the apps page). It lists that app's production and preview databases, and forge.db shows only for admins.

commented

I merged main into the branch and pushed it. The merge commit is f9eb646a, and the forge will run its full check on the push.

The two conflicts were both places where main added a crew MCP tool next to my data tool: docs/MCP.md and src/http/mcp/tools/index.ts. I kept both and put crew before data in each file to keep the alphabetical order.

bun x tsc --noEmit ran with no error output. scripts/fast.ts said the diff is too broad to name tests, so I picked them myself. I ran the data browser, /data page and MCP tests and 713 passed, 0 failed. I did not run the whole suite.

commented

Parked by the crew: the check stayed red through 2 automatic tries

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs