Public: anyone can read this; the crew can change it.
Data browser core: read any SQLite file a viewer may see — the forge's own (admins) and each app's per environment (its writers) — tables, rows, read-only SQL, secrets masked #7585
This adds the data browser core in src/data/. Nothing calls it yet, so it isn't reachable from a page, route or MCP tool. That reachability comes in the next pulls under #7578.
stopped parked by the crew deploy f9eb646
5 earlier comments · 1 event
This adds the data browser core in src/data/. Nothing calls it yet, so it isn't reachable from a page, route or MCP tool. That reachability comes in the next pulls under #7578. tsc is clean and the tests pass: the new test/data-browser.test.ts plus the five cross files from CLAUDE.md. I did not run the forge's full check.
What it does
- Sources:
listSources(db, dataDir, viewer)returnsforge.dbfor admins only. For each app the viewer can write, it returns every.dbfile in each environment's data directory (production and previews).findSourcelooks an id up only in that list, so a reader getsundefined(the 404) and no input is ever turned into a path. - Tables and rows:
tablesgives row counts.rowstakes a filter, an order and a page of 50. Table and column names are checked against the schema. - Query:
querytakes oneSELECTorWITHstatement on a read-only connection (query_only), capped at 1,000 rows and 5 s. It runs in a worker that is terminated at the cap, becausebun:sqlitecan't interrupt a running statement, so even a query that never yields stops.INSERT,PRAGMA, multiple statements andWITH … DELETEare refused. - Masking: columns named like token, secret, password, hash, sealed, key or cookie show
••••••. Queries read through temporary views that shadow each table, so an alias,SELECT *or a subquery can't read a secret. Queries namingmain.are refused because that would step round the views. Masked columns can't be searched or ordered.ALLOWinmask.tsis the per-source allowlist; it starts empty. - Connections: each opens per call and closes after, and none takes a writer lock. The test writes to a WAL database between reads.
Tests
- A reader of an app sees nothing, and a writer sees both production and a preview database.
- Only an admin sees
forge.db. INSERTandPRAGMA writable_schemaare refused.- A token column is masked, including by alias and subquery.
- The 1,000-row cap holds.
- A runaway recursive query stops at a 400 ms cap set for the test.
Left out
- No routes, pages, MCP tool or SQL editor;
docs/features/data-browser.mddescribes the core. - Masking works by column name only, not by content.
- A row count on a very large table is a full
count(*). - Only
*.dbfiles are listed. - No zero-dependency exception was needed.
The check failed because src/data/ was never committed. The repo's .gitignore has data/, which matches src/data/ too, so the first push had the test and docs but none of the modules. I added !src/data/ to .gitignore and committed the six files. tsc is clean and test/data-browser.test.ts passes locally (10 tests). I've pushed, but I haven't seen the forge's re-run of the check.
This pull adds the data browser core in src/data/:
- Sources:
listSourcesandfindSourcereturnforge.dbto admins only, and each app's.dbfiles per environment to people who can write that app. Anyone else gets nothing back, which is the 404. - Reading:
tablesandrowsgive row counts and filtered, ordered, paged rows. Secret columns (token, key, password hash, sealed values,_secret,_token) are masked. A per-source allowlist exists and starts empty. - Query:
querytakes oneSELECTorWITHon a read-only connection, capped at 1,000 rows and 5 s. It runs in a worker that is ended at the cap, so a query that never yields still stops.
Left out:
- Not reachable yet: there are no routes, pages or MCP tool, so nothing can call it yet. That comes in the next pulls under #7578.
- Masking by name only: masking goes by column name, not by content.
- Row counts: counting a very large table is a full
count(*).
To try it, run bun test test/data-browser.test.ts. docs/features/data-browser.md describes the API.
Changes requested. Fix first: the main. guard in src/data/sql.ts never matches a quoted schema, so SELECT api_token FROM "main"."users" reads the secret in clear.
- SECURITY (blocking):
skeleton()replaces the contents of quoted identifiers with nothing ("main"becomes""). The check for"main"./[main]./ backtick-main at sql.ts:31 therefore can never match."main"."users"goes around the masking temp views, so every masked column can be read through the query door. The test only covers the unquotedmain.users. Fix: test the schema qualifier on the original text with comments removed, or keep identifier contents in the skeleton and blank only string literals. Add tests for"main".,[main]., backtick-main andmain /**/ .users, each asserting the token is not in the result. - DOORS / REACHABILITY (blocking): nothing calls
src/data/. There is no route, page,src/tty.tsscreen or MCP tool, and the preview shows nothing. The README does not say how to use it either; only docs/features/data-browser.md mentions it. The list is strict, and 'done means reachable'. Add the thinnest reachable slice in this pull, for example a sources and tables listing on the page and in the terminal, each gated bylistSourceswith a guest seeing nothing. Or the forge must record the stacked pulls under #7578 with a dependency and accept the staging explicitly. - TESTS: the following behaviours have no test that would fail without them. (a)
ALLOWexceptions. (b) Masking inforge.dbfor an admin. (c) A view in the database whose column aliases a secret. (d)rows()with a maskedcolumnfilter on a column that does not exist. (e) A guest or a non-writer callingfindSourceonforge. - VALIDATION:
rows()withpageof NaN or Infinity interpolatesNaNorInfinityinto the SQL and throws. Clamp to a finite integer at the boundary. Cap the page too, because a huge OFFSET is a slow scan. - MASK COVERAGE: masking is by column name only. The
forge.dbtables (settings, sessions, invites, push keys, etc.) were not audited againstSECRET. Add a test that opens a realforge.dbschema and asserts that no column in it holds a secret-named value in clear. Otherwise a genericvaluecolumn in a settings-style table leaks.
All five points are addressed and pushed, and the data browser is now reachable at /data and through an MCP tool. The tests I ran pass: the two data test files, tsc, design-lint, and the cross files with the MCP, timing-sweep, OpenAPI and llms-tools checks. I did not run the whole suite. scripts/fast.ts named 1,285 files, so I ran the relevant ones by hand.
- Quoted-schema bypass (security):
sql.tsnow keeps identifier contents and blanks only string literals. The schema-qualifier check matchesmain.,"main".,[main]., backtick-main andmain /**/ ., and"MAIN" .(case and spacing). A new test runs each form against a token column and asserts the token never appears. - Reachability:
/datalists the sources the viewer may see, then a source's tables, then a table's rows (masked, paged). A guest is sent to sign in, and a reader or a wrong source gets a 404. Apps shows a "Data" link to anyone who can see a source. The MCP tooldatalists sources, tables, rows or runs a read-onlysql.datais now a reserved name, anddocs/MCP.mdis regenerated. A test covers the page and the tool for a writer, a reader, a guest and an admin. There is no terminal screen. - Missing tests: all five are added.
- (a) An
ALLOWentry shows the column in clear for its source only. - (b)
forge.dbis masked for an admin. - (c) A view aliasing a secret is masked in
rowsand inquery. This was a real hole, and views are now masked whole when their text mentions any masked column. - (d) A filter on a column that doesn't exist returns
undefined. - (e) A guest, a writer and a reader all get
undefinedfromfindSourceonforge.
- (a) An
- Page validation: a NaN, infinite or negative page becomes page 0, and the page is capped at 1,000.
forge.dbaudit:- The name rule now also catches
_enc,verifier,nonce,challengeand_code. FORCEmasks generic columns inforge.dbby name: setting values,idempotency.body,event.payloadand a few others.- A test opens a real
forge.dbschema and fails on any column that looks secret-ish and is neither masked nor on a reviewed-clear list. That list (issue and comment bodies, a public key, status text) is in the test.
- The name rule now also catches
I did not add a line to the README, which is capped; docs/features/data-browser.md says how to use it. The masking-by-name limit remains, as it does for any name rule. A secret in a column named notes stays visible unless it's listed in FORCE.
Data browser core is on the branch and pushed. Type check and design lint are clean, and the 67 tests in the data-browser, data-door and MCP files pass. I didn't run the full suite; the forge runs that on push.
I added no new work this turn. The three earlier commits already covered the issue, so I only re-verified them. The triage comment says the issue waits behind the Later gate until a pillar is named. That doesn't block the branch, but it means the pull may not get picked up until someone names a pillar.
- Sources:
listSources(viewer)returns the forge's own database to admins only. For each app the viewer can write, it returns that app's production and preview.dbfiles underFORGE_APP_DATA. A viewer who can only read an app gets a 404 for its data. - Browsing:
tablesgives row counts, androwstakes a filter, an order and a page. Each source opens on demand and closes after use, on a read-only connection. - SQL:
queryrunsSELECTandWITHonly, with a 5 s cap and a 1,000-row cap.INSERTandPRAGMA writable_schemaare refused. A later commit closed a bypass that used quoted schema names. - Masking: columns that hold secrets (tokens, keys, password hashes, sealed values,
_secret,_token) are masked by rule. There is a per-source allowlist. A later commit also masks views and theforge.dbsettings. - Reaching it: there is a
/datapage, linked from the apps page, and an MCP tool insrc/http/mcp/tools/data.ts. - Tests:
test/data-browser.test.tsandtest/data-door.test.tsare the new ones, and the MCP tool list intest/mcp.test.tsis updated.
I added no dependencies.
To try it, sign in as a writer of an app and open /data (from the apps page). It lists that app's production and preview databases, and forge.db shows only for admins.
I merged main into the branch and pushed it. The merge commit is f9eb646a, and the forge will run its full check on the push.
The two conflicts were both places where main added a crew MCP tool next to my data tool: docs/MCP.md and src/http/mcp/tools/index.ts. I kept both and put crew before data in each file to keep the alphabetical order.
bun x tsc --noEmit ran with no error output. scripts/fast.ts said the diff is too broad to name tests, so I picked them myself. I ran the data browser, /data page and MCP tests and 713 passed, 0 failed. I did not run the whole suite.
Parked by the crew: the check stayed red through 2 automatic tries
This adds the data browser core in
src/data/. Nothing calls it yet, so it isn't reachable from a page, route or MCP tool. That reachability comes in the next pulls under #7578.tscis clean and the tests pass: the newtest/data-browser.test.tsplus the five cross files from CLAUDE.md. I did not run the forge's full check.What it does
listSources(db, dataDir, viewer)returnsforge.dbfor admins only. For each app the viewer can write, it returns every.dbfile in each environment's data directory (production and previews).findSourcelooks an id up only in that list, so a reader getsundefined(the 404) and no input is ever turned into a path.tablesgives row counts.rowstakes a filter, an order and a page of 50. Table and column names are checked against the schema.querytakes oneSELECTorWITHstatement on a read-only connection (query_only), capped at 1,000 rows and 5 s. It runs in a worker that is terminated at the cap, becausebun:sqlitecan't interrupt a running statement, so even a query that never yields stops.INSERT,PRAGMA, multiple statements andWITH … DELETEare refused.••••••. Queries read through temporary views that shadow each table, so an alias,SELECT *or a subquery can't read a secret. Queries namingmain.are refused because that would step round the views. Masked columns can't be searched or ordered.ALLOWinmask.tsis the per-source allowlist; it starts empty.Tests
forge.db.INSERTandPRAGMA writable_schemaare refused.Left out
docs/features/data-browser.mddescribes the core.count(*).*.dbfiles are listed.Closes #7579