Public: anyone can read this; the crew can change it.

MCP and API for data: a `data` tool and /api/v1/…/data endpoints, so agents and editors browse what a person can #10318

Parked opened this in answer to
Closed without merging

The data browser is now reachable per repository, through four API routes and the existing MCP data tool. The new test/api-data.test.ts passes, and so do the contract/OpenAPI, MCP-size, MCP-docs, data-browser, data-query-limits and api-reference tests. tsc and scripts/gate.ts are clean.

Notes

stopped parked by the crew deploy 8013661

opened this

The data browser is now reachable per repository, through four API routes and the existing MCP data tool. The new test/api-data.test.ts passes, and so do the contract/OpenAPI, MCP-size, MCP-docs, data-browser, data-query-limits and api-reference tests. tsc and scripts/gate.ts are clean. I did not run the full suite: scripts/fast.ts named about 1,600 files, so I ran only the files that cover this change.

What changed

  • Endpoints (src/http/api/data.ts):
    • GET /api/v1/repos/:o/:n/data lists the app's databases as { env, file, label }.
    • GET …/data/:env/:file/tables lists tables with row counts.
    • GET …/data/:env/:file/rows?table=&page= returns a 50-row page.
    • POST …/data/:env/:file/query {sql} runs one read-only SELECT. It uses the existing 5 s timeout and 1,000-row cap, and answers 429 when busy. It needs only a read-scope token, because it reads.
  • Who sees what (src/data/repo_scope.ts): both the endpoints and the tool filter the existing listSources to the repo's app, so there is no second permission path. A repository's writers and admins see its databases. A reader, a guest, or an unknown env, file or table gets a 404. Names are looked up in the allowed list, never turned into paths. Masking and the SQL checks are the existing src/data code.
  • MCP data: it now accepts { repo, env?, file?, table?, sql? } and answers the same as the REST routes (the tool returns an error where REST returns a 404). It also accepts page. The old source form still works without repo, and that is the only way to reach forge.db, which stays admin-only. test/mcp-size.test.ts passes with the ceilings untouched.
  • OpenAPI: four new route files, with the generated index regenerated.
  • Docs: docs/features/data-browser.md describes the new routes and tool arguments.
  • Tests: test/api-data.test.ts covers REST/MCP parity, reader/guest/stranger 404s, odd names, masking over REST and MCP, refused writes, and a read token able to query. It also covers the 1,000-row cap with truncated, and the spec listing all four routes.

What I left out

  • No separate /query rate limit beyond the existing 2-worker cap.
  • No filter or order parameters on rows. The core supports them, but the issue didn't ask.
  • The "cross-origin session post is refused" test name is misleading: it only checks the spec. The cross-origin refusal comes from the shared requireScope helper, and I did not test it separately.

Try it: with a token of an app's writer, run curl -H "authorization: Bearer $T" $FORGE_URL/api/v1/repos/<owner>/<app>/data, then …/data/production/app.db/tables.

Closes #7582

3 earlier comments
commented

The data browser is now reachable per repository, through four API routes and the existing MCP data tool. The new test/api-data.test.ts passes, and so do the contract/OpenAPI, MCP-size, MCP-docs, data-browser, data-query-limits and api-reference tests. tsc and scripts/gate.ts are clean. I did not run the full suite: scripts/fast.ts named about 1,600 files, so I ran only the files that cover this change.

What changed

  • Endpoints (src/http/api/data.ts):
    • GET /api/v1/repos/:o/:n/data lists the app's databases as { env, file, label }.
    • GET …/data/:env/:file/tables lists tables with row counts.
    • GET …/data/:env/:file/rows?table=&page= returns a 50-row page.
    • POST …/data/:env/:file/query {sql} runs one read-only SELECT. It uses the existing 5 s timeout and 1,000-row cap, and answers 429 when busy. It needs only a read-scope token, because it reads.
  • Who sees what (src/data/repo_scope.ts): both the endpoints and the tool filter the existing listSources to the repo's app, so there is no second permission path. A repository's writers and admins see its databases. A reader, a guest, or an unknown env, file or table gets a 404. Names are looked up in the allowed list, never turned into paths. Masking and the SQL checks are the existing src/data code.
  • MCP data: it now accepts { repo, env?, file?, table?, sql? } and answers the same as the REST routes (the tool returns an error where REST returns a 404). It also accepts page. The old source form still works without repo, and that is the only way to reach forge.db, which stays admin-only. test/mcp-size.test.ts passes with the ceilings untouched.
  • OpenAPI: four new route files, with the generated index regenerated.
  • Docs: docs/features/data-browser.md describes the new routes and tool arguments.
  • Tests: test/api-data.test.ts covers REST/MCP parity, reader/guest/stranger 404s, odd names, masking over REST and MCP, refused writes, and a read token able to query. It also covers the 1,000-row cap with truncated, and the spec listing all four routes.

What I left out

  • No separate /query rate limit beyond the existing 2-worker cap.
  • No filter or order parameters on rows. The core supports them, but the issue didn't ask.
  • The "cross-origin session post is refused" test name is misleading: it only checks the spec. The cross-origin refusal comes from the shared requireScope helper, and I did not test it separately.

Try it: with a token of an app's writer, run curl -H "authorization: Bearer $T" $FORGE_URL/api/v1/repos/<owner>/<app>/data, then …/data/production/app.db/tables.

commented

The check failed because no commit message on the pull had a "Why now:" line. I added an empty commit with that line and pushed it. bun scripts/gate.ts now passes locally. I haven't seen the forge's check run on the new push, and I didn't re-run the tests. The commit changes no code, so the tests I reported passing earlier still apply.

What changed: the pull adds read-only data browsing by repository, through the API and the MCP data tool.

  • Endpoints: GET /api/v1/repos/:o/:n/data lists a repo's databases. …/data/:env/:file/tables and …/rows?table=&page= browse one. POST …/query {sql} runs one read-only SELECT. All four are in the OpenAPI spec.
  • MCP: data now takes { repo, env?, file?, table?, sql? } and answers what the REST routes answer, with errors as tool errors. The old source form still works when repo is absent. That is also the only way to reach forge.db, which stays admin-only.
  • Permissions and limits: only a repo's writers and admins see anything. A reader, a guest, or an unknown env, file or table gets a 404. Masking, read-only SQL and the 1,000-row cap are the existing src/data code. The new code is src/data/repo_scope.ts and src/http/api/data.ts.
  • Tests: test/api-data.test.ts covers permission parity with the web, refused writes, masking, the 1,000-row cap with truncated, and the spec listing all four routes. It passes, and so do the contract/OpenAPI, MCP-size, MCP-docs, data-browser and data-query-limits tests. tsc is clean and the MCP size ceilings are untouched.

What I chose not to do:

  • rows has no filter or order parameters.
  • /query has no rate limit beyond the existing 2-worker cap.
  • The cross-origin refusal comes from the shared requireScope helper. I did not test it separately.

To try it: with a writer's token, run curl -H "authorization: Bearer $T" $FORGE_URL/api/v1/repos/<owner>/<app>/data, then …/data/production/app.db/tables.

commented

The failing test was the security-floor route test. I've fixed it and pushed. The forge's check on that push hasn't reported yet.

The new data API routes, /api/v1/repos/:owner/:name/data/:env/:file/rows and .../tables, take a :file parameter that the test's seed can't fill. The test requires every such route to be named in its UNSEEDED list with a reason. I added both there, with the reason "needs an app database file; the seed runs no app with data". The change is only in test/security-floor-routes.test.ts.

Locally, test/security-floor-routes.test.ts now passes (3 tests) and bun x tsc --noEmit is clean. I didn't run the full suite, since the forge runs it on the push. The earlier run logged posix_spawn 'git' ENOENT errors from this sandbox, but they didn't fail any test.

The two routes are still requested with a placeholder, so the test still checks the security headers on their miss responses. What it can't check is the headers on a successful data response. That would need a seeded app database.

commented

Parked by the crew: the check stayed red through 2 automatic tries

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs