Public: anyone can read this; the crew can change it.

A table in an app's Data tab downloads as CSV, and the same rows are readable through the data tool #12711

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A table in an app's Data tab now downloads as CSV, and the data tool returns the same rows with format: "csv". The new test/data-csv-export.test.ts passes, and so do test/mcp-size.test.ts, test/data-door.test.ts, test/api-data.test.ts and test/mcp-refusal-hints.test.ts.

Notes

shipped production is serving it deploy 77d0004

opened this

A table in an app's Data tab now downloads as CSV, and the data tool returns the same rows with format: "csv". The new test/data-csv-export.test.ts passes, and so do test/mcp-size.test.ts, test/data-door.test.ts, test/api-data.test.ts and test/mcp-refusal-hints.test.ts. bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts are clean. I did not boot the app or open the download in a browser.

  • Web: GET /:owner/:name/data/<table>.csv?source=<id> streams the table with a header row, as an attachment with nosniff. source can be left out when the app has a single database. The table's Data page has a "Download CSV" button for it.
  • Cells: Fields use RFC 4180 quoting with CRLF. Text cells starting with = + - @, tab or CR get a leading '; numbers are left alone, so -5 stays -5. Secret columns stay masked as in the browser.
  • Caps: The web download is capped at 10,000 rows and says so in an x-forge-truncated header. The MCP answer is capped at 1,000 rows and carries rows, truncated and csv.
  • MCP: format is a new argument on the existing data tool, for both the repo+env+file form and the source form. It accepts csv or json; anything else is refused. The description grew by about 30 bytes.
  • Access: Only people who can write to the app can fetch it; everyone else gets the same 404 as the Data tab. The issue says "readers", but the Data tab is writers-only today and I kept that rule rather than widen who can read an app's data.
  • Table name: It is checked against the database's real tables and never interpolated, so sqlite_master and quote-injection names return 404.
  • Not done: There is no download for the /data or /admin/data doors, only the app tab. Filters and sort aren't applied to the download; it is the whole table in rowid order. The MCP CSV can be truncated at 1,000 rows, and there is no paging for it.

To try it, open an app's Data tab as a writer, pick a table and press Download CSV.

I added a Try: commit because the pre-push hook required one.

Closes #12684

Show 1 earlier items

7 earlier comments · 1 event
commented

No change is needed, so I pushed nothing. The deploy log you sent shows the preview building, starting, answering /readyz and ending with done: https://lab--12711--forge-trevato-apps.open-platform.sh.

  • "No tests declared": forge.json has no app.test key, and the repository's tests run under check, not at deploy. My change doesn't touch this.
  • The egress refusal: preview-grace-client--lab--12711…:443 is not in this repository's egress came from the preview-seed data (seedWorld in src/spawn/preview-seed.ts). My change doesn't touch this either, and I won't add a host to forge.json egress to silence it.

The failure may be coming from the check the forge runs on my push, which I can't see from here. If it does fail, send me that log and I'll fix what it names. Locally, bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts are clean, and the touched data and MCP tests pass.

commented

The check on this pull's head fails, and it is not the preview: the deploy log you were shown is a success. These are the failing lines from check run 01m7eq1sq7kp1gpf5gx7ewfyw3 (ended 2026-11-15T04:22), quoted from its log:

gate: rule "before/after looks": this pull changes src/web/pages/data.ts, src/web/pages/data_view.ts but no commit message has a "Looks: <path>" line naming the page of app.looks that shows it; the forge takes the before/after pictures (light and dark) of that page on the preview. Or put a "No look:
gate: rule "a comment line is wrapped at 100 columns": src/data/browse.ts:86 is 111 long; rewrap it
gate: rule "a comment line is wrapped at 100 columns": src/web/pages/data.ts:85 is 101 long; rewrap it
gate: rule "a comment line is wrapped at 100 columns": test/data-csv-export.test.ts:1 is 101 long; rewrap it

Fix exactly these, then push. Nothing else is being asked.

commented

I fixed the three gate failures from the check and pushed the new commit (8400b1a28). After the commit, bun scripts/gate.ts prints nothing, which I took as passing, though I didn't capture its exit code (the rc=0 shown is tail's). bun x tsc --noEmit printed nothing, and test/data-csv-export.test.ts passes (5 tests).

  • Comment lines: I rewrapped the three comments at 100 columns, in src/data/browse.ts:86, src/web/pages/data.ts:85 and test/data-csv-export.test.ts:1. Only comments changed, no code.
  • Before/after looks: I added a No look: line to the commit message instead of a Looks: path. The Data tab lives under an app's own page, and no entry in app.looks reaches an app that has data, so there is no page to point at. If you would rather have a looks page for it, that needs a new entry in forge.json, which I haven't added.
commented

The four failing tests were bookkeeping my change left out. I fixed them and pushed d215e76e8. The four test files that were red (pages-budget, no-polling, security-floor-routes, mcp-table) now pass here, 38 tests with none failing. bun scripts/gate.ts and bun x tsc --noEmit printed nothing. I did not run the full suite; the forge's check does that on the push.

  • New route listed: the new /:owner/:name/data/:file CSV download needs an entry wherever the route walks list routes the fixture can't fill. I added one in test/no-polling.test.ts and test/security-floor-routes.test.ts, each saying it is a download and the seed has no app data. In test/pages-budget.test.ts I raised MAX_UNFIXTURED from 149 to 150.
  • MCP table: the data tool's description changed when it gained format: "csv", so I regenerated docs/MCP.md with bun scripts/mcp-table.ts.

Only tests and docs changed in this push; no app code did.

commented

I think this failure is a timeout on a loaded check machine rather than a defect in the pull, so I changed no code and pushed an empty commit (8abab3353) to re-run the check.

  • What failed: the looks-owner test waits up to 90 s for six preview pictures and gave up after 90.2 s. The same shard ran at 230 s that run, against a 61 s slowest shard in the earlier failing run.
  • Locally: test/looks-owner.test.ts passes, 22 of 22 in 10.9 s.
  • My diff: it adds a CSV download route, the data tool's format: "csv" option, and tests. It changes no look-taking code, and the test never touches /data/:file.
  • Main: the forge's own comparison shows test/looks-owner.test.ts green on main.

If the check fails the same way again, the 90 s wait in test/looks-owner.test.ts (lines 437 and 490) is the thing to look at.

Changes requested — before the latest push

The web and MCP doors work and the security core is sound (table name checked against real tables, never interpolated; secrets masked; formula-safe; 404 for non-writers). Three things block it:

  1. FIX FIRST: both doors. A person can download a table on the page but not in the terminal: add a screen in src/tui/ (entry where AGENTS.md says) that exports the same rows, gated exactly like the page, and add docs/features/<name>.md with the how-to (not README).
  2. Tests do not cover behaviour the change adds: the 10,000-row cap and x-forge-truncated header, the MCP 1,000-row cap with truncated:true, and format json are all untested; the guest assertion is only 'not 200'. Add tests that fail without them.
  3. Safety/cleanliness: the export is built whole in memory with a row cap only, so wide/blob rows can make a 10,000-row answer huge; cap by bytes too. Unlike read-only SQL, the export has no rate limit. format: json is accepted and does nothing; either implement it or drop it.

Box 2 says 'readers' but this head allows writers only, consistent with the Data tab; I accept that and tick it.

commented

I addressed all three review points and pushed them as b62671ce5.

Tests pass for everything touched (145 across the data, both-doors, feature-docs, MCP-table, no-polling and security-floor suites). bun scripts/gate.ts and bun x tsc --noEmit are clean, and design-lint printed no complaint. I did not run the full suite; the forge's check does that.

1. Terminal door and docs

  • On the Data screen, pressing e on a table or one of its rows copies the table to your clipboard as CSV. It reads through listSources, so it is gated like the page. I put it on the existing Data screen rather than a new one, because that screen already has the palette entry and tree.
  • It is capped at 1,000 rows and about 500 KB in the terminal, and the toast says when the export was cut.
  • The how-to is the new docs/features/data-csv.md, linked from docs/features/README.md.

2. Tests that fail without the behaviour

  • A table of 10,050 rows downloads 10,000 and carries x-forge-truncated: 10000; a small table carries no such header.
  • Wide rows stop at the byte cap, and the same table read with a larger byte cap returns all 100 rows.
  • The data tool stops at 1,000 rows and answers truncated: true.
  • format: "json" is now refused.
  • The guest assertion is now exactly 404, and checks the response leaks no cell data.
  • After 20 exports a minute the page answers 429 with retry-after, and the tool answers rate_limited.
  • e in the terminal copies masked CSV and copies nothing on a file.

3. Safety and cleanliness

  • Byte cap: the export now reads rows one at a time and stops at 5,000,000 characters as well as 10,000 rows, marking the result truncated.
  • Rate limit: the page, the tool and the terminal all go through one new function, exportCsv in src/data/export_csv.ts. It shares the per-person 20-a-minute budget that SQL has, and checks it before touching the database.
  • format: I dropped json; only csv is accepted.
  • Button: "Download CSV" now sits in the existing filter toolbar beside Clear, not in its own row.

I left out streaming the response, so a download is still built in memory, now bounded at about 5 MB. I also left out a separate budget for exports; they share SQL's.

commented

I fixed both gate failures and pushed 1c06274e8. bun scripts/gate.ts now prints nothing against the new head, bun x tsc --noEmit is clean, and test/tui-data.test.ts passes (14 tests). I did not run the full suite.

  • Comment wrapping: I rewrapped the comment on the export function in src/tui/screens/data.ts (line 148, 105 columns) to fit 100. Only the comment changed.
  • Pipeline-first rule: this pull adds docs/features/data-csv.md and changes nothing in the pipeline. I put a Direction: line in the commit message saying this is the feature the issue asked for, not pipeline work. The gate only counts it once it is committed, so it still showed both failures before that commit. I made no pipeline change.
commented

This pull adds a CSV export to the Data tab: a table downloads as a CSV file, with a byte cap, a rate limit, and a terminal key. The check's two failures were a stale test, not a code fault.

  • Cause: the export added an e exports a table as CSV hint to the palette's Data entry. test/palette-roster.test.ts still expected the old text in the two roster tests ("a site-admin person is offered the whole roster" and "a site-admin agent is offered the roster without the person-only entries").
  • Change: I updated the expected hint on both lines in that test. The test was wrong, not the code.
  • Checks: palette-roster passes (15 of 15) and bun x tsc --noEmit is clean. I did not run the full suite, since the forge runs it on push.

To try it, open an app's Data tab, pick a table and export it as CSV. In the terminal, open Data, pick a table and press e.

Approved

All earlier asks are met. The terminal door is the e key on the Data screen, gated through listSources, with its how-to in docs/features/data-csv.md and nothing added to README. A 5 MB byte cap and a per-person budget shared with SQL now bound the export. format accepts only csv. The button sits in the existing toolbar as an <a>. Tests now cover the row cap, the byte cap, the MCP cap, rate limiting and the terminal key. Non-blocking: the page download is built in memory (bounded at about 5 MB), so streaming could come later. Access is writers only, which is stricter than the issue says; I accept that because it matches the Data tab.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs