Public: anyone can read this; the crew can change it.

A CSV, TSV or JSON-lines file over 5 MB opens its first 1,000 rows and says how many it did not show #13494

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A large CSV, TSV or JSON-lines file now opens from a bounded prefix of its bytes. It draws its first 1,000 rows and says how many it did not show. What changed Bounded read: a table file over 2 MB is read as its first 2 MB, cut back to the last full line so no half line is drawn.

Notes

shipped production is serving it deploy 8e1361f

opened this

A large CSV, TSV or JSON-lines file now opens from a bounded prefix of its bytes. It draws its first 1,000 rows and says how many it did not show.

What changed

  • Bounded read: a table file over 2 MB is read as its first 2 MB, cut back to the last full line so no half line is drawn. That covers your "over 5 MB" case and the 2–5 MB files too. The cut happens in src/view/local.ts, using the existing blobHead pipe read. The Blob type gains partial, and kinds.ts gains CSV_PREFIX_BYTES, isTablePath and wholeLines.
  • Page text: it reads "Showing 1,000 of about N rows", where N is the row density of the prefix scaled to the file size. When the prefix itself ends inside the window it reads "of more than K rows". It also states the file size and how many bytes were read, and the summary line says "about N rows".
  • Quoted fields: if the cut lands inside a quoted field, the renderer steps back a line at a time (up to 64 lines) until the rest parses.
  • Files that used to show nothing: CSV files between 1 MB and 10 MB were previously dropped by the page's 1 MB render ceiling and showed the over-cap note, and anything over 10 MB hit the CSV cap. capped() now lets table bodies through, and RENDER_CAP.csv is lifted because the read is bounded anyway.
  • Tests: the new test/render-csv-prefix.test.ts commits generated 20 MB .csv and .jsonl files and fetches their pages.

- The source hands over a prefix of at most 2 MB that ends on a newline. - Each page shows "Showing 1,000 of about N rows" within 2 s, with RSS growth under 150 MB. - Unit cases cover the final partial line, "about" versus "more than", and a cut inside a quoted field.

  • Checks run: the new test, render-csv, blob-media-ceiling, render-kinds, fuzz-renderers and render-sqlite pass, tsc is clean, and gate.ts --fix and design-lint ran. I did not run the full suite, and the tests were not run against the unchanged code to confirm they fail without it. The gate rewrapped comments in two files; the commit includes them.

Left out

  • ?all=1 ("Show all rows") does nothing on a prefixed file; the page points to the raw file instead.
  • The row count is an estimate, so a file with very uneven row lengths can be off. It is never presented as exact.
  • Files read through another forge's origin source are unchanged.
  • I did not call keep_plan, because that tool was not available in this session.

Try it: open a CSV or .jsonl file over 2 MB in a repository's code view.

🤖 Generated with Claude Code

Closes #13465

Show 4 earlier items

7 earlier comments · 2 events
commented

I pushed nothing for this and changed nothing. No failing test has been named, so I found no defect to fix.

The head is 98d0ec030, which already includes main. Before I pushed it, bun x tsc --noEmit was clean and the four CSV test files that cover this change passed (74 tests). The forge's own check of that head was cut off or is running again, and it will check the head by itself. If it comes back red on a named test, I'll fix that.

commented

I merged main into the branch and pushed it; the merge had no conflicts, and test/muse-find.test.ts came in from main unchanged.

I ran bun x tsc --noEmit, which was clean. I also ran test/render-csv-prefix.test.ts, test/render-csv.test.ts and test/muse-find.test.ts, which passed (47 tests). I did not run the full suite; the forge's check does that on the push.

The pull itself is unchanged. A CSV, TSV or JSON-lines file over 5 MB opens its first rows and says how much it left out. I did not add a terminal-workspace counterpart.

To try it, open a CSV over 5 MB in the repository file view, or use the preview at https://lab--13494--forge-trevato-apps.open-platform.sh.

commented

<!-- look-pages b53c2a69c9e5 --> Look pages for b53c2a6, from the gate:

  • look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/csv.ts, src/web/render/kinds.ts and 1 more; the forge pictures / (src/web/render/csv.ts), /docs (src/web/render/csv.ts), /explore (src/web/render/csv.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
commented

I added a "Files over 5 MB" section to docs/features/csv-view.md, which should clear the "both doors" failure, and pushed it. bun scripts/gate.ts --fix no longer prints that rule, but it only printed its last line, so I haven't seen all of its output. I didn't re-run the tests, because this push changes only the doc. I haven't seen the forge's check on the new head.

The section covers the first-bytes read, cutting at a whole line, the "of about M rows" estimate, and that sorting and ?all=1 work over the prefix only.

I left out the terminal side. The workspace still applies its own size limit to tables, and the doc says so. The commit message carries a "No door:" line giving the same reason, in case the gate wants an explicit exemption. If you'd rather the terminal show the prefix too, that's a separate change under src/tui/.

commented

<!-- look-pages 06af960f5acf --> Look pages for 06af960, from the gate:

  • look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/csv.ts, src/web/render/kinds.ts and 1 more; the forge pictures / (src/web/render/csv.ts), /docs (src/web/render/csv.ts), /explore (src/web/render/csv.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
commented

A table file over 5 MB (CSV, TSV or JSON-lines) now opens its first 1,000 rows and says how many rows it did not show. That is unchanged from the last push. The only new thing here is an empty commit that answers the look-pages note.

  • What the feedback asked for: nothing needed fixing. The gate's note only inferred which pages to picture, and it says a Looks: or No look: line in a commit message overrides that.
  • What I did: I added a No look: commit saying the prefix view lives on a repository file page. No page in app.looks reaches that view. The 26 inferred pages only import the shared renderer, and its markup for files under the limit is unchanged.
  • If you disagree: if you want the prefix view pictured, a looks route for a sample large table would have to be added to forge.json. I didn't add one.
  • Tests and checks: I ran none this turn, since no code changed. The check that covers this change is the forge's own run on the push.

To try it, open a CSV, TSV or JSONL file over 5 MB on a repository file page in the preview at https://lab--13494--forge-trevato-apps.open-platform.sh.

Changes requested — before the latest push

The prefix read works and is tested, but three things block it. 1) Both doors: the terminal workspace still refuses large tables. The pull says so itself, and the rule is that what the page does the terminal does too. Open the prefix in src/tui/ through the same Blob.partial path, with the same note. 2) Direction: this is a feature pull while the pipeline items (#6534, #6611, #5276 and the rest) are open. Say in the pull why it goes ahead of them, or park it. 3) RENDER_CAP.csv is lifted to 4000 MB for every source, not just the local one that cuts a prefix. Keep the lift scoped to sources that return partial, so another source cannot hand the renderer a whole huge body. Smaller: the note says the same thing as the summary and then the file size twice, so shorten it to one sentence; the quoted-field retry re-parses up to 2 MB 64 times, so cut back to a safe line in one step; sortColumn decodes the whole body a second time; and the 150 MB RSS and 2 s limits in the test are machine-dependent and will flake, so give them margin or measure the bytes read. Accessibility and layout are fine: the note is plain text in the page with no JavaScript needed.

commented

A CSV, TSV or JSON-lines file over 5 MB opens its first 1,000 rows on the page and says what it cut. The terminal workspace now opens the same prefix. Tests pass for the CSV renderer, the prefix tests, and the terminal workspace; tsc and the design lint are clean. I did not run the full suite; the forge runs that on push.

What changed in this round:

  • Terminal: the workspace asks for the same prefix and shows a toast: "Showing the first X of Y, read-only. The rest is in the raw file." It refuses e and the external editor on that tab. A new test opens a roughly 7 MB CSV through the terminal. I dropped the "terminal keeps its own limit" line from docs/features/csv-view.md.
  • Cap scope: RENDER_CAP.csv is back to 10 MB. rendererFor lifts it only for a body the source marked partial, and a test covers both cases.
  • Prefix is opt-in: I found a problem you didn't list. A partial blob handed out by default would have reached edit.ts, config.ts and the API, and a save could have written back a truncated file. The prefix is now only returned when a reader calls blob(ref, path, { prefix: true }). Only the file page and the terminal do. Every other reader gets too_large as before, and a test covers that.
  • Quoted-field retry: the loop of up to 64 re-parses is replaced by safeLines, one pass that cuts at the last line end outside a quote.
  • Decoding: csv() now decodes the body once. The file page's cache-key lookup in repo.ts (sortColumn) still decodes it once more; I left that, because decoding a slice could make the key disagree with the render.
  • Note: one sentence, "Showing 1,000 of about 50,000 rows. The rest is in the raw file."
  • Test limits: the 150 MB RSS and 2 s limits are gone. The test now asserts the page is under 2 MB and that the source returns at most CSV_PREFIX_BYTES ending on a line break.

Direction (your point 2): I have no basis to say why this goes ahead of #6534, #6611 and #5276. I haven't touched those, and the pull doesn't depend on them. Park it or merge it as you decide; that's yours to call.

Not done:

  • A table between 2 and 5 MB now shows a 2 MB prefix where it used to render whole.
  • JSON-lines files open as plain source in the terminal. The terminal has no table view for them.
  • Sorting works over the prefix only.

Look: No look: again. No page in app.looks reaches a repository file page of a table over the size limit.

Approved

Earlier asks are met: the prefix is opt-in per reader (editors still get too_large), the cap is lifted only for a body the source marked partial, the quoted-field retry loop is one pass (safeLines), the note is one sentence, the flaky RSS/clock limits are gone, and the terminal opens the same prefix read-only with a toast and refuses e and the external editor on it. The how-to is in docs/features/csv-view.md, not README. The note reuses the existing muted csv-more paragraph and works with JavaScript off. Two things I would do next: (1) rename the test whose title ends in 'and ' (test/render-csv-prefix.test.ts, the per-file loop), and (2) have the cache-key lookup in repo.ts stop decoding the whole prefix a second time. Non-blocking: a 2-5 MB table now shows a 2 MB prefix where it used to render whole; that is stated in the pull and acceptable.

commented

Left #13465 open: 1 unticked item remain (memory stays bounded; a test feeds a generated 20 MB file under a time budget).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs