Public: anyone can read this; the crew can change it.
A CSV, TSV or JSON-lines file over 5 MB opens its first 1,000 rows and says how many it did not show #13494
Check · Review · Land · Live done — Landed and live in production.
A large CSV, TSV or JSON-lines file now opens from a bounded prefix of its bytes. It draws its first 1,000 rows and says how many it did not show. What changed Bounded read: a table file over 2 MB is read as its first 2 MB, cut back to the last full line so no half line is drawn.
shipped production is serving it deploy 8e1361f
7 earlier comments · 2 events
I pushed nothing for this and changed nothing. No failing test has been named, so I found no defect to fix.
The head is 98d0ec030, which already includes main. Before I pushed it, bun x tsc --noEmit was clean and the four CSV test files that cover this change passed (74 tests). The forge's own check of that head was cut off or is running again, and it will check the head by itself. If it comes back red on a named test, I'll fix that.
I merged main into the branch and pushed it; the merge had no conflicts, and test/muse-find.test.ts came in from main unchanged.
I ran bun x tsc --noEmit, which was clean. I also ran test/render-csv-prefix.test.ts, test/render-csv.test.ts and test/muse-find.test.ts, which passed (47 tests). I did not run the full suite; the forge's check does that on the push.
The pull itself is unchanged. A CSV, TSV or JSON-lines file over 5 MB opens its first rows and says how much it left out. I did not add a terminal-workspace counterpart.
To try it, open a CSV over 5 MB in the repository file view, or use the preview at https://lab--13494--forge-trevato-apps.open-platform.sh.
<!-- look-pages b53c2a69c9e5 --> Look pages for b53c2a6, from the gate:
look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/csv.ts, src/web/render/kinds.ts and 1 more; the forge pictures / (src/web/render/csv.ts), /docs (src/web/render/csv.ts), /explore (src/web/render/csv.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
I added a "Files over 5 MB" section to docs/features/csv-view.md, which should clear the "both doors" failure, and pushed it. bun scripts/gate.ts --fix no longer prints that rule, but it only printed its last line, so I haven't seen all of its output. I didn't re-run the tests, because this push changes only the doc. I haven't seen the forge's check on the new head.
The section covers the first-bytes read, cutting at a whole line, the "of about M rows" estimate, and that sorting and ?all=1 work over the prefix only.
I left out the terminal side. The workspace still applies its own size limit to tables, and the doc says so. The commit message carries a "No door:" line giving the same reason, in case the gate wants an explicit exemption. If you'd rather the terminal show the prefix too, that's a separate change under src/tui/.
<!-- look-pages 06af960f5acf --> Look pages for 06af960, from the gate:
look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/csv.ts, src/web/render/kinds.ts and 1 more; the forge pictures / (src/web/render/csv.ts), /docs (src/web/render/csv.ts), /explore (src/web/render/csv.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
A table file over 5 MB (CSV, TSV or JSON-lines) now opens its first 1,000 rows and says how many rows it did not show. That is unchanged from the last push. The only new thing here is an empty commit that answers the look-pages note.
- What the feedback asked for: nothing needed fixing. The gate's note only inferred which pages to picture, and it says a
Looks:orNo look:line in a commit message overrides that. - What I did: I added a
No look:commit saying the prefix view lives on a repository file page. No page inapp.looksreaches that view. The 26 inferred pages only import the shared renderer, and its markup for files under the limit is unchanged. - If you disagree: if you want the prefix view pictured, a
looksroute for a sample large table would have to be added toforge.json. I didn't add one. - Tests and checks: I ran none this turn, since no code changed. The check that covers this change is the forge's own run on the push.
To try it, open a CSV, TSV or JSONL file over 5 MB on a repository file page in the preview at https://lab--13494--forge-trevato-apps.open-platform.sh.
The prefix read works and is tested, but three things block it. 1) Both doors: the terminal workspace still refuses large tables. The pull says so itself, and the rule is that what the page does the terminal does too. Open the prefix in src/tui/ through the same Blob.partial path, with the same note. 2) Direction: this is a feature pull while the pipeline items (#6534, #6611, #5276 and the rest) are open. Say in the pull why it goes ahead of them, or park it. 3) RENDER_CAP.csv is lifted to 4000 MB for every source, not just the local one that cuts a prefix. Keep the lift scoped to sources that return partial, so another source cannot hand the renderer a whole huge body. Smaller: the note says the same thing as the summary and then the file size twice, so shorten it to one sentence; the quoted-field retry re-parses up to 2 MB 64 times, so cut back to a safe line in one step; sortColumn decodes the whole body a second time; and the 150 MB RSS and 2 s limits in the test are machine-dependent and will flake, so give them margin or measure the bytes read. Accessibility and layout are fine: the note is plain text in the page with no JavaScript needed.
A CSV, TSV or JSON-lines file over 5 MB opens its first 1,000 rows on the page and says what it cut. The terminal workspace now opens the same prefix. Tests pass for the CSV renderer, the prefix tests, and the terminal workspace; tsc and the design lint are clean. I did not run the full suite; the forge runs that on push.
What changed in this round:
- Terminal: the workspace asks for the same prefix and shows a toast: "Showing the first X of Y, read-only. The rest is in the raw file." It refuses
eand the external editor on that tab. A new test opens a roughly 7 MB CSV through the terminal. I dropped the "terminal keeps its own limit" line fromdocs/features/csv-view.md. - Cap scope:
RENDER_CAP.csvis back to 10 MB.rendererForlifts it only for a body the source markedpartial, and a test covers both cases. - Prefix is opt-in: I found a problem you didn't list. A partial blob handed out by default would have reached
edit.ts,config.tsand the API, and a save could have written back a truncated file. The prefix is now only returned when a reader callsblob(ref, path, { prefix: true }). Only the file page and the terminal do. Every other reader getstoo_largeas before, and a test covers that. - Quoted-field retry: the loop of up to 64 re-parses is replaced by
safeLines, one pass that cuts at the last line end outside a quote. - Decoding:
csv()now decodes the body once. The file page's cache-key lookup inrepo.ts(sortColumn) still decodes it once more; I left that, because decoding a slice could make the key disagree with the render. - Note: one sentence, "Showing 1,000 of about 50,000 rows. The rest is in the raw file."
- Test limits: the 150 MB RSS and 2 s limits are gone. The test now asserts the page is under 2 MB and that the source returns at most
CSV_PREFIX_BYTESending on a line break.
Direction (your point 2): I have no basis to say why this goes ahead of #6534, #6611 and #5276. I haven't touched those, and the pull doesn't depend on them. Park it or merge it as you decide; that's yours to call.
Not done:
- A table between 2 and 5 MB now shows a 2 MB prefix where it used to render whole.
- JSON-lines files open as plain source in the terminal. The terminal has no table view for them.
- Sorting works over the prefix only.
Look: No look: again. No page in app.looks reaches a repository file page of a table over the size limit.
Earlier asks are met: the prefix is opt-in per reader (editors still get too_large), the cap is lifted only for a body the source marked partial, the quoted-field retry loop is one pass (safeLines), the note is one sentence, the flaky RSS/clock limits are gone, and the terminal opens the same prefix read-only with a toast and refuses e and the external editor on it. The how-to is in docs/features/csv-view.md, not README. The note reuses the existing muted csv-more paragraph and works with JavaScript off. Two things I would do next: (1) rename the test whose title ends in 'and ' (test/render-csv-prefix.test.ts, the per-file loop), and (2) have the cache-key lookup in repo.ts stop decoding the whole prefix a second time. Non-blocking: a 2-5 MB table now shows a 2 MB prefix where it used to render whole; that is stated in the pull and acceptable.
Left #13465 open: 1 unticked item remain (memory stays bounded; a test feeds a generated 20 MB file under a time budget).
A large CSV, TSV or JSON-lines file now opens from a bounded prefix of its bytes. It draws its first 1,000 rows and says how many it did not show.
What changed
src/view/local.ts, using the existingblobHeadpipe read. TheBlobtype gainspartial, andkinds.tsgainsCSV_PREFIX_BYTES,isTablePathandwholeLines.capped()now lets table bodies through, andRENDER_CAP.csvis lifted because the read is bounded anyway.test/render-csv-prefix.test.tscommits generated 20 MB.csvand.jsonlfiles and fetches their pages.- The source hands over a prefix of at most 2 MB that ends on a newline. - Each page shows "Showing 1,000 of about N rows" within 2 s, with RSS growth under 150 MB. - Unit cases cover the final partial line, "about" versus "more than", and a cut inside a quoted field.
render-csv,blob-media-ceiling,render-kinds,fuzz-renderersandrender-sqlitepass,tscis clean, andgate.ts --fixanddesign-lintran. I did not run the full suite, and the tests were not run against the unchanged code to confirm they fail without it. The gate rewrapped comments in two files; the commit includes them.Left out
?all=1("Show all rows") does nothing on a prefixed file; the page points to the raw file instead.keep_plan, because that tool was not available in this session.Try it: open a CSV or
.jsonlfile over 2 MB in a repository's code view.🤖 Generated with Claude Code
Closes #13465