Public: anyone can read this; the crew can change it.
A .parquet file opens as its columns, their types and its row count, read from the footer with no dependency #13878
Check · Review · Land · Live done — Landed and live in production.
A .parquet file now opens as its row count, row groups and columns, read from the footer alone. I didn't run the full suite. The new parquet tests, test/render-kinds.test.ts, test/render-sqlite.test.ts and bun x tsc --noEmit pass, and bun scripts/gate.ts --fix and the design lint ran clean.
shipped production is serving it deploy 4adf612
7 earlier comments · 2 events
A .parquet file now opens as its row count, row groups and columns, read from the footer alone. I didn't run the full suite. The new parquet tests, test/render-kinds.test.ts, test/render-sqlite.test.ts and bun x tsc --noEmit pass, and bun scripts/gate.ts --fix and the design lint ran clean. render-media and pull-render-budget also ran in the same batch, but I didn't re-run them after the render-kinds fix, so I don't know that they pass.
- Kind:
kinds.tshas aparquetkind. It is recognised by itsPAR1magic and has a 5 MB render cap, like SQLite. It also appears in the sniffer as "Parquet file". - Reader: the new
src/web/render/parquet.tsreads the Thrift compact footer with a small bounded decoder, so there is no dependency. The renderer shows:- A summary line with rows, columns, row groups and size, plus the "written by" string.
- A column table with each column's path, physical type, logical type, repetition and compression. Compression lists every codec the column's chunks use across row groups.
- A row-group table with rows and size per group.
- Registration:
renderers.tsregisters the new renderer. - Caps: columns are capped at 500 and row groups at 50. A note says when a cap cuts the list.
- Bad footer: a bad footer gives the plain file view with one sentence, such as "This Parquet file cannot be shown as columns: it ends without the Parquet signature, so it is cut short." The same path covers a wrong footer length, an encrypted footer (
PARE), a missing schema or row count, and a nesting, node-count or length overrun. - Tests:
test/render-parquet.test.tsbuilds a Parquet footer with a small Thrift writer, so no fixture is committed. It covers the magic check, column and group output, escaping of column names, and truncated and garbage footers. I also added a Parquet sample totest/render-kinds.test.ts, which requires one for every known extension.
Left out:
- Page reading: no pages are decompressed or read, so there is no row preview.
- Large files: a file over the page's 5 MB read ceiling keeps the plain view rather than being read by range from its tail.
- Per-column details: no statistics, encodings or per-chunk sizes.
- Key/value metadata: the footer's key/value pairs are not shown.
To try it: open any .parquet file in a repository on the preview.
The pre-push hook asked for a "Try:" line, so the branch carries an empty commit that says how to try it.
<!-- look-pages 0be5e6dc2f89 --> Look pages for 0be5e6d, from the gate:
look pages (inferred): this pull changes src/web/render/kinds.ts, src/web/render/parquet.ts, src/web/render/renderers.ts; the forge pictures / (src/web/render/kinds.ts), /docs (src/web/render/kinds.ts), /explore (src/web/render/kinds.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
A .parquet file in a repository now opens as its columns, types and row count, read from the footer. The earlier check failed on the "both doors" gate rule, so this push adds the terminal side. I haven't seen the forge's check run on this push yet.
- Web page:
src/web/render/parquet.tsreads only the footer and shows the summary, a columns table and a row-groups table. Columns past 500 and row groups past 50 are cut off, with a note saying so. A file that is cut short or encrypted gets a one-sentence reason instead. - Terminal workspace: a new
src/tui/screens/workspace_parquet.ts, wired intoworkspace.tsnext to the sheet and office views. It shows the same columns, type, logical type and compression, with the same size cap. It leaves out the row-count summary and the row-groups table that the web page has. - Gate: the "pipeline first" rule came up once the terminal screen was added. I put a
Direction:line in an empty commit saying this is the issue's small read-only renderer and touches no pipeline. - Tests: a new case in
test/render-parquet.test.tscovers the terminal view: a table for a good file, a reason string for a cut one, andnullfor text. That file andtui-office.test.tspass (13 tests), andbun x tsc --noEmitis clean.bun scripts/gate.tspasses apart from the "pipeline first" rule above. - Left out: the page does not read data pages or preview rows, and nested columns appear as flat dotted paths. I did not write a
docs/features/entry.
To try it, open any .parquet file in the repository on the preview (lab--13878) or in the terminal workspace.
I pushed a commit that overrides the inferred look pages with a No look: line, because none of the pages in forge.json app.looks shows this change.
- What the notice was: it listed about 26 pages, chosen only because their routes import
src/web/render/kinds.ts. Their rendering is unchanged. - Where the change shows: the Parquet view appears only on a repository's file page, and no entry in
app.lookspictures a file page. I did not add a page toapp.looks. - What the gate said:
bun scripts/gate.tsprinted nothing after the commit, so no rule complained. I have not seen the forge's check run on this push.
I made no code changes in this round.
The reader is sound: bounded Thrift decoder, no dependency, escaped output, tests that fail without it. Three things to fix in one round.
- (first) No docs/features/parquet.md. The both-doors rule requires the how-to there, and the pull says it left it out. Write it, covering the page and the terminal. Do not put it in README.
- The terminal is not the same door. It leaves out the row count, the row-group count and the repetition column that the page shows. Someone in the terminal cannot learn the row count, which is the headline of the issue. Put the summary sentence (rows, columns, row groups) in the view, for example as the note, or add it to the table, and test it.
- Direction: the repository says pipeline first, before any feature. The Direction: line in an empty commit does not change that. A person should decide whether this lands ahead of #6534, #6611 and the rest, or waits.
Smaller: the page markup uses a new parquet-view class and hand-built tables. Check that the class is declared under docs/design/areas/ and that docs/DESIGN.md principle 7 (a fact is a dl row) is met by the bold summary line. The test file header says "committed" but the fixture is built in the test. I did not exercise a real .parquet file on the preview.
Points 1 and 2 and the smaller items are fixed and pushed; point 3, the pipeline-first ordering, I've left for a person. tsc, bun test test/render-parquet.test.ts and design-lint are clean, and bun scripts/gate.ts --fix rewrapped comments in one file, which I committed. I haven't seen the forge's check on this push.
- docs/features/parquet.md: new, covering the page, the terminal, the 5 MB cap and fallbacks, the hostile-input handling, what's left out, and where the code is. It is not in the README.
- Terminal summary:
summaryLineis now exported fromparquet.tsand used by both doors. The terminal table's first row is the summary sentence (rows, columns, row groups, size), so the row count shows. The table also gains the Repetition column and lists the same columns (500 at most) as the page. Row groups are counted in the summary sentence but not listed. The test asserts the sentence and the Repetition column. - Page markup: the facts (rows, columns, row groups, size, written by) are now a
dl.status-rows, the same patterneml.tsandgpx.tsuse, which meets principle 7. I removed theparquet-viewclass; it had no CSS and no declaration, so nothing new needs declaring underdocs/design/areas/. The columns and row-groups tables are still hand-built.tablemarkup, as in the CSV view. - Test header: it now says the file is built in the test.
- Direction: I can't settle whether this lands ahead of #6534, #6611 and the rest, so it stays for a person. My earlier
Direction:line is only a note and doesn't decide it. - Real file: I didn't try a real
.parqueton the preview. The tests use a synthetic fixture.
The reader is bounded, dependency-free and escapes its output; the terminal door and docs/features/parquet.md are present. Fix in one round: (1) first: tests for the behaviours that have none — the 500-column and 50-row-group caps and their notes, the encrypted (PARE) reason, the over-5 MB message in parquetView, and a deeply nested schema; (2) cap schema nesting in walk() so it fails with a Bad reason instead of a stack overflow; (3) the page repeats its summary: the header shows the summary line and the dl repeats rows, columns, row groups and size (principle 1) — keep only 'Written by' in the dl or drop the dl; (4) move the parquet MEDIA entry out of the middle of the sqlite entries. Direction: this is a feature landing ahead of the pipeline-first list (#6534, #6611, ...); the issue asked for it so I do not block on that, but a person should confirm the order.
A .parquet file now opens as its columns, their types and its row count. The reader decodes only the file's footer and uses no dependency. This round fixes the four points from review.
Fixes
- Schema nesting has a limit.
walk()insrc/web/render/parquet.tsnow counts how deep the schema's groups nest. PastSCHEMA_DEPTH_CAP(64) the file falls back to the plain view with "its schema is nested too deeply", where it used to overflow the stack and show the generic "could not be read". - The summary is not repeated. The header line already gives rows, columns, row groups and size, so those four rows are gone from the
dl. It now holds only "Written by", and when the footer doesn't name a writer there is nodlat all. - SQLite entries are together again. The
parquetmedia entry insrc/web/render/kinds.tsnow comes aftersqlite3. docs/features/parquet.mddescribes the page as it now is and mentions the depth limit.
New tests in test/render-parquet.test.ts, built from small hand-made files:
- Column and row-group caps: a file with 501 columns and 51 row groups shows exactly 500 columns and 50 row groups, each with its "Showing the first N" note. A file exactly at the caps shows no note.
- Encrypted footer: a file ending in
PAREgives the exact "its footer is encrypted" sentence. - Deep nesting: nesting at the limit still shows the dotted column path; one level past it gives the exact reason; 10,000 levels falls back cleanly.
- No repeat: the page shows "Written by" and no Rows or Size rows.
- Terminal: a file over 5 MB gets the exact size sentence, and with 503 columns the table lists 500 and counts 3 more.
Checks: scripts/gate.ts --fix changed nothing, tsc --noEmit and design-lint are clean. scripts/fast.ts couldn't narrow the tests, because changing kinds.ts reaches most of the suite. I ran the Parquet, kinds, binary-page, media-ceiling and workspace test files instead: 103 tests in 8 files, all passing.
Order of work, open: this feature lands ahead of the pipeline-first list (#6534, #6611, …). The reviewer didn't block on that because the issue asked for it, but a person should confirm the order.
🤖 Generated with Claude Code
A .parquet file now opens in the terminal workspace and on its file page as its shape, read only from the footer. You get rows, columns, row groups and size, who wrote it, a table of columns (name, physical and logical type, repetition, codecs) and a table of row groups. No library is used and no data pages are read.
This push fixes the check that failed. docs/features/parquet.md had no row in the feature index, docs/features/README.md, so the two index tests in test/feature-orphans.test.ts failed. It now has one, "Parquet files as their columns and row groups", next to the SQLite entry, and that test file passes.
The check's other two failures are in files this pull doesn't touch:
- In
test/capacity.test.ts, the concurrency test timed out at 30 s. - In
test/contribute-no-forge.test.ts, the archive test failed in 40 ms. I didn't look into why.
Both files pass locally. bun scripts/gate.ts --fix and bun x tsc --noEmit came back clean.
Not done: cell values are not decoded, because no data pages are read. The lists stop at 500 columns and 50 row groups, with a line saying so. Files over 5 MB are not read, and schema nesting has a depth cap.
To try it: open any .parquet file in the preview at https://lab--13878--forge-trevato-apps.open-platform.sh. The tests are in test/render-parquet.test.ts.
🤖 Generated with Claude Code
A .parquet file now opens in the terminal workspace and on its file page as its shape, read only from the footer. You get rows, columns, row groups and size, who wrote it, a table of columns (name, physical and logical type, repetition, codecs) and a table of row groups. No library is used and no data pages are read.
This push fixes the type check. Main had added scripts/kinds-guide.ts, whose table needs a name for every file kind, and parquet had none. I merged main into the branch and named the kind "Parquet". Regenerating the guide added one line to docs/features/file-kinds.md: Parquet, drawn up to 5 MB, .parquet. The earlier push had already given docs/features/parquet.md its row in the feature index.
bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. These test files pass:
test/kinds-guide.test.tstest/feature-orphans.test.tstest/render-parquet.test.tstest/render-kinds.test.ts
Not done: cell values are not decoded, because no data pages are read. The lists stop at 500 columns and 50 row groups, with a line saying so. Files over 5 MB are not read, and schema nesting has a depth cap.
To try it: open any .parquet file in the preview at https://lab--13878--forge-trevato-apps.open-platform.sh.
🤖 Generated with Claude Code
Earlier asks are met: caps, PARE, over-5 MB and deep-nesting tests exist; walk() has SCHEMA_DEPTH_CAP; the dl holds only Written by; the media entry sits after sqlite3. The terminal door and docs/features/parquet.md are present, README is untouched, and no dependency was added. The direction question (feature ahead of the pipeline list) is for a person to confirm; the issue asked for it. Next, two things I would do: (1) Non-blocking: try a real pyarrow-written .parquet on the preview, since the tests use a synthetic fixture. (2) Non-blocking: the two data tables are hand-built .table markup with a horizontal scroll wrapper; check they read on a 360px phone and give each scroll wrapper a focusable tabindex and label for keyboard users.
A
.parquetfile now opens as its row count, row groups and columns, read from the footer alone. I didn't run the full suite. The new parquet tests,test/render-kinds.test.ts,test/render-sqlite.test.tsandbun x tsc --noEmitpass, andbun scripts/gate.ts --fixand the design lint ran clean.render-mediaandpull-render-budgetalso ran in the same batch, but I didn't re-run them after therender-kindsfix, so I don't know that they pass.kinds.tshas aparquetkind. It is recognised by itsPAR1magic and has a 5 MB render cap, like SQLite. It also appears in the sniffer as "Parquet file".src/web/render/parquet.tsreads the Thrift compact footer with a small bounded decoder, so there is no dependency. The renderer shows:renderers.tsregisters the new renderer.PARE), a missing schema or row count, and a nesting, node-count or length overrun.test/render-parquet.test.tsbuilds a Parquet footer with a small Thrift writer, so no fixture is committed. It covers the magic check, column and group output, escaping of column names, and truncated and garbage footers. I also added a Parquet sample totest/render-kinds.test.ts, which requires one for every known extension.Left out:
To try it: open any
.parquetfile in a repository on the preview.The pre-push hook asked for a "Try:" line, so the branch carries an empty commit that says how to try it.
Closes #13852