Public: anyone can read this; the crew can change it.

A .parquet file opens as its columns, their types and its row count, read from the footer with no dependency #13878

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13852-a-parquet-file-opens-as-its-columns-thei → main 11 commits fork 7 threads · 7 unresolved

  • Merged. scout/13852-a-parquet-file-opens-as-its-columns-thei was merged into main as ff33af6.

Review

Sign in to review this change.
11 files changed +489 −2
docs/features/README.md +1
@@ -322,6 +322,7 @@ Read with the guide page [Your apps](../guide-your-apps.md).
322322 - [Notebooks, JSON Lines and logs in the terminal](terminal-workspace-kinds.md)
323323 - [Shift a subtitle file](subtitle-shift.md)
324324 - [SQLite files as tables](sqlite-view.md)
325 - [Parquet files as their columns and row groups](parquet.md)
325326 - [Excel workbooks (.xlsx) as a table](xlsx.md)
326327 - [Office documents and layered images (.docx, .pptx, .ora)](office.md)
327328 - [Subtitles as a transcript](subtitles.md)
docs/features/file-kinds.md +1
@@ -12,6 +12,7 @@ The forge draws a file on its page when its kind is below and it is within the s
1212 - **Video**, drawn up to 4 GB: `.mp4`, `.m4v`, `.webm`, `.mov`
1313 - **Table (CSV, TSV, JSON lines)**, drawn up to 10 MB: `.csv`, `.tsv`, `.jsonl`, `.ndjson`
1414 - **SQLite database**, drawn up to 5 MB: `.db`, `.sqlite`, `.sqlite3`
15 - **Parquet**, drawn up to 5 MB: `.parquet`
1516 - **3D model**, drawn up to 100 MB: `.stl`, `.3mf`, `.ply`, `.glb`, `.gltf`, `.obj`
1617 - **Archive**, drawn up to 50 MB: `.zip`, `.tar`, `.tgz`, `.tar.gz`
1718 - **Spreadsheet**, drawn up to 10 MB: `.xlsx`
docs/features/parquet.md added +10
@@ -0,0 +1,10 @@
1 # Parquet files (.parquet)
2
3 A Parquet file opens as its shape, read from the footer alone, with no library and no data pages read.
4
5 - **Page:** a file page shows a summary line (rows, columns, row groups, size), who wrote it, a table of columns (name, physical type, logical type, repetition, compression codecs) and a table of row groups (rows and bytes each). At most 500 columns and 50 row groups are listed, with a line saying so.
6 - **Terminal:** the workspace opens the same file with the same reader and cap (`src/tui/screens/workspace_parquet.ts`) as a table of columns with the same five fields; its first row is the summary sentence (`3 rows · 4 columns · 1 row group · 2 KB`), so the row count is there too. Row groups are counted in that sentence, not listed.
7 - **Gate:** the name and the first bytes (`PAR1`) must both say Parquet, and the file must be at most 5 MB; past that, or when the file is damaged, encrypted or cut short, the page keeps the source view and the terminal shows a sentence saying why. Never an error page.
8 - **Hostile input:** the footer is Thrift compact, decoded by a bounded reader (depth, length and count limits); schema groups nest at most 64 deep; every name is escaped on the page.
9 - **Left out:** the rows themselves, nested schemas as a tree (columns are dotted paths), statistics, encrypted footers.
10 - **Code:** `src/web/render/parquet.ts` (`readFooter`, `summaryLine`, `parquet`), kind `parquet` in `kinds.ts`; test `test/render-parquet.test.ts`.
scripts/kinds-guide.ts +1 −1
@@ -10,7 +10,7 @@ export const END = "<!-- file-kinds:end -->";
1010 /** The name a person says for each kind. A new kind without one fails the type check. */
1111 export const NAMES: Readonly<Record<FileKind, string>> = {
1212 image: "Image", svg: "SVG", pdf: "PDF", html: "HTML page", audio: "Audio", video: "Video", csv: "Table (CSV, TSV, JSON lines)",
13 sqlite: "SQLite database", model3d: "3D model", archive: "Archive", sheet: "Spreadsheet", document: "Word document",
13 sqlite: "SQLite database", parquet: "Parquet", model3d: "3D model", archive: "Archive", sheet: "Spreadsheet", document: "Word document",
1414 slides: "Slides", layers: "Layered image", font: "Font", psd: "Photoshop", config: "Config", markdown: "Markdown",
1515 notebook: "Notebook", patch: "Patch", log: "Log", subtitles: "Subtitles", gpx: "GPS track", har: "HTTP archive",
1616 geojson: "GeoJSON", kml: "KML", calendar: "Calendar", otio: "OpenTimelineIO", edl: "Edit list", fcpxml: "Final Cut XML",
src/tui/screens/workspace.ts +2 −1
@@ -87,6 +87,7 @@ import { isJsonlPath, kindView, type KindLines } from "./workspace_kinds.ts";
8787 import { sheetView } from "./workspace_sheet.ts";
8888 import { sqliteStep, sqliteView, type SqliteAt } from "./workspace_sqlite.ts";
8989 import { officeView } from "./workspace_office.ts";
90 import { parquetView } from "./workspace_parquet.ts";
9091 import { layoutTable, type Grid } from "../ui/grid.ts";
9192 import type { Ctx, Screen } from "../app/app.ts";
9293 import { stripAnsi } from "../term/ansi.ts";
@@ -1449,7 +1450,7 @@ export function workspaceScreen(
14491450 // bytes as text is how a terminal ends up beeping at you.
14501451 const base = looksLikeImage(t.path) ? null : sqliteView(t.path, read.value.body, { table: 0, page: 1 });
14511452 if (base !== null && typeof base !== "string") db = read.value.body;
1452 const book = looksLikeImage(t.path) ? null : (base ?? sheetView(t.path, read.value.body) ?? officeView(t.path, read.value.body));
1453 const book = looksLikeImage(t.path) ? null : (base ?? sheetView(t.path, read.value.body) ?? officeView(t.path, read.value.body) ?? parquetView(t.path, read.value.body));
14531454 if (looksLikeImage(t.path)) picture = read.value.body;
14541455 else if (book !== null && typeof book !== "string") sheet = book;
14551456 else if (typeof book === "string") note = book;
src/tui/screens/workspace_parquet.ts added +32
@@ -0,0 +1,32 @@
1 /**
2 * A .parquet file in the terminal workspace, read the way the page reads it: the same `readFooter`,
3 * drawn as a table of its columns. The same gate too: the name and the bytes must say so
4 * (`fileKind`), under the page's cap, and a file that cannot be read answers its reason.
5 *
6 * `parquetView(path, body)` answers a table, a sentence (the workspace shows it as its note), or
7 * `null` for a file that is not Parquet.
8 */
9
10 import { fileKind, HEAD_BYTES, RENDER_CAP } from "../../web/render/kinds.ts";
11 import { COLUMN_CAP, readFooter, summaryLine } from "../../web/render/parquet.ts";
12 import type { Table } from "../../text/csv.ts";
13
14 export function parquetView(path: string, body: Uint8Array): Table | string | null {
15 if (fileKind(path, body.subarray(0, HEAD_BYTES)) !== "parquet") return null;
16 if (body.length > RENDER_CAP.parquet)

The over-5 MB message and the column cap in the terminal are untested; add both to the terminal test.

17 return `This file is over ${RENDER_CAP.parquet / 1_000_000} MB, too large to open here. Download it instead.`;
18 let p;
19 try {
20 p = readFooter(body);
21 } catch {
22 return "This Parquet file cannot be shown as columns: its footer could not be read.";
23 }
24 const header = ["Column", "Type", "Logical type", "Repetition", "Compression"];
25 // The headline (rows, columns, row groups, size) is the first row, as the page's summary is its
26 // first line.
outdated

Principle 2 and both doors: the terminal drops the row count and row groups the page shows. Add the summary here and a test.

27 const rows = [
28 [summaryLine(p, body.length), "", "", "", ""],
29 ...p.columns.slice(0, COLUMN_CAP).map((c) => [c.path, c.physical, c.logical, c.repetition, c.codecs.join(", ")]),
30 ];
31 return { header, rows, more: Math.max(0, p.columns.length - COLUMN_CAP), delimiter: ",", columns: header.length };
32 }
src/web/render/kinds.ts +5
@@ -22,6 +22,7 @@ export type FileKind =
2222 | "video"
2323 | "csv"
2424 | "sqlite"
25 | "parquet"
2526 | "model3d"
2627 | "archive"
2728 | "sheet"
@@ -95,6 +96,8 @@ export const RENDER_CAP: Readonly<Record<FileKind, number>> = {
9596 csv: 10 * MB,
9697 // Under the source's own 5 MB read ceiling: a larger file is never read.
9798 sqlite: 5 * MB,
99 // Only the footer is read, but the page reads the whole body: under its 5 MB ceiling.
100 parquet: 5 * MB,
98101 model3d: 100 * MB,
99102 archive: 50 * MB,
100103 sheet: 10 * MB,
@@ -222,6 +225,7 @@ const MEDIA: Readonly<Record<string, Media>> = {
222225 db: { kind: "sqlite", name: "SQLite", type: "application/vnd.sqlite3", magic: ascii("SQLite format 3\0") },
223226 sqlite: { kind: "sqlite", name: "SQLite", type: "application/vnd.sqlite3", magic: ascii("SQLite format 3\0") },
outdated

This entry sits between sqlite and sqlite3, splitting the SQLite group. Move it after the sqlite3 line.

224227 sqlite3: { kind: "sqlite", name: "SQLite", type: "application/vnd.sqlite3", magic: ascii("SQLite format 3\0") },
228 parquet: { kind: "parquet", name: "Parquet", type: "application/vnd.apache.parquet", magic: ascii("PAR1") },
225229 stl: { kind: "model3d", name: "STL", type: "model/stl" },
226230 ttf: { kind: "font", name: "TrueType", type: "font/ttf", magic: bytes(0x00, 0x01, 0x00, 0x00) },
227231 otf: { kind: "font", name: "OpenType", type: "font/otf", magic: ascii("OTTO") },
@@ -532,6 +536,7 @@ const SNIFFED: readonly (readonly [string, Magic])[] = [
532536 ["GIF image", ascii("GIF8")],
533537 ["PDF document", ascii("%PDF-")],
534538 ["SQLite database", ascii("SQLite format 3\0")],
539 ["Parquet file", ascii("PAR1")],
535540 ];
536541
537542 /**
src/web/render/parquet.ts added +273
@@ -0,0 +1,273 @@
1 /**
2 * A Parquet file as what its footer says: row count, row groups, and each
3 * column's name, physical and logical type and compression. Only the
4 * footer (Thrift compact `FileMetaData`) is read; no page is decompressed
5 * and nothing is fetched. A footer that is cut short or does not parse falls
6 * back to the plain file view with one sentence.
7 */
8
9 import { escapeHtml, html, raw } from "../html.ts";
10 import { fileSize } from "../ui.ts";
11 import type { Rendered, RenderInput } from "./renderers.ts";
12
13 export const COLUMN_CAP = 500;
14 export const GROUP_CAP = 50;

COLUMN_CAP, GROUP_CAP and the 'Showing the first N' notes have no test; the test would still pass if the caps were removed. Add one with 501 columns and 51 row groups.

15 const NODE_CAP = 200_000;
16 const DEPTH_CAP = 24;
17 /** How deep groups may nest in the schema; past it the file falls back with a reason. */
18 export const SCHEMA_DEPTH_CAP = 64;
19
20 /** A decoded Thrift value: a struct is its fields by id. */
21 type Value = bigint | number | boolean | string | Value[] | Struct;
22 type Struct = Map<number, Value>;
23
24 class Bad extends Error {}
25
26 /** Thrift compact protocol reader over the footer bytes; bounded in depth and node count. */
27 class Reader {
28 pos = 0;
29 nodes = 0;
30 constructor(private readonly b: Uint8Array) {}
31
32 private byte(): number {
33 if (this.pos >= this.b.length) throw new Bad("footer ends early");
34 return this.b[this.pos++]!;
35 }
36
37 varint(): bigint {
38 let n = 0n;
39 for (let shift = 0n; shift < 70n; shift += 7n) {
40 const x = this.byte();
41 n |= BigInt(x & 0x7f) << shift;
42 if ((x & 0x80) === 0) return n;
43 }
44 throw new Bad("bad varint");
45 }
46
47 private zigzag(): bigint {
48 const n = this.varint();
49 return (n >> 1n) ^ -(n & 1n);
50 }
51
52 private binary(): string {
53 const len = Number(this.varint());
54 if (!Number.isSafeInteger(len) || len < 0 || this.pos + len > this.b.length) throw new Bad("string past end");
55 const s = new TextDecoder().decode(this.b.subarray(this.pos, this.pos + len));
56 this.pos += len;
57 return s;
58 }
59
60 private value(type: number, depth: number): Value {
61 if (++this.nodes > NODE_CAP || depth > DEPTH_CAP) throw new Bad("footer too deep");
62 switch (type) {
63 case 1: return true;
64 case 2: return false;
65 case 3: return BigInt(this.byte() << 24 >> 24);
66 case 4: case 5: case 6: return this.zigzag();
67 case 7:
68 if (this.pos + 8 > this.b.length) throw new Bad("footer ends early");
69 this.pos += 8;
70 return 0n;
71 case 8: return this.binary();
72 case 9: case 10: {
73 const h = this.byte();
74 let size = h >> 4;
75 const et = h & 0xf;
76 if (size === 15) size = Number(this.varint());
77 if (!Number.isSafeInteger(size) || size > this.b.length - this.pos + 1) throw new Bad("list too long");
78 const out: Value[] = [];
79 for (let i = 0; i < size; i++) out.push(et === 1 || et === 2 ? this.byte() === 1 : this.value(et, depth + 1));
80 return out;
81 }
82 case 11: {
83 const size = Number(this.varint());
84 if (!Number.isSafeInteger(size) || size > this.b.length) throw new Bad("map too long");
85 if (size > 0) {
86 const kv = this.byte();
87 for (let i = 0; i < size; i++) {
88 this.value(kv >> 4, depth + 1);
89 this.value(kv & 0xf, depth + 1);
90 }
91 }
92 return [];
93 }
94 case 12: return this.struct(depth + 1);
95 default: throw new Bad("unknown type");
96 }
97 }
98
99 struct(depth = 0): Struct {
100 const out: Struct = new Map();
101 let id = 0;
102 for (;;) {
103 const h = this.byte();
104 if (h === 0) return out;
105 const type = h & 0xf;
106 id = h >> 4 === 0 ? Number(this.zigzag()) : id + (h >> 4);
107 out.set(id, this.value(type, depth));
108 }
109 }
110 }
111
112 const num = (v: Value | undefined): number | undefined => (typeof v === "bigint" ? Number(v) : undefined);
113 const str = (v: Value | undefined): string | undefined => (typeof v === "string" ? v : undefined);
114 const list = (v: Value | undefined): Value[] => (Array.isArray(v) ? v : []);
115 const struct = (v: Value | undefined): Struct | undefined => (v instanceof Map ? v : undefined);
116
117 const PHYSICAL = ["BOOLEAN", "INT32", "INT64", "INT96", "FLOAT", "DOUBLE", "BYTE_ARRAY", "FIXED_LEN_BYTE_ARRAY"];
118 const CODEC = ["UNCOMPRESSED", "SNAPPY", "GZIP", "LZO", "BROTLI", "LZ4", "ZSTD", "LZ4_RAW"];
119 const CONVERTED = [
120 "UTF8", "MAP", "MAP_KEY_VALUE", "LIST", "ENUM", "DECIMAL", "DATE", "TIME_MILLIS", "TIME_MICROS",
121 "TIMESTAMP_MILLIS", "TIMESTAMP_MICROS", "UINT_8", "UINT_16", "UINT_32", "UINT_64", "INT_8", "INT_16",
122 "INT_32", "INT_64", "JSON", "BSON", "INTERVAL",
123 ];
124 const LOGICAL: Record<number, string> = {
125 1: "STRING", 2: "MAP", 3: "LIST", 4: "ENUM", 5: "DECIMAL", 6: "DATE", 7: "TIME", 8: "TIMESTAMP",
126 10: "INTEGER", 11: "UNKNOWN", 12: "JSON", 13: "BSON", 14: "UUID", 15: "FLOAT16",
127 };
128
129 export interface Column {
130 readonly path: string;
131 readonly physical: string;
132 readonly logical: string;
133 readonly repetition: string;
134 /** Codecs the column's chunks use across row groups, in first-seen order. */
135 readonly codecs: readonly string[];
136 }
137 export interface Group { readonly rows: number | undefined; readonly bytes: number | undefined; readonly columns: number }
138 export interface Parquet {
139 readonly rows: number;
140 readonly groups: readonly Group[];
141 readonly groupCount: number;
142 readonly columns: readonly Column[];
143 readonly createdBy: string | undefined;
144 }
145
146 function logicalName(e: Struct): string {
147 const u = struct(e.get(10));
148 if (u !== undefined) {
149 const id = [...u.keys()][0];
150 if (id !== undefined) {
151 const inner = struct(u.get(id));
152 let name = LOGICAL[id] ?? "";
153 if (id === 5 && inner) name += `(${num(inner.get(2)) ?? "?"},${num(inner.get(1)) ?? "?"})`;
outdated

The PARE (encrypted footer) branch has no test.

154 if (id === 10 && inner) name = `${inner.get(2) === false ? "U" : ""}INT(${num(inner.get(1)) ?? "?"})`;
155 if ((id === 7 || id === 8) && inner) {
156 const unit = struct(inner.get(2));
157 const u2 = unit === undefined ? undefined : ["MILLIS", "MICROS", "NANOS"][[...unit.keys()][0]! - 1];
158 name += `(${u2 ?? "?"}${inner.get(1) === true ? ",UTC" : ""})`;
159 }
160 if (name !== "") return name;
161 }
162 }
163 const c = num(e.get(6));
164 return c !== undefined ? (CONVERTED[c] ?? "") : "";
165 }
166
167 /**
168 * The footer of a Parquet file as plain data; throws a sentence-sized reason when it cannot be
169 * read.
170 */
171 export function readFooter(body: Uint8Array): Parquet {
172 const n = body.length;
173 const magic = (at: number): string => String.fromCharCode(...body.subarray(at, at + 4));
174 if (n < 12 || magic(0) !== "PAR1") throw new Bad("it does not start with the Parquet signature");
175 if (magic(n - 4) === "PARE") throw new Bad("its footer is encrypted");
outdated

walk() recurses once per nesting level and the only bound is the node cap (200,000), so a footer of nested single-child groups overflows the stack and is reported as the generic 'could not be read'. Pass a depth and throw Bad('its schema is nested too deeply') past DEPTH_CAP, with a test.

176 if (magic(n - 4) !== "PAR1") throw new Bad("it ends without the Parquet signature, so it is cut short");
177 const len = new DataView(body.buffer, body.byteOffset + n - 8, 4).getUint32(0, true);
178 if (len === 0 || len > n - 12) throw new Bad("its footer length is wrong");
179 const meta = new Reader(body.subarray(n - 8 - len, n - 8)).struct();
180
181 const schema = list(meta.get(2)).map(struct);
182 if (schema.length < 2 || schema.some((s) => s === undefined)) throw new Bad("its schema is missing");
183 // Walk the flattened schema tree: the leaves are the columns, in order.
184 const leaves: Column[] = [];
185 let i = 1;
186 const walk = (prefix: string, left: number, depth: number): void => {
187 if (depth > SCHEMA_DEPTH_CAP) throw new Bad("its schema is nested too deeply");
188 while (left-- > 0 && i < schema.length) {
189 const e = schema[i++]!;
190 const name = str(e.get(4)) ?? "?";
191 const path = prefix === "" ? name : `${prefix}.${name}`;
192 const kids = num(e.get(5)) ?? 0;
193 if (kids > 0) {
194 walk(path, kids, depth + 1);
195 continue;
196 }
197 const t = num(e.get(1));
198 let physical = t === undefined ? "?" : (PHYSICAL[t] ?? "?");
199 if (t === 7) physical += `(${num(e.get(2)) ?? "?"})`;
200 leaves.push({ path, physical, logical: logicalName(e), repetition: ["required", "optional", "repeated"][num(e.get(3)) ?? 0] ?? "", codecs: [] });
201 if (leaves.length > COLUMN_CAP * 4) throw new Bad("it has too many columns");
202 }
203 };
204 walk("", num(schema[0]!.get(5)) ?? 0, 0);
205
206 const codecs: string[][] = leaves.map(() => []);
207 const rgs = list(meta.get(4));
208 const groups: Group[] = [];
209 for (const rg of rgs) {
210 const g = struct(rg);
211 if (g === undefined) throw new Bad("a row group is malformed");
212 const chunks = list(g.get(1));
213 if (groups.length < GROUP_CAP) groups.push({ rows: num(g.get(3)), bytes: num(g.get(2)), columns: chunks.length });
214 chunks.forEach((c, k) => {
215 const md = struct(struct(c)?.get(3));
216 const code = num(md?.get(4));
217 const name = code === undefined ? undefined : (CODEC[code] ?? `codec ${code}`);
218 if (name !== undefined && codecs[k] !== undefined && !codecs[k]!.includes(name)) codecs[k]!.push(name);
219 });
220 }
221 const rows = num(meta.get(3));
222 if (rows === undefined) throw new Bad("its row count is missing");
223 return {
224 rows,
225 groups,
226 groupCount: rgs.length,
227 columns: leaves.map((c, k) => ({ ...c, codecs: codecs[k]! })),
228 createdBy: str(meta.get(6)),
229 };
230 }
231
232 const plural = (n: number, one: string): string => `${n.toLocaleString()} ${one}${n === 1 ? "" : "s"}`;
233
234 /** The headline in one sentence: shared by the page and the terminal. */
235 export function summaryLine(p: Parquet, size: number): string {
236 return `${plural(p.rows, "row")} · ${plural(p.columns.length, "column")} · ${plural(p.groupCount, "row group")} · ${fileSize(size)}`;
237 }
238
239 export function parquet(input: RenderInput): Rendered {
240 let p: Parquet;
241 try {
242 p = readFooter(input.body);
243 } catch (e) {
244 const why = e instanceof Bad ? e.message : "its footer could not be read";
245 return { content: html``, lines: 0, fallback: `This Parquet file cannot be shown as columns: ${why}.` };
246 }
247 const summary = summaryLine(p, input.size);
248 const cols = p.columns
249 .slice(0, COLUMN_CAP)
250 .map(
251 (c) =>
252 `<tr><td>${escapeHtml(c.path)}</td><td>${escapeHtml(c.physical)}</td><td class="muted">${escapeHtml(c.logical)}</td><td class="muted">${escapeHtml(c.repetition)}</td><td>${escapeHtml(c.codecs.join(", "))}</td></tr>`,
253 )
254 .join("");
255 const groups = p.groups
outdated

Principle 1: the page already shows summary (rows, columns, row groups, size) in its header, and this dl states the same four facts again. Keep only 'Written by' here, or drop the dl.

256 .map(
257 (g, k) =>
258 `<tr><td class="num">${k + 1}</td><td class="num">${g.rows === undefined ? "" : g.rows.toLocaleString()}</td><td class="num">${g.bytes === undefined ? "" : escapeHtml(fileSize(g.bytes))}</td></tr>`,
259 )
260 .join("");
261 const more = p.columns.length > COLUMN_CAP ? `<p class="muted small">Showing the first ${COLUMN_CAP} columns.</p>` : "";
262 const moreGroups = p.groupCount > GROUP_CAP ? `<p class="muted small">Showing the first ${GROUP_CAP} row groups.</p>` : "";
263 // The header already carries rows, columns, row groups and size (`summary`); only the writer
264 // is new here.
265 const dl =
266 p.createdBy === undefined
267 ? ""
268 : `<dl class="status-rows" aria-label="Written by"><div class="status-row"><dt>Written by</dt><dd>${escapeHtml(p.createdBy.slice(0, 200))}</dd></div></dl>`;
269 const content = `${dl}
270 <div class="csv-scroll"><table class="table" data-dense aria-label="Columns"><thead><tr><th>Column</th><th>Type</th><th>Logical type</th><th>Repetition</th><th>Compression</th></tr></thead><tbody>${cols}</tbody></table></div>${more}
271 <div class="csv-scroll"><table class="table" data-dense aria-label="Row groups"><thead><tr><th class="num">Group</th><th class="num">Rows</th><th class="num">Size</th></tr></thead><tbody>${groups}</tbody></table></div>${moreGroups}`;
272 return { content: html`${raw(content)}`, lines: 0, summary };
273 }
src/web/render/renderers.ts +2
@@ -54,6 +54,7 @@ import { docx } from "./docx.ts";
5454 import { pptx } from "./pptx.ts";
5555 import { ora } from "./ora.ts";
5656 import { sqlite } from "./sqlite.ts";
57 import { parquet } from "./parquet.ts";
5758 import { subtitles } from "./subtitles.ts";
5859 import { video } from "./video.ts";
5960
@@ -123,6 +124,7 @@ export const RENDERERS: Partial<Readonly<Record<FileKind, Renderer>>> = {
123124 html: htmlPreview,
124125 csv,
125126 sqlite,
127 parquet,
126128 audio,
127129 video,
128130 model3d,
test/render-kinds.test.ts +1
@@ -44,6 +44,7 @@ const SAMPLES: Record<string, [Uint8Array, FileKind, string]> = {
4444 db: [enc("SQLite format 3\0"), "sqlite", "application/vnd.sqlite3"],
4545 sqlite: [enc("SQLite format 3\0"), "sqlite", "application/vnd.sqlite3"],
4646 sqlite3: [enc("SQLite format 3\0"), "sqlite", "application/vnd.sqlite3"],
47 parquet: [enc("PAR1"), "parquet", "application/vnd.apache.parquet"],
4748 csv: [enc("a,b\n1,2\n"), "csv", "application/octet-stream"],
4849 tsv: [enc("a\tb\n"), "csv", "application/octet-stream"],
4950 jsonl: [enc("{\"a\":1}\n"), "csv", "application/octet-stream"],
test/render-parquet.test.ts added +161
@@ -0,0 +1,161 @@
1 /** A Parquet file (built in the test) as its footer: rows, row groups, columns; no pages read. */
2 import { describe, expect, test } from "bun:test";
3 import { fileKind, sniffType } from "../src/web/render/kinds.ts";
4 import { RENDERERS } from "../src/web/render/renderers.ts";
5 import { COLUMN_CAP, GROUP_CAP, parquet, SCHEMA_DEPTH_CAP } from "../src/web/render/parquet.ts";
6
7 // A minimal Thrift compact writer, enough for a FileMetaData.
8 const vi = (n: bigint): number[] => {
9 const o: number[] = [];
10 do {
11 let b = Number(n & 0x7fn);
12 n >>= 7n;
13 if (n > 0n) b |= 0x80;
14 o.push(b);
15 } while (n > 0n);
16 return o;
17 };
18 const zz = (n: number): number[] => vi(BigInt(n) >= 0n ? BigInt(n) << 1n : (BigInt(-n) << 1n) - 1n);
19 type F = [id: number, type: number, bytes: number[]];
20 const struct = (fields: F[]): number[] => {
21 let last = 0;
22 const o: number[] = [];
23 for (const [id, t, b] of fields) {
24 o.push(((id - last) << 4) | t, ...b);
25 last = id;
26 }
27 return [...o, 0];
28 };
29 const i32 = (id: number, n: number): F => [id, 5, zz(n)];
30 const i64 = (id: number, n: number): F => [id, 6, zz(n)];
31 const bin = (s: string): number[] => [...vi(BigInt(s.length)), ...new TextEncoder().encode(s)];
32 const text = (id: number, s: string): F => [id, 8, bin(s)];
33 const lst = (id: number, type: number, items: number[][]): F => [id, 9, [items.length < 15 ? (items.length << 4) | type : 0xf0 | type, ...(items.length < 15 ? [] : vi(BigInt(items.length))), ...items.flat()]];
34
35 function file(opts: { cut?: boolean; evil?: string } = {}): Uint8Array {
36 const schema = [
37 struct([text(4, "schema"), i32(5, 3)]),
38 struct([i32(1, 2), i32(3, 0), text(4, "id")]),
39 struct([i32(1, 6), i32(3, 1), text(4, opts.evil ?? "name"), i32(6, 0), [10, 12, struct([[1, 12, [0]]])]]),
40 struct([i32(1, 2), i32(3, 1), text(4, "ts"), [10, 12, struct([[8, 12, struct([[1, 1, []], [2, 12, struct([[2, 12, [0]]])]])]])]]),
41 ];
42 const chunk = (codec: number) => struct([i64(2, 4), [3, 12, struct([i32(1, 1), i32(4, codec), i64(5, 10)])]]);
43 const group = (rows: number, codec: number) => struct([lst(1, 12, [chunk(codec), chunk(codec), chunk(codec)]), i64(2, 2048), i64(3, rows)]);
44 const meta = struct([i32(1, 2), lst(2, 12, schema), i64(3, 30), lst(4, 12, [group(10, 1), group(20, 6)]), text(6, "forge test")]);
45 const out = [...new TextEncoder().encode("PAR1"), 1, 2, 3, 4, ...meta, ...new Uint8Array(new Uint32Array([meta.length]).buffer), ...new TextEncoder().encode("PAR1")];
46 return new Uint8Array(opts.cut ? out.slice(0, out.length - 20) : out);
47 }
48 /** A file from a schema list and row groups; `end` is the trailing signature. */
49 function shaped(schema: number[][], groups: number[][], end = "PAR1"): Uint8Array {
50 const meta = struct([i32(1, 2), lst(2, 12, schema), i64(3, 1), lst(4, 12, groups)]);
51 const enc = new TextEncoder();
52 return new Uint8Array([...enc.encode("PAR1"), ...meta, ...new Uint8Array(new Uint32Array([meta.length]).buffer), ...enc.encode(end)]);
53 }
54 const leaf = (name: string) => struct([i32(1, 1), i32(3, 0), text(4, name)]);
55 const plainGroup = struct([lst(1, 12, []), i64(2, 10), i64(3, 1)]);
56 /** `cols` INT32 columns and `groups` row groups. */
57 const wide = (cols: number, groups: number): Uint8Array =>
58 shaped([struct([text(4, "schema"), i32(5, cols)]), ...Array.from({ length: cols }, (_, k) => leaf(`c${k}`))], Array.from({ length: groups }, () => plainGroup));
59 /** One leaf under `depth` nested single-child groups. */
60 const deep = (depth: number): Uint8Array =>
61 shaped([struct([text(4, "schema"), i32(5, 1)]), ...Array.from({ length: depth }, (_, k) => struct([text(4, `g${k}`), i32(5, 1)])), leaf("x")], [plainGroup]);
62 const draw = (body: Uint8Array) => parquet({ path: "a.parquet", size: body.length, body, media: "/m", download: "/d" });
63
64 describe("parquet kind", () => {
65 test("PAR1 confirms it; a .parquet without it is binary", () => {
66 expect(fileKind("a.parquet", file().subarray(0, 4096))).toBe("parquet");
67 expect(fileKind("a.parquet", new Uint8Array([1, 0, 2, 3]))).toBe("binary");
68 expect(sniffType(file())).toBe("Parquet file");
69 expect(RENDERERS.parquet).toBe(parquet);
70 });
71 });
72
73 describe("parquet page", () => {
74 test("shows rows, row groups and each column's types and compression", () => {
75 const r = draw(file());
76 const h = String(r.content);
77 expect(r.fallback).toBeUndefined();
78 expect(r.summary).toContain("30 rows · 3 columns · 2 row groups");
79 expect(h).toContain("<td>id</td><td>INT64</td>");
80 expect(h).toContain("<td>name</td><td>BYTE_ARRAY</td><td class=\"muted\">STRING</td>");
81 expect(h).toContain("TIMESTAMP(MICROS,UTC)");
82 expect(h).toContain("SNAPPY, ZSTD");
83 expect(h).toContain("forge test");
84 });
85 test("a column name is escaped", () => {
86 const h = String(draw(file({ evil: "<script>x</script>" })).content);
87 expect(h).not.toContain("<script>x");
88 expect(h).toContain("&lt;script&gt;");
89 });
90 test("a truncated file falls back with one sentence", () => {
91 const r = draw(file({ cut: true }));
92 expect(r.fallback).toMatch(/^This Parquet file cannot be shown as columns: .+\.$/);
93 });
94 test("a corrupt footer falls back", () => {
95 const b = file();
96 for (let i = b.length - 60; i < b.length - 8; i++) b[i] = 0xff;
97 expect(draw(b).fallback).toContain("cannot be shown");
98 expect(draw(new TextEncoder().encode("PAR1PAR1PAR1PAR1")).fallback).toContain("cannot be shown");
99 expect(draw(new Uint8Array(0)).fallback).toContain("cannot be shown");
100 });
101 });
102
103 describe("parquet limits", () => {
104 test("past the caps it lists the first 500 columns and 50 row groups and says so", () => {
105 const r = draw(wide(COLUMN_CAP + 1, GROUP_CAP + 1));
106 const h = String(r.content);
107 expect(r.summary).toContain("501 columns · 51 row groups");
108 expect(h).toContain(`<td>c${COLUMN_CAP - 1}</td>`);
109 expect(h).not.toContain(`<td>c${COLUMN_CAP}</td>`);
110 expect(h).toContain(`Showing the first ${COLUMN_CAP} columns.`);
111 expect(h).toContain(`Showing the first ${GROUP_CAP} row groups.`);
112 expect(h.match(/<td class="num">\d+<\/td><td class="num">1<\/td>/g)?.length).toBe(GROUP_CAP);
113 });
114 test("at the caps there is no note", () => {
115 const h = String(draw(wide(COLUMN_CAP, GROUP_CAP)).content);
116 expect(h).not.toContain("Showing the first");
117 });
118 test("an encrypted footer (PARE) says so", () => {
119 expect(draw(wide(1, 1).map((b, i, a) => (i === a.length - 1 ? 0x45 : b))).fallback).toBe("This Parquet file cannot be shown as columns: its footer is encrypted.");
120 });
121 test("a schema nested past the cap falls back with its reason, not a stack overflow", () => {
122 expect(draw(deep(SCHEMA_DEPTH_CAP)).fallback).toBeUndefined();
123 expect(String(draw(deep(SCHEMA_DEPTH_CAP)).content)).toContain(".g1.");
124 expect(draw(deep(SCHEMA_DEPTH_CAP + 1)).fallback).toBe("This Parquet file cannot be shown as columns: its schema is nested too deeply.");
125 expect(draw(deep(10_000)).fallback).toContain("cannot be shown");
126 });
127 test("the summary is not repeated below it: only the writer is", () => {
128 const h = String(draw(file()).content);
129 expect(h).toContain("<dt>Written by</dt><dd>forge test</dd>");
130 expect(h).not.toContain("<dt>Rows</dt>");
131 expect(h).not.toContain("<dt>Size</dt>");
132 expect(String(draw(wide(1, 1)).content)).not.toContain("<dl");
133 });
134 });
135
136 describe("parquet in the terminal workspace", () => {
137 test("over 5 MB is a sentence; past the column cap the rest is counted", async () => {
138 const { parquetView } = await import("../src/tui/screens/workspace_parquet.ts");
139 const big = new Uint8Array(5_000_001);
140 big.set(new TextEncoder().encode("PAR1"));
141 expect(parquetView("a.parquet", big)).toBe("This file is over 5 MB, too large to open here. Download it instead.");
142 const t = parquetView("a.parquet", wide(COLUMN_CAP + 3, 1));
143 if (t === null || typeof t === "string") throw new Error("expected a table");
144 expect(t.rows.length).toBe(COLUMN_CAP + 1);
145 expect(t.rows[COLUMN_CAP]![0]).toBe(`c${COLUMN_CAP - 1}`);
146 expect(t.more).toBe(3);
147 });
148
149 test("a Parquet file is a table of its columns; a cut one answers its reason; text is not Parquet", async () => {
150 const { parquetView } = await import("../src/tui/screens/workspace_parquet.ts");
151 const t = parquetView("a.parquet", file());
152 expect(typeof t).toBe("object");
153 if (t === null || typeof t === "string") throw new Error("expected a table");
154 expect(t.header[0]).toBe("Column");
155 expect(t.rows.map((r) => r[0])).toContain("name");
156 expect(t.header).toContain("Repetition");
157 expect(t.rows[0]![0]).toMatch(/^[\d,]+ rows? · \d+ columns? · \d+ row groups? · /);
158 expect(typeof parquetView("a.parquet", file({ cut: true }))).toBe("string");
159 expect(parquetView("a.txt", new TextEncoder().encode("hello"))).toBeNull();
160 });
161 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs