Public: anyone can read this; the crew can change it.
A .parquet file opens as its columns, their types and its row count, read from the footer with no dependency #13878
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/13852-a-parquet-file-opens-as-its-columns-thei → main 11 commits fork 7 threads · 7 unresolved
docs/features/README.md +1
| @@ -322,6 +322,7 @@ Read with the guide page [Your apps](../guide-your-apps.md). | ||
| 322 | 322 | - [](terminal-workspace-kinds.md) |
| 323 | 323 | - [](subtitle-shift.md) |
| 324 | 324 | - [](sqlite-view.md) |
| 325 | - [](parquet.md) | |
| 325 | 326 | - [](xlsx.md) |
| 326 | 327 | - [](office.md) |
| 327 | 328 | - [](subtitles.md) |
docs/features/file-kinds.md +1
| @@ -12,6 +12,7 @@ The forge draws a file on its page when its kind is below and it is within the s | ||
| 12 | 12 | - **Video**, drawn up to 4 GB: `.mp4`, `.m4v`, `.webm`, `.mov` |
| 13 | 13 | - **Table (CSV, TSV, JSON lines)**, drawn up to 10 MB: `.csv`, `.tsv`, `.jsonl`, `.ndjson` |
| 14 | 14 | - **SQLite database**, drawn up to 5 MB: `.db`, `.sqlite`, `.sqlite3` |
| 15 | - **Parquet**, drawn up to 5 MB: `.parquet` | |
| 15 | 16 | - **3D model**, drawn up to 100 MB: `.stl`, `.3mf`, `.ply`, `.glb`, `.gltf`, `.obj` |
| 16 | 17 | - **Archive**, drawn up to 50 MB: `.zip`, `.tar`, `.tgz`, `.tar.gz` |
| 17 | 18 | - **Spreadsheet**, drawn up to 10 MB: `.xlsx` |
docs/features/parquet.md added +10
| @@ -0,0 +1,10 @@ | ||
| 1 | # Parquet files (.parquet) | |
| 2 | ||
| 3 | A Parquet file opens as its shape, read from the footer alone, with no library and no data pages read. | |
| 4 | ||
| 5 | - **Page:** a file page shows a summary line (rows, columns, row groups, size), who wrote it, a table of columns (name, physical type, logical type, repetition, compression codecs) and a table of row groups (rows and bytes each). At most 500 columns and 50 row groups are listed, with a line saying so. | |
| 6 | - **Terminal:** the workspace opens the same file with the same reader and cap (`src/tui/screens/workspace_parquet.ts`) as a table of columns with the same five fields; its first row is the summary sentence (`3 rows · 4 columns · 1 row group · 2 KB`), so the row count is there too. Row groups are counted in that sentence, not listed. | |
| 7 | - **Gate:** the name and the first bytes (`PAR1`) must both say Parquet, and the file must be at most 5 MB; past that, or when the file is damaged, encrypted or cut short, the page keeps the source view and the terminal shows a sentence saying why. Never an error page. | |
| 8 | - **Hostile input:** the footer is Thrift compact, decoded by a bounded reader (depth, length and count limits); schema groups nest at most 64 deep; every name is escaped on the page. | |
| 9 | - **Left out:** the rows themselves, nested schemas as a tree (columns are dotted paths), statistics, encrypted footers. | |
| 10 | - **Code:** `src/web/render/parquet.ts` (`readFooter`, `summaryLine`, `parquet`), kind `parquet` in `kinds.ts`; test `test/render-parquet.test.ts`. |
scripts/kinds-guide.ts +1 −1
| @@ -10,7 +10,7 @@ export const END = "<!-- file-kinds:end -->"; | ||
| 10 | 10 | /** The name a person says for each kind. A new kind without one fails the type check. */ |
| 11 | 11 | export const NAMES: Readonly<Record<FileKind, string>> = { |
| 12 | 12 | image: "Image", svg: "SVG", pdf: "PDF", html: "HTML page", audio: "Audio", video: "Video", csv: "Table (CSV, TSV, JSON lines)", |
| 13 | sqlite: "SQLite database", model3d: "3D model", archive: "Archive", sheet: "Spreadsheet", document: "Word document", | |
| 13 | sqlite: "SQLite database", parquet: "Parquet", model3d: "3D model", archive: "Archive", sheet: "Spreadsheet", document: "Word document", | |
| 14 | 14 | slides: "Slides", layers: "Layered image", font: "Font", psd: "Photoshop", config: "Config", markdown: "Markdown", |
| 15 | 15 | notebook: "Notebook", patch: "Patch", log: "Log", subtitles: "Subtitles", gpx: "GPS track", har: "HTTP archive", |
| 16 | 16 | geojson: "GeoJSON", kml: "KML", calendar: "Calendar", otio: "OpenTimelineIO", edl: "Edit list", fcpxml: "Final Cut XML", |
src/tui/screens/workspace.ts +2 −1
| @@ -87,6 +87,7 @@ import { isJsonlPath, kindView, type KindLines } from "./workspace_kinds.ts"; | ||
| 87 | 87 | import { sheetView } from "./workspace_sheet.ts"; |
| 88 | 88 | import { sqliteStep, sqliteView, type SqliteAt } from "./workspace_sqlite.ts"; |
| 89 | 89 | import { officeView } from "./workspace_office.ts"; |
| 90 | import { parquetView } from "./workspace_parquet.ts"; | |
| 90 | 91 | import { layoutTable, type Grid } from "../ui/grid.ts"; |
| 91 | 92 | import type { Ctx, Screen } from "../app/app.ts"; |
| 92 | 93 | import { stripAnsi } from "../term/ansi.ts"; |
| @@ -1449,7 +1450,7 @@ export function workspaceScreen( | ||
| 1449 | 1450 | // bytes as text is how a terminal ends up beeping at you. |
| 1450 | 1451 | const base = looksLikeImage(t.path) ? null : sqliteView(t.path, read.value.body, { table: 0, page: 1 }); |
| 1451 | 1452 | if (base !== null && typeof base !== "string") db = read.value.body; |
| 1452 | const book = looksLikeImage(t.path) ? null : (base ?? sheetView(t.path, read.value.body) ?? officeView(t.path, read.value.body)); | |
| 1453 | const book = looksLikeImage(t.path) ? null : (base ?? sheetView(t.path, read.value.body) ?? officeView(t.path, read.value.body) ?? parquetView(t.path, read.value.body)); | |
| 1453 | 1454 | if (looksLikeImage(t.path)) picture = read.value.body; |
| 1454 | 1455 | else if (book !== null && typeof book !== "string") sheet = book; |
| 1455 | 1456 | else if (typeof book === "string") note = book; |
src/tui/screens/workspace_parquet.ts added +32
| @@ -0,0 +1,32 @@ | ||
| 1 | /** | |
| 2 | * A .parquet file in the terminal workspace, read the way the page reads it: the same `readFooter`, | |
| 3 | * drawn as a table of its columns. The same gate too: the name and the bytes must say so | |
| 4 | * (`fileKind`), under the page's cap, and a file that cannot be read answers its reason. | |
| 5 | * | |
| 6 | * `parquetView(path, body)` answers a table, a sentence (the workspace shows it as its note), or | |
| 7 | * `null` for a file that is not Parquet. | |
| 8 | */ | |
| 9 | ||
| 10 | import { fileKind, HEAD_BYTES, RENDER_CAP } from "../../web/render/kinds.ts"; | |
| 11 | import { COLUMN_CAP, readFooter, summaryLine } from "../../web/render/parquet.ts"; | |
| 12 | import type { Table } from "../../text/csv.ts"; | |
| 13 | ||
| 14 | export function parquetView(path: string, body: Uint8Array): Table | string | null { | |
| 15 | if (fileKind(path, body.subarray(0, HEAD_BYTES)) !== "parquet") return null; | |
| 16 | if (body.length > RENDER_CAP.parquet) | |
| 17 | return `This file is over ${RENDER_CAP.parquet / 1_000_000} MB, too large to open here. Download it instead.`; | |
| 18 | let p; | |
| 19 | try { | |
| 20 | p = readFooter(body); | |
| 21 | } catch { | |
| 22 | return "This Parquet file cannot be shown as columns: its footer could not be read."; | |
| 23 | } | |
| 24 | const header = ["Column", "Type", "Logical type", "Repetition", "Compression"]; | |
| 25 | // The headline (rows, columns, row groups, size) is the first row, as the page's summary is its | |
| 26 | // first line. | |
outdated
Principle 2 and both doors: the terminal drops the row count and row groups the page shows. Add the summary here and a test. |
||
| 27 | const rows = [ | |
| 28 | [summaryLine(p, body.length), "", "", "", ""], | |
| 29 | ...p.columns.slice(0, COLUMN_CAP).map((c) => [c.path, c.physical, c.logical, c.repetition, c.codecs.join(", ")]), | |
| 30 | ]; | |
| 31 | return { header, rows, more: Math.max(0, p.columns.length - COLUMN_CAP), delimiter: ",", columns: header.length }; | |
| 32 | } |
src/web/render/kinds.ts +5
| @@ -22,6 +22,7 @@ export type FileKind = | ||
| 22 | 22 | | "video" |
| 23 | 23 | | "csv" |
| 24 | 24 | | "sqlite" |
| 25 | | "parquet" | |
| 25 | 26 | | "model3d" |
| 26 | 27 | | "archive" |
| 27 | 28 | | "sheet" |
| @@ -95,6 +96,8 @@ export const RENDER_CAP: Readonly<Record<FileKind, number>> = { | ||
| 95 | 96 | csv: 10 * MB, |
| 96 | 97 | // Under the source's own 5 MB read ceiling: a larger file is never read. |
| 97 | 98 | sqlite: 5 * MB, |
| 99 | // Only the footer is read, but the page reads the whole body: under its 5 MB ceiling. | |
| 100 | parquet: 5 * MB, | |
| 98 | 101 | model3d: 100 * MB, |
| 99 | 102 | archive: 50 * MB, |
| 100 | 103 | sheet: 10 * MB, |
| @@ -222,6 +225,7 @@ const MEDIA: Readonly<Record<string, Media>> = { | ||
| 222 | 225 | db: { kind: "sqlite", name: "SQLite", type: "application/vnd.sqlite3", magic: ascii("SQLite format 3\0") }, |
| 223 | 226 | sqlite: { kind: "sqlite", name: "SQLite", type: "application/vnd.sqlite3", magic: ascii("SQLite format 3\0") }, |
outdated
This entry sits between |
||
| 224 | 227 | sqlite3: { kind: "sqlite", name: "SQLite", type: "application/vnd.sqlite3", magic: ascii("SQLite format 3\0") }, |
| 228 | parquet: { kind: "parquet", name: "Parquet", type: "application/vnd.apache.parquet", magic: ascii("PAR1") }, | |
| 225 | 229 | stl: { kind: "model3d", name: "STL", type: "model/stl" }, |
| 226 | 230 | ttf: { kind: "font", name: "TrueType", type: "font/ttf", magic: bytes(0x00, 0x01, 0x00, 0x00) }, |
| 227 | 231 | otf: { kind: "font", name: "OpenType", type: "font/otf", magic: ascii("OTTO") }, |
| @@ -532,6 +536,7 @@ const SNIFFED: readonly (readonly [string, Magic])[] = [ | ||
| 532 | 536 | ["GIF image", ascii("GIF8")], |
| 533 | 537 | ["PDF document", ascii("%PDF-")], |
| 534 | 538 | ["SQLite database", ascii("SQLite format 3\0")], |
| 539 | ["Parquet file", ascii("PAR1")], | |
| 535 | 540 | ]; |
| 536 | 541 | |
| 537 | 542 | /** |
src/web/render/parquet.ts added +273
| @@ -0,0 +1,273 @@ | ||
| 1 | /** | |
| 2 | * A Parquet file as what its footer says: row count, row groups, and each | |
| 3 | * column's name, physical and logical type and compression. Only the | |
| 4 | * footer (Thrift compact `FileMetaData`) is read; no page is decompressed | |
| 5 | * and nothing is fetched. A footer that is cut short or does not parse falls | |
| 6 | * back to the plain file view with one sentence. | |
| 7 | */ | |
| 8 | ||
| 9 | import { escapeHtml, html, raw } from "../html.ts"; | |
| 10 | import { fileSize } from "../ui.ts"; | |
| 11 | import type { Rendered, RenderInput } from "./renderers.ts"; | |
| 12 | ||
| 13 | export const COLUMN_CAP = 500; | |
| 14 | export const GROUP_CAP = 50; | |
COLUMN_CAP, GROUP_CAP and the 'Showing the first N' notes have no test; the test would still pass if the caps were removed. Add one with 501 columns and 51 row groups. |
||
| 15 | const NODE_CAP = 200_000; | |
| 16 | const DEPTH_CAP = 24; | |
| 17 | /** How deep groups may nest in the schema; past it the file falls back with a reason. */ | |
| 18 | export const SCHEMA_DEPTH_CAP = 64; | |
| 19 | ||
| 20 | /** A decoded Thrift value: a struct is its fields by id. */ | |
| 21 | type Value = bigint | number | boolean | string | Value[] | Struct; | |
| 22 | type Struct = Map<number, Value>; | |
| 23 | ||
| 24 | class Bad extends Error {} | |
| 25 | ||
| 26 | /** Thrift compact protocol reader over the footer bytes; bounded in depth and node count. */ | |
| 27 | class Reader { | |
| 28 | pos = 0; | |
| 29 | nodes = 0; | |
| 30 | constructor(private readonly b: Uint8Array) {} | |
| 31 | ||
| 32 | private byte(): number { | |
| 33 | if (this.pos >= this.b.length) throw new Bad("footer ends early"); | |
| 34 | return this.b[this.pos++]!; | |
| 35 | } | |
| 36 | ||
| 37 | varint(): bigint { | |
| 38 | let n = 0n; | |
| 39 | for (let shift = 0n; shift < 70n; shift += 7n) { | |
| 40 | const x = this.byte(); | |
| 41 | n |= BigInt(x & 0x7f) << shift; | |
| 42 | if ((x & 0x80) === 0) return n; | |
| 43 | } | |
| 44 | throw new Bad("bad varint"); | |
| 45 | } | |
| 46 | ||
| 47 | private zigzag(): bigint { | |
| 48 | const n = this.varint(); | |
| 49 | return (n >> 1n) ^ -(n & 1n); | |
| 50 | } | |
| 51 | ||
| 52 | private binary(): string { | |
| 53 | const len = Number(this.varint()); | |
| 54 | if (!Number.isSafeInteger(len) || len < 0 || this.pos + len > this.b.length) throw new Bad("string past end"); | |
| 55 | const s = new TextDecoder().decode(this.b.subarray(this.pos, this.pos + len)); | |
| 56 | this.pos += len; | |
| 57 | return s; | |
| 58 | } | |
| 59 | ||
| 60 | private value(type: number, depth: number): Value { | |
| 61 | if (++this.nodes > NODE_CAP || depth > DEPTH_CAP) throw new Bad("footer too deep"); | |
| 62 | switch (type) { | |
| 63 | case 1: return true; | |
| 64 | case 2: return false; | |
| 65 | case 3: return BigInt(this.byte() << 24 >> 24); | |
| 66 | case 4: case 5: case 6: return this.zigzag(); | |
| 67 | case 7: | |
| 68 | if (this.pos + 8 > this.b.length) throw new Bad("footer ends early"); | |
| 69 | this.pos += 8; | |
| 70 | return 0n; | |
| 71 | case 8: return this.binary(); | |
| 72 | case 9: case 10: { | |
| 73 | const h = this.byte(); | |
| 74 | let size = h >> 4; | |
| 75 | const et = h & 0xf; | |
| 76 | if (size === 15) size = Number(this.varint()); | |
| 77 | if (!Number.isSafeInteger(size) || size > this.b.length - this.pos + 1) throw new Bad("list too long"); | |
| 78 | const out: Value[] = []; | |
| 79 | for (let i = 0; i < size; i++) out.push(et === 1 || et === 2 ? this.byte() === 1 : this.value(et, depth + 1)); | |
| 80 | return out; | |
| 81 | } | |
| 82 | case 11: { | |
| 83 | const size = Number(this.varint()); | |
| 84 | if (!Number.isSafeInteger(size) || size > this.b.length) throw new Bad("map too long"); | |
| 85 | if (size > 0) { | |
| 86 | const kv = this.byte(); | |
| 87 | for (let i = 0; i < size; i++) { | |
| 88 | this.value(kv >> 4, depth + 1); | |
| 89 | this.value(kv & 0xf, depth + 1); | |
| 90 | } | |
| 91 | } | |
| 92 | return []; | |
| 93 | } | |
| 94 | case 12: return this.struct(depth + 1); | |
| 95 | default: throw new Bad("unknown type"); | |
| 96 | } | |
| 97 | } | |
| 98 | ||
| 99 | struct(depth = 0): Struct { | |
| 100 | const out: Struct = new Map(); | |
| 101 | let id = 0; | |
| 102 | for (;;) { | |
| 103 | const h = this.byte(); | |
| 104 | if (h === 0) return out; | |
| 105 | const type = h & 0xf; | |
| 106 | id = h >> 4 === 0 ? Number(this.zigzag()) : id + (h >> 4); | |
| 107 | out.set(id, this.value(type, depth)); | |
| 108 | } | |
| 109 | } | |
| 110 | } | |
| 111 | ||
| 112 | const num = (v: Value | undefined): number | undefined => (typeof v === "bigint" ? Number(v) : undefined); | |
| 113 | const str = (v: Value | undefined): string | undefined => (typeof v === "string" ? v : undefined); | |
| 114 | const list = (v: Value | undefined): Value[] => (Array.isArray(v) ? v : []); | |
| 115 | const struct = (v: Value | undefined): Struct | undefined => (v instanceof Map ? v : undefined); | |
| 116 | ||
| 117 | const PHYSICAL = ["BOOLEAN", "INT32", "INT64", "INT96", "FLOAT", "DOUBLE", "BYTE_ARRAY", "FIXED_LEN_BYTE_ARRAY"]; | |
| 118 | const CODEC = ["UNCOMPRESSED", "SNAPPY", "GZIP", "LZO", "BROTLI", "LZ4", "ZSTD", "LZ4_RAW"]; | |
| 119 | const CONVERTED = [ | |
| 120 | "UTF8", "MAP", "MAP_KEY_VALUE", "LIST", "ENUM", "DECIMAL", "DATE", "TIME_MILLIS", "TIME_MICROS", | |
| 121 | "TIMESTAMP_MILLIS", "TIMESTAMP_MICROS", "UINT_8", "UINT_16", "UINT_32", "UINT_64", "INT_8", "INT_16", | |
| 122 | "INT_32", "INT_64", "JSON", "BSON", "INTERVAL", | |
| 123 | ]; | |
| 124 | const LOGICAL: Record<number, string> = { | |
| 125 | 1: "STRING", 2: "MAP", 3: "LIST", 4: "ENUM", 5: "DECIMAL", 6: "DATE", 7: "TIME", 8: "TIMESTAMP", | |
| 126 | 10: "INTEGER", 11: "UNKNOWN", 12: "JSON", 13: "BSON", 14: "UUID", 15: "FLOAT16", | |
| 127 | }; | |
| 128 | ||
| 129 | export interface Column { | |
| 130 | readonly path: string; | |
| 131 | readonly physical: string; | |
| 132 | readonly logical: string; | |
| 133 | readonly repetition: string; | |
| 134 | /** Codecs the column's chunks use across row groups, in first-seen order. */ | |
| 135 | readonly codecs: readonly string[]; | |
| 136 | } | |
| 137 | export interface Group { readonly rows: number | undefined; readonly bytes: number | undefined; readonly columns: number } | |
| 138 | export interface Parquet { | |
| 139 | readonly rows: number; | |
| 140 | readonly groups: readonly Group[]; | |
| 141 | readonly groupCount: number; | |
| 142 | readonly columns: readonly Column[]; | |
| 143 | readonly createdBy: string | undefined; | |
| 144 | } | |
| 145 | ||
| 146 | function logicalName(e: Struct): string { | |
| 147 | const u = struct(e.get(10)); | |
| 148 | if (u !== undefined) { | |
| 149 | const id = [...u.keys()][0]; | |
| 150 | if (id !== undefined) { | |
| 151 | const inner = struct(u.get(id)); | |
| 152 | let name = LOGICAL[id] ?? ""; | |
| 153 | if (id === 5 && inner) name += `(${num(inner.get(2)) ?? "?"},${num(inner.get(1)) ?? "?"})`; | |
outdated
The PARE (encrypted footer) branch has no test. |
||
| 154 | if (id === 10 && inner) name = `${inner.get(2) === false ? "U" : ""}INT(${num(inner.get(1)) ?? "?"})`; | |
| 155 | if ((id === 7 || id === 8) && inner) { | |
| 156 | const unit = struct(inner.get(2)); | |
| 157 | const u2 = unit === undefined ? undefined : ["MILLIS", "MICROS", "NANOS"][[...unit.keys()][0]! - 1]; | |
| 158 | name += `(${u2 ?? "?"}${inner.get(1) === true ? ",UTC" : ""})`; | |
| 159 | } | |
| 160 | if (name !== "") return name; | |
| 161 | } | |
| 162 | } | |
| 163 | const c = num(e.get(6)); | |
| 164 | return c !== undefined ? (CONVERTED[c] ?? "") : ""; | |
| 165 | } | |
| 166 | ||
| 167 | /** | |
| 168 | * The footer of a Parquet file as plain data; throws a sentence-sized reason when it cannot be | |
| 169 | * read. | |
| 170 | */ | |
| 171 | export function readFooter(body: Uint8Array): Parquet { | |
| 172 | const n = body.length; | |
| 173 | const magic = (at: number): string => String.fromCharCode(...body.subarray(at, at + 4)); | |
| 174 | if (n < 12 || magic(0) !== "PAR1") throw new Bad("it does not start with the Parquet signature"); | |
| 175 | if (magic(n - 4) === "PARE") throw new Bad("its footer is encrypted"); | |
outdated
walk() recurses once per nesting level and the only bound is the node cap (200,000), so a footer of nested single-child groups overflows the stack and is reported as the generic 'could not be read'. Pass a depth and throw Bad('its schema is nested too deeply') past DEPTH_CAP, with a test. |
||
| 176 | if (magic(n - 4) !== "PAR1") throw new Bad("it ends without the Parquet signature, so it is cut short"); | |
| 177 | const len = new DataView(body.buffer, body.byteOffset + n - 8, 4).getUint32(0, true); | |
| 178 | if (len === 0 || len > n - 12) throw new Bad("its footer length is wrong"); | |
| 179 | const meta = new Reader(body.subarray(n - 8 - len, n - 8)).struct(); | |
| 180 | ||
| 181 | const schema = list(meta.get(2)).map(struct); | |
| 182 | if (schema.length < 2 || schema.some((s) => s === undefined)) throw new Bad("its schema is missing"); | |
| 183 | // Walk the flattened schema tree: the leaves are the columns, in order. | |
| 184 | const leaves: Column[] = []; | |
| 185 | let i = 1; | |
| 186 | const walk = (prefix: string, left: number, depth: number): void => { | |
| 187 | if (depth > SCHEMA_DEPTH_CAP) throw new Bad("its schema is nested too deeply"); | |
| 188 | while (left-- > 0 && i < schema.length) { | |
| 189 | const e = schema[i++]!; | |
| 190 | const name = str(e.get(4)) ?? "?"; | |
| 191 | const path = prefix === "" ? name : `${prefix}.${name}`; | |
| 192 | const kids = num(e.get(5)) ?? 0; | |
| 193 | if (kids > 0) { | |
| 194 | walk(path, kids, depth + 1); | |
| 195 | continue; | |
| 196 | } | |
| 197 | const t = num(e.get(1)); | |
| 198 | let physical = t === undefined ? "?" : (PHYSICAL[t] ?? "?"); | |
| 199 | if (t === 7) physical += `(${num(e.get(2)) ?? "?"})`; | |
| 200 | leaves.push({ path, physical, logical: logicalName(e), repetition: ["required", "optional", "repeated"][num(e.get(3)) ?? 0] ?? "", codecs: [] }); | |
| 201 | if (leaves.length > COLUMN_CAP * 4) throw new Bad("it has too many columns"); | |
| 202 | } | |
| 203 | }; | |
| 204 | walk("", num(schema[0]!.get(5)) ?? 0, 0); | |
| 205 | ||
| 206 | const codecs: string[][] = leaves.map(() => []); | |
| 207 | const rgs = list(meta.get(4)); | |
| 208 | const groups: Group[] = []; | |
| 209 | for (const rg of rgs) { | |
| 210 | const g = struct(rg); | |
| 211 | if (g === undefined) throw new Bad("a row group is malformed"); | |
| 212 | const chunks = list(g.get(1)); | |
| 213 | if (groups.length < GROUP_CAP) groups.push({ rows: num(g.get(3)), bytes: num(g.get(2)), columns: chunks.length }); | |
| 214 | chunks.forEach((c, k) => { | |
| 215 | const md = struct(struct(c)?.get(3)); | |
| 216 | const code = num(md?.get(4)); | |
| 217 | const name = code === undefined ? undefined : (CODEC[code] ?? `codec ${code}`); | |
| 218 | if (name !== undefined && codecs[k] !== undefined && !codecs[k]!.includes(name)) codecs[k]!.push(name); | |
| 219 | }); | |
| 220 | } | |
| 221 | const rows = num(meta.get(3)); | |
| 222 | if (rows === undefined) throw new Bad("its row count is missing"); | |
| 223 | return { | |
| 224 | rows, | |
| 225 | groups, | |
| 226 | groupCount: rgs.length, | |
| 227 | columns: leaves.map((c, k) => ({ ...c, codecs: codecs[k]! })), | |
| 228 | createdBy: str(meta.get(6)), | |
| 229 | }; | |
| 230 | } | |
| 231 | ||
| 232 | const plural = (n: number, one: string): string => `${n.toLocaleString()} ${one}${n === 1 ? "" : "s"}`; | |
| 233 | ||
| 234 | /** The headline in one sentence: shared by the page and the terminal. */ | |
| 235 | export function summaryLine(p: Parquet, size: number): string { | |
| 236 | return `${plural(p.rows, "row")} · ${plural(p.columns.length, "column")} · ${plural(p.groupCount, "row group")} · ${fileSize(size)}`; | |
| 237 | } | |
| 238 | ||
| 239 | export function parquet(input: RenderInput): Rendered { | |
| 240 | let p: Parquet; | |
| 241 | try { | |
| 242 | p = readFooter(input.body); | |
| 243 | } catch (e) { | |
| 244 | const why = e instanceof Bad ? e.message : "its footer could not be read"; | |
| 245 | return { content: html``, lines: 0, fallback: `This Parquet file cannot be shown as columns: ${why}.` }; | |
| 246 | } | |
| 247 | const summary = summaryLine(p, input.size); | |
| 248 | const cols = p.columns | |
| 249 | .slice(0, COLUMN_CAP) | |
| 250 | .map( | |
| 251 | (c) => | |
| 252 | `<tr><td>${escapeHtml(c.path)}</td><td>${escapeHtml(c.physical)}</td><td class="muted">${escapeHtml(c.logical)}</td><td class="muted">${escapeHtml(c.repetition)}</td><td>${escapeHtml(c.codecs.join(", "))}</td></tr>`, | |
| 253 | ) | |
| 254 | .join(""); | |
| 255 | const groups = p.groups | |
outdated
Principle 1: the page already shows |
||
| 256 | .map( | |
| 257 | (g, k) => | |
| 258 | `<tr><td class="num">${k + 1}</td><td class="num">${g.rows === undefined ? "" : g.rows.toLocaleString()}</td><td class="num">${g.bytes === undefined ? "" : escapeHtml(fileSize(g.bytes))}</td></tr>`, | |
| 259 | ) | |
| 260 | .join(""); | |
| 261 | const more = p.columns.length > COLUMN_CAP ? `<p class="muted small">Showing the first ${COLUMN_CAP} columns.</p>` : ""; | |
| 262 | const moreGroups = p.groupCount > GROUP_CAP ? `<p class="muted small">Showing the first ${GROUP_CAP} row groups.</p>` : ""; | |
| 263 | // The header already carries rows, columns, row groups and size (`summary`); only the writer | |
| 264 | // is new here. | |
| 265 | const dl = | |
| 266 | p.createdBy === undefined | |
| 267 | ? "" | |
| 268 | : `<dl class="status-rows" aria-label="Written by"><div class="status-row"><dt>Written by</dt><dd>${escapeHtml(p.createdBy.slice(0, 200))}</dd></div></dl>`; | |
| 269 | const content = `${dl} | |
| 270 | <div class="csv-scroll"><table class="table" data-dense aria-label="Columns"><thead><tr><th>Column</th><th>Type</th><th>Logical type</th><th>Repetition</th><th>Compression</th></tr></thead><tbody>${cols}</tbody></table></div>${more} | |
| 271 | <div class="csv-scroll"><table class="table" data-dense aria-label="Row groups"><thead><tr><th class="num">Group</th><th class="num">Rows</th><th class="num">Size</th></tr></thead><tbody>${groups}</tbody></table></div>${moreGroups}`; | |
| 272 | return { content: html`${raw(content)}`, lines: 0, summary }; | |
| 273 | } |
src/web/render/renderers.ts +2
| @@ -54,6 +54,7 @@ import { docx } from "./docx.ts"; | ||
| 54 | 54 | import { pptx } from "./pptx.ts"; |
| 55 | 55 | import { ora } from "./ora.ts"; |
| 56 | 56 | import { sqlite } from "./sqlite.ts"; |
| 57 | import { parquet } from "./parquet.ts"; | |
| 57 | 58 | import { subtitles } from "./subtitles.ts"; |
| 58 | 59 | import { video } from "./video.ts"; |
| 59 | 60 | |
| @@ -123,6 +124,7 @@ export const RENDERERS: Partial<Readonly<Record<FileKind, Renderer>>> = { | ||
| 123 | 124 | html: htmlPreview, |
| 124 | 125 | csv, |
| 125 | 126 | sqlite, |
| 127 | parquet, | |
| 126 | 128 | audio, |
| 127 | 129 | video, |
| 128 | 130 | model3d, |
test/render-kinds.test.ts +1
| @@ -44,6 +44,7 @@ const SAMPLES: Record<string, [Uint8Array, FileKind, string]> = { | ||
| 44 | 44 | db: [enc("SQLite format 3\0"), "sqlite", "application/vnd.sqlite3"], |
| 45 | 45 | sqlite: [enc("SQLite format 3\0"), "sqlite", "application/vnd.sqlite3"], |
| 46 | 46 | sqlite3: [enc("SQLite format 3\0"), "sqlite", "application/vnd.sqlite3"], |
| 47 | parquet: [enc("PAR1"), "parquet", "application/vnd.apache.parquet"], | |
| 47 | 48 | csv: [enc("a,b\n1,2\n"), "csv", "application/octet-stream"], |
| 48 | 49 | tsv: [enc("a\tb\n"), "csv", "application/octet-stream"], |
| 49 | 50 | jsonl: [enc("{\"a\":1}\n"), "csv", "application/octet-stream"], |
test/render-parquet.test.ts added +161
| @@ -0,0 +1,161 @@ | ||
| 1 | /** A Parquet file (built in the test) as its footer: rows, row groups, columns; no pages read. */ | |
| 2 | import { describe, expect, test } from "bun:test"; | |
| 3 | import { fileKind, sniffType } from "../src/web/render/kinds.ts"; | |
| 4 | import { RENDERERS } from "../src/web/render/renderers.ts"; | |
| 5 | import { COLUMN_CAP, GROUP_CAP, parquet, SCHEMA_DEPTH_CAP } from "../src/web/render/parquet.ts"; | |
| 6 | ||
| 7 | // A minimal Thrift compact writer, enough for a FileMetaData. | |
| 8 | const vi = (n: bigint): number[] => { | |
| 9 | const o: number[] = []; | |
| 10 | do { | |
| 11 | let b = Number(n & 0x7fn); | |
| 12 | n >>= 7n; | |
| 13 | if (n > 0n) b |= 0x80; | |
| 14 | o.push(b); | |
| 15 | } while (n > 0n); | |
| 16 | return o; | |
| 17 | }; | |
| 18 | const zz = (n: number): number[] => vi(BigInt(n) >= 0n ? BigInt(n) << 1n : (BigInt(-n) << 1n) - 1n); | |
| 19 | type F = [id: number, type: number, bytes: number[]]; | |
| 20 | const struct = (fields: F[]): number[] => { | |
| 21 | let last = 0; | |
| 22 | const o: number[] = []; | |
| 23 | for (const [id, t, b] of fields) { | |
| 24 | o.push(((id - last) << 4) | t, ...b); | |
| 25 | last = id; | |
| 26 | } | |
| 27 | return [...o, 0]; | |
| 28 | }; | |
| 29 | const i32 = (id: number, n: number): F => [id, 5, zz(n)]; | |
| 30 | const i64 = (id: number, n: number): F => [id, 6, zz(n)]; | |
| 31 | const bin = (s: string): number[] => [...vi(BigInt(s.length)), ...new TextEncoder().encode(s)]; | |
| 32 | const text = (id: number, s: string): F => [id, 8, bin(s)]; | |
| 33 | const lst = (id: number, type: number, items: number[][]): F => [id, 9, [items.length < 15 ? (items.length << 4) | type : 0xf0 | type, ...(items.length < 15 ? [] : vi(BigInt(items.length))), ...items.flat()]]; | |
| 34 | ||
| 35 | function file(opts: { cut?: boolean; evil?: string } = {}): Uint8Array { | |
| 36 | const schema = [ | |
| 37 | struct([text(4, "schema"), i32(5, 3)]), | |
| 38 | struct([i32(1, 2), i32(3, 0), text(4, "id")]), | |
| 39 | struct([i32(1, 6), i32(3, 1), text(4, opts.evil ?? "name"), i32(6, 0), [10, 12, struct([[1, 12, [0]]])]]), | |
| 40 | struct([i32(1, 2), i32(3, 1), text(4, "ts"), [10, 12, struct([[8, 12, struct([[1, 1, []], [2, 12, struct([[2, 12, [0]]])]])]])]]), | |
| 41 | ]; | |
| 42 | const chunk = (codec: number) => struct([i64(2, 4), [3, 12, struct([i32(1, 1), i32(4, codec), i64(5, 10)])]]); | |
| 43 | const group = (rows: number, codec: number) => struct([lst(1, 12, [chunk(codec), chunk(codec), chunk(codec)]), i64(2, 2048), i64(3, rows)]); | |
| 44 | const meta = struct([i32(1, 2), lst(2, 12, schema), i64(3, 30), lst(4, 12, [group(10, 1), group(20, 6)]), text(6, "forge test")]); | |
| 45 | const out = [...new TextEncoder().encode("PAR1"), 1, 2, 3, 4, ...meta, ...new Uint8Array(new Uint32Array([meta.length]).buffer), ...new TextEncoder().encode("PAR1")]; | |
| 46 | return new Uint8Array(opts.cut ? out.slice(0, out.length - 20) : out); | |
| 47 | } | |
| 48 | /** A file from a schema list and row groups; `end` is the trailing signature. */ | |
| 49 | function shaped(schema: number[][], groups: number[][], end = "PAR1"): Uint8Array { | |
| 50 | const meta = struct([i32(1, 2), lst(2, 12, schema), i64(3, 1), lst(4, 12, groups)]); | |
| 51 | const enc = new TextEncoder(); | |
| 52 | return new Uint8Array([...enc.encode("PAR1"), ...meta, ...new Uint8Array(new Uint32Array([meta.length]).buffer), ...enc.encode(end)]); | |
| 53 | } | |
| 54 | const leaf = (name: string) => struct([i32(1, 1), i32(3, 0), text(4, name)]); | |
| 55 | const plainGroup = struct([lst(1, 12, []), i64(2, 10), i64(3, 1)]); | |
| 56 | /** `cols` INT32 columns and `groups` row groups. */ | |
| 57 | const wide = (cols: number, groups: number): Uint8Array => | |
| 58 | shaped([struct([text(4, "schema"), i32(5, cols)]), ...Array.from({ length: cols }, (_, k) => leaf(`c${k}`))], Array.from({ length: groups }, () => plainGroup)); | |
| 59 | /** One leaf under `depth` nested single-child groups. */ | |
| 60 | const deep = (depth: number): Uint8Array => | |
| 61 | shaped([struct([text(4, "schema"), i32(5, 1)]), ...Array.from({ length: depth }, (_, k) => struct([text(4, `g${k}`), i32(5, 1)])), leaf("x")], [plainGroup]); | |
| 62 | const draw = (body: Uint8Array) => parquet({ path: "a.parquet", size: body.length, body, media: "/m", download: "/d" }); | |
| 63 | ||
| 64 | describe("parquet kind", () => { | |
| 65 | test("PAR1 confirms it; a .parquet without it is binary", () => { | |
| 66 | expect(fileKind("a.parquet", file().subarray(0, 4096))).toBe("parquet"); | |
| 67 | expect(fileKind("a.parquet", new Uint8Array([1, 0, 2, 3]))).toBe("binary"); | |
| 68 | expect(sniffType(file())).toBe("Parquet file"); | |
| 69 | expect(RENDERERS.parquet).toBe(parquet); | |
| 70 | }); | |
| 71 | }); | |
| 72 | ||
| 73 | describe("parquet page", () => { | |
| 74 | test("shows rows, row groups and each column's types and compression", () => { | |
| 75 | const r = draw(file()); | |
| 76 | const h = String(r.content); | |
| 77 | expect(r.fallback).toBeUndefined(); | |
| 78 | expect(r.summary).toContain("30 rows · 3 columns · 2 row groups"); | |
| 79 | expect(h).toContain("<td>id</td><td>INT64</td>"); | |
| 80 | expect(h).toContain("<td>name</td><td>BYTE_ARRAY</td><td class=\"muted\">STRING</td>"); | |
| 81 | expect(h).toContain("TIMESTAMP(MICROS,UTC)"); | |
| 82 | expect(h).toContain("SNAPPY, ZSTD"); | |
| 83 | expect(h).toContain("forge test"); | |
| 84 | }); | |
| 85 | test("a column name is escaped", () => { | |
| 86 | const h = String(draw(file({ evil: "<script>x</script>" })).content); | |
| 87 | expect(h).not.toContain("<script>x"); | |
| 88 | expect(h).toContain("<script>"); | |
| 89 | }); | |
| 90 | test("a truncated file falls back with one sentence", () => { | |
| 91 | const r = draw(file({ cut: true })); | |
| 92 | expect(r.fallback).toMatch(/^This Parquet file cannot be shown as columns: .+\.$/); | |
| 93 | }); | |
| 94 | test("a corrupt footer falls back", () => { | |
| 95 | const b = file(); | |
| 96 | for (let i = b.length - 60; i < b.length - 8; i++) b[i] = 0xff; | |
| 97 | expect(draw(b).fallback).toContain("cannot be shown"); | |
| 98 | expect(draw(new TextEncoder().encode("PAR1PAR1PAR1PAR1")).fallback).toContain("cannot be shown"); | |
| 99 | expect(draw(new Uint8Array(0)).fallback).toContain("cannot be shown"); | |
| 100 | }); | |
| 101 | }); | |
| 102 | ||
| 103 | describe("parquet limits", () => { | |
| 104 | test("past the caps it lists the first 500 columns and 50 row groups and says so", () => { | |
| 105 | const r = draw(wide(COLUMN_CAP + 1, GROUP_CAP + 1)); | |
| 106 | const h = String(r.content); | |
| 107 | expect(r.summary).toContain("501 columns · 51 row groups"); | |
| 108 | expect(h).toContain(`<td>c${COLUMN_CAP - 1}</td>`); | |
| 109 | expect(h).not.toContain(`<td>c${COLUMN_CAP}</td>`); | |
| 110 | expect(h).toContain(`Showing the first ${COLUMN_CAP} columns.`); | |
| 111 | expect(h).toContain(`Showing the first ${GROUP_CAP} row groups.`); | |
| 112 | expect(h.match(/<td class="num">\d+<\/td><td class="num">1<\/td>/g)?.length).toBe(GROUP_CAP); | |
| 113 | }); | |
| 114 | test("at the caps there is no note", () => { | |
| 115 | const h = String(draw(wide(COLUMN_CAP, GROUP_CAP)).content); | |
| 116 | expect(h).not.toContain("Showing the first"); | |
| 117 | }); | |
| 118 | test("an encrypted footer (PARE) says so", () => { | |
| 119 | expect(draw(wide(1, 1).map((b, i, a) => (i === a.length - 1 ? 0x45 : b))).fallback).toBe("This Parquet file cannot be shown as columns: its footer is encrypted."); | |
| 120 | }); | |
| 121 | test("a schema nested past the cap falls back with its reason, not a stack overflow", () => { | |
| 122 | expect(draw(deep(SCHEMA_DEPTH_CAP)).fallback).toBeUndefined(); | |
| 123 | expect(String(draw(deep(SCHEMA_DEPTH_CAP)).content)).toContain(".g1."); | |
| 124 | expect(draw(deep(SCHEMA_DEPTH_CAP + 1)).fallback).toBe("This Parquet file cannot be shown as columns: its schema is nested too deeply."); | |
| 125 | expect(draw(deep(10_000)).fallback).toContain("cannot be shown"); | |
| 126 | }); | |
| 127 | test("the summary is not repeated below it: only the writer is", () => { | |
| 128 | const h = String(draw(file()).content); | |
| 129 | expect(h).toContain("<dt>Written by</dt><dd>forge test</dd>"); | |
| 130 | expect(h).not.toContain("<dt>Rows</dt>"); | |
| 131 | expect(h).not.toContain("<dt>Size</dt>"); | |
| 132 | expect(String(draw(wide(1, 1)).content)).not.toContain("<dl"); | |
| 133 | }); | |
| 134 | }); | |
| 135 | ||
| 136 | describe("parquet in the terminal workspace", () => { | |
| 137 | test("over 5 MB is a sentence; past the column cap the rest is counted", async () => { | |
| 138 | const { parquetView } = await import("../src/tui/screens/workspace_parquet.ts"); | |
| 139 | const big = new Uint8Array(5_000_001); | |
| 140 | big.set(new TextEncoder().encode("PAR1")); | |
| 141 | expect(parquetView("a.parquet", big)).toBe("This file is over 5 MB, too large to open here. Download it instead."); | |
| 142 | const t = parquetView("a.parquet", wide(COLUMN_CAP + 3, 1)); | |
| 143 | if (t === null || typeof t === "string") throw new Error("expected a table"); | |
| 144 | expect(t.rows.length).toBe(COLUMN_CAP + 1); | |
| 145 | expect(t.rows[COLUMN_CAP]![0]).toBe(`c${COLUMN_CAP - 1}`); | |
| 146 | expect(t.more).toBe(3); | |
| 147 | }); | |
| 148 | ||
| 149 | test("a Parquet file is a table of its columns; a cut one answers its reason; text is not Parquet", async () => { | |
| 150 | const { parquetView } = await import("../src/tui/screens/workspace_parquet.ts"); | |
| 151 | const t = parquetView("a.parquet", file()); | |
| 152 | expect(typeof t).toBe("object"); | |
| 153 | if (t === null || typeof t === "string") throw new Error("expected a table"); | |
| 154 | expect(t.header[0]).toBe("Column"); | |
| 155 | expect(t.rows.map((r) => r[0])).toContain("name"); | |
| 156 | expect(t.header).toContain("Repetition"); | |
| 157 | expect(t.rows[0]![0]).toMatch(/^[\d,]+ rows? · \d+ columns? · \d+ row groups? · /); | |
| 158 | expect(typeof parquetView("a.parquet", file({ cut: true }))).toBe("string"); | |
| 159 | expect(parquetView("a.txt", new TextEncoder().encode("hello"))).toBeNull(); | |
| 160 | }); | |
| 161 | }); |
The over-5 MB message and the column cap in the terminal are untested; add both to the terminal test.