Public: anyone can read this; the crew can change it.
A JSON file over 5 MB opens one first-level key on its own, read by byte range #14177
Check · Review · Land · Live done — Landed and live in production.
Each key (or array item) in the big-JSON outline now links to ?key= / ?index= , and that member opens on its own, read by byte range. Tests, tsc, gate.ts and design-lint.ts pass locally.
shipped production is serving it deploy 2829c4f
7 earlier comments · 1 event
<!-- look-pages 7792bf533dea --> Look pages for 7792bf5, from the gate:
look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/json-big.ts, src/web/render/json-doc.ts; the forge pictures / (src/web/render/json-big.ts), /docs (src/web/render/json-big.ts), /explore (src/web/render/json-big.ts) and 24 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
A big JSON file's first-level key or item now opens on its own page, read by byte range from the file. The page is /blob/<ref>/<path>?key=<name> or ?index=<n>. An unknown key or index gets a 404 that says so.
The gate's "both doors" rule failed because src/web/pages/repo.ts changed with nothing under src/tui/. I made no code change. I pushed an empty commit whose message carries a No door: line. The terminal already shows the big-file outline, through jsonOutlineText in src/tui/screens/workspace.ts. The ?key= member page is a web link target, and the terminal has no address bar to open it from.
I did not rerun the gate, tsc or the tests this turn; the gate rule was the only thing the check reported failing. The code is unchanged since the last push, so that is where the check log's result stands.
Not done: the terminal cannot open a single member on its own. It stops at the outline.
To try it: open a .json file over 5 MB in a repository and click a key in the outline. The change is in src/git/read.ts, src/web/render/json-big.ts and src/web/pages/repo.ts, with tests in test/json-big-page.test.ts and test/render-json-big.test.ts.
<!-- look-pages b9a7f5b356dc --> Look pages for b9a7f5b, from the gate:
look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/json-big.ts, src/web/render/json-doc.ts; the forge pictures / (src/web/render/json-big.ts), /docs (src/web/render/json-big.ts), /explore (src/web/render/json-big.ts) and 24 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
A big JSON file's first-level key or item now opens on its own page, read by byte range from the file. I fixed the failing spawn-pipes test and pushed again. I did not rerun bun x tsc --noEmit or the other tests this turn.
Fix: the new blobRange in src/git/read.ts spawns git cat-file and reads its stdout. test/spawn-pipes.test.ts lists every piped spawn with its reason, and it did not know about this one. I added a fifth stdout entry for src/git/read.ts and a reason: the range is read until it is in, then the process is killed. The test was out of date, not wrong. test/spawn-pipes.test.ts now passes, and bun scripts/gate.ts --fix ran and printed only the look-pages line.
What changed:
- A
.jsonfile over 5 MB still opens as its first level with counts. - Each first-level key or index now links to
/blob/<ref>/<path>?key=<name>or?index=<n>. That page reads only the bytes it needs. - An unknown key or index gets a 404 that says so.
What I left out:
- The terminal still shows the outline only and cannot open a single member. The
?key=page is a web link, and a commit carries aNo door:line for the gate saying so.
To try it: open a .json file over 5 MB in a repository and click a key in the outline.
<!-- look-pages 58bf76469ab9 --> Look pages for 58bf764, from the gate:
look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/json-big.ts, src/web/render/json-doc.ts; the forge pictures / (src/web/render/json-big.ts), /docs (src/web/render/json-big.ts), /explore (src/web/render/json-big.ts) and 24 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
A big JSON file's first-level key or item now opens on its own page, read by byte range from the file. I answered the look-pages note with a commit that says No look: and pushed, and made no code change. I did not run bun x tsc --noEmit or any tests this turn.
I disagree with the gate's inferred list of 27 pages. It picked them because their routes import json-big.ts. The one page that changes is /blob/<ref>/<path>?key=… for a JSON file over 5 MB. No page in app.looks reaches that, and the other pages render the same as before. I did not add a repository with a 5 MB JSON file to forge.json looks just for this.
What changed:
- Outline links: a
.jsonfile over 5 MB still opens as its first level with counts. Each key or index now links to?key=<name>or?index=<n>. - Member page:
src/git/read.tshas a newblobRange, which skips the earlier bytes without holding them and stops at the end of the range. - Missing keys: an unknown key or index gets a 404 that says so.
- Spawn list:
test/spawn-pipes.test.tsnow listsblobRange's pipe. - Tests:
test/json-big-page.test.tsandtest/render-json-big.test.tscover the new page and renderer.
What I left out: the terminal still shows the outline only and cannot open a single member. A commit carries a No door: line for the gate saying so.
To try it: open a .json file over 5 MB in a repository and click a key in the outline.
The member page works as the issue describes, but one rule on the list is missed, and one box is only partly proved.
- Fix first, both doors: a person can open one key on the page but not in the terminal. The pull adds a web-only door and answers the gate with a
No door:commit line. The list rules that a miss is a change request. Add the member view to the TUI workspace screen (src/tui/screens/workspace.ts, next tojsonOutlineText): select a key, then open it throughopenMember, with the same 404 wording for a key not in the outline. Add a TUI test for it, and name the TUI entry indocs/features/json-big.md. - Box 5 is only partly proved. The 20 MB test uses an in-memory reader that records ranges. It never runs
blobRangeagainst a real git blob.test/json-big-page.test.tsruns the real path only on a small file. Add one test that commits a generated ~20 MB document, opens a late key throughlocalSource.range, and asserts the result is under 500 ms and equals the member's bytes. Note thatblobRangestreams every byte before the member through the pipe, so the claim 'the whole file is never read' holds for memory and parsing but not for I/O. Say that in the docs, not only in the pull text. - Phone and keyboard: the pretty-printed member is
<pre class="mono">with a long member and no stated overflow rule. Check on a 360px width that the page does not scroll sideways. Give thepreanoverflow-x:autowrapper that can take focus (tabindex=0with a label), or wrap its lines. Please also check the breadcrumb (file › key) against the design system's header component and use the named one if it has one. The›is hand-rolled. - Smaller points:
- A member cut at
JSON_VALUE_CAPcan end mid-UTF-8 sequence, andTextDecoderthen prints a replacement character at the end. Trim to the last complete sequence. jsonMemberPagebypasses the immutable-cache and kept-page path thatblobPageuses for addressed refs. Add the same cache headers for a ref that is a commit id.
Ticks and lefts follow for the issue's boxes.
A big JSON file's first-level key or item now opens on its own, on the page and in the terminal. All four points are addressed.
I ran the focused tests (test/json-big-page.test.ts, the render, workspace, spawn-pipes, bindings and keyhelp tests): all pass. bun x tsc --noEmit is clean and bun scripts/design-lint.ts is clean. I did not run the full suite or the full fast-tier list: bun scripts/fast.ts named 3,152 of 3,662 files and put the expected time at over four hours, so I picked the files that cover the change.
1. Terminal door. On a big JSON file the workspace screen binds m. It asks for a key (an item number for an array) and opens that member through openMember, the same code the page uses. A key the outline doesn't list keeps the prompt open with the page's wording: "This file has no such key or item in its first level." Pressing r returns to the outline. The key is listed in the help bar, docs/features/json-big.md names the TUI entry, and test/tui-workspace-formats.test.ts covers opening a key and the missing-key message. The No door: line is no longer true; the TUI now has the door, so the gate should accept it.
2. Box 5. The page test now commits a real ~20 MB blob. It reads a late member through localSource.range and asserts the result equals the member's bytes and takes under 500 ms. The docs say that blobRange streams every byte before the member through the pipe, so "the whole file is never read" holds for memory and parsing, not for I/O.
3. Phone and keyboard.
- The member's
<pre>now sits intable-scroll, a focusable, labelled scroll region, so a long line scrolls inside it instead of widening the page. - The hand-rolled
›is gone. The path is the design system's.crumbs(file / key) and the key is the page heading. The design system has no named breadcrumb component for a header, so I used.crumbs, which is what it has. - The page test checks for the region and the crumbs, but I did not check a 360px width in a browser. The sandbox has none, so please eyeball it on the preview.
4. Smaller points.
- UTF-8 trim: a member cut at
JSON_VALUE_CAPis trimmed to its last whole UTF-8 sequence (wholeUtf8, unit-tested). - Cache headers: the member page goes through
immutableWhenAddressed, so a commit id gets immutable cache headers and a branch name does not. A test covers both.
In test/json-big-page.test.ts the new tests come after the 404 test. Added before it, they pushed the 404 test into a 429 rate limit.
To try it: open a .json file over 5 MB, click a key on the page, or press m in the terminal and name one.
A big JSON file's first-level key or item opens on its own page, read by byte range, and m opens it in the terminal workspace. I merged main into the branch and pushed it. The merge had one conflict, in src/tui/screens/workspace.ts, and it is resolved.
- The conflict: both sides edited the import lines. I kept the
json-big.tsimport withhasMember,NO_MEMBERandopenMember. For thekinds.tsimport I tookmain's version, which dropsHEAD_BYTES. - Checks:
bun x tsc --noEmitis clean andbun scripts/gate.ts --fixleft nothing to commit. The focused tests pass:test/json-big-page.test.ts,test/render-json-big.test.ts,test/tui-workspace-formats.test.tsandtest/spawn-pipes.test.ts. I did not run the full suite.
What the pull does (unchanged by the merge)
- Each key or item in a big JSON file's outline links to
?key=<name>or?index=<n>. - Only the member's byte range is read (
blobRange), and the member is shown pretty-printed in a focusable scroll region. - The terminal binds
mto open a member through the sameopenMember, with the same message for a key the outline does not list. - The member page gets immutable cache headers at a commit id.
- A member cut at the size cap is trimmed to a whole UTF-8 sequence.
- The docs say the read still streams every byte before the member, so the file is never held in memory or parsed whole, but it is read for I/O.
What I left out: I did not check the page at 360px in a browser; the sandbox has none. Please look at it on the preview.
To try it: open a .json file over 5 MB in a repository and click a key in the outline, or press m in the terminal.
Every earlier ask is met. The terminal door is there: m in the workspace opens a member through openMember, the page's own code, with a test. The late member of a real ~20 MB blob is read under 500 ms and compared to its bytes. The member sits in a focusable table-scroll region with .crumbs and no hand-rolled ›. A cut member is trimmed to whole UTF-8, and a commit id gets immutable cache headers. The how-to is in docs/features/json-big.md and not in README.
Two things I would do next:
- Non-blocking: git cannot seek, so a late member still streams every byte before it. The docs say so plainly. A cheap next step is a per-request time cap on
blobRange. - Non-blocking: look at the member page at 360px on the preview. The
<pre>scrolls inside its region, but nobody has seen it on a phone yet.
Each key (or array item) in the big-JSON outline now links to
?key=<name>/?index=<n>, and that member opens on its own, read by byte range. Tests,tsc,gate.tsanddesign-lint.tspass locally.scripts/fast.tsnamed 3,152 of 3,662 files, so I ran the JSON-big, blob-window, CSV-prefix and TUI-format tests by hand instead; the forge's full check runs on the push. Thekeep_planandestimatetools were not available in this run, so there is no plan on the issue.What changed
outlineJsoninjson-doc.tsnow records each member's character span.openMemberinjson-big.tsturns that into a byte offset and length and reads only that range through an injected reader.RepoSource.rangeandblobRangeingit/read.tsprovide the reader;repo.tsserves the page.JSON_VALUE_CAP) is pretty-printed.docs/features/json-big.md.What I left out
range, so?key=falls through to the normal page for them.To try it Open a
.jsonfile over 5 MB in a repository and click a key in the outline. Fordata.jsonthat is…/blob/main/data.json?key=alpha.🤖 Generated with Claude Code
Closes #14031