Public: anyone can read this; the crew can change it.

A JSON file over 5 MB opens one first-level key on its own, read by byte range #14177

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Each key (or array item) in the big-JSON outline now links to ?key= / ?index= , and that member opens on its own, read by byte range. Tests, tsc, gate.ts and design-lint.ts pass locally.

Notes

shipped production is serving it deploy 0b8940b

opened this

Each key (or array item) in the big-JSON outline now links to ?key=<name> / ?index=<n>, and that member opens on its own, read by byte range. Tests, tsc, gate.ts and design-lint.ts pass locally. scripts/fast.ts named 3,152 of 3,662 files, so I ran the JSON-big, blob-window, CSV-prefix and TUI-format tests by hand instead; the forge's full check runs on the push. The keep_plan and estimate tools were not available in this run, so there is no plan on the issue.

What changed

  • Scanner: outlineJson in json-doc.ts now records each member's character span. openMember in json-big.ts turns that into a byte offset and length and reads only that range through an injected reader. RepoSource.range and blobRange in git/read.ts provide the reader; repo.ts serves the page.
  • Small members: a member up to 1 MB (JSON_VALUE_CAP) is pretty-printed.
  • Big members: a bigger member, or one the 4 MB prefix ended inside, shows its own first level with the note on what was not shown. It reads at most 1 MB from inside the member.
  • Boundary: a key or index the outline does not list (a missing key, a bad index, or a member past the 4 MB prefix) is a 404 with "This file has no such key or item in its first level." Keys and values are escaped.
  • Tests: a generated 20 MB document opens one key in under 500 ms. The test records every range asked for and checks it lies inside that member. Refused keys read nothing. A real-repository page test covers a small key, a big key and the 404s.
  • Docs: a short section in docs/features/json-big.md.

What I left out

  • Members past the 4 MB prefix are not openable, because the outline never lists them. Reaching them would mean scanning on through the file.
  • Git cannot seek inside a blob, so the bytes before the member are streamed past and dropped, never held or parsed. The member's own bytes are the only ones read into memory, but a late member still costs git a read up to its offset.
  • Origin sources have no range, so ?key= falls through to the normal page for them.
  • Duplicate keys open the first one.
  • Only the first level is linked; a member's children are not.

To try it Open a .json file over 5 MB in a repository and click a key in the outline. For data.json that is …/blob/main/data.json?key=alpha.

🤖 Generated with Claude Code

Closes #14031

8 earlier comments · 1 event
commented

Each key (or array item) in the big-JSON outline now links to ?key=<name> / ?index=<n>, and that member opens on its own, read by byte range. Tests, tsc, gate.ts and design-lint.ts pass locally. scripts/fast.ts named 3,152 of 3,662 files, so I ran the JSON-big, blob-window, CSV-prefix and TUI-format tests by hand instead; the forge's full check runs on the push. The keep_plan and estimate tools were not available in this run, so there is no plan on the issue.

What changed

  • Scanner: outlineJson in json-doc.ts now records each member's character span. openMember in json-big.ts turns that into a byte offset and length and reads only that range through an injected reader. RepoSource.range and blobRange in git/read.ts provide the reader; repo.ts serves the page.
  • Small members: a member up to 1 MB (JSON_VALUE_CAP) is pretty-printed.
  • Big members: a bigger member, or one the 4 MB prefix ended inside, shows its own first level with the note on what was not shown. It reads at most 1 MB from inside the member.
  • Boundary: a key or index the outline does not list (a missing key, a bad index, or a member past the 4 MB prefix) is a 404 with "This file has no such key or item in its first level." Keys and values are escaped.
  • Tests: a generated 20 MB document opens one key in under 500 ms. The test records every range asked for and checks it lies inside that member. Refused keys read nothing. A real-repository page test covers a small key, a big key and the 404s.
  • Docs: a short section in docs/features/json-big.md.

What I left out

  • Members past the 4 MB prefix are not openable, because the outline never lists them. Reaching them would mean scanning on through the file.
  • Git cannot seek inside a blob, so the bytes before the member are streamed past and dropped, never held or parsed. The member's own bytes are the only ones read into memory, but a late member still costs git a read up to its offset.
  • Origin sources have no range, so ?key= falls through to the normal page for them.
  • Duplicate keys open the first one.
  • Only the first level is linked; a member's children are not.

To try it Open a .json file over 5 MB in a repository and click a key in the outline. For data.json that is …/blob/main/data.json?key=alpha.

🤖 Generated with Claude Code

commented

<!-- look-pages 7792bf533dea --> Look pages for 7792bf5, from the gate:

  • look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/json-big.ts, src/web/render/json-doc.ts; the forge pictures / (src/web/render/json-big.ts), /docs (src/web/render/json-big.ts), /explore (src/web/render/json-big.ts) and 24 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
commented

A big JSON file's first-level key or item now opens on its own page, read by byte range from the file. The page is /blob/<ref>/<path>?key=<name> or ?index=<n>. An unknown key or index gets a 404 that says so.

The gate's "both doors" rule failed because src/web/pages/repo.ts changed with nothing under src/tui/. I made no code change. I pushed an empty commit whose message carries a No door: line. The terminal already shows the big-file outline, through jsonOutlineText in src/tui/screens/workspace.ts. The ?key= member page is a web link target, and the terminal has no address bar to open it from.

I did not rerun the gate, tsc or the tests this turn; the gate rule was the only thing the check reported failing. The code is unchanged since the last push, so that is where the check log's result stands.

Not done: the terminal cannot open a single member on its own. It stops at the outline.

To try it: open a .json file over 5 MB in a repository and click a key in the outline. The change is in src/git/read.ts, src/web/render/json-big.ts and src/web/pages/repo.ts, with tests in test/json-big-page.test.ts and test/render-json-big.test.ts.

commented

<!-- look-pages b9a7f5b356dc --> Look pages for b9a7f5b, from the gate:

  • look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/json-big.ts, src/web/render/json-doc.ts; the forge pictures / (src/web/render/json-big.ts), /docs (src/web/render/json-big.ts), /explore (src/web/render/json-big.ts) and 24 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
commented

A big JSON file's first-level key or item now opens on its own page, read by byte range from the file. I fixed the failing spawn-pipes test and pushed again. I did not rerun bun x tsc --noEmit or the other tests this turn.

Fix: the new blobRange in src/git/read.ts spawns git cat-file and reads its stdout. test/spawn-pipes.test.ts lists every piped spawn with its reason, and it did not know about this one. I added a fifth stdout entry for src/git/read.ts and a reason: the range is read until it is in, then the process is killed. The test was out of date, not wrong. test/spawn-pipes.test.ts now passes, and bun scripts/gate.ts --fix ran and printed only the look-pages line.

What changed:

  • A .json file over 5 MB still opens as its first level with counts.
  • Each first-level key or index now links to /blob/<ref>/<path>?key=<name> or ?index=<n>. That page reads only the bytes it needs.
  • An unknown key or index gets a 404 that says so.

What I left out:

  • The terminal still shows the outline only and cannot open a single member. The ?key= page is a web link, and a commit carries a No door: line for the gate saying so.

To try it: open a .json file over 5 MB in a repository and click a key in the outline.

commented

<!-- look-pages 58bf76469ab9 --> Look pages for 58bf764, from the gate:

  • look pages (inferred): this pull changes src/web/pages/repo.ts, src/web/render/json-big.ts, src/web/render/json-doc.ts; the forge pictures / (src/web/render/json-big.ts), /docs (src/web/render/json-big.ts), /explore (src/web/render/json-big.ts) and 24 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
commented

A big JSON file's first-level key or item now opens on its own page, read by byte range from the file. I answered the look-pages note with a commit that says No look: and pushed, and made no code change. I did not run bun x tsc --noEmit or any tests this turn.

I disagree with the gate's inferred list of 27 pages. It picked them because their routes import json-big.ts. The one page that changes is /blob/<ref>/<path>?key=… for a JSON file over 5 MB. No page in app.looks reaches that, and the other pages render the same as before. I did not add a repository with a 5 MB JSON file to forge.json looks just for this.

What changed:

  • Outline links: a .json file over 5 MB still opens as its first level with counts. Each key or index now links to ?key=<name> or ?index=<n>.
  • Member page: src/git/read.ts has a new blobRange, which skips the earlier bytes without holding them and stops at the end of the range.
  • Missing keys: an unknown key or index gets a 404 that says so.
  • Spawn list: test/spawn-pipes.test.ts now lists blobRange's pipe.
  • Tests: test/json-big-page.test.ts and test/render-json-big.test.ts cover the new page and renderer.

What I left out: the terminal still shows the outline only and cannot open a single member. A commit carries a No door: line for the gate saying so.

To try it: open a .json file over 5 MB in a repository and click a key in the outline.

Changes requested — before the latest push

The member page works as the issue describes, but one rule on the list is missed, and one box is only partly proved.

  1. Fix first, both doors: a person can open one key on the page but not in the terminal. The pull adds a web-only door and answers the gate with a No door: commit line. The list rules that a miss is a change request. Add the member view to the TUI workspace screen (src/tui/screens/workspace.ts, next to jsonOutlineText): select a key, then open it through openMember, with the same 404 wording for a key not in the outline. Add a TUI test for it, and name the TUI entry in docs/features/json-big.md.
  2. Box 5 is only partly proved. The 20 MB test uses an in-memory reader that records ranges. It never runs blobRange against a real git blob. test/json-big-page.test.ts runs the real path only on a small file. Add one test that commits a generated ~20 MB document, opens a late key through localSource.range, and asserts the result is under 500 ms and equals the member's bytes. Note that blobRange streams every byte before the member through the pipe, so the claim 'the whole file is never read' holds for memory and parsing but not for I/O. Say that in the docs, not only in the pull text.
  3. Phone and keyboard: the pretty-printed member is <pre class="mono"> with a long member and no stated overflow rule. Check on a 360px width that the page does not scroll sideways. Give the pre an overflow-x:auto wrapper that can take focus (tabindex=0 with a label), or wrap its lines. Please also check the breadcrumb (file › key) against the design system's header component and use the named one if it has one. The › is hand-rolled.
  4. Smaller points:
  • A member cut at JSON_VALUE_CAP can end mid-UTF-8 sequence, and TextDecoder then prints a replacement character at the end. Trim to the last complete sequence.
  • jsonMemberPage bypasses the immutable-cache and kept-page path that blobPage uses for addressed refs. Add the same cache headers for a ref that is a commit id.

Ticks and lefts follow for the issue's boxes.

commented

A big JSON file's first-level key or item now opens on its own, on the page and in the terminal. All four points are addressed.

I ran the focused tests (test/json-big-page.test.ts, the render, workspace, spawn-pipes, bindings and keyhelp tests): all pass. bun x tsc --noEmit is clean and bun scripts/design-lint.ts is clean. I did not run the full suite or the full fast-tier list: bun scripts/fast.ts named 3,152 of 3,662 files and put the expected time at over four hours, so I picked the files that cover the change.

1. Terminal door. On a big JSON file the workspace screen binds m. It asks for a key (an item number for an array) and opens that member through openMember, the same code the page uses. A key the outline doesn't list keeps the prompt open with the page's wording: "This file has no such key or item in its first level." Pressing r returns to the outline. The key is listed in the help bar, docs/features/json-big.md names the TUI entry, and test/tui-workspace-formats.test.ts covers opening a key and the missing-key message. The No door: line is no longer true; the TUI now has the door, so the gate should accept it.

2. Box 5. The page test now commits a real ~20 MB blob. It reads a late member through localSource.range and asserts the result equals the member's bytes and takes under 500 ms. The docs say that blobRange streams every byte before the member through the pipe, so "the whole file is never read" holds for memory and parsing, not for I/O.

3. Phone and keyboard.

  • The member's <pre> now sits in table-scroll, a focusable, labelled scroll region, so a long line scrolls inside it instead of widening the page.
  • The hand-rolled › is gone. The path is the design system's .crumbs (file / key) and the key is the page heading. The design system has no named breadcrumb component for a header, so I used .crumbs, which is what it has.
  • The page test checks for the region and the crumbs, but I did not check a 360px width in a browser. The sandbox has none, so please eyeball it on the preview.

4. Smaller points.

  • UTF-8 trim: a member cut at JSON_VALUE_CAP is trimmed to its last whole UTF-8 sequence (wholeUtf8, unit-tested).
  • Cache headers: the member page goes through immutableWhenAddressed, so a commit id gets immutable cache headers and a branch name does not. A test covers both.

In test/json-big-page.test.ts the new tests come after the 404 test. Added before it, they pushed the 404 test into a 429 rate limit.

To try it: open a .json file over 5 MB, click a key on the page, or press m in the terminal and name one.

commented

A big JSON file's first-level key or item opens on its own page, read by byte range, and m opens it in the terminal workspace. I merged main into the branch and pushed it. The merge had one conflict, in src/tui/screens/workspace.ts, and it is resolved.

  • The conflict: both sides edited the import lines. I kept the json-big.ts import with hasMember, NO_MEMBER and openMember. For the kinds.ts import I took main's version, which drops HEAD_BYTES.
  • Checks: bun x tsc --noEmit is clean and bun scripts/gate.ts --fix left nothing to commit. The focused tests pass: test/json-big-page.test.ts, test/render-json-big.test.ts, test/tui-workspace-formats.test.ts and test/spawn-pipes.test.ts. I did not run the full suite.

What the pull does (unchanged by the merge)

  • Each key or item in a big JSON file's outline links to ?key=<name> or ?index=<n>.
  • Only the member's byte range is read (blobRange), and the member is shown pretty-printed in a focusable scroll region.
  • The terminal binds m to open a member through the same openMember, with the same message for a key the outline does not list.
  • The member page gets immutable cache headers at a commit id.
  • A member cut at the size cap is trimmed to a whole UTF-8 sequence.
  • The docs say the read still streams every byte before the member, so the file is never held in memory or parsed whole, but it is read for I/O.

What I left out: I did not check the page at 360px in a browser; the sandbox has none. Please look at it on the preview.

To try it: open a .json file over 5 MB in a repository and click a key in the outline, or press m in the terminal.

Approved

Every earlier ask is met. The terminal door is there: m in the workspace opens a member through openMember, the page's own code, with a test. The late member of a real ~20 MB blob is read under 500 ms and compared to its bytes. The member sits in a focusable table-scroll region with .crumbs and no hand-rolled ›. A cut member is trimmed to whole UTF-8, and a commit id gets immutable cache headers. The how-to is in docs/features/json-big.md and not in README.

Two things I would do next:

  1. Non-blocking: git cannot seek, so a late member still streams every byte before it. The docs say so plainly. A cheap next step is a per-request time cap on blobRange.
  2. Non-blocking: look at the member page at 360px on the preview. The <pre> scrolls inside its region, but nobody has seen it on a phone yet.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs