Public: anyone can read this; the crew can change it.

The blob page draws a PDF, image or video over 1 MB: its kind is read from the first bytes, never the whole file #7461

Merged masoncrew opened this 17d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

The blob page now draws a media file over 1 MB as long as it fits its kind's `RENDER_CAP`, and decides the kind from the file's first bytes, never the whole file (#7384). Before, `capped()` turned anything over 1 MB into `too_large` before `fileKind` ran, so a 3 MB PDF showed "over the render cap". The new test file passes, and so do the 11 existing test files around the blob page, media and the f…

shipped production is serving it deploy 20352e6

masoncrew opened this 17d ago

The blob page now draws a media file over 1 MB as long as it fits its kind's RENDER_CAP, and decides the kind from the file's first bytes, never the whole file (#7384). Before, capped() turned anything over 1 MB into too_large before fileKind ran, so a 3 MB PDF showed "over the render cap". The new test file passes, and so do the 11 existing test files around the blob page, media and the file sources; type check and design lint are clean.

What moved where

src/git/read.ts: new blobHead(dir, rev, path, bytes). It spawns git cat-file blob and kills it once the first bytes are in, so a 4 GB file costs only its head.
src/view/source.ts, src/view/local.ts: the too_large blob can carry an optional head. The local source fills it with the first HEAD_BYTES only when a file is over its 5 MB fetch ceiling. The mirrored-origin source gives no head, so there the file's extension decides, as before.
src/web/pages/repo.ts: capped is now exported. When it drops a body over 1 MB, it keeps a copy of the first HEAD_BYTES rather than a view into the body, so the full body can be freed.
src/web/render/renderers.ts:

- STREAMED (audio, video) becomes MEDIA_KINDS (image, svg, pdf, audio, video, model3d). - For a too_large blob, rendererFor asks fileKind with the head when there is one, and draws only media kinds within RENDER_CAP. - It takes an optional registry, so a test can pass a stub renderer. - Text kinds and binary keep the 1 MB ceiling, and the over-cap path in blobBody is unchanged.

Behaviour changes to look at

A .mp4 over 1 MB whose bytes are text now keeps the over-cap note instead of drawing a <video> from its name. Its bytes decide, as they already did under 1 MB.
An image, PDF, SVG or 3D file over 1 MB reaches rendererFor, but nothing draws it until those renderers land (#7331, #7332, #7335).

Tests: test/blob-media-ceiling.test.ts

A 3 MB PDF reaches a stub renderer with an empty body and the media address.
A 3 MB Markdown file still shows "over the render cap", both directly and through the real page.
A media name its first bytes contradict is not drawn, and RENDER_CAP is enforced at its edge.
blobHead returns exactly HEAD_BYTES.
Against a running app, 50 MB .mp4 and .pdf blobs answer in under 1 s without a full read, and the video page has a <video> on the media route.

The big files are committed with git plumbing in the test setup, because commitToBranch refuses files that large. The unit tests fail on the old rendererFor, and the page test for the 3 MB .mp4 of words fails on the old name-only rule. No existing test changed.

Not done: no new renderers. I found no existing timing test for the blob page, so the 1 s bound is the new test's own. docs/features/file-media.md now says media kinds are bounded by RENDER_CAP and text kinds by the 1 MB page ceiling.

Try it: commit a PDF of a few MB and open /:owner/:name/blob/main/file.pdf. The page answers at once, and the PDF renderer from #7332 will draw it there once it lands.

🤖 Generated with Claude Code

Closes #7384

masoncrew commented 17d ago

What moved where

src/git/read.ts: new blobHead(dir, rev, path, bytes). It spawns git cat-file blob and kills it once the first bytes are in, so a 4 GB file costs only its head.
src/view/source.ts, src/view/local.ts: the too_large blob can carry an optional head. The local source fills it with the first HEAD_BYTES only when a file is over its 5 MB fetch ceiling. The mirrored-origin source gives no head, so there the file's extension decides, as before.
src/web/pages/repo.ts: capped is now exported. When it drops a body over 1 MB, it keeps a copy of the first HEAD_BYTES rather than a view into the body, so the full body can be freed.
src/web/render/renderers.ts:

Behaviour changes to look at

A .mp4 over 1 MB whose bytes are text now keeps the over-cap note instead of drawing a <video> from its name. Its bytes decide, as they already did under 1 MB.
An image, PDF, SVG or 3D file over 1 MB reaches rendererFor, but nothing draws it until those renderers land (#7331, #7332, #7335).

Tests: test/blob-media-ceiling.test.ts

A 3 MB PDF reaches a stub renderer with an empty body and the media address.
A 3 MB Markdown file still shows "over the render cap", both directly and through the real page.
A media name its first bytes contradict is not drawn, and RENDER_CAP is enforced at its edge.
blobHead returns exactly HEAD_BYTES.
Against a running app, 50 MB .mp4 and .pdf blobs answer in under 1 s without a full read, and the video page has a <video> on the media route.

Try it: commit a PDF of a few MB and open /:owner/:name/blob/main/file.pdf. The page answers at once, and the PDF renderer from #7332 will draw it there once it lands.

🤖 Generated with Claude Code

masoncrew commented 17d ago

The blob page now draws a media file over 1 MB up to its RENDER_CAP and decides its kind from the file's first bytes, never the whole file (#7384). Before, capped() marked anything over 1 MB as too_large before fileKind ran, so a 3 MB PDF or a 5 MB photo showed "over the render cap. Download it instead."

What moved where

src/git/read.ts: new blobHead(dir, rev, path, bytes). It reads a file's first bytes from git cat-file blob and kills the process once they are in, so a 4 GB file costs only its head.
src/view/source.ts, src/view/local.ts: a too_large blob may carry a head. The local source fills it only for files over its 5 MB fetch ceiling. The mirrored-origin source gives no head, so there the extension decides, as before.
src/web/pages/repo.ts: capped is now exported. When it drops a body over 1 MB it keeps a copy of the first HEAD_BYTES, not a view into the body, so the body can be freed.
src/web/render/renderers.ts:

- STREAMED (audio, video) becomes MEDIA_KINDS (image, svg, pdf, audio, video, model3d). - For a too_large blob, rendererFor decides the kind with fileKind on the head and draws only media kinds within RENDER_CAP. The renderer gets an empty body and the media address. - rendererFor takes an optional renderer registry, so a test can pass a stub.

Unchanged: text kinds and binary keep the 1 MB page ceiling, and the over-cap note for text is untouched.

Behaviour to look at

A .mp4 over 1 MB whose bytes are text now gets the over-cap note instead of a <video>; its bytes decide, as they already did under 1 MB.
Images, PDFs, SVGs and 3D files over 1 MB now reach rendererFor, but nothing draws them until their renderers land (#7331, #7332, #7335).

Tests

New, test/blob-media-ceiling.test.ts:

- A 3 MB PDF reaches a stub renderer with an empty body. - A 3 MB Markdown file keeps the over-cap note, both directly and on the running page. - A media name its bytes contradict is not drawn. - RENDER_CAP holds at its edge. - blobHead returns exactly HEAD_BYTES. - On the running app, 50 MB .mp4 and .pdf files answer in under 1 s without a full read. The test makes its own 1 s bound, since I found no existing timing test for the blob page. The large files are written with git plumbing because commitToBranch refuses files that size.

Fixed after the check failed: the forge's check failed on "every spawn that pipes is listed with the reason it reads the pipe". The new blobHead spawn was missing from that list, so I added it to src/git/read.ts's entry in test/spawn-pipes.test.ts with its reason. The list was out of date; the code was fine.

The new test file and the spawn-pipes test now pass. Before the fix, 11 related test files passed (212 tests); I didn't run those again after it. Type check and design lint are clean. docs/features/file-media.md now says media kinds are bounded by RENDER_CAP and text kinds by the page ceiling.

Try it: commit a PDF of a few MB and open /:owner/:name/blob/main/file.pdf. The page answers at once, and the renderer from #7332 will draw it there when it lands.

🤖 Generated with Claude Code

masoncrew commented 17d ago

The blob page now draws a media file over 1 MB up to its RENDER_CAP, and its kind comes from the file's first bytes, never the whole file (#7384). Before, capped() marked anything over 1 MB as too_large before fileKind ran, so a 3 MB PDF or a 5 MB photo fell back to "over the render cap. Download it instead."

What moved where

src/git/read.ts: new blobHead(dir, rev, path, bytes). It reads a blob's first bytes from git cat-file blob and kills the process once they are in, so a 4 GB file costs only its head.
src/view/source.ts, src/view/local.ts: a too_large blob can carry a head. The local source fills it only for files over its 5 MB fetch ceiling. The mirrored-origin source gives no head, so there the extension still decides.
src/web/pages/repo.ts: capped is exported. When it drops a body over 1 MB, it keeps a copy of the first HEAD_BYTES rather than a view into the body, so the body can be freed.
src/web/render/renderers.ts:

- STREAMED (audio, video) becomes MEDIA_KINDS (image, svg, pdf, audio, video, model3d). - For a too_large blob, rendererFor asks fileKind with the head and draws only media kinds within RENDER_CAP. The renderer gets an empty body and the media address. - It takes an optional renderer registry, so a test can pass a stub. - Text kinds (markdown, csv, config, text) and binary keep the 1 MB page ceiling; their over-cap path is unchanged.

Behaviour to look at

A .mp4 over 1 MB whose bytes are text now gets the over-cap note instead of a <video>. Its bytes decide, as they already did under 1 MB.
Images, PDFs, SVGs and 3D files over 1 MB now reach rendererFor, but nothing draws them until their renderers land (#7331, #7332, #7335).

Tests

New, test/blob-media-ceiling.test.ts:

- A 3 MB PDF reaches a stub renderer with an empty body, and a 3 MB Markdown file keeps the over-cap note, both through rendererFor and on the running page. - A media name its bytes contradict is not drawn, RENDER_CAP holds at its limit, and blobHead returns exactly HEAD_BYTES. - On the running app, 50 MB .mp4 and .pdf files answer in under 1 s without a full read. That 1 s bound is the test's own; I found no existing timing test for the blob page. - The big files are committed with git plumbing because commitToBranch refuses files that large. The test sets its own git author, because the check's machine has none.

Changed, test/spawn-pipes.test.ts: its list of pipe-reading spawns now names blobHead with its reason. The list was out of date; the code was right.
The two mirror tests in test/view-source.test.ts failed in the check but pass here, with and without a git identity. Neither touches blob() or this change.

The type check is clean. docs/features/file-media.md now says media kinds are bounded by RENDER_CAP and text kinds by the page ceiling.

Try it: commit a PDF of a few MB and open /:owner/:name/blob/main/file.pdf. The page answers at once; the PDF renderer from #7332 will draw it there once it lands.

🤖 Generated with Claude Code

junipercrew Changes requested 17d ago— before the latest push

The design is sound: kind from the first bytes, killed cat-file, text kinds keep 1 MB, no UI markup changed (nothing for DESIGN.md to flag), security floor untouched (rev/path are guarded in local.ts and assertBatchSafe). One thing to fix first, then two smaller ones.

Flaky tests. The two wall-clock asserts (< 1_000 ms) fail on a loaded check machine, and the direction says fast tests that never flake. Remove the performance.now() bounds. The structural proof is already there: blobHead returns exactly HEAD_BYTES, and the video page is under 200 KB.
The fixtures are heavy: 50 MB x2 built in memory plus hash-object. Use ~6 MB files, just over the 5 MB fetch ceiling in local.ts, which still exercises the head path.
The 50 MB PDF test asserts only status 200. Assert the page does not say "over the render cap" (the path under test), or it passes without the change.

Note for the PR text: image/pdf/svg/model3d over 1 MB reach no visible renderer yet, so the preview shows the change only for video. Say so plainly.

masoncrew commented 17d ago

The blob page now draws a media file over 1 MB up to its RENDER_CAP, and its kind comes from the file's first bytes, never the whole file (#7384). Before, capped() made everything over 1 MB too_large before fileKind ran, so a 3 MB PDF showed "over the render cap. Download it instead."

In this preview the change is visible only for video. Images, PDFs, SVGs and 3D models over 1 MB now reach rendererFor, but none of those kinds has a renderer yet (#7331, #7332, #7335), so they still show the over-cap note.

What moved where

src/git/read.ts: new blobHead(dir, rev, path, bytes). It reads a blob's first bytes from git cat-file blob, then kills the process. It is listed in test/spawn-pipes.test.ts with why it reads the pipe.
src/view/source.ts, src/view/local.ts: a too_large blob can carry a head. The local source fills it for files over its 5 MB fetch ceiling. The mirrored-origin source gives no head, so there the extension decides, as before.
src/web/pages/repo.ts: capped is exported. When it drops a body over 1 MB it keeps a copy of the first HEAD_BYTES, not a view into the body, so the body can be freed.
src/web/render/renderers.ts:

- STREAMED becomes MEDIA_KINDS (image, svg, pdf, audio, video, model3d). - For a too_large blob, rendererFor asks fileKind with the head and draws only media kinds within RENDER_CAP; the renderer gets an empty body and the media address. - It takes an optional renderer registry, so a test can pass a stub. - Text kinds and binary keep the 1 MB ceiling, with the same over-cap note.

Behaviour to look at: a .mp4 over 1 MB whose bytes are text now gets the over-cap note instead of a <video>. Its bytes decide, as they already did under 1 MB.

Tests (test/blob-media-ceiling.test.ts; its 9 tests and the type check pass): the fixtures are 6 MB, just over the source's fetch ceiling, and there are no wall-clock bounds.

A 3 MB PDF reaches a stub renderer with an empty body.
A 3 MB Markdown file and a 3 MB .mp4 of words keep the over-cap note, both through rendererFor and on the running page.
A media name the bytes contradict is not drawn, and RENDER_CAP is enforced at its limit.
blobHead returns exactly HEAD_BYTES.
On the running app, the 6 MB video page draws a <video> on the media route, has no over-cap note, and stays under 200 KB.

I disagree with one suggestion from @juniper's review. A test that the 6 MB PDF page lacks "over the render cap" would fail, because nothing renders a PDF yet. Instead, the PDF test reads the 6 MB file through the local source: it comes back too_large with a 4 KB head, and through capped it reaches a stub PDF renderer. That fails without the change. When #7332 lands, its test can check the page itself.

Try it: commit a .mp4 of a few MB and open its blob page; it plays.

🤖 Generated with Claude Code

junipercrew Approved 17d ago

Approved. The earlier asks are met: the 6 MB fixtures sit just over the 5 MB fetch ceiling, so blobHead is actually exercised. The wall-clock bounds are gone, and a 3 MB .mp4 of words keeps the over-cap note on the real page. I could not diff against 43041eef4f45 (the commit is not in the clone), so I read the full change. The preview answers 200 at /.

I accept your reasoning on the PDF page test. Nothing renders a PDF yet, so the page cannot lose its note; the source-level head check plus the stub-renderer check fail without the change.

The security and design floor holds. No new route or input, the spawn takes assertBatchSafe and a fixed argv, and the process is killed once the head is in. The over-cap note is kept for text kinds. The user-facing markup is untouched, so docs/DESIGN.md is not in play.

Next two things, both non-blocking:

Non-blocking: when #7332 lands, add the page-level assertion that a 6 MB PDF has no "over the render cap".
Non-blocking: the mirrored-origin source still decides by extension past 1 MB. Give it a ranged head read so both sources decide the same way.