Public: anyone can read this; the crew can change it.

A raw file or blob read at a full commit id is cached as immutable, with an ETag and 304 #3989

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/3377-a-raw-file-or-blob-read-at-a-full-commit → main 4 commits fork 1 thread · 1 unresolved

  • Merged. scout/3377-a-raw-file-or-blob-read-at-a-full-commit was merged into main as f9663d3.

Review

Sign in to review this change.
5 files changed +228 −19
README.md +2
@@ -1389,6 +1389,8 @@ The terminal also has an Access command (`:` → Access), a site admin's: it rea
13891389 In the terminal, `:` → System shows the same view (the owner's only): per-core CPU with a block-character history, memory, swap, disk and network rates, and each group's top two processes (no sort; the dialog is read-only), and `:` → Telemetry lists the agents and flips `telemetry:read` for the one you press Enter on.
13901390 An agent the owner grants `telemetry:read` (Settings → Telemetry, one switch per agent, revoked the same way) reads the same through `GET /api/v1/telemetry/{system,flow,sources,history}` and the MCP `flow` and `system` tools; private repositories' checks and apps are renamed `check:private-N` and lose their processes and test files, and `flow` (like `system`, an admin agent account needs the grant too: only a person who is an admin reads without it) names only public repositories' conflicted pulls, every read is listed on that page, and an agent without the grant is refused. `/contribute`, `GET /api/v1/contribute` and the MCP `contribute` tool are open to anyone: what the forge is, the bottleneck now, open issues labelled “help wanted” in public repositories, and how to propose (fork, pull, the same gate). The grant is a table, not a token scope, so no token has to be re-issued.
13911391
1392 **Raw files and caches.** `GET /:owner/:name/raw/:ref/*path` at a full 40-hex commit id is `immutable` for a year: `public, max-age=31536000, immutable` for a public repository, `private, max-age=31536000, immutable` for any other, so a shared cache never keeps private bytes. At a branch or tag it is `private, max-age=0` and revalidates. Every raw answer has the blob's oid as its `ETag`, and a matching `If-None-Match` gets a 304 with the same `Cache-Control`, `nosniff` and sandbox CSP. The HTML blob page is not cached this way (it carries the viewer and a nonce). A public repository that later turns private keeps its year-long `public` copies in shared caches already holding them. See docs/CACHING.md.
1393
13921394 `docs/CACHING.md` is what the forge keeps between requests to make `/map` and
13931395 `/` fast: each map lane's plan and each viewer's feed candidates, stamped with
13941396 SQLite's own write clock so any write is seen on the next read, keyed so one
docs/CACHING.md +32
@@ -382,6 +382,38 @@ warm:
382382 | home KPI row | forge head, UTC hour; admin-gated | 30 s |
383383 | home pinned panel | board blob, panel, forge head; admin-gated | 30 s |
384384
385 ## Raw files at a full commit id (#3381)
386
387 `GET /:owner/:name/raw/:ref/*path` is the one place the forge tells shared
388 caches (browsers, a CDN, a proxy) to keep bytes for a year.
389
390 | `ref` is | repository | `Cache-Control` |
391 | ------------------------- | ---------- | ---------------------------------------- |
392 | 40 hex digits (a commit) | public | `public, max-age=31536000, immutable` |
393 | 40 hex digits (a commit) | not public | `private, max-age=31536000, immutable` |
394 | a branch, tag or anything else | any | `private, max-age=0` (revalidates) |
395
396 - **Why immutable.** A full commit id names one tree forever, so the bytes
397 at a path under it can never change. A branch name moves, so it always
398 revalidates.
399 - **Public vs private.** Only `repo.visibility === "public"` (wired in
400 `rawBlob`) allows `public`. Any other repository is `private`: the
401 browser may keep it, a shared cache may not.
402 - **ETag and 304.** Every raw answer carries the blob's own oid as a strong
403 `ETag`. `If-None-Match` naming it (weak or strong, a list, or `*`) gets a
404 bodiless 304 that keeps the same `Cache-Control`, `nosniff` and sandbox
405 CSP. A branch read changes oid only when the file changes, so it costs
406 one round trip and no body.
407 - **Why the blob page is left out.** `/:owner/:name/blob/...` is HTML: it
408 carries the viewer's name, a CSP nonce and links that change with the
409 repository, so it is not a function of the commit alone. Only the raw
410 bytes are. The origin-browse raw route (`browseRaw`) is also left alone:
411 its source is another host, whose bytes this forge cannot vouch for.
412 - **Known limit.** A public repository that later turns private keeps
413 year-long `public` copies in whatever shared caches already fetched
414 them; the forge cannot recall them. Anyone who needs that undone must
415 purge those caches. A commit id is also not secret once public.
416
385417 ## What is built (#2675)
386418
387419 The map half is #2716's (`web/map_cache.ts`, numbers above). #2675 adds
src/git/read.ts +15 −6
@@ -293,8 +293,14 @@ export type BlobForRender =
293293 readonly kind: "binary";
294294 readonly body: Uint8Array;
295295 readonly size: number;
296 readonly oid: string;
296297 }
297 | { readonly kind: "text"; readonly body: Uint8Array; readonly size: number };
298 | {
299 readonly kind: "text";
300 readonly body: Uint8Array;
301 readonly size: number;
302 readonly oid: string;
303 };
298304
299305 /**
300306 * Read a blob for rendering, size-bounded up front. The object is *stat*ed
@@ -329,9 +335,11 @@ export async function blobForRender(
329335 if (!Number.isSafeInteger(size)) return { kind: "missing" };
330336 if (size > maxBytes) return { kind: "too_large", size };
331337
338 const named = await runGit(["rev-parse", "--verify", spec], { cwd: dir });
339 if (!named.ok) return { kind: "missing" };
332340 const body = await runGitRaw(["cat-file", "blob", spec], { cwd: dir });
333341 if (!body.ok) return { kind: "missing" };
334 return classify(body.value, size);
342 return classify(body.value, size, named.value.trim());
335343 }
336344
337345 /**
@@ -356,17 +364,18 @@ async function viaReader(
356364 if (size > maxBytes) return { kind: "too_large", size };
357365 const object = await reader.object(spec);
358366 if (!object.ok) return { kind: "missing" };
359 return classify(object.value.body, size);
367 return classify(object.value.body, size, object.value.oid);
360368 } catch {
361369 closeReader(dir);
362370 return undefined;
363371 }
364372 }
365373
366 function classify(body: Uint8Array, size: number): BlobForRender {
374 function classify(body: Uint8Array, size: number, oid: string): BlobForRender {
367375 // NUL in the first 8KB is git's own binary heuristic.
368 if (body.subarray(0, 8000).includes(0)) return { kind: "binary", body, size };
369 return { kind: "text", body, size };
376 if (body.subarray(0, 8000).includes(0))
377 return { kind: "binary", body, size, oid };
378 return { kind: "text", body, size, oid };
370379 }
371380
372381 export interface Ref {
src/web/pages/repo.ts +41 −13
@@ -34,6 +34,7 @@ import { blobForRender, type TreeEntry } from "../../git/read.ts";
3434 import { repoDir } from "../../git/repos.ts";
3535 import { estimateTokens } from "../../kernel/tokens-format.ts";
3636 import { safeBack, type Ctx } from "../../http/context.ts";
37 import { holds } from "../../http/etag.ts";
3738 import { json, notFound } from "../../http/respond.ts";
3839 import type { Blob, RepoSource } from "../../view/source.ts";
3940 import {
@@ -913,19 +914,39 @@ function renderedMarkdown(blob: Blob & { kind: "text" }): {
913914 }
914915
915916 /** The bytes, with the headers that stop them ever executing as a page. */
916 export function rawResponse(blob: Blob): Response {
917 export function rawResponse(
918 blob: Blob,
919 cache: {
920 oid?: string;
921 immutable?: boolean;
922 public?: boolean;
923 request?: Request;
924 } = {},
925 ): Response {
917926 if (blob.kind === "too_large") return new Response("", { status: 404 });
918 return new Response(blob.body, {
919 headers: {
920 "content-type":
921 blob.kind === "text"
922 ? "text/plain; charset=utf-8"
923 : "application/octet-stream",
924 "x-content-type-options": "nosniff",
925 "content-security-policy": "default-src 'none'; sandbox",
926 "cache-control": "private, max-age=0",
927 },
928 });
927 // A full commit id names bytes that can never change; a branch name does
928 // not, so it revalidates every time, against the blob's own oid. Only a
929 // public repository's bytes may sit in a shared cache.
930 const etag = cache.oid ? `"${cache.oid}"` : undefined;
931 const control =
932 cache.immutable && etag
933 ? `${cache.public ? "public" : "private"}, max-age=31536000, immutable`
934 : "private, max-age=0";
935 const headers: Record<string, string> = {
936 "content-type":
937 blob.kind === "text"
938 ? "text/plain; charset=utf-8"
939 : "application/octet-stream",
940 "x-content-type-options": "nosniff",
941 "content-security-policy": "default-src 'none'; sandbox",
942 "cache-control": control,
943 };
944 if (etag) {
945 headers.etag = etag;
946 if (cache.request && holds(cache.request, etag))
947 return new Response(null, { status: 304, headers });
948 }
949 return new Response(blob.body, { headers });
929950 }
930951
931952 // GET /:owner/:name — the repo home. An app's home is the app, for every
@@ -1314,5 +1335,12 @@ export async function rawBlob(ctx: Ctx): Promise<Response> {
13141335 const blob = await blobForRender(dir, ref, path, 25_000_000);
13151336 if (blob.kind === "missing" || blob.kind === "too_large")
13161337 return notFoundPage(ctx);
1317 return rawResponse(blob);
1338 return rawResponse(blob, {
1339 oid: blob.oid,
1340 immutable: FULL_OID.test(ref),
1341 public: repo.visibility === "public",
1342 request: ctx.request,
1343 });
13181344 }
1345
1346 const FULL_OID = /^[0-9a-f]{40}$/;

Declare this constant above rawBlob, with the module constants, rather than after the function that uses it.

test/web-ui.test.ts +138
@@ -17,6 +17,7 @@ import { createApp, type App } from "../src/app.ts";
1717 import { getRepo } from "../src/core/repos.ts";
1818 import { listLabels, labelsFor, getIssue } from "../src/core/issues.ts";
1919 import { serve, type Servers } from "../src/http/server.ts";
20 import { rawResponse } from "../src/web/pages/repo.ts";
2021 import { safeNext } from "../src/web/pages/site.ts";
2122 import { shellJs } from "../src/web/client.ts";
2223 import { gitClientEnv } from "./helpers/git-client.ts";
@@ -268,6 +269,143 @@ describe("web ui", () => {
268269
269270 // ---- raw ----
270271
272 async function revParse(spec: string): Promise<string> {
273 const proc = Bun.spawn(["git", "rev-parse", spec], {
274 cwd: join(work, "proj"),
275 env: gitEnv,
276 stdout: "pipe",
277 });
278 return (await new Response(proc.stdout).text()).trim();
279 }
280
281 test("raw at a full commit id is immutable, with the blob oid as ETag and a 304", async () => {
282 const commit = await revParse("main");
283 const blobOid = await revParse("main:a.txt");
284 const url = `${base}/mel/proj/raw/${commit}/a.txt`;
285 const res = await fetch(url);
286 expect(res.status).toBe(200);
287 expect(res.headers.get("cache-control")).toBe(
288 "public, max-age=31536000, immutable",
289 );
290 expect(res.headers.get("etag")).toBe(`"${blobOid}"`);
291 expect(res.headers.get("content-security-policy")).toBe(
292 "default-src 'none'; sandbox",
293 );
294 expect(res.headers.get("x-content-type-options")).toBe("nosniff");
295 await res.text();
296
297 const again = await fetch(url, {
298 headers: { "if-none-match": `"${blobOid}"` },
299 });
300 expect(again.status).toBe(304);
301 expect(await again.text()).toBe("");
302 expect(again.headers.get("cache-control")).toContain("immutable");
303 });
304
305 test("raw at a branch name revalidates against the blob oid", async () => {
306 const blobOid = await revParse("main:a.txt");
307 const res = await fetch(`${base}/mel/proj/raw/main/a.txt`);
308 expect(res.headers.get("cache-control")).toBe("private, max-age=0");
309 expect(res.headers.get("etag")).toBe(`"${blobOid}"`);
310 await res.text();
311 const again = await fetch(`${base}/mel/proj/raw/main/a.txt`, {
312 headers: { "if-none-match": `"${blobOid}"` },
313 });
314 expect(again.status).toBe(304);
315 expect(await again.text()).toBe("");
316 const other = await fetch(`${base}/mel/proj/raw/main/a.txt`, {
317 headers: { "if-none-match": '"0000"' },
318 });
319 expect(other.status).toBe(200);
320 await other.text();
321 });
322
323 test("raw answers 304 to a weak tag, a list holding the tag, and *", () => {
324 const oid = "b".repeat(40);
325 const blob = {
326 kind: "text" as const,
327 body: new TextEncoder().encode("x"),
328 size: 1,
329 oid,
330 };
331 const ask = (inm: string) =>
332 rawResponse(blob, {
333 oid,
334 immutable: true,
335 public: true,
336 request: new Request("http://x/", {
337 headers: { "if-none-match": inm },
338 }),
339 }).status;
340 expect(ask(`W/"${oid}"`)).toBe(304);
341 expect(ask(`"zzz", "${oid}"`)).toBe(304);
342 expect(ask("*")).toBe(304);
343 expect(ask('"zzz"')).toBe(200);
344 });
345
346 test("raw of a repository that is not public is never cached as public", () => {
347 const blob = {
348 kind: "text" as const,
349 body: new TextEncoder().encode("x"),
350 size: 1,
351 oid: "a".repeat(40),
352 };
353 const res = rawResponse(blob, {
354 oid: blob.oid,
355 immutable: true,
356 public: false,
357 });
358 expect(res.headers.get("cache-control")).toBe(
359 "private, max-age=31536000, immutable",
360 );
361 expect(res.headers.get("content-security-policy")).toContain("sandbox");
362 });
363
364 test("raw of a private repository at a full commit id is private, over HTTP", async () => {
365 await fetch(`${base}/new`, {
366 method: "POST",
367 headers: { cookie: owner },
368 body: form({ name: "hush", visibility: "private" }),
369 redirect: "manual",
370 });
371 const dir = join(work, "hush");
372 await mkdir(dir, { recursive: true });
373 await git(["init", "-q", "-b", "main", "."], dir);
374 await writeFile(join(dir, "s.txt"), "secret\n");
375 await git(["add", "."], dir);
376 await git(["commit", "-q", "-m", "s"], dir);
377 await git(
378 [
379 "push",
380 "-q",
381 `http://mel:web-ui-password@localhost:${servers.port}/mel/hush.git`,
382 "main",
383 ],
384 dir,
385 );
386 const proc = Bun.spawn(["git", "rev-parse", "HEAD"], {
387 cwd: dir,
388 env: gitEnv,
389 stdout: "pipe",
390 });
391 const commit = (await new Response(proc.stdout).text()).trim();
392 const url = `${base}/mel/hush/raw/${commit}/s.txt`;
393 const res = await fetch(url, { headers: { cookie: owner } });
394 expect(res.status).toBe(200);
395 expect(res.headers.get("cache-control")).toBe(
396 "private, max-age=31536000, immutable",
397 );
398 expect(res.headers.get("cache-control")).not.toContain("public");
399 await res.text();
400 const etag = res.headers.get("etag")!;
401 const again = await fetch(url, {
402 headers: { cookie: owner, "if-none-match": etag },
403 });
404 expect(again.status).toBe(304);
405 expect(again.headers.get("cache-control")).not.toContain("public");
406 expect((await fetch(url)).status).toBe(404);
407 });
408
271409 test("raw serves bytes as text with nosniff, and conflates not-found", async () => {
272410 const res = await fetch(`${base}/mel/proj/raw/main/a.txt`);
273411 expect(res.status).toBe(200);

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs