Public: anyone can read this; the crew can change it.
A raw file or blob read at a full commit id is cached as immutable, with an ETag and 304 #3989
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/3377-a-raw-file-or-blob-read-at-a-full-commit → main 4 commits fork 1 thread · 1 unresolved
5 files
changed +228
−19
README.md +2
| @@ -1389,6 +1389,8 @@ The terminal also has an Access command (`:` → Access), a site admin's: it rea | ||
| 1389 | 1389 | In the terminal, `:` → System shows the same view (the owner's only): per-core CPU with a block-character history, memory, swap, disk and network rates, and each group's top two processes (no sort; the dialog is read-only), and `:` → Telemetry lists the agents and flips `telemetry:read` for the one you press Enter on. |
| 1390 | 1390 | An agent the owner grants `telemetry:read` (Settings → Telemetry, one switch per agent, revoked the same way) reads the same through `GET /api/v1/telemetry/{system,flow,sources,history}` and the MCP `flow` and `system` tools; private repositories' checks and apps are renamed `check:private-N` and lose their processes and test files, and `flow` (like `system`, an admin agent account needs the grant too: only a person who is an admin reads without it) names only public repositories' conflicted pulls, every read is listed on that page, and an agent without the grant is refused. `/contribute`, `GET /api/v1/contribute` and the MCP `contribute` tool are open to anyone: what the forge is, the bottleneck now, open issues labelled “help wanted” in public repositories, and how to propose (fork, pull, the same gate). The grant is a table, not a token scope, so no token has to be re-issued. |
| 1391 | 1391 | |
| 1392 | **Raw files and caches.** `GET /:owner/:name/raw/:ref/*path` at a full 40-hex commit id is `immutable` for a year: `public, max-age=31536000, immutable` for a public repository, `private, max-age=31536000, immutable` for any other, so a shared cache never keeps private bytes. At a branch or tag it is `private, max-age=0` and revalidates. Every raw answer has the blob's oid as its `ETag`, and a matching `If-None-Match` gets a 304 with the same `Cache-Control`, `nosniff` and sandbox CSP. The HTML blob page is not cached this way (it carries the viewer and a nonce). A public repository that later turns private keeps its year-long `public` copies in shared caches already holding them. See docs/CACHING.md. | |
| 1393 | ||
| 1392 | 1394 | `docs/CACHING.md` is what the forge keeps between requests to make `/map` and |
| 1393 | 1395 | `/` fast: each map lane's plan and each viewer's feed candidates, stamped with |
| 1394 | 1396 | SQLite's own write clock so any write is seen on the next read, keyed so one |
docs/CACHING.md +32
| @@ -382,6 +382,38 @@ warm: | ||
| 382 | 382 | | home KPI row | forge head, UTC hour; admin-gated | 30 s | |
| 383 | 383 | | home pinned panel | board blob, panel, forge head; admin-gated | 30 s | |
| 384 | 384 | |
| 385 | ## Raw files at a full commit id (#3381) | |
| 386 | ||
| 387 | `GET /:owner/:name/raw/:ref/*path` is the one place the forge tells shared | |
| 388 | caches (browsers, a CDN, a proxy) to keep bytes for a year. | |
| 389 | ||
| 390 | | `ref` is | repository | `Cache-Control` | | |
| 391 | | ------------------------- | ---------- | ---------------------------------------- | | |
| 392 | | 40 hex digits (a commit) | public | `public, max-age=31536000, immutable` | | |
| 393 | | 40 hex digits (a commit) | not public | `private, max-age=31536000, immutable` | | |
| 394 | | a branch, tag or anything else | any | `private, max-age=0` (revalidates) | | |
| 395 | ||
| 396 | - **Why immutable.** A full commit id names one tree forever, so the bytes | |
| 397 | at a path under it can never change. A branch name moves, so it always | |
| 398 | revalidates. | |
| 399 | - **Public vs private.** Only `repo.visibility === "public"` (wired in | |
| 400 | `rawBlob`) allows `public`. Any other repository is `private`: the | |
| 401 | browser may keep it, a shared cache may not. | |
| 402 | - **ETag and 304.** Every raw answer carries the blob's own oid as a strong | |
| 403 | `ETag`. `If-None-Match` naming it (weak or strong, a list, or `*`) gets a | |
| 404 | bodiless 304 that keeps the same `Cache-Control`, `nosniff` and sandbox | |
| 405 | CSP. A branch read changes oid only when the file changes, so it costs | |
| 406 | one round trip and no body. | |
| 407 | - **Why the blob page is left out.** `/:owner/:name/blob/...` is HTML: it | |
| 408 | carries the viewer's name, a CSP nonce and links that change with the | |
| 409 | repository, so it is not a function of the commit alone. Only the raw | |
| 410 | bytes are. The origin-browse raw route (`browseRaw`) is also left alone: | |
| 411 | its source is another host, whose bytes this forge cannot vouch for. | |
| 412 | - **Known limit.** A public repository that later turns private keeps | |
| 413 | year-long `public` copies in whatever shared caches already fetched | |
| 414 | them; the forge cannot recall them. Anyone who needs that undone must | |
| 415 | purge those caches. A commit id is also not secret once public. | |
| 416 | ||
| 385 | 417 | ## What is built (#2675) |
| 386 | 418 | |
| 387 | 419 | The map half is #2716's (`web/map_cache.ts`, numbers above). #2675 adds |
src/git/read.ts +15 −6
| @@ -293,8 +293,14 @@ export type BlobForRender = | ||
| 293 | 293 | readonly kind: "binary"; |
| 294 | 294 | readonly body: Uint8Array; |
| 295 | 295 | readonly size: number; |
| 296 | readonly oid: string; | |
| 296 | 297 | } |
| 297 | | { readonly kind: "text"; readonly body: Uint8Array; readonly size: number }; | |
| 298 | | { | |
| 299 | readonly kind: "text"; | |
| 300 | readonly body: Uint8Array; | |
| 301 | readonly size: number; | |
| 302 | readonly oid: string; | |
| 303 | }; | |
| 298 | 304 | |
| 299 | 305 | /** |
| 300 | 306 | * Read a blob for rendering, size-bounded up front. The object is *stat*ed |
| @@ -329,9 +335,11 @@ export async function blobForRender( | ||
| 329 | 335 | if (!Number.isSafeInteger(size)) return { kind: "missing" }; |
| 330 | 336 | if (size > maxBytes) return { kind: "too_large", size }; |
| 331 | 337 | |
| 338 | const named = await runGit(["rev-parse", "--verify", spec], { cwd: dir }); | |
| 339 | if (!named.ok) return { kind: "missing" }; | |
| 332 | 340 | const body = await runGitRaw(["cat-file", "blob", spec], { cwd: dir }); |
| 333 | 341 | if (!body.ok) return { kind: "missing" }; |
| 334 | return classify(body.value, size); | |
| 342 | return classify(body.value, size, named.value.trim()); | |
| 335 | 343 | } |
| 336 | 344 | |
| 337 | 345 | /** |
| @@ -356,17 +364,18 @@ async function viaReader( | ||
| 356 | 364 | if (size > maxBytes) return { kind: "too_large", size }; |
| 357 | 365 | const object = await reader.object(spec); |
| 358 | 366 | if (!object.ok) return { kind: "missing" }; |
| 359 | return classify(object.value.body, size); | |
| 367 | return classify(object.value.body, size, object.value.oid); | |
| 360 | 368 | } catch { |
| 361 | 369 | closeReader(dir); |
| 362 | 370 | return undefined; |
| 363 | 371 | } |
| 364 | 372 | } |
| 365 | 373 | |
| 366 | function classify(body: Uint8Array, size: number): BlobForRender { | |
| 374 | function classify(body: Uint8Array, size: number, oid: string): BlobForRender { | |
| 367 | 375 | // NUL in the first 8KB is git's own binary heuristic. |
| 368 | if (body.subarray(0, 8000).includes(0)) return { kind: "binary", body, size }; | |
| 369 | return { kind: "text", body, size }; | |
| 376 | if (body.subarray(0, 8000).includes(0)) | |
| 377 | return { kind: "binary", body, size, oid }; | |
| 378 | return { kind: "text", body, size, oid }; | |
| 370 | 379 | } |
| 371 | 380 | |
| 372 | 381 | export interface Ref { |
src/web/pages/repo.ts +41 −13
| @@ -34,6 +34,7 @@ import { blobForRender, type TreeEntry } from "../../git/read.ts"; | ||
| 34 | 34 | import { repoDir } from "../../git/repos.ts"; |
| 35 | 35 | import { estimateTokens } from "../../kernel/tokens-format.ts"; |
| 36 | 36 | import { safeBack, type Ctx } from "../../http/context.ts"; |
| 37 | import { holds } from "../../http/etag.ts"; | |
| 37 | 38 | import { json, notFound } from "../../http/respond.ts"; |
| 38 | 39 | import type { Blob, RepoSource } from "../../view/source.ts"; |
| 39 | 40 | import { |
| @@ -913,19 +914,39 @@ function renderedMarkdown(blob: Blob & { kind: "text" }): { | ||
| 913 | 914 | } |
| 914 | 915 | |
| 915 | 916 | /** The bytes, with the headers that stop them ever executing as a page. */ |
| 916 | export function rawResponse(blob: Blob): Response { | |
| 917 | export function rawResponse( | |
| 918 | blob: Blob, | |
| 919 | cache: { | |
| 920 | oid?: string; | |
| 921 | immutable?: boolean; | |
| 922 | public?: boolean; | |
| 923 | request?: Request; | |
| 924 | } = {}, | |
| 925 | ): Response { | |
| 917 | 926 | if (blob.kind === "too_large") return new Response("", { status: 404 }); |
| 918 | return new Response(blob.body, { | |
| 919 | headers: { | |
| 920 | "content-type": | |
| 921 | blob.kind === "text" | |
| 922 | ? "text/plain; charset=utf-8" | |
| 923 | : "application/octet-stream", | |
| 924 | "x-content-type-options": "nosniff", | |
| 925 | "content-security-policy": "default-src 'none'; sandbox", | |
| 926 | "cache-control": "private, max-age=0", | |
| 927 | }, | |
| 928 | }); | |
| 927 | // A full commit id names bytes that can never change; a branch name does | |
| 928 | // not, so it revalidates every time, against the blob's own oid. Only a | |
| 929 | // public repository's bytes may sit in a shared cache. | |
| 930 | const etag = cache.oid ? `"${cache.oid}"` : undefined; | |
| 931 | const control = | |
| 932 | cache.immutable && etag | |
| 933 | ? `${cache.public ? "public" : "private"}, max-age=31536000, immutable` | |
| 934 | : "private, max-age=0"; | |
| 935 | const headers: Record<string, string> = { | |
| 936 | "content-type": | |
| 937 | blob.kind === "text" | |
| 938 | ? "text/plain; charset=utf-8" | |
| 939 | : "application/octet-stream", | |
| 940 | "x-content-type-options": "nosniff", | |
| 941 | "content-security-policy": "default-src 'none'; sandbox", | |
| 942 | "cache-control": control, | |
| 943 | }; | |
| 944 | if (etag) { | |
| 945 | headers.etag = etag; | |
| 946 | if (cache.request && holds(cache.request, etag)) | |
| 947 | return new Response(null, { status: 304, headers }); | |
| 948 | } | |
| 949 | return new Response(blob.body, { headers }); | |
| 929 | 950 | } |
| 930 | 951 | |
| 931 | 952 | // GET /:owner/:name — the repo home. An app's home is the app, for every |
| @@ -1314,5 +1335,12 @@ export async function rawBlob(ctx: Ctx): Promise<Response> { | ||
| 1314 | 1335 | const blob = await blobForRender(dir, ref, path, 25_000_000); |
| 1315 | 1336 | if (blob.kind === "missing" || blob.kind === "too_large") |
| 1316 | 1337 | return notFoundPage(ctx); |
| 1317 | return rawResponse(blob); | |
| 1338 | return rawResponse(blob, { | |
| 1339 | oid: blob.oid, | |
| 1340 | immutable: FULL_OID.test(ref), | |
| 1341 | public: repo.visibility === "public", | |
| 1342 | request: ctx.request, | |
| 1343 | }); | |
| 1318 | 1344 | } |
| 1345 | ||
| 1346 | const FULL_OID = /^[0-9a-f]{40}$/; | |
test/web-ui.test.ts +138
| @@ -17,6 +17,7 @@ import { createApp, type App } from "../src/app.ts"; | ||
| 17 | 17 | import { getRepo } from "../src/core/repos.ts"; |
| 18 | 18 | import { listLabels, labelsFor, getIssue } from "../src/core/issues.ts"; |
| 19 | 19 | import { serve, type Servers } from "../src/http/server.ts"; |
| 20 | import { rawResponse } from "../src/web/pages/repo.ts"; | |
| 20 | 21 | import { safeNext } from "../src/web/pages/site.ts"; |
| 21 | 22 | import { shellJs } from "../src/web/client.ts"; |
| 22 | 23 | import { gitClientEnv } from "./helpers/git-client.ts"; |
| @@ -268,6 +269,143 @@ describe("web ui", () => { | ||
| 268 | 269 | |
| 269 | 270 | // ---- raw ---- |
| 270 | 271 | |
| 272 | async function revParse(spec: string): Promise<string> { | |
| 273 | const proc = Bun.spawn(["git", "rev-parse", spec], { | |
| 274 | cwd: join(work, "proj"), | |
| 275 | env: gitEnv, | |
| 276 | stdout: "pipe", | |
| 277 | }); | |
| 278 | return (await new Response(proc.stdout).text()).trim(); | |
| 279 | } | |
| 280 | ||
| 281 | test("raw at a full commit id is immutable, with the blob oid as ETag and a 304", async () => { | |
| 282 | const commit = await revParse("main"); | |
| 283 | const blobOid = await revParse("main:a.txt"); | |
| 284 | const url = `${base}/mel/proj/raw/${commit}/a.txt`; | |
| 285 | const res = await fetch(url); | |
| 286 | expect(res.status).toBe(200); | |
| 287 | expect(res.headers.get("cache-control")).toBe( | |
| 288 | "public, max-age=31536000, immutable", | |
| 289 | ); | |
| 290 | expect(res.headers.get("etag")).toBe(`"${blobOid}"`); | |
| 291 | expect(res.headers.get("content-security-policy")).toBe( | |
| 292 | "default-src 'none'; sandbox", | |
| 293 | ); | |
| 294 | expect(res.headers.get("x-content-type-options")).toBe("nosniff"); | |
| 295 | await res.text(); | |
| 296 | ||
| 297 | const again = await fetch(url, { | |
| 298 | headers: { "if-none-match": `"${blobOid}"` }, | |
| 299 | }); | |
| 300 | expect(again.status).toBe(304); | |
| 301 | expect(await again.text()).toBe(""); | |
| 302 | expect(again.headers.get("cache-control")).toContain("immutable"); | |
| 303 | }); | |
| 304 | ||
| 305 | test("raw at a branch name revalidates against the blob oid", async () => { | |
| 306 | const blobOid = await revParse("main:a.txt"); | |
| 307 | const res = await fetch(`${base}/mel/proj/raw/main/a.txt`); | |
| 308 | expect(res.headers.get("cache-control")).toBe("private, max-age=0"); | |
| 309 | expect(res.headers.get("etag")).toBe(`"${blobOid}"`); | |
| 310 | await res.text(); | |
| 311 | const again = await fetch(`${base}/mel/proj/raw/main/a.txt`, { | |
| 312 | headers: { "if-none-match": `"${blobOid}"` }, | |
| 313 | }); | |
| 314 | expect(again.status).toBe(304); | |
| 315 | expect(await again.text()).toBe(""); | |
| 316 | const other = await fetch(`${base}/mel/proj/raw/main/a.txt`, { | |
| 317 | headers: { "if-none-match": '"0000"' }, | |
| 318 | }); | |
| 319 | expect(other.status).toBe(200); | |
| 320 | await other.text(); | |
| 321 | }); | |
| 322 | ||
| 323 | test("raw answers 304 to a weak tag, a list holding the tag, and *", () => { | |
| 324 | const oid = "b".repeat(40); | |
| 325 | const blob = { | |
| 326 | kind: "text" as const, | |
| 327 | body: new TextEncoder().encode("x"), | |
| 328 | size: 1, | |
| 329 | oid, | |
| 330 | }; | |
| 331 | const ask = (inm: string) => | |
| 332 | rawResponse(blob, { | |
| 333 | oid, | |
| 334 | immutable: true, | |
| 335 | public: true, | |
| 336 | request: new Request("http://x/", { | |
| 337 | headers: { "if-none-match": inm }, | |
| 338 | }), | |
| 339 | }).status; | |
| 340 | expect(ask(`W/"${oid}"`)).toBe(304); | |
| 341 | expect(ask(`"zzz", "${oid}"`)).toBe(304); | |
| 342 | expect(ask("*")).toBe(304); | |
| 343 | expect(ask('"zzz"')).toBe(200); | |
| 344 | }); | |
| 345 | ||
| 346 | test("raw of a repository that is not public is never cached as public", () => { | |
| 347 | const blob = { | |
| 348 | kind: "text" as const, | |
| 349 | body: new TextEncoder().encode("x"), | |
| 350 | size: 1, | |
| 351 | oid: "a".repeat(40), | |
| 352 | }; | |
| 353 | const res = rawResponse(blob, { | |
| 354 | oid: blob.oid, | |
| 355 | immutable: true, | |
| 356 | public: false, | |
| 357 | }); | |
| 358 | expect(res.headers.get("cache-control")).toBe( | |
| 359 | "private, max-age=31536000, immutable", | |
| 360 | ); | |
| 361 | expect(res.headers.get("content-security-policy")).toContain("sandbox"); | |
| 362 | }); | |
| 363 | ||
| 364 | test("raw of a private repository at a full commit id is private, over HTTP", async () => { | |
| 365 | await fetch(`${base}/new`, { | |
| 366 | method: "POST", | |
| 367 | headers: { cookie: owner }, | |
| 368 | body: form({ name: "hush", visibility: "private" }), | |
| 369 | redirect: "manual", | |
| 370 | }); | |
| 371 | const dir = join(work, "hush"); | |
| 372 | await mkdir(dir, { recursive: true }); | |
| 373 | await git(["init", "-q", "-b", "main", "."], dir); | |
| 374 | await writeFile(join(dir, "s.txt"), "secret\n"); | |
| 375 | await git(["add", "."], dir); | |
| 376 | await git(["commit", "-q", "-m", "s"], dir); | |
| 377 | await git( | |
| 378 | [ | |
| 379 | "push", | |
| 380 | "-q", | |
| 381 | `http://mel:web-ui-password@localhost:${servers.port}/mel/hush.git`, | |
| 382 | "main", | |
| 383 | ], | |
| 384 | dir, | |
| 385 | ); | |
| 386 | const proc = Bun.spawn(["git", "rev-parse", "HEAD"], { | |
| 387 | cwd: dir, | |
| 388 | env: gitEnv, | |
| 389 | stdout: "pipe", | |
| 390 | }); | |
| 391 | const commit = (await new Response(proc.stdout).text()).trim(); | |
| 392 | const url = `${base}/mel/hush/raw/${commit}/s.txt`; | |
| 393 | const res = await fetch(url, { headers: { cookie: owner } }); | |
| 394 | expect(res.status).toBe(200); | |
| 395 | expect(res.headers.get("cache-control")).toBe( | |
| 396 | "private, max-age=31536000, immutable", | |
| 397 | ); | |
| 398 | expect(res.headers.get("cache-control")).not.toContain("public"); | |
| 399 | await res.text(); | |
| 400 | const etag = res.headers.get("etag")!; | |
| 401 | const again = await fetch(url, { | |
| 402 | headers: { cookie: owner, "if-none-match": etag }, | |
| 403 | }); | |
| 404 | expect(again.status).toBe(304); | |
| 405 | expect(again.headers.get("cache-control")).not.toContain("public"); | |
| 406 | expect((await fetch(url)).status).toBe(404); | |
| 407 | }); | |
| 408 | ||
| 271 | 409 | test("raw serves bytes as text with nosniff, and conflates not-found", async () => { |
| 272 | 410 | const res = await fetch(`${base}/mel/proj/raw/main/a.txt`); |
| 273 | 411 | expect(res.status).toBe(200); |
new above
offline
Declare this constant above rawBlob, with the module constants, rather than after the function that uses it.