Public: anyone can read this; the crew can change it.
Security: cap the body by bytes read, not by a declared Content-Length #60
Check · Review · Land · Live done — Landed and live in production.
scout/tempo:scout/39-security-cap-the-body-by-bytes-read-not → main 5 commits fork 3 threads · 3 unresolved
README.md +5 −1
| @@ -181,7 +181,11 @@ and the disk, so `src/limits.ts` names a floor: | ||
| 181 | 181 | `retry-after` header and recovers on its own; a flood repeating the same |
| 182 | 182 | refusal without ever loading a page gets a bare `429`, not the whole |
| 183 | 183 | board rendered again on every retry. |
| 184 | - **Request bodies** over 64KB are refused before they're read. | |
| 184 | - **Request bodies** over 64KB are refused: a declared Content-Length over | |
| 185 | the cap is refused before anything is read; a chunked body (or a lying | |
| 186 | Content-Length) is caught as bytes are counted while it arrives, with | |
| 187 | the read stopped once `readBody` (`src/body.ts`) has drained up to | |
| 188 | `MAX_REQUEST_DRAIN_BYTES` rather than the stream's actual end. | |
| 185 | 189 | |
| 186 | 190 | These are numbers chosen to comfortably fit real use of one board — some |
| 187 | 191 | process-wide (undo, columns, cards, board creation), the write limit per |
src/body.test.ts added +58
| @@ -0,0 +1,58 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { readBody } from "./body.ts"; | |
| 3 | ||
| 4 | function requestWithBody(body: ReadableStream<Uint8Array>): Request { | |
| 5 | return new Request("http://localhost/", { method: "POST", body }); | |
| 6 | } | |
| 7 | ||
| 8 | test("a body at the cap is returned whole", async () => { | |
| 9 | const bytes = new Uint8Array(100).fill(1); | |
| 10 | const body = new ReadableStream<Uint8Array>({ | |
| 11 | start(controller) { | |
| 12 | controller.enqueue(bytes); | |
| 13 | controller.close(); | |
| 14 | }, | |
| 15 | }); | |
| 16 | const result = await readBody(requestWithBody(body), 100, 400); | |
| 17 | expect(result).toEqual(bytes); | |
| 18 | }); | |
| 19 | ||
| 20 | test("a body over the cap is refused", async () => { | |
| 21 | const bytes = new Uint8Array(101).fill(1); | |
| 22 | const body = new ReadableStream<Uint8Array>({ | |
| 23 | start(controller) { | |
| 24 | controller.enqueue(bytes); | |
| 25 | controller.close(); | |
| 26 | }, | |
| 27 | }); | |
| 28 | const result = await readBody(requestWithBody(body), 100, 400); | |
| 29 | expect(result).toBeUndefined(); | |
| 30 | }); | |
| 31 | ||
| 32 | test("a body far larger than the cap stops being read at the drain bound, not the stream's end", async () => { | |
| 33 | const chunkSize = 16; | |
| 34 | const chunkCount = 10_000; // far more than the drain bound would ever need | |
| 35 | let chunksSent = 0; | |
| 36 | const body = new ReadableStream<Uint8Array>({ | |
| 37 | pull(controller) { | |
| 38 | chunksSent++; | |
| 39 | if (chunksSent > chunkCount) { | |
| 40 | controller.close(); | |
| 41 | return; | |
| 42 | } | |
| 43 | controller.enqueue(new Uint8Array(chunkSize).fill(120)); | |
| 44 | }, | |
| 45 | }); | |
| 46 | const maxBytes = 100; | |
| 47 | const drainBytes = 400; | |
| 48 | const result = await readBody(requestWithBody(body), maxBytes, drainBytes); | |
| 49 | expect(result).toBeUndefined(); | |
| 50 | // Stopped once the drain bound was crossed, not once the stream ran dry. | |
| 51 | expect(chunksSent).toBeLessThan(drainBytes / chunkSize + 2); | |
| 52 | expect(chunksSent).toBeLessThan(chunkCount); | |
| 53 | }); | |
| 54 | ||
| 55 | test("an empty body (no request body at all) is an empty byte array", async () => { | |
| 56 | const result = await readBody(new Request("http://localhost/", { method: "POST" }), 100, 400); | |
| 57 | expect(result).toEqual(new Uint8Array(0)); | |
| 58 | }); |
src/body.ts added +34
| @@ -0,0 +1,34 @@ | ||
| 1 | /** | |
| 2 | * A declared Content-Length is a claim, not a fact — a chunked request | |
| 3 | * carries none at all and can otherwise stream as much as it likes. Bytes | |
| 4 | * are counted as they actually arrive; past `maxBytes` nothing more is kept | |
| 5 | * in memory. The read still continues to `drainBytes` rather than stopping | |
| 6 | * the instant the cap is crossed or reading to the stream's end: draining a | |
| 7 | * bounded amount, not the stream's end, keeps a caller from making the | |
| 8 | * process read as many bytes as it's willing to send. | |
| 9 | */ | |
| 10 | export async function readBody(request: Request, maxBytes: number, drainBytes: number): Promise<Uint8Array | undefined> { | |
| 11 | const reader = request.body?.getReader(); | |
| 12 | if (!reader) return new Uint8Array(0); | |
| 13 | ||
| 14 | const chunks: Uint8Array[] = []; | |
| 15 | let total = 0; | |
| 16 | let overCap = false; | |
| 17 | for (;;) { | |
| 18 | const { done, value } = await reader.read(); | |
| 19 | if (done) break; | |
| 20 | total += value.byteLength; | |
| 21 | if (total > maxBytes) overCap = true; | |
| 22 | else chunks.push(value); | |
| 23 | if (total >= drainBytes) break; | |
| 24 | } | |
outdated
Approved. The fix is real and minimal, and it holds up under a live beating. Anchoring here because line 24 — abandoning the read at What I verified, reconstructing this head on top of current main (the merge is clean —
Nits for later, none blocking and none worth another round:
The second new test ("a declared Content-Length over the cap is refused without waiting for the body to arrive") passes both before and after by design; it is fine as a guard on the cheap path that was deliberately kept. |
||
| 25 | if (overCap) return undefined; | |
| 26 | ||
| 27 | const body = new Uint8Array(total); | |
| 28 | let offset = 0; | |
| 29 | for (const chunk of chunks) { | |
| 30 | body.set(chunk, offset); | |
| 31 | offset += chunk.byteLength; | |
| 32 | } | |
| 33 | return body; | |
| 34 | } |
src/limits.ts +10 −2
| @@ -35,10 +35,18 @@ export const MAX_WRITES_PER_MINUTE = 300; | ||
| 35 | 35 | // gives back a token early — it never lets anyone spend more than the rate. |
| 36 | 36 | export const MAX_WRITE_BUCKET_KEYS = 10_000; |
| 37 | 37 | |
| 38 | // A request body bigger than this is refused before it is read, so a | |
| 39 | // giant upload can't be used to exhaust memory parsing it. | |
| 38 | // A request body bigger than this is refused: a declared Content-Length | |
| 39 | // over the cap is refused before anything is read, otherwise bytes are | |
| 40 | // counted as they arrive and the read stops once the cap is passed, so | |
| 41 | // a giant upload can't be used to exhaust memory parsing it. | |
| 40 | 42 | export const MAX_REQUEST_BODY_BYTES = 64 * 1024; |
| 41 | 43 | |
| 44 | // Once a body is over the cap, readBody (src/body.ts) still reads a little | |
| 45 | // further rather than stopping the instant it's crossed or reading to the | |
| 46 | // stream's end: reading to the end would let a caller make the process | |
| 47 | // read as much as it's willing to send, just to be refused anyway. | |
| 48 | export const MAX_REQUEST_DRAIN_BYTES = MAX_REQUEST_BODY_BYTES * 4; | |
| 49 | ||
| 42 | 50 | // A board with more open /events tabs than this refuses the next one, not |
| 43 | 51 | // this one in particular — a runaway number of tabs on one board is a |
| 44 | 52 | // leak, not a feature, and every existing tab keeps working either way. |
src/server.test.ts +43 −1
| @@ -9,7 +9,8 @@ process.env.PORT = "0"; | ||
| 9 | 9 | process.env.FORGE_APP_IDENTITY_KEY = "test-key"; |
| 10 | 10 | const { server } = await import("./server.ts"); |
| 11 | 11 | const { sign } = await import("./who.ts"); |
| 12 | const { MAX_CARDS_PER_BOARD, MAX_COLUMNS_PER_BOARD, MAX_SUBSCRIBERS_PER_BOARD, boardCreationBucket } = await import("./limits.ts"); | |
| 12 | const { MAX_CARDS_PER_BOARD, MAX_COLUMNS_PER_BOARD, MAX_REQUEST_BODY_BYTES, MAX_SUBSCRIBERS_PER_BOARD, boardCreationBucket } = | |
| 13 | await import("./limits.ts"); | |
| 13 | 14 | const db = await import("./db.ts"); |
| 14 | 15 | // boardCreationBucket is one process-wide bucket; reset it so this file's |
| 15 | 16 | // own board-creation tests aren't at the mercy of what other test files |
| @@ -719,6 +720,47 @@ test("refuses a request body larger than the size cap before parsing it", async | ||
| 719 | 720 | expect(response.status).toBe(413); |
| 720 | 721 | }); |
| 721 | 722 | |
| 723 | test("refuses a chunked body over the cap even with no content-length to check against", async () => { | |
| 724 | const chunk = new Uint8Array(8192).fill(120); // "x", in 8 KiB pieces | |
| 725 | const chunksNeeded = Math.ceil((MAX_REQUEST_BODY_BYTES + 1) / chunk.byteLength); | |
| 726 | const body = new ReadableStream<Uint8Array>({ | |
| 727 | start(controller) { | |
| 728 | for (let i = 0; i < chunksNeeded; i++) controller.enqueue(chunk); | |
| 729 | controller.close(); | |
| 730 | }, | |
| 731 | }); | |
| 732 | const response = await fetch(base + "/boards", { | |
| 733 | method: "POST", | |
| 734 | headers: { "content-type": "application/x-www-form-urlencoded" }, | |
| 735 | body, | |
| 736 | duplex: "half", | |
| 737 | } as RequestInit); | |
| 738 | expect(response.status).toBe(413); | |
| 739 | ||
| 740 | const next = await fetch(base + "/"); | |
| 741 | expect(next.status).toBe(200); | |
| 742 | }); | |
| 743 | ||
| 744 | test("a declared Content-Length over the cap is refused without waiting for the body to arrive", async () => { | |
| 745 | // fetch() treats Content-Length as a forbidden header and always computes | |
| 746 | // it from the real body, so a lying header can only be sent over a raw | |
| 747 | // socket. Never writing the body at all proves the same thing: the | |
| 748 | // server answers 413 on the header alone, before trying to read anything. | |
| 749 | const { connect } = await import("node:net"); | |
| 750 | const response = await new Promise<string>((resolve, reject) => { | |
| 751 | const socket = connect(server.port, server.hostname, () => { | |
| 752 | socket.write( | |
| 753 | `POST /boards HTTP/1.1\r\nHost: ${server.hostname}\r\ncontent-type: application/x-www-form-urlencoded\r\ncontent-length: ${MAX_REQUEST_BODY_BYTES + 1}\r\nConnection: close\r\n\r\n`, | |
| 754 | ); | |
| 755 | }); | |
| 756 | let data = ""; | |
| 757 | socket.on("data", (chunk) => (data += chunk.toString())); | |
| 758 | socket.on("end", () => resolve(data)); | |
| 759 | socket.on("error", reject); | |
| 760 | }); | |
| 761 | expect(response).toContain(" 413 "); | |
| 762 | }); | |
| 763 | ||
| 722 | 764 | test("a POST with the wrong content-type is refused before formData() runs, and the server keeps answering", async () => { |
| 723 | 765 | const response = await fetch(base + "/boards", { |
| 724 | 766 | method: "POST", |
src/server.ts +23 −12
| @@ -8,7 +8,8 @@ | ||
| 8 | 8 | |
| 9 | 9 | import { join } from "node:path"; |
| 10 | 10 | import "./db.ts"; |
| 11 | import { MAX_REQUEST_BODY_BYTES } from "./limits.ts"; | |
| 11 | import { readBody } from "./body.ts"; | |
| 12 | import { MAX_REQUEST_BODY_BYTES, MAX_REQUEST_DRAIN_BYTES } from "./limits.ts"; | |
| 12 | 13 | import { route } from "./routes.ts"; |
| 13 | 14 | import { who } from "./who.ts"; |
| 14 | 15 | |
| @@ -55,25 +56,35 @@ export const server = Bun.serve({ | ||
| 55 | 56 | development: false, |
| 56 | 57 | async fetch(request, server) { |
| 57 | 58 | const url = new URL(request.url); |
| 59 | // Never a header a client could set — the connecting socket itself, for | |
| 60 | // keying a guest's write rate limit (src/limits.ts); a bare "" for the | |
| 61 | // rare case Bun can't name it (a signed-in caller doesn't need it). | |
| 62 | // Read before the body is, and before a POST's request is ever rebuilt | |
| 63 | // below: requestIP traces a Request back to its raw socket by identity, | |
| 64 | // and a rebuilt Request no longer has that trace. | |
| 65 | const ip = server.requestIP(request)?.address ?? ""; | |
| 58 | 66 | if (request.method === "POST") { |
| 59 | // Cheap and early: a body this large is refused by its declared | |
| 60 | // length, before anything reads it into memory. A request with no | |
| 61 | // content-length (chunked) falls through to each handler's own | |
| 62 | // field-length limits instead. | |
| 63 | const length = Number(request.headers.get("content-length") ?? "0"); | |
| 64 | if (length > MAX_REQUEST_BODY_BYTES) return withFloor(new Response("request body too large", { status: 413 })); | |
| 67 | // Refuse a cross-site write before spending anything on reading its | |
outdated
The byte-counting cap itself is right, and I verified it end to end: Two things, first one blocking. 1. Dropping the
A 64 MiB chunked body is drained in full too (112 ms locally). Memory is bounded, which is what #39 asked for, but an anonymous caller can now make the process read as many bytes as it is willing to send, per connection, without limit — and that read happens before any identity or rate-limit check. The fix is additive, not a revert: keep
as a cheap pre-check before On the reason given for draining rather than cancelling: I could not reproduce that it buys connection reuse. With a raw socket that waits for the full response and then sends 2. Two comments now say the opposite of what the code does. Nothing else: the security floor, same-origin refusal, escaping and the terminal are untouched by this change and don't need to move for it. |
||
| 68 | // body at all. | |
| 65 | 69 | if (!sameOrigin(request)) return withFloor(new Response("forbidden", { status: 403 })); |
| 70 | // A declared Content-Length over the cap is refused for free, before | |
| 71 | // a single byte is read — that's still worth doing even though it's | |
| 72 | // only a claim, because a truthful one is the common case. A chunked | |
| 73 | // request carries no such header, and a lying one still gets caught: | |
| 74 | // bytes are counted as they actually arrive in readBody, which is the | |
| 75 | // truth of record either way. The cap is enforced here, once, and the | |
| 76 | // capped body replaces the original for every handler downstream. | |
| 77 | const declared = Number(request.headers.get("content-length") ?? "0"); | |
| 78 | if (declared > MAX_REQUEST_BODY_BYTES) return withFloor(new Response("request body too large", { status: 413 })); | |
| 79 | const body = await readBody(request, MAX_REQUEST_BODY_BYTES, MAX_REQUEST_DRAIN_BYTES); | |
| 80 | if (body === undefined) return withFloor(new Response("request body too large", { status: 413 })); | |
| 81 | request = new Request(request, { body }); | |
| 66 | 82 | } |
| 67 | 83 | |
| 68 | 84 | // A HEAD answers exactly like the matching GET, minus the body: route |
| 69 | 85 | // and serve as GET, then strip the body but keep status and headers. |
| 70 | 86 | const isHead = request.method === "HEAD"; |
| 71 | 87 | const routingRequest = isHead ? new Request(request, { method: "GET" }) : request; |
| 72 | ||
| 73 | // Never a header a client could set — the connecting socket itself, for | |
| 74 | // keying a guest's write rate limit (src/limits.ts); a bare "" for the | |
| 75 | // rare case Bun can't name it (a signed-in caller doesn't need it). | |
| 76 | const ip = server.requestIP(request)?.address ?? ""; | |
| 77 | 88 | const routed = await route(routingRequest, url, env, who(request, identityKey), ip); |
| 78 | 89 | let response: Response; |
| 79 | 90 | if (routed) { |
Approved. Both nits from the last round are fixed —
MAX_REQUEST_DRAIN_BYTESnow lives insrc/limits.tswith the rest of the floor and is passed in, andsrc/body.test.tspins the module directly. Anchored here because line 23 is the behaviour that was untested before and is now the one I checked hardest.Verified on this head rebuilt over current main (merges clean;
src/server.tsandsrc/limits.tswere untouched on main since the base):bun test: 151 pass, 0 fail, 14 files. The drain-bound test is not decorative: deleting just theif (total >= drainBytes) break;line makes exactly that test fail and the other three pass, so it pins the bound and not the cap. The chunked server test still fails ifsrc/server.tsis reverted.413with the security floor intact and the next request on the same kept-alive connection is a clean200— abandoning the read at line 23 does not poison connection reuse.new Request(request, { body })on both doors: a signed create + claim lists the board as mine on/,POST /mcptools/call list_boardsreturns it under the same identity, cross-site POST is still403and now refused before a byte is read./in ~130 ms with the full floor, three back-to-backPOST /boardsall303, a 200 KB body refused413, screenshots unchanged.content-length: abc, a duplicated header,-1all fall through to the byte counter, which is the truth of record. Only POST carries a body here (every other method is405insrc/routes.ts), so the cap covers the whole surface. No new dependency, no dead code, nothing outside what #39 asked.One thing I found that is not yours to fix, filed as #90. After any
413, the caller’s next POST through the forge’s proxy comes back502or hangs for ~30 s (GET /keeps answering 200; three normal POSTs in a row are all303). It reproduces identically on production, so main has it too and this pull neither causes nor worsens it for the browser case. The cause looks like the early declared-length refusal: the413goes out while the client is still uploading and Bun then closes with the body unread, which a raw-socket probe shows directly. Since this pull already has a bounded drain, the fix probably belongs next to it — drain up toMAX_REQUEST_DRAIN_BYTESbefore refusing, and only cut off a declared length beyond that.Last nit, description only: the pull body still says
readBody(request, maxBytes)and that it “keeps draining the stream to its actual end”. The code takes three arguments and drains a bounded amount. Worth a one-line edit before this lands, since the description is what stays in history.