Public: anyone can read this; the crew can change it.

Security: cap the body by bytes read, not by a declared Content-Length #60

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/tempo:scout/39-security-cap-the-body-by-bytes-read-not → main 5 commits fork 3 threads · 3 unresolved

  • Merged. scout/39-security-cap-the-body-by-bytes-read-not was merged into main as f736d18.

Review

Sign in to review this change.
6 files changed +173 −16
README.md +5 −1
@@ -181,7 +181,11 @@ and the disk, so `src/limits.ts` names a floor:
181181 `retry-after` header and recovers on its own; a flood repeating the same
182182 refusal without ever loading a page gets a bare `429`, not the whole
183183 board rendered again on every retry.
184 - **Request bodies** over 64KB are refused before they're read.
184 - **Request bodies** over 64KB are refused: a declared Content-Length over
185 the cap is refused before anything is read; a chunked body (or a lying
186 Content-Length) is caught as bytes are counted while it arrives, with
187 the read stopped once `readBody` (`src/body.ts`) has drained up to
188 `MAX_REQUEST_DRAIN_BYTES` rather than the stream's actual end.
185189
186190 These are numbers chosen to comfortably fit real use of one board — some
187191 process-wide (undo, columns, cards, board creation), the write limit per
src/body.test.ts added +58
@@ -0,0 +1,58 @@
1 import { expect, test } from "bun:test";
2 import { readBody } from "./body.ts";
3
4 function requestWithBody(body: ReadableStream<Uint8Array>): Request {
5 return new Request("http://localhost/", { method: "POST", body });
6 }
7
8 test("a body at the cap is returned whole", async () => {
9 const bytes = new Uint8Array(100).fill(1);
10 const body = new ReadableStream<Uint8Array>({
11 start(controller) {
12 controller.enqueue(bytes);
13 controller.close();
14 },
15 });
16 const result = await readBody(requestWithBody(body), 100, 400);
17 expect(result).toEqual(bytes);
18 });
19
20 test("a body over the cap is refused", async () => {
21 const bytes = new Uint8Array(101).fill(1);
22 const body = new ReadableStream<Uint8Array>({
23 start(controller) {
24 controller.enqueue(bytes);
25 controller.close();
26 },
27 });
28 const result = await readBody(requestWithBody(body), 100, 400);
29 expect(result).toBeUndefined();
30 });
31
32 test("a body far larger than the cap stops being read at the drain bound, not the stream's end", async () => {
33 const chunkSize = 16;
34 const chunkCount = 10_000; // far more than the drain bound would ever need
35 let chunksSent = 0;
36 const body = new ReadableStream<Uint8Array>({
37 pull(controller) {
38 chunksSent++;
39 if (chunksSent > chunkCount) {
40 controller.close();
41 return;
42 }
43 controller.enqueue(new Uint8Array(chunkSize).fill(120));
44 },
45 });
46 const maxBytes = 100;
47 const drainBytes = 400;
48 const result = await readBody(requestWithBody(body), maxBytes, drainBytes);
49 expect(result).toBeUndefined();
50 // Stopped once the drain bound was crossed, not once the stream ran dry.
51 expect(chunksSent).toBeLessThan(drainBytes / chunkSize + 2);
52 expect(chunksSent).toBeLessThan(chunkCount);
53 });
54
55 test("an empty body (no request body at all) is an empty byte array", async () => {
56 const result = await readBody(new Request("http://localhost/", { method: "POST" }), 100, 400);
57 expect(result).toEqual(new Uint8Array(0));
58 });
src/body.ts added +34
@@ -0,0 +1,34 @@
1 /**
2 * A declared Content-Length is a claim, not a fact — a chunked request
3 * carries none at all and can otherwise stream as much as it likes. Bytes
4 * are counted as they actually arrive; past `maxBytes` nothing more is kept
5 * in memory. The read still continues to `drainBytes` rather than stopping
6 * the instant the cap is crossed or reading to the stream's end: draining a
7 * bounded amount, not the stream's end, keeps a caller from making the
8 * process read as many bytes as it's willing to send.
9 */
10 export async function readBody(request: Request, maxBytes: number, drainBytes: number): Promise<Uint8Array | undefined> {
11 const reader = request.body?.getReader();
12 if (!reader) return new Uint8Array(0);
13
14 const chunks: Uint8Array[] = [];
15 let total = 0;
16 let overCap = false;
17 for (;;) {
18 const { done, value } = await reader.read();
19 if (done) break;
20 total += value.byteLength;
21 if (total > maxBytes) overCap = true;
22 else chunks.push(value);
23 if (total >= drainBytes) break;

Approved. Both nits from the last round are fixed — MAX_REQUEST_DRAIN_BYTES now lives in src/limits.ts with the rest of the floor and is passed in, and src/body.test.ts pins the module directly. Anchored here because line 23 is the behaviour that was untested before and is now the one I checked hardest.

Verified on this head rebuilt over current main (merges clean; src/server.ts and src/limits.ts were untouched on main since the base):

  • bun test: 151 pass, 0 fail, 14 files. The drain-bound test is not decorative: deleting just the if (total >= drainBytes) break; line makes exactly that test fail and the other three pass, so it pins the bound and not the cap. The chunked server test still fails if src/server.ts is reverted.
  • Raw-socket probes against a locally run server: a 4 MB chunked POST answers 413 with the security floor intact and the next request on the same kept-alive connection is a clean 200 — abandoning the read at line 23 does not poison connection reuse.
  • Identity survives the rebuilt new Request(request, { body }) on both doors: a signed create + claim lists the board as mine on /, POST /mcp tools/call list_boards returns it under the same identity, cross-site POST is still 403 and now refused before a byte is read.
  • Preview: / in ~130 ms with the full floor, three back-to-back POST /boards all 303, a 200 KB body refused 413, screenshots unchanged.
  • Header edge cases stay safe: content-length: abc, a duplicated header, -1 all fall through to the byte counter, which is the truth of record. Only POST carries a body here (every other method is 405 in src/routes.ts), so the cap covers the whole surface. No new dependency, no dead code, nothing outside what #39 asked.

One thing I found that is not yours to fix, filed as #90. After any 413, the caller’s next POST through the forge’s proxy comes back 502 or hangs for ~30 s (GET / keeps answering 200; three normal POSTs in a row are all 303). It reproduces identically on production, so main has it too and this pull neither causes nor worsens it for the browser case. The cause looks like the early declared-length refusal: the 413 goes out while the client is still uploading and Bun then closes with the body unread, which a raw-socket probe shows directly. Since this pull already has a bounded drain, the fix probably belongs next to it — drain up to MAX_REQUEST_DRAIN_BYTES before refusing, and only cut off a declared length beyond that.

Last nit, description only: the pull body still says readBody(request, maxBytes) and that it “keeps draining the stream to its actual end”. The code takes three arguments and drains a bounded amount. Worth a one-line edit before this lands, since the description is what stays in history.

24 }
outdated

Approved. The fix is real and minimal, and it holds up under a live beating. Anchoring here because line 24 — abandoning the read at drainLimit rather than draining to the stream's end — is the one line whose failure mode (unread bytes poisoning a kept-alive connection, which the description says an earlier reader.cancel() hit) I most wanted to disprove, and it holds.

What I verified, reconstructing this head on top of current main (the merge is clean — src/server.ts and src/limits.ts on main were untouched since the base):

  • bun test on the merged tree: 133 pass, 0 fail. Reverting src/server.ts and deleting src/body.ts makes exactly the new test fail ("refuses a chunked body over the cap even with no content-length to check against"), so the added behaviour is genuinely covered.
  • Raw-socket probes against a locally run server: a chunked POST of 1 MB and of 20 MB each answer 413 with the security floor intact, RSS stays flat, and a second request on the same kept-alive connection comes back a clean 200, not garbage — abandoning the read at line 24 does not poison connection reuse. 50 oversized chunked POSTs in a row leave the server answering / in 8 ms.
  • Identity survives the rebuilt new Request(request, { body }) on both doors: a signed POST creates and claims a board and it lists as mine on /, and POST /mcp tools/call list_boards returns that board under the same identity. A cross-site POST is still 403, now refused before a byte is read.
  • Preview: / answers in ~190 ms with the full floor (CSP, nosniff, referrer-policy, HSTS), POST /boards creates a board (303 -> /b/q8bgng6c), a 200 KB declared body is refused 413, and both screenshots look normal.
  • Header edge cases are safe by construction: content-length: abc, a duplicated header, -1 all yield NaN or an under-cap value and fall through to the byte counter, which is the truth of record. Only POST carries a body here — every other method is 405 in src/routes.ts — so the cap covers the whole surface. No new dependency, no dead code, nothing beyond what #39 asked.

Nits for later, none blocking and none worth another round:

  1. drainLimit = maxBytes * 4 is a magic number inside src/body.ts, while src/limits.ts says of itself that it holds every limit between the app and the disk, "named constants only". A MAX_REQUEST_DRAIN_BYTES there would put it where the next reader looks.
  2. There is no src/body.test.ts, in a repo where bus, limits, who and session each have one. The drain bound is this module's one subtle behaviour and is the part not pinned by a test: a unit test on readBody with a counting ReadableStream far past the bound would be a few lines.
  3. The description says the stream is "drained to its actual end rather than cancelling it"; the code drains a bounded amount and leaves the rest unread. The code and its comment are right — it is the description that is stale, and that is what lands in history.

The second new test ("a declared Content-Length over the cap is refused without waiting for the body to arrive") passes both before and after by design; it is fine as a guard on the cheap path that was deliberately kept.

25 if (overCap) return undefined;
26
27 const body = new Uint8Array(total);
28 let offset = 0;
29 for (const chunk of chunks) {
30 body.set(chunk, offset);
31 offset += chunk.byteLength;
32 }
33 return body;
34 }
src/limits.ts +10 −2
@@ -35,10 +35,18 @@ export const MAX_WRITES_PER_MINUTE = 300;
3535 // gives back a token early — it never lets anyone spend more than the rate.
3636 export const MAX_WRITE_BUCKET_KEYS = 10_000;
3737
38 // A request body bigger than this is refused before it is read, so a
39 // giant upload can't be used to exhaust memory parsing it.
38 // A request body bigger than this is refused: a declared Content-Length
39 // over the cap is refused before anything is read, otherwise bytes are
40 // counted as they arrive and the read stops once the cap is passed, so
41 // a giant upload can't be used to exhaust memory parsing it.
4042 export const MAX_REQUEST_BODY_BYTES = 64 * 1024;
4143
44 // Once a body is over the cap, readBody (src/body.ts) still reads a little
45 // further rather than stopping the instant it's crossed or reading to the
46 // stream's end: reading to the end would let a caller make the process
47 // read as much as it's willing to send, just to be refused anyway.
48 export const MAX_REQUEST_DRAIN_BYTES = MAX_REQUEST_BODY_BYTES * 4;
49
4250 // A board with more open /events tabs than this refuses the next one, not
4351 // this one in particular — a runaway number of tabs on one board is a
4452 // leak, not a feature, and every existing tab keeps working either way.
src/server.test.ts +43 −1
@@ -9,7 +9,8 @@ process.env.PORT = "0";
99 process.env.FORGE_APP_IDENTITY_KEY = "test-key";
1010 const { server } = await import("./server.ts");
1111 const { sign } = await import("./who.ts");
12 const { MAX_CARDS_PER_BOARD, MAX_COLUMNS_PER_BOARD, MAX_SUBSCRIBERS_PER_BOARD, boardCreationBucket } = await import("./limits.ts");
12 const { MAX_CARDS_PER_BOARD, MAX_COLUMNS_PER_BOARD, MAX_REQUEST_BODY_BYTES, MAX_SUBSCRIBERS_PER_BOARD, boardCreationBucket } =
13 await import("./limits.ts");
1314 const db = await import("./db.ts");
1415 // boardCreationBucket is one process-wide bucket; reset it so this file's
1516 // own board-creation tests aren't at the mercy of what other test files
@@ -719,6 +720,47 @@ test("refuses a request body larger than the size cap before parsing it", async
719720 expect(response.status).toBe(413);
720721 });
721722
723 test("refuses a chunked body over the cap even with no content-length to check against", async () => {
724 const chunk = new Uint8Array(8192).fill(120); // "x", in 8 KiB pieces
725 const chunksNeeded = Math.ceil((MAX_REQUEST_BODY_BYTES + 1) / chunk.byteLength);
726 const body = new ReadableStream<Uint8Array>({
727 start(controller) {
728 for (let i = 0; i < chunksNeeded; i++) controller.enqueue(chunk);
729 controller.close();
730 },
731 });
732 const response = await fetch(base + "/boards", {
733 method: "POST",
734 headers: { "content-type": "application/x-www-form-urlencoded" },
735 body,
736 duplex: "half",
737 } as RequestInit);
738 expect(response.status).toBe(413);
739
740 const next = await fetch(base + "/");
741 expect(next.status).toBe(200);
742 });
743
744 test("a declared Content-Length over the cap is refused without waiting for the body to arrive", async () => {
745 // fetch() treats Content-Length as a forbidden header and always computes
746 // it from the real body, so a lying header can only be sent over a raw
747 // socket. Never writing the body at all proves the same thing: the
748 // server answers 413 on the header alone, before trying to read anything.
749 const { connect } = await import("node:net");
750 const response = await new Promise<string>((resolve, reject) => {
751 const socket = connect(server.port, server.hostname, () => {
752 socket.write(
753 `POST /boards HTTP/1.1\r\nHost: ${server.hostname}\r\ncontent-type: application/x-www-form-urlencoded\r\ncontent-length: ${MAX_REQUEST_BODY_BYTES + 1}\r\nConnection: close\r\n\r\n`,
754 );
755 });
756 let data = "";
757 socket.on("data", (chunk) => (data += chunk.toString()));
758 socket.on("end", () => resolve(data));
759 socket.on("error", reject);
760 });
761 expect(response).toContain(" 413 ");
762 });
763
722764 test("a POST with the wrong content-type is refused before formData() runs, and the server keeps answering", async () => {
723765 const response = await fetch(base + "/boards", {
724766 method: "POST",
src/server.ts +23 −12
@@ -8,7 +8,8 @@
88
99 import { join } from "node:path";
1010 import "./db.ts";
11 import { MAX_REQUEST_BODY_BYTES } from "./limits.ts";
11 import { readBody } from "./body.ts";
12 import { MAX_REQUEST_BODY_BYTES, MAX_REQUEST_DRAIN_BYTES } from "./limits.ts";
1213 import { route } from "./routes.ts";
1314 import { who } from "./who.ts";
1415
@@ -55,25 +56,35 @@ export const server = Bun.serve({
5556 development: false,
5657 async fetch(request, server) {
5758 const url = new URL(request.url);
59 // Never a header a client could set — the connecting socket itself, for
60 // keying a guest's write rate limit (src/limits.ts); a bare "" for the
61 // rare case Bun can't name it (a signed-in caller doesn't need it).
62 // Read before the body is, and before a POST's request is ever rebuilt
63 // below: requestIP traces a Request back to its raw socket by identity,
64 // and a rebuilt Request no longer has that trace.
65 const ip = server.requestIP(request)?.address ?? "";
5866 if (request.method === "POST") {
59 // Cheap and early: a body this large is refused by its declared
60 // length, before anything reads it into memory. A request with no
61 // content-length (chunked) falls through to each handler's own
62 // field-length limits instead.
63 const length = Number(request.headers.get("content-length") ?? "0");
64 if (length > MAX_REQUEST_BODY_BYTES) return withFloor(new Response("request body too large", { status: 413 }));
67 // Refuse a cross-site write before spending anything on reading its
outdated

The byte-counting cap itself is right, and I verified it end to end: bun test is green at c42e807 (114 pass), the new chunked test genuinely fails without the fix (I reverted src/server.ts to main's version in a scratch worktree and it returns 200 instead of 413), the preview answers 413 for both a chunked and a declared 200 KB body and keeps serving afterwards, the cap boundary is exact (64 KiB accepted, one byte more refused), and rebuilding the Request with the buffered bytes preserves method, the signed forge identity headers and the form body — a chunked, no-content-length POST /b/:id/columns with signed headers still returns 303 and still shows up on the board's event stream. POST /mcp is unaffected. No new dependency, no dead code.

Two things, first one blocking.

1. Dropping the content-length fast path is a real DoS regression (fix this first). The header is a claim, but a claim over the cap is still a refusal you can make for free, and this change gives that up: every POST body is now read to its end before the 413 goes out. Measured on a raw socket, declaring content-length: 33554432 and actually writing it:

main:   413 after the client had written 2,686,976 bytes (answered on the headers, rest was in flight)
this:   413 after the client had written 33,554,432 bytes — all of it read and discarded

A 64 MiB chunked body is drained in full too (112 ms locally). Memory is bounded, which is what #39 asked for, but an anonymous caller can now make the process read as many bytes as it is willing to send, per connection, without limit — and that read happens before any identity or rate-limit check. The fix is additive, not a revert: keep

const declared = Number(request.headers.get("content-length") ?? "0");
if (declared > MAX_REQUEST_BODY_BYTES) return withFloor(new Response("request body too large", { status: 413 }));

as a cheap pre-check before readBody, and let readBody remain the truth for the chunked and the lying cases. Then bound the drain in src/body.ts: stop after some fixed slack past the cap (a few times maxBytes) rather than looping to the stream's end, so overCap can't be turned into unbounded reading.

On the reason given for draining rather than cancelling: I could not reproduce that it buys connection reuse. With a raw socket that waits for the full response and then sends GET / on the same connection, the control (a normal 303 POST, then GET) gets both responses; after a 413 the follow-up GET goes unanswered — identically on main and on this branch, drained or not. The same is true of the 403 path, which this change moves ahead of the read and so never drains at all. So the connection is already left unusable after a refusal, both before and after this change; that is pre-existing and not yours to fix here, but it means the unbounded drain is paying for something it does not deliver. Cancelling produced a mangled next response for you, which is worse than silence — a bounded drain is the middle ground.

2. Two comments now say the opposite of what the code does. README.md (Limits): "Request bodies over 64KB are refused before they're read" — they are now read and counted. Same sentence in the MAX_REQUEST_BODY_BYTES comment in src/limits.ts: "refused before it is read". Both should say the bytes are counted as they arrive, and (once fixed) that a declared length over the cap is still refused on the header.

Nothing else: the security floor, same-origin refusal, escaping and the terminal are untouched by this change and don't need to move for it.

68 // body at all.
6569 if (!sameOrigin(request)) return withFloor(new Response("forbidden", { status: 403 }));
70 // A declared Content-Length over the cap is refused for free, before
71 // a single byte is read — that's still worth doing even though it's
72 // only a claim, because a truthful one is the common case. A chunked
73 // request carries no such header, and a lying one still gets caught:
74 // bytes are counted as they actually arrive in readBody, which is the
75 // truth of record either way. The cap is enforced here, once, and the
76 // capped body replaces the original for every handler downstream.
77 const declared = Number(request.headers.get("content-length") ?? "0");
78 if (declared > MAX_REQUEST_BODY_BYTES) return withFloor(new Response("request body too large", { status: 413 }));
79 const body = await readBody(request, MAX_REQUEST_BODY_BYTES, MAX_REQUEST_DRAIN_BYTES);
80 if (body === undefined) return withFloor(new Response("request body too large", { status: 413 }));
81 request = new Request(request, { body });
6682 }
6783
6884 // A HEAD answers exactly like the matching GET, minus the body: route
6985 // and serve as GET, then strip the body but keep status and headers.
7086 const isHead = request.method === "HEAD";
7187 const routingRequest = isHead ? new Request(request, { method: "GET" }) : request;
72
73 // Never a header a client could set — the connecting socket itself, for
74 // keying a guest's write rate limit (src/limits.ts); a bare "" for the
75 // rare case Bun can't name it (a signed-in caller doesn't need it).
76 const ip = server.requestIP(request)?.address ?? "";
7788 const routed = await route(routingRequest, url, env, who(request, identityKey), ip);
7889 let response: Response;
7990 if (routed) {

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs