Public: anyone can read this; the crew can change it.

Security: a request body has a cap but no deadline — a trickling client holds a handler open #87

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

mason/bazaar:mason/54-security-a-request-body-has-a-cap-but-no → main 1 commit fork

  • Merged. mason/54-security-a-request-body-has-a-cap-but-no was merged into main as 03ab5b0.

Review

Sign in to review this change.
5 files changed +227 −51
AGENTS.md +3 −1
@@ -36,7 +36,9 @@ preview at its own address, deployed from your branch as you push.
3636 escaped; every response carries the security floor in `src/server.ts`
3737 (keep it, extend it); every request body is read through the cap in
3838 `src/http.ts` — bytes counted as they arrive, `413` past it, never a
39 `Content-Length` believed; a POST that is not same-origin is refused; secrets
39 `Content-Length` believed, and one wall-clock budget over the whole read so a
40 trickling client is `408` rather than a held-open handler; a POST that is not
41 same-origin is refused; secrets
4042 come from the environment, never the repository; no `eval`, no shell
4143 built from user text. Who is asking comes from the forge's signed headers,
4244 verified in `src/who.ts` and read only as `ctx.who` — never a password of
README.md +7 −1
@@ -462,7 +462,13 @@ Every request body is read through `readBody` in `src/http.ts`, which counts
462462 the bytes as they arrive and stops at 64 KB. A declared `Content-Length` is a
463463 claim and a chunked request has none, so neither is believed; past the cap the
464464 answer is `413` and the rest is drained (up to eight times the cap, then cut
465 off) so the connection stays usable. That happens in one place — `src/server.ts`
465 off) so the connection stays usable. The cap says how much; a five-second
466 wall-clock budget on the same read says how long, because a client that sends
467 one byte a second is inside the cap for hours. One clock covers the whole body,
468 the drain after a refusal included: when it runs out the read is let go and the
469 answer is `408` (`413` if the body was already past the cap), with
470 `Connection: close`, since the rest of that body is still on the wire and is
471 nobody's next request. That happens in one place — `src/server.ts`
466472 — for every route: a form is parsed from the capped bytes, and the one route
467473 that wants its body whole, Stripe's webhook, which verifies a signature over it,
468474 is capped before it is verified and reads the result as `ctx.body`. A POST that
src/http.ts +96 −42
@@ -40,56 +40,110 @@ export function handOff(location: string, status: 302 | 303 = 303): Response {
4040
4141 // A request body is read up to a cap and no further, counting the bytes as
4242 // they arrive. A declared Content-Length is a claim: a chunked request has
43 // none, and a lie is free. Undefined means "too large" — refuse it with a 413.
44 // Past the cap the rest is drained and dropped so the connection stays usable
45 // for the next request; past eight times the cap it is cut off instead.
43 // none, and a lie is free. Past the cap the rest is drained and dropped so the
44 // connection stays usable for the next request; past eight times the cap it is
45 // cut off instead.
4646 export const BODY_MAX = 64 * 1024;
4747
48 export async function readBody(request: Request, max = BODY_MAX): Promise<Uint8Array | undefined> {
49 const reader = request.body?.getReader();
50 if (reader === undefined) return new Uint8Array();
51 const parts: Uint8Array[] = [];
52 let size = 0;
53 for (;;) {
54 const { done, value } = await reader.read();
55 if (done) break;
56 size += value.byteLength;
57 if (size > max) {
58 for (;;) {
59 const next = await reader.read();
60 if (next.done) return undefined;
61 size += next.value.byteLength;
62 if (size > max * 8) {
63 await reader.cancel();
64 return undefined;
65 }
66 }
67 }
68 parts.push(value);
69 }
70 const body = new Uint8Array(size);
71 let at = 0;
72 for (const part of parts) {
73 body.set(part, at);
74 at += part.byteLength;
75 }
76 return body;
48 // The cap says how much; this says how long. A client that sends one byte a
49 // second is inside the cap for hours, so the whole body — the drain after a
50 // refusal included — gets one wall-clock budget, and the read that runs out of
51 // it is the end of the request. Every honest body of 64 KB arrives in far less
52 // than this, and this is well inside the server's idle timeout, so the answer
53 // a slow client gets is ours and not a dropped connection.
54 export const BODY_MS = 5_000;
55
56 /**
57 * Why a body was refused before a handler ever saw it, as the status to send:
58 * `413` too many bytes, `408` too much time. Anything else a read returns is
59 * the body itself.
60 */
61 export type Refused = 413 | 408;
62
63 export const refusal = (status: Refused): string =>
64 status === 413 ? "that body was too large" : "that body took too long to arrive";
65
66 export const isRefused = (read: unknown): read is Refused => read === 413 || read === 408;
67
68 /** Out of time: the one value the clock below adds to a read's answers. */
69 const LATE = Symbol("late");
70
71 /**
72 * One timer for one body, raced against every read of it. Started when the
73 * first byte is waited for and cleared however the read ends, so a finished
74 * request leaves nothing behind.
75 */
76 function clock(ms: number) {
77 let timer: ReturnType<typeof setTimeout> | undefined;
78 const late = new Promise<typeof LATE>((resolve) => {
79 timer = setTimeout(() => resolve(LATE), ms);
80 timer.unref?.();
81 });
82 return {
83 read: (reader: ReadableStreamDefaultReader<Uint8Array>) => Promise.race([reader.read(), late]),
84 stop: () => clearTimeout(timer),
85 };
7786 }
7887
79 /** The text of a capped body; `undefined` when it was past the cap. */
80 export async function readText(request: Request, max = BODY_MAX): Promise<string | undefined> {
81 const body = await readBody(request, max);
82 return body === undefined ? undefined : new TextDecoder().decode(body);
88 /** The bytes of a body, or the status that refuses it. */
89 export async function readBody(request: Request, max = BODY_MAX, ms = BODY_MS): Promise<Uint8Array | Refused> {
90 const reader = request.body?.getReader();
91 if (reader === undefined) return new Uint8Array();
92 const within = clock(ms);
93 try {
94 const parts: Uint8Array[] = [];
95 let size = 0;
96 for (;;) {
97 const read = await within.read(reader);
98 // Out of time. Let go of the stream — this request is over either way.
99 if (read === LATE) return cut(reader, 408);
100 if (read.done) break;
101 size += read.value.byteLength;
102 if (size > max) {
103 for (;;) {
104 const next = await within.read(reader);
105 // Still too large; the clock only decides how long we keep draining.
106 if (next === LATE) return cut(reader, 413);
107 if (next.done) return 413;
108 size += next.value.byteLength;
109 if (size > max * 8) return cut(reader, 413);
110 }
111 }
112 parts.push(read.value);
113 }
114 const body = new Uint8Array(size);
115 let at = 0;
116 for (const part of parts) {
117 body.set(part, at);
118 at += part.byteLength;
119 }
120 return body;
121 } finally {
122 within.stop();
123 }
124 }
125
126 /** Drop the rest of a body and refuse it. The cancel is not waited on: a stream
127 * that stalled is exactly the one that could stall here too. */
128 function cut(reader: ReadableStreamDefaultReader<Uint8Array>, status: Refused): Refused {
129 reader.cancel().catch(() => {});
130 return status;
131 }
132
133 /** The text of a capped body, or the status that refuses it. */
134 export async function readText(request: Request, max = BODY_MAX, ms = BODY_MS): Promise<string | Refused> {
135 const body = await readBody(request, max, ms);
136 return isRefused(body) ? body : new TextDecoder().decode(body);
83137 }
84138
85139 /**
86 * The form fields of a capped body, decoded the way the browser sent them:
87 * `undefined` when it was past the cap, and a throw when the bytes are not
88 * the form the content type promised.
140 * The form fields of a capped body, decoded the way the browser sent them: the
141 * status that refuses it when it was too large or too slow, and a throw when
142 * the bytes are not the form the content type promised.
89143 */
90 export async function readForm(request: Request, max = BODY_MAX): Promise<FormData | undefined> {
91 const body = await readBody(request, max);
92 if (body === undefined) return undefined;
144 export async function readForm(request: Request, max = BODY_MAX, ms = BODY_MS): Promise<FormData | Refused> {
145 const body = await readBody(request, max, ms);
146 if (isRefused(body)) return body;
93147 return new Request(request.url, {
94148 method: "POST",
95149 headers: { "content-type": request.headers.get("content-type") ?? "" },
src/server.test.ts +98
@@ -1,4 +1,5 @@
11 import { expect, test } from "bun:test";
2 import { connect } from "node:net";
23 import { join } from "node:path";
34 import { freshData } from "./fresh.ts";
45
@@ -12,6 +13,7 @@ delete process.env.STRIPE_SECRET_KEY;
1213 const { stamp } = await import("./who.ts");
1314 const { server } = await import("./server.ts");
1415 const { db } = await import("./db.ts");
16 const { isRefused, readBody, readForm, refusal } = await import("./http.ts");
1517 const limit = await import("./ratelimit.ts");
1618 const live = await import("./live.ts");
1719 const giftcards = await import("./giftcards.ts");
@@ -328,6 +330,102 @@ test("a chunked body is counted as it arrives, not believed by its header", asyn
328330 expect(response.status).toBe(413);
329331 });
330332
333 // --- the deadline on a body --------------------------------------------------
334
335 /** A body that sends what it is given and then never another byte, nor an end. */
336 function stalling(...first: Uint8Array[]): ReadableStream<Uint8Array> {
337 return new ReadableStream({
338 start(controller) {
339 for (const chunk of first) controller.enqueue(chunk);
340 },
341 });
342 }
343
344 const posting = (body: BodyInit): Request =>
345 // @ts-expect-error a streaming body is half duplex; Bun wants it said out loud.
346 new Request("http://bazaar.test/cart/add", { method: "POST", body, duplex: "half" });
347
348 test("a body that stalls is refused when its budget runs out, not waited on", async () => {
349 const started = Date.now();
350 const read = await readBody(posting(stalling(new Uint8Array(8).fill(0x61))), 64 * 1024, 100);
351 expect(read).toBe(408);
352 expect(isRefused(read)).toBe(true);
353 expect(refusal(408)).toContain("too long");
354 expect(Date.now() - started).toBeLessThan(2_000);
355 });
356
357 test("a body with no first byte at all is refused the same way", async () => {
358 expect(await readBody(posting(stalling()), 64 * 1024, 100)).toBe(408);
359 });
360
361 test("the drain after a refusal has the same deadline: too large, and still not waited on", async () => {
362 const started = Date.now();
363 // Past the cap on the first chunk, so the read is in the drain loop — and
364 // then nothing more arrives, and no end ever comes.
365 const read = await readBody(posting(stalling(new Uint8Array(4 * 1024).fill(0x61))), 1024, 100);
366 expect(read).toBe(413);
367 expect(Date.now() - started).toBeLessThan(2_000);
368 });
369
370 test("a body inside its budget is read whole, and a form is still a form", async () => {
371 const body = new URLSearchParams({ variant: "1", quantity: "2" }).toString();
372 const request = new Request("http://bazaar.test/cart/add", {
373 method: "POST",
374 body,
375 headers: { "content-type": "application/x-www-form-urlencoded" },
376 });
377 const form = await readForm(request, 64 * 1024, 5_000);
378 expect(isRefused(form)).toBe(false);
379 expect((form as FormData).get("quantity")).toBe("2");
380 });
381
382 /**
383 * A request written by hand on a socket of its own — `fetch` cannot say "and
384 * now nothing, for as long as you will have me", and a client that walks off
385 * mid-body is the whole point. Resolves with the response as it came off the
386 * wire, or with what had arrived by the time we gave up waiting.
387 */
388 function trickle(head: string, waitMs = 12_000): Promise<{ raw: string; ms: number }> {
389 return new Promise((resolve) => {
390 const started = Date.now();
391 const socket = connect(server.port, server.hostname, () => socket.write(head));
392 socket.on("error", () => {});
393 socket.on("data", (chunk: Buffer) => {
394 socket.destroy();
395 resolve({ raw: chunk.toString(), ms: Date.now() - started });
396 });
397 setTimeout(() => {
398 socket.destroy();
399 resolve({ raw: "", ms: Date.now() - started });
400 }, waitMs).unref?.();
401 });
402 }
403
404 test("a trickling client gets an answer from the server, not a held-open handler", async () => {
405 // Half a form body, a Content-Length that promises the rest, and then
406 // silence: the handler is holding a read that will never finish.
407 const { raw, ms } = await trickle(
408 [
409 "POST /cart/add HTTP/1.1",
410 "Host: bazaar.test",
411 "Sec-Fetch-Site: same-origin",
412 "Content-Type: application/x-www-form-urlencoded",
413 "Content-Length: 40",
414 "",
415 "variant=1&",
416 ].join("\r\n"),
417 );
418 expect(raw).toContain("HTTP/1.1 408");
419 expect(raw).toContain("that body took too long to arrive");
420 // The floor is on a refusal too, and the connection is over with the request:
421 // the rest of that body is still on the wire and is nobody's next request.
422 expect(raw).toContain("X-Content-Type-Options: nosniff");
423 expect(raw.toLowerCase()).toContain("connection: close");
424 // Inside the budget, and long before the server's own idle timeout.
425 expect(ms).toBeGreaterThan(1_000);
426 expect(ms).toBeLessThan(10_000);
427 }, 20_000);
428
331429 test("a body under the cap is still a form", async () => {
332430 const shopper = new Shopper();
333431 const product = await shopper.page("/p/ethiopia-guji");
src/server.ts +23 −7
@@ -30,7 +30,7 @@ import { peerOf, trustedHops } from "./peer.ts";
3030 import { seed } from "./catalog.ts";
3131 import { seed as seedDiscounts } from "./discounts.ts";
3232 import { seed as seedTax } from "./tax.ts";
33 import { HAND_OFF, readForm, readText, text } from "./http.ts";
33 import { HAND_OFF, isRefused, readForm, readText, refusal, text, type Refused } from "./http.ts";
3434 import { returnTo, whoIs } from "./who.ts";
3535 import type { Who } from "./who.ts";
3636 import type { Ctx } from "./context.ts";
@@ -89,6 +89,17 @@ const sameOrigin = (request: Request): boolean => {
8989 return origin === null || origin === publicOrigin;
9090 };
9191
92 /**
93 * The answer to a body we stopped reading. It says close, and means it: bytes
94 * of that body are still on the wire, and the next request on this connection
95 * would read them as its own. A refusal ends the connection with the request.
96 */
97 function refuse(status: Refused): Response {
98 const response = text(refusal(status), status);
99 response.headers.set("connection", "close");
100 return response;
101 }
102
92103 /** Every response leaves through here, so none of them can miss the floor. */
93104 function finish(response: Response, cookies: string[]): Response {
94105 const headers = new Headers(response.headers);
@@ -157,24 +168,29 @@ export const server = Bun.serve({
157168 }
158169
159170 // Every body this app reads goes through src/http.ts, which counts the
160 // bytes as they arrive and stops at the cap — including the one route that
161 // reads its own body, Stripe's webhook, which gets it as ctx.body.
171 // bytes as they arrive and stops at the cap, and holds the whole read to a
172 // wall-clock budget so a trickling client cannot keep a handler open —
173 // including the one route that reads its own body, Stripe's webhook, which
174 // gets it as ctx.body. A refusal comes back as the status to send.
162175 let form: FormData | null = null;
163176 let body: string | null = null;
164177 if (method === "POST" && found.route.raw) {
165 body = (await readText(request)) ?? null;
166 if (body === null) return finish(text("that body was too large", 413), outgoing);
178 const read = await readText(request);
179 if (isRefused(read)) return finish(refuse(read), outgoing);
180 body = read;
167181 }
168182 if (method === "POST" && !found.route.raw) {
169183 const type = request.headers.get("content-type") ?? "";
170184 if (!type.startsWith("application/x-www-form-urlencoded") && !type.startsWith("multipart/form-data"))
171185 return finish(text("expected a form", 415), outgoing);
186 let read: FormData | Refused;
172187 try {
173 form = (await readForm(request)) ?? null;
188 read = await readForm(request);
174189 } catch {
175190 return finish(text("bad form", 400), outgoing);
176191 }
177 if (form === null) return finish(text("that body was too large", 413), outgoing);
192 if (isRefused(read)) return finish(refuse(read), outgoing);
193 form = read;
178194 if (found.route.csrf !== false && !checkCsrf(session, form.get("csrf")))
179195 return finish(text("that form was not from here — go back and try again", 403), outgoing);
180196 }

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs