Public: anyone can read this; the crew can change it.
The code-page worker pool is sized from the machine, and the cold 500/s bench says what it bought #9674
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/9671-the-code-page-worker-pool-is-sized-from → main 2 commits fork 2 threads · 2 unresolved
4 files
changed +127
−3
bench/results/cold-forge-500.md +37
| @@ -68,3 +68,40 @@ Each needs this same bench as its done-test. | ||
| 68 | 68 | - **No code under `src/` changes.** |
| 69 | 69 | - **The kept-pages-across-a-swap idea (#9631, pull #9632) stays dropped.** |
| 70 | 70 | - **The run on the deployed edge is not repeated here.** That is `bench/load.ts` against the deployment (docs/features/load-test.md). |
| 71 | ||
| 72 | ## After the pool (#9671) | |
| 73 | ||
| 74 | **Sizing the code-page worker pool from the machine did not cut refusals beyond noise. The default stays 2.** | |
| 75 | ||
| 76 | `src/web/code_rows_pool.ts` now reads `FORGE_CODE_ROWS_WORKERS`: a number (capped at 8), or `auto` for the machine's available parallelism within [2, 8]. Unset, it is 2, as before. That let both sides run from the same tree with no code change between them. | |
| 77 | ||
| 78 | - **Same command:** `bun bench/code-pages-at-rate.ts --rate 500 --seconds 20 --warm 0 --crawl 3000`. | |
| 79 | - **Same copy:** this clone's HEAD `b4137c8c`, 5,304 files. | |
| 80 | - **Machine:** 80 cores, shared with the load generator, so `auto` is 8. | |
| 81 | - **Order of runs:** 2, 8, 8, 2, then 8, 2. | |
| 82 | ||
| 83 | | | served | refused (429) | served/s | p50 | p95 | | |
| 84 | |---|---|---|---|---|---| | |
| 85 | | 2 workers, run 1 | 7,338 | 2,661 | 367 | 59 ms | 451 ms | | |
| 86 | | 2 workers, run 2 | 7,054 | 2,945 | 353 | 122 ms | 595 ms | | |
| 87 | | 2 workers, run 3 | 7,856 | 2,143 | 393 | — | 458 ms | | |
| 88 | | 8 workers, run 1 | 8,080 | 1,919 | 404 | 26 ms | 414 ms | | |
| 89 | | 8 workers, run 2 | 8,081 | 1,918 | 404 | 88 ms | 382 ms | | |
| 90 | | 8 workers, run 3 | 7,900 | 2,099 | 395 | — | 399 ms | | |
| 91 | ||
| 92 | All refusals were 429. There were no 503s. | |
| 93 | ||
| 94 | Served/refused per second for the first pair (2 workers, then 8 workers, run 1): | |
| 95 | ||
| 96 | - 2 workers, run 1: `233/266 55/445 258/242 330/170 221/279 236/264 226/274 273/227 367/133 405/95 349/151 500/0 385/115 500/0 500/0 499/0 501/0 500/0 499/0 501/0` | |
| 97 | - 8 workers, run 1: `202/297 209/291 280/220 266/234 293/207 302/198 285/215 376/124 489/11 453/47 425/75 500/0 500/0 500/0 500/0 500/0 500/0 500/0 500/0 500/0` | |
| 98 | ||
| 99 | ### What this says | |
| 100 | ||
| 101 | - **Refused:** 8 workers refused 1,918–2,099, a mean of about 1,980. 2 workers refused 2,143–2,945 here, and 1,802–2,463 in the "After" runs above. So the 8-worker range sits inside the 2-worker range. In the last back-to-back pair the gap was 44 (2,099 against 2,143). | |
| 102 | - **p95:** 382–414 ms with 8 workers, against 451–595 ms with 2 here. That is a little lower, but 376–498 ms above was also 2 workers. | |
| 103 | - **The ceiling did not move.** Served/s still levels off at about 400. Seconds 1–4 still refuse 220–300 a second with either size. If two workers at about 5 ms a page had been the limit, eight would have lifted the served rate well past 400. They did not. | |
| 104 | ||
| 105 | **So the pool is not the limit.** That fits the finding above: the refusals come from the strangers' in-flight test (`behindAll`, with the oldest waiting past 250 ms) while git permits are held. The other lever named above, counting queueing at the worker pool rather than at the strangers' gate, has not been tried. | |
| 106 | ||
| 107 | **Memory.** Each worker is about 8 MB of heap when idle (measured with four `code_rows_worker.ts` workers, Bun 1.3.14). The cap of 8 bounds that at about 64 MB idle, more while pages are in flight. |
docs/features/load-test.md +2
| @@ -39,6 +39,8 @@ Only 200 (and 206) answers are samples. A redirect, 401 or 404 is counted by sta | ||
| 39 | 39 | |
| 40 | 40 | Anonymous tree, blob, raw and media pages take a permit from `strangerCodeGate` (`src/http/stranger_code.ts`) before they do any work: at most half the plumbing pool (`gitGate`), `FORGE_STRANGER_CODE_CONCURRENCY` overrides, with at most 256 waiting before the next stranger gets 503 + `Retry-After`. Holding the permit, a page yields one turn of the event loop before it renders, so a queue of strangers' cached pages cannot run back to back while a person's request goes unread. A signed-in request never touches the gate. The admission limit in `strangers.ts` still answers 429 above its per-minute budget. |
| 41 | 41 | |
| 42 | Cold code pages build their rows and encode their bytes on a pool of workers (`src/web/code_rows_pool.ts`): two by default; `FORGE_CODE_ROWS_WORKERS` sets a number (capped at 8, about 8 MB of heap each) or `auto` for the machine's available parallelism within [2, 8]. Eight did not cut cold 500/s refusals beyond noise (bench/results/cold-forge-500.md, "After the pool"). | |
| 43 | ||
| 42 | 44 | `bench/signed-in-under-load.ts` boots a throwaway forge in-process with the Shield `open` (so the whole flood reaches the code pages), seeds a repository and a session, and runs `bench/load.ts` against it with `FORGE_LOAD_COOKIE` set: anonymous code pages open-loop at `--rate`, and a signed-in `/` and repository home every 50 ms beside them. It prints both streams' p50/p95, the event loop's lag and the `gitGate` waits. On a sandbox at 500/s for 15 s: signed-in p95 91 ms with the yield, 22.5 s without it. |
| 43 | 45 | |
| 44 | 46 | `bench/signed-in-under-load.ts` boots a throwaway forge in-process, floods it with anonymous code pages open-loop and times a signed-in `/` and repository home at the same time: |
src/web/code_rows_pool.ts +42 −3
| @@ -1,14 +1,53 @@ | ||
| 1 | 1 | /** |
| 2 | 2 | * Cold code pages highlight and lay out their rows on worker threads, so the |
| 3 | 3 | * main loop keeps serving while a crawl asks for pages nobody has kept (#9630). |
| 4 | * Two workers, started on first use; a job that fails or runs past | |
| 5 | * `JOB_MS` answers undefined and the caller does the work inline. | |
| 4 | * `poolSize()` workers (two; `FORGE_CODE_ROWS_WORKERS` sets a number or `auto`), | |
| 5 | * started on first use; a job that fails or runs past `JOB_MS` answers | |
| 6 | * undefined and the caller does the work inline. | |
| 6 | 7 | */ |
| 8 | import { availableParallelism } from "node:os"; | |
| 7 | 9 | import type { Encoding } from "../http/compress.ts"; |
| 8 | 10 | import type { CodeRows } from "./code_rows.ts"; |
| 9 | 11 | import type { Job } from "./code_rows_worker.ts"; |
| 10 | 12 | |
| 11 | const WORKERS = 2; | |
| 13 | /** The floor: one worker busy on a long file must not stall every cold page. */ | |
| 14 | const MIN_WORKERS = 2; | |
| 15 | /** | |
| 16 | * The cap: each worker is its own JS heap, about 8 MB idle (measured, #9671) | |
| 17 | * and more while it holds a page's bytes to encode. Eight bounds that near | |
| 18 | * 100 MB on any machine and leaves cores for git, which a cold page also waits on. | |
| 19 | */ | |
The comment says near 100 MB, while the bench says about 64 MB idle (8 workers x 8 MB). Say which figure is meant (idle or in flight) so the two agree. |
||
| 20 | const MAX_WORKERS = 8; | |
| 21 | ||
| 22 | /** | |
| 23 | * The default, kept at the floor: sizing from the machine (8 here, on 80 cores) | |
| 24 | * did not cut a cold 500/s crawl's refusals beyond noise, and served/s still | |
| 25 | * levelled near 400 (bench/results/cold-forge-500.md, "After the pool", #9671). | |
| 26 | */ | |
| 27 | const DEFAULT_WORKERS = MIN_WORKERS; | |
| 28 | ||
| 29 | /** The machine's share: its available parallelism clamped to [MIN_WORKERS, MAX_WORKERS]. */ | |
| 30 | export const machineWorkers = (cores: number = availableParallelism()): number => | |
| 31 | Math.min(MAX_WORKERS, Math.max(MIN_WORKERS, Math.floor(cores))); | |
| 32 | ||
| 33 | /** | |
| 34 | * Workers in the pool, read like the git pool's `FORGE_GIT_CONCURRENCY`: | |
| 35 | * `FORGE_CODE_ROWS_WORKERS` as a positive number (floored, capped at | |
| 36 | * MAX_WORKERS), or `auto` for `machineWorkers()`; unset or anything else is | |
| 37 | * DEFAULT_WORKERS. | |
| 38 | */ | |
| 39 | export function poolSize( | |
| 40 | override: string | undefined = process.env.FORGE_CODE_ROWS_WORKERS, | |
| 41 | cores: number = availableParallelism(), | |
| 42 | ): number { | |
| 43 | const text = override?.trim() ?? ""; | |
| 44 | if (text === "auto") return machineWorkers(cores); | |
| 45 | const asked = Number(text); | |
| 46 | if (text !== "" && Number.isFinite(asked) && asked >= 1) return Math.min(MAX_WORKERS, Math.floor(asked)); | |
| 47 | return DEFAULT_WORKERS; | |
| 48 | } | |
| 49 | ||
| 50 | const WORKERS = poolSize(); | |
| 12 | 51 | const JOB_MS = 5_000; |
| 13 | 52 | /** Jobs in flight past which the caller renders inline instead of queueing. */ |
| 14 | 53 | const MAX_PENDING = 64; |
test/code-rows-pool-size.test.ts added +46
| @@ -0,0 +1,46 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { availableParallelism } from "node:os"; | |
| 3 | import { machineWorkers, poolSize } from "../src/web/code_rows_pool.ts"; | |
| 4 | ||
| 5 | test("unset, the pool is two workers whatever the machine", () => { | |
| 6 | expect(poolSize(undefined, 1)).toBe(2); | |
| 7 | expect(poolSize(undefined, 80)).toBe(2); | |
| 8 | }); | |
| 9 | ||
| 10 | test("auto sizes from the machine: at least 2, at most 8", () => { | |
| 11 | expect(machineWorkers(1)).toBe(2); | |
| 12 | expect(machineWorkers(4)).toBe(4); | |
| 13 | expect(machineWorkers(80)).toBe(8); | |
| 14 | expect(poolSize("auto", 1)).toBe(2); | |
| 15 | expect(poolSize("auto", 6)).toBe(6); | |
| 16 | expect(poolSize(" auto ", 80)).toBe(8); | |
| 17 | expect(poolSize("auto")).toBe(Math.min(8, Math.max(2, availableParallelism()))); | |
| 18 | }); | |
| 19 | ||
| 20 | test("a number overrides, floored and capped at 8", () => { | |
| 21 | expect(poolSize("3", 80)).toBe(3); | |
| 22 | expect(poolSize("1", 80)).toBe(1); | |
| 23 | expect(poolSize("5.9", 2)).toBe(5); | |
| 24 | expect(poolSize("64", 2)).toBe(8); | |
| 25 | }); | |
| 26 | ||
| 27 | test("a bad value falls back to the default of two", () => { | |
| 28 | for (const bad of ["", " ", "0", "-2", "0.5", "two", "NaN", "Infinity", "AUTO"]) { | |
| 29 | expect(poolSize(bad, 40)).toBe(2); | |
| 30 | } | |
| 31 | }); | |
| 32 | ||
| 33 | test("FORGE_CODE_ROWS_WORKERS is what poolSize reads", () => { | |
| 34 | const was = process.env.FORGE_CODE_ROWS_WORKERS; | |
| 35 | try { | |
| 36 | process.env.FORGE_CODE_ROWS_WORKERS = "3"; | |
| 37 | expect(poolSize()).toBe(3); | |
| 38 | process.env.FORGE_CODE_ROWS_WORKERS = "junk"; | |
| 39 | expect(poolSize()).toBe(2); | |
| 40 | delete process.env.FORGE_CODE_ROWS_WORKERS; | |
| 41 | expect(poolSize()).toBe(2); | |
| 42 | } finally { | |
| 43 | if (was === undefined) delete process.env.FORGE_CODE_ROWS_WORKERS; | |
| 44 | else process.env.FORGE_CODE_ROWS_WORKERS = was; | |
| 45 | } | |
| 46 | }); |
This line says no code under src/ changes, but this pull changes src/web/code_rows_pool.ts. Reword it to say it was true of the earlier pull.