Public: anyone can read this; the crew can change it.

The forge holds 500 anonymous code pages a second, with people still fast #9405

Closed opened this has it

Landed in pull #9499 · 2 runs · run

opened this

The owner, 2026-11-03: "i really want a focus on speed, efficiency and scalability."

A flood of about 31 anonymous code pages a second took 91% of the forge's answering time. Those pages averaged about 22 s, home reached 7.9 s, and the foundry stopped answering, while the node sat at a load of about 1.7 with most of its memory free. The limit shipped in 0.7.384 sheds this traffic; it doesn't make the pages cheap.

Done when:

5 earlier comments
commented

Experiments, 2026-11-03, on a throwaway copy of 0.7.385 (8 CPUs, shallow 300-commit copy of this repository, no crew, memory disk). Not production, and far less data than production, so read these as best case.

Capacity, code pages on random pinned commits:

AskedServed/sp50p95A person's home, p95
10/s1039 ms70 ms29 ms
30/s2824 ms49 ms26 ms
100/s895.2 s10.2 s7.3 s
300/s8814.5 s27.4 s26.6 s

It saturates near 90 pages a second, and a person waits behind the strangers once it does.

Open questions these raise:

  • The copy handles 30/s at 49 ms p95, while production averaged 22 s at about 31/s. Something production has makes it far slower; find what before fixing pages.
  • The ceiling in 0.7.384 didn't fire at 100 anonymous pages/s here: no 429s, 3.9 s p50, a person's p95 4.4 s. It keys on event-loop lateness, and here the wait isn't on the loop. It should also read in-flight deep pages.

Profile, 30 s at 30/s, share of samples: #getInternalBuffer 13.9%, writeSync 8.8%, spawn 7.8%, prepare 6.9%, toLocaleString 6.1%, callers (src/kernel/slow_sql.ts:31) 4.1%. Candidates: logging synchronously on the main thread, preparing statements afresh, Intl formatting per call, and the slow-SQL diagnostic itself. Not confirmed as causes; profile production's shape before acting.

Chaos:

  • SIGKILL: back in 1.0 s, 20 of 20 acknowledged writes present, quick_check ok. The 516 load errors are likely the load generator sharing the killed container's network, not an outage.
  • Freeze 10 s: healthy 0.1 s after resuming.
  • 59 git children killed under load: no failed requests, p95 61 ms.
  • Disk full: stayed up and a write still returned 201. That likely means SQLite had room already, so write failure on a full disk is still untested.
commented

Parked by the crew: the check stayed red through 2 automatic tries (pull #9461 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

commented

Reviewed pull #9499 at its head: ticked Commit-pinned pages are served without a git walk after the first read.. left open: A repeatable load test, run from a machine that isn't the owner's Mac against a copy of the forge, reports the rate and latency it reached, and its result is kept for each release. — The tool exists and is tested, but no run has been made and no result is kept per release (#9504).; A profile of a tree, blob, raw and media page names where each one's time goes under load, and is attached here before any fix is chosen. — No profile attached (#9503).; Rendering a code page can't hold the main thread: signed-in pages stay under 200 ms at p95 while strangers read at full rate. — Rendering is not moved off the main thread and nothing is measured.; At 500 anonymous code pages a second, nothing is refused and p95 is under 100 ms, with and without the edge cache. — Not measured at 500/s.; A release fails its check if the tested rate falls more than 20% from the last one. — The verdict exists in bench/load.ts --compare but is not wired into the release check (#9504)..

commented

Left open after pull #9499 landed: 5 unticked items remain (A repeatable load test, run from a machine that isn't the owner's Mac against a copy of the forge, reports the rate and latency it reached, and its result is kept for each release.; A profile of a tree, blob, raw and media page names where each one's time goes under load, and is attached here before any fix is chosen.; Rendering a code page can't hold the main thread: signed-in pages stay under 200 ms at p95 while strangers read at full rate.; …). The planner files each as its own issue and closes this one; a person may close it, tick what is done, or say what is left. Until then the crew does not pick it again.

commented

The open boxes are split. Box 1 (repeatable load test, result kept per release) and box 5 (a release fails on a drop of more than 20%) are #9504. Box 2 (profile of tree, blob, raw and media pages) is #9503. Box 3 (signed-in pages under 200 ms p95 while strangers read) is #9513, which waits for the profile. Box 4 (500/s, nothing refused, p95 under 100 ms, with and without the edge cache) is #9514, which waits for #9513 and #9504. Box 4 (commit-pinned pages served without a git walk after the first read) shipped in #9499.

commented

Correction to the box numbers above. In the issue body as written: load test → #9504; profile → #9503; main thread under 200 ms → #9513; commit-pinned pages (ticked, shipped in #9499); 500/s under 100 ms → #9514; release fails on a drop of more than 20% → #9504.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs