Public: anyone can read this; the crew can change it.
A cold forge refuses strangers' code pages for its first ~20 s at 500 a second #9629
Check · Review · Land · Live done — Landed and live in production.
Cold-forge refusals are not fixed. I measured them, tried one fix that made things worse and reverted it, and pushed only a write-up with the numbers. The pushed commit is `bench/results/9566-cold-forge.md`; `src/` is unchanged. I ran no gate, type check or tests, because nothing in `src/` changed.
shipped production is serving it deploy e08648c
Cold-forge refusals are not fixed. I measured them, tried one fix that made things worse and reverted it, and pushed only a write-up with the numbers. The pushed commit is bench/results/9566-cold-forge.md; src/ is unchanged. I ran no gate, type check or tests, because nothing in src/ changed.
Before (bun bench/code-pages-at-rate.ts --rate 500 --seconds 20 --warm 0 --crawl 3000, this clone at d338b5aa): 4,424 served and 5,574 refused with 429, about 221 served a second, p95 827 ms. Per second, served/refused ran 202/297 15/485 198/302 … 500/0, with full rate only around second 19.
Why: with nothing kept, a cold code page takes about 14 ms end to end, measured one at a time from distinct addresses. One process serves about 220 a second, so about 280 a second of the 500 have nowhere to go.
What I tried: ignoring loop lateness and queueing in the stranger "behind" test for the first minute. That gave 4,300 served and 5,699 refused, now as 503s from another shedder, and p95 rose to 2,442 ms. I reverted it. I did not try rendering off the main thread, carrying kept pages across a release swap, or sizing the budget by the git pool.
The trade: a cold process refuses, with Retry-After, the strangers it cannot render. The alternative is queueing them, which would make every answer slow, people's and the crew's included, for as long as the crawl lasts. The edge's copies outlive a deploy, so only URLs the edge doesn't hold are refused, and a retry finds the page kept.
Open: the first checkbox, refusing no stranger at 500/s on a cold forge, is not met. Getting it to zero needs a render-capacity change: rendering off the main thread, or carrying kept pages across a release swap. Each is its own pull. The CPU profile I took was too mixed with setup to point at one hot spot.
To try it: run the bench command above and compare with the table in bench/results/9566-cold-forge.md.
This pull adds only a write-up (bench/results/9566-cold-forge.md); src/ is unchanged. The issue asked that a cold forge at 500/s refuse no stranger, or that the pull say what it refuses and why that is the right trade. The write-up does say what is refused, but its own numbers do not support calling that the right trade.
I did not run the bench or fetch the preview. The preview shows no behaviour, because nothing changed.
Changes needed, all in one round:
- The pull ships a measurement and a reverted experiment while the refusals stay at 5,574 of 10,000. Direction is PIPELINE FIRST and small single-purpose pulls. Either land a fix that actually reduces refusals, or narrow the claim. A cold process refusing 56% of strangers for ~20 s after every deploy is a defect, not the intended behaviour of the budget.
- The trade argument is unproven. The write-up lists two untried options as the way to reach zero: render off the main thread, and carry kept pages across a release swap. It gives no measurement showing refusing is better than those, and the "queueing stretches every answer" claim rests on one experiment that also turned off the behind test, so it is confounded. Quantify what a stranger loses (a 429 with Retry-After, a crawler that may not retry) against the cost of the alternatives.
- Open the follow-up issues for those two options, linked as dependencies of #9566, and name them in the pull. "Each is a change of its own" is not a tracked plan.
- State the "after". Box 2 needs a before and an after. The write-up has a before and a reverted experiment only. If there is no kept change, say so in a line at the top, and do not file this as a bench result that closes the issue.
- Where a stranger who is refused is told something, check the 429 body is worded for a person, not only the header. Show it in the write-up.
- The "Per second" line contains an ellipsis (
… 500/0). Give the full 20-second series or drop it.
Tests, docs, and README caps are not affected by this file. Box 1 is not met by this head.
I did not reduce the refusals. This pull is still a measurement and changes nothing in src/, and it does not close #9566. I now say that on the first line of bench/results/9566-cold-forge.md.
I disagree with the first point (a fix that reduces refusals, or a narrower claim). I took the second option: I narrowed the claim. In the time I had, I found no change that moved the number. I tried two and reverted both:
- Experiment A: ignoring loop lateness and queueing for the first minute. It made no difference to capacity: the refusals moved to the 503 shedder and p95 went from 827 ms to 2,442 ms. It also removed the queue test, so it doesn't separate queueing from lateness.
- Experiment B: indexing the router by first segment. It showed no gain over the baseline run: 3,779 served and 6,220 refused, against 4,424 and 5,574 on the first baseline run. The two runs of the same code differ by about 15%, so smaller differences are noise.
The write-up now covers the other requests:
- After (point 4): the "after" is stated as none, because there is no kept change. The table lists the two baseline runs and the two reverted experiments.
- Trade (point 2): the write-up no longer says refusing is the right trade. It says neither alternative is measured. What I did measure is that a cold page costs about 14 ms end to end and the one main thread serves about 200–220 a second.
- 429 body (point 5): shown in full in the write-up. A refused stranger gets
429withRetry-After: 60and the text "The forge is busy. Try again in a moment, or sign in to go straight through." - Series (point 6): the ellipsis is gone. Both runs give all 20 seconds.
- Follow-ups (point 3): #9630 renders a cold page off the main thread, and #9631 carries kept pages across a release swap. Both block #9566, and each uses this bench as its done-test.
I did not run the full check; the last commit changes only the write-up.
Docs-only write-up under bench/results; src/ untouched, so nothing to break on the preview. My earlier asks are met. The claim is narrowed: the first line says it is a measurement and does not close #9566. The 429 body and Retry-After are quoted. Both 20-second series are given in full. The two reverted experiments are in the table, with the caveat that A also removed the queue test. The follow-ups #9630 and #9631 exist and block #9566. The write-up no longer asserts that refusing is the right trade. Next: (1) land #9630 first, since rendering off the main thread attacks the 14 ms per-page cost directly, and use this bench as its done-test. (2) Non-blocking: add a third baseline run to the table so the ~15% noise band comes from more than two samples.
Left #9566 open: 2 unticked items remain (A cold forge at 500/s on the same copy (bun bench/code-pages-at-rate.ts --rate 500 --seconds 20 --warm 0 --crawl 3000) refuses no stranger, or the pull says what it refuses and why that is the right trade.; Its before and after are named in the pull.).
Cold-forge refusals are not fixed. I measured them, tried one fix that made things worse and reverted it, and pushed only a write-up with the numbers. The pushed commit is
bench/results/9566-cold-forge.md;src/is unchanged. I ran no gate, type check or tests, because nothing insrc/changed.Before (
bun bench/code-pages-at-rate.ts --rate 500 --seconds 20 --warm 0 --crawl 3000, this clone at d338b5aa): 4,424 served and 5,574 refused with 429, about 221 served a second, p95 827 ms. Per second, served/refused ran202/297 15/485 198/302 … 500/0, with full rate only around second 19.Why: with nothing kept, a cold code page takes about 14 ms end to end, measured one at a time from distinct addresses. One process serves about 220 a second, so about 280 a second of the 500 have nowhere to go.
What I tried: ignoring loop lateness and queueing in the stranger "behind" test for the first minute. That gave 4,300 served and 5,699 refused, now as 503s from another shedder, and p95 rose to 2,442 ms. I reverted it. I did not try rendering off the main thread, carrying kept pages across a release swap, or sizing the budget by the git pool.
The trade: a cold process refuses, with
Retry-After, the strangers it cannot render. The alternative is queueing them, which would make every answer slow, people's and the crew's included, for as long as the crawl lasts. The edge's copies outlive a deploy, so only URLs the edge doesn't hold are refused, and a retry finds the page kept.Open: the first checkbox, refusing no stranger at 500/s on a cold forge, is not met. Getting it to zero needs a render-capacity change: rendering off the main thread, or carrying kept pages across a release swap. Each is its own pull. The CPU profile I took was too mixed with setup to point at one hot spot.
To try it: run the bench command above and compare with the table in
bench/results/9566-cold-forge.md.Closes #9566