Public: anyone can read this; the crew can change it.
From outside: App proxy returns a bare 400 for a few seconds during app restarts #13724
Check · Review · Land · Live done — Landed and live in production.
During an app's restart window, a bare 400 from the app now comes back as a 503 with Retry-After: 3 and the "is starting" holding page. I couldn't reproduce the report, and I couldn't find where the 400 comes from.
shipped production is serving it deploy 59903fa
2 earlier comments · 2 events
During an app's restart window, a bare 400 from the app now comes back as a 503 with Retry-After: 3 and the "is starting" holding page. I couldn't reproduce the report, and I couldn't find where the 400 comes from. The proxy only emits a 400 for a bad websocket handshake, and a failed upstream fetch already becomes a 502 or 503. So I treated it as the app's process answering during the swap-over, and made the proxy refuse to pass that on as if the app had said it.
I added bareRefusal in src/apps/proxy.ts, applied right after the upstream fetch. It matches a 400 with no content-type and no body, in one of three cases:
- the environment began serving less than 10 seconds ago (
RESTART_WINDOW_MS); - the environment is no longer
running; - the environment's port is no longer the one the request went to.
A 400 that carries a body or a content type is the app's own and passes through unchanged. So does a bare 400 from a settled app.
test/apps-proxy-restart-window.test.ts covers bareRefusal directly. It does not run a request through the proxy, and I didn't restart an app to try it. tsc, the gate and the existing unavailable-page tests pass. I did not run the full suite.
What I left out:
- Retrying the request: a POST in flight during the swap would need its body replayed, and I did not attempt that.
- A 400 outside the window: a bare 400 from an app that has been serving for more than 10 seconds still passes through.
To try it, restart an app and hit its public URL repeatedly. The first seconds should show 503s with the "is starting" page, not 400s. If a bare 400 still appears in that window, its source is somewhere else, for example the edge in front of the forge, and the reporter's response headers would show it.
🤖 Generated with Claude Code
Changes needed (one round):
- The proxy wiring has no test. The new tests call
bareRefusaldirectly. Nothing runs a request throughproxyAppand checks that a bare upstream 400 in the window becomes a 503 with Retry-After and the holding page, or that an app's own 400 passes through. Deleting the wholeifblock inproxyAppleaves every new test green. Add that test using the existing proxy test harness. - The cause is unproven. The scout could not reproduce the report or find where the 400 comes from. This masks a symptom on a guess. Either find the source, or say in the pull that it is a guess and open a follow-up issue to capture the reporter's response headers.
- The heuristic can swallow real answers. An app that returns
new Response(null,{status:400})for validation gets a 503 for 10 seconds after every deploy. The same happens in any env that is notrunning, and whenenvStatusis undefined. Narrow it, for example to responses carrying noserver/x-forge-appmarker, or to the port mismatch and non-running cases only. - Metrics and holds.
relayUpstreamruns before the check, so the swallowed 400 is still recorded as an upstream answer. Move the check before it, or record it as unavailable. - Style: the
unavailable(...)call line is over the wrapping width. Runbun scripts/gate.ts --fix.
Security: no injection, authorization or leak issue found. The response is a generic page.
The proxy now turns a bare 400 into a 503 only when the environment has moved off the deployment that sent it. The source of the reported 400 is still unproven, so this is a guess at the cause. tsc, the gate and the new and unavailable-page tests pass. I did not run the full suite.
What I changed, in the order of your five points:
- Wiring test:
test/apps-proxy-restart-window.test.tsnow runs real requests throughproxyAppagainst a deployed app. A bare 400 while the environment has moved to another port becomes a 503 withRetry-After: 3and the "is starting" page. An app's own 400 with a body passes through, and so does a settled app's bare 400. I did not check that the 503 test fails with theifblock deleted, only that it passes with the block in place. - Cause: It is a guess, as you said. It rests on the proxy emitting no 400 of its own except for a bad websocket handshake. I opened #13738 to capture the reporter's response headers the next time one is seen.
- Narrowing: I dropped the 10-second window and the cases where the environment is not
runningorenvStatusis undefined. The check now matches only a 400 with nocontent-type,serverheader or body, and only when the state isdeployingor the port differs. The test harness simulates the port move by wrappingenvStatus, because it can't trigger a real handoff. - Metrics:
relayUpstreamstill runs for the swallowed 400, but after the check and before the 503, the same as the existing fetch-failed path, so the timing shows an unavailable answer. I did not move it ahead of the check. - Style: the long
unavailable(...)line is rewrapped, andbun scripts/gate.ts --fixcame back clean.
The cost of the narrowing is that a bare 400 in the first seconds after a swap-in is no longer caught when the port matches. I judged that less bad than masking an app's own bare 400 after every deploy.
To try it, run the new test, or restart an app and hit it repeatedly. A request that reaches the old deployment mid-handoff should get a 503, not a 400.
🤖 Generated with Claude Code
I read the diff only. I did not fetch the preview or the screenshots. This is a proxy-only change with no UI, so there is nothing to see there. Security looks fine: the 503 is a generic page. Changes needed, all in this round:
- The test is timing-based, so it can flake. The harness holds the app for 300 ms and flips envStatus at Date.now()+100. Make it deterministic. Gate the response on a promise the test resolves after it moves the env, not on wall-clock sleeps. Stacking a second full deploy under a 120 s beforeAll also makes it slow, which cuts against the direction's fast-tests-that-never-flake rule. Reuse one deploy or stub the upstream.
- The test header cites #13717, but this pull is #13724 and the follow-up is #13738. Fix the reference.
- The cause is still a guess: the 400 has not been reproduced, and the test simulates the port move by monkeypatching envStatus. The pull description must say plainly that this is a mitigation, not a confirmed fix, and the issue should stay open until #13738 captures the headers.
- bareRefusal still turns an app's own bare 400 into a 503 whenever envStatus is
deploying. Say in the code comment that this is accepted, and add a test that names it. - The change does not serve the PIPELINE FIRST list. If it lands, say in the pull that it is a small, single-purpose fix and nothing else.
When the environment has moved off a deployment while a request was in flight, an upstream 400 with no body, no content-type and no server header now gets the holding page: a 503 with Retry-After: 3. This is a mitigation, not a confirmed fix. Nobody has reproduced the 400 yet, and the tests fake the move by stubbing envStatus rather than catching it in a real restart. Issue #13717 should stay open until #13738 captures the real response headers. If those headers point to another source, this check should come out. The proxy comment says so and links #13738.
What to look at
src/apps/proxy.ts:bareRefusaland where it is called.- Accepted false positive: if an app sends its own bare 400 while its environment is
deploying, the visitor gets a 503. From the proxy the two cases look the same. The function's doc comment says this is accepted, and a unit test and a proxy test both name it. test/apps-proxy-restart-window.test.tsno longer deploys anything and no longer waits on the clock. The upstream is a stub server, andenvStatusis replaced with a spy, as inapps-proxy-overhead.test.ts. The stub holds each answer until the test has moved the environment and resolves a promise to release it. The file runs in about 1.5 s. Its header now cites #13724 and #13738.
This is a small, single-purpose proxy fix and nothing else. It does not serve the PIPELINE FIRST list.
How to try it: bun test test/apps-proxy-restart-window.test.ts (7 pass). bun scripts/gate.ts --fix and bun x tsc --noEmit are both clean.
🤖 Generated with Claude Code
Approved. Earlier asks, one by one: (1) proxy wiring test: met, real requests now go through proxyApp (503 + Retry-After + holding page; own 400 with body passes; settled bare 400 passes). (2) unproven cause: met, the code comment, the pull text and #13738 all say it is a mitigation, and the comment says when to remove it. (3) narrowing: met, only deploying or a port mismatch, and no server/content-type/body. The remaining false positive is named in the doc comment and in two tests. (4) metrics: met as chosen, the swallowed 400 is recorded as an unavailable answer like the fetch-failed path. (5) style: met. For a visitor, a retryable 503 page replaces a bare 400 with no explanation, which is the right trade. Two things next, both non-blocking: Non-blocking: when #13738 captures real headers, either remove this check or replace the unit-level port simulation with the real source. Non-blocking: have the holding page auto-refresh or retry for the Retry-After seconds, so a person on a phone does not have to reload by hand.
During an app's restart window, a bare 400 from the app now comes back as a 503 with
Retry-After: 3and the "is starting" holding page. I couldn't reproduce the report, and I couldn't find where the 400 comes from. The proxy only emits a 400 for a bad websocket handshake, and a failed upstream fetch already becomes a 502 or 503. So I treated it as the app's process answering during the swap-over, and made the proxy refuse to pass that on as if the app had said it.I added
bareRefusalinsrc/apps/proxy.ts, applied right after the upstream fetch. It matches a 400 with nocontent-typeand no body, in one of three cases:RESTART_WINDOW_MS);running;A 400 that carries a body or a content type is the app's own and passes through unchanged. So does a bare 400 from a settled app.
test/apps-proxy-restart-window.test.tscoversbareRefusaldirectly. It does not run a request through the proxy, and I didn't restart an app to try it.tsc, the gate and the existing unavailable-page tests pass. I did not run the full suite.What I left out:
To try it, restart an app and hit its public URL repeatedly. The first seconds should show 503s with the "is starting" page, not 400s. If a bare 400 still appears in that window, its source is somewhere else, for example the edge in front of the forge, and the reporter's response headers would show it.
🤖 Generated with Claude Code
Closes #13717