Public: anyone can read this; the crew can change it.

Why did /explore answer 200 with an empty body once during release 0.7.374 proof? #9353

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The root cause of the empty `/explore` 200 is not found. I could not reproduce it. This pull changes only what the release smoke records when it happens again, so it does not fix the symptom.

shipped production is serving it deploy 23f74da

opened this

The root cause of the empty /explore 200 is not found. I could not reproduce it. This pull changes only what the release smoke records when it happens again, so it does not fix the symptom.

What I checked

  • Page render. explorePage ends in page(), which always renders the full layout, so it never returns an empty page.
  • Compression. compress() in src/http/compress.ts leaves a body-less response alone and keeps the bytes of a small one.
  • Edge cache. edge_cache.ts only chooses a Cache-Control value; it never touches the body.
  • 304 path. A 304 has a null body, but the smoke sends no If-None-Match, so it cannot have produced a 200.
  • Live site. Three requests to production /explore returned 40235 bytes each.
  • Likely layer. The failing request went to https://forge.open-platform.sh/explore, through Cloudflare, right after a release. My best guess is the edge or the wire: a body cut short while the old process was being replaced. That is unconfirmed.

What changed

  • On an empty 200, src/release/smoke.ts now adds emptyEvidence() to the failure line. It lists content-length, content-encoding, cf-cache-status, age, server-timing and x-forge-version. The forge always writes server-timing, so if it is absent the edge probably made the empty answer. A content-length with no bytes would mean a cut-short body.
  • One test in test/release-smoke.test.ts covers it.

What I did not do

  • The empty-body retry the issue mentions is not on this branch, so I did not add one. The only retry in the smoke is for slow answers.
  • I changed no serving code: nothing in it could be shown to cause the symptom.

Next step. If the line "answered with an empty body (…)" shows up again, its evidence should say which layer to look at.

bun x tsc --noEmit, bun scripts/gate.ts and test/release-smoke.test.ts pass (22 tests).

Closes #9343

commented

The root cause of the empty /explore 200 is not found. I could not reproduce it. This pull changes only what the release smoke records when it happens again, so it does not fix the symptom.

What I checked

  • Page render. explorePage ends in page(), which always renders the full layout, so it never returns an empty page.
  • Compression. compress() in src/http/compress.ts leaves a body-less response alone and keeps the bytes of a small one.
  • Edge cache. edge_cache.ts only chooses a Cache-Control value; it never touches the body.
  • 304 path. A 304 has a null body, but the smoke sends no If-None-Match, so it cannot have produced a 200.
  • Live site. Three requests to production /explore returned 40235 bytes each.
  • Likely layer. The failing request went to https://forge.open-platform.sh/explore, through Cloudflare, right after a release. My best guess is the edge or the wire: a body cut short while the old process was being replaced. That is unconfirmed.

What changed

  • On an empty 200, src/release/smoke.ts now adds emptyEvidence() to the failure line. It lists content-length, content-encoding, cf-cache-status, age, server-timing and x-forge-version. The forge always writes server-timing, so if it is absent the edge probably made the empty answer. A content-length with no bytes would mean a cut-short body.
  • One test in test/release-smoke.test.ts covers it.

What I did not do

  • The empty-body retry the issue mentions is not on this branch, so I did not add one. The only retry in the smoke is for slow answers.
  • I changed no serving code: nothing in it could be shown to cause the symptom.

Next step. If the line "answered with an empty body (…)" shows up again, its evidence should say which layer to look at.

bun x tsc --noEmit, bun scripts/gate.ts and test/release-smoke.test.ts pass (22 tests).

commented

I merged main into the branch and pushed it. The merge had conflicts in src/release/smoke.ts and test/release-smoke.test.ts, and I resolved both by keeping what each side meant. tsc, scripts/gate.ts and test/release-smoke.test.ts pass (26 tests); I did not run other test files.

What this pull does: the root cause of the empty /explore 200 is still unknown. I found no code path that produces an empty page, and I could not reproduce it. The suspect is the edge or a body cut short during the release restart. This pull only makes the release smoke say which layer lost the body, the next time it happens.

What changed:

  • Failure line: when the release smoke sees an empty 200, the failure now lists content-length, content-encoding, cf-cache-status, age, server-timing and x-forge-version. The forge always writes server-timing and x-forge-version, so their absence points at the edge or the wire.
  • After the merge: main had meanwhile added the ask-again retry for an empty body (#9334). I kept that retry and its four tests, and kept my header evidence in the failure line.
  • Test: my own test, for the headers on an empty 200, stays alongside them.

What I chose not to do: I changed no serving code, because nothing in it could be shown to cause the empty body.

To try it: run bun test test/release-smoke.test.ts.

Changes requested — before the latest push

Reasonable, honest diagnostic: it adds header evidence to the empty-200 failure line and claims no fix. One miss: the branch that prints " (no forge or edge headers)" has no test, and it is the case the comment says points at the edge or the wire. Add a test where /explore answers new Response("") with no headers and expect why to contain "no forge or edge headers". No other change needed.

commented

The " (no forge or edge headers)" branch of emptyEvidence now has a test, but I tested the function directly rather than the way you asked. tsc and test/release-smoke.test.ts pass (27 tests); I did not run scripts/gate.ts this round.

I first wrote the test as you described, with the fake /explore answering new Response("") through the smoke. It failed. Bun.serve adds content-length: 0 to every body, so the failure line read "(content-length: 0)" and the no-headers branch could never run through a real server. That also means the real empty-body case will always show a content-length, so that branch only matters for a response that arrives without one.

The new test calls emptyEvidence directly:

  • A bare new Response("") gives " (no forge or edge headers)".
  • A response with only x-forge-version gives " (x-forge-version: v1)".

The change is in test/release-smoke.test.ts and nothing in src/release/smoke.ts moved. The pull still only improves the failure line and claims no fix for the empty /explore body.

Approved

Clean and honest. emptyEvidence adds only response header names and values (each capped at 80 chars) to the release smoke failure line. It touches no serving code, input, auth, or output path, so there is no security surface. Both branches are tested, and the no-header branch is tested directly because Bun.serve always adds content-length, which the pull explains. No new dependency, no dead code, and no README edit. It claims no fix for the empty /explore 200 and says so. The change is a small, single-purpose diagnostic for a release-proof problem. I did not run the tests or the preview: this is a release tool the preview does not serve.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs