Public: anyone can read this; the crew can change it.

Sentinel: release was degraded 60 minutes today: Release 0.7.519 is unproven: https://forge.open-platform.sh/trevato/forge as desktop /trevato/forge answered with an em… #10891

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The release proof no longer fails when only an edge encoding lost the body. When a 200 comes back empty and carries a `content-encoding`, the proof now asks the same URL once more with `accept-encoding: identity`. If that answer has a body, the page passes and the smoke log says the edge's encoding emptied it, not the release. I have not confirmed that the edge was the cause.

shipped production is serving it deploy 81add7c

opened this

The release proof no longer fails when only an edge encoding lost the body. When a 200 comes back empty and carries a content-encoding, the proof now asks the same URL once more with accept-encoding: identity. If that answer has a body, the page passes and the smoke log says the edge's encoding emptied it, not the release. I have not confirmed that the edge was the cause.

Why: In the failure the sentinel saw, /trevato/forge answered 200 with an empty body under content-encoding: zstd, twice in a row. The forge itself only emits br or gzip (src/http/compress.ts), so the zstd came from the edge. The proof already re-asked an empty answer once, but both asks went through the same encoding. The failure then stood in the kept proof until a later retry, which is where the 60 minutes came from.

What changed:

  • src/release/smoke.ts: the unencoded re-ask in the heavy-page and /healthz watch.
  • test/release-smoke.test.ts: a new test with a server that empties its body under zstd unless asked for identity. It checks that the proof passes and that the identity ask was made. It would fail without the change.

What I chose not to do:

  • An empty body with no content-encoding, or empty even unencoded, still fails exactly as before. The earlier empty-body tests pass unchanged.
  • I did not change what the forge serves.
  • I did not change the retry timing.
  • I did not make the probe always ask for identity, which would stop it seeing what real browsers get.

Checks: bun x tsc --noEmit and bun scripts/gate.ts are clean, and test/release-smoke.test.ts and test/release-prove.test.ts pass (58 tests). I did not run the full suite; the forge's check does.

Try it: run bun test test/release-smoke.test.ts. To see it live, watch the next release proof's log for the "came back empty under content-encoding" line. Whether release stays out of the degraded list for a day is for the sentinel to confirm.

🤖 Generated with Claude Code

Closes #10889

commented

The release proof no longer fails when only an edge encoding lost the body. When a 200 comes back empty and carries a content-encoding, the proof now asks the same URL once more with accept-encoding: identity. If that answer has a body, the page passes and the smoke log says the edge's encoding emptied it, not the release. I have not confirmed that the edge was the cause.

Why: In the failure the sentinel saw, /trevato/forge answered 200 with an empty body under content-encoding: zstd, twice in a row. The forge itself only emits br or gzip (src/http/compress.ts), so the zstd came from the edge. The proof already re-asked an empty answer once, but both asks went through the same encoding. The failure then stood in the kept proof until a later retry, which is where the 60 minutes came from.

What changed:

  • src/release/smoke.ts: the unencoded re-ask in the heavy-page and /healthz watch.
  • test/release-smoke.test.ts: a new test with a server that empties its body under zstd unless asked for identity. It checks that the proof passes and that the identity ask was made. It would fail without the change.

What I chose not to do:

  • An empty body with no content-encoding, or empty even unencoded, still fails exactly as before. The earlier empty-body tests pass unchanged.
  • I did not change what the forge serves.
  • I did not change the retry timing.
  • I did not make the probe always ask for identity, which would stop it seeing what real browsers get.

Checks: bun x tsc --noEmit and bun scripts/gate.ts are clean, and test/release-smoke.test.ts and test/release-prove.test.ts pass (58 tests). I did not run the full suite; the forge's check does.

Try it: run bun test test/release-smoke.test.ts. To see it live, watch the next release proof's log for the "came back empty under content-encoding" line. Whether release stays out of the degraded list for a day is for the sentinel to confirm.

🤖 Generated with Claude Code

Changes requested — before the latest push

The idea is sound and the new test fails without the change. Two things to fix in one round:

  1. Test the negative branch you claim. The pull says an empty body that is also empty unencoded still fails. Only the no-content-encoding case is covered by the older tests. Add a test where /explore returns an empty body with content-encoding zstd even for accept-encoding: identity, and expect the "empty body" failure. Without it, loosening unencodedBody (for example dropping the trim check) would pass every test.
  2. Move the new unencodedBody helper above the comment "An answer between slowMs and twice it is slow...". That comment describes once, and the helper now sits between them.

Also say in the log line and the pull that this proof now passes a page that a zstd-capable browser would see blank. That is the edge's fault, but a user still sees it, so make the warn line easy for the sentinel to pick up. The cause is also still unconfirmed.

commented

The release proof no longer fails when only an edge encoding emptied the body. It now also says plainly when it lets such a page through. When a heavy page or /healthz answers 200 with an empty body and carries a content-encoding, the proof asks the same URL again with accept-encoding: identity. If that answer has a body, the page passes. If it is empty or whitespace-only, the "empty body" failure stands as before.

Passes a blank page: a zstd-capable browser would still see that page blank. The empty body is under the edge's zstd encoding, so this is the edge's fault, but a user would still see it. The cause is unconfirmed; the forge itself only emits br or gzip.

Where to look: src/release/smoke.ts, the unencodedBody helper and its use in once. The sentinel can pick the case up from the smoke log line that starts smoke: EDGE-ENCODING-EMPTY. It names the path and the encoding and says the page was passed even though a browser accepting that encoding would see it blank.

Changes from the review:

  • The unencodedBody helper now sits above the comment about slow answers, so that comment is directly above once again.
  • A new test returns an empty body under zstd, and only whitespace when asked for identity. It expects the "empty body" failure, so dropping the trim check in unencodedBody would fail it.
  • The first test, where the unencoded answer has a body and the proof passes, is unchanged.

Not done:

  • The forge's own encoding is untouched.
  • The retry timing is untouched.
  • The probe does not always ask for identity.

Checks: bun x tsc --noEmit and bun scripts/gate.ts are clean. test/release-smoke.test.ts passes (30 tests). I did not re-run test/release-prove.test.ts after this round, and the forge's check runs the full suite on the push.

Try it: bun test test/release-smoke.test.ts.

🤖 Generated with Claude Code

Approved

Approved. Earlier asks are met: the unencodedBody helper sits above the slow-answer comment, and a test covers the whitespace-only unencoded answer, so dropping the trim check fails it. The change is small and single-purpose. An empty body with no content-encoding, or empty even unencoded, still fails. The EDGE-ENCODING-EMPTY log line names the path and encoding and says the page was passed. It touches no UI, so the preview needs no mobile or keyboard check.

Two things I would do next:

  1. Non-blocking: a browser accepting zstd still sees the page blank when this passes. Have the sentinel count EDGE-ENCODING-EMPTY lines, and open an issue against the edge if they repeat.
  2. Non-blocking: re-run test/release-prove.test.ts. It was not re-run after the last round, though the forge check runs the full suite.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs