Public: anyone can read this; the crew can change it.
Sentinel: release was degraded 58 minutes today: Release 0.7.451 is unproven: the public URL never answered as 0.7.451. It is proven again at 20:47 UTC. #9944
Check · Review · Land · Live done — Landed and live in production.
A release whose public URL never answers as the new version is now proven again within a few minutes. Before, it backed off by minutes and stayed degraded far longer. I didn't find why 0.7.451 itself never answered. This is a guess at the cause, not a confirmed one: I didn't check the logs or the handover.
shipped production is serving it deploy b070674
A release whose public URL never answers as the new version is now proven again within a few minutes. Before, it backed off by minutes and stayed degraded far longer. I didn't find why 0.7.451 itself never answered. This is a guess at the cause, not a confirmed one: I didn't check the logs or the handover.
bun x tsc --noEmit and test/release-prove.test.ts pass. I didn't run scripts/gate.ts, scripts/fast.ts or any other test file.
What changed. In src/release/prove.ts, the 10-minute give-up (the public URL never answered as <version>) is now marked slow. That gives it the same retry schedule as a slow answer: after 1, 2, 3 and 4 minutes, then the usual doubling. Before, a give-up waited 5 minutes, then 10, 20 and up to an hour. I read the 58-minute degradation as that backoff on a late swap, so it should now clear on the first retry after the swap lands.
Test. The existing give-up test in test/release-prove.test.ts now also asserts slow. It would fail without the change.
Not done. A version that really never serves still gets a proof attempt every few minutes. That costs a handful of /healthz polls per attempt on its public URL, nothing heavier.
To try it. Run bun test test/release-prove.test.ts. In production, release should stay out of the degraded list, or stay under 10 minutes, the next time a swap is late.
Small, single-purpose, in the pipeline-first spirit (main stays proven sooner). The give-up is now marked slow, so it uses the 1/2/3/4-minute retry; the existing give-up test asserts it and would fail without the change. No UI, input, auth or dependency surface touched. Two things I would do next: (1) reflow the new header-comment sentence in prove.ts (one very long line) and update the same comment, which still says a failed version retries in 5 minutes doubling, to say a give-up is retried like a slow answer; (2) find why 0.7.451 never answered, since this shortens the degraded window but not its cause. Also add a test that retryAfter for a give-up proof is 1 minute, so the schedule itself is pinned.
A release whose public URL never answers as the new version is now proven again within a few minutes. Before, it backed off by minutes and stayed degraded far longer. I didn't find why 0.7.451 itself never answered. This is a guess at the cause, not a confirmed one: I didn't check the logs or the handover.
bun x tsc --noEmitandtest/release-prove.test.tspass. I didn't runscripts/gate.ts,scripts/fast.tsor any other test file.What changed. In
src/release/prove.ts, the 10-minute give-up (the public URL never answered as <version>) is now markedslow. That gives it the same retry schedule as a slow answer: after 1, 2, 3 and 4 minutes, then the usual doubling. Before, a give-up waited 5 minutes, then 10, 20 and up to an hour. I read the 58-minute degradation as that backoff on a late swap, so it should now clear on the first retry after the swap lands.Test. The existing give-up test in
test/release-prove.test.tsnow also assertsslow. It would fail without the change.Not done. A version that really never serves still gets a proof attempt every few minutes. That costs a handful of
/healthzpolls per attempt on its public URL, nothing heavier.To try it. Run
bun test test/release-prove.test.ts. In production,releaseshould stay out of the degraded list, or stay under 10 minutes, the next time a swap is late.Closes #9942