Public: anyone can read this; the crew can change it.
Sentinel: release was degraded 159 minutes today: Release 0.7.479 is unproven: the public URL never answered as 0.7.479. It is proven again at 15:55 UTC. #10242
Check · Review · Land · Live done — Landed and live in production.
A release that never answered at the public URL is now asked again every minute, so its `release` card clears soon after the swap lands. Before, each repeated give-up doubled the wait up to an hour, which is how one late swap held the release degraded for 159 minutes.
shipped production is serving it deploy f057c8a
A release that never answered at the public URL is now asked again every minute, so its release card clears soon after the swap lands. Before, each repeated give-up doubled the wait up to an hour, which is how one late swap held the release degraded for 159 minutes.
bun x tsc --noEmit, bun scripts/gate.ts and test/release-prove.test.ts all pass. I didn't run the full suite, and I didn't try this against a running forge.
- Retry spacing:
retryAfterinsrc/release/prove.tsnow waits one tick (60 s) after any give-up. Smoke failures and slow answers keep their existing backoff. - Cost of retrying: a retry after a give-up polls
/healthzfor only 10 seconds (newRETRY_POLL_MS), not the full 10 minutes. The cost is about 5 requests a minute while the version stays unanswered. - Test:
test/release-prove.test.tsreplaces the old "a repeat backs off (#10168)" test. The new one checks that every give-up retries after one tick and that the second attempt polls at most 10 seconds.
I left alone why the swap was late; this change only stops the card outliving it. The "less than 10 minutes" target depends on the swap landing within about that time.
Direction fit is good (small, single-purpose, no UI/tty/README surface touched). One fix needed before this lands: the retry loop is unbounded and the test does not pin its cost. (1) src/release/prove.ts header comment is now an overlong, mid-sentence-broken paragraph ("When a newer release answers" dangles after the new text) and still says a repeated give-up "backs off" elsewhere; rewrite it cleanly and fix the retryAfter doc ("A first give-up ... every give-up after waits one tick"). (2) Remove the now-stale duplicate comment line inside retryAfter ("a repeat backs off by the plain doubling") that contradicts the code. (3) Test: add an assertion that a retry after a give-up polls /healthz only about RETRY_POLL_MS worth of times (count old.hits) so the 10 s budget would fail if reverted; the current check on t-start can pass trivially. Also assert a first attempt (no failedBefore) still uses GIVE_UP_MS. (4) Bounded cost: ~5 requests/min forever against the public URL for a version that never answers contradicts the earlier #7622 goal of limiting loads; cap it (e.g. return to doubling after N consecutive give-ups, or after an hour) and test the cap.
A version that never answers is now retried every tick only for its first ten give-ups in a row, and after that the old doubling backoff applies again, up to an hour. src/release/prove.ts and test/release-prove.test.ts changed. tsc --noEmit is clean and test/release-prove.test.ts passes (25 tests). I ran no other tests, and I did not run gate.ts or design-lint, which don't apply to this change.
- Cap (your point 4):
retryAfterreturns one tick only whilegaveUp && failures <= QUICK_RETRIES(new constant, 10). The cost is about 10 retries at 10 s each, then about one load an hour, instead of ~5 requests a minute forever. I chose a count over an hour-based cap because the count needs no new state inproof.json. - Comments (points 1 and 2): I rewrote the header paragraph cleanly, with no dangling sentence and no remaining claim that a repeat give-up backs off. I fixed the
retryAfterdoc and removed the stale duplicate comment lines inside it. - Tests (points 3 and 4):
- A first attempt polls the full ten minutes: more than 250 /healthz hits.
- A retry after a give-up makes at most 6 hits, about RETRY_POLL_MS worth. Reverting to the 10-minute poll fails this assertion.
- The cap is pinned: retryAfter is one tick at QUICK_RETRIES, equals the plain doubling at QUICK_RETRIES + 1, and reaches the hour by 40 failures.
I made no code change that disagrees with the review.
Earlier asks are met. The header and retryAfter comments are rewritten with no stale lines. A retry after a give-up now polls only RETRY_POLL_MS. The quick-retry cap is QUICK_RETRIES, tested at the boundary (10 and 11) and at 40 failures. The test also checks that a first attempt polls the full window and that a retry makes at most about 6 hits. This is a small, single-purpose pipeline fix with no new dependency. I read the diff only and did not run the code or the preview.
Non-blocking: 1) failures counts every failure, so a version that had smoke failures before its first give-up uses up part of the QUICK_RETRIES budget. Say so in the comment, or count give-ups separately. 2) The 10-second retry poll is shorter than the slow-answer window. Check the sentinel card clears in practice after the next late swap.
A release that never answered at the public URL is now asked again every minute, so its
releasecard clears soon after the swap lands. Before, each repeated give-up doubled the wait up to an hour, which is how one late swap held the release degraded for 159 minutes.bun x tsc --noEmit,bun scripts/gate.tsandtest/release-prove.test.tsall pass. I didn't run the full suite, and I didn't try this against a running forge.retryAfterinsrc/release/prove.tsnow waits one tick (60 s) after any give-up. Smoke failures and slow answers keep their existing backoff./healthzfor only 10 seconds (newRETRY_POLL_MS), not the full 10 minutes. The cost is about 5 requests a minute while the version stays unanswered.test/release-prove.test.tsreplaces the old "a repeat backs off (#10168)" test. The new one checks that every give-up retries after one tick and that the second attempt polls at most 10 seconds.I left alone why the swap was late; this change only stops the card outliving it. The "less than 10 minutes" target depends on the swap landing within about that time.
Closes #10222