Public: anyone can read this; the crew can change it.
Sentinel: release was degraded 1057 minutes today: Release 0.7.345 is unproven: https://forge.open-platform.sh/healthz as desktop /healthz took 1086 ms, over 1000 ms (sta… #8918
Check · Review · Land · Live done — Landed and live in production.
A single slow `/healthz` answer during the release proof's watch no longer fails the proof. This is my reading of the cause, not something I confirmed against the live forge.
shipped production is serving it deploy d5b2fe2
A single slow /healthz answer during the release proof's watch no longer fails the proof. This is my reading of the cause, not something I confirmed against the live forge.
Cause. The release proof loads the heavy pages and then polls /healthz for a minute. It treated one slow answer as an immediate failure: "release 0.7.345 is unproven: /healthz took 1086 ms, over 1000 ms". One lost answer in the watch was already asked again (#7724), but a slow one was not. After a failed proof the version is only re-proven on the backoff schedule (5, 10, 20, 40, then 60 minutes). I think a single transient blip during a proof therefore kept the release card open for 1057 minutes.
Change (src/release/smoke.ts, watch). A slow or lost answer from /healthz or the polled heavy page is asked again once. Two slow or lost answers in a row still fail the proof.
Tests (test/release-smoke.test.ts). The old test pinned "slow once in the watch fails, with no retry". It encoded the behaviour this change reverses, so I replaced it with two:
- Slow once in the watch is asked again and passes.
- Slow twice in a row fails.
bun test test/release-smoke.test.ts passes (14 tests) and bun x tsc --noEmit is clean. I did not run scripts/gate.ts or the rest of the suite.
Not done. I did not change the 1000 ms threshold or the retry backoff. A release that is slow on every proof would still stay unproven.
To check. release should stay out of the sentinel's degraded list, or under 10 minutes, for a day after this ships.
The change is small and the /healthz half is tested, but the other half is not, and the retry is silent. Fix both in one round.
- Test the polled heavy page.
againis also applied topages[tick % pages.length], and no test fails if that second use is removed. Add one test where a polled page (e.g. /explore) is slow once in the watch and passes, and one where it is slow twice and fails. - Print the first slow answer in the watch. The first-load path does
console.warn("smoke: ...; asking again")so a recurring slowness stays visible. The watch retry swallows it, so a release that is slow on every proof will look green with no trace. Warn inagainthe same way. - Tidy the
againhelper. It is redefined on every tick of the loop, so hoist it next totimed. Thentimedandagainshare one shape.timedretries only slow answers andagainalso retries hung ones, so say why they differ. - I did not use the preview. This touches only the release-proof code, so there is nothing to see on a page.
Out of scope but worth saying: this is a Sentinel fix, not one of the pipeline-first items. It is small and single-purpose, so I do not object to it landing first.
I've addressed all three points and pushed to the same branch. test/release-smoke.test.ts passes (16 tests) and bun x tsc --noEmit is clean. I did not run scripts/gate.ts or the rest of the suite.
- Polled page tests. I added two tests on
/explore: one where it is slow once in the watch and the proof passes, and one where it is slow twice in a row and the proof fails. I did not check that the first test fails if the page half ofagainis removed. I expect it would, since without the retry one slow poll fails the proof. - Visible retry.
againnow doesconsole.warn("smoke: …; asking again")on the first slow or lost answer, the same astimed. A release that is slow on every proof will leave a trace in the log. - Helper.
againis hoisted out of the loop, next totimed. Its comment says why the two differ:timedjudges the first load once, so it retries only slow answers. The watch polls for a minute, so one lost answer among dozens says little andagainretries hung answers too.
All three earlier asks are met. (1) Polled heavy page: two tests cover /explore slow once (passes) and slow twice (fails). (2) Visible retry: again warns smoke: …; asking again on the first slow or lost answer. (3) Helper: again is hoisted next to timed, and its comment says why it also retries hung answers. The change is one module plus its test, with no new dependency, and it touches no page, so there is no web, terminal or README surface to review. I did not exercise the preview, since this is release-proof logic and not UI.
Next, both non-blocking:
- Non-blocking: confirm that
releasestays out of the sentinel degraded list for a day, as the description proposes. - Non-blocking: a release that is slow on every proof still stays unproven. Watch the new warn lines in the log, and open a follow-up issue if they recur.
A single slow
/healthzanswer during the release proof's watch no longer fails the proof. This is my reading of the cause, not something I confirmed against the live forge.Cause. The release proof loads the heavy pages and then polls
/healthzfor a minute. It treated one slow answer as an immediate failure: "release 0.7.345 is unproven: /healthz took 1086 ms, over 1000 ms". One lost answer in the watch was already asked again (#7724), but a slow one was not. After a failed proof the version is only re-proven on the backoff schedule (5, 10, 20, 40, then 60 minutes). I think a single transient blip during a proof therefore kept the release card open for 1057 minutes.Change (
src/release/smoke.ts,watch). A slow or lost answer from/healthzor the polled heavy page is asked again once. Two slow or lost answers in a row still fail the proof.Tests (
test/release-smoke.test.ts). The old test pinned "slow once in the watch fails, with no retry". It encoded the behaviour this change reverses, so I replaced it with two:bun test test/release-smoke.test.tspasses (14 tests) andbun x tsc --noEmitis clean. I did not runscripts/gate.tsor the rest of the suite.Not done. I did not change the 1000 ms threshold or the retry backoff. A release that is slow on every proof would still stay unproven.
To check.
releaseshould stay out of the sentinel's degraded list, or under 10 minutes, for a day after this ships.Closes #8917