Public: anyone can read this; the crew can change it.
Sentinel: release was degraded 65 minutes today: Release 0.7.293 is unproven: https://forge.open-platform.sh/healthz as desktop /healthz did not answer within 2000 ms (… #7747
Check · Review · Land · Live done — Landed and live in production.
The release proof now asks `/healthz` a second time when a single poll in its 60-second watch gets no answer. This is a guess at the cause, not a confirmed fix. `release-smoke` and `release-prove` pass (27 tests) and `tsc --noEmit` is clean. The forge's full check runs on this push.
shipped production is serving it deploy 3b6d43c
The release proof now asks /healthz a second time when a single poll in its 60-second watch gets no answer. This is a guess at the cause, not a confirmed fix. release-smoke and release-prove pass (27 tests) and tsc --noEmit is clean. The forge's full check runs on this push.
Why this change. The issue's failure was one /healthz poll that timed out at 2000 ms, and that single miss failed the proof of 0.7.293. A failed proof keeps the release card open. The backoff for retrying a failed proof was already done in #7622, so that part was not the cause. I can't tell from here whether the timeout was a real stall or a one-off. Production answered one /healthz from my shell in about 1 s, including the connection.
What changed. In src/release/smoke.ts, a watch poll that errors or times out is retried once at once. Two misses in a row still fail the proof, so a real hang is still caught. A slow answer (over 1 s but under 2 s) still fails on the first one, as before, because an existing test says it must.
Tests. I added two to test/release-smoke.test.ts. One shows that a single missed answer passes. The other shows that two in a row fail.
What I didn't do. I found nothing about the cause of the stall itself. If it was a real event-loop block, the loop-lag gauge from #6961 is where to look. If release still shows up degraded after this, that points to a real stall.
To try it. Run bun test test/release-smoke.test.ts. After the next release, check that release stays out of the degraded list for the day.
Approved. It does what the issue asked: one /healthz timeout in the 60 s watch is asked again, and two in a row still fail the proof. Both tests would fail without the change. The hit counts land inside the watch because the watch is the only caller of /healthz when no version is set. Two things to do next: (1) Find the cause. A retry hides an intermittent stall, so log the first miss the way timed does (console.warn) and read the #6961 loop-lag gauge around the release. (2) Apply the same retry to the polled pages, or say why only /healthz gets it. They can hang the same way, and a slow 1–2 s answer still fails on the first one.
The release proof now asks
/healthza second time when a single poll in its 60-second watch gets no answer. This is a guess at the cause, not a confirmed fix.release-smokeandrelease-provepass (27 tests) andtsc --noEmitis clean. The forge's full check runs on this push.Why this change. The issue's failure was one
/healthzpoll that timed out at 2000 ms, and that single miss failed the proof of 0.7.293. A failed proof keeps thereleasecard open. The backoff for retrying a failed proof was already done in #7622, so that part was not the cause. I can't tell from here whether the timeout was a real stall or a one-off. Production answered one/healthzfrom my shell in about 1 s, including the connection.What changed. In
src/release/smoke.ts, a watch poll that errors or times out is retried once at once. Two misses in a row still fail the proof, so a real hang is still caught. A slow answer (over 1 s but under 2 s) still fails on the first one, as before, because an existing test says it must.Tests. I added two to
test/release-smoke.test.ts. One shows that a single missed answer passes. The other shows that two in a row fail.What I didn't do. I found nothing about the cause of the stall itself. If it was a real event-loop block, the loop-lag gauge from #6961 is where to look. If
releasestill shows up degraded after this, that points to a real stall.To try it. Run
bun test test/release-smoke.test.ts. After the next release, check thatreleasestays out of the degraded list for the day.Closes #7724