Sentinel: release was degraded 12 minutes today: Release 0.7.415 is unproven: https://forge.open-platform.sh/ as desktop / did not answer within 2000 ms (TimeoutError: … #9493

A release proof that fails because a page did not answer (a hang) is now judged like one that failed on a slow or empty answer. Before, it was treated as a broken release and kept the release card up 5 minutes or more each time.

What happened in #9482: 0.7.415's first load of /, right after boot, did not answer within 2000 ms. The heavy-page load in src/release/smoke.ts asked a slow or empty first answer again, but never a hang, so the proof failed on that one answer. The hang was then kept as an ordinary failure, so the retry waited 5 minutes and then doubled. That gave 12 minutes, then 18 by the time 0.7.417 hit the same thing on /healthz.

• src/release/smoke.ts: Failure gets a hung flag for "did not answer within twice slowMs". The first load of each heavy page (timed) now asks a hung answer again once, as it already did for slow and empty ones. A second hang in a row still fails the proof.
• src/release/prove.ts: a proof whose first failure is a hang is kept with slow: true. It is tried again after 1, 2, 3, 4 minutes and then doubles (the #9238 schedule), instead of 5 minutes doubling to an hour. A release that really hangs still stays unproven, it is just checked more often.
• Tests: two existing tests in test/release-smoke.test.ts pinned the old behaviour, so I changed them:

- "a hung first load is not asked again" is now two tests: one hang is asked again and passes; two hangs in a row fail and are marked hung. - "a hang is a failure whatever Server-Timing would have said" now hangs twice. What it checks still holds: Server-Timing never excuses a hang.

• New test: test/release-prove.test.ts checks that a proof failing on a hang is kept as slow and is due again after a minute.

What I chose not to do: this does not find why / and /healthz stopped answering for 2 s just after boot (a blocked event loop or the edge). Production logs can't be read from here. If it is real, two hangs in a row will still fail the proof and say so.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass, and test/release-smoke.test.ts and test/release-prove.test.ts pass (47 tests). scripts/fast.ts named 76 test files, over its own 5-minute budget. I ran only the six closest to this code (smoke, vitals-release, release-behind, forge-degraded, vitals-degraded, sentinel): 81 pass. The forge's check runs the rest. To try it: bun test test/release-smoke.test.ts test/release-prove.test.ts.

🤖 Generated with Claude Code

Closes #9482

A release proof that fails because a page did not answer (a hang) is now judged like one that failed on a slow or empty answer. Before, it was treated as a broken release and kept the release card up 5 minutes or more each time.

What happened in #9482: 0.7.415's first load of /, right after boot, did not answer within 2000 ms. The heavy-page load in src/release/smoke.ts asked a slow or empty first answer again, but never a hang, so the proof failed on that one answer. The hang was then kept as an ordinary failure, so the retry waited 5 minutes and then doubled. That gave 12 minutes, then 18 by the time 0.7.417 hit the same thing on /healthz.

• src/release/smoke.ts: Failure gets a hung flag for "did not answer within twice slowMs". The first load of each heavy page (timed) now asks a hung answer again once, as it already did for slow and empty ones. A second hang in a row still fails the proof.
• src/release/prove.ts: a proof whose first failure is a hang is kept with slow: true. It is tried again after 1, 2, 3, 4 minutes and then doubles (the #9238 schedule), instead of 5 minutes doubling to an hour. A release that really hangs still stays unproven, it is just checked more often.
• Tests: two existing tests in test/release-smoke.test.ts pinned the old behaviour, so I changed them:

- "a hung first load is not asked again" is now two tests: one hang is asked again and passes; two hangs in a row fail and are marked hung. - "a hang is a failure whatever Server-Timing would have said" now hangs twice. What it checks still holds: Server-Timing never excuses a hang.

• New test: test/release-prove.test.ts checks that a proof failing on a hang is kept as slow and is due again after a minute.

What I chose not to do: this does not find why / and /healthz stopped answering for 2 s just after boot (a blocked event loop or the edge). Production logs can't be read from here. If it is real, two hangs in a row will still fail the proof and say so.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass, and test/release-smoke.test.ts and test/release-prove.test.ts pass (47 tests). scripts/fast.ts named 76 test files, over its own 5-minute budget. I ran only the six closest to this code (smoke, vitals-release, release-behind, forge-degraded, vitals-degraded, sentinel): 81 pass. The forge's check runs the rest. To try it: bun test test/release-smoke.test.ts test/release-prove.test.ts.

🤖 Generated with Claude Code