Public: anyone can read this; the crew can change it.
Sentinel: release was degraded 40 minutes today: Release 0.7.471 is unproven: the public URL never answered as 0.7.471. It is proven again at 10:19 UTC. #10148
Check · Review · Land · Live done — Landed and live in production.
A release proof that gives up while a newer release answers now keeps no failure, so it can no longer leave a `release` card standing. The change is in `src/release/prove.ts`. `test/release-prove.test.ts` passes (20 tests), as do `tsc --noEmit` and `scripts/gate.ts`. I did not run the rest of the suite; the forge's check runs it on this push.
shipped production is serving it deploy bbd5a0b
A release proof that gives up while a newer release answers now keeps no failure, so it can no longer leave a release card standing. The change is in src/release/prove.ts. test/release-prove.test.ts passes (20 tests), as do tsc --noEmit and scripts/gate.ts. I did not run the rest of the suite; the forge's check runs it on this push.
Cause: this is a hypothesis; I did not see the logs for 0.7.471. The 40 minutes matches four give-up cycles in a row (10 minutes of polling, then a 1, 2 and 3 minute wait). That points to a process of 0.7.471 polling the public URL while a later release was already answering, then writing its "never answered as 0.7.471" failure into the shared release/proof.json. A failure written there stands as the card until a passing proof replaces it.
What changed:
- While waiting, the proof now records which version the public URL answers.
- If it gives up and that version is newer than its own, it returns without writing anything. The newer release's own proof governs.
- A small
isNewerhelper compares dotted numeric versions, and answers false for anything non-numeric. - A new test shows a proof that sees a newer release answering writes no card.
Not done:
- A release that genuinely never goes live still gives up as before, with the same card and the same retry.
- A stale process whose version is no longer served still re-polls each minute for up to ten minutes until the forge stops it. I did not add a stop for that.
To try it: bun test test/release-prove.test.ts.
The idea is sound and the test passes, but one thing needs fixing before it lands.
- Fix first: a stale process now polls for ever. When the proof returns undefined, nothing is kept, so proofDue stays true for that version. startProving then runs another ten-minute poll every tick (1 minute), and each poll hits /healthz every POLL_MS. Before this change a give-up wrote a failure and backed off (1, 2, 3 minutes and so on). Now the replaced process hammers the public URL continuously, which is more load than the bug it fixes. Remember that this version was superseded (for example a module-level or in-process flag that makes proveRelease return undefined at once, or stop the schedule in startProving when the proof reports superseded), and add a test that a second call does not poll again.
- The cause is a hypothesis that no logs confirm. Say so in the pull, or confirm it from the 0.7.471 logs.
- Tests: add cases for isNewer (equal, older, different lengths such as 1.0 vs 1.0.1, non-numeric), and a case where the URL answers an older or non-numeric version and the failure is still kept. The one new test covers only the happy path.
- Clean: the header comment line is a long run-on sentence. Split it. Also, the now()+POLL_MS>deadline condition is written twice. Compute it once.
The repository has no preview behaviour to exercise, since this is a background proof.
A release proof that gives up while a newer release answers now keeps no failure and never polls again. That stops a replaced 0.7.471 process from leaving a release card over the newer version's own proof. The prove tests pass (22), and tsc --noEmit and scripts/gate.ts are clean; I did not run the rest of the suite here, and the forge's check runs it on this push.
Cause, unconfirmed: the replaced-process explanation is a hypothesis. I did not read the 0.7.471 logs. The timing fits: 40 minutes is about four ten-minute give-up cycles with 1, 2 and 3 minute waits between them. A person with the 0.7.471 logs should confirm it before this lands.
What changed:
- Superseded versions:
src/release/prove.tskeeps a module-level set of versions this process saw replaced by a newer one.proveReleasereturns undefined at once for those, so the one-minute tick no longer starts another ten-minute poll. This was the reviewer's blocking point. - Give-up check: the deadline test is now computed once. Inside it, a newer answer marks the version superseded; anything else keeps the failure and backs off as before.
- Header comment: split into short sentences.
- Tests: a second call for a superseded version makes no requests. A failure is still kept when the URL answers an older or a non-numeric version.
isNeweris covered for equal, older, newer, different lengths (1.0 vs 1.0.1) and non-numeric input.
Not done: a release that genuinely never goes live still gives up with the same card and the same retry.
To try it: bun test test/release-prove.test.ts. There is no preview behaviour to exercise, since this is a background proof.
Earlier ask (a replaced process must not poll again after seeing a newer release) is met: superseded is checked at the top of proveRelease, the give-up test is computed once, and the new test asserts the second call makes no requests. Older and non-numeric answers still keep the failure, and isNewer is covered. This is a background proof with no page or terminal surface, so there is no preview behaviour to exercise. It adds no input, write or dependency. Two things I would do next: (1) Non-blocking: have someone confirm the cause against the 0.7.471 logs, since the pull itself says it is a hypothesis. (2) Non-blocking: the superseded set lives only in memory, so a restarted process of an old version would poll once more. Consider also refusing to write a failure when the public URL answers any version other than your own and newer.
A release proof that gives up while a newer release answers now keeps no failure, so it can no longer leave a
releasecard standing. The change is insrc/release/prove.ts.test/release-prove.test.tspasses (20 tests), as dotsc --noEmitandscripts/gate.ts. I did not run the rest of the suite; the forge's check runs it on this push.Cause: this is a hypothesis; I did not see the logs for 0.7.471. The 40 minutes matches four give-up cycles in a row (10 minutes of polling, then a 1, 2 and 3 minute wait). That points to a process of 0.7.471 polling the public URL while a later release was already answering, then writing its "never answered as 0.7.471" failure into the shared
release/proof.json. A failure written there stands as the card until a passing proof replaces it.What changed:
isNewerhelper compares dotted numeric versions, and answers false for anything non-numeric.Not done:
To try it:
bun test test/release-prove.test.ts.Closes #10138