Sentinel: release was degraded 178 minutes today: Release 0.7.479 is unproven: the public URL never answered as 0.7.479. It is proven again at 15:55 UTC. #10267

A failed proof for another version no longer keeps the release card open. I couldn't confirm this was the cause of the 178 minutes, because the sandbox can't read production's own proof file or logs.

What changed

• proofNeed in src/release/prove.ts takes the running version. It returns nothing when the last failed proof is for a different version.
• src/kernel/vitals.ts now passes the running version in, so the release card reflects only this process's own proof.
• The proof file is shared across versions and lists only the last proof, which can name a version that is no longer running. Before this change a failure recorded for 0.7.479 kept the card open after a roll-back or a newer boot, until some later proof overwrote it.
• Unstamped (dev) runs keep the old behaviour.
• test/release-prove.test.ts has a new test for this. The tsc run, the release-prove and vitals-release tests, and scripts/gate.ts all pass.

What I left alone

• I did not change the give-up and retry timings, which earlier fixes (#10168, #10222, #10242) already tuned.
• A version that really never answers at its public URL still shows the card. If 0.7.479 was itself running and unreachable for those 178 minutes, this change won't shorten it.

How to check Watch whether release stays out of the degraded list tomorrow. If it shows up again, read the production proof file (release/proof.json) and the release proof log lines for the version it names.

Closes #10248

A failed proof for another version no longer keeps the release card open. I couldn't confirm this was the cause of the 178 minutes, because the sandbox can't read production's own proof file or logs.

What changed

• proofNeed in src/release/prove.ts takes the running version. It returns nothing when the last failed proof is for a different version.
• src/kernel/vitals.ts now passes the running version in, so the release card reflects only this process's own proof.
• The proof file is shared across versions and lists only the last proof, which can name a version that is no longer running. Before this change a failure recorded for 0.7.479 kept the card open after a roll-back or a newer boot, until some later proof overwrote it.
• Unstamped (dev) runs keep the old behaviour.
• test/release-prove.test.ts has a new test for this. The tsc run, the release-prove and vitals-release tests, and scripts/gate.ts all pass.

What I left alone

• I did not change the give-up and retry timings, which earlier fixes (#10168, #10222, #10242) already tuned.
• A version that really never answers at its public URL still shows the card. If 0.7.479 was itself running and unreachable for those 178 minutes, this change won't shorten it.

How to check Watch whether release stays out of the degraded list tomorrow. If it shows up again, read the production proof file (release/proof.json) and the release proof log lines for the version it names.

The release card could stay on "Release 0.7.479 is unproven… It is proven again at 15:55 UTC" for hours after that time had passed. The sentinel's latest note shows it still there at 17:39 UTC, 174 minutes in the day. This pull fixes the two causes:

• The process stopped checking for good. When a process saw a newer release answer at the public URL, it marked its own version as superseded and never polled again. If that roll was then undone, the process kept running, and its last failure stayed on the card with a retry time already in the past. Now the pause after "superseded" lasts only GIVE_UP_MS, then proving starts again. This is in src/release/prove.ts.
• The card could name an older version's failure. src/kernel/vitals.ts and prove.ts now show only the failure of the version that is actually running.

Each fix has a new test in test/release-prove.test.ts that fails without it.

What to look at: the superseded-pause handling in src/release/prove.ts. Once this is merged and the next release goes out, the card should clear on the next successful check. If the 15:55-style line comes back, the sentinel will reopen the issue.

Checks: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts said the diff was too broad to pick tests, so I ran the 39 release and vitals test files by hand: 266 pass, 1 skipped, 1 failed. The failure was test/api-releases.test.ts › "each 200 carries an etag; a repeat gets 304…", which got 304 where it expected 200. It passes 3 out of 3 on its own, on this branch and on main alike, so it's timing-dependent under load and not caused by this change. I left it alone.

🤖 Generated with Claude Code

The release card could stay on "Release 0.7.479 is unproven… It is proven again at 15:55 UTC" for hours after that time had passed. The sentinel's latest note shows it still there at 17:39 UTC, 174 minutes in the day. This pull fixes the two causes:

• The process stopped checking for good. When a process saw a newer release answer at the public URL, it marked its own version as superseded and never polled again. If that roll was then undone, the process kept running, and its last failure stayed on the card with a retry time already in the past. Now the pause after "superseded" lasts only GIVE_UP_MS, then proving starts again. This is in src/release/prove.ts.
• The card could name an older version's failure. src/kernel/vitals.ts and prove.ts now show only the failure of the version that is actually running.

Each fix has a new test in test/release-prove.test.ts that fails without it.

The check's header-width rule failed on my own long comment in prove.ts, now wrapped. It also flagged comments main itself leaves over 160 columns: src/apps/feature_schedule.ts (two lines), test/landing.test.ts, test/looks.test.ts and test/credentials.test.ts. Those are wrapped in their own commit, comment text only, so the rule is green with main merged in. No test was changed to pass.

What to look at: the superseded-pause handling in src/release/prove.ts. Once this is merged and the next release goes out, the card should clear on the next successful check.

Checks: with main merged, bun scripts/gate.ts and bun x tsc --noEmit are clean. test/header-width.test.ts, test/release-prove.test.ts and test/vitals-release.test.ts pass (32 tests). In an earlier run of the 39 release and vitals test files, test/api-releases.test.ts › "each 200 carries an etag; a repeat gets 304…" failed once under load (304 where 200 was expected). It passes on its own on this branch and on main alike, so I left it alone.

🤖 Generated with Claude Code

The release card only shows a failed proof for the version that is actually running, so a failure kept for another version no longer holds the card open. Last time the public URL never answered as 0.7.479, and that card stayed up for 178 minutes.

The fix

• Proof checks: src/release/prove.ts now waits GIVE_UP_MS after a newer release answers before checking again. Before, it never checked again. If the process is still running after that wait, it was not replaced (for example, a roll the launcher undid), so it checks its own version again instead of leaving an old failure and a retry time that has already passed on screen for hours.
• The card: proofNeed(dataDir, running) returns no card when the kept failure is for a version other than the running one. A run from source with no version stamp (dev) keeps the old behaviour. vitals() passes running().version at src/kernel/vitals.ts:519.

Review follow-up: test/vitals-release.test.ts now has two cases that set FORGE_VERSION and call vitals():

• A failed proof kept for 0.7.479 while 0.7.478 runs gives no release card.
• The same proof while 0.7.479 runs (also tested as v0.7.479) still gives the card, linking /board.

Reverting line 519 to proofNeed(input.dataDir) makes the first case fail; with the fix, both test files pass (32 tests), and scripts/gate.ts and tsc --noEmit are clean.

Comments wrapped to fit the header-width check (in feature_schedule.ts and three test files) are in separate commits and don't change behaviour. This is a server-side fix with nothing new on a page or in the terminal, so there's no docs/features entry or terminal screen.

To try it: bun test test/vitals-release.test.ts test/release-prove.test.ts

🤖 Generated with Claude Code

The release card only shows a failed proof for the version that is actually running, so a failure kept for another version no longer holds the card open. Last time the public URL never answered as 0.7.479, and that card stayed up for 178 minutes.

The fix

• Proof checks: src/release/prove.ts now waits GIVE_UP_MS after a newer release answers before checking again. Before, it never checked again. If the process is still running after that wait, it was not replaced (for example, a roll the launcher undid), so it checks its own version again instead of leaving an old failure and a retry time that has already passed on screen for hours.
• The card: proofNeed(dataDir, running) returns no card when the kept failure is for a version other than the running one. A run from source with no version stamp (dev) keeps the old behaviour. vitals() passes running().version at src/kernel/vitals.ts:519.

Tests

• test/release-prove.test.ts covers proofNeed directly.
• test/vitals-release.test.ts drives vitals() with FORGE_VERSION set:

- A failed proof kept for 0.7.479 while 0.7.478 runs gives no release card. - The same proof while 0.7.479 runs still gives the card, linking /board.

• Reverting line 519 to proofNeed(input.dataDir) makes the first case fail.

Merge with main: the only conflicts were the earlier comment rewraps in feature_schedule.ts, credentials.test.ts, landing.test.ts and looks.test.ts. main had already rewrapped the same comments, so I took its version. The pull now changes only the four files of the fix.

After the merge, scripts/gate.ts and tsc --noEmit are clean, and test/release-prove.test.ts plus all test/vitals* files pass (50 tests).

To try it: bun test test/vitals-release.test.ts test/release-prove.test.ts

🤖 Generated with Claude Code

The release card only shows a failed proof for the version that is actually running, so a failure kept for another version no longer holds the card open. Last time the public URL never answered as 0.7.479, and that card stayed up for 178 minutes.

The fix

• Proof checks: src/release/prove.ts now waits GIVE_UP_MS after a newer release answers before checking again. Before, it never checked again. If the process is still running after that wait, it was not replaced (for example, a roll the launcher undid), so it checks its own version again instead of leaving an old failure and a retry time that has already passed on screen for hours.
• The card: proofNeed(dataDir, running) returns no card when the kept failure is for a version other than the running one. A run from source with no version stamp (dev) keeps the old behaviour. vitals() passes running().version at src/kernel/vitals.ts:519.

Tests

• test/release-prove.test.ts covers proofNeed directly.
• test/vitals-release.test.ts drives vitals() with FORGE_VERSION set:

- A failed proof kept for 0.7.479 while 0.7.478 runs gives no release card. - The same proof while 0.7.479 runs still gives the card, linking /board.

• Reverting line 519 to proofNeed(input.dataDir) makes the first case fail.

Fixes for two failures main already has. The check failed on two tests this pull doesn't touch; both come from recent merges to main:

• Boot test (#10274): src/http/plugin_routes.ts reads plugins/opencode/forge.ts at import, and the Dockerfile copies it into the image. test/release-boot.test.ts built its copy of the image without it, so the boot exited with ENOENT. The test was wrong, not the code: it now copies plugins/opencode the way the Dockerfile does. The test keeps its name, which still says "src/ alone".
• Unreached module (#10275): src/spawn/pod.ts has no caller yet; it comes from pull #3743's branch. It is now on scripts/unreached-allow.json, and test/unreached.test.ts fails once the module is reached, so the entry has to come off when its caller lands.

The merge with main took main's rewraps of the comments in feature_schedule.ts and three test files.

scripts/gate.ts and tsc --noEmit are clean. test/unreached.test.ts, test/release-boot.test.ts, test/release-prove.test.ts, test/header-width.test.ts and all test/vitals* files pass.

To try it: bun test test/vitals-release.test.ts test/release-prove.test.ts test/release-boot.test.ts test/unreached.test.ts

🤖 Generated with Claude Code

Release proof: a failed proof now shows as a card only to the process running that version. A process that saw a newer release answer proves its own version again later instead of standing silent.

• src/release/prove.ts:

- proofNeed(dataDir, running) returns nothing when the kept failure is for a different version than the one running. Before, a failure kept for an older version stood on the board for hours after a roll or a newer boot (the 178-minute "release degraded" in #10248). - A version seen replaced by a newer one is skipped only until GIVE_UP_MS has passed, no longer forever. A process still running after that was not replaced (the launcher undid the roll), so it proves its version again.

• src/kernel/vitals.ts: passes the running version to proofNeed.

Tests:

• test/release-prove.test.ts and test/vitals-release.test.ts cover both cases: another version's failure is ignored, the running version's failure is shown, and a replaced version is proved again after the window.
• test/release-boot.test.ts now copies plugins/opencode into the image tree, as the Dockerfile does. src/http/plugin_routes.ts reads that file at boot (#10274), so the boot test was red on main.
• test/crew-loop.test.ts: the suite's crew starts with tickMs: 3_600_000. The first crew tick runs the autonomy pass, which queues its own feedback run on a pull whose verdict still stands. On a loaded shard that tick landed inside "the third changes-requested hands the pull to a person" and changed the counts it checks. Running one autonomy tick between rounds two and three reproduces the failure every time. The test was wrong, not the code; no test in that suite relies on the tick.
• scripts/unreached-allow.json lists src/spawn/pod.ts, which is on main from #10275 and has no caller yet.

Not changed: "a child forge spawned into a cell serves its app" failed on the last check. The child crashed while writing git hooks at boot (src/git/hooks.ts:36, inside reconcile). Then its spawn directory vanished from under the parent (kid.json ENOENT), with 16% of the disk free. This pull touches neither spawn, cells, hooks nor reconcile. The log cuts off the child's actual error, and my sandbox skips the cell suites, so it is not reproduced here. If it fails again on this push, it needs its own issue.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass, and the covering files pass (75 pass, 0 fail, 1 cell e2e skipped here). Those files are crew-loop, release-prove, vitals-release, release-boot, spawn-pod, spawn-cell-e2e and vitals-held-agent.

To try it: on the preview, /board shows no "release" card while the proof kept on disk is for a version other than the running one.

🤖 Generated with Claude Code