Public: anyone can read this; the crew can change it.
The release proof fails on the first try after every restart, never retries, and leaves its probe forge behind #12368
Check · Review · Land · Live done — Landed and live in production.
The release proof now waits for a settled forge and walks the stranger's journey up to three times before it records a failure. A walk that fails also removes its probe forge, or names it if the removal fails. I did not run the full suite or try a restart under load on the stage rig.
shipped production is serving it deploy d73d768
7 earlier comments · 1 event
Merged main into the branch and pushed. The only conflict was in src/release/prove.ts, in the finish helper, and I kept both sides. It now has main's measured footprint variable and my wider signature (tries, leftover probe forges).
After the merge, the tsc type check failed in src/kernel/vitals.ts. main had changed need() to take the kind first, and my probe-forge card still used the old call. I updated that call to the new shape, passing "todo" as the kind. The same merge had also made the probe-card test fail.
The gate and the type check are clean. The four test files covering the change pass (62 tests): release-prove, release-journey, release-now and vitals-release. The forge runs the full check on the push.
Nothing about the pull's behaviour changed from the earlier description.
I found no code fault in this pull behind the red check, so I merged current main (6 new commits, clean merge) and pushed again to re-run it. I haven't seen the new check's verdict yet. This is the pull description, unchanged in substance.
What changed
- The release proof waits for the forge to report ready and its wake-up work to settle before the first walk.
- After a failed walk it tries again a few minutes later, twice at most, before recording a failure. The recorded line says how many walks it took, and
/releasesshows it. - A failed walk removes its probe forge just as a passed one does. If the removal fails, the failure names the forge. It also appears in the needs-you stack with a Remove button.
- An admin can run the proof again from the release's page and through MCP (
release_now). - The 20-second limit is unchanged, so a stranger's forge that is slow on a settled host still fails.
What the failed check showed
- The check log named three failures, and the rest of the list was cut off:
- ssh-port-taken: "main.ts keeps serving / when the SSH port is taken" times out at 5 s here, and it does the same on plain origin/main, so it isn't this pull.
- Cell suites: "a Small child in a real cell…" belongs to a family of cell suites the check log says are red on main too, so they aren't counted against this pull.
- preview-spec: "a deployed preview's findings are…" passes when run alone here, so it looks like a load flake in the check run.
- I couldn't tell whether the other failures in the list were these same cases.
- After the merge I ran
gate,tscand the tests covering this change (release-prove,release-now,release-journey,vitals-release,preview-spec, the autonomy, asks-page and releases-page tests). All passed.scripts/fast.tsreturned exit 2 (diff too broad to name), so I chose those files myself. - I did not run the full suite or the cell and unix-socket suites, which this sandbox skips. The forge's check covers them.
What I left out
- I did not change the SSH-port or cell tests, because they are red on
main.
How to try it
- Restart under load, then open
/releases#proof. It should show a passed proof and the number of walks it took, with noprobe-forge left on the host. - Admins have a re-run action on the same page, and
release_nowdoes the same through MCP.
This pull makes the release proof show when it started and stops it after 15 minutes with a recorded verdict. That covers the 0.7.595 comment. The branch already had the original four asks (settle before the walk, up to three tries, probe cleanup, re-run).
Changed (new in this push)
- Still-proving state. While a proof runs, the forge keeps
release/proving.jsonwith the version and start time. The home page's live-version figure then reads "still proving, started HH:MM UTC" until a verdict replaces it. - Ceiling.
HANG_MSis now 15 minutes in total, down from 25, so it is shorter than the gap between releases. 15 minutes is my pick, not a number from the issue. A proof that outruns it is aborted and recorded as a failed,hungproof ("the proof started at HH:MM UTC did not finish in 15 minutes"). Before, it only loggedrelease proof hungand left the old verdict standing. The next tick tries again within a minute, and the release card says the proof ran out of time. - Test, doc and check.
test/release-prove.test.tscovers the hang verdict, the proving marker being set and cleared, and the ceiling.docs/features/release-heartbeat.mdhas a short section on it. The 37 tests in that file pass andtsc --noEmitis clean.scripts/gate.tsprinted nothing, so I took it as clean.
Left out
- No test renders the home page figure; I only changed its string.
- I did not run the stage-rig restart-under-load check from the issue's "Done when"; that needs the real host.
- I did not run the rest of the suite; the forge's check does that on this push.
Try it: run bun test test/release-prove.test.ts. On a preview, the home page shows "still proving, started …" while a proof runs.
I merged main into this branch with no conflicts and pushed it (merge commit 1798d88ea).
The release proof now retries once when it fails on the first try. src/release/prove.ts and src/release/journey.ts hold the retry. The release-now tool, the home flow, releases and asks pages, the heartbeat doc and docs/MCP.md follow it.
- Passing:
bun x tsc --noEmitis clean andbun scripts/design-lint.tsis clean. The four release test files pass:test/release-journey.test.ts,test/release-now.test.ts,test/release-prove.test.tsandtest/vitals-release.test.ts(63 tests). - One flaky run: the first run had one failure, with a
/exploresmoke timeout in the log. I ran the files again on their own and all 63 passed. That first run overlapped with a full-suite run I started by mistake and then stopped, so I read the failure as load, not a regression. - Fast script:
scripts/fast.tsexited 2 becausesrc/kernel/vitals.tschanged, so I ran the release tests by hand. - Not run: I did not run the full suite; the forge runs it on the push.
- Gate:
scripts/gate.tsprinted nothing. I didn't check its exit code, so I can't say it passed.
I regenerated the release_now row of the tool table in docs/MCP.md and pushed it. That was the only red this pull caused: test/mcp-table.test.ts failed because the doc was stale. It passes now, and bun x tsc --noEmit and bun scripts/gate.ts print nothing. I ran no other test files.
The check log was cut off, so I only saw 7 failures in total. I saw the check's report that six test files are red on main at cc21c84 too, so they aren't counted against this pull: the spawn-cell and small-cell tests, the data browser, the tenancy cell test and the mirror worker. I assume the other failures are those six, but I didn't confirm it.
scripts/fast.ts exited 2 (the diff reaches src/kernel/vitals.ts and is too broad to name tests), so I ran only the one failing file.
Changes requested. The retry, settle and probe-removal work is sound, but these must be fixed in one round:
- FIRST:
HANG_MSis 15 min, the same as the old GIVE_UP_MS+5 min, but the proof now also does up to 3 walks (settle up to 90 s + a 20 s walk + 2 min waits, roughly 8-12 min more) after smoke polling that can itself take 10 min. A release that is slow to come up will be killed and recorded ashungwhile it is still working normally. Make the ceiling cover the worst case (give-up + 3 x (settle + walk) + 2 waits) with a margin, or budget the walks inside it, and test the worst case rather thanexpect(HANG_MS).toBeLessThanOrEqual(15*60_000), which proves nothing. release/proving.jsonis never cleared on boot. If the forge restarts mid-proof and the next proof is not yet due (backoff), the home page says 'still proving, started HH:MM UTC' indefinitely for a failed version. Clear it whenstartProvingstarts (or ignore a marker older than HANG_MS) and add a test.- Tests are missing for behaviour added: the home page 'still proving' figure (
proofWords), the/releaseswalk count inproofCard, and the asks-page Remove button forprobe:cards. Each needs a test that fails without it. - Both doors: an admin can 'Prove it again' on /releases and over MCP, but there is no terminal action for it (src/tui/ releases screen), and the Remove card's page has no terminal equivalent noted. Add the terminal entry where AGENTS.md says, or show that it exists.
- Scope: the proving marker, home-page figure and hang verdict go beyond the issue's three asks (first-try failure, no retry, leftover probe). Keep them only with their tests; otherwise split them out. Pipeline-first direction: this is not on the listed pipeline items, so keep it small.
- Principle 8 (plain words): 'proofs so far' and 'walks in the last one' on /releases are jargon; say 'attempts' in one consistent word.
- docs/features/release-heartbeat.md should state the real total ceiling once fixed, and the
proveargument of release_now should be documented in the doc there too (it only lives in the tool text).
I've addressed all seven points and pushed. The tests I ran are green: the new proof-words test and the release, asks, terminal-releases, mcp-table and home-flow tests. tsc, scripts/gate.ts and design-lint are clean. I did not run the full suite or the cell tests; the forge's check will.
What changed
- Hang ceiling.
HANG_MSis now 30 minutes, derived from a newWORST_CASE_MS. That is 10 min of give-up polling, 3 min of smoke checks, 3 × (90 s settle + 20 s walk), two 2-min waits and 5 min of margin, about 27.5 min in all. A test assertsHANG_MSis at least the sum, replacing the old 15-minute check. The 3 min for smoke is my estimate of the heavy-page watch, not a measured figure. - Stale marker.
- startProving clears proving.json when it starts.
- provingSince ignores a marker older than HANG_MS.
- A tick only writes the marker when a proof is due, so a boot whose retry is still in backoff says nothing.
- A test covers each of these.
- Missing tests.
- proofWords is now exported and covered in test/release-proof-words.test.ts, which also covers the /releases attempt counts in proofCard.
- test/asks-forge-alert.test.ts now checks that a probe: card has a Remove button and other causes don't.
- Terminal door.
pon the terminal releases screen ("prove it again") uses the same gate as the page: a person who may read, never an agent. The test covers it with a failed proof, no failed proof, and an agent. The probe card's Remove already has a terminal equivalent,x(remove one) on the forges screen, and the doc says so. - Plain words. /releases now reads "2 failed attempts so far; the last one tried 3 times".
- Docs.
docs/features/release-heartbeat.mdstates the real ceiling and documentsproveonrelease_now: admin scope, never an agent, and it answersinvalidwhen no failed proof waits.
Scope (point 5). I kept the marker, the home figure and the hang verdict, since they now have tests. I did not split them out, as the hang verdict is what stops a failed proof standing for hours.
The earlier red tests in the cell e2e files were red on main too, so I left them alone.
I merged main into the branch and pushed it. The gate is clean and tsc has no errors. The four test files I ran pass: asks-forge-alert, release-proof-words, release-prove and release-journey. I did not run the full suite; the forge's check does that on push.
The conflict was in forgeCard in src/web/studio/asks_page.ts. main now passes a ForgeNeed that carries its own line, so I used ${d.line} for the sentence. I kept this pull's "Remove" button, shown on probe: causes, next to Health.
Follow-on fixes the merge needed:
test/asks-forge-alert.test.tsbuilt the oldDegradedshape; it now builds a{ line, cause, href }record.proofCardinsrc/web/pages/releases.tstakes the data directory instead of a wholeCtx. That removes theas nevercasts intest/release-proof-words.test.ts, which the gate now refuses.- I marked the one real wait in
test/release-prove.test.tsas// real-timer:, and rewrapped a 110-column comment insrc/tui/screens/releases.ts.
I pushed the merge commit before the gate was clean, so the branch briefly had two gate violations. The follow-up commit fixed them.
I did not rewrite the real-timer wait in test/release-prove.test.ts to use an injected clock. The marker is the minimal fix.
The dead-code test (test/unused-exports.test.ts) was red because of this pull, and I've fixed it and pushed.
WORST_CASE_MS in src/release/prove.ts was exported, but only test/release-prove.test.ts used it. I removed it. The ceiling test already computes the same sum from GIVE_UP_MS, the walks and the 3-minute smoke allowance, so it still checks HANG_MS against the worst healthy proof. It no longer adds the 5-minute margin that WORST_CASE_MS had.
tsc, scripts/gate.ts, test/unused-exports.test.ts and test/release-prove.test.ts all pass locally.
The other reds in the log I did not look at. test/mirror-worker.test.ts and test/boundary-tenancy-cell.test.ts are red on main as well, and the check log didn't name the failing tests beyond the first two. One of those was the S3 mirror bucket-path test. The other was a "surfaces adversarial" test about the dashboard's Apps and O… Neither touches what this pull changed. The forge's next check will show whether they recur.
Earlier asks are met: the hang ceiling is now derived and tested; the stale proving marker is cleared at boot and ignored once older than the ceiling; proofWords, the probe Remove card and the proof card wording have tests; the terminal door is p on the releases screen, gated as POST /releases/prove is, and it is tested; the how-to is in docs/features/release-heartbeat.md, not the README; the unused export is removed. No new security finding on the changed lines: the probe name is regex-checked before it becomes a card href, and the MCP prove path requires an admin-scope token and a person who may roll back. Non-blocking: the MCP path uses the stricter mayRollBack while the page and terminal use mayRead. Pick one gate for the three doors. I did not open the preview.
The release proof now waits for a settled forge and walks the stranger's journey up to three times before it records a failure. A walk that fails also removes its probe forge, or names it if the removal fails. I did not run the full suite or try a restart under load on the stage rig. The release-proof, journey, releases page, vitals, MCP-size, MCP write-scope and release-now tests pass, as do
tsc, the gate and design-lint./readyzto answer 200 and the event loop's lag to fall under 250 ms. If the forge never settles, that try fails with "had not settled". A failed walk is tried again 2 minutes later, three walks at most.proof.jsonand therelease prooflog line recordtries, and/releasesshows the walk count. The sign-up-limit refusal and a probe forge left behind both stop further tries, since another walk would only make another forge. The 20 s forge-ready budget is untouched. I raised the proof's hang limit (HANG_MS) from 15 to 25 minutes so three walks fit under it.proof.jsonasleftover. Each leftover shows in the needs-you stack as a forge card with a Remove button that opens that forge's existing remove page, where one click removes it. The card clears once a later proof leaves none./releasesalready existed. It now has an MCP door:release_now {prove: true}on the existing tool, so there is no new tool and no ceiling raised. It is admin-scope only, refuses agents, and refuses when no proof has failed.I did not touch the 20 s budget. If a settled host takes longer than that, the proof stays red.
The
/releasesbutton still runs the proof at the next tick, within a minute, not instantly.To try it, call
release_now {prove: true}with an admin token, or open/releaseswhile the proof is red.Closes #12367