Public: anyone can read this; the crew can change it.

The release proof fails on the first try after every restart, never retries, and leaves its probe forge behind #12368

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The release proof now waits for a settled forge and walks the stranger's journey up to three times before it records a failure. A walk that fails also removes its probe forge, or names it if the removal fails. I did not run the full suite or try a restart under load on the stage rig.

Notes

shipped production is serving it deploy 7612a18

opened this

The release proof now waits for a settled forge and walks the stranger's journey up to three times before it records a failure. A walk that fails also removes its probe forge, or names it if the removal fails. I did not run the full suite or try a restart under load on the stage rig. The release-proof, journey, releases page, vitals, MCP-size, MCP write-scope and release-now tests pass, as do tsc, the gate and design-lint.

  • Settle and retry: before each walk the proof waits up to 90 s for /readyz to answer 200 and the event loop's lag to fall under 250 ms. If the forge never settles, that try fails with "had not settled". A failed walk is tried again 2 minutes later, three walks at most. proof.json and the release proof log line record tries, and /releases shows the walk count. The sign-up-limit refusal and a probe forge left behind both stop further tries, since another walk would only make another forge. The 20 s forge-ready budget is untouched. I raised the proof's hang limit (HANG_MS) from 15 to 25 minutes so three walks fit under it.
  • Probe cleanup: the walk already removed its forge on both pass and fail, but a failed removal was silent. Now the failure reads "the probe forge probe-xxxx could not be removed", and the name is kept in proof.json as leftover. Each leftover shows in the needs-you stack as a forge card with a Remove button that opens that forge's existing remove page, where one click removes it. The card clears once a later proof leaves none.
  • Re-run: the "Prove it again now" button on /releases already existed. It now has an MCP door: release_now {prove: true} on the existing tool, so there is no new tool and no ceiling raised. It is admin-scope only, refuses agents, and refuses when no proof has failed.

I did not touch the 20 s budget. If a settled host takes longer than that, the proof stays red.

The /releases button still runs the proof at the next tick, within a minute, not instantly.

To try it, call release_now {prove: true} with an admin token, or open /releases while the proof is red.

Closes #12367

9 earlier comments · 1 event
commented

The release proof now waits for a settled forge and walks the stranger's journey up to three times before it records a failure. A walk that fails also removes its probe forge, or names it if the removal fails. I did not run the full suite or try a restart under load on the stage rig. The release-proof, journey, releases page, vitals, MCP-size, MCP write-scope and release-now tests pass, as do tsc, the gate and design-lint.

  • Settle and retry: before each walk the proof waits up to 90 s for /readyz to answer 200 and the event loop's lag to fall under 250 ms. If the forge never settles, that try fails with "had not settled". A failed walk is tried again 2 minutes later, three walks at most. proof.json and the release proof log line record tries, and /releases shows the walk count. The sign-up-limit refusal and a probe forge left behind both stop further tries, since another walk would only make another forge. The 20 s forge-ready budget is untouched. I raised the proof's hang limit (HANG_MS) from 15 to 25 minutes so three walks fit under it.
  • Probe cleanup: the walk already removed its forge on both pass and fail, but a failed removal was silent. Now the failure reads "the probe forge probe-xxxx could not be removed", and the name is kept in proof.json as leftover. Each leftover shows in the needs-you stack as a forge card with a Remove button that opens that forge's existing remove page, where one click removes it. The card clears once a later proof leaves none.
  • Re-run: the "Prove it again now" button on /releases already existed. It now has an MCP door: release_now {prove: true} on the existing tool, so there is no new tool and no ceiling raised. It is admin-scope only, refuses agents, and refuses when no proof has failed.

I did not touch the 20 s budget. If a settled host takes longer than that, the proof stays red.

The /releases button still runs the proof at the next tick, within a minute, not instantly.

To try it, call release_now {prove: true} with an admin token, or open /releases while the proof is red.

commented

The release proof now waits for a settled forge before its first walk. If a walk fails it tries again, up to three walks in all, and it names any probe forge it could not remove. The last push changes only comment wrapping.

What changed

  • Failing gate fixed: the gate's "comment lines wrap at 100 columns" rule failed on eight comment lines I had added. They are rewrapped or shortened in src/release/prove.ts, src/release/journey.ts and src/kernel/vitals.ts. No code changed.
  • Looks rule: the other gate failure, the missing "Looks" line, is covered by a No look: line in the new commit message, since no page renders differently.
  • Proof behaviour:

- The first walk waits for the forge to report ready and its event loop to calm. - A failed walk is tried again a few minutes later, twice at most, before the failure is recorded. - A passed walk, a sign-up-limit refusal, or a probe forge left behind ends the tries. - A probe forge the proof could not remove shows as a needs-you card with a Remove page one tap away.

  • Manual retry: from /releases, an admin can press "Prove it again now" when the proof is red, or call release_now with prove true.

Checked before pushing

  • bun scripts/gate.ts passes and bun x tsc --noEmit reports no errors.
  • The four test files covering the change pass (62 tests): release-prove, release-journey, release-now and vitals-release. The forge runs the full check on the push.

Left out

  • The retry count and waits are fixed constants, not settings.

To try it

  • Open /releases on the preview as an admin while the proof is red and press "Prove it again now".
commented

Merged main into the branch and pushed. The only conflict was in src/release/prove.ts, in the finish helper, and I kept both sides. It now has main's measured footprint variable and my wider signature (tries, leftover probe forges).

After the merge, the tsc type check failed in src/kernel/vitals.ts. main had changed need() to take the kind first, and my probe-forge card still used the old call. I updated that call to the new shape, passing "todo" as the kind. The same merge had also made the probe-card test fail.

The gate and the type check are clean. The four test files covering the change pass (62 tests): release-prove, release-journey, release-now and vitals-release. The forge runs the full check on the push.

Nothing about the pull's behaviour changed from the earlier description.

commented

I found no code fault in this pull behind the red check, so I merged current main (6 new commits, clean merge) and pushed again to re-run it. I haven't seen the new check's verdict yet. This is the pull description, unchanged in substance.

What changed

  • The release proof waits for the forge to report ready and its wake-up work to settle before the first walk.
  • After a failed walk it tries again a few minutes later, twice at most, before recording a failure. The recorded line says how many walks it took, and /releases shows it.
  • A failed walk removes its probe forge just as a passed one does. If the removal fails, the failure names the forge. It also appears in the needs-you stack with a Remove button.
  • An admin can run the proof again from the release's page and through MCP (release_now).
  • The 20-second limit is unchanged, so a stranger's forge that is slow on a settled host still fails.

What the failed check showed

  • The check log named three failures, and the rest of the list was cut off:

- ssh-port-taken: "main.ts keeps serving / when the SSH port is taken" times out at 5 s here, and it does the same on plain origin/main, so it isn't this pull. - Cell suites: "a Small child in a real cell…" belongs to a family of cell suites the check log says are red on main too, so they aren't counted against this pull. - preview-spec: "a deployed preview's findings are…" passes when run alone here, so it looks like a load flake in the check run.

  • I couldn't tell whether the other failures in the list were these same cases.
  • After the merge I ran gate, tsc and the tests covering this change (release-prove, release-now, release-journey, vitals-release, preview-spec, the autonomy, asks-page and releases-page tests). All passed. scripts/fast.ts returned exit 2 (diff too broad to name), so I chose those files myself.
  • I did not run the full suite or the cell and unix-socket suites, which this sandbox skips. The forge's check covers them.

What I left out

  • I did not change the SSH-port or cell tests, because they are red on main.

How to try it

  • Restart under load, then open /releases#proof. It should show a passed proof and the number of walks it took, with no probe- forge left on the host.
  • Admins have a re-run action on the same page, and release_now does the same through MCP.
commented

This pull makes the release proof show when it started and stops it after 15 minutes with a recorded verdict. That covers the 0.7.595 comment. The branch already had the original four asks (settle before the walk, up to three tries, probe cleanup, re-run).

Changed (new in this push)

  • Still-proving state. While a proof runs, the forge keeps release/proving.json with the version and start time. The home page's live-version figure then reads "still proving, started HH:MM UTC" until a verdict replaces it.
  • Ceiling. HANG_MS is now 15 minutes in total, down from 25, so it is shorter than the gap between releases. 15 minutes is my pick, not a number from the issue. A proof that outruns it is aborted and recorded as a failed, hung proof ("the proof started at HH:MM UTC did not finish in 15 minutes"). Before, it only logged release proof hung and left the old verdict standing. The next tick tries again within a minute, and the release card says the proof ran out of time.
  • Test, doc and check. test/release-prove.test.ts covers the hang verdict, the proving marker being set and cleared, and the ceiling. docs/features/release-heartbeat.md has a short section on it. The 37 tests in that file pass and tsc --noEmit is clean. scripts/gate.ts printed nothing, so I took it as clean.

Left out

  • No test renders the home page figure; I only changed its string.
  • I did not run the stage-rig restart-under-load check from the issue's "Done when"; that needs the real host.
  • I did not run the rest of the suite; the forge's check does that on this push.

Try it: run bun test test/release-prove.test.ts. On a preview, the home page shows "still proving, started …" while a proof runs.

commented

I merged main into this branch with no conflicts and pushed it (merge commit 1798d88ea).

The release proof now retries once when it fails on the first try. src/release/prove.ts and src/release/journey.ts hold the retry. The release-now tool, the home flow, releases and asks pages, the heartbeat doc and docs/MCP.md follow it.

  • Passing: bun x tsc --noEmit is clean and bun scripts/design-lint.ts is clean. The four release test files pass: test/release-journey.test.ts, test/release-now.test.ts, test/release-prove.test.ts and test/vitals-release.test.ts (63 tests).
  • One flaky run: the first run had one failure, with a /explore smoke timeout in the log. I ran the files again on their own and all 63 passed. That first run overlapped with a full-suite run I started by mistake and then stopped, so I read the failure as load, not a regression.
  • Fast script: scripts/fast.ts exited 2 because src/kernel/vitals.ts changed, so I ran the release tests by hand.
  • Not run: I did not run the full suite; the forge runs it on the push.
  • Gate: scripts/gate.ts printed nothing. I didn't check its exit code, so I can't say it passed.
commented

I regenerated the release_now row of the tool table in docs/MCP.md and pushed it. That was the only red this pull caused: test/mcp-table.test.ts failed because the doc was stale. It passes now, and bun x tsc --noEmit and bun scripts/gate.ts print nothing. I ran no other test files.

The check log was cut off, so I only saw 7 failures in total. I saw the check's report that six test files are red on main at cc21c84 too, so they aren't counted against this pull: the spawn-cell and small-cell tests, the data browser, the tenancy cell test and the mirror worker. I assume the other failures are those six, but I didn't confirm it.

scripts/fast.ts exited 2 (the diff reaches src/kernel/vitals.ts and is too broad to name tests), so I ran only the one failing file.

Changes requested — before the latest push

Changes requested. The retry, settle and probe-removal work is sound, but these must be fixed in one round:

  1. FIRST: HANG_MS is 15 min, the same as the old GIVE_UP_MS+5 min, but the proof now also does up to 3 walks (settle up to 90 s + a 20 s walk + 2 min waits, roughly 8-12 min more) after smoke polling that can itself take 10 min. A release that is slow to come up will be killed and recorded as hung while it is still working normally. Make the ceiling cover the worst case (give-up + 3 x (settle + walk) + 2 waits) with a margin, or budget the walks inside it, and test the worst case rather than expect(HANG_MS).toBeLessThanOrEqual(15*60_000), which proves nothing.
  2. release/proving.json is never cleared on boot. If the forge restarts mid-proof and the next proof is not yet due (backoff), the home page says 'still proving, started HH:MM UTC' indefinitely for a failed version. Clear it when startProving starts (or ignore a marker older than HANG_MS) and add a test.
  3. Tests are missing for behaviour added: the home page 'still proving' figure (proofWords), the /releases walk count in proofCard, and the asks-page Remove button for probe: cards. Each needs a test that fails without it.
  4. Both doors: an admin can 'Prove it again' on /releases and over MCP, but there is no terminal action for it (src/tui/ releases screen), and the Remove card's page has no terminal equivalent noted. Add the terminal entry where AGENTS.md says, or show that it exists.
  5. Scope: the proving marker, home-page figure and hang verdict go beyond the issue's three asks (first-try failure, no retry, leftover probe). Keep them only with their tests; otherwise split them out. Pipeline-first direction: this is not on the listed pipeline items, so keep it small.
  6. Principle 8 (plain words): 'proofs so far' and 'walks in the last one' on /releases are jargon; say 'attempts' in one consistent word.
  7. docs/features/release-heartbeat.md should state the real total ceiling once fixed, and the prove argument of release_now should be documented in the doc there too (it only lives in the tool text).
commented

I've addressed all seven points and pushed. The tests I ran are green: the new proof-words test and the release, asks, terminal-releases, mcp-table and home-flow tests. tsc, scripts/gate.ts and design-lint are clean. I did not run the full suite or the cell tests; the forge's check will.

What changed

  1. Hang ceiling. HANG_MS is now 30 minutes, derived from a new WORST_CASE_MS. That is 10 min of give-up polling, 3 min of smoke checks, 3 × (90 s settle + 20 s walk), two 2-min waits and 5 min of margin, about 27.5 min in all. A test asserts HANG_MS is at least the sum, replacing the old 15-minute check. The 3 min for smoke is my estimate of the heavy-page watch, not a measured figure.
  2. Stale marker.

- startProving clears proving.json when it starts. - provingSince ignores a marker older than HANG_MS. - A tick only writes the marker when a proof is due, so a boot whose retry is still in backoff says nothing. - A test covers each of these.

  1. Missing tests.

- proofWords is now exported and covered in test/release-proof-words.test.ts, which also covers the /releases attempt counts in proofCard. - test/asks-forge-alert.test.ts now checks that a probe: card has a Remove button and other causes don't.

  1. Terminal door. p on the terminal releases screen ("prove it again") uses the same gate as the page: a person who may read, never an agent. The test covers it with a failed proof, no failed proof, and an agent. The probe card's Remove already has a terminal equivalent, x (remove one) on the forges screen, and the doc says so.
  2. Plain words. /releases now reads "2 failed attempts so far; the last one tried 3 times".
  3. Docs. docs/features/release-heartbeat.md states the real ceiling and documents prove on release_now: admin scope, never an agent, and it answers invalid when no failed proof waits.

Scope (point 5). I kept the marker, the home figure and the hang verdict, since they now have tests. I did not split them out, as the hang verdict is what stops a failed proof standing for hours.

The earlier red tests in the cell e2e files were red on main too, so I left them alone.

commented

I merged main into the branch and pushed it. The gate is clean and tsc has no errors. The four test files I ran pass: asks-forge-alert, release-proof-words, release-prove and release-journey. I did not run the full suite; the forge's check does that on push.

The conflict was in forgeCard in src/web/studio/asks_page.ts. main now passes a ForgeNeed that carries its own line, so I used ${d.line} for the sentence. I kept this pull's "Remove" button, shown on probe: causes, next to Health.

Follow-on fixes the merge needed:

  • test/asks-forge-alert.test.ts built the old Degraded shape; it now builds a { line, cause, href } record.
  • proofCard in src/web/pages/releases.ts takes the data directory instead of a whole Ctx. That removes the as never casts in test/release-proof-words.test.ts, which the gate now refuses.
  • I marked the one real wait in test/release-prove.test.ts as // real-timer:, and rewrapped a 110-column comment in src/tui/screens/releases.ts.

I pushed the merge commit before the gate was clean, so the branch briefly had two gate violations. The follow-up commit fixed them.

I did not rewrite the real-timer wait in test/release-prove.test.ts to use an injected clock. The marker is the minimal fix.

commented

The dead-code test (test/unused-exports.test.ts) was red because of this pull, and I've fixed it and pushed.

WORST_CASE_MS in src/release/prove.ts was exported, but only test/release-prove.test.ts used it. I removed it. The ceiling test already computes the same sum from GIVE_UP_MS, the walks and the 3-minute smoke allowance, so it still checks HANG_MS against the worst healthy proof. It no longer adds the 5-minute margin that WORST_CASE_MS had.

tsc, scripts/gate.ts, test/unused-exports.test.ts and test/release-prove.test.ts all pass locally.

The other reds in the log I did not look at. test/mirror-worker.test.ts and test/boundary-tenancy-cell.test.ts are red on main as well, and the check log didn't name the failing tests beyond the first two. One of those was the S3 mirror bucket-path test. The other was a "surfaces adversarial" test about the dashboard's Apps and O… Neither touches what this pull changed. The forge's next check will show whether they recur.

Approved

Earlier asks are met: the hang ceiling is now derived and tested; the stale proving marker is cleared at boot and ignored once older than the ceiling; proofWords, the probe Remove card and the proof card wording have tests; the terminal door is p on the releases screen, gated as POST /releases/prove is, and it is tested; the how-to is in docs/features/release-heartbeat.md, not the README; the unused export is removed. No new security finding on the changed lines: the probe name is regex-checked before it becomes a card href, and the MCP prove path requires an admin-scope token and a person who may roll back. Non-blocking: the MCP path uses the stricter mayRollBack while the page and terminal use mayRead. Pick one gate for the three doors. I did not open the preview.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs