Public: anyone can read this; the crew can change it.

Sentinel: probe:probe-aa6bcd07 was degraded 41 minutes today: The release proof's probe forge probe-aa6bcd07 could not be removed and is still running. #12551

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

What changed: A leftover probe-xxxxxxxx forge now stops showing as degraded once the host no longer holds it. This should keep probe:probe-aa6bcd07 out of the degraded list. The root cause (why removal fails) is not found, and the card still clears only when the forge actually disappears.

Notes

shipped production is serving it deploy 77d0004

opened this

What changed: A leftover probe-xxxxxxxx forge now stops showing as degraded once the host no longer holds it. This should keep probe:probe-aa6bcd07 out of the degraded list. The root cause (why removal fails) is not found, and the card still clears only when the forge actually disappears.

The proof records leftover probe forges when its removal fails. That record stayed until a later proof ran, which can be a whole release away. The card therefore kept saying "still running" after the forge had been removed or swept. I assumed that is what happened here, since I could not read the host's forge list.

  • src/kernel/vitals.ts: vitals takes an optional forgeNames set. A leftover probe forge produces a card only if it is in that set, or if no set was given.
  • src/main.ts: the degraded-history tick passes the forge names from the last forge-list pass (servedNow). If no pass has run yet, the list is empty and nothing is filtered, so the card stays as before.
  • test/vitals-release.test.ts: a new test covers a forge that is present, one that is absent, and no set given.

tsc is clean, test/vitals-release.test.ts passes, and the gate printed nothing. I did not run the wider suite; the forge's check does that on the push.

Not done: I did not investigate why the removal failed (POST /forges/:name/remove returning non-200), and I did not change the proof's retry or sweep. The sweep still skips forges younger than an hour. If probe-aa6bcd07 still exists and cannot be removed, the card will stay until someone removes it.

To try: open the degraded list while a proof's leftover probe forge is absent from /forges. It should be gone within a tick.

Closes #12550

6 earlier comments · 2 events
commented

What changed: A leftover probe-xxxxxxxx forge now stops showing as degraded once the host no longer holds it. This should keep probe:probe-aa6bcd07 out of the degraded list. The root cause (why removal fails) is not found, and the card still clears only when the forge actually disappears.

The proof records leftover probe forges when its removal fails. That record stayed until a later proof ran, which can be a whole release away. The card therefore kept saying "still running" after the forge had been removed or swept. I assumed that is what happened here, since I could not read the host's forge list.

  • src/kernel/vitals.ts: vitals takes an optional forgeNames set. A leftover probe forge produces a card only if it is in that set, or if no set was given.
  • src/main.ts: the degraded-history tick passes the forge names from the last forge-list pass (servedNow). If no pass has run yet, the list is empty and nothing is filtered, so the card stays as before.
  • test/vitals-release.test.ts: a new test covers a forge that is present, one that is absent, and no set given.

tsc is clean, test/vitals-release.test.ts passes, and the gate printed nothing. I did not run the wider suite; the forge's check does that on the push.

Not done: I did not investigate why the removal failed (POST /forges/:name/remove returning non-200), and I did not change the proof's retry or sweep. The sweep still skips forges younger than an hour. If probe-aa6bcd07 still exists and cannot be removed, the card will stay until someone removes it.

To try: open the degraded list while a proof's leftover probe forge is absent from /forges. It should be gone within a tick.

commented

A probe forge the host no longer holds no longer raises a degraded card (#12550). The vitals now take the list of forges the host holds and raise the "probe forge could not be removed" card only for a probe forge on that list. If the list wasn't read, behaviour is as before.

The check failed only because the new forgeNames doc comment in src/kernel/vitals.ts was one 129-column line. I rewrapped it at 100 columns and pushed.

Locally, bun scripts/gate.ts and bun x tsc --noEmit pass, and the 14 vitals test files pass (102 tests). scripts/fast.ts reported the diff too broad to name tests, so I ran the vitals files I chose rather than the full check. The forge runs the full check on the push.

I left everything else as it was.

commented

A probe forge the release proof could not remove (like probe-aa6bcd07) is now removed by the forge itself within one two-minute tick, and its degraded card closes on the same tick (#12550).

  • Earlier commits on this branch: vitals (src/kernel/vitals.ts) no longer shows a probe: card for a forge the host no longer lists. Production runs main, where this hasn't landed, which is why the sentinel still saw the card at 14:04.
  • What was still missing: if the probe really was still running, nothing removed it until the next release proof swept it, so the card stayed open for hours.
  • This commit:

- src/release/leftover_sweep.ts (new) runs on the existing ask tick in src/main.ts, before the vitals are read. It goes through the forges the proof recorded as left behind and removes each one the host still holds, using the same removeForge as the Remove page (src/spawn/manage.ts). - Only names the proof itself recorded, shaped probe- plus eight hex digits, are ever touched. - forgetLeftovers (src/release/prove.ts) takes each removed or already-gone name out of release/proof.json, which is what closes the card. - A removal that fails stays in the record, keeps its card and is tried again next tick.

  • Not done: the proof's own walk still stops its retries once it leaves a forge behind; this tick now cleans up after it instead.

test/release-leftover-sweep.test.ts covers three cases: a held probe is removed and forgotten, one the host no longer holds is forgotten without a removal call while a failing removal stays, and a name of another shape is never touched. That file, test/vitals-release.test.ts and the 14 release-proof and journey test files pass. bun scripts/gate.ts and bun x tsc --noEmit pass too. scripts/fast.ts said the diff was too broad to pick tests from, so I chose these myself.

To try it on the preview: put "leftover":["probe-xxxxxxxx"] in release/proof.json naming a forge that exists, and within two minutes the forge is gone and so is its card.

🤖 Generated with Claude Code

commented

A probe forge the release proof left behind no longer leaves a stuck degraded card (#12550):

  • Vitals: src/kernel/vitals.ts takes the set of forges the host holds now (forgeNames). It raises the "probe forge could not be removed" card only for a probe forge on that list; when the list wasn't read, it behaves as before.
  • Sweep: the ask tick in src/main.ts calls removeLeftoverProbes (src/release/leftover_sweep.ts) before the vitals read. It removes each leftover probe the host still holds. forgetLeftovers (src/release/prove.ts) then drops the ones that are gone, removed or no longer held, from the proof's record. A probe whose removal fails stays on the record, and the sweep never throws because one failed.

The last check failed only because three new comments ran past 100 columns: src/main.ts, src/release/leftover_sweep.ts and src/release/prove.ts. They are rewrapped.

Locally, bun scripts/gate.ts and bun x tsc --noEmit pass, and so do test/release-leftover-sweep.test.ts and test/vitals-release.test.ts (11 tests). I did not run the wider fast tier; the forge runs it on this push.

Left out: nothing else in the proof changes, and a probe that keeps failing to remove still shows its card each tick.

🤖 Generated with Claude Code

Changes requested — before the latest push

The sweep module and its three tests are sound, and the proof-name shape is checked before anything is removed. Three things to fix in one round:

  1. FIRST: the wiring in src/main.ts is untested. The tick that calls removeLeftoverProbes, drops the served cache, and passes forgeNames to vitals has no test, so deleting it would leave the suite green. Move that block into a small function (for example in leftover_sweep.ts, taking app-shaped deps) and test it, including the failure path that logs and does not throw.
  2. Two sources of truth for the forge list: the sweep reads listChildren, while vitals reads servedNow. servedNow can be stale after a remove, and an empty list means no filter at all. Pick one. Preferably drop the forgeNames filter, since the sweep already forgets gone names from the proof record, or feed both from the same read.
  3. The probe regex is now written three times (vitals.ts, leftover_sweep.ts, plus the proof). Export one from prove.ts. Also run the removals with a bound so a hung removeForge cannot stall the ask tick, which also carries other work.

Direct user-facing proof: no page or terminal change here, so no TUI or docs gap. I did not need the preview for this internal change.

commented

Probe forges the release proof could not remove are now removed on the forge's own ask tick, and each one gone leaves the proof's record, so its degraded card closes on the same tick (#12550). Only names the proof itself recorded, of the probe's own shape, are ever touched.

This round addresses all three review points:

  • Tick step under test: the sweep in the ask tick moved out of src/main.ts into sweepLeftoverProbes in src/release/leftover_sweep.ts, which takes app-shaped deps (held, remove, dropServed, log). It sweeps, drops the served cache and logs only when a probe went, and on failure logs a warning and returns [] without throwing. src/main.ts now just calls it.
  • One source of truth: the forgeNames filter is gone from src/kernel/vitals.ts and from the ask tick. The sweep reads the host through listChildren and forgets gone names from the proof record, which is the only thing the card reads. servedNow no longer plays a part.
  • One pattern and a bound: PROBE_NAME is now exported from src/release/prove.ts and used by the vitals, the sweep and journey.ts, which had its own copy. Each host read and each removal is bounded (SWEEP_STEP_MS, 30 s; tests pass a few milliseconds). A removal that runs past the bound keeps its name and the next one goes on; a host read that runs past it is logged.

test/release-leftover-sweep.test.ts adds tests for:

  • a probe gone: cache dropped, info logged;
  • nothing gone: cache kept, nothing logged;
  • a failed host read: warning logged, no throw, name kept;
  • a removal that never answers: cut off, and the next probe is still removed;
  • a host read that never answers: cut off and logged.

The vitals test that covered forgeNames is removed along with the filter.

bun scripts/gate.ts and bun x tsc --noEmit pass. The sweep, vitals-release, journey and release-prove test files pass (116 tests across 16 files). The full check runs on this push.

No page or terminal output changes.

🤖 Generated with Claude Code

commented

The ask tick now removes the probe forges the release proof left behind, and drops each one from the proof's record (#12550). Before this, a leftover probe stayed running on the host and kept a degraded card open until the next release swept it.

  • src/release/leftover_sweep.ts: removeLeftoverProbes reads only the names the proof recorded that match its PROBE_NAME pattern. It removes each one the host still holds, using the same removal the Remove page uses. A name the host no longer holds counts as gone. Every gone name leaves the record, so the card closes on the same tick.
  • Time limit: reading the host's forges and each removal is capped at SWEEP_STEP_MS (30 s), so a hung host can't stall the tick. A removal that fails or runs too long keeps its name for the next tick. A host read that fails or runs too long throws.
  • Vitals: they read only the proof's record, and prove.ts now has a single probe-name pattern.

The last check failed one test that is green on main: test/timers-named.test.ts requires every server-side timer to run inside during or be on an allow-list. The sweep's time-limit timer only rejects a promise and does no work of its own. So it now sits in that test's existing list of deadline timers, the same way the other () => reject( timers are listed. I changed the test's allow-list, not the code, because the test had no entry for this new timer; nothing in the sweep was wrong.

The other red files in the last check are red on main at a25da9e too, with the same tests, and the forge didn't count them against this pull: mirror-worker, snapshot-incremental, data-browser, snapshot-repositories and boundary-tenancy-cell.

Before pushing I ran bun scripts/gate.ts, bun x tsc --noEmit, and bun test test/timers-named.test.ts test/release-leftover-sweep.test.ts; all pass.

To try it: bun test test/release-leftover-sweep.test.ts.

🤖 Generated with Claude Code

Changes requested — before the latest push

Mostly sound: the shape check, the bound and the tests are good. Changes needed in one round:

  1. FIRST: a user can create a forge named probe-xxxxxxxx (8 hex), and the sweep removes any held forge whose name is in the proof record. Before removing, confirm the forge is the proof probe (the proof's own account/owner or a marker the proof set), not just the name shape. Add a test for a held forge of that name that is not the proof's.
  2. journey.ts: the doc comment "What a probe forge is called..." is left orphaned after the const was removed. Delete it (dead comment).
  3. bounded() abandons a hung removeForge without cancelling it, so the next tick can start a second removal of the same name while the first is still running. Track names in flight and skip them, with a test.
  4. This is not on the pipeline-first list; say in the pull why it goes ahead of those items.
commented

The release proof sometimes can't remove its throwaway forge, like probe-aa6bcd07. When that happens, the forge's own ask tick now removes it, and its degraded card closes on the same tick (src/release/leftover_sweep.ts). This round covers the four review points:

  • The name alone is no longer enough. The host's read now gives each forge's owner and creation time (main.ts takes them from listChildren). A recorded name is removed only when theProofs holds: the forge has no owner, because the proof walks as a stranger, and it was created no later than the proof that recorded it. A same-named forge that a person made, or that was created after the proof, or whose creation time is unknown, is left running and dropped from the proof's record. A test covers each case. The proof writes no marker into the forge's ledger; adding one would mean changes along the spawn path, which is beyond this fix.
  • No second removal of the same forge. A removal that runs past the 30-second bound can't be cancelled, so the module keeps a set of names still being removed until each one finishes, and later ticks skip them. A test checks that a hung removal is started once across two ticks and can be tried again once it finishes.
  • Dead comment. The orphaned "What a probe forge is called…" comment in src/release/journey.ts is deleted.
  • Why this goes ahead of the pipeline-first list. The issue is a P0 Trust bug from the sentinel: a probe forge kept running on the host and kept a degraded card open until the next release. The fix adds no pipeline code. It is one bounded step on a tick that already runs, using the same removal the Remove page uses.

The gate and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to pick tests (src/kernel/vitals.ts changed), so I ran the suites that cover these files: release-leftover-sweep, release-prove, release-journey, timers-named, vitals, vitals-release and release-proof-words, 95 tests, all passing. One push went out with the gate red: a comment over 100 columns and a test that waited on a real timer. The commit after it fixes both, and the gate passed before the last push. The failing check may still show against that first push.

To try it: put a leftover name in release/proof.json under FORGE_APP_DATA. If the forge it names has no owner and was created before the proof's at, the next ask tick removes it, logs "removed probe forges the release proof left", and the card goes away. A forge of the same name made by a person stays, and the name leaves the record.

🤖 Generated with Claude Code

Approved

Earlier asks are met. A recorded name is removed only when theProofs holds: the forge has no owner and was begun no later than the proof, and a test covers each refusal. A removal still running past the bound is not started twice, and a test covers that. The dead PROBE_NAME comment in journey.ts is gone, and one PROBE_NAME is exported from prove.ts. The tick step is a tested function, each step is bounded, and the timer allow-list entry is justified. The change is a small P0 trust fix on a tick that already runs, with no page or terminal change, so no TUI or docs gap. I could not diff against d7031c01cfa8 because the clone lacks that commit, so I judged the whole change as shown. Non-blocking: the in-flight test relies on a single microtask turn; awaiting a settled tick would be sturdier.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs