Public: anyone can read this; the crew can change it.
Sentinel: probe:probe-5893f874 was degraded 25 minutes today: The release proof's probe forge probe-5893f874 could not be removed and is still running. #12557
Check · Review · Land · Live done — Landed and live in production.
The release proof now retries removing its probe forge for about 14 seconds instead of about 1, which should keep probe: cards from appearing when the host is only briefly busy. I haven't checked that against a real busy host.
shipped production is serving it deploy 7612a18
The release proof now retries removing its probe forge for about 14 seconds instead of about 1, which should keep probe: cards from appearing when the host is only briefly busy. I haven't checked that against a real busy host. test/release-journey.test.ts (12 tests) passes, tsc --noEmit is clean and bun scripts/gate.ts printed nothing.
Cause. The proof's walk makes a throwaway probe-xxxxxxxx forge and removes it afterwards. It tried removal three times in about 1.25 s. If the host was busy for longer, the forge stayed up and was recorded as leftover. The probe:probe-5893f874 card then stayed until a later proof ran and found none. The sweep that cleans up stale probes ignores anything under an hour old, so it didn't help either.
Change. REMOVE_BACKOFF_MS in src/release/journey.ts is now [250, 1000, 3000, 10_000], so removal gets five attempts. The sweep passes no backoff, so it still tries each stale forge once. The test I added makes removal fail four times and checks the fifth attempt reports the forge as removed.
Not done. If all five attempts fail, the card still lasts until the next proof. A fuller fix would let the next sweep remove forges named in the last proof's leftover list whatever their age, or give the card its own retry. That is a larger change to src/release/prove.ts. I left it out because the likely cause is a short stall, which longer retries should cover.
Try it. Run bun test test/release-journey.test.ts. On production, probe: cards should stay off /system after the next release proof.
The change does what it says. removeForge now makes one try plus four backed-off retries (about 14 s), the sweep still passes no backoff, and the new test fails on the old two-entry list. It adds no UI, so there is nothing for a phone or keyboard to miss. I judged this from the diff and journey.ts, and did not exercise the preview.
Two things I would do next:
- If all five attempts fail, the probe: card still lasts until the next proof. Let the next sweep remove forges named in the last proof's leftover list whatever their age.
- The 14 s of sleeping in the journey's finally block delays the verdict when the host is down. Check that the release proof's overall time budget still holds.
The release proof now retries removing its probe forge for about 14 seconds instead of about 1, which should keep
probe:cards from appearing when the host is only briefly busy. I haven't checked that against a real busy host.test/release-journey.test.ts(12 tests) passes,tsc --noEmitis clean andbun scripts/gate.tsprinted nothing.Cause. The proof's walk makes a throwaway
probe-xxxxxxxxforge and removes it afterwards. It tried removal three times in about 1.25 s. If the host was busy for longer, the forge stayed up and was recorded as leftover. Theprobe:probe-5893f874card then stayed until a later proof ran and found none. The sweep that cleans up stale probes ignores anything under an hour old, so it didn't help either.Change.
REMOVE_BACKOFF_MSinsrc/release/journey.tsis now[250, 1000, 3000, 10_000], so removal gets five attempts. The sweep passes no backoff, so it still tries each stale forge once. The test I added makes removal fail four times and checks the fifth attempt reports the forge as removed.Not done. If all five attempts fail, the card still lasts until the next proof. A fuller fix would let the next sweep remove forges named in the last proof's leftover list whatever their age, or give the card its own retry. That is a larger change to
src/release/prove.ts. I left it out because the likely cause is a short stall, which longer retries should cover.Try it. Run
bun test test/release-journey.test.ts. On production,probe:cards should stay off/systemafter the next release proof.Closes #12556