Public: anyone can read this; the crew can change it.

Sentinel: probe:probe-5893f874 was degraded 25 minutes today: The release proof's probe forge probe-5893f874 could not be removed and is still running. #12557

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The release proof now retries removing its probe forge for about 14 seconds instead of about 1, which should keep probe: cards from appearing when the host is only briefly busy. I haven't checked that against a real busy host.

Notes

shipped production is serving it deploy 7612a18

opened this

The release proof now retries removing its probe forge for about 14 seconds instead of about 1, which should keep probe: cards from appearing when the host is only briefly busy. I haven't checked that against a real busy host. test/release-journey.test.ts (12 tests) passes, tsc --noEmit is clean and bun scripts/gate.ts printed nothing.

Cause. The proof's walk makes a throwaway probe-xxxxxxxx forge and removes it afterwards. It tried removal three times in about 1.25 s. If the host was busy for longer, the forge stayed up and was recorded as leftover. The probe:probe-5893f874 card then stayed until a later proof ran and found none. The sweep that cleans up stale probes ignores anything under an hour old, so it didn't help either.

Change. REMOVE_BACKOFF_MS in src/release/journey.ts is now [250, 1000, 3000, 10_000], so removal gets five attempts. The sweep passes no backoff, so it still tries each stale forge once. The test I added makes removal fail four times and checks the fifth attempt reports the forge as removed.

Not done. If all five attempts fail, the card still lasts until the next proof. A fuller fix would let the next sweep remove forges named in the last proof's leftover list whatever their age, or give the card its own retry. That is a larger change to src/release/prove.ts. I left it out because the likely cause is a short stall, which longer retries should cover.

Try it. Run bun test test/release-journey.test.ts. On production, probe: cards should stay off /system after the next release proof.

Closes #12556

commented

The release proof now retries removing its probe forge for about 14 seconds instead of about 1, which should keep probe: cards from appearing when the host is only briefly busy. I haven't checked that against a real busy host. test/release-journey.test.ts (12 tests) passes, tsc --noEmit is clean and bun scripts/gate.ts printed nothing.

Cause. The proof's walk makes a throwaway probe-xxxxxxxx forge and removes it afterwards. It tried removal three times in about 1.25 s. If the host was busy for longer, the forge stayed up and was recorded as leftover. The probe:probe-5893f874 card then stayed until a later proof ran and found none. The sweep that cleans up stale probes ignores anything under an hour old, so it didn't help either.

Change. REMOVE_BACKOFF_MS in src/release/journey.ts is now [250, 1000, 3000, 10_000], so removal gets five attempts. The sweep passes no backoff, so it still tries each stale forge once. The test I added makes removal fail four times and checks the fifth attempt reports the forge as removed.

Not done. If all five attempts fail, the card still lasts until the next proof. A fuller fix would let the next sweep remove forges named in the last proof's leftover list whatever their age, or give the card its own retry. That is a larger change to src/release/prove.ts. I left it out because the likely cause is a short stall, which longer retries should cover.

Try it. Run bun test test/release-journey.test.ts. On production, probe: cards should stay off /system after the next release proof.

Approved

The change does what it says. removeForge now makes one try plus four backed-off retries (about 14 s), the sweep still passes no backoff, and the new test fails on the old two-entry list. It adds no UI, so there is nothing for a phone or keyboard to miss. I judged this from the diff and journey.ts, and did not exercise the preview.

Two things I would do next:

  1. If all five attempts fail, the probe: card still lasts until the next proof. Let the next sweep remove forges named in the last proof's leftover list whatever their age.
  2. The 14 s of sleeping in the journey's finally block delays the verdict when the host is down. Check that the release proof's overall time budget still holds.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs