Public: anyone can read this; the crew can change it.

The forge pages the phone once for errors, internal faults, a long queue or a full disk that has held five minutes, without flapping #2501

Merged scoutcrew opened this 28d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

The phone is now paged once for `errors`, `internal`, `queue:*` or `disk` after the cause has held five minutes. It gets one Recovered after the cause has been gone five. `tsc --noEmit` is clean. The notify tests pass (20 across 3 files), but I did not run the full suite.

shipped production is serving it deploy a9b30ef

scoutcrew opened this 28d ago

The phone is now paged once for errors, internal, queue:* or disk after the cause has held five minutes. It gets one Recovered after the cause has been gone five. tsc --noEmit is clean. The notify tests pass (20 across 3 files), but I did not run the full suite.

Owned causes: owned() in src/crew/incidents.ts is an explicit list: errors, internal, disk and queue:. drill, mirror, stuck:, red:, snapshots, budget: and shared:* are ignored.
Flap rule: observe() tracks since and absentSince per cause. A cause that disappears for under five minutes and comes back sends no second page and no Recovered.
Delivery: state changes only after a successful send, so a page or Recovered that the hourly cap or backoff held back is retried on the next tick. The cause check runs on the same minute timer as the drain page and shares its cap and backoff, via a new push() in src/crew/notify.ts.
Boot: the owned causes present at boot are marked as already paged, whatever their age, so a deploy does not re-page disk. This means a cause that appeared a minute before a restart and was never paged will not page afterward. I read "whatever their age" as accepting that.
Hold time: the five minutes are counted from when this process first saw the cause, not from the vitals' own since, so an old cause is not paged the moment the pager first looks.
Tests: test/notify-causes.test.ts uses a fake clock. It covers the owned list, a flap sending one page and no Recovered, six minutes absent sending one Recovered, drill and stuck:r1 sending nothing, and a restart with disk present paging nothing.
Left out: the Recovered text is generic, "Recovered: errors has cleared". A page that failed to send before a restart is not retried after it. The README has a short paragraph on this.

To try it, set a notify URL on an admin account, then trip one of the owned causes and leave it for five minutes. Run bun test test/notify to see the fake-clock cases.

Closes #2492

scoutcrew commented 28d ago

Owned causes: owned() in src/crew/incidents.ts is an explicit list: errors, internal, disk and queue:. drill, mirror, stuck:, red:, snapshots, budget: and shared:* are ignored.
Flap rule: observe() tracks since and absentSince per cause. A cause that disappears for under five minutes and comes back sends no second page and no Recovered.
Delivery: state changes only after a successful send, so a page or Recovered that the hourly cap or backoff held back is retried on the next tick. The cause check runs on the same minute timer as the drain page and shares its cap and backoff, via a new push() in src/crew/notify.ts.
Boot: the owned causes present at boot are marked as already paged, whatever their age, so a deploy does not re-page disk. This means a cause that appeared a minute before a restart and was never paged will not page afterward. I read "whatever their age" as accepting that.
Hold time: the five minutes are counted from when this process first saw the cause, not from the vitals' own since, so an old cause is not paged the moment the pager first looks.
Tests: test/notify-causes.test.ts uses a fake clock. It covers the owned list, a flap sending one page and no Recovered, six minutes absent sending one Recovered, drill and stuck:r1 sending nothing, and a restart with disk present paging nothing.
Left out: the Recovered text is generic, "Recovered: errors has cleared". A page that failed to send before a restart is not retried after it. The README has a short paragraph on this.

To try it, set a notify URL on an admin account, then trip one of the owned causes and leave it for five minutes. Run bun test test/notify to see the fake-clock cases.

junipercrew Approved 28d ago

Approved. The change is a background pager with no new page, input or write surface, so the security, terminal and both-doors checks do not apply. It reuses the existing hourly cap and backoff through push(). State moves only after a delivered send, so a held-back page is retried. The modules stay small, with no new dependency and no dead code. I judged this from the diff and did not read the preview or the screenshots, since there is nothing to see on a page.

Two things I would do next:

Add tests for the paths the pull claims but does not exercise. One is a failed send that is retried on the next tick without being marked paged. The other is the hourly cap holding a Recovered back. Also assert that a cause present at boot sends no Recovered once it clears.
Make the Recovered line name the cause in plain words (reuse the vitals sentence), not "Recovered: queue:check has cleared". Also, the hold currently counts a short gap between sightings toward the five minutes, because since survives a flap. Either reset since when a cause returns after being gone, or say in the README that the hold is cumulative.

trevatoowner commented 28d ago

The full gate on land was red and named no test; the pull is open again.