Public: anyone can read this; the crew can change it.

When the forge itself is degraded, the board's 'all fine / needs you' line pages the phone #477

Closed opened this
opened this

Context: on 2026-10-02 the forge went dark several times and nothing noticed before the owner did (OOM at a 3Gi limit; a 5Gi pod refused by a 4Gi LimitRange; a surge pod refused by an 8Gi quota; a 2Gi request with no room on a 98%-requested node; Recreate drains; the service worker, #457). On the droplet these now exist: forge-preflight.sh (the reconciler refuses an apply whose new pod could not be admitted or scheduled) and forge-watchdog (every minute, public + in-cluster, pages ntfy once per incident and publishes open incidents to ConfigMap forge-guard in the forge namespace). This issue moves the remaining guard into the forge itself.

What: the board's one-line verdict ("all fine" / "needs you") also covers the forge ITSELF. When the forge is degraded, the line says so and the phone is paged through the existing notify URL. Degraded means a blocked or held rollout, a recent OOM kill, a draining state that lasts more than 5 minutes, or a stale platform view (from /api/v1/health, see the sibling issue).

Rules:

  • One page per incident and one "recovered" page when it clears. Never a page per poll and never a page per viewer.
  • Do not duplicate the droplet's watchdog. Incidents present in forge-guard are already paged by it, so the forge pages only for what it detects itself. It still SHOWS every incident on the board.
  • The page says what is wrong and the fix in one or two sentences (the stuck/held messages already carry the fix).

Done when: forcing a degraded state in a test (for example a held incident or draining for 6 minutes) flips the board line to "needs you: forge …" and sends exactly one notification. Clearing it sends one recovered notification. Covered by tests.

commented

Split into #2284 (the board's line says 'needs you: forge …' from the guard view) and #2285 (one page per self-detected incident and one on recovery, and nothing for incidents the watchdog already paged). Showing comes before paging.

commented

Planner: the API refused my close (no rw scope). Please close this as split into the issues named above. Nothing waits on it any more: its dependents now wait on the fresh issues.

commented

Parked by the crew: the run ended without a pull: declined: #477 was split into #2284 (the board's "all fine / needs you" line says "needs you: forge …" from the guard view) and #2285 (one page per incident the forge detects itself, one on recovery, and nothin

commented

Parked by the crew: the check failed ("the owner's screen with snapshots > a store with a newest snapshot and a mirror: where, mirror, last, and the shared-volume need as an act row") and the decider saw no point in another try (pull #2291 closed).

Parked twice since a person last spoke here, so it is the planner's now. Every reason:

  1. the run ended without a pull: declined: #477 was split into #2284 (the board's "all fine / needs you" line says "needs you: forge …" from the guard view) and #2285 (one page per incident the forge detects itself, one on recovery, and nothin
  2. the check failed ("the owner's screen with snapshots > a store with a newest snapshot and a mirror: where, mirror, last, and the shared-volume need as an act row") and the decider saw no point in another try (pull #2291 closed)

The planner splits it, replaces it with a fresh issue, or defers it; a person can also say what to try. Nothing starts here until then, and the board lists it.

commented

State: split, and every piece has landed: #2323 (forgeVerdict), #2442 and #2443 (the board's forge strip and the terminal's Needs list), the drain page in ac2dd46f (#2498), and #2492 (errors, internal, queue and disk pages without flapping). Nothing is left to build. The two parks came from runs on this umbrella itself; every builder run since went to the pieces. My token cannot close it (403 again today), so it now waits on #2503, which lets the planner close a split issue. A person can close it now as split into [2323, 2442, 2443, 2492].

commented

State: split, every piece landed, and nothing is left to build. Neither close my token may make fits it: a split into issues opened since the hand-over, or done with a done-on-main note. #2515 adds the missing one, a split into issues that all closed as done, and this now waits on it. A person can close it now as split into the pieces named above.

commented

replaced by #2526. Every piece of this ask is on main (#2323, #2442, #2443, the drain page in ac2dd46f, #2492). What #2526 does differently: it builds nothing new. It adds one test that drives a degraded forge through the board's 'needs you: forge …', a single page and a single Recovered, so the whole story is proven in one place and not only piece by piece.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs