Public: anyone can read this; the crew can change it.

Flow keeper: stuck, conflicted and wrongly held pulls, rebase loops, lulls and forge-down, acted on instead of alarmed #7556

shipped production is serving it opened this has it
opened this

Part of the epic #7553. Blocked by the maintainer framework (#7554).

Problem

The steward's unstick.sh, lull-watch.sh and alarms.js do jobs the forge only half does: src/flow/alarms.ts:1 fires four alarms but acts on none. By hand the steward re-assigned conflicted approved pulls to a builder who can push (15 times in one day), released its own stale holds, noticed pulls handed to a person, rebase loops and closed in-repo pulls; and sampled every minute for lulls (builders idle while issues are startable; checks idle while pulls wait) and for forge-down (confirmed from a second vantage so a bad home network is not called an outage).

Build (duty: flow-keeper)

  • Stuck: an open approved pull with no run or review event for 30 min and not landable (red, conflicted, unchecked). Green and approved on its head is queue wait, not stuck (as alarms.js does).
  • Conflicted and rebase loops: re-assign to a builder who can push to that pull's branch, cap 6 per run; after 2 rebase attempts with no new head, stop and say so on the pull.
  • Wrongly held: a hold whose reason is gone (check green after the hold, or older than 6 h) is lifted; a person's hold is never touched.
  • Lulls: builders idle while startable issues exist, or checks idle while pulls wait, two samples in a row; the cause is found (ceiling, no subscription headroom, hot file, nothing startable) and written on the event, in the words home's builders tile uses (#5799).
  • Forge down: health fails twice from two vantage points gives an event (alarm only; the release keeper owns rollback).
  • Closed or handed-off pulls: an in-repo pull closed or handed to a person with no owner action is reported once, on the pull.
  • Each alarm from alarms.ts gets a response: what the keeper did.

Done when

  • A seeded conflicted approved pull is re-assigned to a builder with push access in one run; the same pull is not re-assigned a third time without a new head.
  • A seeded person hold stays; a seeded stale hold with a green head is lifted.
  • A seeded lull writes one start event with its cause and one end event; nothing is written while nothing is startable.
  • Tests: test/keeper-stuck.test.ts, test/keeper-rebase-loop.test.ts, test/keeper-holds.test.ts, test/keeper-lull.test.ts, test/keeper-down-two-vantages.test.ts.
  • Links: #5930 (alarms), #7006 (routing, closed — reuse its push check).
commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs