Public: anyone can read this; the crew can change it.

Flow keeper: stuck, conflicted and wrongly held pulls, rebase loops, lulls and forge-down, acted on instead of alarmed #11255

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The `flow-keeper` duty now runs once a minute, registered in `src/maintain/registry.ts`, and leaves a `maintenance_run` row whose findings each carry a probe. The five new test files pass, along with the neighbouring suites I ran (idle-reason, the stuck alarm and other flow-alarm tests, maintain, docs-link, readme-cap). `bun scripts/gate.ts` and `tsc --noEmit` are clean. The forge runs the full ch…

shipped production is serving it deploy 0d144fb

opened this

The flow-keeper duty now runs once a minute, registered in src/maintain/registry.ts, and leaves a maintenance_run row whose findings each carry a probe. The five new test files pass, along with the neighbouring suites I ran (idle-reason, the stuck alarm and other flow-alarm tests, maintain, docs-link, readme-cap). bun scripts/gate.ts and tsc --noEmit are clean. The forge runs the full check on the push. scripts/fast.ts said the diff was too broad to name tests, so I picked those suites by hand.

What it does

  • Stuck: an approved, unheld pull with no run, review or push for 30 minutes that is conflicted, red or unchecked becomes a finding. A green check on its head counts as queue wait, not stuck.
  • Conflicted and rebase loops:

- A conflicted approved pull goes to the builder who can push to its branch, using ownBuilder (the push check #7006 made), at most 6 per run. - After 2 assignments on the same head it stops and comments once on the pull. A new head starts the count again. - A pull a person holds or is assigned to is never moved.

  • Wrongly held: a crew hold is lifted when a check went green on the head after the hold, or when it is over 6 hours old. Only holds placed by agents are read, so a person's hold is never touched.
  • Lulls: two idle samples in a row write one keeper_lull start event with its cause, and the first busy sample writes the end event. The cause uses the words home's builders tile uses (#5799); the ceiling and subscription-headroom checks were pulled out of idle_reason.ts as whatBlocks. Nothing is written while nothing is startable. Checks idle while pulls wait is a lull of its own.
  • Forge down: health failing twice from two vantage points writes forge_down, then forge_up on recovery. It is alarm only.
  • Closed or handed-off pulls: an in-repo crew pull closed with no reason, or assigned to a person, gets one comment on the pull.
  • Alarms: each firing alarm from alarms.ts gets a finding and an action line saying what the keeper did, or that it only reports.

The tests for the first four items cover your "done when" cases; the closed/handed-off report is untested. How-to is in docs/features/flow-keeper.md. Besides the duty, the shared change is DutyContext and the runner, which now pass the running app.

Left out or different from the issue

  • Conflicted detection: the keeper does not ask git. It uses the conflict runs and the finishing pass's notes in flow/conflicted.ts. The keeper only assigns; the dispatcher's assignment pickup starts the run, which I assumed but did not test.
  • Forge-down vantages: they come from FORGE_HEALTH_VANTAGES=name=url,name=url. With fewer than two configured, nothing is sampled, so the outage check does nothing until the variable is set on the forge.
  • Lull streaks and forge-down failure counts: these live in process memory, so a restart delays detection by two samples. Whether a lull is open comes from its events.
  • Comments the keeper writes: they are posted under the name of whoever turned autonomy on, with a line saying so, as the finishing pass's hand-over does.
  • Stuck alarm and green pulls: the existing stuck alarm in alarm_stuck.ts still fires on a green quiet pull. I left it as it is, so the keeper's finding is narrower than the alarm.
  • Unrelated test failure: test/apps-idle.test.ts timed out in my sandbox. It only ran because my file glob matched it. It needs a running preview app, so I did not look further.

Closes #7556

commented

The flow-keeper duty now runs once a minute, registered in src/maintain/registry.ts, and leaves a maintenance_run row whose findings each carry a probe. The five new test files pass, along with the neighbouring suites I ran (idle-reason, the stuck alarm and other flow-alarm tests, maintain, docs-link, readme-cap). bun scripts/gate.ts and tsc --noEmit are clean. The forge runs the full check on the push. scripts/fast.ts said the diff was too broad to name tests, so I picked those suites by hand.

What it does

  • Stuck: an approved, unheld pull with no run, review or push for 30 minutes that is conflicted, red or unchecked becomes a finding. A green check on its head counts as queue wait, not stuck.
  • Conflicted and rebase loops:

- A conflicted approved pull goes to the builder who can push to its branch, using ownBuilder (the push check #7006 made), at most 6 per run. - After 2 assignments on the same head it stops and comments once on the pull. A new head starts the count again. - A pull a person holds or is assigned to is never moved.

  • Wrongly held: a crew hold is lifted when a check went green on the head after the hold, or when it is over 6 hours old. Only holds placed by agents are read, so a person's hold is never touched.
  • Lulls: two idle samples in a row write one keeper_lull start event with its cause, and the first busy sample writes the end event. The cause uses the words home's builders tile uses (#5799); the ceiling and subscription-headroom checks were pulled out of idle_reason.ts as whatBlocks. Nothing is written while nothing is startable. Checks idle while pulls wait is a lull of its own.
  • Forge down: health failing twice from two vantage points writes forge_down, then forge_up on recovery. It is alarm only.
  • Closed or handed-off pulls: an in-repo crew pull closed with no reason, or assigned to a person, gets one comment on the pull.
  • Alarms: each firing alarm from alarms.ts gets a finding and an action line saying what the keeper did, or that it only reports.

The tests for the first four items cover your "done when" cases; the closed/handed-off report is untested. How-to is in docs/features/flow-keeper.md. Besides the duty, the shared change is DutyContext and the runner, which now pass the running app.

Left out or different from the issue

  • Conflicted detection: the keeper does not ask git. It uses the conflict runs and the finishing pass's notes in flow/conflicted.ts. The keeper only assigns; the dispatcher's assignment pickup starts the run, which I assumed but did not test.
  • Forge-down vantages: they come from FORGE_HEALTH_VANTAGES=name=url,name=url. With fewer than two configured, nothing is sampled, so the outage check does nothing until the variable is set on the forge.
  • Lull streaks and forge-down failure counts: these live in process memory, so a restart delays detection by two samples. Whether a lull is open comes from its events.
  • Comments the keeper writes: they are posted under the name of whoever turned autonomy on, with a line saying so, as the finishing pass's hand-over does.
  • Stuck alarm and green pulls: the existing stuck alarm in alarm_stuck.ts still fires on a green quiet pull. I left it as it is, so the keeper's finding is narrower than the alarm.
  • Unrelated test failure: test/apps-idle.test.ts timed out in my sandbox. It only ran because my file glob matched it. It needs a running preview app, so I did not look further.
commented

The failing test, "every kind the source emits is classified", now passes, and I've pushed the fix. I ran only the event-audience tests (17 pass) and tsc --noEmit (clean), not the full suite. The forge's check will run that on this push.

  • Cause: The flow keeper's note() in src/maintain/keeper/record.ts calls append({ kind }) with a parameter, which the audience scan can't read. Its six kinds were also never classified: keeper_said, keeper_rebase, keeper_release, keeper_lull, forge_down and forge_up.
  • Fix: test/event-audience-coverage.test.ts now lists those six kinds for record.ts, the same way it already does for runs.ts. In src/core/event_audience.ts, keeper_*, forge_down and forge_up are now admins-only, matching the existing keeper. kinds that the throughput keeper writes.
  • Not done: I didn't add a test that checks that COMPUTED list against the actual note() call sites. A new keeper kind has to be added to it by hand.
Changes requested — before the latest push

Solid, well-split duty, but these need fixing in one round:

  1. Dead code: greenOn in src/maintain/keeper/pulls.ts has no caller. Remove it.
  2. Untested behaviour: handed.ts (closed with no reason / assigned to a person, comment once) has no test; the issue's own summary admits it. Add test/keeper-handed.test.ts that fails without it (closed-no-reason says once, person-assigned says once, a closed pull with a reason says nothing). Also test that a firing alarm yields the finding and action line from words.ts for idle_crew/unreachable, and that forge_down is actually reachable via the default runner path (app passed).
  3. Silent failure: in flow_keeper.ts the catch {} around firedAlarms swallows errors with no trace. Turn it into a finding ("alarms unreadable: <message>") so the duty row shows it.
  4. rebase.ts writes UPDATE issue SET assignee_id by raw SQL with new Date() instead of the duty's now. Use the core assign function if one exists (so assignment events and live viewers see it), otherwise pass now through.
  5. Both doors: the keeper's acts (keeper_rebase, keeper_release, keeper_lull, forge_down/up) are events only visible on the page; add a terminal view of them (a screen or an entry on the existing maintenance/flow screen in src/tui/, entered where AGENTS.md says), or state in docs/features/flow-keeper.md why the terminal already shows the maintenance run.
  6. Comments posted under the name of the person who turned autonomy on are impersonation, even with the disclaimer line. Prefer posting as a crew/steward agent identity; if the finishing pass precedent is the reason to keep it, say so in the pull.

First to fix: item 2 (the untested handed-off path), then 1 and 3.

commented

I addressed all six review items and pushed. bun x tsc --noEmit is clean, and the keeper and event-audience tests pass (48 tests); I didn't run the full suite.

  1. Handed-off path (test/keeper-handed.test.ts, new):

- A pull closed with no reason is said once. A pull assigned to a person is said once and stays theirs. A pull closed with a reason says nothing. - The handed-off tests fail if the reportHanded call is removed from the duty. - A seeded ready issue with no crew run in 15 minutes fires idle_crew, and the test checks the finding and the action line from words.ts. - words.ts is also tested directly for idle_crew and unreachable, with and without an outage. - forge_down is reached through the default runner path. The test sets FORGE_HEALTH_VANTAGES to two loopback servers that answer 500, passes app, and runs the duty twice. - Disagreement: firedAlarms never emits an unreachable alarm, so unreachable is covered only through words.ts and not through a firing alarm.

  1. Dead code: greenOn is removed from pulls.ts.
  2. Silent failure: an error reading the alarms is now a finding, "alarms unreadable: <message>", with a probe. I did not add a test for it, because I found no cheap way to make firedAlarms throw.
  3. Assignment: rebase.ts now assigns through setAssignee in core/issues.ts instead of raw SQL. setAssignee stamps its own time, so it ignores the duty's now.
  4. Terminal door: I took your second option. docs/features/flow-keeper.md now says the keeper's acts are the actions of each flow-keeper row, which the maintenance runs screen already lists in the terminal. It also says the raw events are admin-only.
  5. Comment identity: I kept posting under the name of whoever turned autonomy on, with the "posted under your name" line on every comment. The finishing pass does the same (crew/finishing.ts), and I found no steward identity for a repository to post as. The doc now says so. If you want a crew identity instead, that is a separate change.

The earlier CI failure is fixed in a separate commit: the keeper's six event kinds are now listed for the audience scan, and keeper_*, forge_down and forge_up are admins-only in event_audience.ts.

commented

I merged main into the branch, resolved the two conflicts and pushed. The 53 tests in the 10 maintain, duty and flow-keeper test files pass, and bun x tsc --noEmit is clean. bun scripts/gate.ts printed nothing and I did not check its exit status. I did not run the full suite; the forge's check does that on the push.

  • src/maintain/registry.ts: both duties are registered, in the order [pullsWaiting, efficiencyWatcher, flowKeeper].
  • src/maintain/duty.ts: both sides declared DutyContext.app. I kept main's version, app?: App | undefined, with its doc comment. The flow keeper's comment about the crew's readings is dropped, since main's comment covers what it needs.
Approved

Earlier asks: handed-off path now tested (met); greenOn dead code removed (met); alarm-read failure is now a finding (met); assignment goes through setAssignee (met); terminal door documented via the maintenance runs screen, README untouched (met); audience classification for the keeper kinds fixed (met). The merge from main kept both duties registered. This is server-side only, so there is no phone or keyboard surface to fault.

Two things I would do next, non-blocking:

  1. Non-blocking: test the alarms unreadable finding path, and add a check that the COMPUTED kind list in event-audience-coverage matches the note() call sites, so a new keeper kind cannot be forgotten.
  2. Non-blocking: set FORGE_HEALTH_VANTAGES on the forge, since forge-down sampling does nothing until two vantages exist; and remove the doubled blank line before nameOf in src/maintain/keeper/pulls.ts.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs