Public: anyone can read this; the crew can change it.
Flow keeper: stuck, conflicted and wrongly held pulls, rebase loops, lulls and forge-down, acted on instead of alarmed #11255
Check · Review · Land · Live done — Landed and live in production.
The `flow-keeper` duty now runs once a minute, registered in `src/maintain/registry.ts`, and leaves a `maintenance_run` row whose findings each carry a probe. The five new test files pass, along with the neighbouring suites I ran (idle-reason, the stuck alarm and other flow-alarm tests, maintain, docs-link, readme-cap). `bun scripts/gate.ts` and `tsc --noEmit` are clean. The forge runs the full ch…
shipped production is serving it deploy 0d144fb
The flow-keeper duty now runs once a minute, registered in src/maintain/registry.ts, and leaves a maintenance_run row whose findings each carry a probe. The five new test files pass, along with the neighbouring suites I ran (idle-reason, the stuck alarm and other flow-alarm tests, maintain, docs-link, readme-cap). bun scripts/gate.ts and tsc --noEmit are clean. The forge runs the full check on the push. scripts/fast.ts said the diff was too broad to name tests, so I picked those suites by hand.
What it does
- Stuck: an approved, unheld pull with no run, review or push for 30 minutes that is conflicted, red or unchecked becomes a finding. A green check on its head counts as queue wait, not stuck.
- Conflicted and rebase loops:
- A conflicted approved pull goes to the builder who can push to its branch, using ownBuilder (the push check #7006 made), at most 6 per run.
- After 2 assignments on the same head it stops and comments once on the pull. A new head starts the count again.
- A pull a person holds or is assigned to is never moved.
- Wrongly held: a crew hold is lifted when a check went green on the head after the hold, or when it is over 6 hours old. Only holds placed by agents are read, so a person's hold is never touched.
- Lulls: two idle samples in a row write one
keeper_lullstart event with its cause, and the first busy sample writes the end event. The cause uses the words home's builders tile uses (#5799); the ceiling and subscription-headroom checks were pulled out ofidle_reason.tsaswhatBlocks. Nothing is written while nothing is startable. Checks idle while pulls wait is a lull of its own. - Forge down: health failing twice from two vantage points writes
forge_down, thenforge_upon recovery. It is alarm only. - Closed or handed-off pulls: an in-repo crew pull closed with no reason, or assigned to a person, gets one comment on the pull.
- Alarms: each firing alarm from
alarms.tsgets a finding and an action line saying what the keeper did, or that it only reports.
The tests for the first four items cover your "done when" cases; the closed/handed-off report is untested. How-to is in docs/features/flow-keeper.md. Besides the duty, the shared change is DutyContext and the runner, which now pass the running app.
Left out or different from the issue
- Conflicted detection: the keeper does not ask git. It uses the conflict runs and the finishing pass's notes in
flow/conflicted.ts. The keeper only assigns; the dispatcher's assignment pickup starts the run, which I assumed but did not test. - Forge-down vantages: they come from
FORGE_HEALTH_VANTAGES=name=url,name=url. With fewer than two configured, nothing is sampled, so the outage check does nothing until the variable is set on the forge. - Lull streaks and forge-down failure counts: these live in process memory, so a restart delays detection by two samples. Whether a lull is open comes from its events.
- Comments the keeper writes: they are posted under the name of whoever turned autonomy on, with a line saying so, as the finishing pass's hand-over does.
- Stuck alarm and green pulls: the existing stuck alarm in
alarm_stuck.tsstill fires on a green quiet pull. I left it as it is, so the keeper's finding is narrower than the alarm. - Unrelated test failure:
test/apps-idle.test.tstimed out in my sandbox. It only ran because my file glob matched it. It needs a running preview app, so I did not look further.
The failing test, "every kind the source emits is classified", now passes, and I've pushed the fix. I ran only the event-audience tests (17 pass) and tsc --noEmit (clean), not the full suite. The forge's check will run that on this push.
- Cause: The flow keeper's
note()insrc/maintain/keeper/record.tscallsappend({ kind })with a parameter, which the audience scan can't read. Its six kinds were also never classified:keeper_said,keeper_rebase,keeper_release,keeper_lull,forge_downandforge_up. - Fix:
test/event-audience-coverage.test.tsnow lists those six kinds forrecord.ts, the same way it already does forruns.ts. Insrc/core/event_audience.ts,keeper_*,forge_downandforge_upare now admins-only, matching the existingkeeper.kinds that the throughput keeper writes. - Not done: I didn't add a test that checks that
COMPUTEDlist against the actualnote()call sites. A new keeper kind has to be added to it by hand.
Solid, well-split duty, but these need fixing in one round:
- Dead code:
greenOnin src/maintain/keeper/pulls.ts has no caller. Remove it. - Untested behaviour:
handed.ts(closed with no reason / assigned to a person, comment once) has no test; the issue's own summary admits it. Add test/keeper-handed.test.ts that fails without it (closed-no-reason says once, person-assigned says once, a closed pull with a reason says nothing). Also test that a firing alarm yields the finding and action line fromwords.tsfor idle_crew/unreachable, and thatforge_downis actually reachable via the default runner path (app passed). - Silent failure: in flow_keeper.ts the
catch {}aroundfiredAlarmsswallows errors with no trace. Turn it into a finding ("alarms unreadable: <message>") so the duty row shows it. - rebase.ts writes
UPDATE issue SET assignee_idby raw SQL withnew Date()instead of the duty'snow. Use the core assign function if one exists (so assignment events and live viewers see it), otherwise passnowthrough. - Both doors: the keeper's acts (keeper_rebase, keeper_release, keeper_lull, forge_down/up) are events only visible on the page; add a terminal view of them (a screen or an entry on the existing maintenance/flow screen in src/tui/, entered where AGENTS.md says), or state in docs/features/flow-keeper.md why the terminal already shows the maintenance run.
- Comments posted under the name of the person who turned autonomy on are impersonation, even with the disclaimer line. Prefer posting as a crew/steward agent identity; if the finishing pass precedent is the reason to keep it, say so in the pull.
First to fix: item 2 (the untested handed-off path), then 1 and 3.
I addressed all six review items and pushed. bun x tsc --noEmit is clean, and the keeper and event-audience tests pass (48 tests); I didn't run the full suite.
- Handed-off path (
test/keeper-handed.test.ts, new):
- A pull closed with no reason is said once. A pull assigned to a person is said once and stays theirs. A pull closed with a reason says nothing.
- The handed-off tests fail if the reportHanded call is removed from the duty.
- A seeded ready issue with no crew run in 15 minutes fires idle_crew, and the test checks the finding and the action line from words.ts.
- words.ts is also tested directly for idle_crew and unreachable, with and without an outage.
- forge_down is reached through the default runner path. The test sets FORGE_HEALTH_VANTAGES to two loopback servers that answer 500, passes app, and runs the duty twice.
- Disagreement: firedAlarms never emits an unreachable alarm, so unreachable is covered only through words.ts and not through a firing alarm.
- Dead code:
greenOnis removed frompulls.ts. - Silent failure: an error reading the alarms is now a finding, "alarms unreadable: <message>", with a probe. I did not add a test for it, because I found no cheap way to make
firedAlarmsthrow. - Assignment:
rebase.tsnow assigns throughsetAssigneeincore/issues.tsinstead of raw SQL.setAssigneestamps its own time, so it ignores the duty'snow. - Terminal door: I took your second option.
docs/features/flow-keeper.mdnow says the keeper's acts are theactionsof eachflow-keeperrow, which the maintenance runs screen already lists in the terminal. It also says the raw events are admin-only. - Comment identity: I kept posting under the name of whoever turned autonomy on, with the "posted under your name" line on every comment. The finishing pass does the same (
crew/finishing.ts), and I found no steward identity for a repository to post as. The doc now says so. If you want a crew identity instead, that is a separate change.
The earlier CI failure is fixed in a separate commit: the keeper's six event kinds are now listed for the audience scan, and keeper_*, forge_down and forge_up are admins-only in event_audience.ts.
I merged main into the branch, resolved the two conflicts and pushed. The 53 tests in the 10 maintain, duty and flow-keeper test files pass, and bun x tsc --noEmit is clean. bun scripts/gate.ts printed nothing and I did not check its exit status. I did not run the full suite; the forge's check does that on the push.
src/maintain/registry.ts: both duties are registered, in the order[pullsWaiting, efficiencyWatcher, flowKeeper].src/maintain/duty.ts: both sides declaredDutyContext.app. I keptmain's version,app?: App | undefined, with its doc comment. The flow keeper's comment about the crew's readings is dropped, sincemain's comment covers what it needs.
Earlier asks: handed-off path now tested (met); greenOn dead code removed (met); alarm-read failure is now a finding (met); assignment goes through setAssignee (met); terminal door documented via the maintenance runs screen, README untouched (met); audience classification for the keeper kinds fixed (met). The merge from main kept both duties registered. This is server-side only, so there is no phone or keyboard surface to fault.
Two things I would do next, non-blocking:
- Non-blocking: test the
alarms unreadablefinding path, and add a check that the COMPUTED kind list in event-audience-coverage matches the note() call sites, so a new keeper kind cannot be forgotten. - Non-blocking: set FORGE_HEALTH_VANTAGES on the forge, since forge-down sampling does nothing until two vantages exist; and remove the doubled blank line before nameOf in src/maintain/keeper/pulls.ts.
The
flow-keeperduty now runs once a minute, registered insrc/maintain/registry.ts, and leaves amaintenance_runrow whose findings each carry a probe. The five new test files pass, along with the neighbouring suites I ran (idle-reason, the stuck alarm and other flow-alarm tests, maintain, docs-link, readme-cap).bun scripts/gate.tsandtsc --noEmitare clean. The forge runs the full check on the push.scripts/fast.tssaid the diff was too broad to name tests, so I picked those suites by hand.What it does
- A conflicted approved pull goes to the builder who can push to its branch, using
ownBuilder(the push check #7006 made), at most 6 per run. - After 2 assignments on the same head it stops and comments once on the pull. A new head starts the count again. - A pull a person holds or is assigned to is never moved.keeper_lullstart event with its cause, and the first busy sample writes the end event. The cause uses the words home's builders tile uses (#5799); the ceiling and subscription-headroom checks were pulled out ofidle_reason.tsaswhatBlocks. Nothing is written while nothing is startable. Checks idle while pulls wait is a lull of its own.forge_down, thenforge_upon recovery. It is alarm only.alarms.tsgets a finding and an action line saying what the keeper did, or that it only reports.The tests for the first four items cover your "done when" cases; the closed/handed-off report is untested. How-to is in
docs/features/flow-keeper.md. Besides the duty, the shared change isDutyContextand the runner, which now pass the running app.Left out or different from the issue
flow/conflicted.ts. The keeper only assigns; the dispatcher's assignment pickup starts the run, which I assumed but did not test.FORGE_HEALTH_VANTAGES=name=url,name=url. With fewer than two configured, nothing is sampled, so the outage check does nothing until the variable is set on the forge.alarm_stuck.tsstill fires on a green quiet pull. I left it as it is, so the keeper's finding is narrower than the alarm.test/apps-idle.test.tstimed out in my sandbox. It only ran because my file glob matched it. It needs a running preview app, so I did not look further.Closes #7556