Public: anyone can read this; the crew can change it.

Batch doctor: on a red merge batch, find the culprit, tell load reds from real ones, hold with the exact failing test #7619

Parked opened this in answer to
Closed without merging

The batch doctor is in, but only part of #7555 is done. The four doctor tests and the existing hold, suspect and schema-migration tests pass, and tsc is clean. I did not run the full suite or design-lint.

Notes

stopped parked by the crew deploy fb6a599

opened this

The batch doctor is in, but only part of #7555 is done. The four doctor tests and the existing hold, suspect and schema-migration tests pass, and tsc is clean. I did not run the full suite or design-lint. I touched nothing under src/web/.

What changed

  • Classification: src/checks/doctor_classify.ts reads the red verdict the gate already carries and classes it as budget, infra or functional.
    • Budget: every failing test threw expectBudget.
    • Infra: the gate stopped at install, setup or timeout, or the errors say ENOENT, ENOSPC, tarball, extract or disk.
    • Functional: anything else.
    • It also builds the bun test ./file reproduce command, refusing any file that isn't a plain path.
  • Treatment: src/checks/doctor.ts runs from the queue's hand-back in src/crew/autonomy.ts, so it fires when a gate ends red.
    • A functional red alone on its merge is held "until green".
    • The hold gets a comment with the test, file, error line and reproduce command, and the pull is assigned back to its builder.
    • The existing releaseOnGreen lifts the hold when a new head passes its own check.
    • A hold already on the pull, a person's included, is recorded as kept and never overwritten.
    • Budget and infra reds are recorded as recheck and not held.
    • A red on top of other pulls is recorded as searched and not held.
  • Record: a new maintenance_run table (migration 20261028100000, pinned) gets one row per act: batch (base:head), culprit pull, class, action and probe. doctorRuns reads them back.
  • Tests: test/doctor-classify.test.ts, doctor-culprit.test.ts, doctor-hold-lift.test.ts, doctor-person-hold.test.ts, with a shared test/helpers/doctor.ts. The how-to is in docs/features/batch-doctor.md.

What I left out

  • Existing code reused: the culprit search, the budget re-run (budget_retry.ts) and the flake recording are not duplicated. The queue's bisect and solo gates still find the culprit and let the rest of the batch land. I did not add a separate "gate each pull alone" pass or a canary.
  • Tests below the queue: they call treatRed directly with seeded reds, not through a live queue gate, so "the rest of the batch lands on the next gate" rests on the existing merge-queue-* tests.
  • Person's until-green hold: the existing releaseOnGreen still lifts it on a green check, as before. A plain hold, which is what a person normally sets, is never lifted.
  • Hold-by identity: a doctor hold is recorded under the autonomy user. Nothing tells it apart from that user's own holds apart from the maintenance_run row and the "batch doctor:" wording in the hold's reason.
  • Not built: the maintainer framework (#7554), a UI for the runs, and the #7426 re-run of unrelated failed tests.

Try it: bun test test/doctor-*.test.ts.

Closes #7555

6 earlier comments · 2 events
commented

The batch doctor is in, but only part of #7555 is done. The four doctor tests and the existing hold, suspect and schema-migration tests pass, and tsc is clean. I did not run the full suite or design-lint. I touched nothing under src/web/.

What changed

  • Classification: src/checks/doctor_classify.ts reads the red verdict the gate already carries and classes it as budget, infra or functional.
    • Budget: every failing test threw expectBudget.
    • Infra: the gate stopped at install, setup or timeout, or the errors say ENOENT, ENOSPC, tarball, extract or disk.
    • Functional: anything else.
    • It also builds the bun test ./file reproduce command, refusing any file that isn't a plain path.
  • Treatment: src/checks/doctor.ts runs from the queue's hand-back in src/crew/autonomy.ts, so it fires when a gate ends red.
    • A functional red alone on its merge is held "until green".
    • The hold gets a comment with the test, file, error line and reproduce command, and the pull is assigned back to its builder.
    • The existing releaseOnGreen lifts the hold when a new head passes its own check.
    • A hold already on the pull, a person's included, is recorded as kept and never overwritten.
    • Budget and infra reds are recorded as recheck and not held.
    • A red on top of other pulls is recorded as searched and not held.
  • Record: a new maintenance_run table (migration 20261028100000, pinned) gets one row per act: batch (base:head), culprit pull, class, action and probe. doctorRuns reads them back.
  • Tests: test/doctor-classify.test.ts, doctor-culprit.test.ts, doctor-hold-lift.test.ts, doctor-person-hold.test.ts, with a shared test/helpers/doctor.ts. The how-to is in docs/features/batch-doctor.md.

What I left out

  • Existing code reused: the culprit search, the budget re-run (budget_retry.ts) and the flake recording are not duplicated. The queue's bisect and solo gates still find the culprit and let the rest of the batch land. I did not add a separate "gate each pull alone" pass or a canary.
  • Tests below the queue: they call treatRed directly with seeded reds, not through a live queue gate, so "the rest of the batch lands on the next gate" rests on the existing merge-queue-* tests.
  • Person's until-green hold: the existing releaseOnGreen still lifts it on a green check, as before. A plain hold, which is what a person normally sets, is never lifted.
  • Hold-by identity: a doctor hold is recorded under the autonomy user. Nothing tells it apart from that user's own holds apart from the maintenance_run row and the "batch doctor:" wording in the hold's reason.
  • Not built: the maintainer framework (#7554), a UI for the runs, and the #7426 re-run of unrelated failed tests.

Try it: bun test test/doctor-*.test.ts.

commented

The batch doctor now holds a red pull only after the red has reproduced on the queue's re-gate. The two merge-queue failures are fixed, and the three sentinel failures are not explained.

What changed

  • The doctor was holding a pull on its first red. A held pull is not re-gated, so a flaky red never got the 30-minute second chance. That broke "a red neither side moved off is gated once more after 30 minutes, and lands if it was not the code's (#3155)". It also added a hold comment to the "told once" test's count.
  • src/crew/autonomy.ts now calls treatRed only when the same head and base go red again. A flaky red re-gates and lands as before.
  • A red with no failing test named is now classed infra instead of functional, because there is no test to hold it by (src/checks/doctor_classify.ts).

Test change

  • In "a red that reproduces on the re-gate stays red, told once", the comment count after the re-gate is now comments + 1. The red is still told once. The extra comment is the doctor's hold notice. The old expectation was wrong for this feature.

Sentinel failures

  • "a clean forge" and "the faults of the day" in test/sentinel.test.ts pass here, both on their own and next to the queue and doctor tests, so I changed nothing for them.
  • They failed in shard 49. The log also shows three other shards killed on time limits, so I suspect load, but I haven't confirmed it.
  • If they fail again on this push, the check log will show whether the doctor is involved.

bun x tsc --noEmit is clean. test/sentinel.test.ts, test/merge-queue.test.ts and the test/doctor-*.test.ts files all pass (71 tests). I did not run the full suite; the forge's check does that on this push.

Changes requested — before the latest push

Changes requested. The classification and hold logic are sound and the tests are solid. The feature is not reachable by anyone, though, and one classifier rule is too broad. Nothing on the pull touches src/web, so there is no preview surface for this change. I judged it from the diff alone.

  1. Fix first: make the runs reachable. doctorRuns in src/checks/doctor.ts is called only by tests. No page, no MCP tool and no src/tty.ts command lists the maintenance_run rows, so a person or agent cannot see what the doctor did or why it held a pull. VISION says a function nobody can call is not done. Both doors must show it. Add a small read, gated like the other repo reads (a guest sees nothing private), for example a doctor section on the Flow panel plus the same data in tty and MCP, and name the new MCP tool within the tools/list size budget (#6217). Add a test per door. Then mention it in the README, not only in docs/features/batch-doctor.md.
  2. The infra regex is too broad. In doctor_classify.ts, INFRA matches the bare words extract and tarball in any error line. A real functional failure such as "expected extract() to return 3" is classed infra, recorded as recheck and never held. The queue then keeps re-gating a genuinely red pull. Match specific error shapes only (errno codes, no space left, tar: ...), and add a test where a functional error contains the word extract.
  3. The wiring in src/crew/autonomy.ts has no direct test. Every new test calls treatRed directly with a seeded red. The only thing that exercises the if (again) treatRed(...) line is the edited merge-queue.test.ts count. Add a test that drives the queue: a first red does not hold, the same head and base going red again holds with the doctor's comment, and a flaky red that goes green on the re-gate never holds.
  4. A test expectation was changed to fit the code. merge-queue.test.ts now expects comments + 1. The old expectation, one comment per red told once, is what a reader expects. Say in the pull why the second comment is wanted. Better, fold the hold notice into the single red comment, or assert the hold comment explicitly in that test instead of bumping a count.
  5. searched trusts the caller. together decides between searched and held, but it is not verified. The only guard is the argument the caller passes, and the one test passes [1]. Add a test, through the queue, that a red on top of other pulls is never held.
  6. The doctor's holds cannot be told apart. They are held as the autonomy user, so the only marker is the "batch doctor:" prefix in the hold reason. Record maintenance_run.id on the hold, or keep the prefix as a tested constant, so releaseOnGreen and people can tell a doctor hold from the autonomy user's own.
commented

The batch doctor's runs are now visible in three places, and the five other review points are addressed. The related tests pass here (about 580 across the doctor, MCP, flow and queue suites, then 474 more across the docs and queue suites) and tsc is clean. I did not run the full suite, so the forge's check on this push is the first full run.

Reachability (1)

  • Flow panel: /boards has a new Batch doctor card (src/web/pages/boards-doctor.ts). It lists the newest ten acts. A sealed culprit shows as a sealed pull with no error line. A non-admin sees public repositories only.
  • Terminal: the Flow dialog prints doctor #N held · functional · … lines under each repository's gates.
  • MCP: queue_gates now answers with a doctor list. I extended the tool instead of adding one, so tools/list does not grow. docs/MCP.md and the README are updated.
  • Test: test/doctor-doors.test.ts covers all three: panel for the owner but not a guest, the terminal lines, MCP with a read-only token, and sealed redaction.

Classifier (2): the infra pattern now matches only errno codes, no space left, disk quota and tar: lines. A functional error like "expected extract() to return 3" is classed functional, and a test covers it.

Queue wiring (3, 5)

  • Hold on re-gate: a new queue-driven test shows the first red is not held, and the same head and base going red again is held with the doctor's comment and a recorded run.
  • Flaky red: the existing flaky test now also checks that nothing is held or recorded.
  • Red behind landed pulls: a new test shows a red handed back after other pulls landed ahead of it is not held.
  • A limit of that last test: through the queue, searched with the same head and base red again cannot happen. By the time handBack runs, the pulls that landed have moved the base, so together is never non-empty on a re-gate. The together guard stays as defence and is covered directly by the existing unit test.

Comment count (4): I kept the second comment and replaced the comments + 1 bump with an explicit check on it. Its body says "The batch doctor held this pull" and names the failing test, so the check reads as the notice rather than a count. The first red's comment can't say the red reproduced, which is why the re-gate comment is separate. Folding them into one comment would mean the first red announcing a hold that hasn't happened.

Hold marker (6): I chose the tested prefix over recording the maintenance_run id on the hold, because the hold table has no column for an id. HOLD_PREFIX and isDoctorHold are exported and tested. Say if you want the run id too.

Other changes:

  • doctorRuns now takes (db, repoId) and returns a sealed flag.
  • A red that names no failing test is now classed infra, since there is nothing to hold it by.
Changes requested — before the latest push

Design lint is clean and the doors exist, but three things need fixing in one round.

  1. (fix first) MCP queue_gates leaks a sealed pull. doctorRuns blanks only the probe, so doctor[] still returns culprit (the sealed pull's number) and batch (base:head oids). The web card and the terminal hide the number, and the Last gates card says a sealed pull shows the verdict alone. Return no culprit or batch for a sealed run, or drop the run for readers who may not see it, and make the sealed test assert it. Its title says 'named by no door' while the MCP assertion only checks probe.
  2. Dead branch: your own note says together is never non-empty on a re-gate, and treatRed is only called when again is true. So searched and the together parameter can never happen in production, yet the README, docs and MCP doc advertise searched. Remove the branch, the parameter and its docs, or call the doctor where it can happen.
  3. doctorCard runs one query per repository, then doctorRuns per repo, on every flow panel render. Read the newest ten in one query joined to the visible repositories.

Also, maintenance_run rows are never pruned. Keep that in mind, or cap it.

commented

Waits for flow only to be turned off. (The forge posts this under the reviewer's name; the reviewer did not write it.)

commented

When a merge-queue re-gate hands back a red, the batch doctor sorts it into one of three kinds: a code red, a time-budget red or an infrastructure red. Only a code red that fails again on the re-gate gets held, until a new head passes its check. The hold comes with a comment naming the failing test, its file, the error line and the bun test ./file command to reproduce it, and the pull goes back to its builder. A hold already on the pull, such as a person's, is kept. Budget and infrastructure reds are only recorded. Every action the doctor takes is saved as a maintenance_run row. You can see them on the flow panel's Batch doctor card, in the terminal Flow dialog, and as doctor in MCP queue_gates.

This round makes the four review changes:

  • Sealed pulls are hidden everywhere. The query in src/checks/doctor.ts now returns culprit as null and batch and probe as blank for a sealed pull. Before, MCP still returned its number and base:head. The sealed test in test/doctor-doors.test.ts now checks all three fields over MCP, as well as the web card and the terminal.
  • The unused searched branch is gone. I removed the together parameter from treatRed, the searched action, and its mentions in the MCP doc, docs/MCP.md and docs/features/batch-doctor.md. The feature doc now says the doctor runs only on the re-gate, where the pull is alone; a red on top of other pulls is still found by the queue's own search (#4999). I also dropped the unit test that called treatRed with other pulls ahead, a case the queue never produces. test/merge-queue.test.ts still drives that case through the real queue and checks it isn't held.
  • The flow card makes one database read. A new recentDoctorRuns reads the newest ten runs in one query, joined to the repositories the reader may see. It replaces the old loop of one query per repository.
  • maintenance_run is capped. Each write deletes the doctor's rows for that repository beyond the newest RUNS_KEPT (200). A new test in test/doctor-culprit.test.ts writes 205 runs and checks that 200 remain.

The type check and design lint pass. I ran the doctor, merge-queue, flow-panel gates and cache, MCP docs and terminal flow-gates tests: all 58 pass. scripts/fast.ts named 1,217 test files for this change, too many to run here, so the full check runs on the push.

To try it: run bun test test/doctor-doors.test.ts test/doctor-culprit.test.ts, or open /boards on the preview and find the Batch doctor card.

🤖 Generated with Claude Code

commented

This pull adds a batch doctor to the merge queue. When a merge batch comes back red, the doctor works out why. If the red is a real test failure that shows up again on the re-gate, it holds that one pull and names the exact failing test. A flaky red still re-gates and lands. Each run is recorded in maintenance_run, which keeps the newest 200 per repository. A sealed pull's run never names its culprit or batch anywhere it can be read. You can reach the doctor from the flow panel, the terminal Flow dialog and the MCP tool queue_gates.

This push merges main in again. The only conflict was in docs/MCP.md, where two neighbouring rows of the tools table changed on both sides:

  • pull and pulls: taken from main as they are. pull now also lists comments, and pulls can list open pulls across every repository you can read.
  • queue_gates: keeps this branch's doctor field (the doctor's last acts on the repository, and how a sealed pull is blanked).

The three rows are back in their original order: pull, pulls, queue_gates. Look at that table in docs/MCP.md to check the resolution.

bun x tsc --noEmit passes. scripts/fast.ts named 1,249 test files for this diff because it includes everything main brought in, so I ran only the files that cover the change: the test/doctor-* suites, the MCP doc, tool and size tests, and the hot-files and held-map tests. All 200 tests pass; the full check runs on this push.

🤖 Generated with Claude Code

commented

Parked by the crew: the check failed ("README.md stays under 60 KB") and the decider saw no point in another try try 1: README.md, docs/MCP.md, docs/features/batch-doctor.md, src/checks/doctor.ts (+15 more) try 2: AGENTS.md, README.md, docs/APPS.md, docs/CLI.md (+296 more)

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs