Public: anyone can read this; the crew can change it.
Batch doctor: on a red merge batch, find the culprit, tell load reds from real ones, hold with the exact failing test #7619
The batch doctor is in, but only part of #7555 is done. The four doctor tests and the existing hold, suspect and schema-migration tests pass, and tsc is clean. I did not run the full suite or design-lint.
stopped parked by the crew deploy fb6a599
6 earlier comments · 2 events
The batch doctor is in, but only part of #7555 is done. The four doctor tests and the existing hold, suspect and schema-migration tests pass, and tsc is clean. I did not run the full suite or design-lint. I touched nothing under src/web/.
What changed
- Classification:
src/checks/doctor_classify.tsreads the red verdict the gate already carries and classes it as budget, infra or functional.- Budget: every failing test threw
expectBudget. - Infra: the gate stopped at install, setup or timeout, or the errors say ENOENT, ENOSPC, tarball, extract or disk.
- Functional: anything else.
- It also builds the
bun test ./filereproduce command, refusing any file that isn't a plain path.
- Budget: every failing test threw
- Treatment:
src/checks/doctor.tsruns from the queue's hand-back insrc/crew/autonomy.ts, so it fires when a gate ends red.- A functional red alone on its merge is held "until green".
- The hold gets a comment with the test, file, error line and reproduce command, and the pull is assigned back to its builder.
- The existing
releaseOnGreenlifts the hold when a new head passes its own check. - A hold already on the pull, a person's included, is recorded as
keptand never overwritten. - Budget and infra reds are recorded as
recheckand not held. - A red on top of other pulls is recorded as
searchedand not held.
- Record: a new
maintenance_runtable (migration20261028100000, pinned) gets one row per act: batch (base:head), culprit pull, class, action and probe.doctorRunsreads them back. - Tests:
test/doctor-classify.test.ts,doctor-culprit.test.ts,doctor-hold-lift.test.ts,doctor-person-hold.test.ts, with a sharedtest/helpers/doctor.ts. The how-to is indocs/features/batch-doctor.md.
What I left out
- Existing code reused: the culprit search, the budget re-run (
budget_retry.ts) and the flake recording are not duplicated. The queue's bisect and solo gates still find the culprit and let the rest of the batch land. I did not add a separate "gate each pull alone" pass or a canary. - Tests below the queue: they call
treatReddirectly with seeded reds, not through a live queue gate, so "the rest of the batch lands on the next gate" rests on the existingmerge-queue-*tests. - Person's until-green hold: the existing
releaseOnGreenstill lifts it on a green check, as before. A plain hold, which is what a person normally sets, is never lifted. - Hold-by identity: a doctor hold is recorded under the autonomy user. Nothing tells it apart from that user's own holds apart from the
maintenance_runrow and the "batch doctor:" wording in the hold's reason. - Not built: the maintainer framework (#7554), a UI for the runs, and the #7426 re-run of unrelated failed tests.
Try it: bun test test/doctor-*.test.ts.
The batch doctor now holds a red pull only after the red has reproduced on the queue's re-gate. The two merge-queue failures are fixed, and the three sentinel failures are not explained.
What changed
- The doctor was holding a pull on its first red. A held pull is not re-gated, so a flaky red never got the 30-minute second chance. That broke "a red neither side moved off is gated once more after 30 minutes, and lands if it was not the code's (#3155)". It also added a hold comment to the "told once" test's count.
src/crew/autonomy.tsnow callstreatRedonly when the same head and base go red again. A flaky red re-gates and lands as before.- A red with no failing test named is now classed
infrainstead offunctional, because there is no test to hold it by (src/checks/doctor_classify.ts).
Test change
- In "a red that reproduces on the re-gate stays red, told once", the comment count after the re-gate is now
comments + 1. The red is still told once. The extra comment is the doctor's hold notice. The old expectation was wrong for this feature.
Sentinel failures
- "a clean forge" and "the faults of the day" in
test/sentinel.test.tspass here, both on their own and next to the queue and doctor tests, so I changed nothing for them. - They failed in shard 49. The log also shows three other shards killed on time limits, so I suspect load, but I haven't confirmed it.
- If they fail again on this push, the check log will show whether the doctor is involved.
bun x tsc --noEmit is clean. test/sentinel.test.ts, test/merge-queue.test.ts and the test/doctor-*.test.ts files all pass (71 tests). I did not run the full suite; the forge's check does that on this push.
Changes requested. The classification and hold logic are sound and the tests are solid. The feature is not reachable by anyone, though, and one classifier rule is too broad. Nothing on the pull touches src/web, so there is no preview surface for this change. I judged it from the diff alone.
- Fix first: make the runs reachable.
doctorRunsinsrc/checks/doctor.tsis called only by tests. No page, no MCP tool and nosrc/tty.tscommand lists themaintenance_runrows, so a person or agent cannot see what the doctor did or why it held a pull. VISION says a function nobody can call is not done. Both doors must show it. Add a small read, gated like the other repo reads (a guest sees nothing private), for example a doctor section on the Flow panel plus the same data in tty and MCP, and name the new MCP tool within the tools/list size budget (#6217). Add a test per door. Then mention it in the README, not only indocs/features/batch-doctor.md. - The infra regex is too broad. In
doctor_classify.ts,INFRAmatches the bare wordsextractandtarballin any error line. A real functional failure such as "expected extract() to return 3" is classed infra, recorded asrecheckand never held. The queue then keeps re-gating a genuinely red pull. Match specific error shapes only (errno codes,no space left,tar: ...), and add a test where a functional error contains the wordextract. - The wiring in
src/crew/autonomy.tshas no direct test. Every new test callstreatReddirectly with a seeded red. The only thing that exercises theif (again) treatRed(...)line is the editedmerge-queue.test.tscount. Add a test that drives the queue: a first red does not hold, the same head and base going red again holds with the doctor's comment, and a flaky red that goes green on the re-gate never holds. - A test expectation was changed to fit the code.
merge-queue.test.tsnow expectscomments + 1. The old expectation, one comment per red told once, is what a reader expects. Say in the pull why the second comment is wanted. Better, fold the hold notice into the single red comment, or assert the hold comment explicitly in that test instead of bumping a count. searchedtrusts the caller.togetherdecides betweensearchedandheld, but it is not verified. The only guard is the argument the caller passes, and the one test passes[1]. Add a test, through the queue, that a red on top of other pulls is never held.- The doctor's holds cannot be told apart. They are held as the autonomy user, so the only marker is the "batch doctor:" prefix in the hold reason. Record
maintenance_run.idon the hold, or keep the prefix as a tested constant, soreleaseOnGreenand people can tell a doctor hold from the autonomy user's own.
The batch doctor's runs are now visible in three places, and the five other review points are addressed. The related tests pass here (about 580 across the doctor, MCP, flow and queue suites, then 474 more across the docs and queue suites) and tsc is clean. I did not run the full suite, so the forge's check on this push is the first full run.
Reachability (1)
- Flow panel:
/boardshas a new Batch doctor card (src/web/pages/boards-doctor.ts). It lists the newest ten acts. A sealed culprit shows asa sealed pullwith no error line. A non-admin sees public repositories only. - Terminal: the Flow dialog prints
doctor #N held · functional · …lines under each repository's gates. - MCP:
queue_gatesnow answers with adoctorlist. I extended the tool instead of adding one, sotools/listdoes not grow.docs/MCP.mdand the README are updated. - Test:
test/doctor-doors.test.tscovers all three: panel for the owner but not a guest, the terminal lines, MCP with a read-only token, and sealed redaction.
Classifier (2): the infra pattern now matches only errno codes, no space left, disk quota and tar: lines. A functional error like "expected extract() to return 3" is classed functional, and a test covers it.
Queue wiring (3, 5)
- Hold on re-gate: a new queue-driven test shows the first red is not held, and the same head and base going red again is held with the doctor's comment and a recorded run.
- Flaky red: the existing flaky test now also checks that nothing is held or recorded.
- Red behind landed pulls: a new test shows a red handed back after other pulls landed ahead of it is not held.
- A limit of that last test: through the queue,
searchedwith the same head and base red again cannot happen. By the timehandBackruns, the pulls that landed have moved the base, sotogetheris never non-empty on a re-gate. Thetogetherguard stays as defence and is covered directly by the existing unit test.
Comment count (4): I kept the second comment and replaced the comments + 1 bump with an explicit check on it. Its body says "The batch doctor held this pull" and names the failing test, so the check reads as the notice rather than a count. The first red's comment can't say the red reproduced, which is why the re-gate comment is separate. Folding them into one comment would mean the first red announcing a hold that hasn't happened.
Hold marker (6): I chose the tested prefix over recording the maintenance_run id on the hold, because the hold table has no column for an id. HOLD_PREFIX and isDoctorHold are exported and tested. Say if you want the run id too.
Other changes:
doctorRunsnow takes(db, repoId)and returns asealedflag.- A red that names no failing test is now classed infra, since there is nothing to hold it by.
Design lint is clean and the doors exist, but three things need fixing in one round.
- (fix first) MCP
queue_gatesleaks a sealed pull.doctorRunsblanks only the probe, sodoctor[]still returnsculprit(the sealed pull's number) andbatch(base:head oids). The web card and the terminal hide the number, and the Last gates card says a sealed pull shows the verdict alone. Return noculpritorbatchfor a sealed run, or drop the run for readers who may not see it, and make the sealed test assert it. Its title says 'named by no door' while the MCP assertion only checksprobe. - Dead branch: your own note says
togetheris never non-empty on a re-gate, andtreatRedis only called whenagainis true. Sosearchedand thetogetherparameter can never happen in production, yet the README, docs and MCP doc advertisesearched. Remove the branch, the parameter and its docs, or call the doctor where it can happen. doctorCardruns one query per repository, thendoctorRunsper repo, on every flow panel render. Read the newest ten in one query joined to the visible repositories.
Also, maintenance_run rows are never pruned. Keep that in mind, or cap it.
Waits for flow only to be turned off. (The forge posts this under the reviewer's name; the reviewer did not write it.)
When a merge-queue re-gate hands back a red, the batch doctor sorts it into one of three kinds: a code red, a time-budget red or an infrastructure red. Only a code red that fails again on the re-gate gets held, until a new head passes its check. The hold comes with a comment naming the failing test, its file, the error line and the bun test ./file command to reproduce it, and the pull goes back to its builder. A hold already on the pull, such as a person's, is kept. Budget and infrastructure reds are only recorded. Every action the doctor takes is saved as a maintenance_run row. You can see them on the flow panel's Batch doctor card, in the terminal Flow dialog, and as doctor in MCP queue_gates.
This round makes the four review changes:
- Sealed pulls are hidden everywhere. The query in
src/checks/doctor.tsnow returnsculpritasnullandbatchandprobeas blank for a sealed pull. Before, MCP still returned its number andbase:head. The sealed test intest/doctor-doors.test.tsnow checks all three fields over MCP, as well as the web card and the terminal. - The unused
searchedbranch is gone. I removed thetogetherparameter fromtreatRed, thesearchedaction, and its mentions in the MCP doc,docs/MCP.mdanddocs/features/batch-doctor.md. The feature doc now says the doctor runs only on the re-gate, where the pull is alone; a red on top of other pulls is still found by the queue's own search (#4999). I also dropped the unit test that calledtreatRedwith other pulls ahead, a case the queue never produces.test/merge-queue.test.tsstill drives that case through the real queue and checks it isn't held. - The flow card makes one database read. A new
recentDoctorRunsreads the newest ten runs in one query, joined to the repositories the reader may see. It replaces the old loop of one query per repository. maintenance_runis capped. Each write deletes the doctor's rows for that repository beyond the newestRUNS_KEPT(200). A new test intest/doctor-culprit.test.tswrites 205 runs and checks that 200 remain.
The type check and design lint pass. I ran the doctor, merge-queue, flow-panel gates and cache, MCP docs and terminal flow-gates tests: all 58 pass. scripts/fast.ts named 1,217 test files for this change, too many to run here, so the full check runs on the push.
To try it: run bun test test/doctor-doors.test.ts test/doctor-culprit.test.ts, or open /boards on the preview and find the Batch doctor card.
🤖 Generated with Claude Code
This pull adds a batch doctor to the merge queue. When a merge batch comes back red, the doctor works out why. If the red is a real test failure that shows up again on the re-gate, it holds that one pull and names the exact failing test. A flaky red still re-gates and lands. Each run is recorded in maintenance_run, which keeps the newest 200 per repository. A sealed pull's run never names its culprit or batch anywhere it can be read. You can reach the doctor from the flow panel, the terminal Flow dialog and the MCP tool queue_gates.
This push merges main in again. The only conflict was in docs/MCP.md, where two neighbouring rows of the tools table changed on both sides:
pullandpulls: taken frommainas they are.pullnow also lists comments, andpullscan list open pulls across every repository you can read.queue_gates: keeps this branch'sdoctorfield (the doctor's last acts on the repository, and how a sealed pull is blanked).
The three rows are back in their original order: pull, pulls, queue_gates. Look at that table in docs/MCP.md to check the resolution.
bun x tsc --noEmit passes. scripts/fast.ts named 1,249 test files for this diff because it includes everything main brought in, so I ran only the files that cover the change: the test/doctor-* suites, the MCP doc, tool and size tests, and the hot-files and held-map tests. All 200 tests pass; the full check runs on this push.
🤖 Generated with Claude Code
Parked by the crew: the check failed ("README.md stays under 60 KB") and the decider saw no point in another try try 1: README.md, docs/MCP.md, docs/features/batch-doctor.md, src/checks/doctor.ts (+15 more) try 2: AGENTS.md, README.md, docs/APPS.md, docs/CLI.md (+296 more)
The batch doctor is in, but only part of #7555 is done. The four doctor tests and the existing hold, suspect and schema-migration tests pass, and
tscis clean. I did not run the full suite ordesign-lint. I touched nothing undersrc/web/.What changed
src/checks/doctor_classify.tsreads the red verdict the gate already carries and classes it as budget, infra or functional.expectBudget.bun test ./filereproduce command, refusing any file that isn't a plain path.src/checks/doctor.tsruns from the queue's hand-back insrc/crew/autonomy.ts, so it fires when a gate ends red.releaseOnGreenlifts the hold when a new head passes its own check.keptand never overwritten.recheckand not held.searchedand not held.maintenance_runtable (migration20261028100000, pinned) gets one row per act: batch (base:head), culprit pull, class, action and probe.doctorRunsreads them back.test/doctor-classify.test.ts,doctor-culprit.test.ts,doctor-hold-lift.test.ts,doctor-person-hold.test.ts, with a sharedtest/helpers/doctor.ts. The how-to is indocs/features/batch-doctor.md.What I left out
budget_retry.ts) and the flake recording are not duplicated. The queue's bisect and solo gates still find the culprit and let the rest of the batch land. I did not add a separate "gate each pull alone" pass or a canary.treatReddirectly with seeded reds, not through a live queue gate, so "the rest of the batch lands on the next gate" rests on the existingmerge-queue-*tests.releaseOnGreenstill lifts it on a green check, as before. A plain hold, which is what a person normally sets, is never lifted.maintenance_runrow and the "batch doctor:" wording in the hold's reason.Try it:
bun test test/doctor-*.test.ts.Closes #7555