Public: anyone can read this; the crew can change it.

Batch doctor: on a red merge batch, find the culprit, tell load reds from real ones, hold with the exact failing test #7555

stopped parked by the crew opened this
opened this

Part of the epic #7553. Blocked by the maintainer framework (#7554).

Problem

When a merge batch goes red, the steward found the culprit by gating each pull alone against main, read the failing test, and decided: load red (re-check) or real red (hold, tell the builder the exact test). The forge already bisects (src/checks/queue.ts:1, #5994), re-runs budget-only reds once (src/checks/budget_retry.ts:1, #7354) and records flakes (src/checks/flake.ts:1). Missing: classifying every red the same way, holding with a precise message, and lifting the hold when the pull is green again. Today the steward greps queue logs for "[queue] gated 1 pull: red" and posts the hold itself.

Build (duty: batch-doctor)

  • Trigger: a queue gate finished red (event, not polling).
  • Classify failing tests: budget/timing (expectBudget), infra (extract/ENOENT/tarball, disk), functional. Load reds are re-checked once serially (reuse budget_retry.ts, do not duplicate) and the result recorded.
  • Culprit: use the queue's own search result; when a batch is red with no single culprit, gate each pull alone on main (canary: one first).
  • A real red alone: hold the pull out of the line and comment on it with the exact failing test, file, error line and the command to reproduce; route the rebase or fix to a builder who can push to that branch (#7006).
  • Lift the hold when a new head passes its own check; never touch a hold a person set.
  • Record batch id, culprit, class, action and the probe (the test output line) in maintenance_run.

Done when

  • A seeded red batch with one culprit ends with that pull held, a comment naming the test, and the rest of the batch landing on the next gate.
  • A seeded budget-only red is re-checked and not held; a seeded functional red alone is held.
  • A hold set by the doctor lifts when the new head is green; a person's hold is untouched.
  • Tests: test/doctor-culprit.test.ts, test/doctor-classify.test.ts (budget, infra, functional), test/doctor-hold-lift.test.ts, test/doctor-person-hold.test.ts.
  • Links, does not duplicate: #7354 (closed), #7426 (re-run unrelated failed tests alone), #4235 (queue memory).
commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

commented

Pulled forward by P0 #7615 (the owner: "much faster. and stabily"). Build the culprit finder and quarantine first, as one service in the queue; it does not need the maintainer framework to run.

commented

Parked by the crew: the check failed ("README.md stays under 60 KB") and the decider saw no point in another try try 1: README.md, docs/MCP.md, docs/features/batch-doctor.md, src/checks/doctor.ts (+15 more) try 2: AGENTS.md, README.md, docs/APPS.md, docs/CLI.md (+296 more) (pull #7619 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs