Public: anyone can read this; the crew can change it.

Rule watcher: gate-only reds, rule breakers with exact fixes, flaky and load-sensitive budget tests #7560

shipped production is serving it opened this has it
opened this

Part of the epic #7553. Blocked by the maintainer framework (#7554).

Problem

Three classes of test failure waste the crew's time and are found by the steward: repo rules that only fail in the gate (#7138, closed: raw ms budgets and unpinned migrations got through a pull's own check), flaky tests (check_flake, src/checks/flake.ts:1, which files one issue at 2 flakes a week), and budget tests that fail only under gate load (#7354, closed; budget_retry.ts). Each fix was one-off; nothing watches the trend, and holds with exact fix instructions are written by hand.

Build (duty: rule-watcher)

  • Gate-only reds: a test that failed in a merge-queue gate but passed in its pull's own check on the same code is recorded as gate-only; two in a week files one issue naming the test and what differs (load, ordering, repo-wide scan), and says whether the rule is in the pull check's impact set (the #7138 class).
  • Rule breakers: when a rule test fails on a pull, comment once with the exact rule, file and line, and the fix (for example use expectBudget, add the migration pin), taken from one small table with an entry per rule. Never a vague "tests failed".
  • Flake trend: a weekly top list from check_flake with counts and the gate minutes each delayed (from #7562); the worst three get issues the crew builds. Quarantine is not allowed: the fix is a deterministic or budget-aware test.
  • Budget tests under load: if the same budget test is re-run and passes 3 times in a week, file a headroom issue with its measured p95 (links #7354).

Done when

  • A seeded gate-only red twice in a week files one issue; once files none.
  • A seeded rule breaker gets one comment with rule, location and fix; pushing again does not repeat an identical comment.
  • The weekly flake summary lists tests ordered by wasted gate minutes.
  • Tests: test/rules-gate-only.test.ts, test/rules-breaker-comment.test.ts, test/rules-flake-trend.test.ts, test/rules-budget-rerun-count.test.ts.
  • Links, does not duplicate: #7138, #7354, #7426 (open), #5642, #6643.
commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs