Public: anyone can read this; the crew can change it.

The merge queue re-runs a batch's budget-only reds once, serially, before calling it red — budget tests under gate load are sinking 8-pull batches #7354

shipped production is serving it opened this has it
opened this

Budget tests are now the top reason batches go red, and they fail only under the gate's own load. Measured 2026-10-27 15:00–18:50Z:

  • 200 branches hold the 50 ms budget red at 15:07, 18:07, 18:39 and 18:44.
  • GET /apps/open … within its budget, a cold plan … stays under 50 ms, warm, the seed's map answers within its budget, signed-out GET /organizations/new names session and render.
  • Every one passes on main and on the pull alone. The 200-branches file passes 7/7 in 4.8 s on a quiet machine, and fails in a gate even at the 4× expectBudget cap.
  • Each red costs a batch of 8 and about 13 minutes. Landing fell from 50+/h to 13–18/h while they recur.

Done when:

  1. The gate tells a budget failure apart from a functional one. Use expectBudget failures, or a budget tag in the test name. When the only reds in a batch are budget tests, re-run just those test files once, serially (--max-concurrency=1, FORGE_TEST_QUIET=1). Pass → the batch is green, and the test is recorded in check_flake. Fail again → red as today.
  2. A functional red still bisects at once, with no retry.
  3. A test drives the queue: a batch whose only failure is a budget test that passes on retry lands. A batch with a functional failure does not.
  4. The tick line counts budget retries passed/failed per hour, so the steward can see when a budget is truly regressed.
commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs