CI lockups: main can never silently go red (pre-push check + auto-revert), bisect by halves, reserved gate capacity, every red names its test, gate logs kept #5994

Owner, 2026-10-25: "this just sounds like more CI lockups that need to be addressed."

What locked CI up today (probes, steward):

1. A test that broke on a date. test/standards.test.ts "releases.json … in the shape a child reads" assumed the release record fits in two pages of MAX_RELEASES (100). 0.7.251's release commit took it past 200, and main went red (Expected 101, got 100).

- The cascade: 0.7.252 HELD ("the forge's own check failed"), and every queue batch built on main went red and was bisected for nothing, for about 1.5 h. - Fixed by 7fb46e6 (steward). - Nothing said so: the steward found it by reading the release log.

1. Bisects waste gates. On a red batch the queue re-forms nested prefixes: 7, 6, 4, 3, 2, 1, all containing the same suspect pulls, gated in parallel. At 21:26 five gates (412 s each) went red together over #5756 + #5757. Then #5756 alone was green (532 s). The real suspect was #5757, or the clash of the two. Six gates are spent to learn one bit.
2. Gates run 2× slower than checks. Gates take 410–600 s against 246 s for the same pull's own check, because gates and 7 pull checks share the machine.
3. Reds that name no test. Main's failed run (01m5tc5p…) had shard 258/259/260 exited 143 without naming a test (killed at a timeout, under load).
4. The gate keeps no log for this repo. keepForgeGate writes data/checks/gate-last.json only when forgeRepo === repo, and the file doesn't exist. Nobody can read why a gate went red; the steward had to re-run the pull alone to find out.
5. A red main holds every pull check for up to 20 min (by design), so (1) froze everything.

Done when:

• Main can't silently go red.

- A release commit runs main's check before it's pushed (release.sh, and the forge's own release step). - If main goes red anyway, the forge reverts the commit that did it within 5 minutes (a revert pull, auto-landed on green) and says so on home (#5930's banner). - A lint flags tests that read growing records (releases, runs, tags) with fixed-size assumptions.

• Bisect by halves, not prefixes. A red batch splits into two disjoint halves, gated in parallel, recursing only into the red half. One suspect costs log₂(n) rounds of 2 gates. A pull red alone gets held at once, with its failing test quoted on it.
• Gates have reserved capacity: they don't share slots with pull checks (a fixed share of the machine, or the second runner), so a gate takes as long as a check.
• Every red names its test. A shard killed at a timeout reports the test it was running, and retries once on a quiet slot before it counts as red. A red with no test is reported as the machine's fault, not the pull's (as the disk-floor rule already does).
• Gate logs are kept per gate (the last 50), readable from the pull and from /flow (#5930). The red line quotes the failing test.
• KPIs on /flow:

From a look at Dagger (dagger.io), 2026-10-25 (steward and a researcher). Verdict: don't adopt it. The pre-1.0 engine wants a near-privileged container, which clashes with our sandbox, and it doesn't touch queue logic. Steal its core idea instead: results addressed by their inputs.

• The key for each test shard and each gate is a hash of its inputs: the tree OID, bun.lock, the shard assignment, the Bun version and the test-runner config.
• The store keeps "passed" under that key. A batch whose merged tree hashes to a tree already gated green skips its gate. A shard whose inputs are unchanged since its last pass skips with "cached".
• What it buys: a re-formed bisect or re-gate of an identical tree costs 0 s instead of ~7 min (pain 2, part of 1).
• No new runtime: git tree OIDs already give us the hash.

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Steward, 19:05Z: the throughput killer, found in the queue log. This is the fix to do first. Landed per hour fell to 2. At 18:39 an 8-pull batch went red. The queue re-formed it as five separate candidates (2, 2, 1, 2, 1 pulls), all gated in parallel on the same main. At 18:50 four of them came back green, but only the first landed (#6314, #6369). That moved main, so the other green verdicts (#6385/#6386, #6388/#6389, #6398) were void and those pulls went back to waiting. That's about 40 check-minutes thrown away per red batch. It also re-gated #6387, which had already gone red alone.

queue.depth's own help says candidates gate "each on top of the one before" (stacked, so both land if green). The re-form after a red doesn't do that. Fix:

1. After a red, re-form as stacked candidates (cand₂ = cand₁ + its pulls, …), so every green piece lands in order, or split in two halves where the second is stacked on the first.
2. Never re-gate a pull that just went red alone on the same head until its head or base changes.
3. A test: an 8-pull batch with one bad pull lands all 7 good ones in at most 2 gate rounds.