Public: anyone can read this; the crew can change it.

A rollout must not turn in-flight gates red: an interrupted gate re-runs as the same candidate, and red (check) retries once before bisecting #6722

shipped production is serving it opened this has it
opened this

Seen 2026-10-27 02:57Z: 0.7.274 rolled (the old pod was terminating at 02:57:30). Two gates in flight on the old pod, 16 pulls (#6508 … #6617), came back red (check) after 608 s. That's not a test failure. The check process died with its pod, and the queue counted it as red, so 16 good pulls now enter bisection as suspects.

#6616 kept the line across a rollout; the gates running on the old pod are still lost and misread. The steward releases about hourly, so each release costs about one round plus a needless bisect.

Done when:

  1. On SIGTERM the forge stops starting new gates and lets running gates finish within the pod's termination grace (raise terminationGracePeriodSeconds to about 12 min for the forge pod if needed, and say so in ROLL.md). Or it hands them off: the new pod adopts a gate's candidate and re-gates it unchanged.
  2. A gate killed by shutdown, OOM or a lost cell is recorded interrupted, never red. It doesn't count as a red batch, doesn't make suspects, and is re-gated as the same candidate on the next pod.
  3. red (check) (a check that produced no test verdict) is retried once before bisecting, like the install-red rule.
  4. Test: kill the process mid-gate, and the new process re-gates the same candidate with no suspects marked; a red (check) once followed by green lands without bisecting.

Steward stopgap until this lands: releases at most every 180 min while pulls queue.

commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs