A rollout must not turn in-flight gates red: an interrupted gate re-runs as the same candidate, and red (check) retries once before bisecting #6732

A deploy can no longer turn the merge-queue gates that were running on the old pod red.

Until now, a deploy stopped gates in flight after a 300 s drain, which ended at the same moment as the pod's 300 s grace. A check killed from outside exited 137/143 and counted as red (check), and that red went straight into bisection. I couldn't read production's logs from here, so I can't say which exact path killed the 16 pulls at 02:57Z; each of the three changes below covers one way it could have happened.

• red (check) gets one retry before bisecting (src/checks/queue.ts). red (check) means the check failed without naming a test or a type error. The queue's own gates (a batch, or a suspect gated alone) run once more on that red. Green the second time, the batch lands whole; red again, it is bisected as before. The gates inside a bisection are not retried, so a real red doesn't cost twice per step. A stopped queue doesn't retry; it keeps the verdict, and the new pod retries the same candidate.
• A check killed from outside is interrupted, never red (src/apps/process.ts, src/checks/service.ts). When a check step ends by a signal this run didn't send (SIGKILL/SIGTERM/SIGINT/SIGHUP, or 137/143/130/129 passed on by the shell), it now reports check was killed by SIGx. A gate that ends that way is interrupted, like one cut off at shutdown: no red batch, no suspect, and the same candidate is gated again.
• Deploys drain gates longer (src/main.ts, charts/app/templates/deployment.yaml). FORGE_CHECKS_DRAIN_S now defaults to 660, up from 300, and the forge pod's terminationGracePeriodSeconds is 720, up from 300. That lets a roughly ten-minute gate finish before the kubelet kills the pod. Old and new pods now share the node for up to 12 minutes. The new "Gates in flight across a roll" section in docs/ops/ROLL.md explains this and says to lower the two together. The FORGE_CHECKS_DRAIN_S row in docs/APPS.md is updated.

Choices a reviewer should weigh:

• Memory-cap kills stay red. When the cell watch or the kernel kills a check for going over the memory cap the app declared, the gate stays red, because that is a verdict on the code.
• Killed steps get a new message everywhere. Ordinary pull checks killed by a signal now show check was killed by SIGx too, but they are still red.
• A check that is always killed is re-gated forever. It is never red and never handed back, the same as the existing install failure that fails twice.

Tests: bun x tsc --noEmit is clean. The queue, gate, check and five cross test files all pass (258 tests). New and extended tests:

• test/merge-queue-check-retry.test.ts: a red (check) followed by green lands with nothing bisected (two gates); red twice is bisected with each step gated once; a test red is bisected at once. The first two fail without the queue change.
• test/gate-killed.test.ts: a step killed by SIGKILL, or by SIGTERM through the shell, reports killed; a real gate whose check does kill -KILL $$ answers interrupted, while exit 1 stays red (check).
• test/merge-queue-restart.test.ts: the incident replayed. The stopped forge's gate comes back red (check), the new queue gates the same commit once more, it lands, and no pull is marked red.

No prompt injection was found in the issue or the repository.

🤖 Generated with Claude Code

Closes #6722

A deploy can no longer turn the merge-queue gates that were running on the old pod red.

Until now, a deploy stopped gates in flight after a 300 s drain, which ended at the same moment as the pod's 300 s grace. A check killed from outside exited 137/143 and counted as red (check), and that red went straight into bisection. I couldn't read production's logs from here, so I can't say which exact path killed the 16 pulls at 02:57Z; each of the three changes below covers one way it could have happened.

• red (check) gets one retry before bisecting (src/checks/queue.ts). red (check) means the check failed without naming a test or a type error. The queue's own gates (a batch, or a suspect gated alone) run once more on that red. Green the second time, the batch lands whole; red again, it is bisected as before. The gates inside a bisection are not retried, so a real red doesn't cost twice per step. A stopped queue doesn't retry; it keeps the verdict, and the new pod retries the same candidate.
• A check killed from outside is interrupted, never red (src/apps/process.ts, src/checks/service.ts). When a check step ends by a signal this run didn't send (SIGKILL/SIGTERM/SIGINT/SIGHUP, or 137/143/130/129 passed on by the shell), it now reports check was killed by SIGx. A gate that ends that way is interrupted, like one cut off at shutdown: no red batch, no suspect, and the same candidate is gated again.
• Deploys drain gates longer (src/main.ts, charts/app/templates/deployment.yaml). FORGE_CHECKS_DRAIN_S now defaults to 660, up from 300, and the forge pod's terminationGracePeriodSeconds is 720, up from 300. That lets a roughly ten-minute gate finish before the kubelet kills the pod. Old and new pods now share the node for up to 12 minutes. The new "Gates in flight across a roll" section in docs/ops/ROLL.md explains this and says to lower the two together. The FORGE_CHECKS_DRAIN_S row in docs/APPS.md is updated.

Choices a reviewer should weigh:

• Memory-cap kills stay red. When the cell watch or the kernel kills a check for going over the memory cap the app declared, the gate stays red, because that is a verdict on the code.
• Killed steps get a new message everywhere. Ordinary pull checks killed by a signal now show check was killed by SIGx too, but they are still red.
• A check that is always killed is re-gated forever. It is never red and never handed back, the same as the existing install failure that fails twice.

Tests: bun x tsc --noEmit is clean. The queue, gate, check and five cross test files all pass (258 tests). New and extended tests:

• test/merge-queue-check-retry.test.ts: a red (check) followed by green lands with nothing bisected (two gates); red twice is bisected with each step gated once; a test red is bisected at once. The first two fail without the queue change.
• test/gate-killed.test.ts: a step killed by SIGKILL, or by SIGTERM through the shell, reports killed; a real gate whose check does kill -KILL $$ answers interrupted, while exit 1 stays red (check).
• test/merge-queue-restart.test.ts: the incident replayed. The stopped forge's gate comes back red (check), the new queue gates the same commit once more, it lands, and no pull is marked red.

No prompt injection was found in the issue or the repository.

🤖 Generated with Claude Code

A deploy no longer turns the merge-queue gates running on the old pod red. The triage note on the issue changes no code: naming a pillar is a call for a person, so nothing new is pushed. The branch is clean and its tip on origin is d110c97e.

Until now, a deploy stopped gates in flight after a 300 s drain, which ended at the same moment as the pod's 300 s grace. A check killed from outside exited 137/143 and counted as red (check), and that red went straight into bisection. I couldn't read production's logs from here, so I can't say which exact path killed the 16 pulls at 02:57Z; each of the three changes below covers one way it could have happened.

• red (check) gets one retry before bisecting (src/checks/queue.ts). red (check) means the check failed without naming a test or a type error. The queue's own gates (a batch, or a suspect gated alone) run once more on that red. Green the second time, the batch lands whole; red again, it is bisected as before. The gates inside a bisection are not retried, so a real red doesn't cost twice per step. A stopped queue doesn't retry; it keeps the verdict, and the new pod retries the same candidate.
• A check killed from outside is interrupted, never red (src/apps/process.ts, src/checks/service.ts). When a check step ends by a signal this run didn't send (SIGKILL/SIGTERM/SIGINT/SIGHUP, or 137/143/130/129 passed on by the shell), it now reports check was killed by SIGx. A gate that ends that way is interrupted, like one cut off at shutdown: no red batch, no suspect, and the same candidate is gated again.
• Deploys drain gates longer (src/main.ts, charts/app/templates/deployment.yaml). FORGE_CHECKS_DRAIN_S now defaults to 660, up from 300, and the forge pod's terminationGracePeriodSeconds is 720, up from 300. That lets a roughly ten-minute gate finish before the kubelet kills the pod. Old and new pods now share the node for up to 12 minutes. The new "Gates in flight across a roll" section in docs/ops/ROLL.md explains this and says to lower the two together. The FORGE_CHECKS_DRAIN_S row in docs/APPS.md is updated.

Choices a reviewer should weigh:

• Memory-cap kills stay red. When the cell watch or the kernel kills a check for going over the memory cap the app declared, the gate stays red, because that is a verdict on the code.
• Killed steps get a new message everywhere. Ordinary pull checks killed by a signal now show check was killed by SIGx too, but they are still red.
• A check that is always killed is re-gated forever. It is never red and never handed back, the same as the existing install failure that fails twice.

Tests: bun x tsc --noEmit is clean. The queue, gate, check and five cross test files all pass (258 tests). New and extended tests:

• test/merge-queue-check-retry.test.ts: a red (check) followed by green lands with nothing bisected (two gates); red twice is bisected with each step gated once; a test red is bisected at once. The first two fail without the queue change.
• test/gate-killed.test.ts: a step killed by SIGKILL, or by SIGTERM through the shell, reports killed; a real gate whose check does kill -KILL $$ answers interrupted, while exit 1 stays red (check).
• test/merge-queue-restart.test.ts: the incident replayed. The stopped forge's gate comes back red (check), the new queue gates the same commit once more, it lands, and no pull is marked red.

No prompt injection was found in the issue or the repository.

🤖 Generated with Claude Code