The merge queue survives a rollout: gates in flight resume on the new pod instead of starting over #6616

The merge queue now remembers the gate in flight across a restart: the new process resumes the same candidate instead of planning the line again. With a gate held open across a simulated restart, the candidate lands with one gate run, not two. Type check is clean; 332 tests across the merge-queue, gate and checks suites, the schema tests and the five cross files pass.

What's kept (new migration 20261026215105-queue-gate.ts, read and written through the new src/checks/queue_store.ts)

• queue_gate: one row per gate. It holds each step's pull, head, commit and tree, the base, the tip and the tree being gated, the check lane's run id (gate-<tip12>), when it started, and the verdict once there is one. Rows that don't parse are ignored, and rows older than a day are pruned.
• queue_red: the red marks, keyed by (pull, head, base). They used to live only in memory.

Queue (src/checks/queue.ts)

• Each gate writes its row when it starts and deletes it once its verdict has been acted on or the gate is abandoned.
• stopQueue() runs first on shutdown (main.ts). The old queue stops acting, but its gates keep running; a verdict that arrives afterwards is written to the row.
• A new queue reads the rows. If the recorded pulls are back at the same heads and main is still at the recorded base, it rebuilds that exact candidate from the recorded commits. It then either acts on the recorded verdict, waits on the lane's run that is still going, or re-runs the gate on the same tip. A row whose base has moved is dropped and the line is planned as before.

Checks lane (src/checks/service.ts, src/checks/landing.ts)

• The service tracks gates in flight by commit, and adopt(commit) hands back the same running gate instead of starting a second one.
• Shutdown change: the checks drain (FORGE_CHECKS_DRAIN_S, 300 s) now waits for queue gates as well as pull checks. Before, gates were cut off immediately. A gate that finishes during the drain counts as a real verdict; only being aborted counts as "cut off".
• adoptGate covers the old process's gate during a release handover. While that gate's check_gate row is still open, the new process re-reads the queue_gate row every 5 s until a verdict is written there. If the row disappears, the old queue acted on it itself, so the new process does nothing and doesn't re-run the gate.

Tests

• test/merge-queue-restart.test.ts: the queue is killed mid-gate and restarted. It checks three outcomes:

- The lane's run is taken over and the same candidate lands, with one gate run. - A verdict that came in after the stop lands the candidate with no gate run at all. - When the run is gone, the very same tip commit is gated again.

It also covers a moved base, red marks, a bad row and the handover polling.

• test/gate-drain.test.ts: the drain waits for a gate, and its red is judged.

No existing test was changed. CREW.md and ARCHITECTURE.md describe the new behaviour.

Not done

• When checks run as cluster Jobs, a Job is still deleted when its forge stops. Production runs checks as plain processes, which die with the pod. So on a full pod replacement the old run's minutes are only saved if the gate finishes within the drain. Otherwise the candidate is gated again, same candidate, no re-planning. Keeping Jobs alive across a pod roll, and letting a new forge attach to them, needs changes in cluster/jobs.ts and a separate issue.
• Resuming only applies to a candidate that was built directly on main. One stacked on another in-flight candidate is planned again.

🤖 Generated with Claude Code

Closes #6611

The merge queue now remembers the gate in flight across a restart: the new process resumes the same candidate instead of planning the line again. With a gate held open across a simulated restart, the candidate lands with one gate run, not two. Type check is clean; 332 tests across the merge-queue, gate and checks suites, the schema tests and the five cross files pass.

What's kept (new migration 20261026215105-queue-gate.ts, read and written through the new src/checks/queue_store.ts)

• queue_gate: one row per gate. It holds each step's pull, head, commit and tree, the base, the tip and the tree being gated, the check lane's run id (gate-<tip12>), when it started, and the verdict once there is one. Rows that don't parse are ignored, and rows older than a day are pruned.
• queue_red: the red marks, keyed by (pull, head, base). They used to live only in memory.

Queue (src/checks/queue.ts)

• Each gate writes its row when it starts and deletes it once its verdict has been acted on or the gate is abandoned.
• stopQueue() runs first on shutdown (main.ts). The old queue stops acting, but its gates keep running; a verdict that arrives afterwards is written to the row.
• A new queue reads the rows. If the recorded pulls are back at the same heads and main is still at the recorded base, it rebuilds that exact candidate from the recorded commits. It then either acts on the recorded verdict, waits on the lane's run that is still going, or re-runs the gate on the same tip. A row whose base has moved is dropped and the line is planned as before.

Checks lane (src/checks/service.ts, src/checks/landing.ts)

• The service tracks gates in flight by commit, and adopt(commit) hands back the same running gate instead of starting a second one.
• Shutdown change: the checks drain (FORGE_CHECKS_DRAIN_S, 300 s) now waits for queue gates as well as pull checks. Before, gates were cut off immediately. A gate that finishes during the drain counts as a real verdict; only being aborted counts as "cut off".
• adoptGate covers the old process's gate during a release handover. While that gate's check_gate row is still open, the new process re-reads the queue_gate row every 5 s until a verdict is written there. If the row disappears, the old queue acted on it itself, so the new process does nothing and doesn't re-run the gate.

Tests

• test/merge-queue-restart.test.ts: the queue is killed mid-gate and restarted. It checks three outcomes:

- The lane's run is taken over and the same candidate lands, with one gate run. - A verdict that came in after the stop lands the candidate with no gate run at all. - When the run is gone, the very same tip commit is gated again.

It also covers a moved base, red marks, a bad row and the handover polling.

• test/gate-drain.test.ts: the drain waits for a gate, and its red is judged.

No existing test was changed. CREW.md and ARCHITECTURE.md describe the new behaviour.

Not done

• When checks run as cluster Jobs, a Job is still deleted when its forge stops. Production runs checks as plain processes, which die with the pod. So on a full pod replacement the old run's minutes are only saved if the gate finishes within the drain. Otherwise the candidate is gated again, same candidate, no re-planning. Keeping Jobs alive across a pod roll, and letting a new forge attach to them, needs changes in cluster/jobs.ts and a separate issue.
• Resuming only applies to a candidate that was built directly on main. One stacked on another in-flight candidate is planned again.

🤖 Generated with Claude Code

The pull description is the same as before, with one note added at the end about the triage message. The triage note asks for the issue to be filed under a pillar, and only the owner can do that. #6611 waits behind the Later gate (#595), and its parent #6001 is closed, so there was no open parent issue to link it under. The branch already holds the finished work, and no code changed in response, so there was nothing new to push.

The merge queue now remembers the gate in flight across a restart: the new process resumes the same candidate instead of planning the line again. With a gate held open across a simulated restart, the candidate lands with one gate run, not two. Type check is clean; 332 tests across the merge-queue, gate and checks suites, the schema tests and the five cross files pass.

What's kept (new migration 20261026215105-queue-gate.ts, read and written through the new src/checks/queue_store.ts)

• queue_gate: one row per gate. It holds each step's pull, head, commit and tree, the base, the tip and the tree being gated, the check lane's run id (gate-<tip12>), when it started, and the verdict once there is one. Rows that don't parse are ignored, and rows older than a day are pruned.
• queue_red: the red marks, keyed by (pull, head, base). They used to live only in memory.

Queue (src/checks/queue.ts)

• Each gate writes its row when it starts and deletes it once its verdict has been acted on or the gate is abandoned.
• stopQueue() runs first on shutdown (main.ts). The old queue stops acting, but its gates keep running; a verdict that arrives afterwards is written to the row.
• A new queue reads the rows. If the recorded pulls are back at the same heads and main is still at the recorded base, it rebuilds that exact candidate from the recorded commits. It then either acts on the recorded verdict, waits on the lane's run that is still going, or re-runs the gate on the same tip. A row whose base has moved is dropped and the line is planned as before.

Checks lane (src/checks/service.ts, src/checks/landing.ts)

• The service tracks gates in flight by commit, and adopt(commit) hands back the same running gate instead of starting a second one.
• Shutdown change: the checks drain (FORGE_CHECKS_DRAIN_S, 300 s) now waits for queue gates as well as pull checks. Before, gates were cut off immediately. A gate that finishes during the drain counts as a real verdict; only being aborted counts as "cut off".
• adoptGate covers the old process's gate during a release handover. While that gate's check_gate row is still open, the new process re-reads the queue_gate row every 5 s until a verdict is written there. If the row disappears, the old queue acted on it itself, so the new process does nothing and doesn't re-run the gate.

Tests

• test/merge-queue-restart.test.ts: the queue is killed mid-gate and restarted. It checks three outcomes:

- The lane's run is taken over and the same candidate lands, with one gate run. - A verdict that came in after the stop lands the candidate with no gate run at all. - When the run is gone, the very same tip commit is gated again.

It also covers a moved base, red marks, a bad row and the handover polling.

• test/gate-drain.test.ts: the drain waits for a gate, and its red is judged.

No existing test was changed. CREW.md and ARCHITECTURE.md describe the new behaviour.

Not done

• When checks run as cluster Jobs, a Job is still deleted when its forge stops. Production runs checks as plain processes, which die with the pod. So on a full pod replacement the old run's minutes are only saved if the gate finishes within the drain. Otherwise the candidate is gated again, same candidate, no re-planning. Keeping Jobs alive across a pod roll, and letting a new forge attach to them, needs changes in cluster/jobs.ts and a separate issue.
• Resuming only applies to a candidate that was built directly on main. One stacked on another in-flight candidate is planned again.

Triage: the automatic note on the issue asks for a pillar. #6611 waits behind the Later gate (#595), and its parent #6001 is closed, so there is no open parent issue to file it under. Naming a pillar or opening the gate is the owner's call.

🤖 Generated with Claude Code

The merge queue no longer loses a gate in flight when the forge rolls; it resumes the same candidate instead of planning the line again. A pull that is held, closed or pushed to while waiting now leaves the line on the next tick. With a gate held open across a simulated restart, the candidate lands with one gate run, not two. Three pulls wait, the middle one is held, and the next batch carries two. Type check is clean. 250 tests across the merge-queue and autonomy suites and the five cross files passed three times in normal order. The two new queue test files also passed six times in random order; the older autonomy and clash suites fail under random order.

What's kept (new migration 20261026215105-queue-gate.ts, read and written through the new src/checks/queue_store.ts)

• queue_gate: one row per gate. It holds each step's pull, head, commit and tree, the base, the tip and the tree being gated, the check lane's run id (gate-<tip12>), when it started, and the verdict once there is one. Rows that don't parse are ignored, and rows older than a day are pruned.
• queue_red: the red marks, keyed by (pull, head, base). They used to live only in memory.

Queue (src/checks/queue.ts)

• Each gate writes its row when it starts and deletes it once its verdict has been acted on or the gate is abandoned.
• stopQueue() runs first on shutdown (main.ts). The old queue stops acting, but its gates keep running; a verdict that arrives afterwards is written to the row.
• A new queue reads the rows. If the recorded pulls are back at the same heads and main is still at the recorded base, it rebuilds that exact candidate from the recorded commits. It then either acts on the recorded verdict, waits on the lane's run that is still going, or re-runs the gate on the same tip. A row whose base has moved is dropped and the line is planned as before.

Checks lane (src/checks/service.ts, src/checks/landing.ts)

• The service tracks gates in flight by commit, and adopt(commit) hands back the same running gate instead of starting a second one.
• Shutdown change: the checks drain (FORGE_CHECKS_DRAIN_S, 300 s) now waits for queue gates as well as pull checks. Before, gates were cut off immediately. A gate that finishes during the drain counts as a real verdict; only being aborted counts as "cut off".
• adoptGate covers the old process's gate during a release handover. While that gate's check_gate row is still open, the new process re-reads the queue_gate row every 5 s until a verdict is written there. If the row disappears, the old queue acted on it itself, so the new process does nothing and doesn't re-run the gate.

Taking pulls out of the line (steward's comment, 22:50Z)

• Why the old check missed it: a sweep in flight waits for its whole line, so the next sweep, and its admission check, doesn't run until then. A hold set on a pull already waiting was never seen.
• pruneLine (src/crew/autonomy.ts) now runs on every tick before the sweep. Using only database rows, it takes out every waiting pull that is:

- held, - closed or merged, - pushed to since it joined, - no longer green on its head, - or has changes requested on that head.

• dropFromLine (src/checks/queue.ts) answers each removed pull false, logs [queue] taken out of the line, and the tick line's waiting count drops with it. A pull already inside a candidate is left to its gate.
• "Lost approval" is read as changes requested on the head. Zero approvals can't be used: a person's own pull lands without any.

Tests

• test/merge-queue-restart.test.ts: the queue is killed mid-gate and restarted. It checks three outcomes:

- The lane's run is taken over and the same candidate lands, with one gate run. - A verdict that came in after the stop lands the candidate with no gate run at all. - When the run is gone, the very same tip commit is gated again.

It also covers a moved base, red marks, a bad row and the handover polling.

• test/gate-drain.test.ts: the drain waits for a gate, and its red is judged.
• test/merge-queue-hold.test.ts:

- Three pulls wait behind a gate and the middle one is held: the line says 2 waiting and the next candidate is exactly the other two. - A pull pushed to and a closed pull leave the line, while a pull that still qualifies stays and lands.

No existing test was changed. CREW.md and ARCHITECTURE.md describe the new behaviour.

Not done

• When checks run as cluster Jobs, a Job is still deleted when its forge stops. Production runs checks as plain processes, which die with the pod. So on a full pod replacement the old run's minutes are only saved if the gate finishes within the drain. Otherwise the candidate is gated again, same candidate, no re-planning. Keeping Jobs alive across a pod roll, and letting a new forge attach to them, needs changes in cluster/jobs.ts and a separate issue.
• Resuming only applies to a candidate that was built directly on main. One stacked on another in-flight candidate is planned again.

Triage: the automatic note on the issue asks for a pillar. #6611 waits behind the Later gate (#595), and its parent #6001 is closed, so there is no open parent issue to file it under. Naming a pillar or opening the gate is the owner's call.

🤖 Generated with Claude Code

I merged main into this branch and resolved the conflicts in src/checks/queue.ts and src/checks/service.ts, keeping both sides. The type check is clean, the 92 gate, landing and merge-queue tests pass, and the 8 tests in test/merge-queue-restart.test.ts pass, including one new test (described below).

• queue.ts: main's counted (#6615) runs a gate once more when it went red for an install, setup or timeout reason. It now wraps this branch's resumable gate run, renamed attempt; counted takes adopt and passes the resumed record to the first try only, so a retry always runs fresh. A verdict recorded before a roll is still acted on with no gate run.
• service.ts: I kept the gateOnce/adopt split from this branch and added main's stage to GateDone and to the gate's result. One difference from main is deliberate: being stopped alone does not mark a gate interrupted, because a gate the drain lets finish is judged (#6611).
• queue_store.ts: this fixes a bug the merge would have caused. verdictOf dropped the new reason field when it read a stored red back, so an install-failure red resumed after a roll would have been bisected as if the code were at fault. It now reads and validates reason, and a record whose reason does not parse counts as no record. The new test in test/merge-queue-restart.test.ts checks that a stored install red keeps its reason after it is read back.

🤖 Generated with Claude Code