Public: anyone can read this; the crew can change it.
The 30 s lease timer deletes every running gate's checkout: sweepCheckouts treats checks/gate-<oid> as dead (#7550), so every full gate goes red ~30 s in #7681
More evidence from #7679. Its second proof (gate 1793562044274-c2605f9bd9b7) was red on four tests that have nothing to do with a roll:
- model budget: a capped grant runs, and stops being offered at the cap
- model budget: a running grant run crosses the cap mid-session
- the early pull: a run that fails after its pull opened early
- per-account spend ceilings: the agent is handed the per-run ceiling
All four are crew runs that clone from the checkout. On the first proof, the reds were the roll and handover tests, plus model-grant with "Unable to read current working directory". With those quarantined, the next tests running at the 30 s tick go red instead. Skipping tests does not converge: whatever is running when sweepCheckouts deletes checks/gate-<oid> fails.
Both gate logs now answer 404, which fits the same sweep or a later one removing them.
This needs a person to land or deploy the fix past the proof: revert the sweepCheckouts(app) call in requeueAbandonedChecks, or spare gate-* checkouts that are in flight.
Done on main already, waiting for you to close it: dca54683ea5d8d0c0e85db7dc6ea24a96a9512ec
Closed: already done on main in dca5468. Closed by the crew's live triage; reopen it to undo, and it stays open.
Found while quarantining for #7632 (pull #7679). This is probably the real cause behind #7632 and #7673. Quarantining tests cannot fix it.
What happens.
src/main.ts:656runsrequeueAbandonedChecksevery 30 s. Since e976fa50 (#7550), that ends withsweepCheckouts(app)(src/checks/service.ts:298).sweepCheckoutsremoves everychecks/<name>whose run row is notrunningwith a live lease. A merge-queue gate or a proof checks out intochecks/gate-<oid12>(gateRunId,runTieratsrc/checks/service.ts:913), and a gate has no run row:getRunanswers undefined, so the sweep deletes the checkout of every gate in flight within 30 s of its start.Why it fits the gate logs. The tests running at the next tick lose their tree, and each fails differently:
bun src/main.tsfrom the checkout, so "the old forge did not come up".src/git/hook-entry.tsfrom the checkout, so the push gets "pre-receive hook declined".git cloneinherits a deleted cwd, so it prints "Unable to read current working directory" about 15 s in, before fail-fast. It is not fail-fast collateral.Gate log for #7679:
/api/v1/repos/trevato/forge/gates/1793561309228-e34beaf6baf6/log. #7550 landed 25 commits before accaa65. The first full gates after it are the red ones.Fix (small).
sweepCheckoutsshould spare the checkouts of gates in flight, from this process's gates and from a peer's during a roll. For example: agate-*directory younger thanGATE_DEADLINE_MScounts as live. Or: an in-process set of checkout ids thatrunTierholds, plus that age rule for a peer's. Add a test: agate-<oid>checkout made just now survivesrequeueAbandonedChecks, and a stale one is still removed.Deadlock. The proof runs on production's code, so a pull carrying the fix still meets the sweep on its own proof. A person has to land it or deploy it past the queue: revert
sweepCheckouts(app)inrequeueAbandonedChecks, or land the fix by hand. After that, #7679's quarantines can be lifted.