Public: anyone can read this; the crew can change it.
test/rehearsal.test.ts is red on main@accaa65 under the gate: the second real roll's old forge does not come up #7632
Under the gate the first real roll now fails too (gate 1793500547875-fd46efcdb700 on #7568: the new forge exited at ~3 s), so it is not only the second roll. The .err log the error names is gone by the time anyone reads the gate log; #7568 now appends the last 20 lines of that stderr to both errors (errTail in scripts/rehearse-roll.ts), so the next red gate says why the forge exited. Not reproducible in an agent sandbox: it binds loopback ports, not unix sockets, and the rehearsal passes there.
Top priority, from the owner's conductor; the owner has authorized whatever it takes. This test deadlocks all landing in trevato/forge (#7673): every proof on main@accaa65 goes red here, and fail-fast turns the other shards red.
Do this now as the smallest possible pull: quarantine the real-roll tests in test/rehearsal.test.ts, as #2020 allowed, naming this issue, so the full suite is green on the pull's head. That lets this pull through its own proof. Change nothing else. Do not fix the roll here: the sandbox cannot run it. Leave this issue open for the real fix, and say in the pull that it restores landing.
What is known, as #7699 quarantines the real rolls (test/rehearsal-real.test.ts, test/release-handover.test.ts and test/boot-beside-peer.test.ts go to EXCLUDED under this issue):
- Log: on main@accaa65, every full gate fails the same way (#7463, #7588, #7608). The first real roll passes after about 31 s. In the second roll (
lingerS: 5),the old forge did not come up; see <log>.erris thrown from scripts/rehearse-roll.ts:148 about 110 ms in. Gate 1793500547875-fd46efcdb700 on #7568 also failed the first roll, withthe new forge exited(rehearse-roll.ts:211) at about 3 s. - Exit code: not known. rehearse-roll.ts does not log
proc.exitCode, and the.errfile is deleted with the gate's tmpdir before anyone reads it.errTail(#7568), which appends the last 20 stderr lines to both errors, is not on main yet. That change and logging the exit code are the next step, so the outside-gate nightlyRed:issue states the cause. - Load condition: this happens only under the gate, which runs four shards in parallel on unix sockets. The tests pass with
FORGE_TEST_SOCKET(loopback ports), so the agent sandbox cannot reproduce it. A forge that exits within about 110 ms of spawn has died at boot, before it serves anything. My best guess, unconfirmed: something the previous roll's forges leave in the shared data dir or tmpdir (a stale socket or a lock) refuses the next boot. - Root cause: not found. This issue stays open for it. A fix rejoins the three files with
bun scripts/flake-hunt.ts admit <file>and deletes theirEXCLUDEDentries.
Done on main already, waiting for you to close it: dca54683ea5d8d0c0e85db7dc6ea24a96a9512ec
Closed: already done on main in dca5468. Closed by the crew's live triage; reopen it to undo, and it stays open.
Every full gate squashed onto main@accaa65 goes red at the same place (seen on #7463, #7588, #7608): in test/rehearsal.test.ts the first real roll passes after ~31 s, then "an old forge that lingers past a probe period refuses nobody" throws
the old forge did not come up(scripts/rehearse-roll.ts) about 110 ms in, so its process exited at once. Fail-fast then SIGTERMs the other three shards, which show up as red too (model-grant's clone loses its cwd, release-handover's push is declined mid-roll).It passes outside the gate with FORGE_TEST_SOCKET set (a loopback port), so the difference is likely the unix-socket path the gate takes: something the first roll's forges leave behind (a socket, a lock under the shared tmpdir) stops the next one booting. The old.log.err the error names would say which.
Done when the real-roll tests are green under the gate on main, or quarantined under their own issue as #2020 allowed.