test/rehearsal-real.test.ts is admitted: ten green flake-hunt runs on the check host, or its red names what did not come up and that is fixed #11178

When a forge in the roll rehearsal doesn't come up, the error now says which process, on which port, how it ended, how long it was waited for, and the last lines it wrote. A forge that died only because another process took its port is started again on a fresh one. test/rehearsal-real.test.ts was already in the gate with no skip: #11114 emptied EXCLUDED, and nothing here changes that.

The one gap: I couldn't run the admission on the check host. Both ten-run admissions ran in the agent sandbox, which uses loopback ports (FORGE_TEST_SOCKET=port) instead of the unix sockets the gate uses. So the ten-on-the-check-host box isn't proven. If the gate goes red again, the message will say why.

What the red said before. Only the old forge did not come up; see <log>.err. That file is deleted with the check's tmpdir, so #7632 never learned why the forge died at boot (~110 ms after start, on the second roll).

What it says now. goneLine (new, in scripts/rehearsal.ts) builds the message. For example: the old forge (pid 4242, port 39001) exited with code 1 112 ms after it started; it wrote (<log>): … followed by the last 20 lines of the forge's stderr, or its stdout log when stderr is empty. It also covers a forge ended by a signal, one still not ready when the 60 s wait ran out, and how many ports it lost. Both the old forge's boot wait and the new forge's "exited" check in scripts/rehearse-roll.ts use it.

The fix, for the likely cause. The rehearsal had its own freePort: it found a free ephemeral port, released it, and let the forge bind it once booted. Under the gate, in that window another shard's listener, or one of the first roll's thousands of outgoing connections, can take the port. The forge then dies at once, which fits the second-roll failure. The rehearsal now uses src/kernel/port.ts (freePort, lostPort, PORT_RETRIES), the same retry that apps and release boot already use:

• A forge whose output says its port was taken is started on a fresh port, at most twice.
• After that, the error is raised and says how many ports it lost.
• A new forge that already joined rotation is never restarted; if it dies, that is the roll failing.

No timeout was raised and nothing is skipped. This cause is a hypothesis that fits the evidence, not one I confirmed on the check host.

Tests. Five new tests in test/rehearsal.test.ts, 14 of 14 passing:

• Three check the message wording, the signal and still-not-ready cases, and that only the last lines are quoted.
• Two run the real rehearse against a fake checkout whose src/main.ts dies at boot. One checks the message names the exit code and what the forge wrote. The other checks that a forge that keeps losing its port is retried exactly PORT_RETRIES times and then named.

Admission: bun scripts/flake-hunt.ts admit test/rehearsal-real.test.ts on the pushed commit, in the agent sandbox:

admission: test/rehearsal-real.test.ts — 10/10 green, random order, under load
  run 1: seed 1704976033 green
  run 2: seed 1018444972 green
  run 3: seed 1654011427 green
  run 4: seed 3999222738 green
  run 5: seed 805058752 green
  run 6: seed 1654055206 green
  run 7: seed 34683828 green
  run 8: seed 2182226109 green
  run 9: seed 690300765 green
  run 10: seed 3742971245 green

An earlier admission, which picked up this change partway through, was also 10/10.

bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts exited 2 (diff too broad to name), so I ran test/rehearsal.test.ts and the admission runs of test/rehearsal-real.test.ts myself. The release workflow and the chart are untouched. The issue asks for a keep_plan tool that wasn't available in this run, so no plan was posted on the issue.

🤖 Generated with Claude Code

Closes #11169

When a forge in the roll rehearsal doesn't come up, the error now says which process, on which port, how it ended, how long it was waited for, and the last lines it wrote. A forge that died only because another process took its port is started again on a fresh one. test/rehearsal-real.test.ts was already in the gate with no skip: #11114 emptied EXCLUDED, and nothing here changes that.

The one gap: I couldn't run the admission on the check host. Both ten-run admissions ran in the agent sandbox, which uses loopback ports (FORGE_TEST_SOCKET=port) instead of the unix sockets the gate uses. So the ten-on-the-check-host box isn't proven. If the gate goes red again, the message will say why.

What the red said before. Only the old forge did not come up; see <log>.err. That file is deleted with the check's tmpdir, so #7632 never learned why the forge died at boot (~110 ms after start, on the second roll).

What it says now. goneLine (new, in scripts/rehearsal.ts) builds the message. For example: the old forge (pid 4242, port 39001) exited with code 1 112 ms after it started; it wrote (<log>): … followed by the last 20 lines of the forge's stderr, or its stdout log when stderr is empty. It also covers a forge ended by a signal, one still not ready when the 60 s wait ran out, and how many ports it lost. Both the old forge's boot wait and the new forge's "exited" check in scripts/rehearse-roll.ts use it.

The fix, for the likely cause. The rehearsal had its own freePort: it found a free ephemeral port, released it, and let the forge bind it once booted. Under the gate, in that window another shard's listener, or one of the first roll's thousands of outgoing connections, can take the port. The forge then dies at once, which fits the second-roll failure. The rehearsal now uses src/kernel/port.ts (freePort, lostPort, PORT_RETRIES), the same retry that apps and release boot already use:

• A forge whose output says its port was taken is started on a fresh port, at most twice.
• After that, the error is raised and says how many ports it lost.
• A new forge that already joined rotation is never restarted; if it dies, that is the roll failing.

No timeout was raised and nothing is skipped. This cause is a hypothesis that fits the evidence, not one I confirmed on the check host.

Tests. Five new tests in test/rehearsal.test.ts, 14 of 14 passing:

• Three check the message wording, the signal and still-not-ready cases, and that only the last lines are quoted.
• Two run the real rehearse against a fake checkout whose src/main.ts dies at boot. One checks the message names the exit code and what the forge wrote. The other checks that a forge that keeps losing its port is retried exactly PORT_RETRIES times and then named.

Admission: bun scripts/flake-hunt.ts admit test/rehearsal-real.test.ts on the pushed commit, in the agent sandbox:

admission: test/rehearsal-real.test.ts — 10/10 green, random order, under load
  run 1: seed 1704976033 green
  run 2: seed 1018444972 green
  run 3: seed 1654011427 green
  run 4: seed 3999222738 green
  run 5: seed 805058752 green
  run 6: seed 1654055206 green
  run 7: seed 34683828 green
  run 8: seed 2182226109 green
  run 9: seed 690300765 green
  run 10: seed 3742971245 green

An earlier admission, which picked up this change partway through, was also 10/10.

bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts exited 2 (diff too broad to name), so I ran test/rehearsal.test.ts and the admission runs of test/rehearsal-real.test.ts myself. The release workflow and the chart are untouched. The issue asks for a keep_plan tool that wasn't available in this run, so no plan was posted on the issue.

🤖 Generated with Claude Code

Left #11169 open: 2 unticked items remain (bun scripts/flake-hunt.ts admit test/rehearsal-real.test.ts is ten green on the check host, and the pull quotes its summary.; Or, when a run is red, the failure message names the process or port that did not come up and how long it waited. That cause is fixed in the rehearsal or the test's setup, not by raising a timeout or skipping the test.).