Sentinel: release was degraded 12 minutes today: Release 0.7.547 is unproven: the public URL never answered as 0.7.547. It is proven again at 19:28 UTC. #11729

When a release takes over the port, the old forge now closes the connections that were already open to it, so the next request reaches the new version. Before this, the public URL could keep answering as the old version for the whole drain. That is how release 0.7.547 went unproven: it was a stale connection, not a broken release.

Cause. A release swap has both processes bind one port with SO_REUSEPORT. The old one then calls stopAccepting, which closes its listener with web.stop(false). That only stops new connections. A connection opened before the swap, like the ones a router keeps open and reuses, is still answered by the old process until it exits. The old process exits only when its drain ends, after the crew's runs in flight have landed. During that time /healthz on the public URL keeps answering with the old x-forge-version, release/prove.ts gives up after ten minutes with "the public URL never answered as 0.7.547", and the release card stays up until the old process exits. I confirmed this with plain Bun.serve: after stop(false), a client that had connected earlier kept hearing the old process every time.

Change (src/http/server.ts). stopAccepting sets a handedOver flag. From then on, every answer the old process still gives carries connection: close, so Bun ends the connection after writing it. The client redials, and the old process's listener is already closed, so the redial lands on the release. Requests in flight, including a slow git push, are still answered to the end. I did not shorten the connections' idle timeout, because that could cut a slow answer mid-stream. WebSocket upgrades (no response object) are left alone.

Test. test/release-handover-keepalive.test.ts opens a connection to an old forge and reuses it. It then brings up a release on the same port and calls stopAccepting. It expects one more answer from the old forge with connection: close, then every later request from the release. It fails without the change and passes with it.

Not changed. The proof's timings in release/prove.ts: the earlier fixes (#10148–#10323) only changed how long it waits and retries, and this fixes why the URL never answered. The SIGTERM drain path, where the listener stays open on purpose for the linger.

Checks run. bun scripts/gate.ts and bun x tsc --noEmit are clean. bun scripts/fast.ts named 1,932 files, because server.ts reaches most of the suite. Instead I ran the handover, swap, self-swap, keeper-swap, swapped-launch, body-limits, limits and release-prove suites: 91 pass, 1 skip, 0 fail. The keep_plan tool was not available in this session, so there is no plan on the issue.

To try it: run bun test test/release-handover-keepalive.test.ts. After the next release, /board should show no "never answered as" release card.

🤖 Generated with Claude Code

Closes #11682

When a release takes over the port, the old forge now closes the connections that were already open to it, so the next request reaches the new version. Before this, the public URL could keep answering as the old version for the whole drain. That is how release 0.7.547 went unproven: it was a stale connection, not a broken release.

Cause. A release swap has both processes bind one port with SO_REUSEPORT. The old one then calls stopAccepting, which closes its listener with web.stop(false). That only stops new connections. A connection opened before the swap, like the ones a router keeps open and reuses, is still answered by the old process until it exits. The old process exits only when its drain ends, after the crew's runs in flight have landed. During that time /healthz on the public URL keeps answering with the old x-forge-version, release/prove.ts gives up after ten minutes with "the public URL never answered as 0.7.547", and the release card stays up until the old process exits. I confirmed this with plain Bun.serve: after stop(false), a client that had connected earlier kept hearing the old process every time.

Change (src/http/server.ts). stopAccepting sets a handedOver flag. From then on, every answer the old process still gives carries connection: close, so Bun ends the connection after writing it. The client redials, and the old process's listener is already closed, so the redial lands on the release. Requests in flight, including a slow git push, are still answered to the end. I did not shorten the connections' idle timeout, because that could cut a slow answer mid-stream. WebSocket upgrades (no response object) are left alone.

Test. test/release-handover-keepalive.test.ts opens a connection to an old forge and reuses it. It then brings up a release on the same port and calls stopAccepting. It expects one more answer from the old forge with connection: close, then every later request from the release. It fails without the change and passes with it.

Not changed. The proof's timings in release/prove.ts: the earlier fixes (#10148–#10323) only changed how long it waits and retries, and this fixes why the URL never answered. The SIGTERM drain path, where the listener stays open on purpose for the linger.

Checks run. bun scripts/gate.ts and bun x tsc --noEmit are clean. bun scripts/fast.ts named 1,932 files, because server.ts reaches most of the suite. Instead I ran the handover, swap, self-swap, keeper-swap, swapped-launch, body-limits, limits and release-prove suites: 91 pass, 1 skip, 0 fail. The keep_plan tool was not available in this session, so there is no plan on the issue.

To try it: run bun test test/release-handover-keepalive.test.ts. After the next release, /board should show no "never answered as" release card.

🤖 Generated with Claude Code