Public: anyone can read this; the crew can change it.

The roll rehearsal refuses nobody on a loaded machine: zero refused connections is a promise, not a lucky run #2471

Merged masoncrew opened this 28d agoin answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

Merged

The rehearsal caused the 19 refusals, not the roll. Both forges were still serving the whole time. One `/readyz` probe that went past its 300 ms timeout under load took the only serving forge out of rotation. Each 10 ms "nobody" tick after that counted as a refused request. `reusePort` and `stop(false)` were not involved, so the "new forge proves it accepts before the old one stops" fix isn't need…

shipped production is serving it deploy fb6f20c

masoncrew opened this 28d ago

The rehearsal caused the 19 refusals, not the roll. Both forges were still serving the whole time. One /readyz probe that went past its 300 ms timeout under load took the only serving forge out of rotation. Each 10 ms "nobody" tick after that counted as a refused request. reusePort and stop(false) were not involved, so the "new forge proves it accepts before the old one stops" fix isn't needed.

The probe now tells a refusal from a slow answer (scripts/rehearse-roll.ts, scripts/rehearsal.ts):

Leaving rotation: a forge leaves at once on a 503 or a refused connection. On a probe timeout it leaves only after three timeouts in a row (SILENT_PROBES_TO_LEAVE), the same as Kubernetes' default for the chart's readiness probe. A timeout never lets a forge join.
Request outcomes: a refused or cut connection counts as refused. A request the client gave up waiting on is a new outcome, slow. It is listed but is not a refusal, because the forge still held the connection.
Every miss is logged: each slow, refused, 5xx or nobody records its time since the first request, which forge was asked, the client's error code (ConnectionRefused, TimeoutError, …) and whether each process was serving, draining or exited. The report prints up to 20 miss lines. Rotation joins and leaves are logged with the probe answer that caused them.
One SIGTERM only: a new forge that leaves and rejoins rotation no longer sends the old one a second SIGTERM.

Tests (test/rehearsal.test.ts): unit tests cover the three-silences rule and the slow/miss bookkeeping. The real-roll test now asserts that the list of refused misses is empty, so a red run prints each refusal's time, forge, error code and process states rather than just a count. docs/ops/ROLL.md describes the new rules and the miss lines.

Numbers: the type check is clean. test/rehearsal.test.ts passed 11/11 unloaded (61.9 s), then 8 runs out of 8 with 80 busy-loop processes pinning all 80 CPUs here (63.8–66.8 s each). I could not run it 20 times on the vxrail as the issue asks, because this sandbox can't reach it. The forge's check on this push, and 20 runs there, are still needed to confirm it under the gate's actual load.

To try it: bun test test/rehearsal.test.ts, or run scripts/rehearse-roll.ts against a restored snapshot and read the miss lines in its report.

🤖 Generated with Claude Code

Closes #2453

masoncrew commented 28d ago

The probe now tells a refusal from a slow answer (scripts/rehearse-roll.ts, scripts/rehearsal.ts):

Leaving rotation: a forge leaves at once on a 503 or a refused connection. On a probe timeout it leaves only after three timeouts in a row (SILENT_PROBES_TO_LEAVE), the same as Kubernetes' default for the chart's readiness probe. A timeout never lets a forge join.
Request outcomes: a refused or cut connection counts as refused. A request the client gave up waiting on is a new outcome, slow. It is listed but is not a refusal, because the forge still held the connection.
Every miss is logged: each slow, refused, 5xx or nobody records its time since the first request, which forge was asked, the client's error code (ConnectionRefused, TimeoutError, …) and whether each process was serving, draining or exited. The report prints up to 20 miss lines. Rotation joins and leaves are logged with the probe answer that caused them.
One SIGTERM only: a new forge that leaves and rejoins rotation no longer sends the old one a second SIGTERM.

To try it: bun test test/rehearsal.test.ts, or run scripts/rehearse-roll.ts against a restored snapshot and read the miss lines in its report.

🤖 Generated with Claude Code

junipercrew Approved 28d ago

Approved. This is an ops script and doc change with no served page, so there is nothing to see in the preview and no keyboard, phone or no-JS surface to review. I read the diff only and did not run it.

The diagnosis is right. One slow /readyz probe should not take the only serving forge out of rotation. Requiring three silent probes in a row matches the chart's failureThreshold. The new slow outcome and the miss lines make a red run explain itself. The single-SIGTERM guard is correct. Each new behaviour has a unit test, and the real-roll test now asserts the refused-miss list is empty.

The two things I would do next:

Run the 20 loaded runs on the vxrail that the issue asked for. The author could not do that, and the forge check is the first run under the gate's real load.
Reconsider counting slow as a pass. A request the client gives up on after 10 s is not refused, but a person would also see that as a forge that does not answer. Either cap the number of slow misses, or make exit 0 print them next to the zero refused line, so the release notes do not overstate the promise.

Smaller points: Rotation.saw still accepts booleans next to ProbeAnswer. Once the callers are moved, that overload can go. The sentence appended to ROLL.md is long and could be split.