Forge releases go dark: the deploy has maxSurge 0 with a 720 s drain, and a failed release deadlocks the quota; surge instead (maxUnavailable 0) #6958

Every forge release is dark until the new pod is ready, and a failed release can deadlock the quota. Found at 11:37–11:42Z on 2026-10-27:

1. The deploy is maxSurge: 0, maxUnavailable: 1. The old pod gets SIGTERM before the new one is ready. With a 720 s grace, the old pod drains, removed from the service, while the new pod boots, so any slow boot is an outage. The quota (98 CPU, pods at 48) has room for a surge pod. It should be maxSurge: 1, maxUnavailable: 0 in the generated manifests.
2. After a rollback, two terminating pods held 96 of the 98 CPU, and the replacement could not be created (exceeded quota) for 12 minutes.
3. The vxrail kubelet did not act on a shortened grace: both pods sat at deletionGracePeriodSeconds=1 for 3+ minutes. Recovery needed crictl stop on the node.

Done when: the generated manifest surges (and the reconciler keeps it that way); a test on the generated manifest asserts maxUnavailable: 0; and the release path's dark time, measured by health probes every 2 s, is 0 for a normal release.

Found the source: it isn't the forge's chart (which says maxSurge: 1, maxUnavailable: 0, and test/roll.test.ts guards it).

It's a platform-side hold from 2026-10-07, a kustomize post-render patch on the forge's release ("Hold fix 2026-10-07: standby-reconcile refused 2Gi surge on pg-grand (vxrail NotReady). maxSurge 0 replaces the pod in place."). The vxrail has been Ready for weeks, and the quota holds two 48-CPU pods (96 ≤ 98), so the hold has outlived its reason. It's what made every release dark since then. With #6732's long drain, it's also what made 0.7.278 run two writers.

Live state now:

• GitOps renders the forge at 0.7.278. The live deploy is 0.7.276, set by hand in recovery.
• standby-reconcile leaves it alone ("live template differs from the last apply … left as is") because the forge's staged entry hasn't changed.
• The trap: any change that re-stages the forge, including removing this patch, would re-apply 0.7.278.

The fix (steward, with the owner's go, since it touches the live forge deploy):

1. In one GitOps change: pin the tag back to 0.7.276 and drop the maxSurge-0 patch, so the deploy changes strategy only and no new pod starts. Watch health every 2 s while it applies.
2. Release again only after #6957, with the surge in place: the new pod must be ready before the old one stops.

Parked by the crew: the run ended without a pull: declined: nothing in this repository needs to change. The chart is already maxSurge: 1, maxUnavailable: 0 in charts/app/templates/deployment.yaml:22-23. test/roll.test.ts:259 asserts that, and `test/roll.

Done on main already, waiting for you to close it: 5a341e543ce4f66dcc1cd34dace9870b5cefa357 629076b4742c80dbdf959c8563b5ec3cedb03597 27dfd6d17724404d6389f703b22dcd59622b0815

Closed: already done on main in 5a341e5, 629076b, 27dfd6d. Closed by the crew's live triage; reopen it to undo, and it stays open.

Reopened: #6961 fixes the forge side (#6957), so a new pod holds its work while the old one drains. But the platform still rolls the forge with maxSurge: 0, so each release stays dark for the new pod's boot (about 14 s), and docs/ops/ROLL.md forbids a dark roll without the owner's yes for that version. This one closes when the live deploy reads maxSurge: 1, maxUnavailable: 0 and a release measures 0 s dark. It's the steward's job, waiting on the owner's go.

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Done (steward, with the owner's go): the 2026-10-07 maxSurge: 0 patch is removed from trevato-apps/gitops apps/forge/prod/manifests.yaml, and the live deploy now reads maxSurge: 1, maxUnavailable: 0. 0.7.279 rolled at 16:07Z with 0 non-200 probes in 246 s (2 s probes), and the promote measured dark ≈ 0 s. verify-release passed: 111 probes under page load over 10 min, worst 5.6 s.

Closed: already done on main in 5a341e5, 629076b, 27dfd6d.