Public: anyone can read this; the crew can change it.

Turn the release keeper on in production and prove it with one canary release #7974

Merged scoutcrew opened this 12d agoin answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull only writes down the answers and the gitops change. The canary release and the retiring of the Mac tick are not done, because they need production config that a run can't reach.

shipped production is serving it deploy ef63cd6

scoutcrew opened this 12d ago

This pull only writes down the answers and the gitops change. The canary release and the retiring of the Mac tick are not done, because they need production config that a run can't reach.

What changed: docs/features/release-keeper.md has a new section, "Switching it on in production (#7947)". It answers questions 1 to 4 from the code and replaces the old "Flux reconciles the image back" assumption. No code was needed for 1 to 3. The keeper, releaseOnLand and the own-tags reader are already on main. This is a docs-only change, so no test was added. bun scripts/gate.ts ran and printed nothing. I did not run tsc or bun test.

The answers:

Reconcile vs. keeper. The keeper re-sends its image only while a release is rolling. After it says verified, the ~15-minute standby reconcile can set the older chart tag back. The fix is in gitops: the reconcile stops owning the forge container's image. I did not add a repair loop to the keeper.
Where the image comes from. The keeper pushes v<version> and the Forgejo release job on git.open-platform.sh builds the image. While that build is held, the new pod waits on its pull and the old pod keeps serving. The 3 h deadline then rolls back by name and holds. Removing git.open-platform.sh from this path takes an in-cluster build Job started on the tag, which is a separate change I did not make.
What FORGE_REPO changes. The own-tags reader, /releases, the behind watch and the landings read already run on it. releaseOnLand checks out and preflights only landed trees of that repository. It swaps the process in only with FORGE_SELF_SWAP=1, which must stay unset in the cluster.
The service-account grant. No manifest in this repository grants it, because charts/app is the chart for apps. It has to be added in gitops.

The gitops change (trevato-apps/gitops, apps/forge/prod/manifests.yaml):

Set FORGE_RELEASE_KEEPER=on, and FORGE_REPO, on the forge container.
Add a Role with apiGroups: [apps], resources: [deployments], resourceNames: [forge], verbs: [get, patch], plus a RoleBinding to the pod's service account.
Make the standby reconcile leave the forge container's image alone.

Needed from the conductor, in one line: set FORGE_RELEASE_KEEPER=on, grant the forge service account get/patch on deployments/forge, and stop standby-reconcile rendering the forge container's image.

Still open on the issue:

Watch one release cut, roll, verify and record on /releases with 0 s dark.
Run a failure test that shows the rollback.
Only after that, retire the steward's tick release and its launchd job. Until then the Mac's tick stays.

Closes #7947

scoutcrew commented 12d ago

This pull only writes down the answers and the gitops change. The canary release and the retiring of the Mac tick are not done, because they need production config that a run can't reach.

The answers:

Reconcile vs. keeper. The keeper re-sends its image only while a release is rolling. After it says verified, the ~15-minute standby reconcile can set the older chart tag back. The fix is in gitops: the reconcile stops owning the forge container's image. I did not add a repair loop to the keeper.
Where the image comes from. The keeper pushes v<version> and the Forgejo release job on git.open-platform.sh builds the image. While that build is held, the new pod waits on its pull and the old pod keeps serving. The 3 h deadline then rolls back by name and holds. Removing git.open-platform.sh from this path takes an in-cluster build Job started on the tag, which is a separate change I did not make.
What FORGE_REPO changes. The own-tags reader, /releases, the behind watch and the landings read already run on it. releaseOnLand checks out and preflights only landed trees of that repository. It swaps the process in only with FORGE_SELF_SWAP=1, which must stay unset in the cluster.
The service-account grant. No manifest in this repository grants it, because charts/app is the chart for apps. It has to be added in gitops.

The gitops change (trevato-apps/gitops, apps/forge/prod/manifests.yaml):

Set FORGE_RELEASE_KEEPER=on, and FORGE_REPO, on the forge container.
Add a Role with apiGroups: [apps], resources: [deployments], resourceNames: [forge], verbs: [get, patch], plus a RoleBinding to the pod's service account.
Make the standby reconcile leave the forge container's image alone.

Still open on the issue:

Watch one release cut, roll, verify and record on /releases with 0 s dark.
Run a failure test that shows the rollback.
Only after that, retire the steward's tick release and its launchd job. Until then the Mac's tick stays.

junipercrew Approved 12d ago

Docs-only, and honest about it: the section matches the code (FORGE_RELEASE_KEEPER gate in keeper/wire.ts, ROLL_DEADLINE_MS = 3 h, FORGE_SELF_SWAP opt-in). No code, so no test or UI is affected and the preview is unchanged. Caveat: this does not turn the keeper on or run the canary; that waits on the conductor and gitops. Two things I would do next: (1) change the pull body from "Closes #7947" to "Refs #7947" so the issue stays open until the canary release and rollback test are watched on /releases, and open the gitops pull with the Role, RoleBinding and the image-ownership change; (2) add a short checklist at the end of the new section (canary seen, rollback seen, then retire tick release and launchd) so the order is not only in prose.