Public: anyone can read this; the crew can change it.
Turn the release keeper on in production and prove it with one canary release #7947
Landed in pull #7974 · 3 runs · run
commented
Answers from the code on main, for the switch-on. Nothing has been switched on: production config can't be reached from a run.
- Reconcile vs. keeper. On main the keeper re-sends its image only while a release is
rolling. Once a release is verified, nothing stops the ~15-minute standby reconcile from setting the older chart tag back. The fix belongs in gitops: the reconcile must stop owning the forge container'simage, so it never rolls the pod. A keeper-side repair loop (tried in #7948) only hides this, and both reviewers asked for it to come out of this issue. - Where the image comes from. The keeper pushes
v<version>, and the Forgejo release job on git.open-platform.sh builds<repo>:<version>. While that build is held, the new pod waits on its image pull, the old pod keeps serving (maxSurge 1, maxUnavailable 0), and the 3 h deadline (ROLL_DEADLINE_MS) rolls back by name and holds. Removing git.open-platform.sh from this path would take an in-cluster build Job started on the tag. That is a separate change. - What
FORGE_REPOchanges. The own-tags reader (release/own_tags.ts),/releases, the behind watch and the landings read already run on it.releaseOnLand(checks/landing.ts) checks out and preflights each landed tree of that repository only, and it swaps the process in only withFORGE_SELF_SWAP=1, which must stay unset in the cluster. - The service-account grant. No manifest in this repository grants it:
charts/appis the chart for apps. It has to be added in gitops.
The gitops change, in apps/forge/prod/manifests.yaml:
FORGE_RELEASE_KEEPER=onon the forge container;- a
RolewithapiGroups: [apps],resources: [deployments],resourceNames: [forge],verbs: [get, patch], plus aRoleBindingto the pod's service account; - the reconcile leaves the forge container's
imagealone.
For the conductor, in one line: set FORGE_RELEASE_KEEPER=on, grant the forge SA get/patch on deployments/forge, and make standby-reconcile stop rendering the forge container's image. After that, the canary can be watched on /releases.
commented
Parked by the crew: @torque's run ended without a pull: the pull closed
Sign in to join this conversation.
new above
offline
The keeper's code is on main (#7794, #7911) but production boots with "the release keeper is off: FORGE_RELEASE_KEEPER is not on", so releases still need the Mac's hourly tick or a Claude Code session. This is not done until a release has gone live with neither. Earlier
#7557closed itself because the code existed; this issue is about production.Answer each one here, from the code and the live setup, before anything is switched on:
trevato-apps/gitopsapps/forge/prod/manifests.yaml. An image set directly on the Deployment can be set back by it after the keeper says verified. How does the keeper keep the version it verified?releaseOnLandand the own-tags reader?Done when: