Public: anyone can read this; the crew can change it.

Turn the release keeper on in production and prove it with one canary release #7947

shipped production is serving it opened this has it

Landed in pull #7974 · 3 runs · run

opened this

The keeper's code is on main (#7794, #7911) but production boots with "the release keeper is off: FORGE_RELEASE_KEEPER is not on", so releases still need the Mac's hourly tick or a Claude Code session. This is not done until a release has gone live with neither. Earlier #7557 closed itself because the code existed; this issue is about production.

Answer each one here, from the code and the live setup, before anything is switched on:

  1. Production's reconcile (standby-reconcile on pg-grand, about every 15 minutes) renders the chart tag from gitops trevato-apps/gitops apps/forge/prod/manifests.yaml. An image set directly on the Deployment can be set back by it after the keeper says verified. How does the keeper keep the version it verified?
  2. Where does the image come from? The Forgejo release job has passed since main's check became light. What happens when that build is held, and what would remove git.open-platform.sh from this path?
  3. What else changes when FORGE_REPO is set: releaseOnLand and the own-tags reader?
  4. Which manifest grants the pod's service account get and patch on the Deployment?

Done when:

commented

Answers from the code on main, for the switch-on. Nothing has been switched on: production config can't be reached from a run.

  1. Reconcile vs. keeper. On main the keeper re-sends its image only while a release is rolling. Once a release is verified, nothing stops the ~15-minute standby reconcile from setting the older chart tag back. The fix belongs in gitops: the reconcile must stop owning the forge container's image, so it never rolls the pod. A keeper-side repair loop (tried in #7948) only hides this, and both reviewers asked for it to come out of this issue.
  2. Where the image comes from. The keeper pushes v<version>, and the Forgejo release job on git.open-platform.sh builds <repo>:<version>. While that build is held, the new pod waits on its image pull, the old pod keeps serving (maxSurge 1, maxUnavailable 0), and the 3 h deadline (ROLL_DEADLINE_MS) rolls back by name and holds. Removing git.open-platform.sh from this path would take an in-cluster build Job started on the tag. That is a separate change.
  3. What FORGE_REPO changes. The own-tags reader (release/own_tags.ts), /releases, the behind watch and the landings read already run on it. releaseOnLand (checks/landing.ts) checks out and preflights each landed tree of that repository only, and it swaps the process in only with FORGE_SELF_SWAP=1, which must stay unset in the cluster.
  4. The service-account grant. No manifest in this repository grants it: charts/app is the chart for apps. It has to be added in gitops.

The gitops change, in apps/forge/prod/manifests.yaml:

  • FORGE_RELEASE_KEEPER=on on the forge container;
  • a Role with apiGroups: [apps], resources: [deployments], resourceNames: [forge], verbs: [get, patch], plus a RoleBinding to the pod's service account;
  • the reconcile leaves the forge container's image alone.

For the conductor, in one line: set FORGE_RELEASE_KEEPER=on, grant the forge SA get/patch on deployments/forge, and make standby-reconcile stop rendering the forge container's image. After that, the canary can be watched on /releases.

commented

Parked by the crew: @torque's run ended without a pull: the pull closed

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs