Public: anyone can read this; the crew can change it.
Turn the release keeper on in production and prove it with one canary release #7974
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/7947-turn-the-release-keeper-on-in-production → main 1 commit fork
1 file
changed +46
−4
docs/features/release-keeper.md +46 −4
| @@ -96,7 +96,49 @@ rows and issues. It has no account that may land a pull or push main. | ||
| 96 | 96 | Rollback is the one action it takes without a canary, because the image it |
| 97 | 97 | sets was already live. |
| 98 | 98 | |
| 99 | Assumption: the keeper promotes by setting the image directly, as the | |
| 100 | steward's rollback did. If Flux reconciles the image back, the keeper sets | |
| 101 | it again on its next tick while the release is rolling. If it never takes, | |
| 102 | the 3 h deadline rolls it back and holds. | |
| 99 | ## Switching it on in production (#7947) | |
| 100 | ||
| 101 | Nothing here is switched on yet: production config is the conductor's, and a | |
| 102 | run cannot reach it. The answers, from the code: | |
| 103 | ||
| 104 | 1. **Reconcile vs. keeper.** The keeper re-sends its image only while a | |
| 105 | release is `rolling`. Once it is verified, the ~15-minute standby | |
| 106 | reconcile (pg-grand) would set the older chart tag back. The fix is in | |
| 107 | gitops, not in a repair loop here: the reconcile must stop owning the | |
| 108 | forge container's `image`, so it never rolls the pod. The chart tag is | |
| 109 | then only the first image of a fresh Deployment. | |
| 110 | 2. **The image.** The keeper pushes `v<version>`; the Forgejo release job on | |
| 111 | git.open-platform.sh builds `<repo>:<version>`. While that build is held | |
| 112 | the new pod waits on its pull, the old pod serves | |
| 113 | (`maxSurge 1, maxUnavailable 0`), and the 3 h deadline (`ROLL_DEADLINE_MS`) | |
| 114 | rolls back by name and holds. Taking git.open-platform.sh off this path | |
| 115 | needs an in-cluster build Job started on the tag: a separate change. | |
| 116 | 3. **`FORGE_REPO`.** `release/own_tags.ts`, `/releases`, the behind watch and | |
| 117 | the landings read already run on it. `releaseOnLand` | |
| 118 | (`checks/landing.ts`) checks out and preflights each landed tree of that | |
| 119 | repository only, and swaps the process in only with `FORGE_SELF_SWAP=1`, | |
| 120 | which must stay unset in the cluster. | |
| 121 | 4. **The grant.** No manifest in this repository grants it (`charts/app` is | |
| 122 | the apps' chart). It is added in gitops. | |
| 123 | ||
| 124 | The gitops change, in `trevato-apps/gitops` `apps/forge/prod/manifests.yaml`: | |
| 125 | ||
| 126 | - `FORGE_RELEASE_KEEPER=on` (and `FORGE_REPO` set) on the forge container; | |
| 127 | - a `Role` with `apiGroups: [apps]`, `resources: [deployments]`, | |
| 128 | `resourceNames: [forge]`, `verbs: [get, patch]`, and a `RoleBinding` to the | |
| 129 | pod's service account; | |
| 130 | - the standby reconcile leaves the forge container's `image` alone. | |
| 131 | ||
| 132 | For the conductor, in one line: set `FORGE_RELEASE_KEEPER=on`, grant the forge | |
| 133 | service account get/patch on `deployments/forge`, and stop standby-reconcile | |
| 134 | rendering the forge container's image. | |
| 135 | ||
| 136 | Then the canary: watch one release cut, roll, verify and record on | |
| 137 | `/releases`, 0 s dark. Only after that retire the steward's `tick release` | |
| 138 | and its `launchd` job. Until then the Mac's tick stays. | |
| 139 | ||
| 140 | Assumption: the keeper promotes by setting the image directly. Without the | |
| 141 | gitops change above, a reconcile can set it back after the keeper says | |
| 142 | verified; with it, nothing does. While a release is rolling the keeper sets | |
| 143 | the image again each tick, and if it never takes, the 3 h deadline rolls it | |
| 144 | back and holds. |
new above
offline