Found during the AWS audit (PR #212, finding L3). This is a live issue on the git.benjivers.com EC2 k3s leader.
Symptom
fleet/insightsprod HelmRelease is UpgradeFailed:
failed early due to stalled resources:
[Deployment/fleet--insights--prod/insights status: 'Failed']
with a recurring pod Warning:
FailedToRetrieveImagePullSecret: Unable to retrieve some image pull secrets (ecr-cred);
attempting to pull the image may not succeed.
The pod happens to still be Running (0 restarts) on the already-pulled image, but the release can't complete and any reschedule would fail to pull. fleet/{www,relay,review} and invoicing/fleet-invoice are healthy — so it's this release/secret, not cluster-wide.
Root cause
Every app on the leader pulls from ECR (<acct>.dkr.ecr.us-east-1.amazonaws.com/...) using an imagePullSecrets: [ecr-cred].
ECR auth tokens expire ~12h. There is no live refresher keeping ecr-cred current, so the secret goes stale / missing and the pull-secret lookup fails.
Why this is NOT a plat/mitosis source change
The string ecr-creddoes not exist anywhere in the mitosis source (grep -rn ecr-cred bin/ deploy/ → nothing).
Mitosis' AWS design is node-IAM-ECR-role pulls, no pull secret (bin/germinate:71-72, :813 — "kubelet pulls via the node's ECR role"). Default k3s (what this leader runs) is meant to pull from the in-cluster Forgejo registry via the 127.0.0.1:31100 mirror (registry-node-config), not ECR.
So the leader was wired to ECR + ecr-credout-of-band; that mechanism lives in the leader's own gitops (plat/gitops on git.benjivers.com), not here.
Fix (lands in the daughter gitops, tracked here)
On git.benjivers.complat/gitops, add (or repair) an ECR-token refresher CronJob — periodically aws ecr get-login-password → recreate the ecr-cred docker-registry secret in each app namespace, on a schedule well under the ~12h expiry (e.g. every 6–8h). (Pattern already used ad hoc in deploy/eks/mirror-images-eks.sh:84-88, which notes the 12h expiry.)
Then flux reconcile / roll fleet--insights--prod to clear the wedged UpgradeFailed.
Should mitosis support an AWS/ECR-backed k3s leader as a first-class profile (shipping the ecr-cred refresher in the germinate gitops), or should the leader be moved back to the default in-cluster-registry path so ECR + ecr-cred isn't needed at all? Right now it's an unsupported, hand-wired config that silently rots every 12h.
Found during the AWS audit (PR #212, finding L3). This is a live issue on the
git.benjivers.comEC2 k3s leader.Symptom
fleet/insightsprod HelmRelease isUpgradeFailed:with a recurring pod Warning:
The pod happens to still be
Running(0 restarts) on the already-pulled image, but the release can't complete and any reschedule would fail to pull.fleet/{www,relay,review}andinvoicing/fleet-invoiceare healthy — so it's this release/secret, not cluster-wide.Root cause
<acct>.dkr.ecr.us-east-1.amazonaws.com/...) using animagePullSecrets: [ecr-cred].ecr-credcurrent, so the secret goes stale / missing and the pull-secret lookup fails.Why this is NOT a plat/mitosis source change
ecr-creddoes not exist anywhere in the mitosis source (grep -rn ecr-cred bin/ deploy/→ nothing).bin/germinate:71-72,:813— "kubelet pulls via the node's ECR role"). Default k3s (what this leader runs) is meant to pull from the in-cluster Forgejo registry via the127.0.0.1:31100mirror (registry-node-config), not ECR.ecr-credout-of-band; that mechanism lives in the leader's own gitops (plat/gitopsongit.benjivers.com), not here.Fix (lands in the daughter gitops, tracked here)
git.benjivers.complat/gitops, add (or repair) an ECR-token refresher CronJob — periodicallyaws ecr get-login-password→ recreate theecr-creddocker-registry secret in each app namespace, on a schedule well under the ~12h expiry (e.g. every 6–8h). (Pattern already used ad hoc indeploy/eks/mirror-images-eks.sh:84-88, which notes the 12h expiry.)flux reconcile/ rollfleet--insights--prodto clear the wedgedUpgradeFailed.Follow-up worth deciding (real plat/mitosis question)
Should mitosis support an AWS/ECR-backed k3s leader as a first-class profile (shipping the ecr-cred refresher in the germinate gitops), or should the leader be moved back to the default in-cluster-registry path so ECR +
ecr-credisn't needed at all? Right now it's an unsupported, hand-wired config that silently rots every 12h.Ref: audit PR #212 →
docs/ops/aws-audit-2026-08-25.md(L3, H9).