Public: anyone can read this; nobody can change it.

Live: fleet/insights prod wedged on stale `ecr-cred` pull secret (git.benjivers.com leader) #215

Closed opened this view on origin

Closed without the crew

opened this view on origin

Found during the AWS audit (PR #212, finding L3). This is a live issue on the git.benjivers.com EC2 k3s leader.

Symptom

fleet/insights prod HelmRelease is UpgradeFailed:

failed early due to stalled resources:
[Deployment/fleet--insights--prod/insights status: 'Failed']

with a recurring pod Warning:

FailedToRetrieveImagePullSecret: Unable to retrieve some image pull secrets (ecr-cred);
attempting to pull the image may not succeed.

The pod happens to still be Running (0 restarts) on the already-pulled image, but the release can't complete and any reschedule would fail to pull. fleet/{www,relay,review} and invoicing/fleet-invoice are healthy — so it's this release/secret, not cluster-wide.

Root cause

  • Every app on the leader pulls from ECR (<acct>.dkr.ecr.us-east-1.amazonaws.com/...) using an imagePullSecrets: [ecr-cred].
  • ECR auth tokens expire ~12h. There is no live refresher keeping ecr-cred current, so the secret goes stale / missing and the pull-secret lookup fails.

Why this is NOT a plat/mitosis source change

  • The string ecr-cred does not exist anywhere in the mitosis source (grep -rn ecr-cred bin/ deploy/ → nothing).
  • Mitosis' AWS design is node-IAM-ECR-role pulls, no pull secret (bin/germinate:71-72, :813 — "kubelet pulls via the node's ECR role"). Default k3s (what this leader runs) is meant to pull from the in-cluster Forgejo registry via the 127.0.0.1:31100 mirror (registry-node-config), not ECR.
  • So the leader was wired to ECR + ecr-cred out-of-band; that mechanism lives in the leader's own gitops (plat/gitops on git.benjivers.com), not here.

Fix (lands in the daughter gitops, tracked here)

  1. On git.benjivers.com plat/gitops, add (or repair) an ECR-token refresher CronJob — periodically aws ecr get-login-password → recreate the ecr-cred docker-registry secret in each app namespace, on a schedule well under the ~12h expiry (e.g. every 6–8h). (Pattern already used ad hoc in deploy/eks/mirror-images-eks.sh:84-88, which notes the 12h expiry.)
  2. Then flux reconcile / roll fleet--insights--prod to clear the wedged UpgradeFailed.

Follow-up worth deciding (real plat/mitosis question)

Should mitosis support an AWS/ECR-backed k3s leader as a first-class profile (shipping the ecr-cred refresher in the germinate gitops), or should the leader be moved back to the default in-cluster-registry path so ECR + ecr-cred isn't needed at all? Right now it's an unsupported, hand-wired config that silently rots every 12h.

Ref: audit PR #212 → docs/ops/aws-audit-2026-08-25.md (L3, H9).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs