Public: anyone can read this; nobody can change it.
AWS deployment audit — EC2 leader + EKS backlog (34 items) #212
A consolidated audit of everything AWS in the mitosis platform, requested 2026-08-25. Adds one doc — docs/ops/aws-audit-2026-08-25.md — a prioritized, checkbox-able backlog. No code changes; this is the list of issues + concrete changes to work through.
stopped closed without shipping
2 earlier comments
L3 resolved — live EC2 leader (git.benjivers.com) swept
op-benjivers is now authorized, so the one gap in the original audit is filled. Pushed to the branch (acf7e7b).
9 apps on the leader — fleet/{www,relay,review,insights} + invoicing/fleet-invoice, all pulling from ECR (<acct>.dkr.ecr.us-east-1.amazonaws.com).
| App | Prod status |
|---|---|
| fleet/www | ✅ healthy (v3) |
| fleet/relay | ✅ healthy (v2) |
| fleet/review | ✅ healthy (v4) |
| invoicing/fleet-invoice | ✅ healthy (v23, app + worker) |
| fleet/insights | 🔴 HelmRelease UpgradeFailed — stalled Deployment + recurring FailedToRetrieveImagePullSecret (ecr-cred) |
New live finding (now L3 in the doc): fleet/insights prod is wedged because the ecr-cred image-pull secret can't be retrieved. The leader pulls everything from ECR and ECR auth tokens expire ~12h, so a static platform/registry-pull-secret.yaml goes stale without a refresher. The pod happens to still be Running (0 restarts) but the release can't complete and a reschedule would fail to pull. Fix: repair/add the ECR-token refresh and roll the release — bumped to step 0 (operational hotfix) in the landing order since it's live right now. Related to germinate H9 (ECR wiring).
PLAT_FORWARD_AUTH-off (L2) is confirmed on both clusters. Everything else in the audit stands.
Filed the two live findings as tracking issues (both root-cause to the daughter-cluster gitops, not this repo's source — the ecr-cred mechanism isn't in mitosis source and the leader pulls from ECR off the default k3s path):
- L3 → #215 —
fleet/insightswedged on staleecr-cred(git.benjivers.com leader); fix = ECR-token refresher CronJob in that cluster'splat/gitops. - L1 → #216 —
open-platform/wwwstuck HelmRelease (git.aws.benjivers.com EKS); fix = pull flux controller logs + reconcile/rollback on that cluster.
Each issue also notes the follow-up plat/mitosis-source question (support AWS/ECR k3s as a first-class profile w/ refresher; HelmRelease install-retry hardening).
Closing per Ben — the current-state audit isn't the artifact wanted. The build-process history (how AWS/AKS/EKS actually got built) supersedes this in #221. The two live app-status items surfaced here remain tracked as #215 (ecr-cred stale) and #216 (open-platform/www stuck) if they're ever needed.
What this is
A consolidated audit of everything AWS in the mitosis platform, requested 2026-08-25. Adds one doc —
docs/ops/aws-audit-2026-08-25.md— a prioritized, checkbox-able backlog. No code changes; this is the list of issues + concrete changes to work through.Scope swept
deploy/aws/— single-node EC2 leader stack →git.benjivers.com(k3s, Free-Plan account)deploy/eks/— managed-EKS stack →git.aws.benjivers.com(billed control plane, us-east-1)bin/germinate+bin/lib.shonfeat/2vcpu-daughter-fixes(incl. uncommitted working-tree changes)git.aws.benjivers.comclusterHeadline findings
4 critical
deploy/eks/stack (behind a live, billing cluster) is untracked in git; exists only on one laptop. (Verified safe to commit —deploy/.gitignorealready excludes state/tfvars.)PLATFORM_EDGE_ISSUERsupport is an unapplied patch, the shipped seed predates it, and certs point at aletsencrypt-dns01issuer that is never created. An eks germination today comes up with untrusted certs on every browser-facing edge.Live cluster (running now)
open-platform/wwwprod is stuck (HelmReleaseProgressing, 0 pods) on the AWS cluster;fleet/wwwis healthy.PLAT_FORWARD_AUTHunset → every app Ingress trips theplat-accesscoverage policy.git.benjivers.comEC2 leader couldn't be inspected (op-benjivers MCP needs OAuth) — action for you to authorize so it can be swept.12 high — open EKS API endpoint (
0.0.0.0/0, taken by the live cluster); credit-guard that fails open and only brakes compute (control plane keeps billing);down.shthat can leave a billable NLB + Route53 zone orphaned; EC2 SSH default0.0.0.0/0; no cost guardrails; unencrypted root EBS with no backups holding the sovereign key; single leader with noprevent_destroy+ destructive user-data replacement; plus germinate ECR/kyverno-mirror/external-dns gaps.12 medium + 6 low — full detail, file:line, and the concrete change for each in the doc, plus a "do not fix — already correct" section and a suggested landing order.
Note
The doc is deliberately a checklist so items can be ticked off (or split into issues) as they land. Nothing here is auto-fixed — flag which items you want turned into PRs and I'll take them in the suggested order (C1/C2 first).
🤖 Generated with Claude Code