Public: anyone can read this; nobody can change it.

AWS deployment audit — EC2 leader + EKS backlog (34 items) #212

Closed opened this view on origin
Closed without merging

A consolidated audit of everything AWS in the mitosis platform, requested 2026-08-25. Adds one doc — docs/ops/aws-audit-2026-08-25.md — a prioritized, checkbox-able backlog. No code changes; this is the list of issues + concrete changes to work through.

Notes

stopped closed without shipping

opened this view on origin

What this is

A consolidated audit of everything AWS in the mitosis platform, requested 2026-08-25. Adds one doc — docs/ops/aws-audit-2026-08-25.md — a prioritized, checkbox-able backlog. No code changes; this is the list of issues + concrete changes to work through.

Scope swept

  • deploy/aws/ — single-node EC2 leader stack → git.benjivers.com (k3s, Free-Plan account)
  • deploy/eks/ — managed-EKS stack → git.aws.benjivers.com (billed control plane, us-east-1)
  • EKS-mode germinate wiring — bin/germinate + bin/lib.sh on feat/2vcpu-daughter-fixes (incl. uncommitted working-tree changes)
  • Live cluster state — pulled from the running git.aws.benjivers.com cluster

Headline findings

4 critical

  • C1 — the entire deploy/eks/ stack (behind a live, billing cluster) is untracked in git; exists only on one laptop. (Verified safe to commit — deploy/.gitignore already excludes state/tfvars.)
  • C2 — EKS Terraform state is local-only (no S3 backend) and holds plaintext secrets; laptop loss = unrecoverable orphaned billing.
  • C3 + C4 — the EKS germinate TLS path is half-landed: PLATFORM_EDGE_ISSUER support is an unapplied patch, the shipped seed predates it, and certs point at a letsencrypt-dns01 issuer that is never created. An eks germination today comes up with untrusted certs on every browser-facing edge.

Live cluster (running now)

  • L1 — open-platform/www prod is stuck (HelmRelease Progressing, 0 pods) on the AWS cluster; fleet/www is healthy.
  • L2 — PLAT_FORWARD_AUTH unset → every app Ingress trips the plat-access coverage policy.
  • L3 — the live git.benjivers.com EC2 leader couldn't be inspected (op-benjivers MCP needs OAuth) — action for you to authorize so it can be swept.

12 high — open EKS API endpoint (0.0.0.0/0, taken by the live cluster); credit-guard that fails open and only brakes compute (control plane keeps billing); down.sh that can leave a billable NLB + Route53 zone orphaned; EC2 SSH default 0.0.0.0/0; no cost guardrails; unencrypted root EBS with no backups holding the sovereign key; single leader with no prevent_destroy + destructive user-data replacement; plus germinate ECR/kyverno-mirror/external-dns gaps.

12 medium + 6 low — full detail, file:line, and the concrete change for each in the doc, plus a "do not fix — already correct" section and a suggested landing order.

Note

The doc is deliberately a checklist so items can be ticked off (or split into issues) as they land. Nothing here is auto-fixed — flag which items you want turned into PRs and I'll take them in the suggested order (C1/C2 first).

🤖 Generated with Claude Code

2 earlier comments
commented

L3 resolved — live EC2 leader (git.benjivers.com) swept

op-benjivers is now authorized, so the one gap in the original audit is filled. Pushed to the branch (acf7e7b).

9 apps on the leader — fleet/{www,relay,review,insights} + invoicing/fleet-invoice, all pulling from ECR (<acct>.dkr.ecr.us-east-1.amazonaws.com).

AppProd status
fleet/www✅ healthy (v3)
fleet/relay✅ healthy (v2)
fleet/review✅ healthy (v4)
invoicing/fleet-invoice✅ healthy (v23, app + worker)
fleet/insights🔴 HelmRelease UpgradeFailed — stalled Deployment + recurring FailedToRetrieveImagePullSecret (ecr-cred)

New live finding (now L3 in the doc): fleet/insights prod is wedged because the ecr-cred image-pull secret can't be retrieved. The leader pulls everything from ECR and ECR auth tokens expire ~12h, so a static platform/registry-pull-secret.yaml goes stale without a refresher. The pod happens to still be Running (0 restarts) but the release can't complete and a reschedule would fail to pull. Fix: repair/add the ECR-token refresh and roll the release — bumped to step 0 (operational hotfix) in the landing order since it's live right now. Related to germinate H9 (ECR wiring).

PLAT_FORWARD_AUTH-off (L2) is confirmed on both clusters. Everything else in the audit stands.

commented

Filed the two live findings as tracking issues (both root-cause to the daughter-cluster gitops, not this repo's source — the ecr-cred mechanism isn't in mitosis source and the leader pulls from ECR off the default k3s path):

  • L3 → #215 — fleet/insights wedged on stale ecr-cred (git.benjivers.com leader); fix = ECR-token refresher CronJob in that cluster's plat/gitops.
  • L1 → #216 — open-platform/www stuck HelmRelease (git.aws.benjivers.com EKS); fix = pull flux controller logs + reconcile/rollback on that cluster.

Each issue also notes the follow-up plat/mitosis-source question (support AWS/ECR k3s as a first-class profile w/ refresher; HelmRelease install-retry hardening).

commented

Closing per Ben — the current-state audit isn't the artifact wanted. The build-process history (how AWS/AKS/EKS actually got built) supersedes this in #221. The two live app-status items surfaced here remain tracked as #215 (ecr-cred stale) and #216 (open-platform/www stuck) if they're ever needed.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs