Public: anyone can read this; nobody can change it.
AWS deployment audit — EC2 leader + EKS backlog (34 items) #212
audit/aws-2026-08-25 → main 2 commits
1 file
changed +340
−0
docs/ops/aws-audit-2026-08-25.md added +340
| @@ -0,0 +1,340 @@ | ||
| 1 | # AWS deployment audit — `git.benjivers.com` (EC2 leader) + `git.aws.benjivers.com` (EKS) | |
| 2 | ||
| 3 | _Audit date: 2026-08-25. Scope: everything AWS in the mitosis platform — the single-node EC2 | |
| 4 | leader stack (`deploy/aws/`, serving `git.benjivers.com`) and the managed-EKS stack | |
| 5 | (`deploy/eks/`, serving `git.aws.benjivers.com`), plus the EKS-mode germinate wiring on | |
| 6 | branch `feat/2vcpu-daughter-fixes` and the live cluster state._ | |
| 7 | ||
| 8 | This document is a backlog, not a fix. Each item: severity, location, why it matters, and the | |
| 9 | concrete change. Check items off as they land. Sensitive specifics (account ID, leader IP) are | |
| 10 | redacted — they live only in gitignored Terraform state, never in this repo. | |
| 11 | ||
| 12 | ## How to read the sources | |
| 13 | ||
| 14 | - **`deploy/aws/`** — EC2 leader (k3s single node, Free-Plan account). Tracked in git. | |
| 15 | - **`deploy/eks/`** — managed EKS (billed control plane, us-east-1). **Currently untracked.** | |
| 16 | - **`bin/germinate` + `bin/lib.sh`** — germinate flow; the `mode=eks` / `CLUSTER_PROFILE=eks` | |
| 17 | path is partly committed on `feat/2vcpu-daughter-fixes` and partly still in the working tree | |
| 18 | (uncommitted: `bin/germinate` +56, `bin/lib.sh` +397). | |
| 19 | ||
| 20 | --- | |
| 21 | ||
| 22 | ## P0 — Critical (blocks safe/correct AWS operation) | |
| 23 | ||
| 24 | - [ ] **C1. `deploy/eks/` is untracked — live EKS infra exists only on one laptop.** | |
| 25 | `deploy/eks/*` (all `??` in `git status`), referenced by tracked code at `bin/lib.sh:733,755,1050` | |
| 26 | and `bin/germinate:70`. The Terraform + `up.sh`/`down.sh`/`mirror-images-eks.sh`/`credit-guard.tf` | |
| 27 | behind a live, billing EKS cluster is uncommitted; a fresh clone of the branch gets germinate/lib | |
| 28 | EKS code that calls files that aren't there, and laptop loss = an orphaned running cluster + NLB | |
| 29 | bleeding credits with no git record. | |
| 30 | **Change:** Commit `deploy/eks/` as a unit alongside the `bin/` changes. Committing is safe — | |
| 31 | `deploy/.gitignore` already excludes `*.tfstate`/`*.tfvars`/`.terraform/`, and a dry-run | |
| 32 | `git add deploy/eks` pulls only source (`.tf`, `.sh`, `README.md`, `UPGRADE.md`, `lambda/`, | |
| 33 | `patches/`, `.terraform.lock.hcl`, `terraform.tfvars.example`). Do **not** force-add state/tfvars. | |
| 34 | ||
| 35 | - [ ] **C2. EKS Terraform state is local-only (no remote backend).** | |
| 36 | `deploy/eks/versions.tf:12-15` (S3 backend commented out); a 329 KB `terraform.tfstate` sits on | |
| 37 | the laptop as the only record of the live cluster and carries plaintext secrets (a | |
| 38 | `random_password.result`, a `tls_certificate`, KMS metadata, account ID ×149). It is correctly | |
| 39 | gitignored, but loss = unrecoverable orphaned billing. | |
| 40 | **Change:** Create a versioned + SSE S3 bucket (TF ≥1.10 `use_lockfile=true`, no DynamoDB needed), | |
| 41 | uncomment the `backend "s3"` block, `tofu init -migrate-state`. Same treatment recommended for the | |
| 42 | EC2 stack (see H8). | |
| 43 | ||
| 44 | - [ ] **C3. EKS germinate TLS path is half-landed — browser edges will render untrusted certs.** | |
| 45 | `deploy/eks/patches/mcp-platform-edge-issuer.patch` (untracked, unapplied) vs `genesis/seed.tar.gz` | |
| 46 | (ships mcp 0.39.2). `trim_gitops_eks` injects `platformEdgeIssuer: letsencrypt-dns01` | |
| 47 | (`bin/lib.sh:915-925`) and `render_front_door` swaps the issuer via `PLATFORM_EDGE_ISSUER` | |
| 48 | (`bin/lib.sh:1970-2010`), but the renderer support for that value lives only in the patch. The seed | |
| 49 | predates it, so on the direct NLB (no Cloudflare TLS termination) every app edge + the front door | |
| 50 | fall back to `plat-local-ca-issuer` → browser TLS errors on sign-in and all app edges. This is the | |
| 51 | exact failure the EKS TLS story exists to prevent. | |
| 52 | **Change:** Land the patch in the mcp source the seed is built from, cut a new seed whose | |
| 53 | `integration.mcp` carries `platformEdgeIssuer` support, and gate germinate on `CLUSTER_PROFILE=eks` | |
| 54 | to assert the seed's mcp tag supports it (fail loud, not silent no-op). | |
| 55 | ||
| 56 | - [ ] **C4. Issuer name mismatch — certs reference a `letsencrypt-dns01` issuer that is never created.** | |
| 57 | `bin/lib.sh:876-879,915-925` and `bin/germinate:85` point platform-edge + app Certificates at a | |
| 58 | ClusterIssuer named `letsencrypt-dns01`, but the cloudflare→route53 swap (`bin/lib.sh:806-834`) | |
| 59 | only rewrites the *solver* inside the seed's existing issuer (named `letsencrypt`) and never renames | |
| 60 | it; `trim_gitops_eks` only otherwise creates `letsencrypt-http01`. Nothing produces | |
| 61 | `letsencrypt-dns01` → every edge Certificate references a nonexistent issuer → certs never issue. | |
| 62 | **Change:** Either rename the seed's DNS-01 issuer to `letsencrypt-dns01` in step 4d and verify, or | |
| 63 | render a fresh route53 `clusterissuer-dns01.yaml` named `letsencrypt-dns01` (as 4f-3 does for | |
| 64 | http01) and add it to the letsencrypt kustomization. Then hard-assert the issuer exists post-trim. | |
| 65 | ||
| 66 | --- | |
| 67 | ||
| 68 | ## Live cluster state (swept 2026-08-25) | |
| 69 | ||
| 70 | **EKS — `git.aws.benjivers.com`:** | |
| 71 | ||
| 72 | - [ ] **L1. `open-platform/www` prod is stuck.** | |
| 73 | HelmRelease `Progressing` ("reconciliation in progress"), `deployment.found=false`, **zero pods** | |
| 74 | (chart `[email protected]+77386226b940`). `fleet/www` prod on the same cluster is healthy, so this is | |
| 75 | app/release-specific, not cluster-wide. Investigate the HelmRelease + source; reconcile or roll back. | |
| 76 | ||
| 77 | **EC2 leader — `git.benjivers.com`** (op-benjivers authorized; swept 2026-08-25). 9 apps — | |
| 78 | `fleet/{www,relay,review,insights}` + `invoicing/fleet-invoice`; all pull from ECR | |
| 79 | (`<acct>.dkr.ecr.us-east-1.amazonaws.com`). Healthy: `fleet/www` (v3), `fleet/relay` (v2), | |
| 80 | `fleet/review` (v4), `invoicing/fleet-invoice` (v23, app+worker). | |
| 81 | ||
| 82 | - [ ] **L3. `fleet/insights` prod HelmRelease is wedged `UpgradeFailed` on a stale ECR pull secret.** | |
| 83 | "failed early due to stalled resources: [Deployment/fleet--insights--prod/insights status: | |
| 84 | 'Failed']", with a recurring `FailedToRetrieveImagePullSecret` warning for `ecr-cred`. The pod | |
| 85 | currently runs (0 restarts) but the release can't complete and any reschedule may fail to pull. | |
| 86 | The leader pulls all images from ECR, whose auth tokens expire ~12h — a static | |
| 87 | `platform/registry-pull-secret.yaml` goes stale without a refresher. | |
| 88 | **Change:** Root-cause the `ecr-cred` refresh (find/repair or add the ECR-token refresh CronJob), | |
| 89 | then `flux reconcile` / roll `fleet--insights--prod`. Also relevant to germinate H9 (ECR wiring). | |
| 90 | ||
| 91 | **Both clusters:** | |
| 92 | ||
| 93 | - [ ] **L2. `PLAT_FORWARD_AUTH` is unset.** | |
| 94 | Every app Ingress on **both** the EC2 leader and the EKS cluster trips kyverno | |
| 95 | `require-forwardauth-on-app-ingress/app-ingress-has-plat-access-middleware` as a coverage warn. | |
| 96 | Benign while default-off, but if forwardAuth is ever enabled without re-rendering overlays, those | |
| 97 | Ingresses serve with **no** auth. Decide: enable + add the `plat-strip,plat-access` chain, or leave | |
| 98 | documented as intentionally off. | |
| 99 | ||
| 100 | --- | |
| 101 | ||
| 102 | ## P1 — High | |
| 103 | ||
| 104 | - [ ] **H1. EKS public API endpoint defaults to `0.0.0.0/0` and the live cluster took the default.** | |
| 105 | `deploy/eks/variables.tf:79` default `["0.0.0.0/0"]`, consumed `main.tf:52-56`; live `tfvars` sets | |
| 106 | only `domain`. Combined with `enable_cluster_creator_admin_permissions=true` (`main.tf:58`) the whole | |
| 107 | cluster's exposure rests on IAM alone with no network boundary. The variable's own description admits | |
| 108 | this is "the top finding in most security reviews." | |
| 109 | **Change:** Set `api_allowed_cidrs` to your egress CIDR(s) and `tofu apply`; make the var have no | |
| 110 | default (force a choice) or have `up.sh`/germinate refuse a wildcard unless `ALLOW_OPEN_API=1`. | |
| 111 | ||
| 112 | - [ ] **H2. Credit-guard fails OPEN and only brakes compute.** | |
| 113 | `deploy/eks/lambda/credit_guard.py:40-64`, `credit-guard.tf`. Three gaps: (a) it only scales node | |
| 114 | groups down — the EKS control plane (~$2.40–3/day) is delete-only and keeps billing when tripped; | |
| 115 | (b) it reads `freetier:GetAccountPlanState`, which exists only on the *new* Free Plan — on a legacy | |
| 116 | account `remaining` defaults to `9999` (line 42) so it silently never trips; (c) on trip it sets | |
| 117 | `maxSize=1`, not 0, leaving autoscaling headroom. | |
| 118 | **Change:** Verify the account returns real `GetAccountPlanState` data; change the `9999` fallback to | |
| 119 | fail-**closed** (trip when unreadable); set `maxSize:0` on trip; add an AWS Budgets action + SNS that | |
| 120 | can fire a cluster-delete runbook for the control-plane cost. Document it as a compute brake, not a | |
| 121 | full stop. | |
| 122 | ||
| 123 | - [ ] **H3. `down.sh` can leave billable orphans (NLB, Route53 zone).** | |
| 124 | `deploy/eks/down.sh:28-63`. The NLB (LB-controller-owned) and Route53 records (external-dns-owned) | |
| 125 | live outside TF state and are deleted via `kubectl` only if the cluster API answers. If the cluster | |
| 126 | is already broken, teardown skips to `tofu destroy`, which hangs on `DependencyViolation` (NLB ENIs | |
| 127 | pin the subnets) or leaves the NLB + hosted zone billing forever. Lines 69-72 only `echo`, they | |
| 128 | don't assert clean. | |
| 129 | **Change:** Add an AWS-CLI fallback that deletes LBs by VPC tag and empties/deletes the hosted zone | |
| 130 | when the cluster is unreachable, before `tofu destroy`; make the final check assert no LB/zone/EBS | |
| 131 | remains. | |
| 132 | ||
| 133 | - [ ] **H4. EC2: `allowed_ssh_cidr` defaults to `0.0.0.0/0` (fail-open).** | |
| 134 | `deploy/aws/variables.tf:35-39`, used `main.tf:90`. Live tfvars overrides to a /32 (box is fine), | |
| 135 | but a fresh apply / lost tfvars / copied stack lands SSH open to the internet on a personal-account | |
| 136 | box. **Change:** Remove the default (make it required) and add a `validation` block rejecting | |
| 137 | `0.0.0.0/0`. | |
| 138 | ||
| 139 | - [ ] **H5. EC2: no Free-Plan cost guardrails.** | |
| 140 | `deploy/aws/main.tf`/`variables.tf` — no `aws_budgets_budget`, no metric alarm, and `instance_type` | |
| 141 | is a free-form string (default `t3a.xlarge`) with no validation. A typo or `ha=true` (3× xlarge) | |
| 142 | silently triples spend with nothing to catch it. **Change:** Add a Budgets budget + SNS notification | |
| 143 | and a `validation` allowlisting Free-Plan-safe instance types. | |
| 144 | ||
| 145 | - [ ] **H6. EC2: root EBS is unencrypted with no snapshots/backup.** | |
| 146 | `deploy/aws/main.tf:153-156` sets only size/type. The root volume holds Postgres, the registry, and | |
| 147 | the sovereign age key (`deploy/lib/mitosis-node.sh:243` writes `/root/fork-$DOMAIN.age`). No | |
| 148 | `encrypted=true`, and no `aws_dlm_lifecycle_policy`/AWS Backup anywhere → at-rest secrets in | |
| 149 | cleartext and total data loss on termination. **Change:** `encrypted=true` on `root_block_device`; | |
| 150 | add a daily EBS snapshot policy. | |
| 151 | ||
| 152 | - [ ] **H7. EC2: single leader, no `prevent_destroy`, and user-data edits replace the machine.** | |
| 153 | `deploy/aws/main.tf:137-162` (instance), `:151` `user_data_replace_on_change=true`. `ha=false` means | |
| 154 | one instance *is* the platform, and any edit to rendered cloud-init replaces it (cloud-init runs | |
| 155 | once) wiping all germinated state; no `lifecycle { prevent_destroy = true }`. With no snapshots (H6), | |
| 156 | a stray change or `terraform destroy` is unrecoverable. **Change:** Add `prevent_destroy=true` to the | |
| 157 | instance/EBS/EIP, document that user-data edits are destructive, pair with H6 snapshots. | |
| 158 | ||
| 159 | - [ ] **H8. EC2: cluster secrets in local, unencrypted, un-backed-up state.** | |
| 160 | `deploy/aws/main.tf:16-19` (`random_password.k3s_token`) → `terraform.tfstate` (local backend, no | |
| 161 | `backend` block in `versions.tf`). k3s join token + SSH pubkey + password results in cleartext on | |
| 162 | the laptop only. **Change:** Same remote-encrypted-backend migration as C2 for the EC2 stack; confirm | |
| 163 | laptop disk encryption and off-device backup meanwhile. | |
| 164 | ||
| 165 | - [ ] **H9. germinate ECR path swallows errors and leaks tags.** | |
| 166 | `bin/germinate:817-818` do ECR login + `describe-or-create-repository` with `>/dev/null 2>&1 ... || | |
| 167 | true`, so an auth/permission failure only surfaces later as an opaque `docker push` FATAL. The | |
| 168 | on-demand `plat/ci-runner` ECR repo also gets no lifecycle policy, unlike `mirror-images-eks.sh`, so | |
| 169 | runner tags accumulate forever. **Change:** Drop `|| true` on login (or WARN with the aws error); | |
| 170 | apply the same `ECR_LIFECYCLE` put-lifecycle-policy to the `plat/ci-runner` repo germinate creates. | |
| 171 | ||
| 172 | - [ ] **H10. kyverno image-redirect regex vs mirror list drift → `ErrImagePull` risk.** | |
| 173 | Redirect regex `bin/lib.sh:1076,1099` vs mirror sources `deploy/eks/mirror-images-eks.sh:31-65`. | |
| 174 | `registry.k8s.io` is in the policy but not mirrored; the `docker.io/library/*` and `minio/*` | |
| 175 | short-name normalization is fragile and must match exactly how each chart writes its image ref. Any | |
| 176 | unmatched ref stays pointed at the public registry, which nodes (by design) can't reach. | |
| 177 | **Change:** Reconcile the redirect allowlist against the actual mirror list from one shared source; | |
| 178 | add a render/coldstart assertion that every mirrored image maps to an ECR path the policy produces. | |
| 179 | ||
| 180 | - [ ] **H11. traefik NLB external-dns hostname hardcoded to `*.open-platform.sh`.** | |
| 181 | `trim_gitops_eks` traefik block injects | |
| 182 | `external-dns.alpha.kubernetes.io/hostname: "*.open-platform.sh,open-platform.sh"` and relies on | |
| 183 | `rewrite_identity_domain` to swap it afterward. That sweep only rewrites the *old* seed domain | |
| 184 | string; a seed whose domain isn't `open-platform.sh` (a re-germinated `sslip.io`/gate seed) leaves | |
| 185 | the literal unrewritten and external-dns publishes to the wrong zone → NLB unreachable at the | |
| 186 | intended host. `up.sh:126-132` re-annotates but a traefik HelmRelease reconcile drops it. | |
| 187 | **Change:** Write the annotation from the seed's actual domain (read from manifest) or substitute | |
| 188 | `$DOMAIN` at inject time. | |
| 189 | ||
| 190 | - [ ] **H12. EKS cert render is guarded on the seed shape and only WARNs on miss.** | |
| 191 | `bin/lib.sh:882-905` (4f-3) runs the route53 swap + http01 render only if the seed's | |
| 192 | `letsencrypt/clusterissuer.yaml` contains `cloudflare`. A lean/renamed/staging seed silently skips | |
| 193 | the whole thing with a WARN and germinate proceeds to a platform whose certs can never issue — the | |
| 194 | failure surfaces far away at cert-manager. **Change:** After trim, FATAL-assert on eks that both a | |
| 195 | `letsencrypt-dns01` and a `letsencrypt-http01` issuer exist, rather than best-effort WARN. | |
| 196 | ||
| 197 | --- | |
| 198 | ||
| 199 | ## P2 — Medium | |
| 200 | ||
| 201 | - [ ] **M1. EKS: orphaned `random_password.rds_master` cleartext secret in state.** | |
| 202 | `deploy/eks/terraform.tfstate` holds it though no RDS resource remains (`main.tf:221-226` notes RDS | |
| 203 | was removed). Dead but increases state sensitivity. **Change:** `tofu state rm | |
| 204 | random_password.rds_master` (regenerate clean state after C2 backend migration); rotate if it was | |
| 205 | ever wired to a real DB. | |
| 206 | ||
| 207 | - [ ] **M2. EKS: `t3a.2xlarge` (8 vCPU, ~$7/day) dominates Free-Plan burn.** | |
| 208 | `deploy/eks/variables.tf:33` default, `main.tf:72`; `node_max=2` (`variables.tf:51`) lets CI bursts | |
| 209 | double it. **Change:** Consider a smaller type or spot capacity; keep `node_max` low; document daily | |
| 210 | burn per size. | |
| 211 | ||
| 212 | - [ ] **M3. EKS: credit-guard Lambda IAM is `Resource: "*"`.** | |
| 213 | `deploy/eks/credit-guard.tf:46-49` — `eks:UpdateNodegroupConfig`/List/Describe on every cluster in | |
| 214 | the account. **Change:** Scope the `eks:*` actions to the `var.cluster_name` ARN pattern; split | |
| 215 | `freetier:GetAccountPlanState` (no resource scoping) into its own statement. | |
| 216 | ||
| 217 | - [ ] **M4. EKS: `enable_cluster_creator_admin_permissions=true` + open API.** | |
| 218 | `deploy/eks/main.tf:58`. Whoever ran `tofu apply` is permanent internet-reachable cluster-admin | |
| 219 | while H1 stands. **Change:** Acceptable for a solo operator once H1 locks the API CIDR; for multi-op, | |
| 220 | manage admins via explicit access entries. | |
| 221 | ||
| 222 | - [ ] **M5. EKS: live `terraform.tfvars` (`domain = "aws.benjivers.com"`) sits in the worktree.** | |
| 223 | `deploy/eks/terraform.tfvars:1`. Already excluded by `deploy/.gitignore` (verified) — the only action | |
| 224 | is to **not** force-add it when committing C1, and keep `terraform.tfvars.example` as the tracked copy. | |
| 225 | ||
| 226 | - [ ] **M6. 2-vCPU kyverno fix lives only in the lean profile.** | |
| 227 | `bin/lib.sh:297-308` (`trim_gitops_lean`, commit `6ff70a9`) pins kyverno replicas 3→1, fixing the | |
| 228 | small-node stall — but `trim_gitops_eks` (which states it must be self-sufficient) doesn't re-assert | |
| 229 | it, so a `full`-profile eks germination on a small node group reintroduces the 3-replica CPU stall. | |
| 230 | **Change:** Replicate the replicas→1 pin (or a node-sizing guard) inside `trim_gitops_eks`, or | |
| 231 | document + assert that eks requires the lean profile. | |
| 232 | ||
| 233 | - [ ] **M7. NS-delegation prerequisite is manual and unchecked.** | |
| 234 | `deploy/eks/up.sh:102` germinates right after `tofu apply` with no check that `aws.benjivers.com` is | |
| 235 | actually delegated/resolving (the documented gate). If delegation isn't live, germinate stalls at the | |
| 236 | first public-cert issuance and wastes the ~60-90m budget with an opaque failure. **Change:** Between | |
| 237 | terraform and germinate, `dig +short NS $DOMAIN` against a public resolver and refuse/prompt until | |
| 238 | delegation is observed; surface `route53_nameservers` prominently. | |
| 239 | ||
| 240 | - [ ] **M8. EC2: DNS/Cloudflare wiring is entirely manual.** | |
| 241 | `deploy/aws/outputs.tf:6-9` just prints a wildcard-A instruction; no `cloudflare_record` resource. | |
| 242 | `*.benjivers.com → leader EIP` lives only in Cloudflare, untracked, and breaks silently on EIP | |
| 243 | re-allocation. **Change:** Add the `cloudflare/cloudflare` provider and manage the wildcard (and | |
| 244 | per-node HA) records off `aws_eip.node[*].public_ip`. | |
| 245 | ||
| 246 | - [ ] **M9. EC2: Forgejo git-over-SSH (2222) is world-open.** | |
| 247 | `deploy/aws/main.tf:92-112`. 80/443 world-open is expected for a public edge; 2222 broadens the auth | |
| 248 | surface on a personal box. **Change:** Restrict 2222 to operator/CI CIDRs if public git-SSH isn't | |
| 249 | needed, or document as intentional. | |
| 250 | ||
| 251 | - [ ] **M10. EC2: no cost-allocation tags / `default_tags`.** | |
| 252 | `deploy/aws/versions.tf:15-17` provider has no `default_tags`; resources carry only `Name`. Hurts | |
| 253 | spend attribution + cleanup on a cost-sensitive account. **Change:** Add provider `default_tags` | |
| 254 | (`ManagedBy=terraform`, `Project=mitosis`, …). | |
| 255 | ||
| 256 | - [ ] **M11. `resolve_seed`/`publish_seed` use inconsistent Forgejo package API base paths.** | |
| 257 | `bin/lib.sh` — `resolve_seed` lists via `/api/v1/packages/plat?...` for `latest` then hits | |
| 258 | `/api/packages/...`; `publish_seed` PUTs `/api/packages/...`. A mismatch means `SEED=latest` silently | |
| 259 | fails to resolve. Low urgency for eks (its `up.sh` uses the bundled `genesis/seed.tar.gz`), but it's | |
| 260 | the documented fetch path. **Change:** Validate the three endpoints against the running forge; error | |
| 261 | clearly when `latest` returns empty. | |
| 262 | ||
| 263 | - [ ] **M12. EC2: subnet AZ not pinned (implicit/driftable).** | |
| 264 | `deploy/aws/main.tf:60-65` omits `availability_zone`. Low blast radius for a single fixed deploy but | |
| 265 | the AZ can drift between applies. **Change:** Optionally pin `availability_zone` for determinism. | |
| 266 | (AMI pinning via region-correct SSM is already correct — no change there.) | |
| 267 | ||
| 268 | --- | |
| 269 | ||
| 270 | ## P3 — Low | |
| 271 | ||
| 272 | - [ ] **Lo1. `mitosis-node.sh` report() ships admin passwords over the network.** | |
| 273 | `deploy/lib/mitosis-node.sh` `report()` lifts Forgejo + Grafana admin passwords from the germinate | |
| 274 | log into a JSON POST to `REPORT_URL` (opt-in, bearer-gated, best-effort, no TLS assertion). | |
| 275 | **Change:** Reject non-HTTPS `REPORT_URL`; document that this channel carries admin creds so | |
| 276 | `REPORT_TOKEN`/endpoint are treated as secret-grade. Also consider tightening | |
| 277 | `--write-kubeconfig-mode 644` (`mitosis-node.sh:141`) to `600` if the VM is ever multi-user. | |
| 278 | ||
| 279 | - [ ] **Lo2. EKS: mirror list + edge-issuer are floating workarounds.** | |
| 280 | `deploy/eks/mirror-images-eks.sh` must be hand-kept in sync with the AKS mirror list and the | |
| 281 | helmrelease pins (drift → silent `ImagePull` fail), and `patches/mcp-platform-edge-issuer.patch` is a | |
| 282 | floating git patch that must be reapplied on every mcp bump (see C3). **Change:** CI check that the | |
| 283 | EKS/AKS mirror lists match the helmrelease pins; upstream the edge-issuer change into the mcp chart | |
| 284 | as a real flag, or add a patch-applies-cleanly check to `up.sh`. | |
| 285 | ||
| 286 | - [ ] **Lo3. germinate ci-runner cross-arch build assumes buildx/emulation.** | |
| 287 | `bin/germinate:820` `docker build --platform linux/amd64` on an arm64 laptop needs buildx + binfmt; | |
| 288 | the `command -v docker` check (`:87`) doesn't verify it. **Change:** Preflight `docker buildx inspect` | |
| 289 | (or verify emulation) and error clearly if amd64 cross-build isn't available. | |
| 290 | ||
| 291 | - [ ] **Lo4. EC2: provider/Terraform version constraints are loose.** | |
| 292 | `deploy/aws/versions.tf:1-12` — `~> 6.0`/`~> 3.6` and `required_version >= 1.6` with no ceiling. The | |
| 293 | committed lock file mitigates, but `init -upgrade` can jump minors. **Change:** Optionally tighten to | |
| 294 | `~> 6.58` and add a Terraform upper bound `< 2.0`. | |
| 295 | ||
| 296 | - [ ] **Lo5. EKS: `credit_guard.py` region hardcoded `us-east-1`.** | |
| 297 | `deploy/eks/lambda/credit_guard.py:40`. Correct-by-necessity (the `freetier` endpoint is | |
| 298 | us-east-1-only) but reads like a portability bug. **Change:** Add a clarifying comment; leave as-is. | |
| 299 | ||
| 300 | - [ ] **Lo6. Repo housekeeping in the worktree.** | |
| 301 | Stray `.DS_Store`, `germinate-aks-poc*.log`, `mirror-run2.log`, `plat-ca.crt`, and | |
| 302 | `fork-aws.benjivers.com.age.backup` sit untracked. Add `.DS_Store` (and `*.log` scratch) to | |
| 303 | `.gitignore`; confirm `fork-*.age*` stays ignored (it is). | |
| 304 | ||
| 305 | --- | |
| 306 | ||
| 307 | ## Already correct — do not "fix" these | |
| 308 | ||
| 309 | - IMDSv2 enforced (`deploy/aws/main.tf:157-159`, `http_tokens=required`). | |
| 310 | - user-data gzip for the EC2 16 KB cap (`main.tf:146-148`), correctly commented. | |
| 311 | - AMI via region-correct SSM pointer — no hardcoded AMI id. | |
| 312 | - `.terraform.lock.hcl` committed with full `h1:`/`zh:` hashes (EC2); should be committed for EKS too. | |
| 313 | - `deploy/.gitignore` correctly excludes `*.tfstate`/`*.tfvars`/`.terraform/` (keeps `*.example`) — | |
| 314 | so C1 is safe to commit. | |
| 315 | - Sensitive vars marked `sensitive = true`; no account ID or leader IP in any tracked file. | |
| 316 | - EIPs decouple DNS from instance lifecycle. | |
| 317 | - `mirror-images-eks.sh` forces `--override-arch amd64` and applies an ECR lifecycle policy | |
| 318 | (expire untagged >7d, keep last 10). | |
| 319 | - The edge-issuer renderer change itself is clean + test-covered — it just needs to land in mcp source | |
| 320 | and a fresh seed (C3), not be a floating patch. | |
| 321 | ||
| 322 | --- | |
| 323 | ||
| 324 | ## Suggested landing order | |
| 325 | ||
| 326 | 0. **L3 (operational hotfix)** — repair the `ecr-cred` refresh so `fleet/insights` (and any future | |
| 327 | pull) stops failing; roll the wedged release. This is live right now. | |
| 328 | 1. **C1 + C2** — commit `deploy/eks/`, migrate EKS state to S3. (Stops the single-laptop SPOF.) | |
| 329 | 2. **C3 + C4** — fix the issuer name + land `PLATFORM_EDGE_ISSUER` in mcp source, cut a new seed. | |
| 330 | (Without these, an eks germination comes up with dead TLS on every browser edge.) | |
| 331 | 3. **H1 / H2 / H3** — lock the API CIDR, make credit-guard fail-closed, harden teardown. | |
| 332 | 4. **EC2 hardening H4–H8** — SSH default, budget guard, EBS encryption + snapshots, prevent_destroy, | |
| 333 | remote state. | |
| 334 | 5. Everything P2/P3 as follow-ups. | |
| 335 | ||
| 336 | ## Status | |
| 337 | ||
| 338 | `op-benjivers` authorized 2026-08-25 — the live EC2 leader has now been swept (see Live cluster state); | |
| 339 | its one live problem is L3 (`fleet/insights` wedged on the stale `ecr-cred`). Nothing here is | |
| 340 | auto-fixed; flag which items to turn into PRs. |
new above
offline