Public: anyone can read this; nobody can change it.

AWS deployment audit — EC2 leader + EKS backlog (34 items) #212

Closed opened this view on origin
Closed without merging

audit/aws-2026-08-25 → main 2 commits

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
1 file changed +340 −0
docs/ops/aws-audit-2026-08-25.md added +340
@@ -0,0 +1,340 @@
1 # AWS deployment audit — `git.benjivers.com` (EC2 leader) + `git.aws.benjivers.com` (EKS)
2
3 _Audit date: 2026-08-25. Scope: everything AWS in the mitosis platform — the single-node EC2
4 leader stack (`deploy/aws/`, serving `git.benjivers.com`) and the managed-EKS stack
5 (`deploy/eks/`, serving `git.aws.benjivers.com`), plus the EKS-mode germinate wiring on
6 branch `feat/2vcpu-daughter-fixes` and the live cluster state._
7
8 This document is a backlog, not a fix. Each item: severity, location, why it matters, and the
9 concrete change. Check items off as they land. Sensitive specifics (account ID, leader IP) are
10 redacted — they live only in gitignored Terraform state, never in this repo.
11
12 ## How to read the sources
13
14 - **`deploy/aws/`** — EC2 leader (k3s single node, Free-Plan account). Tracked in git.
15 - **`deploy/eks/`** — managed EKS (billed control plane, us-east-1). **Currently untracked.**
16 - **`bin/germinate` + `bin/lib.sh`** — germinate flow; the `mode=eks` / `CLUSTER_PROFILE=eks`
17 path is partly committed on `feat/2vcpu-daughter-fixes` and partly still in the working tree
18 (uncommitted: `bin/germinate` +56, `bin/lib.sh` +397).
19
20 ---
21
22 ## P0 — Critical (blocks safe/correct AWS operation)
23
24 - [ ] **C1. `deploy/eks/` is untracked — live EKS infra exists only on one laptop.**
25 `deploy/eks/*` (all `??` in `git status`), referenced by tracked code at `bin/lib.sh:733,755,1050`
26 and `bin/germinate:70`. The Terraform + `up.sh`/`down.sh`/`mirror-images-eks.sh`/`credit-guard.tf`
27 behind a live, billing EKS cluster is uncommitted; a fresh clone of the branch gets germinate/lib
28 EKS code that calls files that aren't there, and laptop loss = an orphaned running cluster + NLB
29 bleeding credits with no git record.
30 **Change:** Commit `deploy/eks/` as a unit alongside the `bin/` changes. Committing is safe —
31 `deploy/.gitignore` already excludes `*.tfstate`/`*.tfvars`/`.terraform/`, and a dry-run
32 `git add deploy/eks` pulls only source (`.tf`, `.sh`, `README.md`, `UPGRADE.md`, `lambda/`,
33 `patches/`, `.terraform.lock.hcl`, `terraform.tfvars.example`). Do **not** force-add state/tfvars.
34
35 - [ ] **C2. EKS Terraform state is local-only (no remote backend).**
36 `deploy/eks/versions.tf:12-15` (S3 backend commented out); a 329 KB `terraform.tfstate` sits on
37 the laptop as the only record of the live cluster and carries plaintext secrets (a
38 `random_password.result`, a `tls_certificate`, KMS metadata, account ID ×149). It is correctly
39 gitignored, but loss = unrecoverable orphaned billing.
40 **Change:** Create a versioned + SSE S3 bucket (TF ≥1.10 `use_lockfile=true`, no DynamoDB needed),
41 uncomment the `backend "s3"` block, `tofu init -migrate-state`. Same treatment recommended for the
42 EC2 stack (see H8).
43
44 - [ ] **C3. EKS germinate TLS path is half-landed — browser edges will render untrusted certs.**
45 `deploy/eks/patches/mcp-platform-edge-issuer.patch` (untracked, unapplied) vs `genesis/seed.tar.gz`
46 (ships mcp 0.39.2). `trim_gitops_eks` injects `platformEdgeIssuer: letsencrypt-dns01`
47 (`bin/lib.sh:915-925`) and `render_front_door` swaps the issuer via `PLATFORM_EDGE_ISSUER`
48 (`bin/lib.sh:1970-2010`), but the renderer support for that value lives only in the patch. The seed
49 predates it, so on the direct NLB (no Cloudflare TLS termination) every app edge + the front door
50 fall back to `plat-local-ca-issuer` → browser TLS errors on sign-in and all app edges. This is the
51 exact failure the EKS TLS story exists to prevent.
52 **Change:** Land the patch in the mcp source the seed is built from, cut a new seed whose
53 `integration.mcp` carries `platformEdgeIssuer` support, and gate germinate on `CLUSTER_PROFILE=eks`
54 to assert the seed's mcp tag supports it (fail loud, not silent no-op).
55
56 - [ ] **C4. Issuer name mismatch — certs reference a `letsencrypt-dns01` issuer that is never created.**
57 `bin/lib.sh:876-879,915-925` and `bin/germinate:85` point platform-edge + app Certificates at a
58 ClusterIssuer named `letsencrypt-dns01`, but the cloudflare→route53 swap (`bin/lib.sh:806-834`)
59 only rewrites the *solver* inside the seed's existing issuer (named `letsencrypt`) and never renames
60 it; `trim_gitops_eks` only otherwise creates `letsencrypt-http01`. Nothing produces
61 `letsencrypt-dns01` → every edge Certificate references a nonexistent issuer → certs never issue.
62 **Change:** Either rename the seed's DNS-01 issuer to `letsencrypt-dns01` in step 4d and verify, or
63 render a fresh route53 `clusterissuer-dns01.yaml` named `letsencrypt-dns01` (as 4f-3 does for
64 http01) and add it to the letsencrypt kustomization. Then hard-assert the issuer exists post-trim.
65
66 ---
67
68 ## Live cluster state (swept 2026-08-25)
69
70 **EKS — `git.aws.benjivers.com`:**
71
72 - [ ] **L1. `open-platform/www` prod is stuck.**
73 HelmRelease `Progressing` ("reconciliation in progress"), `deployment.found=false`, **zero pods**
74 (chart `[email protected]+77386226b940`). `fleet/www` prod on the same cluster is healthy, so this is
75 app/release-specific, not cluster-wide. Investigate the HelmRelease + source; reconcile or roll back.
76
77 **EC2 leader — `git.benjivers.com`** (op-benjivers authorized; swept 2026-08-25). 9 apps —
78 `fleet/{www,relay,review,insights}` + `invoicing/fleet-invoice`; all pull from ECR
79 (`<acct>.dkr.ecr.us-east-1.amazonaws.com`). Healthy: `fleet/www` (v3), `fleet/relay` (v2),
80 `fleet/review` (v4), `invoicing/fleet-invoice` (v23, app+worker).
81
82 - [ ] **L3. `fleet/insights` prod HelmRelease is wedged `UpgradeFailed` on a stale ECR pull secret.**
83 "failed early due to stalled resources: [Deployment/fleet--insights--prod/insights status:
84 'Failed']", with a recurring `FailedToRetrieveImagePullSecret` warning for `ecr-cred`. The pod
85 currently runs (0 restarts) but the release can't complete and any reschedule may fail to pull.
86 The leader pulls all images from ECR, whose auth tokens expire ~12h — a static
87 `platform/registry-pull-secret.yaml` goes stale without a refresher.
88 **Change:** Root-cause the `ecr-cred` refresh (find/repair or add the ECR-token refresh CronJob),
89 then `flux reconcile` / roll `fleet--insights--prod`. Also relevant to germinate H9 (ECR wiring).
90
91 **Both clusters:**
92
93 - [ ] **L2. `PLAT_FORWARD_AUTH` is unset.**
94 Every app Ingress on **both** the EC2 leader and the EKS cluster trips kyverno
95 `require-forwardauth-on-app-ingress/app-ingress-has-plat-access-middleware` as a coverage warn.
96 Benign while default-off, but if forwardAuth is ever enabled without re-rendering overlays, those
97 Ingresses serve with **no** auth. Decide: enable + add the `plat-strip,plat-access` chain, or leave
98 documented as intentionally off.
99
100 ---
101
102 ## P1 — High
103
104 - [ ] **H1. EKS public API endpoint defaults to `0.0.0.0/0` and the live cluster took the default.**
105 `deploy/eks/variables.tf:79` default `["0.0.0.0/0"]`, consumed `main.tf:52-56`; live `tfvars` sets
106 only `domain`. Combined with `enable_cluster_creator_admin_permissions=true` (`main.tf:58`) the whole
107 cluster's exposure rests on IAM alone with no network boundary. The variable's own description admits
108 this is "the top finding in most security reviews."
109 **Change:** Set `api_allowed_cidrs` to your egress CIDR(s) and `tofu apply`; make the var have no
110 default (force a choice) or have `up.sh`/germinate refuse a wildcard unless `ALLOW_OPEN_API=1`.
111
112 - [ ] **H2. Credit-guard fails OPEN and only brakes compute.**
113 `deploy/eks/lambda/credit_guard.py:40-64`, `credit-guard.tf`. Three gaps: (a) it only scales node
114 groups down — the EKS control plane (~$2.40–3/day) is delete-only and keeps billing when tripped;
115 (b) it reads `freetier:GetAccountPlanState`, which exists only on the *new* Free Plan — on a legacy
116 account `remaining` defaults to `9999` (line 42) so it silently never trips; (c) on trip it sets
117 `maxSize=1`, not 0, leaving autoscaling headroom.
118 **Change:** Verify the account returns real `GetAccountPlanState` data; change the `9999` fallback to
119 fail-**closed** (trip when unreadable); set `maxSize:0` on trip; add an AWS Budgets action + SNS that
120 can fire a cluster-delete runbook for the control-plane cost. Document it as a compute brake, not a
121 full stop.
122
123 - [ ] **H3. `down.sh` can leave billable orphans (NLB, Route53 zone).**
124 `deploy/eks/down.sh:28-63`. The NLB (LB-controller-owned) and Route53 records (external-dns-owned)
125 live outside TF state and are deleted via `kubectl` only if the cluster API answers. If the cluster
126 is already broken, teardown skips to `tofu destroy`, which hangs on `DependencyViolation` (NLB ENIs
127 pin the subnets) or leaves the NLB + hosted zone billing forever. Lines 69-72 only `echo`, they
128 don't assert clean.
129 **Change:** Add an AWS-CLI fallback that deletes LBs by VPC tag and empties/deletes the hosted zone
130 when the cluster is unreachable, before `tofu destroy`; make the final check assert no LB/zone/EBS
131 remains.
132
133 - [ ] **H4. EC2: `allowed_ssh_cidr` defaults to `0.0.0.0/0` (fail-open).**
134 `deploy/aws/variables.tf:35-39`, used `main.tf:90`. Live tfvars overrides to a /32 (box is fine),
135 but a fresh apply / lost tfvars / copied stack lands SSH open to the internet on a personal-account
136 box. **Change:** Remove the default (make it required) and add a `validation` block rejecting
137 `0.0.0.0/0`.
138
139 - [ ] **H5. EC2: no Free-Plan cost guardrails.**
140 `deploy/aws/main.tf`/`variables.tf` — no `aws_budgets_budget`, no metric alarm, and `instance_type`
141 is a free-form string (default `t3a.xlarge`) with no validation. A typo or `ha=true` (3× xlarge)
142 silently triples spend with nothing to catch it. **Change:** Add a Budgets budget + SNS notification
143 and a `validation` allowlisting Free-Plan-safe instance types.
144
145 - [ ] **H6. EC2: root EBS is unencrypted with no snapshots/backup.**
146 `deploy/aws/main.tf:153-156` sets only size/type. The root volume holds Postgres, the registry, and
147 the sovereign age key (`deploy/lib/mitosis-node.sh:243` writes `/root/fork-$DOMAIN.age`). No
148 `encrypted=true`, and no `aws_dlm_lifecycle_policy`/AWS Backup anywhere → at-rest secrets in
149 cleartext and total data loss on termination. **Change:** `encrypted=true` on `root_block_device`;
150 add a daily EBS snapshot policy.
151
152 - [ ] **H7. EC2: single leader, no `prevent_destroy`, and user-data edits replace the machine.**
153 `deploy/aws/main.tf:137-162` (instance), `:151` `user_data_replace_on_change=true`. `ha=false` means
154 one instance *is* the platform, and any edit to rendered cloud-init replaces it (cloud-init runs
155 once) wiping all germinated state; no `lifecycle { prevent_destroy = true }`. With no snapshots (H6),
156 a stray change or `terraform destroy` is unrecoverable. **Change:** Add `prevent_destroy=true` to the
157 instance/EBS/EIP, document that user-data edits are destructive, pair with H6 snapshots.
158
159 - [ ] **H8. EC2: cluster secrets in local, unencrypted, un-backed-up state.**
160 `deploy/aws/main.tf:16-19` (`random_password.k3s_token`) → `terraform.tfstate` (local backend, no
161 `backend` block in `versions.tf`). k3s join token + SSH pubkey + password results in cleartext on
162 the laptop only. **Change:** Same remote-encrypted-backend migration as C2 for the EC2 stack; confirm
163 laptop disk encryption and off-device backup meanwhile.
164
165 - [ ] **H9. germinate ECR path swallows errors and leaks tags.**
166 `bin/germinate:817-818` do ECR login + `describe-or-create-repository` with `>/dev/null 2>&1 ... ||
167 true`, so an auth/permission failure only surfaces later as an opaque `docker push` FATAL. The
168 on-demand `plat/ci-runner` ECR repo also gets no lifecycle policy, unlike `mirror-images-eks.sh`, so
169 runner tags accumulate forever. **Change:** Drop `|| true` on login (or WARN with the aws error);
170 apply the same `ECR_LIFECYCLE` put-lifecycle-policy to the `plat/ci-runner` repo germinate creates.
171
172 - [ ] **H10. kyverno image-redirect regex vs mirror list drift → `ErrImagePull` risk.**
173 Redirect regex `bin/lib.sh:1076,1099` vs mirror sources `deploy/eks/mirror-images-eks.sh:31-65`.
174 `registry.k8s.io` is in the policy but not mirrored; the `docker.io/library/*` and `minio/*`
175 short-name normalization is fragile and must match exactly how each chart writes its image ref. Any
176 unmatched ref stays pointed at the public registry, which nodes (by design) can't reach.
177 **Change:** Reconcile the redirect allowlist against the actual mirror list from one shared source;
178 add a render/coldstart assertion that every mirrored image maps to an ECR path the policy produces.
179
180 - [ ] **H11. traefik NLB external-dns hostname hardcoded to `*.open-platform.sh`.**
181 `trim_gitops_eks` traefik block injects
182 `external-dns.alpha.kubernetes.io/hostname: "*.open-platform.sh,open-platform.sh"` and relies on
183 `rewrite_identity_domain` to swap it afterward. That sweep only rewrites the *old* seed domain
184 string; a seed whose domain isn't `open-platform.sh` (a re-germinated `sslip.io`/gate seed) leaves
185 the literal unrewritten and external-dns publishes to the wrong zone → NLB unreachable at the
186 intended host. `up.sh:126-132` re-annotates but a traefik HelmRelease reconcile drops it.
187 **Change:** Write the annotation from the seed's actual domain (read from manifest) or substitute
188 `$DOMAIN` at inject time.
189
190 - [ ] **H12. EKS cert render is guarded on the seed shape and only WARNs on miss.**
191 `bin/lib.sh:882-905` (4f-3) runs the route53 swap + http01 render only if the seed's
192 `letsencrypt/clusterissuer.yaml` contains `cloudflare`. A lean/renamed/staging seed silently skips
193 the whole thing with a WARN and germinate proceeds to a platform whose certs can never issue — the
194 failure surfaces far away at cert-manager. **Change:** After trim, FATAL-assert on eks that both a
195 `letsencrypt-dns01` and a `letsencrypt-http01` issuer exist, rather than best-effort WARN.
196
197 ---
198
199 ## P2 — Medium
200
201 - [ ] **M1. EKS: orphaned `random_password.rds_master` cleartext secret in state.**
202 `deploy/eks/terraform.tfstate` holds it though no RDS resource remains (`main.tf:221-226` notes RDS
203 was removed). Dead but increases state sensitivity. **Change:** `tofu state rm
204 random_password.rds_master` (regenerate clean state after C2 backend migration); rotate if it was
205 ever wired to a real DB.
206
207 - [ ] **M2. EKS: `t3a.2xlarge` (8 vCPU, ~$7/day) dominates Free-Plan burn.**
208 `deploy/eks/variables.tf:33` default, `main.tf:72`; `node_max=2` (`variables.tf:51`) lets CI bursts
209 double it. **Change:** Consider a smaller type or spot capacity; keep `node_max` low; document daily
210 burn per size.
211
212 - [ ] **M3. EKS: credit-guard Lambda IAM is `Resource: "*"`.**
213 `deploy/eks/credit-guard.tf:46-49` — `eks:UpdateNodegroupConfig`/List/Describe on every cluster in
214 the account. **Change:** Scope the `eks:*` actions to the `var.cluster_name` ARN pattern; split
215 `freetier:GetAccountPlanState` (no resource scoping) into its own statement.
216
217 - [ ] **M4. EKS: `enable_cluster_creator_admin_permissions=true` + open API.**
218 `deploy/eks/main.tf:58`. Whoever ran `tofu apply` is permanent internet-reachable cluster-admin
219 while H1 stands. **Change:** Acceptable for a solo operator once H1 locks the API CIDR; for multi-op,
220 manage admins via explicit access entries.
221
222 - [ ] **M5. EKS: live `terraform.tfvars` (`domain = "aws.benjivers.com"`) sits in the worktree.**
223 `deploy/eks/terraform.tfvars:1`. Already excluded by `deploy/.gitignore` (verified) — the only action
224 is to **not** force-add it when committing C1, and keep `terraform.tfvars.example` as the tracked copy.
225
226 - [ ] **M6. 2-vCPU kyverno fix lives only in the lean profile.**
227 `bin/lib.sh:297-308` (`trim_gitops_lean`, commit `6ff70a9`) pins kyverno replicas 3→1, fixing the
228 small-node stall — but `trim_gitops_eks` (which states it must be self-sufficient) doesn't re-assert
229 it, so a `full`-profile eks germination on a small node group reintroduces the 3-replica CPU stall.
230 **Change:** Replicate the replicas→1 pin (or a node-sizing guard) inside `trim_gitops_eks`, or
231 document + assert that eks requires the lean profile.
232
233 - [ ] **M7. NS-delegation prerequisite is manual and unchecked.**
234 `deploy/eks/up.sh:102` germinates right after `tofu apply` with no check that `aws.benjivers.com` is
235 actually delegated/resolving (the documented gate). If delegation isn't live, germinate stalls at the
236 first public-cert issuance and wastes the ~60-90m budget with an opaque failure. **Change:** Between
237 terraform and germinate, `dig +short NS $DOMAIN` against a public resolver and refuse/prompt until
238 delegation is observed; surface `route53_nameservers` prominently.
239
240 - [ ] **M8. EC2: DNS/Cloudflare wiring is entirely manual.**
241 `deploy/aws/outputs.tf:6-9` just prints a wildcard-A instruction; no `cloudflare_record` resource.
242 `*.benjivers.com → leader EIP` lives only in Cloudflare, untracked, and breaks silently on EIP
243 re-allocation. **Change:** Add the `cloudflare/cloudflare` provider and manage the wildcard (and
244 per-node HA) records off `aws_eip.node[*].public_ip`.
245
246 - [ ] **M9. EC2: Forgejo git-over-SSH (2222) is world-open.**
247 `deploy/aws/main.tf:92-112`. 80/443 world-open is expected for a public edge; 2222 broadens the auth
248 surface on a personal box. **Change:** Restrict 2222 to operator/CI CIDRs if public git-SSH isn't
249 needed, or document as intentional.
250
251 - [ ] **M10. EC2: no cost-allocation tags / `default_tags`.**
252 `deploy/aws/versions.tf:15-17` provider has no `default_tags`; resources carry only `Name`. Hurts
253 spend attribution + cleanup on a cost-sensitive account. **Change:** Add provider `default_tags`
254 (`ManagedBy=terraform`, `Project=mitosis`, …).
255
256 - [ ] **M11. `resolve_seed`/`publish_seed` use inconsistent Forgejo package API base paths.**
257 `bin/lib.sh` — `resolve_seed` lists via `/api/v1/packages/plat?...` for `latest` then hits
258 `/api/packages/...`; `publish_seed` PUTs `/api/packages/...`. A mismatch means `SEED=latest` silently
259 fails to resolve. Low urgency for eks (its `up.sh` uses the bundled `genesis/seed.tar.gz`), but it's
260 the documented fetch path. **Change:** Validate the three endpoints against the running forge; error
261 clearly when `latest` returns empty.
262
263 - [ ] **M12. EC2: subnet AZ not pinned (implicit/driftable).**
264 `deploy/aws/main.tf:60-65` omits `availability_zone`. Low blast radius for a single fixed deploy but
265 the AZ can drift between applies. **Change:** Optionally pin `availability_zone` for determinism.
266 (AMI pinning via region-correct SSM is already correct — no change there.)
267
268 ---
269
270 ## P3 — Low
271
272 - [ ] **Lo1. `mitosis-node.sh` report() ships admin passwords over the network.**
273 `deploy/lib/mitosis-node.sh` `report()` lifts Forgejo + Grafana admin passwords from the germinate
274 log into a JSON POST to `REPORT_URL` (opt-in, bearer-gated, best-effort, no TLS assertion).
275 **Change:** Reject non-HTTPS `REPORT_URL`; document that this channel carries admin creds so
276 `REPORT_TOKEN`/endpoint are treated as secret-grade. Also consider tightening
277 `--write-kubeconfig-mode 644` (`mitosis-node.sh:141`) to `600` if the VM is ever multi-user.
278
279 - [ ] **Lo2. EKS: mirror list + edge-issuer are floating workarounds.**
280 `deploy/eks/mirror-images-eks.sh` must be hand-kept in sync with the AKS mirror list and the
281 helmrelease pins (drift → silent `ImagePull` fail), and `patches/mcp-platform-edge-issuer.patch` is a
282 floating git patch that must be reapplied on every mcp bump (see C3). **Change:** CI check that the
283 EKS/AKS mirror lists match the helmrelease pins; upstream the edge-issuer change into the mcp chart
284 as a real flag, or add a patch-applies-cleanly check to `up.sh`.
285
286 - [ ] **Lo3. germinate ci-runner cross-arch build assumes buildx/emulation.**
287 `bin/germinate:820` `docker build --platform linux/amd64` on an arm64 laptop needs buildx + binfmt;
288 the `command -v docker` check (`:87`) doesn't verify it. **Change:** Preflight `docker buildx inspect`
289 (or verify emulation) and error clearly if amd64 cross-build isn't available.
290
291 - [ ] **Lo4. EC2: provider/Terraform version constraints are loose.**
292 `deploy/aws/versions.tf:1-12` — `~> 6.0`/`~> 3.6` and `required_version >= 1.6` with no ceiling. The
293 committed lock file mitigates, but `init -upgrade` can jump minors. **Change:** Optionally tighten to
294 `~> 6.58` and add a Terraform upper bound `< 2.0`.
295
296 - [ ] **Lo5. EKS: `credit_guard.py` region hardcoded `us-east-1`.**
297 `deploy/eks/lambda/credit_guard.py:40`. Correct-by-necessity (the `freetier` endpoint is
298 us-east-1-only) but reads like a portability bug. **Change:** Add a clarifying comment; leave as-is.
299
300 - [ ] **Lo6. Repo housekeeping in the worktree.**
301 Stray `.DS_Store`, `germinate-aks-poc*.log`, `mirror-run2.log`, `plat-ca.crt`, and
302 `fork-aws.benjivers.com.age.backup` sit untracked. Add `.DS_Store` (and `*.log` scratch) to
303 `.gitignore`; confirm `fork-*.age*` stays ignored (it is).
304
305 ---
306
307 ## Already correct — do not "fix" these
308
309 - IMDSv2 enforced (`deploy/aws/main.tf:157-159`, `http_tokens=required`).
310 - user-data gzip for the EC2 16 KB cap (`main.tf:146-148`), correctly commented.
311 - AMI via region-correct SSM pointer — no hardcoded AMI id.
312 - `.terraform.lock.hcl` committed with full `h1:`/`zh:` hashes (EC2); should be committed for EKS too.
313 - `deploy/.gitignore` correctly excludes `*.tfstate`/`*.tfvars`/`.terraform/` (keeps `*.example`) —
314 so C1 is safe to commit.
315 - Sensitive vars marked `sensitive = true`; no account ID or leader IP in any tracked file.
316 - EIPs decouple DNS from instance lifecycle.
317 - `mirror-images-eks.sh` forces `--override-arch amd64` and applies an ECR lifecycle policy
318 (expire untagged >7d, keep last 10).
319 - The edge-issuer renderer change itself is clean + test-covered — it just needs to land in mcp source
320 and a fresh seed (C3), not be a floating patch.
321
322 ---
323
324 ## Suggested landing order
325
326 0. **L3 (operational hotfix)** — repair the `ecr-cred` refresh so `fleet/insights` (and any future
327 pull) stops failing; roll the wedged release. This is live right now.
328 1. **C1 + C2** — commit `deploy/eks/`, migrate EKS state to S3. (Stops the single-laptop SPOF.)
329 2. **C3 + C4** — fix the issuer name + land `PLATFORM_EDGE_ISSUER` in mcp source, cut a new seed.
330 (Without these, an eks germination comes up with dead TLS on every browser edge.)
331 3. **H1 / H2 / H3** — lock the API CIDR, make credit-guard fail-closed, harden teardown.
332 4. **EC2 hardening H4–H8** — SSH default, budget guard, EBS encryption + snapshots, prevent_destroy,
333 remote state.
334 5. Everything P2/P3 as follow-ups.
335
336 ## Status
337
338 `op-benjivers` authorized 2026-08-25 — the live EC2 leader has now been swept (see Live cluster state);
339 its one live problem is L3 (`fleet/insights` wedged on the stale `ecr-cred`). Nothing here is
340 auto-fixed; flag which items to turn into PRs.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs