Public: anyone can read this; nobody can change it.
AWS / Azure-AKS / EKS cloud-deploy build history #221
Check · Review · Land · Live — No review yet.
docs/aws-eks-build-history → main 1 commit
A guest
This pull comes from a guest: its preview runs without the app's variables or a forge token until a person who can write here approves it.
This pull waits for a person who can write here to review it; it never lands on its own.
1 file
changed +266
−0
docs/ops/aws-eks-build-history.md added +266
| @@ -0,0 +1,266 @@ | ||
| 1 | # Mitosis cloud-deploy build history — AWS, Azure/AKS, EKS | |
| 2 | ||
| 3 | _Compiled 2026-08-25 from git history, closed PRs, in-repo build notes, and germination run | |
| 4 | logs. This is the trail of problems hit and fixes made while standing the cloud-deploy paths | |
| 5 | up — the "how the build went." It is the companion to the current-state backlog in | |
| 6 | `aws-audit-2026-08-25.md` (PR #212): that doc is what's still wrong today; this doc is how we | |
| 7 | got here._ | |
| 8 | ||
| 9 | ## The three tracks and how they relate | |
| 10 | ||
| 11 | - **AWS / EC2-k3s** — `deploy/aws`, `deploy/do`, `deploy/lib`. k3s on a bare cloud VM; the | |
| 12 | origin path. Serves the live `git.benjivers.com` leader. | |
| 13 | - **Azure / AKS** — `deploy/azure`, the `azure-gate` CI, `trim_gitops_aks`. The first *managed*- | |
| 14 | Kubernetes ground; where the hard managed-k8s lessons were paid for. | |
| 15 | - **EKS** — `deploy/eks`, `trim_gitops_eks`. Modeled directly on the AKS work and inherits its | |
| 16 | trim structure. Serves `git.aws.benjivers.com`. | |
| 17 | - **Shared spine** — `deploy/lib/mitosis-node.sh`, `bin/lib.sh`, `bin/up`. Node bring-up + the | |
| 18 | gitops-trim library used by every track. | |
| 19 | ||
| 20 | **Inheritance chain:** a lot of what makes AWS/EKS work was actually fixed on the AKS path first | |
| 21 | (managed-k8s trim, DNS-01 issuer swap, registry image-redirect via Kyverno, local-path storage | |
| 22 | strip, control-plane-pin removal) and carried over. `trim_gitops_eks` is a direct descendant of | |
| 23 | `trim_gitops_aks`, and the shared `mitosis-node.sh` fixes apply to all clouds. | |
| 24 | ||
| 25 | ## Timeline at a glance | |
| 26 | ||
| 27 | | Date | Ref | Track | Headline | | |
| 28 | |---|---|---|---| | |
| 29 | | 2026-06-19 | 213faee / 291de83 | AWS | cold-start hardening under `set -u` + nix locale | | |
| 30 | | 2026-06-30 | 270136a | AWS | HYB-1: bare single-VPS k3s + Cloudflare-tunnel edge | | |
| 31 | | 2026-07-18 | da7df45 (#101) | AWS | born multinode-ready (Docker ≥28 raw-table DROP fix) | | |
| 32 | | 2026-08-13 | bd4509f (#120) | AWS+AKS | one command to any cloud (TF + bicep foundation) | | |
| 33 | | 2026-08-14 | 4a42cc4 (#123/#124) | spine | bake static public IP into node.env | | |
| 34 | | 2026-08-17 | ca3e481 (#132) | spine | zero-DNS (`DOMAIN=auto` sslip.io) + report-home | | |
| 35 | | 2026-08-19 | fd01a7e (#139) | AKS | `mode=aks` + `trim_gitops_aks` managed-k8s profile | | |
| 36 | | 2026-08-19 | ebb0864→e4ae956 (#143–#159) | AKS/CI | the azure-gate real-cloud CI saga | | |
| 37 | | 2026-08-19 | e4ae956 (#159) | AKS | cert-zone doubling → certs never issued (the big one) | | |
| 38 | | 2026-08-18 | (#162/#167/#178/#183/#187/#188) | AKS | CIDRs, per-run DNS, coredns/mirror/CNPG, storageClass | | |
| 39 | | ~2026-08-17 | aks-cost-experiment.md | AKS | 2-node → 1-node cost cut; a pile of `az` gotchas | | |
| 40 | | 2026-08-20 | e1c2740 | AWS | gzip user-data (EC2 16 KB cap) | | |
| 41 | | 2026-08-25 | 8001607 | EKS | turnkey hardening + `trim_gitops_eks` | | |
| 42 | ||
| 43 | --- | |
| 44 | ||
| 45 | ## Track A — AWS / EC2-k3s | |
| 46 | ||
| 47 | ### Cold-start hardening (June) | |
| 48 | ||
| 49 | - **213faee — brace `${CLUSTER}` for non-UTF-8 locales.** A UTF-8 ellipsis abutting `$CLUSTER` | |
| 50 | in the germinate hand-off line folded into the variable name under nix-run's locale → | |
| 51 | `CLUSTER: unbound variable` aborted `up` right after k3d creation. Fix: brace-delimit | |
| 52 | `${CLUSTER}`; also gitignore the sovereign `fork-*.age` keys `up` mints. | |
| 53 | - **291de83 — harden cold-start for send-out.** Under `set -u`, an unset `MITOSIS_RECIPE` | |
| 54 | expanded the whole recipe-search list before the in-loop guard ran; a stopped cluster on | |
| 55 | rerun minted a fork key then died; the seed was found only via checkout CWD, not the flake. | |
| 56 | Fix: guard `${MITOSIS_RECIPE:-}`, preflight `docker info` + `k3d cluster start` on reuse, and | |
| 57 | bundle `genesis/` into the flake via `--set MITOSIS_GENESIS`. | |
| 58 | - **270136a — HYB-1: bare-k3s single-VPS + Cloudflare-tunnel edge.** The bare-server fallback | |
| 59 | recipe wasn't copy-paste-correct for a real k3s node, and there was no outbound-only edge for | |
| 60 | a VPS behind a default-deny firewall. Fix: rewrite the recipe (registries.yaml *before* k3s | |
| 61 | start; `--disable=traefik/servicelb --tls-san`; SSH-only firewall) and fork the reach-it card | |
| 62 | on `CUTOVER_TUNNEL` (tunnel → traefik.kube-system.svc; drop the self-signed-CA step since a | |
| 63 | public issuer serves trusted certs). | |
| 64 | ||
| 65 | ### Multinode networking (July) | |
| 66 | ||
| 67 | - **da7df45 (#101) — born multinode-ready.** Docker ≥28 installs raw-table rules that DROP | |
| 68 | directly-routed traffic to container IPs, breaking the routed agent fabric (subnet-router → | |
| 69 | server container). Fix: pre-create the k3d docker network with | |
| 70 | `gateway_mode_ipv4=nat-unprotected` (falling back to plain create on older Docker); survives | |
| 71 | cluster rebuilds. | |
| 72 | ||
| 73 | ### Turnkey cloud IaC (August) | |
| 74 | ||
| 75 | - **bd4509f (#120) — one command to any cloud.** No turnkey IaC existed. Fix: add `deploy/aws` | |
| 76 | Terraform — region-correct Canonical Ubuntu 24.04 AMI via SSM (no AMI table to maintain), VPC | |
| 77 | CIDR clear of k3s pod/service ranges, k3s-token password, security group, EIP + | |
| 78 | `user_data_replace_on_change` for fresh nodes; `-var ha=true` grows 3 etcd servers. (Same PR | |
| 79 | introduced `deploy/azure/main.bicep`.) | |
| 80 | - **ca3e481 (#132) — zero-DNS deploy + node reports home.** A cloud deploy required manual DNS, | |
| 81 | and a headless deploy had no way to observe germination. Fix (shared lib, inherited by all | |
| 82 | clouds): `DOMAIN=auto` derives a self-resolving `<ip-with-dashes>.sslip.io` host *before* k3s | |
| 83 | install (and realigns the registry-mirror host); the node phones home progress/card/failure as | |
| 84 | best-effort JSON POSTs via `REPORT_URL`+`REPORT_TOKEN`. | |
| 85 | - **4a42cc4 (#123/#124) — bake the static public IP into node.env.** Azure IMDS leaves the | |
| 86 | public-IP leaf empty for Standard-SKU IPs so `pub_ip()` returned nothing; the SAN guard then | |
| 87 | silently dropped an empty `PUBLIC_IP`, leaving the k3s API cert with no public-IP SAN. AWS/EKS | |
| 88 | share the same `mitosis-node.sh` path. Fix: `PUBLIC_IP="${PUBLIC_IP:-$(pub_ip)}"` so an | |
| 89 | IaC-rendered static IP wins over live metadata. | |
| 90 | - **e1c2740 — gzip user-data (EC2 16 KB cap).** Rendered cloud-config is ~21 KB but EC2 rejects | |
| 91 | `user_data` >16 KB (DO/Azure allow 64 KB), failing at apply. Fix: `user_data_base64 = | |
| 92 | base64gzip(...)` — cloud-init decompresses gzip transparently; `user_data_replace_on_change` | |
| 93 | still forces a fresh node. | |
| 94 | ||
| 95 | ### AWS operational notes (`deploy/README.md`) | |
| 96 | ||
| 97 | - Elastic IPs are assigned so DNS survives VM stop/start (public IP would otherwise change). | |
| 98 | - The root Ubuntu AMI is pulled from Canonical's official SSM parameter — no AMI table to keep. | |
| 99 | - Day-0 IaC is provisioning-only: don't re-apply TF to change a running platform (user-data | |
| 100 | edits replace the VM). Change a running platform via its own `git.<domain>/plat/gitops`. | |
| 101 | ||
| 102 | --- | |
| 103 | ||
| 104 | ## Track B — Azure / AKS (the managed-k8s precedent) | |
| 105 | ||
| 106 | ### `mode=aks` and the managed-k8s trim | |
| 107 | ||
| 108 | - **fd01a7e (#139) — `mode=vms|aks` in main.bicep + AKS germinate profile.** Azure only | |
| 109 | supported k3s-on-VMs; a lean gitops tree assumes k3s-isms (local-path storage, coredns | |
| 110 | rewrite, control-plane node pins, in-cluster registry mirror) that break on AKS. Fix: add | |
| 111 | `mode=aks` + HA `servers.bicep` carve-out and `trim_gitops_aks` (`CLUSTER_PROFILE=aks`) — keep | |
| 112 | Azure Monitor scrapes, drop coredns/node-fabric, repoint runner + Kyverno image-redirect at | |
| 113 | ACR, and remove control-plane pins (AKS nodes carry no control-plane label → pods otherwise | |
| 114 | permanently unschedulable, observed on the first germination). | |
| 115 | - **local-path storageClass strip (same commit).** AKS has no `local-path` class; a PVC pinning | |
| 116 | it stays Pending, and CNPG's hardcoded class made a first AKS germination "unrecoverable." Fix: | |
| 117 | strip `storageClass(Name): local-path` from CNPG and PVCs so the cluster default (`managed-csi`) | |
| 118 | binds. (Later refined by **#188** — *replace* not *remove* the class, to fix the E4 e2e.) | |
| 119 | ||
| 120 | ### The azure-gate CI saga (a real cloud germination, gated in CI on a NixOS runner) | |
| 121 | ||
| 122 | - **ebb0864 (#143)** — introduce the manual-dispatch gate that actually deploys to Azure in both | |
| 123 | `vms` and `aks` modes (`azure-gate.yml` + `gate.sh` + coldstart-assert + GATE-NOTES). | |
| 124 | - **55ec5e6 (#151)** — `bicep` wasn't on PATH on the NixOS runner; put it on PATH. | |
| 125 | - **039b703 (#154)** — the nixpkgs `az deployment` module *crashes* on NixOS; drive the deploy | |
| 126 | through raw `az rest` instead. | |
| 127 | - **e46b793 (#156)** — teardown `pkill` matched and killed the gate's own shell; narrow it to | |
| 128 | only the kubectl port-forward. | |
| 129 | - **c1ef012 (#158)** — the gate read pass/fail from the *ssh* exit code, giving a false FAIL | |
| 130 | after a real PASS; derive the result from the assert's exit code. | |
| 131 | - **e4ae956 (#159) — the big one: AKS cert-zone doubling.** `trim_gitops_aks`'s azureDNS sweep | |
| 132 | (germinate's `rewrite_identity_domain` mother→daughter sed) doubled the hosted-zone name | |
| 133 | (`gl3.gate.gl3.gate.open-platform.sh`) because a gate daughter's domain *ends with* the mother | |
| 134 | domain, so the already-correct `hostedZoneName` matched on its suffix. cert-manager DNS-01 then | |
| 135 | 403'd → `wildcard-public-tls` never issued → every push/pull to `git.<domain>` x509-failed and | |
| 136 | retried forever. Fix: sentinel-shield values already equal to the daughter domain before the | |
| 137 | sweep (so `$new` can't re-expand `$old`); plus a Cloudflare-403 preflight and a PASS-exits-1 | |
| 138 | fix. **This same doubling bug is the root of the EKS issuer risk tracked in PR #212 (C4).** | |
| 139 | - **#167** — stop the AKS gate touching Cloudflare from the runner; delegate per-run DNS via `az`. | |
| 140 | - **#162** — the seed hardcoded k3s pod/service CIDRs that don't hold on AKS (broke the A4 assert | |
| 141 | and left latent-broken NetworkPolicies); rewrite to AKS CIDRs + surface germinate WARNs. | |
| 142 | - **#178** — the meaty bundle: CoreDNS hairpin resolution, in-cluster image mirror, image | |
| 143 | tag-push, image-redirect rewriting, and CNPG breakage on AKS. | |
| 144 | - **#183 / #187** — fit the e2e in a 55m/60m budget; pin Kyverno sovereign-registry patterns for | |
| 145 | flat daughters. | |
| 146 | - **command-injection catch (GATE-NOTES).** The `ssh_pubkey` workflow input, if interpolated | |
| 147 | into the remote command, was a root-RCE vector (adversarial review). Fix: validate to one | |
| 148 | well-formed key line and pipe over stdin; `claudeCodeOauthToken` is `@secure()`, gate logs | |
| 149 | filtered to `==>|FATAL` so no secret prints. | |
| 150 | ||
| 151 | ### The AKS cost experiment (`deploy/azure/aks-cost-experiment.md`) — 2-node → 1-node, ~62% cut | |
| 152 | ||
| 153 | A string of `az`/AKS gotchas paid for while shrinking the cluster: | |
| 154 | ||
| 155 | - **CNPG PDBs block every node drain.** `az aks nodepool scale/upgrade/resize` fails | |
| 156 | (provisioningState=Failed) because each CNPG cluster has `minAvailable=1` with 1 replica → 0 | |
| 157 | allowed disruptions. Workaround: manually `kubectl cordon` + `drain --disable-eviction` first | |
| 158 | (deletes pods directly; CNPG rebuilds each pg pod in ~1 min). | |
| 159 | - **`nodepool delete-machines` refuses "all" system nodes** (ignores the cordoned node when | |
| 160 | counting) → use plain `nodepool scale --node-count 1` (VMSS scale-in removes the drained, | |
| 161 | highest-ID instance). | |
| 162 | - **`az aks nodepool add` crashes client-side** (`AttributeError … artifact_streaming_profile`, | |
| 163 | azure-cli SDK-model mismatch) even with `--no-wait` → do the raw ARM call via | |
| 164 | `az rest --method put …/agentPools/…`. | |
| 165 | - **Zero B-v2 vCPU quota** (Basv2/Bsv2/Bpsv2) in eastus made the planned B4as_v2 impossible → use | |
| 166 | `Standard_D4as_v7` (had free vCPUs; full-perf AMD). | |
| 167 | - **D4as_v7 attaches max 12 disks** (vs 16 on D4ds_v6) but the cluster has 18 disk PVs → only | |
| 168 | works because hibernated PR apps keep disks detached; else pods stick Pending "exceed max | |
| 169 | volume count." | |
| 170 | - **CRITICAL — CNPG restart-loop under image-redirect.** Kyverno `aks-image-redirect` mutates | |
| 171 | *pod* images to ACR but the CNPG `Cluster` spec still says ghcr.io → CNPG sees spec≠pod and | |
| 172 | restart-loops the primary forever; all 9 tenant clusters 503 with no ready DB. Fix: extend the | |
| 173 | redirect to also mutate `clusters.postgresql.cnpg.io` `spec.imageName`. | |
| 174 | - **Single-node request-pressure.** After scaling to 1 node, CPU *requests* hit 99% (usage ~46%) | |
| 175 | and pods stuck FailedScheduling "Insufficient cpu" → trim kyverno-admission + metrics-server to | |
| 176 | 1 replica (frees ~350m); make durable via gitops, since kubectl-only reverts on the next | |
| 177 | Helm/Flux/addon reconcile. | |
| 178 | - **Disabling managed Prometheus deletes the addon's PodMonitor/ServiceMonitor CRs** — client- | |
| 179 | ready still needs cheap in-cluster alerting. | |
| 180 | - **bicep drift:** `main.bicep` still pins 2×D4ds_v6 + omsagent, so a re-germinate resurrects the | |
| 181 | ~$364/mo baseline — update it to the kept end-state or the savings evaporate. | |
| 182 | ||
| 183 | ### POC run-log lessons (`germinate-aks-poc*.log`, `mirror-run2.log`) | |
| 184 | ||
| 185 | - **mcp chart cold-cache stall.** The mcp chart (v0.35.3) missed the 35m CI window twice (cold | |
| 186 | mirror cache) before landing on attempt 3; the germinate loop re-pushes the tag each retry to | |
| 187 | re-trigger CI and warm the cache. | |
| 188 | - **docker.io 429 on ACR import.** `az acr import` of Docker Hub images (traefik/alpine/caddy/ | |
| 189 | minio) hit HTTP 429 TOOMANYREQUESTS (unauthenticated pull limit) even on the auth-free retry; | |
| 190 | images from ghcr/quay/kyverno/forgejo succeeded. Remedy: rerun (nothing partial is harmful) — | |
| 191 | and **#170** added per-import retry-with-backoff for exactly this. | |
| 192 | - **Flux push-webhook registration** failing during germination is non-fatal (interval polling | |
| 193 | covers it). | |
| 194 | - **Single-node registry chicken-and-egg.** During a 1-node blue/green swap, platform images | |
| 195 | pull from the cluster's *own* Forgejo registry; while forgejo relocated, the registry 503'd → | |
| 196 | ~3 min ImagePullBackOff. Self-heals; plan for it on any single-node move. | |
| 197 | ||
| 198 | --- | |
| 199 | ||
| 200 | ## Track C — EKS (modeled on the AKS precedent) | |
| 201 | ||
| 202 | - **8001607 — turnkey hardening.** The initial `CLUSTER_PROFILE=eks` path used IRSA | |
| 203 | (cluster-specific roles), an in-cluster kube-prometheus-stack, a top-level `disk_size` that was | |
| 204 | a no-op under the custom launch template, a billed-but-unused RDS, a hand-deployed credit-guard | |
| 205 | Lambda, and left an orphaned NLB/zone on teardown; local-path storage and cross-AZ EBS don't | |
| 206 | work on EKS. Fix: IRSA → **EKS Pod Identity** (portable per-client roles); monitoring → | |
| 207 | **CloudWatch Container Insights** (strip in-cluster Prometheus); `block_device_mappings` for | |
| 208 | disk; **single-AZ node pin** so EBS co-locates; drop RDS (optional `enable_rds` toggle); | |
| 209 | credit-guard as **IaC** (budget/Lambda/SNS scale-to-zero); `down.sh` clean teardown; ECR | |
| 210 | lifecycle + mirror drift check; region-portable AZs. | |
| 211 | - **`trim_gitops_eks` (same commit) — mirrors `trim_gitops_aks`.** A lean gitops tree carries | |
| 212 | k3s/Azure assumptions that break on EKS. Fix: strip node-fabric/coredns/cloudflare/monitoring/ | |
| 213 | flow; local-path → **gp3** (EBS CSI default, enforced by a Kyverno admission strip); ACME | |
| 214 | cloudflare → **route53 DNS-01 via IRSA** (cert-manager SA annotated with the role ARN); runner + | |
| 215 | Kyverno `eks-image-redirect` → **ECR** (after `mirror-images-eks.sh`); restrict-image-registries | |
| 216 | learns the ECR registry; BYOD issuer via `PLATFORM_EDGE_ISSUER` + `wildcard-public-tls` swap. | |
| 217 | ||
| 218 | ### EKS operational rules learned (`deploy/eks/UPGRADE.md`) | |
| 219 | ||
| 220 | - **Version drift is a money trap.** Letting an EKS minor reach end-of-standard-support | |
| 221 | auto-enrolls extended support (~$0.50/hr ≈ $360/mo) and eventually force-upgrades. Upgrade | |
| 222 | deliberately, one minor at a time, via the single `k8s_version` knob. | |
| 223 | - **Node roll needs `--force`.** A plain node-group roll wedges on the same single-instance CNPG | |
| 224 | PDBs seen on AKS; use `aws eks update-nodegroup-version … --force` (brief single-node outage). | |
| 225 | - **Single-AZ EBS.** The node group is pinned to one AZ so a replacement node returns in the | |
| 226 | volumes' AZ (EBS can't cross AZ) — otherwise pods won't schedule after a node swap. | |
| 227 | - **Control-plane billing is delete-only.** Scaling nodes to zero saves the node cost but the | |
| 228 | control plane ($0.10/hr) bills 24/7 until `tofu destroy`. | |
| 229 | ||
| 230 | ### EKS status | |
| 231 | ||
| 232 | **No merged EKS PRs exist.** The mode is code-complete but was gated on `aws.benjivers.com` NS | |
| 233 | delegation, so germinate ran out-of-band and the `deploy/eks/` stack is **still untracked in | |
| 234 | git** — the top item in the current-state audit (PR #212, C1). The live TLS/issuer wiring also | |
| 235 | inherits the AKS cert-zone-doubling lineage (PR #212, C3/C4). | |
| 236 | ||
| 237 | --- | |
| 238 | ||
| 239 | ## Cross-cloud spine and CI | |
| 240 | ||
| 241 | - **`mitosis-node.sh`** carries the shared bring-up fixes used by every track: public-IP fallback | |
| 242 | (4a42cc4), `DOMAIN=auto` sslip.io self-resolving host + report-home (ca3e481), and the HA | |
| 243 | static-private-IP scheme (`10.60.0.10/.11/.12`) with round-robin DNS. | |
| 244 | - **#169** — signed upstream propagation: a signed release manifest verified against a seed | |
| 245 | pubkey, baked into germinate, with a P1/P2 gate. | |
| 246 | - **#170** — mirror-images retry-with-backoff (docker.io throttling). | |
| 247 | - **#164** — coldstart canary must be org-owned (fix E3). | |
| 248 | - **#177 / #182** — `FLAT_HOSTS=1` germinate under the mother's apex; strip mother-only surfaces | |
| 249 | from every daughter. | |
| 250 | - **#197** — pause nightly paid-cloud (Azure/AKS) autoruns during active dev to avoid spend. | |
| 251 | ||
| 252 | --- | |
| 253 | ||
| 254 | ## What remained open at the end of the build | |
| 255 | ||
| 256 | Everything unresolved rolled into the current-state audit — see `aws-audit-2026-08-25.md` | |
| 257 | (PR #212). The through-lines from this history that are still open: | |
| 258 | ||
| 259 | - `deploy/eks/` never got committed (C1) and its state is laptop-only (C2). | |
| 260 | - The cert-zone-doubling / issuer-naming lineage from #159 still bites the EKS TLS path (C3/C4). | |
| 261 | - The credit-guard shipped as IaC (8001607) but **fails open** on a legacy account and only | |
| 262 | brakes compute (H2). | |
| 263 | - The ECR mirror-list vs Kyverno-redirect drift the AKS/EKS work introduced has no CI check | |
| 264 | (H10 / Lo2). | |
| 265 | - The two live app-status issues found during the audit are tracked separately as #215 | |
| 266 | (`ecr-cred` stale on the leader) and #216 (`open-platform/www` stuck on EKS). |