Public: anyone can read this; nobody can change it.

DOM-2 [NO-GO / DO NOT MERGE]: germinate on Cilium CNI — evidence it breaks first-try germination #59

Closed opened this view on origin
Closed without merging

DOM-2 (plat/mitosis#37) asked to swap the germination CNI to Cilium. Per the brief's STRICT-honesty clause, this is a NO-GO: on the k3d/k3s germination stack, neither viable Cilium configuration germinates a full platform first-try.

Notes

stopped closed without shipping

opened this view on origin

DOM-2 (plat/mitosis#37) asked to swap the germination CNI to Cilium. Per the brief's STRICT-honesty clause, this is a NO-GO: on the k3d/k3s germination stack, neither viable Cilium configuration germinates a full platform first-try. This branch (bin/up: --flannel-backend=none + --disable-network-policy + --disable-kube-proxy + Cilium 1.17.17 kube-proxy-replacement) is preserved as evidence — do not merge.

Two independent, reproducible blockers (both tested live on vxrail, clusters torn down)

1. kube-proxy PRESENT (Cilium CNI + netpol only): platform reaches the CI-build phase (all 25+ pods Running), then mcp/agents — and every in-cluster-built app image — fail ImagePull. Root cause: the platform's registry mirror is reached by containerd over a hostPort (127.0.0.1:31100, the registry-node-config DaemonSet), and Cilium only programs hostPort when it replaces kube-proxy. Setting enable-host-port=true alone (kpr=false) does not bind the port (verified: node wget 127.0.0.1:31100 → connection refused).

2. kube-proxy REPLACED (kpr=true, auto-enables hostPort): germination hard-fails at 'install Flux + point at gitops'. Flux's 4 controllers + metrics-server + local-path-provisioner all CrashLoopBackOff with x509: certificate signed by unknown authority when dialing the apiserver via the kubernetes ClusterIP. Cilium routing is correct (service maps 10.43.0.1:443 → 172.18.0.2:6443, a pod reaches it and gets 401), but the cert served under kpr routing is not trusted by the pods' ServiceAccount CA bundle (curl --cacert ca.crt → exit 60). bpf.hostLegacyRouting=true did not fix it. This is a known k3s/k3d + Cilium-kpr apiserver-trust incompatibility.

What DOES work on Cilium (proven, shipping separately)

On the kube-proxy-present run, before the registry blocker, I proved the value of Cilium end-to-end:

  • cilium status OK (1.17.17), Controllers healthy.
  • Standard networking.k8s.io/v1 NetworkPolicy still enforces natively: default-deny blocked (HTTP 000/timeout), an allow rule restored it (HTTP 200), endpoint ingress ENFORCEMENT=Enabled.
  • The mcp renderer's CiliumNetworkPolicy toFQDNs egress works: allowlisted api.github.com→200, non-allowlisted api.gitlab.com/pypi.org→BLOCKED, DNS preserved, egress ENFORCEMENT=Enabled.

Recommendation

  • Keep kube-router (the current default CNI) for germination — do not merge this bin/up change.
  • Ship the FQDN-egress renderer (plat/mcp feat/dom-2-fqdn-egress + plat/_app-template feat/dom-2-allowed-hosts): it is additive, gated default-off behind PLAT_FQDN_EGRESS, tested, and proven to work the moment the platform runs on Cilium.
  • Before any future Cilium adoption, the germination seam needs: (a) the registry mirror to stop depending on hostPort (e.g. a NodePort/hostNetwork socat, so kube-proxy can stay), OR (b) the k3s+Cilium-kpr apiserver-cert-trust issue resolved (dedicated apiserver LB SAN / cert handling).
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs