Public: anyone can read this; nobody can change it.

Kyverno generate backlog: new namespaces wait minutes for tenant-defaults LimitRange → pods denied by require-resource-limits #220

Closed opened this view on origin

Closed without the crew

opened this view on origin

Observed 2026-08-25 ~20:05Z on prod while a PR preview (otavert/issue-map pr-4) came up.

Symptom: the preview namespace got its Deployment but every pod was denied: require-resource-limits / require-requests-and-limits — because the namespace had no tenant-defaults-standard LimitRange yet (pr-3, created 45 min earlier, had one). The RS sat in FailedCreate backoff.

Cause: the Kyverno background controller is starved. kubectl get updaterequests -A = 246 Pending (231 for plat-ns-network-governance, 12 plat-ns-resource-governance, 3 plat-ns-dos-bound), the oldest 39 min. The controller is at ~500m CPU looping on Operation cannot be fulfilled on networkpolicies … "default-deny" / "allow-ingress-traefik" / "allow-ingress-from-workflows" — resourceVersion conflicts while synchronize: true rules re-apply NetworkPolicies that something else also writes. Everything queued behind those (the LimitRange generate) waits.

Impact: every new namespace (PR previews, new app bootstraps → v1 previews) is blocked for as long as the backlog takes; the platform looks like 'preview never comes up'.

Unblocked pr-4 by hand: copied the LimitRange from pr-3 into the namespace.

Suggested fix: find the second writer of those NetworkPolicies (org Kustomization? overlay?) and make one owner; or turn synchronize off for the netpol rules; and generate the LimitRange from its own policy so a netpol storm cannot starve it. Consider a background-controller replica bump as a stopgap.

🤖 Generated with Claude Code

commented

Second writer found and removed. 35 pre-POL-2 app manifests in plat/gitops still shipped the four generic NetworkPolicies through Flux (prune: true) while plat-ns-network-governance owns the same objects with synchronize: true — the two controllers rewrote each other's resourceVersion forever, which is the conflict loop pinning the background controller and starving every queued LimitRange generate. gitops f637b377 strips git's copies (140 documents); Kyverno is now the sole owner. The regeneration window is safe: every tenant pod's own allow-egress excludes RFC1918, so nothing can reach a briefly-unpolicied sibling. Verify on the cluster with kubectl get updaterequests -A — the Pending count should drain to ~0; then this closes.

commented

Verified on prod: kubectl get updaterequests -A = 1 (was 246 Pending). The conflict storm is gone with Kyverno as sole owner. Closing.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs