Public: anyone can read this; nobody can change it.
workbenchthis machine
POL-0: Kyverno policy-governance model (design) — plat/mitosis#38 #49
Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.
docs/pol-0-policy-governance → main 1 commit
1 file
changed +273
−0
docs/proposals/policy-governance-kyverno.md added +273
| @@ -0,0 +1,273 @@ | ||
| 1 | # Proposal: Kyverno policy-governance model (POL-0) | |
| 2 | ||
| 3 | **Roadmap:** `POL-0` · gates the whole Kyverno track (`POL-1..POL-5`) · depends on `RBAC-1`. | |
| 4 | ||
| 5 | **The one line.** Kyverno is the **deterministic backend** for governance — `validate` / | |
| 6 | `mutate` / `generate` / `verifyImages`, with `generate ... synchronize: true` _reverting_ | |
| 7 | any tampering. The **agent that authors policy YAML from a natural-language prompt is the | |
| 8 | frontend**. Kyverno is **not** a language-to-system compiler; it is a policy engine whose | |
| 9 | inputs happen to be human- or agent-authored. This proposal decides _who may change the | |
| 10 | policy that governs the platform, and how_ — the governance-of-governance — **before** any | |
| 11 | agent is handed a policy-authoring capability. | |
| 12 | ||
| 13 | ## The problem, precisely | |
| 14 | ||
| 15 | Today **all** governance on mitosis is hand-rolled TypeScript in the MCP overlay renderer. | |
| 16 | `overlayManifests()` (`mcp/src/server.ts:2182`) emits, per app namespace: | |
| 17 | ||
| 18 | - **default-deny + minimal-allow NetworkPolicies** (`server.ts:2458`, `:2467`, `:2491`, `:2510`) — | |
| 19 | cross-tenant chatter blocked, only Traefik ingress + CoreDNS/intra-ns/443 egress. | |
| 20 | - a hardcoded **ResourceQuota** `tenant-quota` and **LimitRange** `tenant-defaults` | |
| 21 | (`server.ts:2391`, `:2414`) — the unannotated baseline tier. | |
| 22 | - the **PodSecurity `restricted`** enforce/warn labels on the namespace (`server.ts:2373`). | |
| 23 | - a **per-ns RBAC** `Role`/`RoleBinding` `agent-dev-env-read` granting the agents SA `get` | |
| 24 | on secrets in _that_ dev namespace only (`agentDevEnvReadRbac`, `server.ts:2827`). | |
| 25 | ||
| 26 | This works — it is the validated bedrock in | |
| 27 | `docs/design/multitenancy-and-agent-identity.md` (ResourceQuota rejects over-limit pods; | |
| 28 | default-deny blocks cross-tenant; PSA `restricted` enforced). But it has two structural limits: | |
| 29 | ||
| 30 | 1. **It is rendered once, then never corrected.** If a tenant (or a compromised run) deletes | |
| 31 | its `default-deny` NetworkPolicy or edits its `ResourceQuota`, nothing puts it back until | |
| 32 | the next overlay render. There is **no drift correction** — the property Kyverno | |
| 33 | `generate ... synchronize: true` exists to provide. | |
| 34 | 2. **Governance is code, not policy.** "Show me the policy, amend it, generate the software | |
| 35 | deterministically" is the vision; today the policy _is_ a 700-line TS function only a | |
| 36 | platform engineer can safely amend. There is no declarative, reviewable, diff-able policy | |
| 37 | surface — and therefore nothing an agent could ever be _safely_ pointed at. | |
| 38 | ||
| 39 | Kyverno closes both. And it is a **cheap Flux seam**: a `platform/kyverno` component plus a | |
| 40 | Flux `Kustomization` with `dependsOn: [cert-manager]` — exactly the shape of the existing | |
| 41 | `pki` / `monitoring` Kustomizations (`gitops/clusters/local/flux-kustomizations.yaml`), which | |
| 42 | already `dependsOn: [{ name: cert-manager }]`. **No `germinate` edit** is required to add the | |
| 43 | engine; germinate only needs to learn the policy _source_ (Decision 1). | |
| 44 | ||
| 45 | But adding an admission-time policy engine that can `generate`, `mutate`, and `deny` | |
| 46 | platform-wide is a strictly larger blast radius than anything today. A bad _per-app_ PR | |
| 47 | harms one tenant. A bad **ClusterPolicy** harms **every** tenant at once. So the engine is | |
| 48 | easy; the **governance of the engine** is the real design, and it is what `POL-0` must fix | |
| 49 | before `POL-1..POL-5` become safe to build. | |
| 50 | ||
| 51 | ## The model | |
| 52 | ||
| 53 | Four moving parts, each mapped to something that already exists on mitosis: | |
| 54 | ||
| 55 | | Part | What it is | Anchored to | | |
| 56 | | --------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------- | | |
| 57 | | **Source** | a git location holding `ClusterPolicy`/`Policy` YAML, reconciled by Flux | the gitops/sops/Flux stack (`flux-kustomizations.yaml`) | | |
| 58 | | **Authoring** | human writes YAML directly, or an agent drafts it from an NL prompt → **PR** | RBAC-1 repo-permission authz (`server.ts:924`, PR #5) | | |
| 59 | | **Gate** | who may _merge_ a policy change (never just _propose_) | Forgejo branch protection + CODEOWNERS + repo permission | | |
| 60 | | **Enforcement** | Kyverno admission webhook: `Audit` or `Enforce`, `failurePolicy` `Ignore`/`Fail` | `authorize()` fail-closed philosophy (`server.ts:987`) | | |
| 61 | ||
| 62 | The governing principle, borrowed verbatim from the multitenancy design: **the change is | |
| 63 | "the user, scoped."** An agent authoring policy can, at most, _open a PR_ that a | |
| 64 | higher-tier human must merge — its blast radius is a _proposal_, never an _enactment_. | |
| 65 | RBAC-1 makes that literal: the agent's run-scoped token is already excluded from | |
| 66 | `commit_files` and `merge_pr` (`AGENT_TOOLS`, `server.ts:857`; the exclusion rationale at | |
| 67 | `server.ts:844`), so an agent _cannot_ self-merge policy even if it authored it. | |
| 68 | ||
| 69 | ## Decisions (with recommendations) | |
| 70 | ||
| 71 | ### Decision 1 — Where policy lives | |
| 72 | ||
| 73 | **Options.** (a) A protected path in **`gitops`** — the same repo and PR flow apps use. | |
| 74 | (b) A **separate `policy` repo** with its own, stricter reviewer set, reconciled by the same | |
| 75 | Flux stack via its own `GitRepository` source. | |
| 76 | ||
| 77 | **Recommendation: a separate `policy` repo (target), with a CODEOWNERS-protected | |
| 78 | `gitops/clusters/*/platform/kyverno/` path as the bootstrap.** | |
| 79 | ||
| 80 | Rationale: a bad `ClusterPolicy` is _platform-wide_ blast radius, categorically unlike a | |
| 81 | per-app PR. `gitops` today mixes platform config and per-app overlays under one reviewer | |
| 82 | policy; folding policy into it means the protection boundary for "one tenant's overlay" and | |
| 83 | "the rule that governs every tenant" is the same boundary. A dedicated repo lets us set a | |
| 84 | **strictly higher** bar — a small `platform-security` CODEOWNERS group, required reviews, | |
| 85 | linear history, signed commits — without over-restricting ordinary app PRs. It also makes | |
| 86 | Decision 2 enforceable _by Forgejo_: "can this actor change policy" becomes "is this actor a | |
| 87 | collaborator on `policy`", a different (higher) grant than app-repo write. | |
| 88 | ||
| 89 | The engine still reconciles identically: add a `GitRepository` for `policy` and a | |
| 90 | `kyverno-policies` Kustomization `dependsOn: [kyverno]` (itself `dependsOn: [cert-manager]`). | |
| 91 | sops/age decryption is unchanged (policies are non-secret, so no `decryption` block needed). | |
| 92 | ||
| 93 | Tradeoff: one more repo + Flux source for germinate/seed to register at daughter birth (a | |
| 94 | one-time addition to the seam, not per-app). Until the crew is trusted to _propose_ policy at | |
| 95 | all, the bootstrap path (a CODEOWNERS-gated dir inside `gitops`) is acceptable and cheaper; | |
| 96 | promote to the separate repo the moment agents get propose access (Decision 2). | |
| 97 | ||
| 98 | ### Decision 2 — May an agent ever _write_ cluster-shape policy? | |
| 99 | ||
| 100 | **The acceptance question, per category: "can an agent's own PR change the policy that | |
| 101 | governs itself?"** | |
| 102 | ||
| 103 | **Recommendation: agents NEVER get direct write (push/merge) to policy — full stop. They may | |
| 104 | only _propose_ via a human-merged PR, and only for a bounded subset of categories. Merging | |
| 105 | policy always requires policy-repo write, a tier strictly above app-repo write (Decision 1).** | |
| 106 | ||
| 107 | Per category: | |
| 108 | ||
| 109 | | Policy category | Agent may **propose** (PR)? | Agent may **merge**? | Rationale | | |
| 110 | | ------------------------------------------------------- | --------------------------- | -------------------- | ------------------------------------------------------------------------------------------------ | | |
| 111 | | Quota / LimitRange **tier** bumps (within a fixed menu) | ✅ yes | ❌ never | Parameterized, bounded; the "tiers as data" knob the code already anticipates (`server.ts:2389`) | | |
| 112 | | `verifyImages` allow-list additions | ✅ yes | ❌ never | Additive supply-chain scope; still human-reviewed | | |
| 113 | | `generate` template _values_ (labels, annotations) | ✅ yes | ❌ never | Cosmetic/bounded; cannot widen security | | |
| 114 | | **default-deny NetworkPolicy** shape | ❌ **human-only** | ❌ never | Loosening egress = cross-tenant blast radius | | |
| 115 | | **PodSecurity baseline** | ❌ **human-only** | ❌ never | Relaxing `restricted` re-opens pod-escape vectors | | |
| 116 | | **RBAC** policies (POL-5) | ❌ **human-only** | ❌ never | Privilege-escalation surface | | |
| 117 | | **`failurePolicy` / audit↔enforce** flips | ❌ **human-only** | ❌ never | Changes the engine's own safety posture | | |
| 118 | ||
| 119 | For human-only categories the agent's ceiling is **filing an issue** requesting the change — | |
| 120 | never a policy PR. This is enforced structurally by RBAC-1: the run-scoped token is off | |
| 121 | `commit_files`/`merge_pr` (`server.ts:844-870`), so an agent cannot even _land the file_, | |
| 122 | let alone merge. Propose-access, when granted, rides a distinct deploy identity with | |
| 123 | policy-repo _read_ + PR-open, never write. | |
| 124 | ||
| 125 | ### Decision 3 — `Enforce` vs `Audit` default for a newly-merged policy | |
| 126 | ||
| 127 | **Recommendation: `Audit`-first for every newly-merged policy, with an explicit, tracked | |
| 128 | promotion to `Enforce` after a bake window and a _separate_ human flip.** | |
| 129 | ||
| 130 | A self-modifying platform must never let one merged YAML turn into an instant admission | |
| 131 | outage. `Audit` reports violations (surfaced via Kyverno PolicyReports) without blocking | |
| 132 | admission, so a bad rule is _observed_, not _enforced_, and the promotion PR is a second, | |
| 133 | deliberate decision. Note `generate` policies still `synchronize: true` and revert drift | |
| 134 | regardless of `Audit`/`Enforce` (that mode governs _admission_ of the triggering resource, | |
| 135 | not the generated resource's lifecycle) — so the drift-correction payoff is **not** delayed | |
| 136 | by audit-first. | |
| 137 | ||
| 138 | **One exception, for parity migrations only:** a policy that is a _provable superset_ of | |
| 139 | behavior already live in `overlayManifests()` — i.e. the POL-2 port of the hand-rolled TS — | |
| 140 | may ship directly in `Enforce`, because it encodes a rule the API server _already_ imposes | |
| 141 | (PSA `restricted`, the quota, default-deny). It changes _where_ the rule lives, not _whether_ | |
| 142 | it binds. Everything genuinely new starts in `Audit`. | |
| 143 | ||
| 144 | Tradeoff: audit-first delays the deterministic _deny_ payoff for new invariants by one bake | |
| 145 | window and one PR. That is the correct price for a platform that rewrites itself. | |
| 146 | ||
| 147 | ### Decision 4 — Kyverno webhook `failurePolicy`: `Ignore` (fail-open) vs `Fail` (fail-closed) | |
| 148 | ||
| 149 | **Recommendation: `Fail` (fail-closed) for the validate/mutate/verifyImages webhooks that | |
| 150 | guard the security baseline — matching `authorize()`'s "deny rather than fail open" | |
| 151 | (`server.ts:987`) — but _narrowly scoped and HA-hardened_, and shipped as a _tested_ decision.** | |
| 152 | ||
| 153 | Fail-open means the instant the Kyverno webhook is unreachable, **all** admission-time | |
| 154 | governance silently evaporates — unsigned images, quota-less pods, and privilege escalations | |
| 155 | sail through exactly when the platform is least healthy. That is unacceptable for an | |
| 156 | adversary-safe posture; it is the same trap `authorize()` explicitly refuses ("Conservative | |
| 157 | default: deny unknown tools rather than fail open", `server.ts:987`). | |
| 158 | ||
| 159 | Fail-closed's cost is real: if Kyverno is fully down, guarded namespaces cannot admit new | |
| 160 | pods → a platform-wide-ish outage. We bound it: | |
| 161 | ||
| 162 | - **Narrow `namespaceSelector`** — the webhook matches only workload namespaces | |
| 163 | (`plat.sh/tier: workload`), never `kube-system` / `flux-system` / `kyverno` itself, so a | |
| 164 | Kyverno outage can never wedge its own recovery or the reconciler that fixes it. | |
| 165 | - **HA Kyverno** (≥ 3 replicas, PDB) + a generous webhook timeout, so a single pod restart | |
| 166 | never trips `Fail`. | |
| 167 | - **A break-glass label** on a namespace to skip the webhook, usable only by a platform admin. | |
| 168 | - **A POL test that kills the webhook** and asserts: admission is _denied_ in a guarded | |
| 169 | workload namespace, while `flux-system`/`kyverno` keep reconciling. The decision is only | |
| 170 | "made" once that test is green. | |
| 171 | ||
| 172 | Tradeoff surfaced: we are explicitly choosing "possible outage when Kyverno is down" over | |
| 173 | "silent loss of all governance when Kyverno is down." Given a control plane meant to be | |
| 174 | handed to a stranger, the former is recoverable and loud; the latter is invisible and | |
| 175 | exploitable. | |
| 176 | ||
| 177 | ## Invariants: what POL-1..POL-5 will enforce | |
| 178 | ||
| 179 | These are the buildable specs `POL-0` unblocks. Each is a set of Kyverno policies in the | |
| 180 | Decision-1 source, defaulting per Decision 3, guarded per Decision 4. | |
| 181 | ||
| 182 | ### POL-1 — Baselines (`validate`) | |
| 183 | ||
| 184 | Admission-time backstops for what the namespace labels / PSA already imply, so the guarantee | |
| 185 | survives even if a label is stripped: | |
| 186 | ||
| 187 | - **PodSecurity `restricted`** as an explicit `validate` (non-root, no added capabilities, | |
| 188 | read-only rootfs, no host namespaces/hostPath) — backstops `server.ts:2373`. | |
| 189 | - **Require resource requests + limits** on every container (backstops `LimitRange`, | |
| 190 | `server.ts:2414`). | |
| 191 | - **Disallow `services.loadbalancers` / `NodePort`** (mirrors quota `:0`, `server.ts:2405`). | |
| 192 | - **Disallow mutable image tags** — require a digest or reject `:latest`. | |
| 193 | - **Require `plat.sh/*` provenance labels** (`managed`, `owner-id`, `app`, `env`, `tier`) | |
| 194 | present at `server.ts:2360`. | |
| 195 | ||
| 196 | ### POL-2 — Generate-migration of `overlayManifests()` governance (`generate`, `synchronize: true`) | |
| 197 | ||
| 198 | Port the hand-rolled TS into Kyverno `generate` ClusterPolicies keyed on the workload-namespace | |
| 199 | label, so each is **recreated on deletion and reverted on tamper** — the drift-correction the | |
| 200 | current once-rendered model lacks: | |
| 201 | ||
| 202 | - the **default-deny + allow-ingress-traefik + allow-egress** NetworkPolicies | |
| 203 | (`server.ts:2458..2540`); | |
| 204 | - the **`tenant-quota` ResourceQuota** and **`tenant-defaults` LimitRange** | |
| 205 | (`server.ts:2391`, `:2414`); | |
| 206 | - the **PSA enforce/warn labels** on the namespace (`server.ts:2373`); | |
| 207 | - the **`agent-dev-env-read` Role/RoleBinding** for dev namespaces | |
| 208 | (`agentDevEnvReadRbac`, `server.ts:2827`). | |
| 209 | ||
| 210 | Invariant: _every_ namespace labelled `plat.sh/tier: workload` always carries this set; | |
| 211 | deleting any member triggers Kyverno to regenerate it. The MCP renderer may then stop | |
| 212 | emitting these (or keep emitting as belt-and-suspenders during migration; parity-tested per | |
| 213 | Decision 3's exception). | |
| 214 | ||
| 215 | ### POL-3 — Supply chain: `verifyImages` / cosign | |
| 216 | ||
| 217 | - Only images **signed by the platform cosign key** _or_ pulled from the **in-cluster | |
| 218 | registry** admit into workload namespaces. | |
| 219 | - **Digest pinning** enforced (mutate tag→digest on admission where resolvable; reject | |
| 220 | unresolvable/unsigned external images). | |
| 221 | - Allow-list additions are the one place agents may _propose_ (Decision 2). | |
| 222 | ||
| 223 | ### POL-4 — Secrets, domains, network (`validate` + `generate`) | |
| 224 | ||
| 225 | - **Secrets** in workload namespaces must be sops-provenanced / carry the `plat.sh/managed` | |
| 226 | label; reject plaintext-injected Secrets outside the sync-env path. | |
| 227 | - **Domains:** an `Ingress`/`HTTPRoute` host must fall within the tenant's owned domain — no | |
| 228 | cross-tenant hostname squatting (the multi-tenancy hostname guarantee, enforced at | |
| 229 | admission rather than only in the renderer). | |
| 230 | - **Network:** reject any tenant-authored `NetworkPolicy` that widens **egress to internal | |
| 231 | ranges** or grants cross-tenant reach; close the **storage-_size_ DOS gap** flagged in the | |
| 232 | multitenancy design (`requests.storage` cap on `local-path`, today only a PVC _count_ cap | |
| 233 | at `server.ts:2407`). | |
| 234 | ||
| 235 | ### POL-5 — RBAC (`validate`) — the admission-layer backstop to RBAC-1 | |
| 236 | ||
| 237 | RBAC-1 (`server.ts:924`, plat/mcp PR #5) gates the _MCP API_ by repo permission. POL-5 is the | |
| 238 | _cluster admission_ backstop for anything that reaches the API server directly: | |
| 239 | ||
| 240 | - **Forbid tenant/app manifests from creating `ClusterRole` / `ClusterRoleBinding`** (no | |
| 241 | cluster-wide grants from a namespace-scoped tenant). | |
| 242 | - **Restrict `Role` grants** — no `secrets` beyond the `agent-dev-env-read` shape; no | |
| 243 | wildcard verbs/resources. | |
| 244 | - **Restrict `RoleBinding` subjects** to same-namespace ServiceAccounts (no binding a | |
| 245 | foreign-namespace or cluster SA into a tenant Role). | |
| 246 | - **Block escalation vectors** (bind to `cluster-admin`, aggregate-label abuse). | |
| 247 | ||
| 248 | ## Tradeoffs (summary) | |
| 249 | ||
| 250 | - **Separate `policy` repo** buys a strictly higher merge bar and a clean "policy-write ≠ | |
| 251 | app-write" tier, at the cost of one more Flux source to bootstrap. Bootstrap path | |
| 252 | (CODEOWNERS dir in `gitops`) de-risks the first mile. | |
| 253 | - **Propose-only agents** keep the deterministic-generation vision honest — the agent is the | |
| 254 | _frontend_, a human merge is the enactment — at the cost of a human in every policy loop. | |
| 255 | Correct until the crew has a track record. | |
| 256 | - **Audit-first** trades one bake window of delayed _deny_ for immunity to self-inflicted | |
| 257 | admission outages; `generate` drift-correction is _not_ delayed. | |
| 258 | - **Fail-closed** trades a possible-outage-when-Kyverno-is-down for no-silent-loss-of-governance, | |
| 259 | bounded by narrow scope + HA + break-glass + a kill-the-webhook test. | |
| 260 | ||
| 261 | ## Open questions | |
| 262 | ||
| 263 | - **Cosign key custody** (POL-3): where the platform signing key lives and how the CI signer | |
| 264 | gets it without widening the god-PAT blast radius — likely a sops-sealed key + a scoped | |
| 265 | signer identity, parallel to the secrets model. | |
| 266 | - **Domain ownership source of truth** (POL-4): the hostname-within-owned-domain check needs a | |
| 267 | queryable owner→domain map; is it Forgejo repo metadata, a CRD, or the MCP registry? | |
| 268 | - **PolicyReport surfacing:** how `Audit` violations reach a human (Forgejo issue? the | |
| 269 | notifications channel from the multitenancy design?) so audit-first is actionable, not silent. | |
| 270 | - **germinate/seed registration:** the exact one-time step by which a daughter is born already | |
| 271 | knowing its `policy` source + `kyverno` Kustomization (the seam addition Decision 1 implies). | |
| 272 | - **Tier menu definition** (Decision 2): the fixed set of quota tiers an agent may propose | |
| 273 | between — data, not code, and who authors that menu. |
new above
offline