Public: anyone can read this; nobody can change it.

workbenchthis machine

POL-0: Kyverno policy-governance model (design) — plat/mitosis#38 #49

Merged opened this view on origin

Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.

Merged

docs/pol-0-policy-governance → main 1 commit

  • Merged. docs/pol-0-policy-governance was merged into main.

Review

Sign in to review this change.
1 file changed +273 −0
docs/proposals/policy-governance-kyverno.md added +273
@@ -0,0 +1,273 @@
1 # Proposal: Kyverno policy-governance model (POL-0)
2
3 **Roadmap:** `POL-0` · gates the whole Kyverno track (`POL-1..POL-5`) · depends on `RBAC-1`.
4
5 **The one line.** Kyverno is the **deterministic backend** for governance — `validate` /
6 `mutate` / `generate` / `verifyImages`, with `generate ... synchronize: true` _reverting_
7 any tampering. The **agent that authors policy YAML from a natural-language prompt is the
8 frontend**. Kyverno is **not** a language-to-system compiler; it is a policy engine whose
9 inputs happen to be human- or agent-authored. This proposal decides _who may change the
10 policy that governs the platform, and how_ — the governance-of-governance — **before** any
11 agent is handed a policy-authoring capability.
12
13 ## The problem, precisely
14
15 Today **all** governance on mitosis is hand-rolled TypeScript in the MCP overlay renderer.
16 `overlayManifests()` (`mcp/src/server.ts:2182`) emits, per app namespace:
17
18 - **default-deny + minimal-allow NetworkPolicies** (`server.ts:2458`, `:2467`, `:2491`, `:2510`) —
19 cross-tenant chatter blocked, only Traefik ingress + CoreDNS/intra-ns/443 egress.
20 - a hardcoded **ResourceQuota** `tenant-quota` and **LimitRange** `tenant-defaults`
21 (`server.ts:2391`, `:2414`) — the unannotated baseline tier.
22 - the **PodSecurity `restricted`** enforce/warn labels on the namespace (`server.ts:2373`).
23 - a **per-ns RBAC** `Role`/`RoleBinding` `agent-dev-env-read` granting the agents SA `get`
24 on secrets in _that_ dev namespace only (`agentDevEnvReadRbac`, `server.ts:2827`).
25
26 This works — it is the validated bedrock in
27 `docs/design/multitenancy-and-agent-identity.md` (ResourceQuota rejects over-limit pods;
28 default-deny blocks cross-tenant; PSA `restricted` enforced). But it has two structural limits:
29
30 1. **It is rendered once, then never corrected.** If a tenant (or a compromised run) deletes
31 its `default-deny` NetworkPolicy or edits its `ResourceQuota`, nothing puts it back until
32 the next overlay render. There is **no drift correction** — the property Kyverno
33 `generate ... synchronize: true` exists to provide.
34 2. **Governance is code, not policy.** "Show me the policy, amend it, generate the software
35 deterministically" is the vision; today the policy _is_ a 700-line TS function only a
36 platform engineer can safely amend. There is no declarative, reviewable, diff-able policy
37 surface — and therefore nothing an agent could ever be _safely_ pointed at.
38
39 Kyverno closes both. And it is a **cheap Flux seam**: a `platform/kyverno` component plus a
40 Flux `Kustomization` with `dependsOn: [cert-manager]` — exactly the shape of the existing
41 `pki` / `monitoring` Kustomizations (`gitops/clusters/local/flux-kustomizations.yaml`), which
42 already `dependsOn: [{ name: cert-manager }]`. **No `germinate` edit** is required to add the
43 engine; germinate only needs to learn the policy _source_ (Decision 1).
44
45 But adding an admission-time policy engine that can `generate`, `mutate`, and `deny`
46 platform-wide is a strictly larger blast radius than anything today. A bad _per-app_ PR
47 harms one tenant. A bad **ClusterPolicy** harms **every** tenant at once. So the engine is
48 easy; the **governance of the engine** is the real design, and it is what `POL-0` must fix
49 before `POL-1..POL-5` become safe to build.
50
51 ## The model
52
53 Four moving parts, each mapped to something that already exists on mitosis:
54
55 | Part | What it is | Anchored to |
56 | --------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------- |
57 | **Source** | a git location holding `ClusterPolicy`/`Policy` YAML, reconciled by Flux | the gitops/sops/Flux stack (`flux-kustomizations.yaml`) |
58 | **Authoring** | human writes YAML directly, or an agent drafts it from an NL prompt → **PR** | RBAC-1 repo-permission authz (`server.ts:924`, PR #5) |
59 | **Gate** | who may _merge_ a policy change (never just _propose_) | Forgejo branch protection + CODEOWNERS + repo permission |
60 | **Enforcement** | Kyverno admission webhook: `Audit` or `Enforce`, `failurePolicy` `Ignore`/`Fail` | `authorize()` fail-closed philosophy (`server.ts:987`) |
61
62 The governing principle, borrowed verbatim from the multitenancy design: **the change is
63 "the user, scoped."** An agent authoring policy can, at most, _open a PR_ that a
64 higher-tier human must merge — its blast radius is a _proposal_, never an _enactment_.
65 RBAC-1 makes that literal: the agent's run-scoped token is already excluded from
66 `commit_files` and `merge_pr` (`AGENT_TOOLS`, `server.ts:857`; the exclusion rationale at
67 `server.ts:844`), so an agent _cannot_ self-merge policy even if it authored it.
68
69 ## Decisions (with recommendations)
70
71 ### Decision 1 — Where policy lives
72
73 **Options.** (a) A protected path in **`gitops`** — the same repo and PR flow apps use.
74 (b) A **separate `policy` repo** with its own, stricter reviewer set, reconciled by the same
75 Flux stack via its own `GitRepository` source.
76
77 **Recommendation: a separate `policy` repo (target), with a CODEOWNERS-protected
78 `gitops/clusters/*/platform/kyverno/` path as the bootstrap.**
79
80 Rationale: a bad `ClusterPolicy` is _platform-wide_ blast radius, categorically unlike a
81 per-app PR. `gitops` today mixes platform config and per-app overlays under one reviewer
82 policy; folding policy into it means the protection boundary for "one tenant's overlay" and
83 "the rule that governs every tenant" is the same boundary. A dedicated repo lets us set a
84 **strictly higher** bar — a small `platform-security` CODEOWNERS group, required reviews,
85 linear history, signed commits — without over-restricting ordinary app PRs. It also makes
86 Decision 2 enforceable _by Forgejo_: "can this actor change policy" becomes "is this actor a
87 collaborator on `policy`", a different (higher) grant than app-repo write.
88
89 The engine still reconciles identically: add a `GitRepository` for `policy` and a
90 `kyverno-policies` Kustomization `dependsOn: [kyverno]` (itself `dependsOn: [cert-manager]`).
91 sops/age decryption is unchanged (policies are non-secret, so no `decryption` block needed).
92
93 Tradeoff: one more repo + Flux source for germinate/seed to register at daughter birth (a
94 one-time addition to the seam, not per-app). Until the crew is trusted to _propose_ policy at
95 all, the bootstrap path (a CODEOWNERS-gated dir inside `gitops`) is acceptable and cheaper;
96 promote to the separate repo the moment agents get propose access (Decision 2).
97
98 ### Decision 2 — May an agent ever _write_ cluster-shape policy?
99
100 **The acceptance question, per category: "can an agent's own PR change the policy that
101 governs itself?"**
102
103 **Recommendation: agents NEVER get direct write (push/merge) to policy — full stop. They may
104 only _propose_ via a human-merged PR, and only for a bounded subset of categories. Merging
105 policy always requires policy-repo write, a tier strictly above app-repo write (Decision 1).**
106
107 Per category:
108
109 | Policy category | Agent may **propose** (PR)? | Agent may **merge**? | Rationale |
110 | ------------------------------------------------------- | --------------------------- | -------------------- | ------------------------------------------------------------------------------------------------ |
111 | Quota / LimitRange **tier** bumps (within a fixed menu) | ✅ yes | ❌ never | Parameterized, bounded; the "tiers as data" knob the code already anticipates (`server.ts:2389`) |
112 | `verifyImages` allow-list additions | ✅ yes | ❌ never | Additive supply-chain scope; still human-reviewed |
113 | `generate` template _values_ (labels, annotations) | ✅ yes | ❌ never | Cosmetic/bounded; cannot widen security |
114 | **default-deny NetworkPolicy** shape | ❌ **human-only** | ❌ never | Loosening egress = cross-tenant blast radius |
115 | **PodSecurity baseline** | ❌ **human-only** | ❌ never | Relaxing `restricted` re-opens pod-escape vectors |
116 | **RBAC** policies (POL-5) | ❌ **human-only** | ❌ never | Privilege-escalation surface |
117 | **`failurePolicy` / audit↔enforce** flips | ❌ **human-only** | ❌ never | Changes the engine's own safety posture |
118
119 For human-only categories the agent's ceiling is **filing an issue** requesting the change —
120 never a policy PR. This is enforced structurally by RBAC-1: the run-scoped token is off
121 `commit_files`/`merge_pr` (`server.ts:844-870`), so an agent cannot even _land the file_,
122 let alone merge. Propose-access, when granted, rides a distinct deploy identity with
123 policy-repo _read_ + PR-open, never write.
124
125 ### Decision 3 — `Enforce` vs `Audit` default for a newly-merged policy
126
127 **Recommendation: `Audit`-first for every newly-merged policy, with an explicit, tracked
128 promotion to `Enforce` after a bake window and a _separate_ human flip.**
129
130 A self-modifying platform must never let one merged YAML turn into an instant admission
131 outage. `Audit` reports violations (surfaced via Kyverno PolicyReports) without blocking
132 admission, so a bad rule is _observed_, not _enforced_, and the promotion PR is a second,
133 deliberate decision. Note `generate` policies still `synchronize: true` and revert drift
134 regardless of `Audit`/`Enforce` (that mode governs _admission_ of the triggering resource,
135 not the generated resource's lifecycle) — so the drift-correction payoff is **not** delayed
136 by audit-first.
137
138 **One exception, for parity migrations only:** a policy that is a _provable superset_ of
139 behavior already live in `overlayManifests()` — i.e. the POL-2 port of the hand-rolled TS —
140 may ship directly in `Enforce`, because it encodes a rule the API server _already_ imposes
141 (PSA `restricted`, the quota, default-deny). It changes _where_ the rule lives, not _whether_
142 it binds. Everything genuinely new starts in `Audit`.
143
144 Tradeoff: audit-first delays the deterministic _deny_ payoff for new invariants by one bake
145 window and one PR. That is the correct price for a platform that rewrites itself.
146
147 ### Decision 4 — Kyverno webhook `failurePolicy`: `Ignore` (fail-open) vs `Fail` (fail-closed)
148
149 **Recommendation: `Fail` (fail-closed) for the validate/mutate/verifyImages webhooks that
150 guard the security baseline — matching `authorize()`'s "deny rather than fail open"
151 (`server.ts:987`) — but _narrowly scoped and HA-hardened_, and shipped as a _tested_ decision.**
152
153 Fail-open means the instant the Kyverno webhook is unreachable, **all** admission-time
154 governance silently evaporates — unsigned images, quota-less pods, and privilege escalations
155 sail through exactly when the platform is least healthy. That is unacceptable for an
156 adversary-safe posture; it is the same trap `authorize()` explicitly refuses ("Conservative
157 default: deny unknown tools rather than fail open", `server.ts:987`).
158
159 Fail-closed's cost is real: if Kyverno is fully down, guarded namespaces cannot admit new
160 pods → a platform-wide-ish outage. We bound it:
161
162 - **Narrow `namespaceSelector`** — the webhook matches only workload namespaces
163 (`plat.sh/tier: workload`), never `kube-system` / `flux-system` / `kyverno` itself, so a
164 Kyverno outage can never wedge its own recovery or the reconciler that fixes it.
165 - **HA Kyverno** (≥ 3 replicas, PDB) + a generous webhook timeout, so a single pod restart
166 never trips `Fail`.
167 - **A break-glass label** on a namespace to skip the webhook, usable only by a platform admin.
168 - **A POL test that kills the webhook** and asserts: admission is _denied_ in a guarded
169 workload namespace, while `flux-system`/`kyverno` keep reconciling. The decision is only
170 "made" once that test is green.
171
172 Tradeoff surfaced: we are explicitly choosing "possible outage when Kyverno is down" over
173 "silent loss of all governance when Kyverno is down." Given a control plane meant to be
174 handed to a stranger, the former is recoverable and loud; the latter is invisible and
175 exploitable.
176
177 ## Invariants: what POL-1..POL-5 will enforce
178
179 These are the buildable specs `POL-0` unblocks. Each is a set of Kyverno policies in the
180 Decision-1 source, defaulting per Decision 3, guarded per Decision 4.
181
182 ### POL-1 — Baselines (`validate`)
183
184 Admission-time backstops for what the namespace labels / PSA already imply, so the guarantee
185 survives even if a label is stripped:
186
187 - **PodSecurity `restricted`** as an explicit `validate` (non-root, no added capabilities,
188 read-only rootfs, no host namespaces/hostPath) — backstops `server.ts:2373`.
189 - **Require resource requests + limits** on every container (backstops `LimitRange`,
190 `server.ts:2414`).
191 - **Disallow `services.loadbalancers` / `NodePort`** (mirrors quota `:0`, `server.ts:2405`).
192 - **Disallow mutable image tags** — require a digest or reject `:latest`.
193 - **Require `plat.sh/*` provenance labels** (`managed`, `owner-id`, `app`, `env`, `tier`)
194 present at `server.ts:2360`.
195
196 ### POL-2 — Generate-migration of `overlayManifests()` governance (`generate`, `synchronize: true`)
197
198 Port the hand-rolled TS into Kyverno `generate` ClusterPolicies keyed on the workload-namespace
199 label, so each is **recreated on deletion and reverted on tamper** — the drift-correction the
200 current once-rendered model lacks:
201
202 - the **default-deny + allow-ingress-traefik + allow-egress** NetworkPolicies
203 (`server.ts:2458..2540`);
204 - the **`tenant-quota` ResourceQuota** and **`tenant-defaults` LimitRange**
205 (`server.ts:2391`, `:2414`);
206 - the **PSA enforce/warn labels** on the namespace (`server.ts:2373`);
207 - the **`agent-dev-env-read` Role/RoleBinding** for dev namespaces
208 (`agentDevEnvReadRbac`, `server.ts:2827`).
209
210 Invariant: _every_ namespace labelled `plat.sh/tier: workload` always carries this set;
211 deleting any member triggers Kyverno to regenerate it. The MCP renderer may then stop
212 emitting these (or keep emitting as belt-and-suspenders during migration; parity-tested per
213 Decision 3's exception).
214
215 ### POL-3 — Supply chain: `verifyImages` / cosign
216
217 - Only images **signed by the platform cosign key** _or_ pulled from the **in-cluster
218 registry** admit into workload namespaces.
219 - **Digest pinning** enforced (mutate tag→digest on admission where resolvable; reject
220 unresolvable/unsigned external images).
221 - Allow-list additions are the one place agents may _propose_ (Decision 2).
222
223 ### POL-4 — Secrets, domains, network (`validate` + `generate`)
224
225 - **Secrets** in workload namespaces must be sops-provenanced / carry the `plat.sh/managed`
226 label; reject plaintext-injected Secrets outside the sync-env path.
227 - **Domains:** an `Ingress`/`HTTPRoute` host must fall within the tenant's owned domain — no
228 cross-tenant hostname squatting (the multi-tenancy hostname guarantee, enforced at
229 admission rather than only in the renderer).
230 - **Network:** reject any tenant-authored `NetworkPolicy` that widens **egress to internal
231 ranges** or grants cross-tenant reach; close the **storage-_size_ DOS gap** flagged in the
232 multitenancy design (`requests.storage` cap on `local-path`, today only a PVC _count_ cap
233 at `server.ts:2407`).
234
235 ### POL-5 — RBAC (`validate`) — the admission-layer backstop to RBAC-1
236
237 RBAC-1 (`server.ts:924`, plat/mcp PR #5) gates the _MCP API_ by repo permission. POL-5 is the
238 _cluster admission_ backstop for anything that reaches the API server directly:
239
240 - **Forbid tenant/app manifests from creating `ClusterRole` / `ClusterRoleBinding`** (no
241 cluster-wide grants from a namespace-scoped tenant).
242 - **Restrict `Role` grants** — no `secrets` beyond the `agent-dev-env-read` shape; no
243 wildcard verbs/resources.
244 - **Restrict `RoleBinding` subjects** to same-namespace ServiceAccounts (no binding a
245 foreign-namespace or cluster SA into a tenant Role).
246 - **Block escalation vectors** (bind to `cluster-admin`, aggregate-label abuse).
247
248 ## Tradeoffs (summary)
249
250 - **Separate `policy` repo** buys a strictly higher merge bar and a clean "policy-write ≠
251 app-write" tier, at the cost of one more Flux source to bootstrap. Bootstrap path
252 (CODEOWNERS dir in `gitops`) de-risks the first mile.
253 - **Propose-only agents** keep the deterministic-generation vision honest — the agent is the
254 _frontend_, a human merge is the enactment — at the cost of a human in every policy loop.
255 Correct until the crew has a track record.
256 - **Audit-first** trades one bake window of delayed _deny_ for immunity to self-inflicted
257 admission outages; `generate` drift-correction is _not_ delayed.
258 - **Fail-closed** trades a possible-outage-when-Kyverno-is-down for no-silent-loss-of-governance,
259 bounded by narrow scope + HA + break-glass + a kill-the-webhook test.
260
261 ## Open questions
262
263 - **Cosign key custody** (POL-3): where the platform signing key lives and how the CI signer
264 gets it without widening the god-PAT blast radius — likely a sops-sealed key + a scoped
265 signer identity, parallel to the secrets model.
266 - **Domain ownership source of truth** (POL-4): the hostname-within-owned-domain check needs a
267 queryable owner→domain map; is it Forgejo repo metadata, a CRD, or the MCP registry?
268 - **PolicyReport surfacing:** how `Audit` violations reach a human (Forgejo issue? the
269 notifications channel from the multitenancy design?) so audit-first is actionable, not silent.
270 - **germinate/seed registration:** the exact one-time step by which a daughter is born already
271 knowing its `policy` source + `kyverno` Kustomization (the seam addition Decision 1 implies).
272 - **Tier menu definition** (Decision 2): the fixed set of quota tiers an agent may propose
273 between — data, not code, and who authors that menu.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs