Public: anyone can read this; nobody can change it.

workbenchthis machine

SEC-3: secrets threat model + reviewer FAQ (docs only) #54

Merged opened this view on origin

Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.

Merged

feat/sec-3-secrets-threat-model → main 1 commit

  • Merged. feat/sec-3-secrets-threat-model was merged into main.

Review

Sign in to review this change.
1 file changed +310 −0
docs/security/secrets-threat-model.md added +310
@@ -0,0 +1,310 @@
1 # Secrets threat model & reviewer FAQ (SEC-3)
2
3 A single-page account of how a Mitosis platform holds secrets: what is
4 encrypted, what key decrypts it, who ever sees plaintext and at what moment, how
5 the one key is backed up and rotated, and why we deliberately do **not** run
6 Vault (yet). It is written so a reviewer who has never read the source can state
7 those facts correctly. Every claim anchors to the code that enforces it
8 (`file:line`), so the doc stays honest as the code moves.
9
10 Scope: the platform's own secrets (under `clusters/**` in the gitops repo) and an
11 app's self-serve secrets. Out of scope: TLS/PKI internals, and the Forgejo RBAC
12 model (see `docs/design/forgejo-identity-to-postgres-rbac.md` and the RBAC
13 track).
14
15 ---
16
17 ## Problem
18
19 A self-replicating platform has an awkward secrets requirement. It must:
20
21 1. **Carry secrets in git.** Flux reconciles the platform from a gitops repo, so
22 every secret a workload needs — DB passwords, registry creds, OAuth
23 client secrets, the Forgejo admin password — lives, encrypted, in that repo.
24 2. **Re-seal itself on every germination.** A daughter platform is forged from a
25 parent's seed. If the daughter kept the parent's secrets (or the parent's
26 key), it would not be sovereign: the parent could read the child. So the fork
27 path mints a fresh key and re-seals or regenerates _everything_ to it
28 (`bin/lib.sh:207` `fork_regen_secrets`, `bin/lib.sh:338`
29 `fork_verify_all_sealed`).
30 3. **Let app owners set their own secrets** without ever handing the platform a
31 long-lived readable copy — and without a human hand-editing sealed YAML.
32
33 The design answer is **SOPS + age**: one age keypair per platform, a git-native
34 encrypted store, and a single CI seam where plaintext is briefly handled. The
35 rest of this doc is the threat model of that answer.
36
37 ---
38
39 ## The sovereignty invariant: one key seals everything
40
41 **Every** encrypted secret under `clusters/**` in a platform's gitops repo is
42 sealed to exactly **one** age recipient — the platform's _sovereign_ (fork) key,
43 minted at germination (`bin/germinate:72`) and never shared with the parent or
44 any sibling. There is no escrow, no second recipient, no parent copy.
45
46 - `.sops.yaml` at the gitops repo root lists that one recipient for both
47 creation rules (platform + apps). The fork path rewrites it to the sovereign
48 key (`bin/lib.sh:156` `fork_rekey_sops`; the rotation multi-recipient variant
49 is `bin/lib.sh:737` `_rotate_set_recipients`).
50 - SOPS encrypts only the **values** under `data`/`stringData`
51 (`encrypted-regex ^(data|stringData)$`, see the `_fork_seal` note at
52 `bin/lib.sh:170`). Manifest structure, key _names_, namespaces, and labels stay
53 cleartext — so a reader of the repo learns _which_ secrets exist and _where_,
54 but no values.
55 - Flux decrypts in-cluster with the same key, held as the `sops-age` Secret in
56 `flux-system` (created at `bin/germinate:296`; wired as the Kustomization
57 `decryption.secretRef` at `bin/lib.sh:111` and `bin/lib.sh:116`).
58
59 **Invariant:** the sovereign key decrypts everything; nothing else decrypts
60 anything. It is enforced empirically, not by trusting the recipient string —
61 `fork_verify_all_sealed` (`bin/lib.sh:338`) walks every `^sops:` file and both
62 (a) decrypts it with the fork key _file_ and (b) requires the fork pubkey be a
63 listed recipient; germination aborts if any file fails (`bin/germinate:118`).
64 This is what stops the classic fork leak: a parent-sealed secret riding along in
65 the seed (e.g. the agents dispatcher token, called out at `bin/lib.sh:251`) that
66 the daughter's Flux could neither decrypt nor should ever possess.
67
68 **Consequence — a single point of failure by design.** Lose the sovereign key
69 and every sealed secret is unrecoverable; hold it and you hold the whole
70 platform. That SPOF is the deliberate cost of sovereignty (no escrow party can
71 read a daughter). The mitigations are custody, backup, and rotation — below — not
72 a second decryptor.
73
74 ### What decrypts what (reviewer table)
75
76 | Ciphertext | Where | Sole decryptor | Who holds that key |
77 | -------------------------------------------------------------- | ------------- | ------------------------------------------- | --------------------------------------------------------------------------- |
78 | `clusters/**/*.yaml` sops values (platform + app secrets) | gitops repo | sovereign age key | operator's `fork-<domain>.age` on disk; Flux's `sops-age` Secret in-cluster |
79 | in-cluster `sops-age` Secret | `flux-system` | (it _is_ the key) | cluster admins / anything that can read that namespace's secrets |
80 | app self-serve secrets (`<app>-user-env`, `<app>` dev overlay) | gitops repo | sovereign age key (same key) | as above |
81 | a Forgejo Actions secret (the source of a self-serve value) | Forgejo | write-only; only that repo's CI can read it | the app owner set it; nobody reads it back |
82
83 There is exactly one key to reason about. That is the point.
84
85 ---
86
87 ## Who sees plaintext, and when
88
89 Encrypted-at-rest is easy; the real question is the **plaintext exposure
90 windows**. There are three, and only three.
91
92 ### 1. The operator's workstation, at germination/rotation
93
94 `fork_regen_secrets` (`bin/lib.sh:207`) generates fresh values (`openssl rand …`)
95 and pipes them through `_fork_seal` (`bin/lib.sh:180`), which writes a
96 `stringData` manifest and immediately `sops --encrypt --in-place`. Plaintext
97 exists only in the shell's memory/pipe for the instant before encryption; it is
98 never written to disk unencrypted and never committed. Identity- and PAT-derived
99 fields are set later with `sops set` (`bin/lib.sh:369`, `:384`, `:413`,
100 `:432`), which edits the encrypted file in place. The sovereign **key file**
101 itself (`fork-<domain>.age`) does sit in cleartext on the operator's disk
102 (`chmod 600`, `bin/germinate:72`) — that is the custody problem, addressed by
103 backup/rotation below.
104
105 ### 2. The app's own CI runner, transiently, in the sync-env job
106
107 This is the crux of the self-serve design and the most common reviewer
108 question.
109
110 **Forgejo Actions secrets are write-only.** An app owner sets `STRIPE_KEY` under
111 their repo's _Settings → Actions → Secrets_; neither the platform nor any human
112 can ever read that value back through Forgejo. The only principal that can read
113 it is that repo's _own_ CI job, via the `${{ toJSON(secrets) }}` expression
114 (`_app-template/.forgejo/workflows/sync-env.yml:4` explains the model;
115 `:85` is the read).
116
117 So the platform never _pulls_ app secrets. Instead a `sync-env` workflow on the
118 app repo — the one place that _can_ read them — hands them to the platform's
119 sealing code (`secrets-cli.js`, run in a container at
120 `_app-template/.forgejo/workflows/sync-env.yml:100`). That CLI
121 (`mcp/src/secrets-cli.ts`) drops the runner/registry-injected tokens
122 (`REGISTRY_TOKEN`, `*_TOKEN`, `PLAT_`-reserved; `mcp/src/secrets-cli.ts:54`,
123 `:112`), then calls the _same_ `syncAppEnv` (`mcp/src/server.ts:1829`) the platform
124 uses everywhere: clone gitops → write a `stringData` Secret → `sops` encrypt in
125 place → commit → push (`mcp/src/server.ts:1850`, `:1870`, `:1950`).
126
127 **Exposure window:** the plaintext lives _only_ inside that ephemeral CI job's
128 process and memory for the duration of one sealing run, then is sealed to the
129 sovereign key and pushed. It is never persisted unencrypted, never logged (the
130 reconciliation table prints key **names** and scope, never values —
131 `mcp/src/secrets-cli.ts:148`), and never leaves the cluster. The platform's
132 long-lived components hold only ciphertext.
133
134 > **Trust note.** "The platform sees plaintext only transiently in the sync CI
135 > job" is precise: the CI job _is_ platform-operated code (`secrets-cli.js`
136 > pulled from the platform's registry) running with an admin PAT
137 > (`mcp/src/server.ts:1838` — CI acts as admin; gitops history is the audit
138 > log). The trust boundary is "the sealing code + the cluster it runs in," not
139 > "no platform component ever touches plaintext." A platform operator who can
140 > run pods in the cluster can already read the `sops-age` Secret and decrypt
141 > everything; the sync job grants no capability beyond that.
142
143 ### 3. Inside the running app pod
144
145 Flux decrypts the sealed `<app>-user-env` Secret in-cluster and the chart mounts
146 it as `extraEnvFrom` (`mcp/src/server.ts:2434`). The app process sees its own
147 secrets as env vars — necessarily, and only its own. Kubernetes namespace
148 isolation and NetworkPolicies keep one app out of another's Secret.
149
150 **Nowhere else.** Not in Forgejo (write-only), not in the platform DB, not in
151 logs, not in the gitops repo at rest.
152
153 ---
154
155 ## The dev/prod (`PLAT_DEV`) scope trust boundary
156
157 By default every self-serve secret is **production-only**: sealed into the app's
158 runtime env and nothing else. An app owner can additionally mark a secret
159 **dev-shared** by listing its name in a Forgejo Actions _Variable_ (not a secret,
160 so it is readable) called `PLAT_DEV`
161 (`_app-template/.forgejo/workflows/sync-env.yml:21`, `:91`). The split is done in
162 CI: `secrets-cli` partitions the merged env into `prodEnv` / `devEnv` by
163 upper-case name match (`mcp/src/secrets-cli.ts:141`) and a `PLAT_DEV` name with
164 no matching secret is a **hard error** (`mcp/src/secrets-cli.ts:130`) — you cannot
165 silently mis-scope.
166
167 The two scopes land in **different sealed stores with different readers**
168 (`mcp/src/server.ts:1892`):
169
170 - **prod** → `<app>-user-env` in the app's prod namespace, consumed only by the
171 running app pod.
172 - **dev** → a _secret-only_ overlay at `<owner>--<name>--dev` with **no
173 workload**, plus a narrowly-scoped RoleBinding letting the **agents**
174 ServiceAccount read _that one namespace's_ secrets
175 (`mcp/src/server.ts:1911`). This is what lets the coding agent exercise the app
176 against real services while it builds, without ever seeing the prod secret.
177
178 **Boundary statement:** a dev-shared secret is deliberately visible to the
179 platform's build-time agent; a prod-only secret is not. Marking a value
180 dev-shared _widens_ its reader set to the agent — so a client's _live_ payment
181 key stays prod-only, while a _test_ key can be dev-shared for the agent to use.
182 Emptying `PLAT_DEV` tears the dev overlay down so no stale dev secret lingers
183 (`mcp/src/server.ts:1924`, `mcp/src/secrets-cli.ts` re-runs on every push). Both
184 stores are still sealed to the one sovereign key; the boundary is _who is
185 authorized to read the decrypted value in-cluster_, enforced by namespace RBAC,
186 not a second key.
187
188 ---
189
190 ## Rotation & backup posture (post-SEC-1)
191
192 The one-key SPOF is managed by three operations, all in `bin/lib.sh` and
193 documented operationally in `docs/ops/age-key-rotation.md`.
194
195 - **Custody gate at germination.** `germinate` refuses to build until the
196 operator acknowledges custody of the sovereign key (`fork_backup_key`,
197 `bin/lib.sh:681`, called at `bin/germinate:81`). On a TTY it prompts for a typed
198 `YES`; in automation `FORK_KEY_ACK=1` acknowledges non-interactively. It never
199 silently skips the step — no key, no platform.
200 - **Backup.** `FORK_KEY_BACKUP=<path>` writes a copy; `FORK_KEY_BACKUP_RECIPIENT=age1…`
201 age-encrypts that backup to a **separate offline** identity the operator holds
202 (`bin/lib.sh:696`), so the backup is ciphertext at rest and stealing it is
203 useless without the offline key. The plaintext-copy fallback warns loudly
204 (`bin/lib.sh:705`). Sovereignty is unchanged: the on-disk key remains the sole
205 decryptor.
206 - **Rotation.** `rotate_age_key` (`bin/lib.sh:809`) mints a new keypair and
207 re-seals every `clusters/**` file to it **preserving values** (via
208 `sops updatekeys`, `bin/lib.sh:749` — not the regenerate-fresh path), rotates the
209 data key so a party who cached the _old_ data key can read nothing
210 (`bin/lib.sh:764` `_rotate_datakey`), swaps the in-cluster `sops-age` Secret, and
211 gates on `fork_verify_all_sealed`. The ordering is lock-out-safe: the new
212 recipient is **added** (both keys valid) and pushed first, the in-cluster key
213 is swapped, then the old recipient is removed and the data key rotated
214 (`bin/lib.sh:832`, `:848`) — Flux can decrypt at every step. A
215 `ROTATE_LOCAL_ONLY=1` dry-run proves value-preservation offline.
216
217 **Threats this covers.** Lost workstation → recover from the offline-encrypted
218 backup. Suspected key compromise → rotate (data-key rotation defeats a cached
219 data key) _and_ rotate every external credential the platform holds, since
220 rotation re-seals _containers_; a leaked _value_ must be replaced, and old git
221 history stays readable by whoever had the old key (`docs/ops/age-key-rotation.md`
222 § "Suspected compromise"). **Threats it does not cover.** An attacker who reads
223 the live `sops-age` Secret (i.e. already has cluster-admin) has the key; rotation
224 is remediation, not prevention, against an already-compromised cluster.
225
226 ---
227
228 ## The Vault non-decision
229
230 **We deliberately do not run HashiCorp Vault (or any external secret manager),
231 and this is a considered choice, not an omission.**
232
233 The tempting pitch is "put secrets in Vault instead of sealed git." It does not
234 fit the invariant:
235
236 - **It does not remove the SPOF; it moves and multiplies it.** Vault's own
237 unseal keys / root token are the new sovereign secret — now you have _both_ an
238 age key (Flux/SOPS don't disappear overnight) _and_ Vault's unseal material to
239 guard. A single-operator platform gains a second thing to lose.
240 - **It adds stateful, always-on risk.** Vault is a HA stateful service with its
241 own storage backend, unseal lifecycle, upgrade/backup story, and audit
242 surface. Sealed git is _stateless at rest_: the store is the repo, the only
243 live secret is one in-cluster key, and the whole thing survives a cluster
244 rebuild by re-applying gitops. Adding Vault trades a one-file custody problem
245 for an operational service the operator must keep alive to boot an app.
246 - **It breaks self-replication cleanly.** Germination forks a platform in ~18
247 minutes with `age-keygen` + `sops` (`bin/germinate:72`). A per-daughter Vault
248 (bootstrap, unseal, seed, and prove sovereign) is a large regression in the
249 thing that makes Mitosis Mitosis.
250 - **Sovereignty is already met.** The requirements Vault would satisfy —
251 encrypted at rest, per-platform isolation, write-only owner secrets, auditable
252 change history (gitops commits, attributed via SEC-2 trailers,
253 `_app-template/.forgejo/workflows/sync-env.yml:92`) — are already met by the
254 sealed-git model at far lower operational cost.
255
256 **Triggers to revisit (build Vault when _any_ holds):**
257
258 1. **Multi-operator quorum.** More than one operator must be prevented from
259 unilaterally decrypting — i.e. we need Shamir-split unseal / m-of-n. A single
260 age key cannot express quorum; that is the point where Vault (or `age` with a
261 plugin + real ceremony) earns its keep.
262 2. **Secret TTLs / automatic expiry.** A requirement that secrets _expire_ and
263 force rotation on a clock, rather than rotate on operator action.
264 3. **Dynamic/leased credentials.** Per-request DB users, short-lived cloud creds,
265 PKI issued on demand — capabilities sealed-git structurally cannot provide
266 (it stores static values).
267
268 Until one of those is a real client requirement, Vault is **all cost, no
269 marginal sovereignty**. Defer.
270
271 ---
272
273 ## Tradeoffs
274
275 - **One key = simplest possible trust story, at the cost of a hard SPOF.** We buy
276 reviewability and true per-platform sovereignty; we pay with "lose the key,
277 lose the platform," mitigated by mandatory-acknowledged backup + rotation.
278 - **Write-only Forgejo secrets = no readable central store, at the cost of a
279 transient CI plaintext window.** The platform can never leak what it cannot
280 read; the price is that one ephemeral, admin-run CI job must handle plaintext
281 to seal it.
282 - **Sealed git = stateless, auditable, replication-friendly, at the cost of no
283 TTLs/leases/quorum.** Those are exactly the Vault triggers above.
284 - **`stringData` before sealing = human-readable pre-encryption + `sops set`
285 edits, at the cost that key _names_/structure are cleartext in the repo** (an
286 attacker learns the shape of the secret inventory, never the values).
287
288 ---
289
290 ## Open questions
291
292 - **In-cluster `sops-age` blast radius.** Anything that can read the
293 `flux-system/sops-age` Secret holds the sovereign key (`bin/germinate:296`).
294 Should the key be sealed to a KMS/enclave so even cluster-admin cannot exfil it
295 cleartext? (Would soften the "cluster-admin ⇒ game over" note above; interacts
296 with the POL/RBAC tracks.)
297 - **Old-history readability after rotation.** Rotation does not rewrite git
298 history, so a party with the _old_ key + old commits can still read old
299 ciphertext (`docs/ops/age-key-rotation.md` § compromise). Is a
300 history-rewrite / fresh-repo cutover worth automating for the compromise path?
301 - **Dev-scope reader is the agents SA, cluster-wide identity.** `PLAT_DEV`
302 grants the agents ServiceAccount read on the dev namespace
303 (`mcp/src/server.ts:1911`). As RBAC-2 forwardAuth and per-team identity land, is
304 namespace-scoped RoleBinding still the right granularity, or should dev secrets
305 bind to a per-app agent identity?
306 - **CI job compromise.** The sync-env job runs platform code with an admin PAT
307 (`mcp/src/server.ts:1838`). A malicious app repo cannot read _other_ apps'
308 secrets (Forgejo write-only + namespace isolation), but should the sealing job
309 run with a _scoped_ token rather than admin, to shrink its authority to
310 "seal into this one app's overlay"?

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs