Public: anyone can read this; nobody can change it.
workbenchthis machine
SEC-3: secrets threat model + reviewer FAQ (docs only) #54
Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.
feat/sec-3-secrets-threat-model → main 1 commit
1 file
changed +310
−0
docs/security/secrets-threat-model.md added +310
| @@ -0,0 +1,310 @@ | ||
| 1 | # Secrets threat model & reviewer FAQ (SEC-3) | |
| 2 | ||
| 3 | A single-page account of how a Mitosis platform holds secrets: what is | |
| 4 | encrypted, what key decrypts it, who ever sees plaintext and at what moment, how | |
| 5 | the one key is backed up and rotated, and why we deliberately do **not** run | |
| 6 | Vault (yet). It is written so a reviewer who has never read the source can state | |
| 7 | those facts correctly. Every claim anchors to the code that enforces it | |
| 8 | (`file:line`), so the doc stays honest as the code moves. | |
| 9 | ||
| 10 | Scope: the platform's own secrets (under `clusters/**` in the gitops repo) and an | |
| 11 | app's self-serve secrets. Out of scope: TLS/PKI internals, and the Forgejo RBAC | |
| 12 | model (see `docs/design/forgejo-identity-to-postgres-rbac.md` and the RBAC | |
| 13 | track). | |
| 14 | ||
| 15 | --- | |
| 16 | ||
| 17 | ## Problem | |
| 18 | ||
| 19 | A self-replicating platform has an awkward secrets requirement. It must: | |
| 20 | ||
| 21 | 1. **Carry secrets in git.** Flux reconciles the platform from a gitops repo, so | |
| 22 | every secret a workload needs — DB passwords, registry creds, OAuth | |
| 23 | client secrets, the Forgejo admin password — lives, encrypted, in that repo. | |
| 24 | 2. **Re-seal itself on every germination.** A daughter platform is forged from a | |
| 25 | parent's seed. If the daughter kept the parent's secrets (or the parent's | |
| 26 | key), it would not be sovereign: the parent could read the child. So the fork | |
| 27 | path mints a fresh key and re-seals or regenerates _everything_ to it | |
| 28 | (`bin/lib.sh:207` `fork_regen_secrets`, `bin/lib.sh:338` | |
| 29 | `fork_verify_all_sealed`). | |
| 30 | 3. **Let app owners set their own secrets** without ever handing the platform a | |
| 31 | long-lived readable copy — and without a human hand-editing sealed YAML. | |
| 32 | ||
| 33 | The design answer is **SOPS + age**: one age keypair per platform, a git-native | |
| 34 | encrypted store, and a single CI seam where plaintext is briefly handled. The | |
| 35 | rest of this doc is the threat model of that answer. | |
| 36 | ||
| 37 | --- | |
| 38 | ||
| 39 | ## The sovereignty invariant: one key seals everything | |
| 40 | ||
| 41 | **Every** encrypted secret under `clusters/**` in a platform's gitops repo is | |
| 42 | sealed to exactly **one** age recipient — the platform's _sovereign_ (fork) key, | |
| 43 | minted at germination (`bin/germinate:72`) and never shared with the parent or | |
| 44 | any sibling. There is no escrow, no second recipient, no parent copy. | |
| 45 | ||
| 46 | - `.sops.yaml` at the gitops repo root lists that one recipient for both | |
| 47 | creation rules (platform + apps). The fork path rewrites it to the sovereign | |
| 48 | key (`bin/lib.sh:156` `fork_rekey_sops`; the rotation multi-recipient variant | |
| 49 | is `bin/lib.sh:737` `_rotate_set_recipients`). | |
| 50 | - SOPS encrypts only the **values** under `data`/`stringData` | |
| 51 | (`encrypted-regex ^(data|stringData)$`, see the `_fork_seal` note at | |
| 52 | `bin/lib.sh:170`). Manifest structure, key _names_, namespaces, and labels stay | |
| 53 | cleartext — so a reader of the repo learns _which_ secrets exist and _where_, | |
| 54 | but no values. | |
| 55 | - Flux decrypts in-cluster with the same key, held as the `sops-age` Secret in | |
| 56 | `flux-system` (created at `bin/germinate:296`; wired as the Kustomization | |
| 57 | `decryption.secretRef` at `bin/lib.sh:111` and `bin/lib.sh:116`). | |
| 58 | ||
| 59 | **Invariant:** the sovereign key decrypts everything; nothing else decrypts | |
| 60 | anything. It is enforced empirically, not by trusting the recipient string — | |
| 61 | `fork_verify_all_sealed` (`bin/lib.sh:338`) walks every `^sops:` file and both | |
| 62 | (a) decrypts it with the fork key _file_ and (b) requires the fork pubkey be a | |
| 63 | listed recipient; germination aborts if any file fails (`bin/germinate:118`). | |
| 64 | This is what stops the classic fork leak: a parent-sealed secret riding along in | |
| 65 | the seed (e.g. the agents dispatcher token, called out at `bin/lib.sh:251`) that | |
| 66 | the daughter's Flux could neither decrypt nor should ever possess. | |
| 67 | ||
| 68 | **Consequence — a single point of failure by design.** Lose the sovereign key | |
| 69 | and every sealed secret is unrecoverable; hold it and you hold the whole | |
| 70 | platform. That SPOF is the deliberate cost of sovereignty (no escrow party can | |
| 71 | read a daughter). The mitigations are custody, backup, and rotation — below — not | |
| 72 | a second decryptor. | |
| 73 | ||
| 74 | ### What decrypts what (reviewer table) | |
| 75 | ||
| 76 | | Ciphertext | Where | Sole decryptor | Who holds that key | | |
| 77 | | -------------------------------------------------------------- | ------------- | ------------------------------------------- | --------------------------------------------------------------------------- | | |
| 78 | | `clusters/**/*.yaml` sops values (platform + app secrets) | gitops repo | sovereign age key | operator's `fork-<domain>.age` on disk; Flux's `sops-age` Secret in-cluster | | |
| 79 | | in-cluster `sops-age` Secret | `flux-system` | (it _is_ the key) | cluster admins / anything that can read that namespace's secrets | | |
| 80 | | app self-serve secrets (`<app>-user-env`, `<app>` dev overlay) | gitops repo | sovereign age key (same key) | as above | | |
| 81 | | a Forgejo Actions secret (the source of a self-serve value) | Forgejo | write-only; only that repo's CI can read it | the app owner set it; nobody reads it back | | |
| 82 | ||
| 83 | There is exactly one key to reason about. That is the point. | |
| 84 | ||
| 85 | --- | |
| 86 | ||
| 87 | ## Who sees plaintext, and when | |
| 88 | ||
| 89 | Encrypted-at-rest is easy; the real question is the **plaintext exposure | |
| 90 | windows**. There are three, and only three. | |
| 91 | ||
| 92 | ### 1. The operator's workstation, at germination/rotation | |
| 93 | ||
| 94 | `fork_regen_secrets` (`bin/lib.sh:207`) generates fresh values (`openssl rand …`) | |
| 95 | and pipes them through `_fork_seal` (`bin/lib.sh:180`), which writes a | |
| 96 | `stringData` manifest and immediately `sops --encrypt --in-place`. Plaintext | |
| 97 | exists only in the shell's memory/pipe for the instant before encryption; it is | |
| 98 | never written to disk unencrypted and never committed. Identity- and PAT-derived | |
| 99 | fields are set later with `sops set` (`bin/lib.sh:369`, `:384`, `:413`, | |
| 100 | `:432`), which edits the encrypted file in place. The sovereign **key file** | |
| 101 | itself (`fork-<domain>.age`) does sit in cleartext on the operator's disk | |
| 102 | (`chmod 600`, `bin/germinate:72`) — that is the custody problem, addressed by | |
| 103 | backup/rotation below. | |
| 104 | ||
| 105 | ### 2. The app's own CI runner, transiently, in the sync-env job | |
| 106 | ||
| 107 | This is the crux of the self-serve design and the most common reviewer | |
| 108 | question. | |
| 109 | ||
| 110 | **Forgejo Actions secrets are write-only.** An app owner sets `STRIPE_KEY` under | |
| 111 | their repo's _Settings → Actions → Secrets_; neither the platform nor any human | |
| 112 | can ever read that value back through Forgejo. The only principal that can read | |
| 113 | it is that repo's _own_ CI job, via the `${{ toJSON(secrets) }}` expression | |
| 114 | (`_app-template/.forgejo/workflows/sync-env.yml:4` explains the model; | |
| 115 | `:85` is the read). | |
| 116 | ||
| 117 | So the platform never _pulls_ app secrets. Instead a `sync-env` workflow on the | |
| 118 | app repo — the one place that _can_ read them — hands them to the platform's | |
| 119 | sealing code (`secrets-cli.js`, run in a container at | |
| 120 | `_app-template/.forgejo/workflows/sync-env.yml:100`). That CLI | |
| 121 | (`mcp/src/secrets-cli.ts`) drops the runner/registry-injected tokens | |
| 122 | (`REGISTRY_TOKEN`, `*_TOKEN`, `PLAT_`-reserved; `mcp/src/secrets-cli.ts:54`, | |
| 123 | `:112`), then calls the _same_ `syncAppEnv` (`mcp/src/server.ts:1829`) the platform | |
| 124 | uses everywhere: clone gitops → write a `stringData` Secret → `sops` encrypt in | |
| 125 | place → commit → push (`mcp/src/server.ts:1850`, `:1870`, `:1950`). | |
| 126 | ||
| 127 | **Exposure window:** the plaintext lives _only_ inside that ephemeral CI job's | |
| 128 | process and memory for the duration of one sealing run, then is sealed to the | |
| 129 | sovereign key and pushed. It is never persisted unencrypted, never logged (the | |
| 130 | reconciliation table prints key **names** and scope, never values — | |
| 131 | `mcp/src/secrets-cli.ts:148`), and never leaves the cluster. The platform's | |
| 132 | long-lived components hold only ciphertext. | |
| 133 | ||
| 134 | > **Trust note.** "The platform sees plaintext only transiently in the sync CI | |
| 135 | > job" is precise: the CI job _is_ platform-operated code (`secrets-cli.js` | |
| 136 | > pulled from the platform's registry) running with an admin PAT | |
| 137 | > (`mcp/src/server.ts:1838` — CI acts as admin; gitops history is the audit | |
| 138 | > log). The trust boundary is "the sealing code + the cluster it runs in," not | |
| 139 | > "no platform component ever touches plaintext." A platform operator who can | |
| 140 | > run pods in the cluster can already read the `sops-age` Secret and decrypt | |
| 141 | > everything; the sync job grants no capability beyond that. | |
| 142 | ||
| 143 | ### 3. Inside the running app pod | |
| 144 | ||
| 145 | Flux decrypts the sealed `<app>-user-env` Secret in-cluster and the chart mounts | |
| 146 | it as `extraEnvFrom` (`mcp/src/server.ts:2434`). The app process sees its own | |
| 147 | secrets as env vars — necessarily, and only its own. Kubernetes namespace | |
| 148 | isolation and NetworkPolicies keep one app out of another's Secret. | |
| 149 | ||
| 150 | **Nowhere else.** Not in Forgejo (write-only), not in the platform DB, not in | |
| 151 | logs, not in the gitops repo at rest. | |
| 152 | ||
| 153 | --- | |
| 154 | ||
| 155 | ## The dev/prod (`PLAT_DEV`) scope trust boundary | |
| 156 | ||
| 157 | By default every self-serve secret is **production-only**: sealed into the app's | |
| 158 | runtime env and nothing else. An app owner can additionally mark a secret | |
| 159 | **dev-shared** by listing its name in a Forgejo Actions _Variable_ (not a secret, | |
| 160 | so it is readable) called `PLAT_DEV` | |
| 161 | (`_app-template/.forgejo/workflows/sync-env.yml:21`, `:91`). The split is done in | |
| 162 | CI: `secrets-cli` partitions the merged env into `prodEnv` / `devEnv` by | |
| 163 | upper-case name match (`mcp/src/secrets-cli.ts:141`) and a `PLAT_DEV` name with | |
| 164 | no matching secret is a **hard error** (`mcp/src/secrets-cli.ts:130`) — you cannot | |
| 165 | silently mis-scope. | |
| 166 | ||
| 167 | The two scopes land in **different sealed stores with different readers** | |
| 168 | (`mcp/src/server.ts:1892`): | |
| 169 | ||
| 170 | - **prod** → `<app>-user-env` in the app's prod namespace, consumed only by the | |
| 171 | running app pod. | |
| 172 | - **dev** → a _secret-only_ overlay at `<owner>--<name>--dev` with **no | |
| 173 | workload**, plus a narrowly-scoped RoleBinding letting the **agents** | |
| 174 | ServiceAccount read _that one namespace's_ secrets | |
| 175 | (`mcp/src/server.ts:1911`). This is what lets the coding agent exercise the app | |
| 176 | against real services while it builds, without ever seeing the prod secret. | |
| 177 | ||
| 178 | **Boundary statement:** a dev-shared secret is deliberately visible to the | |
| 179 | platform's build-time agent; a prod-only secret is not. Marking a value | |
| 180 | dev-shared _widens_ its reader set to the agent — so a client's _live_ payment | |
| 181 | key stays prod-only, while a _test_ key can be dev-shared for the agent to use. | |
| 182 | Emptying `PLAT_DEV` tears the dev overlay down so no stale dev secret lingers | |
| 183 | (`mcp/src/server.ts:1924`, `mcp/src/secrets-cli.ts` re-runs on every push). Both | |
| 184 | stores are still sealed to the one sovereign key; the boundary is _who is | |
| 185 | authorized to read the decrypted value in-cluster_, enforced by namespace RBAC, | |
| 186 | not a second key. | |
| 187 | ||
| 188 | --- | |
| 189 | ||
| 190 | ## Rotation & backup posture (post-SEC-1) | |
| 191 | ||
| 192 | The one-key SPOF is managed by three operations, all in `bin/lib.sh` and | |
| 193 | documented operationally in `docs/ops/age-key-rotation.md`. | |
| 194 | ||
| 195 | - **Custody gate at germination.** `germinate` refuses to build until the | |
| 196 | operator acknowledges custody of the sovereign key (`fork_backup_key`, | |
| 197 | `bin/lib.sh:681`, called at `bin/germinate:81`). On a TTY it prompts for a typed | |
| 198 | `YES`; in automation `FORK_KEY_ACK=1` acknowledges non-interactively. It never | |
| 199 | silently skips the step — no key, no platform. | |
| 200 | - **Backup.** `FORK_KEY_BACKUP=<path>` writes a copy; `FORK_KEY_BACKUP_RECIPIENT=age1…` | |
| 201 | age-encrypts that backup to a **separate offline** identity the operator holds | |
| 202 | (`bin/lib.sh:696`), so the backup is ciphertext at rest and stealing it is | |
| 203 | useless without the offline key. The plaintext-copy fallback warns loudly | |
| 204 | (`bin/lib.sh:705`). Sovereignty is unchanged: the on-disk key remains the sole | |
| 205 | decryptor. | |
| 206 | - **Rotation.** `rotate_age_key` (`bin/lib.sh:809`) mints a new keypair and | |
| 207 | re-seals every `clusters/**` file to it **preserving values** (via | |
| 208 | `sops updatekeys`, `bin/lib.sh:749` — not the regenerate-fresh path), rotates the | |
| 209 | data key so a party who cached the _old_ data key can read nothing | |
| 210 | (`bin/lib.sh:764` `_rotate_datakey`), swaps the in-cluster `sops-age` Secret, and | |
| 211 | gates on `fork_verify_all_sealed`. The ordering is lock-out-safe: the new | |
| 212 | recipient is **added** (both keys valid) and pushed first, the in-cluster key | |
| 213 | is swapped, then the old recipient is removed and the data key rotated | |
| 214 | (`bin/lib.sh:832`, `:848`) — Flux can decrypt at every step. A | |
| 215 | `ROTATE_LOCAL_ONLY=1` dry-run proves value-preservation offline. | |
| 216 | ||
| 217 | **Threats this covers.** Lost workstation → recover from the offline-encrypted | |
| 218 | backup. Suspected key compromise → rotate (data-key rotation defeats a cached | |
| 219 | data key) _and_ rotate every external credential the platform holds, since | |
| 220 | rotation re-seals _containers_; a leaked _value_ must be replaced, and old git | |
| 221 | history stays readable by whoever had the old key (`docs/ops/age-key-rotation.md` | |
| 222 | § "Suspected compromise"). **Threats it does not cover.** An attacker who reads | |
| 223 | the live `sops-age` Secret (i.e. already has cluster-admin) has the key; rotation | |
| 224 | is remediation, not prevention, against an already-compromised cluster. | |
| 225 | ||
| 226 | --- | |
| 227 | ||
| 228 | ## The Vault non-decision | |
| 229 | ||
| 230 | **We deliberately do not run HashiCorp Vault (or any external secret manager), | |
| 231 | and this is a considered choice, not an omission.** | |
| 232 | ||
| 233 | The tempting pitch is "put secrets in Vault instead of sealed git." It does not | |
| 234 | fit the invariant: | |
| 235 | ||
| 236 | - **It does not remove the SPOF; it moves and multiplies it.** Vault's own | |
| 237 | unseal keys / root token are the new sovereign secret — now you have _both_ an | |
| 238 | age key (Flux/SOPS don't disappear overnight) _and_ Vault's unseal material to | |
| 239 | guard. A single-operator platform gains a second thing to lose. | |
| 240 | - **It adds stateful, always-on risk.** Vault is a HA stateful service with its | |
| 241 | own storage backend, unseal lifecycle, upgrade/backup story, and audit | |
| 242 | surface. Sealed git is _stateless at rest_: the store is the repo, the only | |
| 243 | live secret is one in-cluster key, and the whole thing survives a cluster | |
| 244 | rebuild by re-applying gitops. Adding Vault trades a one-file custody problem | |
| 245 | for an operational service the operator must keep alive to boot an app. | |
| 246 | - **It breaks self-replication cleanly.** Germination forks a platform in ~18 | |
| 247 | minutes with `age-keygen` + `sops` (`bin/germinate:72`). A per-daughter Vault | |
| 248 | (bootstrap, unseal, seed, and prove sovereign) is a large regression in the | |
| 249 | thing that makes Mitosis Mitosis. | |
| 250 | - **Sovereignty is already met.** The requirements Vault would satisfy — | |
| 251 | encrypted at rest, per-platform isolation, write-only owner secrets, auditable | |
| 252 | change history (gitops commits, attributed via SEC-2 trailers, | |
| 253 | `_app-template/.forgejo/workflows/sync-env.yml:92`) — are already met by the | |
| 254 | sealed-git model at far lower operational cost. | |
| 255 | ||
| 256 | **Triggers to revisit (build Vault when _any_ holds):** | |
| 257 | ||
| 258 | 1. **Multi-operator quorum.** More than one operator must be prevented from | |
| 259 | unilaterally decrypting — i.e. we need Shamir-split unseal / m-of-n. A single | |
| 260 | age key cannot express quorum; that is the point where Vault (or `age` with a | |
| 261 | plugin + real ceremony) earns its keep. | |
| 262 | 2. **Secret TTLs / automatic expiry.** A requirement that secrets _expire_ and | |
| 263 | force rotation on a clock, rather than rotate on operator action. | |
| 264 | 3. **Dynamic/leased credentials.** Per-request DB users, short-lived cloud creds, | |
| 265 | PKI issued on demand — capabilities sealed-git structurally cannot provide | |
| 266 | (it stores static values). | |
| 267 | ||
| 268 | Until one of those is a real client requirement, Vault is **all cost, no | |
| 269 | marginal sovereignty**. Defer. | |
| 270 | ||
| 271 | --- | |
| 272 | ||
| 273 | ## Tradeoffs | |
| 274 | ||
| 275 | - **One key = simplest possible trust story, at the cost of a hard SPOF.** We buy | |
| 276 | reviewability and true per-platform sovereignty; we pay with "lose the key, | |
| 277 | lose the platform," mitigated by mandatory-acknowledged backup + rotation. | |
| 278 | - **Write-only Forgejo secrets = no readable central store, at the cost of a | |
| 279 | transient CI plaintext window.** The platform can never leak what it cannot | |
| 280 | read; the price is that one ephemeral, admin-run CI job must handle plaintext | |
| 281 | to seal it. | |
| 282 | - **Sealed git = stateless, auditable, replication-friendly, at the cost of no | |
| 283 | TTLs/leases/quorum.** Those are exactly the Vault triggers above. | |
| 284 | - **`stringData` before sealing = human-readable pre-encryption + `sops set` | |
| 285 | edits, at the cost that key _names_/structure are cleartext in the repo** (an | |
| 286 | attacker learns the shape of the secret inventory, never the values). | |
| 287 | ||
| 288 | --- | |
| 289 | ||
| 290 | ## Open questions | |
| 291 | ||
| 292 | - **In-cluster `sops-age` blast radius.** Anything that can read the | |
| 293 | `flux-system/sops-age` Secret holds the sovereign key (`bin/germinate:296`). | |
| 294 | Should the key be sealed to a KMS/enclave so even cluster-admin cannot exfil it | |
| 295 | cleartext? (Would soften the "cluster-admin ⇒ game over" note above; interacts | |
| 296 | with the POL/RBAC tracks.) | |
| 297 | - **Old-history readability after rotation.** Rotation does not rewrite git | |
| 298 | history, so a party with the _old_ key + old commits can still read old | |
| 299 | ciphertext (`docs/ops/age-key-rotation.md` § compromise). Is a | |
| 300 | history-rewrite / fresh-repo cutover worth automating for the compromise path? | |
| 301 | - **Dev-scope reader is the agents SA, cluster-wide identity.** `PLAT_DEV` | |
| 302 | grants the agents ServiceAccount read on the dev namespace | |
| 303 | (`mcp/src/server.ts:1911`). As RBAC-2 forwardAuth and per-team identity land, is | |
| 304 | namespace-scoped RoleBinding still the right granularity, or should dev secrets | |
| 305 | bind to a per-app agent identity? | |
| 306 | - **CI job compromise.** The sync-env job runs platform code with an admin PAT | |
| 307 | (`mcp/src/server.ts:1838`). A malicious app repo cannot read _other_ apps' | |
| 308 | secrets (Forgejo write-only + namespace isolation), but should the sealing job | |
| 309 | run with a _scoped_ token rather than admin, to shrink its authority to | |
| 310 | "seal into this one app's overlay"? |
new above
offline