Public: anyone can read this; the crew can change it.
Each row of docs/THREATS.md names the test or gate that enforces it, and a test fails on a row that names none #11874
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/11865-each-row-of-docs-threats-md-names-the-te → main 1 commit fork
6 files
changed +222
−116
docs/THREATS.md +51 −47
| @@ -1,10 +1,14 @@ | ||
| 1 | 1 | # Threats |
| 2 | 2 | |
| 3 | 3 | Who would attack this forge, what each wants, and what stands in the way. |
| 4 | Every protection is one bullet ending in a `Test:` line that names the test | |
| 5 | file proving it, or says `none yet` plainly. `test/threats-doc.test.ts` checks | |
| 6 | that every cited file exists and prints how many gaps remain. A protection | |
| 7 | with no test is a gap, not a secret: the count is the to-do list. | |
| 4 | Every protection is one bullet ending in one `Enforced by:` line that names | |
| 5 | the test file (`test/….test.ts`) or the gate (`scripts/….ts`) that enforces | |
| 6 | it, or says `none yet` plainly, with `(#N)` when an open issue owns the gap. | |
| 7 | Each row of the internal-calls table names its enforcer in its last cell the | |
| 8 | same way. `test/threats-enforced.test.ts` fails on a row with no such line or | |
| 9 | a named path that does not exist, and prints how many rows say `none yet`. A | |
| 10 | protection nothing enforces is a gap, not a secret: the count is the to-do | |
| 11 | list, and only editing a row lowers it. | |
| 8 | 12 | |
| 9 | 13 | ## A snooper on the network |
| 10 | 14 | |
| @@ -13,16 +17,16 @@ talks to whom. | ||
| 13 | 17 | |
| 14 | 18 | - Sessions are HttpOnly cookies, `Secure` when the request came over https, so |
| 15 | 19 | script cannot read one and a plain-http hop does not carry it. |
| 16 | Test: `test/own-session.test.ts` | |
| 20 | Enforced by: `test/own-session.test.ts` | |
| 17 | 21 | - Tokens are shown once; the database keeps only their sha256, so a copy of the |
| 18 | 22 | index does not replay as a login. |
| 19 | Test: `test/tokens.test.ts` | |
| 23 | Enforced by: `test/tokens.test.ts` | |
| 20 | 24 | - TLS is terminated by the edge in front of the forge, and certificates are |
| 21 | 25 | issued only for names the forge vouches for (`on_demand_tls` ask). |
| 22 | Test: none yet | |
| 26 | Enforced by: none yet | |
| 23 | 27 | - Between the edge and the forge process, and between forge and cell, traffic is |
| 24 | 28 | plain HTTP on one machine or a pod network. Nothing encrypts it. |
| 25 | Test: none yet | |
| 29 | Enforced by: none yet | |
| 26 | 30 | |
| 27 | 31 | ## A stolen token |
| 28 | 32 | |
| @@ -30,25 +34,25 @@ Wants: to act as the person or agent it was issued to, for as long as it works. | ||
| 30 | 34 | |
| 31 | 35 | - Tokens carry a scope and an optional expiry; a narrower token cannot do what a |
| 32 | 36 | wider one can. |
| 33 | Test: `test/tokens.test.ts` | |
| 37 | Enforced by: `test/tokens.test.ts` | |
| 34 | 38 | - A crew run's secrets are named in the brief, opened per run, redacted from the |
| 35 | 39 | log, and released when the run ends. |
| 36 | Test: `test/run-secrets.test.ts` | |
| 40 | Enforced by: `test/run-secrets.test.ts` | |
| 37 | 41 | - A passport (`fp1.…`) is bound to one audience and scope set and dies in |
| 38 | 42 | minutes, so a copy lifted from a run is useful to one door briefly. |
| 39 | Test: `test/passport.test.ts` | |
| 43 | Enforced by: `test/passport.test.ts` | |
| 40 | 44 | - Sign-in attempts are throttled per client, with a whole-forge cap beside it. |
| 41 | Test: `test/signin-throttle.test.ts` | |
| 45 | Enforced by: `test/signin-throttle.test.ts` | |
| 42 | 46 | - Stored provider tokens (origins) are sealed under a key in `secret.key`, never |
| 43 | 47 | returned by any route. |
| 44 | Test: `test/origins-security-auth.test.ts` | |
| 48 | Enforced by: `test/origins-security-auth.test.ts` | |
| 45 | 49 | - The SSH host key and the database files are created 0600; a host key group- |
| 46 | 50 | or world-readable is refused, and a database file found too open is a |
| 47 | 51 | degraded finding naming the chmod, never a silent change and never a stop. |
| 48 | Test: `test/file-modes.test.ts` | |
| 52 | Enforced by: `test/file-modes.test.ts` | |
| 49 | 53 | - Revoking every token of one account at once, and telling its owner a token was |
| 50 | 54 | used from somewhere new. |
| 51 | Test: none yet | |
| 55 | Enforced by: none yet | |
| 52 | 56 | |
| 53 | 57 | ## A compromised vendor |
| 54 | 58 | |
| @@ -57,24 +61,24 @@ trusts: a package, a model provider, a hosting provider, an origin it syncs from | ||
| 57 | 61 | |
| 58 | 62 | - The forge has no runtime dependencies, so there is no package to poison in the |
| 59 | 63 | forge itself. |
| 60 | Test: `test/deps.test.ts` | |
| 64 | Enforced by: `test/deps.test.ts` | |
| 61 | 65 | - An app's code runs in a cell: read-only machine, empty data directory and |
| 62 | 66 | home, own pid and network namespaces, memory, CPU and disk watched. A |
| 63 | 67 | dependency an app pulls in gets no more than the app does. |
| 64 | Test: `test/contain.test.ts` | |
| 68 | Enforced by: `test/contain.test.ts` | |
| 65 | 69 | - The only way out of a cell is the egress proxy, which reaches the hosts the |
| 66 | 70 | repository declared and refuses names that resolve to private, loopback or |
| 67 | 71 | metadata addresses. |
| 68 | Test: `test/apps-adversarial.test.ts` | |
| 72 | Enforced by: `test/apps-adversarial.test.ts` | |
| 69 | 73 | - A connected origin's replies are validated at the boundary and not trusted for |
| 70 | 74 | identity. |
| 71 | Test: `test/origins-security-provider.test.ts` | |
| 75 | Enforced by: `test/origins-security-provider.test.ts` | |
| 72 | 76 | - Pushes from an origin cannot rewrite what the forge protects. |
| 73 | Test: `test/origins-security-push.test.ts` | |
| 77 | Enforced by: `test/origins-security-push.test.ts` | |
| 74 | 78 | - Backups are encrypted before they leave the machine. |
| 75 | Test: `test/snapshot-store.test.ts` | |
| 79 | Enforced by: `test/snapshot-store.test.ts` | |
| 76 | 80 | - A review of lockfile changes (`bun.lock`) before they reach an app's cell. |
| 77 | Test: none yet | |
| 81 | Enforced by: none yet | |
| 78 | 82 | |
| 79 | 83 | ## A hostile agent inside a run |
| 80 | 84 | |
| @@ -86,81 +90,81 @@ person approved. | ||
| 86 | 90 | host, and never another repository's app. The |
| 87 | 91 | forge's repositories, index, key and every other agent's clone are denied by |
| 88 | 92 | name and by allow-list, and a listing that fails refuses the run. |
| 89 | Test: `test/crew-sandbox.test.ts` | |
| 93 | Enforced by: `test/crew-sandbox.test.ts` | |
| 90 | 94 | - The one place a run writes beside its clone and temp directory is its |
| 91 | 95 | repository's bun install cache (`crew/.cache/<repo id>/bun`), which every |
| 92 | 96 | run on that repository can write: a run can poison what the next run on the |
| 93 | 97 | same repository installs. It is per repository: the cache root is denied whole |
| 94 | 98 | and only this repository's directory re-allowed, so a cache made later is |
| 95 | 99 | denied too, and deleting the repository removes it. |
| 96 | Test: `test/crew-install-cache.test.ts` | |
| 100 | Enforced by: `test/crew-install-cache.test.ts` | |
| 97 | 101 | - A check's cell reads its repository's shared bun cache |
| 98 | 102 | (`check-install/<repo id>/bun`) through a read-only bind and cannot write |
| 99 | 103 | it, so one pull cannot plant a package the gate on main installs. The |
| 100 | 104 | forge fills it on the host with `--ignore-scripts`, and reads no |
| 101 | 105 | `bunfig.toml` or `.npmrc` of the pull's. |
| 102 | Test: `test/check-install.test.ts` | |
| 106 | Enforced by: `test/check-install.test.ts` | |
| 103 | 107 | - An app environment's cell reads that same cache read-only and |
| 104 | 108 | copies from it, so a preview cannot plant a package production installs; |
| 105 | 109 | the forge fills it through the same lockfile rule as a check's. |
| 106 | Test: `test/app-install-cache.test.ts` | |
| 110 | Enforced by: `test/app-install-cache.test.ts` | |
| 107 | 111 | - The forge fills that cache only from a lockfile whose every package is a |
| 108 | 112 | registry `name@version` with a sha512 integrity (`checks/fillable.ts`): |
| 109 | 113 | the host never fetches a URL a pull chose and reads no file, link or |
| 110 | 114 | workspace a pull points at, an override's or resolution's included. |
| 111 | Test: `test/check-fillable.test.ts` | |
| 115 | Enforced by: `test/check-fillable.test.ts` | |
| 112 | 116 | - The same policy holds against a real bubblewrap, not only as a string. |
| 113 | Test: `test/crew-sandbox-real.test.ts` | |
| 117 | Enforced by: `test/crew-sandbox-real.test.ts` | |
| 114 | 118 | - An agent cannot push to the default branch or merge; a person merges. |
| 115 | Test: `test/protection.test.ts` | |
| 119 | Enforced by: `test/protection.test.ts` | |
| 116 | 120 | - An outside agent that drives a person's account cannot push past the pulls |
| 117 | 121 | either: a protected default takes a push from its admins only, so every |
| 118 | 122 | other writer lands through a pull, its gate and its review. |
| 119 | Test: `test/protected-default-push.test.ts` | |
| 123 | Enforced by: `test/protected-default-push.test.ts` | |
| 120 | 124 | - Merge gates (checks, review) are enforced in the merge path, not by convention. |
| 121 | Test: `test/security-merge.test.ts` | |
| 125 | Enforced by: `test/security-merge.test.ts` | |
| 122 | 126 | - Spend, time and turns per run are limited. |
| 123 | Test: `test/crew-limits.test.ts` | |
| 127 | Enforced by: `test/crew-limits.test.ts` | |
| 124 | 128 | - Rendered output escapes everything an agent wrote. |
| 125 | Test: `test/security-xss.test.ts` | |
| 129 | Enforced by: `test/security-xss.test.ts` | |
| 126 | 130 | |
| 127 | 131 | ## An automated attacker at scale |
| 128 | 132 | |
| 129 | 133 | Wants: accounts, spam, resources, or a way to exhaust the forge. |
| 130 | 134 | |
| 131 | 135 | - Request bodies are capped at the boundary. |
| 132 | Test: `test/body-limits.test.ts` | |
| 136 | Enforced by: `test/body-limits.test.ts` | |
| 133 | 137 | - Parsers survive generated hostile input. |
| 134 | Test: `test/fuzz-parsers.test.ts` | |
| 138 | Enforced by: `test/fuzz-parsers.test.ts` | |
| 135 | 139 | - Anonymous spawn is rate limited per client address. |
| 136 | Test: `test/spawn-anon.test.ts` | |
| 140 | Enforced by: `test/spawn-anon.test.ts` | |
| 137 | 141 | - Every door a stranger can reach that costs (API reads, clones, code |
| 138 | 142 | search on the page, the API and the terminal's guest palette, event |
| 139 | 143 | streams, spawn plans) is limited per address and for every stranger |
| 140 | 144 | together (`src/http/strangers.ts`, docs/ARCHITECTURE.md, concurrency |
| 141 | 145 | invariant 7); a signed-in push keeps its share of the pool. |
| 142 | Test: `test/strangers.test.ts` | |
| 146 | Enforced by: `test/strangers.test.ts` | |
| 143 | 147 | - The client address is the socket's peer unless that peer is a configured |
| 144 | 148 | trusted proxy; `X-Forwarded-For` is read from the right, past trusted hops. |
| 145 | Test: `test/client-address.test.ts` | |
| 149 | Enforced by: `test/client-address.test.ts` | |
| 146 | 150 | - A new-account guard can refuse repository creation and forks for young |
| 147 | 151 | accounts. |
| 148 | Test: `test/guard-new-accounts.test.ts` | |
| 152 | Enforced by: `test/guard-new-accounts.test.ts` | |
| 149 | 153 | - Sign-up has a ceiling per address and for the whole forge, per minute and per |
| 150 | 154 | hour, and an admin can close it at once. |
| 151 | Test: `test/guest-contribute.test.ts` | |
| 155 | Enforced by: `test/guest-contribute.test.ts` | |
| 152 | 156 | - A guest owns only a contribution fork of a public repository. It holds no |
| 153 | 157 | settings, secrets or app, nothing runs for it, and its storage, forks and |
| 154 | 158 | pulls are limited (`docs/ROLES.md`). |
| 155 | Test: `test/guest-contribute.test.ts` | |
| 159 | Enforced by: `test/guest-contribute.test.ts` | |
| 156 | 160 | - A guest's pull runs nothing until a person who can write the repository |
| 157 | 161 | approves that exact head, and the crew never lands it. |
| 158 | Test: `test/guest-contribute.test.ts` | |
| 162 | Enforced by: `test/guest-contribute.test.ts` | |
| 159 | 163 | - Every limit is a setting with its reason, changed by an admin only. |
| 160 | Test: `test/limits.test.ts` | |
| 164 | Enforced by: `test/limits.test.ts` | |
| 161 | 165 | - Distributed attacks across many addresses: only the whole-forge caps stand |
| 162 | 166 | here. |
| 163 | Test: none yet | |
| 167 | Enforced by: none yet | |
| 164 | 168 | |
| 165 | 169 | ## Internal calls |
| 166 | 170 | |
| @@ -171,11 +175,11 @@ else runs as that user on the host network (agents and apps are sandboxed or in | ||
| 171 | 175 | cells). Only one swap is made per process, so two landings never leave two |
| 172 | 176 | releases serving. The swap is refused unless `net.ipv4.tcp_migrate_req=1`, for |
| 173 | 177 | without it the old listener closing resets queued connections. |
| 174 | Test: `test/release-swap.test.ts` | |
| 178 | Enforced by: `test/release-swap.test.ts` | |
| 175 | 179 | |
| 176 | 180 | How each part proves itself to the next, today. |
| 177 | 181 | |
| 178 | | Call | Authenticated by | Test | | |
| 182 | | Call | Authenticated by | Enforced by | | |
| 179 | 183 | | --- | --- | --- | |
| 180 | 184 | | runner → forge | the run's scoped token (`FORGE_TOKEN`) and a few-minute passport (`FORGE_PASSPORT`) | `test/passport-edge.test.ts` | |
| 181 | 185 | | git hook → forge | per-boot secret in `x-forge-hook-secret` on the internal listener; a call without it is refused and counted | `test/crew-sandbox.test.ts` | |
| @@ -197,7 +201,7 @@ What it can reach: everything a child forge can reach, no more: its own data dir | ||
| 197 | 201 | - The cell limits, sandbox, egress list and hidden parent paths are built by the |
| 198 | 202 | parent from outside, never read from the child's repository, so a child's own |
| 199 | 203 | code is held exactly like any app. |
| 200 | Test: `test/own-code.test.ts` | |
| 204 | Enforced by: `test/own-code.test.ts` | |
| 201 | 205 | |
| 202 | 206 | ## Honeypots (later) |
| 203 | 207 |
src/core/threats.ts +80 −7
| @@ -1,17 +1,28 @@ | ||
| 1 | 1 | /** |
| 2 | * The protections in docs/THREATS.md, as plain data: each `Test:` line (after a | |
| 3 | * bullet or a paragraph) and each table row naming a call, with the test file | |
| 4 | * cited, or none. | |
| 2 | * The protections in docs/THREATS.md, as plain data: each `Enforced by:` line | |
| 3 | * (after a bullet or a paragraph) and each table row naming a call, with the | |
| 4 | * enforcer cited, or none. | |
| 5 | 5 | */ |
| 6 | 6 | |
| 7 | 7 | export interface Protection { |
| 8 | 8 | readonly adversary: string; |
| 9 | 9 | readonly text: string; |
| 10 | /** The cited `test/*.test.ts`, or undefined for "none yet". */ | |
| 10 | /** | |
| 11 | * The cited enforcer, a `test/*.test.ts` or a `scripts/*.ts` gate, or | |
| 12 | * undefined for "none yet". A gate never appears in a gate's per-file | |
| 13 | * results, so its standing reads "untested". | |
| 14 | */ | |
| 11 | 15 | readonly test: string | undefined; |
| 12 | 16 | } |
| 13 | 17 | |
| 14 | const CITED = /^`(test\/[A-Za-z0-9_.-]+\.test\.ts)`$/; | |
| 18 | /** The line that closes each protection. */ | |
| 19 | export const ENFORCED_BY = /^\s*Enforced by: /; | |
| 20 | ||
| 21 | /** One cited enforcer: a test file or a gate under scripts/, in backticks. */ | |
| 22 | export const CITED = /^`((?:test\/[A-Za-z0-9_.-]+\.test\.ts)|(?:scripts\/[A-Za-z0-9_./-]+\.ts))`$/; | |
| 23 | ||
| 24 | /** A gap: nothing enforces the row yet, optionally with the open issue owning it. */ | |
| 25 | export const NONE_YET = /^none yet(?: \(#\d+\))?$/; | |
| 15 | 26 | |
| 16 | 27 | export function parseThreats(md: string): Protection[] { |
| 17 | 28 | const out: Protection[] = []; |
| @@ -34,10 +45,10 @@ export function parseThreats(md: string): Protection[] { | ||
| 34 | 45 | adversary = line.slice(3).trim(); |
| 35 | 46 | } else if (line.startsWith("- ")) { |
| 36 | 47 | bullet = [line.slice(2)]; |
| 37 | } else if (/^\s*Test: /.test(line)) { | |
| 48 | } else if (ENFORCED_BY.test(line)) { | |
| 38 | 49 | if (bullet === undefined && para.length > 0) bullet = para; |
| 39 | 50 | para = []; |
| 40 | close(line.replace(/^\s*Test: /, "").trim()); | |
| 51 | close(line.replace(ENFORCED_BY, "").trim()); | |
| 41 | 52 | } else if (/^\s+\S/.test(line) && bullet !== undefined) { |
| 42 | 53 | bullet.push(line.trim()); |
| 43 | 54 | } else if (line.startsWith("| ") && line.includes("→")) { |
| @@ -54,6 +65,68 @@ export function parseThreats(md: string): Protection[] { | ||
| 54 | 65 | return out; |
| 55 | 66 | } |
| 56 | 67 | |
| 68 | /** What `enforcement` finds in the doc. */ | |
| 69 | export interface Enforcement { | |
| 70 | /** One sentence per row that breaks the rule, naming the row. */ | |
| 71 | readonly problems: string[]; | |
| 72 | /** How many rows say `none yet`. */ | |
| 73 | readonly noneYet: number; | |
| 74 | /** Every enforcer path cited, in order, repeats kept. */ | |
| 75 | readonly cited: string[]; | |
| 76 | } | |
| 77 | ||
| 78 | /** | |
| 79 | * Hold docs/THREATS.md to its rule: every bullet (outside a `(later)` section, | |
| 80 | * whose bullets are preconditions, not protections) ends in one `Enforced by:` | |
| 81 | * line, every call row names its enforcer in its last cell, each enforcer is | |
| 82 | * a path `exists` says is there or `none yet`. | |
| 83 | */ | |
| 84 | export function enforcement(md: string, exists: (path: string) => boolean): Enforcement { | |
| 85 | const problems: string[] = []; | |
| 86 | const cited: string[] = []; | |
| 87 | let noneYet = 0; | |
| 88 | let later = false; | |
| 89 | let open: string | undefined; | |
| 90 | const unclosed = (): void => { | |
| 91 | if (open !== undefined) problems.push(`"${open}" has no Enforced by: line`); | |
| 92 | open = undefined; | |
| 93 | }; | |
| 94 | const judge = (row: string, value: string): void => { | |
| 95 | if (NONE_YET.test(value)) { | |
| 96 | noneYet++; | |
| 97 | return; | |
| 98 | } | |
| 99 | const path = CITED.exec(value)?.[1]; | |
| 100 | if (path === undefined) problems.push(`"${row}" names neither a test, a scripts/ gate nor "none yet": ${value}`); | |
| 101 | else if (!exists(path)) problems.push(`"${row}" names ${path}, which does not exist`); | |
| 102 | else cited.push(path); | |
| 103 | }; | |
| 104 | for (const line of md.split("\n")) { | |
| 105 | if (line.startsWith("## ")) { | |
| 106 | unclosed(); | |
| 107 | later = /\(later\)\s*$/.test(line); | |
| 108 | } else if (later) { | |
| 109 | continue; | |
| 110 | } else if (line.startsWith("- ")) { | |
| 111 | unclosed(); | |
| 112 | open = line.slice(2, 62); | |
| 113 | } else if (ENFORCED_BY.test(line)) { | |
| 114 | judge(open ?? "the paragraph above", line.replace(ENFORCED_BY, "").trim()); | |
| 115 | open = undefined; | |
| 116 | } else if (/^\s*Test: /.test(line)) { | |
| 117 | problems.push(`"${open ?? line.trim()}" uses the old Test: line; say Enforced by:`); | |
| 118 | open = undefined; | |
| 119 | } else if (line.startsWith("| ") && line.includes("→")) { | |
| 120 | const cells = line.split("|").map((c) => c.trim()); | |
| 121 | judge(cells[1] ?? line, cells.at(-2) ?? ""); | |
| 122 | } else if (!/^\s+\S/.test(line)) { | |
| 123 | unclosed(); | |
| 124 | } | |
| 125 | } | |
| 126 | unclosed(); | |
| 127 | return { problems, noneYet, cited }; | |
| 128 | } | |
| 129 | ||
| 57 | 130 | export type Standing = "green" | "red" | "untested" | "no test yet"; |
| 58 | 131 | |
| 59 | 132 | /** One protection against the last gate: a cited file that did not run is untested. */ |
test/checks-resilience.test.ts +2 −2
| @@ -15,9 +15,9 @@ import { readResilience, type Row } from "../src/checks/resilience.ts"; | ||
| 15 | 15 | const DOC = readFileSync(join(import.meta.dir, "..", "docs", "THREATS.md"), "utf8"); |
| 16 | 16 | const UNKNOWN = new Set(["untested", "no test yet"]); |
| 17 | 17 | |
| 18 | // Counted from the document without its parser: one protection per `Test:` | |
| 18 | // Counted from the document without its parser: one protection per `Enforced by:` | |
| 19 | 19 | // line, one per table row naming a call. |
| 20 | const TESTS = DOC.split("\n").filter((l) => /^\s*Test: /.test(l)).length; | |
| 20 | const TESTS = DOC.split("\n").filter((l) => /^\s*Enforced by: /.test(l)).length; | |
| 21 | 21 | const CALLS = DOC.split("\n").filter((l) => l.startsWith("| ") && l.includes("→")).length; |
| 22 | 22 | |
| 23 | 23 | let dataDir: string; |
test/resilience.test.ts +6 −6
| @@ -29,9 +29,9 @@ const DOC = `# Threats | ||
| 29 | 29 | ## A stolen token |
| 30 | 30 | |
| 31 | 31 | - Tokens carry a scope. |
| 32 | Test: \`test/tokens.test.ts\` | |
| 32 | Enforced by: \`test/tokens.test.ts\` | |
| 33 | 33 | - Revoking every token at once. |
| 34 | Test: none yet | |
| 34 | Enforced by: none yet | |
| 35 | 35 | |
| 36 | 36 | | call | note | test | |
| 37 | 37 | | --- | --- | --- | |
| @@ -63,17 +63,17 @@ describe("resilience", () => { | ||
| 63 | 63 | ]); |
| 64 | 64 | }); |
| 65 | 65 | |
| 66 | test("the real docs/THREATS.md yields one protection per Test: line and per → row", async () => { | |
| 66 | test("the real docs/THREATS.md yields one protection per Enforced by: line and per → row", async () => { | |
| 67 | 67 | const md = await Bun.file(join(import.meta.dir, "..", "docs", "THREATS.md")).text(); |
| 68 | const tests = md.split("\n").filter((l) => /^\s*Test: /.test(l)).length; | |
| 68 | const tests = md.split("\n").filter((l) => /^\s*Enforced by: /.test(l)).length; | |
| 69 | 69 | const rows = md.split("\n").filter((l) => l.startsWith("| ") && l.includes("→")).length; |
| 70 | 70 | const got = parseThreats(md); |
| 71 | 71 | expect(got.length).toBe(tests + rows); |
| 72 | 72 | expect(got.some((p) => p.test === "test/release-swap.test.ts" && p.text.includes("SO_REUSEPORT"))).toBe(true); |
| 73 | 73 | }); |
| 74 | 74 | |
| 75 | test("an unindented Test: line closes the paragraph above it", () => { | |
| 76 | const got = parseThreats("## X\n\nA prose protection.\nTest: `test/a.test.ts`\n"); | |
| 75 | test("an unindented Enforced by: line closes the paragraph above it", () => { | |
| 76 | const got = parseThreats("## X\n\nA prose protection.\nEnforced by: `test/a.test.ts`\n"); | |
| 77 | 77 | expect(got.map((p) => [p.text, p.test])).toEqual([["A prose protection.", "test/a.test.ts"]]); |
| 78 | 78 | }); |
| 79 | 79 |
test/threats-doc.test.ts deleted −54
| @@ -1,54 +0,0 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { existsSync, readFileSync } from "node:fs"; | |
| 3 | import { join } from "node:path"; | |
| 4 | ||
| 5 | const root = join(import.meta.dir, ".."); | |
| 6 | const doc = readFileSync(join(root, "docs/THREATS.md"), "utf8"); | |
| 7 | ||
| 8 | test("the doc covers each adversary, the internal calls and honeypots", () => { | |
| 9 | for (const h of [ | |
| 10 | "## A snooper on the network", | |
| 11 | "## A stolen token", | |
| 12 | "## A compromised vendor", | |
| 13 | "## A hostile agent inside a run", | |
| 14 | "## An automated attacker at scale", | |
| 15 | "## Internal calls", | |
| 16 | "## Honeypots (later)", | |
| 17 | ]) | |
| 18 | expect(doc).toContain(h); | |
| 19 | for (const call of ["runner → forge", "cell → proxy", "edge → pod", "child → parent"]) | |
| 20 | expect(doc).toContain(call); | |
| 21 | expect(doc).toContain('`kind: "honeypot"`'); | |
| 22 | }); | |
| 23 | ||
| 24 | test("every test file the doc cites exists", () => { | |
| 25 | const cited = new Set( | |
| 26 | [...doc.matchAll(/`(test\/[A-Za-z0-9_.-]+\.test\.ts)`/g)].map((m) => m[1]!), | |
| 27 | ); | |
| 28 | expect(cited.size).toBeGreaterThan(10); | |
| 29 | for (const file of cited) expect(existsSync(join(root, file))).toBe(true); | |
| 30 | }); | |
| 31 | ||
| 32 | test("every protection ends in a test file or 'none yet', and the gaps are counted", () => { | |
| 33 | const lines = doc.split("\n"); | |
| 34 | const tests = lines.filter((l) => /^\s+Test: /.test(l)); | |
| 35 | // Each bullet outside the honeypot list has exactly one Test: line. | |
| 36 | const honeypots = doc.indexOf("## Honeypots"); | |
| 37 | const bullets = lines.filter((l) => l.startsWith("- ")).length; | |
| 38 | const inHoneypots = doc.slice(honeypots).split("\n").filter((l) => l.startsWith("- ")).length; | |
| 39 | expect(tests.length).toBe(bullets - inHoneypots); | |
| 40 | let gaps = 0; | |
| 41 | for (const l of tests) { | |
| 42 | const what = l.replace(/^\s+Test: /, ""); | |
| 43 | if (what === "none yet") gaps++; | |
| 44 | else expect(what).toMatch(/^`test\/[^`]+\.test\.ts`$/); | |
| 45 | } | |
| 46 | // Table rows name their test in the last cell. | |
| 47 | for (const l of lines.filter((l) => l.startsWith("| ") && l.includes("→"))) { | |
| 48 | const cell = l.split("|").at(-2)!.trim(); | |
| 49 | if (cell === "none yet") gaps++; | |
| 50 | else expect(cell).toMatch(/^`test\/[^`]+\.test\.ts`$/); | |
| 51 | } | |
| 52 | expect(gaps).toBeGreaterThan(0); | |
| 53 | console.log(`threats: ${gaps} protection${gaps === 1 ? "" : "s"} with no test yet`); | |
| 54 | }); |
test/threats-enforced.test.ts added +83
| @@ -0,0 +1,83 @@ | ||
| 1 | /** | |
| 2 | * docs/THREATS.md names what enforces each protection: one `Enforced by:` line | |
| 3 | * per bullet, the last cell of each call row, a test file or a scripts/ gate | |
| 4 | * that exists, or `none yet`. The `none yet` count is printed so it is seen, | |
| 5 | * and only editing a row lowers it. | |
| 6 | */ | |
| 7 | import { expect, test } from "bun:test"; | |
| 8 | import { existsSync, readFileSync } from "node:fs"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { enforcement, parseThreats } from "../src/core/threats.ts"; | |
| 11 | ||
| 12 | const root = join(import.meta.dir, ".."); | |
| 13 | const doc = readFileSync(join(root, "docs/THREATS.md"), "utf8"); | |
| 14 | const exists = (path: string): boolean => existsSync(join(root, path)); | |
| 15 | ||
| 16 | test("the doc covers each adversary, the internal calls and honeypots", () => { | |
| 17 | for (const h of [ | |
| 18 | "## A snooper on the network", | |
| 19 | "## A stolen token", | |
| 20 | "## A compromised vendor", | |
| 21 | "## A hostile agent inside a run", | |
| 22 | "## An automated attacker at scale", | |
| 23 | "## Internal calls", | |
| 24 | "## Honeypots (later)", | |
| 25 | ]) | |
| 26 | expect(doc).toContain(h); | |
| 27 | for (const call of ["runner → forge", "cell → proxy", "edge → pod", "child → parent"]) | |
| 28 | expect(doc).toContain(call); | |
| 29 | expect(doc).toContain('`kind: "honeypot"`'); | |
| 30 | }); | |
| 31 | ||
| 32 | test("every row names an enforcer that exists, or says none yet; the gaps are counted", () => { | |
| 33 | const got = enforcement(doc, exists); | |
| 34 | expect(got.problems).toEqual([]); | |
| 35 | expect(got.cited.length).toBeGreaterThan(10); | |
| 36 | // Every row the resilience page lists was judged here, none skipped. | |
| 37 | expect(got.cited.length + got.noneYet).toBe(parseThreats(doc).length); | |
| 38 | console.log(`threats: ${got.noneYet} row${got.noneYet === 1 ? "" : "s"} enforced by none yet`); | |
| 39 | }); | |
| 40 | ||
| 41 | const FIXTURE = (rows: string): string => `# Threats\n\n## An adversary\n\n${rows}\n`; | |
| 42 | const always = (): boolean => true; | |
| 43 | ||
| 44 | test("a bullet with no Enforced by: line fails, naming the bullet", () => { | |
| 45 | const got = enforcement(FIXTURE("- Tokens carry a scope.\n- Sessions are HttpOnly.\n Enforced by: `test/a.test.ts`"), always); | |
| 46 | expect(got.problems).toEqual(['"Tokens carry a scope." has no Enforced by: line']); | |
| 47 | }); | |
| 48 | ||
| 49 | test("a named path that does not exist fails", () => { | |
| 50 | const got = enforcement(FIXTURE("- A.\n Enforced by: `test/gone.test.ts`"), () => false); | |
| 51 | expect(got.problems).toEqual(['"A." names test/gone.test.ts, which does not exist']); | |
| 52 | }); | |
| 53 | ||
| 54 | test("an enforcer that is neither a test, a scripts/ gate nor none yet fails", () => { | |
| 55 | const got = enforcement(FIXTURE("- A.\n Enforced by: review\n- B.\n Enforced by: `src/a.ts`"), always); | |
| 56 | expect(got.problems.length).toBe(2); | |
| 57 | }); | |
| 58 | ||
| 59 | test("the old Test: line fails", () => { | |
| 60 | const got = enforcement(FIXTURE("- A.\n Test: `test/a.test.ts`"), always); | |
| 61 | expect(got.problems).toEqual(['"A." uses the old Test: line; say Enforced by:']); | |
| 62 | }); | |
| 63 | ||
| 64 | test("a scripts/ gate counts as an enforcer; none yet, with or without its issue, is counted", () => { | |
| 65 | const got = enforcement( | |
| 66 | FIXTURE( | |
| 67 | "- A.\n Enforced by: `scripts/gate.ts`\n- B.\n Enforced by: none yet\n- C.\n Enforced by: none yet (#12)\n\n" + | |
| 68 | "| Call | How | Enforced by |\n| --- | --- | --- |\n| a → b | c | none yet |\n| b → c | d | `test/x.test.ts` |", | |
| 69 | ), | |
| 70 | always, | |
| 71 | ); | |
| 72 | expect(got).toEqual({ problems: [], noneYet: 3, cited: ["scripts/gate.ts", "test/x.test.ts"] }); | |
| 73 | }); | |
| 74 | ||
| 75 | test("a call row whose last cell names no enforcer fails", () => { | |
| 76 | const got = enforcement(FIXTURE("| Call | How | Enforced by |\n| --- | --- | --- |\n| a → b | c | |"), always); | |
| 77 | expect(got.problems.length).toBe(1); | |
| 78 | }); | |
| 79 | ||
| 80 | test("bullets under a (later) heading are preconditions, not rows", () => { | |
| 81 | const got = enforcement("# T\n\n## Honeypots (later)\n\n- its egress must be none;\n", always); | |
| 82 | expect(got).toEqual({ problems: [], noneYet: 0, cited: [] }); | |
| 83 | }); |
new above
offline