Public: anyone can read this; the crew can change it.
deepsec findings file as sealed reports linked to their pull, matched against ones already filed #15044
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/15025-deepsec-findings-file-as-sealed-reports → main 4 commits fork 3 threads · 3 unresolved
7 files
changed +871
−0
docs/features/deepsec-findings.md added +41
| @@ -0,0 +1,41 @@ | ||
| 1 | --- | |
| 2 | topic: When something is wrong | |
| 3 | row: deepsec findings become sealed reports under P0 Trust | |
| 4 | --- | |
| 5 | # deepsec findings become sealed reports under P0 Trust | |
| 6 | ||
| 7 | This page is for people who write a repository and run the deepsec security scanner on it. Each finding deepsec makes becomes one sealed issue under the repository's P0 Trust umbrella. A sealed issue is shown only to writers and the crew until the fix lands. If you file the same findings again, nothing is filed twice. | |
| 8 | ||
| 9 | ## File a run | |
| 10 | ||
| 11 | Run deepsec's `process` and `revalidate` steps first. Then hand its data directory to the forge: | |
| 12 | ||
| 13 | sh | |
| 14 | FORGE_URL=https://your.forge FORGE_TOKEN=<a read-write token> \ | |
| 15 | bun scripts/deepsec-file.ts .deepsec/data/forge --pull 123 | |
| 16 | ||
| 17 | ||
| 18 | - The directory defaults to `.deepsec/data/forge`. | |
| 19 | - `--repo owner/name` names the repository. Without it the script reads `FORGE_REPO`, then falls back to `trevato/forge`. | |
| 20 | - `--pull N` names the pull whose check ran deepsec. `--commit <oid>` names the commit it read. `--tool "deepsec 2.3.16 (model)"` names the scanner. You can leave each of them out. | |
| 21 | - The token must be read-write and belong to someone who writes the repository. Anyone else gets `403`. | |
| 22 | ||
| 23 | The script sends only the fields the forge reads. It posts them in batches of at most 100 findings, and the runs go with the first batch. It skips symbolic links, so it reads nothing outside the directory. It prints one line per batch, for example `deepsec: 3 findings, 2 filed as sealed reports (#41, #42), 1 already filed (#17); spent $1.20; does not block the gate until its false-positive rate is measured.` | |
| 24 | ||
| 25 | ## What a report says | |
| 26 | ||
| 27 | Each report is titled with the finding's title. Its body gives the severity, the file and lines, whether revalidation confirmed it, what deepsec saw, the fix it suggests and what its recheck said. If you named a pull, the body names it too. A report is done when a test reproduces the attack, fails before the fix and passes after, and the matching `docs/THREATS.md` row names that test. If a finding is not exploitable, close its report and give the reason. | |
| 28 | ||
| 29 | A finding is matched by its title and its file. The file must be the whole path, so `src/a.ts` never matches `src/a.tsx`. Only reports that this feature opened count. If one is still open with both, the run counts the finding as already filed and does not open another. | |
| 30 | ||
| 31 | ## The API | |
| 32 | ||
| 33 | `POST /api/v1/repos/:owner/:name/deepsec` takes `{files, runs, pull?, commit?, tool?}`. `files` and `runs` are deepsec's own records, at most 5,000 of each. One request files at most 100 findings, the most severe first. The line says how many more it did not file, so you can send them in another request. The answer is the same line the script prints. A malformed body gets `422` and says which field is wrong. | |
| 34 | ||
| 35 | ## In the terminal | |
| 36 | ||
| 37 | Filing is not a terminal screen: a script or a check runs it. The reports it opens are ordinary sealed issues. In the terminal they show in the repository's issues and under the P0 Trust pillar, like every other report. | |
| 38 | ||
| 39 | ## Not yet | |
| 40 | ||
| 41 | A finding does not block the gate yet. It will block it once deepsec's false-positive rate has been measured over two weeks of runs. A pull's check step (#15024) and a weekly sweep (#15026) will call the same door. Today you run the script yourself. |
scripts/deepsec-file.ts added +183
| @@ -0,0 +1,183 @@ | ||
| 1 | /** | |
| 2 | * File what deepsec found as sealed reports (#15025). | |
| 3 | * | |
| 4 | * FORGE_URL=… FORGE_TOKEN=… bun scripts/deepsec-file.ts [dir] [--repo owner/name] [--pull N] | |
| 5 | * [--commit oid] [--tool "deepsec 2.3.16 (model)"] | |
| 6 | * | |
| 7 | * Reads deepsec's data directory (default `.deepsec/data/forge`) after | |
| 8 | * `process` and `revalidate`: every file record with findings, and | |
| 9 | * `runs/*.json` for what was spent. Posts them to | |
| 10 | * `POST /api/v1/repos/:owner/:name/deepsec`, which files each new finding | |
| 11 | * sealed under P0 Trust and matches each one already open. Prints the line | |
| 12 | * the forge answers for each batch. The token must write the repository. | |
| 13 | * | |
| 14 | * It sends only what the door reads (#15044): each record trimmed to the | |
| 15 | * fields `checks/deepsec_findings.ts` reads, its text capped, and the records | |
| 16 | * posted in batches of at most `FINDINGS_MAX` findings and `BATCH_BYTES` | |
| 17 | * bytes, so no batch is past the door's body cap or its findings cap. The runs | |
| 18 | * go with the first batch only, so what was spent is said once. A symbolic | |
| 19 | * link in the data directory is skipped: nothing outside it is read. | |
| 20 | */ | |
| 21 | ||
| 22 | import { lstatSync, readdirSync, readFileSync } from "node:fs"; | |
| 23 | import { join } from "node:path"; | |
| 24 | import { clean, FINDINGS_MAX, spentIn, TEXT_MAX } from "../src/checks/deepsec_findings.ts"; | |
| 25 | import { BODY_MAX } from "../src/http/limits.ts"; | |
| 26 | ||
| 27 | const FILES_MAX = 5_000; | |
| 28 | const BYTES_MAX = 2_000_000; | |
| 29 | const DEPTH_MAX = 12; | |
| 30 | /** | |
| 31 | * A quarter of the door's body cap: room for the JSON around the records and for multibyte text. | |
| 32 | */ | |
| 33 | export const BATCH_BYTES = Math.floor(BODY_MAX / 4); | |
| 34 | ||
| 35 | function fail(message: string): never { | |
| 36 | console.error(`deepsec-file: ${message}`); | |
| 37 | process.exit(1); | |
| 38 | } | |
| 39 | ||
| 40 | /** Every `.json` under `dir`, bounded; `runs/` at the top is left to `runRecords`. */ | |
| 41 | function jsonFiles(dir: string, depth = 0, acc: string[] = []): string[] { | |
| 42 | if (depth > DEPTH_MAX || acc.length >= FILES_MAX) return acc; | |
| 43 | let names: string[]; | |
| 44 | try { | |
| 45 | names = readdirSync(dir).sort(); | |
| 46 | } catch { | |
| 47 | return acc; | |
| 48 | } | |
| 49 | for (const name of names) { | |
| 50 | if (acc.length >= FILES_MAX) break; | |
| 51 | if (depth === 0 && name === "runs") continue; | |
| 52 | const path = join(dir, name); | |
| 53 | const st = lstatSync(path, { throwIfNoEntry: false }); | |
| 54 | if (st === undefined || st.isSymbolicLink()) continue; | |
| 55 | if (st.isDirectory()) jsonFiles(path, depth + 1, acc); | |
| 56 | else if (st.isFile() && name.endsWith(".json") && st.size <= BYTES_MAX) acc.push(path); | |
| 57 | } | |
| 58 | return acc; | |
| 59 | } | |
| 60 | ||
| 61 | function parse(path: string): unknown { | |
| 62 | try { | |
| 63 | return JSON.parse(readFileSync(path, "utf8")); | |
| 64 | } catch { | |
| 65 | return undefined; | |
| 66 | } | |
| 67 | } | |
| 68 | ||
| 69 | const hasFindings = (v: unknown): boolean => | |
| 70 | typeof v === "object" && v !== null && Array.isArray((v as { findings?: unknown }).findings) && (v as { findings: unknown[] }).findings.length > 0; | |
| 71 | ||
| 72 | const obj = (v: unknown): Record<string, unknown> => | |
| 73 | typeof v === "object" && v !== null && !Array.isArray(v) ? (v as Record<string, unknown>) : {}; | |
| 74 | const text = (v: unknown): string | undefined => (typeof v === "string" ? clean(v, TEXT_MAX) : undefined); | |
| 75 | ||
| 76 | /** | |
| 77 | * A recheck as the door reads it: its verdict, reasoning and adjusted severity, or its bare word. | |
| 78 | */ | |
| 79 | function trimRecheck(v: unknown): unknown { | |
| 80 | if (typeof v === "string") return clean(v, 200); | |
| 81 | const r = obj(v); | |
| 82 | return { verdict: text(r["verdict"]), reasoning: text(r["reasoning"]), adjustedSeverity: text(r["adjustedSeverity"]) }; | |
| 83 | } | |
| 84 | ||
| 85 | /** One file record cut to the fields the door reads, at most `FINDINGS_MAX` findings. */ | |
| 86 | export function trimRecord(raw: unknown): { filePath: unknown; findings: unknown[] } { | |
| 87 | const r = obj(raw); | |
| 88 | const findings = Array.isArray(r["findings"]) ? r["findings"].slice(0, FINDINGS_MAX) : []; | |
| 89 | return { | |
| 90 | filePath: typeof r["filePath"] === "string" ? r["filePath"] : r["path"], | |
| 91 | findings: findings.map((item) => { | |
| 92 | const f = obj(item); | |
| 93 | const recheck = f["revalidation"] ?? f["revalidate"]; | |
| 94 | return { | |
| 95 | severity: text(f["severity"]), | |
| 96 | title: text(f["title"]), | |
| 97 | description: text(f["description"]), | |
| 98 | lineNumbers: f["lineNumbers"] ?? f["lines"], | |
| 99 | recommendation: text(f["recommendation"] ?? f["fix"]), | |
| 100 | ...(recheck === undefined ? {} : { revalidation: trimRecheck(recheck) }), | |
| 101 | }; | |
| 102 | }), | |
| 103 | }; | |
| 104 | } | |
| 105 | ||
| 106 | /** deepsec's records under `dir`: file records with findings, trimmed, and what each run spent. */ | |
| 107 | export function readWorkspace(dir: string): { files: ReturnType<typeof trimRecord>[]; runs: unknown[] } { | |
| 108 | const files = jsonFiles(dir).map(parse).filter(hasFindings).map(trimRecord); | |
| 109 | const runs = jsonFiles(join(dir, "runs")) | |
| 110 | .map(parse) | |
| 111 | .filter((r) => r !== undefined) | |
| 112 | .map((r) => ({ totalCostUsd: spentIn([r]) })); | |
| 113 | return { files, runs }; | |
| 114 | } | |
| 115 | ||
| 116 | /** | |
| 117 | * The records as bodies the door accepts: each at most `FINDINGS_MAX` findings and `maxBytes` of | |
| 118 | * records, the runs with the first. A record is never split; one trimmed record fits by itself. | |
| 119 | */ | |
| 120 | export function batches( | |
| 121 | records: { files: readonly { findings: readonly unknown[] }[]; runs: readonly unknown[] }, | |
| 122 | maxBytes = BATCH_BYTES, | |
| 123 | ): { files: unknown[]; runs: unknown[] }[] { | |
| 124 | const out: { files: unknown[]; runs: unknown[] }[] = []; | |
| 125 | let files: unknown[] = []; | |
| 126 | let count = 0; | |
| 127 | let bytes = 0; | |
| 128 | for (const rec of records.files) { | |
| 129 | const size = Buffer.byteLength(JSON.stringify(rec)); | |
| 130 | if (files.length > 0 && (count + rec.findings.length > FINDINGS_MAX || bytes + size > maxBytes)) { | |
| 131 | out.push({ files, runs: [] }); | |
| 132 | files = []; | |
| 133 | count = 0; | |
| 134 | bytes = 0; | |
| 135 | } | |
| 136 | files.push(rec); | |
| 137 | count += rec.findings.length; | |
| 138 | bytes += size; | |
| 139 | } | |
| 140 | if (files.length > 0 || out.length === 0) out.push({ files, runs: [] }); | |
| 141 | out[0] = { files: out[0]!.files, runs: [...records.runs] }; | |
| 142 | return out; | |
| 143 | } | |
| 144 | ||
| 145 | function flag(args: string[], name: string): string | undefined { | |
| 146 | const at = args.indexOf(name); | |
| 147 | if (at === -1) return undefined; | |
| 148 | const value = args[at + 1]; | |
| 149 | if (value === undefined) fail(`${name} needs a value`); | |
| 150 | args.splice(at, 2); | |
| 151 | return value; | |
| 152 | } | |
| 153 | ||
| 154 | if (import.meta.main) { | |
| 155 | const args = process.argv.slice(2); | |
| 156 | const repo = flag(args, "--repo") ?? process.env["FORGE_REPO"] ?? "trevato/forge"; | |
| 157 | const pull = flag(args, "--pull"); | |
| 158 | const commit = flag(args, "--commit"); | |
| 159 | const tool = flag(args, "--tool"); | |
| 160 | const dir = args[0] ?? ".deepsec/data/forge"; | |
| 161 | const url = process.env["FORGE_URL"]; | |
| 162 | const token = process.env["FORGE_TOKEN"]; | |
| 163 | if (!url || !token) fail("needs FORGE_URL and FORGE_TOKEN"); | |
| 164 | if (!/^[\w.-]+\/[\w.-]+$/.test(repo)) fail("--repo is owner/name"); | |
| 165 | if (pull !== undefined && !/^\d+$/.test(pull)) fail("--pull is a number"); | |
| 166 | const records = readWorkspace(dir); | |
| 167 | for (const batch of batches(records)) { | |
| 168 | const res = await fetch(`${url}/api/v1/repos/${repo}/deepsec`, { | |
| 169 | method: "POST", | |
| 170 | headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, | |
| 171 | body: JSON.stringify({ | |
| 172 | ...batch, | |
| 173 | ...(pull !== undefined ? { pull: Number(pull) } : {}), | |
| 174 | ...(commit !== undefined ? { commit } : {}), | |
| 175 | ...(tool !== undefined ? { tool } : {}), | |
| 176 | }), | |
| 177 | signal: AbortSignal.timeout(60_000), | |
| 178 | }); | |
| 179 | const answer = (await res.json().catch(() => ({}))) as { line?: string; message?: string }; | |
| 180 | if (!res.ok) fail(`${res.status}: ${answer.message ?? "refused"}`); | |
| 181 | console.log(answer.line ?? "filed"); | |
| 182 | } | |
| 183 | } |
src/checks/deepsec.ts added +144
| @@ -0,0 +1,144 @@ | ||
| 1 | /** | |
| 2 | * deepsec's findings become sealed reports (#15025). Each new finding is | |
| 3 | * filed as a sealed issue under the P0 Trust umbrella, the way the | |
| 4 | * 2026-11-20 sweep's were (#14796); one that came from a pull's check names | |
| 5 | * that pull. A finding already filed and still open, the same file and the | |
| 6 | * same title, is matched instead, so a rerun never files it twice. | |
| 7 | * | |
| 8 | * A finding does not block the gate. Until its false-positive rate has been | |
| 9 | * measured over two weeks of runs, a red verdict on a guess would hold real | |
| 10 | * work; the line this returns says so. A pass reads at most `FINDINGS_MAX` | |
| 11 | * findings, the most severe, and says how many it did not file. | |
| 12 | * | |
| 13 | * Whatever ran deepsec hands in the records it wrote and gets back one line | |
| 14 | * for its log: today `scripts/deepsec-file.ts` through | |
| 15 | * `POST /api/v1/repos/:owner/:name/deepsec` (`http/api/deepsec.ts`); next | |
| 16 | * the check step on a pull's changed files (#15024) and the weekly sweep | |
| 17 | * (#15026). | |
| 18 | */ | |
| 19 | ||
| 20 | import type { App } from "../app.ts"; | |
| 21 | import { linkIssue } from "../actions/dependencies.ts"; | |
| 22 | import { openIssue } from "../actions/issues.ts"; | |
| 23 | import { getIssue, type Issue } from "../core/issues.ts"; | |
| 24 | import { findPillar, pillarsOf } from "../core/pillars.ts"; | |
| 25 | import type { Repo } from "../core/repos.ts"; | |
| 26 | import { getUserById } from "../core/users.ts"; | |
| 27 | import { FINDINGS_MAX, findingsIn, spentIn, type DeepsecFinding } from "./deepsec_findings.ts"; | |
| 28 | ||
| 29 | /** The pillar a sealed report stands under. */ | |
| 30 | export const PILLAR = "P0 Trust"; | |
| 31 | ||
| 32 | export interface Filed { | |
| 33 | /** Numbers of the reports opened by this pass. */ | |
| 34 | readonly opened: readonly number[]; | |
| 35 | /** Numbers of the open reports a finding matched. */ | |
| 36 | readonly matched: readonly number[]; | |
| 37 | readonly spentUsd: number; | |
| 38 | /** Findings past the cap, not filed by this pass. */ | |
| 39 | readonly dropped: number; | |
| 40 | /** One line for the check log. */ | |
| 41 | readonly line: string; | |
| 42 | } | |
| 43 | ||
| 44 | export interface Source { | |
| 45 | /** The pull whose check ran deepsec; absent for the weekly sweep. */ | |
| 46 | readonly pull?: number; | |
| 47 | /** The commit deepsec read. */ | |
| 48 | readonly commit?: string; | |
| 49 | /** `deepsec 2.3.16 (claude-opus-5-5)`: what to credit, when known. */ | |
| 50 | readonly tool?: string; | |
| 51 | readonly now?: Date; | |
| 52 | } | |
| 53 | ||
| 54 | /** Quoted text that pings nobody and closes nothing: `@` and `#` lose their reach. */ | |
| 55 | const inert = (text: string): string => text.replace(/@(?=\w)/g, "@\u200b").replace(/#(?=\d)/g, "#\u200b"); | |
| 56 | ||
| 57 | /** The sealed report a finding makes. Pure, so it is tested. */ | |
| 58 | export function reportFor(f: DeepsecFinding, source: Source): { title: string; body: string } { | |
| 59 | const where = f.lines.length > 0 ? `${f.file}:${f.lines.join(",")}` : f.file; | |
| 60 | const date = (source.now ?? new Date()).toISOString().slice(0, 10); | |
| 61 | const commit = source.commit !== undefined && /^[0-9a-f]{7,64}$/.test(source.commit) ? ` on ${source.commit.slice(0, 9)}` : ""; | |
| 62 | const tool = source.tool !== undefined ? inert(source.tool.replace(/[^\w .()/:-]/g, "").slice(0, 80)) : "deepsec"; | |
| 63 | const body = [ | |
| 64 | `### ${f.severity}: ${inert(f.title)}`, | |
| 65 | "", | |
| 66 | `\`${where}\` · ${f.confirmed ? "confirmed by revalidation" : "not yet rechecked"}`, | |
| 67 | "", | |
| 68 | inert(f.description), | |
| 69 | ...(f.fix === "" ? [] : ["", `**Fix.** ${inert(f.fix)}`]), | |
| 70 | ...(f.reasoning === "" ? [] : ["", `**Recheck.** ${inert(f.reasoning)}`]), | |
| 71 | "", | |
| 72 | ...(source.pull !== undefined ? [`From the deepsec check on pull #${source.pull}.`, ""] : []), | |
| 73 | "**Done when:** a test reproduces the attack, fails before the fix and passes after; the matching `docs/THREATS.md` row names that test; a finding judged not exploitable is closed here with the reason.", | |
| 74 | "", | |
| 75 | `Found by ${tool}${commit}, ${date}. Sealed: the detail stays with writers and the crew until the fix lands. ` + | |
| 76 | "It does not block the gate until its false-positive rate has been measured over two weeks.", | |
| 77 | ].join("\n"); | |
| 78 | return { title: f.title, body }; | |
| 79 | } | |
| 80 | ||
| 81 | /** | |
| 82 | * The open report this door already filed for this finding: the same title, and the file as the | |
| 83 | * whole code span `reportFor` writes (`` `src/a.ts` `` or `` `src/a.ts:12` ``, never `src/a.tsx`). | |
| 84 | * Only a report this door opened counts: sealed, in the owner's name, shaped as `reportFor` writes | |
| 85 | * it; an issue that merely mentions the file and title is not one. | |
| 86 | */ | |
| 87 | export function filedFor(app: App, repo: Repo, f: DeepsecFinding): Issue | undefined { | |
| 88 | const row = app.db | |
| 89 | .query<{ number: number }, [string, string, string, string, string]>( | |
| 90 | `SELECT number FROM issue | |
| 91 | WHERE repo_id = ?1 AND kind = 'issue' AND state = 'open' AND sealed = 1 AND author_id = ?2 | |
| 92 | AND title = ?3 AND substr(body, 1, 4) = '### ' | |
| 93 | AND instr(body, char(10) || 'Found by ') > 0 | |
| 94 | AND (instr(body, ?4) > 0 OR instr(body, ?5) > 0) | |
| 95 | ORDER BY number LIMIT 1`, | |
| 96 | ) | |
| 97 | .get(repo.id, repo.ownerId, f.title, `\`${f.file}\``, `\`${f.file}:`); | |
| 98 | return row === null ? undefined : getIssue(app.db, repo.id, row.number); | |
outdated
Loose |
||
| 99 | } | |
| 100 | ||
| 101 | const dollars = (usd: number): string => `$${usd.toFixed(2)}`; | |
| 102 | ||
| 103 | /** What deepsec wrote, as it wrote it: its file records and its run records, each untrusted. */ | |
| 104 | export interface Records { | |
| 105 | readonly files: readonly unknown[]; | |
| 106 | readonly runs: readonly unknown[]; | |
| 107 | } | |
| 108 | ||
| 109 | /** The log line a pass ends with. */ | |
| 110 | export function lineFor(found: number, opened: readonly number[], matched: readonly number[], usd: number, dropped = 0): string { | |
| 111 | const refs = (ns: readonly number[]) => (ns.length > 0 ? ` (${ns.map((n) => `#${n}`).join(", ")})` : ""); | |
| 112 | const what = | |
| 113 | found === 0 | |
| 114 | ? "deepsec: no findings" | |
| 115 | : `deepsec: ${found} finding${found === 1 ? "" : "s"}, ${opened.length} filed as sealed reports${refs(opened)}, ${matched.length} already filed${refs(matched)}`; | |
| 116 | const left = dropped > 0 ? `; ${dropped} more past the cap of ${FINDINGS_MAX} not filed: send them in another pass` : ""; | |
| 117 | return `${what}${left}; spent ${dollars(usd)}; does not block the gate until its false-positive rate is measured.`; | |
| 118 | } | |
| 119 | ||
| 120 | /** | |
| 121 | * File what deepsec found in `dataDir` against `repo`: each new finding a | |
| 122 | * sealed report under P0 Trust, opened in the owner's name like the sweep's; | |
| 123 | * each one already open matched. Returns what it did, and one line saying so. | |
| 124 | */ | |
| 125 | export function fileFindings(app: App, repo: Repo, records: Records, source: Source = {}): Filed { | |
| 126 | const { findings, dropped } = findingsIn(records.files); | |
| 127 | const usd = spentIn(records.runs); | |
| 128 | const owner = getUserById(app.db, repo.ownerId); | |
| 129 | const umbrella = findPillar(pillarsOf(app.db, repo).pillars, PILLAR)?.umbrella; | |
| 130 | const opened: number[] = []; | |
| 131 | const matched: number[] = []; | |
| 132 | for (const f of findings) { | |
| 133 | const known = filedFor(app, repo, f); | |
| 134 | if (known !== undefined) { | |
| 135 | if (!matched.includes(known.number)) matched.push(known.number); | |
| 136 | continue; | |
| 137 | } | |
| 138 | const made = openIssue(app, owner, repo, { ...reportFor(f, source), sealed: true }); | |
| 139 | if (!made.ok) continue; | |
| 140 | opened.push(made.value.number); | |
| 141 | if (umbrella !== undefined) linkIssue(app, owner, repo, umbrella, made.value); | |
| 142 | } | |
| 143 | return { opened, matched, spentUsd: usd, dropped, line: lineFor(findings.length, opened, matched, usd, dropped) }; | |
| 144 | } |
src/checks/deepsec_findings.ts added +149
| @@ -0,0 +1,149 @@ | ||
| 1 | /** | |
| 2 | * What deepsec found (#15025): the file records it writes under its data | |
| 3 | * directory (`.deepsec/data/forge/`) after `process` and `revalidate`, and | |
| 4 | * the run records that say what it spent. Plain data in, plain data out. | |
| 5 | * | |
| 6 | * deepsec is an agent that read untrusted code, so everything it wrote is | |
| 7 | * untrusted text: each record is checked at this boundary, a path must be a | |
| 8 | * plain repository path, text is stripped of control characters and capped, | |
| 9 | * and a finding its own recheck called a false positive is dropped. A record | |
| 10 | * that does not fit is skipped, never fatal: one bad record must not hide | |
| 11 | * the rest. | |
| 12 | */ | |
| 13 | ||
| 14 | export type Severity = "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "BUG"; | |
| 15 | ||
| 16 | export interface DeepsecFinding { | |
| 17 | /** Repository path, `src/http/respond.ts`. */ | |
| 18 | readonly file: string; | |
| 19 | readonly title: string; | |
| 20 | readonly severity: Severity; | |
| 21 | readonly lines: readonly number[]; | |
| 22 | readonly description: string; | |
| 23 | readonly fix: string; | |
| 24 | /** The recheck's verdict when `revalidate` ran on it. */ | |
| 25 | readonly confirmed: boolean; | |
| 26 | readonly reasoning: string; | |
| 27 | } | |
| 28 | ||
| 29 | const SEVERITIES = new Set<string>(["CRITICAL", "HIGH", "MEDIUM", "LOW", "BUG"]); | |
| 30 | /** A repository path: no `..`, no leading `/`, no spaces or quotes to break out of a code span. */ | |
| 31 | const PATH = /^(?!.*(?:^|\/)\.\.(?:\/|$))[A-Za-z0-9_.@+-]+(?:\/[A-Za-z0-9_.@+-]+)*$/; | |
| 32 | // eslint-disable-next-line no-control-regex | |
| 33 | const CONTROL = /[\u0000-\u0008\u000b-\u001f\u007f\u202A-\u202E\u2066-\u2069]/g; | |
| 34 | const TITLE_MAX = 200; | |
| 35 | export const TEXT_MAX = 6_000; | |
| 36 | const PATH_MAX = 300; | |
| 37 | /** More file records than one pass reads. */ | |
| 38 | export const FILES_MAX = 5_000; | |
| 39 | /** | |
| 40 | * More findings than one pass files (#15044): each is an issue and a link, so | |
| 41 | * 5,000 records of 100 findings must not open 500,000 of them in one request. | |
| 42 | * The most severe are kept; the line says how many were not filed. | |
| 43 | */ | |
| 44 | export const FINDINGS_MAX = 100; | |
| 45 | ||
| 46 | /** Untrusted text made safe to quote: no control or bidi characters, capped. */ | |
| 47 | export function clean(value: unknown, max: number): string { | |
| 48 | if (typeof value !== "string") return ""; | |
| 49 | const text = value.replace(CONTROL, "").trim(); | |
| 50 | return text.length > max ? `${text.slice(0, max - 1)}…` : text; | |
| 51 | } | |
| 52 | ||
| 53 | const record = (v: unknown): Record<string, unknown> | undefined => | |
| 54 | typeof v === "object" && v !== null && !Array.isArray(v) ? (v as Record<string, unknown>) : undefined; | |
| 55 | ||
| 56 | function lineNumbers(v: unknown): number[] { | |
| 57 | if (!Array.isArray(v)) return []; | |
| 58 | return v.filter((n): n is number => Number.isInteger(n) && n > 0 && n < 10_000_000).slice(0, 20); | |
| 59 | } | |
| 60 | ||
| 61 | /** The recheck's word on a finding: true, false, or undefined when it was not rechecked. */ | |
| 62 | function verdictOf(v: unknown): boolean | undefined { | |
| 63 | const r = record(v); | |
| 64 | const word = typeof r?.["verdict"] === "string" ? r["verdict"].toLowerCase() : typeof v === "string" ? v.toLowerCase() : ""; | |
| 65 | if (/false[-_ ]?positive|rejected|not[-_ ]?exploitable/.test(word)) return false; | |
| 66 | if (/true[-_ ]?positive|confirmed|valid/.test(word)) return true; | |
| 67 | return undefined; | |
| 68 | } | |
| 69 | ||
| 70 | /** One file record's findings, each checked; what fails a check is left out. */ | |
| 71 | export function findingsOf(raw: unknown): DeepsecFinding[] { | |
| 72 | const file = record(raw); | |
| 73 | if (file === undefined || !Array.isArray(file["findings"])) return []; | |
| 74 | const path = typeof file["filePath"] === "string" ? file["filePath"] : file["path"]; | |
| 75 | if (typeof path !== "string") return []; | |
| 76 | const rel = path.replace(/^\.\//, ""); | |
| 77 | if (rel.length > PATH_MAX || !PATH.test(rel)) return []; | |
| 78 | const out: DeepsecFinding[] = []; | |
| 79 | for (const item of file["findings"].slice(0, 100)) { | |
| 80 | const f = record(item); | |
| 81 | if (f === undefined) continue; | |
| 82 | const recheck = f["revalidation"] ?? f["revalidate"]; | |
| 83 | const verdict = verdictOf(recheck); | |
| 84 | if (verdict === false) continue; | |
| 85 | const adjusted = record(recheck)?.["adjustedSeverity"]; | |
| 86 | const severity = String(typeof adjusted === "string" ? adjusted : f["severity"] ?? "").toUpperCase(); | |
| 87 | if (!SEVERITIES.has(severity)) continue; | |
| 88 | const title = clean(f["title"], TITLE_MAX); | |
| 89 | if (title === "") continue; | |
| 90 | out.push({ | |
| 91 | file: rel, | |
| 92 | title, | |
| 93 | severity: severity as Severity, | |
| 94 | lines: lineNumbers(f["lineNumbers"] ?? f["lines"]), | |
| 95 | description: clean(f["description"], TEXT_MAX), | |
| 96 | fix: clean(f["recommendation"] ?? f["fix"], TEXT_MAX), | |
| 97 | confirmed: verdict === true, | |
| 98 | reasoning: clean(record(recheck)?.["reasoning"], TEXT_MAX), | |
| 99 | }); | |
| 100 | } | |
| 101 | return out; | |
| 102 | } | |
| 103 | ||
| 104 | /** What makes two findings one: the file and the title. */ | |
| 105 | const matchKey = (f: Pick<DeepsecFinding, "file" | "title">): string => `${f.file}\0${f.title}`; | |
| 106 | ||
| 107 | /** What one pass files: the most severe findings, and how many past the cap were left out. */ | |
outdated
No cap on the total findings returned. 5,000 records times 100 findings is 500,000 issues from one request. Cap it here. |
||
| 108 | export interface Findings { | |
| 109 | readonly findings: readonly DeepsecFinding[]; | |
| 110 | readonly dropped: number; | |
| 111 | } | |
| 112 | ||
| 113 | /** | |
| 114 | * Every finding in deepsec's file records, once each by file and title, most severe first, at most | |
| 115 | * {@link FINDINGS_MAX}. | |
| 116 | */ | |
| 117 | export function findingsIn(records: readonly unknown[]): Findings { | |
| 118 | const seen = new Set<string>(); | |
| 119 | const out: DeepsecFinding[] = []; | |
| 120 | for (const raw of records.slice(0, FILES_MAX)) { | |
| 121 | for (const f of findingsOf(raw)) { | |
| 122 | const key = matchKey(f); | |
| 123 | if (seen.has(key)) continue; | |
| 124 | seen.add(key); | |
| 125 | out.push(f); | |
| 126 | } | |
| 127 | } | |
| 128 | const rank = ["CRITICAL", "HIGH", "MEDIUM", "LOW", "BUG"]; | |
| 129 | out.sort((a, b) => rank.indexOf(a.severity) - rank.indexOf(b.severity)); | |
| 130 | return { findings: out.slice(0, FINDINGS_MAX), dropped: Math.max(0, out.length - FINDINGS_MAX) }; | |
| 131 | } | |
| 132 | ||
| 133 | /** A run record's `totalCostUsd`, at its top or one level down (`stats`, `meta`). */ | |
| 134 | function costOf(raw: unknown): number { | |
| 135 | const r = record(raw); | |
| 136 | if (r === undefined) return 0; | |
| 137 | const direct = r["totalCostUsd"]; | |
| 138 | if (typeof direct === "number") return Number.isFinite(direct) && direct >= 0 ? direct : 0; | |
| 139 | for (const v of Object.values(r)) { | |
| 140 | const inner = record(v)?.["totalCostUsd"]; | |
| 141 | if (typeof inner === "number" && Number.isFinite(inner) && inner >= 0) return inner; | |
| 142 | } | |
| 143 | return 0; | |
| 144 | } | |
| 145 | ||
| 146 | /** What deepsec's runs spent, in dollars: the sum of the run records' `totalCostUsd`. */ | |
| 147 | export function spentIn(runs: readonly unknown[]): number { | |
| 148 | return runs.slice(0, FILES_MAX).reduce<number>((sum, run) => sum + costOf(run), 0); | |
| 149 | } |
src/http/api.ts +2
| @@ -422,6 +422,7 @@ import { buildRoutes } from "./api/builds.ts"; | ||
| 422 | 422 | import { buildShotRoutes } from "./api/build_shots.ts"; |
| 423 | 423 | import { buildCartRoutes } from "./api/build_cart.ts"; |
| 424 | 424 | import { iterationRoutes } from "./api/iterations.ts"; |
| 425 | import { deepsecRoutes } from "./api/deepsec.ts"; | |
| 425 | 426 | import { appNoteRoutes } from "./api/app_notes.ts"; |
| 426 | 427 | import { reportRoutes } from "./api/reports.ts"; |
| 427 | 428 | import { offerHandler, offerStatusHandler } from "./offer_routes.ts"; |
| @@ -4101,6 +4102,7 @@ export function apiRoutes(router: Router<Handler>, isPage: (path: string) => boo | ||
| 4101 | 4102 | buildShotRoutes(router); |
| 4102 | 4103 | buildCartRoutes(router); |
| 4103 | 4104 | iterationRoutes(router); |
| 4105 | deepsecRoutes(router); | |
| 4104 | 4106 | |
| 4105 | 4107 | egressGrantRoutes(router); |
| 4106 | 4108 | reachAskRoutes(router); |
src/http/api/deepsec.ts added +51
| @@ -0,0 +1,51 @@ | ||
| 1 | /** | |
| 2 | * `POST /api/v1/repos/:owner/:name/deepsec` (#15025): file what a deepsec | |
| 3 | * run found as sealed reports under P0 Trust (`checks/deepsec.ts`). The body | |
| 4 | * is deepsec's own records, `{files, runs}`, as `scripts/deepsec-file.ts` | |
| 5 | * reads them from `.deepsec/data/forge/`, with the pull whose check ran it, | |
| 6 | * the commit it read and the tool that read it, each optional. | |
| 7 | * | |
| 8 | * Only someone who writes the repository files here: a report opens in the | |
| 9 | * owner's name, sealed, and a reader must not be able to open one. What the | |
| 10 | * records say is checked in `deepsec_findings.ts`; what this door checks is | |
| 11 | * their shape, the pull and the commit. | |
| 12 | */ | |
| 13 | ||
| 14 | import { fileFindings } from "../../checks/deepsec.ts"; | |
| 15 | import { FILES_MAX } from "../../checks/deepsec_findings.ts"; | |
| 16 | import { getPull } from "../../core/pulls.ts"; | |
| 17 | import { canWrite } from "../../core/repos.ts"; | |
| 18 | import { loadRepo } from "../api.ts"; | |
| 19 | import type { Handler } from "../context.ts"; | |
| 20 | import { apiError, body, forbidden, json, notFound, requireScope } from "../respond.ts"; | |
| 21 | import type { Router } from "../router.ts"; | |
| 22 | ||
| 23 | export function deepsecRoutes(router: Router<Handler>): void { | |
| 24 | router.add("POST", "/api/v1/repos/:owner/:name/deepsec", async (ctx) => { | |
| 25 | const denied = requireScope(ctx, "rw"); | |
| 26 | if (denied) return denied; | |
| 27 | const repo = loadRepo(ctx); | |
| 28 | if (!repo) return notFound(); | |
| 29 | if (!canWrite(ctx.app.db, repo, ctx.auth.user)) return forbidden(); | |
| 30 | const input = await body(ctx.request); | |
| 31 | const files = input["files"]; | |
| 32 | const runs = input["runs"] ?? []; | |
| 33 | if (!Array.isArray(files) || !Array.isArray(runs) || files.length > FILES_MAX || runs.length > FILES_MAX) | |
| 34 | return apiError("invalid", 422, `files and runs are lists of deepsec's records, at most ${FILES_MAX} each`); | |
| 35 | const pull = input["pull"]; | |
| 36 | if (pull !== undefined && (!Number.isSafeInteger(pull) || getPull(ctx.app.db, repo.id, pull as number) === undefined)) | |
| 37 | return apiError("invalid", 422, "pull names no pull in this repository"); | |
| 38 | const commit = input["commit"]; | |
| 39 | if (commit !== undefined && (typeof commit !== "string" || !/^[0-9a-f]{7,64}$/.test(commit))) | |
| 40 | return apiError("invalid", 422, "commit is a hexadecimal object id"); | |
| 41 | const tool = input["tool"]; | |
| 42 | if (tool !== undefined && (typeof tool !== "string" || tool.length > 80)) | |
| 43 | return apiError("invalid", 422, "tool is at most 80 characters"); | |
| 44 | const filed = fileFindings(ctx.app, repo, { files, runs }, { | |
| 45 | ...(pull !== undefined ? { pull: pull as number } : {}), | |
| 46 | ...(commit !== undefined ? { commit } : {}), | |
| 47 | ...(tool !== undefined ? { tool } : {}), | |
| 48 | }); | |
| 49 | return json(filed, 200); | |
| 50 | }); | |
| 51 | } |
test/deepsec-findings.test.ts added +301
| @@ -0,0 +1,301 @@ | ||
| 1 | /** | |
| 2 | * deepsec's findings file as sealed reports (#15025): read from its data | |
| 3 | * directory, each new one a sealed issue under P0 Trust naming its pull, | |
| 4 | * each one already filed matched, what was spent said, and the gate never | |
| 5 | * held by them. | |
| 6 | */ | |
| 7 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 8 | import { mkdtemp, mkdir, rm, symlink, writeFile } from "node:fs/promises"; | |
| 9 | import { tmpdir } from "node:os"; | |
| 10 | import { join } from "node:path"; | |
| 11 | import { createApp, type App } from "../src/app.ts"; | |
| 12 | import { openIssue } from "../src/actions/issues.ts"; | |
| 13 | import { createRepo } from "../src/actions/repos.ts"; | |
| 14 | import { fileFindings, lineFor, reportFor } from "../src/checks/deepsec.ts"; | |
| 15 | import { FINDINGS_MAX, findingsIn, findingsOf, spentIn } from "../src/checks/deepsec_findings.ts"; | |
| 16 | import { createToken } from "../src/core/tokens.ts"; | |
| 17 | import { serve, type Servers } from "../src/http/server.ts"; | |
| 18 | import { batches, readWorkspace, trimRecord } from "../scripts/deepsec-file.ts"; | |
| 19 | import { BODY_MAX } from "../src/http/limits.ts"; | |
| 20 | import { blockerNumbersFor } from "../src/core/dependencies.ts"; | |
| 21 | import { getIssue, type Issue } from "../src/core/issues.ts"; | |
| 22 | import type { Repo } from "../src/core/repos.ts"; | |
| 23 | import { createUser, getUserByName, type User } from "../src/core/users.ts"; | |
| 24 | ||
| 25 | const finding = (over: Record<string, unknown> = {}) => ({ | |
| 26 | severity: "HIGH", | |
| 27 | title: "Token compared with ==", | |
| 28 | description: "The token is compared with == so timing leaks it. Ask @alice, see #12.", | |
| 29 | lineNumbers: [12, 14], | |
| 30 | recommendation: "Use timingSafeEqual.", | |
| 31 | revalidation: { verdict: "true-positive", reasoning: "Reachable from the API." }, | |
| 32 | ...over, | |
| 33 | }); | |
| 34 | ||
| 35 | async function workspace(files: Record<string, unknown>, runs: Record<string, unknown> = {}): Promise<string> { | |
| 36 | const dir = await mkdtemp(join(tmpdir(), "forge-deepsec-data-")); | |
| 37 | for (const [name, body] of Object.entries(files)) { | |
| 38 | await mkdir(join(dir, "files", name, ".."), { recursive: true }); | |
| 39 | await writeFile(join(dir, "files", name), JSON.stringify(body)); | |
| 40 | } | |
| 41 | await mkdir(join(dir, "runs"), { recursive: true }); | |
| 42 | for (const [name, body] of Object.entries(runs)) await writeFile(join(dir, "runs", name), JSON.stringify(body)); | |
| 43 | return dir; | |
| 44 | } | |
| 45 | ||
| 46 | describe("reading what deepsec wrote", () => { | |
| 47 | test("a record's findings are checked at the boundary, false positives dropped", () => { | |
| 48 | const out = findingsOf({ | |
| 49 | filePath: "src/http/auth.ts", | |
| 50 | findings: [ | |
| 51 | finding(), | |
| 52 | finding({ title: "Rejected", revalidation: { verdict: "false-positive" } }), | |
| 53 | finding({ title: "", severity: "HIGH" }), | |
| 54 | finding({ title: "Odd severity", severity: "SPICY" }), | |
| 55 | finding({ title: "Lowered", revalidation: { verdict: "true-positive", adjustedSeverity: "MEDIUM" } }), | |
| 56 | ], | |
| 57 | }); | |
| 58 | expect(out.map((f) => [f.title, f.severity])).toEqual([ | |
| 59 | ["Token compared with ==", "HIGH"], | |
| 60 | ["Lowered", "MEDIUM"], | |
| 61 | ]); | |
| 62 | expect(out[0]!.lines).toEqual([12, 14]); | |
| 63 | expect(out[0]!.confirmed).toBe(true); | |
| 64 | }); | |
| 65 | ||
| 66 | test("a path that climbs out or is absolute files nothing; control characters are stripped", () => { | |
| 67 | expect(findingsOf({ filePath: "../etc/passwd", findings: [finding()] })).toEqual([]); | |
| 68 | expect(findingsOf({ filePath: "/etc/passwd", findings: [finding()] })).toEqual([]); | |
| 69 | expect(findingsOf({ filePath: "src/a b.ts", findings: [finding()] })).toEqual([]); | |
| 70 | expect(findingsOf("nonsense")).toEqual([]); | |
| 71 | const [f] = findingsOf({ filePath: "src/a.ts", findings: [finding({ title: "Bad\u202Eti\u0007tle" })] }); | |
| 72 | expect(f!.title).toBe("Badtitle"); | |
| 73 | }); | |
| 74 | ||
| 75 | test("the whole directory is read once each by file and title, most severe first, skipping what does not parse", async () => { | |
| 76 | const dir = await workspace({ | |
| 77 | "src/a.ts.json": { filePath: "src/a.ts", findings: [finding({ severity: "MEDIUM", title: "M" }), finding({ severity: "CRITICAL", title: "C" })] }, | |
| 78 | "src/b.ts.json": { filePath: "src/b.ts", findings: [finding({ title: "M" })] }, | |
| 79 | "src/dup.json": { filePath: "src/a.ts", findings: [finding({ severity: "MEDIUM", title: "M" })] }, | |
| 80 | }); | |
| 81 | await writeFile(join(dir, "files", "broken.json"), "{not json"); | |
| 82 | try { | |
| 83 | expect(findingsIn(readWorkspace(dir).files).findings.map((f) => `${f.severity} ${f.file} ${f.title}`)).toEqual([ | |
| 84 | "CRITICAL src/a.ts C", | |
| 85 | "HIGH src/b.ts M", | |
| 86 | "MEDIUM src/a.ts M", | |
| 87 | ]); | |
| 88 | } finally { | |
| 89 | await rm(dir, { recursive: true, force: true }); | |
| 90 | } | |
| 91 | }); | |
| 92 | ||
| 93 | test("what was spent is the sum of the runs' totalCostUsd, at the top or one level down", async () => { | |
| 94 | const dir = await workspace({}, { | |
| 95 | "a.json": { totalCostUsd: 1.25 }, | |
| 96 | "b.json": { stats: { totalCostUsd: 0.5 } }, | |
| 97 | "c.json": { totalCostUsd: -4 }, | |
| 98 | "d.json": { nothing: true }, | |
| 99 | }); | |
| 100 | try { | |
| 101 | expect(spentIn(readWorkspace(dir).runs)).toBeCloseTo(1.75); | |
| 102 | expect(spentIn(readWorkspace(join(dir, "missing")).runs)).toBe(0); | |
| 103 | } finally { | |
| 104 | await rm(dir, { recursive: true, force: true }); | |
| 105 | } | |
| 106 | }); | |
| 107 | ||
| 108 | test("a report quotes the finding without pinging anyone or naming another issue, and names its pull", () => { | |
| 109 | const [f] = findingsOf({ filePath: "src/http/auth.ts", findings: [finding()] }); | |
| 110 | const { title, body } = reportFor(f!, { pull: 77, commit: "9c5ddf488abc", now: new Date("2026-11-21T00:00:00Z") }); | |
| 111 | expect(title).toBe("Token compared with =="); | |
| 112 | expect(body).toContain("`src/http/auth.ts:12,14` · confirmed by revalidation"); | |
| 113 | expect(body).toContain("From the deepsec check on pull #77."); | |
| 114 | expect(body).toContain("on 9c5ddf488, 2026-11-21"); | |
| 115 | expect(body).not.toContain("@alice"); | |
| 116 | expect(body).not.toMatch(/see #12/); | |
| 117 | }); | |
| 118 | }); | |
| 119 | ||
| 120 | describe("a pass is bounded", () => { | |
| 121 | test("one pass reads at most FINDINGS_MAX findings, the most severe, and counts the rest", () => { | |
| 122 | const records = Array.from({ length: 3 }, (_, r) => ({ | |
| 123 | filePath: `src/f${r}.ts`, | |
| 124 | findings: Array.from({ length: 100 }, (_, i) => finding({ title: `T${i}`, severity: r === 2 ? "CRITICAL" : "LOW" })), | |
| 125 | })); | |
| 126 | const { findings, dropped } = findingsIn(records); | |
| 127 | expect(findings).toHaveLength(FINDINGS_MAX); | |
| 128 | expect(findings.every((f) => f.severity === "CRITICAL")).toBe(true); | |
| 129 | expect(dropped).toBe(300 - FINDINGS_MAX); | |
| 130 | expect(lineFor(findings.length, [], [], 0, dropped)).toContain(`200 more past the cap of ${FINDINGS_MAX} not filed`); | |
| 131 | }); | |
| 132 | ||
| 133 | test("the script skips symbolic links, so nothing outside the data directory is read", async () => { | |
| 134 | const outside = await workspace({ "secret.json": { filePath: "src/secret.ts", findings: [finding({ title: "Outside" })] } }); | |
| 135 | const dir = await workspace({ "src/a.ts.json": { filePath: "src/a.ts", findings: [finding()] } }); | |
| 136 | try { | |
| 137 | await symlink(join(outside, "files"), join(dir, "files", "linked")); | |
| 138 | await symlink(join(outside, "files", "secret.json"), join(dir, "files", "secret.json")); | |
| 139 | expect(findingsIn(readWorkspace(dir).files).findings.map((f) => f.title)).toEqual(["Token compared with =="]); | |
| 140 | } finally { | |
| 141 | await rm(dir, { recursive: true, force: true }); | |
| 142 | await rm(outside, { recursive: true, force: true }); | |
| 143 | } | |
| 144 | }); | |
| 145 | ||
| 146 | test("the script trims a record to what the door reads and batches under both caps, runs once", () => { | |
| 147 | const big = "x".repeat(50_000); | |
| 148 | const trimmed = trimRecord({ filePath: "src/a.ts", junk: big, findings: [finding({ description: big, extra: big })] }); | |
| 149 | expect(Object.keys(trimmed)).toEqual(["filePath", "findings"]); | |
| 150 | expect(JSON.stringify(trimmed).length).toBeLessThan(10_000); | |
| 151 | expect(findingsOf(trimmed)[0]!.confirmed).toBe(true); | |
| 152 | ||
| 153 | const files = Array.from({ length: 5 }, (_, r) => trimRecord({ filePath: `src/f${r}.ts`, findings: Array.from({ length: 60 }, (_, i) => finding({ title: `T${i}` })) })); | |
| 154 | const out = batches({ files, runs: [{ totalCostUsd: 1 }] }); | |
| 155 | expect(out).toHaveLength(5); | |
| 156 | expect(out.every((b) => b.files.reduce<number>((n, f) => n + (f as { findings: unknown[] }).findings.length, 0) <= FINDINGS_MAX)).toBe(true); | |
| 157 | expect(out.map((b) => b.runs.length)).toEqual([1, 0, 0, 0, 0]); | |
| 158 | const small = batches({ files: files.slice(0, 1), runs: [] }, 1); | |
| 159 | expect(small).toHaveLength(1); | |
| 160 | expect(batches({ files: [trimRecord({ filePath: "a", findings: [finding()] }), trimRecord({ filePath: "b", findings: [finding()] })], runs: [] }, 100)).toHaveLength(2); | |
| 161 | }); | |
| 162 | }); | |
| 163 | ||
| 164 | describe("filing findings as sealed reports", () => { | |
| 165 | let dataDir: string; | |
| 166 | let app: App; | |
| 167 | let alice: User; | |
| 168 | let repo: Repo; | |
| 169 | let umbrella: Issue; | |
| 170 | let servers: Servers; | |
| 171 | ||
| 172 | beforeAll(async () => { | |
| 173 | dataDir = await mkdtemp(join(tmpdir(), "forge-deepsec-")); | |
| 174 | app = createApp({ dataDir, port: 0 }); | |
| 175 | const made = await createUser(app.db, { | |
| 176 | origin: "seed", | |
| 177 | trusted: true, | |
| 178 | username: "alice", | |
| 179 | email: "[email protected]", | |
| 180 | password: "a-good-passphrase", | |
| 181 | adminIfFirst: true, | |
| 182 | }); | |
| 183 | if (!made.ok) throw new Error(made.error); | |
| 184 | alice = getUserByName(app.db, "alice")!; | |
| 185 | const created = await createRepo(app, alice, { name: "engine", visibility: "public" }); | |
| 186 | if (!created.ok) throw new Error(String(created.error)); | |
| 187 | repo = created.value; | |
| 188 | const u = openIssue(app, alice, repo, { title: "P0 Trust: be trusted" }); | |
| 189 | if (!u.ok) throw new Error(String(u.error)); | |
| 190 | umbrella = u.value; | |
| 191 | servers = await serve(app); | |
| 192 | }); | |
| 193 | afterAll(async () => { | |
| 194 | await servers.stop(); | |
| 195 | app.db.close(); | |
| 196 | await rm(dataDir, { recursive: true, force: true }); | |
| 197 | }); | |
| 198 | ||
| 199 | test("a new finding is a sealed report under P0 Trust naming its pull; a rerun matches it; spend is said; the gate is not held", async () => { | |
| 200 | const dir = await workspace( | |
| 201 | { | |
| 202 | "src/http/auth.ts.json": { filePath: "src/http/auth.ts", findings: [finding(), finding({ title: "Second", severity: "MEDIUM" })] }, | |
| 203 | }, | |
| 204 | { "r1.json": { totalCostUsd: 0.54 } }, | |
| 205 | ); | |
| 206 | try { | |
| 207 | const first = fileFindings(app, repo, readWorkspace(dir), { pull: 77 }); | |
| 208 | expect(first.opened).toHaveLength(2); | |
| 209 | expect(first.matched).toEqual([]); | |
| 210 | expect(first.dropped).toBe(0); | |
| 211 | expect(first.line).toContain("2 findings, 2 filed as sealed reports"); | |
| 212 | expect(first.line).toContain("spent $0.54"); | |
| 213 | expect(first.line).toContain("does not block the gate"); | |
| 214 | for (const n of first.opened) { | |
| 215 | const issue = getIssue(app.db, repo.id, n)!; | |
| 216 | expect(issue.sealed).toBe(true); | |
| 217 | expect(issue.body).toContain("pull #77"); | |
| 218 | } | |
| 219 | expect(blockerNumbersFor(app.db, [umbrella.id]).get(umbrella.id)).toEqual(expect.arrayContaining(first.opened)); | |
| 220 | ||
| 221 | const again = fileFindings(app, repo, readWorkspace(dir), { pull: 78 }); | |
| 222 | expect(again.opened).toEqual([]); | |
| 223 | expect([...again.matched].sort()).toEqual([...first.opened].sort()); | |
| 224 | expect(again.line).toContain("0 filed as sealed reports, 2 already filed"); | |
| 225 | } finally { | |
| 226 | await rm(dir, { recursive: true, force: true }); | |
| 227 | } | |
| 228 | }); | |
| 229 | ||
| 230 | test("the same title in another file is its own report", async () => { | |
| 231 | const dir = await workspace({ | |
| 232 | "src/http/other.ts.json": { filePath: "src/http/other.ts", findings: [finding()] }, | |
| 233 | }); | |
| 234 | try { | |
| 235 | const filed = fileFindings(app, repo, readWorkspace(dir)); | |
| 236 | expect(filed.opened).toHaveLength(1); | |
| 237 | expect(getIssue(app.db, repo.id, filed.opened[0]!)!.body).not.toContain("From the deepsec check on pull"); | |
| 238 | } finally { | |
| 239 | await rm(dir, { recursive: true, force: true }); | |
| 240 | } | |
| 241 | }); | |
| 242 | ||
| 243 | test("a report for src/a.ts is not matched by one for src/a.tsx, nor by an issue that only mentions it", async () => { | |
| 244 | const tsx = fileFindings(app, repo, { files: [{ filePath: "src/m.tsx", findings: [finding({ title: "Loose" })] }], runs: [] }); | |
| 245 | expect(tsx.opened).toHaveLength(1); | |
| 246 | const mention = openIssue(app, alice, repo, { | |
| 247 | title: "Loose", | |
| 248 | body: "### HIGH: Loose\n\n`src/m.ts` came up\n\nFound by hand", | |
| 249 | }); | |
| 250 | if (!mention.ok) throw new Error(String(mention.error)); | |
| 251 | const ts = fileFindings(app, repo, { files: [{ filePath: "src/m.ts", findings: [finding({ title: "Loose" })] }], runs: [] }); | |
| 252 | expect(ts.matched).toEqual([]); | |
| 253 | expect(ts.opened).toHaveLength(1); | |
| 254 | const again = fileFindings(app, repo, { files: [{ filePath: "src/m.ts", findings: [finding({ title: "Loose", lineNumbers: [] })] }], runs: [] }); | |
| 255 | expect(again.matched).toEqual(ts.opened); | |
| 256 | }); | |
| 257 | ||
| 258 | test("the API files for someone who writes the repository, and refuses a reader and a pull that is not one", async () => { | |
| 259 | const made = await createUser(app.db, { | |
| 260 | origin: "seed", | |
| 261 | trusted: true, | |
| 262 | username: "bob", | |
| 263 | email: "[email protected]", | |
| 264 | password: "a-good-passphrase", | |
| 265 | }); | |
| 266 | if (!made.ok) throw new Error(made.error); | |
| 267 | const bob = getUserByName(app.db, "bob")!; | |
| 268 | const post = (token: string, payload: unknown) => | |
| 269 | fetch(`${servers.url}/api/v1/repos/alice/engine/deepsec`, { | |
| 270 | method: "POST", | |
| 271 | headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, | |
| 272 | body: JSON.stringify(payload), | |
| 273 | }); | |
| 274 | const files = [{ filePath: "src/http/api.ts", findings: [finding({ title: "Over the API" })] }]; | |
| 275 | const writer = createToken(app.db, alice.id, "rw", "rw").token; | |
| 276 | const reader = createToken(app.db, bob.id, "rw", "rw").token; | |
| 277 | ||
| 278 | expect((await post(reader, { files })).status).toBe(403); | |
| 279 | expect((await post(writer, { files, pull: 99_999 })).status).toBe(422); | |
| 280 | expect((await post(writer, { files: "nope" })).status).toBe(422); | |
| 281 | expect((await post(writer, { files, commit: "not hex" })).status).toBe(422); | |
| 282 | expect((await post(writer, { files, tool: "t".repeat(81) })).status).toBe(422); | |
| 283 | const huge = await fetch(`${servers.url}/api/v1/repos/alice/engine/deepsec`, { | |
| 284 | method: "POST", | |
| 285 | headers: { authorization: `Bearer ${writer}`, "content-type": "application/json" }, | |
| 286 | body: JSON.stringify({ files: [{ filePath: "src/x.ts", findings: [finding({ description: "y".repeat(BODY_MAX) })] }] }), | |
| 287 | }); | |
| 288 | expect(huge.status).toBe(413); | |
| 289 | ||
| 290 | const noRuns = await post(writer, { files: [{ filePath: "src/http/norun.ts", findings: [finding({ title: "No runs" })] }] }); | |
| 291 | expect(noRuns.status).toBe(200); | |
| 292 | expect(((await noRuns.json()) as { line: string }).line).toContain("spent $0.00"); | |
| 293 | ||
| 294 | const res = await post(writer, { files, runs: [{ totalCostUsd: 2 }], commit: "9c5ddf488" }); | |
| 295 | expect(res.status).toBe(200); | |
| 296 | const filed = (await res.json()) as { opened: number[]; line: string }; | |
| 297 | expect(filed.opened).toHaveLength(1); | |
| 298 | expect(filed.line).toContain("spent $2.00"); | |
| 299 | expect(getIssue(app.db, repo.id, filed.opened[0]!)!.sealed).toBe(true); | |
| 300 | }); | |
| 301 | }); |
new above
offline
statSyncfollows symlinks. UselstatSyncand skip links so nothing outside the directory is read.