Public: anyone can read this; the crew can change it.

deepsec findings file as sealed reports linked to their pull, matched against ones already filed #15044

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/15025-deepsec-findings-file-as-sealed-reports → main 4 commits fork 3 threads · 3 unresolved

  • Merged. scout/15025-deepsec-findings-file-as-sealed-reports was merged into main as 2f6f4f8.

Review

Sign in to review this change.
7 files changed +871 −0
docs/features/deepsec-findings.md added +41
@@ -0,0 +1,41 @@
1 ---
2 topic: When something is wrong
3 row: deepsec findings become sealed reports under P0 Trust
4 ---
5 # deepsec findings become sealed reports under P0 Trust
6
7 This page is for people who write a repository and run the deepsec security scanner on it. Each finding deepsec makes becomes one sealed issue under the repository's P0 Trust umbrella. A sealed issue is shown only to writers and the crew until the fix lands. If you file the same findings again, nothing is filed twice.
8
9 ## File a run
10
11 Run deepsec's `process` and `revalidate` steps first. Then hand its data directory to the forge:
12
13 ```sh
14 FORGE_URL=https://your.forge FORGE_TOKEN=<a read-write token> \
15 bun scripts/deepsec-file.ts .deepsec/data/forge --pull 123
16 ```
17
18 - The directory defaults to `.deepsec/data/forge`.
19 - `--repo owner/name` names the repository. Without it the script reads `FORGE_REPO`, then falls back to `trevato/forge`.
20 - `--pull N` names the pull whose check ran deepsec. `--commit <oid>` names the commit it read. `--tool "deepsec 2.3.16 (model)"` names the scanner. You can leave each of them out.
21 - The token must be read-write and belong to someone who writes the repository. Anyone else gets `403`.
22
23 The script sends only the fields the forge reads. It posts them in batches of at most 100 findings, and the runs go with the first batch. It skips symbolic links, so it reads nothing outside the directory. It prints one line per batch, for example `deepsec: 3 findings, 2 filed as sealed reports (#41, #42), 1 already filed (#17); spent $1.20; does not block the gate until its false-positive rate is measured.`
24
25 ## What a report says
26
27 Each report is titled with the finding's title. Its body gives the severity, the file and lines, whether revalidation confirmed it, what deepsec saw, the fix it suggests and what its recheck said. If you named a pull, the body names it too. A report is done when a test reproduces the attack, fails before the fix and passes after, and the matching `docs/THREATS.md` row names that test. If a finding is not exploitable, close its report and give the reason.
28
29 A finding is matched by its title and its file. The file must be the whole path, so `src/a.ts` never matches `src/a.tsx`. Only reports that this feature opened count. If one is still open with both, the run counts the finding as already filed and does not open another.
30
31 ## The API
32
33 `POST /api/v1/repos/:owner/:name/deepsec` takes `{files, runs, pull?, commit?, tool?}`. `files` and `runs` are deepsec's own records, at most 5,000 of each. One request files at most 100 findings, the most severe first. The line says how many more it did not file, so you can send them in another request. The answer is the same line the script prints. A malformed body gets `422` and says which field is wrong.
34
35 ## In the terminal
36
37 Filing is not a terminal screen: a script or a check runs it. The reports it opens are ordinary sealed issues. In the terminal they show in the repository's issues and under the P0 Trust pillar, like every other report.
38
39 ## Not yet
40
41 A finding does not block the gate yet. It will block it once deepsec's false-positive rate has been measured over two weeks of runs. A pull's check step (#15024) and a weekly sweep (#15026) will call the same door. Today you run the script yourself.
scripts/deepsec-file.ts added +183
@@ -0,0 +1,183 @@
1 /**
2 * File what deepsec found as sealed reports (#15025).
3 *
4 * FORGE_URL=… FORGE_TOKEN=… bun scripts/deepsec-file.ts [dir] [--repo owner/name] [--pull N]
5 * [--commit oid] [--tool "deepsec 2.3.16 (model)"]
6 *
7 * Reads deepsec's data directory (default `.deepsec/data/forge`) after
8 * `process` and `revalidate`: every file record with findings, and
9 * `runs/*.json` for what was spent. Posts them to
10 * `POST /api/v1/repos/:owner/:name/deepsec`, which files each new finding
11 * sealed under P0 Trust and matches each one already open. Prints the line
12 * the forge answers for each batch. The token must write the repository.
13 *
14 * It sends only what the door reads (#15044): each record trimmed to the
15 * fields `checks/deepsec_findings.ts` reads, its text capped, and the records
16 * posted in batches of at most `FINDINGS_MAX` findings and `BATCH_BYTES`
17 * bytes, so no batch is past the door's body cap or its findings cap. The runs
18 * go with the first batch only, so what was spent is said once. A symbolic
19 * link in the data directory is skipped: nothing outside it is read.
20 */
21
22 import { lstatSync, readdirSync, readFileSync } from "node:fs";
23 import { join } from "node:path";
24 import { clean, FINDINGS_MAX, spentIn, TEXT_MAX } from "../src/checks/deepsec_findings.ts";
25 import { BODY_MAX } from "../src/http/limits.ts";
26
27 const FILES_MAX = 5_000;
28 const BYTES_MAX = 2_000_000;
29 const DEPTH_MAX = 12;
30 /**
31 * A quarter of the door's body cap: room for the JSON around the records and for multibyte text.
32 */
33 export const BATCH_BYTES = Math.floor(BODY_MAX / 4);
34
35 function fail(message: string): never {
36 console.error(`deepsec-file: ${message}`);
37 process.exit(1);
outdated

statSync follows symlinks. Use lstatSync and skip links so nothing outside the directory is read.

38 }
39
40 /** Every `.json` under `dir`, bounded; `runs/` at the top is left to `runRecords`. */
41 function jsonFiles(dir: string, depth = 0, acc: string[] = []): string[] {
42 if (depth > DEPTH_MAX || acc.length >= FILES_MAX) return acc;
43 let names: string[];
44 try {
45 names = readdirSync(dir).sort();
46 } catch {
47 return acc;
48 }
49 for (const name of names) {
50 if (acc.length >= FILES_MAX) break;
51 if (depth === 0 && name === "runs") continue;
52 const path = join(dir, name);
53 const st = lstatSync(path, { throwIfNoEntry: false });
54 if (st === undefined || st.isSymbolicLink()) continue;
55 if (st.isDirectory()) jsonFiles(path, depth + 1, acc);
56 else if (st.isFile() && name.endsWith(".json") && st.size <= BYTES_MAX) acc.push(path);
57 }
58 return acc;
59 }
60
61 function parse(path: string): unknown {
62 try {
63 return JSON.parse(readFileSync(path, "utf8"));
64 } catch {
65 return undefined;
66 }
67 }
68
69 const hasFindings = (v: unknown): boolean =>
70 typeof v === "object" && v !== null && Array.isArray((v as { findings?: unknown }).findings) && (v as { findings: unknown[] }).findings.length > 0;
71
72 const obj = (v: unknown): Record<string, unknown> =>
73 typeof v === "object" && v !== null && !Array.isArray(v) ? (v as Record<string, unknown>) : {};
74 const text = (v: unknown): string | undefined => (typeof v === "string" ? clean(v, TEXT_MAX) : undefined);
75
76 /**
77 * A recheck as the door reads it: its verdict, reasoning and adjusted severity, or its bare word.
78 */
79 function trimRecheck(v: unknown): unknown {
80 if (typeof v === "string") return clean(v, 200);
81 const r = obj(v);
82 return { verdict: text(r["verdict"]), reasoning: text(r["reasoning"]), adjustedSeverity: text(r["adjustedSeverity"]) };
83 }
84
85 /** One file record cut to the fields the door reads, at most `FINDINGS_MAX` findings. */
86 export function trimRecord(raw: unknown): { filePath: unknown; findings: unknown[] } {
87 const r = obj(raw);
88 const findings = Array.isArray(r["findings"]) ? r["findings"].slice(0, FINDINGS_MAX) : [];
89 return {
90 filePath: typeof r["filePath"] === "string" ? r["filePath"] : r["path"],
91 findings: findings.map((item) => {
92 const f = obj(item);
93 const recheck = f["revalidation"] ?? f["revalidate"];
94 return {
95 severity: text(f["severity"]),
96 title: text(f["title"]),
97 description: text(f["description"]),
98 lineNumbers: f["lineNumbers"] ?? f["lines"],
99 recommendation: text(f["recommendation"] ?? f["fix"]),
100 ...(recheck === undefined ? {} : { revalidation: trimRecheck(recheck) }),
101 };
102 }),
103 };
104 }
105
106 /** deepsec's records under `dir`: file records with findings, trimmed, and what each run spent. */
107 export function readWorkspace(dir: string): { files: ReturnType<typeof trimRecord>[]; runs: unknown[] } {
108 const files = jsonFiles(dir).map(parse).filter(hasFindings).map(trimRecord);
109 const runs = jsonFiles(join(dir, "runs"))
110 .map(parse)
111 .filter((r) => r !== undefined)
112 .map((r) => ({ totalCostUsd: spentIn([r]) }));
113 return { files, runs };
114 }
115
116 /**
117 * The records as bodies the door accepts: each at most `FINDINGS_MAX` findings and `maxBytes` of
118 * records, the runs with the first. A record is never split; one trimmed record fits by itself.
119 */
120 export function batches(
121 records: { files: readonly { findings: readonly unknown[] }[]; runs: readonly unknown[] },
122 maxBytes = BATCH_BYTES,
123 ): { files: unknown[]; runs: unknown[] }[] {
124 const out: { files: unknown[]; runs: unknown[] }[] = [];
125 let files: unknown[] = [];
126 let count = 0;
127 let bytes = 0;
128 for (const rec of records.files) {
129 const size = Buffer.byteLength(JSON.stringify(rec));
130 if (files.length > 0 && (count + rec.findings.length > FINDINGS_MAX || bytes + size > maxBytes)) {
131 out.push({ files, runs: [] });
132 files = [];
133 count = 0;
134 bytes = 0;
135 }
136 files.push(rec);
137 count += rec.findings.length;
138 bytes += size;
139 }
140 if (files.length > 0 || out.length === 0) out.push({ files, runs: [] });
141 out[0] = { files: out[0]!.files, runs: [...records.runs] };
142 return out;
143 }
144
145 function flag(args: string[], name: string): string | undefined {
146 const at = args.indexOf(name);
147 if (at === -1) return undefined;
148 const value = args[at + 1];
149 if (value === undefined) fail(`${name} needs a value`);
150 args.splice(at, 2);
151 return value;
152 }
153
154 if (import.meta.main) {
155 const args = process.argv.slice(2);
156 const repo = flag(args, "--repo") ?? process.env["FORGE_REPO"] ?? "trevato/forge";
157 const pull = flag(args, "--pull");
158 const commit = flag(args, "--commit");
159 const tool = flag(args, "--tool");
160 const dir = args[0] ?? ".deepsec/data/forge";
161 const url = process.env["FORGE_URL"];
162 const token = process.env["FORGE_TOKEN"];
163 if (!url || !token) fail("needs FORGE_URL and FORGE_TOKEN");
164 if (!/^[\w.-]+\/[\w.-]+$/.test(repo)) fail("--repo is owner/name");
165 if (pull !== undefined && !/^\d+$/.test(pull)) fail("--pull is a number");
166 const records = readWorkspace(dir);
167 for (const batch of batches(records)) {
168 const res = await fetch(`${url}/api/v1/repos/${repo}/deepsec`, {
169 method: "POST",
170 headers: { authorization: `Bearer ${token}`, "content-type": "application/json" },
171 body: JSON.stringify({
172 ...batch,
173 ...(pull !== undefined ? { pull: Number(pull) } : {}),
174 ...(commit !== undefined ? { commit } : {}),
175 ...(tool !== undefined ? { tool } : {}),
176 }),
177 signal: AbortSignal.timeout(60_000),
178 });
179 const answer = (await res.json().catch(() => ({}))) as { line?: string; message?: string };
180 if (!res.ok) fail(`${res.status}: ${answer.message ?? "refused"}`);
181 console.log(answer.line ?? "filed");
182 }
183 }
src/checks/deepsec.ts added +144
@@ -0,0 +1,144 @@
1 /**
2 * deepsec's findings become sealed reports (#15025). Each new finding is
3 * filed as a sealed issue under the P0 Trust umbrella, the way the
4 * 2026-11-20 sweep's were (#14796); one that came from a pull's check names
5 * that pull. A finding already filed and still open, the same file and the
6 * same title, is matched instead, so a rerun never files it twice.
7 *
8 * A finding does not block the gate. Until its false-positive rate has been
9 * measured over two weeks of runs, a red verdict on a guess would hold real
10 * work; the line this returns says so. A pass reads at most `FINDINGS_MAX`
11 * findings, the most severe, and says how many it did not file.
12 *
13 * Whatever ran deepsec hands in the records it wrote and gets back one line
14 * for its log: today `scripts/deepsec-file.ts` through
15 * `POST /api/v1/repos/:owner/:name/deepsec` (`http/api/deepsec.ts`); next
16 * the check step on a pull's changed files (#15024) and the weekly sweep
17 * (#15026).
18 */
19
20 import type { App } from "../app.ts";
21 import { linkIssue } from "../actions/dependencies.ts";
22 import { openIssue } from "../actions/issues.ts";
23 import { getIssue, type Issue } from "../core/issues.ts";
24 import { findPillar, pillarsOf } from "../core/pillars.ts";
25 import type { Repo } from "../core/repos.ts";
26 import { getUserById } from "../core/users.ts";
27 import { FINDINGS_MAX, findingsIn, spentIn, type DeepsecFinding } from "./deepsec_findings.ts";
28
29 /** The pillar a sealed report stands under. */
30 export const PILLAR = "P0 Trust";
31
32 export interface Filed {
33 /** Numbers of the reports opened by this pass. */
34 readonly opened: readonly number[];
35 /** Numbers of the open reports a finding matched. */
36 readonly matched: readonly number[];
37 readonly spentUsd: number;
38 /** Findings past the cap, not filed by this pass. */
39 readonly dropped: number;
40 /** One line for the check log. */
41 readonly line: string;
42 }
43
44 export interface Source {
45 /** The pull whose check ran deepsec; absent for the weekly sweep. */
46 readonly pull?: number;
47 /** The commit deepsec read. */
48 readonly commit?: string;
49 /** `deepsec 2.3.16 (claude-opus-5-5)`: what to credit, when known. */
50 readonly tool?: string;
51 readonly now?: Date;
52 }
53
54 /** Quoted text that pings nobody and closes nothing: `@` and `#` lose their reach. */
55 const inert = (text: string): string => text.replace(/@(?=\w)/g, "@\u200b").replace(/#(?=\d)/g, "#\u200b");
56
57 /** The sealed report a finding makes. Pure, so it is tested. */
58 export function reportFor(f: DeepsecFinding, source: Source): { title: string; body: string } {
59 const where = f.lines.length > 0 ? `${f.file}:${f.lines.join(",")}` : f.file;
60 const date = (source.now ?? new Date()).toISOString().slice(0, 10);
61 const commit = source.commit !== undefined && /^[0-9a-f]{7,64}$/.test(source.commit) ? ` on ${source.commit.slice(0, 9)}` : "";
62 const tool = source.tool !== undefined ? inert(source.tool.replace(/[^\w .()/:-]/g, "").slice(0, 80)) : "deepsec";
63 const body = [
64 `### ${f.severity}: ${inert(f.title)}`,
65 "",
66 `\`${where}\` · ${f.confirmed ? "confirmed by revalidation" : "not yet rechecked"}`,
67 "",
68 inert(f.description),
69 ...(f.fix === "" ? [] : ["", `**Fix.** ${inert(f.fix)}`]),
70 ...(f.reasoning === "" ? [] : ["", `**Recheck.** ${inert(f.reasoning)}`]),
71 "",
72 ...(source.pull !== undefined ? [`From the deepsec check on pull #${source.pull}.`, ""] : []),
73 "**Done when:** a test reproduces the attack, fails before the fix and passes after; the matching `docs/THREATS.md` row names that test; a finding judged not exploitable is closed here with the reason.",
74 "",
75 `Found by ${tool}${commit}, ${date}. Sealed: the detail stays with writers and the crew until the fix lands. ` +
76 "It does not block the gate until its false-positive rate has been measured over two weeks.",
77 ].join("\n");
78 return { title: f.title, body };
79 }
80
81 /**
82 * The open report this door already filed for this finding: the same title, and the file as the
83 * whole code span `reportFor` writes (`` `src/a.ts` `` or `` `src/a.ts:12` ``, never `src/a.tsx`).
84 * Only a report this door opened counts: sealed, in the owner's name, shaped as `reportFor` writes
85 * it; an issue that merely mentions the file and title is not one.
86 */
87 export function filedFor(app: App, repo: Repo, f: DeepsecFinding): Issue | undefined {
88 const row = app.db
89 .query<{ number: number }, [string, string, string, string, string]>(
90 `SELECT number FROM issue
91 WHERE repo_id = ?1 AND kind = 'issue' AND state = 'open' AND sealed = 1 AND author_id = ?2
92 AND title = ?3 AND substr(body, 1, 4) = '### '
93 AND instr(body, char(10) || 'Found by ') > 0
94 AND (instr(body, ?4) > 0 OR instr(body, ?5) > 0)
95 ORDER BY number LIMIT 1`,
96 )
97 .get(repo.id, repo.ownerId, f.title, `\`${f.file}\``, `\`${f.file}:`);
98 return row === null ? undefined : getIssue(app.db, repo.id, row.number);
outdated

Loose instr match: src/a.ts also matches src/a.tsx, or any open issue that mentions it. Match the whole code span instead.

99 }
100
101 const dollars = (usd: number): string => `$${usd.toFixed(2)}`;
102
103 /** What deepsec wrote, as it wrote it: its file records and its run records, each untrusted. */
104 export interface Records {
105 readonly files: readonly unknown[];
106 readonly runs: readonly unknown[];
107 }
108
109 /** The log line a pass ends with. */
110 export function lineFor(found: number, opened: readonly number[], matched: readonly number[], usd: number, dropped = 0): string {
111 const refs = (ns: readonly number[]) => (ns.length > 0 ? ` (${ns.map((n) => `#${n}`).join(", ")})` : "");
112 const what =
113 found === 0
114 ? "deepsec: no findings"
115 : `deepsec: ${found} finding${found === 1 ? "" : "s"}, ${opened.length} filed as sealed reports${refs(opened)}, ${matched.length} already filed${refs(matched)}`;
116 const left = dropped > 0 ? `; ${dropped} more past the cap of ${FINDINGS_MAX} not filed: send them in another pass` : "";
117 return `${what}${left}; spent ${dollars(usd)}; does not block the gate until its false-positive rate is measured.`;
118 }
119
120 /**
121 * File what deepsec found in `dataDir` against `repo`: each new finding a
122 * sealed report under P0 Trust, opened in the owner's name like the sweep's;
123 * each one already open matched. Returns what it did, and one line saying so.
124 */
125 export function fileFindings(app: App, repo: Repo, records: Records, source: Source = {}): Filed {
126 const { findings, dropped } = findingsIn(records.files);
127 const usd = spentIn(records.runs);
128 const owner = getUserById(app.db, repo.ownerId);
129 const umbrella = findPillar(pillarsOf(app.db, repo).pillars, PILLAR)?.umbrella;
130 const opened: number[] = [];
131 const matched: number[] = [];
132 for (const f of findings) {
133 const known = filedFor(app, repo, f);
134 if (known !== undefined) {
135 if (!matched.includes(known.number)) matched.push(known.number);
136 continue;
137 }
138 const made = openIssue(app, owner, repo, { ...reportFor(f, source), sealed: true });
139 if (!made.ok) continue;
140 opened.push(made.value.number);
141 if (umbrella !== undefined) linkIssue(app, owner, repo, umbrella, made.value);
142 }
143 return { opened, matched, spentUsd: usd, dropped, line: lineFor(findings.length, opened, matched, usd, dropped) };
144 }
src/checks/deepsec_findings.ts added +149
@@ -0,0 +1,149 @@
1 /**
2 * What deepsec found (#15025): the file records it writes under its data
3 * directory (`.deepsec/data/forge/`) after `process` and `revalidate`, and
4 * the run records that say what it spent. Plain data in, plain data out.
5 *
6 * deepsec is an agent that read untrusted code, so everything it wrote is
7 * untrusted text: each record is checked at this boundary, a path must be a
8 * plain repository path, text is stripped of control characters and capped,
9 * and a finding its own recheck called a false positive is dropped. A record
10 * that does not fit is skipped, never fatal: one bad record must not hide
11 * the rest.
12 */
13
14 export type Severity = "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | "BUG";
15
16 export interface DeepsecFinding {
17 /** Repository path, `src/http/respond.ts`. */
18 readonly file: string;
19 readonly title: string;
20 readonly severity: Severity;
21 readonly lines: readonly number[];
22 readonly description: string;
23 readonly fix: string;
24 /** The recheck's verdict when `revalidate` ran on it. */
25 readonly confirmed: boolean;
26 readonly reasoning: string;
27 }
28
29 const SEVERITIES = new Set<string>(["CRITICAL", "HIGH", "MEDIUM", "LOW", "BUG"]);
30 /** A repository path: no `..`, no leading `/`, no spaces or quotes to break out of a code span. */
31 const PATH = /^(?!.*(?:^|\/)\.\.(?:\/|$))[A-Za-z0-9_.@+-]+(?:\/[A-Za-z0-9_.@+-]+)*$/;
32 // eslint-disable-next-line no-control-regex
33 const CONTROL = /[\u0000-\u0008\u000b-\u001f\u007f\u202A-\u202E\u2066-\u2069]/g;
34 const TITLE_MAX = 200;
35 export const TEXT_MAX = 6_000;
36 const PATH_MAX = 300;
37 /** More file records than one pass reads. */
38 export const FILES_MAX = 5_000;
39 /**
40 * More findings than one pass files (#15044): each is an issue and a link, so
41 * 5,000 records of 100 findings must not open 500,000 of them in one request.
42 * The most severe are kept; the line says how many were not filed.
43 */
44 export const FINDINGS_MAX = 100;
45
46 /** Untrusted text made safe to quote: no control or bidi characters, capped. */
47 export function clean(value: unknown, max: number): string {
48 if (typeof value !== "string") return "";
49 const text = value.replace(CONTROL, "").trim();
50 return text.length > max ? `${text.slice(0, max - 1)}…` : text;
51 }
52
53 const record = (v: unknown): Record<string, unknown> | undefined =>
54 typeof v === "object" && v !== null && !Array.isArray(v) ? (v as Record<string, unknown>) : undefined;
55
56 function lineNumbers(v: unknown): number[] {
57 if (!Array.isArray(v)) return [];
58 return v.filter((n): n is number => Number.isInteger(n) && n > 0 && n < 10_000_000).slice(0, 20);
59 }
60
61 /** The recheck's word on a finding: true, false, or undefined when it was not rechecked. */
62 function verdictOf(v: unknown): boolean | undefined {
63 const r = record(v);
64 const word = typeof r?.["verdict"] === "string" ? r["verdict"].toLowerCase() : typeof v === "string" ? v.toLowerCase() : "";
65 if (/false[-_ ]?positive|rejected|not[-_ ]?exploitable/.test(word)) return false;
66 if (/true[-_ ]?positive|confirmed|valid/.test(word)) return true;
67 return undefined;
68 }
69
70 /** One file record's findings, each checked; what fails a check is left out. */
71 export function findingsOf(raw: unknown): DeepsecFinding[] {
72 const file = record(raw);
73 if (file === undefined || !Array.isArray(file["findings"])) return [];
74 const path = typeof file["filePath"] === "string" ? file["filePath"] : file["path"];
75 if (typeof path !== "string") return [];
76 const rel = path.replace(/^\.\//, "");
77 if (rel.length > PATH_MAX || !PATH.test(rel)) return [];
78 const out: DeepsecFinding[] = [];
79 for (const item of file["findings"].slice(0, 100)) {
80 const f = record(item);
81 if (f === undefined) continue;
82 const recheck = f["revalidation"] ?? f["revalidate"];
83 const verdict = verdictOf(recheck);
84 if (verdict === false) continue;
85 const adjusted = record(recheck)?.["adjustedSeverity"];
86 const severity = String(typeof adjusted === "string" ? adjusted : f["severity"] ?? "").toUpperCase();
87 if (!SEVERITIES.has(severity)) continue;
88 const title = clean(f["title"], TITLE_MAX);
89 if (title === "") continue;
90 out.push({
91 file: rel,
92 title,
93 severity: severity as Severity,
94 lines: lineNumbers(f["lineNumbers"] ?? f["lines"]),
95 description: clean(f["description"], TEXT_MAX),
96 fix: clean(f["recommendation"] ?? f["fix"], TEXT_MAX),
97 confirmed: verdict === true,
98 reasoning: clean(record(recheck)?.["reasoning"], TEXT_MAX),
99 });
100 }
101 return out;
102 }
103
104 /** What makes two findings one: the file and the title. */
105 const matchKey = (f: Pick<DeepsecFinding, "file" | "title">): string => `${f.file}\0${f.title}`;
106
107 /** What one pass files: the most severe findings, and how many past the cap were left out. */
outdated

No cap on the total findings returned. 5,000 records times 100 findings is 500,000 issues from one request. Cap it here.

108 export interface Findings {
109 readonly findings: readonly DeepsecFinding[];
110 readonly dropped: number;
111 }
112
113 /**
114 * Every finding in deepsec's file records, once each by file and title, most severe first, at most
115 * {@link FINDINGS_MAX}.
116 */
117 export function findingsIn(records: readonly unknown[]): Findings {
118 const seen = new Set<string>();
119 const out: DeepsecFinding[] = [];
120 for (const raw of records.slice(0, FILES_MAX)) {
121 for (const f of findingsOf(raw)) {
122 const key = matchKey(f);
123 if (seen.has(key)) continue;
124 seen.add(key);
125 out.push(f);
126 }
127 }
128 const rank = ["CRITICAL", "HIGH", "MEDIUM", "LOW", "BUG"];
129 out.sort((a, b) => rank.indexOf(a.severity) - rank.indexOf(b.severity));
130 return { findings: out.slice(0, FINDINGS_MAX), dropped: Math.max(0, out.length - FINDINGS_MAX) };
131 }
132
133 /** A run record's `totalCostUsd`, at its top or one level down (`stats`, `meta`). */
134 function costOf(raw: unknown): number {
135 const r = record(raw);
136 if (r === undefined) return 0;
137 const direct = r["totalCostUsd"];
138 if (typeof direct === "number") return Number.isFinite(direct) && direct >= 0 ? direct : 0;
139 for (const v of Object.values(r)) {
140 const inner = record(v)?.["totalCostUsd"];
141 if (typeof inner === "number" && Number.isFinite(inner) && inner >= 0) return inner;
142 }
143 return 0;
144 }
145
146 /** What deepsec's runs spent, in dollars: the sum of the run records' `totalCostUsd`. */
147 export function spentIn(runs: readonly unknown[]): number {
148 return runs.slice(0, FILES_MAX).reduce<number>((sum, run) => sum + costOf(run), 0);
149 }
src/http/api.ts +2
@@ -422,6 +422,7 @@ import { buildRoutes } from "./api/builds.ts";
422422 import { buildShotRoutes } from "./api/build_shots.ts";
423423 import { buildCartRoutes } from "./api/build_cart.ts";
424424 import { iterationRoutes } from "./api/iterations.ts";
425 import { deepsecRoutes } from "./api/deepsec.ts";
425426 import { appNoteRoutes } from "./api/app_notes.ts";
426427 import { reportRoutes } from "./api/reports.ts";
427428 import { offerHandler, offerStatusHandler } from "./offer_routes.ts";
@@ -4101,6 +4102,7 @@ export function apiRoutes(router: Router<Handler>, isPage: (path: string) => boo
41014102 buildShotRoutes(router);
41024103 buildCartRoutes(router);
41034104 iterationRoutes(router);
4105 deepsecRoutes(router);
41044106
41054107 egressGrantRoutes(router);
41064108 reachAskRoutes(router);
src/http/api/deepsec.ts added +51
@@ -0,0 +1,51 @@
1 /**
2 * `POST /api/v1/repos/:owner/:name/deepsec` (#15025): file what a deepsec
3 * run found as sealed reports under P0 Trust (`checks/deepsec.ts`). The body
4 * is deepsec's own records, `{files, runs}`, as `scripts/deepsec-file.ts`
5 * reads them from `.deepsec/data/forge/`, with the pull whose check ran it,
6 * the commit it read and the tool that read it, each optional.
7 *
8 * Only someone who writes the repository files here: a report opens in the
9 * owner's name, sealed, and a reader must not be able to open one. What the
10 * records say is checked in `deepsec_findings.ts`; what this door checks is
11 * their shape, the pull and the commit.
12 */
13
14 import { fileFindings } from "../../checks/deepsec.ts";
15 import { FILES_MAX } from "../../checks/deepsec_findings.ts";
16 import { getPull } from "../../core/pulls.ts";
17 import { canWrite } from "../../core/repos.ts";
18 import { loadRepo } from "../api.ts";
19 import type { Handler } from "../context.ts";
20 import { apiError, body, forbidden, json, notFound, requireScope } from "../respond.ts";
21 import type { Router } from "../router.ts";
22
23 export function deepsecRoutes(router: Router<Handler>): void {
24 router.add("POST", "/api/v1/repos/:owner/:name/deepsec", async (ctx) => {
25 const denied = requireScope(ctx, "rw");
26 if (denied) return denied;
27 const repo = loadRepo(ctx);
28 if (!repo) return notFound();
29 if (!canWrite(ctx.app.db, repo, ctx.auth.user)) return forbidden();
30 const input = await body(ctx.request);
31 const files = input["files"];
32 const runs = input["runs"] ?? [];
33 if (!Array.isArray(files) || !Array.isArray(runs) || files.length > FILES_MAX || runs.length > FILES_MAX)
34 return apiError("invalid", 422, `files and runs are lists of deepsec's records, at most ${FILES_MAX} each`);
35 const pull = input["pull"];
36 if (pull !== undefined && (!Number.isSafeInteger(pull) || getPull(ctx.app.db, repo.id, pull as number) === undefined))
37 return apiError("invalid", 422, "pull names no pull in this repository");
38 const commit = input["commit"];
39 if (commit !== undefined && (typeof commit !== "string" || !/^[0-9a-f]{7,64}$/.test(commit)))
40 return apiError("invalid", 422, "commit is a hexadecimal object id");
41 const tool = input["tool"];
42 if (tool !== undefined && (typeof tool !== "string" || tool.length > 80))
43 return apiError("invalid", 422, "tool is at most 80 characters");
44 const filed = fileFindings(ctx.app, repo, { files, runs }, {
45 ...(pull !== undefined ? { pull: pull as number } : {}),
46 ...(commit !== undefined ? { commit } : {}),
47 ...(tool !== undefined ? { tool } : {}),
48 });
49 return json(filed, 200);
50 });
51 }
test/deepsec-findings.test.ts added +301
@@ -0,0 +1,301 @@
1 /**
2 * deepsec's findings file as sealed reports (#15025): read from its data
3 * directory, each new one a sealed issue under P0 Trust naming its pull,
4 * each one already filed matched, what was spent said, and the gate never
5 * held by them.
6 */
7 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
8 import { mkdtemp, mkdir, rm, symlink, writeFile } from "node:fs/promises";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { createApp, type App } from "../src/app.ts";
12 import { openIssue } from "../src/actions/issues.ts";
13 import { createRepo } from "../src/actions/repos.ts";
14 import { fileFindings, lineFor, reportFor } from "../src/checks/deepsec.ts";
15 import { FINDINGS_MAX, findingsIn, findingsOf, spentIn } from "../src/checks/deepsec_findings.ts";
16 import { createToken } from "../src/core/tokens.ts";
17 import { serve, type Servers } from "../src/http/server.ts";
18 import { batches, readWorkspace, trimRecord } from "../scripts/deepsec-file.ts";
19 import { BODY_MAX } from "../src/http/limits.ts";
20 import { blockerNumbersFor } from "../src/core/dependencies.ts";
21 import { getIssue, type Issue } from "../src/core/issues.ts";
22 import type { Repo } from "../src/core/repos.ts";
23 import { createUser, getUserByName, type User } from "../src/core/users.ts";
24
25 const finding = (over: Record<string, unknown> = {}) => ({
26 severity: "HIGH",
27 title: "Token compared with ==",
28 description: "The token is compared with == so timing leaks it. Ask @alice, see #12.",
29 lineNumbers: [12, 14],
30 recommendation: "Use timingSafeEqual.",
31 revalidation: { verdict: "true-positive", reasoning: "Reachable from the API." },
32 ...over,
33 });
34
35 async function workspace(files: Record<string, unknown>, runs: Record<string, unknown> = {}): Promise<string> {
36 const dir = await mkdtemp(join(tmpdir(), "forge-deepsec-data-"));
37 for (const [name, body] of Object.entries(files)) {
38 await mkdir(join(dir, "files", name, ".."), { recursive: true });
39 await writeFile(join(dir, "files", name), JSON.stringify(body));
40 }
41 await mkdir(join(dir, "runs"), { recursive: true });
42 for (const [name, body] of Object.entries(runs)) await writeFile(join(dir, "runs", name), JSON.stringify(body));
43 return dir;
44 }
45
46 describe("reading what deepsec wrote", () => {
47 test("a record's findings are checked at the boundary, false positives dropped", () => {
48 const out = findingsOf({
49 filePath: "src/http/auth.ts",
50 findings: [
51 finding(),
52 finding({ title: "Rejected", revalidation: { verdict: "false-positive" } }),
53 finding({ title: "", severity: "HIGH" }),
54 finding({ title: "Odd severity", severity: "SPICY" }),
55 finding({ title: "Lowered", revalidation: { verdict: "true-positive", adjustedSeverity: "MEDIUM" } }),
56 ],
57 });
58 expect(out.map((f) => [f.title, f.severity])).toEqual([
59 ["Token compared with ==", "HIGH"],
60 ["Lowered", "MEDIUM"],
61 ]);
62 expect(out[0]!.lines).toEqual([12, 14]);
63 expect(out[0]!.confirmed).toBe(true);
64 });
65
66 test("a path that climbs out or is absolute files nothing; control characters are stripped", () => {
67 expect(findingsOf({ filePath: "../etc/passwd", findings: [finding()] })).toEqual([]);
68 expect(findingsOf({ filePath: "/etc/passwd", findings: [finding()] })).toEqual([]);
69 expect(findingsOf({ filePath: "src/a b.ts", findings: [finding()] })).toEqual([]);
70 expect(findingsOf("nonsense")).toEqual([]);
71 const [f] = findingsOf({ filePath: "src/a.ts", findings: [finding({ title: "Bad\u202Eti\u0007tle" })] });
72 expect(f!.title).toBe("Badtitle");
73 });
74
75 test("the whole directory is read once each by file and title, most severe first, skipping what does not parse", async () => {
76 const dir = await workspace({
77 "src/a.ts.json": { filePath: "src/a.ts", findings: [finding({ severity: "MEDIUM", title: "M" }), finding({ severity: "CRITICAL", title: "C" })] },
78 "src/b.ts.json": { filePath: "src/b.ts", findings: [finding({ title: "M" })] },
79 "src/dup.json": { filePath: "src/a.ts", findings: [finding({ severity: "MEDIUM", title: "M" })] },
80 });
81 await writeFile(join(dir, "files", "broken.json"), "{not json");
82 try {
83 expect(findingsIn(readWorkspace(dir).files).findings.map((f) => `${f.severity} ${f.file} ${f.title}`)).toEqual([
84 "CRITICAL src/a.ts C",
85 "HIGH src/b.ts M",
86 "MEDIUM src/a.ts M",
87 ]);
88 } finally {
89 await rm(dir, { recursive: true, force: true });
90 }
91 });
92
93 test("what was spent is the sum of the runs' totalCostUsd, at the top or one level down", async () => {
94 const dir = await workspace({}, {
95 "a.json": { totalCostUsd: 1.25 },
96 "b.json": { stats: { totalCostUsd: 0.5 } },
97 "c.json": { totalCostUsd: -4 },
98 "d.json": { nothing: true },
99 });
100 try {
101 expect(spentIn(readWorkspace(dir).runs)).toBeCloseTo(1.75);
102 expect(spentIn(readWorkspace(join(dir, "missing")).runs)).toBe(0);
103 } finally {
104 await rm(dir, { recursive: true, force: true });
105 }
106 });
107
108 test("a report quotes the finding without pinging anyone or naming another issue, and names its pull", () => {
109 const [f] = findingsOf({ filePath: "src/http/auth.ts", findings: [finding()] });
110 const { title, body } = reportFor(f!, { pull: 77, commit: "9c5ddf488abc", now: new Date("2026-11-21T00:00:00Z") });
111 expect(title).toBe("Token compared with ==");
112 expect(body).toContain("`src/http/auth.ts:12,14` · confirmed by revalidation");
113 expect(body).toContain("From the deepsec check on pull #77.");
114 expect(body).toContain("on 9c5ddf488, 2026-11-21");
115 expect(body).not.toContain("@alice");
116 expect(body).not.toMatch(/see #12/);
117 });
118 });
119
120 describe("a pass is bounded", () => {
121 test("one pass reads at most FINDINGS_MAX findings, the most severe, and counts the rest", () => {
122 const records = Array.from({ length: 3 }, (_, r) => ({
123 filePath: `src/f${r}.ts`,
124 findings: Array.from({ length: 100 }, (_, i) => finding({ title: `T${i}`, severity: r === 2 ? "CRITICAL" : "LOW" })),
125 }));
126 const { findings, dropped } = findingsIn(records);
127 expect(findings).toHaveLength(FINDINGS_MAX);
128 expect(findings.every((f) => f.severity === "CRITICAL")).toBe(true);
129 expect(dropped).toBe(300 - FINDINGS_MAX);
130 expect(lineFor(findings.length, [], [], 0, dropped)).toContain(`200 more past the cap of ${FINDINGS_MAX} not filed`);
131 });
132
133 test("the script skips symbolic links, so nothing outside the data directory is read", async () => {
134 const outside = await workspace({ "secret.json": { filePath: "src/secret.ts", findings: [finding({ title: "Outside" })] } });
135 const dir = await workspace({ "src/a.ts.json": { filePath: "src/a.ts", findings: [finding()] } });
136 try {
137 await symlink(join(outside, "files"), join(dir, "files", "linked"));
138 await symlink(join(outside, "files", "secret.json"), join(dir, "files", "secret.json"));
139 expect(findingsIn(readWorkspace(dir).files).findings.map((f) => f.title)).toEqual(["Token compared with =="]);
140 } finally {
141 await rm(dir, { recursive: true, force: true });
142 await rm(outside, { recursive: true, force: true });
143 }
144 });
145
146 test("the script trims a record to what the door reads and batches under both caps, runs once", () => {
147 const big = "x".repeat(50_000);
148 const trimmed = trimRecord({ filePath: "src/a.ts", junk: big, findings: [finding({ description: big, extra: big })] });
149 expect(Object.keys(trimmed)).toEqual(["filePath", "findings"]);
150 expect(JSON.stringify(trimmed).length).toBeLessThan(10_000);
151 expect(findingsOf(trimmed)[0]!.confirmed).toBe(true);
152
153 const files = Array.from({ length: 5 }, (_, r) => trimRecord({ filePath: `src/f${r}.ts`, findings: Array.from({ length: 60 }, (_, i) => finding({ title: `T${i}` })) }));
154 const out = batches({ files, runs: [{ totalCostUsd: 1 }] });
155 expect(out).toHaveLength(5);
156 expect(out.every((b) => b.files.reduce<number>((n, f) => n + (f as { findings: unknown[] }).findings.length, 0) <= FINDINGS_MAX)).toBe(true);
157 expect(out.map((b) => b.runs.length)).toEqual([1, 0, 0, 0, 0]);
158 const small = batches({ files: files.slice(0, 1), runs: [] }, 1);
159 expect(small).toHaveLength(1);
160 expect(batches({ files: [trimRecord({ filePath: "a", findings: [finding()] }), trimRecord({ filePath: "b", findings: [finding()] })], runs: [] }, 100)).toHaveLength(2);
161 });
162 });
163
164 describe("filing findings as sealed reports", () => {
165 let dataDir: string;
166 let app: App;
167 let alice: User;
168 let repo: Repo;
169 let umbrella: Issue;
170 let servers: Servers;
171
172 beforeAll(async () => {
173 dataDir = await mkdtemp(join(tmpdir(), "forge-deepsec-"));
174 app = createApp({ dataDir, port: 0 });
175 const made = await createUser(app.db, {
176 origin: "seed",
177 trusted: true,
178 username: "alice",
179 email: "[email protected]",
180 password: "a-good-passphrase",
181 adminIfFirst: true,
182 });
183 if (!made.ok) throw new Error(made.error);
184 alice = getUserByName(app.db, "alice")!;
185 const created = await createRepo(app, alice, { name: "engine", visibility: "public" });
186 if (!created.ok) throw new Error(String(created.error));
187 repo = created.value;
188 const u = openIssue(app, alice, repo, { title: "P0 Trust: be trusted" });
189 if (!u.ok) throw new Error(String(u.error));
190 umbrella = u.value;
191 servers = await serve(app);
192 });
193 afterAll(async () => {
194 await servers.stop();
195 app.db.close();
196 await rm(dataDir, { recursive: true, force: true });
197 });
198
199 test("a new finding is a sealed report under P0 Trust naming its pull; a rerun matches it; spend is said; the gate is not held", async () => {
200 const dir = await workspace(
201 {
202 "src/http/auth.ts.json": { filePath: "src/http/auth.ts", findings: [finding(), finding({ title: "Second", severity: "MEDIUM" })] },
203 },
204 { "r1.json": { totalCostUsd: 0.54 } },
205 );
206 try {
207 const first = fileFindings(app, repo, readWorkspace(dir), { pull: 77 });
208 expect(first.opened).toHaveLength(2);
209 expect(first.matched).toEqual([]);
210 expect(first.dropped).toBe(0);
211 expect(first.line).toContain("2 findings, 2 filed as sealed reports");
212 expect(first.line).toContain("spent $0.54");
213 expect(first.line).toContain("does not block the gate");
214 for (const n of first.opened) {
215 const issue = getIssue(app.db, repo.id, n)!;
216 expect(issue.sealed).toBe(true);
217 expect(issue.body).toContain("pull #77");
218 }
219 expect(blockerNumbersFor(app.db, [umbrella.id]).get(umbrella.id)).toEqual(expect.arrayContaining(first.opened));
220
221 const again = fileFindings(app, repo, readWorkspace(dir), { pull: 78 });
222 expect(again.opened).toEqual([]);
223 expect([...again.matched].sort()).toEqual([...first.opened].sort());
224 expect(again.line).toContain("0 filed as sealed reports, 2 already filed");
225 } finally {
226 await rm(dir, { recursive: true, force: true });
227 }
228 });
229
230 test("the same title in another file is its own report", async () => {
231 const dir = await workspace({
232 "src/http/other.ts.json": { filePath: "src/http/other.ts", findings: [finding()] },
233 });
234 try {
235 const filed = fileFindings(app, repo, readWorkspace(dir));
236 expect(filed.opened).toHaveLength(1);
237 expect(getIssue(app.db, repo.id, filed.opened[0]!)!.body).not.toContain("From the deepsec check on pull");
238 } finally {
239 await rm(dir, { recursive: true, force: true });
240 }
241 });
242
243 test("a report for src/a.ts is not matched by one for src/a.tsx, nor by an issue that only mentions it", async () => {
244 const tsx = fileFindings(app, repo, { files: [{ filePath: "src/m.tsx", findings: [finding({ title: "Loose" })] }], runs: [] });
245 expect(tsx.opened).toHaveLength(1);
246 const mention = openIssue(app, alice, repo, {
247 title: "Loose",
248 body: "### HIGH: Loose\n\n`src/m.ts` came up\n\nFound by hand",
249 });
250 if (!mention.ok) throw new Error(String(mention.error));
251 const ts = fileFindings(app, repo, { files: [{ filePath: "src/m.ts", findings: [finding({ title: "Loose" })] }], runs: [] });
252 expect(ts.matched).toEqual([]);
253 expect(ts.opened).toHaveLength(1);
254 const again = fileFindings(app, repo, { files: [{ filePath: "src/m.ts", findings: [finding({ title: "Loose", lineNumbers: [] })] }], runs: [] });
255 expect(again.matched).toEqual(ts.opened);
256 });
257
258 test("the API files for someone who writes the repository, and refuses a reader and a pull that is not one", async () => {
259 const made = await createUser(app.db, {
260 origin: "seed",
261 trusted: true,
262 username: "bob",
263 email: "[email protected]",
264 password: "a-good-passphrase",
265 });
266 if (!made.ok) throw new Error(made.error);
267 const bob = getUserByName(app.db, "bob")!;
268 const post = (token: string, payload: unknown) =>
269 fetch(`${servers.url}/api/v1/repos/alice/engine/deepsec`, {
270 method: "POST",
271 headers: { authorization: `Bearer ${token}`, "content-type": "application/json" },
272 body: JSON.stringify(payload),
273 });
274 const files = [{ filePath: "src/http/api.ts", findings: [finding({ title: "Over the API" })] }];
275 const writer = createToken(app.db, alice.id, "rw", "rw").token;
276 const reader = createToken(app.db, bob.id, "rw", "rw").token;
277
278 expect((await post(reader, { files })).status).toBe(403);
279 expect((await post(writer, { files, pull: 99_999 })).status).toBe(422);
280 expect((await post(writer, { files: "nope" })).status).toBe(422);
281 expect((await post(writer, { files, commit: "not hex" })).status).toBe(422);
282 expect((await post(writer, { files, tool: "t".repeat(81) })).status).toBe(422);
283 const huge = await fetch(`${servers.url}/api/v1/repos/alice/engine/deepsec`, {
284 method: "POST",
285 headers: { authorization: `Bearer ${writer}`, "content-type": "application/json" },
286 body: JSON.stringify({ files: [{ filePath: "src/x.ts", findings: [finding({ description: "y".repeat(BODY_MAX) })] }] }),
287 });
288 expect(huge.status).toBe(413);
289
290 const noRuns = await post(writer, { files: [{ filePath: "src/http/norun.ts", findings: [finding({ title: "No runs" })] }] });
291 expect(noRuns.status).toBe(200);
292 expect(((await noRuns.json()) as { line: string }).line).toContain("spent $0.00");
293
294 const res = await post(writer, { files, runs: [{ totalCostUsd: 2 }], commit: "9c5ddf488" });
295 expect(res.status).toBe(200);
296 const filed = (await res.json()) as { opened: number[]; line: string };
297 expect(filed.opened).toHaveLength(1);
298 expect(filed.line).toContain("spent $2.00");
299 expect(getIssue(app.db, repo.id, filed.opened[0]!)!.sealed).toBe(true);
300 });
301 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs