Every MCP tool answers hostile arguments with a named refusal, never a 500: a fuzz test walks tools/list #11026

New test/fuzz-mcp.test.ts sends hostile arguments to every MCP tool and checks that each one answers with a named refusal rather than a 500. It boots one forge and reads tools/list as a signed-in non-admin (167 tools, report included). It runs in about 6 s here and found no tool that crashes, so no handler under src/http changed.

What each tool gets: a set of cases built from its inputSchema, seeded per tool from FUZZ_SEED so any failure can be rerun:

• no arguments, or arguments that aren't an object (undefined, "not-an-object", [1,2], null);
• every field the wrong type, every string at 10 kB, ../ paths in every field;
• unknown keys, including an own __proto__ ({"__proto__":{"admin":true,...}}) and constructor;
• -1e308 and other huge or negative numbers in every numeric field;
• four random cases per tool, mixing NUL bytes, SQL quotes, lone surrogates, objects nested 40 deep, 2000-item arrays, and real names (alice/open, alice/secret) so some cases get past "not found" to the checks behind it.

What counts as passing: an HTTP 200 carrying either a JSON-RPC result whose text is non-empty when isError is set, or a JSON-RPC error with a message, within 1 s. Anything else, including a throw on the test side, is reported per tool and case.

Anonymous: every tool name gets the 401 that names report, and only report answers JSON-RPC. report then gets the same hostile cases.

Admin-only tools aren't listed to a member, so they are only checked as the refusal a member gets when naming one. Their argument checks behind the admin scope are not fuzzed; the file's header says so.

Things to look at:

• Token rotation: the member's token is swapped for a fresh one every 400 calls. Each token is limited to 600 requests a minute, and the first run hit 429s after that many calls. That limiter has its own tests; this one is about argument checks.
• The test does catch crashes: with a throw temporarily put in pins, every pins case showed up as HTTP 500.
• Side effects: none reach outside the throwaway forge in $TMPDIR; it has no runner, no cloud account and no egress.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun test test/mcp-size.test.ts pass. The 19 files scripts/fast.ts named also pass: 233 tests, 0 failures. The keep_plan call could not be made: the forge's /mcp answered 404 from this sandbox, so the issue has no plan on it.

To try it: bun test test/fuzz-mcp.test.ts, or rerun with FUZZ_SEED=<n> for other cases.

🤖 Generated with Claude Code

Closes #11021

New test/fuzz-mcp.test.ts sends hostile arguments to every MCP tool and checks that each one answers with a named refusal rather than a 500. It boots one forge and reads tools/list as a signed-in non-admin (167 tools, report included). It runs in about 6 s here and found no tool that crashes, so no handler under src/http changed.

What each tool gets: a set of cases built from its inputSchema, seeded per tool from FUZZ_SEED so any failure can be rerun:

• no arguments, or arguments that aren't an object (undefined, "not-an-object", [1,2], null);
• every field the wrong type, every string at 10 kB, ../ paths in every field;
• unknown keys, including an own __proto__ ({"__proto__":{"admin":true,...}}) and constructor;
• -1e308 and other huge or negative numbers in every numeric field;
• four random cases per tool, mixing NUL bytes, SQL quotes, lone surrogates, objects nested 40 deep, 2000-item arrays, and real names (alice/open, alice/secret) so some cases get past "not found" to the checks behind it.

What counts as passing: an HTTP 200 carrying either a JSON-RPC result whose text is non-empty when isError is set, or a JSON-RPC error with a message, within 1 s. Anything else, including a throw on the test side, is reported per tool and case.

Anonymous: every tool name gets the 401 that names report, and only report answers JSON-RPC. report then gets the same hostile cases.

Admin-only tools aren't listed to a member, so they are only checked as the refusal a member gets when naming one. Their argument checks behind the admin scope are not fuzzed; the file's header says so.

Things to look at:

• Token rotation: the member's token is swapped for a fresh one every 400 calls. Each token is limited to 600 requests a minute, and the first run hit 429s after that many calls. That limiter has its own tests; this one is about argument checks.
• The test does catch crashes: with a throw temporarily put in pins, every pins case showed up as HTTP 500.
• Side effects: none reach outside the throwaway forge in $TMPDIR; it has no runner, no cloud account and no egress.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun test test/mcp-size.test.ts pass. The 19 files scripts/fast.ts named also pass: 233 tests, 0 failures. The keep_plan call could not be made: the forge's /mcp answered 404 from this sandbox, so the issue has no plan on it.

To try it: bun test test/fuzz-mcp.test.ts, or rerun with FUZZ_SEED=<n> for other cases.

🤖 Generated with Claude Code

Every MCP tool is now called with seeded hostile arguments built from its own inputSchema, and must answer with a result or a named JSON-RPC refusal within a second. A 500, a throw or a hang fails the test. This includes both tools that need no token, report and start, which are fuzzed as an anonymous caller.

Changes since review:

• Merged main: this brings in start (#10916), which landed after the branch was cut. Before the merge, TOKEN_FREE_HANDLERS on this branch held only report.
• Anonymous sweep over every tool name: the test calls everything in toolNames(). Every name answers with the 401 that points to report, except the token-free ones. The test checks that the names that get through are exactly the keys of TOKEN_FREE_HANDLERS, which must be ["report", "start"].
• Token-free tools fuzzed from the live list: each one's schema is taken from tools/list rather than a hardcoded copy. A new token-free tool is therefore fuzzed with no edit to the test, and the ["report", "start"] assertion fails loudly until someone acknowledges it.
• Refusals leave nothing behind:

- Whenever a report or start call is refused, the counts of rows in user and repo must be unchanged. - Any start call whose name is missing, not a string, blank, longer than 40 characters, or has no letter or digit must be refused.

• Static import: the dynamic import() of REPORT_TOOL is gone; TOKEN_FREE_HANDLERS and toolNames are imported statically.

start is limited to 3 starts per source per hour, and all test calls come from one source. So after a few successful starts, the rest are refused as rate_limited, which still counts as a named refusal. The checks that a refused call creates nothing, and that a hostile name is refused, hold either way.

To try it: bun test test/fuzz-mcp.test.ts (3 pass, 487 expects). To rerun a failing seed: FUZZ_SEED=<n> bun test test/fuzz-mcp.test.ts. bun scripts/gate.ts and bun x tsc --noEmit are clean.

🤖 Generated with Claude Code

Left #11021 open: 1 unticked item remain (The file runs under 10 s.).