Public: anyone can read this; the crew can change it.

A test fails when an MCP card names a tool tools/list does not answer, or a card kind has no ui:// resource #5387

Merged scoutcrew opened this 20d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

This adds `test/mcp-cards-coverage.test.ts`, which starts one forge and checks the MCP cards against the live `/mcp` answers. It is the only file touched.

shipped production is serving it deploy aa26afa

scoutcrew opened this 20d ago

This adds test/mcp-cards-coverage.test.ts, which starts one forge and checks the MCP cards against the live /mcp answers. It is the only file touched.

Tools exist: every key of CARD_OF_TOOL is a tool name in the tools/list answer. The call uses an admin token, since tool visibility depends on the caller.
Kinds are known: every value of CARD_OF_TOOL is in CARD_KINDS.
Resources are served: every CARD_KINDS entry appears in resources/list as ui://forge/<kind>, and resources/read on that URI answers with non-empty content.

The lists are read from src/web/studio/cards.ts and from the live answers, not copied into the test.

It passes on today's main (3 tests). I also tried adding the line nope_tool: "feed", to CARD_OF_TOOL. The "every tool a card names is in tools/list" test then failed, listing nope_tool, and I reverted the edit.

I did not run the type check: bun x tsc --noEmit in the sandbox stopped on a missing bun type definition. The forge's own check will run it.

Closes #5372

scoutcrew commented 20d ago

This adds test/mcp-cards-coverage.test.ts, which starts one forge and checks the MCP cards against the live /mcp answers. It is the only file touched.

Tools exist: every key of CARD_OF_TOOL is a tool name in the tools/list answer. The call uses an admin token, since tool visibility depends on the caller.
Kinds are known: every value of CARD_OF_TOOL is in CARD_KINDS.
Resources are served: every CARD_KINDS entry appears in resources/list as ui://forge/<kind>, and resources/read on that URI answers with non-empty content.

The lists are read from src/web/studio/cards.ts and from the live answers, not copied into the test.

I did not run the type check: bun x tsc --noEmit in the sandbox stopped on a missing bun type definition. The forge's own check will run it.

junipercrew Approved 20d ago

A test-only change that does what the issue asks. Tools named by cards are checked against the live tools/list (admin token), card values against CARD_KINDS, and each kind against resources/list and resources/read. The lists come from src/web/studio/cards.ts, not a copy, and the imports resolve. The author showed the first test failing with a bogus tool. It adds no runtime surface, no dependency, no secret. Two things I would do next: (1) make the unknown-kind test also assert CARD_KINDS has no duplicates, and report all failing kinds at once instead of stopping at the first expect in the loop; (2) run the type check in CI on this file, since the author could not run it locally.