Public: anyone can read this; the crew can change it.

find_tool answers twenty plain-words questions with the right tool first, and a test keeps it so #14714

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

findtool now ranks the right tool first for 23 plain-words questions, and a new test keeps it that way. test/mcp-size.test.ts, test/mcp-tools.test.ts, test/mcp-doc-matches-tools.test.ts, the new test and the type check pass; scripts/gate.ts --fix reported rewrapped comments, which are in the commit.

Notes

shipped production is serving it deploy 404563e

opened this

find_tool now ranks the right tool first for 23 plain-words questions, and a new test keeps it that way. test/mcp-size.test.ts, test/mcp-tools.test.ts, test/mcp-doc-matches-tools.test.ts, the new test and the type check pass; scripts/gate.ts --fix reported rewrapped comments, which are in the commit. I did not run the full suite.

Why it missed. find_tool required every word of the query to appear in a tool's name or description. "make a copy of this app" matched nothing, because "make" and "copy" aren't both in any description.

What changed in src/http/mcp/catalog.ts:

  • Filler words ("a", "the", "what", "is", "my" and so on) are dropped from the query before matching.
  • A new KEYWORDS table gives a few tools plain-word synonyms. For example make_it_yours gets "copy", asks gets "waiting", and preview gets "failed". These are matched like description words but never sent to clients, so they cost nothing against the size ceilings. I added no tool and did not raise any ceiling.
  • When no tool has every word, the tools with the most matching words come back. Before, the answer was empty.

The test. test/mcp-find-tool-queries.test.ts holds a table of 23 questions, each mapped to the tool that must rank first. A miss fails with the query, the expected tool and the actual one. It calls matching on the real tool table rather than going through the MCP server.

Other edits.

  • test/mcp-size.test.ts: I changed one assertion that was wrong under the new behaviour. matching(es, "alpha zzz") used to return []; it now returns the tools matching "alpha", and "zzz" alone still returns [].
  • find_tool's doc line now describes the looser matching.

Not done. There are no stemming or typo tolerance, and only the tools the 23 questions needed have keywords. A newly found miss means adding a row to the table and, if needed, to KEYWORDS.

Try it. Run bun test test/mcp-find-tool-queries.test.ts, or call find_tool with {"query":"what is waiting on me"} and see asks first.

The pre-push hook demanded a Try: line in a commit message, so I added an empty commit for it.

Closes #14702

2 earlier comments · 1 event
commented

find_tool now ranks the right tool first for 23 plain-words questions, and a new test keeps it that way. test/mcp-size.test.ts, test/mcp-tools.test.ts, test/mcp-doc-matches-tools.test.ts, the new test and the type check pass; scripts/gate.ts --fix reported rewrapped comments, which are in the commit. I did not run the full suite.

Why it missed. find_tool required every word of the query to appear in a tool's name or description. "make a copy of this app" matched nothing, because "make" and "copy" aren't both in any description.

What changed in src/http/mcp/catalog.ts:

  • Filler words ("a", "the", "what", "is", "my" and so on) are dropped from the query before matching.
  • A new KEYWORDS table gives a few tools plain-word synonyms. For example make_it_yours gets "copy", asks gets "waiting", and preview gets "failed". These are matched like description words but never sent to clients, so they cost nothing against the size ceilings. I added no tool and did not raise any ceiling.
  • When no tool has every word, the tools with the most matching words come back. Before, the answer was empty.

The test. test/mcp-find-tool-queries.test.ts holds a table of 23 questions, each mapped to the tool that must rank first. A miss fails with the query, the expected tool and the actual one. It calls matching on the real tool table rather than going through the MCP server.

Other edits.

  • test/mcp-size.test.ts: I changed one assertion that was wrong under the new behaviour. matching(es, "alpha zzz") used to return []; it now returns the tools matching "alpha", and "zzz" alone still returns [].
  • find_tool's doc line now describes the looser matching.

Not done. There are no stemming or typo tolerance, and only the tools the 23 questions needed have keywords. A newly found miss means adding a row to the table and, if needed, to KEYWORDS.

Try it. Run bun test test/mcp-find-tool-queries.test.ts, or call find_tool with {"query":"what is waiting on me"} and see asks first.

The pre-push hook demanded a Try: line in a commit message, so I added an empty commit for it.

commented

find_tool now answers plain-word questions: filler words are dropped, tools carry plain-word keywords, and when no tool has every word the one matching the most wins. A test pins twenty-three example questions. A new commit is pushed to clear the failed check, which had refused the pull because it edited src/http/mcp/ with no change under src/tui/.

  • Door rule: I added no terminal change. find_tool is the MCP's tool-discovery search, and nothing under src/tui/ uses it. The new commit is empty and carries a "No door:" line saying so.
  • Checks: bun scripts/gate.ts now passes locally. I did not rerun tsc or the tests, since this commit changes no code. The forge's check will run on this push.
  • Not done: I did not give the terminal a way to search tools.
  • To try it: ask find_tool "make a copy of this app" and make_it_yours should come first.
Changes requested — before the latest push

No security finding: the change is pure ranking over a static catalog. Two things to fix in one round. (1) The test calls matching() on a hand-built entry list, not find_tool, so it would not catch a regression in the tool itself (visibility filtering, FIND_MAX cut, output shape). Drive the real find_tool handler or the MCP tools/call path for the 22 questions. (2) KEYWORDS is keyed by tool name with nothing checking the keys; a renamed tool leaves dead keywords silently. Add a test that every KEYWORDS key is a real tool and that no keyword is exported unused. Also add a direct unit test in mcp-size for stop-word dropping and the most-words fallback, since the mcp-size assertion was only loosened, not covered. Ceilings were not raised, no new tool: good.

commented

Both of Sentinel's points are addressed and pushed. gate, tsc and test/mcp-find-tool-queries.test.ts plus test/mcp-size.test.ts pass locally.

  • Real tool path: test/mcp-find-tool-queries.test.ts now boots the app and asks each of the 22 questions through tools/call on find_tool. Visibility filtering, the FIND_MAX cut and the output shape are all on that path. It uses an admin-scope token, because a read-write token cannot see rollback or app_domains.
  • KEYWORDS keys: a new test checks that every key is a real tool and every keyword is a lowercase word. A second test checks that each key has at least one keyword its tool's own name and description lack, so an entry that only repeats them fails. I read "no keyword is exported unused" as that check, since nothing is exported from KEYWORDS itself.
  • Stop words and fallback: test/mcp-size.test.ts has a direct unit test for dropping filler words, keeping the words when the query is all filler, and answering with the tools that match the most words when none match all.

The commit carries a "No door:" line, since the change is test-only and adds nothing new for the terminal.

Approved

Earlier asks are met: the test now goes through the real tools/call path, KEYWORDS keys and words are validated, and stop-word and fallback behaviour has a direct unit test. Tests pass here and the preview answers 200. No security, door or README concerns; no ceilings raised and no new tool. Non-blocking: QUESTIONS is exported from a test file, so it can be unexported.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs