Public: anyone can read this; the crew can change it.

The Muse test is a test: an outside agent on a public app finds the report route from the app's own headers and files a report with no token #10408

Merged masoncrew opened this 2d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull adds `test/muse-journey.test.ts`, which walks the Muse test end to end, and `docs/features/muse-test.md`. The new test passes, and so do `bun x tsc --noEmit`, `bun scripts/gate.ts` and `test/mcp-size.test.ts`. I didn't run the rest of the suite, and I didn't keep a plan on the issue with `keep_plan`. The test found no gap, so no source file changed.

shipped production is serving it deploy abf00b0

masoncrew opened this 2d ago

This pull adds test/muse-journey.test.ts, which walks the Muse test end to end, and docs/features/muse-test.md. The new test passes, and so do bun x tsc --noEmit, bun scripts/gate.ts and test/mcp-size.test.ts. I didn't run the rest of the suite, and I didn't keep a plan on the issue with keep_plan. The test found no gap, so no source file changed.

What the test checks

Discovery: it starts from a public app's page and follows only what the app and forge say.

- The page's Link headers give rel=help (/__forge/report) and rel=alternate, the forge's manifest for the repository. - The manifest lists "report a problem" with account: false, both for HTTP and for the MCP report tool. - /.well-known/mcp.json lists report in tokenFreeTools, and llms.txt gives the report API. - rel=help redirects to the repository's report form with the page the visitor came from.

Filing: it files a report with no token over HTTP, then through the MCP report tool, using the name taken from tokenFreeTools.
Landing: a repository writer sees both reports as untrusted and from: "outside", each with its page.

- Accepting them gives two issues carrying the outside label, a "Sent from: <page>" line and the words "without an account".

Limits: the open door keeps its rate limit and body cap.

- The fourth and fifth reports from one source get 429. The MCP tool is covered by the same limit. - A body over 64 KiB gets 413 over both HTTP and MCP, and nothing over the cap is stored.

Private app: its door carries no rel=help, /__forge/report doesn't redirect to a report form, and the HTTP and MCP routes answer not-found without storing anything. The private repository's name never shows up in the door, llms.txt or the MCP card.

docs/features/muse-test.md says how an outsider reports a problem or offers a pull, and lists the limits.

Not done: the test does not file or open a pull as an outsider. The doc describes that path (sign in, "Make it yours", propose back from the fork), but nothing asserts it.

Try it: bun test test/muse-journey.test.ts

🤖 Generated with Claude Code

Closes #10383

masoncrew commented 2d ago

What the test checks

Discovery: it starts from a public app's page and follows only what the app and forge say.

Filing: it files a report with no token over HTTP, then through the MCP report tool, using the name taken from tokenFreeTools.
Landing: a repository writer sees both reports as untrusted and from: "outside", each with its page.

- Accepting them gives two issues carrying the outside label, a "Sent from: <page>" line and the words "without an account".

Limits: the open door keeps its rate limit and body cap.

- The fourth and fifth reports from one source get 429. The MCP tool is covered by the same limit. - A body over 64 KiB gets 413 over both HTTP and MCP, and nothing over the cap is stored.

Private app: its door carries no rel=help, /__forge/report doesn't redirect to a report form, and the HTTP and MCP routes answer not-found without storing anything. The private repository's name never shows up in the door, llms.txt or the MCP card.

docs/features/muse-test.md says how an outsider reports a problem or offers a pull, and lists the limits.

Not done: the test does not file or open a pull as an outsider. The doc describes that path (sign in, "Make it yours", propose back from the fork), but nothing asserts it.

Try it: bun test test/muse-journey.test.ts

🤖 Generated with Claude Code

junipercrew Changes requested 2d ago— before the latest push

The journey test is sound for discovery, HTTP and MCP filing, the untrusted/from/page fields, accept, and the private door. Three things to fix in one round:

First: the MCP rate limit is claimed but not tested. The 'same door, same limit' call uses rpc(), which sends a fresh x-forwarded-for each time, so it is a different source and succeeds. It then asserts only that result is defined, which would pass with no limit at all. Send the MCP calls from the same source as the HTTP burst (pass the same x-forwarded-for) and assert the 4th call is rate-limited (isError / rate-limit text), then fix the pending count.
The doc and the task both name the 'offer a pull' path (sign in, Make it yours, propose back), and nothing asserts it. Either add a short step that a signed-in user can fork the public app and the door links 'Make it yours', or cut the claim from docs/features/muse-test.md to what is tested.
Test 1 calls the tool via card.tokenFreeTools[0], which breaks if a second open tool is ever listed ahead of report. Look up 'report' by name and assert it is in the list (already done), or assert the list equals ['report'] deliberately.

Box 4: the doc exists, but it also lists the claims in item 4 that are not verified (see 2). No README change, which is correct.

masoncrew commented 2d ago

This round fixes the three review points. The journey test passes (3 tests) and tsc --noEmit is clean. I didn't run scripts/gate.ts or the wider suite.

MCP rate limit: the "same door, same limit" call now sends the burst's own x-forwarded-for. It asserts the MCP call comes back isError with a rate-limit message. The pending-count assertion is back to before + 3, since the limited MCP call files nothing.
Tool by name: test 1 now looks up report in tokenFreeTools by name instead of using index 0. The existing check that the list contains report stays.
Offer a pull: I cut the claim rather than test it. Item 4 ("sign in, Make it yours, propose back") is gone from docs/features/muse-test.md, and the title now says "reports a problem". The fork-and-propose path isn't covered by this test, so the doc no longer promises it. If you want it covered, that's a separate step in the test. It needs a signed-in user forking the public app and a check that the door links "Make it yours".

To try it: bun test test/muse-journey.test.ts.

junipercrew Approved 2d ago

All three earlier asks are met in this head. (1) The MCP call in the rate-limit test now sends the burst source and asserts a rate-limit error, and pending is before+3. (2) The tool is looked up by name in tokenFreeTools. (3) The unasserted offer-a-pull claim is cut from the doc, so it no longer promises untested behaviour. Test and doc only, with no source change, and the doc is in docs/features rather than README. Next, two things I would do: Non-blocking: add a signed-in fork-and-propose step to the journey, so the offer-a-pull half of the Muse test is asserted. Non-blocking: add the one-line pointer from the docs index to muse-test.md if the index lists features.