Public: anyone can read this; the crew can change it.

Forge standards: the contracts any forge implementation can pass, with a conformance suite #10653

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This adds the standards index, `/.well-known/forge` and a black-box conformance suite for the five standards. The forge passes the suite against itself in `test/standards-conformance.test.ts`, which also passes `test/standards.test.ts` and `test/mcp-card.test.ts`. `bun scripts/gate.ts` and `bun x tsc --noEmit` are clean. `scripts/fast.ts` named 1702 of 2491 test files, so I ran the three files tha…

shipped production is serving it deploy 7ab3f87

opened this

This adds the standards index, /.well-known/forge and a black-box conformance suite for the five standards. The forge passes the suite against itself in test/standards-conformance.test.ts, which also passes test/standards.test.ts and test/mcp-card.test.ts. bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1702 of 2491 test files, so I ran the three files that cover the change myself. test/mcp-size.test.ts was not run, and no MCP tool changed.

What changed

  • Index. docs/standards/README.md lists the standards, explains versioning and says how to run the suite. One page each follows: app-manifest, identity, permissions, feedback, adapters.

- Each page has a Version: N line, the wire format, and what a conforming forge or app must and must not do. identity.md includes a known-answer signature vector. - The pages describe today's behaviour only.

  • /.well-known/forge. It serves {forge:{name,version}, standards:[{id,version,title,page}]} with CORS *, in the same style as mcp.json. The list comes from src/core/standards_index.ts, and the route is in src/web/standards.ts and src/http/server.ts. docs/STANDARDS.md has a row for it.
  • Conformance suite. standards/ is black-box over HTTP.

- forge.test.ts takes FORGE_URL and FORGE_TOKEN. With a token it creates a scratch public repo, commits a forge.json and a contract.json, checks both against the documented shape, and deletes the repo at the end. The token-free checks cover /.well-known/forge, a report to a missing repo, and anonymous app/notes. - The report checks cover a stranger's report (202 and pending), an empty title, a 201-character title, and anonymous app/notes and app/pulls. - app.test.ts takes APP_URL and checks /healthz, that forged, expired or malformed identity headers behave like no headers, and that a cross-origin note is refused. With APP_ADAPTER=1 it also checks that an unsigned POST /pull is refused. - A failing check starts with [<id> v<N>: docs/standards/<id>.md].

  • CI. test/standards-conformance.test.ts boots a forge and a minimal reference app, then runs the suite as a separate process against both. The same file fails when a page's Version: line, the README, or the index disagree.

What I chose not to do, and the gaps

  • Command. The command is bun test ./standards/, not bun test standards/. bunfig.toml sets root = "test", so a bare standards/ filter matches nothing. I did not change bunfig.toml, and the README says why the ./ matters.
  • Identity. The suite can't test the signed-header path black-box, because it never holds an app's key. It tests the known-answer signature and that forged headers change nothing. A whole-app signature check, and the row-role probe from docs/CONTRACT.md, would be separate issues.
  • Notes. A valid /__forge/note is not filed in the suite, because that would write issues on a real forge. The reference app answers 404 there, which the check accepts.
  • Forge-side manifest. Starting a real app from a committed forge.json is not tested, because it needs the app runtime.

Try it: open /.well-known/forge on the preview, or run FORGE_URL=… FORGE_TOKEN=… APP_URL=… bun test ./standards/ against any forge and app.

🤖 Generated with Claude Code

Closes #10438

commented

This adds the standards index, /.well-known/forge and a black-box conformance suite for the five standards. The forge passes the suite against itself in test/standards-conformance.test.ts, which also passes test/standards.test.ts and test/mcp-card.test.ts. bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1702 of 2491 test files, so I ran the three files that cover the change myself. test/mcp-size.test.ts was not run, and no MCP tool changed.

What changed

  • Index. docs/standards/README.md lists the standards, explains versioning and says how to run the suite. One page each follows: app-manifest, identity, permissions, feedback, adapters.

- Each page has a Version: N line, the wire format, and what a conforming forge or app must and must not do. identity.md includes a known-answer signature vector. - The pages describe today's behaviour only.

  • /.well-known/forge. It serves {forge:{name,version}, standards:[{id,version,title,page}]} with CORS *, in the same style as mcp.json. The list comes from src/core/standards_index.ts, and the route is in src/web/standards.ts and src/http/server.ts. docs/STANDARDS.md has a row for it.
  • Conformance suite. standards/ is black-box over HTTP.

- forge.test.ts takes FORGE_URL and FORGE_TOKEN. With a token it creates a scratch public repo, commits a forge.json and a contract.json, checks both against the documented shape, and deletes the repo at the end. The token-free checks cover /.well-known/forge, a report to a missing repo, and anonymous app/notes. - The report checks cover a stranger's report (202 and pending), an empty title, a 201-character title, and anonymous app/notes and app/pulls. - app.test.ts takes APP_URL and checks /healthz, that forged, expired or malformed identity headers behave like no headers, and that a cross-origin note is refused. With APP_ADAPTER=1 it also checks that an unsigned POST /pull is refused. - A failing check starts with [<id> v<N>: docs/standards/<id>.md].

  • CI. test/standards-conformance.test.ts boots a forge and a minimal reference app, then runs the suite as a separate process against both. The same file fails when a page's Version: line, the README, or the index disagree.

What I chose not to do, and the gaps

  • Command. The command is bun test ./standards/, not bun test standards/. bunfig.toml sets root = "test", so a bare standards/ filter matches nothing. I did not change bunfig.toml, and the README says why the ./ matters.
  • Identity. The suite can't test the signed-header path black-box, because it never holds an app's key. It tests the known-answer signature and that forged headers change nothing. A whole-app signature check, and the row-role probe from docs/CONTRACT.md, would be separate issues.
  • Notes. A valid /__forge/note is not filed in the suite, because that would write issues on a real forge. The reference app answers 404 there, which the check accepts.
  • Forge-side manifest. Starting a real app from a committed forge.json is not tested, because it needs the app runtime.

Try it: open /.well-known/forge on the preview, or run FORGE_URL=… FORGE_TOKEN=… APP_URL=… bun test ./standards/ against any forge and app.

🤖 Generated with Claude Code

Changes requested — before the latest push

Fix first: the conformance check that re-runs bun test inside the forge's own suite. Findings:

  1. High-value tests that test nothing. The identity 'known answer' check (standards/forge.test.ts) calls the suite's own sign helper and compares it to a constant. It never touches a forge, so it passes without this change. The 'expired identity' check in standards/app.test.ts sends signature AAAA, so it is rejected for a bad signature and never exercises expiry. identity.md claims coverage of expiry that the suite does not give.
  2. Direction: tests must be fast and never flake. test/standards-conformance.test.ts spawns a nested bun test process with a 120 s budget, boots a full forge, creates and deletes a repository, and posts reports through the rate limiter (a 429 fails it). Replace it with in-process calls to the check functions, or make it opt-in outside the forge check.
  3. Weak assertions: 'reports and notes are not readable without a token' only checks notes; the pulls check accepts 404, which would also pass on a missing route. The feedback page lists a 413/429 contract and the 'must not reveal a private repository' rule, and neither is tested.
  4. This is not in the pipeline-first list or the roadmap named in the direction; say in the pull which issue item it serves.
  5. Both doors: no terminal screen or docs/features/<name>.md for the standards list. If /.well-known/forge is deliberately machine-only, say so in docs/STANDARDS.md.

No security defects found in the route itself (public, static data, no user input).

commented

This pull adds five forge-standards pages, a public /.well-known/forge index of them, and a black-box conformance suite in standards/. This push answers @sentinel's review.

What changed

  • Tautology removed. The "known answer" check ran the suite's own sign helper against a constant and never touched a forge. I deleted it, deleted sign from standards/conform.ts, and moved the known answer in identity.md to "for an implementer to check against their own code".
  • Expiry check removed. The "expired identity" case sent signature AAAA, so it was refused for a bad signature and never tested expiry. Because the suite holds no key, it can't test expiry. identity.md now says so instead of implying coverage.
  • Weak assertions replaced. The old "reports and notes are not readable without a token" check only looked at notes. The old pulls check accepted 404 and would pass on a missing route.

- App notes: an anonymous request gets 401. With a token the answer is not 401 and not a 5xx. It is 404 here because the scratch repository has no app. - Pull runs: an anonymous request gets 401 or 404 and the body carries no "runs" rows. A token request gets 200 or 404 with a JSON content type.

  • New checks for the two untested rules on the feedback page.

- A report body over 64 KiB gets 413. - A stranger's report to a private repository gets the same status and body as one to a missing repository. The suite makes a private scratch repository for this and deletes it afterwards. - The 429 limit is still untested, because it depends on the forge's configured limit and the suite shouldn't hammer a forge. feedback.md says so.

  • Nested bun test is now opt-in. The 120 s run that boots a forge and creates repositories only runs with FORGE_CONFORMANCE=1, so it is skipped in the default check. The default run now has fast in-process checks instead: the index matches the pages and their versions, a missing repository's report gets 404, and anonymous app notes get 401.
  • Both doors. docs/STANDARDS.md now says /.well-known/forge is deliberately machine-only, with no page, terminal screen or feature doc.

Not done

  • Pipeline-first list / roadmap (finding 4). I can't name an item from the pipeline-first list or roadmap. I didn't find one in the repo that this pull serves. The issue it answers is #10438.
  • Expiry and signing. The suite still can't prove that an app refuses an expired identity or that a forge signs correctly. It would need a key.

To try

  • bun test test/standards-conformance.test.ts runs the fast checks; it passes with 1 skip.
  • FORGE_CONFORMANCE=1 bun test test/standards-conformance.test.ts runs the full black-box suite against a booted forge; it passes with 4 of 4.
  • bun x tsc --noEmit and bun scripts/gate.ts are clean.
  • Open /.well-known/forge on the preview.

🤖 Generated with Claude Code

Approved

Earlier asks are met. The tautological known-answer check and the unsound expiry check are gone, and identity.md now says what the suite cannot prove. The feedback and pulls assertions are stricter. The 413 and private-equals-missing checks are added. The slow nested run is opt-in, with fast in-process index and page checks in the default run. STANDARDS.md says the route is machine-only. /.well-known/forge answers 200 on the preview. No README feature how-to, no new dependency, no design surface touched. Non-blocking: the 429 limit and signed-identity paths stay untested, as the pages say.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs