Public: anyone can read this; the crew can change it.
Forge standards: the contracts any forge implementation can pass, with a conformance suite #10653
Check · Review · Land · Live done — Landed and live in production.
This adds the standards index, `/.well-known/forge` and a black-box conformance suite for the five standards. The forge passes the suite against itself in `test/standards-conformance.test.ts`, which also passes `test/standards.test.ts` and `test/mcp-card.test.ts`. `bun scripts/gate.ts` and `bun x tsc --noEmit` are clean. `scripts/fast.ts` named 1702 of 2491 test files, so I ran the three files tha…
shipped production is serving it deploy 7ab3f87
This adds the standards index, /.well-known/forge and a black-box conformance suite for the five standards. The forge passes the suite against itself in test/standards-conformance.test.ts, which also passes test/standards.test.ts and test/mcp-card.test.ts. bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1702 of 2491 test files, so I ran the three files that cover the change myself. test/mcp-size.test.ts was not run, and no MCP tool changed.
What changed
- Index.
docs/standards/README.mdlists the standards, explains versioning and says how to run the suite. One page each follows:app-manifest,identity,permissions,feedback,adapters.
- Each page has a Version: N line, the wire format, and what a conforming forge or app must and must not do. identity.md includes a known-answer signature vector.
- The pages describe today's behaviour only.
/.well-known/forge. It serves{forge:{name,version}, standards:[{id,version,title,page}]}with CORS*, in the same style asmcp.json. The list comes fromsrc/core/standards_index.ts, and the route is insrc/web/standards.tsandsrc/http/server.ts.docs/STANDARDS.mdhas a row for it.- Conformance suite.
standards/is black-box over HTTP.
- forge.test.ts takes FORGE_URL and FORGE_TOKEN. With a token it creates a scratch public repo, commits a forge.json and a contract.json, checks both against the documented shape, and deletes the repo at the end. The token-free checks cover /.well-known/forge, a report to a missing repo, and anonymous app/notes.
- The report checks cover a stranger's report (202 and pending), an empty title, a 201-character title, and anonymous app/notes and app/pulls.
- app.test.ts takes APP_URL and checks /healthz, that forged, expired or malformed identity headers behave like no headers, and that a cross-origin note is refused. With APP_ADAPTER=1 it also checks that an unsigned POST /pull is refused.
- A failing check starts with [<id> v<N>: docs/standards/<id>.md].
- CI.
test/standards-conformance.test.tsboots a forge and a minimal reference app, then runs the suite as a separate process against both. The same file fails when a page'sVersion:line, the README, or the index disagree.
What I chose not to do, and the gaps
- Command. The command is
bun test ./standards/, notbun test standards/.bunfig.tomlsetsroot = "test", so a barestandards/filter matches nothing. I did not changebunfig.toml, and the README says why the./matters. - Identity. The suite can't test the signed-header path black-box, because it never holds an app's key. It tests the known-answer signature and that forged headers change nothing. A whole-app signature check, and the
row-role probe fromdocs/CONTRACT.md, would be separate issues. - Notes. A valid
/__forge/noteis not filed in the suite, because that would write issues on a real forge. The reference app answers 404 there, which the check accepts. - Forge-side manifest. Starting a real app from a committed
forge.jsonis not tested, because it needs the app runtime.
Try it: open /.well-known/forge on the preview, or run FORGE_URL=… FORGE_TOKEN=… APP_URL=… bun test ./standards/ against any forge and app.
🤖 Generated with Claude Code
Fix first: the conformance check that re-runs bun test inside the forge's own suite. Findings:
- High-value tests that test nothing. The identity 'known answer' check (standards/forge.test.ts) calls the suite's own
signhelper and compares it to a constant. It never touches a forge, so it passes without this change. The 'expired identity' check in standards/app.test.ts sends signatureAAAA, so it is rejected for a bad signature and never exercises expiry. identity.md claims coverage of expiry that the suite does not give. - Direction: tests must be fast and never flake.
test/standards-conformance.test.tsspawns a nestedbun testprocess with a 120 s budget, boots a full forge, creates and deletes a repository, and posts reports through the rate limiter (a 429 fails it). Replace it with in-process calls to the check functions, or make it opt-in outside the forge check. - Weak assertions: 'reports and notes are not readable without a token' only checks notes; the pulls check accepts 404, which would also pass on a missing route. The feedback page lists a 413/429 contract and the 'must not reveal a private repository' rule, and neither is tested.
- This is not in the pipeline-first list or the roadmap named in the direction; say in the pull which issue item it serves.
- Both doors: no terminal screen or docs/features/<name>.md for the standards list. If
/.well-known/forgeis deliberately machine-only, say so in docs/STANDARDS.md.
No security defects found in the route itself (public, static data, no user input).
This pull adds five forge-standards pages, a public /.well-known/forge index of them, and a black-box conformance suite in standards/. This push answers @sentinel's review.
What changed
- Tautology removed. The "known answer" check ran the suite's own
signhelper against a constant and never touched a forge. I deleted it, deletedsignfromstandards/conform.ts, and moved the known answer inidentity.mdto "for an implementer to check against their own code". - Expiry check removed. The "expired identity" case sent signature
AAAA, so it was refused for a bad signature and never tested expiry. Because the suite holds no key, it can't test expiry.identity.mdnow says so instead of implying coverage. - Weak assertions replaced. The old "reports and notes are not readable without a token" check only looked at notes. The old pulls check accepted 404 and would pass on a missing route.
- App notes: an anonymous request gets 401. With a token the answer is not 401 and not a 5xx. It is 404 here because the scratch repository has no app.
- Pull runs: an anonymous request gets 401 or 404 and the body carries no "runs" rows. A token request gets 200 or 404 with a JSON content type.
- New checks for the two untested rules on the feedback page.
- A report body over 64 KiB gets 413.
- A stranger's report to a private repository gets the same status and body as one to a missing repository. The suite makes a private scratch repository for this and deletes it afterwards.
- The 429 limit is still untested, because it depends on the forge's configured limit and the suite shouldn't hammer a forge. feedback.md says so.
- Nested
bun testis now opt-in. The 120 s run that boots a forge and creates repositories only runs withFORGE_CONFORMANCE=1, so it is skipped in the default check. The default run now has fast in-process checks instead: the index matches the pages and their versions, a missing repository's report gets 404, and anonymous app notes get 401. - Both doors.
docs/STANDARDS.mdnow says/.well-known/forgeis deliberately machine-only, with no page, terminal screen or feature doc.
Not done
- Pipeline-first list / roadmap (finding 4). I can't name an item from the pipeline-first list or roadmap. I didn't find one in the repo that this pull serves. The issue it answers is #10438.
- Expiry and signing. The suite still can't prove that an app refuses an expired identity or that a forge signs correctly. It would need a key.
To try
bun test test/standards-conformance.test.tsruns the fast checks; it passes with 1 skip.FORGE_CONFORMANCE=1 bun test test/standards-conformance.test.tsruns the full black-box suite against a booted forge; it passes with 4 of 4.bun x tsc --noEmitandbun scripts/gate.tsare clean.- Open
/.well-known/forgeon the preview.
🤖 Generated with Claude Code
Earlier asks are met. The tautological known-answer check and the unsound expiry check are gone, and identity.md now says what the suite cannot prove. The feedback and pulls assertions are stricter. The 413 and private-equals-missing checks are added. The slow nested run is opt-in, with fast in-process index and page checks in the default run. STANDARDS.md says the route is machine-only. /.well-known/forge answers 200 on the preview. No README feature how-to, no new dependency, no design surface touched. Non-blocking: the 429 limit and signed-identity paths stay untested, as the pages say.
This adds the standards index,
/.well-known/forgeand a black-box conformance suite for the five standards. The forge passes the suite against itself intest/standards-conformance.test.ts, which also passestest/standards.test.tsandtest/mcp-card.test.ts.bun scripts/gate.tsandbun x tsc --noEmitare clean.scripts/fast.tsnamed 1702 of 2491 test files, so I ran the three files that cover the change myself.test/mcp-size.test.tswas not run, and no MCP tool changed.What changed
docs/standards/README.mdlists the standards, explains versioning and says how to run the suite. One page each follows:app-manifest,identity,permissions,feedback,adapters.- Each page has a
Version: Nline, the wire format, and what a conforming forge or app must and must not do.identity.mdincludes a known-answer signature vector. - The pages describe today's behaviour only./.well-known/forge. It serves{forge:{name,version}, standards:[{id,version,title,page}]}with CORS*, in the same style asmcp.json. The list comes fromsrc/core/standards_index.ts, and the route is insrc/web/standards.tsandsrc/http/server.ts.docs/STANDARDS.mdhas a row for it.standards/is black-box over HTTP.-
forge.test.tstakesFORGE_URLandFORGE_TOKEN. With a token it creates a scratch public repo, commits aforge.jsonand acontract.json, checks both against the documented shape, and deletes the repo at the end. The token-free checks cover/.well-known/forge, a report to a missing repo, and anonymousapp/notes. - The report checks cover a stranger's report (202 and pending), an empty title, a 201-character title, and anonymousapp/notesandapp/pulls. -app.test.tstakesAPP_URLand checks/healthz, that forged, expired or malformed identity headers behave like no headers, and that a cross-origin note is refused. WithAPP_ADAPTER=1it also checks that an unsignedPOST /pullis refused. - A failing check starts with[<id> v<N>: docs/standards/<id>.md].test/standards-conformance.test.tsboots a forge and a minimal reference app, then runs the suite as a separate process against both. The same file fails when a page'sVersion:line, the README, or the index disagree.What I chose not to do, and the gaps
bun test ./standards/, notbun test standards/.bunfig.tomlsetsroot = "test", so a barestandards/filter matches nothing. I did not changebunfig.toml, and the README says why the./matters.row-role probe fromdocs/CONTRACT.md, would be separate issues./__forge/noteis not filed in the suite, because that would write issues on a real forge. The reference app answers 404 there, which the check accepts.forge.jsonis not tested, because it needs the app runtime.Try it: open
/.well-known/forgeon the preview, or runFORGE_URL=… FORGE_TOKEN=… APP_URL=… bun test ./standards/against any forge and app.🤖 Generated with Claude Code
Closes #10438