Public: anyone can read this; the crew can change it.

llms.txt is cacheable and role-aware: public by default, richer with a token, with an ETag that changes only when its words do #12334

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

An anonymous /llms.txt read is now public, max-age=300 with a strong ETag, and a signed-in read stays private, no-cache. The ETag already existed. The only header change is for readers without an account.

Notes

shipped production is serving it deploy d73d768

opened this

An anonymous /llms.txt read is now public, max-age=300 with a strong ETag, and a signed-in read stays private, no-cache. The ETag already existed. The only header change is for readers without an account. tsc and the llms and standards tests pass (100 pass, 1 skipped). bun scripts/gate.ts printed nothing in the sandbox (the one sandbox violation was a blocked outbound connection to ::1), so I can't say it passed. Pushed.

  • Headers: llmsTxt in src/web/standards.ts picks the cache header by whether the reader is signed in. Both variants keep Vary: Cookie, Authorization, and If-None-Match still answers 304 with the matching ETag.
  • Stale ETag fix: the kept body was never rebuilt when the tool list changed, so the ETag could not change either. setMcpTools in src/web/llms_tools.ts now bumps a version, and the body cache keys on it.
  • Tests: the new test/llms-etag.test.ts covers three things:

- Two anonymous reads share an ETag and a 304 follows. - The token body is diffed against the public one: it carries extra lines, the public body contains neither the token nor the username, and the ETags differ. - Changing the tool list changes the anonymous ETag.

  • One test changed: test/standards.test.ts asserted the old anonymous header, private, no-cache. That was the old behaviour this issue replaces, so I updated it to public, max-age=300.

I did not narrow the signed-in tool list to what each token's scope may call. Signed-in readers still get the full index, as before, and the second checkbox ("adds the tools that token may call") is met only in the sense that tokens get more than strangers. A per-scope list would need a rule for what a scope may call, which I did not find in this code. Say if you want that as a follow-up.

To try it, run curl -i <origin>/llms.txt, then repeat it with the returned ETag in If-None-Match to get a 304. Then repeat with authorization: Bearer <token> to see the private header and the longer tool list.

The push hook asked for a commit with a Try: line, so there is an empty commit carrying it.

Closes #12322

3 earlier comments · 1 event
commented

An anonymous /llms.txt read is now public, max-age=300 with a strong ETag, and a signed-in read stays private, no-cache. The ETag already existed. The only header change is for readers without an account. tsc and the llms and standards tests pass (100 pass, 1 skipped). bun scripts/gate.ts printed nothing in the sandbox (the one sandbox violation was a blocked outbound connection to ::1), so I can't say it passed. Pushed.

  • Headers: llmsTxt in src/web/standards.ts picks the cache header by whether the reader is signed in. Both variants keep Vary: Cookie, Authorization, and If-None-Match still answers 304 with the matching ETag.
  • Stale ETag fix: the kept body was never rebuilt when the tool list changed, so the ETag could not change either. setMcpTools in src/web/llms_tools.ts now bumps a version, and the body cache keys on it.
  • Tests: the new test/llms-etag.test.ts covers three things:

- Two anonymous reads share an ETag and a 304 follows. - The token body is diffed against the public one: it carries extra lines, the public body contains neither the token nor the username, and the ETags differ. - Changing the tool list changes the anonymous ETag.

  • One test changed: test/standards.test.ts asserted the old anonymous header, private, no-cache. That was the old behaviour this issue replaces, so I updated it to public, max-age=300.

I did not narrow the signed-in tool list to what each token's scope may call. Signed-in readers still get the full index, as before, and the second checkbox ("adds the tools that token may call") is met only in the sense that tokens get more than strangers. A per-scope list would need a rule for what a scope may call, which I did not find in this code. Say if you want that as a follow-up.

To try it, run curl -i <origin>/llms.txt, then repeat it with the returned ETag in If-None-Match to get a 304. Then repeat with authorization: Bearer <token> to see the private header and the longer tool list.

The push hook asked for a commit with a Try: line, so there is an empty commit carrying it.

commented

Both gate failures are fixed and pushed. The gate now prints nothing locally, tsc is clean, and the llms-etag and standards tests pass. It only gave me clean output after I committed, so the earlier runs were checking the old commit.

  • as never: test/llms-etag.test.ts now builds a typed Listed record, with writes: false, instead of casting.
  • Comment length: the two long comment lines in src/web/standards.ts (the one at line 403 and the one I added above the cache header) are rewrapped under 100 columns.

The behaviour is unchanged from the last push.

  • Anonymous read: /llms.txt answers public, max-age=300 with a strong ETag, and If-None-Match gets a 304.
  • Signed-in read: private, no-cache, with the full tool list.
  • Tool-list change: changing the tool list changes the anonymous ETag.

Still not done: the signed-in list is the full index, not narrowed to what each token's scope may call.

Changes requested — before the latest push

Anonymous caching works and is tested; the second box is not met. Fix first: make the token read list only the tools that token may call, and test it. Also: README untouched (good), but docs/features should note llms.txt caching if a how-to exists there. The signed-in body today is the full index, so a read-only token and an rw token see the same tools; the issue asks for the tools that token may call.

commented

The token read of /llms.txt now lists only the tools that token may call, and the body is cached per reader kind. bun x tsc --noEmit is clean and test/llms-etag.test.ts and test/standards.test.ts pass (33 tests). scripts/gate.ts and scripts/fast.ts don't exist in this clone, so I ran neither.

  • Scope rule: callableBy(tool, {scope, isAdmin}) in src/web/llms_tools.ts owns it, mirroring the MCP dispatch in src/http/mcp.ts:

- adminOnly tools need an admin account. - needsAdminScope tools need an admin-scope token. - A write needs rw, unless the tool can only plan or read (dryByDefault, readsWithout, readsWhen), which a read-only token may still call. - A cookie session has no scope and counts as admin scope, as Auth documents.

  • Caching: the kept body in src/web/standards.ts is keyed by origin and reader kind (anon, ro, rw, admin, plus +admin for an admin account) instead of signedIn. The per-reader tool sections in llms_tools.ts are cached the same way and cleared whenever setMcpTools runs, so a changed tool list still bumps the version and changes the ETag.
  • Tests: in test/llms-etag.test.ts, a read-only token lists fewer tools than an rw one, and everything only rw adds is a write or admin tool. A second test covers callableBy directly.
  • Docs: docs/features has no page for llms.txt, so I updated the llms.txt paragraph in docs/MCP.md. It now says the signed-in list is filtered by what the token may call, and describes the ETag and the public 5-minute cache for strangers. The README is untouched.

Not changed: I kept the module-level version counter. The reader-keyed section cache is reset with it, so no separate per-scope version is needed.

A read-only token on an admin account still sees adminOnly tools, because dispatch lets that token call them (the tool, not the token, is admin-gated there). It hides the needsAdminScope ones.

commented

I merged main into the branch, resolved the conflict, and pushed. tsc --noEmit is clean and the llms/standards tests (42) pass. I did not run the full suite; the forge's check does that on push.

The conflict. It was in src/web/llms_tools.ts, inside setMcpTools. I kept both sides:

  • From main: the open tools in TOKEN_FREE_TOOLS (explore, search) are listed with report and start and left out of the "more tools" count.
  • From this pull: the version bump, the list of every tool, and the per-reader section cache that is cleared when the list is set.

Test change. test/llms-tools.test.ts and test/llms-anonymous.test.ts came from main and expected a signed-in reader to see every tool. They built an admin account with an rw token, which this pull now correctly hides the admin-scope tools from, so both failed. Their intent was "everything is listed", so I switched their token to admin scope. The tests were wrong, not the code.

Gate. The gate flagged a 101-column comment in src/web/standards.ts, which I rewrapped. It also wants a Looks: or No look: line in a commit message. I gave No look: on the rewrap commit, since only the text of /llms.txt changes for a signed-in reader. I did not re-run the gate after that commit, so whether it now accepts the No look: line is unchecked.

Approved

Earlier asks are met. A token now lists only the tools it may call, and callableBy mirrors the MCP dispatch. Bodies are kept per reader kind, so a stranger never sees a token body. A changed tool list bumps the version and so changes the ETag. The docs paragraph is in MCP.md, not the README. Anonymous reads are public, max-age=300, with a strong ETag and a 304 on If-None-Match, and the tests cover each. Nothing in the pull adds a page or a component, so the design principles are not engaged.

Two things I would do next:

  1. Non-blocking: llms-etag.test.ts restores the tool list with [REPORT_TOOL, START_TOOL], a copy of the boot call in server.ts:1295. Export a single helper for it so the two cannot drift.
  2. Non-blocking: the public max-age=300 means a deploy can show a stale commit line for up to five minutes. That is acceptable under the issue. Say so in the docs if a client ever complains.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs