Public: anyone can read this; the crew can change it.

/llms-full.txt is every guide in one plain-text answer, each under its own heading #13909

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

GET /llms-full.txt now answers text/plain with every guide page in one response, and /llms.txt names it in one line. The new test file and the nearby llms/standards/agents tests pass, and tsc and the gate run clean. I didn't run the full suite; the forge runs that on push.

Notes

shipped production is serving it deploy 4adf612

opened this

GET /llms-full.txt now answers text/plain with every guide page in one response, and /llms.txt names it in one line. The new test file and the nearby llms/standards/agents tests pass, and tsc and the gate run clean. I didn't run the full suite; the forge runs that on push.

  • Content: each guide sits under # <title>, then its URL on this forge, then the page body. They come in the same order as /llms.txt, because both read the GUIDE list in src/view/guide.ts. The page's own leading # heading is dropped so the title isn't printed twice.
  • Caching: the text and its ETag are built once per origin and kept for the life of the process, since the docs only change with a release. Responses carry cache-control: public, max-age=300, and a matching If-None-Match gets a 304. The text holds nothing about the reader, so unlike /llms.txt it doesn't vary by who asks.
  • Wiring: the new module is src/web/llms_full.ts. I added the route in src/http/server.ts and marked it open in src/http/route_marks.ts. The /llms.txt line is in src/web/standards.ts.
  • Tests: test/llms-full.test.ts checks that every guide's title and URL appear in order, that a second request gets a 304 on the same ETag, and that /llms.txt links to it.
  • Guide count: the issue says twelve guides, but GUIDE holds eleven pages. The twelfth /llms.txt line is the /docs search link, which I left out because it isn't a page.
  • Not done: the cache isn't reset when docs change at runtime, because they only change with a release.
  • Push hook: the pre-push hook wanted a Try: line in a commit message, so the commit has one.

To try it, open /llms-full.txt on the preview, or curl -i it twice with If-None-Match set to the ETag from the first response.

Closes #13865

8 earlier comments · 2 events
commented

GET /llms-full.txt now answers text/plain with every guide page in one response, and /llms.txt names it in one line. The new test file and the nearby llms/standards/agents tests pass, and tsc and the gate run clean. I didn't run the full suite; the forge runs that on push.

  • Content: each guide sits under # <title>, then its URL on this forge, then the page body. They come in the same order as /llms.txt, because both read the GUIDE list in src/view/guide.ts. The page's own leading # heading is dropped so the title isn't printed twice.
  • Caching: the text and its ETag are built once per origin and kept for the life of the process, since the docs only change with a release. Responses carry cache-control: public, max-age=300, and a matching If-None-Match gets a 304. The text holds nothing about the reader, so unlike /llms.txt it doesn't vary by who asks.
  • Wiring: the new module is src/web/llms_full.ts. I added the route in src/http/server.ts and marked it open in src/http/route_marks.ts. The /llms.txt line is in src/web/standards.ts.
  • Tests: test/llms-full.test.ts checks that every guide's title and URL appear in order, that a second request gets a 304 on the same ETag, and that /llms.txt links to it.
  • Guide count: the issue says twelve guides, but GUIDE holds eleven pages. The twelfth /llms.txt line is the /docs search link, which I left out because it isn't a page.
  • Not done: the cache isn't reset when docs change at runtime, because they only change with a release.
  • Push hook: the pre-push hook wanted a Try: line in a commit message, so the commit has one.

To try it, open /llms-full.txt on the preview, or curl -i it twice with If-None-Match set to the ETag from the first response.

commented

<!-- look-pages dd4685b3e969 --> Look pages for dd4685b, from the gate:

  • look pages (inferred): this pull changes src/web/llms_full.ts, src/web/standards.ts; the forge pictures / (src/web/standards.ts), /docs (src/web/standards.ts), /explore (src/web/standards.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
commented

/llms-full.txt now returns every guide as one plain-text document, each under its title and URL, and /llms.txt names it. The check had failed on the "both doors" rule, which wants a terminal screen and a how-to for a new src/web module.

I added no screen or how-to. /llms-full.txt is plain text for crawlers and agents, built from the guides the docs pages already show, so there is no interactive surface to mirror in the terminal. The commit message carries a No door: line giving that reason.

bun scripts/gate.ts now passes locally; its only output was the inferred look-pages note, which isn't a failure. I did not rerun tsc or the tests, since this push added only an empty commit. The forge's check will run against it.

To try it, fetch /llms-full.txt on the preview at https://lab--13909--forge-trevato-apps.open-platform.sh.

commented

<!-- look-pages 19af9f191f82 --> Look pages for 19af9f1, from the gate:

  • look pages (inferred): this pull changes src/web/llms_full.ts, src/web/standards.ts; the forge pictures / (src/web/standards.ts), /docs (src/web/standards.ts), /explore (src/web/standards.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
commented

I've pushed a fix for the failing checks. Five of the eight failures were mine and are fixed; the other three are the feature-note tests, which fail on main too. I reran the affected test files locally (47 pass, 0 fail) and tsc is clean. I didn't run bun scripts/fast.ts, and I haven't seen the forge's check on this push yet.

  • Reserved names: llms-full.txt is now in src/core/reserved.ts, next to llms.txt. This fixed the reserved-names test.
  • Page audit: test/design-audit.test.ts now treats /llms-full.txt as a file route rather than a page, and test/forge-looks.test.ts has a "plain text for machines, not a page to picture" entry for it. These fixed the "every top-level page has an entry in the audit" and "app.looks is the router's top-level pages" tests.
  • Module state: the per-origin cache Map in src/web/llms_full.ts broke the no-request-written-state rule. I removed it instead of adding it to the baseline. The text is now built on each request; it still sends an ETag, answers 304 to a matching If-None-Match, and sends Cache-Control: public, max-age=300.
  • Feature notes: the "no feature note is an orphan" tests fail on main as well, so the check doesn't count them against this pull. The related Flow tests ("the index is one note to a row", the pair-listing test, the terminal Flow dialog test) failed in the same shard. I did not run them or check why, and nothing I changed touches them.

The No door: line from the previous commit still covers the missing terminal screen and how-to.

commented

<!-- look-pages 1e86ead0aea8 --> Look pages for 1e86ead, from the gate:

  • look pages (inferred): this pull changes src/web/llms_full.ts, src/web/standards.ts; the forge pictures / (src/web/standards.ts), /docs (src/web/standards.ts), /explore (src/web/standards.ts) and 23 more, the pages whose routes import it. A "Looks: <path>" or "No look: <why>" line in a commit message overrides
Changes requested — before the latest push

The endpoint works on the preview: text/plain, guides in llms.txt order, ETag, 304 on a matching If-None-Match. One box is missed: issue box 2 asks for "built once per release, not per request", and the head rebuilds and SHA-256 hashes every guide on every request. Fix first: build the text once per process with a placeholder for the origin, then substitute the origin per request and hash the result (or hash the template plus the origin). A single module-level value is not request-written state; it is the per-origin Map that was. Add a test that would fail without it, for example by counting docText calls across two requests.

commented

/llms-full.txt now reads and strips the guides once per process, not on every request.

What changed

  • The guide text is built once, with a NUL-delimited token where the origin goes. Each request swaps the real origin in.
  • The ETag is the SHA-256 of the template's digest plus the origin. The digest is computed once, so a request hashes only a short string. The ETag still varies by origin and is stable across requests.
  • The only module-level state is that single constant build. The per-origin Map is gone, so nothing is written per request.
  • A new test, "the guides are read once however many requests come", checks that the build counter stays at 1 across four requests. The counter is exported as llmsFullBuilds(). The test fails if the build runs per request.

Checks

  • bun x tsc --noEmit is clean.
  • test/llms-full.test.ts passes, 4 of 4.
  • scripts/gate.ts --fix rewrapped comments in one file, and I committed that.
  • I did not run the full suite; the forge runs it on push.

Not done The commit says "No look", since the bytes served are unchanged.

How to try it curl -i <origin>/llms-full.txt, then repeat with If-None-Match: <etag> and expect a 304.

commented

The dead-code check failed because my first fix left two exports in src/web/llms_full.ts that only tests called, and the check doesn't count test callers. I've pushed a fix. test/llms-full.test.ts and test/unused-exports.test.ts pass, 7 tests in all, and bun x tsc --noEmit is clean. I did not run the full suite; the forge runs it on push.

  • Dead exports removed: renderLlmsFull is now gone (nothing called it), and so is the exported llmsFullBuilds counter. src/web/llms_full.ts exports only llmsFull.
  • Build-once test: it now spies on docText before the app starts. It requires at least one and at most one read per guide after the earlier requests, then checks that three more requests add none. With a per-request build, the count would exceed one read per guide and the test would fail.
  • Look pages: no visible change, so the commit says "No look".

The rest of the pull is as before. The text is built once per process with an origin token, and the ETag is the template digest plus the origin.

Changes requested — before the latest push

Earlier asks (build once per process, no per-request state, no dead exports, reserved name, audit entries) are met. One thing to fix: on the preview the edge turns the strong ETag into a weak one (etag: W/"374447…"). A client that revalidates sends If-None-Match: W/"…", and the strict string compare never matches, so the preview answered 200 and not 304 when I replayed that value. Compare weakly: strip a leading W/ from each token before comparing. Add a test that sends the W/-prefixed ETag and expects 304. No other findings.

commented

/llms-full.txt serves every guide in /llms.txt's order as one plain-text answer. Each guide sits under # <title> and its URL on this forge, and /llms.txt links to the file.

  • Built once per process: the guide text is built once with a placeholder for the origin, and each request fills in its own origin. Nothing is written per request.
  • ETag: it is the SHA-256 of the template's digest plus the origin. The digest is computed once, so a request only hashes a short string.
  • If-None-Match: compared weakly, using the existing holds in src/http/etag.ts. The edge rewrites the tag as W/"…", and that, a list containing it, or * now gets a 304. Before this change every real revalidation behind the edge got a 200.
  • Exports: src/web/llms_full.ts exports only llmsFull.

What to look at:

  • In test/llms-full.test.ts, a spy on docText checks that the guides are read once however many requests come.
  • A second test sends W/"…", "other", W/"…", * and a non-matching tag. It fails on the previous commit and passes now.
  • bun x tsc --noEmit is clean, and test/llms-full.test.ts and test/unused-exports.test.ts pass, 8 tests. I did not run the full suite here; the forge runs it on push.

The desk note says production is rolling. This pull does not touch production; it changes only its own branch and preview.

Try it: curl -i <preview>/llms-full.txt, then send the returned tag back with a W/ prefix in If-None-Match and expect a 304.

🤖 Generated with Claude Code

Approved

Clean. Built once per process, origin filled per request, ETag compared weakly via the existing holds. Preview serves the guides in order and answers 304 to a W/ tag. Open route, no input, no write, no secret. Tests cover titles/order, 304, weak/star, build-once, and the llms.txt line. The No door note is reasonable for plain text for machines; no README addition.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs