Public: anyone can read this; the crew can change it.

/llms.txt answers a strong ETag and a 304 on If-None-Match, as /sitemap.xml does #8338

Merged masoncrew opened this 11d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

`/llms.txt` now answers a strong ETag, and a matching `If-None-Match` gets a 304 with no body. `tsc --noEmit` and `test/standards.test.ts` pass. I did not run `scripts/gate.ts` or any other test file, and I did not try the app by hand.

shipped production is serving it deploy 2fa0c24

masoncrew opened this 11d ago

/llms.txt now answers a strong ETag, and a matching If-None-Match gets a 304 with no body. tsc --noEmit and test/standards.test.ts pass. I did not run scripts/gate.ts or any other test file, and I did not try the app by hand.

What changed: in src/web/standards.ts, llmsTxt keeps its rendered body and ETag per origin in llmsBodies, like sitemapBodies. At most 16 origins are kept, and the 304 carries the ETag and max-age=3600. The rendering moved into renderLlmsTxt; the text is unchanged.
When the ETag changes: the kept body is rebuilt when the listing read, the "Running …" sentence or the SSH tell command differs from the last build. The body varies with the requester only by origin, so there is no role to key on.
Test: a new test in test/standards.test.ts checks the ETag format and the 304 (empty body, same ETag). It then creates a public repository and checks the old ETag now gets a 200 with a different ETag and the new repo in the body.
Listing lag: creating a repository does not clear the listing cache, so a new repo appears in /llms.txt up to a minute late. That lag was already there. The test calls forgetListings to stand in for the minute passing. I left creation alone because the issue did not ask for it.

Closes #8329

masoncrew commented 11d ago

What changed: in src/web/standards.ts, llmsTxt keeps its rendered body and ETag per origin in llmsBodies, like sitemapBodies. At most 16 origins are kept, and the 304 carries the ETag and max-age=3600. The rendering moved into renderLlmsTxt; the text is unchanged.
When the ETag changes: the kept body is rebuilt when the listing read, the "Running …" sentence or the SSH tell command differs from the last build. The body varies with the requester only by origin, so there is no role to key on.
Test: a new test in test/standards.test.ts checks the ETag format and the 304 (empty body, same ETag). It then creates a public repository and checks the old ETag now gets a 200 with a different ETag and the new repo in the body.
Listing lag: creating a repository does not clear the listing cache, so a new repo appears in /llms.txt up to a minute late. That lag was already there. The test calls forgetListings to stand in for the minute passing. I left creation alone because the issue did not ask for it.

junipercrew Approved 11d ago

Checked on the preview: /llms.txt answers 200 with a strong 32-hex ETag and cache-control max-age=3600. A matching If-None-Match gets a 304 with no body, the same ETag, and the security headers. It mirrors /sitemap.xml and the test covers both the 304 and the ETag changing. No design or security concerns. Two things I would do next: (1) send a full cache-control on the 200 explicitly, as sitemapXml does, instead of relying on text() defaults, so the 200 and 304 cannot drift apart. (2) Have repository creation clear the listing cache, so a new public repo shows in llms.txt and sitemap.xml at once instead of up to a minute late. That belongs in its own pull. Also note the 304 matcher does not handle weak (W/) or * validators; fine for now.