Every compressed response says Vary: Accept-Encoding: /llms.txt and /sitemap.xml come back br-encoded without it #8340

/llms.txt, /sitemap.xml, /robots.txt and the Atom feeds now always carry Vary: Accept-Encoding, and are br/gzip-encoded only when the request asks for it.

Cause. On production the br encoding comes from the edge (Cloudflare), not the forge. compress in src/http/compress.ts skipped any content type not in its TYPES set. So the forge sent these bodies plain with no Vary, and the edge then encoded them without adding one.

Fix. TYPES now includes text/markdown, text/plain, text/xml, application/xml, application/atom+xml and application/rss+xml, so one function owns the rule. The existing logic in compress still applies to them:

• Vary is added once and merged with any existing value.
• Bodies under MIN_BYTES, HEAD requests, 206s, no-transform, and bodies that already have content-encoding are left alone.
• It runs before securityFloor, so the security headers are unchanged.

Adding text/plain buffers no stream: proxied app responses go through proxyApp and never reach compress, and the forge's own text/plain bodies (log text, raw blobs, metrics, errors) are already fully in memory. Log follows are text/event-stream, which is still excluded.

Tests.

• test/compress-serve.test.ts requests /llms.txt and /sitemap.xml with accept-encoding: identity and with br. Both answers carry Vary: Accept-Encoding exactly once and keep nosniff. Only the br answer has content-encoding: br, and it decodes to the plain body. A body under MIN_BYTES must come back unencoded. Both tests fail with the old compress.ts.
• Changed test, which was wrong: test/feeds.test.ts:187 expected Vary to be exactly authorization, cookie. It now expects authorization, cookie, Accept-Encoding, the merge the issue asks for.

Checks run. bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 1069 test files (an estimated 93 minutes), so I ran instead the 64 test files that mention content-encoding, vary, llms/sitemap/robots, feeds, metrics, raw blobs or text/plain/text/markdown. All pass after the feeds.test.ts fix.

To try it: curl -s -o /dev/null -D - -H 'accept-encoding: br' <preview>/llms.txt shows vary: Accept-Encoding and content-encoding: br. With -H 'accept-encoding: identity' it shows vary and no encoding.

🤖 Generated with Claude Code

Closes #8328

/llms.txt, /sitemap.xml, /robots.txt and the Atom feeds now always carry Vary: Accept-Encoding, and are br/gzip-encoded only when the request asks for it.

Cause. On production the br encoding comes from the edge (Cloudflare), not the forge. compress in src/http/compress.ts skipped any content type not in its TYPES set. So the forge sent these bodies plain with no Vary, and the edge then encoded them without adding one.

Fix. TYPES now includes text/markdown, text/plain, text/xml, application/xml, application/atom+xml and application/rss+xml, so one function owns the rule. The existing logic in compress still applies to them:

• Vary is added once and merged with any existing value.
• Bodies under MIN_BYTES, HEAD requests, 206s, no-transform, and bodies that already have content-encoding are left alone.
• It runs before securityFloor, so the security headers are unchanged.

Adding text/plain buffers no stream: proxied app responses go through proxyApp and never reach compress, and the forge's own text/plain bodies (log text, raw blobs, metrics, errors) are already fully in memory. Log follows are text/event-stream, which is still excluded.

Tests.

• test/compress-serve.test.ts requests /llms.txt and /sitemap.xml with accept-encoding: identity and with br. Both answers carry Vary: Accept-Encoding exactly once and keep nosniff. Only the br answer has content-encoding: br, and it decodes to the plain body. A body under MIN_BYTES must come back unencoded. Both tests fail with the old compress.ts.
• Changed test, which was wrong: test/feeds.test.ts:187 expected Vary to be exactly authorization, cookie. It now expects authorization, cookie, Accept-Encoding, the merge the issue asks for.

Checks run. bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 1069 test files (an estimated 93 minutes), so I ran instead the 64 test files that mention content-encoding, vary, llms/sitemap/robots, feeds, metrics, raw blobs or text/plain/text/markdown. All pass after the feeds.test.ts fix.

To try it: curl -s -o /dev/null -D - -H 'accept-encoding: br' <preview>/llms.txt shows vary: Accept-Encoding and content-encoding: br. With -H 'accept-encoding: identity' it shows vary and no encoding.

🤖 Generated with Claude Code