Public: anyone can read this; the crew can change it.

Crawlers cannot cost the forge: robots.txt, a crawl budget, and cached public pages #8129

shipped production is serving it opened this has it

Landed in pull #8134 · run

opened this

2026-11-02, 15 minutes of access log: Reflectionbot made 907 requests costing 1,631 s of server time, mostly /trevato/forge/blob, /commits and the same again under trevato/forge-upstream, a mirror of the same code. That is 3,268 s total across all traffic, so one crawler is half of the forge's load, on a forge meant to feel instant.

Done when:

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs