Crawl budget (#8129)
/robots.txtdisallowsDEEP_SEGMENTS(blob, raw, commits, commit, blame, compare, diff, tree, history) at every ref (and/origins/), generated from the same list the budget's deep paths use, keeps home, repository roots and docs open, and asksCrawl-delay: 10.src/http/crawl.ts: an unauthenticated GET/HEAD whose User-Agent names a crawler, or is absent, is counted against three budgets, and refused429withRetry-Afterbefore any page work when any is spent: its client address (peer.ts: the socket peer, an IPv6 client by /64; the only key a client cannot write) at 60 a minute and 6 on the deep paths; its agent (the User-Agent's product token) as a second cap at 120 and 12, since anyone can claimGooglebot; and all such crawlers together at 240 and 24, so total crawler cost has a ceiling whatever they rotate. Agent names past 256 are counted asother, so rotating names cannot grow memory. A spoofer of a real crawler's name is stopped by its own address budget first. While the event loop is late (average wake > 50 ms) the budgets divide by 4. A signed-in person or any request carrying a token is never counted. A browser-looking User-Agent from a rotating-address client is not a crawler here;stranger_reads.tsbounds those.- A blob or commit page at a full object id, read with no one signed in (the resolved auth, not header presence) on a public repository, answers
public, max-age=31536000, s-maxage=86400, immutablewithVary: Cookie, Authorization: the browser keeps it a year, a shared cache a day, because a repository can later turn private or be deleted and a shared cache would go on serving it. Raw already did. - Usage counts carry a
crawlerrole. The crawlers' share of answering time over the last 15 minutes, the refusals by agent and by which budget refused (address, agent, shared) are read atGET /api/v1/admin/crawlers(admin) and in the terminal::→ "Crawlers" (a forge admin who is a person).
Not done: a panel on a board page (the performance board has no page yet; the share is the API above), and a budget by time spent rather than by request count.
Anonymous readers with a browser User-Agent (#8199)
- Anyone with no session and no token, whatever the User-Agent, is counted on the deep pages (
DEEP_SEGMENTS):STRANGER_BUDGETis 30 a minute per address (an IPv6 client by /64), divided by 4 while the loop is late. There is no bucket for all such readers together: a handful of addresses could spend it and refuse every other person; the aggregate is held by the tree cache and the loop-lag divisor. The explorer's lazy folder expansion (?fragment=explorer) is not counted. Past it: 429 withRetry-Afterbefore page work. Other pages are not counted for browser UAs; a signed-in person or token never is. /tree/:reffor an anonymous reader of a public repository resolves the branch to its commit from the ref files (no process), then serves the rendered listing and explorer from a cache keyed on repo, commit, ref, path and?all(src/web/pages/tree_cache.ts, 60 s, 32 MB). A push moves the commit, so the next read misses; only the ref switcher's menu can be up to 60 s old. A miss reads git as before (ls-treeetc. are child processes, off the main thread).- Intended: only this anonymous path links by commit; a signed-in person's listing keeps the branch name in its links (
linkRefis set only there). - That listing's links (entries and README) name the full commit, so the blob and tree pages behind them carry the immutable headers; a tree at a full id is
immutablefor anonymous readers too. - The crawler board (
/api/v1/admin/crawlers, terminal "Crawlers") addsdeep_stranger_requests,deep_stranger_msandrefused_by_budget.stranger.
- Who used the answering time is on home's "Traffic, 15 min" tile and at
/settings/traffic(admin), from minute totals the performance record keeps as each request is added (perf/traffic.ts): exact under a flood, no address kept, only its /16 or /32 and a salted sketch.GET /api/v1/admin/kpis/probe(admin) answers every home number with what it counts and its raw inputs read again past the home cache.