trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

Crawlers cannot cost the forge: robots.txt, a crawl budget, and cached public pages #8134

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/8129-crawlers-cannot-cost-the-forge-robots-tx → main 5 commits fork 2 threads · 2 unresolved

  • Merged. torque/8129-crawlers-cannot-cost-the-forge-robots-tx was merged into main as db53585.

Review

Sign in to review this change.
21 files changed +651 −7
README.md +2
@@ -88,6 +88,8 @@ Signed in, `/` paints its frame from memory and swaps in what a write has made s
8888
8989 In the terminal, the System screen (palette: System) shows a forge admin who is a person one more line, `telemetry:read held by: …` (the agents holding the grant), and `g` there picks an agent and asks `grant <agent> telemetry:read? y/n` (or `take back …`; the default is n). `y` calls `setTelemetryGrant`, as MCP `telemetry_grant` does; anyone that tool refuses sees neither the line nor the key hint, and `g` answers them in the tool's words.
9090
91 Crawlers (`docs/features/crawl-budget.md`): an unauthenticated crawler is held to a budget per address, per agent and all together, answered `429` past it; a forge admin reads their share of answering time and who was refused at `GET /api/v1/admin/crawlers` and in the terminal's palette as "Crawlers".
92
9193 In the terminal, the Capacity screen (palette: Capacity, a forge admin who is a person) lists every guard with its live value and bounds, as /settings/capacity does; `e` on one asks for a value (empty restores the default), shows `key: from → to` and calls `changeCapacity` only on `y` (default n). No row is read-only here.
9294
9395 An admin's home and the terminal's Flow dialog also say when the forge is clogged and how long the last clog took to notice and to clear: `docs/features/clog-watch.md`. A fork whose parent is archived shows one quiet line instead of its ahead/behind count, and its admin can detach it from the parent (Settings › Danger zone, the terminal's About `x`, MCP `detach`): `docs/features/detach.md`.
docs/features/crawl-budget.md added +8
@@ -0,0 +1,8 @@
1 # Crawl budget (#8129)
2
3 - `/robots.txt` disallows `DEEP_SEGMENTS` (blob, raw, commits, commit, blame, compare, diff, tree, history) at every ref (and `/origins/`), generated from the same list the budget's deep paths use, keeps home, repository roots and docs open, and asks `Crawl-delay: 10`.
4 - `src/http/crawl.ts`: an unauthenticated GET/HEAD whose User-Agent names a crawler, or is absent, is counted against three budgets, and refused `429` with `Retry-After` before any page work when any is spent: its **client address** (`peer.ts`: the socket peer, an IPv6 client by /64; the only key a client cannot write) at 60 a minute and 6 on the deep paths; its **agent** (the User-Agent's product token) as a second cap at 120 and 12, since anyone can claim `Googlebot`; and **all such crawlers together** at 240 and 24, so total crawler cost has a ceiling whatever they rotate. Agent names past 256 are counted as `other`, so rotating names cannot grow memory. A spoofer of a real crawler's name is stopped by its own address budget first. While the event loop is late (average wake > 50 ms) the budgets divide by 4. A signed-in person or any request carrying a token is never counted. A browser-looking User-Agent from a rotating-address client is not a crawler here; `stranger_reads.ts` bounds those.
5 - A blob or commit page at a full object id, read with no one signed in (the resolved auth, not header presence) on a public repository, answers `public, max-age=31536000, s-maxage=86400, immutable` with `Vary: Cookie, Authorization`: the browser keeps it a year, a shared cache a day, because a repository can later turn private or be deleted and a shared cache would go on serving it. Raw already did.
6 - Usage counts carry a `crawler` role. The crawlers' share of answering time over the last 15 minutes, the refusals by agent and by which budget refused (address, agent, shared) are read at `GET /api/v1/admin/crawlers` (admin) and in the terminal: `:` → "Crawlers" (a forge admin who is a person).
7
8 Not done: a panel on a board page (the performance board has no page yet; the share is the API above), and a budget by time spent rather than by request count.
src/contract/openapi/routes/getApiV1AdminCrawlers.ts added +10
@@ -0,0 +1,10 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "get",
5 path: "/api/v1/admin/crawlers",
6 summary:
7 "Read the crawlers' share of the last 15 minutes of answering time (a person admin's token, any scope); answers { window_minutes, requests, crawler_requests, ms, crawler_ms, share, refused: [{ agent, refused }], refused_by_budget: { address, agent, shared } }",
8 auth: true,
9 refusals: { not_found: "the token is not a person who administers the forge" },
10 };
src/contract/openapi/routes/index.ts +1
@@ -30,6 +30,7 @@ export { route as deleteApiV1UserPinsOwnerName } from "./deleteApiV1UserPinsOwne
3030 export { route as deleteApiV1UsersUsernameSessionsId } from "./deleteApiV1UsersUsernameSessionsId.ts";
3131 export { route as getApiV1AdminAccounts } from "./getApiV1AdminAccounts.ts";
3232 export { route as getApiV1AdminCapacity } from "./getApiV1AdminCapacity.ts";
33 export { route as getApiV1AdminCrawlers } from "./getApiV1AdminCrawlers.ts";
3334 export { route as getApiV1AdminDrills } from "./getApiV1AdminDrills.ts";
3435 export { route as getApiV1AdminInvites } from "./getApiV1AdminInvites.ts";
3536 export { route as getApiV1AdminLimits } from "./getApiV1AdminLimits.ts";
src/core/ratelimit.ts +5
@@ -189,6 +189,11 @@ export class RateLimiter {
189189 this.lastSweep = 0;
190190 }
191191
192 /** Whether `key` is being tracked. */
193 has(key: string): boolean {
194 return this.hits.has(key);
195 }
196
192197 /** How many keys are being tracked — for the leak test, not for callers. */
193198 get size(): number {
194199 return this.hits.size;
src/core/usage_count.ts +1 −1
@@ -17,7 +17,7 @@ import type { Database } from "bun:sqlite";
1717 import { withoutWaiting } from "../kernel/db.ts";
1818 import { quietly } from "./write_clock.ts";
1919
20 export const ROLES = ["anon", "user", "admin", "agent"] as const;
20 export const ROLES = ["anon", "user", "admin", "agent", "crawler"] as const;
2121 export const SURFACES = ["api", "mcp", "web"] as const;
2222 export const METHODS = [
2323 "GET",
src/http/addressed.ts added +33
@@ -0,0 +1,33 @@
1 /**
2 * Commit-addressed pages are immutable (#8129, #8016): a blob or a commit at
3 * a full object id names bytes that can never change, so for an anonymous
4 * reader of a public repository the page may sit in a shared cache for a
5 * while. Only the anonymous render qualifies: the caller says, from the
6 * resolved auth (a cookie, a token or the edge's signed headers alike), that
7 * no one is behind the request; anyone else gets a page with their topbar,
8 * which `privateWhenSignedIn` keeps private. `Vary` makes a cache keep the two
9 * apart. The browser keeps it a year (the bytes cannot change), but a shared
10 * cache only a day (`s-maxage`): a repository can later turn private or be
11 * deleted, and a shared cache must not go on serving it for long.
12 */
13
14 /** How long a shared cache may keep an addressed page, seconds. */
15 export const SHARED_MAX_AGE_S = 86_400;
16
17 const FULL_ID = /^(?:[0-9a-f]{40}|[0-9a-f]{64})$/;
18
19 /** Whether `rev` is a full object id and not a branch, tag or abbreviation. */
20 export const isFullId = (rev: string): boolean => FULL_ID.test(rev);
21
22 export function immutableWhenAddressed(
sentinelcrew outdated

Decide anonymity from the resolved auth, not header presence; and consider s-maxage short rather than a year for content whose repo may become private.
23 anonymous: boolean,
24 repo: { readonly visibility: string },
25 rev: string,
26 response: Response,
27 ): Response {
28 if (response.status !== 200 || repo.visibility !== "public" || !isFullId(rev)) return response;
29 if (!anonymous) return response;
30 response.headers.set("cache-control", `public, max-age=31536000, s-maxage=${SHARED_MAX_AGE_S}, immutable`);
31 response.headers.append("vary", "Cookie, Authorization");
32 return response;
33 }
src/http/api/admin.ts +10
@@ -6,6 +6,7 @@
66 */
77
88
9 import { crawlReport } from "../../perf/crawlers.ts";
910 import { MAX_SECONDS, sampleStacks } from "../../perf/sample.ts";
1011 import { listAccounts, removeAccount, retireAndRemoveAccount, retirementPlan, suspendAccount, unsuspendAccount, verifyAccount } from "../../actions/accounts.ts";
1112 import { ADMIN_WORDS } from "../../actions/admin_words.ts";
@@ -61,6 +62,15 @@ export function adminRoutes(router: Router<Handler>): void {
6162 return sampled ? json(sampled) : apiError("busy", 409, "a sample is already running");
6263 });
6364
65 // The crawlers' share of the last quarter hour's answering time, and who
66 // was refused (#8129): the number the performance board reads.
67 router.add("GET", "/api/v1/admin/crawlers", (ctx) => {
68 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
69 const denied = accountAdmin(ctx, "ro");
70 if (denied) return denied;
71 return json(crawlReport(ctx.app));
72 });
73
6474 // The forge's capacity (`core/capacity.ts`): read and changed live by an
6575 // admin, never by an agent; anyone else is told the door is not there.
6676 router.add("GET", "/api/v1/admin/capacity", (ctx) => {
src/http/crawl.ts added +203
@@ -0,0 +1,203 @@
1 /**
2 * What an unauthenticated crawler may spend (#8129).
3 *
4 * On 2026-11-02 one crawler made 907 requests in 15 minutes, mostly blob and
5 * commit pages, and cost half the forge's server time. `robots.txt` asks
6 * politely; this is the mechanical half. A request with no session and no
7 * token whose User-Agent names itself a crawler (or names nothing) is counted
8 * against three budgets and, past any of them, answered 429 with Retry-After
9 * before any page work: its client address (`peer.ts`, the socket's peer, the
10 * only key a client cannot write; an IPv6 client by its /64), its agent (the
11 * product token of the User-Agent, a second cap only, since anyone can
12 * send `Googlebot`), and one shared budget for every such crawler, so their
13 * total cost has a ceiling however they rotate. A browser-looking User-Agent
14 * is not counted here; `stranger_reads.ts` bounds those. Deep paths (blob, raw, commits, blame, diffs at every ref) have a
15 * far smaller budget than the rest. When the event loop is late the budgets
16 * shrink, so a crawler is the first to wait. A signed-in person or a token is
17 * never counted here: the check is made only for a request `resolveAuth`
18 * found no one behind. In memory, like `strangers.ts`: a restart forgives.
19 */
20
21 import type { App } from "../app.ts";
22 import { RateLimiter } from "../core/ratelimit.ts";
23 import { startLoopLag } from "../kernel/loop_lag.ts";
24 import { bucketOf, clientAddress } from "./peer.ts";
25
26 /** Requests per minute per client address: any page, and the deep ones that cost a git walk. */
27 export const CRAWL_BUDGET = { page: 60, deep: 6 } as const;
28 /** The second cap per agent name (a name is the client's word, so it is looser than an address). */
29 export const AGENT_BUDGET = { page: 120, deep: 12 } as const;
30 /** The ceiling for every crawler together. */
31 export const SHARED_BUDGET = { page: 240, deep: 24 } as const;
32 /** The budgets divide by this while the event loop runs late. */
33 export const BEHIND_DIVISOR = 4;
34 /** An average wake later than this (ms) is "behind". */
35 export const BEHIND_MS = 50;
36 /** `Crawl-delay` in robots.txt, seconds. */
37 export const CRAWL_DELAY_S = 10;
38 const MAX_AGENTS = 256;
39 /** The agent name every name past `MAX_AGENTS` is counted under, so rotating names cannot grow the limiters. */
40 const OTHER = "other";
41
42 /** A User-Agent that says it is a crawler. Not curl, not a browser, not an HTTP library. */
43 const CRAWLER = /bot\b|bot[/ ;)-]|crawl|spider|slurp|scrap|archiver|reflection|facebookexternalhit|bingpreview|ia_archiver/i;
44
45 /** Paths whose answer costs a git walk or a diff: the ones robots.txt disallows. */
46 export const DEEP_SEGMENTS = ["blob", "raw", "commits", "commit", "blame", "compare", "diff", "tree", "history"] as const;
47 const DEEP = new RegExp(`^/(?:origins/[^/]+/r/)?[^/]+/[^/]+/(?:${DEEP_SEGMENTS.join("|")})(?:/|$)`);
48
49 /** What a crawler never spends from: the files it is asked to read, and the probes. */
50 const FREE = /^\/(?:robots\.txt|sitemap\.xml|healthz|readyz|assets\/)/;
51
52 /** The product token of a User-Agent, lowercase and bounded: `Mozilla/5.0 (compatible; Reflectionbot/1.0)` is `reflectionbot`. */
53 export function agentOf(userAgent: string): string {
54 const named = /([a-z][a-z0-9._-]{1,38}bot|[a-z][a-z0-9._-]{0,38}(?:crawler|spider|slurp|archiver))/i.exec(userAgent);
55 const token = named?.[1] ?? /^\s*([a-z][a-z0-9._-]{0,39})/i.exec(userAgent)?.[1] ?? "unknown";
56 return token.toLowerCase();
57 }
58
59 /** Whether this request says it is a crawler and is one the budget counts. */
60 export function isCrawler(request: Request): boolean {
61 if (request.method !== "GET" && request.method !== "HEAD") return false;
62 const ua = request.headers.get("user-agent") ?? "";
63 // No User-Agent at all is a script hiding, not a browser.
64 return ua.length === 0 || CRAWLER.test(ua.slice(0, 300));
65 }
66
67 type Scope = "address" | "agent" | "shared";
68 interface Pair {
69 readonly page: RateLimiter;
70 readonly deep: RateLimiter;
71 }
72 interface State {
73 readonly address: Pair;
74 readonly agent: Pair;
75 readonly shared: Pair;
76 readonly refused: Map<string, number>;
77 readonly refusedBy: Record<Scope, number>;
78 readonly clock: { now: () => number; behind: boolean };
79 }
80
81 const states = new WeakMap<App, State>();
82 let lateMs = 0;
83 let lagStarted = false;
84
85 /** Start measuring lateness on the first crawler; an exponential average, so one slow wake is not "behind". */
86 function watchLag(): void {
87 if (lagStarted) return;
88 lagStarted = true;
89 startLoopLag(100, undefined, (late) => {
90 lateMs = lateMs * 0.8 + late * 0.2;
91 });
92 }
93
94 /** Whether the event loop is running late enough that a crawler should wait. */
95 export const mainThreadBehind = (): boolean => lateMs > BEHIND_MS;
96
97 const limits = (perMinute: number) => ({ perMinute, adminPerMinute: perMinute, enabled: true });
98
99 function stateOf(app: App): State {
100 let state = states.get(app);
101 if (state === undefined) {
102 const clock = { now: Date.now, behind: false };
103 const pair = (b: { page: number; deep: number }): Pair => ({
104 page: new RateLimiter(limits(b.page), () => clock.now()),
105 deep: new RateLimiter(limits(b.deep), () => clock.now()),
106 });
107 state = {
108 address: pair(CRAWL_BUDGET),
109 agent: pair(AGENT_BUDGET),
sentinelcrew outdated

High: agent comes from the client's User-Agent and is the only bucket key. Spoofable in both directions (exhaust a real crawler's budget, or evade by rotating UA). Key on the socket address and keep UA as a secondary cap. Also unbounded limiter keys.
110 shared: pair(SHARED_BUDGET),
111 refused: new Map(),
112 refusedBy: { address: 0, agent: 0, shared: 0 },
113 clock,
114 };
115 states.set(app, state);
116 }
117 return state;
118 }
119
120 /** Test seam: count by `now`, and say whether the main thread is behind. */
121 export function setCrawlClock(app: App, now: () => number, behind = false): void {
122 const { clock } = stateOf(app);
123 clock.now = now;
124 clock.behind = behind;
125 }
126
127 export interface CrawlVerdict {
128 readonly retryAfter: number;
129 readonly agent: string;
130 /** Which budget ran out. */
131 readonly scope: Scope;
132 }
133
134 /**
135 * Count one unauthenticated request. Undefined: let it through (not a
136 * crawler, or inside its budget). Otherwise the refusal to answer with.
137 */
138 export function takeCrawl(app: App, request: Request, pathname: string): CrawlVerdict | undefined {
139 if (!isCrawler(request) || FREE.test(pathname)) return undefined;
140 watchLag();
141 const state = stateOf(app);
142 const behind = state.clock.behind || mainThreadBehind();
143 const named = agentOf(request.headers.get("user-agent") ?? "");
144 const scale = behind ? BEHIND_DIVISOR : 1;
145 const deep = DEEP.test(pathname);
146 const scaled = (budget: number) => limits(Math.max(1, Math.floor(budget / scale)));
147 // A name nobody has used yet, once the table is full, is counted as "other".
148 const agent = state.agent.page.has(named) || state.agent.page.size < MAX_AGENTS ? named : OTHER;
149 const budgets: [Scope, Pair, string, { page: number; deep: number }][] = [
150 ["address", state.address, bucketOf(clientAddress(request)), CRAWL_BUDGET],
151 ["agent", state.agent, agent, AGENT_BUDGET],
152 ["shared", state.shared, "all", SHARED_BUDGET],
153 ];
154 const taken: { limiter: RateLimiter; key: string }[] = [];
155 for (const [scope, pair, key, budget] of budgets) {
156 pair.page.setLimits(scaled(budget.page));
157 pair.deep.setLimits(scaled(budget.deep));
158 const counted = deep ? [pair.page, pair.deep] : [pair.page];
159 for (const limiter of counted) {
160 const v = limiter.take(key, "ro");
161 if (v.allowed) {
162 taken.push({ limiter, key });
163 continue;
164 }
165 // A refused request spends nothing anywhere: give back what it took.
166 for (const t of taken) t.limiter.refund(t.key);
167 if (state.refused.has(agent) || state.refused.size < MAX_AGENTS)
168 state.refused.set(agent, (state.refused.get(agent) ?? 0) + 1);
169 state.refusedBy[scope]++;
170 return { retryAfter: v.retryAfter, agent, scope };
171 }
172 }
173 return undefined;
174 }
175
176 /** Requests refused so far in this process, by agent, most first. */
177 export const crawlRefusals = (app: App): { agent: string; refused: number }[] =>
178 [...stateOf(app).refused]
179 .map(([agent, refused]) => ({ agent, refused }))
180 .sort((a, b) => b.refused - a.refused);
181
182 /** How many agent names the limiters track, for the test of the bound. */
183 export const crawlAgentCount = (app: App): number => stateOf(app).agent.page.size;
184
185 /** How many refusals each budget has given, for the board. */
186 export const crawlRefusalScopes = (app: App): Record<Scope, number> => ({ ...stateOf(app).refusedBy });
187
188 /** The refusal: small, plain, and not worth caching past the wait. */
189 export function crawlRefused(verdict: CrawlVerdict): Response {
190 return new Response("Too many requests from this crawler: see /robots.txt for what to skip.\n", {
191 status: 429,
192 headers: {
193 "retry-after": String(verdict.retryAfter),
194 "content-type": "text/plain; charset=utf-8",
195 "cache-control": "no-store",
196 },
197 });
198 }
199
200 const noted = new WeakSet<Request>();
201 /** The server says this request was a crawler's, for the usage count and the performance record. */
202 export const noteCrawler = (request: Request): void => void noted.add(request);
203 export const wasCrawler = (request: Request): boolean => noted.has(request);
src/http/server.ts +13 −1
@@ -76,6 +76,7 @@ import {
7676 verifyPasswordThrottled,
7777 } from "../core/users.ts";
7878 import { activeEdgeToken, edgeIdentity } from "./edge.ts";
79 import { crawlRefused, isCrawler, noteCrawler, takeCrawl, wasCrawler } from "./crawl.ts";
7980 import { liveResponse, readyResponse } from "./health.ts";
8081 import { running } from "../kernel/version.ts";
8182 import { runGit } from "../git/env.ts";
@@ -2262,7 +2263,7 @@ export async function serve(app: App): Promise<Servers> {
22622263 const end = performance.now();
22632264 perfRecord.add(
22642265 tag.pattern,
2265 { at: Date.now(), ms, signedIn: tag.role !== "anon", phases: parseServerTiming(header) },
2266 { at: Date.now(), ms, signedIn: tag.role !== "anon" && tag.role !== "crawler", crawler: wasCrawler(request), phases: parseServerTiming(header) },
22662267 );
22672268 loopWatch.finished(tag.pattern, started, end);
22682269 } catch {
@@ -2627,6 +2628,17 @@ async function answerGet(
26272628 noteCaller(request, app.db, auth.user.id, auth.sessionId);
26282629 }
26292630
2631 // A crawler nobody is signed in behind spends from its agent's budget and
2632 // is refused before any page work (`crawl.ts`); a person or token never is.
2633 if (auth.user === undefined && !request.headers.has("authorization")) {
2634 const crawl = takeCrawl(app, request, url.pathname);
2635 if (isCrawler(request)) {
2636 noteCrawler(request);
2637 noteUsage(request, { pattern: matched.pattern, role: "crawler" });
2638 }
2639 if (crawl !== undefined) return crawlRefused(crawl);
2640 }
2641
26302642 // A cookie's identity never acts for another site: one door for every route.
26312643 const originRefusal = originGate(app, request, url, auth);
26322644 if (originRefusal !== undefined) return originRefusal;
src/perf/crawlers.ts added +24
@@ -0,0 +1,24 @@
1 /**
2 * What crawlers cost the forge, as one plain object (#8129): their share of
3 * the last quarter hour's answering time and who was refused. The admin API
4 * and the terminal's Crawlers screen both read it.
5 */
6
7 import type { App } from "../app.ts";
8 import { crawlRefusalScopes, crawlRefusals } from "../http/crawl.ts";
9 import { perfRecord } from "./live.ts";
10 import { WINDOW_MS, crawlerShare } from "./record.ts";
11
12 export function crawlReport(app: App, now = Date.now()) {
13 const share = crawlerShare(perfRecord, now - WINDOW_MS);
14 return {
15 window_minutes: WINDOW_MS / 60_000,
16 requests: share.requests,
17 crawler_requests: share.crawlerRequests,
18 ms: Math.round(share.ms),
19 crawler_ms: Math.round(share.crawlerMs),
20 share: Math.round(share.share * 1000) / 1000,
21 refused: crawlRefusals(app).slice(0, 50),
22 refused_by_budget: crawlRefusalScopes(app),
23 };
24 }
src/perf/record.ts +27
@@ -25,6 +25,8 @@ export interface Sample {
2525 readonly at: number;
2626 readonly ms: number;
2727 readonly signedIn: boolean;
28 /** An unauthenticated request whose User-Agent names a crawler (`http/crawl.ts`). */
29 readonly crawler?: boolean;
2830 readonly phases: readonly PhaseSample[];
2931 }
3032
@@ -128,3 +130,28 @@ export function parseServerTiming(header: string): PhaseSample[] {
128130 }
129131 return out;
130132 }
133
134 export interface CrawlerShare {
135 readonly requests: number;
136 readonly crawlerRequests: number;
137 /** Summed time to answer, ms, of every request and of the crawlers' alone. */
138 readonly ms: number;
139 readonly crawlerMs: number;
140 /** crawlerMs / ms, 0 when nothing was answered. */
141 readonly share: number;
142 }
143
144 /** What share of the server's answering time crawlers took, from `since` on. */
145 export function crawlerShare(record: PerfRecord, since: number): CrawlerShare {
146 let requests = 0, crawlerRequests = 0, ms = 0, crawlerMs = 0;
147 for (const pattern of record.patterns())
148 for (const s of record.traffic(pattern, since)) {
149 requests++;
150 ms += s.ms;
151 if (s.crawler === true) {
152 crawlerRequests++;
153 crawlerMs += s.ms;
154 }
155 }
156 return { requests, crawlerRequests, ms, crawlerMs, share: ms > 0 ? crawlerMs / ms : 0 };
157 }
src/tui/forge.ts +9
@@ -56,6 +56,7 @@ import { signInsScreen } from "./screens/signins.ts";
5656 import { originAppsScreen } from "./screens/originapps.ts";
5757 import { addEntries } from "./screens/entries.ts";
5858 import { capacityScreen, mayCapacity } from "./screens/capacity.ts";
59 import { crawlersScreen } from "./screens/crawlers.ts";
5960 import { forgeCeiling } from "../flow/ceiling.ts";
6061 import { flowLines, flowScreen } from "./screens/flow.ts";
6162 import { readHeld } from "../flow/held.ts";
@@ -342,6 +343,14 @@ export function forgeCommands(session: Session): (ctx: Ctx) => Command[] {
342343 hint: "concurrency and guards, live; e changes a settable one",
343344 run: () => ctx.push(capacityScreen(session, ctx)),
344345 });
346 // What crawlers cost and who was refused, as GET /api/v1/admin/crawlers: the capacity screen's readers.
347 if (mayCapacity(session))
348 commands.push({
349 id: "crawlers",
350 title: "Crawlers",
351 hint: "their share of answering time, and who was refused",
352 run: () => ctx.push(crawlersScreen(session, ctx)),
353 });
345354 // The forge's API operations, as /api lists them; Enter runs a GET: for anyone.
346355 commands.push({
347356 id: "api",
src/tui/screens/crawlers.ts added +44
@@ -0,0 +1,44 @@
1 /**
2 * What crawlers cost the forge in the terminal, a forge admin's who is a
3 * person: their share of the last quarter hour's answering time, which budget
4 * refused them, and each agent refused, as GET /api/v1/admin/crawlers says it.
5 * Read-only.
6 */
7
8 import { crawlReport } from "../../perf/crawlers.ts";
9 import type { Ctx, Screen } from "../app/app.ts";
10 import { binding } from "../app/keys.ts";
11 import type { Session } from "../session.ts";
12 import { style } from "../term/style.ts";
13 import { truncate } from "../term/width.ts";
14 import { fill, len, split } from "../ui/layout.ts";
15 import { ListState, drawList, type ListRow } from "./common.ts";
16 import { mayCapacity } from "./capacity.ts";
17
18 export function crawlersScreen(f: Session, ctx: Ctx): Screen {
19 const state = new ListState();
20 const lines = (): ListRow[] => {
21 if (!mayCapacity(f)) return [];
22 const r = crawlReport(f.app);
23 const b = r.refused_by_budget;
24 return [
25 { title: `Share of answering time: ${Math.round(r.share * 100)}%`, meta: `${r.crawler_ms} of ${r.ms} ms, ${r.crawler_requests} of ${r.requests} requests, last ${r.window_minutes} min` },
26 { title: "Refused (429) by budget", meta: `address ${b.address} · agent ${b.agent} · all crawlers ${b.shared}` },
27 ...r.refused.map((a) => ({ title: a.agent, meta: `${a.refused} refused` })),
28 ];
29 };
30 const screen: Screen = {
31 title: "Crawlers",
32 keys: [binding(["j", "k"], "move cursor")],
33 render(area, s) {
34 const c = ctx.theme.c;
35 const [head, list] = split(area, "v", [len(1), fill(1)]);
36 s.text(head!.x, head!.y, truncate("unauthenticated crawlers: counted per address, per agent, and all together", head!.w), style(c.mutedFg, c.bg), head!.x + head!.w);
37 drawList(s, list!, c, state, lines());
38 },
39 key(e) {
40 return state.key(e, lines().length);
41 },
42 };
43 return screen;
44 }
src/web/pages/commits.ts +2 −1
@@ -10,6 +10,7 @@
1010 * list can do neither, so the live browse pages by cursor and says so.
1111 */
1212
13 import { immutableWhenAddressed } from "../../http/addressed.ts";
1314 import { checkOfCommit, verdictsOfCommits } from "../../actions/runs.ts";
1415 import type { CommitCheck } from "../../checks/commit_check.ts";
1516 import { diffCommit, type Diff } from "../../git/diff.ts";
@@ -350,7 +351,7 @@ export async function commitPage(ctx: Ctx): Promise<Response> {
350351 body,
351352 });
352353 notePhase(ctx.request, "render", performance.now() - rendering);
353 return response;
354 return immutableWhenAddressed(ctx.auth.user === undefined, repo, rev, response);
354355 }
355356
356357 // ---- the live-browse pages ----
src/web/pages/repo.ts +2 −1
@@ -8,6 +8,7 @@
88 * path segment and can never smuggle a path of its own.
99 */
1010
11 import { immutableWhenAddressed } from "../../http/addressed.ts";
1112 import { behindLine, followingLine } from "./upstream-follow.ts";
1213 import { upstreamOffer } from "../../actions/offer_upstream.ts";
1314 import { offerUpstream } from "./offer-upstream.ts";
@@ -1537,7 +1538,7 @@ export async function blobPage(ctx: Ctx): Promise<Response> {
15371538 ...(isCsvPath(path) ? { extraJs: csvJs } : {}),
15381539 });
15391540 notePhase(ctx.request, "render", paneMs + performance.now() - pageStart);
1540 return response;
1541 return immutableWhenAddressed(ctx.auth.user === undefined, repo, ref, response);
15411542 }
15421543
15431544 /**
src/web/standards.ts +7
@@ -24,6 +24,7 @@
2424 * FORGE_SECURITY_CONTACT.
2525 */
2626
27 import { CRAWL_DELAY_S, DEEP_SEGMENTS } from "../http/crawl.ts";
2728 import { markPng, type MarkShape } from "../design/mark.ts";
2829 import { metaThemeColor } from "../design/themes.ts";
2930 import { BODY_MAX, TITLE_MAX } from "../core/reports.ts";
@@ -159,6 +160,12 @@ export function robotsTxt(ctx: Ctx): Response {
159160 "Disallow: /settings",
160161 "Disallow: /search",
161162 "Disallow: /api/",
163 // The deep paths, at every ref and in every mirror: each is a git walk
164 // (#8129). Repository roots, docs and the home page stay open.
165 ...DEEP_SEGMENTS.map((segment) => `Disallow: /*/*/${segment}`),
166 "Disallow: /origins/",
167 "Allow: /$",
168 `Crawl-delay: ${CRAWL_DELAY_S}`,
162169 "",
163170 `Sitemap: ${ctx.url.origin}/sitemap.xml`,
164171 "",
test/crawl-budget.test.ts added +178
@@ -0,0 +1,178 @@
1 /** #8129: robots.txt names the deep paths, a crawler past its budget gets 429 before page work, a person never does, addressed pages are immutable. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
8 import { getRepo } from "../src/core/repos.ts";
9 import { createSession } from "../src/core/sessions.ts";
10 import { createUser, type User } from "../src/core/users.ts";
11 import { AGENT_BUDGET, CRAWL_BUDGET, DEEP_SEGMENTS, SHARED_BUDGET, agentOf, crawlAgentCount, isCrawler, setCrawlClock, takeCrawl } from "../src/http/crawl.ts";
12 import { serve, type Servers } from "../src/http/server.ts";
13 import { crawlerShare, createPerfRecord } from "../src/perf/record.ts";
14
15 let dataDir: string;
16 let app: App;
17 let servers: Servers;
18 let alice: User;
19 let sha = "";
20
21 const BOT = "Mozilla/5.0 (compatible; Reflectionbot/1.0; +https://example.test/bot)";
22 const PASSWORD = "a-good-passphrase";
23 /** A token for `username`, minted through the API as a client would. */
24 const mintToken = async (username: string, scope: string): Promise<string> => {
25 const res = await fetch(`${servers.url}/api/v1/tokens`, {
26 method: "POST",
27 headers: { "content-type": "application/json" },
28 body: JSON.stringify({ adminConfirm: scope === "admin", username, password: PASSWORD, name: "t", scope }),
29 });
30 return ((await res.json()) as { token: string }).token;
31 };
32 const get = (path: string, headers: Record<string, string> = {}) => fetch(`${servers.url}${path}`, { headers, redirect: "manual" });
33
34 beforeAll(async () => {
35 dataDir = await mkdtemp(join(tmpdir(), "forge-crawl-"));
36 app = createApp({ dataDir, port: 0 });
37 servers = await serve(app);
38 const u = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", isAdmin: true });
39 if (!u.ok) throw new Error(u.error);
40 alice = u.value;
41 const r = await createRepo(app, alice, { name: "open", visibility: "public" });
42 if (!r.ok) throw new Error(String(r.error));
43 const c = await commitToBranch(app, alice, getRepo(app.db, "alice", "open")!, { message: "Start", files: [{ path: "a.txt", content: "hi\n" }] });
44 if (!c.ok) throw new Error(String(c.error));
45 const git = Bun.spawnSync(["git", "--git-dir", join(dataDir, "repos", "alice", "open.git"), "rev-parse", "HEAD"]);
46 sha = git.stdout.toString().trim();
47 });
48
49 afterAll(async () => {
50 servers.stop();
51 app.close();
52 await rm(dataDir, { recursive: true, force: true });
53 });
54
55 test("robots.txt disallows the deep paths at every ref, asks for a crawl delay, and leaves roots open", async () => {
56 const lines = (await (await get("/robots.txt")).text()).split("\n");
57 for (const segment of DEEP_SEGMENTS) expect(lines).toContain(`Disallow: /*/*/${segment}`);
58 expect(lines).toContain("Disallow: /origins/");
59 expect(lines).toContain("Crawl-delay: 10");
60 expect(lines).not.toContain("Disallow: /");
61 });
62
63 test("a crawler is told by its User-Agent's product token, a browser and curl are not crawlers", () => {
64 expect(agentOf(BOT)).toBe("reflectionbot");
65 expect(isCrawler(new Request("http://x/", { headers: { "user-agent": BOT } }))).toBe(true);
66 expect(isCrawler(new Request("http://x/", { headers: { "user-agent": "Mozilla/5.0 Firefox/130.0" } }))).toBe(false);
67 expect(isCrawler(new Request("http://x/", { headers: { "user-agent": "curl/8.0" } }))).toBe(false);
68 expect(isCrawler(new Request("http://x/", { method: "POST", headers: { "user-agent": BOT } }))).toBe(false);
69 });
70
71 const from = (ip: string, ua = BOT) => ({ "user-agent": ua, "x-forwarded-for": ip });
72
73 test("past its deep budget an address gets 429 with Retry-After; a person on the same path is not throttled", async () => {
74 let t = 1_000_000;
75 setCrawlClock(app, () => t);
76 const deep = `/alice/open/blob/${sha}/a.txt`;
77 for (let i = 0; i < CRAWL_BUDGET.deep; i++) expect((await get(deep, from("10.0.0.1"))).status).toBe(200);
78 const refused = await get(deep, from("10.0.0.1"));
79 expect(refused.status).toBe(429);
80 expect(Number(refused.headers.get("retry-after"))).toBeGreaterThanOrEqual(1);
81 expect(refused.headers.get("x-content-type-options")).toBe("nosniff");
82 // The same address is refused the whole deep family, not just that URL, and not by rotating its name.
83 expect((await get("/alice/open/commits", from("10.0.0.1"))).status).toBe(429);
84 expect((await get(deep, from("10.0.0.1", "Rotatedbot/9.9"))).status).toBe(429);
85 expect((await get(deep, { "x-forwarded-for": "10.0.0.1", "user-agent": "" })).status).toBe(429);
86 // Another address has its own budget.
87 expect((await get(deep, from("10.0.0.2"))).status).toBe(200);
88 // A person — no crawler UA — is never counted.
89 for (let i = 0; i < CRAWL_BUDGET.deep * 3; i++) expect((await get(deep, from("10.0.0.1", "Mozilla/5.0 Firefox/130.0"))).status).toBe(200);
90 // The budget is a minute: it comes back.
91 t += 61_000;
92 expect((await get(deep, from("10.0.0.1"))).status).toBe(200);
93 });
94
95 test("a token is never counted, even with a bot User-Agent", async () => {
96 const token = await mintToken("alice", "admin");
97 const deep = `/alice/open/blob/${sha}/a.txt`;
98 for (let i = 0; i < CRAWL_BUDGET.deep * 3; i++)
99 expect((await get(deep, { ...from("10.0.0.9"), authorization: `Bearer ${token}` })).status).toBe(200);
100 });
101
102 test("a spoofed name cannot spend a real crawler's budget: the address is spent first", async () => {
103 setCrawlClock(app, () => 2_000_000);
104 const deep = `/alice/open/blob/${sha}/a.txt`;
105 // One script sends the real crawler's name from one address: it stops at the address budget.
106 let ok = 0;
107 for (let i = 0; i < 40; i++) if ((await get(deep, from("10.1.0.1", "Googlebot/2.1"))).status === 200) ok++;
108 expect(ok).toBe(CRAWL_BUDGET.deep);
109 // The real one, from its own address, still gets in.
110 expect((await get(deep, from("10.1.0.2", "Googlebot/2.1"))).status).toBe(200);
111 });
112
113 test("many addresses sharing one name meet the agent cap; many names meet the shared ceiling", async () => {
114 setCrawlClock(app, () => 3_000_000);
115 const deep = `/alice/open/blob/${sha}/a.txt`;
116 let ok = 0;
117 for (let i = 0; i < AGENT_BUDGET.deep + 6; i++) if ((await get(deep, from(`10.2.0.${i + 1}`, "Swarmbot/1.0"))).status === 200) ok++;
118 expect(ok).toBe(AGENT_BUDGET.deep);
119 setCrawlClock(app, () => 4_000_000);
120 ok = 0;
121 for (let i = 0; i < SHARED_BUDGET.deep + 6; i++) if ((await get(deep, from(`10.3.${i}.1`, `Name${i}bot/1.0`))).status === 200) ok++;
122 expect(ok).toBe(SHARED_BUDGET.deep);
123 });
124
125 test("rotating names does not grow the limiters without bound", () => {
126 setCrawlClock(app, () => 6_000_000);
127 for (let i = 0; i < 600; i++)
128 takeCrawl(app, new Request("http://x/a/b", { headers: { "user-agent": `Spin${i}bot/1.0` } }), "/a/b");
129 expect(crawlAgentCount(app)).toBeLessThanOrEqual(257);
130 });
131
132 test("while the main thread is behind a crawler's budget shrinks", async () => {
133 let t = 7_000_000;
134 setCrawlClock(app, () => t, true);
135 const deep = `/alice/open/commits`;
136 const allowed = Math.floor(CRAWL_BUDGET.deep / 4);
137 for (let i = 0; i < allowed; i++) expect((await get(deep, { "user-agent": "Lagbot/1.0", "x-forwarded-for": "10.4.0.1" })).status).toBe(200);
138 expect((await get(deep, { "user-agent": "Lagbot/1.0", "x-forwarded-for": "10.4.0.1" })).status).toBe(429);
139 setCrawlClock(app, () => t, false);
140 });
141
142 test("a blob or commit at a full sha is immutable for an anonymous reader; a branch name is not", async () => {
143 const blob = await get(`/alice/open/blob/${sha}/a.txt`);
144 expect(blob.headers.get("cache-control")).toBe("public, max-age=31536000, s-maxage=86400, immutable");
145 expect(blob.headers.get("vary")).toContain("Cookie");
146 const commit = await get(`/alice/open/commit/${sha}`);
147 expect(commit.headers.get("cache-control")).toBe("public, max-age=31536000, s-maxage=86400, immutable");
148 const branch = await get(`/alice/open/blob/main/a.txt`);
149 expect(branch.headers.get("cache-control") ?? "").not.toContain("immutable");
150 // A cookie that carries no identity is still anonymous; one that does is not.
151 const theme = await get(`/alice/open/blob/${sha}/a.txt`, { cookie: "forge_theme=dark" });
152 expect(theme.headers.get("cache-control")).toContain("immutable");
153 const signedIn = await get(`/alice/open/blob/${sha}/a.txt`, { cookie: `forge_session=${createSession(app.db, alice.id).token}` });
154 expect(signedIn.headers.get("cache-control") ?? "").not.toContain("public");
155 });
156
157 test("the performance record knows the crawlers' share of answering time", () => {
158 const record = createPerfRecord();
159 const at = Date.now();
160 record.add("/:owner/:name/blob/:ref/*path", { at, ms: 300, signedIn: false, crawler: true, phases: [] });
161 record.add("/:owner/:name/blob/:ref/*path", { at, ms: 100, signedIn: true, phases: [] });
162 const share = crawlerShare(record, at - 1000);
163 expect(share).toEqual({ requests: 2, crawlerRequests: 1, ms: 400, crawlerMs: 300, share: 0.75 });
164 });
165
166 test("the crawler board door is an admin's: a non-admin and an anonymous caller are told it is not there, an admin gets the shape", async () => {
167 expect((await get("/api/v1/admin/crawlers")).status).toBe(404);
168 const bob = await createUser(app.db, { origin: "seed", trusted: true, username: "bob", email: "[email protected]", password: "a-good-passphrase" });
169 if (!bob.ok) throw new Error(bob.error);
170 const bobToken = await mintToken("bob", "read");
171 expect((await get("/api/v1/admin/crawlers", { authorization: `Bearer ${bobToken}` })).status).toBe(404);
172 const body = (await (await get("/api/v1/admin/crawlers", { authorization: `Bearer ${await mintToken("alice", "admin")}` })).json()) as any;
173 expect(body.window_minutes).toBe(15);
174 expect(body.refused.length).toBeGreaterThan(0);
175 expect(body.refused_by_budget.address).toBeGreaterThan(0);
176 expect(body.refused_by_budget.agent).toBeGreaterThan(0);
177 expect(body.refused_by_budget.shared).toBeGreaterThan(0);
178 });
test/forge-webspec.test.ts +7 −3
@@ -24,12 +24,15 @@ quietLogs();
2424
2525 const PATHS = ["/", "/signin", "/signup", "/explore", "/alice/notes", "/alice/notes/issues", "/alice/notes/issues/1", "/alice/notes/tree/main", "/alice/notes/blob/main/a.txt"];
2626
27 const CACHING = "https://specification.website/#caching";
28
2729 /** `check path` -> [section, why it stands]. Every one is expected on every page it names; one fixed fails the test, one new fails it too. */
28 const KNOWN: Record<string, [string, string]> = {};
30 const KNOWN: Record<string, [string, string]> = {
31 "cache-html /alice/notes/commit/<sha>": [CACHING, "commit-addressed pages are immutable by design (#8129)"],
32 };
2933
3034 /** The owner's pages: signed in, so no-store by design; webspec's caching findings on them are not asserted away, they are named. */
3135 const SIGNED_IN = ["/map", "/system", "/boards", "/settings/crew", "/alice/notes/pulls/2", "/settings", "/audit", "/settings/backups", "/fleet/graph"];
32 const CACHING = "https://specification.website/#caching";
3336 const NO_ETAG = "signed-in HTML is no-store by design: it is per-person, so there is no validator to revalidate against";
3437 const KNOWN_SIGNED_IN: Record<string, [string, string]> = {
3538 "etag /system": [CACHING, NO_ETAG],
@@ -98,7 +101,8 @@ const through = (signedIn: boolean) => async (input: string, init?: RequestInit)
98101
99102 test("the forge's signed-out pages raise exactly the findings named in KNOWN", async () => {
100103 const found = await webspec("http://forge.test", PATHS, through(false), { site: true });
101 const keyOf = (f: { check: string; path: string }) => `${f.check} ${f.path}`;
104 // #8129: a page at a full commit id names bytes that never change, so the edge may keep it (#8016); the id differs per run, so the path is keyed as `<sha>`.
105 const keyOf = (f: { check: string; path: string }) => `${f.check} ${f.path.replace(/\/commit\/[0-9a-f]{40}$/, "/commit/<sha>")}`;
102106 // New: not named, or citing another section than the one named.
103107 expect(found.filter((f) => KNOWN[keyOf(f)]?.[0] !== f.section).map((f) => `${keyOf(f)}: ${f.message}`)).toEqual([]);
104108 // Fixed: named but no longer found, so the entry must go.
test/palette-roster.test.ts +1
@@ -77,6 +77,7 @@ test("a site-admin person is offered the whole roster", () => {
7777 ["telemetry", "Telemetry", "which agents may read flow and system; enter flips the grant"],
7878 ["capacity", "Capacity", "concurrency and guards, live; e changes a settable one"],
7979 ["resilience", "Resilience", "where snapshots go and the mirror; each protection green, red or untested"],
80 ["crawlers", "Crawlers", "their share of answering time, and who was refused"],
8081 ["api", "API", "the forge's endpoints; run a GET"],
8182 ["changelog", "Changelog", "the last 30 releases and their pulls"],
8283 ["releases", "Releases", "the live release and the kept ones"],
test/tui-crawlers.test.ts added +64
@@ -0,0 +1,64 @@
1 /** #8129: the crawler share and refusals in the terminal, a forge admin's who is a person; a member is offered nothing. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { createUser, type User } from "../src/core/users.ts";
8 import { setCrawlClock, takeCrawl } from "../src/http/crawl.ts";
9 import { createForgeApp, forgeCommands } from "../src/tui/forge.ts";
10 import { crawlersScreen } from "../src/tui/screens/crawlers.ts";
11 import { headlessTerminal } from "../src/tui/term/terminal.ts";
12 import { quietLogs } from "./helpers/globals.ts";
13 import { settleApp } from "./helpers/tui.ts";
14
15 quietLogs();
16
17 let dir: string;
18 let app: App;
19 let carol: User;
20 let bob: User;
21 const identity = (user: User) => ({ sshUsername: user.username, remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
22
23 beforeAll(async () => {
24 dir = await mkdtemp(join(tmpdir(), "forge-tui-crawlers-"));
25 app = createApp({ dataDir: dir, port: 0 });
26 const mk = async (username: string, isAdmin: boolean) => {
27 const r = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", isAdmin });
28 if (!r.ok) throw new Error(r.error);
29 return r.value;
30 };
31 carol = await mk("carol", true);
32 bob = await mk("bob", false);
33 }, 30_000);
34
35 afterAll(async () => {
36 app.close();
37 await rm(dir, { recursive: true, force: true });
38 });
39
40 test("only an admin who is a person is offered Crawlers", () => {
41 const door = (u: User) => {
42 const { app: tui, session } = createForgeApp(app, identity(u), { terminal: headlessTerminal(80, 24), exitHooks: false });
43 const ids = forgeCommands(session)(tui.ctx).map((c) => c.id);
44 tui.stop?.();
45 return ids;
46 };
47 expect(door(carol)).toContain("crawlers");
48 expect(door(bob)).not.toContain("crawlers");
49 });
50
51 test("the screen shows the share, the budgets that refused and the agent refused", async () => {
52 setCrawlClock(app, () => 1_000_000);
53 const req = () => new Request("http://x/a/b/blob/main/f", { headers: { "user-agent": "Screenbot/1.0" } });
54 for (let i = 0; i < 8; i++) takeCrawl(app, req(), "/a/b/blob/main/f");
55 const { app: tui, session } = createForgeApp(app, identity(carol), { terminal: headlessTerminal(140, 20), exitHooks: false });
56 tui.start();
57 tui.ctx.push(crawlersScreen(session, tui.ctx));
58 await settleApp(tui);
59 const text = tui.screen.toText();
60 expect(text).toContain("Share of answering time");
61 expect(text).toContain("address 2");
62 expect(text).toContain("screenbot");
63 tui.stop?.();
64 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs