trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns13g rEventsFlows

Anonymous readers cannot cost the forge either: deep git pages share one budget, and /tree answers from a cache #8203

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/8199-anonymous-readers-cannot-cost-the-forge → main 2 commits fork

  • Merged. mason/8199-anonymous-readers-cannot-cost-the-forge was merged into main as d18d601.

Review

Sign in to review this change.
11 files changed +336 −28
docs/features/crawl-budget.md +8
@@ -6,3 +6,11 @@
66 - Usage counts carry a `crawler` role. The crawlers' share of answering time over the last 15 minutes, the refusals by agent and by which budget refused (address, agent, shared) are read at `GET /api/v1/admin/crawlers` (admin) and in the terminal: `:` → "Crawlers" (a forge admin who is a person).
77
88 Not done: a panel on a board page (the performance board has no page yet; the share is the API above), and a budget by time spent rather than by request count.
9
10 ## Anonymous readers with a browser User-Agent (#8199)
11
12 - Anyone with no session and no token, whatever the User-Agent, is counted on the deep pages (`DEEP_SEGMENTS`): `STRANGER_BUDGET` is 30 a minute per address (an IPv6 client by /64), divided by 4 while the loop is late. There is no bucket for all such readers together: a handful of addresses could spend it and refuse every other person; the aggregate is held by the tree cache and the loop-lag divisor. The explorer's lazy folder expansion (`?fragment=explorer`) is not counted. Past it: 429 with `Retry-After` before page work. Other pages are not counted for browser UAs; a signed-in person or token never is.
13 - `/tree/:ref` for an anonymous reader of a public repository resolves the branch to its commit from the ref files (no process), then serves the rendered listing and explorer from a cache keyed on repo, commit, ref, path and `?all` (`src/web/pages/tree_cache.ts`, 60 s, 32 MB). A push moves the commit, so the next read misses; only the ref switcher's menu can be up to 60 s old. A miss reads git as before (`ls-tree` etc. are child processes, off the main thread).
14 - Intended: only this anonymous path links by commit; a signed-in person's listing keeps the branch name in its links (`linkRef` is set only there).
15 - That listing's links (entries and README) name the full commit, so the blob and tree pages behind them carry the immutable headers; a tree at a full id is `immutable` for anonymous readers too.
16 - The crawler board (`/api/v1/admin/crawlers`, terminal "Crawlers") adds `deep_stranger_requests`, `deep_stranger_ms` and `refused_by_budget.stranger`.
src/contract/openapi/routes/getApiV1AdminCrawlers.ts +1 −1
@@ -4,7 +4,7 @@ export const route: Route = {
44 method: "get",
55 path: "/api/v1/admin/crawlers",
66 summary:
7 "Read the crawlers' share of the last 15 minutes of answering time (a person admin's token, any scope); answers { window_minutes, requests, crawler_requests, ms, crawler_ms, share, refused: [{ agent, refused }], refused_by_budget: { address, agent, shared } }",
7 "Read the crawlers' share of the last 15 minutes of answering time (a person admin's token, any scope); answers { window_minutes, requests, crawler_requests, ms, crawler_ms, deep_stranger_requests, deep_stranger_ms, share, refused: [{ agent, refused }], refused_by_budget: { address, agent, shared, stranger } }",
88 auth: true,
99 refusals: { not_found: "the token is not a person who administers the forge" },
1010 };
src/http/crawl.ts +45 −4
@@ -11,7 +11,7 @@
1111 * product token of the User-Agent, a second cap only, since anyone can
1212 * send `Googlebot`), and one shared budget for every such crawler, so their
1313 * total cost has a ceiling however they rotate. A browser-looking User-Agent
14 * is not counted here; `stranger_reads.ts` bounds those. Deep paths (blob, raw, commits, blame, diffs at every ref) have a
14 * is counted only on the deep paths, with a looser budget of its own. Deep paths (blob, raw, commits, blame, diffs at every ref) have a
1515 * far smaller budget than the rest. When the event loop is late the budgets
1616 * shrink, so a crawler is the first to wait. A signed-in person or a token is
1717 * never counted here: the check is made only for a request `resolveAuth`
@@ -29,6 +29,15 @@ export const CRAWL_BUDGET = { page: 60, deep: 6 } as const;
2929 export const AGENT_BUDGET = { page: 120, deep: 12 } as const;
3030 /** The ceiling for every crawler together. */
3131 export const SHARED_BUDGET = { page: 240, deep: 24 } as const;
32 /**
33 * Deep pages per minute, per address, for a reader whose User-Agent looks like
34 * a browser (#8199): anyone can write that header, so what it may spend on the
35 * pages that cost a git walk is bound too. Far looser than a crawler's, since
36 * people click. There is deliberately no bucket for all such readers
37 * together: a few addresses could spend it and shut every other person out.
38 * The aggregate is held by the tree cache and by the loop-lag divisor.
39 */
40 export const STRANGER_BUDGET = { address: 30 } as const;
3241 /** The budgets divide by this while the event loop runs late. */
3342 export const BEHIND_DIVISOR = 4;
3443 /** An average wake later than this (ms) is "behind". */
@@ -46,6 +55,9 @@ const CRAWLER = /bot\b|bot[/ ;)-]|crawl|spider|slurp|scrap|archiver|reflection|f
4655 export const DEEP_SEGMENTS = ["blob", "raw", "commits", "commit", "blame", "compare", "diff", "tree", "history"] as const;
4756 const DEEP = new RegExp(`^/(?:origins/[^/]+/r/)?[^/]+/[^/]+/(?:${DEEP_SEGMENTS.join("|")})(?:/|$)`);
4857
58 /** Whether `pathname` is a page that costs a git walk or a diff. */
59 export const isDeepPath = (pathname: string): boolean => DEEP.test(pathname);
60
4961 /** What a crawler never spends from: the files it is asked to read, and the probes. */
5062 const FREE = /^\/(?:robots\.txt|sitemap\.xml|healthz|readyz|assets\/)/;
5163
@@ -64,7 +76,7 @@ export function isCrawler(request: Request): boolean {
6476 return ua.length === 0 || CRAWLER.test(ua.slice(0, 300));
6577 }
6678
67 type Scope = "address" | "agent" | "shared";
79 type Scope = "address" | "agent" | "shared" | "stranger";
6880 interface Pair {
6981 readonly page: RateLimiter;
7082 readonly deep: RateLimiter;
@@ -73,6 +85,8 @@ interface State {
7385 readonly address: Pair;
7486 readonly agent: Pair;
7587 readonly shared: Pair;
88 /** Deep pages read by browser-looking strangers, per address. */
89 readonly strangerAddress: RateLimiter;
7690 readonly refused: Map<string, number>;
7791 readonly refusedBy: Record<Scope, number>;
7892 readonly clock: { now: () => number; behind: boolean };
@@ -108,8 +122,9 @@ function stateOf(app: App): State {
108122 address: pair(CRAWL_BUDGET),
109123 agent: pair(AGENT_BUDGET),
110124 shared: pair(SHARED_BUDGET),
125 strangerAddress: new RateLimiter(limits(STRANGER_BUDGET.address), () => clock.now()),
111126 refused: new Map(),
112 refusedBy: { address: 0, agent: 0, shared: 0 },
127 refusedBy: { address: 0, agent: 0, shared: 0, stranger: 0 },
113128 clock,
114129 };
115130 states.set(app, state);
@@ -136,7 +151,8 @@ export interface CrawlVerdict {
136151 * crawler, or inside its budget). Otherwise the refusal to answer with.
137152 */
138153 export function takeCrawl(app: App, request: Request, pathname: string): CrawlVerdict | undefined {
139 if (!isCrawler(request) || FREE.test(pathname)) return undefined;
154 if (!isCrawler(request)) return takeStrangerDeep(app, request, pathname);
155 if (FREE.test(pathname)) return undefined;
140156 watchLag();
141157 const state = stateOf(app);
142158 const behind = state.clock.behind || mainThreadBehind();
@@ -173,6 +189,26 @@ export function takeCrawl(app: App, request: Request, pathname: string): CrawlVe
173189 return undefined;
174190 }
175191
192 /**
193 * A browser-looking reader with no session and no token, on a deep page:
194 * counted per address, divided while the loop is behind, refused before any
195 * page work (#8199). The explorer's lazy folder expansion
196 * (`?fragment=explorer`) is a click, not a page, and is not counted.
197 */
198 function takeStrangerDeep(app: App, request: Request, pathname: string): CrawlVerdict | undefined {
199 if (request.method !== "GET" && request.method !== "HEAD") return undefined;
200 if (!DEEP.test(pathname)) return undefined;
201 if (new URL(request.url).searchParams.get("fragment") === "explorer") return undefined;
202 watchLag();
203 const state = stateOf(app);
204 const scale = state.clock.behind || mainThreadBehind() ? BEHIND_DIVISOR : 1;
205 state.strangerAddress.setLimits(limits(Math.max(1, Math.floor(STRANGER_BUDGET.address / scale))));
206 const own = state.strangerAddress.take(bucketOf(clientAddress(request)), "ro");
207 if (own.allowed) return undefined;
208 state.refusedBy.stranger++;
209 return { retryAfter: own.retryAfter, agent: "browser", scope: "stranger" };
210 }
211
176212 /** Requests refused so far in this process, by agent, most first. */
177213 export const crawlRefusals = (app: App): { agent: string; refused: number }[] =>
178214 [...stateOf(app).refused]
@@ -201,3 +237,8 @@ const noted = new WeakSet<Request>();
201237 /** The server says this request was a crawler's, for the usage count and the performance record. */
202238 export const noteCrawler = (request: Request): void => void noted.add(request);
203239 export const wasCrawler = (request: Request): boolean => noted.has(request);
240
241 const notedDeep = new WeakSet<Request>();
242 /** The server says this was a browser-looking stranger's read of a deep page, for the crawler board. */
243 export const noteDeepStranger = (request: Request): void => void notedDeep.add(request);
244 export const wasDeepStranger = (request: Request): boolean => notedDeep.has(request);
src/http/server.ts +3 −3
@@ -76,7 +76,7 @@ import {
7676 verifyPasswordThrottled,
7777 } from "../core/users.ts";
7878 import { activeEdgeToken, edgeIdentity } from "./edge.ts";
79 import { crawlRefused, isCrawler, noteCrawler, takeCrawl, wasCrawler } from "./crawl.ts";
79 import { crawlRefused, isCrawler, isDeepPath, noteCrawler, noteDeepStranger, takeCrawl, wasCrawler, wasDeepStranger } from "./crawl.ts";
8080 import { liveResponse, readyResponse } from "./health.ts";
8181 import { running } from "../kernel/version.ts";
8282 import { runGit } from "../git/env.ts";
@@ -2268,7 +2268,7 @@ export async function serve(app: App): Promise<Servers> {
22682268 const end = performance.now();
22692269 perfRecord.add(
22702270 tag.pattern,
2271 { at: Date.now(), ms, signedIn: tag.role !== "anon" && tag.role !== "crawler", crawler: wasCrawler(request), phases: parseServerTiming(header) },
2271 { at: Date.now(), ms, signedIn: tag.role !== "anon" && tag.role !== "crawler", crawler: wasCrawler(request), deepStranger: wasDeepStranger(request), phases: parseServerTiming(header) },
22722272 );
22732273 loopWatch.finished(tag.pattern, started, end);
22742274 } catch {
@@ -2640,7 +2640,7 @@ async function answerGet(
26402640 if (isCrawler(request)) {
26412641 noteCrawler(request);
26422642 noteUsage(request, { pattern: matched.pattern, role: "crawler" });
2643 }
2643 } else if (isDeepPath(url.pathname) && url.searchParams.get("fragment") !== "explorer") noteDeepStranger(request);
26442644 if (crawl !== undefined) return crawlRefused(crawl);
26452645 }
26462646
src/perf/crawlers.ts +2
@@ -17,6 +17,8 @@ export function crawlReport(app: App, now = Date.now()) {
1717 crawler_requests: share.crawlerRequests,
1818 ms: Math.round(share.ms),
1919 crawler_ms: Math.round(share.crawlerMs),
20 deep_stranger_requests: share.strangerRequests,
21 deep_stranger_ms: Math.round(share.strangerMs),
2022 share: Math.round(share.share * 1000) / 1000,
2123 refused: crawlRefusals(app).slice(0, 50),
2224 refused_by_budget: crawlRefusalScopes(app),
src/perf/record.ts +10 −2
@@ -27,6 +27,8 @@ export interface Sample {
2727 readonly signedIn: boolean;
2828 /** An unauthenticated request whose User-Agent names a crawler (`http/crawl.ts`). */
2929 readonly crawler?: boolean;
30 /** An unauthenticated, browser-looking read of a deep git page (#8199). */
31 readonly deepStranger?: boolean;
3032 readonly phases: readonly PhaseSample[];
3133 }
3234
@@ -138,13 +140,16 @@ export interface CrawlerShare {
138140 /** Summed time to answer, ms, of every request and of the crawlers' alone. */
139141 readonly ms: number;
140142 readonly crawlerMs: number;
143 /** The same for anonymous readers with a browser's User-Agent on the deep git pages (#8199). */
144 readonly strangerRequests: number;
145 readonly strangerMs: number;
141146 /** crawlerMs / ms, 0 when nothing was answered. */
142147 readonly share: number;
143148 }
144149
145150 /** What share of the server's answering time crawlers took, from `since` on. */
146151 export function crawlerShare(record: PerfRecord, since: number): CrawlerShare {
147 let requests = 0, crawlerRequests = 0, ms = 0, crawlerMs = 0;
152 let requests = 0, crawlerRequests = 0, ms = 0, crawlerMs = 0, strangerRequests = 0, strangerMs = 0;
148153 for (const pattern of record.patterns())
149154 for (const s of record.traffic(pattern, since)) {
150155 requests++;
@@ -152,7 +157,10 @@ export function crawlerShare(record: PerfRecord, since: number): CrawlerShare {
152157 if (s.crawler === true) {
153158 crawlerRequests++;
154159 crawlerMs += s.ms;
160 } else if (s.deepStranger === true) {
161 strangerRequests++;
162 strangerMs += s.ms;
155163 }
156164 }
157 return { requests, crawlerRequests, ms, crawlerMs, share: ms > 0 ? crawlerMs / ms : 0 };
165 return { requests, crawlerRequests, ms, crawlerMs, strangerRequests, strangerMs, share: ms > 0 ? crawlerMs / ms : 0 };
158166 }
src/tui/screens/crawlers.ts +2 −1
@@ -23,7 +23,8 @@ export function crawlersScreen(f: Session, ctx: Ctx): Screen {
2323 const b = r.refused_by_budget;
2424 return [
2525 { title: `Share of answering time: ${Math.round(r.share * 100)}%`, meta: `${r.crawler_ms} of ${r.ms} ms, ${r.crawler_requests} of ${r.requests} requests, last ${r.window_minutes} min` },
26 { title: "Refused (429) by budget", meta: `address ${b.address} · agent ${b.agent} · all crawlers ${b.shared}` },
26 { title: `Browser-looking strangers on deep git pages: ${r.deep_stranger_requests} requests`, meta: `${r.deep_stranger_ms} ms, last ${r.window_minutes} min` },
27 { title: "Refused (429) by budget", meta: `address ${b.address} · agent ${b.agent} · all crawlers ${b.shared} · browser strangers ${b.stranger}` },
2728 ...r.refused.map((a) => ({ title: a.agent, meta: `${a.refused} refused` })),
2829 ];
2930 };
src/web/pages/repo.ts +48 −16
@@ -9,6 +9,7 @@
99 */
1010
1111 import { immutableWhenAddressed } from "../../http/addressed.ts";
12 import { cachedTree, keepTree, resolveCommit } from "./tree_cache.ts";
1213 import { behindLine, followingLine } from "./upstream-follow.ts";
1314 import { upstreamOffer } from "../../actions/offer_upstream.ts";
1415 import { offerUpstream } from "./offer-upstream.ts";
@@ -915,13 +916,16 @@ export async function renderCode(
915916 rail?: Html;
916917 /** The README, when the caller has already read it (and timed the read). */
917918 readme?: Html;
919 /** What the listing's links name instead of `ref`: a full commit id, whose pages a cache may keep (#8199). */
920 linkRef?: string;
918921 },
919922 ): Promise<Html> {
920923 const base = scope.base;
921924 const { ref, path } = o;
925 const linked = o.linkRef ?? ref;
922926 const readme =
923927 o.readme ??
924 (await readmeFor(scope.source, ref, o.entries, path, `${base}/blob/${encodeURIComponent(ref)}`));
928 (await readmeFor(scope.source, linked, o.entries, path, `${base}/blob/${encodeURIComponent(linked)}`));
925929 const historyParams = new URLSearchParams({ ref });
926930 if (path !== "") historyParams.set("path", path);
927931 const bar =
@@ -954,7 +958,7 @@ export async function renderCode(
954958 })}
955959 </div>`;
956960 const main = html`${bar} ${o.extra ?? null}
957 ${treeListing(base, ref, path, o.entries, o.all === true)}
961 ${treeListing(base, linked, path, o.entries, o.all === true)}
958962 <div class="mt">${readme}</div>`;
959963 return o.rail === undefined
960964 ? main
@@ -1441,11 +1445,35 @@ export async function treePage(ctx: Ctx): Promise<Response> {
14411445 const path = ctx.params.path ?? "";
14421446 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
14431447 const git = gitClock(ctx.request);
1444 const entries = await git.read(() => scope.source.tree(ref, path));
1448 const fragmentOnly = ctx.url.searchParams.get("fragment") === "explorer";
1449 const all = ctx.url.searchParams.get("all") === "1";
1450
1451 // An anonymous reader of a public repository is served the listing of the
1452 // commit the branch names, kept under that commit (`tree_cache.ts`), and
1453 // its links name the commit, so the edge may keep what they lead to.
1454 const anonymous = ctx.auth.user === undefined && repo.visibility === "public";
1455 const commit =
1456 anonymous && !fragmentOnly && isSafeRef(ref)
1457 ? await resolveCommit(repoDir(ctx.app.reposRoot, repo.ownerName, repo.name), ref)
1458 : undefined;
1459 if (commit !== undefined) {
1460 const hit = cachedTree(repo.id, commit, ref, path, all);
1461 if (hit !== undefined) {
1462 git.done();
1463 return immutableWhenAddressed(
1464 true,
1465 repo,
1466 ref,
1467 codePage(ctx, scope, { title: `${path || ref} · ${repo.name}`, ref, explorer: hit.explorer, content: hit.content }),
1468 );
1469 }
1470 }
1471
1472 const entries = await git.read(() => scope.source.tree(commit ?? ref, path));
14451473 if (!entries.ok) return notFoundPage(ctx);
14461474
14471475 // The explorer's lazy expansion asks for exactly one directory.
1448 if (ctx.url.searchParams.get("fragment") === "explorer") {
1476 if (fragmentOnly) {
14491477 git.done();
14501478 return fragment(explorerFragment(scope.base, ref, path, entries.value), {
14511479 request: ctx.request,
@@ -1461,23 +1489,27 @@ export async function treePage(ctx: Ctx): Promise<Response> {
14611489 git.done();
14621490 // `render` includes the README read: it is fetched while the page is built.
14631491 const began = performance.now();
1492 const explorerHtml = explorerColumn(scope, explorer, {
1493 finder: true,
1494 actions: newFileAction(scope, ref, path),
1495 });
1496 const content = await renderCode(scope, {
1497 refs,
1498 ref,
1499 path,
1500 entries: entries.value,
1501 all,
1502 ...(commit === undefined ? {} : { linkRef: commit }),
1503 });
1504 if (commit !== undefined) keepTree(repo.id, commit, ref, path, all, { explorer: explorerHtml, content });
14641505 const response = codePage(ctx, scope, {
14651506 title: `${path || ref} · ${repo.name}`,
14661507 ref,
1467 explorer: explorerColumn(scope, explorer, {
1468 finder: true,
1469 actions: newFileAction(scope, ref, path),
1470 }),
1471 content: await renderCode(scope, {
1472 refs,
1473 ref,
1474 path,
1475 entries: entries.value,
1476 all: ctx.url.searchParams.get("all") === "1",
1477 }),
1508 explorer: explorerHtml,
1509 content,
14781510 });
14791511 notePhase(ctx.request, "render", performance.now() - began);
1480 return response;
1512 return immutableWhenAddressed(anonymous, repo, ref, response);
14811513 }
14821514
14831515 // GET /:owner/:name/blob/:ref/*path
src/web/pages/tree_cache.ts added +48
@@ -0,0 +1,48 @@
1 /**
2 * The rendered listing of one directory at one commit (#8199).
3 *
4 * A branch page costs a tree read, a ref list, the explorer's walk and the
5 * README render, and strangers ask for the same few directories again and
6 * again. For an anonymous reader of a public repository the markup depends
7 * on nothing but the commit, the path and the ref spelled in the URL, so it
8 * is kept under that key. The branch is resolved to its commit once, from the
9 * ref files (no process); a push moves the branch, so the next request keys
10 * on the new commit and the old entry simply ages out. The ref switcher's
11 * menu is the one part a new branch elsewhere can change: it is at most
12 * `TTL_MS` stale. Nothing here is truth: any entry can be dropped.
13 */
14
15 import { isFullId } from "../../http/addressed.ts";
16 import { refTip } from "../../git/read.ts";
17 import { ViewCache } from "../../view/cache.ts";
18 import type { Html } from "../html.ts";
19
20 export const TTL_MS = 60_000;
21
22 export interface RenderedTree {
23 readonly explorer: Html;
24 readonly content: Html;
25 }
26
27 const cache = new ViewCache({ maxEntries: 512, maxBytes: 32 * 1024 * 1024 });
28
29 /** The commit `ref` names: a full id itself, a branch by its ref file; undefined for anything else (the caller reads git). */
30 export async function resolveCommit(dir: string, ref: string): Promise<string | undefined> {
31 if (isFullId(ref)) return ref;
32 return refTip(dir, `refs/heads/${ref}`);
33 }
34
35 const keyOf = (repoId: string, commit: string, ref: string, path: string, all: boolean): string =>
36 [repoId, commit, ref, path, all ? "all" : "window"].join("\0");
37
38 export const cachedTree = (repoId: string, commit: string, ref: string, path: string, all: boolean) =>
39 cache.get<RenderedTree>(keyOf(repoId, commit, ref, path, all));
40
41 export function keepTree(repoId: string, commit: string, ref: string, path: string, all: boolean, tree: RenderedTree): void {
42 cache.set(keyOf(repoId, commit, ref, path, all), tree, TTL_MS, tree.explorer.text.length + tree.content.text.length);
43 }
44
45 export const treeCacheStats = () => cache.stats();
46
47 /** Test seam: start from nothing. */
48 export const clearTreeCache = (): void => cache.clear();
test/anonymous-deep.test.ts added +168
@@ -0,0 +1,168 @@
1 /** #8199: a browser-looking stranger is bound on deep pages, /tree is served from a per-commit cache whose links name the commit, and the board shows it. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { mkdtemp, rename, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
8 import { getRepo } from "../src/core/repos.ts";
9 import { createSession } from "../src/core/sessions.ts";
10 import { createUser, type User } from "../src/core/users.ts";
11 import { BEHIND_DIVISOR, STRANGER_BUDGET, setCrawlClock } from "../src/http/crawl.ts";
12 import { serve, type Servers } from "../src/http/server.ts";
13 import { crawlerShare, createPerfRecord } from "../src/perf/record.ts";
14 import { crawlReport } from "../src/perf/crawlers.ts";
15 import { clearTreeCache, treeCacheStats } from "../src/web/pages/tree_cache.ts";
16
17 let dataDir: string;
18 let app: App;
19 let servers: Servers;
20 let alice: User;
21 let sha = "";
22
23 const BROWSER = "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 Chrome/130.0 Safari/537.36";
24 const from = (ip: string, extra: Record<string, string> = {}) => ({ "user-agent": BROWSER, "x-forwarded-for": ip, ...extra });
25 const get = (path: string, headers: Record<string, string> = {}) => fetch(`${servers.url}${path}`, { headers, redirect: "manual" });
26
27 beforeAll(async () => {
28 dataDir = await mkdtemp(join(tmpdir(), "forge-anon-deep-"));
29 app = createApp({ dataDir, port: 0 });
30 servers = await serve(app);
31 const u = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", isAdmin: true });
32 if (!u.ok) throw new Error(u.error);
33 alice = u.value;
34 const r = await createRepo(app, alice, { name: "open", visibility: "public" });
35 if (!r.ok) throw new Error(String(r.error));
36 const c = await commitToBranch(app, alice, getRepo(app.db, "alice", "open")!, { message: "Start", files: [{ path: "a.txt", content: "hi\n" }, { path: "README.md", content: "# Open\n" }] });
37 if (!c.ok) throw new Error(String(c.error));
38 const head = Bun.spawnSync(["git", "--git-dir", join(dataDir, "repos", "alice", "open.git"), "rev-parse", "HEAD"]);
39 sha = head.stdout.toString().trim();
40 });
41
42 afterAll(async () => {
43 servers.stop();
44 app.close();
45 await rm(dataDir, { recursive: true, force: true });
46 });
47
48 test("a browser User-Agent with no session shares a per-address and an overall deep budget; a signed-in person is never counted", async () => {
49 const t = 5_000_000;
50 setCrawlClock(app, () => t);
51 const deep = `/alice/open/blob/${sha}/a.txt`;
52 for (let i = 0; i < STRANGER_BUDGET.address; i++) expect((await get(deep, from("10.1.0.1"))).status).toBe(200);
53 const refused = await get(deep, from("10.1.0.1"));
54 expect(refused.status).toBe(429);
55 expect(Number(refused.headers.get("retry-after"))).toBeGreaterThanOrEqual(1);
56 // another address is not held back by it; the home page is not deep
57 expect((await get(deep, from("10.1.0.2"))).status).toBe(200);
58 expect((await get("/alice/open", from("10.1.0.1"))).status).toBe(200);
59 // a person on the refused address is never counted
60 const cookie = `forge_session=${createSession(app.db, alice.id).token}`;
61 expect((await get(deep, from("10.1.0.1", { cookie }))).status).toBe(200);
62 });
63
64 test("a quiet address is still served after many noisy ones: there is no bucket for everyone together", async () => {
65 setCrawlClock(app, () => 6_000_000);
66 const deep = `/alice/open/blob/${sha}/a.txt`;
67 for (let n = 0; n < 6; n++)
68 for (let i = 0; i < STRANGER_BUDGET.address + 1; i++) await get(deep, from(`10.2.0.${n + 1}`));
69 expect((await get(deep, from("10.2.9.9"))).status).toBe(200);
70 });
71
72 test("the budget divides while the loop is behind, and the explorer's folder expansion is not counted", async () => {
73 setCrawlClock(app, () => 6_500_000, true);
74 const deep = `/alice/open/blob/${sha}/a.txt`;
75 const allowed = Math.floor(STRANGER_BUDGET.address / BEHIND_DIVISOR);
76 for (let i = 0; i < allowed; i++) expect((await get(deep, from("10.4.0.1"))).status).toBe(200);
77 expect((await get(deep, from("10.4.0.1"))).status).toBe(429);
78 for (let i = 0; i < 40; i++) expect((await get("/alice/open/tree/main?fragment=explorer", from("10.4.0.2"))).status).toBe(200);
79 setCrawlClock(app, () => 6_500_000, false);
80 });
81
82 test("the board counts the browser-looking traffic and the refusals", () => {
83 const record = createPerfRecord();
84 const at = Date.now();
85 record.add("/:owner/:name/tree/:ref", { at, ms: 7000, signedIn: false, deepStranger: true, phases: [] });
86 expect(crawlerShare(record, at - 1000)).toMatchObject({ strangerRequests: 1, strangerMs: 7000, crawlerRequests: 0 });
87 expect(crawlReport(app).refused_by_budget.stranger).toBeGreaterThan(0);
88 });
89
90 test("a branch page resolves to its commit once, serves from the cache, links by commit, and a push shows at once", async () => {
91 setCrawlClock(app, () => 9_000_000);
92 clearTreeCache();
93 const first = await get("/alice/open/tree/main", from("10.3.0.1"));
94 expect(first.status).toBe(200);
95 const body = await first.text();
96 expect(body).toContain(`/alice/open/blob/${sha}/a.txt`);
97 expect(first.headers.get("cache-control") ?? "").not.toContain("immutable");
98 expect(treeCacheStats().entries).toBe(1);
99 const second = await get("/alice/open/tree/main", from("10.3.0.1"));
100 expect(await second.text()).toContain(`/alice/open/blob/${sha}/a.txt`);
101 expect(treeCacheStats().hits).toBeGreaterThanOrEqual(1);
102 // a commit-addressed listing is immutable for the edge
103 const addressed = await get(`/alice/open/tree/${sha}`, from("10.3.0.1"));
104 expect(addressed.headers.get("cache-control")).toContain("immutable");
105 // a signed-in person is neither served from nor given the shared cache
106 const cookie = `forge_session=${createSession(app.db, alice.id).token}`;
107 const mine = await get(`/alice/open/tree/${sha}`, from("10.3.0.1", { cookie }));
108 expect(mine.headers.get("cache-control") ?? "").not.toContain("public");
109 // a new commit moves the branch: the next read keys on it
110 const c = await commitToBranch(app, alice, getRepo(app.db, "alice", "open")!, { message: "More", files: [{ path: "b.txt", content: "b\n" }] });
111 if (!c.ok) throw new Error(String(c.error));
112 const after = await (await get("/alice/open/tree/main", from("10.3.0.1"))).text();
113 expect(after).toContain("b.txt");
114 });
115
116 test("a hit does not read git; a signed-in reader, a tag and ?all=1 behave as the cache says", async () => {
117 setCrawlClock(app, () => 12_000_000);
118 clearTreeCache();
119 const gitDir = join(dataDir, "repos", "alice", "open.git");
120 Bun.spawnSync(["git", "--git-dir", gitDir, "tag", "v1"]);
121 const anon = from("10.5.0.1");
122 await get("/alice/open/tree/main", anon);
123 expect(treeCacheStats().entries).toBe(1);
124
125 // the same read with the objects gone still answers: nothing was read from git
126 await rename(join(gitDir, "objects"), join(gitDir, "objects.away"));
127 try {
128 const hit = await get("/alice/open/tree/main", anon);
129 expect(hit.status).toBe(200);
130 expect(await hit.text()).toContain("a.txt");
131 expect((await get("/alice/open/tree/main/nowhere", anon)).status).toBe(404);
132 } finally {
133 await rename(join(gitDir, "objects.away"), join(gitDir, "objects"));
134 }
135
136 // ?all=1 has its own key
137 await get("/alice/open/tree/main?all=1", anon);
138 expect(treeCacheStats().entries).toBe(2);
139
140 // a tag is not resolved: it is read from git and kept out of the cache
141 const tagged = await get("/alice/open/tree/v1", anon);
142 expect(tagged.status).toBe(200);
143 expect(treeCacheStats().entries).toBe(2);
144
145 // a signed-in person neither reads nor fills the cache, and links by branch name
146 const before = treeCacheStats();
147 const cookie = `forge_session=${createSession(app.db, alice.id).token}`;
148 const mine = await (await get("/alice/open/tree/main", from("10.5.0.1", { cookie }))).text();
149 expect(mine).toContain("/alice/open/blob/main/a.txt");
150 expect(treeCacheStats()).toEqual(before);
151
152 // a private repository is not cached either, for its owner
153 const p = await createRepo(app, alice, { name: "hidden", visibility: "private" });
154 if (!p.ok) throw new Error(String(p.error));
155 await commitToBranch(app, alice, getRepo(app.db, "alice", "hidden")!, { message: "x", files: [{ path: "s.txt", content: "s\n" }] });
156 expect((await get("/alice/hidden/tree/main", from("10.5.0.1", { cookie }))).status).toBe(200);
157 expect((await get("/alice/hidden/tree/main", anon)).status).toBe(404);
158 expect(treeCacheStats().entries).toBe(2);
159 });
160
161 test("the pages behind the listing's commit links carry the immutable headers for an anonymous reader", async () => {
162 setCrawlClock(app, () => 13_000_000);
163 const body = await (await get("/alice/open/tree/main", from("10.6.0.1"))).text();
164 const blob = /\/alice\/open\/blob\/[0-9a-f]{40}\/a\.txt/.exec(body)?.[0];
165 expect(blob).toBeDefined();
166 expect((await get(blob!, from("10.6.0.1"))).headers.get("cache-control")).toContain("immutable");
167 expect((await get(blob!.replace("/blob/", "/tree/").replace("/a.txt", ""), from("10.6.0.1"))).headers.get("cache-control")).toContain("immutable");
168 });
test/crawl-budget.test.ts +1 −1
@@ -160,7 +160,7 @@ test("the performance record knows the crawlers' share of answering time", () =>
160160 record.add("/:owner/:name/blob/:ref/*path", { at, ms: 300, signedIn: false, crawler: true, phases: [] });
161161 record.add("/:owner/:name/blob/:ref/*path", { at, ms: 100, signedIn: true, phases: [] });
162162 const share = crawlerShare(record, at - 1000);
163 expect(share).toEqual({ requests: 2, crawlerRequests: 1, ms: 400, crawlerMs: 300, share: 0.75 });
163 expect(share).toEqual({ requests: 2, crawlerRequests: 1, ms: 400, crawlerMs: 300, strangerRequests: 0, strangerMs: 0, share: 0.75 });
164164 });
165165
166166 test("the crawler board door is an admin's: a non-admin and an anonymous caller are told it is not there, an admin gets the shape", async () => {

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs