An admin can see what was scraped and work out who is doing it #9464

An admin can now open /settings/traffic/scrapes (linked from Traffic) to see what was read, which networks it came from, and which requests look like one actor.

I ran tsc, the design lint, the gate, the new test/scrape-evidence.test.ts and about 45 related test files, and all passed. I didn't run the full suite, because the diff was too broad for fast.ts to narrow, and I didn't try the page in a running preview.

What it does

• What was read: reads are counted per repository, ref and path, and kept in a new read_hour table for 7 days with no address stored. The page offers 1, 3 or 7 days. For each repository it shows the share of its files read and flags paths whose names look like secrets (.env, id_rsa and so on). That flag is a guess from the name; the file contents are never read.
• Who it is: each address prefix shows network, operator, country and abuse contact from a new network table on the forge. An admin fills it by pasting range, name, operator, country, abuse contact lines, and bad lines are refused with the reason. No address goes to a third party, and a prefix with no entry says the evidence stops there.
• Capture: one click starts a capture that lasts 10 minutes or 2000 requests. It is held in memory only, so a restart or the expiry removes the full addresses. It keeps no cookie, token or query string, and it skips signed-in people. It can be stopped and deleted early.
• Grouping: requests are grouped by user agent, header names and order, language and related headers. Each group shows its size, rate, addresses, commits walked, pace and Cloudflare bot score if sent. Each group states a confidence (low, medium or high) with its reasons. A single address, or a trait every browser of one version shares, stays low.
• Findings: each group links to the requests behind it. "Export for an abuse report" downloads a plain-text file with the networks, abuse contacts, addresses and sample requests. Every finding repeats that it shows a network and a pattern, not who is behind it.
• Access: everything is admin-only, writes need a same-origin form post, and agent accounts can't start a capture or import networks. Capture start and network imports are recorded as events.

Left out

• No bundled whois or ASN database: the table is filled by hand or by paste, because there is no offline data source in the repo.
• Cloudflare's bot score is read from a cf-bot-score or x-bot-score header only if the edge sends one.
• There is one capture at a time, and requests to /settings/traffic are not captured.
• The group list shows the first 20 groups and the request list shows the first 100.
• Coverage is distinct paths read over files at HEAD, and only the 8 busiest repositories are measured.

To try it: sign in as an admin, open Settings → Traffic → "What was scraped, and who", paste a network line, start a capture, then open a group and export it.

I added no dependencies, and the migration is pinned.

Closes #9406

An admin can now open /settings/traffic/scrapes (linked from Traffic) to see what was read, which networks it came from, and which requests look like one actor.

I ran tsc, the design lint, the gate, the new test/scrape-evidence.test.ts and about 45 related test files, and all passed. I didn't run the full suite, because the diff was too broad for fast.ts to narrow, and I didn't try the page in a running preview.

What it does

• What was read: reads are counted per repository, ref and path, and kept in a new read_hour table for 7 days with no address stored. The page offers 1, 3 or 7 days. For each repository it shows the share of its files read and flags paths whose names look like secrets (.env, id_rsa and so on). That flag is a guess from the name; the file contents are never read.
• Who it is: each address prefix shows network, operator, country and abuse contact from a new network table on the forge. An admin fills it by pasting range, name, operator, country, abuse contact lines, and bad lines are refused with the reason. No address goes to a third party, and a prefix with no entry says the evidence stops there.
• Capture: one click starts a capture that lasts 10 minutes or 2000 requests. It is held in memory only, so a restart or the expiry removes the full addresses. It keeps no cookie, token or query string, and it skips signed-in people. It can be stopped and deleted early.
• Grouping: requests are grouped by user agent, header names and order, language and related headers. Each group shows its size, rate, addresses, commits walked, pace and Cloudflare bot score if sent. Each group states a confidence (low, medium or high) with its reasons. A single address, or a trait every browser of one version shares, stays low.
• Findings: each group links to the requests behind it. "Export for an abuse report" downloads a plain-text file with the networks, abuse contacts, addresses and sample requests. Every finding repeats that it shows a network and a pattern, not who is behind it.
• Access: everything is admin-only, writes need a same-origin form post, and agent accounts can't start a capture or import networks. Capture start and network imports are recorded as events.

Left out

• No bundled whois or ASN database: the table is filled by hand or by paste, because there is no offline data source in the repo.
• Cloudflare's bot score is read from a cf-bot-score or x-bot-score header only if the edge sends one.
• There is one capture at a time, and requests to /settings/traffic are not captured.
• The group list shows the first 20 groups and the request list shows the first 100.
• Coverage is distinct paths read over files at HEAD, and only the 8 busiest repositories are measured.

To try it: sign in as an admin, open Settings → Traffic → "What was scraped, and who", paste a network line, start a capture, then open a group and export it.

I added no dependencies, and the migration is pinned.

The gate failed because no commit message had a "Why now:" or "Try:" line. I added both in an empty commit and pushed it. I haven't re-run the gate or the tests since, and I changed no code, so the forge's check on the new push is the next confirmation.

An admin can now open /settings/traffic/scrapes (linked from Traffic) to see what was scraped and work out who is doing it.

• What was read: reads are counted by repository, ref and path for 1, 3 or 7 days, with the share of each repository's files covered. Paths whose names look like keys or credentials are flagged. That flag is a guess from the name, since file contents are never read.
• Who it is: each address prefix shows network, operator, country and abuse contact from a network table on the forge. An admin fills it by pasting range, name, operator, country, abuse contact lines, so no visitor address goes to a third party. A prefix with no entry says the evidence stops there.
• One actor across addresses: one click starts a capture of 10 minutes or 2000 requests. It is held in memory only, keeps no cookies, tokens or query strings, skips signed-in people, and is gone when it expires or is stopped. Requests are grouped by user agent and header shape. Each group shows size, rate, addresses, commits walked and Cloudflare's bot score if sent, with a stated confidence and its reasons.
• Findings and export: each group links to the requests behind it and exports as a plain-text abuse report with the networks' abuse contacts. Every finding says it shows a network and a pattern, not who is behind it.
• Access: everything is admin-only, and writes need a same-origin form post. Agent accounts can't start a capture or import networks.

Left out:

• There is no bundled whois or ASN database; the network table is filled by hand.
• Only one capture runs at a time.
• The group list shows the first 20 groups and the request list the first 100.
• Coverage is measured only for the 8 busiest repositories.

I added no dependencies, and the new migration is pinned.

To try it: sign in as an admin and open Settings → Traffic → "What was scraped, and who". Paste a network line, start a capture, then open a group and export it.

I fixed the two failures that came from my change and pushed the fix as b2f6f69d. I haven't checked the other two, and I haven't seen the forge's check run on the push yet.

• Design audit: /settings/traffic/scrapes had no entry in docs/design/audit.md, so I added one. /settings/traffic/scrapes/export is a file download, not a page, so test/design-audit.test.ts now excludes it.
• Server-Timing sweep: the scrapes page didn't name the session and render phases. It now does. The sweep test (test/server-timing-sweep.test.ts) now lists the export route among the routes that aren't pages. I changed both tests because they needed to know about the new routes, not because they were wrong.
• The MCP feed-card test and the "tokens bought" test: both failed in shard 20, and neither is in code I touched. I ran the MCP tests and test/returns.test.ts alone and they passed, so I treated these two as flakes. I didn't run the returns test by its exact name.

Before pushing I ran bun scripts/gate.ts, bun scripts/design-lint.ts, bun x tsc --noEmit, and the design-audit, server-timing, scrape-evidence, MCP and returns tests. Everything was clean.

I've addressed the three review points and pushed the fix as 18255e5e. The forge's check on that push hasn't run yet. Locally the gate, tsc, the design lint and the scrape-evidence, design-audit and server-timing tests pass. So do the README-cap and terminal-screen tests I picked. I didn't run the full suite.

What changed

• README: the how-to now lives in docs/features/scrapes.md, and README.md is back to its original size.
• Terminal screen: src/tui/screens/scrapes.ts is new and registered in src/tui/screens/entries.ts. It is read-only for admins. It lists each repository's reads over the last day, with any key-looking paths, plus the groups from a running capture. Starting a capture and exporting stay on the web page, since the abuse report is a download.
• Imports: the duplicate perf/live.ts, perf/traffic.ts and perf/reads.ts imports in src/http/server.ts are merged into one line each.
• read_hour growth: an hour now holds at most 500 distinct rows (HOUR_KEYS_MAX), counted across all the writes that hour gets. Later keys fold into that repository's (others) row. readsReport now sums by repository and path, and by repository and ref, in SQL instead of loading every row in the window. A new test in test/scrape-evidence.test.ts covers the hourly cap.

One addition you didn't ask for

• Cloudflare's bot score and country headers are now believed only when the request came through a proxy the operator trusts, via a new wasViaEdge in src/http/peer.ts. Otherwise a visitor could set those headers themselves.

Nothing in the review was disputed.

I fixed one of the two failing tests and pushed the fix as ee2574d4. The forge's check on that push hasn't run yet.

• Palette roster test: this failure was my change. The new Scrapes terminal screen shows up in a site admin's palette, but test/palette-roster.test.ts didn't list it. I added the entry to the admin roster. The test was out of date, not wrong, and it passes now.
• Model-grant test (test/model-grant.test.ts, "a running grant run crosses the cap mid-session…"): nothing in my change touches it, so I left it alone. I ran the file alone and it passes (7/7). I'm treating it as a flake, a timing-sensitive test that fails when its shard is under load. If it fails again on this push, I'll look at it properly.