See where people come from and where they go: referrers, campaigns and a traffic path #9529

This adds a sources view for the forge and for each app, with a path to a goal, outbound-click counts, a probe behind every number and a daily GitHub pull. The new tests pass, and so do tsc, scripts/gate.ts, and the openapi, design-audit, design-map, design-lint, measure and apps-measure suites. I ran those suites, not the whole suite, and the forge's check runs the full one on push. The keep_plan tool wasn't available to me, so the issue has no plan list.

What changed

• Recording: the page-view beacon now also sends the referrer, the query string and the host of the last external link clicked. The server keeps only:

- the referrer's host; - utm_source, utm_medium and utm_campaign, restricted to a safe alphabet; - a clicked host that is in the named set (github.com and espodev.com; FORGE_OUTBOUND_LINKS replaces it).

No cookie, no address, no third party. The visitor is the existing daily-salted hash. Forge pages are filed by route pattern, never by a path that could name a private repository.

• Off the request path: a beacon request only pushes a row onto a bounded in-memory buffer. A timer writes it every 10 seconds in one transaction (src/perf/views_store.ts). The sources view reads it with one indexed query, and a test checks the query plan.
• Scrapers: crawlers, headless browsers, HTTP libraries and empty user agents are flagged and shown under their own tab, separate from people. The page says how each was counted.
• The view: /settings/traffic/sources is for admins, with ?app=<id> for any app. /:owner/:name/settings/app/traffic is for whoever administers the app's repository. It shows visits per day, sources, mediums, campaigns, landing pages, where people left and outbound clicks. Clicking a value narrows every other panel. The page states what it can't see.
• Path to a goal: you name steps such as /, /explore, out:github.com. The table shows visits and drop-off at each step, and the source filter sets where it starts.
• Probe: every number links to a new admin endpoint, GET /api/v1/admin/traffic/probe. It counts the number again in SQL by a different route and reports whether the two agree. It is documented in the OpenAPI contract.
• GitHub: with FORGE_GITHUB_TOKEN and FORGE_GITHUB_REPOS=owner/name,… set, the forge pulls referrers, views and clones once a day into github_traffic. Answers are validated, a refusal is recorded without throwing, and the token is never stored. I added api.github.com to the forge.json egress list.
• Docs and migration: docs/features/traffic-sources.md, the audit entry in docs/design/audit.md, and migration 20261108150000-traffic-views with its pin.

Not done

• Existing apps: apps already scaffolded keep their old static/beacon.js, which still works but sends no referrer data until it is replaced with the template's copy.
• No-script visitors: nothing counts them, so they stay invisible.
• Funnel recount: the probe recounts the path in the same order-matching logic as the page, but over the raw views instead of the built visits.
• GitHub pull untested live: it is tested with a fake fetch only, since this sandbox can't reach GitHub.
• Beacon SQL: the forge's existing /_measure door still does its old per-request SQL for page vitals. I left that alone and kept my new recording buffered.

To try it: sign in as an admin and open /settings/traffic, then follow "Where people come from". Visit with ?utm_source=post&utm_campaign=x and leave a page to see a row appear within about 10 seconds. Click any number to see its probe.

🤖 Generated with Claude Code

Closes #9427

This adds a sources view for the forge and for each app, with a path to a goal, outbound-click counts, a probe behind every number and a daily GitHub pull. The new tests pass, and so do tsc, scripts/gate.ts, and the openapi, design-audit, design-map, design-lint, measure and apps-measure suites. I ran those suites, not the whole suite, and the forge's check runs the full one on push. The keep_plan tool wasn't available to me, so the issue has no plan list.

What changed

• Recording: the page-view beacon now also sends the referrer, the query string and the host of the last external link clicked. The server keeps only:

- the referrer's host; - utm_source, utm_medium and utm_campaign, restricted to a safe alphabet; - a clicked host that is in the named set (github.com and espodev.com; FORGE_OUTBOUND_LINKS replaces it).

No cookie, no address, no third party. The visitor is the existing daily-salted hash. Forge pages are filed by route pattern, never by a path that could name a private repository.

• Off the request path: a beacon request only pushes a row onto a bounded in-memory buffer. A timer writes it every 10 seconds in one transaction (src/perf/views_store.ts). The sources view reads it with one indexed query, and a test checks the query plan.
• Scrapers: crawlers, headless browsers, HTTP libraries and empty user agents are flagged and shown under their own tab, separate from people. The page says how each was counted.
• The view: /settings/traffic/sources is for admins, with ?app=<id> for any app. /:owner/:name/settings/app/traffic is for whoever administers the app's repository. It shows visits per day, sources, mediums, campaigns, landing pages, where people left and outbound clicks. Clicking a value narrows every other panel. The page states what it can't see.
• Path to a goal: you name steps such as /, /explore, out:github.com. The table shows visits and drop-off at each step, and the source filter sets where it starts.
• Probe: every number links to a new admin endpoint, GET /api/v1/admin/traffic/probe. It counts the number again in SQL by a different route and reports whether the two agree. It is documented in the OpenAPI contract.
• GitHub: with FORGE_GITHUB_TOKEN and FORGE_GITHUB_REPOS=owner/name,… set, the forge pulls referrers, views and clones once a day into github_traffic. Answers are validated, a refusal is recorded without throwing, and the token is never stored. I added api.github.com to the forge.json egress list.
• Docs and migration: docs/features/traffic-sources.md, the audit entry in docs/design/audit.md, and migration 20261108150000-traffic-views with its pin.

Not done

• Existing apps: apps already scaffolded keep their old static/beacon.js, which still works but sends no referrer data until it is replaced with the template's copy.
• No-script visitors: nothing counts them, so they stay invisible.
• Funnel recount: the probe recounts the path in the same order-matching logic as the page, but over the raw views instead of the built visits.
• GitHub pull untested live: it is tested with a fake fetch only, since this sandbox can't reach GitHub.
• Beacon SQL: the forge's existing /_measure door still does its old per-request SQL for page vitals. I left that alone and kept my new recording buffered.

To try it: sign in as an admin and open /settings/traffic, then follow "Where people come from". Visit with ?utm_source=post&utm_campaign=x and leave a page to see a row appear within about 10 seconds. Click any number to see its probe.

🤖 Generated with Claude Code

The check failed on two gate rules that read commit messages, not code. I added both lines in a new commit (77c343a8) and pushed it. I didn't re-run the gate or any tests after pushing, so I haven't confirmed the rules now pass. The earlier code and tests are unchanged.

• "Why now": the owner asked for referrers, campaigns and a traffic path in #9427.
• "Try": sign in as admin on the preview, open /settings/traffic/sources, and click any number to reach /api/v1/admin/traffic/probe.

The commit is empty, so it adds no files. I couldn't amend the earlier commit without a force push, which isn't allowed here.

I fixed the failing test and pushed the fix (d0ceda54).

The test "every GET page names session and render in Server-Timing" failed because the two new sources pages never reported a render phase. /settings/traffic/sources and /:owner/:name/settings/app/traffic now do, including the not-found case when a repo has no app.

After the change, test/server-timing-sweep.test.ts and test/traffic-sources.test.ts pass (16 tests), tsc reports nothing and scripts/gate.ts is clean. I did not re-run the rest of the fast tier here. The forge's check runs that on the push.

Review points 1–5 are addressed and pushed. The gate, design lint and tsc are clean. The traffic and apps-measure suites and the palette and hot-read suites pass here. I did not run the full suite; the forge runs it on the push.

What changed

• Terminal door (1). New src/tui/screens/trafficview.ts, named apart from the followed-sources screen. It shows the page's figures: source, medium, campaign, landing, exit and outbound, the pages-per-visit line, and a path to a goal.

- Keys: w window, s people/scrapers, Enter narrows to the row under the cursor, c clears, t types path steps. - Gates: an admin gets the palette entry "Traffic sources" for the forge, added to ENTRIES. A repository admin gets "<owner>/<name> — Traffic sources" while the repository is open, and only if the repository is an app. A guest is offered neither, and the screen draws nothing for them. - Docs: docs/features/traffic-sources.md and AGENTS.md name the entry.

• Forged outbound clicks (2). The page's "What it cannot see" now says outbound clicks are reported by the visitor's browser and can be forged, so read them as a hint. The feature doc says the same, and so does the terminal screen.
• Memory cost (3). readVisits now holds its result per scope and window for 15 seconds, and flushViews drops it when views are written. A page and its probe calls share one read. The probe still recounts in SQL, so it stays an independent check.
• Client-supplied paths (4). The flush keeps at most 300 views per visitor per scope per day (VISITOR_DAY_MAX). One client posting invented paths can no longer fill the 50,000-row day cap and crowd out real views. I did not normalise paths, because the per-visitor cap bounds the damage without hiding real pages.
• Tests (5).

- New tests: the per-visitor cap, cache reuse and invalidation, the terminal gating, and a non-admin getting 404 on both pages. The admin page is checked for CSP, X-Content-Type-Options and Referrer-Policy. - Preview 303: the page does not redirect an admin session; the 303 you saw was most likely your token being sent to sign-in. The page needs a session cookie, not just a bearer token.

Left out

• The terminal does not call the probe or show GitHub's own traffic; those stay on the web page.
• The terminal has no forge-admin view of a single app (?app=). It has the forge scope and the per-app scope for a repository admin.
• The new tests call readableScope with a stub session, not a full terminal session, so the screen's key handling is untested.

Try it: open the palette as an admin and choose "Traffic sources". Or open /settings/traffic/sources on the preview with a session cookie.

🤖 Generated with Claude Code

I fixed the two failing roster tests and pushed. test/palette-roster.test.ts now passes (15 of 15) and bun x tsc --noEmit is clean. I didn't run bun scripts/gate.ts, the other tests or the full suite here; the forge's check will run them on this push.

The traffic-sources work added a "Traffic sources" entry to the terminal palette for site admins. Both tests list the palette entries in order and didn't include it. The code was right and the test was wrong. I added the entry (traffic-sources, "Traffic sources", "where people come from and where they go, for the forge") before "Who can do what" in the site-admin person list and the site-admin agent list. Only the test file changed.