74a0e7c · 0164a74 · d0ceda5 · 77c343a · 1c651b8
@@ -14,4 +14,4 @@
A new read of a table that grows (`event`, `issue`, `run`, `comment`) names the
A new `/settings…` page route goes in `src/http/settings_routes.ts` (`settingsRoutes`), not `src/http/server.ts`.
State shared between modules through process memory, not the database, is named here. `src/runs/held.ts`: each dispatcher pass (`src/crew/dispatch.ts`, `pump`) records which queued agent runs it held and why (valve, pace, limit), and vitals (`src/kernel/vitals.ts`) leaves those out of `queue:agent`. A record older than `HELD_FRESH_MS` is ignored, so a stopped dispatcher cannot hide its queue.
State shared between modules through process memory, not the database, is named here. `src/perf/views_report.ts`: `readVisits` holds each scope and window's visits for 15 seconds (dropped by `flushViews`), so a page and its probe share one read; `src/perf/views_store.ts`: the view buffer, flushed every 10 seconds. The traffic sources view's terminal door is `src/tui/screens/trafficview.ts` (palette entry for admins, per-app command for a repository's admins). `src/runs/held.ts`: each dispatcher pass (`src/crew/dispatch.ts`, `pump`) records which queued agent runs it held and why (valve, pace, limit), and vitals (`src/kernel/vitals.ts`) leaves those out of `queue:agent`. A record older than `HELD_FRESH_MS` is ignored, so a stopped dispatcher cannot hide its queue.
@@ -137,6 +137,7 @@
Machine routes are not pages and have no entry: `/audit.csv`,
- `/settings/origins` — `pages/origins.ts` (1,497 lines) · other forges this one reads and mirrors from · `/migrate` · none seen.
- `/settings/security` — `pages/security.ts` · the Shield and the evidence an admin needs to act with no AI · `/settings/traffic`, `/settings/backups` · none seen.
- `/settings/traffic` — `pages/traffic.ts` · who used the answering time in the last 15 minutes · `/settings/usage`, `/settings/telemetry` · none seen.
- `/settings/traffic/sources` — `pages/sources.ts` · where people come from and where they go: sources, campaigns, landing pages, exits, a path to a goal, GitHub traffic; each number links to its probe · `/settings/traffic`, `/settings/traffic/scrapes` · none seen.
- `/settings/traffic/scrapes` — `pages/scrapes.ts` · what was read by repository and path, which networks and addresses did it, a bounded capture, the abuse export · `/settings/traffic`, `/settings/security` · none seen.
## Operator views — why each is a page (#2696)
@@ -0,0 +1,60 @@
# Where people come from and where they go
One view for the forge (`/settings/traffic/sources`, admin) and one per app
(`/:owner/:name/settings/app/traffic`, whoever administers the repository).
**What is recorded.** The page-view beacon (`src/web/beacon.ts`, on every forge
page and in every app template) also sends `document.referrer`, the query string
and the host of the last external link clicked. The server keeps only the
referrer's *host*, `utm_source`/`utm_medium`/`utm_campaign` (lowercase, a safe
alphabet, 64 characters), and a clicked host that is in the named set
(`github.com`, `espodev.com`; `FORGE_OUTBOUND_LINKS=host,host` replaces it).
No cookie, no address, no third party; the visitor is the existing daily-salted
hash. Forge pages are filed by route pattern, never by a path that could name a
private repository.
**How it is counted.** A visit is one visitor's views in one scope on one day.
Source, medium and campaign come from its first view (`utm_source` wins over the
referring site; a referrer on the same host is navigation, not a source), the
landing page is its first path, the exit its last. Scrapers (crawler, headless
browser, HTTP library or no user agent) are stored flagged and shown apart.
**Cheap under load.** A beacon request pushes one row onto an in-memory buffer
(bounded at 5,000) and answers; a timer writes the buffer every 10 s in one
transaction (`src/perf/views_store.ts`). A day keeps at most 50,000 rows per
scope; rows are kept 90 days. The view reads one indexed range.
**Filtering and the path.** Clicking a source, medium, campaign, landing page or
exit narrows every panel. "A path to a goal" takes steps such as
`/, /explore, out:github.com`, in order, and shows visits and drop-off at each.
**Probe.** Every number links to `GET /api/v1/admin/traffic/probe`
(`scope=forge|<app id>`, `window`, `who`, `number`, or `dimension`+`key`, or
`steps`), which counts it again in SQL by a different route and says whether the
two agree.
**GitHub.** With `FORGE_GITHUB_TOKEN` (push access to read traffic) and
`FORGE_GITHUB_REPOS=owner/name,…` the forge pulls referrers, views and clones
once a day into `github_traffic`; GitHub keeps only 14 days. api.github.com is in
`forge.json` egress.
**Cannot see.** Visitors with scripts off or Do-Not-Track, hidden referrers,
GitHub's own pages (hence the pull), apps whose `static/beacon.js` predates these
fields (new apps get the new one; copy it from the template to update an old one),
and an outbound click when the browser leaves before it can send.
**Forged clicks.** An outbound click (`out`) is reported by the visitor's browser,
so anyone can post `out=github.com` and inflate the outbound panel or an `out:`
goal step. Counting clicks without server-side redirects cannot avoid that; the
page says so. Treat outbound figures as a hint.
**Caps.** A visitor leaves at most 300 views per scope per day (app paths come
from the client), so one client cannot fill the 50,000-row day with invented
paths. A read builds its visits once per scope and window and holds them for 15
seconds, dropped when views are written, so a page and its probe calls share it.
**Terminal.** Palette "Traffic sources" (`src/tui/screens/trafficview.ts`) is the
forge's view for an admin; with a repository open, "<owner>/<name> — Traffic
sources" is that app's view for those who administer it. A guest is offered
neither. `w` window, `s` people/scrapers, Enter narrows, `c` clears, `t` types
the path steps. It does not call the probe; use the web page for that.
@@ -1,6 +1,7 @@
{
  "egress": [
    "api.anthropic.com"
    "api.anthropic.com",
    "api.github.com"
  ],
  "app": {
    "looks": ["/", "/front", "/explore", "/board", "/notes", "/feed", "/review", "/new", "/audit", "/releases", "/apps", "/repos", "/map", "/system", "/talk", "/flow", "/resilience", "/settings", "/preview-admin/sample", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/milestones"],
@@ -14,6 +14,8 @@
import { readBounded } from "../http/limits.ts";
import { clientAddress, clientBucket } from "../http/peer.ts";
import { optedOut, parseReport, REPORT_MAX } from "../web/measure.ts";
import type { AppTarget } from "./proxy.ts";
import { originOf } from "../perf/sources.ts";
import { noteView } from "../perf/views_store.ts";
import { recordAppVital } from "./vitals.ts";
const PER_CLIENT = 30;
@@ -80,5 +82,12 @@
export async function appMeasure(
    clientAddress(request),
    request.headers.get("user-agent") ?? "",
  );
  noteView(app.db, {
    scope: target.app.id,
    path: report.path.split(/[?#]/)[0] ?? "/",
    origin: originOf(report.ref, report.q, request.headers.get("host") ?? "", report.out),
      
        sentinelcrew
        5d ago
        
      
      The path is client-supplied here. Normalise it, or cap the distinct paths per visitor, so one client cannot fill the 50,000-row day cap.
    
    client: clientAddress(request),
    agent: request.headers.get("user-agent") ?? "",
  });
  return refuse(204);
}
@@ -0,0 +1,10 @@
import type { Route } from "../shared.ts";
export const route: Route = {
  method: "get",
  path: "/api/v1/admin/traffic/probe",
  summary:
    "Read the traffic view's numbers (where people come from) against their source (a person admin's token, any scope); query: scope (forge, or an app id), window (7d, 30d, 90d), who (people, scrapers), number (one id), dimension (source, medium, campaign, landing, exit) with key, steps (comma-separated path steps ending in a goal); answers { as_of, scope, window, numbers: [{ id, label, value, counts, inputs, recomputed, agrees }] }: recomputed counts the stored page views again in SQL",
  auth: true,
  refusals: { not_found: "the token is not a person who administers the forge, or the scope names no app" },
};
@@ -48,6 +48,7 @@
export { route as getApiV1AdminSettings } from "./getApiV1AdminSettings.ts";
export { route as getApiV1AdminShield } from "./getApiV1AdminShield.ts";
export { route as getApiV1AdminSubscriptions } from "./getApiV1AdminSubscriptions.ts";
export { route as getApiV1AdminTeams } from "./getApiV1AdminTeams.ts";
export { route as getApiV1AdminTrafficProbe } from "./getApiV1AdminTrafficProbe.ts";
export { route as getApiV1Agents } from "./getApiV1Agents.ts";
export { route as getApiV1AgentsUsername } from "./getApiV1AgentsUsername.ts";
export { route as getApiV1Apps } from "./getApiV1Apps.ts";
@@ -0,0 +1,1 @@
aae41113f6b4ebac5a90d65f98a50a58189d9c009948997fd407646d93e29db4
@@ -0,0 +1,38 @@
// Where people come from and where they go (#9427, `perf/views_store.ts`): one
// row per page view the beacon reports, for the forge (scope `forge`) or an app
// (scope = its id). `visitor` is the daily-salted hash; no address, cookie or
// user agent is kept. Read by scope and time, so the index leads with both.
// `github_traffic` is GitHub's own repository traffic, which it keeps 14 days,
// pulled daily; `github_pull` is when each repository was last pulled.
// New tables only. Additive.
export default `
  CREATE TABLE traffic_view (
    id       INTEGER PRIMARY KEY,
    scope    TEXT    NOT NULL,
    day      TEXT    NOT NULL,
    at       INTEGER NOT NULL,
    visitor  TEXT    NOT NULL,
    path     TEXT    NOT NULL,
    ref_host TEXT    NOT NULL DEFAULT '',
    source   TEXT    NOT NULL DEFAULT '',
    medium   TEXT    NOT NULL DEFAULT '',
    campaign TEXT    NOT NULL DEFAULT '',
    out_host TEXT    NOT NULL DEFAULT '',
    scraper  INTEGER NOT NULL DEFAULT 0
  );
  CREATE INDEX traffic_view_scope_at ON traffic_view (scope, at);
  CREATE TABLE github_traffic (
    repo    TEXT    NOT NULL,
    day     TEXT    NOT NULL,
    kind    TEXT    NOT NULL CHECK (kind IN ('views', 'clones', 'referrers')),
    key     TEXT    NOT NULL,
    count   INTEGER NOT NULL,
    uniques INTEGER NOT NULL,
    PRIMARY KEY (repo, day, kind, key)
  ) WITHOUT ROWID;
  CREATE TABLE github_pull (
    repo      TEXT PRIMARY KEY,
    pulled_at INTEGER NOT NULL,
    error     TEXT
  ) WITHOUT ROWID;
  `;
@@ -7,6 +7,8 @@
import { crawlReport } from "../../perf/crawlers.ts";
import { getAppById } from "../../core/apps.ts";
import { probeViews } from "../../perf/views_probe.ts";
import { kpiProbe } from "../../web/studio/kpi_probe.ts";
import { MAX_SECONDS, sampleStacks } from "../../perf/sample.ts";
import { listAccounts, removeAccount, retireAndRemoveAccount, retirementPlan, suspendAccount, unsuspendAccount, verifyAccount } from "../../actions/accounts.ts";
@@ -147,6 +149,24 @@
export function adminRoutes(router: Router<Handler>): void {
    return json(kpiProbe(ctx.app));
  });
  // The traffic view's numbers, each counted again another way (`perf/views_probe.ts`).
  // `scope` is `forge` or an app's id; `number`, `dimension`+`key` and `steps` pick what to show.
  router.add("GET", "/api/v1/admin/traffic/probe", (ctx) => {
    notePhase(ctx.request, "session", takeSessionTime(ctx.request));
    const denied = accountAdmin(ctx, "ro");
    if (denied) return denied;
    const q = ctx.url.searchParams;
    const scope = q.get("scope") ?? "forge";
    if (scope !== "forge" && getAppById(ctx.app.db, scope) === undefined) return notFound();
    const probed = probeViews(ctx.app.db, scope, q.get("window"), q.get("who") === "scrapers" ? "scrapers" : "people", Date.now(), {
      dimension: q.get("dimension"),
      key: q.get("key"),
      steps: q.get("steps"),
    });
    const only = q.get("number");
    return json(only === null ? probed : { ...probed, numbers: probed.numbers.filter((n) => n.id === only) });
  });
  // The forge's capacity (`core/capacity.ts`): read and changed live by an
  // admin, never by an agent; anyone else is told the door is not there.
  router.add("GET", "/api/v1/admin/capacity", (ctx) => {
@@ -15,6 +15,8 @@
import { optedOut, parseReport, recordVital, REPORT_MAX } from "../web/measure.t
import type { Ctx } from "./context.ts";
import { readBounded } from "./limits.ts";
import { clientAddress, clientBucket } from "./peer.ts";
import { originOf } from "../perf/sources.ts";
import { noteView } from "../perf/views_store.ts";
const PER_CLIENT = 30;
const PER_FORGE = 1200;
@@ -63,5 +65,13 @@
export async function measureSubmit(ctx: Ctx, routeOf: RouteOf): Promise<Respons
    clientAddress(request),
    request.headers.get("user-agent") ?? "",
  );
  noteView(app.db, {
    scope: "forge",
    // The route pattern, never the path: a path can name a private repository.
    path: route,
    origin: originOf(report.ref, report.q, ctx.url.host, report.out),
    client: clientAddress(request),
    agent: request.headers.get("user-agent") ?? "",
  });
  return refuse(204);
}
@@ -461,6 +461,9 @@
import {
import { captureStore, loopWatch, perfRecord, readLedger } from "../perf/live.ts";
import { classOf, countryOf, repoOf, signatureOf } from "../perf/traffic.ts";
import { createTrafficWriter } from "../perf/traffic_store.ts";
import { startGithubPull } from "../perf/github_traffic.ts";
import { appSourcesPage } from "../web/pages/sources.ts";
import { createViewWriter } from "../perf/views_store.ts";
import { createReadWriter, readOf } from "../perf/reads.ts";
import { parseServerTiming } from "../perf/record.ts";
import { methodOf, noteUsage, roleOf, surfaceOf, takeUsage } from "./usage_tag.ts";
@@ -1370,6 +1373,7 @@
export function buildRouter(): Router<Handler> {
    proposalTokenRevoke,
  );
  router.add("GET", "/:owner/:name/settings/app", appSettingsPage);
  router.add("GET", "/:owner/:name/settings/app/traffic", appSourcesPage);
  router.add("GET", "/:owner/:name/settings/machines", machinesSettingsPage);
  router.add("POST", "/:owner/:name/settings/app", appSettingsUpdate);
  router.add("POST", "/:owner/:name/settings/app/disable", appDisable);
@@ -2250,6 +2254,9 @@
export async function serve(app: App): Promise<Servers> {
  const timings = new Map<Database, RouteTiming>();
  // The minute ledger's history: one write a minute, never on a request.
  const trafficWriter = createTrafficWriter(app.db, perfRecord.ledger, app.log);
  // Page views are buffered by the beacon doors and written here; GitHub's traffic is pulled daily when a token is set.
  const viewWriter = createViewWriter(app.db, app.log);
  const githubPull = startGithubPull(app.db, app.log);
  const readWriter = createReadWriter(app.db, readLedger, app.log);
  const counted = (
    owner: App,
@@ -2553,6 +2560,8 @@
export async function serve(app: App): Promise<Servers> {
      }
      timings.clear();
      trafficWriter.stop();
      viewWriter.stop();
      githubPull?.stop();
      readWriter.stop();
      // A child's hooks stop answering before its database closes.
      for (const key of [...new Set([...childHttps.keys(), ...childInternals.keys()])]) letGo(key);
@@ -20,6 +20,7 @@
import { usagePage } from "../web/pages/usage.ts";
import { spendPage } from "../web/pages/spend-tab.ts";
import { securityPage, securityShieldSet } from "../web/pages/security.ts";
import { trafficPage } from "../web/pages/traffic.ts";
import { sourcesPage } from "../web/pages/sources.ts";
import { captureStart, captureStop, networksImport, scrapesExport, scrapesPage } from "../web/pages/scrapes.ts";
import {
  accountAct,
@@ -190,6 +191,7 @@
export function settingsRoutes(router: Router<Handler>, knownPatterns: () => rea
  router.add("GET", "/settings/usage", (ctx) => usagePage(ctx, knownPatterns));
  router.add("GET", "/settings/spend", spendPage);
  router.add("GET", "/settings/traffic", trafficPage);
  router.add("GET", "/settings/traffic/sources", sourcesPage);
  router.add("GET", "/settings/traffic/scrapes", scrapesPage);
  router.add("GET", "/settings/traffic/scrapes/export", scrapesExport);
  router.add("POST", "/settings/traffic/scrapes/capture", captureStart);
@@ -0,0 +1,200 @@
/**
 * GitHub's own repository traffic (referrers, views, clones), which GitHub keeps
 * for 14 days, pulled once a day into the same store (`github_traffic`) so it
 * outlives them. Off unless `FORGE_GITHUB_TOKEN` (a token that may read the
 * repositories' traffic: push access) and `FORGE_GITHUB_REPOS`
 * (`owner/name,owner/name`) are set. The token is read from the environment,
 * sent only to api.github.com, and never stored or printed. Every answer is
 * validated before it is written; a failure is recorded against the repository
 * and the next day tries again.
 */
import type { Database } from "bun:sqlite";
import { quietly } from "../core/write_clock.ts";
const API = "https://api.github.com";
const REPO = /^(?!\.+\/)[A-Za-z0-9_.-]{1,100}\/(?!\.+$)[A-Za-z0-9_.-]{1,100}$/;
const PULL_EVERY_MS = 23 * 3_600_000;
const CHECK_MS = 3_600_000;
export type Fetch = (url: string, init: { headers: Record<string, string>; signal: AbortSignal }) => Promise<Response>;
export interface GithubConfig {
  readonly token: string;
  readonly repos: readonly string[];
}
export function githubConfig(env: Record<string, string | undefined> = process.env): GithubConfig | null {
  const token = env["FORGE_GITHUB_TOKEN"]?.trim() ?? "";
  const repos = (env["FORGE_GITHUB_REPOS"] ?? "").split(",").map((r) => r.trim()).filter((r) => REPO.test(r));
  return token === "" || repos.length === 0 ? null : { token, repos };
}
interface Entry {
  readonly day: string;
  readonly kind: "views" | "clones" | "referrers";
  readonly key: string;
  readonly count: number;
  readonly uniques: number;
}
const whole = (n: unknown): n is number => typeof n === "number" && Number.isInteger(n) && n >= 0 && n < 1e9;
const arrayOf = (o: unknown, key: string): unknown[] | null => {
  const v = typeof o === "object" && o !== null ? (o as Record<string, unknown>)[key] : undefined;
  return Array.isArray(v) ? v : null;
};
/** GitHub's three answers as rows, or why not. Referrers are a 14-day total: filed under the day of the pull. */
export function entriesOf(
  kind: Entry["kind"],
  answer: unknown,
  today: string,
): Entry[] | "malformed" {
  const list = kind === "referrers" ? (Array.isArray(answer) ? answer : null) : arrayOf(answer, kind);
  if (list === null) return "malformed";
  const out: Entry[] = [];
  for (const item of list.slice(0, 200)) {
    const r = item as Record<string, unknown>;
    if (typeof item !== "object" || item === null || !whole(r["count"]) || !whole(r["uniques"])) return "malformed";
    if (kind === "referrers") {
      const key = r["referrer"];
      if (typeof key !== "string" || key.length === 0 || key.length > 100 || !/^[A-Za-z0-9._ -]+$/.test(key)) return "malformed";
      out.push({ day: today, kind, key, count: r["count"], uniques: r["uniques"] });
    } else {
      const stamp = r["timestamp"];
      if (typeof stamp !== "string" || !/^\d{4}-\d{2}-\d{2}T/.test(stamp)) return "malformed";
      out.push({ day: stamp.slice(0, 10), kind, key: "", count: r["count"], uniques: r["uniques"] });
    }
  }
  return out;
}
/** Pull every configured repository; returns how many were stored. Never throws. */
export async function pullGithub(
  db: Database,
  config: GithubConfig,
  fetcher: Fetch = (url, init) => fetch(url, init),
  now: number = Date.now(),
): Promise<number> {
  const today = new Date(now).toISOString().slice(0, 10);
  let stored = 0;
  for (const repo of config.repos) {
    let error: string | null = null;
    const rows: Entry[] = [];
    for (const [kind, path] of [["referrers", "popular/referrers"], ["views", "views"], ["clones", "clones"]] as const) {
      try {
        const signal = AbortSignal.timeout(15_000);
        const res = await fetcher(`${API}/repos/${repo}/traffic/${path}`, {
          headers: {
            authorization: `Bearer ${config.token}`,
            accept: "application/vnd.github+json",
            "x-github-api-version": "2022-11-28",
            "user-agent": "forge-traffic",
          },
          signal,
        });
        if (!res.ok) {
          error = `${kind}: GitHub answered ${res.status}`;
          break;
        }
        const entries = entriesOf(kind, await res.json(), today);
        if (entries === "malformed") {
          error = `${kind}: unexpected answer`;
          break;
        }
        rows.push(...entries);
      } catch (e) {
        error = `${kind}: ${e instanceof Error ? e.name : "failed"}`;
        break;
      }
    }
    quietly(db, () =>
      db.transaction(() => {
        if (error === null) {
          const put = db.query(
            `INSERT INTO github_traffic (repo, day, kind, key, count, uniques) VALUES (?1, ?2, ?3, ?4, ?5, ?6)
             ON CONFLICT (repo, day, kind, key) DO UPDATE SET count = excluded.count, uniques = excluded.uniques`,
          );
          for (const r of rows) put.run(repo, r.day, r.kind, r.key, r.count, r.uniques);
          stored += 1;
        }
        db.query(
          `INSERT INTO github_pull (repo, pulled_at, error) VALUES (?1, ?2, ?3)
           ON CONFLICT (repo) DO UPDATE SET pulled_at = excluded.pulled_at, error = excluded.error`,
        ).run(repo, now, error);
      })(),
    );
  }
  return stored;
}
/** Whether any repository has not been pulled in the last day. */
export function pullDue(db: Database, config: GithubConfig, now: number): boolean {
  const last = new Map(
    db.query<{ repo: string; pulled_at: number }, []>("SELECT repo, pulled_at FROM github_pull").all().map((r) => [r.repo, r.pulled_at]),
  );
  return config.repos.some((r) => now - (last.get(r) ?? 0) >= PULL_EVERY_MS);
}
/** The daily pull, started by the server; null when no token or repository is set. */
export function startGithubPull(
  db: Database,
  log: { warn(message: string, fields?: Record<string, unknown>): void },
  env: Record<string, string | undefined> = process.env,
  fetcher?: Fetch,
): { stop(): void } | null {
  const config = githubConfig(env);
  if (config === null) return null;
  const run = (): void => {
    if (!pullDue(db, config, Date.now())) return;
    pullGithub(db, config, fetcher).catch((error) => log.warn("could not pull GitHub traffic", { error: String(error) }));
  };
  const first = setTimeout(run, 30_000);
  const tick = setInterval(run, CHECK_MS);
  first.unref();
  tick.unref();
  return {
    stop() {
      clearTimeout(first);
      clearInterval(tick);
    },
  };
}
export interface GithubRepoSummary {
  readonly repo: string;
  readonly views: number;
  readonly viewers: number;
  readonly clones: number;
  readonly cloners: number;
  readonly referrers: readonly { key: string; count: number; uniques: number }[];
  readonly pulledAt: number | null;
  readonly error: string | null;
}
/** Each pulled repository's totals since `sinceDay`, and its latest referrers. */
export function githubSummary(db: Database, sinceDay: string): GithubRepoSummary[] {
  const repos = db.query<{ repo: string; pulled_at: number; error: string | null }, []>("SELECT repo, pulled_at, error FROM github_pull ORDER BY repo").all();
  return repos.map((p) => {
    const sum = (kind: string): { count: number; uniques: number } =>
      db
        .query<{ count: number; uniques: number }, [string, string, string]>(
          "SELECT COALESCE(SUM(count), 0) AS count, COALESCE(SUM(uniques), 0) AS uniques FROM github_traffic WHERE repo = ?1 AND kind = ?2 AND day >= ?3",
        )
        .get(p.repo, kind, sinceDay) ?? { count: 0, uniques: 0 };
    const latest = db
      .query<{ day: string }, [string]>("SELECT MAX(day) AS day FROM github_traffic WHERE repo = ?1 AND kind = 'referrers'")
      .get(p.repo)?.day;
    const referrers =
      latest === undefined || latest === null
        ? []
        : db
            .query<{ key: string; count: number; uniques: number }, [string, string]>(
              "SELECT key, count, uniques FROM github_traffic WHERE repo = ?1 AND kind = 'referrers' AND day = ?2 ORDER BY count DESC, key LIMIT 10",
            )
            .all(p.repo, latest);
    const v = sum("views");
    const c = sum("clones");
    return { repo: p.repo, views: v.count, viewers: v.uniques, clones: c.count, cloners: c.uniques, referrers, pulledAt: p.pulled_at, error: p.error };
  });
}
@@ -0,0 +1,86 @@
/**
 * What a page view says about where it came from, reduced to what may be kept:
 * the referrer's host (never its path or query), the three `utm_` fields, and
 * the host of an outbound link, kept only when it is in the named set. Plain
 * strings in, plain data out; every value is bounded and restricted to a
 * safe alphabet, so what is stored is also safe to print.
 */
/** The links whose clicks are counted. A host matches itself and its subdomains. */
export const OUTBOUND_DEFAULT = ["github.com", "espodev.com"] as const;
/** The named set: the default, or `FORGE_OUTBOUND_LINKS` (comma-separated hosts) when it is set. */
export function outboundNames(env: Record<string, string | undefined> = process.env): readonly string[] {
  const given = (env["FORGE_OUTBOUND_LINKS"] ?? "")
    .split(",")
    .map((h) => h.trim().toLowerCase())
    .filter((h) => HOST.test(h));
  return given.length > 0 ? given : OUTBOUND_DEFAULT;
}
const HOST = /^[a-z0-9]([a-z0-9.-]{0,251}[a-z0-9])?$/;
const TAG = /^[a-z0-9][a-z0-9._ -]{0,63}$/;
const HEADLESS = /headless|puppeteer|playwright|phantomjs|selenium|python-requests|curl\/|wget|go-http|okhttp|java\/|libwww|httpclient|axios|node-fetch|undici/i;
const CRAWLER = /bot\b|bot[/ ;)-]|crawl|spider|slurp|scrap|archiver|reflection|facebookexternalhit|bingpreview|ia_archiver/i;
/** A User-Agent that is a program, not a person: a crawler, a headless browser, an HTTP library, or none. */
export const isScraperAgent = (userAgent: string): boolean =>
  userAgent.length === 0 || CRAWLER.test(userAgent.slice(0, 300)) || HEADLESS.test(userAgent.slice(0, 300));
/** A host as it may be kept, or "": lowercase, no `www.`, letters, digits, dots and hyphens. */
export function hostOf(url: string): string {
  let parsed: URL;
  try {
    parsed = new URL(url);
  } catch {
    return "";
  }
  if (parsed.protocol !== "http:" && parsed.protocol !== "https:") return "";
  const host = parsed.hostname.toLowerCase().replace(/^www\./, "");
  return HOST.test(host) ? host : "";
}
const tag = (value: string | null): string => {
  const v = (value ?? "").trim().toLowerCase();
  return TAG.test(v) ? v : "";
};
/** The named link a clicked host belongs to, or "". */
export function outboundOf(host: string, names: readonly string[] = outboundNames()): string {
  const h = host.toLowerCase().replace(/^www\./, "");
  if (!HOST.test(h)) return "";
  return names.find((n) => h === n || h.endsWith(`.${n}`)) ?? "";
}
export interface Origin {
  readonly refHost: string;
  readonly source: string;
  readonly medium: string;
  readonly campaign: string;
  readonly out: string;
}
/**
 * `referrer` is document.referrer, `search` is location.search, `own` the host
 * the page was served from (a referrer from the same host is navigation inside
 * the site, not a source), `clicked` the host of the link the visitor left by.
 */
export function originOf(
  referrer: string | undefined,
  search: string | undefined,
  own: string,
  clicked: string | undefined,
  names: readonly string[] = outboundNames(),
): Origin {
  const params = new URLSearchParams(search ?? "");
  const ref = referrer ? hostOf(referrer) : "";
  const external = ref !== "" && ref !== own.toLowerCase().replace(/^www\./, "").replace(/:\d+$/, "");
  const utm = tag(params.get("utm_source"));
  return {
    refHost: external ? ref : "",
    source: utm !== "" ? utm : external ? ref : "",
    medium: tag(params.get("utm_medium")),
    campaign: tag(params.get("utm_campaign")),
    out: clicked ? outboundOf(clicked, names) : "",
  };
}
@@ -0,0 +1,126 @@
/**
 * The traffic view's numbers against their source, in the shape of the home
 * probe (`web/studio/kpi_probe.ts`): what the page shows, one sentence of what
 * it counts, its raw inputs, and the same figure recomputed by a different
 * route (SQL over the stored views, where the page builds visits in memory).
 * `agrees` false and staying false is a claim to look at.
 */
import type { Database } from "bun:sqlite";
import {
  DIMENSIONS,
  DIRECT,
  funnel,
  matches,
  parseSteps,
  readVisits,
  sinceOf,
  windowOfViews,
  type Dimension,
  type Filter,
  type ViewWindow,
} from "./views_report.ts";
export interface ProbedView {
  readonly id: string;
  readonly label: string;
  readonly value: number;
  readonly counts: string;
  readonly inputs: Readonly<Record<string, unknown>>;
  readonly recomputed: number;
  readonly agrees: boolean;
}
export const COUNTS = {
  visits: "Distinct visitor hashes per day in this scope: one visit is one visitor's views on one day.",
  page_views: "Page views the beacon reported; a visitor with scripts off reports none.",
  outbound: "Page views that carried a click on a link in the named set, one per view.",
  scrapers: "Visits whose user agent is a crawler, a headless browser, an HTTP library or empty, counted apart from people.",
  cell: "Visits whose first view (source, medium, campaign, landing) or last view (exit) has this value.",
  funnel: "Visits that did this step after every step before it, other pages allowed between.",
} as const;
const made = (id: string, label: string, counts: string, value: number, recomputed: number, inputs: Record<string, unknown>): ProbedView => ({
  id, label, value, counts, inputs, recomputed, agrees: value === recomputed,
});
const COLUMN: Record<Dimension, { column: string; rank: "rn" | "rl" }> = {
  source: { column: "source", rank: "rn" },
  medium: { column: "medium", rank: "rn" },
  campaign: { column: "campaign", rank: "rn" },
  landing: { column: "path", rank: "rn" },
  exit: { column: "path", rank: "rl" },
};
export function probeViews(
  db: Database,
  scope: string,
  windowRaw: string | null,
  who: "people" | "scrapers",
  now: number,
  ask: { dimension?: string | null; key?: string | null; steps?: string | null } = {},
): { as_of: string; scope: string; window: ViewWindow; numbers: ProbedView[] } {
  const window = windowOfViews(windowRaw);
  const since = sinceOf(now, window);
  const flag = who === "scrapers" ? 1 : 0;
  const visits = readVisits(db, scope, since);
  const mine = visits.filter((v) => matches(v, { who }));
  const one = (sql: string, ...args: (string | number)[]): number =>
    db.query<{ n: number }, (string | number)[]>(sql).get(...args)?.n ?? 0;
  const numbers: ProbedView[] = [
    made("visits", "Visits", COUNTS.visits, mine.length,
      one("SELECT COUNT(*) AS n FROM (SELECT DISTINCT day, visitor FROM traffic_view WHERE scope = ?1 AND at >= ?2 AND scraper = ?3)", scope, since, flag),
      { scope, window, who }),
    made("page_views", "Page views", COUNTS.page_views, mine.reduce((n, v) => n + v.pages, 0),
      one("SELECT COUNT(*) AS n FROM traffic_view WHERE scope = ?1 AND at >= ?2 AND scraper = ?3", scope, since, flag),
      { scope, window, who }),
    made("outbound", "Outbound clicks", COUNTS.outbound, mine.reduce((n, v) => n + v.outs.length, 0),
      one("SELECT COUNT(*) AS n FROM traffic_view WHERE scope = ?1 AND at >= ?2 AND scraper = ?3 AND out_host != ''", scope, since, flag),
      { scope, window, who }),
    made("scrapers", "Scraper visits", COUNTS.scrapers, visits.filter((v) => v.scraper).length,
      one("SELECT COUNT(*) AS n FROM (SELECT DISTINCT day, visitor FROM traffic_view WHERE scope = ?1 AND at >= ?2 AND scraper = 1)", scope, since),
      { scope, window }),
  ];
  const dimension = DIMENSIONS.find((d) => d === ask.dimension);
  if (dimension !== undefined && typeof ask.key === "string") {
    const key = ask.key === DIRECT && dimension === "source" ? "" : ask.key;
    const filter: Filter = { who, [dimension]: key };
    const { column, rank } = COLUMN[dimension];
    numbers.push(
      made("cell", `${dimension} ${ask.key}`, COUNTS.cell, visits.filter((v) => matches(v, filter)).length,
        one(
          `WITH h AS (SELECT ${column} AS c,
                  ROW_NUMBER() OVER (PARTITION BY day, visitor ORDER BY at, id) AS rn,
                  ROW_NUMBER() OVER (PARTITION BY day, visitor ORDER BY at DESC, id DESC) AS rl
             FROM traffic_view WHERE scope = ?1 AND at >= ?2 AND scraper = ?3)
           SELECT COUNT(*) AS n FROM h WHERE ${rank} = 1 AND c = ?4`,
          scope, since, flag, key,
        ),
        { scope, window, who, dimension, key }),
    );
  }
  const steps = parseSteps(ask.steps ?? null);
  if (steps.length > 0) {
    const rows = funnel(db, scope, window, { who }, steps, now);
    // The second route: for each visitor-day, walk the stored views in order.
    const reached = new Map<string, number>();
    const stored = db
      .query<{ day: string; visitor: string; path: string; out_host: string }, [string, number, number]>(
        "SELECT day, visitor, path, out_host FROM traffic_view WHERE scope = ?1 AND at >= ?2 AND scraper = ?3 ORDER BY at, id",
      )
      .all(scope, since, flag);
    const at = (e: string, s: string): boolean =>
      s === "/" ? e === "/" : s.startsWith("out:") ? e === s : e === s || e.startsWith(s.endsWith("/") ? s : `${s}/`);
    for (const r of stored) {
      const k = `${r.day}\0${r.visitor}`;
      let n = reached.get(k) ?? 0;
      for (const e of r.out_host === "" ? [r.path] : [r.path, `out:${r.out_host}`]) if (n < steps.length && at(e, steps[n]!)) n += 1;
      reached.set(k, n);
    }
    rows.forEach((row, i) => {
      const again = i === 0 ? reached.size : [...reached.values()].filter((n) => n >= i).length;
      numbers.push(made(`funnel_${i}`, row.step, COUNTS.funnel, row.visits, again, { scope, window, who, steps }));
    });
  }
  return { as_of: new Date(now).toISOString(), scope, window, numbers };
}
@@ -0,0 +1,233 @@
/**
 * The traffic view's figures (#9427): visits made from the stored page views,
 * narrowed by a filter, summarised into the panels, and a path from a source
 * through named steps to a goal. Plain data in, plain data out.
 *
 * A visit is one visitor's views in one scope on one day: the visitor hash is
 * salted daily, so a visitor cannot be followed past midnight, and a return in
 * the same day is the same visit. Its source, medium and campaign are those of
 * its first view; its landing page is the first path and its exit the last.
 * Scrapers (`sources.ts` `isScraperAgent`) are counted apart and never in the
 * people's figures.
 */
import type { Database } from "bun:sqlite";
export const DAY_MS = 86_400_000;
export const WINDOWS = { "7d": 7, "30d": 30, "90d": 90 } as const;
export type ViewWindow = keyof typeof WINDOWS;
export const windowOfViews = (raw: string | null): ViewWindow => (raw === "30d" || raw === "90d" ? raw : "7d");
/** The most views one read looks at. */
export const READ_MAX = 200_000;
export const DIMENSIONS = ["source", "medium", "campaign", "landing", "exit"] as const;
export type Dimension = (typeof DIMENSIONS)[number];
export interface Filter {
  readonly who: "people" | "scrapers";
  readonly source?: string;
  readonly medium?: string;
  readonly campaign?: string;
  readonly landing?: string;
  readonly exit?: string;
}
export interface Visit {
  readonly day: string;
  readonly visitor: string;
  readonly source: string;
  readonly medium: string;
  readonly campaign: string;
  readonly landing: string;
  readonly exit: string;
  readonly pages: number;
  /** Paths in order, each followed by `out:<host>` when that view carried a click. */
  readonly events: readonly string[];
  readonly outs: readonly string[];
  readonly scraper: boolean;
}
interface ViewRow {
  day: string;
  visitor: string;
  path: string;
  source: string;
  medium: string;
  campaign: string;
  out_host: string;
  scraper: number;
}
export const sinceOf = (now: number, window: ViewWindow): number => now - WINDOWS[window] * DAY_MS;
const HELD_MS = 15_000;
const held = new WeakMap<Database, Map<string, { at: number; visits: Visit[] }>>();
      
        sentinelcrew
        5d ago
        outdated
      
      This loads up to 200,000 rows into memory on every render and probe call. Aggregate in SQL, or cache the result briefly.
    
/** Forget the visits held for a short while; the store calls it after it writes. */
export const forgetVisits = (db: Database): void => void held.delete(db);
/**
 * The visits of a scope since a time. A read builds up to `READ_MAX` rows, so
 * the result is held for `HELD_MS` per scope and minute: a page and its probe
 * calls share one read, and a write (`forgetVisits`) drops what is held.
 */
export function readVisits(db: Database, scope: string, since: number): Visit[] {
  const key = `${scope}\0${Math.floor(since / 60_000)}`;
  let mine = held.get(db);
  if (mine === undefined) held.set(db, (mine = new Map()));
  const now = Date.now();
  const hit = mine.get(key);
  if (hit !== undefined && now - hit.at < HELD_MS) return hit.visits;
  if (mine.size >= 64) mine.clear();
  const visits = buildVisits(db, scope, since);
  mine.set(key, { at: now, visits });
  return visits;
}
function buildVisits(db: Database, scope: string, since: number): Visit[] {
  const rows = db
    .query<ViewRow, [string, number, number]>(
      `SELECT day, visitor, path, source, medium, campaign, out_host, scraper
         FROM traffic_view WHERE scope = ?1 AND at >= ?2 ORDER BY at, id LIMIT ?3`,
    )
    .all(scope, since, READ_MAX);
  const made = new Map<string, { first: ViewRow; last: ViewRow; events: string[]; outs: string[]; pages: number }>();
  for (const r of rows) {
    const key = `${r.day}\0${r.visitor}\0${r.scraper}`;
    let v = made.get(key);
    if (v === undefined) {
      v = { first: r, last: r, events: [], outs: [], pages: 0 };
      made.set(key, v);
    }
    v.last = r;
    v.pages += 1;
    v.events.push(r.path);
    if (r.out_host !== "") {
      v.events.push(`out:${r.out_host}`);
      v.outs.push(r.out_host);
    }
  }
  return [...made.values()].map((v) => ({
    day: v.first.day,
    visitor: v.first.visitor,
    source: v.first.source,
    medium: v.first.medium,
    campaign: v.first.campaign,
    landing: v.first.path,
    exit: v.last.path,
    pages: v.pages,
    events: v.events,
    outs: v.outs,
    scraper: v.first.scraper === 1,
  }));
}
export function matches(v: Visit, f: Filter): boolean {
  return (
    v.scraper === (f.who === "scrapers") &&
    (f.source === undefined || v.source === f.source) &&
    (f.medium === undefined || v.medium === f.medium) &&
    (f.campaign === undefined || v.campaign === f.campaign) &&
    (f.landing === undefined || v.landing === f.landing) &&
    (f.exit === undefined || v.exit === f.exit)
  );
}
export interface Counted {
  readonly key: string;
  readonly visits: number;
}
function tally(visits: readonly Visit[], of: (v: Visit) => string, limit = 10): Counted[] {
  const by = new Map<string, number>();
  for (const v of visits) by.set(of(v), (by.get(of(v)) ?? 0) + 1);
  return [...by]
    .map(([key, n]) => ({ key, visits: n }))
    .sort((a, b) => b.visits - a.visits || a.key.localeCompare(b.key))
    .slice(0, limit);
}
export interface SourcesReport {
  readonly visits: number;
  readonly pageViews: number;
  readonly pagesPerVisit: number;
  /** The same window's scrapers, whatever the filter. */
  readonly scrapers: number;
  readonly people: number;
  readonly days: readonly Counted[];
  readonly sources: readonly Counted[];
  readonly mediums: readonly Counted[];
  readonly campaigns: readonly Counted[];
  readonly landings: readonly Counted[];
  readonly exits: readonly Counted[];
  readonly outbound: readonly Counted[];
}
export const DIRECT = "(direct)";
export function sourcesReport(db: Database, scope: string, window: ViewWindow, filter: Filter, now: number): SourcesReport {
  const all = readVisits(db, scope, sinceOf(now, window));
  const mine = all.filter((v) => matches(v, filter));
  const pageViews = mine.reduce((n, v) => n + v.pages, 0);
  const days = Array.from({ length: WINDOWS[window] }, (_, i) =>
    new Date(sinceOf(now, window) + (i + 1) * DAY_MS).toISOString().slice(0, 10),
  );
  const perDay = new Map(tally(mine, (v) => v.day, 1000).map((c) => [c.key, c.visits]));
  const outs = new Map<string, number>();
  for (const v of mine) for (const o of v.outs) outs.set(o, (outs.get(o) ?? 0) + 1);
  return {
    visits: mine.length,
    pageViews,
    pagesPerVisit: mine.length === 0 ? 0 : Math.round((pageViews / mine.length) * 10) / 10,
    scrapers: all.filter((v) => v.scraper).length,
    people: all.filter((v) => !v.scraper).length,
    days: days.map((key) => ({ key, visits: perDay.get(key) ?? 0 })),
    sources: tally(mine, (v) => v.source || DIRECT),
    mediums: tally(mine.filter((v) => v.medium !== ""), (v) => v.medium),
    campaigns: tally(mine.filter((v) => v.campaign !== ""), (v) => v.campaign),
    landings: tally(mine, (v) => v.landing),
    exits: tally(mine, (v) => v.exit),
    outbound: [...outs].map(([key, visits]) => ({ key, visits })).sort((a, b) => b.visits - a.visits || a.key.localeCompare(b.key)),
  };
}
export interface FunnelStep {
  readonly step: string;
  readonly visits: number;
  /** Visits lost since the step before, and their share of it. */
  readonly lost: number;
  readonly lostShare: number;
}
export const STEPS_MAX = 6;
const STEP = /^(?:out:[a-z0-9.-]{1,253}|\/[A-Za-z0-9/_:.~%+-]{0,199})$/;
/** The steps a person typed: a path (a prefix, `/` meaning the front page only) or `out:<host>`. Invalid ones are dropped. */
export function parseSteps(raw: string | null): string[] {
  return (raw ?? "")
    .split(",")
    .map((s) => s.trim())
    .filter((s) => STEP.test(s))
    .slice(0, STEPS_MAX);
}
const hit = (event: string, step: string): boolean =>
  step === "/" ? event === "/" : step.startsWith("out:") ? event === step : event === step || event.startsWith(step.endsWith("/") ? step : `${step}/`);
/** Visits that did each step in order (other pages may come between), with drop-off. The first row is every visit the filter keeps. */
export function funnel(db: Database, scope: string, window: ViewWindow, filter: Filter, steps: readonly string[], now: number): FunnelStep[] {
  const mine = readVisits(db, scope, sinceOf(now, window)).filter((v) => matches(v, filter));
  const progress = mine.map((v) => {
    let reached = 0;
    for (const e of v.events) if (reached < steps.length && hit(e, steps[reached]!)) reached += 1;
    return reached;
  });
  const rows: FunnelStep[] = [{ step: "From this source", visits: mine.length, lost: 0, lostShare: 0 }];
  let before = mine.length;
  steps.forEach((step, i) => {
    const n = progress.filter((p) => p > i).length;
    rows.push({ step, visits: n, lost: before - n, lostShare: before === 0 ? 0 : Math.round(((before - n) / before) * 100) });
    before = n;
  });
  return rows;
}
@@ -0,0 +1,141 @@
/**
 * Page views, kept for the traffic view (`views_report.ts`). A request only
 * pushes a plain row onto an in-memory buffer (`noteView`, no SQL); a timer
 * writes the whole buffer in one transaction (`flushViews`), off the request
 * path. The buffer is bounded, so a flood drops views rather than memory, and
 * a day keeps at most `DAY_MAX` rows per scope. Rows are kept 90 days.
 *
 * The visitor is the existing daily-salted hash (`web/measure.ts` `saltFor`),
 * with the scope; the salt is read once a day, not per view.
 */
import type { Database } from "bun:sqlite";
import { quietly } from "../core/write_clock.ts";
import { sha256 } from "../kernel/hash.ts";
import { withoutWaiting } from "../kernel/db.ts";
import { dayOf, saltFor } from "../web/measure.ts";
import { forgetVisits } from "./views_report.ts";
import { isScraperAgent, type Origin } from "./sources.ts";
export const BUFFER_MAX = 5_000;
export const DAY_MAX = 50_000;
/** Views one visitor may leave in a scope in a day: one client cannot crowd the day out with invented paths. */
export const VISITOR_DAY_MAX = 300;
export const KEEP_DAYS = 90;
const FLUSH_MS = 10_000;
interface Row {
  readonly scope: string;
  readonly day: string;
  readonly at: number;
  readonly visitor: string;
  readonly path: string;
  readonly origin: Origin;
  readonly scraper: boolean;
}
const buffers = new WeakMap<Database, Row[]>();
const salts = new WeakMap<Database, { day: string; salt: string }>();
function saltOf(db: Database, at: number): string {
  const day = dayOf(at);
  const held = salts.get(db);
  if (held?.day === day) return held.salt;
  const salt = saltFor(db, at);
  salts.set(db, { day, salt });
  return salt;
}
/** Push one view; false when the buffer is full and it was dropped. No SQL after the day's first. */
export function noteView(
  db: Database,
  view: { scope: string; path: string; origin: Origin; client: string; agent: string; at?: number },
): boolean {
  const at = view.at ?? Date.now();
  let buffer = buffers.get(db);
  if (buffer === undefined) {
    buffer = [];
    buffers.set(db, buffer);
  }
  if (buffer.length >= BUFFER_MAX) return false;
  buffer.push({
    scope: view.scope,
    day: dayOf(at),
    at,
    visitor: sha256(`${saltOf(db, at)}\0views\0${view.scope}\0${view.client}\0${view.agent}`).slice(0, 32),
    path: view.path.slice(0, 200),
    origin: view.origin,
    scraper: isScraperAgent(view.agent),
  });
  return true;
}
/** Views waiting to be written. */
export const pendingViews = (db: Database): number => buffers.get(db)?.length ?? 0;
/** Write what is buffered, in one transaction; on failure the rows stay for the next try. */
export function flushViews(db: Database, now: number = Date.now()): number {
  const buffer = buffers.get(db);
  if (buffer === undefined || buffer.length === 0) return 0;
  const taken = buffer.slice();
  withoutWaiting(
    db,
    db.transaction(() =>
      quietly(db, () => {
        const insert = db.query(
          `INSERT INTO traffic_view (scope, day, at, visitor, path, ref_host, source, medium, campaign, out_host, scraper)
           VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)`,
        );
        const held = new Map<string, number>();
        const heldVisitor = new Map<string, number>();
        const ofVisitor = db.query<{ n: number }, [string, string, number]>(
          "SELECT COUNT(*) AS n FROM traffic_view WHERE scope = ?1 AND visitor = ?2 AND at >= ?3 AND at < ?3 + 86400000",
        );
        const count = db.query<{ n: number }, [string, number]>(
          "SELECT COUNT(*) AS n FROM traffic_view WHERE scope = ?1 AND at >= ?2 AND at < ?2 + 86400000",
        );
        for (const r of taken) {
          const key = `${r.scope}\0${r.day}`;
          if (!held.has(key)) held.set(key, count.get(r.scope, Date.parse(`${r.day}T00:00:00Z`))?.n ?? 0);
          const n = held.get(key)!;
          if (n >= DAY_MAX) continue;
          const who = `${key}\0${r.visitor}`;
          if (!heldVisitor.has(who)) heldVisitor.set(who, ofVisitor.get(r.scope, r.visitor, Date.parse(`${r.day}T00:00:00Z`))?.n ?? 0);
          const mine = heldVisitor.get(who)!;
          if (mine >= VISITOR_DAY_MAX) continue;
          heldVisitor.set(who, mine + 1);
          held.set(key, n + 1);
          const o = r.origin;
          insert.run(r.scope, r.day, r.at, r.visitor, r.path, o.refHost, o.source, o.medium, o.campaign, o.out, r.scraper ? 1 : 0);
        }
        db.query("DELETE FROM traffic_view WHERE at < ?1").run(now - KEEP_DAYS * 86_400_000);
      }),
    ),
  );
  buffer.splice(0, taken.length);
  forgetVisits(db);
  return taken.length;
}
/** The flush timer, for the server; `stop` writes what is left. */
export function createViewWriter(
  db: Database,
  log: { warn(message: string, fields?: Record<string, unknown>): void },
  flushMs: number = FLUSH_MS,
): { stop(): void } {
  const run = (): void => {
    try {
      flushViews(db);
    } catch (error) {
      log.warn("could not write page views", { error: String(error) });
    }
  };
  const tick = setInterval(run, flushMs);
  tick.unref();
  return {
    stop() {
      clearInterval(tick);
      run();
    },
  };
}
@@ -22,6 +22,7 @@
import { entry as crewHandover } from "./crewhandover.ts";
import { entry as crewSplits } from "./crewsplits.ts";
import { entry as domains } from "./domains.ts";
import { entry as edgeOnly } from "./edgeonly.ts";
import { entry as traffic } from "./trafficview.ts";
import { entry as feed } from "./feed.ts";
import { entry as forgeView } from "./forgeview.ts";
import { entry as front } from "./front.ts";
@@ -81,6 +82,7 @@
export const ENTRIES: readonly PaletteEntry[] = [
  valves,
  access,
  edgeOnly,
  traffic,
  audit,
  secrets,
  crewSplits,
@@ -0,0 +1,182 @@
/**
 * Where people come from, in the terminal (#9427): the web page's reads
 * (`web/pages/sources.ts`) from the same figures (`perf/views_report.ts`),
 * gated the same way. A forge admin opens the forge's view from the palette;
 * a repository's admin opens that app's view from the palette while the
 * repository is open; anyone else is offered nothing and, were the screen
 * reached, sees no rows.
 *
 * `w` changes the window, `s` flips people and scrapers, Enter narrows every
 * panel to the value under the cursor, `c` clears the narrowing, `t` types
 * the steps of a path to a goal. Read-only.
 */
import { getAppByRepo } from "../../core/apps.ts";
import { canAdmin, type Repo } from "../../core/repos.ts";
import {
  DIMENSIONS,
  DIRECT,
  WINDOWS,
  funnel,
  parseSteps,
  sourcesReport,
  windowOfViews,
  type Counted,
  type Dimension,
  type Filter,
  type ViewWindow,
} from "../../perf/views_report.ts";
import type { Ctx, Screen } from "../app/app.ts";
import type { Command } from "../app/palette.ts";
import { binding } from "../app/keys.ts";
import { style } from "../term/style.ts";
import { truncate } from "../term/width.ts";
import type { Session } from "../session.ts";
import { fill, len, split } from "../ui/layout.ts";
import { ListState, drawList, promptScreen, type ListRow } from "./common.ts";
import type { PaletteEntry } from "./entries.ts";
/** The scope this viewer may read, or undefined: the forge for an admin, the app for its repository's admin. */
export function readableScope(f: Session, repo?: Repo): string | undefined {
  if (f.user === undefined) return undefined;
  if (repo === undefined) return f.user.isAdmin ? "forge" : undefined;
  if (!canAdmin(f.db, repo, f.user)) return undefined;
  return getAppByRepo(f.db, repo.id)?.id;
}
interface Line {
  readonly row: ListRow;
  readonly dimension?: Dimension;
  readonly key?: string;
}
const PANELS: ReadonlyArray<[string, keyof ReturnType<typeof sourcesReport>, Dimension | null]> = [
  ["Source", "sources", "source"],
  ["Medium", "mediums", "medium"],
  ["Campaign", "campaigns", "campaign"],
  ["Landing", "landings", "landing"],
  ["Left at", "exits", "exit"],
  ["Outbound", "outbound", null],
];
export function trafficLines(f: Session, scope: string, window: ViewWindow, filter: Filter, steps: readonly string[], now = Date.now()): Line[] {
  const r = sourcesReport(f.db, scope, window, filter, now);
  const out: Line[] = [
    {
      row: {
        title: `${filter.who === "scrapers" ? "Scraper visits" : "Visits"}: ${r.visits} · page views ${r.pageViews} · ${r.pagesPerVisit} pages a visit`,
        meta: `${WINDOWS[window]} days · ${filter.who === "scrapers" ? `${r.people} people` : `${r.scrapers} scrapers`} kept apart`,
      },
    },
  ];
  for (const [title, field, dimension] of PANELS) {
    for (const c of r[field] as readonly Counted[])
      out.push({
        row: { title: `${title}: ${c.key}`, meta: `${c.visits}` },
        ...(dimension === null ? {} : { dimension, key: c.key }),
      });
  }
  if (steps.length > 0)
    for (const s of funnel(f.db, scope, window, filter, steps, now))
      out.push({ row: { title: `Path: ${s.step}`, meta: `${s.visits} visits · lost ${s.lost} (${s.lostShare}%)` } });
  out.push({
    row: {
      title: "Not seen",
      note: "Outbound clicks are reported by the visitor's browser and can be forged; scripts off, Do-Not-Track and hidden referrers are not counted.",
    },
  });
  return out;
}
export function trafficScreen(f: Session, ctx: Ctx, repo?: Repo): Screen {
  const state = new ListState();
  let window: ViewWindow = "7d";
  let who: Filter["who"] = "people";
  let narrow: Partial<Record<Dimension, string>> = {};
  let steps: string[] = [];
  const scope = (): string | undefined => readableScope(f, repo);
  const filter = (): Filter => ({ who, ...narrow });
  const lines = (): Line[] => {
    const s = scope();
    return s === undefined ? [] : trafficLines(f, s, window, filter(), steps);
  };
  const screen: Screen = {
    title: repo === undefined ? "Traffic sources" : `${repo.ownerName}/${repo.name} — Traffic sources`,
    keys: [
      binding(["j", "k"], "move cursor"),
      binding(["w"], "window"),
      binding(["s"], "people/scrapers"),
      binding(["enter"], "narrow"),
      binding(["c"], "clear"),
      binding(["t"], "path steps"),
    ],
    render(area, s) {
      const c = ctx.theme.c;
      const [head, list] = split(area, "v", [len(1), fill(1)]);
      const on = DIMENSIONS.filter((d) => narrow[d] !== undefined).map((d) => `${d}=${narrow[d] === "" ? DIRECT : narrow[d]}`);
      s.text(head!.x, head!.y, truncate(`${WINDOWS[window]} days · ${who}${on.length > 0 ? ` · ${on.join(" ")}` : ""}${steps.length > 0 ? ` · path ${steps.join(",")}` : ""}`, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w);
      drawList(s, list!, c, state, lines().map((l) => l.row));
    },
    key(e) {
      if (scope() === undefined) return false;
      const all = lines();
      if (state.key(e, all.length)) return true;
      if (e.key === "w") {
        window = windowOfViews(window === "7d" ? "30d" : window === "30d" ? "90d" : "7d");
        return true;
      }
      if (e.key === "s") {
        who = who === "people" ? "scrapers" : "people";
        return true;
      }
      if (e.key === "c") {
        narrow = {};
        return true;
      }
      if (e.key === "enter") {
        const at = all[state.index];
        if (at?.dimension !== undefined && at.key !== undefined) narrow = { ...narrow, [at.dimension]: at.key === DIRECT ? "" : at.key };
        return true;
      }
      if (e.key === "t") {
        ctx.push(
          promptScreen(ctx, screen, {
            title: "Path to a goal",
            label: "Steps, in order",
            placeholder: "/, /explore, out:github.com",
            value: steps.join(","),
            onSubmit: (value) => {
              steps = parseSteps(value);
              return undefined;
            },
          }),
        );
        return true;
      }
      return false;
    },
  };
  return screen;
}
/** The palette's forge-wide view, to site admins. */
export const entry: PaletteEntry = {
  id: "traffic-sources",
  title: "Traffic sources",
  hint: "where people come from and where they go, for the forge",
  offered: (f) => readableScope(f) !== undefined,
  run: (f, ctx) => ctx.push(trafficScreen(f, ctx)),
};
/** The palette's per-app view while a repository is open, to the repository's admins. */
export function trafficCommands(f: Session, ctx: Ctx, repo: Repo | undefined): Command[] {
  if (repo === undefined || readableScope(f, repo) === undefined) return [];
  return [
    {
      id: "traffic-sources-app",
      title: `${repo.ownerName}/${repo.name} — Traffic sources`,
      hint: "where this app's visitors come from and go",
      run: () => ctx.push(trafficScreen(f, ctx, repo)),
    },
  ];
}
@@ -138,6 +138,7 @@
import { iterationsCommands } from "./iterations.ts";
import { labelsCommands } from "./labels.ts";
import { appEnvCommands } from "./appenv.ts";
import { autonomyCommands } from "./autonomy.ts";
import { trafficCommands } from "./trafficview.ts";
import { externalEditor, editorHint, hasExternalEditor } from "./external.ts";
import { finderScreen } from "./finder.ts";
import { issuesScreen } from "./issues.ts";
@@ -1456,6 +1457,7 @@
export function workspaceScreen(
      ...iterationsCommands(f, ctx, repo),
      ...appEnvCommands(f, ctx, repo),
      ...autonomyCommands(f, ctx, repo),
      ...trafficCommands(f, ctx, repo),
    ],
    get title() {
      return sourceName(src);
@@ -8,16 +8,20 @@
 * being measured belongs to the page that was loaded. Nothing
 * is sent under Do-Not-Track or with the `forge_measure=off` cookie; the
 * server checks both again. A metric it could not measure is 0 (or absent).
 * It also sends the referrer, the query string and the host of the last external
 * link clicked; the server keeps only the referrer's host, the three `utm_`
 * fields, and a clicked host that is in the named set (`perf/sources.ts`).
 */
export const beaconJs = `!function(n,d,w){
if(n.doNotTrack=="1"||/(^|; )forge_measure=off/.test(d.cookie)||!n.sendBeacon||!w.PerformanceObserver)return;
var c=0,i=0,l=0,x=0,a=location.pathname;
var c=0,i=0,l=0,x=0,u="",a=location.pathname;
function o(t,f,e){try{new PerformanceObserver(function(s){s.getEntries().forEach(f)}).observe(Object.assign({type:t,buffered:true},e))}catch(_){}}
o("largest-contentful-paint",function(e){l=e.startTime});
o("layout-shift",function(e){e.hadRecentInput||(c+=e.value)});
o("event",function(e){e.interactionId&&e.duration>i&&(i=e.duration)},{durationThreshold:40});
d.addEventListener("click",function(e){var t=e.target,k=t&&t.closest&&t.closest("a[href]");k&&k.hostname&&k.hostname!=location.hostname&&(u=k.hostname)},true);
function s(){if(x)return;x=1;var v=performance.getEntriesByType("navigation")[0]||{};
n.sendBeacon("/_measure",JSON.stringify({path:a,ttfb:v.responseStart||0,load:v.loadEventEnd||0,lcp:l,inp:i,cls:c}))}
n.sendBeacon("/_measure",JSON.stringify({path:a,ttfb:v.responseStart||0,load:v.loadEventEnd||0,lcp:l,inp:i,cls:c,ref:d.referrer.slice(0,300),q:location.search.slice(0,300),out:u}))}
d.addEventListener("visibilitychange",function(){d.visibilityState=="hidden"&&s()});
w.addEventListener("pagehide",s)
}(navigator,document,window)`;
@@ -32,9 +32,22 @@
export interface Report {
  readonly lcp: number | null;
  readonly inp: number | null;
  readonly cls: number;
  /** document.referrer, location.search and the host of the link the visitor left by; absent from an older beacon. */
  readonly ref?: string;
  readonly q?: string;
  readonly out?: string;
}
const FIELDS = new Set(["path", "ttfb", "load", "lcp", "inp", "cls"]);
const FIELDS = new Set(["path", "ttfb", "load", "lcp", "inp", "cls", "ref", "q", "out"]);
/** An optional text field: absent is fine, a wrong kind or a long or control-laden one is malformed. */
function optionalText(body: Record<string, unknown>, key: string, max: number): string | undefined | "malformed" {
  const given = body[key];
  if (given === undefined) return undefined;
  // eslint-disable-next-line no-control-regex
  if (typeof given !== "string" || given.length > max || /[\u0000-\u001f\u007f]/.test(given)) return "malformed";
  return given;
}
const bounded = (value: unknown, max: number): number | undefined =>
  typeof value === "number" && Number.isFinite(value) && value >= 0 && value <= max
@@ -71,6 +84,10 @@
export function parseReport(text: string): Report | "malformed" {
    // 0 is how the beacon says "not measured".
    times[key] = value === 0 ? null : value;
  }
  const ref = optionalText(body, "ref", 300);
  const q = optionalText(body, "q", 300);
  const out = optionalText(body, "out", 253);
  if (ref === "malformed" || q === "malformed" || out === "malformed") return "malformed";
  return {
    path,
    ttfb: times["ttfb"] ?? null,
@@ -78,6 +95,9 @@
export function parseReport(text: string): Report | "malformed" {
    lcp: times["lcp"] ?? null,
    inp: times["inp"] ?? null,
    cls,
    ...(ref === undefined ? {} : { ref }),
    ...(q === undefined ? {} : { q }),
    ...(out === undefined ? {} : { out }),
  };
}
@@ -796,7 +796,8 @@
export function appVisitorsView(v: AppVitals): Html {
      : `p75 load each day over the last 30 days, latest ${ms(latest ?? undefined)}, range ${ms(Math.min(...known))} to ${ms(Math.max(...known))}`,
  );
  return card(
    html`<p class="muted small mb">
    html`<p class="small mb"><a href="app/traffic">Where visitors come from and where they go</a></p>
      <p class="muted small mb">
        Counted by the beacon the template ships, through the forge: no
        address is kept, and a visitor is the same only within a day, so the
        7 day figure adds the days up.
@@ -0,0 +1,298 @@
/**
 * Where people come from and where they go (#9427): one view for the forge
 * (`/settings/traffic/sources`, admin) and one per app
 * (`/:owner/:name/settings/app/traffic`, those who administer it). Visits over
 * time, sources, mediums, campaigns, landing pages and exits; choosing a value
 * narrows every other panel; a path from a source through named steps to a
 * goal; GitHub's own traffic for the forge. Every number links to the probe
 * (`/api/v1/admin/traffic/probe`) that counts it again another way. Built from
 * stored views (`perf/views_store.ts`) in one indexed read.
 */
import { getAppById } from "../../core/apps.ts";
import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
import type { Ctx } from "../../http/context.ts";
import { githubSummary } from "../../perf/github_traffic.ts";
import { outboundNames } from "../../perf/sources.ts";
import {
  DIMENSIONS,
  DIRECT,
  WINDOWS,
  funnel,
  parseSteps,
  sinceOf,
  sourcesReport,
  windowOfViews,
  type Counted,
  type Dimension,
  type Filter,
  type ViewWindow,
} from "../../perf/views_report.ts";
import { notFoundPage, settingsLayout, signInFirst } from "../chrome.ts";
import { escapeHtml, html, raw, type Html } from "../html.ts";
import { page } from "../layout.ts";
import { btn, card, empty, seg } from "../ui.ts";
import { accountSections } from "./origins.ts";
import { settingsPageResponse, timedAdminRepo } from "./settings.ts";
import { getAppByRepo } from "../../core/apps.ts";
const CHART_W = 600;
const CHART_H = 72;
const PROBE = "/api/v1/admin/traffic/probe";
interface Query {
  readonly scope: string;
  readonly base: string;
  readonly extra: Record<string, string>;
  readonly window: ViewWindow;
  readonly filter: Filter;
  readonly steps: string[];
}
const text = (v: string | null): string | undefined => (v !== null && v.length > 0 && v.length <= 200 ? v : undefined);
function queryOf(url: URL, scope: string, base: string, extra: Record<string, string>): Query {
  const p = url.searchParams;
  const pick = (k: string): { [k: string]: string } => {
    const v = text(p.get(k));
    return v === undefined ? {} : { [k]: v };
  };
  const source = p.get("source") === DIRECT ? { source: "" } : pick("source");
  return {
    scope,
    base,
    extra,
    window: windowOfViews(p.get("window")),
    filter: {
      who: p.get("who") === "scrapers" ? "scrapers" : "people",
      ...source,
      ...pick("medium"),
      ...pick("campaign"),
      ...pick("landing"),
      ...pick("exit"),
    },
    steps: parseSteps(p.get("steps")),
  };
}
/** This view's address with the given parameters changed; a null removes one. */
function href(q: Query, change: Record<string, string | null>): string {
  const params = new URLSearchParams(q.extra);
  params.set("window", q.window);
  if (q.filter.who === "scrapers") params.set("who", "scrapers");
  for (const d of DIMENSIONS) {
    const v = q.filter[d];
    if (v !== undefined) params.set(d, d === "source" && v === "" ? DIRECT : v);
  }
  if (q.steps.length > 0) params.set("steps", q.steps.join(","));
  for (const [k, v] of Object.entries(change)) v === null ? params.delete(k) : params.set(k, v);
  return `${q.base}?${params}`;
}
const probe = (q: Query, ask: Record<string, string>): string =>
  `${PROBE}?${new URLSearchParams({ scope: q.scope, window: q.window, who: q.filter.who, ...ask })}`;
const figure = (q: Query, n: number, ask: Record<string, string>): Html =>
  html`<a href="${probe(q, ask)}" title="How this was counted">${n}</a>`;
function columns(days: readonly Counted[], label: string): Html {
  const top = Math.max(1, ...days.map((d) => d.visits));
  const step = CHART_W / Math.max(1, days.length);
  const bars = days.map((d, i) => {
    const h = (d.visits / top) * CHART_H;
    return h <= 0 ? "" : `<rect x="${(i * step).toFixed(1)}" y="${(CHART_H - h).toFixed(1)}" width="${Math.max(1, step - 1).toFixed(1)}" height="${h.toFixed(1)}" fill="currentColor" fill-opacity="0.7"><title>${escapeHtml(d.key)}: ${d.visits}</title></rect>`;
  });
  return raw(`<svg viewBox="0 0 ${CHART_W} ${CHART_H}" width="${CHART_W}" height="${CHART_H}" role="img" aria-label="${escapeHtml(label)}">${bars.join("")}</svg>`);
}
function panel(q: Query, title: string, rows: readonly Counted[], dimension: Dimension | null): Html {
  return card(
    rows.length === 0
      ? html`<p class="muted small">None in the window.</p>`
      : html`<div class="table-scroll">
          <table class="table" data-dense>
            <caption class="sr-only">${title}</caption>
            <thead>
              <tr><th scope="col">${title}</th><th scope="col" class="num">Visits</th></tr>
            </thead>
            <tbody>
              ${rows.map(
                (r) => html`<tr>
                  <td class="break">${dimension === null
                    ? r.key
                    : html`<a href="${href(q, { [dimension]: r.key })}" title="Narrow every panel to this">${r.key}</a>`}</td>
                  <td class="num">${dimension === null
                    ? figure(q, r.visits, { number: "outbound" })
                    : figure(q, r.visits, { dimension, key: r.key })}</td>
                </tr>`,
              )}
            </tbody>
          </table>
        </div>`,
  );
}
function filters(q: Query): Html {
  const on = DIMENSIONS.filter((d) => q.filter[d] !== undefined);
  if (on.length === 0) return html``;
  return html`<p class="small mb">
    Narrowed to
    ${on.map((d) => html`<strong>${d}</strong> ${q.filter[d] === "" ? DIRECT : q.filter[d]} <a href="${href(q, { [d]: null })}">(clear)</a> `)}
  </p>`;
}
function path(q: Query, now: number, db: Ctx["app"]["db"]): Html {
  const rows = q.steps.length === 0 ? [] : funnel(db, q.scope, q.window, q.filter, q.steps, now);
  return card(
    html`<p class="muted small mb">
        Name the steps a visit takes, in order, ending in a goal: a path such as
        <code>/signup</code> (a path also counts its sub-pages) or an outbound
        click such as <code>out:github.com</code>. Other pages may come between.
        Choose a source above to start from it.
      </p>
      <form class="row-fields" method="get" action="${q.base}">
        ${Object.entries({ ...q.extra, window: q.window, ...(q.filter.who === "scrapers" ? { who: "scrapers" } : {}), ...Object.fromEntries(DIMENSIONS.flatMap((d) => (q.filter[d] === undefined ? [] : [[d, q.filter[d] === "" && d === "source" ? DIRECT : q.filter[d]!]]))) }).map(
          ([k, v]) => html`<input type="hidden" name="${k}" value="${v}" />`,
        )}
        <input class="input mono" name="steps" value="${q.steps.join(",")}" placeholder="/, /explore, out:github.com" aria-label="Steps, comma separated" />
        ${btn("Follow", { size: "sm", variant: "primary" })}
      </form>
      ${rows.length === 0
        ? null
        : html`<div class="table-scroll mt">
            <table class="table" data-dense>
              <caption class="sr-only">Visits at each step and who dropped off</caption>
              <thead>
                <tr><th scope="col">Step</th><th scope="col" class="num">Visits</th><th scope="col" class="num">Lost</th></tr>
              </thead>
              <tbody>
                ${rows.map(
                  (r, i) => html`<tr>
                    <td class="break">${r.step}</td>
                    <td class="num">${figure(q, r.visits, { steps: q.steps.join(","), number: `funnel_${i}` })}</td>
                    <td class="num">${i === 0 ? "–" : html`${r.lost} (${r.lostShare}%)`}</td>
                  </tr>`,
                )}
              </tbody>
            </table>
          </div>`}`,
  );
}
function github(db: Ctx["app"]["db"], q: Query, now: number): Html {
  const rows = githubSummary(db, new Date(sinceOf(now, q.window)).toISOString().slice(0, 10));
  if (rows.length === 0)
    return html`<p class="muted small mt">
      GitHub's own traffic is not pulled: set <code>FORGE_GITHUB_TOKEN</code> (a token that may read repository
      traffic) and <code>FORGE_GITHUB_REPOS</code> (<code>owner/name,…</code>) and it is stored daily, since GitHub keeps only 14 days.
    </p>`;
  return card(
    html`<div class="table-scroll">
      <table class="table" data-dense>
        <caption class="sr-only">GitHub repository traffic, pulled daily</caption>
        <thead>
          <tr><th scope="col">Repository</th><th scope="col" class="num">Views</th><th scope="col" class="num">Viewers</th><th scope="col" class="num">Clones</th><th scope="col" class="num">Cloners</th><th scope="col">Top referrers, last pull</th></tr>
        </thead>
        <tbody>
          ${rows.map(
            (r) => html`<tr>
              <td class="break">${r.repo}</td>
              <td class="num">${r.views}</td><td class="num">${r.viewers}</td>
              <td class="num">${r.clones}</td><td class="num">${r.cloners}</td>
              <td class="break">${r.error !== null ? `Last pull failed: ${r.error}` : r.referrers.map((x) => `${x.key} ${x.count}`).join(", ") || "none"}</td>
            </tr>`,
          )}
        </tbody>
      </table>
    </div>`,
    { head: html`<h2>GitHub</h2>` },
  );
}
export function sourcesBody(ctx: Ctx, q: Query, label: string): Html {
  const db = ctx.app.db;
  const now = Date.now();
  const r = sourcesReport(db, q.scope, q.window, q.filter, now);
  const scraping = q.filter.who === "scrapers";
  const empties = r.people + r.scrapers === 0;
  return html`<div class="mb">
      ${seg(Object.keys(WINDOWS).map((w) => ({ label: `${WINDOWS[w as ViewWindow]} days`, href: href(q, { window: w }), current: w === q.window })))}
      ${seg([
        { label: "People", href: href(q, { who: null }), current: !scraping },
        { label: "Scrapers", href: href(q, { who: "scrapers" }), current: scraping },
      ])}
    </div>
    ${filters(q)}
    ${empties
      ? empty({ icon: "users", title: `No visits recorded for ${label} in ${WINDOWS[q.window]} days`, note: "Views are written every few seconds; a page counts when it is left." })
      : html`${card(
            html`<dl class="stats">
                <div><dt>${scraping ? "Scraper visits" : "Visits"}</dt><dd>${figure(q, r.visits, { number: scraping ? "scrapers" : "visits" })}</dd></div>
                <div><dt>Pages per visit</dt><dd><a href="${probe(q, { number: "page_views" })}" title="How this was counted">${r.pagesPerVisit}</a></dd></div>
                <div><dt>Page views</dt><dd>${figure(q, r.pageViews, { number: "page_views" })}</dd></div>
                <div><dt>${scraping ? "People, kept apart" : "Scrapers, kept apart"}</dt><dd>${figure(q, scraping ? r.people : r.scrapers, { number: scraping ? "visits" : "scrapers" })}</dd></div>
              </dl>
              <div class="table-scroll muted mt">${columns(r.days, `Visits each day over the last ${WINDOWS[q.window]} days`)}</div>`,
          )}
          ${panel(q, "Source", r.sources, "source")}
          ${panel(q, "Medium (utm_medium)", r.mediums, "medium")}
          ${panel(q, "Campaign (utm_campaign)", r.campaigns, "campaign")}
          ${panel(q, "Landing page", r.landings, "landing")}
          ${panel(q, "Where people left (last page)", r.exits, "exit")}
          ${panel(q, `Outbound clicks to ${outboundNames().join(", ")}`, r.outbound, null)}`}
    <h2 class="mt">A path to a goal</h2>
    ${path(q, now, db)}
    ${q.scope === "forge" ? github(db, q, now) : null}
    <p class="small muted mt">
      <strong>How it was counted.</strong> A visit is one visitor's views on one day: a visitor is a hash of the address and
      browser under a salt that changes daily and is then deleted, so no address or cookie is kept, nobody is followed
      across days, and the 7- or 30-day figures add days up. A visit's source, medium and campaign are those of its first
      page (<code>utm_source</code> wins over the referring site; a referrer from this site is not a source; none is "${DIRECT}").
      Scrapers are told by user agent (crawlers, headless browsers, HTTP libraries, none at all) and counted apart.
      ${q.scope === "forge" ? "Forge pages are filed by their route (/:owner/:name), never by a path that could name a private repository." : null}
    </p>
    <p class="small muted">
      <strong>What it cannot see.</strong> Visitors with scripts off or who ask not to be measured (Do-Not-Track), anyone whose
      browser hides the referrer, pages that do not run the beacon (GitHub's own pages, other people's sites; GitHub's
      figures are pulled in separately above), apps whose <code>beacon.js</code> predates the referrer fields, and an outbound
      click when the browser leaves before it can send. Outbound clicks are reported by the visitor's own browser,
      so anyone can post one that never happened: read them as a hint, not a count. Click a number to see how it was counted.
    </p>`;
}
/** GET /settings/traffic/sources — the forge's view, or `?app=<id>` for any app; admins only. */
export function sourcesPage(ctx: Ctx): Response {
  if (!ctx.auth.user) return signInFirst(ctx);
  if (!ctx.auth.user.isAdmin) return notFoundPage(ctx);
  notePhase(ctx.request, "session", takeSessionTime(ctx.request));
  const asked = ctx.url.searchParams.get("app");
  const record = asked === null ? undefined : getAppById(ctx.app.db, asked);
  if (asked !== null && record === undefined) return notFoundPage(ctx);
  const q = queryOf(ctx.url, record?.id ?? "forge", "/settings/traffic/sources", record === undefined ? {} : { app: record.id });
  const label = record === undefined ? "this forge" : "this app";
  const rendering = performance.now();
  const response = page({
    ctx,
    title: "Sources",
    body: html`<div class="page-head"><h1>Where people come from</h1></div>
      ${settingsLayout(accountSections("traffic", { admin: true }), sourcesBody(ctx, q, label), "Account settings")}`,
  });
  notePhase(ctx.request, "render", performance.now() - rendering);
  return response;
}
/** GET /:owner/:name/settings/app/traffic — one app's view, for those who administer it. */
export function appSourcesPage(ctx: Ctx): Response {
  const repo = timedAdminRepo(ctx);
  if (!repo) return notFoundPage(ctx);
  const record = getAppByRepo(ctx.app.db, repo.id);
  if (record === undefined) {
    notePhase(ctx.request, "render", 0);
    return notFoundPage(ctx);
  }
  const q = queryOf(ctx.url, record.id, `/${ctx.params.owner}/${ctx.params.name}/settings/app/traffic`, {});
  const rendering = performance.now();
  const response = settingsPageResponse(ctx, repo, "app", "Where people come from", sourcesBody(ctx, q, "this app"));
  notePhase(ctx.request, "render", performance.now() - rendering);
  return response;
}
@@ -171,6 +171,9 @@
export function trafficPage(ctx: Ctx): Response {
          ${top("Address prefix", tops.prefixes, t)}
          ${top("Country", tops.countries, t)}
          ${top("Request signature", tops.signatures, t)}
          <p class="mt">
            <a href="/settings/traffic/sources">Where people come from and where they go: sources, campaigns, a path to a goal</a>
          </p>
          <p class="mt">
            <a href="/settings/traffic/scrapes">What was scraped, and who: reads, networks, captures</a>
          </p>
@@ -107,6 +107,17 @@
describe("a report through the proxy", () => {
    expect((await report("notes", { ...good, app: ids["diary"] })).status).toBe(400);
  });
  test("its referrer and campaign are buffered against that app, as host and utm fields only (#9427)", async () => {
    const { flushViews } = await import("../src/perf/views_store.ts");
    await report("notes", { ...good, path: "/a?x=1", ref: "https://espodev.com/blog?secret=1", q: "?utm_source=post&utm_campaign=c1&token=zzz", out: "github.com" });
    flushViews(app.db);
    const row = app.db
      .query<Record<string, unknown>, [string]>("SELECT path, ref_host, source, campaign, out_host FROM traffic_view WHERE scope = ?1 ORDER BY id DESC LIMIT 1")
      .get(ids["notes"]!);
    expect(row).toEqual({ path: "/a", ref_host: "espodev.com", source: "post", campaign: "c1", out_host: "github.com" });
    expect(app.db.query("SELECT 1 FROM traffic_view WHERE scope = ?1").all(ids["diary"]!)).toEqual([]);
  });
  test("keeps no address and refuses a foreign origin, a bad body and an opted-out visitor", async () => {
    expect((await report("notes", good, { origin: "http://evil.test" })).status).toBe(403);
    expect((await report("notes", "nope")).status).toBe(400);
@@ -125,6 +125,7 @@
test("a site-admin person is offered the whole roster", () => {
    ["bulk", "Bulk actions", "your last bulk actions; u undoes one inside its day"],
    ["subscriptions", "Crew subscriptions", "each account's budget and rest; wake one, set its budget, add or remove one"],
    ["edge-only-accounts", "Edge-only accounts", "who would be locked out if the platform edge sign-in went off"],
    ["traffic-sources", "Traffic sources", "where people come from and where they go, for the forge"],
    ["audit", "Who can do what", "each app's doors from its contract.json: yes, own or no"],
    ["archived", "Archived repositories", "put away, not gone: open one or wake it"],
    ["cloud", "Cloud accounts", "provider tokens, what they are used for, today's calls"],
@@ -190,6 +191,7 @@
test("a site-admin agent is offered the roster without the person-only entries",
    ["proposal-tokens", "Proposal tokens", "mint, list and revoke what a child proposes with"],
    ["can", "What I can do", "your role and what it allows here"],
    ["bugs", "Bugs", "open bugs and the bug clock"],
    ["traffic-sources", "Traffic sources", "where people come from and where they go, for the forge"],
    ["audit", "Who can do what", "each app's doors from its contract.json: yes, own or no"],
    ["archived", "Archived repositories", "put away, not gone: open one or wake it"],
    ["cloud", "Cloud accounts", "provider tokens, what they are used for, today's calls"],
@@ -0,0 +1,285 @@
/**
 * Where people come from and where they go (#9427): the beacon's referrer and
 * campaign fields are reduced to what may be kept, buffered off the request
 * path, told apart from scrapers, narrowed by a filter, followed through a
 * path to a goal, and every number is counted again by the probe.
 */
import { afterAll, beforeAll, describe, expect, test } from "bun:test";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createApp, type App } from "../src/app.ts";
import { createSession } from "../src/core/sessions.ts";
import { createUser } from "../src/core/users.ts";
import { serve, type Servers } from "../src/http/server.ts";
import { entriesOf, githubSummary, pullGithub, githubConfig, type Fetch } from "../src/perf/github_traffic.ts";
import { hostOf, originOf, outboundOf, isScraperAgent } from "../src/perf/sources.ts";
import { probeViews } from "../src/perf/views_probe.ts";
import { funnel, parseSteps, readVisits, sourcesReport } from "../src/perf/views_report.ts";
import { VISITOR_DAY_MAX, flushViews, noteView, pendingViews } from "../src/perf/views_store.ts";
import { parseReport } from "../src/web/measure.ts";
import { entry, readableScope, trafficLines } from "../src/tui/screens/trafficview.ts";
import { beaconJs } from "../src/web/beacon.ts";
let dir: string;
let app: App;
let servers: Servers;
let cookie: string;
const PERSON = "Mozilla/5.0 (X11; Linux x86_64) Chrome/130.0 Safari/537.36";
const BOT = "Mozilla/5.0 (compatible; Reflectionbot/1.0)";
beforeAll(async () => {
  dir = await mkdtemp(join(tmpdir(), "forge-sources-"));
  app = createApp({ dataDir: dir, port: 0 });
  servers = await serve(app);
  const made = await createUser(app.db, { origin: "seed", trusted: true, username: "ada", email: "[email protected]", password: "correct-horse-battery", isAdmin: true });
  if (!made.ok) throw new Error(made.error);
  cookie = `forge_session=${createSession(app.db, made.value.id).token}`;
});
afterAll(async () => {
  await servers.stop();
  await rm(dir, { recursive: true, force: true });
});
const at = Date.now();
const view = (scope: string, client: string, path: string, o: { ref?: string; q?: string; out?: string } = {}, agent = PERSON, when = at) =>
  noteView(app.db, { scope, path, origin: originOf(o.ref, o.q, "forge.test", o.out), client, agent, at: when });
describe("what is kept of a referrer", () => {
  test("only hosts, only the three utm fields, only named outbound links", () => {
    expect(hostOf("https://www.Espodev.com/blog/post?token=secret#x")).toBe("espodev.com");
    expect(hostOf("javascript:alert(1)")).toBe("");
    expect(hostOf("https://bad host/")).toBe("");
    const o = originOf("https://news.example/item?id=1", "?utm_source=Post&utm_medium=social&utm_campaign=launch-1&other=1", "forge.test", "github.com");
    expect(o).toEqual({ refHost: "news.example", source: "post", medium: "social", campaign: "launch-1", out: "github.com" });
    expect(originOf("https://forge.test/explore", "", "forge.test:3000", undefined).source).toBe("");
    expect(originOf(undefined, "?utm_source=<script>", "x", "evil.example")).toEqual({ refHost: "", source: "", medium: "", campaign: "", out: "" });
    expect(outboundOf("gist.github.com")).toBe("github.com");
    expect(outboundOf("notgithub.com")).toBe("");
  });
  test("a program is not a person", () => {
    expect(isScraperAgent(BOT)).toBe(true);
    expect(isScraperAgent("")).toBe(true);
    expect(isScraperAgent("HeadlessChrome/120")).toBe(true);
    expect(isScraperAgent(PERSON)).toBe(false);
  });
  test("the beacon's new fields are optional, bounded and the old beacon still passes", () => {
    const base = { path: "/", cls: 0 };
    expect(parseReport(JSON.stringify(base))).toEqual({ path: "/", ttfb: null, load: null, lcp: null, inp: null, cls: 0 });
    expect(parseReport(JSON.stringify({ ...base, ref: "https://a.test/", q: "?utm_source=x", out: "github.com" }))).toMatchObject({ ref: "https://a.test/", out: "github.com" });
    expect(parseReport(JSON.stringify({ ...base, ref: "x".repeat(301) }))).toBe("malformed");
    expect(parseReport(JSON.stringify({ ...base, out: 5 }))).toBe("malformed");
    expect(beaconJs).toContain("document".slice(0, 0) + "referrer");
    expect(new TextEncoder().encode(beaconJs).length).toBeLessThan(2048);
  });
});
describe("buffered views, visits and filters", () => {
  test("a view costs no SQL until the flush, and the flush writes them in one go", () => {
    // Ada from a post (campaign), reads two pages and leaves for GitHub.
    view("forge", "10.0.0.1", "/", { ref: "https://blog.example/post", q: "?utm_source=post&utm_medium=social&utm_campaign=launch" }, PERSON, at);
    view("forge", "10.0.0.1", "/explore", { ref: "https://forge.test/" }, PERSON, at + 1000);
    view("forge", "10.0.0.1", "/explore", { out: "github.com" }, PERSON, at + 2000);
    // Bo comes from the personal site and bounces.
    view("forge", "10.0.0.2", "/", { ref: "https://espodev.com/" }, PERSON, at + 3000);
    // Cy arrives direct and signs up.
    view("forge", "10.0.0.3", "/", {}, PERSON, at + 4000);
    view("forge", "10.0.0.3", "/signup", {}, PERSON, at + 5000);
    // A scraper, and another scope.
    view("forge", "10.0.0.9", "/", {}, BOT, at + 6000);
    view("app1", "10.0.0.1", "/", { ref: "https://espodev.com/" }, PERSON, at + 7000);
    const before = app.db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM traffic_view").get()!.n;
    expect(before).toBe(0);
    expect(pendingViews(app.db)).toBe(8);
    expect(flushViews(app.db, at + 10_000)).toBe(8);
    expect(pendingViews(app.db)).toBe(0);
    expect(app.db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM traffic_view").get()!.n).toBe(8);
  });
  test("visits take the first view's source and the last view's exit; scrapers stay apart; scopes stay apart", () => {
    const visits = readVisits(app.db, "forge", at - 1000);
    expect(visits.length).toBe(4);
    const r = sourcesReport(app.db, "forge", "7d", { who: "people" }, at + 20_000);
    expect(r.visits).toBe(3);
    expect(r.scrapers).toBe(1);
    expect(r.pageViews).toBe(6);
    expect(r.pagesPerVisit).toBe(2);
    expect(r.sources.map((s) => `${s.key}:${s.visits}`).sort()).toEqual(["(direct):1", "espodev.com:1", "post:1"]);
    expect(r.campaigns).toEqual([{ key: "launch", visits: 1 }]);
    expect(r.landings[0]).toEqual({ key: "/", visits: 3 });
    expect(r.exits.map((e) => e.key).sort()).toEqual(["/", "/explore", "/signup"]);
    expect(r.outbound).toEqual([{ key: "github.com", visits: 1 }]);
    expect(r.days.at(-1)?.visits).toBeGreaterThan(0);
    expect(sourcesReport(app.db, "forge", "7d", { who: "scrapers" }, at + 20_000).visits).toBe(1);
    expect(sourcesReport(app.db, "app1", "7d", { who: "people" }, at + 20_000).visits).toBe(1);
  });
  test("choosing a value narrows every other panel", () => {
    const r = sourcesReport(app.db, "forge", "7d", { who: "people", source: "post" }, at + 20_000);
    expect(r.visits).toBe(1);
    expect(r.exits).toEqual([{ key: "/explore", visits: 1 }]);
    expect(r.campaigns).toEqual([{ key: "launch", visits: 1 }]);
    expect(r.outbound).toEqual([{ key: "github.com", visits: 1 }]);
    const direct = sourcesReport(app.db, "forge", "7d", { who: "people", source: "" }, at + 20_000);
    expect(direct.exits).toEqual([{ key: "/signup", visits: 1 }]);
  });
  test("a path from a source to a goal counts each step and what was lost", () => {
    expect(parseSteps("/, /signup, out:github.com, nonsense, ../x")).toEqual(["/", "/signup", "out:github.com"]);
    const all = funnel(app.db, "forge", "7d", { who: "people" }, ["/", "/signup"], at + 20_000);
    expect(all.map((s) => s.visits)).toEqual([3, 3, 1]);
    expect(all[2]).toMatchObject({ lost: 2, lostShare: 67 });
    const fromPost = funnel(app.db, "forge", "7d", { who: "people", source: "post" }, ["/explore", "out:github.com"], at + 20_000);
    expect(fromPost.map((s) => s.visits)).toEqual([1, 1, 1]);
    const wrongOrder = funnel(app.db, "forge", "7d", { who: "people" }, ["out:github.com", "/"], at + 20_000);
    expect(wrongOrder.map((s) => s.visits)).toEqual([3, 1, 0]);
  });
  test("the probe counts every number again by another route and they agree", () => {
    const p = probeViews(app.db, "forge", "7d", "people", at + 20_000, { dimension: "source", key: "post", steps: "/,/signup" });
    const ids = p.numbers.map((n) => n.id);
    expect(ids).toEqual(expect.arrayContaining(["visits", "page_views", "outbound", "scrapers", "cell", "funnel_0", "funnel_2"]));
    for (const n of p.numbers) expect({ id: n.id, agrees: n.agrees }).toEqual({ id: n.id, agrees: true });
    expect(p.numbers.find((n) => n.id === "cell")?.value).toBe(1);
    const exit = probeViews(app.db, "forge", "7d", "people", at + 20_000, { dimension: "exit", key: "/explore" });
    expect(exit.numbers.find((n) => n.id === "cell")).toMatchObject({ value: 1, recomputed: 1 });
    const direct = probeViews(app.db, "forge", "7d", "people", at + 20_000, { dimension: "source", key: "(direct)" });
    expect(direct.numbers.find((n) => n.id === "cell")).toMatchObject({ value: 1, recomputed: 1 });
  });
  test("the buffer is bounded and a read of visits walks the index", () => {
    const plan = app.db
      .query("EXPLAIN QUERY PLAN SELECT day, visitor, path, source, medium, campaign, out_host, scraper FROM traffic_view WHERE scope = ?1 AND at >= ?2 ORDER BY at, id LIMIT ?3")
      .all("x", 0, 5)
      .map((r) => (r as { detail: string }).detail)
      .join("\n");
    expect(plan).toContain("traffic_view_scope_at");
    expect(plan).not.toMatch(/^SCAN (?!.*USING)/m);
  });
});
describe("through the doors", () => {
  test("a report on the forge's own door is buffered with its route, not its path, and the page and probe answer an admin alone", async () => {
    const res = await fetch(`${servers.url}/_measure`, {
      method: "POST",
      headers: { origin: servers.url, "user-agent": PERSON },
      body: JSON.stringify({ path: "/explore", cls: 0, ref: "https://news.example/x", q: "?utm_source=hn&utm_campaign=show" }),
    });
    expect(res.status).toBe(204);
    expect(pendingViews(app.db)).toBe(1);
    flushViews(app.db);
    const row = app.db.query<{ path: string; source: string; campaign: string }, []>("SELECT path, source, campaign FROM traffic_view WHERE source = 'hn'").get()!;
    expect(row).toEqual({ path: "/explore", source: "hn", campaign: "show" });
    const page = await fetch(`${servers.url}/settings/traffic/sources?window=7d&source=hn&steps=/explore,out:github.com`, { headers: { cookie } });
    expect(page.status).toBe(200);
    const body = await page.text();
    expect(body).toContain("What it cannot see");
    expect(body).toContain("Scrapers are told by user agent");
    expect(body).toContain("/api/v1/admin/traffic/probe?");
    expect(body).toContain("Narrowed to");
    expect(await (await fetch(`${servers.url}/settings/traffic/sources`)).text()).not.toContain("How it was counted");
    expect((await fetch(`${servers.url}/settings/traffic/sources?app=nope`, { headers: { cookie } })).status).toBe(404);
    const probe = await fetch(`${servers.url}/api/v1/admin/traffic/probe?scope=forge&number=visits`, { headers: { cookie } });
    expect(probe.status).toBe(200);
    const json = (await probe.json()) as { numbers: { id: string; agrees: boolean }[] };
    expect(json.numbers.map((n) => n.id)).toEqual(["visits"]);
    expect(json.numbers[0]!.agrees).toBe(true);
    expect(await (await fetch(`${servers.url}/api/v1/admin/traffic/probe`)).text()).not.toContain("agrees");
  });
});
describe("GitHub's own traffic", () => {
  const answers: Record<string, unknown> = {
    referrers: [{ referrer: "espodev.com", count: 9, uniques: 4 }],
    views: { count: 5, uniques: 3, views: [{ timestamp: "2026-11-06T00:00:00Z", count: 5, uniques: 3 }] },
    clones: { count: 2, uniques: 1, clones: [{ timestamp: "2026-11-07T00:00:00Z", count: 2, uniques: 1 }] },
  };
  const seen: { url: string; auth: string }[] = [];
  const fake: Fetch = async (url, init) => {
    seen.push({ url, auth: init.headers["authorization"] ?? "" });
    const kind = url.includes("referrers") ? "referrers" : url.includes("views") ? "views" : "clones";
    return Response.json(answers[kind]);
  };
  test("needs a token and a valid repository, validates answers, and stores them", async () => {
    expect(githubConfig({})).toBeNull();
    expect(githubConfig({ FORGE_GITHUB_TOKEN: "t", FORGE_GITHUB_REPOS: "../x, a/b" })?.repos).toEqual(["a/b"]);
    expect(entriesOf("views", { views: [{ timestamp: "nope", count: 1, uniques: 1 }] }, "2026-11-08")).toBe("malformed");
    expect(entriesOf("referrers", [{ referrer: "<b>", count: 1, uniques: 1 }], "2026-11-08")).toBe("malformed");
    const stored = await pullGithub(app.db, { token: "tok", repos: ["trevato/forge"] }, fake, Date.parse("2026-11-08T03:00:00Z"));
    expect(stored).toBe(1);
    expect(seen.map((s) => s.url)).toEqual([
      "https://api.github.com/repos/trevato/forge/traffic/popular/referrers",
      "https://api.github.com/repos/trevato/forge/traffic/views",
      "https://api.github.com/repos/trevato/forge/traffic/clones",
    ]);
    expect(seen[0]!.auth).toBe("Bearer tok");
    const [s] = githubSummary(app.db, "2026-11-01");
    expect(s).toMatchObject({ repo: "trevato/forge", views: 5, viewers: 3, clones: 2, error: null });
    expect(s!.referrers).toEqual([{ key: "espodev.com", count: 9, uniques: 4 }]);
    // Pulling again the same day replaces rather than doubles.
    await pullGithub(app.db, { token: "tok", repos: ["trevato/forge"] }, fake, Date.parse("2026-11-08T05:00:00Z"));
    expect(githubSummary(app.db, "2026-11-01")[0]!.views).toBe(5);
    expect(JSON.stringify(app.db.query("SELECT * FROM github_pull").all())).not.toContain("tok");
  });
  test("a refusal is recorded, not thrown, and stores nothing new", async () => {
    const refuse: Fetch = async () => new Response("no", { status: 403 });
    expect(await pullGithub(app.db, { token: "tok", repos: ["a/b"] }, refuse)).toBe(0);
    const row = app.db.query<{ error: string }, []>("SELECT error FROM github_pull WHERE repo = 'a/b'").get()!;
    expect(row.error).toBe("referrers: GitHub answered 403");
  });
});
describe("review: caps, cache, gates and the terminal door", () => {
  test("one visitor cannot fill a day with invented paths", () => {
    const scope = "capped-scope";
    for (let i = 0; i < VISITOR_DAY_MAX + 20; i++) view(scope, "10.9.9.9", `/junk-${i}`);
    view(scope, "10.1.1.1", "/real");
    flushViews(app.db);
    const n = () =>
      app.db.query<{ n: number }, [string]>("SELECT COUNT(*) AS n FROM traffic_view WHERE scope = ?1").get(scope)!.n;
    expect(n()).toBe(VISITOR_DAY_MAX + 1);
    expect(app.db.query("SELECT 1 FROM traffic_view WHERE scope = ?1 AND path = '/real'").all(scope)).toHaveLength(1);
  });
  test("a read is held briefly and a write drops what is held", () => {
    const scope = "held-scope";
    const since = Date.now() - 86_400_000;
    view(scope, "10.2.2.2", "/a");
    flushViews(app.db);
    const first = readVisits(app.db, scope, since);
    expect(readVisits(app.db, scope, since)).toBe(first);
    view(scope, "10.3.3.3", "/b");
    flushViews(app.db);
    expect(readVisits(app.db, scope, since)).toHaveLength(2);
  });
  test("a non-admin gets 404 on both pages and the probe; the pages carry the security floor", async () => {
    const made = await createUser(app.db, { origin: "seed", trusted: true, username: "guest1", email: "[email protected]", password: "correct-horse-battery", isAdmin: false });
    if (!made.ok) throw new Error(made.error);
    const plain = `forge_session=${createSession(app.db, made.value.id).token}`;
    for (const path of ["/settings/traffic/sources", "/ada/none/settings/app/traffic"])
      expect((await fetch(`${servers.url}${path}`, { headers: { cookie: plain } })).status).toBe(404);
    expect((await fetch(`${servers.url}/api/v1/admin/traffic/probe?scope=forge`, { headers: { cookie: plain } })).status).not.toBe(200);
    const ok = await fetch(`${servers.url}/settings/traffic/sources`, { headers: { cookie } });
    expect(ok.status).toBe(200);
    expect(ok.headers.get("x-content-type-options")).toBe("nosniff");
    expect(ok.headers.get("referrer-policy")).toBeTruthy();
    expect(ok.headers.get("content-security-policy")).toBeTruthy();
    expect(await ok.text()).toContain("reported by the visitor");
  });
  test("the terminal reads what the page reads, to the same people", () => {
    const admin = app.db.query<{ id: string }, []>("SELECT id FROM user WHERE username = 'ada'").get()!;
    const asAdmin = { db: app.db, app, user: { id: admin.id, isAdmin: true } } as never;
    const asGuest = { db: app.db, app, user: { id: "x", isAdmin: false } } as never;
    expect(readableScope(asAdmin)).toBe("forge");
    expect(readableScope(asGuest)).toBeUndefined();
    expect(readableScope({ db: app.db, app } as never)).toBeUndefined();
    expect(entry.offered(asGuest)).toBe(false);
    expect(trafficLines(asAdmin, "forge", "7d", { who: "people" }, ["/"]).length).toBeGreaterThan(1);
  });
});
		@@ -14,4 +14,4 @@ A new read of a table that grows (`event`, `issue`, `run`, `comment`) names the
14	14
15	15	A new `/settings…` page route goes in `src/http/settings_routes.ts` (`settingsRoutes`), not `src/http/server.ts`.
16	16
17		State shared between modules through process memory, not the database, is named here. `src/runs/held.ts`: each dispatcher pass (`src/crew/dispatch.ts`, `pump`) records which queued agent runs it held and why (valve, pace, limit), and vitals (`src/kernel/vitals.ts`) leaves those out of `queue:agent`. A record older than `HELD_FRESH_MS` is ignored, so a stopped dispatcher cannot hide its queue.
	17	State shared between modules through process memory, not the database, is named here. `src/perf/views_report.ts`: `readVisits` holds each scope and window's visits for 15 seconds (dropped by `flushViews`), so a page and its probe share one read; `src/perf/views_store.ts`: the view buffer, flushed every 10 seconds. The traffic sources view's terminal door is `src/tui/screens/trafficview.ts` (palette entry for admins, per-app command for a repository's admins). `src/runs/held.ts`: each dispatcher pass (`src/crew/dispatch.ts`, `pump`) records which queued agent runs it held and why (valve, pace, limit), and vitals (`src/kernel/vitals.ts`) leaves those out of `queue:agent`. A record older than `HELD_FRESH_MS` is ignored, so a stopped dispatcher cannot hide its queue.
		@@ -137,6 +137,7 @@ Machine routes are not pages and have no entry: `/audit.csv`,
137	137	- `/settings/origins` — `pages/origins.ts` (1,497 lines) · other forges this one reads and mirrors from · `/migrate` · none seen.
138	138	- `/settings/security` — `pages/security.ts` · the Shield and the evidence an admin needs to act with no AI · `/settings/traffic`, `/settings/backups` · none seen.
139	139	- `/settings/traffic` — `pages/traffic.ts` · who used the answering time in the last 15 minutes · `/settings/usage`, `/settings/telemetry` · none seen.
	140	- `/settings/traffic/sources` — `pages/sources.ts` · where people come from and where they go: sources, campaigns, landing pages, exits, a path to a goal, GitHub traffic; each number links to its probe · `/settings/traffic`, `/settings/traffic/scrapes` · none seen.
140	141	- `/settings/traffic/scrapes` — `pages/scrapes.ts` · what was read by repository and path, which networks and addresses did it, a bounded capture, the abuse export · `/settings/traffic`, `/settings/security` · none seen.
141	142
142	143	## Operator views — why each is a page (#2696)
		@@ -0,0 +1,60 @@
	1	# Where people come from and where they go
	2
	3	One view for the forge (`/settings/traffic/sources`, admin) and one per app
	4	(`/:owner/:name/settings/app/traffic`, whoever administers the repository).
	5
	6	What is recorded. The page-view beacon (`src/web/beacon.ts`, on every forge
	7	page and in every app template) also sends `document.referrer`, the query string
	8	and the host of the last external link clicked. The server keeps only the
	9	referrer's host, `utm_source`/`utm_medium`/`utm_campaign` (lowercase, a safe
	10	alphabet, 64 characters), and a clicked host that is in the named set
	11	(`github.com`, `espodev.com`; `FORGE_OUTBOUND_LINKS=host,host` replaces it).
	12	No cookie, no address, no third party; the visitor is the existing daily-salted
	13	hash. Forge pages are filed by route pattern, never by a path that could name a
	14	private repository.
	15
	16	How it is counted. A visit is one visitor's views in one scope on one day.
	17	Source, medium and campaign come from its first view (`utm_source` wins over the
	18	referring site; a referrer on the same host is navigation, not a source), the
	19	landing page is its first path, the exit its last. Scrapers (crawler, headless
	20	browser, HTTP library or no user agent) are stored flagged and shown apart.
	21
	22	Cheap under load. A beacon request pushes one row onto an in-memory buffer
	23	(bounded at 5,000) and answers; a timer writes the buffer every 10 s in one
	24	transaction (`src/perf/views_store.ts`). A day keeps at most 50,000 rows per
	25	scope; rows are kept 90 days. The view reads one indexed range.
	26
	27	Filtering and the path. Clicking a source, medium, campaign, landing page or
	28	exit narrows every panel. "A path to a goal" takes steps such as
	29	`/, /explore, out:github.com`, in order, and shows visits and drop-off at each.
	30
	31	Probe. Every number links to `GET /api/v1/admin/traffic/probe`
	32	(`scope=forge\|<app id>`, `window`, `who`, `number`, or `dimension`+`key`, or
	33	`steps`), which counts it again in SQL by a different route and says whether the
	34	two agree.
	35
	36	GitHub. With `FORGE_GITHUB_TOKEN` (push access to read traffic) and
	37	`FORGE_GITHUB_REPOS=owner/name,…` the forge pulls referrers, views and clones
	38	once a day into `github_traffic`; GitHub keeps only 14 days. api.github.com is in
	39	`forge.json` egress.
	40
	41	Cannot see. Visitors with scripts off or Do-Not-Track, hidden referrers,
	42	GitHub's own pages (hence the pull), apps whose `static/beacon.js` predates these
	43	fields (new apps get the new one; copy it from the template to update an old one),
	44	and an outbound click when the browser leaves before it can send.
	45
	46	Forged clicks. An outbound click (`out`) is reported by the visitor's browser,
	47	so anyone can post `out=github.com` and inflate the outbound panel or an `out:`
	48	goal step. Counting clicks without server-side redirects cannot avoid that; the
	49	page says so. Treat outbound figures as a hint.
	50
	51	Caps. A visitor leaves at most 300 views per scope per day (app paths come
	52	from the client), so one client cannot fill the 50,000-row day with invented
	53	paths. A read builds its visits once per scope and window and holds them for 15
	54	seconds, dropped when views are written, so a page and its probe calls share it.
	55
	56	Terminal. Palette "Traffic sources" (`src/tui/screens/trafficview.ts`) is the
	57	forge's view for an admin; with a repository open, "<owner>/<name> — Traffic
	58	sources" is that app's view for those who administer it. A guest is offered
	59	neither. `w` window, `s` people/scrapers, Enter narrows, `c` clears, `t` types
	60	the path steps. It does not call the probe; use the web page for that.
		@@ -1,6 +1,7 @@
1	1	{
2	2	"egress": [
3		"api.anthropic.com"
	3	"api.anthropic.com",
	4	"api.github.com"
4	5	],
5	6	"app": {
6	7	"looks": ["/", "/front", "/explore", "/board", "/notes", "/feed", "/review", "/new", "/audit", "/releases", "/apps", "/repos", "/map", "/system", "/talk", "/flow", "/resilience", "/settings", "/preview-admin/sample", "/preview-admin/sample/pulls/3", "/preview-admin/sample/next", "/preview-admin/sample/bets", "/preview-admin/sample/waves", "/preview-admin/sample/milestones"],
		@@ -14,6 +14,8 @@ import { readBounded } from "../http/limits.ts";
14	14	import { clientAddress, clientBucket } from "../http/peer.ts";
15	15	import { optedOut, parseReport, REPORT_MAX } from "../web/measure.ts";
16	16	import type { AppTarget } from "./proxy.ts";
	17	import { originOf } from "../perf/sources.ts";
	18	import { noteView } from "../perf/views_store.ts";
17	19	import { recordAppVital } from "./vitals.ts";
18	20
19	21	const PER_CLIENT = 30;
		@@ -80,5 +82,12 @@ export async function appMeasure(
80	82	clientAddress(request),
81	83	request.headers.get("user-agent") ?? "",
82	84	);
	85	noteView(app.db, {
	86	scope: target.app.id,
	87	path: report.path.split(/[?#]/)[0] ?? "/",
	88	origin: originOf(report.ref, report.q, request.headers.get("host") ?? "", report.out),
		sentinelcrew 5d ago The path is client-supplied here. Normalise it, or cap the distinct paths per visitor, so one client cannot fill the 50,000-row day cap.
	89	client: clientAddress(request),
	90	agent: request.headers.get("user-agent") ?? "",
	91	});
83	92	return refuse(204);
84	93	}
		@@ -0,0 +1,10 @@
	1	import type { Route } from "../shared.ts";
	2
	3	export const route: Route = {
	4	method: "get",
	5	path: "/api/v1/admin/traffic/probe",
	6	summary:
	7	"Read the traffic view's numbers (where people come from) against their source (a person admin's token, any scope); query: scope (forge, or an app id), window (7d, 30d, 90d), who (people, scrapers), number (one id), dimension (source, medium, campaign, landing, exit) with key, steps (comma-separated path steps ending in a goal); answers { as_of, scope, window, numbers: [{ id, label, value, counts, inputs, recomputed, agrees }] }: recomputed counts the stored page views again in SQL",
	8	auth: true,
	9	refusals: { not_found: "the token is not a person who administers the forge, or the scope names no app" },
	10	};
		@@ -48,6 +48,7 @@ export { route as getApiV1AdminSettings } from "./getApiV1AdminSettings.ts";
48	48	export { route as getApiV1AdminShield } from "./getApiV1AdminShield.ts";
49	49	export { route as getApiV1AdminSubscriptions } from "./getApiV1AdminSubscriptions.ts";
50	50	export { route as getApiV1AdminTeams } from "./getApiV1AdminTeams.ts";
	51	export { route as getApiV1AdminTrafficProbe } from "./getApiV1AdminTrafficProbe.ts";
51	52	export { route as getApiV1Agents } from "./getApiV1Agents.ts";
52	53	export { route as getApiV1AgentsUsername } from "./getApiV1AgentsUsername.ts";
53	54	export { route as getApiV1Apps } from "./getApiV1Apps.ts";
		@@ -0,0 +1,1 @@
	1	aae41113f6b4ebac5a90d65f98a50a58189d9c009948997fd407646d93e29db4
		@@ -0,0 +1,38 @@
	1	// Where people come from and where they go (#9427, `perf/views_store.ts`): one
	2	// row per page view the beacon reports, for the forge (scope `forge`) or an app
	3	// (scope = its id). `visitor` is the daily-salted hash; no address, cookie or
	4	// user agent is kept. Read by scope and time, so the index leads with both.
	5	// `github_traffic` is GitHub's own repository traffic, which it keeps 14 days,
	6	// pulled daily; `github_pull` is when each repository was last pulled.
	7	// New tables only. Additive.
	8	export default `
	9	CREATE TABLE traffic_view (
	10	id INTEGER PRIMARY KEY,
	11	scope TEXT NOT NULL,
	12	day TEXT NOT NULL,
	13	at INTEGER NOT NULL,
	14	visitor TEXT NOT NULL,
	15	path TEXT NOT NULL,
	16	ref_host TEXT NOT NULL DEFAULT '',
	17	source TEXT NOT NULL DEFAULT '',
	18	medium TEXT NOT NULL DEFAULT '',
	19	campaign TEXT NOT NULL DEFAULT '',
	20	out_host TEXT NOT NULL DEFAULT '',
	21	scraper INTEGER NOT NULL DEFAULT 0
	22	);
	23	CREATE INDEX traffic_view_scope_at ON traffic_view (scope, at);
	24	CREATE TABLE github_traffic (
	25	repo TEXT NOT NULL,
	26	day TEXT NOT NULL,
	27	kind TEXT NOT NULL CHECK (kind IN ('views', 'clones', 'referrers')),
	28	key TEXT NOT NULL,
	29	count INTEGER NOT NULL,
	30	uniques INTEGER NOT NULL,
	31	PRIMARY KEY (repo, day, kind, key)
	32	) WITHOUT ROWID;
	33	CREATE TABLE github_pull (
	34	repo TEXT PRIMARY KEY,
	35	pulled_at INTEGER NOT NULL,
	36	error TEXT
	37	) WITHOUT ROWID;
	38	`;
		@@ -7,6 +7,8 @@
7	7
8	8
9	9	import { crawlReport } from "../../perf/crawlers.ts";
	10	import { getAppById } from "../../core/apps.ts";
	11	import { probeViews } from "../../perf/views_probe.ts";
10	12	import { kpiProbe } from "../../web/studio/kpi_probe.ts";
11	13	import { MAX_SECONDS, sampleStacks } from "../../perf/sample.ts";
12	14	import { listAccounts, removeAccount, retireAndRemoveAccount, retirementPlan, suspendAccount, unsuspendAccount, verifyAccount } from "../../actions/accounts.ts";
		@@ -147,6 +149,24 @@ export function adminRoutes(router: Router<Handler>): void {
147	149	return json(kpiProbe(ctx.app));
148	150	});
149	151
	152	// The traffic view's numbers, each counted again another way (`perf/views_probe.ts`).
	153	// `scope` is `forge` or an app's id; `number`, `dimension`+`key` and `steps` pick what to show.
	154	router.add("GET", "/api/v1/admin/traffic/probe", (ctx) => {
	155	notePhase(ctx.request, "session", takeSessionTime(ctx.request));
	156	const denied = accountAdmin(ctx, "ro");
	157	if (denied) return denied;
	158	const q = ctx.url.searchParams;
	159	const scope = q.get("scope") ?? "forge";
	160	if (scope !== "forge" && getAppById(ctx.app.db, scope) === undefined) return notFound();
	161	const probed = probeViews(ctx.app.db, scope, q.get("window"), q.get("who") === "scrapers" ? "scrapers" : "people", Date.now(), {
	162	dimension: q.get("dimension"),
	163	key: q.get("key"),
	164	steps: q.get("steps"),
	165	});
	166	const only = q.get("number");
	167	return json(only === null ? probed : { ...probed, numbers: probed.numbers.filter((n) => n.id === only) });
	168	});
	169
150	170	// The forge's capacity (`core/capacity.ts`): read and changed live by an
151	171	// admin, never by an agent; anyone else is told the door is not there.
152	172	router.add("GET", "/api/v1/admin/capacity", (ctx) => {