trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

See and steer the flows upstream and downstream: dependencies, sources, forks, followers and code scanning #9537

Merged scoutcrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/9434-see-and-steer-the-flows-upstream-and-dow → main 3 commits fork

  • Merged. scout/9434-see-and-steer-the-flows-upstream-and-dow was merged into main as 6fe60b0.

Review

Sign in to review this change.
17 files changed +797 −0
docs/features/flows.md added +11
@@ -0,0 +1,11 @@
1 # Flows
2
3 `/<owner>/<repo>/flows` is one picture per repository (#9434): on the left what flows in — the origin it was forked from, the upstream and sources it follows, the npm packages its `package.json` pins with the newest release upstream — and on the right what flows out: its forks and the mirror it pushes to. Each line has its state and when it last moved.
4
5 - **No crew, no model.** Reading pins, comparing versions (`src/flows/semver.ts`), the gap (patch, minor, major), and the page are deterministic. Test: `test/flows.test.ts`, no credential.
6 - **Read upstream.** An admin of the repository taps *Read upstream releases now* (same-origin POST); the radar asks the npm registry for each exact pin and stores the answer in `dep_latest`. The page itself never touches the network.
7 - **Doors.** A **Flows** entry in the repository header's More menu (and the palette, from the same `repoSteps` list), and the terminal's *Flows of a repository* verb (`src/tui/screens/flows.ts`). Both read through `readFlows` (`canRead`, `visibleForks`) and start the radar through `startRadar` (`src/flows/start.ts`): admin only, never an agent.
8 - **Bounded.** A run asks four packages at a time within 30 s total; a repository has one run at a time; one identity gets four runs a minute. Followed and upstream URLs are shown without userinfo, query or fragment.
9 - **Pins.** Only exact floors (`1.2.3`, `^1.2.3`, `~1.2.3`) are compared; tags and URLs are skipped. Names are validated before they reach a URL.
10
11 Left out: the CHANGELOG sections between pin and newest (dropped until a lookup can read one within a byte bound); a schedule and the issue the crew files from a report; advisories and code scanning (and the release block on a new high finding); acting on a line in place (update, follow, send back); GitHub release tags; the all-repositories admin view; "last moved" for forks. Each is its own pull on top of `dep_latest` and `readFlows`.
src/core/migration-pins/20261108140000-dep-latest.pin added +1
@@ -0,0 +1,1 @@
1 9140de6d2c8b874429288fea80000e75bf33daa8fe5ed5a1824f7378d0644d8b
src/core/migrations/20261108140000-dep-latest.ts added +14
@@ -0,0 +1,14 @@
1 // What the dependency radar last read about one package a repository pins
2 // (#9434): the newest version upstream and when it was read. One row per
3 // repository and package.
4 // New table only. Additive.
5 export default `
6 CREATE TABLE dep_latest (
7 repo_id TEXT NOT NULL REFERENCES repo(id) ON DELETE CASCADE,
8 name TEXT NOT NULL,
9 latest TEXT NOT NULL,
10 error TEXT NOT NULL DEFAULT '',
11 checked_at TEXT NOT NULL,
12 PRIMARY KEY (repo_id, name)
13 );
14 `;
src/flows/pins.ts added +35
@@ -0,0 +1,35 @@
1 /** The packages a repository pins, read from its package.json text (#9434). */
2
3 export interface Pin {
4 readonly name: string;
5 /** What package.json asks for, as written. */
6 readonly range: string;
7 /** The exact version it asks for, or "" when the range names none (`latest`, a URL, a tag). */
8 readonly version: string;
9 readonly dev: boolean;
10 }
11
12 /** An npm package name; anything else is never sent to a registry. */
13 export const NPM_NAME = /^(?:@[a-z0-9][a-z0-9._-]{0,213}\/)?[a-z0-9][a-z0-9._-]{0,213}$/;
14
15 const FLOOR = /^[\^~>=\s]*v?(\d{1,9}\.\d{1,9}\.\d{1,9}(?:-[0-9A-Za-z.-]{1,64})?)$/;
16
17 export function readPins(packageJson: string): Pin[] {
18 let parsed: unknown;
19 try {
20 parsed = JSON.parse(packageJson);
21 } catch {
22 return [];
23 }
24 if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return [];
25 const out: Pin[] = [];
26 for (const [field, dev] of [["dependencies", false], ["devDependencies", true]] as const) {
27 const block = (parsed as Record<string, unknown>)[field];
28 if (typeof block !== "object" || block === null || Array.isArray(block)) continue;
29 for (const [name, range] of Object.entries(block)) {
30 if (typeof range !== "string" || !NPM_NAME.test(name)) continue;
31 out.push({ name, range, version: FLOOR.exec(range)?.[1] ?? "", dev });
32 }
33 }
34 return out.sort((a, b) => a.name.localeCompare(b.name));
35 }
src/flows/radar.ts added +108
@@ -0,0 +1,108 @@
1 /**
2 * The dependency radar's deterministic core (#9434): read what a repository
3 * pins, ask upstream for the newest version of each, keep the answer. No
4 * model and no credential: the same pins and the same registry give the same
5 * rows. The lookup is injected so a test (and a forge with no outbound
6 * network) never reaches the registry.
7 */
8
9 import type { Database } from "bun:sqlite";
10 import { NPM_NAME, type Pin } from "./pins.ts";
11 import { gapOf, parseSemver, type Gap } from "./semver.ts";
12
13 export interface Upstream {
14 readonly latest: string;
15 }
16
17 /** Asks upstream about one package: its newest version, or why it could not say. */
18 export type Lookup = (name: string) => Promise<Upstream | string>;
19
20 export interface DepLine {
21 readonly name: string;
22 readonly range: string;
23 readonly pinned: string;
24 readonly latest: string;
25 readonly gap: Gap;
26 readonly dev: boolean;
27 readonly error: string;
28 readonly checkedAt: string | null;
29 }
30
31 const MAX_PINS = 200;
32 /** Registry asks in flight at once, and the whole run's budget: a run never holds a worker for minutes. */
33 export const CONCURRENCY = 4;
34 export const RUN_DEADLINE_MS = 30_000;
35
36 /** The npm registry's answer for one package; the name is validated before it is put in a URL. */
37 export const npmLookup: Lookup = async (name) => {
38 if (!NPM_NAME.test(name)) return "not a package name";
39 try {
40 const res = await fetch(`https://registry.npmjs.org/${name.replace("/", "%2F")}/latest`, {
41 signal: AbortSignal.timeout(5000),
42 headers: { accept: "application/json" },
43 });
44 if (!res.ok) return `registry answered ${res.status}`;
45 const body: unknown = await res.json();
46 const version = typeof body === "object" && body !== null ? (body as { version?: unknown }).version : undefined;
47 return typeof version === "string" && parseSemver(version) ? { latest: version } : "no version in the answer";
48 } catch {
49 return "registry not reachable";
50 }
51 };
52
53 /**
54 * Read upstream for every exact pin and store the answers, a few at a time and
55 * within a total deadline; a pin not reached keeps its last row. Returns how
56 * many rows were written.
57 */
58 export async function refreshRadar(
59 db: Database,
60 repoId: string,
61 pins: readonly Pin[],
62 lookup: Lookup,
63 now = new Date(),
64 deadlineMs = RUN_DEADLINE_MS,
65 ): Promise<number> {
66 const put = db.query(
67 `INSERT INTO dep_latest (repo_id, name, latest, error, checked_at) VALUES (?1, ?2, ?3, ?4, ?5)
68 ON CONFLICT (repo_id, name) DO UPDATE SET latest = excluded.latest,
69 error = excluded.error, checked_at = excluded.checked_at`,
70 );
71 const todo = pins.slice(0, MAX_PINS).filter((p) => p.version !== "");
72 const stop = Date.now() + deadlineMs;
73 let next = 0;
74 let written = 0;
75 const worker = async (): Promise<void> => {
76 while (next < todo.length && Date.now() < stop) {
77 const pin = todo[next++]!;
78 const answer = await lookup(pin.name);
79 const at = now.toISOString();
80 if (typeof answer === "string") put.run(repoId, pin.name, "", answer.slice(0, 200), at);
81 else put.run(repoId, pin.name, answer.latest, "", at);
82 written++;
83 }
84 };
85 await Promise.all(Array.from({ length: Math.min(CONCURRENCY, todo.length) }, worker));
86 return written;
87 }
88
89 interface Row { name: string; latest: string; error: string; checked_at: string }
90
91 /** Each pin with what the radar last read, outdated first. */
92 export function readRadar(db: Database, repoId: string, pins: readonly Pin[]): DepLine[] {
93 const rows = new Map(
94 db.query<Row, [string]>("SELECT name, latest, error, checked_at FROM dep_latest WHERE repo_id = ?1").all(repoId).map((r) => [r.name, r]),
95 );
96 const rank: Record<Gap, number> = { major: 0, minor: 1, patch: 2, none: 3 };
97 return pins
98 .map((p): DepLine => {
99 const r = rows.get(p.name);
100 const a = parseSemver(p.version);
101 const b = parseSemver(r?.latest ?? "");
102 return {
103 name: p.name, range: p.range, pinned: p.version, latest: r?.latest ?? "", dev: p.dev,
104 gap: a && b ? gapOf(a, b) : "none", error: r?.error ?? "", checkedAt: r?.checked_at ?? null,
105 };
106 })
107 .sort((x, y) => rank[x.gap] - rank[y.gap] || x.name.localeCompare(y.name));
108 }
src/flows/run.ts added +21
@@ -0,0 +1,21 @@
1 /** Run the radar for one repository: read package.json at the default branch's tip, ask upstream, store. */
2
3 import type { App } from "../app.ts";
4 import { readFileAt } from "../apps/checkout.ts";
5 import type { Repo } from "../core/repos.ts";
6 import { resolveOid } from "../git/merge.ts";
7 import { repoDir } from "../git/repos.ts";
8 import { readPins, type Pin } from "./pins.ts";
9 import { refreshRadar, type Lookup } from "./radar.ts";
10
11 export async function pinsOf(app: App, repo: Repo): Promise<Pin[]> {
12 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
13 const tip = await resolveOid(dir, `refs/heads/${repo.defaultBranch}`);
14 if (tip === undefined) return [];
15 const text = await readFileAt(dir, tip, "package.json");
16 return text === undefined ? [] : readPins(text);
17 }
18
19 export async function runRadar(app: App, repo: Repo, lookup: Lookup): Promise<number> {
20 return refreshRadar(app.db, repo.id, await pinsOf(app, repo), lookup);
21 }
src/flows/semver.ts added +38
@@ -0,0 +1,38 @@
1 /** Semantic versions, compared without a library (#9434). Plain data in, plain data out. */
2
3 export interface Semver {
4 readonly major: number;
5 readonly minor: number;
6 readonly patch: number;
7 readonly pre: string;
8 }
9
10 export type Gap = "none" | "patch" | "minor" | "major";
11
12 const RE = /^v?(\d{1,9})\.(\d{1,9})\.(\d{1,9})(?:-([0-9A-Za-z.-]{1,64}))?(?:\+[0-9A-Za-z.-]{1,64})?$/;
13
14 /** The version in `text` (a leading `v` allowed), or undefined when it is not one. */
15 export function parseSemver(text: string): Semver | undefined {
16 const m = RE.exec(text.trim());
17 if (!m) return undefined;
18 return { major: Number(m[1]), minor: Number(m[2]), patch: Number(m[3]), pre: m[4] ?? "" };
19 }
20
21 /** Negative when a < b, zero when equal, positive when a > b. A release outranks its prerelease. */
22 export function compareSemver(a: Semver, b: Semver): number {
23 if (a.major !== b.major) return a.major - b.major;
24 if (a.minor !== b.minor) return a.minor - b.minor;
25 if (a.patch !== b.patch) return a.patch - b.patch;
26 if (a.pre === b.pre) return 0;
27 if (a.pre === "") return 1;
28 if (b.pre === "") return -1;
29 return a.pre < b.pre ? -1 : 1;
30 }
31
32 /** How far `latest` is ahead of `pinned`: none when it is the same or behind. */
33 export function gapOf(pinned: Semver, latest: Semver): Gap {
34 if (compareSemver(latest, pinned) <= 0) return "none";
35 if (latest.major !== pinned.major) return "major";
36 if (latest.minor !== pinned.minor) return "minor";
37 return "patch";
38 }
src/flows/start.ts added +41
@@ -0,0 +1,41 @@
1 /**
2 * Starting the radar (#9434), one door for the page and the terminal: only an
3 * admin of the repository (never an agent) may, a repository has one run at a
4 * time, and one identity gets a few runs a minute.
5 */
6
7 import type { App } from "../app.ts";
8 import { RateLimiter } from "../core/ratelimit.ts";
9 import { canAdmin, type Repo } from "../core/repos.ts";
10 import type { User } from "../core/users.ts";
11 import { npmLookup, type Lookup } from "./radar.ts";
12 import { runRadar } from "./run.ts";
13
14 export type Started = { ok: true; read: number } | { ok: false; why: string };
15
16 const running = new Set<string>();
17 const RUNS_PER_MINUTE = 4;
18 const defaultLimiter = new RateLimiter({ perMinute: RUNS_PER_MINUTE, adminPerMinute: RUNS_PER_MINUTE, enabled: true });
19
20 /** May this person start a run: the same check on every door. */
21 export const mayStartRadar = (app: App, repo: Repo, user: User | undefined): user is User =>
22 user !== undefined && user.kind !== "agent" && canAdmin(app.db, repo, user);
23
24 export async function startRadar(
25 app: App,
26 repo: Repo,
27 user: User | undefined,
28 lookup: Lookup = npmLookup,
29 limiter: RateLimiter = defaultLimiter,
30 ): Promise<Started> {
31 if (!mayStartRadar(app, repo, user)) return { ok: false, why: "Only an admin of this repository can do this." };
32 if (running.has(repo.id)) return { ok: false, why: "A read is already running for this repository." };
33 const verdict = limiter.take(`flows:${user.id}`, "rw");
34 if (!verdict.allowed) return { ok: false, why: `Too many reads: try again in ${verdict.retryAfter}s.` };
35 running.add(repo.id);
36 try {
37 return { ok: true, read: await runRadar(app, repo, lookup) };
38 } finally {
39 running.delete(repo.id);
40 }
41 }
src/flows/view.ts added +95
@@ -0,0 +1,95 @@
1 /**
2 * One repository's flows as plain data (#9434): what comes in (its upstream,
3 * the sources it follows, the packages it pins) and what goes out (forks,
4 * the mirror it pushes to). Read from state the forge already keeps; no
5 * network and no model.
6 */
7
8 import type { Database } from "bun:sqlite";
9 import { canRead, getRepoById, visibleForks, type Repo } from "../core/repos.ts";
10 import type { User } from "../core/users.ts";
11
12 export interface FlowLine {
13 readonly kind: "origin" | "source" | "upstream" | "fork" | "mirror";
14 readonly label: string;
15 /** In-forge link, when the line is a repository here. */
16 readonly href: string;
17 readonly state: string;
18 /** When it last moved (ISO), or null. */
19 readonly at: string | null;
20 readonly problem: boolean;
21 }
22
23 export interface FlowView {
24 readonly inbound: FlowLine[];
25 readonly outbound: FlowLine[];
26 }
27
28 /** A followed URL as shown to any reader: no userinfo (`user:token@`), no query, no fragment. */
29 export function plainUrl(raw: string): string {
30 try {
31 const u = new URL(raw);
32 u.username = "";
33 u.password = "";
34 u.search = "";
35 u.hash = "";
36 return u.toString();
37 } catch {
38 // Not a URL (scp-style `user@host:path`): drop anything before the last `@`.
39 return raw.replace(/^[^@/]*@/, "");
40 }
41 }
42
43 const base = (r: Repo) => `/${r.ownerName}/${r.name}`;
44
45 function behindWords(behind: number, error: string): string {
46 if (error !== "") return "could not be read";
47 return behind === 0 ? "up to date" : `${behind} ${behind === 1 ? "change" : "changes"} behind`;
48 }
49
50 export function readFlows(db: Database, repo: Repo, viewer: User | undefined): FlowView {
51 const inbound: FlowLine[] = [];
52 const outbound: FlowLine[] = [];
53 if (repo.forkParentId !== null) {
54 const parent = getRepoById(db, repo.forkParentId);
55 if (parent && canRead(db, parent, viewer)) {
56 inbound.push({ kind: "origin", label: `${parent.ownerName}/${parent.name}`, href: base(parent), state: "forked from", at: null, problem: false });
57 }
58 }
59 const up = db
60 .query<{ url: string; branch: string; enabled: number; checked_at: string | null; behind: number; error: string }, [string]>(
61 "SELECT url, branch, enabled, checked_at, behind, error FROM repo_upstream WHERE repo_id = ?1",
62 )
63 .get(repo.id);
64 if (up) {
65 inbound.push({
66 kind: "upstream", label: `${plainUrl(up.url)} (${up.branch})`, href: "", at: up.checked_at, problem: up.error !== "",
67 state: up.enabled === 0 ? "not following" : behindWords(up.behind, up.error),
68 });
69 }
70 for (const s of db
71 .query<{ url: string; branch: string; enabled: number; checked_at: string | null; behind: number; error: string }, [string]>(
72 "SELECT url, branch, enabled, checked_at, behind, error FROM repo_source WHERE repo_id = ?1 ORDER BY created_at",
73 )
74 .all(repo.id)) {
75 inbound.push({
76 kind: "source", label: `${plainUrl(s.url)} (${s.branch})`, href: "", at: s.checked_at, problem: s.error !== "",
77 state: s.enabled === 0 ? "not following" : behindWords(s.behind, s.error),
78 });
79 }
80 for (const f of visibleForks(db, repo.id, viewer)) {
81 outbound.push({ kind: "fork", label: `${f.ownerName}/${f.name}`, href: base(f), state: "fork", at: null, problem: false });
82 }
83 const mirror = db
84 .query<{ full_name: string; pushed_at: string | null; last_error: string }, [string]>(
85 "SELECT full_name, pushed_at, last_error FROM migrate_mirror WHERE repo_id = ?1",
86 )
87 .get(repo.id);
88 if (mirror) {
89 outbound.push({
90 kind: "mirror", label: mirror.full_name, href: "", at: mirror.pushed_at, problem: mirror.last_error !== "",
91 state: mirror.last_error !== "" ? "last push failed" : "mirrored",
92 });
93 }
94 return { inbound, outbound };
95 }
src/http/server.ts +3
@@ -149,6 +149,7 @@ import { reconnectSubmit } from "../web/pages/frozen.ts";
149149 import { activityAtom, changelogAtom } from "../web/feeds.ts";
150150 import { auditCsv, auditPage, auditProbeSubmit } from "../web/pages/audit.ts";
151151 import { apiPage, appApiPage } from "../web/pages/api.ts";
152 import { flowsCheck, flowsPage } from "../web/pages/flows.ts";
152153 import { betAct, betPage, betPropose, betsPage } from "../web/pages/bets.ts";
153154 import { changelogJson, changelogPage } from "../web/pages/changelog.ts";
154155 import { releaseHoldSubmit, liftSubmit, releasesPage, rollbackSubmit } from "../web/pages/releases.ts";
@@ -1129,6 +1130,8 @@ export function buildRouter(): Router<Handler> {
11291130 router.add("GET", "/:owner/:name/week", weekPage);
11301131 router.add("POST", "/:owner/:name/week", weekSubmit);
11311132 router.add("POST", "/:owner/:name/issues/:number/first", firstSubmit);
1133 router.add("GET", "/:owner/:name/flows", flowsPage);
1134 router.add("POST", "/:owner/:name/flows/check", flowsCheck);
11321135 router.add("GET", "/:owner/:name/bets", betsPage);
11331136 router.add("POST", "/:owner/:name/bets/propose", betPropose);
11341137 router.add("GET", "/:owner/:name/bets/:id", betPage);
src/tui/forge.ts +7
@@ -80,6 +80,7 @@ import { stalledScreen } from "./screens/stalled.ts";
8080 import { issuesScreen, newIssueScreen } from "./screens/issues.ts";
8181 import { limbsScreen } from "./screens/limbs.ts";
8282 import { nextScreen } from "./screens/next.ts";
83 import { flowsScreen } from "./screens/flows.ts";
8384 import { weekScreen } from "./screens/week.ts";
8485 import { machinesScreen } from "./screens/machines.ts";
8586 import { newPullScreen } from "./screens/newpull.ts";
@@ -144,6 +145,12 @@ const REPO_VERBS: readonly {
144145 title: "The week in a repository",
145146 open: (f, ctx, repo) => weekScreen(f, ctx, repo),
146147 },
148 {
149 // What flows in and out of a repository, and the radar's key for its admin (#9434).
150 id: "flows",
151 title: "Flows of a repository",
152 open: (f, ctx, repo) => flowsScreen(f, ctx, repo),
153 },
147154 {
148155 // Settings → Machines' other door: the machines and last shell sessions, writers only (#9471).
149156 id: "machines",
src/tui/screens/flows.ts added +65
@@ -0,0 +1,65 @@
1 /**
2 * Flows in the terminal (#9434): the page's lines from the same reads —
3 * `readFlows` gates forks and the origin with `canRead`/`visibleForks`, and
4 * `startRadar` gates the `r` key with the page's admin check — so this door
5 * decides nothing of its own. Access is read at every draw.
6 */
7
8 import { canRead, type Repo } from "../../core/repos.ts";
9 import { readRadar, type DepLine } from "../../flows/radar.ts";
10 import { pinsOf } from "../../flows/run.ts";
11 import { mayStartRadar, startRadar } from "../../flows/start.ts";
12 import { readFlows, type FlowLine } from "../../flows/view.ts";
13 import type { Ctx, Screen } from "../app/app.ts";
14 import { binding } from "../app/keys.ts";
15 import type { Session } from "../session.ts";
16 import { ListState, drawList, runAsync, type ListRow } from "./common.ts";
17
18 const flowRow = (side: string) => (l: FlowLine): ListRow => ({
19 title: `${side} ${l.label}`,
20 secondary: l.state,
21 ...(l.problem ? { tone: "danger" as const } : {}),
22 meta: l.at === null ? "never moved" : l.at.slice(0, 16).replace("T", " "),
23 });
24
25 const depRow = (d: DepLine): ListRow => ({
26 title: `pkg ${d.name}`,
27 secondary: d.error !== "" ? "not read" : d.gap === "none" ? (d.checkedAt === null ? "not read yet" : "current") : `${d.gap} update`,
28 ...(d.error !== "" || d.gap === "major" ? { tone: "danger" as const } : {}),
29 meta: `${d.pinned === "" ? d.range : `pinned ${d.pinned}`}${d.latest === "" ? "" : ` · newest ${d.latest}`}`,
30 });
31
32 export function flowsScreen(f: Session, ctx: Ctx, repo: Repo): Screen {
33 const state = new ListState();
34 let deps: DepLine[] = [];
35 const visible = (): boolean => canRead(f.db, repo, f.user);
36 const rows = (): ListRow[] => {
37 if (!visible()) return [];
38 const view = readFlows(f.db, repo, f.user);
39 return [...view.inbound.map(flowRow("in ")), ...deps.map(depRow), ...view.outbound.map(flowRow("out"))];
40 };
41 const load = async (): Promise<void> => {
42 deps = visible() ? readRadar(f.db, repo.id, await pinsOf(f.app, repo)) : [];
43 ctx.invalidate();
44 };
45 return {
46 title: "Flows",
47 keys: [binding(["j", "k"], "move cursor"), ...(mayStartRadar(f.app, repo, f.user) ? [binding(["r"], "read upstream now")] : []), binding(["esc"], "back")],
48 load,
49 render(area, s) {
50 drawList(s, area, ctx.theme.c, state, rows());
51 },
52 key(e) {
53 if (e.key === "r") {
54 runAsync(ctx, async () => {
55 const started = await startRadar(f.app, repo, f.user);
56 if (started.ok) ctx.toast(`Read upstream for ${started.read} packages.`);
57 else ctx.toast(started.why, "danger");
58 await load();
59 });
60 return true;
61 }
62 return state.key(e, rows().length);
63 },
64 };
65 }
src/web/pages/flows.ts added +80
@@ -0,0 +1,80 @@
1 /**
2 * Flows (#9434): `/:owner/:name/flows` is one picture per repository — what
3 * flows in (origin, followed sources, pinned packages with their newest
4 * release) on one side and what flows out (forks, mirrors) on the other,
5 * each with its state and when it last moved. An admin can ask the radar to
6 * read upstream now. Server-rendered, no script; the read is from state the
7 * forge keeps, so it answers without the network or a model.
8 */
9
10 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
11 import { readRadar } from "../../flows/radar.ts";
12 import { pinsOf } from "../../flows/run.ts";
13 import { mayStartRadar, startRadar } from "../../flows/start.ts";
14 import { readFlows, type FlowLine } from "../../flows/view.ts";
15 import type { Ctx } from "../../http/context.ts";
16 import { crumbsFor, loadRepo, notFoundPage, pageHead, repoBase, repoHeader, repoPalette } from "../chrome.ts";
17 import { readForm } from "../form.ts";
18 import { html, type Html } from "../html.ts";
19 import { flashTo, page } from "../layout.ts";
20 import { badge, btn, card, listCard, row, time } from "../ui.ts";
21
22 const lineRow = (l: FlowLine): Html =>
23 row({
24 ...(l.href === "" ? {} : { href: l.href }),
25 title: html`${l.label}`,
26 meta: html`${l.at === null ? "never moved" : time(l.at)}`,
27 trail: badge(l.state, { tone: l.problem ? "danger" : "default" }),
28 });
29
30 function side(title: string, lines: readonly Html[], empty: string): Html {
31 return card(lines.length === 0 ? html`<p class="muted">${empty}</p>` : listCard([...lines]), { head: html`${title}`, pad: lines.length === 0 });
32 }
33
34 // GET /:owner/:name/flows
35 export async function flowsPage(ctx: Ctx): Promise<Response> {
36 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
37 const repo = loadRepo(ctx);
38 if (!repo) return notFoundPage(ctx);
39 const rendering = performance.now();
40 const flows = readFlows(ctx.app.db, repo, ctx.auth.user);
41 const deps = readRadar(ctx.app.db, repo.id, await pinsOf(ctx.app, repo));
42 const admin = mayStartRadar(ctx.app, repo, ctx.auth.user);
43 const depRows = deps.map((d) =>
44 row({
45 title: html`${d.name}`,
46 meta: html`${d.pinned === "" ? d.range : `pinned ${d.pinned}`}${d.latest === "" ? "" : ` · newest ${d.latest}`}${d.checkedAt === null ? "" : html` · read ${time(d.checkedAt)}`}`,
47 trail: badge(d.error !== "" ? "not read" : d.gap === "none" ? (d.checkedAt === null ? "not read yet" : "current") : `${d.gap} update`, {
48 tone: d.error !== "" || d.gap === "major" ? "danger" : d.gap === "none" ? "default" : "attn",
49 }),
50 }),
51 );
52 const body = html`${repoHeader(ctx, repo, "flows")}
53 ${pageHead("Flows", { sub: "What comes in on the left, what goes out on the right." })}
54 <div class="split">
55 <div>
56 ${side("Flows in", flows.inbound.map(lineRow), "Nothing flows in: no origin and no followed source.")}
57 ${side("Packages", depRows, "No pinned packages.")}
58 </div>
59 <div>${side("Flows out", flows.outbound.map(lineRow), "Nothing flows out: no forks and no mirror.")}</div>
60 </div>
61 ${admin
62 ? html`<form method="post" action="${repoBase(repo)}/flows/check" class="mt">${btn("Read upstream releases now", { variant: "secondary" })}</form>`
63 : null}`;
64 const response = page({ ctx, title: `Flows · ${repo.name}`, crumbs: crumbsFor(repo), palette: repoPalette(ctx, repo), body });
65 notePhase(ctx.request, "render", performance.now() - rendering);
66 return response;
67 }
68
69 // POST /:owner/:name/flows/check
70 export async function flowsCheck(ctx: Ctx): Promise<Response> {
71 const repo = loadRepo(ctx);
72 if (!repo) return notFoundPage(ctx);
73 const form = await readForm(ctx, "rw");
74 if (form instanceof Response) return form;
75 const back = `${repoBase(repo)}/flows`;
76 const started = await startRadar(ctx.app, repo, ctx.auth.user);
77 if (!started.ok) return flashTo(ctx, back, { tone: "danger", message: started.why });
78 const n = started.read;
79 return flashTo(ctx, back, { message: n === 0 ? "No exact pins to read." : `Read upstream for ${n} ${n === 1 ? "package" : "packages"}.` });
80 }
src/web/repo_steps.ts +3
@@ -27,6 +27,7 @@ export type RepoTab =
2727 | "app"
2828 | "runs"
2929 | "events"
30 | "flows"
3031 | "api"
3132 | "settings"
3233 /** A page that belongs to no tab (waves): nothing is marked current. */
@@ -114,6 +115,8 @@ function work(ctx: Ctx, base: string, repo: Repo | undefined): Steps {
114115 });
115116 }
116117 steps.push({ key: "events", label: "Events", href: `${base}/events` });
118 // What flows in and out: origin, sources, pins, forks (#9434). The page gates itself like the repo.
119 steps.push({ key: "flows", label: "Flows", href: `${base}/flows` });
117120 // An app that serves /openapi.json gets its API reference. The spec is
118121 // read in the background and remembered, so a header never waits.
119122 if (record !== undefined && canEnterApp(ctx.app.db, repo, record, ctx.auth.user)) {
test/flows-tui.test.ts added +89
@@ -0,0 +1,89 @@
1 /** Flows in the terminal (#9434): the page's gates, from the same reads. */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createRepo } from "../src/actions/repos.ts";
7 import { createApp, type App as Forge } from "../src/app.ts";
8 import type { Repo } from "../src/core/repos.ts";
9 import { createUser, type User } from "../src/core/users.ts";
10 import { createForgeApp } from "../src/tui/forge.ts";
11 import type { TuiIdentity } from "../src/tui/session.ts";
12 import { headlessTerminal } from "../src/tui/term/terminal.ts";
13 import { quietLogs } from "./helpers/globals.ts";
14 import { settleApp } from "./helpers/tui.ts";
15
16 quietLogs();
17
18 describe("flows in the terminal", () => {
19 let dir: string;
20 let forge: Forge;
21 let ada: User;
22 let eve: User;
23 let repo: Repo;
24
25 function boot(user: User | undefined) {
26 const terminal = headlessTerminal(120, 32);
27 const identity: TuiIdentity = { sshUsername: "t", remoteAddress: "127.0.0.1", user };
28 const { app } = createForgeApp(forge, identity, { terminal, exitHooks: false });
29 app.start();
30 const key = (k: string) => app.dispatch({ type: "key", key: k, ctrl: k.startsWith("ctrl+"), alt: false, shift: false });
31 const press = async (...keys: string[]) => {
32 for (const k of keys) key(k.startsWith("ctrl+") ? k.slice(5) : k);
33 await settleApp(app);
34 };
35 const open = async () => {
36 await settleApp(app);
37 for (const k of ["ctrl+k"]) app.dispatch({ type: "key", key: k.slice(5), ctrl: true, alt: false, shift: false });
38 await settleApp(app);
39 for (const ch of "Flows of a repository") key(ch);
40 await press("enter");
41 for (const ch of "ada/arm") key(ch);
42 await press("enter");
43 };
44 return { text: () => app.screen.toText(), press, open, stop: () => app.stop() };
45 }
46
47 beforeAll(async () => {
48 dir = await mkdtemp(join(tmpdir(), "forge-flows-tui-"));
49 forge = createApp({ dataDir: dir, port: 0 });
50 const make = async (username: string, adminIfFirst: boolean): Promise<User> => {
51 const made = await createUser(forge.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst });
52 if (!made.ok) throw new Error(made.error);
53 return made.value;
54 };
55 ada = await make("ada", true);
56 eve = await make("eve", false);
57 const made = await createRepo(forge, ada, { name: "arm", owner: "ada", visibility: "public", description: "" });
58 if (!made.ok) throw new Error(JSON.stringify(made.error));
59 repo = made.value;
60 forge.db.query("INSERT INTO repo_source (id, repo_id, url, branch, by_id, created_at) VALUES ('s1', ?1, 'https://u:[email protected]/x.git', 'main', ?2, '2026-11-01T00:00:00Z')").run(repo.id, ada.id);
61 }, 60_000);
62 afterAll(async () => {
63 forge?.close();
64 await rm(dir, { recursive: true, force: true });
65 });
66
67 test("an admin sees the lines without userinfo and the radar key; a reader does not get the key", async () => {
68 const admin = boot(ada);
69 await admin.open();
70 expect(admin.text()).toContain("example.com/x.git");
71 expect(admin.text()).not.toContain("tok3n");
72 admin.stop();
73 const reader = boot(eve);
74 await reader.open();
75 expect(reader.text()).toContain("example.com/x.git");
76 await reader.press("r");
77 expect(reader.text()).toContain("Only an admin of this repository can do this.");
78 reader.stop();
79 });
80
81 test("a private repository shows a guest nothing", async () => {
82 forge.db.query("UPDATE repo SET visibility = 'private' WHERE id = ?").run(repo.id);
83 const guest = boot(undefined);
84 await guest.open();
85 expect(guest.text()).not.toContain("example.com/x.git");
86 guest.stop();
87 forge.db.query("UPDATE repo SET visibility = 'public' WHERE id = ?").run(repo.id);
88 });
89 });
test/flows.test.ts added +182
@@ -0,0 +1,182 @@
1 /** Flows (#9434): versions, pins, changelog, the radar and the page — all with no crew and no model credential. */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { createHmac } from "node:crypto";
4 import { mkdtemp, rm } from "node:fs/promises";
5 import { tmpdir } from "node:os";
6 import { join } from "node:path";
7 import { createApp, type App } from "../src/app.ts";
8 import { getRepo } from "../src/core/repos.ts";
9 import { readPins } from "../src/flows/pins.ts";
10 import { readRadar, refreshRadar } from "../src/flows/radar.ts";
11 import { compareSemver, gapOf, parseSemver } from "../src/flows/semver.ts";
12 import { RateLimiter } from "../src/core/ratelimit.ts";
13 import { getUserById, type User } from "../src/core/users.ts";
14 import { startRadar } from "../src/flows/start.ts";
15 import { plainUrl, readFlows } from "../src/flows/view.ts";
16 import { serve, type Servers } from "../src/http/server.ts";
17 import { PREVIEW_ADMIN, PREVIEW_REPO } from "../src/spawn/preview-looks.ts";
18 import { seedPreview } from "../src/spawn/preview-seed.ts";
19
20 const v = (s: string) => parseSemver(s)!;
21
22 describe("semver", () => {
23 test("parses and rejects", () => {
24 expect(parseSemver("v1.2.3")).toEqual({ major: 1, minor: 2, patch: 3, pre: "" });
25 expect(parseSemver("1.2")).toBeUndefined();
26 expect(parseSemver("latest")).toBeUndefined();
27 });
28 test("orders, a release above its prerelease", () => {
29 expect(compareSemver(v("1.10.0"), v("1.9.9"))).toBeGreaterThan(0);
30 expect(compareSemver(v("2.0.0-rc.1"), v("2.0.0"))).toBeLessThan(0);
31 });
32 test("names the gap", () => {
33 expect(gapOf(v("1.2.3"), v("2.0.0"))).toBe("major");
34 expect(gapOf(v("1.2.3"), v("1.3.0"))).toBe("minor");
35 expect(gapOf(v("1.2.3"), v("1.2.4"))).toBe("patch");
36 expect(gapOf(v("1.2.3"), v("1.2.3"))).toBe("none");
37 expect(gapOf(v("1.2.3"), v("1.0.0"))).toBe("none");
38 });
39 });
40
41 describe("pins and changelog", () => {
42 test("reads exact floors, skips tags and bad names", () => {
43 const pins = readPins(JSON.stringify({ dependencies: { zed: "^1.2.3", "bad name": "1.0.0", tagged: "latest" }, devDependencies: { "@a/b": "~0.4.0" } }));
44 expect(pins.map((p) => [p.name, p.version, p.dev])).toEqual([["@a/b", "0.4.0", true], ["tagged", "", false], ["zed", "1.2.3", false]]);
45 expect(readPins("{nope")).toEqual([]);
46 });
47 });
48
49 let dir: string;
50 let app: App;
51 let servers: Servers;
52 const KEY = "flows-test-key";
53 beforeAll(async () => {
54 dir = await mkdtemp(join(tmpdir(), "forge-flows-"));
55 app = createApp({ dataDir: dir, port: 0, previewIdentityKey: KEY });
56 const seeded = await seedPreview(app, { FORGE_APP_ENV: "preview" });
57 if (!seeded.ok) throw new Error(seeded.error);
58 servers = await serve(app);
59 }, 30_000);
60 afterAll(async () => {
61 servers.stop();
62 app.close();
63 await rm(dir, { recursive: true, force: true });
64 });
65
66 describe("the radar and the view", () => {
67 test("stores what upstream says, outdated first, an unreachable one marked", async () => {
68 const repo = getRepo(app.db, PREVIEW_ADMIN, PREVIEW_REPO)!;
69 const pins = readPins(JSON.stringify({ dependencies: { a: "1.0.0", b: "2.0.0", c: "1.0.0" } }));
70 const n = await refreshRadar(app.db, repo.id, pins, async (name) =>
71 name === "a" ? { latest: "1.0.0" } : name === "b" ? { latest: "3.1.0" } : "registry not reachable");
72 expect(n).toBe(3);
73 const lines = readRadar(app.db, repo.id, pins);
74 expect(lines.map((l) => [l.name, l.gap, l.error])).toEqual([["b", "major", ""], ["a", "none", ""], ["c", "none", "registry not reachable"]]);
75 });
76 test("lists forks out and a followed source in", () => {
77 const repo = getRepo(app.db, PREVIEW_ADMIN, PREVIEW_REPO)!;
78 app.db.query("INSERT INTO repo_source (id, repo_id, url, branch, by_id, created_at, behind) VALUES ('s1', ?1, 'https://example.com/x.git', 'main', 'u1', '2026-11-01T00:00:00Z', 2)").run(repo.id);
79 const view = readFlows(app.db, repo, undefined);
80 expect(view.inbound.find((l) => l.kind === "source")?.state).toBe("2 changes behind");
81 });
82 test("a followed URL shows no userinfo, query or fragment", () => {
83 const repo = getRepo(app.db, PREVIEW_ADMIN, PREVIEW_REPO)!;
84 app.db.query("INSERT INTO repo_source (id, repo_id, url, branch, by_id, created_at) VALUES ('s2', ?1, 'https://bob:[email protected]/y.git?token=zz#f', 'main', 'u1', '2026-11-02T00:00:00Z')").run(repo.id);
85 app.db.query("INSERT OR REPLACE INTO repo_upstream (repo_id, url, branch, by_id, updated_at) VALUES (?1, 'https://al:[email protected]/z.git', 'main', 'u1', '2026-11-02T00:00:00Z')").run(repo.id);
86 const text = JSON.stringify(readFlows(app.db, repo, undefined));
87 expect(text).toContain("example.org/y.git");
88 expect(text).toContain("example.net/z.git");
89 for (const secret of ["sekret", "bob", "pw@", "token=zz"]) expect(text).not.toContain(secret);
90 expect(plainUrl("[email protected]:me/r.git")).toBe("host.example:me/r.git");
91 });
92 });
93
94 describe("starting the radar", () => {
95 const repoNow = () => getRepo(app.db, PREVIEW_ADMIN, PREVIEW_REPO)!;
96 const userNamed = (name: string) => app.db.query<{ id: string }, [string]>("SELECT id FROM user WHERE username = ?").get(name);
97 const fresh = () => new RateLimiter({ perMinute: 2, adminPerMinute: 2, enabled: true });
98
99 test("refuses a non-admin and an agent, and writes nothing", async () => {
100 const repo = repoNow();
101 const before = app.db.query("SELECT COUNT(*) AS n FROM dep_latest WHERE repo_id = ?").get(repo.id);
102 const lookup = async () => ({ latest: "9.9.9" });
103 const stranger = { ...getUserById(app.db, userNamed(PREVIEW_ADMIN)!.id)!, id: "nobody", isAdmin: false } as User;
104 const agent = { ...getUserById(app.db, userNamed(PREVIEW_ADMIN)!.id)!, kind: "agent" } as User;
105 for (const who of [undefined, stranger, agent]) {
106 const r = await startRadar(app, repo, who, lookup, fresh());
107 expect(r).toEqual({ ok: false, why: "Only an admin of this repository can do this." });
108 }
109 expect(app.db.query("SELECT COUNT(*) AS n FROM dep_latest WHERE repo_id = ?").get(repo.id)).toEqual(before);
110 });
111
112 test("an admin's run stores the answers; a second at once is refused; the rate limit holds", async () => {
113 const repo = repoNow();
114 const admin = getUserById(app.db, userNamed(PREVIEW_ADMIN)!.id)!;
115 // pinsOf reads the repository; stand in for it with the pins the run stores through refreshRadar.
116 let release!: () => void;
117 const gate = new Promise<void>((r) => (release = r));
118 const slow = async () => { await gate; return { latest: "1.1.0" }; };
119 const limiter = fresh();
120 const first = startRadar(app, repo, admin, slow, limiter);
121 await Promise.resolve();
122 const second = await startRadar(app, repo, admin, slow, limiter);
123 release();
124 expect((await first).ok).toBe(true);
125 expect(second).toEqual({ ok: false, why: "A read is already running for this repository." });
126 const limited = fresh();
127 await startRadar(app, repo, admin, slow, limited);
128 await startRadar(app, repo, admin, slow, limited);
129 const third = await startRadar(app, repo, admin, slow, limited);
130 expect(third.ok).toBe(false);
131 expect(!third.ok && third.why).toMatch(/^Too many reads/);
132 });
133
134 test("a run is bounded: limited concurrency and a total deadline", async () => {
135 const repo = repoNow();
136 const pins = readPins(JSON.stringify({ dependencies: Object.fromEntries(Array.from({ length: 20 }, (_, i) => [`p${i}`, "1.0.0"])) }));
137 let live = 0;
138 let peak = 0;
139 const lookup = async () => { live++; peak = Math.max(peak, live); await Bun.sleep(5); live--; return { latest: "1.0.1" }; };
140 expect(await refreshRadar(app.db, repo.id, pins, lookup)).toBe(20);
141 expect(peak).toBeLessThanOrEqual(4);
142 const slowOnes = async () => { await Bun.sleep(30); return { latest: "1.0.1" }; };
143 const done = await refreshRadar(app.db, repo.id, pins, slowOnes, new Date(), 50);
144 expect(done).toBeLessThan(20);
145 });
146 });
147
148 describe("the page", () => {
149 const sign = (id: string, name: string, role: string) => {
150 const said = `${id}|${name}|${role}|${Date.now() + 60_000}`;
151 return { "x-forge-identity": said, "x-forge-signature": createHmac("sha256", KEY).update(said).digest("base64url") };
152 };
153 const url = () => `${servers.url}/${PREVIEW_ADMIN}/${PREVIEW_REPO}/flows`;
154
155 test("answers for an admin with the action and a tab in the header; a cross-origin POST is refused and changes nothing", async () => {
156 const headers = sign("u1", "trevato", "admin");
157 const res = await fetch(url(), { headers });
158 expect(res.status).toBe(200);
159 const text = await res.text();
160 expect(text).toContain("Flows in");
161 expect(text).toContain("Flows out");
162 expect(text).toContain("Read upstream releases now");
163 expect(text).toContain(`href="/${PREVIEW_ADMIN}/${PREVIEW_REPO}/flows"`);
164 const repo = getRepo(app.db, PREVIEW_ADMIN, PREVIEW_REPO)!;
165 const before = app.db.query("SELECT * FROM dep_latest WHERE repo_id = ? ORDER BY name").all(repo.id);
166 const post = await fetch(`${url()}/check`, { method: "POST", redirect: "manual", headers: { ...headers, origin: "https://evil.example" } });
167 expect(post.status).toBe(403);
168 expect(app.db.query("SELECT * FROM dep_latest WHERE repo_id = ? ORDER BY name").all(repo.id)).toEqual(before);
169 });
170
171 test("a private repository is hidden from a guest", async () => {
172 const repo = getRepo(app.db, PREVIEW_ADMIN, PREVIEW_REPO)!;
173 app.db.query("UPDATE repo SET visibility = 'private' WHERE id = ?").run(repo.id);
174 try {
175 const guest = await fetch(url());
176 expect(guest.status).toBe(404);
177 expect(await guest.text()).not.toContain("Flows in");
178 } finally {
179 app.db.query("UPDATE repo SET visibility = 'public' WHERE id = ?").run(repo.id);
180 }
181 });
182 });
test/palette-roster.test.ts +4
@@ -57,6 +57,7 @@ test("a site-admin person is offered the whole roster", () => {
5757 ["milestones", "Milestones in a repository", "owner/name"],
5858 ["next-up", "Next up in a repository", "owner/name"],
5959 ["week", "The week in a repository", "owner/name"],
60 ["flows", "Flows of a repository", "owner/name"],
6061 ["machines", "Machines of a repository", "owner/name"],
6162 ["repo-usage", "Usage of a repository", "owner/name"],
6263 ["new-issue", "New issue", "owner/name"],
@@ -158,6 +159,7 @@ test("a site-admin agent is offered the roster without the person-only entries",
158159 ["milestones", "Milestones in a repository", "owner/name"],
159160 ["next-up", "Next up in a repository", "owner/name"],
160161 ["week", "The week in a repository", "owner/name"],
162 ["flows", "Flows of a repository", "owner/name"],
161163 ["machines", "Machines of a repository", "owner/name"],
162164 ["repo-usage", "Usage of a repository", "owner/name"],
163165 ["new-issue", "New issue", "owner/name"],
@@ -223,6 +225,7 @@ test("a member person is offered the signed-in roster without the admin entries"
223225 ["milestones", "Milestones in a repository", "owner/name"],
224226 ["next-up", "Next up in a repository", "owner/name"],
225227 ["week", "The week in a repository", "owner/name"],
228 ["flows", "Flows of a repository", "owner/name"],
226229 ["machines", "Machines of a repository", "owner/name"],
227230 ["repo-usage", "Usage of a repository", "owner/name"],
228231 ["new-issue", "New issue", "owner/name"],
@@ -283,6 +286,7 @@ test("signed out, the palette offers sign-in and the public entries", () => {
283286 ["milestones", "Milestones in a repository", "owner/name"],
284287 ["next-up", "Next up in a repository", "owner/name"],
285288 ["week", "The week in a repository", "owner/name"],
289 ["flows", "Flows of a repository", "owner/name"],
286290 ["machines", "Machines of a repository", "owner/name"],
287291 ["repo-usage", "Usage of a repository", "owner/name"],
288292 ["new-issue", "New issue", "owner/name"],

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs