Public: anyone can read this; the crew can change it.

The repository map's API names what holds each ready issue: hot_file with the file, run_cap, open_pull and the rest #6862

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/6441-the-repository-map-s-api-names-what-hold → main 3 commits fork 7 threads · 7 unresolved

  • Merged. scout/6441-the-repository-map-s-api-names-what-hold was merged into main as da020fc.

Review

Sign in to review this change.
8 files changed +269 −14
docs/features/map-held.md added +5
@@ -0,0 +1,5 @@
1 # What holds a ready issue
2
3 `GET /api/v1/repos/:owner/:name/map` and the MCP `map` tool give each `ready` entry `held`: `null`, or `{ by, detail }` with `by` one of the names in `src/crew/held.ts` (`open_pull`, `hot_file`, `run_cap`, …). For `hot_file`, `detail` names the file and its count of open pulls. `startable` lists the ready numbers nothing holds.
4
5 Holds are worked out for the viewer: a pull they may not see (a sealed one) neither counts toward a hot file nor holds an issue as `open_pull`. So a guest's `startable` can list an issue the crew will not start; read it as what the viewer can see is free, not as the crew's queue. The direction issue is never in `startable` but shows `held: null`. Hot-file counts come from memory, so a cached map can lag them by up to a minute.
src/crew/held.ts +79 −3
@@ -30,11 +30,12 @@
3030
3131 import type { App } from "../app.ts";
3232 import { listCrewRoles, type Agent, type CrewRole } from "../core/agents.ts";
33 import { readyNow, repoGraph, type Ranked } from "../core/dependencies.ts";
33 import { readyNow, repoGraph, type Graph, type Ranked } from "../core/dependencies.ts";
3434 import { alreadyDone, BY_CREW, DONE_LIKE } from "../core/done_on_main.ts";
3535 import type { Issue } from "../core/issues.ts";
3636 import { withPlanner } from "../core/limb_replan.ts";
3737 import type { Repo } from "../core/repos.ts";
38 import { sealedRowByNumber } from "../core/sealed.ts";
3839 import { listActive, type Run } from "../core/runs.ts";
3940 import { getUserById } from "../core/users.ts";
4041 import type { Stuck } from "./brief.ts";
@@ -120,15 +121,17 @@ export function heldBy(
120121 ready: readonly Ranked[],
121122 crew: readonly { agent: Agent; role: CrewRole }[],
122123 active: readonly Run[],
124 admit: (pullNumber: number) => boolean = () => true,
123125 ): Held {
124126 const busyIssues = new Set(active.map((r) => r.issueId));
125 const proposed = proposedWork(app.db, repo);
127 // Only the pulls the viewer may see count: a sealed pull holds nothing for a guest.
128 const proposed = new Map([...proposedWork(app.db, repo)].filter(([, pull]) => admit(pull.number)));

New behaviour with no test. Add a sealed crew pull that closes a public ready issue. A guest should get held ≠ open_pull and an admin should get open_pull. As it stands, reverting this line breaks no test.

126129 // An issue a planner or a reviewer holds is theirs, not a builder's.
127130 const planners = new Set(
128131 crew.filter((c) => c.role !== "builder").map((c) => c.agent.user.id),
129132 );
130133 const handed = withPlanner(app.db, repo);
131 const hot = hotFiles(repo.id);
134 const hot = hotFiles(repo.id, admit);
132135 const pins = new Map(heldFiles(app.db, repo.id).map((h) => [h.path, h]));
133136 const pinned = new Map<number, FileHold>();
134137 // The hold an admin set on a file the issue names, if any: the first by hold order.
@@ -211,3 +214,76 @@ export function stuckOf(held: Held): Stuck | undefined {
211214 );
212215 return issues.length === 0 ? undefined : { held: heldWords(held), issues };
213216 }
217
218 /** What holds a ready issue, as the map's API says it. */
219 export interface MapHold {
220 readonly by: HoldName;
221 readonly detail: string;
222 }
223
outdated

open_pull comes from proposedWork, which doesn't filter on sealed. A public ready issue that a sealed pull closes shows as held to a guest. Filter by the viewer's scope.

224 const HOLD_WORDS: Readonly<Record<HoldName, string>> = {
225 running: "a run is in flight on it",
226 open_pull: "an open pull already proposes it",
227 planner: "the planner or a reviewer holds it",
228 with_planner: "its limb waits on the planner",
229 done_on_main: "found done on the base branch; waits for a person to close it",
230 stalled: "a builder ran on it and left without a pull",
231 run_cap: "it has used its runs for the day",
232 parked_twice: "parked twice; it is the planner's",
233 red_fix: "a fix for a red main covers it",
234 hot_file: "a file it names is hot",
235 };
236
237 /**
238 * The hold on each held ready issue, by number, and the numbers none holds.
239 * `hot_file` names the file and its open pulls. Memory and the database only:
240 * no git call. `issues` is the viewer's own ready list, so a sealed issue the
241 * viewer may not see is never named.
242 */
243 export function mapHolds(
244 repo: Repo,
outdated

High: hotFiles counts sealed pulls too (learnPullFiles reads every open pull). This count goes out to every reader of the map, guests included, so it reveals that a hidden pull touches this path. Count only the pulls the viewer's sealed scope admits.

245 held: Held,
246 admit: (pullNumber: number) => boolean = () => true,
247 ): { holds: Map<number, MapHold>; startable: number[] } {
248 const hot = hotFiles(repo.id, admit);
249 const holds = new Map<number, MapHold>();
250 for (const name of HOLDS) {
251 for (const issue of held.by[name]) {
252 let detail = HOLD_WORDS[name];
253 if (name === "hot_file") {
254 const pin = held.pinned.get(issue.number);
outdated

This count is the leak in practice: an anonymous N open pulls that is larger than the open pulls the reader can see.

255 const top = hotOverlap(issue, hot)[0];

No test covers the admin-pin branch. Seed one file hold on a file a ready issue names, and assert that the detail is <path> is held by an admin.

256 if (pin !== undefined) detail = `${pin.path} is held by an admin`;
257 else if (top !== undefined)
258 detail = `${top.path} is in ${top.pulls} open ${top.pulls === 1 ? "pull" : "pulls"}`;
259 }
260 holds.set(issue.number, { by: name, detail });
261 }
262 }
263 return { holds, startable: held.startable.map((r) => r.issue.number) };
264 }
265
266 /** The pulls a viewer of `scope` may see: all with the sealed scope, else the unsealed and their own. */
267 function admits(app: App, repo: Repo, scope: { sealed?: true; sealedBy?: string }) {
268 if (scope.sealed === true) return () => true;
269 return (n: number): boolean => {
270 const row = sealedRowByNumber(app.db, repo.id, n);
271 return row === undefined || !row.sealed || row.authorId === scope.sealedBy;
272 };
273 }
274
275 /** The map's holds for one viewer: shared by the route and the MCP `map` tool. */
276 export function mapHeldFor(
277 app: App,
278 repo: Repo,
279 graph: Graph,
280 scope: { sealed?: true; sealedBy?: string },
281 ): ReturnType<typeof mapHolds> {
282 const admit = admits(app, repo, scope);
283 const active = listActive(app.db, "agent").filter((r) => r.repoId === repo.id);
284 return mapHolds(
285 repo,
286 heldBy(app, repo, readyNow(graph), listCrewRoles(app.db, repo.id), active, admit),
287 admit,
288 );
289 }
src/crew/hot_files.ts +11 −5
@@ -108,18 +108,24 @@ export async function learnPullFiles(app: App, repo: Repo, now: number = Date.no
108108 }
109109
110110 /** Every file of `repoId` with how many open pulls touch it, narrow pulls only. */
111 export function pullCounts(repoId: string): Map<string, number> {
111 export function pullCounts(
112 repoId: string,
113 admit: (pullNumber: number) => boolean = () => true,
114 ): Map<string, number> {
112115 const counts = new Map<string, number>();
113 for (const { files } of touched.get(repoId)?.values() ?? []) {
114 if (files.length > WIDE_PULL) continue;
116 for (const { files, number } of touched.get(repoId)?.values() ?? []) {
117 if (files.length > WIDE_PULL || !admit(number)) continue;
115118 for (const file of files) counts.set(file, (counts.get(file) ?? 0) + 1);
116119 }
117120 return counts;
118121 }
119122
120123 /** Every file of `repo` that `HOT_PULLS` or more open pulls touch, with how many. */
121 export function hotFiles(repoId: string): Map<string, number> {
122 const counts = pullCounts(repoId);
124 export function hotFiles(
125 repoId: string,
126 admit?: (pullNumber: number) => boolean,
127 ): Map<string, number> {
128 const counts = pullCounts(repoId, admit);
123129 for (const [file, n] of counts) if (n < HOT_PULLS) counts.delete(file);
124130 return counts;
125131 }
src/http/api.ts +4 −2
@@ -350,6 +350,7 @@ import {
350350 } from "../actions/dependencies.ts";
351351 import { openPlan, parsePlan, proposePlan } from "../actions/plan.ts";
352352 import { dependenciesOf, repoGraph } from "../core/dependencies.ts";
353 import { mapHeldFor } from "../crew/held.ts";
353354 import { pillarView } from "../core/pillars.ts";
354355 import { askToLiveFor } from "../web/ask_to_live.ts";
355356 import { decisionStats, listDecisions } from "../core/decisions.ts";
@@ -3858,12 +3859,13 @@ export function apiRoutes(router: Router<Handler>): void {
38583859 const ask = timePhase(request, "asktolive", () =>
38593860 askToLiveFor(ctx.app, repo, pillars, ctx.auth.user),
38603861 );
3861 return timePhase(request, "render", () => mapJson(repo, graph, marks, pillars, ask));
3862 const held = timePhase(request, "held", () => mapHeldFor(ctx.app, repo, graph, scope));
outdated

Pass scope through here so the holds are computed for this viewer. Then put this block in a helper the MCP map tool can call as well (src/http/mcp/tools/map.ts still calls mapJson without held).

3863 return timePhase(request, "render", () => mapJson(repo, graph, marks, pillars, ask, held));
38623864 },
38633865 );
38643866 // A snapshot skips the phases; they are still named, as hits at no cost.
38653867 if (!built)
3866 for (const name of ["graph", "marks", "pillars", "asktolive", "render"])
3868 for (const name of ["graph", "marks", "pillars", "asktolive", "held", "render"])
38673869 notePhase(request, name, 0, hit);
38683870 return json(value);
38693871 });
src/http/mcp/tools/map.ts +4 −3
@@ -1,5 +1,6 @@
11 import { sealedScope } from "../../../core/sealed.ts";
22 import { repoGraph } from "../../../core/dependencies.ts";
3 import { mapHeldFor } from "../../../crew/held.ts";
34 import { pillarView } from "../../../core/pillars.ts";
45 import { askToLiveFor } from "../../../web/ask_to_live.ts";
56 import { listActiveRuns } from "../../../core/runs.ts";
@@ -20,9 +21,8 @@ export const tool: Tool = {
2021 async run(ctx, user, args) {
2122 const repo = planRepo(ctx, user, args);
2223 if (isAnswer(repo)) return repo;
23 const graph = repoGraph(ctx.app.db, repo.id, {
24 ...sealedScope(ctx.app.db, repo, user),
25 });
24 const scope = sealedScope(ctx.app.db, repo, user);
25 const graph = repoGraph(ctx.app.db, repo.id, { ...scope });
2626 const pillars = pillarView(ctx.app.db, repo);
2727 return good(
2828 mapJson(
@@ -36,6 +36,7 @@ export const tool: Tool = {
3636 ),
3737 pillars,
3838 askToLiveFor(ctx.app, repo, pillars, user),
39 mapHeldFor(ctx.app, repo, graph, scope),
3940 ),
4041 );
4142 },
src/web/pages/map.ts +5
@@ -19,6 +19,7 @@
1919 * can see, one lane each.
2020 */
2121
22 import type { MapHold } from "../../crew/held.ts";
2223 import { sealedScope } from "../../core/sealed.ts";
2324 import type { Database } from "bun:sqlite";
2425 import {
@@ -1482,6 +1483,7 @@ export function mapJson(
14821483 marks: Marks = NO_MARKS,
14831484 pillars?: PillarView,
14841485 ask: readonly AskToLiveCount[] = [],
1486 held?: { holds: ReadonlyMap<number, MapHold>; startable: readonly number[] },
14851487 ): Record<string, unknown> {
14861488 const ready = readyNow(graph);
14871489 const readySet = new Set(ready.map((r) => r.issue.id));
@@ -1508,7 +1510,10 @@ export function mapJson(
15081510 ready: ready.map((r) => ({
15091511 ...brief(r.issue),
15101512 holdsUp: r.holdsUp,
1513 ...(held === undefined ? {} : { held: held.holds.get(r.issue.number) ?? null }),
15111514 })),
1515 // The ready numbers no filter holds: what a builder may start now.
1516 ...(held === undefined ? {} : { startable: held.startable }),
15121517 blockedOn: blockedOn(graph).map((r) => ({
15131518 ...brief(r.issue),
15141519 holdsUp: r.holdsUp,
test/map-api-timing.test.ts +1 −1
@@ -52,6 +52,6 @@ test("the repository map names its phases in order, with durations", async () =>
5252 expect(body.ready.length).toBeGreaterThan(0);
5353 const parts = (res.headers.get("server-timing") ?? "").split(", ");
5454 const names = parts.map((p) => p.split(";")[0]!);
55 expect(names.filter((n) => n !== "app")).toEqual(["session", "graph", "marks", "pillars", "asktolive", "render"]);
55 expect(names.filter((n) => n !== "app")).toEqual(["session", "graph", "marks", "pillars", "asktolive", "held", "render"]);
5656 for (const p of parts) expect(Number(/dur=([\d.]+)/.exec(p)?.[1])).not.toBeNaN();
5757 });
test/map-held.test.ts added +160
@@ -0,0 +1,160 @@
1 /** GET …/map names what holds each ready issue (#6441): `held`, and `startable` for the rest. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
8 import { openPull } from "../src/actions/pulls.ts";
9 import { holdFile } from "../src/crew/file_holds.ts";
10 import { getUserByName } from "../src/core/users.ts";
11 import { createIssue } from "../src/core/issues.ts";
12 import { HOT_PULLS, rememberPullFiles } from "../src/crew/hot_files.ts";
13 import { serve, type Servers } from "../src/http/server.ts";
14 import { seedAdmin } from "./helpers/admin.ts";
15
16 let dataDir = "";
17 let app: App, servers: Servers, repoId = "", bearer = "";
18 const nums: Record<string, number> = {};
19
20 beforeAll(async () => {
21 dataDir = await mkdtemp(join(tmpdir(), "forge-mapheld-"));
22 app = createApp({ dataDir, port: 0 });
23 servers = await serve(app);
24 const admin = await seedAdmin(app, { username: "alice", email: "[email protected]", password: "correct-horse-battery" });
25 if (!admin.ok) throw new Error(admin.error);
26 const made = await createRepo(app, admin.value, { name: "notes", visibility: "public", description: "" });
27 if (!made.ok) throw new Error(String(made.error));
28 repoId = made.value.id;
29 rememberPullFiles(
30 repoId,
31 Array.from({ length: HOT_PULLS }, (_, i) => ({ number: i + 1, files: ["src/hot.ts", `src/own${i}.ts`] })),
32 );
33 const add = (key: string, title: string, body = "", sealed = false) => {
34 const r = createIssue(app.db, { repoId, authorId: admin.value.id, title, body, sealed });
35 if (!r.ok) throw new Error(r.error);
36 nums[key] = r.value.number;
37 };
38 add("hot", "Add a flag to src/hot.ts");
39 add("free", "Tidy src/cold.ts");
40 add("secret", "Leak in src/hot.ts", "", true);
41 // A sealed pull touching the same file: only a viewer who may see it counts it.
42 rememberPullFiles(repoId, [
43 ...Array.from({ length: HOT_PULLS }, (_, i) => ({ number: 100 + i, files: ["src/hot.ts"] })),
44 { number: nums.secret!, files: ["src/hot.ts"] },
45 ]);
46 const minted = await fetch(`${servers.url}/api/v1/tokens`, {
47 method: "POST",
48 headers: { "content-type": "application/json" },
49 body: JSON.stringify({ adminConfirm: true, username: "alice", password: "correct-horse-battery", name: "cli", scope: "admin" }),
50 });
51 bearer = ((await minted.json()) as { token: string }).token;
52 });
53
54 afterAll(async () => {
55 rememberPullFiles(repoId, []);
56 servers.stop?.();
57 app.close();
58 await rm(dataDir, { recursive: true, force: true });
59 });
60
61 test("a ready issue holding a hot file says so, and startable lists only the free one", async () => {
outdated

This passes on main as well: a sealed issue was never in an anonymous ready list. Seed a sealed pull touching src/hot.ts and assert that the guest's count leaves it out.

62 const res = await fetch(`${servers.url}/api/v1/repos/alice/notes/map`);
63 const body = (await res.json()) as {
64 ready: { number: number; held: { by: string; detail: string } | null }[];
65 startable: number[];
66 };
67 const of = (n: number) => body.ready.find((r) => r.number === n)!;
68 expect(of(nums.hot!).held?.by).toBe("hot_file");
69 expect(of(nums.hot!).held?.detail).toContain("src/hot.ts");
70 expect(of(nums.hot!).held?.detail).toContain(`${HOT_PULLS} open pulls`);
71 expect(of(nums.free!).held).toBeNull();
72 expect(body.startable).toEqual([nums.free!]);
73 });
74
75 test("an anonymous reader is told nothing of a sealed issue", async () => {
76 const res = await fetch(`${servers.url}/api/v1/repos/alice/notes/map`);
77 const text = await res.text();
78 expect(text).not.toContain("Leak in");
79 expect(JSON.parse(text).ready.some((r: { number: number }) => r.number === nums.secret)).toBe(false);
80 expect(JSON.parse(text).startable).not.toContain(nums.secret);
81 });
82
83 type Ready = { ready: { number: number; held: { by: string; detail: string } | null }[] };
84
85 test("a sealed pull is not counted for a guest, and is for a viewer who may see it", async () => {
86 const guest = (await (await fetch(`${servers.url}/api/v1/repos/alice/notes/map`)).json()) as Ready;
87 const admin = (await (
88 await fetch(`${servers.url}/api/v1/repos/alice/notes/map`, { headers: { authorization: `Bearer ${bearer}` } })
89 ).json()) as Ready;
90 const detail = (m: Ready) => m.ready.find((r) => r.number === nums.hot!)?.held?.detail;
91 expect(detail(guest)).toBe(`src/hot.ts is in ${HOT_PULLS} open pulls`);
92 expect(detail(admin)).toBe(`src/hot.ts is in ${HOT_PULLS + 1} open pulls`);
93 });
94
95 test("the MCP map tool carries held and startable too", async () => {
96 const res = await fetch(`${servers.url}/mcp`, {
97 method: "POST",
98 headers: {
99 "content-type": "application/json",
100 accept: "application/json, text/event-stream",
101 authorization: `Bearer ${bearer}`,
102 },
103 body: JSON.stringify({
104 jsonrpc: "2.0",
105 id: 1,
106 method: "tools/call",
107 params: { name: "map", arguments: { app: "alice/notes" } },
108 }),
109 });
110 const body: any = await res.json();
111 const map = JSON.parse(body.result.content[0].text) as Ready & { startable: number[] };
112 expect(map.ready.find((r) => r.number === nums.hot!)?.held?.by).toBe("hot_file");
113 expect(map.startable).toEqual([nums.free!]);
114 });
115
116 test("an open pull a guest cannot see holds nothing for the guest; an admin's map says open_pull", async () => {
117 const alice = getUserByName(app.db, "alice")!;
118 const made = await createRepo(app, alice, { name: "pulls", visibility: "public", description: "" });
119 if (!made.ok) throw new Error(String(made.error));
120 const repo = made.value;
121 const mk = (title: string) => {
122 const r = createIssue(app.db, { repoId: repo.id, authorId: alice.id, title, body: "" });
123 if (!r.ok) throw new Error(r.error);
124 return r.value;
125 };
126 const closed = mk("Proposed by a sealed pull");
127 const pinned = mk("Edit src/pinned.ts");
128 const seeded = await commitToBranch(app, alice, repo, {
129 message: "Start",
130 files: [{ path: "README.md", content: "# pulls\n" }],
131 });
132 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
133 const branch = await commitToBranch(app, alice, repo, {
134 branch: "feature/sealed",
135 base: "main",
136 message: "Fix",
137 files: [{ path: "fix.ts", content: "export const n = 1;\n" }],
138 });
139 if (!branch.ok) throw new Error(JSON.stringify(branch.error));
140 const pull = await openPull(app, alice, repo, {
141 title: "Fix",
142 body: `Closes #${closed.number}`,
143 baseBranch: "main",
144 headBranch: "feature/sealed",
145 });
146 if (!pull.ok) throw new Error(String(pull.error));
147 // The pull is an issue row: seal it the way a security fix is.
148 app.db.query("UPDATE issue SET sealed = 1 WHERE repo_id = ?1 AND number = ?2").run(repo.id, pull.value.number);
149 expect(holdFile(app.db, repo.id, "src/pinned.ts", alice.id).ok).toBe(true);
150
151 const read = async (headers: Record<string, string>) =>
152 (await (await fetch(`${servers.url}/api/v1/repos/alice/pulls/map`, { headers })).json()) as Ready;
153 const guest = await read({});
154 const admin = await read({ authorization: `Bearer ${bearer}` });
155 const held = (m: Ready, n: number) => m.ready.find((r) => r.number === n)?.held;
156 expect(held(guest, closed.number)).toBeNull();
157 expect(held(admin, closed.number)?.by).toBe("open_pull");
158 expect(held(guest, pinned.number)?.detail).toBe("src/pinned.ts is held by an admin");
159 expect(held(admin, pinned.number)?.by).toBe("hot_file");
160 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs