Public: anyone can read this; the crew can change it.

Fewer wire hops: a crew run does its work locally, and every hop it still makes is counted #14651

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/14650-fewer-wire-hops-a-crew-run-does-its-work → main 4 commits fork

  • Merged. scout/14650-fewer-wire-hops-a-crew-run-does-its-work was merged into main as 29659c0.

Review

Sign in to review this change.
6 files changed +236 −1
docs/HOPS.md added +65
@@ -0,0 +1,65 @@
1 # A crew run's wire hops
2
3 The owner's direction (2026-11-20, #14650): do the work on this machine, and let
4 only the model call cross the uplink. The outage of #14614 came from one hop
5 that did not need to be one.
6
7 Every run's log carries one line just before its `timing:` line:
8
9 hops: 4, off-host: 1 — clone loopback · model internet · agent loopback · push loopback
10
11 `src/crew/hops.ts` tallies them and names where each goes from the URL's host
12 alone (no DNS lookup): `in-process` (a unix socket), `loopback`, `lan` (private
13 ranges, cluster names and any name with no dot) or `internet`. `lan` and
14 `internet` are off the host. A dotless name counts as `lan`, so a public alias
15 with no dot in it would be miscounted.
16
17 Each hop is counted once its step has succeeded: the clone or fetch after
18 `prepareClone` returns, the push after git accepts it, and the model and agent
19 hops once the session has taken a turn. A run that fails in `prepareClone` logs
20 no clone. The model and agent hops are counted per session, not per request:
21 the agent's own traffic is not observed, only the address it was given.
22
23 When a hop other than the model call leaves the host, a second line names it:
24
25 stray hops: agent internet
26
27 ## The map
28
29 | Hop | Made by | Goes to | Per run |
30 | ------------------------------------ | -------------- | ---------------------------------------------- | --------------- |
31 | mirror fetch and clone (`clone`) | the forge | `localUrl`, loopback; objects borrowed from the local mirror (`crew/mirror.ts`) | one, on a fresh clone |
32 | fetch of a kept clone (`fetch`) | the forge | `localUrl`, loopback | one, on a resumed run |
33 | model call (`model`) | the agent | the account's gateway, or api.anthropic.com | one session (many requests) |
34 | the agent's git and API (`agent`) | the agent | `FORGE_URL`, through the sandbox's proxy | one session (many requests) |
35 | push for the agent (`push`) | the forge | `localUrl`, loopback | only when the agent left commits unpushed |
36 | check run, preview deploy | the forge | in-process, after the push lands | not the run's own: counted by the check and the deploy |
37
38 ## Before and after
39
40 Before (#14614): the forge's own mirror fetches were paced like uplink traffic
41 and timed out. They already went to `localUrl`. #14605 keeps the mirror when a
42 fetch fails, and the clone borrows its objects (`crew/mirror.ts`). Nobody
43 measured hop counts before this change, so there is no "before" column. The
44 counts below are what the code expects a run to log, not measured values:
45
46 | Run | hops (expected) | off-host (expected) |
47 | ------------------------- | --------------- | ------------------------ |
48 | fresh clone, forge pushes | 4 | 1 (model) when `FORGE_URL` is loopback; 2 (model, agent) when it is the forge's public name |
49 | fresh clone, agent pushes | 3 | the same |
50 | resumed clone | 3 or 4 | the same |
51
52 In production `FORGE_URL` is the forge's public name, so every real run today
53 logs `off-host: 2` and a `stray hops: agent internet` line. The agent's
54 remotes and `FORGE_URL` carry that name because inside the shell sandbox
55 `localhost` is the sandbox's own (`prepareClone` says why). The next step is to
56 route that name to the loopback inside the sandbox's proxy. Until then the line
57 counts the hop rather than hiding it.
58
59 ## The guard
60
61 `test/crew-hops.test.ts` holds the reach of each kind of host and the
62 `stray hops:` line. `test/crew-run-timing.test.ts` runs a whole stand-in run
63 and fails when its log has a `stray hops:` line. The stand-in's `FORGE_URL` is
64 loopback, so this guards the stand-in only. It cannot fail for the production
65 agent hop above, which the stray line reports instead.
docs/REFERENCE.md +1
@@ -543,6 +543,7 @@ This page is the front door. Everything else is a file of its own under
543543 - [`docs/CONTRACT.md`](CONTRACT.md) — who may do what, as a table: an app's `contract.json`, the one evaluator every app decides with, the audit that reads it from git, the probe that knocks on the running app, and the wall a person alone crosses.
544544 - [`docs/WEBSPEC-PREVIEWS.md`](WEBSPEC-PREVIEWS.md) — the website specification run over every preview, report-only.
545545 - [`docs/CREW.md`](CREW.md) — the crew: agents, how a run works, review, live triage, the merge queue, spend, roles and invites, the sandbox, passports, and the direction a vision pull will write.
546 - [`docs/HOPS.md`](HOPS.md) — every network hop in a crew run, where it goes, and the `hops: n, off-host: m` line each run logs.
546547 - [`docs/SENTINEL.md`](SENTINEL.md) — the forge hunting its own faults; [`docs/HUNT.md`](HUNT.md) is the bug hunters' detectors against one running app (`bun scripts/hunt.ts <origin>`).
547548 - The `flow` label: ready issues labelled `flow` go first (`readyNow` in `src/core/dependencies.ts` is the one place); when none is ready, builders take other ready work as usual. There is no switch and no state. A writer puts an issue first with the **Flow first** button on the issue and on each row of its list (`POST …/issues/:number/flow`), with `f` on the terminal issue screen, or with MCP `labels` and `action: "add"` (all three make the label on first use, for a writer only); the map's "Ready now" shows labelled work on top, marked `flow`.
548549 - Outsiders directing the crew: the terminal's About card (`I`) has a Crew row saying whether people who cannot write may direct the crew (the web page's words). A repository admin who is a person presses `d` to see the plan, then enter to confirm (esc changes nothing); it calls `setCrewOutsiders`, as MCP `outsiders` does. The key is `d`, not `o`, because `o` is the fork's upstream offer on the same card and `d` never clashes with it; a writer who is not an admin and an agent are not offered `d`.
src/crew/hops.ts added +86
@@ -0,0 +1,86 @@
1 /**
2 * Every network hop a crew run makes, counted, and where each one goes.
3 *
4 * The owner's direction (#14650): do the work on this machine and let only
5 * the model call cross the uplink. A run tallies each hop it makes as it
6 * makes it (once each step has succeeded), and its log carries one `hops: n, off-host: m` line
7 * before the timing line, so a run that went off the host for anything but the model says so.
8 * docs/HOPS.md is the map.
9 *
10 * Plain data in, plain data out: where a URL goes is read from its host
11 * name alone, never from a DNS lookup, so the count costs nothing.
12 */
13
14 /** Where a hop goes, nearest first. Only `internet` leaves the host's network. */
15 export type Reach = "in-process" | "loopback" | "lan" | "internet";
16
17 /** What a hop is for. `model` is the one kind allowed off the host. */
18 export type HopKind = "clone" | "fetch" | "push" | "agent" | "model";
19
20 export interface Hop {
21 readonly kind: HopKind;
22 readonly reach: Reach;
23 }
24
25 const LOOPBACK = /^(localhost|.*\.localhost|127(\.\d{1,3}){3}|::1|0\.0\.0\.0)$/i;
26 const LAN =
27 /^(10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){2}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){2}|169\.254(\.\d{1,3}){2}|f[cd][0-9a-f]{2}:.*|fe80:.*|[^.:]+|.*\.(local|internal|lan|svc|cluster\.local))$/i;
28
29 /**
30 * Where `url` goes. A unix socket (`unix:` or a path) is in-process for the
31 * machine's purposes; a name with no dot, a private range or a cluster name
32 * is the LAN; anything else, or anything unreadable, is the internet.
33 */
34 export function reachOf(url: string): Reach {
35 const raw = url.trim();
36 if (raw.startsWith("unix:") || raw.startsWith("/")) return "in-process";
37 let host: string;
38 try {
39 host = new URL(raw).hostname.replace(/^\[|\]$/g, "");
40 } catch {
41 return "internet";
42 }
43 if (host === "") return "internet";
44 if (LOOPBACK.test(host)) return "loopback";
45 if (LAN.test(host)) return "lan";
46 return "internet";
47 }
48
49 export const offHost = (hop: Hop): boolean =>
50 hop.reach === "internet" || hop.reach === "lan";
51
52 /** One run's tally, in the order the hops were made. */
53 export class Hops {
54 private readonly made: Hop[] = [];
55
56 record(kind: HopKind, url: string): void {
57 this.made.push({ kind, reach: reachOf(url) });
58 }
59
60 list(): readonly Hop[] {
61 return this.made;
62 }
63
64 /** The off-host hops that are not model calls: what the guard forbids. */
65 strays(): readonly Hop[] {
66 return this.made.filter((h) => offHost(h) && h.kind !== "model");
67 }
68
69 /** `stray hops: agent internet`, or nothing when only model calls left the host. */
70 strayLine(): string | undefined {
71 const strays = this.strays();
72 return strays.length === 0
73 ? undefined
74 : `stray hops: ${strays.map((h) => `${h.kind} ${h.reach}`).join(" · ")}`;
75 }
76
77 /**
78 * `hops: 4, off-host: 1 — clone loopback · push loopback · …`: the count,
79 * the count that left the host, then each hop with where it went.
80 */
81 line(): string {
82 const off = this.made.filter(offHost).length;
83 const each = this.made.map((h) => `${h.kind} ${h.reach}`).join(" · ");
84 return `hops: ${this.made.length}, off-host: ${off}${each === "" ? "" : ` — ${each}`}`;
85 }
86 }
src/crew/runner.ts +26 −1
@@ -102,6 +102,7 @@ import { deleteToken } from "../core/tokens.ts";
102102 import { runGit } from "../git/env.ts";
103103 import { installPinHook } from "./pin_hook.ts";
104104 import { cloneWithMirror } from "./mirror.ts";
105 import { Hops } from "./hops.ts";
105106 import { isSafeRef } from "../git/guard.ts";
106107 import { runGitCapture } from "../git/env.ts";
107108 import { looksForEnv } from "../core/looks.ts";
@@ -405,6 +406,8 @@ interface Ran {
405406 claimed: boolean;
406407 inHarness: boolean;
407408 inWrapUp: boolean;
409 /** Every network hop the run made, for its `hops:` line (#14650). */
410 hops: Hops;
408411 }
409412
410413 /**
@@ -418,7 +421,7 @@ export async function executeAgentRun(
418421 options: CrewOptions,
419422 ): Promise<Pause | undefined> {
420423 const watch = new Stopwatch(options.now ?? Date.now);
421 const began = { claimed: false, inHarness: false, inWrapUp: false };
424 const began = { claimed: false, inHarness: false, inWrapUp: false, hops: new Hops() };
422425 try {
423426 return await runClaimed(app, run, options, watch, began);
424427 } finally {
@@ -428,6 +431,12 @@ export async function executeAgentRun(
428431 if (began.inHarness) watch.lap("harness");
429432 if (began.inWrapUp) watch.lap("wrap-up");
430433 if (began.claimed) {
434 // Where the run's traffic went, just before where its time went.
435 appendRunLog(app.config.dataDir, run.id, began.hops.line());
436 // A forge hop that left the host is named on a line of its own, so a
437 // reader of the log (and the guard test) sees it without parsing.
438 const stray = began.hops.strayLine();
439 if (stray !== undefined) appendRunLog(app.config.dataDir, run.id, stray);
431440 appendRunLog(
432441 app.config.dataDir,
433442 run.id,
@@ -1207,6 +1216,7 @@ async function work(
12071216 fork.value.name,
12081217 );
12091218 // A run that was paused carries on in the clone it left, changes and all.
1219 const refresh = existsSync(join(dir, ".git")) ? "fetch" : "clone";
12101220 const prepared = await prepareClone(
12111221 app,
12121222 options,
@@ -1221,6 +1231,8 @@ async function work(
12211231 );
12221232 watch.lap("clone");
12231233 if (!prepared.ok) return { state: "failed", summary: prepared.error };
1234 // Counted once made: a clone that failed is no hop on the line.
1235 ran.hops.record(refresh, options.localUrl);
12241236 // Dependencies before the agent starts, so the type check has its types;
12251237 // a failure is logged and the run goes on.
12261238 const deps = await installDependencies(
@@ -1764,6 +1776,16 @@ async function work(
17641776 : { prompt: "Continue where you stopped.", sessionId: last.sessionId },
17651777 );
17661778 await early?.stop();
1779 // Once the session took a turn it made the model call and could reach
1780 // the forge at the address it is told (`FORGE_URL`, its remotes): both
1781 // are counted then, never for a session that never started.
1782 if (result.turns > 0) {
1783 ran.hops.record(
1784 "model",
1785 picked.kind === "ready" && picked.apiBase ? picked.apiBase : "https://api.anthropic.com",
1786 );
1787 ran.hops.record("agent", options.forgeUrl);
1788 }
17671789 const down = outagePause(result, result.costUsd);
17681790 if (down !== undefined) return down;
17691791 noteRunProgress(app.db, run.id, {
@@ -1969,6 +1991,7 @@ async function work(
19691991 write,
19701992 [...Object.values(grant.env), ...modelSecrets],
19711993 run.variantGroup,
1994 ran.hops,
19721995 );
19731996 // An outage that outlasted every pause is named as one, not as a push
19741997 // that found nothing: the board and the sentinel read the difference.
@@ -2299,6 +2322,7 @@ async function landWork(
22992322 write: (line: string) => void,
23002323 granted: readonly string[] = [],
23012324 variantGroup: string | null = null,
2325 hops?: Hops,
23022326 ): Promise<{ pushed: boolean; summary: string }> {
23032327 // A model may echo a granted value into a file; the tree is scrubbed
23042328 // before the forge commits it, and the agent's own commits are searched
@@ -2387,6 +2411,7 @@ async function landWork(
23872411 );
23882412 if (!pushed.ok)
23892413 write(`push refused: ${pushed.error.stderr.slice(0, 300)}`);
2414 else hops?.record("push", options.localUrl);
23902415 remote = await resolveOid(forkBare, `refs/heads/${branch}`);
23912416 }
23922417 }
test/crew-hops.test.ts added +45
@@ -0,0 +1,45 @@
1 /**
2 * Where a crew run's hops go (#14650): read from the URL alone, and only a
3 * model call may leave the host.
4 */
5
6 import { describe, expect, test } from "bun:test";
7 import { Hops, reachOf } from "../src/crew/hops.ts";
8
9 describe("reachOf", () => {
10 test("names each reach from the host alone", () => {
11 expect(reachOf("unix:/data/forge.sock")).toBe("in-process");
12 expect(reachOf("http://127.0.0.1:3000")).toBe("loopback");
13 expect(reachOf("http://localhost:8080/x.git")).toBe("loopback");
14 expect(reachOf("http://[::1]:3000")).toBe("loopback");
15 expect(reachOf("http://10.0.3.7:3000")).toBe("lan");
16 expect(reachOf("http://forge.svc.cluster.local")).toBe("lan");
17 expect(reachOf("http://forge:3000")).toBe("lan");
18 expect(reachOf("https://forge.open-platform.sh")).toBe("internet");
19 expect(reachOf("https://api.anthropic.com")).toBe("internet");
20 expect(reachOf("http://[2001:db8::1]/")).toBe("internet");
21 expect(reachOf("not a url")).toBe("internet");
22 });
23 });
24
25 describe("a run's hop tally", () => {
26 test("counts every hop and the ones that left the host", () => {
27 const hops = new Hops();
28 hops.record("clone", "http://127.0.0.1:3000/a/b.git");
29 hops.record("model", "https://api.anthropic.com");
30 hops.record("push", "http://127.0.0.1:3000/a/b.git");
31 expect(hops.line()).toBe(
32 "hops: 3, off-host: 1 — clone loopback · model internet · push loopback",
33 );
34 expect(hops.strays()).toEqual([]);
35 expect(hops.strayLine()).toBeUndefined();
36 });
37
38 test("a forge hop that leaves the host is a stray", () => {
39 const hops = new Hops();
40 hops.record("agent", "https://forge.open-platform.sh");
41 expect(hops.strays()).toEqual([{ kind: "agent", reach: "internet" }]);
42 expect(hops.strayLine()).toBe("stray hops: agent internet");
43 expect(new Hops().line()).toBe("hops: 0, off-host: 0");
44 });
45 });
test/crew-run-timing.test.ts +13
@@ -62,6 +62,19 @@ describe("an agent run's timing line", () => {
6262 );
6363 }, 30_000);
6464
65 test("a finished run counts its hops, and only the model call leaves the host (#14650)", async () => {
66 const { run, lines } = await runOn("Count the hops");
67 expect(run.state).toBe("ok");
68 const hops = lines.filter((l) => l.startsWith("hops:"));
69 expect(hops).toHaveLength(1);
70 // Just before the timing line, which stays last.
71 expect(lines.at(-2)).toBe(hops[0]!);
72 expect(hops[0]!).toMatch(/^hops: \d+, off-host: 1 — clone loopback · model internet · agent loopback/);
73 // The stand-in's FORGE_URL is loopback, so no forge hop strays; this
74 // guards the stand-in, not production (docs/HOPS.md).
75 expect(lines.some((l) => l.startsWith("stray hops:"))).toBe(false);
76 }, 30_000);
77
6578 test("a pull opened at the first push gets its line before the end line", async () => {
6679 rmSync(join(standIn.dataDir, "release"), { force: true });
6780 // The stand-in lingers after its push; release it once the pull is open.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs