Public: anyone can read this; the crew can change it.
Fewer wire hops: a crew run does its work locally, and every hop it still makes is counted #14651
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/14650-fewer-wire-hops-a-crew-run-does-its-work → main 4 commits fork
6 files
changed +236
−1
docs/HOPS.md added +65
| @@ -0,0 +1,65 @@ | ||
| 1 | # A crew run's wire hops | |
| 2 | ||
| 3 | The owner's direction (2026-11-20, #14650): do the work on this machine, and let | |
| 4 | only the model call cross the uplink. The outage of #14614 came from one hop | |
| 5 | that did not need to be one. | |
| 6 | ||
| 7 | Every run's log carries one line just before its `timing:` line: | |
| 8 | ||
| 9 | hops: 4, off-host: 1 — clone loopback · model internet · agent loopback · push loopback | |
| 10 | ||
| 11 | `src/crew/hops.ts` tallies them and names where each goes from the URL's host | |
| 12 | alone (no DNS lookup): `in-process` (a unix socket), `loopback`, `lan` (private | |
| 13 | ranges, cluster names and any name with no dot) or `internet`. `lan` and | |
| 14 | `internet` are off the host. A dotless name counts as `lan`, so a public alias | |
| 15 | with no dot in it would be miscounted. | |
| 16 | ||
| 17 | Each hop is counted once its step has succeeded: the clone or fetch after | |
| 18 | `prepareClone` returns, the push after git accepts it, and the model and agent | |
| 19 | hops once the session has taken a turn. A run that fails in `prepareClone` logs | |
| 20 | no clone. The model and agent hops are counted per session, not per request: | |
| 21 | the agent's own traffic is not observed, only the address it was given. | |
| 22 | ||
| 23 | When a hop other than the model call leaves the host, a second line names it: | |
| 24 | ||
| 25 | stray hops: agent internet | |
| 26 | ||
| 27 | ## The map | |
| 28 | ||
| 29 | | Hop | Made by | Goes to | Per run | | |
| 30 | | ------------------------------------ | -------------- | ---------------------------------------------- | --------------- | | |
| 31 | | mirror fetch and clone (`clone`) | the forge | `localUrl`, loopback; objects borrowed from the local mirror (`crew/mirror.ts`) | one, on a fresh clone | | |
| 32 | | fetch of a kept clone (`fetch`) | the forge | `localUrl`, loopback | one, on a resumed run | | |
| 33 | | model call (`model`) | the agent | the account's gateway, or api.anthropic.com | one session (many requests) | | |
| 34 | | the agent's git and API (`agent`) | the agent | `FORGE_URL`, through the sandbox's proxy | one session (many requests) | | |
| 35 | | push for the agent (`push`) | the forge | `localUrl`, loopback | only when the agent left commits unpushed | | |
| 36 | | check run, preview deploy | the forge | in-process, after the push lands | not the run's own: counted by the check and the deploy | | |
| 37 | ||
| 38 | ## Before and after | |
| 39 | ||
| 40 | Before (#14614): the forge's own mirror fetches were paced like uplink traffic | |
| 41 | and timed out. They already went to `localUrl`. #14605 keeps the mirror when a | |
| 42 | fetch fails, and the clone borrows its objects (`crew/mirror.ts`). Nobody | |
| 43 | measured hop counts before this change, so there is no "before" column. The | |
| 44 | counts below are what the code expects a run to log, not measured values: | |
| 45 | ||
| 46 | | Run | hops (expected) | off-host (expected) | | |
| 47 | | ------------------------- | --------------- | ------------------------ | | |
| 48 | | fresh clone, forge pushes | 4 | 1 (model) when `FORGE_URL` is loopback; 2 (model, agent) when it is the forge's public name | | |
| 49 | | fresh clone, agent pushes | 3 | the same | | |
| 50 | | resumed clone | 3 or 4 | the same | | |
| 51 | ||
| 52 | In production `FORGE_URL` is the forge's public name, so every real run today | |
| 53 | logs `off-host: 2` and a `stray hops: agent internet` line. The agent's | |
| 54 | remotes and `FORGE_URL` carry that name because inside the shell sandbox | |
| 55 | `localhost` is the sandbox's own (`prepareClone` says why). The next step is to | |
| 56 | route that name to the loopback inside the sandbox's proxy. Until then the line | |
| 57 | counts the hop rather than hiding it. | |
| 58 | ||
| 59 | ## The guard | |
| 60 | ||
| 61 | `test/crew-hops.test.ts` holds the reach of each kind of host and the | |
| 62 | `stray hops:` line. `test/crew-run-timing.test.ts` runs a whole stand-in run | |
| 63 | and fails when its log has a `stray hops:` line. The stand-in's `FORGE_URL` is | |
| 64 | loopback, so this guards the stand-in only. It cannot fail for the production | |
| 65 | agent hop above, which the stray line reports instead. |
docs/REFERENCE.md +1
| @@ -543,6 +543,7 @@ This page is the front door. Everything else is a file of its own under | ||
| 543 | 543 | - [](CONTRACT.md) — who may do what, as a table: an app's `contract.json`, the one evaluator every app decides with, the audit that reads it from git, the probe that knocks on the running app, and the wall a person alone crosses. |
| 544 | 544 | - [](WEBSPEC-PREVIEWS.md) — the website specification run over every preview, report-only. |
| 545 | 545 | - [](CREW.md) — the crew: agents, how a run works, review, live triage, the merge queue, spend, roles and invites, the sandbox, passports, and the direction a vision pull will write. |
| 546 | - [](HOPS.md) — every network hop in a crew run, where it goes, and the `hops: n, off-host: m` line each run logs. | |
| 546 | 547 | - [](SENTINEL.md) — the forge hunting its own faults; [](HUNT.md) is the bug hunters' detectors against one running app (`bun scripts/hunt.ts <origin>`). |
| 547 | 548 | - The `flow` label: ready issues labelled `flow` go first (`readyNow` in `src/core/dependencies.ts` is the one place); when none is ready, builders take other ready work as usual. There is no switch and no state. A writer puts an issue first with the **Flow first** button on the issue and on each row of its list (`POST …/issues/:number/flow`), with `f` on the terminal issue screen, or with MCP `labels` and `action: "add"` (all three make the label on first use, for a writer only); the map's "Ready now" shows labelled work on top, marked `flow`. |
| 548 | 549 | - Outsiders directing the crew: the terminal's About card (`I`) has a Crew row saying whether people who cannot write may direct the crew (the web page's words). A repository admin who is a person presses `d` to see the plan, then enter to confirm (esc changes nothing); it calls `setCrewOutsiders`, as MCP `outsiders` does. The key is `d`, not `o`, because `o` is the fork's upstream offer on the same card and `d` never clashes with it; a writer who is not an admin and an agent are not offered `d`. |
src/crew/hops.ts added +86
| @@ -0,0 +1,86 @@ | ||
| 1 | /** | |
| 2 | * Every network hop a crew run makes, counted, and where each one goes. | |
| 3 | * | |
| 4 | * The owner's direction (#14650): do the work on this machine and let only | |
| 5 | * the model call cross the uplink. A run tallies each hop it makes as it | |
| 6 | * makes it (once each step has succeeded), and its log carries one `hops: n, off-host: m` line | |
| 7 | * before the timing line, so a run that went off the host for anything but the model says so. | |
| 8 | * docs/HOPS.md is the map. | |
| 9 | * | |
| 10 | * Plain data in, plain data out: where a URL goes is read from its host | |
| 11 | * name alone, never from a DNS lookup, so the count costs nothing. | |
| 12 | */ | |
| 13 | ||
| 14 | /** Where a hop goes, nearest first. Only `internet` leaves the host's network. */ | |
| 15 | export type Reach = "in-process" | "loopback" | "lan" | "internet"; | |
| 16 | ||
| 17 | /** What a hop is for. `model` is the one kind allowed off the host. */ | |
| 18 | export type HopKind = "clone" | "fetch" | "push" | "agent" | "model"; | |
| 19 | ||
| 20 | export interface Hop { | |
| 21 | readonly kind: HopKind; | |
| 22 | readonly reach: Reach; | |
| 23 | } | |
| 24 | ||
| 25 | const LOOPBACK = /^(localhost|.*\.localhost|127(\.\d{1,3}){3}|::1|0\.0\.0\.0)$/i; | |
| 26 | const LAN = | |
| 27 | /^(10(\.\d{1,3}){3}|192\.168(\.\d{1,3}){2}|172\.(1[6-9]|2\d|3[01])(\.\d{1,3}){2}|169\.254(\.\d{1,3}){2}|f[cd][0-9a-f]{2}:.*|fe80:.*|[^.:]+|.*\.(local|internal|lan|svc|cluster\.local))$/i; | |
| 28 | ||
| 29 | /** | |
| 30 | * Where `url` goes. A unix socket (`unix:` or a path) is in-process for the | |
| 31 | * machine's purposes; a name with no dot, a private range or a cluster name | |
| 32 | * is the LAN; anything else, or anything unreadable, is the internet. | |
| 33 | */ | |
| 34 | export function reachOf(url: string): Reach { | |
| 35 | const raw = url.trim(); | |
| 36 | if (raw.startsWith("unix:") || raw.startsWith("/")) return "in-process"; | |
| 37 | let host: string; | |
| 38 | try { | |
| 39 | host = new URL(raw).hostname.replace(/^\[|\]$/g, ""); | |
| 40 | } catch { | |
| 41 | return "internet"; | |
| 42 | } | |
| 43 | if (host === "") return "internet"; | |
| 44 | if (LOOPBACK.test(host)) return "loopback"; | |
| 45 | if (LAN.test(host)) return "lan"; | |
| 46 | return "internet"; | |
| 47 | } | |
| 48 | ||
| 49 | export const offHost = (hop: Hop): boolean => | |
| 50 | hop.reach === "internet" || hop.reach === "lan"; | |
| 51 | ||
| 52 | /** One run's tally, in the order the hops were made. */ | |
| 53 | export class Hops { | |
| 54 | private readonly made: Hop[] = []; | |
| 55 | ||
| 56 | record(kind: HopKind, url: string): void { | |
| 57 | this.made.push({ kind, reach: reachOf(url) }); | |
| 58 | } | |
| 59 | ||
| 60 | list(): readonly Hop[] { | |
| 61 | return this.made; | |
| 62 | } | |
| 63 | ||
| 64 | /** The off-host hops that are not model calls: what the guard forbids. */ | |
| 65 | strays(): readonly Hop[] { | |
| 66 | return this.made.filter((h) => offHost(h) && h.kind !== "model"); | |
| 67 | } | |
| 68 | ||
| 69 | /** `stray hops: agent internet`, or nothing when only model calls left the host. */ | |
| 70 | strayLine(): string | undefined { | |
| 71 | const strays = this.strays(); | |
| 72 | return strays.length === 0 | |
| 73 | ? undefined | |
| 74 | : `stray hops: ${strays.map((h) => `${h.kind} ${h.reach}`).join(" · ")}`; | |
| 75 | } | |
| 76 | ||
| 77 | /** | |
| 78 | * `hops: 4, off-host: 1 — clone loopback · push loopback · …`: the count, | |
| 79 | * the count that left the host, then each hop with where it went. | |
| 80 | */ | |
| 81 | line(): string { | |
| 82 | const off = this.made.filter(offHost).length; | |
| 83 | const each = this.made.map((h) => `${h.kind} ${h.reach}`).join(" · "); | |
| 84 | return `hops: ${this.made.length}, off-host: ${off}${each === "" ? "" : ` — ${each}`}`; | |
| 85 | } | |
| 86 | } |
src/crew/runner.ts +26 −1
| @@ -102,6 +102,7 @@ import { deleteToken } from "../core/tokens.ts"; | ||
| 102 | 102 | import { runGit } from "../git/env.ts"; |
| 103 | 103 | import { installPinHook } from "./pin_hook.ts"; |
| 104 | 104 | import { cloneWithMirror } from "./mirror.ts"; |
| 105 | import { Hops } from "./hops.ts"; | |
| 105 | 106 | import { isSafeRef } from "../git/guard.ts"; |
| 106 | 107 | import { runGitCapture } from "../git/env.ts"; |
| 107 | 108 | import { looksForEnv } from "../core/looks.ts"; |
| @@ -405,6 +406,8 @@ interface Ran { | ||
| 405 | 406 | claimed: boolean; |
| 406 | 407 | inHarness: boolean; |
| 407 | 408 | inWrapUp: boolean; |
| 409 | /** Every network hop the run made, for its `hops:` line (#14650). */ | |
| 410 | hops: Hops; | |
| 408 | 411 | } |
| 409 | 412 | |
| 410 | 413 | /** |
| @@ -418,7 +421,7 @@ export async function executeAgentRun( | ||
| 418 | 421 | options: CrewOptions, |
| 419 | 422 | ): Promise<Pause | undefined> { |
| 420 | 423 | const watch = new Stopwatch(options.now ?? Date.now); |
| 421 | const began = { claimed: false, inHarness: false, inWrapUp: false }; | |
| 424 | const began = { claimed: false, inHarness: false, inWrapUp: false, hops: new Hops() }; | |
| 422 | 425 | try { |
| 423 | 426 | return await runClaimed(app, run, options, watch, began); |
| 424 | 427 | } finally { |
| @@ -428,6 +431,12 @@ export async function executeAgentRun( | ||
| 428 | 431 | if (began.inHarness) watch.lap("harness"); |
| 429 | 432 | if (began.inWrapUp) watch.lap("wrap-up"); |
| 430 | 433 | if (began.claimed) { |
| 434 | // Where the run's traffic went, just before where its time went. | |
| 435 | appendRunLog(app.config.dataDir, run.id, began.hops.line()); | |
| 436 | // A forge hop that left the host is named on a line of its own, so a | |
| 437 | // reader of the log (and the guard test) sees it without parsing. | |
| 438 | const stray = began.hops.strayLine(); | |
| 439 | if (stray !== undefined) appendRunLog(app.config.dataDir, run.id, stray); | |
| 431 | 440 | appendRunLog( |
| 432 | 441 | app.config.dataDir, |
| 433 | 442 | run.id, |
| @@ -1207,6 +1216,7 @@ async function work( | ||
| 1207 | 1216 | fork.value.name, |
| 1208 | 1217 | ); |
| 1209 | 1218 | // A run that was paused carries on in the clone it left, changes and all. |
| 1219 | const refresh = existsSync(join(dir, ".git")) ? "fetch" : "clone"; | |
| 1210 | 1220 | const prepared = await prepareClone( |
| 1211 | 1221 | app, |
| 1212 | 1222 | options, |
| @@ -1221,6 +1231,8 @@ async function work( | ||
| 1221 | 1231 | ); |
| 1222 | 1232 | watch.lap("clone"); |
| 1223 | 1233 | if (!prepared.ok) return { state: "failed", summary: prepared.error }; |
| 1234 | // Counted once made: a clone that failed is no hop on the line. | |
| 1235 | ran.hops.record(refresh, options.localUrl); | |
| 1224 | 1236 | // Dependencies before the agent starts, so the type check has its types; |
| 1225 | 1237 | // a failure is logged and the run goes on. |
| 1226 | 1238 | const deps = await installDependencies( |
| @@ -1764,6 +1776,16 @@ async function work( | ||
| 1764 | 1776 | : { prompt: "Continue where you stopped.", sessionId: last.sessionId }, |
| 1765 | 1777 | ); |
| 1766 | 1778 | await early?.stop(); |
| 1779 | // Once the session took a turn it made the model call and could reach | |
| 1780 | // the forge at the address it is told (`FORGE_URL`, its remotes): both | |
| 1781 | // are counted then, never for a session that never started. | |
| 1782 | if (result.turns > 0) { | |
| 1783 | ran.hops.record( | |
| 1784 | "model", | |
| 1785 | picked.kind === "ready" && picked.apiBase ? picked.apiBase : "https://api.anthropic.com", | |
| 1786 | ); | |
| 1787 | ran.hops.record("agent", options.forgeUrl); | |
| 1788 | } | |
| 1767 | 1789 | const down = outagePause(result, result.costUsd); |
| 1768 | 1790 | if (down !== undefined) return down; |
| 1769 | 1791 | noteRunProgress(app.db, run.id, { |
| @@ -1969,6 +1991,7 @@ async function work( | ||
| 1969 | 1991 | write, |
| 1970 | 1992 | [...Object.values(grant.env), ...modelSecrets], |
| 1971 | 1993 | run.variantGroup, |
| 1994 | ran.hops, | |
| 1972 | 1995 | ); |
| 1973 | 1996 | // An outage that outlasted every pause is named as one, not as a push |
| 1974 | 1997 | // that found nothing: the board and the sentinel read the difference. |
| @@ -2299,6 +2322,7 @@ async function landWork( | ||
| 2299 | 2322 | write: (line: string) => void, |
| 2300 | 2323 | granted: readonly string[] = [], |
| 2301 | 2324 | variantGroup: string | null = null, |
| 2325 | hops?: Hops, | |
| 2302 | 2326 | ): Promise<{ pushed: boolean; summary: string }> { |
| 2303 | 2327 | // A model may echo a granted value into a file; the tree is scrubbed |
| 2304 | 2328 | // before the forge commits it, and the agent's own commits are searched |
| @@ -2387,6 +2411,7 @@ async function landWork( | ||
| 2387 | 2411 | ); |
| 2388 | 2412 | if (!pushed.ok) |
| 2389 | 2413 | write(`push refused: ${pushed.error.stderr.slice(0, 300)}`); |
| 2414 | else hops?.record("push", options.localUrl); | |
| 2390 | 2415 | remote = await resolveOid(forkBare, `refs/heads/${branch}`); |
| 2391 | 2416 | } |
| 2392 | 2417 | } |
test/crew-hops.test.ts added +45
| @@ -0,0 +1,45 @@ | ||
| 1 | /** | |
| 2 | * Where a crew run's hops go (#14650): read from the URL alone, and only a | |
| 3 | * model call may leave the host. | |
| 4 | */ | |
| 5 | ||
| 6 | import { describe, expect, test } from "bun:test"; | |
| 7 | import { Hops, reachOf } from "../src/crew/hops.ts"; | |
| 8 | ||
| 9 | describe("reachOf", () => { | |
| 10 | test("names each reach from the host alone", () => { | |
| 11 | expect(reachOf("unix:/data/forge.sock")).toBe("in-process"); | |
| 12 | expect(reachOf("http://127.0.0.1:3000")).toBe("loopback"); | |
| 13 | expect(reachOf("http://localhost:8080/x.git")).toBe("loopback"); | |
| 14 | expect(reachOf("http://[::1]:3000")).toBe("loopback"); | |
| 15 | expect(reachOf("http://10.0.3.7:3000")).toBe("lan"); | |
| 16 | expect(reachOf("http://forge.svc.cluster.local")).toBe("lan"); | |
| 17 | expect(reachOf("http://forge:3000")).toBe("lan"); | |
| 18 | expect(reachOf("https://forge.open-platform.sh")).toBe("internet"); | |
| 19 | expect(reachOf("https://api.anthropic.com")).toBe("internet"); | |
| 20 | expect(reachOf("http://[2001:db8::1]/")).toBe("internet"); | |
| 21 | expect(reachOf("not a url")).toBe("internet"); | |
| 22 | }); | |
| 23 | }); | |
| 24 | ||
| 25 | describe("a run's hop tally", () => { | |
| 26 | test("counts every hop and the ones that left the host", () => { | |
| 27 | const hops = new Hops(); | |
| 28 | hops.record("clone", "http://127.0.0.1:3000/a/b.git"); | |
| 29 | hops.record("model", "https://api.anthropic.com"); | |
| 30 | hops.record("push", "http://127.0.0.1:3000/a/b.git"); | |
| 31 | expect(hops.line()).toBe( | |
| 32 | "hops: 3, off-host: 1 — clone loopback · model internet · push loopback", | |
| 33 | ); | |
| 34 | expect(hops.strays()).toEqual([]); | |
| 35 | expect(hops.strayLine()).toBeUndefined(); | |
| 36 | }); | |
| 37 | ||
| 38 | test("a forge hop that leaves the host is a stray", () => { | |
| 39 | const hops = new Hops(); | |
| 40 | hops.record("agent", "https://forge.open-platform.sh"); | |
| 41 | expect(hops.strays()).toEqual([{ kind: "agent", reach: "internet" }]); | |
| 42 | expect(hops.strayLine()).toBe("stray hops: agent internet"); | |
| 43 | expect(new Hops().line()).toBe("hops: 0, off-host: 0"); | |
| 44 | }); | |
| 45 | }); |
test/crew-run-timing.test.ts +13
| @@ -62,6 +62,19 @@ describe("an agent run's timing line", () => { | ||
| 62 | 62 | ); |
| 63 | 63 | }, 30_000); |
| 64 | 64 | |
| 65 | test("a finished run counts its hops, and only the model call leaves the host (#14650)", async () => { | |
| 66 | const { run, lines } = await runOn("Count the hops"); | |
| 67 | expect(run.state).toBe("ok"); | |
| 68 | const hops = lines.filter((l) => l.startsWith("hops:")); | |
| 69 | expect(hops).toHaveLength(1); | |
| 70 | // Just before the timing line, which stays last. | |
| 71 | expect(lines.at(-2)).toBe(hops[0]!); | |
| 72 | expect(hops[0]!).toMatch(/^hops: \d+, off-host: 1 — clone loopback · model internet · agent loopback/); | |
| 73 | // The stand-in's FORGE_URL is loopback, so no forge hop strays; this | |
| 74 | // guards the stand-in, not production (docs/HOPS.md). | |
| 75 | expect(lines.some((l) => l.startsWith("stray hops:"))).toBe(false); | |
| 76 | }, 30_000); | |
| 77 | ||
| 65 | 78 | test("a pull opened at the first push gets its line before the end line", async () => { |
| 66 | 79 | rmSync(join(standIn.dataDir, "release"), { force: true }); |
| 67 | 80 | // The stand-in lingers after its push; release it once the pull is open. |
new above
offline