Public: anyone can read this; the crew can change it.

The security machine carries its investigation tools installed, not just named #9480

Parked opened this
Closed without merging

scout/forge:scout/9470-the-security-machine-carries-its-investi → main 10 commits fork 3 threads · 3 unresolved

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
13 files changed +683 −28
docs/features/machines.md +50 −6
@@ -41,8 +41,51 @@ cannot widen its own machine by editing the file on its branch):
4141
4242 ## What a run gets
4343
44 `FORGE_MACHINE` and `FORGE_MACHINE_TOOLS` (names only) in its environment, a
45 line in its brief naming the tools its machine carries, and its `apps` reach.
44 `FORGE_MACHINE` and `FORGE_MACHINE_TOOLS` (the installed tools' names) in its
45 environment, the installed tools on its `PATH`, a line in its brief naming
46 what is on `PATH` and what is declared but missing, and its `apps` reach.
47
48 ## Installed tools (#9470)
49
50 `src/crew/machine_tools.ts` pins each tool to a versioned release asset
51 (`TOOL_PINS`); `machine-tools.lock.json`, beside `package.json`, holds each
52 asset's sha256. The operator installs them; a run never downloads anything:
53
54 ```sh
55 bun scripts/machine-tools.ts pin # sums the pinned assets into the lock, for a reviewed pull
56 FORGE_DATA=/data bun scripts/machine-tools.ts install # downloads, refuses any sum that is not the lock's, unpacks
57 ```
58
59 Each lands in `<data>/crew/.tools/<tool>-<version>/` with `.verified.json`
60 beside it: the archive's sum, and the executable's own sum and size. A run
61 re-hashes the executable before it trusts it, so a binary changed after
62 install is named as missing, not put on `PATH`. The installer reads at most
63 512 MiB per download, lists every archive before unpacking it, and refuses
64 the whole archive when a member is a link or device, is absolute, or climbs
65 out with `..`; `tar` runs with `--no-same-owner`. Every run's sandbox denies `crew/.tools`; a machine that names a
66 tool, and finds it installed with the lock's sum, has that one directory
67 re-allowed for reading (never writing) and put first on `PATH`. A run in
68 `standard` neither reads nor finds them. Reach is unchanged: the hosts a run
69 may connect to are the same with or without tools, so a scanner can aim at
70 the repository and its own app only, and the forge's secrets stay denied.
71 A declared tool that is not installed is named as missing in the brief,
72 never claimed.
73
74 | Tool | Pinned | Why a security machine carries it |
75 | --- | --- | --- |
76 | gitleaks 8.18.4 | linux x64 tarball | finds secrets committed to the repository, the first thing to rule out |
77 | trivy 0.50.1 | linux 64-bit tarball | dependency and image vulnerabilities from the lockfiles |
78 | osv-scanner 1.9.2 | linux amd64 binary | a second opinion on the lockfiles, from the OSV database |
79 | nuclei 3.3.5 | linux amd64 zip | template checks against the repository's own running app |
80 | testssl.sh 3.0.9 | source tag tarball | the app's TLS protocols and ciphers (needs bash and openssl on the host) |
81 | zap | left out | a JVM application: no single pinned artifact runs without a Java runtime |
82 | nmap | left out | no upstream static Linux build; a distribution package cannot be pinned |
83
84 The lock ships empty: the sums were not taken in the run that added this
85 (its sandbox could not reach the release host), so until a `pin` pull lands
86 nothing installs and the brief says so. A test holds the lock to all or
87 nothing: once one sum lands, every pinned tool needs one, and this paragraph
88 must go.
4689
4790 ## A shell into a machine (#9471)
4891
@@ -87,7 +130,8 @@ repository's machines:
87130
88131 ## Not done yet (split from #9418)
89132
90 Disk ceilings are not declared (#9460); the tools are declared, not yet
91 installed in an image; the browser terminal (ghostty-web) that speaks to the
92 shell socket is not built; the shell's cell does not reach the repository's
93 apps even when its machine says `apps`.
133 Disk ceilings are not declared (#9460); zap and nmap are not installed
134 (above), and the shell's cell does not carry the machine's tools; the browser
135 terminal (ghostty-web) that speaks to the shell socket is not built; the
136 shell's cell does not reach the repository's apps even when its machine says
137 `apps`.
machine-tools.lock.json added +1
@@ -0,0 +1,1 @@
1 {}

Empty lock: no tool can be verified, so none is ever installed or on PATH. Commit the real sha256 for gitleaks, trivy, osv-scanner, nuclei and testssl.sh here.

scripts/machine-tools.ts added +150
@@ -0,0 +1,150 @@
1 /**
2 * Installs the machines' investigation tools (#9470) under the forge's
3 * data, from the pinned releases in `src/crew/machine_tools.ts`. Run by the
4 * operator, never by a crew run (a run downloads nothing):
5 *
6 * bun scripts/machine-tools.ts pin # sums the pinned assets into machine-tools.lock.json, for a reviewed pull
7 * bun scripts/machine-tools.ts install # FORGE_DATA=/data: downloads, checks each sum against the lock, unpacks
8 *
9 * An archive whose sum is not the lock's is refused and nothing is unpacked.
10 * A download is read up to MAX_ASSET_BYTES and refused past it. An archive
11 * is listed before it is unpacked, and refused whole when any member is not
12 * a plain file or directory, is absolute, or climbs out with `..`. The
13 * archive tools are run with fixed arguments on the forge's own paths.
14 */
15
16 import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs";
17 import { join } from "node:path";
18 import {
19 LOCK_PATH,
20 lockKey,
21 MAX_ASSET_BYTES,
22 readLock,
23 sha256Hex as sha256,
24 TOOL_PINS,
25 toolDir,
26 UNPINNED,
27 unsafeMember,
28 VERIFIED_FILE,
outdated

The download is read fully into memory with no byte cap. Stream it and stop past a fixed limit; also validate archive member paths before extracting (no absolute, no ..).

29 type ToolPin,
30 type Verified,
31 } from "../src/crew/machine_tools.ts";
32
33 /** The asset's bytes, streamed and refused past MAX_ASSET_BYTES. */
34 async function fetchAsset(pin: ToolPin): Promise<Uint8Array> {
35 const response = await fetch(pin.url, { redirect: "follow" });
36 if (!response.ok || response.body === null) throw new Error(`${pin.url}: ${response.status}`);
37 const declared = Number(response.headers.get("content-length") ?? "0");
38 if (declared > MAX_ASSET_BYTES) throw new Error(`${pin.url}: ${declared} bytes is past the cap`);
39 const chunks: Uint8Array[] = [];
40 let total = 0;
41 const reader = response.body.getReader();
42 for (;;) {
43 const { done, value } = await reader.read();
44 if (done) break;
45 total += value.byteLength;
46 if (total > MAX_ASSET_BYTES) {
47 await reader.cancel();
48 throw new Error(`${pin.url}: past ${MAX_ASSET_BYTES} bytes, refused`);
49 }
50 chunks.push(value);
51 }
52 const out = new Uint8Array(total);
53 let at = 0;
54 for (const chunk of chunks) {
55 out.set(chunk, at);
56 at += chunk.byteLength;
57 }
58 return out;
59 }
60
61 async function lines(argv: string[]): Promise<string[]> {
62 const p = Bun.spawn(argv, { stdout: "pipe", stderr: "inherit" });
63 const text = await new Response(p.stdout).text();
64 if ((await p.exited) !== 0) throw new Error(`${argv[0]} could not list the archive`);
65 return text.split("\n").filter((l) => l !== "");
66 }
67
68 /** Every member of the archive is safe to unpack, or it throws naming the first that is not. */
69 async function checkMembers(kind: "tar.gz" | "zip", archive: string): Promise<void> {
70 // Names one per line, and the long listing's first column for each one's type.
71 const names = kind === "zip" ? await lines(["unzip", "-Z1", archive]) : await lines(["tar", "-tzf", archive]);
72 const long = kind === "zip" ? await lines(["unzip", "-Zs", archive]) : await lines(["tar", "-tvzf", archive]);
73 const types = (kind === "zip" ? long.filter((l) => /^[-dl?a-z][-rwxsStT]{9}/.test(l)) : long).map((l) => l[0] ?? "?");
74 if (types.length !== names.length) throw new Error("the archive's two listings disagree");
75 names.forEach((name, i) => {
76 const why = unsafeMember(name.replace(/\/$/, ""), types[i] ?? "?");
77 if (why !== undefined) throw new Error(`refused: ${why}`);
78 });
79 }
80
81 async function unpack(tool: string, pin: ToolPin, bytes: Uint8Array, sum: string, dataDir: string) {
82 const dest = toolDir(dataDir, tool, pin);
83 const staging = `${dest}.new`;
84 rmSync(staging, { recursive: true, force: true });
85 mkdirSync(staging, { recursive: true });
86 if (pin.kind === "binary") {
87 const exe = join(staging, pin.binDir, pin.exe);
88 writeFileSync(exe, bytes);
89 chmodSync(exe, 0o755);
90 } else {
91 const archive = join(staging, `.asset.${pin.kind}`);
92 writeFileSync(archive, bytes);
93 await checkMembers(pin.kind, archive);
94 const argv =
95 pin.kind === "zip"
96 ? ["unzip", "-q", "-n", archive, "-d", staging]
97 : ["tar", "--no-same-owner", "--no-overwrite-dir", "-xzf", archive, "-C", staging];
98 const code = await Bun.spawn(argv, { stdout: "inherit", stderr: "inherit" }).exited;
99 rmSync(archive, { force: true });
100 if (code !== 0) throw new Error(`${tool}: ${argv[0]} exited ${code}`);
101 }
102 const exe = join(staging, pin.binDir, pin.exe);
103 const verified: Verified = { archive: sum, exe: sha256(readFileSync(exe)), size: statSync(exe).size };
104 writeFileSync(join(staging, VERIFIED_FILE), `${JSON.stringify(verified)}\n`);
105 rmSync(dest, { recursive: true, force: true });
106 renameSync(staging, dest);
107 }
108
109 async function main(command: string | undefined): Promise<number> {
110 for (const [tool, why] of Object.entries(UNPINNED)) console.log(`${tool}: left out — ${why}`);
111 if (command === "pin") {
112 const lock: Record<string, string> = {};
113 for (const [tool, pin] of Object.entries(TOOL_PINS)) {
114 lock[lockKey(tool, pin)] = sha256(await fetchAsset(pin));
115 console.log(`${lockKey(tool, pin)} ${lock[lockKey(tool, pin)]}`);
116 }
117 writeFileSync(LOCK_PATH, `${JSON.stringify(lock, null, 2)}\n`);
118 return 0;
119 }
120 if (command === "install") {
121 const dataDir = process.env["FORGE_DATA"];
122 if (dataDir === undefined || dataDir === "") {
123 console.error("FORGE_DATA must name the forge's data directory");
124 return 1;
125 }
126 const lock = readLock();
127 let failed = 0;
128 for (const [tool, pin] of Object.entries(TOOL_PINS)) {
129 const want = lock[lockKey(tool, pin)];
130 if (want === undefined) {
131 console.log(`${tool}: skipped — no sum in machine-tools.lock.json`);
132 continue;
133 }
134 const bytes = await fetchAsset(pin);
135 const got = sha256(bytes);
136 if (got !== want) {
137 console.error(`${tool}: refused — sha256 ${got} is not the lock's ${want}`);
138 failed++;
139 continue;
140 }
141 await unpack(tool, pin, bytes, want, dataDir);
142 console.log(`${tool} ${pin.version}: installed`);
143 }
144 return failed === 0 ? 0 : 1;
145 }
146 console.error("usage: bun scripts/machine-tools.ts pin|install");
147 return 2;
148 }
149
150 process.exit(await main(process.argv[2]));
src/crew/brief.ts +6 −3
@@ -8,6 +8,7 @@
88 */
99
1010 import { machineNote, type Machine } from "./machines.ts";
11 import type { MachineKit } from "./machine_tools.ts";
1112 import type { OwnApp } from "./app_hosts.ts";
1213 import type { Owed } from "./team_share.ts";
1314 import type { Agent, CrewRole } from "../core/agents.ts";
@@ -53,6 +54,8 @@ export interface BriefInput {
5354 readonly role?: CrewRole | undefined;
5455 /** The machine this role's shell runs in (`crew/machines.ts`). */
5556 readonly machine?: Machine | undefined;
57 /** What that machine really carries here (`crew/machine_tools.ts`). */
58 readonly kit?: MachineKit | undefined;
5659 /**
5760 * The policy in force: the app's contract as a table, when it has one.
5861 * A builder is told the room it works in before it starts, so the doors
@@ -104,8 +107,8 @@ export function fenced(text: string, info = ""): string[] {
104107 }
105108
106109 /** The brief line naming a machine's tools; none for a machine without any. */
107 function machineLines(machine: Machine | undefined): string[] {
108 const note = machine === undefined ? undefined : machineNote(machine);
110 function machineLines(machine: Machine | undefined, kit: MachineKit | undefined): string[] {
111 const note = machine === undefined ? undefined : machineNote(machine, kit);
109112 return note === undefined ? [] : [note, ""];
110113 }
111114
@@ -132,7 +135,7 @@ export function systemPrompt(input: BriefInput): string {
132135 "",
133136 "Read docs/VISION.md first: it says what the forge is for and the order of work. Done means reachable: a function nobody can call is not done. Where the repository has docs/PRINCIPLES.md, read it too: what it holds to, and what enforces each.",
134137 "",
135 ...machineLines(input.machine),
138 ...machineLines(input.machine, input.kit),
136139 "How to finish:",
137140 "- Commit your work with clear messages and push: `git push`.",
138141 "- If `forge.json` declares a `check`, the forge runs it against every push to your pull, and that verdict is what lands it; a failing check comes back to you with its log.",
src/crew/machine_tools.ts added +234
@@ -0,0 +1,234 @@
1 /**
2 * The investigation tools a machine carries, installed (#9470): where each
3 * one lives, from which pinned release, and whether it is really there.
4 *
5 * A tool is installed by the operator (`scripts/machine-tools.ts install`),
6 * never by a run: the archive at its versioned release URL is checked
7 * against the sha256 in `machine-tools.lock.json` (reviewed in a pull like
8 * any code) and unpacked into `<data>/crew/.tools/<tool>-<version>/`, with
9 * a record beside it of the archive's sum and the executable's own sum and
10 * size. A run trusts the install only while the executable still hashes to
11 * what was recorded: a binary changed after install is not installed. That directory is under `crew/` but
12 * not any agent's, so every run's sandbox denies it (`crew/sandbox.ts`);
13 * a machine that carries the tool has its directory re-allowed and put on
14 * `PATH`, read-only. A run in `standard` neither sees nor finds them.
15 *
16 * Left out, and named: `zap` (a JVM application with no single pinned
17 * artifact that runs without a Java runtime) and `nmap` (no upstream
18 * static Linux build; a distribution package cannot be pinned here).
19 */
20
21 import { existsSync, readFileSync, statSync } from "node:fs";
22 import { join } from "node:path";
23 import { crewRoot } from "./workdir.ts";
24 import type { Machine } from "./machines.ts";
25
26 export interface ToolPin {
27 readonly version: string;
28 /** The versioned release asset; never a "latest" link. */
29 readonly url: string;
30 /** How the asset unpacks: a tarball, a zip, or the executable itself. */
31 readonly kind: "tar.gz" | "zip" | "binary";
32 /** The directory inside the install that holds the executable. */
33 readonly binDir: string;
34 /** The executable's name in `binDir`. */
35 readonly exe: string;
36 }
37
38 export const TOOL_PINS: Readonly<Record<string, ToolPin>> = {
39 gitleaks: {
40 version: "8.18.4",
41 url: "https://github.com/gitleaks/gitleaks/releases/download/v8.18.4/gitleaks_8.18.4_linux_x64.tar.gz",
42 kind: "tar.gz",
43 binDir: ".",
44 exe: "gitleaks",
45 },
46 trivy: {
47 version: "0.50.1",
48 url: "https://github.com/aquasecurity/trivy/releases/download/v0.50.1/trivy_0.50.1_Linux-64bit.tar.gz",
49 kind: "tar.gz",
50 binDir: ".",
51 exe: "trivy",
52 },
53 "osv-scanner": {
54 version: "1.9.2",
55 url: "https://github.com/google/osv-scanner/releases/download/v1.9.2/osv-scanner_linux_amd64",
56 kind: "binary",
57 binDir: ".",
58 exe: "osv-scanner",
59 },
60 nuclei: {
61 version: "3.3.5",
62 url: "https://github.com/projectdiscovery/nuclei/releases/download/v3.3.5/nuclei_3.3.5_linux_amd64.zip",
63 kind: "zip",
64 binDir: ".",
65 exe: "nuclei",
66 },
67 "testssl.sh": {
68 version: "3.0.9",
69 url: "https://github.com/drwetter/testssl.sh/archive/refs/tags/v3.0.9.tar.gz",
70 kind: "tar.gz",
71 binDir: "testssl.sh-3.0.9",
72 exe: "testssl.sh",
73 },
74 };
75
76 /** Named in `TOOLS` but never installed, and why. */
77 export const UNPINNED: Readonly<Record<string, string>> = {
78 zap: "a JVM application: no single pinned artifact runs without a Java runtime",
79 nmap: "no upstream static Linux build; a distribution package cannot be pinned",
80 };
81
82 /** The record the installer writes once an archive matched the lock. */
83 export const VERIFIED_FILE = ".verified.json";
84
85 /** What the installer recorded: the archive's sum and the executable's. */
86 export interface Verified {
87 readonly archive: string;
88 readonly exe: string;
89 readonly size: number;
90 }
91
92 /** The largest asset the installer will read; past it the download is refused. */
93 export const MAX_ASSET_BYTES = 512 * 1024 * 1024;
94
95 export const sha256Hex = (bytes: Uint8Array): string =>
96 new Bun.CryptoHasher("sha256").update(bytes).digest("hex");
97
98 /**
99 * Why an archive member may not be unpacked, or undefined when it may: only
100 * plain files and directories, by relative paths that stay inside.
101 */
102 export function unsafeMember(name: string, type: string): string | undefined {
103 if (type !== "-" && type !== "d") return `${name}: not a plain file or directory`;
104 if (name === "" || name.startsWith("/")) return `${name}: absolute path`;
105 if (name.split("/").includes("..")) return `${name}: climbs out with ..`;
106 if (/[\\\0]/.test(name)) return `${name}: backslash or NUL in the path`;
107 return undefined;
108 }
109
110 function readVerified(dir: string): Verified | undefined {
111 try {
112 const doc = JSON.parse(readFileSync(join(dir, VERIFIED_FILE), "utf8")) as Record<string, unknown>;
113 const { archive, exe, size } = doc;
114 if (typeof archive !== "string" || !SHA_RE.test(archive)) return undefined;
115 if (typeof exe !== "string" || !SHA_RE.test(exe)) return undefined;
116 if (typeof size !== "number" || !Number.isSafeInteger(size) || size <= 0) return undefined;
117 return { archive, exe, size };
118 } catch {
119 return undefined;
120 }
121 }
122
123 /** The executable still is what the installer recorded: same size, same sum. */
124 function stillVerified(path: string, want: Verified): boolean {
125 try {
126 if (statSync(path).size !== want.size) return false;
127 return sha256Hex(readFileSync(path)) === want.exe;
128 } catch {
129 return false;
130 }
131 }
132
133 export const toolsRoot = (dataDir: string): string => join(crewRoot(dataDir), ".tools");
134
135 export const toolDir = (dataDir: string, tool: string, pin: ToolPin): string =>
136 join(toolsRoot(dataDir), `${tool}-${pin.version}`);
137
138 const SHA_RE = /^[0-9a-f]{64}$/;
139
140 /** The lock's sums, validated: a malformed entry is no pin at all. */
141 export function parseLock(text: string | undefined): Record<string, string> {
142 if (text === undefined) return {};
143 let root: unknown;
144 try {
145 root = JSON.parse(text);
146 } catch {
147 return {};
148 }
149 if (typeof root !== "object" || root === null || Array.isArray(root)) return {};
150 const out: Record<string, string> = {};
151 for (const [key, value] of Object.entries(root as Record<string, unknown>))
outdated

Only the .sha256 marker is compared, not the executable. A modified or replaced binary still passes. Record a digest of the executable at install and re-check it here.

152 if (typeof value === "string" && SHA_RE.test(value)) out[key] = value;
153 return out;
154 }
155
156 /** The lock's key for a tool at its pinned version. */
157 export const lockKey = (tool: string, pin: ToolPin): string => `${tool}@${pin.version}`;
158
159 /** The forge's own lock, beside its package.json: never a run's clone. */
160 export const LOCK_PATH = join(import.meta.dir, "..", "..", "machine-tools.lock.json");
161
162 export function readLock(path = LOCK_PATH): Record<string, string> {
163 try {
164 return parseLock(readFileSync(path, "utf8"));
165 } catch {
166 return {};
167 }
168 }
169
170 export interface MachineKit {
171 /** Tools installed and verified, with the directory put on PATH. */
172 readonly present: readonly {
173 readonly tool: string;
174 /** The tool's install, re-allowed for reading in the sandbox. */
175 readonly dir: string;
176 /** The directory put on PATH. */
177 readonly binDir: string;
178 }[];
179 /** Tools the machine names that are not on PATH, and why. */
180 readonly missing: readonly { readonly tool: string; readonly why: string }[];
181 }
182
183 /**
184 * What a machine's shell really carries: each named tool whose install
185 * holds the sum the lock pins and its executable. Nothing else is claimed.
186 */
187 export function machineKit(
188 dataDir: string,
189 machine: Machine,
190 lock: Readonly<Record<string, string>> = readLock(),
191 ): MachineKit {
192 const present: { tool: string; dir: string; binDir: string }[] = [];
193 const missing: { tool: string; why: string }[] = [];
194 for (const tool of machine.tools) {
195 const pin = TOOL_PINS[tool];
196 if (pin === undefined) {
197 missing.push({ tool, why: UNPINNED[tool] ?? "not pinned" });
198 continue;
199 }
200 const sum = lock[lockKey(tool, pin)];
201 if (sum === undefined) {
202 missing.push({ tool, why: "no checksum in machine-tools.lock.json yet" });
203 continue;
204 }
205 const dir = toolDir(dataDir, tool, pin);
206 const binDir = join(dir, pin.binDir);
207 const exe = join(binDir, pin.exe);
208 const verified = readVerified(dir);
209 if (verified?.archive !== sum || !isExecutable(exe)) {
210 missing.push({ tool, why: "not installed on this forge" });
211 continue;
212 }
213 if (!stillVerified(exe, verified)) {
214 missing.push({ tool, why: "changed since it was installed: reinstall it" });
215 continue;
216 }
217 present.push({ tool, dir, binDir });
218 }
219 return { present, missing };
220 }
221
222 function isExecutable(path: string): boolean {
223 if (!existsSync(path)) return false;
224 try {
225 return (statSync(path).mode & 0o111) !== 0;
226 } catch {
227 return false;
228 }
229 }
230
231 /** The run's PATH: the machine's tool directories first, then the forge's own. */
232 export function kitPath(kit: MachineKit, base: string): string {
233 return [...kit.present.map((p) => p.binDir), base].join(":");
234 }
src/crew/machines.ts +24 −10
@@ -20,6 +20,7 @@
2020
2121 import { CREW_ROLES, type CrewRole } from "../core/agents.ts";
2222 import { err, ok, type Result } from "../kernel/result.ts";
23 import type { MachineKit } from "./machine_tools.ts";
2324
2425 /**
2526 * What a machine may reach beyond the forge itself (always reachable: the
@@ -205,20 +206,33 @@ export function machineLimits(machine: Machine): { memoryMiB?: number; cpus?: nu
205206 };
206207 }
207208
208 /** The run's environment for its machine: names only, nothing secret. */
209 export function machineEnv(machine: Machine): Record<string, string> {
209 /**
210 * The run's environment for its machine: names only, nothing secret.
211 * `FORGE_MACHINE_TOOLS` names what is installed, not what was declared.
212 */
213 export function machineEnv(machine: Machine, kit?: MachineKit): Record<string, string> {
210214 return {
211215 FORGE_MACHINE: machine.name,
212 FORGE_MACHINE_TOOLS: machine.tools.join(","),
216 FORGE_MACHINE_TOOLS: (kit === undefined ? [] : kit.present.map((p) => p.tool)).join(","),
213217 };
214218 }
215219
216 /** One line for the brief: where the shell runs and what it was given. */
217 export function machineNote(machine: Machine): string | undefined {
220 /**
221 * One line for the brief: where the shell runs and what is on its PATH.
222 * Only an installed tool is claimed; a declared one that is not is named
223 * as missing, so the brief never promises a command that is not there.
224 */
225 export function machineNote(machine: Machine, kit?: MachineKit): string | undefined {
218226 if (machine.tools.length === 0) return undefined;
219 return (
220 `Your shell runs in the \`${machine.name}\` machine. Tools it is meant to carry (use those on PATH; none is installed for you, and each may be aimed only at the forge's own copies): ` +
221 machine.tools.map((t) => `${t} (${TOOLS[t]})`).join("; ") +
222 "."
223 );
227 const present = kit?.present.map((p) => p.tool) ?? [];
228 const missing = machine.tools.filter((t) => !present.includes(t));
229 const has =
230 present.length === 0
231 ? "None of its tools is installed on this forge."
232 : "On your PATH, to be aimed only at the forge's own copies (the repository and its own app): " +
233 present.map((t) => `${t} (${TOOLS[t]})`).join("; ") +
234 ".";
235 const lacks =
236 missing.length === 0 ? "" : ` Not installed here, so do not count on them: ${missing.join(", ")}.`;
237 return `Your shell runs in the \`${machine.name}\` machine. ${has}${lacks}`;
224238 }
src/crew/runner.ts +10 −1
@@ -36,6 +36,7 @@ import { appsRuntime } from "../apps/runtime.ts";
3636 import { chooseModel, modelFacts } from "./model_choice.ts";
3737 import { ownApp, ownAppUrls } from "./app_hosts.ts";
3838 import { machineEnv, machineFor, machineLimits, reachableApps } from "./machines.ts";
39 import { kitPath, machineKit } from "./machine_tools.ts";
3940 import { repoMachines } from "./machines_read.ts";
4041 import { authorsInRepoPull } from "./authors_pull.ts";
4142 import { crewRole, getAgent, type Agent } from "../core/agents.ts";
@@ -1095,10 +1096,14 @@ async function work(
10951096 // The machine this seat runs in, from the default branch's forge.json.
10961097 const role = crewRole(app.db, repo.id, agent.user.id);
10971098 const machine = machineFor(await repoMachines(app, repo), role);
1099 // What the machine really carries on this forge: only these are on PATH
1100 // and named in the brief (`crew/machine_tools.ts`).
1101 const kit = machineKit(app.config.dataDir, machine);
10981102 const brief = {
10991103 agent,
11001104 repo,
11011105 machine,
1106 kit,
11021107 issue,
11031108 pull,
11041109 branch,
@@ -1326,7 +1331,10 @@ async function work(
13261331 ...passportEnv(app.config.dataDir),
13271332 ...credentialEnv(picked),
13281333 ...grant.env,
1329 ...machineEnv(machine),
1334 ...machineEnv(machine, kit),
1335 ...(kit.present.length === 0
1336 ? {}
1337 : { PATH: kitPath(kit, process.env["PATH"] ?? "/usr/local/bin:/usr/bin:/bin") }),
13301338 ...(options.configDir === undefined
13311339 ? {}
13321340 : { CLAUDE_CONFIG_DIR: options.configDir }),
@@ -1355,6 +1363,7 @@ async function work(
13551363 enforced: options.sandboxEnforced,
13561364 runId: run.id,
13571365 repoId: repo.id,
1366 toolDirs: kit.present.map((p) => p.dir),
13581367 });
13591368 const sandbox = shellPolicy();
13601369 // The run's own temp directory, never the machine's shared one; it goes
src/crew/sandbox.ts +13 −1
@@ -250,6 +250,12 @@ export function sandboxPolicy(input: {
250250 runId?: string | undefined;
251251 /** The repository the run is on: its install cache is writable (`installCache`). */
252252 repoId?: string | undefined;
253 /**
254 * The installed tool directories the run's machine carries
255 * (`crew/machine_tools.ts`): re-allowed for reading inside the denied
256 * `crew/.tools`, never written. A machine without tools sees none.
257 */
258 toolDirs?: readonly string[] | undefined;
253259 }): SandboxPolicy {
254260 const hosts = new Set<string>(["registry.npmjs.org"]);
255261 for (const url of input.urls) {
@@ -297,6 +303,9 @@ export function sandboxPolicy(input: {
297303 ...others(crew, input.agentName),
298304 // The mirrors, by name: one a later run creates is not in the listing.
299305 join(crew, ".mirrors"),
306 // The machines' installed tools, by name: re-allowed (`readAllow`) only
307 // for the directories the run's own machine carries.
308 join(crew, ".tools"),
300309 // This agent's other runs' temp directories.
301310 ...others(join(own, ".t"), tmpDir === undefined ? "" : basename(tmpDir)),
302311 // The machine's temp directory: every other session's files, and the
@@ -313,7 +322,10 @@ export function sandboxPolicy(input: {
313322 allowHosts: [...hosts],
314323 denyRead: deny,
315324 enforced: input.enforced ?? true,
316 readAllow: installCache === undefined ? [] : [installCache],
325 readAllow: [
326 ...(installCache === undefined ? [] : [installCache]),
327 ...(input.toolDirs ?? []).filter((dir) => isAbsolute(dir)),
328 ],
317329 ...(tmpDir === undefined ? {} : { tmpDir }),
318330 ...(installCache === undefined ? {} : { installCache }),
319331 };
src/http/server.ts −3
@@ -544,7 +544,6 @@ import {
544544 shellTarget,
545545 stopShells,
546546 withShells,
547 closeStaleSessions,
548547 type ForgeSocket,
549548 type ShellSocket,
550549 } from "../machines/shell.ts";
@@ -2161,8 +2160,6 @@ export function forgeHttp(app: App): ForgeHttp {
21612160
21622161 export async function serve(app: App): Promise<Servers> {
21632162 const router = buildRouter();
2164 // No shell outlives its forge: a row a stop or crash left open is closed.
2165 closeStaleSessions(app.db);
21662163 const apiRouter = buildApiRouter();
21672164 await clearStaleSockets(app);
21682165 // One limiter per listener: its state is a minute of timestamps, and two
src/main.ts +5
@@ -98,6 +98,7 @@ import { stallLine } from "./perf/loop.ts";
9898 import { bootStall, gitRecord, loopWatch, perfRecord } from "./perf/live.ts";
9999 import { timeStatements } from "./kernel/slow_sql.ts";
100100 import { waitForPeer } from "./runs/peer.ts";
101 import { closeStaleSessions } from "./machines/shell.ts";
101102
102103 // `forge hook <name>` re-enters here from a git hook stub. It relays to the
103104 // running forge over its socket and exits — it must never boot a server.
@@ -425,6 +426,10 @@ const earlySignals = ["SIGINT", "SIGTERM", "SIGHUP"] as const;
425426 for (const signal of earlySignals) process.once(signal, leaveEarly);
426427 await waitForPeer(app.db, app.log);
427428 for (const signal of earlySignals) process.off(signal, leaveEarly);
429 // No shell outlives its forge: a row a stop or crash left open is closed.
430 // After the peer lets go, never in `serve`: a draining forge holds the write
431 // lock (the UPDATE would stall the loop) and its own shells are still live.
432 closeStaleSessions(app.db);
428433 // A repository's own check, on every push a pull or the default branch takes.
429434 const checks =
430435 process.env.FORGE_CHECKS === "0"
test/crew-machine-tools.test.ts added +174
@@ -0,0 +1,174 @@
1 /**
2 * The security machine carries its tools installed, not just named (#9470):
3 * a verified install is on the machine's PATH and re-allowed in its
4 * sandbox; `standard` neither sees nor finds it; a tool that is not there
5 * is not claimed.
6 */
7 import { afterAll, expect, test } from "bun:test";
8 import { appendFileSync, chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { machineNote, SECURITY, STANDARD } from "../src/crew/machines.ts";
12 import {
13 kitPath,
14 lockKey,
15 machineKit,
16 parseLock,
17 LOCK_PATH,
18 readLock,
19 sha256Hex,
20 TOOL_PINS,
21 toolDir,
22 toolsRoot,
23 UNPINNED,
24 unsafeMember,
25 VERIFIED_FILE,
26 } from "../src/crew/machine_tools.ts";
27 import { sandboxPolicy } from "../src/crew/sandbox.ts";
28 import { TOOLS } from "../src/crew/machines.ts";
29
30 const data = mkdtempSync(join(tmpdir(), "machine-tools-"));
31 afterAll(() => rmSync(data, { recursive: true, force: true }));
32 mkdirSync(join(data, "crew", "scout"), { recursive: true });
33
34 const SUM = "a".repeat(64);
35 const pin = TOOL_PINS["gitleaks"]!;
36 const lock = { [lockKey("gitleaks", pin)]: SUM };
37
38 /** A stand-in gitleaks, installed as the installer leaves one. */
39 function install(sum: string): string {
40 const dir = toolDir(data, "gitleaks", pin);
41 mkdirSync(dir, { recursive: true });
42 const exe = join(dir, pin.exe);
43 writeFileSync(exe, "#!/bin/sh\necho \"v8.18.4\"\n");
44 chmodSync(exe, 0o755);
45 const record = { archive: sum, exe: sha256Hex(readFileSync(exe)), size: statSync(exe).size };
46 writeFileSync(join(dir, VERIFIED_FILE), JSON.stringify(record));
47 return dir;
48 }
49
50 test("every tool a machine can name is pinned to a versioned release or left out with a reason", () => {
51 for (const tool of Object.keys(TOOLS)) {
52 const p = TOOL_PINS[tool];
53 if (p === undefined) {
54 expect(UNPINNED[tool]).toBeTruthy();
55 continue;
56 }
57 expect(p.url).toStartWith("https://");
58 expect(p.url).toContain(p.version);
59 expect(p.url).not.toContain("latest");
60 }
61 // The forge's own lock parses; a malformed sum is no pin.
62 expect(typeof readLock()).toBe("object");
63 expect(parseLock('{"gitleaks@1":"nope","trivy@2":"' + SUM + '"}')).toEqual({ "trivy@2": SUM });
64 expect(parseLock("not json")).toEqual({});
65 });
66
67 test("a tool is claimed only when its install holds the lock's sum", () => {
68 expect(machineKit(data, SECURITY, lock).present).toEqual([]);
69 install("b".repeat(64));
70 expect(machineKit(data, SECURITY, lock).present).toEqual([]);
71 expect(machineKit(data, SECURITY, {}).present).toEqual([]);
72 const dir = install(SUM);
73 const kit = machineKit(data, SECURITY, lock);
74 expect(kit.present).toEqual([{ tool: "gitleaks", dir, binDir: join(dir, ".") }]);
75 expect(kit.missing.map((m) => m.tool)).toEqual(SECURITY.tools.filter((t) => t !== "gitleaks"));
76 expect(machineKit(data, STANDARD, lock)).toEqual({ present: [], missing: [] });
77 // The brief claims what is there and names what is not.
78 const note = machineNote(SECURITY, kit)!;
79 expect(note).toContain("gitleaks (");
80 expect(note).toContain("Not installed here");
81 expect(note).toMatch(/do not count on them: .*zap/);
82 });
83
84 test("the sandbox's mounts for each machine: security reads its tools, standard none, secrets stay denied", () => {
85 const dir = install(SUM);
86 const policyFor = (machine: typeof SECURITY) =>
87 sandboxPolicy({
88 dataDir: data,
89 agentName: "scout",
90 urls: ["http://127.0.0.1:3000"],
91 toolDirs: machineKit(data, machine, lock).present.map((p) => p.dir),
92 });
93 const standard = policyFor(STANDARD);
94 const security = policyFor(SECURITY);
95 for (const policy of [standard, security]) {
96 expect(policy.denyRead).toContain(toolsRoot(data));
97 for (const secret of ["secret.key", "forge.db", "repos", "ssh"])
98 expect(policy.denyRead).toContain(join(data, secret));
99 expect(policy.readAllow.some((p) => p.endsWith("secret.key"))).toBe(false);
100 }
101 expect(standard.readAllow.some((p) => p.startsWith(toolsRoot(data)))).toBe(false);
102 expect(security.readAllow).toContain(dir);
103 // Reach is unchanged by the tools: the same hosts either way.
104 expect(security.allowHosts).toEqual(standard.allowHosts);
105 });
106
107 test("a run in security finds `gitleaks version` on PATH; one in standard cannot", async () => {
108 install(SUM);
109 const base = "/usr/bin:/bin";
110 const run = async (machine: typeof SECURITY) => {
111 const PATH = kitPath(machineKit(data, machine, lock), base);
112 const found = Bun.which("gitleaks", { PATH });
113 if (found === null) return null;
114 const p = Bun.spawn([found, "version"], { env: { PATH }, stdout: "pipe" });
115 return (await new Response(p.stdout).text()).trim();
116 };
117 expect(await run(SECURITY)).toBe("v8.18.4");
118 expect(await run(STANDARD)).toBeNull();
119 });
120
121 test("an executable changed after install is not claimed, even with the record in place", () => {
122 const dir = install(SUM);
123 expect(machineKit(data, SECURITY, lock).present.map((p) => p.tool)).toEqual(["gitleaks"]);
124 // Same size, other bytes: the sum catches it.
125 const exe = join(dir, pin.exe);
126 writeFileSync(exe, "#!/bin/sh\necho \"v6.66.6\"\n");
127 const kit = machineKit(data, SECURITY, lock);
128 expect(kit.present).toEqual([]);
129 expect(kit.missing.find((m) => m.tool === "gitleaks")?.why).toContain("changed since it was installed");
130 // Longer: the size catches it.
131 install(SUM);
132 appendFileSync(exe, "rm -rf /\n");
133 expect(machineKit(data, SECURITY, lock).present).toEqual([]);
134 // A record that is not the installer's is no install.
135 install(SUM);
136 writeFileSync(join(dir, VERIFIED_FILE), `${SUM}\n`);
137 expect(machineKit(data, SECURITY, lock).present).toEqual([]);
138 });
139
140 test("an archive member is unpacked only as a plain file or directory inside the install", () => {
141 expect(unsafeMember("gitleaks", "-")).toBeUndefined();
142 expect(unsafeMember("testssl.sh-3.0.9/etc", "d")).toBeUndefined();
143 expect(unsafeMember("./README.md", "-")).toBeUndefined();
144 for (const [name, type] of [
145 ["/etc/passwd", "-"],
146 ["../../secret.key", "-"],
147 ["a/../../b", "-"],
148 ["..", "d"],
149 ["link", "l"],
150 ["hard", "h"],
151 ["dev", "c"],
152 ["a\\..\\b", "-"],
153 ["", "-"],
154 ] as const)
155 expect(unsafeMember(name, type)).toBeString();
156 });
157
158 test("the lock pins every tool or none, and an empty lock is said out loud", () => {
159 const keys = Object.entries(TOOL_PINS).map(([tool, p]) => lockKey(tool, p)).sort();
160 const sums = readLock();
161 const raw = JSON.parse(readFileSync(LOCK_PATH, "utf8")) as Record<string, unknown>;
162 // Every entry is a valid sum for a pinned tool at its pinned version.
163 expect(Object.keys(sums).sort()).toEqual(Object.keys(raw).sort());
164 for (const key of Object.keys(sums)) expect(keys).toContain(key);
165 const doc = readFileSync(join(import.meta.dir, "..", "docs", "features", "machines.md"), "utf8");
166 if (Object.keys(sums).length === 0) {
167 // Shipping no sums is a declared state: the docs say nothing installs.
168 expect(doc).toContain("The lock ships empty");
169 } else {
170 // Once any sum lands, every pinned tool has one and the docs stop saying it is empty.
171 expect(Object.keys(sums).sort()).toEqual(keys);
172 expect(doc).not.toContain("The lock ships empty");
173 }
174 });
test/crew-machines.test.ts +13 −3
@@ -82,14 +82,24 @@ describe("machines", () => {
8282
8383 test("a machine's environment carries names, never a secret", () => {
8484 expect(machineEnv(SECURITY).FORGE_MACHINE).toBe("security");
85 expect(machineEnv(SECURITY).FORGE_MACHINE_TOOLS).toContain("gitleaks");
85 // Names what is installed, not what was declared (#9470).
86 expect(machineEnv(SECURITY).FORGE_MACHINE_TOOLS).toBe("");
87 expect(
88 machineEnv(SECURITY, { present: [{ tool: "gitleaks", dir: "/d", binDir: "/d" }], missing: [] })
89 .FORGE_MACHINE_TOOLS,
90 ).toBe("gitleaks");
8691 expect(Object.keys(machineEnv(SECURITY)).sort()).toEqual(["FORGE_MACHINE", "FORGE_MACHINE_TOOLS"]);
8792 });
8893
8994 test("the brief names the tools a machine carries, and says nothing for the standard one", () => {
9095 expect(machineNote(STANDARD)).toBeUndefined();
91 expect(machineNote(SECURITY)).toContain("gitleaks");
92 expect(machineNote(SECURITY)).toContain("forge's own copies");
96 expect(machineNote(SECURITY)).toContain("None of its tools is installed");
97 expect(machineNote(SECURITY)).not.toContain("On your PATH");
98 const kit = { present: [{ tool: "gitleaks", dir: "/d", binDir: "/d" }], missing: [] };
99 expect(machineNote(SECURITY, kit)).toContain("On your PATH");
100 expect(machineNote(SECURITY, kit)).toContain("gitleaks (");
101 expect(machineNote(SECURITY, kit)).toContain("forge's own copies");
102 expect(machineNote(SECURITY, kit)).toContain("do not count on them: trivy");
93103 });
94104 });
95105
test/machine-shell.test.ts +3 −1
@@ -57,8 +57,10 @@ beforeAll(async () => {
5757 repo = made.value;
5858 const shut = await createRepo(app.db, app.reposRoot, { ownerId: alice.id, name: "shut", visibility: "private" });
5959 if (!shut.ok) throw new Error(String(shut.error));
60 // A row a crashed forge left open, closed when this one boots.
60 // A row a crashed forge left open, closed when this one boots: main.ts
61 // does it once any draining peer has let go, before work starts.
6162 staleRow = recordStart(app.db, repo.id, "standard", alice.id);
63 closeStaleSessions(app.db);
6264 servers = await serve(app);
6365 base = servers.url.replace(/^http/, "ws");
6466 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs