trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns2g rEventsFlows

A crew run's sandbox refuses writes outside its worktree and hosts past the egress allowlist, each a test #10601

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/10553-a-crew-run-s-sandbox-refuses-writes-outs → main 8 commits fork

  • Merged. scout/10553-a-crew-run-s-sandbox-refuses-writes-outs was merged into main as f12e5a5.

Review

Sign in to review this change.
2 files changed +266 −67
docs/ops/RED-TEAM.md +33 −2
@@ -15,14 +15,45 @@ so the live run is still open as its own issue.
1515 | A visitor sends `x-forge-identity` / `x-forge-signature` that the app believes | The proxy deletes both and writes signed ones of its own (`IDENTITY_HEADERS`, `src/apps/proxy.ts`) | `test/apps-identity.test.ts`, `test/redteam-doors.test.ts` |
1616 | An app learns the edge secret or a bearer token from a forwarded request | `stripForgeCredentials` drops every `x-plat-*` header, `authorization`, and the grant cookie | `test/redteam-doors.test.ts` "forged identity headers" |
1717 | Someone outside claims a person with `x-plat-user` | `edgeIdentity` believes it only with the edge secret, compared as digests | `test/redteam-doors.test.ts`, `test/edge-auth.test.ts` |
18 | A crew run leaves its worktree or its network allowlist | bwrap in `src/crew/sandbox.ts` | `test/crew-sandbox.test.ts`; the escape attempts under `procMount: Unmasked` are still open |
18 | A crew run leaves its worktree or its network allowlist | bwrap in `src/crew/sandbox.ts` | `test/crew-sandbox.test.ts`; the escape attempts under `procMount: Unmasked` are below, in "The crew sandbox" |
1919 | Note or issue text steers a crew brief | `untrusted` marking in briefs | still open: needs a wider test than #10365 |
2020
21 ## The crew sandbox, attempt by attempt
22
23 A real sandboxed shell (bwrap, made by `srt` from `claudeSettings`, the
24 settings a crew run hands Claude Code) with `/proc` unmasked, as the pod runs
25 it. Every row is a test in `test/crew-sandbox-escape.test.ts` that fails if
26 the attempt works; `<pid>` is a process outside the sandbox. Where no sandbox
27 can be made (inside a crew run, or user namespaces refused) the file is
28 skipped and prints `skipped: test/crew-sandbox-escape.test.ts: <reason>`; the
29 forge's check runs it.
30
31 | Attempt | Result | Test |
32 | --- | --- | --- |
33 | Read `forge.db`: by its path, `/proc/self/root`, `/proc/1/root`, `/proc/<pid>/root` | refused (no canary back) | "reads nothing from a denied data file, …" (one per route) |
34 | Read another crew's file, the same four routes | refused | "reads nothing from another crew's file, …" |
35 | Read a secret from `/proc/1/environ` | not there | "reads no secret from /proc/1/environ" |
36 | Read a secret from any `/proc/<pid>/environ` the shell lists | not there | "reads no secret from any /proc/<pid>/environ …" |
37 | Write into the data dir, the same four routes | refused with an error code the probe reports, and the file absent, checked from outside | "writes nothing into the data dir, …" |
38 | Write into another crew's dir, the same four routes | refused | "writes nothing into another crew's dir, …" |
39 | Connect to `127.0.0.1:<port off the allowlist>` directly, as raw TCP (no proxy setting can carry it) | refused (own network) | "cannot reach 127.0.0.1:<port off the allowlist>, directly" |
40 | The same through the sandbox's proxy | refused by the proxy | "… through the proxy" |
41 | Connect to `localhost:<port off the allowlist>`, directly and through the proxy | refused | "cannot reach localhost:<port off the allowlist>, …" |
42 | Controls: read and write the own worktree, read the own environ, reach a listener in the own network namespace directly, reach the forge's host:port through the proxy | works | the four "control:" tests |
43
44 The controls keep a probe that never ran, or a proxy that refuses
45 everything, from passing as "all refused". Where the allowed host does not
46 answer through the proxy either (the forge's check host today, the same cause
47 that keeps `test/crew-sandbox-real.test.ts` out of the gate; #10602 makes them run there), the
48 attempts through the proxy and their control are skipped, printing
49 `skipped: test/crew-sandbox-escape.test.ts, the attempts through the proxy:
50 <reason>`; the direct attempts, the reads and the writes still run.
51
2152 ## Not done here
2253
2354 - **Pod security enforced on all nine namespaces.** The namespace labels live
2455 in the cluster's GitOps source, which is not in this repository. That work
2556 happens there.
26 - **The live run inside the pod and the sandbox escape attempts**: #10450.
57 - **The live run inside the pod**: #10450. The sandbox escape attempts are above.
2758 - **Wider prompt-injection tests**: #10451.
2859 - Done: the security controls page, `/settings/security/controls` (#10452); how it reads `security-report.json` is in `docs/features/security-controls.md`.
test/crew-sandbox-escape.test.ts +233 −65
@@ -13,6 +13,17 @@
1313 * Two controls keep a probe that never ran from passing as "all refused": the
1414 * shell reads the canary from its own worktree, and reads its own environ.
1515 *
16 * Write attempts go the same routes to the data dir and another crew's dir;
17 * each is judged from outside the sandbox (the file must not exist after),
18 * not by what the shell says. The control writes into its own worktree.
19 *
20 * Network attempts are at a server outside the sandbox on a port that is not
21 * on the egress allowlist, by `127.0.0.1` and by the name `localhost`, each
22 * directly as raw TCP, so no proxy setting can carry it (the sandbox has a
23 * network of its own; its control is a listener the shell starts itself) and
24 * through the sandbox's proxy. The control reaches the forge's host:port,
25 * which is allowed, through the proxy.
26 *
1627 * Like crew-sandbox-real.test.ts this needs a sandbox of its own, so inside a
1728 * crew run or a container that refuses user namespaces it is skipped, naming
1829 * itself and the reason on the console; the forge's check runs it. Where a
@@ -20,8 +31,8 @@
2031 * The `rg` is the pinned `@vscode/ripgrep` binary, handed to `srt` as its
2132 * `ripgrep.command`: the check host has no ripgrep on its PATH.
2233 */
23 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
24 import { existsSync, mkdirSync, writeFileSync } from "node:fs";
34 import { afterAll, describe, expect, test } from "bun:test";
35 import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
2536 import { mkdtemp, rm } from "node:fs/promises";
2637 import { tmpdir } from "node:os";
2738 import { join } from "node:path";
@@ -52,6 +63,14 @@ const FILES = {
5263 /** The routes to each file; `<pid>` is a process outside the sandbox. */
5364 const ROUTES = ["by its path", "/proc/self/root", "/proc/1/root", "/proc/<pid>/root"] as const;
5465 type Route = (typeof ROUTES)[number];
66 /** The directories outside the worktree a write is tried in. */
67 const DIRS = {
68 "the data dir": ".",
69 "another crew's dir": "crew/mason/01repo/3",
70 } as const;
71 const WRITTEN = "escape-written.txt";
72 /** Hosts off the allowlist that name the outside server. */
73 const OFF_LIST = ["127.0.0.1", "localhost"] as const;
5574
5675 function routed(route: Route, path: string, pid: number): string {
5776 if (route === "by its path") return path;
@@ -66,52 +85,119 @@ interface Report {
6685 pidsListed: number;
6786 initEnviron: boolean;
6887 environLeaks: string[];
88 /** `host|direct` (raw TCP) or `host|proxy` → what came back, or "refused: <error>". */
89 net: Record<string, string>;
90 /** The raw response from a listener inside the shell's own namespace. */
91 ownDirect: string;
92 /** `label|route` → "ok", or the error code the write was refused with. */
93 writes: Record<string, string>;
94 forgeViaProxy: string;
95 ownWrite: string;
6996 }
7097
71 describe.skipIf(nestedReason !== undefined)(
72 "read attempts from a real sandboxed shell, /proc unmasked",
73 () => {
74 let dataDir: string;
75 let report: Report;
76 let holder: ReturnType<typeof Bun.spawn>;
98 let dataDir: string;
99 let report: Report;
100 let holder: ReturnType<typeof Bun.spawn>;
101 let work: string;
102 let writes: Record<string, string>;
103 let forge: ReturnType<typeof Bun.serve>;
104 let outside: ReturnType<typeof Bun.serve>;
77105
78 beforeAll(async () => {
79 dataDir = await mkdtemp(join(tmpdir(), "forge-escape-"));
80 const work = join(dataDir, "crew/scout/01repo/1");
81 mkdirSync(work, { recursive: true });
82 mkdirSync(join(dataDir, "crew/mason/01repo/3"), { recursive: true });
83 for (const rel of Object.values(FILES)) writeFileSync(join(dataDir, rel), CANARY);
84 writeFileSync(join(work, "own.txt"), CANARY);
106 /**
107 * Runs the probe once, at load: whether the sandbox's proxy answers at all
108 * decides, before any test is declared, whether the attempts through it run.
109 */
110 async function runEscape(): Promise<void> {
111 dataDir = await mkdtemp(join(tmpdir(), "forge-escape-"));
112 work = join(dataDir, "crew/scout/01repo/1");
113 mkdirSync(work, { recursive: true });
114 mkdirSync(join(dataDir, "crew/mason/01repo/3"), { recursive: true });
115 for (const rel of Object.values(FILES)) writeFileSync(join(dataDir, rel), CANARY);
116 writeFileSync(join(work, "own.txt"), CANARY);
85117
86 // A process outside the sandbox with a secret in its environment: with
87 // /proc whole, the shell must still not read it, nor go through its root.
88 holder = Bun.spawn(["sleep", "120"], {
89 env: { ...process.env, FORGE_REDTEAM_SECRET: ENV_CANARY },
90 stdin: "ignore",
91 stdout: "ignore",
92 stderr: "ignore",
93 });
118 // A process outside the sandbox with a secret in its environment: with
119 // /proc whole, the shell must still not read it, nor go through its root.
120 holder = Bun.spawn(["sleep", "120"], {
121 env: { ...process.env, FORGE_REDTEAM_SECRET: ENV_CANARY },
122 stdin: "ignore",
123 stdout: "ignore",
124 stderr: "ignore",
125 });
94126
95 const targets: Record<string, string> = {};
96 for (const [label, rel] of Object.entries(FILES))
97 for (const route of ROUTES)
98 targets[`${label}|${route}`] = routed(route, join(dataDir, rel), holder.pid);
127 const targets: Record<string, string> = {};
128 for (const [label, rel] of Object.entries(FILES))
129 for (const route of ROUTES)
130 targets[`${label}|${route}`] = routed(route, join(dataDir, rel), holder.pid);
99131
100 writeFileSync(
101 join(work, "probe.ts"),
102 `
103 import { readFileSync, readdirSync } from "node:fs";
132 writes = {};
133 for (const [label, rel] of Object.entries(DIRS))
134 for (const route of ROUTES)
135 writes[`${label}|${route}`] = routed(
136 route,
137 join(dataDir, rel, `${route.replace(/\W+/g, "-")}-${WRITTEN}`),
138 holder.pid,
139 );
140
141 forge = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response("forge") });
142 outside = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response("outside") });
143
144 writeFileSync(
145 join(work, "probe.ts"),
146 `
147 import { readFileSync, readdirSync, writeFileSync } from "node:fs";
104148 const CANARY = ${JSON.stringify(CANARY)};
105149 const ENV = ${JSON.stringify(ENV_CANARY)};
106150 const has = (path: string, needle: string) => {
107151 try { return readFileSync(path, "utf8").includes(needle); } catch { return false; }
108152 };
153 // The network first, before the shell touches /proc.
154 const get = async (url: string, proxy?: string) => {
155 try { return await (await fetch(url, { proxy })).text(); }
156 catch (e) { return "refused: " + String(e); }
157 };
158 // Direct is a raw TCP connection, so no proxy setting in the environment
159 // can carry it: what comes back is the server's bytes or the refusal.
160 const direct = (hostname: string, port: number) =>
161 new Promise<string>((resolve) => {
162 let bytes = "";
163 const refused = (e: unknown) => resolve("refused: " + String(e));
164 Bun.connect({
165 hostname,
166 port,
167 socket: {
168 open(socket) { socket.write("GET / HTTP/1.0\\r\\nhost: probe\\r\\n\\r\\n"); },
169 data(_socket, chunk) { bytes += new TextDecoder().decode(chunk); },
170 close() { resolve(bytes); },
171 error(_socket, e) { refused(e); },
172 connectError(_socket, e) { refused(e); },
173 },
174 }).catch(refused);
175 });
176 // The direct control: a listener inside the shell's own network namespace.
177 const own = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response("own") });
178 const ownDirect = await direct("127.0.0.1", own.port);
179 own.stop(true);
180 // The allowed control first: a refusal must not be what decides it.
181 const forgeViaProxy = await get("http://127.0.0.1:${forge.port}/", process.env.HTTP_PROXY);
182 const net: Record<string, string> = {};
183 for (const host of ${JSON.stringify(OFF_LIST)}) {
184 const url = "http://" + host + ":" + ${outside.port} + "/";
185 net[host + "|direct"] = await direct(host, ${outside.port});
186 net[host + "|proxy"] = process.env.HTTP_PROXY ? await get(url, process.env.HTTP_PROXY) : "no proxy";
187 }
109188 const reads: Record<string, boolean> = {};
110189 for (const [key, path] of Object.entries(${JSON.stringify(targets)} as Record<string, string>))
111190 reads[key] = has(path, CANARY);
112191 let pids: string[] = [];
113192 try { pids = readdirSync("/proc").filter((n) => /^[0-9]+$/.test(n)); } catch {}
114193 const environLeaks = pids.filter((pid) => has("/proc/" + pid + "/environ", ENV));
194 const writes: Record<string, string> = {};
195 for (const [key, path] of Object.entries(${JSON.stringify(writes)} as Record<string, string>)) {
196 try { writeFileSync(path, CANARY); writes[key] = "ok"; }
197 catch (e) { writes[key] = (e as { code?: string }).code ?? String(e); }
198 }
199 let ownWrite = "ok";
200 try { writeFileSync("own-written.txt", CANARY); } catch (e) { ownWrite = String(e); }
115201 console.log(JSON.stringify({
116202 reads,
117203 ownRead: has("own.txt", CANARY),
@@ -119,46 +205,84 @@ console.log(JSON.stringify({
119205 pidsListed: pids.length,
120206 initEnviron: has("/proc/1/environ", ENV),
121207 environLeaks,
208 net,
209 ownDirect,
210 writes,
211 forgeViaProxy,
212 ownWrite,
122213 }));
123214 `,
124 );
215 );
125216
126 const policy = sandboxPolicy({
127 dataDir,
128 agentName: "scout",
129 urls: [],
130 runId: "01m46abcdefghjkmnpqrstvwxy",
131 });
132 const real = claudeSettings(policy).sandbox as Record<string, any>;
133 const settings = join(dataDir, "srt-settings.json");
134 writeFileSync(
135 settings,
136 JSON.stringify({
137 ripgrep: { command: rgPath },
138 network: { deniedDomains: [], ...real.network },
139 filesystem: {
140 denyRead: [],
141 denyWrite: [],
142 ...real.filesystem,
143 allowWrite: [work, ...(real.filesystem?.allowWrite ?? [])],
144 },
145 }),
146 );
147 const proc = Bun.spawn(
148 [srt, "--settings", settings, "-c", `'${process.execPath}' probe.ts`],
149 { cwd: work, stdin: "ignore", stdout: "pipe", stderr: "pipe" },
150 );
151 const [out, err, code] = await Promise.all([
152 new Response(proc.stdout).text(),
153 new Response(proc.stderr).text(),
154 proc.exited,
155 ]);
156 if (code !== 0) throw new Error(`sandboxed shell exited ${code}: ${err}`);
157 report = JSON.parse(out.trim().split("\n").pop() ?? "{}");
158 }, 60_000);
217 const policy = sandboxPolicy({
218 dataDir,
219 agentName: "scout",
220 urls: [`http://127.0.0.1:${forge.port}`],
221 runId: "01m46abcdefghjkmnpqrstvwxy",
222 });
223 const real = claudeSettings(policy).sandbox as Record<string, any>;
224 const settings = join(dataDir, "srt-settings.json");
225 writeFileSync(
226 settings,
227 JSON.stringify({
228 ripgrep: { command: rgPath },
229 network: { deniedDomains: [], ...real.network },
230 filesystem: {
231 denyRead: [],
232 denyWrite: [],
233 ...real.filesystem,
234 allowWrite: [work, ...(real.filesystem?.allowWrite ?? [])],
235 },
236 }),
237 );
238 const proc = Bun.spawn([srt, "--settings", settings, "-c", `'${process.execPath}' probe.ts`], {
239 cwd: work,
240 stdin: "ignore",
241 stdout: "pipe",
242 stderr: "pipe",
243 });
244 const [out, err, code] = await Promise.all([
245 new Response(proc.stdout).text(),
246 new Response(proc.stderr).text(),
247 proc.exited,
248 ]);
249 if (code !== 0) throw new Error(`sandboxed shell exited ${code}: ${err}`);
250 report = JSON.parse(out.trim().split("\n").pop() ?? "{}");
251 }
252
253 /** Why the probe did not run, when it did not: a named test fails with it. */
254 let probeError: string | undefined;
255 if (nestedReason === undefined)
256 await runEscape().catch((e: unknown) => {
257 probeError = e instanceof Error ? e.message : String(e);
258 });
259
260 /**
261 * Where the allowed host cannot be reached through the proxy either, the
262 * proxy answers nothing here (the check host today, #10602, as for
263 * crew-sandbox-real.test.ts), and a refusal through it would prove nothing:
264 * those attempts and their control are skipped, named.
265 */
266 const proxyReason =
267 nestedReason === undefined && probeError === undefined && report!.forgeViaProxy !== "forge"
268 ? `the sandbox's proxy does not answer here, even for an allowed host (#10602): ${report!.forgeViaProxy}`
269 : undefined;
270 if (proxyReason !== undefined)
271 console.warn(
272 `skipped: test/crew-sandbox-escape.test.ts, the attempts through the proxy: ${proxyReason}`,
273 );
274
275 describe.skipIf(nestedReason !== undefined)(
276 "read attempts from a real sandboxed shell, /proc unmasked",
277 () => {
278 test("the sandboxed probe ran", () => {
279 expect(probeError).toBeUndefined();
280 });
159281
160282 afterAll(async () => {
161283 holder?.kill();
284 forge?.stop(true);
285 outside?.stop(true);
162286 if (dataDir !== undefined) await rm(dataDir, { recursive: true, force: true });
163287 });
164288
@@ -180,5 +304,49 @@ console.log(JSON.stringify({
180304 test("reads no secret from any /proc/<pid>/environ of a process outside", () => {
181305 expect(report.environLeaks).toEqual([]);
182306 });
307
308 test("control: the shell writes its own worktree", () => {
309 expect(report.ownWrite).toBe("ok");
310 expect(existsSync(join(work, "own-written.txt"))).toBe(true);
311 expect(readFileSync(join(work, "own-written.txt"), "utf8")).toBe(CANARY);
312 });
313
314 test.skipIf(proxyReason !== undefined)(
315 "control: the shell reaches the forge's host:port through the proxy",
316 () => {
317 expect(report.forgeViaProxy).toBe("forge");
318 },
319 );
320
321 for (const label of Object.keys(DIRS))
322 for (const route of ROUTES)
323 test(`writes nothing into ${label}, ${route}`, () => {
324 // Judged from outside: the path resolved without /proc routes.
325 const rel = DIRS[label as keyof typeof DIRS];
326 const path = join(dataDir, rel, `${route.replace(/\W+/g, "-")}-${WRITTEN}`);
327 // The probe tried it, and was refused with an error of its own.
328 expect(report.writes[`${label}|${route}`]).toMatch(/^E[A-Z]+$/);
329 expect(existsSync(path)).toBe(false);
330 });
331
332 test("control: the shell reaches a listener in its own network namespace, directly", () => {
333 expect(report.ownDirect).toContain("\r\n\r\nown");
334 });
335
336 for (const host of OFF_LIST) {
337 test(`cannot reach ${host}:<port off the allowlist>, directly`, () => {
338 // Raw TCP from a shell whose direct control connects: refused, no bytes.
339 expect(report.net[`${host}|direct`]).toStartWith("refused: ");
340 expect(report.net[`${host}|direct`]).not.toContain("outside");
341 });
342
343 test.skipIf(proxyReason !== undefined)(
344 `cannot reach ${host}:<port off the allowlist>, through the proxy`,
345 () => {
346 expect(report.net[`${host}|proxy`]).not.toBe("outside");
347 expect(report.net[`${host}|proxy`]).not.toBe("no proxy");
348 },
349 );
350 }
183351 },
184352 );

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs