Public: anyone can read this; the crew can change it.

A check run as a cluster job holds the same egress wall as one on the box #13272

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13251-a-check-run-as-a-cluster-job-holds-the-s → main 6 commits fork 3 threads · 3 unresolved

  • Merged. scout/13251-a-check-run-as-a-cluster-job-holds-the-s was merged into main as fc1f404.

Review

Sign in to review this change.
14 files changed +518 −115
docs/APPS.md +7 −4
@@ -913,10 +913,12 @@ records `placement` and the `node` its pod ran on. A malformed setting leaves
913913 `process` in place. No cell wraps a Job. The pod mounts only its own
914914 checkout (`subPath` = the run id), mounts no service-account token, runs
915915 non-root without privilege escalation, and carries the check's limits as
916 `resources.limits`. **Egress is not enforced by the forge on a Job**: the
917 run is recorded as uncontained ("egress not enforced on a cluster job"), and
918 the pod carries the label `forge-check=true` so an operator's NetworkPolicy
919 can select it and hold what `egress` declares.
916 `resources.limits`. **Egress is held as in a cell**: the forge opens its
917 egress proxy for the run on `FORGE_CHECKS_EGRESS_GATE` and points the pod's
918 `HTTPS_PROXY` at it, so an undeclared host is refused in the forge's own
919 words, and the run is recorded as enforced; the pod carries the label
920 `forge-check=true` for the NetworkPolicy that leaves it no other way out
921 (docs/CLUSTER.md). Without a gate no check runs as a Job.
920922
921923 Checks share `FORGE_CHECKS_CONCURRENCY` slots (default `1`), and when one
922924 frees, it goes to the most recently requested check waiting, not the one
@@ -1811,6 +1813,7 @@ a link to the milestone (`docs/PLANNING.md`); its progress never does.
18111813 | `FORGE_CREW_CONCURRENCY` | 1 per 3 GiB above 2 GiB, 2–16 | agent runs at once (live: Settings → Capacity) |
18121814 | `FORGE_CHECKS_CONCURRENCY` | `1` | checks at once (live: Settings → Capacity) |
18131815 | `FORGE_CHECKS_PLACEMENT` | `process` | `job` runs a check's commands as a cluster Job (see Placement above) |
1816 | `FORGE_CHECKS_EGRESS_GATE` | unset | the address a Job's egress gate listens on; without it no check runs as a Job |
18141817 | `FORGE_CHECKS_DRAIN_S` | `10` | seconds a roll lets running checks finish first |
18151818 | `FORGE_CREW_DRAIN_S` | `10` | seconds a roll lets crew runs finish before they are queued again |
18161819 | `FORGE_CHECK_SHARDS` | the machine's cores | test shards per check (live: Settings → Capacity) |
docs/CLUSTER.md +32 −13
@@ -50,31 +50,50 @@ Read once at boot by `src/checks/placement.ts`. A malformed value leaves
5050 | `FORGE_CHECKS_CLAIM` | `forge-checks` | a ReadWriteMany claim holding the checks root; required |
5151 | `FORGE_CHECKS_NODE_SELECTOR` | `pool=checks,disk=ssd` | `key=value,key=value`; optional |
5252 | `FORGE_CHECKS_TOLERATIONS` | `[{"key":"dedicated","operator":"Equal","value":"checks","effect":"NoSchedule"}]` | JSON list, at most 20; optional |
53 | `FORGE_CHECKS_EGRESS_GATE` | `10.42.0.17` (the forge pod's IP) | the address the forge's per-run egress gate listens on and a Job's pod dials; without it no check runs as a Job |
5354
54 ## Only while egress is reported
55 ## Egress through the forge's gate
5556
56 The lane applies only while `cells.egress` is `report`. Under `enforce` (the
57 default) a Job cannot hold egress, so every check stays in a cell: the run
58 records `placement: process` and its log says
59 `placement: job skipped, egress is enforced and a Job cannot hold it`. The
60 setting is read on every run, so an admin's change takes effect on the next
61 check.
57 A Job holds the same egress wall as a cell (#13251). For each run the forge
58 opens the proxy a cell's only way out is (`apps/egress.ts`, through
59 `src/checks/job_gate.ts`) on `FORGE_CHECKS_EGRESS_GATE`, at the first free
60 port of 47100-47199 (`JOB_GATE_PORTS`: one per Job running at once), and
61 gives the pod `HTTP_PROXY`/`HTTPS_PROXY` pointing at it (`NO_PROXY` empty).
62 The proxy URL carries a secret made for that run; a request to the gate
63 without it as `Proxy-Authorization` is answered `407` and dials nothing, so
64 another pod that reaches the port cannot use the forge as its way out. It forwards to the forge, the registry and what `forge.json`
65 `egress` declares, and refuses anything else with `403` and
66 `refused by this forge: <host> is not in this app's forge.json egress`; an
67 outside contributor's check is enforced whatever `cells.egress` says. The
68 run records the gate with a note (`JOB_GATE_NOTE`) that the rest rests on
69 the cluster's NetworkPolicy, which the forge cannot see, and its page says
70 so. The gate closes when the Job ends.
6271
63 ## Label for a NetworkPolicy
64
65 The forge does not enforce egress on a Job. Every check pod carries the label
66 `forge-check: "true"` (`JOB_LABELS` in `src/checks/job_step.ts`), so a
67 NetworkPolicy can select it and hold what a check declares:
72 The variables only point the way: the wall is a NetworkPolicy that lets a
73 check pod reach the forge pod on the gate's ports and nothing else. Name the
74 range, never the whole pod: on any port a check pod could dial the forge's
75 own listeners around the gate. Keep nothing else of the forge pod listening
76 in 47100-47199. Every check pod carries the label `forge-check: "true"`
77 (`JOB_LABELS` in `src/checks/job_step.ts`):
6878
6979 ```yaml
7080 podSelector:
7181 matchLabels: { forge-check: "true" }
82 policyTypes: [Egress]
83 egress:
84 - to: [{ podSelector: { matchLabels: { app: forge } } }]
85 ports: [{ protocol: TCP, port: 47100, endPort: 47199 }]
7286 ```
7387
88 Set `FORGE_CHECKS_EGRESS_GATE` only once that policy is in place. Without
89 it the cluster cannot honour `egress`, so no check runs as a Job: each stays
90 in a cell, records `placement: process`, and its log says
91 `placement: job skipped, this cluster names no egress gate for a Job (FORGE_CHECKS_EGRESS_GATE)`.
92
7493 ## Confirming the lane is on
7594
7695 1. Push to a pull, or re-run its check.
77 2. With `cells.egress` at `report`, the run's page says "ran as a cluster job on <node>"; a `process` run says
96 2. The run's page says "ran as a cluster job on <node>"; a `process` run says
7897 nothing.
7998 3. In the cluster, `kubectl get jobs,pods -l forge-check=true` shows the Job
8099 while it runs (it is deleted when the run ends).
src/apps/contain.ts +61 −20
@@ -57,7 +57,10 @@ import { CELL_MEMORY_FLOOR_MIB, CELL_MEMORY_RESERVE_MIB, cellMemoryCeilingMiB }
5757 import { cpuWatch } from "./cpu_rule.ts";
5858 import { dutyTick, newDuty, resumeNow, type Duty, type DutyLater, type Signaller } from "./cpu_duty.ts";
5959
60 /** `ceiling`: a declaration was cut to the machine's maximum; `declared`: forge.json's own; `default`: neither. */
60 /**
61 * `ceiling`: a declaration was cut to the machine's maximum; `declared`: forge.json's own;
62 * `default`: neither.
63 */
6164 export type CeilingFrom = "env" | "pod" | "default" | "size";
6265 export type MemorySource = "ceiling" | "declared" | "default";
6366
@@ -112,7 +115,10 @@ export const OWN_SIZE_SHARE = 3 / 4;
112115
113116 const ownShare = (mib: number): number => Math.max(1, Math.floor(mib * OWN_SIZE_SHARE));
114117
115 /** This forge's size when it is a child in a cell (`FORGE_CELL_SIZE`, set by its parent: `cellEnv`); else none. */
118 /**
119 * This forge's size when it is a child in a cell (`FORGE_CELL_SIZE`, set by its parent: `cellEnv`);
120 * else none.
121 */
116122 function ownSize(): Size | undefined {
117123 const raw = process.env["FORGE_CELL_SIZE"];
118124 return isSize(raw) ? raw : undefined;
@@ -186,7 +192,10 @@ export function memoryCeilingSource(
186192 return ceiling(env, podFile, size).from;
187193 }
188194
189 /** The machine's ceiling, held under {@link OWN_SIZE_SHARE} of the forge's own size when it is a child. */
195 /**
196 * The machine's ceiling, held under {@link OWN_SIZE_SHARE} of the forge's own size when it is a
197 * child.
198 */
190199 function ceiling(
191200 env: string | undefined,
192201 podFile: string,
@@ -253,7 +262,9 @@ function positive(raw: string | undefined): number | undefined {
253262 return Number.isFinite(n) && n > 0 ? n : undefined;
254263 }
255264
256 /** The defaults for `kind`, raised or lowered by a declaration, never past the machine's maximum. */
265 /**
266 * The defaults for `kind`, raised or lowered by a declaration, never past the machine's maximum.
267 */
257268 export function resolveLimits(
258269 kind: "app" | "step",
259270 declared: DeclaredLimits | undefined,
@@ -323,7 +334,9 @@ export interface CellSpec {
323334 /** Origins always reachable: the forge's own URL. */
324335 readonly reach: readonly string[];
325336 readonly limits: Limits;
326 /** Read-only paths to keep visible when they sit under a hidden directory (a forge's own code). */
337 /**
338 * Read-only paths to keep visible when they sit under a hidden directory (a forge's own code).
339 */
327340 readonly reveal?: readonly string[] | undefined;
328341 /** The name the cell's processes see as the host's (`cellHostname`); "cell" when unset. */
329342 readonly hostname?: string | undefined;
@@ -333,7 +346,10 @@ export interface CellSpec {
333346 readonly terminal?: boolean | undefined;
334347 /** Told of every undeclared host let through while egress is only reported. */
335348 readonly onUndeclared?: ((host: string, port: number) => void) | undefined;
336 /** Egress entries allowed since the cell started, asked on every connection (`apps/egress_refused.ts`). */
349 /**
350 * Egress entries allowed since the cell started, asked on every connection
351 * (`apps/egress_refused.ts`).
352 */
337353 readonly allowed?: (() => readonly string[]) | undefined;
338354 /** Told of every refusal by policy, host and port, and whether it is private. */
339355 readonly onRefused?: ((host: string, port: number, isPrivate: boolean) => void) | undefined;
@@ -395,7 +411,10 @@ export interface Cell {
395411 readonly breach: () => string | undefined;
396412 /** Stop watching and close the cell's sockets. */
397413 readonly close: () => void;
398 /** The cell's own cgroup directory, where the kernel holds its caps; absent when only the watch does. */
414 /**
415 * The cell's own cgroup directory, where the kernel holds its caps; absent when only the watch
416 * does.
417 */
399418 readonly cgroupDir?: string;
400419 }
401420
@@ -614,7 +633,8 @@ for p in $loops; do (socat TCP-LISTEN:$p,bind=127.0.0.1,fork,reuseaddr UNIX-CONN
614633 i=0; while [ $i -lt 200 ] && ! grep -q ':${PROXY_PORT.toString(16).toUpperCase().padStart(4, "0")} 00000000:0000 0A' /proc/net/tcp 2>/dev/null; do sleep 0.01; i=$((i+1)); done
615634 exec "$@"`;
616635
617 function origin(url: string): { host: string; port: number } | undefined {
636 /** A URL's host and port, as the egress proxy holds an origin. */
637 export function origin(url: string): { host: string; port: number } | undefined {
618638 try {
619639 const u = new URL(url);
620640 const port = Number(u.port || (u.protocol === "https:" ? 443 : 80));
@@ -644,8 +664,9 @@ export function cellMounts(
644664 kind: (path: string) => "dir" | "file" | undefined = pathKind,
645665 real: (path: string) => string = realPath,
646666 ): { hide: string[]; blank: string[]; reveal: string[] } {
647 // Each under the name it was given and under the name it really has: a data directory configured through
648 // a link is the same directory at its real path, and a path is compared as text from here on.
667 // Each under the name it was given and under the name it really has: a data directory configured
668 // through a link is the same directory at its real path, and a path is compared as text from here
669 // on.
649670 const hidden = [...new Set([...spec.hide, ...machine].flatMap((path) => [path, real(path)]))]
650671 // A path that is not there has nothing to hide (and cannot be mounted
651672 // over on a read-only root).
@@ -660,7 +681,8 @@ export function cellMounts(
660681 const blank = hidden.filter((path) => kind(path) === "file");
661682 // The runtime itself must stay visible when it lives in a hidden home
662683 // (`~/.bun/bin/bun` on a developer's machine).
663 // Kept when it sits under something hidden by either name: the forge's own code is known by its real path.
684 // Kept when it sits under something hidden by either name: the forge's own code is known by its
685 // real path.
664686 const reveal = [dirname(process.execPath), ...(spec.reveal ?? [])].filter((path) =>
665687 hide.some((h) => within(path, h) || within(real(path), h)),
666688 );
@@ -676,7 +698,10 @@ function realPath(path: string): string {
676698 }
677699 }
678700
679 /** The paths of `asked` that lie inside no other: each of the rest is already writable through the one it lies in. */
701 /**
702 * The paths of `asked` that lie inside no other: each of the rest is already writable through the
703 * one it lies in.
704 */
680705 function bindRoots(asked: readonly string[]): string[] {
681706 return asked.filter((path) => !asked.some((other) => other !== path && within(path, other)));
682707 }
@@ -883,8 +908,8 @@ export function openCell(
883908 const asked = [...new Set([spec.dir, spec.home, ...spec.writable])];
884909 const roots = bindRoots(asked);
885910 for (const path of roots) mkdirSync(path, { recursive: true });
886 // A place inside one of those (a child forge's home, in its own directory) is made if it can be; what is
887 // there under its name is the cell's own business, and is never mounted by name.
911 // A place inside one of those (a child forge's home, in its own directory) is made if it can be;
912 // what is there under its name is the cell's own business, and is never mounted by name.
888913 for (const path of asked) {
889914 if (roots.includes(path)) continue;
890915 try {
@@ -1120,7 +1145,10 @@ function readMountinfo(): string {
11201145
11211146 /** How often the process table is read, and over how long CPU is averaged. */
11221147 export const WATCH = { tickMs: 1_000, cpuWindowMs: 30_000, diskEveryTicks: 15 };
1123 /** A cell's process limit: the watch's every tick, and the kernel's `pids.max` where pids is delegated. */
1148 /**
1149 * A cell's process limit: the watch's every tick, and the kernel's `pids.max` where pids is
1150 * delegated.
1151 */
11241152 export const PROCESSES_MAX = 512;
11251153 const CLK_TCK = 100;
11261154
@@ -1133,7 +1161,10 @@ interface Watched {
11331161 ticks: number;
11341162 /** A disk walk is out for this cell. */
11351163 measuring?: boolean;
1136 /** When the cell began to be watched, and since when its cores have held over the stop line without a break (#11657). */
1164 /**
1165 * When the cell began to be watched, and since when its cores have held over the stop line
1166 * without a break (#11657).
1167 */
11371168 readonly startedAt: number;
11381169 overSince: number | null;
11391170 readonly key: number;
@@ -1157,7 +1188,9 @@ const watched = new Set<Watched>();
11571188 let timer: ReturnType<typeof setInterval> | undefined;
11581189 let nextKey = 1;
11591190
1160 /** A cell's memory cap, bytes, and who holds it: the kernel (`apps/cgroup.ts`) or the watch alone. */
1191 /**
1192 * A cell's memory cap, bytes, and who holds it: the kernel (`apps/cgroup.ts`) or the watch alone.
1193 */
11611194 export interface MemoryCap {
11621195 readonly bytes: number;
11631196 readonly by: "kernel" | "watch";
@@ -1169,7 +1202,9 @@ export function memoryCapHolder(): { by: "kernel" } | { by: "watch"; reason: str
11691202 return host.ok ? { by: "kernel" } : { by: "watch", reason: host.reason };
11701203 }
11711204
1172 /** Whether cells' process counts are the kernel's here (`pids.max`), or why they are watched only. */
1205 /**
1206 * Whether cells' process counts are the kernel's here (`pids.max`), or why they are watched only.
1207 */
11731208 export function processCapHolder(host: CgroupHost = cgroupHost()): { by: "kernel" } | { by: "watch"; reason: string } {
11741209 if (!host.ok) return { by: "watch", reason: host.reason };
11751210 return pidsDelegated(host)
@@ -1424,7 +1459,10 @@ export function diskEntriesMax(diskMiB: number): number {
14241459 return Number.isFinite(diskMiB) && diskMiB > 0 ? Math.max(DISK_ENTRIES_FLOOR, Math.floor(diskMiB * 256)) : DISK_ENTRIES_FLOOR;
14251460 }
14261461
1427 /** Why a cell is stopped for its disk, from what a walk of its directories found; undefined while it is within its limit. */
1462 /**
1463 * Why a cell is stopped for its disk, from what a walk of its directories found; undefined while it
1464 * is within its limit.
1465 */
14281466 export function diskBreach(walk: DiskWalk, diskMiB: number): string | undefined {
14291467 if (walk.capped)
14301468 return `holds more than ${diskEntriesMax(diskMiB).toLocaleString("en-US")} files and directories: too many to count toward its ${diskMiB} MiB of disk`;
@@ -1663,7 +1701,10 @@ function decide(now: number, trees: readonly TreeReading[]): void {
16631701 }
16641702 }
16651703
1666 /** The root process of every labelled cell being watched, for grouping the process table by what launched it (`system/view.ts`). */
1704 /**
1705 * The root process of every labelled cell being watched, for grouping the process table by what
1706 * launched it (`system/view.ts`).
1707 */
16671708 export function cellRoots(): { label: string; root: number; cap: MemoryCap }[] {
16681709 const out: { label: string; root: number; cap: MemoryCap }[] = [];
16691710 for (const e of watched) if (e.label !== undefined) out.push({ label: e.label, root: e.root, cap: e.cap });
src/apps/egress.ts +62 −13
@@ -20,6 +20,7 @@
2020 import { storedText } from "../kernel/blocked.ts";
2121 import { lookup } from "node:dns/promises";
2222 import { isIP } from "node:net";
23 import { timingSafeEqual } from "node:crypto";
2324 import type { Socket } from "bun";
2425 import { neverGrantable, PAYMENT_HOSTS } from "../core/egress_grants.ts";
2526 import { addressBytes } from "../kernel/cidr.ts";
@@ -43,15 +44,24 @@ export interface EgressPolicy {
4344 readonly report?: ((host: string, port: number) => void) | undefined;
4445 /** Whose proxy this is (`app:<slug>/<env>`), for the connection gauges. */
4546 readonly label?: string | undefined;
46 /** More `egress` entries, asked on every connection: what an owner allowed since the cell started (#10456). */
47 /**
48 * More `egress` entries, asked on every connection: what an owner allowed since the cell started
49 * (#10456).
50 */
4751 readonly allowed?: (() => readonly string[]) | undefined;
48 /** Told of every connection refused by policy (not one that merely failed), with whether the destination is private. */
52 /**
53 * Told of every connection refused by policy (not one that merely failed), with whether the
54 * destination is private.
55 */
4956 readonly refused?: ((host: string, port: number, isPrivate: boolean) => void) | undefined;
5057 /** Most connections open to one destination at once; default 8. */
5158 readonly maxPerDestination?: number | undefined;
5259 /** Close a connection that moved no bytes for this long; default 5 min. */
5360 readonly idleMs?: number | undefined;
54 /** Bytes held for a slow client; a quarter of it pauses the upstream, and past all of it once paused the client is cut. Default 4 MiB. */
61 /**
62 * Bytes held for a slow client; a quarter of it pauses the upstream, and past all of it once
63 * paused the client is cut. Default 4 MiB.
64 */
5565 readonly queueMax?: number | undefined;
5666 /**
5767 * The proxy of the cell this forge itself runs in (`upstreamProxy`): a
@@ -61,8 +71,29 @@ export interface EgressPolicy {
6171 * resolves).
6272 */
6373 readonly upstream?: { readonly host: string; readonly port: number } | undefined;
64 /** Name to addresses: the system resolver unless a test passes its own. Never read from the environment. */
74 /**
75 * Name to addresses: the system resolver unless a test passes its own. Never read from the
76 * environment.
77 */
6578 readonly resolve?: ((host: string) => Promise<readonly string[]>) | undefined;
79 /**
80 * The `Proxy-Authorization` value every request must carry (a Job's gate on a network others
81 * share, `checks/job_gate.ts`); without it the proxy answers `407` and dials nothing. Unset for a
82 * cell, whose socket only the cell can reach.
83 */
84 readonly credential?: string | undefined;
85 }
86
87 /**
88 * Whether a request head's `Proxy-Authorization` is exactly `credential`, compared in constant
89 * time.
90 */
91 function authorized(headers: readonly string[], credential: string): boolean {
92 const line = headers.find((h) => /^proxy-authorization:/i.test(h));
93 if (line === undefined) return false;
94 const given = encoder.encode(line.slice(line.indexOf(":") + 1).trim());
95 const want = encoder.encode(credential);
96 return given.length === want.length && timingSafeEqual(given, want);
6697 }
6798
6899 /**
@@ -100,6 +131,8 @@ export function cellProxy(env: Record<string, string | undefined> = process.env)
100131 }
101132
102133 export interface Egress {
134 /** The port listened on, for a `{ tcp }` listener; absent on a socket. */
135 readonly port?: number | undefined;
103136 close(): void;
104137 }
105138
@@ -108,7 +141,9 @@ const MAX_PER_DESTINATION = 8;
108141 /** A public destination's cap: an install opens dozens of parallel fetches to one registry. */
109142 export const MAX_PER_PUBLIC_DESTINATION = 64;
110143 const IDLE_MS = 5 * 60_000;
111 /** Bytes held for a client that does not read before its upstream is cut, once the pause has landed. */
144 /**
145 * Bytes held for a client that does not read before its upstream is cut, once the pause has landed.
146 */
112147 const QUEUE_MAX = 4 * 1024 * 1024;
113148 /**
114149 * A backlog past this share of `queueMax` pauses the side that fills it until
@@ -303,7 +338,10 @@ export interface NpmTopRepo {
303338 readonly bytes: number;
304339 }
305340
306 /** The metered repository the registry sent most to in the last hour; none when no metered install fetched. */
341 /**
342 * The metered repository the registry sent most to in the last hour; none when no metered install
343 * fetched.
344 */
307345 export function npmTopRepoLastHour(now = Date.now()): NpmTopRepo | undefined {
308346 const minute = Math.floor(now / 60_000);
309347 let top: NpmTopRepo | undefined;
@@ -362,7 +400,10 @@ export function resetEgressTotals(): void {
362400 /** The ports a declared host may be reached on when its entry names none. */
363401 export const WEB_PORTS: readonly number[] = [80, 443];
364402
365 /** What stands in front of `--<host>` in a hosted forge's name, or its app's: one DNS label's worth, never a dot. */
403 /**
404 * What stands in front of `--<host>` in a hosted forge's name, or its app's: one DNS label's worth,
405 * never a dot.
406 */
366407 const BESIDE_LABEL = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?$/;
367408
368409 /**
@@ -625,6 +666,11 @@ async function open(
625666 // tunnel, the rewritten head for a forwarded request.
626667 let prefix = new Uint8Array(0);
627668 let tunnel = false;
669 if (policy.credential !== undefined && !authorized(headers, policy.credential)) {
670 log("egress refused: no credential for this gate");
671 refuse(client, "407 Proxy Authentication Required", "this gate takes only its own run's credential");
672 return;
673 }
628674 if (method.toUpperCase() === "CONNECT") {
629675 const at = target.lastIndexOf(":");
630676 host = target.slice(0, at);
@@ -677,8 +723,9 @@ async function open(
677723 }
678724 const destination = `${host.toLowerCase()}:${port}`;
679725 const count = tally(policy.label ?? "", destination);
680 // The cap guards a granted private device (a Pi's camera, #3719); a public host such as the package
681 // registry is reached by many parallel requests from one install, so it gets the wider cap.
726 // The cap guards a granted private device (a Pi's camera, #3719); a public host such as the
727 // package registry is reached by many parallel requests from one install, so it gets the wider
728 // cap.
682729 const cap = policy.maxPerDestination ?? (policy.upstream !== undefined || isPublicAddress(verdict.address) ? MAX_PER_PUBLIC_DESTINATION : MAX_PER_DESTINATION);
683730 if (count.open >= cap) {
684731 log(`egress capped: ${policy.label ?? "app"} already has ${count.open} connections to ${destination}`);
@@ -804,11 +851,12 @@ async function open(
804851 }
805852
806853 /**
807 * Listen at `path` and forward only what `policy` allows. `{ tcp }` (a
808 * loopback port) exists for the tests, where a unix socket cannot be bound.
854 * Listen at `path` and forward only what `policy` allows. `{ tcp }` is a
855 * port on loopback (the tests, where a unix socket cannot be bound) or on
856 * `hostname`: the address a cluster Job's pod dials (`checks/job_gate.ts`).
809857 */
810858 export function startEgress(
811 path: string | { tcp: number },
859 path: string | { tcp: number; hostname?: string },
812860 policy: EgressPolicy,
813861 log: (line: string) => void = () => {},
814862 ): Egress {
@@ -875,7 +923,7 @@ export function startEgress(
875923 const server =
876924 typeof path === "string"
877925 ? Bun.listen<Conn>({ unix: path, socket: handlers })
878 : Bun.listen<Conn>({ hostname: "127.0.0.1", port: path.tcp, socket: handlers });
926 : Bun.listen<Conn>({ hostname: path.hostname ?? "127.0.0.1", port: path.tcp, socket: handlers });
879927 // A tunnel that moves nothing (a client that stopped reading, an upstream
880928 // that stopped sending) is cut, so it cannot hold an upstream open.
881929 const idleMs = policy.idleMs ?? IDLE_MS;
@@ -894,6 +942,7 @@ export function startEgress(
894942 );
895943 sweep.unref();
896944 return {
945 port: "port" in server ? server.port : undefined,
897946 close() {
898947 clearInterval(sweep);
899948 for (const client of live) {
src/checks/job_gate.ts added +91
@@ -0,0 +1,91 @@
1 /**

Security: the gate takes any connection that reaches its port, with no credential. Require a per-run Proxy-Authorization secret (carried in the proxy URL given to the pod) so another pod on the network cannot use the forge as an egress proxy; test that a CONNECT without it is refused.

2 * The egress wall around a check run as a cluster Job (#13251): the same
3 * proxy a cell's only way out is (`apps/egress.ts`), opened for one run on
4 * the address the Job's pod dials (`FORGE_CHECKS_EGRESS_GATE`), and the
5 * proxy variables that point the pod at it. It forwards to the forge and to
6 * what `forge.json` `egress` declares, and refuses everything else with
7 * "refused by this forge: <host> is not in this app's forge.json egress".
8 * The operator's NetworkPolicy (docs/CLUSTER.md) lets a `forge-check` pod
9 * reach nothing but this gate's ports, so the variables are not a courtesy.
10 *
11 * The address is on a network other pods share, so the gate takes only its
12 * own run: a secret made for the run rides in the proxy URL the pod gets,
13 * and a request without it as `Proxy-Authorization` is answered `407`.
14 * It listens on a port of {@link JOB_GATE_PORTS}, a fixed range the
15 * NetworkPolicy names, so the policy need not open the forge's other ports.
16 */
17 import { randomBytes } from "node:crypto";
18 import { PROXY_VARS, startEgress, type EgressPolicy } from "../apps/egress.ts";
19 import { err, ok, type Result } from "../kernel/result.ts";
20
21 /**
22 * What a Job's run records of its boundary. The forge holds the gate and the
23 * proxy variables; that the pod can reach nothing else is the cluster's
24 * NetworkPolicy, which the forge cannot see, and the note says so.
25 */
26 export const JOB_GATE_NOTE = "The forge holds its egress; the cluster's NetworkPolicy holds the rest.";
27
28 /**
29 * The ports a gate listens on, one per Job running at once; docs/CLUSTER.md's policy names them.
30 */
31 export const JOB_GATE_PORTS = { from: 47100, to: 47199 } as const;
32
33 /** The user name in a gate's proxy URL; the secret is its password. */
34 const GATE_USER = "forge";
35
36 /**
37 * The gate's policy, as a cell's: the forge, the registry and what forge.json
38 * declares. Undeclared hosts are let through and noted only under
39 * `cells.egress=report`, and never for an outside contributor's check.
40 */
41 export function jobGatePolicy(input: {
42 readonly hosts: readonly string[];
43 readonly origins: readonly { host: string; port: number }[];
44 readonly label: string;
45 readonly outside: boolean;
46 readonly setting: "enforce" | "report";
47 readonly note: (host: string, port: number) => void;
48 }): EgressPolicy {
49 return {
50 hosts: input.hosts,
51 origins: input.origins,
52 label: input.label,
53 report: !input.outside && input.setting === "report" ? input.note : undefined,
54 };
55 }
56
57 export type JobGate = {
58 /** The variables the Job's container gets, beside its `CI` and `FORGE_*`. */
59 readonly env: Record<string, string>;
60 close(): void;
61 };
62
63 /**
64 * Open the gate on `address`, at the first free port of `ports`; the reason when it cannot
65 * listen.
66 */
67 export function openJobGate(
68 address: string,
69 policy: EgressPolicy,
70 log: (line: string) => void = () => {},
71 ports: { readonly from: number; readonly to: number } = JOB_GATE_PORTS,
72 ): Result<JobGate, string> {
73 const secret = randomBytes(24).toString("hex");
74 const held: EgressPolicy = { ...policy, credential: `Basic ${btoa(`${GATE_USER}:${secret}`)}` };
75 let gate;
76 let last = "no port in the range";
77 for (let port = ports.from; port <= ports.to && gate === undefined; port++) {
78 try {
79 gate = startEgress({ tcp: port, hostname: address }, held, log);
80 } catch (error) {
81 last = error instanceof Error ? error.message : String(error);
82 }
83 }
84 if (gate === undefined)
85 return err(`the egress gate could not listen on ${address}, ports ${ports.from}-${ports.to}: ${last}`);
86 const url = `http://${GATE_USER}:${secret}@${address}:${gate.port}`;
87 const env: Record<string, string> = {};
88 for (const name of PROXY_VARS) env[name] = name.toLowerCase() === "no_proxy" ? "" : url;
89 const open = gate;
90 return ok({ env, close: () => open.close() });
91 }
src/checks/job_step.ts +7 −1
@@ -28,11 +28,17 @@ export async function runJobStep(
2828 /** The Job's own output, apart from the echoed commands. */
2929 output: string[];
3030 onNode: (node: string) => void;
31 /** The proxy variables of this run's egress gate (`checks/job_gate.ts`). */
32 proxy?: Record<string, string>;
3133 options?: JobOptions;
3234 },
3335 ): Promise<Result<void, string>> {
3436 // Only the check's own variables cross to the cluster, never the forge's.
35 const env = Object.fromEntries(Object.entries(input.env).filter(([k]) => k === "CI" || k.startsWith("FORGE_")));
37 // Its way out is the run's egress gate, and nothing else (#13251).
38 const env = {
39 ...Object.fromEntries(Object.entries(input.env).filter(([k]) => k === "CI" || k.startsWith("FORGE_"))),
40 ...input.proxy,
41 };
3642 const spec: JobSpec = {
3743 name: `check-${input.runId.toLowerCase().replace(/[^a-z0-9]/g, "-")}`.slice(0, 53).replace(/-+$/, ""),
3844 image: placement.image,
src/checks/placement.ts +20 −1
@@ -8,6 +8,9 @@
88 * FORGE_CHECKS_CLAIM a ReadWriteMany claim holding the checks root
99 * FORGE_CHECKS_NODE_SELECTOR key=value,key=value
1010 * FORGE_CHECKS_TOLERATIONS a JSON list of tolerations
11 * FORGE_CHECKS_EGRESS_GATE the forge's address a Job's pod dials for its
12 * egress (`checks/job_gate.ts`); without it no
13 * check runs as a Job
1114 */
1215 import type { Toleration } from "../cluster/jobs.ts";
1316
@@ -19,8 +22,13 @@ export type Placement =
1922 readonly claim: string;
2023 readonly nodeSelector: Record<string, string>;
2124 readonly tolerations: Toleration[];
25 /** The address the forge's per-run egress proxy listens on and the pod dials. */
26 readonly gate?: string;
2227 };
2328
29 /** An IPv4 address or a DNS name: what a pod can dial and the forge can bind. */
30 const GATE = /^(?=.{1,253}$)[A-Za-z0-9]([-A-Za-z0-9]{0,61}[A-Za-z0-9])?(\.[A-Za-z0-9]([-A-Za-z0-9]{0,61}[A-Za-z0-9])?)*$/;
31
2432 const LABEL = /^[A-Za-z0-9]([-A-Za-z0-9_./]{0,61}[A-Za-z0-9])?$/;
2533 const EFFECTS = ["NoSchedule", "PreferNoSchedule", "NoExecute"];
2634
@@ -73,8 +81,10 @@ export function placementFromEnv(env: Record<string, string | undefined> = proce
7381 const nodeSelector = parseSelector(env["FORGE_CHECKS_NODE_SELECTOR"] ?? "");
7482 const tolerations = parseTolerations(env["FORGE_CHECKS_TOLERATIONS"]?.trim() || "[]");
7583 // A job placement that cannot be built is not guessed at.
84 const gate = env["FORGE_CHECKS_EGRESS_GATE"]?.trim() ?? "";
7685 if (image === "" || /\s/.test(image) || claim === "" || !nodeSelector || !tolerations) return { mode: "process" };
77 return { mode: "job", image, claim, nodeSelector, tolerations };
86 if (gate !== "" && !GATE.test(gate)) return { mode: "process" };
87 return { mode: "job", image, claim, nodeSelector, tolerations, ...(gate === "" ? {} : { gate }) };
7888 }
7989
8090 /**
@@ -88,3 +98,12 @@ export function placementWords(run: { placement: string; node: string }): string
8898 const node = run.node.replace(/[^\x21-\x7e]/g, "").slice(0, 253);
8999 return node === "" ? "ran as a cluster job" : `ran as a cluster job on ${node}`;
90100 }
101
102 /**
103 * The sandbox words for a check that ran as a Job, the page's and the terminal's (#13251): two
104 * plain sentences, the second what the run recorded of its wall (`JOB_GATE_NOTE`).
105 */
106 export function jobSandboxWords(note: string): string {
107 const rest = note.replace(/[^\x20-\x7e]/g, "").trim();
108 return rest === "" ? "Held by the cluster: the check ran as a Job." : `Held by the cluster: the check ran as a Job. ${rest}`;
109 }
src/checks/service.ts +85 −38
@@ -124,6 +124,7 @@ import {
124124 } from "../core/capacity.ts";
125125 import {
126126 cellsAvailable,
127 origin,
127128 limitWarnings,
128129 resolveLimits,
129130 STEP_HOSTS,
@@ -131,7 +132,8 @@ import {
131132 type CellSpec,
132133 } from "../apps/contain.ts";
133134 import { summariseCheckFailure } from "./summary.ts";
134 import type { Result } from "../kernel/result.ts";
135 import { err, type Result } from "../kernel/result.ts";
136 import { JOB_GATE_NOTE, jobGatePolicy, openJobGate } from "./job_gate.ts";
135137 import { placementFromEnv, type Placement } from "./placement.ts";
136138 import { refuseSmallPod, type PodRoom } from "./pod_room.ts";
137139 import { JOB_VANISHED, runJobStep } from "./job_step.ts";
@@ -198,7 +200,10 @@ export interface ChecksOptions {
198200 readonly job?: JobOptions | undefined;
199201 /** How long a transient install failure waits for its retry. @default INSTALL_RETRY_MS */
200202 readonly installRetryMs?: number | undefined;
201 /** Runs the check step itself; tests stand one in for a stop the cell's watch makes (#11617). @default runStep */
203 /**
204 * Runs the check step itself; tests stand one in for a stop the cell's watch makes
205 * (#11617). @default runStep
206 */
202207 readonly checkStep?: typeof runStep | undefined;
203208 /**
204209 * Where the pod's memory.max is read and the forge's size
@@ -232,7 +237,10 @@ export interface GateDone {
232237 }
233238
234239 export interface ChecksHandle {
235 /** Ask for a check of one commit; the answer is the run, or nothing when the commit declares no check. */
240 /**
241 * Ask for a check of one commit; the answer is the run, or nothing when the commit declares no
242 * check.
243 */
236244 request(
237245 repo: Repo,
238246 env: string,
@@ -450,7 +458,10 @@ const STORES: ReadonlySet<string> = new Set([IMPACT_DIR, KEPT_DIR, "learned", "g
450458 /** A gate's checkout (`gateRunId`): it has no run, only its `check_gate` row. */
451459 const GATE_DIR = /^gate-([0-9a-f]{12})$/;
452460
453 /** Whether a gate's checkout, this forge's or a peer's during a roll, is younger than a gate may run. */
461 /**
462 * Whether a gate's checkout, this forge's or a peer's during a roll, is younger than a gate may
463 * run.
464 */
454465 function gateMayBeRunning(path: string, now: number): boolean {
455466 try {
456467 return now - lstatSync(path).mtimeMs < GATE_DEADLINE_MS;
@@ -535,7 +546,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
535546 >();
536547 /** Checks already told why they wait, so the log says it once. */
537548 const waiting = new Set<string>();
538 /** Queued checks a full disk holds that have said so once (#11799); apart from `waiting`, so neither line hides the other. */
549 /**
550 * Queued checks a full disk holds that have said so once (#11799); apart from `waiting`, so
551 * neither line hides the other.
552 */
539553 const diskWaiting = new Set<string>();
540554 const memoryWaiting = new Set<string>();
541555 /** The verification-lane run in flight, which holds no check slot. */
@@ -689,7 +703,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
689703 */
690704 /** `pickNext` among `queued`, passing over any whose account is at its limit. */
691705 function startable(queued: readonly Run[]): Run | undefined {
692 // At 80% used no check starts (#11799): they stay queued, one line each, and the tick asks again.
706 // At 80% used no check starts (#11799): they stay queued, one line each, and the tick asks
707 // again.
693708 const full = usedRefusal(app.config.dataDir);
694709 if (full !== undefined) {
695710 for (const run of queued)
@@ -1131,7 +1146,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
11311146 readonly runId?: string;
11321147 readonly commitOid: string;
11331148 readonly pull?: Pull | undefined;
1134 /** `run`: a push to a branch; `fast`: a pull's push; `full`: on land; `verify`: main after landing. */
1149 /**
1150 * `run`: a push to a branch; `fast`: a pull's push; `full`: on land; `verify`: main after
1151 * landing.
1152 */
11351153 readonly tier: "run" | "fast" | "full" | "verify";
11361154 /** A bisection step: only these test files run (`FORGE_CHECK_ONLY`). */
11371155 readonly only?: readonly string[];
@@ -1143,7 +1161,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
11431161 readonly keepImpact?: boolean;
11441162 /** The default branch's own run: only its log may open issues. */
11451163 readonly fileOrderDependent?: boolean;
1146 /** The default branch's own run: a failure of one or two tests is re-run alone before it counts as red. */
1164 /**
1165 * The default branch's own run: a failure of one or two tests is re-run alone before it counts
1166 * as red.
1167 */
11471168 readonly isMain?: boolean;
11481169 readonly write: (line: string) => void;
11491170 readonly signal: AbortSignal;
@@ -1235,7 +1256,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
12351256 // (`checks/held.ts`). The branch's own supplies only its install step.
12361257 let manifest = declared.value;
12371258 let baseTip: string | undefined;
1238 /** An outside agent's pull nobody has approved: its code is checked, inside a cell or not at all. */
1259 /**
1260 * An outside agent's pull nobody has approved: its code is checked, inside a cell or not at
1261 * all.
1262 */
12391263 let outside = false;
12401264 if (x.pull !== undefined) {
12411265 baseTip = await resolveOid(bare, `refs/heads/${x.pull.baseBranch}`);
@@ -1278,7 +1302,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
12781302 write(OUTSIDE_CHECK_LINE(x.pull.baseBranch));
12791303 }
12801304 } else if (outsideAgentWaits(app.db, x.pull)) {
1281 // No gate on the base branch to hold it to: an outside contributor's pull does not bring its own.
1305 // No gate on the base branch to hold it to: an outside contributor's pull does not bring
1306 // its own.
12821307 return { ok: false, error: OUTSIDE_NO_GATE(x.pull.baseBranch) };
12831308 }
12841309 // What it asked for and does not have yet, said where a refused request is looked for.
@@ -1366,17 +1391,17 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13661391 // The check is repository code: it runs in a cell of its own
13671392 // (`apps/contain.ts`), its checkout and a home its only places to write,
13681393 // the forge, the registry and what it declared its only way out.
1369 // A Job cannot hold egress (#1428), so while it is enforced the check
1370 // stays in a cell. Read per run: an admin can flip it at runtime.
1371 // Nor can it hold an outside contributor's check to anything: that one always runs in a cell.
1372 const fellBack = placement.mode === "job" && (outside || egressSetting(app.db) === "enforce");
1394 // A Job holds egress only through a gate the forge opens for it
1395 // (`checks/job_gate.ts`, #13251): a cluster that names none cannot
1396 // honour forge.json, so the check stays in a cell, never unrestricted.
1397 const fellBack = placement.mode === "job" && placement.gate === undefined;
13731398 const placed: Placement = fellBack ? { mode: "process" } : placement;
13741399 const job = placed.mode === "job" ? placed : undefined;
13751400 if (x.runId !== undefined)
13761401 noteRunProgress(app.db, x.runId, { placement: placed.mode });
13771402 if (fellBack)
1378 write("placement: job skipped, egress is enforced and a Job cannot hold it");
1379 // A Job is its own boundary: no cell here, the cluster holds it.
1403 write("placement: job skipped, this cluster names no egress gate for a Job (FORGE_CHECKS_EGRESS_GATE)");
1404 // A Job is its own boundary: no cell here, the cluster and its gate hold it.
13801405 const cells = job === undefined ? await cellsAvailable() : undefined;
13811406 if (cells !== undefined) {
13821407 if (x.runId !== undefined)
@@ -1386,12 +1411,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13861411 if (!cells.ok)
13871412 write(`uncontained here: ${cells.reason}; the check runs unrestricted`);
13881413 } else {
1389 // The cluster holds a Job, but not as a cell does: the egress a
1390 // check declared is not enforced here, and the run says so.
13911414 if (x.runId !== undefined)
1392 noteRunProgress(app.db, x.runId, {
1393 sandbox: { enforced: false, note: "egress not enforced on a cluster job" },
1394 });
1415 noteRunProgress(app.db, x.runId, { sandbox: { enforced: true, note: JOB_GATE_NOTE } });
13951416 write("placement: job");
13961417 }
13971418 const home = `${dir}.home`;
@@ -1455,22 +1476,45 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
14551476 const checkLines: string[] = [];
14561477 // A bisection step runs only some files: what it shows green says nothing of the rest.
14571478 const wholeVerify = x.tier === "verify" && x.only === undefined;
1458 if (job !== undefined) {
1459 ran = await runJobStep(job, {
1460 runId: run.id,
1461 dir,
1462 commands: manifest.install === null ? [command] : [manifest.install, command],
1463 env,
1464 log: write,
1465 output: checkLines,
1466 signal,
1467 timeoutMs: manifest.timeout * 1000,
1468 limits: resolveLimits("step", manifest.limits),
1469 onNode: (node) => {
1470 if (x.runId !== undefined) noteRunProgress(app.db, x.runId, { node });
1471 },
1472 ...(options.job === undefined ? {} : { options: options.job }),
1473 });
1479 // The gate is held to what a cell holds: the forge, the registry and
1480 // what forge.json declares, an outside contributor's always enforced.
1481 const gate = job?.gate === undefined
1482 ? undefined
1483 : openJobGate(
1484 job.gate,
1485 jobGatePolicy({
1486 hosts: [...STEP_HOSTS, ...manifest.egress],
1487 origins: [options.forgeUrl].map(origin).filter((o) => o !== undefined),
1488 label: `check:${run.id}`,
1489 outside,
1490 setting: egressSetting(app.db),
1491 note: (host, port) => noteReached(app.config.dataDir, repo.id, host, port),
1492 }),
1493 write,
1494 );
1495 if (gate !== undefined && !gate.ok) {
1496 ran = err(gate.error);
1497 } else if (job !== undefined) {
1498 try {
1499 ran = await runJobStep(job, {
1500 runId: run.id,
1501 dir,
1502 commands: manifest.install === null ? [command] : [manifest.install, command],
1503 env,
1504 log: write,
1505 output: checkLines,
1506 signal,
1507 timeoutMs: manifest.timeout * 1000,
1508 limits: resolveLimits("step", manifest.limits),
1509 onNode: (node) => {
1510 if (x.runId !== undefined) noteRunProgress(app.db, x.runId, { node });
1511 },
1512 ...(gate === undefined ? {} : { proxy: gate.value.env }),
1513 ...(options.job === undefined ? {} : { options: options.job }),
1514 });
1515 } finally {
1516 gate?.value.close();
1517 }
14741518 } else {
14751519 if (manifest.install !== null) {
14761520 const install = manifest.install;
@@ -2053,7 +2097,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
20532097 }
20542098 }
20552099
2056 /** A gate asked of a forge already stopping is not started: the forge that carries on asks it again. */
2100 /**
2101 * A gate asked of a forge already stopping is not started: the forge that carries on asks it
2102 * again.
2103 */
20572104 function gateRun(repo: Repo, pull: Pull, commitOid: string, signal?: AbortSignal): Promise<GateDone> {
20582105 if (stopped) return Promise.resolve({ ok: false, error: INTERRUPTED, lines: [], interrupted: true });
20592106 const controller = new AbortController();
src/tui/screens/runlog.ts +12 −1
@@ -19,6 +19,7 @@ import { emptyState } from "../ui/widgets.ts";
1919 import { subscribeRunLog } from "../../runs/log.ts";
2020 import { Scroll, live, openOnWeb } from "./common.ts";
2121 import { costText, envLabel, runTone } from "./runwords.ts";
22 import { jobSandboxWords } from "../../checks/placement.ts";
2223
2324 const LOG_LINES = 200;
2425
@@ -102,8 +103,14 @@ export function runLogScreen(
102103 });
103104 return;
104105 }
105 const [head, body] = split(area, "v", [len(1), fill(1)]);
106106 const run = view.run;
107 // A Job's wall rests on the cluster as well as the forge: the terminal says so, as the page
108 // does. One sentence a row, so a narrow terminal cuts neither.
109 const wall =
110 run.placement === "job" && run.sandboxEnforced === true ? jobSandboxWords(run.sandboxNote).split(/(?<=\.) /) : [];
111 const rows = split(area, "v", [len(1), ...wall.map(() => len(1)), fill(1)]);
112 const head = rows[0];
113 const body = rows.at(-1);
107114 const tone = runTone(run.state);
108115 const state = [
109116 run.state,
@@ -121,6 +128,10 @@ export function runLogScreen(
121128 style(tone === undefined ? c.mutedFg : c.fg, c.bg),
122129 head!.x + head!.w,
123130 );
131 wall.forEach((words, i) => {
132 const row = rows[i + 1]!;
133 s.text(row.x, row.y, truncate(words, row.w), style(c.mutedFg, c.bg), row.x + row.w);
134 });
124135 if (lines.length === 0) {
125136 emptyState(s, body!, c, {
126137 title: "No log yet",
src/web/studio/sandbox.ts +4 −2
@@ -6,7 +6,7 @@
66 */
77
88 import type { Run } from "../../core/runs.ts";
9 import { placementWords } from "../../checks/placement.ts";
9 import { jobSandboxWords, placementWords } from "../../checks/placement.ts";
1010 import { html, type Html } from "../html.ts";
1111 import { badge } from "../ui.ts";
1212
@@ -23,7 +23,9 @@ export function sandboxLine(run: Run): Html | null {
2323 return html`<p class="muted small" id="run-sandbox">
2424 ${agent
2525 ? "Sandboxed: the shell was held to the policy."
26 : "Contained: the repository's code ran in a cell of its own, with limits."}
26 : run.placement === "job"
27 ? jobSandboxWords(run.sandboxNote)
outdated

Principle 8: the note is a long clause in parentheses. Write two plain sentences, e.g. 'Held by the cluster: the check ran as a Job. The forge holds its egress; the cluster's NetworkPolicy holds the rest.' Also add the same line to the terminal run screen.

28 : "Contained: the repository's code ran in a cell of its own, with limits."}
2729 </p>`;
2830 return html`<p class="small" id="run-sandbox">
2931 ${badge(agent ? "Unsandboxed" : "Uncontained", { tone: "danger" })}
test/checks-job-vanish.test.ts +1 −4
@@ -10,7 +10,6 @@ import { createApp, type App } from "../src/app.ts";
1010 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
1111 import { startChecks, type ChecksHandle } from "../src/checks/service.ts";
1212 import { VANISHED_SUMMARY, listRuns } from "../src/core/runs.ts";
13 import { setCapacity } from "../src/core/capacity.ts";
1413 import { createUser } from "../src/core/users.ts";
1514 import type { Repo } from "../src/core/repos.ts";
1615 import { serve, type Servers } from "../src/http/server.ts";
@@ -46,15 +45,13 @@ describe("a vanished check Job", () => {
4645 beforeAll(async () => {
4746 dataDir = await mkdtemp(join(tmpdir(), "forge-vanish-"));
4847 app = createApp({ dataDir, port: 0 });
49 // Jobs run only while egress is reported (#3190).
50 setCapacity(app.db, "cells.egress", "report", "test");
5148 servers = await serve(app);
5249 const fetch = async (_u: string, init: RequestInit) =>
5350 init.method === "GET" ? new Response("{}", { status: 404 }) : Response.json({});
5451 checks = startChecks(app, {
5552 forgeUrl: servers.url,
5653 concurrency: 1,
57 placement: { mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [] },
54 placement: { mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [], gate: "127.0.0.1" },
5855 job: { fetch: fetch as never, credentials: cred, pollMs: 1 },
5956 });
6057 const a = await createUser(app.db, {
test/checks-placement.test.ts +117 −16
@@ -12,7 +12,8 @@ import { commitToBranch, createRepo } from "../src/actions/repos.ts";
1212 import { parseSelector, parseTolerations, placementFromEnv } from "../src/checks/placement.ts";
1313 import { startChecks, type ChecksHandle } from "../src/checks/service.ts";
1414 import { runRows } from "../src/tui/screens/runlist.ts";
15 import { placementLine } from "../src/web/studio/sandbox.ts";
15 import { placementLine, sandboxLine } from "../src/web/studio/sandbox.ts";
16 import { JOB_GATE_NOTE, JOB_GATE_PORTS, jobGatePolicy, openJobGate } from "../src/checks/job_gate.ts";
1617 import { runPayload } from "../src/actions/runs.ts";
1718 import { placementWords } from "../src/checks/placement.ts";
1819 import { listRuns, type Run } from "../src/core/runs.ts";
@@ -51,11 +52,24 @@ describe("placement from the environment", () => {
5152 nodeSelector: { pool: "ci", arch: "arm64" },
5253 tolerations: [{ key: "ci", operator: "Exists", effect: "NoSchedule" }],
5354 });
55 expect(placementFromEnv({ ...job, FORGE_CHECKS_EGRESS_GATE: "10.0.0.7" })).toMatchObject({ mode: "job", gate: "10.0.0.7" });
56 expect(placementFromEnv({ ...job, FORGE_CHECKS_EGRESS_GATE: "forge.svc; rm" })).toEqual({ mode: "process" });
57 expect(placementFromEnv(job)).not.toHaveProperty("gate");
5458 expect(parseSelector("a")).toBeUndefined();
5559 expect(parseTolerations("{}")).toBeUndefined();
5660 });
5761 });
5862
63 describe("a Job's gate policy", () => {
64 const base = { hosts: ["registry.npmjs.org"], origins: [], label: "check:r", note: () => {} };
65 test("reports undeclared hosts only under report, and never for an outside contributor", () => {
66 expect(jobGatePolicy({ ...base, outside: false, setting: "report" }).report).toBe(base.note);
67 expect(jobGatePolicy({ ...base, outside: false, setting: "enforce" }).report).toBeUndefined();
68 expect(jobGatePolicy({ ...base, outside: true, setting: "report" }).report).toBeUndefined();
69 expect(jobGatePolicy({ ...base, outside: true, setting: "report" }).hosts).toEqual(["registry.npmjs.org"]);
70 });
71 });
72
5973 describe("checks placed", () => {
6074 let dataDir: string;
6175 let app: App;
@@ -65,11 +79,52 @@ describe("checks placed", () => {
6579 let repo: Repo;
6680 const calls: { method: string; url: string; body?: string }[] = [];
6781 let exit = 0;
82 /** What the Job's pod heard, through its proxy, from an undeclared host. */
83 const reached: string[] = [];
84 /** What another pod heard, dialling the same gate without the run's credential. */
85 const stranger: string[] = [];
86
87 // What the check in the pod does: dial the host through the proxy it was given.
88 async function connectThrough(proxy: string, target: string, credential = true): Promise<string> {
89 const { hostname, port, username, password } = new URL(proxy);
90 const auth = username === "" ? "" : `Proxy-Authorization: Basic ${btoa(`${username}:${password}`)}\r\n`;
91 let heard = "";
92 const done = Promise.withResolvers<void>();
93 const socket = await Bun.connect({
94 hostname,
95 port: Number(port),
96 socket: {
97 data(_s, chunk) {
98 heard += new TextDecoder().decode(chunk);
99 if (heard.includes("\r\n\r\n")) done.resolve();
100 },
101 close() {
102 done.resolve();
103 },
104 error() {
105 done.resolve();
106 },
107 },
108 });
109 socket.write(`CONNECT ${target} HTTP/1.1\r\nHost: ${target}\r\n${credential ? auth : ""}\r\n`);
110 // The proxy answers or closes; a hang is the test's own timeout.
111 await done.promise;
112 socket.end();
113 return heard;
114 }
68115
69116 // A stand-in kube API: the Job succeeds at once, its pod on node n7.
70117 const fetch = async (url: string, init: RequestInit) => {
71118 calls.push({ method: String(init.method), url, ...(typeof init.body === "string" ? { body: init.body } : {}) });
72119 const path = new URL(url).pathname;
120 if (init.method === "POST" && typeof init.body === "string") {
121 const env: { name: string; value: string }[] = JSON.parse(init.body).spec.template.spec.containers[0].env;
122 const proxy = env.find((e) => e.name === "HTTPS_PROXY")?.value;
123 if (proxy !== undefined) {
124 reached.push(await connectThrough(proxy, "undeclared.example:443"));
125 stranger.push(await connectThrough(proxy, "undeclared.example:443", false));
126 }
127 }
73128 if (init.method === "POST" || init.method === "DELETE") return Response.json({});
74129 if (path.endsWith("/log")) return new Response("from the job\nsecond line\n");
75130 if (path.endsWith("/pods"))
@@ -146,13 +201,10 @@ describe("checks placed", () => {
146201 claim: "checks-root",
147202 nodeSelector: { pool: "ci" },
148203 tolerations: [{ key: "ci", operator: "Exists" }],
204 gate: "127.0.0.1",
149205 }),
150206 );
151207 afterAll(down);
152 // Egress is enforced by default, which keeps checks in cells.
153 beforeAll(() => {
154 setCapacity(app.db, "cells.egress", "report", alice.id);
155 });
156208
157209 test("a check runs as a Job: log in the run, node and placement recorded", async () => {
158210 const run = await settled(await push("exit 0"));
@@ -172,13 +224,47 @@ describe("checks placed", () => {
172224 expect(pod.automountServiceAccountToken).toBe(false);
173225 expect(pod.containers[0].resources.limits.memory).toMatch(/Mi$/);
174226 expect(made.spec.template.metadata.labels["forge-check"]).toBe("true");
175 expect(run.sandboxEnforced).toBe(false);
176 expect(run.sandboxNote).toBe("egress not enforced on a cluster job");
227 expect(run.sandboxEnforced).toBe(true);
228 expect(run.sandboxNote).toBe(JOB_GATE_NOTE);
229 const line = sandboxLine(run)!.toString();
230 expect(line).toContain("ran as a Job");
231 expect(line).toContain("cluster&#39;s NetworkPolicy");
232 expect(line).not.toContain("in a cell");
177233 expect(pod.containers[0].image).toBe("oven/bun:1");
outdated

Not awaited, so this assertion never fails the test. Use await expect(connectThrough(proxy, "undeclared.example:443")).rejects.toThrow();.

178234 expect(pod.containers[0].command.at(-1)).toBe("exit 0");
179235 expect(calls.some((c) => c.method === "DELETE")).toBe(true);
180236 });
181237
238 test("the Job's way out is the forge's gate, which refuses an undeclared host in its own words", async () => {
239 reached.length = 0;
240 stranger.length = 0;
241 calls.length = 0;
242 const run = await settled(await push("exit 0 # reach out"));
243 expect(run.placement).toBe("job");
244 const env: { name: string; value: string }[] = JSON.parse(calls.find((c) => c.method === "POST")!.body!).spec
245 .template.spec.containers[0].env;
246 const proxy = env.find((e) => e.name === "HTTPS_PROXY")!.value;
247 // The run's own secret rides in the URL, on a port of the range the NetworkPolicy names.
248 const url = new URL(proxy);
249 expect(url.hostname).toBe("127.0.0.1");
250 expect(Number(url.port)).toBeGreaterThanOrEqual(JOB_GATE_PORTS.from);
251 expect(Number(url.port)).toBeLessThanOrEqual(JOB_GATE_PORTS.to);
252 expect(url.password).toMatch(/^[0-9a-f]{48}$/);
253 expect(env.find((e) => e.name === "NO_PROXY")!.value).toBe("");
254 expect(reached).toHaveLength(1);
255 expect(reached[0]).toStartWith("HTTP/1.1 403");
256 expect(reached[0]).toContain(
257 "refused by this forge: undeclared.example:443 is not in this app's forge.json egress",
258 );
259 // Another pod on the network that reaches the port but not the secret is refused, dialling
260 // nothing.
261 expect(stranger).toHaveLength(1);
262 expect(stranger[0]).toStartWith("HTTP/1.1 407");
263 expect(stranger[0]).toContain("X-Forge-Egress: refused");
264 // The gate lives as long as the run: closed once the Job is done.
265 await expect(connectThrough(proxy, "undeclared.example:443")).rejects.toThrow();
266 });
267
182268 test("the page, the terminal and the events say where a job ran; a process run says nothing", async () => {
183269 const run = await settled(await push("exit 0 # page"));
184270 expect(run.placement).toBe("job");
@@ -214,11 +300,33 @@ describe("checks placed", () => {
214300 });
215301 });
216302
217 describe("job under enforce", () => {
303 describe("job whose gate cannot listen", () => {
304 // 192.0.2.1 is TEST-NET-1: no interface here holds it, so nothing can listen on it.
305 beforeAll(() =>
306 boot({ mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [], gate: "192.0.2.1" }),
307 );
308 afterAll(down);
309
310 test("the gate says why, and the check fails with that reason, calling no cluster", async () => {
311 const policy = jobGatePolicy({ hosts: [], origins: [], label: "t", outside: false, setting: "enforce", note: () => {} });
312 const got = openJobGate("192.0.2.1", policy, () => {}, { from: 47190, to: 47191 });
313 expect(got.ok).toBe(false);
314 if (!got.ok) expect(got.error).toContain("the egress gate could not listen on 192.0.2.1, ports 47190-47191");
315 calls.length = 0;
316 const run = await settled(await push("exit 0 # no gate"));
317 expect(run.state).toBe("failed");
318 expect(readRunLog(dataDir, run.id).lines.join("\n")).toContain("the egress gate could not listen on 192.0.2.1");
319 expect(calls.filter((c) => c.method === "POST")).toHaveLength(0);
320 });
321 });
322
323 describe("job without an egress gate", () => {
218324 beforeAll(() => boot({ mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [] }));
219325 afterAll(down);
220326
221327 test("never calls the cluster: the check runs in a cell and says why", async () => {
328 // Report or enforce, a Job with no gate would run unrestricted: it never runs.
329 setCapacity(app.db, "cells.egress", "report", alice.id);
222330 calls.length = 0;
223331 const run = await settled(await push("echo in a cell"));
224332 expect(run.state).toBe("ok");
@@ -226,18 +334,11 @@ describe("checks placed", () => {
226334 expect(run.node).toBe("");
227335 expect(calls).toHaveLength(0);
228336 const log = readRunLog(dataDir, run.id).lines.join("\n");
229 expect(log).toContain("placement: job skipped, egress is enforced and a Job cannot hold it");
337 expect(log).toContain("placement: job skipped, this cluster names no egress gate for a Job");
230338 expect(log).toContain("in a cell");
231339 const page = await (await globalThis.fetch(`${servers.url}/alice/lib/runs/${run.id}`)).text();
232340 expect(page).not.toContain("cluster job");
233341 });
234
235 test("is read per run: flipping to report sends the next check to a Job", async () => {
236 setCapacity(app.db, "cells.egress", "report", alice.id);
237 const run = await settled(await push("echo flipped"));
238 expect(run.placement).toBe("job");
239 expect(calls.some((c) => c.method === "POST")).toBe(true);
240 });
241342 });
242343
243344 describe("process", () => {
test/cluster-doc.test.ts +1 −1
@@ -9,7 +9,7 @@ const read = (f: string) => readFileSync(join(root, f), "utf8");
99
1010 test("the doc names every FORGE_CHECKS_* variable placement.ts reads", () => {
1111 const vars = new Set([...read("src/checks/placement.ts").matchAll(/env\["(FORGE_CHECKS_[A-Z_]+)"\]/g)].map((m) => m[1]!));
12 expect(vars.size).toBe(5);
12 expect(vars.size).toBe(6);
1313 for (const v of vars) expect(doc).toContain(`\`${v}\``);
1414 });
1515
test/tui-runlog.test.ts +18 −1
@@ -7,7 +7,8 @@ import { appendFileSync } from "node:fs";
77 import { announceRun } from "../src/actions/runs.ts";
88 import { createRepo } from "../src/actions/repos.ts";
99 import { createApp, type App as Forge } from "../src/app.ts";
10 import { claimRun, createRun, finishRun } from "../src/core/runs.ts";
10 import { claimRun, createRun, finishRun, noteRunProgress } from "../src/core/runs.ts";
11 import { JOB_GATE_NOTE } from "../src/checks/job_gate.ts";
1112 import type { Repo } from "../src/core/repos.ts";
1213 import { createUser, type User } from "../src/core/users.ts";
1314 import { appendRunLog, runLogPath } from "../src/runs/log.ts";
@@ -74,6 +75,22 @@ test("a finished run shows its state and its last 200 lines, at the end", async
7475 expect(app.screen.toText()).not.toContain("\nline-100\n");
7576 });
7677
78 test("a check that ran as a Job says, under its state, that its wall rests on the cluster too (#13251)", async () => {
79 const run = createRun(forge.db, { repoId: repo.id, kind: "check" });
80 claimRun(forge.db, run.id);
81 noteRunProgress(forge.db, run.id, { placement: "job", sandbox: { enforced: true, note: JOB_GATE_NOTE } });
82 appendRunLog(dir, run.id, "from the job");
83 finishRun(forge.db, run.id, { state: "ok", summary: "green" });
84 const text = (await open(alice, run.id)).screen.toText();
85 expect(text).toContain("Held by the cluster: the check ran as a Job.");
86 expect(text).toContain("NetworkPolicy");
87 expect(text).toContain("from the job");
88 const here = createRun(forge.db, { repoId: repo.id, kind: "check" });
89 claimRun(forge.db, here.id);
90 finishRun(forge.db, here.id, { state: "ok", summary: "green" });
91 expect((await open(alice, here.id)).screen.toText()).not.toContain("Held by the cluster");
92 });
93
7794 test("a log the viewer may not read is not found and nothing of it is drawn", async () => {
7895 const run = createRun(forge.db, { repoId: repo.id, kind: "agent" });
7996 claimRun(forge.db, run.id);

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs