Public: anyone can read this; the crew can change it.
A check run as a cluster job holds the same egress wall as one on the box #13272
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/13251-a-check-run-as-a-cluster-job-holds-the-s → main 6 commits fork 3 threads · 3 unresolved
14 files
changed +518
−115
docs/APPS.md +7 −4
| @@ -913,10 +913,12 @@ records `placement` and the `node` its pod ran on. A malformed setting leaves | ||
| 913 | 913 | `process` in place. No cell wraps a Job. The pod mounts only its own |
| 914 | 914 | checkout (`subPath` = the run id), mounts no service-account token, runs |
| 915 | 915 | non-root without privilege escalation, and carries the check's limits as |
| 916 | `resources.limits`. **Egress is not enforced by the forge on a Job**: the | |
| 917 | run is recorded as uncontained ("egress not enforced on a cluster job"), and | |
| 918 | the pod carries the label `forge-check=true` so an operator's NetworkPolicy | |
| 919 | can select it and hold what `egress` declares. | |
| 916 | `resources.limits`. **Egress is held as in a cell**: the forge opens its | |
| 917 | egress proxy for the run on `FORGE_CHECKS_EGRESS_GATE` and points the pod's | |
| 918 | `HTTPS_PROXY` at it, so an undeclared host is refused in the forge's own | |
| 919 | words, and the run is recorded as enforced; the pod carries the label | |
| 920 | `forge-check=true` for the NetworkPolicy that leaves it no other way out | |
| 921 | (docs/CLUSTER.md). Without a gate no check runs as a Job. | |
| 920 | 922 | |
| 921 | 923 | Checks share `FORGE_CHECKS_CONCURRENCY` slots (default `1`), and when one |
| 922 | 924 | frees, it goes to the most recently requested check waiting, not the one |
| @@ -1811,6 +1813,7 @@ a link to the milestone (`docs/PLANNING.md`); its progress never does. | ||
| 1811 | 1813 | | `FORGE_CREW_CONCURRENCY` | 1 per 3 GiB above 2 GiB, 2–16 | agent runs at once (live: Settings → Capacity) | |
| 1812 | 1814 | | `FORGE_CHECKS_CONCURRENCY` | `1` | checks at once (live: Settings → Capacity) | |
| 1813 | 1815 | | `FORGE_CHECKS_PLACEMENT` | `process` | `job` runs a check's commands as a cluster Job (see Placement above) | |
| 1816 | | `FORGE_CHECKS_EGRESS_GATE` | unset | the address a Job's egress gate listens on; without it no check runs as a Job | | |
| 1814 | 1817 | | `FORGE_CHECKS_DRAIN_S` | `10` | seconds a roll lets running checks finish first | |
| 1815 | 1818 | | `FORGE_CREW_DRAIN_S` | `10` | seconds a roll lets crew runs finish before they are queued again | |
| 1816 | 1819 | | `FORGE_CHECK_SHARDS` | the machine's cores | test shards per check (live: Settings → Capacity) | |
docs/CLUSTER.md +32 −13
| @@ -50,31 +50,50 @@ Read once at boot by `src/checks/placement.ts`. A malformed value leaves | ||
| 50 | 50 | | `FORGE_CHECKS_CLAIM` | `forge-checks` | a ReadWriteMany claim holding the checks root; required | |
| 51 | 51 | | `FORGE_CHECKS_NODE_SELECTOR` | `pool=checks,disk=ssd` | `key=value,key=value`; optional | |
| 52 | 52 | | `FORGE_CHECKS_TOLERATIONS` | `[{"key":"dedicated","operator":"Equal","value":"checks","effect":"NoSchedule"}]` | JSON list, at most 20; optional | |
| 53 | | `FORGE_CHECKS_EGRESS_GATE` | `10.42.0.17` (the forge pod's IP) | the address the forge's per-run egress gate listens on and a Job's pod dials; without it no check runs as a Job | | |
| 53 | 54 | |
| 54 | ## Only while egress is reported | |
| 55 | ## Egress through the forge's gate | |
| 55 | 56 | |
| 56 | The lane applies only while `cells.egress` is `report`. Under `enforce` (the | |
| 57 | default) a Job cannot hold egress, so every check stays in a cell: the run | |
| 58 | records `placement: process` and its log says | |
| 59 | `placement: job skipped, egress is enforced and a Job cannot hold it`. The | |
| 60 | setting is read on every run, so an admin's change takes effect on the next | |
| 61 | check. | |
| 57 | A Job holds the same egress wall as a cell (#13251). For each run the forge | |
| 58 | opens the proxy a cell's only way out is (`apps/egress.ts`, through | |
| 59 | `src/checks/job_gate.ts`) on `FORGE_CHECKS_EGRESS_GATE`, at the first free | |
| 60 | port of 47100-47199 (`JOB_GATE_PORTS`: one per Job running at once), and | |
| 61 | gives the pod `HTTP_PROXY`/`HTTPS_PROXY` pointing at it (`NO_PROXY` empty). | |
| 62 | The proxy URL carries a secret made for that run; a request to the gate | |
| 63 | without it as `Proxy-Authorization` is answered `407` and dials nothing, so | |
| 64 | another pod that reaches the port cannot use the forge as its way out. It forwards to the forge, the registry and what `forge.json` | |
| 65 | `egress` declares, and refuses anything else with `403` and | |
| 66 | `refused by this forge: <host> is not in this app's forge.json egress`; an | |
| 67 | outside contributor's check is enforced whatever `cells.egress` says. The | |
| 68 | run records the gate with a note (`JOB_GATE_NOTE`) that the rest rests on | |
| 69 | the cluster's NetworkPolicy, which the forge cannot see, and its page says | |
| 70 | so. The gate closes when the Job ends. | |
| 62 | 71 | |
| 63 | ## Label for a NetworkPolicy | |
| 64 | ||
| 65 | The forge does not enforce egress on a Job. Every check pod carries the label | |
| 66 | `forge-check: "true"` (`JOB_LABELS` in `src/checks/job_step.ts`), so a | |
| 67 | NetworkPolicy can select it and hold what a check declares: | |
| 72 | The variables only point the way: the wall is a NetworkPolicy that lets a | |
| 73 | check pod reach the forge pod on the gate's ports and nothing else. Name the | |
| 74 | range, never the whole pod: on any port a check pod could dial the forge's | |
| 75 | own listeners around the gate. Keep nothing else of the forge pod listening | |
| 76 | in 47100-47199. Every check pod carries the label `forge-check: "true"` | |
| 77 | (`JOB_LABELS` in `src/checks/job_step.ts`): | |
| 68 | 78 | |
| 69 | 79 | yaml |
| 70 | 80 | podSelector: |
| 71 | 81 | matchLabels: { forge-check: "true" } |
| 82 | policyTypes: [Egress] | |
| 83 | egress: | |
| 84 | - to: [{ podSelector: { matchLabels: { app: forge } } }] | |
| 85 | ports: [{ protocol: TCP, port: 47100, endPort: 47199 }] | |
| 72 | 86 | |
| 73 | 87 | |
| 88 | Set `FORGE_CHECKS_EGRESS_GATE` only once that policy is in place. Without | |
| 89 | it the cluster cannot honour `egress`, so no check runs as a Job: each stays | |
| 90 | in a cell, records `placement: process`, and its log says | |
| 91 | `placement: job skipped, this cluster names no egress gate for a Job (FORGE_CHECKS_EGRESS_GATE)`. | |
| 92 | ||
| 74 | 93 | ## Confirming the lane is on |
| 75 | 94 | |
| 76 | 95 | 1. Push to a pull, or re-run its check. |
| 77 | 2. With `cells.egress` at `report`, the run's page says "ran as a cluster job on <node>"; a `process` run says | |
| 96 | 2. The run's page says "ran as a cluster job on <node>"; a `process` run says | |
| 78 | 97 | nothing. |
| 79 | 98 | 3. In the cluster, `kubectl get jobs,pods -l forge-check=true` shows the Job |
| 80 | 99 | while it runs (it is deleted when the run ends). |
src/apps/contain.ts +61 −20
| @@ -57,7 +57,10 @@ import { CELL_MEMORY_FLOOR_MIB, CELL_MEMORY_RESERVE_MIB, cellMemoryCeilingMiB } | ||
| 57 | 57 | import { cpuWatch } from "./cpu_rule.ts"; |
| 58 | 58 | import { dutyTick, newDuty, resumeNow, type Duty, type DutyLater, type Signaller } from "./cpu_duty.ts"; |
| 59 | 59 | |
| 60 | /** `ceiling`: a declaration was cut to the machine's maximum; `declared`: forge.json's own; `default`: neither. */ | |
| 60 | /** | |
| 61 | * `ceiling`: a declaration was cut to the machine's maximum; `declared`: forge.json's own; | |
| 62 | * `default`: neither. | |
| 63 | */ | |
| 61 | 64 | export type CeilingFrom = "env" | "pod" | "default" | "size"; |
| 62 | 65 | export type MemorySource = "ceiling" | "declared" | "default"; |
| 63 | 66 | |
| @@ -112,7 +115,10 @@ export const OWN_SIZE_SHARE = 3 / 4; | ||
| 112 | 115 | |
| 113 | 116 | const ownShare = (mib: number): number => Math.max(1, Math.floor(mib * OWN_SIZE_SHARE)); |
| 114 | 117 | |
| 115 | /** This forge's size when it is a child in a cell (`FORGE_CELL_SIZE`, set by its parent: `cellEnv`); else none. */ | |
| 118 | /** | |
| 119 | * This forge's size when it is a child in a cell (`FORGE_CELL_SIZE`, set by its parent: `cellEnv`); | |
| 120 | * else none. | |
| 121 | */ | |
| 116 | 122 | function ownSize(): Size | undefined { |
| 117 | 123 | const raw = process.env["FORGE_CELL_SIZE"]; |
| 118 | 124 | return isSize(raw) ? raw : undefined; |
| @@ -186,7 +192,10 @@ export function memoryCeilingSource( | ||
| 186 | 192 | return ceiling(env, podFile, size).from; |
| 187 | 193 | } |
| 188 | 194 | |
| 189 | /** The machine's ceiling, held under {@link OWN_SIZE_SHARE} of the forge's own size when it is a child. */ | |
| 195 | /** | |
| 196 | * The machine's ceiling, held under {@link OWN_SIZE_SHARE} of the forge's own size when it is a | |
| 197 | * child. | |
| 198 | */ | |
| 190 | 199 | function ceiling( |
| 191 | 200 | env: string | undefined, |
| 192 | 201 | podFile: string, |
| @@ -253,7 +262,9 @@ function positive(raw: string | undefined): number | undefined { | ||
| 253 | 262 | return Number.isFinite(n) && n > 0 ? n : undefined; |
| 254 | 263 | } |
| 255 | 264 | |
| 256 | /** The defaults for `kind`, raised or lowered by a declaration, never past the machine's maximum. */ | |
| 265 | /** | |
| 266 | * The defaults for `kind`, raised or lowered by a declaration, never past the machine's maximum. | |
| 267 | */ | |
| 257 | 268 | export function resolveLimits( |
| 258 | 269 | kind: "app" | "step", |
| 259 | 270 | declared: DeclaredLimits | undefined, |
| @@ -323,7 +334,9 @@ export interface CellSpec { | ||
| 323 | 334 | /** Origins always reachable: the forge's own URL. */ |
| 324 | 335 | readonly reach: readonly string[]; |
| 325 | 336 | readonly limits: Limits; |
| 326 | /** Read-only paths to keep visible when they sit under a hidden directory (a forge's own code). */ | |
| 337 | /** | |
| 338 | * Read-only paths to keep visible when they sit under a hidden directory (a forge's own code). | |
| 339 | */ | |
| 327 | 340 | readonly reveal?: readonly string[] | undefined; |
| 328 | 341 | /** The name the cell's processes see as the host's (`cellHostname`); "cell" when unset. */ |
| 329 | 342 | readonly hostname?: string | undefined; |
| @@ -333,7 +346,10 @@ export interface CellSpec { | ||
| 333 | 346 | readonly terminal?: boolean | undefined; |
| 334 | 347 | /** Told of every undeclared host let through while egress is only reported. */ |
| 335 | 348 | readonly onUndeclared?: ((host: string, port: number) => void) | undefined; |
| 336 | /** Egress entries allowed since the cell started, asked on every connection (`apps/egress_refused.ts`). */ | |
| 349 | /** | |
| 350 | * Egress entries allowed since the cell started, asked on every connection | |
| 351 | * (`apps/egress_refused.ts`). | |
| 352 | */ | |
| 337 | 353 | readonly allowed?: (() => readonly string[]) | undefined; |
| 338 | 354 | /** Told of every refusal by policy, host and port, and whether it is private. */ |
| 339 | 355 | readonly onRefused?: ((host: string, port: number, isPrivate: boolean) => void) | undefined; |
| @@ -395,7 +411,10 @@ export interface Cell { | ||
| 395 | 411 | readonly breach: () => string | undefined; |
| 396 | 412 | /** Stop watching and close the cell's sockets. */ |
| 397 | 413 | readonly close: () => void; |
| 398 | /** The cell's own cgroup directory, where the kernel holds its caps; absent when only the watch does. */ | |
| 414 | /** | |
| 415 | * The cell's own cgroup directory, where the kernel holds its caps; absent when only the watch | |
| 416 | * does. | |
| 417 | */ | |
| 399 | 418 | readonly cgroupDir?: string; |
| 400 | 419 | } |
| 401 | 420 | |
| @@ -614,7 +633,8 @@ for p in $loops; do (socat TCP-LISTEN:$p,bind=127.0.0.1,fork,reuseaddr UNIX-CONN | ||
| 614 | 633 | i=0; while [ $i -lt 200 ] && ! grep -q ':${PROXY_PORT.toString(16).toUpperCase().padStart(4, "0")} 00000000:0000 0A' /proc/net/tcp 2>/dev/null; do sleep 0.01; i=$((i+1)); done |
| 615 | 634 | exec "$@"`; |
| 616 | 635 | |
| 617 | function origin(url: string): { host: string; port: number } | undefined { | |
| 636 | /** A URL's host and port, as the egress proxy holds an origin. */ | |
| 637 | export function origin(url: string): { host: string; port: number } | undefined { | |
| 618 | 638 | try { |
| 619 | 639 | const u = new URL(url); |
| 620 | 640 | const port = Number(u.port || (u.protocol === "https:" ? 443 : 80)); |
| @@ -644,8 +664,9 @@ export function cellMounts( | ||
| 644 | 664 | kind: (path: string) => "dir" | "file" | undefined = pathKind, |
| 645 | 665 | real: (path: string) => string = realPath, |
| 646 | 666 | ): { hide: string[]; blank: string[]; reveal: string[] } { |
| 647 | // Each under the name it was given and under the name it really has: a data directory configured through | |
| 648 | // a link is the same directory at its real path, and a path is compared as text from here on. | |
| 667 | // Each under the name it was given and under the name it really has: a data directory configured | |
| 668 | // through a link is the same directory at its real path, and a path is compared as text from here | |
| 669 | // on. | |
| 649 | 670 | const hidden = [...new Set([...spec.hide, ...machine].flatMap((path) => [path, real(path)]))] |
| 650 | 671 | // A path that is not there has nothing to hide (and cannot be mounted |
| 651 | 672 | // over on a read-only root). |
| @@ -660,7 +681,8 @@ export function cellMounts( | ||
| 660 | 681 | const blank = hidden.filter((path) => kind(path) === "file"); |
| 661 | 682 | // The runtime itself must stay visible when it lives in a hidden home |
| 662 | 683 | // (`~/.bun/bin/bun` on a developer's machine). |
| 663 | // Kept when it sits under something hidden by either name: the forge's own code is known by its real path. | |
| 684 | // Kept when it sits under something hidden by either name: the forge's own code is known by its | |
| 685 | // real path. | |
| 664 | 686 | const reveal = [dirname(process.execPath), ...(spec.reveal ?? [])].filter((path) => |
| 665 | 687 | hide.some((h) => within(path, h) || within(real(path), h)), |
| 666 | 688 | ); |
| @@ -676,7 +698,10 @@ function realPath(path: string): string { | ||
| 676 | 698 | } |
| 677 | 699 | } |
| 678 | 700 | |
| 679 | /** The paths of `asked` that lie inside no other: each of the rest is already writable through the one it lies in. */ | |
| 701 | /** | |
| 702 | * The paths of `asked` that lie inside no other: each of the rest is already writable through the | |
| 703 | * one it lies in. | |
| 704 | */ | |
| 680 | 705 | function bindRoots(asked: readonly string[]): string[] { |
| 681 | 706 | return asked.filter((path) => !asked.some((other) => other !== path && within(path, other))); |
| 682 | 707 | } |
| @@ -883,8 +908,8 @@ export function openCell( | ||
| 883 | 908 | const asked = [...new Set([spec.dir, spec.home, ...spec.writable])]; |
| 884 | 909 | const roots = bindRoots(asked); |
| 885 | 910 | for (const path of roots) mkdirSync(path, { recursive: true }); |
| 886 | // A place inside one of those (a child forge's home, in its own directory) is made if it can be; what is | |
| 887 | // there under its name is the cell's own business, and is never mounted by name. | |
| 911 | // A place inside one of those (a child forge's home, in its own directory) is made if it can be; | |
| 912 | // what is there under its name is the cell's own business, and is never mounted by name. | |
| 888 | 913 | for (const path of asked) { |
| 889 | 914 | if (roots.includes(path)) continue; |
| 890 | 915 | try { |
| @@ -1120,7 +1145,10 @@ function readMountinfo(): string { | ||
| 1120 | 1145 | |
| 1121 | 1146 | /** How often the process table is read, and over how long CPU is averaged. */ |
| 1122 | 1147 | export const WATCH = { tickMs: 1_000, cpuWindowMs: 30_000, diskEveryTicks: 15 }; |
| 1123 | /** A cell's process limit: the watch's every tick, and the kernel's `pids.max` where pids is delegated. */ | |
| 1148 | /** | |
| 1149 | * A cell's process limit: the watch's every tick, and the kernel's `pids.max` where pids is | |
| 1150 | * delegated. | |
| 1151 | */ | |
| 1124 | 1152 | export const PROCESSES_MAX = 512; |
| 1125 | 1153 | const CLK_TCK = 100; |
| 1126 | 1154 | |
| @@ -1133,7 +1161,10 @@ interface Watched { | ||
| 1133 | 1161 | ticks: number; |
| 1134 | 1162 | /** A disk walk is out for this cell. */ |
| 1135 | 1163 | measuring?: boolean; |
| 1136 | /** When the cell began to be watched, and since when its cores have held over the stop line without a break (#11657). */ | |
| 1164 | /** | |
| 1165 | * When the cell began to be watched, and since when its cores have held over the stop line | |
| 1166 | * without a break (#11657). | |
| 1167 | */ | |
| 1137 | 1168 | readonly startedAt: number; |
| 1138 | 1169 | overSince: number | null; |
| 1139 | 1170 | readonly key: number; |
| @@ -1157,7 +1188,9 @@ const watched = new Set<Watched>(); | ||
| 1157 | 1188 | let timer: ReturnType<typeof setInterval> | undefined; |
| 1158 | 1189 | let nextKey = 1; |
| 1159 | 1190 | |
| 1160 | /** A cell's memory cap, bytes, and who holds it: the kernel (`apps/cgroup.ts`) or the watch alone. */ | |
| 1191 | /** | |
| 1192 | * A cell's memory cap, bytes, and who holds it: the kernel (`apps/cgroup.ts`) or the watch alone. | |
| 1193 | */ | |
| 1161 | 1194 | export interface MemoryCap { |
| 1162 | 1195 | readonly bytes: number; |
| 1163 | 1196 | readonly by: "kernel" | "watch"; |
| @@ -1169,7 +1202,9 @@ export function memoryCapHolder(): { by: "kernel" } | { by: "watch"; reason: str | ||
| 1169 | 1202 | return host.ok ? { by: "kernel" } : { by: "watch", reason: host.reason }; |
| 1170 | 1203 | } |
| 1171 | 1204 | |
| 1172 | /** Whether cells' process counts are the kernel's here (`pids.max`), or why they are watched only. */ | |
| 1205 | /** | |
| 1206 | * Whether cells' process counts are the kernel's here (`pids.max`), or why they are watched only. | |
| 1207 | */ | |
| 1173 | 1208 | export function processCapHolder(host: CgroupHost = cgroupHost()): { by: "kernel" } | { by: "watch"; reason: string } { |
| 1174 | 1209 | if (!host.ok) return { by: "watch", reason: host.reason }; |
| 1175 | 1210 | return pidsDelegated(host) |
| @@ -1424,7 +1459,10 @@ export function diskEntriesMax(diskMiB: number): number { | ||
| 1424 | 1459 | return Number.isFinite(diskMiB) && diskMiB > 0 ? Math.max(DISK_ENTRIES_FLOOR, Math.floor(diskMiB * 256)) : DISK_ENTRIES_FLOOR; |
| 1425 | 1460 | } |
| 1426 | 1461 | |
| 1427 | /** Why a cell is stopped for its disk, from what a walk of its directories found; undefined while it is within its limit. */ | |
| 1462 | /** | |
| 1463 | * Why a cell is stopped for its disk, from what a walk of its directories found; undefined while it | |
| 1464 | * is within its limit. | |
| 1465 | */ | |
| 1428 | 1466 | export function diskBreach(walk: DiskWalk, diskMiB: number): string | undefined { |
| 1429 | 1467 | if (walk.capped) |
| 1430 | 1468 | return `holds more than ${diskEntriesMax(diskMiB).toLocaleString("en-US")} files and directories: too many to count toward its ${diskMiB} MiB of disk`; |
| @@ -1663,7 +1701,10 @@ function decide(now: number, trees: readonly TreeReading[]): void { | ||
| 1663 | 1701 | } |
| 1664 | 1702 | } |
| 1665 | 1703 | |
| 1666 | /** The root process of every labelled cell being watched, for grouping the process table by what launched it (`system/view.ts`). */ | |
| 1704 | /** | |
| 1705 | * The root process of every labelled cell being watched, for grouping the process table by what | |
| 1706 | * launched it (`system/view.ts`). | |
| 1707 | */ | |
| 1667 | 1708 | export function cellRoots(): { label: string; root: number; cap: MemoryCap }[] { |
| 1668 | 1709 | const out: { label: string; root: number; cap: MemoryCap }[] = []; |
| 1669 | 1710 | for (const e of watched) if (e.label !== undefined) out.push({ label: e.label, root: e.root, cap: e.cap }); |
src/apps/egress.ts +62 −13
| @@ -20,6 +20,7 @@ | ||
| 20 | 20 | import { storedText } from "../kernel/blocked.ts"; |
| 21 | 21 | import { lookup } from "node:dns/promises"; |
| 22 | 22 | import { isIP } from "node:net"; |
| 23 | import { timingSafeEqual } from "node:crypto"; | |
| 23 | 24 | import type { Socket } from "bun"; |
| 24 | 25 | import { neverGrantable, PAYMENT_HOSTS } from "../core/egress_grants.ts"; |
| 25 | 26 | import { addressBytes } from "../kernel/cidr.ts"; |
| @@ -43,15 +44,24 @@ export interface EgressPolicy { | ||
| 43 | 44 | readonly report?: ((host: string, port: number) => void) | undefined; |
| 44 | 45 | /** Whose proxy this is (`app:<slug>/<env>`), for the connection gauges. */ |
| 45 | 46 | readonly label?: string | undefined; |
| 46 | /** More `egress` entries, asked on every connection: what an owner allowed since the cell started (#10456). */ | |
| 47 | /** | |
| 48 | * More `egress` entries, asked on every connection: what an owner allowed since the cell started | |
| 49 | * (#10456). | |
| 50 | */ | |
| 47 | 51 | readonly allowed?: (() => readonly string[]) | undefined; |
| 48 | /** Told of every connection refused by policy (not one that merely failed), with whether the destination is private. */ | |
| 52 | /** | |
| 53 | * Told of every connection refused by policy (not one that merely failed), with whether the | |
| 54 | * destination is private. | |
| 55 | */ | |
| 49 | 56 | readonly refused?: ((host: string, port: number, isPrivate: boolean) => void) | undefined; |
| 50 | 57 | /** Most connections open to one destination at once; default 8. */ |
| 51 | 58 | readonly maxPerDestination?: number | undefined; |
| 52 | 59 | /** Close a connection that moved no bytes for this long; default 5 min. */ |
| 53 | 60 | readonly idleMs?: number | undefined; |
| 54 | /** Bytes held for a slow client; a quarter of it pauses the upstream, and past all of it once paused the client is cut. Default 4 MiB. */ | |
| 61 | /** | |
| 62 | * Bytes held for a slow client; a quarter of it pauses the upstream, and past all of it once | |
| 63 | * paused the client is cut. Default 4 MiB. | |
| 64 | */ | |
| 55 | 65 | readonly queueMax?: number | undefined; |
| 56 | 66 | /** |
| 57 | 67 | * The proxy of the cell this forge itself runs in (`upstreamProxy`): a |
| @@ -61,8 +71,29 @@ export interface EgressPolicy { | ||
| 61 | 71 | * resolves). |
| 62 | 72 | */ |
| 63 | 73 | readonly upstream?: { readonly host: string; readonly port: number } | undefined; |
| 64 | /** Name to addresses: the system resolver unless a test passes its own. Never read from the environment. */ | |
| 74 | /** | |
| 75 | * Name to addresses: the system resolver unless a test passes its own. Never read from the | |
| 76 | * environment. | |
| 77 | */ | |
| 65 | 78 | readonly resolve?: ((host: string) => Promise<readonly string[]>) | undefined; |
| 79 | /** | |
| 80 | * The `Proxy-Authorization` value every request must carry (a Job's gate on a network others | |
| 81 | * share, `checks/job_gate.ts`); without it the proxy answers `407` and dials nothing. Unset for a | |
| 82 | * cell, whose socket only the cell can reach. | |
| 83 | */ | |
| 84 | readonly credential?: string | undefined; | |
| 85 | } | |
| 86 | ||
| 87 | /** | |
| 88 | * Whether a request head's `Proxy-Authorization` is exactly `credential`, compared in constant | |
| 89 | * time. | |
| 90 | */ | |
| 91 | function authorized(headers: readonly string[], credential: string): boolean { | |
| 92 | const line = headers.find((h) => /^proxy-authorization:/i.test(h)); | |
| 93 | if (line === undefined) return false; | |
| 94 | const given = encoder.encode(line.slice(line.indexOf(":") + 1).trim()); | |
| 95 | const want = encoder.encode(credential); | |
| 96 | return given.length === want.length && timingSafeEqual(given, want); | |
| 66 | 97 | } |
| 67 | 98 | |
| 68 | 99 | /** |
| @@ -100,6 +131,8 @@ export function cellProxy(env: Record<string, string | undefined> = process.env) | ||
| 100 | 131 | } |
| 101 | 132 | |
| 102 | 133 | export interface Egress { |
| 134 | /** The port listened on, for a `{ tcp }` listener; absent on a socket. */ | |
| 135 | readonly port?: number | undefined; | |
| 103 | 136 | close(): void; |
| 104 | 137 | } |
| 105 | 138 | |
| @@ -108,7 +141,9 @@ const MAX_PER_DESTINATION = 8; | ||
| 108 | 141 | /** A public destination's cap: an install opens dozens of parallel fetches to one registry. */ |
| 109 | 142 | export const MAX_PER_PUBLIC_DESTINATION = 64; |
| 110 | 143 | const IDLE_MS = 5 * 60_000; |
| 111 | /** Bytes held for a client that does not read before its upstream is cut, once the pause has landed. */ | |
| 144 | /** | |
| 145 | * Bytes held for a client that does not read before its upstream is cut, once the pause has landed. | |
| 146 | */ | |
| 112 | 147 | const QUEUE_MAX = 4 * 1024 * 1024; |
| 113 | 148 | /** |
| 114 | 149 | * A backlog past this share of `queueMax` pauses the side that fills it until |
| @@ -303,7 +338,10 @@ export interface NpmTopRepo { | ||
| 303 | 338 | readonly bytes: number; |
| 304 | 339 | } |
| 305 | 340 | |
| 306 | /** The metered repository the registry sent most to in the last hour; none when no metered install fetched. */ | |
| 341 | /** | |
| 342 | * The metered repository the registry sent most to in the last hour; none when no metered install | |
| 343 | * fetched. | |
| 344 | */ | |
| 307 | 345 | export function npmTopRepoLastHour(now = Date.now()): NpmTopRepo | undefined { |
| 308 | 346 | const minute = Math.floor(now / 60_000); |
| 309 | 347 | let top: NpmTopRepo | undefined; |
| @@ -362,7 +400,10 @@ export function resetEgressTotals(): void { | ||
| 362 | 400 | /** The ports a declared host may be reached on when its entry names none. */ |
| 363 | 401 | export const WEB_PORTS: readonly number[] = [80, 443]; |
| 364 | 402 | |
| 365 | /** What stands in front of `--<host>` in a hosted forge's name, or its app's: one DNS label's worth, never a dot. */ | |
| 403 | /** | |
| 404 | * What stands in front of `--<host>` in a hosted forge's name, or its app's: one DNS label's worth, | |
| 405 | * never a dot. | |
| 406 | */ | |
| 366 | 407 | const BESIDE_LABEL = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?$/; |
| 367 | 408 | |
| 368 | 409 | /** |
| @@ -625,6 +666,11 @@ async function open( | ||
| 625 | 666 | // tunnel, the rewritten head for a forwarded request. |
| 626 | 667 | let prefix = new Uint8Array(0); |
| 627 | 668 | let tunnel = false; |
| 669 | if (policy.credential !== undefined && !authorized(headers, policy.credential)) { | |
| 670 | log("egress refused: no credential for this gate"); | |
| 671 | refuse(client, "407 Proxy Authentication Required", "this gate takes only its own run's credential"); | |
| 672 | return; | |
| 673 | } | |
| 628 | 674 | if (method.toUpperCase() === "CONNECT") { |
| 629 | 675 | const at = target.lastIndexOf(":"); |
| 630 | 676 | host = target.slice(0, at); |
| @@ -677,8 +723,9 @@ async function open( | ||
| 677 | 723 | } |
| 678 | 724 | const destination = `${host.toLowerCase()}:${port}`; |
| 679 | 725 | const count = tally(policy.label ?? "", destination); |
| 680 | // The cap guards a granted private device (a Pi's camera, #3719); a public host such as the package | |
| 681 | // registry is reached by many parallel requests from one install, so it gets the wider cap. | |
| 726 | // The cap guards a granted private device (a Pi's camera, #3719); a public host such as the | |
| 727 | // package registry is reached by many parallel requests from one install, so it gets the wider | |
| 728 | // cap. | |
| 682 | 729 | const cap = policy.maxPerDestination ?? (policy.upstream !== undefined || isPublicAddress(verdict.address) ? MAX_PER_PUBLIC_DESTINATION : MAX_PER_DESTINATION); |
| 683 | 730 | if (count.open >= cap) { |
| 684 | 731 | log(`egress capped: ${policy.label ?? "app"} already has ${count.open} connections to ${destination}`); |
| @@ -804,11 +851,12 @@ async function open( | ||
| 804 | 851 | } |
| 805 | 852 | |
| 806 | 853 | /** |
| 807 | * Listen at `path` and forward only what `policy` allows. `{ tcp }` (a | |
| 808 | * loopback port) exists for the tests, where a unix socket cannot be bound. | |
| 854 | * Listen at `path` and forward only what `policy` allows. `{ tcp }` is a | |
| 855 | * port on loopback (the tests, where a unix socket cannot be bound) or on | |
| 856 | * `hostname`: the address a cluster Job's pod dials (`checks/job_gate.ts`). | |
| 809 | 857 | */ |
| 810 | 858 | export function startEgress( |
| 811 | path: string | { tcp: number }, | |
| 859 | path: string | { tcp: number; hostname?: string }, | |
| 812 | 860 | policy: EgressPolicy, |
| 813 | 861 | log: (line: string) => void = () => {}, |
| 814 | 862 | ): Egress { |
| @@ -875,7 +923,7 @@ export function startEgress( | ||
| 875 | 923 | const server = |
| 876 | 924 | typeof path === "string" |
| 877 | 925 | ? Bun.listen<Conn>({ unix: path, socket: handlers }) |
| 878 | : Bun.listen<Conn>({ hostname: "127.0.0.1", port: path.tcp, socket: handlers }); | |
| 926 | : Bun.listen<Conn>({ hostname: path.hostname ?? "127.0.0.1", port: path.tcp, socket: handlers }); | |
| 879 | 927 | // A tunnel that moves nothing (a client that stopped reading, an upstream |
| 880 | 928 | // that stopped sending) is cut, so it cannot hold an upstream open. |
| 881 | 929 | const idleMs = policy.idleMs ?? IDLE_MS; |
| @@ -894,6 +942,7 @@ export function startEgress( | ||
| 894 | 942 | ); |
| 895 | 943 | sweep.unref(); |
| 896 | 944 | return { |
| 945 | port: "port" in server ? server.port : undefined, | |
| 897 | 946 | close() { |
| 898 | 947 | clearInterval(sweep); |
| 899 | 948 | for (const client of live) { |
src/checks/job_gate.ts added +91
| @@ -0,0 +1,91 @@ | ||
| 1 | /** | |
| 2 | * The egress wall around a check run as a cluster Job (#13251): the same | |
| 3 | * proxy a cell's only way out is (`apps/egress.ts`), opened for one run on | |
| 4 | * the address the Job's pod dials (`FORGE_CHECKS_EGRESS_GATE`), and the | |
| 5 | * proxy variables that point the pod at it. It forwards to the forge and to | |
| 6 | * what `forge.json` `egress` declares, and refuses everything else with | |
| 7 | * "refused by this forge: <host> is not in this app's forge.json egress". | |
| 8 | * The operator's NetworkPolicy (docs/CLUSTER.md) lets a `forge-check` pod | |
| 9 | * reach nothing but this gate's ports, so the variables are not a courtesy. | |
| 10 | * | |
| 11 | * The address is on a network other pods share, so the gate takes only its | |
| 12 | * own run: a secret made for the run rides in the proxy URL the pod gets, | |
| 13 | * and a request without it as `Proxy-Authorization` is answered `407`. | |
| 14 | * It listens on a port of {@link JOB_GATE_PORTS}, a fixed range the | |
| 15 | * NetworkPolicy names, so the policy need not open the forge's other ports. | |
| 16 | */ | |
| 17 | import { randomBytes } from "node:crypto"; | |
| 18 | import { PROXY_VARS, startEgress, type EgressPolicy } from "../apps/egress.ts"; | |
| 19 | import { err, ok, type Result } from "../kernel/result.ts"; | |
| 20 | ||
| 21 | /** | |
| 22 | * What a Job's run records of its boundary. The forge holds the gate and the | |
| 23 | * proxy variables; that the pod can reach nothing else is the cluster's | |
| 24 | * NetworkPolicy, which the forge cannot see, and the note says so. | |
| 25 | */ | |
| 26 | export const JOB_GATE_NOTE = "The forge holds its egress; the cluster's NetworkPolicy holds the rest."; | |
| 27 | ||
| 28 | /** | |
| 29 | * The ports a gate listens on, one per Job running at once; docs/CLUSTER.md's policy names them. | |
| 30 | */ | |
| 31 | export const JOB_GATE_PORTS = { from: 47100, to: 47199 } as const; | |
| 32 | ||
| 33 | /** The user name in a gate's proxy URL; the secret is its password. */ | |
| 34 | const GATE_USER = "forge"; | |
| 35 | ||
| 36 | /** | |
| 37 | * The gate's policy, as a cell's: the forge, the registry and what forge.json | |
| 38 | * declares. Undeclared hosts are let through and noted only under | |
| 39 | * `cells.egress=report`, and never for an outside contributor's check. | |
| 40 | */ | |
| 41 | export function jobGatePolicy(input: { | |
| 42 | readonly hosts: readonly string[]; | |
| 43 | readonly origins: readonly { host: string; port: number }[]; | |
| 44 | readonly label: string; | |
| 45 | readonly outside: boolean; | |
| 46 | readonly setting: "enforce" | "report"; | |
| 47 | readonly note: (host: string, port: number) => void; | |
| 48 | }): EgressPolicy { | |
| 49 | return { | |
| 50 | hosts: input.hosts, | |
| 51 | origins: input.origins, | |
| 52 | label: input.label, | |
| 53 | report: !input.outside && input.setting === "report" ? input.note : undefined, | |
| 54 | }; | |
| 55 | } | |
| 56 | ||
| 57 | export type JobGate = { | |
| 58 | /** The variables the Job's container gets, beside its `CI` and `FORGE_*`. */ | |
| 59 | readonly env: Record<string, string>; | |
| 60 | close(): void; | |
| 61 | }; | |
| 62 | ||
| 63 | /** | |
| 64 | * Open the gate on `address`, at the first free port of `ports`; the reason when it cannot | |
| 65 | * listen. | |
| 66 | */ | |
| 67 | export function openJobGate( | |
| 68 | address: string, | |
| 69 | policy: EgressPolicy, | |
| 70 | log: (line: string) => void = () => {}, | |
| 71 | ports: { readonly from: number; readonly to: number } = JOB_GATE_PORTS, | |
| 72 | ): Result<JobGate, string> { | |
| 73 | const secret = randomBytes(24).toString("hex"); | |
| 74 | const held: EgressPolicy = { ...policy, credential: `Basic ${btoa(`${GATE_USER}:${secret}`)}` }; | |
| 75 | let gate; | |
| 76 | let last = "no port in the range"; | |
| 77 | for (let port = ports.from; port <= ports.to && gate === undefined; port++) { | |
| 78 | try { | |
| 79 | gate = startEgress({ tcp: port, hostname: address }, held, log); | |
| 80 | } catch (error) { | |
| 81 | last = error instanceof Error ? error.message : String(error); | |
| 82 | } | |
| 83 | } | |
| 84 | if (gate === undefined) | |
| 85 | return err(`the egress gate could not listen on ${address}, ports ${ports.from}-${ports.to}: ${last}`); | |
| 86 | const url = `http://${GATE_USER}:${secret}@${address}:${gate.port}`; | |
| 87 | const env: Record<string, string> = {}; | |
| 88 | for (const name of PROXY_VARS) env[name] = name.toLowerCase() === "no_proxy" ? "" : url; | |
| 89 | const open = gate; | |
| 90 | return ok({ env, close: () => open.close() }); | |
| 91 | } |
src/checks/job_step.ts +7 −1
| @@ -28,11 +28,17 @@ export async function runJobStep( | ||
| 28 | 28 | /** The Job's own output, apart from the echoed commands. */ |
| 29 | 29 | output: string[]; |
| 30 | 30 | onNode: (node: string) => void; |
| 31 | /** The proxy variables of this run's egress gate (`checks/job_gate.ts`). */ | |
| 32 | proxy?: Record<string, string>; | |
| 31 | 33 | options?: JobOptions; |
| 32 | 34 | }, |
| 33 | 35 | ): Promise<Result<void, string>> { |
| 34 | 36 | // Only the check's own variables cross to the cluster, never the forge's. |
| 35 | const env = Object.fromEntries(Object.entries(input.env).filter(([k]) => k === "CI" || k.startsWith("FORGE_"))); | |
| 37 | // Its way out is the run's egress gate, and nothing else (#13251). | |
| 38 | const env = { | |
| 39 | ...Object.fromEntries(Object.entries(input.env).filter(([k]) => k === "CI" || k.startsWith("FORGE_"))), | |
| 40 | ...input.proxy, | |
| 41 | }; | |
| 36 | 42 | const spec: JobSpec = { |
| 37 | 43 | name: `check-${input.runId.toLowerCase().replace(/[^a-z0-9]/g, "-")}`.slice(0, 53).replace(/-+$/, ""), |
| 38 | 44 | image: placement.image, |
src/checks/placement.ts +20 −1
| @@ -8,6 +8,9 @@ | ||
| 8 | 8 | * FORGE_CHECKS_CLAIM a ReadWriteMany claim holding the checks root |
| 9 | 9 | * FORGE_CHECKS_NODE_SELECTOR key=value,key=value |
| 10 | 10 | * FORGE_CHECKS_TOLERATIONS a JSON list of tolerations |
| 11 | * FORGE_CHECKS_EGRESS_GATE the forge's address a Job's pod dials for its | |
| 12 | * egress (`checks/job_gate.ts`); without it no | |
| 13 | * check runs as a Job | |
| 11 | 14 | */ |
| 12 | 15 | import type { Toleration } from "../cluster/jobs.ts"; |
| 13 | 16 | |
| @@ -19,8 +22,13 @@ export type Placement = | ||
| 19 | 22 | readonly claim: string; |
| 20 | 23 | readonly nodeSelector: Record<string, string>; |
| 21 | 24 | readonly tolerations: Toleration[]; |
| 25 | /** The address the forge's per-run egress proxy listens on and the pod dials. */ | |
| 26 | readonly gate?: string; | |
| 22 | 27 | }; |
| 23 | 28 | |
| 29 | /** An IPv4 address or a DNS name: what a pod can dial and the forge can bind. */ | |
| 30 | const GATE = /^(?=.{1,253}$)[A-Za-z0-9]([-A-Za-z0-9]{0,61}[A-Za-z0-9])?(\.[A-Za-z0-9]([-A-Za-z0-9]{0,61}[A-Za-z0-9])?)*$/; | |
| 31 | ||
| 24 | 32 | const LABEL = /^[A-Za-z0-9]([-A-Za-z0-9_./]{0,61}[A-Za-z0-9])?$/; |
| 25 | 33 | const EFFECTS = ["NoSchedule", "PreferNoSchedule", "NoExecute"]; |
| 26 | 34 | |
| @@ -73,8 +81,10 @@ export function placementFromEnv(env: Record<string, string | undefined> = proce | ||
| 73 | 81 | const nodeSelector = parseSelector(env["FORGE_CHECKS_NODE_SELECTOR"] ?? ""); |
| 74 | 82 | const tolerations = parseTolerations(env["FORGE_CHECKS_TOLERATIONS"]?.trim() || "[]"); |
| 75 | 83 | // A job placement that cannot be built is not guessed at. |
| 84 | const gate = env["FORGE_CHECKS_EGRESS_GATE"]?.trim() ?? ""; | |
| 76 | 85 | if (image === "" || /\s/.test(image) || claim === "" || !nodeSelector || !tolerations) return { mode: "process" }; |
| 77 | return { mode: "job", image, claim, nodeSelector, tolerations }; | |
| 86 | if (gate !== "" && !GATE.test(gate)) return { mode: "process" }; | |
| 87 | return { mode: "job", image, claim, nodeSelector, tolerations, ...(gate === "" ? {} : { gate }) }; | |
| 78 | 88 | } |
| 79 | 89 | |
| 80 | 90 | /** |
| @@ -88,3 +98,12 @@ export function placementWords(run: { placement: string; node: string }): string | ||
| 88 | 98 | const node = run.node.replace(/[^\x21-\x7e]/g, "").slice(0, 253); |
| 89 | 99 | return node === "" ? "ran as a cluster job" : `ran as a cluster job on ${node}`; |
| 90 | 100 | } |
| 101 | ||
| 102 | /** | |
| 103 | * The sandbox words for a check that ran as a Job, the page's and the terminal's (#13251): two | |
| 104 | * plain sentences, the second what the run recorded of its wall (`JOB_GATE_NOTE`). | |
| 105 | */ | |
| 106 | export function jobSandboxWords(note: string): string { | |
| 107 | const rest = note.replace(/[^\x20-\x7e]/g, "").trim(); | |
| 108 | return rest === "" ? "Held by the cluster: the check ran as a Job." : `Held by the cluster: the check ran as a Job. ${rest}`; | |
| 109 | } |
src/checks/service.ts +85 −38
| @@ -124,6 +124,7 @@ import { | ||
| 124 | 124 | } from "../core/capacity.ts"; |
| 125 | 125 | import { |
| 126 | 126 | cellsAvailable, |
| 127 | origin, | |
| 127 | 128 | limitWarnings, |
| 128 | 129 | resolveLimits, |
| 129 | 130 | STEP_HOSTS, |
| @@ -131,7 +132,8 @@ import { | ||
| 131 | 132 | type CellSpec, |
| 132 | 133 | } from "../apps/contain.ts"; |
| 133 | 134 | import { summariseCheckFailure } from "./summary.ts"; |
| 134 | import type { Result } from "../kernel/result.ts"; | |
| 135 | import { err, type Result } from "../kernel/result.ts"; | |
| 136 | import { JOB_GATE_NOTE, jobGatePolicy, openJobGate } from "./job_gate.ts"; | |
| 135 | 137 | import { placementFromEnv, type Placement } from "./placement.ts"; |
| 136 | 138 | import { refuseSmallPod, type PodRoom } from "./pod_room.ts"; |
| 137 | 139 | import { JOB_VANISHED, runJobStep } from "./job_step.ts"; |
| @@ -198,7 +200,10 @@ export interface ChecksOptions { | ||
| 198 | 200 | readonly job?: JobOptions | undefined; |
| 199 | 201 | /** How long a transient install failure waits for its retry. @default INSTALL_RETRY_MS */ |
| 200 | 202 | readonly installRetryMs?: number | undefined; |
| 201 | /** Runs the check step itself; tests stand one in for a stop the cell's watch makes (#11617). @default runStep */ | |
| 203 | /** | |
| 204 | * Runs the check step itself; tests stand one in for a stop the cell's watch makes | |
| 205 | * (#11617). @default runStep | |
| 206 | */ | |
| 202 | 207 | readonly checkStep?: typeof runStep | undefined; |
| 203 | 208 | /** |
| 204 | 209 | * Where the pod's memory.max is read and the forge's size |
| @@ -232,7 +237,10 @@ export interface GateDone { | ||
| 232 | 237 | } |
| 233 | 238 | |
| 234 | 239 | export interface ChecksHandle { |
| 235 | /** Ask for a check of one commit; the answer is the run, or nothing when the commit declares no check. */ | |
| 240 | /** | |
| 241 | * Ask for a check of one commit; the answer is the run, or nothing when the commit declares no | |
| 242 | * check. | |
| 243 | */ | |
| 236 | 244 | request( |
| 237 | 245 | repo: Repo, |
| 238 | 246 | env: string, |
| @@ -450,7 +458,10 @@ const STORES: ReadonlySet<string> = new Set([IMPACT_DIR, KEPT_DIR, "learned", "g | ||
| 450 | 458 | /** A gate's checkout (`gateRunId`): it has no run, only its `check_gate` row. */ |
| 451 | 459 | const GATE_DIR = /^gate-([0-9a-f]{12})$/; |
| 452 | 460 | |
| 453 | /** Whether a gate's checkout, this forge's or a peer's during a roll, is younger than a gate may run. */ | |
| 461 | /** | |
| 462 | * Whether a gate's checkout, this forge's or a peer's during a roll, is younger than a gate may | |
| 463 | * run. | |
| 464 | */ | |
| 454 | 465 | function gateMayBeRunning(path: string, now: number): boolean { |
| 455 | 466 | try { |
| 456 | 467 | return now - lstatSync(path).mtimeMs < GATE_DEADLINE_MS; |
| @@ -535,7 +546,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 535 | 546 | >(); |
| 536 | 547 | /** Checks already told why they wait, so the log says it once. */ |
| 537 | 548 | const waiting = new Set<string>(); |
| 538 | /** Queued checks a full disk holds that have said so once (#11799); apart from `waiting`, so neither line hides the other. */ | |
| 549 | /** | |
| 550 | * Queued checks a full disk holds that have said so once (#11799); apart from `waiting`, so | |
| 551 | * neither line hides the other. | |
| 552 | */ | |
| 539 | 553 | const diskWaiting = new Set<string>(); |
| 540 | 554 | const memoryWaiting = new Set<string>(); |
| 541 | 555 | /** The verification-lane run in flight, which holds no check slot. */ |
| @@ -689,7 +703,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 689 | 703 | */ |
| 690 | 704 | /** `pickNext` among `queued`, passing over any whose account is at its limit. */ |
| 691 | 705 | function startable(queued: readonly Run[]): Run | undefined { |
| 692 | // At 80% used no check starts (#11799): they stay queued, one line each, and the tick asks again. | |
| 706 | // At 80% used no check starts (#11799): they stay queued, one line each, and the tick asks | |
| 707 | // again. | |
| 693 | 708 | const full = usedRefusal(app.config.dataDir); |
| 694 | 709 | if (full !== undefined) { |
| 695 | 710 | for (const run of queued) |
| @@ -1131,7 +1146,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1131 | 1146 | readonly runId?: string; |
| 1132 | 1147 | readonly commitOid: string; |
| 1133 | 1148 | readonly pull?: Pull | undefined; |
| 1134 | /** `run`: a push to a branch; `fast`: a pull's push; `full`: on land; `verify`: main after landing. */ | |
| 1149 | /** | |
| 1150 | * `run`: a push to a branch; `fast`: a pull's push; `full`: on land; `verify`: main after | |
| 1151 | * landing. | |
| 1152 | */ | |
| 1135 | 1153 | readonly tier: "run" | "fast" | "full" | "verify"; |
| 1136 | 1154 | /** A bisection step: only these test files run (`FORGE_CHECK_ONLY`). */ |
| 1137 | 1155 | readonly only?: readonly string[]; |
| @@ -1143,7 +1161,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1143 | 1161 | readonly keepImpact?: boolean; |
| 1144 | 1162 | /** The default branch's own run: only its log may open issues. */ |
| 1145 | 1163 | readonly fileOrderDependent?: boolean; |
| 1146 | /** The default branch's own run: a failure of one or two tests is re-run alone before it counts as red. */ | |
| 1164 | /** | |
| 1165 | * The default branch's own run: a failure of one or two tests is re-run alone before it counts | |
| 1166 | * as red. | |
| 1167 | */ | |
| 1147 | 1168 | readonly isMain?: boolean; |
| 1148 | 1169 | readonly write: (line: string) => void; |
| 1149 | 1170 | readonly signal: AbortSignal; |
| @@ -1235,7 +1256,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1235 | 1256 | // (`checks/held.ts`). The branch's own supplies only its install step. |
| 1236 | 1257 | let manifest = declared.value; |
| 1237 | 1258 | let baseTip: string | undefined; |
| 1238 | /** An outside agent's pull nobody has approved: its code is checked, inside a cell or not at all. */ | |
| 1259 | /** | |
| 1260 | * An outside agent's pull nobody has approved: its code is checked, inside a cell or not at | |
| 1261 | * all. | |
| 1262 | */ | |
| 1239 | 1263 | let outside = false; |
| 1240 | 1264 | if (x.pull !== undefined) { |
| 1241 | 1265 | baseTip = await resolveOid(bare, `refs/heads/${x.pull.baseBranch}`); |
| @@ -1278,7 +1302,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1278 | 1302 | write(OUTSIDE_CHECK_LINE(x.pull.baseBranch)); |
| 1279 | 1303 | } |
| 1280 | 1304 | } else if (outsideAgentWaits(app.db, x.pull)) { |
| 1281 | // No gate on the base branch to hold it to: an outside contributor's pull does not bring its own. | |
| 1305 | // No gate on the base branch to hold it to: an outside contributor's pull does not bring | |
| 1306 | // its own. | |
| 1282 | 1307 | return { ok: false, error: OUTSIDE_NO_GATE(x.pull.baseBranch) }; |
| 1283 | 1308 | } |
| 1284 | 1309 | // What it asked for and does not have yet, said where a refused request is looked for. |
| @@ -1366,17 +1391,17 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1366 | 1391 | // The check is repository code: it runs in a cell of its own |
| 1367 | 1392 | // (`apps/contain.ts`), its checkout and a home its only places to write, |
| 1368 | 1393 | // the forge, the registry and what it declared its only way out. |
| 1369 | // A Job cannot hold egress (#1428), so while it is enforced the check | |
| 1370 | // stays in a cell. Read per run: an admin can flip it at runtime. | |
| 1371 | // Nor can it hold an outside contributor's check to anything: that one always runs in a cell. | |
| 1372 | const fellBack = placement.mode === "job" && (outside || egressSetting(app.db) === "enforce"); | |
| 1394 | // A Job holds egress only through a gate the forge opens for it | |
| 1395 | // (`checks/job_gate.ts`, #13251): a cluster that names none cannot | |
| 1396 | // honour forge.json, so the check stays in a cell, never unrestricted. | |
| 1397 | const fellBack = placement.mode === "job" && placement.gate === undefined; | |
| 1373 | 1398 | const placed: Placement = fellBack ? { mode: "process" } : placement; |
| 1374 | 1399 | const job = placed.mode === "job" ? placed : undefined; |
| 1375 | 1400 | if (x.runId !== undefined) |
| 1376 | 1401 | noteRunProgress(app.db, x.runId, { placement: placed.mode }); |
| 1377 | 1402 | if (fellBack) |
| 1378 | write("placement: job skipped, egress is enforced and a Job cannot hold it"); | |
| 1379 | // A Job is its own boundary: no cell here, the cluster holds it. | |
| 1403 | write("placement: job skipped, this cluster names no egress gate for a Job (FORGE_CHECKS_EGRESS_GATE)"); | |
| 1404 | // A Job is its own boundary: no cell here, the cluster and its gate hold it. | |
| 1380 | 1405 | const cells = job === undefined ? await cellsAvailable() : undefined; |
| 1381 | 1406 | if (cells !== undefined) { |
| 1382 | 1407 | if (x.runId !== undefined) |
| @@ -1386,12 +1411,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1386 | 1411 | if (!cells.ok) |
| 1387 | 1412 | write(`uncontained here: ${cells.reason}; the check runs unrestricted`); |
| 1388 | 1413 | } else { |
| 1389 | // The cluster holds a Job, but not as a cell does: the egress a | |
| 1390 | // check declared is not enforced here, and the run says so. | |
| 1391 | 1414 | if (x.runId !== undefined) |
| 1392 | noteRunProgress(app.db, x.runId, { | |
| 1393 | sandbox: { enforced: false, note: "egress not enforced on a cluster job" }, | |
| 1394 | }); | |
| 1415 | noteRunProgress(app.db, x.runId, { sandbox: { enforced: true, note: JOB_GATE_NOTE } }); | |
| 1395 | 1416 | write("placement: job"); |
| 1396 | 1417 | } |
| 1397 | 1418 | const home = `${dir}.home`; |
| @@ -1455,22 +1476,45 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1455 | 1476 | const checkLines: string[] = []; |
| 1456 | 1477 | // A bisection step runs only some files: what it shows green says nothing of the rest. |
| 1457 | 1478 | const wholeVerify = x.tier === "verify" && x.only === undefined; |
| 1458 | if (job !== undefined) { | |
| 1459 | ran = await runJobStep(job, { | |
| 1460 | runId: run.id, | |
| 1461 | dir, | |
| 1462 | commands: manifest.install === null ? [command] : [manifest.install, command], | |
| 1463 | env, | |
| 1464 | log: write, | |
| 1465 | output: checkLines, | |
| 1466 | signal, | |
| 1467 | timeoutMs: manifest.timeout * 1000, | |
| 1468 | limits: resolveLimits("step", manifest.limits), | |
| 1469 | onNode: (node) => { | |
| 1470 | if (x.runId !== undefined) noteRunProgress(app.db, x.runId, { node }); | |
| 1471 | }, | |
| 1472 | ...(options.job === undefined ? {} : { options: options.job }), | |
| 1473 | }); | |
| 1479 | // The gate is held to what a cell holds: the forge, the registry and | |
| 1480 | // what forge.json declares, an outside contributor's always enforced. | |
| 1481 | const gate = job?.gate === undefined | |
| 1482 | ? undefined | |
| 1483 | : openJobGate( | |
| 1484 | job.gate, | |
| 1485 | jobGatePolicy({ | |
| 1486 | hosts: [...STEP_HOSTS, ...manifest.egress], | |
| 1487 | origins: [options.forgeUrl].map(origin).filter((o) => o !== undefined), | |
| 1488 | label: `check:${run.id}`, | |
| 1489 | outside, | |
| 1490 | setting: egressSetting(app.db), | |
| 1491 | note: (host, port) => noteReached(app.config.dataDir, repo.id, host, port), | |
| 1492 | }), | |
| 1493 | write, | |
| 1494 | ); | |
| 1495 | if (gate !== undefined && !gate.ok) { | |
| 1496 | ran = err(gate.error); | |
| 1497 | } else if (job !== undefined) { | |
| 1498 | try { | |
| 1499 | ran = await runJobStep(job, { | |
| 1500 | runId: run.id, | |
| 1501 | dir, | |
| 1502 | commands: manifest.install === null ? [command] : [manifest.install, command], | |
| 1503 | env, | |
| 1504 | log: write, | |
| 1505 | output: checkLines, | |
| 1506 | signal, | |
| 1507 | timeoutMs: manifest.timeout * 1000, | |
| 1508 | limits: resolveLimits("step", manifest.limits), | |
| 1509 | onNode: (node) => { | |
| 1510 | if (x.runId !== undefined) noteRunProgress(app.db, x.runId, { node }); | |
| 1511 | }, | |
| 1512 | ...(gate === undefined ? {} : { proxy: gate.value.env }), | |
| 1513 | ...(options.job === undefined ? {} : { options: options.job }), | |
| 1514 | }); | |
| 1515 | } finally { | |
| 1516 | gate?.value.close(); | |
| 1517 | } | |
| 1474 | 1518 | } else { |
| 1475 | 1519 | if (manifest.install !== null) { |
| 1476 | 1520 | const install = manifest.install; |
| @@ -2053,7 +2097,10 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 2053 | 2097 | } |
| 2054 | 2098 | } |
| 2055 | 2099 | |
| 2056 | /** A gate asked of a forge already stopping is not started: the forge that carries on asks it again. */ | |
| 2100 | /** | |
| 2101 | * A gate asked of a forge already stopping is not started: the forge that carries on asks it | |
| 2102 | * again. | |
| 2103 | */ | |
| 2057 | 2104 | function gateRun(repo: Repo, pull: Pull, commitOid: string, signal?: AbortSignal): Promise<GateDone> { |
| 2058 | 2105 | if (stopped) return Promise.resolve({ ok: false, error: INTERRUPTED, lines: [], interrupted: true }); |
| 2059 | 2106 | const controller = new AbortController(); |
src/tui/screens/runlog.ts +12 −1
| @@ -19,6 +19,7 @@ import { emptyState } from "../ui/widgets.ts"; | ||
| 19 | 19 | import { subscribeRunLog } from "../../runs/log.ts"; |
| 20 | 20 | import { Scroll, live, openOnWeb } from "./common.ts"; |
| 21 | 21 | import { costText, envLabel, runTone } from "./runwords.ts"; |
| 22 | import { jobSandboxWords } from "../../checks/placement.ts"; | |
| 22 | 23 | |
| 23 | 24 | const LOG_LINES = 200; |
| 24 | 25 | |
| @@ -102,8 +103,14 @@ export function runLogScreen( | ||
| 102 | 103 | }); |
| 103 | 104 | return; |
| 104 | 105 | } |
| 105 | const [head, body] = split(area, "v", [len(1), fill(1)]); | |
| 106 | 106 | const run = view.run; |
| 107 | // A Job's wall rests on the cluster as well as the forge: the terminal says so, as the page | |
| 108 | // does. One sentence a row, so a narrow terminal cuts neither. | |
| 109 | const wall = | |
| 110 | run.placement === "job" && run.sandboxEnforced === true ? jobSandboxWords(run.sandboxNote).split(/(?<=\.) /) : []; | |
| 111 | const rows = split(area, "v", [len(1), ...wall.map(() => len(1)), fill(1)]); | |
| 112 | const head = rows[0]; | |
| 113 | const body = rows.at(-1); | |
| 107 | 114 | const tone = runTone(run.state); |
| 108 | 115 | const state = [ |
| 109 | 116 | run.state, |
| @@ -121,6 +128,10 @@ export function runLogScreen( | ||
| 121 | 128 | style(tone === undefined ? c.mutedFg : c.fg, c.bg), |
| 122 | 129 | head!.x + head!.w, |
| 123 | 130 | ); |
| 131 | wall.forEach((words, i) => { | |
| 132 | const row = rows[i + 1]!; | |
| 133 | s.text(row.x, row.y, truncate(words, row.w), style(c.mutedFg, c.bg), row.x + row.w); | |
| 134 | }); | |
| 124 | 135 | if (lines.length === 0) { |
| 125 | 136 | emptyState(s, body!, c, { |
| 126 | 137 | title: "No log yet", |
src/web/studio/sandbox.ts +4 −2
| @@ -6,7 +6,7 @@ | ||
| 6 | 6 | */ |
| 7 | 7 | |
| 8 | 8 | import type { Run } from "../../core/runs.ts"; |
| 9 | import { placementWords } from "../../checks/placement.ts"; | |
| 9 | import { jobSandboxWords, placementWords } from "../../checks/placement.ts"; | |
| 10 | 10 | import { html, type Html } from "../html.ts"; |
| 11 | 11 | import { badge } from "../ui.ts"; |
| 12 | 12 | |
| @@ -23,7 +23,9 @@ export function sandboxLine(run: Run): Html | null { | ||
| 23 | 23 | return html`<p class="muted small" id="run-sandbox"> |
| 24 | 24 | ${agent |
| 25 | 25 | ? "Sandboxed: the shell was held to the policy." |
| 26 | : "Contained: the repository's code ran in a cell of its own, with limits."} | |
| 26 | : run.placement === "job" | |
| 27 | ? jobSandboxWords(run.sandboxNote) | |
outdated
Principle 8: the note is a long clause in parentheses. Write two plain sentences, e.g. 'Held by the cluster: the check ran as a Job. The forge holds its egress; the cluster's NetworkPolicy holds the rest.' Also add the same line to the terminal run screen. |
||
| 28 | : "Contained: the repository's code ran in a cell of its own, with limits."} | |
| 27 | 29 | </p>`; |
| 28 | 30 | return html`<p class="small" id="run-sandbox"> |
| 29 | 31 | ${badge(agent ? "Unsandboxed" : "Uncontained", { tone: "danger" })} |
test/checks-job-vanish.test.ts +1 −4
| @@ -10,7 +10,6 @@ import { createApp, type App } from "../src/app.ts"; | ||
| 10 | 10 | import { commitToBranch, createRepo } from "../src/actions/repos.ts"; |
| 11 | 11 | import { startChecks, type ChecksHandle } from "../src/checks/service.ts"; |
| 12 | 12 | import { VANISHED_SUMMARY, listRuns } from "../src/core/runs.ts"; |
| 13 | import { setCapacity } from "../src/core/capacity.ts"; | |
| 14 | 13 | import { createUser } from "../src/core/users.ts"; |
| 15 | 14 | import type { Repo } from "../src/core/repos.ts"; |
| 16 | 15 | import { serve, type Servers } from "../src/http/server.ts"; |
| @@ -46,15 +45,13 @@ describe("a vanished check Job", () => { | ||
| 46 | 45 | beforeAll(async () => { |
| 47 | 46 | dataDir = await mkdtemp(join(tmpdir(), "forge-vanish-")); |
| 48 | 47 | app = createApp({ dataDir, port: 0 }); |
| 49 | // Jobs run only while egress is reported (#3190). | |
| 50 | setCapacity(app.db, "cells.egress", "report", "test"); | |
| 51 | 48 | servers = await serve(app); |
| 52 | 49 | const fetch = async (_u: string, init: RequestInit) => |
| 53 | 50 | init.method === "GET" ? new Response("{}", { status: 404 }) : Response.json({}); |
| 54 | 51 | checks = startChecks(app, { |
| 55 | 52 | forgeUrl: servers.url, |
| 56 | 53 | concurrency: 1, |
| 57 | placement: { mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [] }, | |
| 54 | placement: { mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [], gate: "127.0.0.1" }, | |
| 58 | 55 | job: { fetch: fetch as never, credentials: cred, pollMs: 1 }, |
| 59 | 56 | }); |
| 60 | 57 | const a = await createUser(app.db, { |
test/checks-placement.test.ts +117 −16
| @@ -12,7 +12,8 @@ import { commitToBranch, createRepo } from "../src/actions/repos.ts"; | ||
| 12 | 12 | import { parseSelector, parseTolerations, placementFromEnv } from "../src/checks/placement.ts"; |
| 13 | 13 | import { startChecks, type ChecksHandle } from "../src/checks/service.ts"; |
| 14 | 14 | import { runRows } from "../src/tui/screens/runlist.ts"; |
| 15 | import { placementLine } from "../src/web/studio/sandbox.ts"; | |
| 15 | import { placementLine, sandboxLine } from "../src/web/studio/sandbox.ts"; | |
| 16 | import { JOB_GATE_NOTE, JOB_GATE_PORTS, jobGatePolicy, openJobGate } from "../src/checks/job_gate.ts"; | |
| 16 | 17 | import { runPayload } from "../src/actions/runs.ts"; |
| 17 | 18 | import { placementWords } from "../src/checks/placement.ts"; |
| 18 | 19 | import { listRuns, type Run } from "../src/core/runs.ts"; |
| @@ -51,11 +52,24 @@ describe("placement from the environment", () => { | ||
| 51 | 52 | nodeSelector: { pool: "ci", arch: "arm64" }, |
| 52 | 53 | tolerations: [{ key: "ci", operator: "Exists", effect: "NoSchedule" }], |
| 53 | 54 | }); |
| 55 | expect(placementFromEnv({ ...job, FORGE_CHECKS_EGRESS_GATE: "10.0.0.7" })).toMatchObject({ mode: "job", gate: "10.0.0.7" }); | |
| 56 | expect(placementFromEnv({ ...job, FORGE_CHECKS_EGRESS_GATE: "forge.svc; rm" })).toEqual({ mode: "process" }); | |
| 57 | expect(placementFromEnv(job)).not.toHaveProperty("gate"); | |
| 54 | 58 | expect(parseSelector("a")).toBeUndefined(); |
| 55 | 59 | expect(parseTolerations("{}")).toBeUndefined(); |
| 56 | 60 | }); |
| 57 | 61 | }); |
| 58 | 62 | |
| 63 | describe("a Job's gate policy", () => { | |
| 64 | const base = { hosts: ["registry.npmjs.org"], origins: [], label: "check:r", note: () => {} }; | |
| 65 | test("reports undeclared hosts only under report, and never for an outside contributor", () => { | |
| 66 | expect(jobGatePolicy({ ...base, outside: false, setting: "report" }).report).toBe(base.note); | |
| 67 | expect(jobGatePolicy({ ...base, outside: false, setting: "enforce" }).report).toBeUndefined(); | |
| 68 | expect(jobGatePolicy({ ...base, outside: true, setting: "report" }).report).toBeUndefined(); | |
| 69 | expect(jobGatePolicy({ ...base, outside: true, setting: "report" }).hosts).toEqual(["registry.npmjs.org"]); | |
| 70 | }); | |
| 71 | }); | |
| 72 | ||
| 59 | 73 | describe("checks placed", () => { |
| 60 | 74 | let dataDir: string; |
| 61 | 75 | let app: App; |
| @@ -65,11 +79,52 @@ describe("checks placed", () => { | ||
| 65 | 79 | let repo: Repo; |
| 66 | 80 | const calls: { method: string; url: string; body?: string }[] = []; |
| 67 | 81 | let exit = 0; |
| 82 | /** What the Job's pod heard, through its proxy, from an undeclared host. */ | |
| 83 | const reached: string[] = []; | |
| 84 | /** What another pod heard, dialling the same gate without the run's credential. */ | |
| 85 | const stranger: string[] = []; | |
| 86 | ||
| 87 | // What the check in the pod does: dial the host through the proxy it was given. | |
| 88 | async function connectThrough(proxy: string, target: string, credential = true): Promise<string> { | |
| 89 | const { hostname, port, username, password } = new URL(proxy); | |
| 90 | const auth = username === "" ? "" : `Proxy-Authorization: Basic ${btoa(`${username}:${password}`)}\r\n`; | |
| 91 | let heard = ""; | |
| 92 | const done = Promise.withResolvers<void>(); | |
| 93 | const socket = await Bun.connect({ | |
| 94 | hostname, | |
| 95 | port: Number(port), | |
| 96 | socket: { | |
| 97 | data(_s, chunk) { | |
| 98 | heard += new TextDecoder().decode(chunk); | |
| 99 | if (heard.includes("\r\n\r\n")) done.resolve(); | |
| 100 | }, | |
| 101 | close() { | |
| 102 | done.resolve(); | |
| 103 | }, | |
| 104 | error() { | |
| 105 | done.resolve(); | |
| 106 | }, | |
| 107 | }, | |
| 108 | }); | |
| 109 | socket.write(`CONNECT ${target} HTTP/1.1\r\nHost: ${target}\r\n${credential ? auth : ""}\r\n`); | |
| 110 | // The proxy answers or closes; a hang is the test's own timeout. | |
| 111 | await done.promise; | |
| 112 | socket.end(); | |
| 113 | return heard; | |
| 114 | } | |
| 68 | 115 | |
| 69 | 116 | // A stand-in kube API: the Job succeeds at once, its pod on node n7. |
| 70 | 117 | const fetch = async (url: string, init: RequestInit) => { |
| 71 | 118 | calls.push({ method: String(init.method), url, ...(typeof init.body === "string" ? { body: init.body } : {}) }); |
| 72 | 119 | const path = new URL(url).pathname; |
| 120 | if (init.method === "POST" && typeof init.body === "string") { | |
| 121 | const env: { name: string; value: string }[] = JSON.parse(init.body).spec.template.spec.containers[0].env; | |
| 122 | const proxy = env.find((e) => e.name === "HTTPS_PROXY")?.value; | |
| 123 | if (proxy !== undefined) { | |
| 124 | reached.push(await connectThrough(proxy, "undeclared.example:443")); | |
| 125 | stranger.push(await connectThrough(proxy, "undeclared.example:443", false)); | |
| 126 | } | |
| 127 | } | |
| 73 | 128 | if (init.method === "POST" || init.method === "DELETE") return Response.json({}); |
| 74 | 129 | if (path.endsWith("/log")) return new Response("from the job\nsecond line\n"); |
| 75 | 130 | if (path.endsWith("/pods")) |
| @@ -146,13 +201,10 @@ describe("checks placed", () => { | ||
| 146 | 201 | claim: "checks-root", |
| 147 | 202 | nodeSelector: { pool: "ci" }, |
| 148 | 203 | tolerations: [{ key: "ci", operator: "Exists" }], |
| 204 | gate: "127.0.0.1", | |
| 149 | 205 | }), |
| 150 | 206 | ); |
| 151 | 207 | afterAll(down); |
| 152 | // Egress is enforced by default, which keeps checks in cells. | |
| 153 | beforeAll(() => { | |
| 154 | setCapacity(app.db, "cells.egress", "report", alice.id); | |
| 155 | }); | |
| 156 | 208 | |
| 157 | 209 | test("a check runs as a Job: log in the run, node and placement recorded", async () => { |
| 158 | 210 | const run = await settled(await push("exit 0")); |
| @@ -172,13 +224,47 @@ describe("checks placed", () => { | ||
| 172 | 224 | expect(pod.automountServiceAccountToken).toBe(false); |
| 173 | 225 | expect(pod.containers[0].resources.limits.memory).toMatch(/Mi$/); |
| 174 | 226 | expect(made.spec.template.metadata.labels["forge-check"]).toBe("true"); |
| 175 | expect(run.sandboxEnforced).toBe(false); | |
| 176 | expect(run.sandboxNote).toBe("egress not enforced on a cluster job"); | |
| 227 | expect(run.sandboxEnforced).toBe(true); | |
| 228 | expect(run.sandboxNote).toBe(JOB_GATE_NOTE); | |
| 229 | const line = sandboxLine(run)!.toString(); | |
| 230 | expect(line).toContain("ran as a Job"); | |
| 231 | expect(line).toContain("cluster's NetworkPolicy"); | |
| 232 | expect(line).not.toContain("in a cell"); | |
| 177 | 233 | expect(pod.containers[0].image).toBe("oven/bun:1"); |
outdated
Not awaited, so this assertion never fails the test. Use |
||
| 178 | 234 | expect(pod.containers[0].command.at(-1)).toBe("exit 0"); |
| 179 | 235 | expect(calls.some((c) => c.method === "DELETE")).toBe(true); |
| 180 | 236 | }); |
| 181 | 237 | |
| 238 | test("the Job's way out is the forge's gate, which refuses an undeclared host in its own words", async () => { | |
| 239 | reached.length = 0; | |
| 240 | stranger.length = 0; | |
| 241 | calls.length = 0; | |
| 242 | const run = await settled(await push("exit 0 # reach out")); | |
| 243 | expect(run.placement).toBe("job"); | |
| 244 | const env: { name: string; value: string }[] = JSON.parse(calls.find((c) => c.method === "POST")!.body!).spec | |
| 245 | .template.spec.containers[0].env; | |
| 246 | const proxy = env.find((e) => e.name === "HTTPS_PROXY")!.value; | |
| 247 | // The run's own secret rides in the URL, on a port of the range the NetworkPolicy names. | |
| 248 | const url = new URL(proxy); | |
| 249 | expect(url.hostname).toBe("127.0.0.1"); | |
| 250 | expect(Number(url.port)).toBeGreaterThanOrEqual(JOB_GATE_PORTS.from); | |
| 251 | expect(Number(url.port)).toBeLessThanOrEqual(JOB_GATE_PORTS.to); | |
| 252 | expect(url.password).toMatch(/^[0-9a-f]{48}$/); | |
| 253 | expect(env.find((e) => e.name === "NO_PROXY")!.value).toBe(""); | |
| 254 | expect(reached).toHaveLength(1); | |
| 255 | expect(reached[0]).toStartWith("HTTP/1.1 403"); | |
| 256 | expect(reached[0]).toContain( | |
| 257 | "refused by this forge: undeclared.example:443 is not in this app's forge.json egress", | |
| 258 | ); | |
| 259 | // Another pod on the network that reaches the port but not the secret is refused, dialling | |
| 260 | // nothing. | |
| 261 | expect(stranger).toHaveLength(1); | |
| 262 | expect(stranger[0]).toStartWith("HTTP/1.1 407"); | |
| 263 | expect(stranger[0]).toContain("X-Forge-Egress: refused"); | |
| 264 | // The gate lives as long as the run: closed once the Job is done. | |
| 265 | await expect(connectThrough(proxy, "undeclared.example:443")).rejects.toThrow(); | |
| 266 | }); | |
| 267 | ||
| 182 | 268 | test("the page, the terminal and the events say where a job ran; a process run says nothing", async () => { |
| 183 | 269 | const run = await settled(await push("exit 0 # page")); |
| 184 | 270 | expect(run.placement).toBe("job"); |
| @@ -214,11 +300,33 @@ describe("checks placed", () => { | ||
| 214 | 300 | }); |
| 215 | 301 | }); |
| 216 | 302 | |
| 217 | describe("job under enforce", () => { | |
| 303 | describe("job whose gate cannot listen", () => { | |
| 304 | // 192.0.2.1 is TEST-NET-1: no interface here holds it, so nothing can listen on it. | |
| 305 | beforeAll(() => | |
| 306 | boot({ mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [], gate: "192.0.2.1" }), | |
| 307 | ); | |
| 308 | afterAll(down); | |
| 309 | ||
| 310 | test("the gate says why, and the check fails with that reason, calling no cluster", async () => { | |
| 311 | const policy = jobGatePolicy({ hosts: [], origins: [], label: "t", outside: false, setting: "enforce", note: () => {} }); | |
| 312 | const got = openJobGate("192.0.2.1", policy, () => {}, { from: 47190, to: 47191 }); | |
| 313 | expect(got.ok).toBe(false); | |
| 314 | if (!got.ok) expect(got.error).toContain("the egress gate could not listen on 192.0.2.1, ports 47190-47191"); | |
| 315 | calls.length = 0; | |
| 316 | const run = await settled(await push("exit 0 # no gate")); | |
| 317 | expect(run.state).toBe("failed"); | |
| 318 | expect(readRunLog(dataDir, run.id).lines.join("\n")).toContain("the egress gate could not listen on 192.0.2.1"); | |
| 319 | expect(calls.filter((c) => c.method === "POST")).toHaveLength(0); | |
| 320 | }); | |
| 321 | }); | |
| 322 | ||
| 323 | describe("job without an egress gate", () => { | |
| 218 | 324 | beforeAll(() => boot({ mode: "job", image: "oven/bun:1", claim: "checks-root", nodeSelector: {}, tolerations: [] })); |
| 219 | 325 | afterAll(down); |
| 220 | 326 | |
| 221 | 327 | test("never calls the cluster: the check runs in a cell and says why", async () => { |
| 328 | // Report or enforce, a Job with no gate would run unrestricted: it never runs. | |
| 329 | setCapacity(app.db, "cells.egress", "report", alice.id); | |
| 222 | 330 | calls.length = 0; |
| 223 | 331 | const run = await settled(await push("echo in a cell")); |
| 224 | 332 | expect(run.state).toBe("ok"); |
| @@ -226,18 +334,11 @@ describe("checks placed", () => { | ||
| 226 | 334 | expect(run.node).toBe(""); |
| 227 | 335 | expect(calls).toHaveLength(0); |
| 228 | 336 | const log = readRunLog(dataDir, run.id).lines.join("\n"); |
| 229 | expect(log).toContain("placement: job skipped, egress is enforced and a Job cannot hold it"); | |
| 337 | expect(log).toContain("placement: job skipped, this cluster names no egress gate for a Job"); | |
| 230 | 338 | expect(log).toContain("in a cell"); |
| 231 | 339 | const page = await (await globalThis.fetch(`${servers.url}/alice/lib/runs/${run.id}`)).text(); |
| 232 | 340 | expect(page).not.toContain("cluster job"); |
| 233 | 341 | }); |
| 234 | ||
| 235 | test("is read per run: flipping to report sends the next check to a Job", async () => { | |
| 236 | setCapacity(app.db, "cells.egress", "report", alice.id); | |
| 237 | const run = await settled(await push("echo flipped")); | |
| 238 | expect(run.placement).toBe("job"); | |
| 239 | expect(calls.some((c) => c.method === "POST")).toBe(true); | |
| 240 | }); | |
| 241 | 342 | }); |
| 242 | 343 | |
| 243 | 344 | describe("process", () => { |
test/cluster-doc.test.ts +1 −1
| @@ -9,7 +9,7 @@ const read = (f: string) => readFileSync(join(root, f), "utf8"); | ||
| 9 | 9 | |
| 10 | 10 | test("the doc names every FORGE_CHECKS_* variable placement.ts reads", () => { |
| 11 | 11 | const vars = new Set([...read("src/checks/placement.ts").matchAll(/env\["(FORGE_CHECKS_[A-Z_]+)"\]/g)].map((m) => m[1]!)); |
| 12 | expect(vars.size).toBe(5); | |
| 12 | expect(vars.size).toBe(6); | |
| 13 | 13 | for (const v of vars) expect(doc).toContain(`\`${v}\``); |
| 14 | 14 | }); |
| 15 | 15 |
test/tui-runlog.test.ts +18 −1
| @@ -7,7 +7,8 @@ import { appendFileSync } from "node:fs"; | ||
| 7 | 7 | import { announceRun } from "../src/actions/runs.ts"; |
| 8 | 8 | import { createRepo } from "../src/actions/repos.ts"; |
| 9 | 9 | import { createApp, type App as Forge } from "../src/app.ts"; |
| 10 | import { claimRun, createRun, finishRun } from "../src/core/runs.ts"; | |
| 10 | import { claimRun, createRun, finishRun, noteRunProgress } from "../src/core/runs.ts"; | |
| 11 | import { JOB_GATE_NOTE } from "../src/checks/job_gate.ts"; | |
| 11 | 12 | import type { Repo } from "../src/core/repos.ts"; |
| 12 | 13 | import { createUser, type User } from "../src/core/users.ts"; |
| 13 | 14 | import { appendRunLog, runLogPath } from "../src/runs/log.ts"; |
| @@ -74,6 +75,22 @@ test("a finished run shows its state and its last 200 lines, at the end", async | ||
| 74 | 75 | expect(app.screen.toText()).not.toContain("\nline-100\n"); |
| 75 | 76 | }); |
| 76 | 77 | |
| 78 | test("a check that ran as a Job says, under its state, that its wall rests on the cluster too (#13251)", async () => { | |
| 79 | const run = createRun(forge.db, { repoId: repo.id, kind: "check" }); | |
| 80 | claimRun(forge.db, run.id); | |
| 81 | noteRunProgress(forge.db, run.id, { placement: "job", sandbox: { enforced: true, note: JOB_GATE_NOTE } }); | |
| 82 | appendRunLog(dir, run.id, "from the job"); | |
| 83 | finishRun(forge.db, run.id, { state: "ok", summary: "green" }); | |
| 84 | const text = (await open(alice, run.id)).screen.toText(); | |
| 85 | expect(text).toContain("Held by the cluster: the check ran as a Job."); | |
| 86 | expect(text).toContain("NetworkPolicy"); | |
| 87 | expect(text).toContain("from the job"); | |
| 88 | const here = createRun(forge.db, { repoId: repo.id, kind: "check" }); | |
| 89 | claimRun(forge.db, here.id); | |
| 90 | finishRun(forge.db, here.id, { state: "ok", summary: "green" }); | |
| 91 | expect((await open(alice, here.id)).screen.toText()).not.toContain("Held by the cluster"); | |
| 92 | }); | |
| 93 | ||
| 77 | 94 | test("a log the viewer may not read is not found and nothing of it is drawn", async () => { |
| 78 | 95 | const run = createRun(forge.db, { repoId: repo.id, kind: "agent" }); |
| 79 | 96 | claimRun(forge.db, run.id); |
new above
offline
Security: the gate takes any connection that reaches its port, with no credential. Require a per-run Proxy-Authorization secret (carried in the proxy URL given to the pod) so another pod on the network cannot use the forge as an egress proxy; test that a CONNECT without it is refused.