Public: anyone can read this; the crew can change it.

A pull's check can run on a runner: a separate process that needs only the forge's URL and a token #13823

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

trevato/checks-runner-placement → main 2 commits 1 thread · 1 unresolved

  • Merged. trevato/checks-runner-placement was merged into main as e6292aa.

Review

Sign in to review this change.
32 files changed +2428 −18
AGENTS.md +2
@@ -22,6 +22,8 @@ State shared between modules through process memory, not the database, is named
2222
2323 The check queue hands work to a runner through `src/checks/runner.ts` (#13652, R0); its module header says how. Keep database handles, paths and callbacks out of `CheckJob`.
2424
25 `src/checks/runner_jobs.ts`: `deskOf(app)`, per forge, the runners waiting in a claim (`claimers`) and the checks waiting on a job a runner may hold (`ears`, by job id), with `FORGE_CHECKS_PLACEMENT=runner` (#13652, R1; `docs/features/runners.md`). At most one entry per open claim request and per check in flight; gone on restart, and nothing is lost with it: what is claimed is a row (`check_job`), and a check whose forge restarted is queued again as any interrupted check is. `RunnerJob` is what crosses the wire: keep the bare repository's path (`check_job.dir`) out of it.
26
2527 `src/perf/loop_worst.ts`: `loopWorst`, the worst event-loop lag of the last 10 s in ten one-second slots, fed by one unref'd 50 ms timer that the first `serve` starts (`startLoopWorst`), and read by every response's `lag;dur` and by `/healthz` asked for JSON (`loopLagMs`). Tests listing a request's phases read `requestTiming` (`test/helpers/request-timing.ts`), which leaves `lag` out.
2628
2729 `src/perf/stall_blame.ts` (`stallBlame` in `src/perf/live.ts`): each loop stall over 200 ms as `{ what, ms, at }` in a fixed ring of 256, fed by `main.ts`'s stall watcher after `loopWatch` names it, read by /system's worst-five card and `/healthz`'s `loopWorstWhat`.
docs/APPS.md +5 −2
@@ -918,7 +918,10 @@ egress proxy for the run on `FORGE_CHECKS_EGRESS_GATE` and points the pod's
918918 `HTTPS_PROXY` at it, so an undeclared host is refused in the forge's own
919919 words, and the run is recorded as enforced; the pod carries the label
920920 `forge-check=true` for the NetworkPolicy that leaves it no other way out
921 (docs/CLUSTER.md). Without a gate no check runs as a Job.
921 (docs/CLUSTER.md). Without a gate no check runs as a Job. `runner` offers a
922 pull's check to a runner that joined, a separate process with the forge's URL
923 and a token, and runs it here when none takes it
924 ([docs/features/runners.md](features/runners.md)).
922925
923926 Checks share `FORGE_CHECKS_CONCURRENCY` slots (default `1`), and when one
924927 frees, it goes to the most recently requested check waiting, not the one
@@ -1816,7 +1819,7 @@ a link to the milestone (`docs/PLANNING.md`); its progress never does.
18161819 | `FORGE_PREVIEW_IDLE_DAYS` | the profit slider | days a pull may sit untouched before its preview sleeps; `0` = never; unset follows the slider (3 at 50) |
18171820 | `FORGE_CREW_CONCURRENCY` | 1 per 3 GiB above 2 GiB, 2–16 | agent runs at once (live: Settings → Capacity) |
18181821 | `FORGE_CHECKS_CONCURRENCY` | `1` | checks at once (live: Settings → Capacity) |
1819 | `FORGE_CHECKS_PLACEMENT` | `process` | `job` runs a check's commands as a cluster Job (see Placement above) |
1822 | `FORGE_CHECKS_PLACEMENT` | `process` | `job` runs a check's commands as a cluster Job, `runner` offers a pull's check to a runner that joined (see Placement above) |
18201823 | `FORGE_CHECKS_EGRESS_GATE` | unset | the address a Job's egress gate listens on; without it no check runs as a Job |
18211824 | `FORGE_CHECKS_DRAIN_S` | `10` | seconds a roll lets running checks finish first |
18221825 | `FORGE_CREW_DRAIN_S` | `10` | seconds a roll lets crew runs finish before they are queued again |
docs/features/README.md +1
@@ -210,6 +210,7 @@ Read with the guide page [Ship and undo](../guide-ship-and-undo.md).
210210 - [Pin screenshots](pin-screenshots.md)
211211 - [Pulls untouched for 14 days](stale-pulls.md)
212212 - [Ranked checks](ranked-checks.md)
213 - [Runners: a check on another machine, with a URL and a token](runners.md)
213214 - [A repository stuck behind its pulls](stuck-pile.md)
214215 - [Release heartbeat and the main-ahead alarm](release-heartbeat.md)
215216 - [Release keeper](release-keeper.md)
docs/features/runners.md added +110
@@ -0,0 +1,110 @@
1 # Runners
2
3 A runner is a separate process that takes checks off the forge. It can be on
4 the forge's machine, on another machine or in a rented one. It needs two
5 things: the forge's URL and a token. Nothing dials in to it, it shares no disk
6 with the forge, and it never sees the forge's data directory or database.
7
8 Off by default. With `FORGE_CHECKS_PLACEMENT=runner` the forge offers a check's
9 steps to runners, and **never needs one**: with no runner connected the check
10 runs on the forge's machine at once, as `process` does; when runners are
11 connected and none takes the job in 5 seconds it runs on the forge's machine
12 too.
13
14 ## Start one, in two lines
15
16 An admin enrols it (admin scope; the token is shown this once):
17
18 ```sh
19 curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" -H 'content-type: application/json' -d '{"name":"mac"}' $FORGE_URL/api/v1/admin/runners
20 ```
21
22 On the machine that will run checks, from a checkout of the forge or its image:
23
24 ```sh
25 FORGE_URL=https://your.forge FORGE_RUNNER_TOKEN=forge_runner_… bun src/runner/main.ts
26 ```
27
28 `GET /api/v1/admin/runners` lists the enrolled runners and when each last
29 asked for work; `DELETE /api/v1/admin/runners/<id>` removes one: its token
30 stops at once and a job it held is queued again. A check's page and the
31 terminal's run list say "Ran on runner mac" for a check one took.
32
33 The machine needs what the checks need: `git` and `bun`, and whatever the
34 repository's check calls. The forge's image has all of it, and `bubblewrap`
35 and `socat` for cells.
36
37 `FORGE_RUNNER_DIR` is where it keeps its scratch and its store (default: a
38 `forge-runner` directory under the temp directory), `FORGE_RUNNER_JOBS` how
39 many jobs it runs at once (default 1).
40
41 ## What a job is
42
43 Plain data, made by the forge from the repository's `forge.json` as it would
44 run the check itself:
45
46 | Field | What |
47 | ---------------------- | ----------------------------------------------------------------------------- |
48 | `commit` | the exact commit under test: for a pull, its merge into the base as it stands |
49 | `base`, `depth` | the commit a fast check diffs against, and how deep to fetch |
50 | `steps` | the recipe in order: `install`, then `check`, each a command |
51 | `env` | the check's own variables (`CI`, `FORGE_*`), never the forge's |
52 | `limits`, `timeoutS` | cores, memory and disk as declared; seconds a step may take |
53 | `egress`, `egressMode` | the hosts the steps may reach besides the forge |
54 | `cell` | `required` when the code is an outside contributor's: no cell, no run |
55 | `cache` | names a runner may keep things under: the repository, the tree, the base |
56
57 With the job comes a **lease** of 30 seconds, renewed by every report, and a
58 **job token** that works for this one job while the lease lives.
59
60 ## What a runner does
61
62 1. Claims: `POST /api/v1/runner/claim` with its runner token, held open until a
63 job is offered. Of two runners asking at once, one gets a job.
64 2. Fetches the job's commit from the job's own git door
65 (`/api/v1/runner/jobs/<id>/git`) with the job token, into a bare mirror it
66 keeps per repository, so the next job fetches only new objects.
67 3. Runs the steps in the cell the forge uses (bubblewrap on Linux): the
68 checkout and a home are the only places to write, the forge and the declared
69 hosts the only way out. Where the machine has no cells it says so, runs the
70 steps as plain processes, and hands back any job that requires a cell.
71 4. Streams the log every second (`POST /api/v1/runner/jobs/<id>`), reports each
72 step's result, and removes its scratch.
73
74 A runner that goes silent loses its lease: the job is queued again, and the
75 check's try count does not move. Twice, and the forge runs it itself.
76
77 ## What a runner decides, and what it does not
78
79 The forge keeps the queue, the claim on the check, the budget and the record.
80 A runner's answer stands only as: green, the check command's exit code, or its
81 timeout. The forge runs the check on its own machine when the install failed
82 on the runner, when the step ended any other way (a signal, a limit, a full
83 disk there), and when one or two tests failed: those are re-run alone to tell
84 a flake from a red, which a runner cannot do yet.
85
86 Only a **pull's check** goes to a runner. The default branch's own check and a
87 verification after landing keep what they measured (the build cache, the
88 impact map, kept results) from the checkout, and a runner cannot send those
89 back yet, so they stay on the forge's machine. For the same reason a runner's
90 check starts without them: correct, and slower than the forge's own.
91
92 ## What a runner can and cannot see
93
94 - **Can:** the commit of each job it claims and that commit's history; the
95 names and tips of that repository's branches (the advertisement); the job's
96 recipe and variables. It can append to that check's log and end that job.
97 - **Cannot:** fetch any other object of that repository, or anything of
98 another repository; push; read an issue, a page, the API or the MCP. Its two
99 tokens are not account tokens: every other door treats them as no credential
100 at all. A job token dies with its lease.
101 - A runner token can claim any job on the desk, so enrol machines you trust
102 with every repository whose checks run here. Without cells, a check runs as
103 the runner's own user on that machine.
104
105 A runner's requests are counted like a stranger's (600 a minute from one
106 address); one job is about 60 a minute.
107
108 Code: `src/checks/runner_jobs.ts` (the desk), `src/checks/runner_step.ts` (the
109 forge's side), `src/runner/main.ts` (the runner), `src/http/api/runners.ts`.
110 Tests: `test/checks-runner-placement.test.ts`. Epic: #13652.
scripts/unreached.ts +13 −4
@@ -5,8 +5,8 @@
55 * what main carries today; `test/unreached.test.ts` keeps it shrinking.
66 *
77 * An import is a static `from "./x"`, a side-effect `import "./x"`, or an
8 * `import("./x")` (quotes or backticks, any whitespace), resolved against the importing file's directory.
9 * Imports from `scripts/` count as reaching what they name.
8 * `import("./x")` (quotes or backticks, any whitespace), resolved against the importing file's
9 * directory. Imports from `scripts/` count as reaching what they name.
1010 */
1111
1212 import { readFileSync } from "node:fs";
@@ -21,13 +21,19 @@ export const ENTRIES = [
2121 "src/chaos/run.ts",
2222 "src/release/boot.ts",
2323 "src/git/hook-entry.ts",
24 // A runner's own process, started on its machine: `bun src/runner/main.ts`
25 // (docs/features/runners.md).
26 "src/runner/main.ts",
2427 // Run by hand: `bun run src/tui/gallery.ts` prints every widget.
2528 "src/tui/gallery.ts",
2629 // Bundled for the browser by Bun.build in src/web/asset.ts, named there by a template path.
2730 "src/web/highlight-client.ts",
2831 ];
2932
30 /** Scripts the check and operators run, and the benches run at release, are callers too: their imports count. */
33 /**
34 * Scripts the check and operators run, and the benches run at release, are callers too: their
35 * imports count.
36 */
3137 const SCRIPTS = ["scripts/**/*.ts", "bench/**/*.ts"];
3238
3339 /** Not modules: tests, and migrations, which the runner finds by listing the directory. */
@@ -35,7 +41,10 @@ const NOT_MODULES = /(?:\.test\.ts$|^src\/core\/migrations\/)/;
3541
3642 const SPEC = /(?:\bfrom\s*|\bimport\s*\(?\s*)["'`](\.{1,2}\/[^"'`]*)["'`]/g;
3743
38 /** A module named as a URL beside this one, the way a `Bun.build` entry point is: `new URL("./x.ts", import.meta.url)`. */
44 /**
45 * A module named as a URL beside this one, the way a `Bun.build` entry point is:
46 * `new URL("./x.ts", import.meta.url)`.
47 */
3948 const URL_SPEC = /\bnew\s+URL\(\s*["'`](\.{1,2}\/[^"'`]*)["'`]\s*,\s*import\.meta\.url\s*\)/g;
4049
4150 /** The src/ file a relative specifier names, among `files`, or undefined. */
src/actions/runners.ts added +59
@@ -0,0 +1,59 @@
1 /**
2 * An admin enrols, lists and removes runners (#13652): machines that take a check's steps off
3 * the forge (`checks/runner_jobs.ts`). Each enrolment and removal is an event, which is the
4 * audit; the token is in none of them. Only a person who is a forge admin may: anyone else is
5 * told it is not there.
6 */
7
8 import type { App } from "../app.ts";
9 import {
10 enrolRunner,
11 listRunners,
12 removeRunner,
13 wakeClaimers,
14 type Enrolled,
15 } from "../checks/runner_jobs.ts";
16 import type { User } from "../core/users.ts";
17 import { err, ok, type Result } from "../kernel/result.ts";
18
19 const admin = (actor: User | undefined): actor is User =>
20 actor !== undefined && actor.isAdmin && actor.kind === "user";
21
22 export function enrol(
23 app: App,
24 actor: User | undefined,
25 name: string,
26 ): Result<{ runner: Enrolled; token: string }, "not_found"> {
27 if (!admin(actor)) return err("not_found");
28 const made = enrolRunner(app.db, name, actor.id);
29 app.bus.append({
30 kind: "runner.enrolled",
31 actorId: actor.id,
32 payload: { id: made.runner.id, name: made.runner.name },
33 });
34 return ok(made);
35 }
36
37 export function list(
38 app: App,
39 actor: User | undefined,
40 ): Result<Enrolled[], "not_found"> {
41 return admin(actor) ? ok(listRunners(app.db)) : err("not_found");
42 }
43
44 export function remove(
45 app: App,
46 actor: User | undefined,
47 runnerId: string,
48 ): Result<void, "not_found"> {
49 if (!admin(actor)) return err("not_found");
50 if (!removeRunner(app.db, runnerId)) return err("not_found");
51 // A claim it holds open ends now, refused; a check waiting on its job looks again.
52 wakeClaimers(app);
53 app.bus.append({
54 kind: "runner.removed",
55 actorId: actor.id,
56 payload: { id: runnerId },
57 });
58 return ok();
59 }
src/checks/history.ts +1 −1
@@ -15,7 +15,7 @@ import { isSafeOid } from "../git/guard.ts";
1515 export const MAX_PULL_COMMITS = 500;
1616
1717 /** How deep to fetch the head so every commit in `base..head` arrives (at least 1). */
18 async function headDepth(bare: string, base: string, head: string): Promise<number> {
18 export async function headDepth(bare: string, base: string, head: string): Promise<number> {
1919 const counted = await runGit(["rev-list", "--count", `${base}..${head}`], { cwd: bare, timeoutMs: 30_000 });
2020 const n = counted.ok ? Number.parseInt(counted.value.trim(), 10) : NaN;
2121 // Unknown (a base the bare lacks) keeps the old one-deep fetch: the diff still answers.
src/checks/placement.ts +14 −5
@@ -1,9 +1,11 @@
11 /**
22 * Where a check's commands run: `process` (the default, a cell on the
3 * forge's own machine) or `job` (a Kubernetes Job, through
4 * `cluster/jobs.ts`). Read from the forge's environment once, validated at
5 * this boundary; anything malformed leaves the default in place.
6 * FORGE_CHECKS_PLACEMENT process | job
3 * forge's own machine), `job` (a Kubernetes Job, through
4 * `cluster/jobs.ts`) or `runner` (a runner that joined, `runner_step.ts`;
5 * with none connected the check runs as `process`). Read from the forge's
6 * environment once, validated at this boundary; anything malformed leaves
7 * the default in place.
8 * FORGE_CHECKS_PLACEMENT process | job | runner
79 * FORGE_CHECKS_IMAGE the Job's image
810 * FORGE_CHECKS_CLAIM a ReadWriteMany claim holding the checks root
911 * FORGE_CHECKS_NODE_SELECTOR key=value,key=value
@@ -24,6 +26,11 @@ export type Placement =
2426 readonly tolerations: Toleration[];
2527 /** The address the forge's per-run egress proxy listens on and the pod dials. */
2628 readonly gate?: string;
29 }
30 | {
31 readonly mode: "runner";
32 /** How long a job waits for a runner to take it; tests shorten it. @default RUNNER_WAIT_MS */
33 readonly waitMs?: number;
2734 };
2835
2936 /** An IPv4 address or a DNS name: what a pod can dial and the forge can bind. */
@@ -75,6 +82,7 @@ export function parseTolerations(text: string): Toleration[] | undefined {
7582 }
7683
7784 export function placementFromEnv(env: Record<string, string | undefined> = process.env): Placement {
85 if (env["FORGE_CHECKS_PLACEMENT"]?.trim() === "runner") return { mode: "runner" };
7886 if (env["FORGE_CHECKS_PLACEMENT"]?.trim() !== "job") return { mode: "process" };
7987 const image = env["FORGE_CHECKS_IMAGE"]?.trim() ?? "";
8088 const claim = env["FORGE_CHECKS_CLAIM"]?.trim() ?? "";
@@ -94,8 +102,9 @@ export function placementFromEnv(env: Record<string, string | undefined> = proce
94102 * here as well as escaped by whatever shows it.
95103 */
96104 export function placementWords(run: { placement: string; node: string }): string {
97 if (run.placement !== "job") return "";
105 if (run.placement !== "job" && run.placement !== "runner") return "";
98106 const node = run.node.replace(/[^\x21-\x7e]/g, "").slice(0, 253);
107 if (run.placement === "runner") return node === "" ? "ran on a runner" : `ran on runner ${node}`;
99108 return node === "" ? "ran as a cluster job" : `ran as a cluster job on ${node}`;
100109 }
101110
src/checks/runner_jobs.ts added +471
@@ -0,0 +1,471 @@
1 /**
2 * The desk between a check and the runners that may take its steps (#13652, R1).
3 *
4 * The check queue is still the scheduler: it picks a run, claims it and keeps its lease, as
5 * before. With `FORGE_CHECKS_PLACEMENT=runner` the steps that run would spawn are instead put on
6 * this desk as a job, and a runner (`src/runner/main.ts`, a separate process that holds nothing
7 * of the forge's but its URL and a token) may claim it. A job is plain data (`RunnerJob`): it
8 * survives `JSON.stringify`, and names no path, handle or callback of this forge.
9 *
10 * Two credentials, neither a row of `token`, so neither is anyone's account anywhere else:
11 * a runner token (`forge_runner_…`, made by an admin, kept as a hash in `check_runner`) claims
12 * jobs and nothing more; a job token (`forge_job_…`, made by each claim, kept as a hash on the
13 * job's row) fetches that job's repository and reports on that job, while its lease lives.
14 *
15 * A claim is the UPDATE's WHERE on `state`, as `claimRun` is. A lease that lapses puts the job
16 * back as queued: the run it serves is untouched, so nothing counts as a try. Rows are deleted
17 * when the check that offered them ends; the table holds only what is in flight.
18 */
19
20 import type { Database } from "bun:sqlite";
21 import type { App } from "../app.ts";
22 import { sha256 } from "../kernel/hash.ts";
23 import { id, secret } from "../kernel/id.ts";
24 import { during } from "../perf/jobs.ts";
25
26 export const RUNNER_TOKEN_PREFIX = "forge_runner_";
27 export const JOB_TOKEN_PREFIX = "forge_job_";
28
29 /** How long a claim holds without a word from its runner. */
30 export const LEASE_MS = 30_000;
31 /** A runner that asked for work this recently is connected. */
32 export const SEEN_FRESH_MS = 60_000;
33
34 export interface RunnerStepSpec {
35 readonly name: "install" | "check";
36 readonly command: string;
37 }
38
39 /** A check's steps as a runner is sent them. */
40 export interface RunnerJob {
41 readonly id: string;
42 /** The check run it serves, as the log names it. */
43 readonly run: string;
44 /** `owner/name`, for the runner's own log; the job's git door needs no name. */
45 readonly repo: string;
46 /** The exact commit under test: for a pull, its merge into the base as it stands. */
47 readonly commit: string;
48 /** The commit the fast tier diffs against (`FORGE_CHECK_BASE`), fetched one deep. */
49 readonly base?: string;
50 /** How deep `commit` is fetched so every commit of `base..commit` arrives. */
51 readonly depth: number;
52 /** The recipe, in order: what the service would have run, from `forge.json`. */
53 readonly steps: readonly RunnerStepSpec[];
54 /** The check's own variables (`CI`, `FORGE_*`); never the forge's environment. */
55 readonly env: Readonly<Record<string, string>>;
56 /** What `forge.json` declares; the runner holds them to its own machine. */
57 readonly limits: {
58 readonly cpus?: number;
59 readonly memoryMiB?: number;
60 readonly diskMiB?: number;
61 };
62 /** Seconds one step may take. */
63 readonly timeoutS: number;
64 /** Hosts the steps may reach besides the forge. */
65 readonly egress: readonly string[];
66 readonly egressMode: "enforce" | "report";
67 /** `required`: a runner without cells must hand it back. */
68 readonly cell: "required" | "preferred";
69 /**
70 * Names a runner may keep things under between jobs: its mirror and package cache by `repo`,
71 * a tree it has seen by `tree`, what the base's check left by `base` ("" when there is none).
72 */
73 readonly cache: {
74 readonly repo: string;
75 readonly tree: string;
76 readonly base: string;
77 };
78 }
79
80 /** What a runner reports of one step. */
81 export interface RunnerStepResult {
82 readonly name: string;
83 readonly ok: boolean;
84 /** `runStep`'s own words when it did not end ok. */
85 readonly error?: string;
86 readonly ms: number;
87 }
88
89 export interface RunnerResult {
90 readonly steps: readonly RunnerStepResult[];
91 /** Whether the steps ran in a cell, and why not when they did not. */
92 readonly contained: boolean;
93 readonly note: string;
94 }
95
96 export interface Enrolled {
97 readonly id: string;
98 readonly name: string;
99 readonly createdAt: string;
100 /** When it last asked for work, in milliseconds; 0 for never. */
101 readonly seenAt: number;
102 }
103
104 type RunnerRow = {
105 id: string;
106 name: string;
107 created_at: string;
108 seen_at: number;
109 };
110 const toEnrolled = (r: RunnerRow): Enrolled => ({
111 id: r.id,
112 name: r.name,
113 createdAt: r.created_at,
114 seenAt: r.seen_at,
115 });
116
117 /** A name as it is kept: printable, at most 40 characters, "runner" when nothing is left. */
118 export const runnerName = (raw: string): string =>
119 raw.replace(/[^\x21-\x7e]/g, "").slice(0, 40) || "runner";
120
121 /** Enrol a runner. The token is returned once; the row keeps its hash. */
122 export function enrolRunner(
123 db: Database,
124 name: string,
125 createdBy: string,
126 ): { runner: Enrolled; token: string } {
127 const runnerId = id();
128 const token = `${RUNNER_TOKEN_PREFIX}${secret()}`;
129 db.query(
130 "INSERT INTO check_runner (id, name, token_hash, created_by) VALUES (?1, ?2, ?3, ?4)",
131 ).run(runnerId, runnerName(name), sha256(token), createdBy);
132 return { runner: getRunner(db, runnerId)!, token };
133 }
134
135 export function getRunner(
136 db: Database,
137 runnerId: string,
138 ): Enrolled | undefined {
139 const row = db
140 .query<
141 RunnerRow,
142 [string]
143 >("SELECT id, name, created_at, seen_at FROM check_runner WHERE id = ?1")
144 .get(runnerId);
145 return row === null ? undefined : toEnrolled(row);
146 }
147
148 export function listRunners(db: Database): Enrolled[] {
149 return db
150 .query<
151 RunnerRow,
152 []
153 >("SELECT id, name, created_at, seen_at FROM check_runner ORDER BY id LIMIT 200")
154 .all()
155 .map(toEnrolled);
156 }
157
158 /**
159 * Remove a runner: its token stops at once, and what it held is queued again as a lapsed lease
160 * is.
161 */
162 export function removeRunner(db: Database, runnerId: string): boolean {
163 return db.transaction(() => {
164 const gone =
165 db.query("DELETE FROM check_runner WHERE id = ?1").run(runnerId).changes >
166 0;
167 if (gone)
168 db.query(
169 "UPDATE check_job SET state = 'queued', runner_id = NULL, token_hash = NULL, lease_until = 0 WHERE runner_id = ?1 AND state = 'claimed'",
170 ).run(runnerId);
171 return gone;
172 })();
173 }
174
175 /** The runner a presented token is, noting that it asked; undefined for anything else. */
176 export function runnerByToken(
177 db: Database,
178 presented: string,
179 now: number,
180 ): Enrolled | undefined {
181 if (!presented.startsWith(RUNNER_TOKEN_PREFIX)) return undefined;
182 const row = db
183 .query<
184 RunnerRow,
185 [string]
186 >("SELECT id, name, created_at, seen_at FROM check_runner WHERE token_hash = ?1")
187 .get(sha256(presented));
188 if (row === null) return undefined;
189 db.query("UPDATE check_runner SET seen_at = ?2 WHERE id = ?1").run(
190 row.id,
191 now,
192 );
193 return toEnrolled({ ...row, seen_at: now });
194 }
195
196 /** Whether any runner asked for work within `SEEN_FRESH_MS`. */
197 export function runnersAround(db: Database, now: number): boolean {
198 return (
199 db
200 .query<
201 { n: number },
202 [number]
203 >("SELECT 1 AS n FROM check_runner WHERE seen_at > ?1 LIMIT 1")
204 .get(now - SEEN_FRESH_MS) !== null
205 );
206 }
207
208 /**
209 * Put a check's steps on the desk. `dir` is the bare repository the commit is in; it stays here.
210 */
211 export function offerJob(
212 db: Database,
213 input: {
214 runId: string;
215 repoId: string;
216 dir: string;
217 job: Omit<RunnerJob, "id">;
218 },
219 ): RunnerJob {
220 const job: RunnerJob = { ...input.job, id: id() };
221 db.query(
222 "INSERT INTO check_job (id, run_id, repo_id, dir, spec, needs_cell) VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
223 ).run(
224 job.id,
225 input.runId,
226 input.repoId,
227 input.dir,
228 JSON.stringify(job),
229 job.cell === "required" ? 1 : 0,
230 );
231 return job;
232 }
233
234 export interface Claim {
235 readonly job: RunnerJob;
236 /** The job's own credential: shown once, here. */
237 readonly token: string;
238 readonly leaseMs: number;
239 }
240
241 /**
242 * Take the oldest queued job this runner can run. The UPDATE's WHERE is the claim: of two
243 * runners asking at once, one changes the row and the other moves on to the next.
244 */
245 export function claimJob(
246 db: Database,
247 runner: { id: string; cells: boolean },
248 now: number,
249 leaseMs: number = LEASE_MS,
250 ): Claim | undefined {
251 const queued = db
252 .query<
253 { id: string; spec: string },
254 [number]
255 >("SELECT id, spec FROM check_job WHERE state = 'queued' AND (needs_cell = 0 OR ?1 = 1) ORDER BY id LIMIT 8")
256 .all(runner.cells ? 1 : 0);
257 for (const row of queued) {
258 const token = `${JOB_TOKEN_PREFIX}${secret()}`;
259 const taken =
260 db
261 .query(
262 `UPDATE check_job SET state = 'claimed', runner_id = ?2, token_hash = ?3, lease_until = ?4,
263 claims = claims + 1 WHERE id = ?1 AND state = 'queued'`,
264 )
265 .run(row.id, runner.id, sha256(token), now + leaseMs).changes > 0;
266 if (taken)
267 return { job: JSON.parse(row.spec) as RunnerJob, token, leaseMs };
268 }
269 return undefined;
270 }
271
272 export interface HeldJob {
273 readonly id: string;
274 readonly runId: string;
275 readonly repoId: string;
276 /** The bare repository the job's commit is in. */
277 readonly dir: string;
278 /** The only commits its git door serves: the job's commit, and its base when it has one. */
279 readonly wants: readonly string[];
280 }
281
282 /**
283 * The job a presented token holds: this job, claimed, its lease alive. Anything else (another
284 * job's token, a lapsed or withdrawn claim, a runner's or an account's token) is undefined.
285 * Its runner is noted as seen: one at work is connected, though it is not asking for more.
286 */
287 export function jobByToken(db: Database, jobId: string, presented: string, now: number): HeldJob | undefined {
288 if (!presented.startsWith(JOB_TOKEN_PREFIX)) return undefined;
289 const row = db
290 .query<{ id: string; run_id: string; repo_id: string; dir: string; spec: string }, [string, string, number]>(
291 `SELECT id, run_id, repo_id, dir, spec FROM check_job
292 WHERE id = ?1 AND token_hash = ?2 AND state = 'claimed' AND lease_until > ?3`,
293 )
294 .get(jobId, sha256(presented), now);
295 if (row === null) return undefined;
296 db.query("UPDATE check_runner SET seen_at = ?2 WHERE id = (SELECT runner_id FROM check_job WHERE id = ?1)").run(
297 row.id,
298 now,
299 );
300 const job = JSON.parse(row.spec) as RunnerJob;
301 return {
302 id: row.id,
303 runId: row.run_id,
304 repoId: row.repo_id,
305 dir: row.dir,
306 wants: [job.commit, ...(job.base === undefined ? [] : [job.base])].map((oid) => oid.toLowerCase()),
307 };
308 }
309
310 /** Push a held job's lease out; false when the claim is no longer this one's to renew. */
311 export function renewJob(
312 db: Database,
313 jobId: string,
314 now: number,
315 leaseMs: number = LEASE_MS,
316 ): boolean {
317 return (
318 db
319 .query(
320 "UPDATE check_job SET lease_until = ?2 WHERE id = ?1 AND state = 'claimed' AND lease_until > ?3",
321 )
322 .run(jobId, now + leaseMs, now).changes > 0
323 );
324 }
325
326 /** Record a held job's answer: `done` with its result, or `returned` with the runner's reason. */
327 export function settleJob(
328 db: Database,
329 jobId: string,
330 end: { done: RunnerResult } | { returned: string },
331 ): boolean {
332 const done = "done" in end;
333 return (
334 db
335 .query(
336 "UPDATE check_job SET state = ?2, result = ?3, token_hash = NULL WHERE id = ?1 AND state = 'claimed'",
337 )
338 .run(
339 jobId,
340 done ? "done" : "returned",
341 done ? JSON.stringify(end.done) : end.returned.slice(0, 400),
342 ).changes > 0
343 );
344 }
345
346 /**
347 * Every claim whose lease ran out goes back to queued, its token dead. Returns their ids. The
348 * check each serves is still running under the forge's own lease: nothing is counted as a try.
349 */
350 export function lapseJobs(db: Database, now: number): string[] {
351 return db
352 .query<{ id: string }, [number]>(
353 `UPDATE check_job SET state = 'queued', runner_id = NULL, token_hash = NULL, lease_until = 0
354 WHERE state = 'claimed' AND lease_until <= ?1 RETURNING id`,
355 )
356 .all(now)
357 .map((r) => r.id);
358 }
359
360 /** Take a job nobody claimed back off the desk; false when a runner got there first. */
361 export function withdrawJob(db: Database, jobId: string): boolean {
362 return (
363 db
364 .query("DELETE FROM check_job WHERE id = ?1 AND state = 'queued'")
365 .run(jobId).changes > 0
366 );
367 }
368
369 /** The check that offered the job has ended: the row goes, and with it any claim. */
370 export function dropJob(db: Database, jobId: string): void {
371 db.query("DELETE FROM check_job WHERE id = ?1").run(jobId);
372 }
373
374 /** Jobs whose check is no longer running (a forge that died mid-check): swept at boot. */
375 export function sweepJobs(db: Database): number {
376 return db
377 .query(
378 "DELETE FROM check_job WHERE run_id NOT IN (SELECT id FROM run WHERE kind = 'check' AND state = 'running')",
379 )
380 .run().changes;
381 }
382
383 export interface JobView {
384 readonly state: "queued" | "claimed" | "done" | "returned";
385 readonly claims: number;
386 /** The runner holding it, by name; "" when none does. */
387 readonly runner: string;
388 readonly result: string | null;
389 }
390
391 export function viewJob(db: Database, jobId: string): JobView | undefined {
392 const row = db
393 .query<
394 {
395 state: JobView["state"];
396 claims: number;
397 runner: string | null;
398 result: string | null;
399 },
400 [string]
401 >(
402 `SELECT j.state, j.claims, r.name AS runner, j.result
403 FROM check_job j LEFT JOIN check_runner r ON r.id = j.runner_id WHERE j.id = ?1`,
404 )
405 .get(jobId);
406 return row === null
407 ? undefined
408 : {
409 state: row.state,
410 claims: row.claims,
411 runner: row.runner ?? "",
412 result: row.result,
413 };
414 }
415
416 /** What the check waiting on a job hears from the runner's requests, in this process. */
417 export interface JobEar {
418 /** Something about the job changed: look again. */
419 readonly wake: () => void;
420 /** Lines the runner streamed, and the step they belong to. */
421 readonly lines: (step: string, lines: readonly string[]) => void;
422 }
423
424 /**
425 * This forge's desk in memory: the runners waiting in a claim, and the checks waiting on a job.
426 * Both are gone with the process, and nothing is lost with them: the rows say what is claimed.
427 */
428 export interface Desk {
429 readonly claimers: Set<() => void>;
430 readonly ears: Map<string, JobEar>;
431 }
432
433 const desks = new WeakMap<App, Desk>();
434
435 export function deskOf(app: App): Desk {
436 let desk = desks.get(app);
437 if (desk === undefined) {
438 desk = { claimers: new Set(), ears: new Map() };
439 desks.set(app, desk);
440 }
441 return desk;
442 }
443
444 /** Tell every runner waiting in a claim that the desk changed. */
445 export function wakeClaimers(app: App): void {
446 for (const wake of [...deskOf(app).claimers]) wake();
447 }
448
449 /**
450 * Wait until the desk changes, `ms` pass or `signal` aborts, whichever is first. A runner's
451 * long poll rests here between looks.
452 */
453 export function nextOffer(
454 app: App,
455 ms: number,
456 signal: AbortSignal,
457 ): Promise<void> {
458 if (ms <= 0 || signal.aborted) return Promise.resolve();
459 const desk = deskOf(app);
460 return new Promise((resolve) => {
461 const done = (): void => {
462 clearTimeout(timer);
463 desk.claimers.delete(done);
464 signal.removeEventListener("abort", done);
465 resolve();
466 };
467 const timer = setTimeout(() => during("runner-wait", done), ms);
468 desk.claimers.add(done);
469 signal.addEventListener("abort", done, { once: true });
470 });
471 }
src/checks/runner_step.ts added +274
@@ -0,0 +1,274 @@
1 /**
2 * A check's steps on a runner (`checks.placement: runner`, #13652 R1): the forge's side.
3 *
4 * The check stays this forge's: claimed, leased, budgeted and recorded here, as it is while a
5 * cluster Job runs it. `runRunnerStep` puts the steps on the desk (`runner_jobs.ts`) and waits.
6 * The forge never needs a runner: when none is connected, none takes the job within the wait, the
7 * one that took it goes silent twice or hands it back, the answer is "not taken" and the caller
8 * runs the check on this machine as `process` would.
9 *
10 * What a runner's answer may decide is narrow on purpose (`runnerVerdict`): green, the check
11 * command's own exit code, or its timeout. Anything else is the runner machine's, and the forge
12 * runs the check itself.
13 */
14
15 import type { App } from "../app.ts";
16 import { killedStep } from "../apps/process.ts";
17 import { during } from "../perf/jobs.ts";
18 import { err, ok, type Result } from "../kernel/result.ts";
19 import { parseBunTestFailure } from "./bun-test.ts";
20 import { rerunnable } from "./flake.ts";
21 import { checkStage } from "./gate_reason.ts";
22 import type { CheckOutcome, CheckTier } from "./runner.ts";
23 import {
24 deskOf,
25 dropJob,
26 lapseJobs,
27 offerJob,
28 runnersAround,
29 viewJob,
30 wakeClaimers,
31 withdrawJob,
32 type RunnerJob,
33 type RunnerResult,
34 type RunnerStepResult,
35 type RunnerStepSpec,
36 } from "./runner_jobs.ts";
37 import { summariseCheckFailure } from "./summary.ts";
38
39 /** How long a job waits for a runner to take it before the forge runs the check itself. */
40 export const RUNNER_WAIT_MS = 5_000;
41 /** A job whose runner went silent this many times is not offered again. */
42 export const MAX_CLAIMS = 2;
43 /** How often a waiting check looks at its job when nothing woke it. */
44 const LOOK_MS = 1_000;
45
46 /**
47 * Whether a runner may take this check. Only one that hands nothing back but its log and its
48 * verdict: a pull's check. The default branch's own run and a verification keep what they
49 * measured (the cache, the impact map, kept results) from the checkout, and a runner has no way
50 * to send those yet, so they stay on the forge's machine.
51 */
52 export function runnerTakes(x: {
53 readonly tier: CheckTier;
54 readonly keep?: boolean | undefined;
55 readonly keepImpact?: boolean | undefined;
56 readonly isMain?: boolean | undefined;
57 readonly only?: readonly string[] | undefined;
58 }): boolean {
59 return (
60 (x.tier === "fast" || x.tier === "run") &&
61 x.keep !== true &&
62 x.keepImpact !== true &&
63 x.isMain !== true &&
64 x.only === undefined
65 );
66 }
67
68 export type RunnerStepEnd =
69 | {
70 readonly taken: true;
71 readonly ran: Result<void, string>;
72 /** The step that did not end ok. */
73 readonly failed?: string;
74 readonly runner: string;
75 readonly contained: boolean;
76 readonly note: string;
77 }
78 | { readonly taken: false; readonly why: string };
79
80 export interface RunnerStepInput {
81 readonly runId: string;
82 readonly repoId: string;
83 /** The bare repository the commit is in: what the job's git door serves. */
84 readonly dir: string;
85 readonly job: Omit<RunnerJob, "id">;
86 readonly log: (line: string) => void;
87 /** The check step's own output, apart from the echoed command. */
88 readonly output: string[];
89 readonly signal: AbortSignal;
90 /** @default RUNNER_WAIT_MS */
91 readonly waitMs?: number | undefined;
92 /** Tests stand a clock in. @default Date.now */
93 readonly now?: (() => number) | undefined;
94 }
95
96 function parseResult(text: string | null): RunnerResult | undefined {
97 try {
98 const value = JSON.parse(text ?? "") as RunnerResult;
99 return Array.isArray(value.steps) ? value : undefined;
outdated

A result with steps: [] passes here and becomes green. Require the check step (and install when offered) to be present, ok and named as in the job, or return taken:false.

100 } catch {
101 return undefined;
102 }
103 }
104
105 /**
106 * What a runner's step results say, held to the steps it was asked for: the same names in the
107 * same order, every one ok, or ok up to one that failed and nothing after it. Anything else (no
108 * steps, a step left out, a name of its own, a step after a failure) is no answer, and the forge
109 * runs the check itself: an empty or partial result is never green.
110 */
111 export function readSteps(
112 asked: readonly RunnerStepSpec[],
113 said: readonly RunnerStepResult[],
114 ): { ran: Result<void, string>; failed?: string } | undefined {
115 if (said.length === 0 || said.length > asked.length) return undefined;
116 for (const [i, step] of said.entries()) {
117 if (step.name !== asked[i]!.name || typeof step.ok !== "boolean") return undefined;
118 if (step.ok) continue;
119 return i === said.length - 1
120 ? { ran: err(step.error ?? `${step.name} failed`), failed: step.name }
121 : undefined;
122 }
123 return said.length === asked.length ? { ran: ok() } : undefined;
124 }
125
126 export async function runRunnerStep(
127 app: App,
128 input: RunnerStepInput,
129 ): Promise<RunnerStepEnd> {
130 const now = input.now ?? Date.now;
131 const waitMs = input.waitMs ?? RUNNER_WAIT_MS;
132 const desk = deskOf(app);
133 // Nobody to offer it to: not a second is spent waiting.
134 if (desk.claimers.size === 0 && !runnersAround(app.db, now()))
135 return { taken: false, why: "no runner is connected" };
136 const job = offerJob(app.db, {
137 runId: input.runId,
138 repoId: input.repoId,
139 dir: input.dir,
140 job: input.job,
141 });
142 let wake: () => void = () => {};
143 let sawEcho = false;
144 desk.ears.set(job.id, {
145 wake: () => wake(),
146 lines: (step, lines) => {
147 for (const line of lines) {
148 // `runStep` echoes the command first; that echo is not the command's own output.
149 if (step === "check") {
150 if (sawEcho) input.output.push(line);
151 sawEcho = true;
152 }
153 input.log(line);
154 }
155 },
156 });
157 /** Rest until a request of the runner's wakes this, a second passes or the check is stopped. */
158 const rest = (ms: number): Promise<void> =>
159 new Promise((resolve) => {
160 const done = (): void => {
161 clearTimeout(timer);
162 input.signal.removeEventListener("abort", done);
163 resolve();
164 };
165 const timer = setTimeout(() => during("runner-wait", done), ms);
166 wake = done;
167 input.signal.addEventListener("abort", done, { once: true });
168 });
169 let offeredAt = now();
170 let holder = "";
171 try {
172 wakeClaimers(app);
173 for (;;) {
174 if (input.signal.aborted)
175 return {
176 taken: true,
177 ran: err("check cancelled"),
178 runner: holder,
179 contained: false,
180 note: "",
181 };
182 // Every lapsed lease on the desk, this job's among them, goes back to queued.
183 lapseJobs(app.db, now());
184 const view = viewJob(app.db, job.id);
185 if (view === undefined)
186 return { taken: false, why: "the job left the desk" };
187 if (view.state === "queued" && holder !== "") {
188 input.log(
189 `runner ${holder} went silent: the job is queued again, and that is not a try`,
190 );
191 // What a half-run printed is not the check's output.
192 input.output.length = 0;
193 sawEcho = false;
194 holder = "";
195 offeredAt = now();
196 wakeClaimers(app);
197 }
198 if (view.state === "claimed" && view.runner !== holder) {
199 holder = view.runner;
200 input.log(`placement: runner ${holder}`);
201 }
202 if (view.state === "done") {
203 const result = parseResult(view.result);
204 const read = result === undefined ? undefined : readSteps(input.job.steps, result.steps);
205 if (result === undefined || read === undefined)
206 return { taken: false, why: `runner ${holder} sent a result that does not match the job's steps` };
207 return {
208 taken: true,
209 ...read,
210 runner: holder,
211 contained: result.contained === true,
212 note:
213 typeof result.note === "string" ? result.note.slice(0, 200) : "",
214 };
215 }
216 if (view.state === "returned")
217 return {
218 taken: false,
219 why: `runner ${holder} handed it back: ${view.result ?? ""}`,
220 };
221 if (
222 view.state === "queued" &&
223 (view.claims >= MAX_CLAIMS || now() - offeredAt >= waitMs)
224 ) {
225 // A runner that claims in this instant keeps it: the DELETE's WHERE decides.
226 if (withdrawJob(app.db, job.id))
227 return {
228 taken: false,
229 why:
230 view.claims >= MAX_CLAIMS
231 ? `its runners went silent ${view.claims} times`
232 : `no runner took it in ${Math.round(waitMs / 1000)}s`,
233 };
234 continue;
235 }
236 // Queued, the next look is no later than the end of its wait.
237 const left = view.state === "queued" ? waitMs - (now() - offeredAt) : LOOK_MS;
238 await rest(Math.min(Math.max(left, 1), LOOK_MS));
239 }
240 } finally {
241 desk.ears.delete(job.id);
242 dropJob(app.db, job.id);
243 }
244 }
245
246 /** A step error that is the check's own verdict: its exit code, or its time. */
247 const VERDICT = /^check (?:exited with code \d+|took longer than \d+s)/;
248
249 /**
250 * What a runner's answer decides. An outcome stands as the check's; a string is why it does not,
251 * and the forge runs the check itself:
252 * - the install failed there (this machine's install knows a registry's bad minute from a red);
253 * - the step ended by anything but its own exit or time (a signal, a limit, a full disk on the
254 * runner);
255 * - one or two tests failed, which this machine re-runs alone to tell a flake from a red, and a
256 * runner cannot yet.
257 */
258 export function runnerVerdict(
259 end: Extract<RunnerStepEnd, { taken: true }>,
260 lines: readonly string[],
261 aborted: boolean,
262 ): CheckOutcome | string {
263 if (aborted) return { ok: false, error: "check cancelled" };
264 if (end.ran.ok) return { ok: true };
265 const error = end.ran.error;
266 if (end.failed !== "check")
267 return `the ${end.failed ?? "job"} did not finish on runner ${end.runner} (${error})`;
268 if (!VERDICT.test(error) || killedStep(error))
269 return `runner ${end.runner} could not finish the check (${error})`;
270 if (rerunnable(lines, parseBunTestFailure(lines)) !== undefined)
271 return `one or two tests failed on runner ${end.runner}: re-run here, where a failure is tried alone`;
272 const { summary, detail } = summariseCheckFailure(lines, error);
273 return { ok: false, error: summary, detail, stage: checkStage(error) };
274 }
src/checks/service.ts +74 −2
@@ -68,7 +68,7 @@ import { checkOnApproval } from "./approval.ts";
6868 import { BASE_MOVED, dueGreen, newestBaseFirst } from "./base_moved.ts";
6969 import { checkKey, provenBy, recordProven } from "./proven.ts";
7070 import { beginGate, endGate, gateHolding } from "./gate_record.ts";
71 import { giveHistory } from "./history.ts";
71 import { giveHistory, headDepth } from "./history.ts";
7272 import { readLearned } from "./learned.ts";
7373 import { keepCache, restoreCache } from "./cache.ts";
7474 import { MAX_FILED, clearedFiles, fileOrderDependent } from "./order_dependent.ts";
@@ -139,6 +139,8 @@ import { JOB_GATE_NOTE, jobGatePolicy, openJobGate } from "./job_gate.ts";
139139 import { placementFromEnv, type Placement } from "./placement.ts";
140140 import { refuseSmallPod, type PodRoom } from "./pod_room.ts";
141141 import { JOB_VANISHED, runJobStep } from "./job_step.ts";
142 import { runRunnerStep, runnerTakes, runnerVerdict } from "./runner_step.ts";
143 import { sweepJobs } from "./runner_jobs.ts";
142144 import {
143145 cellInstall,
144146 checkInstallCache,
@@ -606,6 +608,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
606608 capacityCount(app.db, "checks.concurrency", booted);
607609 const gate = options.gate ?? new Gate(capacity());
608610 const placement = options.placement ?? placementFromEnv();
611 // Jobs a forge that died left on the runners' desk: their checks are queued again elsewhere.
612 if (placement.mode === "runner") sweepJobs(app.db);
609613 /**
610614 * Full gates on land in flight: not slots in the queue, but stopped with
611615 * it — after the drain, as a check is. By commit, so a merge queue that
@@ -1365,6 +1369,73 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13651369 write(`already proven green by ${earlier}`);
13661370 return { ok: true, proven: earlier };
13671371 }
1372 // A runner that joined takes the steps (`runner_step.ts`, #13652): only a check that hands
1373 // back nothing but its log and its verdict, and only while a runner is connected. Not taken,
1374 // or an answer that is not the check's own verdict, and it runs here as it always has.
1375 if (placement.mode === "runner" && x.runId !== undefined && runnerTakes(x)) {
1376 const lines: string[] = [];
1377 const sent = await runRunnerStep(app, {
1378 runId: x.runId,
1379 repoId: repo.id,
1380 dir: bare,
1381 job: {
1382 run: x.runId,
1383 repo: `${repo.ownerName}/${repo.name}`,
1384 commit: run.commitOid,
1385 ...(base === undefined ? {} : { base }),
1386 depth: base === undefined ? 1 : await headDepth(bare, base, run.commitOid),
1387 steps: [
1388 ...(manifest.install === null ? [] : [{ name: "install" as const, command: manifest.install }]),
1389 { name: "check" as const, command },
1390 ],
1391 // Only the check's own variables cross, never the forge's; the runner adds its cell's.
1392 env: {
1393 CI: "1",
1394 FORGE_URL: options.forgeUrl,
1395 FORGE_CHECK: run.id,
1396 FORGE_COMMIT: x.commitOid,
1397 FORGE_CHECK_IMPACT: IMPACT_FILE,
1398 FORGE_CHECK_KEPT: KEPT_FILE,
1399 ...(base === undefined ? {} : { FORGE_CHECK_BASE: base, FORGE_CHECK_LEARNED: learned }),
1400 FORGE_CELLS: "off",
1401 },
1402 limits: {
1403 ...(manifest.limits.cpus === undefined ? {} : { cpus: manifest.limits.cpus }),
1404 ...(manifest.limits.memoryMiB === undefined ? {} : { memoryMiB: manifest.limits.memoryMiB }),
1405 ...(manifest.limits.diskMiB === undefined ? {} : { diskMiB: manifest.limits.diskMiB }),
1406 },
1407 timeoutS: manifest.timeout,
1408 egress: [...STEP_HOSTS, ...manifest.egress],
1409 egressMode: outside ? "enforce" : egressSetting(app.db),
1410 // An outside contributor's code runs in a cell or not at all, there as here.
1411 cell: outside ? "required" : "preferred",
1412 cache: { repo: repo.id, tree: tree.ok ? tree.value.trim() : "", base: builtOnBase ?? "" },
1413 },
1414 log: write,
1415 output: lines,
1416 signal,
1417 waitMs: placement.waitMs,
1418 });
1419 const verdict = sent.taken ? runnerVerdict(sent, lines, signal.aborted) : sent.why;
1420 if (typeof verdict === "string") write(`placement: runner; ${verdict}; running here`);
1421 else {
1422 if (sent.taken && sent.runner !== "")
1423 noteRunProgress(app.db, x.runId, {
1424 placement: "runner",
1425 node: sent.runner,
1426 sandbox: { enforced: sent.contained, note: sent.note },
1427 });
1428 try {
1429 recordCrashFlakes(app.db, app.bus, repo, lines, { runId: run.id, commit: run.commitOid, pull: true }, verdict.ok);
1430 } catch (error) {
1431 write(`crash flakes not recorded: ${String(error)}`);
1432 }
1433 if (!verdict.ok) return verdict;
1434 if (key !== undefined) recordProven(app.db, key, x.id);
1435 const cleared = clearedIn(lines);
1436 return { ok: true, ...(cleared === undefined ? {} : { cleared }) };
1437 }
1438 }
13681439 if (await reclaimIfFull(app)) write("disk under its free floor: swept unused checkouts");
13691440 const dir = join(checksRoot(app.config.dataDir), run.id);
13701441 mkdirSync(dir, { recursive: true });
@@ -1409,7 +1480,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
14091480 // (`checks/job_gate.ts`, #13251): a cluster that names none cannot
14101481 // honour forge.json, so the check stays in a cell, never unrestricted.
14111482 const fellBack = placement.mode === "job" && placement.gate === undefined;
1412 const placed: Placement = fellBack ? { mode: "process" } : placement;
1483 // A check a runner did not take runs here as `process` does.
1484 const placed: Placement = fellBack || placement.mode === "runner" ? { mode: "process" } : placement;
14131485 const job = placed.mode === "job" ? placed : undefined;
14141486 if (x.runId !== undefined)
14151487 noteRunProgress(app.db, x.runId, { placement: placed.mode });
src/contract/openapi/routes/deleteApiV1AdminRunnersId.ts added +11
@@ -0,0 +1,11 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "delete",
5 path: "/api/v1/admin/runners/{id}",
6 summary:
7 "Remove a runner by id (admin scope): its token stops at once and a job it held is queued again; 404 when there is none",
8 auth: true,
9 role: "admin",
10 status: 204,
11 };
src/contract/openapi/routes/getApiV1AdminRunners.ts added +10
@@ -0,0 +1,10 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "get",
5 path: "/api/v1/admin/runners",
6 summary:
7 "The enrolled runners, each with when it last asked for work (seenAt, milliseconds; 0 for never); answers { items, next_cursor }",
8 auth: true,
9 role: "admin",
10 };
src/contract/openapi/routes/getApiV1RunnerJobsIdGitInfoRefs.ts added +11
@@ -0,0 +1,11 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "get",
5 path: "/api/v1/runner/jobs/{id}/git/info/refs",
6 summary:
7 "The git door of one job (smart HTTP, fetch only): the ref advertisement of the job's repository. Takes that job's token as the bearer; ?service=git-upload-pack",
8 auth: false,
9 role: "anonymous",
10 query: { service: "string" },
11 };
src/contract/openapi/routes/index.ts +7
@@ -2,6 +2,7 @@
22 // One line per route, sorted: a new route is a new file and one new line here.
33 export { route as deleteApiV1AdminAccountsUsername } from "./deleteApiV1AdminAccountsUsername.ts";
44 export { route as deleteApiV1AdminInvitesId } from "./deleteApiV1AdminInvitesId.ts";
5 export { route as deleteApiV1AdminRunnersId } from "./deleteApiV1AdminRunnersId.ts";
56 export { route as deleteApiV1AgentsUsername } from "./deleteApiV1AgentsUsername.ts";
67 export { route as deleteApiV1BoardsSlug } from "./deleteApiV1BoardsSlug.ts";
78 export { route as deleteApiV1CodesId } from "./deleteApiV1CodesId.ts";
@@ -54,6 +55,7 @@ export { route as getApiV1AdminPinned } from "./getApiV1AdminPinned.ts";
5455 export { route as getApiV1AdminProfile } from "./getApiV1AdminProfile.ts";
5556 export { route as getApiV1AdminQuotas } from "./getApiV1AdminQuotas.ts";
5657 export { route as getApiV1AdminRequests } from "./getApiV1AdminRequests.ts";
58 export { route as getApiV1AdminRunners } from "./getApiV1AdminRunners.ts";
5759 export { route as getApiV1AdminSettings } from "./getApiV1AdminSettings.ts";
5860 export { route as getApiV1AdminShield } from "./getApiV1AdminShield.ts";
5961 export { route as getApiV1AdminSubscriptions } from "./getApiV1AdminSubscriptions.ts";
@@ -207,6 +209,7 @@ export { route as getApiV1ReposOwnerNameVision } from "./getApiV1ReposOwnerNameV
207209 export { route as getApiV1ReposOwnerNameWaves } from "./getApiV1ReposOwnerNameWaves.ts";
208210 export { route as getApiV1ReposOwnerNameWavesVersion } from "./getApiV1ReposOwnerNameWavesVersion.ts";
209211 export { route as getApiV1Returns } from "./getApiV1Returns.ts";
212 export { route as getApiV1RunnerJobsIdGitInfoRefs } from "./getApiV1RunnerJobsIdGitInfoRefs.ts";
210213 export { route as getApiV1Search } from "./getApiV1Search.ts";
211214 export { route as getApiV1SecretsScope } from "./getApiV1SecretsScope.ts";
212215 export { route as getApiV1TelemetryFlow } from "./getApiV1TelemetryFlow.ts";
@@ -285,6 +288,7 @@ export { route as postApiV1AdminPinnedNameUpdate } from "./postApiV1AdminPinnedN
285288 export { route as postApiV1AdminReleasesShaRollback } from "./postApiV1AdminReleasesShaRollback.ts";
286289 export { route as postApiV1AdminRequestsUsernameAllow } from "./postApiV1AdminRequestsUsernameAllow.ts";
287290 export { route as postApiV1AdminRequestsUsernameDecline } from "./postApiV1AdminRequestsUsernameDecline.ts";
291 export { route as postApiV1AdminRunners } from "./postApiV1AdminRunners.ts";
288292 export { route as postApiV1Agents } from "./postApiV1Agents.ts";
289293 export { route as postApiV1Apps } from "./postApiV1Apps.ts";
290294 export { route as postApiV1AsksAdmin } from "./postApiV1AsksAdmin.ts";
@@ -409,6 +413,9 @@ export { route as postApiV1ReposOwnerNameSpend } from "./postApiV1ReposOwnerName
409413 export { route as postApiV1ReposOwnerNameSync } from "./postApiV1ReposOwnerNameSync.ts";
410414 export { route as postApiV1ReposOwnerNameTransfer } from "./postApiV1ReposOwnerNameTransfer.ts";
411415 export { route as postApiV1ReposOwnerNameUpstreamCheck } from "./postApiV1ReposOwnerNameUpstreamCheck.ts";
416 export { route as postApiV1RunnerClaim } from "./postApiV1RunnerClaim.ts";
417 export { route as postApiV1RunnerJobsId } from "./postApiV1RunnerJobsId.ts";
418 export { route as postApiV1RunnerJobsIdGitGitUploadPack } from "./postApiV1RunnerJobsIdGitGitUploadPack.ts";
412419 export { route as postApiV1Sessions } from "./postApiV1Sessions.ts";
413420 export { route as postApiV1Share } from "./postApiV1Share.ts";
414421 export { route as postApiV1Start } from "./postApiV1Start.ts";
src/contract/openapi/routes/postApiV1AdminRunners.ts added +13
@@ -0,0 +1,13 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: "/api/v1/admin/runners",
6 summary:
7 "Enrol a runner: a machine that takes checks off this forge when FORGE_CHECKS_PLACEMENT=runner (admin scope); answers its row and its token, shown this once",
8 auth: true,
9 role: "admin",
10 body: { name: "string" },
11 status: 201,
12 reply: { id: "string", name: "string", createdAt: "string", seenAt: "integer", token: "string" },
13 };
src/contract/openapi/routes/postApiV1RunnerClaim.ts added +13
@@ -0,0 +1,13 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: "/api/v1/runner/claim",
6 summary:
7 "A runner asks for a check to run. Takes a runner token as the bearer (from POST /api/v1/admin/runners), never an account's. Held open up to wait seconds (25 at most); answers { job, token, lease } with the job's own token, or 204 when nothing was offered",
8 auth: false,
9 role: "anonymous",
10 body: { wait: "integer", cells: "boolean" },
11 alsoStatus: [204],
12 refusals: { unauthorized: "401: not a runner token this forge enrolled" },
13 };
src/contract/openapi/routes/postApiV1RunnerJobsId.ts added +12
@@ -0,0 +1,12 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: "/api/v1/runner/jobs/{id}",
6 summary:
7 "A runner reports on the job it holds. Takes that job's token as the bearer. lines (of one step) are appended to the check's log; done { steps, contained, note } ends it with a result, abandon hands it back; every report renews the lease and answers { lease } or { ended }",
8 auth: false,
9 role: "anonymous",
10 body: { step: "string", lines: "array", done: "object", abandon: "string" },
11 refusals: { not_found: "404: no such job, not this token's job, or its lease is over: stop the job", invalid_result: "422: done is not { steps: [{ name, ok, error?, ms }], contained, note }" },
12 };
src/contract/openapi/routes/postApiV1RunnerJobsIdGitGitUploadPack.ts added +10
@@ -0,0 +1,10 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: "/api/v1/runner/jobs/{id}/git/git-upload-pack",
6 summary:
7 "The git door of one job: the fetch itself, held to the commit the job names and its base; any other object is refused. Takes that job's token as the bearer",
8 auth: false,
9 role: "anonymous",
10 };
src/contract/tags.ts +1
@@ -44,6 +44,7 @@ const ALIAS: Readonly<Record<string, string>> = {
4444 secrets: "tokens",
4545 sessions: "tokens",
4646 "first-admin": "admin",
47 runner: "meta",
4748 health: "meta",
4849 version: "meta",
4950 releases: "meta",
src/core/event_audience.ts +2
@@ -239,6 +239,8 @@ export function classifiedAudience(kind: string): EventAudience | undefined {
239239 kind.startsWith("mail.") ||
240240 // A restore drill's verdict is the forge owner's (`snapshot/service.ts`).
241241 kind.startsWith("drill.") ||
242 // Which machines take this forge's checks is its admin's (`actions/runners.ts`).
243 kind.startsWith("runner.") ||
242244 // A snapshot moving is the forge owner's too, and has no repository.
243245 kind.startsWith("backup.") ||
244246 // Preparing for a power-off and ending it are the forge owner's (`maintenance/service.ts`).
src/core/migration-pins/20261116005600-check-runner.pin added +1
@@ -0,0 +1,1 @@
1 024e3f117e4ca43c33ee3054c4186a9365fead6e46619ec7801c11f852a2cd43
src/core/migrations/20261116005600-check-runner.ts added +34
@@ -0,0 +1,34 @@
1 // Runners (#13652, R1): machines that take a check's steps off the forge. `check_runner` is one
2 // enrolled runner and the hash of its token; `check_job` is a check's steps while a runner may
3 // hold them: a row lives from the offer to the answer and is deleted then, so the table stays as
4 // small as the checks in flight. A claim is the UPDATE's WHERE on `state`; `lease_until` is
5 // milliseconds since the epoch. `check_job_state` is what the claim walks (oldest queued first),
6 // `check_job_run` what a check's end deletes by.
7 // Additive: two tables and their indexes.
8 export default `
9 CREATE TABLE check_runner (
10 id TEXT PRIMARY KEY,
11 name TEXT NOT NULL,
12 token_hash TEXT NOT NULL UNIQUE,
13 created_by TEXT NOT NULL,
14 created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
15 seen_at INTEGER NOT NULL DEFAULT 0
16 );
17 CREATE TABLE check_job (
18 id TEXT PRIMARY KEY,
19 run_id TEXT NOT NULL,
20 repo_id TEXT NOT NULL,
21 dir TEXT NOT NULL,
22 spec TEXT NOT NULL,
23 needs_cell INTEGER NOT NULL DEFAULT 0,
24 state TEXT NOT NULL DEFAULT 'queued',
25 runner_id TEXT,
26 token_hash TEXT,
27 lease_until INTEGER NOT NULL DEFAULT 0,
28 claims INTEGER NOT NULL DEFAULT 0,
29 result TEXT
30 );
31 CREATE INDEX check_job_state ON check_job (state, id);
32 CREATE INDEX check_job_run ON check_job (run_id);
33 CREATE UNIQUE INDEX check_job_token ON check_job (token_hash) WHERE token_hash IS NOT NULL;
34 `;
src/git/smart-http.ts +20 −4
@@ -22,6 +22,9 @@ import { OWN_REFS_ONLY, reachableFromOwnRefs, refusalLine, wantsOf } from "./own
2222
2323 export type GitService = "git-upload-pack" | "git-receive-pack";
2424
25 /** What a fetch held to `onlyWants` is told when it asks for anything else. */
26 export const ONLY_WANTS = "this credential fetches only the commits its job names";
27
2528 export function isGitService(value: string | null): value is GitService {
2629 return value === "git-upload-pack" || value === "git-receive-pack";
2730 }
@@ -97,6 +100,12 @@ export interface SmartHttpOptions {
97100 * wants must be reachable from the fork's own refs before git is given it.
98101 */
99102 ownRefsOnly?: boolean;
103 /**
104 * The only objects this fetch may want, by id (a runner's job token: its
105 * commit and its base, `checks/runner_jobs.ts`). Spoken in protocol v0 like
106 * a held reader's, so every want is seen before git is given it.
107 */
108 onlyWants?: readonly string[];
100109 }
101110
102111 /** How long a fetch's negotiation may go without a byte before it ends. */
@@ -144,12 +153,18 @@ export async function serviceRpc(
144153 // A held reader's request is read whole first (it is the negotiation, a few
145154 // kilobytes): what it wants is asked of the fork's own refs, then git gets it.
146155 let read: Uint8Array | undefined;
147 if (options.ownRefsOnly === true && service === "git-upload-pack") {
156 const only = options.onlyWants;
157 if ((options.ownRefsOnly === true || only !== undefined) && service === "git-upload-pack") {
148158 read = await negotiation(request);
149159 const asked = read === undefined ? "bad" : wantsOf(read);
150 if (asked === "bad" || !(await reachableFromOwnRefs(repoDir, asked.wants))) {
160 const beyond = asked !== "bad" && only !== undefined && asked.wants.some((want) => !only.includes(want));
161 if (
162 asked === "bad" ||
163 beyond ||
164 (options.ownRefsOnly === true && !(await reachableFromOwnRefs(repoDir, asked.wants)))
165 ) {
151166 options.release?.();
152 return new Response(refusalLine(OWN_REFS_ONLY), {
167 return new Response(refusalLine(beyond ? ONLY_WANTS : OWN_REFS_ONLY), {
153168 headers: {
154169 "content-type": `application/x-${service}-result`,
155170 "cache-control": "no-cache",
@@ -212,7 +227,8 @@ async function spawnService(
212227 // Anything else silently falls back to v0.
213228 const protocol = request.headers.get("git-protocol");
214229 // A held reader is answered in v0 whatever was asked: v2 serves any object by id.
215 if (options.ownRefsOnly !== true && protocol !== null && /^[A-Za-z0-9=:._-]{1,200}$/.test(protocol)) {
230 const held = options.ownRefsOnly === true || options.onlyWants !== undefined;
231 if (!held && protocol !== null && /^[A-Za-z0-9=:._-]{1,200}$/.test(protocol)) {
216232 env.GIT_PROTOCOL = protocol;
217233 }
218234
src/http/api.ts +2
@@ -392,6 +392,7 @@ import { visionRoutes } from "./api/vision.ts";
392392 import { businessPlanRoutes } from "./api/business_plan.ts";
393393 import { tokenRoutes } from "./api/tokens.ts";
394394 import { accountAdmin, adminRoutes } from "./api/admin.ts";
395 import { runnerRoutes } from "./api/runners.ts";
395396 import { maintenanceRoutes } from "./api/maintenance.ts";
396397 import { maintenanceRunsRoutes } from "./api/maintenance_runs.ts";
397398 import { diskReadingRoutes } from "./api/disk_readings.ts";
@@ -1301,6 +1302,7 @@ export function apiRoutes(router: Router<Handler>): void {
13011302 userRoutes(router);
13021303 forgeOriginRoutes(router);
13031304 adminRoutes(router);
1305 runnerRoutes(router);
13041306 maintenanceRoutes(router);
13051307 maintenanceRunsRoutes(router);
13061308 diskReadingRoutes(router);
src/http/api/runners.ts added +207
@@ -0,0 +1,207 @@
1 /**
2 * Runners over the API (#13652): `/api/v1/admin/runners` is an admin's (enrol, list, remove), and
3 * `/api/v1/runner/…` is a runner's own door: claim a job, report on it, fetch its commit.
4 *
5 * The runner routes take neither an account nor an account's token. Each reads its own
6 * credential from `Authorization: Bearer` (`checks/runner_jobs.ts`): a runner token claims, a
7 * job token reports on and fetches its one job while its lease lives. Every other answer is the
8 * same 404, so a guessed job id says nothing. To the rest of the forge both credentials are a
9 * stranger's header: no page, no API route and no git door knows them.
10 */
11
12 import { enrol, list, remove } from "../../actions/runners.ts";
13 import {
14 LEASE_MS,
15 claimJob,
16 deskOf,
17 getRunner,
18 jobByToken,
19 nextOffer,
20 renewJob,
21 runnerByToken,
22 settleJob,
23 type RunnerResult,
24 } from "../../checks/runner_jobs.ts";
25 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
26 import { advertiseRefs, serviceRpc } from "../../git/smart-http.ts";
27 import type { Ctx, Handler } from "../context.ts";
28 import { STRANGER_MAX } from "../limits.ts";
29 import {
30 apiError,
31 body,
32 boundedBody,
33 json,
34 notFound,
35 str,
36 unauthorized,
37 } from "../respond.ts";
38 import type { Router } from "../router.ts";
39 import { accountAdmin } from "./admin.ts";
40
41 /** The longest a claim waits for a job, in seconds: under every proxy's own patience. */
42 export const CLAIM_WAIT_MAX_S = 25;
43 /** The most one report carries: a second or two of a loud check's output. */
44 const REPORT_MAX = 2 * 1024 * 1024;
45 const LINES_MAX = 5_000;
46
47 const bearer = (request: Request): string =>
48 /^Bearer (\S+)$/i.exec(request.headers.get("authorization") ?? "")?.[1] ?? "";
49
50 const lease = { seconds: LEASE_MS / 1000 };
51
52 /** A report's result, read field by field: nothing a runner sends is kept as it came. */
53 function readResult(raw: unknown): RunnerResult | undefined {
54 if (typeof raw !== "object" || raw === null) return undefined;
55 const from = raw as Record<string, unknown>;
56 if (!Array.isArray(from.steps) || from.steps.length > 8) return undefined;
57 const steps = [];
58 for (const s of from.steps as Record<string, unknown>[]) {
59 if (
60 typeof s !== "object" ||
61 s === null ||
62 typeof s.name !== "string" ||
63 typeof s.ok !== "boolean"
64 )
65 return undefined;
66 steps.push({
67 name: s.name.slice(0, 40),
68 ok: s.ok,
69 ...(typeof s.error === "string"
70 ? { error: s.error.slice(0, 2_000) }
71 : {}),
72 ms:
73 typeof s.ms === "number" && Number.isFinite(s.ms)
74 ? Math.max(0, Math.round(s.ms))
75 : 0,
76 });
77 }
78 return {
79 steps,
80 contained: from.contained === true,
81 note: str(from.note).slice(0, 200),
82 };
83 }
84
85 export function runnerRoutes(router: Router<Handler>): void {
86 router.add("POST", "/api/v1/admin/runners", async (ctx) => {
87 const denied = accountAdmin(ctx, "admin");
88 if (denied) return denied;
89 const input = await body(ctx.request);
90 const made = enrol(ctx.app, ctx.auth.user, str(input.name));
91 if (!made.ok) return notFound();
92 return json({ ...made.value.runner, token: made.value.token }, 201);
93 });
94
95 router.add("GET", "/api/v1/admin/runners", (ctx) => {
96 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
97 const denied = accountAdmin(ctx, "ro");
98 if (denied) return denied;
99 const seen = list(ctx.app, ctx.auth.user);
100 return seen.ok
101 ? json({ items: seen.value, next_cursor: null })
102 : notFound();
103 });
104
105 router.add("DELETE", "/api/v1/admin/runners/:id", (ctx) => {
106 const denied = accountAdmin(ctx, "admin");
107 if (denied) return denied;
108 const gone = remove(ctx.app, ctx.auth.user, ctx.params.id!);
109 return gone.ok ? new Response(null, { status: 204 }) : notFound();
110 });
111
112 // A runner asks for work. Held open until a job is offered or `wait` seconds pass (204).
113 router.add("POST", "/api/v1/runner/claim", async (ctx) => {
114 const runner = runnerByToken(ctx.app.db, bearer(ctx.request), Date.now());
115 if (runner === undefined) return unauthorized();
116 const input = await boundedBody(ctx.request, STRANGER_MAX);
117 if (input === "too_large")
118 return apiError("too_large", 413, `at most ${STRANGER_MAX} bytes`);
119 const wait =
120 Math.min(Math.max(Number(input.wait) || 0, 0), CLAIM_WAIT_MAX_S) * 1000;
121 const until = Date.now() + wait;
122 const stop = AbortSignal.any([ctx.request.signal, ctx.app.closing.signal]);
123 for (;;) {
124 // Removed while it waited: its token is no longer one.
125 if (getRunner(ctx.app.db, runner.id) === undefined) return unauthorized();
126 const claim = claimJob(
127 ctx.app.db,
128 { id: runner.id, cells: input.cells === true },
129 Date.now(),
130 );
131 if (claim !== undefined) {
132 deskOf(ctx.app).ears.get(claim.job.id)?.wake();
133 return json({ job: claim.job, token: claim.token, lease });
134 }
135 const left = until - Date.now();
136 if (left <= 0 || stop.aborted) return new Response(null, { status: 204 });
137 await nextOffer(ctx.app, left, stop);
138 }
139 });
140
141 /** The job this request's token holds, with the check waiting on it here; else nothing. */
142 const held = (ctx: Ctx) => {
143 const job = jobByToken(
144 ctx.app.db,
145 ctx.params.id!,
146 bearer(ctx.request),
147 Date.now(),
148 );
149 const ear =
150 job === undefined ? undefined : deskOf(ctx.app).ears.get(job.id);
151 return job === undefined || ear === undefined ? undefined : { job, ear };
152 };
153
154 // A runner reports on its job: log lines, then the end. Every report renews the lease.
155 router.add("POST", "/api/v1/runner/jobs/:id", async (ctx) => {
156 const mine = held(ctx);
157 if (mine === undefined) return notFound();
158 const input = await boundedBody(ctx.request, REPORT_MAX);
159 if (input === "too_large")
160 return apiError("too_large", 413, `at most ${REPORT_MAX} bytes`);
161 const lines = Array.isArray(input.lines)
162 ? input.lines
163 .filter((l): l is string => typeof l === "string")
164 .slice(0, LINES_MAX)
165 : [];
166 if (lines.length > 0) mine.ear.lines(str(input.step), lines);
167 if (typeof input.abandon === "string") {
168 settleJob(ctx.app.db, mine.job.id, { returned: input.abandon });
169 mine.ear.wake();
170 return json({ ended: true });
171 }
172 if (input.done !== undefined) {
173 const result = readResult(input.done);
174 if (result === undefined)
175 return apiError(
176 "invalid_result",
177 422,
178 "done takes { steps: [{ name, ok, error?, ms }], contained, note }",
179 );
180 settleJob(ctx.app.db, mine.job.id, { done: result });
181 mine.ear.wake();
182 return json({ ended: true });
183 }
184 if (!renewJob(ctx.app.db, mine.job.id, Date.now())) return notFound();
185 return json({ lease });
186 });
187
188 // The job's git door: a fetch of its repository, held to the commits the job names.
189 router.add("GET", "/api/v1/runner/jobs/:id/git/info/refs", (ctx) => {
190 const mine = held(ctx);
191 if (
192 mine === undefined ||
193 ctx.url.searchParams.get("service") !== "git-upload-pack"
194 )
195 return notFound();
196 return advertiseRefs(mine.job.dir, "git-upload-pack", ctx.request, {
197 onlyWants: mine.job.wants,
198 });
199 });
200 router.add("POST", "/api/v1/runner/jobs/:id/git/git-upload-pack", (ctx) => {
201 const mine = held(ctx);
202 if (mine === undefined) return notFound();
203 return serviceRpc(mine.job.dir, "git-upload-pack", ctx.request, {
204 onlyWants: mine.job.wants,
205 });
206 });
207 }
src/http/route_marks.ts +2
@@ -85,6 +85,8 @@ export const HEADS: Readonly<Record<string, Mark | "mixed">> = {
8585 "/api/v1/releases": "open",
8686 "/api/v1/repos": "open",
8787 "/api/v1/returns": "open",
88 // A runner's own door: each route reads a runner's or a job's token, never an account.
89 "/api/v1/runner": "open",
8890 "/api/v1/search": "open",
8991 "/api/v1/secrets": "open",
9092 "/api/v1/sessions": "open",
src/runner/main.ts added +453
@@ -0,0 +1,453 @@
1 /**
2 * A runner (#13652, R1): a process that takes checks off a forge. It needs two things of the
3 * forge, its URL and a runner token, and nothing else: no shared disk, no database, no cluster.
4 *
5 * FORGE_URL=https://forge.example FORGE_RUNNER_TOKEN=forge_runner_… bun src/runner/main.ts
6 *
7 * It asks the forge for a job (a long poll), fetches the job's commit over the job's own git door
8 * with the job's token, exports the tree as the forge's own checkout does, runs the recipe's steps
9 * in the cell the forge uses (bubblewrap, where this machine has it), streams the log back as it
10 * comes, reports each step's result and removes its scratch. Nothing dials in to a runner.
11 *
12 * What it keeps between jobs, under its own directory: one bare mirror and one package cache per
13 * repository, so a second job fetches only new objects and installs from disk.
14 *
15 * FORGE_RUNNER_DIR where its scratch and store live. @default $TMPDIR/forge-runner
16 * FORGE_RUNNER_JOBS how many jobs it runs at once. @default 1
17 * FORGE_RUNNER_ONCE `1`: one claim, then exit (a test, a cron line)
18 * FORGE_CELLS `off`: never use cells here; jobs that require one go back to the forge
19 */
20
21 import { existsSync, mkdirSync } from "node:fs";
22 import { rm } from "node:fs/promises";
23 import { tmpdir } from "node:os";
24 import { join } from "node:path";
25 import { exportTree } from "../apps/checkout.ts";
26 import {
27 cellsAvailable,
28 resolveLimits,
29 type CellSpec,
30 } from "../apps/contain.ts";
31 import { childEnv, runStep } from "../apps/process.ts";
32 import { scratchEnv } from "../checks/disk_full.ts";
33 import { giveHistory } from "../checks/history.ts";
34 import { cellInstall, prefillInstallCache } from "../checks/install.ts";
35 import type {
36 Claim,
37 RunnerStepResult,
38 } from "../checks/runner_jobs.ts";
39 import { runGit } from "../git/env.ts";
40 import { isSafeOid } from "../git/guard.ts";
41 import { during } from "../perf/jobs.ts";
42
43 /** How often a job's lines go to the forge; each report renews the lease. */
44 const REPORT_MS = 1_000;
45 /** The most lines one report carries. */
46 const REPORT_LINES = 2_000;
47 /** How long a claim may wait at the forge, in seconds. */
48 const CLAIM_WAIT_S = 20;
49 /** The pause after a claim the forge could not answer. */
50 const RETRY_MS = 5_000;
51 const FETCH_MS = 600_000;
52
53 export interface RunnerOptions {
54 readonly forgeUrl: string;
55 readonly token: string;
56 /** Where the scratch and the store live. */
57 readonly root: string;
58 readonly jobs?: number | undefined;
59 readonly once?: boolean | undefined;
60 /** Stops it: no new claim, and a job in flight is handed back. */
61 readonly signal?: AbortSignal | undefined;
62 readonly say?: ((line: string) => void) | undefined;
63 }
64
65 const pause = (ms: number, signal: AbortSignal): Promise<void> =>
66 new Promise((resolve) => {
67 if (signal.aborted) return resolve();
68 const done = (): void => {
69 clearTimeout(timer);
70 signal.removeEventListener("abort", done);
71 resolve();
72 };
73 const timer = setTimeout(() => during("runner-wait", done), ms);
74 signal.addEventListener("abort", done, { once: true });
75 });
76
77 /** Only the check's own variables: a job can name `CI` and `FORGE_*`, never `PATH` or a proxy. */
78 function checkVars(
79 env: Readonly<Record<string, string>>,
80 ): Record<string, string> {
81 const out: Record<string, string> = {};
82 for (const [key, value] of Object.entries(env ?? {}))
83 if (
84 (key === "CI" || /^FORGE_[A-Z0-9_]+$/.test(key)) &&
85 typeof value === "string" &&
86 !key.startsWith("FORGE_RUNNER")
87 )
88 out[key] = value;
89 return out;
90 }
91
92 /** A name of the forge's as a directory name of ours. */
93 const dirName = (raw: string): string =>
94 raw.replace(/[^A-Za-z0-9_-]/g, "").slice(0, 80) || "none";
95
96 export async function runRunner(options: RunnerOptions): Promise<void> {
97 const say = options.say ?? ((line: string) => console.log(line));
98 const forge = options.forgeUrl.replace(/\/+$/, "");
99 const stopping = options.signal ?? new AbortController().signal;
100 const jobsRoot = join(options.root, "jobs");
101 // What a runner that died mid-job left: no job is using it now.
102 await rm(jobsRoot, { recursive: true, force: true }).catch(() => {});
103 mkdirSync(jobsRoot, { recursive: true });
104 const cells = await cellsAvailable();
105 say(
106 cells.ok
107 ? `runner for ${forge}: steps run in a cell`
108 : `runner for ${forge}: no cells here (${cells.reason}); steps run as plain processes, and a job that requires a cell goes back to the forge`,
109 );
110 /** One fetch at a time into a repository's mirror. */
111 const mirrors = new Map<string, Promise<unknown>>();
112
113 const post = (
114 path: string,
115 token: string,
116 body: unknown,
117 signal?: AbortSignal,
118 ): Promise<Response> =>
119 fetch(`${forge}${path}`, {
120 method: "POST",
121 headers: {
122 authorization: `Bearer ${token}`,
123 "content-type": "application/json",
124 },
125 body: JSON.stringify(body),
126 ...(signal === undefined ? {} : { signal }),
127 });
128
129 async function work(claim: Pick<Claim, "job" | "token">): Promise<void> {
130 const { job, token } = claim;
131 const started = Date.now();
132 const jobDir = join(jobsRoot, dirName(job.id));
133 const dir = join(jobDir, "tree");
134 const home = `${dir}.home`;
135 /** Aborted when the forge no longer wants the job, or this runner is stopping. */
136 const stop = new AbortController();
137 const onStopping = (): void => stop.abort();
138 stopping.addEventListener("abort", onStopping, { once: true });
139 if (stopping.aborted) stop.abort();
140 let dropped = false;
141 const waiting: { step: string; lines: string[] }[] = [];
142 const line = (step: string, text: string): void => {
143 const last = waiting.at(-1);
144 if (
145 last !== undefined &&
146 last.step === step &&
147 last.lines.length < REPORT_LINES
148 )
149 last.lines.push(text);
150 else waiting.push({ step, lines: [text] });
151 };
152 /**
153 * Send what has been said, in order; with nothing to say, the report is the lease's renewal.
154 */
155 const flush = async (): Promise<void> => {
156 const batches =
157 waiting.length === 0
158 ? [{ step: "", lines: [] as string[] }]
159 : waiting.splice(0);
160 for (const [i, batch] of batches.entries()) {
161 try {
162 const res = await post(`/api/v1/runner/jobs/${job.id}`, token, batch);
163 if (res.status === 404) {
164 dropped = true;
165 stop.abort();
166 return;
167 }
168 if (!res.ok) throw new Error(`the forge answered ${res.status}`);
169 } catch {
170 // The forge could not be reached: these lines wait for the next report.
171 waiting.unshift(...batches.slice(i));
172 return;
173 }
174 }
175 };
176 let reporting = true;
177 /** Ends the rest between two reports, so the last one is not kept waiting. */
178 const quiet = new AbortController();
179 const reports = (async () => {
180 while (reporting) {
181 await flush();
182 if (reporting)
183 await pause(REPORT_MS, quiet.signal);
184 }
185 })();
186 const end = async (body: Record<string, unknown>): Promise<void> => {
187 reporting = false;
188 quiet.abort();
189 await reports;
190 await flush();
191 if (dropped) return;
192 // The answer is the job: a forge that did not hear it is told again, while the lease lasts.
193 for (let tries = 0; tries < 5; tries++) {
194 const res = await post(`/api/v1/runner/jobs/${job.id}`, token, body).catch(() => undefined);
195 if (res !== undefined && (res.ok || res.status === 404 || res.status === 422)) return;
196 await pause(REPORT_MS, new AbortController().signal);
197 }
198 };
199 try {
200 if (
201 !isSafeOid(job.commit) ||
202 (job.base !== undefined && !isSafeOid(job.base))
203 )
204 return await end({
205 abandon: "the job names something that is not a commit id",
206 });
207 if (job.cell === "required" && !cells.ok)
208 return await end({
209 abandon: `no cells here (${cells.reason}), and this job requires one`,
210 });
211 // The commit, over the job's own git door, into this repository's mirror.
212 const mirror = join(
213 options.root,
214 "store",
215 "git",
216 `${dirName(job.cache.repo)}.git`,
217 );
218 const auth = {
219 GIT_CONFIG_COUNT: "1",
220 GIT_CONFIG_KEY_0: "http.extraHeader",
221 GIT_CONFIG_VALUE_0: `Authorization: Bearer ${token}`,
222 };
223 const url = `${forge}/api/v1/runner/jobs/${job.id}/git`;
224 const depth = Math.min(
225 Math.max(Math.trunc(Number(job.depth)) || 1, 1),
226 500,
227 );
228 const fetched = (mirrors.get(mirror) ?? Promise.resolve()).then(
229 async (): Promise<string | undefined> => {
230 if (!existsSync(join(mirror, "HEAD"))) {
231 mkdirSync(mirror, { recursive: true });
232 const made = await runGit(["init", "--quiet", "--bare", mirror]);
233 if (!made.ok) return `git init: ${made.error.stderr || "failed"}`;
234 }
235 for (const [oid, deep, kept] of [
236 ...(job.base === undefined ? [] : [[job.base, 1, "base"] as const]),
237 [job.commit, depth, "commit"] as const,
238 ]) {
239 const got = await runGit(
240 ["fetch", "--quiet", "--no-tags", `--depth=${deep}`, url, oid],
241 {
242 cwd: mirror,
243 env: auth,
244 timeoutMs: FETCH_MS,
245 signal: stop.signal,
246 },
247 );
248 if (!got.ok)
249 return `git fetch ${oid.slice(0, 7)}: ${got.error.stderr.trim().slice(0, 300) || "failed"}`;
250 // A fetch offers what its refs reach as already had: the last base and the last
251 // commit are kept under a ref, so the next fetch brings only what is new.
252 await runGit(["update-ref", `refs/runner/${kept}`, oid], { cwd: mirror });
253 }
254 return undefined;
255 },
256 );
257 mirrors.set(
258 mirror,
259 fetched.catch(() => {}),
260 );
261 const refused = await fetched;
262 if (refused !== undefined) return await end({ abandon: refused });
263 mkdirSync(jobDir, { recursive: true });
264 const exported = await exportTree(mirror, job.commit, dir);
265 if (!exported.ok) return await end({ abandon: exported.error });
266 if (job.base !== undefined) {
267 const given = await giveHistory(mirror, dir, job.base, job.commit);
268 if (!given.ok) return await end({ abandon: given.error });
269 }
270 line(
271 "runner",
272 `runner: ${job.commit.slice(0, 7)} fetched and checked out in ${((Date.now() - started) / 1000).toFixed(1)}s`,
273 );
274 const limits = resolveLimits("step", job.limits);
275 const cell: CellSpec | undefined = cells.ok
276 ? {
277 dir,
278 writable: [],
279 home,
280 // The runner's own store and scratch, its home and the temp directory look empty
281 // inside.
282 hide: [options.root, process.env["HOME"] ?? "", tmpdir()].filter(
283 (p) => p !== "",
284 ),
285 egress: (job.egress ?? []).filter((h) => typeof h === "string"),
286 reach: [...new Set([forge, job.env?.["FORGE_URL"] ?? forge])],
287 limits,
288 egressMode: job.egressMode === "report" ? "report" : "enforce",
289 label: `check:${job.run}`,
290 onUndeclared: (host, port) =>
291 line(
292 "runner",
293 `runner: reached ${host}:${port}, which forge.json does not declare`,
294 ),
295 }
296 : undefined;
297 if (cell === undefined)
298 line(
299 "runner",
300 `uncontained on this runner: ${cells.ok ? "" : cells.reason}; the check runs unrestricted`,
301 );
302 const env = childEnv({
303 ...checkVars(job.env),
304 FORGE_CELL_CPUS: String(limits.cpus),
305 ...scratchEnv(cell === undefined ? undefined : home),
306 });
307 const timeoutMs =
308 Math.min(Math.max(Number(job.timeoutS) || 900, 1), 7_200) * 1000;
309 const warm = join(
310 options.root,
311 "store",
312 "install",
313 dirName(job.cache.repo),
314 );
315 const steps: RunnerStepResult[] = [];
316 for (const step of job.steps) {
317 if (step.name !== "install" && step.name !== "check") continue;
318 const from = Date.now();
319 const log = (text: string): void => line(step.name, text);
320 let ran;
321 if (step.name === "install" && cell !== undefined) {
322 // As on the forge: this machine fills the repository's cache without running scripts,
323 // and the cell installs from it read-only, so no job writes what the next one installs.
324 const fill = await prefillInstallCache({
325 cache: warm,
326 dir,
327 env,
328 log,
329 signal: stop.signal,
330 timeoutMs,
331 });
332 const filled = fill.ok && fill.value;
333 if (!fill.ok) log(`cache not filled: ${fill.error}`);
334 const use = cellInstall(step.command, env, filled ? warm : undefined);
335 ran = await runStep("install", use.command, {
336 cwd: dir,
337 env: use.env,
338 log,
339 signal: stop.signal,
340 timeoutMs,
341 cell: filled ? { ...cell, reveal: [warm] } : cell,
342 });
343 } else {
344 ran = await runStep(step.name, step.command, {
345 cwd: dir,
346 // Uncontained, the repository's package cache is shared between its jobs.
347 env:
348 step.name === "install"
349 ? { ...env, BUN_INSTALL_CACHE_DIR: warm }
350 : env,
351 log,
352 signal: stop.signal,
353 timeoutMs,
354 cell,
355 });
356 }
357 steps.push({
358 name: step.name,
359 ok: ran.ok,
360 ...(ran.ok ? {} : { error: ran.error }),
361 ms: Date.now() - from,
362 });
363 if (!ran.ok) break;
364 }
365 if (stopping.aborted && !dropped)
366 return await end({ abandon: "the runner is stopping" });
367 await end({
368 done: {
369 steps,
370 contained: cell !== undefined,
371 note: cells.ok ? "" : cells.reason,
372 },
373 });
374 const failed = steps.find((s) => !s.ok);
375 say(
376 `job ${job.id} (${job.repo}@${job.commit.slice(0, 7)}): ${dropped ? "dropped by the forge" : failed === undefined ? "ok" : `${failed.name} failed`} in ${((Date.now() - started) / 1000).toFixed(1)}s`,
377 );
378 } catch (error) {
379 await end({
380 abandon: `the runner failed: ${error instanceof Error ? error.message : String(error)}`,
381 });
382 say(`job ${job.id}: ${String(error)}`);
383 } finally {
384 reporting = false;
385 quiet.abort();
386 stopping.removeEventListener("abort", onStopping);
387 await rm(jobDir, { recursive: true, force: true }).catch((error) =>
388 say(`scratch not removed: ${String(error)}`),
389 );
390 }
391 }
392
393 async function loop(): Promise<void> {
394 while (!stopping.aborted) {
395 let res: Response;
396 try {
397 res = await post(
398 "/api/v1/runner/claim",
399 options.token,
400 { wait: options.once === true ? 0 : CLAIM_WAIT_S, cells: cells.ok },
401 stopping,
402 );
403 } catch {
404 if (stopping.aborted || options.once === true) return;
405 await pause(RETRY_MS, stopping);
406 continue;
407 }
408 if (res.status === 401)
409 throw new Error(
410 "the forge does not know this runner token (removed, or mistyped)",
411 );
412 if (res.status === 200)
413 await work((await res.json()) as Pick<Claim, "job" | "token">);
414 else {
415 await res.body?.cancel().catch(() => {});
416 if (res.status !== 204) await pause(RETRY_MS, stopping);
417 }
418 if (options.once === true) return;
419 }
420 }
421
422 const n = Math.min(Math.max(Math.trunc(options.jobs ?? 1), 1), 64);
423 await Promise.all(Array.from({ length: n }, () => loop()));
424 }
425
426 if (import.meta.main) {
427 const forgeUrl = process.env["FORGE_URL"] ?? "";
428 const token = process.env["FORGE_RUNNER_TOKEN"] ?? "";
429 if (!/^https?:\/\//.test(forgeUrl) || token === "") {
430 console.error(
431 "usage: FORGE_URL=https://your.forge FORGE_RUNNER_TOKEN=forge_runner_… bun src/runner/main.ts",
432 );
433 process.exit(2);
434 }
435 // The token is this process's alone: nothing it starts may read it from the environment.
436 delete process.env["FORGE_RUNNER_TOKEN"];
437 const stopping = new AbortController();
438 for (const sig of ["SIGINT", "SIGTERM"] as const)
439 process.on(sig, () => stopping.abort());
440 try {
441 await runRunner({
442 forgeUrl,
443 token,
444 root: process.env["FORGE_RUNNER_DIR"] ?? join(tmpdir(), "forge-runner"),
445 jobs: Number(process.env["FORGE_RUNNER_JOBS"] ?? "1") || 1,
446 once: process.env["FORGE_RUNNER_ONCE"] === "1",
447 signal: stopping.signal,
448 });
449 } catch (error) {
450 console.error(error instanceof Error ? error.message : String(error));
451 process.exit(1);
452 }
453 }
test/checks-runner-placement.test.ts added +592
@@ -0,0 +1,592 @@
1 /**
2 * `FORGE_CHECKS_PLACEMENT=runner` (#13652, R1): a pull's check as a job a runner claims, runs and
3 * reports on, with nothing of the forge's but its URL and a token. The runner here is the real
4 * program, a separate process. A silent runner and a stolen job token are played over HTTP.
5 */
6 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
7 import { mkdirSync } from "node:fs";
8 import { join } from "node:path";
9 import { openPull } from "../src/actions/pulls.ts";
10 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
11 import { placementFromEnv, placementWords } from "../src/checks/placement.ts";
12 import {
13 LEASE_MS,
14 claimJob,
15 deskOf,
16 dropJob,
17 enrolRunner,
18 getRunner,
19 jobByToken,
20 lapseJobs,
21 offerJob,
22 renewJob,
23 viewJob,
24 withdrawJob,
25 type RunnerJob,
26 } from "../src/checks/runner_jobs.ts";
27 import { readSteps, runnerTakes, runnerVerdict } from "../src/checks/runner_step.ts";
28 import { startChecks } from "../src/checks/service.ts";
29 import { createToken } from "../src/core/tokens.ts";
30 import { ok, err } from "../src/kernel/result.ts";
31 import { readRunLog } from "../src/runs/log.ts";
32 import { checkJson, checksForge } from "./helpers/checks-forge.ts";
33 import { gitClientEnv } from "./helpers/git-client.ts";
34 import { quietLogs } from "./helpers/globals.ts";
35
36 quietLogs();
37
38 const f = checksForge("forge-runner-");
39 /** The check every pull of `lib` gets from main: green unless the pull adds a file named RED. */
40 const CHECK = "echo ran-the-check; test ! -f RED";
41
42 let adminToken = "";
43 const api = (path: string, token: string, body?: unknown): Promise<Response> =>
44 fetch(`${f.servers.url}${path}`, {
45 method: body === undefined ? "GET" : "POST",
46 headers: {
47 authorization: `Bearer ${token}`,
48 "content-type": "application/json",
49 },
50 ...(body === undefined ? {} : { body: JSON.stringify(body) }),
51 });
52
53 /** A pull of `lib` from a new branch `name` that adds `file`; its head commit. */
54 async function pull(name: string, file: string): Promise<string> {
55 const made = await commitToBranch(f.app, f.alice, f.repo, {
56 branch: name,
57 message: `Add ${file}`,
58 files: [{ path: file, content: `${name}\n` }],
59 });
60 if (!made.ok) throw new Error(JSON.stringify(made.error));
61 const opened = await openPull(f.app, f.alice, f.repo, {
62 title: name,
63 body: "",
64 baseBranch: "main",
65 headBranch: name,
66 });
67 if (!opened.ok) throw new Error(JSON.stringify(opened.error));
68 return made.value.commitOid;
69 }
70
71 const logOf = (runId: string): string =>
72 readRunLog(f.app.config.dataDir, runId).lines.join("\n");
73
74 const job: Omit<RunnerJob, "id"> = {
75 run: "run-1",
76 repo: "alice/lib",
77 commit: "a".repeat(40),
78 base: "b".repeat(40),
79 depth: 3,
80 steps: [
81 { name: "install", command: "bun install" },
82 { name: "check", command: "bun test" },
83 ],
84 env: { CI: "1", FORGE_CHECK: "run-1" },
85 limits: { cpus: 2, memoryMiB: 2048 },
86 timeoutS: 900,
87 egress: ["registry.npmjs.org"],
88 egressMode: "enforce",
89 cell: "preferred",
90 cache: { repo: "repo-1", tree: "c".repeat(40), base: "b".repeat(40) },
91 };
92
93 beforeAll(async () => {
94 // The forge under test offers its checks to runners, and waits only briefly for one.
95 await f.checks.stop();
96 f.checks = startChecks(f.app, {
97 forgeUrl: f.servers.url,
98 concurrency: 1,
99 placement: { mode: "runner", waitMs: 300 },
100 });
101 adminToken = createToken(f.app.db, f.alice.id, "admin", "admin").token;
102 const gate = await commitToBranch(f.app, f.alice, f.repo, {
103 message: "Declare the check",
104 files: [{ path: "forge.json", content: checkJson(CHECK) }],
105 });
106 if (!gate.ok) throw new Error(JSON.stringify(gate.error));
107 await f.settled((r) => r.commitOid === gate.value.commitOid, 30_000);
108 });
109
110 describe("the setting", () => {
111 test("runner is a placement only when asked for, and a run says where it ran", () => {
112 expect(placementFromEnv({})).toEqual({ mode: "process" });
113 expect(placementFromEnv({ FORGE_CHECKS_PLACEMENT: "runner" })).toEqual({
114 mode: "runner",
115 });
116 expect(placementFromEnv({ FORGE_CHECKS_PLACEMENT: "runners" })).toEqual({
117 mode: "process",
118 });
119 expect(placementWords({ placement: "runner", node: "mac" })).toBe(
120 "ran on runner mac",
121 );
122 expect(placementWords({ placement: "process", node: "" })).toBe("");
123 });
124
125 test("only a check that hands back nothing but its verdict goes to a runner", () => {
126 expect(
127 runnerTakes({
128 tier: "fast",
129 keep: false,
130 keepImpact: false,
131 isMain: false,
132 }),
133 ).toBe(true);
134 expect(
135 runnerTakes({ tier: "run", keep: true, keepImpact: true, isMain: true }),
136 ).toBe(false);
137 expect(runnerTakes({ tier: "verify", keep: false, keepImpact: true })).toBe(
138 false,
139 );
140 expect(runnerTakes({ tier: "full" })).toBe(false);
141 expect(runnerTakes({ tier: "fast", only: ["a.test.ts"] })).toBe(false);
142 });
143
144 test("a result is held to the steps the job asked for: an empty or partial one is no answer", () => {
145 const asked = job.steps;
146 const yes = (name: string) => ({ name, ok: true, ms: 1 });
147 const no = (name: string) => ({ name, ok: false, error: `${name} exited with code 1`, ms: 1 });
148 expect(readSteps(asked, [yes("install"), yes("check")])).toEqual({ ran: ok() });
149 expect(readSteps(asked, [yes("install"), no("check")])).toEqual({ ran: err("check exited with code 1"), failed: "check" });
150 expect(readSteps(asked, [no("install")])).toEqual({ ran: err("install exited with code 1"), failed: "install" });
151 // Never green: nothing run, the check left out, a step of its own, a step after a failure.
152 expect(readSteps(asked, [])).toBeUndefined();
153 expect(readSteps(asked, [yes("install")])).toBeUndefined();
154 expect(readSteps(asked, [yes("check")])).toBeUndefined();
155 expect(readSteps(asked, [yes("install"), yes("lint")])).toBeUndefined();
156 expect(readSteps(asked, [no("install"), yes("check")])).toBeUndefined();
157 expect(readSteps(asked, [yes("install"), yes("check"), yes("check")])).toBeUndefined();
158 });
159
160 test("a runner's answer stands only as green, the check's exit code or its timeout", () => {
161 const end = {
162 taken: true as const,
163 runner: "mac",
164 contained: true,
165 note: "",
166 };
167 expect(runnerVerdict({ ...end, ran: ok() }, [], false)).toEqual({
168 ok: true,
169 });
170 expect(
171 runnerVerdict(
172 { ...end, ran: err("check exited with code 1"), failed: "check" },
173 ["boom"],
174 false,
175 ),
176 ).toMatchObject({ ok: false, stage: "check" });
177 expect(
178 runnerVerdict(
179 { ...end, ran: err("check took longer than 900s"), failed: "check" },
180 [],
181 false,
182 ),
183 ).toMatchObject({ ok: false, stage: "timeout" });
184 // The machine's, not the check's: the forge runs it itself.
185 expect(
186 typeof runnerVerdict(
187 { ...end, ran: err("install exited with code 1"), failed: "install" },
188 [],
189 false,
190 ),
191 ).toBe("string");
192 expect(
193 typeof runnerVerdict(
194 { ...end, ran: err("check was killed by SIGKILL"), failed: "check" },
195 [],
196 false,
197 ),
198 ).toBe("string");
199 expect(
200 typeof runnerVerdict(
201 {
202 ...end,
203 ran: err("check stopped: it used more memory than its limit"),
204 failed: "check",
205 },
206 [],
207 false,
208 ),
209 ).toBe("string");
210 });
211 });
212
213 describe("the desk", () => {
214 test("a job survives the wire and the desk unchanged, and names nothing of the forge's disk", () => {
215 const runner = enrolRunner(f.app.db, "wire", f.alice.id).runner;
216 const offered = offerJob(f.app.db, {
217 runId: "run-1",
218 repoId: "repo-1",
219 dir: "/data/repos/alice/lib.git",
220 job,
221 });
222 expect(JSON.parse(JSON.stringify(offered))).toEqual(offered);
223 expect(offered).toEqual({ ...job, id: offered.id });
224 const claim = claimJob(f.app.db, { id: runner.id, cells: true }, 1_000);
225 expect(claim?.job).toEqual(offered);
226 expect(JSON.stringify(claim)).not.toContain("/data/repos");
227 expect(claim?.token).toStartWith("forge_job_");
228 expect(withdrawJob(f.app.db, offered.id)).toBe(false);
229 dropJob(f.app.db, offered.id);
230 });
231
232 test("of two runners asking at once, one takes the job; a job that needs a cell goes to one that has them", () => {
233 const one = enrolRunner(f.app.db, "one", f.alice.id).runner;
234 const two = enrolRunner(f.app.db, "two", f.alice.id).runner;
235 const offered = offerJob(f.app.db, {
236 runId: "run-2",
237 repoId: "repo-1",
238 dir: "/x",
239 job: { ...job, cell: "required" },
240 });
241 expect(
242 claimJob(f.app.db, { id: one.id, cells: false }, 1_000),
243 ).toBeUndefined();
244 const first = claimJob(f.app.db, { id: one.id, cells: true }, 1_000);
245 const second = claimJob(f.app.db, { id: two.id, cells: true }, 1_000);
246 expect(first?.job.id).toBe(offered.id);
247 expect(second).toBeUndefined();
248 expect(viewJob(f.app.db, offered.id)).toMatchObject({
249 state: "claimed",
250 claims: 1,
251 runner: "one",
252 });
253 dropJob(f.app.db, offered.id);
254 });
255
256 test("a lease that runs out queues the job again: the old token is dead and another runner takes it", () => {
257 const one = enrolRunner(f.app.db, "sleepy", f.alice.id).runner;
258 const two = enrolRunner(f.app.db, "awake", f.alice.id).runner;
259 const offered = offerJob(f.app.db, {
260 runId: "run-3",
261 repoId: "repo-1",
262 dir: "/x",
263 job,
264 });
265 const first = claimJob(f.app.db, { id: one.id, cells: true }, 1_000)!;
266 expect(jobByToken(f.app.db, offered.id, first.token, 1_001)?.wants).toEqual(
267 [job.commit, job.base!],
268 );
269 // A runner at work is connected, though it is not asking for more.
270 expect(getRunner(f.app.db, one.id)?.seenAt).toBe(1_001);
271 // A report in time pushes the lease out; silence past it loses the job.
272 expect(renewJob(f.app.db, offered.id, 1_000 + LEASE_MS - 1)).toBe(true);
273 expect(lapseJobs(f.app.db, 1_000 + LEASE_MS)).not.toContain(offered.id);
274 expect(lapseJobs(f.app.db, 1_000 + 2 * LEASE_MS)).toContain(offered.id);
275 expect(viewJob(f.app.db, offered.id)).toMatchObject({
276 state: "queued",
277 runner: "",
278 });
279 expect(
280 jobByToken(f.app.db, offered.id, first.token, 1_000 + 2 * LEASE_MS),
281 ).toBeUndefined();
282 expect(renewJob(f.app.db, offered.id, 1_000 + 2 * LEASE_MS)).toBe(false);
283 const second = claimJob(
284 f.app.db,
285 { id: two.id, cells: true },
286 1_000 + 2 * LEASE_MS,
287 )!;
288 expect(second.job.id).toBe(offered.id);
289 expect(second.token).not.toBe(first.token);
290 expect(withdrawJob(f.app.db, offered.id)).toBe(false);
291 dropJob(f.app.db, offered.id);
292 });
293 });
294
295 describe("a pull's check with FORGE_CHECKS_PLACEMENT=runner", () => {
296 test("with no runner connected the forge runs it itself, at once", async () => {
297 // The runners enrolled above never asked for work: none is connected.
298 const oid = await pull("alone", "a.txt");
299 const run = await f.settled((r) => r.commitOid === oid, 30_000);
300 expect(run.state).toBe("ok");
301 expect(run.placement).toBe("process");
302 expect(logOf(run.id)).toContain(
303 "placement: runner; no runner is connected; running here",
304 );
305 });
306
307 test("a runner that goes silent loses the job, which is no try of the check; nobody else takes it and the forge runs it", async () => {
308 const silent = enrolRunner(f.app.db, "silent", f.alice.id);
309 // It asks first and is held open, as a runner is; then the pull is opened.
310 const asked = api("/api/v1/runner/claim", silent.token, {
311 wait: 20,
312 cells: true,
313 });
314 await f.until(
315 () => deskOf(f.app).claimers.size > 0,
316 "the runner waiting in its claim",
317 );
318 const oid = await pull("silent", "s.txt");
319 const claim = (await (await asked).json()) as {
320 job: RunnerJob;
321 token: string;
322 lease: { seconds: number };
323 };
324 expect(claim.job.commit).not.toBe(oid);
325 expect(claim.job.steps).toEqual([{ name: "check", command: CHECK }]);
326 expect(claim.lease.seconds).toBe(LEASE_MS / 1000);
327 // Its lease runs out with no report (the clock, moved by hand).
328 expect(lapseJobs(f.app.db, Date.now() + LEASE_MS + 1)).toEqual([
329 claim.job.id,
330 ]);
331 deskOf(f.app).ears.get(claim.job.id)?.wake();
332 expect(
333 (
334 await api(`/api/v1/runner/jobs/${claim.job.id}`, claim.token, {
335 step: "check",
336 lines: ["late"],
337 })
338 ).status,
339 ).toBe(404);
340 const run = await f.settled((r) => r.commitOid === oid, 30_000);
341 expect(run.state).toBe("ok");
342 expect(run.attempt).toBe(0);
343 expect(f.checksOf(oid).length).toBe(1);
344 const log = logOf(run.id);
345 expect(log).toContain("placement: runner silent");
346 expect(log).toContain(
347 "runner silent went silent: the job is queued again, and that is not a try",
348 );
349 expect(log).toContain("no runner took it");
350 expect(log).not.toContain("late");
351 });
352
353 test("a job token reaches its own job and nothing else: no other repository, no other object, no other job, no claim", async () => {
354 const secret = await createRepo(f.app, f.alice, {
355 name: "secret",
356 visibility: "private",
357 });
358 if (!secret.ok) throw new Error(String(secret.error));
359 const hidden = await commitToBranch(f.app, f.alice, secret.value, {
360 message: "Seed",
361 files: [{ path: "S.md", content: "s\n" }],
362 });
363 if (!hidden.ok) throw new Error(JSON.stringify(hidden.error));
364 const thief = enrolRunner(f.app.db, "thief", f.alice.id);
365 const asked = api("/api/v1/runner/claim", thief.token, {
366 wait: 20,
367 cells: true,
368 });
369 await f.until(
370 () => deskOf(f.app).claimers.size > 0,
371 "the runner waiting in its claim",
372 );
373 const oid = await pull("stolen", "t.txt");
374 const claim = (await (await asked).json()) as {
375 job: RunnerJob;
376 token: string;
377 };
378 const door = `/api/v1/runner/jobs/${claim.job.id}`;
379 const git = async (
380 url: string,
381 want: string,
382 ): Promise<{ code: number; err: string }> => {
383 const dir = join(
384 f.dataDir,
385 `fetch-${Math.random().toString(36).slice(2)}`,
386 );
387 mkdirSync(dir);
388 const env = gitClientEnv({
389 GIT_CONFIG_COUNT: "1",
390 GIT_CONFIG_KEY_0: "http.extraHeader",
391 GIT_CONFIG_VALUE_0: `Authorization: Bearer ${claim.token}`,
392 });
393 const init = Bun.spawn(["git", "init", "--quiet", "--bare", dir], {
394 env,
395 stdout: "ignore",
396 stderr: "ignore",
397 });
398 await init.exited;
399 const proc = Bun.spawn(
400 ["git", "fetch", "--quiet", "--no-tags", "--depth=1", url, want],
401 { cwd: dir, env, stdout: "ignore", stderr: "pipe" },
402 );
403 const [text, code] = await Promise.all([
404 new Response(proc.stderr).text(),
405 proc.exited,
406 ]);
407 return { code, err: text };
408 };
409 // Its own commit, over its own door: yes.
410 expect(
411 (await git(`${f.servers.url}${door}/git`, claim.job.commit)).code,
412 ).toBe(0);
413 // Another object of the same repository (the pull's own head, which is not what the job names):
414 // no.
415 const other = await git(`${f.servers.url}${door}/git`, oid);
416 expect(other.code).not.toBe(0);
417 expect(other.err).toContain("fetches only the commits its job names");
418 // Another repository, by its own git door or through this job's: no.
419 expect(
420 (await git(`${f.servers.url}/alice/secret.git`, hidden.value.commitOid))
421 .code,
422 ).not.toBe(0);
423 expect(
424 (await git(`${f.servers.url}${door}/git`, hidden.value.commitOid)).code,
425 ).not.toBe(0);
426 // The API, the account, another job, a claim: it is no credential there.
427 expect((await api("/api/v1/repos/alice/secret", claim.token)).status).toBe(
428 404,
429 );
430 expect((await api("/api/v1/user", claim.token)).status).toBe(401);
431 expect(
432 (await api("/api/v1/runner/claim", claim.token, { wait: 0 })).status,
433 ).toBe(401);
434 expect(
435 (
436 await api("/api/v1/runner/jobs/01NOTAJOB", claim.token, {
437 lines: ["x"],
438 })
439 ).status,
440 ).toBe(404);
441 const elsewhere = offerJob(f.app.db, {
442 runId: "run-9",
443 repoId: "repo-9",
444 dir: "/x",
445 job,
446 });
447 expect(
448 (
449 await api(`/api/v1/runner/jobs/${elsewhere.id}`, claim.token, {
450 lines: ["x"],
451 })
452 ).status,
453 ).toBe(404);
454 expect(withdrawJob(f.app.db, elsewhere.id)).toBe(true);
455 // Nor is a runner token, or an account's, a job's.
456 expect((await api(door, thief.token, { lines: ["x"] })).status).toBe(404);
457 expect((await api(door, adminToken, { lines: ["x"] })).status).toBe(404);
458 expect(
459 (await api("/api/v1/runner/claim", adminToken, { wait: 0 })).status,
460 ).toBe(401);
461 expect((await api("/api/v1/user", thief.token)).status).toBe(401);
462 // Handed back, the forge runs it.
463 expect(
464 (await api(door, claim.token, { abandon: "done looking" })).status,
465 ).toBe(200);
466 const run = await f.settled((r) => r.commitOid === oid, 30_000);
467 expect(run.state).toBe("ok");
468 expect(logOf(run.id)).toContain(
469 "runner thief handed it back: done looking; running here",
470 );
471 expect((await api(door, claim.token, { lines: ["x"] })).status).toBe(404);
472 });
473
474 test("a runner that says done with no steps does not make the check green: the forge runs it", async () => {
475 const lazy = enrolRunner(f.app.db, "lazy", f.alice.id);
476 const asked = api("/api/v1/runner/claim", lazy.token, { wait: 20, cells: true });
477 await f.until(() => deskOf(f.app).claimers.size > 0, "the runner waiting in its claim");
478 const oid = await pull("lazy", "l.txt");
479 const claim = (await (await asked).json()) as { job: RunnerJob; token: string };
480 const done = await api(`/api/v1/runner/jobs/${claim.job.id}`, claim.token, { done: { steps: [], contained: true, note: "" } });
481 expect(done.status).toBe(200);
482 const run = await f.settled((r) => r.commitOid === oid, 30_000);
483 expect(run.state).toBe("ok");
484 // Green because this machine ran the command, not because the runner said so.
485 expect(run.placement).toBe("process");
486 const log = logOf(run.id);
487 expect(log).toContain("runner lazy sent a result that does not match the job's steps; running here");
488 expect(log).toContain("ran-the-check");
489 });
490
491 describe("with a runner process connected", () => {
492 let proc: ReturnType<typeof Bun.spawn> | undefined;
493 let runnerId = "";
494
495 beforeAll(async () => {
496 const made = await api("/api/v1/admin/runners", adminToken, {
497 name: "box",
498 });
499 expect(made.status).toBe(201);
500 const enrolled = (await made.json()) as {
501 id: string;
502 name: string;
503 token: string;
504 };
505 expect(enrolled.token).toStartWith("forge_runner_");
506 runnerId = enrolled.id;
507 // The runner is given the forge's URL, its token and a directory of its own: nothing else.
508 proc = Bun.spawn(
509 [
510 process.execPath,
511 join(import.meta.dir, "..", "src", "runner", "main.ts"),
512 ],
513 {
514 env: {
515 PATH: process.env["PATH"] ?? "/usr/bin:/bin",
516 HOME: join(f.dataDir, "runner-home"),
517 FORGE_URL: f.servers.url,
518 FORGE_RUNNER_TOKEN: enrolled.token,
519 FORGE_RUNNER_DIR: join(f.dataDir, "runner"),
520 FORGE_CELLS: "off",
521 },
522 stdout: "ignore",
523 stderr: "ignore",
524 },
525 );
526 await f.until(
527 () => deskOf(f.app).claimers.size > 0,
528 "the runner process waiting in its claim",
529 );
530 });
531 afterAll(async () => {
532 proc?.kill();
533 await proc?.exited;
534 });
535
536 test("a green check runs on the runner and lands in the run with its log", async () => {
537 const oid = await pull("green", "g.txt");
538 const run = await f.settled((r) => r.commitOid === oid, 30_000);
539 expect(run.state).toBe("ok");
540 expect(run.placement).toBe("runner");
541 expect(run.node).toBe("box");
542 expect(run.sandboxEnforced).toBe(false);
543 const log = logOf(run.id);
544 expect(log).toContain("placement: runner box");
545 expect(log).toContain(`$ ${CHECK}`);
546 expect(log).toContain("ran-the-check");
547 expect(log).toContain("uncontained on this runner: FORGE_CELLS=off");
548 expect(log).not.toContain("running here");
549 // Nothing was checked out on the forge's side, and the desk is empty again.
550 expect(
551 f.app.db
552 .query("SELECT COUNT(*) AS n FROM check_job WHERE run_id = ?1")
553 .get(run.id),
554 ).toEqual({ n: 0 });
555 });
556
557 test("a red check is red, with the command's output and its exit code", async () => {
558 const oid = await pull("red", "RED");
559 const run = await f.settled((r) => r.commitOid === oid, 30_000);
560 expect(run.state).toBe("failed");
561 expect(run.placement).toBe("runner");
562 expect(run.summary).toStartWith("check failed");
563 expect(logOf(run.id)).toContain("ran-the-check");
564 });
565
566 test("an admin lists it as seen, and a removed runner's token stops", async () => {
567 const listed = (await (
568 await api("/api/v1/admin/runners", adminToken)
569 ).json()) as { items: { id: string; name: string; seenAt: number }[] };
570 const box = listed.items.find((r) => r.id === runnerId);
571 expect(box?.name).toBe("box");
572 expect(box!.seenAt).toBeGreaterThan(0);
573 expect(JSON.stringify(listed)).not.toContain("forge_runner_");
574 // A member is not told the door exists.
575 const member = createToken(f.app.db, f.bob.id, "bob", "admin").token;
576 expect([403, 404]).toContain(
577 (await api("/api/v1/admin/runners", member)).status,
578 );
579 expect(
580 (
581 await fetch(`${f.servers.url}/api/v1/admin/runners/${runnerId}`, {
582 method: "DELETE",
583 headers: { authorization: `Bearer ${adminToken}` },
584 })
585 ).status,
586 ).toBe(204);
587 // Its next claim is refused, and the process ends saying so.
588 expect(await proc!.exited).toBe(1);
589 proc = undefined;
590 });
591 });
592 });
test/job-slices.test.ts +1
@@ -261,6 +261,7 @@ function jobs(): Record<string, () => unknown> {
261261 * on.
262262 */
263263 const UNTIMED: Record<string, string> = {
264 "runner-wait": "one-shot timers that only resolve a promise: a runner's claim held open, a check's look at its job each second (checks/runner_jobs.ts, runner_step.ts) and the runner program's own pauses; off unless FORGE_CHECKS_PLACEMENT=runner, and checks-runner-placement.test.ts drives all three",
264265 "drill-watch": "a /proc read of the drill child's memory and a state write every WATCH_MS, only while a restore drill runs in its own process; drill-job tests drive runDrillInChild directly",
265266 desk: "expires notes past their time: one indexed delete on the desk table every 30 s (actions/desk.ts); the fixture has no notes",
266267 ledger: "walks the cost ledger's hash chain at boot and once a day (#12273): file reads proportional to the ledger, off the request path; the fixture has no ledger",
test/pages-budget.test.ts +1
@@ -66,6 +66,7 @@ const NOT_PAGES = new Set([
6666 "/events", "/system/stream", "/settings/teams/stream", "/fleet/graph/live",
6767 "/settings/traffic/scrapes/export", "/settings/backups/live", "/:owner/:name/stream",
6868 "/settings/origins/flows/:id/stream", "/:owner/:name/archive/:file", // a tarball download, not a page
69 "/api/v1/runner/jobs/:id/git/info/refs", // a runner's git door for one job (#13652), not a page
6970 ]);
7071
7172 /** Patterns with an id in them, and the path the fixture fills them with. */
test/security-floor-routes.test.ts +1
@@ -71,6 +71,7 @@ const CART = "the seed pins no build with a cart (test/build-cart.test.ts covers
7171 const UNSEEDED: Record<string, string> = {
7272 "/:owner/-/teams/:team": ORG,
7373 "/api/v1/repos/:owner/:name/drop/:file": "a file name in an app's private drop; the seed drops none",
74 "/api/v1/runner/jobs/:id/git/info/refs": "a job a runner holds, read with that job's own token; the seed offers none (test/checks-runner-placement.test.ts covers it)",
7475 "/claim/start/:token": "needs a one-time start claim token, which the seed does not mint (test/start.test.ts covers it)",
7576 "/signin/access/:token": "needs a one-time access link an admin minted",
7677 "/talk/:id": TALK,

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs