Public: anyone can read this; the crew can change it.
A pull's check can run on a runner: a separate process that needs only the forge's URL and a token #13823
Check · Review · Land · Live done — Landed and live in production.
trevato/checks-runner-placement → main 2 commits 1 thread · 1 unresolved
32 files
changed +2428
−18
AGENTS.md +2
| @@ -22,6 +22,8 @@ State shared between modules through process memory, not the database, is named | ||
| 22 | 22 | |
| 23 | 23 | The check queue hands work to a runner through `src/checks/runner.ts` (#13652, R0); its module header says how. Keep database handles, paths and callbacks out of `CheckJob`. |
| 24 | 24 | |
| 25 | `src/checks/runner_jobs.ts`: `deskOf(app)`, per forge, the runners waiting in a claim (`claimers`) and the checks waiting on a job a runner may hold (`ears`, by job id), with `FORGE_CHECKS_PLACEMENT=runner` (#13652, R1; `docs/features/runners.md`). At most one entry per open claim request and per check in flight; gone on restart, and nothing is lost with it: what is claimed is a row (`check_job`), and a check whose forge restarted is queued again as any interrupted check is. `RunnerJob` is what crosses the wire: keep the bare repository's path (`check_job.dir`) out of it. | |
| 26 | ||
| 25 | 27 | `src/perf/loop_worst.ts`: `loopWorst`, the worst event-loop lag of the last 10 s in ten one-second slots, fed by one unref'd 50 ms timer that the first `serve` starts (`startLoopWorst`), and read by every response's `lag;dur` and by `/healthz` asked for JSON (`loopLagMs`). Tests listing a request's phases read `requestTiming` (`test/helpers/request-timing.ts`), which leaves `lag` out. |
| 26 | 28 | |
| 27 | 29 | `src/perf/stall_blame.ts` (`stallBlame` in `src/perf/live.ts`): each loop stall over 200 ms as `{ what, ms, at }` in a fixed ring of 256, fed by `main.ts`'s stall watcher after `loopWatch` names it, read by /system's worst-five card and `/healthz`'s `loopWorstWhat`. |
docs/APPS.md +5 −2
| @@ -918,7 +918,10 @@ egress proxy for the run on `FORGE_CHECKS_EGRESS_GATE` and points the pod's | ||
| 918 | 918 | `HTTPS_PROXY` at it, so an undeclared host is refused in the forge's own |
| 919 | 919 | words, and the run is recorded as enforced; the pod carries the label |
| 920 | 920 | `forge-check=true` for the NetworkPolicy that leaves it no other way out |
| 921 | (docs/CLUSTER.md). Without a gate no check runs as a Job. | |
| 921 | (docs/CLUSTER.md). Without a gate no check runs as a Job. `runner` offers a | |
| 922 | pull's check to a runner that joined, a separate process with the forge's URL | |
| 923 | and a token, and runs it here when none takes it | |
| 924 | ([](features/runners.md)). | |
| 922 | 925 | |
| 923 | 926 | Checks share `FORGE_CHECKS_CONCURRENCY` slots (default `1`), and when one |
| 924 | 927 | frees, it goes to the most recently requested check waiting, not the one |
| @@ -1816,7 +1819,7 @@ a link to the milestone (`docs/PLANNING.md`); its progress never does. | ||
| 1816 | 1819 | | `FORGE_PREVIEW_IDLE_DAYS` | the profit slider | days a pull may sit untouched before its preview sleeps; `0` = never; unset follows the slider (3 at 50) | |
| 1817 | 1820 | | `FORGE_CREW_CONCURRENCY` | 1 per 3 GiB above 2 GiB, 2–16 | agent runs at once (live: Settings → Capacity) | |
| 1818 | 1821 | | `FORGE_CHECKS_CONCURRENCY` | `1` | checks at once (live: Settings → Capacity) | |
| 1819 | | `FORGE_CHECKS_PLACEMENT` | `process` | `job` runs a check's commands as a cluster Job (see Placement above) | | |
| 1822 | | `FORGE_CHECKS_PLACEMENT` | `process` | `job` runs a check's commands as a cluster Job, `runner` offers a pull's check to a runner that joined (see Placement above) | | |
| 1820 | 1823 | | `FORGE_CHECKS_EGRESS_GATE` | unset | the address a Job's egress gate listens on; without it no check runs as a Job | |
| 1821 | 1824 | | `FORGE_CHECKS_DRAIN_S` | `10` | seconds a roll lets running checks finish first | |
| 1822 | 1825 | | `FORGE_CREW_DRAIN_S` | `10` | seconds a roll lets crew runs finish before they are queued again | |
docs/features/README.md +1
| @@ -210,6 +210,7 @@ Read with the guide page [Ship and undo](../guide-ship-and-undo.md). | ||
| 210 | 210 | - [](pin-screenshots.md) |
| 211 | 211 | - [](stale-pulls.md) |
| 212 | 212 | - [](ranked-checks.md) |
| 213 | - [](runners.md) | |
| 213 | 214 | - [](stuck-pile.md) |
| 214 | 215 | - [](release-heartbeat.md) |
| 215 | 216 | - [](release-keeper.md) |
docs/features/runners.md added +110
| @@ -0,0 +1,110 @@ | ||
| 1 | # Runners | |
| 2 | ||
| 3 | A runner is a separate process that takes checks off the forge. It can be on | |
| 4 | the forge's machine, on another machine or in a rented one. It needs two | |
| 5 | things: the forge's URL and a token. Nothing dials in to it, it shares no disk | |
| 6 | with the forge, and it never sees the forge's data directory or database. | |
| 7 | ||
| 8 | Off by default. With `FORGE_CHECKS_PLACEMENT=runner` the forge offers a check's | |
| 9 | steps to runners, and **never needs one**: with no runner connected the check | |
| 10 | runs on the forge's machine at once, as `process` does; when runners are | |
| 11 | connected and none takes the job in 5 seconds it runs on the forge's machine | |
| 12 | too. | |
| 13 | ||
| 14 | ## Start one, in two lines | |
| 15 | ||
| 16 | An admin enrols it (admin scope; the token is shown this once): | |
| 17 | ||
| 18 | sh | |
| 19 | curl -X POST -H "Authorization: Bearer " -H 'content-type: application/json' -d '{"name":"mac"}' /api/v1/admin/runners | |
| 20 | ||
| 21 | ||
| 22 | On the machine that will run checks, from a checkout of the forge or its image: | |
| 23 | ||
| 24 | sh | |
| 25 | FORGE_URL=https://your.forge FORGE_RUNNER_TOKEN=forge_runner_… bun src/runner/main.ts | |
| 26 | ||
| 27 | ||
| 28 | `GET /api/v1/admin/runners` lists the enrolled runners and when each last | |
| 29 | asked for work; `DELETE /api/v1/admin/runners/<id>` removes one: its token | |
| 30 | stops at once and a job it held is queued again. A check's page and the | |
| 31 | terminal's run list say "Ran on runner mac" for a check one took. | |
| 32 | ||
| 33 | The machine needs what the checks need: `git` and `bun`, and whatever the | |
| 34 | repository's check calls. The forge's image has all of it, and `bubblewrap` | |
| 35 | and `socat` for cells. | |
| 36 | ||
| 37 | `FORGE_RUNNER_DIR` is where it keeps its scratch and its store (default: a | |
| 38 | `forge-runner` directory under the temp directory), `FORGE_RUNNER_JOBS` how | |
| 39 | many jobs it runs at once (default 1). | |
| 40 | ||
| 41 | ## What a job is | |
| 42 | ||
| 43 | Plain data, made by the forge from the repository's `forge.json` as it would | |
| 44 | run the check itself: | |
| 45 | ||
| 46 | | Field | What | | |
| 47 | | ---------------------- | ----------------------------------------------------------------------------- | | |
| 48 | | `commit` | the exact commit under test: for a pull, its merge into the base as it stands | | |
| 49 | | `base`, `depth` | the commit a fast check diffs against, and how deep to fetch | | |
| 50 | | `steps` | the recipe in order: `install`, then `check`, each a command | | |
| 51 | | `env` | the check's own variables (`CI`, `FORGE_*`), never the forge's | | |
| 52 | | `limits`, `timeoutS` | cores, memory and disk as declared; seconds a step may take | | |
| 53 | | `egress`, `egressMode` | the hosts the steps may reach besides the forge | | |
| 54 | | `cell` | `required` when the code is an outside contributor's: no cell, no run | | |
| 55 | | `cache` | names a runner may keep things under: the repository, the tree, the base | | |
| 56 | ||
| 57 | With the job comes a **lease** of 30 seconds, renewed by every report, and a | |
| 58 | **job token** that works for this one job while the lease lives. | |
| 59 | ||
| 60 | ## What a runner does | |
| 61 | ||
| 62 | 1. Claims: `POST /api/v1/runner/claim` with its runner token, held open until a | |
| 63 | job is offered. Of two runners asking at once, one gets a job. | |
| 64 | 2. Fetches the job's commit from the job's own git door | |
| 65 | (`/api/v1/runner/jobs/<id>/git`) with the job token, into a bare mirror it | |
| 66 | keeps per repository, so the next job fetches only new objects. | |
| 67 | 3. Runs the steps in the cell the forge uses (bubblewrap on Linux): the | |
| 68 | checkout and a home are the only places to write, the forge and the declared | |
| 69 | hosts the only way out. Where the machine has no cells it says so, runs the | |
| 70 | steps as plain processes, and hands back any job that requires a cell. | |
| 71 | 4. Streams the log every second (`POST /api/v1/runner/jobs/<id>`), reports each | |
| 72 | step's result, and removes its scratch. | |
| 73 | ||
| 74 | A runner that goes silent loses its lease: the job is queued again, and the | |
| 75 | check's try count does not move. Twice, and the forge runs it itself. | |
| 76 | ||
| 77 | ## What a runner decides, and what it does not | |
| 78 | ||
| 79 | The forge keeps the queue, the claim on the check, the budget and the record. | |
| 80 | A runner's answer stands only as: green, the check command's exit code, or its | |
| 81 | timeout. The forge runs the check on its own machine when the install failed | |
| 82 | on the runner, when the step ended any other way (a signal, a limit, a full | |
| 83 | disk there), and when one or two tests failed: those are re-run alone to tell | |
| 84 | a flake from a red, which a runner cannot do yet. | |
| 85 | ||
| 86 | Only a **pull's check** goes to a runner. The default branch's own check and a | |
| 87 | verification after landing keep what they measured (the build cache, the | |
| 88 | impact map, kept results) from the checkout, and a runner cannot send those | |
| 89 | back yet, so they stay on the forge's machine. For the same reason a runner's | |
| 90 | check starts without them: correct, and slower than the forge's own. | |
| 91 | ||
| 92 | ## What a runner can and cannot see | |
| 93 | ||
| 94 | - **Can:** the commit of each job it claims and that commit's history; the | |
| 95 | names and tips of that repository's branches (the advertisement); the job's | |
| 96 | recipe and variables. It can append to that check's log and end that job. | |
| 97 | - **Cannot:** fetch any other object of that repository, or anything of | |
| 98 | another repository; push; read an issue, a page, the API or the MCP. Its two | |
| 99 | tokens are not account tokens: every other door treats them as no credential | |
| 100 | at all. A job token dies with its lease. | |
| 101 | - A runner token can claim any job on the desk, so enrol machines you trust | |
| 102 | with every repository whose checks run here. Without cells, a check runs as | |
| 103 | the runner's own user on that machine. | |
| 104 | ||
| 105 | A runner's requests are counted like a stranger's (600 a minute from one | |
| 106 | address); one job is about 60 a minute. | |
| 107 | ||
| 108 | Code: `src/checks/runner_jobs.ts` (the desk), `src/checks/runner_step.ts` (the | |
| 109 | forge's side), `src/runner/main.ts` (the runner), `src/http/api/runners.ts`. | |
| 110 | Tests: `test/checks-runner-placement.test.ts`. Epic: #13652. |
scripts/unreached.ts +13 −4
| @@ -5,8 +5,8 @@ | ||
| 5 | 5 | * what main carries today; `test/unreached.test.ts` keeps it shrinking. |
| 6 | 6 | * |
| 7 | 7 | * An import is a static `from "./x"`, a side-effect `import "./x"`, or an |
| 8 | * `import("./x")` (quotes or backticks, any whitespace), resolved against the importing file's directory. | |
| 9 | * Imports from `scripts/` count as reaching what they name. | |
| 8 | * `import("./x")` (quotes or backticks, any whitespace), resolved against the importing file's | |
| 9 | * directory. Imports from `scripts/` count as reaching what they name. | |
| 10 | 10 | */ |
| 11 | 11 | |
| 12 | 12 | import { readFileSync } from "node:fs"; |
| @@ -21,13 +21,19 @@ export const ENTRIES = [ | ||
| 21 | 21 | "src/chaos/run.ts", |
| 22 | 22 | "src/release/boot.ts", |
| 23 | 23 | "src/git/hook-entry.ts", |
| 24 | // A runner's own process, started on its machine: `bun src/runner/main.ts` | |
| 25 | // (docs/features/runners.md). | |
| 26 | "src/runner/main.ts", | |
| 24 | 27 | // Run by hand: `bun run src/tui/gallery.ts` prints every widget. |
| 25 | 28 | "src/tui/gallery.ts", |
| 26 | 29 | // Bundled for the browser by Bun.build in src/web/asset.ts, named there by a template path. |
| 27 | 30 | "src/web/highlight-client.ts", |
| 28 | 31 | ]; |
| 29 | 32 | |
| 30 | /** Scripts the check and operators run, and the benches run at release, are callers too: their imports count. */ | |
| 33 | /** | |
| 34 | * Scripts the check and operators run, and the benches run at release, are callers too: their | |
| 35 | * imports count. | |
| 36 | */ | |
| 31 | 37 | const SCRIPTS = ["scripts/**/*.ts", "bench/**/*.ts"]; |
| 32 | 38 | |
| 33 | 39 | /** Not modules: tests, and migrations, which the runner finds by listing the directory. */ |
| @@ -35,7 +41,10 @@ const NOT_MODULES = /(?:\.test\.ts$|^src\/core\/migrations\/)/; | ||
| 35 | 41 | |
| 36 | 42 | const SPEC = /(?:\bfrom\s*|\bimport\s*\(?\s*)["'`](\.{1,2}\/[^"'`]*)["'`]/g; |
| 37 | 43 | |
| 38 | /** A module named as a URL beside this one, the way a `Bun.build` entry point is: `new URL("./x.ts", import.meta.url)`. */ | |
| 44 | /** | |
| 45 | * A module named as a URL beside this one, the way a `Bun.build` entry point is: | |
| 46 | * `new URL("./x.ts", import.meta.url)`. | |
| 47 | */ | |
| 39 | 48 | const URL_SPEC = /\bnew\s+URL\(\s*["'`](\.{1,2}\/[^"'`]*)["'`]\s*,\s*import\.meta\.url\s*\)/g; |
| 40 | 49 | |
| 41 | 50 | /** The src/ file a relative specifier names, among `files`, or undefined. */ |
src/actions/runners.ts added +59
| @@ -0,0 +1,59 @@ | ||
| 1 | /** | |
| 2 | * An admin enrols, lists and removes runners (#13652): machines that take a check's steps off | |
| 3 | * the forge (`checks/runner_jobs.ts`). Each enrolment and removal is an event, which is the | |
| 4 | * audit; the token is in none of them. Only a person who is a forge admin may: anyone else is | |
| 5 | * told it is not there. | |
| 6 | */ | |
| 7 | ||
| 8 | import type { App } from "../app.ts"; | |
| 9 | import { | |
| 10 | enrolRunner, | |
| 11 | listRunners, | |
| 12 | removeRunner, | |
| 13 | wakeClaimers, | |
| 14 | type Enrolled, | |
| 15 | } from "../checks/runner_jobs.ts"; | |
| 16 | import type { User } from "../core/users.ts"; | |
| 17 | import { err, ok, type Result } from "../kernel/result.ts"; | |
| 18 | ||
| 19 | const admin = (actor: User | undefined): actor is User => | |
| 20 | actor !== undefined && actor.isAdmin && actor.kind === "user"; | |
| 21 | ||
| 22 | export function enrol( | |
| 23 | app: App, | |
| 24 | actor: User | undefined, | |
| 25 | name: string, | |
| 26 | ): Result<{ runner: Enrolled; token: string }, "not_found"> { | |
| 27 | if (!admin(actor)) return err("not_found"); | |
| 28 | const made = enrolRunner(app.db, name, actor.id); | |
| 29 | app.bus.append({ | |
| 30 | kind: "runner.enrolled", | |
| 31 | actorId: actor.id, | |
| 32 | payload: { id: made.runner.id, name: made.runner.name }, | |
| 33 | }); | |
| 34 | return ok(made); | |
| 35 | } | |
| 36 | ||
| 37 | export function list( | |
| 38 | app: App, | |
| 39 | actor: User | undefined, | |
| 40 | ): Result<Enrolled[], "not_found"> { | |
| 41 | return admin(actor) ? ok(listRunners(app.db)) : err("not_found"); | |
| 42 | } | |
| 43 | ||
| 44 | export function remove( | |
| 45 | app: App, | |
| 46 | actor: User | undefined, | |
| 47 | runnerId: string, | |
| 48 | ): Result<void, "not_found"> { | |
| 49 | if (!admin(actor)) return err("not_found"); | |
| 50 | if (!removeRunner(app.db, runnerId)) return err("not_found"); | |
| 51 | // A claim it holds open ends now, refused; a check waiting on its job looks again. | |
| 52 | wakeClaimers(app); | |
| 53 | app.bus.append({ | |
| 54 | kind: "runner.removed", | |
| 55 | actorId: actor.id, | |
| 56 | payload: { id: runnerId }, | |
| 57 | }); | |
| 58 | return ok(); | |
| 59 | } |
src/checks/history.ts +1 −1
| @@ -15,7 +15,7 @@ import { isSafeOid } from "../git/guard.ts"; | ||
| 15 | 15 | export const MAX_PULL_COMMITS = 500; |
| 16 | 16 | |
| 17 | 17 | /** How deep to fetch the head so every commit in `base..head` arrives (at least 1). */ |
| 18 | async function headDepth(bare: string, base: string, head: string): Promise<number> { | |
| 18 | export async function headDepth(bare: string, base: string, head: string): Promise<number> { | |
| 19 | 19 | const counted = await runGit(["rev-list", "--count", `${base}..${head}`], { cwd: bare, timeoutMs: 30_000 }); |
| 20 | 20 | const n = counted.ok ? Number.parseInt(counted.value.trim(), 10) : NaN; |
| 21 | 21 | // Unknown (a base the bare lacks) keeps the old one-deep fetch: the diff still answers. |
src/checks/placement.ts +14 −5
| @@ -1,9 +1,11 @@ | ||
| 1 | 1 | /** |
| 2 | 2 | * Where a check's commands run: `process` (the default, a cell on the |
| 3 | * forge's own machine) or `job` (a Kubernetes Job, through | |
| 4 | * `cluster/jobs.ts`). Read from the forge's environment once, validated at | |
| 5 | * this boundary; anything malformed leaves the default in place. | |
| 6 | * FORGE_CHECKS_PLACEMENT process | job | |
| 3 | * forge's own machine), `job` (a Kubernetes Job, through | |
| 4 | * `cluster/jobs.ts`) or `runner` (a runner that joined, `runner_step.ts`; | |
| 5 | * with none connected the check runs as `process`). Read from the forge's | |
| 6 | * environment once, validated at this boundary; anything malformed leaves | |
| 7 | * the default in place. | |
| 8 | * FORGE_CHECKS_PLACEMENT process | job | runner | |
| 7 | 9 | * FORGE_CHECKS_IMAGE the Job's image |
| 8 | 10 | * FORGE_CHECKS_CLAIM a ReadWriteMany claim holding the checks root |
| 9 | 11 | * FORGE_CHECKS_NODE_SELECTOR key=value,key=value |
| @@ -24,6 +26,11 @@ export type Placement = | ||
| 24 | 26 | readonly tolerations: Toleration[]; |
| 25 | 27 | /** The address the forge's per-run egress proxy listens on and the pod dials. */ |
| 26 | 28 | readonly gate?: string; |
| 29 | } | |
| 30 | | { | |
| 31 | readonly mode: "runner"; | |
| 32 | /** How long a job waits for a runner to take it; tests shorten it. @default RUNNER_WAIT_MS */ | |
| 33 | readonly waitMs?: number; | |
| 27 | 34 | }; |
| 28 | 35 | |
| 29 | 36 | /** An IPv4 address or a DNS name: what a pod can dial and the forge can bind. */ |
| @@ -75,6 +82,7 @@ export function parseTolerations(text: string): Toleration[] | undefined { | ||
| 75 | 82 | } |
| 76 | 83 | |
| 77 | 84 | export function placementFromEnv(env: Record<string, string | undefined> = process.env): Placement { |
| 85 | if (env["FORGE_CHECKS_PLACEMENT"]?.trim() === "runner") return { mode: "runner" }; | |
| 78 | 86 | if (env["FORGE_CHECKS_PLACEMENT"]?.trim() !== "job") return { mode: "process" }; |
| 79 | 87 | const image = env["FORGE_CHECKS_IMAGE"]?.trim() ?? ""; |
| 80 | 88 | const claim = env["FORGE_CHECKS_CLAIM"]?.trim() ?? ""; |
| @@ -94,8 +102,9 @@ export function placementFromEnv(env: Record<string, string | undefined> = proce | ||
| 94 | 102 | * here as well as escaped by whatever shows it. |
| 95 | 103 | */ |
| 96 | 104 | export function placementWords(run: { placement: string; node: string }): string { |
| 97 | if (run.placement !== "job") return ""; | |
| 105 | if (run.placement !== "job" && run.placement !== "runner") return ""; | |
| 98 | 106 | const node = run.node.replace(/[^\x21-\x7e]/g, "").slice(0, 253); |
| 107 | if (run.placement === "runner") return node === "" ? "ran on a runner" : `ran on runner ${node}`; | |
| 99 | 108 | return node === "" ? "ran as a cluster job" : `ran as a cluster job on ${node}`; |
| 100 | 109 | } |
| 101 | 110 |
src/checks/runner_jobs.ts added +471
| @@ -0,0 +1,471 @@ | ||
| 1 | /** | |
| 2 | * The desk between a check and the runners that may take its steps (#13652, R1). | |
| 3 | * | |
| 4 | * The check queue is still the scheduler: it picks a run, claims it and keeps its lease, as | |
| 5 | * before. With `FORGE_CHECKS_PLACEMENT=runner` the steps that run would spawn are instead put on | |
| 6 | * this desk as a job, and a runner (`src/runner/main.ts`, a separate process that holds nothing | |
| 7 | * of the forge's but its URL and a token) may claim it. A job is plain data (`RunnerJob`): it | |
| 8 | * survives `JSON.stringify`, and names no path, handle or callback of this forge. | |
| 9 | * | |
| 10 | * Two credentials, neither a row of `token`, so neither is anyone's account anywhere else: | |
| 11 | * a runner token (`forge_runner_…`, made by an admin, kept as a hash in `check_runner`) claims | |
| 12 | * jobs and nothing more; a job token (`forge_job_…`, made by each claim, kept as a hash on the | |
| 13 | * job's row) fetches that job's repository and reports on that job, while its lease lives. | |
| 14 | * | |
| 15 | * A claim is the UPDATE's WHERE on `state`, as `claimRun` is. A lease that lapses puts the job | |
| 16 | * back as queued: the run it serves is untouched, so nothing counts as a try. Rows are deleted | |
| 17 | * when the check that offered them ends; the table holds only what is in flight. | |
| 18 | */ | |
| 19 | ||
| 20 | import type { Database } from "bun:sqlite"; | |
| 21 | import type { App } from "../app.ts"; | |
| 22 | import { sha256 } from "../kernel/hash.ts"; | |
| 23 | import { id, secret } from "../kernel/id.ts"; | |
| 24 | import { during } from "../perf/jobs.ts"; | |
| 25 | ||
| 26 | export const RUNNER_TOKEN_PREFIX = "forge_runner_"; | |
| 27 | export const JOB_TOKEN_PREFIX = "forge_job_"; | |
| 28 | ||
| 29 | /** How long a claim holds without a word from its runner. */ | |
| 30 | export const LEASE_MS = 30_000; | |
| 31 | /** A runner that asked for work this recently is connected. */ | |
| 32 | export const SEEN_FRESH_MS = 60_000; | |
| 33 | ||
| 34 | export interface RunnerStepSpec { | |
| 35 | readonly name: "install" | "check"; | |
| 36 | readonly command: string; | |
| 37 | } | |
| 38 | ||
| 39 | /** A check's steps as a runner is sent them. */ | |
| 40 | export interface RunnerJob { | |
| 41 | readonly id: string; | |
| 42 | /** The check run it serves, as the log names it. */ | |
| 43 | readonly run: string; | |
| 44 | /** `owner/name`, for the runner's own log; the job's git door needs no name. */ | |
| 45 | readonly repo: string; | |
| 46 | /** The exact commit under test: for a pull, its merge into the base as it stands. */ | |
| 47 | readonly commit: string; | |
| 48 | /** The commit the fast tier diffs against (`FORGE_CHECK_BASE`), fetched one deep. */ | |
| 49 | readonly base?: string; | |
| 50 | /** How deep `commit` is fetched so every commit of `base..commit` arrives. */ | |
| 51 | readonly depth: number; | |
| 52 | /** The recipe, in order: what the service would have run, from `forge.json`. */ | |
| 53 | readonly steps: readonly RunnerStepSpec[]; | |
| 54 | /** The check's own variables (`CI`, `FORGE_*`); never the forge's environment. */ | |
| 55 | readonly env: Readonly<Record<string, string>>; | |
| 56 | /** What `forge.json` declares; the runner holds them to its own machine. */ | |
| 57 | readonly limits: { | |
| 58 | readonly cpus?: number; | |
| 59 | readonly memoryMiB?: number; | |
| 60 | readonly diskMiB?: number; | |
| 61 | }; | |
| 62 | /** Seconds one step may take. */ | |
| 63 | readonly timeoutS: number; | |
| 64 | /** Hosts the steps may reach besides the forge. */ | |
| 65 | readonly egress: readonly string[]; | |
| 66 | readonly egressMode: "enforce" | "report"; | |
| 67 | /** `required`: a runner without cells must hand it back. */ | |
| 68 | readonly cell: "required" | "preferred"; | |
| 69 | /** | |
| 70 | * Names a runner may keep things under between jobs: its mirror and package cache by `repo`, | |
| 71 | * a tree it has seen by `tree`, what the base's check left by `base` ("" when there is none). | |
| 72 | */ | |
| 73 | readonly cache: { | |
| 74 | readonly repo: string; | |
| 75 | readonly tree: string; | |
| 76 | readonly base: string; | |
| 77 | }; | |
| 78 | } | |
| 79 | ||
| 80 | /** What a runner reports of one step. */ | |
| 81 | export interface RunnerStepResult { | |
| 82 | readonly name: string; | |
| 83 | readonly ok: boolean; | |
| 84 | /** `runStep`'s own words when it did not end ok. */ | |
| 85 | readonly error?: string; | |
| 86 | readonly ms: number; | |
| 87 | } | |
| 88 | ||
| 89 | export interface RunnerResult { | |
| 90 | readonly steps: readonly RunnerStepResult[]; | |
| 91 | /** Whether the steps ran in a cell, and why not when they did not. */ | |
| 92 | readonly contained: boolean; | |
| 93 | readonly note: string; | |
| 94 | } | |
| 95 | ||
| 96 | export interface Enrolled { | |
| 97 | readonly id: string; | |
| 98 | readonly name: string; | |
| 99 | readonly createdAt: string; | |
| 100 | /** When it last asked for work, in milliseconds; 0 for never. */ | |
| 101 | readonly seenAt: number; | |
| 102 | } | |
| 103 | ||
| 104 | type RunnerRow = { | |
| 105 | id: string; | |
| 106 | name: string; | |
| 107 | created_at: string; | |
| 108 | seen_at: number; | |
| 109 | }; | |
| 110 | const toEnrolled = (r: RunnerRow): Enrolled => ({ | |
| 111 | id: r.id, | |
| 112 | name: r.name, | |
| 113 | createdAt: r.created_at, | |
| 114 | seenAt: r.seen_at, | |
| 115 | }); | |
| 116 | ||
| 117 | /** A name as it is kept: printable, at most 40 characters, "runner" when nothing is left. */ | |
| 118 | export const runnerName = (raw: string): string => | |
| 119 | raw.replace(/[^\x21-\x7e]/g, "").slice(0, 40) || "runner"; | |
| 120 | ||
| 121 | /** Enrol a runner. The token is returned once; the row keeps its hash. */ | |
| 122 | export function enrolRunner( | |
| 123 | db: Database, | |
| 124 | name: string, | |
| 125 | createdBy: string, | |
| 126 | ): { runner: Enrolled; token: string } { | |
| 127 | const runnerId = id(); | |
| 128 | const token = `${RUNNER_TOKEN_PREFIX}${secret()}`; | |
| 129 | db.query( | |
| 130 | "INSERT INTO check_runner (id, name, token_hash, created_by) VALUES (?1, ?2, ?3, ?4)", | |
| 131 | ).run(runnerId, runnerName(name), sha256(token), createdBy); | |
| 132 | return { runner: getRunner(db, runnerId)!, token }; | |
| 133 | } | |
| 134 | ||
| 135 | export function getRunner( | |
| 136 | db: Database, | |
| 137 | runnerId: string, | |
| 138 | ): Enrolled | undefined { | |
| 139 | const row = db | |
| 140 | .query< | |
| 141 | RunnerRow, | |
| 142 | [string] | |
| 143 | >("SELECT id, name, created_at, seen_at FROM check_runner WHERE id = ?1") | |
| 144 | .get(runnerId); | |
| 145 | return row === null ? undefined : toEnrolled(row); | |
| 146 | } | |
| 147 | ||
| 148 | export function listRunners(db: Database): Enrolled[] { | |
| 149 | return db | |
| 150 | .query< | |
| 151 | RunnerRow, | |
| 152 | [] | |
| 153 | >("SELECT id, name, created_at, seen_at FROM check_runner ORDER BY id LIMIT 200") | |
| 154 | .all() | |
| 155 | .map(toEnrolled); | |
| 156 | } | |
| 157 | ||
| 158 | /** | |
| 159 | * Remove a runner: its token stops at once, and what it held is queued again as a lapsed lease | |
| 160 | * is. | |
| 161 | */ | |
| 162 | export function removeRunner(db: Database, runnerId: string): boolean { | |
| 163 | return db.transaction(() => { | |
| 164 | const gone = | |
| 165 | db.query("DELETE FROM check_runner WHERE id = ?1").run(runnerId).changes > | |
| 166 | 0; | |
| 167 | if (gone) | |
| 168 | db.query( | |
| 169 | "UPDATE check_job SET state = 'queued', runner_id = NULL, token_hash = NULL, lease_until = 0 WHERE runner_id = ?1 AND state = 'claimed'", | |
| 170 | ).run(runnerId); | |
| 171 | return gone; | |
| 172 | })(); | |
| 173 | } | |
| 174 | ||
| 175 | /** The runner a presented token is, noting that it asked; undefined for anything else. */ | |
| 176 | export function runnerByToken( | |
| 177 | db: Database, | |
| 178 | presented: string, | |
| 179 | now: number, | |
| 180 | ): Enrolled | undefined { | |
| 181 | if (!presented.startsWith(RUNNER_TOKEN_PREFIX)) return undefined; | |
| 182 | const row = db | |
| 183 | .query< | |
| 184 | RunnerRow, | |
| 185 | [string] | |
| 186 | >("SELECT id, name, created_at, seen_at FROM check_runner WHERE token_hash = ?1") | |
| 187 | .get(sha256(presented)); | |
| 188 | if (row === null) return undefined; | |
| 189 | db.query("UPDATE check_runner SET seen_at = ?2 WHERE id = ?1").run( | |
| 190 | row.id, | |
| 191 | now, | |
| 192 | ); | |
| 193 | return toEnrolled({ ...row, seen_at: now }); | |
| 194 | } | |
| 195 | ||
| 196 | /** Whether any runner asked for work within `SEEN_FRESH_MS`. */ | |
| 197 | export function runnersAround(db: Database, now: number): boolean { | |
| 198 | return ( | |
| 199 | db | |
| 200 | .query< | |
| 201 | { n: number }, | |
| 202 | [number] | |
| 203 | >("SELECT 1 AS n FROM check_runner WHERE seen_at > ?1 LIMIT 1") | |
| 204 | .get(now - SEEN_FRESH_MS) !== null | |
| 205 | ); | |
| 206 | } | |
| 207 | ||
| 208 | /** | |
| 209 | * Put a check's steps on the desk. `dir` is the bare repository the commit is in; it stays here. | |
| 210 | */ | |
| 211 | export function offerJob( | |
| 212 | db: Database, | |
| 213 | input: { | |
| 214 | runId: string; | |
| 215 | repoId: string; | |
| 216 | dir: string; | |
| 217 | job: Omit<RunnerJob, "id">; | |
| 218 | }, | |
| 219 | ): RunnerJob { | |
| 220 | const job: RunnerJob = { ...input.job, id: id() }; | |
| 221 | db.query( | |
| 222 | "INSERT INTO check_job (id, run_id, repo_id, dir, spec, needs_cell) VALUES (?1, ?2, ?3, ?4, ?5, ?6)", | |
| 223 | ).run( | |
| 224 | job.id, | |
| 225 | input.runId, | |
| 226 | input.repoId, | |
| 227 | input.dir, | |
| 228 | JSON.stringify(job), | |
| 229 | job.cell === "required" ? 1 : 0, | |
| 230 | ); | |
| 231 | return job; | |
| 232 | } | |
| 233 | ||
| 234 | export interface Claim { | |
| 235 | readonly job: RunnerJob; | |
| 236 | /** The job's own credential: shown once, here. */ | |
| 237 | readonly token: string; | |
| 238 | readonly leaseMs: number; | |
| 239 | } | |
| 240 | ||
| 241 | /** | |
| 242 | * Take the oldest queued job this runner can run. The UPDATE's WHERE is the claim: of two | |
| 243 | * runners asking at once, one changes the row and the other moves on to the next. | |
| 244 | */ | |
| 245 | export function claimJob( | |
| 246 | db: Database, | |
| 247 | runner: { id: string; cells: boolean }, | |
| 248 | now: number, | |
| 249 | leaseMs: number = LEASE_MS, | |
| 250 | ): Claim | undefined { | |
| 251 | const queued = db | |
| 252 | .query< | |
| 253 | { id: string; spec: string }, | |
| 254 | [number] | |
| 255 | >("SELECT id, spec FROM check_job WHERE state = 'queued' AND (needs_cell = 0 OR ?1 = 1) ORDER BY id LIMIT 8") | |
| 256 | .all(runner.cells ? 1 : 0); | |
| 257 | for (const row of queued) { | |
| 258 | const token = `${JOB_TOKEN_PREFIX}${secret()}`; | |
| 259 | const taken = | |
| 260 | db | |
| 261 | .query( | |
| 262 | `UPDATE check_job SET state = 'claimed', runner_id = ?2, token_hash = ?3, lease_until = ?4, | |
| 263 | claims = claims + 1 WHERE id = ?1 AND state = 'queued'`, | |
| 264 | ) | |
| 265 | .run(row.id, runner.id, sha256(token), now + leaseMs).changes > 0; | |
| 266 | if (taken) | |
| 267 | return { job: JSON.parse(row.spec) as RunnerJob, token, leaseMs }; | |
| 268 | } | |
| 269 | return undefined; | |
| 270 | } | |
| 271 | ||
| 272 | export interface HeldJob { | |
| 273 | readonly id: string; | |
| 274 | readonly runId: string; | |
| 275 | readonly repoId: string; | |
| 276 | /** The bare repository the job's commit is in. */ | |
| 277 | readonly dir: string; | |
| 278 | /** The only commits its git door serves: the job's commit, and its base when it has one. */ | |
| 279 | readonly wants: readonly string[]; | |
| 280 | } | |
| 281 | ||
| 282 | /** | |
| 283 | * The job a presented token holds: this job, claimed, its lease alive. Anything else (another | |
| 284 | * job's token, a lapsed or withdrawn claim, a runner's or an account's token) is undefined. | |
| 285 | * Its runner is noted as seen: one at work is connected, though it is not asking for more. | |
| 286 | */ | |
| 287 | export function jobByToken(db: Database, jobId: string, presented: string, now: number): HeldJob | undefined { | |
| 288 | if (!presented.startsWith(JOB_TOKEN_PREFIX)) return undefined; | |
| 289 | const row = db | |
| 290 | .query<{ id: string; run_id: string; repo_id: string; dir: string; spec: string }, [string, string, number]>( | |
| 291 | `SELECT id, run_id, repo_id, dir, spec FROM check_job | |
| 292 | WHERE id = ?1 AND token_hash = ?2 AND state = 'claimed' AND lease_until > ?3`, | |
| 293 | ) | |
| 294 | .get(jobId, sha256(presented), now); | |
| 295 | if (row === null) return undefined; | |
| 296 | db.query("UPDATE check_runner SET seen_at = ?2 WHERE id = (SELECT runner_id FROM check_job WHERE id = ?1)").run( | |
| 297 | row.id, | |
| 298 | now, | |
| 299 | ); | |
| 300 | const job = JSON.parse(row.spec) as RunnerJob; | |
| 301 | return { | |
| 302 | id: row.id, | |
| 303 | runId: row.run_id, | |
| 304 | repoId: row.repo_id, | |
| 305 | dir: row.dir, | |
| 306 | wants: [job.commit, ...(job.base === undefined ? [] : [job.base])].map((oid) => oid.toLowerCase()), | |
| 307 | }; | |
| 308 | } | |
| 309 | ||
| 310 | /** Push a held job's lease out; false when the claim is no longer this one's to renew. */ | |
| 311 | export function renewJob( | |
| 312 | db: Database, | |
| 313 | jobId: string, | |
| 314 | now: number, | |
| 315 | leaseMs: number = LEASE_MS, | |
| 316 | ): boolean { | |
| 317 | return ( | |
| 318 | db | |
| 319 | .query( | |
| 320 | "UPDATE check_job SET lease_until = ?2 WHERE id = ?1 AND state = 'claimed' AND lease_until > ?3", | |
| 321 | ) | |
| 322 | .run(jobId, now + leaseMs, now).changes > 0 | |
| 323 | ); | |
| 324 | } | |
| 325 | ||
| 326 | /** Record a held job's answer: `done` with its result, or `returned` with the runner's reason. */ | |
| 327 | export function settleJob( | |
| 328 | db: Database, | |
| 329 | jobId: string, | |
| 330 | end: { done: RunnerResult } | { returned: string }, | |
| 331 | ): boolean { | |
| 332 | const done = "done" in end; | |
| 333 | return ( | |
| 334 | db | |
| 335 | .query( | |
| 336 | "UPDATE check_job SET state = ?2, result = ?3, token_hash = NULL WHERE id = ?1 AND state = 'claimed'", | |
| 337 | ) | |
| 338 | .run( | |
| 339 | jobId, | |
| 340 | done ? "done" : "returned", | |
| 341 | done ? JSON.stringify(end.done) : end.returned.slice(0, 400), | |
| 342 | ).changes > 0 | |
| 343 | ); | |
| 344 | } | |
| 345 | ||
| 346 | /** | |
| 347 | * Every claim whose lease ran out goes back to queued, its token dead. Returns their ids. The | |
| 348 | * check each serves is still running under the forge's own lease: nothing is counted as a try. | |
| 349 | */ | |
| 350 | export function lapseJobs(db: Database, now: number): string[] { | |
| 351 | return db | |
| 352 | .query<{ id: string }, [number]>( | |
| 353 | `UPDATE check_job SET state = 'queued', runner_id = NULL, token_hash = NULL, lease_until = 0 | |
| 354 | WHERE state = 'claimed' AND lease_until <= ?1 RETURNING id`, | |
| 355 | ) | |
| 356 | .all(now) | |
| 357 | .map((r) => r.id); | |
| 358 | } | |
| 359 | ||
| 360 | /** Take a job nobody claimed back off the desk; false when a runner got there first. */ | |
| 361 | export function withdrawJob(db: Database, jobId: string): boolean { | |
| 362 | return ( | |
| 363 | db | |
| 364 | .query("DELETE FROM check_job WHERE id = ?1 AND state = 'queued'") | |
| 365 | .run(jobId).changes > 0 | |
| 366 | ); | |
| 367 | } | |
| 368 | ||
| 369 | /** The check that offered the job has ended: the row goes, and with it any claim. */ | |
| 370 | export function dropJob(db: Database, jobId: string): void { | |
| 371 | db.query("DELETE FROM check_job WHERE id = ?1").run(jobId); | |
| 372 | } | |
| 373 | ||
| 374 | /** Jobs whose check is no longer running (a forge that died mid-check): swept at boot. */ | |
| 375 | export function sweepJobs(db: Database): number { | |
| 376 | return db | |
| 377 | .query( | |
| 378 | "DELETE FROM check_job WHERE run_id NOT IN (SELECT id FROM run WHERE kind = 'check' AND state = 'running')", | |
| 379 | ) | |
| 380 | .run().changes; | |
| 381 | } | |
| 382 | ||
| 383 | export interface JobView { | |
| 384 | readonly state: "queued" | "claimed" | "done" | "returned"; | |
| 385 | readonly claims: number; | |
| 386 | /** The runner holding it, by name; "" when none does. */ | |
| 387 | readonly runner: string; | |
| 388 | readonly result: string | null; | |
| 389 | } | |
| 390 | ||
| 391 | export function viewJob(db: Database, jobId: string): JobView | undefined { | |
| 392 | const row = db | |
| 393 | .query< | |
| 394 | { | |
| 395 | state: JobView["state"]; | |
| 396 | claims: number; | |
| 397 | runner: string | null; | |
| 398 | result: string | null; | |
| 399 | }, | |
| 400 | [string] | |
| 401 | >( | |
| 402 | `SELECT j.state, j.claims, r.name AS runner, j.result | |
| 403 | FROM check_job j LEFT JOIN check_runner r ON r.id = j.runner_id WHERE j.id = ?1`, | |
| 404 | ) | |
| 405 | .get(jobId); | |
| 406 | return row === null | |
| 407 | ? undefined | |
| 408 | : { | |
| 409 | state: row.state, | |
| 410 | claims: row.claims, | |
| 411 | runner: row.runner ?? "", | |
| 412 | result: row.result, | |
| 413 | }; | |
| 414 | } | |
| 415 | ||
| 416 | /** What the check waiting on a job hears from the runner's requests, in this process. */ | |
| 417 | export interface JobEar { | |
| 418 | /** Something about the job changed: look again. */ | |
| 419 | readonly wake: () => void; | |
| 420 | /** Lines the runner streamed, and the step they belong to. */ | |
| 421 | readonly lines: (step: string, lines: readonly string[]) => void; | |
| 422 | } | |
| 423 | ||
| 424 | /** | |
| 425 | * This forge's desk in memory: the runners waiting in a claim, and the checks waiting on a job. | |
| 426 | * Both are gone with the process, and nothing is lost with them: the rows say what is claimed. | |
| 427 | */ | |
| 428 | export interface Desk { | |
| 429 | readonly claimers: Set<() => void>; | |
| 430 | readonly ears: Map<string, JobEar>; | |
| 431 | } | |
| 432 | ||
| 433 | const desks = new WeakMap<App, Desk>(); | |
| 434 | ||
| 435 | export function deskOf(app: App): Desk { | |
| 436 | let desk = desks.get(app); | |
| 437 | if (desk === undefined) { | |
| 438 | desk = { claimers: new Set(), ears: new Map() }; | |
| 439 | desks.set(app, desk); | |
| 440 | } | |
| 441 | return desk; | |
| 442 | } | |
| 443 | ||
| 444 | /** Tell every runner waiting in a claim that the desk changed. */ | |
| 445 | export function wakeClaimers(app: App): void { | |
| 446 | for (const wake of [...deskOf(app).claimers]) wake(); | |
| 447 | } | |
| 448 | ||
| 449 | /** | |
| 450 | * Wait until the desk changes, `ms` pass or `signal` aborts, whichever is first. A runner's | |
| 451 | * long poll rests here between looks. | |
| 452 | */ | |
| 453 | export function nextOffer( | |
| 454 | app: App, | |
| 455 | ms: number, | |
| 456 | signal: AbortSignal, | |
| 457 | ): Promise<void> { | |
| 458 | if (ms <= 0 || signal.aborted) return Promise.resolve(); | |
| 459 | const desk = deskOf(app); | |
| 460 | return new Promise((resolve) => { | |
| 461 | const done = (): void => { | |
| 462 | clearTimeout(timer); | |
| 463 | desk.claimers.delete(done); | |
| 464 | signal.removeEventListener("abort", done); | |
| 465 | resolve(); | |
| 466 | }; | |
| 467 | const timer = setTimeout(() => during("runner-wait", done), ms); | |
| 468 | desk.claimers.add(done); | |
| 469 | signal.addEventListener("abort", done, { once: true }); | |
| 470 | }); | |
| 471 | } |
src/checks/runner_step.ts added +274
| @@ -0,0 +1,274 @@ | ||
| 1 | /** | |
| 2 | * A check's steps on a runner (`checks.placement: runner`, #13652 R1): the forge's side. | |
| 3 | * | |
| 4 | * The check stays this forge's: claimed, leased, budgeted and recorded here, as it is while a | |
| 5 | * cluster Job runs it. `runRunnerStep` puts the steps on the desk (`runner_jobs.ts`) and waits. | |
| 6 | * The forge never needs a runner: when none is connected, none takes the job within the wait, the | |
| 7 | * one that took it goes silent twice or hands it back, the answer is "not taken" and the caller | |
| 8 | * runs the check on this machine as `process` would. | |
| 9 | * | |
| 10 | * What a runner's answer may decide is narrow on purpose (`runnerVerdict`): green, the check | |
| 11 | * command's own exit code, or its timeout. Anything else is the runner machine's, and the forge | |
| 12 | * runs the check itself. | |
| 13 | */ | |
| 14 | ||
| 15 | import type { App } from "../app.ts"; | |
| 16 | import { killedStep } from "../apps/process.ts"; | |
| 17 | import { during } from "../perf/jobs.ts"; | |
| 18 | import { err, ok, type Result } from "../kernel/result.ts"; | |
| 19 | import { parseBunTestFailure } from "./bun-test.ts"; | |
| 20 | import { rerunnable } from "./flake.ts"; | |
| 21 | import { checkStage } from "./gate_reason.ts"; | |
| 22 | import type { CheckOutcome, CheckTier } from "./runner.ts"; | |
| 23 | import { | |
| 24 | deskOf, | |
| 25 | dropJob, | |
| 26 | lapseJobs, | |
| 27 | offerJob, | |
| 28 | runnersAround, | |
| 29 | viewJob, | |
| 30 | wakeClaimers, | |
| 31 | withdrawJob, | |
| 32 | type RunnerJob, | |
| 33 | type RunnerResult, | |
| 34 | type RunnerStepResult, | |
| 35 | type RunnerStepSpec, | |
| 36 | } from "./runner_jobs.ts"; | |
| 37 | import { summariseCheckFailure } from "./summary.ts"; | |
| 38 | ||
| 39 | /** How long a job waits for a runner to take it before the forge runs the check itself. */ | |
| 40 | export const RUNNER_WAIT_MS = 5_000; | |
| 41 | /** A job whose runner went silent this many times is not offered again. */ | |
| 42 | export const MAX_CLAIMS = 2; | |
| 43 | /** How often a waiting check looks at its job when nothing woke it. */ | |
| 44 | const LOOK_MS = 1_000; | |
| 45 | ||
| 46 | /** | |
| 47 | * Whether a runner may take this check. Only one that hands nothing back but its log and its | |
| 48 | * verdict: a pull's check. The default branch's own run and a verification keep what they | |
| 49 | * measured (the cache, the impact map, kept results) from the checkout, and a runner has no way | |
| 50 | * to send those yet, so they stay on the forge's machine. | |
| 51 | */ | |
| 52 | export function runnerTakes(x: { | |
| 53 | readonly tier: CheckTier; | |
| 54 | readonly keep?: boolean | undefined; | |
| 55 | readonly keepImpact?: boolean | undefined; | |
| 56 | readonly isMain?: boolean | undefined; | |
| 57 | readonly only?: readonly string[] | undefined; | |
| 58 | }): boolean { | |
| 59 | return ( | |
| 60 | (x.tier === "fast" || x.tier === "run") && | |
| 61 | x.keep !== true && | |
| 62 | x.keepImpact !== true && | |
| 63 | x.isMain !== true && | |
| 64 | x.only === undefined | |
| 65 | ); | |
| 66 | } | |
| 67 | ||
| 68 | export type RunnerStepEnd = | |
| 69 | | { | |
| 70 | readonly taken: true; | |
| 71 | readonly ran: Result<void, string>; | |
| 72 | /** The step that did not end ok. */ | |
| 73 | readonly failed?: string; | |
| 74 | readonly runner: string; | |
| 75 | readonly contained: boolean; | |
| 76 | readonly note: string; | |
| 77 | } | |
| 78 | | { readonly taken: false; readonly why: string }; | |
| 79 | ||
| 80 | export interface RunnerStepInput { | |
| 81 | readonly runId: string; | |
| 82 | readonly repoId: string; | |
| 83 | /** The bare repository the commit is in: what the job's git door serves. */ | |
| 84 | readonly dir: string; | |
| 85 | readonly job: Omit<RunnerJob, "id">; | |
| 86 | readonly log: (line: string) => void; | |
| 87 | /** The check step's own output, apart from the echoed command. */ | |
| 88 | readonly output: string[]; | |
| 89 | readonly signal: AbortSignal; | |
| 90 | /** @default RUNNER_WAIT_MS */ | |
| 91 | readonly waitMs?: number | undefined; | |
| 92 | /** Tests stand a clock in. @default Date.now */ | |
| 93 | readonly now?: (() => number) | undefined; | |
| 94 | } | |
| 95 | ||
| 96 | function parseResult(text: string | null): RunnerResult | undefined { | |
| 97 | try { | |
| 98 | const value = JSON.parse(text ?? "") as RunnerResult; | |
| 99 | return Array.isArray(value.steps) ? value : undefined; | |
| 100 | } catch { | |
| 101 | return undefined; | |
| 102 | } | |
| 103 | } | |
| 104 | ||
| 105 | /** | |
| 106 | * What a runner's step results say, held to the steps it was asked for: the same names in the | |
| 107 | * same order, every one ok, or ok up to one that failed and nothing after it. Anything else (no | |
| 108 | * steps, a step left out, a name of its own, a step after a failure) is no answer, and the forge | |
| 109 | * runs the check itself: an empty or partial result is never green. | |
| 110 | */ | |
| 111 | export function readSteps( | |
| 112 | asked: readonly RunnerStepSpec[], | |
| 113 | said: readonly RunnerStepResult[], | |
| 114 | ): { ran: Result<void, string>; failed?: string } | undefined { | |
| 115 | if (said.length === 0 || said.length > asked.length) return undefined; | |
| 116 | for (const [i, step] of said.entries()) { | |
| 117 | if (step.name !== asked[i]!.name || typeof step.ok !== "boolean") return undefined; | |
| 118 | if (step.ok) continue; | |
| 119 | return i === said.length - 1 | |
| 120 | ? { ran: err(step.error ?? `${step.name} failed`), failed: step.name } | |
| 121 | : undefined; | |
| 122 | } | |
| 123 | return said.length === asked.length ? { ran: ok() } : undefined; | |
| 124 | } | |
| 125 | ||
| 126 | export async function runRunnerStep( | |
| 127 | app: App, | |
| 128 | input: RunnerStepInput, | |
| 129 | ): Promise<RunnerStepEnd> { | |
| 130 | const now = input.now ?? Date.now; | |
| 131 | const waitMs = input.waitMs ?? RUNNER_WAIT_MS; | |
| 132 | const desk = deskOf(app); | |
| 133 | // Nobody to offer it to: not a second is spent waiting. | |
| 134 | if (desk.claimers.size === 0 && !runnersAround(app.db, now())) | |
| 135 | return { taken: false, why: "no runner is connected" }; | |
| 136 | const job = offerJob(app.db, { | |
| 137 | runId: input.runId, | |
| 138 | repoId: input.repoId, | |
| 139 | dir: input.dir, | |
| 140 | job: input.job, | |
| 141 | }); | |
| 142 | let wake: () => void = () => {}; | |
| 143 | let sawEcho = false; | |
| 144 | desk.ears.set(job.id, { | |
| 145 | wake: () => wake(), | |
| 146 | lines: (step, lines) => { | |
| 147 | for (const line of lines) { | |
| 148 | // `runStep` echoes the command first; that echo is not the command's own output. | |
| 149 | if (step === "check") { | |
| 150 | if (sawEcho) input.output.push(line); | |
| 151 | sawEcho = true; | |
| 152 | } | |
| 153 | input.log(line); | |
| 154 | } | |
| 155 | }, | |
| 156 | }); | |
| 157 | /** Rest until a request of the runner's wakes this, a second passes or the check is stopped. */ | |
| 158 | const rest = (ms: number): Promise<void> => | |
| 159 | new Promise((resolve) => { | |
| 160 | const done = (): void => { | |
| 161 | clearTimeout(timer); | |
| 162 | input.signal.removeEventListener("abort", done); | |
| 163 | resolve(); | |
| 164 | }; | |
| 165 | const timer = setTimeout(() => during("runner-wait", done), ms); | |
| 166 | wake = done; | |
| 167 | input.signal.addEventListener("abort", done, { once: true }); | |
| 168 | }); | |
| 169 | let offeredAt = now(); | |
| 170 | let holder = ""; | |
| 171 | try { | |
| 172 | wakeClaimers(app); | |
| 173 | for (;;) { | |
| 174 | if (input.signal.aborted) | |
| 175 | return { | |
| 176 | taken: true, | |
| 177 | ran: err("check cancelled"), | |
| 178 | runner: holder, | |
| 179 | contained: false, | |
| 180 | note: "", | |
| 181 | }; | |
| 182 | // Every lapsed lease on the desk, this job's among them, goes back to queued. | |
| 183 | lapseJobs(app.db, now()); | |
| 184 | const view = viewJob(app.db, job.id); | |
| 185 | if (view === undefined) | |
| 186 | return { taken: false, why: "the job left the desk" }; | |
| 187 | if (view.state === "queued" && holder !== "") { | |
| 188 | input.log( | |
| 189 | `runner ${holder} went silent: the job is queued again, and that is not a try`, | |
| 190 | ); | |
| 191 | // What a half-run printed is not the check's output. | |
| 192 | input.output.length = 0; | |
| 193 | sawEcho = false; | |
| 194 | holder = ""; | |
| 195 | offeredAt = now(); | |
| 196 | wakeClaimers(app); | |
| 197 | } | |
| 198 | if (view.state === "claimed" && view.runner !== holder) { | |
| 199 | holder = view.runner; | |
| 200 | input.log(`placement: runner ${holder}`); | |
| 201 | } | |
| 202 | if (view.state === "done") { | |
| 203 | const result = parseResult(view.result); | |
| 204 | const read = result === undefined ? undefined : readSteps(input.job.steps, result.steps); | |
| 205 | if (result === undefined || read === undefined) | |
| 206 | return { taken: false, why: `runner ${holder} sent a result that does not match the job's steps` }; | |
| 207 | return { | |
| 208 | taken: true, | |
| 209 | ...read, | |
| 210 | runner: holder, | |
| 211 | contained: result.contained === true, | |
| 212 | note: | |
| 213 | typeof result.note === "string" ? result.note.slice(0, 200) : "", | |
| 214 | }; | |
| 215 | } | |
| 216 | if (view.state === "returned") | |
| 217 | return { | |
| 218 | taken: false, | |
| 219 | why: `runner ${holder} handed it back: ${view.result ?? ""}`, | |
| 220 | }; | |
| 221 | if ( | |
| 222 | view.state === "queued" && | |
| 223 | (view.claims >= MAX_CLAIMS || now() - offeredAt >= waitMs) | |
| 224 | ) { | |
| 225 | // A runner that claims in this instant keeps it: the DELETE's WHERE decides. | |
| 226 | if (withdrawJob(app.db, job.id)) | |
| 227 | return { | |
| 228 | taken: false, | |
| 229 | why: | |
| 230 | view.claims >= MAX_CLAIMS | |
| 231 | ? `its runners went silent ${view.claims} times` | |
| 232 | : `no runner took it in ${Math.round(waitMs / 1000)}s`, | |
| 233 | }; | |
| 234 | continue; | |
| 235 | } | |
| 236 | // Queued, the next look is no later than the end of its wait. | |
| 237 | const left = view.state === "queued" ? waitMs - (now() - offeredAt) : LOOK_MS; | |
| 238 | await rest(Math.min(Math.max(left, 1), LOOK_MS)); | |
| 239 | } | |
| 240 | } finally { | |
| 241 | desk.ears.delete(job.id); | |
| 242 | dropJob(app.db, job.id); | |
| 243 | } | |
| 244 | } | |
| 245 | ||
| 246 | /** A step error that is the check's own verdict: its exit code, or its time. */ | |
| 247 | const VERDICT = /^check (?:exited with code \d+|took longer than \d+s)/; | |
| 248 | ||
| 249 | /** | |
| 250 | * What a runner's answer decides. An outcome stands as the check's; a string is why it does not, | |
| 251 | * and the forge runs the check itself: | |
| 252 | * - the install failed there (this machine's install knows a registry's bad minute from a red); | |
| 253 | * - the step ended by anything but its own exit or time (a signal, a limit, a full disk on the | |
| 254 | * runner); | |
| 255 | * - one or two tests failed, which this machine re-runs alone to tell a flake from a red, and a | |
| 256 | * runner cannot yet. | |
| 257 | */ | |
| 258 | export function runnerVerdict( | |
| 259 | end: Extract<RunnerStepEnd, { taken: true }>, | |
| 260 | lines: readonly string[], | |
| 261 | aborted: boolean, | |
| 262 | ): CheckOutcome | string { | |
| 263 | if (aborted) return { ok: false, error: "check cancelled" }; | |
| 264 | if (end.ran.ok) return { ok: true }; | |
| 265 | const error = end.ran.error; | |
| 266 | if (end.failed !== "check") | |
| 267 | return `the ${end.failed ?? "job"} did not finish on runner ${end.runner} (${error})`; | |
| 268 | if (!VERDICT.test(error) || killedStep(error)) | |
| 269 | return `runner ${end.runner} could not finish the check (${error})`; | |
| 270 | if (rerunnable(lines, parseBunTestFailure(lines)) !== undefined) | |
| 271 | return `one or two tests failed on runner ${end.runner}: re-run here, where a failure is tried alone`; | |
| 272 | const { summary, detail } = summariseCheckFailure(lines, error); | |
| 273 | return { ok: false, error: summary, detail, stage: checkStage(error) }; | |
| 274 | } |
src/checks/service.ts +74 −2
| @@ -68,7 +68,7 @@ import { checkOnApproval } from "./approval.ts"; | ||
| 68 | 68 | import { BASE_MOVED, dueGreen, newestBaseFirst } from "./base_moved.ts"; |
| 69 | 69 | import { checkKey, provenBy, recordProven } from "./proven.ts"; |
| 70 | 70 | import { beginGate, endGate, gateHolding } from "./gate_record.ts"; |
| 71 | import { giveHistory } from "./history.ts"; | |
| 71 | import { giveHistory, headDepth } from "./history.ts"; | |
| 72 | 72 | import { readLearned } from "./learned.ts"; |
| 73 | 73 | import { keepCache, restoreCache } from "./cache.ts"; |
| 74 | 74 | import { MAX_FILED, clearedFiles, fileOrderDependent } from "./order_dependent.ts"; |
| @@ -139,6 +139,8 @@ import { JOB_GATE_NOTE, jobGatePolicy, openJobGate } from "./job_gate.ts"; | ||
| 139 | 139 | import { placementFromEnv, type Placement } from "./placement.ts"; |
| 140 | 140 | import { refuseSmallPod, type PodRoom } from "./pod_room.ts"; |
| 141 | 141 | import { JOB_VANISHED, runJobStep } from "./job_step.ts"; |
| 142 | import { runRunnerStep, runnerTakes, runnerVerdict } from "./runner_step.ts"; | |
| 143 | import { sweepJobs } from "./runner_jobs.ts"; | |
| 142 | 144 | import { |
| 143 | 145 | cellInstall, |
| 144 | 146 | checkInstallCache, |
| @@ -606,6 +608,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 606 | 608 | capacityCount(app.db, "checks.concurrency", booted); |
| 607 | 609 | const gate = options.gate ?? new Gate(capacity()); |
| 608 | 610 | const placement = options.placement ?? placementFromEnv(); |
| 611 | // Jobs a forge that died left on the runners' desk: their checks are queued again elsewhere. | |
| 612 | if (placement.mode === "runner") sweepJobs(app.db); | |
| 609 | 613 | /** |
| 610 | 614 | * Full gates on land in flight: not slots in the queue, but stopped with |
| 611 | 615 | * it — after the drain, as a check is. By commit, so a merge queue that |
| @@ -1365,6 +1369,73 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1365 | 1369 | write(`already proven green by ${earlier}`); |
| 1366 | 1370 | return { ok: true, proven: earlier }; |
| 1367 | 1371 | } |
| 1372 | // A runner that joined takes the steps (`runner_step.ts`, #13652): only a check that hands | |
| 1373 | // back nothing but its log and its verdict, and only while a runner is connected. Not taken, | |
| 1374 | // or an answer that is not the check's own verdict, and it runs here as it always has. | |
| 1375 | if (placement.mode === "runner" && x.runId !== undefined && runnerTakes(x)) { | |
| 1376 | const lines: string[] = []; | |
| 1377 | const sent = await runRunnerStep(app, { | |
| 1378 | runId: x.runId, | |
| 1379 | repoId: repo.id, | |
| 1380 | dir: bare, | |
| 1381 | job: { | |
| 1382 | run: x.runId, | |
| 1383 | repo: `${repo.ownerName}/${repo.name}`, | |
| 1384 | commit: run.commitOid, | |
| 1385 | ...(base === undefined ? {} : { base }), | |
| 1386 | depth: base === undefined ? 1 : await headDepth(bare, base, run.commitOid), | |
| 1387 | steps: [ | |
| 1388 | ...(manifest.install === null ? [] : [{ name: "install" as const, command: manifest.install }]), | |
| 1389 | { name: "check" as const, command }, | |
| 1390 | ], | |
| 1391 | // Only the check's own variables cross, never the forge's; the runner adds its cell's. | |
| 1392 | env: { | |
| 1393 | CI: "1", | |
| 1394 | FORGE_URL: options.forgeUrl, | |
| 1395 | FORGE_CHECK: run.id, | |
| 1396 | FORGE_COMMIT: x.commitOid, | |
| 1397 | FORGE_CHECK_IMPACT: IMPACT_FILE, | |
| 1398 | FORGE_CHECK_KEPT: KEPT_FILE, | |
| 1399 | ...(base === undefined ? {} : { FORGE_CHECK_BASE: base, FORGE_CHECK_LEARNED: learned }), | |
| 1400 | FORGE_CELLS: "off", | |
| 1401 | }, | |
| 1402 | limits: { | |
| 1403 | ...(manifest.limits.cpus === undefined ? {} : { cpus: manifest.limits.cpus }), | |
| 1404 | ...(manifest.limits.memoryMiB === undefined ? {} : { memoryMiB: manifest.limits.memoryMiB }), | |
| 1405 | ...(manifest.limits.diskMiB === undefined ? {} : { diskMiB: manifest.limits.diskMiB }), | |
| 1406 | }, | |
| 1407 | timeoutS: manifest.timeout, | |
| 1408 | egress: [...STEP_HOSTS, ...manifest.egress], | |
| 1409 | egressMode: outside ? "enforce" : egressSetting(app.db), | |
| 1410 | // An outside contributor's code runs in a cell or not at all, there as here. | |
| 1411 | cell: outside ? "required" : "preferred", | |
| 1412 | cache: { repo: repo.id, tree: tree.ok ? tree.value.trim() : "", base: builtOnBase ?? "" }, | |
| 1413 | }, | |
| 1414 | log: write, | |
| 1415 | output: lines, | |
| 1416 | signal, | |
| 1417 | waitMs: placement.waitMs, | |
| 1418 | }); | |
| 1419 | const verdict = sent.taken ? runnerVerdict(sent, lines, signal.aborted) : sent.why; | |
| 1420 | if (typeof verdict === "string") write(`placement: runner; ${verdict}; running here`); | |
| 1421 | else { | |
| 1422 | if (sent.taken && sent.runner !== "") | |
| 1423 | noteRunProgress(app.db, x.runId, { | |
| 1424 | placement: "runner", | |
| 1425 | node: sent.runner, | |
| 1426 | sandbox: { enforced: sent.contained, note: sent.note }, | |
| 1427 | }); | |
| 1428 | try { | |
| 1429 | recordCrashFlakes(app.db, app.bus, repo, lines, { runId: run.id, commit: run.commitOid, pull: true }, verdict.ok); | |
| 1430 | } catch (error) { | |
| 1431 | write(`crash flakes not recorded: ${String(error)}`); | |
| 1432 | } | |
| 1433 | if (!verdict.ok) return verdict; | |
| 1434 | if (key !== undefined) recordProven(app.db, key, x.id); | |
| 1435 | const cleared = clearedIn(lines); | |
| 1436 | return { ok: true, ...(cleared === undefined ? {} : { cleared }) }; | |
| 1437 | } | |
| 1438 | } | |
| 1368 | 1439 | if (await reclaimIfFull(app)) write("disk under its free floor: swept unused checkouts"); |
| 1369 | 1440 | const dir = join(checksRoot(app.config.dataDir), run.id); |
| 1370 | 1441 | mkdirSync(dir, { recursive: true }); |
| @@ -1409,7 +1480,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle { | ||
| 1409 | 1480 | // (`checks/job_gate.ts`, #13251): a cluster that names none cannot |
| 1410 | 1481 | // honour forge.json, so the check stays in a cell, never unrestricted. |
| 1411 | 1482 | const fellBack = placement.mode === "job" && placement.gate === undefined; |
| 1412 | const placed: Placement = fellBack ? { mode: "process" } : placement; | |
| 1483 | // A check a runner did not take runs here as `process` does. | |
| 1484 | const placed: Placement = fellBack || placement.mode === "runner" ? { mode: "process" } : placement; | |
| 1413 | 1485 | const job = placed.mode === "job" ? placed : undefined; |
| 1414 | 1486 | if (x.runId !== undefined) |
| 1415 | 1487 | noteRunProgress(app.db, x.runId, { placement: placed.mode }); |
src/contract/openapi/routes/deleteApiV1AdminRunnersId.ts added +11
| @@ -0,0 +1,11 @@ | ||
| 1 | import type { Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "delete", | |
| 5 | path: "/api/v1/admin/runners/{id}", | |
| 6 | summary: | |
| 7 | "Remove a runner by id (admin scope): its token stops at once and a job it held is queued again; 404 when there is none", | |
| 8 | auth: true, | |
| 9 | role: "admin", | |
| 10 | status: 204, | |
| 11 | }; |
src/contract/openapi/routes/getApiV1AdminRunners.ts added +10
| @@ -0,0 +1,10 @@ | ||
| 1 | import type { Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "get", | |
| 5 | path: "/api/v1/admin/runners", | |
| 6 | summary: | |
| 7 | "The enrolled runners, each with when it last asked for work (seenAt, milliseconds; 0 for never); answers { items, next_cursor }", | |
| 8 | auth: true, | |
| 9 | role: "admin", | |
| 10 | }; |
src/contract/openapi/routes/getApiV1RunnerJobsIdGitInfoRefs.ts added +11
| @@ -0,0 +1,11 @@ | ||
| 1 | import type { Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "get", | |
| 5 | path: "/api/v1/runner/jobs/{id}/git/info/refs", | |
| 6 | summary: | |
| 7 | "The git door of one job (smart HTTP, fetch only): the ref advertisement of the job's repository. Takes that job's token as the bearer; ?service=git-upload-pack", | |
| 8 | auth: false, | |
| 9 | role: "anonymous", | |
| 10 | query: { service: "string" }, | |
| 11 | }; |
src/contract/openapi/routes/index.ts +7
| @@ -2,6 +2,7 @@ | ||
| 2 | 2 | // One line per route, sorted: a new route is a new file and one new line here. |
| 3 | 3 | export { route as deleteApiV1AdminAccountsUsername } from "./deleteApiV1AdminAccountsUsername.ts"; |
| 4 | 4 | export { route as deleteApiV1AdminInvitesId } from "./deleteApiV1AdminInvitesId.ts"; |
| 5 | export { route as deleteApiV1AdminRunnersId } from "./deleteApiV1AdminRunnersId.ts"; | |
| 5 | 6 | export { route as deleteApiV1AgentsUsername } from "./deleteApiV1AgentsUsername.ts"; |
| 6 | 7 | export { route as deleteApiV1BoardsSlug } from "./deleteApiV1BoardsSlug.ts"; |
| 7 | 8 | export { route as deleteApiV1CodesId } from "./deleteApiV1CodesId.ts"; |
| @@ -54,6 +55,7 @@ export { route as getApiV1AdminPinned } from "./getApiV1AdminPinned.ts"; | ||
| 54 | 55 | export { route as getApiV1AdminProfile } from "./getApiV1AdminProfile.ts"; |
| 55 | 56 | export { route as getApiV1AdminQuotas } from "./getApiV1AdminQuotas.ts"; |
| 56 | 57 | export { route as getApiV1AdminRequests } from "./getApiV1AdminRequests.ts"; |
| 58 | export { route as getApiV1AdminRunners } from "./getApiV1AdminRunners.ts"; | |
| 57 | 59 | export { route as getApiV1AdminSettings } from "./getApiV1AdminSettings.ts"; |
| 58 | 60 | export { route as getApiV1AdminShield } from "./getApiV1AdminShield.ts"; |
| 59 | 61 | export { route as getApiV1AdminSubscriptions } from "./getApiV1AdminSubscriptions.ts"; |
| @@ -207,6 +209,7 @@ export { route as getApiV1ReposOwnerNameVision } from "./getApiV1ReposOwnerNameV | ||
| 207 | 209 | export { route as getApiV1ReposOwnerNameWaves } from "./getApiV1ReposOwnerNameWaves.ts"; |
| 208 | 210 | export { route as getApiV1ReposOwnerNameWavesVersion } from "./getApiV1ReposOwnerNameWavesVersion.ts"; |
| 209 | 211 | export { route as getApiV1Returns } from "./getApiV1Returns.ts"; |
| 212 | export { route as getApiV1RunnerJobsIdGitInfoRefs } from "./getApiV1RunnerJobsIdGitInfoRefs.ts"; | |
| 210 | 213 | export { route as getApiV1Search } from "./getApiV1Search.ts"; |
| 211 | 214 | export { route as getApiV1SecretsScope } from "./getApiV1SecretsScope.ts"; |
| 212 | 215 | export { route as getApiV1TelemetryFlow } from "./getApiV1TelemetryFlow.ts"; |
| @@ -285,6 +288,7 @@ export { route as postApiV1AdminPinnedNameUpdate } from "./postApiV1AdminPinnedN | ||
| 285 | 288 | export { route as postApiV1AdminReleasesShaRollback } from "./postApiV1AdminReleasesShaRollback.ts"; |
| 286 | 289 | export { route as postApiV1AdminRequestsUsernameAllow } from "./postApiV1AdminRequestsUsernameAllow.ts"; |
| 287 | 290 | export { route as postApiV1AdminRequestsUsernameDecline } from "./postApiV1AdminRequestsUsernameDecline.ts"; |
| 291 | export { route as postApiV1AdminRunners } from "./postApiV1AdminRunners.ts"; | |
| 288 | 292 | export { route as postApiV1Agents } from "./postApiV1Agents.ts"; |
| 289 | 293 | export { route as postApiV1Apps } from "./postApiV1Apps.ts"; |
| 290 | 294 | export { route as postApiV1AsksAdmin } from "./postApiV1AsksAdmin.ts"; |
| @@ -409,6 +413,9 @@ export { route as postApiV1ReposOwnerNameSpend } from "./postApiV1ReposOwnerName | ||
| 409 | 413 | export { route as postApiV1ReposOwnerNameSync } from "./postApiV1ReposOwnerNameSync.ts"; |
| 410 | 414 | export { route as postApiV1ReposOwnerNameTransfer } from "./postApiV1ReposOwnerNameTransfer.ts"; |
| 411 | 415 | export { route as postApiV1ReposOwnerNameUpstreamCheck } from "./postApiV1ReposOwnerNameUpstreamCheck.ts"; |
| 416 | export { route as postApiV1RunnerClaim } from "./postApiV1RunnerClaim.ts"; | |
| 417 | export { route as postApiV1RunnerJobsId } from "./postApiV1RunnerJobsId.ts"; | |
| 418 | export { route as postApiV1RunnerJobsIdGitGitUploadPack } from "./postApiV1RunnerJobsIdGitGitUploadPack.ts"; | |
| 412 | 419 | export { route as postApiV1Sessions } from "./postApiV1Sessions.ts"; |
| 413 | 420 | export { route as postApiV1Share } from "./postApiV1Share.ts"; |
| 414 | 421 | export { route as postApiV1Start } from "./postApiV1Start.ts"; |
src/contract/openapi/routes/postApiV1AdminRunners.ts added +13
| @@ -0,0 +1,13 @@ | ||
| 1 | import type { Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "post", | |
| 5 | path: "/api/v1/admin/runners", | |
| 6 | summary: | |
| 7 | "Enrol a runner: a machine that takes checks off this forge when FORGE_CHECKS_PLACEMENT=runner (admin scope); answers its row and its token, shown this once", | |
| 8 | auth: true, | |
| 9 | role: "admin", | |
| 10 | body: { name: "string" }, | |
| 11 | status: 201, | |
| 12 | reply: { id: "string", name: "string", createdAt: "string", seenAt: "integer", token: "string" }, | |
| 13 | }; |
src/contract/openapi/routes/postApiV1RunnerClaim.ts added +13
| @@ -0,0 +1,13 @@ | ||
| 1 | import type { Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "post", | |
| 5 | path: "/api/v1/runner/claim", | |
| 6 | summary: | |
| 7 | "A runner asks for a check to run. Takes a runner token as the bearer (from POST /api/v1/admin/runners), never an account's. Held open up to wait seconds (25 at most); answers { job, token, lease } with the job's own token, or 204 when nothing was offered", | |
| 8 | auth: false, | |
| 9 | role: "anonymous", | |
| 10 | body: { wait: "integer", cells: "boolean" }, | |
| 11 | alsoStatus: [204], | |
| 12 | refusals: { unauthorized: "401: not a runner token this forge enrolled" }, | |
| 13 | }; |
src/contract/openapi/routes/postApiV1RunnerJobsId.ts added +12
| @@ -0,0 +1,12 @@ | ||
| 1 | import type { Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "post", | |
| 5 | path: "/api/v1/runner/jobs/{id}", | |
| 6 | summary: | |
| 7 | "A runner reports on the job it holds. Takes that job's token as the bearer. lines (of one step) are appended to the check's log; done { steps, contained, note } ends it with a result, abandon hands it back; every report renews the lease and answers { lease } or { ended }", | |
| 8 | auth: false, | |
| 9 | role: "anonymous", | |
| 10 | body: { step: "string", lines: "array", done: "object", abandon: "string" }, | |
| 11 | refusals: { not_found: "404: no such job, not this token's job, or its lease is over: stop the job", invalid_result: "422: done is not { steps: [{ name, ok, error?, ms }], contained, note }" }, | |
| 12 | }; |
src/contract/openapi/routes/postApiV1RunnerJobsIdGitGitUploadPack.ts added +10
| @@ -0,0 +1,10 @@ | ||
| 1 | import type { Route } from "../shared.ts"; | |
| 2 | ||
| 3 | export const route: Route = { | |
| 4 | method: "post", | |
| 5 | path: "/api/v1/runner/jobs/{id}/git/git-upload-pack", | |
| 6 | summary: | |
| 7 | "The git door of one job: the fetch itself, held to the commit the job names and its base; any other object is refused. Takes that job's token as the bearer", | |
| 8 | auth: false, | |
| 9 | role: "anonymous", | |
| 10 | }; |
src/contract/tags.ts +1
| @@ -44,6 +44,7 @@ const ALIAS: Readonly<Record<string, string>> = { | ||
| 44 | 44 | secrets: "tokens", |
| 45 | 45 | sessions: "tokens", |
| 46 | 46 | "first-admin": "admin", |
| 47 | runner: "meta", | |
| 47 | 48 | health: "meta", |
| 48 | 49 | version: "meta", |
| 49 | 50 | releases: "meta", |
src/core/event_audience.ts +2
| @@ -239,6 +239,8 @@ export function classifiedAudience(kind: string): EventAudience | undefined { | ||
| 239 | 239 | kind.startsWith("mail.") || |
| 240 | 240 | // A restore drill's verdict is the forge owner's (`snapshot/service.ts`). |
| 241 | 241 | kind.startsWith("drill.") || |
| 242 | // Which machines take this forge's checks is its admin's (`actions/runners.ts`). | |
| 243 | kind.startsWith("runner.") || | |
| 242 | 244 | // A snapshot moving is the forge owner's too, and has no repository. |
| 243 | 245 | kind.startsWith("backup.") || |
| 244 | 246 | // Preparing for a power-off and ending it are the forge owner's (`maintenance/service.ts`). |
src/core/migration-pins/20261116005600-check-runner.pin added +1
| @@ -0,0 +1,1 @@ | ||
| 1 | 024e3f117e4ca43c33ee3054c4186a9365fead6e46619ec7801c11f852a2cd43 |
src/core/migrations/20261116005600-check-runner.ts added +34
| @@ -0,0 +1,34 @@ | ||
| 1 | // Runners (#13652, R1): machines that take a check's steps off the forge. `check_runner` is one | |
| 2 | // enrolled runner and the hash of its token; `check_job` is a check's steps while a runner may | |
| 3 | // hold them: a row lives from the offer to the answer and is deleted then, so the table stays as | |
| 4 | // small as the checks in flight. A claim is the UPDATE's WHERE on `state`; `lease_until` is | |
| 5 | // milliseconds since the epoch. `check_job_state` is what the claim walks (oldest queued first), | |
| 6 | // `check_job_run` what a check's end deletes by. | |
| 7 | // Additive: two tables and their indexes. | |
| 8 | export default ` | |
| 9 | CREATE TABLE check_runner ( | |
| 10 | id TEXT PRIMARY KEY, | |
| 11 | name TEXT NOT NULL, | |
| 12 | token_hash TEXT NOT NULL UNIQUE, | |
| 13 | created_by TEXT NOT NULL, | |
| 14 | created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')), | |
| 15 | seen_at INTEGER NOT NULL DEFAULT 0 | |
| 16 | ); | |
| 17 | CREATE TABLE check_job ( | |
| 18 | id TEXT PRIMARY KEY, | |
| 19 | run_id TEXT NOT NULL, | |
| 20 | repo_id TEXT NOT NULL, | |
| 21 | dir TEXT NOT NULL, | |
| 22 | spec TEXT NOT NULL, | |
| 23 | needs_cell INTEGER NOT NULL DEFAULT 0, | |
| 24 | state TEXT NOT NULL DEFAULT 'queued', | |
| 25 | runner_id TEXT, | |
| 26 | token_hash TEXT, | |
| 27 | lease_until INTEGER NOT NULL DEFAULT 0, | |
| 28 | claims INTEGER NOT NULL DEFAULT 0, | |
| 29 | result TEXT | |
| 30 | ); | |
| 31 | CREATE INDEX check_job_state ON check_job (state, id); | |
| 32 | CREATE INDEX check_job_run ON check_job (run_id); | |
| 33 | CREATE UNIQUE INDEX check_job_token ON check_job (token_hash) WHERE token_hash IS NOT NULL; | |
| 34 | `; |
src/git/smart-http.ts +20 −4
| @@ -22,6 +22,9 @@ import { OWN_REFS_ONLY, reachableFromOwnRefs, refusalLine, wantsOf } from "./own | ||
| 22 | 22 | |
| 23 | 23 | export type GitService = "git-upload-pack" | "git-receive-pack"; |
| 24 | 24 | |
| 25 | /** What a fetch held to `onlyWants` is told when it asks for anything else. */ | |
| 26 | export const ONLY_WANTS = "this credential fetches only the commits its job names"; | |
| 27 | ||
| 25 | 28 | export function isGitService(value: string | null): value is GitService { |
| 26 | 29 | return value === "git-upload-pack" || value === "git-receive-pack"; |
| 27 | 30 | } |
| @@ -97,6 +100,12 @@ export interface SmartHttpOptions { | ||
| 97 | 100 | * wants must be reachable from the fork's own refs before git is given it. |
| 98 | 101 | */ |
| 99 | 102 | ownRefsOnly?: boolean; |
| 103 | /** | |
| 104 | * The only objects this fetch may want, by id (a runner's job token: its | |
| 105 | * commit and its base, `checks/runner_jobs.ts`). Spoken in protocol v0 like | |
| 106 | * a held reader's, so every want is seen before git is given it. | |
| 107 | */ | |
| 108 | onlyWants?: readonly string[]; | |
| 100 | 109 | } |
| 101 | 110 | |
| 102 | 111 | /** How long a fetch's negotiation may go without a byte before it ends. */ |
| @@ -144,12 +153,18 @@ export async function serviceRpc( | ||
| 144 | 153 | // A held reader's request is read whole first (it is the negotiation, a few |
| 145 | 154 | // kilobytes): what it wants is asked of the fork's own refs, then git gets it. |
| 146 | 155 | let read: Uint8Array | undefined; |
| 147 | if (options.ownRefsOnly === true && service === "git-upload-pack") { | |
| 156 | const only = options.onlyWants; | |
| 157 | if ((options.ownRefsOnly === true || only !== undefined) && service === "git-upload-pack") { | |
| 148 | 158 | read = await negotiation(request); |
| 149 | 159 | const asked = read === undefined ? "bad" : wantsOf(read); |
| 150 | if (asked === "bad" || !(await reachableFromOwnRefs(repoDir, asked.wants))) { | |
| 160 | const beyond = asked !== "bad" && only !== undefined && asked.wants.some((want) => !only.includes(want)); | |
| 161 | if ( | |
| 162 | asked === "bad" || | |
| 163 | beyond || | |
| 164 | (options.ownRefsOnly === true && !(await reachableFromOwnRefs(repoDir, asked.wants))) | |
| 165 | ) { | |
| 151 | 166 | options.release?.(); |
| 152 | return new Response(refusalLine(OWN_REFS_ONLY), { | |
| 167 | return new Response(refusalLine(beyond ? ONLY_WANTS : OWN_REFS_ONLY), { | |
| 153 | 168 | headers: { |
| 154 | 169 | "content-type": `application/x-${service}-result`, |
| 155 | 170 | "cache-control": "no-cache", |
| @@ -212,7 +227,8 @@ async function spawnService( | ||
| 212 | 227 | // Anything else silently falls back to v0. |
| 213 | 228 | const protocol = request.headers.get("git-protocol"); |
| 214 | 229 | // A held reader is answered in v0 whatever was asked: v2 serves any object by id. |
| 215 | if (options.ownRefsOnly !== true && protocol !== null && /^[A-Za-z0-9=:._-]{1,200}$/.test(protocol)) { | |
| 230 | const held = options.ownRefsOnly === true || options.onlyWants !== undefined; | |
| 231 | if (!held && protocol !== null && /^[A-Za-z0-9=:._-]{1,200}$/.test(protocol)) { | |
| 216 | 232 | env.GIT_PROTOCOL = protocol; |
| 217 | 233 | } |
| 218 | 234 |
src/http/api.ts +2
| @@ -392,6 +392,7 @@ import { visionRoutes } from "./api/vision.ts"; | ||
| 392 | 392 | import { businessPlanRoutes } from "./api/business_plan.ts"; |
| 393 | 393 | import { tokenRoutes } from "./api/tokens.ts"; |
| 394 | 394 | import { accountAdmin, adminRoutes } from "./api/admin.ts"; |
| 395 | import { runnerRoutes } from "./api/runners.ts"; | |
| 395 | 396 | import { maintenanceRoutes } from "./api/maintenance.ts"; |
| 396 | 397 | import { maintenanceRunsRoutes } from "./api/maintenance_runs.ts"; |
| 397 | 398 | import { diskReadingRoutes } from "./api/disk_readings.ts"; |
| @@ -1301,6 +1302,7 @@ export function apiRoutes(router: Router<Handler>): void { | ||
| 1301 | 1302 | userRoutes(router); |
| 1302 | 1303 | forgeOriginRoutes(router); |
| 1303 | 1304 | adminRoutes(router); |
| 1305 | runnerRoutes(router); | |
| 1304 | 1306 | maintenanceRoutes(router); |
| 1305 | 1307 | maintenanceRunsRoutes(router); |
| 1306 | 1308 | diskReadingRoutes(router); |
src/http/api/runners.ts added +207
| @@ -0,0 +1,207 @@ | ||
| 1 | /** | |
| 2 | * Runners over the API (#13652): `/api/v1/admin/runners` is an admin's (enrol, list, remove), and | |
| 3 | * `/api/v1/runner/…` is a runner's own door: claim a job, report on it, fetch its commit. | |
| 4 | * | |
| 5 | * The runner routes take neither an account nor an account's token. Each reads its own | |
| 6 | * credential from `Authorization: Bearer` (`checks/runner_jobs.ts`): a runner token claims, a | |
| 7 | * job token reports on and fetches its one job while its lease lives. Every other answer is the | |
| 8 | * same 404, so a guessed job id says nothing. To the rest of the forge both credentials are a | |
| 9 | * stranger's header: no page, no API route and no git door knows them. | |
| 10 | */ | |
| 11 | ||
| 12 | import { enrol, list, remove } from "../../actions/runners.ts"; | |
| 13 | import { | |
| 14 | LEASE_MS, | |
| 15 | claimJob, | |
| 16 | deskOf, | |
| 17 | getRunner, | |
| 18 | jobByToken, | |
| 19 | nextOffer, | |
| 20 | renewJob, | |
| 21 | runnerByToken, | |
| 22 | settleJob, | |
| 23 | type RunnerResult, | |
| 24 | } from "../../checks/runner_jobs.ts"; | |
| 25 | import { notePhase, takeSessionTime } from "../../core/route_timing.ts"; | |
| 26 | import { advertiseRefs, serviceRpc } from "../../git/smart-http.ts"; | |
| 27 | import type { Ctx, Handler } from "../context.ts"; | |
| 28 | import { STRANGER_MAX } from "../limits.ts"; | |
| 29 | import { | |
| 30 | apiError, | |
| 31 | body, | |
| 32 | boundedBody, | |
| 33 | json, | |
| 34 | notFound, | |
| 35 | str, | |
| 36 | unauthorized, | |
| 37 | } from "../respond.ts"; | |
| 38 | import type { Router } from "../router.ts"; | |
| 39 | import { accountAdmin } from "./admin.ts"; | |
| 40 | ||
| 41 | /** The longest a claim waits for a job, in seconds: under every proxy's own patience. */ | |
| 42 | export const CLAIM_WAIT_MAX_S = 25; | |
| 43 | /** The most one report carries: a second or two of a loud check's output. */ | |
| 44 | const REPORT_MAX = 2 * 1024 * 1024; | |
| 45 | const LINES_MAX = 5_000; | |
| 46 | ||
| 47 | const bearer = (request: Request): string => | |
| 48 | /^Bearer (\S+)$/i.exec(request.headers.get("authorization") ?? "")?.[1] ?? ""; | |
| 49 | ||
| 50 | const lease = { seconds: LEASE_MS / 1000 }; | |
| 51 | ||
| 52 | /** A report's result, read field by field: nothing a runner sends is kept as it came. */ | |
| 53 | function readResult(raw: unknown): RunnerResult | undefined { | |
| 54 | if (typeof raw !== "object" || raw === null) return undefined; | |
| 55 | const from = raw as Record<string, unknown>; | |
| 56 | if (!Array.isArray(from.steps) || from.steps.length > 8) return undefined; | |
| 57 | const steps = []; | |
| 58 | for (const s of from.steps as Record<string, unknown>[]) { | |
| 59 | if ( | |
| 60 | typeof s !== "object" || | |
| 61 | s === null || | |
| 62 | typeof s.name !== "string" || | |
| 63 | typeof s.ok !== "boolean" | |
| 64 | ) | |
| 65 | return undefined; | |
| 66 | steps.push({ | |
| 67 | name: s.name.slice(0, 40), | |
| 68 | ok: s.ok, | |
| 69 | ...(typeof s.error === "string" | |
| 70 | ? { error: s.error.slice(0, 2_000) } | |
| 71 | : {}), | |
| 72 | ms: | |
| 73 | typeof s.ms === "number" && Number.isFinite(s.ms) | |
| 74 | ? Math.max(0, Math.round(s.ms)) | |
| 75 | : 0, | |
| 76 | }); | |
| 77 | } | |
| 78 | return { | |
| 79 | steps, | |
| 80 | contained: from.contained === true, | |
| 81 | note: str(from.note).slice(0, 200), | |
| 82 | }; | |
| 83 | } | |
| 84 | ||
| 85 | export function runnerRoutes(router: Router<Handler>): void { | |
| 86 | router.add("POST", "/api/v1/admin/runners", async (ctx) => { | |
| 87 | const denied = accountAdmin(ctx, "admin"); | |
| 88 | if (denied) return denied; | |
| 89 | const input = await body(ctx.request); | |
| 90 | const made = enrol(ctx.app, ctx.auth.user, str(input.name)); | |
| 91 | if (!made.ok) return notFound(); | |
| 92 | return json({ ...made.value.runner, token: made.value.token }, 201); | |
| 93 | }); | |
| 94 | ||
| 95 | router.add("GET", "/api/v1/admin/runners", (ctx) => { | |
| 96 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); | |
| 97 | const denied = accountAdmin(ctx, "ro"); | |
| 98 | if (denied) return denied; | |
| 99 | const seen = list(ctx.app, ctx.auth.user); | |
| 100 | return seen.ok | |
| 101 | ? json({ items: seen.value, next_cursor: null }) | |
| 102 | : notFound(); | |
| 103 | }); | |
| 104 | ||
| 105 | router.add("DELETE", "/api/v1/admin/runners/:id", (ctx) => { | |
| 106 | const denied = accountAdmin(ctx, "admin"); | |
| 107 | if (denied) return denied; | |
| 108 | const gone = remove(ctx.app, ctx.auth.user, ctx.params.id!); | |
| 109 | return gone.ok ? new Response(null, { status: 204 }) : notFound(); | |
| 110 | }); | |
| 111 | ||
| 112 | // A runner asks for work. Held open until a job is offered or `wait` seconds pass (204). | |
| 113 | router.add("POST", "/api/v1/runner/claim", async (ctx) => { | |
| 114 | const runner = runnerByToken(ctx.app.db, bearer(ctx.request), Date.now()); | |
| 115 | if (runner === undefined) return unauthorized(); | |
| 116 | const input = await boundedBody(ctx.request, STRANGER_MAX); | |
| 117 | if (input === "too_large") | |
| 118 | return apiError("too_large", 413, `at most ${STRANGER_MAX} bytes`); | |
| 119 | const wait = | |
| 120 | Math.min(Math.max(Number(input.wait) || 0, 0), CLAIM_WAIT_MAX_S) * 1000; | |
| 121 | const until = Date.now() + wait; | |
| 122 | const stop = AbortSignal.any([ctx.request.signal, ctx.app.closing.signal]); | |
| 123 | for (;;) { | |
| 124 | // Removed while it waited: its token is no longer one. | |
| 125 | if (getRunner(ctx.app.db, runner.id) === undefined) return unauthorized(); | |
| 126 | const claim = claimJob( | |
| 127 | ctx.app.db, | |
| 128 | { id: runner.id, cells: input.cells === true }, | |
| 129 | Date.now(), | |
| 130 | ); | |
| 131 | if (claim !== undefined) { | |
| 132 | deskOf(ctx.app).ears.get(claim.job.id)?.wake(); | |
| 133 | return json({ job: claim.job, token: claim.token, lease }); | |
| 134 | } | |
| 135 | const left = until - Date.now(); | |
| 136 | if (left <= 0 || stop.aborted) return new Response(null, { status: 204 }); | |
| 137 | await nextOffer(ctx.app, left, stop); | |
| 138 | } | |
| 139 | }); | |
| 140 | ||
| 141 | /** The job this request's token holds, with the check waiting on it here; else nothing. */ | |
| 142 | const held = (ctx: Ctx) => { | |
| 143 | const job = jobByToken( | |
| 144 | ctx.app.db, | |
| 145 | ctx.params.id!, | |
| 146 | bearer(ctx.request), | |
| 147 | Date.now(), | |
| 148 | ); | |
| 149 | const ear = | |
| 150 | job === undefined ? undefined : deskOf(ctx.app).ears.get(job.id); | |
| 151 | return job === undefined || ear === undefined ? undefined : { job, ear }; | |
| 152 | }; | |
| 153 | ||
| 154 | // A runner reports on its job: log lines, then the end. Every report renews the lease. | |
| 155 | router.add("POST", "/api/v1/runner/jobs/:id", async (ctx) => { | |
| 156 | const mine = held(ctx); | |
| 157 | if (mine === undefined) return notFound(); | |
| 158 | const input = await boundedBody(ctx.request, REPORT_MAX); | |
| 159 | if (input === "too_large") | |
| 160 | return apiError("too_large", 413, `at most ${REPORT_MAX} bytes`); | |
| 161 | const lines = Array.isArray(input.lines) | |
| 162 | ? input.lines | |
| 163 | .filter((l): l is string => typeof l === "string") | |
| 164 | .slice(0, LINES_MAX) | |
| 165 | : []; | |
| 166 | if (lines.length > 0) mine.ear.lines(str(input.step), lines); | |
| 167 | if (typeof input.abandon === "string") { | |
| 168 | settleJob(ctx.app.db, mine.job.id, { returned: input.abandon }); | |
| 169 | mine.ear.wake(); | |
| 170 | return json({ ended: true }); | |
| 171 | } | |
| 172 | if (input.done !== undefined) { | |
| 173 | const result = readResult(input.done); | |
| 174 | if (result === undefined) | |
| 175 | return apiError( | |
| 176 | "invalid_result", | |
| 177 | 422, | |
| 178 | "done takes { steps: [{ name, ok, error?, ms }], contained, note }", | |
| 179 | ); | |
| 180 | settleJob(ctx.app.db, mine.job.id, { done: result }); | |
| 181 | mine.ear.wake(); | |
| 182 | return json({ ended: true }); | |
| 183 | } | |
| 184 | if (!renewJob(ctx.app.db, mine.job.id, Date.now())) return notFound(); | |
| 185 | return json({ lease }); | |
| 186 | }); | |
| 187 | ||
| 188 | // The job's git door: a fetch of its repository, held to the commits the job names. | |
| 189 | router.add("GET", "/api/v1/runner/jobs/:id/git/info/refs", (ctx) => { | |
| 190 | const mine = held(ctx); | |
| 191 | if ( | |
| 192 | mine === undefined || | |
| 193 | ctx.url.searchParams.get("service") !== "git-upload-pack" | |
| 194 | ) | |
| 195 | return notFound(); | |
| 196 | return advertiseRefs(mine.job.dir, "git-upload-pack", ctx.request, { | |
| 197 | onlyWants: mine.job.wants, | |
| 198 | }); | |
| 199 | }); | |
| 200 | router.add("POST", "/api/v1/runner/jobs/:id/git/git-upload-pack", (ctx) => { | |
| 201 | const mine = held(ctx); | |
| 202 | if (mine === undefined) return notFound(); | |
| 203 | return serviceRpc(mine.job.dir, "git-upload-pack", ctx.request, { | |
| 204 | onlyWants: mine.job.wants, | |
| 205 | }); | |
| 206 | }); | |
| 207 | } |
src/http/route_marks.ts +2
| @@ -85,6 +85,8 @@ export const HEADS: Readonly<Record<string, Mark | "mixed">> = { | ||
| 85 | 85 | "/api/v1/releases": "open", |
| 86 | 86 | "/api/v1/repos": "open", |
| 87 | 87 | "/api/v1/returns": "open", |
| 88 | // A runner's own door: each route reads a runner's or a job's token, never an account. | |
| 89 | "/api/v1/runner": "open", | |
| 88 | 90 | "/api/v1/search": "open", |
| 89 | 91 | "/api/v1/secrets": "open", |
| 90 | 92 | "/api/v1/sessions": "open", |
src/runner/main.ts added +453
| @@ -0,0 +1,453 @@ | ||
| 1 | /** | |
| 2 | * A runner (#13652, R1): a process that takes checks off a forge. It needs two things of the | |
| 3 | * forge, its URL and a runner token, and nothing else: no shared disk, no database, no cluster. | |
| 4 | * | |
| 5 | * FORGE_URL=https://forge.example FORGE_RUNNER_TOKEN=forge_runner_… bun src/runner/main.ts | |
| 6 | * | |
| 7 | * It asks the forge for a job (a long poll), fetches the job's commit over the job's own git door | |
| 8 | * with the job's token, exports the tree as the forge's own checkout does, runs the recipe's steps | |
| 9 | * in the cell the forge uses (bubblewrap, where this machine has it), streams the log back as it | |
| 10 | * comes, reports each step's result and removes its scratch. Nothing dials in to a runner. | |
| 11 | * | |
| 12 | * What it keeps between jobs, under its own directory: one bare mirror and one package cache per | |
| 13 | * repository, so a second job fetches only new objects and installs from disk. | |
| 14 | * | |
| 15 | * FORGE_RUNNER_DIR where its scratch and store live. @default $TMPDIR/forge-runner | |
| 16 | * FORGE_RUNNER_JOBS how many jobs it runs at once. @default 1 | |
| 17 | * FORGE_RUNNER_ONCE `1`: one claim, then exit (a test, a cron line) | |
| 18 | * FORGE_CELLS `off`: never use cells here; jobs that require one go back to the forge | |
| 19 | */ | |
| 20 | ||
| 21 | import { existsSync, mkdirSync } from "node:fs"; | |
| 22 | import { rm } from "node:fs/promises"; | |
| 23 | import { tmpdir } from "node:os"; | |
| 24 | import { join } from "node:path"; | |
| 25 | import { exportTree } from "../apps/checkout.ts"; | |
| 26 | import { | |
| 27 | cellsAvailable, | |
| 28 | resolveLimits, | |
| 29 | type CellSpec, | |
| 30 | } from "../apps/contain.ts"; | |
| 31 | import { childEnv, runStep } from "../apps/process.ts"; | |
| 32 | import { scratchEnv } from "../checks/disk_full.ts"; | |
| 33 | import { giveHistory } from "../checks/history.ts"; | |
| 34 | import { cellInstall, prefillInstallCache } from "../checks/install.ts"; | |
| 35 | import type { | |
| 36 | Claim, | |
| 37 | RunnerStepResult, | |
| 38 | } from "../checks/runner_jobs.ts"; | |
| 39 | import { runGit } from "../git/env.ts"; | |
| 40 | import { isSafeOid } from "../git/guard.ts"; | |
| 41 | import { during } from "../perf/jobs.ts"; | |
| 42 | ||
| 43 | /** How often a job's lines go to the forge; each report renews the lease. */ | |
| 44 | const REPORT_MS = 1_000; | |
| 45 | /** The most lines one report carries. */ | |
| 46 | const REPORT_LINES = 2_000; | |
| 47 | /** How long a claim may wait at the forge, in seconds. */ | |
| 48 | const CLAIM_WAIT_S = 20; | |
| 49 | /** The pause after a claim the forge could not answer. */ | |
| 50 | const RETRY_MS = 5_000; | |
| 51 | const FETCH_MS = 600_000; | |
| 52 | ||
| 53 | export interface RunnerOptions { | |
| 54 | readonly forgeUrl: string; | |
| 55 | readonly token: string; | |
| 56 | /** Where the scratch and the store live. */ | |
| 57 | readonly root: string; | |
| 58 | readonly jobs?: number | undefined; | |
| 59 | readonly once?: boolean | undefined; | |
| 60 | /** Stops it: no new claim, and a job in flight is handed back. */ | |
| 61 | readonly signal?: AbortSignal | undefined; | |
| 62 | readonly say?: ((line: string) => void) | undefined; | |
| 63 | } | |
| 64 | ||
| 65 | const pause = (ms: number, signal: AbortSignal): Promise<void> => | |
| 66 | new Promise((resolve) => { | |
| 67 | if (signal.aborted) return resolve(); | |
| 68 | const done = (): void => { | |
| 69 | clearTimeout(timer); | |
| 70 | signal.removeEventListener("abort", done); | |
| 71 | resolve(); | |
| 72 | }; | |
| 73 | const timer = setTimeout(() => during("runner-wait", done), ms); | |
| 74 | signal.addEventListener("abort", done, { once: true }); | |
| 75 | }); | |
| 76 | ||
| 77 | /** Only the check's own variables: a job can name `CI` and `FORGE_*`, never `PATH` or a proxy. */ | |
| 78 | function checkVars( | |
| 79 | env: Readonly<Record<string, string>>, | |
| 80 | ): Record<string, string> { | |
| 81 | const out: Record<string, string> = {}; | |
| 82 | for (const [key, value] of Object.entries(env ?? {})) | |
| 83 | if ( | |
| 84 | (key === "CI" || /^FORGE_[A-Z0-9_]+$/.test(key)) && | |
| 85 | typeof value === "string" && | |
| 86 | !key.startsWith("FORGE_RUNNER") | |
| 87 | ) | |
| 88 | out[key] = value; | |
| 89 | return out; | |
| 90 | } | |
| 91 | ||
| 92 | /** A name of the forge's as a directory name of ours. */ | |
| 93 | const dirName = (raw: string): string => | |
| 94 | raw.replace(/[^A-Za-z0-9_-]/g, "").slice(0, 80) || "none"; | |
| 95 | ||
| 96 | export async function runRunner(options: RunnerOptions): Promise<void> { | |
| 97 | const say = options.say ?? ((line: string) => console.log(line)); | |
| 98 | const forge = options.forgeUrl.replace(/\/+$/, ""); | |
| 99 | const stopping = options.signal ?? new AbortController().signal; | |
| 100 | const jobsRoot = join(options.root, "jobs"); | |
| 101 | // What a runner that died mid-job left: no job is using it now. | |
| 102 | await rm(jobsRoot, { recursive: true, force: true }).catch(() => {}); | |
| 103 | mkdirSync(jobsRoot, { recursive: true }); | |
| 104 | const cells = await cellsAvailable(); | |
| 105 | say( | |
| 106 | cells.ok | |
| 107 | ? `runner for ${forge}: steps run in a cell` | |
| 108 | : `runner for ${forge}: no cells here (${cells.reason}); steps run as plain processes, and a job that requires a cell goes back to the forge`, | |
| 109 | ); | |
| 110 | /** One fetch at a time into a repository's mirror. */ | |
| 111 | const mirrors = new Map<string, Promise<unknown>>(); | |
| 112 | ||
| 113 | const post = ( | |
| 114 | path: string, | |
| 115 | token: string, | |
| 116 | body: unknown, | |
| 117 | signal?: AbortSignal, | |
| 118 | ): Promise<Response> => | |
| 119 | fetch(`${forge}${path}`, { | |
| 120 | method: "POST", | |
| 121 | headers: { | |
| 122 | authorization: `Bearer ${token}`, | |
| 123 | "content-type": "application/json", | |
| 124 | }, | |
| 125 | body: JSON.stringify(body), | |
| 126 | ...(signal === undefined ? {} : { signal }), | |
| 127 | }); | |
| 128 | ||
| 129 | async function work(claim: Pick<Claim, "job" | "token">): Promise<void> { | |
| 130 | const { job, token } = claim; | |
| 131 | const started = Date.now(); | |
| 132 | const jobDir = join(jobsRoot, dirName(job.id)); | |
| 133 | const dir = join(jobDir, "tree"); | |
| 134 | const home = `${dir}.home`; | |
| 135 | /** Aborted when the forge no longer wants the job, or this runner is stopping. */ | |
| 136 | const stop = new AbortController(); | |
| 137 | const onStopping = (): void => stop.abort(); | |
| 138 | stopping.addEventListener("abort", onStopping, { once: true }); | |
| 139 | if (stopping.aborted) stop.abort(); | |
| 140 | let dropped = false; | |
| 141 | const waiting: { step: string; lines: string[] }[] = []; | |
| 142 | const line = (step: string, text: string): void => { | |
| 143 | const last = waiting.at(-1); | |
| 144 | if ( | |
| 145 | last !== undefined && | |
| 146 | last.step === step && | |
| 147 | last.lines.length < REPORT_LINES | |
| 148 | ) | |
| 149 | last.lines.push(text); | |
| 150 | else waiting.push({ step, lines: [text] }); | |
| 151 | }; | |
| 152 | /** | |
| 153 | * Send what has been said, in order; with nothing to say, the report is the lease's renewal. | |
| 154 | */ | |
| 155 | const flush = async (): Promise<void> => { | |
| 156 | const batches = | |
| 157 | waiting.length === 0 | |
| 158 | ? [{ step: "", lines: [] as string[] }] | |
| 159 | : waiting.splice(0); | |
| 160 | for (const [i, batch] of batches.entries()) { | |
| 161 | try { | |
| 162 | const res = await post(`/api/v1/runner/jobs/${job.id}`, token, batch); | |
| 163 | if (res.status === 404) { | |
| 164 | dropped = true; | |
| 165 | stop.abort(); | |
| 166 | return; | |
| 167 | } | |
| 168 | if (!res.ok) throw new Error(`the forge answered ${res.status}`); | |
| 169 | } catch { | |
| 170 | // The forge could not be reached: these lines wait for the next report. | |
| 171 | waiting.unshift(...batches.slice(i)); | |
| 172 | return; | |
| 173 | } | |
| 174 | } | |
| 175 | }; | |
| 176 | let reporting = true; | |
| 177 | /** Ends the rest between two reports, so the last one is not kept waiting. */ | |
| 178 | const quiet = new AbortController(); | |
| 179 | const reports = (async () => { | |
| 180 | while (reporting) { | |
| 181 | await flush(); | |
| 182 | if (reporting) | |
| 183 | await pause(REPORT_MS, quiet.signal); | |
| 184 | } | |
| 185 | })(); | |
| 186 | const end = async (body: Record<string, unknown>): Promise<void> => { | |
| 187 | reporting = false; | |
| 188 | quiet.abort(); | |
| 189 | await reports; | |
| 190 | await flush(); | |
| 191 | if (dropped) return; | |
| 192 | // The answer is the job: a forge that did not hear it is told again, while the lease lasts. | |
| 193 | for (let tries = 0; tries < 5; tries++) { | |
| 194 | const res = await post(`/api/v1/runner/jobs/${job.id}`, token, body).catch(() => undefined); | |
| 195 | if (res !== undefined && (res.ok || res.status === 404 || res.status === 422)) return; | |
| 196 | await pause(REPORT_MS, new AbortController().signal); | |
| 197 | } | |
| 198 | }; | |
| 199 | try { | |
| 200 | if ( | |
| 201 | !isSafeOid(job.commit) || | |
| 202 | (job.base !== undefined && !isSafeOid(job.base)) | |
| 203 | ) | |
| 204 | return await end({ | |
| 205 | abandon: "the job names something that is not a commit id", | |
| 206 | }); | |
| 207 | if (job.cell === "required" && !cells.ok) | |
| 208 | return await end({ | |
| 209 | abandon: `no cells here (${cells.reason}), and this job requires one`, | |
| 210 | }); | |
| 211 | // The commit, over the job's own git door, into this repository's mirror. | |
| 212 | const mirror = join( | |
| 213 | options.root, | |
| 214 | "store", | |
| 215 | "git", | |
| 216 | `${dirName(job.cache.repo)}.git`, | |
| 217 | ); | |
| 218 | const auth = { | |
| 219 | GIT_CONFIG_COUNT: "1", | |
| 220 | GIT_CONFIG_KEY_0: "http.extraHeader", | |
| 221 | GIT_CONFIG_VALUE_0: `Authorization: Bearer ${token}`, | |
| 222 | }; | |
| 223 | const url = `${forge}/api/v1/runner/jobs/${job.id}/git`; | |
| 224 | const depth = Math.min( | |
| 225 | Math.max(Math.trunc(Number(job.depth)) || 1, 1), | |
| 226 | 500, | |
| 227 | ); | |
| 228 | const fetched = (mirrors.get(mirror) ?? Promise.resolve()).then( | |
| 229 | async (): Promise<string | undefined> => { | |
| 230 | if (!existsSync(join(mirror, "HEAD"))) { | |
| 231 | mkdirSync(mirror, { recursive: true }); | |
| 232 | const made = await runGit(["init", "--quiet", "--bare", mirror]); | |
| 233 | if (!made.ok) return `git init: ${made.error.stderr || "failed"}`; | |
| 234 | } | |
| 235 | for (const [oid, deep, kept] of [ | |
| 236 | ...(job.base === undefined ? [] : [[job.base, 1, "base"] as const]), | |
| 237 | [job.commit, depth, "commit"] as const, | |
| 238 | ]) { | |
| 239 | const got = await runGit( | |
| 240 | ["fetch", "--quiet", "--no-tags", `--depth=${deep}`, url, oid], | |
| 241 | { | |
| 242 | cwd: mirror, | |
| 243 | env: auth, | |
| 244 | timeoutMs: FETCH_MS, | |
| 245 | signal: stop.signal, | |
| 246 | }, | |
| 247 | ); | |
| 248 | if (!got.ok) | |
| 249 | return `git fetch ${oid.slice(0, 7)}: ${got.error.stderr.trim().slice(0, 300) || "failed"}`; | |
| 250 | // A fetch offers what its refs reach as already had: the last base and the last | |
| 251 | // commit are kept under a ref, so the next fetch brings only what is new. | |
| 252 | await runGit(["update-ref", `refs/runner/${kept}`, oid], { cwd: mirror }); | |
| 253 | } | |
| 254 | return undefined; | |
| 255 | }, | |
| 256 | ); | |
| 257 | mirrors.set( | |
| 258 | mirror, | |
| 259 | fetched.catch(() => {}), | |
| 260 | ); | |
| 261 | const refused = await fetched; | |
| 262 | if (refused !== undefined) return await end({ abandon: refused }); | |
| 263 | mkdirSync(jobDir, { recursive: true }); | |
| 264 | const exported = await exportTree(mirror, job.commit, dir); | |
| 265 | if (!exported.ok) return await end({ abandon: exported.error }); | |
| 266 | if (job.base !== undefined) { | |
| 267 | const given = await giveHistory(mirror, dir, job.base, job.commit); | |
| 268 | if (!given.ok) return await end({ abandon: given.error }); | |
| 269 | } | |
| 270 | line( | |
| 271 | "runner", | |
| 272 | `runner: ${job.commit.slice(0, 7)} fetched and checked out in ${((Date.now() - started) / 1000).toFixed(1)}s`, | |
| 273 | ); | |
| 274 | const limits = resolveLimits("step", job.limits); | |
| 275 | const cell: CellSpec | undefined = cells.ok | |
| 276 | ? { | |
| 277 | dir, | |
| 278 | writable: [], | |
| 279 | home, | |
| 280 | // The runner's own store and scratch, its home and the temp directory look empty | |
| 281 | // inside. | |
| 282 | hide: [options.root, process.env["HOME"] ?? "", tmpdir()].filter( | |
| 283 | (p) => p !== "", | |
| 284 | ), | |
| 285 | egress: (job.egress ?? []).filter((h) => typeof h === "string"), | |
| 286 | reach: [...new Set([forge, job.env?.["FORGE_URL"] ?? forge])], | |
| 287 | limits, | |
| 288 | egressMode: job.egressMode === "report" ? "report" : "enforce", | |
| 289 | label: `check:${job.run}`, | |
| 290 | onUndeclared: (host, port) => | |
| 291 | line( | |
| 292 | "runner", | |
| 293 | `runner: reached ${host}:${port}, which forge.json does not declare`, | |
| 294 | ), | |
| 295 | } | |
| 296 | : undefined; | |
| 297 | if (cell === undefined) | |
| 298 | line( | |
| 299 | "runner", | |
| 300 | `uncontained on this runner: ${cells.ok ? "" : cells.reason}; the check runs unrestricted`, | |
| 301 | ); | |
| 302 | const env = childEnv({ | |
| 303 | ...checkVars(job.env), | |
| 304 | FORGE_CELL_CPUS: String(limits.cpus), | |
| 305 | ...scratchEnv(cell === undefined ? undefined : home), | |
| 306 | }); | |
| 307 | const timeoutMs = | |
| 308 | Math.min(Math.max(Number(job.timeoutS) || 900, 1), 7_200) * 1000; | |
| 309 | const warm = join( | |
| 310 | options.root, | |
| 311 | "store", | |
| 312 | "install", | |
| 313 | dirName(job.cache.repo), | |
| 314 | ); | |
| 315 | const steps: RunnerStepResult[] = []; | |
| 316 | for (const step of job.steps) { | |
| 317 | if (step.name !== "install" && step.name !== "check") continue; | |
| 318 | const from = Date.now(); | |
| 319 | const log = (text: string): void => line(step.name, text); | |
| 320 | let ran; | |
| 321 | if (step.name === "install" && cell !== undefined) { | |
| 322 | // As on the forge: this machine fills the repository's cache without running scripts, | |
| 323 | // and the cell installs from it read-only, so no job writes what the next one installs. | |
| 324 | const fill = await prefillInstallCache({ | |
| 325 | cache: warm, | |
| 326 | dir, | |
| 327 | env, | |
| 328 | log, | |
| 329 | signal: stop.signal, | |
| 330 | timeoutMs, | |
| 331 | }); | |
| 332 | const filled = fill.ok && fill.value; | |
| 333 | if (!fill.ok) log(`cache not filled: ${fill.error}`); | |
| 334 | const use = cellInstall(step.command, env, filled ? warm : undefined); | |
| 335 | ran = await runStep("install", use.command, { | |
| 336 | cwd: dir, | |
| 337 | env: use.env, | |
| 338 | log, | |
| 339 | signal: stop.signal, | |
| 340 | timeoutMs, | |
| 341 | cell: filled ? { ...cell, reveal: [warm] } : cell, | |
| 342 | }); | |
| 343 | } else { | |
| 344 | ran = await runStep(step.name, step.command, { | |
| 345 | cwd: dir, | |
| 346 | // Uncontained, the repository's package cache is shared between its jobs. | |
| 347 | env: | |
| 348 | step.name === "install" | |
| 349 | ? { ...env, BUN_INSTALL_CACHE_DIR: warm } | |
| 350 | : env, | |
| 351 | log, | |
| 352 | signal: stop.signal, | |
| 353 | timeoutMs, | |
| 354 | cell, | |
| 355 | }); | |
| 356 | } | |
| 357 | steps.push({ | |
| 358 | name: step.name, | |
| 359 | ok: ran.ok, | |
| 360 | ...(ran.ok ? {} : { error: ran.error }), | |
| 361 | ms: Date.now() - from, | |
| 362 | }); | |
| 363 | if (!ran.ok) break; | |
| 364 | } | |
| 365 | if (stopping.aborted && !dropped) | |
| 366 | return await end({ abandon: "the runner is stopping" }); | |
| 367 | await end({ | |
| 368 | done: { | |
| 369 | steps, | |
| 370 | contained: cell !== undefined, | |
| 371 | note: cells.ok ? "" : cells.reason, | |
| 372 | }, | |
| 373 | }); | |
| 374 | const failed = steps.find((s) => !s.ok); | |
| 375 | say( | |
| 376 | `job ${job.id} (${job.repo}@${job.commit.slice(0, 7)}): ${dropped ? "dropped by the forge" : failed === undefined ? "ok" : `${failed.name} failed`} in ${((Date.now() - started) / 1000).toFixed(1)}s`, | |
| 377 | ); | |
| 378 | } catch (error) { | |
| 379 | await end({ | |
| 380 | abandon: `the runner failed: ${error instanceof Error ? error.message : String(error)}`, | |
| 381 | }); | |
| 382 | say(`job ${job.id}: ${String(error)}`); | |
| 383 | } finally { | |
| 384 | reporting = false; | |
| 385 | quiet.abort(); | |
| 386 | stopping.removeEventListener("abort", onStopping); | |
| 387 | await rm(jobDir, { recursive: true, force: true }).catch((error) => | |
| 388 | say(`scratch not removed: ${String(error)}`), | |
| 389 | ); | |
| 390 | } | |
| 391 | } | |
| 392 | ||
| 393 | async function loop(): Promise<void> { | |
| 394 | while (!stopping.aborted) { | |
| 395 | let res: Response; | |
| 396 | try { | |
| 397 | res = await post( | |
| 398 | "/api/v1/runner/claim", | |
| 399 | options.token, | |
| 400 | { wait: options.once === true ? 0 : CLAIM_WAIT_S, cells: cells.ok }, | |
| 401 | stopping, | |
| 402 | ); | |
| 403 | } catch { | |
| 404 | if (stopping.aborted || options.once === true) return; | |
| 405 | await pause(RETRY_MS, stopping); | |
| 406 | continue; | |
| 407 | } | |
| 408 | if (res.status === 401) | |
| 409 | throw new Error( | |
| 410 | "the forge does not know this runner token (removed, or mistyped)", | |
| 411 | ); | |
| 412 | if (res.status === 200) | |
| 413 | await work((await res.json()) as Pick<Claim, "job" | "token">); | |
| 414 | else { | |
| 415 | await res.body?.cancel().catch(() => {}); | |
| 416 | if (res.status !== 204) await pause(RETRY_MS, stopping); | |
| 417 | } | |
| 418 | if (options.once === true) return; | |
| 419 | } | |
| 420 | } | |
| 421 | ||
| 422 | const n = Math.min(Math.max(Math.trunc(options.jobs ?? 1), 1), 64); | |
| 423 | await Promise.all(Array.from({ length: n }, () => loop())); | |
| 424 | } | |
| 425 | ||
| 426 | if (import.meta.main) { | |
| 427 | const forgeUrl = process.env["FORGE_URL"] ?? ""; | |
| 428 | const token = process.env["FORGE_RUNNER_TOKEN"] ?? ""; | |
| 429 | if (!/^https?:\/\//.test(forgeUrl) || token === "") { | |
| 430 | console.error( | |
| 431 | "usage: FORGE_URL=https://your.forge FORGE_RUNNER_TOKEN=forge_runner_… bun src/runner/main.ts", | |
| 432 | ); | |
| 433 | process.exit(2); | |
| 434 | } | |
| 435 | // The token is this process's alone: nothing it starts may read it from the environment. | |
| 436 | delete process.env["FORGE_RUNNER_TOKEN"]; | |
| 437 | const stopping = new AbortController(); | |
| 438 | for (const sig of ["SIGINT", "SIGTERM"] as const) | |
| 439 | process.on(sig, () => stopping.abort()); | |
| 440 | try { | |
| 441 | await runRunner({ | |
| 442 | forgeUrl, | |
| 443 | token, | |
| 444 | root: process.env["FORGE_RUNNER_DIR"] ?? join(tmpdir(), "forge-runner"), | |
| 445 | jobs: Number(process.env["FORGE_RUNNER_JOBS"] ?? "1") || 1, | |
| 446 | once: process.env["FORGE_RUNNER_ONCE"] === "1", | |
| 447 | signal: stopping.signal, | |
| 448 | }); | |
| 449 | } catch (error) { | |
| 450 | console.error(error instanceof Error ? error.message : String(error)); | |
| 451 | process.exit(1); | |
| 452 | } | |
| 453 | } |
test/checks-runner-placement.test.ts added +592
| @@ -0,0 +1,592 @@ | ||
| 1 | /** | |
| 2 | * `FORGE_CHECKS_PLACEMENT=runner` (#13652, R1): a pull's check as a job a runner claims, runs and | |
| 3 | * reports on, with nothing of the forge's but its URL and a token. The runner here is the real | |
| 4 | * program, a separate process. A silent runner and a stolen job token are played over HTTP. | |
| 5 | */ | |
| 6 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 7 | import { mkdirSync } from "node:fs"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import { openPull } from "../src/actions/pulls.ts"; | |
| 10 | import { commitToBranch, createRepo } from "../src/actions/repos.ts"; | |
| 11 | import { placementFromEnv, placementWords } from "../src/checks/placement.ts"; | |
| 12 | import { | |
| 13 | LEASE_MS, | |
| 14 | claimJob, | |
| 15 | deskOf, | |
| 16 | dropJob, | |
| 17 | enrolRunner, | |
| 18 | getRunner, | |
| 19 | jobByToken, | |
| 20 | lapseJobs, | |
| 21 | offerJob, | |
| 22 | renewJob, | |
| 23 | viewJob, | |
| 24 | withdrawJob, | |
| 25 | type RunnerJob, | |
| 26 | } from "../src/checks/runner_jobs.ts"; | |
| 27 | import { readSteps, runnerTakes, runnerVerdict } from "../src/checks/runner_step.ts"; | |
| 28 | import { startChecks } from "../src/checks/service.ts"; | |
| 29 | import { createToken } from "../src/core/tokens.ts"; | |
| 30 | import { ok, err } from "../src/kernel/result.ts"; | |
| 31 | import { readRunLog } from "../src/runs/log.ts"; | |
| 32 | import { checkJson, checksForge } from "./helpers/checks-forge.ts"; | |
| 33 | import { gitClientEnv } from "./helpers/git-client.ts"; | |
| 34 | import { quietLogs } from "./helpers/globals.ts"; | |
| 35 | ||
| 36 | quietLogs(); | |
| 37 | ||
| 38 | const f = checksForge("forge-runner-"); | |
| 39 | /** The check every pull of `lib` gets from main: green unless the pull adds a file named RED. */ | |
| 40 | const CHECK = "echo ran-the-check; test ! -f RED"; | |
| 41 | ||
| 42 | let adminToken = ""; | |
| 43 | const api = (path: string, token: string, body?: unknown): Promise<Response> => | |
| 44 | fetch(`${f.servers.url}${path}`, { | |
| 45 | method: body === undefined ? "GET" : "POST", | |
| 46 | headers: { | |
| 47 | authorization: `Bearer ${token}`, | |
| 48 | "content-type": "application/json", | |
| 49 | }, | |
| 50 | ...(body === undefined ? {} : { body: JSON.stringify(body) }), | |
| 51 | }); | |
| 52 | ||
| 53 | /** A pull of `lib` from a new branch `name` that adds `file`; its head commit. */ | |
| 54 | async function pull(name: string, file: string): Promise<string> { | |
| 55 | const made = await commitToBranch(f.app, f.alice, f.repo, { | |
| 56 | branch: name, | |
| 57 | message: `Add ${file}`, | |
| 58 | files: [{ path: file, content: `${name}\n` }], | |
| 59 | }); | |
| 60 | if (!made.ok) throw new Error(JSON.stringify(made.error)); | |
| 61 | const opened = await openPull(f.app, f.alice, f.repo, { | |
| 62 | title: name, | |
| 63 | body: "", | |
| 64 | baseBranch: "main", | |
| 65 | headBranch: name, | |
| 66 | }); | |
| 67 | if (!opened.ok) throw new Error(JSON.stringify(opened.error)); | |
| 68 | return made.value.commitOid; | |
| 69 | } | |
| 70 | ||
| 71 | const logOf = (runId: string): string => | |
| 72 | readRunLog(f.app.config.dataDir, runId).lines.join("\n"); | |
| 73 | ||
| 74 | const job: Omit<RunnerJob, "id"> = { | |
| 75 | run: "run-1", | |
| 76 | repo: "alice/lib", | |
| 77 | commit: "a".repeat(40), | |
| 78 | base: "b".repeat(40), | |
| 79 | depth: 3, | |
| 80 | steps: [ | |
| 81 | { name: "install", command: "bun install" }, | |
| 82 | { name: "check", command: "bun test" }, | |
| 83 | ], | |
| 84 | env: { CI: "1", FORGE_CHECK: "run-1" }, | |
| 85 | limits: { cpus: 2, memoryMiB: 2048 }, | |
| 86 | timeoutS: 900, | |
| 87 | egress: ["registry.npmjs.org"], | |
| 88 | egressMode: "enforce", | |
| 89 | cell: "preferred", | |
| 90 | cache: { repo: "repo-1", tree: "c".repeat(40), base: "b".repeat(40) }, | |
| 91 | }; | |
| 92 | ||
| 93 | beforeAll(async () => { | |
| 94 | // The forge under test offers its checks to runners, and waits only briefly for one. | |
| 95 | await f.checks.stop(); | |
| 96 | f.checks = startChecks(f.app, { | |
| 97 | forgeUrl: f.servers.url, | |
| 98 | concurrency: 1, | |
| 99 | placement: { mode: "runner", waitMs: 300 }, | |
| 100 | }); | |
| 101 | adminToken = createToken(f.app.db, f.alice.id, "admin", "admin").token; | |
| 102 | const gate = await commitToBranch(f.app, f.alice, f.repo, { | |
| 103 | message: "Declare the check", | |
| 104 | files: [{ path: "forge.json", content: checkJson(CHECK) }], | |
| 105 | }); | |
| 106 | if (!gate.ok) throw new Error(JSON.stringify(gate.error)); | |
| 107 | await f.settled((r) => r.commitOid === gate.value.commitOid, 30_000); | |
| 108 | }); | |
| 109 | ||
| 110 | describe("the setting", () => { | |
| 111 | test("runner is a placement only when asked for, and a run says where it ran", () => { | |
| 112 | expect(placementFromEnv({})).toEqual({ mode: "process" }); | |
| 113 | expect(placementFromEnv({ FORGE_CHECKS_PLACEMENT: "runner" })).toEqual({ | |
| 114 | mode: "runner", | |
| 115 | }); | |
| 116 | expect(placementFromEnv({ FORGE_CHECKS_PLACEMENT: "runners" })).toEqual({ | |
| 117 | mode: "process", | |
| 118 | }); | |
| 119 | expect(placementWords({ placement: "runner", node: "mac" })).toBe( | |
| 120 | "ran on runner mac", | |
| 121 | ); | |
| 122 | expect(placementWords({ placement: "process", node: "" })).toBe(""); | |
| 123 | }); | |
| 124 | ||
| 125 | test("only a check that hands back nothing but its verdict goes to a runner", () => { | |
| 126 | expect( | |
| 127 | runnerTakes({ | |
| 128 | tier: "fast", | |
| 129 | keep: false, | |
| 130 | keepImpact: false, | |
| 131 | isMain: false, | |
| 132 | }), | |
| 133 | ).toBe(true); | |
| 134 | expect( | |
| 135 | runnerTakes({ tier: "run", keep: true, keepImpact: true, isMain: true }), | |
| 136 | ).toBe(false); | |
| 137 | expect(runnerTakes({ tier: "verify", keep: false, keepImpact: true })).toBe( | |
| 138 | false, | |
| 139 | ); | |
| 140 | expect(runnerTakes({ tier: "full" })).toBe(false); | |
| 141 | expect(runnerTakes({ tier: "fast", only: ["a.test.ts"] })).toBe(false); | |
| 142 | }); | |
| 143 | ||
| 144 | test("a result is held to the steps the job asked for: an empty or partial one is no answer", () => { | |
| 145 | const asked = job.steps; | |
| 146 | const yes = (name: string) => ({ name, ok: true, ms: 1 }); | |
| 147 | const no = (name: string) => ({ name, ok: false, error: `${name} exited with code 1`, ms: 1 }); | |
| 148 | expect(readSteps(asked, [yes("install"), yes("check")])).toEqual({ ran: ok() }); | |
| 149 | expect(readSteps(asked, [yes("install"), no("check")])).toEqual({ ran: err("check exited with code 1"), failed: "check" }); | |
| 150 | expect(readSteps(asked, [no("install")])).toEqual({ ran: err("install exited with code 1"), failed: "install" }); | |
| 151 | // Never green: nothing run, the check left out, a step of its own, a step after a failure. | |
| 152 | expect(readSteps(asked, [])).toBeUndefined(); | |
| 153 | expect(readSteps(asked, [yes("install")])).toBeUndefined(); | |
| 154 | expect(readSteps(asked, [yes("check")])).toBeUndefined(); | |
| 155 | expect(readSteps(asked, [yes("install"), yes("lint")])).toBeUndefined(); | |
| 156 | expect(readSteps(asked, [no("install"), yes("check")])).toBeUndefined(); | |
| 157 | expect(readSteps(asked, [yes("install"), yes("check"), yes("check")])).toBeUndefined(); | |
| 158 | }); | |
| 159 | ||
| 160 | test("a runner's answer stands only as green, the check's exit code or its timeout", () => { | |
| 161 | const end = { | |
| 162 | taken: true as const, | |
| 163 | runner: "mac", | |
| 164 | contained: true, | |
| 165 | note: "", | |
| 166 | }; | |
| 167 | expect(runnerVerdict({ ...end, ran: ok() }, [], false)).toEqual({ | |
| 168 | ok: true, | |
| 169 | }); | |
| 170 | expect( | |
| 171 | runnerVerdict( | |
| 172 | { ...end, ran: err("check exited with code 1"), failed: "check" }, | |
| 173 | ["boom"], | |
| 174 | false, | |
| 175 | ), | |
| 176 | ).toMatchObject({ ok: false, stage: "check" }); | |
| 177 | expect( | |
| 178 | runnerVerdict( | |
| 179 | { ...end, ran: err("check took longer than 900s"), failed: "check" }, | |
| 180 | [], | |
| 181 | false, | |
| 182 | ), | |
| 183 | ).toMatchObject({ ok: false, stage: "timeout" }); | |
| 184 | // The machine's, not the check's: the forge runs it itself. | |
| 185 | expect( | |
| 186 | typeof runnerVerdict( | |
| 187 | { ...end, ran: err("install exited with code 1"), failed: "install" }, | |
| 188 | [], | |
| 189 | false, | |
| 190 | ), | |
| 191 | ).toBe("string"); | |
| 192 | expect( | |
| 193 | typeof runnerVerdict( | |
| 194 | { ...end, ran: err("check was killed by SIGKILL"), failed: "check" }, | |
| 195 | [], | |
| 196 | false, | |
| 197 | ), | |
| 198 | ).toBe("string"); | |
| 199 | expect( | |
| 200 | typeof runnerVerdict( | |
| 201 | { | |
| 202 | ...end, | |
| 203 | ran: err("check stopped: it used more memory than its limit"), | |
| 204 | failed: "check", | |
| 205 | }, | |
| 206 | [], | |
| 207 | false, | |
| 208 | ), | |
| 209 | ).toBe("string"); | |
| 210 | }); | |
| 211 | }); | |
| 212 | ||
| 213 | describe("the desk", () => { | |
| 214 | test("a job survives the wire and the desk unchanged, and names nothing of the forge's disk", () => { | |
| 215 | const runner = enrolRunner(f.app.db, "wire", f.alice.id).runner; | |
| 216 | const offered = offerJob(f.app.db, { | |
| 217 | runId: "run-1", | |
| 218 | repoId: "repo-1", | |
| 219 | dir: "/data/repos/alice/lib.git", | |
| 220 | job, | |
| 221 | }); | |
| 222 | expect(JSON.parse(JSON.stringify(offered))).toEqual(offered); | |
| 223 | expect(offered).toEqual({ ...job, id: offered.id }); | |
| 224 | const claim = claimJob(f.app.db, { id: runner.id, cells: true }, 1_000); | |
| 225 | expect(claim?.job).toEqual(offered); | |
| 226 | expect(JSON.stringify(claim)).not.toContain("/data/repos"); | |
| 227 | expect(claim?.token).toStartWith("forge_job_"); | |
| 228 | expect(withdrawJob(f.app.db, offered.id)).toBe(false); | |
| 229 | dropJob(f.app.db, offered.id); | |
| 230 | }); | |
| 231 | ||
| 232 | test("of two runners asking at once, one takes the job; a job that needs a cell goes to one that has them", () => { | |
| 233 | const one = enrolRunner(f.app.db, "one", f.alice.id).runner; | |
| 234 | const two = enrolRunner(f.app.db, "two", f.alice.id).runner; | |
| 235 | const offered = offerJob(f.app.db, { | |
| 236 | runId: "run-2", | |
| 237 | repoId: "repo-1", | |
| 238 | dir: "/x", | |
| 239 | job: { ...job, cell: "required" }, | |
| 240 | }); | |
| 241 | expect( | |
| 242 | claimJob(f.app.db, { id: one.id, cells: false }, 1_000), | |
| 243 | ).toBeUndefined(); | |
| 244 | const first = claimJob(f.app.db, { id: one.id, cells: true }, 1_000); | |
| 245 | const second = claimJob(f.app.db, { id: two.id, cells: true }, 1_000); | |
| 246 | expect(first?.job.id).toBe(offered.id); | |
| 247 | expect(second).toBeUndefined(); | |
| 248 | expect(viewJob(f.app.db, offered.id)).toMatchObject({ | |
| 249 | state: "claimed", | |
| 250 | claims: 1, | |
| 251 | runner: "one", | |
| 252 | }); | |
| 253 | dropJob(f.app.db, offered.id); | |
| 254 | }); | |
| 255 | ||
| 256 | test("a lease that runs out queues the job again: the old token is dead and another runner takes it", () => { | |
| 257 | const one = enrolRunner(f.app.db, "sleepy", f.alice.id).runner; | |
| 258 | const two = enrolRunner(f.app.db, "awake", f.alice.id).runner; | |
| 259 | const offered = offerJob(f.app.db, { | |
| 260 | runId: "run-3", | |
| 261 | repoId: "repo-1", | |
| 262 | dir: "/x", | |
| 263 | job, | |
| 264 | }); | |
| 265 | const first = claimJob(f.app.db, { id: one.id, cells: true }, 1_000)!; | |
| 266 | expect(jobByToken(f.app.db, offered.id, first.token, 1_001)?.wants).toEqual( | |
| 267 | [job.commit, job.base!], | |
| 268 | ); | |
| 269 | // A runner at work is connected, though it is not asking for more. | |
| 270 | expect(getRunner(f.app.db, one.id)?.seenAt).toBe(1_001); | |
| 271 | // A report in time pushes the lease out; silence past it loses the job. | |
| 272 | expect(renewJob(f.app.db, offered.id, 1_000 + LEASE_MS - 1)).toBe(true); | |
| 273 | expect(lapseJobs(f.app.db, 1_000 + LEASE_MS)).not.toContain(offered.id); | |
| 274 | expect(lapseJobs(f.app.db, 1_000 + 2 * LEASE_MS)).toContain(offered.id); | |
| 275 | expect(viewJob(f.app.db, offered.id)).toMatchObject({ | |
| 276 | state: "queued", | |
| 277 | runner: "", | |
| 278 | }); | |
| 279 | expect( | |
| 280 | jobByToken(f.app.db, offered.id, first.token, 1_000 + 2 * LEASE_MS), | |
| 281 | ).toBeUndefined(); | |
| 282 | expect(renewJob(f.app.db, offered.id, 1_000 + 2 * LEASE_MS)).toBe(false); | |
| 283 | const second = claimJob( | |
| 284 | f.app.db, | |
| 285 | { id: two.id, cells: true }, | |
| 286 | 1_000 + 2 * LEASE_MS, | |
| 287 | )!; | |
| 288 | expect(second.job.id).toBe(offered.id); | |
| 289 | expect(second.token).not.toBe(first.token); | |
| 290 | expect(withdrawJob(f.app.db, offered.id)).toBe(false); | |
| 291 | dropJob(f.app.db, offered.id); | |
| 292 | }); | |
| 293 | }); | |
| 294 | ||
| 295 | describe("a pull's check with FORGE_CHECKS_PLACEMENT=runner", () => { | |
| 296 | test("with no runner connected the forge runs it itself, at once", async () => { | |
| 297 | // The runners enrolled above never asked for work: none is connected. | |
| 298 | const oid = await pull("alone", "a.txt"); | |
| 299 | const run = await f.settled((r) => r.commitOid === oid, 30_000); | |
| 300 | expect(run.state).toBe("ok"); | |
| 301 | expect(run.placement).toBe("process"); | |
| 302 | expect(logOf(run.id)).toContain( | |
| 303 | "placement: runner; no runner is connected; running here", | |
| 304 | ); | |
| 305 | }); | |
| 306 | ||
| 307 | test("a runner that goes silent loses the job, which is no try of the check; nobody else takes it and the forge runs it", async () => { | |
| 308 | const silent = enrolRunner(f.app.db, "silent", f.alice.id); | |
| 309 | // It asks first and is held open, as a runner is; then the pull is opened. | |
| 310 | const asked = api("/api/v1/runner/claim", silent.token, { | |
| 311 | wait: 20, | |
| 312 | cells: true, | |
| 313 | }); | |
| 314 | await f.until( | |
| 315 | () => deskOf(f.app).claimers.size > 0, | |
| 316 | "the runner waiting in its claim", | |
| 317 | ); | |
| 318 | const oid = await pull("silent", "s.txt"); | |
| 319 | const claim = (await (await asked).json()) as { | |
| 320 | job: RunnerJob; | |
| 321 | token: string; | |
| 322 | lease: { seconds: number }; | |
| 323 | }; | |
| 324 | expect(claim.job.commit).not.toBe(oid); | |
| 325 | expect(claim.job.steps).toEqual([{ name: "check", command: CHECK }]); | |
| 326 | expect(claim.lease.seconds).toBe(LEASE_MS / 1000); | |
| 327 | // Its lease runs out with no report (the clock, moved by hand). | |
| 328 | expect(lapseJobs(f.app.db, Date.now() + LEASE_MS + 1)).toEqual([ | |
| 329 | claim.job.id, | |
| 330 | ]); | |
| 331 | deskOf(f.app).ears.get(claim.job.id)?.wake(); | |
| 332 | expect( | |
| 333 | ( | |
| 334 | await api(`/api/v1/runner/jobs/${claim.job.id}`, claim.token, { | |
| 335 | step: "check", | |
| 336 | lines: ["late"], | |
| 337 | }) | |
| 338 | ).status, | |
| 339 | ).toBe(404); | |
| 340 | const run = await f.settled((r) => r.commitOid === oid, 30_000); | |
| 341 | expect(run.state).toBe("ok"); | |
| 342 | expect(run.attempt).toBe(0); | |
| 343 | expect(f.checksOf(oid).length).toBe(1); | |
| 344 | const log = logOf(run.id); | |
| 345 | expect(log).toContain("placement: runner silent"); | |
| 346 | expect(log).toContain( | |
| 347 | "runner silent went silent: the job is queued again, and that is not a try", | |
| 348 | ); | |
| 349 | expect(log).toContain("no runner took it"); | |
| 350 | expect(log).not.toContain("late"); | |
| 351 | }); | |
| 352 | ||
| 353 | test("a job token reaches its own job and nothing else: no other repository, no other object, no other job, no claim", async () => { | |
| 354 | const secret = await createRepo(f.app, f.alice, { | |
| 355 | name: "secret", | |
| 356 | visibility: "private", | |
| 357 | }); | |
| 358 | if (!secret.ok) throw new Error(String(secret.error)); | |
| 359 | const hidden = await commitToBranch(f.app, f.alice, secret.value, { | |
| 360 | message: "Seed", | |
| 361 | files: [{ path: "S.md", content: "s\n" }], | |
| 362 | }); | |
| 363 | if (!hidden.ok) throw new Error(JSON.stringify(hidden.error)); | |
| 364 | const thief = enrolRunner(f.app.db, "thief", f.alice.id); | |
| 365 | const asked = api("/api/v1/runner/claim", thief.token, { | |
| 366 | wait: 20, | |
| 367 | cells: true, | |
| 368 | }); | |
| 369 | await f.until( | |
| 370 | () => deskOf(f.app).claimers.size > 0, | |
| 371 | "the runner waiting in its claim", | |
| 372 | ); | |
| 373 | const oid = await pull("stolen", "t.txt"); | |
| 374 | const claim = (await (await asked).json()) as { | |
| 375 | job: RunnerJob; | |
| 376 | token: string; | |
| 377 | }; | |
| 378 | const door = `/api/v1/runner/jobs/${claim.job.id}`; | |
| 379 | const git = async ( | |
| 380 | url: string, | |
| 381 | want: string, | |
| 382 | ): Promise<{ code: number; err: string }> => { | |
| 383 | const dir = join( | |
| 384 | f.dataDir, | |
| 385 | `fetch-${Math.random().toString(36).slice(2)}`, | |
| 386 | ); | |
| 387 | mkdirSync(dir); | |
| 388 | const env = gitClientEnv({ | |
| 389 | GIT_CONFIG_COUNT: "1", | |
| 390 | GIT_CONFIG_KEY_0: "http.extraHeader", | |
| 391 | GIT_CONFIG_VALUE_0: `Authorization: Bearer ${claim.token}`, | |
| 392 | }); | |
| 393 | const init = Bun.spawn(["git", "init", "--quiet", "--bare", dir], { | |
| 394 | env, | |
| 395 | stdout: "ignore", | |
| 396 | stderr: "ignore", | |
| 397 | }); | |
| 398 | await init.exited; | |
| 399 | const proc = Bun.spawn( | |
| 400 | ["git", "fetch", "--quiet", "--no-tags", "--depth=1", url, want], | |
| 401 | { cwd: dir, env, stdout: "ignore", stderr: "pipe" }, | |
| 402 | ); | |
| 403 | const [text, code] = await Promise.all([ | |
| 404 | new Response(proc.stderr).text(), | |
| 405 | proc.exited, | |
| 406 | ]); | |
| 407 | return { code, err: text }; | |
| 408 | }; | |
| 409 | // Its own commit, over its own door: yes. | |
| 410 | expect( | |
| 411 | (await git(`${f.servers.url}${door}/git`, claim.job.commit)).code, | |
| 412 | ).toBe(0); | |
| 413 | // Another object of the same repository (the pull's own head, which is not what the job names): | |
| 414 | // no. | |
| 415 | const other = await git(`${f.servers.url}${door}/git`, oid); | |
| 416 | expect(other.code).not.toBe(0); | |
| 417 | expect(other.err).toContain("fetches only the commits its job names"); | |
| 418 | // Another repository, by its own git door or through this job's: no. | |
| 419 | expect( | |
| 420 | (await git(`${f.servers.url}/alice/secret.git`, hidden.value.commitOid)) | |
| 421 | .code, | |
| 422 | ).not.toBe(0); | |
| 423 | expect( | |
| 424 | (await git(`${f.servers.url}${door}/git`, hidden.value.commitOid)).code, | |
| 425 | ).not.toBe(0); | |
| 426 | // The API, the account, another job, a claim: it is no credential there. | |
| 427 | expect((await api("/api/v1/repos/alice/secret", claim.token)).status).toBe( | |
| 428 | 404, | |
| 429 | ); | |
| 430 | expect((await api("/api/v1/user", claim.token)).status).toBe(401); | |
| 431 | expect( | |
| 432 | (await api("/api/v1/runner/claim", claim.token, { wait: 0 })).status, | |
| 433 | ).toBe(401); | |
| 434 | expect( | |
| 435 | ( | |
| 436 | await api("/api/v1/runner/jobs/01NOTAJOB", claim.token, { | |
| 437 | lines: ["x"], | |
| 438 | }) | |
| 439 | ).status, | |
| 440 | ).toBe(404); | |
| 441 | const elsewhere = offerJob(f.app.db, { | |
| 442 | runId: "run-9", | |
| 443 | repoId: "repo-9", | |
| 444 | dir: "/x", | |
| 445 | job, | |
| 446 | }); | |
| 447 | expect( | |
| 448 | ( | |
| 449 | await api(`/api/v1/runner/jobs/${elsewhere.id}`, claim.token, { | |
| 450 | lines: ["x"], | |
| 451 | }) | |
| 452 | ).status, | |
| 453 | ).toBe(404); | |
| 454 | expect(withdrawJob(f.app.db, elsewhere.id)).toBe(true); | |
| 455 | // Nor is a runner token, or an account's, a job's. | |
| 456 | expect((await api(door, thief.token, { lines: ["x"] })).status).toBe(404); | |
| 457 | expect((await api(door, adminToken, { lines: ["x"] })).status).toBe(404); | |
| 458 | expect( | |
| 459 | (await api("/api/v1/runner/claim", adminToken, { wait: 0 })).status, | |
| 460 | ).toBe(401); | |
| 461 | expect((await api("/api/v1/user", thief.token)).status).toBe(401); | |
| 462 | // Handed back, the forge runs it. | |
| 463 | expect( | |
| 464 | (await api(door, claim.token, { abandon: "done looking" })).status, | |
| 465 | ).toBe(200); | |
| 466 | const run = await f.settled((r) => r.commitOid === oid, 30_000); | |
| 467 | expect(run.state).toBe("ok"); | |
| 468 | expect(logOf(run.id)).toContain( | |
| 469 | "runner thief handed it back: done looking; running here", | |
| 470 | ); | |
| 471 | expect((await api(door, claim.token, { lines: ["x"] })).status).toBe(404); | |
| 472 | }); | |
| 473 | ||
| 474 | test("a runner that says done with no steps does not make the check green: the forge runs it", async () => { | |
| 475 | const lazy = enrolRunner(f.app.db, "lazy", f.alice.id); | |
| 476 | const asked = api("/api/v1/runner/claim", lazy.token, { wait: 20, cells: true }); | |
| 477 | await f.until(() => deskOf(f.app).claimers.size > 0, "the runner waiting in its claim"); | |
| 478 | const oid = await pull("lazy", "l.txt"); | |
| 479 | const claim = (await (await asked).json()) as { job: RunnerJob; token: string }; | |
| 480 | const done = await api(`/api/v1/runner/jobs/${claim.job.id}`, claim.token, { done: { steps: [], contained: true, note: "" } }); | |
| 481 | expect(done.status).toBe(200); | |
| 482 | const run = await f.settled((r) => r.commitOid === oid, 30_000); | |
| 483 | expect(run.state).toBe("ok"); | |
| 484 | // Green because this machine ran the command, not because the runner said so. | |
| 485 | expect(run.placement).toBe("process"); | |
| 486 | const log = logOf(run.id); | |
| 487 | expect(log).toContain("runner lazy sent a result that does not match the job's steps; running here"); | |
| 488 | expect(log).toContain("ran-the-check"); | |
| 489 | }); | |
| 490 | ||
| 491 | describe("with a runner process connected", () => { | |
| 492 | let proc: ReturnType<typeof Bun.spawn> | undefined; | |
| 493 | let runnerId = ""; | |
| 494 | ||
| 495 | beforeAll(async () => { | |
| 496 | const made = await api("/api/v1/admin/runners", adminToken, { | |
| 497 | name: "box", | |
| 498 | }); | |
| 499 | expect(made.status).toBe(201); | |
| 500 | const enrolled = (await made.json()) as { | |
| 501 | id: string; | |
| 502 | name: string; | |
| 503 | token: string; | |
| 504 | }; | |
| 505 | expect(enrolled.token).toStartWith("forge_runner_"); | |
| 506 | runnerId = enrolled.id; | |
| 507 | // The runner is given the forge's URL, its token and a directory of its own: nothing else. | |
| 508 | proc = Bun.spawn( | |
| 509 | [ | |
| 510 | process.execPath, | |
| 511 | join(import.meta.dir, "..", "src", "runner", "main.ts"), | |
| 512 | ], | |
| 513 | { | |
| 514 | env: { | |
| 515 | PATH: process.env["PATH"] ?? "/usr/bin:/bin", | |
| 516 | HOME: join(f.dataDir, "runner-home"), | |
| 517 | FORGE_URL: f.servers.url, | |
| 518 | FORGE_RUNNER_TOKEN: enrolled.token, | |
| 519 | FORGE_RUNNER_DIR: join(f.dataDir, "runner"), | |
| 520 | FORGE_CELLS: "off", | |
| 521 | }, | |
| 522 | stdout: "ignore", | |
| 523 | stderr: "ignore", | |
| 524 | }, | |
| 525 | ); | |
| 526 | await f.until( | |
| 527 | () => deskOf(f.app).claimers.size > 0, | |
| 528 | "the runner process waiting in its claim", | |
| 529 | ); | |
| 530 | }); | |
| 531 | afterAll(async () => { | |
| 532 | proc?.kill(); | |
| 533 | await proc?.exited; | |
| 534 | }); | |
| 535 | ||
| 536 | test("a green check runs on the runner and lands in the run with its log", async () => { | |
| 537 | const oid = await pull("green", "g.txt"); | |
| 538 | const run = await f.settled((r) => r.commitOid === oid, 30_000); | |
| 539 | expect(run.state).toBe("ok"); | |
| 540 | expect(run.placement).toBe("runner"); | |
| 541 | expect(run.node).toBe("box"); | |
| 542 | expect(run.sandboxEnforced).toBe(false); | |
| 543 | const log = logOf(run.id); | |
| 544 | expect(log).toContain("placement: runner box"); | |
| 545 | expect(log).toContain(`$ ${CHECK}`); | |
| 546 | expect(log).toContain("ran-the-check"); | |
| 547 | expect(log).toContain("uncontained on this runner: FORGE_CELLS=off"); | |
| 548 | expect(log).not.toContain("running here"); | |
| 549 | // Nothing was checked out on the forge's side, and the desk is empty again. | |
| 550 | expect( | |
| 551 | f.app.db | |
| 552 | .query("SELECT COUNT(*) AS n FROM check_job WHERE run_id = ?1") | |
| 553 | .get(run.id), | |
| 554 | ).toEqual({ n: 0 }); | |
| 555 | }); | |
| 556 | ||
| 557 | test("a red check is red, with the command's output and its exit code", async () => { | |
| 558 | const oid = await pull("red", "RED"); | |
| 559 | const run = await f.settled((r) => r.commitOid === oid, 30_000); | |
| 560 | expect(run.state).toBe("failed"); | |
| 561 | expect(run.placement).toBe("runner"); | |
| 562 | expect(run.summary).toStartWith("check failed"); | |
| 563 | expect(logOf(run.id)).toContain("ran-the-check"); | |
| 564 | }); | |
| 565 | ||
| 566 | test("an admin lists it as seen, and a removed runner's token stops", async () => { | |
| 567 | const listed = (await ( | |
| 568 | await api("/api/v1/admin/runners", adminToken) | |
| 569 | ).json()) as { items: { id: string; name: string; seenAt: number }[] }; | |
| 570 | const box = listed.items.find((r) => r.id === runnerId); | |
| 571 | expect(box?.name).toBe("box"); | |
| 572 | expect(box!.seenAt).toBeGreaterThan(0); | |
| 573 | expect(JSON.stringify(listed)).not.toContain("forge_runner_"); | |
| 574 | // A member is not told the door exists. | |
| 575 | const member = createToken(f.app.db, f.bob.id, "bob", "admin").token; | |
| 576 | expect([403, 404]).toContain( | |
| 577 | (await api("/api/v1/admin/runners", member)).status, | |
| 578 | ); | |
| 579 | expect( | |
| 580 | ( | |
| 581 | await fetch(`${f.servers.url}/api/v1/admin/runners/${runnerId}`, { | |
| 582 | method: "DELETE", | |
| 583 | headers: { authorization: `Bearer ${adminToken}` }, | |
| 584 | }) | |
| 585 | ).status, | |
| 586 | ).toBe(204); | |
| 587 | // Its next claim is refused, and the process ends saying so. | |
| 588 | expect(await proc!.exited).toBe(1); | |
| 589 | proc = undefined; | |
| 590 | }); | |
| 591 | }); | |
| 592 | }); |
test/job-slices.test.ts +1
| @@ -261,6 +261,7 @@ function jobs(): Record<string, () => unknown> { | ||
| 261 | 261 | * on. |
| 262 | 262 | */ |
| 263 | 263 | const UNTIMED: Record<string, string> = { |
| 264 | "runner-wait": "one-shot timers that only resolve a promise: a runner's claim held open, a check's look at its job each second (checks/runner_jobs.ts, runner_step.ts) and the runner program's own pauses; off unless FORGE_CHECKS_PLACEMENT=runner, and checks-runner-placement.test.ts drives all three", | |
| 264 | 265 | "drill-watch": "a /proc read of the drill child's memory and a state write every WATCH_MS, only while a restore drill runs in its own process; drill-job tests drive runDrillInChild directly", |
| 265 | 266 | desk: "expires notes past their time: one indexed delete on the desk table every 30 s (actions/desk.ts); the fixture has no notes", |
| 266 | 267 | ledger: "walks the cost ledger's hash chain at boot and once a day (#12273): file reads proportional to the ledger, off the request path; the fixture has no ledger", |
test/pages-budget.test.ts +1
| @@ -66,6 +66,7 @@ const NOT_PAGES = new Set([ | ||
| 66 | 66 | "/events", "/system/stream", "/settings/teams/stream", "/fleet/graph/live", |
| 67 | 67 | "/settings/traffic/scrapes/export", "/settings/backups/live", "/:owner/:name/stream", |
| 68 | 68 | "/settings/origins/flows/:id/stream", "/:owner/:name/archive/:file", // a tarball download, not a page |
| 69 | "/api/v1/runner/jobs/:id/git/info/refs", // a runner's git door for one job (#13652), not a page | |
| 69 | 70 | ]); |
| 70 | 71 | |
| 71 | 72 | /** Patterns with an id in them, and the path the fixture fills them with. */ |
test/security-floor-routes.test.ts +1
| @@ -71,6 +71,7 @@ const CART = "the seed pins no build with a cart (test/build-cart.test.ts covers | ||
| 71 | 71 | const UNSEEDED: Record<string, string> = { |
| 72 | 72 | "/:owner/-/teams/:team": ORG, |
| 73 | 73 | "/api/v1/repos/:owner/:name/drop/:file": "a file name in an app's private drop; the seed drops none", |
| 74 | "/api/v1/runner/jobs/:id/git/info/refs": "a job a runner holds, read with that job's own token; the seed offers none (test/checks-runner-placement.test.ts covers it)", | |
| 74 | 75 | "/claim/start/:token": "needs a one-time start claim token, which the seed does not mint (test/start.test.ts covers it)", |
| 75 | 76 | "/signin/access/:token": "needs a one-time access link an admin minted", |
| 76 | 77 | "/talk/:id": TALK, |
new above
offline
A result with
steps: []passes here and becomes green. Require the check step (and install when offered) to be present, ok and named as in the job, or return taken:false.