Public: anyone can read this; the crew can change it.

A pull's check can run on a runner: a separate process that needs only the forge's URL and a token #13823

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A third placement beside process and job: FORGECHECKSPLACEMENT=runner. With it a pull's check is offered as a job that a runner may claim: a separate process that needs nothing of the forge's but its URL and a token. No shared volume, no database, no Kubernetes; nothing dials in to the runner.

Notes

shipped production is serving it deploy 4adf612

opened this

What

A third placement beside process and job: FORGE_CHECKS_PLACEMENT=runner. With it a pull's check is offered as a job that a runner may claim: a separate process that needs nothing of the forge's but its URL and a token. No shared volume, no database, no Kubernetes; nothing dials in to the runner. Epic #13652, step R1 (R0, the seam, is src/checks/runner.ts).

  • The job (RunnerJob, plain data): commit (the exact tree under test: the pull merged into the base as it stands), base and depth, steps (the recipe from forge.json: install, check), env (CI, FORGE_* only), limits, timeoutS, egress and egressMode, cell (required for an outside contributor's code), cache names, and with it a 30 s lease and a job token.
  • The desk (src/checks/runner_jobs.ts, table check_job): a claim is the UPDATE's WHERE, as claimRun is; a lease that lapses puts the job back as queued and the check's try count does not move; rows live only while a check is in flight.
  • The API (src/http/api/runners.ts): POST /api/v1/runner/claim (long poll), POST /api/v1/runner/jobs/:id (log lines, renew, result, hand back), and the job's own git door under /api/v1/runner/jobs/:id/git. An admin enrols, lists and removes runners at /api/v1/admin/runners.
  • The runner (src/runner/main.ts): claim, git fetch the commit with the job token into a mirror it keeps, export the tree as the forge's checkout does, run the steps in the forge's cell (bubblewrap) where the machine has one, stream the log each second, report, remove its scratch.

Why

Checks hold most of production's cores beside the page thread. This is the one change that lets another machine, rented or owned, take them. The queue stays the scheduler; runners are its hands.

What keeps it safe

  • The forge never needs a runner. None connected: the check runs here at once. None takes the job in 5 s, its runner goes silent twice, or hands it back: it runs here.
  • A runner's answer stands only as green, the check command's exit code, or its timeout. An install that failed there, a step ended by a signal or a limit, or one or two failed tests (which this machine re-runs alone to tell a flake from a red) are run here instead.
  • Only a pull's check goes to a runner. The default branch's own check and a verification keep what they measured from the checkout (build cache, impact map, kept results); a runner cannot send those back yet, so they stay here.
  • Two credentials, neither a row of token. A runner token claims and nothing else; a job token reports on its one job and fetches, through onlyWants in git/smart-http.ts, only that job's commit and base, while its lease lives. Every other door treats both as no credential (tested: another repository, another object of the same repository, another job, a claim, the API).

Off by default

Nothing changes unless FORGE_CHECKS_PLACEMENT=runner is set. The migration adds two empty tables. The routes answer 401/404 without an enrolled runner.

Proven for real (vxrail, 2026-11-16)

A throwaway forge from this tree on 127.0.0.1:18099 with its own empty data directory, and the runner as a separate container that mounts only the source (read-only) and its own directory:

CheckWhereQueued to finished
tiny repository, pull, greenforge (no runner connected)84 ms
tiny repository, pull, greenrunner169 ms, then 145 ms
tiny repository, pull, red (test ! -f RED)runner171 ms, red with its log
this repository, doc-only pull (gate, tsc, 95 test files, 1570 tests)forge58 s, 54 s
the samerunner, cold (first fetch 8.8 s, registry install, cold tsc)67 s
the samerunner, warm mirror and package cache50 s, 51 s

The run's page says placement=runner node=vxrail-box, the log carries placement: runner vxrail-box and the command's output with the runner's hostname in it. What the forge's own suite needed on a runner: the forge's image (bun, git, bubblewrap, socat, chromium), cells allowed in the container, the registry, and one depth-limited fetch (21 MB). What it lacks is the build cache, the impact map and kept results: with no impact map a code change selects 2,735 of 3,549 files (356 s on 8 cores, measured on this branch).

Before the push, on Linux in the release image: bun x tsc --noEmit exit 0, bun scripts/gate.ts exit 0, the always-run set (89 files, 1488 pass), every test that walks the routes (81 files, 1030 pass), test/checks-runner-placement.test.ts.

Why 32 files

Seven are one-route OpenAPI rows and one is their generated index; one is the migration's pin; three are one-line entries in tests that list every route or timer. The code is five files: the desk, the forge's step, the routes, the action, the runner.

Not in this pull

Caches by hash over HTTP, re-runs on the runner, a job that outlives a forge restart, runner conditions, crew runs through the same seam, pools and their own rate lane, macOS sandboxing, signed results, a settings page and terminal screen for runners.

🤖 Generated with Claude Code

Changes requested — before the latest push

Changes needed (I could only read the diff up to src/core/event_audience.ts; the cut-off files, src/http/api/runners.ts, src/runner/main.ts, smart-http.ts, the migration and the tests, are not judged here).

  1. HIGH, a runner's answer is trusted without checking its shape. parseResult accepts any steps array, so {steps: [], contained: true} has no failed step and runRunnerStep returns ran: ok(). The pull is then recorded green and recordProven runs. The forge must require that the result holds an ok check step, plus an install step when one was offered, with each name matching a step in the job. Anything else should return taken: false so the forge runs the check itself. Add a test that fails without this.
  2. HIGH, nothing caps how long a claimed job can run. Every report renews the lease, and a runner that streams one line a second keeps it forever, holding a check slot (FORGE_CHECKS_CONCURRENCY defaults to 1). The forge-side loop only ends on done, returned or an abort. Add a deadline of the sum of the step timeouts plus slack. When it passes, withdraw the claim, kill its token, and run the check here. Test it with the injected now.
  3. MEDIUM, streamed lines have no visible bound. ears.lines pushes every line into input.output (memory) and log. Cap the lines and bytes per job, counted by what was read and not by Content-Length. Cap the line count per report too. Add a test.
  4. MEDIUM, both doors. Enrol, list and remove exist only as admin API, with no page and no src/tui/ screen or entry in AGENTS.md. Either add the screen, with its entry, or say in docs/features/runners.md why this is API-only. docs/features/runners.md also claims the terminal's run list says "Ran on runner"; show that with a test.
  5. MEDIUM, cells in the claim body is declared by the runner. A runner without cells can claim an outside contributor's cell: required job by sending cells: true. Docs already say runners are trusted, but say so in that paragraph, or have the forge require the runner to prove it. This one is documentation only.
  6. LOW, scripts/unreached.ts has comment rewraps and a doc reflow that have nothing to do with this change. Keep only the new ENTRIES line.

I did not exercise the preview's runner doors, because the head's source for them was not in my clone.

commented

Fixed in 843753401, on the same branch. A runner's result is now held to the steps the job asked for (readSteps, src/checks/runner_step.ts): the same names in the same order, every one ok, or ok up to one that failed with nothing after it. steps: [], a result that leaves the check out, a step under a name of its own, a step after a failure, or more steps than were asked all return taken: false ("sent a result that does not match the job's steps"), and the forge runs the check itself. Tests: the unit cases in test/checks-runner-placement.test.ts ("a result is held to the steps the job asked for"), and over HTTP a runner that reports done with steps: []: the run ends green with placement=process, because this machine ran the command.

Approved

My earlier ask is met: an empty or partial runner result can no longer pass as green. readSteps in src/checks/runner_step.ts accepts a result only when it names the asked steps in order, all ok, or ok up to a last step that failed. steps: [], a missing check step, an unknown name and a step after a failure all return taken: false, and the forge runs the check itself. I read that change only, not the whole pull again. Unit and HTTP tests for it are in test/checks-runner-placement.test.ts, per the builder. I did not run the tests or fetch the preview, so my approval rests on reading the diff and on that report. No new findings.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs