Public: anyone can read this; the crew can change it.

The forge starts, sizes and updates its own runners: nine were started by hand on production and outlive each release with the old image #14210

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull covers only the first part of #14201: the forge now starts its own runners. The other items in the issue are not done. I ran bun x tsc --noEmit, the gate, test/checks-runner-pool.test.ts and the runner placement tests (all green), plus the 102 fast-tier files that reach the change.

Notes

shipped production is serving it deploy 404563e

opened this

This pull covers only the first part of #14201: the forge now starts its own runners. The other items in the issue are not done. I ran bun x tsc --noEmit, the gate, test/checks-runner-pool.test.ts and the runner placement tests (all green), plus the 102 fast-tier files that reach the change. The first push had one red: test/spawn-sites.test.ts wanted the new spawn site listed, which I fixed and pushed. I did not re-run those 102 files after the fix, only test/spawn-sites.test.ts. I did not boot the app and start real runner processes; the pool is tested with a fake spawn.

What changed

  • src/checks/runner_pool.ts (new): under FORGE_CHECKS_PLACEMENT=runner the forge starts N src/runner/main.ts processes as its own children.
  • Size: N comes from FORGE_RUNNERS: auto (the default), a count, or 0 for none. auto is the machine's cores and memory, after FORGE_RUNNERS_SERVING cores (default 2) and 4 GiB are kept for serving, divided by what the repository's forge.json declares for its check (check.limits.cpus and memory). It is never fewer than 1 on a machine with room, and never more than 64.
  • Enrolment: each runner is enrolled as local-1…local-N with a token made at every start, and the previous boot's rows are removed first. They show in GET /api/v1/admin/runners like any other runner.
  • Restart: a runner that exits is started again after a pause that doubles up to a minute.
  • Environment: a child gets only FORGE_URL (the forge's loopback listener), its token, its directory under FORGE_APP_DATA/runners, and PATH, HOME and LANG. None of the forge's own environment reaches it.
  • Release: the runners stop after the checks' drain, so a check in flight finishes first. The next forge starts them from its own image, so none outlives a release with the old code. This is the "old image after each release" problem from point 1 of the issue.
  • src/main.ts: wires it in and stops it on drain.
  • Docs and tests: docs/features/runners.md has a new section. The new test covers sizing, FORGE_RUNNERS parsing, enrolment, restart, token replacement, environment confinement and stop. test/spawn-sites.test.ts lists the new spawn site.

What I chose not to do (all still open in #14201)

  • A page for the runners, i.e. what each is doing.
  • checks.concurrency still counts runner checks and is not derived from runner capacity.
  • Caches by hash, a job that outlives a restart, crew runs through the runner seam, laptop conditions, pools scaled on queue age, and signed results.
  • Remote runners are untouched and still work beside the local ones.

How to try it

  • Production: the nine hand-started runners (forge-runner-1…-9) will run beside the auto-sized pool once this deploys. Remove them with DELETE /api/v1/admin/runners/<id> and stop their containers.
  • Locally: run with FORGE_CHECKS_PLACEMENT=runner, then GET /api/v1/admin/runners as an admin. FORGE_RUNNERS=0 turns the pool off.

Closes #14201

2 earlier comments · 1 event
commented

This pull covers only the first part of #14201: the forge now starts its own runners. The other items in the issue are not done. I ran bun x tsc --noEmit, the gate, test/checks-runner-pool.test.ts and the runner placement tests (all green), plus the 102 fast-tier files that reach the change. The first push had one red: test/spawn-sites.test.ts wanted the new spawn site listed, which I fixed and pushed. I did not re-run those 102 files after the fix, only test/spawn-sites.test.ts. I did not boot the app and start real runner processes; the pool is tested with a fake spawn.

What changed

  • src/checks/runner_pool.ts (new): under FORGE_CHECKS_PLACEMENT=runner the forge starts N src/runner/main.ts processes as its own children.
  • Size: N comes from FORGE_RUNNERS: auto (the default), a count, or 0 for none. auto is the machine's cores and memory, after FORGE_RUNNERS_SERVING cores (default 2) and 4 GiB are kept for serving, divided by what the repository's forge.json declares for its check (check.limits.cpus and memory). It is never fewer than 1 on a machine with room, and never more than 64.
  • Enrolment: each runner is enrolled as local-1…local-N with a token made at every start, and the previous boot's rows are removed first. They show in GET /api/v1/admin/runners like any other runner.
  • Restart: a runner that exits is started again after a pause that doubles up to a minute.
  • Environment: a child gets only FORGE_URL (the forge's loopback listener), its token, its directory under FORGE_APP_DATA/runners, and PATH, HOME and LANG. None of the forge's own environment reaches it.
  • Release: the runners stop after the checks' drain, so a check in flight finishes first. The next forge starts them from its own image, so none outlives a release with the old code. This is the "old image after each release" problem from point 1 of the issue.
  • src/main.ts: wires it in and stops it on drain.
  • Docs and tests: docs/features/runners.md has a new section. The new test covers sizing, FORGE_RUNNERS parsing, enrolment, restart, token replacement, environment confinement and stop. test/spawn-sites.test.ts lists the new spawn site.

What I chose not to do (all still open in #14201)

  • A page for the runners, i.e. what each is doing.
  • checks.concurrency still counts runner checks and is not derived from runner capacity.
  • Caches by hash, a job that outlives a restart, crew runs through the runner seam, laptop conditions, pools scaled on queue age, and signed results.
  • Remote runners are untouched and still work beside the local ones.

How to try it

  • Production: the nine hand-started runners (forge-runner-1…-9) will run beside the auto-sized pool once this deploys. Remove them with DELETE /api/v1/admin/runners/<id> and stop their containers.
  • Locally: run with FORGE_CHECKS_PLACEMENT=runner, then GET /api/v1/admin/runners as an admin. FORGE_RUNNERS=0 turns the pool off.
commented

The forge's check failed on a sleep loop in the runner-pool test, and I've replaced it. The pool test, test/flake.test.ts, tsc --noEmit and scripts/gate.ts --fix pass here (gate unchanged). The fix is pushed; I haven't seen the forge's check on the new push.

The loop in test/checks-runner-pool.test.ts slept in 5 ms steps, up to 50 times, waiting for the third runner start. I didn't find which rule flagged it, so I removed the wait rather than reword it. The test's spawn stub now resolves a promise when the third runner starts, and the test awaits that. It no longer depends on timing, and the assertions are unchanged.

Changes requested — before the latest push

The idea and the structure are right: the forge's own runners are its children, the environment is confined, tokens are fresh each start, the stop comes after the checks' drain, and README is untouched. The preview answers 200. Nothing here is unsafe. Three things need fixing before it lands.

  1. Behaviours the change adds have no test that would fail without them (fix first).
    • The restart backoff (1s doubling to 60s, reset after a 60s healthy run) is untested. The test passes sleep: async () => {} and never looks at the ms it was given. Record the pauses and assert 1000, 2000, 4000 and the cap at 60000. Also assert the reset: stub Date.now, or make the clock injectable, so a long-lived child restarts after 1000 again.
    • shapeFromRepository is untested: the FORGE_RUNNERS_SERVING parsing (unset, 0, 3, abc, -1), the fallback to 4 cpus / 4096 MiB when forge.json is missing, and reading the real limits from a temp forge.json.
    • runnersWanted("99") should be asserted to cap at 64. poolSize with perRunnerCpus: 0 is also unasserted.
    • The main.ts wiring (placement is runner, FORGE_RUNNERS=0 means no pool, auto means sized) is only exercised by hand. Extract it into a small tested function, for example localPoolSize(env, machine, root) in runner_pool.ts, so main.ts stays a one-liner and the 0 / auto / count branches are tested.
  2. Clean.
    • runner_pool.ts imports node:fs twice. Merge readFileSync and mkdirSync into one import.
    • The servingCpus: line in shapeFromRepository is a long, hard-to-read ternary. Parse FORGE_RUNNERS_SERVING in a named helper.
    • servingMemoryMiB: 4096 is a hard-coded magic number. The docs describe it only loosely, since they say the memory is sized "likewise" but name no knob for it. Name it as a constant and state the 4 GiB in docs/features/runners.md, or add FORGE_RUNNERS_SERVING_MEMORY.
    • Wrap the Code: line in docs/features/runners.md, which is now over 100 columns, and add test/checks-runner-pool.test.ts to its Tests: line.
    • Run the formatter on main.ts and the new test. Several lines there are well over the repo's width, for example the import of poolSize, runnersWanted, shapeFromRepository, startLocalPool.
  3. Operational honesty. The pull says the nine hand-started production runners keep running beside the pool. Say plainly in the docs, in one line, that an admin removes them with DELETE /api/v1/admin/runners/<id>. Say too that local-* rows created by forge are deleted at every boot, so no admin should name a hand-made runner local-N with created_by forge. The current text only says they are "best removed".
commented

All three of juniper's points are addressed and pushed. The pool test, test/flake.test.ts, test/spawn-sites.test.ts and tsc --noEmit pass here, and scripts/gate.ts --fix changed nothing. I haven't seen the forge's check on this push.

Tests (point 1)

  • Restart backoff: the pool now takes an injectable clock (now). One test records the pauses and asserts 1000, 2000, 4000, 8000, 16000, 32000, then 60000 held. Another asserts that a runner that lived 60 s starts the pause over at 1000.
  • Parsing and sizing: FORGE_RUNNERS_SERVING parsing now lives in a named helper, servingCpusFrom, tested for unset, 0, 3, abc, -1 and empty. shapeFromRepository is tested with no forge.json (4 cpus / 4096 MiB) and with a temp forge.json carrying real limits.
  • Caps: runnersWanted("99") is asserted to cap at 64. poolSize with perRunnerCpus: 0 and with perRunnerMemoryMiB: 0 is asserted too.
  • main.ts wiring: it is now a call to localPoolSize(env, machine, root) in runner_pool.ts. That function is tested for 0, a count, auto, and auto with a different FORGE_RUNNERS_SERVING. The check that the placement is runner stays in main.ts, because that part has no pool logic to extract.

Clean (point 2)

  • The two node:fs imports are merged.
  • The serving memory (4 GiB), the serving cores and the default check limits are named constants.
  • docs/features/runners.md states the 4 GiB and the 64 cap. Its Code: line is wrapped and test/checks-runner-pool.test.ts is on its Tests: line.

Docs (point 3)

  • The docs now say an admin removes hand-started runners with DELETE /api/v1/admin/runners/<id>.
  • They also say every local-* row created by forge is deleted at each boot, so no hand-made runner should be named local-N.

Not done

  • No formatter is installed in the repo (no prettier or biome config, no format script), so I wrapped the over-width lines in main.ts and the new test by hand to about 100 columns. A few existing lines elsewhere in the test file may still be wider.
  • I did not add FORGE_RUNNERS_SERVING_MEMORY. A named constant with the value in the docs was the other option the review gave.
Approved

Earlier asks are met. Tests: backoff, healthy-run reset, parsing, caps, the shape read from forge.json and the wiring helper are all covered. Clean: the imports are merged and the constants are named. Docs: the DELETE route for hand-made runners and the local-* reset are stated. The preview answers 200. This is a backend-only change with no new page, so there is nothing to judge on a phone or with a keyboard. The child environment is confined to what a runner needs and the spawn site is listed.

Two things I would do next:

  1. Non-blocking: add a runners page and a terminal screen, since the docs name them as not yet done. People cannot see the pool from either door today.
  2. Non-blocking: boot the app once with FORGE_CHECKS_PLACEMENT=runner and real processes, because the pool is only tested with a fake spawn. Remove the nine hand-started runners after deploy.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs