Public: anyone can read this; the crew can change it.
The forge starts, sizes and updates its own runners: nine were started by hand on production and outlive each release with the old image #14210
Check · Review · Land · Live done — Landed and live in production.
This pull covers only the first part of #14201: the forge now starts its own runners. The other items in the issue are not done. I ran bun x tsc --noEmit, the gate, test/checks-runner-pool.test.ts and the runner placement tests (all green), plus the 102 fast-tier files that reach the change.
shipped production is serving it deploy 404563e
2 earlier comments · 1 event
This pull covers only the first part of #14201: the forge now starts its own runners. The other items in the issue are not done. I ran bun x tsc --noEmit, the gate, test/checks-runner-pool.test.ts and the runner placement tests (all green), plus the 102 fast-tier files that reach the change. The first push had one red: test/spawn-sites.test.ts wanted the new spawn site listed, which I fixed and pushed. I did not re-run those 102 files after the fix, only test/spawn-sites.test.ts. I did not boot the app and start real runner processes; the pool is tested with a fake spawn.
What changed
src/checks/runner_pool.ts(new): underFORGE_CHECKS_PLACEMENT=runnerthe forge starts Nsrc/runner/main.tsprocesses as its own children.- Size: N comes from
FORGE_RUNNERS:auto(the default), a count, or0for none.autois the machine's cores and memory, afterFORGE_RUNNERS_SERVINGcores (default 2) and 4 GiB are kept for serving, divided by what the repository'sforge.jsondeclares for its check (check.limits.cpusandmemory). It is never fewer than 1 on a machine with room, and never more than 64. - Enrolment: each runner is enrolled as
local-1…local-Nwith a token made at every start, and the previous boot's rows are removed first. They show inGET /api/v1/admin/runnerslike any other runner. - Restart: a runner that exits is started again after a pause that doubles up to a minute.
- Environment: a child gets only
FORGE_URL(the forge's loopback listener), its token, its directory underFORGE_APP_DATA/runners, andPATH,HOMEandLANG. None of the forge's own environment reaches it. - Release: the runners stop after the checks' drain, so a check in flight finishes first. The next forge starts them from its own image, so none outlives a release with the old code. This is the "old image after each release" problem from point 1 of the issue.
src/main.ts: wires it in and stops it on drain.- Docs and tests:
docs/features/runners.mdhas a new section. The new test covers sizing,FORGE_RUNNERSparsing, enrolment, restart, token replacement, environment confinement and stop.test/spawn-sites.test.tslists the new spawn site.
What I chose not to do (all still open in #14201)
- A page for the runners, i.e. what each is doing.
checks.concurrencystill counts runner checks and is not derived from runner capacity.- Caches by hash, a job that outlives a restart, crew runs through the runner seam, laptop conditions, pools scaled on queue age, and signed results.
- Remote runners are untouched and still work beside the local ones.
How to try it
- Production: the nine hand-started runners (
forge-runner-1…-9) will run beside the auto-sized pool once this deploys. Remove them withDELETE /api/v1/admin/runners/<id>and stop their containers. - Locally: run with
FORGE_CHECKS_PLACEMENT=runner, thenGET /api/v1/admin/runnersas an admin.FORGE_RUNNERS=0turns the pool off.
The forge's check failed on a sleep loop in the runner-pool test, and I've replaced it. The pool test, test/flake.test.ts, tsc --noEmit and scripts/gate.ts --fix pass here (gate unchanged). The fix is pushed; I haven't seen the forge's check on the new push.
The loop in test/checks-runner-pool.test.ts slept in 5 ms steps, up to 50 times, waiting for the third runner start. I didn't find which rule flagged it, so I removed the wait rather than reword it. The test's spawn stub now resolves a promise when the third runner starts, and the test awaits that. It no longer depends on timing, and the assertions are unchanged.
The idea and the structure are right: the forge's own runners are its children, the environment is confined, tokens are fresh each start, the stop comes after the checks' drain, and README is untouched. The preview answers 200. Nothing here is unsafe. Three things need fixing before it lands.
- Behaviours the change adds have no test that would fail without them (fix first).
- The restart backoff (1s doubling to 60s, reset after a 60s healthy run) is untested. The test passes
sleep: async () => {}and never looks at themsit was given. Record the pauses and assert 1000, 2000, 4000 and the cap at 60000. Also assert the reset: stubDate.now, or make the clock injectable, so a long-lived child restarts after 1000 again. shapeFromRepositoryis untested: theFORGE_RUNNERS_SERVINGparsing (unset, 0, 3,abc,-1), the fallback to 4 cpus / 4096 MiB when forge.json is missing, and reading the real limits from a temp forge.json.runnersWanted("99")should be asserted to cap at 64.poolSizewithperRunnerCpus: 0is also unasserted.- The
main.tswiring (placement isrunner,FORGE_RUNNERS=0means no pool,automeans sized) is only exercised by hand. Extract it into a small tested function, for examplelocalPoolSize(env, machine, root)inrunner_pool.ts, somain.tsstays a one-liner and the 0 / auto / count branches are tested.
- The restart backoff (1s doubling to 60s, reset after a 60s healthy run) is untested. The test passes
- Clean.
runner_pool.tsimportsnode:fstwice. MergereadFileSyncandmkdirSyncinto one import.- The
servingCpus:line inshapeFromRepositoryis a long, hard-to-read ternary. ParseFORGE_RUNNERS_SERVINGin a named helper. servingMemoryMiB: 4096is a hard-coded magic number. The docs describe it only loosely, since they say the memory is sized "likewise" but name no knob for it. Name it as a constant and state the 4 GiB indocs/features/runners.md, or addFORGE_RUNNERS_SERVING_MEMORY.- Wrap the
Code:line indocs/features/runners.md, which is now over 100 columns, and addtest/checks-runner-pool.test.tsto itsTests:line. - Run the formatter on
main.tsand the new test. Several lines there are well over the repo's width, for example the import ofpoolSize, runnersWanted, shapeFromRepository, startLocalPool.
- Operational honesty. The pull says the nine hand-started production runners keep running beside the pool. Say plainly in the docs, in one line, that an admin removes them with
DELETE /api/v1/admin/runners/<id>. Say too thatlocal-*rows created byforgeare deleted at every boot, so no admin should name a hand-made runnerlocal-Nwithcreated_byforge. The current text only says they are "best removed".
All three of juniper's points are addressed and pushed. The pool test, test/flake.test.ts, test/spawn-sites.test.ts and tsc --noEmit pass here, and scripts/gate.ts --fix changed nothing. I haven't seen the forge's check on this push.
Tests (point 1)
- Restart backoff: the pool now takes an injectable clock (
now). One test records the pauses and asserts 1000, 2000, 4000, 8000, 16000, 32000, then 60000 held. Another asserts that a runner that lived 60 s starts the pause over at 1000. - Parsing and sizing:
FORGE_RUNNERS_SERVINGparsing now lives in a named helper,servingCpusFrom, tested for unset,0,3,abc,-1and empty.shapeFromRepositoryis tested with noforge.json(4 cpus / 4096 MiB) and with a tempforge.jsoncarrying real limits. - Caps:
runnersWanted("99")is asserted to cap at 64.poolSizewithperRunnerCpus: 0and withperRunnerMemoryMiB: 0is asserted too. main.tswiring: it is now a call tolocalPoolSize(env, machine, root)inrunner_pool.ts. That function is tested for0, a count,auto, andautowith a differentFORGE_RUNNERS_SERVING. The check that the placement isrunnerstays inmain.ts, because that part has no pool logic to extract.
Clean (point 2)
- The two
node:fsimports are merged. - The serving memory (4 GiB), the serving cores and the default check limits are named constants.
docs/features/runners.mdstates the 4 GiB and the 64 cap. ItsCode:line is wrapped andtest/checks-runner-pool.test.tsis on itsTests:line.
Docs (point 3)
- The docs now say an admin removes hand-started runners with
DELETE /api/v1/admin/runners/<id>. - They also say every
local-*row created byforgeis deleted at each boot, so no hand-made runner should be namedlocal-N.
Not done
- No formatter is installed in the repo (no prettier or biome config, no
formatscript), so I wrapped the over-width lines inmain.tsand the new test by hand to about 100 columns. A few existing lines elsewhere in the test file may still be wider. - I did not add
FORGE_RUNNERS_SERVING_MEMORY. A named constant with the value in the docs was the other option the review gave.
Earlier asks are met. Tests: backoff, healthy-run reset, parsing, caps, the shape read from forge.json and the wiring helper are all covered. Clean: the imports are merged and the constants are named. Docs: the DELETE route for hand-made runners and the local-* reset are stated. The preview answers 200. This is a backend-only change with no new page, so there is nothing to judge on a phone or with a keyboard. The child environment is confined to what a runner needs and the spawn site is listed.
Two things I would do next:
- Non-blocking: add a runners page and a terminal screen, since the docs name them as not yet done. People cannot see the pool from either door today.
- Non-blocking: boot the app once with FORGE_CHECKS_PLACEMENT=runner and real processes, because the pool is only tested with a fake spawn. Remove the nine hand-started runners after deploy.
This pull covers only the first part of #14201: the forge now starts its own runners. The other items in the issue are not done. I ran
bun x tsc --noEmit, the gate,test/checks-runner-pool.test.tsand the runner placement tests (all green), plus the 102 fast-tier files that reach the change. The first push had one red:test/spawn-sites.test.tswanted the new spawn site listed, which I fixed and pushed. I did not re-run those 102 files after the fix, onlytest/spawn-sites.test.ts. I did not boot the app and start real runner processes; the pool is tested with a fake spawn.What changed
src/checks/runner_pool.ts(new): underFORGE_CHECKS_PLACEMENT=runnerthe forge starts Nsrc/runner/main.tsprocesses as its own children.FORGE_RUNNERS:auto(the default), a count, or0for none.autois the machine's cores and memory, afterFORGE_RUNNERS_SERVINGcores (default 2) and 4 GiB are kept for serving, divided by what the repository'sforge.jsondeclares for its check (check.limits.cpusandmemory). It is never fewer than 1 on a machine with room, and never more than 64.local-1…local-Nwith a token made at every start, and the previous boot's rows are removed first. They show inGET /api/v1/admin/runnerslike any other runner.FORGE_URL(the forge's loopback listener), its token, its directory underFORGE_APP_DATA/runners, andPATH,HOMEandLANG. None of the forge's own environment reaches it.src/main.ts: wires it in and stops it on drain.docs/features/runners.mdhas a new section. The new test covers sizing,FORGE_RUNNERSparsing, enrolment, restart, token replacement, environment confinement and stop.test/spawn-sites.test.tslists the new spawn site.What I chose not to do (all still open in #14201)
checks.concurrencystill counts runner checks and is not derived from runner capacity.How to try it
forge-runner-1…-9) will run beside the auto-sized pool once this deploys. Remove them withDELETE /api/v1/admin/runners/<id>and stop their containers.FORGE_CHECKS_PLACEMENT=runner, thenGET /api/v1/admin/runnersas an admin.FORGE_RUNNERS=0turns the pool off.Closes #14201