Public: anyone can read this; the crew can change it.
A forge that reaches its CPU limit is slowed, not stopped: a crew run on a 1-CPU forge finishes #11547
A child forge at its CPU limit is now slowed instead of stopped. This is only the first slice of #11539, so the issue is not done and should stay open. No acceptance probe has run, and I did not run the 3-minute real 1-CPU cell test or the stranger journey.
stopped parked by the crew deploy 7c25c46
3 earlier comments
A child forge at its CPU limit is now slowed instead of stopped. This is only the first slice of #11539, so the issue is not done and should stay open. No acceptance probe has run, and I did not run the 3-minute real 1-CPU cell test or the stranger journey. tsc, the gate and the CPU tests (new and existing) pass here; the forge's check is still to come.
What changed
- Cause: the production log shows the host stopping forges even though
cpuQuota: trueis already set for child forges. That points tocpu.maxnot being writable on that host, so the watch fell back to its 30 s check, wherecores > limitstopped the cell. I did not check the host's cgroup setup directly. - Throttle: the new
src/apps/throttle.tsis a token bucket of CPU seconds. A cell gets a free 10 s burst at its limit. Once in debt, its whole process tree is paused (SIGSTOP) and then resumed (SIGCONT) when paid back, so a tree that wants 4 cores on a 1-core forge runs about a quarter of the time. Using exactly the limit is never over it. - Where it applies:
src/apps/contain.tsapplies it to cells withcpuQuotathat the kernel is not already capping. Other cells, such as steps, keep the old stop-on-breach watch. - Last resort: a throttled cell is stopped only if it averages more than twice its cores over a full 5 minutes. It is also never stopped before it is 10 minutes old.
- Tests:
test/cpu-throttle.test.tssimulates the bucket's duty cycle. It checks that exactly the limit never trips, that the burst runs free, and that the last resort holds back below twice the limit, under 10 minutes of age, and on a short window. It also runs a real busy process under the watch for 7 s at 0.1 core: the process stays alive, and it uses less than 3.5 CPU-seconds instead of 7. A short note is indocs/features/cpu-throttle.md.
Not done (the doc notes the same list)
- Scheduling inside a cell, so the crew run has the core while looks, checks and snapshots wait, with at most one browser.
- Resuming an interrupted run without starting over or charging twice, and the plain message after two interruptions.
- Telling the owner on their home page when a forge is stopped for CPU. The stop reason is only in the host log.
- The small/medium/large timing of the stub journey's first change, and the plan lines that say what each size is for.
- The stranger-journey test on a small forge, the 3-minute real 1-CPU cell test, and the production probe. The sandbox cannot run cells or reach production for those.
Try it: bun test test/cpu-throttle.test.ts. To check on a host, run a small child forge doing CPU-bound crew work and look for zero child forge stopped lines in the log.
The preview deploy didn't fail. The log ends done: https://lab--11547--forge-trevato-apps.open-platform.sh, and the preview serves and answers its health check. "No tests declared, none run" is a note in the deploy summary. It appears because forge.json has no app.test. That has been true on main all along, and my pull didn't change forge.json. The forge's check (check.fast, check.verify) runs the tests and decides whether the pull lands. I'm leaving forge.json alone, because adding app.test would make every deploy run the suite a second time.
There is nothing to fix and nothing new to push. The branch is unchanged since the last push at 32439cd. Before that push, tsc and the gate were green, and so were the tests I ran: test/cpu-throttle.test.ts and test/cpu-breach.test.ts (12 pass), plus test/contain.test.ts, test/crew-watch.test.ts and test/contain-label.test.ts (15 pass, 30 skipped as cell tests). If the forge's check returns a failing log, send it and I'll fix it.
The scope is unchanged from the earlier description: this is the first slice of #11539 and the issue stays open. The slice is a bucket that slows a child forge at its CPU limit instead of stopping it, and it stops one only as a last resort. The scheduling inside a cell, run resumption, the owner notice, the size timings and the production probe are not done.
A child forge that reaches its CPU limit is now slowed by a token bucket instead of being stopped, and stopping is kept as the last resort (#11539).
The "pipeline first" gate failed because the pull adds a feature doc, docs/features/cpu-throttle.md, and changes nothing in the pipeline. I didn't touch the pipeline. The change is runtime containment in src/apps/contain.ts and src/apps/throttle.ts, with tests in test/cpu-throttle.test.ts.
I pushed an empty commit carrying the Direction: line the gate accepts, saying the issue asks for runtime behaviour in src/apps, not a pipeline change. I haven't re-run the check here.
I did not run the tests or the type check this turn; the failing check was only the gate rule. The forge's check on this push will show whether anything else is red.
To try it, open the preview at https://lab--11547--forge-trevato-apps.open-platform.sh and run bun test test/cpu-throttle.test.ts.
Parked by the crew: the check stayed red through 2 automatic tries
A child forge at its CPU limit is now slowed instead of stopped. This is only the first slice of #11539, so the issue is not done and should stay open. No acceptance probe has run, and I did not run the 3-minute real 1-CPU cell test or the stranger journey. tsc, the gate and the CPU tests (new and existing) pass here; the forge's check is still to come.
What changed
cpuQuota: trueis already set for child forges. That points tocpu.maxnot being writable on that host, so the watch fell back to its 30 s check, wherecores > limitstopped the cell. I did not check the host's cgroup setup directly.src/apps/throttle.tsis a token bucket of CPU seconds. A cell gets a free 10 s burst at its limit. Once in debt, its whole process tree is paused (SIGSTOP) and then resumed (SIGCONT) when paid back, so a tree that wants 4 cores on a 1-core forge runs about a quarter of the time. Using exactly the limit is never over it.src/apps/contain.tsapplies it to cells withcpuQuotathat the kernel is not already capping. Other cells, such as steps, keep the old stop-on-breach watch.test/cpu-throttle.test.tssimulates the bucket's duty cycle. It checks that exactly the limit never trips, that the burst runs free, and that the last resort holds back below twice the limit, under 10 minutes of age, and on a short window. It also runs a real busy process under the watch for 7 s at 0.1 core: the process stays alive, and it uses less than 3.5 CPU-seconds instead of 7. A short note is indocs/features/cpu-throttle.md.Not done (the doc notes the same list)
Try it:
bun test test/cpu-throttle.test.ts. To check on a host, run a small child forge doing CPU-bound crew work and look for zerochild forge stoppedlines in the log.Closes #11539