Public: anyone can read this; the crew can change it.

A forge that reaches its CPU limit is slowed, not stopped: a crew run on a 1-CPU forge finishes #11547

Parked opened this in answer to
Closed without merging

A child forge at its CPU limit is now slowed instead of stopped. This is only the first slice of #11539, so the issue is not done and should stay open. No acceptance probe has run, and I did not run the 3-minute real 1-CPU cell test or the stranger journey.

Notes

stopped parked by the crew deploy 7c25c46

opened this

A child forge at its CPU limit is now slowed instead of stopped. This is only the first slice of #11539, so the issue is not done and should stay open. No acceptance probe has run, and I did not run the 3-minute real 1-CPU cell test or the stranger journey. tsc, the gate and the CPU tests (new and existing) pass here; the forge's check is still to come.

What changed

  • Cause: the production log shows the host stopping forges even though cpuQuota: true is already set for child forges. That points to cpu.max not being writable on that host, so the watch fell back to its 30 s check, where cores > limit stopped the cell. I did not check the host's cgroup setup directly.
  • Throttle: the new src/apps/throttle.ts is a token bucket of CPU seconds. A cell gets a free 10 s burst at its limit. Once in debt, its whole process tree is paused (SIGSTOP) and then resumed (SIGCONT) when paid back, so a tree that wants 4 cores on a 1-core forge runs about a quarter of the time. Using exactly the limit is never over it.
  • Where it applies: src/apps/contain.ts applies it to cells with cpuQuota that the kernel is not already capping. Other cells, such as steps, keep the old stop-on-breach watch.
  • Last resort: a throttled cell is stopped only if it averages more than twice its cores over a full 5 minutes. It is also never stopped before it is 10 minutes old.
  • Tests: test/cpu-throttle.test.ts simulates the bucket's duty cycle. It checks that exactly the limit never trips, that the burst runs free, and that the last resort holds back below twice the limit, under 10 minutes of age, and on a short window. It also runs a real busy process under the watch for 7 s at 0.1 core: the process stays alive, and it uses less than 3.5 CPU-seconds instead of 7. A short note is in docs/features/cpu-throttle.md.

Not done (the doc notes the same list)

  • Scheduling inside a cell, so the crew run has the core while looks, checks and snapshots wait, with at most one browser.
  • Resuming an interrupted run without starting over or charging twice, and the plain message after two interruptions.
  • Telling the owner on their home page when a forge is stopped for CPU. The stop reason is only in the host log.
  • The small/medium/large timing of the stub journey's first change, and the plan lines that say what each size is for.
  • The stranger-journey test on a small forge, the 3-minute real 1-CPU cell test, and the production probe. The sandbox cannot run cells or reach production for those.

Try it: bun test test/cpu-throttle.test.ts. To check on a host, run a small child forge doing CPU-bound crew work and look for zero child forge stopped lines in the log.

Closes #11539

3 earlier comments
commented

A child forge at its CPU limit is now slowed instead of stopped. This is only the first slice of #11539, so the issue is not done and should stay open. No acceptance probe has run, and I did not run the 3-minute real 1-CPU cell test or the stranger journey. tsc, the gate and the CPU tests (new and existing) pass here; the forge's check is still to come.

What changed

  • Cause: the production log shows the host stopping forges even though cpuQuota: true is already set for child forges. That points to cpu.max not being writable on that host, so the watch fell back to its 30 s check, where cores > limit stopped the cell. I did not check the host's cgroup setup directly.
  • Throttle: the new src/apps/throttle.ts is a token bucket of CPU seconds. A cell gets a free 10 s burst at its limit. Once in debt, its whole process tree is paused (SIGSTOP) and then resumed (SIGCONT) when paid back, so a tree that wants 4 cores on a 1-core forge runs about a quarter of the time. Using exactly the limit is never over it.
  • Where it applies: src/apps/contain.ts applies it to cells with cpuQuota that the kernel is not already capping. Other cells, such as steps, keep the old stop-on-breach watch.
  • Last resort: a throttled cell is stopped only if it averages more than twice its cores over a full 5 minutes. It is also never stopped before it is 10 minutes old.
  • Tests: test/cpu-throttle.test.ts simulates the bucket's duty cycle. It checks that exactly the limit never trips, that the burst runs free, and that the last resort holds back below twice the limit, under 10 minutes of age, and on a short window. It also runs a real busy process under the watch for 7 s at 0.1 core: the process stays alive, and it uses less than 3.5 CPU-seconds instead of 7. A short note is in docs/features/cpu-throttle.md.

Not done (the doc notes the same list)

  • Scheduling inside a cell, so the crew run has the core while looks, checks and snapshots wait, with at most one browser.
  • Resuming an interrupted run without starting over or charging twice, and the plain message after two interruptions.
  • Telling the owner on their home page when a forge is stopped for CPU. The stop reason is only in the host log.
  • The small/medium/large timing of the stub journey's first change, and the plan lines that say what each size is for.
  • The stranger-journey test on a small forge, the 3-minute real 1-CPU cell test, and the production probe. The sandbox cannot run cells or reach production for those.

Try it: bun test test/cpu-throttle.test.ts. To check on a host, run a small child forge doing CPU-bound crew work and look for zero child forge stopped lines in the log.

commented

The preview deploy didn't fail. The log ends done: https://lab--11547--forge-trevato-apps.open-platform.sh, and the preview serves and answers its health check. "No tests declared, none run" is a note in the deploy summary. It appears because forge.json has no app.test. That has been true on main all along, and my pull didn't change forge.json. The forge's check (check.fast, check.verify) runs the tests and decides whether the pull lands. I'm leaving forge.json alone, because adding app.test would make every deploy run the suite a second time.

There is nothing to fix and nothing new to push. The branch is unchanged since the last push at 32439cd. Before that push, tsc and the gate were green, and so were the tests I ran: test/cpu-throttle.test.ts and test/cpu-breach.test.ts (12 pass), plus test/contain.test.ts, test/crew-watch.test.ts and test/contain-label.test.ts (15 pass, 30 skipped as cell tests). If the forge's check returns a failing log, send it and I'll fix it.

The scope is unchanged from the earlier description: this is the first slice of #11539 and the issue stays open. The slice is a bucket that slows a child forge at its CPU limit instead of stopping it, and it stops one only as a last resort. The scheduling inside a cell, run resumption, the owner notice, the size timings and the production probe are not done.

commented

A child forge that reaches its CPU limit is now slowed by a token bucket instead of being stopped, and stopping is kept as the last resort (#11539).

The "pipeline first" gate failed because the pull adds a feature doc, docs/features/cpu-throttle.md, and changes nothing in the pipeline. I didn't touch the pipeline. The change is runtime containment in src/apps/contain.ts and src/apps/throttle.ts, with tests in test/cpu-throttle.test.ts.

I pushed an empty commit carrying the Direction: line the gate accepts, saying the issue asks for runtime behaviour in src/apps, not a pipeline change. I haven't re-run the check here.

I did not run the tests or the type check this turn; the failing check was only the gate rule. The forge's check on this push will show whether anything else is red.

To try it, open the preview at https://lab--11547--forge-trevato-apps.open-platform.sh and run bun test test/cpu-throttle.test.ts.

commented

Parked by the crew: the check stayed red through 2 automatic tries

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs