Public: anyone can read this; the crew can change it.

A forge that reaches its CPU limit is slowed, not stopped: a crew run on a 1-CPU forge finishes #11539

Closed opened this
opened this

Evidence (host log, production 0.7.540, forge probe-kqzv, size small = 1 CPU).

23:36:00Z child forge stopped name=probe-kqzv why="used 2.9 cores for 31s, over its limit of 1; the busiest process is bwrap (pid 2488834, 44s of CPU)"
23:36:20Z [crew] crew is up …
23:37:29Z child forge stopped … why="used 1.0 cores for 31s, over its limit of 1; the busiest process is bun (pid 2557579, 19s of CPU)"
23:37:53Z forge is up … restarts=2 unclean=2
23:38:53Z [crew] run queued again … why="interrupted by a restart; queued again"
23:39:01Z child forge stopped … why="used 1.0 cores for 31s, over its limit of 1; the busiest process is bun"
23:40:35Z child forge stopped … why="used 1.1 cores for 31s, over its limit of 1 …"
23:41:00Z forge is up … restarts=4 unclean=4

The busiest process is the crew's own sandbox (bwrap), then the forge's bun, with chromium running beside them. A crew run is the product, and it uses a core or more by design. The host answers by stopping the whole forge after 31 s, the run is interrupted and queued again, and the cycle repeats about every 90 s until the credit is gone. Note used 1.0 cores … over its limit of 1: using exactly the limit counts as over it. The looks fix (#11509) is live and is not the cause here.

What.

  • Enforce a cell's CPU size by throttling (a cgroup cpu.max quota on the cell's process tree, or the nearest equivalent where cgroups are not writable), so a forge at its limit runs slower and nothing is killed.
  • Stopping for CPU remains only as a last resort for a forge the throttle cannot hold (for example more than twice its limit for five minutes), never at or near the limit, and never for a forge younger than ten minutes. When it happens the forge's owner is told on their home page what happened and why.
  • Inside a cell, work is scheduled to fit its size: on 1 CPU the crew run has the core; looks, checks and snapshots wait until no run is active, and at most one browser exists at a time.
  • A run interrupted by a host stop does not start over from nothing and does not charge the credit twice for the same work; after two interruptions in a row the forge says so plainly instead of looping.
  • Sizes tell the truth on the plans: if a size cannot run the crew at a usable speed even when throttled, the plan's line says what it is for. Measure it: time for the stub journey's first change on small, medium and large.

Acceptance.

  • A test on a real 1-CPU cell runs a CPU-bound sandboxed job for three minutes: the cell is never stopped, its CPU stays at its quota, and the job finishes.
  • The stranger journey (#11158) passes on a small forge: first ask to a built change, with zero child forge stopped lines for that forge.
  • A production probe after release on a throwaway small forge shows the same in the host's log; paste the lines in the pull. Do not close this until that probe passes.
commented

Parked by the crew: the check stayed red through 2 automatic tries (pull #11540 closed; its branch torque/11539-a-forge-that-reaches-its-cpu-limit-is-sl at b282d42548d5 keeps the work so far). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

try 1: docs/features/cpu-throttle.md, src/apps/contain.ts, src/apps/throttle.ts, src/checks/service.ts, src/crew/autonomy.ts, src/crew/dispatch.ts, src/crew/in… try 2: src/apps/contain.ts, test/cpu-throttle.test.ts try 3: src/core/capacity.ts, test/shards.test.ts

commented

Parked by the crew: the check stayed red through 2 automatic tries (pull #11547 closed; its branch scout/11539-a-forge-that-reaches-its-cpu-limit-is-sl at 7c25c46b64b3 keeps the work so far).

Parked twice since a person last spoke here, so it is the planner's now. Every reason:

  1. the check stayed red through 2 automatic tries (pull #11540 closed; its branch torque/11539-a-forge-that-reaches-its-cpu-limit-is-sl at b282d42548d5 keeps the work so far)
  2. the check stayed red through 2 automatic tries (pull #11547 closed; its branch scout/11539-a-forge-that-reaches-its-cpu-limit-is-sl at 7c25c46b64b3 keeps the work so far)

The planner splits it, replaces it with a fresh issue, or defers it; a person can also say what to try. Nothing starts here until then, and the board lists it.

try 1: docs/features/cpu-throttle.md, src/apps/contain.ts, src/apps/throttle.ts, test/cpu-throttle.test.ts try 2: no files changed

commented

Split into #11558 (the breach rule: never at the limit, never in the first ten minutes, only past twice the limit for five minutes), #11559 (where cpu.max cannot be written, the forge pauses and resumes the tree instead of stopping it — the likely production cause, since child forges already ask for cpu.max), #11560 (two interruptions in a row stop the requeue loop and charge once), #11561 (on 1 CPU the crew run has the core), and #11562 behind the Later gate (sizes measured on the plans page). The production probe you asked for is the acceptance of #11559, for a person to run after release. The parked branches keep the earlier attempts for reference.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs