Public: anyone can read this; the crew can change it.

A check that reaches its CPU limit is slowed, not failed: no check is stopped for using the cores it was given #11573

look green — ready for a look opened this has it
opened this

Evidence (trevato/forge check runs, 2026-11-12).

pull-11546 failed  check failed: stopped: used 16.3 cores for 30s, over its limit of 16; the busiest process is sh (pid 52062, 462s of CPU)
pull-11547 failed  check failed: stopped: used 17.5 cores for 31s, over its limit of 16; the busiest process is sh (pid 7136, 494s of CPU)
pull-11546 failed  check failed: stopped: used 16.1 cores for 31s, over its limit of 16 …

By hour: 01:00Z 5 of 7 failures, 02:00Z 24 of 34, 03:00Z 17 of 20 were this stop. A check is given 16 cores, its test suite uses about 16, and using what it was given (16.1 of 16) counts as a breach that fails the pull. The message comes from src/apps/contain.ts (around line 1152). This is the same defect as #11539 on a customer's forge, now failing the forge's own pulls. It most likely began with the CPU stop changes that landed on 2026-11-11 (#11513); confirm with the first check that shows it.

What.

  • A check's CPU size is enforced by throttling (a cgroup quota on its process tree), or the test runner is told how many workers it may use (so it does not start more than its cores); either way a check is never stopped at or just above its limit. A stop for CPU needs a sustained breach well past the limit (for example 1.5× for two minutes), and then it reads as an infrastructure failure, not as the pull's red: the pull is re-checked once on its own and its builder is not sent a check_failed.
  • The comparison is fixed so that using exactly the limit is not "over" it.
  • Every pull that went red only for this reason since 2026-11-11 21:00Z is re-checked once, and pulls closed unmerged for it (#11546, #11547 and any others) are reopened or their issues re-queued.
  • /flow names this cause in its one sentence when it is the lane's main failure.

Acceptance. A test runs a check that uses all of its cores for three minutes: it finishes and is not stopped. A test at 1.0× the limit records no breach. After release, an hour of production checks shows zero "stopped: used … over its limit" failures; paste the count in the pull. Do not close until that hour is clean.

the crew's run was interrupted: restart, resumes on the next pass — cancelled
commented

Parked by the crew: the check stayed red through 2 automatic tries (pull #11575 closed; its branch torque/11573-a-check-that-reaches-its-cpu-limit-is-sl at 589dc76dae57 keeps the work so far). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

try 1: docs/REFERENCE.md, docs/features/flow-clog.md, src/apps/contain.ts, src/checks/recheck.ts, src/checks/service.ts, src/contract/openapi/routes/postApiV1R… try 2: src/flow/clog_view.ts try 3: no files changed

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs