A check that reaches its CPU limit is slowed, not failed: no check is stopped for using the cores it was given #11575

Checks are no longer failed for using the cores they were given. The CPU watch now stops a cell only when it runs past 1.5× its limit for two minutes straight, and even then a check is re-queued once as an infrastructure failure instead of going red.

Why it happened. Checks run where the kernel's CPU cap (cpu.max) isn't applied, so only the watch in src/apps/contain.ts enforces cores. It stopped a cell when cores > limit over a 30 s average, so 16.1 of 16 cores counted as a breach. That comparison predates #11513, which only changed the message (it added the busiest-process name). So the stops are likely not a regression from #11513, but I haven't found the first check that shows them, which the issue asked for.

What changed

• The watch (src/apps/contain.ts): new CPU_STOP = { factor: 1.5, sustainMs: 120_000 }, plus cpuOver and cpuTick as plain functions. A dip back under the line restarts the two-minute count. Using exactly the limit is not "over" it. The message now reads over 1.5× its limit of N. Cells under cpu.max are still throttled and never stopped. This rule covers every watched cell, apps included, not just checks.
• Infrastructure, not red (src/checks/service.ts, src/core/runs.ts): when the runner's own stop line (<step> stopped: it used … cores …) ends a check, the check is cancelled with CPU_STOP_SUMMARY and queued once more. There is no failed-test re-run and no check_failed. A second stop in a row fails as before. The match is anchored to the start of the runner's error, and a step's own output only appears after "exited with code", so a pull can't print its way into a re-check.
• Re-checking pulls already red: POST /api/v1/repos/:owner/:name/checks/recheck also takes {"cause":"cpu"}. It re-asks every open pull whose latest check on its head failed only on a CPU stop, matching both the old and new wording. Any other cause is a 400 invalid_cause. It's admin only, uses the same-origin check and allows at most 50 per call. The OpenAPI route and docs/REFERENCE.md are updated.
• /flow: the clog sentence leads with "N open pulls went red only because the CPU watch stopped their checks" when that's at least half of the open red pulls and more than one (redCauses, cpuIsMain in src/flow/clog_view.ts).

Tests

• test/cpu-breach.test.ts: three minutes of ticks at 16 of 16, and at jittering values just above 16, are never stopped. 1.0×, 16.1 and 17.5 of 16 record no breach. 2× is stopped at 120 s, and a dip restarts the count. Also covers matching the stop line and the stored summaries.
• test/checks-recheck.test.ts: the cause path, including the 400 and 404 cases.
• test/flow-clog-cpu.test.ts: the /flow sentence.
• test/contain.test.ts: the real-cell CPU test now expects the new line and that cpuStopped recognises it.

Not done

• No pulls were re-checked or reopened, including #11546 and #11547. Doing it before this ships would just hit the old watch again. After release, an admin runs {"cause":"cpu"} for open pulls. Pulls closed unmerged for this need a person to reopen them or re-queue their issues; this change doesn't automate that.
• The issue's acceptance, an hour of production checks with zero "stopped: used … over its limit" failures, can only be counted after release. The issue should stay open until that count is pasted here.
• The keep_plan tool wasn't available in this run, so the issue has no plan posted.

Run here: bun scripts/gate.ts, bun x tsc --noEmit, and 57 test files covering limits, checks, recheck, flow-clog and OpenAPI: 687 pass, 0 fail. 31 tests were skipped, among them the suites that need real cells (the cell limits in contain.test.ts, the cgroup ones), because this sandbox uses a loopback socket and can't run them. The forge's check runs them on this push. scripts/fast.ts named 2356 files, too many to run here, so I picked the files above.

Try: on the preview, as an admin, POST /api/v1/repos/trevato/forge/checks/recheck with {"cause":"cpu"} returns {queued: [...]}, and /flow leads with the CPU stops while they're most of the red.

🤖 Generated with Claude Code

Closes #11573

Checks are no longer failed for using the cores they were given. The CPU watch now stops a cell only when it runs past 1.5× its limit for two minutes straight, and even then a check is re-queued once as an infrastructure failure instead of going red.

Why it happened. Checks run where the kernel's CPU cap (cpu.max) isn't applied, so only the watch in src/apps/contain.ts enforces cores. It stopped a cell when cores > limit over a 30 s average, so 16.1 of 16 cores counted as a breach. That comparison predates #11513, which only changed the message (it added the busiest-process name). So the stops are likely not a regression from #11513, but I haven't found the first check that shows them, which the issue asked for.

What changed

• The watch (src/apps/contain.ts): new CPU_STOP = { factor: 1.5, sustainMs: 120_000 }, plus cpuOver and cpuTick as plain functions. A dip back under the line restarts the two-minute count. Using exactly the limit is not "over" it. The message now reads over 1.5× its limit of N. Cells under cpu.max are still throttled and never stopped. This rule covers every watched cell, apps included, not just checks.
• Infrastructure, not red (src/checks/service.ts, src/core/runs.ts): when the runner's own stop line (<step> stopped: it used … cores …) ends a check, the check is cancelled with CPU_STOP_SUMMARY and queued once more. There is no failed-test re-run and no check_failed. A second stop in a row fails as before. The match is anchored to the start of the runner's error, and a step's own output only appears after "exited with code", so a pull can't print its way into a re-check.
• Re-checking pulls already red: POST /api/v1/repos/:owner/:name/checks/recheck also takes {"cause":"cpu"}. It re-asks every open pull whose latest check on its head failed only on a CPU stop, matching both the old and new wording. Any other cause is a 400 invalid_cause. It's admin only, uses the same-origin check and allows at most 50 per call. The OpenAPI route and docs/REFERENCE.md are updated.
• /flow: the clog sentence leads with "N open pulls went red only because the CPU watch stopped their checks" when that's at least half of the open red pulls and more than one (redCauses, cpuIsMain in src/flow/clog_view.ts).

Tests

• test/cpu-breach.test.ts: three minutes of ticks at 16 of 16, and at jittering values just above 16, are never stopped. 1.0×, 16.1 and 17.5 of 16 record no breach. 2× is stopped at 120 s, and a dip restarts the count. Also covers matching the stop line and the stored summaries.
• test/checks-recheck.test.ts: the cause path, including the 400 and 404 cases.
• test/flow-clog-cpu.test.ts: the /flow sentence.
• test/contain.test.ts: the real-cell CPU test now expects the new line and that cpuStopped recognises it.

Not done

• No pulls were re-checked or reopened, including #11546 and #11547. Doing it before this ships would just hit the old watch again. After release, an admin runs {"cause":"cpu"} for open pulls. Pulls closed unmerged for this need a person to reopen them or re-queue their issues; this change doesn't automate that.
• The issue's acceptance, an hour of production checks with zero "stopped: used … over its limit" failures, can only be counted after release. The issue should stay open until that count is pasted here.
• The keep_plan tool wasn't available in this run, so the issue has no plan posted.

Run here: bun scripts/gate.ts, bun x tsc --noEmit, and 57 test files covering limits, checks, recheck, flow-clog and OpenAPI: 687 pass, 0 fail. 31 tests were skipped, among them the suites that need real cells (the cell limits in contain.test.ts, the cgroup ones), because this sandbox uses a loopback socket and can't run them. The forge's check runs them on this push. scripts/fast.ts named 2356 files, too many to run here, so I picked the files above.

Try: on the preview, as an admin, POST /api/v1/repos/trevato/forge/checks/recheck with {"cause":"cpu"} returns {queued: [...]}, and /flow leads with the CPU stops while they're most of the red.

🤖 Generated with Claude Code

A check that the CPU watch stops is no longer red for the pull's code (#11573).

• When the watch stops a cell: only once the cell's 30 s average CPU has stayed above 1.5× its limit for two minutes. A cell using the cores it was given, such as 16.1 of 16, is never stopped. The rule is cpuOver and cpuTick in src/apps/contain.ts.
• What happens to a stopped check: cpuStopped recognises the runner's own stop line, matched only at the start of the error, so a step's output can't fake one. The check is then treated as an infrastructure failure and queued once more (src/checks/service.ts); its builder is not told.
• Pulls already red from a stop: an admin can re-check all of them with {cause: "cpu"} on the re-check endpoint.
• On /flow: the one sentence about what clogs the lane names the CPU watch's stops when they cause most of the red pulls.

Look at cpuTick's reset when the average drops back under the line, and at the anchored pattern in cpuStopped.

Not done: the kernel's own CPU limit is unchanged, and so is the memory, disk and process-count side of the watch.

The last check failed on two things:

• rule header-width: the header comment of src/flow/clog_view.ts ran past 160 columns. It is now wrapped and that test passes.
• test/sentinel-shared-settle.test.ts: 15 tests failed on shard 91, which the runner marked as failing only beside other files. This pull doesn't touch the sentinel. Here the file passes on its own, and three times beside the four test files this pull changes. I couldn't reproduce the failure, so I left it alone; if it fails again on the forge's check, it needs its own issue.

scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 2410 test files for this diff, too many to run here, so I ran the 15 files that cover the change and they all pass (header-width, rule-tests-always, contain, cpu-breach, checks-recheck, flow-clog*, sentinel-shared-settle).

To try it: bun test test/cpu-breach.test.ts test/checks-recheck.test.ts test/flow-clog-cpu.test.ts. The cpu-breach tests feed three minutes of watch readings through cpuTick without starting a process.

🤖 Generated with Claude Code

A check that the CPU watch stops is no longer red for the pull's code (#11573).

• When the watch stops a cell: only once the cell's 30 s average CPU has stayed above 1.5× its limit for two minutes. A cell using the cores it was given, such as 16.1 of 16, is never stopped. The rule is cpuOver and cpuTick in src/apps/contain.ts.
• What happens to a stopped check: cpuStopped recognises the runner's own stop line, matched only at the start of the error, so a step's output can't fake one. The check is then treated as an infrastructure failure and queued once more (src/checks/service.ts); its builder is not told.
• Pulls already red from a stop: an admin can re-check all of them with {cause: "cpu"} on the re-check endpoint.
• On /flow: the one sentence about what clogs the lane names the CPU watch's stops when they cause most of the red pulls.
• Header comment: src/flow/clog_view.ts's header is wrapped to the 160-column limit.

Look at cpuTick's reset when the average drops back under the line, and at the anchored pattern in cpuStopped.

Not done: the kernel's own CPU limit is unchanged, and so is the memory, disk and process-count side of the watch.

The last check failed on this exact bug: "stopped: used 16.3 cores for 31s, over its limit of 16". That wording comes from main's watch; this pull's watch says "over 1.5× its limit". Nothing in the pull needed changing, so the newest commit is empty and only runs the check again. Until this merges, any check on the forge, this one included, can still be stopped the same way.

An earlier check also failed 15 tests in test/sentinel-shared-settle.test.ts on one shard, marked as failing only beside other files. This pull doesn't touch the sentinel. Here the file passes on its own, and three times beside the four test files this pull changes; if it fails again on the forge, it needs its own issue.

scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 2410 test files for this diff, too many to run here, so I ran the 15 files that cover the change and they all pass (header-width, rule-tests-always, contain, cpu-breach, checks-recheck, flow-clog*, sentinel-shared-settle).

To try it: bun test test/cpu-breach.test.ts test/checks-recheck.test.ts test/flow-clog-cpu.test.ts. The cpu-breach tests feed three minutes of watch readings through cpuTick without starting a process.

🤖 Generated with Claude Code

Parked by the crew: the check stayed red through 2 automatic tries