Checks at once and shards per check default to what the pod's CPUs hold, so a big machine needs no override #2551

The defaults for checks at once and shards per check now follow the pod's CPU count. On the vxrail's 80 cores that's 4 shards and 6 checks, the values the owner chose, so the sentinel stops reporting them as away from their defaults. The type check passes, and so do the capacity, sentinel, asks, limits, board, pipeline, shards and checks suites. The full check runs on the push.

What changed

• cpuLimit() (src/core/capacity.ts): reads cgroup v2 cpu.max as quota ÷ period, rounded down, at least 1 and never more than the machine has. If there's no quota (max), the file is unreadable or the contents are garbage, it uses os.availableParallelism(). It's built like memoryLimitBytes().
• New defaults (shardsForCpus, checksForCpus): shards are min(4, cpus); checks at once are floor(cpus / (shards × 3)), kept between 1 and 16. So 80 CPUs give 4 and 6, 16 give 4 and 1, and 4 give 4 and 1.
• What still wins: capacityDefault uses these only when the environment variable doesn't set a valid value, and a stored value still overrides both. The sentinel and the settings asks already compare against capacityDefault, so they pick up the new defaults with no change of their own.
• src/main.ts starts the checks service at the CPU-sized default instead of a fixed 1.
• Help lines: both now say how their default is reached.

Things to look at

• Shard count is now always passed down. capacityEnv used to set FORGE_CHECK_SHARDS only when someone had set a value. Now it always passes the shard count in force. Without this, clearing the owner's stored 4 would have sent checks back to one shard per cell CPU (up to 32), so clearing it would have changed behaviour. For a check, the default is sized to what its cell holds (workerCpus(cpuLimit(), cellCpus) from scripts/shards.ts), so a small cell is never handed the pod's count. A stored or environment value is passed as it is, as before.
• Possibly slower check: the forge's own check, in a 16-CPU cell, now defaults to 4 shards; before, it sized itself to the cell (12). If that's too slow, set checks.shards higher.
• Tests pin the CPU count. The test preload calls a new provideCpusForTest(4) so the defaults are the same on any machine. This sandbox reports 48 CPUs, which would have made the stored "4" in the sentinel tests equal the default.
• One existing test changed because it encoded the old behaviour: test/capacity.test.ts expected an empty env, and "" in the change log, after a reset. It now expects the sized default, and clears FORGE_CHECK_SHARDS from the environment while it checks.
• New tests (test/capacity-cpus.test.ts):

- the hand-calculated table for 1, 2, 4, 16, 80 and 1000 CPUs - the environment winning over the CPUs - a stored value overriding both the default and a cell's smaller count - the sentinel ignoring 4 and 6 on 80 CPUs and flagging 8 shards against "default 4" - cpuLimit with fractional, over-machine, max, garbage and missing cpu.max files

Not done

• No cgroup v1 CPU fallback (cpu.cfs_quota_us); the issue asked for v2 only.
• The owner's stored values on production are left in place. Clearing them is now safe and changes nothing on the vxrail.

To try it: bun test test/capacity-cpus.test.ts. On /settings/capacity, clear checks.shards and checks.concurrency: the defaults shown are what this pod's CPUs give.

🤖 Generated with Claude Code

Closes #2549

The defaults for checks at once and shards per check now follow the pod's CPU count. On the vxrail's 80 cores that's 4 shards and 6 checks, the values the owner chose, so the sentinel stops reporting them as away from their defaults. The type check passes, and so do the capacity, sentinel, asks, limits, board, pipeline, shards and checks suites. The full check runs on the push.

What changed

• cpuLimit() (src/core/capacity.ts): reads cgroup v2 cpu.max as quota ÷ period, rounded down, at least 1 and never more than the machine has. If there's no quota (max), the file is unreadable or the contents are garbage, it uses os.availableParallelism(). It's built like memoryLimitBytes().
• New defaults (shardsForCpus, checksForCpus): shards are min(4, cpus); checks at once are floor(cpus / (shards × 3)), kept between 1 and 16. So 80 CPUs give 4 and 6, 16 give 4 and 1, and 4 give 4 and 1.
• What still wins: capacityDefault uses these only when the environment variable doesn't set a valid value, and a stored value still overrides both. The sentinel and the settings asks already compare against capacityDefault, so they pick up the new defaults with no change of their own.
• src/main.ts starts the checks service at the CPU-sized default instead of a fixed 1.
• Help lines: both now say how their default is reached.

Things to look at

• Shard count is now always passed down. capacityEnv used to set FORGE_CHECK_SHARDS only when someone had set a value. Now it always passes the shard count in force. Without this, clearing the owner's stored 4 would have sent checks back to one shard per cell CPU (up to 32), so clearing it would have changed behaviour. For a check, the default is sized to what its cell holds (workerCpus(cpuLimit(), cellCpus) from scripts/shards.ts), so a small cell is never handed the pod's count. A stored or environment value is passed as it is, as before.
• Possibly slower check: the forge's own check, in a 16-CPU cell, now defaults to 4 shards; before, it sized itself to the cell (12). If that's too slow, set checks.shards higher.
• Tests pin the CPU count. The test preload calls a new provideCpusForTest(4) so the defaults are the same on any machine. This sandbox reports 48 CPUs, which would have made the stored "4" in the sentinel tests equal the default.
• One existing test changed because it encoded the old behaviour: test/capacity.test.ts expected an empty env, and "" in the change log, after a reset. It now expects the sized default, and clears FORGE_CHECK_SHARDS from the environment while it checks.
• New tests (test/capacity-cpus.test.ts):

- the hand-calculated table for 1, 2, 4, 16, 80 and 1000 CPUs - the environment winning over the CPUs - a stored value overriding both the default and a cell's smaller count - the sentinel ignoring 4 and 6 on 80 CPUs and flagging 8 shards against "default 4" - cpuLimit with fractional, over-machine, max, garbage and missing cpu.max files

Not done

• No cgroup v1 CPU fallback (cpu.cfs_quota_us); the issue asked for v2 only.
• The owner's stored values on production are left in place. Clearing them is now safe and changes nothing on the vxrail.

To try it: bun test test/capacity-cpus.test.ts. On /settings/capacity, clear checks.shards and checks.concurrency: the defaults shown are what this pod's CPUs give.

🤖 Generated with Claude Code

The defaults for checks at once and shards per check now follow the pod's CPU count. On the vxrail's 80 cores that gives 4 shards and 6 checks, the values the owner chose, so the sentinel stops reporting them as away from their defaults. This push also fixes the restore race behind the last red check. The full check hasn't run on it yet.

The CPU-sized defaults

• cpuLimit() (src/core/capacity.ts): reads cgroup v2 cpu.max as quota ÷ period, rounded down, at least 1 and never more than the machine has. If there's no quota (max), the file is unreadable or the contents are garbage, it uses os.availableParallelism().
• New defaults (shardsForCpus, checksForCpus): shards are min(4, cpus); checks at once are floor(cpus / (shards × 3)), kept between 1 and 16. So 80 CPUs give 4 and 6, 16 give 4 and 1, and 4 give 4 and 1.
• What still wins: the environment variable, then a stored value. The sentinel and the settings asks already compare against capacityDefault, so they use the new defaults with no change.
• src/main.ts starts the checks service at the CPU-sized default instead of a fixed 1.
• Help lines: both now say how their default is reached.

Things to look at

• Shard count is always passed down now. capacityEnv used to set FORGE_CHECK_SHARDS only when someone had set a value; now it always passes the count in force. Without this, clearing the owner's stored 4 would have sent checks back to one shard per cell CPU. A check's default is sized to what its cell holds (workerCpus(cpuLimit(), cellCpus)), so a small cell is never handed the pod's count.
• Possibly slower check: the forge's own check, in a 16-CPU cell, now defaults to 4 shards instead of the 12 it used to give itself. If that's too slow, set checks.shards higher.
• Tests pin the CPU count. The test preload calls a new provideCpusForTest(4), so the defaults are the same on any machine.
• One existing test changed because it encoded the old behaviour: test/capacity.test.ts expected an empty capacityEnv and "" in the change log after a reset. It now expects the sized default.
• New tests (test/capacity-cpus.test.ts):

- the hand-calculated table for 1, 2, 4, 16, 80 and 1000 CPUs - the environment winning over the CPUs - a stored value overriding both the default and a cell's smaller count - the sentinel ignoring 4 and 6 on 80 CPUs and flagging 8 shards against "default 4" - cpuLimit with fractional, over-machine, max, garbage and missing cpu.max files

The restore race that failed the check

• Cause: "repositories come back in the manifest's order" failed because of a real race in src/snapshot/restore.ts, which is on main too. Repositories are cloned four at a time, and the restore threw whichever clone failed first. On a loaded runner alice/second could fail before alice/first.
• Fix: after a failure no new clones start, the ones already running finish, and the error names the failure earliest in the manifest.
• Test: a new one in test/snapshot-restore-parallel.test.ts makes the first clone fail last. It fails without the fix and passes with it.
• Shard 14 is unexplained. It was stopped (exit 143) without naming a test. That check ran on production's main code, so this branch didn't choose its shard count, and I made no change for it.

Not done

• No cgroup v1 CPU fallback (cpu.cfs_quota_us); the issue asked for v2 only.
• The owner's stored values on production are left in place. Clearing them is now safe and changes nothing on the vxrail.

To try it: bun test test/capacity-cpus.test.ts test/snapshot-restore-parallel.test.ts. Both pass here, as does the type check. Then, on /settings/capacity, clear checks.shards and checks.concurrency: the defaults shown are what this pod's CPUs give.

🤖 Generated with Claude Code

The defaults for checks at once and shards per check now follow the pod's CPU count. On the vxrail's 80 cores that gives 4 shards and 6 checks, the values the owner chose, so the sentinel stops reporting them as away from their defaults. The branch now has main merged in. After the merge, the type check passes and 144 tests across the capacity, sentinel, asks, limits, board, pipeline, checks and snapshot-restore suites pass; the forge's full check runs on this push.

The CPU-sized defaults

• cpuLimit() (src/core/capacity.ts): reads cgroup v2 cpu.max as quota ÷ period, rounded down, at least 1 and never more than the machine has. If there's no quota (max), the file is unreadable or the contents are garbage, it uses os.availableParallelism().
• New defaults (shardsForCpus, checksForCpus): shards are min(4, cpus); checks at once are floor(cpus / (shards × 3)), kept between 1 and 16. So 80 CPUs give 4 and 6, 16 give 4 and 1, and 4 give 4 and 1.
• What still wins: the environment variable, then a stored value. The sentinel and the settings asks already compare against capacityDefault, so they use the new defaults with no change.
• src/main.ts starts the checks service at the CPU-sized default instead of a fixed 1.
• Help lines: both now say how their default is reached.

Merging main

• capacityDefault now holds both sides: main's crew concurrency sized from memory (crewForMemory) and this branch's check settings sized from CPUs.
• src/main.ts: main had added the same capacityDefault import, so the duplicate is gone.
• The restore race: main fixed the same race this branch had fixed (it named the clone that failed first, not the earliest in the manifest), so src/snapshot/restore.ts is main's version. The test added here, where clones fail at once, is kept and passes against main's code.

Things to look at

• Shard count is always passed down now. capacityEnv used to set FORGE_CHECK_SHARDS only when someone had set a value; now it always passes the count in force. Without this, clearing the owner's stored 4 would have sent checks back to one shard per cell CPU. A check's default is sized to what its cell holds (workerCpus(cpuLimit(), cellCpus)), so a small cell is never handed the pod's count.
• Possibly slower check: the forge's own check, in a 16-CPU cell, now defaults to 4 shards instead of the 12 it used to give itself. If that's too slow, set checks.shards higher.
• Tests pin the CPU count. The test preload calls a new provideCpusForTest(4), so the defaults are the same on any machine.
• One existing test changed because it encoded the old behaviour: test/capacity.test.ts expected an empty capacityEnv and "" in the change log after a reset. It now expects the sized default.
• New tests (test/capacity-cpus.test.ts):

- the hand-calculated table for 1, 2, 4, 16, 80 and 1000 CPUs - the environment winning over the CPUs - a stored value overriding both the default and a cell's smaller count - the sentinel ignoring 4 and 6 on 80 CPUs and flagging 8 shards against "default 4" - cpuLimit with fractional, over-machine, max, garbage and missing cpu.max files

Not done

• No cgroup v1 CPU fallback (cpu.cfs_quota_us); the issue asked for v2 only.
• The owner's stored values on production are left in place. Clearing them is now safe and changes nothing on the vxrail.

To try it: bun test test/capacity-cpus.test.ts. Then, on /settings/capacity, clear checks.shards and checks.concurrency: the defaults shown are what this pod's CPUs give.

🤖 Generated with Claude Code