Sentinel: red:01m23vygyffx5hg2f5vm8r74sr was degraded 122 minutes today: trevato/forge: main is red at 1aecf60 — stopped: used 16.1 cores for 30s, over its limit of 16; 5 pull checks wait for … #2162

The gate's test runner now sizes itself to the check's limits as declared in forge.json, not to the host's core count. Before, main's check was being stopped at 1aecf60 ("used 16.1 cores for 30s, over its limit of 16"), and every pull check queued behind it.

Cause. scripts/test-shards.ts ran one worker per CPU that availableParallelism() reports. That is the host's count, not the limit of the cell the check runs in (the sandbox the forge gives each check). The cell's limit is enforced by a watchdog in src/apps/contain.ts, which stops a check whose processes average more than check.limits.cpus cores over 30 s. Nothing in /proc or /sys shows that limit. So the gate ran 16 or more workers against a 16-core limit. Each worker goes above one core at its peaks (Bun's own threads, git children, test forges it boots), so the 30 s average crossed 16.

Change.

• scripts/shards.ts: a new pure workerCpus(machine, declared) caps workers at CELL_CPU_SHARE (3/4) of the declared CPUs. It never goes above the machine's count or below one, and with no declared limit it uses the machine's count. On the 16-CPU cell the gate now runs 12 workers at once.
• scripts/test-shards.ts: reads check.limits from forge.json with the forge's own parseCheck and passes the CPUs through workerCpus. The declared memory (8 GiB) joins the cgroup limits that memoryLimit already reads, because the same watchdog enforces it. At 512 MiB per worker that still allows 16, so it doesn't change the count today.
• scripts/gate-summary.ts: doc comments only. The summary's "N CPUs" is now the count the workers were sized to.
• FORGE_CHECK_SHARDS still overrides all of this.

Tests. test/shards.test.ts has two new tests:

• 12 workers for a 64-core host in a 16-CPU cell, plus the edge cases.
• The repo's real forge.json check runs fewer workers than it declares CPUs.

No existing test changed. shardCount behaves as before, and the existing "16 CPUs and 8 GiB run 16" case still holds for the function on its own. Type check is clean, and bun test test/shards.test.ts test/gate-summary.test.ts passes (55/55).

Not done.

• The watchdog still stops a check rather than slowing it down. Changing what a breach does is a forge-wide behaviour change, not part of this fix.
• The forge doesn't hand checks their limits through the environment. Reading forge.json works on the next push without the forge having to deploy first.

To try it: the next full check's log should start test-shards: N files, one bun test each, 12 at a time. The sentinel's mark is red:01m23vygyffx5hg2f5vm8r74sr staying off the degraded list for a day; 3/4 is my estimate of enough headroom, not a measured value.

🤖 Generated with Claude Code

Closes #2161

Test workers are now sized to 3/4 of the check cell's declared CPUs, so the check stops using more CPU than its cell allows: one worker per declared CPU averaged 16.1 of 16 cores and stopped main (#2161). This merge also brings in main's change #2260 for the same problem, and both limits now apply together.

• What this branch does: workerCpus (in scripts/shards.ts) caps the CPU count at floor(check.limits.cpus × 3/4) from forge.json, never below 1. The memory bound also counts the cell's declared memoryMiB.
• What #2260 added on main: shardCountFromEnv reads FORGE_CELL_CPUS, which the forge sets for a check, and caps the shard count at one less than that number. It still honours the FORGE_CHECK_SHARDS override.
• The conflict: both sides changed the import list in scripts/test-shards.ts. The script now imports workerCpus and shardCountFromEnv, and passes the workerCpus result into shardCountFromEnv. The plain shardCount import is gone because nothing uses it. The two caps are separate, so the lower one wins.

No tests changed. The type check is clean, and test/shards.test.ts and test/check-tiers.test.ts pass (92 tests). I didn't run the full suite; the forge's check runs it on this push.

🤖 Generated with Claude Code

The full gate on land was red; the pull is open again. Red tests:

• what the open web expects > releases.json is the shipped releases, newest first, in the shape a child reads
• what the open web expects > releases.json is the shipped releases, newest first, in the shape a child reads [1.34ms] [shard 161]

The gate now gets the cell's CPU count from one place, FORGE_CELL_CPUS, and runs 3/4 that many workers. The 16-CPU check cell gets 12, not the 16 that averaged 16.1 cores and stopped main (#2161). This also fixes the red releases.json test, which broke on main itself.

• One source for the CPU count (@juniper 1): scripts/test-shards.ts no longer reads check.limits.cpus from forge.json. It passes availableParallelism() to shardCountFromEnv, and shardCount caps on FORGE_CELL_CPUS via workerCpus instead of cpus − 1. Across all cell sizes, floor(3/4 × cpus) is never higher than cpus − 1, so it replaces that cap rather than adding a second one. forge.json is still read for the cell's declared memory: nothing passes that in the environment, and /sys may not show it.
• Test for the combined cap (@juniper 2): a new test in test/shards.test.ts pins that the lowest bound wins. For a 16-CPU cell on a 64-core host the result is 12. A smaller machine (8), less memory (4 GiB → 8) or fewer files (5) gives a lower count, and FORGE_CHECK_SHARDS still wins over all of them.
• One expectation in main's tests changed on purpose: the #2257 test expected shardCount(80, 400, …, "16") to give 15. The new rule gives 12, and 15 workers still peak past 16 cores. The "4" → 3 case and main's other FORGE_CELL_CPUS tests are unchanged.
• The red test (@trevato): release 0.7.152 brought releases.json to 101 entries, but a child reads at most MAX_RELEASES (100). The test in test/standards.test.ts was wrong in assuming a child reads every entry. It now checks that the server sends the whole list, that a child reads the newest min(n, MAX_RELEASES), and that the entries past 100 are in the same shape. MAX_RELEASES is now exported from src/core/parent_releases.ts for this; the endpoint's behaviour is unchanged.

The type check is clean, and the shards, check-tiers, gate-summary and standards suites pass (112 tests). I didn't run the full suite; the forge's check runs it on this push.

🤖 Generated with Claude Code

The test runner now gives the check 3/4 of its cell's CPU limit as workers, read only from FORGE_CELL_CPUS. On the 16-CPU check cell it runs 12 at once. With one worker per CPU, main's check averaged 16.1 of 16 cores over 30 s and the cell's watchdog in src/apps/contain.ts stopped it (#2161). This also fixes the releases.json test that went red on main at release 0.7.152.

The failed check ran on the previous head, 49ef6efe, and its only failure was that releases.json test. The fix was already in the last push (309a8679), so nothing new was pushed for this round, and that head's check is still queued. Here, the shards, check-tiers and standards suites pass (102 tests), as do the 37 tests that touch parent_releases.

What to look at:

• Worker count: scripts/test-shards.ts passes availableParallelism() to shardCountFromEnv and no longer reads the CPU count from forge.json. In scripts/shards.ts, shardCount now caps on FORGE_CELL_CPUS through workerCpus instead of cpus − 1. floor(3/4 × cpus) is never higher than cpus − 1, so it replaces that cap rather than adding a second one.
• Memory: forge.json is still read for the cell's declared memory, which nothing passes in the environment. It joins the cgroup limits in memoryLimit.
• New test for the combined cap: in test/shards.test.ts, the lowest bound wins. A 16-CPU cell on a 64-core host gives 12; a smaller machine, less memory or fewer files gives less; FORGE_CHECK_SHARDS still overrides everything.
• One existing test changed on purpose: the #2257 test expected 15 workers for FORGE_CELL_CPUS=16; it now expects 12, because 15 workers still peak past 16 cores. Its other expectations are unchanged.
• The releases.json test was wrong: it assumed a child reads every release, and release 0.7.152 made 101 of them while a child reads at most MAX_RELEASES (100). test/standards.test.ts now checks that the server sends the whole list, a child reads the newest min(n, MAX_RELEASES), and the entries past 100 have the same shape. MAX_RELEASES is now exported from src/core/parent_releases.ts; the endpoint itself is unchanged.

To try it: bun test test/shards.test.ts test/standards.test.ts. On the check cell, the gate summary shows 12 shards.

🤖 Generated with Claude Code

The test runner now gives the check 3/4 of its cell's CPU limit as workers, read only from FORGE_CELL_CPUS. On the 16-CPU check cell it runs 12 at once. With one worker per CPU, main's check averaged 16.1 of 16 cores over 30 s and the cell's watchdog in src/apps/contain.ts stopped it (#2161).

This push merges main again. Main had fixed the same releases.json test independently, and that is where the two conflicts were:

• src/core/parent_releases.ts: both sides exported MAX_RELEASES; main's doc comment on it is kept.
• test/standards.test.ts: main's assertion is kept, that a child reads the newest min(n, MAX_RELEASES) while the record keeps every release. This branch's checks are kept too: the server sends the whole list, and the entries past the newest 100 have the same shape a child reads.

The type check is clean. The standards, shards, check-tiers and parent_releases suites pass (130 tests); I didn't run the full suite, which the forge's check runs on this push.

What to look at:

• Worker count: scripts/test-shards.ts passes availableParallelism() to shardCountFromEnv and no longer reads the CPU count from forge.json. In scripts/shards.ts, shardCount now caps on FORGE_CELL_CPUS through workerCpus instead of cpus − 1. floor(3/4 × cpus) is never higher than cpus − 1, so it replaces that cap rather than adding a second one.
• Memory: forge.json is still read for the cell's declared memory, which nothing passes in the environment. It joins the cgroup limits in memoryLimit.
• New test for the combined cap: in test/shards.test.ts, the lowest bound wins. A 16-CPU cell on a 64-core host gives 12; a smaller machine, less memory or fewer files gives less; FORGE_CHECK_SHARDS still overrides everything.
• One existing test changed on purpose: the #2257 test expected 15 workers for FORGE_CELL_CPUS=16; it now expects 12, because 15 workers still peak past 16 cores. Its other expectations are unchanged.

To try it: bun test test/shards.test.ts test/standards.test.ts. On the check cell, the gate summary shows 12 shards.

🤖 Generated with Claude Code