Cause of #3116 and #3105: main's own check is stopped for memory, so main goes red and every pull's check is held behind it.
Evidence (run logs, 2026-10-20)
Six main checks were stopped between 03:16 and 04:26 at 4105–4668 MiB against the 4096 MiB cell cap: 9f67d37d, 8a963c51, e9d5eb78, 42df069e, c9aa7d6c, 83bef118. Three ran alone and still went over, so this isn't two checks fighting for memory.
Each one is test-shards: measuring ~630 files under coverage, one bun test each, 4 at a time, and each is stopped at about 3 min, right when shards 36–40 run together: snapshot-uplink, looks, apps-slow-test, apps-adversarial-teardown and the crew-* suites. The runs that pass print the same "4 at a time" and peak just under the cap.
The 4 comes from the CPU limit, not memory.shardCount in scripts/shards.ts takes the least of workerCpus(…, FORGE_CELL_CPUS) and memory / WORKER_MEMORY_BYTES. At 512 MiB a worker, even the real 4096 MiB cap allows 8 workers, which is more than 4. So #3111 (resolving the declared 8192 MiB to the granted 4096 MiB) is right but leaves the count at 4 and doesn't fix this on its own.
WORKER_MEMORY_BYTES (512 MiB) is the figure for a plain bun test. A worker under --coverage that starts forges and apps uses about twice that. The cell measures resident memory summed across the process tree (apps/contain.tstick).
Mechanical fix
scripts/shards.ts: give a measuring run (--impact, one file per process under coverage) its own per-worker figure, about 1.25 GiB from the peaks above. Pass it to shardCount from scripts/test-shards.ts when measure is set, against the cell's resolved memory (the #3111 change). At 4096 MiB that gives 3 workers.
Test in test/shards.test.ts: 16 CPUs, FORGE_CELL_CPUS giving 4, and 4096 MiB measuring gives fewer than 4 shards. The same inputs not measuring still give 4. The test fails without the change.
When fixed: three main checks in a row finish without a "used … MiB of memory" stop, and red:01m23vygyffx5hg2f5vm8r74sr and queue:check leave the degraded list.
Cause of #3116 and #3105: main's own check is stopped for memory, so main goes red and every pull's check is held behind it.
Evidence (run logs, 2026-10-20)
test-shards: measuring ~630 files under coverage, one bun test each, 4 at a time, and each is stopped at about 3 min, right when shards 36–40 run together:snapshot-uplink,looks,apps-slow-test,apps-adversarial-teardownand thecrew-*suites. The runs that pass print the same "4 at a time" and peak just under the cap.shardCountinscripts/shards.tstakes the least ofworkerCpus(…, FORGE_CELL_CPUS)andmemory / WORKER_MEMORY_BYTES. At 512 MiB a worker, even the real 4096 MiB cap allows 8 workers, which is more than 4. So #3111 (resolving the declared 8192 MiB to the granted 4096 MiB) is right but leaves the count at 4 and doesn't fix this on its own.bun test. A worker under--coveragethat starts forges and apps uses about twice that. The cell measures resident memory summed across the process tree (apps/contain.tstick).Mechanical fix
scripts/shards.ts: give a measuring run (--impact, one file per process under coverage) its own per-worker figure, about 1.25 GiB from the peaks above. Pass it toshardCountfromscripts/test-shards.tswhenmeasureis set, against the cell's resolved memory (the #3111 change). At 4096 MiB that gives 3 workers.test/shards.test.ts: 16 CPUs,FORGE_CELL_CPUSgiving 4, and 4096 MiB measuring gives fewer than 4 shards. The same inputs not measuring still give 4. The test fails without the change.When fixed: three main checks in a row finish without a "used … MiB of memory" stop, and
red:01m23vygyffx5hg2f5vm8r74srandqueue:checkleave the degraded list.