A check never takes the forge down: each check cell has a real memory cap, and test processes stay small #2426

Each check cell can now run under a kernel memory cap, but it only switches on once the operator makes a cgroup writable for the forge. In production today the forge's /sys/fs/cgroup is mounted read-only and owned by root (I checked from this sandbox, which runs inside the forge pod), so cells there stay watched only and /system says why. The new end-to-end test that allocates past the cap is skipped here and on the forge's check for the same reason, so the kernel kill has not run on any machine yet.

The kernel cap (new src/apps/cgroup.ts)

• Each cell gets its own cgroup with memory.max set to its memory limit, so a cell that goes over is killed by the kernel on its own. Swap is off for it, and the whole cell is killed together.
• The forge moves its own processes into a separate group first, because cgroup v2 won't hand memory to child groups while processes sit in the parent.
• A cell joins its group before it starts, so everything it launches is capped. If it can't join, it exits 125 and does not run uncapped. The path and the command are passed as arguments, never as shell text.
• When the kernel kills a step, the step's error says so: check stopped: it reached its 8192 MiB memory cap, and the kernel killed it. Before, it showed only a bare exit 137.
• It switches on by itself only when the forge is at the root of its container's cgroup; elsewhere FORGE_CELL_CGROUP names the cgroup to use, and off turns it off. This stops bun test on a developer's machine from rearranging their login session.
• On boot it removes cell cgroups left by a forge process that has exited, never those of one still finishing its checks during a roll.
• The once-a-second watch is unchanged and still runs underneath.

Test files per process (scripts/shards.ts, scripts/test-shards.ts)

• splitCapped gives no bun test process more than FILES_PER_PROCESS = 40 files. The runner starts more, smaller processes and runs at most the shard count at once.
• Why 40: peak memory measured here on Bun 1.3.14 (not 1.3.13, and not on the vxrail) was 548 MiB for 40 light files and 549 MiB for 120. The peak follows the heaviest file rather than the file count, so 40 stays well under 1 GiB.
• I did not try --isolate.

/system

• Each cell's memory now shows beside its cap, marked as kernel-capped or watched.
• A line under the machine summary says whether the kernel holds the caps, or why it doesn't.

Tests

• New test/cgroup.test.ts covers the cgroup layout on a stand-in directory, a refused or read-only cgroup, joining before start (same pid, refuses on failure, path never run as script), reading back kernel kills, and the cap on /system.
• The end-to-end case runs two checks side by side: one fills 1 GiB faster than the watch can see under a 96 MiB cap and must fail with the kernel message; the other must pass.
• I widened one existing test, test/contain.test.ts "over its memory limit": where the kernel cap is on, the kernel kill is a correct second reason. The code was not wrong.
• test/shards.test.ts and test/system-telemetry.test.ts gained cases.
• Type check and the touched test files pass, along with the cross files CLAUDE.md lists; the app boots and / and /healthz answer 200.

Left for the operator

• Make the forge pod's cgroup writable by the forge's user, or point FORGE_CELL_CGROUP at a cgroup that is. docs/APPS.md describes how.
• Only once /system reports "kernel cap" should checks.concurrency go back from 6 to 9. I did not change that live setting.

🤖 Generated with Claude Code

Closes #2410

Each check cell can now run under a kernel memory cap, but it only switches on once the operator makes a cgroup writable for the forge. In production today the forge's /sys/fs/cgroup is mounted read-only and owned by root (I checked from this sandbox, which runs inside the forge pod), so cells there stay watched only and /system says why. The new end-to-end test that allocates past the cap is skipped here and on the forge's check for the same reason, so the kernel kill has not run on any machine yet.

The kernel cap (new src/apps/cgroup.ts)

• Each cell gets its own cgroup with memory.max set to its memory limit, so a cell that goes over is killed by the kernel on its own. Swap is off for it, and the whole cell is killed together.
• The forge moves its own processes into a separate group first, because cgroup v2 won't hand memory to child groups while processes sit in the parent.
• A cell joins its group before it starts, so everything it launches is capped. If it can't join, it exits 125 and does not run uncapped. The path and the command are passed as arguments, never as shell text.
• When the kernel kills a step, the step's error says so: check stopped: it reached its 8192 MiB memory cap, and the kernel killed it. Before, it showed only a bare exit 137.
• It switches on by itself only when the forge is at the root of its container's cgroup; elsewhere FORGE_CELL_CGROUP names the cgroup to use, and off turns it off. This stops bun test on a developer's machine from rearranging their login session.
• On boot it removes cell cgroups left by a forge process that has exited, never those of one still finishing its checks during a roll.
• The once-a-second watch is unchanged and still runs underneath.

Test files per process (scripts/shards.ts, scripts/test-shards.ts)

• splitCapped gives no bun test process more than FILES_PER_PROCESS = 40 files. The runner starts more, smaller processes and runs at most the shard count at once.
• Why 40: peak memory measured here on Bun 1.3.14 (not 1.3.13, and not on the vxrail) was 548 MiB for 40 light files and 549 MiB for 120. The peak follows the heaviest file rather than the file count, so 40 stays well under 1 GiB.
• I did not try --isolate.

/system

• Each cell's memory now shows beside its cap, marked as kernel-capped or watched.
• A line under the machine summary says whether the kernel holds the caps, or why it doesn't.

Tests

• New test/cgroup.test.ts covers the cgroup layout on a stand-in directory, a refused or read-only cgroup, joining before start (same pid, refuses on failure, path never run as script), reading back kernel kills, and the cap on /system.
• The end-to-end case runs two checks side by side: one fills 1 GiB faster than the watch can see under a 96 MiB cap and must fail with the kernel message; the other must pass.
• I widened one existing test, test/contain.test.ts "over its memory limit": where the kernel cap is on, the kernel kill is a correct second reason. The code was not wrong.
• test/shards.test.ts and test/system-telemetry.test.ts gained cases.
• Type check and the touched test files pass, along with the cross files CLAUDE.md lists; the app boots and / and /healthz answer 200.

Left for the operator

• Make the forge pod's cgroup writable by the forge's user, or point FORGE_CELL_CGROUP at a cgroup that is. docs/APPS.md describes how.
• Only once /system reports "kernel cap" should checks.concurrency go back from 6 to 9. I did not change that live setting.

🤖 Generated with Claude Code

This pull gives each check cell its own cgroup v2 with memory.max set to the cell's memory limit. Swap is off for the cell (memory.swap.max = 0) and memory.oom.group = 1 is set, so a cell that goes over its cap is OOM-killed as a whole, alone. The cell's process enters that cgroup before bwrap starts. If it cannot enter, it exits with 125 rather than run without a cap. When the kernel kills a cell, that becomes the step's error.

A single bun test process now runs at most 40 files (scripts/shards.ts), which keeps each process's memory small. /system shows each cell's memory beside its cap and says who holds the cap: the kernel, or only the once-a-second watch. It gives the reason when it is the watch.

This does not fix #2410 in production yet. The forge's /sys/fs/cgroup is read-only there, so cells fall back to the watch, and the end-to-end test of the kernel kill is skipped on every machine. #2432 covers giving the forge a writable cgroup and running that test on the forge's check. #2410 now waits for it.

Changes since review:

• Dead code removed: CellCgroup.current() (which read memory.current) and its test lines are gone. maxBytes is now used outside the test: the watch's cap in src/apps/contain.ts is the cgroup's own maxBytes, so the figure /system shows is the one the kernel enforces.
• Terminal: the terminal's System dialog (src/tui/screens/system.ts) uses the same view as /system and the API, and now shows each cell's line as, for example, check:RUN1 3 procs 412 MiB of 8.0 GiB kernel (or watched). test/system-telemetry.test.ts checks that line and that every line fits the dialog's width.

The type check passes, and so do test/cgroup.test.ts, test/system-telemetry.test.ts, test/tui-system.test.ts and test/contain.test.ts. 32 tests in them were skipped in this sandbox: the kernel-kill test because the cgroup is read-only here, and the others because they need a unix socket. The forge's check runs those.

To try it: on the preview, open /system as the owner. #cell-caps says whether the kernel or the watch holds the caps here, and each check row in the table shows its memory as "X of 8.0 GiB". In the terminal, the System command shows the same.

🤖 Generated with Claude Code