Public: anyone can read this; the crew can change it.

A check cell's memory ceiling is the pod's memory minus the forge's own use, so a check dies before the forge does #12540

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A check cell's memory ceiling is now the pod's memory limit, minus the forge's own memory use, minus a 64 MiB reserve, and never more than 75% of the pod. Before, a cell could take 75% of the pod, which let a check bring down the whole forge on a 384 MiB pod (#12092, finding 1). FORGECELLMEMORYMAXMIB still overrides.

Notes

shipped production is serving it deploy 7612a18

opened this

A check cell's memory ceiling is now the pod's memory limit, minus the forge's own memory use, minus a 64 MiB reserve, and never more than 75% of the pod. Before, a cell could take 75% of the pod, which let a check bring down the whole forge on a 384 MiB pod (#12092, finding 1). FORGE_CELL_MEMORY_MAX_MIB still overrides.

  • The rule (src/core/capacity.ts): cellMemoryCeilingMiB(podMiB, forgeMiB, share = 0.75, reserve = 64) = ⌊min(pod × share, pod − forge − reserve)⌋, and never below 0. It is a pure function. The reserve is 64 MiB and the floor is 128 MiB.

- 384 MiB pod, forge at 192: min(288, 128) = 128. - 512 MiB pod: min(384, 256) = 256. - 2048 MiB pod: min(1536, 1792) = 1536, so the 75% cap still applies on big pods.

  • The forge's size: forgeResidentMiB() in src/apps/contain.ts reads the forge's live memory use (RSS) and never counts it under the measured 192 MiB. podShareMiB and the wording on the settings page and in the logs now describe the new rule.
  • The refusal (src/checks/service.ts): if the ceiling read from the pod is under 128 MiB, the check stops before install. Its log says too small to check here: … with both numbers. It is reported as the machine's problem: queued once more, then marked interrupted, never red.

- It doesn't apply when an admin set FORGE_CELL_MEMORY_MAX_MIB, to a child forge held to its own size, or to cluster Jobs.

Tests (test/memory-ceiling.test.ts):

  • Hand-checked values for 384, 512 and 2048 MiB pods, each with a second, larger forge size.
  • A check that the forge plus the cell never goes past the pod less the reserve.
  • A pod the forge fills gives 0; under the floor gives the refusal; the variable still overrides.
  • The 384/512 expectations of #12346 changed from 192/320 to 128/256, because the formula is new.
  • test/settings-check-memory.test.ts asserted the old wording exactly; I updated it. The test was right for the old text, not wrong.
  • The check that the service refuses before install only reads the source file for the call's position. The service always reads the real /sys/fs/cgroup/memory.max, so a real end-to-end test can't fake a small pod.

Run here: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2675 files for this change, so I picked the covering suites myself: memory-ceiling, settings-check-memory, child-step-ceiling, shards, capacity, tui-check-memory, check-install, commit-check, checks-disk-red, gate-interrupted, shards-run, check-flake, build-limits and memory-hold all pass (3 skipped by the sandbox). docs/OPERATING.md now states the rule.

Not done:

  • No other cell limit (CPU, disk, time) is touched.
  • src/actions/limits.ts is unchanged; the ceiling isn't an admin setting.
  • The settings-page sentence caches its number per declaration, so it can lag the forge's current size. The limit a check actually gets is computed fresh at each start.
  • The keep_plan and estimate tools weren't available in this run, so the issue has no plan and no time estimate recorded.

Try it: on a pod whose memory.max is 384 MiB, open a pull. While the forge stays near 192 MiB, the check runs in a 128 MiB cell. Once it grows past about 192 MiB, the check log starts with too small to check here instead of the forge being OOM-killed.

🤖 Generated with Claude Code

Closes #12519

3 earlier comments · 1 event
commented

A check cell's memory ceiling is now the pod's memory limit, minus the forge's own memory use, minus a 64 MiB reserve, and never more than 75% of the pod. Before, a cell could take 75% of the pod, which let a check bring down the whole forge on a 384 MiB pod (#12092, finding 1). FORGE_CELL_MEMORY_MAX_MIB still overrides.

  • The rule (src/core/capacity.ts): cellMemoryCeilingMiB(podMiB, forgeMiB, share = 0.75, reserve = 64) = ⌊min(pod × share, pod − forge − reserve)⌋, and never below 0. It is a pure function. The reserve is 64 MiB and the floor is 128 MiB.

- 384 MiB pod, forge at 192: min(288, 128) = 128. - 512 MiB pod: min(384, 256) = 256. - 2048 MiB pod: min(1536, 1792) = 1536, so the 75% cap still applies on big pods.

  • The forge's size: forgeResidentMiB() in src/apps/contain.ts reads the forge's live memory use (RSS) and never counts it under the measured 192 MiB. podShareMiB and the wording on the settings page and in the logs now describe the new rule.
  • The refusal (src/checks/service.ts): if the ceiling read from the pod is under 128 MiB, the check stops before install. Its log says too small to check here: … with both numbers. It is reported as the machine's problem: queued once more, then marked interrupted, never red.

- It doesn't apply when an admin set FORGE_CELL_MEMORY_MAX_MIB, to a child forge held to its own size, or to cluster Jobs.

Tests (test/memory-ceiling.test.ts):

  • Hand-checked values for 384, 512 and 2048 MiB pods, each with a second, larger forge size.
  • A check that the forge plus the cell never goes past the pod less the reserve.
  • A pod the forge fills gives 0; under the floor gives the refusal; the variable still overrides.
  • The 384/512 expectations of #12346 changed from 192/320 to 128/256, because the formula is new.
  • test/settings-check-memory.test.ts asserted the old wording exactly; I updated it. The test was right for the old text, not wrong.
  • The check that the service refuses before install only reads the source file for the call's position. The service always reads the real /sys/fs/cgroup/memory.max, so a real end-to-end test can't fake a small pod.

Run here: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2675 files for this change, so I picked the covering suites myself: memory-ceiling, settings-check-memory, child-step-ceiling, shards, capacity, tui-check-memory, check-install, commit-check, checks-disk-red, gate-interrupted, shards-run, check-flake, build-limits and memory-hold all pass (3 skipped by the sandbox). docs/OPERATING.md now states the rule.

Not done:

  • No other cell limit (CPU, disk, time) is touched.
  • src/actions/limits.ts is unchanged; the ceiling isn't an admin setting.
  • The settings-page sentence caches its number per declaration, so it can lag the forge's current size. The limit a check actually gets is computed fresh at each start.
  • The keep_plan and estimate tools weren't available in this run, so the issue has no plan and no time estimate recorded.

Try it: on a pod whose memory.max is 384 MiB, open a pull. While the forge stays near 192 MiB, the check runs in a 128 MiB cell. Once it grows past about 192 MiB, the check log starts with too small to check here instead of the forge being OOM-killed.

🤖 Generated with Claude Code

Changes requested — before the latest push

The formula, the floor and the wording are sound, and tooSmallToCheck is a clean pure seam. One thing to fix, since the checklist is strict on tests.

  1. The service refusal has no behavioural test. The last describe in test/memory-ceiling.test.ts reads src/checks/service.ts as text and compares indexOf positions. It passes if the call is dead code, if the result is ignored, or if the string is moved into a comment, and it breaks on any reformat. The claim that a real test cannot fake a small pod is not a reason: move the refuse-or-go decision into a small function that takes the pod file (or the ceiling) as an argument, for example in src/checks/ or next to tooSmallToCheck. It should return the { ok:false, infrastructure:true, error } result and write the log line. Test that function directly with a 384 MiB pod file and a forge at 230, and assert that no step runs. Or add an injectable option to ChecksOptions and run startChecks with a fake step runner that must never be called. Delete the source-grep test.
  1. Smaller point: ceilingWords says "75% of the pod memory.max, at most the pod less the forge own and a reserve". Read by a person on the settings page, that is muddled. Say plainly: "the pod memory.max less this forge resident size and a 64 MiB reserve, at most 75% of it".

Boxes 1 and 2 are met; box 3 is met in behaviour but its test is the weak one above. I did not exercise the refusal on the preview, since the pod there is not small. No TUI or README change is needed for this change.

commented

A check cell's memory ceiling is now the pod's memory.max less the forge's resident size and a 64 MiB reserve, capped at 75% of the pod. If that leaves less than 128 MiB, the check is refused before any step starts, as a machine problem rather than a red. This push addresses the review on #12540.

  • The refusal is its own small function. The refuse-or-go decision is now refuseSmallPod in src/checks/pod_room.ts. It takes the pod file and the forge's size as arguments, writes the log line, and returns { ok: false, infrastructure: true, error }. src/checks/service.ts returns that result straight away. ChecksOptions gains a podRoom option so tests can supply a small pod; unset, it reads the real /sys/fs/cgroup/memory.max and measures the forge.
  • The service is tested by behaviour. In the new test/check-pod-too-small.test.ts, startChecks runs against a 384 MiB pod with the forge at 230 MiB (384 − 230 − 64 = 90 MiB). Its check step counts its calls. The test asserts:

- the step is never called and no $ command line is logged; - the first run is cancelled and queued again; - the second run fails; - both logs say "too small to check here … 90 MiB".

With the refusal removed from the service, this test fails.

  • The function is tested directly. test/memory-ceiling.test.ts has three new tests of refuseSmallPod: 384 MiB with the forge at 230 is refused and logs the line once; with the forge at 192 it goes and logs nothing; an admin's FORGE_CELL_MEMORY_MAX_MIB is never refused. The test that read service.ts as text is deleted.
  • Plain wording. The settings page now says "the pod's memory.max less this forge's resident size and a 64 MiB reserve, at most 75% of it". The two assertions on the old wording in test/settings-check-memory.test.ts were updated to match.

bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts named 2676 test files, because service.ts reaches almost everything, so I ran the eight files that cover this change; all pass. The full check runs on the push.

🤖 Generated with Claude Code

commented

A check cell's memory ceiling is now worked out from the pod: the pod's memory.max, less the forge's resident size, less a 64 MiB reserve. If that leaves less than 128 MiB, the check stops before any step and logs "too small to check here" with the figure, for example 384 − 230 − 64 = 90 MiB. The first refusal counts as the machine's fault: the run is cancelled and queued once more, not marked red. If the retry is refused too, it fails with the same reason in its log, like any other red.

  • Where it lives: the refusal is in checks/pod_room.ts, which startChecks uses. It takes a podRoom option so tests can stand in a small pod.
  • The test: test/check-pod-too-small.test.ts uses a check step that must never be called. It asserts that no step ran, that the first run is cancelled and the second failed, and that both logs give the reason.

The last commit fixes what the gate refused on the previous push. The podRoom doc comment in src/checks/service.ts is now wrapped at 100 columns. The test still polls the run list on the real clock, but now says why: the checks service clones with real git and picks up runs on its own timer. A clock could be injected through the service's tickMs option, but that would not remove the wait for git. This follows the // real-timer: notes in other tests rather than injecting a clock.

Gate and type check pass, and test/check-pod-too-small.test.ts passes. scripts/fast.ts said the diff was too broad to pick tests from, so this is the only test file run here; the forge's check runs the full set.

🤖 Generated with Claude Code

commented

A check cell's memory ceiling is now the pod's memory.max, less the forge's resident size, less a 64 MiB reserve. When that leaves under 128 MiB, the check stops before any step runs and logs "too small to check here" with the figure, for example 384 − 230 − 64 = 90 MiB. The first refusal counts as the machine's: the run is cancelled and queued once more, not marked red. If the next run is refused too, it fails with the same reason in its log, like any other red.

  • Where it lives: the refusal is in checks/pod_room.ts, which startChecks calls. A podRoom option lets tests stand in a small pod.
  • The test: test/check-pod-too-small.test.ts uses a check step that must never be called. It asserts that no step ran, that the first run was cancelled and the second failed, and that both logs give the reason.
  • Gate fixes: the podRoom doc comment in src/checks/service.ts is wrapped at 100 columns.
  • The test's wait: the loop still waits on the real clock, with a // real-timer: note saying why: the checks service clones with real git on its own timer. When the wait runs out it now throws, so it can't fall through to the asserts. That is the case test/flake.test.ts refuses.

The gate and the type check pass. check-pod-too-small, flake and system-parts all pass (42 tests); system-parts was named in the last log and is green here. scripts/fast.ts said the diff was too broad to pick tests from, so those three are the only files run here. The snapshot, studio-app-timing and boundary-tenancy-cell tests in the last log are red on main too, and the check did not count them against this pull.

🤖 Generated with Claude Code

Approved

Earlier asks are met. The service refusal is now its own module (src/checks/pod_room.ts) and is tested through startChecks with a step that must never run (test/check-pod-too-small.test.ts). The source-text test is gone, the wait throws on timeout, and the settings wording tests are updated. The preview answers 200. This change adds no page or TUI surface, so the both-doors and README rules do not apply, and docs/OPERATING.md states the rule. I could not diff against 033b3c8377b3 (the commit is not in the clone), so I judged the head as a whole. Two things to do next: (1) Non-blocking: the settings page caches the ceiling per declaration, so it can lag the forge current size; compute it fresh on render. (2) Non-blocking: the 1 MiB floor in podShareMiB hides a 0 from cellMemoryCeilingMiB; the check refuses first, but app cells on a tiny pod still get a 1 MiB limit, so consider refusing them with a clear line too.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs