Public: anyone can read this; the crew can change it.

/api/v1/health and /system report the cgroup's memory and CPU limit, not the host's #12539

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

On a host with a 512 MiB cgroup limit, /api/v1/health and /system now report the cgroup's CPU and memory limits instead of the host's, and both say "limited by cgroup". The footprint reading on /system used to report the host's core count; it now reports the cgroup's limit.

Notes

shipped production is serving it deploy e5f3b40

opened this

On a host with a 512 MiB cgroup limit, /api/v1/health and /system now report the cgroup's CPU and memory limits instead of the host's, and both say "limited by cgroup". The footprint reading on /system used to report the host's core count; it now reports the cgroup's limit.

What changed

  • src/core/capacity.ts: cgroupLimits(files, host) is a new pure function. It takes the contents of cgroup v2's memory.max and cpu.max and of cgroup v1's memory.limit_in_bytes, plus the host's size, and returns { cpus, memoryBytes, cpuLimited, memoryLimited, limitedBy }. Any of these falls back to the host: "max", a missing file, a malformed value, v1's "unset" number, or a limit not below the host. machineLimits(dir) reads the files and passes them in. The existing cpuLimit and memoryLimitBytes now go through it, so there is one parser. It lives here rather than in footprint.ts because footprint already imports capacity, and the other way round would make an import cycle.
  • src/system/footprint.ts: the reading takes its CPUs and memory from machineLimits(), and the Machine line ends in "limited by cgroup" when the cgroup set them. The new limitedBy field is optional, so footprints saved before this change still load.
  • src/kernel/vitals.ts: health has a new machine: { cpus, memoryBytes, limitedBy } field, shown to site admins. memory.totalBytes follows the same limit. Tests can pass in their own limits.
  • src/system/view.ts and src/web/pages/system.ts: the view sets limitedBy: "cgroup" when the cgroup holds cores or memory under the host's. The summary row then shows a "limited by cgroup" pill next to the core count.

Tests

  • test/footprint.test.ts covers:

- fixture contents for cgroup v2, v1, unset, missing and malformed files, and a limit at or above the host; - reading a cgroup directory; - the footprint's Machine line; - the pill on /system; - health reporting 512 MiB, not the host's 64 GiB, when a 512 MiB limit is given (this one fails without the change).

  • test/vitals.test.ts adds machine to the list of fields a site admin's health answer has.
  • The type check and scripts/gate.ts pass, and design-lint is clean. scripts/fast.ts said the diff was too broad to pick tests for, so I ran 20 test files myself (footprint, vitals, health, capacity, system, tui-system, release-prove): all pass.

Left out: the terminal's system screen is unchanged. It already shows the limited core count, but it does not say "limited by cgroup".

To try it: as a site admin, GET /api/v1/health and read machine. On a preview with a memory limit, /system shows the pill next to the cores.

🤖 Generated with Claude Code

Closes #12518

6 earlier comments · 1 event
commented

On a host with a 512 MiB cgroup limit, /api/v1/health and /system now report the cgroup's CPU and memory limits instead of the host's, and both say "limited by cgroup". The footprint reading on /system used to report the host's core count; it now reports the cgroup's limit.

What changed

  • src/core/capacity.ts: cgroupLimits(files, host) is a new pure function. It takes the contents of cgroup v2's memory.max and cpu.max and of cgroup v1's memory.limit_in_bytes, plus the host's size, and returns { cpus, memoryBytes, cpuLimited, memoryLimited, limitedBy }. Any of these falls back to the host: "max", a missing file, a malformed value, v1's "unset" number, or a limit not below the host. machineLimits(dir) reads the files and passes them in. The existing cpuLimit and memoryLimitBytes now go through it, so there is one parser. It lives here rather than in footprint.ts because footprint already imports capacity, and the other way round would make an import cycle.
  • src/system/footprint.ts: the reading takes its CPUs and memory from machineLimits(), and the Machine line ends in "limited by cgroup" when the cgroup set them. The new limitedBy field is optional, so footprints saved before this change still load.
  • src/kernel/vitals.ts: health has a new machine: { cpus, memoryBytes, limitedBy } field, shown to site admins. memory.totalBytes follows the same limit. Tests can pass in their own limits.
  • src/system/view.ts and src/web/pages/system.ts: the view sets limitedBy: "cgroup" when the cgroup holds cores or memory under the host's. The summary row then shows a "limited by cgroup" pill next to the core count.

Tests

  • test/footprint.test.ts covers:

- fixture contents for cgroup v2, v1, unset, missing and malformed files, and a limit at or above the host; - reading a cgroup directory; - the footprint's Machine line; - the pill on /system; - health reporting 512 MiB, not the host's 64 GiB, when a 512 MiB limit is given (this one fails without the change).

  • test/vitals.test.ts adds machine to the list of fields a site admin's health answer has.
  • The type check and scripts/gate.ts pass, and design-lint is clean. scripts/fast.ts said the diff was too broad to pick tests for, so I ran 20 test files myself (footprint, vitals, health, capacity, system, tui-system, release-prove): all pass.

Left out: the terminal's system screen is unchanged. It already shows the limited core count, but it does not say "limited by cgroup".

To try it: as a site admin, GET /api/v1/health and read machine. On a preview with a memory limit, /system shows the pill next to the cores.

🤖 Generated with Claude Code

commented

On a host whose cgroup caps the forge (512 MiB, say), /api/v1/health and /system now report that cgroup's CPU and memory limits instead of the host's, and both say "limited by cgroup".

Why this push: the preview deploy was fine and is still running. What failed was the pull's check, on the gate's 100-column rule for comments: src/kernel/vitals.ts:145 was 101 columns and src/core/capacity.ts:310 was 103. Both doc comments are rewrapped; no code changed. The gate also wanted a before/after page for the /system change, so the commit message carries Looks: /system.

What changed overall

  • src/core/capacity.ts: a new pure function, cgroupLimits(files, host), reads cgroup v2's memory.max and cpu.max, plus v1's memory.limit_in_bytes as the memory fallback. Each value falls back to the host's when the file says "max", is missing or malformed, holds v1's "unset" number, or is not below the host. machineLimits(dir) reads the files and passes them in. The existing cpuLimit and memoryLimitBytes now go through it, so there is one parser.
  • src/system/footprint.ts: the footprint reading on /system takes its CPUs and memory from machineLimits() (it used to show the host's core count) and ends the Machine line with "limited by cgroup" when the cgroup set them. The new limitedBy field is optional, so footprints saved before this change still load.
  • src/kernel/vitals.ts: /api/v1/health gains machine: { cpus, memoryBytes, limitedBy }, where limitedBy is "cgroup" or "host".
  • src/system/view.ts and src/web/pages/system.ts: the "limited by cgroup" pill beside the cores, using existing design classes only.

Not done: cgroup v1's CPU quota files are not read; on v1, the CPU count falls back to the host's.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts pass. bun scripts/fast.ts said the diff was too broad to pick tests from, so I ran these myself: test/footprint.test.ts, test/vitals.test.ts, and the capacity, health and system test files (21 files, 146 tests), all passing.

To try it: as a site admin, GET /api/v1/health and read machine, then open /system and look for the pill beside the cores.

🤖 Generated with Claude Code

commented

/api/v1/health and /system now report the cgroup's CPU and memory limits and say "limited by cgroup" when the pod is capped below the host (#12518). cgroupLimits in src/core/capacity.ts is a pure function that takes the contents of cpu.max, memory.max and v1's memory.limit_in_bytes and decides the limits. machineLimits reads those files, and cpuLimit and memoryLimitBytes now just call it, so they behave as before. The tests in test/footprint.test.ts feed it fixture file contents.

Fix for the red check: this pull's code didn't cause the test/review-wait.test.ts failure; leftover state from earlier test files did.

  • Cause: the asking tally in src/perf/asking.ts is one object shared by every test file in a process. When capacity-timing.test.ts runs earlier in the same process, its cli token repeatedly requesting /settings/capacity stays in that tally. With enough total traffic it shows up as "cli made 36% of the serving thread's time" on /asks (review-wait) and adds a "Who is asking" section to /system (system-parts.test.ts). I reproduced both failures by running the capacity, system and health test files together.
  • Fix, tests only: review-wait and system-parts now call asking.forget() before they start, as forge-needs-one-stack.test.ts already does, and capacity-timing clears it when it finishes. No test's expectations changed.

The branch also merges current main.

Not done: other tests that send heavy token traffic don't clear the tally yet. A preload hook that clears it for every file would be broader than this fix needs.

To try it, as a site admin, GET /api/v1/health and read machine, or open /system and look for the "limited by cgroup" pill beside the cores.

Checked locally: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts pass. The capacity, system, health, footprint, vitals and review-wait test files pass together in both orders (256 tests in the run with review-wait last). scripts/fast.ts exited 2 because the diff touches src/kernel/vitals.ts, so I picked those files myself.

🤖 Generated with Claude Code

commented

/api/v1/health and /system report the machine's capacity from its cgroup limits. The changes are in src/core/capacity.ts, a new machine key in vitals (src/kernel/vitals.ts), and the footprint and view behind /system. Tests are in test/footprint.test.ts and test/vitals.test.ts.

This push brings in the latest main:

  • test/vitals.test.ts: this branch and main each added a vitals key in the same line. The resolution keeps both: the expected key list now has this branch's machine and main's schedules.
  • Test resets dropped: the branch had reset the process-wide "who is asking" tally at the start of review-wait and system-parts, and at the end of capacity-timing. main (e33973d66) now gives each forge its own tally through askingOf(app), so the shared asking export those resets imported no longer exists. Those three files now match main again.

Checks: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. scripts/fast.ts exited 2 (diff too broad), so I ran the capacity, vitals, footprint, system and health test files by hand: 360 pass, 1 fail. The failure is test/boards-healthz-gap.test.ts, whose hook timed out at the 5 s limit while 50 files ran together. It passes run on its own (5.0 s), and this branch doesn't change it.

To try it: as an admin, GET /api/v1/health and look for the machine key, or open /system.

🤖 Generated with Claude Code

Changes requested — before the latest push

The core is sound: cgroupLimits is pure, tested with fixture contents, and health plus the footprint carry the limit. Three things to fix in one round.

  1. (first) Both doors: the terminal system screen in src/tui/ does not say "limited by cgroup". What the page shows, the terminal must too. Show the same pill/line there from view.limitedBy, with a test in the tui-system test.
  2. No how-to: add docs/features/system.md (or the existing system feature doc) saying that /system and /api/v1/health report the cgroup limit, what machine.limitedBy means, and the v1 CPU caveat. Do not put it in README.
  3. cpuLimit passes a fake host of memoryBytes: 0 to cgroupLimits, which leans on the host.memoryBytes <= 0 branch only to serve that call. Read the CPU alone instead, or pass totalmem(), so the pure function has no sentinel behaviour.

Preview note: /system on the preview shows no cgroup pill, as expected where no limit is set below the host; I could not see the limited state there, so the unit tests carry it.

Box 2 is left only because the terminal half of /system is missing; box 1 and 3 are met.

commented

/api/v1/health and /system now report the memory and CPU limit set by the forge's cgroup instead of the host's size. This round makes the three changes asked for in review, after merging the latest main (it merged cleanly).

  • The terminal says it too: the System screen's first line reads CPU … over N cores · limited by cgroup · load … when view.limitedBy is "cgroup", matching the page's pill (src/tui/screens/system.ts). test/tui-system.test.ts checks both cases: limited, and the host's own size, where nothing mentions cgroup.
  • No sentinel host: the cpu.max parsing is now its own pure function, cpusOf(cpuMax, hostCpus). cpuLimit calls it directly, and cgroupLimits uses it too, so the memoryBytes: 0 fake host and the host.memoryBytes <= 0 branch are both gone (src/core/capacity.ts).
  • How-to: the new docs/features/system.md covers:

- which cgroup files are read, and what happens when one is missing, unset or malformed; - what each place shows: health's machine key (admins only), machine.limitedBy, the /system pill, the footprint fact and the terminal line; - the cgroup v1 caveat: only the memory limit is read, not the CPU quota (cpu.cfs_quota_us).

The README is unchanged.

  • Test addition: test/vitals.test.ts now also checks that a reader who isn't an admin gets no machine key, which the doc states.

The gate, tsc --noEmit and the design lint are clean. The tui-system, footprint, vitals, system-crew, system-btop, system-parts and capacity test files all pass (93 tests). The limited state still shows only in unit tests, because the preview's pod sets no limit below the host.

🤖 Generated with Claude Code

commented

The /system report and /api/v1/health now take the machine's limits from its cgroup instead of from the host. Inside a cell or pod, the CPU and memory reported are what the forge is actually allowed to use, not the size of the physical machine. cpuLimit reads cpu.max on its own (src/core/capacity.ts), and the footprint and vitals use those cgroup values (src/system/footprint.ts, src/kernel/vitals.ts). The web page and the terminal's System screen say when a figure is "limited by cgroup". The how-to for this is docs/features/system.md.

The gate's "pipeline first" rule failed because adding docs/features/system.md counts as a new feature, and this pull touches no pipeline code. This is a correctness fix to a surface that already exists, with no new page or screen, so the last commit is empty and carries only a Direction: line saying that. The pull stays inside the waiver's size limit: one added feature file, 10 changed files.

Checked here: bun scripts/gate.ts, bun x tsc --noEmit, and test/footprint.test.ts, test/tui-system.test.ts and test/vitals.test.ts (58 pass, 0 fail). To try it, open /system on the preview, or the System screen in the terminal, and look for "limited by cgroup" next to CPU and memory.

🤖 Generated with Claude Code

commented

/api/v1/health and /system now report the CPU and memory limits from the pod's cgroup, not the host's, and say whether a cgroup limit or the host's own size sets them. machineLimits() and cgroupLimits() in src/core/capacity.ts read cpu.max and memory.max (then v1's memory.limit_in_bytes). vitals adds a machine key, the footprint uses those limits, and the terminal screen says "limited by cgroup" when one applies.

This update merges main, which had moved memoryLimitBytes into its own module, src/system/memory_limit.ts, so the static web can size its cache without loading capacity.ts (#12522). The merge keeps that move:

  • src/core/capacity.ts: the branch's copy of memoryLimitBytes is gone. totalmem is imported directly, since machineLimits still uses it.
  • src/kernel/vitals.ts: machineLimits and MachineLimits come from capacity.ts, and memoryLimitBytes from system/memory_limit.ts.
  • src/system/footprint.ts: keeps the branch's machineLimits and main's largePageCache import. availableParallelism is dropped because the limits now supply the CPU count.
  • test/vitals.test.ts: the expected health keys now include both the branch's machine and main's optional.

Two readers of the memory limit remain: vitals' pod-memory figure uses memoryLimitBytes(), and the machine report uses machineLimits(). Both read the same cgroup files and only differ on malformed contents. I did not merge them into one function.

bun scripts/gate.ts and bun x tsc --noEmit are clean. The vitals, footprint, capacity, boundary-limits, system and cache tests pass: 296 tests across 44 files.

To try it, open /system on the preview, or call /api/v1/health with a site-admin token, and look at machine.

🤖 Generated with Claude Code

Approved

Earlier asks are met: the terminal System screen says "limited by cgroup" and is tested; the cpu.max parsing is its own pure function (cpusOf) with no sentinel host; docs/features/system.md carries the how-to and README is untouched; a non-admin reader gets no machine key, and a test checks it. The pill uses existing components, a plain pill, with no new markup, and it works with JavaScript off.

Next, both non-blocking: (1) Non-blocking: vitals still has two readers of the memory limit (memoryLimitBytes and machineLimits). They differ on malformed files, so fold them into one soon. (2) Non-blocking: cgroup v1 CPU quota (cpu.cfs_quota_us) is not read. The doc states this caveat, but a follow-up issue would keep it from being forgotten.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs