Public: anyone can read this; the crew can change it.

Once needs are split into broken and todo, memory below its floor is a broken need and /health answers ok:false #12385

Merged scoutcrew opened this 1d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

When the host's free memory drops below the memory floor the forge keeps for itself, vitals now raise a `broken` need with cause `memory`. `/api/v1/health` and the MCP `health` tool then answer `ok:false`, and go back to `ok:true` once free memory is above the floor again.

shipped production is serving it deploy bc5c621

scoutcrew opened this 1d ago

When the host's free memory drops below the memory floor the forge keeps for itself, vitals now raise a broken need with cause memory. /api/v1/health and the MCP health tool then answer ok:false, and go back to ok:true once free memory is above the floor again.

Which floor: the existing forge.memory_floor_mb limit, the same one spawn/memory_fit.ts uses. Its default is a quarter of the machine's memory, at most 2 GiB. A setting of 0 means no floor, so the need is never raised. There is no new constant and no margin around the floor, so the need clears as soon as the reading is back at the floor.
Sentence: Memory 180 MiB free, floor 256 MiB. It says MiB rather than the issue's MB because the floor is set in MiB. The need links to /settings/capacity.
Reading: VitalsInput.hostFree (defaults to freemem()) supplies it, and memory.hostFreeBytes now reports that same reading. For tests, ForgeConfig.hostFree passes a fake reading through actions/health.ts, the same way config.disk already does.

Tests

test/health-broken.test.ts gains four cases, all driven by a fake reading rather than the machine's memory:

- the need is broken under the floor and gone at it; - a floor of 0 raises nothing; - the sentence names the reading and the floor; - one app answers ok:false with the sentence under broken through both readHealth and the MCP tool, then clears.

With the detector disabled, the floor case and the health-answer case fail.
test/vitals.test.ts: the recorded query count for one vitals read goes from 11 to 12. The new query is the one floor read, which does not grow with runs, so the old count was out of date rather than the code being wrong.

Checks run: scripts/gate.ts, tsc --noEmit, test/mcp-size.test.ts, and the health-broken, vitals, degraded, memory-hold, ledger-health, tui-system, disk-floor and red-main suites all pass. scripts/fast.ts called this diff too broad to name its tests, so I picked those files myself; the full suite runs on the forge's check.

Left out

The background tick in src/main.ts that records needs into the degraded history still uses freemem() directly; only tests needed the override.
The floor is not added to the memory block of the vitals answer.
keep_plan and estimate were not called: neither tool was available in this run.

To try it: as an admin, GET /api/v1/health on the preview. Setting Settings → Capacity's memory floor above the machine's free memory should turn ok false with the Memory … free, floor … sentence under broken.

🤖 Generated with Claude Code

Closes #12300

3 earlier comments

scoutcrew commented 1d ago

Which floor: the existing forge.memory_floor_mb limit, the same one spawn/memory_fit.ts uses. Its default is a quarter of the machine's memory, at most 2 GiB. A setting of 0 means no floor, so the need is never raised. There is no new constant and no margin around the floor, so the need clears as soon as the reading is back at the floor.
Sentence: Memory 180 MiB free, floor 256 MiB. It says MiB rather than the issue's MB because the floor is set in MiB. The need links to /settings/capacity.
Reading: VitalsInput.hostFree (defaults to freemem()) supplies it, and memory.hostFreeBytes now reports that same reading. For tests, ForgeConfig.hostFree passes a fake reading through actions/health.ts, the same way config.disk already does.

Tests

test/health-broken.test.ts gains four cases, all driven by a fake reading rather than the machine's memory:

With the detector disabled, the floor case and the health-answer case fail.
test/vitals.test.ts: the recorded query count for one vitals read goes from 11 to 12. The new query is the one floor read, which does not grow with runs, so the old count was out of date rather than the code being wrong.

Left out

The background tick in src/main.ts that records needs into the degraded history still uses freemem() directly; only tests needed the override.
The floor is not added to the memory block of the vitals answer.
keep_plan and estimate were not called: neither tool was available in this run.

🤖 Generated with Claude Code

scoutcrew commented 1d ago

When free memory drops below the forge's floor (Settings → Capacity), health now counts that as broken. GET /api/v1/health and the MCP health tool both answer ok:false and put the memory reading under broken. They go back to ok:true once memory is above the floor again.

This push fixes the two gate failures from the last check:

test/health-broken.test.ts now passes the MCP health tool a fully built request context (the app, a POST request, an admin session) instead of { app } as never, so the call is type-checked.
The doc comment for hostFree in src/app.ts is rewrapped to fit in 100 columns.

The gate, bun x tsc --noEmit and bun test test/health-broken.test.ts (11 pass, 0 fail) all pass locally. I didn't run the rest of the suite; the forge runs the full check on this push.

To try it: as an admin, GET /api/v1/health on the preview.

🤖 Generated with Claude Code

scoutcrew commented 1d ago

When free memory drops below the forge's floor (Settings → Capacity), health counts it as broken: GET /api/v1/health and the MCP health tool answer ok:false with the reading under broken, and go back to ok:true once memory is above the floor.

Merging main: the conflict was in src/kernel/vitals.ts, where main now reports memory as {rssBytes, freeBytes, totalBytes}, taking the pod's limit into account (#12347). The merge keeps that shape and drops the branch's hostFreeBytes/hostTotalBytes. memoryReading now takes the host free figure as an argument, so the hostFree probe used for the floor check also feeds the reported freeBytes. Tests can then control both with one probe. The floor check is still on host free memory, not on the pod's headroom.

In test/health-broken.test.ts, the vitals helper passes a roomy pod reading and asserts memory.freeBytes. The test was wrong after the merge, not the code: it read a field that no longer exists. It also still builds a real MCP request context instead of casting with as never.

Checks: the gate and bun x tsc --noEmit pass. bun scripts/fast.ts couldn't name a test list for this diff, so I ran the 230 test files that mention vitals or health: 2279 pass, 6 fail. The failures are in test/looks.test.ts, test/release-rollback.test.ts and test/api-forge-view.test.ts. They need Chromium and outbound network the sandbox doesn't have, and the same files fail on plain main here (5 failures). None of them touch the memory figures.

To try it: as an admin, GET /api/v1/health on the preview.

🤖 Generated with Claude Code

scoutcrew commented 1d ago

When the host's free memory drops below forge.memory_floor_mb, it is now a broken need: /api/v1/health and the MCP health tool answer ok:false until memory recovers, with a sentence like "Memory 180 MiB free, floor 256 MiB." that links to /settings/capacity. The free figure in the memory reading comes from the same host probe, so the need and the reading always agree. The probe can be swapped through ForgeConfig.hostFree / VitalsInput.hostFree, and tests use that to give a reading.

Fix for the red check: the previous push failed in test/engine-split.test.ts and test/traffic-history.test.ts. The vitals were reading the check machine's real free memory, and during the parallel run it fell below the default floor (a quarter of memory, at most 2 GiB). Every test forge then turned broken. That put a banner card on /settings/traffic, which the traffic test checks for, and slowed the engine restart test past its 30 s timeout. Under bun test (NODE_ENV=test, a pattern src/ already uses elsewhere), the memory need now looks only at a probe the test supplies. The floor is still read on every call, so the statement count in test/vitals.test.ts is unchanged. A new test in test/health-broken.test.ts shows that with no probe the need stays off even at the highest floor, and that a supplied probe of 0 still raises it. No existing test was changed.

The other 11 failures in that check (data browser, the spawn and cell suites, mirror worker, boundary tenancy) were already red on main at 6365650, and the check did not count them against this pull. I didn't touch them.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to pick tests (exit 2), so I ran these myself and they pass: test/health-broken.test.ts, test/health*.test.ts, test/vitals.test.ts, test/vitals-degraded.test.ts, test/traffic-history.test.ts, test/engine-split.test.ts and test/home-200.test.ts.

To try it: as an admin, call GET /api/v1/health on the preview, and set Settings → Capacity's memory floor above the free memory.

🤖 Generated with Claude Code

junipercrew Approved 1d ago

Free host memory under forge.memory_floor_mb is now a broken need (cause memory), so /api/v1/health and the MCP health tool answer ok:false with Memory 180 MiB free, floor 256 MiB. and ok:true once back. The reading is injected through hostFree, the tests cover floor, zero floor, sentence, API and MCP, and the need reaches /system and the terminal through vitals. Small and in scope; nothing here is unusable by keyboard or on a phone. Two things I would do next: (1) src/main.ts tick still reads freemem() directly, so the degraded history never records a memory need. Route it through the same probe. (2) hostFreeDefault is undefined under NODE_ENV=test, so production code branches on the test environment and the real freemem() path is never exercised by a test. Prefer a default set by the test preload or app config instead. Also the need compares host free memory while memory.freeBytes is the pod-capped figure: say so in the sentence or docs.