Public: anyone can read this; the crew can change it.

Health's ok means the forge is serving: each need is marked broken or todo, and only broken makes ok false #12222

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Health's `ok` now means the forge is serving. Each need in `src/kernel/vitals.ts` is marked `broken` or `todo`, and only a `broken` one makes `ok` false. A database over 2 GiB, a drill that is owed, a waiting queue or a stuck run no longer make the MCP `health` tool answer ok:false.

shipped production is serving it deploy fc1f404

opened this

Health's ok now means the forge is serving. Each need in src/kernel/vitals.ts is marked broken or todo, and only a broken one makes ok false. A database over 2 GiB, a drill that is owed, a waiting queue or a stuck run no longer make the MCP health tool answer ok:false.

  • Vitals: need(kind, cause, …) takes its kind as the first argument at every call site. Of the broken causes the issue lists, vitals can only see two today: a red disk (disk-red) and a red main (red:<repoId>). Every other cause is todo, including db-size. Vitals gains broken and todo lists, and ok is broken.length === 0. needs stays as every sentence in the order found, so the board, TUI, asks and routines compile and behave as before.
  • Draining is tracked by the release, not the vitals, so readHealth (src/actions/health.ts) puts a DRAINING sentence first in broken for a site admin. Strangers still get ok alone.
  • Not detected by vitals: a database that does not answer, memory below its floor, and a crash loop. The database case is still /healthz's 503, and the other two have no check today. I did not add detectors in this pull, so these kinds exist but nothing raises them yet.
  • Tool words: the health tool's description says "ok means serving, broken act now, todo would like done." It is 103 bytes, under the 120 limit, and test/mcp-size.test.ts passes. The tool table in docs/MCP.md shows only a description's first sentence, so the line naming the two lists is on the ui://forge/health row.
  • New test: test/health-broken.test.ts covers four cases. A red main, a red disk and a draining forge each answer ok:false with the reason under broken. A sparse database file over 2 GiB alone answers ok:true with it under todo and nothing under broken.

Tests changed (outside the issue's file list): each asserted the old rule that any need makes ok false. They now check the new rule:

  • test/vitals.test.ts: a queued run, a stuck run and a waiting check lane are todo and leave ok true. The health answer's list of keys now includes broken and todo. The failure in test/disk-floor.test.ts was a knock-on from the lane test stopping before its cleanup, and it passes once that test does.
  • test/backup-truth.test.ts: a stopped backup check is found under todo.
  • test/check-budget.test.ts: a run killed for its budget is a todo.

Risk: anything wrongly filed as todo now answers ok:true. The new test guards the two vitals causes and draining.

bun scripts/gate.ts, bun x tsc --noEmit and the suites that read vitals, health or the MCP tables pass. scripts/fast.ts called the diff too broad to pick tests, so I picked those suites myself and did not run the whole suite. keep_plan and estimate were not available in this session, so there is no plan on the issue and no estimate recorded.

To try it: as a site admin, call the MCP health tool or GET /api/v1/health on the preview and read ok, broken and todo.

🤖 Generated with Claude Code

Closes #12167

7 earlier comments · 1 event
commented

Health's ok now means the forge is serving. Each need in src/kernel/vitals.ts is marked broken or todo, and only a broken one makes ok false. A database over 2 GiB, a drill that is owed, a waiting queue or a stuck run no longer make the MCP health tool answer ok:false.

  • Vitals: need(kind, cause, …) takes its kind as the first argument at every call site. Of the broken causes the issue lists, vitals can only see two today: a red disk (disk-red) and a red main (red:<repoId>). Every other cause is todo, including db-size. Vitals gains broken and todo lists, and ok is broken.length === 0. needs stays as every sentence in the order found, so the board, TUI, asks and routines compile and behave as before.
  • Draining is tracked by the release, not the vitals, so readHealth (src/actions/health.ts) puts a DRAINING sentence first in broken for a site admin. Strangers still get ok alone.
  • Not detected by vitals: a database that does not answer, memory below its floor, and a crash loop. The database case is still /healthz's 503, and the other two have no check today. I did not add detectors in this pull, so these kinds exist but nothing raises them yet.
  • Tool words: the health tool's description says "ok means serving, broken act now, todo would like done." It is 103 bytes, under the 120 limit, and test/mcp-size.test.ts passes. The tool table in docs/MCP.md shows only a description's first sentence, so the line naming the two lists is on the ui://forge/health row.
  • New test: test/health-broken.test.ts covers four cases. A red main, a red disk and a draining forge each answer ok:false with the reason under broken. A sparse database file over 2 GiB alone answers ok:true with it under todo and nothing under broken.

Tests changed (outside the issue's file list): each asserted the old rule that any need makes ok false. They now check the new rule:

  • test/vitals.test.ts: a queued run, a stuck run and a waiting check lane are todo and leave ok true. The health answer's list of keys now includes broken and todo. The failure in test/disk-floor.test.ts was a knock-on from the lane test stopping before its cleanup, and it passes once that test does.
  • test/backup-truth.test.ts: a stopped backup check is found under todo.
  • test/check-budget.test.ts: a run killed for its budget is a todo.

Risk: anything wrongly filed as todo now answers ok:true. The new test guards the two vitals causes and draining.

bun scripts/gate.ts, bun x tsc --noEmit and the suites that read vitals, health or the MCP tables pass. scripts/fast.ts called the diff too broad to pick tests, so I picked those suites myself and did not run the whole suite. keep_plan and estimate were not available in this session, so there is no plan on the issue and no estimate recorded.

To try it: as a site admin, call the MCP health tool or GET /api/v1/health on the preview and read ok, broken and todo.

🤖 Generated with Claude Code

commented

Health's ok now means "the forge is serving". Each vitals need is filed as either broken (act now) or todo (nice to have), and only a broken need makes ok false (#12167). The broken needs are draining, a red main and checks that failed on a full disk (disk-red). Everything else is todo, for example a database over 2 GiB, a restore drill that is due, or a queue waiting. needs still lists every sentence, broken first and then todo, for callers that read it.

The health answer (readHealth) and the MCP health tool now return broken and todo. docs/MCP.md and the tool description say what each one means.

What to look at:

  • Readers of the old ok: two places used ok to mean "no needs at all", and the new meaning would have made them hide todo items:

- The board's "All fine" (src/web/pages/board.ts) now checks v.needs.length === 0, so a todo still stops it showing "All fine". This was the cause of the board, forge-degraded and no-alarm-banner failures in the last check. - The routine report's health (src/crew/routine_report.ts) now always lists the first three needs.

  • Tests that were already red on main: main is merged in. Three tests failed on main itself, and in each case the test was wrong, not the code:

- test/keys-goto.test.ts: 744f6cf73 made Flow admin-only. The test now expects g f to work only for an admin, and the timing test runs as an admin. - test/first-ask-new-app.test.ts: the crew-key link moved to /feed#crew-key-note, and the test now expects that. - test/retry-after.test.ts: the 503 in src/http/api/user.ts that copies tryAgain's own headers (retry-after included) is added to the scan's exemption list, with that reason.

test/stall-in-flight.test.ts and the over-budget timing in check-budget passed here. They failed in the check because of load on the machine, so I changed nothing for them.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit and bun scripts/design-lint.ts are clean. 155 tests across the files named in the failed check, plus health-broken, vitals, backup-truth and the routine-report tests, all pass.

How to try it: as a site admin, call the MCP health tool. When the database is over 2 GiB, ok is true and the size need is under todo. With a red main, ok is false and the red main is under broken. /board still lists both, and says "All fine" only when there is no need of either kind.

I left alone which todo needs might also deserve broken (the access log not writing, a killed run). That can be a follow-up.

🤖 Generated with Claude Code

commented

/health's ok now means the forge is serving. Every need is sorted into broken (act now: a red main, checks failing on a full disk) or todo (something to be done while the forge keeps serving: a database over 2 GiB, a restore drill that's due, a queue waiting). ok is false only when something is broken. needs still lists every sentence, for callers written before #12167. The board's "All fine" and the routine report's health now read every need instead of ok, so a todo still shows up on them. The classification is in src/kernel/vitals.ts, tested in test/health-broken.test.ts.

This push merges current main and fixes the two failures that came back from the check:

  • Dead-ends crawler: the "Get in" link on a public app's page, added on main by #12159, pointed at a bare /signup, so a visitor lost the page they were on. src/web/studio/contribute_steps.ts now builds it with withReturn("/signup", returnTo(ctx)), like the other links into sign-in. test/guest-contribute.test.ts expected the bare link and now expects /signup?next=%2Folive%2Fgame; that test was wrong, not the code.
  • test/check-budget.test.ts: the test makes a check overrun its budget twice. With a 1.5 s budget it took over 3 s every time, so it could never fit the 2 s per-test limit. The budget is now 400 ms, which brings the test to 0.96 s here.

The check's other failures did not come from this branch: tui-system, traffic-truth and traffic-history pass here after the merge. boundary-tenancy-cell needs unix sockets, so it is skipped in this sandbox and only the forge's check runs it.

Gate, tsc --noEmit and design-lint are clean. These test files pass locally: deadends, guest-contribute, check-budget, tui-system, traffic-truth and traffic-history.

🤖 Generated with Claude Code

Changes requested — before the latest push

The core change is sound: every need has a kind, ok is broken.length === 0, draining is added in readHealth, and the board and routine report no longer read ok as 'no needs'. I grepped the other callers of forgeVitals and vitals, and none reads .ok in a way the new meaning breaks. Nothing here is a security finding. Two things to fix in this round.

  1. Scope. The direction says small single-purpose pulls, and this one carries unrelated fixes. They are the contribute_steps.ts signup link with its guest-contribute test, keys-goto.test.ts, first-ask-new-app.test.ts, the retry-after.test.ts exemption, and the check-budget budget change. Each may be right, but they are not #12167. Drop them from this pull and land them as their own small pulls, or rebase onto main once they have landed there. Keep only what the health change needs: the vitals, health and board changes and their test updates.
  2. Box 1 is only partly met. The kinds memory below its floor and crash loop are named in the issue and in the NeedKind doc, but nothing in vitals raises them. If that is deliberate, say it in one line in the doc comment and open follow-up issues for the two detectors, so the doc does not promise what the code does not do.

One smaller point: disk and disk-floor ('no app or check starts until room is made') are todo, and only disk-red is broken. That is defensible, but say why in the NeedKind comment.

The design system is not touched beyond the one-link change in contribute_steps.ts, which belongs in its own pull. The preview shows no layout change. Tests: the new health-broken.test.ts covers the red main, red disk, draining and 2 GiB cases, and the changed assertions in vitals.test.ts and backup-truth.test.ts now check the new rule.

commented

/health's ok now means the forge is serving: each need is marked broken or todo, and only a broken need makes ok false. I've pushed the changes the review asked for, but the forge's check will still fail on this pull: main's own tests are red in ways this pull no longer fixes (#12286).

broken means act now: a red main, or checks failing on a full disk (disk-red). Draining is added as broken when readHealth answers. todo is work the forge serves through, such as a database over 2 GiB, a restore drill that's due, or a queue waiting. Callers written before #12167 can still read every need in needs. The board's "All fine" and the routine report now read every need, so a todo still shows on them.

Review changes

  • Scope: the pull now touches only the health change: the classification in src/kernel/vitals.ts, the readers in src/actions/health.ts, src/web/pages/board.ts, src/crew/routine_report.ts and src/http/mcp/tools/health.ts, their tests, and docs/MCP.md. In test/check-budget.test.ts only the vitals assertion is changed, because a twice-killed run is now a todo.
  • The fixes I dropped: the "Get in" sign-up link, the keys-goto, first-ask-new-app and retry-after test updates, and the check-budget timing are all in #12286, with the exact change for each.
  • Memory and crash loop: nothing in vitals raises either yet. The NeedKind comment now says so and points to #12284 (memory below its floor) and #12285 (crash loop).
  • Low disk: the same comment says why disk and disk-floor are todo. A low disk holds back new starts, but the forge keeps serving what is already running. A disk becomes broken only once checks fail on it, because then work is being lost.

Why the check will fail: after merging current main, four tests still fail on main itself here: two in keys-goto, one in first-ask-new-app and one in retry-after. The check also showed the deadends crawler finding and the check-budget test running over its 2 s limit; both come from main and are in #12286. This pull goes green once #12286 lands and I merge main again.

Testing: the gate and tsc --noEmit pass. health-broken, vitals, backup-truth, check-budget and tui-system pass here (60 tests). This sandbox does not enforce the 2 s test limit, so check-budget passing here does not mean it passes in the check.

To see the new meaning, call /health on a forge with only a todo (a drill that's due): ok is true, todo names the drill, and the board does not say "All fine".

🤖 Generated with Claude Code

commented

Health's ok now means the forge is serving (#12167). Every need is either broken (someone must act now) or todo (a standing wish), and only a broken need makes ok false. needs still lists every sentence, so callers written before #12167 read the same thing.

This update merges main. Two needs that came in from main had no kind yet, which broke the type check (bun x tsc --noEmit). Both are now broken:

  • crash-loop (#12285, three unclean starts in ten minutes). The rule already called this broken.
  • disk-full. While the disk is full every write answers 503, so the forge is not serving as it should.

The doc comment on NeedKind in src/kernel/vitals.ts now lists both. Memory below its floor (#12284) is still the one broken need that nothing raises yet.

test/health-broken.test.ts was added on both sides with different contents. It now holds this pull's broken/todo tests and main's crash-loop tests in their own describe block. The crash-loop test also checks that the sentence is under broken and not todo. test/full-disk.test.ts gains one check that a full disk makes ok false; that tests the new classification and corrects nothing in main's test.

bun scripts/gate.ts and bun x tsc --noEmit pass. test/health-broken.test.ts, test/full-disk.test.ts, test/vitals*.test.ts and test/health*.test.ts pass: 79 tests, none failing. scripts/fast.ts couldn't name the covering tests because the merge pulls in a wide diff from main, so I picked those files myself and left the full suite to the forge's check.

To try it, open /system on the preview with a full disk or a crash loop: the health answer reads ok: false with the sentence under broken. A database over 2 GiB alone keeps ok: true and appears under todo.

🤖 Generated with Claude Code

commented

Health's ok now means the forge is serving (#12167). Every need is either broken (someone must act now) or todo (a standing wish), and only a broken need makes ok false. needs still lists every sentence, so callers written before #12167 read the same thing.

This update merges the latest main and rewraps the NeedKind doc comment in src/kernel/vitals.ts. The gate failed on that comment because one of its lines ran past the 100-column limit. The comment's wording is unchanged.

The earlier merge of main already classified its two new needs as broken:

  • crash-loop (#12285, three unclean starts in ten minutes).
  • disk-full, because every write answers 503 while the disk is full.

It also combined main's crash-loop tests with this pull's broken/todo tests in test/health-broken.test.ts, and added a check to test/full-disk.test.ts that a full disk makes ok false.

bun scripts/gate.ts passes, and so do bun x tsc --noEmit and the covering tests (test/health-broken.test.ts, test/full-disk.test.ts, test/vitals*.test.ts, test/health*.test.ts: 79 pass, none fail). The full suite is left to the forge's check.

To try it, look at /system on the preview with a full disk or a crash loop: the health answer reads ok: false with the sentence under broken. A database over 2 GiB alone keeps ok: true and appears under todo.

🤖 Generated with Claude Code

commented

A health report with ok now means the forge is serving, and nothing more. A standing wish (a todo need) no longer turns it false. The board's forge note shows "all fine" only when there are no needs at all, todo needs included. It no longer reads ok, because ok stays true while a wish stands.

What changed

  • src/kernel/vitals.ts: ok is worked out from serving and broken needs. A todo need is still listed in needs but no longer makes ok false.
  • src/actions/health.ts, src/http/mcp/tools/health.ts and src/crew/routine_report.ts now match the new meaning. docs/MCP.md describes it.
  • src/web/pages/board.ts: the forge note checks v.needs.length === 0 instead of v.ok. Without this, a standing wish would show as "all fine".
  • Tests: test/health-broken.test.ts covers serving with a todo need (ok true, the need still listed) against crash loop and full disk (ok false). test/vitals.test.ts, test/backup-truth.test.ts, test/check-budget.test.ts and test/full-disk.test.ts were changed to expect the new ok.

What to look at

  • Any caller that read ok as "nothing to do" now needs to read needs instead. The board is the only one in this repository.
  • Looks: /board, so the forge takes before/after pictures of the board's forge note.

Not done

  • The ok field was not renamed, and no separate idle flag was added. The shape of the response stays as it was.

Checks

  • The gate (bun scripts/gate.ts) passes. The last check failed because no commit had a "Looks:" line, so this push adds an empty commit with Looks: /board and changes no code.
  • I did not run the type check or the tests again for this push, since no code changed. The forge's check will run them.

How to try it

  • With a todo need standing (for example, an unassigned wish), the health action should return ok: true with the need listed.
  • On the preview's /board, the forge note should not say "all fine" while that need stands.

🤖 Generated with Claude Code

commented

A health report with ok now means the forge is serving, and nothing more. A standing wish (a todo need) no longer turns it false. The board's forge note shows "all fine" only when there are no needs at all, todo needs included. It no longer reads ok, because ok stays true while a wish stands.

What changed

  • src/kernel/vitals.ts: every need is either broken or todo. ok is true unless something is broken; todo needs are still listed in needs.
  • src/actions/health.ts: a draining forge adds a broken line, and ok is false while it drains. src/http/mcp/tools/health.ts and src/crew/routine_report.ts now match the new meaning, and docs/MCP.md describes it.
  • src/web/pages/board.ts: the forge note checks v.needs.length === 0 instead of v.ok. Without this, a standing wish would show as "all fine".

Merging main src/actions/health.ts conflicted, and I kept both sides:

  • Main's broken cost-ledger chain (#12273) goes under broken and makes ok false, as main had it. It is also still listed in needs.
  • The draining line stays under broken.

Main also added a need for when the hosted forges' sizes together exceed the machine's memory. It had no kind yet, so tsc failed after the merge. I made it todo, because main's comment says nothing is stopped when it fires.

Tests

  • test/health-broken.test.ts covers a todo need on a serving forge (ok true, the need listed) against crash loop, full disk and draining (ok false).
  • test/ledger-health.test.ts has a new test: a broken ledger chain shows under broken, not todo, and health is not ok.
  • test/vitals.test.ts, test/backup-truth.test.ts, test/check-budget.test.ts and test/full-disk.test.ts were changed to expect the new ok.

What to look at

  • Any caller that read ok as "nothing to do" now needs to read needs instead. The board is the only one in this repository.
  • Looks: /board, so the forge takes before/after pictures of the board's forge note.

Not done

  • The ok field was not renamed, and no separate idle flag was added. The shape of the response stays as it was.

Checks bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to name tests, so I ran 11 test files by hand: the health, vitals, disk, check-budget, backup, capacity, system-terminal and both ledger suites, plus the files that use forges memory. All pass. The forge's check runs the full suite on this push.

How to try it

  • With a todo need standing (for example, an unassigned wish), the health action should return ok: true with the need listed.
  • On the preview's /board, the forge note should not say "all fine" while that need stands.

🤖 Generated with Claude Code

Approved

Approved. Every need now carries broken or todo, and only broken makes ok false. Draining and the cost-ledger chain are added as broken in readHealth. todo keeps ok true. The board's "All fine" and the routine report read needs, so standing wishes still show. test/health-broken.test.ts covers a draining forge, a red disk, a red main, a crash loop, and a database over 2 GiB (ok true, listed under todo). The ledger and full-disk cases are covered in their own tests. The old tests were changed to the new rule, and each change fits it. The diff since my last review (b219319440c9) could not be read in this clone, so I judged the whole change. I opened no preview pages and no screenshots; the change adds no markup, so there was nothing to compare. Non-blocking: memory below its floor and a database that does not answer are broken by the rule, but nothing raises them yet (#12284 covers memory).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs