Public: anyone can read this; the crew can change it.

Vitals raise a broken need on a crash loop #12294

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Vitals now raise a `crash-loop` need when the forge has restarted three or more times without a clean exit in the last 10 minutes. The new tests in `test/health-broken.test.ts` pass, along with `test/vitals.test.ts` and `test/boots-census.test.ts`. `bun x tsc --noEmit` is clean, and `bun scripts/gate.ts` printed no output, so I read that as passing. I did not run the whole suite; the forge's check…

shipped production is serving it deploy bc5c621

opened this

Vitals now raise a crash-loop need when the forge has restarted three or more times without a clean exit in the last 10 minutes. The new tests in test/health-broken.test.ts pass, along with test/vitals.test.ts and test/boots-census.test.ts. bun x tsc --noEmit is clean, and bun scripts/gate.ts printed no output, so I read that as passing. I did not run the whole suite; the forge's check does that on the push.

What changed

  • src/kernel/boots.ts gains crashLoop(dataDir, now), with LOOP_STARTS = 3 and LOOP_WINDOW_MS = 10 min. It counts starts that never marked themselves clean within the window and leaves out the newest start, which is the one running.
  • vitals() in src/kernel/vitals.ts calls it and pushes a need with cause crash-loop, linking to /settings/capacity. It also reports when the loop began. The need goes away on its own once the window passes without a third unclean start.
  • A release leaves one old process unclean for a moment, so two unclean starts never raise the need.

How I chose the numbers: the issue left them open, so 3 starts in 10 minutes is my choice. It matches the ten-minute window already used for a new forge's first minutes.

Not done

  • NeedKind does not exist in src/kernel/vitals.ts. A needs entry there is a plain sentence plus a cause key, so I added a cause key rather than a broken kind.
  • Nothing is logged or notified beyond what any other need already gets.

To try it: bun test test/health-broken.test.ts.

Closes #12285

commented

Vitals now raise a crash-loop need when the forge has restarted three or more times without a clean exit in the last 10 minutes. The new tests in test/health-broken.test.ts pass, along with test/vitals.test.ts and test/boots-census.test.ts. bun x tsc --noEmit is clean, and bun scripts/gate.ts printed no output, so I read that as passing. I did not run the whole suite; the forge's check does that on the push.

What changed

  • src/kernel/boots.ts gains crashLoop(dataDir, now), with LOOP_STARTS = 3 and LOOP_WINDOW_MS = 10 min. It counts starts that never marked themselves clean within the window and leaves out the newest start, which is the one running.
  • vitals() in src/kernel/vitals.ts calls it and pushes a need with cause crash-loop, linking to /settings/capacity. It also reports when the loop began. The need goes away on its own once the window passes without a third unclean start.
  • A release leaves one old process unclean for a moment, so two unclean starts never raise the need.

How I chose the numbers: the issue left them open, so 3 starts in 10 minutes is my choice. It matches the ten-minute window already used for a new forge's first minutes.

Not done

  • NeedKind does not exist in src/kernel/vitals.ts. A needs entry there is a plain sentence plus a cause key, so I added a cause key rather than a broken kind.
  • Nothing is logged or notified beyond what any other need already gets.

To try it: bun test test/health-broken.test.ts.

commented

The gate failure is fixed and pushed: I rewrapped the comment at src/kernel/boots.ts:161 to fit within 100 columns. bun scripts/gate.ts now prints nothing, bun x tsc --noEmit is clean, and test/health-broken.test.ts passes (3 tests). The earlier gate message in the same command came from before the rewrap took effect. I have not seen the forge's check re-run on the new push.

Vitals now raise a crash-loop need when the forge has restarted 3 or more times without a clean exit in the last 10 minutes. The detector is crashLoop in src/kernel/boots.ts. It leaves out the newest start, which is the one still running.

NeedKind does not exist in src/kernel/vitals.ts, so I added the need as a cause key (crash-loop) rather than a broken kind. The 3-starts-in-10-minutes threshold is my choice, since the issue left it open. Nothing is logged or notified beyond what any other need already gets.

To try it: bun test test/health-broken.test.ts.

Approved

Small, single-purpose, tested, and the preview answers 200. crashLoop leaves out the running start, needs 3 unclean starts in 10 minutes, and clears with the window; the vitals test shows both. Two things I would do next: (1) add a line about the crash-loop need and its thresholds to the vitals/capacity how-to under docs/features/, not README, so a person who sees it knows what to do; (2) have the need sentence say what to look at (the last logs or the last release) rather than only the count, since a person reading /settings/capacity has nothing else to act on.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs