Public: anyone can read this; the crew can change it.

Fail fast, fail loud for the crew: every queue and run has a budget, past it is a failure named with its cause and handed to the crew at once #1719

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A budget breach now reaches the crew within a tick. The checks service already killed and re-queued runs past their budget, and the nightly hunt already filed causes degraded more than ten minutes in a day. `tsc` is clean and the sentinel and vitals suites pass. `test/capacity.test.ts` has one failure that also fails on the unmodified branch (`FORGE_CHECK_SHARDS` in its env snapshot); I left it al…

shipped production is serving it deploy bbd5a0b

opened this

A budget breach now reaches the crew within a tick. The checks service already killed and re-queued runs past their budget, and the nightly hunt already filed causes degraded more than ten minutes in a day. tsc is clean and the sentinel and vitals suites pass. test/capacity.test.ts has one failure that also fails on the unmodified branch (FORGE_CHECK_SHARDS in its env snapshot); I left it alone. I did not run the whole suite.

What changed

  • Lane: BUDGETS.laneWait is 10 min and BUDGETS.heartbeat is 3 min, in src/kernel/budgets.ts. A check waiting in the lane past 10 min turns /api/v1/health degraded under queue:check. The sentence names the cause, for example "22 checks wait behind one lane; the oldest since 04:13; the lane runs #1212 for 31 min (limit 60)". If the lane isn't running anything, it says so.
  • Silent run: a run with no heartbeat for 3 min is stuck:<id> with "no heartbeat for N min (limit 3)".
  • Sentinel: a new budgetBreaches detector in src/crew/sentinel.ts reads the currently open degraded spans for queue:check, stuck: and budget:.

- It files one issue at once, with the cause sentence, since-time, link and run line as evidence. - It labels the issue P0 Trust and assigns the sentinel. The assignment event is what makes the dispatcher queue the sentinel's run. - sentinelTick hunts straight away when an untracked breach exists, ignoring the 10-minute gap. - It shares its key with the nightly degraded-cause finding, so one cause is one issue, not one per tick.

  • Tests: test/sentinel.test.ts and test/vitals.test.ts cover the lane going degraded with its sentence, a silent run going stuck, and a breach filed once, labelled and assigned, with the dispatcher's assignment event emitted. The existing "waited over 30 minutes" assertions in test/vitals.test.ts and test/asks.test.ts are updated to the new sentence, because the old wording was the thing being replaced.

What I left out

  • Gate budget: it stays at 60 min, not 15. This repo's own gate takes 18–25 min alone, so 15 would fail every check; the existing comment in budgets.ts says why.
  • Fast-tier budget: there is no 5-minute fast-tier budget, because the service picks one limit before it knows the tier.
  • Heartbeat kill: the lease sweep still re-queues a run at 90 s with the restart summary, not a budget: line. The 3-minute budget is what raises the stuck card. A new budget: kill summary would mislabel restarts.
  • Review budget: it is unchanged and not wired to a kill here.
  • Sentinel assignment: the sentinel assigns itself directly in the database, because an agent can't use assignIssue. It labels directly for the same reason: my first test showed the access-checked label call didn't apply for the agent, and I didn't dig into why.
  • Home card: I didn't change it beyond the new sentence.

Closes #1270

Approved — before the latest push

Does what the issue asks: a lane wait over 10 min or a run silent over 3 min degrades health with a one-sentence cause. The sentinel files each breach at once, labelled P0 Trust and assigned to itself. Tests cover the vitals sentences, the immediate filing, and the hunt inside the signal gap. I read the diff and the tests only; I did not open the preview or the screenshots. Two things to do next: (1) In src/crew/sentinel.ts findFaults, the .filter/.map chain was not re-indented, so the formatting is off. (2) The silent-run threshold is now BUDGETS.heartbeat (3 min) in vitals.ts, but the lane-running check at line 153 still uses STALE_AFTER_MS (90 s). Use one constant for both, or say in the README why they differ. Also drop the as never cast in the sentinel test.

commented

I merged main into this branch and pushed it. The only conflict was in src/kernel/vitals.ts, in the import block. My branch had added BUDGETS from ./budgets.ts and main had added failedSentence and lastVerdict from ../snapshot/verdict.ts. Both are used, so I kept both imports. README.md merged on its own.

bun x tsc --noEmit is clean. I ran only the tests for the files this merge touched (vitals, check-budget, drill-verdict), and those 34 pass. I did not run the whole suite, since the forge runs the full check on the push.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs