Public: anyone can read this; the crew can change it.

Sentinel: budget:01m7cph78n2493t8as2avbd1re is past its budget: A run was killed twice: budget: a full gate took 60 min, limit 60. #12823

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A budget-kill card for a run killed under an old, since-raised limit no longer shows. It now clears when the next run of the same kind passes, or when the limit has been raised since. Cause: run 01m7cph78n2493t8as2avbd1re was a scheduled verification of main, killed at the old 60 min limit.

Notes

shipped production is serving it deploy 5bb6da3

opened this

A budget-kill card for a run killed under an old, since-raised limit no longer shows. It now clears when the next run of the same kind passes, or when the limit has been raised since.

  • Cause: run 01m7cph78n2493t8as2avbd1re was a scheduled verification of main, killed at the old 60 min limit. #12499 already gave verification its own 150 min limit.
  • Why the card stayed: the vitals show any budget kill for 24 hours. The earlier landed fix (#12626) only clears the card once a later run of the same kind and environment passes. A verification takes up to 150 min, so that can be hours away. That pull's description also named a "limit raised" fix (limitSinceRaised) that never landed.
  • What changed:

- src/kernel/budgets.ts gains limitInSummary and limitSinceRaised. They read the limit named in a kill's summary ("… limit 60") and compare it with what that run would get today. A verification is judged against BUDGETS.verify, everything else against BUDGETS.gate. - src/kernel/vitals.ts skips the card when that limit has since been raised. - test/budget-raised.test.ts is new. It covers a kill under an old limit, a kill under today's limit, and a summary that names no limit.

  • Not done:

- I did not make verification faster. A kill under today's limit still shows for its day. - The summary text is parsed with a regex. It is tied to budgetSummary's format.

  • Checks: bun x tsc --noEmit, test/budget-raised.test.ts and test/vitals.test.ts (20 tests) pass. I ran bun scripts/gate.ts before pushing but did not read its output, so I can't say it passed. I did not run the full suite; the forge's check does that on push.
  • To try: after deploy, open /settings/capacity or the health endpoint. budget:01m7cph78n2493t8as2avbd1re should be gone.

Closes #12811

5 earlier comments · 1 event
commented

A budget-kill card for a run killed under an old, since-raised limit no longer shows. It now clears when the next run of the same kind passes, or when the limit has been raised since.

  • Cause: run 01m7cph78n2493t8as2avbd1re was a scheduled verification of main, killed at the old 60 min limit. #12499 already gave verification its own 150 min limit.
  • Why the card stayed: the vitals show any budget kill for 24 hours. The earlier landed fix (#12626) only clears the card once a later run of the same kind and environment passes. A verification takes up to 150 min, so that can be hours away. That pull's description also named a "limit raised" fix (limitSinceRaised) that never landed.
  • What changed:

- src/kernel/budgets.ts gains limitInSummary and limitSinceRaised. They read the limit named in a kill's summary ("… limit 60") and compare it with what that run would get today. A verification is judged against BUDGETS.verify, everything else against BUDGETS.gate. - src/kernel/vitals.ts skips the card when that limit has since been raised. - test/budget-raised.test.ts is new. It covers a kill under an old limit, a kill under today's limit, and a summary that names no limit.

  • Not done:

- I did not make verification faster. A kill under today's limit still shows for its day. - The summary text is parsed with a regex. It is tied to budgetSummary's format.

  • Checks: bun x tsc --noEmit, test/budget-raised.test.ts and test/vitals.test.ts (20 tests) pass. I ran bun scripts/gate.ts before pushing but did not read its output, so I can't say it passed. I did not run the full suite; the forge's check does that on push.
  • To try: after deploy, open /settings/capacity or the health endpoint. budget:01m7cph78n2493t8as2avbd1re should be gone.
commented

The preview itself is up: / answers 200 in 0.35 s, and the log you pasted ends with the app running normally. That log shows only periodic SQL and process statistics, with no error, so I can't tell what made the deploy "fail". The line "no tests declared, none run" is the likely cause. The forge.json app block declares no test command, and I found nothing in the repository that fails on a boot. I changed nothing, since there is no defect on my side to fix.

The tests that cover my change pass: test/budget-raised.test.ts and test/vitals.test.ts (20 tests). The forge's own check runs the rest on the push.

If the forge needs the app to declare a test command for the preview to count as passing, tell me the field it expects in forge.json and I will add it. I did not want to guess at a schema.

commented

The degraded list no longer shows a card for a run the budget killed under a limit that has since been raised. Run 01m7cph78n2493t8as2avbd1re was a scheduled verification of main, killed at the old 60 min limit. #12499 gave verification its own 150 min limit, which fixed the real cause. The card stayed because #12626 landed a different fix from the one its description named: it only clears the card once a later run of the same kind and environment passes, and a verification can take 150 min.

What changed

  • src/kernel/budgets.ts gains limitInSummary and limitSinceRaised. They read the limit a budget kill's summary names ("… limit 60") and compare it with the limit that run would get today: BUDGETS.verify for a verification, BUDGETS.gate otherwise.
  • src/kernel/vitals.ts skips a budget-kill card when that limit has since been raised. The check from #12626 stays as it was.
  • test/budget-raised.test.ts is new. It covers a verification killed under the old 60 min limit, a kill under today's limit, and a summary that names no limit.

bun x tsc --noEmit, test/budget-raised.test.ts and test/vitals.test.ts (20 pass) ran green. bun scripts/gate.ts ran before the push, but its result was not read back. The full suite was not run here; the forge's check runs it on push. The live-triage comment on the issue (filed under P0 Trust) asked for no code change.

What I chose not to do

  • I did not make verification itself faster. A kill under today's 150 min limit is a real signal and still shows for its day.
  • I did not split off a separate builder issue: the fix fits in one small pull.
  • The limit is parsed out of the summary text, so this depends on the format budgetSummary writes. A run that recorded its limit in its own column would be sturdier, but that needs a migration.

How to try it After deploy, open /settings/capacity or /healthz. budget:01m7cph78n2493t8as2avbd1re should be gone from the degraded list.

Closes #12811

🤖 Generated with Claude Code

commented

When a run is killed twice for its budget, vitals now hides its card if the kill happened under a limit that has since been raised (#12811). Each budget kill's summary records the limit it ran under ("… limit 60"). If today's limit for that run is higher, the kill is history and the card is dropped. Verifications are measured against BUDGETS.verify; every other run is measured against BUDGETS.gate.

The fix in this push: the first version always compared against BUDGETS. test/check-budget.test.ts starts the checks service with budgetMs: 400, so its kill reads "limit 1", and vitals took that as "since raised" and hid a card it should show. Now limitSinceRaised takes the limits as an argument (RunLimits, defaulting to BUDGETS), and VitalsInput.runLimits passes in the limits the service actually runs under. I changed check-budget.test.ts to pass the same limits it gives startChecks. The test wasn't wrong: vitals simply had no way to know the service's override until now.

What to look at: src/kernel/budgets.ts (limitInSummary, limitSinceRaised) and the budget-failure loop in src/kernel/vitals.ts.

Not done: production starts the checks service with the default budgets, so production calls to vitals don't pass runLimits. The landing path's swapBudgetMs is a separate limit and doesn't feed into this.

How to try it:

  • bun test test/budget-raised.test.ts test/check-budget.test.ts passes. One new case shows a 400 ms kill counts as "since raised" against the 60 min default, but not against a 400 ms limit.
  • Gate and tsc --noEmit are clean.
  • scripts/fast.ts called the diff too broad to narrow down (exit 2), so I ran the 72 test files that touch vitals, budgets or budget summaries: 692 pass, 1 fail. The failure is in test/test-budget-run.test.ts (the looks lane where Chromium starts). It also fails with this change removed; it needs Chromium, which this sandbox doesn't have.
  • The check's other five failures (threats-enforced, shards-keep-going, loop-stall) were reported red on main at b2d34ff.

🤖 Generated with Claude Code

commented

When a run is killed twice for its budget, vitals now hides its card if the kill happened under a limit that has since been raised (#12811). Each budget kill's summary records the limit it ran under ("… limit 60"). If today's limit for that run is higher, the kill is history and the card is dropped. Verifications are measured against BUDGETS.verify; every other run is measured against BUDGETS.gate.

Fixes since the first push:

  • Limits passed in: vitals now uses the limits the checks service actually runs under. limitSinceRaised takes them as an argument (RunLimits, defaulting to BUDGETS), and VitalsInput.runLimits passes them in. Before this, test/check-budget.test.ts started the service with a 400 ms budget, so its kill read "limit 1", and vitals took that as "since raised" and hid a card it should show. That test now passes the same limits it gives startChecks. The test wasn't wrong: vitals had no way to know the service's override.
  • Lint: a comment in that test mentioned "400 ms", which the no-raw-wall-clock-budget lint in test/flake.test.ts (#4429) reads as a speed budget. I reworded the comment; the code is unchanged.

What to look at: src/kernel/budgets.ts (limitInSummary, limitSinceRaised) and the budget-failure loop in src/kernel/vitals.ts.

Not done: production starts the checks service with the default budgets, so production calls to vitals don't pass runLimits.

How to try it:

  • bun test test/budget-raised.test.ts test/check-budget.test.ts test/flake.test.ts passes (44 tests). One case in budget-raised shows a 400 ms kill counts as "since raised" against the 60 min default, but not against a 400 ms limit.
  • Gate and tsc --noEmit are clean.
  • The last check had eight failures, and flake.test.ts was the only one it counted against this pull. threats-enforced and shards-keep-going are red on main at 8dc19f4 too. I didn't rerun the other five (crew concurrency, the --impact map, the terminal's shift, the prefixes table, the small-child journey); none of them touches vitals or budgets.
  • In an earlier run here, test/test-budget-run.test.ts failed with or without this change, because it needs Chromium and this sandbox has none.

🤖 Generated with Claude Code

Changes requested — before the latest push

The idea is sound and the preview is unaffected (no UI change). Two things to fix in one round:

  1. The vitals wiring is untested. Nothing fails if the limitSinceRaised skip in vitals.ts is deleted: budget-raised.test.ts tests only the helper, and check-budget.test.ts only checks the card still shows under a matching limit. Add a vitals-level test: a budget-killed run with summary budget: a full gate took 60 min, limit 60, env verify, and no runLimits, must give no budget:<id> need; the same run with runLimits {gate:60min, verify:60min} must give the card.
  2. Verification has two env names (verify and verify-bisect, see src/flow/lanes_read.ts:424). The helper judges only verify against limits.verify, so a killed verify-bisect run is compared to the gate limit. Use the same predicate as lanes_read, or say why bisect is separate, and test it.

Smaller: the regex is tied to the budgetSummary format. Put the format and the parser side by side with a round-trip test (the first test partly does).

commented

When a run is killed twice for its budget, vitals now hides its card if the kill happened under a limit that has since been raised (#12811). Each budget kill's summary records the limit it ran under ("… limit 60"). If today's limit for that run is higher, the kill is history and the card is dropped.

Changes from review:

  • One limit, shared: checkLimitMs(env, limits) in src/kernel/budgets.ts is now what the checks service kills at (src/checks/service.ts) and what vitals judges a kill by, so the two can't drift.

- A verify run gets BUDGETS.verify; every other check gets the gate limit.

  • Review runs: an agent run's kill comes from a reviewer, so limitSinceRaised now takes the run and judges those against BUDGETS.review. Before, a review kill at "limit 10" was compared to the 60 min gate and always read as since raised, so its card never showed.
  • Limits from the services: VitalsInput.runLimits (type BudgetLimits, defaulting to BUDGETS) passes in the limits the services actually run under.
  • Format and parser: budgetSummary and limitInSummary sit side by side, and a round-trip test covers 1, 10, 60 and 150 min.
  • Vitals-level test: in test/budget-raised.test.ts, a failed verify run summarised budget: a full gate took 60 min, limit 60 gives no card with default limits. With limits {gate: 60 min, verify: 60 min} it gives one. With the skip in vitals.ts removed, that test fails.

Where I disagree: I didn't treat verify-bisect like verify. The service has always killed a bisection step at the gate limit; it runs a few named files, not the whole gate one file per process. Judging bisect against limits.verify (150 min) would hide cards for kills that still happen today. Bisect is judged by the limit it actually runs under, and a test pins that. If bisect should get the verify limit, that belongs in checkLimitMs, so the service and vitals change together.

How to try it:

  • bun test test/budget-raised.test.ts test/check-budget.test.ts test/flake.test.ts passes (48 tests).
  • Gate and tsc --noEmit are clean.
  • I also ran every test file that imports vitals, budgets or the checks service: 956 pass, 1 fail. The failure is test/check-install.test.ts (installing from a filled read-only cache), which also fails with this change removed; I didn't track down why it fails in the sandbox.

The desk note "Releasing trevato/forge: production is rolling" doesn't affect this: pushing the branch only redeploys this pull's preview, and production isn't touched until it merges.

🤖 Generated with Claude Code

Approved

Every earlier ask is met on the diff since 29c34618bf3c. The service and vitals now share one limit through checkLimitMs. Review kills are judged by BUDGETS.review. runLimits carries the service limits into vitals. The round-trip test pins budgetSummary against limitInSummary. A vitals-level test fails without the skip. Your verify-bisect argument holds: bisect runs under the gate limit, a test pins it, and one shared function keeps service and vitals from drifting. There is no UI surface, so nothing in the preview or pictures is affected. I did not open the preview, the pictures or the changed-line diff: this verdict rests on the diff in the task and the conversation. Next two things, both non-blocking: 1) Non-blocking: production calls to vitals do not pass runLimits. If the service is ever started with a custom budget in production, pass it through so vitals matches. 2) Non-blocking: store the limit in a run column instead of parsing the summary text. That needs a migration, so it belongs in a separate pull.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs