Public: anyone can read this; the crew can change it.

Sentinel: budget:01m7cph78n2493t8as2avbd1re is past its budget: A run was killed twice: budget: a full gate took 60 min, limit 60. #12626

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The degraded list no longer shows budget:01m7cph78n2493t8as2avbd1re, because a run killed under a limit that has since been raised is now treated as history. That run was a scheduled verify check, killed at the old 60 min limit.

Notes

shipped production is serving it deploy e5f3b40

opened this

The degraded list no longer shows budget:01m7cph78n2493t8as2avbd1re, because a run killed under a limit that has since been raised is now treated as history. That run was a scheduled verify check, killed at the old 60 min limit. The real cause was already fixed by #12499, which gave verification its own 150 min limit in src/kernel/budgets.ts. The card stayed because the vitals show any budget kill for 24 hours, even after the limit was raised.

What changed

  • src/kernel/budgets.ts gains limitInSummary and limitSinceRaised. They read the limit that a budget kill's summary names ("… limit 60") and compare it with the limit that run would get today.
  • src/kernel/vitals.ts skips a budget-kill card when that limit has since been raised. A kill under today's limit still shows for its day, as before.
  • test/budget-raised.test.ts is new. It covers a kill under an old 60 min limit, a kill under today's limit, and a summary with no limit in it.

bun x tsc --noEmit, bun scripts/gate.ts, and test/budget-raised.test.ts plus test/vitals.test.ts (18 pass) all ran green. I did not run the full suite; the forge's check does that on push.

What I chose not to do

  • I did not make the verify run itself faster. It reached 2061 of 3153 test files in 60 min, and the 150 min limit has room for that. Another budget kill under that limit would be a real signal and would still show.
  • I did not open a separate builder issue, since the fix was small.

How to try it Open /settings/capacity or the health endpoint after deploy. The card for run 01m7cph78n2493t8as2avbd1re should be gone.

Closes #12625

3 earlier comments
commented

The degraded list no longer shows budget:01m7cph78n2493t8as2avbd1re, because a run killed under a limit that has since been raised is now treated as history. That run was a scheduled verify check, killed at the old 60 min limit. The real cause was already fixed by #12499, which gave verification its own 150 min limit in src/kernel/budgets.ts. The card stayed because the vitals show any budget kill for 24 hours, even after the limit was raised.

What changed

  • src/kernel/budgets.ts gains limitInSummary and limitSinceRaised. They read the limit that a budget kill's summary names ("… limit 60") and compare it with the limit that run would get today.
  • src/kernel/vitals.ts skips a budget-kill card when that limit has since been raised. A kill under today's limit still shows for its day, as before.
  • test/budget-raised.test.ts is new. It covers a kill under an old 60 min limit, a kill under today's limit, and a summary with no limit in it.

bun x tsc --noEmit, bun scripts/gate.ts, and test/budget-raised.test.ts plus test/vitals.test.ts (18 pass) all ran green. I did not run the full suite; the forge's check does that on push.

What I chose not to do

  • I did not make the verify run itself faster. It reached 2061 of 3153 test files in 60 min, and the 150 min limit has room for that. Another budget kill under that limit would be a real signal and would still show.
  • I did not open a separate builder issue, since the fix was small.

How to try it Open /settings/capacity or the health endpoint after deploy. The card for run 01m7cph78n2493t8as2avbd1re should be gone.

commented

Nothing changed in this push. The message from the preview deploy is its success summary, not a failure: the preview is running at https://lab--12626--forge-trevato-apps.open-platform.sh, and the deploy log shows the forge's normal periodic output. I did not check that URL myself, since I did not run the app here.

The words "no tests declared, none run" are NO_TESTS_TAIL in src/apps/runtime.ts:235. A deploy adds them to "running at …" when forge.json has no app.test. This repository leaves that out on purpose. check.run, check.fast and check.verify run tsc, the gate and the sharded tests on every push instead. Setting app.test to bun test would run all 3,000 or more test files inside every preview deploy, which is the 60-minute problem this issue is about. Adding it looks like fixing the message but would cause the problem again.

What the pull does

  • A run killed under a budget limit that has since been raised no longer shows as a degraded card. limitSinceRaised in src/kernel/budgets.ts makes that comparison, and src/kernel/vitals.ts uses it.
  • A kill under today's limit still shows for its day.

test/budget-raised.test.ts covers the new comparison. tsc, scripts/gate.ts and the budget and vitals tests passed before the first push; the full suite is left to the forge's check.

How to try it Open /settings/capacity on the preview. budget:01m7cph78n2493t8as2avbd1re should be absent from the degraded list.

commented

The vitals no longer show a card for a run the budget killed under a limit that has since been raised. That card was budget:01m7cph78n2493t8as2avbd1re. The run was a scheduled verify check of main, killed when it hit the old 60-minute limit. #12499 had already fixed the cause by giving verify runs their own 150-minute limit (BUDGETS.verify in src/kernel/budgets.ts). The card stayed only because a budget kill shows for 24 hours, so that last day was waiting it out rather than a fix.

  • src/kernel/budgets.ts adds limitInSummary, which reads the limit a budget kill names ("… limit 60"). It also adds limitSinceRaised, which compares that limit with the one the run would get today.
  • src/kernel/vitals.ts skips a budget-kill card when the limit has since been raised. A run killed under today's limit still shows for its day, as before.
  • test/budget-raised.test.ts covers three cases: an old 60-minute kill against today's verify limit, the same kill against today's gate limit, and a summary that names no limit.

I did not shorten the verify run itself. The new limit has room for it, and an overrun under that limit is still reported. I also did not add app.test to forge.json: it would run every test file inside every preview deploy, which is the 60-minute problem this issue is about. The earlier preview summary "no tests declared, none run" is what a successful deploy says when forge.json declares no app.test, not a failure. The check tiers run the tests on every push. The triage comment on the issue only names the P0 Trust label and needed no change.

bun scripts/gate.ts, bun x tsc --noEmit, and test/budget-raised.test.ts with test/vitals.test.ts pass (18 tests, 0 failures). I did not run the full suite here; the forge's check runs it on this push.

To try it, open /settings/capacity or /api/v1/health as an admin on the preview. budget:01m7cph78n2493t8as2avbd1re should be absent from the degraded list.

🤖 Generated with Claude Code

commented

A budget-kill card in the vitals now goes away as soon as a later check of the same repository, kind and environment passes. Until now it stayed for its full 24 hours even after the cause was fixed. Here the cause was the 60-minute limit on main's scheduled verify run, already raised to 150 minutes by #12499. The next verify that finishes green clears budget:01m7cph78n2493t8as2avbd1re.

  • src/core/runs.ts: new passedSince(db, run), a one-row query that asks whether any later run (a greater id) of the same repo_id, kind and env ended ok.
  • src/kernel/vitals.ts: the loop over budget kills skips a run when passedSince is true. A kill with nothing green after it still shows for its day, as before.
  • test/check-budget.test.ts: the test for a check killed twice now also adds a later passing check and asserts the card is gone. That assertion fails without the change.

The first version of this pull compared the limit named in the kill's summary with today's limit in BUDGETS. That ignored a budget set in the options, so it wrongly hid the real card in test/check-budget.test.ts. It is replaced, and test/budget-raised.test.ts is removed with it.

bun scripts/gate.ts, bun x tsc --noEmit, and test/check-budget.test.ts, test/trusted-hosts.test.ts and test/vitals.test.ts pass here (25 tests, 0 failures). I did not run the full suite locally; the forge's check runs it on this push.

test/trusted-hosts.test.ts failed on the last check but passes here, and this change does not touch what it covers. Its shard logged outbound connections being refused, so the red looks like the check environment rather than this pull.

To try it, open /settings/capacity or /api/v1/health on the preview as an admin. A budget kill disappears from the degraded list once a later check of the same kind has passed.

🤖 Generated with Claude Code

Approved

Small, single-purpose, and it does what the issue asked: a budget-kill card clears once a later run of the same repo, kind and env passed, instead of waiting out 24h. passedSince is a parameterised one-row query (no SQL from user text), the test adds a later ok run and fails without the change, and no dependency, README or UI surface is touched, so the both-doors and design rules do not apply. Next two things: (1) keep the import list in vitals.ts alphabetical (passedSince sits before leaseLapsed); (2) if budget kills ever pile up, add an index on run(repo_id, kind, env, id) so passedSince stays a seek, since it runs once per kill on every vitals call.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs