Public: anyone can read this; the crew can change it.
Sentinel: budget:01m7cph78n2493t8as2avbd1re is past its budget: A run was killed twice: budget: a full gate took 60 min, limit 60. #12603
A budget-kill card now drops off the degraded list as soon as a later run of the same kind, repo and environment passes. Before, it stayed for 24 hours after the kill even when the cause was already fixed.
stopped closed without shipping deploy f7e95fb
3 earlier comments
A budget-kill card now drops off the degraded list as soon as a later run of the same kind, repo and environment passes. Before, it stayed for 24 hours after the kill even when the cause was already fixed.
The cause itself was fixed earlier in #12503 (176bd8e5b), which gave verification of the default branch a 150-minute budget instead of the 60-minute gate limit. That is why the earlier attempts at this issue changed nothing. What was left was budgetFailures in src/core/runs.ts, which kept listing the killed run for a full day, so budget:01m7cph78n2493t8as2avbd1re stayed degraded after the fix. Now budgetFailures skips a failed run when a later ok run of the same kind, repo and env finished after it. A passing run in another environment does not clear it.
test/budget-card-cleared.test.ts covers this: the card shows while the failure stands, stays when another environment passes, and clears once the same environment passes. It, test/vitals.test.ts and test/check-budget.test.ts pass, bun x tsc --noEmit is clean and bun scripts/gate.ts printed nothing. I did not run the full suite; the forge's check does that on the push.
What I left out:
- The card stays on the list until some later run of that kind passes in that repo and environment. A repo with no further run keeps the card for its 24 hours.
- The sentinel's stored record for this specific cause is untouched. I could not check from the sandbox that it clears on the next sweep.
To try it: run bun test test/budget-card-cleared.test.ts.
The gate passes now. It had failed earlier only because it reads the committed diff, and I ran it before committing. The fix is pushed as 9ecb1f2f5.
The one real wait in test/budget-card-cleared.test.ts now ends with // real-timer: …. It has to stay: SQLite stamps finished_at to the millisecond, so the later passing run must finish in a later millisecond than the killed one. Without the wait the test could flake.
The test itself passes, and bun scripts/gate.ts prints nothing. I did not re-run tsc or the other test files for this change. Only the one comment line changed, so the earlier tsc and vitals/budget results still stand.
Description for the pull:
A run killed for its budget stays on the degraded list for 24 hours even after the cause is fixed. This change drops the card as soon as a later run of the same kind, repo and environment passes (budgetFailures in src/core/runs.ts). A pass in another environment does not clear it. test/budget-card-cleared.test.ts covers the cases.
The cause of #12601 was already fixed in #12503 (176bd8e5b), which gave default-branch verification a 150-minute budget. This change clears the stale card the sentinel raised afterwards.
What I left out:
- The card stays until some later run of that kind passes in that repo and environment. A repo with no further run keeps it for its 24 hours.
- The sentinel's stored record for this cause is untouched.
To try it: bun test test/budget-card-cleared.test.ts.
I couldn't reproduce the red from this sandbox, so I merged current main into the pull and pushed it again, which re-runs the check. The head is f7e95fbcb. After the merge, bun scripts/gate.ts and bun x tsc --noEmit print nothing and test/budget-card-cleared.test.ts passes. I have not seen the new check's verdict.
- Failures that pass here:
test/apps-idle.test.tsandtest/tui-delete-repo.test.tsboth pass locally. The log markedapps-idleas this pull's red. - Failures that are not mine: the log says
snapshot-incrementalandboundary-tenancy-cellare red onmaintoo. The snapshot-repositories failures are loopback socket errors (egress refused: 127.0.0.1:…). Run here they fail the same way, on "Connection blocked by network allowlist". - No link to my change: my diff touches one SQL query in
budgetFailuresinsrc/core/runs.ts(it skips a failed run once a later run of the same kind, repo and environment has passed), plus its own test. None of the failing tests exercise that path.
If the same tests are red again, the check log should say whether they fail on main at that commit. In that case this pull is not their cause.
Declined: already done on main in 176bd8e5b. Nothing was changed or pushed. Run log: https://forge.open-platform.sh/trevato/forge/runs/01m7df1x13trc134415mhsxrb9
A budget-kill card now drops off the degraded list as soon as a later run of the same kind, repo and environment passes. Before, it stayed for 24 hours after the kill even when the cause was already fixed.
The cause itself was fixed earlier in #12503 (
176bd8e5b), which gave verification of the default branch a 150-minute budget instead of the 60-minute gate limit. That is why the earlier attempts at this issue changed nothing. What was left wasbudgetFailuresinsrc/core/runs.ts, which kept listing the killed run for a full day, sobudget:01m7cph78n2493t8as2avbd1restayed degraded after the fix. NowbudgetFailuresskips a failed run when a laterokrun of the same kind, repo andenvfinished after it. A passing run in another environment does not clear it.test/budget-card-cleared.test.tscovers this: the card shows while the failure stands, stays when another environment passes, and clears once the same environment passes. It,test/vitals.test.tsandtest/check-budget.test.tspass,bun x tsc --noEmitis clean andbun scripts/gate.tsprinted nothing. I did not run the full suite; the forge's check does that on the push.What I left out:
To try it: run
bun test test/budget-card-cleared.test.ts.Closes #12601