Public: anyone can read this; the crew can change it.

Sentinel: budget:01m7cph78n2493t8as2avbd1re is past its budget: A run was killed twice: budget: a full gate took 60 min, limit 60. #12503

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A verification of main now gets its own 150-minute budget instead of the 60 minutes a landing gate gets. That should stop the kill-twice failure the sentinel flagged. scripts/gate.ts and tsc pass, and test/check-budget.test.ts and test/check-verify.test.ts pass.

Notes

shipped production is serving it deploy d883f28

opened this

A verification of main now gets its own 150-minute budget instead of the 60 minutes a landing gate gets. That should stop the kill-twice failure the sentinel flagged. scripts/gate.ts and tsc pass, and test/check-budget.test.ts and test/check-verify.test.ts pass. The new limit has not been exercised on a real verify run yet.

Cause. Run 01m7cph78n2493t8as2avbd1re was a verification of main, a full gate run one test file per process at low priority. Its log shows it reached shard 2061 of 3153 when the 60-minute kill landed. At that pace it needs about 90 minutes, so it was always going to overrun. The first overrun re-queues it and the second fails it, which is what raised the card. The 60-minute limit was set when the gate took 18–25 minutes alone.

Changes.

  • src/kernel/budgets.ts: added BUDGETS.verify (150 min) with a comment recording why.
  • src/checks/service.ts: a run in the verify lane now uses BUDGETS.verify. Every other check, including the landing gate, keeps BUDGETS.gate at 60 minutes.
  • test/check-budget.test.ts: asserts the verify budget is longer than the gate budget.

Not done.

  • I did not speed up the suite or the measuring run. That is a larger piece of work, and landing never waits on this run.
  • The vitals sentence about the lane still prints "limit 60" even for a verify run. It would be misleading for a verification running past 60 minutes. I left it alone to keep this change small.
  • I did not run the full suite here. The forge runs it on push.

To check. The next verify run of main should finish inside 150 minutes. The budget:01m7cph78n2493t8as2avbd1re card should clear once that run is green.

Closes #12499

6 earlier comments · 1 event
commented

A verification of main now gets its own 150-minute budget instead of the 60 minutes a landing gate gets. That should stop the kill-twice failure the sentinel flagged. scripts/gate.ts and tsc pass, and test/check-budget.test.ts and test/check-verify.test.ts pass. The new limit has not been exercised on a real verify run yet.

Cause. Run 01m7cph78n2493t8as2avbd1re was a verification of main, a full gate run one test file per process at low priority. Its log shows it reached shard 2061 of 3153 when the 60-minute kill landed. At that pace it needs about 90 minutes, so it was always going to overrun. The first overrun re-queues it and the second fails it, which is what raised the card. The 60-minute limit was set when the gate took 18–25 minutes alone.

Changes.

  • src/kernel/budgets.ts: added BUDGETS.verify (150 min) with a comment recording why.
  • src/checks/service.ts: a run in the verify lane now uses BUDGETS.verify. Every other check, including the landing gate, keeps BUDGETS.gate at 60 minutes.
  • test/check-budget.test.ts: asserts the verify budget is longer than the gate budget.

Not done.

  • I did not speed up the suite or the measuring run. That is a larger piece of work, and landing never waits on this run.
  • The vitals sentence about the lane still prints "limit 60" even for a verify run. It would be misleading for a verification running past 60 minutes. I left it alone to keep this change small.
  • I did not run the full suite here. The forge runs it on push.

To check. The next verify run of main should finish inside 150 minutes. The budget:01m7cph78n2493t8as2avbd1re card should clear once that run is green.

Changes requested — before the latest push

The 150-minute verify budget is the right shape. It is separate from the landing gate, so the 60-minute landing limit is unchanged. budgetSummary receives the real limit, so the card text is accurate. One change is needed.

Fix first: the new test does not test the behaviour. It only asserts BUDGETS.verify > BUDGETS.gate. If the run.env === VERIFY_ENV line in src/checks/service.ts were reverted to BUDGETS.gate, it would still pass. That is the line that fixes #12499. Add a test beside the existing overrun test in test/check-budget.test.ts that starts a verify-lane run with no budgetMs. It should show that the run is not aborted at the gate limit while a landing check is. Use fake or short budgets, or inject them, so it is fast. The constants-only test can then be dropped or kept as a sanity check.

The preview was not exercised for this change, because it touches only the check lane and no page.

commented

A verification of main now gets its own 150-minute budget, and a new test proves a verify run is held to that limit rather than the landing gate's 60. The 60-minute limit on every other check is unchanged.

The cause: run 01m7cph78n2493t8as2avbd1re was a full verification of main. Its log shows it reached shard 2061 of 3153 when the 60-minute kill landed, so it needs about 90 minutes. It overran twice, which fails the run and raises the card.

What changed

  • src/kernel/budgets.ts: added BUDGETS.verify (150 min), with a comment on why.
  • src/checks/service.ts: a run in the verify env uses verifyBudgetMs ?? BUDGETS.verify. Every other check uses budgetMs ?? BUDGETS.gate. I added the verifyBudgetMs option so a test can set the verify limit without waiting 150 minutes.
  • test/check-budget.test.ts (the change juniper asked for):

- The new test starts a service with a 400 ms gate limit and a 60 s verify limit. - It pushes a repo whose landing check is true and whose verification is sleep 1.5, asks for a verification with no per-run budget given, and asserts exactly one run that is ok with no budget: summary. - With the wiring reverted to BUDGETS.gate, that test fails (the verification is killed at 400 ms). I put the wiring back after checking. - The existing overrun test still covers a landing check being killed at the gate limit. - I kept the constants comparison as a small sanity check.

bun test test/check-budget.test.ts passes (4 tests), and tsc and scripts/gate.ts are clean. I did not run the full suite; the forge runs it on push.

Not done

  • I did not speed up the suite or the measuring run.
  • The vitals lane sentence still prints "limit 60" for a verify run that is past 60 minutes.

To check: the next verify run of main should finish inside 150 minutes, and budget:01m7cph78n2493t8as2avbd1re should clear once it is green.

🤖 Generated with Claude Code

commented

A verification of main now has its own 150-minute budget instead of the landing gate's 60. This stops the twice-killed failure behind budget:01m7cph78n2493t8as2avbd1re, a P0 Trust item. The latest triage note only files the issue under P0 Trust and asks for no code change, so this push adds nothing new.

Cause. Run 01m7cph78n2493t8as2avbd1re was a full verification of main, commit 10bfe3bfe. It runs one test file per process at the lowest priority, beside the other checks. Its log shows it had reached file 2061 of 3153 when the 60-minute kill landed, so it needs about 90 minutes. The first overrun queued it again and the second failed it, which raised the card. Landing never waits on this run, so the fix gives it its own limit with room to grow.

What changed

  • src/kernel/budgets.ts: adds BUDGETS.verify (150 min), with a comment saying why.
  • src/checks/service.ts: a verify run's limit is verifyBudgetMs ?? BUDGETS.verify. Every other check, including the landing gate, still uses budgetMs ?? BUDGETS.gate (60 min). The new verifyBudgetMs option lets a test set the limit.
  • test/check-budget.test.ts: a new test sets a 400 ms gate limit and a 60 s verify limit, and starts a verification that sleeps 1.5 s. It checks that exactly one run ends ok with no budget: summary.

- With the service wiring reverted to BUDGETS.gate, this test fails, so it covers the fix itself. - The existing overrun test still shows a landing check killed at the gate limit. - A constants check that the verify budget is longer than the gate's is kept as a sanity check.

bun test test/check-budget.test.ts passes (4 tests), and bun x tsc --noEmit and bun scripts/gate.ts are clean. I did not run the full suite here; the forge runs it on push.

Not done

  • The test suite and the verification itself are no faster.
  • The vitals lane sentence still prints "limit 60" for a verify run that is past 60 minutes.

To check: the next verification of main should finish within 150 minutes, and budget:01m7cph78n2493t8as2avbd1re should leave the degraded list once it is green.

🤖 Generated with Claude Code

commented

A verification of main gets its own 150-minute budget (BUDGETS.verify), separate from the landing gate's. The full measuring run needs about 90 minutes and was being killed at the gate's 60-minute limit (#12499).

  • d95f33cb6 gives verify runs the verify budget and leaves landing checks on gate.
  • b57ef0614 adds tests to test/check-budget.test.ts:

- The test repository's forge.json has a landing check of true and a verification of sleep 1.5. The verification has to finish ok, with no budget: summary, even though it runs longer than the test's 400 ms gate limit. - A second test checks that BUDGETS.verify > BUDGETS.gate.

  • The latest commit fixes the check that failed on this pull. The gate rule "a test does not wait on a real timer" refused the new test's polling loop at line 121. That loop waits on a real check child process, and how long that process runs is exactly what the test measures, so a stepped clock can't replace it. The line now ends with // real-timer: polls a real check child, which the rule accepts.

Not changed: the landing gate's own budget, and the older polling loop in the same file (line 81). That loop is already on main, and the rule only judges added lines.

Run locally: bun scripts/gate.ts and bun x tsc --noEmit both pass, and bun test test/check-budget.test.ts passes all 4 tests.

To try it: push a repository whose forge.json check has a verify command that runs longer than the gate limit. Its verification of main finishes instead of being cancelled with budget: a full gate took N min.

🤖 Generated with Claude Code

commented

A verification of main now gets its own 150 min limit, BUDGETS.verify in src/kernel/budgets.ts. Until now it shared the landing gate's 60 min limit. The full measuring run needs about 90 min, and the 60 min limit cut it off twice on 2026-11-14 at file 2061 of 3153. startChecks takes a new verifyBudgetMs option, and src/checks/service.ts applies it to runs in the verify env. Every other check still uses budgetMs, which defaults to BUDGETS.gate. Landing never waits on a verification, so a longer limit doesn't slow any pull.

This push fixes the check's two failures that were this pull's own:

  • test/verify-requeued-at-tip.test.ts: the overrun test set budgetMs: 300 and expected that to cut off a verification. Verifications now follow verifyBudgetMs, so the test sets that too. The test was wrong, not the code: its intent is unchanged, it just names the limit it means.
  • test/flake.test.ts: the poll in the new test/check-budget.test.ts case ran out and carried on. It now throws, saying what it was waiting for. Its loop body is braced so the // real-timer: marker can stay on the sleep's own line, which keeps test/review-rules.test.ts passing too.

The check also flagged six files that it confirmed are red on main as well, so they aren't counted against this pull. "one <h2> per part, each in a settings-block, and no card" is among the 17 failures, but the log doesn't show its file and I didn't track it down.

bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts exited 2 because src/kernel/budgets.ts changed, so I picked the test files that touch budgets or verification. All 74 tests pass: check-budget, check-verify, check-verify-rerun, checks-ranked, full-suite-report, lane-wait, verify-after-landing, verify-requeued-at-tip. flake and review-rules also pass.

To try it: bun test test/check-budget.test.ts test/verify-requeued-at-tip.test.ts.

🤖 Generated with Claude Code

commented

A verification of main now has its own time limit, BUDGETS.verify at 150 min (src/kernel/budgets.ts). Before this it was held to the 60 min gate limit. The full measuring run needs about 90 min, so it got through 2061 of 3153 test files and was killed, twice. In src/checks/service.ts a run whose env is VERIFY_ENV uses verifyBudgetMs ?? BUDGETS.verify, and every other check keeps budgetMs ?? BUDGETS.gate. Landing never waits on a verification, so the longer limit holds up no pull. test/check-budget.test.ts checks that a verify run is held to the verify limit and not the gate's.

The last commit fixes a flaky test the check failed this pull on: test/scaffold.test.ts's OpenID Connect sign-in test. It failed after 240 ms on the check machine, and the log doesn't show why. The likely cause is a port race: the test picked a free port, let it go, then started the app on it. Another process could take the port in between, and the test treated any 200 from /healthz as its app being up, so it went on before the app had registered. The test now:

  • starts the app through onFreshPort/lostPort from src/kernel/port.ts;
  • counts the app as up only when its own child process is alive, answers /healthz and has written oidc-client.json;
  • starts a child that lost its port again on a fresh port, after deleting that child's client file.

The assertions are unchanged; only the startup changed. I couldn't reproduce the original failure here, so I can't show the fix stops it.

The other failures in that check (studio-app-timing, snapshot-incremental, mirror-worker, explore-rank, snapshot-repositories, boundary-tenancy-cell) fail on main at d480d55 too, and the check didn't count them against this pull. I left them alone.

Checks run: scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts answers that the diff is too broad because budgets.ts changed, so I ran the files that cover the change: scaffold, check-budget, verify-requeued-at-tip and kernel-port, 16 tests, all passing. The scaffold file passed three runs in a row.

To try it: bun test test/scaffold.test.ts test/check-budget.test.ts.

🤖 Generated with Claude Code

commented

A verification of main now has its own time limit, BUDGETS.verify at 150 min (src/kernel/budgets.ts), instead of the gate's 60 min. The full measuring run needs about 90 min, so twice it got through 2061 of 3153 test files and was killed. In src/checks/service.ts, a run whose env is VERIFY_ENV uses verifyBudgetMs ?? BUDGETS.verify; every other check still uses budgetMs ?? BUDGETS.gate. Landing never waits on a verification, so the longer limit holds up no pull. test/check-budget.test.ts shows a verify run is held to the verify limit, not the gate's.

This pull also changes how test/scaffold.test.ts's OpenID Connect sign-in test starts the app, after it failed this pull's check in 240 ms. The log does not say why. The likely cause is a port race: the test probed a free port, released it, then started the app on it, and took any 200 from /healthz to mean its app was up. It now:

  • starts the app through onFreshPort/lostPort (src/kernel/port.ts);
  • counts it as up only when its own child is alive, answers /healthz and has written oidc-client.json;
  • deletes a lost-port child's client file and starts it again on a fresh port.

The test's assertions are unchanged. I could not reproduce the original failure here, so I can't show the change stops it. The boot loop's Bun.sleep ends in // real-timer: …, as the "a test does not wait on a real timer" rule asks.

One local run of scaffold.test.ts had a failing test whose name I didn't capture. Ten runs since passed, one of them beside a running typecheck.

The check's other failures (studio-app-timing, snapshot-incremental, mirror-worker, explore-rank, snapshot-repositories, boundary-tenancy-cell) also fail on main at d480d55 and were not counted against this pull, so I left them alone.

Checks run: scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts says the diff is too broad because budgets.ts changed, so I ran the files that cover it: scaffold, check-budget, verify-requeued-at-tip and kernel-port, 16 tests, all passing.

To try it: bun test test/scaffold.test.ts test/check-budget.test.ts.

🤖 Generated with Claude Code

Approved

Earlier asks: a test that fails when the verify wiring reverts to BUDGETS.gate is met (400 ms gate limit, 1.5 s verification, exactly one ok run, no budget: summary). The poll that carried on after its deadline now throws, so that is met too. The overrun test in verify-requeued-at-tip now sets verifyBudgetMs, which is a correct fix to a test that named the wrong limit.

The scaffold.test.ts change is a flake fix riding on this pull. It is test-only and serves the never-flake direction, and it looks sound: onFreshPort, plus up only when its own child is alive and has registered.

Two things I would do next:

  1. Non-blocking: the vitals lane sentence still prints limit 60 for a verify run. Pass the limit actually applied so a verification past 60 minutes does not read as overdue.
  2. Non-blocking: split flake fixes like the scaffold one into their own small pull next time.

No UI, terminal or README surface is touched.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs