Sentinel: queue:check was degraded 75 minutes today: 9 checks wait behind one lane; the oldest since 04:44; the lane runs #4032 for 2 min (limit 60). #4062

I made two changes to the check lane. Together they remove the two policy causes of today's queue:check minutes I could find in the run history (/api/v1/repos/trevato/forge/runs, 01:30–07:00). The rest was a real overload that a code change won't fix (listed at the end). Type check and the related suites (checks, merge-queue, red-main, check-list, check-land-main and the two new files) pass; the forge's check hasn't run on these commits yet.

1. A passed-over check moves to the front at half the lane's budget (src/checks/service.ts) pickNext starts the newest check first, unless an older one has waited STARVATION_MS. That limit was 10 minutes, the same as BUDGETS.laneWait, the point at which the lane counts as degraded. So under load, every check the newest-first order skipped waited just over 10 minutes before it moved up, even when the lane was keeping up. From 04:30 to 05:00 the queue drained, but checks still waited 10.1–13.9 minutes; that is the "oldest since 04:44" in the issue. STARVATION_MS is now BUDGETS.laneWait / 2. test/lane-wait.test.ts replays that half hour (seven slots, four-minute checks, nine queued, one more every 36 s): the worst wait was 12 minutes before and is 8 now. The existing pickNext test works off the constant and passes unchanged.

2. Main's check after a land waits 30 seconds before it may start (LAND_SETTLE_MS, landSettleMs) The merge queue gates one candidate (up to eight pulls on top of each other), then lands them seconds apart. Each land asked for a full check of main and started it at once. At 05:56 that meant six eight-and-a-half-minute checks of in-between commits filling six of seven slots, and pulls waited 8 minutes behind them. Now that check is queued with a not_before, and schedule skips it until then. Each step's check is replaced by the next one while it is still queued. Only the candidate's last commit gets checked, and the merge queue's own check already passed on that exact tree, so its main check reuses that result instead of running again. test/check-land-settle.test.ts lands two pulls back to back and requires the first one's main check to be cancelled before it starts; it fails without the change. The option defaults to 0, so other tests that start the checks service are unaffected; src/main.ts sets the 30 seconds. docs/APPS.md describes both rules.

What I didn't change, though these also cost lane time today:

• A request at 03:14 queued 47 checks at once.
• Three restarts cut off 19 running checks, about 130 lane-minutes.
• About 200 lane-minutes went to re-running approved pulls after main moved.
• The package registry was failing from 02:30 to 02:52.

The first and last are real overloads. Rechecking approved pulls against a moved main is deliberate. The vitals sentence still says "one lane" while seven checks run at once; that is wording only.

To try it: bun test test/lane-wait.test.ts test/check-land-settle.test.ts test/check-land-main.test.ts. On production, once this lands, /settings/capacity should show queue:check less often, and a merge-queue land should show main checks "superseded by …" rather than several running together.

🤖 Generated with Claude Code

Closes #4061

I made two changes to the check lane. Together they remove the two policy causes of today's queue:check minutes I could find in the run history (/api/v1/repos/trevato/forge/runs, 01:30–07:00). The rest was a real overload that a code change won't fix (listed at the end). Type check and the related suites (checks, merge-queue, red-main, check-list, check-land-main and the two new files) pass; the forge's check hasn't run on these commits yet.

1. A passed-over check moves to the front at half the lane's budget (src/checks/service.ts) pickNext starts the newest check first, unless an older one has waited STARVATION_MS. That limit was 10 minutes, the same as BUDGETS.laneWait, the point at which the lane counts as degraded. So under load, every check the newest-first order skipped waited just over 10 minutes before it moved up, even when the lane was keeping up. From 04:30 to 05:00 the queue drained, but checks still waited 10.1–13.9 minutes; that is the "oldest since 04:44" in the issue. STARVATION_MS is now BUDGETS.laneWait / 2. test/lane-wait.test.ts replays that half hour (seven slots, four-minute checks, nine queued, one more every 36 s): the worst wait was 12 minutes before and is 8 now. The existing pickNext test works off the constant and passes unchanged.

2. Main's check after a land waits 30 seconds before it may start (LAND_SETTLE_MS, landSettleMs) The merge queue gates one candidate (up to eight pulls on top of each other), then lands them seconds apart. Each land asked for a full check of main and started it at once. At 05:56 that meant six eight-and-a-half-minute checks of in-between commits filling six of seven slots, and pulls waited 8 minutes behind them. Now that check is queued with a not_before, and schedule skips it until then. Each step's check is replaced by the next one while it is still queued. Only the candidate's last commit gets checked, and the merge queue's own check already passed on that exact tree, so its main check reuses that result instead of running again. test/check-land-settle.test.ts lands two pulls back to back and requires the first one's main check to be cancelled before it starts; it fails without the change. The option defaults to 0, so other tests that start the checks service are unaffected; src/main.ts sets the 30 seconds. docs/APPS.md describes both rules.

What I didn't change, though these also cost lane time today:

• A request at 03:14 queued 47 checks at once.
• Three restarts cut off 19 running checks, about 130 lane-minutes.
• About 200 lane-minutes went to re-running approved pulls after main moved.
• The package registry was failing from 02:30 to 02:52.

The first and last are real overloads. Rechecking approved pulls against a moved main is deliberate. The vitals sentence still says "one lane" while seven checks run at once; that is wording only.

To try it: bun test test/lane-wait.test.ts test/check-land-settle.test.ts test/check-land-main.test.ts. On production, once this lands, /settings/capacity should show queue:check less often, and a merge-queue land should show main checks "superseded by …" rather than several running together.

🤖 Generated with Claude Code