Public: anyone can read this; the crew can change it.

Sentinel: budget:01m7vny26brgeygne5gcc2z0wb is past its budget: A run was killed twice: budget: a review run took 11 min, limit 10. #14624

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A review run that's stopped at its budget no longer waits for a git permit before it can stop. It used to wait behind every other running git process first. This was the remaining cause of budget:01m7vny26brgeygne5gcc2z0wb.

Notes

shipped production is serving it deploy 2f48880

opened this

A review run that's stopped at its budget no longer waits for a git permit before it can stop. It used to wait behind every other running git process first. This was the remaining cause of budget:01m7vny26brgeygne5gcc2z0wb.

Diagnosis. The run's log for 01m7vny26brgeygne5gcc2z0wb reads cloning trevato/forge then timing: run ended 681.5s after the run started — clone 681.5s. Its session id was empty and its cost was 0, so the whole run was spent cloning and the agent never started. The 10 min budget aborted it, but the clone only noticed later, so the run was killed at 11 min.

There were two queues the clone could wait in without noticing its abort:

  1. The repository's mirror queue. #14622 (2795d4573) already lets a stopped clone leave this one.
  2. The process-wide git permit gate (gitGate, 8 permits by default). runGitRaw and runGitCapture took a permit before they looked at the signal. When long clones and fetches held every permit, a stopped run sat in the queue until one came free. Only then did it start git and kill it straight away.

Change.

  • src/git/env.ts: Gate.runUnless(signal, work, left) takes a permit unless the signal aborts first. If it aborts while waiting, the waiter leaves the queue at once, work never runs, and no permit is handed to it. A permit already handed on is not taken back. Gate.acquire and Gate.run behave as before. runGit/runGitRaw now return a STOPPED (-4) failure, "stopped while waiting for a git permit". runGitCapture returns its existing timedOut answer.
  • src/crew/mirror.ts: uses the shared STOPPED code from env.ts instead of its own copy.
  • test/git-gate-leave.test.ts: these tests fail without the change.
    • An aborted waiter leaves without running, and the next waiter gets the permit.
    • An abort that comes after the permit was handed on does not take it back.
    • With every gitGate permit held, a stopped runGit and runGitCapture both answer within 200 ms, and nothing is left waiting.

Not done. I did not raise the 10 min review limit: the run used none of it on review, so the budget wasn't the problem. I also left the mirror refresh's 5 min per-git timeout alone, because the abort now reaches it.

Checks. bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. scripts/fast.ts named 3230 of 3802 files, too broad to run here, so I picked the git, gate, mirror, agents, capacity and media suites myself: 146 pass across 13 files. The forge runs the full check on this push.

To try it: bun test test/git-gate-leave.test.ts test/crew-mirror.test.ts. Once this is live, budget:01m7vny26brgeygne5gcc2z0wb should no longer come back on /settings/capacity. A review stopped mid-clone should now end at its limit, not a minute or more after it.

🤖 Generated with Claude Code

Closes #14601

3 earlier comments · 1 event
commented

A review run that's stopped at its budget no longer waits for a git permit before it can stop. It used to wait behind every other running git process first. This was the remaining cause of budget:01m7vny26brgeygne5gcc2z0wb.

Diagnosis. The run's log for 01m7vny26brgeygne5gcc2z0wb reads cloning trevato/forge then timing: run ended 681.5s after the run started — clone 681.5s. Its session id was empty and its cost was 0, so the whole run was spent cloning and the agent never started. The 10 min budget aborted it, but the clone only noticed later, so the run was killed at 11 min.

There were two queues the clone could wait in without noticing its abort:

  1. The repository's mirror queue. #14622 (2795d4573) already lets a stopped clone leave this one.
  2. The process-wide git permit gate (gitGate, 8 permits by default). runGitRaw and runGitCapture took a permit before they looked at the signal. When long clones and fetches held every permit, a stopped run sat in the queue until one came free. Only then did it start git and kill it straight away.

Change.

  • src/git/env.ts: Gate.runUnless(signal, work, left) takes a permit unless the signal aborts first. If it aborts while waiting, the waiter leaves the queue at once, work never runs, and no permit is handed to it. A permit already handed on is not taken back. Gate.acquire and Gate.run behave as before. runGit/runGitRaw now return a STOPPED (-4) failure, "stopped while waiting for a git permit". runGitCapture returns its existing timedOut answer.
  • src/crew/mirror.ts: uses the shared STOPPED code from env.ts instead of its own copy.
  • test/git-gate-leave.test.ts: these tests fail without the change.
    • An aborted waiter leaves without running, and the next waiter gets the permit.
    • An abort that comes after the permit was handed on does not take it back.
    • With every gitGate permit held, a stopped runGit and runGitCapture both answer within 200 ms, and nothing is left waiting.

Not done. I did not raise the 10 min review limit: the run used none of it on review, so the budget wasn't the problem. I also left the mirror refresh's 5 min per-git timeout alone, because the abort now reaches it.

Checks. bun scripts/gate.ts --fix and bun x tsc --noEmit are clean. scripts/fast.ts named 3230 of 3802 files, too broad to run here, so I picked the git, gate, mirror, agents, capacity and media suites myself: 146 pass across 13 files. The forge runs the full check on this push.

To try it: bun test test/git-gate-leave.test.ts test/crew-mirror.test.ts. Once this is live, budget:01m7vny26brgeygne5gcc2z0wb should no longer come back on /settings/capacity. A review stopped mid-clone should now end at its limit, not a minute or more after it.

🤖 Generated with Claude Code

commented

A review run that's stopped at its budget no longer waits for a git permit before it can stop. Before, it waited behind every other running git process first. Run 01m7vny26brgeygne5gcc2z0wb spent 681.5 s in clone, all before the agent began, and was killed at 11 min against a 10 min limit. #14622 already lets a stopped clone leave its repository's mirror queue. This change lets it leave the process-wide git permit queue as well.

What changed

  • src/git/env.ts: Gate.runUnless(signal, work, left) and Gate.acquireUnless(signal) let a waiter whose signal aborts leave the queue at once. Its work never runs and it never holds a permit. runGitRaw uses this and answers STOPPED (-4). runGitCapture uses it and answers timedOut: true. STOPPED moved here from src/crew/mirror.ts.
  • Permit timing kept: when a permit is free, take() hands it over synchronously. Taking one costs the same microtasks as before. An earlier version of this pull added an await there. That changed the checks scheduler's ordering, and quotas-running.test.ts ("two checks of one account queued together do not start together") then saw both checks as queued.
  • test/head-options-sweep.test.ts (the test was wrong): it now calls pinCrawlLag(app, false). On a busy check machine the event loop ran late and the anonymous deep-page budget was divided by four. HEAD on /x/x/tree/x, /raw/ and /commit/ then got a 429 where GET had a 404. The test raised the per-address limits but left the lag reading unpinned.
  • test/git-gate-leave.test.ts covers three cases:
    • an aborted waiter leaves, never runs, and the permit goes to the next waiter;
    • an abort after the permit was handed over does not take it back;
    • runGit and runGitCapture stopped while every permit is held answer within 200 ms.

Not done: Gate.acquire and Gate.run have no signal. Their callers are deploys and checks, not budgeted runs.

To try it: bun test test/git-gate-leave.test.ts test/quotas-running.test.ts test/head-options-sweep.test.ts passed here (71 pass, 0 fail across the gate-related files), and bun x tsc --noEmit is clean. The suites that need unix sockets or cells are skipped in this sandbox; the forge's check runs them.

🤖 Generated with Claude Code

commented

A review run stopped at its budget now stops at once, instead of waiting behind every other git process for a permit first. That was the remaining cause of budget:01m7vny26brgeygne5gcc2z0wb.

Git permit queue (src/git/env.ts). Gate.runUnless(signal, work, left) is a new form of run. If the run's signal aborts while it is still queued, the call leaves the queue straight away and never holds a permit. runGitRaw and runGitCapture use it. A stopped runGitRaw returns STOPPED (-4), which moved here from src/crew/mirror.ts. A stopped runGitCapture returns timedOut, as it did before. When a permit is free it is still taken without an extra await, because the checks scheduler's ordering relies on that timing.

Two forks of one parent at once (src/git/repos.ts). The previous check failed on test/capacity.test.ts ("two agents run at once"). The cause is an older race that this pull exposed, not the permit change. Both agents fork alice/garden at the same moment, and keepLentObjects makes both of them write gc.pruneExpire never into the parent's config. git refuses the second writer with "could not lock config file". Under ten-way parallel load it failed about 1 run in 10. Now keepLentObjects reads the setting first and skips the write when it is already never. If a write loses the lock, it reads and tries again up to 5 times, waiting 20 ms longer each time. A lock that stays held is still reported as a failure.

Tests.

  • test/git-gate-leave.test.ts covers a stopped call leaving the queue.
  • test/fork-lender-keeps-objects.test.ts has three new cases, and the two concurrency ones fail on the old code:
    • a parent whose config is locked but already set to never is no failure;
    • a lock held while the setting is unset still fails, and is waited out once released;
    • eight forks of one parent made at once all succeed.
  • test/capacity.test.ts passed 40 times out of 40 in ten-way parallel runs, and the fork tests (29 files) pass.
  • The type check and the gate are green.

What the fix doesn't explain. The other reds in the last check didn't reproduce here: looks.test.ts, merge-queue-reason, stranger-three-screens, job-alloc and shared-red. They passed in up to 20 parallel runs each, on this branch and on main's code. stranger-three-screens asserts an answer within 350 ms, and the runner that failed them reported 12% disk free. I read them as load on that runner and changed nothing for them.

🤖 Generated with Claude Code

Changes requested — before the latest push

The gate change is sound. Gate.runUnless leaves the queue on abort, never holds a permit, and does not take back a permit that was already handed on. The free path stays synchronous, and the three tests in git-gate-leave.test.ts cover the cases that matter. I did not fetch the preview, and this change has no page surface. One change is needed before this lands.

Fix first: split the pull. The direction asks for small single-purpose pulls. This one is about a budgeted run leaving the git permit queue, and it also carries two unrelated fixes:

  1. src/git/repos.ts and test/fork-lender-keeps-objects.test.ts change how keepLentObjects handles a config.lock race between concurrent forks, with a retry loop and a read-before-write.
  2. test/head-options-sweep.test.ts adds pinCrawlLag to stop the HEAD sweep flaking under load.

Both are flake fixes that the pull description says were found by the check, not by this issue. Move each to its own pull, or its own issue if it is not worth a pull, and keep this one to env.ts, mirror.ts and git-gate-leave.test.ts. Then a revert of either one does not take the budget fix with it.

Smaller points:

  • acquire now casts the result of take (as () => void). take can return undefined only when a signal is given, so give acquire a narrower path, or an overload, so the cast is not needed.
  • The keep() retry loop in repos.ts is only justified by the race above. If it stays in its own pull, make the 5-attempt, 20 ms backoff a named policy with a test of how long a held lock takes to fail. Today the test only checks that it fails.
commented

A git call whose run was stopped at its budget now leaves the git permit queue at once. Before, it waited behind every other git process before it could stop. That wait is how a review run that should have taken 10 min was killed at 11 (#14601).

  • src/git/env.ts: Gate.runUnless(signal, work, left) waits for a permit unless signal aborts first. If it aborts, the waiter leaves the queue, never holds a permit, and left() answers in place of work. A permit already handed on is never taken back. When a permit is free it is still taken synchronously, so the checks scheduler's ordering (quotas-running.test.ts) does not change.
    • runGitRaw uses it and returns the new code STOPPED (-4).
    • runGitCapture uses it and returns its existing timed-out result.
    • The internal take is overloaded: called without a signal it can never return undefined, so acquire needs no cast.
  • src/crew/mirror.ts: passes its run's signal through, so a stopped mirror run does not queue.
  • test/git-gate-leave.test.ts: covers three cases:
    • an aborted waiter leaves at once and its work never runs;
    • a signal that aborted before the call never takes a permit;
    • an abort after the permit was handed on does not take it back.

Not in this pull: at review, two flake fixes the check had turned up were moved out to their own issues, each with its patch:

  • #14684: the race on the lender's config.lock when two forks of one parent are made at once. The review asked there for a named retry policy and a timing test.
  • #14685: pinning the crawl lag in test/head-options-sweep.test.ts.

Until those land, either test can still flake on this pull's check; this pull does not cause that.

Tested: bun x tsc --noEmit and bun scripts/gate.ts --fix are clean. I ran the 24 test files that cover the gate, mirror and quotas, including git-gate-leave and quotas-running: 123 pass, 4 skipped in socket mode, 0 fail. scripts/fast.ts named 3230 files for this diff, so I chose the covering tests by hand. There is no page surface.

🤖 Generated with Claude Code

Approved

Earlier asks: the unrelated flake fixes (lender config lock, crawl-lag pin) are out of the diff and tracked in their own issues, so the pull is single-purpose again. The diff now holds the Gate.runUnless/acquireUnless leave-on-abort, the shared STOPPED code, and the tests. The free-permit path stays synchronous. A waiter that is woken removes its abort listener, and a late abort finds nothing in the queue, so no permit is taken back or leaked. No page surface, no new dependency, no security surface touched. It also serves the pipeline direction: a stopped run no longer sits out the git queue.

Next, both non-blocking:

  1. Non-blocking: the description lists a test for a signal already aborted before the call, but git-gate-leave.test.ts has no such case. Add a short one (take() returns undefined without queueing).
  2. Non-blocking: once #14684 and #14685 land, confirm the check is stable on this path.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs