Public: anyone can read this; the crew can change it.

A crew run paused for credit or a rate limit gives up its slot and says why and when it resumes #11580

opened this

Check · Review · Land · Live — Its check is red. Next: builder, its check is red.

Ready to land

The `run paused` log line now names its cause, and the paused run's card now shows the provider's own wording with the resume time. I did not change how slots are counted, because I found no code path where a paused run holds one.

4 files · +82 −1

stopped failed before it answered deploy d46c3f1

Check failed: check failed: stopped: used 16.6 cores for 31s, over its limit of 16; the busie…Check of d46c3f1checked against main at f21b126fast tier: red in 36s

Website spec: 23 findings (report only)

Look

9 pages beside main · 0 new to this pull

/apps

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/explore

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/feed

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/front

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/map

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/notes

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/releases

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/repos

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

/review

main

at f21b126 ·

this pull

In the dark

main

at f21b126 ·

this pull

at d46c3f1 ·

opened this

The run paused log line now names its cause, and the paused run's card now shows the provider's own wording with the resume time. I did not change how slots are counted, because I found no code path where a paused run holds one.

What I found about slots. pump counts only running runs against concurrency. A paused run is cancelled, then re-queued with a notBefore time, and listQueued skips it until that time. So I could not reproduce the 17-minute hold from the code. The test below shows a queued run starting while a paused one waits. If the hold happens in production, the cause is something other than this path, and I did not find it.

What changed

  • Pause in src/crew/runner.ts now carries the run's own pause summary, for example paused: <account>'s 5h window is full until 00:52; resumes 00:52. Before, the card fell back to a generic "paused on the provider's limit" line.
  • pauseCause() in src/crew/pacing.ts reads that summary and returns a word: credit, rate limit, network or other. resumeLater in src/crew/dispatch.ts puts it in the log line as cause=.

Cause classification. It is keyword matching on the summary text. The credit keywords are a guess at what a provider says, so the word may be wrong for a message I did not anticipate.

Tests. test/crew-pause-slot.test.ts runs a real crew with concurrency 1. Run A is queued and paused for an hour, and run B is assigned. B starts and runs, while A stays queued with its notBefore time and its summary. The same file checks that pauseCause names credit, rate limit and network.

  • Run: bun test test/crew-pause-slot.test.ts passed 2 of 2, and bun x tsc --noEmit and bun scripts/gate.ts both passed. I did not run the full suite; the forge runs it on the push.
  • Not changed: I added no new card markup, since the card already shows the run's summary. The test does not assert how the card renders.

Closes #11564

commented

The run paused log line now names its cause, and the paused run's card now shows the provider's own wording with the resume time. I did not change how slots are counted, because I found no code path where a paused run holds one.

What I found about slots. pump counts only running runs against concurrency. A paused run is cancelled, then re-queued with a notBefore time, and listQueued skips it until that time. So I could not reproduce the 17-minute hold from the code. The test below shows a queued run starting while a paused one waits. If the hold happens in production, the cause is something other than this path, and I did not find it.

What changed

  • Pause in src/crew/runner.ts now carries the run's own pause summary, for example paused: <account>'s 5h window is full until 00:52; resumes 00:52. Before, the card fell back to a generic "paused on the provider's limit" line.
  • pauseCause() in src/crew/pacing.ts reads that summary and returns a word: credit, rate limit, network or other. resumeLater in src/crew/dispatch.ts puts it in the log line as cause=.

Cause classification. It is keyword matching on the summary text. The credit keywords are a guess at what a provider says, so the word may be wrong for a message I did not anticipate.

Tests. test/crew-pause-slot.test.ts runs a real crew with concurrency 1. Run A is queued and paused for an hour, and run B is assigned. B starts and runs, while A stays queued with its notBefore time and its summary. The same file checks that pauseCause names credit, rate limit and network.

  • Run: bun test test/crew-pause-slot.test.ts passed 2 of 2, and bun x tsc --noEmit and bun scripts/gate.ts both passed. I did not run the full suite; the forge runs it on the push.
  • Not changed: I added no new card markup, since the card already shows the run's summary. The test does not assert how the card renders.
commented

Nothing in the code needed fixing, so I pushed nothing. The deploy log you sent doesn't show a failure.

  • The log: it ends with the preview "running", and "no tests declared, none run" is a note, not an error. The ERROR line about the restore drill (nowhere to drill from) comes from the host forge's own background job. It isn't from this app or this change.
  • Checks run locally: bun scripts/gate.ts and bun x tsc --noEmit both pass. 15 tests across 4 files pass with 0 failures, including test/crew-pause-slot.test.ts, which shows a paused run leaves the only slot to a queued one. I picked those test files myself because scripts/fast.ts named 615 files, too broad for this sandbox.
  • Left running: a background job from my first, broader test attempt may still be running. I did not use its output.

The pull is unchanged from d46c3f19f. A paused run's log line now names its cause, and its card keeps the provider's words. If the forge's check still reports red on this commit, send me that log, because the deploy log alone doesn't name a cause.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs