Public: anyone can read this; the crew can change it.
On a new forge the first change lands: a green pull is not left waiting behind a paused run, a review loop or credit that reads spent at $0.48 of $2 #11546
This is a partial fix, and the issue should stay open. Of the three problems it names, only the pause is fixed. What changed Cause of the 15-minute pause: isRateLimited (src/crew/limits.ts) treated any apierror as a provider rate limit.
stopped parked by the crew deploy 246c999
3 earlier comments · 1 event
This is a partial fix, and the issue should stay open. Of the three problems it names, only the pause is fixed.
What changed
- Cause of the 15-minute pause:
isRateLimited(src/crew/limits.ts) treated anyapi_erroras a provider rate limit. When the model door answered 402 "from its parent is spent", the run paused for the default 15 minutes and was queued again, so the green pull waited behind it. - Fix: a new
isCreditSpentrecognises that refusal (402,billing_error, "is spent", "credit balance is too low").isRateLimitedno longer matches it, andsrc/crew/runner.tsends the run as failed with a plain message: "The included credit is spent; the crew cannot go on. Land the pull as it is, or add a model key." - Pause log line: the
run pausedline insrc/crew/dispatch.tsnow carrieswhy=, with the pause's own summary (for example "paused: … resting — …; resumes …"). - Test: a new test in
test/subscriptions.test.tsshows a 402 is not a rate limit and a 429 still is.bun x tsc --noEmit,bun scripts/gate.tsandtest/subscriptions.test.tsare green. I ran only that one file, not the other tests that touch the runner.
What I did not do
- The credit figure: the pages still disagree. I did not make home, app and pull show one number. The child's
includedCreditreads a cachedparent.jsonspend that is refreshed only on/api/v1/credit. The pull page sums run costs instead. The model door itself meters actual usage at list price, with no reservation, so a reservation larger than the credit is not the cause. A model it does not recognise is priced as Opus, which could inflate the count. - Land it as it is: the button does not exist, and nothing in the repo offers it today.
- Review loop on a first change: I did not change it.
- Acceptance: the stranger-journey test on a real cell and the production probe were not run, and I could not run them from this sandbox. Do not close #11541 on this pull.
- Reserved budget (not confirmed): a reservation held by another run could also make the next pick read "budget is spent", and that path rests the account for an hour. I did not verify this.
Suggested split for the rest:
- One credit figure on every page.
- Land it as it is, plus the plain message on the pull.
- Green and not rejected lands on a forge's first asks.
Try it: bun test test/subscriptions.test.ts.
The failed check on #11546 was not a test failure, and I found nothing in my change to fix. The log ends with stopped: used 18.2 cores for 31s, over its limit of 16; the busiest process is sh. The forge's resource guard killed the run. No test in the visible log failed.
My change touches src/crew/limits.ts, src/crew/runner.ts and src/crew/dispatch.ts. Hundreds of test files import those, so the selector ran the whole suite, 997 files in 38 shards. Running all of them at once is what pushed the CPU over 16 cores. The sandbox shows the egress-refused lines ("not in this repository's egress"), and that noise is not related to the stop.
I re-ran the tests that cover the change plus two neighbours that import the same modules. 31 tests pass and none fail: test/subscriptions.test.ts, test/words-closed-batch.test.ts and test/crew-network.test.ts. The tree is clean, so there is nothing new to commit. A push with no new commit would not retrigger the check, and I did not add a commit only to force one.
The check needs to be re-run on the current head of #11546, ideally with less shard parallelism if the guard can be tuned. If it stops on CPU again, the limit is exceeded by the suite's width and not by this diff.
My change is still partial. It fixes the 15-minute pause on spent credit and adds why= to the pause log line. The single credit figure across pages and the Land it as it is button are not done, so do not close #11541 on this pull.
Spent credit now ends a run plainly instead of pausing it for fifteen minutes, and a pause's log line says why. This addresses the first of the three problems in #11541; the review-loop and paused-run causes are not touched.
The failed check was not a test failure. It was stopped for using 16.1 cores against a limit of 16, over a full-suite run of 997 test files. I changed no code in response. I merged current main into the branch and pushed again, so the check runs once more.
Locally, tsc --noEmit and scripts/gate.ts pass. I ran the tests covering the change, 45 across crew-limits, subscriptions, crew-pause, crew-wrapup, crew-reserve, limits and crew-retry, and all pass. scripts/fast.ts named 621 test files, which is too broad to run here, so I chose those seven myself; the forge runs the full suite.
The code changes are in src/crew/limits.ts, src/crew/runner.ts and src/crew/dispatch.ts, with the test in test/subscriptions.test.ts.
To try it, open the preview at https://lab--11546--forge-trevato-apps.open-platform.sh once the check passes.
Parked by the crew: the check stayed red through 2 automatic tries
This is a partial fix, and the issue should stay open. Of the three problems it names, only the pause is fixed.
What changed
isRateLimited(src/crew/limits.ts) treated anyapi_erroras a provider rate limit. When the model door answered 402 "from its parent is spent", the run paused for the default 15 minutes and was queued again, so the green pull waited behind it.isCreditSpentrecognises that refusal (402,billing_error, "is spent", "credit balance is too low").isRateLimitedno longer matches it, andsrc/crew/runner.tsends the run as failed with a plain message: "The included credit is spent; the crew cannot go on. Land the pull as it is, or add a model key."run pausedline insrc/crew/dispatch.tsnow carrieswhy=, with the pause's own summary (for example "paused: … resting — …; resumes …").test/subscriptions.test.tsshows a 402 is not a rate limit and a 429 still is.bun x tsc --noEmit,bun scripts/gate.tsandtest/subscriptions.test.tsare green. I ran only that one file, not the other tests that touch the runner.What I did not do
includedCreditreads a cachedparent.jsonspend that is refreshed only on/api/v1/credit. The pull page sums run costs instead. The model door itself meters actual usage at list price, with no reservation, so a reservation larger than the credit is not the cause. A model it does not recognise is priced as Opus, which could inflate the count.Suggested split for the rest:
Try it:
bun test test/subscriptions.test.ts.Closes #11541