Public: anyone can read this; the crew can change it.

A child forge whose only run is paused on a limit may sleep: it no longer holds a slot of the host's awake ceiling for days #11566

Merged grokbot opened this 16h ago

Check · Review · Land · Live done — Landed and live in production.

Merged

### Why `crewAtWork` (added in #10364) keeps a child forge awake while its crew has a run `queued` or `running`. A queued run that is paused on a limit (`not_before` ahead, e.g. a provider's weekly limit) also counted, so such a child never slept and held one slot of the host's `FORGE_CHILD_AWAKE_MAX` ceiling for days. On 2026-11-11 the grokbot-agency child got 429 "the forge is full, try again sh…

shipped production is serving it deploy e1bbdcb

grokbot opened this 16h ago

Why

crewAtWork (added in #10364) keeps a child forge awake while its crew has a run queued or running. A queued run that is paused on a limit (not_before ahead, e.g. a provider's weekly limit) also counted, so such a child never slept and held one slot of the host's FORGE_CHILD_AWAKE_MAX ceiling for days. On 2026-11-11 the grokbot-agency child got 429 "the forge is full, try again shortly", which is what this kind of leak produces.

What

A queued run counts as work only once its not_before is past (or unset). A paused run starts when the forge next wakes, as before. test/spawn-cell.test.ts covers the paused, due and running cases.

Validation

validate on 3fa3d3fc: flaky in 283s (forge.json check.fast, verbatim, in a scrubbed env).

bun install --frozen-lockfile: ok (exit 0, 0s)
forge.json check.fast: FLAKY (exit 1, 283s)

trevatoowner Changes requested 2h ago— before the latest push

Thank you for this; the direction is right (a child whose only run is paused on a limit should be allowed to sleep, and it frees the host's awake ceiling). Three things before it lands, all in src/spawn/cell.ts crewAtWork:

Nothing wakes the child when not_before falls due. The comment says the run "starts when the forge next wakes", which in practice means a visitor. A paused run could then sleep past its time indefinitely. Either arm a wake for the earliest not_before when the child goes to sleep, or say in the code where that already happens and add a test that a child asleep with a run due at T is awake and running it shortly after T.
A missing not_before column now reads as "no work". The query sits under the same catch { return false } as before; on an older child's schema it throws, a genuinely running run is counted as nothing, the child sleeps and the run is killed. On error, fall back to the old test (any queued or running run counts as work).
The time comparison is a string compare (not_before <= ISO). Confirm the crew writes that column as a UTC ISO string with the same precision, or compare as dates; add one test with the format the dispatcher really writes.

The check is red with no run recorded on this head; I have asked for a re-check, so do not chase that one.

grokbot commented 57m ago

Thanks. All three are addressed in ed63fb6c (merged with main as 0307ddd8):

Waking when not_before falls due. When the idle sweep puts a cell child to sleep, sleepIdleChildren reads the earliest not_before of its queued runs (nextRunDue in src/spawn/cell.ts) and arms a wake for that moment (wakeWhenDue in src/spawn/host.ts). At that time the cell starts through settleCell and the child's own dispatcher picks up the run. The wake follows the host's limits. It does nothing if the child is already awake, if its billing is none, or if it was demoted or removed. At the awake ceiling or below the disk floor it tries again a minute later. A wait longer than setTimeout's limit is re-armed in steps. closeHostedChildren clears all armed wakes. New test: "a cell put to sleep with a run due at T is woken at T, not before, and its run then counts as work".
Older schema. If the not_before query throws, crewAtWork now falls back to the old test: any queued or running run counts as work. Only a database that can't be opened at all still answers no. New test: "on a schema older than not_before a queued or running run is still work, never nothing". It uses a run table without the column.
Time format. I kept the dispatcher's own rule (not_before <= now as text, as core/runs.ts queueDepth and the eligibility query use), so this says a run is due exactly when the dispatcher would start it. Every writer stores the column as Date#toISOString() (UTC, milliseconds, Z). resumeLater gets it from pause.resumeAt, which the runner sets from toISOString() or from subscription.resting_until, and that is written with until.toISOString() in crew/limits.ts. New test: "a run paused as the dispatcher writes it is due at its moment, not a millisecond before…". It queues the run through createRun with that format and checks that crewAtWork agrees with queueDepth at T−1 ms and at T.

I merged main into the branch instead of force-pushing. That makes the head its own combined build, so the shallow history the check gets carries the commit that has the Try line. Local validate: flaky in 444s (forge.json check.fast; the gate passes, and the reds are unrelated box flakes that pass when rerun: chromium, egress DNS, terminal timing).

trevatoowner commented 55m ago

Thank you, @grokbot: pull #11566 landed in 7f8ff39713db.

3 commits by grokbot touched 3 files, landed as a merge, so your commits are ancestors of main and git merge main on your side stays clean.

What changed from your head on the way:

conflicts resolved: 1 merge of main into your branch (0307ddd8ff15 Merge main into fix/cell-sleep-paused-runs so the head is its own combined build)