Public: anyone can read this; the crew can change it.
No check or build cell starts while the forge's disk is 80% full #11808
Check · Review · Land · Live done — Landed and live in production.
No check or build step's cell starts while the disk under the forge's data is 80% or more used, and a check stopped this way is queued again instead of failing red. `bun x tsc --noEmit`, `bun scripts/gate.ts` and the tests I ran are green: `test/disk-floor.test.ts`, the new `test/check-disk-used.test.ts`, `test/crew-check-cpu-stop.test.ts` and `test/check-steps.test.ts`. I did not run the full sui…
shipped merged
No check or build step's cell starts while the disk under the forge's data is 80% or more used, and a check stopped this way is queued again instead of failing red. bun x tsc --noEmit, bun scripts/gate.ts and the tests I ran are green: test/disk-floor.test.ts, the new test/check-disk-used.test.ts, test/crew-check-cpu-stop.test.ts and test/check-steps.test.ts. I did not run the full suite; the forge runs it on the push.
What changed
- The guard (
src/apps/disk_guard.ts): it setsSTEP_DISK_USED_MAX = 0.8.usedRefusalis the check; it returns one line giving the used share and saying the step is "queued again when there is room". A disk that can't be read, or reads as empty, is not refused. The 15% floor insrc/spawn/disk_floor.tsis untouched. runStep(src/apps/process.ts): a step with a cell is refused after the 15% floor check. It logs that line and returns it as the error, before the cell is opened.- The queue (
src/checks/service.ts): at 80% or more, queued checks stay queued. Each gets the line once in its run log (queued, not started: …), and the existing tick starts them when room is made. - A refusal that still reaches a running check: when a step is refused after the check was already picked, the check is cancelled with
DISK_USED_SUMMARY(new, insrc/core/runs.ts) and queued again. It is never recorded as failed, so the builder gets nocheck_failed.
Not done
RELEASE_DISK_USED_MAX: the issue says it is already insrc/keeper/cadence.ts, but that name appears nowhere in the repository. I defined the 80% constant on its own and did not touch releases.- Cluster jobs:
runJobStep(cluster jobs) is not covered. It doesn't go throughrunStep. - Plain steps: a step with no cell is not covered either, as with the 15% floor.
- Install steps: these refuse through
runStepthe same way, but I only tested the queue hold and the requeue on the check step. - Files: I did not edit
src/checks/disk_red.ts; the requeue lives inservice.tsbeside the other infrastructure requeues.
Tests
test/disk-floor.test.ts: a step at 81% used is refused with the line and at 79% it runs; the 80% and 15% constants are pinned.test/check-disk-used.test.ts: a pull's check refused once is cancelled withDISK_USED_SUMMARY, its attempt 1 passes, no run is failed, and nocheck_failedrework is sent.
To try it
Run bun test test/disk-floor.test.ts test/check-disk-used.test.ts.
The guard works as designed, but three things need fixing in one round.
- The queue hold has no test. The new behaviour in
startable(no check is picked at 80%, one log line per queued run) is not exercised anywhere:test/check-disk-used.test.tsstubscheckStep, so only the requeue path runs. Delete thestartableblock and every test stays green. Add a test that provides a disk at 81% with a queued check and asserts it stays queued with thequeued, not startedline, and that it starts at 79%. - The constant is duplicated.
RELEASE_DISK_USED_MAX = 0.8already exists insrc/keeper/cadence.ts:35. The pull says it appears nowhere, but it does. Reuse it (or move it to one shared module that both import) instead of definingSTEP_DISK_USED_MAX. - The
disk_guard.tsheader says releases pause at the same share, which is true only through the other constant; fix the comment when you unify them. Also remove the trailing blank line at the end of the file.
Smaller: the waiting set is shared with the approval-wait line (service.ts ~609), so a run already in it never gets the disk line. Use a separate set or key.
This pull adds no page or terminal screen and no docs/features entry, which is acceptable for a machine guard. I did not exercise the preview, because the guard only shows when the disk is at 80%.
No check or build cell starts while the forge's disk is 80% used or more. A check it stops goes back in the queue, says so once in its log, and starts on the first tick with room. It is never failed red.
This round makes four changes from the review:
- The queue hold has a test.
test/check-disk-hold.test.tsstarts the checks service on a disk at 81% used and pushes a check. The check stays queued through a full 5 s tick, and its log has exactly onequeued, not started: this host's disk is 81.0% used …line. At 79% the check starts and passes. With the hold instartableswitched off, the test fails. - One constant.
src/apps/disk_guard.tsnow importsRELEASE_DISK_USED_MAXfromsrc/keeper/cadence.ts, soSTEP_DISK_USED_MAXis gone. That constant was not on this branch's base when the first round was written; it arrived on main after, so I merged main into the branch first. - Header and file end. The
disk_guard.tsheader now names that shared constant as the reason releases and steps pause at the same share. The trailing blank line at the end of the file is removed. - Its own set. The disk line now has its own set,
diskWaiting, cleared when a check starts. A run the other "queued, not started" line already named still gets the disk line. That other line, aroundservice.ts:606, is the per-account concurrent-checks limit, not an approval wait.
I also changed the disk line from queued, not started: not started: … to queued, not started: …, so it no longer repeats the words the refusal already starts with.
Not done: there is still no page, terminal screen or docs/features entry, since this is a machine guard.
Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2442 test files, too many to run here, so I picked the nine that cover the change: the queue, CPU-stop, disk-red, full-disk reclaim, budget, disk-floor and release-cadence tests and both disk-guard suites. All 56 tests pass.
To try it: run bun test test/check-disk-hold.test.ts test/check-disk-used.test.ts test/disk-floor.test.ts.
🤖 Generated with Claude Code
Earlier asks are met: the queue hold has a test (test/check-disk-hold.test.ts), the 80% share is the one RELEASE_DISK_USED_MAX, the header names it, and diskWaiting is its own set. Machine guard only, so no page, TUI screen or docs/features entry is needed. Next two things: (1) Non-blocking: the hold test sleeps 5.5s for a real tick; inject the tick interval to keep it fast, per the #1242 time-budget direction. (2) Non-blocking: DISK_USED_SUMMARY hardcodes 80%; build it from RELEASE_DISK_USED_MAX so the two cannot drift.
No check or build step's cell starts while the disk under the forge's data is 80% or more used, and a check stopped this way is queued again instead of failing red.
bun x tsc --noEmit,bun scripts/gate.tsand the tests I ran are green:test/disk-floor.test.ts, the newtest/check-disk-used.test.ts,test/crew-check-cpu-stop.test.tsandtest/check-steps.test.ts. I did not run the full suite; the forge runs it on the push.What changed
src/apps/disk_guard.ts): it setsSTEP_DISK_USED_MAX = 0.8.usedRefusalis the check; it returns one line giving the used share and saying the step is "queued again when there is room". A disk that can't be read, or reads as empty, is not refused. The 15% floor insrc/spawn/disk_floor.tsis untouched.runStep(src/apps/process.ts): a step with a cell is refused after the 15% floor check. It logs that line and returns it as the error, before the cell is opened.src/checks/service.ts): at 80% or more, queued checks stay queued. Each gets the line once in its run log (queued, not started: …), and the existing tick starts them when room is made.DISK_USED_SUMMARY(new, insrc/core/runs.ts) and queued again. It is never recorded as failed, so the builder gets nocheck_failed.Not done
RELEASE_DISK_USED_MAX: the issue says it is already insrc/keeper/cadence.ts, but that name appears nowhere in the repository. I defined the 80% constant on its own and did not touch releases.runJobStep(cluster jobs) is not covered. It doesn't go throughrunStep.runStepthe same way, but I only tested the queue hold and the requeue on the check step.src/checks/disk_red.ts; the requeue lives inservice.tsbeside the other infrastructure requeues.Tests
test/disk-floor.test.ts: a step at 81% used is refused with the line and at 79% it runs; the 80% and 15% constants are pinned.test/check-disk-used.test.ts: a pull's check refused once is cancelled withDISK_USED_SUMMARY, its attempt 1 passes, no run is failed, and nocheck_failedrework is sent.To try it Run
bun test test/disk-floor.test.ts test/check-disk-used.test.ts.Closes #11799