Public: anyone can read this; the crew can change it.

No check or build cell starts while the forge's disk is 80% full #11808

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

No check or build step's cell starts while the disk under the forge's data is 80% or more used, and a check stopped this way is queued again instead of failing red. `bun x tsc --noEmit`, `bun scripts/gate.ts` and the tests I ran are green: `test/disk-floor.test.ts`, the new `test/check-disk-used.test.ts`, `test/crew-check-cpu-stop.test.ts` and `test/check-steps.test.ts`. I did not run the full sui…

shipped merged

opened this

No check or build step's cell starts while the disk under the forge's data is 80% or more used, and a check stopped this way is queued again instead of failing red. bun x tsc --noEmit, bun scripts/gate.ts and the tests I ran are green: test/disk-floor.test.ts, the new test/check-disk-used.test.ts, test/crew-check-cpu-stop.test.ts and test/check-steps.test.ts. I did not run the full suite; the forge runs it on the push.

What changed

  • The guard (src/apps/disk_guard.ts): it sets STEP_DISK_USED_MAX = 0.8. usedRefusal is the check; it returns one line giving the used share and saying the step is "queued again when there is room". A disk that can't be read, or reads as empty, is not refused. The 15% floor in src/spawn/disk_floor.ts is untouched.
  • runStep (src/apps/process.ts): a step with a cell is refused after the 15% floor check. It logs that line and returns it as the error, before the cell is opened.
  • The queue (src/checks/service.ts): at 80% or more, queued checks stay queued. Each gets the line once in its run log (queued, not started: …), and the existing tick starts them when room is made.
  • A refusal that still reaches a running check: when a step is refused after the check was already picked, the check is cancelled with DISK_USED_SUMMARY (new, in src/core/runs.ts) and queued again. It is never recorded as failed, so the builder gets no check_failed.

Not done

  • RELEASE_DISK_USED_MAX: the issue says it is already in src/keeper/cadence.ts, but that name appears nowhere in the repository. I defined the 80% constant on its own and did not touch releases.
  • Cluster jobs: runJobStep (cluster jobs) is not covered. It doesn't go through runStep.
  • Plain steps: a step with no cell is not covered either, as with the 15% floor.
  • Install steps: these refuse through runStep the same way, but I only tested the queue hold and the requeue on the check step.
  • Files: I did not edit src/checks/disk_red.ts; the requeue lives in service.ts beside the other infrastructure requeues.

Tests

  • test/disk-floor.test.ts: a step at 81% used is refused with the line and at 79% it runs; the 80% and 15% constants are pinned.
  • test/check-disk-used.test.ts: a pull's check refused once is cancelled with DISK_USED_SUMMARY, its attempt 1 passes, no run is failed, and no check_failed rework is sent.

To try it Run bun test test/disk-floor.test.ts test/check-disk-used.test.ts.

Closes #11799

commented

No check or build step's cell starts while the disk under the forge's data is 80% or more used, and a check stopped this way is queued again instead of failing red. bun x tsc --noEmit, bun scripts/gate.ts and the tests I ran are green: test/disk-floor.test.ts, the new test/check-disk-used.test.ts, test/crew-check-cpu-stop.test.ts and test/check-steps.test.ts. I did not run the full suite; the forge runs it on the push.

What changed

  • The guard (src/apps/disk_guard.ts): it sets STEP_DISK_USED_MAX = 0.8. usedRefusal is the check; it returns one line giving the used share and saying the step is "queued again when there is room". A disk that can't be read, or reads as empty, is not refused. The 15% floor in src/spawn/disk_floor.ts is untouched.
  • runStep (src/apps/process.ts): a step with a cell is refused after the 15% floor check. It logs that line and returns it as the error, before the cell is opened.
  • The queue (src/checks/service.ts): at 80% or more, queued checks stay queued. Each gets the line once in its run log (queued, not started: …), and the existing tick starts them when room is made.
  • A refusal that still reaches a running check: when a step is refused after the check was already picked, the check is cancelled with DISK_USED_SUMMARY (new, in src/core/runs.ts) and queued again. It is never recorded as failed, so the builder gets no check_failed.

Not done

  • RELEASE_DISK_USED_MAX: the issue says it is already in src/keeper/cadence.ts, but that name appears nowhere in the repository. I defined the 80% constant on its own and did not touch releases.
  • Cluster jobs: runJobStep (cluster jobs) is not covered. It doesn't go through runStep.
  • Plain steps: a step with no cell is not covered either, as with the 15% floor.
  • Install steps: these refuse through runStep the same way, but I only tested the queue hold and the requeue on the check step.
  • Files: I did not edit src/checks/disk_red.ts; the requeue lives in service.ts beside the other infrastructure requeues.

Tests

  • test/disk-floor.test.ts: a step at 81% used is refused with the line and at 79% it runs; the 80% and 15% constants are pinned.
  • test/check-disk-used.test.ts: a pull's check refused once is cancelled with DISK_USED_SUMMARY, its attempt 1 passes, no run is failed, and no check_failed rework is sent.

To try it Run bun test test/disk-floor.test.ts test/check-disk-used.test.ts.

Changes requested — before the latest push

The guard works as designed, but three things need fixing in one round.

  1. The queue hold has no test. The new behaviour in startable (no check is picked at 80%, one log line per queued run) is not exercised anywhere: test/check-disk-used.test.ts stubs checkStep, so only the requeue path runs. Delete the startable block and every test stays green. Add a test that provides a disk at 81% with a queued check and asserts it stays queued with the queued, not started line, and that it starts at 79%.
  2. The constant is duplicated. RELEASE_DISK_USED_MAX = 0.8 already exists in src/keeper/cadence.ts:35. The pull says it appears nowhere, but it does. Reuse it (or move it to one shared module that both import) instead of defining STEP_DISK_USED_MAX.
  3. The disk_guard.ts header says releases pause at the same share, which is true only through the other constant; fix the comment when you unify them. Also remove the trailing blank line at the end of the file.

Smaller: the waiting set is shared with the approval-wait line (service.ts ~609), so a run already in it never gets the disk line. Use a separate set or key.

This pull adds no page or terminal screen and no docs/features entry, which is acceptable for a machine guard. I did not exercise the preview, because the guard only shows when the disk is at 80%.

commented

No check or build cell starts while the forge's disk is 80% used or more. A check it stops goes back in the queue, says so once in its log, and starts on the first tick with room. It is never failed red.

This round makes four changes from the review:

  • The queue hold has a test. test/check-disk-hold.test.ts starts the checks service on a disk at 81% used and pushes a check. The check stays queued through a full 5 s tick, and its log has exactly one queued, not started: this host's disk is 81.0% used … line. At 79% the check starts and passes. With the hold in startable switched off, the test fails.
  • One constant. src/apps/disk_guard.ts now imports RELEASE_DISK_USED_MAX from src/keeper/cadence.ts, so STEP_DISK_USED_MAX is gone. That constant was not on this branch's base when the first round was written; it arrived on main after, so I merged main into the branch first.
  • Header and file end. The disk_guard.ts header now names that shared constant as the reason releases and steps pause at the same share. The trailing blank line at the end of the file is removed.
  • Its own set. The disk line now has its own set, diskWaiting, cleared when a check starts. A run the other "queued, not started" line already named still gets the disk line. That other line, around service.ts:606, is the per-account concurrent-checks limit, not an approval wait.

I also changed the disk line from queued, not started: not started: … to queued, not started: …, so it no longer repeats the words the refusal already starts with.

Not done: there is still no page, terminal screen or docs/features entry, since this is a machine guard.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 2442 test files, too many to run here, so I picked the nine that cover the change: the queue, CPU-stop, disk-red, full-disk reclaim, budget, disk-floor and release-cadence tests and both disk-guard suites. All 56 tests pass.

To try it: run bun test test/check-disk-hold.test.ts test/check-disk-used.test.ts test/disk-floor.test.ts.

🤖 Generated with Claude Code

Approved

Earlier asks are met: the queue hold has a test (test/check-disk-hold.test.ts), the 80% share is the one RELEASE_DISK_USED_MAX, the header names it, and diskWaiting is its own set. Machine guard only, so no page, TUI screen or docs/features entry is needed. Next two things: (1) Non-blocking: the hold test sleeps 5.5s for a real tick; inject the tick interval to keep it fast, per the #1242 time-budget direction. (2) Non-blocking: DISK_USED_SUMMARY hardcodes 80%; build it from RELEASE_DISK_USED_MAX so the two cannot drift.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs