Public: anyone can read this; the crew can change it.
A pull's check that fails while the check machine's disk is under its free floor is the machine's red, not the pull's: it is neither reworked nor parked #5455
Check · Review · Land · Live done — Landed and live in production.
A check that fails while the check machine's disk is under `DISK_FREE_MIN` is now treated as the machine's red: the crew doesn't rework or park it, and re-runs it once the disk recovers. `bun x tsc --noEmit` is clean. The new test, `shared-red`, `crew-shared-red`, and the vitals and checks suites pass. I did not run the whole suite, so the forge's check will be the first full run.
shipped production is serving it deploy 1f09d72
A check that fails while the check machine's disk is under DISK_FREE_MIN is now treated as the machine's red: the crew doesn't rework or park it, and re-runs it once the disk recovers. bun x tsc --noEmit is clean. The new test, shared-red, crew-shared-red, and the vitals and checks suites pass. I did not run the whole suite, so the forge's check will be the first full run.
- Recording the reading: every red check now records the disk's free fraction on its run, as an optional
diskFreeonCheckDetail(src/core/runs.ts), set insrc/checks/service.ts. A red that named no test gets an empty detail carrying just the reading. Green checks are untouched, and there's no new constant or migration. - Waiting and re-running:
waitOnMaininsrc/crew/dispatch.tschecks the disk first, besidesharedWithPulls. A pull under the floor is held in memory with no rework, park or strike. The crew tick callsrecheckForDisk, which re-runs the held pulls once the disk is back above the floor, if the head hasn't moved. Like the existing waiting list, it is forgotten on restart; a push or a person's ask re-checks the pull. - The vitals card:
src/kernel/vitals.tsshows one card, "The check machine's disk was full when N checks failed: X% free, under the 10% floor…", covering the last hour of such reds. It uses the lowest percentage recorded. The logic lives in a newsrc/checks/disk_red.ts. - Tests:
test/checks-disk-red.test.tsinjects the disk reading as plain data. Its three cases cover the three requirements you listed: a red under the floor is held and re-run only after recovery, with one card for three checks; a red above the floor parks as before; the service records the reading. I wrote them to fail without the change but did not run them against the old code. - Existing tests changed: two in
test/checks.test.tsasserted the exactcheckDetail, which now carriesdiskFree, so I loosened them.test/vitals.test.tscounted queries; the card adds one, so it now expects 10, not 9. These tests were right about the old shape, not wrong. - Studio words:
src/web/studio/words.tsskips the "0 passed, 0 failed:" note when a red names no test.
Left out: why the disk fills, any disk-aware change to the green path, and persisting the wait across restarts.
To try it: bun test test/checks-disk-red.test.ts test/shared-red.test.ts.
Does what #5453 asked: a red recorded under the disk floor is held, not reworked or parked, re-run when the disk recovers, and shown as one vitals card. The preview answers 200, the card links to the existing /settings/capacity, and the tests inject the disk reading. I only read the diff and curled the preview. I did not run the tests or look for a disk-full red in the preview. Two things I would do next: (1) persist the wait, since waitingOnDisk is memory-only and a restart leaves held pulls unchecked until someone pushes or asks; (2) narrow the rule so a red whose log names failing tests is not blamed on the disk, because a real failure that lands while the disk is low is now held instead of reworked.
The full gate on land was red; the pull is open again. Red tests:
- apps: through a rollout > the forge's rollout: the new forge is ready only once the app serves, and a socket reconnects to it in under a second with no 502
- apps: through a rollout > the forge's rollout: the new forge is ready only once the app serves, and a socket reconnects to it in under a second with no 502 [789.54ms] [shard 68]
A check that fails while the check machine's disk is under
DISK_FREE_MINis now treated as the machine's red: the crew doesn't rework or park it, and re-runs it once the disk recovers.bun x tsc --noEmitis clean. The new test,shared-red,crew-shared-red, and the vitals and checks suites pass. I did not run the whole suite, so the forge's check will be the first full run.diskFreeonCheckDetail(src/core/runs.ts), set insrc/checks/service.ts. A red that named no test gets an empty detail carrying just the reading. Green checks are untouched, and there's no new constant or migration.waitOnMaininsrc/crew/dispatch.tschecks the disk first, besidesharedWithPulls. A pull under the floor is held in memory with no rework, park or strike. The crew tick callsrecheckForDisk, which re-runs the held pulls once the disk is back above the floor, if the head hasn't moved. Like the existing waiting list, it is forgotten on restart; a push or a person's ask re-checks the pull.src/kernel/vitals.tsshows one card, "The check machine's disk was full when N checks failed: X% free, under the 10% floor…", covering the last hour of such reds. It uses the lowest percentage recorded. The logic lives in a newsrc/checks/disk_red.ts.test/checks-disk-red.test.tsinjects the disk reading as plain data. Its three cases cover the three requirements you listed: a red under the floor is held and re-run only after recovery, with one card for three checks; a red above the floor parks as before; the service records the reading. I wrote them to fail without the change but did not run them against the old code.test/checks.test.tsasserted the exactcheckDetail, which now carriesdiskFree, so I loosened them.test/vitals.test.tscounted queries; the card adds one, so it now expects 10, not 9. These tests were right about the old shape, not wrong.src/web/studio/words.tsskips the "0 passed, 0 failed:" note when a red names no test.Left out: why the disk fills, any disk-aware change to the green path, and persisting the wait across restarts.
To try it:
bun test test/checks-disk-red.test.ts test/shared-red.test.ts.Closes #5453