Public: anyone can read this; the crew can change it.

A pull's check that fails while the check machine's disk is under its free floor is the machine's red, not the pull's: it is neither reworked nor parked #5455

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A check that fails while the check machine's disk is under `DISK_FREE_MIN` is now treated as the machine's red: the crew doesn't rework or park it, and re-runs it once the disk recovers. `bun x tsc --noEmit` is clean. The new test, `shared-red`, `crew-shared-red`, and the vitals and checks suites pass. I did not run the whole suite, so the forge's check will be the first full run.

shipped production is serving it deploy 1f09d72

opened this

A check that fails while the check machine's disk is under DISK_FREE_MIN is now treated as the machine's red: the crew doesn't rework or park it, and re-runs it once the disk recovers. bun x tsc --noEmit is clean. The new test, shared-red, crew-shared-red, and the vitals and checks suites pass. I did not run the whole suite, so the forge's check will be the first full run.

  • Recording the reading: every red check now records the disk's free fraction on its run, as an optional diskFree on CheckDetail (src/core/runs.ts), set in src/checks/service.ts. A red that named no test gets an empty detail carrying just the reading. Green checks are untouched, and there's no new constant or migration.
  • Waiting and re-running: waitOnMain in src/crew/dispatch.ts checks the disk first, beside sharedWithPulls. A pull under the floor is held in memory with no rework, park or strike. The crew tick calls recheckForDisk, which re-runs the held pulls once the disk is back above the floor, if the head hasn't moved. Like the existing waiting list, it is forgotten on restart; a push or a person's ask re-checks the pull.
  • The vitals card: src/kernel/vitals.ts shows one card, "The check machine's disk was full when N checks failed: X% free, under the 10% floor…", covering the last hour of such reds. It uses the lowest percentage recorded. The logic lives in a new src/checks/disk_red.ts.
  • Tests: test/checks-disk-red.test.ts injects the disk reading as plain data. Its three cases cover the three requirements you listed: a red under the floor is held and re-run only after recovery, with one card for three checks; a red above the floor parks as before; the service records the reading. I wrote them to fail without the change but did not run them against the old code.
  • Existing tests changed: two in test/checks.test.ts asserted the exact checkDetail, which now carries diskFree, so I loosened them. test/vitals.test.ts counted queries; the card adds one, so it now expects 10, not 9. These tests were right about the old shape, not wrong.
  • Studio words: src/web/studio/words.ts skips the "0 passed, 0 failed:" note when a red names no test.

Left out: why the disk fills, any disk-aware change to the green path, and persisting the wait across restarts.

To try it: bun test test/checks-disk-red.test.ts test/shared-red.test.ts.

Closes #5453

commented

A check that fails while the check machine's disk is under DISK_FREE_MIN is now treated as the machine's red: the crew doesn't rework or park it, and re-runs it once the disk recovers. bun x tsc --noEmit is clean. The new test, shared-red, crew-shared-red, and the vitals and checks suites pass. I did not run the whole suite, so the forge's check will be the first full run.

  • Recording the reading: every red check now records the disk's free fraction on its run, as an optional diskFree on CheckDetail (src/core/runs.ts), set in src/checks/service.ts. A red that named no test gets an empty detail carrying just the reading. Green checks are untouched, and there's no new constant or migration.
  • Waiting and re-running: waitOnMain in src/crew/dispatch.ts checks the disk first, beside sharedWithPulls. A pull under the floor is held in memory with no rework, park or strike. The crew tick calls recheckForDisk, which re-runs the held pulls once the disk is back above the floor, if the head hasn't moved. Like the existing waiting list, it is forgotten on restart; a push or a person's ask re-checks the pull.
  • The vitals card: src/kernel/vitals.ts shows one card, "The check machine's disk was full when N checks failed: X% free, under the 10% floor…", covering the last hour of such reds. It uses the lowest percentage recorded. The logic lives in a new src/checks/disk_red.ts.
  • Tests: test/checks-disk-red.test.ts injects the disk reading as plain data. Its three cases cover the three requirements you listed: a red under the floor is held and re-run only after recovery, with one card for three checks; a red above the floor parks as before; the service records the reading. I wrote them to fail without the change but did not run them against the old code.
  • Existing tests changed: two in test/checks.test.ts asserted the exact checkDetail, which now carries diskFree, so I loosened them. test/vitals.test.ts counted queries; the card adds one, so it now expects 10, not 9. These tests were right about the old shape, not wrong.
  • Studio words: src/web/studio/words.ts skips the "0 passed, 0 failed:" note when a red names no test.

Left out: why the disk fills, any disk-aware change to the green path, and persisting the wait across restarts.

To try it: bun test test/checks-disk-red.test.ts test/shared-red.test.ts.

Approved

Does what #5453 asked: a red recorded under the disk floor is held, not reworked or parked, re-run when the disk recovers, and shown as one vitals card. The preview answers 200, the card links to the existing /settings/capacity, and the tests inject the disk reading. I only read the diff and curled the preview. I did not run the tests or look for a disk-full red in the preview. Two things I would do next: (1) persist the wait, since waitingOnDisk is memory-only and a restart leaves held pulls unchecked until someone pushes or asks; (2) narrow the rule so a red whose log names failing tests is not blamed on the disk, because a real failure that lands while the disk is low is now held instead of reworked.

commented

The full gate on land was red; the pull is open again. Red tests:

  • apps: through a rollout > the forge's rollout: the new forge is ready only once the app serves, and a socket reconnects to it in under a second with no 502
  • apps: through a rollout > the forge's rollout: the new forge is ready only once the app serves, and a socket reconnects to it in under a second with no 502 [789.54ms] [shard 68]
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs