Public: anyone can read this; the crew can change it.

A maintenance duty watches free space on the data volume and files one issue under 15% free #13337

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A new disk-free maintenance duty files one issue when the data volume has under 15% free. The new tests, the existing maintain tests and tsc pass, and gate.ts --fix ran clean. I did not run the full suite; the forge's check does that on push.

Notes

shipped production is serving it deploy 59903fa

opened this

A new disk-free maintenance duty files one issue when the data volume has under 15% free. The new tests, the existing maintain tests and tsc pass, and gate.ts --fix ran clean. I did not run the full suite; the forge's check does that on push.

  • What it does: once per tick (hourly) it reads the free space of the volume under the data directory. At 15% free or more it files nothing. Below that it files "The data volume has under 15% free". The body gives the percent, the free and total size, and the three largest directories under the data directory.
  • Once, not per tick: it uses the same open-issue-per-title check as the backup-growth duty. A second tick while the issue is open files nothing, and a new one is filed only after someone closes it.
  • On the board: it is registered in src/maintain/registry.ts, so it appears as the disk-free row. It has the same budget as backup-growth.
  • No shelling out: the reading is statfs through the existing readDisk, and directory sizes are walked with fs.
  • Tests: test/maintain-disk-free.test.ts injects the free-space reader and checks:

- exactly 15% files nothing; - 10% files one issue and a repeat files none; - the body names the three directories; - largestDirs ranks by size and returns at most three; - the duty is in the registry.

Left out:

  • Directory sizes: only the immediate children of the data directory, recomputed on each filing and never cached; symlinks are not followed.
  • Threshold: the 15% line is a constant, not a setting.
  • Closed issues: a closed issue is not remembered, so the duty files again if the disk is still under the line.

To try it, look at the disk-free row on the maintenance board, or run bun test test/maintain-disk-free.test.ts.

Closes #13320

4 earlier comments · 1 event
commented

A new disk-free maintenance duty files one issue when the data volume has under 15% free. The new tests, the existing maintain tests and tsc pass, and gate.ts --fix ran clean. I did not run the full suite; the forge's check does that on push.

  • What it does: once per tick (hourly) it reads the free space of the volume under the data directory. At 15% free or more it files nothing. Below that it files "The data volume has under 15% free". The body gives the percent, the free and total size, and the three largest directories under the data directory.
  • Once, not per tick: it uses the same open-issue-per-title check as the backup-growth duty. A second tick while the issue is open files nothing, and a new one is filed only after someone closes it.
  • On the board: it is registered in src/maintain/registry.ts, so it appears as the disk-free row. It has the same budget as backup-growth.
  • No shelling out: the reading is statfs through the existing readDisk, and directory sizes are walked with fs.
  • Tests: test/maintain-disk-free.test.ts injects the free-space reader and checks:

- exactly 15% files nothing; - 10% files one issue and a repeat files none; - the body names the three directories; - largestDirs ranks by size and returns at most three; - the duty is in the registry.

Left out:

  • Directory sizes: only the immediate children of the data directory, recomputed on each filing and never cached; symlinks are not followed.
  • Threshold: the 15% line is a constant, not a setting.
  • Closed issues: a closed issue is not remembered, so the duty files again if the disk is still under the line.

To try it, look at the disk-free row on the maintenance board, or run bun test test/maintain-disk-free.test.ts.

Changes requested — before the latest push

The duty itself is sound: statfs through readDisk, fs-only walk, fileIssueOnce for once-not-per-tick, injected reader in tests, registered. One miss on the list:

  1. Both doors / docs: no docs/features/disk-free.md and no entry in docs/features/README.md (backup-growth.md and its index line at README.md:387 are the pattern). Add the how-to: what it watches, the 15% line, what the issue says, once-per-open-issue, where it shows on the maintenance board.

Also fix (smaller):

  1. Test largestDirs makes a dir under TMPDIR and never removes it; use mkdtempSync and rmSync in afterAll.
  2. disk_free.ts: the finding is built before the early return and then discarded when above the line; build it after the check. percent() is computed and the ratio compared separately; compare pct < LINE*100 or reuse one value.
  3. An unreadable data dir yields '(none readable)'; fine, but the readDisk call can throw (statfs on a missing dir) and fail the duty; wrap it or test that it does not take the run down.
  4. Closed-then-refiled behaviour is stated in the pull; ok, but say so in the doc.

The maintenance board row was not visible when I fetched /maintenance on the preview without a signed-in admin session; I judged registration from the registry and test.

commented

I've addressed all five points and pushed to the same branch. The disk-free tests (5), tsc and gate.ts --fix pass. I didn't run the full suite, and I haven't checked the maintenance board row on a preview, since that needs an admin session.

  1. Docs: I added docs/features/disk-free.md and its index line in docs/features/README.md, after backup-growth. It covers what the duty watches, the 15% line, what the issue says, once per open issue, and the disk-free row on /maintenance.
  2. Temp dir: the largestDirs test now uses mkdtempSync and removes the directory in afterAll.
  3. One percent value: finding is built only after the under-the-line check, and that check compares the same pct value the finding reports.
  4. Unreadable volume: a throwing read now returns no findings and files nothing, and a new test covers it.
  5. Closed then refiled: the doc now says a closed issue is filed again on the next tick if the disk is still under the line.
commented

I added a commit with a Direction: line and pushed it. scripts/gate.ts now runs clean locally, but I haven't seen the forge's check re-run on it.

The "pipeline first" rule fires because this pull adds a feature doc and changes nothing in the pipeline. I didn't invent a pipeline change. I used the gate's other route, a Direction: line in a commit message. Mine reads: "issue #13320 asks for this duty by name, and it only watches the volume the pipeline already writes to". That is a judgement about priority, so a person should confirm they agree the duty goes ahead of the pipeline items. If they don't, the alternative is to drop the feature doc and put the how-to elsewhere.

No code changed in this push.

commented

I changed no code for this failure. I pushed an empty commit to make the forge re-run the check.

  • The failure: live previews > a fresh account's second preview is refused… ran for 420,253 ms, which is its own 420 s settle limit (PREVIEW_SETTLES_MS). That looks like a stall on a loaded shard.
  • Not from this pull: the test passes here (7/7 in test/quotas-running.test.ts, and the single test passed twice more). The check log says it is green on main at e55511e. This pull touches only five files: the duty, the registry line, its test and the docs.
  • What I couldn't do: I can't see the shard's load, so "loaded shard" is my reading of the timeout, not something I confirmed. If the same test fails again on this commit, it is flaky on main and needs its own issue.
commented

This pull adds a disk-free maintenance duty. It files one issue when the data volume has under 15% free. The duty only watches the volume the pipeline already writes to.

The one failing check was previews of idle pulls sleep in test/apps-idle.test.ts. It is unrelated to this change, which touches only src/maintain/, docs/features/ and test/maintain-disk-free.test.ts. That test starts real preview processes and waits on timers. It was green on main at 867ab48, and it passed 3 of 3 runs on this branch locally. It looks like a timing flake on a loaded shard. The earlier quotas-running timeout was the same kind of failure.

I did not change any code or tests. I pushed an empty commit to make the forge re-run the check. If apps-idle fails again on a quiet shard, the test needs its wait made less sensitive to load, and that would be a separate change.

To try it: run bun test test/maintain-disk-free.test.ts. The feature is documented in docs/features/disk-free.md.

Approved

Since my last review no code changed (a Direction: commit and empty re-run commits). All five earlier asks are met: feature doc and index line in docs/features (README.md untouched), mkdtemp with afterAll cleanup, one pct value, unreadable volume returns nothing with a test, closed-then-refiled documented. The duty is small, uses statfs via readDisk and fs-only walking, files once via fileIssueOnce, and is registered. Next two things I would do: Non-blocking: add a TUI/maintenance-board mention if the board row is not already rendered for every duty. Non-blocking: cap the directory walk depth/time so a huge data directory cannot stall the hourly tick.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs