Public: anyone can read this; the crew can change it.

Find what fills the check machine disk (follow-up to #6945) #6948

Parked opened this
Closed without merging

This pull doesn't find what filled the disk: my sandbox can't read production's data directory. What it does is make /settings/capacity show what is filling the disk. It also caps the only data-directory caches I found that the code never prunes, and fixes a sweep that was deleting the checks' kept files.

Notes

stopped parked by the crew

opened this

This pull doesn't find what filled the disk: my sandbox can't read production's data directory. What it does is make /settings/capacity show what is filling the disk. It also caps the only data-directory caches I found that the code never prunes, and fixes a sweep that was deleting the checks' kept files. Whether disk-red stays off the degraded list for a day can only be judged in production once this lands, so the issue should stay open until then.

What changed

  • The Disk card (src/kernel/disk_use.ts): /settings/capacity now opens with a Disk card. It shows free space out of the total, then the biggest entries under FORGE_DATA, biggest first, with the three biggest things inside each (for example crew/.cache).
    • It counts like du: blocks on disk, a hard-linked file once, no symlinks, no other mounted filesystems.
    • The walk runs in the background and its result is kept for 30 minutes, so the page never waits for it.
    • listCapacity carries the reading, so GET /api/v1/admin/capacity (disk) and the MCP capacity tool name the same consumers. The how-to is in docs/features/disk-use.md.
  • Install-cache ceiling (src/kernel/cache_bound.ts): bun's cache keeps every package version it has ever fetched, and nothing pruned crew/.cache/<repo>/bun or check-install/<repo>/bun. Before an install, the cache is now measured at most once an hour and emptied completely if it is over 4 GiB. For checks this happens under the existing install lock. For the crew it is written in the run log.
  • Checkout sweep fix: sweepCheckouts treated every name under checks/ as a run id. So on every boot it deleted checks/gates, learned, impact and gate-last.json. It also deleted <run>.home for checks still running, which matters more once #6946 runs the sweep before each check. It now skips those four names and ties a .home to its run. Any other leftover under checks/ is still removed, so the existing test in test/checks.test.ts passes unchanged.

Left out on purpose

  • I didn't prune run logs, looks, backups or snapshots. Run logs are never pruned in the code, but deleting run history is a product decision, so read the Disk card on production first, then bound whatever it shows at the top in a follow-up.
  • The 4 GiB ceiling is a guess. A workdir's node_modules here is only 80 MB, so the install caches may well not be the main grower.
  • Emptying crew/.cache while another run on the same repository is installing can fail that one install. The run starts anyway, as it already does after a failed install.

Tests: new tests in test/disk-use.test.ts, test/cache-bound.test.ts and test/checks-sweep-kept.test.ts; the sweep test fails on main. The type check, these, test/checks.test.ts, the capacity and MCP-doc tests, and the five cross files pass. I didn't run the full gate; the forge runs it on the push.

To try it: open /settings/capacity on the preview as an admin. The first load says "Measuring"; reload after a minute to see the list. Or run bun test test/disk-use.test.ts test/cache-bound.test.ts test/checks-sweep-kept.test.ts.

🤖 Generated with Claude Code

Closes #6947

commented

This pull doesn't find what filled the disk: my sandbox can't read production's data directory. What it does is make /settings/capacity show what is filling the disk. It also caps the only data-directory caches I found that the code never prunes, and fixes a sweep that was deleting the checks' kept files. Whether disk-red stays off the degraded list for a day can only be judged in production once this lands, so the issue should stay open until then.

What changed

  • The Disk card (src/kernel/disk_use.ts): /settings/capacity now opens with a Disk card. It shows free space out of the total, then the biggest entries under FORGE_DATA, biggest first, with the three biggest things inside each (for example crew/.cache).
    • It counts like du: blocks on disk, a hard-linked file once, no symlinks, no other mounted filesystems.
    • The walk runs in the background and its result is kept for 30 minutes, so the page never waits for it.
    • listCapacity carries the reading, so GET /api/v1/admin/capacity (disk) and the MCP capacity tool name the same consumers. The how-to is in docs/features/disk-use.md.
  • Install-cache ceiling (src/kernel/cache_bound.ts): bun's cache keeps every package version it has ever fetched, and nothing pruned crew/.cache/<repo>/bun or check-install/<repo>/bun. Before an install, the cache is now measured at most once an hour and emptied completely if it is over 4 GiB. For checks this happens under the existing install lock. For the crew it is written in the run log.
  • Checkout sweep fix: sweepCheckouts treated every name under checks/ as a run id. So on every boot it deleted checks/gates, learned, impact and gate-last.json. It also deleted <run>.home for checks still running, which matters more once #6946 runs the sweep before each check. It now skips those four names and ties a .home to its run. Any other leftover under checks/ is still removed, so the existing test in test/checks.test.ts passes unchanged.

Left out on purpose

  • I didn't prune run logs, looks, backups or snapshots. Run logs are never pruned in the code, but deleting run history is a product decision, so read the Disk card on production first, then bound whatever it shows at the top in a follow-up.
  • The 4 GiB ceiling is a guess. A workdir's node_modules here is only 80 MB, so the install caches may well not be the main grower.
  • Emptying crew/.cache while another run on the same repository is installing can fail that one install. The run starts anyway, as it already does after a failed install.

Tests: new tests in test/disk-use.test.ts, test/cache-bound.test.ts and test/checks-sweep-kept.test.ts; the sweep test fails on main. The type check, these, test/checks.test.ts, the capacity and MCP-doc tests, and the five cross files pass. I didn't run the full gate; the forge runs it on the push.

To try it: open /settings/capacity on the preview as an admin. The first load says "Measuring"; reload after a minute to see the list. Or run bun test test/disk-use.test.ts test/cache-bound.test.ts test/checks-sweep-kept.test.ts.

🤖 Generated with Claude Code

commented

Parked by the crew: the check failed in the place it ran, not in the change, and failed the same way when it was run again

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs