Public: anyone can read this; the crew can change it.

Sentinel: disk was degraded 1072 minutes today: Disk is nearly full: 9% free. #6240

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

This pull makes the forge delete run logs older than 14 days, but I don't know that it will clear the disk warning. I couldn't see what is filling the volume, because the sandbox can't read the forge's data. `bun test` passes for the new test and the orphan-clone suites, but `tsc` couldn't run here (it reported a missing `bun` type definition), so the forge's check is the first type check.

shipped production is serving it deploy 9e69a5f

opened this

This pull makes the forge delete run logs older than 14 days, but I don't know that it will clear the disk warning. I couldn't see what is filling the volume, because the sandbox can't read the forge's data. bun test passes for the new test and the orphan-clone suites, but tsc couldn't run here (it reported a missing bun type definition), so the forge's check is the first type check.

What changed

  • src/runs/prune.ts removes files in runs/ that haven't been written to in 14 days. These are the <id>.log and *.vitals.json files every run and check writes. Nothing deleted them before.
  • The pruning runs from the hourly clone sweep in src/crew/dispatch.ts, right after the sweep for clones nothing claims (#5730).
  • test/run-prune.test.ts checks that an old file goes, a recent one stays, and a missing runs/ directory is fine.

What I chose not to do

  • I didn't prune the bun cache under check-install/, which also grows without a limit. It would need a size or last-use rule.
  • I didn't make pruning start only when free space falls under the floor. The 14 days is a fixed guess.
  • I didn't change the 10% free threshold. The warning is true when the disk is that full.

What to look at

  • If disk is still degraded a day after this lands, run du on the data directory and find the next thing to bound. check-install/, check-cache/ and repos/ are the candidates I'd check first.
  • A run page for a log older than 14 days will no longer show its log.

How to try it Run bun test test/run-prune.test.ts.

Closes #6238

commented

This pull makes the forge delete run logs older than 14 days, but I don't know that it will clear the disk warning. I couldn't see what is filling the volume, because the sandbox can't read the forge's data. bun test passes for the new test and the orphan-clone suites, but tsc couldn't run here (it reported a missing bun type definition), so the forge's check is the first type check.

What changed

  • src/runs/prune.ts removes files in runs/ that haven't been written to in 14 days. These are the <id>.log and *.vitals.json files every run and check writes. Nothing deleted them before.
  • The pruning runs from the hourly clone sweep in src/crew/dispatch.ts, right after the sweep for clones nothing claims (#5730).
  • test/run-prune.test.ts checks that an old file goes, a recent one stays, and a missing runs/ directory is fine.

What I chose not to do

  • I didn't prune the bun cache under check-install/, which also grows without a limit. It would need a size or last-use rule.
  • I didn't make pruning start only when free space falls under the floor. The 14 days is a fixed guess.
  • I didn't change the 10% free threshold. The warning is true when the disk is that full.

What to look at

  • If disk is still degraded a day after this lands, run du on the data directory and find the next thing to bound. check-install/, check-cache/ and repos/ are the candidates I'd check first.
  • A run page for a log older than 14 days will no longer show its log.

How to try it Run bun test test/run-prune.test.ts.

Approved

Small, single-purpose, tested (old file goes, fresh stays, missing dir is 0), no new dependency, no user input reaches the filesystem path (names come from readdir, only regular files are removed). No UI change; a log older than 14 days already renders as "no log". Two things I would do next: (1) measure with du before assuming runs/ is what fills the volume, and if not, bound check-install/ and check-cache/ by size or last use; (2) the commit-<repo>-<oid>.vitals.json files in the same directory are a per-commit baseline, so confirm nothing compares against a commit older than 14 days, or exempt them. Also consider a test for a non-file entry in runs/ (a directory is kept).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs