A data volume that is filling up
The disk-free maintenance duty (#13320) files one issue when the volume under
FORGE_APP_DATA has under 15% free, so the forge says it is filling up days before it is full.
- What it watches. Free space as statfs reports it (what an unprivileged writer can
still use) against the volume's size, read once per hourly tick. A volume that cannot be read (the data directory is missing) yields no finding and does not fail the run.
- The line. Under 15% free files; exactly 15% or more is quiet.
- What the issue says. "The data volume has under 15% free": the percent, the free and
total size, and the three largest directories directly under the data directory, sized with fs only (symlinks not followed, no process started).
- One issue. While one is open, later ticks file none (
fileIssueOnce). - Closing it. Closed while the disk is still under the line, the next tick files a new
one: the duty remembers nothing but the open issue.
- Where it shows. One row,
disk-free, on the maintenance board (/maintenance, admin). - The duty runs hourly and spends no money.
What gives space back (#13514)
The hourly clone sweep (src/crew/dispatch.ts) also removes node_modules from a clone kept for
an open issue or pull that nothing has touched for a day and no live run holds
(sweepIdleDependencies, src/crew/orphans.ts). It is about two thirds of a clone; sources and
.git stay, and installDependencies rebuilds it from the shared cache when a run resumes.
Run logs are kept a week (#13569)
runs/ held 15 GiB at the last count. pruneRunFiles (src/runs/prune.ts) now removes a run
log or vitals file seven days after its last write, not fourteen; the run pages already say "no
log" for a missing one.
Run logs have a size cap (#13576)
pruneRunFiles also keeps runs/ under 8 GiB (RUN_FILES_MAX_BYTES): after the age sweep, the
oldest remaining files go until the total fits.
A dormant preview gives back its checkouts (#13631)
A preview whose pull went quiet is stopped (dormant) and waking it deploys again with a fresh
checkout and install. dropCheckouts (src/apps/dormant_checkout.ts) removes its checkout
directories (<env>/<sha7>, node_modules and all) as it goes dormant, in the idle sweep and at
boot. Its data and home stay.
Its build cache goes with the checkouts (#13915). dropPreviewBuildCache
(src/apps/build_cache.ts) removes the environment's build-cache directory when it is dormant
and no deploy is running or queued, so a wake builds from scratch. Production's build cache is
never dropped. The size of what this gives back is not measured here: the cache is a cargo
target directory per preview, and the volume's own count shows whether it was enough.
A clone idle for a day goes whole (#13734, #13843, #14007)
sweepStaleClones (src/crew/orphans.ts) removes a kept clone, sources and .git, whose .git
nothing has touched for a day (CLONE_IDLE_MS; it was seven, then three; about 150 clones of ~150 MB were holding 22.8 GiB) and no live run holds. The window is counted from .git alone, so
trimming node_modules at one day does not restart it. The remote is truth: a run fetches and
resets to it, and clones afresh when the directory is gone.
A clone is kept, and counted in the log, when git status --porcelain is not empty or a local
branch has commits no remote ref has (holdsUnpushedWork; fixed argv, GIT_OPTIONAL_LOCKS=0 so the
check does not stamp the clone). Git that cannot answer also keeps it.
Checkouts nothing serves go at boot (#13772)
At boot, reconcileApp calls dropStrayCheckouts (src/apps/runtime.ts): in each environment on
disk, a checkout directory that is not the commit serving, resting, deploying or queued goes, and
so does what a failed or interrupted deploy left. One touched within STRAY_CHECKOUT_AGE_MS (ten minutes, src/apps/dormant_checkout.ts) stays.
This is a partial step on #13735: it gives back what strays hold but does not by itself bring free space
above 15%. test/stray-checkouts.test.ts proves the kept commits are in state when the sweep runs.