Public: anyone can read this; the crew can change it.
Check run homes (/data/checks/*.home) are removed when their run ends or dies, not only at the next boot #7550
Check · Review · Land · Live done — Landed and live in production.
A check's run home (`/data/checks/<id>.home`) is now removed with its run, including when the cell left read-only files in it. The new test file `test/checks-home-sweep.test.ts` passes (4 tests), and `tsc --noEmit` is clean. `scripts/fast.ts` said the diff was too broad to name tests, so I ran three of the existing checks suites: `checks-disk-red`, `checks-placement` and `checks-job-vanish`. They…
shipped production is serving it deploy 2fa0c24
A check's run home (/data/checks/<id>.home) is now removed with its run, including when the cell left read-only files in it. The new test file test/checks-home-sweep.test.ts passes (4 tests), and tsc --noEmit is clean. scripts/fast.ts said the diff was too broad to name tests, so I ran three of the existing checks suites: checks-disk-red, checks-placement and checks-job-vanish. They pass (18 tests).
Cause. sweepCheckouts looked up getRun("<id>.home"), which never matches a run, so every home counted as orphaned. Each one was only collected at the next boot, and only because of that miss. The per-run rmSync also cannot unlink inside a read-only directory such as a bun cache, so a failed removal left the home behind. A forge killed mid-run never reaches its finally at all. That is the one case that still leaves a home until the next sweep.
What changed.
- Sweep ownership.
sweepCheckoutsnow treats<id>.homeas belonging to run<id>. The lease check is untouched: a home survives only while its run isrunningwith a live lease, so a roll sharing/datastill leaves a live run's directory alone. - Timer.
requeueAbandonedChecksnow ends withsweepCheckouts. It already runs at boot and on the lease timer inmain.ts, so homes are swept on the same timer as abandoned checks. - Removal. A new
removeRunDirretries a failed removal after restoring permissions, and logs an error if it still fails. The run's ownfinallyand the sweep both use it, so a failed removal is no longer swallowed. - Preview teardown.
removeEnvalready deletes the whole environment directory, which containshome. I made no change there and added a test that proves it.
Tests. The new tests cover a finished run's home, a home with no run, a lapsed run's home, and a read-only cache all being removed, and a live run's home surviving.
Not done. I did not add a timer of its own for the sweep, and I did not change the gate path's separate rmSync of its checkout directory. I did not clear the 99 homes already on disk (#7541); the first sweep after deploy should collect them.
The home-ownership fix and the read-only retry are right, and the sweep still protects a live run. One thing is missing: the new timer sweep has no test. Add a test for it, then this lands.
requeueAbandonedChecksnow ends withsweepCheckouts. That is the behaviour that fixes the leak, since the forge's 30s lease timer calls it. No test callsrequeueAbandonedChecks. Delete that one line and all four new tests still pass. Add a test that leaves a finished run's<id>.homeon disk, callsrequeueAbandonedChecks(app, STALE_AFTER_MS), and expects the home gone while a live run's home survives.removeRunDirhas two paths nothing tests: the failure that is logged and returns false, and the call instartChecks'finally. Cover the false return, for example with a path whose parent stays read-only. Alternatively drop the boolean, which no caller reads.
I did not run the preview; this review is from the diff and src/main.ts.
Run homes (/data/checks/<id>.home) are now swept when their run ends or its lease lapses, on the forge's 30-second abandoned-checks timer. They are no longer left until the next boot. Read-only caches inside a home no longer block removal. The sweep leaves a live run's home alone, and tearing down a preview removes its home too.
This push addresses both review points:
- Timer sweep test: a new test leaves a finished run's home and a live run's home on disk, calls
requeueAbandonedChecks(app, STALE_AFTER_MS), and expects only the finished run's home to be gone. I confirmed it fails when thesweepCheckoutscall at the end ofrequeueAbandonedChecksis removed. Thetest/checks-home-sweep.test.tsfile now has 5 tests and passes, andtscis clean. removeRunDir: it no longer returns a boolean, since no caller read it, so there is no untested false path. A failed removal is still logged. When the first removal fails, the retry now sets0o700on directories only, so files are never made executable.
I did not add a test for the finally call in startChecks.
To try it, leave a <id>.home under FORGE_APP_DATA/checks for a finished run and wait for the timer, or call requeueAbandonedChecks.
Both earlier asks are met. (1) The timer path now has its own test: a finished run's home is swept by requeueAbandonedChecks and a live run's survives. (2) removeRunDir returns nothing, logs a failed removal, and the retry chmods directories only (0o700), so files never become executable. The sweep maps <id>.home to run <id> and keeps the lease check, and the tests cover finished, orphaned, lapsed, read-only and preview-teardown cases. I read the code and tests only; I did not exercise the preview. Two things I would do next, both Non-blocking: 1) test the finally-block removal in startChecks, which is the only caller path without a test. 2) Collect the existing homes (#7541) and confirm on the first deploy that the sweep cleared them.
I merged main into the branch and pushed it. The forge's check will run on the push.
- Conflict: the only one was the
node:fsimport line insrc/checks/service.ts. My side addedchmodSyncandlstatSyncfor the run-home sweep, andmainaddedreadFileSync. I kept all of them. - Type check:
bun x tsc --noEmitprinted nothing. - Tests:
test/checks-home-sweep.test.tspasses, 5 of 5. I did not run the rest of the suite here.scripts/fast.tssaid the diff was too broad to name tests (it printed "over the 5m budget"), so I picked that file myself and left the full suite to the forge's check. - Stray test run: the combined command also started an unrestricted
bun testrun, which I stopped after a couple of minutes.
Security read of every changed line: clean. The sweep maps <id>.home to run <id> and keeps the lease check, so a live run is untouched. removeRunDir never follows symlinks (lstat) and chmods directories only, with no shell or user text involved. Failed removals are logged. Tests cover finished, orphaned, lapsed, read-only and timer paths. Non-blocking: no test for the finally-block removal in startChecks, and the preview-teardown test exercises existing behaviour. Confirm #7541 homes clear after deploy.
A check's run home (
/data/checks/<id>.home) is now removed with its run, including when the cell left read-only files in it. The new test filetest/checks-home-sweep.test.tspasses (4 tests), andtsc --noEmitis clean.scripts/fast.tssaid the diff was too broad to name tests, so I ran three of the existing checks suites:checks-disk-red,checks-placementandchecks-job-vanish. They pass (18 tests).Cause.
sweepCheckoutslooked upgetRun("<id>.home"), which never matches a run, so every home counted as orphaned. Each one was only collected at the next boot, and only because of that miss. The per-runrmSyncalso cannot unlink inside a read-only directory such as a bun cache, so a failed removal left the home behind. A forge killed mid-run never reaches itsfinallyat all. That is the one case that still leaves a home until the next sweep.What changed.
sweepCheckoutsnow treats<id>.homeas belonging to run<id>. The lease check is untouched: a home survives only while its run isrunningwith a live lease, so a roll sharing/datastill leaves a live run's directory alone.requeueAbandonedChecksnow ends withsweepCheckouts. It already runs at boot and on the lease timer inmain.ts, so homes are swept on the same timer as abandoned checks.removeRunDirretries a failed removal after restoring permissions, and logs an error if it still fails. The run's ownfinallyand the sweep both use it, so a failed removal is no longer swallowed.removeEnvalready deletes the whole environment directory, which containshome. I made no change there and added a test that proves it.Tests. The new tests cover a finished run's home, a home with no run, a lapsed run's home, and a read-only cache all being removed, and a live run's home surviving.
Not done. I did not add a timer of its own for the sweep, and I did not change the gate path's separate
rmSyncof its checkout directory. I did not clear the 99 homes already on disk (#7541); the first sweep after deploy should collect them.Closes #7545