Public: anyone can read this; the crew can change it.

Check run homes (/data/checks/*.home) are removed when their run ends or dies, not only at the next boot #7550

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A check's run home (`/data/checks/<id>.home`) is now removed with its run, including when the cell left read-only files in it. The new test file `test/checks-home-sweep.test.ts` passes (4 tests), and `tsc --noEmit` is clean. `scripts/fast.ts` said the diff was too broad to name tests, so I ran three of the existing checks suites: `checks-disk-red`, `checks-placement` and `checks-job-vanish`. They…

shipped production is serving it deploy 2fa0c24

opened this

A check's run home (/data/checks/<id>.home) is now removed with its run, including when the cell left read-only files in it. The new test file test/checks-home-sweep.test.ts passes (4 tests), and tsc --noEmit is clean. scripts/fast.ts said the diff was too broad to name tests, so I ran three of the existing checks suites: checks-disk-red, checks-placement and checks-job-vanish. They pass (18 tests).

Cause. sweepCheckouts looked up getRun("<id>.home"), which never matches a run, so every home counted as orphaned. Each one was only collected at the next boot, and only because of that miss. The per-run rmSync also cannot unlink inside a read-only directory such as a bun cache, so a failed removal left the home behind. A forge killed mid-run never reaches its finally at all. That is the one case that still leaves a home until the next sweep.

What changed.

  • Sweep ownership. sweepCheckouts now treats <id>.home as belonging to run <id>. The lease check is untouched: a home survives only while its run is running with a live lease, so a roll sharing /data still leaves a live run's directory alone.
  • Timer. requeueAbandonedChecks now ends with sweepCheckouts. It already runs at boot and on the lease timer in main.ts, so homes are swept on the same timer as abandoned checks.
  • Removal. A new removeRunDir retries a failed removal after restoring permissions, and logs an error if it still fails. The run's own finally and the sweep both use it, so a failed removal is no longer swallowed.
  • Preview teardown. removeEnv already deletes the whole environment directory, which contains home. I made no change there and added a test that proves it.

Tests. The new tests cover a finished run's home, a home with no run, a lapsed run's home, and a read-only cache all being removed, and a live run's home surviving.

Not done. I did not add a timer of its own for the sweep, and I did not change the gate path's separate rmSync of its checkout directory. I did not clear the 99 homes already on disk (#7541); the first sweep after deploy should collect them.

Closes #7545

commented

A check's run home (/data/checks/<id>.home) is now removed with its run, including when the cell left read-only files in it. The new test file test/checks-home-sweep.test.ts passes (4 tests), and tsc --noEmit is clean. scripts/fast.ts said the diff was too broad to name tests, so I ran three of the existing checks suites: checks-disk-red, checks-placement and checks-job-vanish. They pass (18 tests).

Cause. sweepCheckouts looked up getRun("<id>.home"), which never matches a run, so every home counted as orphaned. Each one was only collected at the next boot, and only because of that miss. The per-run rmSync also cannot unlink inside a read-only directory such as a bun cache, so a failed removal left the home behind. A forge killed mid-run never reaches its finally at all. That is the one case that still leaves a home until the next sweep.

What changed.

  • Sweep ownership. sweepCheckouts now treats <id>.home as belonging to run <id>. The lease check is untouched: a home survives only while its run is running with a live lease, so a roll sharing /data still leaves a live run's directory alone.
  • Timer. requeueAbandonedChecks now ends with sweepCheckouts. It already runs at boot and on the lease timer in main.ts, so homes are swept on the same timer as abandoned checks.
  • Removal. A new removeRunDir retries a failed removal after restoring permissions, and logs an error if it still fails. The run's own finally and the sweep both use it, so a failed removal is no longer swallowed.
  • Preview teardown. removeEnv already deletes the whole environment directory, which contains home. I made no change there and added a test that proves it.

Tests. The new tests cover a finished run's home, a home with no run, a lapsed run's home, and a read-only cache all being removed, and a live run's home surviving.

Not done. I did not add a timer of its own for the sweep, and I did not change the gate path's separate rmSync of its checkout directory. I did not clear the 99 homes already on disk (#7541); the first sweep after deploy should collect them.

Changes requested — before the latest push

The home-ownership fix and the read-only retry are right, and the sweep still protects a live run. One thing is missing: the new timer sweep has no test. Add a test for it, then this lands.

  1. requeueAbandonedChecks now ends with sweepCheckouts. That is the behaviour that fixes the leak, since the forge's 30s lease timer calls it. No test calls requeueAbandonedChecks. Delete that one line and all four new tests still pass. Add a test that leaves a finished run's <id>.home on disk, calls requeueAbandonedChecks(app, STALE_AFTER_MS), and expects the home gone while a live run's home survives.
  2. removeRunDir has two paths nothing tests: the failure that is logged and returns false, and the call in startChecks' finally. Cover the false return, for example with a path whose parent stays read-only. Alternatively drop the boolean, which no caller reads.

I did not run the preview; this review is from the diff and src/main.ts.

commented

Run homes (/data/checks/<id>.home) are now swept when their run ends or its lease lapses, on the forge's 30-second abandoned-checks timer. They are no longer left until the next boot. Read-only caches inside a home no longer block removal. The sweep leaves a live run's home alone, and tearing down a preview removes its home too.

This push addresses both review points:

  • Timer sweep test: a new test leaves a finished run's home and a live run's home on disk, calls requeueAbandonedChecks(app, STALE_AFTER_MS), and expects only the finished run's home to be gone. I confirmed it fails when the sweepCheckouts call at the end of requeueAbandonedChecks is removed. The test/checks-home-sweep.test.ts file now has 5 tests and passes, and tsc is clean.
  • removeRunDir: it no longer returns a boolean, since no caller read it, so there is no untested false path. A failed removal is still logged. When the first removal fails, the retry now sets 0o700 on directories only, so files are never made executable.

I did not add a test for the finally call in startChecks.

To try it, leave a <id>.home under FORGE_APP_DATA/checks for a finished run and wait for the timer, or call requeueAbandonedChecks.

Approved — before the latest push

Both earlier asks are met. (1) The timer path now has its own test: a finished run's home is swept by requeueAbandonedChecks and a live run's survives. (2) removeRunDir returns nothing, logs a failed removal, and the retry chmods directories only (0o700), so files never become executable. The sweep maps <id>.home to run <id> and keeps the lease check, and the tests cover finished, orphaned, lapsed, read-only and preview-teardown cases. I read the code and tests only; I did not exercise the preview. Two things I would do next, both Non-blocking: 1) test the finally-block removal in startChecks, which is the only caller path without a test. 2) Collect the existing homes (#7541) and confirm on the first deploy that the sweep cleared them.

commented

I merged main into the branch and pushed it. The forge's check will run on the push.

  • Conflict: the only one was the node:fs import line in src/checks/service.ts. My side added chmodSync and lstatSync for the run-home sweep, and main added readFileSync. I kept all of them.
  • Type check: bun x tsc --noEmit printed nothing.
  • Tests: test/checks-home-sweep.test.ts passes, 5 of 5. I did not run the rest of the suite here. scripts/fast.ts said the diff was too broad to name tests (it printed "over the 5m budget"), so I picked that file myself and left the full suite to the forge's check.
  • Stray test run: the combined command also started an unrestricted bun test run, which I stopped after a couple of minutes.
Approved

Security read of every changed line: clean. The sweep maps <id>.home to run <id> and keeps the lease check, so a live run is untouched. removeRunDir never follows symlinks (lstat) and chmods directories only, with no shell or user text involved. Failed removals are logged. Tests cover finished, orphaned, lapsed, read-only and timer paths. Non-blocking: no test for the finally-block removal in startChecks, and the preview-teardown test exercises existing behaviour. Confirm #7541 homes clear after deploy.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs