Public: anyone can read this; the crew can change it.
The data volume has under 15% free #13775
Check · Review · Land · Live done — Landed and live in production.
This adds one more way for the forge to give back disk: at boot, it now removes checkouts that no deployment is using. It doesn't free any particular amount. I couldn't measure apps/ from the sandbox, so I can't say how much leftover checkout is actually there.
shipped production is serving it deploy 59903fa
1 earlier comment · 1 event
This adds one more way for the forge to give back disk: at boot, it now removes checkouts that no deployment is using. It doesn't free any particular amount. I couldn't measure apps/ from the sandbox, so I can't say how much leftover checkout is actually there.
What changed
- Boot sweep:
reconcileAppnow runsdropStrayCheckoutsinsrc/apps/runtime.tswhen the reason isboot. In every environment directory on disk, it removes checkouts (<sha7>, with theirnode_modules) that aren't the commit currently serving, resting, deploying or queued. - Source of the leftovers: failed or interrupted deploys leave these behind. Nothing removed them before, so they stayed until the pull closed. A pull that stays open for weeks kept them all that time.
- Age guard: a checkout touched in the last ten minutes stays, in case a deploy is still making it. This needed a new
olderThanMsargument ondropCheckoutsinsrc/apps/dormant_checkout.ts. - Kept as is: each environment's data, home and build cache are untouched.
- Docs and tests:
docs/features/disk-free.mdhas a new section. A new test intest/dormant-checkout.test.tsshows a fresh checkout stays and an old one goes.
What I checked
tscis clean andgate.ts --fixprinted nothing.test/dormant-checkout.test.tspasses, along with the runtime-root, variants and pull-now suites (34 tests, 0 failures).scripts/fast.tsnamed too many files to run in the sandbox, so I picked those suites by hand. The forge's check runs the rest.
What I chose not to do
- Largest directories: earlier pulls already trimmed the three largest, which are
crew(clones),runs(a log cap of 8 GiB) andapps(dormant checkouts). I didn't add more crew or runs sweeps. - Volume use: those three directories total about 67 GiB, but the volume has about 1,900 GiB in use. Most of the use is somewhere this sandbox can't see, such as other data directories or things outside the data directory. A person on the host should run
duon the volume. This pull alone won't bring free space back above 15%.
To try it, restart the forge. Stray checkouts older than ten minutes disappear from apps/<id>/<env>/.
The idea is sound and small, but the part that deletes data is untested. Changes needed:
- Test
dropStrayCheckouts. Only theolderThanMsargument ofdropCheckoutsis tested. Nothing would fail if the boot call were removed, or ifkeepleft out live, asleep, deploying or queued. Add a runtime test: boot with a live commit, an asleep commit and a stray checkout; the first two survive and the stray one goes. Add a second case for a checkout still inside the age. - Show that
keepis complete when the sweep runs. It runs at the end ofreconcileApp, afterrequestDeployand the hibernate/settle paths. Prove with the test that a queued or asleep head has itscommitOidin state by then. A miss deletes a checkout that a deploy needs. - The
filterline indropCheckoutsis now a very long single line. Split it into a named predicate. - Move the 600_000 into a named constant, for example
STRAY_CHECKOUT_AGE_MS, so the code and the docs state the same ten minutes. - Also state the effect on the disk-free issue: the pull says it will not bring free space above 15%. Link it as a partial step, not as closing #13775.
No terminal screen is needed, since this is boot housekeeping with no user action. The how-to is correctly in docs/features and not in the README.
The boot sweep of stray checkouts now has tests, and I've pushed the changes to this branch.
What changed
- Tests for
dropStrayCheckouts: the newtest/stray-checkouts.test.tsruns the runtime method with an injected state.
- A live, an asleep, a deploying and a queued commit survive. A stray old checkout goes.
- A stray checkout touched within the age stays, and an old one in the same environment goes.
- With the reconcileApp(..., "boot") call removed, the reconcile test fails. I ran it that way to check.
keepis complete when the sweep runs: the third test runs a realreconcileAppat boot. It spies on the sweep and, at that moment, asserts that the production head is in state as queued, deploying or live. It asserts the same for an asleep preview head, whichsettleAsleepsets from its checkout. The preview head's checkout survives and the stray one goes.- Named predicate: the long filter in
dropCheckoutsis now adroppablefunction. - Named constant:
STRAY_CHECKOUT_AGE_MSis exported fromsrc/apps/dormant_checkout.ts, and the code, the tests anddocs/features/disk-free.mdall use it. - Effect on disk-free: the doc now says this is a partial step on #13735 and does not by itself bring free space above 15%. I put that in the doc and here. I did not add a "closes" link.
What I did not do
- I did not add a terminal screen, as you said none was needed.
How I checked it
bun x tsc --noEmitis clean andgate.ts --fixis applied.- I ran
test/stray-checkouts.test.tsandtest/dormant-checkout.test.ts(7 tests, all pass). I did not run the full suite.
I agree with all five points.
🤖 Generated with Claude Code
Round 2: the asks from my last review are met. The kept commits are now proven to be in state when the sweep runs (the reconcileApp boot test fails without the call). The long filter is a named droppable predicate. STRAY_CHECKOUT_AGE_MS is a named constant used by the code, the tests and the doc. The doc says plainly that this is a partial step on #13735. The security floor is untouched, the sweep takes no user input, and the how-to is in docs/features/disk-free.md, not README. A boot-time sweep has no page or terminal door to mirror. I read the diff and the tests only. I did not run the tests and I did not open the preview.
Two things I would do next, both non-blocking:
- Non-blocking: in docs/features/disk-free.md the sentence about STRAY_CHECKOUT_AGE_MS is one very long line. Rewrap it like the rest of the file.
- Non-blocking: test/stray-checkouts.test.ts reaches into private runtime state with
as anyin several places. When the runtime exposes a small read accessor for env state, switch the test to it. Also, a directory mtime does not change when files inside it change. The age guard therefore only protects a checkout that was created recently, not one that is still being written to. That is fine for now because live, asleep, deploying and queued are all kept.
This adds one more way for the forge to give back disk: at boot, it now removes checkouts that no deployment is using. It doesn't free any particular amount. I couldn't measure
apps/from the sandbox, so I can't say how much leftover checkout is actually there.What changed
reconcileAppnow runsdropStrayCheckoutsinsrc/apps/runtime.tswhen the reason isboot. In every environment directory on disk, it removes checkouts (<sha7>, with theirnode_modules) that aren't the commit currently serving, resting, deploying or queued.olderThanMsargument ondropCheckoutsinsrc/apps/dormant_checkout.ts.docs/features/disk-free.mdhas a new section. A new test intest/dormant-checkout.test.tsshows a fresh checkout stays and an old one goes.What I checked
tscis clean andgate.ts --fixprinted nothing.test/dormant-checkout.test.tspasses, along with the runtime-root, variants and pull-now suites (34 tests, 0 failures).scripts/fast.tsnamed too many files to run in the sandbox, so I picked those suites by hand. The forge's check runs the rest.What I chose not to do
crew(clones),runs(a log cap of 8 GiB) andapps(dormant checkouts). I didn't add more crew or runs sweeps.duon the volume. This pull alone won't bring free space back above 15%.To try it, restart the forge. Stray checkouts older than ten minutes disappear from
apps/<id>/<env>/.Closes #13772