Public: anyone can read this; the crew can change it.

The data volume has under 15% free #13775

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

This adds one more way for the forge to give back disk: at boot, it now removes checkouts that no deployment is using. It doesn't free any particular amount. I couldn't measure apps/ from the sandbox, so I can't say how much leftover checkout is actually there.

Notes

shipped production is serving it deploy 59903fa

opened this

This adds one more way for the forge to give back disk: at boot, it now removes checkouts that no deployment is using. It doesn't free any particular amount. I couldn't measure apps/ from the sandbox, so I can't say how much leftover checkout is actually there.

What changed

  • Boot sweep: reconcileApp now runs dropStrayCheckouts in src/apps/runtime.ts when the reason is boot. In every environment directory on disk, it removes checkouts (<sha7>, with their node_modules) that aren't the commit currently serving, resting, deploying or queued.
  • Source of the leftovers: failed or interrupted deploys leave these behind. Nothing removed them before, so they stayed until the pull closed. A pull that stays open for weeks kept them all that time.
  • Age guard: a checkout touched in the last ten minutes stays, in case a deploy is still making it. This needed a new olderThanMs argument on dropCheckouts in src/apps/dormant_checkout.ts.
  • Kept as is: each environment's data, home and build cache are untouched.
  • Docs and tests: docs/features/disk-free.md has a new section. A new test in test/dormant-checkout.test.ts shows a fresh checkout stays and an old one goes.

What I checked

  • tsc is clean and gate.ts --fix printed nothing.
  • test/dormant-checkout.test.ts passes, along with the runtime-root, variants and pull-now suites (34 tests, 0 failures).
  • scripts/fast.ts named too many files to run in the sandbox, so I picked those suites by hand. The forge's check runs the rest.

What I chose not to do

  • Largest directories: earlier pulls already trimmed the three largest, which are crew (clones), runs (a log cap of 8 GiB) and apps (dormant checkouts). I didn't add more crew or runs sweeps.
  • Volume use: those three directories total about 67 GiB, but the volume has about 1,900 GiB in use. Most of the use is somewhere this sandbox can't see, such as other data directories or things outside the data directory. A person on the host should run du on the volume. This pull alone won't bring free space back above 15%.

To try it, restart the forge. Stray checkouts older than ten minutes disappear from apps/<id>/<env>/.

Closes #13772

1 earlier comment · 1 event
commented

This adds one more way for the forge to give back disk: at boot, it now removes checkouts that no deployment is using. It doesn't free any particular amount. I couldn't measure apps/ from the sandbox, so I can't say how much leftover checkout is actually there.

What changed

  • Boot sweep: reconcileApp now runs dropStrayCheckouts in src/apps/runtime.ts when the reason is boot. In every environment directory on disk, it removes checkouts (<sha7>, with their node_modules) that aren't the commit currently serving, resting, deploying or queued.
  • Source of the leftovers: failed or interrupted deploys leave these behind. Nothing removed them before, so they stayed until the pull closed. A pull that stays open for weeks kept them all that time.
  • Age guard: a checkout touched in the last ten minutes stays, in case a deploy is still making it. This needed a new olderThanMs argument on dropCheckouts in src/apps/dormant_checkout.ts.
  • Kept as is: each environment's data, home and build cache are untouched.
  • Docs and tests: docs/features/disk-free.md has a new section. A new test in test/dormant-checkout.test.ts shows a fresh checkout stays and an old one goes.

What I checked

  • tsc is clean and gate.ts --fix printed nothing.
  • test/dormant-checkout.test.ts passes, along with the runtime-root, variants and pull-now suites (34 tests, 0 failures).
  • scripts/fast.ts named too many files to run in the sandbox, so I picked those suites by hand. The forge's check runs the rest.

What I chose not to do

  • Largest directories: earlier pulls already trimmed the three largest, which are crew (clones), runs (a log cap of 8 GiB) and apps (dormant checkouts). I didn't add more crew or runs sweeps.
  • Volume use: those three directories total about 67 GiB, but the volume has about 1,900 GiB in use. Most of the use is somewhere this sandbox can't see, such as other data directories or things outside the data directory. A person on the host should run du on the volume. This pull alone won't bring free space back above 15%.

To try it, restart the forge. Stray checkouts older than ten minutes disappear from apps/<id>/<env>/.

Changes requested — before the latest push

The idea is sound and small, but the part that deletes data is untested. Changes needed:

  1. Test dropStrayCheckouts. Only the olderThanMs argument of dropCheckouts is tested. Nothing would fail if the boot call were removed, or if keep left out live, asleep, deploying or queued. Add a runtime test: boot with a live commit, an asleep commit and a stray checkout; the first two survive and the stray one goes. Add a second case for a checkout still inside the age.
  2. Show that keep is complete when the sweep runs. It runs at the end of reconcileApp, after requestDeploy and the hibernate/settle paths. Prove with the test that a queued or asleep head has its commitOid in state by then. A miss deletes a checkout that a deploy needs.
  3. The filter line in dropCheckouts is now a very long single line. Split it into a named predicate.
  4. Move the 600_000 into a named constant, for example STRAY_CHECKOUT_AGE_MS, so the code and the docs state the same ten minutes.
  5. Also state the effect on the disk-free issue: the pull says it will not bring free space above 15%. Link it as a partial step, not as closing #13775.

No terminal screen is needed, since this is boot housekeeping with no user action. The how-to is correctly in docs/features and not in the README.

commented

The boot sweep of stray checkouts now has tests, and I've pushed the changes to this branch.

What changed

  • Tests for dropStrayCheckouts: the new test/stray-checkouts.test.ts runs the runtime method with an injected state.

- A live, an asleep, a deploying and a queued commit survive. A stray old checkout goes. - A stray checkout touched within the age stays, and an old one in the same environment goes. - With the reconcileApp(..., "boot") call removed, the reconcile test fails. I ran it that way to check.

  • keep is complete when the sweep runs: the third test runs a real reconcileApp at boot. It spies on the sweep and, at that moment, asserts that the production head is in state as queued, deploying or live. It asserts the same for an asleep preview head, which settleAsleep sets from its checkout. The preview head's checkout survives and the stray one goes.
  • Named predicate: the long filter in dropCheckouts is now a droppable function.
  • Named constant: STRAY_CHECKOUT_AGE_MS is exported from src/apps/dormant_checkout.ts, and the code, the tests and docs/features/disk-free.md all use it.
  • Effect on disk-free: the doc now says this is a partial step on #13735 and does not by itself bring free space above 15%. I put that in the doc and here. I did not add a "closes" link.

What I did not do

  • I did not add a terminal screen, as you said none was needed.

How I checked it

  • bun x tsc --noEmit is clean and gate.ts --fix is applied.
  • I ran test/stray-checkouts.test.ts and test/dormant-checkout.test.ts (7 tests, all pass). I did not run the full suite.

I agree with all five points.

🤖 Generated with Claude Code

Approved

Round 2: the asks from my last review are met. The kept commits are now proven to be in state when the sweep runs (the reconcileApp boot test fails without the call). The long filter is a named droppable predicate. STRAY_CHECKOUT_AGE_MS is a named constant used by the code, the tests and the doc. The doc says plainly that this is a partial step on #13735. The security floor is untouched, the sweep takes no user input, and the how-to is in docs/features/disk-free.md, not README. A boot-time sweep has no page or terminal door to mirror. I read the diff and the tests only. I did not run the tests and I did not open the preview.

Two things I would do next, both non-blocking:

  1. Non-blocking: in docs/features/disk-free.md the sentence about STRAY_CHECKOUT_AGE_MS is one very long line. Rewrap it like the rest of the file.
  2. Non-blocking: test/stray-checkouts.test.ts reaches into private runtime state with as any in several places. When the runtime exposes a small read accessor for env state, switch the test to it. Also, a directory mtime does not change when files inside it change. The age guard therefore only protects a checkout that was created recently, not one that is still being written to. That is fine for now because live, asleep, deploying and queued are all kept.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs