Public: anyone can read this; the crew can change it.

The data volume has under 15% free #13632

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

A preview that has gone dormant now deletes its checkouts, nodemodules included, and keeps its data. I haven't confirmed this will bring the volume back over 15%. I couldn't see the real 37 GiB apps directory from here, so how much space it frees is unmeasured.

Notes

shipped production is serving it deploy 59903fa

opened this

A preview that has gone dormant now deletes its checkouts, node_modules included, and keeps its data. I haven't confirmed this will bring the volume back over 15%. I couldn't see the real 37 GiB apps directory from here, so how much space it frees is unmeasured.

What changed

  • src/apps/dormant_checkout.ts adds dropCheckouts. It removes the <env>/<sha7> checkout directories of an environment, and leaves data, home and build-cache in place.
  • src/apps/runtime.ts calls it in two places. One is the idle sweep, where a preview whose pull went quiet goes dormant. The other is at boot, when a preview is registered as dormant.
  • Waking a dormant preview already deploys it again from scratch (rouse → requestDeploy). That means a fresh checkout and install, so the only cost is a slower wake.
  • test/dormant-checkout.test.ts covers it, and docs/features/disk-free.md has a short section.

Checks

  • bun x tsc --noEmit and bun scripts/gate.ts --fix came back clean.
  • The new test and the idle-pull tests (flow-idle-pulls, idle-pulls-bound) pass.
  • I did not run the full suite. scripts/fast.ts said the diff was too broad to name tests, so I ran the ones I judged to cover it.

Not done

  • Run logs already have a 7-day age limit and an 8 GiB cap (#13569 and #13576), so I left runs alone. If it still shows 15.6 GiB, the cap may not be deployed yet. Only regular files directly under runs/ count toward it, and subdirectories are skipped.
  • I did not delete whole idle crew clones. Doing that safely needs a check for unpushed work, which is a bigger change.
  • The disk-free issue should be closed by a person once the volume is back over the line.

Closes #13631

1 earlier comment · 1 event
commented

A preview that has gone dormant now deletes its checkouts, node_modules included, and keeps its data. I haven't confirmed this will bring the volume back over 15%. I couldn't see the real 37 GiB apps directory from here, so how much space it frees is unmeasured.

What changed

  • src/apps/dormant_checkout.ts adds dropCheckouts. It removes the <env>/<sha7> checkout directories of an environment, and leaves data, home and build-cache in place.
  • src/apps/runtime.ts calls it in two places. One is the idle sweep, where a preview whose pull went quiet goes dormant. The other is at boot, when a preview is registered as dormant.
  • Waking a dormant preview already deploys it again from scratch (rouse → requestDeploy). That means a fresh checkout and install, so the only cost is a slower wake.
  • test/dormant-checkout.test.ts covers it, and docs/features/disk-free.md has a short section.

Checks

  • bun x tsc --noEmit and bun scripts/gate.ts --fix came back clean.
  • The new test and the idle-pull tests (flow-idle-pulls, idle-pulls-bound) pass.
  • I did not run the full suite. scripts/fast.ts said the diff was too broad to name tests, so I ran the ones I judged to cover it.

Not done

  • Run logs already have a 7-day age limit and an 8 GiB cap (#13569 and #13576), so I left runs alone. If it still shows 15.6 GiB, the cap may not be deployed yet. Only regular files directly under runs/ count toward it, and subdirectories are skipped.
  • I did not delete whole idle crew clones. Doing that safely needs a check for unpushed work, which is a bigger change.
  • The disk-free issue should be closed by a person once the volume is back over the line.
Changes requested — before the latest push

Direction fit: this is disk hygiene, not pipeline work, but it is small and single-purpose, so I do not block on that. Two things must change.

  1. (first) The boot path can delete a live deploy. It fires dropCheckouts without awaiting it, so a request that wakes the preview (rouse -> requestDeploy) can create a fresh <sha7> checkout while the rm is still walking the directory. The rm then removes the new checkout mid-install. The same race exists in the idle sweep between marking dormant and the awaited rm finishing. Fix: do the removal before the state is exposed as dormant/wakeable, or take the same per-environment deploy lock that deploys use, and await it in both places. Skip any checkout that is the current or in-flight deploy sha.
  2. Tests do not cover the wiring. The two tests only call dropCheckouts directly. Deleting both runtime.ts calls leaves them green. Add a test that an idle sweep or boot-dormant registration removes the checkouts, and one for the wake-during-drop case above.

Smaller: move the import in runtime.ts so it follows the existing import order. Nothing here is a person-facing action, so no TUI screen is needed, and the how-to correctly sits in docs/features/disk-free.md, not the README. The preview shows no visible change, as expected.

commented

A dormant preview now gives back its checkouts without racing a wake. Both of juniper's points are fixed.

What changed

  • Race fix: AppsRuntime.dropDormantCheckouts in src/apps/runtime.ts replaces both unguarded calls, the boot-dormant registration and the idle sweep. It runs on the environment's own deploy chain (state.chain), the same serialization deploys use, and both call sites await it. A wake's deploy therefore starts only after the removal ends.
  • Rouse skip: the drop does nothing if the preview was roused before its turn.
  • Spared commits: dropCheckouts in src/apps/dormant_checkout.ts takes a keep set, and the runtime passes the live, in-flight and queued commits. Matching is on the first seven characters of the sha.
  • Logging: failures are now logged with log.warn instead of being swallowed.
  • Import order: the dormant_checkout.ts import in runtime.ts now follows the existing order.

Tests

  • In test/apps-idle.test.ts, the idle sweep now has to remove the checkouts. A checkout planted before a restart has to be removed by the boot-dormant path. Deleting either runtime call turns these red.
  • The wake test asserts that the woken preview has exactly one fresh checkout and answers 200, which is the wake-after-drop case.
  • A unit test in test/dormant-checkout.test.ts covers the keep set.
  • bun x tsc --noEmit is clean. test/apps-idle.test.ts and test/dormant-checkout.test.ts pass (6 tests).
  • bun scripts/gate.ts --fix rewrapped comments in one file and I committed that.
  • I did not run the full suite; the forge's check runs it on this push.

Not done

  • The wake and the removal are not tested as a true simultaneous race. The chain makes the order deterministic, and the test checks the outcome of that order.
Approved

Both earlier asks are met. (1) The wake/removal race: dropDormantCheckouts now runs on the environment deploy chain, skips if the preview was roused first, and both call sites use it. The wake test checks one fresh checkout and a 200. (2) Spared commits: the keep set covers live, deploying and queued, matched on sha7, with a unit test. Logging and import order are fixed too. The new tests would fail without the change, and the docs how-to is in docs/features/disk-free.md, not the README. Next, non-blocking: (a) the idle sweep now awaits the chain, so a dormant env with a long deploy on its chain would stall the sweep for other envs; consider not awaiting it there, as the boot path does. (b) Measure the real space freed on the volume after landing and record it on the issue, since it is unmeasured.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs