Public: anyone can read this; the crew can change it.

The release step prunes old release images itself: the newest 15, the live one and the known-good rollback are kept #11845

opened this

Check · Review · Land · Live — Its check is queued. Next: reviewer, pushed since the last review.

Ready to land

After a release is verified live, the keeper now prunes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the `release.verified_live` event. This replaces the `forge-prune-images` host timer on pg-grand.

11 files · +375 −13

working queued to check deploy 32409d0

Check queued 40mCheck of 32409d0fast tier: queued · full gate runs on land

Website spec: 23 findings (report only)

Look

9 pages beside main · 0 new to this pull

/apps

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/explore

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/feed

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/front

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/map

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/notes

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/releases

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/repos

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

/review

main

at 3f03250 ·

this pull

In the dark

main

at 3f03250 ·

this pull

at daad645 · · the head has moved since; a new picture follows its deploy

opened this

After a release is verified live, the keeper now prunes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the release.verified_live event. This replaces the forge-prune-images host timer on pg-grand.

What changed

  • The rule is toPrune in src/keeper/prune.ts. It takes the stored releases with their ages and the ids that must stay, and returns the ids to delete, oldest first. It does no file or registry work.
  • The Deployer contract (src/keeper/deploy.ts) has a new optional store with three methods: list, idOf(ref) and remove(id). The keeper uses idOf to match the live ref, the known-good ref and image() to ids in the store.
  • The swapper's store (src/keeper/swapper.ts) is the trees under releases/<sha>/, dated by their mtime. It skips anything that isn't a 40-hex name, such as good/ and .partial folders. It refuses to remove the commit it is running or the one it handed over to, whatever it is asked.
  • The keeper (src/keeper/keeper.ts) prunes after the release is marked verified and before it sends the event. The event carries pruned: string[]. If the store refuses partway, the event also carries pruneError and lists what was already removed. The release stays verified either way. If the live ref has no id in the store, nothing is pruned.
  • Footing (src/system/footing.ts): the host-timers row now names only forge-watchdog and forge-github-mirror. A new release-prune row (find: self) puts pruning on the vxrail. The docs/OPERATING.md table is regenerated from footingTable(), its prose is updated, and docs/features/release-keeper.md gains a step 5.

What I chose not to do

  • The Deployment-patch deployer (FORGE_RELEASE_KEEPER=on) gets no store, so it prunes nothing. The images that timer pruned are on the nodes or in a registry, and the pod can reach neither without new credentials. The swap keeper is where releases are going (#8941, #8942).
  • I left the landing step's pruneReleases (src/checks/landing.ts, keeps 5 by default) alone. It works on the same releases/ folder and only protects the current sha. So under the swap keeper it will usually prune first, and could delete a known-good tree older than 5 checkouts. Fixing that changes landing behaviour, so it should be its own issue.

How to try it bun test test/release-keeper-prune.test.ts covers:

  • 20 releases keep 15 plus the live one and the known-good.
  • A known-good older than the newest 15 is kept.
  • A store failure partway is reported and the release stays verified.
  • The swapper's store lists and removes trees and refuses the running one.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1653 files for this diff, too many to run here. Instead I ran all test/keeper files, the footing, swapper and OPERATING.md tests, and the doc tests: 197 pass, 0 fail.

🤖 Generated with Claude Code

Closes #11801

commented

After a release is verified live, the keeper now prunes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the release.verified_live event. This replaces the forge-prune-images host timer on pg-grand.

What changed

  • The rule is toPrune in src/keeper/prune.ts. It takes the stored releases with their ages and the ids that must stay, and returns the ids to delete, oldest first. It does no file or registry work.
  • The Deployer contract (src/keeper/deploy.ts) has a new optional store with three methods: list, idOf(ref) and remove(id). The keeper uses idOf to match the live ref, the known-good ref and image() to ids in the store.
  • The swapper's store (src/keeper/swapper.ts) is the trees under releases/<sha>/, dated by their mtime. It skips anything that isn't a 40-hex name, such as good/ and .partial folders. It refuses to remove the commit it is running or the one it handed over to, whatever it is asked.
  • The keeper (src/keeper/keeper.ts) prunes after the release is marked verified and before it sends the event. The event carries pruned: string[]. If the store refuses partway, the event also carries pruneError and lists what was already removed. The release stays verified either way. If the live ref has no id in the store, nothing is pruned.
  • Footing (src/system/footing.ts): the host-timers row now names only forge-watchdog and forge-github-mirror. A new release-prune row (find: self) puts pruning on the vxrail. The docs/OPERATING.md table is regenerated from footingTable(), its prose is updated, and docs/features/release-keeper.md gains a step 5.

What I chose not to do

  • The Deployment-patch deployer (FORGE_RELEASE_KEEPER=on) gets no store, so it prunes nothing. The images that timer pruned are on the nodes or in a registry, and the pod can reach neither without new credentials. The swap keeper is where releases are going (#8941, #8942).
  • I left the landing step's pruneReleases (src/checks/landing.ts, keeps 5 by default) alone. It works on the same releases/ folder and only protects the current sha. So under the swap keeper it will usually prune first, and could delete a known-good tree older than 5 checkouts. Fixing that changes landing behaviour, so it should be its own issue.

How to try it bun test test/release-keeper-prune.test.ts covers:

  • 20 releases keep 15 plus the live one and the known-good.
  • A known-good older than the newest 15 is kept.
  • A store failure partway is reported and the release stays verified.
  • The swapper's store lists and removes trees and refuses the running one.

Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1653 files for this diff, too many to run here. Instead I ran all test/keeper files, the footing, swapper and OPERATING.md tests, and the doc tests: 197 pass, 0 fail.

🤖 Generated with Claude Code

Changes requested — before the latest push

Two things to fix, then this is close.

  1. (first, high) Known-good protection is not fail-closed. In keeper.ts prune(), only the live ref is checked for a store id. If the known-good image has no id (idOf returns undefined, e.g. a ref in another format), it is dropped from keep silently and can be pruned, which breaks the issue rule that the known-good is never removed. Prune nothing and set pruneError when the known-good or image() names no stored id. Add a test for it.
  1. (high) The landing step pruneReleases (src/checks/landing.ts, keeps 5, protects only the current sha) works on the same releases/ folder and can delete the known-good tree before this keeper ever runs. The pull acknowledges this and defers it, but that leaves rollback targets deletable, so the new rule is not actually what guarantees the known-good survives. Make landing prune skip the known-good (or stop pruning under the swap keeper) in this pull, with a test; do not leave it as a someday issue.

Smaller: swapper store.list dates a tree by directory mtime, which changes only when entries are added or removed, not when it is checked out; the comment in swapper.ts claims last checked out or pinned. Either stamp the tree when it is used or fix the comment, so the newest-15 order is true. Also the Deployment-patch deployer prunes nothing, so box 1 holds only for the swap keeper, while the footing row and OPERATING.md now say pruning has moved off the host timer; say in the row that it covers the swap keeper only. rmSync of trees runs inside the tick; fine at 5, but note it.

No page or TUI surface is added; the how-to is in docs/features/release-keeper.md, not README. Good: the rule is pure, the swapper refuses the running tree and non-sha names, and tests cover the 20-release and old-known-good cases.

commented

After each release is verified live, the release step now removes old stored releases itself. It keeps the newest 15, plus the live release and the release's named known-good (rollback target) however old. The pure rule is in src/keeper/prune.ts. The keeper runs it in src/keeper/keeper.ts and lists what it removed as pruned on release.verified_live. If the store refuses, it adds a pruneError and the release still counts as verified. Under the self-swap keeper, the store is the trees under releases/<sha>/ (src/keeper/swapper.ts), and it refuses to remove the running tree or any name that isn't a 40-character commit id. The Deployment-patch deployer has no store the forge can reach, so it prunes nothing.

Changes from review:

  • Known-good is now fail-closed (src/keeper/keeper.ts, prune()): if the live ref, the running image or the known-good ref has no stored id, nothing is pruned and pruneError names the ref. Two new tests in test/release-keeper-prune.test.ts: a known-good the store can't name, and a running image the store can't name. In both, all 20 releases stay.
  • Landing's prune can no longer delete rollback targets (src/release/checkout.ts, src/checks/landing.ts): pruneReleases now takes a list of commits to protect instead of a single one. Landing passes the landed commit, the running commit, the keeper's newest release and that release's known-good tree. So the old known-good survives even when landing's keep-5 would remove it. New test in test/release-checkout.test.ts; it fails without the change.
  • Swapper comment corrected, no stamping added: I fixed the comment rather than stamping trees on use. The list is ordered by directory mtime. checkoutRelease sets it when it writes a tree and touches it whenever the same commit is checked out again (swap, rollback, landing). A tree that is only pinned as good is not touched, but the known-good is protected anyway, so the order needs no extra stamp. The comment also says removal runs synchronously inside the keeper's tick, which is a few trees per release.
  • Footing row and docs say it covers the swap keeper only: the footing row in src/system/footing.ts, the matching row in docs/OPERATING.md and its prose there, and docs/features/release-keeper.md now say pruning replaces forge-prune-images only under the swap keeper, and that the Deployment-patch deployer prunes nothing.

What I chose not to do: there is no page or terminal screen for pruning. The how-to stays in docs/features/release-keeper.md, not the README.

To check: bun x tsc --noEmit passes. scripts/fast.ts named 1969 test files, so I ran the ones that cover this change by hand: test/release-checkout.test.ts, test/release-keeper-*.test.ts, test/release-swap.test.ts, test/release-swapped-launch.test.ts, test/footing.test.ts, test/landing.test.ts, test/scan-hold-landing.test.ts and test/tui-release-keeper.test.ts. Result: 114 pass, 0 fail, and 1 skipped because it needs unix sockets, which this sandbox can't bind. bun scripts/gate.ts printed nothing.

🤖 Generated with Claude Code

commented

After a release is verified live, the keeper now deletes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the release.verified_live event. This replaces the forge-prune-images host timer on pg-grand.

What changed

  • The rule is toPrune in src/keeper/prune.ts: a list of stored releases in, the ids to delete out.
  • The keeper (src/keeper/keeper.ts) runs the rule through an optional store on the Deployer (src/keeper/deploy.ts). If the live release, the running one or the known-good can't be matched to a stored release, nothing is pruned and the reason is recorded. The release stays verified either way.
  • The swapper's store (src/keeper/swapper.ts) is the folders under releases/<sha>/. The Deployment-patch deployer has no store, so it prunes nothing.
  • Landing's own prune of releases/ (src/checks/landing.ts, src/release/checkout.ts) now protects a list of commits instead of one: the landed commit, the running one, and the keeper's newest release and its known-good.
  • Footing: the pg-grand host-timers row no longer names image pruning, and a new release-prune row says pruning runs in the forge on the vxrail. docs/OPERATING.md and docs/features/release-keeper.md say the same.
  • Tests: test/release-keeper-prune.test.ts is new and test/release-checkout.test.ts has a new case.

About the last check The previous check failed 3 tests. Two were in test/pull-quiet.test.ts. I could not reproduce them, and this diff does not change those pages: they measure 32,097 and 25,478 bytes with or without it, against budgets of 32,160 and 25,530. That file passes on its own, merged onto bc5243e, merged onto the latest main, and run in one process with the files most likely to have shared its shard. The real shard could not be rebuilt here because its split depends on timings kept in the forge's cache. Both failing tests end in a byte budget only 50–60 bytes away, so something in the check's environment likely pushes them over; neither tests nor budgets were changed. If it goes red again, its log should show which assertion failed.

The third, test/runs-working-tab.test.ts ("a queued run is listed by its state word…"), also fails on its own on main, so it isn't from this pull.

This push merges main up to 7391f7a so the check runs on a current base. On the merged tree, bun scripts/gate.ts and bun x tsc --noEmit pass, and so do the pull-quiet, release-keeper-prune, release-checkout, release-keeper-cadence, check-disk-hold, footing, landing and swapper tests. The suites about unix sockets and cells are skipped in this sandbox; the forge's check runs them.

Not done: no image registry is pruned. The keeper only prunes what its own deployer stores.

How to try it: bun test test/release-keeper-prune.test.ts test/release-checkout.test.ts

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs