Public: anyone can read this; the crew can change it.
The release step prunes old release images itself: the newest 15, the live one and the known-good rollback are kept #11845
Check · Review · Land · Live — Its check is queued.
After a release is verified live, the keeper now prunes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the `release.verified_live` event. This replaces the `forge-prune-images` host timer on pg-grand.
11 files · +375 −13
working queued to check deploy 32409d0
Check queued 40mCheck of 32409d0fast tier: queued · full gate runs on land
Website spec: 23 findings (report only)
- / <a href="/admin-ada0/admin/app/production/open"> has no accessible name — names
- / <a href="/bad-name-12dd/bad-name/app/production/open"> has no accessible name — names
- /front <a href="/admin-ada0/admin/app/production/open"> has no accessible name — names
- /front <a href="/bad-name-12dd/bad-name/app/production/open"> has no accessible name — names
- /board http://127.0.0.1:46141/board answered 303 — https://specification.website/
- /notes http://127.0.0.1:46141/notes answered 303 — https://specification.website/
- /feed http://127.0.0.1:46141/feed answered 303 — https://specification.website/
- /review http://127.0.0.1:46141/review answered 303 — https://specification.website/
- /releases http://127.0.0.1:46141/releases answered 303 — https://specification.website/
- /repos http://127.0.0.1:46141/repos answered 303 — https://specification.website/
- /map http://127.0.0.1:46141/map answered 303 — https://specification.website/
- /system http://127.0.0.1:46141/system answered 303 — https://specification.website/
- /talk http://127.0.0.1:46141/talk answered 303 — https://specification.website/
- /flow http://127.0.0.1:46141/flow answered 303 — https://specification.website/
- /resilience http://127.0.0.1:46141/resilience answered 303 — https://specification.website/
- /boards?help http://127.0.0.1:46141/boards?help answered 303 — https://specification.website/
- /settings/crew http://127.0.0.1:46141/settings/crew answered 303 — https://specification.website/
- /settings/crew?help http://127.0.0.1:46141/settings/crew?help answered 303 — https://specification.website/
- /preview-admin/sample expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/next expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/bets expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/waves expected exactly one <h1>, found 2 — headings
- /preview-admin/sample/milestones expected exactly one <h1>, found 2 — headings
Look
9 pages beside main · 0 new to this pull
/apps
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/explore
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/feed
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/front
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/map
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/notes
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/releases
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/repos
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
/review
In the dark
at daad645 · · the head has moved since; a new picture follows its deploy
After a release is verified live, the keeper now prunes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the release.verified_live event. This replaces the forge-prune-images host timer on pg-grand.
What changed
- The rule is
toPruneinsrc/keeper/prune.ts. It takes the stored releases with their ages and the ids that must stay, and returns the ids to delete, oldest first. It does no file or registry work. - The
Deployercontract (src/keeper/deploy.ts) has a new optionalstorewith three methods:list,idOf(ref)andremove(id). The keeper usesidOfto match the live ref, the known-good ref andimage()to ids in the store. - The swapper's store (
src/keeper/swapper.ts) is the trees underreleases/<sha>/, dated by their mtime. It skips anything that isn't a 40-hex name, such asgood/and.partialfolders. It refuses to remove the commit it is running or the one it handed over to, whatever it is asked. - The keeper (
src/keeper/keeper.ts) prunes after the release is marked verified and before it sends the event. The event carriespruned: string[]. If the store refuses partway, the event also carriespruneErrorand lists what was already removed. The release stays verified either way. If the live ref has no id in the store, nothing is pruned. - Footing (
src/system/footing.ts): the host-timers row now names onlyforge-watchdogandforge-github-mirror. A newrelease-prunerow (find: self) puts pruning on the vxrail. Thedocs/OPERATING.mdtable is regenerated fromfootingTable(), its prose is updated, anddocs/features/release-keeper.mdgains a step 5.
What I chose not to do
- The Deployment-patch deployer (
FORGE_RELEASE_KEEPER=on) gets no store, so it prunes nothing. The images that timer pruned are on the nodes or in a registry, and the pod can reach neither without new credentials. The swap keeper is where releases are going (#8941, #8942). - I left the landing step's
pruneReleases(src/checks/landing.ts, keeps 5 by default) alone. It works on the samereleases/folder and only protects the current sha. So under the swap keeper it will usually prune first, and could delete a known-good tree older than 5 checkouts. Fixing that changes landing behaviour, so it should be its own issue.
How to try it
bun test test/release-keeper-prune.test.ts covers:
- 20 releases keep 15 plus the live one and the known-good.
- A known-good older than the newest 15 is kept.
- A store failure partway is reported and the release stays verified.
- The swapper's store lists and removes trees and refuses the running one.
Checks run: bun scripts/gate.ts and bun x tsc --noEmit are clean. scripts/fast.ts named 1653 files for this diff, too many to run here. Instead I ran all test/keeper files, the footing, swapper and OPERATING.md tests, and the doc tests: 197 pass, 0 fail.
🤖 Generated with Claude Code
Two things to fix, then this is close.
- (first, high) Known-good protection is not fail-closed. In keeper.ts prune(), only the live ref is checked for a store id. If the known-good image has no id (idOf returns undefined, e.g. a ref in another format), it is dropped from keep silently and can be pruned, which breaks the issue rule that the known-good is never removed. Prune nothing and set pruneError when the known-good or image() names no stored id. Add a test for it.
- (high) The landing step pruneReleases (src/checks/landing.ts, keeps 5, protects only the current sha) works on the same releases/ folder and can delete the known-good tree before this keeper ever runs. The pull acknowledges this and defers it, but that leaves rollback targets deletable, so the new rule is not actually what guarantees the known-good survives. Make landing prune skip the known-good (or stop pruning under the swap keeper) in this pull, with a test; do not leave it as a someday issue.
Smaller: swapper store.list dates a tree by directory mtime, which changes only when entries are added or removed, not when it is checked out; the comment in swapper.ts claims last checked out or pinned. Either stamp the tree when it is used or fix the comment, so the newest-15 order is true. Also the Deployment-patch deployer prunes nothing, so box 1 holds only for the swap keeper, while the footing row and OPERATING.md now say pruning has moved off the host timer; say in the row that it covers the swap keeper only. rmSync of trees runs inside the tick; fine at 5, but note it.
No page or TUI surface is added; the how-to is in docs/features/release-keeper.md, not README. Good: the rule is pure, the swapper refuses the running tree and non-sha names, and tests cover the 20-release and old-known-good cases.
After each release is verified live, the release step now removes old stored releases itself. It keeps the newest 15, plus the live release and the release's named known-good (rollback target) however old. The pure rule is in src/keeper/prune.ts. The keeper runs it in src/keeper/keeper.ts and lists what it removed as pruned on release.verified_live. If the store refuses, it adds a pruneError and the release still counts as verified. Under the self-swap keeper, the store is the trees under releases/<sha>/ (src/keeper/swapper.ts), and it refuses to remove the running tree or any name that isn't a 40-character commit id. The Deployment-patch deployer has no store the forge can reach, so it prunes nothing.
Changes from review:
- Known-good is now fail-closed (
src/keeper/keeper.ts,prune()): if the live ref, the running image or the known-good ref has no stored id, nothing is pruned andpruneErrornames the ref. Two new tests intest/release-keeper-prune.test.ts: a known-good the store can't name, and a running image the store can't name. In both, all 20 releases stay. - Landing's prune can no longer delete rollback targets (
src/release/checkout.ts,src/checks/landing.ts):pruneReleasesnow takes a list of commits to protect instead of a single one. Landing passes the landed commit, the running commit, the keeper's newest release and that release's known-good tree. So the old known-good survives even when landing's keep-5 would remove it. New test intest/release-checkout.test.ts; it fails without the change. - Swapper comment corrected, no stamping added: I fixed the comment rather than stamping trees on use. The list is ordered by directory mtime.
checkoutReleasesets it when it writes a tree and touches it whenever the same commit is checked out again (swap, rollback, landing). A tree that is only pinned as good is not touched, but the known-good is protected anyway, so the order needs no extra stamp. The comment also says removal runs synchronously inside the keeper's tick, which is a few trees per release. - Footing row and docs say it covers the swap keeper only: the footing row in
src/system/footing.ts, the matching row indocs/OPERATING.mdand its prose there, anddocs/features/release-keeper.mdnow say pruning replacesforge-prune-imagesonly under the swap keeper, and that the Deployment-patch deployer prunes nothing.
What I chose not to do: there is no page or terminal screen for pruning. The how-to stays in docs/features/release-keeper.md, not the README.
To check: bun x tsc --noEmit passes. scripts/fast.ts named 1969 test files, so I ran the ones that cover this change by hand: test/release-checkout.test.ts, test/release-keeper-*.test.ts, test/release-swap.test.ts, test/release-swapped-launch.test.ts, test/footing.test.ts, test/landing.test.ts, test/scan-hold-landing.test.ts and test/tui-release-keeper.test.ts. Result: 114 pass, 0 fail, and 1 skipped because it needs unix sockets, which this sandbox can't bind. bun scripts/gate.ts printed nothing.
🤖 Generated with Claude Code
After a release is verified live, the keeper now deletes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the release.verified_live event. This replaces the forge-prune-images host timer on pg-grand.
What changed
- The rule is
toPruneinsrc/keeper/prune.ts: a list of stored releases in, the ids to delete out. - The keeper (
src/keeper/keeper.ts) runs the rule through an optionalstoreon theDeployer(src/keeper/deploy.ts). If the live release, the running one or the known-good can't be matched to a stored release, nothing is pruned and the reason is recorded. The release stays verified either way. - The swapper's store (
src/keeper/swapper.ts) is the folders underreleases/<sha>/. The Deployment-patch deployer has no store, so it prunes nothing. - Landing's own prune of
releases/(src/checks/landing.ts,src/release/checkout.ts) now protects a list of commits instead of one: the landed commit, the running one, and the keeper's newest release and its known-good. - Footing: the pg-grand host-timers row no longer names image pruning, and a new
release-prunerow says pruning runs in the forge on the vxrail.docs/OPERATING.mdanddocs/features/release-keeper.mdsay the same. - Tests:
test/release-keeper-prune.test.tsis new andtest/release-checkout.test.tshas a new case.
About the last check
The previous check failed 3 tests. Two were in test/pull-quiet.test.ts. I could not reproduce them, and this diff does not change those pages: they measure 32,097 and 25,478 bytes with or without it, against budgets of 32,160 and 25,530. That file passes on its own, merged onto bc5243e, merged onto the latest main, and run in one process with the files most likely to have shared its shard. The real shard could not be rebuilt here because its split depends on timings kept in the forge's cache. Both failing tests end in a byte budget only 50–60 bytes away, so something in the check's environment likely pushes them over; neither tests nor budgets were changed. If it goes red again, its log should show which assertion failed.
The third, test/runs-working-tab.test.ts ("a queued run is listed by its state word…"), also fails on its own on main, so it isn't from this pull.
This push merges main up to 7391f7a so the check runs on a current base. On the merged tree, bun scripts/gate.ts and bun x tsc --noEmit pass, and so do the pull-quiet, release-keeper-prune, release-checkout, release-keeper-cadence, check-disk-hold, footing, landing and swapper tests. The suites about unix sockets and cells are skipped in this sandbox; the forge's check runs them.
Not done: no image registry is pruned. The keeper only prunes what its own deployer stores.
How to try it: bun test test/release-keeper-prune.test.ts test/release-checkout.test.ts
🤖 Generated with Claude Code
After a release is verified live, the keeper now prunes its own old releases. It keeps the newest 15, plus the live release and the release's named known-good, however old that one is. The list of what was removed goes on the
release.verified_liveevent. This replaces theforge-prune-imageshost timer on pg-grand.What changed
toPruneinsrc/keeper/prune.ts. It takes the stored releases with their ages and the ids that must stay, and returns the ids to delete, oldest first. It does no file or registry work.Deployercontract (src/keeper/deploy.ts) has a new optionalstorewith three methods:list,idOf(ref)andremove(id). The keeper usesidOfto match the live ref, the known-good ref andimage()to ids in the store.src/keeper/swapper.ts) is the trees underreleases/<sha>/, dated by their mtime. It skips anything that isn't a 40-hex name, such asgood/and.partialfolders. It refuses to remove the commit it is running or the one it handed over to, whatever it is asked.src/keeper/keeper.ts) prunes after the release is marked verified and before it sends the event. The event carriespruned: string[]. If the store refuses partway, the event also carriespruneErrorand lists what was already removed. The release stays verified either way. If the live ref has no id in the store, nothing is pruned.src/system/footing.ts): the host-timers row now names onlyforge-watchdogandforge-github-mirror. A newrelease-prunerow (find: self) puts pruning on the vxrail. Thedocs/OPERATING.mdtable is regenerated fromfootingTable(), its prose is updated, anddocs/features/release-keeper.mdgains a step 5.What I chose not to do
FORGE_RELEASE_KEEPER=on) gets no store, so it prunes nothing. The images that timer pruned are on the nodes or in a registry, and the pod can reach neither without new credentials. The swap keeper is where releases are going (#8941, #8942).pruneReleases(src/checks/landing.ts, keeps 5 by default) alone. It works on the samereleases/folder and only protects the current sha. So under the swap keeper it will usually prune first, and could delete a known-good tree older than 5 checkouts. Fixing that changes landing behaviour, so it should be its own issue.How to try it
bun test test/release-keeper-prune.test.tscovers:Checks run:
bun scripts/gate.tsandbun x tsc --noEmitare clean.scripts/fast.tsnamed 1653 files for this diff, too many to run here. Instead I ran alltest/keeperfiles, the footing, swapper and OPERATING.md tests, and the doc tests: 197 pass, 0 fail.🤖 Generated with Claude Code
Closes #11801