Public: anyone can read this; the crew can change it.
The release step prunes old release images itself: the newest 15, the live one and the known-good rollback are kept #11845
Check · Review · Land · Live — Its check is queued.
mason/forge:mason/11801-the-release-step-prunes-old-release-imag → main 5 commits fork 1 thread · 1 unresolved
11 files
changed +375
−13
docs/OPERATING.md +3 −2
| @@ -122,7 +122,8 @@ down. | ||
| 122 | 122 | | (d) release | registry-node, a registry on the vxrail | vxrail | It goes with the forge, which runs beside it. | — | — | |
| 123 | 123 | | (d) release | forge-release-tick, a host timer every 5 min | pg-grand | Nothing is released; production keeps the release it has. | 7 | #11686, #8942 | |
| 124 | 124 | | (d) release | standby-reconcile, a host timer every 2 min that applies the forge's manifests from Forgejo; Flux's Kustomization for the forge is suspended | pg-grand | A changed manifest is not applied; the running pod is untouched. | 7 | #8948, #11686 | |
| 125 | | (d) release | Host timers forge-watchdog, forge-github-mirror and forge-prune-images | pg-grand | The watchdog, the GitHub mirror and image pruning stop. | 7 | #11686 | | |
| 125 | | (d) release | Host timers forge-watchdog and forge-github-mirror | pg-grand | The watchdog and the GitHub mirror stop. | 7 | #11686 | | |
| 126 | | (d) release | Pruning swapped-in release trees, under the self-swap keeper only: the release keeper keeps the newest 15, the live one and the known-good after each verified release, in place of the host timer forge-prune-images (#11801); the Deployment-patch deployer prunes nothing | vxrail | It goes with the forge, which runs beside it. | — | — | | |
| 126 | 127 | | (e) back up | The backup namespace's CronJobs (FORGE_BACKUP_LANES) | pg-grand | No new backup is taken. | 6 | #11686 | |
| 127 | 128 | | (e) back up | MinIO, where the backup lanes store the forge's backups | pg-grand | No new backup is stored, and those it holds go with the machine. | 6 | #11686 | |
| 128 | 129 | |
| @@ -230,7 +231,7 @@ rollback. #11803 (tunnel straight to the forge), #11802 (its own sign-in) and | ||
| 230 | 231 | operator this move takes out. |
| 231 | 232 | - **MinIO** is copied once into the off-site bucket. |
| 232 | 233 | - **What else pg-grand runs for the forge:** the backup CronJobs go to |
| 233 | #11805 and `forge-prune-images` to #11801. `forge-release-tick` and | |
| 234 | #11805; `forge-prune-images` is replaced by the swap keeper's own pruning (#11801; the Deployment-patch deployer prunes nothing). `forge-release-tick` and | |
| 234 | 235 | `standby-reconcile` go to the swap (#8941, #8942). `forge-watchdog` is |
| 235 | 236 | replaced by `Restart=always` and the launcher. `plat-auth`, the signup |
| 236 | 237 | gate and Traefik leave the forge's path with #11802 and #11803. |
docs/features/release-keeper.md +12
| @@ -53,6 +53,18 @@ keyboard. It does what the steward's `release.sh`, `verify-after.sh` and | ||
| 53 | 53 | by name (`keeper/rollback.ts`). There is no "undo to previous": on |
| 54 | 54 | 2026-10-27 a `rollout undo` put the broken version back for two minutes. |
| 55 | 55 | A release that is not live 3 h after its cut is rolled back the same way. |
| 56 | 5. **Prune** (`keeper/prune.ts`, #11801). Once a release is verified live, | |
| 57 | the keeper removes the deployer's stored releases beyond the newest 15, | |
| 58 | never the live one or the release's named known-good, however old. Under | |
| 59 | the self-swap keeper these are the trees under `releases/<sha>/`; the | |
| 60 | commits removed are `pruned` on `release.verified_live` (with | |
| 61 | `pruneError` when the store refused, and the release stays verified). | |
| 62 | The Deployment-patch deployer keeps no store the forge can reach, so it | |
| 63 | prunes nothing. If the live, running or known-good release names no | |
| 64 | stored one, it prunes nothing and says so in `pruneError`. Landing's own | |
| 65 | prune of `releases/` (keep `FORGE_RELEASES_KEEP`) never removes the | |
| 66 | keeper's newest release or its known-good tree. Removal runs inside the | |
| 67 | tick. Under the swap keeper this replaces the host timer `forge-prune-images`. | |
| 56 | 68 | |
| 57 | 69 | Every step is an event on the outbox: `release.cut`, `release.released`, |
| 58 | 70 | `release.verified_live`, `release.rolled_back` and `release.held`. |
src/checks/landing.ts +15 −1
| @@ -36,6 +36,8 @@ import { gateRecorded, gateRunning, recordedVerdict } from "./queue_store.ts"; | ||
| 36 | 36 | import { checksFor, type GateDone } from "./service.ts"; |
| 37 | 37 | import { landVerdict, type LandVerdict } from "./tiers.ts"; |
| 38 | 38 | import { ownProbeHeaders } from "../kernel/probe.ts"; |
| 39 | import { listKeeperReleases } from "../keeper/record.ts"; | |
| 40 | import { treeOf } from "../keeper/swapper.ts"; | |
| 39 | 41 | |
| 40 | 42 | /** |
| 41 | 43 | * The full gate, once, on a commit that is what main would become — one |
| @@ -144,6 +146,18 @@ async function preflightTrusted(dir: string, dataDir: string): Promise<Verdict> | ||
| 144 | 146 | return first.ok ? first : preflightRelease(dir, dataDir); |
| 145 | 147 | } |
| 146 | 148 | |
| 149 | /** | |
| 150 | * The trees the release keeper may still need (#11801): the one running, the | |
| 151 | * keeper's newest release and that release's named known-good, the rollback | |
| 152 | * target. Landing's prune never | |
| 153 | * removes them, so the keeper's own rule is what decides when they go. | |
| 154 | */ | |
| 155 | function keeperTrees(app: App): string[] { | |
| 156 | const newest = listKeeperReleases(app.db, 1)[0]; | |
| 157 | const trees = [running().commit, newest?.sha, newest === undefined ? undefined : treeOf(newest.knownGood.image)]; | |
| 158 | return trees.filter((sha): sha is string => typeof sha === "string"); | |
| 159 | } | |
| 160 | ||
| 147 | 161 | /** Landings queue behind one another: two checkouts never share releases/. */ |
| 148 | 162 | let turn: Promise<void> = Promise.resolve(); |
| 149 | 163 | |
| @@ -171,7 +185,7 @@ async function cut( | ||
| 171 | 185 | ok: verdict.ok, |
| 172 | 186 | line: verdict.line, |
| 173 | 187 | }); |
| 174 | pruneReleases(dataDir, app.config.releasesKeep ?? DEFAULT_KEEP, sha); | |
| 188 | pruneReleases(dataDir, app.config.releasesKeep ?? DEFAULT_KEEP, [sha, ...keeperTrees(app)]); | |
| 175 | 189 | if (!verdict.ok) { |
| 176 | 190 | app.log.warn("release preflight failed", { |
| 177 | 191 | sha: sha.slice(0, 12), |
src/keeper/deploy.ts +13
| @@ -13,8 +13,21 @@ | ||
| 13 | 13 | */ |
| 14 | 14 | |
| 15 | 15 | import type { Credentials } from "../cluster/jobs.ts"; |
| 16 | import type { Stored } from "./prune.ts"; | |
| 17 | ||
| 18 | /** Where a deployer keeps its releases, so the keeper can prune them (#11801). */ | |
| 19 | export interface ReleaseStore { | |
| 20 | /** Every release stored now, with when it was made. */ | |
| 21 | list(): Promise<Stored[]>; | |
| 22 | /** The store id a reference names (`image()`, `refFor`, a known-good image); undefined when it names none here. */ | |
| 23 | idOf(ref: string): string | undefined; | |
| 24 | /** Delete one stored release. */ | |
| 25 | remove(id: string): Promise<void>; | |
| 26 | } | |
| 16 | 27 | |
| 17 | 28 | export interface Deployer { |
| 29 | /** Its stored releases, when it keeps any the keeper should prune. */ | |
| 30 | readonly store?: ReleaseStore; | |
| 18 | 31 | /** The image the Deployment's forge container names now. */ |
| 19 | 32 | image(): Promise<string>; |
| 20 | 33 | /** Name this image on the forge container; the Deployment rolls to it with surge. */ |
src/keeper/keeper.ts +37 −2
| @@ -18,12 +18,15 @@ | ||
| 18 | 18 | * 4. **Verified live**, or **roll back** to the named known-good image |
| 19 | 19 | * (`rollback.ts`; there is no undo) and **hold** with the numbers |
| 20 | 20 | * (`hold.ts`). A release with seconds dark files one issue. |
| 21 | * 5. **Prune**: once verified live, the deployer's stored releases beyond | |
| 22 | * the newest 15 are removed, never the live one or the release's named | |
| 23 | * known-good (`prune.ts`, #11801); the list is on `release.verified_live`. | |
| 21 | 24 | * |
| 22 | 25 | * Each step is an event on the outbox: `release.cut`, `release.released`, |
| 23 | 26 | * `release.verified_live`, `release.rolled_back`, `release.held`. |
| 24 | 27 | * |
| 25 | 28 | * Guardrail: the keeper's only writes are a tag, the Deployment's image, |
| 26 | * its own rows, and issues. It holds no account that may land a pull or | |
| 29 | * its own rows, issues, and the removal of its own old releases. It holds no account that may land a pull or | |
| 27 | 30 | * push main, and it has no code path to either. |
| 28 | 31 | */ |
| 29 | 32 | |
| @@ -35,6 +38,7 @@ import { imageAt, type Deployer } from "./deploy.ts"; | ||
| 35 | 38 | import { blocking, liftFixed, names, setCause, setHold, standingHolds, type Hold } from "./hold.ts"; |
| 36 | 39 | import { stallReason, STALL_HOLD_MS } from "../perf/boot_stall.ts"; |
| 37 | 40 | import { countRollProbe, dropCut, finish, getKeeperRelease, inFlight, keepStall, keepTally, listKeeperReleases, markLive, releaseBefore, recordCut, type KeeperRelease } from "./record.ts"; |
| 41 | import { toPrune } from "./prune.ts"; | |
| 38 | 42 | import { rollBackToNamed } from "./rollback.ts"; |
| 39 | 43 | import type { Probe, Tally, Verdict } from "./verify.ts"; |
| 40 | 44 | import type { Step } from "./beat.ts"; |
| @@ -132,7 +136,8 @@ async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<Step> | ||
| 132 | 136 | finish(p.db, r.id, "verified", stalled ?? "", p.now()); |
| 133 | 137 | p.deployer.verified?.(r.version); |
| 134 | 138 | const lifted = liftFixed(p.db, r.version, p.now()); |
| 135 | p.bus.append({ kind: "release.verified_live", repoId: p.repo.id, payload: { ...numbers, holdsLifted: lifted } }); | |
| 139 | const pruned = await prune(p, r); | |
| 140 | p.bus.append({ kind: "release.verified_live", repoId: p.repo.id, payload: { ...numbers, holdsLifted: lifted, ...pruned } }); | |
| 136 | 141 | fileDark(p, getKeeperRelease(p.db, r.id)!); |
| 137 | 142 | return { line: `${r.version} verified live: ${kept.probes} probes, worst ${kept.worstMs} ms, ${kept.secondsDark} s dark${stalled === undefined ? "" : `; ${stalled}`}`, outcome: "cut" }; |
| 138 | 143 | } |
| @@ -142,6 +147,36 @@ async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<Step> | ||
| 142 | 147 | } |
| 143 | 148 | } |
| 144 | 149 | |
| 150 | /** | |
| 151 | * Remove the deployer's stored releases the rule names (#11801): the live one | |
| 152 | * (the release just verified, and whatever the deployer runs now) and the | |
| 153 | * release's named known-good always stay. When any of those three names no | |
| 154 | * stored release, or the store cannot be read, it prunes nothing and says why; | |
| 155 | * the release stays verified either way. | |
| 156 | */ | |
| 157 | async function prune(p: KeeperPorts, r: KeeperRelease): Promise<{ pruned: string[]; pruneError?: string }> { | |
| 158 | const store = p.deployer.store; | |
| 159 | if (store === undefined) return { pruned: [] }; | |
| 160 | const pruned: string[] = []; | |
| 161 | try { | |
| 162 | const refs = [refFor(p, r.knownGood.image, r.version, r.sha), r.knownGood.image, await p.deployer.image()]; | |
| 163 | const keep: string[] = []; | |
| 164 | for (const ref of refs) { | |
| 165 | const id = store.idOf(ref); | |
| 166 | // Fail closed: a live, running or known-good release the store cannot name could be the one pruned. | |
| 167 | if (id === undefined) return { pruned: [], pruneError: `${ref} names no stored release` }; | |
| 168 | keep.push(id); | |
| 169 | } | |
| 170 | for (const id of toPrune(await store.list(), keep)) { | |
| 171 | await store.remove(id); | |
| 172 | pruned.push(id); | |
| 173 | } | |
| 174 | return { pruned }; | |
| 175 | } catch (error) { | |
| 176 | return { pruned, pruneError: error instanceof Error ? error.message : String(error) }; | |
| 177 | } | |
| 178 | } | |
| 179 | ||
| 145 | 180 | /** |
| 146 | 181 | * A release's worst stall in its first five minutes is kept, shown beside the last |
| 147 | 182 | * release's, and over {@link STALL_HOLD_MS} holds the next cut with the reason |
src/keeper/prune.ts added +31
| @@ -0,0 +1,31 @@ | ||
| 1 | /** | |
| 2 | * Which stored releases go (#11801): the release step prunes its own old | |
| 3 | * images, or under the self-swap keeper its swapped-in trees, once a | |
| 4 | * release is verified live. It replaces `forge-prune-images`, a host timer | |
| 5 | * on pg-grand. | |
| 6 | * | |
| 7 | * Plain data in, a list to delete out: the stored releases with when each | |
| 8 | * was made, and the ones that must stay (the live release, the release's | |
| 9 | * named known-good). The newest {@link KEEP_NEWEST} stay too; everything | |
| 10 | * else is named, oldest first. Nothing here touches a disk or a registry. | |
| 11 | */ | |
| 12 | ||
| 13 | /** How many of the newest stored releases are kept, besides the live one and the known-good. */ | |
| 14 | export const KEEP_NEWEST = 15; | |
| 15 | ||
| 16 | /** A stored release: its id in the store (a commit for a tree, a tag for an image) and when it was made, epoch ms. */ | |
| 17 | export interface Stored { | |
| 18 | readonly id: string; | |
| 19 | readonly at: number; | |
| 20 | } | |
| 21 | ||
| 22 | /** The ids to delete: all but the `newest` most recent, never one in `keep`. Oldest first. */ | |
| 23 | export function toPrune(found: readonly Stored[], keep: readonly string[], newest: number = KEEP_NEWEST): string[] { | |
| 24 | const limit = Math.max(1, Math.floor(Number.isFinite(newest) ? newest : KEEP_NEWEST)); | |
| 25 | // One entry per id, at its newest. | |
| 26 | const latest = new Map<string, number>(); | |
| 27 | for (const s of found) latest.set(s.id, Math.max(s.at, latest.get(s.id) ?? -Infinity)); | |
| 28 | const ordered = [...latest].sort(([a, at], [b, bt]) => bt - at || (a < b ? -1 : a > b ? 1 : 0)).map(([id]) => id); | |
| 29 | const kept = new Set(keep); | |
| 30 | return ordered.slice(limit).filter((id) => !kept.has(id)).reverse(); | |
| 31 | } |
src/keeper/swapper.ts +40 −3
| @@ -22,25 +22,38 @@ | ||
| 22 | 22 | * - **Refusals**: a commit refused (dependency, red preflight) is remembered |
| 23 | 23 | * per sha and logged once, so the next minute does not check it out and boot |
| 24 | 24 | * a preflight again. |
| 25 | * - **Pruning** (#11801): `store` lists the trees under `releases/<sha>/` | |
| 26 | * by the directory's mtime, which `checkoutRelease` sets when it writes the | |
| 27 | * tree and touches each time the same commit is checked out again (a swap, | |
| 28 | * a rollback, a landing); a tree only pinned as good is not touched. It | |
| 29 | * removes one by commit, synchronously, inside the keeper's tick (a few | |
| 30 | * trees once a release, so the tick is not held long); | |
| 31 | * the keeper decides which (`prune.ts`). The tree this process runs and the | |
| 32 | * one it handed over to are never removed, whatever it is asked. | |
| 25 | 33 | * - **Restart**: once verified, `verified` keeps the tree as the last good |
| 26 | 34 | * release over this image's version (`release/launch.ts`), so a pod |
| 27 | 35 | * restart comes back on it, not on the older image's version. |
| 28 | 36 | */ |
| 29 | 37 | |
| 30 | import { cpSync, existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; | |
| 38 | import { cpSync, existsSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; | |
| 31 | 39 | import { join } from "node:path"; |
| 32 | 40 | import { err, type Result } from "../kernel/result.ts"; |
| 33 | import { releaseDir, type Verdict } from "../release/checkout.ts"; | |
| 41 | import { releaseDir, releasesRoot, type Verdict } from "../release/checkout.ts"; | |
| 34 | 42 | import { keepGood } from "../release/launch.ts"; |
| 35 | 43 | import { liveSchema, releaseSchema } from "../release/rollback.ts"; |
| 36 | 44 | import { beginSwap, endSwap, startRelease, swapBlocked } from "../release/swap.ts"; |
| 37 | import type { Deployer } from "./deploy.ts"; | |
| 45 | import type { Deployer, ReleaseStore } from "./deploy.ts"; | |
| 46 | import type { Stored } from "./prune.ts"; | |
| 38 | 47 | |
| 39 | 48 | const REF = /^tree\/([0-9a-f]{40}):([0-9A-Za-z][0-9A-Za-z.+_-]{0,63})$/; |
| 40 | 49 | const TREE_PARTS = ["src", "package.json", "tsconfig.json"]; |
| 41 | 50 | /** A tree that has served: a rollback target, not a candidate, so it is not preflighted again. */ |
| 42 | 51 | const LIVE_MARK = ".was-live"; |
| 43 | 52 | const DEP_FIELDS = ["dependencies", "devDependencies", "optionalDependencies", "peerDependencies"]; |
| 53 | const SHA = /^[0-9a-f]{40}$/; | |
| 54 | ||
| 55 | /** The commit a swapper ref (`tree/<sha>:<version>`) names; undefined for any other ref. */ | |
| 56 | export const treeOf = (ref: string): string | undefined => REF.exec(ref)?.[1]; | |
| 44 | 57 | |
| 45 | 58 | export interface SwapperInput { |
| 46 | 59 | readonly dataDir: string; |
| @@ -137,7 +150,31 @@ export function swapper(i: SwapperInput): Deployer { | ||
| 137 | 150 | return known < live ? `${sha} predates schema ${live}; rolling back would need a down-migration` : undefined; |
| 138 | 151 | } |
| 139 | 152 | |
| 153 | const store: ReleaseStore = { | |
| 154 | async list() { | |
| 155 | const root = releasesRoot(i.dataDir); | |
| 156 | if (!existsSync(root)) return []; | |
| 157 | const found: Stored[] = []; | |
| 158 | for (const name of readdirSync(root)) { | |
| 159 | if (!SHA.test(name)) continue; | |
| 160 | try { | |
| 161 | found.push({ id: name, at: statSync(join(root, name)).mtimeMs }); | |
| 162 | } catch { | |
| 163 | // Removed between the listing and the look. | |
| 164 | } | |
| 165 | } | |
| 166 | return found; | |
| 167 | }, | |
| 168 | idOf: treeOf, | |
| 169 | async remove(sha) { | |
| 170 | if (!SHA.test(sha)) throw new Error("not a commit id"); | |
| 171 | if (sha === i.commit() || (handedTo !== undefined && REF.exec(handedTo)?.[1] === sha)) throw new Error(`${sha.slice(0, 12)} is running`); | |
| 172 | rmSync(releaseDir(i.dataDir, sha), { recursive: true, force: true }); | |
| 173 | }, | |
| 174 | }; | |
| 175 | ||
| 140 | 176 | return { |
| 177 | store, | |
| 141 | 178 | async image() { |
| 142 | 179 | return handedTo ?? here(); |
| 143 | 180 | }, |
src/release/checkout.ts +7 −3
| @@ -53,8 +53,12 @@ export async function checkoutRelease( | ||
| 53 | 53 | } |
| 54 | 54 | } |
| 55 | 55 | |
| 56 | /** Remove all but the `keep` newest releases, and `keep` is at least one. Returns what went. */ | |
| 57 | export function pruneReleases(dataDir: string, keep: number, current: string): string[] { | |
| 56 | /** | |
| 57 | * Remove all but the `keep` newest releases, and `keep` is at least one; the | |
| 58 | * commits in `protect` (the landed one, the running one, the keeper's | |
| 59 | * rollback target) stay however old. Returns what went. | |
| 60 | */ | |
| 61 | export function pruneReleases(dataDir: string, keep: number, protect: readonly string[]): string[] { | |
| 58 | 62 | const root = releasesRoot(dataDir); |
| 59 | 63 | if (!existsSync(root)) return []; |
| 60 | 64 | const limit = Math.max(1, Math.floor(keep)); |
| @@ -63,7 +67,7 @@ export function pruneReleases(dataDir: string, keep: number, current: string): s | ||
| 63 | 67 | .map((name) => ({ name, at: statSync(join(root, name)).mtimeMs })) |
| 64 | 68 | .sort((a, b) => b.at - a.at || (a.name < b.name ? -1 : 1)); |
| 65 | 69 | const kept = new Set(found.slice(0, limit).map((r) => r.name)); |
| 66 | kept.add(current); | |
| 70 | for (const sha of protect) kept.add(sha); | |
| 67 | 71 | const gone: string[] = []; |
| 68 | 72 | for (const { name } of found) { |
| 69 | 73 | if (kept.has(name)) continue; |
src/system/footing.ts +11 −2
| @@ -244,13 +244,22 @@ export const FOOTING: readonly Piece[] = [ | ||
| 244 | 244 | { |
| 245 | 245 | id: "host-timers", |
| 246 | 246 | need: "release", |
| 247 | what: "Host timers forge-watchdog, forge-github-mirror and forge-prune-images", | |
| 247 | what: "Host timers forge-watchdog and forge-github-mirror", | |
| 248 | 248 | find: { kind: "host", unit: "forge-watchdog" }, |
| 249 | 249 | machine: "pg-grand", |
| 250 | ifDies: "The watchdog, the GitHub mirror and image pruning stop.", | |
| 250 | ifDies: "The watchdog and the GitHub mirror stop.", | |
| 251 | 251 | issues: [11686], |
| 252 | 252 | breaks: 7, |
| 253 | 253 | }, |
| 254 | { | |
| 255 | id: "release-prune", | |
| 256 | need: "release", | |
| 257 | what: "Pruning swapped-in release trees, under the self-swap keeper only: the release keeper keeps the newest 15, the live one and the known-good after each verified release, in place of the host timer forge-prune-images (#11801); the Deployment-patch deployer prunes nothing", | |
| 258 | find: { kind: "self" }, | |
| 259 | machine: "vxrail", | |
| 260 | ifDies: SAME_DEATH, | |
| 261 | issues: [], | |
| 262 | }, | |
| 254 | 263 | { |
| 255 | 264 | id: "backup-lanes", |
| 256 | 265 | need: "backup", |
test/release-checkout.test.ts +14
| @@ -18,6 +18,7 @@ import { listComments } from "../src/core/issues.ts"; | ||
| 18 | 18 | import type { Pull } from "../src/core/pulls.ts"; |
| 19 | 19 | import type { Repo } from "../src/core/repos.ts"; |
| 20 | 20 | import { createUser, type User } from "../src/core/users.ts"; |
| 21 | import { recordCut } from "../src/keeper/record.ts"; | |
| 21 | 22 | import { releaseDir } from "../src/release/checkout.ts"; |
| 22 | 23 | import { releaseEnv } from "../src/release/env.ts"; |
| 23 | 24 | import { listReleases } from "../src/release/record.ts"; |
| @@ -110,6 +111,19 @@ describe("release checkout on landing", () => { | ||
| 110 | 111 | expect(existsSync(releaseDir(dataDir, first))).toBe(false); |
| 111 | 112 | }, 60_000); |
| 112 | 113 | |
| 114 | test("never prunes the release keeper's known-good tree, however old (#11801)", async () => { | |
| 115 | const good = listReleases(app.db)[0]!.sha; | |
| 116 | const old = new Date(Date.now() - 3_600_000); | |
| 117 | utimesSync(releaseDir(dataDir, good), old, old); | |
| 118 | recordCut(app.db, { version: "0.7.9", sha: "f".repeat(40), knownGood: { version: "0.7.8", image: `tree/${good}:0.7.8` }, at: Date.now() }); | |
| 119 | for (const n of ["four", "five"]) { | |
| 120 | const sha = await land(GOOD, n); | |
| 121 | await releaseOnLand(app, alice, forge, pull, sha); | |
| 122 | } | |
| 123 | expect(existsSync(releaseDir(dataDir, good))).toBe(true); | |
| 124 | expect(readdirSync(join(dataDir, "releases")).filter((d) => /^[0-9a-f]{40}$/.test(d))).toHaveLength(3); | |
| 125 | }, 60_000); | |
| 126 | ||
| 113 | 127 | test("a failing preflight is recorded red, logged, and told to the pull", async () => { |
| 114 | 128 | const sha = await land(BAD, "red"); |
| 115 | 129 | const release = await releaseOnLand(app, alice, forge, pull, sha); |
test/release-keeper-prune.test.ts added +192
| @@ -0,0 +1,192 @@ | ||
| 1 | /** | |
| 2 | * The release step prunes its own old releases (#11801): the rule is plain | |
| 3 | * data in and a list out, the keeper runs it once a release is verified live | |
| 4 | * and puts the list on `release.verified_live`, and the swapper's store is the | |
| 5 | * trees under `releases/<sha>/`. | |
| 6 | */ | |
| 7 | ||
| 8 | import { afterAll, describe, expect, test } from "bun:test"; | |
| 9 | import { existsSync, mkdirSync, mkdtempSync, rmSync, utimesSync } from "node:fs"; | |
| 10 | import { tmpdir } from "node:os"; | |
| 11 | import { join } from "node:path"; | |
| 12 | import { ok } from "../src/kernel/result.ts"; | |
| 13 | import { tagOf, type ReleaseStore } from "../src/keeper/deploy.ts"; | |
| 14 | import { tick } from "../src/keeper/keeper.ts"; | |
| 15 | import { KEEP_NEWEST, toPrune, type Stored } from "../src/keeper/prune.ts"; | |
| 16 | import { swapper } from "../src/keeper/swapper.ts"; | |
| 17 | import { releaseDir } from "../src/release/checkout.ts"; | |
| 18 | import { keeperForge } from "./helpers/keeper.ts"; | |
| 19 | ||
| 20 | const dirs: string[] = []; | |
| 21 | afterAll(() => { | |
| 22 | for (const d of dirs) rmSync(d, { recursive: true, force: true }); | |
| 23 | }); | |
| 24 | ||
| 25 | /** `n` releases, `r1` oldest, one minute apart. */ | |
| 26 | const releases = (n: number): Stored[] => Array.from({ length: n }, (_, i) => ({ id: `r${i + 1}`, at: 60_000 * (i + 1) })); | |
| 27 | ||
| 28 | describe("the rule", () => { | |
| 29 | test("20 releases keep the newest 15 plus the live one and the known-good", () => { | |
| 30 | const gone = toPrune(releases(20), ["r20", "r19"]); | |
| 31 | expect(KEEP_NEWEST).toBe(15); | |
| 32 | expect(gone).toEqual(["r1", "r2", "r3", "r4", "r5"]); | |
| 33 | }); | |
| 34 | ||
| 35 | test("a known-good older than the newest 15 is kept, and so is a live one", () => { | |
| 36 | expect(toPrune(releases(20), ["r20", "r2"])).toEqual(["r1", "r3", "r4", "r5"]); | |
| 37 | expect(toPrune(releases(20), ["r3", "r2"])).toEqual(["r1", "r4", "r5"]); | |
| 38 | }); | |
| 39 | ||
| 40 | test("15 or fewer prune nothing; order in, ties and repeats do not change the answer", () => { | |
| 41 | expect(toPrune(releases(15), [])).toEqual([]); | |
| 42 | expect(toPrune([], ["r1"])).toEqual([]); | |
| 43 | const shuffled = [...releases(20)].reverse(); | |
| 44 | expect(toPrune(shuffled, ["r20"])).toEqual(["r1", "r2", "r3", "r4", "r5"]); | |
| 45 | // A repeated id counts once, at its newest. | |
| 46 | expect(toPrune([...releases(16), { id: "r1", at: 999_999_999 }], [])).toEqual(["r2"]); | |
| 47 | }); | |
| 48 | ||
| 49 | test("a nonsense limit keeps at least one", () => { | |
| 50 | expect(toPrune(releases(3), [], 0)).toEqual(["r1", "r2"]); | |
| 51 | expect(toPrune(releases(17), [], Number.NaN)).toEqual(["r1", "r2"]); | |
| 52 | }); | |
| 53 | }); | |
| 54 | ||
| 55 | /** A registry of tags, as the fake Deployment's images name them. */ | |
| 56 | function fakeStore(tags: Stored[]) { | |
| 57 | const held = new Map(tags.map((t) => [t.id, t.at])); | |
| 58 | const store: ReleaseStore = { | |
| 59 | list: async () => [...held].map(([id, at]) => ({ id, at })), | |
| 60 | idOf: (ref) => tagOf(ref) || undefined, | |
| 61 | async remove(id) { | |
| 62 | held.delete(id); | |
| 63 | }, | |
| 64 | }; | |
| 65 | return { store, held }; | |
| 66 | } | |
| 67 | ||
| 68 | describe("the keeper prunes after verified live", () => { | |
| 69 | async function verifiedWith(tags: Stored[], store?: (s: ReleaseStore) => ReleaseStore) { | |
| 70 | const f = await keeperForge(); | |
| 71 | const fake = fakeStore(tags); | |
| 72 | Object.assign(f.ports.deployer, { store: store === undefined ? fake.store : store(fake.store) }); | |
| 73 | await tick(f.ports); // cut 0.7.284, known good 0.7.283 | |
| 74 | f.world.live = "0.7.284"; | |
| 75 | const line = await tick(f.ports); | |
| 76 | expect(line).toContain("0.7.284 verified live"); | |
| 77 | return { f, held: fake.held, event: f.events("release.verified_live")[0]! }; | |
| 78 | } | |
| 79 | ||
| 80 | // 0.7.265 … 0.7.284, one minute apart: the live one newest, its known good 0.7.283 next. | |
| 81 | const twenty = Array.from({ length: 20 }, (_, i) => ({ id: `0.7.${265 + i}`, at: 60_000 * (i + 1) })); | |
| 82 | ||
| 83 | test("of 20 releases the 5 oldest go, and the event names them", async () => { | |
| 84 | const { held, event } = await verifiedWith(twenty); | |
| 85 | expect(event["pruned"]).toEqual(["0.7.265", "0.7.266", "0.7.267", "0.7.268", "0.7.269"]); | |
| 86 | expect(event["pruneError"]).toBeUndefined(); | |
| 87 | expect(held.size).toBe(15); | |
| 88 | expect(held.has("0.7.284")).toBe(true); | |
| 89 | expect(held.has("0.7.283")).toBe(true); | |
| 90 | }); | |
| 91 | ||
| 92 | test("a known-good older than the newest 15 stays", async () => { | |
| 93 | // 0.7.283 was stored first; twenty newer ones were stored since. | |
| 94 | const tags = [{ id: "0.7.283", at: 1 }, ...Array.from({ length: 20 }, (_, i) => ({ id: `0.7.${300 + i}`, at: 60_000 * (i + 1) })), { id: "0.7.284", at: 9e9 }]; | |
| 95 | const { held, event } = await verifiedWith(tags); | |
| 96 | expect(event["pruned"]).toEqual(["0.7.300", "0.7.301", "0.7.302", "0.7.303", "0.7.304", "0.7.305"]); | |
| 97 | expect(held.has("0.7.283")).toBe(true); | |
| 98 | expect(held.has("0.7.284")).toBe(true); | |
| 99 | expect(held.size).toBe(16); | |
| 100 | }); | |
| 101 | ||
| 102 | test("a store that fails says why on the event, keeps what it did, and the release stays verified", async () => { | |
| 103 | const { f, event } = await verifiedWith(twenty, (s) => ({ | |
| 104 | ...s, | |
| 105 | async remove(id) { | |
| 106 | if (id === "0.7.266") throw new Error("registry said 503"); | |
| 107 | await s.remove(id); | |
| 108 | }, | |
| 109 | })); | |
| 110 | expect(event["pruned"]).toEqual(["0.7.265"]); | |
| 111 | expect(event["pruneError"]).toBe("registry said 503"); | |
| 112 | expect(f.app.db.query<{ stage: string }, []>("SELECT stage FROM keeper_release").get()!.stage).toBe("verified"); | |
| 113 | }); | |
| 114 | ||
| 115 | test("a known-good the store cannot name prunes nothing and says why", async () => { | |
| 116 | const { held, event } = await verifiedWith(twenty, (s) => ({ ...s, idOf: (ref) => (ref.endsWith(":0.7.283") ? undefined : s.idOf(ref)) })); | |
| 117 | expect(event["pruned"]).toEqual([]); | |
| 118 | expect(String(event["pruneError"])).toContain("names no stored release"); | |
| 119 | expect(held.size).toBe(20); | |
| 120 | }); | |
| 121 | ||
| 122 | test("a running image the store cannot name prunes nothing", async () => { | |
| 123 | const f = await keeperForge(); | |
| 124 | const fake = fakeStore(twenty); | |
| 125 | Object.assign(f.ports.deployer, { store: { ...fake.store, idOf: (ref: string) => (ref.startsWith("other/") ? undefined : fake.store.idOf(ref)) } }); | |
| 126 | await tick(f.ports); | |
| 127 | f.world.live = "0.7.284"; | |
| 128 | Object.assign(f.ports.deployer, { image: async () => "other/forge@sha256:abc" }); | |
| 129 | await tick(f.ports); | |
| 130 | const event = f.events("release.verified_live")[0]!; | |
| 131 | expect(event["pruned"]).toEqual([]); | |
| 132 | expect(String(event["pruneError"])).toContain("other/forge@sha256:abc"); | |
| 133 | expect(fake.held.size).toBe(20); | |
| 134 | }); | |
| 135 | ||
| 136 | test("a deployer without a store prunes nothing", async () => { | |
| 137 | const f = await keeperForge(); | |
| 138 | await tick(f.ports); | |
| 139 | f.world.live = "0.7.284"; | |
| 140 | await tick(f.ports); | |
| 141 | expect(f.events("release.verified_live")[0]!["pruned"]).toEqual([]); | |
| 142 | }); | |
| 143 | }); | |
| 144 | ||
| 145 | describe("the swapper's store", () => { | |
| 146 | const sha = (n: number) => n.toString(16).padStart(40, "0"); | |
| 147 | ||
| 148 | function swapStore(running: string) { | |
| 149 | const dataDir = mkdtempSync(join(tmpdir(), "forge-prune-")); | |
| 150 | dirs.push(dataDir); | |
| 151 | const deployer = swapper({ | |
| 152 | dataDir, | |
| 153 | port: 0, | |
| 154 | treeRoot: dataDir, | |
| 155 | version: () => "0.7.1", | |
| 156 | commit: () => running, | |
| 157 | env: {}, | |
| 158 | checkout: async (s) => ok(releaseDir(dataDir, s)), | |
| 159 | preflight: async () => ({ ok: true, line: "" }), | |
| 160 | drain: () => {}, | |
| 161 | }); | |
| 162 | return { dataDir, store: deployer.store! }; | |
| 163 | } | |
| 164 | ||
| 165 | test("lists the trees under releases/ by when each was touched, and nothing else", async () => { | |
| 166 | const { dataDir, store } = swapStore(sha(99)); | |
| 167 | for (let n = 1; n <= 3; n++) { | |
| 168 | mkdirSync(releaseDir(dataDir, sha(n)), { recursive: true }); | |
| 169 | utimesSync(releaseDir(dataDir, sha(n)), n, n); | |
| 170 | } | |
| 171 | mkdirSync(join(dataDir, "releases", "good", "0.7.1"), { recursive: true }); | |
| 172 | mkdirSync(join(dataDir, "releases", `.${sha(4)}.partial`), { recursive: true }); | |
| 173 | const found = (await store.list()).sort((a, b) => a.at - b.at); | |
| 174 | expect(found).toEqual([1, 2, 3].map((n) => ({ id: sha(n), at: n * 1000 }))); | |
| 175 | expect(store.idOf(`tree/${sha(2)}:0.7.2`)).toBe(sha(2)); | |
| 176 | expect(store.idOf("reg.test/o/forge:0.7.2")).toBeUndefined(); | |
| 177 | }); | |
| 178 | ||
| 179 | test("removes a tree by commit, never the running one or a path", async () => { | |
| 180 | const { dataDir, store } = swapStore(sha(1)); | |
| 181 | for (const n of [1, 2]) mkdirSync(releaseDir(dataDir, sha(n)), { recursive: true }); | |
| 182 | await store.remove(sha(2)); | |
| 183 | expect(existsSync(releaseDir(dataDir, sha(2)))).toBe(false); | |
| 184 | await expect(store.remove(sha(1))).rejects.toThrow("is running"); | |
| 185 | await expect(store.remove("../good")).rejects.toThrow("not a commit id"); | |
| 186 | expect(existsSync(releaseDir(dataDir, sha(1)))).toBe(true); | |
| 187 | }); | |
| 188 | ||
| 189 | test("an empty data directory lists nothing", async () => { | |
| 190 | expect(await swapStore(sha(1)).store.list()).toEqual([]); | |
| 191 | }); | |
| 192 | }); |
new above
offline
A known-good (or image()) that maps to no id is filtered out of keep here and can then be pruned. Fail closed: if any of the three refs has no id, prune nothing and return pruneError.