82c453c · 6a9be2d · 29eef47 · 6f759d3 · 36389a2 · d18d601
@@ -545,7 +545,7 @@
What the crew's tokens bought is on the board (`/board/returns`), in MCP `return
The terminal's palette has `Signed in` for anyone signed in: your browsers, tokens and this terminal's key, with when each was last seen; `d` signs the selected browser out after a confirm, as `/settings` and MCP `sessions` do, and on a token revokes it after a confirm (Keep is the default) through the same `deleteToken` action as Settings and the API, so it stops working at once (`docs/CREW.md`, "Where you are signed in").
The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases on the forge's own one-minute timer, with no model. It is off until `FORGE_RELEASE_KEEPER=on`, and the pod's service account may `get` and `patch` the forge Deployment.
The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases on the forge's own one-minute timer, with no model. It is off until `FORGE_RELEASE_KEEPER=on`, and the pod's service account may `get` and `patch` the forge Deployment. `FORGE_RELEASE_KEEPER=swap` needs no cluster: it swaps main's tree in place (`docs/features/ship-itself.md`). Its tag is the forge's own and is not pushed to an origin, so a cut is invisible to Forgejo-side tools. /releases, the terminal's Release keeper and MCP `release_keeper` show each release's short commit and its seconds from landed to live; /changelog's live line says it too.
- **Cut:** it tags main's head as `v<patch + 1>` when main has moved past a live tag, nothing is in flight, the merge queue is still, 30 minutes have passed since its last cut and no hold stands.
- **Verify:** it records the known-good version and image before the tag, sets the Deployment's image, and has the new version spend ten minutes under page load probing `/healthz` each second.
@@ -142,7 +142,7 @@
for a tunnel and a forge that sleeps.
| `quotas` | `{ key?, value?, dryRun? }` (forge admin): the per-account quotas `repositories`, `apps`, `storage` (MB), `previews`, `runs`, `checks`, each a number or null for none. `repositories` bounds repos on create, fork and copy; `apps` the apps made; `storage` the MB of repos plus app data, asked on push, create and fork; `previews` live previews; `runs` concurrent agent runs; `checks` concurrent checks. Each is asked before the thing is made or started and a forge admin is never limited; with `key` and `value` (`""` clears), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes |
| `read_file` | a file (utf-8, or base64 for binary) or a directory listing at a ref, as GET …/contents; MP4/MOV/M4A files also carry about: {kind, seconds, width?, height?, says} |  |
| `release_hold` | (admin scope) `{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent | yes |
| `release_keeper` | `{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases |  |
| `release_keeper` | `{}`: what the release keeper (#7557) did: its releases newest first (version, commit (short sha), stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason, landedToLiveSeconds — from the merge of its pull to the new version answering, null until it has (#8042)) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases |  |
| `releases` | `{shipped: true}`: what went live, for anyone: `version`, `commit`, `live` (newest first: `repo`, `number`, `title`, `at`) and `onItsWay` (landed, not yet in the running commit), the list /changelog shows. `{}`: the forge's kept releases newest first: short sha, version, the pull that landed it, when, and for the live one `live: true` and `since`; for whoever may read /releases, others get its refusal; no rollback, a person does that on the page |  |
| `repo` | `{ repo }`: one repository's description, visibility, autonomy, `audience { level, sentence }` and `provenance` (where a migrated one came from, `null` when hidden), the same as `GET /api/v1/repos/:owner/:name` |  |
| `repo_labels` | `{ repo, action?, name?, color?, id? }`: list (`list`, the default, paged by `limit` 1–200 and `cursor`, as `{ items, next_cursor }`, each label with `open`, its open issues and pulls, for a writer; a read-only token may), create (`create`, `name`, `color` six hex digits), rename or recolour (`update`, `id`, `name` and/or `color`, as the labels page) or remove (`remove`, `id`) the repository's labels, as `GET`/`POST …/labels` and `PATCH`/`DELETE …/labels/:id`; create, update and remove need an admin token and a repository admin | yes |
@@ -2,4 +2,4 @@
A §5 `.table` too wide for a narrow screen sits inside a horizontal scroller, so the page itself never scrolls sideways and only the table's figures move.
- `.table-scroll` — the wrapper `<div>` around a wide `.table`: `overflow-x: auto` with a thin scrollbar, styled in `src/web/style/components.ts`. Pair it with `.table[data-sticky-lead]` to keep each row's name in view while the figures scroll. Written by `src/web/pages/account.ts`, `src/web/pages/audit.ts`, `src/web/pages/boards-held.ts`, `src/web/pages/crew.ts`, `src/web/pages/flow-panel.ts`, `src/web/pages/passkeys.ts`, `src/web/pages/proposal-tokens.ts`, `src/web/pages/settings.ts`, `src/web/pages/spend.ts`, `src/web/pages/subscriptions.ts`, `src/web/pages/system.ts`, `fleetGraphSection`'s figures table in `src/web/fleet-graph.ts`, and `vitalsTable` in `src/web/studio/vitals.ts`.
- `.table-scroll` — the wrapper `<div>` around a wide `.table`: `overflow-x: auto` with a thin scrollbar, styled in `src/web/style/components.ts`. Pair it with `.table[data-sticky-lead]` to keep each row's name in view while the figures scroll. Written by `src/web/pages/account.ts`, `src/web/pages/audit.ts`, `src/web/pages/boards-held.ts`, `src/web/pages/crew.ts`, `src/web/pages/flow-panel.ts`, `src/web/pages/keeper-panel.ts`, `src/web/pages/passkeys.ts`, `src/web/pages/proposal-tokens.ts`, `src/web/pages/settings.ts`, `src/web/pages/spend.ts`, `src/web/pages/subscriptions.ts`, `src/web/pages/system.ts`, `fleetGraphSection`'s figures table in `src/web/fleet-graph.ts`, and `vitalsTable` in `src/web/studio/vitals.ts`.
@@ -0,0 +1,34 @@
# The forge ships itself (#8042)
One path, in the forge: **landing → on cadence the keeper takes main's tree → preflight → swap in place → verify under load → known good, or swap back and hold.** No image, no registry, no Forgejo, no Kubernetes roll, no steward.
## Switch
`FORGE_REPO=<owner>/<name> FORGE_SELF_SWAP=1 FORGE_RELEASE_KEEPER=swap` (the conductor's one line), on a kernel with `net.ipv4.tcp_migrate_req=1` and with the SSH terminal off (`FORGE_SSH_PORT=0`) until it can be handed over. `FORGE_RELEASE_KEEPER=on` is the old Deployment-patch path and stays until production has moved.
## What happens
1. `releaseOnLand` checks each landed tree out to `releases/<sha>/` (`checks/landing.ts`).
2. Every minute (`SWAP_INTERVAL_MS`) the keeper (`keeper/keeper.ts`) compares main with the newest tag. `swapper.admits` refuses a cut that could not be swapped: SSH held, the kernel setting missing, a dependency added, a red `--preflight`. Nothing is tagged or recorded then, and the line says why.
3. The cut records the known good as `tree/<commit>:<version>`, tags `v<version>` in the forge's own repository only (no origin push, so Forgejo-side tools never see the cut), and `swapper.setImage` pins the running tree under `releases/<commit>/`, starts the new tree beside this process on the same port (`release/swap.ts`, SO_REUSEPORT, `FORGE_VERSION` and `FORGE_COMMIT` set), waits until health names its commit, and drains this process.
4. The new process finds a release in flight, spends ten minutes under page load (`keeper/verify.ts`) and either calls `verified` (the tree becomes the last good release) or sets the recorded known-good tree by name, which is the same swap, and holds.
## A pod restart
`release/launch.ts` is the container's first process. `keepGood` records a swapped tree with `base`, the image version it was swapped over, in `release/good.json`. When the same image restarts, the launcher starts that tree first, and a ready older image never replaces it as the last good. A different image version is a deliberate roll (a new base): it runs, and is kept over the swapped tree. A swapped tree that does not become ready in the grace gives way to the image and the rollback card says so. The swapped process is not the launcher's child, so it writes `release/serving.json` (`release/serving.ts`) and the launcher stays up while that pid lives and signals it on SIGTERM; when it dies the container exits and restarts on the newest good tree.
## Two writers, apps and runs
The handover is the existing one (#2040): the new process listens beside the old; the old stops accepting at once, stops its timers (throughput keeper, merge queue, deliveries), lets crew runs in flight finish and land, and closes after its in-flight requests. Runs hold leases that either process renews (`failAbandonedRuns` only fails a lapsed lease), apps are restarted by the new process and the old stops its own after the linger. The pause is measured, not assumed: during the roll the old process probes `/healthz` each second and counts non-200s into `keeper_release` (`roll_probes`, `roll_non200`, `seconds_dark`), shown on /releases; `test/release-keeper-swap.test.ts` drives a client through two swaps and asserts 0 refused requests where the kernel migrates queued connections.
## Dependencies
A release's directory has no `node_modules`, and installing into it before preflight would run a package manager's scripts inside the forge. So a tree whose package.json names any dependency is not swapped: the keeper does not cut it and says a new image is needed (the image is the base). Today `package.json` has none.
## Reading it
/releases, the terminal's Release keeper and MCP `release_keeper`: version, short commit, cut time, **landed to live** (merge of the landed pull to the new version answering), known good, probes, dark seconds. /changelog: the live line adds "Landed to live in …" for the release the forge runs.
## What can be deleted once production runs it
The image-per-release path (release workflow image build and push, the `Dockerfile` `VERSION`/`COMMIT` stamping per release), `keeper/deploy.ts` `kubeDeployer` and `FORGE_KEEPER_DEPLOYMENT`/`CONTAINER`, the promote scripts and `scripts/cut-release.ts`'s image half, the droplet's `forge-release-tick` on pg-grand, and the chart's service-account patch rule.
@@ -1,11 +1,12 @@
import { mayRead } from "../../../release/gate.ts";
import { blocking, listHolds, standingHolds } from "../../../keeper/hold.ts";
import { withLandedToLive } from "../../../keeper/landed.ts";
import { listKeeperReleases } from "../../../keeper/record.ts";
import { bad, good, type Tool } from "../shared.ts";
export const tool: Tool = {
  name: "release_keeper",
  doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases",
  doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, commit (short sha), stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason, landedToLiveSeconds — from the merge of its pull to the new version answering, null until it has (#8042)) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases",
  description: "The release keeper's releases, holds and what blocks the next cut.",
  inputSchema: { type: "object", properties: {} },
  writes: false,
@@ -15,7 +16,7 @@
export const tool: Tool = {
    const db = ctx.app.db;
    return good({
      blocked: blocking(standingHolds(db)) ?? null,
      releases: listKeeperReleases(db),
      releases: withLandedToLive(db, listKeeperReleases(db)),
      holds: listHolds(db),
    });
  },
@@ -19,6 +19,12 @@
export interface Deployer {
  image(): Promise<string>;
  /** Name this image on the forge container; the Deployment rolls to it with surge. */
  setImage(image: string): Promise<void>;
  /** The reference that names `version` at `sha` (a swap names a tree, not a tag); the keeper falls back to `imageAt`. */
  refFor?(version: string, sha: string): string;
  /** Why `sha` cannot be released by this deployer, checked before anything is cut; undefined when it can. */
  admits?(sha: string): Promise<string | undefined>;
  /** `version` read well under load and is the known good: keep it for a restart. */
  verified?(version: string): void;
}
/** An image reference: registry/path:tag or @digest, no space, no flag. */
@@ -7,7 +7,9 @@
 * 1. **Cut** on cadence (`cadence.ts`): the known-good version and image
 *    are recorded, then the annotated tag `v<version>` is made on main's
 *    head — a tag, never a commit, so no gate in flight is voided (#6640).
 * 2. **Promote**: the Deployment is set to the new version's image; it
 * 2. **Promote**: the Deployment is set to the new version's image (or,
 *    with `swapper.ts` as the deployer, #8042, main's tree is swapped in beside
 *    this process on the same port: no image, no registry, no roll); it
 *    rolls with surge (`maxSurge 1, maxUnavailable 0`). The old forge
 *    probes `/healthz` each second while it still serves (`rollProbe`),
 *    so a roll that went dark says how long.
@@ -119,6 +121,7 @@
async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<string
    const numbers = { version: r.version, probes: kept.probes, pages: kept.pages, worstMs: kept.worstMs, secondsDark: kept.secondsDark, rollNon200: kept.rollNon200 };
    if (verdict.ok) {
      finish(p.db, r.id, "verified", "", p.now());
      p.deployer.verified?.(r.version);
      const lifted = liftFixed(p.db, r.version, p.now());
      p.bus.append({ kind: "release.verified_live", repoId: p.repo.id, payload: { ...numbers, holdsLifted: lifted } });
      fileDark(p, getKeeperRelease(p.db, r.id)!);
@@ -160,7 +163,7 @@
export async function tick(p: KeeperPorts): Promise<string> {
    if (flying.stage === "rolling" && p.now() - Date.parse(flying.cutAt) > ROLL_DEADLINE_MS)
      return rollBack(p, flying, `${flying.version} was not live ${ROLL_DEADLINE_MS / 3_600_000} h after its cut`);
    // A promote that did not take (the API refused, or something set the image back) is sent again.
    const want = imageAt(flying.knownGood.image, flying.version);
    const want = refFor(p, flying.knownGood.image, flying.version, flying.sha);
    if (flying.stage === "rolling" && (await p.deployer.image()) !== want) await promote(p, flying.version, want);
    return `${flying.version} is ${flying.stage}`;
  }
@@ -180,17 +183,23 @@
export async function tick(p: KeeperPorts): Promise<string> {
    ...(p.minIntervalMs === undefined ? {} : { minIntervalMs: p.minIntervalMs }),
  });
  if (!decision.cut || main === undefined) return decision.cut ? "main is unreadable" : decision.why;
  // A tree the deployer cannot release (dependencies changed, preflight red) is not cut: no tag, no row.
  const refused = await p.deployer.admits?.(main);
      
        junipercrew
        12d ago
        
      
      A refusal is not remembered: a red preflight or dependency change reruns checkout and a booted preflight every minute. Cache the refusal per sha (and log it once).
    
  if (refused !== undefined) return `not cut: ${refused}`;
  const image = await p.deployer.image();
  const knownGood = { version: p.live(), image };
  // The row first, the tag second: a crash between them leaves a `rolling` row
  // whose tag the next tick sends (`tagged`), never a tag the keeper has no row for.
  const r = recordCut(p.db, { version: decision.version, sha: main, knownGood, at: p.now() });
  const refused = await tagged(p, r);
  if (refused !== undefined) return refused;
  const sent = await promote(p, r.version, imageAt(image, r.version));
  const tagRefused = await tagged(p, r);
  if (tagRefused !== undefined) return tagRefused;
  const sent = await promote(p, r.version, refFor(p, image, r.version, main));
  return `cut ${r.version} at ${main.slice(0, 12)}; ${sent}; known good: ${knownGood.version}`;
}
const refFor = (p: KeeperPorts, image: string, version: string, sha: string): string =>
  p.deployer.refFor?.(version, sha) ?? imageAt(image, version);
/** Make the release's tag; when it is refused the cut is dropped and the line says why. */
async function tagged(p: KeeperPorts, r: KeeperRelease): Promise<string | undefined> {
  const refused = await p.tag(r.version, r.sha);
@@ -0,0 +1,55 @@
/**
 * Seconds from landed to live (#8042): when the commit a release took was
 * merged, and when the new version first answered. Read from rows the forge
 * already keeps — the landing's `release` row and its pull's merge time, the
 * keeper's `live_at` — so there is nothing new to migrate. A commit that
 * landed without a pull of this forge counts from its checkout.
 */
import type { Database } from "bun:sqlite";
import { listKeeperReleases, type KeeperRelease } from "./record.ts";
/** When `sha` landed, epoch ms: its pull's merge, else the first time a landing checked it out. */
function landedAt(db: Database, sha: string): number | undefined {
  const row = db
    .query<{ at: string | null }, [string]>(
      `SELECT COALESCE(pull.merged_at, release.created_at) AS at FROM release
       LEFT JOIN issue ON issue.repo_id = release.repo_id AND issue.number = release.pull_number
       LEFT JOIN pull ON pull.issue_id = issue.id
       WHERE release.sha = ?1 ORDER BY release.id LIMIT 1`,
    )
    .get(sha);
  const ms = row?.at == null ? NaN : Date.parse(row.at);
  return Number.isFinite(ms) ? ms : undefined;
}
/** Whole seconds from landing to the new version answering; undefined while it has not, or when the landing is unknown. */
export function landedToLive(db: Database, r: Pick<KeeperRelease, "sha" | "liveAt">): number | undefined {
  if (r.liveAt === null) return undefined;
  const landed = landedAt(db, r.sha);
  const live = Date.parse(r.liveAt);
  if (landed === undefined || !Number.isFinite(live)) return undefined;
  return Math.max(0, Math.round((live - landed) / 1000));
}
/** "1 min 12 s", "45 s". */
export function secondsWords(s: number): string {
  return s < 60 ? `${s} s` : `${Math.floor(s / 60)} min ${s % 60} s`;
}
      
        sentinelcrew
        12d ago
        
      
      Expose these figures to the terminal and MCP too (tui keeper screen, release_keeper tool), each with a test.
    
/** The newest release that went live and stayed (or is being verified), with its landed-to-live seconds. */
export function newestLive(db: Database): { version: string; sha: string; seconds: number } | undefined {
  for (const r of listKeeperReleases(db, 10)) {
    if (r.stage === "rolled_back") continue;
    const seconds = landedToLive(db, r);
    if (seconds !== undefined) return { version: r.version, sha: r.sha, seconds };
  }
  return undefined;
}
/** A keeper release with its short commit and landed-to-live seconds (null until it answered): what the card, the terminal and MCP show. */
export type LandedRelease = KeeperRelease & { readonly commit: string; readonly landedToLiveSeconds: number | null };
export function withLandedToLive(db: Database, releases: readonly KeeperRelease[]): LandedRelease[] {
  return releases.map((r) => ({ ...r, commit: r.sha.slice(0, 7), landedToLiveSeconds: landedToLive(db, r) ?? null }));
}
@@ -0,0 +1,195 @@
/**
 * The keeper's hands without a cluster (#8042): the `Deployer` that swaps a
 * tree in place instead of patching a Deployment's image. A reference is
 * `tree/<sha>:<version>`; setting it checks the commit out under
 * `releases/<sha>/`, starts it beside this process on the same port
 * (`release/swap.ts`, SO_REUSEPORT, handed over once health names the commit),
 * and drains this process. Nothing here needs a registry, a Forgejo, a
 * Kubernetes API or a person's machine.
 *
 * - **Known good**: before the first swap away from a tree, that tree is
 *   pinned under `releases/<commit>/` (an image's own tree included), so the
 *   rollback is the same swap to a tree that was already live.
 * - **Dependencies**: a release is swapped in only when its package.json
 *   names no dependency. Its directory has no node_modules, and installing
 *   into it before preflight would run a package manager's scripts inside the
 *   forge; so a tree that adds one is not cut (`admits`), and the line says a
 *   new image is needed. The image only has to be a base.
 * - **Rollback**: a tree that has served is swapped back to only when it
 *   knows every migration the live database has applied (the guard of
 *   `--rollback`, `release/rollback.ts`); an older tree is refused and the
 *   line says so, since this forge has no down-migration.
 * - **Refusals**: a commit refused (dependency, red preflight) is remembered
 *   per sha and logged once, so the next minute does not check it out and boot
 *   a preflight again.
 * - **Restart**: once verified, `verified` keeps the tree as the last good
 *   release over this image's version (`release/launch.ts`), so a pod
 *   restart comes back on it, not on the older image's version.
 */
import { cpSync, existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { err, type Result } from "../kernel/result.ts";
import { releaseDir, type Verdict } from "../release/checkout.ts";
import { keepGood } from "../release/launch.ts";
import { liveSchema, releaseSchema } from "../release/rollback.ts";
import { beginSwap, endSwap, startRelease, swapBlocked } from "../release/swap.ts";
import type { Deployer } from "./deploy.ts";
const REF = /^tree\/([0-9a-f]{40}):([0-9A-Za-z][0-9A-Za-z.+_-]{0,63})$/;
const TREE_PARTS = ["src", "package.json", "tsconfig.json"];
/** A tree that has served: a rollback target, not a candidate, so it is not preflighted again. */
const LIVE_MARK = ".was-live";
const DEP_FIELDS = ["dependencies", "devDependencies", "optionalDependencies", "peerDependencies"];
export interface SwapperInput {
  readonly dataDir: string;
  readonly port: number;
  /** The tree this process runs from: the directory holding `src/main.ts`. */
  readonly treeRoot: string;
  readonly version: () => string;
  readonly commit: () => string | null;
  readonly env: Record<string, string | undefined>;
  /** Write `sha`'s tree to `releases/<sha>/`; the directory. */
  readonly checkout: (sha: string) => Promise<Result<string, string>>;
  readonly preflight: (dir: string) => Promise<Verdict>;
  /** Let go of the port and drain, once the new tree is well. */
  readonly drain: () => void;
  readonly probe?: Readonly<Record<string, string>>;
  readonly budgetMs?: number;
  readonly now?: () => number;
}
/** Why `dir`'s package.json cannot run without an install; undefined when it names no dependency. */
export function dependencyChange(dir: string): string | undefined {
  let pkg: Record<string, unknown>;
  try {
    pkg = JSON.parse(readFileSync(join(dir, "package.json"), "utf8")) as Record<string, unknown>;
  } catch {
    return "the tree has no readable package.json";
  }
  for (const field of DEP_FIELDS) {
    const deps = pkg[field];
    if (typeof deps === "object" && deps !== null && Object.keys(deps).length > 0)
      return `${field} names ${Object.keys(deps).slice(0, 3).join(", ")}: a dependency change needs a new image, which is the base this swaps over`;
  }
  return undefined;
}
export function swapper(i: SwapperInput): Deployer {
  const now = i.now ?? Date.now;
  /** Set once this process has handed over: what it then reports as running, so the keeper does not promote again. */
  let handedTo: string | undefined;
  const admitted = new Set<string>();
  /** Commits refused for good (their tree does not change), with the reason. */
  const refused = new Map<string, string>();
  const here = (): string => {
    const commit = i.commit();
    if (commit === null || !/^[0-9a-f]{40}$/.test(commit)) throw new Error("the running commit is not stamped, so there is no known good tree to name");
    return `tree/${commit}:${i.version()}`;
  };
  /** Keep the running tree where a rollback finds it. */
  function pin(): void {
    const commit = i.commit();
    if (commit === null || !/^[0-9a-f]{40}$/.test(commit)) return;
    const target = releaseDir(i.dataDir, commit);
    if (existsSync(target)) {
      writeFileSync(join(target, LIVE_MARK), "");
      return;
    }
    const staging = `${target}.pin-${process.pid}.partial`;
    rmSync(staging, { recursive: true, force: true });
    mkdirSync(staging, { recursive: true });
    for (const part of TREE_PARTS)
      if (existsSync(join(i.treeRoot, part))) cpSync(join(i.treeRoot, part), join(staging, part), { recursive: true });
    writeFileSync(join(staging, LIVE_MARK), "");
    renameSync(staging, target);
  }
  const refuse = (sha: string, why: string): Result<string, string> => {
    refused.set(sha, why);
    console.error(`keeper: ${sha.slice(0, 12)} is not swapped in: ${why}`);
    return err(why);
  };
  async function prepare(sha: string, forward: boolean): Promise<Result<string, string>> {
    const before = forward ? refused.get(sha) : undefined;
    if (before !== undefined) return err(before);
    const out = await i.checkout(sha);
    if (!out.ok) return err(`checkout failed: ${out.error}`);
    if (!forward) return out;
    if (admitted.has(sha)) return out;
    const deps = dependencyChange(out.value);
    if (deps !== undefined) return refuse(sha, deps);
    const verdict = await i.preflight(out.value);
    if (!verdict.ok) return refuse(sha, `preflight failed: ${verdict.line}`);
    admitted.add(sha);
    return out;
  }
  /** The guard of `--rollback`: a tree that knows fewer migrations than the live database has applied does not run on it. */
  async function schemaRefusal(sha: string, dir: string): Promise<string | undefined> {
    const known = await releaseSchema(dir);
    const live = liveSchema(i.dataDir);
    if (known === undefined) return live === 0 ? undefined : `${sha} has no readable schema; not rolling back to it`;
    return known < live ? `${sha} predates schema ${live}; rolling back would need a down-migration` : undefined;
  }
  return {
    async image() {
      return handedTo ?? here();
    },
    refFor: (version, sha) => `tree/${sha}:${version}`,
    async admits(sha) {
      // Nothing is cut that this process could not swap in: the same refusals as the swap itself.
      const blocked = swapBlocked();
      if (blocked !== undefined) return blocked;
      const r = await prepare(sha, true);
      return r.ok ? undefined : r.error;
    },
    async setImage(ref) {
      const m = REF.exec(ref);
      if (m === null) throw new Error("not a tree reference");
      const [, sha, version] = m as unknown as [string, string, string];
      if (handedTo === ref) return;
      // A tree that has served (pinned when it was swapped away from) is a rollback and is not checked again; any other is admitted, as at the cut.
      const backward = existsSync(join(releaseDir(i.dataDir, sha), LIVE_MARK));
      const dir = await prepare(sha, !backward);
      if (!dir.ok) throw new Error(dir.error);
      if (backward) {
        const stopped = await schemaRefusal(sha, dir.value);
        if (stopped !== undefined) throw new Error(stopped);
      }
      const turn = beginSwap();
      if (!turn.ok) throw new Error(turn.line);
      let swapped = false;
      try {
        pin();
        const env = { ...i.env, FORGE_DATA: i.dataDir };
        const started = await startRelease({
          dir: dir.value,
          sha,
          port: i.port,
          env,
          version,
          ...(i.probe === undefined ? {} : { probe: i.probe }),
          ...(i.budgetMs === undefined ? {} : { budgetMs: i.budgetMs }),
        });
        if (!started.ok) throw new Error(started.line);
        swapped = true;
      } finally {
        endSwap(swapped);
      }
      handedTo = ref;
      i.drain();
    },
    verified(version) {
      const commit = i.commit();
      const base = i.env["FORGE_BASE_VERSION"] || i.env["FORGE_VERSION"] || null;
      if (commit === null || base === null) return;
      keepGood(i.dataDir, i.treeRoot, version, commit, now(), base);
    },
  };
}
@@ -3,12 +3,17 @@
 * repository for main and the tags, the merge queue for "mid-landing", the
 * kube API for the Deployment, and the public URL for the probes.
 *
 * On when `FORGE_RELEASE_KEEPER=on`, `FORGE_REPO` names a repository here,
 * and the forge runs in a cluster. The Deployment and its container are
 * `FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER` (both default
 * `forge`); the pod's service account needs `get` and `patch` on it.
 * Two modes. `FORGE_RELEASE_KEEPER=on`, with `FORGE_REPO` naming a
 * repository here and the forge in a cluster, patches a Deployment: its name
 * and container are `FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER`
 * (both default `forge`), and the pod's service account needs `get` and
 * `patch` on it.
 *
 * `FORGE_RELEASE_KEEPER=swap` (#8042) needs no cluster and wants
 * `FORGE_SELF_SWAP=1`: the keeper swaps main's tree in place (`swapper.ts`).
 */
import { join } from "node:path";
import type { App } from "../app.ts";
import { originPush } from "../actions/branches.ts";
import { serviceAccount } from "../cluster/jobs.ts";
@@ -22,7 +27,10 @@
import { repoDir } from "../git/repos.ts";
import { ownProbeHeaders } from "../kernel/probe.ts";
import { running } from "../kernel/version.ts";
import { compareVersions, readTags, recordFor } from "../release/shipped.ts";
import { checkoutRelease, preflightRelease } from "../release/checkout.ts";
import { handover } from "../release/swap.ts";
import { kubeDeployer } from "./deploy.ts";
import { swapper } from "./swapper.ts";
import type { KeeperPorts } from "./keeper.ts";
import { timedGet, verifyUnderLoad } from "./verify.ts";
@@ -31,16 +39,26 @@
const OID = /^[0-9a-f]{40}$/;
const VERSION = /^\d+\.\d+\.\d+$/;
const ZERO = "0".repeat(40);
export type KeeperEnv = { on: true; deployment: string; container: string } | { on: false; why: string };
export type KeeperEnv =
  | { on: true; mode: "kube"; deployment: string; container: string }
  | { on: true; mode: "swap" }
  | { on: false; why: string };
/** In swap mode a release is only the swap, so the cadence is a minute, not half an hour. */
export const SWAP_INTERVAL_MS = 60_000;
/** Whether the keeper runs, from the environment. */
export function keeperEnv(env: Record<string, string | undefined>): KeeperEnv {
  if (env.FORGE_RELEASE_KEEPER === "swap") {
    if (env.FORGE_SELF_SWAP !== "1") return { on: false, why: "FORGE_RELEASE_KEEPER=swap needs FORGE_SELF_SWAP=1" };
    return { on: true, mode: "swap" };
  }
  if (env.FORGE_RELEASE_KEEPER !== "on") return { on: false, why: "FORGE_RELEASE_KEEPER is not on" };
  if (!env.KUBERNETES_SERVICE_HOST) return { on: false, why: "not running in a cluster" };
  const deployment = env.FORGE_KEEPER_DEPLOYMENT || "forge";
  const container = env.FORGE_KEEPER_CONTAINER || "forge";
  if (!LABEL.test(deployment) || !LABEL.test(container)) return { on: false, why: "FORGE_KEEPER_DEPLOYMENT and FORGE_KEEPER_CONTAINER must be DNS labels" };
  return { on: true, deployment, container };
  return { on: true, mode: "kube", deployment, container };
}
/** The first active site admin that is a person: the account the page load signs in as. */
@@ -56,17 +74,33 @@
export function loadPages(repo: Pick<Repo, "ownerName" | "name">): string[] {
  return ["/boards", `/${encodeURIComponent(repo.ownerName)}/${encodeURIComponent(repo.name)}`, "/explore", "/"];
}
export function keeperPorts(app: App, input: { forgeUrl: string; deployment: string; container: string }): KeeperPorts | string {
export function keeperPorts(app: App, input: { forgeUrl: string; deployment: string; container: string; swap?: boolean }): KeeperPorts | string {
  const named = app.config.forgeRepo;
  if (named === undefined) return "FORGE_REPO is not set";
  const [owner = "", name = ""] = named.split("/");
  const repo = getRepo(app.db, owner, name);
  if (repo === undefined) return `FORGE_REPO names ${named}, which is no repository on this forge`;
  const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
  const deployer = kubeDeployer({ credentials: serviceAccount(), deployment: input.deployment, container: input.container });
  const deployer =
    input.swap === true
      ? swapper({
          dataDir: app.config.dataDir,
          port: app.config.port ?? 0,
          treeRoot: join(import.meta.dir, "..", ".."),
          version: () => running().version,
          commit: () => running().commit,
          env: process.env,
          checkout: (sha) => checkoutRelease(app.config.dataDir, dir, sha),
          preflight: (tree) => preflightRelease(tree, app.config.dataDir),
          drain: () => handover.drain?.(),
          probe: ownProbeHeaders(app.config.dataDir),
          ...(app.config.swapBudgetMs === undefined ? {} : { budgetMs: app.config.swapBudgetMs }),
        })
      : kubeDeployer({ credentials: serviceAccount(), deployment: input.deployment, container: input.container });
  const probeHeaders = ownProbeHeaders(app.config.dataDir);
  const at = (version: string) => runGit(["rev-parse", "--verify", "--quiet", `refs/tags/v${version}^{commit}`], { cwd: dir });
  return {
    ...(input.swap === true ? { minIntervalMs: SWAP_INTERVAL_MS } : {}),
    db: app.db,
    bus: app.bus,
    repo: { id: repo.id, ownerId: repo.ownerId },
@@ -99,7 +133,8 @@
export function keeperPorts(app: App, input: { forgeUrl: string; deployment: str
      if (!made.ok) return made.error.stderr || "git refused the tag";
      const tagOid = await runGit(["rev-parse", `refs/tags/v${version}`], { cwd: dir });
      const owner = getUserById(app.db, repo.ownerId);
      if (!tagOid.ok || owner === undefined) return undefined;
      // A swap needs no origin: the tag is the forge's own record of the release.
      if (!tagOid.ok || owner === undefined || input.swap === true) return undefined;
      // A tag the origin did not take is taken back here too: the keeper drops the cut, and a
      // tag left behind would be the newest, never live, and stall the cadence.
      const untag = async (why: string) => (await runGit(["tag", "--delete", `v${version}`], { cwd: dir }), why);
@@ -658,7 +658,12 @@
if (app.config.forgeRepo) {
// known-good image and hold. Off unless FORGE_RELEASE_KEEPER=on in a cluster (keeper/wire.ts).
const keeperOn = keeperEnv(process.env);
if (keeperOn.on) {
  const ports = keeperPorts(app, { forgeUrl, deployment: keeperOn.deployment, container: keeperOn.container });
  const ports = keeperPorts(
    app,
    keeperOn.mode === "swap"
      ? { forgeUrl, deployment: "", container: "", swap: true }
      : { forgeUrl, deployment: keeperOn.deployment, container: keeperOn.container },
  );
  if (typeof ports === "string") app.log.warn("the release keeper is off", { why: ports });
  else {
    let said = "";
@@ -17,6 +17,15 @@
 * seconds after its crash, and a hang costs the grace (150 s, inside the
 * chart's three-minute startup probe).
 *
 * A swapped release (#8042) is a good tree too: the keeper that verified it
 * keeps it with `base`, the image version it was swapped over. On a restart
 * of that same image the launcher starts the swapped tree first, so an older
 * image never silently rolls production back; only a different image (a
 * deliberate roll) takes precedence, and a swapped tree that is not ready in
 * the grace falls back to the image, recorded as any rollback is. The
 * process that took over is not the launcher's child; `serving.ts` names it
 * and the launcher stays up as long as it does.
 *
 * Node built-ins only, on purpose: it is the one part of a bad release that
 * has to work.
 */
@@ -25,6 +34,7 @@
import { cpSync, existsSync, mkdirSync, readdirSync, readFileSync, renameSync, r
import { dirname, join } from "node:path";
import type { Subprocess } from "bun";
import { hkdfSync } from "node:crypto";
import { clearServing, follow, signalServing } from "./serving.ts";
export const READY_GRACE_MS = 150_000;
const POLL_MS = 500;
@@ -38,6 +48,8 @@
export interface Good {
  readonly version: string;
  readonly commit: string | null;
  readonly at: string;
  /** The image version this was swapped in over; null for a tree the launcher itself started. */
  readonly base: string | null;
}
export interface RolledBack {
@@ -78,13 +90,27 @@
export function lastGood(dataDir: string): Good | undefined {
  const version = g?.["version"];
  if (typeof version !== "string" || !VERSION.test(version) || typeof g?.["at"] !== "string") return undefined;
  if (!existsSync(join(goodTree(dataDir, version), "src", "main.ts"))) return undefined;
  return { version, commit: typeof g["commit"] === "string" ? g["commit"] : null, at: g["at"] };
  const base = g["base"];
  return {
    version,
    commit: typeof g["commit"] === "string" ? g["commit"] : null,
    at: g["at"],
    base: typeof base === "string" && VERSION.test(base) ? base : null,
  };
}
/** Keep `tree` as the last good release; drop the one it replaces and any standing rollback. */
export function keepGood(dataDir: string, tree: string, version: string, commit: string | null, now = Date.now()): void {
  if (!VERSION.test(version)) return;
  if (lastGood(dataDir)?.version !== version) {
export function keepGood(
  dataDir: string,
  tree: string,
  version: string,
  commit: string | null,
  now = Date.now(),
  base: string | null = null,
): void {
  if (!VERSION.test(version) || (base !== null && !VERSION.test(base))) return;
  const kept = lastGood(dataDir);
  if (kept?.version !== version || kept.commit !== commit || kept.base !== base) {
    const target = goodTree(dataDir, version);
    const staging = `${target}.partial-${process.pid}`;
    rmSync(staging, { recursive: true, force: true });
@@ -93,7 +119,7 @@
export function keepGood(dataDir: string, tree: string, version: string, commit:
      if (existsSync(join(tree, part))) cpSync(join(tree, part), join(staging, part), { recursive: true });
    rmSync(target, { recursive: true, force: true });
    renameSync(staging, target);
    writeJson(goodFile(dataDir), { version, commit, at: new Date(now).toISOString() });
    writeJson(goodFile(dataDir), { version, commit, at: new Date(now).toISOString(), base });
  }
  for (const name of readdirSync(join(releaseDir(dataDir), "good")))
    if (name !== version) rmSync(join(releaseDir(dataDir), "good", name), { recursive: true, force: true });
@@ -190,6 +216,8 @@
function start(input: LaunchInput, tree: string, version: string, commit: string
  for (const [k, v] of Object.entries(input.env)) if (v !== undefined) env[k] = v;
  env["FORGE_VERSION"] = version;
  env["FORGE_COMMIT"] = commit ?? "";
  // What a swap in this process's descendants is over: the image, not the tree running now.
  env["FORGE_BASE_VERSION"] = input.version;
  const child = Bun.spawn([process.execPath, join(tree, "src", "main.ts")], {
    cwd: tree,
    env,
@@ -201,42 +229,65 @@
function start(input: LaunchInput, tree: string, version: string, commit: string
  return child;
}
/** Wait for `child` to answer health as `version`: undefined once it does, else why it did not. */
async function becomesReady(input: LaunchInput, child: Subprocess, version: string): Promise<string | undefined> {
  const now = input.now ?? Date.now;
  const grace = input.graceMs ?? READY_GRACE_MS;
  const began = now();
  for (;;) {
    if (input.stopping?.()) return "asked to stop before it was ready";
    if (gone(child)) return `it exited with ${child.exitCode ?? child.signalCode} before it was ready`;
    if ((await versionAt(input.port, probeHeaders(input.dataDir))) === version) return undefined;
    if (now() - began >= grace) return `not ready within ${Math.round(grace / 1000)}s`;
    await Bun.sleep(POLL_MS);
  }
}
/**
 * Start this image's forge and answer once it is ready (`release`), was
 * replaced by the last good one (`rollback`), or neither runs (`none`).
 * The child returned is the one to wait on.
 * The child returned is the one to wait on. A swapped tree kept over this
 * very image goes first (see the header).
 */
export async function launch(input: LaunchInput): Promise<Launched> {
  const now = input.now ?? Date.now;
  const grace = input.graceMs ?? READY_GRACE_MS;
  const release = start(input, input.tree, input.version, input.commit);
  const began = now();
  let why: string;
  for (;;) {
  const swapped = lastGood(input.dataDir);
  let passed: { version: string; why: string } | undefined;
  if (swapped !== undefined && swapped.base === input.version && swapped.version !== input.version) {
    const first = start(input, goodTree(input.dataDir, swapped.version), swapped.version, swapped.commit);
    const failed = await becomesReady(input, first, swapped.version);
    if (failed === undefined)
      return { ran: "release", child: first, line: `${swapped.version} is ready: the swapped release kept over image ${input.version}` };
    await stop(first);
    if (input.stopping?.()) return { ran: "none", line: "asked to stop before it was ready" };
    if (gone(release)) {
      why = `it exited with ${release.exitCode ?? release.signalCode} before it was ready`;
      break;
    passed = { version: swapped.version, why: failed };
  }
  const release = start(input, input.tree, input.version, input.commit);
  const why = await becomesReady(input, release, input.version);
  if (why === undefined) {
    try {
      keepGood(input.dataDir, input.tree, input.version, input.commit, now());
    } catch (error) {
      // Serving matters more than keeping: the next version falls back to the one before.
      return { ran: "release", child: release, line: `${input.version} is ready; not kept: ${String(error).slice(0, 200)}` };
    }
    if ((await versionAt(input.port, probeHeaders(input.dataDir))) === input.version) {
      try {
        keepGood(input.dataDir, input.tree, input.version, input.commit, now());
      } catch (error) {
        // Serving matters more than keeping: the next version falls back to the one before.
        return { ran: "release", child: release, line: `${input.version} is ready; not kept: ${String(error).slice(0, 200)}` };
      }
      return { ran: "release", child: release, line: `${input.version} is ready and kept as the last good release` };
    // The swapped tree that would not start: the image runs in its place, and the forge says so.
    if (passed !== undefined) {
      writeJson(rolledFile(input.dataDir), {
        from: passed.version,
        to: input.version,
        at: new Date(now()).toISOString(),
        why: passed.why,
        paged: false,
      } satisfies RolledBack);
      return { ran: "rollback", child: release, line: `${passed.version}: ${passed.why}; the image ${input.version} runs instead` };
    }
    if (now() - began >= grace) {
      why = `not ready within ${Math.round(grace / 1000)}s`;
      break;
    }
    await Bun.sleep(POLL_MS);
    return { ran: "release", child: release, line: `${input.version} is ready and kept as the last good release` };
  }
  await stop(release);
  if (input.stopping?.()) return { ran: "none", line: "asked to stop before it was ready" };
  const good = lastGood(input.dataDir);
  if (good === undefined || good.version === input.version)
  if (good === undefined || good.version === input.version || passed?.version === good.version)
    return { ran: "none", line: `${input.version}: ${why}; no earlier release is kept to roll back to` };
  writeJson(rolledFile(input.dataDir), {
    from: input.version,
@@ -256,10 +307,14 @@
if (import.meta.main) {
  const graceS = Number(env["FORGE_READY_GRACE_S"] ?? "");
  let current: Subprocess | undefined;
  let stopping = false;
  const dataHere = env["FORGE_DATA"] || "data";
  // Whatever an earlier container left names processes that are not ours.
  clearServing(dataHere);
  for (const signal of ["SIGTERM", "SIGINT"] as const)
    process.on(signal, () => {
      stopping = true;
      current?.kill(signal);
      signalServing(dataHere, signal);
    });
  const launched = await launch({
    tree: join(import.meta.dir, "..", ".."),
@@ -275,5 +330,10 @@
if (import.meta.main) {
    stopping: () => stopping,
  });
  console.error(`launch: ${launched.line}`);
      
        junipercrew
        12d ago
        
      
      Redundant alias of dataHere; use dataHere.
    
  process.exit(launched.ran === "none" ? (stopping ? 0 : 1) : await launched.child.exited);
  if (launched.ran === "none") process.exit(stopping ? 0 : 1);
  const code = await launched.child.exited;
  // A swapped release took over and the process we started has drained: this container is that
  // process's now, and goes when it does (the pod then restarts on the newest good tree).
  await follow(dataHere, launched.child.pid);
  process.exit(code);
}
@@ -0,0 +1,72 @@
/**
 * Who serves the port now (#8042). A swapped release is started by the
 * process it replaces, so it is not the launcher's child; the launcher
 * (`launch.ts`) would see its own child drain and exit, and a container whose
 * first process exits takes the release with it. So the process that took
 * over writes `release/serving.json` ({pid, sha}), and the launcher stays up
 * while that pid lives and passes it SIGTERM when it is told to stop. The
 * launcher clears the file on start, so a pid is only ever one this
 * container started. Node built-ins only, as `launch.ts` is.
 */
import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
export interface Serving {
  readonly pid: number;
  readonly sha: string;
}
const file = (dataDir: string): string => join(dataDir, "release", "serving.json");
export function writeServing(dataDir: string, s: Serving): void {
  const path = file(dataDir);
  mkdirSync(dirname(path), { recursive: true });
  const tmp = `${path}.tmp-${process.pid}`;
  writeFileSync(tmp, JSON.stringify(s));
  renameSync(tmp, path);
}
export function readServing(dataDir: string): Serving | undefined {
  try {
    const v = JSON.parse(readFileSync(file(dataDir), "utf8")) as { pid?: unknown; sha?: unknown };
    if (!Number.isInteger(v.pid) || (v.pid as number) <= 1 || typeof v.sha !== "string") return undefined;
    return { pid: v.pid as number, sha: v.sha };
  } catch {
    return undefined;
  }
}
export function clearServing(dataDir: string): void {
  if (existsSync(file(dataDir))) rmSync(file(dataDir), { force: true });
}
export function alive(pid: number): boolean {
  try {
    process.kill(pid, 0);
    return true;
  } catch (error) {
    // EPERM: it exists and is not ours to signal.
    return (error as NodeJS.ErrnoException).code === "EPERM";
  }
}
/** Pass `signal` to the process that took over from `ownPid`, if one lives. */
export function signalServing(dataDir: string, signal: NodeJS.Signals, ownPid?: number): void {
  const s = readServing(dataDir);
  if (s === undefined || s.pid === ownPid || !alive(s.pid)) return;
  try {
    process.kill(s.pid, signal);
  } catch {
    // Gone between the look and the signal.
  }
}
/** Return once the process serving is `ownPid` again or none lives: the swapped release is gone. */
export async function follow(dataDir: string, ownPid: number, pollMs = 1000): Promise<void> {
  for (;;) {
    const s = readServing(dataDir);
    if (s === undefined || s.pid === ownPid || !alive(s.pid)) return;
    await new Promise((r) => setTimeout(r, pollMs));
  }
}
@@ -18,6 +18,7 @@
import { existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { writeServing } from "./serving.ts";
export const SWAP_BUDGET_MS = 60_000;
const POLL_MS = 250;
@@ -32,6 +33,8 @@
export interface SwapInput {
  /** The environment the new forge starts with: the old one's, as it is. */
  env: Record<string, string | undefined>;
  budgetMs?: number;
  /** The version the new process runs as (`FORGE_VERSION`); the environment's own when absent. */
  version?: string;
  /** Sent with each health read (`kernel/probe.ts`), so the answer names its commit. */
  probe?: Readonly<Record<string, string>>;
}
@@ -62,6 +65,7 @@
export async function startRelease(input: SwapInput): Promise<Swapped> {
  const env: Record<string, string> = {};
  for (const [k, v] of Object.entries(input.env)) if (v !== undefined) env[k] = v;
  env["FORGE_COMMIT"] = input.sha;
  if (input.version !== undefined) env["FORGE_VERSION"] = input.version;
  env["PORT"] = String(input.port);
  env["FORGE_SELF_SWAP"] = "1";
  const proc = Bun.spawn([process.execPath, entry], {
@@ -81,6 +85,8 @@
export async function startRelease(input: SwapInput): Promise<Swapped> {
      return { ok: false, line: `the new process exited with ${exited} before it was healthy` };
    if (healthyAs(await readHealth(input.port, input.probe ?? {}), input.sha)) {
      proc.unref();
      // The launcher of a container waits on this pid now (`serving.ts`).
      if (env["FORGE_DATA"]) writeServing(env["FORGE_DATA"], { pid: proc.pid, sha: input.sha });
      return { ok: true, pid: proc.pid };
    }
    await Bun.sleep(POLL_MS);
@@ -164,22 +170,21 @@
let underWay = false;
export function beginSwap(): { ok: true } | { ok: false; line: string; tell: boolean } {
  if (underWay)
    return { ok: false, line: "a swap is already under way or done in this process", tell: false };
  if (handover.ssh === true)
    return {
      ok: false,
      line: "this process holds the SSH terminal, which a new process cannot take over yet; restart the forge to run this tree",
      tell: true,
    };
  if (!migratesQueuedConnections())
    return {
      ok: false,
      line: "the kernel would reset connections queued on the old listener when it closes; set net.ipv4.tcp_migrate_req=1 (Linux 5.14+) to swap, or restart the forge to run this tree",
      tell: true,
    };
  const blocked = swapBlocked();
  if (blocked !== undefined) return { ok: false, line: blocked, tell: true };
  underWay = true;
  return { ok: true };
}
/** Why this process cannot swap at all (SSH held, queued connections not migrated); undefined when it can. */
export function swapBlocked(): string | undefined {
  if (handover.ssh === true)
    return "this process holds the SSH terminal, which a new process cannot take over yet; restart the forge to run this tree";
  if (!migratesQueuedConnections())
    return "the kernel would reset connections queued on the old listener when it closes; set net.ipv4.tcp_migrate_req=1 (Linux 5.14+) to swap, or restart the forge to run this tree";
  return undefined;
}
/** A swap that failed gives the turn back; one that worked keeps it, for this process is leaving. */
export function endSwap(swapped: boolean): void {
  if (!swapped) underWay = false;
@@ -1,14 +1,16 @@
/**
 * The release keeper in the terminal (#7557), the /releases card's read:
 * what holds the next cut, then each release the keeper cut with its stage,
 * the known-good version it rolls back to and what the probes counted.
 * its short commit, landed to live (#8042), the known-good version it rolls
 * back to and what the probes counted.
 * Gated by `release/gate.ts` as the page is. `h` holds releases with a
 * reason and `h` with `lift N` lifts an owner's hold, through the same
 * functions the card and MCP `release_hold` call.
 */
import { blocking, holdAs, liftAs, standingHolds } from "../../keeper/hold.ts";
import { listKeeperReleases, type KeeperRelease, type Stage } from "../../keeper/record.ts";
import { secondsWords, withLandedToLive, type LandedRelease } from "../../keeper/landed.ts";
import { listKeeperReleases, type Stage } from "../../keeper/record.ts";
import { mayRead, mayRollBack } from "../../release/gate.ts";
import type { Ctx, Screen } from "../app/app.ts";
import { binding } from "../app/keys.ts";
@@ -29,22 +31,24 @@
const STAGE: Record<Stage, { label: string; tone: Tone }> = {
export interface KeeperRead {
  /** What holds the next cut, or the line that says nothing does. */
  readonly blocked: string;
  readonly releases: readonly KeeperRelease[];
  readonly releases: readonly LandedRelease[];
}
/** The card's content for this person; undefined for anyone the page would 404. */
export function readKeeper(f: Session): KeeperRead | undefined {
  if (!mayRead(f.app, f.user)) return undefined;
  return { blocked: blocking(standingHolds(f.db)) ?? "Nothing holds the next cut.", releases: listKeeperReleases(f.db) };
  return { blocked: blocking(standingHolds(f.db)) ?? "Nothing holds the next cut.", releases: withLandedToLive(f.db, listKeeperReleases(f.db)) };
}
export function keeperRows(releases: readonly KeeperRelease[]): ListRow[] {
export function keeperRows(releases: readonly LandedRelease[]): ListRow[] {
  return releases.map((r) => ({
    title: r.version,
    tone: STAGE[r.stage].tone,
    meta: metaOf(
      STAGE[r.stage].label,
      r.commit,
      `cut ${ago(r.cutAt)}`,
      r.landedToLiveSeconds === null ? "not live yet" : `landed to live ${secondsWords(r.landedToLiveSeconds)}`,
      `known good ${r.knownGood.version}`,
      `roll ${r.rollNon200}/${r.rollProbes} non-200`,
      `${r.probes} probes, ${r.pages} pages`,
@@ -8,6 +8,7 @@
import { notePhase, takeSessionTime, timePhaseAsync } from "../../core/route_timing.ts";
import type { Ctx } from "../../http/context.ts";
import { newestLive, secondsWords } from "../../keeper/landed.ts";
import { dayHeading, dayOf, daysJs, shippedToday } from "../../release/days.ts";
import { readLandings, type Landing, type Landings } from "../../release/landings.ts";
import { liveRegionsJs } from "../client.ts";
@@ -109,7 +110,13 @@
export const localDaysJs = /* js */ `
`;
/** The region: the top line, what is on its way, and what went live by day. */
export function shippedRegion(read: Landings, choice: string, now: Date = new Date(), zone = "UTC"): Html {
export function shippedRegion(
  read: Landings,
  choice: string,
  now: Date = new Date(),
  zone = "UTC",
  toLive?: { version: string; seconds: number },
): Html {
  const keep = (l: Landing) => choice === "all" || l.repo === choice;
  const live = read.live.filter(keep);
  const onItsWay = read.onItsWay.filter(keep);
@@ -120,7 +127,7 @@
export function shippedRegion(read: Landings, choice: string, now: Date = new Da
  const version =
    read.version !== "dev" ? (commit === null ? read.version : html`${read.version} · ${commit}`) : (commit ?? "a development build");
  const top = html`<p class="small muted" id="shipped-top">
    Live: ${version}. <span id="shipped-today">${shippedToday(today, zone === "UTC")}</span>
    Live: ${version}.${toLive !== undefined && toLive.version === read.version ? html` Landed to live in ${secondsWords(toLive.seconds)}.` : null} <span id="shipped-today">${shippedToday(today, zone === "UTC")}</span>
  </p>`;
  const way =
    onItsWay.length === 0
@@ -147,7 +154,7 @@
export async function changelogPage(ctx: Ctx): Promise<Response> {
    ...others.map((r) => ({ label: r, href: `/changelog?repo=${encodeURIComponent(r)}`, current: choice === r })),
  ]);
  const body = html`${pageHead("Changelog", { sub: "What went live, as it happens." })}${filters}
    <div id="shipped" data-live="${LIVE}" aria-live="polite">${shippedRegion(read, choice)}</div>`;
    <div id="shipped" data-live="${LIVE}" aria-live="polite">${shippedRegion(read, choice, new Date(), "UTC", newestLive(ctx.app.db))}</div>`;
  const response = page({ ctx, title: "Changelog", body, scriptJs: `${liveRegionsJs}\n${localDaysJs}` });
  notePhase(ctx.request, "render", performance.now() - rendering);
  return response;
@@ -6,6 +6,7 @@
import type { Database } from "bun:sqlite";
import { blocking, MAX_REASON, standingHolds } from "../../keeper/hold.ts";
import { secondsWords, withLandedToLive } from "../../keeper/landed.ts";
import { listKeeperReleases, type Stage } from "../../keeper/record.ts";
import { html, when, type Html } from "../html.ts";
import { badge, btn, card, time, type Tone } from "../ui.ts";
@@ -19,28 +20,29 @@
const STAGE: Record<Stage, { label: string; tone: Tone }> = {
/** `mayHold` is `mayRollBack`: the card then offers the owner's hold and its lift. */
export function keeperPanel(db: Database, mayHold = false): Html | undefined {
  const releases = listKeeperReleases(db);
  const releases = withLandedToLive(db, listKeeperReleases(db));
  const holds = standingHolds(db);
  const blocked = blocking(holds);
  if (releases.length === 0 && blocked === undefined && !mayHold) return undefined;
  const rows = releases.map((r) => {
    const s = STAGE[r.stage];
    return html`<tr>
      <td>${r.version} ${badge(s.label, { tone: s.tone })}</td>
      <td>${r.version} <span class="small muted">${r.commit}</span> ${badge(s.label, { tone: s.tone })}</td>
      <td>${time(r.cutAt)}</td>
      <td>${r.landedToLiveSeconds === null ? "–" : secondsWords(r.landedToLiveSeconds)}</td>
      <td>${r.knownGood.version}</td>
      <td>${r.rollNon200} of ${r.rollProbes}</td>
      <td>${r.probes} / ${r.pages}</td>
      <td>${r.worstMs} ms</td>
      <td>${r.secondsDark} s</td>
    </tr>${when(r.reason !== "", () => html`<tr><td colspan="7" class="small muted">${r.reason}</td></tr>`)}`;
    </tr>${when(r.reason !== "", () => html`<tr><td colspan="8" class="small muted">${r.reason}</td></tr>`)}`;
  });
  return card(html`<h2>Release keeper</h2>
    <p class="small ${blocked === undefined ? "muted" : ""}" id="keeper-blocked">${blocked ?? "Nothing holds the next cut."}</p>
    ${when(rows.length > 0, () => html`<table class="table">
      <thead><tr><th>Version</th><th>Cut</th><th>Known good</th><th>Roll non-200</th><th>Probes / pages</th><th>Worst</th><th>Dark</th></tr></thead>
    ${when(rows.length > 0, () => html`<div class="table-scroll" role="region" tabindex="0" aria-label="Keeper releases"><table class="table" data-sticky-lead>
      <thead><tr><th>Version</th><th>Cut</th><th>Landed to live</th><th>Known good</th><th>Roll non-200</th><th>Probes / pages</th><th>Worst</th><th>Dark</th></tr></thead>
      <tbody>${rows}</tbody>
      
        sentinelcrew
        12d ago
        outdated
      
      Eight-column table: wrap in the named scroll wrapper / responsive table pattern so it works at 360px.
    
    </table>`)}
    </table></div>`)}
    ${when(mayHold, () => html`${holds.filter((h) => h.kind === "owner").map(
      (h) => html`<form method="post" action="/releases/hold/${h.id}/lift" class="small">Owner hold ${h.id}: ${h.reason} ${btn("Lift", { type: "submit" })}</form>`,
    )}
@@ -134,7 +134,11 @@
test("warm home holds p95 under the stated budget", async () => {
  panelCache(f.app.db).clear();
  const missed = await counted();
  expect(missed.hits).toEqual({ kpis: false, panels: false });
  const missOnly = [...missed.ran.keys()].filter((sql) => !warm.ran.has(sql));
  // Reads only: a route-timing flush that lands inside this one request (its INSERTs, PRAGMAs,
  // total_changes) is a write any request may carry, not the cache missing.
  const missOnly = [...missed.ran.keys()].filter(
    (sql) => !warm.ran.has(sql) && /^\s*SELECT\b/i.test(sql) && !sql.includes("total_changes()"),
  );
  // The KPI reads (eight on this fixture) and the pinned panel's own query.
  expect(missOnly.length).toBeGreaterThan(5);
  expect(missOnly).toContain('SELECT "kind" AS "group", COUNT(*) AS "value" FROM "run" GROUP BY 1 ORDER BY 1 LIMIT 1000');
@@ -0,0 +1,68 @@
/**
 * Landed to live and the short commit at every door (#8042): the /releases
 * card, the terminal's Release keeper and MCP `release_keeper` read the same
 * figure from `keeper/landed.ts`. Pull 7 merged at noon; the keeper cut at
 * once, and 0.7.284 answered 42 s later, so each door says 42 s.
 */
import { describe, expect, test } from "bun:test";
import type { Ctx as HttpCtx } from "../src/http/context.ts";
import { tool as keeperTool } from "../src/http/mcp/tools/release_keeper.ts";
import { tick } from "../src/keeper/keeper.ts";
import type { LandedRelease } from "../src/keeper/landed.ts";
import type { User } from "../src/core/users.ts";
import { createForgeApp } from "../src/tui/forge.ts";
import { keeperScreen } from "../src/tui/screens/keeper.ts";
import { headlessTerminal } from "../src/tui/term/terminal.ts";
import { keeperPanel } from "../src/web/pages/keeper-panel.ts";
import { quietLogs } from "./helpers/globals.ts";
import { B, keeperForge } from "./helpers/keeper.ts";
import { settleApp } from "./helpers/tui.ts";
quietLogs();
const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
/** B landed through pull 7 at the clock's noon, was cut, and answered as 0.7.284 42 s after it merged. */
async function liveForge() {
  const f = await keeperForge();
  const merged = new Date(f.world.now).toISOString();
  f.app.db
    .query(`INSERT INTO release (sha, repo_id, pull_number, ok, line, created_at) VALUES (?1, ?2, 7, 1, 'preflight ok', ?3)`)
    .run(B, f.repo.id, merged);
  expect(await tick(f.ports)).toContain("cut 0.7.284");
  f.world.now = Date.parse(merged) + 42_000;
  f.world.live = "0.7.284";
  await tick(f.ports);
  return f;
}
describe("landed to live at every door", () => {
  test("MCP release_keeper gives each release its short commit and landed-to-live seconds", async () => {
    const f = await liveForge();
    const read = await keeperTool.run({ app: f.app } as unknown as HttpCtx, f.user, {});
    if (!read.ok) throw new Error("release_keeper refused an admin");
    const [first] = (read.value as { releases: LandedRelease[] }).releases;
    expect(first).toMatchObject({ version: "0.7.284", commit: "bbbbbbb", landedToLiveSeconds: 42 });
  });
  test("the terminal's Release keeper shows the short commit and landed to live", async () => {
    const f = await liveForge();
    const terminal = headlessTerminal(200, 20);
    const { app: tui, session } = createForgeApp(f.app, identity(f.user), { terminal, exitHooks: false });
    tui.start();
    tui.ctx.push(keeperScreen(session, tui.ctx));
    await settleApp(tui);
    const text = tui.screen.toText();
    tui.stop?.();
    expect(text).toContain("bbbbbbb");
    expect(text).toContain("landed to live 42 s");
  });
  test("/releases puts the eight-column table in the named scroller, with the figure in it", async () => {
    const f = await liveForge();
    const card = String(keeperPanel(f.app.db));
    expect(card).toMatch(/<div class="table-scroll" role="region" tabindex="0" aria-label="Keeper releases"><table class="table" data-sticky-lead>/);
    expect(card).toContain('<span class="small muted">bbbbbbb</span>');
    expect(card).toContain("<td>42 s</td>");
  });
});
@@ -0,0 +1,343 @@
/**
 * The keeper ships by swapping, not by image (#8042). On a hosted child
 * forge's shape: a running tree, a landed tree, a keeper that cuts on cadence,
 * swaps it in beside the running process on the same port, verifies it under
 * load and keeps it as known good; then a bad release is swapped in, fails
 * its verify, and the known-good tree is swapped back and held. The stand-in
 * forges are a few lines that answer health as the commit and version they
 * are told; the swap, the pin, the keep and the keeper are the real code.
 */
import { afterAll, describe, expect, test } from "bun:test";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { err, ok } from "../src/kernel/result.ts";
import { standingHolds } from "../src/keeper/hold.ts";
import { landedToLive, newestLive, secondsWords } from "../src/keeper/landed.ts";
import { tick } from "../src/keeper/keeper.ts";
import { listKeeperReleases } from "../src/keeper/record.ts";
import { keeperEnv, SWAP_INTERVAL_MS } from "../src/keeper/wire.ts";
import { swapper, dependencyChange } from "../src/keeper/swapper.ts";
import { keepGood, lastGood } from "../src/release/launch.ts";
import { releaseDir } from "../src/release/checkout.ts";
import { liveSchema } from "../src/release/rollback.ts";
import { readServing } from "../src/release/serving.ts";
import { endSwap, handover } from "../src/release/swap.ts";
import { keeperForge, A, B, C } from "./helpers/keeper.ts";
const STAND_IN = `
Bun.serve({
  port: Number(process.env.PORT),
  reusePort: true,
  fetch: () => Response.json({ ok: true, draining: false, commit: process.env.FORGE_COMMIT, version: process.env.FORGE_VERSION }),
});
`;
const realMigrate = (() => {
  try {
    return readFileSync("/proc/sys/net/ipv4/tcp_migrate_req", "utf8").trim() === "1";
  } catch {
    return false;
  }
})();
handover.migrate = true;
const dirs: string[] = [];
const pids = new Set<number>();
afterAll(() => {
  // The swap turn and the kernel switch are this process's: leave them as other files find them.
  endSwap(false);
  delete handover.migrate;
  for (const pid of pids) {
    try {
      process.kill(pid, "SIGKILL");
    } catch {}
  }
  for (const d of dirs) rmSync(d, { recursive: true, force: true });
});
/** `migrations` is how many the tree knows: more than any live database, unless a test says fewer. */
function tree(root: string, main = STAND_IN, pkg = "{}", migrations = 9999): void {
  mkdirSync(join(root, "src", "core"), { recursive: true });
  writeFileSync(join(root, "src", "core", "schema.ts"), `export const migrations = new Array(${migrations}).fill(0);\n`);
  writeFileSync(join(root, "src", "main.ts"), main);
  writeFileSync(join(root, "package.json"), pkg);
}
async function freePort(): Promise<number> {
  const s = Bun.serve({ port: 0, fetch: () => new Response("") });
  const port = s.port ?? 0;
  s.stop(true);
  return port;
}
async function health(port: number): Promise<{ commit?: string; version?: string } | undefined> {
  try {
    const res = await fetch(`http://127.0.0.1:${port}/api/v1/health`, { headers: { connection: "close" }, signal: AbortSignal.timeout(3_000) });
    return (await res.json()) as { commit?: string; version?: string };
  } catch {
    return undefined;
  }
}
/** A forge running tree A on a port, with a swapper that checks landed trees out of `landed`. */
async function hosted(landed: Record<string, { main?: string; pkg?: string; preflight?: boolean }>) {
  const f = await keeperForge();
  const base = mkdtempSync(join(tmpdir(), "forge-keeper-swap-"));
  dirs.push(base);
  const dataDir = f.app.config.dataDir;
  const running = join(base, "running");
  tree(running);
  const port = await freePort();
  const old = Bun.serve({ port, reusePort: true, fetch: () => Response.json({ ok: true, draining: false, commit: A, version: "0.7.283" }) });
  const now = { commit: A as string, version: "0.7.283" };
  let before: number | undefined = undefined;
  const drained: string[] = [];
  const build = () => swapper({
    dataDir,
    port,
    treeRoot: running,
    version: () => now.version,
    commit: () => now.commit,
    env: { PATH: process.env.PATH, FORGE_VERSION: "0.7.283", FORGE_BASE_VERSION: "0.7.283" },
    checkout: async (sha) => {
      const spec = landed[sha];
      if (spec === undefined) return err("no such commit");
      const dir = releaseDir(dataDir, sha);
      if (!existsSync(dir)) tree(dir, spec.main, spec.pkg);
      return ok(dir);
    },
    preflight: async (dir) => {
      const sha = dir.split("/").pop()!;
      return landed[sha]?.preflight === false ? { ok: false, line: "boot failed: (fixture)" } : { ok: true, line: "preflight ok" };
    },
    drain: () => {
      drained.push(now.commit);
      // The process that handed over lets go of the port; the one before it, started by a swap, is stopped.
      if (before === undefined) old.stop(true);
      else
        try {
          process.kill(before, "SIGTERM");
        } catch {}
      before = readServing(dataDir)?.pid;
      if (before !== undefined) pids.add(before);
    },
  });
  const h = {
    f,
    ports: { ...f.ports, deployer: build(), minIntervalMs: SWAP_INTERVAL_MS } as typeof f.ports,
    dataDir,
    port,
    now,
    drained,
    running,
    /** The swapped-in process is a new process: its own swapper, its own turn at the swap. */
    become(commit: string, version: string) {
      now.commit = commit;
      now.version = version;
      endSwap(false);
      h.ports = { ...h.ports, deployer: build() };
    },
  };
  return h;
}
describe("the keeper swaps main's tree in; a pod restart comes back on it", () => {
  test("land, swap, verify, bad release, swap back: no refused request, known good kept, held", async () => {
    const h = await hosted({ [B]: {}, [C]: {} });
    const { f } = h;
    // A client that hits the port for the whole story.
    let stop = false;
    const seen = { ok: 0, bad: 0 };
    const hammer = (async () => {
      while (!stop) {
        const got = await health(h.port);
        if (got?.commit === undefined) seen.bad++;
        else seen.ok++;
        await Bun.sleep(5);
      }
    })();
    // Land: main moves to B (a landing the forge already checked out and recorded).
    f.land(7, "A change");
    f.app.db.query(`INSERT INTO release (sha, repo_id, pull_number, ok, line, created_at) VALUES (?1, ?2, 7, 1, 'preflight ok', ?3)`).run(B, f.repo.id, new Date(f.world.now).toISOString());
    f.world.now += 61_000; // the cadence minute
    const cut = await tick(h.ports);
    expect(cut).toContain("cut 0.7.284");
    expect(cut).toContain(`promoted to tree/${B}:0.7.284`);
    // The new process answers on the same port as its own commit and version, beside the draining one.
    expect(await (async () => { for (let i = 0; i < 40; i++) { const g = await health(h.port); if (g?.commit === B) return g; await Bun.sleep(50); } })()).toMatchObject({ commit: B, version: "0.7.284" });
    expect(h.drained).toEqual([A]);
    expect(readServing(h.dataDir)?.sha).toBe(B);
    // The tree it left is pinned where a rollback finds it, marked as having served.
    expect(existsSync(join(releaseDir(h.dataDir, A), "src", "main.ts"))).toBe(true);
    expect(existsSync(join(releaseDir(h.dataDir, A), ".was-live"))).toBe(true);
    // Verify: the new process is the keeper now.
    h.become(B, "0.7.284");
    f.world.live = "0.7.284";
    f.world.now += 5_000;
    expect(await tick(h.ports)).toContain("0.7.284 verified live");
    expect(lastGood(h.dataDir)).toMatchObject({ version: "0.7.284", commit: B, base: "0.7.283" });
    expect(existsSync(join(h.dataDir, "release", "good", "0.7.284", "src", "main.ts"))).toBe(true);
    const [first] = listKeeperReleases(f.app.db);
    expect(first).toMatchObject({ stage: "verified", knownGood: { version: "0.7.283", image: `tree/${A}:0.7.283` } });
    // Landed to live: from the merge to the new version answering, the page's number.
    // Merged, 61 s to the cadence's minute, 5 s for the new process to answer.
    expect(landedToLive(f.app.db, first!)).toBe(66);
    expect(newestLive(f.app.db)).toMatchObject({ version: "0.7.284", sha: B });
    // A bad release: C boots and answers, then fails under load.
    f.world.main = C;
    f.world.now += 61_000;
    expect(await tick(h.ports)).toContain(`promoted to tree/${C}:0.7.285`);
    h.become(C, "0.7.285");
    f.world.live = "0.7.285";
    f.world.healthz = (n) => (n >= 6 ? { status: 503, ms: 4 } : { status: 200, ms: 4 });
    const verdict = await tick(h.ports);
    expect(verdict).toContain("0.7.285 failed its verify");
    expect(verdict).toContain(`rolled back to 0.7.284 (tree/${B}:0.7.284)`);
    // Swapped back: the pinned B tree serves again as 0.7.284.
    const back = await (async () => { for (let i = 0; i < 60; i++) { const g = await health(h.port); if (g?.commit === B && g.version === "0.7.284") return g; await Bun.sleep(50); } })();
    expect(back).toMatchObject({ commit: B, version: "0.7.284" });
    expect(standingHolds(f.app.db)[0]).toMatchObject({ kind: "failed", version: "0.7.285" });
    // What a restart would run is still the verified B, not the bad C and not the image.
    expect(lastGood(h.dataDir)).toMatchObject({ version: "0.7.284", commit: B, base: "0.7.283" });
    stop = true;
    await hammer;
    expect(seen.ok).toBeGreaterThan(10);
    // The promise of 0 refused requests holds where the kernel migrates queued connections (the check runs there).
    if (realMigrate) expect(seen.bad).toBe(0);
  }, 60_000);
  test("a tree that adds a dependency is not cut: no tag, no row, and the line asks for a new image", async () => {
    const h = await hosted({ [B]: { pkg: JSON.stringify({ dependencies: { left: "1.0.0" } }) } });
    h.f.world.now += 61_000;
    const line = await tick(h.ports);
    expect(line).toContain("not cut: dependencies names left");
    expect(line).toContain("needs a new image");
    expect(h.f.world.tags).toHaveLength(0);
    expect(listKeeperReleases(h.f.app.db)).toHaveLength(0);
    expect(h.drained).toEqual([]);
    expect(dependencyChange(join(h.running))).toBeUndefined();
  });
  test("a process that cannot swap (it holds the SSH terminal) cuts nothing", async () => {
    const h = await hosted({ [B]: {} });
    h.f.world.now += 61_000;
    handover.ssh = true;
    try {
      expect(await tick(h.ports)).toContain("not cut: this process holds the SSH terminal");
    } finally {
      delete handover.ssh;
    }
    expect(h.f.world.tags).toHaveLength(0);
  });
  test("a tree whose preflight is red is not cut", async () => {
    const h = await hosted({ [B]: { preflight: false } });
    h.f.world.now += 61_000;
    expect(await tick(h.ports)).toContain("not cut: preflight failed: boot failed");
    expect(h.f.world.tags).toHaveLength(0);
  });
  test("a release that will not come up leaves the running process serving and the cut sent again", async () => {
    const h = await hosted({ [B]: { main: `await Bun.sleep(60000)` } });
    h.f.world.now += 61_000;
    const line = await tick({ ...h.ports, deployer: swapper({ ...swapOf(h), budgetMs: 800 }) });
    expect(line).toContain("refused");
    expect((await health(h.port))?.commit).toBe(A);
    expect(h.drained).toEqual([]);
  }, 20_000);
  test("a refusal is remembered per commit: two ticks check out and preflight once", async () => {
    const h = await hosted({ [B]: {} });
    h.f.world.now += 61_000;
    let checkouts = 0;
    let preflights = 0;
    const base = swapOf(h);
    const deployer = swapper({
      ...base,
      checkout: async (sha: string) => {
        checkouts++;
        tree(releaseDir(h.dataDir, sha));
        return ok(releaseDir(h.dataDir, sha));
      },
      preflight: async () => {
        preflights++;
        return { ok: false, line: "boot failed: (fixture)" };
      },
    });
    const ports = { ...h.ports, deployer };
    expect(await tick(ports)).toContain("not cut: preflight failed");
    h.f.world.now += 61_000;
    expect(await tick(ports)).toContain("not cut: preflight failed");
    expect(preflights).toBe(1);
    expect(checkouts).toBe(1);
  });
  test("a swap back to a tree that knows fewer migrations than the database is refused, and says why", async () => {
    const h = await hosted({});
    const live = liveSchema(h.dataDir);
    expect(live).toBeGreaterThan(1);
    // B has served (pinned and marked) but predates the live schema; C is as new.
    tree(releaseDir(h.dataDir, B), STAND_IN, "{}", live - 1);
    writeFileSync(join(releaseDir(h.dataDir, B), ".was-live"), "");
    // C never answers: one that did would win the swap on a fast machine and be left on the port.
    tree(releaseDir(h.dataDir, C), `await Bun.sleep(60000)`, "{}", live);
    writeFileSync(join(releaseDir(h.dataDir, C), ".was-live"), "");
    endSwap(false);
    const deployer = swapper({ ...swapOf(h), checkout: async (sha: string) => ok(releaseDir(h.dataDir, sha)), budgetMs: 800 });
    await expect(deployer.setImage(`tree/${B}:0.7.284`)).rejects.toThrow(`${B} predates schema ${live}; rolling back would need a down-migration`);
    expect(h.drained).toEqual([]);
    expect((await health(h.port))?.commit).toBe(A);
    // A tree with the live schema is not refused by the guard (this one then fails to answer in its short budget, a different line).
    await expect(deployer.setImage(`tree/${C}:0.7.285`)).rejects.not.toThrow("predates schema");
  }, 20_000);
  test("the running commit must be stamped to name a known good", async () => {
    const h = await hosted({ [B]: {} });
    h.now.commit = "";
    await expect(h.ports.deployer.image()).rejects.toThrow("not stamped");
  });
});
/** The pieces `hosted` built the swapper from, for a variant of it. */
function swapOf(h: Awaited<ReturnType<typeof hosted>>) {
  return {
    dataDir: h.dataDir,
    port: h.port,
    treeRoot: h.running,
    version: () => h.now.version,
    commit: () => h.now.commit,
    env: { PATH: process.env.PATH },
    checkout: async (sha: string) => {
      tree(releaseDir(h.dataDir, sha), `await Bun.sleep(60000)`);
      return ok(releaseDir(h.dataDir, sha));
    },
    preflight: async () => ({ ok: true, line: "ok" }),
    drain: () => h.drained.push("x"),
  };
}
test("swap mode needs no cluster, only the swap switch", () => {
  expect(keeperEnv({ FORGE_RELEASE_KEEPER: "swap", FORGE_SELF_SWAP: "1" })).toEqual({ on: true, mode: "swap" });
  expect(keeperEnv({ FORGE_RELEASE_KEEPER: "swap" })).toMatchObject({ on: false });
  expect(keeperEnv({ FORGE_RELEASE_KEEPER: "on", KUBERNETES_SERVICE_HOST: "10.0.0.1" })).toMatchObject({ on: true, mode: "kube" });
  expect(SWAP_INTERVAL_MS).toBe(60_000);
});
test("seconds are said in words", () => {
  expect(secondsWords(45)).toBe("45 s");
  expect(secondsWords(72)).toBe("1 min 12 s");
});
test("keeping a swapped tree names the image it is over", () => {
  const dir = mkdtempSync(join(tmpdir(), "forge-keep-"));
  dirs.push(dir);
  const t = join(dir, "t");
  tree(t);
  keepGood(dir, t, "0.7.290", B, 1, "0.7.283");
  expect(lastGood(dir)?.base).toBe("0.7.283");
});
@@ -154,7 +154,10 @@
test("a record that is not one reads as none", () => {
test("the image starts the launcher, and the launcher imports nothing a release could break", async () => {
  const repo = join(import.meta.dir, "..");
  expect(await Bun.file(join(repo, "Dockerfile")).text()).toContain('CMD ["bun", "src/release/launch.ts"]');
  const source = await Bun.file(join(repo, "src/release/launch.ts")).text();
  const imports = new Bun.Transpiler({ loader: "ts" }).scanImports(source).map((i) => i.path);
  expect(imports.every((p) => p.startsWith("node:") || p === "bun")).toBe(true);
  // serving.ts (#8042) is the launcher's own sibling, node built-ins only, so the rule holds for both.
  for (const [file, sibling] of [["launch.ts", "./serving.ts"], ["serving.ts", undefined]] as const) {
    const source = await Bun.file(join(repo, "src/release", file)).text();
    const imports = new Bun.Transpiler({ loader: "ts" }).scanImports(source).map((i) => i.path);
    expect(imports.every((p) => p.startsWith("node:") || p === "bun" || p === sibling)).toBe(true);
  }
});
@@ -0,0 +1,124 @@
/**
 * A pod restart comes back on the newest good swapped tree (#8042). The image
 * is a base: while the same image restarts, the swapped release kept over it
 * runs first, and an older image's version never overwrites it. A different
 * image (a deliberate roll) is the newer base; a swapped tree that will not
 * start gives way to the image, and the forge says so.
 */
import { afterAll, afterEach, beforeAll, expect, test } from "bun:test";
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { Subprocess } from "bun";
import { keepGood, lastGood, launch, rolledBack, type Launched } from "../src/release/launch.ts";
import { clearServing, follow, readServing, writeServing } from "../src/release/serving.ts";
let root: string;
let n = 0;
const running: Subprocess[] = [];
beforeAll(() => {
  root = mkdtempSync(join(tmpdir(), "forge-swapped-launch-"));
});
afterEach(async () => {
  for (const child of running.splice(0)) {
    child.kill("SIGKILL");
    await child.exited;
  }
});
afterAll(() => rmSync(root, { recursive: true, force: true }));
function tree(main: string): string {
  const dir = join(root, `tree-${++n}`);
  mkdirSync(join(dir, "src"), { recursive: true });
  writeFileSync(join(dir, "src", "main.ts"), main);
  writeFileSync(join(dir, "package.json"), "{}");
  return dir;
}
const SERVES = `
Bun.serve({ hostname: "127.0.0.1", port: Number(process.env.PORT), fetch(req) {
  if (new URL(req.url).pathname !== "/api/v1/health") return new Response("", { status: 404 });
  return Response.json({ ok: true, version: process.env.FORGE_VERSION, commit: process.env.FORGE_COMMIT || null, base: process.env.FORGE_BASE_VERSION });
} });
`;
const THROWS = `throw new Error("(test fixture) the swapped tree does not start");\n`;
async function run(dataDir: string, dir: string, version: string, graceMs = 30_000): Promise<Launched & { port: number }> {
  const probe = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response("") });
  const port = probe.port ?? 0;
  await probe.stop(true);
  const launched = await launch({
    tree: dir,
    dataDir,
    port,
    version,
    commit: null,
    env: { PATH: process.env.PATH, PORT: String(port) },
    graceMs,
    started: (child) => running.push(child),
  });
  return { ...launched, port };
}
const healthAt = async (port: number) =>
  (await (await fetch(`http://127.0.0.1:${port}/api/v1/health`)).json()) as { version: string; commit: string; base: string };
const SWAPPED = "5".repeat(40);
test("the same image restarting runs the swapped release kept over it, and keeps it", async () => {
  const data = join(root, "data-same");
  keepGood(data, tree(SERVES), "0.7.290", SWAPPED, Date.now(), "0.7.283");
  expect(lastGood(data)).toMatchObject({ version: "0.7.290", commit: SWAPPED, base: "0.7.283" });
  // The image is the older 0.7.283; its tree throws, so only the swapped tree can answer.
  const launched = await run(data, tree(THROWS), "0.7.283");
  expect(launched.ran).toBe("release");
  expect(await healthAt(launched.port)).toMatchObject({ version: "0.7.290", commit: SWAPPED, base: "0.7.283" });
  expect(lastGood(data)).toMatchObject({ version: "0.7.290", base: "0.7.283" });
  expect(rolledBack(data)).toBeUndefined();
});
test("a stale image that is ready does not replace the swapped release as the last good one", async () => {
  const data = join(root, "data-stale");
  keepGood(data, tree(SERVES), "0.7.290", SWAPPED, Date.now(), "0.7.283");
  await run(data, tree(SERVES), "0.7.283");
  expect(lastGood(data)?.version).toBe("0.7.290");
});
test("a different image is a deliberate roll: it runs, and is kept over the swapped tree", async () => {
  const data = join(root, "data-newer");
  keepGood(data, tree(SERVES), "0.7.290", SWAPPED, Date.now(), "0.7.283");
  const launched = await run(data, tree(SERVES), "0.7.300");
  expect(launched.ran).toBe("release");
  expect(await healthAt(launched.port)).toMatchObject({ version: "0.7.300" });
  expect(lastGood(data)).toMatchObject({ version: "0.7.300", base: null });
});
test("a swapped tree that does not start gives way to the image, recorded as a rollback", async () => {
  const data = join(root, "data-broken");
  keepGood(data, tree(THROWS), "0.7.290", SWAPPED, Date.now(), "0.7.283");
  const launched = await run(data, tree(SERVES), "0.7.283");
  expect(launched.ran).toBe("rollback");
  expect(launched.line).toContain("0.7.290: it exited with 1 before it was ready; the image 0.7.283 runs instead");
  expect(await healthAt(launched.port)).toMatchObject({ version: "0.7.283" });
  expect(rolledBack(data)).toMatchObject({ from: "0.7.290", to: "0.7.283", paged: false });
});
test("the launcher follows the process that took over, and lets go when it is gone", async () => {
  const data = join(root, "data-follow");
  clearServing(data);
  expect(readServing(data)).toBeUndefined();
  const heir = Bun.spawn([process.execPath, "-e", "setTimeout(() => {}, 700)"], { stdout: "ignore", stderr: "ignore" });
  writeServing(data, { pid: heir.pid, sha: SWAPPED });
  expect(readServing(data)).toEqual({ pid: heir.pid, sha: SWAPPED });
  const began = Date.now();
  await follow(data, 1, 50);
  expect(Date.now() - began).toBeGreaterThanOrEqual(400);
  await heir.exited;
  // Its own child serving again (or nobody) ends the wait at once.
  writeServing(data, { pid: 4242, sha: SWAPPED });
  const own = Date.now();
  await follow(data, 4242, 50);
  expect(Date.now() - own).toBeLessThan(200);
  clearServing(data);
  expect(readServing(data)).toBeUndefined();
});
@@ -61,6 +61,8 @@
function plantRun(
}
beforeAll(async () => {
  // The error ledger is the process's: a file run before this one in the same shard leaves its lines there.
  clearErrors();
  dataDir = await mkdtemp(join(tmpdir(), "forge-sentinel-"));
  app = createApp({ dataDir, port: 0 });
  const a = await createUser(app.db, {
		@@ -545,7 +545,7 @@ What the crew's tokens bought is on the board (`/board/returns`), in MCP `return
545	545
546	546	The terminal's palette has `Signed in` for anyone signed in: your browsers, tokens and this terminal's key, with when each was last seen; `d` signs the selected browser out after a confirm, as `/settings` and MCP `sessions` do, and on a token revokes it after a confirm (Keep is the default) through the same `deleteToken` action as Settings and the API, so it stops working at once (`docs/CREW.md`, "Where you are signed in").
547	547
548		The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases on the forge's own one-minute timer, with no model. It is off until `FORGE_RELEASE_KEEPER=on`, and the pod's service account may `get` and `patch` the forge Deployment.
	548	The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases on the forge's own one-minute timer, with no model. It is off until `FORGE_RELEASE_KEEPER=on`, and the pod's service account may `get` and `patch` the forge Deployment. `FORGE_RELEASE_KEEPER=swap` needs no cluster: it swaps main's tree in place (`docs/features/ship-itself.md`). Its tag is the forge's own and is not pushed to an origin, so a cut is invisible to Forgejo-side tools. /releases, the terminal's Release keeper and MCP `release_keeper` show each release's short commit and its seconds from landed to live; /changelog's live line says it too.
549	549
550	550	- Cut: it tags main's head as `v<patch + 1>` when main has moved past a live tag, nothing is in flight, the merge queue is still, 30 minutes have passed since its last cut and no hold stands.
551	551	- Verify: it records the known-good version and image before the tag, sets the Deployment's image, and has the new version spend ten minutes under page load probing `/healthz` each second.
		@@ -142,7 +142,7 @@ for a tunnel and a forge that sleeps.
142	142	\| `quotas` \| `{ key?, value?, dryRun? }` (forge admin): the per-account quotas `repositories`, `apps`, `storage` (MB), `previews`, `runs`, `checks`, each a number or null for none. `repositories` bounds repos on create, fork and copy; `apps` the apps made; `storage` the MB of repos plus app data, asked on push, create and fork; `previews` live previews; `runs` concurrent agent runs; `checks` concurrent checks. Each is asked before the thing is made or started and a forge admin is never limited; with `key` and `value` (`""` clears), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token \| yes \|
143	143	\| `read_file` \| a file (utf-8, or base64 for binary) or a directory listing at a ref, as GET …/contents; MP4/MOV/M4A files also carry about: {kind, seconds, width?, height?, says} \| \|
144	144	\| `release_hold` \| (admin scope) `{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent \| yes \|
145		\| `release_keeper` \| `{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases \| \|
	145	\| `release_keeper` \| `{}`: what the release keeper (#7557) did: its releases newest first (version, commit (short sha), stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason, landedToLiveSeconds — from the merge of its pull to the new version answering, null until it has (#8042)) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases \| \|
146	146	\| `releases` \| `{shipped: true}`: what went live, for anyone: `version`, `commit`, `live` (newest first: `repo`, `number`, `title`, `at`) and `onItsWay` (landed, not yet in the running commit), the list /changelog shows. `{}`: the forge's kept releases newest first: short sha, version, the pull that landed it, when, and for the live one `live: true` and `since`; for whoever may read /releases, others get its refusal; no rollback, a person does that on the page \| \|
147	147	\| `repo` \| `{ repo }`: one repository's description, visibility, autonomy, `audience { level, sentence }` and `provenance` (where a migrated one came from, `null` when hidden), the same as `GET /api/v1/repos/:owner/:name` \| \|
148	148	\| `repo_labels` \| `{ repo, action?, name?, color?, id? }`: list (`list`, the default, paged by `limit` 1–200 and `cursor`, as `{ items, next_cursor }`, each label with `open`, its open issues and pulls, for a writer; a read-only token may), create (`create`, `name`, `color` six hex digits), rename or recolour (`update`, `id`, `name` and/or `color`, as the labels page) or remove (`remove`, `id`) the repository's labels, as `GET`/`POST …/labels` and `PATCH`/`DELETE …/labels/:id`; create, update and remove need an admin token and a repository admin \| yes \|
		@@ -2,4 +2,4 @@
2	2
3	3	A §5 `.table` too wide for a narrow screen sits inside a horizontal scroller, so the page itself never scrolls sideways and only the table's figures move.
4	4
5		- `.table-scroll` — the wrapper `<div>` around a wide `.table`: `overflow-x: auto` with a thin scrollbar, styled in `src/web/style/components.ts`. Pair it with `.table[data-sticky-lead]` to keep each row's name in view while the figures scroll. Written by `src/web/pages/account.ts`, `src/web/pages/audit.ts`, `src/web/pages/boards-held.ts`, `src/web/pages/crew.ts`, `src/web/pages/flow-panel.ts`, `src/web/pages/passkeys.ts`, `src/web/pages/proposal-tokens.ts`, `src/web/pages/settings.ts`, `src/web/pages/spend.ts`, `src/web/pages/subscriptions.ts`, `src/web/pages/system.ts`, `fleetGraphSection`'s figures table in `src/web/fleet-graph.ts`, and `vitalsTable` in `src/web/studio/vitals.ts`.
	5	- `.table-scroll` — the wrapper `<div>` around a wide `.table`: `overflow-x: auto` with a thin scrollbar, styled in `src/web/style/components.ts`. Pair it with `.table[data-sticky-lead]` to keep each row's name in view while the figures scroll. Written by `src/web/pages/account.ts`, `src/web/pages/audit.ts`, `src/web/pages/boards-held.ts`, `src/web/pages/crew.ts`, `src/web/pages/flow-panel.ts`, `src/web/pages/keeper-panel.ts`, `src/web/pages/passkeys.ts`, `src/web/pages/proposal-tokens.ts`, `src/web/pages/settings.ts`, `src/web/pages/spend.ts`, `src/web/pages/subscriptions.ts`, `src/web/pages/system.ts`, `fleetGraphSection`'s figures table in `src/web/fleet-graph.ts`, and `vitalsTable` in `src/web/studio/vitals.ts`.
		@@ -0,0 +1,34 @@
	1	# The forge ships itself (#8042)
	2
	3	One path, in the forge: landing → on cadence the keeper takes main's tree → preflight → swap in place → verify under load → known good, or swap back and hold. No image, no registry, no Forgejo, no Kubernetes roll, no steward.
	4
	5	## Switch
	6
	7	`FORGE_REPO=<owner>/<name> FORGE_SELF_SWAP=1 FORGE_RELEASE_KEEPER=swap` (the conductor's one line), on a kernel with `net.ipv4.tcp_migrate_req=1` and with the SSH terminal off (`FORGE_SSH_PORT=0`) until it can be handed over. `FORGE_RELEASE_KEEPER=on` is the old Deployment-patch path and stays until production has moved.
	8
	9	## What happens
	10
	11	1. `releaseOnLand` checks each landed tree out to `releases/<sha>/` (`checks/landing.ts`).
	12	2. Every minute (`SWAP_INTERVAL_MS`) the keeper (`keeper/keeper.ts`) compares main with the newest tag. `swapper.admits` refuses a cut that could not be swapped: SSH held, the kernel setting missing, a dependency added, a red `--preflight`. Nothing is tagged or recorded then, and the line says why.
	13	3. The cut records the known good as `tree/<commit>:<version>`, tags `v<version>` in the forge's own repository only (no origin push, so Forgejo-side tools never see the cut), and `swapper.setImage` pins the running tree under `releases/<commit>/`, starts the new tree beside this process on the same port (`release/swap.ts`, SO_REUSEPORT, `FORGE_VERSION` and `FORGE_COMMIT` set), waits until health names its commit, and drains this process.
	14	4. The new process finds a release in flight, spends ten minutes under page load (`keeper/verify.ts`) and either calls `verified` (the tree becomes the last good release) or sets the recorded known-good tree by name, which is the same swap, and holds.
	15
	16	## A pod restart
	17
	18	`release/launch.ts` is the container's first process. `keepGood` records a swapped tree with `base`, the image version it was swapped over, in `release/good.json`. When the same image restarts, the launcher starts that tree first, and a ready older image never replaces it as the last good. A different image version is a deliberate roll (a new base): it runs, and is kept over the swapped tree. A swapped tree that does not become ready in the grace gives way to the image and the rollback card says so. The swapped process is not the launcher's child, so it writes `release/serving.json` (`release/serving.ts`) and the launcher stays up while that pid lives and signals it on SIGTERM; when it dies the container exits and restarts on the newest good tree.
	19
	20	## Two writers, apps and runs
	21
	22	The handover is the existing one (#2040): the new process listens beside the old; the old stops accepting at once, stops its timers (throughput keeper, merge queue, deliveries), lets crew runs in flight finish and land, and closes after its in-flight requests. Runs hold leases that either process renews (`failAbandonedRuns` only fails a lapsed lease), apps are restarted by the new process and the old stops its own after the linger. The pause is measured, not assumed: during the roll the old process probes `/healthz` each second and counts non-200s into `keeper_release` (`roll_probes`, `roll_non200`, `seconds_dark`), shown on /releases; `test/release-keeper-swap.test.ts` drives a client through two swaps and asserts 0 refused requests where the kernel migrates queued connections.
	23
	24	## Dependencies
	25
	26	A release's directory has no `node_modules`, and installing into it before preflight would run a package manager's scripts inside the forge. So a tree whose package.json names any dependency is not swapped: the keeper does not cut it and says a new image is needed (the image is the base). Today `package.json` has none.
	27
	28	## Reading it
	29
	30	/releases, the terminal's Release keeper and MCP `release_keeper`: version, short commit, cut time, landed to live (merge of the landed pull to the new version answering), known good, probes, dark seconds. /changelog: the live line adds "Landed to live in …" for the release the forge runs.
	31
	32	## What can be deleted once production runs it
	33
	34	The image-per-release path (release workflow image build and push, the `Dockerfile` `VERSION`/`COMMIT` stamping per release), `keeper/deploy.ts` `kubeDeployer` and `FORGE_KEEPER_DEPLOYMENT`/`CONTAINER`, the promote scripts and `scripts/cut-release.ts`'s image half, the droplet's `forge-release-tick` on pg-grand, and the chart's service-account patch rule.
		@@ -1,11 +1,12 @@
1	1	import { mayRead } from "../../../release/gate.ts";
2	2	import { blocking, listHolds, standingHolds } from "../../../keeper/hold.ts";
	3	import { withLandedToLive } from "../../../keeper/landed.ts";
3	4	import { listKeeperReleases } from "../../../keeper/record.ts";
4	5	import { bad, good, type Tool } from "../shared.ts";
5	6
6	7	export const tool: Tool = {
7	8	name: "release_keeper",
8		doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases",
	9	doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, commit (short sha), stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason, landedToLiveSeconds — from the merge of its pull to the new version answering, null until it has (#8042)) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases",
9	10	description: "The release keeper's releases, holds and what blocks the next cut.",
10	11	inputSchema: { type: "object", properties: {} },
11	12	writes: false,
		@@ -15,7 +16,7 @@ export const tool: Tool = {
15	16	const db = ctx.app.db;
16	17	return good({
17	18	blocked: blocking(standingHolds(db)) ?? null,
18		releases: listKeeperReleases(db),
	19	releases: withLandedToLive(db, listKeeperReleases(db)),
19	20	holds: listHolds(db),
20	21	});
21	22	},
		@@ -19,6 +19,12 @@ export interface Deployer {
19	19	image(): Promise<string>;
20	20	/** Name this image on the forge container; the Deployment rolls to it with surge. */
21	21	setImage(image: string): Promise<void>;
	22	/** The reference that names `version` at `sha` (a swap names a tree, not a tag); the keeper falls back to `imageAt`. */
	23	refFor?(version: string, sha: string): string;
	24	/** Why `sha` cannot be released by this deployer, checked before anything is cut; undefined when it can. */
	25	admits?(sha: string): Promise<string \| undefined>;
	26	/** `version` read well under load and is the known good: keep it for a restart. */
	27	verified?(version: string): void;
22	28	}
23	29
24	30	/** An image reference: registry/path:tag or @digest, no space, no flag. */
		@@ -7,7 +7,9 @@
7	7	* 1. Cut on cadence (`cadence.ts`): the known-good version and image
8	8	* are recorded, then the annotated tag `v<version>` is made on main's
9	9	* head — a tag, never a commit, so no gate in flight is voided (#6640).
10		* 2. Promote: the Deployment is set to the new version's image; it
	10	* 2. Promote: the Deployment is set to the new version's image (or,
	11	* with `swapper.ts` as the deployer, #8042, main's tree is swapped in beside
	12	* this process on the same port: no image, no registry, no roll); it
11	13	* rolls with surge (`maxSurge 1, maxUnavailable 0`). The old forge
12	14	* probes `/healthz` each second while it still serves (`rollProbe`),
13	15	* so a roll that went dark says how long.
		@@ -119,6 +121,7 @@ async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<string
119	121	const numbers = { version: r.version, probes: kept.probes, pages: kept.pages, worstMs: kept.worstMs, secondsDark: kept.secondsDark, rollNon200: kept.rollNon200 };
120	122	if (verdict.ok) {
121	123	finish(p.db, r.id, "verified", "", p.now());
	124	p.deployer.verified?.(r.version);
122	125	const lifted = liftFixed(p.db, r.version, p.now());
123	126	p.bus.append({ kind: "release.verified_live", repoId: p.repo.id, payload: { ...numbers, holdsLifted: lifted } });
124	127	fileDark(p, getKeeperRelease(p.db, r.id)!);
		@@ -160,7 +163,7 @@ export async function tick(p: KeeperPorts): Promise<string> {
160	163	if (flying.stage === "rolling" && p.now() - Date.parse(flying.cutAt) > ROLL_DEADLINE_MS)
161	164	return rollBack(p, flying, `${flying.version} was not live ${ROLL_DEADLINE_MS / 3_600_000} h after its cut`);
162	165	// A promote that did not take (the API refused, or something set the image back) is sent again.
163		const want = imageAt(flying.knownGood.image, flying.version);
	166	const want = refFor(p, flying.knownGood.image, flying.version, flying.sha);
164	167	if (flying.stage === "rolling" && (await p.deployer.image()) !== want) await promote(p, flying.version, want);
165	168	return `${flying.version} is ${flying.stage}`;
166	169	}
		@@ -180,17 +183,23 @@ export async function tick(p: KeeperPorts): Promise<string> {
180	183	...(p.minIntervalMs === undefined ? {} : { minIntervalMs: p.minIntervalMs }),
181	184	});
182	185	if (!decision.cut \|\| main === undefined) return decision.cut ? "main is unreadable" : decision.why;
	186	// A tree the deployer cannot release (dependencies changed, preflight red) is not cut: no tag, no row.
	187	const refused = await p.deployer.admits?.(main);
		junipercrew 12d ago A refusal is not remembered: a red preflight or dependency change reruns checkout and a booted preflight every minute. Cache the refusal per sha (and log it once).
	188	if (refused !== undefined) return `not cut: ${refused}`;
183	189	const image = await p.deployer.image();
184	190	const knownGood = { version: p.live(), image };
185	191	// The row first, the tag second: a crash between them leaves a `rolling` row
186	192	// whose tag the next tick sends (`tagged`), never a tag the keeper has no row for.
187	193	const r = recordCut(p.db, { version: decision.version, sha: main, knownGood, at: p.now() });
188		const refused = await tagged(p, r);
189		if (refused !== undefined) return refused;
190		const sent = await promote(p, r.version, imageAt(image, r.version));
	194	const tagRefused = await tagged(p, r);
	195	if (tagRefused !== undefined) return tagRefused;
	196	const sent = await promote(p, r.version, refFor(p, image, r.version, main));
191	197	return `cut ${r.version} at ${main.slice(0, 12)}; ${sent}; known good: ${knownGood.version}`;
192	198	}
193	199
	200	const refFor = (p: KeeperPorts, image: string, version: string, sha: string): string =>
	201	p.deployer.refFor?.(version, sha) ?? imageAt(image, version);
	202
194	203	/** Make the release's tag; when it is refused the cut is dropped and the line says why. */
195	204	async function tagged(p: KeeperPorts, r: KeeperRelease): Promise<string \| undefined> {
196	205	const refused = await p.tag(r.version, r.sha);
		@@ -0,0 +1,55 @@
	1	/**
	2	* Seconds from landed to live (#8042): when the commit a release took was
	3	* merged, and when the new version first answered. Read from rows the forge
	4	* already keeps — the landing's `release` row and its pull's merge time, the
	5	* keeper's `live_at` — so there is nothing new to migrate. A commit that
	6	* landed without a pull of this forge counts from its checkout.
	7	*/
	8
	9	import type { Database } from "bun:sqlite";
	10	import { listKeeperReleases, type KeeperRelease } from "./record.ts";
	11
	12	/** When `sha` landed, epoch ms: its pull's merge, else the first time a landing checked it out. */
	13	function landedAt(db: Database, sha: string): number \| undefined {
	14	const row = db
	15	.query<{ at: string \| null }, [string]>(
	16	`SELECT COALESCE(pull.merged_at, release.created_at) AS at FROM release
	17	LEFT JOIN issue ON issue.repo_id = release.repo_id AND issue.number = release.pull_number
	18	LEFT JOIN pull ON pull.issue_id = issue.id
	19	WHERE release.sha = ?1 ORDER BY release.id LIMIT 1`,
	20	)
	21	.get(sha);
	22	const ms = row?.at == null ? NaN : Date.parse(row.at);
	23	return Number.isFinite(ms) ? ms : undefined;
	24	}
	25
	26	/** Whole seconds from landing to the new version answering; undefined while it has not, or when the landing is unknown. */
	27	export function landedToLive(db: Database, r: Pick<KeeperRelease, "sha" \| "liveAt">): number \| undefined {
	28	if (r.liveAt === null) return undefined;
	29	const landed = landedAt(db, r.sha);
	30	const live = Date.parse(r.liveAt);
	31	if (landed === undefined \|\| !Number.isFinite(live)) return undefined;
	32	return Math.max(0, Math.round((live - landed) / 1000));
	33	}
	34
	35	/** "1 min 12 s", "45 s". */
	36	export function secondsWords(s: number): string {
	37	return s < 60 ? `${s} s` : `${Math.floor(s / 60)} min ${s % 60} s`;
	38	}
		sentinelcrew 12d ago Expose these figures to the terminal and MCP too (tui keeper screen, release_keeper tool), each with a test.
	39
	40	/** The newest release that went live and stayed (or is being verified), with its landed-to-live seconds. */
	41	export function newestLive(db: Database): { version: string; sha: string; seconds: number } \| undefined {
	42	for (const r of listKeeperReleases(db, 10)) {
	43	if (r.stage === "rolled_back") continue;
	44	const seconds = landedToLive(db, r);
	45	if (seconds !== undefined) return { version: r.version, sha: r.sha, seconds };
	46	}
	47	return undefined;
	48	}
	49
	50	/** A keeper release with its short commit and landed-to-live seconds (null until it answered): what the card, the terminal and MCP show. */
	51	export type LandedRelease = KeeperRelease & { readonly commit: string; readonly landedToLiveSeconds: number \| null };
	52
	53	export function withLandedToLive(db: Database, releases: readonly KeeperRelease[]): LandedRelease[] {
	54	return releases.map((r) => ({ ...r, commit: r.sha.slice(0, 7), landedToLiveSeconds: landedToLive(db, r) ?? null }));
	55	}
		@@ -0,0 +1,195 @@
	1	/**
	2	* The keeper's hands without a cluster (#8042): the `Deployer` that swaps a
	3	* tree in place instead of patching a Deployment's image. A reference is
	4	* `tree/<sha>:<version>`; setting it checks the commit out under
	5	* `releases/<sha>/`, starts it beside this process on the same port
	6	* (`release/swap.ts`, SO_REUSEPORT, handed over once health names the commit),
	7	* and drains this process. Nothing here needs a registry, a Forgejo, a
	8	* Kubernetes API or a person's machine.
	9	*
	10	* - Known good: before the first swap away from a tree, that tree is
	11	* pinned under `releases/<commit>/` (an image's own tree included), so the
	12	* rollback is the same swap to a tree that was already live.
	13	* - Dependencies: a release is swapped in only when its package.json
	14	* names no dependency. Its directory has no node_modules, and installing
	15	* into it before preflight would run a package manager's scripts inside the
	16	* forge; so a tree that adds one is not cut (`admits`), and the line says a
	17	* new image is needed. The image only has to be a base.
	18	* - Rollback: a tree that has served is swapped back to only when it
	19	* knows every migration the live database has applied (the guard of
	20	* `--rollback`, `release/rollback.ts`); an older tree is refused and the
	21	* line says so, since this forge has no down-migration.
	22	* - Refusals: a commit refused (dependency, red preflight) is remembered
	23	* per sha and logged once, so the next minute does not check it out and boot
	24	* a preflight again.
	25	* - Restart: once verified, `verified` keeps the tree as the last good
	26	* release over this image's version (`release/launch.ts`), so a pod
	27	* restart comes back on it, not on the older image's version.
	28	*/
	29
	30	import { cpSync, existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
	31	import { join } from "node:path";
	32	import { err, type Result } from "../kernel/result.ts";
	33	import { releaseDir, type Verdict } from "../release/checkout.ts";
	34	import { keepGood } from "../release/launch.ts";
	35	import { liveSchema, releaseSchema } from "../release/rollback.ts";
	36	import { beginSwap, endSwap, startRelease, swapBlocked } from "../release/swap.ts";
	37	import type { Deployer } from "./deploy.ts";
	38
	39	const REF = /^tree\/([0-9a-f]{40}):([0-9A-Za-z][0-9A-Za-z.+_-]{0,63})$/;
	40	const TREE_PARTS = ["src", "package.json", "tsconfig.json"];
	41	/** A tree that has served: a rollback target, not a candidate, so it is not preflighted again. */
	42	const LIVE_MARK = ".was-live";
	43	const DEP_FIELDS = ["dependencies", "devDependencies", "optionalDependencies", "peerDependencies"];
	44
	45	export interface SwapperInput {
	46	readonly dataDir: string;
	47	readonly port: number;
	48	/** The tree this process runs from: the directory holding `src/main.ts`. */
	49	readonly treeRoot: string;
	50	readonly version: () => string;
	51	readonly commit: () => string \| null;
	52	readonly env: Record<string, string \| undefined>;
	53	/** Write `sha`'s tree to `releases/<sha>/`; the directory. */
	54	readonly checkout: (sha: string) => Promise<Result<string, string>>;
	55	readonly preflight: (dir: string) => Promise<Verdict>;
	56	/** Let go of the port and drain, once the new tree is well. */
	57	readonly drain: () => void;
	58	readonly probe?: Readonly<Record<string, string>>;
	59	readonly budgetMs?: number;
	60	readonly now?: () => number;
	61	}
	62
	63	/** Why `dir`'s package.json cannot run without an install; undefined when it names no dependency. */
	64	export function dependencyChange(dir: string): string \| undefined {
	65	let pkg: Record<string, unknown>;
	66	try {
	67	pkg = JSON.parse(readFileSync(join(dir, "package.json"), "utf8")) as Record<string, unknown>;
	68	} catch {
	69	return "the tree has no readable package.json";
	70	}
	71	for (const field of DEP_FIELDS) {
	72	const deps = pkg[field];
	73	if (typeof deps === "object" && deps !== null && Object.keys(deps).length > 0)
	74	return `${field} names ${Object.keys(deps).slice(0, 3).join(", ")}: a dependency change needs a new image, which is the base this swaps over`;
	75	}
	76	return undefined;
	77	}
	78
	79	export function swapper(i: SwapperInput): Deployer {
	80	const now = i.now ?? Date.now;
	81	/** Set once this process has handed over: what it then reports as running, so the keeper does not promote again. */
	82	let handedTo: string \| undefined;
	83	const admitted = new Set<string>();
	84	/** Commits refused for good (their tree does not change), with the reason. */
	85	const refused = new Map<string, string>();
	86
	87	const here = (): string => {
	88	const commit = i.commit();
	89	if (commit === null \|\| !/^[0-9a-f]{40}$/.test(commit)) throw new Error("the running commit is not stamped, so there is no known good tree to name");
	90	return `tree/${commit}:${i.version()}`;
	91	};
	92
	93	/** Keep the running tree where a rollback finds it. */
	94	function pin(): void {
	95	const commit = i.commit();
	96	if (commit === null \|\| !/^[0-9a-f]{40}$/.test(commit)) return;
	97	const target = releaseDir(i.dataDir, commit);
	98	if (existsSync(target)) {
	99	writeFileSync(join(target, LIVE_MARK), "");
	100	return;
	101	}
	102	const staging = `${target}.pin-${process.pid}.partial`;
	103	rmSync(staging, { recursive: true, force: true });
	104	mkdirSync(staging, { recursive: true });
	105	for (const part of TREE_PARTS)
	106	if (existsSync(join(i.treeRoot, part))) cpSync(join(i.treeRoot, part), join(staging, part), { recursive: true });
	107	writeFileSync(join(staging, LIVE_MARK), "");
	108	renameSync(staging, target);
	109	}
	110
	111	const refuse = (sha: string, why: string): Result<string, string> => {
	112	refused.set(sha, why);
	113	console.error(`keeper: ${sha.slice(0, 12)} is not swapped in: ${why}`);
	114	return err(why);
	115	};
	116
	117	async function prepare(sha: string, forward: boolean): Promise<Result<string, string>> {
	118	const before = forward ? refused.get(sha) : undefined;
	119	if (before !== undefined) return err(before);
	120	const out = await i.checkout(sha);
	121	if (!out.ok) return err(`checkout failed: ${out.error}`);
	122	if (!forward) return out;
	123	if (admitted.has(sha)) return out;
	124	const deps = dependencyChange(out.value);
	125	if (deps !== undefined) return refuse(sha, deps);
	126	const verdict = await i.preflight(out.value);
	127	if (!verdict.ok) return refuse(sha, `preflight failed: ${verdict.line}`);
	128	admitted.add(sha);
	129	return out;
	130	}
	131
	132	/** The guard of `--rollback`: a tree that knows fewer migrations than the live database has applied does not run on it. */
	133	async function schemaRefusal(sha: string, dir: string): Promise<string \| undefined> {
	134	const known = await releaseSchema(dir);
	135	const live = liveSchema(i.dataDir);
	136	if (known === undefined) return live === 0 ? undefined : `${sha} has no readable schema; not rolling back to it`;
	137	return known < live ? `${sha} predates schema ${live}; rolling back would need a down-migration` : undefined;
	138	}
	139
	140	return {
	141	async image() {
	142	return handedTo ?? here();
	143	},
	144	refFor: (version, sha) => `tree/${sha}:${version}`,
	145	async admits(sha) {
	146	// Nothing is cut that this process could not swap in: the same refusals as the swap itself.
	147	const blocked = swapBlocked();
	148	if (blocked !== undefined) return blocked;
	149	const r = await prepare(sha, true);
	150	return r.ok ? undefined : r.error;
	151	},
	152	async setImage(ref) {
	153	const m = REF.exec(ref);
	154	if (m === null) throw new Error("not a tree reference");
	155	const [, sha, version] = m as unknown as [string, string, string];
	156	if (handedTo === ref) return;
	157	// A tree that has served (pinned when it was swapped away from) is a rollback and is not checked again; any other is admitted, as at the cut.
	158	const backward = existsSync(join(releaseDir(i.dataDir, sha), LIVE_MARK));
	159	const dir = await prepare(sha, !backward);
	160	if (!dir.ok) throw new Error(dir.error);
	161	if (backward) {
	162	const stopped = await schemaRefusal(sha, dir.value);
	163	if (stopped !== undefined) throw new Error(stopped);
	164	}
	165	const turn = beginSwap();
	166	if (!turn.ok) throw new Error(turn.line);
	167	let swapped = false;
	168	try {
	169	pin();
	170	const env = { ...i.env, FORGE_DATA: i.dataDir };
	171	const started = await startRelease({
	172	dir: dir.value,
	173	sha,
	174	port: i.port,
	175	env,
	176	version,
	177	...(i.probe === undefined ? {} : { probe: i.probe }),
	178	...(i.budgetMs === undefined ? {} : { budgetMs: i.budgetMs }),
	179	});
	180	if (!started.ok) throw new Error(started.line);
	181	swapped = true;
	182	} finally {
	183	endSwap(swapped);
	184	}
	185	handedTo = ref;
	186	i.drain();
	187	},
	188	verified(version) {
	189	const commit = i.commit();
	190	const base = i.env["FORGE_BASE_VERSION"] \|\| i.env["FORGE_VERSION"] \|\| null;
	191	if (commit === null \|\| base === null) return;
	192	keepGood(i.dataDir, i.treeRoot, version, commit, now(), base);
	193	},
	194	};
	195	}
		@@ -3,12 +3,17 @@
3	3	* repository for main and the tags, the merge queue for "mid-landing", the
4	4	* kube API for the Deployment, and the public URL for the probes.
5	5	*
6		* On when `FORGE_RELEASE_KEEPER=on`, `FORGE_REPO` names a repository here,
7		* and the forge runs in a cluster. The Deployment and its container are
8		* `FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER` (both default
9		* `forge`); the pod's service account needs `get` and `patch` on it.
	6	* Two modes. `FORGE_RELEASE_KEEPER=on`, with `FORGE_REPO` naming a
	7	* repository here and the forge in a cluster, patches a Deployment: its name
	8	* and container are `FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER`
	9	* (both default `forge`), and the pod's service account needs `get` and
	10	* `patch` on it.
	11	*
	12	* `FORGE_RELEASE_KEEPER=swap` (#8042) needs no cluster and wants
	13	* `FORGE_SELF_SWAP=1`: the keeper swaps main's tree in place (`swapper.ts`).
10	14	*/
11	15
	16	import { join } from "node:path";
12	17	import type { App } from "../app.ts";
13	18	import { originPush } from "../actions/branches.ts";
14	19	import { serviceAccount } from "../cluster/jobs.ts";
		@@ -22,7 +27,10 @@ import { repoDir } from "../git/repos.ts";
22	27	import { ownProbeHeaders } from "../kernel/probe.ts";
23	28	import { running } from "../kernel/version.ts";
24	29	import { compareVersions, readTags, recordFor } from "../release/shipped.ts";
	30	import { checkoutRelease, preflightRelease } from "../release/checkout.ts";
	31	import { handover } from "../release/swap.ts";
25	32	import { kubeDeployer } from "./deploy.ts";
	33	import { swapper } from "./swapper.ts";
26	34	import type { KeeperPorts } from "./keeper.ts";
27	35	import { timedGet, verifyUnderLoad } from "./verify.ts";
28	36
		@@ -31,16 +39,26 @@ const OID = /^[0-9a-f]{40}$/;
31	39	const VERSION = /^\d+\.\d+\.\d+$/;
32	40	const ZERO = "0".repeat(40);
33	41
34		export type KeeperEnv = { on: true; deployment: string; container: string } \| { on: false; why: string };
	42	export type KeeperEnv =
	43	\| { on: true; mode: "kube"; deployment: string; container: string }
	44	\| { on: true; mode: "swap" }
	45	\| { on: false; why: string };
	46
	47	/** In swap mode a release is only the swap, so the cadence is a minute, not half an hour. */
	48	export const SWAP_INTERVAL_MS = 60_000;
35	49
36	50	/** Whether the keeper runs, from the environment. */
37	51	export function keeperEnv(env: Record<string, string \| undefined>): KeeperEnv {
	52	if (env.FORGE_RELEASE_KEEPER === "swap") {
	53	if (env.FORGE_SELF_SWAP !== "1") return { on: false, why: "FORGE_RELEASE_KEEPER=swap needs FORGE_SELF_SWAP=1" };
	54	return { on: true, mode: "swap" };
	55	}
38	56	if (env.FORGE_RELEASE_KEEPER !== "on") return { on: false, why: "FORGE_RELEASE_KEEPER is not on" };
39	57	if (!env.KUBERNETES_SERVICE_HOST) return { on: false, why: "not running in a cluster" };
40	58	const deployment = env.FORGE_KEEPER_DEPLOYMENT \|\| "forge";
41	59	const container = env.FORGE_KEEPER_CONTAINER \|\| "forge";
42	60	if (!LABEL.test(deployment) \|\| !LABEL.test(container)) return { on: false, why: "FORGE_KEEPER_DEPLOYMENT and FORGE_KEEPER_CONTAINER must be DNS labels" };
43		return { on: true, deployment, container };
	61	return { on: true, mode: "kube", deployment, container };
44	62	}
45	63
46	64	/** The first active site admin that is a person: the account the page load signs in as. */
		@@ -56,17 +74,33 @@ export function loadPages(repo: Pick<Repo, "ownerName" \| "name">): string[] {
56	74	return ["/boards", `/${encodeURIComponent(repo.ownerName)}/${encodeURIComponent(repo.name)}`, "/explore", "/"];
57	75	}
58	76
59		export function keeperPorts(app: App, input: { forgeUrl: string; deployment: string; container: string }): KeeperPorts \| string {
	77	export function keeperPorts(app: App, input: { forgeUrl: string; deployment: string; container: string; swap?: boolean }): KeeperPorts \| string {
60	78	const named = app.config.forgeRepo;
61	79	if (named === undefined) return "FORGE_REPO is not set";
62	80	const [owner = "", name = ""] = named.split("/");
63	81	const repo = getRepo(app.db, owner, name);
64	82	if (repo === undefined) return `FORGE_REPO names ${named}, which is no repository on this forge`;
65	83	const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
66		const deployer = kubeDeployer({ credentials: serviceAccount(), deployment: input.deployment, container: input.container });
	84	const deployer =
	85	input.swap === true
	86	? swapper({
	87	dataDir: app.config.dataDir,
	88	port: app.config.port ?? 0,
	89	treeRoot: join(import.meta.dir, "..", ".."),
	90	version: () => running().version,
	91	commit: () => running().commit,
	92	env: process.env,
	93	checkout: (sha) => checkoutRelease(app.config.dataDir, dir, sha),
	94	preflight: (tree) => preflightRelease(tree, app.config.dataDir),
	95	drain: () => handover.drain?.(),
	96	probe: ownProbeHeaders(app.config.dataDir),
	97	...(app.config.swapBudgetMs === undefined ? {} : { budgetMs: app.config.swapBudgetMs }),
	98	})
	99	: kubeDeployer({ credentials: serviceAccount(), deployment: input.deployment, container: input.container });
67	100	const probeHeaders = ownProbeHeaders(app.config.dataDir);
68	101	const at = (version: string) => runGit(["rev-parse", "--verify", "--quiet", `refs/tags/v${version}^{commit}`], { cwd: dir });
69	102	return {
	103	...(input.swap === true ? { minIntervalMs: SWAP_INTERVAL_MS } : {}),
70	104	db: app.db,
71	105	bus: app.bus,
72	106	repo: { id: repo.id, ownerId: repo.ownerId },
		@@ -99,7 +133,8 @@ export function keeperPorts(app: App, input: { forgeUrl: string; deployment: str
99	133	if (!made.ok) return made.error.stderr \|\| "git refused the tag";
100	134	const tagOid = await runGit(["rev-parse", `refs/tags/v${version}`], { cwd: dir });
101	135	const owner = getUserById(app.db, repo.ownerId);
102		if (!tagOid.ok \|\| owner === undefined) return undefined;
	136	// A swap needs no origin: the tag is the forge's own record of the release.
	137	if (!tagOid.ok \|\| owner === undefined \|\| input.swap === true) return undefined;
103	138	// A tag the origin did not take is taken back here too: the keeper drops the cut, and a
104	139	// tag left behind would be the newest, never live, and stall the cadence.
105	140	const untag = async (why: string) => (await runGit(["tag", "--delete", `v${version}`], { cwd: dir }), why);