3568570 · b36ac13 · 1d25e88 · 1716143 · fd29865 · 6ed2e67
@@ -455,6 +455,16 @@
What the crew's tokens bought is on the board (`/board/returns`), in MCP `return
The terminal's palette has `Signed in` for anyone signed in: your browsers, tokens and this terminal's key, with when each was last seen; `d` signs the selected browser out after a confirm, as `/settings` and MCP `sessions` do, and on a token revokes it after a confirm (Keep is the default) through the same `deleteToken` action as Settings and the API, so it stops working at once (`docs/CREW.md`, "Where you are signed in").
The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases on the forge's own one-minute timer, with no model. It is off until `FORGE_RELEASE_KEEPER=on`, and the pod's service account may `get` and `patch` the forge Deployment.
- **Cut:** it tags main's head as `v<patch + 1>` when main has moved past a live tag, nothing is in flight, the merge queue is still, 30 minutes have passed since its last cut and no hold stands.
- **Verify:** it records the known-good version and image before the tag, sets the Deployment's image, and has the new version spend ten minutes under page load probing `/healthz` each second.
- **Roll back and hold:** three bad probes in a row hold further cuts with the numbers and set the recorded known-good image back by name, never "undo". A pull saying `release hold <N>` lets the fix through, and that release verifying lifts the hold.
- **Owner's hold:** set and lifted from the card, the terminal (`h`) or MCP `release_hold`, by a person who may roll back, never an agent.
- **Where to see it:** the **Release keeper** card on `/releases`, `Release keeper` in the terminal's palette, the **Maintenance** board under `/boards`, and MCP `release_keeper`, each gated as `/releases` is: a guest sees none of it.
How it works and how to turn it on: [`docs/features/release-keeper.md`](docs/features/release-keeper.md).
`/changelog` lists the last 30 releases of `FORGE_REPO` (public only) with their pulls, for anyone; the terminal's palette has `Changelog` for the same list. In the terminal, `/` takes a word and keeps the releases whose version or pull titles contain it (`esc` clears it); Enter on a release lists its pulls, and Enter on a pull opens it (`docs/features/changelog.md`).
A repository admin cuts a wave from two doors, both through `cutWaveFor` (`src/actions/cut_wave.ts`): Settings → *Cut a wave*, and `c` on the terminal's waves screen, which says how many pulls would go in, asks `cut wave N? y/n` (`y` and `n` choose outright) and shows the draft pull's number (Enter opens it). Each opens a draft pull listing the pulls merged since the last wave; nothing merged, an earlier wave that does not parse, or a viewer who is not an admin is refused in the action's words, writing nothing.
@@ -138,6 +138,8 @@
for a tunnel and a forge that sleeps.
| `queue_gates` | `{ repo, limit? }`: the merge queue's recent gates, newest first (`limit` 1–100, default 20), as `GET …/queue/gates` answers: each gate's pulls, state, reason, counts and failing tests by name with their first error lines, a test red in two gates that share no pull flagged `flaky` |  |
| `quotas` | `{ key?, value?, dryRun? }` (forge admin): the per-account quotas `repositories`, `apps`, `storage` (MB), `previews`, `runs`, `checks`, each a number or null for none. `repositories` bounds repos on create, fork and copy; `apps` the apps made; `storage` the MB of repos plus app data, asked on push, create and fork; `previews` live previews; `runs` concurrent agent runs; `checks` concurrent checks. Each is asked before the thing is made or started and a forge admin is never limited; with `key` and `value` (`""` clears), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token | yes |
| `read_file` | a file (utf-8, or base64 for binary) or a directory listing at a ref, as GET …/contents; MP4/MOV/M4A files also carry about: {kind, seconds, width?, height?, says} |  |
| `release_hold` | `{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent | yes |
| `release_keeper` | `{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases |  |
| `releases` | `{shipped: true}`: what went live, for anyone: `version`, `commit`, `live` (newest first: `repo`, `number`, `title`, `at`) and `onItsWay` (landed, not yet in the running commit), the list /changelog shows. `{}`: the forge's kept releases newest first: short sha, version, the pull that landed it, when, and for the live one `live: true` and `since`; for whoever may read /releases, others get its refusal; no rollback, a person does that on the page |  |
| `repo` | `{ repo }`: one repository's description, visibility, autonomy, `audience { level, sentence }` and `provenance` (where a migrated one came from, `null` when hidden), the same as `GET /api/v1/repos/:owner/:name` |  |
| `repo_labels` | `{ repo, action?, name?, color?, id? }`: list (`list`, the default, paged by `limit` 1–200 and `cursor`, as `{ items, next_cursor }`; a read-only token may), create (`create`, `name`, `color` six hex digits), rename or recolour (`update`, `id`, `name` and/or `color`, as the labels page) or remove (`remove`, `id`) the repository's labels, as `GET`/`POST …/labels` and `DELETE …/labels/:id`; create, update and remove need an admin token and a repository admin | yes |
@@ -0,0 +1,102 @@
# Release keeper
The release keeper (#7557) cuts, promotes, verifies and, when it has to,
rolls back the forge's own releases. It is plain code on the forge's own
timer, with no model, so releases keep going out when no one is at the
keyboard. It does what the steward's `release.sh`, `verify-after.sh` and
`verify-release.sh` did. The code is `src/keeper/`.
## What it does, each minute
1. **Cut on cadence** (`keeper/cadence.ts`). It cuts when all of these hold:
   - main has moved past the newest `v*` tag;
   - no keeper release is rolling or verifying;
   - the newest tag is live, or the keeper rolled it back;
   - no merge-queue gate is running;
   - 30 minutes have passed since the last cut;
   - no hold stands.
   Before anything moves, it records the known-good version and the exact
   image the Deployment runs. Then it makes the annotated tag
   `v<version>` (patch + 1) on main's head, with the same record
   `scripts/cut-release.ts` writes, and pushes the tag to the repository's
   origin when it has one. The cut is a tag, never a commit, so it voids no
   gate in flight (#6640). The row is written before the tag: if the forge
   stops between the two, the next tick sends the tag, and a refused tag
   drops the row (and the local tag) so nothing waits on a release that
   never left.
2. **Promote.** It sets the Deployment's forge container to
   `<same image repository>:<version>`. The chart rolls with
   `maxSurge 1, maxUnavailable 0`. If the image is not built yet, the new
   pod waits on its pull while the old one keeps serving. While the roll
   runs, the old forge probes `/healthz` through the public URL once a
   second and counts every probe that got no 200.
3. **Verify under page load** (`keeper/verify.ts`). The new forge sees
   that it is the version in flight and spends 10 minutes on two things at
   once:
   - it fetches `/boards`, the forge's repository page, `/explore` and `/`,
     one at a time, signed in as the first site admin (a session made for
     the verify and deleted after it);
   - it probes `/healthz` once a second.
   A probe is bad when it is not a 200 or takes over 10 s. Three bad probes
   in a row fail the release.
4. **Verified live**, or **roll back and hold**. On a failure, the keeper
   first writes a hold with the version, the reason and the probe numbers.
   Then it sets the Deployment to the known-good image recorded at the cut,
   by name (`keeper/rollback.ts`). There is no "undo to previous": on
   2026-10-27 a `rollout undo` put the broken version back for two minutes.
   A release that is not live 3 h after its cut is rolled back the same way.
Every step is an event on the outbox: `release.cut`, `release.released`,
`release.verified_live`, `release.rolled_back` and `release.held`.
**Seconds dark** counts the probe-seconds without a 200, through the roll
and the verify together. The target is 0. A release with any dark seconds
files one issue, `Release <version> was dark for <n> s`.
## Holds
- **A failed verify's hold** blocks every cut. Land a pull whose title or
  body says the hold by number (`release hold 3`). Naming the version alone
  (`0.7.284`) does not count: changelogs and revert notes mention it too.
  The keeper then cuts the fix, and the hold is lifted when that release
  verifies.
- **An owner's hold** is set with MCP `release_hold {reason}`, with the Hold form on the `/releases` card, or with `h` on the terminal screen, and lifted
  (`release_hold {lift: <id>}`, the card's Lift button, `h` then `lift N`) — one function, `holdAs`/`liftAs`, behind all three.
  It needs a person who may roll back (MCP: on an admin-scope token); an agent never can.
## Where to see it
- The terminal: **Release keeper** in the command palette shows the same
  read as the card, for the same people.
- `/releases`: the **Release keeper** card says what holds the next cut and
  lists each release with its stage, the known-good version, the roll's
  non-200s, the probes and pages, the worst latency and the seconds dark.
- The **Maintenance** board (`/boards`, from the `keeperReleases` source)
  charts seconds dark and worst latency per release, and releases by
  outcome.
- MCP `release_keeper` returns the same data as JSON.
## Turning it on
The keeper is off unless all of these are true:
- `FORGE_RELEASE_KEEPER=on`;
- `FORGE_REPO` names a repository on this forge;
- the forge runs in a cluster.
`FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER` name the Deployment
and its container (both default `forge`). The pod's service account needs
`get` and `patch` on that Deployment; the chart does not grant them. The
boot log says why it is off.
Guardrail: the keeper writes only a tag, the Deployment's image, its own
rows and issues. It has no account that may land a pull or push main.
Rollback is the one action it takes without a canary, because the image it
sets was already live.
Assumption: the keeper promotes by setting the image directly, as the
steward's rollback did. If Flux reconciles the image back, the keeper sets
it again on its next tick while the release is rolling. If it never takes,
the 3 h deadline rolls it back and holds.
@@ -285,6 +285,28 @@
export const SOURCES: Readonly<Record<string, Source>> = {
      reports: "reports kept that day (production only)",
    },
  },
  // The release keeper's releases (#7557), one row each: what the probes
  // counted through the roll and the ten minutes under page load. Seconds
  // dark is the target-zero number. The stop reason is free text, not listed.
  keeperReleases: {
    table: "keeper_release",
    owner: "admin",
    time: "cut_at",
    columns: {
      version: "text", stage: "text", cut_at: "time", live_at: "time", finished_at: "time",
      roll_probes: "number", roll_non200: "number", probes: "number", pages: "number",
      bad_in_row: "number", worst_ms: "number", seconds_dark: "number",
    },
    docs: {
      stage: "rolling, verifying, verified (verified live) or rolled_back",
      roll_non200: "/healthz probes, one a second, that got no 200 while the roll ran (target 0 with surge)",
      probes: "/healthz probes in the ten minutes under page load",
      pages: "pages fetched as an admin under load, one at a time",
      bad_in_row: "the most bad probes in a row (non-200 or over 10 s); 3 fails the release",
      worst_ms: "the slowest probe or page, ms",
      seconds_dark: "probe-seconds with no 200, roll and verify together (target 0)",
    },
  },
  // Forges made by the one-click path (#2914): one row per spawn event
  // (`actions/spawn.ts`, `spawn/live.ts`). The forge's name and the failure's
  // words are not listed; the step is one of a fixed few.
@@ -1,5 +1,5 @@
/**
 * The boards the forge offers a new install: Usage, Serving, Crew, Releases, Apps, App restarts, Security, Throughput, Forges and Reach. They are plain
 * The boards the forge offers a new install: Usage, Serving, Crew, Releases, Maintenance, Apps, App restarts, Security, Throughput, Forges and Reach. They are plain
 * boards in `forge/boards` once written, so a person edits or deletes them
 * like any other; `.offered` (a JSON list of slugs, committed with the
 * boards) records what was offered so a deletion sticks and an edit is
@@ -83,6 +83,18 @@
export const STARTING_BOARDS: Readonly<Record<string, Board>> = {
      panel("Decider time per hour (ms)", "line", { source: "decisions", aggregate: "avg", column: "ms", groupBy: HOUR, series: "kind" }, 2),
    ],
  },
  // The release keeper (#7557): every release dark for 0 s is the target.
  maintenance: {
    title: "Maintenance",
    range: "30d",
    refresh: "1m",
    panels: [
      panel("Seconds dark per release", "bar", { source: "keeperReleases", aggregate: "max", column: "seconds_dark", groupBy: "version" }, 2),
      panel("Worst latency per release (ms)", "bar", { source: "keeperReleases", aggregate: "max", column: "worst_ms", groupBy: "version" }, 2),
      panel("Releases by outcome", "bar", { source: "keeperReleases", aggregate: "count", groupBy: "stage" }),
      panel("Seconds dark (30 days)", "number", { source: "keeperReleases", aggregate: "sum", column: "seconds_dark" }),
    ],
  },
  apps: {
    title: "Apps",
    range: "7d",
@@ -177,6 +177,8 @@
export function classifiedAudience(kind: string): EventAudience | undefined {
    kind === "repo.crew_outsiders" ||
    // A clog starting names the forge's own pile-up: the owner's to hear (#6524).
    kind.startsWith("clog.") ||
    // The release keeper's steps name images and probe numbers: the forge owner's (`keeper/keeper.ts`, #7557).
    kind.startsWith("release.") ||
    // A file held against new work is its admins' book-keeping (#6522).
    kind === "repo.file_held" ||
    kind === "repo.file_released"
@@ -0,0 +1,1 @@
e24c6b4f77b34f5ba3d6f043f2da2788dd0b2e239ebfd5970e12c3b99b1016be
@@ -0,0 +1,39 @@
// The release keeper (#7557): one row per release it cut, with the
// known-good image it rolls back to and what the probes counted, and the
// holds that stop further cuts. `kind` is `failed` (a verify that failed,
// lifted by a verified release after a pull naming it lands) or `owner`
// (set and lifted by an admin).
export default `
  CREATE TABLE keeper_release (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    version TEXT NOT NULL UNIQUE,
    sha TEXT NOT NULL,
    known_good_version TEXT NOT NULL,
    known_good_image TEXT NOT NULL,
    stage TEXT NOT NULL,
    cut_at TEXT NOT NULL,
    live_at TEXT,
    finished_at TEXT,
    roll_probes INTEGER NOT NULL DEFAULT 0,
    roll_non200 INTEGER NOT NULL DEFAULT 0,
    probes INTEGER NOT NULL DEFAULT 0,
    pages INTEGER NOT NULL DEFAULT 0,
    bad_in_row INTEGER NOT NULL DEFAULT 0,
    worst_ms INTEGER NOT NULL DEFAULT 0,
    seconds_dark INTEGER NOT NULL DEFAULT 0,
    reason TEXT NOT NULL DEFAULT ''
  );
  CREATE TABLE keeper_hold (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    kind TEXT NOT NULL,
    version TEXT NOT NULL,
    reason TEXT NOT NULL,
    probes INTEGER NOT NULL DEFAULT 0,
    bad_in_row INTEGER NOT NULL DEFAULT 0,
    worst_ms INTEGER NOT NULL DEFAULT 0,
    created_at TEXT NOT NULL,
    cause_pull INTEGER,
    lifted_at TEXT,
    lifted_by TEXT
  );
`;
@@ -108,6 +108,8 @@
export { tool as pulls } from "./pulls.ts";
export { tool as queue_gates } from "./queue_gates.ts";
export { tool as quotas } from "./quotas.ts";
export { tool as read_file } from "./read_file.ts";
export { tool as release_hold } from "./release_hold.ts";
export { tool as release_keeper } from "./release_keeper.ts";
export { tool as releases } from "./releases.ts";
export { tool as repo } from "./repo.ts";
export { tool as repo_labels } from "./repo_labels.ts";
@@ -0,0 +1,24 @@
import { holdAs, liftAs, MAX_REASON } from "../../../keeper/hold.ts";
import { bad, good, str, type Tool } from "../shared.ts";
export const tool: Tool = {
  name: "release_hold",
  doc: "`{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent",
  description: "Hold or release the release keeper's cuts (admin).",
  inputSchema: {
    type: "object",
    properties: { reason: { type: "string", maxLength: MAX_REASON }, lift: { type: "integer", minimum: 1 } },
  },
  writes: true,
  needsAdminScope: true,
  run(ctx, user, args) {
    const id = args["lift"];
    if (id !== undefined) {
      if (typeof id !== "number") return bad("invalid", "lift is a hold's id");
      const lifted = liftAs(ctx.app, user, id);
      return lifted.ok ? good({ lifted: lifted.value }) : bad(lifted.error, lifted.message);
    }
    const held = holdAs(ctx.app, user, str(args["reason"]));
    return held.ok ? good({ hold: held.value }) : bad(held.error, held.message);
  },
};
@@ -0,0 +1,22 @@
import { mayRead } from "../../../release/gate.ts";
import { blocking, listHolds, standingHolds } from "../../../keeper/hold.ts";
import { listKeeperReleases } from "../../../keeper/record.ts";
import { bad, good, type Tool } from "../shared.ts";
export const tool: Tool = {
  name: "release_keeper",
  doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases",
  description: "The release keeper's releases, holds and what blocks the next cut.",
  inputSchema: { type: "object", properties: {} },
  writes: false,
  run(ctx, user) {
    if (!mayRead(ctx.app, user))
      return bad("forbidden", "the release history is for the forge's administrators and its repository's readers");
    const db = ctx.app.db;
    return good({
      blocked: blocking(standingHolds(db)) ?? null,
      releases: listKeeperReleases(db),
      holds: listHolds(db),
    });
  },
};
@@ -163,7 +163,7 @@
import {
} from "../web/pages/backups.ts";
import { apiPage, appApiPage } from "../web/pages/api.ts";
import { changelogPage } from "../web/pages/changelog.ts";
import { releasesPage, rollbackSubmit } from "../web/pages/releases.ts";
import { releaseHoldSubmit, liftSubmit, releasesPage, rollbackSubmit } from "../web/pages/releases.ts";
import {
  cloudAdd,
  cloudAudit,
@@ -795,6 +795,8 @@
export function buildRouter(): Router<Handler> {
  router.add("GET", "/changelog", changelogPage);
  router.add("GET", "/:owner/:name/api", appApiPage);
  router.add("POST", "/releases/:sha/rollback", rollbackSubmit);
  router.add("POST", "/releases/hold", releaseHoldSubmit);
  router.add("POST", "/releases/hold/:id/lift", liftSubmit);
  router.add("GET", "/audit.csv", auditCsv);
  router.add("POST", "/audit/:owner/:name/probe", auditProbeSubmit);
  router.add("POST", "/new", newRepoSubmit);
@@ -0,0 +1,65 @@
/**
 * When the release keeper cuts (#7557): plain data in, one answer out.
 *
 *   cut ⇔ main ≠ the newest tag's commit
 *       ∧ no release of the keeper's is in flight (cut, rolling, verifying)
 *       ∧ the live version is the newest tag's, or the keeper rolled that
 *         tag back (nothing still promoting)
 *       ∧ no merge-queue gate is running (the queue is not mid-landing)
 *       ∧ now − the last cut ≥ the minimum interval
 *       ∧ no hold stands (a failed verify's, or an owner's)
 *
 * The first rule that refuses names itself, so the record says why not.
 * The version is the newest tag's with its patch number plus one.
 */
/** Two releases are at least this far apart: 30 minutes. */
export const MIN_INTERVAL_MS = 30 * 60_000;
export interface CadenceInput {
  /** main's head commit. */
  readonly main: string;
  /** The newest release tag: its version and the commit it names. */
  readonly newest: { readonly version: string; readonly sha: string } | undefined;
  /** The version this forge runs. */
  readonly live: string;
  /** The keeper rolled the newest tag back: it is over, not promoting. */
  readonly settled?: boolean;
  /** A keeper release not yet verified or rolled back. */
  readonly inFlight: boolean;
  /** Merge-queue gates running now. */
  readonly gating: number;
  /** When the keeper last cut, epoch ms. */
  readonly lastCutAt: number | undefined;
  /** The reason of a standing hold, when one stands. */
  readonly hold: string | undefined;
  readonly now: number;
  readonly minIntervalMs?: number;
}
export type Cadence = { readonly cut: true; readonly version: string } | { readonly cut: false; readonly why: string };
const SEMVER = /^(\d+)\.(\d+)\.(\d+)$/;
/** The next patch version: `0.7.283` → `0.7.284`; undefined when `version` is not major.minor.patch. */
export function nextVersion(version: string): string | undefined {
  const m = SEMVER.exec(version);
  if (m === null) return undefined;
  return `${m[1]}.${m[2]}.${Number(m[3]) + 1}`;
}
export function decideCut(input: CadenceInput): Cadence {
  const no = (why: string): Cadence => ({ cut: false, why });
  if (input.hold !== undefined) return no(`held: ${input.hold}`);
  if (input.newest === undefined) return no("no release tag to follow; cut the first one by hand");
  if (input.main === input.newest.sha) return no(`main is still ${input.newest.version}`);
  if (input.inFlight) return no("a release is still rolling or being verified");
  if (input.live !== input.newest.version && input.settled !== true) return no(`${input.newest.version} is not live yet (live: ${input.live})`);
  if (input.gating > 0) return no(`the merge queue is landing (${input.gating} gate${input.gating === 1 ? "" : "s"} running)`);
  const wait = input.minIntervalMs ?? MIN_INTERVAL_MS;
  if (input.lastCutAt !== undefined && input.now - input.lastCutAt < wait)
    return no(`the last cut was ${Math.floor((input.now - input.lastCutAt) / 60_000)} min ago; the interval is ${wait / 60_000} min`);
  const version = nextVersion(input.newest.version);
  if (version === undefined) return no(`${input.newest.version} is not major.minor.patch`);
  return { cut: true, version };
}
@@ -0,0 +1,89 @@
/**
 * The release keeper's hands on the cluster (#7557): read the image the
 * forge's Deployment runs, set it to a named image, and say whether a roll
 * to an image has finished. That is all. There is no "undo to previous":
 * on 2026-10-27 a `rollout undo` put the broken version back for two
 * minutes, because "previous" is whatever the controller last held. A
 * rollback here sets the image recorded before the cut, by name
 * (`keeper/rollback.ts`).
 *
 * Spoken over `fetch` to the kube API with the pod's service account
 * (`cluster/jobs.ts` `serviceAccount`), as a strategic-merge patch of the
 * one container's image, the same request `kubectl set image` sends.
 */
import type { Credentials } from "../cluster/jobs.ts";
export interface Deployer {
  /** The image the Deployment's forge container names now. */
  image(): Promise<string>;
  /** Name this image on the forge container; the Deployment rolls to it with surge. */
  setImage(image: string): Promise<void>;
}
/** An image reference: registry/path:tag or @digest, no space, no flag. */
const IMAGE = /^[a-z0-9][a-z0-9._\-/:]{0,254}(@sha256:[a-f0-9]{64})?$/;
const LABEL = /^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$/;
export const isImage = (s: string): boolean => IMAGE.test(s) && !s.includes("..");
/** The image `image` names with its tag replaced by `version`: `reg:5000/o/forge:0.7.1` → `reg:5000/o/forge:0.7.2`. */
export function imageAt(image: string, version: string): string {
  const repo = image.split("@")[0]!;
  const slash = repo.lastIndexOf("/");
  const colon = repo.lastIndexOf(":");
  return `${colon > slash ? repo.slice(0, colon) : repo}:${version}`;
}
/** The tag an image names, or "" when it names none. */
export function tagOf(image: string): string {
  const repo = image.split("@")[0]!;
  const colon = repo.lastIndexOf(":");
  return colon > repo.lastIndexOf("/") ? repo.slice(colon + 1) : "";
}
type Fetch = (url: string, init: RequestInit & { tls?: { ca: string } }) => Promise<Response>;
interface DeploymentJson {
  metadata?: { generation?: number };
  spec?: { replicas?: number; template?: { spec?: { containers?: { name?: string; image?: string }[] } } };
  status?: { observedGeneration?: number; replicas?: number; updatedReplicas?: number; availableReplicas?: number };
}
export function kubeDeployer(input: {
  credentials: Credentials;
  deployment: string;
  container: string;
  fetch?: Fetch;
}): Deployer {
  const { credentials: c, deployment, container } = input;
  if (!LABEL.test(deployment) || !LABEL.test(container) || !LABEL.test(c.namespace))
    throw new Error("deployment, container and namespace must be DNS labels");
  const call = input.fetch ?? (fetch as unknown as Fetch);
  const url = `${c.server}/apis/apps/v1/namespaces/${c.namespace}/deployments/${deployment}`;
  const auth = { authorization: `Bearer ${c.token}` };
  const tls = c.ca === undefined ? {} : { tls: { ca: c.ca } };
  const read = async (): Promise<DeploymentJson> => {
    const res = await call(url, { headers: auth, ...tls });
    if (!res.ok) throw new Error(`reading deployment ${deployment}: HTTP ${res.status}`);
    return (await res.json()) as DeploymentJson;
  };
  const imageOf = (d: DeploymentJson): string => {
    const found = d.spec?.template?.spec?.containers?.find((k) => k.name === container)?.image;
    if (found === undefined) throw new Error(`deployment ${deployment} has no container ${container}`);
    return found;
  };
  return {
    image: async () => imageOf(await read()),
    async setImage(image) {
      if (!isImage(image)) throw new Error("not an image reference");
      const res = await call(url, {
        method: "PATCH",
        headers: { ...auth, "content-type": "application/strategic-merge-patch+json" },
        body: JSON.stringify({ spec: { template: { spec: { containers: [{ name: container, image }] } } } }),
        ...tls,
      });
      if (!res.ok) throw new Error(`setting ${deployment} to ${image}: HTTP ${res.status}`);
    },
  };
}
@@ -0,0 +1,150 @@
/**
 * Release holds (#7557): while one stands, the keeper cuts nothing.
 *
 * - `failed`: a verify failed. It carries the version, the reason and the
 *   probe numbers. A pull that lands saying `release hold 4` is its cause —
 *   the hold by number, never the bare version, which changelogs and revert
 *   notes mention too. With a cause named the keeper may cut again; the
 *   hold is lifted when that release verifies.
 * - `owner`: an admin set it, with a reason; an admin lifts it.
 */
import type { Database } from "bun:sqlite";
import type { App } from "../app.ts";
import type { User } from "../core/users.ts";
import { running } from "../kernel/version.ts";
import { mayRollBack } from "../release/gate.ts";
export type HoldKind = "failed" | "owner";
export interface Hold {
  readonly id: number;
  readonly kind: HoldKind;
  readonly version: string;
  readonly reason: string;
  readonly probes: number;
  readonly badInRow: number;
  readonly worstMs: number;
  readonly createdAt: string;
  readonly causePull: number | null;
  readonly liftedAt: string | null;
  readonly liftedBy: string | null;
}
interface Row {
  id: number;
  kind: HoldKind;
  version: string;
  reason: string;
  probes: number;
  bad_in_row: number;
  worst_ms: number;
  created_at: string;
  cause_pull: number | null;
  lifted_at: string | null;
  lifted_by: string | null;
}
const toHold = (r: Row): Hold => ({
  id: r.id,
  kind: r.kind,
  version: r.version,
  reason: r.reason,
  probes: r.probes,
  badInRow: r.bad_in_row,
  worstMs: r.worst_ms,
  createdAt: r.created_at,
  causePull: r.cause_pull,
  liftedAt: r.lifted_at,
  liftedBy: r.lifted_by,
});
/** A reason is one line a person reads: at most this long. */
export const MAX_REASON = 500;
export function setHold(
  db: Database,
  input: { kind: HoldKind; version: string; reason: string; probes?: number; badInRow?: number; worstMs?: number; at: number },
): Hold {
  const reason = input.reason.replace(/[\u0000-\u001f\u007f]+/g, " ").trim().slice(0, MAX_REASON);
  return toHold(
    db
      .query<Row, [string, string, string, number, number, number, string]>(
        `INSERT INTO keeper_hold (kind, version, reason, probes, bad_in_row, worst_ms, created_at)
         VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) RETURNING *`,
      )
      .get(input.kind, input.version, reason, input.probes ?? 0, input.badInRow ?? 0, input.worstMs ?? 0, new Date(input.at).toISOString())!,
  );
}
/** The holds not lifted, oldest first. */
export function standingHolds(db: Database): Hold[] {
  return db.query<Row, []>(`SELECT * FROM keeper_hold WHERE lifted_at IS NULL ORDER BY id`).all().map(toHold);
}
/** What stops a cut, in one line; undefined when nothing does. A failed hold whose cause has landed lets the fix through. */
export function blocking(holds: readonly Hold[]): string | undefined {
  const h = holds.find((x) => x.kind === "owner" || x.causePull === null);
  if (h === undefined) return undefined;
  return h.kind === "owner"
    ? `the owner holds releases: ${h.reason}`
    : `${h.version} failed its verify (${h.reason}); waiting for a pull saying "release hold ${h.id}" to land`;
}
/** Does this pull's text say it is the hold's cause? Only `release hold N` does; a mere mention of the version does not. */
export function names(hold: Hold, text: string): boolean {
  return new RegExp(`(^|[^0-9a-z])release hold #?${hold.id}(?![0-9])`, "i").test(text);
}
export function setCause(db: Database, id: number, pull: number): void {
  db.query(`UPDATE keeper_hold SET cause_pull = ?2 WHERE id = ?1 AND cause_pull IS NULL`).run(id, pull);
}
export function lift(db: Database, id: number, by: string, at: number): boolean {
  return db.query(`UPDATE keeper_hold SET lifted_at = ?3, lifted_by = ?2 WHERE id = ?1 AND lifted_at IS NULL`).run(id, by, new Date(at).toISOString())
    .changes === 1;
}
/** A verified release lifts every failed hold whose cause has landed. */
export function liftFixed(db: Database, version: string, at: number): number {
  return db
    .query(`UPDATE keeper_hold SET lifted_at = ?2, lifted_by = ?1 WHERE kind = 'failed' AND cause_pull IS NOT NULL AND lifted_at IS NULL`)
    .run(`verified ${version}`, new Date(at).toISOString()).changes;
}
/** Every hold, newest first. */
export function listHolds(db: Database, limit = 20): Hold[] {
  return db.query<Row, [number]>(`SELECT * FROM keeper_hold ORDER BY id DESC LIMIT ?1`).all(limit).map(toHold);
}
export type OwnerHoldResult<T> = { ok: true; value: T } | { ok: false; error: "forbidden" | "invalid" | "not_found" | "refused"; message: string };
/**
 * An owner's hold, set by a person who may roll back (`release/gate.ts`):
 * the one function the MCP tool, the /releases card and the terminal call,
 * so the refusal, the sanitised reason and the event are the same in all
 * three. A refusal writes no hold and no event.
 */
export function holdAs(app: App, user: User | undefined, reason: string): OwnerHoldResult<Hold> {
  if (user === undefined || !mayRollBack(app, user))
    return { ok: false, error: "forbidden", message: "holding releases is for a person who may write the forge's repository and administer it, not an agent" };
  const clean = reason.replace(/[\u0000-\u001f\u007f]+/g, " ").trim();
  if (clean === "" || clean.length > MAX_REASON) return { ok: false, error: "invalid", message: `reason is one line of 1 to ${MAX_REASON} characters` };
  const hold = setHold(app.db, { kind: "owner", version: running().version, reason: clean, at: Date.now() });
  app.bus.append({ kind: "release.held", actorId: user.id, payload: { hold: hold.id, reason: hold.reason, owner: true } });
  return { ok: true, value: hold };
}
/** Lift an owner's hold by id; a failed verify's hold is lifted only by a verified release. */
export function liftAs(app: App, user: User | undefined, id: number): OwnerHoldResult<number> {
  if (user === undefined || !mayRollBack(app, user))
    return { ok: false, error: "forbidden", message: "holding releases is for a person who may write the forge's repository and administer it, not an agent" };
  if (!Number.isInteger(id) || id < 1) return { ok: false, error: "invalid", message: "lift is a hold's id" };
  const hold = standingHolds(app.db).find((h) => h.id === id);
  if (hold === undefined) return { ok: false, error: "not_found", message: "no standing hold has that id" };
  if (hold.kind !== "owner")
    return { ok: false, error: "refused", message: "a failed verify's hold is lifted by a verified release after a pull saying \"release hold N\" lands" };
  lift(app.db, id, user.username, Date.now());
  app.bus.append({ kind: "release.hold_lifted", actorId: user.id, payload: { hold: id } });
  return { ok: true, value: id };
}
@@ -0,0 +1,227 @@
/**
 * The release keeper (#7557): plain code, no model, on the forge's own
 * timer, so releases keep going out when no one is at the keyboard. It is
 * the steward's `release.sh`, `verify-after.sh` and `verify-release.sh`
 * moved inside:
 *
 * 1. **Cut** on cadence (`cadence.ts`): the known-good version and image
 *    are recorded, then the annotated tag `v<version>` is made on main's
 *    head — a tag, never a commit, so no gate in flight is voided (#6640).
 * 2. **Promote**: the Deployment is set to the new version's image; it
 *    rolls with surge (`maxSurge 1, maxUnavailable 0`). The old forge
 *    probes `/healthz` each second while it still serves (`rollProbe`),
 *    so a roll that went dark says how long.
 * 3. **Verify**: the new forge, finding itself the version in flight,
 *    spends ten minutes under page load (`verify.ts`).
 * 4. **Verified live**, or **roll back** to the named known-good image
 *    (`rollback.ts`; there is no undo) and **hold** with the numbers
 *    (`hold.ts`). A release with seconds dark files one issue.
 *
 * Each step is an event on the outbox: `release.cut`, `release.released`,
 * `release.verified_live`, `release.rolled_back`, `release.held`.
 *
 * Guardrail: the keeper's only writes are a tag, the Deployment's image,
 * its own rows, and issues. It holds no account that may land a pull or
 * push main, and it has no code path to either.
 */
import type { Database } from "bun:sqlite";
import type { EventBus } from "../core/events.ts";
import { createIssue } from "../core/issues.ts";
import { decideCut } from "./cadence.ts";
import { imageAt, type Deployer } from "./deploy.ts";
import { blocking, liftFixed, names, setCause, setHold, standingHolds, type Hold } from "./hold.ts";
import { countRollProbe, dropCut, finish, getKeeperRelease, inFlight, keepTally, listKeeperReleases, markLive, recordCut, type KeeperRelease } from "./record.ts";
import { rollBackToNamed } from "./rollback.ts";
import type { Probe, Tally, Verdict } from "./verify.ts";
/** A release not live this long after its cut never rolled: 3 h (the tag's check may take 2 h, the build some more). */
export const ROLL_DEADLINE_MS = 3 * 3_600_000;
export interface KeeperPorts {
  readonly db: Database;
  readonly bus: EventBus;
  /** The forge's own repository, for events and the issues the keeper files. */
  readonly repo: { readonly id: string; readonly ownerId: string };
  readonly now: () => number;
  /** The version this process runs. */
  readonly live: () => string;
  readonly main: () => Promise<string | undefined>;
  readonly newest: () => Promise<{ version: string; sha: string } | undefined>;
  /** Merge-queue gates running now. */
  readonly gating: () => number;
  readonly deployer: Deployer;
  /** Make the annotated tag `v<version>` on `sha` and send it where the release workflow starts; why not. */
  readonly tag: (version: string, sha: string) => Promise<string | undefined>;
  /** One `/healthz` probe through the public URL. */
  readonly probe: () => Promise<Probe>;
  /** Ten minutes under page load. */
  readonly verify: (onProbe: (t: Tally) => void) => Promise<Verdict>;
  readonly minIntervalMs?: number;
}
/** Pulls of the forge's repository merged since `since`: number, title and body. */
function landedSince(db: Database, repoId: string, since: string): { number: number; text: string }[] {
  return db
    .query<{ number: number; title: string; body: string | null }, [string, string]>(
      `SELECT issue.number, issue.title, issue.body FROM pull JOIN issue ON issue.id = pull.issue_id
       WHERE issue.repo_id = ?1 AND pull.merged_at IS NOT NULL AND pull.merged_at > ?2 ORDER BY pull.merged_at LIMIT 500`,
    )
    .all(repoId, since)
    .map((r) => ({ number: r.number, text: `${r.title}\n${r.body ?? ""}` }));
}
/** Name the landed pull that fixes each failed hold. */
function findCauses(p: KeeperPorts, holds: readonly Hold[]): void {
  for (const h of holds) {
    if (h.kind !== "failed" || h.causePull !== null) continue;
    const fix = landedSince(p.db, p.repo.id, h.createdAt).find((x) => names(h, x.text));
    if (fix !== undefined) setCause(p.db, h.id, fix.number);
  }
}
/** One issue for a release that went dark, unless one by its title is open. */
function fileDark(p: KeeperPorts, r: KeeperRelease): number | undefined {
  if (r.secondsDark <= 0) return undefined;
  const title = `Release ${r.version} was dark for ${r.secondsDark} s`;
  const open = p.db
    .query<{ n: number }, [string, string]>(`SELECT COUNT(*) AS n FROM issue WHERE repo_id = ?1 AND title = ?2 AND state = 'open'`)
    .get(p.repo.id, title)!.n;
  if (open > 0) return undefined;
  const made = createIssue(p.db, {
    repoId: p.repo.id,
    authorId: p.repo.ownerId,
    title,
    body:
      `_Filed by the release keeper, not written by a person._\n\n` +
      `${r.version} (${r.sha.slice(0, 12)}) did not answer \`/healthz\` with a 200 for ${r.secondsDark} probe-seconds: ` +
      `${r.rollNon200} of ${r.rollProbes} during the roll, the rest of ${r.probes} under page load; worst ${r.worstMs} ms. ` +
      `The target is 0 s: the forge rolls with \`maxSurge 1, maxUnavailable 0\`. ` +
      `${r.stage === "rolled_back" ? `It was rolled back to ${r.knownGood.version}.` : "It stayed live."}\n\n` +
      `Files: src/keeper/keeper.ts, charts/app/templates/deployment.yaml`,
  });
  if (!made.ok) return undefined;
  p.bus.append({ kind: "issue.opened", repoId: p.repo.id, actorId: p.repo.ownerId, payload: { number: made.value.number } });
  return made.value.number;
}
/** The release each keeper is verifying; keyed by its ports, so one keeper (or one test) never sees another's. */
const verifying = new WeakMap<KeeperPorts, number>();
/** The verify of an in-flight release this process is the new version of, to its end. */
async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<string> {
  if (verifying.has(p)) return `verifying ${r.version}`;
  verifying.set(p, r.id);
  try {
    const verdict = await p.verify((t) => keepTally(p.db, r.id, t, r.rollNon200));
    keepTally(p.db, r.id, verdict.tally, r.rollNon200);
    const kept = getKeeperRelease(p.db, r.id)!;
    const numbers = { version: r.version, probes: kept.probes, pages: kept.pages, worstMs: kept.worstMs, secondsDark: kept.secondsDark, rollNon200: kept.rollNon200 };
    if (verdict.ok) {
      finish(p.db, r.id, "verified", "", p.now());
      const lifted = liftFixed(p.db, r.version, p.now());
      p.bus.append({ kind: "release.verified_live", repoId: p.repo.id, payload: { ...numbers, holdsLifted: lifted } });
      fileDark(p, getKeeperRelease(p.db, r.id)!);
      return `${r.version} verified live: ${kept.probes} probes, worst ${kept.worstMs} ms, ${kept.secondsDark} s dark`;
    }
    return await rollBack(p, kept, verdict.reason);
  } finally {
    verifying.delete(p);
  }
}
/** Hold with the numbers, then set the named known-good image. The hold first: this process may be the one replaced. */
async function rollBack(p: KeeperPorts, r: KeeperRelease, reason: string): Promise<string> {
  const hold = setHold(p.db, { kind: "failed", version: r.version, reason, probes: r.probes, badInRow: r.badInRow, worstMs: r.worstMs, at: p.now() });
  p.bus.append({ kind: "release.held", repoId: p.repo.id, payload: { version: r.version, hold: hold.id, reason, probes: r.probes, badInRow: r.badInRow, worstMs: r.worstMs } });
  const back = await rollBackToNamed(p.deployer, r.knownGood);
  const line = back.ok ? `rolled back to ${r.knownGood.version} (${back.image})` : `rollback failed: ${back.line}`;
  finish(p.db, r.id, "rolled_back", `${reason}; ${line}`, p.now());
  p.bus.append({ kind: "release.rolled_back", repoId: p.repo.id, payload: { version: r.version, to: r.knownGood.version, image: r.knownGood.image, ok: back.ok } });
  fileDark(p, getKeeperRelease(p.db, r.id)!);
  return `${r.version} failed its verify: ${reason}; ${line}; hold ${hold.id} set`;
}
/** One step of the keeper; the line says what it did or why it did nothing. */
export async function tick(p: KeeperPorts): Promise<string> {
  const holds = standingHolds(p.db);
  findCauses(p, holds);
  const flying = inFlight(p.db);
  if (flying !== undefined) {
    if (p.live() === flying.version) {
      markLive(p.db, flying.id, p.now());
      return verifyAndDecide(p, getKeeperRelease(p.db, flying.id)!);
    }
    // A cut recorded but never tagged (the process died between the two): tag it now, or drop it.
    if (flying.stage === "rolling" && (await p.newest())?.version !== flying.version) {
      const refused = await tagged(p, flying);
      if (refused !== undefined) return refused;
    }
    if (flying.stage === "rolling" && p.now() - Date.parse(flying.cutAt) > ROLL_DEADLINE_MS)
      return rollBack(p, flying, `${flying.version} was not live ${ROLL_DEADLINE_MS / 3_600_000} h after its cut`);
    // A promote that did not take (the API refused, or something set the image back) is sent again.
    const want = imageAt(flying.knownGood.image, flying.version);
    if (flying.stage === "rolling" && (await p.deployer.image()) !== want) await promote(p, flying.version, want);
    return `${flying.version} is ${flying.stage}`;
  }
  const main = await p.main();
  const newest = await p.newest();
  const last = listKeeperReleases(p.db, 1)[0];
  const decision = decideCut({
    main: main ?? "",
    newest,
    live: p.live(),
    settled: newest !== undefined && rolledBack(p.db, newest.version),
    inFlight: false,
    gating: p.gating(),
    lastCutAt: last === undefined ? undefined : Date.parse(last.cutAt),
    hold: blocking(standingHolds(p.db)),
    now: p.now(),
    ...(p.minIntervalMs === undefined ? {} : { minIntervalMs: p.minIntervalMs }),
  });
  if (!decision.cut || main === undefined) return decision.cut ? "main is unreadable" : decision.why;
  const image = await p.deployer.image();
  const knownGood = { version: p.live(), image };
  // The row first, the tag second: a crash between them leaves a `rolling` row
  // whose tag the next tick sends (`tagged`), never a tag the keeper has no row for.
  const r = recordCut(p.db, { version: decision.version, sha: main, knownGood, at: p.now() });
  const refused = await tagged(p, r);
  if (refused !== undefined) return refused;
  const sent = await promote(p, r.version, imageAt(image, r.version));
  return `cut ${r.version} at ${main.slice(0, 12)}; ${sent}; known good: ${knownGood.version}`;
}
/** Make the release's tag; when it is refused the cut is dropped and the line says why. */
async function tagged(p: KeeperPorts, r: KeeperRelease): Promise<string | undefined> {
  const refused = await p.tag(r.version, r.sha);
  if (refused !== undefined) {
    dropCut(p.db, r.id);
    return `not cut: ${refused}`;
  }
  p.bus.append({ kind: "release.cut", repoId: p.repo.id, payload: { version: r.version, sha: r.sha, knownGood: r.knownGood.version, image: r.knownGood.image } });
  return undefined;
}
/** Set the Deployment to the new version's image; it rolls with surge. */
async function promote(p: KeeperPorts, version: string, image: string): Promise<string> {
  try {
    await p.deployer.setImage(image);
  } catch (error) {
    return `promote to ${image} refused (${error instanceof Error ? error.message : String(error)}); sent again next tick`;
  }
  p.bus.append({ kind: "release.released", repoId: p.repo.id, payload: { version, image } });
  return `promoted to ${image}`;
}
/** The keeper rolled this version back: its tag is settled, not still promoting. */
function rolledBack(db: Database, version: string): boolean {
  return db.query<{ n: number }, [string]>(`SELECT COUNT(*) AS n FROM keeper_release WHERE version = ?1 AND stage = 'rolled_back'`).get(version)!.n > 0;
}
/** The old forge's probe while a release rolls: one `/healthz`, counted toward the roll. */
export async function rollProbe(p: Pick<KeeperPorts, "db" | "live" | "probe">): Promise<void> {
  const r = inFlight(p.db);
  if (r === undefined || r.stage !== "rolling" || p.live() === r.version) return;
  const got = await p.probe();
  countRollProbe(p.db, r.id, got.status === 200);
}
@@ -0,0 +1,139 @@
/**
 * The keeper's releases (#7557): one row per version it cut, from the cut
 * to `verified` or `rolled_back`, with the known-good image recorded before
 * the cut and what the probes counted. Plain rows in, plain rows out.
 *
 * Stages: `rolling` (tagged and promoted; the old forge still serves),
 * `verifying` (the new version answers; ten minutes under page load),
 * `verified` (verified live), `rolled_back` (set back to the known-good
 * image, a hold stands).
 */
import type { Database } from "bun:sqlite";
import type { Tally } from "./verify.ts";
export type Stage = "rolling" | "verifying" | "verified" | "rolled_back";
export const IN_FLIGHT: readonly Stage[] = ["rolling", "verifying"];
export interface KeeperRelease {
  readonly id: number;
  readonly version: string;
  readonly sha: string;
  readonly knownGood: { readonly version: string; readonly image: string };
  readonly stage: Stage;
  readonly cutAt: string;
  readonly liveAt: string | null;
  readonly finishedAt: string | null;
  readonly rollProbes: number;
  readonly rollNon200: number;
  readonly probes: number;
  readonly pages: number;
  readonly badInRow: number;
  readonly worstMs: number;
  readonly secondsDark: number;
  readonly reason: string;
}
interface Row {
  id: number;
  version: string;
  sha: string;
  known_good_version: string;
  known_good_image: string;
  stage: Stage;
  cut_at: string;
  live_at: string | null;
  finished_at: string | null;
  roll_probes: number;
  roll_non200: number;
  probes: number;
  pages: number;
  bad_in_row: number;
  worst_ms: number;
  seconds_dark: number;
  reason: string;
}
const toRelease = (r: Row): KeeperRelease => ({
  id: r.id,
  version: r.version,
  sha: r.sha,
  knownGood: { version: r.known_good_version, image: r.known_good_image },
  stage: r.stage,
  cutAt: r.cut_at,
  liveAt: r.live_at,
  finishedAt: r.finished_at,
  rollProbes: r.roll_probes,
  rollNon200: r.roll_non200,
  probes: r.probes,
  pages: r.pages,
  badInRow: r.bad_in_row,
  worstMs: r.worst_ms,
  secondsDark: r.seconds_dark,
  reason: r.reason,
});
const iso = (ms: number) => new Date(ms).toISOString();
/** The cut, written before the tag is pushed: the known-good image is on record before anything can need it. */
export function recordCut(
  db: Database,
  input: { version: string; sha: string; knownGood: { version: string; image: string }; at: number },
): KeeperRelease {
  return toRelease(
    db
      .query<Row, [string, string, string, string, string]>(
        `INSERT INTO keeper_release (version, sha, known_good_version, known_good_image, stage, cut_at)
         VALUES (?1, ?2, ?3, ?4, 'rolling', ?5) RETURNING *`,
      )
      .get(input.version, input.sha, input.knownGood.version, input.knownGood.image, iso(input.at))!,
  );
}
/** A cut whose tag was refused never happened: its row goes, so the cadence is not left waiting on it. */
export function dropCut(db: Database, id: number): void {
  db.query(`DELETE FROM keeper_release WHERE id = ?1 AND stage = 'rolling' AND live_at IS NULL`).run(id);
}
/** One more `/healthz` probe while the roll runs. */
export function countRollProbe(db: Database, id: number, ok: boolean): void {
  db.query(
    `UPDATE keeper_release SET roll_probes = roll_probes + 1, roll_non200 = roll_non200 + ?2,
       seconds_dark = seconds_dark + ?2 WHERE id = ?1 AND stage = 'rolling'`,
  ).run(id, ok ? 0 : 1);
}
/** The new version answers: the verify begins. Only one forge moves it, so a second caller gets false. */
export function markLive(db: Database, id: number, at: number): boolean {
  return db.query(`UPDATE keeper_release SET stage = 'verifying', live_at = ?2 WHERE id = ?1 AND stage = 'rolling'`).run(id, iso(at))
    .changes === 1;
}
/** The verify's numbers so far: the seconds dark it saw are added to the roll's. */
export function keepTally(db: Database, id: number, t: Tally, rollDark: number): void {
  db.query(
    `UPDATE keeper_release SET probes = ?2, pages = ?3, bad_in_row = ?4, worst_ms = ?5, seconds_dark = ?6 WHERE id = ?1`,
  ).run(id, t.probes, t.pages, t.worstRun, t.worstMs, rollDark + t.secondsDark);
}
export function finish(db: Database, id: number, stage: "verified" | "rolled_back", reason: string, at: number): void {
  db.query(`UPDATE keeper_release SET stage = ?2, reason = ?3, finished_at = ?4 WHERE id = ?1`).run(id, stage, reason, iso(at));
}
/** Newest first. */
export function listKeeperReleases(db: Database, limit = 20): KeeperRelease[] {
  return db.query<Row, [number]>(`SELECT * FROM keeper_release ORDER BY id DESC LIMIT ?1`).all(limit).map(toRelease);
}
/** The release the keeper has not finished with, if any. */
export function inFlight(db: Database): KeeperRelease | undefined {
  const row = db
    .query<Row, []>(`SELECT * FROM keeper_release WHERE stage IN ('rolling', 'verifying') ORDER BY id DESC LIMIT 1`)
    .get();
  return row === null ? undefined : toRelease(row);
}
export function getKeeperRelease(db: Database, id: number): KeeperRelease | undefined {
  const row = db.query<Row, [number]>(`SELECT * FROM keeper_release WHERE id = ?1`).get(id);
  return row === null ? undefined : toRelease(row);
}
@@ -0,0 +1,26 @@
/**
 * The one rollback the release keeper has (#7557): set the Deployment to
 * the known-good image recorded before the cut, by name. Never "the
 * previous revision": that is whatever the controller last held, and on
 * 2026-10-27 it was the broken version. This is the one action the keeper
 * takes without a canary, because the image it sets was already live.
 */
import { isImage, type Deployer } from "./deploy.ts";
export interface KnownGood {
  readonly version: string;
  readonly image: string;
}
export type RolledBack = { readonly ok: true; readonly image: string } | { readonly ok: false; readonly line: string };
export async function rollBackToNamed(deployer: Deployer, good: KnownGood): Promise<RolledBack> {
  if (!isImage(good.image)) return { ok: false, line: `the recorded known-good image of ${good.version} is not an image reference` };
  try {
    await deployer.setImage(good.image);
    return { ok: true, image: good.image };
  } catch (error) {
    return { ok: false, line: `could not set ${good.image}: ${error instanceof Error ? error.message : String(error)}` };
  }
}
@@ -0,0 +1,136 @@
/**
 * Proving a release live (#7557), as the steward's `verify-release.sh` did:
 * for ten minutes, the pages people open are fetched one at a time as an
 * admin (/boards, the forge's repository, /explore, /) while `/healthz` is
 * probed once a second. A probe is bad when it is not a 200 or takes over
 * 10 s; three bad probes in a row fail the release.
 *
 * Each probe stands for one second, so the seconds dark are the probes
 * that got no 200 — a slow 200 is slow, not dark. The same tally counts
 * the roll (`roll: true`), where the target is zero with surge.
 *
 * `tally` is plain data in, plain data out. `verifyUnderLoad` drives it
 * with the fetch, clock and sleep it is handed, so a test runs ten minutes
 * of probes in no time.
 */
/** A probe slower than this is bad: 10 s. */
export const SLOW_MS = 10_000;
/** This many bad probes in a row fail a release. */
export const FAIL_AFTER = 3;
/** How long a release is held under page load: 10 minutes. */
export const VERIFY_MS = 10 * 60_000;
/** One `/healthz` probe a second. */
export const PROBE_EVERY_MS = 1000;
export interface Probe {
  /** The HTTP status; 0 when nothing answered. */
  readonly status: number;
  readonly ms: number;
}
export interface Tally {
  readonly probes: number;
  readonly non200: number;
  /** Bad probes in a row now, and the most there have been. */
  readonly badInRow: number;
  readonly worstRun: number;
  readonly worstMs: number;
  readonly secondsDark: number;
  /** Pages fetched under load, and how many of them were not a 200. */
  readonly pages: number;
  readonly pagesBad: number;
}
export const EMPTY: Tally = { probes: 0, non200: 0, badInRow: 0, worstRun: 0, worstMs: 0, secondsDark: 0, pages: 0, pagesBad: 0 };
export const isBad = (p: Probe): boolean => p.status !== 200 || p.ms > SLOW_MS;
/** The tally with one more `/healthz` probe. */
export function tally(t: Tally, p: Probe): Tally {
  const bad = isBad(p);
  const badInRow = bad ? t.badInRow + 1 : 0;
  return {
    ...t,
    probes: t.probes + 1,
    non200: t.non200 + (p.status === 200 ? 0 : 1),
    badInRow,
    worstRun: Math.max(t.worstRun, badInRow),
    worstMs: Math.max(t.worstMs, Math.round(p.ms)),
    secondsDark: t.secondsDark + (p.status === 200 ? 0 : 1),
  };
}
/** The tally with one more page fetched under load; its latency counts toward the worst. */
export function tallyPage(t: Tally, p: Probe): Tally {
  return { ...t, pages: t.pages + 1, pagesBad: t.pagesBad + (isBad(p) ? 1 : 0), worstMs: Math.max(t.worstMs, Math.round(p.ms)) };
}
export const failed = (t: Tally): boolean => t.worstRun >= FAIL_AFTER;
/** Why a failed tally failed, with its numbers, in one line. */
export function failure(t: Tally): string {
  return `${t.worstRun} bad /healthz probes in a row (non-200 or over ${SLOW_MS / 1000} s) after ${t.probes} probes; ${t.non200} non-200, worst ${t.worstMs} ms`;
}
export interface VerifyPorts {
  /** One GET: the status (0 when nothing answered) and how long it took. */
  readonly get: (path: string, asAdmin: boolean) => Promise<Probe>;
  readonly now: () => number;
  readonly sleep: (ms: number) => Promise<void>;
  /** The pages fetched in turn, one at a time. */
  readonly pages: readonly string[];
  readonly durationMs?: number;
  /** Told after each probe, so the record keeps up while the verify runs. */
  readonly onProbe?: (t: Tally) => void;
}
export type Verdict = { readonly ok: boolean; readonly tally: Tally; readonly reason: string };
/** Probe `/healthz` each second under page load until the time is up or three bad probes in a row fail it. */
export async function verifyUnderLoad(ports: VerifyPorts, start: Tally = EMPTY): Promise<Verdict> {
  const end = ports.now() + (ports.durationMs ?? VERIFY_MS);
  let t = start;
  let stop = false;
  // The load: one page at a time, the next when the last has answered.
  const load = (async () => {
    for (let i = 0; !stop && ports.pages.length > 0; i++) {
      const page = await ports.get(ports.pages[i % ports.pages.length]!, true);
      t = tallyPage(t, page);
      if (ports.now() >= end) break;
    }
  })();
  while (ports.now() < end) {
    const began = ports.now();
    t = tally(t, await ports.get("/healthz", false));
    ports.onProbe?.(t);
    if (failed(t)) break;
    await ports.sleep(Math.max(0, PROBE_EVERY_MS - (ports.now() - began)));
  }
  stop = true;
  await load;
  return failed(t) ? { ok: false, tally: t, reason: failure(t) } : { ok: true, tally: t, reason: "" };
}
/** A GET through `fetch` that never throws: a refused connection or a timeout is status 0. */
export function timedGet(
  base: string,
  headers: (asAdmin: boolean) => Record<string, string>,
  fetcher: typeof fetch = fetch,
  now: () => number = () => performance.now(),
): (path: string, asAdmin: boolean) => Promise<Probe> {
  return async (path, asAdmin) => {
    const began = now();
    try {
      const res = await fetcher(new URL(path, base), {
        headers: headers(asAdmin),
        redirect: "manual",
        signal: AbortSignal.timeout(SLOW_MS + 5000),
      });
      await res.arrayBuffer();
      return { status: res.status, ms: now() - began };
    } catch {
      return { status: 0, ms: now() - began };
    }
  };
}
@@ -0,0 +1,130 @@
/**
 * The release keeper's ports, from a running forge (#7557): the forge's own
 * repository for main and the tags, the merge queue for "mid-landing", the
 * kube API for the Deployment, and the public URL for the probes.
 *
 * On when `FORGE_RELEASE_KEEPER=on`, `FORGE_REPO` names a repository here,
 * and the forge runs in a cluster. The Deployment and its container are
 * `FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER` (both default
 * `forge`); the pod's service account needs `get` and `patch` on it.
 */
import type { App } from "../app.ts";
import { originPush } from "../actions/branches.ts";
import { serviceAccount } from "../cluster/jobs.ts";
import { getRepo, type Repo } from "../core/repos.ts";
import { createSession, destroySession } from "../core/sessions.ts";
import { getUserById, type User } from "../core/users.ts";
import { gatesInFlight } from "../checks/queue.ts";
import { migrationFiles, NUMBERED } from "../core/schema.ts";
import { runGit } from "../git/env.ts";
import { repoDir } from "../git/repos.ts";
import { ownProbeHeaders } from "../kernel/probe.ts";
import { running } from "../kernel/version.ts";
import { compareVersions, readTags, recordFor } from "../release/shipped.ts";
import { kubeDeployer } from "./deploy.ts";
import type { KeeperPorts } from "./keeper.ts";
import { timedGet, verifyUnderLoad } from "./verify.ts";
const LABEL = /^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$/;
const OID = /^[0-9a-f]{40}$/;
const VERSION = /^\d+\.\d+\.\d+$/;
const ZERO = "0".repeat(40);
export type KeeperEnv = { on: true; deployment: string; container: string } | { on: false; why: string };
/** Whether the keeper runs, from the environment. */
export function keeperEnv(env: Record<string, string | undefined>): KeeperEnv {
  if (env.FORGE_RELEASE_KEEPER !== "on") return { on: false, why: "FORGE_RELEASE_KEEPER is not on" };
  if (!env.KUBERNETES_SERVICE_HOST) return { on: false, why: "not running in a cluster" };
  const deployment = env.FORGE_KEEPER_DEPLOYMENT || "forge";
  const container = env.FORGE_KEEPER_CONTAINER || "forge";
  if (!LABEL.test(deployment) || !LABEL.test(container)) return { on: false, why: "FORGE_KEEPER_DEPLOYMENT and FORGE_KEEPER_CONTAINER must be DNS labels" };
  return { on: true, deployment, container };
}
/** The first active site admin that is a person: the account the page load signs in as. */
function anAdmin(app: App): User | undefined {
  const row = app.db
    .query<{ id: string }, []>(`SELECT id FROM user WHERE is_admin = 1 AND is_agent = 0 ORDER BY created_at LIMIT 1`)
    .get();
  return row === null ? undefined : getUserById(app.db, row.id);
}
/** The pages a release is verified under, one at a time: the ones people open. */
export function loadPages(repo: Pick<Repo, "ownerName" | "name">): string[] {
  return ["/boards", `/${encodeURIComponent(repo.ownerName)}/${encodeURIComponent(repo.name)}`, "/explore", "/"];
}
export function keeperPorts(app: App, input: { forgeUrl: string; deployment: string; container: string }): KeeperPorts | string {
  const named = app.config.forgeRepo;
  if (named === undefined) return "FORGE_REPO is not set";
  const [owner = "", name = ""] = named.split("/");
  const repo = getRepo(app.db, owner, name);
  if (repo === undefined) return `FORGE_REPO names ${named}, which is no repository on this forge`;
  const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
  const deployer = kubeDeployer({ credentials: serviceAccount(), deployment: input.deployment, container: input.container });
  const probeHeaders = ownProbeHeaders(app.config.dataDir);
  const at = (version: string) => runGit(["rev-parse", "--verify", "--quiet", `refs/tags/v${version}^{commit}`], { cwd: dir });
  return {
    db: app.db,
    bus: app.bus,
    repo: { id: repo.id, ownerId: repo.ownerId },
    now: () => Date.now(),
    live: () => running().version,
    async main() {
      const head = await runGit(["rev-parse", "--verify", "--quiet", `refs/heads/${repo.defaultBranch}^{commit}`], { cwd: dir });
      return head.ok && OID.test(head.value) ? head.value : undefined;
    },
    async newest() {
      const tags = await readTags(dir);
      if (!tags.ok) return undefined;
      const top = [...tags.value].filter((t) => VERSION.test(t.version)).sort((a, b) => compareVersions(b.version, a.version))[0];
      if (top === undefined) return undefined;
      const sha = await at(top.version);
      return sha.ok && OID.test(sha.value) ? { version: top.version, sha: sha.value } : undefined;
    },
    gating: gatesInFlight,
    deployer,
    async tag(version, sha) {
      if (!VERSION.test(version) || !OID.test(sha)) return "not a version and a commit";
      // The record `scripts/cut-release.ts` writes. The migrations are this process's: the keeper cuts only
      // when main has moved past a live tag, and each forge cuts the code it was built from at the earliest.
      const record = JSON.stringify(recordFor(version, migrationFiles, NUMBERED));
      const made = await runGit(["tag", "--annotate", "--file=-", `v${version}`, sha], {
        cwd: dir,
        stdin: `${record}\n`,
        env: { GIT_COMMITTER_NAME: "release keeper", GIT_COMMITTER_EMAIL: "[email protected]" },
      });
      if (!made.ok) return made.error.stderr || "git refused the tag";
      const tagOid = await runGit(["rev-parse", `refs/tags/v${version}`], { cwd: dir });
      const owner = getUserById(app.db, repo.ownerId);
      if (!tagOid.ok || owner === undefined) return undefined;
      // A tag the origin did not take is taken back here too: the keeper drops the cut, and a
      // tag left behind would be the newest, never live, and stall the cadence.
      const untag = async (why: string) => (await runGit(["tag", "--delete", `v${version}`], { cwd: dir }), why);
      const push = await originPush(app, owner, repo);
      if (!push.ok) return untag(`the origin refused v${version}: ${push.error}`);
      if (push.value === undefined) return undefined;
      const sent = await push.value([{ ref: `refs/tags/v${version}`, oldOid: ZERO, newOid: tagOid.value }]);
      return sent.ok ? undefined : untag(`pushing v${version} to the origin failed: ${sent.error}`);
    },
    probe: () => timedGet(input.forgeUrl, () => probeHeaders)("/healthz", false),
    async verify(onProbe) {
      const admin = anAdmin(app);
      const session = admin === undefined ? undefined : createSession(app.db, admin.id);
      const cookie = session === undefined ? {} : { cookie: `forge_session=${session.token}` };
      try {
        return await verifyUnderLoad({
          get: timedGet(input.forgeUrl, (asAdmin) => (asAdmin ? cookie : probeHeaders)),
          now: () => Date.now(),
          sleep: (ms) => Bun.sleep(ms),
          pages: loadPages(repo),
          onProbe,
        });
      } finally {
        if (session !== undefined) destroySession(app.db, session.token);
      }
    },
  };
}
@@ -70,6 +70,8 @@
import { proveRunning } from "./release/prove.ts";
import { CHECK_EVERY_MS, setBehind, watchBehind } from "./release/behind.ts";
import { readLandings } from "./release/landings.ts";
import { ownTagsReader } from "./release/own_tags.ts";
import { rollProbe, tick as keeperTick } from "./keeper/keeper.ts";
import { keeperEnv, keeperPorts } from "./keeper/wire.ts";
import { snapshotPlace } from "./actions/health.ts";
import { capacityDefault } from "./core/capacity.ts";
import { bootCounts, recordBoot, recordCleanExit } from "./kernel/boots.ts";
@@ -627,6 +629,26 @@
if (app.config.forgeRepo) {
  void readLandings(app);
  setInterval(() => void readLandings(app), 60_000);
}
// The release keeper (#7557): cut on cadence, promote, verify under page load, roll back to the named
// known-good image and hold. Off unless FORGE_RELEASE_KEEPER=on in a cluster (keeper/wire.ts).
const keeperOn = keeperEnv(process.env);
if (keeperOn.on) {
  const ports = keeperPorts(app, { forgeUrl, deployment: keeperOn.deployment, container: keeperOn.container });
  if (typeof ports === "string") app.log.warn("the release keeper is off", { why: ports });
  else {
    let said = "";
    const step = () =>
      void keeperTick(ports)
        .then((line) => {
          if (line !== said) app.log.info("release keeper", { line });
          said = line;
        })
        .catch((error) => app.log.warn("release keeper step failed", { error: String(error) }));
    step();
    setInterval(step, 60_000);
    setInterval(() => void rollProbe(ports).catch(() => {}), 1000);
  }
} else app.log.info("the release keeper is off", { why: keeperOn.why });
// Only an agent is on a crew (the `repo_crew_agents_only` trigger refuses anyone else). A person
// put on one before that is reported here for the owner to take off — the
// forge never removes it on its own.
@@ -62,6 +62,7 @@
import { noteRead, telemetryReader } from "../actions/telemetry.ts";
import { queuesFor, systemFor } from "../system/telemetry.ts";
import { apiScreen } from "./screens/api.ts";
import { changelogScreen } from "./screens/changelog.ts";
import { keeperScreen } from "./screens/keeper.ts";
import { releasesScreen } from "./screens/releases.ts";
import { mayRead } from "../release/gate.ts";
import { fileHoldsPrompt } from "./screens/fileholds.ts";
@@ -333,6 +334,14 @@
export function forgeCommands(session: Session): (ctx: Ctx) => Command[] {
        hint: "the live release and the kept ones",
        run: () => ctx.push(releasesScreen(session, ctx)),
      });
    // The release keeper's card (#7557), gated as /releases is.
    if (mayRead(session.app, session.user))
      commands.push({
        id: "keeper",
        title: "Release keeper",
        hint: "what holds the next cut, and each release it cut with its probes",
        run: () => ctx.push(keeperScreen(session, ctx)),
      });
    // Stalled flow steps the viewer can unblock, as the home's cards show them (#1807).
    if (session.signedIn)
      commands.push({
@@ -0,0 +1,117 @@
/**
 * The release keeper in the terminal (#7557), the /releases card's read:
 * what holds the next cut, then each release the keeper cut with its stage,
 * the known-good version it rolls back to and what the probes counted.
 * Gated by `release/gate.ts` as the page is. `h` holds releases with a
 * reason and `h` with `lift N` lifts an owner's hold, through the same
 * functions the card and MCP `release_hold` call.
 */
import { blocking, holdAs, liftAs, standingHolds } from "../../keeper/hold.ts";
import { listKeeperReleases, type KeeperRelease, type Stage } from "../../keeper/record.ts";
import { mayRead, mayRollBack } from "../../release/gate.ts";
import type { Ctx, Screen } from "../app/app.ts";
import { binding } from "../app/keys.ts";
import type { Session } from "../session.ts";
import { style } from "../term/style.ts";
import { truncate, wrap } from "../term/width.ts";
import { fill, len, split } from "../ui/layout.ts";
import { emptyState, type Tone } from "../ui/widgets.ts";
import { ListState, ago, drawList, metaOf, promptScreen, type ListRow } from "./common.ts";
const STAGE: Record<Stage, { label: string; tone: Tone }> = {
  rolling: { label: "rolling", tone: "default" },
  verifying: { label: "verifying", tone: "default" },
  verified: { label: "verified live", tone: "success" },
  rolled_back: { label: "rolled back", tone: "danger" },
};
export interface KeeperRead {
  /** What holds the next cut, or the line that says nothing does. */
  readonly blocked: string;
  readonly releases: readonly KeeperRelease[];
}
/** The card's content for this person; undefined for anyone the page would 404. */
export function readKeeper(f: Session): KeeperRead | undefined {
  if (!mayRead(f.app, f.user)) return undefined;
  return { blocked: blocking(standingHolds(f.db)) ?? "Nothing holds the next cut.", releases: listKeeperReleases(f.db) };
}
export function keeperRows(releases: readonly KeeperRelease[]): ListRow[] {
  return releases.map((r) => ({
    title: r.version,
    tone: STAGE[r.stage].tone,
    meta: metaOf(
      STAGE[r.stage].label,
      `cut ${ago(r.cutAt)}`,
      `known good ${r.knownGood.version}`,
      `roll ${r.rollNon200}/${r.rollProbes} non-200`,
      `${r.probes} probes, ${r.pages} pages`,
      `worst ${r.worstMs} ms`,
      `${r.secondsDark} s dark`,
    ),
  }));
}
/** One line of the hold prompt: a reason holds, `lift N` lifts; the sentence to show. */
export function holdLine(f: Session, text: string): { ok: boolean; says: string } {
  const lifting = /^lift\s+#?(\d+)$/i.exec(text.trim());
  if (lifting !== null) {
    const done = liftAs(f.app, f.user, Number(lifting[1]));
    return done.ok ? { ok: true, says: `Hold ${done.value} lifted.` } : { ok: false, says: done.message };
  }
  const held = holdAs(f.app, f.user, text);
  return held.ok ? { ok: true, says: `Releases held: ${held.value.reason}` } : { ok: false, says: held.message };
}
export function keeperScreen(f: Session, ctx: Ctx): Screen {
  const state = new ListState();
  let read: KeeperRead | undefined;
  const refresh = (): void => {
    read = readKeeper(f);
    state.clamp(read?.releases.length ?? 0);
    ctx.invalidate();
  };
  const screen: Screen = {
    title: "Release keeper",
    keys: [binding(["j", "k"], "move"), ...(mayRollBack(f.app, f.user) ? [binding(["h"], "hold / lift")] : []), binding(["esc", "q"], "close")],
    load: async () => refresh(),
    render(area, s) {
      const c = ctx.theme.c;
      if (read === undefined) {
        emptyState(s, area, c, { title: "Not yours to see", body: "The release keeper is for the forge's administrators and its repository's readers." });
        return;
      }
      // What holds the cut ends with what to land ("release hold N"): wrapped, up to three lines, never cut short of it.
      const lines = wrap(read.blocked, area.w).slice(0, 3);
      const [head, body] = split(area, "v", [len(lines.length + 1), fill(1)]);
      lines.forEach((l, i) => s.text(head!.x, head!.y + i, truncate(l, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w));
      if (read.releases.length === 0) {
        emptyState(s, body!, c, { title: "No keeper releases yet", body: "The keeper cuts when main has moved past a live tag (FORGE_RELEASE_KEEPER=on)." });
        return;
      }
      drawList(s, body!, c, state, keeperRows(read.releases));
    },
    key(e) {
      if (e.key === "esc" || e.key === "q") ctx.pop();
      else if (e.key === "h" && mayRollBack(f.app, f.user))
        ctx.push(
          promptScreen(ctx, screen, {
            title: "Hold releases",
            label: "a reason holds the next cut · lift N lifts an owner hold",
            onSubmit: (text) => {
              const done = holdLine(f, text);
              if (!done.ok) return done.says;
              ctx.toast(done.says);
              refresh();
              return undefined;
            },
          }),
        );
      else state.key(e, read?.releases.length ?? 0);
      return true;
    },
  };
  return screen;
}
@@ -0,0 +1,50 @@
/**
 * The release keeper on /releases (#7557): what stops the next cut, and
 * each release it cut with its stage, the known-good version it rolls back
 * to, and what the probes counted. Nothing when it has done nothing.
 */
import type { Database } from "bun:sqlite";
import { blocking, MAX_REASON, standingHolds } from "../../keeper/hold.ts";
import { listKeeperReleases, type Stage } from "../../keeper/record.ts";
import { html, when, type Html } from "../html.ts";
import { badge, btn, card, time, type Tone } from "../ui.ts";
const STAGE: Record<Stage, { label: string; tone: Tone }> = {
  rolling: { label: "rolling", tone: "attn" },
  verifying: { label: "verifying", tone: "attn" },
  verified: { label: "verified live", tone: "success" },
  rolled_back: { label: "rolled back", tone: "danger" },
};
/** `mayHold` is `mayRollBack`: the card then offers the owner's hold and its lift. */
export function keeperPanel(db: Database, mayHold = false): Html | undefined {
  const releases = listKeeperReleases(db);
  const holds = standingHolds(db);
  const blocked = blocking(holds);
  if (releases.length === 0 && blocked === undefined && !mayHold) return undefined;
  const rows = releases.map((r) => {
    const s = STAGE[r.stage];
    return html`<tr>
      <td>${r.version} ${badge(s.label, { tone: s.tone })}</td>
      <td>${time(r.cutAt)}</td>
      <td>${r.knownGood.version}</td>
      <td>${r.rollNon200} of ${r.rollProbes}</td>
      <td>${r.probes} / ${r.pages}</td>
      <td>${r.worstMs} ms</td>
      <td>${r.secondsDark} s</td>
    </tr>${when(r.reason !== "", () => html`<tr><td colspan="7" class="small muted">${r.reason}</td></tr>`)}`;
  });
  return card(html`<h2>Release keeper</h2>
    <p class="small ${blocked === undefined ? "muted" : ""}" id="keeper-blocked">${blocked ?? "Nothing holds the next cut."}</p>
    ${when(rows.length > 0, () => html`<table class="table">
      <thead><tr><th>Version</th><th>Cut</th><th>Known good</th><th>Roll non-200</th><th>Probes / pages</th><th>Worst</th><th>Dark</th></tr></thead>
      <tbody>${rows}</tbody>
    </table>`)}
    ${when(mayHold, () => html`${holds.filter((h) => h.kind === "owner").map(
      (h) => html`<form method="post" action="/releases/hold/${h.id}/lift" class="small">Owner hold ${h.id}: ${h.reason} ${btn("Lift", { type: "submit" })}</form>`,
    )}
    <form method="post" action="/releases/hold" class="small">
      <label>Hold releases <input name="reason" maxlength="${MAX_REASON}" required placeholder="why"></label> ${btn("Hold", { type: "submit" })}
    </form>`)}`);
}
@@ -19,6 +19,8 @@
import { readForm } from "../form.ts";
import { html, when, type Html } from "../html.ts";
import { flashTo, page } from "../layout.ts";
import { badge, btn, card, empty, sha, time } from "../ui.ts";
import { holdAs, liftAs } from "../../keeper/hold.ts";
import { keeperPanel } from "./keeper-panel.ts";
/** The live line (`liveSince`) for the home, or nothing when no release is known to be live. */
export function liveLine(ctx: Ctx): Html | undefined {
@@ -81,7 +83,7 @@
export async function releasesPage(ctx: Ctx): Promise<Response> {
  const response = page({
    ctx,
    title: "Releases",
    body: html`<div class="page-head"><h1>Releases</h1></div>${body}`,
    body: html`<div class="page-head"><h1>Releases</h1></div>${keeperPanel(ctx.app.db, writer)}${body}`,
  });
  notePhase(ctx.request, "render", performance.now() - rendering);
  return response;
@@ -101,3 +103,23 @@
export async function rollbackSubmit(ctx: Ctx): Promise<Response> {
  if (!result.ok) return refuse(result.line);
  return flashTo(ctx, "/releases", { message: `Rolled back to ${target.slice(0, 7)}; it reads healthy.` });
}
/** POST /releases/hold: an owner's hold with a reason, through the function MCP calls. */
export async function releaseHoldSubmit(ctx: Ctx): Promise<Response> {
  const form = await readForm(ctx, "rw");
  if (form instanceof Response) return form;
  const held = holdAs(ctx.app, ctx.auth.user, String(form.get("reason") ?? ""));
  return held.ok
    ? flashTo(ctx, "/releases", { message: `Releases held: ${held.value.reason}` })
    : flashTo(ctx, "/releases", { tone: "danger", message: held.message });
}
/** POST /releases/hold/:id/lift: lift an owner's hold. */
export async function liftSubmit(ctx: Ctx): Promise<Response> {
  const form = await readForm(ctx, "rw");
  if (form instanceof Response) return form;
  const lifted = liftAs(ctx.app, ctx.auth.user, Number(ctx.params["id"]));
  return lifted.ok
    ? flashTo(ctx, "/releases", { message: `Hold ${lifted.value} lifted.` })
    : flashTo(ctx, "/releases", { tone: "danger", message: lifted.message });
}
@@ -36,10 +36,10 @@
test("a forge with no person to found the repo writes nothing", async () => {
  expect(await offerStartingBoards(bare)).toEqual([]);
});
test("a fresh boot writes the ten starting boards, and every panel runs", async () => {
  expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
test("a fresh boot writes the eleven starting boards, and every panel runs", async () => {
  expect((await offerStartingBoards(app)).sort()).toEqual(["apps", "crew", "forges", "maintenance", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
  const boards = await listBoards(app.reposRoot);
  expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
  expect(boards.map((b) => b.slug)).toEqual(["apps", "crew", "forges", "maintenance", "reach", "releases", "restarts", "security", "serving", "throughput", "usage"]);
  let panels = 0;
  for (const { board } of boards) {
    for (const p of board.panels) {
@@ -0,0 +1,101 @@
/** A forge, its own repository and the release keeper's ports with a fake cluster and clock (#7557). */
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createApp } from "../../src/app.ts";
import { createRepo } from "../../src/core/repos.ts";
import { createUser } from "../../src/core/users.ts";
import { id } from "../../src/kernel/id.ts";
import type { Deployer } from "../../src/keeper/deploy.ts";
import type { KeeperPorts } from "../../src/keeper/keeper.ts";
import { verifyUnderLoad, type Probe } from "../../src/keeper/verify.ts";
export const A = "a".repeat(40);
export const B = "b".repeat(40);
export const C = "c".repeat(40);
/** A Deployment that remembers every image set on it. Its revision history holds a broken image "undo" would bring back. */
export function fakeDeployer(image: string) {
  const state = { image, set: [] as string[], history: ["reg.test/o/forge:0.7.277"] };
  const deployer: Deployer = {
    image: async () => state.image,
    async setImage(next) {
      state.history.push(state.image);
      state.image = next;
      state.set.push(next);
    },
  };
  return { deployer, state };
}
export async function keeperForge(opts: { healthz?: (n: number) => Probe } = {}) {
  const app = createApp({ dataDir: mkdtempSync(join(tmpdir(), "forge-keeper-")), port: 0, forgeRepo: "alice/forge" });
  const u = await createUser(app.db, { username: "alice", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true, adminIfFirst: true });
  if (!u.ok) throw new Error("user");
  const repo = await createRepo(app.db, app.reposRoot, { ownerId: u.value.id, name: "forge", visibility: "public", description: "f" });
  if (!repo.ok) throw new Error("repo");
  const world = {
    now: Date.parse("2026-10-27T12:00:00.000Z"),
    live: "0.7.283",
    main: B,
    newest: { version: "0.7.283", sha: A } as { version: string; sha: string } | undefined,
    gating: 0,
    tags: [] as { version: string; sha: string }[],
    /** When set, the tag is refused with this line. */
    tagRefused: undefined as string | undefined,
    /** Why the next tag is refused; undefined lets it through. */
    refuseTag: undefined as string | undefined,
    healthz: opts.healthz ?? (() => ({ status: 200, ms: 12 })),
    pages: [] as string[],
  };
  const { deployer, state } = fakeDeployer("reg.test/o/forge:0.7.283");
  let probes = 0;
  const ports: KeeperPorts = {
    db: app.db,
    bus: app.bus,
    repo: { id: repo.value.id, ownerId: u.value.id },
    now: () => world.now,
    live: () => world.live,
    main: async () => world.main,
    newest: async () => world.newest,
    gating: () => world.gating,
    deployer,
    // As the forge's repository does: a tag made is the newest tag read back.
    async tag(version, sha) {
      if (world.refuseTag !== undefined) return world.refuseTag;
      world.tags.push({ version, sha });
      world.newest = { version, sha };
      return undefined;
    },
    probe: async () => world.healthz(probes++),
    verify: (onProbe) =>
      verifyUnderLoad({
        get: async (path) => {
          if (path === "/healthz") {
            const p = world.healthz(probes++);
            world.now += p.ms;
            return p;
          }
          world.pages.push(path);
          world.now += 40;
          return { status: 200, ms: 40 };
        },
        now: () => world.now,
        sleep: async (ms) => {
          world.now += ms;
        },
        pages: ["/boards", "/alice/forge", "/explore", "/"],
        onProbe,
      }),
  };
  /** A pull of the forge's repository, merged now. */
  const land = (n: number, title: string, body = "") => {
    const issueId = id();
    const at = new Date(world.now).toISOString();
    app.db.query(`INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at) VALUES (?1, ?2, ?3, 'pull', ?4, ?5, ?6, 'closed', ?7)`).run(issueId, repo.value.id, n, title, body, u.value.id, at);
    app.db.query(`INSERT INTO pull (issue_id, base_branch, head_branch, base_oid, head_oid, head_repo_id, merged_at) VALUES (?1, 'main', ?2, '', '', ?3, ?4)`).run(issueId, `b/${n}`, repo.value.id, at);
  };
  const events = (kind: string) =>
    app.db.query<{ payload: string }, [string]>(`SELECT payload FROM event WHERE kind = ?1 ORDER BY id`).all(kind).map((e) => JSON.parse(e.payload) as Record<string, unknown>);
  return { app, user: u.value, repo: repo.value, world, ports, state, land, events };
}
@@ -31,6 +31,7 @@
const ADMIN_ARGS: Record<string, Record<string, unknown>> = {
  subscription_wake: { id: "u1" },
  subscription_headroom: { id: "u1", percent: 30 },
  prompts: { repo: "alice/game", action: "remove", id: "p1" },
  release_hold: { reason: "freeze for the demo" },
  telemetry_grant: { agent: "robo", on: true },
  hooks: { repo: "alice/game", action: "add", url: "https://example.com/h", secret: "0123456789abcdef0123" },
  app_domains: { repo: "alice/game", action: "add", host: "game.example.com" },
@@ -203,6 +203,7 @@
describe("the forge over MCP", () => {
      "edit",
      "close",
      "read_file",
      "release_keeper",
      "write_file",
      "asks",
      "asks_act",
@@ -77,6 +77,7 @@
test("a site-admin person is offered the whole roster", () => {
    ["api", "API", "the forge's endpoints; run a GET"],
    ["changelog", "Changelog", "the last 30 releases and their pulls"],
    ["releases", "Releases", "the live release and the kept ones"],
    ["keeper", "Release keeper", "what holds the next cut, and each release it cut with its probes"],
    ["stalled", "Stalled steps", "pulls whose check is red or was never asked for; enter re-runs"],
    ["file-holds", "Held files", "hold new work on a file, release it, or list the holds"],
    ["access", "Access", "what outside agents tried"],
@@ -135,6 +136,7 @@
test("a site-admin agent is offered the roster without the person-only entries",
    ["api", "API", "the forge's endpoints; run a GET"],
    ["changelog", "Changelog", "the last 30 releases and their pulls"],
    ["releases", "Releases", "the live release and the kept ones"],
    ["keeper", "Release keeper", "what holds the next cut, and each release it cut with its probes"],
    ["stalled", "Stalled steps", "pulls whose check is red or was never asked for; enter re-runs"],
    ["file-holds", "Held files", "hold new work on a file, release it, or list the holds"],
    ["secrets", "Secrets", "sealed secrets by name; set or remove one"],
@@ -0,0 +1,98 @@
// When the release keeper cuts (#7557): each rule refuses on its own, and a cut is a tag, never a commit to main.
import { describe, expect, test } from "bun:test";
import { decideCut, MIN_INTERVAL_MS, nextVersion, type CadenceInput } from "../src/keeper/cadence.ts";
import { tick } from "../src/keeper/keeper.ts";
import { listKeeperReleases, recordCut } from "../src/keeper/record.ts";
import { A, B, keeperForge } from "./helpers/keeper.ts";
const NOW = Date.parse("2026-10-27T12:00:00.000Z");
const ready: CadenceInput = {
  main: B,
  newest: { version: "0.7.283", sha: A },
  live: "0.7.283",
  inFlight: false,
  gating: 0,
  lastCutAt: NOW - MIN_INTERVAL_MS,
  hold: undefined,
  now: NOW,
};
describe("cadence", () => {
  test("the next version is the patch plus one", () => {
    expect(nextVersion("0.7.283")).toBe("0.7.284");
    expect(nextVersion("1.0.9")).toBe("1.0.10");
    expect(nextVersion("0.7")).toBeUndefined();
  });
  test("main past a live tag, nothing in flight, the queue still, the interval passed, no hold: cut", () => {
    expect(decideCut(ready)).toEqual({ cut: true, version: "0.7.284" });
  });
  test("each rule refuses alone, and says which", () => {
    const why = (o: Partial<CadenceInput>) => {
      const d = decideCut({ ...ready, ...o });
      return d.cut ? "" : d.why;
    };
    expect(why({ main: A })).toBe("main is still 0.7.283");
    expect(why({ inFlight: true })).toContain("still rolling");
    expect(why({ live: "0.7.282" })).toBe("0.7.283 is not live yet (live: 0.7.282)");
    expect(why({ gating: 2 })).toBe("the merge queue is landing (2 gates running)");
    // One minute short of 30 is refused; exactly 30 passes (above).
    expect(why({ lastCutAt: NOW - MIN_INTERVAL_MS + 60_000 })).toBe("the last cut was 29 min ago; the interval is 30 min");
    expect(why({ hold: "owner: freeze for the demo" })).toBe("held: owner: freeze for the demo");
    expect(why({ newest: undefined })).toContain("no release tag");
  });
  test("a tag the keeper rolled back is settled: the fix after it may be cut", () => {
    expect(decideCut({ ...ready, newest: { version: "0.7.284", sha: A }, live: "0.7.283", settled: true })).toEqual({ cut: true, version: "0.7.285" });
  });
  test("the keeper's cut is a tag on main's head, and records the known-good version and image first", async () => {
    const f = await keeperForge();
    const line = await tick(f.ports);
    expect(line).toContain("cut 0.7.284");
    expect(f.world.tags).toEqual([{ version: "0.7.284", sha: B }]);
    const [r] = listKeeperReleases(f.app.db);
    expect(r).toMatchObject({ version: "0.7.284", sha: B, stage: "rolling", knownGood: { version: "0.7.283", image: "reg.test/o/forge:0.7.283" } });
    expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284"]);
    expect(f.events("release.cut")).toHaveLength(1);
    expect(f.events("release.released")).toEqual([{ version: "0.7.284", image: "reg.test/o/forge:0.7.284" }]);
    // While it rolls, a second tick cuts nothing more.
    f.world.main = "d".repeat(40);
    f.world.now += 2 * MIN_INTERVAL_MS;
    expect(await tick(f.ports)).toBe("0.7.284 is rolling");
    expect(f.world.tags).toHaveLength(1);
  });
  test("the queue mid-landing holds the cut", async () => {
    const f = await keeperForge();
    f.world.gating = 1;
    expect(await tick(f.ports)).toContain("merge queue is landing");
    expect(f.world.tags).toHaveLength(0);
    expect(f.state.set).toHaveLength(0);
  });
  test("the cut is recorded before the tag: a crash between them is tagged on the next tick, not stranded", async () => {
    const f = await keeperForge();
    // The process died after the row and before the tag.
    recordCut(f.app.db, { version: "0.7.284", sha: B, knownGood: { version: "0.7.283", image: "reg.test/o/forge:0.7.283" }, at: f.world.now });
    expect(f.world.tags).toHaveLength(0);
    expect(await tick(f.ports)).toBe("0.7.284 is rolling");
    expect(f.world.tags).toEqual([{ version: "0.7.284", sha: B }]);
    expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284"]);
    // Tagged once: the next tick does not tag it again.
    await tick(f.ports);
    expect(f.world.tags).toHaveLength(1);
  });
  test("a refused tag drops the cut, so the cadence is not left waiting on a release that never was", async () => {
    const f = await keeperForge();
    f.world.refuseTag = "the origin refused v0.7.284";
    expect(await tick(f.ports)).toBe("not cut: the origin refused v0.7.284");
    expect(listKeeperReleases(f.app.db)).toHaveLength(0);
    expect(f.state.set).toHaveLength(0);
    expect(f.events("release.cut")).toHaveLength(0);
    f.world.refuseTag = undefined;
    expect(await tick(f.ports)).toContain("cut 0.7.284");
  });
});
@@ -0,0 +1,107 @@
// The keeper's record is the admins' (#7557): at each door that reads it — /releases, the terminal
// palette and screen, MCP `release_keeper` — a guest gets none of it and an admin gets all of it.
import { describe, expect, test } from "bun:test";
import { createSession } from "../src/core/sessions.ts";
import { createToken } from "../src/core/tokens.ts";
import { createUser, type User } from "../src/core/users.ts";
import { serve } from "../src/http/server.ts";
import { tick } from "../src/keeper/keeper.ts";
import type { Ctx } from "../src/tui/app/app.ts";
import { forgeCommands, Session } from "../src/tui/forge.ts";
import { readKeeper } from "../src/tui/screens/keeper.ts";
import { quietLogs } from "./helpers/globals.ts";
import { keeperForge } from "./helpers/keeper.ts";
quietLogs();
const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
/** 0.7.284 failed its verify and was rolled back to 0.7.283; a failed hold stands. A guest account `bob` beside the admin `alice`. */
async function seeded() {
  const f = await keeperForge();
  await tick(f.ports);
  f.world.live = "0.7.284";
  f.world.healthz = (n) => (n >= 5 ? { status: 503, ms: 2 } : { status: 200, ms: 2 });
  await tick(f.ports);
  const bob = await createUser(f.app.db, { username: "bob", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true });
  if (!bob.ok) throw new Error("user");
  return { ...f, guest: bob.value };
}
describe("a guest sees no keeper data; an admin sees all of it", () => {
  test("/releases", async () => {
    const f = await seeded();
    const servers = await serve(f.app);
    try {
      const get = (user?: User) =>
        fetch(`${servers.url}/releases`, {
          headers: user === undefined ? {} : { cookie: `forge_session=${createSession(f.app.db, user.id).token}` },
          redirect: "manual",
        });
      const signedOut = await get();
      expect(signedOut.status).toBe(303);
      expect(signedOut.headers.get("location")).toContain("/signin");
      expect(await signedOut.text()).not.toContain("0.7.284");
      const guest = await get(f.guest);
      expect([403, 404]).toContain(guest.status);
      const guestBody = await guest.text();
      expect(guestBody).not.toContain("Release keeper");
      expect(guestBody).not.toContain("0.7.284");
      const admin = await get(f.user);
      expect(admin.status).toBe(200);
      const body = await admin.text();
      expect(body).toContain("Release keeper");
      expect(body).toContain("0.7.284");
      expect(body).toContain("0.7.283");
      expect(body).toContain("rolled back");
      expect(body).toContain("0.7.284 failed its verify");
    } finally {
      servers.stop();
    }
  });
  test("the terminal palette and screen", async () => {
    const f = await seeded();
    const offered = (user?: User) => forgeCommands(new Session(f.app, identity(user)))({} as Ctx).some((c) => c.id === "keeper");
    for (const who of [undefined, f.guest]) {
      expect(offered(who)).toBe(false);
      expect(readKeeper(new Session(f.app, identity(who)))).toBeUndefined();
    }
    expect(offered(f.user)).toBe(true);
    const read = readKeeper(new Session(f.app, identity(f.user)))!;
    expect(read.blocked).toContain("0.7.284 failed its verify");
    expect(read.releases.map((r) => [r.version, r.stage, r.knownGood.version])).toEqual([["0.7.284", "rolled_back", "0.7.283"]]);
  });
  test("MCP release_keeper", async () => {
    const f = await seeded();
    const servers = await serve(f.app);
    try {
      const call = async (user: User) => {
        const res = await fetch(`${servers.url}/mcp`, {
          method: "POST",
          headers: {
            "content-type": "application/json",
            accept: "application/json, text/event-stream",
            authorization: `Bearer ${createToken(f.app.db, user.id, "ro", "ro").token}`,
          },
          body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "release_keeper", arguments: {} } }),
        });
        const body = (await res.json()) as { result: { isError?: boolean; content: { text: string }[] } };
        return { isError: body.result.isError === true, text: body.result.content[0]!.text };
      };
      const guest = await call(f.guest);
      expect(guest.isError).toBe(true);
      expect(guest.text).not.toContain("0.7.284");
      const admin = await call(f.user);
      expect(admin.isError).toBe(false);
      const value = JSON.parse(admin.text);
      expect(value.blocked).toContain("0.7.284 failed its verify");
      expect(value.releases.map((r: { version: string; knownGood: { version: string } }) => [r.version, r.knownGood.version])).toEqual([["0.7.284", "0.7.283"]]);
      expect(value.holds).toHaveLength(1);
      expect(value.holds[0].kind).toBe("failed");
    } finally {
      servers.stop();
    }
  });
});
@@ -0,0 +1,115 @@
// An owner's hold from each door — the /releases card, the terminal screen and MCP — runs the one function
// in src/keeper/hold.ts; a refusal (an agent, a non-admin, a cross-origin post) writes no hold and no event (#7557).
import { describe, expect, test } from "bun:test";
import { createSession } from "../src/core/sessions.ts";
import { createUser, type User } from "../src/core/users.ts";
import { holdAs, liftAs, listHolds, standingHolds } from "../src/keeper/hold.ts";
import { serve } from "../src/http/server.ts";
import { createForgeApp, Session } from "../src/tui/forge.ts";
import { holdLine, keeperScreen } from "../src/tui/screens/keeper.ts";
import { keeperPanel } from "../src/web/pages/keeper-panel.ts";
import { headlessTerminal } from "../src/tui/term/terminal.ts";
import { quietLogs } from "./helpers/globals.ts";
import { keeperForge } from "./helpers/keeper.ts";
import { settleApp } from "./helpers/tui.ts";
quietLogs();
const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
const heldEvents = (f: Awaited<ReturnType<typeof keeperForge>>) => f.events("release.held").length + f.events("release.hold_lifted").length;
describe("the shared hold functions", () => {
  test("an agent and a non-admin are refused, and a refusal writes no hold and no event", async () => {
    const f = await keeperForge();
    const bob = await createUser(f.app.db, { username: "bob", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true });
    if (!bob.ok) throw new Error("user");
    const owner = holdAs(f.app, f.user, "freeze");
    expect(owner.ok).toBe(true);
    const before = heldEvents(f);
    for (const who of [{ ...f.user, kind: "agent" as const }, bob.value, undefined]) {
      expect(holdAs(f.app, who, "stop").ok).toBe(false);
      expect(liftAs(f.app, who, 1).ok).toBe(false);
    }
    expect(listHolds(f.app.db)).toHaveLength(1);
    expect(standingHolds(f.app.db)).toHaveLength(1);
    expect(heldEvents(f)).toBe(before);
  });
  test("control characters are stripped and an empty reason is refused", async () => {
    const f = await keeperForge();
    const h = holdAs(f.app, f.user, "a\u0007b\nc");
    expect(h.ok && h.value.reason).toBe("a b c");
    expect(holdAs(f.app, f.user, "  ").ok).toBe(false);
  });
});
describe("the /releases card", () => {
  test("offers the form to whoever may roll back, and posts through the shared function", async () => {
    const f = await keeperForge();
    const servers = await serve(f.app);
    try {
      expect(String(keeperPanel(f.app.db, false))).not.toContain("/releases/hold");
      expect(String(keeperPanel(f.app.db, true))).toContain('action="/releases/hold"');
      const cookie = `forge_session=${createSession(f.app.db, f.user.id).token}`;
      const post = (path: string, fields: Record<string, string>, origin = servers.url) =>
        fetch(`${servers.url}${path}`, { method: "POST", headers: { cookie, origin }, body: new URLSearchParams(fields), redirect: "manual" });
      const cross = await post("/releases/hold", { reason: "x" }, "https://evil.test");
      expect(cross.status).toBe(403);
      expect(standingHolds(f.app.db)).toHaveLength(0);
      const res = await post("/releases/hold", { reason: "freeze for the demo" });
      expect(res.status).toBe(303);
      expect(standingHolds(f.app.db).map((h) => [h.kind, h.reason])).toEqual([["owner", "freeze for the demo"]]);
      expect(String(keeperPanel(f.app.db, true))).toContain("/releases/hold/1/lift");
      expect((await post("/releases/hold/1/lift", {})).status).toBe(303);
      expect(standingHolds(f.app.db)).toHaveLength(0);
    } finally {
      servers.stop();
    }
  });
  test("a guest and a member who may not administer are refused and nothing is written", async () => {
    const f = await keeperForge();
    const servers = await serve(f.app);
    try {
      const bob = await createUser(f.app.db, { username: "bob", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true });
      if (!bob.ok) throw new Error("user");
      const cookie = `forge_session=${createSession(f.app.db, bob.value.id).token}`;
      for (const headers of [{ origin: servers.url }, { cookie, origin: servers.url }]) {
        const res = await fetch(`${servers.url}/releases/hold`, { method: "POST", headers, body: new URLSearchParams({ reason: "x" }), redirect: "manual" });
        expect(res.status).not.toBe(303);
      }
      expect(listHolds(f.app.db)).toHaveLength(0);
      expect(heldEvents(f)).toBe(0);
    } finally {
      servers.stop();
    }
  });
});
describe("the terminal screen", () => {
  test("h holds with a reason and `lift N` lifts it", async () => {
    const f = await keeperForge();
    const session = new Session(f.app, identity(f.user));
    expect(holdLine(session, "freeze").ok).toBe(true);
    expect(standingHolds(f.app.db)).toHaveLength(1);
    expect(holdLine(session, "lift 1").ok).toBe(true);
    expect(standingHolds(f.app.db)).toHaveLength(0);
    expect(holdLine(new Session(f.app, identity(undefined)), "freeze").ok).toBe(false);
    expect(listHolds(f.app.db)).toHaveLength(1);
  });
  test("the key is on the screen and opens the prompt", async () => {
    const f = await keeperForge();
    const terminal = headlessTerminal(120, 20);
    const { app: tui, session } = createForgeApp(f.app, identity(f.user), { terminal, exitHooks: false });
    tui.start();
    const screen = keeperScreen(session, tui.ctx);
    tui.ctx.push(screen);
    await settleApp(tui);
    screen.key!({ key: "h" } as never);
    await settleApp(tui);
    const text = tui.screen.toText();
    tui.stop?.();
    expect(text).toContain("Hold releases");
  });
});
@@ -0,0 +1,105 @@
// A failed verify holds further cuts until a pull naming the cause lands and a verified release lifts it;
// an owner's hold is set and lifted by an admin, never an agent (#7557).
import { describe, expect, test } from "bun:test";
import type { Ctx } from "../src/http/context.ts";
import { tool as holdTool } from "../src/http/mcp/tools/release_hold.ts";
import { tool as keeperTool } from "../src/http/mcp/tools/release_keeper.ts";
import { blocking, names, setHold, standingHolds, type Hold } from "../src/keeper/hold.ts";
import { tick } from "../src/keeper/keeper.ts";
import { listKeeperReleases } from "../src/keeper/record.ts";
import { keeperPanel } from "../src/web/pages/keeper-panel.ts";
import { keeperForge } from "./helpers/keeper.ts";
/** A forge whose 0.7.284 failed its verify and was rolled back to 0.7.283. */
async function heldForge() {
  const f = await keeperForge();
  await tick(f.ports);
  f.world.live = "0.7.284";
  f.world.healthz = (n) => (n >= 5 ? { status: 503, ms: 2 } : { status: 200, ms: 2 });
  await tick(f.ports);
  f.world.live = "0.7.283";
  f.world.healthz = () => ({ status: 200, ms: 2 });
  f.world.newest = { version: "0.7.284", sha: f.world.main };
  f.world.main = "e".repeat(40);
  f.world.now += 3_600_000;
  return f;
}
describe("a failed verify's hold", () => {
  test("names the version and the probe numbers, and blocks the next cut", async () => {
    const f = await heldForge();
    const [h] = standingHolds(f.app.db);
    expect(h).toMatchObject({ kind: "failed", version: "0.7.284", probes: 8, badInRow: 3, causePull: null });
    expect(await tick(f.ports)).toBe(`held: 0.7.284 failed its verify (${h!.reason}); waiting for a pull saying "release hold ${h!.id}" to land`);
  });
  test("a pull that does not name it changes nothing; one that does lets the fix through, and its verify lifts the hold", async () => {
    const f = await heldForge();
    const [h] = standingHolds(f.app.db);
    f.land(10, "Tidy the explore page");
    expect(await tick(f.ports)).toContain("held:");
    // A changelog or a revert note mentions the version; that is no cause.
    f.land(12, "Changelog for 0.7.284", "Reverts the 0.7.284 notes.");
    expect(await tick(f.ports)).toContain("held:");
    expect(standingHolds(f.app.db)[0]!.causePull).toBeNull();
    f.land(11, "Fix the boot stall that sank 0.7.284", `Closes release hold ${h!.id}.`);
    const line = await tick(f.ports);
    expect(line).toContain("cut 0.7.285");
    expect(standingHolds(f.app.db)[0]!.causePull).toBe(11);
    // Not lifted yet: only a verified release lifts it.
    expect(standingHolds(f.app.db)).toHaveLength(1);
    f.world.live = "0.7.285";
    expect(await tick(f.ports)).toContain("0.7.285 verified live");
    expect(standingHolds(f.app.db)).toHaveLength(0);
    expect(listKeeperReleases(f.app.db).map((r) => [r.version, r.stage])).toEqual([["0.7.285", "verified"], ["0.7.284", "rolled_back"]]);
    expect(f.events("release.verified_live")[0]).toMatchObject({ version: "0.7.285", holdsLifted: 1 });
  });
  test("only `release hold N` names the cause: not the bare version, not another hold's number", () => {
    const h = { id: 4, version: "0.7.284" } as Hold;
    expect(names(h, "Fixes release hold #4")).toBe(true);
    expect(names(h, "release hold 4")).toBe(true);
    expect(names(h, "release hold 41")).toBe(false);
    expect(names(h, "after 0.7.2840")).toBe(false);
    expect(names(h, "after 10.7.284")).toBe(false);
    expect(names(h, "0.7.284 hung at boot")).toBe(false);
    expect(names(h, "prerelease hold 4")).toBe(false);
  });
  test("the /releases panel says what holds the next cut, with the numbers", async () => {
    const f = await heldForge();
    const out = String(keeperPanel(f.app.db));
    expect(out).toContain("0.7.284 failed its verify");
    expect(out).toContain("rolled back");
    expect(out).toContain("0.7.283");
  });
});
describe("an owner's hold", () => {
  test("an admin sets it with a reason through MCP; the keeper cuts nothing; only an admin lifts it", async () => {
    const f = await keeperForge();
    const ctx = { app: f.app } as unknown as Ctx;
    const set = await holdTool.run(ctx, f.user, { reason: "freeze for the launch demo" });
    expect(set.ok).toBe(true);
    expect(await tick(f.ports)).toBe("held: the owner holds releases: freeze for the launch demo");
    expect(f.world.tags).toHaveLength(0);
    const read = await keeperTool.run(ctx, f.user, {});
    expect(read.ok && (read.value as { blocked: string }).blocked).toBe("the owner holds releases: freeze for the launch demo");
    const agent = { ...f.user, kind: "agent" as const };
    expect((await holdTool.run(ctx, agent, { lift: 1 })).ok).toBe(false);
    expect((await holdTool.run(ctx, f.user, { lift: 1 })).ok).toBe(true);
    expect(blocking(standingHolds(f.app.db))).toBeUndefined();
    expect(await tick(f.ports)).toContain("cut 0.7.284");
  });
  test("an agent cannot set one, and a failed verify's hold is not lifted by hand", async () => {
    const f = await keeperForge();
    const ctx = { app: f.app } as unknown as Ctx;
    expect((await holdTool.run(ctx, { ...f.user, kind: "agent" as const }, { reason: "stop" })).ok).toBe(false);
    const h = setHold(f.app.db, { kind: "failed", version: "0.7.284", reason: "3 bad", at: f.world.now });
    const tried = await holdTool.run(ctx, f.user, { lift: h.id });
    expect(tried.ok).toBe(false);
    expect(standingHolds(f.app.db)).toHaveLength(1);
    expect((await holdTool.run(ctx, f.user, { reason: "" })).ok).toBe(false);
  });
});
@@ -0,0 +1,97 @@
// The keeper rolls back by setting the known-good image recorded before the cut, by name; there is no undo path (#7557).
import { describe, expect, test } from "bun:test";
import { readdirSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { imageAt, kubeDeployer, tagOf } from "../src/keeper/deploy.ts";
import { tick } from "../src/keeper/keeper.ts";
import { listKeeperReleases } from "../src/keeper/record.ts";
import { rollBackToNamed } from "../src/keeper/rollback.ts";
import { standingHolds } from "../src/keeper/hold.ts";
import { fakeDeployer, keeperForge } from "./helpers/keeper.ts";
const creds = { server: "https://kube.test", token: "t", namespace: "forge" };
describe("the named rollback", () => {
  test("a seeded failing verify sets the recorded known-good image, holds with the numbers, and cuts nothing further", async () => {
    const f = await keeperForge();
    await tick(f.ports);
    expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284"]);
    // The new forge answers, then stops answering /healthz.
    f.world.live = "0.7.284";
    f.world.healthz = (n) => (n >= 20 ? { status: 503, ms: 4 } : { status: 200, ms: 4 });
    const line = await tick(f.ports);
    expect(line).toContain("0.7.284 failed its verify");
    // The image set is the one recorded at the cut, by name — not the controller's "previous" (the broken 0.7.277 here).
    expect(f.state.set).toEqual(["reg.test/o/forge:0.7.284", "reg.test/o/forge:0.7.283"]);
    expect(f.state.image).toBe("reg.test/o/forge:0.7.283");
    const [r] = listKeeperReleases(f.app.db);
    expect(r).toMatchObject({ stage: "rolled_back", probes: 23, badInRow: 3, secondsDark: 3 });
    expect(r!.reason).toContain("rolled back to 0.7.283 (reg.test/o/forge:0.7.283)");
    const [hold] = standingHolds(f.app.db);
    expect(hold).toMatchObject({ kind: "failed", version: "0.7.284", probes: 23, badInRow: 3, worstMs: 40 });
    expect(hold!.reason).toContain("3 bad /healthz probes in a row");
    expect(f.events("release.rolled_back")).toEqual([{ version: "0.7.284", to: "0.7.283", image: "reg.test/o/forge:0.7.283", ok: true }]);
    expect(f.events("release.held")).toHaveLength(1);
    // Back on the known-good version, main moves on, time passes: nothing more is cut.
    f.world.live = "0.7.283";
    f.world.newest = { version: "0.7.284", sha: f.world.main };
    f.world.main = "e".repeat(40);
    f.world.now += 24 * 3_600_000;
    expect(await tick(f.ports)).toContain("held: 0.7.284 failed its verify");
    expect(f.world.tags).toHaveLength(1);
    expect(f.state.set).toHaveLength(2);
  });
  test("a release not live three hours after its cut is set back by name, and held", async () => {
    const f = await keeperForge();
    await tick(f.ports);
    f.world.now += 3 * 3_600_000 + 1;
    expect(await tick(f.ports)).toContain("was not live 3 h after its cut");
    expect(f.state.image).toBe("reg.test/o/forge:0.7.283");
    expect(standingHolds(f.app.db)).toHaveLength(1);
  });
  test("a recorded image that is no image reference is refused, and nothing is set", async () => {
    const { deployer, state } = fakeDeployer("reg.test/o/forge:0.7.284");
    const r = await rollBackToNamed(deployer, { version: "0.7.283", image: "--help" });
    expect(r.ok).toBe(false);
    expect(state.set).toEqual([]);
  });
  test("there is no undo path: the deployer reads or sets a named image — and sends one PATCH naming it", async () => {
    const sent: { url: string; method: string; body: string }[] = [];
    const deployer = kubeDeployer({
      credentials: creds,
      deployment: "forge",
      container: "forge",
      fetch: async (url, init) => {
        sent.push({ url, method: init.method ?? "GET", body: String(init.body ?? "") });
        return Response.json({ spec: { replicas: 1, template: { spec: { containers: [{ name: "forge", image: "reg.test/o/forge:0.7.284" }] } } } });
      },
    });
    expect(Object.keys(deployer).sort()).toEqual(["image", "setImage"]);
    const r = await rollBackToNamed(deployer, { version: "0.7.283", image: "reg.test/o/forge:0.7.283" });
    expect(r.ok).toBe(true);
    expect(sent).toEqual([
      {
        url: "https://kube.test/apis/apps/v1/namespaces/forge/deployments/forge",
        method: "PATCH",
        body: JSON.stringify({ spec: { template: { spec: { containers: [{ name: "forge", image: "reg.test/o/forge:0.7.283" }] } } } }),
      },
    ]);
    // No code under src/keeper asks for a previous revision: no rollout undo, no rollback subresource, no revision number.
    const dir = join(import.meta.dir, "../src/keeper");
    for (const file of readdirSync(dir)) {
      const code = readFileSync(join(dir, file), "utf8").replace(/\/\*[\s\S]*?\*\//g, "").replace(/\/\/.*$/gm, "");
      expect(code.match(/undo|\/rollback(?!\.ts)|revision|previous/i)?.[0], file).toBeUndefined();
    }
  });
  test("an image keeps its registry port when its tag changes", () => {
    expect(imageAt("reg.test:5000/o/forge:0.7.283", "0.7.284")).toBe("reg.test:5000/o/forge:0.7.284");
    expect(imageAt("reg.test:5000/o/forge", "0.7.284")).toBe("reg.test:5000/o/forge:0.7.284");
    expect(imageAt(`reg.test/o/forge:0.7.283@sha256:${"f".repeat(64)}`, "0.7.284")).toBe("reg.test/o/forge:0.7.284");
    expect(tagOf("reg.test:5000/o/forge:0.7.283")).toBe("0.7.283");
    expect(tagOf("reg.test:5000/o/forge")).toBe("");
  });
});
@@ -0,0 +1,140 @@
// Proving a release live (#7557): ten minutes of /healthz once a second under page load; three bad in a row fail it.
import { describe, expect, test } from "bun:test";
import { tick, rollProbe } from "../src/keeper/keeper.ts";
import { listKeeperReleases } from "../src/keeper/record.ts";
import { EMPTY, FAIL_AFTER, SLOW_MS, tally, verifyUnderLoad, type Probe } from "../src/keeper/verify.ts";
import { keeperForge } from "./helpers/keeper.ts";
/** A clock that only the probes and the sleeps move. */
function clocked(healthz: (n: number) => Probe, pages: readonly string[] = [], page: Probe = { status: 200, ms: 0 }) {
  let now = 0;
  let n = 0;
  let open = 0;
  let mostOpen = 0;
  const run = verifyUnderLoad({
    get: async (path) => {
      if (path === "/healthz") {
        const p = healthz(n++);
        now += p.ms;
        return p;
      }
      open++;
      mostOpen = Math.max(mostOpen, open);
      await Promise.resolve();
      now += page.ms;
      open--;
      return page;
    },
    now: () => now,
    sleep: async (ms) => {
      now += ms;
    },
    pages,
  });
  return { run, mostOpen: () => mostOpen };
}
describe("tally", () => {
  test("a bad probe is a non-200 or one over 10 s; a slow 200 is slow, not dark", () => {
    let t = tally(EMPTY, { status: 200, ms: SLOW_MS + 1 });
    expect(t).toMatchObject({ probes: 1, non200: 0, badInRow: 1, worstMs: 10_001, secondsDark: 0 });
    t = tally(t, { status: 502, ms: 3 });
    expect(t).toMatchObject({ probes: 2, non200: 1, badInRow: 2, secondsDark: 1 });
    t = tally(t, { status: 200, ms: 5 });
    expect(t).toMatchObject({ badInRow: 0, worstRun: 2 });
    // Exactly 10 s is not over it.
    expect(tally(EMPTY, { status: 200, ms: SLOW_MS }).badInRow).toBe(0);
  });
});
describe("under page load", () => {
  test("a healthy release: 600 probes in 10 minutes, one a second, and it passes", async () => {
    const v = await clocked(() => ({ status: 200, ms: 0 })).run;
    expect(v.ok).toBe(true);
    expect(v.tally.probes).toBe(600);
    expect(v.tally.secondsDark).toBe(0);
  });
  test("two bad probes in a row pass; three fail at once, with the numbers", async () => {
    const two = await clocked((n) => (n === 10 || n === 11 ? { status: 503, ms: 1 } : { status: 200, ms: 1 })).run;
    expect(two.ok).toBe(true);
    expect(two.tally).toMatchObject({ non200: 2, worstRun: 2, secondsDark: 2 });
    const three = await clocked((n) => (n >= 10 && n < 10 + FAIL_AFTER ? { status: 0, ms: 1 } : { status: 200, ms: 1 })).run;
    expect(three.ok).toBe(false);
    expect(three.tally.probes).toBe(13);
    expect(three.reason).toBe("3 bad /healthz probes in a row (non-200 or over 10 s) after 13 probes; 3 non-200, worst 1 ms");
  });
  test("three probes over 10 s fail it though each was a 200", async () => {
    const v = await clocked((n) => ({ status: 200, ms: n >= 5 && n < 8 ? 10_500 : 2 })).run;
    expect(v.ok).toBe(false);
    expect(v.tally).toMatchObject({ non200: 0, worstMs: 10_500, secondsDark: 0 });
  });
  test("the pages are fetched one at a time, every one of them", async () => {
    const pages = ["/boards", "/alice/forge", "/explore", "/"];
    const c = clocked(() => ({ status: 200, ms: 0 }), pages, { status: 200, ms: 250 });
    const v = await c.run;
    expect(c.mostOpen()).toBe(1);
    expect(v.tally.pages).toBeGreaterThan(pages.length);
    expect(v.tally.worstMs).toBe(250);
  });
});
describe("a seeded healthy release", () => {
  test("ends verified live with its probes and worst latency stored, and an event", async () => {
    const f = await keeperForge({ healthz: () => ({ status: 200, ms: 30 }) });
    await tick(f.ports);
    // The old forge counts the roll: zero non-200 with surge.
    for (let i = 0; i < 5; i++) await rollProbe(f.ports);
    f.world.live = "0.7.284";
    const line = await tick(f.ports);
    expect(line).toContain("0.7.284 verified live");
    const [r] = listKeeperReleases(f.app.db);
    expect(r!.stage).toBe("verified");
    expect(r!.rollProbes).toBe(5);
    expect(r!.rollNon200).toBe(0);
    expect(r!.probes).toBeGreaterThan(500);
    expect(r!.pages).toBeGreaterThan(0);
    expect(r!.worstMs).toBe(40);
    expect(r!.secondsDark).toBe(0);
    expect(r!.liveAt).not.toBeNull();
    const [ev] = f.events("release.verified_live");
    expect(ev).toMatchObject({ version: "0.7.284", probes: r!.probes, worstMs: 40, secondsDark: 0 });
    expect(f.world.pages.slice(0, 4)).toEqual(["/boards", "/alice/forge", "/explore", "/"]);
    // Nothing went dark, so nothing is filed.
    expect(f.app.db.query("SELECT COUNT(*) AS n FROM issue WHERE title LIKE 'Release % dark%'").get()).toEqual({ n: 0 });
  });
  test("a roll that went dark files one issue, and only one", async () => {
    const f = await keeperForge({ healthz: () => ({ status: 200, ms: 5 }) });
    await tick(f.ports);
    const healthy = f.world.healthz;
    f.world.healthz = () => ({ status: 0, ms: 1 });
    for (let i = 0; i < 4; i++) await rollProbe(f.ports);
    f.world.healthz = healthy;
    f.world.live = "0.7.284";
    await tick(f.ports);
    const [r] = listKeeperReleases(f.app.db);
    expect(r).toMatchObject({ stage: "verified", rollNon200: 4, secondsDark: 4 });
    const filed = f.app.db.query<{ title: string; body: string }, []>("SELECT title, body FROM issue WHERE title LIKE 'Release %'").all();
    expect(filed.map((i) => i.title)).toEqual(["Release 0.7.284 was dark for 4 s"]);
    expect(filed[0]!.body).toContain("4 of 4 during the roll");
  });
  test("a closed issue about the same dark time does not stop the next report", async () => {
    const f = await keeperForge({ healthz: () => ({ status: 200, ms: 5 }) });
    f.app.db
      .query(`INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at) VALUES ('old', ?1, 900, 'issue', 'Release 0.7.284 was dark for 4 s', '', ?2, 'closed', ?3)`)
      .run(f.repo.id, f.user.id, new Date(f.world.now).toISOString());
    await tick(f.ports);
    const healthy = f.world.healthz;
    f.world.healthz = () => ({ status: 0, ms: 1 });
    for (let i = 0; i < 4; i++) await rollProbe(f.ports);
    f.world.healthz = healthy;
    f.world.live = "0.7.284";
    await tick(f.ports);
    const open = f.app.db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM issue WHERE title = 'Release 0.7.284 was dark for 4 s' AND state = 'open'").get();
    expect(open).toEqual({ n: 1 });
  });
});
@@ -66,7 +66,10 @@
describe("verify an email from the People screen", () => {
    try {
      expect(h.text()).toMatch(/v +verify/);
      expect(h.text()).toMatch(/bobby.*unverified/);
      // newest first: bobby is the first row
      // Newest first, but alice and bobby can share a created_at millisecond and then
      // sort by name: put the cursor on bobby rather than assume his row.
      if (!/▸ bobby/.test(h.text())) await h.press("j");
      expect(h.text()).toMatch(/▸ bobby/);
      await h.press("v");
      expect(h.text()).toContain("verify bobby's email? y/n");
      await h.press("n");
@@ -0,0 +1,61 @@
/**
 * The release keeper in the terminal (#7557): the palette offers it to whoever
 * may read /releases, and the screen shows the card's read — what holds the
 * next cut, then each release with its stage, known-good version and numbers.
 */
import { describe, expect, test } from "bun:test";
import { tick } from "../src/keeper/keeper.ts";
import type { User } from "../src/core/users.ts";
import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
import type { Ctx } from "../src/tui/app/app.ts";
import { keeperScreen, readKeeper } from "../src/tui/screens/keeper.ts";
import { headlessTerminal } from "../src/tui/term/terminal.ts";
import { quietLogs } from "./helpers/globals.ts";
import { keeperForge } from "./helpers/keeper.ts";
import { settleApp } from "./helpers/tui.ts";
quietLogs();
const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
/** A forge whose 0.7.284 failed its verify and was rolled back to 0.7.283. */
async function heldForge() {
  const f = await keeperForge();
  await tick(f.ports);
  f.world.live = "0.7.284";
  f.world.healthz = (n) => (n >= 5 ? { status: 503, ms: 2 } : { status: 200, ms: 2 });
  await tick(f.ports);
  return f;
}
describe("the release keeper screen", () => {
  test("the palette offers it to an admin, not to a guest", async () => {
    const f = await keeperForge();
    const offered = (user?: User) => forgeCommands(new Session(f.app, identity(user)))({} as Ctx).some((c) => c.id === "keeper");
    expect(offered(f.user)).toBe(true);
    expect(offered(undefined)).toBe(false);
    expect(readKeeper(new Session(f.app, identity(undefined)))).toBeUndefined();
  });
  test("it shows what holds the next cut and the rolled-back release with its numbers", async () => {
    const f = await heldForge();
    const terminal = headlessTerminal(160, 20);
    const { app: tui, session } = createForgeApp(f.app, identity(f.user), { terminal, exitHooks: false });
    tui.start();
    tui.ctx.push(keeperScreen(session, tui.ctx));
    await settleApp(tui);
    const text = tui.screen.toText();
    tui.stop?.();
    expect(text).toContain("0.7.284 failed its verify");
    expect(text.replace(/\s+/g, " ")).toContain(`a pull saying "release hold 1" to land`);
    expect(text).toContain("rolled back");
    expect(text).toContain("known good 0.7.283");
    expect(text).toMatch(/8 probes/);
  });
  test("with nothing cut, it says nothing holds the next cut and that no releases are kept yet", async () => {
    const f = await keeperForge();
    const read = readKeeper(new Session(f.app, identity(f.user)));
    expect(read).toEqual({ blocked: "Nothing holds the next cut.", releases: [] });
  });
});
		@@ -455,6 +455,16 @@ What the crew's tokens bought is on the board (`/board/returns`), in MCP `return
455	455
456	456	The terminal's palette has `Signed in` for anyone signed in: your browsers, tokens and this terminal's key, with when each was last seen; `d` signs the selected browser out after a confirm, as `/settings` and MCP `sessions` do, and on a token revokes it after a confirm (Keep is the default) through the same `deleteToken` action as Settings and the API, so it stops working at once (`docs/CREW.md`, "Where you are signed in").
457	457
	458	The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases on the forge's own one-minute timer, with no model. It is off until `FORGE_RELEASE_KEEPER=on`, and the pod's service account may `get` and `patch` the forge Deployment.
	459
	460	- Cut: it tags main's head as `v<patch + 1>` when main has moved past a live tag, nothing is in flight, the merge queue is still, 30 minutes have passed since its last cut and no hold stands.
	461	- Verify: it records the known-good version and image before the tag, sets the Deployment's image, and has the new version spend ten minutes under page load probing `/healthz` each second.
	462	- Roll back and hold: three bad probes in a row hold further cuts with the numbers and set the recorded known-good image back by name, never "undo". A pull saying `release hold <N>` lets the fix through, and that release verifying lifts the hold.
	463	- Owner's hold: set and lifted from the card, the terminal (`h`) or MCP `release_hold`, by a person who may roll back, never an agent.
	464	- Where to see it: the Release keeper card on `/releases`, `Release keeper` in the terminal's palette, the Maintenance board under `/boards`, and MCP `release_keeper`, each gated as `/releases` is: a guest sees none of it.
	465
	466	How it works and how to turn it on: [`docs/features/release-keeper.md`](docs/features/release-keeper.md).
	467
458	468	`/changelog` lists the last 30 releases of `FORGE_REPO` (public only) with their pulls, for anyone; the terminal's palette has `Changelog` for the same list. In the terminal, `/` takes a word and keeps the releases whose version or pull titles contain it (`esc` clears it); Enter on a release lists its pulls, and Enter on a pull opens it (`docs/features/changelog.md`).
459	469
460	470	A repository admin cuts a wave from two doors, both through `cutWaveFor` (`src/actions/cut_wave.ts`): Settings → Cut a wave, and `c` on the terminal's waves screen, which says how many pulls would go in, asks `cut wave N? y/n` (`y` and `n` choose outright) and shows the draft pull's number (Enter opens it). Each opens a draft pull listing the pulls merged since the last wave; nothing merged, an earlier wave that does not parse, or a viewer who is not an admin is refused in the action's words, writing nothing.
		@@ -138,6 +138,8 @@ for a tunnel and a forge that sleeps.
138	138	\| `queue_gates` \| `{ repo, limit? }`: the merge queue's recent gates, newest first (`limit` 1–100, default 20), as `GET …/queue/gates` answers: each gate's pulls, state, reason, counts and failing tests by name with their first error lines, a test red in two gates that share no pull flagged `flaky` \| \|
139	139	\| `quotas` \| `{ key?, value?, dryRun? }` (forge admin): the per-account quotas `repositories`, `apps`, `storage` (MB), `previews`, `runs`, `checks`, each a number or null for none. `repositories` bounds repos on create, fork and copy; `apps` the apps made; `storage` the MB of repos plus app data, asked on push, create and fork; `previews` live previews; `runs` concurrent agent runs; `checks` concurrent checks. Each is asked before the thing is made or started and a forge admin is never limited; with `key` and `value` (`""` clears), planned first (`dryRun` is true unless `false`); applying needs an admin-scope token \| yes \|
140	140	\| `read_file` \| a file (utf-8, or base64 for binary) or a directory listing at a ref, as GET …/contents; MP4/MOV/M4A files also carry about: {kind, seconds, width?, height?, says} \| \|
	141	\| `release_hold` \| `{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent \| yes \|
	142	\| `release_keeper` \| `{}`: what the release keeper (#7557) did: its releases newest first (version, stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases \| \|
141	143	\| `releases` \| `{shipped: true}`: what went live, for anyone: `version`, `commit`, `live` (newest first: `repo`, `number`, `title`, `at`) and `onItsWay` (landed, not yet in the running commit), the list /changelog shows. `{}`: the forge's kept releases newest first: short sha, version, the pull that landed it, when, and for the live one `live: true` and `since`; for whoever may read /releases, others get its refusal; no rollback, a person does that on the page \| \|
142	144	\| `repo` \| `{ repo }`: one repository's description, visibility, autonomy, `audience { level, sentence }` and `provenance` (where a migrated one came from, `null` when hidden), the same as `GET /api/v1/repos/:owner/:name` \| \|
143	145	\| `repo_labels` \| `{ repo, action?, name?, color?, id? }`: list (`list`, the default, paged by `limit` 1–200 and `cursor`, as `{ items, next_cursor }`; a read-only token may), create (`create`, `name`, `color` six hex digits), rename or recolour (`update`, `id`, `name` and/or `color`, as the labels page) or remove (`remove`, `id`) the repository's labels, as `GET`/`POST …/labels` and `DELETE …/labels/:id`; create, update and remove need an admin token and a repository admin \| yes \|
		@@ -0,0 +1,102 @@
	1	# Release keeper
	2
	3	The release keeper (#7557) cuts, promotes, verifies and, when it has to,
	4	rolls back the forge's own releases. It is plain code on the forge's own
	5	timer, with no model, so releases keep going out when no one is at the
	6	keyboard. It does what the steward's `release.sh`, `verify-after.sh` and
	7	`verify-release.sh` did. The code is `src/keeper/`.
	8
	9	## What it does, each minute
	10
	11	1. Cut on cadence (`keeper/cadence.ts`). It cuts when all of these hold:
	12	- main has moved past the newest `v*` tag;
	13	- no keeper release is rolling or verifying;
	14	- the newest tag is live, or the keeper rolled it back;
	15	- no merge-queue gate is running;
	16	- 30 minutes have passed since the last cut;
	17	- no hold stands.
	18
	19	Before anything moves, it records the known-good version and the exact
	20	image the Deployment runs. Then it makes the annotated tag
	21	`v<version>` (patch + 1) on main's head, with the same record
	22	`scripts/cut-release.ts` writes, and pushes the tag to the repository's
	23	origin when it has one. The cut is a tag, never a commit, so it voids no
	24	gate in flight (#6640). The row is written before the tag: if the forge
	25	stops between the two, the next tick sends the tag, and a refused tag
	26	drops the row (and the local tag) so nothing waits on a release that
	27	never left.
	28	2. Promote. It sets the Deployment's forge container to
	29	`<same image repository>:<version>`. The chart rolls with
	30	`maxSurge 1, maxUnavailable 0`. If the image is not built yet, the new
	31	pod waits on its pull while the old one keeps serving. While the roll
	32	runs, the old forge probes `/healthz` through the public URL once a
	33	second and counts every probe that got no 200.
	34	3. Verify under page load (`keeper/verify.ts`). The new forge sees
	35	that it is the version in flight and spends 10 minutes on two things at
	36	once:
	37	- it fetches `/boards`, the forge's repository page, `/explore` and `/`,
	38	one at a time, signed in as the first site admin (a session made for
	39	the verify and deleted after it);
	40	- it probes `/healthz` once a second.
	41
	42	A probe is bad when it is not a 200 or takes over 10 s. Three bad probes
	43	in a row fail the release.
	44	4. Verified live, or roll back and hold. On a failure, the keeper
	45	first writes a hold with the version, the reason and the probe numbers.
	46	Then it sets the Deployment to the known-good image recorded at the cut,
	47	by name (`keeper/rollback.ts`). There is no "undo to previous": on
	48	2026-10-27 a `rollout undo` put the broken version back for two minutes.
	49	A release that is not live 3 h after its cut is rolled back the same way.
	50
	51	Every step is an event on the outbox: `release.cut`, `release.released`,
	52	`release.verified_live`, `release.rolled_back` and `release.held`.
	53
	54	Seconds dark counts the probe-seconds without a 200, through the roll
	55	and the verify together. The target is 0. A release with any dark seconds
	56	files one issue, `Release <version> was dark for <n> s`.
	57
	58	## Holds
	59
	60	- A failed verify's hold blocks every cut. Land a pull whose title or
	61	body says the hold by number (`release hold 3`). Naming the version alone
	62	(`0.7.284`) does not count: changelogs and revert notes mention it too.
	63	The keeper then cuts the fix, and the hold is lifted when that release
	64	verifies.
	65	- An owner's hold is set with MCP `release_hold {reason}`, with the Hold form on the `/releases` card, or with `h` on the terminal screen, and lifted
	66	(`release_hold {lift: <id>}`, the card's Lift button, `h` then `lift N`) — one function, `holdAs`/`liftAs`, behind all three.
	67	It needs a person who may roll back (MCP: on an admin-scope token); an agent never can.
	68
	69	## Where to see it
	70
	71	- The terminal: Release keeper in the command palette shows the same
	72	read as the card, for the same people.
	73	- `/releases`: the Release keeper card says what holds the next cut and
	74	lists each release with its stage, the known-good version, the roll's
	75	non-200s, the probes and pages, the worst latency and the seconds dark.
	76	- The Maintenance board (`/boards`, from the `keeperReleases` source)
	77	charts seconds dark and worst latency per release, and releases by
	78	outcome.
	79	- MCP `release_keeper` returns the same data as JSON.
	80
	81	## Turning it on
	82
	83	The keeper is off unless all of these are true:
	84
	85	- `FORGE_RELEASE_KEEPER=on`;
	86	- `FORGE_REPO` names a repository on this forge;
	87	- the forge runs in a cluster.
	88
	89	`FORGE_KEEPER_DEPLOYMENT` and `FORGE_KEEPER_CONTAINER` name the Deployment
	90	and its container (both default `forge`). The pod's service account needs
	91	`get` and `patch` on that Deployment; the chart does not grant them. The
	92	boot log says why it is off.
	93
	94	Guardrail: the keeper writes only a tag, the Deployment's image, its own
	95	rows and issues. It has no account that may land a pull or push main.
	96	Rollback is the one action it takes without a canary, because the image it
	97	sets was already live.
	98
	99	Assumption: the keeper promotes by setting the image directly, as the
	100	steward's rollback did. If Flux reconciles the image back, the keeper sets
	101	it again on its next tick while the release is rolling. If it never takes,
	102	the 3 h deadline rolls it back and holds.
		@@ -285,6 +285,28 @@ export const SOURCES: Readonly<Record<string, Source>> = {
285	285	reports: "reports kept that day (production only)",
286	286	},
287	287	},
	288	// The release keeper's releases (#7557), one row each: what the probes
	289	// counted through the roll and the ten minutes under page load. Seconds
	290	// dark is the target-zero number. The stop reason is free text, not listed.
	291	keeperReleases: {
	292	table: "keeper_release",
	293	owner: "admin",
	294	time: "cut_at",
	295	columns: {
	296	version: "text", stage: "text", cut_at: "time", live_at: "time", finished_at: "time",
	297	roll_probes: "number", roll_non200: "number", probes: "number", pages: "number",
	298	bad_in_row: "number", worst_ms: "number", seconds_dark: "number",
	299	},
	300	docs: {
	301	stage: "rolling, verifying, verified (verified live) or rolled_back",
	302	roll_non200: "/healthz probes, one a second, that got no 200 while the roll ran (target 0 with surge)",
	303	probes: "/healthz probes in the ten minutes under page load",
	304	pages: "pages fetched as an admin under load, one at a time",
	305	bad_in_row: "the most bad probes in a row (non-200 or over 10 s); 3 fails the release",
	306	worst_ms: "the slowest probe or page, ms",
	307	seconds_dark: "probe-seconds with no 200, roll and verify together (target 0)",
	308	},
	309	},
288	310	// Forges made by the one-click path (#2914): one row per spawn event
289	311	// (`actions/spawn.ts`, `spawn/live.ts`). The forge's name and the failure's
290	312	// words are not listed; the step is one of a fixed few.
		@@ -1,5 +1,5 @@
1	1	/**
2		* The boards the forge offers a new install: Usage, Serving, Crew, Releases, Apps, App restarts, Security, Throughput, Forges and Reach. They are plain
	2	* The boards the forge offers a new install: Usage, Serving, Crew, Releases, Maintenance, Apps, App restarts, Security, Throughput, Forges and Reach. They are plain
3	3	* boards in `forge/boards` once written, so a person edits or deletes them
4	4	* like any other; `.offered` (a JSON list of slugs, committed with the
5	5	* boards) records what was offered so a deletion sticks and an edit is
		@@ -83,6 +83,18 @@ export const STARTING_BOARDS: Readonly<Record<string, Board>> = {
83	83	panel("Decider time per hour (ms)", "line", { source: "decisions", aggregate: "avg", column: "ms", groupBy: HOUR, series: "kind" }, 2),
84	84	],
85	85	},
	86	// The release keeper (#7557): every release dark for 0 s is the target.
	87	maintenance: {
	88	title: "Maintenance",
	89	range: "30d",
	90	refresh: "1m",
	91	panels: [
	92	panel("Seconds dark per release", "bar", { source: "keeperReleases", aggregate: "max", column: "seconds_dark", groupBy: "version" }, 2),
	93	panel("Worst latency per release (ms)", "bar", { source: "keeperReleases", aggregate: "max", column: "worst_ms", groupBy: "version" }, 2),
	94	panel("Releases by outcome", "bar", { source: "keeperReleases", aggregate: "count", groupBy: "stage" }),
	95	panel("Seconds dark (30 days)", "number", { source: "keeperReleases", aggregate: "sum", column: "seconds_dark" }),
	96	],
	97	},
86	98	apps: {
87	99	title: "Apps",
88	100	range: "7d",
		@@ -177,6 +177,8 @@ export function classifiedAudience(kind: string): EventAudience \| undefined {
177	177	kind === "repo.crew_outsiders" \|\|
178	178	// A clog starting names the forge's own pile-up: the owner's to hear (#6524).
179	179	kind.startsWith("clog.") \|\|
	180	// The release keeper's steps name images and probe numbers: the forge owner's (`keeper/keeper.ts`, #7557).
	181	kind.startsWith("release.") \|\|
180	182	// A file held against new work is its admins' book-keeping (#6522).
181	183	kind === "repo.file_held" \|\|
182	184	kind === "repo.file_released"
		@@ -0,0 +1,1 @@
	1	e24c6b4f77b34f5ba3d6f043f2da2788dd0b2e239ebfd5970e12c3b99b1016be
		@@ -0,0 +1,39 @@
	1	// The release keeper (#7557): one row per release it cut, with the
	2	// known-good image it rolls back to and what the probes counted, and the
	3	// holds that stop further cuts. `kind` is `failed` (a verify that failed,
	4	// lifted by a verified release after a pull naming it lands) or `owner`
	5	// (set and lifted by an admin).
	6	export default `
	7	CREATE TABLE keeper_release (
	8	id INTEGER PRIMARY KEY AUTOINCREMENT,
	9	version TEXT NOT NULL UNIQUE,
	10	sha TEXT NOT NULL,
	11	known_good_version TEXT NOT NULL,
	12	known_good_image TEXT NOT NULL,
	13	stage TEXT NOT NULL,
	14	cut_at TEXT NOT NULL,
	15	live_at TEXT,
	16	finished_at TEXT,
	17	roll_probes INTEGER NOT NULL DEFAULT 0,
	18	roll_non200 INTEGER NOT NULL DEFAULT 0,
	19	probes INTEGER NOT NULL DEFAULT 0,
	20	pages INTEGER NOT NULL DEFAULT 0,
	21	bad_in_row INTEGER NOT NULL DEFAULT 0,
	22	worst_ms INTEGER NOT NULL DEFAULT 0,
	23	seconds_dark INTEGER NOT NULL DEFAULT 0,
	24	reason TEXT NOT NULL DEFAULT ''
	25	);
	26	CREATE TABLE keeper_hold (
	27	id INTEGER PRIMARY KEY AUTOINCREMENT,
	28	kind TEXT NOT NULL,
	29	version TEXT NOT NULL,
	30	reason TEXT NOT NULL,
	31	probes INTEGER NOT NULL DEFAULT 0,
	32	bad_in_row INTEGER NOT NULL DEFAULT 0,
	33	worst_ms INTEGER NOT NULL DEFAULT 0,
	34	created_at TEXT NOT NULL,
	35	cause_pull INTEGER,
	36	lifted_at TEXT,
	37	lifted_by TEXT
	38	);
	39	`;
		@@ -108,6 +108,8 @@ export { tool as pulls } from "./pulls.ts";
108	108	export { tool as queue_gates } from "./queue_gates.ts";
109	109	export { tool as quotas } from "./quotas.ts";
110	110	export { tool as read_file } from "./read_file.ts";
	111	export { tool as release_hold } from "./release_hold.ts";
	112	export { tool as release_keeper } from "./release_keeper.ts";
111	113	export { tool as releases } from "./releases.ts";
112	114	export { tool as repo } from "./repo.ts";
113	115	export { tool as repo_labels } from "./repo_labels.ts";
		@@ -0,0 +1,24 @@
	1	import { holdAs, liftAs, MAX_REASON } from "../../../keeper/hold.ts";
	2	import { bad, good, str, type Tool } from "../shared.ts";
	3
	4	export const tool: Tool = {
	5	name: "release_hold",
	6	doc: "`{ reason }` or `{ lift }`: hold the release keeper (#7557) with a reason, so it cuts nothing until the hold is lifted; or lift an owner hold by its id. A failed verify's hold is lifted by a verified release after a pull saying `release hold N` lands, not here. A person who may roll back, with an admin-scope token, never an agent",
	7	description: "Hold or release the release keeper's cuts (admin).",
	8	inputSchema: {
	9	type: "object",
	10	properties: { reason: { type: "string", maxLength: MAX_REASON }, lift: { type: "integer", minimum: 1 } },
	11	},
	12	writes: true,
	13	needsAdminScope: true,
	14	run(ctx, user, args) {
	15	const id = args["lift"];
	16	if (id !== undefined) {
	17	if (typeof id !== "number") return bad("invalid", "lift is a hold's id");
	18	const lifted = liftAs(ctx.app, user, id);
	19	return lifted.ok ? good({ lifted: lifted.value }) : bad(lifted.error, lifted.message);
	20	}
	21	const held = holdAs(ctx.app, user, str(args["reason"]));
	22	return held.ok ? good({ hold: held.value }) : bad(held.error, held.message);
	23	},
	24	};