e252cc9 · d161fea · 6452b1a · 98a702a · bad1481 · 852ee33
@@ -25,6 +25,8 @@
State shared between modules through process memory, not the database, is named
`src/core/transfer.ts`: the old owner/name pairs a transfer left behind (`repo_redirect`), held per database after the first request reads them and dropped by `transferRepo`, so a request under a path nothing moved from costs no statement (#569).
`src/system/footing.ts`: `footing`, the last reading of what the forge stands on from the kube API (#11685), held 60 s so /system, its terminal screen (`src/tui/screens/footing.ts`, palette entry for admins) and the MCP `system` tool (`kind: footing`) share one read; a probe that throws is not retried for 60 s; one reading, gone on restart.
`src/apps/timer_fire.ts`: `inFlight`, the ids of timers being fired in this process, so a slow app's timer is never fired twice at once; at most the timers due, gone when each settles. The timers themselves are rows (`timer`), read at every boot, never memory.
## Awake ceiling and the boot
@@ -209,7 +209,7 @@
bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
| `subscription_wake` | — | Admin: wake a resting subscription | 168 |
| `subscriptions` | — | Claude accounts: scopes, budgets | 158 |
| `sync` | — | Sync a mirror now; `originId` reconnects it | 197 |
| `system` | `csp`, `routes` | The forge machine now: CPU, memory, disk, network | 286 |
| `system` | `csp`, `routes`, `footing` | The forge machine now: CPU, memory, disk, network | 296 |
| `take_back` | — | Open the pull that undoes a shipped pull | 219 |
| `take_updates` | — | Take the parent forge's updates into a fork | 176 |
| `teams` | — | Crew teams for person admins: charter, dial (0–100) and member agents | 338 |
@@ -89,6 +89,69 @@
Unset, only this machine may say so, and the named header is never read. Everyth
instance _is_ lives under the data directory — the SQLite index, the bare
repos, and the SSH host key, in plain layout. Tar it up and it moves.
## What the forge stands on
Each thing the forge needs to answer a request, sign a person in, build,
release and back up, the machine it runs on, what happens to the forge if
that machine dies, and the issue that removes the dependency. The list is the
steward's read-only inventory of both machines on 2026-11-12 (#11685), the day
pg-grand took the forge off the internet for 24 minutes while the forge itself
ran fine on the vxrail, and the day the owner decided to drop pg-grand
(#11686). Every row on pg-grand is a way that machine takes the forge
down.
<!-- footing:start — written by footingTable() in src/system/footing.ts; test/footing.test.ts holds it equal -->
- **pg-grand**: DigitalOcean. The k3s server, a single control plane with no HA. The owner decided on 2026-11-12 to drop it.
- **vxrail**: A k3s agent joined to pg-grand. The forge runs here.
| Needed to | Piece | Runs on | If that machine dies | Breaks | Removed by |
|---|---|---|---|---|---|
| (a) answer a request | The forge's pod (git.open-platform.sh/trevato-apps/forge:&lt;version&gt;) and its data, PVC forge-data-vxrail (60 Gi) | vxrail | The forge is down, and its data is on that disk. | — | — |
| (a) answer a request | Cloudflare tunnel carrying forge.open-platform.sh (standby/cloudflared), moved to the vxrail by hand on 2026-11-12; its last-applied manifest still says 0 replicas, so a re-render moves it back | vxrail | It goes with the forge, which runs beside it. | — | #8948, #11686 |
| (a) answer a request | The vxrail's own tunnel (standby/cloudflared-forge-home, 2 replicas): it too sends *.open-platform.sh to Traefik; only forge-home-test.open-platform.sh and arm--forge-trevato-apps.open-platform.sh go straight to the forge | vxrail | It goes with the forge, which runs beside it. | — | — |
| (a) answer a request | Traefik (kube-system/traefik): every public request goes tunnel → Traefik → the forge's Service | pg-grand | The forge is off the internet; it keeps running inside. | 1 | #11686, #8946 |
| (a) answer a request | Child and app hosts, Traefik IngressRoutes forge-children (&lt;name&gt;--forge.open-platform.sh), forge-apps (*--forge-trevato-apps.open-platform.sh) and forge-old-host, and game.cheffryjames.com and www.cheffryjames.com in the forge's namespace | pg-grand | No child forge, app or custom domain answers. | 1 | #11686, #8946 |
| (a) answer a request | forge-ssh, a host socket proxy (:2222 → the forge's ssh service) | pg-grand | git over ssh and the terminal stop answering. | 7 | #11686 |
| (a) answer a request | The k3s API server (the vxrail is an agent joined to it) | pg-grand | Running pods keep running, but nothing reschedules, Services stop updating, and no release rolls or Job runs. | 5 | #11686 |
| (b) sign a person in | Sign-in edge: a forwardAuth middleware on the forge's Ingress, answered by mcp/plat-auth | pg-grand | Every request through the edge answers 500, /healthz too. | 2 | #11686 |
| (b) sign a person in | forge-signup-gate, the pod /signup is routed to | pg-grand | No one can sign up. | 3 | #11686 |
| (c) build | Forgejo (git.open-platform.sh) and its Actions, which build the release image | pg-grand | No release image is built. | 4 | #11686, #8941 |
| (c) build | Forgejo's database, a single instance under CloudNativePG (cnpg-system) | pg-grand | Forgejo stops, and with it builds. | 4 | #11686, #8941 |
| (c) build | forgejo-runner-vxrail, a Forgejo Actions runner | vxrail | It goes with the forge, which runs beside it. | — | — |
| (d) release | Forgejo's registry (git.open-platform.sh), which the forge's image is pulled from | pg-grand | A pod that restarts on a node without the image cached cannot start; nothing new is released. | 4 | #11686, #8941, #8942 |
| (d) release | registry-node, a registry on the vxrail | vxrail | It goes with the forge, which runs beside it. | — | — |
| (d) release | forge-release-tick, a host timer every 5 min | pg-grand | Nothing is released; production keeps the release it has. | 7 | #11686, #8942 |
| (d) release | standby-reconcile, a host timer every 2 min that applies the forge's manifests from Forgejo; Flux's Kustomization for the forge is suspended | pg-grand | A changed manifest is not applied; the running pod is untouched. | 7 | #8948, #11686 |
| (d) release | Host timers forge-watchdog, forge-github-mirror and forge-prune-images | pg-grand | The watchdog, the GitHub mirror and image pruning stop. | 7 | #11686 |
| (e) back up | The backup namespace's CronJobs (FORGE_BACKUP_LANES) | pg-grand | No new backup is taken. | 6 | #11686 |
| (e) back up | MinIO, where the backup lanes store the forge's backups | pg-grand | No new backup is stored, and those it holds go with the machine. | 6 | #11686 |
If pg-grand were switched off now, the forge breaks in this order (the Breaks column):
1. Traefik: nothing public reaches the forge
2. plat-auth: no one signs in
3. the signup gate: no one signs up
4. image pulls and releases
5. the control plane: nothing reschedules, Services stop updating
6. local backups
7. the host timers
Also on pg-grand, not in the forge's path: Forgejo and its database, the platform's mcp/plat-auth and www, the CloudNativePG operator with the tenant apps' databases, their local-path volumes pinned to that node, and MinIO.
<!-- footing:end -->
The same list is read from the live cluster on `/system`, under "What the
forge stands on", on the terminal's screen of that name, and from the MCP
`system` tool with `{ kind: "footing" }`; how to read each is in
[docs/features/footing.md](features/footing.md). The probe
(`src/system/footing.ts`) asks the kube API, with the forge's own service
account, for the forge's node, the nodes labelled control-plane, and the pods
in each namespace above; it never lists every pod in the cluster. What the
cluster cannot show keeps its declared machine and says why. To see every
row from the cluster, the forge's service account needs `get` on its own pod,
`list` on `nodes`, and `list` on `pods` in those namespaces (#8948). The
reading is held 60 s, a page waits at most 0.8 s for it, and a probe that
fails is not retried for a minute.
## Health
`GET /api/v1/health` (and the MCP `health` tool) answer a site admin with
@@ -0,0 +1,58 @@
# What the forge stands on (#11685)
The pieces the forge needs to answer a request, sign a person in, build,
release and back up, and the machine each one runs on. Any piece that is not
on the forge's own machine is a way that machine can take the forge down. The
declared list is the table in [`docs/OPERATING.md`](../OPERATING.md#what-the-forge-stands-on);
the three doors below read it against the live cluster.
All three are for a forge admin who is a person (the MCP tool: a forge admin).
None of them waits on the cluster: they show the last reading, held 60 s, and
the declared list while the first one runs.
## On /system
Under **What the forge stands on**, the summary says how many pieces run on
another machine. Open it to see:
- where the forge runs (its pod's node), or why that was not read;
- each piece on another machine, in the order switching pg-grand off would
  break the forge, with what it is needed for, where it runs, what happens if
  that machine stops, its place in that order, and the issues that remove it;
- the pieces on the forge's own machine, by name;
- what each machine is.
"Runs on" is a node name when the cluster showed the piece's pod there. It
reads `pg-grand (declared: …)` when the cluster could not show it, with the
reason: a systemd unit on a host, a CronJob with no pod between runs, a
namespace the service account may not list, or a pod whose namespace is not
recorded. A declared piece is judged by its declared machine.
## In the terminal
Open the palette and choose **What the forge stands on** (it follows
**Main-thread stalls**). The screen lists the same rows as the page: the
forge's node, then each piece on another machine in break order with its
step, need, machine and what happens if it dies, then the steps themselves and
the pieces on the forge's own machine. `j`/`k` scroll, `esc` closes.
## Over MCP
```json
{ "name": "system", "arguments": { "kind": "footing" } }
```
answers `forgeNode`, `unread` (why the forge's node was not read, or null),
`at`, and per piece `{ id, need, what, machine, seen, risk, ifDies, issues,
breaks }`. `seen` is `{ by: "cluster", nodes }` when the cluster showed it and
`{ by: "declared", why }` otherwise; `risk` is true when the piece is not on
the forge's machine. `risks` repeats those pieces as one line each, in break
order. Anyone but a forge admin is refused.
## Keeping it true
The list is `FOOTING` in `src/system/footing.ts`; `test/footing.test.ts` holds
the OPERATING.md table equal to `footingTable()`. Change the list there, then
paste the output of
`bun -e 'import {footingTable} from "./src/system/footing.ts"; console.log(footingTable())'`
between the markers in OPERATING.md.
@@ -7,16 +7,17 @@
import { stallHour, TOP_ROUTES } from "../../../metrics/stalls.ts";
import { lagMinutes, loopWatch, perfRecord } from "../../../perf/live.ts";
import { LAG_BAR_MS, overBar } from "../../../perf/lag_minutes.ts";
import { slowestRoutes } from "../../../perf/watch.ts";
import { footing, riskLines } from "../../../system/footing.ts";
import { good, bad, type Tool } from "../shared.ts";
export const tool: Tool = {
  name: "system",
  doc: "the machine the forge runs on, now: CPU, load, memory, disk, network, processes, `stalls`: the last hour's main-thread stalls by route (`<route> (in flight)` when still running at the wake), or `job:<name>`, as {cause, ms, count}, `readings`: the last 20 minutes' worst loop lag as {at, ms, what} beside `barMs` (100) and `over`, and `readers`: who polls the telemetry and how fast, `{ kind: \"routes\" }`: the 10 slowest budgeted routes as {route, p95Ms, budgetMs, n, slowestPhase}, over budget first; a forge admin, or an agent granted `telemetry:read`. `{ kind: \"csp\" }`: the last 24 h of Content-Security-Policy violations browsers reported, `{ directive, n, newest }` per directive (`newest` the blocked host last counted), most first, counted per UTC hour from `since`, and `scriptErrors` `{ page, where, message, n, last }`, as the security page reads them; a forge admin only",
  doc: "the machine the forge runs on, now: CPU, load, memory, disk, network, processes, `stalls`: the last hour's main-thread stalls by route (`<route> (in flight)` when still running at the wake), or `job:<name>`, as {cause, ms, count}, `readings`: the last 20 minutes' worst loop lag as {at, ms, what} beside `barMs` (100) and `over`, and `readers`: who polls the telemetry and how fast, `{ kind: \"routes\" }`: the 10 slowest budgeted routes as {route, p95Ms, budgetMs, n, slowestPhase}, over budget first; a forge admin, or an agent granted `telemetry:read`. `{ kind: \"csp\" }`: the last 24 h of Content-Security-Policy violations browsers reported, `{ directive, n, newest }` per directive (`newest` the blocked host last counted), most first, counted per UTC hour from `since`, and `scriptErrors` `{ page, where, message, n, last }`, as the security page reads them; a forge admin only. `{ kind: \"footing\" }`: what the forge stands on (#11685), as /system's \"What the forge stands on\": `forgeNode`, and per piece `{ id, need, what, machine, seen, risk, ifDies, issues }` read from the live cluster, with `risks`, one line per piece not on the forge's machine; a forge admin only",
  description:
    "The forge machine now: CPU, memory, disk, network. Admin or telemetry:read; `kind: csp`: 24 h of CSP violations (admin).",
  inputSchema: { type: "object", properties: { kind: { type: "string", enum: ["csp", "routes"] } } },
  inputSchema: { type: "object", properties: { kind: { type: "string", enum: ["csp", "routes", "footing"] } } },
  writes: false,
  run(ctx, user, args) {
  async run(ctx, user, args) {
    if (args["kind"] !== undefined) {
      if (args["kind"] === "routes") {
        const reader = telemetryReader(ctx.app, user);
@@ -24,7 +25,12 @@
export const tool: Tool = {
        noteRead(ctx.app, user, reader, "system");
        return good({ routes: slowestRoutes(perfRecord, Date.now()) });
      }
      if (args["kind"] !== "csp") return bad("invalid", "kind is csp, routes, or absent for the machine");
      if (args["kind"] === "footing") {
        if (!user.isAdmin) return bad("forbidden", "only an admin of this forge reads what it stands on");
        const f = await footing.read();
        return good({ ...f, risks: riskLines(f) });
      }
      if (args["kind"] !== "csp") return bad("invalid", "kind is csp, routes, footing, or absent for the machine");
      if (!user.isAdmin) return bad("forbidden", "only an admin of this forge reads its CSP violations");
      const now = new Date();
      return good({ hours: 24, since: windowStart(24, now).toISOString(), directives: violationsByDirective(ctx.app.db, 24, now), scriptErrors: scriptErrorCounts(ctx.app.db, 24, now) });
@@ -50,6 +50,8 @@
export interface PerfRecord {
  /** When `pattern` was first seen in this process, undefined if never. */
  first(pattern: string): number | undefined;
  patterns(): string[];
  /** Forget every route's samples (not the minute ledger), as `loopWatch.clear` does its stalls: for a test that reads the record whole. */
  clear(): void;
  /** Who used the answering time, by the minute: exact however the samples are thinned. */
  readonly ledger: TrafficLedger;
}
@@ -100,6 +102,10 @@
export function createPerfRecord(valid: (key: string) => boolean = validPattern)
    traffic: (pattern, since) => (real.get(pattern) ?? []).filter((s) => s.at >= since),
    first: (pattern) => firsts.get(pattern),
    patterns: () => [...real.keys()],
    clear() {
      real.clear();
      firsts.clear();
    },
  };
}
@@ -0,0 +1,524 @@
/**
 * What the forge stands on (#11685): every piece the forge needs to answer a
 * request, sign a person in, build, release and back up, where each was
 * found running, and what happens to the forge if that machine dies.
 *
 * `FOOTING` is the declared list, from the steward's read-only inventory of
 * both machines on 2026-11-12, the day a full disk on pg-grand (DigitalOcean)
 * took the forge off the internet for 24 minutes while the forge itself ran on
 * the vxrail, and the owner decided to drop pg-grand. `docs/OPERATING.md` carries the same list as
 * a table (`footingTable` writes it; `test/footing.test.ts` holds the doc to
 * it), so the two cannot drift.
 *
 * `probeFooting` reads the live cluster over the kube API with the forge's
 * own service account: the forge's node (its own pod's `spec.nodeName`), the
 * control-plane nodes, and the node of each piece's pods. A piece the cluster
 * cannot show (a systemd unit on a host, a CronJob with no pod now, a
 * namespace the service account may not list) keeps its declared machine and
 * says so. Any required piece not on the forge's own machine is a risk line.
 * Only namespaced pod lists are asked for, never the whole cluster's.
 *
 * `footing` holds the last reading for 60 s in process memory (named in
 * AGENTS.md), so /system, its terminal screen (`src/tui/screens/footing.ts`)
 * and the MCP answer share one read and a page never waits on the cluster for
 * more than `WAIT_MS`; a probe that throws is not retried for 60 s.
 *
 * The descriptions carry no figure that goes stale (disk fill, sizes, counts):
 * those were true on one day and are not re-read here.
 */
import { serviceAccount, type Credentials } from "../cluster/jobs.ts";
export type Need = "answer" | "sign-in" | "build" | "release" | "backup";
/** The machines in the forge's path, by the name the steward's inventory gives them. */
export type Machine = "pg-grand" | "vxrail";
/** The machine the forge itself runs on, as declared. */
export const FORGE_MACHINE: Machine = "vxrail";
/** What each machine is, from the steward's read-only inventory of 2026-11-12. */
export const MACHINES: Record<Machine, string> = {
  "pg-grand": "DigitalOcean. The k3s server, a single control plane with no HA. The owner decided on 2026-11-12 to drop it.",
  vxrail: "A k3s agent joined to pg-grand. The forge runs here.",
};
export type Find =
  /** Pods of one workload (Deployment, StatefulSet, DaemonSet) in a namespace, or every pod (`*`); namespace `""` is the forge's own. */
  | { readonly kind: "pods"; readonly namespace: string; readonly workload: string }
  /** The forge's own pod. */
  | { readonly kind: "self" }
  /** The Kubernetes API server: the nodes labelled control-plane. */
  | { readonly kind: "control-plane" }
  /** Not in the cluster: a systemd unit on a host; the cluster cannot show it, so the declared machine stands. */
  | { readonly kind: "host"; readonly unit: string }
  /** A pod whose namespace is not recorded: not looked for, since that would list every pod in the cluster. */
  | { readonly kind: "unlocated" };
export interface Piece {
  readonly id: string;
  readonly need: Need;
  readonly what: string;
  readonly find: Find;
  /** Where it ran when the list was written. */
  readonly machine: Machine;
  /** What happens to the forge if that machine dies. */
  readonly ifDies: string;
  /** The issues that remove the dependency; none for a piece on the forge's own machine. */
  readonly issues: readonly number[];
  /** With pg-grand switched off, the step at which this piece fails the forge (1 first). */
  readonly breaks?: number;
}
export const NEEDS: readonly Need[] = ["answer", "sign-in", "build", "release", "backup"];
/** The steps in which the forge breaks if pg-grand is switched off now, in order. */
export const BREAK_STEPS: readonly string[] = [
  "Traefik: nothing public reaches the forge",
  "plat-auth: no one signs in",
  "the signup gate: no one signs up",
  "image pulls and releases",
  "the control plane: nothing reschedules, Services stop updating",
  "local backups",
  "the host timers",
];
const SAME_DEATH = "It goes with the forge, which runs beside it.";
export const FOOTING: readonly Piece[] = [
  {
    id: "forge",
    need: "answer",
    what: "The forge's pod (git.open-platform.sh/trevato-apps/forge:<version>) and its data, PVC forge-data-vxrail (60 Gi)",
    find: { kind: "self" },
    machine: "vxrail",
    ifDies: "The forge is down, and its data is on that disk.",
    issues: [],
  },
  {
    id: "tunnel",
    need: "answer",
    what: "Cloudflare tunnel carrying forge.open-platform.sh (standby/cloudflared), moved to the vxrail by hand on 2026-11-12; its last-applied manifest still says 0 replicas, so a re-render moves it back",
    find: { kind: "pods", namespace: "standby", workload: "cloudflared" },
    machine: "vxrail",
    ifDies: SAME_DEATH,
    issues: [8948, 11686],
  },
  {
    id: "tunnel-home",
    need: "answer",
    what: "The vxrail's own tunnel (standby/cloudflared-forge-home, 2 replicas): it too sends *.open-platform.sh to Traefik; only forge-home-test.open-platform.sh and arm--forge-trevato-apps.open-platform.sh go straight to the forge",
    find: { kind: "pods", namespace: "standby", workload: "cloudflared-forge-home" },
    machine: "vxrail",
    ifDies: SAME_DEATH,
    issues: [],
  },
  {
    id: "traefik",
    need: "answer",
    what: "Traefik (kube-system/traefik): every public request goes tunnel → Traefik → the forge's Service",
    find: { kind: "pods", namespace: "kube-system", workload: "traefik" },
    machine: "pg-grand",
    ifDies: "The forge is off the internet; it keeps running inside.",
    issues: [11686, 8946],
    breaks: 1,
  },
  {
    id: "hosts",
    need: "answer",
    what: "Child and app hosts, Traefik IngressRoutes forge-children (<name>--forge.open-platform.sh), forge-apps (*--forge-trevato-apps.open-platform.sh) and forge-old-host, and game.cheffryjames.com and www.cheffryjames.com in the forge's namespace",
    find: { kind: "pods", namespace: "kube-system", workload: "traefik" },
    machine: "pg-grand",
    ifDies: "No child forge, app or custom domain answers.",
    issues: [11686, 8946],
    breaks: 1,
  },
  {
    id: "forge-ssh",
    need: "answer",
    what: "forge-ssh, a host socket proxy (:2222 → the forge's ssh service)",
    find: { kind: "host", unit: "forge-ssh" },
    machine: "pg-grand",
    ifDies: "git over ssh and the terminal stop answering.",
    issues: [11686],
    breaks: 7,
  },
  {
    id: "control-plane",
    need: "answer",
    what: "The k3s API server (the vxrail is an agent joined to it)",
    find: { kind: "control-plane" },
    machine: "pg-grand",
    ifDies: "Running pods keep running, but nothing reschedules, Services stop updating, and no release rolls or Job runs.",
    issues: [11686],
    breaks: 5,
  },
  {
    id: "plat-auth",
    need: "sign-in",
    what: "Sign-in edge: a forwardAuth middleware on the forge's Ingress, answered by mcp/plat-auth",
    find: { kind: "pods", namespace: "mcp", workload: "plat-auth" },
    machine: "pg-grand",
    ifDies: "Every request through the edge answers 500, /healthz too.",
    issues: [11686],
    breaks: 2,
  },
  {
    id: "signup-gate",
    need: "sign-in",
    what: "forge-signup-gate, the pod /signup is routed to",
    // An Ingress routes only to a Service in its own namespace: the forge's.
    find: { kind: "pods", namespace: "", workload: "forge-signup-gate" },
    machine: "pg-grand",
    ifDies: "No one can sign up.",
    issues: [11686],
    breaks: 3,
  },
  {
    id: "forgejo",
    need: "build",
    what: "Forgejo (git.open-platform.sh) and its Actions, which build the release image",
    find: { kind: "pods", namespace: "forgejo", workload: "*" },
    machine: "pg-grand",
    ifDies: "No release image is built.",
    issues: [11686, 8941],
    breaks: 4,
  },
  {
    id: "cnpg",
    need: "build",
    what: "Forgejo's database, a single instance under CloudNativePG (cnpg-system)",
    find: { kind: "pods", namespace: "cnpg-system", workload: "*" },
    machine: "pg-grand",
    ifDies: "Forgejo stops, and with it builds.",
    issues: [11686, 8941],
    breaks: 4,
  },
  {
    id: "runner",
    need: "build",
    what: "forgejo-runner-vxrail, a Forgejo Actions runner",
    find: { kind: "unlocated" },
    machine: "vxrail",
    ifDies: SAME_DEATH,
    issues: [],
  },
  {
    id: "registry",
    need: "release",
    what: "Forgejo's registry (git.open-platform.sh), which the forge's image is pulled from",
    find: { kind: "pods", namespace: "forgejo", workload: "*" },
    machine: "pg-grand",
    ifDies: "A pod that restarts on a node without the image cached cannot start; nothing new is released.",
    issues: [11686, 8941, 8942],
    breaks: 4,
  },
  {
    id: "registry-node",
    need: "release",
    what: "registry-node, a registry on the vxrail",
    find: { kind: "unlocated" },
    machine: "vxrail",
    ifDies: SAME_DEATH,
    issues: [],
  },
  {
    id: "release-tick",
    need: "release",
    what: "forge-release-tick, a host timer every 5 min",
    find: { kind: "host", unit: "forge-release-tick" },
    machine: "pg-grand",
    ifDies: "Nothing is released; production keeps the release it has.",
    issues: [11686, 8942],
    breaks: 7,
  },
  {
    id: "standby-reconcile",
    need: "release",
    what: "standby-reconcile, a host timer every 2 min that applies the forge's manifests from Forgejo; Flux's Kustomization for the forge is suspended",
    find: { kind: "host", unit: "standby-reconcile" },
    machine: "pg-grand",
    ifDies: "A changed manifest is not applied; the running pod is untouched.",
    issues: [8948, 11686],
    breaks: 7,
  },
  {
    id: "host-timers",
    need: "release",
    what: "Host timers forge-watchdog, forge-github-mirror and forge-prune-images",
    find: { kind: "host", unit: "forge-watchdog" },
    machine: "pg-grand",
    ifDies: "The watchdog, the GitHub mirror and image pruning stop.",
    issues: [11686],
    breaks: 7,
  },
  {
    id: "backup-lanes",
    need: "backup",
    what: "The backup namespace's CronJobs (FORGE_BACKUP_LANES)",
    find: { kind: "pods", namespace: "backup", workload: "*" },
    machine: "pg-grand",
    ifDies: "No new backup is taken.",
    issues: [11686],
    breaks: 6,
  },
  {
    id: "minio",
    need: "backup",
    what: "MinIO, where the backup lanes store the forge's backups",
    find: { kind: "pods", namespace: "minio", workload: "*" },
    machine: "pg-grand",
    ifDies: "No new backup is stored, and those it holds go with the machine.",
    issues: [11686],
    breaks: 6,
  },
];
/** On pg-grand, not in the forge's path: what dropping the machine takes besides. */
export const ALSO_ON_PG_GRAND =
  "Forgejo and its database, the platform's mcp/plat-auth and www, the CloudNativePG operator with the tenant apps' databases, their local-path volumes pinned to that node, and MinIO.";
/** How each piece was placed: seen on nodes, or the declared machine kept and why. */
export type Seen =
  | { readonly by: "cluster"; readonly nodes: readonly string[] }
  | { readonly by: "declared"; readonly why: string };
export interface FootingRow extends Piece {
  readonly seen: Seen;
  /** True when the piece is not on the forge's own machine (or cannot be shown to be). */
  readonly risk: boolean;
}
export interface Footing {
  /** The forge's own node; null outside a cluster or when its pod cannot be read. */
  readonly forgeNode: string | null;
  /** Why the cluster was not read, when it was not. */
  readonly unread: string | null;
  readonly rows: readonly FootingRow[];
  readonly at: string;
}
/** A pod belongs to a workload: `<w>-<rs>-<id>` (Deployment), `<w>-<n>` (StatefulSet), `<w>-<id>` (DaemonSet). */
export function ownedBy(pod: string, workload: string): boolean {
  if (workload === "*") return true;
  if (!pod.startsWith(`${workload}-`)) return false;
  const rest = pod.slice(workload.length + 1);
  return /^[a-z0-9]{5,10}-[a-z0-9]{5}$/.test(rest) || /^\d+$/.test(rest) || /^[a-z0-9]{5}$/.test(rest);
}
/** Rows for the declared list alone: every piece placed on its declared machine. */
export function declaredFooting(why: string, at = new Date()): Footing {
  return {
    forgeNode: null,
    unread: why,
    at: at.toISOString(),
    rows: FOOTING.map((p) => ({ ...p, seen: { by: "declared", why }, risk: p.machine !== FORGE_MACHINE })),
  };
}
type KubeFetch = (url: string, init: RequestInit & { tls?: { ca: string } }) => Promise<Response>;
export interface ProbeOptions {
  fetch?: KubeFetch;
  /** The forge's service account, or how to read it (it throws outside a cluster). */
  credentials?: Credentials | (() => Credentials);
  /** The forge's own pod name; the kubelet sets HOSTNAME to it. */
  podName?: string;
  now?: () => Date;
}
type PodList = { items?: { metadata?: { name?: string }; spec?: { nodeName?: string }; status?: { phase?: string } }[] };
type NodeList = { items?: { metadata?: { name?: string; labels?: Record<string, string> } }[] };
const CONTROL_LABELS = ["node-role.kubernetes.io/control-plane", "node-role.kubernetes.io/master"];
/** Read the live cluster; never throws. Outside a cluster, the declared list with why. */
export async function probeFooting(opts: ProbeOptions = {}): Promise<Footing> {
  const now = opts.now ?? (() => new Date());
  let cred: Credentials;
  try {
    const c = opts.credentials ?? serviceAccount;
    cred = typeof c === "function" ? c() : c;
  } catch (e) {
    return declaredFooting(`not read: ${e instanceof Error ? e.message : String(e)}`, now());
  }
  const doFetch: KubeFetch = opts.fetch ?? ((u, i) => fetch(u, i as RequestInit));
  const get = async <T>(path: string): Promise<{ ok: true; value: T } | { ok: false; status: number | string }> => {
    try {
      const res = await doFetch(`${cred.server}${path}`, {
        method: "GET",
        headers: { authorization: `Bearer ${cred.token}`, accept: "application/json" },
        signal: AbortSignal.timeout(3000),
        ...(cred.ca ? { tls: { ca: cred.ca } } : {}),
      });
      if (!res.ok) return { ok: false, status: res.status };
      return { ok: true, value: (await res.json()) as T };
    } catch (e) {
      return { ok: false, status: e instanceof Error ? e.message : "unreachable" };
    }
  };
  const refused = (s: number | string, what: string): string =>
    s === 403 ? `the forge's service account may not read ${what}` : `${what} not read (${s})`;
  const podName = opts.podName ?? process.env["HOSTNAME"] ?? "";
  const self = podName
    ? await get<{ spec?: { nodeName?: string } }>(`/api/v1/namespaces/${encodeURIComponent(cred.namespace)}/pods/${encodeURIComponent(podName)}`)
    : null;
  const forgeNode = self?.ok ? (self.value.spec?.nodeName ?? null) : null;
  const nodes = await get<NodeList>("/api/v1/nodes");
  const nsOf = (ns: string): string => ns || cred.namespace;
  const namespaces = [...new Set(FOOTING.flatMap((p) => (p.find.kind === "pods" ? [nsOf(p.find.namespace)] : [])))];
  const pods = new Map(
    await Promise.all(namespaces.map(async (ns) => [ns, await get<PodList>(`/api/v1/namespaces/${encodeURIComponent(ns)}/pods`)] as const)),
  );
  const controlNodes = nodes.ok
    ? (nodes.value.items ?? []).filter((n) => CONTROL_LABELS.some((l) => n.metadata?.labels?.[l] !== undefined)).map((n) => n.metadata?.name ?? "").filter(Boolean)
    : null;
  const place = (p: Piece): Seen => {
    const f = p.find;
    if (f.kind === "self") {
      if (forgeNode !== null) return { by: "cluster", nodes: [forgeNode] };
      return { by: "declared", why: self === null ? "the forge's pod name is unknown" : self.ok ? "its pod has no node yet" : refused(self.status, "its own pod") };
    }
    if (f.kind === "control-plane") {
      if (!nodes.ok) return { by: "declared", why: refused(nodes.status, "nodes") };
      return controlNodes!.length ? { by: "cluster", nodes: controlNodes! } : { by: "declared", why: "no node is labelled control-plane" };
    }
    // Nothing in the cluster sees a systemd unit: the declared machine stands, said as such.
    if (f.kind === "host") return { by: "declared", why: `${f.unit} is a systemd unit on a host; the cluster cannot show it` };
    if (f.kind === "unlocated") return { by: "declared", why: "its namespace is not recorded, and the forge does not list every pod in the cluster" };
    const where = nsOf(f.namespace);
    const list = pods.get(where)!;
    if (!list.ok) return { by: "declared", why: refused(list.status, `pods in ${where}`) };
    const on = (list.value.items ?? [])
      .filter((i) => ownedBy(i.metadata?.name ?? "", f.workload) && i.status?.phase !== "Succeeded" && i.status?.phase !== "Failed")
      .map((i) => i.spec?.nodeName ?? "")
      .filter(Boolean);
    return on.length ? { by: "cluster", nodes: [...new Set(on)].sort() } : { by: "declared", why: `no pod running in ${where} now` };
  };
  const rows = FOOTING.map((p): FootingRow => {
    const seen = place(p);
    const risk =
      seen.by === "cluster" && forgeNode !== null
        ? seen.nodes.some((n) => n !== forgeNode)
        : p.machine !== FORGE_MACHINE;
    return { ...p, seen, risk };
  });
  const unread =
    forgeNode !== null ? null
    : self === null ? "the forge's pod name is unknown (HOSTNAME unset)"
    : self.ok ? "the forge's own pod has no node yet"
    : refused(self.status, "its own pod");
  return { forgeNode, unread, rows, at: now().toISOString() };
}
/** Where a row was placed, in words. */
export function whereWords(r: FootingRow): string {
  return r.seen.by === "cluster" ? r.seen.nodes.join(", ") : `${r.machine} (declared: ${r.seen.why})`;
}
/** The rows off the forge's machine, in the order pg-grand switched off would break the forge; ties keep the list's order. */
export function offMachine(f: Footing): FootingRow[] {
  return f.rows
    .filter((r) => r.risk)
    .map((r, i) => ({ r, i }))
    .sort((a, b) => (a.r.breaks ?? 99) - (b.r.breaks ?? 99) || a.i - b.i)
    .map(({ r }) => r);
}
/** The risk lines: one per required piece off the forge's machine, in the order they would break it. */
export function riskLines(f: Footing): string[] {
  return offMachine(f).map((r) => `${r.need}: ${r.what} on ${whereWords(r)}. ${r.ifDies}`);
}
const FRESH_MS = 60_000;
/** After a probe that throws, the next is not started for this long, so a broken cluster costs one try a minute. */
const BACKOFF_MS = 60_000;
const WAIT_MS = 800;
/** The last reading shared by /system, its terminal screen and the MCP answer. */
export function footingCache(probe: () => Promise<Footing> = () => probeFooting(), clock: () => number = Date.now) {
  let last: { at: number; value: Footing } | null = null;
  let pending: Promise<Footing> | null = null;
  let failedAt: number | null = null;
  const stale = (): boolean => !last || clock() - last.at > FRESH_MS;
  const backingOff = (): boolean => failedAt !== null && clock() - failedAt < BACKOFF_MS;
  const refresh = (): Promise<Footing> => {
    pending ??= probe()
      .then((value) => {
        last = { at: clock(), value };
        failedAt = null;
        return value;
      })
      .catch((e: unknown) => {
        failedAt = clock();
        throw e;
      })
      .finally(() => {
        pending = null;
      });
    return pending;
  };
  return {
    /** The last reading, refreshed in the background when older than 60 s; null before the first. */
    peek(): Footing | null {
      if (stale() && !backingOff()) void refresh().catch(() => {});
      return last?.value ?? null;
    },
    /** A fresh-enough reading, waiting at most `waitMs` for the cluster; the declared list when it is slower. */
    async read(waitMs = WAIT_MS): Promise<Footing> {
      if (!stale()) return last!.value;
      if (backingOff()) return last?.value ?? declaredFooting("the cluster did not answer; it is asked again within a minute");
      let deadline: ReturnType<typeof setTimeout> | undefined;
      const timer = new Promise<null>((r) => {
        deadline = setTimeout(() => r(null), waitMs);
      });
      try {
        const got = await Promise.race([refresh().catch(() => null), timer]);
        return got ?? last?.value ?? declaredFooting("the cluster did not answer in time; reload in a moment");
      } finally {
        clearTimeout(deadline);
      }
    },
  };
}
export const footing = footingCache();
const NEED_WORDS: Record<Need, string> = {
  answer: "(a) answer a request",
  "sign-in": "(b) sign a person in",
  build: "(c) build",
  release: "(d) release",
  backup: "(e) back up",
};
/** Markdown text: a pipe would end a cell, and `<name>` would read as a tag. */
const cell = (s: string): string => s.replaceAll("|", "\\|").replaceAll("<", "&lt;").replaceAll(">", "&gt;");
/** What `docs/OPERATING.md` carries between its markers, written from `MACHINES`, `FOOTING` and `BREAK_STEPS`. */
export function footingTable(): string {
  const machines = (Object.keys(MACHINES) as Machine[]).map((m) => `- **${m}**: ${cell(MACHINES[m])}`);
  const head = "| Needed to | Piece | Runs on | If that machine dies | Breaks | Removed by |\n|---|---|---|---|---|---|";
  const lines = NEEDS.flatMap((need) =>
    FOOTING.filter((p) => p.need === need).map(
      (p) =>
        `| ${NEED_WORDS[need]} | ${cell(p.what)} | ${p.machine} | ${cell(p.ifDies)} | ${p.breaks ?? "—"} | ${p.issues.map((n) => `#${n}`).join(", ") || "—"} |`,
    ),
  );
  const steps = BREAK_STEPS.map((s, i) => `${i + 1}. ${cell(s)}`);
  return [
    ...machines,
    "",
    head,
    ...lines,
    "",
    "If pg-grand were switched off now, the forge breaks in this order (the Breaks column):",
    "",
    ...steps,
    "",
    `Also on pg-grand, not in the forge's path: ${cell(ALSO_ON_PG_GRAND)}`,
  ].join("\n");
}
@@ -57,6 +57,7 @@
import { entry as plans } from "./plans.ts";
import { entry as economics } from "./economics.ts";
import { entry as money } from "./money.ts";
import { entry as loopStalls } from "./loopstalls.ts";
import { entry as footing } from "./footing.ts";
import { entry as waiting } from "./waiting.ts";
import { entry as slowPagesEntry } from "./slowpages.ts";
import { entry as maintenance } from "./maintenance.ts";
@@ -122,6 +123,7 @@
export const ENTRIES: readonly PaletteEntry[] = [
  homePins,
  system,
  loopStalls,
  footing,
  waiting,
  slowPagesEntry,
  telemetry,
@@ -0,0 +1,77 @@
/**
 * What the forge stands on, in the terminal (#11685): the rows /system shows
 * under the same heading. First where the forge runs (or why that was not
 * read), then each piece on another machine, in the order switching pg-grand
 * off would break the forge: its step, what it is needed for, the piece, the
 * machine it was found on, and what happens if that machine dies. Last, the
 * pieces on the forge's own machine, by name. The last reading
 * (`footing.peek`), so the screen never waits on the cluster; the declared
 * list while the first read runs. The gate is /system's: a forge admin who is
 * a person. Read-only.
 */
import { BREAK_STEPS, declaredFooting, footing, offMachine, whereWords, type Footing } from "../../system/footing.ts";
import type { Ctx, Screen } from "../app/app.ts";
import { binding } from "../app/keys.ts";
import type { Session } from "../session.ts";
import { style } from "../term/style.ts";
import { pad } from "../term/width.ts";
import { Scroll, drawLines, hitScroll } from "./common.ts";
import type { PaletteEntry } from "./entries.ts";
const mayRead = (session: Session): boolean => session.user?.isAdmin === true && session.user.kind !== "agent";
const current = (): Footing => footing.peek() ?? declaredFooting("the cluster is being read; reopen in a moment");
/** The screen's lines; empty for anyone the gate refuses. */
export function footingLines(session: Session, f: Footing = current()): string[] {
  if (!mayRead(session)) return [];
  const off = offMachine(f);
  const home = f.rows.filter((r) => !r.risk);
  const where =
    f.forgeNode === null ? `The forge's node was not read (${f.unread ?? "unknown"}); machines are as declared.` : `The forge runs on ${f.forgeNode}.`;
  const lines = [where, ""];
  if (off.length === 0) lines.push("Every piece runs on the forge's own machine.");
  else {
    lines.push(`${String(off.length)} of ${String(f.rows.length)} pieces run on another machine. With pg-grand off, they fail in this order:`);
    for (const r of off) {
      const step = r.breaks === undefined ? "-" : String(r.breaks);
      lines.push("", `${pad(step, 3, "right")}  ${pad(r.need, 8)}  ${r.what}`);
      lines.push(`${pad("", 15)}on ${whereWords(r)}`, `${pad("", 15)}if it dies: ${r.ifDies}`);
    }
    lines.push("", "The steps:", ...BREAK_STEPS.map((s, i) => `${pad(String(i + 1), 3, "right")}  ${s}`));
  }
  if (home.length > 0) lines.push("", `On the forge's own machine: ${home.map((r) => r.id).join(", ")}`);
  return lines;
}
export function footingScreen(session: Session, ctx: Ctx): Screen {
  const scroll = new Scroll();
  return {
    title: "What the forge stands on",
    keys: [binding(["j", "k"], "scroll"), binding(["esc", "q"], "back")],
    render(area, s) {
      const c = ctx.theme.c;
      const lines = footingLines(session);
      drawLines(s, area, c, scroll, lines.map((text) => [{ text, style: style(/^\s*\d+ {2}/.test(text) || text === "" ? c.fg : c.mutedFg, c.bg) }]));
      hitScroll(ctx, area, scroll, lines.length);
    },
    key(e) {
      if (scroll.key(e, footingLines(session).length)) return true;
      if (e.key === "esc" || e.key === "q") {
        ctx.pop();
        return true;
      }
      return false;
    },
  };
}
export const entry: PaletteEntry = {
  id: "footing",
  title: "What the forge stands on",
  hint: "each piece on another machine, in the order it would break the forge",
  after: ["stalls", "system"],
  offered: mayRead,
  run: (session, ctx) => ctx.push(footingScreen(session, ctx)),
};
@@ -47,6 +47,7 @@
import { KEPT_PAGE_BYTES, keptPageStats } from "./kept_page.ts";
import { COLUMN_BYTES, PARTS_BYTES, keptPartsStats, treeCacheStats } from "./tree_cache.ts";
import { featureHealth, type FeatureHealth } from "../../system/feature_health.ts";
import { egressReportedLine } from "../../system/egress_line.ts";
import { BREAK_STEPS, MACHINES, declaredFooting, footing, offMachine, whereWords, type Footing, type Machine } from "../../system/footing.ts";
import { badge, chartBox, disclosure, meter, pill, row, time } from "../ui.ts";
import { childDirOf, listChildren } from "../../actions/spawn.ts";
import { childrenUse, realSelf, selfUse, shareOf, sizeWords, useWords, type ChildUseRow, type SelfUse } from "../../spawn/usage.ts";
@@ -204,6 +205,38 @@
export function featuresSection(rows: readonly FeatureHealth[]): Html {
    <p class="small"><a href="/settings/features">Settings › Features</a></p></section>`;
}
/**
 * What the forge stands on (#11685): each piece it needs to answer, sign in,
 * build, release and back up, and where it was found. The summary gives the
 * one count; inside, each piece on another machine, in the order switching
 * pg-grand off would break the forge, one fact a line; then the pieces on the
 * forge's own machine, by name. `system/footing.ts`'s rows.
 */
export function footingSection(f: Footing): Html {
  const off = offMachine(f);
  const home = f.rows.filter((r) => !r.risk);
  // First sight (#11111): the heading and one count stay; the rows open behind one disclosure.
  const count = off.length === 0 ? "Every piece runs on the forge's own machine" : `${off.length} of ${f.rows.length} pieces run on another machine`;
  return html`<section class="settings-block" id="sys-footing" data-off="${String(off.length)}" data-forge-node="${f.forgeNode ?? ""}"><h2>What the forge stands on</h2>
    ${disclosure(count, html`<p class="small">${f.forgeNode === null ? html`The forge's node was not read (${f.unread ?? "unknown"}), so each piece shows the machine written in <code>docs/OPERATING.md</code>.` : html`The forge runs on <span class="mono">${f.forgeNode}</span>.`}
      ${off.length === 0 ? null : "If that machine stops, each piece below stops with it; they are listed in the order the forge would lose them."}</p>
    ${off.length === 0 ? null : html`<ul class="list" aria-label="Pieces on another machine">${off.map((r) =>
      row({
        title: r.what,
        facts: [
          { label: "Needed to", value: r.need },
          { label: "Runs on", value: html`<span class="mono">${whereWords(r)}</span>` },
          { label: "If it stops", value: r.ifDies },
          ...(r.breaks ? [{ label: "Order", value: `${r.breaks} of ${BREAK_STEPS.length}: ${BREAK_STEPS[r.breaks - 1]}` }] : []),
          ...(r.issues.length ? [{ label: "Fixed by", value: r.issues.map((n) => `#${n}`).join(", ") }] : []),
        ],
        data: { piece: r.id, off: "true", seen: r.seen.by, ...(r.breaks ? { breaks: String(r.breaks) } : {}) },
      }),
    )}</ul>`}
    ${home.length === 0 ? null : html`<p class="small muted">On the forge's own machine: ${home.map((r, i) => html`${i ? ", " : ""}<span data-piece="${r.id}" data-off="false">${r.what}</span>`)}.</p>`}
    ${(Object.keys(MACHINES) as Machine[]).map((m) => html`<p class="small muted" data-machine="${m}"><span class="mono">${m}</span>: ${MACHINES[m]}</p>`)}`, { id: "sys-footing-more" })}</section>`;
}
/** The last hour's stalls by route, most milliseconds first; the rows are `stallHour`'s. */
export function stallSection(rows: readonly RouteStall[], worst: Html = html``): Html {
  return html`<section class="settings-block"><h2>Main-thread stalls, last hour</h2>${
@@ -370,6 +403,8 @@
export async function systemPage(ctx: Ctx): Promise<Response> {
  const stalls = timePhase(ctx.request, "stalls", () => stallHour(loopWatch.tally(performance.now())));
  const nowMono = performance.now();
  const worst = stallBlame.worst(nowMono, HOUR_MS, WORST_STALLS);
  // Never waits on the cluster: the last reading, or the declared rows while the first read runs behind.
  const standsOn = footing.peek() ?? declaredFooting("the cluster is being read; reload in a moment");
  const rendering = performance.now();
  const r = regions(v);
  const at = (id: string): Html => raw(r[id]!);
@@ -403,6 +438,7 @@
export async function systemPage(ctx: Ctx): Promise<Response> {
      <div class="js-only small muted">Sort by <button class="btn" data-size="sm" type="button" data-sort="cpu" aria-pressed="true"><kbd>c</kbd> CPU</button>
        <button class="btn" data-size="sm" type="button" data-sort="mem" aria-pressed="false"><kbd>m</kbd> memory</button></div>
      <div id="sys-procs" data-live-hold>${at("sys-procs")}</div></section>
    ${footingSection(standsOn)}
    ${featuresSection(timePhase(ctx.request, "features", () => featureHealth(ctx.app, reader)))}
    ${lagSection(lagMinutes.readings())}
    <div id="sys-stalls">${stallSection(stalls, worstStallsTable(worst, nowMono, Date.now()))}</div>
@@ -0,0 +1,404 @@
/**
 * What the forge stands on (#11685): the table in docs/OPERATING.md is the
 * declared list exactly; the probe places each piece on the node the kube API
 * shows it on and marks every piece off the forge's node a risk; what the
 * cluster cannot show keeps its declared machine and says why; /system shows
 * it under one heading, and the MCP `system` tool answers it to an admin only.
 */
import { afterAll, beforeAll, describe, expect, test } from "bun:test";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createApp, type App } from "../src/app.ts";
import { createToken } from "../src/core/tokens.ts";
import { createUser } from "../src/core/users.ts";
import { serve, type Servers } from "../src/http/server.ts";
import {
  BREAK_STEPS,
  FOOTING,
  declaredFooting,
  footingCache,
  footingTable,
  ownedBy,
  probeFooting,
  riskLines,
  type Footing,
} from "../src/system/footing.ts";
import { footingSection } from "../src/web/pages/system.ts";
import { getUserByName, type User } from "../src/core/users.ts";
import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
import { footingLines, footingScreen } from "../src/tui/screens/footing.ts";
import { headlessTerminal } from "../src/tui/term/terminal.ts";
import { settleApp } from "./helpers/tui.ts";
import { quietLogs } from "./helpers/globals.ts";
quietLogs();
const cred = { server: "https://kube.test", token: "t", namespace: "forge" };
const pod = (name: string, node: string, phase = "Running") => ({ metadata: { name }, spec: { nodeName: node }, status: { phase } });
/**
 * A fake kube API as the steward found it on 2026-11-12: the forge and both tunnels on vxrail-1,
 * the rest on pg-grand, minio forbidden; Traefik moved to vxrail-1.
 */
function fakeKube(over: Record<string, unknown> = {}) {
  const answers: Record<string, unknown> = {
    "/api/v1/namespaces/forge/pods/forge-7d9f8c6b5-abcde": { spec: { nodeName: "vxrail-1" } },
    "/api/v1/nodes": {
      items: [
        { metadata: { name: "pg-grand", labels: { "node-role.kubernetes.io/control-plane": "true" } } },
        { metadata: { name: "vxrail-1", labels: { "plat.sh/site": "vxrail" } } },
      ],
    },
    "/api/v1/namespaces/standby/pods": {
      items: [pod("cloudflared-6c7b9d8f4-x2x9z", "vxrail-1"), pod("cloudflared-forge-home-5f6d7c8b9-q1w2e", "vxrail-1"), pod("cloudflared-forge-home-5f6d7c8b9-r3t4y", "vxrail-1")],
    },
    "/api/v1/namespaces/forge/pods": { items: [pod("forge-7d9f8c6b5-abcde", "vxrail-1"), pod("forge-signup-gate-6d5c4b3a2-sgate", "pg-grand")] },
    "/api/v1/namespaces/kube-system/pods": { items: [pod("traefik-5b8d9c7f6-zzzzz", "vxrail-1"), pod("coredns-1234567890-aaaaa", "pg-grand")] },
    "/api/v1/namespaces/mcp/pods": { items: [pod("plat-auth-7f8e9d6c5-abcde", "pg-grand")] },
    "/api/v1/namespaces/forgejo/pods": { items: [pod("forgejo-0", "pg-grand"), pod("forgejo-db-1", "pg-grand")] },
    "/api/v1/namespaces/flux-system/pods": { items: [pod("source-controller-6b5d4c3f2-qwert", "pg-grand")] },
    "/api/v1/namespaces/backup/pods": { items: [pod("lane-forge-29000000-abcde", "pg-grand", "Succeeded")] },
    "/api/v1/namespaces/minio/pods": 403,
    "/api/v1/namespaces/cnpg-system/pods": { items: [pod("cnpg-controller-manager-7d8f9b6c5-mnbvc", "pg-grand")] },
    ...over,
  };
  const seen: string[] = [];
  const fetch = async (url: string, init: RequestInit) => {
    const path = url.slice(cred.server.length);
    seen.push(path);
    expect((init.headers as Record<string, string>)["authorization"]).toBe("Bearer t");
    expect(init.method).toBe("GET");
    const a = answers[path];
    if (a === undefined) return new Response("{}", { status: 404 });
    if (typeof a === "number") return new Response("{}", { status: a });
    return Response.json(a);
  };
  return { fetch, seen };
}
const byId = (f: Footing, id: string) => f.rows.find((r) => r.id === id)!;
describe("the table in docs/OPERATING.md", () => {
  test("is footingTable() exactly, between the markers", async () => {
    const doc = await readFile(join(import.meta.dir, "..", "docs", "OPERATING.md"), "utf8");
    const m = doc.match(/<!-- footing:start[^>]*-->\n([\s\S]*?)\n<!-- footing:end -->/);
    expect(m).not.toBeNull();
    expect(m![1]).toBe(footingTable());
    expect(doc).toContain("## What the forge stands on");
  });
  test("every need (a) to (e) has a piece; every pg-grand piece names an issue that removes it and the step it breaks at", () => {
    for (const need of ["answer", "sign-in", "build", "release", "backup"]) expect(FOOTING.some((p) => p.need === need)).toBe(true);
    for (const p of FOOTING.filter((p) => p.machine === "pg-grand")) {
      expect(p.issues.length).toBeGreaterThan(0);
      expect(p.breaks! >= 1 && p.breaks! <= BREAK_STEPS.length).toBe(true);
    }
    for (const p of FOOTING.filter((p) => p.machine === "vxrail")) expect(p.breaks).toBeUndefined();
    // Every step of the break order has a piece behind it.
    for (let i = 1; i <= BREAK_STEPS.length; i++) expect(FOOTING.some((p) => p.breaks === i)).toBe(true);
    expect(new Set(FOOTING.map((p) => p.id)).size).toBe(FOOTING.length);
  });
  test("the doc names both machines and escapes what markdown would read as a tag", async () => {
    const t = footingTable();
    expect(t).toContain("- **pg-grand**: DigitalOcean.");
    expect(t).toContain("- **vxrail**: ");
    expect(t).toContain("forge:&lt;version&gt;");
    expect(t).not.toContain("<version>");
    expect(t).toContain("1. Traefik: nothing public reaches the forge");
    // No figure that goes stale is written as if current: no disk fill, sizes or counts of the day.
    expect(t).not.toMatch(/% used|\b\d+(\.\d+)? (GB|TB)\b|\b\d+ (single-instance )?databases\b|\b\d+ (local-path )?volumes\b/);
  });
});
describe("ownedBy", () => {
  test("a Deployment's, StatefulSet's and DaemonSet's pods; not a longer workload's", () => {
    expect(ownedBy("cloudflared-6c7b9d8f4-x2x9z", "cloudflared")).toBe(true);
    expect(ownedBy("forgejo-0", "forgejo")).toBe(true);
    expect(ownedBy("traefik-abcde", "traefik")).toBe(true);
    expect(ownedBy("cloudflared-forge-home-5f6d7c8b9-q1w2e", "cloudflared")).toBe(false);
    expect(ownedBy("plat-auth-x", "plat-auth")).toBe(false);
    expect(ownedBy("anything", "*")).toBe(true);
  });
});
describe("probeFooting", () => {
  test("places each piece on the node it runs on; off the forge's node is a risk", async () => {
    const kube = fakeKube();
    const f = await probeFooting({ fetch: kube.fetch, credentials: cred, podName: "forge-7d9f8c6b5-abcde", now: () => new Date("2026-11-12T12:00:00Z") });
    expect(f.forgeNode).toBe("vxrail-1");
    expect(f.unread).toBeNull();
    expect(f.at).toBe("2026-11-12T12:00:00.000Z");
    // The forge's own pod, and the tunnel moved to the vxrail by hand: on the forge's node.
    expect(byId(f, "forge")).toMatchObject({ seen: { by: "cluster", nodes: ["vxrail-1"] }, risk: false });
    expect(byId(f, "tunnel")).toMatchObject({ seen: { by: "cluster", nodes: ["vxrail-1"] }, risk: false });
    expect(byId(f, "tunnel-home")).toMatchObject({ seen: { by: "cluster", nodes: ["vxrail-1"] }, risk: false });
    // The signup gate is looked for in the forge's own namespace (an Ingress routes only within its own).
    expect(byId(f, "signup-gate")).toMatchObject({ seen: { by: "cluster", nodes: ["pg-grand"] }, risk: true });
    // A pod whose namespace is not recorded is not looked for: no list of every pod in the cluster.
    expect(byId(f, "runner")).toMatchObject({ seen: { by: "declared", why: "its namespace is not recorded, and the forge does not list every pod in the cluster" }, risk: false });
    expect(byId(f, "registry-node")).toMatchObject({ seen: { by: "declared" }, risk: false });
    expect(kube.seen).not.toContain("/api/v1/pods");
    expect(kube.seen.every((p) => p === "/api/v1/nodes" || p.startsWith("/api/v1/namespaces/"))).toBe(true);
    // Traefik moved to the forge's node: no longer a risk, whatever the declared list said.
    expect(byId(f, "traefik")).toMatchObject({ seen: { by: "cluster", nodes: ["vxrail-1"] }, risk: false });
    expect(byId(f, "hosts").risk).toBe(false);
    expect(byId(f, "control-plane")).toMatchObject({ seen: { by: "cluster", nodes: ["pg-grand"] }, risk: true });
    // Nothing in the cluster sees a systemd unit: declared, said why, judged by the declared machine.
    expect(byId(f, "release-tick")).toMatchObject({ seen: { by: "declared", why: "forge-release-tick is a systemd unit on a host; the cluster cannot show it" }, risk: true });
    expect(byId(f, "standby-reconcile")).toMatchObject({ seen: { by: "declared" }, risk: true });
    for (const id of ["forge-ssh", "release-tick", "standby-reconcile", "host-timers"]) expect(byId(f, id).seen.by).toBe("declared");
    expect(byId(f, "plat-auth").risk).toBe(true);
    // Forbidden: the declared machine is kept and the reason named.
    expect(byId(f, "minio").seen).toEqual({ by: "declared", why: "the forge's service account may not read pods in minio" });
    expect(byId(f, "minio").risk).toBe(true);
    // A CronJob between runs: only a finished pod, so declared.
    expect(byId(f, "backup-lanes").seen).toEqual({ by: "declared", why: "no pod running in backup now" });
    // Each namespace is listed once although two pieces live in forgejo.
    expect(kube.seen.filter((p) => p === "/api/v1/namespaces/forgejo/pods").length).toBe(1);
    const off = ["forge-ssh", "control-plane", "plat-auth", "signup-gate", "forgejo", "cnpg", "registry", "release-tick", "standby-reconcile", "host-timers", "backup-lanes", "minio"];
    expect(f.rows.filter((r) => r.risk).map((r) => r.id).sort()).toEqual([...off].sort());
    // In the order pg-grand switched off would break the forge: sign-in first, host timers last.
    const lines = riskLines(f);
    expect(lines.length).toBe(off.length);
    expect(lines[0]).toStartWith("sign-in: Sign-in edge");
    expect(lines[0]).toContain(" on pg-grand. Every request through the edge answers 500, /healthz too.");
    expect(lines[1]).toStartWith("sign-in: forge-signup-gate");
    expect(lines.at(-1)).toStartWith("release: Host timers forge-watchdog");
  });
  test("with every piece on the forge's node, there is no risk line", async () => {
    const everyone = (names: string[]) => ({ items: names.map((n) => pod(n, "pg-grand")) });
    const kube = fakeKube({
      "/api/v1/namespaces/forge/pods/forge-7d9f8c6b5-abcde": { spec: { nodeName: "pg-grand" } },
      "/api/v1/namespaces/standby/pods": everyone(["cloudflared-6c7b9d8f4-x2x9z", "cloudflared-forge-home-5f6d7c8b9-q1w2e"]),
      "/api/v1/namespaces/forge/pods": everyone(["forge-signup-gate-6d5c4b3a2-sgate"]),
      "/api/v1/namespaces/minio/pods": everyone(["minio-0"]),
      "/api/v1/namespaces/backup/pods": everyone(["lane-forge-29000000-abcde"]),
      "/api/v1/namespaces/kube-system/pods": everyone(["traefik-5b8d9c7f6-zzzzz"]),
    });
    // Every piece the cluster can show is on the forge's node; the host units are declared on pg-grand, so they remain.
    const f = await probeFooting({ fetch: kube.fetch, credentials: cred, podName: "forge-7d9f8c6b5-abcde" });
    expect(f.rows.filter((r) => r.risk).map((r) => r.id).sort()).toEqual(["forge-ssh", "host-timers", "release-tick", "standby-reconcile"]);
    expect(f.rows.filter((r) => r.risk).every((r) => r.seen.by === "declared")).toBe(true);
  });
  test("the forge's own namespace refused: the signup gate keeps its declared machine and says why", async () => {
    const kube = fakeKube({ "/api/v1/namespaces/forge/pods": 403 });
    const f = await probeFooting({ fetch: kube.fetch, credentials: cred, podName: "forge-7d9f8c6b5-abcde" });
    expect(byId(f, "signup-gate")).toMatchObject({ seen: { by: "declared", why: "the forge's service account may not read pods in forge" }, risk: true });
  });
  test("outside a cluster: the declared list, said so, every pg-grand row a risk", async () => {
    let fetched = 0;
    const f = await probeFooting({
      podName: "x",
      credentials: () => {
        throw new Error("not running in a cluster");
      },
      fetch: async () => {
        fetched++;
        return Response.json({});
      },
    });
    expect(fetched).toBe(0);
    expect(f.forgeNode).toBeNull();
    expect(f.unread).toBe("not read: not running in a cluster");
    expect(f.rows.every((r) => r.seen.by === "declared" && r.risk === (r.machine !== "vxrail"))).toBe(true);
  });
  test("the forge's own pod unreadable: its node is unknown and the declared machines decide risk", async () => {
    const kube = fakeKube({ "/api/v1/namespaces/forge/pods/forge-7d9f8c6b5-abcde": 403 });
    const f = await probeFooting({ fetch: kube.fetch, credentials: cred, podName: "forge-7d9f8c6b5-abcde" });
    expect(f.forgeNode).toBeNull();
    expect(f.unread).toBe("the forge's service account may not read its own pod");
    expect(byId(f, "traefik").risk).toBe(true);
    expect(byId(f, "forge")).toMatchObject({ seen: { by: "declared", why: "the forge's service account may not read its own pod" }, risk: false });
  });
});
describe("footingCache", () => {
  test("a slow cluster gives the declared list at once, and the reading once it lands", async () => {
    let release!: (f: Footing) => void;
    const live = declaredFooting("live", new Date(0));
    let calls = 0;
    const cache = footingCache(() => {
      calls++;
      return new Promise<Footing>((r) => (release = r));
    });
    expect(cache.peek()).toBeNull();
    const first = await cache.read(5);
    expect(first.unread).toBe("the cluster did not answer in time; reload in a moment");
    expect(calls).toBe(1); // peek's refresh is the one read waited on
    release(live);
    await Bun.sleep(0);
    expect(cache.peek()).toBe(live);
    expect(await cache.read(5)).toBe(live);
    expect(calls).toBe(1);
  });
  test("a probe that throws is not retried for 60 s, by peek or by read", async () => {
    let now = 1_000_000;
    let calls = 0;
    const cache = footingCache(async () => {
      calls++;
      throw new Error("kube down");
    }, () => now);
    expect(cache.peek()).toBeNull();
    await Bun.sleep(0);
    expect(calls).toBe(1);
    now += 30_000;
    expect(cache.peek()).toBeNull();
    expect((await cache.read(5)).unread).toBe("the cluster did not answer; it is asked again within a minute");
    await Bun.sleep(0);
    expect(calls).toBe(1);
    now += 30_001;
    cache.peek();
    await Bun.sleep(0);
    expect(calls).toBe(2);
  });
  test("a fast read leaves no deadline timer behind", async () => {
    const live = declaredFooting("live", new Date(0));
    const cleared: unknown[] = [];
    const real = globalThis.clearTimeout;
    globalThis.clearTimeout = ((t: Parameters<typeof clearTimeout>[0]) => {
      cleared.push(t);
      real(t);
    }) as typeof clearTimeout;
    try {
      expect(await footingCache(async () => live).read(60_000)).toBe(live);
    } finally {
      globalThis.clearTimeout = real;
    }
    expect(cleared.length).toBe(1);
    expect(cleared[0]).toBeDefined();
  });
});
describe("where it shows", () => {
  let dir = "";
  let app: App;
  let servers: Servers;
  const tok: Record<string, string> = {};
  beforeAll(async () => {
    dir = await mkdtemp(join(tmpdir(), "forge-footing-"));
    app = createApp({ dataDir: dir, port: 0 });
    servers = await serve(app);
    for (const [username, first] of [["alice", true], ["bob", false]] as const) {
      const r = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@x.test`, password: "a-good-passphrase", ...(first ? { adminIfFirst: true } : {}) });
      if (!r.ok) throw new Error(r.error);
      tok[username] = createToken(app.db, r.value.id, "t", "admin").token;
    }
  }, 30_000);
  afterAll(async () => {
    servers.stop();
    app.close();
    await rm(dir, { recursive: true, force: true });
  });
  test("the section has one heading, says the count once, and a row of facts per piece off the forge's machine, escaped", async () => {
    const kube = fakeKube();
    const f = await probeFooting({ fetch: kube.fetch, credentials: cred, podName: "forge-7d9f8c6b5-abcde" });
    const out = String(footingSection(f));
    const off = f.rows.filter((r) => r.risk).length;
    expect(out.match(/<h2>What the forge stands on<\/h2>/g)?.length).toBe(1);
    expect(out).toContain(`data-off="${off}"`);
    expect(out.match(new RegExp(`${off} of ${f.rows.length}`, "g"))?.length).toBe(1);
    expect(out).toContain('data-piece="traefik" data-off="false"');
    expect(out).toMatch(/data-piece="plat-auth" data-off="true"/);
    expect(out).toContain('data-breaks="2"');
    // Off-machine rows come in break order, each fact its own dt/dd.
    const order = [...out.matchAll(/data-piece="([a-z-]+)" data-off="true"/g)].map((m) => m[1]);
    expect(order.slice(0, 2)).toEqual(["plat-auth", "signup-gate"]);
    expect(out).toContain("<dt>If it stops</dt>");
    // Plain words, one colour: no badge, and neither "risk" nor "footing" on the page.
    expect(out).not.toContain("badge");
    expect(out.replace(/<[^>]*>/g, "")).not.toMatch(/\brisk|footing/i);
    expect(out).toContain('data-machine="pg-grand"');
    expect(out).toContain("forge:&lt;version&gt;");
    expect(out).toContain("The forge runs on <span class=\"mono\">vxrail-1</span>.");
    const hostile = { ...f, forgeNode: "<img src=x>" };
    expect(String(footingSection(hostile))).not.toContain("<img src=x>");
  });
  test("the owner's /system shows it", async () => {
    const res = await fetch(`${servers.url}/system`, { headers: { authorization: `Bearer ${tok["alice"]}` } });
    expect(res.status).toBe(200);
    const body = await res.text();
    expect(body).toContain("<h2>What the forge stands on</h2>");
    expect(body).toContain('id="sys-footing"');
  });
  test("/system keeps the rows behind one closed disclosure and never waits on the cluster", async () => {
    const t0 = performance.now();
    const res = await fetch(`${servers.url}/system`, { headers: { authorization: `Bearer ${tok["alice"]}` } });
    const body = await res.text();
    expect(performance.now() - t0).toBeLessThan(800);
    const section = body.slice(body.indexOf('id="sys-footing"'), body.indexOf("</section>", body.indexOf('id="sys-footing"')));
    expect(section).toMatch(/<details id="sys-footing-more" class="disclosure">/);
    expect(section).not.toMatch(/<details[^>]*\bopen\b/);
    expect(section.slice(section.indexOf("<details")).match(/data-piece=/g)?.length).toBe(FOOTING.length);
  });
  const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
  test("the terminal screen gives an admin the rows off the forge's machine in break order, with node and if-dies", async () => {
    const f = await probeFooting({ fetch: fakeKube().fetch, credentials: cred, podName: "forge-7d9f8c6b5-abcde" });
    const lines = footingLines(new Session(app, identity(getUserByName(app.db, "alice")!)), f);
    expect(lines[0]).toBe("The forge runs on vxrail-1.");
    const off = f.rows.filter((r) => r.risk);
    expect(lines).toContain(`${off.length} of ${f.rows.length} pieces run on another machine. With pg-grand off, they fail in this order:`);
    const steps = lines.filter((l) => /^ {0,2}\d+ {2}(answer|sign-in|build|release|backup)\b/.test(l));
    expect(steps.length).toBe(off.length);
    expect(steps[0]).toMatch(/^ {2}2 {2}sign-in\s+Sign-in edge/);
    expect(steps.map((l) => Number(l.trim().split(/\s+/)[0]))).toEqual([...steps.map((l) => Number(l.trim().split(/\s+/)[0]))].sort((a, b) => a - b));
    expect(lines).toContain(`${" ".repeat(15)}if it dies: Every request through the edge answers 500, /healthz too.`);
    expect(lines).toContain(`${" ".repeat(15)}on pg-grand (declared: forge-release-tick is a systemd unit on a host; the cluster cannot show it)`);
    expect(lines.at(-1)).toStartWith("On the forge's own machine: forge, tunnel, tunnel-home, traefik, hosts");
  });
  test("a guest, a signed-in non-admin and nobody get no lines and no door", () => {
    const f = declaredFooting("x");
    const doors = (user?: User): string[] => {
      const { app: tui, session } = createForgeApp(app, identity(user), { terminal: headlessTerminal(80, 24), exitHooks: false });
      const ids = forgeCommands(session)(tui.ctx).map((c) => c.id);
      tui.stop?.();
      return ids;
    };
    const alice = getUserByName(app.db, "alice")!;
    expect(doors(alice)).toContain("footing");
    for (const who of [getUserByName(app.db, "bob")!, undefined]) {
      expect(footingLines(new Session(app, identity(who)), f)).toEqual([]);
      expect(doors(who)).not.toContain("footing");
    }
  });
  test("the screen draws for an admin without waiting on the cluster", async () => {
    const { app: tui, session } = createForgeApp(app, identity(getUserByName(app.db, "alice")!), { terminal: headlessTerminal(120, 50), exitHooks: false });
    tui.start();
    tui.ctx.push(footingScreen(session, tui.ctx));
    await settleApp(tui);
    const text = tui.screen.toText();
    expect(text).toContain("What the forge stands on");
    expect(text).toContain("pieces run on another machine");
    tui.stop?.();
  });
  const mcp = async (who: string) => {
    const res = await fetch(`${servers.url}/mcp`, {
      method: "POST",
      headers: { "content-type": "application/json", accept: "application/json, text/event-stream", authorization: `Bearer ${tok[who]}` },
      body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "system", arguments: { kind: "footing" } } }),
    });
    return (await res.json()) as any;
  };
  test("the MCP system tool answers it to an admin, with the risk lines, and refuses anyone else", async () => {
    const out = JSON.parse((await mcp("alice")).result.content[0].text);
    expect(out.rows.map((r: { id: string }) => r.id)).toEqual(FOOTING.map((p) => p.id));
    expect(Array.isArray(out.risks)).toBe(true);
    expect(out.risks.length).toBe(out.rows.filter((r: { risk: boolean }) => r.risk).length);
    const refused = await mcp("bob");
    expect(JSON.stringify(refused)).toContain("only an admin of this forge reads what it stands on");
  });
});
@@ -47,6 +47,8 @@
const call = async (token: string) => {
};
test("routes lists the over-budget route first, with its budget and slowest phase; a non-admin is refused", async () => {
  // perfRecord is the process's own, and a shard-mate's requests leave samples (and patterns, up to the cap) in it.
  perfRecord.clear();
  const at = Date.now();
  for (let i = 0; i < 25; i++) {
    perfRecord.add("/boards", { at, ms: 100, signedIn: true, phases: [{ name: "app", ms: 100, hit: false }] });
@@ -118,6 +118,7 @@
test("a site-admin person is offered the whole roster", () => {
    ["disks", "Disks", "machines' disk readings: 30-day write rate and projected rated-TBW date"],
    ["telemetry", "Telemetry", "which agents may read flow and system; enter flips the grant"],
    ["stalls", "Main-thread stalls", "the last hour's stall ms by route"],
    ["footing", "What the forge stands on", "each piece on another machine, in the order it would break the forge"],
    ["capacity", "Capacity", "concurrency and guards, live; e changes a settable one"],
    ["resilience", "Resilience", "where snapshots go and the mirror; each protection green, red or untested"],
    ["maintenance", "Maintenance", "prepare the forge for a power-off, check the drain, or end maintenance"],
@@ -16,6 +16,16 @@
test("memory is capped", () => {
  for (let i = 0; i < MAX_SAMPLES + 50; i++) r.add("/x", { at, ms: 1, signedIn: true, phases: [] });
  expect(r.traffic("/x", 0).length).toBeLessThanOrEqual(MAX_SAMPLES);
});
test("clear forgets every pattern's samples and first sight, and the record fills again", () => {
  const r = createPerfRecord();
  r.add("/x", { at, ms: 1, signedIn: true, phases: [] });
  r.clear();
  expect(r.patterns()).toEqual([]);
  expect(r.traffic("/x", 0)).toEqual([]);
  expect(r.first("/x")).toBeUndefined();
  r.add("/x", { at: at + 1, ms: 2, signedIn: true, phases: [] });
  expect(r.first("/x")).toBe(at + 1);
});
test("a Server-Timing header becomes phases, hits marked", () => {
  expect(parseServerTiming('app;dur=5.0, kpis;dur=4400.0;desc="hit"')).toEqual([
    { name: "app", ms: 5, hit: false },
@@ -52,6 +52,7 @@
test("one <h2> per part, each in a settings-block, and no card", async () => {
    "Machine",
    "Disks and network",
    "Processes",
    "What the forge stands on",
    "Loop lag, last 20 minutes",
    "Main-thread stalls, last hour",
    "Telemetry readers",
@@ -124,6 +124,7 @@
const ALLOWED: readonly Allowed[] = [
    ["src/snapshot/drill.ts", "() => reject("],
    ["src/snapshot/store.ts", "() => reject("],
    ["src/snapshot/lanes.ts", "() => reject("],
    ["src/system/footing.ts", "() => r(null)"],
    ["src/release/devtools.ts", "() => { if (pending.delete(n))"],
    ["src/release/devtools.ts", "() => { waiting.set(event"],
    ["src/release/prove.ts", "() => resolve(\"hung\")"],
		@@ -25,6 +25,8 @@ State shared between modules through process memory, not the database, is named
25	25
26	26	`src/core/transfer.ts`: the old owner/name pairs a transfer left behind (`repo_redirect`), held per database after the first request reads them and dropped by `transferRepo`, so a request under a path nothing moved from costs no statement (#569).
27	27
	28	`src/system/footing.ts`: `footing`, the last reading of what the forge stands on from the kube API (#11685), held 60 s so /system, its terminal screen (`src/tui/screens/footing.ts`, palette entry for admins) and the MCP `system` tool (`kind: footing`) share one read; a probe that throws is not retried for 60 s; one reading, gone on restart.
	29
28	30	`src/apps/timer_fire.ts`: `inFlight`, the ids of timers being fired in this process, so a slow app's timer is never fired twice at once; at most the timers due, gone when each settles. The timers themselves are rows (`timer`), read at every boot, never memory.
29	31
30	32	## Awake ceiling and the boot
		@@ -209,7 +209,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
209	209	\| `subscription_wake` \| — \| Admin: wake a resting subscription \| 168 \|
210	210	\| `subscriptions` \| — \| Claude accounts: scopes, budgets \| 158 \|
211	211	\| `sync` \| — \| Sync a mirror now; `originId` reconnects it \| 197 \|
212		\| `system` \| `csp`, `routes` \| The forge machine now: CPU, memory, disk, network \| 286 \|
	212	\| `system` \| `csp`, `routes`, `footing` \| The forge machine now: CPU, memory, disk, network \| 296 \|
213	213	\| `take_back` \| — \| Open the pull that undoes a shipped pull \| 219 \|
214	214	\| `take_updates` \| — \| Take the parent forge's updates into a fork \| 176 \|
215	215	\| `teams` \| — \| Crew teams for person admins: charter, dial (0–100) and member agents \| 338 \|
		@@ -89,6 +89,69 @@ Unset, only this machine may say so, and the named header is never read. Everyth
89	89	instance _is_ lives under the data directory — the SQLite index, the bare
90	90	repos, and the SSH host key, in plain layout. Tar it up and it moves.
91	91
	92	## What the forge stands on
	93
	94	Each thing the forge needs to answer a request, sign a person in, build,
	95	release and back up, the machine it runs on, what happens to the forge if
	96	that machine dies, and the issue that removes the dependency. The list is the
	97	steward's read-only inventory of both machines on 2026-11-12 (#11685), the day
	98	pg-grand took the forge off the internet for 24 minutes while the forge itself
	99	ran fine on the vxrail, and the day the owner decided to drop pg-grand
	100	(#11686). Every row on pg-grand is a way that machine takes the forge
	101	down.
	102
	103	<!-- footing:start — written by footingTable() in src/system/footing.ts; test/footing.test.ts holds it equal -->
	104	- pg-grand: DigitalOcean. The k3s server, a single control plane with no HA. The owner decided on 2026-11-12 to drop it.
	105	- vxrail: A k3s agent joined to pg-grand. The forge runs here.
	106
	107	\| Needed to \| Piece \| Runs on \| If that machine dies \| Breaks \| Removed by \|
	108	\|---\|---\|---\|---\|---\|---\|
	109	\| (a) answer a request \| The forge's pod (git.open-platform.sh/trevato-apps/forge:<version>) and its data, PVC forge-data-vxrail (60 Gi) \| vxrail \| The forge is down, and its data is on that disk. \| — \| — \|
	110	\| (a) answer a request \| Cloudflare tunnel carrying forge.open-platform.sh (standby/cloudflared), moved to the vxrail by hand on 2026-11-12; its last-applied manifest still says 0 replicas, so a re-render moves it back \| vxrail \| It goes with the forge, which runs beside it. \| — \| #8948, #11686 \|
	111	\| (a) answer a request \| The vxrail's own tunnel (standby/cloudflared-forge-home, 2 replicas): it too sends *.open-platform.sh to Traefik; only forge-home-test.open-platform.sh and arm--forge-trevato-apps.open-platform.sh go straight to the forge \| vxrail \| It goes with the forge, which runs beside it. \| — \| — \|
	112	\| (a) answer a request \| Traefik (kube-system/traefik): every public request goes tunnel → Traefik → the forge's Service \| pg-grand \| The forge is off the internet; it keeps running inside. \| 1 \| #11686, #8946 \|
	113	\| (a) answer a request \| Child and app hosts, Traefik IngressRoutes forge-children (<name>--forge.open-platform.sh), forge-apps (*--forge-trevato-apps.open-platform.sh) and forge-old-host, and game.cheffryjames.com and www.cheffryjames.com in the forge's namespace \| pg-grand \| No child forge, app or custom domain answers. \| 1 \| #11686, #8946 \|
	114	\| (a) answer a request \| forge-ssh, a host socket proxy (:2222 → the forge's ssh service) \| pg-grand \| git over ssh and the terminal stop answering. \| 7 \| #11686 \|
	115	\| (a) answer a request \| The k3s API server (the vxrail is an agent joined to it) \| pg-grand \| Running pods keep running, but nothing reschedules, Services stop updating, and no release rolls or Job runs. \| 5 \| #11686 \|
	116	\| (b) sign a person in \| Sign-in edge: a forwardAuth middleware on the forge's Ingress, answered by mcp/plat-auth \| pg-grand \| Every request through the edge answers 500, /healthz too. \| 2 \| #11686 \|
	117	\| (b) sign a person in \| forge-signup-gate, the pod /signup is routed to \| pg-grand \| No one can sign up. \| 3 \| #11686 \|
	118	\| (c) build \| Forgejo (git.open-platform.sh) and its Actions, which build the release image \| pg-grand \| No release image is built. \| 4 \| #11686, #8941 \|
	119	\| (c) build \| Forgejo's database, a single instance under CloudNativePG (cnpg-system) \| pg-grand \| Forgejo stops, and with it builds. \| 4 \| #11686, #8941 \|
	120	\| (c) build \| forgejo-runner-vxrail, a Forgejo Actions runner \| vxrail \| It goes with the forge, which runs beside it. \| — \| — \|
	121	\| (d) release \| Forgejo's registry (git.open-platform.sh), which the forge's image is pulled from \| pg-grand \| A pod that restarts on a node without the image cached cannot start; nothing new is released. \| 4 \| #11686, #8941, #8942 \|
	122	\| (d) release \| registry-node, a registry on the vxrail \| vxrail \| It goes with the forge, which runs beside it. \| — \| — \|
	123	\| (d) release \| forge-release-tick, a host timer every 5 min \| pg-grand \| Nothing is released; production keeps the release it has. \| 7 \| #11686, #8942 \|
	124	\| (d) release \| standby-reconcile, a host timer every 2 min that applies the forge's manifests from Forgejo; Flux's Kustomization for the forge is suspended \| pg-grand \| A changed manifest is not applied; the running pod is untouched. \| 7 \| #8948, #11686 \|
	125	\| (d) release \| Host timers forge-watchdog, forge-github-mirror and forge-prune-images \| pg-grand \| The watchdog, the GitHub mirror and image pruning stop. \| 7 \| #11686 \|
	126	\| (e) back up \| The backup namespace's CronJobs (FORGE_BACKUP_LANES) \| pg-grand \| No new backup is taken. \| 6 \| #11686 \|
	127	\| (e) back up \| MinIO, where the backup lanes store the forge's backups \| pg-grand \| No new backup is stored, and those it holds go with the machine. \| 6 \| #11686 \|
	128
	129	If pg-grand were switched off now, the forge breaks in this order (the Breaks column):
	130
	131	1. Traefik: nothing public reaches the forge
	132	2. plat-auth: no one signs in
	133	3. the signup gate: no one signs up
	134	4. image pulls and releases
	135	5. the control plane: nothing reschedules, Services stop updating
	136	6. local backups
	137	7. the host timers
	138
	139	Also on pg-grand, not in the forge's path: Forgejo and its database, the platform's mcp/plat-auth and www, the CloudNativePG operator with the tenant apps' databases, their local-path volumes pinned to that node, and MinIO.
	140	<!-- footing:end -->
	141
	142	The same list is read from the live cluster on `/system`, under "What the
	143	forge stands on", on the terminal's screen of that name, and from the MCP
	144	`system` tool with `{ kind: "footing" }`; how to read each is in
	145	[docs/features/footing.md](features/footing.md). The probe
	146	(`src/system/footing.ts`) asks the kube API, with the forge's own service
	147	account, for the forge's node, the nodes labelled control-plane, and the pods
	148	in each namespace above; it never lists every pod in the cluster. What the
	149	cluster cannot show keeps its declared machine and says why. To see every
	150	row from the cluster, the forge's service account needs `get` on its own pod,
	151	`list` on `nodes`, and `list` on `pods` in those namespaces (#8948). The
	152	reading is held 60 s, a page waits at most 0.8 s for it, and a probe that
	153	fails is not retried for a minute.
	154
92	155	## Health
93	156
94	157	`GET /api/v1/health` (and the MCP `health` tool) answer a site admin with
		@@ -0,0 +1,58 @@
	1	# What the forge stands on (#11685)
	2
	3	The pieces the forge needs to answer a request, sign a person in, build,
	4	release and back up, and the machine each one runs on. Any piece that is not
	5	on the forge's own machine is a way that machine can take the forge down. The
	6	declared list is the table in [`docs/OPERATING.md`](../OPERATING.md#what-the-forge-stands-on);
	7	the three doors below read it against the live cluster.
	8
	9	All three are for a forge admin who is a person (the MCP tool: a forge admin).
	10	None of them waits on the cluster: they show the last reading, held 60 s, and
	11	the declared list while the first one runs.
	12
	13	## On /system
	14
	15	Under What the forge stands on, the summary says how many pieces run on
	16	another machine. Open it to see:
	17
	18	- where the forge runs (its pod's node), or why that was not read;
	19	- each piece on another machine, in the order switching pg-grand off would
	20	break the forge, with what it is needed for, where it runs, what happens if
	21	that machine stops, its place in that order, and the issues that remove it;
	22	- the pieces on the forge's own machine, by name;
	23	- what each machine is.
	24
	25	"Runs on" is a node name when the cluster showed the piece's pod there. It
	26	reads `pg-grand (declared: …)` when the cluster could not show it, with the
	27	reason: a systemd unit on a host, a CronJob with no pod between runs, a
	28	namespace the service account may not list, or a pod whose namespace is not
	29	recorded. A declared piece is judged by its declared machine.
	30
	31	## In the terminal
	32
	33	Open the palette and choose What the forge stands on (it follows
	34	Main-thread stalls). The screen lists the same rows as the page: the
	35	forge's node, then each piece on another machine in break order with its
	36	step, need, machine and what happens if it dies, then the steps themselves and
	37	the pieces on the forge's own machine. `j`/`k` scroll, `esc` closes.
	38
	39	## Over MCP
	40
	41	```json
	42	{ "name": "system", "arguments": { "kind": "footing" } }
	43	```
	44
	45	answers `forgeNode`, `unread` (why the forge's node was not read, or null),
	46	`at`, and per piece `{ id, need, what, machine, seen, risk, ifDies, issues,
	47	breaks }`. `seen` is `{ by: "cluster", nodes }` when the cluster showed it and
	48	`{ by: "declared", why }` otherwise; `risk` is true when the piece is not on
	49	the forge's machine. `risks` repeats those pieces as one line each, in break
	50	order. Anyone but a forge admin is refused.
	51
	52	## Keeping it true
	53
	54	The list is `FOOTING` in `src/system/footing.ts`; `test/footing.test.ts` holds
	55	the OPERATING.md table equal to `footingTable()`. Change the list there, then
	56	paste the output of
	57	`bun -e 'import {footingTable} from "./src/system/footing.ts"; console.log(footingTable())'`
	58	between the markers in OPERATING.md.
		@@ -7,16 +7,17 @@ import { stallHour, TOP_ROUTES } from "../../../metrics/stalls.ts";
7	7	import { lagMinutes, loopWatch, perfRecord } from "../../../perf/live.ts";
8	8	import { LAG_BAR_MS, overBar } from "../../../perf/lag_minutes.ts";
9	9	import { slowestRoutes } from "../../../perf/watch.ts";
	10	import { footing, riskLines } from "../../../system/footing.ts";
10	11	import { good, bad, type Tool } from "../shared.ts";
11	12
12	13	export const tool: Tool = {
13	14	name: "system",
14		doc: "the machine the forge runs on, now: CPU, load, memory, disk, network, processes, `stalls`: the last hour's main-thread stalls by route (`<route> (in flight)` when still running at the wake), or `job:<name>`, as {cause, ms, count}, `readings`: the last 20 minutes' worst loop lag as {at, ms, what} beside `barMs` (100) and `over`, and `readers`: who polls the telemetry and how fast, `{ kind: \"routes\" }`: the 10 slowest budgeted routes as {route, p95Ms, budgetMs, n, slowestPhase}, over budget first; a forge admin, or an agent granted `telemetry:read`. `{ kind: \"csp\" }`: the last 24 h of Content-Security-Policy violations browsers reported, `{ directive, n, newest }` per directive (`newest` the blocked host last counted), most first, counted per UTC hour from `since`, and `scriptErrors` `{ page, where, message, n, last }`, as the security page reads them; a forge admin only",
	15	doc: "the machine the forge runs on, now: CPU, load, memory, disk, network, processes, `stalls`: the last hour's main-thread stalls by route (`<route> (in flight)` when still running at the wake), or `job:<name>`, as {cause, ms, count}, `readings`: the last 20 minutes' worst loop lag as {at, ms, what} beside `barMs` (100) and `over`, and `readers`: who polls the telemetry and how fast, `{ kind: \"routes\" }`: the 10 slowest budgeted routes as {route, p95Ms, budgetMs, n, slowestPhase}, over budget first; a forge admin, or an agent granted `telemetry:read`. `{ kind: \"csp\" }`: the last 24 h of Content-Security-Policy violations browsers reported, `{ directive, n, newest }` per directive (`newest` the blocked host last counted), most first, counted per UTC hour from `since`, and `scriptErrors` `{ page, where, message, n, last }`, as the security page reads them; a forge admin only. `{ kind: \"footing\" }`: what the forge stands on (#11685), as /system's \"What the forge stands on\": `forgeNode`, and per piece `{ id, need, what, machine, seen, risk, ifDies, issues }` read from the live cluster, with `risks`, one line per piece not on the forge's machine; a forge admin only",
15	16	description:
16	17	"The forge machine now: CPU, memory, disk, network. Admin or telemetry:read; `kind: csp`: 24 h of CSP violations (admin).",
17		inputSchema: { type: "object", properties: { kind: { type: "string", enum: ["csp", "routes"] } } },
	18	inputSchema: { type: "object", properties: { kind: { type: "string", enum: ["csp", "routes", "footing"] } } },
18	19	writes: false,
19		run(ctx, user, args) {
	20	async run(ctx, user, args) {
20	21	if (args["kind"] !== undefined) {
21	22	if (args["kind"] === "routes") {
22	23	const reader = telemetryReader(ctx.app, user);
		@@ -24,7 +25,12 @@ export const tool: Tool = {
24	25	noteRead(ctx.app, user, reader, "system");
25	26	return good({ routes: slowestRoutes(perfRecord, Date.now()) });
26	27	}
27		if (args["kind"] !== "csp") return bad("invalid", "kind is csp, routes, or absent for the machine");
	28	if (args["kind"] === "footing") {
	29	if (!user.isAdmin) return bad("forbidden", "only an admin of this forge reads what it stands on");
	30	const f = await footing.read();
	31	return good({ ...f, risks: riskLines(f) });
	32	}
	33	if (args["kind"] !== "csp") return bad("invalid", "kind is csp, routes, footing, or absent for the machine");
28	34	if (!user.isAdmin) return bad("forbidden", "only an admin of this forge reads its CSP violations");
29	35	const now = new Date();
30	36	return good({ hours: 24, since: windowStart(24, now).toISOString(), directives: violationsByDirective(ctx.app.db, 24, now), scriptErrors: scriptErrorCounts(ctx.app.db, 24, now) });
		@@ -50,6 +50,8 @@ export interface PerfRecord {
50	50	/** When `pattern` was first seen in this process, undefined if never. */
51	51	first(pattern: string): number \| undefined;
52	52	patterns(): string[];
	53	/** Forget every route's samples (not the minute ledger), as `loopWatch.clear` does its stalls: for a test that reads the record whole. */
	54	clear(): void;
53	55	/** Who used the answering time, by the minute: exact however the samples are thinned. */
54	56	readonly ledger: TrafficLedger;
55	57	}
		@@ -100,6 +102,10 @@ export function createPerfRecord(valid: (key: string) => boolean = validPattern)
100	102	traffic: (pattern, since) => (real.get(pattern) ?? []).filter((s) => s.at >= since),
101	103	first: (pattern) => firsts.get(pattern),
102	104	patterns: () => [...real.keys()],
	105	clear() {
	106	real.clear();
	107	firsts.clear();
	108	},
103	109	};
104	110	}
105	111
		@@ -57,6 +57,7 @@ import { entry as plans } from "./plans.ts";
57	57	import { entry as economics } from "./economics.ts";
58	58	import { entry as money } from "./money.ts";
59	59	import { entry as loopStalls } from "./loopstalls.ts";
	60	import { entry as footing } from "./footing.ts";
60	61	import { entry as waiting } from "./waiting.ts";
61	62	import { entry as slowPagesEntry } from "./slowpages.ts";
62	63	import { entry as maintenance } from "./maintenance.ts";
		@@ -122,6 +123,7 @@ export const ENTRIES: readonly PaletteEntry[] = [
122	123	homePins,
123	124	system,
124	125	loopStalls,
	126	footing,
125	127	waiting,
126	128	slowPagesEntry,
127	129	telemetry,
		@@ -0,0 +1,77 @@
	1	/**
	2	* What the forge stands on, in the terminal (#11685): the rows /system shows
	3	* under the same heading. First where the forge runs (or why that was not
	4	* read), then each piece on another machine, in the order switching pg-grand
	5	* off would break the forge: its step, what it is needed for, the piece, the
	6	* machine it was found on, and what happens if that machine dies. Last, the
	7	* pieces on the forge's own machine, by name. The last reading
	8	* (`footing.peek`), so the screen never waits on the cluster; the declared
	9	* list while the first read runs. The gate is /system's: a forge admin who is
	10	* a person. Read-only.
	11	*/
	12
	13	import { BREAK_STEPS, declaredFooting, footing, offMachine, whereWords, type Footing } from "../../system/footing.ts";
	14	import type { Ctx, Screen } from "../app/app.ts";
	15	import { binding } from "../app/keys.ts";
	16	import type { Session } from "../session.ts";
	17	import { style } from "../term/style.ts";
	18	import { pad } from "../term/width.ts";
	19	import { Scroll, drawLines, hitScroll } from "./common.ts";
	20	import type { PaletteEntry } from "./entries.ts";
	21
	22	const mayRead = (session: Session): boolean => session.user?.isAdmin === true && session.user.kind !== "agent";
	23
	24	const current = (): Footing => footing.peek() ?? declaredFooting("the cluster is being read; reopen in a moment");
	25
	26	/** The screen's lines; empty for anyone the gate refuses. */
	27	export function footingLines(session: Session, f: Footing = current()): string[] {
	28	if (!mayRead(session)) return [];
	29	const off = offMachine(f);
	30	const home = f.rows.filter((r) => !r.risk);
	31	const where =
	32	f.forgeNode === null ? `The forge's node was not read (${f.unread ?? "unknown"}); machines are as declared.` : `The forge runs on ${f.forgeNode}.`;
	33	const lines = [where, ""];
	34	if (off.length === 0) lines.push("Every piece runs on the forge's own machine.");
	35	else {
	36	lines.push(`${String(off.length)} of ${String(f.rows.length)} pieces run on another machine. With pg-grand off, they fail in this order:`);
	37	for (const r of off) {
	38	const step = r.breaks === undefined ? "-" : String(r.breaks);
	39	lines.push("", `${pad(step, 3, "right")} ${pad(r.need, 8)} ${r.what}`);
	40	lines.push(`${pad("", 15)}on ${whereWords(r)}`, `${pad("", 15)}if it dies: ${r.ifDies}`);
	41	}
	42	lines.push("", "The steps:", ...BREAK_STEPS.map((s, i) => `${pad(String(i + 1), 3, "right")} ${s}`));
	43	}
	44	if (home.length > 0) lines.push("", `On the forge's own machine: ${home.map((r) => r.id).join(", ")}`);
	45	return lines;
	46	}
	47
	48	export function footingScreen(session: Session, ctx: Ctx): Screen {
	49	const scroll = new Scroll();
	50	return {
	51	title: "What the forge stands on",
	52	keys: [binding(["j", "k"], "scroll"), binding(["esc", "q"], "back")],
	53	render(area, s) {
	54	const c = ctx.theme.c;
	55	const lines = footingLines(session);
	56	drawLines(s, area, c, scroll, lines.map((text) => [{ text, style: style(/^\s*\d+ {2}/.test(text) \|\| text === "" ? c.fg : c.mutedFg, c.bg) }]));
	57	hitScroll(ctx, area, scroll, lines.length);
	58	},
	59	key(e) {
	60	if (scroll.key(e, footingLines(session).length)) return true;
	61	if (e.key === "esc" \|\| e.key === "q") {
	62	ctx.pop();
	63	return true;
	64	}
	65	return false;
	66	},
	67	};
	68	}
	69
	70	export const entry: PaletteEntry = {
	71	id: "footing",
	72	title: "What the forge stands on",
	73	hint: "each piece on another machine, in the order it would break the forge",
	74	after: ["stalls", "system"],
	75	offered: mayRead,
	76	run: (session, ctx) => ctx.push(footingScreen(session, ctx)),
	77	};
		@@ -47,6 +47,7 @@ import { KEPT_PAGE_BYTES, keptPageStats } from "./kept_page.ts";
47	47	import { COLUMN_BYTES, PARTS_BYTES, keptPartsStats, treeCacheStats } from "./tree_cache.ts";
48	48	import { featureHealth, type FeatureHealth } from "../../system/feature_health.ts";
49	49	import { egressReportedLine } from "../../system/egress_line.ts";
	50	import { BREAK_STEPS, MACHINES, declaredFooting, footing, offMachine, whereWords, type Footing, type Machine } from "../../system/footing.ts";
50	51	import { badge, chartBox, disclosure, meter, pill, row, time } from "../ui.ts";
51	52	import { childDirOf, listChildren } from "../../actions/spawn.ts";
52	53	import { childrenUse, realSelf, selfUse, shareOf, sizeWords, useWords, type ChildUseRow, type SelfUse } from "../../spawn/usage.ts";
		@@ -204,6 +205,38 @@ export function featuresSection(rows: readonly FeatureHealth[]): Html {
204	205	<p class="small"><a href="/settings/features">Settings › Features</a></p></section>`;
205	206	}
206	207
	208	/**
	209	* What the forge stands on (#11685): each piece it needs to answer, sign in,
	210	* build, release and back up, and where it was found. The summary gives the
	211	* one count; inside, each piece on another machine, in the order switching
	212	* pg-grand off would break the forge, one fact a line; then the pieces on the
	213	* forge's own machine, by name. `system/footing.ts`'s rows.
	214	*/
	215	export function footingSection(f: Footing): Html {
	216	const off = offMachine(f);
	217	const home = f.rows.filter((r) => !r.risk);
	218	// First sight (#11111): the heading and one count stay; the rows open behind one disclosure.
	219	const count = off.length === 0 ? "Every piece runs on the forge's own machine" : `${off.length} of ${f.rows.length} pieces run on another machine`;
	220	return html`<section class="settings-block" id="sys-footing" data-off="${String(off.length)}" data-forge-node="${f.forgeNode ?? ""}"><h2>What the forge stands on</h2>
	221	${disclosure(count, html`<p class="small">${f.forgeNode === null ? html`The forge's node was not read (${f.unread ?? "unknown"}), so each piece shows the machine written in <code>docs/OPERATING.md</code>.` : html`The forge runs on <span class="mono">${f.forgeNode}</span>.`}
	222	${off.length === 0 ? null : "If that machine stops, each piece below stops with it; they are listed in the order the forge would lose them."}</p>
	223	${off.length === 0 ? null : html`<ul class="list" aria-label="Pieces on another machine">${off.map((r) =>
	224	row({
	225	title: r.what,
	226	facts: [
	227	{ label: "Needed to", value: r.need },
	228	{ label: "Runs on", value: html`<span class="mono">${whereWords(r)}</span>` },
	229	{ label: "If it stops", value: r.ifDies },
	230	...(r.breaks ? [{ label: "Order", value: `${r.breaks} of ${BREAK_STEPS.length}: ${BREAK_STEPS[r.breaks - 1]}` }] : []),
	231	...(r.issues.length ? [{ label: "Fixed by", value: r.issues.map((n) => `#${n}`).join(", ") }] : []),
	232	],
	233	data: { piece: r.id, off: "true", seen: r.seen.by, ...(r.breaks ? { breaks: String(r.breaks) } : {}) },
	234	}),
	235	)}</ul>`}
	236	${home.length === 0 ? null : html`<p class="small muted">On the forge's own machine: ${home.map((r, i) => html`${i ? ", " : ""}<span data-piece="${r.id}" data-off="false">${r.what}</span>`)}.</p>`}
	237	${(Object.keys(MACHINES) as Machine[]).map((m) => html`<p class="small muted" data-machine="${m}"><span class="mono">${m}</span>: ${MACHINES[m]}</p>`)}`, { id: "sys-footing-more" })}</section>`;
	238	}
	239
207	240	/** The last hour's stalls by route, most milliseconds first; the rows are `stallHour`'s. */
208	241	export function stallSection(rows: readonly RouteStall[], worst: Html = html``): Html {
209	242	return html`<section class="settings-block"><h2>Main-thread stalls, last hour</h2>${
		@@ -370,6 +403,8 @@ export async function systemPage(ctx: Ctx): Promise<Response> {
370	403	const stalls = timePhase(ctx.request, "stalls", () => stallHour(loopWatch.tally(performance.now())));
371	404	const nowMono = performance.now();
372	405	const worst = stallBlame.worst(nowMono, HOUR_MS, WORST_STALLS);
	406	// Never waits on the cluster: the last reading, or the declared rows while the first read runs behind.
	407	const standsOn = footing.peek() ?? declaredFooting("the cluster is being read; reload in a moment");
373	408	const rendering = performance.now();
374	409	const r = regions(v);
375	410	const at = (id: string): Html => raw(r[id]!);
		@@ -403,6 +438,7 @@ export async function systemPage(ctx: Ctx): Promise<Response> {
403	438	<div class="js-only small muted">Sort by <button class="btn" data-size="sm" type="button" data-sort="cpu" aria-pressed="true"><kbd>c</kbd> CPU</button>
404	439	<button class="btn" data-size="sm" type="button" data-sort="mem" aria-pressed="false"><kbd>m</kbd> memory</button></div>
405	440	<div id="sys-procs" data-live-hold>${at("sys-procs")}</div></section>
	441	${footingSection(standsOn)}
406	442	${featuresSection(timePhase(ctx.request, "features", () => featureHealth(ctx.app, reader)))}
407	443	${lagSection(lagMinutes.readings())}
408	444	<div id="sys-stalls">${stallSection(stalls, worstStallsTable(worst, nowMono, Date.now()))}</div>
		@@ -47,6 +47,8 @@ const call = async (token: string) => {
47	47	};
48	48
49	49	test("routes lists the over-budget route first, with its budget and slowest phase; a non-admin is refused", async () => {
	50	// perfRecord is the process's own, and a shard-mate's requests leave samples (and patterns, up to the cap) in it.
	51	perfRecord.clear();
50	52	const at = Date.now();
51	53	for (let i = 0; i < 25; i++) {
52	54	perfRecord.add("/boards", { at, ms: 100, signedIn: true, phases: [{ name: "app", ms: 100, hit: false }] });