Public: anyone can read this; the crew can change it.
Spin down the Kubernetes platform: everything lives in the forge #8953
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/7667-spin-down-the-kubernetes-platform-everyt → main 1 commit fork
3 files
changed +109
−0
README.md +1
| @@ -560,6 +560,7 @@ This page is the front door. Everything else is a file of its own under | ||
| 560 | 560 | - [](docs/CLOUD.md) — clouds are capacity, never dependency, and the cloud accounts an admin keeps. |
| 561 | 561 | - [](docs/CLUSTER.md) — the check lane as Kubernetes Jobs. |
| 562 | 562 | - [](docs/EXTRACT.md) — the forge on a machine of its own, and the platform it grew up in switched off: what a host owes it, the moves in order, and what is given up. |
| 563 | - [](docs/PLATFORM.md) — everything the forge still takes from open-platform, what replaces each, what is done, and the issue for the next step; the order and the gate for switching the platform off. | |
| 563 | 564 | - [](docs/ops/ROLL.md) — rolling a release. |
| 564 | 565 | |
| 565 | 566 | **Design and records** |
docs/PLATFORM.md added +86
| @@ -0,0 +1,86 @@ | ||
| 1 | # Platform inventory — what the forge still takes from open-platform (#7667) | |
| 2 | ||
| 3 | Owner, 2026-11-02: "this does go hand in hand with wanting everything to be | |
| 4 | forge native now and getting off of the platform and the forge to forge | |
| 5 | origin connections." | |
| 6 | ||
| 7 | This page lists each thing the forge and its apps still take from the | |
| 8 | platform (the Kubernetes cluster, Forgejo at git.open-platform.sh, Flux, | |
| 9 | Kyverno, the Cloudflare edge, the droplet on pg-grand), what replaces it, | |
| 10 | what is already done, and the issue that does the next step. It is written | |
| 11 | from this repository. What lives only in the cluster or in the gitops | |
| 12 | repository (`trevato-apps/gitops`) cannot be read from here and is marked | |
| 13 | **unread**; #8948 reads it. | |
| 14 | ||
| 15 | **Nothing is turned off or removed by this page or by any step it names | |
| 16 | without the owner present. The forge never goes dark.** Every step has its | |
| 17 | rollback written in its issue, and the platform stays up until the last | |
| 18 | step (#8952). | |
| 19 | ||
| 20 | ## What #1432 retired, and what it did not | |
| 21 | ||
| 22 | #1432 (the forge deploys itself) is closed. What it delivered, on main: | |
| 23 | ||
| 24 | - `releaseOnLand` checks every landed tree out to `releases/<sha>/` and | |
| 25 | preflights it (`src/checks/landing.ts`); the keeper cuts, swaps in place | |
| 26 | with SO_REUSEPORT, verifies under load and swaps back by name | |
| 27 | (`src/keeper/`, `src/release/`; `docs/features/ship-itself.md`). | |
| 28 | - A swap needs no image, no registry, no Forgejo and no Kubernetes roll. | |
| 29 | ||
| 30 | What it did **not** retire, because production does not run it yet: | |
| 31 | ||
| 32 | - Production is not switched to `FORGE_RELEASE_KEEPER=swap` | |
| 33 | (`FORGE_REPO`, `FORGE_SELF_SWAP=1`); the swap has not run on a hosted | |
| 34 | child with measured dark seconds (#8374 closed on the question, the drill | |
| 35 | is #8940). Releases still go through a `v*` tag, the Forgejo release | |
| 36 | workflow, an image on the platform registry, a chart, and Flux/standby | |
| 37 | reconcile rolling the pod. | |
| 38 | - The path is still in the tree: `.forgejo/workflows/release.yml`, the | |
| 39 | Dockerfile's `VERSION`/`COMMIT` stamping, `keeper/deploy.ts` | |
| 40 | `kubeDeployer`, `scripts/cut-release.ts`' image half. | |
| 41 | - The droplet's `forge-release-tick` (pg-grand) and the steward's Mac | |
| 42 | `tick release` still exist (**unread** here: only named in | |
| 43 | `docs/features/ship-itself.md` and `release-keeper.md`). | |
| 44 | ||
| 45 | ## The inventory | |
| 46 | ||
| 47 | | # | The forge takes from the platform | Replaced by | Done already | Next step | | |
| 48 | |---|---|---|---|---| | |
| 49 | | 1 | **Image build and registry.** `release.yml` builds on the droplet runner, signs with cosign, pushes image and chart to the Forgejo registry (`forgejo-http.forgejo.svc.cluster.local:3000`). | Self-swap: the tree is the release; no image. | Code on main (#8042, #8054); runs in tests. | Drill on a child (#8940); switch production (#8941); delete the path after a week (#8942). | | |
| 50 | | 2 | **Gitops and reconcile.** `trevato-apps/gitops` `apps/forge/prod/manifests.yaml`, `signup-gate.yaml`, `outsider-gate.yaml`; Flux; Kyverno; the standby reconcile on pg-grand (~15 min) which also resets the forge container's `image`. | A unit file and settings in the forge (env, limits, gates are forge settings); no reconcile. | Nothing: **unread** from here. | Read it into the forge and table every object (#8948). The reconcile stops owning `image` first (precondition of #8941). | | |
| 51 | | 3 | **The pod and its supervisor.** A Deployment on local-path storage, `maxSurge 1`, restarts, a service account for the keeper's Deployment patch. | A box of the forge's own running `bun run src/main.ts` under systemd (`Restart=always`); `release/launch.ts` supervises swaps. Keeper #8374 proved the swap, not the box. | `docs/EXTRACT.md` steps 1–3, `docs/RESTORE.md`, `release/launch.ts`, `good.json`. | A box beside production taking no traffic (#8945). | | |
| 52 | | 4 | **Ingress and TLS.** Cloudflare tunnel and cluster ingress carry `forge.open-platform.sh`, `*--forge.open-platform.sh` children and apps' own domains. | Caddy with a DNS-challenge wildcard and on-demand TLS asking `/__forge/domains/ask`, on the box. | Forge side done: `FORGE_TRUSTED_PROXIES`, `FORGE_CLIENT_IP_HEADER`, domains ask endpoint (`docs/DOMAINS.md`). | Staging name first, cutover runbook with rollback, the owner's go (#8946, after #8945). | | |
| 53 | | 5 | **Identity.** `X-Plat-User` / `X-Plat-Edge-Token` (`src/http/edge.ts`): accounts the edge made have no password. | The forge's own password, passkey and OAuth between forges (#8902). `signin.edge` is the switch; an admin with neither is refused. | Switch exists (`docs/OPERATING.md`); `via: edge` is listed per account. | List edge-only accounts, a passkey for each, then switch off on a preview (#8947). | | |
| 54 | | 6 | **Backup lanes.** The `backup` namespace's CronJobs (`FORGE_BACKUP_LANES`), `forgejo-dump` (suspended, 36 GB), the platform bucket's off-machine mirror. | The forge's own snapshot to its bucket and B2 (`docs/SNAPSHOT.md`), drilled weekly (`docs/RECOVERY.md`). | Snapshot, restore command, weekly drill, backup tile (`backup-truth.md`). | Decide the dump with the owner, a line for every lane, a restore with the platform unreachable (#8949, drill on #8945). | | |
| 55 | | 7 | **Forgejo, where outside people send pulls.** grokbot's #42 to #67, `jmoran/forge-jay` (#8935), tdobbertin-espo's six (#8938) on git.open-platform.sh. | The forge: a real merge for a kept fork (#8935), a URL-to-landed-pull path for an agent (#6359), forge-to-forge origin (`docs/features/forge-origin.md`). | Origins and Bring exist (`docs/ORIGINS.md`). | Tell each person, then disable (not delete) new pulls on the Forgejo repository (#8950; waits for #6359, #8935). | | |
| 56 | | 8 | **Hosted apps.** Apps the platform runs that are not on the forge. The list is in the cluster: **unread**. | Bring (paste a link) or Migrate; GitHub and Vercel go through #6007, not here. | Bring an app whole (#8919). | List with the owner's call per app (#8951). | | |
| 57 | | 9 | **The droplet release tick.** `forge-release-tick` on pg-grand and the Mac's `tick release`. | The keeper (#1). | The keeper's code. | Switched off by the owner on the day #8942 records. | | |
| 58 | | 10 | **Forgejo Actions jobs that are not releases.** `flake-hunt.yml`, `outside-gate.yml`, `sync-env.yml` on platform runners. | Scheduled jobs the forge runs, same verdicts. | `scripts/flake-hunt.ts`, checks and cells. | #8943. | | |
| 59 | | 11 | **Previews of the forge's own repository.** `preview.yml` builds an image, signs it, pushes a chart. | A child forge in a cell (`src/spawn/placement.ts`, kind `cell`). | Apps preview from branch in a cell today. | #8944. | | |
| 60 | ||
| 61 | ## Order and the gate | |
| 62 | ||
| 63 | Cuts are made one at a time, each with its rollback tested before the next. | |
| 64 | The map holds the order: #8941 waits for #8940, #8942 for #8941, #8946 for | |
| 65 | #8945, #8950 for #6359 and #8935. Every item above, and #8951, feeds the | |
| 66 | last: **#8952**, which starts nothing until all the others are done, then: | |
| 67 | ||
| 68 | 1. The owner sets a quiet period (suggested 30 days). | |
| 69 | 2. The platform's access logs, read at the edge, show no request from the | |
| 70 | forge, its apps, its people or its agents for that period; the count is | |
| 71 | posted weekly. | |
| 72 | 3. A restore drill passed in that period that did not touch the platform | |
| 73 | (#8945, #8949). | |
| 74 | 4. A final snapshot of the platform's own state is in B2. | |
| 75 | 5. The owner says go. It is scaled to zero, not deleted, and kept a month. | |
| 76 | ||
| 77 | ## Outside agents | |
| 78 | ||
| 79 | Outside agents work against the forge and are pointed there (#8950). The | |
| 80 | route `trevato-apps/forge` on Forgejo closes only after #6359 gives them a | |
| 81 | way in, and then pulls are disabled, never deleted, with rollback in | |
| 82 | Forgejo's settings. | |
| 83 | ||
| 84 | ## Not here | |
| 85 | ||
| 86 | Moving GitHub and Vercel in is #6007. No cut is started by this page. |
test/platform-doc.test.ts added +22
| @@ -0,0 +1,22 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { existsSync, readFileSync } from "node:fs"; | |
| 3 | import { join } from "node:path"; | |
| 4 | ||
| 5 | // docs/PLATFORM.md (#7667) is the inventory of what the forge still takes | |
| 6 | // from the platform. It stays true to the repository: the files it says | |
| 7 | // can be deleted are really there, and it names the gate. | |
| 8 | const root = join(import.meta.dir, ".."); | |
| 9 | const doc = readFileSync(join(root, "docs/PLATFORM.md"), "utf8"); | |
| 10 | ||
| 11 | test("every file the inventory names exists", () => { | |
| 12 | const paths = [...doc.matchAll(/`((?:docs|src|scripts|\.forgejo)\/[A-Za-z0-9_./-]+\.(?:md|ts|yml))`/g)].map((m) => m[1]!); | |
| 13 | expect(paths.length).toBeGreaterThan(8); | |
| 14 | expect(paths.filter((p) => !existsSync(join(root, p)))).toEqual([]); | |
| 15 | }); | |
| 16 | ||
| 17 | test("it covers each dependency the owner listed, and says nothing is turned off", () => { | |
| 18 | for (const topic of ["Image build and registry", "Gitops and reconcile", "pod and its supervisor", "Ingress and TLS", "Identity", "Backup lanes", "Forgejo, where outside people", "Hosted apps", "release tick"]) | |
| 19 | expect(doc).toContain(topic); | |
| 20 | expect(doc).toContain("without the owner present"); | |
| 21 | expect(doc).toContain("#8952"); | |
| 22 | }); |
new above
offline