Public: anyone can read this; the crew can change it.

Everything the forge needs to be reachable and to release is written down, with where it runs: today six of those things run on the DigitalOcean node #11685

shipped production is serving it trevatoowner opened this 4h ago torquecrew has it

trevatoowner opened this 4h ago

Why. The owner believed the forge was severed from the platform. It is not: on 2026-11-12 a full disk on the DigitalOcean node took the forge off the internet for 24 minutes while the forge itself ran fine on the vxrail.

What is known to live on that node today (verify each, find the rest):

The Cloudflare tunnel that carries forge.open-platform.sh (deployment standby/cloudflared; a second tunnel, cloudflared-forge-home, already runs on the vxrail but did not carry the forge's address).
The sign-in edge in front of the forge (mcp/plat-auth, a forwardAuth; with it down even /healthz answered 500).
Traefik, the ingress the tunnel points at.
Forgejo (git.open-platform.sh) and its database: the forge's release image is built by Forgejo Actions and pulled from Forgejo's registry; other pods' images (plat-auth itself) are pulled from it too.
The release tick (/var/tmp/forge-release/release-tick.sh, a systemd timer) and its admin token.
The Kubernetes control plane: the vxrail is a worker joined to this node.

Also there: backups (MinIO, CloudNativePG), Flux, and the platform's other apps.

What. docs/OPERATING.md gains one table: each thing the forge needs to (a) answer a request, (b) sign a person in, (c) build, (d) release, (e) back up — with the machine it runs on, what happens to the forge if that machine dies, and the issue that removes the dependency. A test or a probe (system tool, /system) lists the same from the live cluster so the table cannot rot: any required piece not on the forge's own machine shows as a risk line.

Acceptance. The table exists and matches the probe; the probe is on /system under one heading, "What the forge stands on". Nothing on the list is a surprise to the owner again.

Plan · 7 of 8 done · 4h ago

Read /system, MCP system tool, cluster/jobs kube client, PLATFORM.md done
src/system/footing.ts: declared list + live probe from the kube API done
/system section "What the forge stands on" with risk lines done
MCP system tool kind footing; terminal System screen lines done
docs/OPERATING.md table; test holds it equal to the list done
Tests: probe against a fake kube API, page, MCP, terminal done
gate, tsc, design-lint, push done
Owner: grant the forge service account get/list nodes and list pods in the listed namespaces so every row is read live ( to do

trevatoowner commented 4h ago

Ground truth from a read-only inventory of both machines by the steward, 2026-11-12 (the crew cannot ssh; use this, and make /system show the parts the forge can see itself). Owner's decision the same day: the DigitalOcean node is to be dropped.

Machines. pg-grand (DigitalOcean): the k3s server; datastore is sqlite via kine (/var/lib/rancher/k3s/server/db/state.db), no HA; disk 154 GB, 84% used. vxrail: a k3s agent joined to it over the tailnet, taint plat.sh/site=vxrail:NoSchedule; disk 2.2 TB, 83% used.

On the vxrail: the forge pod (git.open-platform.sh/trevato-apps/forge:<version>, PVC forge-data-vxrail 60 Gi), standby/cloudflared (moved there by hand on 11-12; its last-applied manifest still says 0 replicas, so a re-render reverts it), standby/cloudflared-forge-home ×2, forgejo-runner-vxrail, registry-node.

On pg-grand, in the forge's path:

Traefik (kube-system/traefik): every public request goes tunnel → Traefik → the forge's Service. Even the vxrail's own tunnel (cloudflared-forge-home) sends *.open-platform.sh to Traefik; only forge-home-test.open-platform.sh and arm--forge-trevato-apps.open-platform.sh go straight to the forge.
Sign-in: Ingress forge has middlewares mcp-plat-strip and mcp-plat-access-trevato-apps-forge-prod, a forwardAuth to plat-auth.mcp.svc/forward?owner=trevato-apps&repo=forge. /signup and POST /api/v1/users go to forge-signup-gate (a busybox pod on pg-grand).
Child and app hosts: IngressRoutes forge-children (<name>--forge.open-platform.sh), forge-apps (*--forge-trevato-apps.open-platform.sh), forge-old-host; also game.cheffryjames.com and www.cheffryjames.com in the forge's namespace.
Image: pulled from Forgejo's registry on pg-grand (git.open-platform.sh, pull secret forgejo-registry); built by Forgejo Actions.
Release: host timer forge-release-tick (5 min) on pg-grand; also forge-watchdog (1 min), forge-github-mirror (hourly), forge-prune-images (hourly, added 11-12), forge-ssh (socket proxy :2222 → the forge's ssh service).
Control plane: the k3s API on pg-grand. With it gone, running pods keep running but nothing reschedules and Services stop updating.
Backups: the forge's data is copied to a MinIO bucket on pg-grand (app--trevato-apps--forge--prod, 2.3 GB). No off-site copy of the forge's data was found. Only Forgejo's database goes off-box (Backblaze B2, daily).
GitOps: Flux's Kustomization for the forge is suspended; standby-reconcile (every 2 min, a host timer) applies manifests from Forgejo on pg-grand.

Also on pg-grand (not the forge): Forgejo and its database, the platform's mcp/plat-auth, www, CloudNativePG operator with 128 single-instance databases across about 100 tenant namespaces (44 deployments above zero replicas, most woken on demand by sablier), 261 local-path volumes all pinned to that node, MinIO on :9000.

If pg-grand were switched off now, the forge breaks in this order: Traefik (unreachable) → plat-auth (no sign-in) → signup gate → image pulls and releases → control plane → local backups → the host timers.