Public: anyone can read this; the crew can change it.

The forge answers, signs people in, builds and releases with the DigitalOcean node switched off #11686

shipped production is serving it trevatoowner opened this 4h ago masoncrew has it

trevatoowner opened this 4h ago

Why. "forge going down needs to never happen again. we need to sort out the deployment and spin down the digital ocean node." One small rented box must not be able to take the forge down, and should not be needed at all.

What, in the order that removes risk fastest.

Reachable. The forge's address is carried by a tunnel that runs beside the forge (the vxrail), two replicas, pointing straight at the forge's service: no Traefik hop and no node other than the forge's own in the path. Moving the address is a DNS and tunnel-config change with the old path kept as a fallback until the new one has carried traffic for a day.
Sign-in without the platform. The forge authenticates people itself; the platform's forwardAuth edge is removed from in front of it. /healthz and public pages never depend on another service. People who signed in through the platform's edge keep their accounts (map them once; nobody is locked out; a test walks an existing account through).
Builds and releases itself. The release image is built on the forge's own machine (the forge's check and build cells already build software) and stored where the forge's node can pull it with the other node off; old images are pruned on a rule (keep the newest 15 and whatever is live or the known-good rollback) by the release step itself, never by hand. The release tick becomes the forge's own keeper (it already has src/keeper, off until now) or a timer on the forge's machine; nothing about a release touches the DigitalOcean node.
Its own control plane. The vxrail runs the forge without a control plane on another machine (its own k3s server, or no Kubernetes at all for the forge: one process, one data directory, a supervisor). Choose the simpler; write why.
Backups leave the building. The forge's data is backed up, encrypted, to object storage that is not on either machine, and a restore is drilled (the restore_drill tool) before anything is switched off.
Disk can never do this again. On every machine the forge uses: an alarm at 70% and 80% disk to the owner's phone, a hard guard that pauses releases and builds at 80%, and the top five directories by size on /system.

The proof. With the owner's go and at a quiet hour: cordon and stop the DigitalOcean node for thirty minutes. The forge answers, a person signs in, a pull lands, a release rolls, a schedule fires. Paste the probe results. Only after that proof and a restored backup is the node a candidate to be switched off.

Acceptance. The thirty-minute proof passes. The table from the inventory issue shows nothing the forge needs on that node.

Plan · 2 of 11 done · 2h ago

Read the footing inventory (#11685) and the keeper; split the six items into issues done
Releases pause at 80% disk used (keeper cadence rule, this pull) done
Builds pause at 80%: #11799 to do
Disk alarms at 70%/80% and top-five directories: #11800 to do
Release step prunes old images: #11801 to do
Sign-in without the forwardAuth edge: #11802 to do
Tunnel straight to the forge, no Traefik: #11803 (with #8946) to do
Own control plane: decide and write, #11804 to do
Off-site encrypted backup and drilled restore: #11805 to do
Self-swap releases on production: #8941 to do
The thirty-minute proof with pg-grand off: #11806 (the owner's go) to do