Public: anyone can read this; the crew can change it.
Backups leave the platform: one last copy of the 36 GB Forgejo dump, the lanes retired one by one, and a restore that never touches it #9060
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/8949-backups-leave-the-platform-one-last-copy → main 3 commits fork 3 threads · 3 unresolved
4 files
changed +63
−1
docs/PLATFORM.md +1 −1
| @@ -51,7 +51,7 @@ What it did **not** retire, because production does not run it yet: | ||
| 51 | 51 | | 3 | **The pod and its supervisor.** A Deployment on local-path storage, `maxSurge 1`, restarts, a service account for the keeper's Deployment patch. | A box of the forge's own running `bun run src/main.ts` under systemd (`Restart=always`); `release/launch.ts` supervises swaps. Keeper #8374 proved the swap, not the box. | `docs/EXTRACT.md` steps 1–3, `docs/RESTORE.md`, `release/launch.ts`, `good.json`. | A box beside production taking no traffic (#8945). | |
| 52 | 52 | | 4 | **Ingress and TLS.** Cloudflare tunnel and cluster ingress carry `forge.open-platform.sh`, `*--forge.open-platform.sh` children and apps' own domains. | Caddy with a DNS-challenge wildcard and on-demand TLS asking `/__forge/domains/ask`, on the box. | Forge side done: `FORGE_TRUSTED_PROXIES`, `FORGE_CLIENT_IP_HEADER`, domains ask endpoint (`docs/DOMAINS.md`). | Staging name first, cutover runbook with rollback, the owner's go (#8946, after #8945). | |
| 53 | 53 | | 5 | **Identity.** `X-Plat-User` / `X-Plat-Edge-Token` (`src/http/edge.ts`): accounts the edge made have no password. | The forge's own password, passkey and OAuth between forges (#8902). `signin.edge` is the switch; an admin with neither is refused. | Switch exists (`docs/OPERATING.md`); `via: edge` is listed per account. | List edge-only accounts, a passkey for each, then switch off on a preview (#8947). | |
| 54 | | 6 | **Backup lanes.** The `backup` namespace's CronJobs (`FORGE_BACKUP_LANES`), `forgejo-dump` (suspended, 36 GB), the platform bucket's off-machine mirror. | The forge's own snapshot to its bucket and B2 (`docs/SNAPSHOT.md`), drilled weekly (`docs/RECOVERY.md`). | Snapshot, restore command, weekly drill, backup tile (`backup-truth.md`). | Decide the dump with the owner, a line for every lane, a restore with the platform unreachable (#8949, drill on #8945). | | |
| 54 | | 6 | **Backup lanes.** The `backup` namespace's CronJobs (`FORGE_BACKUP_LANES`), `forgejo-dump` (suspended, 36 GB), the platform bucket's off-machine mirror. | The forge's own snapshot to its bucket and B2 (`docs/SNAPSHOT.md`), drilled weekly (`docs/RECOVERY.md`). | Snapshot, restore command, weekly drill, backup tile (`backup-truth.md`). | Decide the dump with the owner, a line for every lane, a restore with the platform unreachable (#8949, plan in [](features/backup-truth.md), _Leaving the platform_; drill on #8945). | | |
| 55 | 55 | | 7 | **Forgejo, where outside people send pulls.** grokbot's #42 to #67, `jmoran/forge-jay` (#8935), tdobbertin-espo's six (#8938) on git.open-platform.sh. | The forge: a real merge for a kept fork (#8935), a URL-to-landed-pull path for an agent (#6359), forge-to-forge origin (`docs/features/forge-origin.md`). | Origins and Bring exist (`docs/ORIGINS.md`). | Tell each person, then disable (not delete) new pulls on the Forgejo repository (#8950; waits for #6359, #8935). | |
| 56 | 56 | | 8 | **Hosted apps.** Apps the platform runs that are not on the forge. The list is in the cluster: **unread**. | Bring (paste a link) or Migrate; GitHub and Vercel go through #6007, not here. | Bring an app whole (#8919). | List with the owner's call per app (#8951). | |
| 57 | 57 | | 9 | **The droplet release tick.** `forge-release-tick` on pg-grand and the Mac's `tick release`. | The keeper (#1). | The keeper's code. | Switched off by the owner on the day #8942 records. | |
docs/RESTORE.md +2
| @@ -55,3 +55,5 @@ differs. It writes only into the directory it is given, which must be empty, | ||
| 55 | 55 | refuses `/data`, and leaves the restored copy in place. |
| 56 | 56 | `test/restore-drill-script.test.ts` runs it against a tiny snapshot and a |
| 57 | 57 | stub bucket. |
| 58 | ||
| 59 | With the platform unreachable (the B2 mirror only): [](features/backup-truth.md), _Leaving the platform_. |
docs/SNAPSHOT.md +2
| @@ -249,3 +249,5 @@ sealed variable and an app's SQLite is snapshotted, restored into an empty | ||
| 249 | 249 | directory with the passphrase and booted there as the same forge; restored |
| 250 | 250 | without it, the repositories come back and the sealed variable is reported by |
| 251 | 251 | name. It also checks that the raw key is in no file of the store. |
| 252 | ||
| 253 | The dump, a line per lane and the platform-free drill: [](features/backup-truth.md), _Leaving the platform_. | |
docs/features/backup-truth.md +58
| @@ -76,3 +76,61 @@ is older than the newest whole snapshot or past the window. | ||
| 76 | 76 | |
| 77 | 77 | Not done: the snapshot still runs in the serving process (the database copy is |
| 78 | 78 | in a worker). Tests: `test/snapshot-resume.test.ts`. |
| 79 | ||
| 80 | ## Leaving the platform (#8949) | |
| 81 | ||
| 82 | Nothing here turns anything off. Each step below waits for the owner present; | |
| 83 | the forge and the backup tile stay up and green throughout. | |
| 84 | ||
| 85 | ### The Forgejo dump (`forgejo-dump`, suspended, 36 GB) | |
| 86 | ||
| 87 | **Recommendation, for the owner to confirm in the issue:** keep **one last | |
| 88 | copy**, and let the migrate flow make it redundant for what it covers. | |
| 89 | ||
| 90 | 1. Every Forgejo repository the forge should keep comes in through Migrate or | |
| 91 | Bring (`docs/ORIGINS.md`); once in, the forge's snapshot carries its bundle. | |
| 92 | 2. The dump is also the only copy of what Migrate does not read (issues, pull | |
| 93 | history, users, LFS, packages, Actions secrets on Forgejo). So it is not | |
| 94 | deleted on the strength of step 1. | |
| 95 | 3. One copy goes to **B2** (a prefix of its own, outside the nightly's | |
| 96 | retention). The B2 dump prefix is not a lane, so it is not in | |
| 97 | `FORGE_BACKUP_LANES` and the tile does not read it. A second copy goes | |
outdated
Garbled: rewrite as 'the B2 dump prefix is not a lane, so it is not in FORGE_BACKUP_LANES and the tile does not read it'. |
||
| 98 | to the **owner's disk**; its SHA-256 is written in this issue. | |
| 99 | 4. Only after the owner has opened the copy (restored a repository from it) is | |
| 100 | the CronJob and its volume retired. | |
| 101 | ||
| 102 | Owner's answer: _pending_. Until it is written, the dump stays where it is. | |
| 103 | ||
| 104 | ### A line per lane | |
| 105 | ||
| 106 | The lane list lives in the forge's configuration, never in the repository, so | |
| 107 | the names are filled in from the Resilience dialog (`verdict.lanes` in the MCP | |
| 108 | `backups` tool), one row each. Every row says what it backs up and one of: | |
| 109 | **read after spin-down** (who, how) or **retired** (on what date, by whom). | |
| 110 | ||
| 111 | | Lane | Backs up | After spin-down | | |
| 112 | | --- | --- | --- | | |
| 113 | | `primary` | the forge's snapshot, platform bucket | the app's bucket; read by the in-process check | | |
| 114 | | `mirror` | the same snapshot, B2 | B2; read by the in-process check and `scripts/backup-probe.ts` | | |
| 115 | | each `backup/<name>` | _fill in from the dialog_ | _read after spin-down, or retired_ | | |
| 116 | ||
| 117 | A lane with no reader after spin-down is retired, and removed from | |
| 118 | `FORGE_BACKUP_LANES` in the same step, so the tile does not go red for a lane | |
| 119 | nobody keeps. | |
| 120 | ||
| 121 | ### Restore drill with the platform unreachable | |
| 122 | ||
| 123 | On the box (`scripts/box-install.sh`, `docs/EXTRACT.md`), with the platform's | |
| 124 | network blocked (egress allowed to B2 only, e.g. a firewall rule on the box): | |
| 125 | ||
outdated
Use |
||
| 126 | 1. Point `S3_*` at the **B2 mirror** credentials, not the platform bucket. | |
| 127 | 2. With the B2 `S3_*` values and the passphrase exported, as the `forge` | |
| 128 | user, run `bun scripts/restore-drill.ts <empty-data-dir>` | |
| 129 | ([](../RESTORE.md), _The drill on the box_). Exit 0 is the pass | |
| 130 | condition: it restored, booted, answered `/` and the counts matched. Exit | |
| 131 | 1 is a failed drill, whatever the step it names. | |
| 132 | 3. Check that the backup tile reads green once the check has run. | |
| 133 | 4. Record the date, snapshot name and counts in this issue. | |
| 134 | ||
| 135 | Not done here: the drill itself, the dump decision, and the lane rows. Each | |
| 136 | needs the box or the owner; this section is what they follow. |
new above
offline
Make this a markdown link, as RESTORE.md does.