Public: anyone can read this; the crew can change it.
docs/RECOVERY.md is the one page a person reads at 3 a.m., and a test keeps it true to src/snapshot #1497
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/1480-docs-recovery-md-is-the-one-page-a-perso → main 2 commits fork 5 threads · 5 unresolved
3 files
changed +137
−0
docs/RECOVERY.md added +69
| @@ -0,0 +1,69 @@ | ||
| 1 | # Recovery — read this at 3 a.m. | |
| 2 | ||
| 3 | Print it. Keep it somewhere that is not the forge. `test/recovery-doc.test.ts` | |
| 4 | keeps it true to `src/snapshot/`. | |
| 5 | ||
| 6 | ## Where the copies are | |
| 7 | ||
| 8 | Snapshots are objects in the forge's bucket (`S3_BUCKET`, at `S3_ENDPOINT`, | |
| 9 | region `S3_REGION`) under the prefix `snapshots/<name>/`, one directory per | |
| 10 | snapshot, named like `2026-09-22T08-00-00Z`. The restore command accepts `latest` for | |
| 11 | the newest; no object is called that. | |
| 12 | ||
| 13 | ## What a restore needs | |
| 14 | ||
| 15 | Export these, with the values the forge ran under: | |
| 16 | ||
| 17 | S3_ENDPOINT S3_BUCKET S3_REGION S3_ACCESS_KEY S3_SECRET_KEY | |
| 18 | ||
| 19 | and `FORGE_SNAPSHOT_PASSPHRASE`, which is in the owner's safe and nowhere | |
| 20 | else. Without it the restore still works, but the secret key stays behind and | |
| 21 | every sealed value is set again by hand. A wrong one refuses before writing. | |
| 22 | A machine with `git`, `bun` and a clone of the forge (`bun install`). | |
| 23 | ||
| 24 | ## Route 1 — the restore command | |
| 25 | ||
| 26 | bun run src/snapshot/restore.ts <snapshot-name|latest> <data-dir> | |
| 27 | ||
| 28 | `<data-dir>` must be empty or new; the command refuses otherwise. It prints | |
| 29 | the files and repositories it brought back. | |
| 30 | ||
| 31 | Then boot the forge on that directory, on a side port, off the public one: | |
| 32 | ||
| 33 | FORGE_DATA=<data-dir> PORT=3999 FORGE_SSH_PORT=0 bun run src/main.ts | |
| 34 | ||
| 35 | Look at, in order: | |
| 36 | ||
| 37 | 1. `curl localhost:3999/api/v1/health` answers ok. | |
| 38 | 2. The home page, signed in as the owner: the repositories and the board. | |
| 39 | 3. One repository's issues: the latest ones you remember are there. | |
| 40 | ||
| 41 | If all three hold, point the real service at `<data-dir>`. | |
| 42 | ||
| 43 | ## Route 2 — the objects by hand | |
| 44 | ||
| 45 | Use it when the restore command fails or only one file is wanted. Every | |
| 46 | object is plain, under `snapshots/<name>/`; fetch with any S3 client: | |
| 47 | ||
| 48 | manifest.json what is there | |
| 49 | forge.db the database | |
| 50 | secret.key.wrapped the key, under the passphrase | |
outdated
No openssl/pbkdf2 tarball exists; wrap.ts uses scrypt + AES-GCM on the secret key only. This command cannot work. secret.key.wrapped is listed as always present. Say it exists only when the passphrase was set. |
||
| 51 | ssh/... the SSH host key | |
| 52 | repos/<owner>/<repo>.bundle one git bundle per repository | |
| 53 | apps/<id>/production/data/... each app's data | |
| 54 | runs/... run logs | |
| 55 | ||
| 56 | Put `forge.db` and `ssh/` in a new data directory; turn each bundle back into | |
| 57 | a repository with `git clone --mirror <file> <data-dir>/repos/<owner>/<repo>.git` | |
| 58 | (the layout `src/git/repos.ts` reads). Boot as above. The key is wrapped with | |
| 59 | scrypt and AES-256-GCM (`src/snapshot/wrap.ts`): only `restore.ts` opens it, | |
| 60 | so by hand the sealed values are set again. | |
| 61 | ||
| 62 | ## What a snapshot does NOT hold | |
| 63 | ||
| 64 | - Sealed values, unless the passphrase was set: they stay sealed and are set | |
| 65 | again by hand (the restore lists them by name). | |
| 66 | - App variables, which are sealed the same way. | |
| 67 | - The crew's clones: working copies that agents make again from the | |
| 68 | repositories. | |
| 69 | - Anything outside the data directory: the code, the machine, the bun. |
scripts/gate.ts +1
| @@ -45,6 +45,7 @@ export const GATE: readonly string[] = [ | ||
| 45 | 45 | "test/event-order.test.ts", |
| 46 | 46 | "test/refresh-meta.test.ts", |
| 47 | 47 | "test/restore-card.test.ts", |
| 48 | "test/recovery-doc.test.ts", | |
| 48 | 49 | "test/returns.test.ts", |
| 49 | 50 | "test/review-route.test.ts", |
| 50 | 51 | "test/review-lines.test.ts", |
test/recovery-doc.test.ts added +67
| @@ -0,0 +1,67 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { readFileSync } from "node:fs"; | |
| 3 | import { join } from "node:path"; | |
| 4 | ||
| 5 | const root = join(import.meta.dir, ".."); | |
| 6 | const read = (p: string) => readFileSync(join(root, p), "utf8"); | |
| 7 | const doc = read("docs/RECOVERY.md"); | |
| 8 | ||
| 9 | test("the doc names every variable the store reads", () => { | |
| 10 | const vars = new Set( | |
| 11 | [...read("src/snapshot/store.ts").matchAll(/env\["(S3_[A-Z_]+)"\]/g)].map( | |
| 12 | (m) => m[1]!, | |
| 13 | ), | |
| 14 | ); | |
| 15 | expect(vars.size).toBeGreaterThanOrEqual(5); | |
| 16 | for (const v of vars) expect(doc).toContain(v); | |
| 17 | expect(doc).toContain("FORGE_SNAPSHOT_PASSPHRASE"); | |
| 18 | }); | |
| 19 | ||
| 20 | test("the doc's restore command is the one restore.ts prints", () => { | |
| 21 | const printed = read("src/snapshot/restore.ts").match( | |
| 22 | /"usage: (bun run src\/snapshot\/restore\.ts [^"]+)"/, | |
| 23 | )?.[1]; | |
| 24 | expect(printed).toBeDefined(); | |
| 25 | expect(doc).toContain(` ${printed}\n`); | |
| 26 | }); | |
| 27 | ||
| 28 | test("the doc's prefix and object names are the ones snapshot.ts writes", () => { | |
| 29 | const prefix = read("src/snapshot/snapshot.ts").match( | |
outdated
These assertions check the doc against itself, not against src/snapshot. Derive the prefix from SNAPSHOT_PREFIX and drop the invented strings. |
||
| 30 | /SNAPSHOT_PREFIX = "([^"]+)"/, | |
| 31 | )?.[1]; | |
| 32 | expect(prefix).toBe("snapshots/"); | |
| 33 | expect(doc).toContain(`${prefix}<name>/`); | |
| 34 | const wrapped = read("src/snapshot/wrap.ts").match( | |
| 35 | /WRAPPED_KEY = "([^"]+)"/, | |
| 36 | )?.[1]; | |
| 37 | expect(wrapped).toBeDefined(); | |
| 38 | expect(doc).toContain(wrapped!); | |
These are bare substring checks on snapshot.ts and pass whatever the layout is. Run takeSnapshot against a fixture store and compare the doc's listed paths to the keys written. |
||
| 39 | // Each layout path the doc lists is one snapshot.ts puts, or restore.ts reads. | |
| 40 | const snap = read("src/snapshot/snapshot.ts"); | |
| 41 | for (const p of ["forge.db", "ssh/", "repos/", "apps/", "runs/", "manifest.json"]) | |
| 42 | expect(snap).toContain(p); | |
| 43 | expect(snap).toContain(".bundle"); | |
| 44 | expect(doc).toContain("<repo>.bundle"); | |
| 45 | expect(doc).not.toContain("manual/"); | |
| 46 | expect(doc).not.toContain("openssl"); | |
| 47 | }); | |
| 48 | ||
| 49 | test("`latest` is a name restore.ts accepts, not an object", () => { | |
| 50 | expect(read("src/snapshot/restore.ts")).toContain('name === "latest"'); | |
slice(0, 1200) ties the test to the length of the header comment; a longer header breaks it with no change in meaning. Match the whole file, or a named export. |
||
| 51 | expect(doc).toContain("`latest`"); | |
| 52 | }); | |
| 53 | ||
| 54 | test("what a snapshot lacks is what snapshot.ts leaves out", () => { | |
| 55 | const head = read("src/snapshot/snapshot.ts").slice(0, 1200); | |
| 56 | expect(head).toContain("clones"); | |
| 57 | expect(doc).toContain("clones"); | |
| 58 | // Sealed values and app variables live sealed in the database. | |
| 59 | expect(read("src/core/app_env.ts")).toContain("sealed at rest"); | |
| 60 | expect(read("src/snapshot/restore.ts")).toContain("sealedMissing"); | |
| 61 | expect(doc).toContain("App variables, which are sealed"); | |
| 62 | expect(doc).toContain("/api/v1/health"); | |
| 63 | }); | |
| 64 | ||
| 65 | test("the doc fits one printed side", () => { | |
| 66 | expect(doc.split("\n").length).toBeLessThanOrEqual(70); | |
| 67 | }); |
new above
offline
No code creates manual/<stamp>/ in the bucket. Remove this route or point it at what snapshot.ts really writes.