8389707
@@ -29,12 +29,16 @@
turned off, with what each move costs and what it gives up.
   recipe with no network story. Install the four binaries; make
   `/var/lib/forge` and a user that owns it.
2. **Restore, don't migrate.** The nightly snapshot holds everything that
   cannot be rebuilt — the database, `secret.key`, every repository, every
   app's production data, the crew's logs. On the new box, with the same
   `S3_*` variables:
   `bun run src/snapshot/restore.ts <snapshot> /var/lib/forge`
   The restore sets the key's mode; sessions, tokens, sealed variables and
   sealed subscriptions all keep working because the key travelled.
   cannot be rebuilt — the database, every repository, every app's
   production data, the crew's logs — and the forge's key only wrapped under
   the operator's passphrase. On the new box, with the same `S3_*` variables
   and the passphrase the old forge ran with:
   `FORGE_SNAPSHOT_PASSPHRASE=… bun run src/snapshot/restore.ts <snapshot> /var/lib/forge`
   The restore unwraps the key and sets its mode; sealed variables, secrets
   and sealed subscriptions keep working. The passphrase lives in the forge's
   environment and with the operator — keep a copy outside the bucket. Without
   it repositories and issues restore, and every sealed value is listed by
   name as lost and must be set again (`docs/SNAPSHOT.md`).
3. **Run it as a service.** A systemd unit: `ExecStart=bun run src/main.ts`
   from a checkout at `/opt/forge`, `FORGE_URL=https://forge.example`,
   `FORGE_APP_DOMAIN=forge.example`, `FORGE_DATA=/var/lib/forge`, the crew
@@ -9,9 +9,13 @@
empty data directory. `src/snapshot/`.
- `forge.db`, copied consistently while the forge runs (`VACUUM INTO`,
  never a raw copy of a file in WAL mode).
- `secret.key`, which the database's sealed values need, and `ssh/`, the
  host key that is the forge's identity to every clone. Both come back with
  owner-only modes; the forge refuses a key anyone else can read.
- The key the database's sealed values need, **only wrapped**:
  `secret.key.wrapped`, AES-256-GCM under a key scrypt-derived from the
  operator's passphrase. The raw `secret.key` is never written to the store,
  so whoever reads the bucket holds the data without the key. See
  [The key](#the-key).
- `ssh/`, the host key that is the forge's identity to every clone. It comes
  back with an owner-only mode; the forge refuses a key anyone else can read.
- Every repository as one git bundle of every ref (`repos/<owner>/<name>.bundle`).
  An empty repository has nothing to bundle and is skipped, not failed.
- Every app's production data (`apps/<id>/production/data/…`). SQLite files
@@ -20,11 +24,33 @@
empty data directory. `src/snapshot/`.
  stay out.
- The crew's run logs (`runs/`).
- `manifest.json`: the name, when, the forge's version, counts, every file
  with its size. A restore checks each size.
  with its size, and `key`: `wrapped` or `none` (older snapshots: `raw`). A
  restore checks each size.
What stays out is what a boot or a deploy rebuilds: clones, checks,
pictures, previews.
## The key
The wrapping secret is `FORGE_SNAPSHOT_PASSPHRASE`, in the forge's
environment. It is not in the bucket, the repository or the database; the
operator holds it elsewhere too (a password manager, a sealed envelope).
- **Set**: every snapshot carries `secret.key.wrapped`, and the Backups page
  says the last one did.
- **Unset**: no key travels, and the Backups page says so. Snapshots still
  restore.
- **Lost**: the wrapped key cannot be opened by anyone, including us.
  Repositories, issues, app data and everything unsealed restore; every
  sealed value (forge, org and repo secrets, app variables, origin tokens,
  subscription tokens) is lost and is set again by hand. Choose the
  passphrase knowing that. A passphrase changed on the forge applies to the
  next snapshot; older snapshots stay under the one they were taken with.
Snapshots from before this change hold the raw `secret.key`; they still
restore with the key, and the Backups page tells you to take a new one.
Delete the old ones from the bucket if the bucket is not trusted.
## Where
- On the platform, the app's bucket: the `S3_*` variables the platform puts
@@ -41,9 +67,17 @@
The store is `Bun.S3Client` or a directory behind one small interface
```sh
# same S3_* as the forge runs under, or FORGE_SNAPSHOT_DIR
bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z /path/to/empty/data
FORGE_SNAPSHOT_PASSPHRASE=… \
  bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z /path/to/empty/data
```
With the passphrase the key is unwrapped into `secret.key` (mode 0600) and
sealed values open. A wrong passphrase refuses the restore before anything is
written. With none, the key is left out, repositories and the rest restore,
and the command lists every sealed value by name (`NAME (app slug)`,
`NAME (scope secret)`) that must be set again; the restored forge makes a new
key on first boot.
It refuses a directory that is not empty and a snapshot it cannot find or
that is missing a file. Bundles become bare repositories with the forge's
hooks installed; the mirror's memory of the bundle as an origin is removed.
@@ -52,4 +86,6 @@
it has moved on, reconciles the index and redeploys what runs.
`test/snapshot.test.ts` is the proof: a running forge with history, a
sealed variable and an app's SQLite is snapshotted, restored into an empty
directory, and booted there as the same forge.
directory with the passphrase and booted there as the same forge; restored
without it, the repositories come back and the sealed variable is reported by
name. It also checks that the raw key is in no file of the store.
@@ -166,10 +166,18 @@
const VERSION =
  ((await import("../package.json")) as { version?: string }).version ?? "0";
const snapshotDir = process.env["FORGE_SNAPSHOT_DIR"]?.trim();
const s3 = s3FromEnv();
const snapshotPassphrase =
  process.env["FORGE_SNAPSHOT_PASSPHRASE"] || undefined;
const snapshots = snapshotDir
  ? startSnapshots(app, dirStore(snapshotDir), { version: VERSION })
  ? startSnapshots(app, dirStore(snapshotDir), {
      version: VERSION,
      passphrase: snapshotPassphrase,
    })
  : s3
    ? startSnapshots(app, s3Store(s3), { version: VERSION })
    ? startSnapshots(app, s3Store(s3), {
        version: VERSION,
        passphrase: snapshotPassphrase,
      })
    : undefined;
const crew = appsOn
  ? await startCrew(app, {
@@ -12,6 +12,11 @@
 *
 * with the same `S3_*` variables the forge runs under, or
 * `FORGE_SNAPSHOT_DIR` for a directory store.
 *
 * The key comes back only from its wrapped form and the operator's
 * passphrase (`FORGE_SNAPSHOT_PASSPHRASE`). A wrong passphrase refuses the
 * restore; none leaves the key out, and every sealed value that will not
 * open is reported by name.
 */
import { chmodSync, existsSync, mkdirSync, readdirSync } from "node:fs";
@@ -19,19 +24,50 @@
import { dirname, join } from "node:path";
import { runGit } from "../git/env.ts";
import { installHooks } from "../git/hooks.ts";
import { repoDir } from "../git/repos.ts";
import { SNAPSHOT_PREFIX, type Manifest } from "./snapshot.ts";
import { keyOf, SNAPSHOT_PREFIX, type Manifest } from "./snapshot.ts";
import { unwrapKey, WRAPPED_KEY } from "./wrap.ts";
import { dirStore, s3FromEnv, s3Store, type Store } from "./store.ts";
export interface Restored {
  readonly manifest: Manifest;
  readonly files: number;
  readonly repos: number;
  /** `restored` from the wrapped key, `raw` from an old snapshot, `missing` otherwise. */
  readonly key: "restored" | "raw" | "missing";
  /** Sealed values that will not open without the key, by name. */
  readonly sealedMissing: readonly string[];
}
/** Names of every sealed value in a restored database; never the values. */
async function sealedNames(dbFile: string): Promise<string[]> {
  const { Database } = await import("bun:sqlite");
  const db = new Database(dbFile, { readonly: true });
  try {
    const secrets = db
      .query<{ name: string; scope: string }, []>(
        "SELECT name, scope FROM secret ORDER BY scope, name",
      )
      .all()
      .map((r) => `${r.name} (${r.scope} secret)`);
    const env = db
      .query<{ name: string; slug: string }, []>(
        "SELECT e.name AS name, a.slug AS slug FROM app_env e JOIN app a ON a.id = e.app_id ORDER BY a.slug, e.name",
      )
      .all()
      .map((r) => `${r.name} (app ${r.slug})`);
    return [...secrets, ...env];
  } catch {
    return [];
  } finally {
    db.close();
  }
}
export async function restoreSnapshot(
  store: Store,
  name: string,
  dataDir: string,
  options: { readonly passphrase?: string | undefined } = {},
): Promise<Restored> {
  if (!/^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}-[0-9]{2}-[0-9]{2}Z$/.test(name))
    throw new Error(`not a snapshot name: ${name}`);
@@ -44,6 +80,19 @@
export async function restoreSnapshot(
  if (raw === undefined)
    throw new Error(`no snapshot ${name} in ${store.where}`);
  const manifest = JSON.parse(new TextDecoder().decode(raw)) as Manifest;
  const wrapped = manifest.files.some((f) => f.path === WRAPPED_KEY)
    ? await store.get(prefix + WRAPPED_KEY)
    : undefined;
  let key: Buffer | undefined;
  if (wrapped !== undefined && options.passphrase) {
    key = unwrapKey(wrapped, options.passphrase);
    // Better to stop here than to restore a forge that silently cannot
    // open its values because of a mistyped passphrase.
    if (key === undefined)
      throw new Error(
        "the passphrase does not open this snapshot's wrapped key; check FORGE_SNAPSHOT_PASSPHRASE, or restore without it to leave sealed values behind",
      );
  }
  mkdirSync(dataDir, { recursive: true });
  let files = 0;
@@ -57,6 +106,7 @@
export async function restoreSnapshot(
      throw new Error(
        `snapshot ${name}: ${entry.path} is ${body.byteLength} bytes, not ${entry.bytes}`,
      );
    if (entry.path === WRAPPED_KEY) continue;
    const bundle = entry.path.match(/^repos\/([^/]+)\/([^/]+)\.bundle$/);
    // A bundle is not a file of the forge's; it becomes a repository below.
    const target = bundle
@@ -89,7 +139,18 @@
export async function restoreSnapshot(
  }
  const { rmSync } = await import("node:fs");
  rmSync(join(dataDir, "restore-bundles"), { recursive: true, force: true });
  return { manifest, files, repos };
  if (key !== undefined) {
    const target = join(dataDir, "secret.key");
    await Bun.write(target, key);
    chmodSync(target, 0o600);
    files++;
  }
  const kind = keyOf(manifest);
  const have =
    key !== undefined ? "restored" : kind === "raw" ? "raw" : "missing";
  const sealedMissing =
    have === "missing" ? await sealedNames(join(dataDir, "forge.db")) : [];
  return { manifest, files, repos, key: have, sealedMissing };
}
if (import.meta.main) {
@@ -111,8 +172,16 @@
if (import.meta.main) {
    );
    process.exit(2);
  }
  const done = await restoreSnapshot(store, name, dataDir);
  const done = await restoreSnapshot(store, name, dataDir, {
    passphrase: process.env["FORGE_SNAPSHOT_PASSPHRASE"] || undefined,
  });
  console.log(
    `restored ${done.manifest.name} (forge ${done.manifest.version}): ${done.files} files, ${done.repos} repositories into ${dataDir}`,
  );
  if (done.key === "missing") {
    console.log(
      `the key was not restored: ${done.sealedMissing.length} sealed value(s) will not open and must be set again:`,
    );
    for (const n of done.sealedMissing) console.log(`  ${n}`);
  }
}
@@ -26,6 +26,8 @@
const NIGHTLY_HOUR_UTC = 8;
export interface SnapshotsHandle {
  readonly store: Store;
  /** Whether snapshots carry the key, wrapped under a passphrase. */
  readonly wraps: boolean;
  /** Take one now; waits for any in flight. */
  readonly now: () => Promise<Manifest>;
  /** The newest first, from the store. */
@@ -49,7 +51,11 @@
export function untilNightly(now = Date.now()): number {
export function startSnapshots(
  app: App,
  store: Store,
  options: { readonly version: string; readonly nightly?: boolean } = {
  options: {
    readonly version: string;
    readonly nightly?: boolean;
    readonly passphrase?: string | undefined;
  } = {
    version: "0",
  },
): SnapshotsHandle {
@@ -65,6 +71,7 @@
export function startSnapshots(
          dataDir: app.config.dataDir,
          reposRoot: app.reposRoot,
          version: options.version,
          passphrase: options.passphrase,
        },
        store,
      );
@@ -98,6 +105,7 @@
export function startSnapshots(
  const handle: SnapshotsHandle = {
    store,
    wraps: !!options.passphrase,
    now: take,
    list: () => listSnapshots(store),
    last: () => last,
@@ -109,6 +117,10 @@
export function startSnapshots(
    },
  };
  handles.set(app, handle);
  log.info("snapshots on", { where: store.where, keep: KEEP });
  log.info("snapshots on", {
    where: store.where,
    keep: KEEP,
    key: options.passphrase ? "wrapped" : "none",
  });
  return handle;
}
@@ -27,6 +27,7 @@
import { runGit } from "../git/env.ts";
import { repoDir } from "../git/repos.ts";
import { createLog } from "../kernel/log.ts";
import type { Store } from "./store.ts";
import { unwrapKey, wrapKey, WRAPPED_KEY } from "./wrap.ts";
const log = createLog("snapshot");
@@ -48,13 +49,25 @@
export interface Manifest {
  readonly apps: number;
  readonly files: readonly SnapshotFile[];
  readonly bytes: number;
  /**
   * What the snapshot carries of the forge's key: `wrapped` under the
   * operator's passphrase, `none` (sealed values are lost on restore), or
   * `raw` in snapshots from before the key was wrapped.
   */
  readonly key?: "wrapped" | "none" | "raw";
}
/** What a manifest says of the key, including manifests older than the field. */
export const keyOf = (m: Manifest): "wrapped" | "none" | "raw" =>
  m.key ?? (m.files.some((f) => f.path === "secret.key") ? "raw" : "none");
export interface Snapshotter {
  readonly db: Database;
  readonly dataDir: string;
  readonly reposRoot: string;
  readonly version: string;
  /** The operator's wrapping secret; without it the key stays out. */
  readonly passphrase?: string | undefined;
}
/** `2026-09-22T09-00-00Z`: sortable, and a name a bucket allows. */
@@ -130,7 +143,17 @@
export async function takeSnapshot(
    const dbCopy = join(scratch, "forge.db");
    forge.db.run("VACUUM INTO ?1", [dbCopy]);
    await putFile("forge.db", dbCopy);
    await putFile("secret.key", join(forge.dataDir, "secret.key"));
    // The key travels only wrapped. Without a passphrase it stays home.
    let key: Manifest["key"] = "none";
    const keyFile = Bun.file(join(forge.dataDir, "secret.key"));
    if (forge.passphrase && (await keyFile.exists())) {
      const raw = new Uint8Array(await keyFile.arrayBuffer());
      const wrapped = wrapKey(raw, forge.passphrase);
      if (unwrapKey(wrapped, forge.passphrase) === undefined)
        throw new Error("the wrapped key does not open; snapshot abandoned");
      await put(WRAPPED_KEY, wrapped);
      key = "wrapped";
    }
    for (const f of filesUnder(join(forge.dataDir, "ssh")))
      await putFile(`ssh/${f}`, join(forge.dataDir, "ssh", f));
@@ -204,6 +227,7 @@
export async function takeSnapshot(
      apps: kept,
      files,
      bytes: files.reduce((n, f) => n + f.bytes, 0),
      key,
    };
    await store.put(
      `${prefix}manifest.json`,
@@ -215,6 +239,7 @@
export async function takeSnapshot(
      repos: bundled,
      apps: kept,
      files: files.length,
      key,
      mb: (manifest.bytes / 1_048_576).toFixed(1),
    });
    return manifest;
@@ -0,0 +1,80 @@
/**
 * The forge's key, wrapped for a bucket: AES-256-GCM under a key stretched
 * from an operator's passphrase with scrypt. The passphrase lives in the
 * forge's environment (`FORGE_SNAPSHOT_PASSPHRASE`) and with the operator,
 * never in the bucket, so whoever reads the bucket holds the data without
 * the key that opens its sealed values.
 *
 * The wrapped form is one small JSON file:
 * `{"v":1,"kdf":"scrypt","salt":…,"iv":…,"ct":…}`, base64url. A wrong
 * passphrase and a tampered file both fail the tag check, and say so alike.
 */
import {
  createCipheriv,
  createDecipheriv,
  randomBytes,
  scryptSync,
} from "node:crypto";
export const WRAPPED_KEY = "secret.key.wrapped";
const SALT_BYTES = 16;
const IV_BYTES = 12;
const TAG_BYTES = 16;
/** Interactive-grade cost: one derivation per snapshot, once a night. */
const SCRYPT = { N: 1 << 15, r: 8, p: 1, maxmem: 64 * 1024 * 1024 } as const;
const derive = (passphrase: string, salt: Buffer): Buffer =>
  scryptSync(passphrase, salt, 32, SCRYPT);
export function wrapKey(key: Uint8Array, passphrase: string): string {
  const salt = randomBytes(SALT_BYTES);
  const iv = randomBytes(IV_BYTES);
  const cipher = createCipheriv("aes-256-gcm", derive(passphrase, salt), iv, {
    authTagLength: TAG_BYTES,
  });
  const ct = Buffer.concat([
    cipher.update(key),
    cipher.final(),
    cipher.getAuthTag(),
  ]);
  return JSON.stringify({
    v: 1,
    kdf: "scrypt",
    salt: salt.toString("base64url"),
    iv: iv.toString("base64url"),
    ct: ct.toString("base64url"),
  });
}
/** The key, or undefined when the passphrase is wrong or the file is not ours. */
export function unwrapKey(
  wrapped: Uint8Array | string,
  passphrase: string,
): Buffer | undefined {
  try {
    const text =
      typeof wrapped === "string" ? wrapped : new TextDecoder().decode(wrapped);
    const w = JSON.parse(text) as Record<string, unknown>;
    if (w["v"] !== 1 || w["kdf"] !== "scrypt") return undefined;
    const salt = Buffer.from(String(w["salt"]), "base64url");
    const iv = Buffer.from(String(w["iv"]), "base64url");
    const ct = Buffer.from(String(w["ct"]), "base64url");
    if (salt.length !== SALT_BYTES || iv.length !== IV_BYTES) return undefined;
    if (ct.length <= TAG_BYTES) return undefined;
    const decipher = createDecipheriv(
      "aes-256-gcm",
      derive(passphrase, salt),
      iv,
      { authTagLength: TAG_BYTES },
    );
    decipher.setAuthTag(ct.subarray(ct.length - TAG_BYTES));
    return Buffer.concat([
      decipher.update(ct.subarray(0, ct.length - TAG_BYTES)),
      decipher.final(),
    ]);
  } catch {
    return undefined;
  }
}
@@ -8,6 +8,7 @@
 * and a stopped forge, and a person at a terminal is the one who has both.
 */
import { keyOf } from "../../snapshot/snapshot.ts";
import { snapshotsOf } from "../../snapshot/service.ts";
import type { Ctx } from "../../http/context.ts";
import { notFoundPage, settingsLayout, signInFirst } from "../chrome.ts";
@@ -36,6 +37,46 @@
function shell(ctx: Ctx, body: Html): Response {
  });
}
/**
 * Whether the key travels, said before anyone relies on it. The passphrase
 * is the only way back to sealed values from a bucket; lose it and they
 * are lost.
 */
function keyNote(
  wraps: boolean,
  newest:
    | Awaited<
        ReturnType<NonNullable<ReturnType<typeof snapshotsOf>>["list"]>
      >[number]
    | undefined,
): Html {
  const carried = newest === undefined ? undefined : keyOf(newest);
  const last =
    carried === undefined
      ? ""
      : carried === "wrapped"
        ? " The last snapshot carried a wrapped key."
        : carried === "raw"
          ? " The last snapshot carried the key unwrapped, as older snapshots did; take a new one."
          : " The last snapshot carried no key.";
  return wraps
    ? html`<p>
        ${icon("check")} The key is wrapped under
        <code>FORGE_SNAPSHOT_PASSPHRASE</code> and travels with each
        snapshot.${last} Keep that passphrase somewhere other than this bucket:
        without it a restore brings back repositories and issues, but every
        sealed value (secrets, app variables, origin tokens) is lost and must be
        set again.
      </p>`
    : html`<p class="backups-error">
        ${icon("alert")} Snapshots carry no key: the bucket never holds key and
        data together, and a restore will bring back repositories and issues but
        not sealed values, which must be set again.${last} To carry the key
        wrapped, set <code>FORGE_SNAPSHOT_PASSPHRASE</code> and restart — and
        keep the passphrase elsewhere, because losing it loses those values.
      </p>`;
}
export async function backupsPage(ctx: Ctx): Promise<Response> {
  const denied = requireAdmin(ctx);
  if (denied) return denied;
@@ -61,8 +102,7 @@
export async function backupsPage(ctx: Ctx): Promise<Response> {
  }
  const restore = snapshots[0]
    ? html`<pre class="backups-restore">
bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre
      >`
bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre>`
    : null;
  return shell(
    ctx,
@@ -73,6 +113,7 @@
bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre
          crew's logs — to <code>${svc.store.where}</code>, and keeps the newest
          fourteen.
        </p>
        ${keyNote(svc.wraps, snapshots[0])}
        ${when(
          last.error,
          (error) =>
@@ -100,47 +141,51 @@
bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre
          <h2>Where and when</h2>`,
      },
    )}
    ${snapshots.length === 0
      ? empty({
          icon: "clock",
          title: "None yet",
          note: "The first is taken tonight, or now with the button above.",
        })
      : card(
          html`<ul class="list">
            ${snapshots.map(
              (m) =>
                html`<li>
                  <div class="row">
                    <span class="row-lead">${icon("download")}</span>
                    <span class="row-main">
                      <span class="row-title">${m.name}</span>
                      <span class="row-meta"
                        >${time(m.finishedAt)} · forge ${m.version} ·
                        ${m.repos === 1
                          ? "one repository"
                          : `${m.repos} repositories`},
                        ${m.apps === 1 ? "one app" : `${m.apps} apps`},
                        ${m.files.length} files</span
                      >
                    </span>
                    <span class="row-trail muted">${fileSize(m.bytes)}</span>
                  </div>
                </li>`,
            )}
          </ul>`,
          {
            pad: false,
            head: html`${icon("history")}
              <h2>Snapshots</h2>`,
            foot: html`<span class="muted small"
                >To restore, stop a forge, point it at an empty data directory,
                and run this with the same S3_* variables or
                FORGE_SNAPSHOT_DIR:</span
              >
              ${restore}`,
          },
        )}`,
    ${
      snapshots.length === 0
        ? empty({
            icon: "clock",
            title: "None yet",
            note: "The first is taken tonight, or now with the button above.",
          })
        : card(
            html`<ul class="list">
              ${snapshots.map(
                (m) =>
                  html`<li>
                    <div class="row">
                      <span class="row-lead">${icon("download")}</span>
                      <span class="row-main">
                        <span class="row-title">${m.name}</span>
                        <span class="row-meta"
                          >${time(m.finishedAt)} · forge ${m.version} ·
                          ${
                            m.repos === 1
                              ? "one repository"
                              : `${m.repos} repositories`
                          },
                          ${m.apps === 1 ? "one app" : `${m.apps} apps`},
                          ${m.files.length} files</span
                        >
                      </span>
                      <span class="row-trail muted">${fileSize(m.bytes)}</span>
                    </div>
                  </li>`,
              )}
            </ul>`,
            {
              pad: false,
              head: html`${icon("history")}
                <h2>Snapshots</h2>`,
              foot: html`<span class="muted small"
                  >To restore, stop a forge, point it at an empty data
                  directory, and run this with the same S3_* variables or
                  FORGE_SNAPSHOT_DIR:</span
                >
                ${restore}`,
            },
          )
    }`,
  );
}
@@ -7,7 +7,13 @@
 * restore refuses what it must.
 */
import { afterAll, beforeAll, describe, expect, test } from "bun:test";
import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import {
  existsSync,
  mkdirSync,
  mkdtempSync,
  readFileSync,
  writeFileSync,
} from "node:fs";
import { rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -30,8 +36,11 @@
import {
  snapshotName,
  takeSnapshot,
} from "../src/snapshot/snapshot.ts";
import { unwrapKey, wrapKey } from "../src/snapshot/wrap.ts";
import { dirStore } from "../src/snapshot/store.ts";
const PASS = "correct horse battery staple";
describe("a forge back from a snapshot", () => {
  let root: string;
  let app: App;
@@ -110,6 +119,7 @@
describe("a forge back from a snapshot", () => {
        dataDir: app.config.dataDir,
        reposRoot: app.reposRoot,
        version: "t",
        passphrase: PASS,
      },
      store(),
      new Date("2026-09-22T08:00:00Z"),
@@ -119,7 +129,18 @@
describe("a forge back from a snapshot", () => {
    expect(manifest.apps).toBe(1);
    const paths = manifest.files.map((f) => f.path);
    expect(paths).toContain("forge.db");
    expect(paths).toContain("secret.key");
    // The raw key never reaches the store; only its wrapped form does.
    expect(paths).not.toContain("secret.key");
    expect(paths).toContain("secret.key.wrapped");
    expect(manifest.key).toBe("wrapped");
    const rawKey = readFileSync(join(app.config.dataDir, "secret.key"));
    for (const f of paths)
      if (f !== "forge.db" && !f.includes("/"))
        expect(
          readFileSync(
            join(root, "store", "snapshots", manifest.name, f),
          ).includes(rawKey),
        ).toBe(false);
    expect(paths).toContain("repos/alice/notes.bundle");
    expect(paths).not.toContain("repos/alice/empty.bundle");
    expect(paths).toContain("runs/r1.log");
@@ -160,8 +181,12 @@
describe("a forge back from a snapshot", () => {
  test("restored into an empty directory, a forge booted there is the same forge", async () => {
    const back = join(root, "back");
    const done = await restoreSnapshot(store(), "2026-09-22T08-00-00Z", back);
    const done = await restoreSnapshot(store(), "2026-09-22T08-00-00Z", back, {
      passphrase: PASS,
    });
    expect(done.repos).toBe(1);
    expect(done.key).toBe("restored");
    expect(done.sealedMissing).toEqual([]);
    const restored = createApp({ dataDir: back, port: 0 });
    try {
      expect(getUserByName(restored.db, "alice")?.id).toBe(alice.id);
@@ -201,6 +226,62 @@
describe("a forge back from a snapshot", () => {
    }
  });
  test("without the passphrase repos restore and sealed values are reported by name", async () => {
    const back = join(root, "back-nokey");
    const done = await restoreSnapshot(store(), "2026-09-22T08-00-00Z", back);
    expect(done.key).toBe("missing");
    expect(done.sealedMissing).toEqual(["SECRET_WORD (app notes)"]);
    expect(existsSync(join(back, "secret.key.wrapped"))).toBe(false);
    const restored = createApp({ dataDir: back, port: 0 });
    try {
      expect(getRepo(restored.db, "alice", "notes")).toBeDefined();
      const record = getAppByRepo(
        restored.db,
        getRepo(restored.db, "alice", "notes")!.id,
      )!;
      const env = resolveAppEnv(restored.db, back, record.id, "production");
      expect(env["SECRET_WORD"]).toBeUndefined();
    } finally {
      restored.close();
    }
  });
  test("a wrong passphrase refuses the restore and writes nothing", async () => {
    const back = join(root, "back-wrong");
    await expect(
      restoreSnapshot(store(), "2026-09-22T08-00-00Z", back, {
        passphrase: "not the one",
      }),
    ).rejects.toThrow("passphrase");
    expect(existsSync(back)).toBe(false);
  });
  test("a snapshot taken without a passphrase carries no key at all", async () => {
    const s = dirStore(join(root, "store-nokey"));
    const m = await takeSnapshot(
      {
        db: app.db,
        dataDir: app.config.dataDir,
        reposRoot: app.reposRoot,
        version: "t",
      },
      s,
      new Date("2026-09-23T08:00:00Z"),
    );
    expect(m.key).toBe("none");
    expect(m.files.some((f) => f.path.startsWith("secret.key"))).toBe(false);
  });
  test("wrapping round-trips and fails closed", () => {
    const key = Buffer.alloc(32, 7);
    const w = wrapKey(key, "pw");
    expect(w.includes(key.toString("base64url"))).toBe(false);
    expect(unwrapKey(w, "pw")?.equals(key)).toBe(true);
    expect(unwrapKey(w, "pW")).toBeUndefined();
    expect(unwrapKey("{}", "pw")).toBeUndefined();
    expect(unwrapKey("not json", "pw")).toBeUndefined();
  });
  test("a restore refuses a full directory and an unknown snapshot", async () => {
    await expect(
      restoreSnapshot(store(), "2026-09-22T08-00-00Z", join(root, "back")),
@@ -279,6 +360,11 @@
describe("a forge back from a snapshot", () => {
        ).text();
        expect(listed).toContain("2026-09-22T08-00-00Z");
        expect(listed).toContain("Snapshot now");
        // Off by default: the page says no key travels, and what that costs.
        expect(listed).toContain("Snapshots carry no key");
        expect(listed.replace(/\s+/g, " ")).toContain(
          "The last snapshot carried a wrapped key",
        );
        expect(listed).toContain(
          "bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z",
        );
@@ -295,6 +381,23 @@
describe("a forge back from a snapshot", () => {
      } finally {
        await svc.stop();
      }
      const wrapping = startSnapshots(app, dirStore(join(root, "store")), {
        version: "t",
        nightly: false,
        passphrase: PASS,
      });
      try {
        const on = await (
          await fetch(`${servers.url}/settings/backups`, {
            headers: { cookie },
          })
        ).text();
        expect(on).toContain("FORGE_SNAPSHOT_PASSPHRASE");
        expect(on.replace(/\s+/g, " ")).toContain("every sealed value");
        expect(on).not.toContain("Snapshots carry no key");
      } finally {
        await wrapping.stop();
      }
      // A stranger and a plain user see nothing here.
      expect(
        (await fetch(`${servers.url}/settings/backups`, { redirect: "manual" }))
		@@ -29,12 +29,16 @@ turned off, with what each move costs and what it gives up.
29	29	recipe with no network story. Install the four binaries; make
30	30	`/var/lib/forge` and a user that owns it.
31	31	2. Restore, don't migrate. The nightly snapshot holds everything that
32		cannot be rebuilt — the database, `secret.key`, every repository, every
33		app's production data, the crew's logs. On the new box, with the same
34		`S3_*` variables:
35		`bun run src/snapshot/restore.ts <snapshot> /var/lib/forge`
36		The restore sets the key's mode; sessions, tokens, sealed variables and
37		sealed subscriptions all keep working because the key travelled.
	32	cannot be rebuilt — the database, every repository, every app's
	33	production data, the crew's logs — and the forge's key only wrapped under
	34	the operator's passphrase. On the new box, with the same `S3_*` variables
	35	and the passphrase the old forge ran with:
	36	`FORGE_SNAPSHOT_PASSPHRASE=… bun run src/snapshot/restore.ts <snapshot> /var/lib/forge`
	37	The restore unwraps the key and sets its mode; sealed variables, secrets
	38	and sealed subscriptions keep working. The passphrase lives in the forge's
	39	environment and with the operator — keep a copy outside the bucket. Without
	40	it repositories and issues restore, and every sealed value is listed by
	41	name as lost and must be set again (`docs/SNAPSHOT.md`).
38	42	3. Run it as a service. A systemd unit: `ExecStart=bun run src/main.ts`
39	43	from a checkout at `/opt/forge`, `FORGE_URL=https://forge.example`,
40	44	`FORGE_APP_DOMAIN=forge.example`, `FORGE_DATA=/var/lib/forge`, the crew
		@@ -9,9 +9,13 @@ empty data directory. `src/snapshot/`.
9	9
10	10	- `forge.db`, copied consistently while the forge runs (`VACUUM INTO`,
11	11	never a raw copy of a file in WAL mode).
12		- `secret.key`, which the database's sealed values need, and `ssh/`, the
13		host key that is the forge's identity to every clone. Both come back with
14		owner-only modes; the forge refuses a key anyone else can read.
	12	- The key the database's sealed values need, only wrapped:
	13	`secret.key.wrapped`, AES-256-GCM under a key scrypt-derived from the
	14	operator's passphrase. The raw `secret.key` is never written to the store,
	15	so whoever reads the bucket holds the data without the key. See
	16	[The key](#the-key).
	17	- `ssh/`, the host key that is the forge's identity to every clone. It comes
	18	back with an owner-only mode; the forge refuses a key anyone else can read.
15	19	- Every repository as one git bundle of every ref (`repos/<owner>/<name>.bundle`).
16	20	An empty repository has nothing to bundle and is skipped, not failed.
17	21	- Every app's production data (`apps/<id>/production/data/…`). SQLite files
		@@ -20,11 +24,33 @@ empty data directory. `src/snapshot/`.
20	24	stay out.
21	25	- The crew's run logs (`runs/`).
22	26	- `manifest.json`: the name, when, the forge's version, counts, every file
23		with its size. A restore checks each size.
	27	with its size, and `key`: `wrapped` or `none` (older snapshots: `raw`). A
	28	restore checks each size.
24	29
25	30	What stays out is what a boot or a deploy rebuilds: clones, checks,
26	31	pictures, previews.
27	32
	33	## The key
	34
	35	The wrapping secret is `FORGE_SNAPSHOT_PASSPHRASE`, in the forge's
	36	environment. It is not in the bucket, the repository or the database; the
	37	operator holds it elsewhere too (a password manager, a sealed envelope).
	38
	39	- Set: every snapshot carries `secret.key.wrapped`, and the Backups page
	40	says the last one did.
	41	- Unset: no key travels, and the Backups page says so. Snapshots still
	42	restore.
	43	- Lost: the wrapped key cannot be opened by anyone, including us.
	44	Repositories, issues, app data and everything unsealed restore; every
	45	sealed value (forge, org and repo secrets, app variables, origin tokens,
	46	subscription tokens) is lost and is set again by hand. Choose the
	47	passphrase knowing that. A passphrase changed on the forge applies to the
	48	next snapshot; older snapshots stay under the one they were taken with.
	49
	50	Snapshots from before this change hold the raw `secret.key`; they still
	51	restore with the key, and the Backups page tells you to take a new one.
	52	Delete the old ones from the bucket if the bucket is not trusted.
	53
28	54	## Where
29	55
30	56	- On the platform, the app's bucket: the `S3_*` variables the platform puts
		@@ -41,9 +67,17 @@ The store is `Bun.S3Client` or a directory behind one small interface
41	67
42	68	```sh
43	69	# same S3_* as the forge runs under, or FORGE_SNAPSHOT_DIR
44		bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z /path/to/empty/data
	70	FORGE_SNAPSHOT_PASSPHRASE=… \
	71	bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z /path/to/empty/data
45	72	```
46	73
	74	With the passphrase the key is unwrapped into `secret.key` (mode 0600) and
	75	sealed values open. A wrong passphrase refuses the restore before anything is
	76	written. With none, the key is left out, repositories and the rest restore,
	77	and the command lists every sealed value by name (`NAME (app slug)`,
	78	`NAME (scope secret)`) that must be set again; the restored forge makes a new
	79	key on first boot.
	80
47	81	It refuses a directory that is not empty and a snapshot it cannot find or
48	82	that is missing a file. Bundles become bare repositories with the forge's
49	83	hooks installed; the mirror's memory of the bundle as an origin is removed.
		@@ -52,4 +86,6 @@ it has moved on, reconciles the index and redeploys what runs.
52	86
53	87	`test/snapshot.test.ts` is the proof: a running forge with history, a
54	88	sealed variable and an app's SQLite is snapshotted, restored into an empty
55		directory, and booted there as the same forge.
	89	directory with the passphrase and booted there as the same forge; restored
	90	without it, the repositories come back and the sealed variable is reported by
	91	name. It also checks that the raw key is in no file of the store.
		@@ -166,10 +166,18 @@ const VERSION =
166	166	((await import("../package.json")) as { version?: string }).version ?? "0";
167	167	const snapshotDir = process.env["FORGE_SNAPSHOT_DIR"]?.trim();
168	168	const s3 = s3FromEnv();
	169	const snapshotPassphrase =
	170	process.env["FORGE_SNAPSHOT_PASSPHRASE"] \|\| undefined;
169	171	const snapshots = snapshotDir
170		? startSnapshots(app, dirStore(snapshotDir), { version: VERSION })
	172	? startSnapshots(app, dirStore(snapshotDir), {
	173	version: VERSION,
	174	passphrase: snapshotPassphrase,
	175	})
171	176	: s3
172		? startSnapshots(app, s3Store(s3), { version: VERSION })
	177	? startSnapshots(app, s3Store(s3), {
	178	version: VERSION,
	179	passphrase: snapshotPassphrase,
	180	})
173	181	: undefined;
174	182	const crew = appsOn
175	183	? await startCrew(app, {
		@@ -12,6 +12,11 @@
12	12	*
13	13	* with the same `S3_*` variables the forge runs under, or
14	14	* `FORGE_SNAPSHOT_DIR` for a directory store.
	15	*
	16	* The key comes back only from its wrapped form and the operator's
	17	* passphrase (`FORGE_SNAPSHOT_PASSPHRASE`). A wrong passphrase refuses the
	18	* restore; none leaves the key out, and every sealed value that will not
	19	* open is reported by name.
15	20	*/
16	21
17	22	import { chmodSync, existsSync, mkdirSync, readdirSync } from "node:fs";
		@@ -19,19 +24,50 @@ import { dirname, join } from "node:path";
19	24	import { runGit } from "../git/env.ts";
20	25	import { installHooks } from "../git/hooks.ts";
21	26	import { repoDir } from "../git/repos.ts";
22		import { SNAPSHOT_PREFIX, type Manifest } from "./snapshot.ts";
	27	import { keyOf, SNAPSHOT_PREFIX, type Manifest } from "./snapshot.ts";
	28	import { unwrapKey, WRAPPED_KEY } from "./wrap.ts";
23	29	import { dirStore, s3FromEnv, s3Store, type Store } from "./store.ts";
24	30
25	31	export interface Restored {
26	32	readonly manifest: Manifest;
27	33	readonly files: number;
28	34	readonly repos: number;
	35	/** `restored` from the wrapped key, `raw` from an old snapshot, `missing` otherwise. */
	36	readonly key: "restored" \| "raw" \| "missing";
	37	/** Sealed values that will not open without the key, by name. */
	38	readonly sealedMissing: readonly string[];
	39	}
	40
	41	/** Names of every sealed value in a restored database; never the values. */
	42	async function sealedNames(dbFile: string): Promise<string[]> {
	43	const { Database } = await import("bun:sqlite");
	44	const db = new Database(dbFile, { readonly: true });
	45	try {
	46	const secrets = db
	47	.query<{ name: string; scope: string }, []>(
	48	"SELECT name, scope FROM secret ORDER BY scope, name",
	49	)
	50	.all()
	51	.map((r) => `${r.name} (${r.scope} secret)`);
	52	const env = db
	53	.query<{ name: string; slug: string }, []>(
	54	"SELECT e.name AS name, a.slug AS slug FROM app_env e JOIN app a ON a.id = e.app_id ORDER BY a.slug, e.name",
	55	)
	56	.all()
	57	.map((r) => `${r.name} (app ${r.slug})`);
	58	return [...secrets, ...env];
	59	} catch {
	60	return [];
	61	} finally {
	62	db.close();
	63	}
29	64	}
30	65
31	66	export async function restoreSnapshot(
32	67	store: Store,
33	68	name: string,
34	69	dataDir: string,
	70	options: { readonly passphrase?: string \| undefined } = {},
35	71	): Promise<Restored> {
36	72	if (!/^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}-[0-9]{2}-[0-9]{2}Z$/.test(name))
37	73	throw new Error(`not a snapshot name: ${name}`);
		@@ -44,6 +80,19 @@ export async function restoreSnapshot(
44	80	if (raw === undefined)
45	81	throw new Error(`no snapshot ${name} in ${store.where}`);
46	82	const manifest = JSON.parse(new TextDecoder().decode(raw)) as Manifest;
	83	const wrapped = manifest.files.some((f) => f.path === WRAPPED_KEY)
	84	? await store.get(prefix + WRAPPED_KEY)
	85	: undefined;
	86	let key: Buffer \| undefined;
	87	if (wrapped !== undefined && options.passphrase) {
	88	key = unwrapKey(wrapped, options.passphrase);
	89	// Better to stop here than to restore a forge that silently cannot
	90	// open its values because of a mistyped passphrase.
	91	if (key === undefined)
	92	throw new Error(
	93	"the passphrase does not open this snapshot's wrapped key; check FORGE_SNAPSHOT_PASSPHRASE, or restore without it to leave sealed values behind",
	94	);
	95	}
47	96	mkdirSync(dataDir, { recursive: true });
48	97
49	98	let files = 0;
		@@ -57,6 +106,7 @@ export async function restoreSnapshot(
57	106	throw new Error(
58	107	`snapshot ${name}: ${entry.path} is ${body.byteLength} bytes, not ${entry.bytes}`,
59	108	);
	109	if (entry.path === WRAPPED_KEY) continue;
60	110	const bundle = entry.path.match(/^repos\/([^/]+)\/([^/]+)\.bundle$/);
61	111	// A bundle is not a file of the forge's; it becomes a repository below.
62	112	const target = bundle
		@@ -89,7 +139,18 @@ export async function restoreSnapshot(
89	139	}
90	140	const { rmSync } = await import("node:fs");
91	141	rmSync(join(dataDir, "restore-bundles"), { recursive: true, force: true });
92		return { manifest, files, repos };
	142	if (key !== undefined) {
	143	const target = join(dataDir, "secret.key");
	144	await Bun.write(target, key);
	145	chmodSync(target, 0o600);
	146	files++;
	147	}
	148	const kind = keyOf(manifest);
	149	const have =
	150	key !== undefined ? "restored" : kind === "raw" ? "raw" : "missing";
	151	const sealedMissing =
	152	have === "missing" ? await sealedNames(join(dataDir, "forge.db")) : [];
	153	return { manifest, files, repos, key: have, sealedMissing };
93	154	}
94	155
95	156	if (import.meta.main) {
		@@ -111,8 +172,16 @@ if (import.meta.main) {
111	172	);
112	173	process.exit(2);
113	174	}
114		const done = await restoreSnapshot(store, name, dataDir);
	175	const done = await restoreSnapshot(store, name, dataDir, {
	176	passphrase: process.env["FORGE_SNAPSHOT_PASSPHRASE"] \|\| undefined,
	177	});
115	178	console.log(
116	179	`restored ${done.manifest.name} (forge ${done.manifest.version}): ${done.files} files, ${done.repos} repositories into ${dataDir}`,
117	180	);
	181	if (done.key === "missing") {
	182	console.log(
	183	`the key was not restored: ${done.sealedMissing.length} sealed value(s) will not open and must be set again:`,
	184	);
	185	for (const n of done.sealedMissing) console.log(` ${n}`);
	186	}
118	187	}
		@@ -26,6 +26,8 @@ const NIGHTLY_HOUR_UTC = 8;
26	26
27	27	export interface SnapshotsHandle {
28	28	readonly store: Store;
	29	/** Whether snapshots carry the key, wrapped under a passphrase. */
	30	readonly wraps: boolean;
29	31	/** Take one now; waits for any in flight. */
30	32	readonly now: () => Promise<Manifest>;
31	33	/** The newest first, from the store. */
		@@ -49,7 +51,11 @@ export function untilNightly(now = Date.now()): number {
49	51	export function startSnapshots(
50	52	app: App,
51	53	store: Store,
52		options: { readonly version: string; readonly nightly?: boolean } = {
	54	options: {
	55	readonly version: string;
	56	readonly nightly?: boolean;
	57	readonly passphrase?: string \| undefined;
	58	} = {
53	59	version: "0",
54	60	},
55	61	): SnapshotsHandle {
		@@ -65,6 +71,7 @@ export function startSnapshots(
65	71	dataDir: app.config.dataDir,
66	72	reposRoot: app.reposRoot,
67	73	version: options.version,
	74	passphrase: options.passphrase,
68	75	},
69	76	store,
70	77	);
		@@ -98,6 +105,7 @@ export function startSnapshots(
98	105
99	106	const handle: SnapshotsHandle = {
100	107	store,
	108	wraps: !!options.passphrase,
101	109	now: take,
102	110	list: () => listSnapshots(store),
103	111	last: () => last,
		@@ -109,6 +117,10 @@ export function startSnapshots(
109	117	},
110	118	};
111	119	handles.set(app, handle);
112		log.info("snapshots on", { where: store.where, keep: KEEP });
	120	log.info("snapshots on", {
	121	where: store.where,
	122	keep: KEEP,
	123	key: options.passphrase ? "wrapped" : "none",
	124	});
113	125	return handle;
114	126	}
		@@ -27,6 +27,7 @@ import { runGit } from "../git/env.ts";
27	27	import { repoDir } from "../git/repos.ts";
28	28	import { createLog } from "../kernel/log.ts";
29	29	import type { Store } from "./store.ts";
	30	import { unwrapKey, wrapKey, WRAPPED_KEY } from "./wrap.ts";
30	31
31	32	const log = createLog("snapshot");
32	33
		@@ -48,13 +49,25 @@ export interface Manifest {
48	49	readonly apps: number;
49	50	readonly files: readonly SnapshotFile[];
50	51	readonly bytes: number;
	52	/**
	53	* What the snapshot carries of the forge's key: `wrapped` under the
	54	* operator's passphrase, `none` (sealed values are lost on restore), or
	55	* `raw` in snapshots from before the key was wrapped.
	56	*/
	57	readonly key?: "wrapped" \| "none" \| "raw";
51	58	}
52	59
	60	/** What a manifest says of the key, including manifests older than the field. */
	61	export const keyOf = (m: Manifest): "wrapped" \| "none" \| "raw" =>
	62	m.key ?? (m.files.some((f) => f.path === "secret.key") ? "raw" : "none");
	63
53	64	export interface Snapshotter {
54	65	readonly db: Database;
55	66	readonly dataDir: string;
56	67	readonly reposRoot: string;
57	68	readonly version: string;
	69	/** The operator's wrapping secret; without it the key stays out. */
	70	readonly passphrase?: string \| undefined;
58	71	}
59	72
60	73	/** `2026-09-22T09-00-00Z`: sortable, and a name a bucket allows. */
		@@ -130,7 +143,17 @@ export async function takeSnapshot(
130	143	const dbCopy = join(scratch, "forge.db");
131	144	forge.db.run("VACUUM INTO ?1", [dbCopy]);
132	145	await putFile("forge.db", dbCopy);
133		await putFile("secret.key", join(forge.dataDir, "secret.key"));
	146	// The key travels only wrapped. Without a passphrase it stays home.
	147	let key: Manifest["key"] = "none";
	148	const keyFile = Bun.file(join(forge.dataDir, "secret.key"));
	149	if (forge.passphrase && (await keyFile.exists())) {
	150	const raw = new Uint8Array(await keyFile.arrayBuffer());
	151	const wrapped = wrapKey(raw, forge.passphrase);
	152	if (unwrapKey(wrapped, forge.passphrase) === undefined)
	153	throw new Error("the wrapped key does not open; snapshot abandoned");
	154	await put(WRAPPED_KEY, wrapped);
	155	key = "wrapped";
	156	}
134	157	for (const f of filesUnder(join(forge.dataDir, "ssh")))
135	158	await putFile(`ssh/${f}`, join(forge.dataDir, "ssh", f));
136	159
		@@ -204,6 +227,7 @@ export async function takeSnapshot(
204	227	apps: kept,
205	228	files,
206	229	bytes: files.reduce((n, f) => n + f.bytes, 0),
	230	key,
207	231	};
208	232	await store.put(
209	233	`${prefix}manifest.json`,
		@@ -215,6 +239,7 @@ export async function takeSnapshot(
215	239	repos: bundled,
216	240	apps: kept,
217	241	files: files.length,
	242	key,
218	243	mb: (manifest.bytes / 1_048_576).toFixed(1),
219	244	});
220	245	return manifest;
		@@ -0,0 +1,80 @@
	1	/**
	2	* The forge's key, wrapped for a bucket: AES-256-GCM under a key stretched
	3	* from an operator's passphrase with scrypt. The passphrase lives in the
	4	* forge's environment (`FORGE_SNAPSHOT_PASSPHRASE`) and with the operator,
	5	* never in the bucket, so whoever reads the bucket holds the data without
	6	* the key that opens its sealed values.
	7	*
	8	* The wrapped form is one small JSON file:
	9	* `{"v":1,"kdf":"scrypt","salt":…,"iv":…,"ct":…}`, base64url. A wrong
	10	* passphrase and a tampered file both fail the tag check, and say so alike.
	11	*/
	12
	13	import {
	14	createCipheriv,
	15	createDecipheriv,
	16	randomBytes,
	17	scryptSync,
	18	} from "node:crypto";
	19
	20	export const WRAPPED_KEY = "secret.key.wrapped";
	21
	22	const SALT_BYTES = 16;
	23	const IV_BYTES = 12;
	24	const TAG_BYTES = 16;
	25	/** Interactive-grade cost: one derivation per snapshot, once a night. */
	26	const SCRYPT = { N: 1 << 15, r: 8, p: 1, maxmem: 64 * 1024 * 1024 } as const;
	27
	28	const derive = (passphrase: string, salt: Buffer): Buffer =>
	29	scryptSync(passphrase, salt, 32, SCRYPT);
	30
	31	export function wrapKey(key: Uint8Array, passphrase: string): string {
	32	const salt = randomBytes(SALT_BYTES);
	33	const iv = randomBytes(IV_BYTES);
	34	const cipher = createCipheriv("aes-256-gcm", derive(passphrase, salt), iv, {
	35	authTagLength: TAG_BYTES,
	36	});
	37	const ct = Buffer.concat([
	38	cipher.update(key),
	39	cipher.final(),
	40	cipher.getAuthTag(),
	41	]);
	42	return JSON.stringify({
	43	v: 1,
	44	kdf: "scrypt",
	45	salt: salt.toString("base64url"),
	46	iv: iv.toString("base64url"),
	47	ct: ct.toString("base64url"),
	48	});
	49	}
	50
	51	/** The key, or undefined when the passphrase is wrong or the file is not ours. */
	52	export function unwrapKey(
	53	wrapped: Uint8Array \| string,
	54	passphrase: string,
	55	): Buffer \| undefined {
	56	try {
	57	const text =
	58	typeof wrapped === "string" ? wrapped : new TextDecoder().decode(wrapped);
	59	const w = JSON.parse(text) as Record<string, unknown>;
	60	if (w["v"] !== 1 \|\| w["kdf"] !== "scrypt") return undefined;
	61	const salt = Buffer.from(String(w["salt"]), "base64url");
	62	const iv = Buffer.from(String(w["iv"]), "base64url");
	63	const ct = Buffer.from(String(w["ct"]), "base64url");
	64	if (salt.length !== SALT_BYTES \|\| iv.length !== IV_BYTES) return undefined;
	65	if (ct.length <= TAG_BYTES) return undefined;
	66	const decipher = createDecipheriv(
	67	"aes-256-gcm",
	68	derive(passphrase, salt),
	69	iv,
	70	{ authTagLength: TAG_BYTES },
	71	);
	72	decipher.setAuthTag(ct.subarray(ct.length - TAG_BYTES));
	73	return Buffer.concat([
	74	decipher.update(ct.subarray(0, ct.length - TAG_BYTES)),
	75	decipher.final(),
	76	]);
	77	} catch {
	78	return undefined;
	79	}
	80	}
		@@ -8,6 +8,7 @@
8	8	* and a stopped forge, and a person at a terminal is the one who has both.
9	9	*/
10	10
	11	import { keyOf } from "../../snapshot/snapshot.ts";
11	12	import { snapshotsOf } from "../../snapshot/service.ts";
12	13	import type { Ctx } from "../../http/context.ts";
13	14	import { notFoundPage, settingsLayout, signInFirst } from "../chrome.ts";
		@@ -36,6 +37,46 @@ function shell(ctx: Ctx, body: Html): Response {
36	37	});
37	38	}
38	39
	40	/**
	41	* Whether the key travels, said before anyone relies on it. The passphrase
	42	* is the only way back to sealed values from a bucket; lose it and they
	43	* are lost.
	44	*/
	45	function keyNote(
	46	wraps: boolean,
	47	newest:
	48	\| Awaited<
	49	ReturnType<NonNullable<ReturnType<typeof snapshotsOf>>["list"]>
	50	>[number]
	51	\| undefined,
	52	): Html {
	53	const carried = newest === undefined ? undefined : keyOf(newest);
	54	const last =
	55	carried === undefined
	56	? ""
	57	: carried === "wrapped"
	58	? " The last snapshot carried a wrapped key."
	59	: carried === "raw"
	60	? " The last snapshot carried the key unwrapped, as older snapshots did; take a new one."
	61	: " The last snapshot carried no key.";
	62	return wraps
	63	? html`<p>
	64	${icon("check")} The key is wrapped under
	65	<code>FORGE_SNAPSHOT_PASSPHRASE</code> and travels with each
	66	snapshot.${last} Keep that passphrase somewhere other than this bucket:
	67	without it a restore brings back repositories and issues, but every
	68	sealed value (secrets, app variables, origin tokens) is lost and must be
	69	set again.
	70	</p>`
	71	: html`<p class="backups-error">
	72	${icon("alert")} Snapshots carry no key: the bucket never holds key and
	73	data together, and a restore will bring back repositories and issues but
	74	not sealed values, which must be set again.${last} To carry the key
	75	wrapped, set <code>FORGE_SNAPSHOT_PASSPHRASE</code> and restart — and
	76	keep the passphrase elsewhere, because losing it loses those values.
	77	</p>`;
	78	}
	79
39	80	export async function backupsPage(ctx: Ctx): Promise<Response> {
40	81	const denied = requireAdmin(ctx);
41	82	if (denied) return denied;
		@@ -61,8 +102,7 @@ export async function backupsPage(ctx: Ctx): Promise<Response> {
61	102	}
62	103	const restore = snapshots[0]
63	104	? html`<pre class="backups-restore">
64		bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre
65		>`
	105	bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre>`
66	106	: null;
67	107	return shell(
68	108	ctx,
		@@ -73,6 +113,7 @@ bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre
73	113	crew's logs — to <code>${svc.store.where}</code>, and keeps the newest
74	114	fourteen.
75	115	</p>
	116	${keyNote(svc.wraps, snapshots[0])}
76	117	${when(
77	118	last.error,
78	119	(error) =>
		@@ -100,47 +141,51 @@ bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre
100	141	<h2>Where and when</h2>`,
101	142	},
102	143	)}
103		${snapshots.length === 0
104		? empty({
105		icon: "clock",
106		title: "None yet",
107		note: "The first is taken tonight, or now with the button above.",
108		})
109		: card(
110		html`<ul class="list">
111		${snapshots.map(
112		(m) =>
113		html`<li>
114		<div class="row">
115		<span class="row-lead">${icon("download")}</span>
116		<span class="row-main">
117		<span class="row-title">${m.name}</span>
118		<span class="row-meta"
119		>${time(m.finishedAt)} · forge ${m.version} ·
120		${m.repos === 1
121		? "one repository"
122		: `${m.repos} repositories`},
123		${m.apps === 1 ? "one app" : `${m.apps} apps`},
124		${m.files.length} files</span
125		>
126		</span>
127		<span class="row-trail muted">${fileSize(m.bytes)}</span>
128		</div>
129		</li>`,
130		)}
131		</ul>`,
132		{
133		pad: false,
134		head: html`${icon("history")}
135		<h2>Snapshots</h2>`,
136		foot: html`<span class="muted small"
137		>To restore, stop a forge, point it at an empty data directory,
138		and run this with the same S3_* variables or
139		FORGE_SNAPSHOT_DIR:</span
140		>
141		${restore}`,
142		},
143		)}`,
	144	${
	145	snapshots.length === 0
	146	? empty({
	147	icon: "clock",
	148	title: "None yet",
	149	note: "The first is taken tonight, or now with the button above.",
	150	})
	151	: card(
	152	html`<ul class="list">
	153	${snapshots.map(
	154	(m) =>
	155	html`<li>
	156	<div class="row">
	157	<span class="row-lead">${icon("download")}</span>
	158	<span class="row-main">
	159	<span class="row-title">${m.name}</span>
	160	<span class="row-meta"
	161	>${time(m.finishedAt)} · forge ${m.version} ·
	162	${
	163	m.repos === 1
	164	? "one repository"
	165	: `${m.repos} repositories`
	166	},
	167	${m.apps === 1 ? "one app" : `${m.apps} apps`},
	168	${m.files.length} files</span
	169	>
	170	</span>
	171	<span class="row-trail muted">${fileSize(m.bytes)}</span>
	172	</div>
	173	</li>`,
	174	)}
	175	</ul>`,
	176	{
	177	pad: false,
	178	head: html`${icon("history")}
	179	<h2>Snapshots</h2>`,
	180	foot: html`<span class="muted small"
	181	>To restore, stop a forge, point it at an empty data
	182	directory, and run this with the same S3_* variables or
	183	FORGE_SNAPSHOT_DIR:</span
	184	>
	185	${restore}`,
	186	},
	187	)
	188	}`,
144	189	);
145	190	}
146	191
		@@ -7,7 +7,13 @@
7	7	* restore refuses what it must.
8	8	*/
9	9	import { afterAll, beforeAll, describe, expect, test } from "bun:test";
10		import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
	10	import {
	11	existsSync,
	12	mkdirSync,
	13	mkdtempSync,
	14	readFileSync,
	15	writeFileSync,
	16	} from "node:fs";
11	17	import { rm } from "node:fs/promises";
12	18	import { tmpdir } from "node:os";
13	19	import { join } from "node:path";
		@@ -30,8 +36,11 @@ import {
30	36	snapshotName,
31	37	takeSnapshot,
32	38	} from "../src/snapshot/snapshot.ts";
	39	import { unwrapKey, wrapKey } from "../src/snapshot/wrap.ts";
33	40	import { dirStore } from "../src/snapshot/store.ts";
34	41
	42	const PASS = "correct horse battery staple";
	43
35	44	describe("a forge back from a snapshot", () => {
36	45	let root: string;
37	46	let app: App;
		@@ -110,6 +119,7 @@ describe("a forge back from a snapshot", () => {
110	119	dataDir: app.config.dataDir,
111	120	reposRoot: app.reposRoot,
112	121	version: "t",
	122	passphrase: PASS,
113	123	},
114	124	store(),
115	125	new Date("2026-09-22T08:00:00Z"),
		@@ -119,7 +129,18 @@ describe("a forge back from a snapshot", () => {
119	129	expect(manifest.apps).toBe(1);
120	130	const paths = manifest.files.map((f) => f.path);
121	131	expect(paths).toContain("forge.db");
122		expect(paths).toContain("secret.key");
	132	// The raw key never reaches the store; only its wrapped form does.
	133	expect(paths).not.toContain("secret.key");
	134	expect(paths).toContain("secret.key.wrapped");
	135	expect(manifest.key).toBe("wrapped");
	136	const rawKey = readFileSync(join(app.config.dataDir, "secret.key"));
	137	for (const f of paths)
	138	if (f !== "forge.db" && !f.includes("/"))
	139	expect(
	140	readFileSync(
	141	join(root, "store", "snapshots", manifest.name, f),
	142	).includes(rawKey),
	143	).toBe(false);
123	144	expect(paths).toContain("repos/alice/notes.bundle");
124	145	expect(paths).not.toContain("repos/alice/empty.bundle");
125	146	expect(paths).toContain("runs/r1.log");
		@@ -160,8 +181,12 @@ describe("a forge back from a snapshot", () => {
160	181
161	182	test("restored into an empty directory, a forge booted there is the same forge", async () => {
162	183	const back = join(root, "back");
163		const done = await restoreSnapshot(store(), "2026-09-22T08-00-00Z", back);
	184	const done = await restoreSnapshot(store(), "2026-09-22T08-00-00Z", back, {
	185	passphrase: PASS,
	186	});
164	187	expect(done.repos).toBe(1);
	188	expect(done.key).toBe("restored");
	189	expect(done.sealedMissing).toEqual([]);
165	190	const restored = createApp({ dataDir: back, port: 0 });
166	191	try {
167	192	expect(getUserByName(restored.db, "alice")?.id).toBe(alice.id);
		@@ -201,6 +226,62 @@ describe("a forge back from a snapshot", () => {
201	226	}
202	227	});
203	228
	229	test("without the passphrase repos restore and sealed values are reported by name", async () => {
	230	const back = join(root, "back-nokey");
	231	const done = await restoreSnapshot(store(), "2026-09-22T08-00-00Z", back);
	232	expect(done.key).toBe("missing");
	233	expect(done.sealedMissing).toEqual(["SECRET_WORD (app notes)"]);
	234	expect(existsSync(join(back, "secret.key.wrapped"))).toBe(false);
	235	const restored = createApp({ dataDir: back, port: 0 });
	236	try {
	237	expect(getRepo(restored.db, "alice", "notes")).toBeDefined();
	238	const record = getAppByRepo(
	239	restored.db,
	240	getRepo(restored.db, "alice", "notes")!.id,
	241	)!;
	242	const env = resolveAppEnv(restored.db, back, record.id, "production");
	243	expect(env["SECRET_WORD"]).toBeUndefined();
	244	} finally {
	245	restored.close();
	246	}
	247	});
	248
	249	test("a wrong passphrase refuses the restore and writes nothing", async () => {
	250	const back = join(root, "back-wrong");
	251	await expect(
	252	restoreSnapshot(store(), "2026-09-22T08-00-00Z", back, {
	253	passphrase: "not the one",
	254	}),
	255	).rejects.toThrow("passphrase");
	256	expect(existsSync(back)).toBe(false);
	257	});
	258
	259	test("a snapshot taken without a passphrase carries no key at all", async () => {
	260	const s = dirStore(join(root, "store-nokey"));
	261	const m = await takeSnapshot(
	262	{
	263	db: app.db,
	264	dataDir: app.config.dataDir,
	265	reposRoot: app.reposRoot,
	266	version: "t",
	267	},
	268	s,
	269	new Date("2026-09-23T08:00:00Z"),
	270	);
	271	expect(m.key).toBe("none");
	272	expect(m.files.some((f) => f.path.startsWith("secret.key"))).toBe(false);
	273	});
	274
	275	test("wrapping round-trips and fails closed", () => {
	276	const key = Buffer.alloc(32, 7);
	277	const w = wrapKey(key, "pw");
	278	expect(w.includes(key.toString("base64url"))).toBe(false);
	279	expect(unwrapKey(w, "pw")?.equals(key)).toBe(true);
	280	expect(unwrapKey(w, "pW")).toBeUndefined();
	281	expect(unwrapKey("{}", "pw")).toBeUndefined();
	282	expect(unwrapKey("not json", "pw")).toBeUndefined();
	283	});
	284
204	285	test("a restore refuses a full directory and an unknown snapshot", async () => {
205	286	await expect(
206	287	restoreSnapshot(store(), "2026-09-22T08-00-00Z", join(root, "back")),
		@@ -279,6 +360,11 @@ describe("a forge back from a snapshot", () => {
279	360	).text();
280	361	expect(listed).toContain("2026-09-22T08-00-00Z");
281	362	expect(listed).toContain("Snapshot now");
	363	// Off by default: the page says no key travels, and what that costs.
	364	expect(listed).toContain("Snapshots carry no key");
	365	expect(listed.replace(/\s+/g, " ")).toContain(
	366	"The last snapshot carried a wrapped key",
	367	);
282	368	expect(listed).toContain(
283	369	"bun run src/snapshot/restore.ts 2026-09-22T08-00-00Z",
284	370	);
		@@ -295,6 +381,23 @@ describe("a forge back from a snapshot", () => {
295	381	} finally {
296	382	await svc.stop();
297	383	}
	384	const wrapping = startSnapshots(app, dirStore(join(root, "store")), {
	385	version: "t",
	386	nightly: false,
	387	passphrase: PASS,
	388	});
	389	try {
	390	const on = await (
	391	await fetch(`${servers.url}/settings/backups`, {
	392	headers: { cookie },
	393	})
	394	).text();
	395	expect(on).toContain("FORGE_SNAPSHOT_PASSPHRASE");
	396	expect(on.replace(/\s+/g, " ")).toContain("every sealed value");
	397	expect(on).not.toContain("Snapshots carry no key");
	398	} finally {
	399	await wrapping.stop();
	400	}
298	401	// A stranger and a plain user see nothing here.
299	402	expect(
300	403	(await fetch(`${servers.url}/settings/backups`, { redirect: "manual" }))