Public: anyone can read this; the crew can change it.

The restore drill restores one hosted forge and says which, and a snapshot that left one out is red #13923

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

trevato/backup-hosted-forges → main 1 commit 3 threads · 3 unresolved

  • Merged. trevato/backup-hosted-forges was merged into main as 936cb90.

Review

Sign in to review this change.
10 files changed +562 −12
docs/RECOVERY.md +5 −5
@@ -88,15 +88,15 @@ which line it stops at.
8888 Likely a bad snapshot or a slow store. Check the phase, then Route 1.
8989 - **open** — opening the restored `forge.db`. Likely the file is truncated or
9090 corrupt. Check its size against the snapshot's manifest.
91 - **count** — counting rows and comparing with the counts the snapshot kept.
92 Likely the snapshot was cut mid-write or rows were lost. Check which table
93 the message names and its count in the snapshot.
91 - **count** — counting rows against the counts the snapshot kept. Likely it was
92 cut mid-write or rows were lost. Check the table the message names there.
9493 - **unseal** — opening one sealed value with the restored key. Likely the key
9594 is missing or from another forge, or the passphrase is wrong. Check
9695 `FORGE_SNAPSHOT_PASSPHRASE` and that the snapshot holds the key.
9796 - **health** — booting the restored forge and asking it for health. Likely a
98 migration or config problem, or a boot slower than the limit. Boot it by
99 hand as in Route 1 and read its log.
97 migration or config problem, or a slow boot. Boot it by hand (Route 1); read its log.
98 - **forge** — restoring one hosted forge alone and proving it opens (ledger, integrity check,
99 `git fsck`). Likely a chunk only it needs is gone. It is named: Route 1 with `--forge <name>`.
100100
101101 ### Restore phases
102102
docs/features/hosted-backup.md +6
@@ -27,6 +27,12 @@ A hosted forge's directory is the forge's own to write, and its host reads it fo
2727 - Resilience screen (terminal): the same line, then one line per forge: kept H h ago, or left out and why.
2828 - `/forges`: each row, "kept <age> ago" (under an hour, hours, then days) or "never kept" when no backup holds it, or why the newest left it out.
2929 - `forges` MCP tool and `GET /api/v1/forges`: the same words as `kept` on each made forge.
30 - MCP `backups`: `hosted`, the same sentence and each forge's line.
31 - The backup verdict (home, vitals, MCP `backups`, the Resilience screen): a lane whose newest snapshot left a hosted forge out is red and names it with why. A repository a forge's copy lacks is said there too, as an unknown and not an alarm.
32
33 ## The drill restores one
34
35 Each restore drill, after the host itself, restores one hosted forge alone from the same snapshot (the next one each day and each week, `src/snapshot/drill_hosted.ts`) and proves it opens: its ledger reads and names it, its database passes SQLite's integrity check, and every repository its copy holds comes back and passes `git fsck`. The verdict says which forge ("…passed, with hosted forge glint (1 of 17)"). One that does not come back fails the drill at the step `forge`, by name ([RECOVERY.md](../RECOVERY.md)).
3036
3137 ## Restore one forge
3238
src/http/mcp/tools/backups.ts +11 −4
@@ -4,13 +4,14 @@ import { bodiesOf } from "../../../snapshot/store.ts";
44 import { mirrorNeed } from "../../../snapshot/mirror.ts";
55 import { s3FromEnv } from "../../../snapshot/store.ts";
66 import { listSnapshots } from "../../../snapshot/snapshot.ts";
7 import { hostedWords, keptWords } from "../../../snapshot/hosted.ts";
78 import { backupVerdict } from "../../../snapshot/bucket.ts";
89 import { lastDrill } from "../../../snapshot/drills.ts";
910 import { good, bad, type Tool } from "../shared.ts";
1011
1112 export const tool: Tool = {
1213 name: "backups",
13 doc: "where snapshots go (`destination`: its `place`, `bodies` (whether a snapshot's data is encrypted before upload, in words), and `credentials` (where the bucket's keys are kept: sealed, dir or env; it says nothing of the data), never a credential; `null` with the Backups page's sentence when none), `verdict` (state, line, reasons, lanes with newest object and `bodies`, drill: read from the bucket, red past 26 h or when the newest snapshot lacks a repository), the last fourteen snapshots (`key`, `at`, `bytes`) and the off-box `mirror`'s state; a forge admin's, as /settings/backups",
14 doc: "where snapshots go (`destination`: its `place`, `bodies` (whether a snapshot's data is encrypted before upload, in words), and `credentials` (where the bucket's keys are kept: sealed, dir or env; it says nothing of the data), never a credential; `null` with the Backups page's sentence when none), `verdict` (state, line, reasons, lanes with newest object and `bodies`, drill: read from the bucket, red past 26 h or when the newest snapshot lacks a repository or a hosted forge), the last fourteen snapshots (`key`, `at`, `bytes`), `hosted` (the forges this one hosts in the newest snapshot: `says` as the Backups page says it, and each forge's `says` as /forges does; absent when it names none) and the off-box `mirror`'s state; a forge admin's, as /settings/backups",
1415 description: "Admin: snapshot destination, recent, mirror.",
1516 inputSchema: { type: "object", properties: {} },
1617 writes: false,
@@ -39,10 +40,15 @@ export const tool: Tool = {
3940 });
4041 let snapshots: { key: string; at: string; bytes: number }[] = [];
4142 let listing: string | undefined;
43 // The hosted forges the newest snapshot kept, in the Backups page's words and /forges'
44 // (#13819).
45 let hosted: { says: string; forges: { name: string; says: string }[] } | undefined;
4246 try {
43 snapshots = (await listSnapshots(dest.store))
44 .slice(0, 14)
45 .map((m) => ({ key: m.name, at: m.finishedAt, bytes: m.bytes }));
47 const all = await listSnapshots(dest.store);
48 snapshots = all.slice(0, 14).map((m) => ({ key: m.name, at: m.finishedAt, bytes: m.bytes }));
49 const kept = all[0]?.hosted;
50 const says = hostedWords(kept);
51 if (kept !== undefined && says !== undefined) hosted = { says, forges: kept.map((h) => ({ name: h.name, says: keptWords(kept, h.name, Date.now()) })) };

This line is far too long. Wrap it; bun scripts/gate.ts --fix rewraps comments, but this is code, so wrap it by hand.

4652 } catch (error) {
4753 const sources = { db: ctx.app.db, dataDir: ctx.app.config.dataDir, bus: ctx.app.bus };
4854 const target = dest.source === "sealed" ? sealedTarget(sources) : dest.source === "env" ? s3FromEnv() : undefined;
@@ -53,6 +59,7 @@ export const tool: Tool = {
5359 // `credentials` is where the bucket's keys are kept; `bodies` is what the data itself gets.
5460 destination: { place: dest.store.where, bodies: bodiesOf(dest.store), credentials: dest.source },
5561 snapshots,
62 ...(hosted === undefined ? {} : { hosted }),
5663 ...(listing === undefined ? {} : { listing }),
5764 mirror,
5865 });
src/snapshot/bucket.ts +56 −1
@@ -19,6 +19,7 @@ import { join } from "node:path";
1919 import type { DrillResult } from "./drill.ts";
2020 import { isOffBox } from "./off_box.ts";
2121 import { listSnapshots, notInSnapshot } from "./snapshot.ts";
22 import type { HostedEntry } from "./hosted.ts";
2223 import { scrubReason } from "./destination.ts";
2324 import { bodiesOf, type S3Target, type Store } from "./store.ts";
2425
@@ -47,6 +48,11 @@ export interface LaneRead {
4748 readonly bodies?: string;
4849 /** Repositories the newest snapshot says it holds no bundle of though they held something. */
4950 readonly absent?: readonly string[];
51 /**
52 * The hosted forges in the newest snapshot (#13819): how many it holds a copy of, and those it
53 * names as left out, each with why. Absent when it names none.
54 */
55 readonly hosted?: HostedRead;
5056 /** When this read ended. */
5157 readonly readAt: string;
5258 /** What the lane backs up, one sentence, from the configuration (#9067). */
@@ -55,6 +61,25 @@ export interface LaneRead {
5561 readonly afterSpinDown?: string;
5662 }
5763
64 /** What a lane's newest snapshot says of the forges its forge hosts. */
65 export interface HostedRead {
66 readonly count: number;
67 readonly failed: readonly { readonly name: string; readonly why: string }[];
68 /**
69 * Repositories of the forges it does hold that their copies hold no bundle of though they held
70 * something, each with why; absent when there is none.
71 */
72 readonly absent?: readonly { readonly forge: string; readonly repo: string; readonly why: string }[];
73 }
74
75 /** "2 repositories of hosted forges are not in it: glint: glint/app (why); …"; empty when none. */
76 function hostedAbsentWords(h: HostedRead): string {
77 const out = h.absent ?? [];
78 if (out.length === 0) return "";
79 const named = out.slice(0, 8).map((a) => `${printable(a.forge)}: ${printable(a.repo)} (${printable(a.why)})`).join("; ");
80 return `${out.length} ${out.length === 1 ? "repository" : "repositories"} of hosted forges ${out.length === 1 ? "is" : "are"} not in it: ${named}${out.length > 8 ? `; and ${out.length - 8} more` : ""}`;
81 }
82
5883 /** `retired` in any case: the lane is listed, never read, never red. */
5984 export const isRetired = (afterSpinDown: string | undefined): boolean => afterSpinDown?.trim().toLowerCase() === "retired";
6085
@@ -194,9 +219,23 @@ export async function readLane(
194219 const all = await listSnapshots(store);
195220 const newest = all[0];
196221 const absent = newest === undefined ? [] : notInSnapshot(newest).map((a) => printable(a.repo));
222 const forges: readonly HostedEntry[] = (newest?.hosted ?? []).filter((f) => typeof f === "object" && f !== null && typeof f.name === "string");
223 const held = forges.filter((f) => f.why === undefined);
224 const left = held.flatMap((f) =>
225 (f.absent ?? []).filter((a) => a.empty !== true).map((a) => ({ forge: f.name, repo: String(a.repo), why: String(a.why).slice(0, 200) })),
226 );
227 const hosted: HostedRead | undefined =
228 forges.length === 0
229 ? undefined
230 : {
231 count: held.length,
232 failed: forges.filter((f) => f.why !== undefined).map((f) => ({ name: f.name, why: String(f.why).slice(0, 200) })),
233 ...(left.length === 0 ? {} : { absent: left }),
234 };
197235 return {
198236 ...base,
199237 ...(absent.length === 0 ? {} : { absent }),
238 ...(hosted === undefined ? {} : { hosted }),
200239 newestAt: newest?.finishedAt ?? null,
201240 newestName: newest?.name ?? null,
202241 newestBytes: newest?.bytes ?? null,
@@ -275,6 +314,14 @@ export function judge(
275314 else if (r.absent !== undefined && r.absent.length > 0) {
276315 const n = r.absent.length;
277316 says = `${lane} (${where}) newest snapshot${name} lacks ${n} ${n === 1 ? "repository" : "repositories"}, which a restore would not bring back: ${r.absent.slice(0, 8).join(", ")}${n > 8 ? `, and ${n - 8} more` : ""}`;
317 }
318 // Nor is one that lacks a forge this one hosts: a person's whole forge would not come back.
319 else if (r.hosted !== undefined && r.hosted.failed.length > 0) {
320 const n = r.hosted.failed.length;
321 says = `${lane} (${where}) newest snapshot${name} lacks ${n} hosted ${n === 1 ? "forge" : "forges"}, which a restore would not bring back: ${r.hosted.failed
322 .slice(0, 8)
323 .map((f) => `${printable(f.name)} (${printable(f.why)})`)
324 .join("; ")}${n > 8 ? `; and ${n - 8} more` : ""}`;
278325 } else {
279326 red = false;
280327 says = `${lane} newest${name} ${ageWords(age)} old, ${sizeWords(r.newestBytes ?? 0)}`;
@@ -288,10 +335,16 @@ export function judge(
288335 ? "empty"
289336 : !(now - Date.parse(r.newestAt) <= LATE_MS)
290337 ? "late"
291 : "repositories missing",
338 : r.absent !== undefined && r.absent.length > 0
339 ? "repositories missing"
340 : "hosted forges missing",
292341 );
293342 reasons.push(`${says[0]!.toUpperCase()}${says.slice(1)}.`);
294343 }
344 // A repository of a hosted forge that its copy lacks is said and is not calm, and is not this
345 // forge's alarm either: what it holds is its owner's doing, night after night.
346 if (r.hosted !== undefined && r.error === undefined && (r.hosted.absent ?? []).length > 0)
347 unknowns.push(`${lane[0]!.toUpperCase()}${lane.slice(1)}: ${hostedAbsentWords(r.hosted)}.`);
295348 lanes.push({ ...r, backsUp, afterSpinDown, state: red ? "red" : "green", says, line });
296349 }
297350 if (state !== undefined && !lanes.some((l) => l.state !== "retired")) {
@@ -309,6 +362,8 @@ export function judge(
309362 // The proof's age, said: "passed" for six days is not the same as passed today (#12592).
310363 const proofAge = ageWords(Math.max(0, now - Date.parse(drill.at)));
311364 if (ok) says = `${says} (${proofAge} ago)`;
365 // And the hosted forge it restored alone, by name: one of them has come back (#13819).
366 if (ok && drill.hosted !== undefined) says = `${says}, with hosted forge ${printable(drill.hosted.name)} (1 of ${drill.hosted.of})`;
312367 drillSays = { at: drill.at, ok, says };
313368 if (!ok) {
314369 causes.push(drill.verdict === "green" ? "drill restored nothing" : "drill failed");
src/snapshot/drill.ts +27 −2
@@ -19,6 +19,7 @@ import { bootRestored } from "./boot.ts";
1919 import { forgetKey, open } from "../kernel/secret.ts";
2020 import { countRows, mismatched, type DrillCounts } from "./counts.ts";
2121 import { restoreSnapshot, type Progress } from "./restore.ts";
22 import { drillPick, proveHosted, type HostedProof } from "./drill_hosted.ts";
2223 import { isOffBox } from "./off_box.ts";
2324 import { listSnapshots } from "./snapshot.ts";
2425 import type { Store } from "./store.ts";
@@ -133,7 +134,7 @@ export interface PhaseTime {
133134 readonly ms: number;
134135 }
135136
136 export type DrillStep = "fetch" | "restore" | "open" | "count" | "unseal" | "health";
137 export type DrillStep = "fetch" | "restore" | "open" | "count" | "unseal" | "health" | "forge";
137138
138139 export interface DrillDeps {
139140 readonly store: Store;
@@ -150,6 +151,8 @@ export interface DrillDeps {
150151 readonly boot?: typeof bootRestored | undefined;
151152 /** Told as the restore moves (phase, counts, bytes), so a job can show where it is. */
152153 readonly onProgress?: ((p: Progress) => void) | undefined;
154 /** Test seam: the proof of one hosted forge. */
155 readonly prove?: typeof proveHosted | undefined;
153156 }
154157
155158 export type { DrillCounts };
@@ -182,6 +185,11 @@ export interface DrillResult {
182185 * process.
183186 */
184187 readonly cost?: DrillCost | undefined;
188 /**
189 * The hosted forge this drill restored alone and proved to open (`drill_hosted.ts`); absent when
190 * the snapshot held none, or the drill failed before it.
191 */
192 readonly hosted?: HostedProof | undefined;
185193 }
186194
187195 /** The drill process's wall time and peak resident memory, and the memory limit it ran under. */
@@ -244,6 +252,8 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> {
244252 let restoreEnd: number | undefined;
245253 let bootStart: number | undefined;
246254 let health: DrillResult["health"] = "unreached";
255 let hosted: HostedProof | undefined;
256 let bootEnd: number | undefined;
247257 try {
248258 // One clock for the whole drill, so a hang in any step ends it red.
249259 const left = (): number => Math.max(1, started + budgetMs - tick());
@@ -332,7 +342,21 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> {
332342 budget,
333343 );
334344 health = booted.ok ? "ok" : "failed";
345 bootEnd = tick();
335346 if (!booted.ok) throw new Error(booted.reason);
347
348 // One hosted forge, alone, from the same snapshot (a restore of the host leaves them all in
349 // the store): the next one each day and each week.
350 const held = (newest.hosted ?? []).filter((h) => h.why === undefined);
351 const pick = drillPick(held, clock());
352 if (pick !== undefined) {
353 step = "forge";
354 hosted = await within(
355 (deps.prove ?? proveHosted)(store, newest.name, pick, join(scratch, "forge"), { passphrase: deps.passphrase, of: held.length, leftMs: left }),
356 left(),
357 budget,
358 );
359 }
336360 } catch (error) {
337361 reason =
338362 error instanceof StepTimeout && step === "restore" && progress !== undefined
@@ -361,7 +385,7 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> {
361385 if (restoreEnd !== undefined) {
362386 const now = tick();
363387 phases.push({ phase: "open", ms: Math.round((bootStart ?? now) - restoreEnd) });
364 if (bootStart !== undefined) phases.push({ phase: "boot", ms: Math.round(now - bootStart) });
388 if (bootStart !== undefined) phases.push({ phase: "boot", ms: Math.round((bootEnd ?? now) - bootStart) });
365389 }
366390 return {
367391 at,
@@ -377,6 +401,7 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> {
377401 from: deps.store.where,
378402 offSite: isOffBox(deps.store),
379403 ...(format === undefined ? {} : { format }),
404 ...(hosted === undefined ? {} : { hosted }),
380405 };
381406 }
382407 }
src/snapshot/drill_hosted.ts added +97
@@ -0,0 +1,97 @@
1 /**
2 * The restore drill's proof of one hosted forge (#13819): a snapshot that
3 * holds every hosted forge is a hope until one has come back. Each drill
4 * restores one of them alone (`drillPick`), from its own files and chunks
5 * (`restore.ts` `restoreHosted`), and proves it opens: its host's ledger of it
6 * reads and names it, its database passes SQLite's integrity check, and every
7 * repository its copy holds comes back and passes `git fsck`.
8 *
9 * The repositories fsck'd here were cloned by this forge's own git from the
10 * forge's bundles: they hold its objects and refs and none of its hooks or
11 * configuration, so git runs nothing of the hosted forge's. Its database is
12 * asked one thing, the integrity check, which runs no view or trigger of its.
13 */
14
15 import { Database } from "bun:sqlite";
16 import { readdirSync } from "node:fs";
17 import { join } from "node:path";
18 import { runGit } from "../git/env.ts";
19 import type { HostedEntry } from "./hosted.ts";
20 import { restoreHosted } from "./restore.ts";
21 import type { Store } from "./store.ts";
22
23 /** What a drill says of the hosted forge it restored. */
24 export interface HostedProof {
25 readonly name: string;
26 /** How many hosted forges the snapshot holds: this one stood for them. */
27 readonly of: number;
28 readonly repos: number;
29 /** The bytes its restore wrote. */
30 readonly bytes: number;
31 }
32
33 /** No fsck of one repository is given longer than this. */
34 export const FSCK_MAX_MS = 15 * 60_000;
35
36 /**
37 * The forge the drill restores from a snapshot, in name order: the next one each day, and the next
38 * one each week on the same weekday. The drill is weekly, so a turn by the day alone would give
39 * seven, or fourteen, hosted forges the same one every Sunday.
40 */
41 export function drillPick(hosted: readonly HostedEntry[], at: Date): HostedEntry | undefined {
42 const held = hosted.filter((h) => h.why === undefined).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));

The held forges are already filtered in drill.ts before this call. Filter and sort once, in one place.

43 if (held.length === 0) return undefined;
44 const day = Math.floor(at.getTime() / 86_400_000);
45 return held[(Math.floor(day / 7) + (day % 7)) % held.length];
46 }
47
48 /**
49 * Restore `forge` from snapshot `snapshot` into `dir` and prove it; throws, naming the forge and
50 * what did not hold, when it does not open.
51 */
52 export async function proveHosted(
53 store: Store,
54 snapshot: string,
55 forge: HostedEntry,
56 dir: string,
57 options: { readonly passphrase?: string | undefined; readonly of: number; readonly leftMs?: (() => number) | undefined; readonly git?: typeof runGit | undefined },
58 ): Promise<HostedProof> {
59 const git = options.git ?? runGit;
60 const back = await restoreHosted(store, snapshot, forge.name, dir, { passphrase: options.passphrase, git });
61 // Without its host's ledger of it, a host could not list it or wake it.
62 let kept: { name?: unknown };
63 try {
64 kept = JSON.parse(back.ledger) as { name?: unknown };
65 } catch {
66 kept = {};
67 }
68 if (kept.name !== forge.name) throw new Error(`hosted forge ${forge.name}: the ledger in its copy is not its own`);
69
70 const db = new Database(join(dir, "forge.db"), { readonly: true });
71 try {
72 const checked = db.query<{ integrity_check: string }, []>("PRAGMA integrity_check").all();
73 if (checked.length !== 1 || checked[0]?.integrity_check !== "ok") throw new Error(`hosted forge ${forge.name}: its database does not pass the integrity check`);
74 } finally {
75 db.close();
76 }
77
78 let repos = 0;
79 const root = join(dir, "repos");
80 const names = (at: string): string[] => {
81 try {
82 return readdirSync(at).sort();
83 } catch {
84 return [];
85 }
86 };
87 for (const owner of names(root))
88 for (const entry of names(join(root, owner))) {
89 if (!entry.endsWith(".git")) continue;
90 const timeoutMs = Math.max(1_000, Math.min(FSCK_MAX_MS, options.leftMs?.() ?? FSCK_MAX_MS));
91 const fsck = await git(["fsck", "--no-dangling", "--no-progress"], { cwd: join(root, owner, entry), timeoutMs });
92 if (!fsck.ok) throw new Error(`hosted forge ${forge.name}: repository ${owner}/${entry.slice(0, -".git".length)} does not pass git fsck`);
93 repos++;
94 }
95 if (repos !== forge.repos) throw new Error(`hosted forge ${forge.name}: ${repos} repositories came back, its copy holds ${forge.repos}`);
96 return { name: forge.name, of: options.of, repos, bytes: forge.bytes };
97 }
src/snapshot/drills.ts +20
@@ -6,6 +6,7 @@
66
77 import type { Database } from "bun:sqlite";
88 import type { DrillCost, DrillResult, DrillStep, PhaseTime } from "./drill.ts";
9 import type { HostedProof } from "./drill_hosted.ts";
910
1011 interface Row {
1112 at: string;
@@ -56,6 +57,23 @@ export function parseCost(raw: string | null): { cost?: DrillCost } {
5657 }
5758 }
5859
60 /**
61 * The hosted forge a kept drill restored and proved (#13819); absent when it kept none, or it does
62 * not read.
63 */
64 export function parseHosted(raw: string | null): { hosted?: HostedProof } {
65 try {
66 const h = (JSON.parse(raw ?? "null") as { hosted?: unknown } | null)?.hosted as Record<string, unknown> | undefined;
67 if (typeof h !== "object" || h === null) return {};
68 const whole = (v: unknown): v is number => typeof v === "number" && Number.isSafeInteger(v) && v >= 0;
69 const { name, of, repos, bytes } = h;
70 if (typeof name !== "string" || name.length > 100 || !whole(of) || !whole(repos) || !whole(bytes)) return {};
71 return { hosted: { name, of, repos, bytes } };
72 } catch {
73 return {};
74 }
75 }
76
5977 /**
6078 * The stored JSON as phases; anything that is not a list of known phases with whole ms reads as
6179 * none.
@@ -102,6 +120,7 @@ export function recordDrill(db: Database, result: DrillResult): void {
102120 budgetMs: result.budgetMs,
103121 ...(result.from === undefined ? {} : { from: result.from, offSite: result.offSite === true }),
104122 ...(result.format === undefined ? {} : { format: result.format }),
123 ...(result.hosted === undefined ? {} : { hosted: result.hosted }),
105124 ...(result.cost === undefined
106125 ? {}
107126 : { cost: { elapsedMs: Math.round(result.cost.elapsedMs), peakBytes: Math.round(result.cost.peakBytes), limitBytes: Math.round(result.cost.limitBytes) } }),
@@ -146,5 +165,6 @@ function asResult(row: Row | null): DrillResult | undefined {
146165 budgetMs: kept.budgetMs,
147166 ...parseFrom(row.phases),
148167 ...parseCost(row.phases),
168 ...parseHosted(row.phases),
149169 };
150170 }
test/mcp-backups.test.ts +31
@@ -85,6 +85,37 @@ describe("MCP backups", () => {
8585 expect(out.mirror.place).toBeNull();
8686 });
8787
88 test("the hosted forges the newest snapshot kept are said as the Backups page and /forges say them", async () => {
89 const dir = await mkdtemp(join(tmpdir(), "forge-mcp-backups-hosted-"));
90 process.env["FORGE_SNAPSHOT_DIR"] = dir;
91 const at = new Date(Date.now() - 2 * 3_600_000 - 60_000).toISOString();
92 const manifest = {
93 format: 2,
94 name: "2026-11-16T08-00-00Z",
95 startedAt: at,
96 finishedAt: at,
97 version: "t",
98 repos: 0,
99 apps: 0,
100 files: [],
101 bytes: 0,
102 hosted: [
103 { name: "kid", at, repos: 1, bytes: 9, key: "none" },
104 { name: "sad", at, repos: 0, bytes: 0, key: "none", why: "its database is not a file of its own" },
105 ],
106 };
107 await dirStore(dir).put("snapshots/2026-11-16T08-00-00Z/manifest.json", JSON.stringify(manifest));
108 const out = JSON.parse((await call(admin)).text);
109 expect(out.hosted).toEqual({
110 says: "1 hosted forge in the newest backup; 1 left out (see the log).",
111 forges: [
112 { name: "kid", says: "kept 2 h ago" },
113 { name: "sad", says: "left out of the newest backup: its database is not a file of its own" },
114 ],
115 });
116 await rm(dir, { recursive: true, force: true });
117 });
118
88119 test("a listing that fails is a short sentence with no key in it", async () => {
89120 const secret = "SECRETKEY-do-not-leak";
90121 // The store's client fails at once and echoes both keys, as a real one may:
test/snapshot-hosted-chunks.test.ts added +308
@@ -0,0 +1,308 @@
1 /**
2 * What a hosted forge's backup costs the line, how long it is kept, and that
3 * it comes back (#13819).
4 *
5 * The snapshot holds every hosted forge under `children/<name>/` through the
6 * same chunks as everything else. Proved here, on a host with hosted forges
7 * laid out as a spawn leaves them, over two nights:
8 *
9 * - two forges with the same repository store its bytes once;
10 * - a forge that did not change sends nothing but its key wrapped anew, and a
11 * changed database sends only the chunks that changed;
12 * - a removed forge leaves the next snapshot and stays in the older one;
13 * - retention never sweeps a chunk a kept snapshot's hosted forge needs;
14 * - the restore drill restores one hosted forge alone, proves it opens and
15 * says which, and fails at its own step, by name, when it does not come back;
16 * - a lane whose newest snapshot left a hosted forge out is red, naming it.
17 */
18 import { Database } from "bun:sqlite";
19 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
20 import { cpSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
21 import { rm } from "node:fs/promises";
22 import { tmpdir } from "node:os";
23 import { join } from "node:path";
24 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
25 import { createApp, type App } from "../src/app.ts";
26 import { getRepo } from "../src/core/repos.ts";
27 import { createUser, type User } from "../src/core/users.ts";
28 import { runGit } from "../src/git/env.ts";
29 import { repoDir } from "../src/git/repos.ts";
30 import { judge, readLane, type LaneRead } from "../src/snapshot/bucket.ts";
31 import { CHUNK_PREFIX, chunkKey } from "../src/snapshot/chunks.ts";
32 import { runDrill } from "../src/snapshot/drill.ts";
33 import { drillPick } from "../src/snapshot/drill_hosted.ts";
34 import { restoreHosted } from "../src/snapshot/restore.ts";
35 import { chunksNamed, pruneSnapshots, takeSnapshot, type Manifest } from "../src/snapshot/snapshot.ts";
36 import type { Store } from "../src/snapshot/store.ts";
37 import { verifySnapshot } from "../src/snapshot/verify.ts";
38 import { quietLogs } from "./helpers/globals.ts";
39
40 quietLogs();
41
42 const PASS = "correct horse battery staple";
43 const MIB = 1024 * 1024;
44
45 /** Bytes that do not compress, the same for the same seed. */
46 function noise(bytes: number, seed: number): Uint8Array {
47 const out = new Uint8Array(bytes);
48 const words = new Uint32Array(out.buffer, 0, bytes >> 2);
49 let x = seed >>> 0 || 1;
50 for (let i = 0; i < words.length; i++) {
51 x ^= x << 13;
52 x >>>= 0;
53 x ^= x >>> 17;
54 x ^= x << 5;
55 x >>>= 0;
56 words[i] = x;
57 }
58 return out;
59 }
60
61 /** A store in memory, as a bucket behaves, that remembers every put in order. */
62 function memStore(): Store & { objects: Map<string, Uint8Array>; puts: string[] } {
63 const objects = new Map<string, Uint8Array>();
64 const puts: string[] = [];
65 return {
66 where: "memory",
67 objects,
68 puts,
69 put: async (key, body) => {
70 puts.push(key);
71 objects.set(key, typeof body === "string" ? new TextEncoder().encode(body) : new Uint8Array(body));
72 },
73 get: async (key) => objects.get(key),
74 list: async (prefix) => [...objects.keys()].filter((k) => k.startsWith(prefix)),
75 names: async (prefix) => [
76 ...new Set(
77 [...objects.keys()]
78 .filter((k) => k.startsWith(prefix) && k.indexOf("/", prefix.length) >= 0)
79 .map((k) => k.slice(prefix.length, k.indexOf("/", prefix.length))),
80 ),
81 ],
82 delete: async (key) => void objects.delete(key),
83 };
84 }
85
86 /** A store of its own with the same objects: for a test that deletes, so the others read on. */
87 function copyOf(from: ReturnType<typeof memStore>): ReturnType<typeof memStore> {
88 const to = memStore();
89 for (const [key, body] of from.objects) to.objects.set(key, body);
90 return to;
91 }
92
93 const refsOf = async (dir: string): Promise<string> => {
94 const r = await runGit(["for-each-ref", "--format=%(refname) %(objectname)"], { cwd: dir });
95 if (!r.ok) throw new Error(r.error.stderr);
96 return r.value;
97 };
98
99 describe("two nights of a host with hosted forges", () => {
100 let root: string;
101 let host: App;
102 let store: ReturnType<typeof memStore>;
103 let m1: Manifest;
104 let m2: Manifest;
105 let put1: string[];
106 let put2: string[];
107 let refs = "";
108 let made = 0;
109 const fresh = (what: string): string => join(root, `${what}-${++made}`);
110 const forge = () => ({
111 db: host.db,
112 dataDir: host.config.dataDir,
113 reposRoot: host.reposRoot,
114 version: "t",
115 passphrase: PASS,
116 // Small cuts, so a file of a few MiB is many chunks.
117 contentCut: { min: 16 * 1024, bits: 15, max: 256 * 1024 },
118 fixedBytes: 64 * 1024,
119 });
120 /** One forge's files in a manifest, by their path under its prefix. */
121 const filesOf = (m: Manifest, name: string): Map<string, { chunks: readonly string[]; bytes: number }> =>
122 new Map(m.files.filter((f) => f.path.startsWith(`children/${name}/`)).map((f) => [f.path.slice(`children/${name}/`.length), { chunks: f.chunks ?? [], bytes: f.bytes }]));
123
124 async function hostForge(name: string, bodyId: string): Promise<{ dir: string; open: App; owner: User }> {
125 const dir = join(host.config.dataDir, "children", bodyId);
126 const open = createApp({ dataDir: dir, port: 0 });
127 const user = await createUser(open.db, { origin: "seed", trusted: true, username: name, email: `${name}@forge.local`, password: "a-good-passphrase", adminIfFirst: true });
128 if (!user.ok) throw new Error(user.error);
129 writeFileSync(join(dir, "secret.key"), name[0]!.repeat(32), { mode: 0o600 });
130 const repo = await createRepo(open, user.value, { name: "forge", visibility: "private" });
131 if (!repo.ok) throw new Error(String(repo.error));
132 // A table at the end of its database, large enough for many chunks.
133 open.db.run("CREATE TABLE zz_fill (id INTEGER PRIMARY KEY, body BLOB)");
134 const fill = open.db.prepare("INSERT INTO zz_fill (body) VALUES (?)");
135 for (let i = 0; i < 24; i++) fill.run(noise(32 * 1024, 100 + i + name.charCodeAt(0) * 1000));
136 writeFileSync(join(host.config.dataDir, "spawn", `${name}.json`), JSON.stringify({ name, bodyId, done: true }));
137 return { dir, open, owner: user.value };
138 }
139
140 beforeAll(async () => {
141 root = mkdtempSync(join(tmpdir(), "forge-hosted-chunks-"));
142 host = createApp({ dataDir: join(root, "host"), port: 0 });
143 const keeper = await createUser(host.db, { origin: "seed", trusted: true, username: "keeper", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
144 if (!keeper.ok) throw new Error(keeper.error);
145 mkdirSync(join(host.config.dataDir, "spawn"), { recursive: true });
146
147 // Alpha holds a repository with about three MiB of pack: the fork of the forge's code every
148 // hosted forge carries. Beta carries the same repository, object for object.
149 const alpha = await hostForge("alpha", "body-alpha");
150 const c = await commitToBranch(alpha.open, alpha.owner, getRepo(alpha.open.db, "alpha", "forge")!, {
151 message: "the forge's code",
152 files: [41, 42, 43, 44, 45].map((seed) => ({ path: `pack-${seed}.bin`, content: Buffer.from(noise(600 * 1024, seed)).toString("base64") })),
153 });
154 if (!c.ok) throw new Error(JSON.stringify(c.error));
155 const beta = await hostForge("beta", "body-beta");
156 const from = repoDir(alpha.open.reposRoot, "alpha", "forge");
157 const to = repoDir(beta.open.reposRoot, "beta", "forge");
158 for (const part of ["objects", "refs", "HEAD", "packed-refs"]) {
159 rmSync(join(to, part), { recursive: true, force: true });
160 if (existsSync(join(from, part))) cpSync(join(from, part), join(to, part), { recursive: true });
161 }
162 refs = await refsOf(from);
163 alpha.open.close();
164 beta.open.close();
165
166 store = memStore();
167 m1 = await takeSnapshot(forge(), store, new Date("2026-11-16T08:00:00Z"));
168 put1 = [...store.puts];
169
170 // The night after: alpha's database gained one row at its end and nothing else of it changed;
171 // beta was removed, its ledger and its directory with it.
172 const db = new Database(join(alpha.dir, "forge.db"));
173 db.run("INSERT INTO zz_fill (body) VALUES (?)", [noise(8 * 1024, 999)]);
174 db.close();
175 rmSync(join(host.config.dataDir, "spawn", "beta.json"));
176 rmSync(beta.dir, { recursive: true, force: true });
177 store.puts.length = 0;
178 m2 = await takeSnapshot(forge(), store, new Date("2026-11-17T08:00:00Z"));
179 put2 = [...store.puts];
180 // Two forges made, a pack committed and two snapshots taken: more than a hook's five seconds
181 // when the whole suite shares the machine.
182 }, 60_000);
183 afterAll(async () => {
184 host.close();
185 await rm(root, { recursive: true, force: true });
186 });
187
188 test("two forges with the same repository store its bytes once", () => {
189 expect(m1.hosted?.map((h) => [h.name, h.repos, h.key, h.why])).toEqual([["alpha", 1, "wrapped", undefined], ["beta", 1, "wrapped", undefined]]);
190 const a = filesOf(m1, "alpha").get("repos/alpha/forge.bundle")!;
191 const b = filesOf(m1, "beta").get("repos/beta/forge.bundle")!;
192 // More than two MiB of pack, cut into many chunks, and the very same ones for both.
193 expect(a.bytes).toBeGreaterThan(2 * MIB);
194 expect(a.chunks.length).toBeGreaterThan(10);
195 expect(b.chunks).toEqual(a.chunks);
196 // Each was put once, and no chunk of the snapshot was put twice: beta's bundle sent nothing.
197 for (const id of a.chunks) expect(put1.filter((k) => k === chunkKey(id)).length).toBe(1);
198 const chunkPuts = put1.filter((k) => k.startsWith(CHUNK_PREFIX));
199 expect(new Set(chunkPuts).size).toBe(chunkPuts.length);
200 });
201
202 test("a forge that did not change sends only its key wrapped anew, and a changed database only its changed chunks", () => {
203 const one = filesOf(m1, "alpha");
204 const two = filesOf(m2, "alpha");
205 // Everything but the database is the same chunks as the night before. (The key is wrapped
206 // anew under a fresh salt every time, as the host's own is: one small chunk.)
207 for (const [path, f] of two) if (path !== "forge.db" && path !== "secret.key.wrapped") expect([path, f.chunks]).toEqual([path, one.get(path)!.chunks]);
208 // The database: a row added to its last table changed the chunk with its header, the one with
209 // that table's root page, and the last ones. The rest of it is chunks the store already held.
210 const before = one.get("forge.db")!.chunks;
211 const after = two.get("forge.db")!.chunks;
212 expect(after.length).toBeGreaterThan(20);
213 const changed = after.filter((id) => !before.includes(id));
214 expect(changed.length).toBeGreaterThan(0);
215 expect(changed.length).toBeLessThanOrEqual(5);
216 // What went up for alpha is exactly those chunks and its key: none of its pack again.
217 for (const id of changed) expect(put2).toContain(chunkKey(id));
218 for (const id of before) expect(put2).not.toContain(chunkKey(id));
219 for (const id of two.get("repos/alpha/forge.bundle")!.chunks) expect(put2).not.toContain(chunkKey(id));
220 const mine = new Set([...two.values()].flatMap((f) => [...f.chunks]).map(chunkKey));
221 expect(put2.filter((k) => mine.has(k)).length).toBe(changed.length + 1);
222 });
223
224 test("a removed forge leaves the next snapshot, and the older one still holds it", async () => {
225 expect(m2.hosted?.map((h) => h.name)).toEqual(["alpha"]);
226 expect(m2.files.some((f) => f.path.startsWith("children/beta/"))).toBe(false);
227 const back = fresh("beta-back");
228 const done = await restoreHosted(store, m1.name, "beta", back, { passphrase: PASS });
229 expect([done.repos, done.key]).toEqual([1, "restored"]);
230 expect(await refsOf(repoDir(join(back, "repos"), "beta", "forge"))).toBe(refs);
231 await expect(restoreHosted(store, m2.name, "beta", fresh("none"))).rejects.toThrow(/does not hold beta/);
232 });
233
234 test("retention never sweeps a chunk a kept snapshot's hosted forge needs", async () => {
235 // On a store of its own: the other tests read the one both snapshots are in.
236 const swept = copyOf(store);
237 // Both kept: beta, removed since, is still whole in the older one.
238 expect(await pruneSnapshots(swept, 2, Date.parse("2026-11-17T09:00:00Z"), true)).toEqual([]);
239 for (const m of [m1, m2]) for (const id of chunksNamed(m)) expect(swept.objects.has(chunkKey(id))).toBe(true);
240 // Only the newer kept: the older manifest goes, and with it what only beta needed.
241 const onlyBeta = [...filesOf(m1, "beta").get("forge.db")!.chunks].filter((id) => !chunksNamed(m2).includes(id));
242 expect(onlyBeta.length).toBeGreaterThan(0);
243 expect(await pruneSnapshots(swept, 1, Date.parse("2026-11-17T09:00:00Z"), true)).toEqual([m1.name]);
244 for (const id of onlyBeta) expect(swept.objects.has(chunkKey(id))).toBe(false);
245 // Every chunk the kept snapshot names is there, the pack alpha shared with beta among them,
246 // and nothing else is.
247 for (const id of chunksNamed(m2)) expect(swept.objects.has(chunkKey(id))).toBe(true);
248 expect([...swept.objects.keys()].filter((k) => k.startsWith(CHUNK_PREFIX)).length).toBe(chunksNamed(m2).length);
249 await verifySnapshot(swept, m2.name, { deep: true });
250 const back = fresh("alpha-after-sweep");
251 await restoreHosted(swept, m2.name, "alpha", back, { passphrase: PASS });
252 expect(await refsOf(repoDir(join(back, "repos"), "alpha", "forge"))).toBe(refs);
253 const db = new Database(join(back, "forge.db"), { readonly: true });
254 expect((db.query("SELECT count(*) AS n FROM zz_fill").get() as { n: number }).n).toBe(25);
255 db.close();
256 });
257
258 test("the drill restores one hosted forge alone, proves it opens, and says which", async () => {
259 const at = new Date("2026-11-17T12:00:00Z");
260 const result = await runDrill({ store, dataDir: fresh("drill"), passphrase: PASS, now: () => at, boot: async () => ({ ok: true, reason: "" }) });
261 expect(result.reason).toBeNull();
262 expect(result.verdict).toBe("green");
263 expect(result.hosted).toEqual({ name: "alpha", of: 1, repos: 1, bytes: m2.hosted![0]!.bytes });
264 // The board says the forge the drill brought back.
265 const now = at.getTime() + 3_600_000;
266 const lane = await readLane("primary", store, now);
267 expect(lane.hosted).toEqual({ count: 1, failed: [] });
268 expect(judge({ lanes: [lane], checkedAt: new Date(now).toISOString() }, result, now).drill?.says).toBe("restore drill 2026-11-17 passed (1 h ago), with hosted forge alpha (1 of 1)");
269 });
270
271 test("a hosted forge that does not come back whole fails the drill at its own step, by name", async () => {
272 // One chunk only that forge's database needs is gone from a store of its own.
273 const holed = copyOf(store);
274 const lost = filesOf(m2, "alpha").get("forge.db")!.chunks.at(-1)!;
275 holed.objects.delete(chunkKey(lost));
276 const red = await runDrill({ store: holed, dataDir: fresh("drill-red"), passphrase: PASS, now: () => new Date("2026-11-17T12:00:00Z"), boot: async () => ({ ok: true, reason: "" }) });
277 // The host's own files do not name that chunk, so the host restored; the forge did not.
278 expect([red.verdict, red.failedStep]).toEqual(["red", "forge"]);
279 expect(red.reason).toMatch(new RegExp(`missing chunk ${lost} of forge.db`));
280 expect(red.hosted).toBeUndefined();
281 });
282
283 test("the drill's forge turns by the day and by the week, and never to one that was left out", () => {
284 const entry = m2.hosted![0]!;
285 const seven = ["a", "b", "c", "d", "e", "f", "g"].map((name) => ({ ...entry, name }));
286 expect(drillPick(seven, new Date("2026-11-17T12:00:00Z"))?.name).not.toBe(drillPick(seven, new Date("2026-11-18T12:00:00Z"))?.name);
287 // Seven forges drilled every Sunday are each drilled in turn.
288 const sundays = [0, 1, 2, 3, 4, 5, 6].map((week) => drillPick(seven, new Date(Date.parse("2026-11-22T09:00:00Z") + week * 7 * 86_400_000))!.name);
289 expect([...sundays].sort()).toEqual(["a", "b", "c", "d", "e", "f", "g"]);
290 expect(drillPick([{ ...entry, why: "left out" }], new Date())).toBeUndefined();
291 expect(drillPick([], new Date())).toBeUndefined();
292 });
293
294 test("a lane whose newest snapshot left a hosted forge out is red, naming it; a repository its copy lacks is said, not an alarm", () => {
295 const now = Date.parse("2026-11-17T09:00:00Z");
296 const lane: LaneRead = { lane: "primary", where: "memory", offBox: false, newestAt: "2026-11-17T08:05:00Z", newestName: "2026-11-17T08-00-00Z", newestBytes: 9, count: 1, readAt: new Date(now).toISOString() };
297 const out = judge({ lanes: [{ ...lane, hosted: { count: 2, failed: [{ name: "sad", why: "its database is not a file of its own" }] } }], checkedAt: new Date(now).toISOString() }, undefined, now);
298 expect(out.state).toBe("red");
299 expect(out.note).toBe("hosted forges missing");
300 expect(out.reasons).toEqual(["Primary (memory) newest snapshot 2026-11-17T08-00-00Z lacks 1 hosted forge, which a restore would not bring back: sad (its database is not a file of its own)."]);
301 const amber = judge({ lanes: [{ ...lane, hosted: { count: 2, failed: [], absent: [{ forge: "kid", repo: "kid/app", why: "it holds a link" }] } }], checkedAt: new Date(now).toISOString() }, undefined, now);
302 expect(amber.lanes[0]?.state).toBe("green");
303 expect(amber.state).toBe("unknown");
304 expect(amber.unknowns).toContain("Primary: 1 repository of hosted forges is not in it: kid: kid/app (it holds a link).");
305 // A lane with every hosted forge whole is as it was.
306 expect(judge({ lanes: [{ ...lane, hosted: { count: 2, failed: [] } }], checkedAt: new Date(now).toISOString() }, undefined, now).lanes[0]?.state).toBe("green");
307 });
308 });
test/worker-database-opens.test.ts +1
@@ -64,6 +64,7 @@ const REACHED: Record<string, { opens: number; why: string }> = {
6464 "src/kernel/db.ts": { opens: 2, why: "the forge's own database, opened once by createApp on the thread that boots; the Workers that read it open their own handle and keep it (KEEPS)" },
6565 "src/release/rollback.ts": { opens: 1, why: "liveSchema reads the schema count once for a rollback or a swap, on the serving thread or in the release tool" },
6666 "src/snapshot/drill.ts": { opens: 1, why: "a restore drill opens the restored copy once, as a duty of the serving thread" },
67 "src/snapshot/drill_hosted.ts": { opens: 1, why: "a restore drill opens the one hosted forge it restored once, for its integrity check, in the drill's own process or as a duty of the serving thread; no Worker calls it" },
6768 "src/snapshot/restore.ts": { opens: 1, why: "a restore reads the sealed names of the restored copy once, in the restore tool or a drill on the serving thread" },
6869 "src/snapshot/snapshot.ts": { opens: 1, why: "a snapshot counts the rows of its own copy once, as a duty of the serving thread (a hosted forge's copy is read in a process of its own, snapshot/hosted_copy.ts); the mirror Worker imports this file for the manifest only" },
6970 "src/spawn/child_runs.ts": { opens: 2, why: "called by the reader process (spawn/child_db_reader.ts), never by a thread of the forge; cell.ts re-exports the two for tests that read a file directly" },

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs