Public: anyone can read this; the crew can change it.
The restore drill restores one hosted forge and says which, and a snapshot that left one out is red #13923
Check · Review · Land · Live done — Landed and live in production.
trevato/backup-hosted-forges → main 1 commit 3 threads · 3 unresolved
10 files
changed +562
−12
docs/RECOVERY.md +5 −5
| @@ -88,15 +88,15 @@ which line it stops at. | ||
| 88 | 88 | Likely a bad snapshot or a slow store. Check the phase, then Route 1. |
| 89 | 89 | - **open** — opening the restored `forge.db`. Likely the file is truncated or |
| 90 | 90 | corrupt. Check its size against the snapshot's manifest. |
| 91 | - **count** — counting rows and comparing with the counts the snapshot kept. | |
| 92 | Likely the snapshot was cut mid-write or rows were lost. Check which table | |
| 93 | the message names and its count in the snapshot. | |
| 91 | - **count** — counting rows against the counts the snapshot kept. Likely it was | |
| 92 | cut mid-write or rows were lost. Check the table the message names there. | |
| 94 | 93 | - **unseal** — opening one sealed value with the restored key. Likely the key |
| 95 | 94 | is missing or from another forge, or the passphrase is wrong. Check |
| 96 | 95 | `FORGE_SNAPSHOT_PASSPHRASE` and that the snapshot holds the key. |
| 97 | 96 | - **health** — booting the restored forge and asking it for health. Likely a |
| 98 | migration or config problem, or a boot slower than the limit. Boot it by | |
| 99 | hand as in Route 1 and read its log. | |
| 97 | migration or config problem, or a slow boot. Boot it by hand (Route 1); read its log. | |
| 98 | - **forge** — restoring one hosted forge alone and proving it opens (ledger, integrity check, | |
| 99 | `git fsck`). Likely a chunk only it needs is gone. It is named: Route 1 with `--forge <name>`. | |
| 100 | 100 | |
| 101 | 101 | ### Restore phases |
| 102 | 102 |
docs/features/hosted-backup.md +6
| @@ -27,6 +27,12 @@ A hosted forge's directory is the forge's own to write, and its host reads it fo | ||
| 27 | 27 | - Resilience screen (terminal): the same line, then one line per forge: kept H h ago, or left out and why. |
| 28 | 28 | - `/forges`: each row, "kept <age> ago" (under an hour, hours, then days) or "never kept" when no backup holds it, or why the newest left it out. |
| 29 | 29 | - `forges` MCP tool and `GET /api/v1/forges`: the same words as `kept` on each made forge. |
| 30 | - MCP `backups`: `hosted`, the same sentence and each forge's line. | |
| 31 | - The backup verdict (home, vitals, MCP `backups`, the Resilience screen): a lane whose newest snapshot left a hosted forge out is red and names it with why. A repository a forge's copy lacks is said there too, as an unknown and not an alarm. | |
| 32 | ||
| 33 | ## The drill restores one | |
| 34 | ||
| 35 | Each restore drill, after the host itself, restores one hosted forge alone from the same snapshot (the next one each day and each week, `src/snapshot/drill_hosted.ts`) and proves it opens: its ledger reads and names it, its database passes SQLite's integrity check, and every repository its copy holds comes back and passes `git fsck`. The verdict says which forge ("…passed, with hosted forge glint (1 of 17)"). One that does not come back fails the drill at the step `forge`, by name ([](../RECOVERY.md)). | |
| 30 | 36 | |
| 31 | 37 | ## Restore one forge |
| 32 | 38 |
src/http/mcp/tools/backups.ts +11 −4
| @@ -4,13 +4,14 @@ import { bodiesOf } from "../../../snapshot/store.ts"; | ||
| 4 | 4 | import { mirrorNeed } from "../../../snapshot/mirror.ts"; |
| 5 | 5 | import { s3FromEnv } from "../../../snapshot/store.ts"; |
| 6 | 6 | import { listSnapshots } from "../../../snapshot/snapshot.ts"; |
| 7 | import { hostedWords, keptWords } from "../../../snapshot/hosted.ts"; | |
| 7 | 8 | import { backupVerdict } from "../../../snapshot/bucket.ts"; |
| 8 | 9 | import { lastDrill } from "../../../snapshot/drills.ts"; |
| 9 | 10 | import { good, bad, type Tool } from "../shared.ts"; |
| 10 | 11 | |
| 11 | 12 | export const tool: Tool = { |
| 12 | 13 | name: "backups", |
| 13 | doc: "where snapshots go (`destination`: its `place`, `bodies` (whether a snapshot's data is encrypted before upload, in words), and `credentials` (where the bucket's keys are kept: sealed, dir or env; it says nothing of the data), never a credential; `null` with the Backups page's sentence when none), `verdict` (state, line, reasons, lanes with newest object and `bodies`, drill: read from the bucket, red past 26 h or when the newest snapshot lacks a repository), the last fourteen snapshots (`key`, `at`, `bytes`) and the off-box `mirror`'s state; a forge admin's, as /settings/backups", | |
| 14 | doc: "where snapshots go (`destination`: its `place`, `bodies` (whether a snapshot's data is encrypted before upload, in words), and `credentials` (where the bucket's keys are kept: sealed, dir or env; it says nothing of the data), never a credential; `null` with the Backups page's sentence when none), `verdict` (state, line, reasons, lanes with newest object and `bodies`, drill: read from the bucket, red past 26 h or when the newest snapshot lacks a repository or a hosted forge), the last fourteen snapshots (`key`, `at`, `bytes`), `hosted` (the forges this one hosts in the newest snapshot: `says` as the Backups page says it, and each forge's `says` as /forges does; absent when it names none) and the off-box `mirror`'s state; a forge admin's, as /settings/backups", | |
| 14 | 15 | description: "Admin: snapshot destination, recent, mirror.", |
| 15 | 16 | inputSchema: { type: "object", properties: {} }, |
| 16 | 17 | writes: false, |
| @@ -39,10 +40,15 @@ export const tool: Tool = { | ||
| 39 | 40 | }); |
| 40 | 41 | let snapshots: { key: string; at: string; bytes: number }[] = []; |
| 41 | 42 | let listing: string | undefined; |
| 43 | // The hosted forges the newest snapshot kept, in the Backups page's words and /forges' | |
| 44 | // (#13819). | |
| 45 | let hosted: { says: string; forges: { name: string; says: string }[] } | undefined; | |
| 42 | 46 | try { |
| 43 | snapshots = (await listSnapshots(dest.store)) | |
| 44 | .slice(0, 14) | |
| 45 | .map((m) => ({ key: m.name, at: m.finishedAt, bytes: m.bytes })); | |
| 47 | const all = await listSnapshots(dest.store); | |
| 48 | snapshots = all.slice(0, 14).map((m) => ({ key: m.name, at: m.finishedAt, bytes: m.bytes })); | |
| 49 | const kept = all[0]?.hosted; | |
| 50 | const says = hostedWords(kept); | |
| 51 | if (kept !== undefined && says !== undefined) hosted = { says, forges: kept.map((h) => ({ name: h.name, says: keptWords(kept, h.name, Date.now()) })) }; | |
| 46 | 52 | } catch (error) { |
| 47 | 53 | const sources = { db: ctx.app.db, dataDir: ctx.app.config.dataDir, bus: ctx.app.bus }; |
| 48 | 54 | const target = dest.source === "sealed" ? sealedTarget(sources) : dest.source === "env" ? s3FromEnv() : undefined; |
| @@ -53,6 +59,7 @@ export const tool: Tool = { | ||
| 53 | 59 | // `credentials` is where the bucket's keys are kept; `bodies` is what the data itself gets. |
| 54 | 60 | destination: { place: dest.store.where, bodies: bodiesOf(dest.store), credentials: dest.source }, |
| 55 | 61 | snapshots, |
| 62 | ...(hosted === undefined ? {} : { hosted }), | |
| 56 | 63 | ...(listing === undefined ? {} : { listing }), |
| 57 | 64 | mirror, |
| 58 | 65 | }); |
src/snapshot/bucket.ts +56 −1
| @@ -19,6 +19,7 @@ import { join } from "node:path"; | ||
| 19 | 19 | import type { DrillResult } from "./drill.ts"; |
| 20 | 20 | import { isOffBox } from "./off_box.ts"; |
| 21 | 21 | import { listSnapshots, notInSnapshot } from "./snapshot.ts"; |
| 22 | import type { HostedEntry } from "./hosted.ts"; | |
| 22 | 23 | import { scrubReason } from "./destination.ts"; |
| 23 | 24 | import { bodiesOf, type S3Target, type Store } from "./store.ts"; |
| 24 | 25 | |
| @@ -47,6 +48,11 @@ export interface LaneRead { | ||
| 47 | 48 | readonly bodies?: string; |
| 48 | 49 | /** Repositories the newest snapshot says it holds no bundle of though they held something. */ |
| 49 | 50 | readonly absent?: readonly string[]; |
| 51 | /** | |
| 52 | * The hosted forges in the newest snapshot (#13819): how many it holds a copy of, and those it | |
| 53 | * names as left out, each with why. Absent when it names none. | |
| 54 | */ | |
| 55 | readonly hosted?: HostedRead; | |
| 50 | 56 | /** When this read ended. */ |
| 51 | 57 | readonly readAt: string; |
| 52 | 58 | /** What the lane backs up, one sentence, from the configuration (#9067). */ |
| @@ -55,6 +61,25 @@ export interface LaneRead { | ||
| 55 | 61 | readonly afterSpinDown?: string; |
| 56 | 62 | } |
| 57 | 63 | |
| 64 | /** What a lane's newest snapshot says of the forges its forge hosts. */ | |
| 65 | export interface HostedRead { | |
| 66 | readonly count: number; | |
| 67 | readonly failed: readonly { readonly name: string; readonly why: string }[]; | |
| 68 | /** | |
| 69 | * Repositories of the forges it does hold that their copies hold no bundle of though they held | |
| 70 | * something, each with why; absent when there is none. | |
| 71 | */ | |
| 72 | readonly absent?: readonly { readonly forge: string; readonly repo: string; readonly why: string }[]; | |
| 73 | } | |
| 74 | ||
| 75 | /** "2 repositories of hosted forges are not in it: glint: glint/app (why); …"; empty when none. */ | |
| 76 | function hostedAbsentWords(h: HostedRead): string { | |
| 77 | const out = h.absent ?? []; | |
| 78 | if (out.length === 0) return ""; | |
| 79 | const named = out.slice(0, 8).map((a) => `${printable(a.forge)}: ${printable(a.repo)} (${printable(a.why)})`).join("; "); | |
| 80 | return `${out.length} ${out.length === 1 ? "repository" : "repositories"} of hosted forges ${out.length === 1 ? "is" : "are"} not in it: ${named}${out.length > 8 ? `; and ${out.length - 8} more` : ""}`; | |
| 81 | } | |
| 82 | ||
| 58 | 83 | /** `retired` in any case: the lane is listed, never read, never red. */ |
| 59 | 84 | export const isRetired = (afterSpinDown: string | undefined): boolean => afterSpinDown?.trim().toLowerCase() === "retired"; |
| 60 | 85 | |
| @@ -194,9 +219,23 @@ export async function readLane( | ||
| 194 | 219 | const all = await listSnapshots(store); |
| 195 | 220 | const newest = all[0]; |
| 196 | 221 | const absent = newest === undefined ? [] : notInSnapshot(newest).map((a) => printable(a.repo)); |
| 222 | const forges: readonly HostedEntry[] = (newest?.hosted ?? []).filter((f) => typeof f === "object" && f !== null && typeof f.name === "string"); | |
| 223 | const held = forges.filter((f) => f.why === undefined); | |
| 224 | const left = held.flatMap((f) => | |
| 225 | (f.absent ?? []).filter((a) => a.empty !== true).map((a) => ({ forge: f.name, repo: String(a.repo), why: String(a.why).slice(0, 200) })), | |
| 226 | ); | |
| 227 | const hosted: HostedRead | undefined = | |
| 228 | forges.length === 0 | |
| 229 | ? undefined | |
| 230 | : { | |
| 231 | count: held.length, | |
| 232 | failed: forges.filter((f) => f.why !== undefined).map((f) => ({ name: f.name, why: String(f.why).slice(0, 200) })), | |
| 233 | ...(left.length === 0 ? {} : { absent: left }), | |
| 234 | }; | |
| 197 | 235 | return { |
| 198 | 236 | ...base, |
| 199 | 237 | ...(absent.length === 0 ? {} : { absent }), |
| 238 | ...(hosted === undefined ? {} : { hosted }), | |
| 200 | 239 | newestAt: newest?.finishedAt ?? null, |
| 201 | 240 | newestName: newest?.name ?? null, |
| 202 | 241 | newestBytes: newest?.bytes ?? null, |
| @@ -275,6 +314,14 @@ export function judge( | ||
| 275 | 314 | else if (r.absent !== undefined && r.absent.length > 0) { |
| 276 | 315 | const n = r.absent.length; |
| 277 | 316 | says = `${lane} (${where}) newest snapshot${name} lacks ${n} ${n === 1 ? "repository" : "repositories"}, which a restore would not bring back: ${r.absent.slice(0, 8).join(", ")}${n > 8 ? `, and ${n - 8} more` : ""}`; |
| 317 | } | |
| 318 | // Nor is one that lacks a forge this one hosts: a person's whole forge would not come back. | |
| 319 | else if (r.hosted !== undefined && r.hosted.failed.length > 0) { | |
| 320 | const n = r.hosted.failed.length; | |
| 321 | says = `${lane} (${where}) newest snapshot${name} lacks ${n} hosted ${n === 1 ? "forge" : "forges"}, which a restore would not bring back: ${r.hosted.failed | |
| 322 | .slice(0, 8) | |
| 323 | .map((f) => `${printable(f.name)} (${printable(f.why)})`) | |
| 324 | .join("; ")}${n > 8 ? `; and ${n - 8} more` : ""}`; | |
| 278 | 325 | } else { |
| 279 | 326 | red = false; |
| 280 | 327 | says = `${lane} newest${name} ${ageWords(age)} old, ${sizeWords(r.newestBytes ?? 0)}`; |
| @@ -288,10 +335,16 @@ export function judge( | ||
| 288 | 335 | ? "empty" |
| 289 | 336 | : !(now - Date.parse(r.newestAt) <= LATE_MS) |
| 290 | 337 | ? "late" |
| 291 | : "repositories missing", | |
| 338 | : r.absent !== undefined && r.absent.length > 0 | |
| 339 | ? "repositories missing" | |
| 340 | : "hosted forges missing", | |
| 292 | 341 | ); |
| 293 | 342 | reasons.push(`${says[0]!.toUpperCase()}${says.slice(1)}.`); |
| 294 | 343 | } |
| 344 | // A repository of a hosted forge that its copy lacks is said and is not calm, and is not this | |
| 345 | // forge's alarm either: what it holds is its owner's doing, night after night. | |
| 346 | if (r.hosted !== undefined && r.error === undefined && (r.hosted.absent ?? []).length > 0) | |
| 347 | unknowns.push(`${lane[0]!.toUpperCase()}${lane.slice(1)}: ${hostedAbsentWords(r.hosted)}.`); | |
| 295 | 348 | lanes.push({ ...r, backsUp, afterSpinDown, state: red ? "red" : "green", says, line }); |
| 296 | 349 | } |
| 297 | 350 | if (state !== undefined && !lanes.some((l) => l.state !== "retired")) { |
| @@ -309,6 +362,8 @@ export function judge( | ||
| 309 | 362 | // The proof's age, said: "passed" for six days is not the same as passed today (#12592). |
| 310 | 363 | const proofAge = ageWords(Math.max(0, now - Date.parse(drill.at))); |
| 311 | 364 | if (ok) says = `${says} (${proofAge} ago)`; |
| 365 | // And the hosted forge it restored alone, by name: one of them has come back (#13819). | |
| 366 | if (ok && drill.hosted !== undefined) says = `${says}, with hosted forge ${printable(drill.hosted.name)} (1 of ${drill.hosted.of})`; | |
| 312 | 367 | drillSays = { at: drill.at, ok, says }; |
| 313 | 368 | if (!ok) { |
| 314 | 369 | causes.push(drill.verdict === "green" ? "drill restored nothing" : "drill failed"); |
src/snapshot/drill.ts +27 −2
| @@ -19,6 +19,7 @@ import { bootRestored } from "./boot.ts"; | ||
| 19 | 19 | import { forgetKey, open } from "../kernel/secret.ts"; |
| 20 | 20 | import { countRows, mismatched, type DrillCounts } from "./counts.ts"; |
| 21 | 21 | import { restoreSnapshot, type Progress } from "./restore.ts"; |
| 22 | import { drillPick, proveHosted, type HostedProof } from "./drill_hosted.ts"; | |
| 22 | 23 | import { isOffBox } from "./off_box.ts"; |
| 23 | 24 | import { listSnapshots } from "./snapshot.ts"; |
| 24 | 25 | import type { Store } from "./store.ts"; |
| @@ -133,7 +134,7 @@ export interface PhaseTime { | ||
| 133 | 134 | readonly ms: number; |
| 134 | 135 | } |
| 135 | 136 | |
| 136 | export type DrillStep = "fetch" | "restore" | "open" | "count" | "unseal" | "health"; | |
| 137 | export type DrillStep = "fetch" | "restore" | "open" | "count" | "unseal" | "health" | "forge"; | |
| 137 | 138 | |
| 138 | 139 | export interface DrillDeps { |
| 139 | 140 | readonly store: Store; |
| @@ -150,6 +151,8 @@ export interface DrillDeps { | ||
| 150 | 151 | readonly boot?: typeof bootRestored | undefined; |
| 151 | 152 | /** Told as the restore moves (phase, counts, bytes), so a job can show where it is. */ |
| 152 | 153 | readonly onProgress?: ((p: Progress) => void) | undefined; |
| 154 | /** Test seam: the proof of one hosted forge. */ | |
| 155 | readonly prove?: typeof proveHosted | undefined; | |
| 153 | 156 | } |
| 154 | 157 | |
| 155 | 158 | export type { DrillCounts }; |
| @@ -182,6 +185,11 @@ export interface DrillResult { | ||
| 182 | 185 | * process. |
| 183 | 186 | */ |
| 184 | 187 | readonly cost?: DrillCost | undefined; |
| 188 | /** | |
| 189 | * The hosted forge this drill restored alone and proved to open (`drill_hosted.ts`); absent when | |
| 190 | * the snapshot held none, or the drill failed before it. | |
| 191 | */ | |
| 192 | readonly hosted?: HostedProof | undefined; | |
| 185 | 193 | } |
| 186 | 194 | |
| 187 | 195 | /** The drill process's wall time and peak resident memory, and the memory limit it ran under. */ |
| @@ -244,6 +252,8 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> { | ||
| 244 | 252 | let restoreEnd: number | undefined; |
| 245 | 253 | let bootStart: number | undefined; |
| 246 | 254 | let health: DrillResult["health"] = "unreached"; |
| 255 | let hosted: HostedProof | undefined; | |
| 256 | let bootEnd: number | undefined; | |
| 247 | 257 | try { |
| 248 | 258 | // One clock for the whole drill, so a hang in any step ends it red. |
| 249 | 259 | const left = (): number => Math.max(1, started + budgetMs - tick()); |
| @@ -332,7 +342,21 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> { | ||
| 332 | 342 | budget, |
| 333 | 343 | ); |
| 334 | 344 | health = booted.ok ? "ok" : "failed"; |
| 345 | bootEnd = tick(); | |
| 335 | 346 | if (!booted.ok) throw new Error(booted.reason); |
| 347 | ||
| 348 | // One hosted forge, alone, from the same snapshot (a restore of the host leaves them all in | |
| 349 | // the store): the next one each day and each week. | |
| 350 | const held = (newest.hosted ?? []).filter((h) => h.why === undefined); | |
| 351 | const pick = drillPick(held, clock()); | |
| 352 | if (pick !== undefined) { | |
| 353 | step = "forge"; | |
| 354 | hosted = await within( | |
| 355 | (deps.prove ?? proveHosted)(store, newest.name, pick, join(scratch, "forge"), { passphrase: deps.passphrase, of: held.length, leftMs: left }), | |
| 356 | left(), | |
| 357 | budget, | |
| 358 | ); | |
| 359 | } | |
| 336 | 360 | } catch (error) { |
| 337 | 361 | reason = |
| 338 | 362 | error instanceof StepTimeout && step === "restore" && progress !== undefined |
| @@ -361,7 +385,7 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> { | ||
| 361 | 385 | if (restoreEnd !== undefined) { |
| 362 | 386 | const now = tick(); |
| 363 | 387 | phases.push({ phase: "open", ms: Math.round((bootStart ?? now) - restoreEnd) }); |
| 364 | if (bootStart !== undefined) phases.push({ phase: "boot", ms: Math.round(now - bootStart) }); | |
| 388 | if (bootStart !== undefined) phases.push({ phase: "boot", ms: Math.round((bootEnd ?? now) - bootStart) }); | |
| 365 | 389 | } |
| 366 | 390 | return { |
| 367 | 391 | at, |
| @@ -377,6 +401,7 @@ export async function runDrill(deps: DrillDeps): Promise<DrillResult> { | ||
| 377 | 401 | from: deps.store.where, |
| 378 | 402 | offSite: isOffBox(deps.store), |
| 379 | 403 | ...(format === undefined ? {} : { format }), |
| 404 | ...(hosted === undefined ? {} : { hosted }), | |
| 380 | 405 | }; |
| 381 | 406 | } |
| 382 | 407 | } |
src/snapshot/drill_hosted.ts added +97
| @@ -0,0 +1,97 @@ | ||
| 1 | /** | |
| 2 | * The restore drill's proof of one hosted forge (#13819): a snapshot that | |
| 3 | * holds every hosted forge is a hope until one has come back. Each drill | |
| 4 | * restores one of them alone (`drillPick`), from its own files and chunks | |
| 5 | * (`restore.ts` `restoreHosted`), and proves it opens: its host's ledger of it | |
| 6 | * reads and names it, its database passes SQLite's integrity check, and every | |
| 7 | * repository its copy holds comes back and passes `git fsck`. | |
| 8 | * | |
| 9 | * The repositories fsck'd here were cloned by this forge's own git from the | |
| 10 | * forge's bundles: they hold its objects and refs and none of its hooks or | |
| 11 | * configuration, so git runs nothing of the hosted forge's. Its database is | |
| 12 | * asked one thing, the integrity check, which runs no view or trigger of its. | |
| 13 | */ | |
| 14 | ||
| 15 | import { Database } from "bun:sqlite"; | |
| 16 | import { readdirSync } from "node:fs"; | |
| 17 | import { join } from "node:path"; | |
| 18 | import { runGit } from "../git/env.ts"; | |
| 19 | import type { HostedEntry } from "./hosted.ts"; | |
| 20 | import { restoreHosted } from "./restore.ts"; | |
| 21 | import type { Store } from "./store.ts"; | |
| 22 | ||
| 23 | /** What a drill says of the hosted forge it restored. */ | |
| 24 | export interface HostedProof { | |
| 25 | readonly name: string; | |
| 26 | /** How many hosted forges the snapshot holds: this one stood for them. */ | |
| 27 | readonly of: number; | |
| 28 | readonly repos: number; | |
| 29 | /** The bytes its restore wrote. */ | |
| 30 | readonly bytes: number; | |
| 31 | } | |
| 32 | ||
| 33 | /** No fsck of one repository is given longer than this. */ | |
| 34 | export const FSCK_MAX_MS = 15 * 60_000; | |
| 35 | ||
| 36 | /** | |
| 37 | * The forge the drill restores from a snapshot, in name order: the next one each day, and the next | |
| 38 | * one each week on the same weekday. The drill is weekly, so a turn by the day alone would give | |
| 39 | * seven, or fourteen, hosted forges the same one every Sunday. | |
| 40 | */ | |
| 41 | export function drillPick(hosted: readonly HostedEntry[], at: Date): HostedEntry | undefined { | |
| 42 | const held = hosted.filter((h) => h.why === undefined).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)); | |
The held forges are already filtered in |
||
| 43 | if (held.length === 0) return undefined; | |
| 44 | const day = Math.floor(at.getTime() / 86_400_000); | |
| 45 | return held[(Math.floor(day / 7) + (day % 7)) % held.length]; | |
| 46 | } | |
| 47 | ||
| 48 | /** | |
| 49 | * Restore `forge` from snapshot `snapshot` into `dir` and prove it; throws, naming the forge and | |
| 50 | * what did not hold, when it does not open. | |
| 51 | */ | |
| 52 | export async function proveHosted( | |
| 53 | store: Store, | |
| 54 | snapshot: string, | |
| 55 | forge: HostedEntry, | |
| 56 | dir: string, | |
| 57 | options: { readonly passphrase?: string | undefined; readonly of: number; readonly leftMs?: (() => number) | undefined; readonly git?: typeof runGit | undefined }, | |
| 58 | ): Promise<HostedProof> { | |
| 59 | const git = options.git ?? runGit; | |
| 60 | const back = await restoreHosted(store, snapshot, forge.name, dir, { passphrase: options.passphrase, git }); | |
| 61 | // Without its host's ledger of it, a host could not list it or wake it. | |
| 62 | let kept: { name?: unknown }; | |
| 63 | try { | |
| 64 | kept = JSON.parse(back.ledger) as { name?: unknown }; | |
| 65 | } catch { | |
| 66 | kept = {}; | |
| 67 | } | |
| 68 | if (kept.name !== forge.name) throw new Error(`hosted forge ${forge.name}: the ledger in its copy is not its own`); | |
| 69 | ||
| 70 | const db = new Database(join(dir, "forge.db"), { readonly: true }); | |
| 71 | try { | |
| 72 | const checked = db.query<{ integrity_check: string }, []>("PRAGMA integrity_check").all(); | |
| 73 | if (checked.length !== 1 || checked[0]?.integrity_check !== "ok") throw new Error(`hosted forge ${forge.name}: its database does not pass the integrity check`); | |
| 74 | } finally { | |
| 75 | db.close(); | |
| 76 | } | |
| 77 | ||
| 78 | let repos = 0; | |
| 79 | const root = join(dir, "repos"); | |
| 80 | const names = (at: string): string[] => { | |
| 81 | try { | |
| 82 | return readdirSync(at).sort(); | |
| 83 | } catch { | |
| 84 | return []; | |
| 85 | } | |
| 86 | }; | |
| 87 | for (const owner of names(root)) | |
| 88 | for (const entry of names(join(root, owner))) { | |
| 89 | if (!entry.endsWith(".git")) continue; | |
| 90 | const timeoutMs = Math.max(1_000, Math.min(FSCK_MAX_MS, options.leftMs?.() ?? FSCK_MAX_MS)); | |
| 91 | const fsck = await git(["fsck", "--no-dangling", "--no-progress"], { cwd: join(root, owner, entry), timeoutMs }); | |
| 92 | if (!fsck.ok) throw new Error(`hosted forge ${forge.name}: repository ${owner}/${entry.slice(0, -".git".length)} does not pass git fsck`); | |
| 93 | repos++; | |
| 94 | } | |
| 95 | if (repos !== forge.repos) throw new Error(`hosted forge ${forge.name}: ${repos} repositories came back, its copy holds ${forge.repos}`); | |
| 96 | return { name: forge.name, of: options.of, repos, bytes: forge.bytes }; | |
| 97 | } |
src/snapshot/drills.ts +20
| @@ -6,6 +6,7 @@ | ||
| 6 | 6 | |
| 7 | 7 | import type { Database } from "bun:sqlite"; |
| 8 | 8 | import type { DrillCost, DrillResult, DrillStep, PhaseTime } from "./drill.ts"; |
| 9 | import type { HostedProof } from "./drill_hosted.ts"; | |
| 9 | 10 | |
| 10 | 11 | interface Row { |
| 11 | 12 | at: string; |
| @@ -56,6 +57,23 @@ export function parseCost(raw: string | null): { cost?: DrillCost } { | ||
| 56 | 57 | } |
| 57 | 58 | } |
| 58 | 59 | |
| 60 | /** | |
| 61 | * The hosted forge a kept drill restored and proved (#13819); absent when it kept none, or it does | |
| 62 | * not read. | |
| 63 | */ | |
| 64 | export function parseHosted(raw: string | null): { hosted?: HostedProof } { | |
| 65 | try { | |
| 66 | const h = (JSON.parse(raw ?? "null") as { hosted?: unknown } | null)?.hosted as Record<string, unknown> | undefined; | |
| 67 | if (typeof h !== "object" || h === null) return {}; | |
| 68 | const whole = (v: unknown): v is number => typeof v === "number" && Number.isSafeInteger(v) && v >= 0; | |
| 69 | const { name, of, repos, bytes } = h; | |
| 70 | if (typeof name !== "string" || name.length > 100 || !whole(of) || !whole(repos) || !whole(bytes)) return {}; | |
| 71 | return { hosted: { name, of, repos, bytes } }; | |
| 72 | } catch { | |
| 73 | return {}; | |
| 74 | } | |
| 75 | } | |
| 76 | ||
| 59 | 77 | /** |
| 60 | 78 | * The stored JSON as phases; anything that is not a list of known phases with whole ms reads as |
| 61 | 79 | * none. |
| @@ -102,6 +120,7 @@ export function recordDrill(db: Database, result: DrillResult): void { | ||
| 102 | 120 | budgetMs: result.budgetMs, |
| 103 | 121 | ...(result.from === undefined ? {} : { from: result.from, offSite: result.offSite === true }), |
| 104 | 122 | ...(result.format === undefined ? {} : { format: result.format }), |
| 123 | ...(result.hosted === undefined ? {} : { hosted: result.hosted }), | |
| 105 | 124 | ...(result.cost === undefined |
| 106 | 125 | ? {} |
| 107 | 126 | : { cost: { elapsedMs: Math.round(result.cost.elapsedMs), peakBytes: Math.round(result.cost.peakBytes), limitBytes: Math.round(result.cost.limitBytes) } }), |
| @@ -146,5 +165,6 @@ function asResult(row: Row | null): DrillResult | undefined { | ||
| 146 | 165 | budgetMs: kept.budgetMs, |
| 147 | 166 | ...parseFrom(row.phases), |
| 148 | 167 | ...parseCost(row.phases), |
| 168 | ...parseHosted(row.phases), | |
| 149 | 169 | }; |
| 150 | 170 | } |
test/mcp-backups.test.ts +31
| @@ -85,6 +85,37 @@ describe("MCP backups", () => { | ||
| 85 | 85 | expect(out.mirror.place).toBeNull(); |
| 86 | 86 | }); |
| 87 | 87 | |
| 88 | test("the hosted forges the newest snapshot kept are said as the Backups page and /forges say them", async () => { | |
| 89 | const dir = await mkdtemp(join(tmpdir(), "forge-mcp-backups-hosted-")); | |
| 90 | process.env["FORGE_SNAPSHOT_DIR"] = dir; | |
| 91 | const at = new Date(Date.now() - 2 * 3_600_000 - 60_000).toISOString(); | |
| 92 | const manifest = { | |
| 93 | format: 2, | |
| 94 | name: "2026-11-16T08-00-00Z", | |
| 95 | startedAt: at, | |
| 96 | finishedAt: at, | |
| 97 | version: "t", | |
| 98 | repos: 0, | |
| 99 | apps: 0, | |
| 100 | files: [], | |
| 101 | bytes: 0, | |
| 102 | hosted: [ | |
| 103 | { name: "kid", at, repos: 1, bytes: 9, key: "none" }, | |
| 104 | { name: "sad", at, repos: 0, bytes: 0, key: "none", why: "its database is not a file of its own" }, | |
| 105 | ], | |
| 106 | }; | |
| 107 | await dirStore(dir).put("snapshots/2026-11-16T08-00-00Z/manifest.json", JSON.stringify(manifest)); | |
| 108 | const out = JSON.parse((await call(admin)).text); | |
| 109 | expect(out.hosted).toEqual({ | |
| 110 | says: "1 hosted forge in the newest backup; 1 left out (see the log).", | |
| 111 | forges: [ | |
| 112 | { name: "kid", says: "kept 2 h ago" }, | |
| 113 | { name: "sad", says: "left out of the newest backup: its database is not a file of its own" }, | |
| 114 | ], | |
| 115 | }); | |
| 116 | await rm(dir, { recursive: true, force: true }); | |
| 117 | }); | |
| 118 | ||
| 88 | 119 | test("a listing that fails is a short sentence with no key in it", async () => { |
| 89 | 120 | const secret = "SECRETKEY-do-not-leak"; |
| 90 | 121 | // The store's client fails at once and echoes both keys, as a real one may: |
test/snapshot-hosted-chunks.test.ts added +308
| @@ -0,0 +1,308 @@ | ||
| 1 | /** | |
| 2 | * What a hosted forge's backup costs the line, how long it is kept, and that | |
| 3 | * it comes back (#13819). | |
| 4 | * | |
| 5 | * The snapshot holds every hosted forge under `children/<name>/` through the | |
| 6 | * same chunks as everything else. Proved here, on a host with hosted forges | |
| 7 | * laid out as a spawn leaves them, over two nights: | |
| 8 | * | |
| 9 | * - two forges with the same repository store its bytes once; | |
| 10 | * - a forge that did not change sends nothing but its key wrapped anew, and a | |
| 11 | * changed database sends only the chunks that changed; | |
| 12 | * - a removed forge leaves the next snapshot and stays in the older one; | |
| 13 | * - retention never sweeps a chunk a kept snapshot's hosted forge needs; | |
| 14 | * - the restore drill restores one hosted forge alone, proves it opens and | |
| 15 | * says which, and fails at its own step, by name, when it does not come back; | |
| 16 | * - a lane whose newest snapshot left a hosted forge out is red, naming it. | |
| 17 | */ | |
| 18 | import { Database } from "bun:sqlite"; | |
| 19 | import { afterAll, beforeAll, describe, expect, test } from "bun:test"; | |
| 20 | import { cpSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; | |
| 21 | import { rm } from "node:fs/promises"; | |
| 22 | import { tmpdir } from "node:os"; | |
| 23 | import { join } from "node:path"; | |
| 24 | import { commitToBranch, createRepo } from "../src/actions/repos.ts"; | |
| 25 | import { createApp, type App } from "../src/app.ts"; | |
| 26 | import { getRepo } from "../src/core/repos.ts"; | |
| 27 | import { createUser, type User } from "../src/core/users.ts"; | |
| 28 | import { runGit } from "../src/git/env.ts"; | |
| 29 | import { repoDir } from "../src/git/repos.ts"; | |
| 30 | import { judge, readLane, type LaneRead } from "../src/snapshot/bucket.ts"; | |
| 31 | import { CHUNK_PREFIX, chunkKey } from "../src/snapshot/chunks.ts"; | |
| 32 | import { runDrill } from "../src/snapshot/drill.ts"; | |
| 33 | import { drillPick } from "../src/snapshot/drill_hosted.ts"; | |
| 34 | import { restoreHosted } from "../src/snapshot/restore.ts"; | |
| 35 | import { chunksNamed, pruneSnapshots, takeSnapshot, type Manifest } from "../src/snapshot/snapshot.ts"; | |
| 36 | import type { Store } from "../src/snapshot/store.ts"; | |
| 37 | import { verifySnapshot } from "../src/snapshot/verify.ts"; | |
| 38 | import { quietLogs } from "./helpers/globals.ts"; | |
| 39 | ||
| 40 | quietLogs(); | |
| 41 | ||
| 42 | const PASS = "correct horse battery staple"; | |
| 43 | const MIB = 1024 * 1024; | |
| 44 | ||
| 45 | /** Bytes that do not compress, the same for the same seed. */ | |
| 46 | function noise(bytes: number, seed: number): Uint8Array { | |
| 47 | const out = new Uint8Array(bytes); | |
| 48 | const words = new Uint32Array(out.buffer, 0, bytes >> 2); | |
| 49 | let x = seed >>> 0 || 1; | |
| 50 | for (let i = 0; i < words.length; i++) { | |
| 51 | x ^= x << 13; | |
| 52 | x >>>= 0; | |
| 53 | x ^= x >>> 17; | |
| 54 | x ^= x << 5; | |
| 55 | x >>>= 0; | |
| 56 | words[i] = x; | |
| 57 | } | |
| 58 | return out; | |
| 59 | } | |
| 60 | ||
| 61 | /** A store in memory, as a bucket behaves, that remembers every put in order. */ | |
| 62 | function memStore(): Store & { objects: Map<string, Uint8Array>; puts: string[] } { | |
| 63 | const objects = new Map<string, Uint8Array>(); | |
| 64 | const puts: string[] = []; | |
| 65 | return { | |
| 66 | where: "memory", | |
| 67 | objects, | |
| 68 | puts, | |
| 69 | put: async (key, body) => { | |
| 70 | puts.push(key); | |
| 71 | objects.set(key, typeof body === "string" ? new TextEncoder().encode(body) : new Uint8Array(body)); | |
| 72 | }, | |
| 73 | get: async (key) => objects.get(key), | |
| 74 | list: async (prefix) => [...objects.keys()].filter((k) => k.startsWith(prefix)), | |
| 75 | names: async (prefix) => [ | |
| 76 | ...new Set( | |
| 77 | [...objects.keys()] | |
| 78 | .filter((k) => k.startsWith(prefix) && k.indexOf("/", prefix.length) >= 0) | |
| 79 | .map((k) => k.slice(prefix.length, k.indexOf("/", prefix.length))), | |
| 80 | ), | |
| 81 | ], | |
| 82 | delete: async (key) => void objects.delete(key), | |
| 83 | }; | |
| 84 | } | |
| 85 | ||
| 86 | /** A store of its own with the same objects: for a test that deletes, so the others read on. */ | |
| 87 | function copyOf(from: ReturnType<typeof memStore>): ReturnType<typeof memStore> { | |
| 88 | const to = memStore(); | |
| 89 | for (const [key, body] of from.objects) to.objects.set(key, body); | |
| 90 | return to; | |
| 91 | } | |
| 92 | ||
| 93 | const refsOf = async (dir: string): Promise<string> => { | |
| 94 | const r = await runGit(["for-each-ref", "--format=%(refname) %(objectname)"], { cwd: dir }); | |
| 95 | if (!r.ok) throw new Error(r.error.stderr); | |
| 96 | return r.value; | |
| 97 | }; | |
| 98 | ||
| 99 | describe("two nights of a host with hosted forges", () => { | |
| 100 | let root: string; | |
| 101 | let host: App; | |
| 102 | let store: ReturnType<typeof memStore>; | |
| 103 | let m1: Manifest; | |
| 104 | let m2: Manifest; | |
| 105 | let put1: string[]; | |
| 106 | let put2: string[]; | |
| 107 | let refs = ""; | |
| 108 | let made = 0; | |
| 109 | const fresh = (what: string): string => join(root, `${what}-${++made}`); | |
| 110 | const forge = () => ({ | |
| 111 | db: host.db, | |
| 112 | dataDir: host.config.dataDir, | |
| 113 | reposRoot: host.reposRoot, | |
| 114 | version: "t", | |
| 115 | passphrase: PASS, | |
| 116 | // Small cuts, so a file of a few MiB is many chunks. | |
| 117 | contentCut: { min: 16 * 1024, bits: 15, max: 256 * 1024 }, | |
| 118 | fixedBytes: 64 * 1024, | |
| 119 | }); | |
| 120 | /** One forge's files in a manifest, by their path under its prefix. */ | |
| 121 | const filesOf = (m: Manifest, name: string): Map<string, { chunks: readonly string[]; bytes: number }> => | |
| 122 | new Map(m.files.filter((f) => f.path.startsWith(`children/${name}/`)).map((f) => [f.path.slice(`children/${name}/`.length), { chunks: f.chunks ?? [], bytes: f.bytes }])); | |
| 123 | ||
| 124 | async function hostForge(name: string, bodyId: string): Promise<{ dir: string; open: App; owner: User }> { | |
| 125 | const dir = join(host.config.dataDir, "children", bodyId); | |
| 126 | const open = createApp({ dataDir: dir, port: 0 }); | |
| 127 | const user = await createUser(open.db, { origin: "seed", trusted: true, username: name, email: `${name}@forge.local`, password: "a-good-passphrase", adminIfFirst: true }); | |
| 128 | if (!user.ok) throw new Error(user.error); | |
| 129 | writeFileSync(join(dir, "secret.key"), name[0]!.repeat(32), { mode: 0o600 }); | |
| 130 | const repo = await createRepo(open, user.value, { name: "forge", visibility: "private" }); | |
| 131 | if (!repo.ok) throw new Error(String(repo.error)); | |
| 132 | // A table at the end of its database, large enough for many chunks. | |
| 133 | open.db.run("CREATE TABLE zz_fill (id INTEGER PRIMARY KEY, body BLOB)"); | |
| 134 | const fill = open.db.prepare("INSERT INTO zz_fill (body) VALUES (?)"); | |
| 135 | for (let i = 0; i < 24; i++) fill.run(noise(32 * 1024, 100 + i + name.charCodeAt(0) * 1000)); | |
| 136 | writeFileSync(join(host.config.dataDir, "spawn", `${name}.json`), JSON.stringify({ name, bodyId, done: true })); | |
| 137 | return { dir, open, owner: user.value }; | |
| 138 | } | |
| 139 | ||
| 140 | beforeAll(async () => { | |
| 141 | root = mkdtempSync(join(tmpdir(), "forge-hosted-chunks-")); | |
| 142 | host = createApp({ dataDir: join(root, "host"), port: 0 }); | |
| 143 | const keeper = await createUser(host.db, { origin: "seed", trusted: true, username: "keeper", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true }); | |
| 144 | if (!keeper.ok) throw new Error(keeper.error); | |
| 145 | mkdirSync(join(host.config.dataDir, "spawn"), { recursive: true }); | |
| 146 | ||
| 147 | // Alpha holds a repository with about three MiB of pack: the fork of the forge's code every | |
| 148 | // hosted forge carries. Beta carries the same repository, object for object. | |
| 149 | const alpha = await hostForge("alpha", "body-alpha"); | |
| 150 | const c = await commitToBranch(alpha.open, alpha.owner, getRepo(alpha.open.db, "alpha", "forge")!, { | |
| 151 | message: "the forge's code", | |
| 152 | files: [41, 42, 43, 44, 45].map((seed) => ({ path: `pack-${seed}.bin`, content: Buffer.from(noise(600 * 1024, seed)).toString("base64") })), | |
| 153 | }); | |
| 154 | if (!c.ok) throw new Error(JSON.stringify(c.error)); | |
| 155 | const beta = await hostForge("beta", "body-beta"); | |
| 156 | const from = repoDir(alpha.open.reposRoot, "alpha", "forge"); | |
| 157 | const to = repoDir(beta.open.reposRoot, "beta", "forge"); | |
| 158 | for (const part of ["objects", "refs", "HEAD", "packed-refs"]) { | |
| 159 | rmSync(join(to, part), { recursive: true, force: true }); | |
| 160 | if (existsSync(join(from, part))) cpSync(join(from, part), join(to, part), { recursive: true }); | |
| 161 | } | |
| 162 | refs = await refsOf(from); | |
| 163 | alpha.open.close(); | |
| 164 | beta.open.close(); | |
| 165 | ||
| 166 | store = memStore(); | |
| 167 | m1 = await takeSnapshot(forge(), store, new Date("2026-11-16T08:00:00Z")); | |
| 168 | put1 = [...store.puts]; | |
| 169 | ||
| 170 | // The night after: alpha's database gained one row at its end and nothing else of it changed; | |
| 171 | // beta was removed, its ledger and its directory with it. | |
| 172 | const db = new Database(join(alpha.dir, "forge.db")); | |
| 173 | db.run("INSERT INTO zz_fill (body) VALUES (?)", [noise(8 * 1024, 999)]); | |
| 174 | db.close(); | |
| 175 | rmSync(join(host.config.dataDir, "spawn", "beta.json")); | |
| 176 | rmSync(beta.dir, { recursive: true, force: true }); | |
| 177 | store.puts.length = 0; | |
| 178 | m2 = await takeSnapshot(forge(), store, new Date("2026-11-17T08:00:00Z")); | |
| 179 | put2 = [...store.puts]; | |
| 180 | // Two forges made, a pack committed and two snapshots taken: more than a hook's five seconds | |
| 181 | // when the whole suite shares the machine. | |
| 182 | }, 60_000); | |
| 183 | afterAll(async () => { | |
| 184 | host.close(); | |
| 185 | await rm(root, { recursive: true, force: true }); | |
| 186 | }); | |
| 187 | ||
| 188 | test("two forges with the same repository store its bytes once", () => { | |
| 189 | expect(m1.hosted?.map((h) => [h.name, h.repos, h.key, h.why])).toEqual([["alpha", 1, "wrapped", undefined], ["beta", 1, "wrapped", undefined]]); | |
| 190 | const a = filesOf(m1, "alpha").get("repos/alpha/forge.bundle")!; | |
| 191 | const b = filesOf(m1, "beta").get("repos/beta/forge.bundle")!; | |
| 192 | // More than two MiB of pack, cut into many chunks, and the very same ones for both. | |
| 193 | expect(a.bytes).toBeGreaterThan(2 * MIB); | |
| 194 | expect(a.chunks.length).toBeGreaterThan(10); | |
| 195 | expect(b.chunks).toEqual(a.chunks); | |
| 196 | // Each was put once, and no chunk of the snapshot was put twice: beta's bundle sent nothing. | |
| 197 | for (const id of a.chunks) expect(put1.filter((k) => k === chunkKey(id)).length).toBe(1); | |
| 198 | const chunkPuts = put1.filter((k) => k.startsWith(CHUNK_PREFIX)); | |
| 199 | expect(new Set(chunkPuts).size).toBe(chunkPuts.length); | |
| 200 | }); | |
| 201 | ||
| 202 | test("a forge that did not change sends only its key wrapped anew, and a changed database only its changed chunks", () => { | |
| 203 | const one = filesOf(m1, "alpha"); | |
| 204 | const two = filesOf(m2, "alpha"); | |
| 205 | // Everything but the database is the same chunks as the night before. (The key is wrapped | |
| 206 | // anew under a fresh salt every time, as the host's own is: one small chunk.) | |
| 207 | for (const [path, f] of two) if (path !== "forge.db" && path !== "secret.key.wrapped") expect([path, f.chunks]).toEqual([path, one.get(path)!.chunks]); | |
| 208 | // The database: a row added to its last table changed the chunk with its header, the one with | |
| 209 | // that table's root page, and the last ones. The rest of it is chunks the store already held. | |
| 210 | const before = one.get("forge.db")!.chunks; | |
| 211 | const after = two.get("forge.db")!.chunks; | |
| 212 | expect(after.length).toBeGreaterThan(20); | |
| 213 | const changed = after.filter((id) => !before.includes(id)); | |
| 214 | expect(changed.length).toBeGreaterThan(0); | |
| 215 | expect(changed.length).toBeLessThanOrEqual(5); | |
| 216 | // What went up for alpha is exactly those chunks and its key: none of its pack again. | |
| 217 | for (const id of changed) expect(put2).toContain(chunkKey(id)); | |
| 218 | for (const id of before) expect(put2).not.toContain(chunkKey(id)); | |
| 219 | for (const id of two.get("repos/alpha/forge.bundle")!.chunks) expect(put2).not.toContain(chunkKey(id)); | |
| 220 | const mine = new Set([...two.values()].flatMap((f) => [...f.chunks]).map(chunkKey)); | |
| 221 | expect(put2.filter((k) => mine.has(k)).length).toBe(changed.length + 1); | |
| 222 | }); | |
| 223 | ||
| 224 | test("a removed forge leaves the next snapshot, and the older one still holds it", async () => { | |
| 225 | expect(m2.hosted?.map((h) => h.name)).toEqual(["alpha"]); | |
| 226 | expect(m2.files.some((f) => f.path.startsWith("children/beta/"))).toBe(false); | |
| 227 | const back = fresh("beta-back"); | |
| 228 | const done = await restoreHosted(store, m1.name, "beta", back, { passphrase: PASS }); | |
| 229 | expect([done.repos, done.key]).toEqual([1, "restored"]); | |
| 230 | expect(await refsOf(repoDir(join(back, "repos"), "beta", "forge"))).toBe(refs); | |
| 231 | await expect(restoreHosted(store, m2.name, "beta", fresh("none"))).rejects.toThrow(/does not hold beta/); | |
| 232 | }); | |
| 233 | ||
| 234 | test("retention never sweeps a chunk a kept snapshot's hosted forge needs", async () => { | |
| 235 | // On a store of its own: the other tests read the one both snapshots are in. | |
| 236 | const swept = copyOf(store); | |
| 237 | // Both kept: beta, removed since, is still whole in the older one. | |
| 238 | expect(await pruneSnapshots(swept, 2, Date.parse("2026-11-17T09:00:00Z"), true)).toEqual([]); | |
| 239 | for (const m of [m1, m2]) for (const id of chunksNamed(m)) expect(swept.objects.has(chunkKey(id))).toBe(true); | |
| 240 | // Only the newer kept: the older manifest goes, and with it what only beta needed. | |
| 241 | const onlyBeta = [...filesOf(m1, "beta").get("forge.db")!.chunks].filter((id) => !chunksNamed(m2).includes(id)); | |
| 242 | expect(onlyBeta.length).toBeGreaterThan(0); | |
| 243 | expect(await pruneSnapshots(swept, 1, Date.parse("2026-11-17T09:00:00Z"), true)).toEqual([m1.name]); | |
| 244 | for (const id of onlyBeta) expect(swept.objects.has(chunkKey(id))).toBe(false); | |
| 245 | // Every chunk the kept snapshot names is there, the pack alpha shared with beta among them, | |
| 246 | // and nothing else is. | |
| 247 | for (const id of chunksNamed(m2)) expect(swept.objects.has(chunkKey(id))).toBe(true); | |
| 248 | expect([...swept.objects.keys()].filter((k) => k.startsWith(CHUNK_PREFIX)).length).toBe(chunksNamed(m2).length); | |
| 249 | await verifySnapshot(swept, m2.name, { deep: true }); | |
| 250 | const back = fresh("alpha-after-sweep"); | |
| 251 | await restoreHosted(swept, m2.name, "alpha", back, { passphrase: PASS }); | |
| 252 | expect(await refsOf(repoDir(join(back, "repos"), "alpha", "forge"))).toBe(refs); | |
| 253 | const db = new Database(join(back, "forge.db"), { readonly: true }); | |
| 254 | expect((db.query("SELECT count(*) AS n FROM zz_fill").get() as { n: number }).n).toBe(25); | |
| 255 | db.close(); | |
| 256 | }); | |
| 257 | ||
| 258 | test("the drill restores one hosted forge alone, proves it opens, and says which", async () => { | |
| 259 | const at = new Date("2026-11-17T12:00:00Z"); | |
| 260 | const result = await runDrill({ store, dataDir: fresh("drill"), passphrase: PASS, now: () => at, boot: async () => ({ ok: true, reason: "" }) }); | |
| 261 | expect(result.reason).toBeNull(); | |
| 262 | expect(result.verdict).toBe("green"); | |
| 263 | expect(result.hosted).toEqual({ name: "alpha", of: 1, repos: 1, bytes: m2.hosted![0]!.bytes }); | |
| 264 | // The board says the forge the drill brought back. | |
| 265 | const now = at.getTime() + 3_600_000; | |
| 266 | const lane = await readLane("primary", store, now); | |
| 267 | expect(lane.hosted).toEqual({ count: 1, failed: [] }); | |
| 268 | expect(judge({ lanes: [lane], checkedAt: new Date(now).toISOString() }, result, now).drill?.says).toBe("restore drill 2026-11-17 passed (1 h ago), with hosted forge alpha (1 of 1)"); | |
| 269 | }); | |
| 270 | ||
| 271 | test("a hosted forge that does not come back whole fails the drill at its own step, by name", async () => { | |
| 272 | // One chunk only that forge's database needs is gone from a store of its own. | |
| 273 | const holed = copyOf(store); | |
| 274 | const lost = filesOf(m2, "alpha").get("forge.db")!.chunks.at(-1)!; | |
| 275 | holed.objects.delete(chunkKey(lost)); | |
| 276 | const red = await runDrill({ store: holed, dataDir: fresh("drill-red"), passphrase: PASS, now: () => new Date("2026-11-17T12:00:00Z"), boot: async () => ({ ok: true, reason: "" }) }); | |
| 277 | // The host's own files do not name that chunk, so the host restored; the forge did not. | |
| 278 | expect([red.verdict, red.failedStep]).toEqual(["red", "forge"]); | |
| 279 | expect(red.reason).toMatch(new RegExp(`missing chunk ${lost} of forge.db`)); | |
| 280 | expect(red.hosted).toBeUndefined(); | |
| 281 | }); | |
| 282 | ||
| 283 | test("the drill's forge turns by the day and by the week, and never to one that was left out", () => { | |
| 284 | const entry = m2.hosted![0]!; | |
| 285 | const seven = ["a", "b", "c", "d", "e", "f", "g"].map((name) => ({ ...entry, name })); | |
| 286 | expect(drillPick(seven, new Date("2026-11-17T12:00:00Z"))?.name).not.toBe(drillPick(seven, new Date("2026-11-18T12:00:00Z"))?.name); | |
| 287 | // Seven forges drilled every Sunday are each drilled in turn. | |
| 288 | const sundays = [0, 1, 2, 3, 4, 5, 6].map((week) => drillPick(seven, new Date(Date.parse("2026-11-22T09:00:00Z") + week * 7 * 86_400_000))!.name); | |
| 289 | expect([...sundays].sort()).toEqual(["a", "b", "c", "d", "e", "f", "g"]); | |
| 290 | expect(drillPick([{ ...entry, why: "left out" }], new Date())).toBeUndefined(); | |
| 291 | expect(drillPick([], new Date())).toBeUndefined(); | |
| 292 | }); | |
| 293 | ||
| 294 | test("a lane whose newest snapshot left a hosted forge out is red, naming it; a repository its copy lacks is said, not an alarm", () => { | |
| 295 | const now = Date.parse("2026-11-17T09:00:00Z"); | |
| 296 | const lane: LaneRead = { lane: "primary", where: "memory", offBox: false, newestAt: "2026-11-17T08:05:00Z", newestName: "2026-11-17T08-00-00Z", newestBytes: 9, count: 1, readAt: new Date(now).toISOString() }; | |
| 297 | const out = judge({ lanes: [{ ...lane, hosted: { count: 2, failed: [{ name: "sad", why: "its database is not a file of its own" }] } }], checkedAt: new Date(now).toISOString() }, undefined, now); | |
| 298 | expect(out.state).toBe("red"); | |
| 299 | expect(out.note).toBe("hosted forges missing"); | |
| 300 | expect(out.reasons).toEqual(["Primary (memory) newest snapshot 2026-11-17T08-00-00Z lacks 1 hosted forge, which a restore would not bring back: sad (its database is not a file of its own)."]); | |
| 301 | const amber = judge({ lanes: [{ ...lane, hosted: { count: 2, failed: [], absent: [{ forge: "kid", repo: "kid/app", why: "it holds a link" }] } }], checkedAt: new Date(now).toISOString() }, undefined, now); | |
| 302 | expect(amber.lanes[0]?.state).toBe("green"); | |
| 303 | expect(amber.state).toBe("unknown"); | |
| 304 | expect(amber.unknowns).toContain("Primary: 1 repository of hosted forges is not in it: kid: kid/app (it holds a link)."); | |
| 305 | // A lane with every hosted forge whole is as it was. | |
| 306 | expect(judge({ lanes: [{ ...lane, hosted: { count: 2, failed: [] } }], checkedAt: new Date(now).toISOString() }, undefined, now).lanes[0]?.state).toBe("green"); | |
| 307 | }); | |
| 308 | }); |
test/worker-database-opens.test.ts +1
| @@ -64,6 +64,7 @@ const REACHED: Record<string, { opens: number; why: string }> = { | ||
| 64 | 64 | "src/kernel/db.ts": { opens: 2, why: "the forge's own database, opened once by createApp on the thread that boots; the Workers that read it open their own handle and keep it (KEEPS)" }, |
| 65 | 65 | "src/release/rollback.ts": { opens: 1, why: "liveSchema reads the schema count once for a rollback or a swap, on the serving thread or in the release tool" }, |
| 66 | 66 | "src/snapshot/drill.ts": { opens: 1, why: "a restore drill opens the restored copy once, as a duty of the serving thread" }, |
| 67 | "src/snapshot/drill_hosted.ts": { opens: 1, why: "a restore drill opens the one hosted forge it restored once, for its integrity check, in the drill's own process or as a duty of the serving thread; no Worker calls it" }, | |
| 67 | 68 | "src/snapshot/restore.ts": { opens: 1, why: "a restore reads the sealed names of the restored copy once, in the restore tool or a drill on the serving thread" }, |
| 68 | 69 | "src/snapshot/snapshot.ts": { opens: 1, why: "a snapshot counts the rows of its own copy once, as a duty of the serving thread (a hosted forge's copy is read in a process of its own, snapshot/hosted_copy.ts); the mirror Worker imports this file for the manifest only" }, |
| 69 | 70 | "src/spawn/child_runs.ts": { opens: 2, why: "called by the reader process (spawn/child_db_reader.ts), never by a thread of the forge; cell.ts re-exports the two for tests that read a file directly" }, |
new above
offline
This line is far too long. Wrap it;
bun scripts/gate.ts --fixrewraps comments, but this is code, so wrap it by hand.