trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

Nightly off-box backups fit the uplink: the upload's deadline follows its size, in parts, compressed #2538

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/2536-nightly-off-box-backups-fit-the-uplink-t → main 3 commits fork 3 threads · 3 unresolved

  • Merged. torque/2536-nightly-off-box-backups-fit-the-uplink-t was merged into main as cf49857.

Review

Sign in to review this change.
14 files changed +898 −49
docs/SNAPSHOT.md +37 −4
@@ -8,7 +8,23 @@ empty data directory. `src/snapshot/`.
88 ## What a snapshot holds
99
1010 - `forge.db`, copied consistently while the forge runs (`VACUUM INTO`,
11 never a raw copy of a file in WAL mode).
11 never a raw copy of a file in WAL mode), then zstd-compressed and cut in
12 16 MB parts: `forge.db.zst.0000`, `forge.db.zst.0001`, … Each part is its
13 own put with its own deadline and retries, so a slow uplink carries it
14 and a dropped connection costs one part, not the database. The manifest's
15 `packed` says what the parts make and how big that is; a restore joins
16 them, unpacks, and refuses a size that does not match. The log line
17 `database packed` gives each night's ratio. Snapshots from before this
18 hold a plain `forge.db` and still restore. The parts go up through the
19 snapshot's upload queue like every other object: at most eight objects
20 or 64 MB in flight, so four 16 MB parts at a time.
21
22 Packing and restoring both hold the database in memory: the copy, its
23 compressed form (the parts are views of it, not copies), and on restore
24 the joined parts and the unpacked file, about three times the database
25 at the peak (a 334 MB database at about 1 GB). A database that outgrows
26 the app's memory limit needs a streaming pack; until then that limit,
27 not the uplink, is the ceiling.
1228 - The key the database's sealed values need, **only wrapped**:
1329 `secret.key.wrapped`, AES-256-GCM under a key scrypt-derived from the
1430 operator's passphrase. The raw `secret.key` is never written to the store,
@@ -24,8 +40,8 @@ empty data directory. `src/snapshot/`.
2440 stay out.
2541 - The crew's run logs (`runs/`).
2642 - `manifest.json`: the name, when, the forge's version, counts, every file
27 with its size, and `key`: `wrapped` or `none` (older snapshots: `raw`). A
28 restore checks each size.
43 with its size, `packed` (the files that went up in parts), and `key`:
44 `wrapped` or `none` (older snapshots: `raw`). A restore checks each size.
2945
3046 What stays out is what a boot or a deploy rebuilds: clones, checks,
3147 pictures, previews.
@@ -86,7 +102,24 @@ mirror is copied from the primary store, not written in parallel, so a snapshot
86102 that never landed on the primary is not seen by it.
87103
88104 The store is `Bun.S3Client` or a directory behind one small interface
89 (`src/snapshot/store.ts`): put, get, list, delete. Nothing else.
105 (`src/snapshot/store.ts`): put, get, list, delete. Nothing else. Every call
106 is tried five times; an attempt that has not answered by its deadline counts
107 as failed. A put's deadline follows its size: its bytes at a floor of
108 5 Mbit/s, plus 60 s. A fixed 180 s once failed every attempt at a 334 MB
109 database behind a 15 Mbit/s uplink (#2536).
110
111 ## When a snapshot fails
112
113 Its outcome is kept in `snapshot/state.json` under the data directory, so a
114 restart forgets nothing. A failure shows straight away as a card on the board
115 (`snapshot-failed`, pointing at Backups), in the terminal's dashboard among
116 the forge's needs, and as a red line on the Backups page with the reason and
117 since when. The forge tries again within the hour, not the next night. The
118 next snapshot that lands, nightly or on demand, clears both.
119
120 A forge given nowhere to put a snapshot has not failed: nothing is kept, no
121 card is raised, and nothing is retried sooner than the nightly. The Backups
122 page says there is no destination.
90123
91124 ## The restore drill
92125
src/kernel/vitals.ts +5
@@ -23,6 +23,7 @@ import {
2323 import { STALE_AFTER_MS } from "../runs/lease.ts";
2424 import { BUDGETS } from "./budgets.ts";
2525 import { lastMirror, mirrorNeed } from "../snapshot/mirror.ts";
26 import { lastSnapshotOutcome, snapshotNeed } from "../snapshot/outcome.ts";
2627 import { drillNeed, firstSeen as drillFirstSeen, lastVerdict } from "../snapshot/verdict.ts";
2728 import { newestDrills } from "../core/domain_drills.ts";
2829 import { DOMAINS } from "./domains.ts";
@@ -384,6 +385,10 @@ export function vitals(input: VitalsInput): Vitals {
384385 ))
385386 need(owed.cause, owed.sentence, owed.since, "/");
386387
388 // A snapshot that failed is a card from that morning until one lands.
389 const failing = snapshotNeed(lastSnapshotOutcome(input.dataDir));
390 if (failing !== undefined) need("snapshot-failed", failing.sentence, failing.since, "/settings/backups");
391
387392 // A snapshot that did not land off the box, or a mirror a day behind, is a card until it does.
388393 const behind = mirrorNeed(lastMirror(input.dataDir), now);
389394 if (behind !== undefined) need("mirror", behind.sentence, behind.since, "/settings/backups");
src/snapshot/outcome.ts added +70
@@ -0,0 +1,70 @@
1 /**
2 * How the last snapshot went, kept where the board can read it. A nightly
3 * that failed used to be one line in the log and nothing else: two nights
4 * of no off-box backup went by unseen (2026-10-17, #2536). Each attempt's
5 * outcome is one small file under the data directory
6 * (`snapshot/state.json`), written when it ends, so vitals and Backups read
7 * it without the network and a restart forgets nothing. A failure is a
8 * card from the moment it is written until a snapshot lands.
9 */
10
11 import { mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
12 import { join } from "node:path";
13
14 export interface SnapshotOutcome {
15 /** When the last attempt ended. */
16 readonly at: string;
17 /** The newest snapshot that landed, and when; absent until one has. */
18 readonly landed?: string;
19 readonly landedAt?: string;
20 /** Set while attempts are failing: since when, and the last one's reason. */
21 readonly failingSince?: string;
22 readonly error?: string;
23 }
24
25 const stateFile = (dataDir: string): string => join(dataDir, "snapshot", "state.json");
26
27 /** The outcome kept, or undefined when none is or the file is not one. */
28 export function lastSnapshotOutcome(dataDir: string): SnapshotOutcome | undefined {
29 try {
30 const s = JSON.parse(readFileSync(stateFile(dataDir), "utf8")) as SnapshotOutcome;
31 if (typeof s.at !== "string") return undefined;
32 return s;
33 } catch {
34 return undefined;
35 }
36 }
37
38 /** Keep an attempt's end: a name that landed, or the error it failed with. Returns what was kept. */
39 export function recordSnapshotOutcome(
40 dataDir: string,
41 result: { readonly landed: string } | { readonly error: string },
42 at: string = new Date().toISOString(),
43 ): SnapshotOutcome {
44 const prev = lastSnapshotOutcome(dataDir);
45 const kept = {
46 ...(prev?.landed === undefined ? {} : { landed: prev.landed }),
47 ...(prev?.landedAt === undefined ? {} : { landedAt: prev.landedAt }),
48 };
49 const next: SnapshotOutcome =
50 "landed" in result
51 ? { at, landed: result.landed, landedAt: at }
52 : { at, ...kept, failingSince: prev?.failingSince ?? at, error: result.error };
53 mkdirSync(join(dataDir, "snapshot"), { recursive: true });
54 const tmp = `${stateFile(dataDir)}.tmp`;
55 writeFileSync(tmp, JSON.stringify(next));
56 renameSync(tmp, stateFile(dataDir));
57 return next;
58 }
59
60 /** The card a failing snapshot is, or undefined while the last one landed. */
61 export function snapshotNeed(
62 s: SnapshotOutcome | undefined,
63 ): { sentence: string; since: string } | undefined {
64 if (s?.failingSince === undefined) return undefined;
65 const last = s.landedAt ? ` The newest that landed is from ${s.landedAt.slice(0, 10)}.` : "";
66 return {
67 sentence: `The last snapshot failed: ${(s.error ?? "unknown reason").replace(/\.$/, "")}.${last} The forge tries again within the hour.`,
68 since: s.failingSince,
69 };
70 }
src/snapshot/parts.ts added +108
@@ -0,0 +1,108 @@
1 /**
2 * A large file packed for a slow uplink: zstd over the whole of it, then
3 * cut into parts of `PART_BYTES`, each its own object. Each part is one put
4 * with its own deadline and its own retries, so a link that drops halfway
5 * loses one part's progress, never the file's; and the mirror, which copies
6 * object by object, moves parts the same way.
7 *
8 * The parts of `forge.db` are `forge.db.zst.0000`, `forge.db.zst.0001`, …;
9 * the manifest lists them as files, like any other, and says once in
10 * `packed` what they make together and how big that is. A restore reads
11 * the parts back in order, joins them, and unpacks to exactly the bytes
12 * that went in, or refuses.
13 *
14 * Plain data in, plain data out: no store, no files.
15 */
16
17 /** A part's size: one put carries it within its own deadline at the floor rate (27 s at 5 Mbit/s). */
18 export const PART_BYTES = 16 * 1_048_576;
19
20 /** What the manifest says of a file that went up in parts. */
21 export interface PackedFile {
22 /** The file's own path, as a restore writes it back. */
23 readonly path: string;
24 /** Its size as it was, before compression. */
25 readonly bytes: number;
26 /** Its size compressed: the parts' sizes added. */
27 readonly packedBytes: number;
28 readonly parts: number;
29 }
30
31 /** The key of part `n` of `path`, under a snapshot's prefix. */
32 export const partPath = (path: string, n: number): string =>
33 `${path}.zst.${String(n).padStart(4, "0")}`;
34
35 /** Every part path of a packed file, in order. */
36 export const partPaths = (p: PackedFile): string[] =>
37 Array.from({ length: p.parts }, (_, n) => partPath(p.path, n));
38
39 /** A path a restore may write under its data directory: relative, and never climbing out. */
40 export const safePath = (path: string): boolean =>
41 path !== "" &&
42 !path.startsWith("/") &&
43 !path.includes("\\") &&
44 !path.split("/").some((s) => s === "" || s === "." || s === "..");
45
46 /**
47 * `body` compressed and cut in parts. The parts are views of one buffer,
48 * not copies; an empty body is one empty-content part, so every packed file
49 * has at least one.
50 */
51 export async function pack(
52 path: string,
53 body: Uint8Array,
54 partBytes = PART_BYTES,
55 ): Promise<{ packed: PackedFile; parts: Uint8Array[] }> {
56 if (!safePath(path)) throw new Error(`refusing to pack ${path}`);
57 if (!(partBytes > 0)) throw new Error(`a part of ${partBytes} bytes`);
58 const zst = new Uint8Array(await Bun.zstdCompress(body));
59 const parts: Uint8Array[] = [];
60 for (let at = 0; at < zst.byteLength || parts.length === 0; at += partBytes)
61 parts.push(zst.subarray(at, at + partBytes));
62 return {
63 packed: {
64 path,
65 bytes: body.byteLength,
66 packedBytes: zst.byteLength,
67 parts: parts.length,
68 },
69 parts,
70 };
71 }
72
73 /** The file the parts make, checked against what the manifest says it was. */
74 export async function unpack(
75 p: PackedFile,
76 parts: readonly Uint8Array[],
77 ): Promise<Uint8Array> {
78 if (parts.length !== p.parts)
79 throw new Error(`${p.path} came back in ${parts.length} parts, not ${p.parts}`);
80 const total = parts.reduce((n, b) => n + b.byteLength, 0);
81 if (total !== p.packedBytes)
82 throw new Error(`${p.path}'s parts are ${total} bytes, not ${p.packedBytes}`);
83 const joined = new Uint8Array(total);
84 let at = 0;
85 for (const b of parts) {
86 joined.set(b, at);
87 at += b.byteLength;
88 }
89 const body = new Uint8Array(await Bun.zstdDecompress(joined));
90 if (body.byteLength !== p.bytes)
91 throw new Error(`${p.path} unpacks to ${body.byteLength} bytes, not ${p.bytes}`);
92 return body;
93 }
94
95 /** Whether a manifest's `packed` entry reads whole: the restore trusts nothing else. */
96 export function isPackedFile(v: unknown): v is PackedFile {
97 if (typeof v !== "object" || v === null) return false;
98 const p = v as Record<string, unknown>;
99 const count = (n: unknown): boolean => Number.isSafeInteger(n) && (n as number) >= 0;
100 return (
101 typeof p["path"] === "string" &&
102 safePath(p["path"]) &&
103 count(p["bytes"]) &&
104 count(p["packedBytes"]) &&
105 count(p["parts"]) &&
106 (p["parts"] as number) >= 1
107 );
108 }
src/snapshot/restore.ts +35
@@ -16,6 +16,10 @@
1616 * with the same `S3_*` variables the forge runs under, or
1717 * `FORGE_SNAPSHOT_DIR` for a directory store.
1818 *
19 * A file that went up compressed in parts (the database, `parts.ts`)
20 * comes back whole: its parts read in order, joined, unpacked, and checked
21 * against the size the manifest gives it.
22 *
1923 * The key comes back only from its wrapped form and the operator's
2024 * passphrase (`FORGE_SNAPSHOT_PASSPHRASE`). A wrong passphrase refuses the
2125 * restore; none leaves the key out, and every sealed value that will not
@@ -36,6 +40,7 @@ import {
3640 type Manifest,
3741 } from "./snapshot.ts";
3842 import { unwrapKey, WRAPPED_KEY } from "./wrap.ts";
43 import { isPackedFile, partPaths, unpack, type PackedFile } from "./parts.ts";
3944 import { resolveDestination } from "./destination.ts";
4045 import type { Store } from "./store.ts";
4146
@@ -104,6 +109,14 @@ export async function restoreSnapshot(
104109 "the passphrase does not open this snapshot's wrapped key; check FORGE_SNAPSHOT_PASSPHRASE, or restore without it to leave sealed values behind",
105110 );
106111 }
112 const packed: readonly unknown[] = manifest.packed ?? [];
113 if (!packed.every(isPackedFile))
junipercrew

Needs tests: a legacy manifest with no packed (plain forge.db), a malformed packed entry, and a missing part. None of these paths is covered.
114 throw new Error(`snapshot ${name}: a packed file in its manifest does not read whole`);
115 const partOf = new Map<string, { file: PackedFile; n: number }>();
116 for (const file of packed)
117 for (const [n, path] of partPaths(file).entries())
118 partOf.set(path, { file, n });
119 const gathered = new Map<string, Uint8Array[]>();
107120 mkdirSync(dataDir, { recursive: true });
108121
109122 let files = 0;
@@ -118,6 +131,14 @@ export async function restoreSnapshot(
118131 `snapshot ${name}: ${entry.path} is ${body.byteLength} bytes, not ${entry.bytes}`,
119132 );
120133 if (entry.path === WRAPPED_KEY) continue;
134 // A part waits for the rest of its file.
135 const part = partOf.get(entry.path);
136 if (part !== undefined) {
137 const got = gathered.get(part.file.path) ?? [];
138 got[part.n] = body;
139 gathered.set(part.file.path, got);
140 continue;
141 }
121142 // A day's run logs in one object come back as the logs they were.
122143 if (isRunBundle(entry.path)) {
123144 for (const run of unpackRuns(body)) {
@@ -143,6 +164,20 @@ export async function restoreSnapshot(
143164 if (bundle)
144165 bundles.push({ owner: bundle[1]!, name: bundle[2]!, file: target });
145166 }
167 for (const file of packed) {
168 const got = gathered.get(file.path) ?? [];
169 const parts = partPaths(file).map((path, n) => {
170 const body = got[n];
171 if (body === undefined)
172 throw new Error(`snapshot ${name} is missing ${path}`);
173 return body;
174 });
175 gathered.delete(file.path);
176 const target = join(dataDir, file.path);
177 mkdirSync(dirname(target), { recursive: true });
178 await Bun.write(target, await unpack(file, parts));
179 files++;
180 }
146181 for (const b of bundles) {
147182 const dir = repoDir(join(dataDir, "repos"), b.owner, b.name);
148183 mkdirSync(dirname(dir), { recursive: true });
src/snapshot/service.ts +48 −13
@@ -5,7 +5,9 @@
55 * snapshots and says so on its settings page, rather than pretending.
66 *
77 * Runs are serialized: a nightly that meets an on-demand one waits its
8 * turn, and a forge stopping waits for the one in flight.
8 * turn, and a forge stopping waits for the one in flight. A run that fails
9 * is kept (`outcome.ts`) and shows on the board at once, and the next try
10 * is within the hour, not the next night.
911 */
1012
1113 import type { App } from "../app.ts";
@@ -23,12 +25,26 @@ import { runDrill, type DrillDeps, type DrillResult } from "./drill.ts";
2325 import { drillDue } from "./due.ts";
2426 import { lastDrill, recordDrill } from "./drills.ts";
2527 import { lastMirror, mirrorPass, MIRROR_EVERY_MS, recordMirror, type MirrorState } from "./mirror.ts";
28 import { recordSnapshotOutcome } from "./outcome.ts";
2629 import type { Store } from "./store.ts";
2730
2831 const log = createLog("snapshots");
2932
3033 /** When the nightly snapshot is taken, in UTC: before the probe round. */
3134 const NIGHTLY_HOUR_UTC = 8;
35 /** After a failed snapshot, the next try is this soon (or the nightly, if sooner). */
36 export const RETRY_MS = 3_600_000;
37
38 /**
39 * A forge given nowhere to put a snapshot. Not a failure: the Backups page
40 * says so calmly, so it is never kept as an outcome, never a card, and
41 * never retried sooner than the nightly.
42 */
43 class Nowhere extends Error {
44 constructor() {
45 super("this forge has nowhere to put a snapshot");
46 }
47 }
3248
3349 export interface SnapshotsHandle {
3450 /** The store in effect now; undefined when the forge has been given none. */
@@ -94,6 +110,8 @@ export function startSnapshots(
94110 /** Test seam: how often the mirror is looked at. */
95111 readonly mirrorEveryMs?: number | undefined;
96112 readonly nightly?: boolean;
113 /** Test seam: how soon a failed snapshot is tried again. */
114 readonly retryMs?: number | undefined;
97115 readonly passphrase?: string | undefined;
98116 /** Test seam: the drill itself. */
99117 readonly drill?: ((deps: DrillDeps) => Promise<DrillResult>) | undefined;
@@ -122,12 +140,13 @@ export function startSnapshots(
122140 let last: { manifest?: Manifest; error?: string; at?: string } = {};
123141
124142 const take = (): Promise<Manifest> => {
125 const run = (inFlight ?? Promise.resolve()).then(async () => {
143 // After the one before it, however that ended: chained on a failure as it
144 // was, every later run failed at once with the old error until a restart.
145 const run = (inFlight ?? Promise.resolve()).catch(() => {}).then(async () => {
126146 // Resolved per run, so a destination saved on the Backups page is the
127147 // very next snapshot's.
128148 const store = current();
129 if (store === undefined)
130 throw new Error("this forge has nowhere to put a snapshot");
149 if (store === undefined) throw new Nowhere();
131150 running = { objects: 0, bytes: 0, elapsedMs: 0 };
132151 const manifest = await takeSnapshot(
133152 {
@@ -153,11 +172,19 @@ export function startSnapshots(
153172 inFlight = run.then(
junipercrew outdated

This error path also runs when there is no store at all. Skip keep() and the hourly arm for that case, so an unconfigured forge stays calm.
154173 (m) => {
155174 last = { manifest: m, at: m.finishedAt };
175 keep({ landed: m.name });
176 arm(untilNightly());
156177 return m;
157178 },
158179 (error) => {
159180 last = { error: String(error), at: new Date().toISOString() };
181 if (error instanceof Nowhere) {
182 arm(untilNightly());
183 throw error;
184 }
160185 log.error("snapshot failed", { error: String(error) });
186 keep({ error: error instanceof Error ? error.message : String(error) });
187 arm(Math.min(options.retryMs ?? RETRY_MS, untilNightly()));
161188 throw error;
162189 },
163190 );
@@ -221,16 +248,24 @@ export function startSnapshots(
221248 return drilling;
222249 };
223250
224 let timer: ReturnType<typeof setTimeout> | undefined;
225 const schedule = (): void => {
226 if (options.nightly === false || stopped) return;
227 timer = setTimeout(() => {
228 take()
229 .catch(() => {})
230 .finally(schedule);
231 }, untilNightly());
251 const keep = (result: { landed: string } | { error: string }): void => {
252 try {
253 recordSnapshotOutcome(app.config.dataDir, result);
254 } catch (error) {
255 log.warn("snapshot outcome not kept; the board will not show it", { error: String(error) });
256 }
232257 };
233 schedule();
258
259 // One timer for the next snapshot: the nightly, or sooner after a failure.
260 // Every run that ends sets it again, whoever asked for the run.
261 let timer: ReturnType<typeof setTimeout> | undefined;
262 function arm(ms: number): void {
263 if (timer !== undefined) clearTimeout(timer);
264 timer = undefined;
265 if (options.nightly === false || stopped) return;
266 timer = setTimeout(() => void take().catch(() => {}), ms);
267 }
268 arm(untilNightly());
234269
235270 // The board only reads; the time the forge first owed a drill is kept here.
236271 try {
src/snapshot/snapshot.ts +31 −5
@@ -12,10 +12,13 @@
1212 * checks, previews, pictures.
1313 *
1414 * A snapshot is named by the moment it began, in UTC, so a listing sorts
15 * itself. Nothing is compressed or bundled beyond git's own bundles and
16 * the crew's older run logs, one compressed bundle per day: a bucket costs
17 * a round trip per object, so thousands of tiny logs are one object a day.
18 * A restore is reading files back, and a person can read the manifest.
15 * itself. Little is compressed or bundled beyond git's own bundles: the
16 * crew's older run logs, one compressed bundle per day, because a bucket
17 * costs a round trip per object and thousands of tiny logs are one object
18 * a day; and the database, compressed and cut in parts (`parts.ts`),
19 * because whole it was more than a home uplink carries in one put's
20 * deadline. A restore is reading files back, and a person can read the
21 * manifest.
1922 *
2023 * Objects go up through a bounded queue, eight in flight; the manifest
2124 * lists them in the order they were handed over, not the order they landed.
@@ -32,6 +35,7 @@ import { runGit } from "../git/env.ts";
3235 import { repoDir } from "../git/repos.ts";
3336 import { createLog } from "../kernel/log.ts";
3437 import { countRows, type DrillCounts } from "./counts.ts";
38 import { pack, partPath, PART_BYTES, type PackedFile } from "./parts.ts";
3539 import type { Store } from "./store.ts";
3640 import { unwrapKey, wrapKey, WRAPPED_KEY } from "./wrap.ts";
3741
@@ -63,6 +67,8 @@ export interface Manifest {
6367 * `raw` in snapshots from before the key was wrapped.
6468 */
6569 readonly key?: "wrapped" | "none" | "raw";
70 /** Files that went up compressed in parts, each listed in `files` by its parts. Absent before the database was packed. */
71 readonly packed?: readonly PackedFile[];
6672 }
6773
6874 /** What a manifest says of the key, including manifests older than the field. */
@@ -162,6 +168,8 @@ export interface Snapshotter {
162168 readonly onLand?: ((p: Progress) => void) | undefined;
163169 /** Overrides of the progress cadence, for tests. */
164170 readonly progressEvery?: { objects?: number; ms?: number } | undefined;
171 /** Override of the database's part size, for tests. */
172 readonly partBytes?: number | undefined;
165173 }
166174
167175 /** `2026-09-22T09-00-00Z`: sortable, and a name a bucket allows. */
@@ -216,6 +224,7 @@ export async function takeSnapshot(
216224 const prefix = `${SNAPSHOT_PREFIX}${name}/`;
217225 const scratch = mkdtempSync(join(tmpdir(), "forge-snapshot-"));
218226 const files: SnapshotFile[] = [];
227 const packed: PackedFile[] = [];
219228 let dbCounts: DrillCounts | undefined;
220229 const started = Date.now();
221230 const everyObjects = forge.progressEvery?.objects ?? PROGRESS_OBJECTS;
@@ -305,7 +314,23 @@ export async function takeSnapshot(
305314 } finally {
306315 copy.close();
307316 }
308 await putFile("forge.db", dbCopy);
317 // Compressed and in parts: each part one put, with its own deadline and
318 // retries, so a slow uplink carries it and a dropped one costs a part.
319 const database = await pack(
320 "forge.db",
321 new Uint8Array(await Bun.file(dbCopy).arrayBuffer()),
322 forge.partBytes ?? PART_BYTES,
323 );
324 packed.push(database.packed);
325 log.info("database packed", {
326 mb: (database.packed.bytes / 1_048_576).toFixed(1),
327 packedMb: (database.packed.packedBytes / 1_048_576).toFixed(1),
328 ratio: (database.packed.bytes / Math.max(1, database.packed.packedBytes)).toFixed(2),
329 parts: database.packed.parts,
330 });
331 for (const [n, part] of database.parts.entries())
332 await put(partPath("forge.db", n), part);
junipercrew

Parts are put one at a time, not through the queue of eight as the pull says. Use the queue, or correct the claim.
333 rmSync(dbCopy, { force: true });
309334 // The key travels only wrapped. Without a passphrase it stays home.
310335 let key: Manifest["key"] = "none";
311336 const keyFile = Bun.file(join(forge.dataDir, "secret.key"));
@@ -434,6 +459,7 @@ export async function takeSnapshot(
434459 bytes: files.reduce((n, f) => n + f.bytes, 0),
435460 ...(dbCounts === undefined ? {} : { counts: dbCounts }),
436461 key,
462 packed,
437463 };
438464 await store.put(
439465 `${prefix}manifest.json`,
src/snapshot/store.ts +43 −9
@@ -51,17 +51,40 @@ export function s3FromEnv(
5151 };
5252 }
5353
54 /** The slowest uplink a put may assume: 5 Mbit/s, a third of the home line it runs behind. */
55 export const FLOOR_BYTES_PER_S = 5_000_000 / 8;
56 /** What every attempt gets on top of its bytes' time: the handshake, the server's thought, a queue ahead of it. */
57 export const SLACK_MS = 60_000;
58 /** How long a read, a listing or a delete may go unanswered. */
59 export const CALL_DEADLINE_MS = 180_000;
60
61 /** How long a put of `bytes` may take before it counts as failed: its bytes at the floor rate, plus slack. */
62 export const deadlineFor = (
63 bytes: number,
64 floorBytesPerS = FLOOR_BYTES_PER_S,
65 slackMs = SLACK_MS,
66 ): number => slackMs + Math.ceil((bytes / floorBytesPerS) * 1000);
67
68 const sizeOf = (body: Uint8Array | string): number =>
69 typeof body === "string" ? Buffer.byteLength(body) : body.byteLength;
70
5471 /**
5572 * A store that tries again. Object stores answer a burst, a flake or a
5673 * moment of maintenance with a 5xx — Backblaze says so in its own docs —
5774 * and one such answer in a thousand puts used to fail the whole snapshot
5875 * with "S3Error: an unexpected error has occurred" and no key named. Every
5976 * call is tried `attempts` times with doubling waits, and what finally
60 * fails says which key it was. An attempt that has not answered within
61 * `deadlineMs` counts as failed too: a connection that died quietly never
77 * fails says which key it was. An attempt that has not answered within its
78 * deadline counts as failed too: a connection that died quietly never
6279 * settles on its own, and once held a snapshot on its first object for an
6380 * hour with the network fine (2026-10-13), so the next attempt opens a
6481 * fresh request instead.
82 *
83 * A put's deadline follows its size (`deadlineFor`): a fixed 180 s once
84 * failed every attempt at a 334 MB database behind a 15 Mbit/s uplink
85 * (2026-10-17). `deadlineMs` fixes every deadline instead, for tests;
86 * `floorBytesPerS` and `slackMs` scale the size rule, for tests that run a
87 * slow link at a thousand times its speed.
6588 */
6689 export function withRetries(
6790 inner: Store,
@@ -69,14 +92,23 @@ export function withRetries(
6992 attempts?: number;
7093 baseMs?: number;
7194 deadlineMs?: number;
95 floorBytesPerS?: number;
96 slackMs?: number;
7297 sleep?: (ms: number) => Promise<void>;
7398 } = {},
7499 ): Store {
75100 const attempts = Math.max(1, options.attempts ?? 5);
76101 const baseMs = options.baseMs ?? 500;
77 const deadlineMs = options.deadlineMs ?? 180_000;
102 const fixed = options.deadlineMs;
103 const callMs = fixed ?? CALL_DEADLINE_MS;
104 const putMs = (body: Uint8Array | string): number =>
105 fixed ?? deadlineFor(sizeOf(body), options.floorBytesPerS, options.slackMs);
78106 const sleep = options.sleep ?? ((ms) => Bun.sleep(ms));
79 async function tried<T>(what: string, fn: () => Promise<T>): Promise<T> {
107 async function tried<T>(
108 what: string,
109 deadlineMs: number,
110 fn: () => Promise<T>,
111 ): Promise<T> {
80112 let last: unknown;
81113 for (let n = 0; n < attempts; n++) {
82114 try {
@@ -93,11 +125,13 @@ export function withRetries(
93125 return {
94126 where: inner.where,
95127 ...(inner.dir === undefined ? {} : { dir: inner.dir }),
96 put: (key, body) => tried(`put ${key}`, () => inner.put(key, body)),
97 get: (key) => tried(`get ${key}`, () => inner.get(key)),
98 list: (prefix) => tried(`list ${prefix}`, () => inner.list(prefix)),
99 names: (prefix) => tried(`names ${prefix}`, () => inner.names(prefix)),
100 delete: (key) => tried(`delete ${key}`, () => inner.delete(key)),
128 put: (key, body) =>
129 tried(`put ${key}`, putMs(body), () => inner.put(key, body)),
130 get: (key) => tried(`get ${key}`, callMs, () => inner.get(key)),
131 list: (prefix) => tried(`list ${prefix}`, callMs, () => inner.list(prefix)),
132 names: (prefix) =>
133 tried(`names ${prefix}`, callMs, () => inner.names(prefix)),
134 delete: (key) => tried(`delete ${key}`, callMs, () => inner.delete(key)),
101135 };
102136 }
103137
src/web/pages/backups.ts +16 −6
@@ -20,6 +20,7 @@ import { resolveDestination, sealedSet } from "../../snapshot/destination.ts";
2020 import { keyOf } from "../../snapshot/snapshot.ts";
2121 import { lastDrill } from "../../snapshot/drills.ts";
2222 import { snapshotsOf } from "../../snapshot/service.ts";
23 import { lastSnapshotOutcome } from "../../snapshot/outcome.ts";
2324 import type { Ctx } from "../../http/context.ts";
2425 import { notFoundPage, settingsLayout, signInFirst } from "../chrome.ts";
2526 import { field, readForm, type Form } from "../form.ts";
@@ -183,9 +184,18 @@ export async function backupsPage(ctx: Ctx): Promise<Response> {
183184 note: "This forge was given nowhere to put them. On the platform an app's bucket (S3_*) is the place; on a machine, FORGE_SNAPSHOT_DIR names a directory. Seal a bucket below, or restart with one set, and a snapshot is taken every night at 08:00 UTC.",
184185 })}${form}`,
185186 );
187 // A failure is kept on disk, so it is shown after a restart too; this
188 // process's own failure is newer when the file could not be written.
189 const kept = lastSnapshotOutcome(ctx.app.config.dataDir);
190 const last = svc.last();
191 const failing =
192 kept?.failingSince !== undefined
193 ? { at: kept.at, since: kept.failingSince, error: kept.error ?? "unknown reason" }
194 : last.error !== undefined && (kept === undefined || (last.at ?? "") > kept.at)
195 ? { at: last.at ?? "", since: last.at ?? "", error: last.error }
196 : undefined;
186197 // The store is somewhere else; a listing that fails is a sentence on the
187198 // page, never a page that fails.
188 const last = svc.last();
189199 let snapshots: Awaited<ReturnType<typeof svc.list>> = [];
190200 let listing: string | undefined;
191201 try {
@@ -227,11 +237,11 @@ bun run src/snapshot/restore.ts ${snapshots[0].name} /path/to/empty/data</pre>`
227237 </p>`,
228238 )}
229239 ${when(
230 last.error,
231 (error) =>
232 html`<p class="backups-error">
233 ${icon("alert")} The last snapshot failed ${time(last.at ?? "")}:
234 ${error}
240 failing,
241 (f) =>
242 html`<p class="backups-error" role="alert">
243 ${icon("alert")} The last snapshot failed ${time(f.at)}, failing
244 since ${time(f.since)}: ${f.error}
235245 </p>`,
236246 )}
237247 ${when(
test/snapshot-drill.test.ts +20 −10
@@ -14,6 +14,7 @@ import { createApp, type App } from "../src/app.ts";
1414 import { createIssue } from "../src/core/issues.ts";
1515 import { createUser } from "../src/core/users.ts";
1616 import { runDrill } from "../src/snapshot/drill.ts";
17 import { pack, partPath } from "../src/snapshot/parts.ts";
1718 import { takeSnapshot } from "../src/snapshot/snapshot.ts";
1819 import { dirStore } from "../src/snapshot/store.ts";
1920
@@ -74,6 +75,8 @@ describe("the restore drill", () => {
7475 reposRoot: app.reposRoot,
7576 version: "t",
7677 passphrase: PASS,
78 // Small parts, so the drill restores a database that went up in several.
79 partBytes: 4096,
7780 },
7881 store(),
7982 new Date("2026-09-22T08:00:00Z"),
@@ -99,6 +102,10 @@ describe("the restore drill", () => {
99102 expect(r.verdict).toBe("green");
100103 expect(r.failedStep).toBeNull();
101104 expect(r.snapshotId).toBe("2026-09-22T08-00-00Z");
105 const manifest = JSON.parse(
106 new TextDecoder().decode((await store().get("snapshots/2026-09-22T08-00-00Z/manifest.json"))!),
107 );
108 expect(manifest.packed[0].parts).toBeGreaterThan(1);
102109 expect(r.counts).toEqual({ repositories: 1, issues: 2, users: 1, pulls: 0, apps: 1 });
103110 expect(r.ms).toBeGreaterThanOrEqual(0);
104111 expect(scratchLeft()).toEqual([]);
@@ -118,23 +125,26 @@ describe("the restore drill", () => {
118125 (await s.get("snapshots/2026-09-22T08-00-00Z/manifest.json"))!,
119126 ),
120127 );
121 const bytes = manifest.files.find(
122 (f: { path: string }) => f.path === "forge.db",
123 ).bytes;
124 // Same size, so the restore accepts it; only opening it can tell.
125 await s.put(
126 "snapshots/2026-09-23T08-00-00Z/forge.db",
127 new Uint8Array(bytes).fill(7),
128 const db = manifest.packed.find(
129 (p: { path: string }) => p.path === "forge.db",
128130 );
129 for (const f of manifest.files)
130 if (f.path !== "forge.db")
131 // Same size, packed the same way, so the restore accepts it; only opening it can tell.
132 const junk = await pack("forge.db", new Uint8Array(db.bytes).fill(7));
133 for (const [n, part] of junk.parts.entries())
134 await s.put(`snapshots/2026-09-23T08-00-00Z/${partPath("forge.db", n)}`, part);
135 const files = [
136 ...junk.parts.map((part, n) => ({ path: partPath("forge.db", n), bytes: part.byteLength })),
137 ...manifest.files.filter((f: { path: string }) => !f.path.startsWith("forge.db.zst.")),
138 ];
139 for (const f of files)
140 if (!f.path.startsWith("forge.db.zst."))
131141 await s.put(
132142 `snapshots/2026-09-23T08-00-00Z/${f.path}`,
133143 (await s.get(`snapshots/2026-09-22T08-00-00Z/${f.path}`))!,
134144 );
135145 await s.put(
136146 "snapshots/2026-09-23T08-00-00Z/manifest.json",
137 JSON.stringify({ ...manifest, name: "2026-09-23T08-00-00Z" }),
147 JSON.stringify({ ...manifest, name: "2026-09-23T08-00-00Z", files, packed: [junk.packed] }),
138148 );
139149 const r = await runDrill(deps());
140150 expect(r.snapshotId).toBe("2026-09-23T08-00-00Z");
test/snapshot-outcome.test.ts added +152
@@ -0,0 +1,152 @@
1 /**
2 * A failed snapshot is seen the same morning (#2536): it is kept under the
3 * data directory, vitals carries it as a card and Backups as a red line,
4 * both pointing at the reason, and the forge tries again within the hour
5 * rather than the next night. A snapshot that lands clears it.
6 */
7
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { mkdtempSync } from "node:fs";
10 import { rm } from "node:fs/promises";
11 import { tmpdir } from "node:os";
12 import { join } from "node:path";
13 import { forgeNeeds, forgeVitals } from "../src/actions/health.ts";
14 import { createApp, type App } from "../src/app.ts";
15 import { createUser, type User } from "../src/core/users.ts";
16 import { serve, type Servers } from "../src/http/server.ts";
17 import { lastSnapshotOutcome, recordSnapshotOutcome, snapshotNeed } from "../src/snapshot/outcome.ts";
18 import { startSnapshots } from "../src/snapshot/service.ts";
19 import { dirStore, type Store } from "../src/snapshot/store.ts";
20
21 const failed = (app: App) => forgeVitals(app).degraded.find((d) => d.cause === "snapshot-failed");
22
23 describe("the outcome kept", () => {
24 let dir: string;
25 beforeAll(() => {
26 dir = mkdtempSync(join(tmpdir(), "forge-outcome-"));
27 });
28 afterAll(async () => {
29 await rm(dir, { recursive: true, force: true });
30 });
31
32 test("none kept is no card; a failure is one from its first morning; one that lands clears it", () => {
33 expect(snapshotNeed(lastSnapshotOutcome(dir))).toBeUndefined();
34 recordSnapshotOutcome(dir, { landed: "2026-10-16T08-00-00Z" }, "2026-10-16T08:40:00.000Z");
35 expect(snapshotNeed(lastSnapshotOutcome(dir))).toBeUndefined();
36 recordSnapshotOutcome(dir, { error: "could not put forge.db: no answer in 180 s." }, "2026-10-17T08:45:00.000Z");
37 recordSnapshotOutcome(dir, { error: "could not put forge.db: no answer in 180 s." }, "2026-10-18T08:00:00.000Z");
38 const need = snapshotNeed(lastSnapshotOutcome(dir));
39 expect(need?.since).toBe("2026-10-17T08:45:00.000Z");
40 expect(need?.sentence).toBe(
41 "The last snapshot failed: could not put forge.db: no answer in 180 s. The newest that landed is from 2026-10-16. The forge tries again within the hour.",
42 );
43 recordSnapshotOutcome(dir, { landed: "2026-10-18T09-00-00Z" }, "2026-10-18T09:20:00.000Z");
44 expect(snapshotNeed(lastSnapshotOutcome(dir))).toBeUndefined();
45 expect(lastSnapshotOutcome(dir)?.landed).toBe("2026-10-18T09-00-00Z");
46 });
47 });
48
49 describe("a nightly that fails", () => {
50 let root: string;
51 let app: App;
52 let servers: Servers;
53 let cookie: string;
54 let admin: User;
55
56 beforeAll(async () => {
57 root = mkdtempSync(join(tmpdir(), "forge-outcome-svc-"));
58 app = createApp({ dataDir: join(root, "data"), port: 0 });
59 const a = await createUser(app.db, {
60 origin: "seed",
61 trusted: true,
62 username: "alice",
63 email: "[email protected]",
64 password: "a-good-passphrase",
65 adminIfFirst: true,
66 });
67 if (!a.ok) throw new Error(a.error);
68 admin = a.value;
69 servers = await serve(app);
70 const signedIn = await fetch(`${servers.url}/signin`, {
71 method: "POST",
72 headers: { origin: servers.url },
73 body: new URLSearchParams({ username: "alice", password: "a-good-passphrase" }),
74 redirect: "manual",
75 });
76 cookie = (signedIn.headers.get("set-cookie") ?? "").split(";")[0]!;
77 });
78
79 afterAll(async () => {
80 servers.stop();
81 app.close();
82 await rm(root, { recursive: true, force: true });
83 });
84
85 test("is a card on vitals and a red line on Backups at once, and the next try lands and clears both", async () => {
86 const inner = dirStore(join(root, "store"));
87 let broken = true;
88 const store: Store = {
89 ...inner,
90 put: async (key, body) => {
91 if (broken && key.includes("forge.db")) throw new Error("no answer in 180 s");
92 return inner.put(key, body);
93 },
94 };
95 const svc = startSnapshots(app, store, { version: "t", retryMs: 300 });
96 try {
97 await expect(svc.now()).rejects.toThrow(/no answer in 180 s/);
98 const card = failed(app);
99 expect(card?.sentence).toContain("The last snapshot failed");
100 expect(card?.sentence).toContain("no answer in 180 s");
101 expect(card?.href).toBe("/settings/backups");
102 // The terminal's dashboard prints the forge's needs: the card is there too.
103 expect(forgeNeeds(app, admin)?.some((l) => l.includes("The last snapshot failed"))).toBe(true);
104 const page = await (await fetch(`${servers.url}/settings/backups`, { headers: { cookie } })).text();
105 expect(page).toContain("backups-error");
106 expect(page.replace(/\s+/g, " ")).toContain("The last snapshot failed");
107 expect(page).toContain("no answer in 180 s");
108
109 // Nobody asks again: the forge does, within the retry, not the next night.
110 broken = false;
111 const until = Date.now() + 10_000;
112 while (lastSnapshotOutcome(app.config.dataDir)?.failingSince !== undefined && Date.now() < until)
113 await Bun.sleep(50);
114 expect(lastSnapshotOutcome(app.config.dataDir)?.landed).toBeDefined();
115 expect(failed(app)).toBeUndefined();
116 const after = await (await fetch(`${servers.url}/settings/backups`, { headers: { cookie } })).text();
117 expect(after.replace(/\s+/g, " ")).not.toContain("The last snapshot failed");
118 } finally {
119 await svc.stop();
120 }
121 }, 30_000);
122 });
123
124 describe("a forge given nowhere to put a snapshot", () => {
125 let root: string;
126 let app: App;
127
128 beforeAll(() => {
129 root = mkdtempSync(join(tmpdir(), "forge-outcome-none-"));
130 app = createApp({ dataDir: join(root, "data"), port: 0 });
131 });
132
133 afterAll(async () => {
134 app.close();
135 await rm(root, { recursive: true, force: true });
136 });
137
138 test("has not failed: asking refuses, but nothing is kept, no card is raised, and nothing is retried within the hour", async () => {
139 const svc = startSnapshots(app, () => undefined, { version: "t", retryMs: 50 });
140 try {
141 await expect(svc.now()).rejects.toThrow(/nowhere to put a snapshot/);
142 expect(lastSnapshotOutcome(app.config.dataDir)).toBeUndefined();
143 expect(failed(app)).toBeUndefined();
144 // A short retry would have run again by now, and every run stamps `last`.
145 const before = svc.last().at;
146 await Bun.sleep(300);
147 expect(svc.last().at).toBe(before);
148 } finally {
149 await svc.stop();
150 }
151 });
152 });
test/snapshot-restore-packed.test.ts added +190
@@ -0,0 +1,190 @@
1 /**
2 * Restore around the packed database (#2536). A snapshot from before the
3 * database was packed — a plain `forge.db`, no `packed` — still restores
4 * byte for byte; a manifest whose `packed` entry does not read whole, or a
5 * snapshot missing one of its parts, is refused before any database is
6 * written; and a snapshot hands its parts to the upload queue, several in
7 * flight at once, not one after another.
8 */
9
10 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
11 import { Database } from "bun:sqlite";
12 import { existsSync, mkdtempSync, readFileSync } from "node:fs";
13 import { rm } from "node:fs/promises";
14 import { tmpdir } from "node:os";
15 import { join } from "node:path";
16 import { createApp, type App } from "../src/app.ts";
17 import { createIssue } from "../src/core/issues.ts";
18 import { createUser } from "../src/core/users.ts";
19 import { createRepo } from "../src/actions/repos.ts";
20 import { pack, partPath } from "../src/snapshot/parts.ts";
21 import { restoreSnapshot } from "../src/snapshot/restore.ts";
22 import { takeSnapshot } from "../src/snapshot/snapshot.ts";
23 import { dirStore, type Store } from "../src/snapshot/store.ts";
24
25 const NAME = "2026-10-16T08-00-00Z";
26 const prefix = `snapshots/${NAME}/`;
27
28 /** A small real database, as the bytes of its file. */
29 function databaseBytes(dir: string): Uint8Array {
30 const file = join(dir, "made.db");
31 const db = new Database(file);
32 db.run("create table notes (body text)");
33 for (let n = 0; n < 200; n++) db.run("insert into notes values (?)", [`note ${n} `.repeat(20)]);
34 db.close();
35 return new Uint8Array(readFileSync(file));
36 }
37
38 const manifestOf = (files: { path: string; bytes: number }[], extra: object = {}): string =>
39 JSON.stringify({
40 name: NAME,
41 startedAt: "2026-10-16T08:00:00.000Z",
42 finishedAt: "2026-10-16T08:01:00.000Z",
43 version: "t",
44 repos: 0,
45 apps: 0,
46 files,
47 bytes: files.reduce((n, f) => n + f.bytes, 0),
48 key: "none",
49 ...extra,
50 });
51
52 describe("restoring around the packed database", () => {
53 let root: string;
54 let db: Uint8Array;
55 let n = 0;
56 const fresh = (): { store: Store; dataDir: string } => {
57 n++;
58 return { store: dirStore(join(root, `store-${n}`)), dataDir: join(root, `data-${n}`) };
59 };
60
61 beforeAll(() => {
62 root = mkdtempSync(join(tmpdir(), "forge-restore-packed-"));
63 db = databaseBytes(root);
64 });
65
66 afterAll(async () => {
67 await rm(root, { recursive: true, force: true });
68 });
69
70 test("a snapshot from before packing — a plain forge.db, no packed — restores byte for byte", async () => {
71 const { store, dataDir } = fresh();
72 await store.put(`${prefix}forge.db`, db);
73 await store.put(`${prefix}manifest.json`, manifestOf([{ path: "forge.db", bytes: db.byteLength }]));
74 const restored = await restoreSnapshot(store, NAME, dataDir);
75 expect(restored.files).toBe(1);
76 expect(new Uint8Array(await Bun.file(join(dataDir, "forge.db")).arrayBuffer())).toEqual(new Uint8Array(db));
77 });
78
79 test("a packed one, in several parts, restores byte for byte", async () => {
80 const { store, dataDir } = fresh();
81 const { packed, parts } = await pack("forge.db", db, 1024);
82 expect(parts.length).toBeGreaterThan(1);
83 for (const [i, part] of parts.entries()) await store.put(prefix + partPath("forge.db", i), part);
84 const files = parts.map((part, i) => ({ path: partPath("forge.db", i), bytes: part.byteLength }));
85 await store.put(`${prefix}manifest.json`, manifestOf(files, { packed: [packed] }));
86 await restoreSnapshot(store, NAME, dataDir);
87 expect(new Uint8Array(await Bun.file(join(dataDir, "forge.db")).arrayBuffer())).toEqual(new Uint8Array(db));
88 });
89
90 test("a packed entry that does not read whole is refused, and no database is written", async () => {
91 const { packed, parts } = await pack("forge.db", db, 1024);
92 const files = parts.map((part, i) => ({ path: partPath("forge.db", i), bytes: part.byteLength }));
93 for (const bad of [
94 { ...packed, parts: 0 },
95 { ...packed, bytes: -1 },
96 { ...packed, path: "../forge.db" },
97 { ...packed, packedBytes: "12" },
98 null,
99 ]) {
100 const { store, dataDir } = fresh();
101 for (const [i, part] of parts.entries()) await store.put(prefix + partPath("forge.db", i), part);
102 await store.put(`${prefix}manifest.json`, manifestOf(files, { packed: [bad] }));
103 await expect(restoreSnapshot(store, NAME, dataDir)).rejects.toThrow(/does not read whole/);
104 expect(existsSync(join(dataDir, "forge.db"))).toBe(false);
105 }
106 });
107
108 test("a part missing from the store, or from the manifest's files, is refused by name", async () => {
109 const { packed, parts } = await pack("forge.db", db, 1024);
110 const files = parts.map((part, i) => ({ path: partPath("forge.db", i), bytes: part.byteLength }));
111 const last = partPath("forge.db", parts.length - 1);
112
113 // Listed, but not in the store.
114 const a = fresh();
115 for (const [i, part] of parts.slice(0, -1).entries()) await a.store.put(prefix + partPath("forge.db", i), part);
116 await a.store.put(`${prefix}manifest.json`, manifestOf(files, { packed: [packed] }));
117 await expect(restoreSnapshot(a.store, NAME, a.dataDir)).rejects.toThrow(`is missing ${last}`);
118 expect(existsSync(join(a.dataDir, "forge.db"))).toBe(false);
119
120 // In the store, but left out of the files: the parts that came back are not the whole.
121 const b = fresh();
122 for (const [i, part] of parts.entries()) await b.store.put(prefix + partPath("forge.db", i), part);
123 await b.store.put(`${prefix}manifest.json`, manifestOf(files.slice(0, -1), { packed: [packed] }));
124 await expect(restoreSnapshot(b.store, NAME, b.dataDir)).rejects.toThrow(`is missing ${last}`);
125 expect(existsSync(join(b.dataDir, "forge.db"))).toBe(false);
126 });
127 });
128
129 describe("a snapshot's parts go up through the queue", () => {
130 let root: string;
131 let app: App;
132
133 beforeAll(async () => {
134 root = mkdtempSync(join(tmpdir(), "forge-parts-queue-"));
135 app = createApp({ dataDir: join(root, "live"), port: 0 });
136 const a = await createUser(app.db, {
137 origin: "seed",
138 trusted: true,
139 username: "alice",
140 email: "[email protected]",
141 password: "a-good-passphrase",
142 adminIfFirst: true,
143 });
144 if (!a.ok) throw new Error(a.error);
145 const repo = await createRepo(app, a.value, { name: "notes", visibility: "private" });
146 if (!repo.ok) throw new Error(String(repo.error));
147 for (let i = 0; i < 50; i++)
148 createIssue(app.db, { repoId: repo.value.id, authorId: a.value.id, title: `issue ${i} ${crypto.randomUUID()}` });
149 });
150
151 afterAll(async () => {
152 app.close();
153 await rm(root, { recursive: true, force: true });
154 });
155
156 test("several parts are in flight at once, never more than eight, and they restore", async () => {
157 const inner = dirStore(join(root, "store"));
158 let flying = 0;
159 let most = 0;
160 const store: Store = {
161 ...inner,
162 put: async (key, body) => {
163 if (!key.includes("forge.db.zst.")) return inner.put(key, body);
164 most = Math.max(most, ++flying);
165 try {
166 await Bun.sleep(20);
167 await inner.put(key, body);
168 } finally {
169 flying--;
170 }
171 },
172 };
173 const m = await takeSnapshot(
174 { db: app.db, dataDir: app.config.dataDir, reposRoot: app.reposRoot, version: "t", partBytes: 2048 },
175 store,
176 new Date("2026-10-18T08:00:00Z"),
177 );
178 expect(m.packed?.[0]?.parts).toBeGreaterThan(8);
179 expect(most).toBeGreaterThan(1);
180 expect(most).toBeLessThanOrEqual(8);
181 const dataDir = join(root, "restored");
182 await restoreSnapshot(inner, m.name, dataDir);
183 const back = new Database(join(dataDir, "forge.db"), { readonly: true });
184 try {
185 expect((back.query("select count(*) as n from issue").get() as { n: number }).n).toBe(50);
186 } finally {
187 back.close();
188 }
189 });
190 });
test/snapshot.test.ts +5 −2
@@ -132,14 +132,17 @@ describe("a forge back from a snapshot", () => {
132132 expect(manifest.repos).toBe(1);
133133 expect(manifest.apps).toBe(1);
134134 const paths = manifest.files.map((f) => f.path);
135 expect(paths).toContain("forge.db");
135 // The database goes up compressed, in parts, and the manifest says what they make.
136 expect(paths).toContain("forge.db.zst.0000");
137 expect(paths).not.toContain("forge.db");
138 expect(manifest.packed?.map((p) => p.path)).toEqual(["forge.db"]);
136139 // The raw key never reaches the store; only its wrapped form does.
137140 expect(paths).not.toContain("secret.key");
138141 expect(paths).toContain("secret.key.wrapped");
139142 expect(manifest.key).toBe("wrapped");
140143 const rawKey = readFileSync(join(app.config.dataDir, "secret.key"));
141144 for (const f of paths)
142 if (f !== "forge.db" && !f.includes("/"))
145 if (!f.startsWith("forge.db") && !f.includes("/"))
143146 expect(
144147 readFileSync(
145148 join(root, "store", "snapshots", manifest.name, f),

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs