trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns11g rEventsFlows

Backups can't lie: a backup that's late or dead turns red in front of the owner, and a snapshot survives a release #8753

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/8742-backups-can-t-lie-a-backup-that-s-late-o → main 5 commits fork

  • Merged. torque/8742-backups-can-t-lie-a-backup-that-s-late-o was merged into main as 2b51aed.

Review

Sign in to review this change.
15 files changed +740 −26
docs/features/backup-truth.md added +9
@@ -0,0 +1,9 @@
1 # Backups that cannot lie (#8742)
2
3 - **Read from the bucket.** `src/snapshot/bucket.ts` lists the primary and mirror stores every 5 minutes, in the serving process (no CronJob to die). Home's backup tile, vitals (one `backup` card) and the MCP `backups` tool's `verdict` all call `backupVerdict`.
4 - **Three states.** `green`, `red`, and `unknown`: nothing read yet since boot, or no restore drill yet. Unknown is never a calm tile and never an alarm; only red raises the one `backup` card. A store error is scrubbed of keys and presigned queries before it is kept.
5 - **Red when:** the newest snapshot is over 26 h old; a lane cannot be read or holds nothing; the newest restore drill failed, did not run, or brought back no rows; or the check itself has not run for 20 min (dead man's switch).
6 - **Catch-up.** A boot with the newest snapshot older than the last 08:00 UTC slot takes one two minutes later (`snapshotDue`). A snapshot cut short by a release has no manifest, so it counts as lost.
7 - **In the terminal.** The Resilience dialog (palette: Resilience, admins) opens with the same verdict, each lane's newest, and every reason. On the web it is home's backup tile and `/settings/backups`; for an agent, `verdict` in the MCP `backups` tool.
8 - **Off the serving thread.** `VACUUM INTO` runs in a worker (`vacuum.ts`).
9 - **Not done, tracked:** platform/K8s lanes on home (#8805); an off-process probe and snapshots outside the serving process or resumable (#8806). Catch-up retakes a lost snapshot whole.
src/flow/kpi_words.ts +1 −1
@@ -143,7 +143,7 @@ export function kpiLines(k: Kpis, live: { sha: string; since: string } | null, n
143143 `Median check time ${checkTime(k.medianCheckMs)}`,
144144 `Builders running ${buildersValue(k.builders)}${k.builders.slots === 0 ? ", no slots set" : ""}${k.builders.idle === null ? "" : ` · ${k.builders.idle.word}`}`,
145145 `Crew spend today ${spendValue(k.spend)}${spendLineNote(k.spend)}`,
146 `Last off-box backup ${backupAge(k.backupAt, now)}${backupOverdue(k.backupAt, now) ? ", overdue" : ""}`,
146 k.backup ? k.backup.line : `Last off-box backup ${backupAge(k.backupAt, now)}${backupOverdue(k.backupAt, now) ? ", overdue" : ""}`,
147147 `Visitors ${visitorsValue(k.reach)}, ${visitorsNote(k.reach)} ${trendText(k.reach.visitorDays)} 30 days`,
148148 `Accounts ${usersValue(k.reach)} signed in today`,
149149 `Page speed ${k.reach.loadP75 === null ? "no page views measured yet" : `loads in ${speedWords(k.reach.loadP75)}`}, ${reactNote(k.reach)}`,
src/flow/kpis.ts +10 −1
@@ -16,6 +16,8 @@
1616 * Those are not counted.
1717 */
1818
19 import { backupVerdict, isWatched, type BackupVerdict } from "../snapshot/bucket.ts";
20 import { lastDrill } from "../snapshot/drills.ts";
1921 import type { Database } from "bun:sqlite";
2022 import { spentTodayEverywhere } from "../crew/autonomy.ts";
2123 import { ceilingHitAt, ceilingReached } from "./ceiling.ts";
@@ -46,6 +48,8 @@ export interface Kpis {
4648 readonly spend: { readonly todayUsd: number; readonly ceilingUsd: number | null; /** When the run that crossed the cap was created (ISO); null while under it. */ readonly hitAt?: string | null };
4749 /** When a snapshot was last known off the machine (a bucket destination or the mirror, `off_box.ts`); null when never. */
4850 readonly backupAt: string | null;
51 /** The verdict on what is in the bucket, read by this process (`snapshot/bucket.ts`); null when the forge is not watching one. */
52 readonly backup?: BackupVerdict | null;
4953 /** Accounts, visitors and page speed (`reach.ts`). */
5054 readonly reach: Reach;
5155 }
@@ -163,10 +167,15 @@ export function readKpis(
163167
164168 const reach = time("kpis.reach", () => readReach(db, now));
165169
166 const backupAt = time("kpis.backup", () => (forge.dataDir === undefined ? null : lastOffBox(forge.dataDir)));
170 const backup = time("kpis.backup", () =>
171 forge.dataDir !== undefined && isWatched(forge.dataDir) ? backupVerdict(forge.dataDir, lastDrill(db), end) : null,
172 );
173 // Read from the bucket when it is watched; a job's own report only where nothing watches.
174 const backupAt = backup !== null ? backup.newestAt : forge.dataDir === undefined ? null : lastOffBox(forge.dataDir);
167175
168176 return {
169177 backupAt,
178 backup,
170179 builders,
171180 reach,
172181 spend,
src/http/mcp/tools/backups.ts +6 −1
@@ -3,11 +3,13 @@ import { resolveDestination, scrubReason, sealedTarget } from "../../../snapshot
33 import { mirrorNeed } from "../../../snapshot/mirror.ts";
44 import { s3FromEnv } from "../../../snapshot/store.ts";
55 import { listSnapshots } from "../../../snapshot/snapshot.ts";
6 import { backupVerdict } from "../../../snapshot/bucket.ts";
7 import { lastDrill } from "../../../snapshot/drills.ts";
68 import { good, bad, type Tool } from "../shared.ts";
79
810 export const tool: Tool = {
911 name: "backups",
10 doc: "where snapshots go (`destination`: `kind` sealed, dir or env, and `place`, never a credential; `null` with the Backups page's sentence when none), the last fourteen snapshots (`key`, `at`, `bytes`) and the off-box `mirror`'s state; a forge admin's, as /settings/backups",
12 doc: "where snapshots go (`destination`: `kind` sealed, dir or env, and `place`, never a credential; `null` with the Backups page's sentence when none), `verdict` (state, line, reasons, lanes with newest object, drill: read from the bucket, red past 26 h), the last fourteen snapshots (`key`, `at`, `bytes`) and the off-box `mirror`'s state; a forge admin's, as /settings/backups",
1113 description: "Admin: snapshot destination, recent, mirror.",
1214 inputSchema: { type: "object", properties: {} },
1315 writes: false,
@@ -25,8 +27,10 @@ export const tool: Tool = {
2527 syncedAt: state?.syncedAt ?? null,
2628 says: need?.sentence ?? null,
2729 };
30 const verdict = backupVerdict(ctx.app.config.dataDir, lastDrill(ctx.app.db));
2831 if (!dest)
2932 return good({
33 verdict,
3034 destination: null,
3135 says: "nothing — no snapshots are taken",
3236 snapshots: [],
@@ -44,6 +48,7 @@ export const tool: Tool = {
4448 listing = scrubReason(target, error);
4549 }
4650 return good({
51 verdict,
4752 destination: { kind: dest.source, place: dest.store.where },
4853 snapshots,
4954 ...(listing === undefined ? {} : { listing }),
src/kernel/vitals.ts +8
@@ -24,6 +24,8 @@ import { STALE_AFTER_MS } from "../runs/lease.ts";
2424 import { ownRepoPath } from "./own_repo.ts";
2525 import { BUDGETS } from "./budgets.ts";
2626 import { openDbFiles } from "./file_modes.ts";
27 import { backupVerdict, isWatched } from "../snapshot/bucket.ts";
28 import { lastDrill } from "../snapshot/drills.ts";
2729 import { lastMirror, mirrorNeed } from "../snapshot/mirror.ts";
2830 import { lastSnapshotOutcome, snapshotNeed } from "../snapshot/outcome.ts";
2931 import { drillNeed, firstSeen as drillFirstSeen, lastVerdict } from "../snapshot/verdict.ts";
@@ -474,6 +476,12 @@ export function vitals(input: VitalsInput): Vitals {
474476 const failing = snapshotNeed(lastSnapshotOutcome(input.dataDir));
475477 if (failing !== undefined) need("snapshot-failed", failing.sentence, failing.since, "/settings/backups");
476478
479 // The bucket itself, read by this process: late, unreadable, a failed drill, or a check that stopped is one card.
480 if (isWatched(input.dataDir)) {
481 const backup = backupVerdict(input.dataDir, lastDrill(input.db), now);
482 if (backup.state === "red") need("backup", backup.reasons.join(" "), backup.checkedAt ?? undefined, "/settings/backups");
483 }
484
477485 // A snapshot that did not land off the box, or a mirror a day behind, is a card until it does.
478486 const behind = mirrorNeed(lastMirror(input.dataDir), now);
479487 if (behind !== undefined) need("mirror", behind.sentence, behind.since, "/settings/backups");
src/snapshot/bucket.ts added +273
@@ -0,0 +1,273 @@
1 /**
2 * What is actually in the bucket (#8742). Every other backup signal here is
3 * a job's own report of itself; when the job dies, so does its report, and
4 * the owner was told nothing for 41 days. This reads the store itself —
5 * newest snapshot, its size, how many — on a timer that belongs to the
6 * forge's own process, and judges it: older than 26 h is red, a lane that
7 * cannot be read is red, a restore drill that failed or brought back no data
8 * is red, and a read that has not happened for `CHECK_STALE_MS` is red too
9 * (the dead man's switch: a check that stops is itself the alarm).
10 *
11 * The reads are kept in memory per data directory; `backupVerdict` is
12 * synchronous, so home, vitals and the MCP's `backups` say the same words
13 * without touching the network. Plain data in, plain data out.
14 */
15
16 import type { DrillResult } from "./drill.ts";
17 import { isOffBox } from "./off_box.ts";
18 import { listSnapshots } from "./snapshot.ts";
19 import { scrubReason } from "./destination.ts";
20 import type { S3Target, Store } from "./store.ts";
21
22 /** A newest snapshot older than this is red: the nightly is 24 h apart, plus a slow one's slack. */
23 export const LATE_MS = 26 * 3_600_000;
24 /** How often the bucket is read. */
25 export const READ_EVERY_MS = 5 * 60_000;
26 /** No read for this long, however it went, is red. */
27 export const CHECK_STALE_MS = 20 * 60_000;
28
29 export interface LaneRead {
30 /** `primary`, `mirror`; a platform lane would add its own name. */
31 readonly lane: string;
32 readonly where: string;
33 readonly offBox: boolean;
34 readonly newestAt: string | null;
35 readonly newestName: string | null;
36 readonly newestBytes: number | null;
37 readonly count: number;
38 /** Set when the store could not be read: the reason, never a credential. */
39 readonly error?: string;
40 /** When this read ended. */
41 readonly readAt: string;
42 }
43
44 export interface LaneVerdict extends LaneRead {
45 readonly state: "green" | "red";
46 readonly says: string;
47 }
48
49 export interface BackupVerdict {
50 /** `unknown` is nothing read yet, or nothing yet proving a restore: never calm, never an alarm. */
51 readonly state: "green" | "red" | "unknown";
52 /** A few words for a tile's note: the first thing wrong, or what is known. */
53 readonly note: string;
54 /** The one line home shows. */
55 readonly line: string;
56 /** Every reason it is red, one sentence each; empty when green or unknown. */
57 readonly reasons: readonly string[];
58 /** Why it is unknown, one sentence each; empty otherwise. */
59 readonly unknowns: readonly string[];
60 readonly lanes: readonly LaneVerdict[];
61 readonly drill: { readonly at: string; readonly ok: boolean; readonly says: string } | null;
62 /** The newest snapshot across the lanes. */
63 readonly newestAt: string | null;
64 /** When the bucket was last read; null when it has not been. */
65 readonly checkedAt: string | null;
66 }
67
68 const reads = new Map<string, { lanes: LaneRead[]; checkedAt: string; since: string }>();
69 const watching = new Map<string, string>();
70
71 /** Whether a bucket watch has been started for this data directory. */
72 export const isWatched = (dataDir: string): boolean => watching.has(dataDir);
73
74 export function ageWords(ms: number): string {
75 const min = Math.max(0, Math.floor(ms / 60_000));
76 if (min < 1) return "just now";
77 if (min < 60) return `${min} min`;
78 if (min < 2880) return `${Math.floor(min / 60)} h`;
79 return `${Math.floor(min / 1440)} d`;
80 }
81
82 export function sizeWords(bytes: number): string {
83 if (bytes < 1024) return `${bytes} B`;
84 if (bytes < 1_048_576) return `${(bytes / 1024).toFixed(1)} KiB`;
85 if (bytes < 1_073_741_824) return `${(bytes / 1_048_576).toFixed(1)} MiB`;
86 return `${(bytes / 1_073_741_824).toFixed(2)} GiB`;
87 }
88
89 /** An error as one short sentence with no key and no presigned query in it. */
90 export function scrubError(targets: readonly (S3Target | undefined)[], error: unknown): string {
91 let reason = String(error instanceof Error ? error.message : error);
92 for (const t of targets) reason = scrubReason(t, reason);
93 reason = reason
94 .replace(/(https?:\/\/)[^\s/@]*@/g, "$1")
95 .replace(/\?[^\s"']*(X-Amz-|Signature=|AWSAccessKeyId=)[^\s"']*/gi, "?…");
96 return reason.slice(0, 200);
97 }
98
99 /** Read one store: its newest snapshot by manifest. Never throws; a failure is the read. */
100 export async function readLane(
101 lane: string,
102 store: Store,
103 now = Date.now(),
104 /** The credentials this store was opened with, struck out of any error; asked only when a read fails, since opening sealed secrets is audited. */
105 targets: () => readonly (S3Target | undefined)[] = () => [],
106 ): Promise<LaneRead> {
107 const base = { lane, where: store.where, offBox: isOffBox(store), readAt: new Date(now).toISOString() };
108 try {
109 const all = await listSnapshots(store);
110 const newest = all[0];
111 return {
112 ...base,
113 newestAt: newest?.finishedAt ?? null,
114 newestName: newest?.name ?? null,
115 newestBytes: newest?.bytes ?? null,
116 count: all.length,
117 };
118 } catch (error) {
119 return { ...base, newestAt: null, newestName: null, newestBytes: null, count: 0, error: scrubError(targets(), error) };
120 }
121 }
122
123 /** Keep what a pass read. */
124 export function keepReads(dataDir: string, lanes: LaneRead[], now = Date.now()): void {
125 const prev = reads.get(dataDir);
126 reads.set(dataDir, { lanes, checkedAt: new Date(now).toISOString(), since: prev?.since ?? new Date(now).toISOString() });
127 }
128
129 export function forgetReads(dataDir: string): void {
130 reads.delete(dataDir);
131 }
132
133 /** Is the drill's word good: it passed, and real data came back. */
134 export function drillCounts(d: Pick<DrillResult, "verdict" | "counts" | "health">): boolean {
135 const c = d.counts;
136 const rows = c.repositories + c.issues + c.users + c.pulls + c.apps;
137 return d.verdict === "green" && d.health === "ok" && rows > 0;
138 }
139
140 /**
141 * The verdict, from what was read and the newest drill. `booted` is when
142 * this process began watching, so a check that has never run is not red
143 * until it has had `CHECK_STALE_MS` to.
144 */
145 export function judge(
146 state: { lanes: readonly LaneRead[]; checkedAt: string } | undefined,
147 drill: DrillResult | undefined,
148 now: number,
149 booted: number = now,
150 ): BackupVerdict {
151 const reasons: string[] = [];
152 /** A few words for each reason, for a tile's note. */
153 const causes: string[] = [];
154 const unknowns: string[] = [];
155 const lanes: LaneVerdict[] = [];
156 let checkedAt: string | null = state?.checkedAt ?? null;
157 const sinceCheck = checkedAt === null ? now - booted : now - Date.parse(checkedAt);
158 if (sinceCheck > CHECK_STALE_MS) {
159 causes.push("check stopped");
160 reasons.push(
161 checkedAt === null
162 ? "The backup check has never run, so nothing says a backup exists."
163 : `The backup check has not run for ${ageWords(sinceCheck)}; it is the alarm, and it is down.`,
164 );
165 } else if (checkedAt === null) unknowns.push("The bucket has not been read yet, so nothing is known about the backups.");
166 for (const r of state?.lanes ?? []) {
167 let says: string;
168 let red = true;
169 if (r.error !== undefined) says = `${r.lane} (${r.where}) cannot be read: ${r.error}`;
170 else if (r.newestAt === null) says = `${r.lane} (${r.where}) holds no snapshot`;
171 else {
172 const age = now - Date.parse(r.newestAt);
173 if (!(age <= LATE_MS)) says = `${r.lane} (${r.where}) newest snapshot is ${ageWords(age)} old`;
174 else {
175 red = false;
176 says = `${r.lane} newest ${ageWords(age)} old, ${sizeWords(r.newestBytes ?? 0)}`;
177 }
178 }
179 if (red) {
180 causes.push(r.error !== undefined ? "unreadable" : r.newestAt === null ? "empty" : "late");
181 reasons.push(`${says[0]!.toUpperCase()}${says.slice(1)}.`);
182 }
183 lanes.push({ ...r, state: red ? "red" : "green", says });
184 }
185 if (state !== undefined && state.lanes.length === 0) {
186 causes.push("no destination");
187 reasons.push("No place is given for snapshots; none is being taken.");
188 }
189 let drillSays: BackupVerdict["drill"] = null;
190 if (drill !== undefined) {
191 const ok = drillCounts(drill);
192 const says = ok
193 ? `restore drill ${drill.at.slice(0, 10)} passed`
194 : drill.verdict === "green"
195 ? `restore drill ${drill.at.slice(0, 10)} brought back no data`
196 : `restore drill ${drill.at.slice(0, 10)} failed${drill.failedStep ? ` at ${drill.failedStep}` : ""}`;
197 drillSays = { at: drill.at, ok, says };
198 if (!ok) {
199 causes.push(drill.verdict === "green" ? "drill restored nothing" : "drill failed");
200 reasons.push(`The last ${says}.`);
201 }
202 } else if (checkedAt !== null) unknowns.push("No restore drill has run, so nothing shows a snapshot restores.");
203 const newest = lanes.map((l) => l.newestAt).filter((t): t is string => t !== null).sort().at(-1) ?? null;
204 const primary = lanes[0];
205 const head =
206 primary?.newestAt != null && primary.error === undefined
207 ? `Backup ${ageWords(now - Date.parse(primary.newestAt))} old, ${sizeWords(primary.newestBytes ?? 0)}`
208 : "No backup in the bucket";
209 const red = reasons.length > 0;
210 const unknown = !red && unknowns.length > 0;
211 const line = `${head}; ${drillSays?.says ?? "no restore drill yet"}`;
212 if (state === undefined) checkedAt = null;
213 const sub = lanes[0]?.newestBytes != null ? sizeWords(lanes[0].newestBytes) : "";
214 const note = red
215 ? causes[0]!
216 : unknown
217 ? checkedAt === null
218 ? "not checked yet"
219 : "no restore drill yet"
220 : `${sub} · drill passed`;
221 return {
222 state: red ? "red" : unknown ? "unknown" : "green",
223 note,
224 line: checkedAt === null && !red ? "Backups not checked yet" : line,
225 reasons,
226 unknowns,
227 lanes,
228 drill: drillSays,
229 newestAt: newest,
230 checkedAt,
231 };
232 }
233
234 /** The verdict now, from what the watch last read. */
235 export function backupVerdict(dataDir: string, drill: DrillResult | undefined, now = Date.now()): BackupVerdict {
236 const booted = Date.parse(watching.get(dataDir) ?? "") || now;
237 return judge(reads.get(dataDir), drill, now, booted);
238 }
239
240 /**
241 * Read the lanes now and then every `everyMs`, in this process, so the
242 * check cannot die with a CronJob. Returns a stop.
243 */
244 export function watchBucket(
245 dataDir: string,
246 lanes: () => readonly { lane: string; store: Store | undefined }[],
247 everyMs: number = READ_EVERY_MS,
248 firstMs = 5_000,
249 /** The credentials in use, struck out of any error a read keeps. */
250 targets: () => readonly (S3Target | undefined)[] = () => [],
251 ): { stop: () => void; pass: () => Promise<void> } {
252 watching.set(dataDir, new Date().toISOString());
253 let stopped = false;
254 let timer: ReturnType<typeof setTimeout> | undefined;
255 const pass = async (): Promise<void> => {
256 const stores = lanes().filter((l): l is { lane: string; store: Store } => l.store !== undefined);
257 keepReads(dataDir, await Promise.all(stores.map((l) => readLane(l.lane, l.store, Date.now(), targets))));
258 };
259 const next = (ms: number): void => {
260 if (stopped) return;
261 timer = setTimeout(() => void pass().catch(() => {}).finally(() => next(everyMs)), ms);
262 };
263 next(Math.min(firstMs, everyMs));
264 return {
265 pass,
266 stop: () => {
267 stopped = true;
268 if (timer !== undefined) clearTimeout(timer);
269 watching.delete(dataDir);
270 forgetReads(dataDir);
271 },
272 };
273 }
src/snapshot/due.ts +18
@@ -44,3 +44,21 @@ export function drillDue(
4444 const began = Date.parse(since ?? "");
4545 return Number.isFinite(began) && began < slot;
4646 }
47
48 /** The latest 08:00 UTC at or before `now`, in ms: the nightly slot. */
49 export function lastNightly(now: number): number {
50 const d = new Date(now);
51 d.setUTCHours(8, 0, 0, 0);
52 return d.getTime() > now ? d.getTime() - DAY : d.getTime();
53 }
54
55 /**
56 * Whether a snapshot is owed at `now`: the newest one finished (`newestAt`,
57 * undefined when the store holds none) before the latest nightly slot. A
58 * snapshot takes the better part of an hour, so one cut short by a release
59 * at 08:21 is owed again on the next boot (#8742).
60 */
61 export function snapshotDue(newestAt: string | undefined, now: number): boolean {
62 const at = Date.parse(newestAt ?? "");
63 return !Number.isFinite(at) || at < lastNightly(now);
64 }
src/snapshot/service.ts +85 −4
@@ -22,12 +22,14 @@ import {
2222 type Progress,
2323 } from "./snapshot.ts";
2424 import { describeTimes, runDrill, type DrillDeps, type DrillResult } from "./drill.ts";
25 import { BOOT_RED_FLOOR_MS, drillDue, RED_RETRY_MS } from "./due.ts";
25 import { BOOT_RED_FLOOR_MS, drillDue, RED_RETRY_MS, snapshotDue } from "./due.ts";
26 import { sealedTarget } from "./destination.ts";
27 import { watchBucket } from "./bucket.ts";
2628 import { lastDrill, recordDrill } from "./drills.ts";
27 import { lastMirror, mirrorPass, MIRROR_EVERY_MS, recordMirror, type MirrorState } from "./mirror.ts";
29 import { lastMirror, mirrorPass, mirrorTarget, MIRROR_EVERY_MS, recordMirror, type MirrorState } from "./mirror.ts";
2830 import { isOffBox } from "./off_box.ts";
2931 import { noteOffBox, recordSnapshotOutcome } from "./outcome.ts";
30 import type { Store } from "./store.ts";
32 import { s3FromEnv, type Store } from "./store.ts";
3133
3234 const log = createLog("snapshots");
3335
@@ -76,6 +78,10 @@ export interface SnapshotsHandle {
7678 readonly mirrored: () => MirrorState | undefined;
7779 /** Forget the resolved store; the next use resolves again (a new destination was saved). */
7880 readonly forget: () => void;
81 /** Read the bucket now, as the watch does; resolves when the read is kept. */
82 readonly checkBucket: () => Promise<void>;
83 /** Take a snapshot if the store lacks the night's; says whether it did. For a boot, and for tests. */
84 readonly takeIfDue: () => Promise<boolean>;
7985 readonly stop: () => Promise<void>;
8086 }
8187
@@ -87,6 +93,8 @@ export const snapshotsOf = (app: App): SnapshotsHandle | undefined =>
8793 const DRILL_DAY_UTC = 0;
8894 const DRILL_HOUR_UTC = 9;
8995 const CATCH_UP_MS = 5 * 60_000;
96 /** A snapshot due on boot is taken this long after it, once the forge is serving. */
97 const SNAPSHOT_CATCH_UP_MS = 2 * 60_000;
9098
9199 export function untilDrill(now = Date.now()): number {
92100 const next = new Date(now);
@@ -124,6 +132,12 @@ export function startSnapshots(
124132 readonly catchUpMs?: number | undefined;
125133 readonly redRetryMs?: number | undefined;
126134 readonly bootRedFloorMs?: number | undefined;
135 /** Test seams: how soon after boot a due snapshot is taken, and how often the bucket is read. */
136 readonly snapshotCatchUpMs?: number | undefined;
137 readonly bucketEveryMs?: number | undefined;
138 readonly bucketFirstMs?: number | undefined;
139 /** Test seam: the clock `takeIfDue` judges by. */
140 readonly now?: (() => number) | undefined;
127141 } = {
128142 version: "0",
129143 },
@@ -145,6 +159,7 @@ export function startSnapshots(
145159 };
146160 let stopped = false;
147161 let inFlight: Promise<Manifest> | undefined;
162 let active = 0;
148163 let running: Progress | undefined;
149164 let last: { manifest?: Manifest; error?: string; at?: string } = {};
150165
@@ -208,9 +223,34 @@ export function startSnapshots(
208223 throw error;
209224 },
210225 );
226 active++;
227 inFlight.then(
228 () => void active--,
229 () => void active--,
230 );
211231 return inFlight;
212232 };
213233
234 // A restart takes the night's snapshot if the store does not hold it: a
235 // release at 08:21 killed the one in flight and nothing took it again (#8742).
236 // The manifest is written last, so a snapshot cut short is not one.
237 const takeIfDue = async (): Promise<boolean> => {
238 if (stopped || active > 0) return false;
239 const store = current();
240 if (store === undefined) return false;
241 let newest: Manifest | undefined;
242 try {
243 newest = (await listSnapshots(store))[0];
244 } catch (error) {
245 log.warn("snapshot catch-up: store unreadable, not taking one now", { error: String(error) });
246 return false;
247 }
248 if (stopped || active > 0 || !snapshotDue(newest?.finishedAt, (options.now ?? Date.now)())) return false;
249 log.info("snapshot due or lost; taking one", { newest: newest?.name ?? "none" });
250 await take().catch(() => {});
251 return true;
252 };
253
214254 // Single-flight: two passes at once would copy the same objects twice.
215255 let mirroring: Promise<MirrorState | undefined> | undefined;
216256 const mirrorNow = (): Promise<MirrorState | undefined> => {
@@ -259,11 +299,26 @@ export function startSnapshots(
259299 const store = current();
260300 if (store === undefined)
261301 throw new Error("this forge has nowhere to drill from");
302 // A drill that could not run is not a drill that passed: it is kept red,
303 // so the board and the alarm say so instead of the log alone (#8742).
262304 const result = await (options.drill ?? runDrill)({
263305 store,
264306 dataDir: app.config.dataDir,
265307 passphrase: options.passphrase,
266 });
308 }).catch(
309 (error): DrillResult => ({
310 at: new Date().toISOString(),
311 snapshotId: null,
312 ms: 0,
313 counts: { repositories: 0, issues: 0, users: 0, pulls: 0, apps: 0 },
314 health: "unreached",
315 verdict: "red",
316 failedStep: "restore",
317 phases: [],
318 budgetMs: 0,
319 reason: `the drill did not run: ${String(error).slice(0, 200)}`,
320 }),
321 );
267322 recordDrill(app.db, result);
268323 // The card and the recovery issue hear of every drill, whoever asked.
269324 const posted = await reportVerdict(app.config.dataDir, result);
@@ -355,6 +410,28 @@ export function startSnapshots(
355410 };
356411 scheduleMirror(true);
357412
413 const catchUpSnapshot =
414 options.nightly === false ? undefined : setTimeout(() => void takeIfDue(), options.snapshotCatchUpMs ?? SNAPSHOT_CATCH_UP_MS);
415
416 // The bucket is read by this process, on its own clock, whatever any job says.
417 const bucket = watchBucket(
418 app.config.dataDir,
419 () => [
420 { lane: "primary", store: current() },
421 { lane: "mirror", store: currentMirror() },
422 ],
423 options.bucketEveryMs,
424 options.bucketFirstMs,
425 () => {
426 const sources = { db: app.db, dataDir: app.config.dataDir, bus: app.bus };
427 try {
428 return [sealedTarget(sources), s3FromEnv(), mirrorTarget(sources)?.target];
429 } catch {
430 return [s3FromEnv()];
431 }
432 },
433 );
434
358435 const handle: SnapshotsHandle = {
359436 get store() {
360437 return current();
@@ -374,6 +451,8 @@ export function startSnapshots(
374451 },
375452 mirrorNow,
376453 mirrorFresh,
454 checkBucket: bucket.pass,
455 takeIfDue,
377456 mirrored: () => lastMirror(app.config.dataDir),
378457 forget: () => {
379458 resolved = undefined;
@@ -386,6 +465,8 @@ export function startSnapshots(
386465 if (catchUp !== undefined) clearTimeout(catchUp);
387466 if (redRetry !== undefined) clearTimeout(redRetry);
388467 if (mirrorTimer !== undefined) clearTimeout(mirrorTimer);
468 if (catchUpSnapshot !== undefined) clearTimeout(catchUpSnapshot);
469 bucket.stop();
389470 if (mirroring !== undefined) await mirroring.catch(() => {});
390471 if (inFlight !== undefined) await inFlight.catch(() => {});
391472 if (handles.get(app) === handle) handles.delete(app);
src/snapshot/snapshot.ts +3 −13
@@ -25,6 +25,7 @@
2525 */
2626
2727 import { Database } from "bun:sqlite";
28 import { vacuumFile, vacuumInto } from "./vacuum.ts";
2829 import { mkdtempSync, readdirSync, rmSync, statSync } from "node:fs";
2930 import { tmpdir } from "node:os";
3031 import { join, relative } from "node:path";
@@ -182,17 +183,6 @@ export const snapshotName = (at: Date): string =>
182183 const SQLITE = /\.(db|sqlite|sqlite3)$/;
183184 const SIDE_FILE = /\.(db|sqlite|sqlite3)-(wal|shm|journal)$/;
184185
185 /** A consistent copy of a SQLite file, made by SQLite itself. */
186 async function vacuumInto(source: string, target: string): Promise<void> {
187 const { Database } = await import("bun:sqlite");
188 const db = new Database(source, { readonly: true });
189 try {
190 db.run("VACUUM INTO ?1", [target]);
191 } finally {
192 db.close();
193 }
194 }
195
196186 /** Every file under `dir`, relative to it. */
197187 function filesUnder(dir: string): string[] {
198188 const out: string[] = [];
@@ -305,7 +295,7 @@ export async function takeSnapshot(
305295 try {
306296 // The database, consistently, from the live connection.
307297 const dbCopy = join(scratch, "forge.db");
308 forge.db.run("VACUUM INTO ?1", [dbCopy]);
298 await vacuumInto(forge.db, dbCopy);
309299 const copy = new Database(dbCopy, { readonly: true });
310300 try {
311301 dbCounts = countRows(copy);
@@ -386,7 +376,7 @@ export async function takeSnapshot(
386376 `app-${app.id}-${f.replace(/[\/\\]/g, "__")}`,
387377 );
388378 try {
389 await vacuumInto(source, copy);
379 await vacuumFile(source, copy);
390380 await putFile(`apps/${app.id}/production/data/${f}`, copy);
391381 } catch (error) {
392382 log.warn("app database not copied consistently; copied as a file", {
src/snapshot/vacuum.ts added +35
@@ -0,0 +1,35 @@
1 /**
2 * The consistent copy of the database, made off the serving thread. `VACUUM
3 * INTO` held the main thread 8.8 s on 2026-11-02 (#8742): every page, every
4 * request, frozen for as long as the copy took. A worker opens the same file
5 * on its own connection and copies it there; WAL gives it a consistent read
6 * while the forge keeps writing. An in-memory database has no file for
7 * another connection to open, and is copied in place (tests only).
8 */
9
10 import type { Database } from "bun:sqlite";
11 import type { VacuumJob } from "./vacuum_worker.ts";
12
13 export async function vacuumInto(db: Database, to: string): Promise<void> {
14 const from = db.filename;
15 if (from === "" || from === ":memory:") {
16 db.run("VACUUM INTO ?1", [to]);
17 return;
18 }
19 await vacuumFile(from, to);
20 }
21
22 /** The same for a database file nobody holds open here. */
23 export async function vacuumFile(from: string, to: string): Promise<void> {
24 const worker = new Worker(new URL("./vacuum_worker.ts", import.meta.url).href);
25 try {
26 await new Promise<void>((resolve, reject) => {
27 worker.onmessage = (e: MessageEvent<{ error?: string }>) =>
28 e.data.error === undefined ? resolve() : reject(new Error(e.data.error));
29 worker.onerror = (e) => reject(new Error(e.message));
30 worker.postMessage({ from, to } satisfies VacuumJob);
31 });
32 } finally {
33 worker.terminate();
34 }
35 }
src/snapshot/vacuum_worker.ts added +26
@@ -0,0 +1,26 @@
1 /** The worker side of `vacuum.ts`: copy `from` into `to` and say so. */
2
3 import { Database } from "bun:sqlite";
4
5 /** What the worker is asked: the file to read and the copy to make. */
6 export interface VacuumJob {
7 readonly from: string;
8 readonly to: string;
9 }
10
11 declare const self: Worker;
12
13 self.onmessage = (e: MessageEvent<VacuumJob>) => {
14 try {
15 const db = new Database(e.data.from, { readonly: true });
16 try {
17 db.run("PRAGMA busy_timeout = 30000");
18 db.run("VACUUM INTO ?1", [e.data.to]);
19 } finally {
20 db.close();
21 }
22 postMessage({});
23 } catch (error) {
24 postMessage({ error: String(error) });
25 }
26 };
src/tui/screens/resilience.ts +14
@@ -30,6 +30,8 @@ import {
3030 type DestinationInput,
3131 } from "../../actions/destination.ts";
3232 import { forgetMirror, saveMirror } from "../../actions/mirror.ts";
33 import { backupVerdict } from "../../snapshot/bucket.ts";
34 import { lastDrill } from "../../snapshot/drills.ts";
3335 import { resolveDestination } from "../../snapshot/destination.ts";
3436 import { snapshotsOf } from "../../snapshot/service.ts";
3537 import type { Ctx, Screen } from "../app/app.ts";
@@ -71,6 +73,17 @@ export function mirrorLine(f: Session): string {
7173 return where === undefined ? "Mirror: none." : `Mirror: ${where}`;
7274 }
7375
76 /** The bucket's verdict, as home and the MCP's `backups` say it: the line, each lane's newest, what is wrong. */
77 export function verdictLines(f: Session): string[] {
78 const v = backupVerdict(f.app.config.dataDir, lastDrill(f.app.db));
79 const mark = v.state === "red" ? "RED" : v.state === "unknown" ? "unknown" : "ok";
80 return [
81 ...wrap(`Backups ${mark}: ${v.line}`, INNER),
82 ...v.lanes.flatMap((l) => wrap(` ${l.state === "red" ? "!" : " "} ${l.says}`, INNER)),
83 ...[...v.reasons, ...v.unknowns].flatMap((r) => wrap(` - ${r}`, INNER)),
84 ];
85 }
86
7487 const STEPS: readonly { key: keyof DestinationInput; label: string; placeholder?: string; mask?: true }[] = [
7588 { key: "endpoint", label: "Endpoint, an http(s) URL", placeholder: "https://s3.example.com" },
7689 { key: "bucket", label: "Bucket" },
@@ -238,6 +251,7 @@ export function resilienceScreen(f: Session, ctx: Ctx): Screen {
238251 const out: string[] = [];
239252 if (asking !== undefined) out.push(QUESTION[asking], "");
240253 else if (said !== "") out.push(...wrap(said, INNER), "");
254 out.push(...verdictLines(f), "");
241255 out.push(...wrap(destinationLine(f), INNER), ...wrap(mirrorLine(f), INNER));
242256 if (note !== "") out.push(...wrap(note, INNER));
243257 if (mayChange()) out.push("d sets where snapshots go, x forgets it.", "m sets the mirror, f forgets it.", "t tests where snapshots go.");
src/web/studio/kpi_row.ts +4 −4
@@ -92,7 +92,7 @@ export function liveNow(): { sha: string; at: string } | null {
9292
9393 export function kpiRowFor(k: Kpis, live: { sha: string; at: string } | null, now: Date = new Date()): Html {
9494 const idle = k.builders.idle;
95 const overdue = backupOverdue(k.backupAt, now);
95 const overdue = k.backup ? k.backup.state === "red" : backupOverdue(k.backupAt, now);
9696 const spendWords = spendCeilingWords(k.spend);
9797 const rate = k.landed / k.landedHours.length;
9898 return html`<section class="stats kpi-row mb" aria-label="The last 24 hours, as of this page load">
@@ -142,10 +142,10 @@ export function kpiRowFor(k: Kpis, live: { sha: string; at: string } | null, now
142142 ${tile(
143143 "Last off-box backup",
144144 "/settings/backups",
145 backupAge(k.backupAt, now),
145 k.backup?.state === "unknown" && k.backupAt === null ? "—" : backupAge(k.backupAt, now),
146146 backupFull(k.backupAt),
147 k.backupAt === null ? "no backup off the machine yet" : overdue ? "overdue, more than 24 h" : "off the machine",
148 overdue ? "Overdue: the last off-box backup is more than 24 h old" : "Age of the last off-box backup",
147 k.backup ? k.backup.note : k.backupAt === null ? "no backup off the machine yet" : overdue ? "overdue, more than 24 h" : "off the machine",
148 k.backup ? (k.backup.reasons[0] ?? k.backup.unknowns[0] ?? k.backup.line) : overdue ? "Overdue: the last off-box backup is more than 24 h old" : "Age of the last off-box backup",
149149 null,
150150 overdue ? "attn" : null,
151151 )}
test/backup-truth.test.ts added +245
@@ -0,0 +1,245 @@
1 /**
2 * Backups can't lie (#8742): a snapshot lost to a restart is taken again, the
3 * bucket is judged from what is in it, a check that stops is red, a drill
4 * that brought back nothing is red, and the database copy does not freeze
5 * the serving thread.
6 */
7
8 import { afterAll, beforeAll, expect, test } from "bun:test";
9 import { Database } from "bun:sqlite";
10 import { existsSync } from "node:fs";
11 import { mkdtemp, rm } from "node:fs/promises";
12 import { tmpdir } from "node:os";
13 import { join } from "node:path";
14 import { createApp, type App } from "../src/app.ts";
15 import { vitals } from "../src/kernel/vitals.ts";
16 import { backupVerdict, CHECK_STALE_MS, judge, keepReads, LATE_MS, readLane, watchBucket, type LaneRead } from "../src/snapshot/bucket.ts";
17 import type { DrillResult } from "../src/snapshot/drill.ts";
18 import { lastNightly, snapshotDue } from "../src/snapshot/due.ts";
19 import { snapshotsOf, startSnapshots } from "../src/snapshot/service.ts";
20 import { listSnapshots } from "../src/snapshot/snapshot.ts";
21 import { dirStore } from "../src/snapshot/store.ts";
22 import { verdictLines } from "../src/tui/screens/resilience.ts";
23 import { vacuumInto } from "../src/snapshot/vacuum.ts";
24
25 let dataDir: string;
26 let app: App;
27 beforeAll(async () => {
28 dataDir = await mkdtemp(join(tmpdir(), "forge-backup-truth-"));
29 app = createApp({ dataDir, port: 0 });
30 });
31 afterAll(async () => {
32 app.db.close();
33 await rm(dataDir, { recursive: true, force: true });
34 });
35
36 const T = Date.parse("2026-11-02T12:00:00Z");
37 const lane = (newestAt: string | null, over: Partial<LaneRead> = {}): LaneRead => ({
38 lane: "primary",
39 where: "bucket/forge",
40 offBox: true,
41 newestAt,
42 newestName: newestAt,
43 newestBytes: 5_000_000,
44 count: 3,
45 readAt: new Date(T).toISOString(),
46 ...over,
47 });
48 const drill = (over: Partial<DrillResult> = {}): DrillResult => ({
49 at: "2026-11-01T09:00:00Z",
50 snapshotId: "s",
51 ms: 1,
52 counts: { repositories: 4, issues: 9, users: 2, pulls: 1, apps: 1 },
53 health: "ok",
54 verdict: "green",
55 failedStep: null,
56 phases: [],
57 budgetMs: 1,
58 reason: null,
59 ...over,
60 });
61 const fresh = (lanes: LaneRead[], at = T) => ({ lanes, checkedAt: new Date(at).toISOString() });
62
63 test("a snapshot is due when the newest finished before the last 08:00 slot, or there is none", () => {
64 expect(new Date(lastNightly(T)).toISOString()).toBe("2026-11-02T08:00:00.000Z");
65 expect(snapshotDue(undefined, T)).toBe(true);
66 expect(snapshotDue("2026-11-01T08:55:00Z", T)).toBe(true);
67 expect(snapshotDue("2026-11-02T08:55:00Z", T)).toBe(false);
68 // Before today's slot, yesterday's is the one owed.
69 expect(snapshotDue("2026-11-01T08:55:00Z", Date.parse("2026-11-02T07:00:00Z"))).toBe(false);
70 });
71
72 test("a newest snapshot over 26 h old is red; at 25 h it is green", () => {
73 const at = (h: number) => new Date(T - h * 3_600_000).toISOString();
74 const late = judge(fresh([lane(at(26.5))]), drill(), T);
75 expect(late.state).toBe("red");
76 expect(late.reasons.join(" ")).toContain("26 h old");
77 const ok = judge(fresh([lane(at(25))]), drill(), T);
78 expect(ok.state).toBe("green");
79 expect(ok.line).toContain("4.8 MiB");
80 expect(LATE_MS).toBe(26 * 3_600_000);
81 });
82
83 test("every lane is shown with its newest object, and one late lane makes it red", () => {
84 const v = judge(fresh([lane("2026-11-02T08:50:00Z"), lane("2026-10-20T08:50:00Z", { lane: "mirror" })]), drill(), T);
85 expect(v.lanes.map((l) => [l.lane, l.state])).toEqual([["primary", "green"], ["mirror", "red"]]);
86 expect(v.state).toBe("red");
87 });
88
89 test("a lane that cannot be read, or holds nothing, is red", () => {
90 expect(judge(fresh([lane(null, { error: "403" })]), drill(), T).state).toBe("red");
91 expect(judge(fresh([lane(null)]), drill(), T).reasons[0]).toContain("holds no snapshot");
92 });
93
94 test("a check that has not run is red: the dead man's switch", () => {
95 const stale = judge(fresh([lane("2026-11-02T08:50:00Z")], T - CHECK_STALE_MS - 60_000), drill(), T);
96 expect(stale.state).toBe("red");
97 expect(stale.reasons.join(" ")).toContain("has not run");
98 const never = judge(undefined, drill(), T, T - CHECK_STALE_MS - 1);
99 expect(never.state).toBe("red");
100 // Just booted: not red yet, and not calm either — nothing is known.
101 const booting = judge(undefined, drill(), T, T - 1000);
102 expect(booting.reasons).toEqual([]);
103 expect(booting.state).toBe("unknown");
104 expect(booting.line).toBe("Backups not checked yet");
105 expect(booting.note).toBe("not checked yet");
106 });
107
108 test("a drill that failed, or passed with no data back, is red", () => {
109 const l = [lane("2026-11-02T08:50:00Z")];
110 expect(judge(fresh(l), drill({ verdict: "red", failedStep: "open", health: "unreached" }), T).state).toBe("red");
111 const empty = judge(fresh(l), drill({ counts: { repositories: 0, issues: 0, users: 0, pulls: 0, apps: 0 } }), T);
112 expect(empty.state).toBe("red");
113 expect(empty.drill?.says).toContain("no data");
114 expect(judge(fresh(l), drill(), T).state).toBe("green");
115 });
116
117 test("a bucket read with no restore drill yet is unknown, never green, and raises no alarm", () => {
118 const v = judge(fresh([lane("2026-11-02T08:50:00Z")]), undefined, T);
119 expect(v.state).toBe("unknown");
120 expect(v.unknowns.join(" ")).toContain("No restore drill");
121 expect(v.reasons).toEqual([]);
122 });
123
124 test("a store's error keeps no credential, presigned query or userinfo", async () => {
125 const secret = "SUPERSECRETKEY123";
126 const bad = {
127 ...dirStore(dataDir),
128 where: "bucket/forge",
129 names: () =>
130 Promise.reject(new Error(`403 for https://u:[email protected]/b/k?X-Amz-Signature=abcdef&X-Amz-Credential=AKIA now ${secret} ${secret}`)),
131 };
132 const r = await readLane("primary", bad, T, () => [{ endpoint: "https://s3.example.com", bucket: "b", accessKeyId: "AKIAX", secretAccessKey: secret }]);
133 expect(r.error).not.toContain(secret);
134 expect(r.error).not.toContain("abcdef");
135 expect(r.error).not.toContain("u:p@");
136 expect(JSON.stringify(judge(fresh([r]), drill(), T))).not.toContain(secret);
137 });
138
139 test("the terminal's Resilience dialog shows the verdict, the lanes and what is wrong", async () => {
140 const w = watchBucket(dataDir, () => [], 3_600_000, 3_600_000);
141 try {
142 keepReads(dataDir, [lane(new Date(Date.now() - 40 * 3_600_000).toISOString())]);
143 const lines = verdictLines({ app } as never).join("\n");
144 expect(lines).toContain("Backups RED");
145 expect(lines).toContain("primary");
146 expect(lines).toContain("old");
147 } finally {
148 w.stop();
149 }
150 });
151
152 test("vitals raises one backup card from the bucket, and none while it is fine", () => {
153 const w = watchBucket(dataDir, () => [], 3_600_000, 3_600_000);
154 try {
155 const now = Date.now();
156 keepReads(dataDir, [lane(new Date(now - 30 * 3_600_000).toISOString())], now);
157 const v = vitals({ db: app.db, signals: app.signals, dataDir, now });
158 expect(v.degraded.filter((d) => d.cause === "backup")).toHaveLength(1);
159 expect(backupVerdict(dataDir, undefined, now + LATE_MS * 2).state).toBe("red");
160 keepReads(dataDir, [lane(new Date(now - 3_600_000).toISOString())], now);
161 expect(vitals({ db: app.db, signals: app.signals, dataDir, now }).degraded.some((d) => d.cause === "backup")).toBe(false);
162 } finally {
163 w.stop();
164 }
165 });
166
167 test("readLane reads the store itself, and a store that fails is a red read, not a throw", async () => {
168 const dir = await mkdtemp(join(tmpdir(), "forge-lane-"));
169 try {
170 const empty = await readLane("primary", dirStore(dir));
171 expect(empty.newestAt).toBeNull();
172 expect(empty.count).toBe(0);
173 const broken = await readLane("primary", { ...dirStore(dir), names: () => Promise.reject(new Error("boom")) });
174 expect(broken.error).toContain("boom");
175 } finally {
176 await rm(dir, { recursive: true, force: true });
177 }
178 });
179
180 test("a restart with a snapshot lost mid-run takes one shortly after boot", async () => {
181 const dir = await mkdtemp(join(tmpdir(), "forge-lost-"));
182 const store = dirStore(dir);
183 // What a release at 08:21 leaves: objects of a snapshot, and no manifest.
184 await store.put("snapshots/2026-11-02T08-00-00Z/forge.db.0", new Uint8Array([1, 2, 3]));
185 expect(await listSnapshots(store)).toHaveLength(0);
186 const svc = startSnapshots(app, store, { version: "t", snapshotCatchUpMs: 20, bucketEveryMs: 3_600_000, bucketFirstMs: 3_600_000 });
187 try {
188 const until = Date.now() + 20_000;
189 while ((await listSnapshots(store)).length === 0 && Date.now() < until) await Bun.sleep(50);
190 expect((await listSnapshots(store)).length).toBe(1);
191 // The bucket watch then reads it from the store.
192 await svc.checkBucket();
193 const v = backupVerdict(dataDir, undefined);
194 expect(v.lanes[0]?.count).toBe(1);
195 expect(snapshotsOf(app)).toBe(svc);
196 } finally {
197 await svc.stop();
198 await rm(dir, { recursive: true, force: true });
199 }
200 }, 30_000);
201
202 test("a boot with the night's snapshot in the store takes none; a day later it takes one", async () => {
203 const dir = await mkdtemp(join(tmpdir(), "forge-kept-"));
204 const store = dirStore(dir);
205 const first = startSnapshots(app, store, { version: "t", nightly: false });
206 const made = await first.now();
207 await first.stop();
208 const done = Date.parse(made.finishedAt);
209 let clock = done + 1000;
210 const svc = startSnapshots(app, store, { version: "t", nightly: false, now: () => clock });
211 try {
212 expect(snapshotDue(made.finishedAt, clock)).toBe(false);
213 expect(await svc.takeIfDue()).toBe(false);
214 expect(await listSnapshots(store)).toHaveLength(1);
215 clock = done + 25 * 3_600_000;
216 expect(snapshotDue(made.finishedAt, clock)).toBe(true);
217 // Snapshot names are by wall second; wait one out so the second does not overwrite the first.
218 await Bun.sleep(1100);
219 expect(await svc.takeIfDue()).toBe(true);
220 expect(await listSnapshots(store)).toHaveLength(2);
221 } finally {
222 await svc.stop();
223 await rm(dir, { recursive: true, force: true });
224 }
225 }, 30_000);
226
227 test("copying the database does not hold the serving thread (#7559)", async () => {
228 const file = join(dataDir, "big.db");
229 const db = new Database(file);
230 db.run("PRAGMA journal_mode = WAL");
231 db.run("CREATE TABLE t (x BLOB)");
232 const blob = new Uint8Array(100_000);
233 db.transaction(() => {
234 for (let i = 0; i < 1500; i++) db.query("INSERT INTO t VALUES (?1)").run(blob);
235 })();
236 const to = join(dataDir, "big-copy.db");
237 // Inline, VACUUM INTO has finished before the call returns; off the thread, it has not begun.
238 const done = vacuumInto(db, to);
239 expect(existsSync(to)).toBe(false);
240 await done;
241 const copy = new Database(to, { readonly: true });
242 expect(copy.query<{ n: number }, []>("SELECT count(*) AS n FROM t").get()?.n).toBe(1500);
243 copy.close();
244 db.close();
245 }, 60_000);
test/mcp-restore-drill.test.ts +3 −2
@@ -101,11 +101,12 @@ describe("MCP restore_drill", () => {
101101 expect(r.error).toBe(false);
102102 expect(JSON.parse(r.text)).toEqual({ verdict: "red", failedStep: "unseal", says: "The drill failed at unseal." });
103103 });
104 test("a drill that throws is an answer, not an error", async () => {
104 test("a drill that throws is kept as a red drill, not as nothing (#8742)", async () => {
105105 mode = "throw";
106106 const r = await call(token("alice"));
107107 expect(r.error).toBe(false);
108 expect(JSON.parse(r.text)).toEqual({ verdict: "none", says: "The drill could not run: Error: store went away" });
108 expect(JSON.parse(r.text)).toEqual({ verdict: "red", failedStep: "restore", says: "The drill failed at restore." });
109 expect(lastDrill(app.db)?.reason).toContain("store went away");
109110 });
110111 test("an admin's read-only token is refused and nothing runs", async () => {
111112 const r = await call(token("alice", "ro"));

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs