trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

Every platform backup lane names its newest object and its age, read from its store, on the Resilience screen and in MCP backups #8831

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/8811-every-platform-backup-lane-names-its-new → main 3 commits fork 1 thread · 1 unresolved

  • Merged. torque/8811-every-platform-backup-lane-names-its-new was merged into main as 08b0f8d.

Review

Sign in to review this change.
7 files changed +693 −7
docs/SNAPSHOT.md +4
@@ -121,6 +121,10 @@ as failed. A put's deadline follows its size: its bytes at a floor of
121121 5 Mbit/s, plus 60 s. A fixed 180 s once failed every attempt at a 334 MB
122122 database behind a 15 Mbit/s uplink (#2536).
123123
124 ## Backup lanes
125
126 The platform's backup lanes (`FORGE_BACKUP_LANES`) and how each is judged: docs/features/backup-truth.md, "Platform backup lanes".
127
124128 ## Progress
125129
126130 A snapshot uploads eight objects at a time. Run logs older than a day go up
docs/features/backup-truth.md +31 −1
@@ -6,7 +6,37 @@
66 - **Catch-up.** A boot with the newest snapshot older than the last 08:00 UTC slot takes one two minutes later (`snapshotDue`). A snapshot cut short by a release has no manifest, so it counts as lost.
77 - **In the terminal.** The Resilience dialog (palette: Resilience, admins) opens with the same verdict, each lane's newest, and every reason. On the web it is home's backup tile and `/settings/backups`; for an agent, `verdict` in the MCP `backups` tool.
88 - **Off the serving thread.** `VACUUM INTO` runs in a worker (`vacuum.ts`).
9 - **Not done, tracked:** platform/K8s lanes on home (#8805); snapshots outside the serving process or resumable (#8828; the off-process probe is below). Catch-up retakes a lost snapshot whole.
9 - **Not done, tracked:** snapshots outside the serving process or resumable (#8828; the off-process probe is below). Catch-up retakes a lost snapshot whole.
10
11 ## Platform backup lanes (#8811)
12
13 Every other place a backup lands — one per CronJob in the `backup` namespace —
14 is a lane the platform declares in `FORGE_BACKUP_LANES`, a JSON list kept in
15 the forge's configuration, never in this repository:
16
17 ```json
18 [{ "name": "backup/postgres", "endpoint": "https://s3.example.com", "bucket": "platform-backups",
19 "prefix": "postgres/", "region": "us-east-1", "accessKey": "…", "secretKey": "…" }]
20 ```
21
22 - **List-only keys.** `src/snapshot/lanes.ts` only lists; give each lane keys
23 that can do nothing else.
24 - **Read on the watch's clock.** Each pass reads every lane beside `primary` and
25 `mirror`, and the verdict names each one's newest object, size and age.
26 - **Bounded, never a full listing.** One delimited listing per folder level (at
27 most three pages of 1,000), walking only into the folder whose name sorts
28 last, since backup names carry their date. A level wider than that is read
29 in part: a fresh object found still proves the lane fresh; a stale one, or
30 none, is red and says to name a narrower prefix.
31 - **Red when:** the store does not answer within 10 s, nothing is under the
32 prefix, or the newest object is over 26 h old. A lane the configuration
33 names but that does not read whole is red with what is wrong, never with its
34 values.
35 - **Nothing unsafe printed.** Errors are scrubbed of keys, and every key or
36 error the store wrote has its control bytes replaced with `?` before it
37 reaches a terminal or an MCP client.
38 - **Where to see it.** MCP `backups` → `verdict.lanes`; the terminal's
39 Resilience dialog; home's backup tile reads the same verdict.
1040
1141 ## The probe outside the forge (#8806, part 1)
1242
src/snapshot/bucket.ts +24 −5
@@ -86,6 +86,13 @@ export function sizeWords(bytes: number): string {
8686 return `${(bytes / 1_073_741_824).toFixed(2)} GiB`;
8787 }
8888
89 /**
90 * Text a store wrote (an object key, its error), safe to print: every C0/C1
91 * control byte and bidi override becomes `?`, so an ESC in a key never
92 * reaches a terminal or an MCP client's screen (#8811).
93 */
94 export const printable = (text: string): string => text.replace(/[\u0000-\u001f\u007f-\u009f\u202a-\u202e\u2066-\u2069]/g, "?");
95
8996 /** An error as one short sentence with no key and no presigned query in it. */
9097 export function scrubError(targets: readonly (S3Target | undefined)[], error: unknown): string {
9198 let reason = String(error instanceof Error ? error.message : error);
@@ -166,14 +173,19 @@ export function judge(
166173 for (const r of state?.lanes ?? []) {
167174 let says: string;
sentinelcrew outdated

newestName is a key written by the store, so it can hold control bytes; sanitise before it joins the line that the terminal and MCP print.
168175 let red = true;
169 if (r.error !== undefined) says = `${r.lane} (${r.where}) cannot be read: ${r.error}`;
170 else if (r.newestAt === null) says = `${r.lane} (${r.where}) holds no snapshot`;
176 // Every word a store or the configuration supplied is made printable before it joins the line.
177 const lane = printable(r.lane);
178 const where = printable(r.where);
179 if (r.error !== undefined) says = `${lane} (${where}) cannot be read: ${printable(r.error)}`;
180 else if (r.newestAt === null) says = `${lane} (${where}) holds no snapshot`;
171181 else {
172182 const age = now - Date.parse(r.newestAt);
173 if (!(age <= LATE_MS)) says = `${r.lane} (${r.where}) newest snapshot is ${ageWords(age)} old`;
183 // The newest object by name, so a person can find it in the store (#8811).
184 const name = r.newestName === null ? "" : ` ${printable(r.newestName)}`;
185 if (!(age <= LATE_MS)) says = `${lane} (${where}) newest snapshot${name} is ${ageWords(age)} old`;
174186 else {
175187 red = false;
176 says = `${r.lane} newest ${ageWords(age)} old, ${sizeWords(r.newestBytes ?? 0)}`;
188 says = `${lane} newest${name} ${ageWords(age)} old, ${sizeWords(r.newestBytes ?? 0)}`;
177189 }
178190 }
179191 if (red) {
@@ -248,13 +260,20 @@ export function watchBucket(
248260 firstMs = 5_000,
249261 /** The credentials in use, struck out of any error a read keeps. */
250262 targets: () => readonly (S3Target | undefined)[] = () => [],
263 /** The platform's own lanes (`lanes.ts`), read after the forge's and judged beside them. */
264 declared: (now: number) => Promise<readonly LaneRead[]> = async () => [],
251265 ): { stop: () => void; pass: () => Promise<void> } {
252266 watching.set(dataDir, new Date().toISOString());
253267 let stopped = false;
254268 let timer: ReturnType<typeof setTimeout> | undefined;
255269 const pass = async (): Promise<void> => {
256270 const stores = lanes().filter((l): l is { lane: string; store: Store } => l.store !== undefined);
257 keepReads(dataDir, await Promise.all(stores.map((l) => readLane(l.lane, l.store, Date.now(), targets))));
271 const now = Date.now();
272 const [own, more] = await Promise.all([
273 Promise.all(stores.map((l) => readLane(l.lane, l.store, now, targets))),
274 declared(now).catch(() => []),
275 ]);
276 keepReads(dataDir, [...own, ...more]);
258277 };
259278 const next = (ms: number): void => {
260279 if (stopped) return;
src/snapshot/lanes.ts added +258
@@ -0,0 +1,258 @@
1 /**
2 * The platform's backup lanes, read from their stores (#8811).
3 *
4 * The forge's own lanes (`primary`, `mirror`) are read by manifest in
5 * `bucket.ts`. Every other place backups land — one per CronJob in the
6 * `backup` namespace — the platform declares in `FORGE_BACKUP_LANES`, a JSON
7 * list of `{ name, endpoint, bucket, prefix, region?, accessKey, secretKey }`
8 * kept in the forge's configuration, never in this repository, with keys that
9 * can only list. This module only lists: it holds no put and no delete. Each
10 * lane is read here into `bucket.ts`'s `LaneRead`, the bucket watch keeps it
11 * beside the forge's own, and `judge` says it: red when its store does not
12 * answer, when nothing is under its prefix, or past 26 h. Home, vitals, MCP
13 * `backups` and the terminal's Resilience screen all read that one verdict.
14 *
15 * Never a full listing. One delimited listing per folder level, at most
16 * `PAGES` pages of 1,000 each, and only the folder whose name sorts last is
17 * walked into: backup names carry their date (`2026-11-02T…`), so the last
18 * name is the newest. The newest object is the one with the latest
19 * LastModified among those read. When a level has more than `PAGES` pages
20 * the read is bounded: a fresh object found still proves the lane fresh; a
21 * stale one or none does not prove it stale, and the lane says so.
22 *
23 * Credentials never leave: a lane's place is `host/bucket/prefix`, and an
24 * error is one line with the lane's keys struck out (`scrubError`).
25 */
26
27 import { LATE_MS, printable, scrubError, type LaneRead } from "./bucket.ts";
28 import { isOffBox } from "./off_box.ts";
29 import { s3Where, type S3Target } from "./store.ts";
30
31 /** Pages read at one folder level before the read counts as bounded. */
32 export const PAGES = 3;
33 /** Entries asked for per page; S3's own maximum. */
34 export const PAGE_KEYS = 1_000;
35 /** Folder levels walked into at most. */
36 export const DEPTH = 6;
37 /** How long one lane may take to answer, all its pages together. */
38 export const LANE_DEADLINE_MS = 10_000;
39 /** Lanes read from configuration at most. */
40 export const MAX_LANES = 32;
41
42 export interface LaneObject {
43 readonly key: string;
44 readonly bytes: number;
45 /** LastModified, ISO 8601. */
46 readonly at: string;
47 }
48
49 /** One delimited page: the objects at this level and the folders under it. */
50 export interface LanePage {
51 readonly objects: readonly LaneObject[];
52 readonly folders: readonly string[];
53 /** Continue with this, when there is more. */
54 readonly next?: string | undefined;
55 }
56
57 /** A store as a lane sees it: one listing page at a time, nothing else. */
58 export interface Lister {
59 /** The store's place, never a key. */
60 readonly where: string;
61 page(prefix: string, next?: string): Promise<LanePage>;
62 }
63
64 /** A lane as declared: where it lists, never yet read. */
65 export interface Lane {
66 readonly name: string;
67 readonly prefix: string;
68 readonly lister: Lister;
69 /** Struck out of every error this lane says. */
70 readonly target?: S3Target | undefined;
71 }
72
73 /** The newest object under `prefix`, walking only the last-named folder at each level; `seen` is how many objects were read. */
74 export async function newestUnder(
75 lister: Lister,
76 prefix: string,
77 ): Promise<{ newest: LaneObject | undefined; bounded: boolean; seen: number }> {
78 let newest: LaneObject | undefined;
79 let bounded = false;
80 let seen = 0;
81 let at = prefix;
82 for (let depth = 0; depth < DEPTH; depth++) {
83 let last: string | undefined;
84 let next: string | undefined;
85 let pages = 0;
86 do {
87 const page = await lister.page(at, next);
88 pages++;
89 seen += page.objects.length;
90 for (const o of page.objects) if (newest === undefined || Date.parse(o.at) > Date.parse(newest.at)) newest = o;
91 for (const f of page.folders) if (last === undefined || f > last) last = f;
92 next = page.next;
93 } while (next !== undefined && pages < PAGES);
94 if (next !== undefined) bounded = true;
95 if (last === undefined) break;
96 at = last;
97 }
98 return { newest, bounded, seen };
99 }
100
101 function deadline<T>(work: Promise<T>, ms: number): Promise<T> {
102 let timer: ReturnType<typeof setTimeout> | undefined;
103 const late = new Promise<never>((_, reject) => {
104 timer = setTimeout(() => reject(new Error(`no answer in ${Math.round(ms / 1000)} s`)), ms);
105 });
106 work.catch(() => {});
107 return Promise.race([work, late]).finally(() => clearTimeout(timer));
108 }
109
110 /** Read one lane into the watch's shape. Never throws: a store that refuses is a read with its reason. */
111 export async function readObjectLane(lane: Lane, now: number, deadlineMs = LANE_DEADLINE_MS): Promise<LaneRead> {
112 const where = lane.prefix === "" ? lane.lister.where : `${lane.lister.where}/${lane.prefix}`;
113 const base = { lane: lane.name, where, offBox: isOffBox({ where: lane.lister.where }), readAt: new Date(now).toISOString() };
114 const none = { newestAt: null, newestName: null, newestBytes: null, count: 0 };
115 let found: Awaited<ReturnType<typeof newestUnder>>;
116 try {
117 found = await deadline(newestUnder(lane.lister, lane.prefix), deadlineMs);
118 } catch (error) {
119 // One line whatever the store said: a proxy's refusal ends in a newline.
120 return { ...base, ...none, error: printable(scrubError([lane.target], error).replace(/\s+/g, " ").trim()) };
121 }
122 const o = found.newest;
123 const read = o === undefined ? none : { newestAt: o.at, newestName: o.key, newestBytes: o.bytes, count: found.seen };
124 // A bounded read proves a lane fresh, never stale or empty: it says what it could not see.
125 if (found.bounded && (o === undefined || now - Date.parse(o.at) > LATE_MS))
126 return { ...base, ...read, error: `only the first ${PAGES * PAGE_KEYS} entries of a folder were read; name a narrower prefix` };
127 return { ...base, ...read };
128 }
129
130 /** A bucket listed one delimited page at a time, one attempt per call. */
131 export function s3Lister(target: S3Target): Lister {
132 const client = new Bun.S3Client({
133 endpoint: target.endpoint,
134 bucket: target.bucket,
135 accessKeyId: target.accessKeyId,
136 secretAccessKey: target.secretAccessKey,
137 ...(target.region === undefined ? {} : { region: target.region }),
138 });
139 return {
140 where: s3Where(target),
141 async page(prefix, next) {
142 const page = await client.list({
143 prefix,
144 delimiter: "/",
145 maxKeys: PAGE_KEYS,
146 ...(next === undefined ? {} : { continuationToken: next }),
147 });
148 return {
149 objects: (page.contents ?? []).flatMap((c) => {
150 const at = c.lastModified === undefined ? NaN : Date.parse(c.lastModified);
151 // A key is the store's text: no control byte of it is kept.
152 return Number.isFinite(at) ? [{ key: printable(c.key), bytes: c.size ?? 0, at: new Date(at).toISOString() }] : [];
153 }),
154 folders: (page.commonPrefixes ?? []).map((p) => p.prefix),
155 next: page.isTruncated ? page.nextContinuationToken : undefined,
156 };
157 },
158 };
159 }
160
161 const NAME = /^[A-Za-z0-9][A-Za-z0-9 ._:/-]{0,63}$/;
162 const BUCKET = /^[A-Za-z0-9][A-Za-z0-9._-]{0,62}$/;
163
164 const text = (v: unknown): string | undefined => (typeof v === "string" && v.trim() !== "" ? v.trim() : undefined);
165
166 /** One declared lane, or why it is refused; the sentence never carries a value but the name. */
167 function declared(raw: unknown, i: number): { name: string; target: S3Target; prefix: string } | { name: string; refused: string } {
168 const o = typeof raw === "object" && raw !== null && !Array.isArray(raw) ? (raw as Record<string, unknown>) : {};
169 const given = text(o["name"]);
170 const name = given !== undefined && NAME.test(given) ? given : `lane ${i + 1}`;
171 if (given === undefined || !NAME.test(given)) return { name, refused: "its name is missing or not 1–64 letters, digits, space . _ : / -" };
172 const endpoint = text(o["endpoint"]);
173 let url: URL | undefined;
174 try {
175 url = endpoint === undefined ? undefined : new URL(endpoint);
176 } catch {
177 url = undefined;
178 }
179 if (url === undefined || (url.protocol !== "https:" && url.protocol !== "http:") || url.username !== "" || url.password !== "")
180 return { name, refused: "its endpoint is not an http(s) URL" };
181 const bucket = text(o["bucket"]);
182 if (bucket === undefined || !BUCKET.test(bucket)) return { name, refused: "its bucket is missing or not a bucket name" };
183 const prefix = o["prefix"] === undefined ? "" : o["prefix"];
184 // Control bytes and a leading slash never reach a listing.
185 if (typeof prefix !== "string" || prefix.length > 512 || /[\u0000-\u001f\u007f]/.test(prefix) || prefix.startsWith("/"))
186 return { name, refused: "its prefix is not a string of at most 512 printable characters" };
187 const accessKeyId = text(o["accessKey"]);
188 const secretAccessKey = text(o["secretKey"]);
189 if (accessKeyId === undefined || secretAccessKey === undefined) return { name, refused: "its accessKey or secretKey is missing" };
190 const region = o["region"] === undefined ? undefined : text(o["region"]);
191 if (o["region"] !== undefined && (region === undefined || !/^[A-Za-z0-9-]{1,32}$/.test(region)))
192 return { name, refused: "its region is not a region name" };
193 return { name, prefix, target: { endpoint: url.href.replace(/\/$/, ""), bucket, accessKeyId, secretAccessKey, region } };
194 }
195
196 export interface Declared {
197 readonly lanes: readonly { name: string; target: S3Target; prefix: string }[];
198 /** Lanes the configuration names but that do not read whole: each a red lane on the screen. */
199 readonly refused: readonly { name: string; says: string }[];
200 }
201
202 /** `FORGE_BACKUP_LANES` read at the boundary: every field checked, a bad lane refused by name, never echoed. */
203 export function parseLanes(raw: string | undefined): Declared {
204 if (raw === undefined || raw.trim() === "") return { lanes: [], refused: [] };
205 let list: unknown;
206 try {
207 list = JSON.parse(raw);
208 } catch {
209 list = undefined;
210 }
211 if (!Array.isArray(list))
212 return { lanes: [], refused: [{ name: "FORGE_BACKUP_LANES", says: "configuration: not a JSON list of lanes" }] };
213 const lanes: Declared["lanes"][number][] = [];
214 const refused: { name: string; says: string }[] = [];
215 const seen = new Set<string>();
216 list.slice(0, MAX_LANES).forEach((raw, i) => {
217 const d = declared(raw, i);
218 if ("refused" in d) refused.push({ name: d.name, says: `configuration: ${d.refused}` });
219 else if (seen.has(d.name)) refused.push({ name: d.name, says: "configuration: a second lane of this name" });
220 else {
221 seen.add(d.name);
222 lanes.push(d);
223 }
224 });
225 if (list.length > MAX_LANES)
226 refused.push({ name: "FORGE_BACKUP_LANES", says: `configuration: ${list.length} lanes, only the first ${MAX_LANES} read` });
227 return { lanes, refused };
228 }
229
230 /**
231 * Every declared lane read at once, a lane the configuration refuses kept as
232 * a read with its reason. The bucket watch calls this on each pass.
233 */
234 export async function readDeclaredLanes(
235 env: Record<string, string | undefined> = process.env,
236 now = Date.now(),
237 /** A lister for a bucket; tests hand a fake. */
238 listerFor: (target: S3Target) => Lister = s3Lister,
239 ): Promise<LaneRead[]> {
240 const { lanes, refused } = parseLanes(env["FORGE_BACKUP_LANES"]);
241 const read = await Promise.all(
242 lanes.map((d) => readObjectLane({ name: d.name, prefix: d.prefix, lister: listerFor(d.target), target: d.target }, now)),
243 );
244 return [
245 ...read,
246 ...refused.map((r) => ({
247 lane: r.name,
248 where: "FORGE_BACKUP_LANES",
249 offBox: false,
250 newestAt: null,
251 newestName: null,
252 newestBytes: null,
253 count: 0,
254 error: r.says,
255 readAt: new Date(now).toISOString(),
256 })),
257 ];
258 }
src/snapshot/service.ts +2
@@ -25,6 +25,7 @@ import { describeTimes, runDrill, type DrillDeps, type DrillResult } from "./dri
2525 import { BOOT_RED_FLOOR_MS, drillDue, RED_RETRY_MS, snapshotDue } from "./due.ts";
2626 import { sealedTarget } from "./destination.ts";
2727 import { watchBucket } from "./bucket.ts";
28 import { readDeclaredLanes } from "./lanes.ts";
2829 import { lastDrill, recordDrill } from "./drills.ts";
2930 import { lastMirror, mirrorPass, mirrorTarget, MIRROR_EVERY_MS, recordMirror, type MirrorState } from "./mirror.ts";
3031 import { isOffBox } from "./off_box.ts";
@@ -430,6 +431,7 @@ export function startSnapshots(
430431 return [s3FromEnv()];
431432 }
432433 },
434 (now) => readDeclaredLanes(process.env, now),
433435 );
434436
435437 const handle: SnapshotsHandle = {
src/snapshot/store.ts +5 −1
@@ -184,6 +184,10 @@ export function s3Store(target: S3Target, attempts?: number): Store {
184184 return withRetries(rawS3Store(target), attempts === undefined ? {} : { attempts });
185185 }
186186
187 /** A bucket's place for a page and a log: `host[:port]/bucket`, never a key. */
188 export const s3Where = (target: Pick<S3Target, "endpoint" | "bucket">): string =>
189 `${target.endpoint.replace(/^https?:\/\//, "")}/${target.bucket}`;
190
187191 function rawS3Store(target: S3Target): Store {
188192 const client = new Bun.S3Client({
189193 endpoint: target.endpoint,
@@ -193,7 +197,7 @@ function rawS3Store(target: S3Target): Store {
193197 ...(target.region === undefined ? {} : { region: target.region }),
194198 });
195199 return {
196 where: `${target.endpoint.replace(/^https?:\/\//, "")}/${target.bucket}`,
200 where: s3Where(target),
197201 async put(key, body) {
198202 await client.write(key, body);
199203 },
test/backup-lanes.test.ts added +369
@@ -0,0 +1,369 @@
1 /**
2 * Backup lanes (#8811): each platform lane's newest object and age, read
3 * from its store with a fake one — fresh, 27 h old, empty, refusing —
4 * bounded to the last-named folder, judged by the one bucket verdict home,
5 * MCP `backups` and Resilience read, and never a credential in the answer.
6 */
7
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { createApp, type App } from "../src/app.ts";
13 import { createToken } from "../src/core/tokens.ts";
14 import { createUser } from "../src/core/users.ts";
15 import { serve, type Servers } from "../src/http/server.ts";
16 import { backupVerdict, judge, LATE_MS, watchBucket, type LaneRead } from "../src/snapshot/bucket.ts";
17 import type { DrillResult } from "../src/snapshot/drill.ts";
18 import {
19 PAGES,
20 newestUnder,
21 parseLanes,
22 readDeclaredLanes,
23 readObjectLane,
24 s3Lister,
25 type LanePage,
26 type Lister,
27 } from "../src/snapshot/lanes.ts";
28 import { dirStore } from "../src/snapshot/store.ts";
29 import { verdictLines } from "../src/tui/screens/resilience.ts";
30
31 const NOW = Date.parse("2026-11-02T12:00:00Z");
32 const H = 3_600_000;
33 const SECRET = "sk-very-secret-lane-key";
34 const ACCESS = "AKIA-lane-access";
35
36 /** A bucket in memory: keys with times, listed one delimited level at a time, `perPage` entries a page. */
37 function fakeStore(objects: Record<string, { bytes: number; at: number }>, perPage = 1000) {
38 const calls: string[] = [];
39 const lister: Lister = {
40 where: "s3.test/bucket",
41 async page(prefix, next) {
42 calls.push(prefix);
43 const entries = new Map<string, LanePage["objects"][number] | "folder">();
44 for (const [key, o] of Object.entries(objects)) {
45 if (!key.startsWith(prefix)) continue;
46 const rest = key.slice(prefix.length);
47 const slash = rest.indexOf("/");
48 if (slash === -1) entries.set(key, { key, bytes: o.bytes, at: new Date(o.at).toISOString() });
49 else entries.set(prefix + rest.slice(0, slash + 1), "folder");
50 }
51 const sorted = [...entries.keys()].sort();
52 const from = next === undefined ? 0 : Number(next);
53 const slice = sorted.slice(from, from + perPage);
54 return {
55 objects: slice.flatMap((k) => {
56 const e = entries.get(k)!;
57 return e === "folder" ? [] : [e];
58 }),
59 folders: slice.filter((k) => entries.get(k) === "folder"),
60 next: from + perPage < sorted.length ? String(from + perPage) : undefined,
61 };
62 },
63 };
64 return { lister, calls };
65 }
66
67 const refusing: Lister = {
68 where: "s3.test/bucket",
69 async page() {
70 throw new Error(`403 Forbidden for ${ACCESS} signed with ${SECRET}\n`);
71 },
72 };
73
74 const target = { endpoint: "https://s3.test", bucket: "bucket", accessKeyId: ACCESS, secretAccessKey: SECRET };
75
76 const drill: DrillResult = {
77 at: "2026-11-02T09:00:00Z",
78 snapshotId: "s",
79 ms: 1,
80 counts: { repositories: 4, issues: 9, users: 2, pulls: 1, apps: 1 },
81 health: "ok",
82 verdict: "green",
83 failedStep: null,
84 phases: [],
85 budgetMs: 1,
86 reason: null,
87 };
88 /** The one verdict, over these reads, now. */
89 const verdict = (lanes: LaneRead[]) => judge({ lanes, checkedAt: new Date(NOW).toISOString() }, drill, NOW);
90
91 describe("one lane", () => {
92 test("a fresh lane is green and names its newest object, size and age", async () => {
93 const { lister } = fakeStore({
94 "pg/2026-11-01.sql.gz": { bytes: 2048, at: NOW - 27 * H },
95 "pg/2026-11-02.sql.gz": { bytes: 3 * 1024 * 1024, at: NOW - 3 * H },
96 });
97 const r = await readObjectLane({ name: "backup/postgres", prefix: "pg/", lister }, NOW);
98 expect(r).toMatchObject({
99 lane: "backup/postgres",
100 where: "s3.test/bucket/pg/",
101 offBox: true,
102 newestName: "pg/2026-11-02.sql.gz",
103 newestBytes: 3 * 1024 * 1024,
104 newestAt: new Date(NOW - 3 * H).toISOString(),
105 });
106 const v = verdict([r]);
107 expect(v.state).toBe("green");
108 expect(v.lanes[0]!.says).toBe("backup/postgres newest pg/2026-11-02.sql.gz 3 h old, 3.0 MiB");
109 });
110
111 test("a lane 27 h old is red and says it is late, naming the object it found", async () => {
112 const { lister } = fakeStore({ "pg/old.sql.gz": { bytes: 100, at: NOW - 27 * H } });
113 const v = verdict([await readObjectLane({ name: "backup/postgres", prefix: "pg/", lister }, NOW)]);
114 expect(v.state).toBe("red");
115 expect(v.reasons).toEqual(["Backup/postgres (s3.test/bucket/pg/) newest snapshot pg/old.sql.gz is 27 h old."]);
116 });
117
118 test("exactly 26 h old is green; a minute past is red", async () => {
119 const at = (ms: number) => fakeStore({ "pg/x.sql": { bytes: 1, at: NOW - ms } }).lister;
120 const edge = await readObjectLane({ name: "edge", prefix: "pg/", lister: at(LATE_MS) }, NOW);
121 expect(verdict([edge]).lanes[0]!.state).toBe("green");
122 const past = await readObjectLane({ name: "edge", prefix: "pg/", lister: at(LATE_MS + 60_000) }, NOW);
123 expect(verdict([past]).lanes[0]!.state).toBe("red");
124 });
125
126 test("an ESC or other control byte a store wrote never reaches the line", () => {
127 const base = { lane: "backup/x", where: "s3.test/b", offBox: true, newestBytes: 1, count: 1, readAt: new Date(NOW).toISOString() };
128 const v = verdict([
129 { ...base, newestAt: new Date(NOW - H).toISOString(), newestName: "pg/\u001b[2J\u001b]0;owned\u0007dump\u202e.sql" },
130 { ...base, newestAt: null, newestName: null, error: "denied \u001b[31mred\u009b" },
131 ]);
132 const text = JSON.stringify([v.lanes.map((l) => l.says), v.reasons]);
133 expect(text).not.toMatch(/\\u00(1b|07|9b)|\\u202e/);
134 expect(v.lanes[0]!.says).toBe("backup/x newest pg/?[2J?]0;owned?dump?.sql 1 h old, 1 B");
135 expect(v.lanes[1]!.says).toBe("backup/x (s3.test/b) cannot be read: denied ?[31mred?");
136 });
137
138 test("an empty lane is red and says it holds nothing", async () => {
139 const { lister } = fakeStore({ "other/x": { bytes: 1, at: NOW } });
140 const r = await readObjectLane({ name: "backup/redis", prefix: "redis/", lister }, NOW);
141 expect(r.newestAt).toBeNull();
142 expect(verdict([r]).reasons).toEqual(["Backup/redis (s3.test/bucket/redis/) holds no snapshot."]);
143 });
144
145 test("a store that refuses is red, says so on one line, and never carries its keys", async () => {
146 const r = await readObjectLane({ name: "backup/postgres", prefix: "pg/", lister: refusing, target }, NOW);
147 expect(r.error).toBe("403 Forbidden for … signed with …");
148 const v = verdict([r]);
149 expect(v.state).toBe("red");
150 expect(v.reasons[0]).toContain("cannot be read: 403 Forbidden");
151 expect(JSON.stringify(v)).not.toContain(SECRET);
152 expect(JSON.stringify(v)).not.toContain(ACCESS);
153 });
154
155 test("a store that never answers is red within the lane's deadline", async () => {
156 const hung: Lister = { where: "s3.test/bucket", page: () => new Promise(() => {}) };
157 const r = await readObjectLane({ name: "slow", prefix: "", lister: hung }, NOW, 20);
158 expect(r.error).toContain("no answer");
159 });
160 });
161
162 describe("bounded: never a full listing", () => {
163 test("only the last-named folder is walked into, one page per level", async () => {
164 const objects: Record<string, { bytes: number; at: number }> = {};
165 for (let d = 1; d <= 30; d++) {
166 const day = `2026-10-${String(d).padStart(2, "0")}`;
167 for (const part of ["db", "repos", "manifest.json"]) objects[`backups/${day}/${part}`] = { bytes: d, at: NOW - (31 - d) * H };
168 }
169 const { lister, calls } = fakeStore(objects);
170 const r = await readObjectLane({ name: "velero", prefix: "backups/", lister }, NOW);
171 expect(calls).toEqual(["backups/", "backups/2026-10-30/"]);
172 expect(r.newestName?.startsWith("backups/2026-10-30/")).toBe(true);
173 expect(r.error).toBeUndefined();
174 });
175
176 test("a level wider than its pages stops at the bound, and a stale find there says so", async () => {
177 const objects: Record<string, { bytes: number; at: number }> = {};
178 for (let i = 0; i < 50; i++) objects[`flat/${String(i).padStart(3, "0")}`] = { bytes: 1, at: NOW - 40 * H + i * 1000 };
179 const { lister, calls } = fakeStore(objects, 10);
180 const found = await newestUnder(lister, "flat/");
181 expect(calls).toHaveLength(PAGES);
182 expect(found.bounded).toBe(true);
183 const r = await readObjectLane({ name: "flat", prefix: "flat/", lister }, NOW);
184 expect(r.error).toContain("name a narrower prefix");
185 });
186 });
187
188 describe("configuration", () => {
189 const lane = { name: "backup/postgres", endpoint: "https://s3.test", bucket: "backups", prefix: "pg/", accessKey: ACCESS, secretKey: SECRET };
190
191 test("a whole lane reads; nothing set is no lanes", () => {
192 expect(parseLanes(undefined)).toEqual({ lanes: [], refused: [] });
193 const got = parseLanes(JSON.stringify([lane]));
194 expect(got.refused).toEqual([]);
195 expect(got.lanes).toEqual([
196 { name: "backup/postgres", prefix: "pg/", target: { endpoint: "https://s3.test", bucket: "backups", accessKeyId: ACCESS, secretAccessKey: SECRET, region: undefined } },
197 ]);
198 });
199
200 test("a lane that does not read whole is refused by name, and no value of it is said", () => {
201 const got = parseLanes(
202 JSON.stringify([
203 { ...lane, endpoint: "file:///etc/passwd" },
204 { ...lane, name: "nokey", secretKey: "" },
205 { ...lane, name: "ctl", prefix: "a\u0000b" },
206 { ...lane, name: "creds", endpoint: `https://${ACCESS}:${SECRET}@s3.test` },
207 lane,
208 lane,
209 ]),
210 );
211 expect(got.lanes.map((l) => l.name)).toEqual(["backup/postgres"]);
212 expect(got.refused.map((r) => r.name)).toEqual(["backup/postgres", "nokey", "ctl", "creds", "backup/postgres"]);
213 expect(JSON.stringify(got.refused)).not.toContain(SECRET);
214 expect(parseLanes("{not json").refused[0]!.says).toBe("configuration: not a JSON list of lanes");
215 });
216
217 test("readDeclaredLanes reads every declared lane; a refused one is a red read with its reason", async () => {
218 const { lister } = fakeStore({ "pg/a.sql": { bytes: 5, at: NOW - H } });
219 const read = await readDeclaredLanes(
220 { FORGE_BACKUP_LANES: JSON.stringify([lane, { ...lane, name: "broken", bucket: "" }]) },
221 NOW,
222 () => lister,
223 );
224 const v = verdict(read);
225 expect(v.lanes.map((l) => [l.lane, l.state])).toEqual([
226 ["backup/postgres", "green"],
227 ["broken", "red"],
228 ]);
229 expect(v.lanes[1]!.says).toContain("configuration: its bucket is missing");
230 expect(JSON.stringify(v)).not.toContain(SECRET);
231 });
232 });
233
234 describe("the bucket watch reads the declared lanes beside the forge's own", () => {
235 let dir = "";
236 let app: App;
237 let servers: Servers;
238 let admin = "";
239 beforeAll(async () => {
240 dir = await mkdtemp(join(tmpdir(), "forge-lanes-watch-"));
241 app = createApp({ dataDir: dir, port: 0 });
242 servers = await serve(app);
243 const made = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-long-lanes-passphrase", isAdmin: true });
244 if (!made.ok) throw new Error(made.error);
245 admin = createToken(app.db, made.value.id, "t", "ro").token;
246 });
247 afterAll(async () => {
248 servers.stop();
249 app.close();
250 await rm(dir, { recursive: true, force: true });
251 });
252
253 test("one pass: primary and every platform lane in the verdict home, MCP and Resilience read", async () => {
254 const { lister } = fakeStore({ "pg/a.sql": { bytes: 5, at: Date.now() - 27 * H } });
255 const w = watchBucket(
256 dir,
257 () => [{ lane: "primary", store: dirStore(join(dir, "store")) }],
258 3_600_000,
259 3_600_000,
260 () => [],
261 (now) => readDeclaredLanes({ FORGE_BACKUP_LANES: JSON.stringify([{ name: "backup/postgres", endpoint: "https://s3.test", bucket: "b", prefix: "pg/", accessKey: ACCESS, secretKey: SECRET }]) }, now, () => lister),
262 );
263 try {
264 await w.pass();
265 const v = backupVerdict(dir, drill);
266 expect(v.lanes.map((l) => [l.lane, l.state])).toEqual([
267 ["primary", "red"],
268 ["backup/postgres", "red"],
269 ]);
270 const lines = verdictLines({ app } as never);
271 // The dialog wraps at its width; the words are what count.
272 expect(lines.join(" ").replace(/\s+/g, " ")).toContain("! backup/postgres (s3.test/bucket/pg/) newest snapshot pg/a.sql is 27 h old");
273 expect(lines.join("\n")).not.toContain(SECRET);
274 } finally {
275 w.stop();
276 }
277 });
278
279 test("MCP backups returns the platform lane in verdict.lanes, with its newest object and no key", async () => {
280 const { lister } = fakeStore({ "pg/2026-11-02.sql.gz": { bytes: 2048, at: Date.now() - 2 * H } });
281 const w = watchBucket(dir, () => [], 3_600_000, 3_600_000, () => [], (now) =>
282 readDeclaredLanes({ FORGE_BACKUP_LANES: JSON.stringify([{ name: "backup/postgres", endpoint: "https://s3.test", bucket: "b", prefix: "pg/", accessKey: ACCESS, secretKey: SECRET }]) }, now, () => lister),
283 );
284 try {
285 await w.pass();
286 const res = await fetch(`${servers.url}/mcp`, {
287 method: "POST",
288 headers: { "content-type": "application/json", accept: "application/json, text/event-stream", authorization: `Bearer ${admin}` },
289 body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "backups", arguments: {} } }),
290 });
291 const text = ((await res.json()) as any).result.content[0].text as string;
292 const lane = JSON.parse(text).verdict.lanes.find((l: { lane: string }) => l.lane === "backup/postgres");
293 expect(lane).toMatchObject({ state: "green", where: "s3.test/bucket/pg/", newestName: "pg/2026-11-02.sql.gz", newestBytes: 2048 });
294 expect(lane.says).toBe("backup/postgres newest pg/2026-11-02.sql.gz 2 h old, 2.0 KiB");
295 expect(text).not.toContain(SECRET);
296 expect(text).not.toContain(ACCESS);
297 } finally {
298 w.stop();
299 }
300 });
301
302 test("a declared reader that throws costs the pass nothing: the forge's own lanes are still kept", async () => {
303 const w = watchBucket(dir, () => [{ lane: "primary", store: dirStore(join(dir, "store")) }], 3_600_000, 3_600_000, () => [], () =>
304 Promise.reject(new Error("boom")),
305 );
306 try {
307 await w.pass();
308 expect(backupVerdict(dir, drill).lanes.map((l) => l.lane)).toEqual(["primary"]);
309 } finally {
310 w.stop();
311 }
312 });
313 });
314
315 describe("the S3 lister asks for one delimited page", () => {
316 const realS3 = Bun.S3Client;
317 const asked: Record<string, unknown>[] = [];
318 let made: Record<string, unknown> | undefined;
319 let refuse = false;
320 beforeAll(() => {
321 (Bun as any).S3Client = class {
322 constructor(options: Record<string, unknown>) {
323 made = options;
324 }
325 async list(options: Record<string, unknown>) {
326 asked.push(options);
327 if (refuse) throw new Error(`AccessDenied: key ${ACCESS} refused`);
328 if (options["prefix"] === "pg/")
329 return {
330 contents: [{ key: "pg/a.sql", lastModified: new Date(NOW - 2 * H).toISOString(), size: 42 }],
331 commonPrefixes: [{ prefix: "pg/2026-11-01/" }, { prefix: "pg/2026-11-02/" }],
332 isTruncated: false,
333 };
334 if (options["prefix"] === "esc/")
335 return { contents: [{ key: "esc/\u001b[2Jdump.sql", lastModified: new Date(NOW - H).toISOString(), size: 1 }], isTruncated: false };
336 if (options["prefix"] === "pg/2026-11-02/")
337 return { contents: [{ key: "pg/2026-11-02/dump.sql", lastModified: new Date(NOW - H).toISOString(), size: 4096 }], isTruncated: false };
338 return { isTruncated: false };
339 }
340 };
341 });
342 afterAll(() => {
343 (Bun as any).S3Client = realS3;
344 });
345
346 test("newest found by walking the last folder; delimiter and 1,000 keys asked each time", async () => {
347 const r = await readObjectLane({ name: "postgres", prefix: "pg/", lister: s3Lister(target), target }, NOW);
348 expect(made).toMatchObject({ endpoint: "https://s3.test", bucket: "bucket" });
349 expect(r).toMatchObject({ newestName: "pg/2026-11-02/dump.sql", newestBytes: 4096, newestAt: new Date(NOW - H).toISOString() });
350 expect(r.error).toBeUndefined();
351 expect(asked).toEqual([
352 { prefix: "pg/", delimiter: "/", maxKeys: 1000 },
353 { prefix: "pg/2026-11-02/", delimiter: "/", maxKeys: 1000 },
354 ]);
355 });
356
357 test("a key holding an ESC is kept with the byte replaced", async () => {
358 const r = await readObjectLane({ name: "esc", prefix: "esc/", lister: s3Lister(target), target }, NOW);
359 expect(r.newestName).toBe("esc/?[2Jdump.sql");
360 });
361
362 test("a refusal is red, and the access key it echoes is struck out", async () => {
363 refuse = true;
364 const r = await readObjectLane({ name: "postgres", prefix: "pg/", lister: s3Lister(target), target }, NOW);
365 expect(r.error).toBe("AccessDenied: key … refused");
366 expect(JSON.stringify(r)).not.toContain(ACCESS);
367 });
368 });
369

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs