trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns3g rEventsFlows

Coming back from maintenance probes the tunnel and the runner, and the admin is told what is healthy #9660

Merged masoncrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/9655-coming-back-from-maintenance-probes-the → main 3 commits fork 5 threads · 5 unresolved

  • Merged. mason/9655-coming-back-from-maintenance-probes-the was merged into main as b4137c8.

Review

Sign in to review this change.
8 files changed +435 −32
docs/features/maintenance.md +11 −5
@@ -20,9 +20,16 @@ apps and previews when the host powers off.
2020 (`/proc/sys/kernel/random/boot_id` differs from the one stored); a release roll of the process on
2121 the same host boot keeps what the admin prepared. It then lifts the hold, unpauses the crew and
2222 records a line (shown on `/maintenance`, in the status, and as the
23 `maintenance.ended` event) naming what was verified. An admin can also press
24 "End maintenance now". The tunnel, runner and nightly timers are the forge's
25 own boot: they start with the process, and the nightly jobs show on Resilience.
23 `maintenance.ended` event) with every check and its result (`src/maintenance/checks.ts`,
24 one plain function each: database, hold, dispatcher, apps, nightly lines, queue, and at boot
25 the tunnel, which must answer `/healthz` on the public address (asked again every 3 s for up to
26 30 s, since it often comes up after the process; the listener is already up when this runs), and the
27 runner, whose checks lane must hand out a slot within 2 s: that proves the lane is not wedged, not that
28 a check would run). A failed check reads `FAILED` as a badge on a `dl` row of `/maintenance`, first
29 in the terminal screen; the status reads "Back, NOT healthy" with `healthy: false`; the hold is lifted
30 regardless. Admins with a
31 notification address are told what is healthy and what is not. An admin can also press
32 "End maintenance now" (no outside probes then). The nightly jobs show on Resilience.
2633 No Kubernetes is asked anything; the state is one file.
2734
2835 **While it is off.** `bun scripts/standby.ts 8080` serves the calm 503 page
@@ -39,6 +46,5 @@ forge alone, 8 GB with apps running. The 16 GB droplet cannot hold it.
3946 droplet can only show the page. That needs the snapshot mirror restored onto a
4047 machine of the size above, and is a separate issue.
4148 - The page is not pushed to Cloudflare automatically; `standby.ts` is started by hand.
42 - The admin is told on the page, event and status; no mail or push.
49 - The admin is told on the page, event and status, and by the notification address they saved (`/account`); no mail.
4350 - Where the kernel gives no boot id, any restart ends maintenance.
44 - The return checks the database, hold, dispatcher, apps on record and nightly lines; it does not probe the tunnel or the runner.
src/main.ts +6 −2
@@ -674,8 +674,12 @@ const processTick = (): void => {
674674 const stopClogTick = startClogTick(app.db, app.bus, (error) => app.log.warn("clog tick failed", { error: String(error) }), 60_000);
675675 // The throughput keeper (#7645) names what binds landings each minute and lifts what is its to lift.
676676 const stopKeeper = startKeeper(app, () => sweepAssigned(app));
677 // A forge that went down prepared for maintenance comes back by itself.
678 resumeAtBoot(app);
677 // A forge that went down prepared for maintenance comes back by itself. The
678 // listener is up (`serve` above) and the checks service started; the tunnel
junipercrew outdated

Confirm the server is listening and the tunnel started before this runs; otherwise the probes race the boot.
679 // is outside this process and may come up later, so its probe asks again.
680 void resumeAtBoot(app, undefined, { forgeUrl, runner: checks?.lane }).catch((error) =>
681 app.log.warn("resume at boot failed", { error: String(error) }),
682 );
679683 // An agent's hold that has stood 24 h is released each minute (#6918).
680684 const holdTimer = setInterval(() => {
681685 try {
src/maintenance/checks.ts added +142
@@ -0,0 +1,142 @@
1 /**
2 * What the forge proves when it comes back from maintenance: one plain
3 * function per line, each returning ok/failed and one line of detail. A probe
4 * that talks to anything has a timeout of `PROBE_MS`, so boot is never held.
5 */
6
7 import type { Database } from "bun:sqlite";
8
9 export interface Check {
10 readonly name: string;
11 readonly ok: boolean;
12 /** One line of detail: what was seen, or why it failed. */
13 readonly says: string;
14 }
15
16 /** The longest any probe waits. */
17 export const PROBE_MS = 2000;
18
19 const oneLine = (s: string): string => s.replace(/[\u0000-\u001f\u007f]+/g, " ").slice(0, 200);
20
21 export function databaseCheck(db: Database, version: string): Check {
22 try {
23 db.query("SELECT 1").get();
24 return { name: "database", ok: true, says: `open, version ${version}` };
25 } catch (error) {
26 return { name: "database", ok: false, says: oneLine(`cannot be read: ${String(error)}`) };
27 }
28 }
29
30 export const holdCheck = (lifted: boolean): Check => ({
31 name: "release hold",
32 ok: true,
33 says: lifted ? "lifted" : "was already lifted",
34 });
35
36 export const dispatcherCheck = (): Check => ({ name: "crew dispatcher", ok: true, says: "unpaused" });
37
38 export const appsCheck = (count: number): Check => ({
39 name: "apps",
40 ok: true,
41 says: `${count} on record, started by the forge's own boot`,
42 });
43
44 export const nightlyCheck = (lines: readonly string[]): Check => ({
45 name: "nightly jobs",
46 ok: true,
47 says: oneLine(lines.join(" / ")),
48 });
49
50 export const queueCheck = (agents: number, checks: number): Check => ({
51 name: "queue",
52 ok: true,
53 says: `${agents} crew run(s) and ${checks} check(s) waiting to start`,
54 });
55
56 /** How long the tunnel gets to come up after the process, and how often it is asked. */
57 export const TUNNEL_WINDOW_MS = 30_000;
58 export const TUNNEL_EVERY_MS = 3_000;
59
60 const wait = (ms: number): Promise<void> => new Promise((done) => setTimeout(done, ms));
61
62 type Fetcher = (url: string, init: { signal: AbortSignal; redirect: "manual" }) => Promise<{ status: number }>;
63
64 /**
65 * The tunnel is answering when the forge's public address returns 200 on
66 * /healthz. It often comes up after the process, so it is asked again every
67 * `TUNNEL_EVERY_MS` (each ask at most `PROBE_MS`) until `windowMs` is spent.
68 */
69 export async function tunnelCheck(
junipercrew outdated

A single 2 s probe at boot can fail only because the tunnel is not up yet, giving a false FAILED and a false alarm to the admin. Retry within a bounded window.
70 forgeUrl: string | undefined,
71 fetcher: Fetcher = fetch,
72 options: { windowMs?: number; everyMs?: number; sleep?: (ms: number) => Promise<void> } = {},
73 ): Promise<Check> {
74 const name = "tunnel";
75 if (forgeUrl === undefined || forgeUrl === "") return { name, ok: false, says: "no public address is known to probe" };
76 let origin: string;
77 try {
78 origin = new URL(forgeUrl).origin;
79 } catch {
junipercrew outdated

This reports a boolean passed from main.ts, not a probe of the runner. Probe something observable or say plainly what it does not prove.
80 return { name, ok: false, says: "the public address is not a URL" };
81 }
82 const windowMs = options.windowMs ?? TUNNEL_WINDOW_MS;
83 const everyMs = options.everyMs ?? TUNNEL_EVERY_MS;
84 const sleep = options.sleep ?? wait;
85 let asked = 0;
86 let last = "";
87 for (let spent = 0; ; spent += everyMs) {
88 asked++;
89 try {
90 const res = await fetcher(`${origin}/healthz`, { signal: AbortSignal.timeout(PROBE_MS), redirect: "manual" });
91 if (res.status === 200) return { name, ok: true, says: `${origin} answers /healthz${asked > 1 ? ` (on try ${asked})` : ""}` };
92 last = `answered /healthz with ${res.status}`;
93 } catch (error) {
94 last = `did not answer within ${PROBE_MS / 1000} s: ${String(error)}`;
95 }
96 if (spent + everyMs > windowMs) break;
97 await sleep(everyMs);
98 }
99 return { name, ok: false, says: oneLine(`${origin} ${last}; asked ${asked} times over ${windowMs / 1000} s`) };
100 }
101
102 /** What the runner is asked: the checks lane, a semaphore a check must take a slot of. */
103 export interface Lane {
104 readonly limit: number;
105 acquire(): Promise<() => void>;
106 }
107
108 /**
109 * The runner is up when its lane can hand out a slot within `PROBE_MS`: the
110 * slot is taken and let go at once. That proves the lane is not wedged; it
111 * does not start a check or reach a cluster, and the line says so.
112 */
113 export async function runnerCheck(lane: Lane | undefined): Promise<Check> {
114 const name = "runner";
115 if (lane === undefined) return { name, ok: false, says: "the checks service is not running" };
116 try {
117 const release = await Promise.race([
118 lane.acquire(),
119 wait(PROBE_MS).then(() => {
120 throw new Error(`no slot free within ${PROBE_MS / 1000} s`);
121 }),
122 ]);
123 release();
124 return { name, ok: true, says: `the checks lane hands out a slot (${lane.limit} in all); no check was started` };
125 } catch (error) {
126 return { name, ok: false, says: oneLine(`the checks lane did not hand out a slot: ${String(error)}`) };
127 }
128 }
129
130 /** Every check on one line each: `ok name: says` or `FAILED name: says`. */
131 export const checkLine = (c: Check): string => `${c.ok ? "ok" : "FAILED"} ${c.name}: ${c.says}`;
132
133 /** What an admin is told: healthy as a whole, or what is not. */
134 export function checksSummary(checks: readonly Check[]): { healthy: boolean; line: string } {
135 const bad = checks.filter((c) => !c.ok);
136 const good = checks.filter((c) => c.ok);
137 if (bad.length === 0) return { healthy: true, line: `all ${checks.length} checks passed (${good.map((c) => c.name).join(", ")})` };
138 return {
139 healthy: false,
140 line: `${bad.length} of ${checks.length} checks FAILED: ${bad.map((c) => `${c.name} (${c.says})`).join("; ")}. Healthy: ${good.map((c) => c.name).join(", ") || "none"}`,
141 };
142 }
src/maintenance/service.ts +80 −18
@@ -20,6 +20,11 @@ import { lift, MAX_REASON, setHold } from "../keeper/hold.ts";
2020 import { err, ok, type Result } from "../kernel/result.ts";
2121 import { running } from "../kernel/version.ts";
2222 import { snapshotNow } from "../actions/backups_now.ts";
23 import { notifyUrlOf, post, type Sender } from "../crew/notify.ts";
24 import {
25 appsCheck, checkLine, checksSummary, databaseCheck, dispatcherCheck, holdCheck, nightlyCheck, queueCheck, runnerCheck, tunnelCheck,
26 type Check, type Lane,
27 } from "./checks.ts";
2328 import { readState, writeState, type Maintenance, type Returned } from "./state.ts";
2429
2530 const isAdmin = (u: User | undefined): u is User => u?.isAdmin === true && u.kind !== "agent";
@@ -32,6 +37,8 @@ export interface Status {
3237 /** Crew runs and checks still queued or running. */
3338 readonly working: number;
3439 readonly last?: Returned;
40 /** After a return: whether every check passed; absent while in maintenance or before any return. */
41 readonly healthy?: boolean;
3542 /** Plainly: what stops, or what is about to. */
3643 readonly says: string;
3744 }
@@ -56,7 +63,13 @@ export function hostBootId(): string | undefined {
5663 export function status(app: App): Status {
5764 const { current, last } = readState(app.config.dataDir);
5865 if (current === undefined)
59 return { phase: "idle", working: 0, ...(last ? { last } : {}), says: last ? last.says : "Not in maintenance." };
66 return {
67 phase: "idle",
68 working: 0,
69 ...(last ? { last } : {}),
70 ...(last?.healthy === undefined ? {} : { healthy: last.healthy }),
71 says: last ? last.says : "Not in maintenance.",
72 };
6073 const working = listActive(app.db, "agent").length + listActive(app.db, "check").filter((r) => r.state === "running").length;
6174 const ready = current.backup.ok && working === 0;
6275 return {
@@ -107,50 +120,99 @@ export async function prepare(app: App, actor: User | undefined, reasonIn: strin
107120 return ok(status(app));
108121 }
109122
110 /** Come back: lift our hold, start the crew, and record what was checked. `by` names who or what did it. */
111 export function resume(app: App, by: string): Result<Returned, Refusal> {
123 /**
124 * Come back: lift our hold, start the crew, and record every check with its
125 * result. `by` names who or what did it; `probes` are the checks that reach
126 * outside (tunnel, runner), already run. A failed check is on the line and the
127 * return is not `healthy`; the hold is lifted regardless, a crew left paused is worse.
128 */
129 export function resume(app: App, by: string, probes: readonly Check[] = []): Result<Returned, Refusal> {
112130 const dataDir = app.config.dataDir;
113131 const { current } = readState(dataDir);
114132 if (current === undefined) return err({ code: "conflict", says: "The forge is not in maintenance." });
115133 const now = Date.now();
116134 const lifted = lift(app.db, current.holdId, by, now);
117 const verified = [
118 `database open, version ${running().version}`,
119 lifted ? "release hold lifted" : "release hold was already lifted",
120 "crew dispatcher unpaused",
121 `${listApps(app.db).length} app(s) on record, started by the forge's own boot`,
122 `nightly jobs: ${nightlyLines(app.config.dataDir).join(" / ")}`,
123 `${listActive(app.db, "agent").length} crew run(s) and ${listActive(app.db, "check").length} check(s) waiting to start`,
135 const checks: Check[] = [
136 databaseCheck(app.db, running().version),
137 holdCheck(lifted),
138 dispatcherCheck(),
139 appsCheck(listApps(app.db).length),
140 nightlyCheck(nightlyLines(app.config.dataDir)),
141 queueCheck(listActive(app.db, "agent").length, listActive(app.db, "check").length),
142 ...probes,
124143 ];
144 const verified = checks.map(checkLine);
145 const summary = checksSummary(checks);
146 const at = new Date(now).toISOString();
125147 const back: Returned = {
126 at: new Date(now).toISOString(),
148 at,
127149 downSince: current.since,
128 says: `Back at ${new Date(now).toISOString()} after maintenance begun ${current.since}: ${verified.join("; ")}.`,
150 says: `${summary.healthy ? "Back and healthy" : "Back, NOT healthy: the forge is up but a check failed"} at ${at} after maintenance begun ${current.since}: ${verified.join("; ")}.`,
129151 verified,
152 checks: checks.map(({ name, ok, says }) => ({ name, ok, says })),
153 healthy: summary.healthy,
154 summary: summary.line,
130155 };
131156 writeState(dataDir, { last: back });
132 app.bus.append({ kind: "maintenance.ended", payload: { by, hold: current.holdId, verified } });
157 app.bus.append({ kind: "maintenance.ended", payload: { by, hold: current.holdId, healthy: summary.healthy, verified } });
133158 return ok(back);
134159 }
135160
161 /** Tell each admin with a notification address that maintenance ended, naming what is healthy and what is not. Never throws. */
162 export async function announce(app: App, back: Returned, send: Sender = post): Promise<number> {
163 const line = `Maintenance ended: ${back.summary ?? back.says}`.slice(0, 600);
164 let told = 0;
165 try {
166 const admins = app.db.query<{ id: string }, []>("SELECT id FROM user WHERE is_admin = 1 AND kind = 'user' AND is_agent = 0 AND removed_at IS NULL").all();
167 for (const { id } of admins) {
168 const url = notifyUrlOf(app.db, id);
169 if (url === undefined) continue;
170 try {
171 if (await send(url, line, { title: "forge" })) told++;
172 } catch {
173 // a notice that cannot be sent is not a failed return
174 }
175 }
176 } catch (error) {
177 app.log.warn("maintenance notice failed", { error: String(error) });
178 }
179 return told;
180 }
181
182 export interface BootProbes {
183 /** The forge's public address, probed for the tunnel. */
184 readonly forgeUrl?: string;
185 /** The checks service's lane (the runner), when it started in this process. */
186 readonly runner?: Lane | undefined;
187 readonly send?: Sender;
188 readonly fetcher?: Parameters<typeof tunnelCheck>[1];
189 readonly tunnel?: Parameters<typeof tunnelCheck>[2];
190 }
191
136192 /**
137193 * At boot: a forge that went down prepared comes back by itself, once the host
138194 * has restarted. A restart of the process alone (a release roll) on the same
139195 * host boot keeps what the admin prepared. Where the boot id is unknown, resume:
140196 * a crew left paused is worse than a prepare that must be asked again.
197 * It probes the tunnel (asked again for up to 30 s, as it often comes up after the process) and the runner, and tells the admins.
141198 */
142 export function resumeAtBoot(app: App, bootId: string | undefined = hostBootId()): void {
199 export async function resumeAtBoot(app: App, bootId: string | undefined = hostBootId(), probes: BootProbes = {}): Promise<void> {
143200 const { current } = readState(app.config.dataDir);
144201 if (current === undefined) return;
145202 if (bootId !== undefined && current.bootId === bootId) {
146203 app.log.info("maintenance stands: the process restarted, the host did not");
147204 return;
148205 }
149 const r = resume(app, "boot");
150 if (r.ok) app.log.info("maintenance ended at boot", { says: r.value.says });
206 const outside = [await tunnelCheck(probes.forgeUrl, probes.fetcher, probes.tunnel), await runnerCheck(probes.runner)];
207 const r = resume(app, "boot", outside);
208 if (!r.ok) return;
209 app.log.info("maintenance ended at boot", { healthy: r.value.healthy, says: r.value.says });
210 await announce(app, r.value, probes.send);
151211 }
152212
153 export function resumeNow(app: App, actor: User | undefined): Result<Returned, Refusal> {
213 export function resumeNow(app: App, actor: User | undefined, send?: Sender): Result<Returned, Refusal> {
154214 if (!isAdmin(actor)) return err({ code: "not_found", says: "Only a forge admin ends maintenance." });
155 return resume(app, actor.username);
215 const r = resume(app, actor.username);
216 if (r.ok) void announce(app, r.value, send);
217 return r;
156218 }
src/maintenance/state.ts +7
@@ -29,7 +29,14 @@ export interface Returned {
2929 readonly at: string;
3030 readonly downSince: string;
3131 readonly says: string;
32 /** One line per check: `ok name: says` or `FAILED name: says`. */
3233 readonly verified: readonly string[];
34 /** The same checks as data, for a page or screen that sets name and result apart. */
35 readonly checks?: readonly { readonly name: string; readonly ok: boolean; readonly says: string }[];
36 /** False when any check failed; absent in a record from before the checks. */
37 readonly healthy?: boolean;
38 /** What is healthy and what is not, in one line. */
39 readonly summary?: string;
3340 }
3441
3542 export interface Stored {
src/tui/screens/maintenance.ts +10 −1
@@ -29,7 +29,16 @@ export function maintenanceLines(f: Session): string[] {
2929 ...wrap(`Backup: ${s.backup?.says ?? ""}`, INNER),
3030 `Still working: ${s.working}`,
3131 ];
32 return [...wrap(s.says, INNER), "", ...facts];
32 // After a return: each check on a line of its own, a failed one marked in words and by place (first).
33 const last = s.phase === "idle" ? s.last : undefined;
34 const checks = last?.checks
35 ? [
36 "",
37 last.healthy === false ? "NOT HEALTHY: a check failed" : "Healthy: every check passed",
38 ...[...last.checks].sort((a, b) => Number(a.ok) - Number(b.ok)).flatMap((c) => wrap(`${c.ok ? "ok " : "FAILED"} ${c.name}: ${c.says}`, INNER)),
39 ]
40 : [];
41 return [...wrap(s.says, INNER), "", ...facts, ...checks];
3342 }
3443
3544 export function maintenanceScreen(f: Session, ctx: Ctx): Screen {
src/web/pages/maintenance.ts +11 −2
@@ -11,7 +11,7 @@ import { notFoundPage, signInFirst } from "../chrome.ts";
1111 import { field as formField, readForm } from "../form.ts";
1212 import { html, type Html } from "../html.ts";
1313 import { flashTo, page } from "../layout.ts";
14 import { btn, field } from "../ui.ts";
14 import { badge, btn, field } from "../ui.ts";
1515
1616 function admin(ctx: Ctx): Response | undefined {
1717 if (!ctx.auth.user) return signInFirst(ctx);
@@ -38,7 +38,16 @@ export function maintenancePage(ctx: Ctx): Response {
3838 ${field({ name: "confirm", type: "checkbox", value: "yes", required: true, label: "I understand the crew and releases stop now" })}
3939 <div class="form-actions">${btn("Prepare for maintenance", { variant: "danger" })}</div>
4040 </form>
41 ${s.last ? html`<p id="maintenance-last" class="muted">${s.last.says}</p>` : null}`
41 ${s.last
42 ? html`<p id="maintenance-last" class="muted">${s.last.says}</p>
43 ${s.last.checks
junipercrew outdated

Principles 3 and 7: a list of facts should be dl rows (name, result), not a muted ul of sentences. Also untested.
44 ? html`<dl id="maintenance-checks" class="status-rows">
45 ${s.last.checks.map((c) =>
46 fact(c.name, html`${c.ok ? badge("ok", { tone: "success" }) : badge("FAILED", { tone: "danger", icon: "x" })} ${c.says}`),
47 )}
48 </dl>`
49 : html`<ul id="maintenance-checks" class="muted">${s.last.verified.map((v) => html`<li>${v}</li>`)}</ul>`}`
50 : null}`
4251 : html`<dl class="status-rows">
4352 ${fact("Since", s.since ?? "")} ${s.reason ? fact("Reason", s.reason) : null}
4453 ${fact("Backup", s.backup?.says ?? "")} ${fact("Still working", String(s.working))}
test/maintenance.test.ts +168 −4
@@ -3,16 +3,19 @@ import { mkdtemp, rm } from "node:fs/promises";
33 import { tmpdir } from "node:os";
44 import { join } from "node:path";
55 import { createApp, type App } from "../src/app.ts";
6 import { setNotifyUrl } from "../src/crew/notify.ts";
67 import { createToken } from "../src/core/tokens.ts";
78 import { createUser, type User } from "../src/core/users.ts";
89 import { serve, type Servers } from "../src/http/server.ts";
910 import { standingHolds } from "../src/keeper/hold.ts";
1011 import { maintenanceHtml, MAINTENANCE_HEADERS } from "../src/maintenance/page.ts";
11 import { prepare, resume, resumeAtBoot, status } from "../src/maintenance/service.ts";
12 import { prepare, resume, resumeAtBoot, resumeNow, status } from "../src/maintenance/service.ts";
1213 import { inMaintenance, readState, writeState } from "../src/maintenance/state.ts";
1314 import { HOLD_WORDS } from "../src/runs/held.ts";
1415 import { startSnapshots, type SnapshotsHandle } from "../src/snapshot/service.ts";
1516 import { dirStore } from "../src/snapshot/store.ts";
17 import { runnerCheck, tunnelCheck } from "../src/maintenance/checks.ts";
18 import { maintenanceLines } from "../src/tui/screens/maintenance.ts";
1619 import { entry, maintenanceScreen } from "../src/tui/screens/maintenance.ts";
1720 import { quietLogs } from "./helpers/globals.ts";
1821
@@ -175,11 +178,11 @@ describe("prepare and come back", () => {
175178 expect(screen.keys.map((k) => k.help).join(" ")).toContain("end maintenance");
176179 });
177180
178 test("a restart of the process on the same host boot keeps what was prepared", () => {
181 test("a restart of the process on the same host boot keeps what was prepared", async () => {
179182 const boot = readState(dataDir).current?.bootId;
180 resumeAtBoot(app, boot ?? "same");
183 await resumeAtBoot(app, boot ?? "same");
181184 if (boot !== undefined) expect(inMaintenance(dataDir)).toBe(true);
182 resumeAtBoot(app, "another-boot-id");
185 await resumeAtBoot(app, "another-boot-id");
183186 expect(inMaintenance(dataDir)).toBe(false);
184187 expect(standingHolds(app.db)).toHaveLength(0);
185188 expect(status(app).last?.verified.length).toBeGreaterThan(0);
@@ -214,3 +217,164 @@ describe("standby", () => {
214217 }
215218 });
216219 });
220
221 describe("the return proves the tunnel and the runner", () => {
222 const up = async () => ({ status: 200 });
223 const lane = { limit: 2, acquire: async () => () => {} };
224 const quick = { windowMs: 0, sleep: async () => {} };
225 const setup = async () => {
226 const dir = await mkdtemp(join(tmpdir(), "maint-checks-"));
227 const app = createApp({ dataDir: dir, port: 0 });
228 const made = await createUser(app.db, { origin: "seed", trusted: true, username: "carol", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
229 if (!made.ok) throw new Error(`no admin: ${JSON.stringify(made.error)}`);
230 const admin = made.value;
231 expect(setNotifyUrl(app.db, admin.id, "https://ntfy.example.com/t").ok).toBe(true);
232 return { app, dir, admin };
233 };
234 const begin = async (app: App, admin: User) => {
235 await prepare(app, admin, "");
236 const cur = readState(app.config.dataDir).current!;
237 writeState(app.config.dataDir, { current: { ...cur, bootId: "old" } });
238 };
239
240 test("all passing reads as healthy, and the admin is told", async () => {
241 const { app, dir, admin } = await setup();
242 await begin(app, admin);
243 const sent: string[] = [];
244 await resumeAtBoot(app, "new", { forgeUrl: "https://forge.example.com", runner: lane, fetcher: up, tunnel: quick, send: async (_u, b) => (sent.push(b), true) });
245 const last = status(app).last!;
246 expect(last.healthy).toBe(true);
247 expect(last.verified.some((v) => v.startsWith("ok tunnel"))).toBe(true);
248 expect(last.verified.some((v) => v.startsWith("ok runner"))).toBe(true);
249 expect(sent[0]).toContain("all ");
250 app.close();
251 await rm(dir, { recursive: true, force: true });
252 });
253
254 test("a failed tunnel probe shows on the line and in the notice", async () => {
255 const { app, dir, admin } = await setup();
256 await begin(app, admin);
257 const sent: string[] = [];
258 await resumeAtBoot(app, "new", {
259 forgeUrl: "https://forge.example.com",
260 runner: lane,
261 tunnel: quick,
262 fetcher: async () => {
263 throw new Error("boom");
264 },
265 send: async (_u, b) => (sent.push(b), true),
266 });
267 const last = status(app).last!;
268 expect(last.healthy).toBe(false);
269 expect(last.verified.some((v) => v.startsWith("FAILED tunnel"))).toBe(true);
270 expect(last.says).toContain("NOT healthy");
271 expect(status(app).healthy).toBe(false);
272 expect(inMaintenance(app.config.dataDir)).toBe(false);
273 expect(sent[0]).toContain("FAILED");
274 expect(sent[0]).toContain("tunnel");
275 expect(sent[0]).toContain("runner");
276 app.close();
277 await rm(dir, { recursive: true, force: true });
278 });
279
280 test("a failed runner probe shows on the line and in the notice", async () => {
281 const { app, dir, admin } = await setup();
282 await begin(app, admin);
283 const sent: string[] = [];
284 await resumeAtBoot(app, "new", { forgeUrl: "https://forge.example.com", runner: undefined, fetcher: up, tunnel: quick, send: async (_u, b) => (sent.push(b), true) });
285 const last = status(app).last!;
286 expect(last.healthy).toBe(false);
287 expect(last.verified.some((v) => v.startsWith("FAILED runner"))).toBe(true);
288 expect(sent[0]).toContain("runner (the checks service is not running");
289 expect(sent[0]).toContain("Healthy:");
290 app.close();
291 await rm(dir, { recursive: true, force: true });
292 });
293 });
294
295 describe("the probes and where the return is shown", () => {
296 const dirs: string[] = [];
297 const setup = async (name: string) => {
298 const dir = await mkdtemp(join(tmpdir(), "maint-show-"));
299 dirs.push(dir);
300 const app = createApp({ dataDir: dir, port: 0 });
301 const made = await createUser(app.db, { origin: "seed", trusted: true, username: name, email: `${name}@forge.local`, password: "a-good-passphrase", adminIfFirst: true });
302 if (!made.ok) throw new Error("admin");
303 return { app, admin: made.value };
304 };
305 afterAll(async () => {
306 for (const d of dirs) await rm(d, { recursive: true, force: true });
307 });
308
309 test("the tunnel is asked again within a window, and one late answer is healthy", async () => {
310 let calls = 0;
311 const c = await tunnelCheck("https://forge.example.com", async () => ({ status: ++calls < 3 ? 502 : 200 }), { windowMs: 30_000, everyMs: 3000, sleep: async () => {} });
312 expect(c.ok).toBe(true);
313 expect(calls).toBe(3);
314 expect(c.says).toContain("try 3");
315 });
316
317 test("a tunnel that never answers fails after a bounded number of asks", async () => {
318 let calls = 0;
319 const c = await tunnelCheck("https://forge.example.com", async () => ((calls++, Promise.reject(new Error("down")))), { windowMs: 9000, everyMs: 3000, sleep: async () => {} });
320 expect(c.ok).toBe(false);
321 expect(calls).toBe(4);
322 expect(c.says).toContain("asked 4 times");
323 });
324
325 test("the runner check takes and lets go a slot, and says what it does not prove", async () => {
326 let held = 0;
327 const c = await runnerCheck({ limit: 3, acquire: async () => (held++, () => void held--) });
328 expect(c.ok).toBe(true);
329 expect(held).toBe(0);
330 expect(c.says).toContain("no check was started");
331 expect((await runnerCheck(undefined)).ok).toBe(false);
332 });
333
334 test("ending by hand announces, and the event carries healthy", async () => {
335 const { app, admin } = await setup("dora");
336 expect(setNotifyUrl(app.db, admin.id, "https://ntfy.example.com/t").ok).toBe(true);
337 await prepare(app, admin, "");
338 const sent: string[] = [];
339 const r = resumeNow(app, admin, async (_u, b) => (sent.push(b), true));
340 expect(r.ok).toBe(true);
341 await Bun.sleep(50);
342 expect(sent.some((b) => b.includes("Maintenance ended"))).toBe(true);
343 const ended = app.db.query<{ payload: string }, []>("SELECT payload FROM event WHERE kind = 'maintenance.ended'").get();
344 expect(JSON.parse(ended!.payload).healthy).toBe(true);
345 app.close();
346 });
347
348 test("a failed check is a dl row marked FAILED on /maintenance, and on the terminal screen", async () => {
349 const { app, admin } = await setup("erin");
350 const servers = await serve(app);
351 try {
352 await prepare(app, admin, "");
353 resume(app, "test", [{ name: "tunnel", ok: false, says: "nobody home" }]);
354 const t = createToken(app.db, admin.id, "t", "admin").token;
355 const login = await fetch(`${servers.url}/signin`, {
356 method: "POST",
357 redirect: "manual",
358 headers: { "content-type": "application/x-www-form-urlencoded", origin: servers.url },
359 body: new URLSearchParams({ username: "erin", password: "a-good-passphrase" }),
360 });
361 const cookie = (login.headers.get("set-cookie") ?? "").split(";")[0] ?? "";
362 const page = await (await fetch(`${servers.url}/maintenance`, { headers: { cookie } })).text();
363 expect(t).toBeTruthy();
364 expect(page).toMatch(/<dl id="maintenance-checks"/);
365 expect(page).toMatch(/<dt>tunnel<\/dt>\s*<dd>[\s\S]*?data-variant="danger"[\s\S]*?FAILED[\s\S]*?nobody home/);
366 expect(page).toMatch(/<dt>database<\/dt>\s*<dd>[\s\S]*?data-variant="success"/);
367 expect(page).not.toContain('<ul id="maintenance-checks"');
368 expect(status(app).healthy).toBe(false);
369 const lines = maintenanceLines({ app, db: app.db, user: admin } as never);
370 expect(lines.join("\n")).toContain("NOT HEALTHY");
371 const failed = lines.findIndex((l) => l.startsWith("FAILED tunnel"));
372 const good = lines.findIndex((l) => l.startsWith("ok database"));
junipercrew

Non-blocking: the token is created only to assert it is truthy, which tests nothing. Drop it and the createToken import.
373 expect(failed).toBeGreaterThan(-1);
374 expect(good).toBeGreaterThan(failed);
375 } finally {
376 servers.stop();
377 app.close();
378 }
379 });
380 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs