trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns4g rEventsFlows

A nightly off-box backup older than 26 hours is a card on /flow with the one action that retries it #10933

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/10924-a-nightly-off-box-backup-older-than-26-h → main 7 commits fork 3 threads · 3 unresolved

  • Merged. torque/10924-a-nightly-off-box-backup-older-than-26-h was merged into main as eee0685.

Review

Sign in to review this change.
26 files changed +399 −31
docs/features/backup-overdue.md added +11
@@ -0,0 +1,11 @@
1 # An overdue off-box backup (#10924)
2
3 When the newest copy of the forge that left the machine is more than 26 hours old (a nightly plus two hours), `flow/alarm_backup.ts` fires the `backup` flow alarm. It reads two small files under the data directory (the snapshot outcome and the mirror's record), never the network, and says one sentence: how old the newest off-box backup is, or that none has landed, and the last failure's cause. A forge that has never sent a copy off the box and never failed to is not alarmed.
4
5 - **On the web.** `/flow` (a forge admin) leads with one row: that sentence and **Retry the backup now** (`POST /flow/backup/retry`). Home and /boards draw it as a line like the other flow alarms; their cache is keyed on the backup's state, so a cleared backup leaves Home at once.
6 - **In the terminal.** The palette's **Valves** (a forge admin who is a person) shows the same sentence on its first line, with `b` to retry. The Flow dialog and the dashboard carry it in their alarm lines.
7 - **One retry.** Both doors call `actions/backup_retry.ts`, which takes one snapshot through `snapshotNow` (the service's own queue, so two never run at once).
8 - **Honest about where it landed.** The alarm clears only when a copy is off the machine. A snapshot that landed in a local directory is said to have "stayed on this machine, so it is not an off-box backup"; the card stays while an older off-box copy is the newest.
9 - **Callers.** `firedAlarms` takes `dataDir` as a required option (`null` for a unit over a bare db), so a door cannot silently lose the alarm.
10
11 Tests: `test/alarm-backup.test.ts`, `test/backup-retry.test.ts` (a real snapshot into a directory; the Valves screen).
src/actions/backup_retry.ts added +29
@@ -0,0 +1,29 @@
1 /**
2 * The retry the off-box backup alarm offers (#10924), for both doors: one
3 * snapshot now through `snapshotNow`, then the truth about the alarm. A
4 * snapshot that landed in a directory on this machine does not count as off-box,
5 * and the answer says so instead of reporting a success the card contradicts.
6 */
7
8 import type { App } from "../app.ts";
9 import type { User } from "../core/users.ts";
10 import { isOffBox } from "../snapshot/off_box.ts";
11 import { snapshotsOf } from "../snapshot/service.ts";
12 import { snapshotNow } from "./backups_now.ts";
13
14 export interface BackupRetry {
15 /** False when the card will still stand: the snapshot failed, or it did not leave the machine. */
16 readonly cleared: boolean;
17 /** The refusal code, when the actor may not; otherwise absent. */
18 readonly refused?: "not_found";
19 readonly says: string;
20 }
21
22 export async function retryBackup(app: App, actor: User | undefined): Promise<BackupRetry> {
23 const taken = await snapshotNow(app, actor);
24 if (!taken.ok) return { cleared: false, ...(taken.error.code === "not_found" ? { refused: "not_found" as const } : {}), says: taken.error.says };
25 const store = snapshotsOf(app)?.store;
26 if (store !== undefined && !isOffBox(store))
27 return { cleared: false, says: `${taken.value.says} It stayed on this machine, so it is not an off-box backup; the card clears when the mirror or a bucket destination holds a new one.` };
28 return { cleared: true, says: taken.value.says };
29 }
src/crew/notify.ts +1 −1
@@ -299,7 +299,7 @@ export function alarmLines(
299299 fired?: readonly FiredAlarm[],
300300 ): { id: string; line: string; items: readonly string[]; link: string }[] {
301301 if (!user.isAdmin || user.kind !== "user") return [];
302 const alarms = fired ?? [...firedAlarms(app.db, new Date(now), { ceilingUsd: forgeCeiling(app.config) }), ...reachAlarms(app)];
302 const alarms = fired ?? [...firedAlarms(app.db, new Date(now), { ceilingUsd: forgeCeiling(app.config), dataDir: app.config.dataDir }), ...reachAlarms(app)];
303303 return alarms.map((a) => {
304304 const id = `alarm:${a.key}@${a.since}`;
305305 return { id, line: a.says, items: [id], link: a.href };
src/flow/alarm_backup.ts added +56
@@ -0,0 +1,56 @@
1 /**
2 * The "off-box backup" flow alarm (#10924): the newest copy of the forge to
3 * have left the machine is more than 26 hours old, so last night's nightly
4 * did not land. Reads what Home reads (`snapshot/off_box.ts`) and the
5 * failure the outcome files kept; no network. Pure of the clock: `now` is
6 * passed in. A forge that has never sent a copy off the box and has never
7 * failed to is not alarmed: it has no off-box backup to be late.
8 */
9
10 import { lastMirror } from "../snapshot/mirror.ts";
11 import { lastOffBox } from "../snapshot/off_box.ts";
12 import { lastSnapshotOutcome } from "../snapshot/outcome.ts";
13
14 /** A nightly plus two hours of grace. */
15 export const BACKUP_ALARM_MS = 26 * 3_600_000;
16
17 export interface BackupAlarm {
18 readonly fired: boolean;
19 /** When the backup became overdue; only when fired. */
20 readonly since?: string;
21 /** The last failure's cause as one line; only when fired and one is known. */
22 readonly cause?: string;
23 readonly says: string;
24 }
25
26 /** One line, bounded: the first line of an error, 160 characters at most. */
27 function oneLine(text: string): string {
28 const line = (text.split("\n")[0] ?? "").replace(/\s+/g, " ").trim();
29 return line.length > 160 ? `${line.slice(0, 159)}…` : line;
30 }
31
32 /** The cause of the latest failure, from the snapshot attempt or the mirror pass, whichever failed last. */
33 export function backupCause(dataDir: string): string | undefined {
34 const snap = lastSnapshotOutcome(dataDir);
35 const mirror = lastMirror(dataDir);
36 const found: { at: number; text: string }[] = [];
37 if (snap?.failingSince !== undefined && snap.error) found.push({ at: Date.parse(snap.at), text: `The snapshot failed: ${snap.error}` });
38 const mirrorWhy = mirror?.sourceError ?? mirror?.error;
39 if (mirror !== undefined && (mirror.failingSince !== undefined || mirror.sourceError !== undefined) && mirrorWhy)
40 found.push({ at: Date.parse(mirror.attemptAt), text: `The mirror failed: ${mirrorWhy}` });
41 if (found.length === 0) return undefined;
42 found.sort((a, b) => (Number.isNaN(b.at) ? 0 : b.at) - (Number.isNaN(a.at) ? 0 : a.at));
43 return oneLine(found[0]!.text);
44 }
45
46 export function alarmBackup(dataDir: string, now: Date): BackupAlarm {
47 const newest = lastOffBox(dataDir);
48 const failing = lastSnapshotOutcome(dataDir)?.failingSince ?? lastMirror(dataDir)?.failingSince;
49 if (newest === null && failing === undefined) return { fired: false, says: "No off-box backup has been taken or failed." };
50 const age = newest === null ? Number.POSITIVE_INFINITY : now.getTime() - Date.parse(newest);
51 if (age <= BACKUP_ALARM_MS) return { fired: false, says: "The newest off-box backup is under 26 hours old." };
52 const since = newest === null ? failing! : new Date(Date.parse(newest) + BACKUP_ALARM_MS).toISOString();
53 const cause = backupCause(dataDir);
54 const held = newest === null ? "No off-box backup has landed" : `The newest off-box backup is ${String(Math.floor(age / 3_600_000))} h old`;
55 return { fired: true, since, ...(cause === undefined ? {} : { cause }), says: `${held}.${cause === undefined ? "" : ` ${cause}`}` };
56 }
src/flow/alarms.ts +6 −2
@@ -12,6 +12,7 @@ import { redHref, redMain } from "../checks/red_main.ts";
1212 import { getRepo } from "../core/repos.ts";
1313 import { ownRepoPath } from "../kernel/own_repo.ts";
1414 import { bugClockAlarm } from "./bugs.ts";
15 import { alarmBackup } from "./alarm_backup.ts";
1516 import { alarmFromApp } from "./alarm_from_app.ts";
1617 import { alarmIdleCrew } from "./alarm_idle_crew.ts";
1718 import { noReleaseAlarm } from "./alarm_no_release.ts";
@@ -21,7 +22,7 @@ import { alarmSilent } from "./alarm_silent.ts";
2122 import { alarmStuck } from "./alarm_stuck.ts";
2223
2324 export interface FiredAlarm {
24 readonly key: "landing_stopped" | "red_main" | "stuck" | "idle_crew" | "no_release" | "bug_clock" | "silent" | "repeat" | "from_app" | "unreachable";
25 readonly key: "landing_stopped" | "red_main" | "stuck" | "idle_crew" | "no_release" | "bug_clock" | "silent" | "repeat" | "from_app" | "unreachable" | "backup";
2526 readonly says: string;
2627 readonly since: string;
2728 /** A same-origin path to the item the alarm names. */
@@ -38,7 +39,8 @@ function redMainHref(db: Database): string {
3839 export function firedAlarms(
3940 db: Database,
4041 now: Date,
41 opts: { readonly publicOnly?: boolean; readonly ceilingUsd?: number | null } = {},
42 /** `dataDir` is required so a caller cannot forget it and lose the backup alarm; `null` says there is none to read (a unit over a bare db). */
sentinelcrew outdated

Optional dataDir lets callers drop the alarm silently (TUI flow already does). Make it required.
43 opts: { readonly dataDir: string | null; readonly publicOnly?: boolean; readonly ceilingUsd?: number | null },
4244 ): FiredAlarm[] {
4345 const out: FiredAlarm[] = [];
4446 const [owner, name] = ownRepoPath().split("/");
@@ -58,6 +60,8 @@ export function firedAlarms(
5860 if (repeat.fired) out.push({ key: "repeat", says: repeat.says, since: repeat.since!, href: `/${repeat.repo}/issues/${repeat.number}` });
5961 const fromApp = alarmFromApp(db, now);
6062 if (fromApp.fired) out.push({ key: "from_app", says: fromApp.says, since: fromApp.since!, href: `/${fromApp.repo}/issues/${fromApp.number}` });
63 const backup = opts.dataDir === null ? undefined : alarmBackup(opts.dataDir, now);
64 if (backup?.fired) out.push({ key: "backup", says: backup.says, since: backup.since!, href: "/flow" });
6165 const idle = alarmIdleCrew(db, now, { ceilingUsd: opts.ceilingUsd ?? null });
6266 if (idle.fired) out.push({ key: "idle_crew", says: idle.says, since: idle.since!, href: "/boards" });
6367 }
src/http/server.ts +2 −1
@@ -310,7 +310,7 @@ import { appNowFragment } from "../web/studio/now.ts";
310310 import { appStateFragment } from "../web/studio/previews.ts";
311311 import { changesList, tellsList } from "../web/studio/lists.ts";
312312 import { dependencySubmit } from "../web/pages/thread.ts";
313 import { flowPage, suggestionTake, valveSet } from "../web/pages/flow.ts";
313 import { backupRetry, flowPage, suggestionTake, valveSet } from "../web/pages/flow.ts";
314314 import { heldApply, heldPage, liftHoldSubmit } from "../web/pages/gate.ts";
315315 import {
316316 assigneeSubmit,
@@ -937,6 +937,7 @@ export function buildRouter(): Router<Handler> {
937937 // /flow is the week's valves (#8698); a query string is an old pipeline drill, which went to /boards.
938938 router.add("GET", "/flow", flowPage);
939939 router.add("POST", "/flow/valve", valveSet);
940 router.add("POST", "/flow/backup/retry", backupRetry);
940941 router.add("POST", "/flow/suggestion", suggestionTake);
941942 router.add("GET", "/system", systemPage);
942943 router.add("GET", "/system/stream", systemStream);
src/snapshot/off_box.ts +3 −3
@@ -11,7 +11,7 @@
1111 */
1212
1313 import { lastMirror } from "./mirror.ts";
14 import { lastSnapshotOutcome } from "./outcome.ts";
14 import { lastSnapshotOutcome, offBoxAtOf } from "./outcome.ts";
1515 import type { Store } from "./store.ts";
1616
1717 /** The host of a bucket's `where` (`host[:port]/bucket`), lower-cased, without its port or brackets. */
@@ -40,8 +40,8 @@ export function lastOffBox(dataDir: string): string | null {
4040 const mirrored = lastMirror(dataDir)?.newestAt;
4141 if (mirrored !== undefined && Number.isFinite(Date.parse(mirrored))) times.push(mirrored);
4242 const kept = lastSnapshotOutcome(dataDir);
43 if (kept?.offBox === true && kept.landedAt !== undefined && Number.isFinite(Date.parse(kept.landedAt)))
44 times.push(kept.landedAt);
43 const landed = offBoxAtOf(kept);
44 if (landed !== undefined && Number.isFinite(Date.parse(landed))) times.push(landed);
4545 if (times.length === 0) return null;
4646 return times.reduce((a, b) => (Date.parse(b) > Date.parse(a) ? b : a));
4747 }
src/snapshot/outcome.ts +20 −1
@@ -19,6 +19,11 @@ export interface SnapshotOutcome {
1919 readonly landedAt?: string;
2020 /** Whether the store it landed in is off the machine (a bucket, not a directory); absent when not known. */
2121 readonly offBox?: boolean;
22 /**
23 * When a snapshot last landed off the machine; kept across landings in a local directory,
24 * so moving the destination home does not erase the age of the last real off-box copy (#10924).
25 */
26 readonly offBoxAt?: string;
2227 /** Set while attempts are failing: since when, and the last one's reason. */
2328 readonly failingSince?: string;
2429 readonly error?: string;
@@ -37,6 +42,13 @@ export function lastSnapshotOutcome(dataDir: string): SnapshotOutcome | undefine
3742 }
3843 }
3944
45 /** When this outcome's last off-box landing was: the kept time, or the landing itself when it was off the machine. */
46 export function offBoxAtOf(s: SnapshotOutcome | undefined): string | undefined {
47 if (s === undefined) return undefined;
48 if (s.offBoxAt !== undefined) return s.offBoxAt;
49 return s.offBox === true ? s.landedAt : undefined;
50 }
51
4052 /** Keep an attempt's end: a name that landed, or the error it failed with. Returns what was kept. */
4153 export function recordSnapshotOutcome(
4254 dataDir: string,
@@ -48,10 +60,17 @@ export function recordSnapshotOutcome(
4860 ...(prev?.landed === undefined ? {} : { landed: prev.landed }),
4961 ...(prev?.landedAt === undefined ? {} : { landedAt: prev.landedAt }),
5062 ...(prev?.offBox === undefined ? {} : { offBox: prev.offBox }),
63 ...(offBoxAtOf(prev) === undefined ? {} : { offBoxAt: offBoxAtOf(prev)! }),
5164 };
5265 const next: SnapshotOutcome =
5366 "landed" in result
54 ? { at, landed: result.landed, landedAt: at, ...(result.offBox === undefined ? {} : { offBox: result.offBox }) }
67 ? {
68 at,
69 landed: result.landed,
70 landedAt: at,
71 ...(result.offBox === undefined ? {} : { offBox: result.offBox }),
72 ...(result.offBox === true ? { offBoxAt: at } : offBoxAtOf(prev) === undefined ? {} : { offBoxAt: offBoxAtOf(prev)! }),
73 }
5574 : { at, ...kept, failingSince: prev?.failingSince ?? at, error: result.error };
5675 write(dataDir, next);
5776 return next;
src/system/telemetry.ts +1 −1
@@ -117,7 +117,7 @@ function flowBody(x: {
117117 // Declined build runs of the last 24 h by reason, each with the runs behind it.
118118 declines: readDeclines(app.db, now, reader.kind === "agent"),
119119 // The four alarms (#5930) that are firing now; empty when none is.
120 alarms: firedAlarms(app.db, now, { publicOnly: reader.kind === "agent" }),
120 alarms: firedAlarms(app.db, now, { publicOnly: reader.kind === "agent", dataDir: app.config.dataDir }),
121121 // Review runs in flight per reviewer, cut to public repositories for an agent (#4733).
122122 reviewing: { says: reviewingSays(busy), reviewers: busy },
123123 conflicted: { says: conflictedSays(conflicted, now.getTime()), pulls: conflicted },
src/tui/forge.ts +1
@@ -465,6 +465,7 @@ export function forgeCommands(session: Session): (ctx: Ctx) => Command[] {
465465 forgeCeiling(session.app.config),
466466 readVerified(session.app, flowReader.kind === "owner"),
467467 keeper,
468 session.app.config.dataDir,
468469 ), keeper.ask && user ? { ask: keeper.ask, answer: () => answerKeeper(session.app, keeper.ask!.id, user.id) } : undefined));
469470 },
470471 });
src/tui/screens/dashboard.ts +1 −1
@@ -406,7 +406,7 @@ export function dashboardScreen(f: Session, ctx: Ctx): Screen {
406406 ...weekBlock(f.app, now),
407407 ...kpiLines(readKpis(f.db, now, { ...kpiForge(f.app), idle: () => idleReason(f.app, now) }), commit === null ? null : { sha: commit, since: ago(BOOTED) }),
408408 vitalsLine(readVitals(f.db, now)),
409 ...firedAlarms(f.db, now, { ceilingUsd: forgeCeiling(f.app.config) }).map((a) => a.says),
409 ...firedAlarms(f.db, now, { ceilingUsd: forgeCeiling(f.app.config), dataDir: f.app.config.dataDir }).map((a) => a.says),
410410 ...(cleared === undefined ? [] : [cleared]),
411411 ];
412412 // "Yours" is ownership plus the orgs you are on a team of; typing in the
src/tui/screens/flow.ts +3 −1
@@ -74,6 +74,8 @@ export function flowLines(
7474 verified?: readonly VerifiedRow[],
7575 /** The keeper's cause and last act (`readKeeper`, already cut to what the reader may see). */
7676 keeper?: KeeperRead,
77 /** Where the backup outcome files are kept; without it the off-box backup alarm cannot be read (the terminal passes the app's). */
78 dataDir: string | null = null,
7779 ): string[] {
7880 const stages = readPipeline(db, now);
7981 const flow = readThroughput(db, stages.map((s) => s.key), now);
@@ -100,7 +102,7 @@ export function flowLines(
100102 "",
101103 ...wrap(vitalsLine(vitalsFor(readVitals(db, now, agent !== undefined), agent === undefined)), inner),
102104 ...(agent === undefined ? clogLines(readClog(db, now), now.getTime()).flatMap((l) => wrap(l, inner)) : []),
103 ...firedAlarms(db, now, { publicOnly: agent !== undefined, ceilingUsd }).flatMap((a) => wrap(a.says, inner)),
105 ...firedAlarms(db, now, { publicOnly: agent !== undefined, ceilingUsd, dataDir }).flatMap((a) => wrap(a.says, inner)),
104106 "",
105107 ...wrap(parkedSays(fig.parked), inner),
106108 "",
src/tui/screens/valves.ts +30 −3
@@ -5,8 +5,13 @@
55 * `x` shuts it, `o` opens it, `t` takes the suggestion beside it. A turn goes
66 * through `actions/valves.ts`, so it is kept, announced as `valve.turned` and
77 * reaches every open page and screen; the screen refreshes on that event.
8 * An overdue off-box backup (#10924, `flow/alarm_backup.ts`) is the first
9 * line, as /flow draws it, and `b` retries it through `actions/backup_retry.ts`,
10 * the same action as the page's button.
811 */
912
13 import { retryBackup } from "../../actions/backup_retry.ts";
14 import { alarmBackup } from "../../flow/alarm_backup.ts";
1015 import { acceptSuggestion, turnValveFor } from "../../actions/valves.ts";
1116 import { readPipes, type Pipes, type ValveRow } from "../../flow/pipes.ts";
1217 import { levelWords } from "../../flow/valve.ts";
@@ -17,7 +22,7 @@ import { style } from "../term/style.ts";
1722 import { truncate } from "../term/width.ts";
1823 import { fill, len, split } from "../ui/layout.ts";
1924 import type { PaletteEntry } from "./entries.ts";
20 import { ListState, drawList, live, type ListRow } from "./common.ts";
25 import { ListState, drawList, live, runAsync, type ListRow } from "./common.ts";
2126 import { mayCapacity } from "./capacity.ts";
2227
2328 const STEP = 10;
@@ -39,6 +44,12 @@ export function valvesScreen(f: Session, ctx: Ctx): Screen {
3944 pipes = mayCapacity(f) && f.user !== undefined ? readPipes(f.db, f.user.id) : undefined;
4045 ctx.invalidate();
4146 };
47 let retrying = false;
48 /** The overdue off-box backup's sentence, read fresh from the outcome files; empty when none is overdue. */
49 const overdue = (): string => {
50 const a = alarmBackup(f.app.config.dataDir, new Date());
51 return a.fired ? `${a.says} b retries it now.` : "";
52 };
4253 const valves = (): readonly ValveRow[] => pipes?.valves ?? [];
4354 const here = (): ValveRow | undefined => valves()[state.index];
4455 const turn = (to: (v: ValveRow) => number): void => {
@@ -56,6 +67,7 @@ export function valvesScreen(f: Session, ctx: Ctx): Screen {
5667 binding(["x"], "shut"),
5768 binding(["o"], "open"),
5869 binding(["t"], "take the suggestion"),
70 binding(["b"], "retry an overdue off-box backup"),
5971 ],
6072 load(signal) {
6173 live(f, signal, { kinds: ["valve.turned"] }, reload);
@@ -64,7 +76,9 @@ export function valvesScreen(f: Session, ctx: Ctx): Screen {
6476 },
6577 render(area, s) {
6678 const c = ctx.theme.c;
67 const [head, list] = split(area, "v", [len(1), fill(1)]);
79 const late = overdue();
80 const [warn, head, list] = split(area, "v", [len(late === "" ? 0 : 1), len(1), fill(1)]);
81 if (late !== "") s.text(warn!.x, warn!.y, truncate(late, warn!.w), style(c.fg, c.bg), warn!.x + warn!.w);
6882 const spent = pipes === undefined ? "" : `$${pipes.usd.toFixed(2)} spent in 7 days, ${String(pipes.shipped)} shipped; `;
6983 s.text(head!.x, head!.y, truncate(`${spent}the crew does what the valves say, from the next tick`, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w);
7084 drawList(s, list!, c, state, valves().map(rowOf));
@@ -74,7 +88,20 @@ export function valvesScreen(f: Session, ctx: Ctx): Screen {
7488 else if (e.key === "l") turn((v) => Math.min(100, v.level + STEP));
7589 else if (e.key === "x") turn(() => 0);
7690 else if (e.key === "o") turn(() => 100);
77 else if (e.key === "t") {
91 else if (e.key === "b") {
92 if (retrying || overdue() === "") return true;
93 retrying = true;
94 ctx.toast("Taking a snapshot…");
95 runAsync(ctx, async () => {
96 try {
97 const r = await retryBackup(f.app, f.user);
98 ctx.toast(r.says, r.cleared ? undefined : "attn");
99 } finally {
100 retrying = false;
101 ctx.invalidate();
102 }
103 });
104 } else if (e.key === "t") {
78105 const v = here();
79106 if (v === undefined) return true;
80107 if (v.suggestion === undefined) ctx.toast("no suggestion here", "attn");
src/web/pages/boards-view.ts +1 −1
@@ -123,7 +123,7 @@ export async function boardsList(ctx: Ctx): Promise<Response> {
123123 body: html`<div class="page-head"><h1>Boards</h1>
124124 ${btn("Flow", { href: "/flow", variant: "ghost" })}
125125 ${btn("Explore the data", { href: "/boards/data", variant: "ghost" })}</div>
126 ${alarmLines(cachedAlarms(ctx.app.db, new Date(), forgeCeiling(ctx.app.config)))}
126 ${alarmLines(cachedAlarms(ctx.app.db, new Date(), forgeCeiling(ctx.app.config), ctx.app.config.dataDir))}
127127 ${body} ${make} ${drift} ${flow} ${boardsHoldsCard(ctx.app.db, ctx.auth.user, true)}`,
128128 });
129129 notePhase(ctx.request, "render", performance.now() - rendering);
src/web/pages/flow.ts +23
@@ -7,6 +7,8 @@
77 * `tui/screens/valves.ts`, on the same `readPipes` and the same actions.
88 */
99
10 import { retryBackup } from "../../actions/backup_retry.ts";
11 import { alarmBackup, type BackupAlarm } from "../../flow/alarm_backup.ts";
1012 import { acceptSuggestion, turnValveFor, type ValveError } from "../../actions/valves.ts";
1113 import type { Ctx } from "../../http/context.ts";
1214 import { readPipes, type ValveRow } from "../../flow/pipes.ts";
@@ -74,6 +76,16 @@ function valveRow(v: ValveRow) {
7476 });
7577 }
7678
79 /** The overdue off-box backup as one row: the sentence (with the last failure's cause) and the one action that retries. Nothing when none is overdue. */
80 export function backupRow(alarm: BackupAlarm) {
81 if (!alarm.fired) return html``;
82 return html`<ul class="list mb">${row({
sentinelcrew outdated

Principle 7: a card box for a single fact. Use a row. Principle 1: heading, says and 'Overdue since' repeat each other; one sentence and the button.
83 lead: icon("alert"),
84 title: html`${alarm.says}`,
85 meta: html`<form method="post" action="/flow/backup/retry">${btn("Retry the backup now", { type: "submit", variant: "primary", size: "sm" })}</form>`,
86 })}</ul>`;
87 }
88
7789 // GET /flow
7890 export function flowPage(ctx: Ctx): Response {
7991 if (ctx.url.search !== "") return new Response(null, { status: 301, headers: { location: `/boards${ctx.url.search}` } });
@@ -84,6 +96,7 @@ export function flowPage(ctx: Ctx): Response {
8496 const p = readPipes(ctx.app.db, user.id);
8597 const body = html`<div class="page-head"><h1>Flow</h1>${btn("Boards", { href: "/boards", variant: "ghost" })}</div>
8698 <p class="mb">Money flows from the left to shipped on the right, as wide as this week's real spend. Every repository has a valve: drag it to give the repository a share of the week, all the way down to shut. The crew does exactly what the valves say, from the next tick, and nothing but you turns one.</p>
99 ${backupRow(alarmBackup(ctx.app.config.dataDir, new Date()))}
87100 ${pipesSvg(p)}
88101 <p class="muted small">${money(p.usd)} spent in the last 7 days; ${plural(p.shipped, "pull")} shipped.</p>
89102 ${p.valves.length === 0
@@ -121,3 +134,13 @@ export async function suggestionTake(ctx: Ctx): Promise<Response> {
121134 if (form instanceof Response) return form;
122135 return done(ctx, acceptSuggestion(ctx.app, ctx.auth.user, field(form, "repo")));
123136 }
137
138 // POST /flow/backup/retry: one snapshot now, through the service's own queue, so two never run at once.
139 export async function backupRetry(ctx: Ctx): Promise<Response> {
140 if (!ctx.auth.user) return signInFirst(ctx);
141 const form = await readForm(ctx, "admin");
sentinelcrew outdated

Retry exists only here. Add the same action as a TUI screen entry and docs/features/<name>.md. Also test a real snapshot, and handle a local landing that leaves the card standing.
142 if (form instanceof Response) return form;
143 const retried = await retryBackup(ctx.app, ctx.auth.user);
144 if (retried.refused) return notFoundPage(ctx);
145 return flashTo(ctx, "/flow", retried.cleared ? { message: retried.says } : { tone: "danger", message: retried.says });
146 }
src/web/studio/alarm_lines.ts +7 −4
@@ -5,6 +5,7 @@
55 * checked, because the words name private repositories.
66 */
77 import type { Database } from "bun:sqlite";
8 import { alarmBackup } from "../../flow/alarm_backup.ts";
89 import { firedAlarms, type FiredAlarm } from "../../flow/alarms.ts";
910 import { writeStamp } from "../../core/write_clock.ts";
1011 import { Stamped } from "../../kernel/stamped.ts";
@@ -14,15 +15,17 @@ import { HOME_MAX_AGE_MS } from "./home_cache.ts";
1415 const caches = new WeakMap<Database, Stamped<FiredAlarm[]>>();
1516
1617 /** Valid like the vitals: while the write clock holds, 30 seconds at most. */
17 export function cachedAlarms(db: Database, now: Date, ceilingUsd: number | null): FiredAlarm[] {
18 export function cachedAlarms(db: Database, now: Date, ceilingUsd: number | null, dataDir: string): FiredAlarm[] {
1819 const stamp = writeStamp(db);
19 if (stamp === undefined) return firedAlarms(db, now, { ceilingUsd });
20 if (stamp === undefined) return firedAlarms(db, now, { ceilingUsd, dataDir });
2021 let cache = caches.get(db);
2122 if (cache === undefined) {
22 cache = new Stamped<FiredAlarm[]>({ maxEntries: 2, maxAgeMs: HOME_MAX_AGE_MS });
23 cache = new Stamped<FiredAlarm[]>({ maxEntries: 8, maxAgeMs: HOME_MAX_AGE_MS });
2324 caches.set(db, cache);
2425 }
25 return cache.get("alarms", stamp, () => firedAlarms(db, now, { ceilingUsd }));
26 // The backup alarm reads files, not the db the stamp watches: its state is part of the key, so a cleared backup leaves Home at once.
27 const backup = alarmBackup(dataDir, now);
28 return cache.get(`alarms:${backup.fired ? backup.since : "-"}:${backup.cause ?? ""}`, stamp, () => firedAlarms(db, now, { ceilingUsd, dataDir }));
2629 }
2730
2831 export function alarmLines(alarms: readonly FiredAlarm[]): Html {
src/web/studio/home.ts +1 −1
@@ -344,7 +344,7 @@ export async function studioHome(
344344 const waiting = await homeRegion(ctx.app.db, ctx.request, user.id, "waiting", homeVariant(ctx.url), access, draws.waiting);
345345 const held = timePhase(ctx.request, "held", () => heldLine(ctx, user));
346346 const alarms = user.isAdmin
347 ? alarmLines(cachedAlarms(ctx.app.db, new Date(), forgeCeiling(ctx.app.config)))
347 ? alarmLines(cachedAlarms(ctx.app.db, new Date(), forgeCeiling(ctx.app.config), ctx.app.config.dataDir))
348348 : null;
349349 const visitedAt = Date.now();
350350 const before = timePhase(ctx.request, "visit", () =>
test/alarm-backup.test.ts added +106
@@ -0,0 +1,106 @@
1 /** The off-box backup alarm (#10924): raises at 26 h, names the last failure's cause, clears when a backup lands; /flow draws the card with one retry. */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { createUser } from "../src/core/users.ts";
8 import { alarmBackup } from "../src/flow/alarm_backup.ts";
9 import { firedAlarms } from "../src/flow/alarms.ts";
10 import { serve, type Servers } from "../src/http/server.ts";
11 import { recordSnapshotOutcome } from "../src/snapshot/outcome.ts";
12
13 const NOW = new Date("2026-11-10T12:00:00.000Z");
14 const hoursAgo = (h: number) => new Date(NOW.getTime() - h * 3_600_000).toISOString();
15
16 let dir: string;
17 let app: App;
18 let servers: Servers;
19 beforeAll(async () => {
20 dir = await mkdtemp(join(tmpdir(), "forge-alarm-backup-"));
21 app = createApp({ dataDir: dir, port: 0 });
22 servers = await serve(app);
23 for (const [username, adminIfFirst] of [["alice", true], ["bob", false]] as const) {
24 const u = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst });
25 if (!u.ok) throw new Error(u.error);
26 }
27 });
28 afterAll(async () => {
29 servers.stop();
30 await rm(dir, { recursive: true, force: true });
31 });
32
33 const fresh = () => mkdtemp(join(tmpdir(), "forge-ab-"));
34
35 describe("alarmBackup", () => {
36 test("says nothing for a forge that has neither sent a copy off the box nor failed to", async () => {
37 expect(alarmBackup(await fresh(), NOW).fired).toBe(false);
38 });
39
40 test("raises past 26 h with the cause line, and not at 25 h", async () => {
41 const d = await fresh();
42 recordSnapshotOutcome(d, { landed: "s1", offBox: true }, hoursAgo(25));
43 expect(alarmBackup(d, NOW).fired).toBe(false);
44 recordSnapshotOutcome(d, { error: "S3 PUT refused: 403 AccessDenied\nstack…" }, hoursAgo(1));
45 const late = new Date(NOW.getTime() + 2 * 3_600_000);
46 const a = alarmBackup(d, late);
47 expect(a.fired).toBe(true);
48 expect(a.since).toBe(new Date(Date.parse(hoursAgo(25)) + 26 * 3_600_000).toISOString());
49 expect(a.cause).toBe("The snapshot failed: S3 PUT refused: 403 AccessDenied");
50 expect(a.says).toBe("The newest off-box backup is 27 h old. The snapshot failed: S3 PUT refused: 403 AccessDenied");
51 });
52
53 test("a failure with nothing ever off the box raises from the failure", async () => {
54 const d = await fresh();
55 recordSnapshotOutcome(d, { error: "no route to bucket" }, hoursAgo(3));
56 const a = alarmBackup(d, NOW);
57 expect(a.fired).toBe(true);
58 expect(a.since).toBe(hoursAgo(3));
59 expect(a.says).toBe("No off-box backup has landed. The snapshot failed: no route to bucket");
60 });
61
62 test("clears when a backup lands off the box", async () => {
63 const d = await fresh();
64 recordSnapshotOutcome(d, { landed: "s1", offBox: true }, hoursAgo(40));
65 recordSnapshotOutcome(d, { error: "boom" }, hoursAgo(10));
66 expect(alarmBackup(d, NOW).fired).toBe(true);
67 recordSnapshotOutcome(d, { landed: "s2", offBox: true }, hoursAgo(0.5));
68 expect(alarmBackup(d, NOW).fired).toBe(false);
69 });
70
71 test("firedAlarms carries it once given the data directory", async () => {
72 const d = await fresh();
73 recordSnapshotOutcome(d, { error: "boom" }, hoursAgo(3));
74 expect(firedAlarms(app.db, NOW, { dataDir: d }).some((a) => a.key === "backup")).toBe(true);
75 expect(firedAlarms(app.db, NOW, { dataDir: d, publicOnly: true }).some((a) => a.key === "backup")).toBe(false);
76 });
77 });
78
79 describe("/flow backup card", () => {
80 const login = async (username: string) => {
81 const res = await fetch(`${servers.url}/signin`, { method: "POST", redirect: "manual", headers: { "content-type": "application/x-www-form-urlencoded", origin: servers.url }, body: new URLSearchParams({ username, password: "a-good-passphrase" }) });
82 return (res.headers.get("set-cookie") ?? "").split(";")[0] ?? "";
83 };
84 const retry = (cookie: string, origin = servers.url) =>
85 fetch(`${servers.url}/flow/backup/retry`, { method: "POST", redirect: "manual", headers: { cookie, origin, "content-type": "application/x-www-form-urlencoded" }, body: "" });
86
87 test("draws the cause and one retry while overdue, and nothing once cleared", async () => {
88 const cookie = await login("alice");
89 recordSnapshotOutcome(dir, { error: "bucket unreachable" }, hoursAgo(3));
90 const text = await (await fetch(`${servers.url}/flow`, { headers: { cookie } })).text();
91 expect(text).toContain("No off-box backup has landed");
92 expect(text).toContain("The snapshot failed: bucket unreachable");
93 expect(text.match(/action="\/flow\/backup\/retry"/g)?.length).toBe(1);
94 recordSnapshotOutcome(dir, { landed: "s9", offBox: true }, new Date().toISOString());
95 const after = await (await fetch(`${servers.url}/flow`, { headers: { cookie } })).text();
96 expect(after).not.toContain("/flow/backup/retry");
97 });
98
99 test("retry answers an admin with a flash back to /flow, and refuses a stranger and a foreign origin", async () => {
100 const res = await retry(await login("alice"));
101 expect(res.status).toBe(303);
102 expect(res.headers.get("location")).toBe("/flow");
103 expect((await retry(await login("bob"))).status).toBe(404);
104 expect((await retry(await login("alice"), "https://evil.example")).status).toBeGreaterThanOrEqual(400);
105 });
106 });
test/alarm_from_app.test.ts +1 −1
@@ -93,7 +93,7 @@ describe("a person's note from an app", () => {
9393 expect(unassigned.fired).toBe(true);
9494 expect(unassigned.says).toContain(`alice/shop#${n}`);
9595 expect(unassigned.says).toContain("unassigned");
96 expect(firedAlarms(app.db, new Date(at + 16 * MIN)).map((a) => a.key)).toContain("from_app");
96 expect(firedAlarms(app.db, new Date(at + 16 * MIN), { dataDir: null }).map((a) => a.key)).toContain("from_app");
9797 // Assigned: quiet until the hour.
9898 app.db.query("UPDATE issue SET assignee_id = ?3 WHERE repo_id = ?1 AND number = ?2").run(repo.id, n, builder.id);
9999 expect(alarmFromApp(app.db, new Date(at + 59 * MIN)).fired).toBe(false);
test/backup-retry.test.ts added +82
@@ -0,0 +1,82 @@
1 /** #10924 round two: the retry is honest when the store is a local directory, the terminal has the same retry, and Home drops a cleared backup at once. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { retryBackup } from "../src/actions/backup_retry.ts";
8 import { createUser, type User } from "../src/core/users.ts";
9 import { alarmBackup } from "../src/flow/alarm_backup.ts";
10 import { recordSnapshotOutcome } from "../src/snapshot/outcome.ts";
11 import { startSnapshots, type SnapshotsHandle } from "../src/snapshot/service.ts";
12 import { dirStore } from "../src/snapshot/store.ts";
13 import { cachedAlarms } from "../src/web/studio/alarm_lines.ts";
14 import { createForgeApp } from "../src/tui/forge.ts";
15 import { headlessTerminal } from "../src/tui/term/terminal.ts";
16 import { quietLogs } from "./helpers/globals.ts";
17 import { settleApp, settleUntil } from "./helpers/tui.ts";
18
19 quietLogs();
20
21 let dir: string;
22 let app: App;
23 let svc: SnapshotsHandle;
24 let alice: User;
25 let bob: User;
26 const identity = (user: User) => ({ sshUsername: user.username, remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
27
28 beforeAll(async () => {
29 dir = await mkdtemp(join(tmpdir(), "forge-backup-retry-"));
30 app = createApp({ dataDir: dir, port: 0 });
31 const mk = async (username: string, adminIfFirst: boolean) => {
32 const r = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst });
33 if (!r.ok) throw new Error(r.error);
34 return r.value;
35 };
36 alice = await mk("alice", true);
37 bob = await mk("bob", false);
38 svc = startSnapshots(app, dirStore(join(dir, "store")), { version: "t", nightly: false });
39 }, 30_000);
40
41 afterAll(async () => {
42 await svc.stop();
43 app.close();
44 await rm(dir, { recursive: true, force: true });
45 });
46
47 test("a real snapshot into a local directory does not clear the alarm, and the retry says so", async () => {
48 recordSnapshotOutcome(dir, { landed: "old", offBox: true }, new Date(Date.now() - 40 * 3_600_000).toISOString());
49 recordSnapshotOutcome(dir, { error: "bucket unreachable" }, new Date(Date.now() - 3 * 3_600_000).toISOString());
50 expect(alarmBackup(dir, new Date()).fired).toBe(true);
51 const r = await retryBackup(app, alice);
52 expect(r.cleared).toBe(false);
53 expect(r.says).toContain("It stayed on this machine, so it is not an off-box backup");
54 expect(alarmBackup(dir, new Date()).fired).toBe(true);
55 });
56
57 test("a stranger is refused", async () => {
58 expect((await retryBackup(app, bob)).refused).toBe("not_found");
59 });
60
61 test("Home's cached alarms drop a backup that has cleared, inside the 30 s", () => {
62 const d = dir;
63 recordSnapshotOutcome(d, { error: "boom" }, new Date(Date.now() - 3 * 3_600_000).toISOString());
64 expect(cachedAlarms(app.db, new Date(), null, d).some((a) => a.key === "backup")).toBe(true);
65 recordSnapshotOutcome(d, { landed: "s", offBox: true }, new Date().toISOString());
66 expect(cachedAlarms(app.db, new Date(), null, d).some((a) => a.key === "backup")).toBe(false);
67 });
68
69 test("the Valves screen shows the overdue backup and b retries it", async () => {
70 recordSnapshotOutcome(dir, { landed: "old", offBox: true }, new Date(Date.now() - 40 * 3_600_000).toISOString());
71 recordSnapshotOutcome(dir, { error: "bucket unreachable" }, new Date(Date.now() - 3 * 3_600_000).toISOString());
72 const { app: tui } = createForgeApp(app, identity(alice), { terminal: headlessTerminal(160, 24), exitHooks: false });
73 tui.start();
74 tui.dispatch({ type: "key", key: "k", ctrl: true, alt: false, shift: false });
75 for (const ch of "Valves") tui.dispatch({ type: "key", key: ch, ctrl: false, alt: false, shift: false });
76 await settleApp(tui);
77 tui.dispatch({ type: "key", key: "enter", ctrl: false, alt: false, shift: false });
78 await settleUntil(tui, "b retries it now", { timeoutMs: 4_000 });
79 tui.dispatch({ type: "key", key: "b", ctrl: false, alt: false, shift: false });
80 await settleUntil(tui, "stayed on this machine", { timeoutMs: 8_000 });
81 tui.stop?.();
82 });
test/flow-alarm-lines.test.ts +1 −1
@@ -67,7 +67,7 @@ describe("alarm lines", () => {
6767 expect(flowFor(app, { kind: "owner" }).alarms.map((a) => a.key)).toEqual(["no_release"]);
6868 const none = await seed();
6969 expect(flowFor(none.app, { kind: "owner" }).alarms).toEqual([]);
70 expect(firedAlarms(none.app.db, new Date())).toEqual([]);
70 expect(firedAlarms(none.app.db, new Date(), { dataDir: null })).toEqual([]);
7171 });
7272
7373 test("the owner is told once when an alarm starts firing, not again while it stands", async () => {
test/flow-bugs.test.ts +1 −1
@@ -58,7 +58,7 @@ describe("bug clock", () => {
5858 expect(a.fired).toBe(true);
5959 expect(a.says).toContain("and 1 more");
6060 expect(countBugs(app.db, NOW)).toEqual({ open: 4, overdue: 2 });
61 const fired = firedAlarms(app.db, NOW).filter((x) => x.key === "bug_clock");
61 const fired = firedAlarms(app.db, NOW, { dataDir: null }).filter((x) => x.key === "bug_clock");
6262 expect(fired).toHaveLength(1);
6363 expect(fired[0]!.href).toMatch(/^\/alice\/r\/issues\/\d+$/);
6464 });
test/gate-bypass.test.ts +4 −4
@@ -160,15 +160,15 @@ describe("landing past a broken gate", () => {
160160 expect(landingStopped(app.db, repo.id)).toBeUndefined();
161161 gate(t + 2, "red", ["the gate > is broken", "other"]);
162162 expect(landingStopped(app.db, repo.id)?.gates).toBe(3);
163 const alarm = firedAlarms(app.db, new Date()).find((a) => a.key === "landing_stopped");
163 const alarm = firedAlarms(app.db, new Date(), { dataDir: null }).find((a) => a.key === "landing_stopped");
164164 expect(alarm?.says).toStartWith("Landing has stopped:");
165165 expect(alarm?.says.split(". ").length).toBe(1);
166166 });
167167
168168 test("the alarm is the owner's: a public-only reader never sees it", () => {
169169 stop();
170 expect(firedAlarms(app.db, new Date()).some((a) => a.key === "landing_stopped")).toBe(true);
171 expect(firedAlarms(app.db, new Date(), { publicOnly: true }).some((a) => a.key === "landing_stopped")).toBe(false);
170 expect(firedAlarms(app.db, new Date(), { dataDir: null }).some((a) => a.key === "landing_stopped")).toBe(true);
171 expect(firedAlarms(app.db, new Date(), { dataDir: null, publicOnly: true }).some((a) => a.key === "landing_stopped")).toBe(false);
172172 });
173173
174174 test("three red gates of different causes, or mixed with a green, are not a stop", () => {
@@ -191,7 +191,7 @@ describe("landing past a broken gate", () => {
191191 await green(next);
192192 expect((await mergePull(app, owner, repo, next, "squash")).ok).toBe(true);
193193 expect(landingStopped(app.db, repo.id)).toBeUndefined();
194 expect(firedAlarms(app.db, new Date()).some((a) => a.key === "landing_stopped")).toBe(false);
194 expect(firedAlarms(app.db, new Date(), { dataDir: null }).some((a) => a.key === "landing_stopped")).toBe(false);
195195 });
196196
197197 test("core: refused while landing is running, for an outsider, an agent, a short reason; then it lands and records", async () => {
test/mcp-system-stalls.test.ts +4 −2
@@ -34,8 +34,10 @@ afterAll(async () => {
3434 });
3535
3636 test("a stall under a named job is read through system, at most 10, most ms first", async () => {
37 // Let every request other tests in this process finished fall outside the 300 ms window, so only the job ran across it.
38 await Bun.sleep(320);
3739 during("test-job-10907", () => 1);
38 loopWatch.stalled(5_000, performance.now());
40 loopWatch.stalled(300, performance.now());
3941 const res = await fetch(`${servers.url}/mcp`, {
4042 method: "POST",
4143 headers: { "content-type": "application/json", accept: "application/json, text/event-stream", authorization: `Bearer ${token}` },
@@ -44,7 +46,7 @@ test("a stall under a named job is read through system, at most 10, most ms firs
4446 const body = (await res.json()) as any;
4547 const out = JSON.parse(body.result.content[0].text);
4648 expect(out.stalls.length).toBeLessThanOrEqual(10);
47 expect(out.stalls).toContainEqual({ cause: "job:test-job-10907", ms: 5000, count: 1 });
49 expect(out.stalls).toContainEqual({ cause: "job:test-job-10907", ms: 300, count: 1 });
4850 const ms = out.stalls.map((s: any) => s.ms);
4951 expect(ms).toEqual([...ms].sort((a, b) => b - a));
5052 });
test/mcp-write-scope.test.ts +2
@@ -99,6 +99,8 @@ describe("a token that may only read cannot write through any tool", () => {
9999 /** Minimal arguments that make the call a real write, not a plan or a list. */
100100 const writing = (t: ReturnType<typeof toolRegistry>[number]) => {
101101 if (t.name in ADMIN_ARGS) return ADMIN_ARGS[t.name]!;
102 // `flow` reads unless it is asked to update a drifting pull (#10959).
103 if (t.name === "flow") return { action: "update", kind: "drift", repo: "alice/game", pull: 1 };
102104 const args: Record<string, unknown> = { dryRun: false, action: "x" };
103105 if (t.readsWithout !== undefined) args[[t.readsWithout].flat()[0]!] = "x";
104106 return args;
test/run-state.test.ts +2 −2
@@ -115,7 +115,7 @@ describe("silence raises one alarm", () => {
115115 test("two silent runs are one alarm, naming the one silent longest; the shown state is on the API row", async () => {
116116 const older = runOn(held("g"), "running", 3 * SILENT_MS);
117117 runOn(held("h"), "running", 2 * SILENT_MS);
118 const silent = firedAlarms(app.db, new Date()).filter((a) => a.key === "silent");
118 const silent = firedAlarms(app.db, new Date(), { dataDir: null }).filter((a) => a.key === "silent");
119119 expect(silent).toHaveLength(1);
120120 expect(silent[0]!.href).toEndWith(`/runs/${older.id}`);
121121 const token = createToken(app.db, alice.id, "t", "ro").token;
@@ -137,7 +137,7 @@ describe("the same issue repeating runs is stopped", () => {
137137 expect(queueAgentRun(app, fresh(), issue, bot, "assigned", alice.id)).toBeUndefined();
138138 expect(listComments(app.db, issue.id).filter((c) => c.body === REPEAT_NOTE)).toHaveLength(1);
139139 expect(getIssueById(app.db, issue.id)!.assigneeId).toBeNull();
140 expect(firedAlarms(app.db, new Date()).filter((a) => a.key === "repeat")).toHaveLength(1);
140 expect(firedAlarms(app.db, new Date(), { dataDir: null }).filter((a) => a.key === "repeat")).toHaveLength(1);
141141 });
142142 test("runs spread beyond ten minutes do not trip it; a reader's word does not lift it, a writer's does", async () => {
143143 const slow = held("Slow");

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs