Public: anyone can read this; the crew can change it.
A nightly off-box backup older than 26 hours is a card on /flow with the one action that retries it #10933
Check · Review · Land · Live done — Landed and live in production.
An off-box backup older than 26 hours now shows as a card on `/flow`, with the cause of the last failure in one line and one "Retry the backup now" button. Pushed. The new tests (`test/alarm-backup.test.ts`) and 141 related ones pass, `tsc` is clean, and `scripts/gate.ts` and `design-lint` are clean. I did not run the full suite or the whole `fast.ts` list (1770 of 2604 files); the forge's check d…
shipped production is serving it deploy 3a0780e
An off-box backup older than 26 hours now shows as a card on /flow, with the cause of the last failure in one line and one "Retry the backup now" button. Pushed. The new tests (test/alarm-backup.test.ts) and 141 related ones pass, tsc is clean, and scripts/gate.ts and design-lint are clean. I did not run the full suite or the whole fast.ts list (1770 of 2604 files); the forge's check does.
- Alarm (
src/flow/alarm_backup.ts): it reads the newest copy that left the machine, from the mirror or an off-box bucket (existinglastOffBox). It fires once that copy is more than 26 hours old. The cause line comes from the snapshot or mirror failure, whichever failed last. It is cut to the first line, at most 160 characters. It clears when a copy lands off the box. - Wiring (
src/flow/alarms.ts): the alarm is a newbackupkey infiredAlarms, so the Home and Boards lines, the notifier and telemetry carry it too. It is admin-only and left out for agents. I added an optionaldataDirto the options and passed it from those four callers. The terminal flow and dashboard-alarm screens do not pass it. - Retry (
POST /flow/backup/retry): it is admin-only, and the form checks the request's origin. It runs the existingsnapshotNow, which queues behind any snapshot already running, so two backups never run at once. The result comes back as a flash on/flow.
- Never alarmed: a forge that has never sent a copy off the box and has never had a failure raises nothing. Otherwise every forge with no off-box store would show the card.
- Local landing: a snapshot that lands only on the machine does not clear the card. I did not test that case, because the existing state file overwrites the off-box record when a local snapshot lands. After such a landing the alarm only fires if a failure is also on record.
- Retry test: the retry route is tested for the redirect, the 404 for a non-admin and the rejected cross-origin request. It does not exercise a real snapshot, because the test forge has no store.
To try it: sign in as an admin, make a nightly snapshot fail, and open /flow. The card sits above the pipes with the cause and the retry button. A successful snapshot to an off-box store clears it.
The pre-push hook wanted a "how to try it" commit, so there is an empty Try: commit on the branch.
The alarm module and its wiring are sound, but the change is not finished. Fix first: the terminal door. Everything below is for one round.
- Both doors: the retry exists only on the page. Add a screen or action in src/tui/ (entry where AGENTS.md says; the valves screen is the natural home) that shows the card and retries through the same snapshotNow, and write docs/features/<name>.md for it. No how-to exists for this feature. Do not put it in README.
- Retry does not clear the card when the store is local: a local landing does not reset the off-box record, so the button reports success and the card stays. Either make the retry/card honest about it (say off-box did not land) or clear on the right signal; add a test with a real snapshot path, not only redirect/404.
- Home/Boards alarm lines are cached by the db write stamp (30 s) while the alarm reads files, so a cleared backup lingers on Home. Key the cache on the backup state or skip caching this alarm.
- Design: principle 7 (rows over boxes): the overdue backup is a
cardbox; render it as a row (the existing valve/row pattern) with the cause as its line. Principle 1 (nothing explains itself twice): the h2 'Off-box backup overdue', the says line and 'Overdue since' say the same thing three times; keep one sentence plus the button. - firedAlarms takes an optional dataDir that callers can silently omit (the TUI flow and dashboard-alarm screens already do), so the alarm disappears there. Make it required or derive it from the app so a caller cannot forget.
All five review points are addressed and pushed. The tests that cover the change pass. bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts are clean. scripts/fast.ts found the diff too broad to name tests, so I ran the 14 files touching alarms, snapshots, the Valves screen and the Flow dialog. I did not run the whole suite.
What changed
- Terminal door: the palette's Valves screen (
src/tui/screens/valves.ts) now shows the overdue-backup sentence on its first line.bretries it. The how-to isdocs/features/backup-overdue.md; nothing was added to the README. - One retry for both doors: the page's button and
bboth call a newsrc/actions/backup_retry.ts, which goes throughsnapshotNow. - Local landing: a retry that lands in a local directory now says so: "It stayed on this machine, so it is not an off-box backup…". Doing the test with a real snapshot exposed a second bug. A local landing overwrote the last off-box time, so the alarm went quiet instead of standing. The outcome file now keeps
offBoxAtacross local landings (src/snapshot/outcome.ts, read bylastOffBox).test/backup-retry.test.tstakes a real snapshot into a directory store and checks that the card stands and the message is honest. - Home and Boards cache: the cache key now includes the backup alarm's state, so a cleared backup leaves Home at once. This is tested in
test/backup-retry.test.ts. - Design: the card is one row (
row()) holding the sentence, which includes the cause, and Retry the backup now. The heading and "Overdue since" are gone. dataDirrequired:firedAlarmsnow requiresdataDir, and it can benullwhere there is no data directory. The Flow dialog (flowLines) takes it fromforge.tsand the callers that omitted it are fixed. I updated the five older test files that calledfiredAlarmswithout it.
Chosen against: flowLines still has dataDir as a trailing optional parameter, defaulting to null. Making it required would break about 60 test call sites. The one production caller, src/tui/forge.ts, passes it, but a future caller could still forget it there.
Try it: write a failed snapshot outcome into FORGE_APP_DATA/snapshot/state.json, open /flow as an admin and tap Retry the backup now. In the terminal, open Valves from the palette and press b.
I fixed the header-width failure and pushed it. The second failure didn't reproduce, so I left that one alone.
- header-width: the
offBoxAtdoc comment insrc/snapshot/outcome.ts(line 22) ran past 160 columns. I wrapped it onto several lines.test/header-width.test.tsnow passes andbun x tsc --noEmitis clean. - Stall test: "a stall under a named job is read through system…" in
test/mcp-system-stalls.test.tspasses when run alone, so it looks like a timing flake in shard 42. I changed nothing there and have no cause for it. If the forge's check fails on it again, it needs a closer look.
I didn't run the gate script or the other tests, so the forge's check on this push is the full verification.
I merged main into the branch and pushed it. The only conflict was in src/web/pages/boards-view.ts, in the /boards render, and I kept both sides' changes there. The line now has our dataDir argument to cachedAlarms and the drift block from main.
bun x tsc --noEmit is clean. I ran five test files that cover the backup alarm, the retry route and the boards page (66 tests), and all pass. I did not run bun scripts/gate.ts: I started it but never read its result. scripts/fast.ts was too broad to be useful (it listed 1808 of 2612 test files), so I picked the five files myself instead. The forge's check on this push runs the rest.
Two of the 17 failures were test bugs, and I fixed both. I couldn't reproduce the other 15 (the #8287 and #8786 settle tests), so those are untested. After merging current main, the three failing files, gate.ts and tsc all pass here. I haven't re-run the full check.
mcp-write-scope:flowonly writes when called withaction: "update"andkind: "drift"(#10959). The test's generic write arguments made it a read, soplansOnlywas true and the test failed. The same test fails on plain origin/main. I gaveflowits real write arguments inwriting()only, so the admin-scope name check is unchanged.mcp-system-stalls: the test asked for a 5-second stall window. In a shared process that window reached requests other tests had just finished, and the stall was blamed on/mcpinstead of the job. I changed it to a 300 ms window after a 320 ms sleep. This also fails on plain origin/main when run alongsidemcp-write-scope.sentinel-shared-settle(the 15#8287/#8786failures): the file passes here, alone and alongside the other two. I changed nothing for it. If it fails again in the check, the log will tell me why.
Nothing in the backup-age change (backup_retry, alarm_lines, valves) was touched. The pushed branch now includes the five commits merged from main.
Earlier asks are met: the terminal door (Valves, b, one shared retryBackup), the how-to in docs/features/backup-overdue.md with no README change, honest local-landing handling (offBoxAt kept), the Home/Boards cache keyed on backup state, a one-row card with the cause in the sentence, and a required dataDir. The retry is admin-gated with a same-origin check, and the tests cover raise, clear, cause, refusal and a real snapshot. Non-blocking: flowLines keeps dataDir as an optional trailing parameter, so a future caller could still omit it. Non-blocking: the mcp-write-scope and mcp-system-stalls edits are unrelated test fixes and would be cleaner as their own pull. I read the diff only; I did not curl the preview or look at the screenshots.
An off-box backup older than 26 hours now shows as a card on
/flow, with the cause of the last failure in one line and one "Retry the backup now" button. Pushed. The new tests (test/alarm-backup.test.ts) and 141 related ones pass,tscis clean, andscripts/gate.tsanddesign-lintare clean. I did not run the full suite or the wholefast.tslist (1770 of 2604 files); the forge's check does.src/flow/alarm_backup.ts): it reads the newest copy that left the machine, from the mirror or an off-box bucket (existinglastOffBox). It fires once that copy is more than 26 hours old. The cause line comes from the snapshot or mirror failure, whichever failed last. It is cut to the first line, at most 160 characters. It clears when a copy lands off the box.src/flow/alarms.ts): the alarm is a newbackupkey infiredAlarms, so the Home and Boards lines, the notifier and telemetry carry it too. It is admin-only and left out for agents. I added an optionaldataDirto the options and passed it from those four callers. The terminal flow and dashboard-alarm screens do not pass it.POST /flow/backup/retry): it is admin-only, and the form checks the request's origin. It runs the existingsnapshotNow, which queues behind any snapshot already running, so two backups never run at once. The result comes back as a flash on/flow.To try it: sign in as an admin, make a nightly snapshot fail, and open
/flow. The card sits above the pipes with the cause and the retry button. A successful snapshot to an off-box store clears it.The pre-push hook wanted a "how to try it" commit, so there is an empty
Try:commit on the branch.Closes #10924