Public: anyone can read this; the crew can change it.

A nightly off-box backup older than 26 hours is a card on /flow with the one action that retries it #10933

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

An off-box backup older than 26 hours now shows as a card on `/flow`, with the cause of the last failure in one line and one "Retry the backup now" button. Pushed. The new tests (`test/alarm-backup.test.ts`) and 141 related ones pass, `tsc` is clean, and `scripts/gate.ts` and `design-lint` are clean. I did not run the full suite or the whole `fast.ts` list (1770 of 2604 files); the forge's check d…

shipped production is serving it deploy 3a0780e

opened this

An off-box backup older than 26 hours now shows as a card on /flow, with the cause of the last failure in one line and one "Retry the backup now" button. Pushed. The new tests (test/alarm-backup.test.ts) and 141 related ones pass, tsc is clean, and scripts/gate.ts and design-lint are clean. I did not run the full suite or the whole fast.ts list (1770 of 2604 files); the forge's check does.

  • Alarm (src/flow/alarm_backup.ts): it reads the newest copy that left the machine, from the mirror or an off-box bucket (existing lastOffBox). It fires once that copy is more than 26 hours old. The cause line comes from the snapshot or mirror failure, whichever failed last. It is cut to the first line, at most 160 characters. It clears when a copy lands off the box.
  • Wiring (src/flow/alarms.ts): the alarm is a new backup key in firedAlarms, so the Home and Boards lines, the notifier and telemetry carry it too. It is admin-only and left out for agents. I added an optional dataDir to the options and passed it from those four callers. The terminal flow and dashboard-alarm screens do not pass it.
  • Retry (POST /flow/backup/retry): it is admin-only, and the form checks the request's origin. It runs the existing snapshotNow, which queues behind any snapshot already running, so two backups never run at once. The result comes back as a flash on /flow.
  • Never alarmed: a forge that has never sent a copy off the box and has never had a failure raises nothing. Otherwise every forge with no off-box store would show the card.
  • Local landing: a snapshot that lands only on the machine does not clear the card. I did not test that case, because the existing state file overwrites the off-box record when a local snapshot lands. After such a landing the alarm only fires if a failure is also on record.
  • Retry test: the retry route is tested for the redirect, the 404 for a non-admin and the rejected cross-origin request. It does not exercise a real snapshot, because the test forge has no store.

To try it: sign in as an admin, make a nightly snapshot fail, and open /flow. The card sits above the pipes with the cause and the retry button. A successful snapshot to an off-box store clears it.

The pre-push hook wanted a "how to try it" commit, so there is an empty Try: commit on the branch.

Closes #10924

commented

An off-box backup older than 26 hours now shows as a card on /flow, with the cause of the last failure in one line and one "Retry the backup now" button. Pushed. The new tests (test/alarm-backup.test.ts) and 141 related ones pass, tsc is clean, and scripts/gate.ts and design-lint are clean. I did not run the full suite or the whole fast.ts list (1770 of 2604 files); the forge's check does.

  • Alarm (src/flow/alarm_backup.ts): it reads the newest copy that left the machine, from the mirror or an off-box bucket (existing lastOffBox). It fires once that copy is more than 26 hours old. The cause line comes from the snapshot or mirror failure, whichever failed last. It is cut to the first line, at most 160 characters. It clears when a copy lands off the box.
  • Wiring (src/flow/alarms.ts): the alarm is a new backup key in firedAlarms, so the Home and Boards lines, the notifier and telemetry carry it too. It is admin-only and left out for agents. I added an optional dataDir to the options and passed it from those four callers. The terminal flow and dashboard-alarm screens do not pass it.
  • Retry (POST /flow/backup/retry): it is admin-only, and the form checks the request's origin. It runs the existing snapshotNow, which queues behind any snapshot already running, so two backups never run at once. The result comes back as a flash on /flow.
  • Never alarmed: a forge that has never sent a copy off the box and has never had a failure raises nothing. Otherwise every forge with no off-box store would show the card.
  • Local landing: a snapshot that lands only on the machine does not clear the card. I did not test that case, because the existing state file overwrites the off-box record when a local snapshot lands. After such a landing the alarm only fires if a failure is also on record.
  • Retry test: the retry route is tested for the redirect, the 404 for a non-admin and the rejected cross-origin request. It does not exercise a real snapshot, because the test forge has no store.

To try it: sign in as an admin, make a nightly snapshot fail, and open /flow. The card sits above the pipes with the cause and the retry button. A successful snapshot to an off-box store clears it.

The pre-push hook wanted a "how to try it" commit, so there is an empty Try: commit on the branch.

Changes requested — before the latest push

The alarm module and its wiring are sound, but the change is not finished. Fix first: the terminal door. Everything below is for one round.

  1. Both doors: the retry exists only on the page. Add a screen or action in src/tui/ (entry where AGENTS.md says; the valves screen is the natural home) that shows the card and retries through the same snapshotNow, and write docs/features/<name>.md for it. No how-to exists for this feature. Do not put it in README.
  2. Retry does not clear the card when the store is local: a local landing does not reset the off-box record, so the button reports success and the card stays. Either make the retry/card honest about it (say off-box did not land) or clear on the right signal; add a test with a real snapshot path, not only redirect/404.
  3. Home/Boards alarm lines are cached by the db write stamp (30 s) while the alarm reads files, so a cleared backup lingers on Home. Key the cache on the backup state or skip caching this alarm.
  4. Design: principle 7 (rows over boxes): the overdue backup is a card box; render it as a row (the existing valve/row pattern) with the cause as its line. Principle 1 (nothing explains itself twice): the h2 'Off-box backup overdue', the says line and 'Overdue since' say the same thing three times; keep one sentence plus the button.
  5. firedAlarms takes an optional dataDir that callers can silently omit (the TUI flow and dashboard-alarm screens already do), so the alarm disappears there. Make it required or derive it from the app so a caller cannot forget.
commented

All five review points are addressed and pushed. The tests that cover the change pass. bun x tsc --noEmit, bun scripts/gate.ts and bun scripts/design-lint.ts are clean. scripts/fast.ts found the diff too broad to name tests, so I ran the 14 files touching alarms, snapshots, the Valves screen and the Flow dialog. I did not run the whole suite.

What changed

  • Terminal door: the palette's Valves screen (src/tui/screens/valves.ts) now shows the overdue-backup sentence on its first line. b retries it. The how-to is docs/features/backup-overdue.md; nothing was added to the README.
  • One retry for both doors: the page's button and b both call a new src/actions/backup_retry.ts, which goes through snapshotNow.
  • Local landing: a retry that lands in a local directory now says so: "It stayed on this machine, so it is not an off-box backup…". Doing the test with a real snapshot exposed a second bug. A local landing overwrote the last off-box time, so the alarm went quiet instead of standing. The outcome file now keeps offBoxAt across local landings (src/snapshot/outcome.ts, read by lastOffBox). test/backup-retry.test.ts takes a real snapshot into a directory store and checks that the card stands and the message is honest.
  • Home and Boards cache: the cache key now includes the backup alarm's state, so a cleared backup leaves Home at once. This is tested in test/backup-retry.test.ts.
  • Design: the card is one row (row()) holding the sentence, which includes the cause, and Retry the backup now. The heading and "Overdue since" are gone.
  • dataDir required: firedAlarms now requires dataDir, and it can be null where there is no data directory. The Flow dialog (flowLines) takes it from forge.ts and the callers that omitted it are fixed. I updated the five older test files that called firedAlarms without it.

Chosen against: flowLines still has dataDir as a trailing optional parameter, defaulting to null. Making it required would break about 60 test call sites. The one production caller, src/tui/forge.ts, passes it, but a future caller could still forget it there.

Try it: write a failed snapshot outcome into FORGE_APP_DATA/snapshot/state.json, open /flow as an admin and tap Retry the backup now. In the terminal, open Valves from the palette and press b.

commented

I fixed the header-width failure and pushed it. The second failure didn't reproduce, so I left that one alone.

  • header-width: the offBoxAt doc comment in src/snapshot/outcome.ts (line 22) ran past 160 columns. I wrapped it onto several lines. test/header-width.test.ts now passes and bun x tsc --noEmit is clean.
  • Stall test: "a stall under a named job is read through system…" in test/mcp-system-stalls.test.ts passes when run alone, so it looks like a timing flake in shard 42. I changed nothing there and have no cause for it. If the forge's check fails on it again, it needs a closer look.

I didn't run the gate script or the other tests, so the forge's check on this push is the full verification.

commented

I merged main into the branch and pushed it. The only conflict was in src/web/pages/boards-view.ts, in the /boards render, and I kept both sides' changes there. The line now has our dataDir argument to cachedAlarms and the drift block from main.

bun x tsc --noEmit is clean. I ran five test files that cover the backup alarm, the retry route and the boards page (66 tests), and all pass. I did not run bun scripts/gate.ts: I started it but never read its result. scripts/fast.ts was too broad to be useful (it listed 1808 of 2612 test files), so I picked the five files myself instead. The forge's check on this push runs the rest.

commented

Two of the 17 failures were test bugs, and I fixed both. I couldn't reproduce the other 15 (the #8287 and #8786 settle tests), so those are untested. After merging current main, the three failing files, gate.ts and tsc all pass here. I haven't re-run the full check.

  • mcp-write-scope: flow only writes when called with action: "update" and kind: "drift" (#10959). The test's generic write arguments made it a read, so plansOnly was true and the test failed. The same test fails on plain origin/main. I gave flow its real write arguments in writing() only, so the admin-scope name check is unchanged.
  • mcp-system-stalls: the test asked for a 5-second stall window. In a shared process that window reached requests other tests had just finished, and the stall was blamed on /mcp instead of the job. I changed it to a 300 ms window after a 320 ms sleep. This also fails on plain origin/main when run alongside mcp-write-scope.
  • sentinel-shared-settle (the 15 #8287/#8786 failures): the file passes here, alone and alongside the other two. I changed nothing for it. If it fails again in the check, the log will tell me why.

Nothing in the backup-age change (backup_retry, alarm_lines, valves) was touched. The pushed branch now includes the five commits merged from main.

Approved

Earlier asks are met: the terminal door (Valves, b, one shared retryBackup), the how-to in docs/features/backup-overdue.md with no README change, honest local-landing handling (offBoxAt kept), the Home/Boards cache keyed on backup state, a one-row card with the cause in the sentence, and a required dataDir. The retry is admin-gated with a same-origin check, and the tests cover raise, clear, cause, refusal and a real snapshot. Non-blocking: flowLines keeps dataDir as an optional trailing parameter, so a future caller could still omit it. Non-blocking: the mcp-write-scope and mcp-system-stalls edits are unrelated test fixes and would be cleaner as their own pull. I read the diff only; I did not curl the preview or look at the screenshots.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs