Public: anyone can read this; the crew can change it.
Home says the last off-box backup is never - confirm backups run, or make them run #3717
Check · Review · Land · Live done — Landed and live in production.
Home's "Last off-box backup" tile now counts snapshots that land in a bucket, not just the second "mirror" bucket. Before this, a forge whose nightly lands in a bucket every morning read "never" if no mirror was set up.
shipped production is serving it deploy 23f74da
Home's "Last off-box backup" tile now counts snapshots that land in a bucket, not just the second "mirror" bucket. Before this, a forge whose nightly lands in a bucket every morning read "never" if no mirror was set up.
Cause. readKpis read only mirror/state.json, which is written only when a second bucket is sealed or FORGE_MIRROR_S3_* is set. The forge's main snapshot destination on the platform is the app's own S3_* bucket, which is already off the machine. Its landings, kept in snapshot/state.json, were never counted. The terminal dashboard also called readKpis without the data directory, so its figure always said "never".
Change.
src/snapshot/off_box.ts:isOffBox(store)says whether a store is off the machine. A directory never is, on any volume, and neither is a bucket on loopback (localhost,127.x,::1).lastOffBox(dataDir)returns the later of two times: the newest snapshot the mirror holds, and the newest that landed in a destination that is off the machine.src/snapshot/outcome.tsrecordsoffBoxon each landing and keeps it through failures.noteOffBoxfills it in once for an outcome written before this change.src/snapshot/service.tssets it on every snapshot and runsnoteOffBoxat the first mirror pass, about 30 s after boot, so production shows the correct age right after deploy rather than after the next 08:00 UTC nightly.src/flow/kpis.tsreadslastOffBox.kpiForgeinsrc/tui/screens/dashboard.tsnow passesdataDir.- README and
docs/SNAPSHOT.mddescribe the new source.
Not done. I couldn't read production's snapshot or drill state from the sandbox, so I haven't confirmed that the nightlies are landing. The fix is to the tile's source either way. If they are failing, the existing snapshot-failed card and the red line on Backups already show it, and the tile will keep saying overdue. The restore check is the existing weekly drill (#2667); nothing changed there.
Tests. test/snapshot-off-box.test.ts covers isOffBox on bucket, directory and loopback addresses, and the lastOffBox cases: no outcome, a directory landing, a bucket landing, a failure after it, mirror and destination each being the newer, and the fill-in for an older outcome. It also takes real snapshots through startSnapshots into a directory and into a store that looks like a bucket, and checks readKpis via kpiForge. That test, the other snapshot and home-KPI suites, and the five cross files CLAUDE.md lists all pass, and tsc --noEmit is clean.
To try it: on the preview, take a snapshot from Settings → Backups. With an S3 destination, home's tile should read "0 min ago" (or close to it); with a directory, it stays "never".
🤖 Generated with Claude Code
Does what #3701 asked: the tile now reads the newer of the mirror and a landing in a bucket, the terminal dashboard passes dataDir so both doors agree, tests fail without the change (directory vs bucket vs loopback, failure keeps offBox, newer-wins, one-time fill-in, real startSnapshots into both store kinds), README and SNAPSHOT.md updated, no new dependency. I could not reach the preview from my sandbox (host denied), so I judged from source and tests only.
Two things I would do next: (1) the fill-in for old outcomes guesses the store in effect at boot; if the destination changed since, the tile can be wrong until the next landing, so say so on Backups or drop the guess after one nightly. (2) Treat in-cluster hosts (*.svc, private ranges) as an explicit decision in isOffBox with a test, since minio.platform.svc currently counts as off-box. Also the new README sentence is one very long line; wrap it.
Home's "Last off-box backup" tile now counts snapshots that land in a bucket, not only those in the second "mirror" bucket. Before this, a forge whose nightly landed in a bucket every morning still read "never" if no mirror was set up. Because #3701 is filed under P0 Trust, this pull fixes where the tile gets its figure and leaves the alarms that already exist for failed nightlies as they are.
Cause. readKpis only read mirror/state.json, and that file is written only when a second bucket is sealed or FORGE_MIRROR_S3_* is set. On the platform, the forge's main snapshot destination is the app's own S3_* bucket, which is already off the machine. Its landings are recorded in snapshot/state.json, and the tile never read them. Separately, the terminal dashboard called readKpis without the data directory, so its figure always said "never".
Change.
src/snapshot/off_box.ts:
- isOffBox(store) says whether a store is off the machine. A directory never counts, whatever volume it is on. A bucket on loopback (localhost, 127.x, ::1) does not count either.
- lastOffBox(dataDir) returns whichever is later: the newest snapshot the mirror holds, or the newest that landed in an off-machine destination.
src/snapshot/outcome.tsrecordsoffBoxon each landing and keeps it through later failures.noteOffBoxfills it in, once, for an outcome written before this change.src/snapshot/service.tssetsoffBoxon every snapshot. It runsnoteOffBoxat the first mirror pass, about 30 s after boot, so production shows the right age soon after deploy instead of waiting for the next 08:00 UTC nightly.src/flow/kpis.tsreadslastOffBox.kpiForgeinsrc/tui/screens/dashboard.tsnow passesdataDir.- README and
docs/SNAPSHOT.mddescribe where the figure now comes from.
Not done. The sandbox cannot read production's snapshot or drill state, so this pull does not confirm that the nightlies are landing. The tile's source was wrong either way, so the fix stands on its own. If the nightlies are failing, the existing snapshot-failed card and the red line on Backups already show it, and the tile will keep saying overdue. Restores are still checked by the existing weekly drill (#2667), which this pull does not touch. The branch is behind main, but it merges with no conflicts.
Tests. test/snapshot-off-box.test.ts covers:
isOffBoxfor a bucket, a directory and loopback addresses.lastOffBoxwith no outcome, a directory landing, a bucket landing, and a failure after a landing.- Each of the mirror and the destination being the newer of the two.
- The fill-in for an outcome written before this change.
- Real snapshots taken through
startSnapshotsinto a directory and into a store that looks like a bucket, with the result checked throughreadKpisviakpiForge.
That suite passes, along with the other snapshot and home-KPI suites and the five cross files CLAUDE.md lists. tsc --noEmit is clean.
To try it: on the preview, take a snapshot from Settings → Backups. With an S3 destination, the home tile should read "0 min ago" or close to it. With a directory destination, it stays "never".
🤖 Generated with Claude Code
Closes #3701
Home's "Last off-box backup" tile now counts snapshots that land in a bucket, not just the second "mirror" bucket. Before this, a forge whose nightly lands in a bucket every morning read "never" if no mirror was set up.
Cause.
readKpisread onlymirror/state.json, which is written only when a second bucket is sealed orFORGE_MIRROR_S3_*is set. The forge's main snapshot destination on the platform is the app's ownS3_*bucket, which is already off the machine. Its landings, kept insnapshot/state.json, were never counted. The terminal dashboard also calledreadKpiswithout the data directory, so its figure always said "never".Change.
src/snapshot/off_box.ts:isOffBox(store)says whether a store is off the machine. A directory never is, on any volume, and neither is a bucket on loopback (localhost,127.x,::1).lastOffBox(dataDir)returns the later of two times: the newest snapshot the mirror holds, and the newest that landed in a destination that is off the machine.src/snapshot/outcome.tsrecordsoffBoxon each landing and keeps it through failures.noteOffBoxfills it in once for an outcome written before this change.src/snapshot/service.tssets it on every snapshot and runsnoteOffBoxat the first mirror pass, about 30 s after boot, so production shows the correct age right after deploy rather than after the next 08:00 UTC nightly.src/flow/kpis.tsreadslastOffBox.kpiForgeinsrc/tui/screens/dashboard.tsnow passesdataDir.docs/SNAPSHOT.mddescribe the new source.Not done. I couldn't read production's snapshot or drill state from the sandbox, so I haven't confirmed that the nightlies are landing. The fix is to the tile's source either way. If they are failing, the existing
snapshot-failedcard and the red line on Backups already show it, and the tile will keep saying overdue. The restore check is the existing weekly drill (#2667); nothing changed there.Tests.
test/snapshot-off-box.test.tscoversisOffBoxon bucket, directory and loopback addresses, and thelastOffBoxcases: no outcome, a directory landing, a bucket landing, a failure after it, mirror and destination each being the newer, and the fill-in for an older outcome. It also takes real snapshots throughstartSnapshotsinto a directory and into a store that looks like a bucket, and checksreadKpisviakpiForge. That test, the other snapshot and home-KPI suites, and the five cross files CLAUDE.md lists all pass, andtsc --noEmitis clean.To try it: on the preview, take a snapshot from Settings → Backups. With an S3 destination, home's tile should read "0 min ago" (or close to it); with a directory, it stays "never".
🤖 Generated with Claude Code
Closes #3701