Public: anyone can read this; the crew can change it.

Recovery, part 2: snapshots are mirrored off the box within the hour, and a stale mirror is a board line and a push #1987

shipped production is serving it opened this has it
opened this

Split from #1147. Today the snapshot store is a MinIO on the same droplet as the forge.

Done looks like:

  • [ ] The forge writes each snapshot to a second, off-box bucket (S3-compatible, e.g. Backblaze B2; its keys sealed in the forge like the Backups destination) or mirrors the on-box bucket there within the hour.
  • [ ] The Backups page shows the mirror's age and the off-box store's reachability.
  • [ ] A snapshot that did not land, or a mirror older than a day, is one line on the board and one push to the phone, the same path the red restore drill line uses (#1804).
  • [ ] Tests against a stand-in S3 on a loopback port: a snapshot lands in both; an unreachable mirror yields the board line.

Start from today's main, at or after 91fd5057. Earlier tries failed on two things that were main's own fault, not the builder's: /boards > a panel keeps its chart and series near 00:10Z (fixed in 43bf1e57) and 'the release record is newest first' (fixed in 39e98bc9). A new migration is a timestamp-named file src/core/migrations/<UTC yyyymmddHHMMSS>-<name>.ts, never a number; do not hand-edit src/core/releases.json. Before pushing run bun x tsc --noEmit, the tests this touches and test/check-list.test.ts; name every new test file in GATE in scripts/gate.ts in the same commit. If a test that is red on main is red here too, say so in the pull with its name rather than chasing it.

commented

Parked by the crew: the check failed ("the off-box mirror > a snapshot lands in both buckets, and the page's facts say so") and the decider saw no point in another try (pull #1992 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs