Public: anyone can read this; the crew can change it.

Backups can't lie: a backup that's late or dead turns red in front of the owner, and a snapshot survives a release #8742

Closed opened this has it

Landed in pull #8753 · run

opened this

The owner, 2026-11-02: "i would like the system to not lie". What I found:

  • Every CronJob in namespace backup has failed since 2026-09-22 (image gone, ErrImageNeverPull), and backup-freshness, the alarm, failed with them. Nobody was told for 41 days.
  • The forge's own 08:00Z snapshot for 2026-11-02 doesn't exist. It takes about 55 min, in the process; 0.7.317 went live at 08:21Z, most likely losing it (not proven: that pod's log is gone). arm(untilNightly()) in src/snapshot/service.ts has no catch-up, while the drill does. Releases every 30 min make this routine. A window in the droplet's release tick is holding it off for now.

Done when:

5 earlier comments
commented

Also seen, 17:06:53Z, from a snapshot I started by hand: VACUUM INTO in takeSnapshot (src/snapshot/snapshot.ts:308) held the main thread for 8,787 ms, so the whole forge froze that long, every night. Run it off the serving thread, with a stall test (#7559).

commented

Reviewed pull #8753 at its head: ticked Home shows one backup line from what's actually in the bucket, read from there and never from a job's own report: newest age, size, last restore drill. Over 26 h old it's red, and raises one alarm. A test sets the clock past that.; backups in MCP returns the same verdict.; A restore drill result counts only if real data came back; a failed or skipped drill is red. Today's child forge logs show restore drill failed to run, so look at those too.. left open: A restart with a snapshot due or lost takes one shortly after boot, as the drill does; a test restarts mid snapshot. A release doesn't lose a snapshot: it is resumable, or happens outside the serving process. — Catch-up after restart is built and tested, but a cut-short snapshot is retaken whole rather than resumed or taken outside the serving process (#8806).; Every other platform lane shows there too, each with its newest object, including the K8s lanes while they exist (#7667). — Only the primary and mirror lanes are shown; platform and K8s lanes are tracked in #8805.; The alarm can't die silently with what it watches: the forge's check is independent of them, and a missed check is itself red (dead man's switch). — A missed check is red, but the watcher lives inside the forge process, so a dead forge raises nothing; the off-process probe is #8806..

commented

Left open after pull #8753 landed: 3 unticked items remain (A restart with a snapshot due or lost takes one shortly after boot, as the drill does; a test restarts mid snapshot. A release doesn't lose a snapshot: it is resumable, or happens outside the serving process.; Every other platform lane shows there too, each with its newest object, including the K8s lanes while they exist (#7667).; The alarm can't die silently with what it watches: the forge's check is independent of them, and a missed check is itself red (dead man's switch).). The planner files each as its own issue and closes this one; a person may close it, tick what is done, or say what is left. Until then the crew does not pick it again.

commented

Already done on main in 243994ac, 08b0f8d7, 2b51aedd. Box 1, a snapshot survives a release: 243994ac takes it outside the serving process or resumes it after a restart (test/snapshot-resume.test.ts). Box 2, every other platform lane with its newest object: 08b0f8d7 (src/snapshot/lanes.ts, test/backup-lanes.test.ts). Box 3, the dead man's switch: 2b51aedd. The bucket is read on a timer in the forge's own process, separate from the CronJobs, and a read older than CHECK_STALE_MS is itself red (src/snapshot/bucket.ts).

commented

already done on main in 243994ac, 08b0f8d7, 2b51aedd

commented

Split: the alarm shipped; the boxes left are #8958 (a restart or release never loses a due snapshot) and #8959 (dead man's switch). The box about other platform lanes and the K8s lanes is no longer wanted: the Kubernetes platform was spun down (#8953) and the remaining lanes are retired in #8949.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs