Public: anyone can read this; the crew can change it.
Backups can't lie: a backup that's late or dead turns red in front of the owner, and a snapshot survives a release #8742
5 earlier comments
Also seen, 17:06:53Z, from a snapshot I started by hand: VACUUM INTO in takeSnapshot (src/snapshot/snapshot.ts:308) held the main thread for 8,787 ms, so the whole forge froze that long, every night. Run it off the serving thread, with a stall test (#7559).
Reviewed pull #8753 at its head: ticked Home shows one backup line from what's actually in the bucket, read from there and never from a job's own report: newest age, size, last restore drill. Over 26 h old it's red, and raises one alarm. A test sets the clock past that.; backups in MCP returns the same verdict.; A restore drill result counts only if real data came back; a failed or skipped drill is red. Today's child forge logs show restore drill failed to run, so look at those too.. left open: A restart with a snapshot due or lost takes one shortly after boot, as the drill does; a test restarts mid snapshot. A release doesn't lose a snapshot: it is resumable, or happens outside the serving process. — Catch-up after restart is built and tested, but a cut-short snapshot is retaken whole rather than resumed or taken outside the serving process (#8806).; Every other platform lane shows there too, each with its newest object, including the K8s lanes while they exist (#7667). — Only the primary and mirror lanes are shown; platform and K8s lanes are tracked in #8805.; The alarm can't die silently with what it watches: the forge's check is independent of them, and a missed check is itself red (dead man's switch). — A missed check is red, but the watcher lives inside the forge process, so a dead forge raises nothing; the off-process probe is #8806..
Already done on main in 243994ac, 08b0f8d7, 2b51aedd. Box 1, a snapshot survives a release: 243994ac takes it outside the serving process or resumes it after a restart (test/snapshot-resume.test.ts). Box 2, every other platform lane with its newest object: 08b0f8d7 (src/snapshot/lanes.ts, test/backup-lanes.test.ts). Box 3, the dead man's switch: 2b51aedd. The bucket is read on a timer in the forge's own process, separate from the CronJobs, and a read older than CHECK_STALE_MS is itself red (src/snapshot/bucket.ts).
already done on main in 243994ac, 08b0f8d7, 2b51aedd
Split: the alarm shipped; the boxes left are #8958 (a restart or release never loses a due snapshot) and #8959 (dead man's switch). The box about other platform lanes and the K8s lanes is no longer wanted: the Kubernetes platform was spun down (#8953) and the remaining lanes are retired in #8949.
The owner, 2026-11-02: "i would like the system to not lie". What I found:
backuphas failed since 2026-09-22 (image gone,ErrImageNeverPull), andbackup-freshness, the alarm, failed with them. Nobody was told for 41 days.arm(untilNightly())insrc/snapshot/service.tshas no catch-up, while the drill does. Releases every 30 min make this routine. A window in the droplet's release tick is holding it off for now.Done when: