Backups can't lie: a backup that's late or dead turns red in front of the owner, and a snapshot survives a release #8753

Most of #8742 is done, but I did not build resumable snapshots, the K8s lanes (#7667), or an off-process dead man's switch. bun scripts/gate.ts, bun x tsc --noEmit, bun scripts/design-lint.ts, test/backup-truth.test.ts and 15 related suites pass here. The 15 suites include test/mcp-size.test.ts. I did not run the full suite, so the forge's check will.

What changed

• Catch-up: a boot whose newest snapshot finished before the last 08:00Z slot takes one two minutes later (snapshotDue in src/snapshot/due.ts). A snapshot cut short by a release has no manifest, so it counts as lost. A test restarts on a store holding only half a snapshot and sees one land; another restarts with today's snapshot present and sees none taken.
• Bucket truth: src/snapshot/bucket.ts lists the primary and mirror stores every 5 minutes inside the forge process. A CronJob dying cannot take it down, because no job reports on the bucket. Home's backup tile, vitals and the MCP backups tool (new verdict field) all use one function, backupVerdict. It shows newest age, size, each lane's newest snapshot and the last drill.
• Red conditions:

- the newest snapshot is over 26 h old (a test sets the clock past it); - a lane is unreadable or holds nothing; - the last drill failed, did not run, or restored no rows; - the check itself has not run for 20 minutes (the dead man's switch, tested).

• One alarm: vitals raises a single backup card.
• Drills: a drill that throws is now stored as a red drill. Before, it only logged restore drill failed to run and left no record. This changes test/mcp-restore-drill.test.ts: it expected a "could not run" answer from the MCP, and I rewrote it to expect a red drill, because the old expectation was the lie.
• Stall: VACUUM INTO now runs in a worker (src/snapshot/vacuum.ts), for the snapshot copy and for app databases. The test fails if the copy runs inline. I did not measure the 8.8 s freeze itself.

Not done

• Resumable snapshots: a snapshot interrupted by a release is retaken whole after boot rather than resumed.
• K8s lanes (#7667): nothing in this repo knows them. A lane is one more entry in the list passed to watchBucket.
• Dead man's switch: it lives inside the forge, so it catches a stopped check but not a dead forge. That needs an external probe.
• Drill logs: I could not see the child forge's restore drill failed to run logs. The failure is now recorded as a red drill with its reason, which is where to look next.
• No drill yet: this shows as "no restore drill yet" and is not red; the existing eight-day drill card still covers it.

I added docs/features/backup-truth.md rather than a README section. To try it, open home as an admin and look at the backup tile, or call the backups tool.

Closes #8742

Most of #8742 is done, but I did not build resumable snapshots, the K8s lanes (#7667), or an off-process dead man's switch. bun scripts/gate.ts, bun x tsc --noEmit, bun scripts/design-lint.ts, test/backup-truth.test.ts and 15 related suites pass here. The 15 suites include test/mcp-size.test.ts. I did not run the full suite, so the forge's check will.

What changed

• Catch-up: a boot whose newest snapshot finished before the last 08:00Z slot takes one two minutes later (snapshotDue in src/snapshot/due.ts). A snapshot cut short by a release has no manifest, so it counts as lost. A test restarts on a store holding only half a snapshot and sees one land; another restarts with today's snapshot present and sees none taken.
• Bucket truth: src/snapshot/bucket.ts lists the primary and mirror stores every 5 minutes inside the forge process. A CronJob dying cannot take it down, because no job reports on the bucket. Home's backup tile, vitals and the MCP backups tool (new verdict field) all use one function, backupVerdict. It shows newest age, size, each lane's newest snapshot and the last drill.
• Red conditions:

- the newest snapshot is over 26 h old (a test sets the clock past it); - a lane is unreadable or holds nothing; - the last drill failed, did not run, or restored no rows; - the check itself has not run for 20 minutes (the dead man's switch, tested).

• One alarm: vitals raises a single backup card.
• Drills: a drill that throws is now stored as a red drill. Before, it only logged restore drill failed to run and left no record. This changes test/mcp-restore-drill.test.ts: it expected a "could not run" answer from the MCP, and I rewrote it to expect a red drill, because the old expectation was the lie.
• Stall: VACUUM INTO now runs in a worker (src/snapshot/vacuum.ts), for the snapshot copy and for app databases. The test fails if the copy runs inline. I did not measure the 8.8 s freeze itself.

Not done

• Resumable snapshots: a snapshot interrupted by a release is retaken whole after boot rather than resumed.
• K8s lanes (#7667): nothing in this repo knows them. A lane is one more entry in the list passed to watchBucket.
• Dead man's switch: it lives inside the forge, so it catches a stopped check but not a dead forge. That needs an external probe.
• Drill logs: I could not see the child forge's restore drill failed to run logs. The failure is now recorded as a red drill with its reason, which is where to look next.
• No drill yet: this shows as "no restore drill yet" and is not red; the existing eight-day drill card still covers it.

I added docs/features/backup-truth.md rather than a README section. To try it, open home as an admin and look at the backup tile, or call the backups tool.

I fixed the three failing tests and pushed. test/unreached.test.ts, test/backup-truth.test.ts and test/stale_reds.test.ts now pass here (22 tests). I did not run the full suite, and tsc was clean.

• Unreached module: src/snapshot/vacuum_worker.ts is started with new Worker(new URL(...)), so nothing imported it and the check flagged it. It now exports a VacuumJob type that src/snapshot/vacuum.ts imports to type the message it sends. I did not add it to the allow-list.
• The two stale_reds tests: this was a wrong test, not a bug in my change. It fixed its clock at 2026-11-02T12:00Z, but it creates its pulls at the real current time. Once the wall clock passed about 17:02Z, the "human word" that restarts the tries was newer than the faked runs. The pull was re-checked instead of parked, and the flow-report test failed with it. The test now takes its start time from new Date() instead of a fixed date.

This is why the change to test/stale_reds.test.ts is not part of the backup work. The failure is not caused by my change, but it only shows up when the clock is late in the day, so it would have hit any pull run then.

I had also run stale_reds on a clean main checkout, and it passed there. I did not work out why. The likely reason is that my first run came a few minutes before the 17:02 cutoff.

I merged main into the branch and pushed it. After the merge, tsc, scripts/gate.ts and four test files pass here (stale_reds, backup-truth, unreached and mcp-size, 26 tests). I did not run the full suite.

The only conflict was test/stale_reds.test.ts. main had already fixed the same fixed-date clock problem I fixed in my last push, with T0 = new Date(). The code was identical, so I kept main's version, including its comment. Nothing else needed resolving.

Neither failure comes from this pull, and I changed no code to address them. I merged the latest main (clean, no conflicts) and pushed it. tsc, test/backup-truth.test.ts and test/unreached.test.ts pass after the merge.

• "a row wears a dot… under its byte budget" (test/issues-quiet.test.ts): this fails on origin/main itself, so the pull can't fix it. The issues page comes to 51,705 bytes on main and 51,385 on my branch, against a 50,000 budget (measured at 49,753 when #8722 landed). My branch was smaller than main before the merge, so nothing I changed adds to it. I did not raise the budget or trim the page. The fix belongs to whoever owns #8722's page weight, or to a decision to move the budget. After the merge it still fails here at 51,705 bytes, so I expect the next check to fail on it again until main is fixed.
• "a sleeping preview woken at the limit" (test/quotas-running.test.ts): this looks like a timeout flake on a loaded shard, since the log shows it took 120 s. Run alone here it passes (7 of 7, 18 s). It has no connection to the files I touched.

All six points are addressed and pushed, apart from the two box gaps in item 5, which are now tracked as issues. tsc, scripts/gate.ts, scripts/design-lint.ts and 12 related test files pass (118 tests, including test/mcp-size.test.ts). I did not run the full suite, so the forge's check will.

1. Green while nothing is known: the verdict now has a third state, unknown. Before the first bucket read, home shows "—" with the note "not checked yet". A read with no restore drill yet is also unknown ("no restore drill yet"). Only red raises the alarm, and only green is calm. The just-booted test now asserts unknown, and a new test covers the no-drill case.
2. Unscrubbed errors: readLane now strips the sealed, env and mirror keys with scrubReason, plus presigned query strings and userinfo in URLs. Those credentials are only looked up when a read fails, because opening sealed secrets is audited. A new test puts a secret in the error and checks it is absent from the lane and from the verdict.
3. Terminal: the Resilience dialog (palette entry "Resilience", admins) now opens with the verdict, each lane's newest snapshot and every reason. I added a test, and docs/features/backup-truth.md says how to reach it from the web, the terminal and the MCP.
4. Tests:

- Both restart tests now use an injected clock (now option and takeIfDue() on the handle) and always assert. The second one checks that a boot with the night's snapshot in the store takes none, and that one a day later takes one. - The bucket assertion awaits checkBucket() instead of sleeping.

1. Gaps: I opened #8805 (platform/K8s lanes on home) and #8806 (an off-process probe, and snapshots taken outside the serving process or resumable). Both are linked as blockers of #8742. Catch-up still retakes a lost snapshot whole.
2. Design: the tile note is now a few words ("late", "check stopped", "drill failed", or "4.8 MiB · drill passed"). Red uses the existing attn tone, with no "RED" in caps. The title attribute carries only the first reason. The Resilience dialog still says "RED" in text; that is a terminal-only label, since the dialog has no tone to use.

Two failures from the last check (issues-quiet byte budget, quotas-running) are unchanged and not caused by this pull. issues-quiet also fails on main. I did not touch either.

Left #8742 open: 3 unticked items remain (A restart with a snapshot due or lost takes one shortly after boot, as the drill does; a test restarts mid snapshot. A release doesn't lose a snapshot: it is resumable, or happens outside the serving process.; Every other platform lane shows there too, each with its newest object, including the K8s lanes while they exist (#7667).; The alarm can't die silently with what it watches: the forge's check is independent of them, and a missed check is itself red (dead man's switch).).