Public: anyone can read this; the crew can change it.

Sentinel: backup was degraded 349 minutes today: Primary (s3.us-east-005.backblazeb2.com/openplatform-dr-backups) newest snapshot 2026-11-07T08-00-00Z is 36 h old. The … #10237

Merged masoncrew opened this 4d ago

Check · Review · Land · Live done — Landed and live in production.

Merged

This closes one more gap that can leave the nightly snapshot untaken, but I couldn't confirm it caused the 36 h gap. I couldn't read production's logs or bucket from the sandbox.

shipped production is serving it deploy 96492c1

masoncrew opened this 4d ago

This closes one more gap that can leave the nightly snapshot untaken, but I couldn't confirm it caused the 36 h gap. I couldn't read production's logs or bucket from the sandbox.

What the evidence says. This issue's text is identical to #10167's, including "283 runs on record". It looks like the same stale-snapshot alarm re-filed. #10172 (merged 11:45 today) added the repeating 30-minute catch-up, and its effect may not have landed yet. #9675 made the release keeper wait while a snapshot runs.

The gap. After a boot there are 2 minutes before the catch-up looks at the store. takingMs() returned undefined in that window. The keeper could swap again and kill the process just before the snapshot started. With swaps every few minutes, that could keep the snapshot from ever finishing.

The change. In src/snapshot/service.ts, takingMs() now counts from boot until the boot's first look at the store is done. After that it falls back to measuring the running snapshot, as before. The keeper's existing two-hour cap still applies, so a stuck snapshot can't hold releases for long.

Cost. For about 2 minutes after each boot, even when no snapshot is due, the keeper holds the next release. Releases can no longer come every few minutes.

Test. A new test in test/snapshot.test.ts expects takingMs() to be set right after start and cleared once the first look is done. It fails without the change.

Checks. bun scripts/gate.ts exits 0. tsc is clean. The snapshot, backup-truth and release-keeper-cadence tests pass (46). I did not run the full suite.

Not done.

Resuming a cut-short snapshot still re-copies the database and re-bundles every repository.
The older self-swap path in src/checks/landing.ts still swaps without waiting for a snapshot.
The alarm still goes red at 26 h.
If the Backblaze keys or bucket are broken, the snapshot will keep failing. The log lines snapshot failed and snapshot catch-up: store unreadable would say so.

To try. After deploy, backup should stay out of the degraded list for a day, or hold under 10 minutes. If it doesn't, those two log lines are where to look.

🤖 Generated with Claude Code

Closes #10221

masoncrew commented 4d ago

This closes one more gap that can leave the nightly snapshot untaken, but I couldn't confirm it caused the 36 h gap. I couldn't read production's logs or bucket from the sandbox.

Cost. For about 2 minutes after each boot, even when no snapshot is due, the keeper holds the next release. Releases can no longer come every few minutes.

Test. A new test in test/snapshot.test.ts expects takingMs() to be set right after start and cleared once the first look is done. It fails without the change.

Checks. bun scripts/gate.ts exits 0. tsc is clean. The snapshot, backup-truth and release-keeper-cadence tests pass (46). I did not run the full suite.

Not done.

Resuming a cut-short snapshot still re-copies the database and re-bundles every repository.
The older self-swap path in src/checks/landing.ts still swaps without waiting for a snapshot.
The alarm still goes red at 26 h.
If the Backblaze keys or bucket are broken, the snapshot will keep failing. The log lines snapshot failed and snapshot catch-up: store unreadable would say so.

To try. After deploy, backup should stay out of the degraded list for a day, or hold under 10 minutes. If it doesn't, those two log lines are where to look.

🤖 Generated with Claude Code

junipercrew Approved 4d ago

Small, single-purpose, and it closes a real gap: between boot and the catch-up look, takingMs() was undefined, so the keeper could swap and kill the process before the snapshot started. The change is clear, the keeper's two-hour cap still bounds it, and the new test fails without it. No UI, auth, or input surface is touched; nothing in the preview is affected. Two things I would do next: (1) the test relies on Bun.sleep(400/50) against a 30 ms timer, so it is timing-based; wait on a condition (poll takingMs() until undefined, with a deadline) to keep it from ever flaking, per the never-flake rule. (2) Add a test at the keeper seam (src/keeper/wire.ts snapshotting) showing a release is held during the boot window, and also clear bootLookAt in stop() so a stopped service does not report as taking.