Sentinel: backup was degraded 41 minutes today: Primary (s3.us-east-005.backblazeb2.com/openplatform-dr-backups) newest snapshot 2026-11-07T08-00-00Z is 26 h old. The … #9675

The release keeper no longer cuts a release while a snapshot is being taken. After two hours a snapshot counts as stuck and stops holding releases.

Cause. With FORGE_RELEASE_KEEPER=swap, the keeper can cut as often as once a minute (SWAP_INTERVAL_MS). On 2026-11-08, main landed 8–14 merges an hour from 08:00. Each swap drains the process the nightly snapshot runs in. A resumed snapshot only skips repository and run bundles; it still re-copies the database and re-bundles every repository. So with a swap every few minutes it never finished, and primary's newest snapshot stayed at 2026-11-07T08-00-00Z. That earlier one landed because nothing merged in its window on 11-07. This is inferred from the merge timestamps on main and the code; I could not read production's logs or bucket from the sandbox.

Change

• src/snapshot/service.ts: SnapshotsHandle.takingMs() says how long the snapshot in flight has run, or undefined when none is.
• src/keeper/cadence.ts: decideCut gets a new rule. While a snapshot is in flight and has run less than SNAPSHOT_WAIT_MS (2 h), it refuses with "a snapshot is being taken (N min in); a release would cut it short". The keeper's log line says the same.
• src/keeper/keeper.ts and src/keeper/wire.ts: an optional snapshotting port, wired to snapshotsOf(app)?.takingMs().
• docs/features/release-keeper.md lists the new rule.

Not done

• The older self-swap path in src/checks/landing.ts (without the keeper) still swaps without waiting; production runs the keeper.
• A swap can still land in the 2 minutes between a boot and the catch-up snapshot starting. Once that snapshot starts, it holds the next cut.
• Resuming a snapshot still re-copies the database and re-bundles every repository.
• The alarm itself is unchanged: it still goes red at 26 h.

Tests. test/release-keeper-cadence.test.ts checks the rule, the two-hour cap, and a keeper tick held and then released. test/snapshot.test.ts checks that takingMs is set during a snapshot and cleared after it. bun scripts/gate.ts and bun x tsc --noEmit pass. I ran 34 test files covering the keeper, snapshots, backups, the docs and import cost rather than all 118 that fast.ts named (about 9 minutes): 226 pass, 2 fail. The 2 failures are in test/snapshot-destination.test.ts; they fail the same way without this change, because the sandbox blocks outbound 127.0.0.1:1.

To check it's fixed: on a busy merge morning, the keeper's log should show the "a snapshot is being taken" line after 08:00, a cut once the nightly lands, and backup should stay out of the sentinel's degraded list.

🤖 Generated with Claude Code

Closes #9653

The release keeper no longer cuts a release while a snapshot is being taken. After two hours a snapshot counts as stuck and stops holding releases.

Cause. With FORGE_RELEASE_KEEPER=swap, the keeper can cut as often as once a minute (SWAP_INTERVAL_MS). On 2026-11-08, main landed 8–14 merges an hour from 08:00. Each swap drains the process the nightly snapshot runs in. A resumed snapshot only skips repository and run bundles; it still re-copies the database and re-bundles every repository. So with a swap every few minutes it never finished, and primary's newest snapshot stayed at 2026-11-07T08-00-00Z. That earlier one landed because nothing merged in its window on 11-07. This is inferred from the merge timestamps on main and the code; I could not read production's logs or bucket from the sandbox.

Change

• src/snapshot/service.ts: SnapshotsHandle.takingMs() says how long the snapshot in flight has run, or undefined when none is.
• src/keeper/cadence.ts: decideCut gets a new rule. While a snapshot is in flight and has run less than SNAPSHOT_WAIT_MS (2 h), it refuses with "a snapshot is being taken (N min in); a release would cut it short". The keeper's log line says the same.
• src/keeper/keeper.ts and src/keeper/wire.ts: an optional snapshotting port, wired to snapshotsOf(app)?.takingMs().
• docs/features/release-keeper.md lists the new rule.

Not done

• The older self-swap path in src/checks/landing.ts (without the keeper) still swaps without waiting; production runs the keeper.
• A swap can still land in the 2 minutes between a boot and the catch-up snapshot starting. Once that snapshot starts, it holds the next cut.
• Resuming a snapshot still re-copies the database and re-bundles every repository.
• The alarm itself is unchanged: it still goes red at 26 h.

Tests. test/release-keeper-cadence.test.ts checks the rule, the two-hour cap, and a keeper tick held and then released. test/snapshot.test.ts checks that takingMs is set during a snapshot and cleared after it. bun scripts/gate.ts and bun x tsc --noEmit pass. I ran 34 test files covering the keeper, snapshots, backups, the docs and import cost rather than all 118 that fast.ts named (about 9 minutes): 226 pass, 2 fail. The 2 failures are in test/snapshot-destination.test.ts; they fail the same way without this change, because the sandbox blocks outbound 127.0.0.1:1.

To check it's fixed: on a busy merge morning, the keeper's log should show the "a snapshot is being taken" line after 08:00, a cut once the nightly lands, and backup should stay out of the sentinel's degraded list.

🤖 Generated with Claude Code