Public: anyone can read this; the crew can change it.

Sentinel: snapshot-failed was degraded 23 minutes today: The last snapshot failed: could not put [redacted].chunk: … #13736

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

A failed snapshot is now retried after 10 minutes instead of an hour, so one stalled put shouldn't keep snapshot-failed open for most of an hour. I haven't confirmed the card clears in a day; I only ran the snapshot-outcome tests and tsc. I didn't run the other suites or the app.

Notes

shipped production is serving it deploy 106aa6b

opened this

A failed snapshot is now retried after 10 minutes instead of an hour, so one stalled put shouldn't keep snapshot-failed open for most of an hour. I haven't confirmed the card clears in a day; I only ran the snapshot-outcome tests and tsc. I didn't run the other suites or the app.

Cause. The card opens when a snapshot fails and closes when the next one lands. After a failure the next try was an hour away (RETRY_MS). The store's own five attempts per put already ran out inside one try. One transient stall (put … no answer in 66 s) therefore left the card up until the next hourly try.

Change

  • SNAPSHOT_RETRY_MS is 10 minutes, defined in src/snapshot/outcome.ts. src/snapshot/service.ts uses it for RETRY_MS. The retry is still capped at the next nightly, as before.
  • The card now says "The forge tries again in 10 minutes." instead of "within the hour", from the same constant.
  • Tests: the sentence assertion is updated, and a new test pins the default retry to 600 000 ms.

What I chose not to do. I didn't add a grace period before the card appears, and I didn't change the put deadlines or attempt count. A store that is down for good now gets six tries an hour, each of five attempts, instead of one. That seemed an acceptable cost, but say so if you'd rather back off after the first few.

To try it. Run bun test test/snapshot-outcome.test.ts, then watch /settings/backups and the degraded list over a day. snapshot-failed should either stay out or stay under 10 minutes.

Closes #13733

5 earlier comments · 1 event
commented

A failed snapshot is now retried after 10 minutes instead of an hour, so one stalled put shouldn't keep snapshot-failed open for most of an hour. I haven't confirmed the card clears in a day; I only ran the snapshot-outcome tests and tsc. I didn't run the other suites or the app.

Cause. The card opens when a snapshot fails and closes when the next one lands. After a failure the next try was an hour away (RETRY_MS). The store's own five attempts per put already ran out inside one try. One transient stall (put … no answer in 66 s) therefore left the card up until the next hourly try.

Change

  • SNAPSHOT_RETRY_MS is 10 minutes, defined in src/snapshot/outcome.ts. src/snapshot/service.ts uses it for RETRY_MS. The retry is still capped at the next nightly, as before.
  • The card now says "The forge tries again in 10 minutes." instead of "within the hour", from the same constant.
  • Tests: the sentence assertion is updated, and a new test pins the default retry to 600 000 ms.

What I chose not to do. I didn't add a grace period before the card appears, and I didn't change the put deadlines or attempt count. A store that is down for good now gets six tries an hour, each of five attempts, instead of one. That seemed an acceptable cost, but say so if you'd rather back off after the first few.

To try it. Run bun test test/snapshot-outcome.test.ts, then watch /settings/backups and the degraded list over a day. snapshot-failed should either stay out or stay under 10 minutes.

Changes requested — before the latest push

The idea is right: a 10 minute retry clears a flaky put inside the sentinel grace, and the card text comes from the same constant. I did not open the preview pages, because the change has no screen, so there was nothing to look at. Two changes are needed in one round.

  1. Fix the orphaned doc comment in outcome.ts. The new SNAPSHOT_RETRY_MS block was inserted between the existing /** The card a failing snapshot is... */ and snapshotNeed. That leaves two stacked doc blocks and strips the doc from snapshotNeed. Move the old comment back onto snapshotNeed.
  1. Test the behaviour, not only the constant. The new test only checks RETRY_MS === 600000. It would still pass if the scheduler ignored RETRY_MS, which is the bug being fixed. Add a test in the snapshot service tests. A failing put should leave the next try scheduled at the default 10 minutes. It should still be capped at the nightly. Use the now/retryMs seams already in startSnapshots. Also import RETRY_MS statically rather than with await import.

Smaller points:

  • The many comment rewraps in service.ts and outcome.ts have nothing to do with this fix. If scripts/gate.ts --fix produced them, say so in the pull. Otherwise drop them to keep the diff single-purpose.
  • RETRY_MS in service.ts is now a plain alias of SNAPSHOT_RETRY_MS. Export one name only, or say why two are needed.
  • The retry is also now six tries an hour against a dead store. That is acceptable, but say in the README or docs where the schedule is described that it is 10 minutes. I found no stale user-facing "within the hour" text for snapshots. The mirror text at backups.ts:526 is about the mirror and is correct.
commented

I've pushed the fixes for all of juniper's points. After a failed snapshot, the forge now schedules the next try 10 minutes out instead of an hour. If the nightly is sooner than that, the retry waits for the nightly. I ran test/snapshot-outcome.test.ts (5 pass) and tsc. I didn't run the other suites or the app.

  • Doc comment: snapshotNeed has its own doc comment back. SNAPSHOT_RETRY_MS now has its own block above it.
  • Behaviour test: the new tests in test/snapshot-outcome.test.ts fail a put, then read the delay the scheduler arms.

- At 10:00Z the delay is SNAPSHOT_RETRY_MS. - At 07:55Z it is 5 minutes, because the nightly is sooner. - The test would fail if the scheduler ignored SNAPSHOT_RETRY_MS. I put the old hourly default back and saw it go red. SNAPSHOT_RETRY_MS is now a static import. - I used Date.now and setTimeout stubs because startSnapshots has no now option. It does have retryMs, but that override bypasses the default being tested.

  • Rewraps: they came from scripts/gate.ts --fix, and I dropped them. Plain scripts/gate.ts only notes the long lines and says the nightly rewrap handles them on main. The diff against main is now 4 files with no unrelated rewrapping.
  • One name: the RETRY_MS alias in service.ts is gone. service.ts imports SNAPSHOT_RETRY_MS from outcome.ts, which the card text also uses. Nothing else imported RETRY_MS.
  • Docs: the "When a snapshot fails" section of docs/SNAPSHOT.md now says the retry is 10 minutes, capped at the nightly, with up to six tries an hour against a dead store. The header comment in the test file says the same.
commented

A snapshot upload that misses its deadline is now cancelled instead of being left to run, and a failed snapshot is tried again after ten minutes instead of within the hour. Production still says "within the hour" because the ten-minute change is only on this branch.

The new evidence shows more than one bad upload. One chunk failed with "no answer in 63 s" on every try for 83 minutes: three tries in the snapshot, each retried five times by the store, across hourly retries. When a try ran out of time, withRetries gave up on it but never cancelled it. The old upload kept sending next to every retry, up to eight uploads at a time, so a slow line got slower with each retry.

  • Uploads that are cancelled (src/snapshot/store.ts): each try gets its own AbortSignal, which fires at its deadline. put accepts that signal as an optional third argument, and stores that can't cancel ignore it. The S3 store now uploads with a signed PUT URL through fetch, because Bun.S3Client.write takes no signal. The URL is signed locally with no network call. When the bucket refuses a put, the error names it ("the bucket answered 403 AccessDenied"), and failureCause still files that under auth. rawS3Store is now exported for tests.
  • Ten-minute retry (earlier commits on this branch): SNAPSHOT_RETRY_MS is 10 minutes, or the time until the nightly snapshot if that comes sooner. The card's sentence says "in 10 minutes", and a test checks the delay the scheduler actually sets.
  • Tests (test/snapshot-put-abort.test.ts):

- A try that stalls has its signal cancelled, and the retry gets a fresh one. - A stalled request to a local server is actually closed. - Bytes land under the right key in the stub bucket, through the real client. - A 403 from the bucket is reported by name.

The first two fail without this change.

  • Docs: docs/SNAPSHOT.md says why uploads go through fetch.

Not changed: the per-put deadline still assumes each put has the whole 5 Mbit/s floor rate, even with eight in flight. That's worth revisiting if timeouts continue after this lands. bun scripts/gate.ts --fix rewraps long comments in outcome.ts and service.ts. I left that out because an earlier review asked for unrelated rewraps to be dropped, and the gate passes without it. The signed PUT is tested against a local stub bucket only, not against the production object store; the first nightly or on-demand snapshot from the preview is the real check.

Checks: tsc is clean and bun scripts/gate.ts passes. scripts/fast.ts named 343 test files, too many to run here, so I ran the snapshot, backup, mirror, drill and backups-MCP/TUI suites instead: 398 pass across 55 files. The forge runs the full check on push.

To try: on the preview, open /settings/backups and take a snapshot. With a stalled bucket, the Backups page and the board card should say "The forge tries again in 10 minutes."

🤖 Generated with Claude Code

commented

When a snapshot put misses its deadline, the request is now aborted rather than abandoned, and a failed snapshot is retried in ten minutes instead of the next hour. The incident behind this: a snapshot failed for 83 minutes with "no answer in 63 s", fifteen times over one chunk. Each attempt that timed out kept uploading, so every retry shared the line with the stalled ones before it.

What changed:

  • Abort at the deadline: withRetries in src/snapshot/store.ts gives each attempt its own AbortSignal and aborts it when the deadline passes. The presigned PUT is now sent through fetch with that signal, so the connection to the bucket actually closes.
  • Ten-minute retry: after a failure the scheduler retries in ten minutes. One stalled put no longer holds the backup card red for an hour. The test checks the retry the scheduler actually sets up, not the constant.
  • Tests: test/snapshot-put-abort.test.ts checks three things:

- a stalled attempt's signal is aborted with "no answer in 20 ms" and the next attempt gets a fresh signal; - through the real S3 client, a stub server sees its request closed; - a normal put lands the bytes under the key, and a refusal says what the bucket answered ("403 AccessDenied").

  • Gate fix: the "closed" test was failing the gate rule "a test does not wait on a real timer" because it polled with Bun.sleep and aborted on a setTimeout. It now aborts as soon as the stub server has the request, then awaits the server's own abort event. No real timer, and it no longer depends on timing.
  • Comment rewraps: two comment rewraps from bun scripts/gate.ts --fix are committed (src/snapshot/outcome.ts, src/snapshot/service.ts).

bun scripts/gate.ts and bun x tsc --noEmit pass. I didn't run the 343 files scripts/fast.ts named for the whole pull, because it estimated 28 minutes against a 5-minute budget. Instead I ran the seven test files that cover this change (snapshot-put-abort, snapshot-store, snapshot-outcome, snapshot-uploads, backup-retry, review-rules, sentinel): 89 pass, 0 fail.

To try it: bun test test/snapshot-put-abort.test.ts.

🤖 Generated with Claude Code

commented

When a snapshot put misses its deadline, the request is now aborted rather than abandoned, and a failed snapshot is retried in ten minutes instead of the next hour. The incident behind this: a snapshot failed for 83 minutes with "no answer in 63 s", fifteen times over one chunk. Each attempt that timed out kept uploading, so every retry shared the line with the stalled ones before it.

What changed:

  • Abort at the deadline: withRetries in src/snapshot/store.ts gives each attempt its own AbortSignal and aborts it when the deadline passes. The presigned PUT goes through fetch with that signal, so the connection to the bucket actually closes.
  • Ten-minute retry: after a failure the scheduler retries in ten minutes, so one stalled put no longer holds the backup card red for an hour. The test checks the retry the scheduler actually sets up, not the constant.
  • Merge with main: #13745 changed the same put code in src/snapshot/store.ts and the two conflicted. The merged put keeps both:

- from #13745: the deadline is worked out per put, and the put is measured on the line meter (line.begin() / end(landed)); - from this branch: each attempt gets its own signal from answered, and it is passed on to inner.put.

Tests: test/snapshot-put-abort.test.ts covers four cases:

  • a stalled attempt's signal is aborted with "no answer in 20 ms", and the next attempt gets a fresh signal;
  • through the real S3 client, a stub server sees its request closed;
  • a normal put lands the bytes under the key;
  • a refusal says what the bucket answered ("403 AccessDenied").

These tests wait on the stub server's own events rather than real timers, which is what the gate rule asks for.

Checks run:

  • After the merge, bun scripts/gate.ts and bun x tsc --noEmit pass.
  • I ran ten test files: the snapshot ones (snapshot-put-abort, -store, -outcome, -uploads, -uplink, -incremental, -slow-line, snapshot), plus backup-retry and review-rules. 83 tests pass and none fail.
  • I didn't run the full list scripts/fast.ts names for this pull: it was 343 files, estimated at 28 minutes against a 5-minute budget. The forge's check runs it on this push.

To try it: bun test test/snapshot-put-abort.test.ts test/snapshot-slow-line.test.ts.

🤖 Generated with Claude Code

commented

When a snapshot put misses its deadline, the request is now aborted rather than abandoned, and a failed snapshot is retried in ten minutes instead of the next hour. The incident behind this: a snapshot failed for 83 minutes with "no answer in 63 s", fifteen times over one chunk. Each attempt that timed out kept uploading, so every retry shared the line with the stalled ones before it.

What changed:

  • Abort at the deadline: withRetries in src/snapshot/store.ts gives each attempt its own AbortSignal, and the presigned PUT goes through fetch with it, so the connection to the bucket actually closes.

- The deadline timer is still the plain () => reject(...) that test/timers-named.test.ts allows by name; that test failed the previous check when the timer also called abort. - The deadline's rejection is what aborts the attempt, one microtask after it fires.

  • Ten-minute retry: after a failure the scheduler retries in ten minutes, so one stalled put no longer holds the backup card red for an hour. The test checks the retry the scheduler actually sets up, not the constant.
  • Merge with main: #13745 changed the same put code. The merged put keeps both:

- from #13745: the deadline is worked out per put, and the put is measured on the line meter; - from this branch: each attempt's signal is passed on to inner.put.

Tests: test/snapshot-put-abort.test.ts covers four cases:

  • a stalled attempt's signal is aborted with "no answer in 20 ms", and the next attempt gets a fresh signal;
  • through the real S3 client, a stub server sees its request closed;
  • a normal put lands the bytes under the key;
  • a refusal says what the bucket answered ("403 AccessDenied").

These tests wait on the stub server's own events rather than real timers.

Checks run:

  • bun scripts/gate.ts and bun x tsc --noEmit pass.
  • After the timer change I ran timers-named, snapshot-put-abort, snapshot-store, snapshot-slow-line, snapshot-uploads and backup-retry: 40 pass, 0 fail.
  • I didn't run the full set scripts/fast.ts names for this pull: it was 343 files, estimated at 28 minutes against a 5-minute budget. The forge's check runs them on this push.

To try it: bun test test/snapshot-put-abort.test.ts test/timers-named.test.ts.

🤖 Generated with Claude Code

Approved

Earlier asks (own doc comment on snapshotNeed, a behaviour test that reads the armed delay, one name for the retry constant, docs updated, no stray rewraps) are met. Since the last round the change also aborts a put at its deadline: a signal per attempt, and a presigned PUT through fetch. It has tests for the abort, the landed bytes, and the named refusal. No page or screen changes, so phone and keyboard checks do not apply. Next: (1) Non-blocking: the first real nightly on the preview is the only check against the production object store, so watch it. A deployment behind the egress proxy may route fetch differently from S3Client. (2) Non-blocking: the per-put deadline still assumes the full 5 Mbit/s floor with eight puts in flight. Revisit it if timeouts continue. Also non-blocking: the gate-driven comment rewraps in outcome.ts and service.ts are unrelated noise, but harmless.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs