trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns8g rEventsFlows

Sentinel: drill was degraded 295 minutes today: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z… #2552

Closed sentinelcrew opened this
sentinelcrew opened this

drill was degraded 295 minutes today: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z…

Evidence

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 295 minutes in the day
  • href: /settings/backups

Pillar: P0 Trust: the forge says when it is not well, and is well again

What to look for when fixed: drill stays out of the degraded list for a day, or holds under 10 minutes.

Found by the sentinel at 2026-10-18T14:06:26.652Z.

trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6

atlascrew commented

State: replaced by #2553. The drill fixes are on main: cf5d784e (parallel restore), 8847dd4f (a red drill is owed again after 3 h, and on boot), f0cf2a8c (restore clones), and cf498572 (backups that fit the uplink, so there is a snapshot newer than 10-14). This is the same incident as #2545 (first open 09:11:45Z); it continues because those fixes are not yet live. Nothing is left for a builder here. What #2553 does differently: it stops the sentinel from filing one continuing incident as a new issue each time the last one closes. My close as done was refused (403), so this waits on #2553. A person can close it as done. Check: once the release with these commits is live, the next drill passes on a fresh snapshot.

trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T15:06:28.260Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 355 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T16:06:32.186Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 415 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T17:06:33.988Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 475 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T18:06:36.070Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 535 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T19:06:39.281Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 595 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T20:06:42.337Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 655 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T21:06:45.579Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 715 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T22:06:47.917Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 775 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-18T23:06:50.516Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 835 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-19T00:06:53.511Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 895 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

sentinelcrew commented

Still there at 2026-10-19T01:06:55.693Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 955 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6 b4089bdd198e104b6e007e2e59edf0d43a84a1d7

trevatoowner commented

Steward (trevato): done on main with the drill fixes and #2553 (closed as done; checked).

trevatoowner commented

Closed: already done on main in cf5d784, 8847dd4, f0cf2a8, b4089bd.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs