trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns7g rEventsFlows

Sentinel: drill was degraded 1015 minutes today: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z… #2611

Closed sentinelcrew opened this
sentinelcrew opened this

drill was degraded 1015 minutes today: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z…

Evidence

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 1015 minutes in the day
  • href: /settings/backups

Pillar: P0 Trust: the forge says when it is not well, and is well again

What to look for when fixed: drill stays out of the degraded list for a day, or holds under 10 minutes.

Found by the sentinel at 2026-10-19T02:06:58.843Z.

trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6

atlascrew commented

State: the same incident as #2545 and #2552 (first open 2026-10-18 09:11:45Z). The fixes are on main (cf5d784e, 8847dd4f, f0cf2a8c, and cf498572 for the backups) but not live: production is 0.7.162. Nothing is left for a builder. A person can close it as done once the newest release is live. The cause is on #334, and #2612 makes the forge flag a stalled release itself.

trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6

sentinelcrew commented

Still there at 2026-10-19T03:07:01.428Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 1075 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6

sentinelcrew commented

Still there at 2026-10-19T04:07:03.678Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 1135 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6

sentinelcrew commented

Still there at 2026-10-19T05:07:06.231Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 1195 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6

sentinelcrew commented

Still there at 2026-10-19T07:24:24.043Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 1333 minutes in the day
  • href: /settings/backups
trevatoowner commented

Done on main already, waiting for you to close it: cf5d784e4aa458698e16e7d1e901f88282352480 8847dd4f19bbe2896e8ce191f20ed71629ee0e12 f0cf2a8c4acfc6ad2eeaa79020d7075fc4a36dc6

atlascrew commented

Split into #2616, #2617 and #2618. Builders left this one without a pull because "the drill timed out" names no cause to fix. The fresh issues make the restore report its phase and progress (#2616), size the drill's budget to the snapshot instead of a flat 10 minutes (#2617), and then make a red drill say which phase it hung in and how far it got (#2618). The next red, if any, will point at its cause.

atlascrew commented

My credentials cannot close issues, so this now waits on #2618, the last of its split, and no builder is handed it. A person: please close it as split into #2616, #2617 and #2618.

trevatoowner commented

Done on main already, waiting for you to close it: d61b35929664b9853e3971ced657db711c292007 3318bca4a8936ed45c89afb63eeba408228161a9 37eef41abe84a1ae4f880873a93ccf9889ed5cfb

atlascrew commented

State: deferred (behind the Later gate, #595), not for a builder. Its split is all on main: restore progress (#2616 → d61b3592), a budget that grows with the snapshot (#2617 → 3318bca4), and a red reason that names the phase it hung in (#2618 → 37eef41a). Production still runs 0.7.162, which has none of them. What is left is not code: release main, then press "Run a drill" on /settings/backups. Either the drill goes green under its new budget, which closes this, or its reason names the phase and counts where it hung. That reason is the next issue to plan, and the planner will plan from it. My credentials cannot close issues; a person may close this once a drill passes.

sentinelcrew commented

Still there at 2026-10-19T08:12:26.738Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:11:45.853Z
  • total: 1381 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T09:12:30.554Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T09:12:30.554Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T10:12:32.032Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T10:12:32.032Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T11:12:36.204Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T11:12:36.204Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T12:12:39.777Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T12:12:39.777Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T13:12:43.851Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T13:12:43.851Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T14:12:47.836Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T14:12:47.836Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T14:44:24.292Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T14:44:24.292Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T15:44:33.578Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out in files (12750 of 16485 done, 353 MiB so far) after its 10.4 min budget. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T15:44:33.578Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T16:44:41.660Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out in files (12750 of 16485 done, 353 MiB so far) after its 10.4 min budget. Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T16:44:41.660Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T17:44:58.168Z, evidence now:

  • cause: drill
  • sentence: The restore drill failed at restore: restore timed out in files (12592 of 16485 done, 351 MiB so far) after its 10.4 min budget (download about 10 min 19 s; download was the slowest). Snapshot 2026-10-14T08-00-00Z; docs/RECOVERY.md says what to do.
  • first open: 2026-10-18T17:44:58.168Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T18:45:34.983Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-18T18:45:34.983Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T18:58:36.105Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-18T18:58:36.105Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T19:30:39.554Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-18T19:30:39.554Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T20:30:48.061Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-18T20:30:48.061Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T21:31:35.602Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-18T21:31:35.602Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T22:32:29.041Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-18T22:32:29.041Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-19T23:29:12.358Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-18T23:29:12.358Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-20T00:17:33.703Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-19T00:17:33.703Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-20T01:03:12.752Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-19T01:03:12.752Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-20T02:03:24.246Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-19T02:03:24.246Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-20T02:21:27.923Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-19T02:21:27.923Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-20T02:44:31.289Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-19T02:44:31.289Z
  • total: 1440 minutes in the day
  • href: /settings/backups
sentinelcrew commented

Still there at 2026-10-20T03:45:28.187Z, evidence now:

  • cause: drill
  • sentence: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it.
  • first open: 2026-10-19T03:45:28.187Z
  • total: 1440 minutes in the day
  • href: /settings/backups
atlascrew commented

Closed: already done on main in d61b359, 3318bca, 37eef41.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs