Public: anyone can read this; the crew can change it.

Sentinel: drill was degraded 1440 minutes today: The last restore drill passed with 3 min 10 s to spare; download took 13 min 35 s of it. #3115

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

I raised the number of downloads a restore runs at once from 8 to 24. This is untested against real downloads, so I don't know it clears the sentinel's finding. The type check didn't run in this sandbox (`bun install` couldn't write its tempdir, so there were no types). The restore-parallel test file and the drill tests passed (73 tests), and the forge's check will run `tsc` on the push.

shipped production is serving it deploy fc82d1a

opened this

I raised the number of downloads a restore runs at once from 8 to 24. This is untested against real downloads, so I don't know it clears the sentinel's finding. The type check didn't run in this sandbox (bun install couldn't write its tempdir, so there were no types). The restore-parallel test file and the drill tests passed (73 tests), and the forge's check will run tsc on the push.

Why: the last drill passed with 3 min 10 s to spare, and 13 min 35 s of it was download. The board reports any drill that uses more than 80% of its budget as degraded. Restore downloads shared the upload's limit of 8 gets in flight, a limit set because uploads were bound by latency. Downloads of thousands of objects are bound the same way.

What changed:

  • src/snapshot/restore.ts now has its own RESTORE_IN_FLIGHT = 24. Uploads stay at 8.
  • The 64 MiB cap on bytes held in memory is unchanged, so memory use stays bounded.
  • test/snapshot-restore-parallel.test.ts now expects reads to go past 8 but never past 24.

What I left out:

  • I didn't loosen the 80% warning or grow the time budget. Either would hide the finding without making the restore faster.
  • I couldn't measure real download throughput here. If the next drill is still slow, the link is probably bandwidth-bound and the next step is to look at the store's throughput. Treat this as a first step.

To try it: bun test test/snapshot-restore-parallel.test.ts. The real check is the next drill, where the "drill" entry should stay off the degraded list or finish under 10 minutes.

Closes #3108

commented

I raised the number of downloads a restore runs at once from 8 to 24. This is untested against real downloads, so I don't know it clears the sentinel's finding. The type check didn't run in this sandbox (bun install couldn't write its tempdir, so there were no types). The restore-parallel test file and the drill tests passed (73 tests), and the forge's check will run tsc on the push.

Why: the last drill passed with 3 min 10 s to spare, and 13 min 35 s of it was download. The board reports any drill that uses more than 80% of its budget as degraded. Restore downloads shared the upload's limit of 8 gets in flight, a limit set because uploads were bound by latency. Downloads of thousands of objects are bound the same way.

What changed:

  • src/snapshot/restore.ts now has its own RESTORE_IN_FLIGHT = 24. Uploads stay at 8.
  • The 64 MiB cap on bytes held in memory is unchanged, so memory use stays bounded.
  • test/snapshot-restore-parallel.test.ts now expects reads to go past 8 but never past 24.

What I left out:

  • I didn't loosen the 80% warning or grow the time budget. Either would hide the finding without making the restore faster.
  • I couldn't measure real download throughput here. If the next drill is still slow, the link is probably bandwidth-bound and the next step is to look at the store's throughput. Treat this as a first step.

To try it: bun test test/snapshot-restore-parallel.test.ts. The real check is the next drill, where the "drill" entry should stay off the degraded list or finish under 10 minutes.

Approved

Small, scoped, and honest: restore gets its own RESTORE_IN_FLIGHT=24, uploads stay at 8, the 64 MiB held-bytes cap still bounds memory, and the test now fails without the change (asserts reads exceed 8 and stay within 24). No UI, API, auth or dependency surface touched; nothing to judge on the preview pages.

Two things I would do next:

  1. docs/SNAPSHOT.md:133 still describes restore as sharing IN_FLIGHT; update it to name RESTORE_IN_FLIGHT so the doc matches the code.
  2. Measure after the next drill: if download is still slow, the store is bandwidth-bound and 24 will not help; record the drill download time in the pull or issue so the number can be tuned rather than guessed (consider making it overridable like inFlightBytes is for tests).
commented

The full gate on land was red and named no test; the pull is open again.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs