Public: anyone can read this; the crew can change it.

The restore drill's time limit grows with the size of the snapshot it restores #2620

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The restore drill's time limit now grows with the snapshot it restores. Before, every snapshot got a flat 10 minutes; now `drillBudgetMs(bytes)` in `src/snapshot/drill.ts` gives 10 min plus 60 s per GiB (2^30 bytes), capped at 45 min, and `runDrill` uses it whenever `deps.timeoutMs` is unset. `bun x tsc --noEmit` passes, and `test/snapshot-drill.test.ts`, `test/drill-weekly.test.ts` and `test/dril…

shipped production is serving it deploy e209d31

opened this

The restore drill's time limit now grows with the snapshot it restores. Before, every snapshot got a flat 10 minutes; now drillBudgetMs(bytes) in src/snapshot/drill.ts gives 10 min plus 60 s per GiB (2^30 bytes), capped at 45 min, and runDrill uses it whenever deps.timeoutMs is unset. bun x tsc --noEmit passes, and test/snapshot-drill.test.ts, test/drill-weekly.test.ts and test/drill-verdict.test.ts pass (37 tests).

  • How the budget is set: the drill still runs on one clock from fetch to health. Finding the newest snapshot runs under the 10 min floor, since the size isn't known yet. Once the manifest is read, the budget is recomputed from its bytes, or from the sum of its files when that total is missing or not a number. A size that isn't a finite, non-negative number counts as 0.
  • What a red drill says: the reason names the whole drill's budget and the snapshot's size, for example restore timed out: no answer within its 14.1 min budget for a 4.1 GiB snapshot. It no longer shows no answer in 600000 ms. Budgets read as min, s or ms, and sizes in binary units with one decimal.
  • Tests: the budget for 0 bytes (10 min), 4 GiB (14 min), 100 GiB (capped at 45 min) and bad sizes. A stubbed hanging restore of a 4.1 GiB snapshot uses Bun's fake timers: it is still running at 14 min, goes red just past 14.1 min, and its reason names that budget. With the budget line removed, this test fails with "10 min budget".
  • Changed test: the existing "whole milliseconds" test checked the old reason text, which this issue replaces, so I updated that test rather than the code. It now checks that an explicit timeoutMs: 1500.75 reads as 1.5 s budget for a … snapshot.
  • Left alone: the stub reasons in test/drill-weekly.test.ts are fixed sample strings that only get passed through, so they don't need to match the new wording.
  • Docs: docs/RECOVERY.md now says how long the drill allows and gives a sample red reason.

🤖 Generated with Claude Code

Closes #2617

commented

The restore drill's time limit now grows with the snapshot it restores. Before, every snapshot got a flat 10 minutes; now drillBudgetMs(bytes) in src/snapshot/drill.ts gives 10 min plus 60 s per GiB (2^30 bytes), capped at 45 min, and runDrill uses it whenever deps.timeoutMs is unset. bun x tsc --noEmit passes, and test/snapshot-drill.test.ts, test/drill-weekly.test.ts and test/drill-verdict.test.ts pass (37 tests).

  • How the budget is set: the drill still runs on one clock from fetch to health. Finding the newest snapshot runs under the 10 min floor, since the size isn't known yet. Once the manifest is read, the budget is recomputed from its bytes, or from the sum of its files when that total is missing or not a number. A size that isn't a finite, non-negative number counts as 0.
  • What a red drill says: the reason names the whole drill's budget and the snapshot's size, for example restore timed out: no answer within its 14.1 min budget for a 4.1 GiB snapshot. It no longer shows no answer in 600000 ms. Budgets read as min, s or ms, and sizes in binary units with one decimal.
  • Tests: the budget for 0 bytes (10 min), 4 GiB (14 min), 100 GiB (capped at 45 min) and bad sizes. A stubbed hanging restore of a 4.1 GiB snapshot uses Bun's fake timers: it is still running at 14 min, goes red just past 14.1 min, and its reason names that budget. With the budget line removed, this test fails with "10 min budget".
  • Changed test: the existing "whole milliseconds" test checked the old reason text, which this issue replaces, so I updated that test rather than the code. It now checks that an explicit timeoutMs: 1500.75 reads as 1.5 s budget for a … snapshot.
  • Left alone: the stub reasons in test/drill-weekly.test.ts are fixed sample strings that only get passed through, so they don't need to match the new wording.
  • Docs: docs/RECOVERY.md now says how long the drill allows and gives a sample red reason.

🤖 Generated with Claude Code

commented

The restore drill's time limit now grows with the snapshot. drillBudgetMs(bytes) in src/snapshot/drill.ts gives 10 min plus 60 s per GiB of the snapshot's bytes, capped at 45 min, and runDrill uses it whenever deps.timeoutMs is unset. bun x tsc --noEmit passes, and test/snapshot-drill.test.ts and test/recovery-doc.test.ts pass together (19 tests).

  • How the budget is set: finding the newest snapshot still runs under the 10 min floor, because its size isn't known yet. Once the manifest is read, the budget is recomputed from its bytes. If that is missing or not a number, it uses the sum of the files' sizes. A size that isn't a finite, non-negative number counts as 0.
  • What a red drill says: the reason names the whole drill's budget and the snapshot's size, for example restore timed out: no answer within its 14.1 min budget for a 4.1 GiB snapshot. Budgets read as min, s or ms and sizes in binary units with one decimal, so no raw float of milliseconds appears.
  • Tests: the budget is checked for 0 bytes (10 min), 4 GiB (14 min), 100 GiB (capped at 45 min) and bad sizes. A hanging restore of a 4.1 GiB snapshot, run on Bun's fake timers, is still going at 14 min and goes red just past 14.1 min with that budget in its reason. Without the budget change this test fails, reporting a "10 min budget".
  • Changed test: the old "whole milliseconds" test checked the reason text this issue replaces, so the test was wrong rather than the code. It now checks that an explicit timeoutMs: 1500.75 reads as a 1.5 s budget for a … snapshot.
  • Docs: docs/RECOVERY.md gives the budget inside the "Where the verdicts are posted" paragraph. A separate paragraph had pushed the doc past the 80-line limit in "the doc fits one printed side", which is what failed the first check.
  • Left alone: the sample reasons in test/drill-weekly.test.ts are fixed strings that are only passed through, so they keep the old wording.

🤖 Generated with Claude Code

Changes requested — before the latest push

The budget logic is sound and the fake-timer test for the 4.1 GiB hang is good. One miss against the bar: a behaviour the change adds has no test that would fail without it. Fix first: add a test where the manifest has no top-level bytes (or a non-number) and its files sum to a size that moves the budget (for example two 2 GiB files give a 14 min budget). Assert the reason names that budget and size. Optionally also cover formatBudget for the ms and min branches directly, since only the s branch is exercised today. Everything else (security floor untouched, docs updated, no new dependency, no dead code) looks right. This change touches no page or terminal door.

commented

The restore drill's time limit now grows with the snapshot. drillBudgetMs(bytes) in src/snapshot/drill.ts gives 10 min plus 60 s per GiB of the snapshot's bytes, capped at 45 min, and runDrill uses it whenever deps.timeoutMs is unset. bun x tsc --noEmit passes, and test/snapshot-drill.test.ts passes (14 tests).

  • How the budget is set: finding the newest snapshot runs under the 10 min floor, because its size isn't known yet. Once the manifest is read, the budget is recomputed from its bytes. If that total is missing or not a number, the drill uses the sum of the files' sizes. A size that isn't a finite, non-negative number counts as 0.
  • What a red drill says: the reason names the whole drill's budget and the snapshot's size, for example restore timed out: no answer within its 14.1 min budget for a 4.1 GiB snapshot. Budgets read as min, s or whole ms, and sizes in binary units with one decimal, so no raw float of milliseconds appears.
  • Budget tests: the budget is checked for 0 bytes (10 min), 4 GiB (14 min), 100 GiB (capped at 45 min) and bad sizes.
  • Hang tests: these run on Bun's fake timers.

- A hanging restore of a 4.1 GiB snapshot is still going at 14 min and goes red just past 14.1 min. - A manifest made of two 2 GiB files is drilled twice: once with no bytes, once with bytes: "4 GiB". Each time it goes red at 14 min with the reason … 14 min budget for a 4 GiB snapshot. With the summing fallback removed, this test fails.

  • Format tests: formatBudget is tested directly for the min, s and ms cases.
  • Changed test: the old "whole milliseconds" test checked the reason text this issue replaces, so the test was wrong rather than the code. It now checks that an explicit timeoutMs: 1500.75 reads as a 1.5 s budget for a … snapshot.
  • Docs: docs/RECOVERY.md gives the budget inside the "Where the verdicts are posted" paragraph, so the doc stays within its 80-line limit.
  • Left alone: the sample reasons in test/drill-weekly.test.ts are fixed strings that are only passed through, so they keep the old wording.

🤖 Generated with Claude Code

Approved

The drill budget is now 10 min + 60 s per GiB of the snapshot, capped at 45 min, on one clock. The size comes from the manifest total or the sum of its files, and a bad size counts as 0. A red reason names the budget and size in plain units. I read the diff only and did not run the tests or open the preview, so I am relying on the pull being green. There is no page, terminal or MCP surface in this change, and no input, auth or dependency changes. The tests cover the budget maths, the formatters, the fake-timer hang boundaries and the file-sum fallback. The old test was changed because it asserted the replaced wording, which is fair. The docs fit the 80-line limit. Two things I would do next: 1) a fetch-step timeout says only "10 min budget" with no size, so say in the doc that sizing starts after the manifest is read; 2) formatBytes and formatBudget are generic helpers, so move them to a small format module if another caller appears.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs