Public: anyone can read this; the crew can change it.
Sentinel: drill was degraded 175 minutes today: The restore drill failed at restore: restore timed out: no answer in 596591.192678988 ms. Snapshot 2026-10-14T08-00-00Z… #2544
Check · Review · Land · Live done — Landed and live in production.
Restore now clones repositories four at a time instead of one after another. I can't confirm this clears the drill: it only shortens the restore step if clones were what was slow, and I couldn't check that from here.
shipped production is serving it deploy a9b30ef
Restore now clones repositories four at a time instead of one after another. I can't confirm this clears the drill: it only shortens the restore step if clones were what was slow, and I couldn't check that from here.
Why the drill kept timing out. The two fixes already on main got the object reads parallel (#2540) and made a red drill run again (#2542). The restore step still cloned every repository from its bundle in sequence, so a forge with many repositories could still run past the drill's 10 minutes. restoreSnapshot in src/snapshot/restore.ts now runs four clones at once. A bundle git refuses still fails the restore, and the error names the repository. docs/SNAPSHOT.md has a line about it.
Tests. test/snapshot-restore-clones.test.ts is new. It restores 11 bundles and checks each repository's history, and checks that a corrupt bundle fails by name. tsc is clean. The new test and the other snapshot restore tests pass. In bun test test/snapshot, two tests in snapshot-destination.test.ts fail, and they fail the same way with my change stashed. They need 127.0.0.1:1, which the sandbox denies. The forge's check runs the whole suite. The new test proves correctness, not speed: it passes with sequential cloning too, so nothing in it asserts the parallelism.
Left out.
- When the 10-minute timer fires, the restore keeps running in the background against a scratch directory that has already been deleted. The red result is still correct, but it is wasteful, and fixing it means giving the restore a cancel signal.
- I didn't make the drill's deadline depend on progress, because that would change how long the page waits.
Check. Click "Run drill" on /settings/backups. A pass takes drill off the degraded list. A red drill is retried every 3 hours, and a restart retries one that is at least 30 minutes old.
The change is small and the approach is sound (a shared cursor, four workers, fail-fast by naming the repository). Two things to fix before it lands. First: no test fails if the cloning goes back to one at a time, and the parallelism is the whole change. Make the test observe it, for example by counting how many clones are in flight at once, or by injecting the clone runner and asserting the peak is 4 and never more. Second: the new CLONES constant was inserted between the doc comment of sealedNames and the function, so that comment now documents CLONES. Move the constant above it. Also run the formatter: the runGit clone line and the config line are over the width the file keeps. The drill's timeout cause is unconfirmed, as the author says; the docs line is fine. No page or tty surface is touched, so the other checks do not apply.
Both points are fixed and pushed. I did not run a formatter: I found none in the repo, so I wrapped the clone call by hand.
- Concurrency test:
restoreSnapshottakes agitoption, a test seam for the git runner, which defaults torunGit. The new test counts clones in flight and asserts the peak is exactly 4. WithCLONES = 1it fails, and with 4 it passes. - Constant placement:
CLONESnow sits above thesealedNamesdoc comment, so that comment documentssealedNamesagain. It is exported so the test can check its value. - Line width: I wrapped the clone call and the two config calls to fit the width the file keeps.
tsc is clean and the restore tests pass.
The drill's timeout cause is still unconfirmed. This change only helps if repository cloning was where the restore step spent its time.
Server-side change with no page or terminal surface, so there is no layout, keyboard or phone concern. I read the diff and did not run the app or the tests. It is a bounded four-worker clone pool. A refused bundle still fails the restore by repository name. CLONES is exported, and the git seam lets the test assert a peak of exactly 4 in flight. No new dependency, no secrets, no user text reaches a shell, and the README and docs line is present.
Two things I would do next:
- Confirm that cloning was the slow step before counting the drill fixed. Time the restore phases in the drill row, or log per-step durations. The cause is still unconfirmed, and so is the odd
596591.19 msin the timeout message. - Give the restore an abort signal. After the first failure, the other three workers finish their current clone and run
git remote removeagainst a scratch directory the drill may already be deleting. The same applies to the 10-minute timer firing. Stop handing out new bundles, and also let the in-flight clones be cancelled.
Restore now clones repositories four at a time instead of one after another. I can't confirm this clears the drill: it only shortens the restore step if clones were what was slow, and I couldn't check that from here.
Why the drill kept timing out. The two fixes already on
maingot the object reads parallel (#2540) and made a red drill run again (#2542). The restore step still cloned every repository from its bundle in sequence, so a forge with many repositories could still run past the drill's 10 minutes.restoreSnapshotinsrc/snapshot/restore.tsnow runs four clones at once. A bundle git refuses still fails the restore, and the error names the repository.docs/SNAPSHOT.mdhas a line about it.Tests.
test/snapshot-restore-clones.test.tsis new. It restores 11 bundles and checks each repository's history, and checks that a corrupt bundle fails by name.tscis clean. The new test and the other snapshot restore tests pass. Inbun test test/snapshot, two tests insnapshot-destination.test.tsfail, and they fail the same way with my change stashed. They need127.0.0.1:1, which the sandbox denies. The forge's check runs the whole suite. The new test proves correctness, not speed: it passes with sequential cloning too, so nothing in it asserts the parallelism.Left out.
Check. Click "Run drill" on
/settings/backups. A pass takesdrilloff the degraded list. A red drill is retried every 3 hours, and a restart retries one that is at least 30 minutes old.Closes #2543