A restore drill script restores a snapshot onto the forge box from the bucket alone and compares counts with production #9058

scripts/restore-drill.ts <empty-data-dir> [--production URL] restores the newest snapshot using only what the restore card names: the S3_* variables (or FORGE_SNAPSHOT_DIR) and FORGE_SNAPSHOT_PASSPHRASE. It then boots the restored forge on a free side port, checks that / answers below 400, and counts the repositories and apps anyone can see on it. It prints those beside production's counts from the same anonymous /api/v1/repos and /api/v1/apps calls (default https://forge.open-platform.sh). It exits 1 if any step fails or any count differs.

The script writes only into the directory it is given: the restore already refuses a directory that isn't empty, and the script also refuses /data. It touches nothing else of the platform. docs/RESTORE.md describes it under "The drill on the box".

What to look at:

• Booting the restored forge: bootRestored (src/snapshot/boot.ts) takes an optional whileUp(base) callback. It runs once the restored forge reads healthy and before it is killed, and anything it throws becomes the reason the boot failed. The existing callers are unchanged.
• Public counts: src/snapshot/public_counts.ts counts what an anonymous visitor sees, following next_cursor. Production's public API only shows public rows, so both sides are counted that way.
• The preview seed: the restored forge boots with FORGE_APP_ENV=preview, which seeds a preview-admin/sample repository on top of the restored data. That made about one test run in four red, and on the box it would have put the restored side one repository ahead of production. The restored side now leaves out rows the preview admin owns.

Tests: test/restore-drill-script.test.ts takes a tiny snapshot (one public and one private repository) and serves it from a stub S3 bucket (path-style HEAD, GET and ListObjectsV2). It runs the script as a subprocess against a stub production and checks the green output. Unit tests cover the argument checks, the comparison lines, and following the cursor while leaving out the preview admin's rows.

bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to name tests (exit 2), so I ran the ones that cover it: restore-drill-script, restore-card, drill-verdict and release-boot. They passed three runs in a row (30 tests).

Not done here:

• The issue's last box, posting the output of one run on the box, needs the box and its bucket credentials; whoever runs it posts that.
• I couldn't call keep_plan because it isn't available in this run, so the issue has no plan.
• The restored copy is left in place, since it is the box's own.
• A real box restoring last night's snapshot will show a red difference for anything made on production since. The issue asks for that comparison to fail when counts differ, so it does.

To try it on the box, as the forge user with the card's S3_* lines and the passphrase exported:

bun scripts/restore-drill.ts /var/lib/forge-drill

🤖 Generated with Claude Code

Closes #9046

scripts/restore-drill.ts <empty-data-dir> [--production URL] restores the newest snapshot using only what the restore card names: the S3_* variables (or FORGE_SNAPSHOT_DIR) and FORGE_SNAPSHOT_PASSPHRASE. It then boots the restored forge on a free side port, checks that / answers below 400, and counts the repositories and apps anyone can see on it. It prints those beside production's counts from the same anonymous /api/v1/repos and /api/v1/apps calls (default https://forge.open-platform.sh). It exits 1 if any step fails or any count differs.

The script writes only into the directory it is given: the restore already refuses a directory that isn't empty, and the script also refuses /data. It touches nothing else of the platform. docs/RESTORE.md describes it under "The drill on the box".

What to look at:

• Booting the restored forge: bootRestored (src/snapshot/boot.ts) takes an optional whileUp(base) callback. It runs once the restored forge reads healthy and before it is killed, and anything it throws becomes the reason the boot failed. The existing callers are unchanged.
• Public counts: src/snapshot/public_counts.ts counts what an anonymous visitor sees, following next_cursor. Production's public API only shows public rows, so both sides are counted that way.
• The preview seed: the restored forge boots with FORGE_APP_ENV=preview, which seeds a preview-admin/sample repository on top of the restored data. That made about one test run in four red, and on the box it would have put the restored side one repository ahead of production. The restored side now leaves out rows the preview admin owns.

Tests: test/restore-drill-script.test.ts takes a tiny snapshot (one public and one private repository) and serves it from a stub S3 bucket (path-style HEAD, GET and ListObjectsV2). It runs the script as a subprocess against a stub production and checks the green output. Unit tests cover the argument checks, the comparison lines, and following the cursor while leaving out the preview admin's rows.

bun scripts/gate.ts and bun x tsc --noEmit pass. scripts/fast.ts said the diff was too broad to name tests (exit 2), so I ran the ones that cover it: restore-drill-script, restore-card, drill-verdict and release-boot. They passed three runs in a row (30 tests).

Not done here:

• The issue's last box, posting the output of one run on the box, needs the box and its bucket credentials; whoever runs it posts that.
• I couldn't call keep_plan because it isn't available in this run, so the issue has no plan.
• The restored copy is left in place, since it is the box's own.
• A real box restoring last night's snapshot will show a red difference for anything made on production since. The issue asks for that comparison to fail when counts differ, so it does.

To try it on the box, as the forge user with the card's S3_* lines and the passphrase exported:

bun scripts/restore-drill.ts /var/lib/forge-drill

🤖 Generated with Claude Code

Left #9046 open: 1 unticked item remain (Its output from one run on the box is posted on this issue by whoever runs it.).