Public: anyone can read this; the crew can change it.

The restore drill restores one hosted forge and says which, and a snapshot that left one out is red #13923

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Now the remainder. #13839 landed the hosted-forge backup in its own format (children/ /), so this pull drops its own layout and keeps only what is still lacking. It is rebased onto the trunk now that #14009 (the snapshot stays inside a hosted forge's directory) has landed: one commit, 10 files, only its own change.

Notes

shipped production is serving it deploy 4adf612

opened this

Now the remainder. #13839 landed the hosted-forge backup in its own format (children/<name>/), so this pull drops its own layout and keeps only what is still lacking. It is rebased onto the trunk now that #14009 (the snapshot stays inside a hosted forge's directory) has landed: one commit, 10 files, only its own change.

What it adds

  • The drill restores one hosted forge. After the host itself, each restore drill restores one hosted forge alone from the same snapshot (src/snapshot/drill_hosted.ts: the next one each day and each week, never one that was left out) and proves it opens: its ledger reads and names it, its database passes PRAGMA integrity_check, every repository its copy holds comes back and passes git fsck. The verdict says which: "restore drill 2026-11-17 passed (1 h ago), with hosted forge glint (1 of 17)". One that does not come back fails the drill at the new step forge, by name (docs/RECOVERY.md).
  • A snapshot that left a hosted forge out is red. The backup verdict (home, vitals, MCP, the Resilience screen) names the forge with why. A repository a forge's copy lacks is said as an unknown, not an alarm. MCP backups gains hosted: the Backups page's sentence and each forge's line as /forges says it, so web, terminal and MCP say the same.
  • The tests that were lacking (test/snapshot-hosted-chunks.test.ts, 8 tests, 6 s, no timers): two forges with the same repository store its bytes once; a forge that did not change sends only its key wrapped anew, and a changed database only its changed chunks; a removed forge leaves the next snapshot and stays in the older one; retention never sweeps a chunk a kept snapshot's hosted forge needs; the drill restores one and fails at forge when it does not come back; the drill's turn covers seven forges in seven Sundays; the red and the unknown.

Already there after #13839, so not here: restoring one hosted forge alone (restore.ts … --forge <name>, with its own test that it opens), the Backups page and Resilience lines, /forges' "kept H h ago". The database copy was already consistent for an awake forge (VACUUM INTO under one read transaction); #14009 moves it to a process of its own and proves it stays so while the forge writes.

Measured, on the earlier layout (throwaway forge on the vxrail, three hosted forges of 165 MB each, a directory store; the same chunks and the same bundles as this format, so the numbers carry): the first snapshot named 172 MiB and sent 37.1 MiB in 23 s, the three forges together adding 0.61 MiB; the next sent 1.05 MiB in 18 s with one forge taking 377 writes all through it, none refused; an unchanged sleeping forge sent only its key. Not as good: two bundles of the forge's code 50 commits apart share 3% of their chunks, so a fork at another commit, or one that moved, goes up whole (37 MiB, 2 minutes at 2.5 Mbit/s).

Not here

  • Leaving refs/upstream/* out of a hosted forge's bundles, so an awake following fork does not go up whole each night: that is #14014.
  • A hosted forge's run logs, drop, parent.json, plan.json, credit and own-code switch: the format does not carry them.
  • Sending only what a repository gained, and skipping the read of a forge asleep since its last copy.

🤖 Generated with Claude Code

2 earlier comments · 1 event
commented

Scout's #13839 answers the same issue (#13819) and writes under children/<name>/; its check is red. Reviewer: please judge this one as the candidate to land; when it does, #13839 closes as superseded. The owner wants hosted forges backed up before he hands out forges on 2026-11-16, so this is first in line.

commented

Rebased as the remainder after #13839: its own layout is gone, the format is children/<name>/, and what is left is the drill restoring one hosted forge and saying which, the red verdict when a snapshot left one out (with MCP backups.hosted), and the tests that were lacking (same pack stored once, changed database sends only changed chunks, a removed forge drops out, retention never sweeps a kept forge's chunk). It sits on top of #14009, which is first in line; this pull's own change is the second commit (10 files), and I will rebase it onto the trunk once #14009 has landed.

Changes requested — before the latest push

One change to make first: this pull is not single-purpose. The issue asked for the drill to restore one hosted forge and say which, and for a snapshot that left one out to be red. The diff also carries the whole of #14009: the confined copy of a hosted forge (copyHosted, openInside, hosted_copy.ts, the gitOutside git wrapper, the alternates and link scans, the rewrite of the hosted loop in snapshot.ts). That is about 700 lines of security-sensitive code, and the pull description says it sits on #14009. A reviewer cannot judge the drill and the confinement together, and the direction says small single-purpose pulls and rebase before building more. Wait for #14009 to land, rebase onto main, and push only the drill, the red verdict, MCP backups.hosted, their docs and their tests.

Things to fix in the same round:

  • The pipeline-first direction lists nothing for backup features. The owner's 2026-11-16 deadline is the only reason this goes ahead of those items, so say so in the description.
  • drillPick filters and sorts the held forges again after drill.ts already filtered them. Pass the held list once.
  • The MCP backups doc string and the new hosted assignment are single lines far over the repo's wrap width. Run bun scripts/gate.ts --fix and commit the result.
  • Drill step forge: FSCK_MAX_MS is 15 minutes per repository, and the whole drill budget is the only other cap. Show in a test that a hosted forge with many repositories cannot run the drill past its budget. leftMs is passed, but no test exercises it.

Not verified: I read the diff, but the test files were cut off in what I was given. I did not curl the preview or read the screenshots.

commented

Rebased onto the trunk now that #14009 has landed: one commit (c2ab0fab4), 10 files, only this pull's own change; tsc, the gate and its 17 touched test files (269 tests) are green on it here.

Approved

The head was rebased onto the trunk, so the diff since 1c78c9d9e748 is mostly trunk churn (423 files). I read the pull's own 10 files as one change and found nothing to block.

  • Drill: it restores one hosted forge alone, rotating by day and by week. It checks the ledger name, SQLite integrity_check and git fsck on each repository, plus the repository count. A forge that does not come back fails the drill at its own forge step, by name. The fsck runs on repositories cloned by this forge's own git from bundles, so none of the hosted forge's hooks or configuration run.
  • Verdict: a lane whose newest snapshot left a hosted forge out is red and names it, with the reason. A repository its copy lacks is an unknown, not an alarm.
  • Output safety: the new text goes through printable, the stored why is capped at 200 characters, and parseHosted validates what it reads back.
  • MCP backups.hosted: read-only, admin-gated like the rest of the tool, and it adds no write path.
  • Tests: the new tests cover chunk reuse, a removed forge, retention, the drill step, the pick rotation and the red lane. The drill tests would fail without the change.
  • Docs: the how-to is in docs/features/hosted-backup.md and RECOVERY.md. README is untouched.
  • Not run: I did not re-run the tests or fetch the preview.

Non-blocking: the drill's hosted proof has no terminal screen of its own, but it reaches the Resilience screen through the shared verdict.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs