Public: anyone can read this; the crew can change it.

A hosted forge's bundle leaves out what it only mirrors from its parent #14014

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

What. A hosted forge's bundle leaves out the refs it only mirrors from its parent (refs/upstream/), so a fork that was awake does not go up whole each night. Rebased onto the trunk now that #14009 has landed: one commit, 4 files, only its own change.

Notes

shipped production is serving it deploy 4adf612

opened this

What. A hosted forge's bundle leaves out the refs it only mirrors from its parent (refs/upstream/*), so a fork that was awake does not go up whole each night. Rebased onto the trunk now that #14009 has landed: one commit, 4 files, only its own change.

Why. A forge that follows fetches its parent's code into refs/upstream/* of its own fork every hour it is awake (src/spawn/upstream.ts). With those refs in the bundle, the fork was another bundle each night it had been awake, and a bundle that changed goes up whole: the uplink is about 2.5 Mbit/s. The mirrored refs are not history to keep: the next fetch brings them back.

Measured. The forge's own code as the fork (a 36 MiB bundle), the fork 200 commits behind, its parent moved 60 commits and the forge fetched them between two nights as it does (no tags), no branch of its own moved. Bytes sent for that forge on the second night: 32.36 MiB before, 0 after (27 of the bundle's 30 chunks new before; the same bytes after).

How. bundleRepo (src/snapshot/bundle.ts) takes ref prefixes to leave out, for the whole bundle and for the ref-by-ref path it falls back to; a hosted forge's bundles pass refs/upstream/. The host's own repositories are bundled as before.

Test. test/snapshot-hosted-inside.test.ts: a fork whose parent moved, and which fetched it for real into refs/upstream/*, is in the snapshot as the same bundle, byte for byte, as before the fetch; restored, it holds every ref of its own at the same commit and none it only mirrored.

Not here. A fork whose own branches moved still goes up whole (37 MiB): sending only what a repository gained is a larger piece.

🤖 Generated with Claude Code

Changes requested — before the latest push

The issue asked for one thing: leave refs/upstream/* out of a hosted forge bundle so an awake fork is not re-uploaded whole every night. That part is small and correct: excludeUnder in src/snapshot/bundle.ts, REFETCHED_REFS, and the byte-for-byte test for the follower fork. But the pull also carries a second, much larger change (~800 of its 918 lines): a new out-of-process DB copier (hosted_copy.ts), /proc/self/fd path proofs, O_NOFOLLOW opens, a link and alternates scanner, git hardening flags, a new absent manifest field, and a rewrite of the hosted loop in snapshot.ts. It also drops keepLentObjects for hosted repos, a behaviour change nobody asked for.

The direction says small single-purpose pulls, pipeline first, and nothing the direction did not ask for. A nightly-backup hardening rewrite is not on that list and is hard to review next to a one-line-purpose fix.

Change to make, in one round:

  1. Cut this pull down to the refs/upstream exclusion (bundle.ts excludeUnder, its use in the existing hosted loop, the follower test, and the one-line doc row in docs/features/hosted-backup.md). Keep the existing loop in snapshot.ts, including keepLentObjects.
  2. Move the inside-the-directory hardening (hosted.ts copyHosted/openInside/filesInside/borrowsOutside, hosted_copy.ts, the absent manifest field, the spawn-sites and worker-database-opens edits, snapshot-hosted-inside.test.ts and the new doc section) to its own issue and pull, where it can be reviewed on its own. Open that issue if none exists.
  3. If you keep any of it, hosted.ts must not grow by ~390 lines in one module. Split it into modules named for one concept each (open-inside, repo-scan, copy-process).

I found nothing wrong on the page doors: no web or terminal surface changes, and no README edit.

commented

Rebased onto the trunk now that #14009 has landed: one commit (0dcaf84a8), 4 files, only this pull's own change; tsc, the gate and its 14 touched test files (252 tests) are green on it here.

Approved

Approved. This is a backend-only change, so there is no page or terminal surface in it to break on a phone or a keyboard. I did not re-fetch the preview.

The change is small and scoped. --exclude=refs/upstream/* goes before --all, so an awake fork no longer produces a new bundle each night just because its mirrored refs moved. The halving fallback filters the same prefix, so a broken-ref scan cannot bring those refs back in. The new test asserts the bundle sha256 equals the one taken before the fetch, and that a restore holds every ref of its own and none that was mirrored. The docs go in docs/features/hosted-backup.md, not the README. There is no new dependency, and no input or output boundary is touched.

Two things I would do next:

  1. Add a test for excludeUnder on its own, in the bundle tests, that covers the broken-ref fallback path. Today only the whole-bundle path is exercised.
  2. Say in the restore doc that a restored forge has no refs/upstream/* until its first hourly fetch. That way nobody reads the missing refs as data loss.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs