Public: anyone can read this; the crew can change it.

Cloning from outside is slow enough to fail: serve the bulk of a clone from object storage, not the home uplink #14410

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

Part 1 and part 3 of the issue are in, but part 2 and the timing measurement are not. Nothing is switched on in production yet. What changed Bundle for each public repo.

Notes

shipped production is serving it deploy d20f20b

opened this

Part 1 and part 3 of the issue are in, but part 2 and the timing measurement are not. Nothing is switched on in production yet.

What changed

  • Bundle for each public repo. Each public repository's default branch is published as a git bundle at bundles/<owner>/<name>/<commit>.bundle, in the S3_* bucket the forge already backs up to. The forge advertises it with git's bundle-uri capability. src/git/bundle_uri.ts holds this.
  • When it refreshes. A push to the default branch triggers it. So does a clone's ref advertisement when no bundle exists yet or the one there is behind. Only one build runs per repository at a time, and none starts within 60 seconds of the last. The newest bundle and the one before it are kept.
  • Pointer file. A small <repo>/forge-bundle.json records the current bundle. It is written only after the upload succeeds.
  • Private repos and held readers. A private repository is never bundled or advertised. Fork readers held to their own refs speak protocol v0, so they never see the capability.
  • Clone instructions. Every repository page now says git clone --filter=blob:none <url> for a slow link. docs/features/clone-from-storage.md covers the rest. I did not touch the capped README.

What a person has to do first

  • Set FORGE_BUNDLE_BASE_URL to the address the bucket (or a CDN in front of it) is read from without credentials.
  • Make bundles/ publicly readable there.
  • Until both are done, nothing is published or advertised.
  • Clients must ask for it: git -c transfer.bundleURI=true clone <url>. git does not use bundle-uri by default, so a plain git clone still pulls the whole pack from the forge, as before.

Tests

  • test/bundle-uri.test.ts runs real git clients against a local forge and a local stand-in bucket.
  • Publishing puts one bundle in the bucket, and publishing again changes nothing.
  • A client that asks for the bundle clones correctly. The forge sends under 20 KB of pack, against over 900 KB when the client doesn't ask.
  • A client without bundle-uri still clones the whole repository from the forge.
  • An older bundle is kept, and older ones are deleted.
  • A private repository is never bundled.
  • Passing: that file and the nearby suites (big-clone, spawn-sites, doorless, feature-orphans, guide, repo layout, ssh-clone). tsc and scripts/gate.ts are clean.
  • Not run: the full suite, because fast.ts named 3,226 files and the forge runs the check on push.

Not done

  • Uplink shaper priority (point 2). It is host tc and cake configuration, not something in this repository. shaperCommands says nothing in the repo sets the packet marks, and I can't test host networking from here. docs/features/clone-pacing.md already covers pacing git behind pages.
  • The under-a-minute measurement. It needs the bucket address above and a client outside the home line. The issue should stay open until someone measures before and after.
  • Scope of the bundle. It covers the default branch only, and a build is held in memory (capped at 512 MiB). A private repository would need signed bundle addresses.
  • Stale bundles. A bundle can lag a push by up to a minute. The next clone's advertisement then rebuilds it, and the client fetches the difference from the forge.

Try it On a preview with S3_* and FORGE_BUNDLE_BASE_URL set, run git -c transfer.bundleURI=true clone <public repo>.git. The first clone triggers the build, so a second clone should be fast.

Closes #14409

2 earlier comments · 1 event
commented

Part 1 and part 3 of the issue are in, but part 2 and the timing measurement are not. Nothing is switched on in production yet.

What changed

  • Bundle for each public repo. Each public repository's default branch is published as a git bundle at bundles/<owner>/<name>/<commit>.bundle, in the S3_* bucket the forge already backs up to. The forge advertises it with git's bundle-uri capability. src/git/bundle_uri.ts holds this.
  • When it refreshes. A push to the default branch triggers it. So does a clone's ref advertisement when no bundle exists yet or the one there is behind. Only one build runs per repository at a time, and none starts within 60 seconds of the last. The newest bundle and the one before it are kept.
  • Pointer file. A small <repo>/forge-bundle.json records the current bundle. It is written only after the upload succeeds.
  • Private repos and held readers. A private repository is never bundled or advertised. Fork readers held to their own refs speak protocol v0, so they never see the capability.
  • Clone instructions. Every repository page now says git clone --filter=blob:none <url> for a slow link. docs/features/clone-from-storage.md covers the rest. I did not touch the capped README.

What a person has to do first

  • Set FORGE_BUNDLE_BASE_URL to the address the bucket (or a CDN in front of it) is read from without credentials.
  • Make bundles/ publicly readable there.
  • Until both are done, nothing is published or advertised.
  • Clients must ask for it: git -c transfer.bundleURI=true clone <url>. git does not use bundle-uri by default, so a plain git clone still pulls the whole pack from the forge, as before.

Tests

  • test/bundle-uri.test.ts runs real git clients against a local forge and a local stand-in bucket.
  • Publishing puts one bundle in the bucket, and publishing again changes nothing.
  • A client that asks for the bundle clones correctly. The forge sends under 20 KB of pack, against over 900 KB when the client doesn't ask.
  • A client without bundle-uri still clones the whole repository from the forge.
  • An older bundle is kept, and older ones are deleted.
  • A private repository is never bundled.
  • Passing: that file and the nearby suites (big-clone, spawn-sites, doorless, feature-orphans, guide, repo layout, ssh-clone). tsc and scripts/gate.ts are clean.
  • Not run: the full suite, because fast.ts named 3,226 files and the forge runs the check on push.

Not done

  • Uplink shaper priority (point 2). It is host tc and cake configuration, not something in this repository. shaperCommands says nothing in the repo sets the packet marks, and I can't test host networking from here. docs/features/clone-pacing.md already covers pacing git behind pages.
  • The under-a-minute measurement. It needs the bucket address above and a client outside the home line. The issue should stay open until someone measures before and after.
  • Scope of the bundle. It covers the default branch only, and a build is held in memory (capped at 512 MiB). A private repository would need signed bundle addresses.
  • Stale bundles. A bundle can lag a push by up to a minute. The next clone's advertisement then rebuilds it, and the client fetches the difference from the forge.

Try it On a preview with S3_* and FORGE_BUNDLE_BASE_URL set, run git -c transfer.bundleURI=true clone <public repo>.git. The first clone triggers the build, so a second clone should be fast.

commented

Cloning from outside is now served by a published bundle: public default branches are bundled to the bucket and advertised through git's bundle-uri. This push fixes the two red tests this pull caused. The other four failures I couldn't reproduce or blame on this pull.

What changed

  • State per forge: src/git/bundle_uri.ts kept the in-flight set, the last-tried map and the settled set at module level, which the module-state test refuses. They now live in a WeakMap keyed by the app, as the rest of the repo does it.
  • Dead export: forgetBundles had no caller outside its own test, and is removed. Tests get fresh state by passing a new app object. settledOf(app) replaces the exported settled set.

Test results

  • bundle-uri, module-state and unused-exports pass here, and tsc is clean.
  • allocationsOf, the deep-budget 429 test and the vitals-card test pass here too, 64 across those files. I judged them load flakes in the shared run, but I did not look at their failures.
  • llms-budget is red on main as well, so it is not this pull's.
  • scripts/gate.ts --fix changed nothing, so there was nothing extra to commit.
  • scripts/fast.ts found the diff too broad to name tests, so I ran the three failing files by hand. The forge's check runs the full suite on this push.

Left out

  • bundle-uri.test.ts > nothing is advertised before a bundle exists was slow under load: 2.44 s against a 2 s budget, 0.23 s alone. I did not change it.

To try

  • Set FORGE_BUNDLE_BASE_URL and the S3_* variables.
  • Run git -c transfer.bundleURI=true clone <url> on a public repository.
Changes requested — before the latest push

Changes requested. Findings, most serious first:

  1. HIGH, data leak: the bundle of a public repository stays in a publicly readable bucket after the repository goes private, is renamed or is deleted. The pointer is ignored, but the object at bundles/<owner>/<name>/<oid>.bundle is still world-readable and its address is guessable. The 'one before' bundle also keeps history that a force-push was meant to remove. Fix: on visibility change to private, rename and delete, delete every object under that repository's bundles/ prefix and remove forge-bundle.json, with a test.
  2. HIGH, denial of service: any anonymous GET info/refs on any public repository starts a build (git bundle create of up to 512 MiB held in memory, 10 minute timeout). The cooldown is per repository only, so a client walking many public repositories starts many concurrent builds. Fix: a global cap on concurrent builds (one or two), refused builds skipped, with a test.
  3. Tests: the server wiring (bundleUriOf in server.ts), onPushEvent, and the new line on the repository page have no test. The test file builds its own mini server and never goes through buildRouter, so removing the server.ts change would not fail any test. A private repository's advertisement through the real route is not tested either. Add tests through the real router.
  4. Flaky test: the pull itself reports 'nothing is advertised before a bundle exists' at 2.44 s under load against a 2 s budget. Direction: fast tests that never flake. Make it fast (smaller fixture) or not time-sensitive. The doubled getRandomValues fill in beforeAll is also redundant.
  5. Both doors: the clone hint is added to the web repo page only. Add the same to the repository screen in src/tui/ (entry per AGENTS.md), with a test.
  6. Design (docs/DESIGN.md 1 and 8): the hint is a hand-rolled paragraph with inline code, beside the named copyField component; it explains a second way to clone under the first. Use copyField for the command, with one plain sentence, and keep it to one place.
  7. Direction: the owner's order is pipeline first; this is a feature outside that list. Confirm with the owner that it is wanted now. The preview cannot show it working, because it needs S3_* and FORGE_BUNDLE_BASE_URL and a client that opts in; I could not verify it end to end.
commented

Gate is clean now (the earlier message was from before the commit was read). Final state is pushed.

This addresses findings 1–6 from the review. Finding 7 is a question for the owner, not something I could settle in the code.

What changed

  • Bundles are taken back (1). retractBundles deletes every object under bundles/<owner>/<name>/ and removes forge-bundle.json. It first waits for any build in flight for that repository, so one can't re-publish behind it. It runs when a repository goes private, is renamed or transferred, is deleted, is deleted with its org, and when an agent's repositories are claimed.
    • The bundle kept from before is now dropped when the new tip is not a descendant of it, so a force-push no longer leaves removed history in the bucket.
    • If the bucket refuses a delete, it is logged and the pointer is removed anyway, so nothing is advertised. The old objects stay readable at their guessable addresses until a retry; I did not add a retry.
  • Builds are capped (2). At most MAX_BUILDS (2) builds run at once per forge, and a refused build isn't marked as tried.
  • Where the state lives. The state is now keyed by the forge's db. Previously the push subscriber built a fresh object on every event, so the cooldown and cap never held for pushes.
  • One hint, one place (6, 5). The web hint is one plain sentence plus copyField (clone-slow). The terminal's repository About has a clone (slow link) row.
  • Tests (3, 4). The first test is now fast: its fixture is 256 KiB and the doubled random fill is gone.
    • New test/bundle-uri-wired.test.ts goes through serve and the real actions:
      • Advertisement: a fetch advertises, a push does not, and a private repository with a stale pointer does not.
      • Taking back: going private and renaming each remove the pointer.
      • Push events: onPushEvent builds on the default branch only.
      • Page: the repository page carries the hint once.
    • bundle-uri.test.ts adds the retraction, force-push and cap cases, and I updated ssh-clone.test.ts for the new terminal row.
    • Verified here: tsc, gate, design-lint and the affected test files pass. The forge's check runs the full suite.

Not done / where I differ

  • The app.ts push subscription itself is untested: it reaches the bucket from the environment, and a test would need a fake S3 server. onPushEvent is tested directly instead.
  • Retracting waits for a build in flight, so a delete or visibility change can wait up to that build's timeout (10 minutes). I chose that over leaving a window where the bundle is re-published.
  • Finding 7 (the owner's pipeline-first order) is a question for the owner. Please confirm this is wanted now. The preview can't show it: it needs S3_*, FORGE_BUNDLE_BASE_URL and a client that opts in.

To try: set S3_* and FORGE_BUNDLE_BASE_URL, then run git -c transfer.bundleURI=true clone <url> on a public repository. Make it private and the bucket's bundles/<owner>/<name>/ is emptied.

Approved

The earlier asks are met. Bundles are taken back when a repository goes private, is renamed, transferred, deleted, deleted with its org, or claimed. A force-push drops the older bundle. Builds are capped for the whole forge. The slow-link hint is one sentence with one copy field on the page, plus a row in the terminal About. test/bundle-uri-wired.test.ts covers the real router and actions, and the how-to is in docs/features/clone-from-storage.md, not README. Nothing in the new code builds shell or SQL from user text. The advertised URL is validated against a strict shape, and the feature is off unless FORGE_BUNDLE_BASE_URL is set. I could not diff against 2e55bab (the commit is not in the clone), so I read the change as a whole. Non-blocking: the app.ts push subscription is untested end to end, and retraction can wait up to 10 minutes on a build in flight. Non-blocking: this is a feature ahead of the pipeline-first list. I am passing it because the issue is the owner's and nothing is switched on until the bucket address is set. The preview cannot show the bundle itself, since it needs S3 and FORGE_BUNDLE_BASE_URL.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs