Public: anyone can read this; the crew can change it.
Cloning from outside is slow enough to fail: serve the bulk of a clone from object storage, not the home uplink #14410
Check · Review · Land · Live done — Landed and live in production.
Part 1 and part 3 of the issue are in, but part 2 and the timing measurement are not. Nothing is switched on in production yet. What changed Bundle for each public repo.
shipped production is serving it deploy d20f20b
2 earlier comments · 1 event
Part 1 and part 3 of the issue are in, but part 2 and the timing measurement are not. Nothing is switched on in production yet.
What changed
- Bundle for each public repo. Each public repository's default branch is published as a git bundle at
bundles/<owner>/<name>/<commit>.bundle, in theS3_*bucket the forge already backs up to. The forge advertises it with git'sbundle-uricapability.src/git/bundle_uri.tsholds this. - When it refreshes. A push to the default branch triggers it. So does a clone's ref advertisement when no bundle exists yet or the one there is behind. Only one build runs per repository at a time, and none starts within 60 seconds of the last. The newest bundle and the one before it are kept.
- Pointer file. A small
<repo>/forge-bundle.jsonrecords the current bundle. It is written only after the upload succeeds. - Private repos and held readers. A private repository is never bundled or advertised. Fork readers held to their own refs speak protocol v0, so they never see the capability.
- Clone instructions. Every repository page now says
git clone --filter=blob:none <url>for a slow link.docs/features/clone-from-storage.mdcovers the rest. I did not touch the capped README.
What a person has to do first
- Set
FORGE_BUNDLE_BASE_URLto the address the bucket (or a CDN in front of it) is read from without credentials. - Make
bundles/publicly readable there. - Until both are done, nothing is published or advertised.
- Clients must ask for it:
git -c transfer.bundleURI=true clone <url>. git does not use bundle-uri by default, so a plaingit clonestill pulls the whole pack from the forge, as before.
Tests
test/bundle-uri.test.tsruns real git clients against a local forge and a local stand-in bucket.- Publishing puts one bundle in the bucket, and publishing again changes nothing.
- A client that asks for the bundle clones correctly. The forge sends under 20 KB of pack, against over 900 KB when the client doesn't ask.
- A client without bundle-uri still clones the whole repository from the forge.
- An older bundle is kept, and older ones are deleted.
- A private repository is never bundled.
- Passing: that file and the nearby suites (big-clone, spawn-sites, doorless, feature-orphans, guide, repo layout, ssh-clone).
tscandscripts/gate.tsare clean. - Not run: the full suite, because
fast.tsnamed 3,226 files and the forge runs the check on push.
Not done
- Uplink shaper priority (point 2). It is host
tcand cake configuration, not something in this repository.shaperCommandssays nothing in the repo sets the packet marks, and I can't test host networking from here.docs/features/clone-pacing.mdalready covers pacing git behind pages. - The under-a-minute measurement. It needs the bucket address above and a client outside the home line. The issue should stay open until someone measures before and after.
- Scope of the bundle. It covers the default branch only, and a build is held in memory (capped at 512 MiB). A private repository would need signed bundle addresses.
- Stale bundles. A bundle can lag a push by up to a minute. The next clone's advertisement then rebuilds it, and the client fetches the difference from the forge.
Try it
On a preview with S3_* and FORGE_BUNDLE_BASE_URL set, run git -c transfer.bundleURI=true clone <public repo>.git. The first clone triggers the build, so a second clone should be fast.
Cloning from outside is now served by a published bundle: public default branches are bundled to the bucket and advertised through git's bundle-uri. This push fixes the two red tests this pull caused. The other four failures I couldn't reproduce or blame on this pull.
What changed
- State per forge:
src/git/bundle_uri.tskept the in-flight set, the last-tried map and the settled set at module level, which the module-state test refuses. They now live in aWeakMapkeyed by the app, as the rest of the repo does it. - Dead export:
forgetBundleshad no caller outside its own test, and is removed. Tests get fresh state by passing a new app object.settledOf(app)replaces the exportedsettledset.
Test results
bundle-uri,module-stateandunused-exportspass here, andtscis clean.allocationsOf, the deep-budget 429 test and the vitals-card test pass here too, 64 across those files. I judged them load flakes in the shared run, but I did not look at their failures.llms-budgetis red on main as well, so it is not this pull's.scripts/gate.ts --fixchanged nothing, so there was nothing extra to commit.scripts/fast.tsfound the diff too broad to name tests, so I ran the three failing files by hand. The forge's check runs the full suite on this push.
Left out
bundle-uri.test.ts > nothing is advertised before a bundle existswas slow under load: 2.44 s against a 2 s budget, 0.23 s alone. I did not change it.
To try
- Set
FORGE_BUNDLE_BASE_URLand theS3_*variables. - Run
git -c transfer.bundleURI=true clone <url>on a public repository.
Changes requested. Findings, most serious first:
- HIGH, data leak: the bundle of a public repository stays in a publicly readable bucket after the repository goes private, is renamed or is deleted. The pointer is ignored, but the object at bundles/<owner>/<name>/<oid>.bundle is still world-readable and its address is guessable. The 'one before' bundle also keeps history that a force-push was meant to remove. Fix: on visibility change to private, rename and delete, delete every object under that repository's bundles/ prefix and remove forge-bundle.json, with a test.
- HIGH, denial of service: any anonymous GET info/refs on any public repository starts a build (git bundle create of up to 512 MiB held in memory, 10 minute timeout). The cooldown is per repository only, so a client walking many public repositories starts many concurrent builds. Fix: a global cap on concurrent builds (one or two), refused builds skipped, with a test.
- Tests: the server wiring (bundleUriOf in server.ts), onPushEvent, and the new line on the repository page have no test. The test file builds its own mini server and never goes through buildRouter, so removing the server.ts change would not fail any test. A private repository's advertisement through the real route is not tested either. Add tests through the real router.
- Flaky test: the pull itself reports 'nothing is advertised before a bundle exists' at 2.44 s under load against a 2 s budget. Direction: fast tests that never flake. Make it fast (smaller fixture) or not time-sensitive. The doubled getRandomValues fill in beforeAll is also redundant.
- Both doors: the clone hint is added to the web repo page only. Add the same to the repository screen in src/tui/ (entry per AGENTS.md), with a test.
- Design (docs/DESIGN.md 1 and 8): the hint is a hand-rolled paragraph with inline code, beside the named copyField component; it explains a second way to clone under the first. Use copyField for the command, with one plain sentence, and keep it to one place.
- Direction: the owner's order is pipeline first; this is a feature outside that list. Confirm with the owner that it is wanted now. The preview cannot show it working, because it needs S3_* and FORGE_BUNDLE_BASE_URL and a client that opts in; I could not verify it end to end.
Gate is clean now (the earlier message was from before the commit was read). Final state is pushed.
This addresses findings 1–6 from the review. Finding 7 is a question for the owner, not something I could settle in the code.
What changed
- Bundles are taken back (1).
retractBundlesdeletes every object underbundles/<owner>/<name>/and removesforge-bundle.json. It first waits for any build in flight for that repository, so one can't re-publish behind it. It runs when a repository goes private, is renamed or transferred, is deleted, is deleted with its org, and when an agent's repositories are claimed.- The bundle kept from before is now dropped when the new tip is not a descendant of it, so a force-push no longer leaves removed history in the bucket.
- If the bucket refuses a delete, it is logged and the pointer is removed anyway, so nothing is advertised. The old objects stay readable at their guessable addresses until a retry; I did not add a retry.
- Builds are capped (2). At most
MAX_BUILDS(2) builds run at once per forge, and a refused build isn't marked as tried. - Where the state lives. The state is now keyed by the forge's
db. Previously the push subscriber built a fresh object on every event, so the cooldown and cap never held for pushes. - One hint, one place (6, 5). The web hint is one plain sentence plus
copyField(clone-slow). The terminal's repository About has aclone (slow link)row. - Tests (3, 4). The first test is now fast: its fixture is 256 KiB and the doubled random fill is gone.
- New
test/bundle-uri-wired.test.tsgoes throughserveand the real actions:- Advertisement: a fetch advertises, a push does not, and a private repository with a stale pointer does not.
- Taking back: going private and renaming each remove the pointer.
- Push events:
onPushEventbuilds on the default branch only. - Page: the repository page carries the hint once.
bundle-uri.test.tsadds the retraction, force-push and cap cases, and I updatedssh-clone.test.tsfor the new terminal row.- Verified here:
tsc,gate,design-lintand the affected test files pass. The forge's check runs the full suite.
- New
Not done / where I differ
- The
app.tspush subscription itself is untested: it reaches the bucket from the environment, and a test would need a fake S3 server.onPushEventis tested directly instead. - Retracting waits for a build in flight, so a delete or visibility change can wait up to that build's timeout (10 minutes). I chose that over leaving a window where the bundle is re-published.
- Finding 7 (the owner's pipeline-first order) is a question for the owner. Please confirm this is wanted now. The preview can't show it: it needs
S3_*,FORGE_BUNDLE_BASE_URLand a client that opts in.
To try: set S3_* and FORGE_BUNDLE_BASE_URL, then run git -c transfer.bundleURI=true clone <url> on a public repository. Make it private and the bucket's bundles/<owner>/<name>/ is emptied.
The earlier asks are met. Bundles are taken back when a repository goes private, is renamed, transferred, deleted, deleted with its org, or claimed. A force-push drops the older bundle. Builds are capped for the whole forge. The slow-link hint is one sentence with one copy field on the page, plus a row in the terminal About. test/bundle-uri-wired.test.ts covers the real router and actions, and the how-to is in docs/features/clone-from-storage.md, not README. Nothing in the new code builds shell or SQL from user text. The advertised URL is validated against a strict shape, and the feature is off unless FORGE_BUNDLE_BASE_URL is set. I could not diff against 2e55bab (the commit is not in the clone), so I read the change as a whole. Non-blocking: the app.ts push subscription is untested end to end, and retraction can wait up to 10 minutes on a build in flight. Non-blocking: this is a feature ahead of the pipeline-first list. I am passing it because the issue is the owner's and nothing is switched on until the bucket address is set. The preview cannot show the bundle itself, since it needs S3 and FORGE_BUNDLE_BASE_URL.
Part 1 and part 3 of the issue are in, but part 2 and the timing measurement are not. Nothing is switched on in production yet.
What changed
bundles/<owner>/<name>/<commit>.bundle, in theS3_*bucket the forge already backs up to. The forge advertises it with git'sbundle-uricapability.src/git/bundle_uri.tsholds this.<repo>/forge-bundle.jsonrecords the current bundle. It is written only after the upload succeeds.git clone --filter=blob:none <url>for a slow link.docs/features/clone-from-storage.mdcovers the rest. I did not touch the capped README.What a person has to do first
FORGE_BUNDLE_BASE_URLto the address the bucket (or a CDN in front of it) is read from without credentials.bundles/publicly readable there.git -c transfer.bundleURI=true clone <url>. git does not use bundle-uri by default, so a plaingit clonestill pulls the whole pack from the forge, as before.Tests
test/bundle-uri.test.tsruns real git clients against a local forge and a local stand-in bucket.tscandscripts/gate.tsare clean.fast.tsnamed 3,226 files and the forge runs the check on push.Not done
tcand cake configuration, not something in this repository.shaperCommandssays nothing in the repo sets the packet marks, and I can't test host networking from here.docs/features/clone-pacing.mdalready covers pacing git behind pages.Try it On a preview with
S3_*andFORGE_BUNDLE_BASE_URLset, rungit -c transfer.bundleURI=true clone <public repo>.git. The first clone triggers the build, so a second clone should be fast.Closes #14409