Large files never matter: content in a bucket, refs in the database, stock git everywhere #4481

Owner, 2026-10-21: "large files … I don't want that to be an issue … the forge is set up for everything to be bucketable … S3 or Cloudflare or both to get solid use out of free tiers … cold storage … any app that needs to run is just a lambda … I want our simple setup running super efficiently and scalably first." The research team (3 researchers: prior art, this codebase, provider economics) reported 2026-10-21. This issue is the design and the order of work.

What the prior art actually does

• Epic's Lore (MIT, Rust, pre-1.0; open-sourced 2026-06-17, github.com/EpicGames/lore). It is not a git rewrite: it is a separate, centralized VCS with no git compatibility.

- Content is split into chunks (FastCDC or fixed-size, about 64 KiB average), addressed by BLAKE3 hash in a Merkle tree. One immutable store holds everything. - A small mutable store holds only branch heads, updated by compare-and-swap. - Clients fetch pieces on demand (byte ranges, "views" of paths); the server keeps hot, warm and cold tiers. - Whether its server uses S3 is unconfirmed (it has a lore-aws crate).

• Cursor's "Git at any scale" (cursor.com/blog/git-at-any-scale, 2026-08-18; Origin on top of Continuity):

- Every push is one object in a write-ahead log in S3-compatible storage. It becomes visible when an index pointer is updated by compare-and-swap. - Each host's ordinary git repo on NVMe is only a cache, rebuildable from the bucket and evicted when idle. - Rendezvous hashing picks the primary, with no routing database. Only the primary compacts; replicas download the compacted packs. - It does not address large files: blobs stay in packs.

• Steal-worthy: git partial clone and promisor remotes; bundle-URI (clone from static packs on a CDN); Hugging Face Xet (content-defined chunking dedup on S3 behind an LFS-compatible bridge); the Git LFS locking API for unmergeable binaries.

Principles for the forge (stock git clients keep working, every one)

1. Refs and metadata are small and mutable; content is immutable, content-addressed, and lives in a bucket. Refs stay in SQLite or Postgres, with compare-and-swap through transactions.
2. The bucket is durable; local disk is a cache. A repo on the node can be rebuilt from the bucket and evicted when idle (the Continuity pattern). Always store packs, never loose objects: loose objects multiply request costs and latency.
3. Big blobs never ride in the clone. Partial clone with a promisor fetches them on demand. Later, presigned bucket URLs serve big blobs straight from R2.
4. Clones are served from static bundles (bundle-URI on R2 and the CDN), so the forge's CPU and uplink stop being the limit.
5. Uploads stream to the bucket (multipart), and the OID is verified on completion. Nothing is buffered whole in memory.
6. LFS is a compatibility door, not the design. It uses the same bucket, presigned URLs and the locking API, for CAD tools that expect it.
7. Chunked dedup (CDC) at the storage layer only. It is invisible on the wire.

Where the forge is today (file:line, from the code)

• Repos: bare repos at <dataDir>/repos (src/app.ts:221); smart HTTP pipes git upload-pack/receive-pack --stateless-rpc (src/git/smart-http.ts:129). There is no scheduled gc or repack.
• Database: SQLite at <dataDir>/forge.db.
• Already bucket-shaped: snapshots use a Store interface with s3Store (Bun.S3Client, any endpoint, so R2 should work) and dirStore (src/snapshot/store.ts:12-30, :183, :242). It is the only bucket code, and it is local to snapshots.
• Logs: run logs are append-and-tail files (src/runs/log.ts:21-24).
• App data: directories, including app SQLite (src/apps/data.ts:58-70).

Order of work (each is its own issue, crew-sized)

1. #4482: clones that never die, plus partial clone. Unblocks the arm's CAD today with stock git.
2. One BlobStore. Promote src/snapshot/store.ts to a shared module: streaming put and get, range reads, presigned GET and PUT, a storage-class hint, and fs plus S3-compatible implementations (R2, S3 and B2 all speak S3). Snapshots move onto it unchanged; finished run logs and attachments archive to it.
3. Big blobs from the bucket. At push time, blobs over a policy size (default 10 MB) are also written to the BlobStore by OID. Lazy fetches of them answer with presigned R2 URLs. (Investigate whether git's packfile-uris or a promisor remote on HTTPS can carry them; otherwise the forge proxies them.) Add a per-repo "large files" panel showing sizes and where they live.
4. The repo is durable in the bucket. Each push writes the new pack to the bucket before it is acknowledged; refs stay in the database with compare-and-swap. A node can rebuild any repo from the bucket, and idle repos can be evicted. Scheduled repack and compaction only on the repo's primary. This is what makes a second forge node trivial.
5. Clones from bundles. A nightly bundle per active repo on R2 is advertised via bundle-URI.
6. LFS door. The batch API and locks on the same BlobStore.
7. Cold tier. Lifecycle rules: R2 Standard for hot (free egress), R2 Infrequent Access after 30 days for old packs, S3 Glacier Deep Archive (about $1 per TB-month) for backups only.

Providers (prices read 2026-10-21; see the research notes)

• R2: $0.015/GB-month, free egress. The free tier is 10 GB, 1M Class A and 10M Class B ops per month. Infrequent Access is $0.01/GB-month (30-day minimum).
• S3: Standard about $0.023/GB-month, egress about $0.09/GB, Deep Archive about $0.001/GB-month (180-day minimum). New accounts get $200 in credits now, not a permanent free tier.
• B2: $6.95/TB-month, free egress through Cloudflare.
• The sketch: at 50 GB and 1 TB egress a month, R2 plus Workers is about $6/month. S3 with CloudFront egress can reach about $95.

Recommendation: R2 is the hot backbone; S3 Deep Archive or B2 holds cold backups. Refs stay in our database.

Apps as lambdas: what breaks (decision for the owner, not built here)

Stateless HTTP apps fit Workers or Lambda. Three things don't:

• Long-lived WebSockets. These need Durable Objects with hibernation.
• Private devices. The arm talks to a Pi on the tailnet, and Workers can't reach 100.x without Cloudflare Tunnel or Workers VPC.
• Runtime. Our apps are Bun, and Workers' runtime (workerd) is not Bun.

So cells stay the runtime for now. A "stateless app" profile, built on Workers, is a later option once apps declare that they are stateless.

Parked by the crew: the run ended without a pull: declined: #4481 is a design and ordering issue, not a change to make. Step 1 already exists as #4482, so I didn't touch the code or push anything. Instead I split steps 2–7 into one-run issues and linked them i

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.