trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns11g rEventsFlows

Big repos always clone: transfers end on stalled progress, not a 5-minute clock, and partial clone works #4501

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/4482-big-repos-always-clone-transfers-end-on → main 2 commits fork 1 thread · 1 unresolved

  • Merged. mason/4482-big-repos-always-clone-transfers-end-on was merged into main as ef526c6.

Review

Sign in to review this change.
9 files changed +820 −34
README.md +4
@@ -299,6 +299,10 @@ ssh -p 2222 localhost # the terminal UI
299299 git clone ssh://localhost:2222/you/hello.git # git, same port
300300 ```
301301
302 A big repository clones over either transport. To skip its large files until a
303 checkout needs them, run `git clone --filter=blob:none <url>`. See
304 [`docs/GIT.md`](docs/GIT.md).
305
302306 **After a GitHub migration** you choose what happens to the original: leave it
303307 alone (the default), archive it, or keep it as a read-only mirror. The choice is
304308 the "Afterwards" field on `/migrate`, in the terminal's migrate screen, and
docs/GIT.md added +53
@@ -0,0 +1,53 @@
1 # Cloning over git
2
3 The forge speaks git's smart HTTP (`https://<forge>/<owner>/<name>.git`) and
4 git over SSH (`ssh://<forge>:<port>/<owner>/<name>.git`), protocol v0 and v2,
5 with stock git and no extension. `src/git/smart-http.ts` and
6 `src/git/smart-ssh.ts` spawn `git upload-pack` and `git receive-pack` and pipe
7 bytes. Who may read or write is decided before git starts (`canRead`,
8 `canWrite`), the same way on both transports.
9
10 ## A big repository
11
12 A repository with large files (CAD models, datasets, media) clones in full:
13
14 ```sh
15 git clone https://<forge>/<owner>/<name>.git
16 ```
17
18 To skip the big files up front, ask for a partial clone. The commits and
19 trees arrive at once. A file's contents are fetched the first time git needs
20 them: at checkout, and later when you switch to a commit that has a file
21 you have not fetched yet.
22
23 ```sh
24 git clone --filter=blob:none https://<forge>/<owner>/<name>.git # no file contents until needed
25 git clone --filter=blob:limit=1m https://<forge>/<owner>/<name>.git # files over 1 MiB until needed
26 ```
27
28 Add `--no-checkout` to fetch nothing but history. `git checkout <branch>`
29 then fetches only that branch's files. The same flags work over SSH.
30
31 The forge runs `upload-pack` with `uploadpack.allowFilter` and
32 `uploadpack.allowAnySHA1InWant` (as `-c` options, so server-side git still
33 reads no global or system config). A later fetch of a missing file is an
34 ordinary upload-pack request for an object id, and it passes the same read
35 check as a clone. Without read access to the repository you cannot fetch any
36 of its objects, by name or by id (`test/big-clone.test.ts`,
37 `test/git-ssh.test.ts`).
38
39 ## When a clone ends
40
41 A clone ends when it stops moving, not on a clock (`src/git/progress.ts`). If
42 fewer than 16 KiB reach the client in 30 seconds, the forge ends the
43 transfer: git is killed and the response is cut, so a client that stopped
44 reading frees both its transfer permit and what the forge had buffered for
45 it. A clone that keeps moving runs for up to two hours. Git's own keep-alive,
46 sent every 5 seconds while it is still preparing the pack, counts as moving.
47 A push ends after ten minutes. Once its pack has arrived, the forge runs
48 index-pack and the hooks with nothing to send, and that quiet is not a stall.
49
50 Signed-in clones and anonymous ones end by the same rule. Anonymous ones
51 still take a share of the transfer pool first (`src/http/strangers.ts`: four
52 at once per address, half the pool for every stranger together), and a
53 negotiation whose request body stops arriving for 30 seconds still ends.
src/git/progress.ts added +226
@@ -0,0 +1,226 @@
1 /**
2 * When a clone ends: on a lack of progress, not on a clock (#4482).
3 *
4 * A wall-clock kill is wrong both ways. It cuts off a large pack moving
5 * steadily over a slow link, which is exactly the clone that most needs to
6 * finish, and it lets a client that has stopped reading keep its transfer
7 * permit until the clock runs out. So a fetch is judged by what it moved:
8 * if fewer than `minBytes` went through in the last `windowMs`, it has
9 * stalled and is ended. A generous absolute cap still bounds the worst case.
10 *
11 * One exception reads the stream, and only five bytes of it. While
12 * pack-objects is still counting and compressing, upload-pack has no data
13 * to send. Every `uploadpack.keepAlive` seconds it sends an empty side-band
14 * packet (`0005\x01`) instead. That packet only reaches the meter once the
15 * client has read what came before it, so it shows that git is working and
16 * the client is reading. It counts as progress.
17 */
18
19 /** What a fetch must move to count as moving. */
20 export interface Stall {
21 /** The window progress is judged over. */
22 readonly windowMs: number;
23 /** Fewer bytes than this through the window, and the transfer has stalled. */
24 readonly minBytes: number;
25 }
26
27 /** Fewer than 16 KiB in 30 s: stalled. */
28 export const STALL: Stall = { windowMs: 30_000, minBytes: 16 * 1024 };
29
30 /** However well it moves, a fetch ends after two hours. */
31 export const TRANSFER_CAP_MS = 2 * 60 * 60_000;
32
33 /** The window is kept in this many slices, so a stall is caught at most a sixth of a window late. */
34 const SLICES = 6;
35
36 /** Git's side-band keep-alive: an empty packet on band 1. */
37 const KEEPALIVE = [0x30, 0x30, 0x30, 0x35, 0x01];
38
39 export interface Progress {
40 /** Bytes that moved, in either direction. */
41 moved(bytes: number): void;
42 /** Stop watching: the transfer ended. Idempotent. */
43 stop(): void;
44 }
45
46 /**
47 * Watch a transfer and call `kill` once it stalls, or once `capMs` has
48 * passed. The first judgement is made a whole window after the start, so a
49 * transfer gets a full window to begin.
50 */
51 export function watchProgress(
52 stall: Stall,
53 capMs: number,
54 kill: () => void,
55 ): Progress {
56 const slices: number[] = [];
57 let current = 0;
58 const tick = setInterval(
59 () => {
60 slices.push(current);
61 current = 0;
62 if (slices.length > SLICES) slices.shift();
63 if (slices.length < SLICES) return;
64 const moved = slices.reduce((sum, n) => sum + n, 0);
65 if (moved < stall.minBytes) {
66 stop();
67 kill();
68 }
69 },
70 Math.max(1, Math.floor(stall.windowMs / SLICES)),
71 );
72 const cap = setTimeout(() => {
73 stop();
74 kill();
75 }, capMs);
76 const stop = (): void => {
77 clearInterval(tick);
78 clearTimeout(cap);
79 };
80 return {
81 moved(bytes) {
82 current += bytes;
83 },
84 stop,
85 };
86 }
87
88 /** A metered stream, and how to end it early. */
89 export interface Metered {
90 readonly stream: ReadableStream<Uint8Array>;
91 /**
92 * End the stream now. What is still queued is dropped, the source is
93 * cancelled, and the reader sees a clean end: a truncated pack, which git
94 * reports as an early EOF.
95 */
96 cut(): void;
97 }
98
99 /** The most handed to the reader at once, so the count follows the reader. */
100 const PIECE = 64 * 1024;
101
102 /**
103 * `stream`, counted as it is read. Bun reads a child's stdout as fast as
104 * the child writes it, so one chunk can be most of a pack. Each chunk is
105 * handed on in pieces, one per read, and a piece is counted when the
106 * reader takes it. A keep-alive counts as a whole window's worth of bytes.
107 */
108 export function metered(
109 stream: ReadableStream<Uint8Array>,
110 progress: Progress,
111 stall: Stall,
112 ): Metered {
113 const reader = stream.getReader();
114 let chunk = new Uint8Array(0);
115 let offset = 0;
116 let ended = false;
117 let controller: ReadableStreamDefaultController<Uint8Array> | undefined;
118 const end = (): void => {
119 if (ended) return;
120 ended = true;
121 progress.stop();
122 try {
123 controller?.close();
124 } catch {
125 // Already closed.
126 }
127 };
128 const out = new ReadableStream<Uint8Array>(
129 {
130 start(c) {
131 controller = c;
132 },
133 async pull(c) {
134 if (offset >= chunk.length) {
135 const next = await reader.read();
136 if (ended) return;
137 if (next.done) return end();
138 // A spawn pipe's chunk may be a view over a buffer Bun reuses,
139 // and this one is held until the reader takes it: keep a copy.
140 chunk = next.value.slice();
141 offset = 0;
142 if (isKeepAlive(chunk)) progress.moved(stall.minBytes);
143 }
144 const piece = chunk.subarray(offset, offset + PIECE);
145 offset += piece.length;
146 progress.moved(piece.length);
147 c.enqueue(piece);
148 },
149 cancel(reason) {
150 end();
151 return reader.cancel(reason);
152 },
153 },
154 { highWaterMark: 0 },
155 );
156 return {
157 stream: out,
158 cut: () => {
159 end();
160 void reader.cancel().catch(() => {});
161 },
162 };
163 }
164
165 /** How long a push may run before it is killed. */
166 export const PUSH_TIMEOUT_MS = 600_000;
167
168 export interface Deadline {
169 /** The stream to answer with: git's stdout, metered for a fetch. */
170 readonly stdout: ReadableStream<Uint8Array>;
171 /** Bytes the client sent (counted only for a fetch). */
172 moved(bytes: number): void;
173 /**
174 * Git exited. A push's clock stops. A fetch's keeps running until its
175 * stream is read to the end: Bun reads a child's stdout as fast as the
176 * child writes it, so git can exit with most of the pack still buffered
177 * here, and sending that buffer is still the transfer.
178 */
179 exited(): void;
180 /** The transfer is over, however it ended: stop every clock. Idempotent. */
181 stop(): void;
182 }
183
184 /**
185 * The deadline a pack transfer runs under, shared by smart HTTP and SSH.
186 * A fetch ends when it stalls, or after `timeoutMs` (default two hours)
187 * whatever it is doing. Ending it kills git and cuts the stream, which also
188 * frees the buffer of a client that stopped reading. A push ends after
189 * `timeoutMs` (default ten minutes): once its pack is in, index-pack and
190 * the hooks run with nothing to send, and that quiet is not a stall.
191 */
192 export function transferDeadline(
193 fetch: boolean,
194 stdout: ReadableStream<Uint8Array>,
195 options: { timeoutMs?: number; stall?: Stall },
196 kill: () => void,
197 ): Deadline {
198 if (fetch) {
199 const stall = options.stall ?? STALL;
200 let cut = (): void => {};
201 const progress = watchProgress(
202 stall,
203 options.timeoutMs ?? TRANSFER_CAP_MS,
204 () => {
205 kill();
206 cut();
207 },
208 );
209 const meter = metered(stdout, progress, stall);
210 cut = meter.cut;
211 return {
212 stdout: meter.stream,
213 moved: progress.moved,
214 exited: () => {},
215 stop: progress.stop,
216 };
217 }
218 const timer = setTimeout(kill, options.timeoutMs ?? PUSH_TIMEOUT_MS);
219 const stop = (): void => clearTimeout(timer);
220 return { stdout, moved: () => {}, exited: stop, stop };
221 }
222
223 function isKeepAlive(chunk: Uint8Array): boolean {
224 if (chunk.length < KEEPALIVE.length) return false;
225 return KEEPALIVE.every((byte, i) => chunk[i] === byte);
226 }
src/git/smart-http.ts +68 −17
@@ -16,6 +16,7 @@ import {
1616 readBounded,
1717 } from "../http/limits.ts";
1818 import { gitEnv, transferGate } from "./env.ts";
19 import { transferDeadline, type Stall } from "./progress.ts";
1920
2021 export type GitService = "git-upload-pack" | "git-receive-pack";
2122
@@ -30,11 +31,45 @@ function pktLine(payload: string): string {
3031
3132 const FLUSH = "0000";
3233
34 /**
35 * Upload-pack serves partial clones (`git clone --filter=blob:none`) and
36 * the blob fetches they make later, which want an object by id and not a
37 * ref. Protocol v2 already allows any want; this lets v0 clients do the
38 * same. Passed with `-c` so the blanked global and system config
39 * (`gitEnv`) stay blank. A want is still only served after the repository
40 * read check, like every other want. The keep-alive is git's default,
41 * pinned here because `progress.ts` relies on it.
42 */
43 const UPLOAD_PACK_CONFIG = [
44 "-c",
45 "uploadpack.allowFilter=true",
46 "-c",
47 "uploadpack.allowAnySHA1InWant=true",
sentinelcrew

Low, not blocking. With any-SHA1 wants, a fork that borrows through alternates serves every object in its parent's store by id. detachFork only runs on delete. So if a public fork's parent later goes private, objects pushed to the parent afterwards can be fetched through the public fork by anyone who knows their oid. Protocol v2 already allowed this before this pull, so the change doesn't widen it. It is worth a follow-up: detach a fork, or refuse it, when the visibility gap opens.
48 "-c",
49 "uploadpack.keepAlive=5",
50 ] as const;
51
52 /** The argv that runs `service`: `git [config] <subcommand> …args`. */
53 export function serviceCommand(
54 service: GitService,
55 args: readonly string[],
56 ): string[] {
57 // "git-upload-pack" the service is "upload-pack" the subcommand.
58 const subcommand = service.slice(4);
59 const config = service === "git-upload-pack" ? UPLOAD_PACK_CONFIG : [];
60 return ["git", ...config, subcommand, ...args];
61 }
62
3363 export interface SmartHttpOptions {
3464 /** Extra env for the spawned git process (pusher identity for hooks, etc.). */
3565 env?: Record<string, string>;
36 /** Kill a wedged transfer after this long. @default 600_000 */
66 /**
67 * Kill the transfer after this long, however well it moves. @default
68 * 7_200_000 for a fetch (`progress.ts`), 600_000 for a push
69 */
3770 timeoutMs?: number;
71 /** End a fetch that moves less than this (`progress.ts`). @default STALL */
72 stall?: Stall;
3873 /**
3974 * End a fetch whose request body stops arriving for this long: a client
4075 * that opens a negotiation and trickles it holds a transfer permit the
@@ -59,14 +94,14 @@ export async function advertiseRefs(
5994 request: Request,
6095 options: SmartHttpOptions = {},
6196 ): Promise<Response> {
62 const proc = await spawnService(repoDir, service, request, options, [
97 const { stdout } = await spawnService(repoDir, service, request, options, [
6398 "--advertise-refs",
6499 ]);
65100
66101 const preamble = new TextEncoder().encode(
67102 pktLine(`# service=${service}\n`) + FLUSH,
68103 );
69 const body = prepend(preamble, proc.stdout);
104 const body = prepend(preamble, stdout);
70105
71106 return new Response(body, {
72107 headers: {
@@ -87,12 +122,18 @@ export async function serviceRpc(
87122 request: Request,
88123 options: SmartHttpOptions = {},
89124 ): Promise<Response> {
90 const proc = await spawnService(repoDir, service, request, options, []);
125 const { proc, stdout } = await spawnService(
126 repoDir,
127 service,
128 request,
129 options,
130 [],
131 );
91132 feedStdin(proc, service, request, options.idleMs ?? FETCH_IDLE_MS).catch(() =>
92133 proc.kill("SIGKILL"),
93134 );
94135
95 return new Response(proc.stdout, {
136 return new Response(stdout, {
96137 headers: {
97138 "content-type": `application/x-${service}-result`,
98139 "cache-control": "no-cache",
@@ -113,6 +154,9 @@ export async function serviceRpc(
113154 * holds its permit across the pre-receive hook, and that hook calls back
114155 * into the forge for a branch-protection check that needs plumbing permits
115156 * of its own. One shared pool would deadlock the first time pushes filled it.
157 *
158 * A fetch ends when it stops moving, a push on a clock (`progress.ts`);
159 * `stdout` is the stream to answer with, metered for a fetch.
116160 */
117161 async function spawnService(
118162 repoDir: string,
@@ -126,9 +170,6 @@ async function spawnService(
126170 permit();
127171 options.release?.();
128172 };
129 // "git-upload-pack" the service is "upload-pack" the subcommand.
130 const subcommand = service.slice(4);
131
132173 const env: Record<string, string> = { ...options.env };
133174 // Forward the client's protocol request, but only within git's own
134175 // grammar (colon-separated key[=value] tokens) and bounded — never an
@@ -140,7 +181,7 @@ async function spawnService(
140181 }
141182
142183 const spawn = () =>
143 Bun.spawn(["git", subcommand, "--stateless-rpc", ...extraArgs, repoDir], {
184 Bun.spawn(serviceCommand(service, ["--stateless-rpc", ...extraArgs, repoDir]), {
144185 env: gitEnv(env),
145186 stdin: "pipe",
146187 stdout: "pipe",
@@ -157,25 +198,35 @@ async function spawnService(
157198 throw error;
158199 }
159200
160 const timer = setTimeout(
161 () => proc.kill("SIGKILL"),
162 options.timeoutMs ?? 600_000,
201 const kill = (): void => {
202 proc.kill("SIGKILL");
203 };
204 const deadline = transferDeadline(
205 service === "git-upload-pack",
206 proc.stdout,
207 options,
208 kill,
163209 );
164210 // A transfer under way is the account's, and ends with its standing
165211 // (\`http/caller.ts\`): a clone does not run on after a suspension.
166 const unhold = holdForCaller(request, () => proc.kill("SIGKILL"));
212 const unhold = holdForCaller(request, kill);
167213 void proc.exited.finally(() => {
168 clearTimeout(timer);
214 deadline.exited();
169215 unhold();
170216 release();
171217 });
172 request.signal.addEventListener("abort", () => proc.kill("SIGKILL"));
218 // A client that hung up has nothing left to deliver or to watch.
219 const abort = (): void => {
220 kill();
221 deadline.stop();
222 };
223 request.signal.addEventListener("abort", abort);
173224 // The client may have hung up while this call was queued behind other
174225 // transfers, in which case the listener above will never fire — nothing
175226 // is left to abort. Check once, now that there is a process to kill.
176 if (request.signal.aborted) proc.kill("SIGKILL");
227 if (request.signal.aborted) abort();
177228
178 return proc;
229 return { proc, stdout: deadline.stdout };
179230 }
180231
181232 /**
src/git/smart-ssh.ts +26 −13
@@ -11,7 +11,8 @@
1111
1212 import { isSafeName } from "./guard.ts";
1313 import { gitEnv } from "./env.ts";
14 import type { GitService } from "./smart-http.ts";
14 import { transferDeadline, type Stall } from "./progress.ts";
15 import { serviceCommand, type GitService } from "./smart-http.ts";
1516
1617 export type { GitService };
1718
@@ -114,8 +115,13 @@ export interface GitChannel {
114115 export interface GitServiceOptions {
115116 /** Extra env for the spawned git process (hook identity, GIT_PROTOCOL). */
116117 env?: Record<string, string>;
117 /** Kill a wedged transfer after this long. @default 600_000 */
118 /**
119 * Kill the transfer after this long, however well it moves. @default
120 * 7_200_000 for a fetch (`progress.ts`), 600_000 for a push
121 */
118122 timeoutMs?: number;
123 /** End a fetch that moves less than this (`progress.ts`). @default STALL */
124 stall?: Stall;
119125 }
120126
121127 /**
@@ -123,6 +129,8 @@ export interface GitServiceOptions {
123129 * Output is written under backpressure: git's stdout can produce a pack far
124130 * faster than the peer's window drains it, and awaiting `drain()` between
125131 * chunks is what keeps a clone from buffering the whole pack in memory.
132 * A fetch ends when it stops moving, in either direction (`progress.ts`);
133 * a push keeps a wall clock, as over HTTP.
126134 */
127135 export function runGitService(
128136 dir: string,
@@ -130,20 +138,25 @@ export function runGitService(
130138 channel: GitChannel,
131139 options: GitServiceOptions = {},
132140 ): Promise<number> {
133 // "git-upload-pack" the service is "upload-pack" the subcommand.
134 const proc = Bun.spawn(["git", service.slice(4), dir], {
141 const proc = Bun.spawn(serviceCommand(service, [dir]), {
135142 env: gitEnv(options.env),
136143 stdin: "pipe",
137144 stdout: "pipe",
138145 stderr: "pipe",
139146 });
140147
141 const timer = setTimeout(
142 () => proc.kill("SIGKILL"),
143 options.timeoutMs ?? 600_000,
148 const kill = (): void => {
149 proc.kill("SIGKILL");
150 };
151 const { stdout, moved, stop } = transferDeadline(
152 service === "git-upload-pack",
153 proc.stdout,
154 options,
155 kill,
144156 );
145157
146158 channel.onData((chunk) => {
159 moved(chunk.length);
147160 try {
148161 proc.stdin.write(chunk);
149162 void proc.stdin.flush();
@@ -164,11 +177,11 @@ export function runGitService(
164177 });
165178 // A *close* is different: the peer is gone, so every byte git still has to
166179 // give has nowhere to go. Without this the child sits in `upload-pack`
167 // writing into a dead channel until the ten-minute timeout reaps it — one
168 // idle git per disconnected clone.
180 // writing into a dead channel until its deadline reaps it — one idle git
181 // per disconnected clone.
169182 channel.onClose(() => {
170 clearTimeout(timer);
171 proc.kill("SIGKILL");
183 stop();
184 kill();
172185 });
173186
174187 const pump = async (
@@ -186,11 +199,11 @@ export function runGitService(
186199
187200 return (async () => {
188201 const [, , code] = await Promise.all([
189 pump(proc.stdout, (data) => channel.write(data)),
202 pump(stdout, (data) => channel.write(data)),
190203 pump(proc.stderr, (data) => channel.writeStderr(data)),
191204 proc.exited,
192205 ]);
193 clearTimeout(timer);
206 stop();
194207 channel.exit(code);
195208 return code;
196209 })();
src/http/server.ts +2 −2
@@ -1461,8 +1461,8 @@ export function knownPatterns(): string[] {
14611461 /**
14621462 * A transfer someone signed in makes runs as it always has; a stranger's
14631463 * takes a share of the transfer pool first (`strangers.ts`) — refused with
1464 * a 429 when its address or every stranger together has none left — and
1465 * runs on a shorter clock. Called once access is decided, so a private
1464 * a 429 when its address or every stranger together has none left. Either
1465 * one ends once it stops moving (`git/progress.ts`). Called once access is decided, so a private
14661466 * repository still answers a stranger with the credential challenge.
14671467 */
14681468 async function asStranger(
src/http/strangers.ts +5 −2
@@ -37,7 +37,10 @@ export interface StrangerLimits {
3737 * kept for pushes and signed-in clones. At least one permit.
3838 */
3939 readonly transferShare: number;
40 /** A stranger's transfer is killed after this long (a signed-in one: 600 s). */
40 /**
41 * A stranger's fetch is killed after this long, however well it moves.
42 * It is killed sooner once it stalls (`git/progress.ts`), as anyone's is.
43 */
4144 readonly transferTimeoutMs: number;
4245 /** A fetch's request body that stops arriving for this long ends the fetch. */
4346 readonly transferIdleMs: number;
@@ -60,7 +63,7 @@ export const STRANGER_LIMITS: StrangerLimits = {
6063 forge: { api: 6000, git: 600, code: 60, plan: 60 },
6164 transfersPerAddress: 4,
6265 transferShare: 0.5,
63 transferTimeoutMs: 300_000,
66 transferTimeoutMs: 7_200_000,
6467 transferIdleMs: 30_000,
6568 codeAtOnce: 1,
6669 streamsPerAddress: 8,
test/big-clone.test.ts added +399
@@ -0,0 +1,399 @@
1 /**
2 * Big repositories always clone (#4482). A fetch ends once it stops moving,
3 * not on a wall clock, so a large pack over a slow link finishes while a
4 * client that stops reading is still cut off. Upload-pack serves partial
5 * clones, so `git clone --filter=blob:none` skips the big blobs and a later
6 * checkout fetches them. A fetch by object id passes the same read check as
7 * a clone.
8 *
9 * Time is scaled down. The real window is 30 s and a CAD repository is
10 * hundreds of megabytes; here the window is one second and the pack is 4 MiB
11 * through a 1 MiB/s link, so the clone lasts several windows, longer than
12 * the wall clock it is also run against.
13 */
14 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
15 import { mkdtemp, rm } from "node:fs/promises";
16 import { tmpdir } from "node:os";
17 import { join } from "node:path";
18 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
19 import { createApp, type App } from "../src/app.ts";
20 import { createToken } from "../src/core/tokens.ts";
21 import { runGit } from "../src/git/env.ts";
22 import { metered, watchProgress, type Stall } from "../src/git/progress.ts";
23 import { initBareRepo, repoDir } from "../src/git/repos.ts";
24 import {
25 advertiseRefs,
26 isGitService,
27 serviceRpc,
28 type SmartHttpOptions,
29 } from "../src/git/smart-http.ts";
30 import { serve, type Servers } from "../src/http/server.ts";
31 import { unwrap } from "../src/kernel/result.ts";
32 import { seedAdmin } from "./helpers/admin.ts";
33 import { gitClientEnv } from "./helpers/git-client.ts";
34
35 interface Run {
36 code: number;
37 out: string;
38 err: string;
39 }
40
41 async function git(args: string[], cwd: string): Promise<Run> {
42 const proc = Bun.spawn(["git", ...args], {
43 cwd,
44 env: gitClientEnv(),
45 stdout: "pipe",
46 stderr: "pipe",
47 });
48 const [out, err, code] = await Promise.all([
49 new Response(proc.stdout).text(),
50 new Response(proc.stderr).text(),
51 proc.exited,
52 ]);
53 return { code, out: out.trim(), err: err.trim() };
54 }
55
56 async function ok(args: string[], cwd: string): Promise<string> {
57 const run = await git(args, cwd);
58 if (run.code !== 0) throw new Error(`git ${args.join(" ")}: ${run.err}`);
59 return run.out;
60 }
61
62 /** Object ids a partial clone does not have, without fetching them. */
63 async function missing(dir: string): Promise<string[]> {
64 const out = await ok(["rev-list", "--objects", "--missing=print", "--all"], dir);
65 return out
66 .split("\n")
67 .filter((line) => line.startsWith("?"))
68 .map((line) => line.slice(1));
69 }
70
71 function randomBytes(size: number): Uint8Array {
72 const out = new Uint8Array(size);
73 for (let at = 0; at < size; at += 65_536)
74 crypto.getRandomValues(out.subarray(at, Math.min(size, at + 65_536)));
75 return out;
76 }
77
78 describe("watchProgress", () => {
79 const stall: Stall = { windowMs: 300, minBytes: 1_000 };
80
81 test("a transfer that keeps moving is never killed", async () => {
82 let killed = false;
83 const progress = watchProgress(stall, 60_000, () => (killed = true));
84 const feed = setInterval(() => progress.moved(400), 50);
85 await Bun.sleep(1_200);
86 clearInterval(feed);
87 progress.stop();
88 expect(killed).toBe(false);
89 });
90
91 test("a transfer that stalls is killed within a window and a slice", async () => {
92 let killedAt = 0;
93 const start = Date.now();
94 const progress = watchProgress(stall, 60_000, () => (killedAt = Date.now()));
95 // A trickle under the floor is a stall too.
96 const feed = setInterval(() => progress.moved(10), 50);
97 await Bun.sleep(1_000);
98 clearInterval(feed);
99 progress.stop();
100 expect(killedAt).toBeGreaterThan(0);
101 expect(killedAt - start).toBeGreaterThanOrEqual(stall.windowMs - 20);
102 expect(killedAt - start).toBeLessThan(stall.windowMs * 2 + 200);
103 });
104
105 test("the cap ends a transfer however well it moves", async () => {
106 let killed = false;
107 const progress = watchProgress(stall, 400, () => (killed = true));
108 const feed = setInterval(() => progress.moved(10_000), 50);
109 await Bun.sleep(700);
110 clearInterval(feed);
111 progress.stop();
112 expect(killed).toBe(true);
113 });
114
115 test("git's keep-alive counts as progress; other bytes count as themselves", async () => {
116 let moved = 0;
117 const progress = { moved: (n: number) => (moved += n), stop: () => {} };
118 const source = new ReadableStream<Uint8Array>({
119 start(controller) {
120 controller.enqueue(new Uint8Array([0x30, 0x30, 0x30, 0x35, 0x01]));
121 controller.enqueue(new TextEncoder().encode("0006\x01a"));
122 controller.close();
123 },
124 });
125 const out = await new Response(metered(source, progress, stall).stream).arrayBuffer();
126 expect(out.byteLength).toBe(11);
127 expect(moved).toBe(stall.minBytes + 11);
128 });
129 });
130
131 /** Delays each chunk as a link of `bytesPerSecond` would. */
132 function throttled(
133 body: ReadableStream<Uint8Array>,
134 bytesPerSecond: number,
135 ): ReadableStream<Uint8Array> {
136 const reader = body.getReader();
137 return new ReadableStream<Uint8Array>(
138 {
139 async pull(controller) {
140 const { done, value } = await reader.read();
141 if (done) return controller.close();
142 await Bun.sleep((value.length * 1_000) / bytesPerSecond);
143 controller.enqueue(value);
144 },
145 cancel: (reason) => reader.cancel(reason),
146 },
147 { highWaterMark: 0 },
148 );
149 }
150
151 /**
152 * Passes `after` bytes on, then stops reading for `pauseMs`, like a client
153 * that went away, and reads on after it. A transfer still alive then
154 * finishes; one that was ended hands back a cut-off pack.
155 */
156 function pausing(
157 body: ReadableStream<Uint8Array>,
158 after: number,
159 pauseMs: number,
160 ): ReadableStream<Uint8Array> {
161 const reader = body.getReader();
162 let passed = 0;
163 let paused = false;
164 return new ReadableStream<Uint8Array>(
165 {
166 async pull(controller) {
167 if (passed >= after && !paused) {
168 paused = true;
169 await Bun.sleep(pauseMs);
170 }
171 const { done, value } = await reader.read();
172 if (done) return controller.close();
173 passed += value.length;
174 controller.enqueue(value);
175 },
176 cancel: (reason) => reader.cancel(reason),
177 },
178 { highWaterMark: 0 },
179 );
180 }
181
182 describe("a big repository over smart http", () => {
183 const stall: Stall = { windowMs: 1_000, minBytes: 16 * 1024 };
184 const BIG = 4 * 1024 * 1024;
185 let root: string;
186 let work: string;
187 let server: ReturnType<typeof Bun.serve>;
188 let base: string;
189 let big: Uint8Array;
190 /** How this test's fetch is answered: its options and its link. */
191 let fetchOptions: SmartHttpOptions = {};
192 let link = (body: ReadableStream<Uint8Array>) => body;
193
194 beforeAll(async () => {
195 root = await mkdtemp(join(tmpdir(), "forge-bigclone-"));
196 work = await mkdtemp(join(tmpdir(), "forge-bigclone-work-"));
197 const bare = repoDir(root, "ada", "arm");
198 unwrap(await initBareRepo(bare));
199 const seed = join(work, "seed");
200 await ok(["init", "-q", "-b", "main", seed], work);
201 big = randomBytes(BIG);
202 await Bun.write(join(seed, "base.step"), big);
203 await Bun.write(join(seed, "README.md"), "# arm\n");
204 await ok(["add", "."], seed);
205 await ok(["commit", "-q", "-m", "cad"], seed);
206 await ok(["push", "-q", bare, "main"], seed);
207
208 server = Bun.serve({
209 port: 0,
210 idleTimeout: 0,
211 async fetch(request) {
212 const url = new URL(request.url);
213 const dir = repoDir(root, "ada", "arm");
214 if (url.pathname === "/ada/arm.git/info/refs") {
215 const service = url.searchParams.get("service");
216 if (!isGitService(service)) return new Response("no", { status: 400 });
217 return advertiseRefs(dir, service, request);
218 }
219 if (url.pathname === "/ada/arm.git/git-upload-pack") {
220 const response = await serviceRpc(dir, "git-upload-pack", request, fetchOptions);
221 return new Response(link(response.body!), { headers: response.headers });
222 }
223 return new Response("not found", { status: 404 });
224 },
225 });
226 base = `http://127.0.0.1:${server.port}/ada/arm.git`;
227 });
228
229 afterAll(async () => {
230 server.stop(true);
231 await rm(root, { recursive: true, force: true });
232 await rm(work, { recursive: true, force: true });
233 });
234
235 test("a clone through a 1 MiB/s link outlasts several windows and finishes", async () => {
236 fetchOptions = { stall, timeoutMs: 60_000 };
237 link = (body) => throttled(body, 1024 * 1024);
238 try {
239 const start = Date.now();
240 const run = await git(["clone", "-q", base, "slow"], work);
241 expect(run.err).toBe("");
242 expect(run.code).toBe(0);
243 expect(Date.now() - start).toBeGreaterThan(stall.windowMs * 3);
244 const got = new Uint8Array(await Bun.file(join(work, "slow", "base.step")).arrayBuffer());
245 expect(got.length).toBe(BIG);
246 expect(Bun.hash(got)).toBe(Bun.hash(big));
247 } finally {
248 fetchOptions = {};
249 link = (body) => body;
250 }
251 }, 60_000);
252
253 test("the same clone under a wall clock is cut off: the clock was the bug", async () => {
254 fetchOptions = { stall, timeoutMs: 2_000 };
255 link = (body) => throttled(body, 1024 * 1024);
256 try {
257 const run = await git(["clone", "-q", base, "clocked"], work);
258 expect(run.code).not.toBe(0);
259 } finally {
260 fetchOptions = {};
261 link = (body) => body;
262 }
263 }, 60_000);
264
265 test("a client that stops reading for longer than a window is cut off", async () => {
266 fetchOptions = { stall, timeoutMs: 60_000 };
267 link = (body) => pausing(body, 64 * 1024, stall.windowMs * 3);
268 try {
269 const run = await git(["clone", "-q", base, "stalled"], work);
270 expect(run.code).not.toBe(0);
271 expect(run.err).toContain("early EOF");
272 } finally {
273 fetchOptions = {};
274 link = (body) => body;
275 }
276 }, 60_000);
277
278 test("a pause shorter than a window is not a stall", async () => {
279 fetchOptions = { stall, timeoutMs: 60_000 };
280 link = (body) => pausing(body, 64 * 1024, stall.windowMs / 3);
281 try {
282 const run = await git(["clone", "-q", base, "paused"], work);
283 expect(run.code).toBe(0);
284 } finally {
285 fetchOptions = {};
286 link = (body) => body;
287 }
288 }, 60_000);
289
290 for (const version of [0, 2]) {
291 test(`--filter=blob:limit=1m over protocol v${version} leaves the big blob behind, and checkout fetches it`, async () => {
292 const dir = `limit-v${version}`;
293 const run = await git(
294 ["-c", `protocol.version=${version}`, "clone", "-q", "--no-checkout", "--filter=blob:limit=1m", base, dir],
295 work,
296 );
297 expect(run.err).not.toContain("filtering not recognized");
298 expect(run.code).toBe(0);
299 const clone = join(work, dir);
300 const bigOid = await ok(["rev-parse", "HEAD:base.step"], clone);
301 const readmeOid = await ok(["rev-parse", "HEAD:README.md"], clone);
302 const absent = await missing(clone);
303 expect(absent).toContain(bigOid);
304 expect(absent).not.toContain(readmeOid);
305
306 await ok(["-c", `protocol.version=${version}`, "reset", "-q", "--hard", "HEAD"], clone);
307 expect(await missing(clone)).toEqual([]);
308 const got = new Uint8Array(await Bun.file(join(clone, "base.step")).arrayBuffer());
309 expect(Bun.hash(got)).toBe(Bun.hash(big));
310 });
311
312 test(`--filter=blob:none over protocol v${version} clones and checks out`, async () => {
313 const dir = `none-v${version}`;
314 const run = await git(
315 ["-c", `protocol.version=${version}`, "clone", "-q", "--filter=blob:none", base, dir],
316 work,
317 );
318 expect(run.err).not.toContain("filtering not recognized");
319 expect(run.code).toBe(0);
320 expect(await Bun.file(join(work, dir, "README.md")).text()).toBe("# arm\n");
321 expect((await Bun.file(join(work, dir, "base.step")).arrayBuffer()).byteLength).toBe(BIG);
322 });
323 }
324 });
325
326 describe("a fetch by object id is a read like any other", () => {
327 let dataDir: string;
328 let work: string;
329 let app: App;
330 let servers: Servers;
331 let token: string;
332 let secretOid: string;
333
334 beforeAll(async () => {
335 dataDir = await mkdtemp(join(tmpdir(), "forge-bigclone-app-"));
336 work = await mkdtemp(join(tmpdir(), "forge-bigclone-app-work-"));
337 app = createApp({ dataDir, port: 0 });
338 servers = await serve(app);
339 const owner = await seedAdmin(app, {
340 username: "ada",
341 email: "[email protected]",
342 password: "owner-passphrase",
343 });
344 if (!owner.ok) throw new Error(owner.error);
345 token = createToken(app.db, owner.value.id, "t", "ro").token;
346 const vault = await createRepo(app, owner.value, { name: "vault" });
347 const open = await createRepo(app, owner.value, { name: "open", visibility: "public" });
348 if (!vault.ok || !open.ok) throw new Error("repo creation failed");
349 await commitToBranch(app, owner.value, vault.value, {
350 message: "seed",
351 files: [{ path: "secret", content: "the vault's secret\n" }],
352 });
353 await commitToBranch(app, owner.value, open.value, {
354 message: "seed",
355 files: [{ path: "README.md", content: "# open\n" }],
356 });
357 secretOid = unwrap(
358 await runGit(["rev-parse", "main:secret"], { cwd: repoDir(app.reposRoot, "ada", "vault") }),
359 );
360 });
361
362 afterAll(async () => {
363 servers.stop();
364 app.close();
365 await rm(dataDir, { recursive: true, force: true });
366 await rm(work, { recursive: true, force: true });
367 });
368
369 const url = (name: string, auth = ""): string =>
370 `${servers.url.replace("://", `://${auth}`)}/ada/${name}.git`;
371
372 test("a stranger cannot fetch a private repository's blob by its id", async () => {
373 const dir = join(work, "stranger");
374 await ok(["init", "-q", dir], work);
375 for (const version of [0, 2]) {
376 const run = await git(["-c", `protocol.version=${version}`, "fetch", url("vault"), secretOid], dir);
377 expect(run.code).not.toBe(0);
378 }
379 const lookup = await git(["cat-file", "-e", secretOid], dir);
380 expect(lookup.code).not.toBe(0);
381 });
382
383 test("a partial clone of a public repository cannot reach into a private one", async () => {
384 const run = await git(["clone", "-q", "--filter=blob:none", url("open"), "open"], work);
385 expect(run.code).toBe(0);
386 const lazy = await git(["cat-file", "-p", secretOid], join(work, "open"));
387 expect(lazy.code).not.toBe(0);
388 expect(lazy.out).not.toContain("secret");
389 });
390
391 test("a reader's partial clone of the private repository fetches the blob lazily", async () => {
392 const dir = join(work, "reader");
393 const auth = `ada:${token}@`;
394 const run = await git(["clone", "-q", "--no-checkout", "--filter=blob:none", url("vault", auth), dir], work);
395 expect(run.code).toBe(0);
396 expect(await missing(dir)).toContain(secretOid);
397 expect(await ok(["cat-file", "-p", secretOid], dir)).toBe("the vault's secret");
398 });
399 });
test/git-ssh.test.ts +37
@@ -181,6 +181,43 @@ describe.skipIf(NO_OPENSSH)("git over ssh", () => {
181181 expect(run.err).not.toContain("permission");
182182 });
183183
184 test("a guest's partial clone leaves the blobs behind, and checkout fetches them (#4482)", async () => {
185 for (const version of [0, 2]) {
186 const dir = `guest-partial-v${version}`;
187 const run = await git(
188 ["-c", `protocol.version=${version}`, "clone", "--no-checkout", "--filter=blob:none", url("alice", "beacon"), dir],
189 { cwd: work, key: strangerKey },
190 );
191 expect(run.err).not.toContain("filtering not recognized");
192 expect(run.code).toBe(0);
193 const clone = join(work, dir);
194 const readme = await git(["rev-parse", "HEAD:README.md"], { cwd: clone, key: strangerKey });
195 const missing = await git(["rev-list", "--objects", "--missing=print", "--all"], {
196 cwd: clone,
197 key: strangerKey,
198 });
199 expect(missing.out.split("\n")).toContain(`?${readme.out}`);
200 const reset = await git(["-c", `protocol.version=${version}`, "reset", "-q", "--hard", "HEAD"], {
201 cwd: clone,
202 key: strangerKey,
203 });
204 expect(reset.code).toBe(0);
205 expect(await readFile(join(clone, "README.md"), "utf8")).toContain("Beacon");
206 }
207 });
208
209 test("a guest cannot fetch a private repo's blob by its id (#4482)", async () => {
210 const vault = repoDir(forge.reposRoot, "alice", "vault");
211 const oid = await runGit(["rev-parse", "main:secret"], { cwd: vault });
212 if (!oid.ok) throw new Error("no secret blob");
213 const dir = join(work, "guest-oid");
214 expect((await git(["init", "-q", dir], { cwd: work, key: strangerKey })).code).toBe(0);
215 const run = await git(["fetch", url("alice", "vault"), oid.value], { cwd: dir, key: strangerKey });
216 expect(run.code).not.toBe(0);
217 expect(run.err).toContain("repository not found");
218 expect((await git(["cat-file", "-e", oid.value], { cwd: dir, key: strangerKey })).code).not.toBe(0);
219 });
220
184221 test("a writer pushes, and the push lands in the outbox", async () => {
185222 const clone = await git(["clone", url("alice", "beacon"), "alice-beacon"], {
186223 cwd: work,

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs