trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns18g rEventsFlows

A pull opened on the Forgejo mirror reaches trunk as a real pull with its author, and the mirror stays a plain copy #9039

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/9011-a-pull-opened-on-the-forgejo-mirror-reac → main 3 commits fork

  • Merged. scout/9011-a-pull-opened-on-the-forgejo-mirror-reac was merged into main as ad06e48.

Review

Sign in to review this change.
16 files changed +949 −28
docs/ORIGINS.md +25
@@ -792,6 +792,31 @@ rights on GitHub: without them the migration still succeeds and says the
792792 repository was not archived, so you can do it there yourself. Only repositories
793793 read through a connected origin have an original to change; a pasted URL does not.
794794
795 **Pulls opened on the mirror come here** (#9011, `src/origins/mirror-pulls.ts`).
796 Someone who finds the mirror and opens a pull there is not lost: each open
797 pull on the mirror is fetched (`refs/pull/<n>/head`, vetted and pinned like the
798 mirror push) into `mirror-pull/<n>` here and opened as a pull on the default
799 branch (or the mirror pull's base, when it exists here), under the account
800 that connected the origin. It carries its author's name (`origin_author`) and
801 a link back (`origin_html_url`), so it is a guest pull: no secrets, no crew,
802 until a person approves it — unless that author is a trusted source on the
803 mirror's host (`pull_source`; `guestWaits` asks `foldsIn`, so lapsed trust
804 waits again). The author and link are written with the pull's first row, so
805 nothing that hears `pull.opened` ever sees it as an ordinary pull. The fetch
806 is capped at 200 MiB of new objects (`src/git/capped-fetch.ts`: it lands in a
807 scratch repository borrowing trunk's objects, and only a fetch under the cap
808 is moved in). One comment on the mirror pull says where it went. A head that
809 moves on the mirror moves the pull here while it is open, and clears any
810 guest approval: the head a person approved is not the one that would run. The mirror is never pushed to except the trunk refs the
811 mirror push already sends; the comment and a webhook are the only writes. It
812 runs by event — on boot the mirror is asked once for a webhook on its pull
813 events (`watch`, Forgejo's `X-Forgejo-Signature` is accepted on
814 `/api/v1/repos/:owner/:name/origin-hook`, signed with the repository's derived
815 secret) — and by a sweep every five minutes for anything missed. Both key on
816 the mirror pull's URL and run one at a time per repository, so they never
817 make two copies of one pull. `test/mirror-pulls.test.ts`, against the fake
818 Forgejo.
819
795820 **Migrate from the terminal.** Press `M` on the dashboard (or pick "Migrate a
796821 repository" in the palette, `ctrl+k`). The list is your connected origins'
797822 repositories, private ones marked; `/` filters it and `u` pastes a URL instead.
src/actions/pulls.ts +3
@@ -122,6 +122,8 @@ export interface OpenPullInput {
122122 revertsIssueId?: string;
123123 /** The variant group the pull joins (`actions/tell.ts`); absent for an ordinary pull. */
124124 variantGroup?: string;
125 /** Who opened it on another forge, and where; native repos only (`CreatePullInput.origin`). */
126 origin?: { author: string; htmlUrl: string };
125127 /**
126128 * A person's deliberate yes to fixing a sealed report from a branch that
127129 * is already public (this repository, or a public fork). Without it such
@@ -297,6 +299,7 @@ export async function openPull(
297299 : {}),
298300 ...(input.revertsIssueId ? { revertsIssueId: input.revertsIssueId } : {}),
299301 ...(input.variantGroup ? { variantGroup: input.variantGroup } : {}),
302 ...(input.origin ? { origin: input.origin } : {}),
300303 },
301304 );
302305 if (!created.ok) return created;
src/core/pulls.ts +12
@@ -181,6 +181,12 @@ export interface CreatePullInput {
181181 revertsIssueId?: string;
182182 /** The variant group the pull joins; absent for an ordinary pull. */
183183 variantGroup?: string;
184 /**
185 * Who opened it on another forge, and where (#9011): written with the
186 * issue row, so the pull is a guest (`origins/guest.ts`) from its first
187 * moment and nothing can start it with secrets before it is marked.
188 */
189 origin?: { author: string; htmlUrl: string };
184190 }
185191
186192 /**
@@ -292,6 +298,12 @@ export async function createPull(
292298 sealed,
293299 });
294300 if (!issue.ok) return err(issue.error);
301 if (input.origin !== undefined)
302 db.query("UPDATE issue SET origin_author = ?1, origin_html_url = ?2 WHERE id = ?3").run(
303 input.origin.author,
304 input.origin.htmlUrl,
305 issue.value.id,
306 );
295307
296308 const headRepoId = input.headRepoId ?? input.repoId;
297309 db.query(
src/git/capped-fetch.ts added +112
@@ -0,0 +1,112 @@
1 /**
2 * A fetch of a head nobody here vetted (a read-only mirror's pull, #9011),
3 * capped in bytes. git's fetch has no `maxInputSize` of its own, so it runs
4 * into a throwaway bare repository beside the real one that borrows the
5 * real one's objects (`objects/info/alternates`): only what is new lands
6 * there. A watchdog kills the fetch once that passes the cap, and only a
7 * fetch that finished under it has its objects moved into the repository
8 * and is named by `ref`. Anything else leaves the repository as it was.
9 */
10
11 import { mkdir, mkdtemp, readdir, rename, rm, stat } from "node:fs/promises";
12 import { join } from "node:path";
13 import { runGitCapture } from "./env.ts";
14 import { isSafeOid } from "./guard.ts";
15 import { resolveOid } from "./merge.ts";
16
17 export interface CappedFetch {
18 /** The repository fetched into. */
19 readonly dir: string;
20 readonly url: string;
21 /** The remote ref, e.g. `refs/pull/3/head`. */
22 readonly from: string;
23 /** The local ref it lands at, once it is under the cap. */
24 readonly ref: string;
25 readonly maxBytes: number;
26 readonly timeoutMs: number;
27 /** The vetted, pinned env for the remote (`vettedEnv`). */
28 readonly env: Record<string, string>;
29 }
30
31 export type CappedFetchResult =
32 | { readonly ok: true; readonly oid: string }
33 | { readonly ok: false; readonly reason: "too_large" | "failed" };
34
35 const WATCH_MS = 100;
36
37 /** Bytes under `root`, every file. */
38 async function bytesUnder(root: string): Promise<number> {
39 let total = 0;
40 for (const entry of await readdir(root, { recursive: true, withFileTypes: true }).catch(() => [])) {
41 if (!entry.isFile()) continue;
42 const size = await stat(join(entry.parentPath, entry.name)).then((s) => s.size, () => 0);
43 total += size;
44 }
45 return total;
46 }
47
48 /** Move every object file from `from` into `to`, keeping its path; one already there stays. */
49 async function migrate(from: string, to: string): Promise<void> {
50 const entries = (await readdir(from, { recursive: true, withFileTypes: true })).filter(
51 (e) => !join(e.parentPath, e.name).startsWith(join(from, "info")),
52 );
53 // Packs: the index after its pack, so no reader sees an index without one.
54 const files = entries
55 .filter((e) => e.isFile())
56 .map((e) => join(e.parentPath, e.name).slice(from.length + 1))
57 .sort((a, b) => Number(a.endsWith(".idx")) - Number(b.endsWith(".idx")));
58 for (const rel of files) {
59 const target = join(to, rel);
60 await mkdir(join(target, ".."), { recursive: true });
61 if (await stat(target).then(() => true, () => false)) continue;
62 await rename(join(from, rel), target);
63 }
64 }
65
66 export async function cappedFetch(input: CappedFetch): Promise<CappedFetchResult> {
67 const objects = join(input.dir, "objects");
68 const scratch = await mkdtemp(join(input.dir, "incoming-"));
69 const fresh = join(scratch, "objects");
70 const abort = new AbortController();
71 let tooLarge = false;
72 const newBytes = async (): Promise<number> =>
73 (await bytesUnder(fresh)) - (await bytesUnder(join(fresh, "info")));
74 const watch = setInterval(() => {
75 void newBytes().then((n) => {
76 if (n > input.maxBytes && !abort.signal.aborted) {
77 tooLarge = true;
78 abort.abort();
79 }
80 });
81 }, WATCH_MS);
82 try {
83 const made = await runGitCapture(["init", "--quiet", "--bare", scratch]);
84 if (made.code !== 0) return { ok: false, reason: "failed" };
85 await mkdir(join(fresh, "info"), { recursive: true });
86 await Bun.write(join(fresh, "info", "alternates"), `${objects}\n`);
87 const fetched = await runGitCapture(
88 [
89 "fetch",
90 "--no-tags",
91 "--no-write-fetch-head",
92 "--no-recurse-submodules",
93 "--quiet",
94 "--",
95 input.url,
96 `+${input.from}:refs/incoming`,
97 ],
98 { cwd: scratch, env: input.env, timeoutMs: input.timeoutMs, signal: abort.signal },
99 );
100 clearInterval(watch);
101 if (tooLarge || (await newBytes()) > input.maxBytes) return { ok: false, reason: "too_large" };
102 if (fetched.code !== 0) return { ok: false, reason: "failed" };
103 const oid = await resolveOid(scratch, "refs/incoming");
104 if (oid === undefined || !isSafeOid(oid)) return { ok: false, reason: "failed" };
105 await migrate(fresh, objects);
106 const named = await runGitCapture(["update-ref", input.ref, oid], { cwd: input.dir });
107 return named.code === 0 ? { ok: true, oid } : { ok: false, reason: "failed" };
108 } finally {
109 clearInterval(watch);
110 await rm(scratch, { recursive: true, force: true });
111 }
112 }
src/http/api.ts +7 −2
@@ -55,7 +55,8 @@ import { isProbe } from "../kernel/probe.ts";
5555 import { PRIVATE_ADDRESS_REFUSED } from "../kernel/address_words.ts";
5656 import { forgeState, PARENT_HEADER } from "../spawn/clients.ts";
5757 import { familyView } from "../spawn/family.ts";
58 import { HOOK_BODY_MAX, HOOK_SIGNATURE_HEADER, receiveOriginHook } from "../origins/hook.ts";
58 import { HOOK_BODY_MAX, receiveOriginHook, signatureOf } from "../origins/hook.ts";
59 import { receiveMirrorHook } from "../origins/mirror-pulls.ts";
5960 import { receiveBeat, SIGNATURE_HEADER } from "../spawn/heartbeat.ts";
6061 import { takeOrTell } from "../actions/follow.ts";
6162 import { noteRead, telemetryReader, type Reader } from "../actions/telemetry.ts";
@@ -3296,7 +3297,11 @@ export function apiRoutes(router: Router<Handler>): void {
32963297 const bytes = await readBounded(ctx.request, HOOK_BODY_MAX);
32973298 if (!repo || bytes === "too_large") return notFound();
32983299 const raw = new TextDecoder().decode(bytes);
3299 const kept = await receiveOriginHook(ctx.app, repo, raw, ctx.request.headers.get(HOOK_SIGNATURE_HEADER));
3300 const signature = signatureOf(ctx.request.headers);
3301 // The origin this repository mirrors, or the mirror it pushes to (#9011).
3302 const kept =
3303 (await receiveOriginHook(ctx.app, repo, raw, signature)) ||
3304 (await receiveMirrorHook(ctx.app, repo, raw, signature));
33003305 return kept ? json({ ok: true }) : notFound();
33013306 });
33023307
src/main.ts +4
@@ -50,6 +50,7 @@ import { RED_HOLD_MS } from "./checks/red_main.ts";
5050 import { stopQueue } from "./checks/queue.ts";
5151 import { startLooks } from "./looks/service.ts";
5252 import { startMirrorOut } from "./origins/mirror-out.ts";
53 import { startMirrorPulls } from "./origins/mirror-pulls.ts";
5354 import { startProbes } from "./contract/probes.ts";
5455 import { startNightly } from "./nightly/service.ts";
5556 import { startSnapshots } from "./snapshot/service.ts";
@@ -443,6 +444,8 @@ const checks =
443444 });
444445 // Migrated repositories kept as a read-only mirror push to their source.
445446 const mirrorOut = startMirrorOut(app);
447 // …and take the pulls opened on that mirror as pulls here (#9011).
448 const mirrorPulls = startMirrorPulls(app);
446449 // The look: a picture of every environment that comes up green, taken
447450 // through this forge's own listener.
448451 const looks = appsOn ? startLooks(app, { localUrl: servers.url }) : undefined;
@@ -859,6 +862,7 @@ function drain(signal: NodeJS.Signals): Promise<unknown> {
859862 // longer ends it: a person's stop cuts checks off (queued again) at once.
860863 checks?.stop({ drain: signal === "SIGTERM" }),
861864 mirrorOut(),
865 mirrorPulls(),
862866 looks?.stop(),
863867 probes?.stop(),
864868 nightly?.stop(),
src/origins/guest.ts +20 −4
@@ -5,14 +5,17 @@
55 * repository's secrets, and the crew does not review it, until a person —
66 * never an agent — who can write the repository approves it once. The
77 * approval is for the pull, not a head: a later push by the same author
8 * runs with the secrets too. Whether a head's code may run at all is still
9 * `core/pull_trust.ts`'s question, asked per head. Pure data.
8 * runs with the secrets too — except a head moved on a read-only mirror,
9 * which clears it (`forgetGuestApproval`, #9011). Whether a head's code
10 * may run at all is still `core/pull_trust.ts`'s question, asked per head.
11 * Pure data.
1012 */
1113
1214 import type { Database } from "bun:sqlite";
1315 import { getIssueById } from "../core/issues.ts";
1416 import type { Pull } from "../core/pulls.ts";
1517 import { canWrite, type Repo } from "../core/repos.ts";
18 import { foldsIn } from "../core/trusted_sources.ts";
1619 import type { User } from "../core/users.ts";
1720 import { err, ok, type Result } from "../kernel/result.ts";
1821
@@ -37,9 +40,13 @@ export function guestApproval(
3740 : { approvedBy: row.approved_by, approvedAt: row.approved_at };
3841 }
3942
40 /** A guest nobody has approved yet: no secrets, no crew. */
43 /**
44 * A guest nobody has approved yet: no secrets, no crew. A guest whose
45 * account is a trusted source on its forge (`pull_source`, #9011) does not
46 * wait: trust is that approval, for as long as it lasts.
47 */
4148 export function guestWaits(db: Database, pull: Pull): boolean {
42 return isGuestPull(db, pull) && guestApproval(db, pull.id) === undefined;
49 return isGuestPull(db, pull) && guestApproval(db, pull.id) === undefined && !foldsIn(db, pull.id);
4350 }
4451
4552 /** Whether `user` may approve a guest on `repo`: a person who can write it. */
@@ -65,3 +72,12 @@ export function approveGuest(
6572 .run(pull.id, user.id).changes;
6673 return ok(changed === 0 ? "already" : "approved");
6774 }
75
76 /**
77 * Clear the guest `pull`'s approval: its head moved somewhere no one here
78 * pushes (a read-only mirror's pull, #9011), so the head a person approved
79 * is not the one that would run. It waits again until someone looks.
80 */
81 export function forgetGuestApproval(db: Database, pullId: string): void {
82 db.query("DELETE FROM guest_approval WHERE pull_id = ?1").run(pullId);
83 }
src/origins/hook-secret.ts added +38
@@ -0,0 +1,38 @@
1 /**
2 * What an origin's webhook delivery is signed with and sent to, shared by
3 * the receiver (`hook.ts`) and the mirror's watcher (`mirror-pulls.ts`) so
4 * neither imports the other. The secret is derived from the forge's key and
5 * the repository, so nothing is stored.
6 */
7
8 import { createHmac } from "node:crypto";
9 import type { App } from "../app.ts";
10 import type { Repo } from "../core/repos.ts";
11 import { loadKey } from "../kernel/secret.ts";
12
13 export const HOOK_SIGNATURE_HEADER = "x-forge-signature-256";
14
15 /** The secret the origin signs this mirror's deliveries with. */
16 export function hookSecret(app: App, repo: Repo): string {
17 return createHmac("sha256", loadKey(app.config.dataDir))
18 .update(`origin-hook:${repo.id}`)
19 .digest("hex");
20 }
21
22 /** Where the origin delivers: https only, because the origin refuses the rest. */
23 export function hookUrl(app: App, repo: Repo): string | undefined {
24 const base = app.config.url;
25 if (base === undefined || !/^https:\/\//i.test(base)) return undefined;
26 return `${base.replace(/\/+$/, "")}/api/v1/repos/${encodeURIComponent(repo.ownerName)}/${encodeURIComponent(repo.name)}/origin-hook`;
27 }
28
29 /**
30 * The signature a delivery carries, as `sha256=<hex>`: ours, or the bare hex
31 * Forgejo and Gitea send from a mirror's webhook.
32 */
33 export function signatureOf(headers: Headers): string | null {
34 const ours = headers.get(HOOK_SIGNATURE_HEADER);
35 if (ours !== null) return ours;
36 const hex = headers.get("x-forgejo-signature") ?? headers.get("x-gitea-signature");
37 return hex !== null && /^[0-9a-f]{64}$/i.test(hex) ? `sha256=${hex.toLowerCase()}` : null;
38 }
src/origins/hook.ts +3 −17
@@ -9,33 +9,19 @@
99 * derived from the forge's key and the repository, so nothing is stored.
1010 */
1111
12 import { createHmac } from "node:crypto";
1312 import type { App } from "../app.ts";
1413 import { verifySignature } from "../core/webhooks.ts";
1514 import { originForRepo } from "../core/origins.ts";
1615 import type { Repo } from "../core/repos.ts";
17 import { loadKey } from "../kernel/secret.ts";
1816 import { bustOrigin } from "../view/cache.ts";
1917 import { providerForOrigin } from "./credential.ts";
18 import { HOOK_SIGNATURE_HEADER, hookSecret, hookUrl, signatureOf } from "./hook-secret.ts";
2019 import { refreshOriginPull } from "./pull-check.ts";
2120
21 export { HOOK_SIGNATURE_HEADER, hookSecret, hookUrl, signatureOf };
22
2223 /** The most a delivery may say; it names a pull and little else. */
2324 export const HOOK_BODY_MAX = 256 * 1024;
24 export const HOOK_SIGNATURE_HEADER = "x-forge-signature-256";
25
26 /** The secret the origin signs this mirror's deliveries with. */
27 export function hookSecret(app: App, repo: Repo): string {
28 return createHmac("sha256", loadKey(app.config.dataDir))
29 .update(`origin-hook:${repo.id}`)
30 .digest("hex");
31 }
32
33 /** Where the origin delivers: https only, because the origin refuses the rest. */
34 export function hookUrl(app: App, repo: Repo): string | undefined {
35 const base = app.config.url;
36 if (base === undefined || !/^https:\/\//i.test(base)) return undefined;
37 return `${base.replace(/\/+$/, "")}/api/v1/repos/${encodeURIComponent(repo.ownerName)}/${encodeURIComponent(repo.name)}/origin-hook`;
38 }
3925
4026 /** The pull number a delivery names: ours, Gitea's or GitHub's shape. */
4127 export function numberIn(body: unknown): number | undefined {
src/origins/mirror-pulls.ts added +341
@@ -0,0 +1,341 @@
1 /**
2 * Mirror pulls in (#9011): a repository that pushes to a read-only mirror
3 * (`mirror-out.ts`) takes the pulls people open *there* as pulls here. The
4 * mirror stays a plain copy: nothing is pushed to it but trunk's own refs —
5 * here we only read its open pulls, fetch each head into
6 * `refs/heads/mirror-pull/<n>`, and leave one comment on the mirror pull
7 * saying where it went.
8 *
9 * A pull brought in names its author (`origin_author`) and links back
10 * (`origin_html_url`), so it is a guest pull (`guest.ts`): no secrets, no
11 * crew, until a person approves it — unless its author is a trusted source
12 * on the mirror's host, which `pull_source` records.
13 *
14 * It runs by event (the mirror's webhook, `hook.ts`) and by a sweep for
15 * anything missed. Both key on the mirror pull's URL, and one run at a time
16 * per repository, so the two never make two copies of one pull.
17 */
18
19 import type { App } from "../app.ts";
20 import { openPull } from "../actions/pulls.ts";
21 import { getMirrorTarget, type MirrorTarget } from "../core/migrate_mirror.ts";
22 import { getOriginById } from "../core/origins.ts";
23 import { getPull, retargetPull } from "../core/pulls.ts";
24 import { getRepoById, type Repo } from "../core/repos.ts";
25 import { cleanAccount, cleanHost, recordPullSource } from "../core/trusted_sources.ts";
26 import { getUserById } from "../core/users.ts";
27 import { cappedFetch } from "../git/capped-fetch.ts";
28 import { resolveOid, writeRef } from "../git/merge.ts";
29 import { REMOTE_PROTOCOLS } from "../git/remote-env.ts";
30 import { repoDir } from "../git/repos.ts";
31 import { createLog } from "../kernel/log.ts";
32 import { providerForOrigin } from "./credential.ts";
33 import { forgetGuestApproval } from "./guest.ts";
34 import { vettedEnv } from "./git-mirror.ts";
35 import { hookSecret, hookUrl } from "./hook-secret.ts";
36 import { verifySignature } from "../core/webhooks.ts";
37 import type { Provider, RemoteIssue } from "./types.ts";
38
39 const log = createLog("mirror-pulls");
40
41 /** How often every mirror is swept for a pull its webhook did not announce. */
42 export const MIRROR_PULLS_EVERY_MS = 5 * 60_000;
43 /** Pages of open pulls read per sweep; a mirror with more is read next time. */
44 const MAX_PAGES = 5;
45 const FETCH_TIMEOUT_MS = 120_000;
46 /** The most a mirror pull's head may bring that trunk does not already hold. */
47 export const MAX_FETCH_BYTES = 200 * 1024 * 1024;
48 const MAX_TITLE = 200;
49 const MAX_BODY = 60_000;
50
51 /** The branch here that carries mirror pull `n`'s head. */
52 export const mirrorPullBranch = (n: number): string => `mirror-pull/${n}`;
53
54 /** Fetch the mirror's `refs/pull/<n>/head` into `ref`; the oid it landed at. */
55 export type FetchHead = (
56 app: App,
57 repo: Repo,
58 target: MirrorTarget,
59 provider: Provider,
60 number: number,
61 ref: string,
62 ) => Promise<string | undefined>;
63
64 export interface MirrorPullsDeps {
65 readonly provider?: Provider;
66 readonly fetchHead?: FetchHead;
67 }
68
69 export interface MirrorPullsReport {
70 /** Local numbers of the pulls opened this run. */
71 readonly opened: number[];
72 /** Local numbers of the pulls whose head moved on the mirror. */
73 readonly moved: number[];
74 readonly error: string;
75 }
76
77 const running = new Map<string, Promise<MirrorPullsReport>>();
78
79 /**
80 * Bring `repo`'s mirror's open pulls in. A second caller while one runs
81 * waits for it and then runs again, so an event heard mid-run is not lost.
82 */
83 export async function bringMirrorPulls(
84 app: App,
85 repo: Repo,
86 deps: MirrorPullsDeps = {},
87 ): Promise<MirrorPullsReport> {
88 const before = running.get(repo.id);
89 const run = (before ?? Promise.resolve())
90 .catch(() => undefined)
91 .then(() => bringOnce(app, repo, deps));
92 running.set(repo.id, run);
93 try {
94 return await run;
95 } finally {
96 if (running.get(repo.id) === run) running.delete(repo.id);
97 }
98 }
99
100 async function bringOnce(app: App, repo: Repo, deps: MirrorPullsDeps): Promise<MirrorPullsReport> {
101 const opened: number[] = [];
102 const moved: number[] = [];
103 const done = (error = ""): MirrorPullsReport => ({ opened, moved, error });
104 const target = getMirrorTarget(app.db, repo.id);
105 if (!target) return done();
106 const origin = getOriginById(app.db, target.originId);
107 if (!origin) return done("the mirror's origin was disconnected");
108 const opener = getUserById(app.db, origin.userId);
109 if (!opener) return done("the mirror's origin has no account here");
110 let provider = deps.provider;
111 if (provider === undefined) {
112 const made = providerForOrigin(app, origin);
113 if (!made.ok) return done("the mirror's origin needs to be reconnected");
114 provider = made.value;
115 }
116 const host = cleanHost(origin.webUrl) ?? "";
117 const fetchHead = deps.fetchHead ?? fetchFromMirror;
118 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
119
120 let cursor: string | undefined;
121 for (let page = 0; page < MAX_PAGES; page++) {
122 const listed = await provider.listIssues(target.fullName, {
123 kind: "pull",
124 state: "open",
125 ...(cursor === undefined ? {} : { cursor }),
126 });
127 if (!listed.ok) return done(`the mirror's pulls could not be read: ${listed.error.message}`);
128 for (const remote of listed.value.items) {
129 if (remote.pull === undefined || !isMirrorPullUrl(remote.htmlUrl)) continue;
130 try {
131 await bringOne(app, repo, { target, provider, fetchHead, dir, host, opener, opened, moved }, remote);
132 } catch (error) {
133 log.error("mirror pull not brought", { repo: repo.id, number: remote.number, error: String(error) });
134 }
135 }
136 if (listed.value.next === null) break;
137 cursor = listed.value.next;
138 }
139 return done();
140 }
141
142 interface Run {
143 readonly target: MirrorTarget;
144 readonly provider: Provider;
145 readonly fetchHead: FetchHead;
146 readonly dir: string;
147 readonly host: string;
148 readonly opener: NonNullable<ReturnType<typeof getUserById>>;
149 readonly opened: number[];
150 readonly moved: number[];
151 }
152
153 /** A link back is an http(s) URL and nothing else, because a page renders it. */
154 const isMirrorPullUrl = (url: string): boolean => {
155 const parsed = URL.parse(url);
156 return parsed !== null && (parsed.protocol === "https:" || parsed.protocol === "http:") && url.length <= 500;
157 };
158
159 /** The number of the pull here that a mirror pull became, by its URL. */
160 export function broughtFrom(app: App, repoId: string, url: string): number | undefined {
161 return (
162 app.db
163 .query<{ number: number }, [string, string]>(
164 "SELECT number FROM issue WHERE repo_id = ?1 AND kind = 'pull' AND origin_html_url = ?2",
165 )
166 .get(repoId, url)?.number ?? undefined
167 );
168 }
169
170 async function bringOne(app: App, repo: Repo, run: Run, remote: RemoteIssue): Promise<void> {
171 const number = remote.number;
172 const branch = mirrorPullBranch(number);
173 const known = broughtFrom(app, repo.id, remote.htmlUrl);
174 if (known !== undefined) {
175 // Already here: follow a head that moved, while the pull is open here.
176 const pull = getPull(app.db, repo.id, known);
177 if (!pull || pull.state !== "open" || pull.mergedAt !== null) return;
178 if (pull.headOid === remote.pull?.headOid) return;
179 const oid = await run.fetchHead(app, repo, run.target, run.provider, number, `refs/heads/${branch}`);
180 if (oid === undefined || oid === pull.headOid) return;
181 // Nobody here pushed this head: an approval given to the last one does
182 // not carry over to it.
183 forgetGuestApproval(app.db, pull.id);
184 retargetPull(app.db, pull.id, branch, oid);
185 await writeRef(run.dir, `refs/pull/${pull.number}/head`, oid);
186 run.moved.push(pull.number);
187 return;
188 }
189
190 const oid = await run.fetchHead(app, repo, run.target, run.provider, number, `refs/heads/${branch}`);
191 if (oid === undefined) return;
192 // A run that raced this one may have brought it while we fetched.
193 if (broughtFrom(app, repo.id, remote.htmlUrl) !== undefined) return;
194 const base = remote.pull?.baseBranch ?? "";
195 const baseBranch =
196 base !== "" && (await resolveOid(run.dir, `refs/heads/${base}`)) !== undefined ? base : repo.defaultBranch;
197 const author = oneLine(remote.author, 100) || "someone";
198 const title = oneLine(remote.title, MAX_TITLE) || `${run.target.fullName}#${number}`;
199 const body = `From @${author} on ${remote.htmlUrl}\n\n${remote.body}`.slice(0, MAX_BODY);
200 // A guest from its first row: the origin is written with the pull, never after.
201 const made = await openPull(app, run.opener, repo, {
202 title,
203 body,
204 baseBranch,
205 headBranch: branch,
206 origin: { author, htmlUrl: remote.htmlUrl },
207 });
208 if (!made.ok) {
209 log.warn("mirror pull not opened", { repo: repo.id, number, error: String(made.error) });
210 return;
211 }
212 const pull = made.value;
213 const account = cleanAccount(author);
214 if (run.host !== "" && account !== undefined)
215 recordPullSource(app.db, { pullId: pull.id, host: run.host, account, via: [] });
216 run.opened.push(pull.number);
217
218 const here = pullUrl(app, repo, pull.number);
219 const said = await run.provider.createComment(
220 run.target.fullName,
221 number,
222 `This pull continues on ${here ?? `${repo.ownerName}/${repo.name}#${pull.number}`}, where it is reviewed and landed. This repository is a mirror; what lands there comes back here.`,
223 );
224 if (!said.ok) log.warn("mirror pull comment failed", { repo: repo.id, number, error: said.error.message });
225 }
226
227 function pullUrl(app: App, repo: Repo, number: number): string | undefined {
228 const base = app.config.url;
229 if (base === undefined || !/^https?:\/\//i.test(base)) return undefined;
230 return `${base.replace(/\/+$/, "")}/${encodeURIComponent(repo.ownerName)}/${encodeURIComponent(repo.name)}/pulls/${number}`;
231 }
232
233 const oneLine = (text: string, max: number): string =>
234 text.replace(/[\u0000-\u001f\u007f]+/g, " ").trim().slice(0, max);
235
236 /**
237 * The real fetch: the mirror's clone URL, vetted and pinned, with the
238 * origin's credential, and capped in bytes (`capped-fetch.ts`).
239 */
240 const fetchFromMirror: FetchHead = async (app, repo, target, provider, number, ref) => {
241 const authEnv = await provider.gitAuthEnv(target.cloneUrl);
242 const pinned = await vettedEnv(target.cloneUrl, authEnv, REMOTE_PROTOCOLS);
243 if (!pinned.ok) return undefined;
244 const fetched = await cappedFetch({
245 dir: repoDir(app.reposRoot, repo.ownerName, repo.name),
246 url: target.cloneUrl,
247 from: `refs/pull/${number}/head`,
248 ref,
249 maxBytes: MAX_FETCH_BYTES,
250 timeoutMs: FETCH_TIMEOUT_MS,
251 env: pinned.value,
252 });
253 if (!fetched.ok) log.warn("mirror pull head not fetched", { repo: repo.id, number, reason: fetched.reason });
254 return fetched.ok ? fetched.oid : undefined;
255 };
256
257 /**
258 * A delivery from the read-only mirror this repository pushes to: someone
259 * opened or moved a pull there. Believed for "look again" only — the mirror's
260 * open pulls are read and brought in (below), not answered from
261 * the body. The origin is not kept waiting for the fetch.
262 */
263 export async function receiveMirrorHook(
264 app: App,
265 repo: Repo,
266 raw: string,
267 signature: string | null,
268 deps: MirrorPullsDeps = {},
269 ): Promise<boolean> {
270 if (signature === null || !getMirrorTarget(app.db, repo.id)) return false;
271 if (!(await verifySignature(hookSecret(app, repo), raw, signature))) return false;
272 void bringMirrorPulls(app, repo, deps).catch((error: unknown) =>
273 app.log.error("mirror pulls failed", { repoId: repo.id, error: String(error) }),
274 );
275 return true;
276 }
277
278 /**
279 * Ask the mirror this repository pushes to for its pull events, once.
280 * Best effort: the sweep still finds what a missing hook would have said.
281 */
282 export async function watchMirror(app: App, repo: Repo, provider: Provider): Promise<void> {
283 const target = getMirrorTarget(app.db, repo.id);
284 const url = hookUrl(app, repo);
285 if (!target || url === undefined || provider.watch === undefined) return;
286 await provider.watch(target.fullName, { url, secret: hookSecret(app, repo) });
287 }
288
289 /** Every repository that pushes to a mirror. */
290 function mirroredRepos(app: App): Repo[] {
291 return app.db
292 .query<{ repo_id: string }, []>("SELECT repo_id FROM migrate_mirror")
293 .all()
294 .map((r) => getRepoById(app.db, r.repo_id))
295 .filter((r): r is Repo => r !== undefined);
296 }
297
298 /** One sweep over every mirror. Never throws. */
299 export async function sweepMirrorPulls(app: App, deps: MirrorPullsDeps = {}): Promise<void> {
300 for (const repo of mirroredRepos(app)) {
301 try {
302 const report = await bringMirrorPulls(app, repo, deps);
303 if (report.error !== "") log.warn("mirror pulls", { repo: repo.id, error: report.error });
304 } catch (error) {
305 log.error("mirror pulls sweep failed", { repo: repo.id, error: String(error) });
306 }
307 }
308 }
309
310 /** Ask each mirror for its webhook, once per boot. Never throws. */
311 async function watchAll(app: App): Promise<void> {
312 for (const repo of mirroredRepos(app)) {
313 const target = getMirrorTarget(app.db, repo.id);
314 const origin = target === undefined ? undefined : getOriginById(app.db, target.originId);
315 const provider = origin === undefined ? undefined : providerForOrigin(app, origin);
316 if (!provider?.ok) continue;
317 try {
318 await watchMirror(app, repo, provider.value);
319 } catch (error) {
320 log.warn("mirror webhook not set", { repo: repo.id, error: String(error) });
321 }
322 }
323 }
324
325 /** The sweep on a clock, once on boot. Returns the stop. */
326 export function startMirrorPulls(app: App): () => void {
327 let busy = false;
328 let first = true;
329 const tick = (): void => {
330 if (busy) return;
331 busy = true;
332 const watching = first ? watchAll(app) : Promise.resolve();
333 first = false;
334 void watching.then(() => sweepMirrorPulls(app)).finally(() => {
335 busy = false;
336 });
337 };
338 const timer = setInterval(tick, MIRROR_PULLS_EVERY_MS);
339 tick();
340 return () => clearInterval(timer);
341 }
src/origins/providers/forge.ts +1 −1
@@ -182,7 +182,7 @@ export class ForgeProvider extends ForgejoProvider {
182182 return succeed(states);
183183 }
184184
185 async watch(fullName: string, hook: { url: string; secret: string }): Promise<ProviderResult<void>> {
185 override async watch(fullName: string, hook: { url: string; secret: string }): Promise<ProviderResult<void>> {
186186 const base = this.repoPath(fullName);
187187 if (base === undefined) return fail("not_found", `not a repository name: ${fullName}`);
188188 const listed = await this.api.call({ path: `${base}/hooks` });
src/origins/providers/forgejo.ts +26
@@ -729,6 +729,32 @@ export class ForgejoProvider implements Provider {
729729 return succeed(toComment(result.value.data));
730730 }
731731
732 /**
733 * A webhook on the repository for its pull events (#9011), signed with
734 * `secret`; Forgejo and Gitea sign as `X-Forgejo-Signature` /
735 * `X-Gitea-Signature`, bare hex. One already pointing at `url` is kept.
736 */
737 async watch(fullName: string, hook: { url: string; secret: string }): Promise<ProviderResult<void>> {
738 const named = splitFullName(fullName);
739 if (!named) return fail("not_found", `not a repository name: ${fullName}`);
740 const path = `/repos/${seg(named.owner)}/${seg(named.name)}/hooks`;
741 const listed = await this.api.call({ path });
742 if (!listed.ok) return listed;
743 if (asArray(listed.value.data).some((row) => asString(field(field(row, "config"), "url")) === hook.url))
744 return succeed(undefined);
745 const made = await this.api.call({
746 path,
747 method: "POST",
748 body: {
749 type: "gitea",
750 active: true,
751 events: ["pull_request"],
752 config: { url: hook.url, content_type: "json", secret: hook.secret },
753 },
754 });
755 return made.ok ? succeed(undefined) : made;
756 }
757
732758 async setState(
733759 fullName: string,
734760 number: number,
src/web/pages/pull-guest.ts +2 −4
@@ -11,8 +11,7 @@ import type { Pull } from "../../core/pulls.ts";
1111 import type { Repo } from "../../core/repos.ts";
1212 import { approveGuestPull } from "../../actions/pulls.ts";
1313 import {
14 guestApproval,
15 isGuestPull,
14 guestWaits,
1615 mayApproveGuest,
1716 } from "../../origins/guest.ts";
1817 import { repoBase, signInFirst } from "../chrome.ts";
@@ -29,8 +28,7 @@ export const GUEST_APPROVED = "Approved as a guest: its preview starts again wit
2928 /** The guest notice; empty for a pull from this forge, or one already approved. */
3029 export function guestCard(ctx: Ctx, repo: Repo, pull: Pull, action: string): Html {
3130 if (pull.state !== "open" || pull.mergedAt !== null) return html``;
32 if (!isGuestPull(ctx.app.db, pull)) return html``;
33 if (guestApproval(ctx.app.db, pull.id) !== undefined) return html``;
31 if (!guestWaits(ctx.app.db, pull)) return html``;
3432 const writer = mayApproveGuest(ctx.app.db, repo, ctx.auth.user);
3533 return card(
3634 html`<p>${GUEST_WAITS}</p>
test/capped-fetch.test.ts added +76
@@ -0,0 +1,76 @@
1 /**
2 * A head fetched from somewhere nobody vetted is capped in bytes (#9011):
3 * over the cap, the repository is as it was — no ref, no objects, no
4 * scratch left behind; under it, the head is named and its objects are the
5 * repository's own. Real git, over file://.
6 */
7
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { randomBytes } from "node:crypto";
10 import { mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
11 import { tmpdir } from "node:os";
12 import { join } from "node:path";
13 import { cappedFetch } from "../src/git/capped-fetch.ts";
14 import { runGitCapture } from "../src/git/env.ts";
15 import { resolveOid } from "../src/git/merge.ts";
16
17 const git = async (cwd: string, ...args: string[]): Promise<string> => {
18 const env = { GIT_AUTHOR_NAME: "t", GIT_AUTHOR_EMAIL: "t@t", GIT_COMMITTER_NAME: "t", GIT_COMMITTER_EMAIL: "t@t" };
19 const run = await runGitCapture(args, { cwd, env });
20 if (run.code !== 0) throw new Error(`git ${args.join(" ")}: ${run.stderr}`);
21 return run.stdout.trim();
22 };
23
24 describe("capped fetch", () => {
25 let root: string;
26 let work: string;
27 let dest: string;
28 let head: string;
29
30 beforeAll(async () => {
31 root = await mkdtemp(join(tmpdir(), "forge-capped-fetch-"));
32 work = join(root, "work");
33 dest = join(root, "dest.git");
34 await git(root, "init", "--quiet", "-b", "main", work);
35 await writeFile(join(work, "README.md"), "hi\n");
36 await git(work, "add", ".");
37 await git(work, "commit", "--quiet", "-m", "start");
38 await git(root, "clone", "--quiet", "--bare", work, dest);
39 // The pull: half a megabyte nothing can compress.
40 await writeFile(join(work, "big.bin"), randomBytes(512 * 1024));
41 await git(work, "add", ".");
42 await git(work, "commit", "--quiet", "-m", "big");
43 head = await git(work, "rev-parse", "HEAD");
44 await git(work, "update-ref", "refs/pull/1/head", head);
45 });
46
47 afterAll(async () => {
48 await rm(root, { recursive: true, force: true });
49 });
50
51 const fetch = (maxBytes: number) =>
52 cappedFetch({
53 dir: dest,
54 url: `file://${work}`,
55 from: "refs/pull/1/head",
56 ref: "refs/heads/mirror-pull/1",
57 maxBytes,
58 timeoutMs: 30_000,
59 env: {},
60 });
61
62 test("over the cap, the repository is as it was", async () => {
63 expect(await fetch(64 * 1024)).toEqual({ ok: false, reason: "too_large" });
64 expect(await resolveOid(dest, "refs/heads/mirror-pull/1")).toBeUndefined();
65 expect((await runGitCapture(["cat-file", "-e", head], { cwd: dest })).code).not.toBe(0);
66 expect((await readdir(dest)).filter((n) => n.startsWith("incoming-"))).toEqual([]);
67 });
68
69 test("under the cap, the head is named and its objects are the repository's", async () => {
70 expect(await fetch(8 * 1024 * 1024)).toEqual({ ok: true, oid: head });
71 expect(await resolveOid(dest, "refs/heads/mirror-pull/1")).toBe(head);
72 expect((await runGitCapture(["cat-file", "-e", `${head}:big.bin`], { cwd: dest })).code).toBe(0);
73 expect((await runGitCapture(["fsck", "--no-dangling"], { cwd: dest })).code).toBe(0);
74 expect((await readdir(dest)).filter((n) => n.startsWith("incoming-"))).toEqual([]);
75 });
76 });
test/fake-origin.ts +15
@@ -199,6 +199,8 @@ export interface FakeState {
199199 conditionalHits: number;
200200 /** Refuse the next write with this message. */
201201 refuseWrite: string;
202 /** Forgejo webhooks as created, keyed `owner/name`. */
203 hooks: Map<string, Record<string, unknown>[]>;
202204 /**
203205 * How many `getRepo` calls a freshly forked repository 404s for before it
204206 * appears. GitHub creates a fork asynchronously and answers 202 with a
@@ -272,6 +274,7 @@ export function newFakeState(
272274 etagsEnabled: false,
273275 conditionalHits: 0,
274276 refuseWrite: "",
277 hooks: new Map(),
275278 forkDelayReads: 0,
276279 pendingForks: new Map(),
277280 refuseFork: "",
@@ -495,6 +498,18 @@ function route(
495498 return jsonResponse({ message: "Not Found" }, 404);
496499 }
497500
501 // Forgejo's webhooks: listed, and made with a `config` holding url and secret.
502 if (tail === "/hooks" && state.kind !== "github") {
503 const hooks = state.hooks.get(fullName) ?? [];
504 if (method === "GET") return jsonResponse(hooks);
505 if (method === "POST") {
506 if (state.refuseWrite !== "") return refuse(state);
507 const made = { id: state.nextId++, ...(parse(bodyText) as Record<string, unknown>) };
508 state.hooks.set(fullName, [...hooks, made]);
509 return jsonResponse(made, 201);
510 }
511 }
512
498513 if (tail === "" && method === "PATCH") {
499514 if (repo.admin === false)
500515 return jsonResponse({ message: "Must have admin rights to Repository." }, 403);
test/mirror-pulls.test.ts added +264
@@ -0,0 +1,264 @@
1 /**
2 * A pull opened on the read-only Forgejo mirror becomes a pull here (#9011):
3 * its title, its author's name and a link back; a guest unless the author is
4 * a trusted source; one pull however often the event and the sweep fire; a
5 * head that moves there moves here. The mirror gets one comment and its
6 * webhook, never a push. Against the fake Forgejo; the git fetch is the one
7 * seam, standing in for `refs/pull/<n>/head`.
8 */
9
10 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
11 import { createHmac } from "node:crypto";
12 import { mkdtemp, rm } from "node:fs/promises";
13 import { tmpdir } from "node:os";
14 import { join } from "node:path";
15 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
16 import { createApp, type App } from "../src/app.ts";
17 import { getIssue } from "../src/core/issues.ts";
18 import { setMirrorTarget } from "../src/core/migrate_mirror.ts";
19 import { createOrigin } from "../src/core/origins.ts";
20 import { getPull } from "../src/core/pulls.ts";
21 import type { Repo } from "../src/core/repos.ts";
22 import { trustSource, untrust } from "../src/core/trusted_sources.ts";
23 import { createUser, type User } from "../src/core/users.ts";
24 import { resolveOid } from "../src/git/merge.ts";
25 import { repoDir } from "../src/git/repos.ts";
26 import { approveGuest, guestWaits } from "../src/origins/guest.ts";
27 import { hookSecret, receiveOriginHook, signatureOf } from "../src/origins/hook.ts";
28 import { bringMirrorPulls, broughtFrom, receiveMirrorHook, watchMirror, type FetchHead } from "../src/origins/mirror-pulls.ts";
29 import { ForgejoProvider } from "../src/origins/providers/forgejo.ts";
30 import { staticCredential } from "../src/origins/types.ts";
31 import { newFakeState, serveFakeForge, type FakeForge, type FakeState } from "./fake-origin.ts";
32 import { quietLogs } from "./helpers/globals.ts";
33
34 quietLogs();
35
36 const TOKEN = "tok-mirror-1234";
37
38 describe("pulls opened on the mirror come here", () => {
39 let dataDir: string;
40 let app: App;
41 let alice: User;
42 let repo: Repo;
43 let fake: FakeForge;
44 let state: FakeState;
45 let fetches = 0;
46
47 // The mirror's `refs/pull/<n>/head`: a commit on the branch the pull rides
48 // here, a new one only when the mirror's head sha moved.
49 const served = new Map<number, string>();
50 const fetchHead: FetchHead = async (a, r, _t, _p, number, ref) => {
51 const dir = repoDir(a.reposRoot, r.ownerName, r.name);
52 const sha = state.issues.get("trevato-apps/forge")?.find((i) => i.number === number)?.pull?.headSha ?? "";
53 if (served.get(number) === sha) return resolveOid(dir, ref);
54 served.set(number, sha);
55 fetches++;
56 const branch = ref.replace(/^refs\/heads\//, "");
57 const made = await commitToBranch(a, alice, r, {
58 branch,
59 message: `Mirror pull ${number}, fetch ${fetches}`,
60 files: [{ path: `m${number}-${fetches}.md`, content: "from the mirror\n" }],
61 });
62 if (!made.ok) return undefined;
63 return resolveOid(dir, ref);
64 };
65 const provider = () =>
66 new ForgejoProvider(
67 staticCredential({ kind: "forgejo", apiUrl: fake.apiUrl, webUrl: fake.url, token: TOKEN }),
68 );
69 const bring = () => bringMirrorPulls(app, repo, { provider: provider(), fetchHead });
70 const openOnMirror = (number: number, user: string, headSha = `${number}`.padStart(40, "a")) => {
71 const all = state.issues.get("trevato-apps/forge") ?? [];
72 all.push({
73 id: 500 + number,
74 number,
75 title: `Fix ${number}`,
76 body: "It was broken.",
77 user,
78 pull: { base: "main", head: `fix-${number}`, headRepo: `${user}/forge`, headSha },
79 });
80 state.issues.set("trevato-apps/forge", all);
81 };
82 const broughtCount = () =>
83 app.db
84 .query<{ n: number }, [string]>(
85 "SELECT count(*) AS n FROM issue WHERE repo_id = ?1 AND kind = 'pull' AND origin_html_url IS NOT NULL",
86 )
87 .get(repo.id)!.n;
88
89 beforeAll(async () => {
90 dataDir = await mkdtemp(join(tmpdir(), "forge-mirror-pulls-"));
91 app = createApp({ dataDir, port: 0, url: "https://forge.test" });
92 state = newFakeState("forgejo", "trevato", TOKEN);
93 state.repos.push({ id: 1, fullName: "trevato-apps/forge", cloneUrl: "http://127.0.0.1/none.git", admin: true });
94 fake = serveFakeForge(state);
95 const made = await createUser(app.db, {
96 origin: "seed",
97 trusted: true,
98 username: "alice",
99 email: "[email protected]",
100 password: "a-good-passphrase",
101 adminIfFirst: true,
102 });
103 if (!made.ok) throw new Error(made.error);
104 alice = made.value;
105 const r = await createRepo(app, alice, { name: "forge", visibility: "public" });
106 if (!r.ok) throw new Error(String(r.error));
107 repo = r.value;
108 const seeded = await commitToBranch(app, alice, repo, {
109 message: "Start",
110 files: [{ path: "README.md", content: "forge\n" }],
111 });
112 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
113 const origin = createOrigin(app.db, dataDir, {
114 userId: alice.id,
115 kind: "forgejo",
116 apiUrl: fake.apiUrl,
117 webUrl: fake.url,
118 login: "trevato",
119 token: TOKEN,
120 });
121 if (!origin.ok) throw new Error(origin.error);
122 setMirrorTarget(app.db, {
123 repoId: repo.id,
124 originId: origin.value.id,
125 fullName: "trevato-apps/forge",
126 cloneUrl: `${fake.url}/trevato-apps/forge.git`,
127 tips: {},
128 });
129 }, 60_000);
130
131 afterAll(async () => {
132 fake.stop();
133 app.close();
134 await rm(dataDir, { recursive: true, force: true });
135 });
136
137 test("a mirror pull becomes a guest pull with its author and a link back, and the mirror is told where", async () => {
138 openOnMirror(1, "tdobbertin-espo");
139 // Whatever starts a check or a preview hears `pull.opened`: by then the
140 // pull must already be a guest, or it could start with secrets.
141 const seen: Array<string | null> = [];
142 const stop = app.bus.subscribe((event) => {
143 if (event.kind !== "pull.opened" || event.repoId !== repo.id) return;
144 const number = (event.payload as { number: number }).number;
145 const issue = getIssue(app.db, repo.id, number)!;
146 seen.push(issue.originAuthor);
147 expect(guestWaits(app.db, getPull(app.db, repo.id, number)!)).toBe(true);
148 });
149 const report = await bring();
150 stop();
151 expect(seen).toEqual(["tdobbertin-espo"]);
152 expect(report.error).toBe("");
153 expect(report.opened.length).toBe(1);
154 const pull = getPull(app.db, repo.id, report.opened[0]!)!;
155 expect(pull.title).toBe("Fix 1");
156 expect(pull.headBranch).toBe("mirror-pull/1");
157 expect(pull.baseBranch).toBe("main");
158 const issue = getIssue(app.db, repo.id, pull.number)!;
159 expect(issue.originAuthor).toBe("tdobbertin-espo");
160 expect(issue.originUrl).toBe(`${fake.url}/1`);
161 expect(issue.body).toContain(`From @tdobbertin-espo on ${fake.url}/1`);
162 expect(guestWaits(app.db, pull)).toBe(true);
163
164 const said = state.comments.get("trevato-apps/forge") ?? [];
165 expect(said.length).toBe(1);
166 expect(said[0]!.issueNumber).toBe(1);
167 expect(said[0]!.body).toContain(`https://forge.test/alice/forge/pulls/${pull.number}`);
168 // The only write the mirror saw is that comment.
169 const writes = state.requests.filter((q) => q.method !== "GET");
170 expect(writes.map((q) => `${q.method} ${q.path}`)).toEqual([
171 "POST /api/v1/repos/trevato-apps/forge/issues/1/comments",
172 ]);
173 });
174
175 test("the event and the sweep firing together still make one pull", async () => {
176 openOnMirror(2, "someone");
177 const before = broughtCount();
178 const [a, b] = await Promise.all([bring(), bring()]);
179 expect(a.opened.length + b.opened.length).toBe(1);
180 expect(broughtCount()).toBe(before + 1);
181 await bring();
182 expect(broughtCount()).toBe(before + 1);
183 expect((state.comments.get("trevato-apps/forge") ?? []).filter((c) => c.issueNumber === 2).length).toBe(1);
184 });
185
186 test("a trusted source's mirror pull does not wait as a guest", async () => {
187 const host = new URL(fake.url).host;
188 const until = new Date(Date.now() + 7 * 86_400_000).toISOString().slice(0, 10);
189 const trusted = trustSource(app.db, { host, account: "friend", until, setBy: alice.id });
190 expect(typeof trusted).toBe("object");
191 openOnMirror(3, "friend");
192 const report = await bring();
193 const pull = getPull(app.db, repo.id, report.opened[0]!)!;
194 expect(getIssue(app.db, repo.id, pull.number)!.originAuthor).toBe("friend");
195 expect(guestWaits(app.db, pull)).toBe(false);
196
197 // Trust that lapses (or is taken back) makes it wait again.
198 app.db
199 .query("UPDATE trusted_source SET until = ?1 WHERE host = ?2 AND account = 'friend'")
200 .run(new Date(Date.now() - 60_000).toISOString(), host);
201 expect(guestWaits(app.db, pull)).toBe(true);
202 const renewed = trustSource(app.db, { host, account: "friend", until, setBy: alice.id });
203 expect(guestWaits(app.db, pull)).toBe(false);
204 if (typeof renewed === "object") untrust(app.db, renewed.id);
205 expect(guestWaits(app.db, pull)).toBe(true);
206 });
207
208 test("an untrusted guest still waits while someone else is trusted", async () => {
209 const pull = getPull(app.db, repo.id, broughtFrom(app, repo.id, `${fake.url}/1`)!)!;
210 expect(guestWaits(app.db, pull)).toBe(true);
211 });
212
213 test("a head that moves on the mirror moves the pull here", async () => {
214 const mirror = state.issues.get("trevato-apps/forge")!.find((i) => i.number === 1)!;
215 const number = app.db
216 .query<{ number: number }, [string]>("SELECT number FROM issue WHERE origin_html_url = ?1")
217 .get(`${fake.url}/1`)!.number;
218 const approved = getPull(app.db, repo.id, number)!;
219 const was = approved.headOid;
220 expect(approveGuest(app.db, repo, approved, alice).ok).toBe(true);
221 expect(guestWaits(app.db, approved)).toBe(false);
222 mirror.pull = { ...mirror.pull!, headSha: "b".repeat(40) };
223 const report = await bring();
224 expect(report.moved).toEqual([number]);
225 const now = getPull(app.db, repo.id, number)!;
226 // The head a person approved is not the one that would run now.
227 expect(guestWaits(app.db, now)).toBe(true);
228 expect(now.headOid).not.toBe(was);
229 expect(await resolveOid(repoDir(app.reposRoot, repo.ownerName, repo.name), `refs/pull/${number}/head`)).toBe(
230 now.headOid,
231 );
232 });
233
234 test("the mirror's webhook is set once, and its signed delivery is believed", async () => {
235 await watchMirror(app, repo, provider());
236 await watchMirror(app, repo, provider());
237 const hooks = state.hooks.get("trevato-apps/forge") ?? [];
238 expect(hooks.length).toBe(1);
239 const config = hooks[0]!["config"] as Record<string, string>;
240 expect(config["url"]).toBe("https://forge.test/api/v1/repos/alice/forge/origin-hook");
241 expect(config["secret"]).toBe(hookSecret(app, repo));
242
243 const raw = JSON.stringify({ action: "opened", number: 9 });
244 const mac = createHmac("sha256", hookSecret(app, repo)).update(raw).digest("hex");
245 const good = signatureOf(new Headers({ "x-forgejo-signature": mac }));
246 expect(good).toBe(`sha256=${mac}`);
247 // Not an origin's delivery: this repository mirrors nothing.
248 expect(await receiveOriginHook(app, repo, raw, good)).toBe(false);
249 const bad = signatureOf(new Headers({ "x-forgejo-signature": "0".repeat(64) }));
250 expect(await receiveMirrorHook(app, repo, raw, bad, { provider: provider(), fetchHead })).toBe(false);
251 expect(await receiveMirrorHook(app, repo, raw, null, { provider: provider(), fetchHead })).toBe(false);
252 expect(signatureOf(new Headers({ "x-forgejo-signature": "not hex" }))).toBeNull();
253
254 // A signed delivery brings the pull it announced.
255 openOnMirror(9, "hooked");
256 const before = broughtCount();
257 expect(await receiveMirrorHook(app, repo, raw, good, { provider: provider(), fetchHead })).toBe(true);
258 // Runs on one repository queue: this one waits for the delivery's, and finds nothing left.
259 const after = await bring();
260 expect(after.opened).toEqual([]);
261 expect(broughtCount()).toBe(before + 1);
262 expect(getIssue(app.db, repo.id, broughtFrom(app, repo.id, `${fake.url}/9`)!)!.originAuthor).toBe("hooked");
263 });
264 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs