trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns7g rEventsFlows

The packs' upstreams are read-only forge mirrors under the packs owner, re-synced on a schedule #10589

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/10530-the-packs-upstreams-are-read-only-forge → main 5 commits fork 2 threads · 2 unresolved

  • Merged. scout/10530-the-packs-upstreams-are-read-only-forge was merged into main as 9e162c0.

Review

Sign in to review this change.
14 files changed +576 −7
docs/features/packs.md +6
@@ -10,3 +10,9 @@ Two ship in `packs/`: **games** (2048, hextris, react-tetris) and **business** (
1010 - Code: `src/packs/`, `src/web/pages/packs.ts`, `scripts/pack.ts`. Tests: `test/packs.test.ts`, `test/packs-up.test.ts` (the whole path against a fake source for both packs, `up()` itself with a real child forge on a free port, the owner reused on a second run, a stored credential, a partial deploy finished, a new upstream commit showing as drift), `test/packs-hands.test.ts` (the git credential is scoped to the forge and never sent to a pack's source; only 409 means "exists"; the terminal screen).
1111
1212 Not done yet: a hosted "deploy this pack" press, a stable follow link from each app back to its source, the guest sign-up gate and the crew filtering guest pulls on automode (owner decisions on #9428).
13
14 ## The upstream mirrors (#10530)
15
16 An admin brings each https `source` in `packs/*/pack.json` in once as a public repository `packs/<app>` (owner `packs`, an account nobody signs in as), from any of three doors: the "Mirror the upstreams" button on `/packs`, the "Mirror the packs' upstreams" screen in the terminal palette (admins only), or `repo_settings` with `{ mirrorPacks: true }` over MCP, which answers `{ made, kept, failed, says }`. The request only reads each upstream's default branch, creates the repository and its follow, and stops at 20 seconds in all: what it did not reach is listed as failed ("press again"). The first sync is the scheduled follow's, as is every later one; it moves the default branch to the upstream's tip instead of opening a pull, and a failed sync is noted on the follow's row. Running it again leaves existing mirrors as they are.
17
18 Read-only is enforced where writes are decided: `canWrite` is false for every `packs/*` repository except for its owner, which only the follow acts as, so pushes (http and ssh, told why), web file edits, the contents and branch API, and opening or merging a pull are all refused, admins included. Open on purpose: issues and comments, and a repository admin's settings (visibility, delete, transfer). A mirror tracks the branch that was the upstream's HEAD when it was made; if the upstream renames its default branch, delete the mirror and mirror again.
src/actions/upstream_follow.ts +11
@@ -44,6 +44,7 @@ import { isSafeOid, isSafeRef } from "../git/guard.ts";
4444 import { aheadCount, mergeBase, mergeable, resolveOid, writeRef } from "../git/merge.ts";
4545 import { REMOTE_PROTOCOLS } from "../git/remote-env.ts";
4646 import { repoDir } from "../git/repos.ts";
47 import { isPacksMirror } from "../packs/owner.ts";
4748 import {
4849 BIG_FOLD_COMMITS,
4950 ownHost,
@@ -412,6 +413,16 @@ async function checkTarget(app: App, repo: Repo, follow: Target): Promise<Result
412413 }
413414 const staged = await resolveOid(dir, ref);
414415 if (staged === undefined) return failed("git");
416 // A packs mirror is a copy, not a fork: its default branch is the upstream's, moved, never proposed.
417 if (follow.sourceId === undefined && isPacksMirror(repo)) {
418 if (
419 (await resolveOid(dir, `refs/heads/${repo.defaultBranch}`)) !== staged &&
420 !(await writeRef(dir, `refs/heads/${repo.defaultBranch}`, staged))
421 )
422 return failed("git", "The mirror's branch could not be moved.");
423 follow.note({ at, tip: staged, behind: 0, error: "" });
424 return ok({ tip: staged, behind: 0, outcome: "none" });
425 }
415426 const main = await resolveOid(dir, `refs/heads/${repo.defaultBranch}`);
416427 if (main === undefined) return failed("git", "The default branch has no commits yet.");
417428 const behind = await aheadCount(dir, main, staged);
src/core/repos.ts +3
@@ -5,6 +5,7 @@
55 * repo. Deletion inverts the order — row first, directory second.
66 */
77
8 import { isPacksMirror } from "../packs/owner.ts";
89 import { contributionForksSince, isContributionFork } from "./contribute.ts";
910 import type { Database } from "bun:sqlite";
1011 import { rmSync } from "node:fs";
@@ -1116,6 +1117,8 @@ export function canWrite(
11161117 repo: Repo,
11171118 user: User | undefined,
11181119 ): boolean {
1120 // A packs mirror is a copy of an upstream: only the follow, acting as its owner, moves it (#10530).
1121 if (isPacksMirror(repo)) return user !== undefined && user.id === repo.ownerId;
11191122 if (personalOrDirect(repo, user)) return true;
11201123 return accessAllows(orgAccessFor(db, repo, user), "write");
11211124 }
src/http/mcp/tools/repo_settings.ts +8 −3
@@ -1,13 +1,14 @@
11 import { updateRepoSettings } from "../../../actions/repos.ts";
22 import { TRANSFER_SAID, transferRepository } from "../../../actions/transfer.ts";
3 import { mirrorPacks, reportWords } from "../../../packs/mirror.ts";
34 import { good, bad, type Tool, REPO_ARG, repoArg, isAnswer } from "../shared.ts";
45 import { VISIBILITY_WORDS } from "../../../core/arg_words.ts";
56
67 export const tool: Tool = {
78 name: "repo_settings",
8 doc: "`{ repo, description?, visibility?, newOwner? }` (admin): `newOwner` moves the repository to that account or org as `POST /api/v1/repos/:owner/:name/transfer` does, with its codes and sentences (`name_taken`, `same_owner`, `move_failed`, `forbidden_owner`, `unknown_owner`) and answers `{ repo, description, visibility }` at the new path, and cannot be combined with the others; otherwise change a repository's description and visibility as `PATCH /api/v1/repos/:owner/:name` does, for the same callers (a repository admin); answers `{ repo, description, visibility }` (with none to change it only reads); a visibility other than public or private is `invalid`",
9 doc: "`{ mirrorPacks: true }` (admin) brings each https source in packs/*/pack.json in once as a read-only `packs/<app>` that follows its upstream, and answers `{ made, kept, failed, says }`; running it again makes nothing new. `{ repo, description?, visibility?, newOwner? }` (admin): `newOwner` moves the repository to that account or org as `POST /api/v1/repos/:owner/:name/transfer` does, with its codes and sentences (`name_taken`, `same_owner`, `move_failed`, `forbidden_owner`, `unknown_owner`) and answers `{ repo, description, visibility }` at the new path, and cannot be combined with the others; otherwise change a repository's description and visibility as `PATCH /api/v1/repos/:owner/:name` does, for the same callers (a repository admin); answers `{ repo, description, visibility }` (with none to change it only reads); a visibility other than public or private is `invalid`",
910 description:
10 "Read, change or move (newOwner) a repo: PATCH /api/v1/repos/:owner/:name (admin).",
11 "Read, change or move (newOwner) a repo: PATCH /api/v1/repos/:owner/:name (admin); mirrorPacks mirrors the packs' upstreams.",
1112 inputSchema: {
1213 type: "object",
1314 properties: {
@@ -15,12 +16,16 @@ export const tool: Tool = {
1516 description: { type: "string", maxLength: 2000 },
1617 visibility: { type: "string", enum: ["public", "private"] },
1718 newOwner: { type: "string", maxLength: 64 },
19 mirrorPacks: { type: "boolean" },
1820 },
19 required: ["repo"],
2021 },
2122 writes: true,
2223 needsAdminScope: true,
2324 async run(ctx, user, args) {
25 if (args["mirrorPacks"] === true) {
26 const done = await mirrorPacks(ctx.app, user);
27 return done.ok ? good({ ...done.value, says: reportWords(done.value) }) : bad("not_found", "no such tool here");
28 }
2429 const repo = repoArg(ctx, user, args);
2530 if (isAnswer(repo)) return repo;
2631 const { description, visibility, newOwner } = args;
src/http/server.ts +4 −1
@@ -283,7 +283,8 @@ import { liveChipsFragment, liveMovingFragment } from "../web/studio/living.ts";
283283 import { HOME_REGIONS } from "../web/studio/home_last.ts";
284284 import { serviceWorker } from "../web/sw.ts";
285285 import { explorePage } from "../web/studio/explore.ts";
286 import { packPage, packsPage } from "../web/pages/packs.ts";
286 import { isPacksMirror, MIRROR_PUSH_REFUSED } from "../packs/owner.ts";
287 import { packPage, packsMirror, packsPage } from "../web/pages/packs.ts";
287288 import {
288289 makeItYoursPage,
289290 makeItYoursSubmit,
@@ -822,6 +823,7 @@ export function buildRouter(): Router<Handler> {
822823 router.add("GET", "/explore", explorePage);
823824 // Forge packs: premade forges to deploy, and the way to help build their apps.
824825 router.add("GET", "/packs", packsPage);
826 router.add("POST", "/packs/mirror", packsMirror);
825827 router.add("GET", "/packs/:name", packPage);
826828 router.add("POST", "/front", frontUpdate);
827829 router.add("POST", "/home/archive", homeArchive);
@@ -1658,6 +1660,7 @@ async function withGitAccess(
16581660 if (!repo) return ctx.auth.user ? notFound() : gitChallenge();
16591661 if (write) {
16601662 if (!ctx.auth.user) return gitChallenge();
1663 if (isPacksMirror(repo) && canRead(ctx.app.db, repo, ctx.auth.user)) return gitRefused(ctx, MIRROR_PUSH_REFUSED);
16611664 if (
16621665 !canWrite(ctx.app.db, repo, ctx.auth.user) ||
16631666 !scopeAllows(ctx.auth.scope ?? "ro", "rw")
src/packs/mirror.ts added +147
@@ -0,0 +1,147 @@
1 /**
2 * Bringing the packs' upstreams in (#10530): each https `source` named in
3 * packs/*\/pack.json becomes one public repository `packs/<app>` that follows
4 * its upstream through the follow the forge already has
5 * (`actions/upstream_follow.ts`). The request only creates the repository and
6 * its follow row; the scheduled follow makes the first sync. The follow moves a mirror's default branch
7 * to the upstream's tip on every check and opens no pull; nobody pushes to
8 * one (`owner.ts`). An admin's action, and idempotent: a mirror that exists
9 * is kept as it is, so running it twice makes nothing new.
10 */
11
12 import type { App } from "../app.ts";
13 import { createRepo, getRepo } from "../core/repos.ts";
14 import { putFollow, upstreamUrl } from "../core/upstream_follow.ts";
15 import { createUser, getUserByName, type User } from "../core/users.ts";
16 import { runGitCapture } from "../git/env.ts";
17 import { isSafeName, isSafeRef } from "../git/guard.ts";
18 import { REMOTE_PROTOCOLS } from "../git/remote-env.ts";
19 import { vettedEnv } from "../origins/git-mirror.ts";
20 import { err, ok, type Result } from "../kernel/result.ts";
21 import { catalog } from "./catalog.ts";
22 import { PACKS_OWNER } from "./owner.ts";
23
24 export interface MirrorReport {
25 /** Mirrors made by this run, as `packs/<app>`. */
26 readonly made: readonly string[];
27 /** Mirrors that were already there, left as they are. */
28 readonly kept: readonly string[];
29 /** Apps that could not be brought in, and the one line why. */
30 readonly failed: readonly { readonly app: string; readonly why: string }[];
31 }
32
33 /** The whole run's time: what it has not reached by then is left for the next press. */
34 export const MIRROR_BUDGET_MS = 20_000;
35 const PER_UPSTREAM_MS = 8_000;
36
37 export const MIRROR_FORBIDDEN = "Only an admin brings the packs' upstreams in.";
38
39 /** The mirrors' owner: an account nobody signs in as, made on first use. */
40 async function mirrorOwner(app: App): Promise<User | undefined> {
41 const found = getUserByName(app.db, PACKS_OWNER);
42 if (found !== undefined) return found;
43 const made = await createUser(app.db, {
44 origin: "seed",
45 trusted: true,
46 allowReserved: true,
47 username: PACKS_OWNER,
48 email: `${PACKS_OWNER}@forge.invalid`,
49 displayName: "Forge packs",
50 });
51 return made.ok ? made.value : getUserByName(app.db, PACKS_OWNER);
52 }
53
54 /** The upstream's own default branch, or `main` when it does not say. */
55 async function upstreamBranch(url: string, timeoutMs: number): Promise<Result<string, string>> {
56 const pinned = await vettedEnv(url, {}, REMOTE_PROTOCOLS);
57 if (!pinned.ok) return err(pinned.error.message);
58 const listed = await runGitCapture(["ls-remote", "--symref", "--", url, "HEAD"], {
59 env: pinned.value,
60 timeoutMs,
61 });
62 if (listed.code !== 0)
63 return err(listed.stderr.split("\n").find((l) => l.trim() !== "")?.slice(0, 200) ?? "the upstream could not be read");
64 const named = /^ref: refs\/heads\/(\S+)\tHEAD$/m.exec(listed.stdout)?.[1];
65 return ok(named !== undefined && isSafeRef(named) && named.length <= 200 ? named : "main");
66 }
67
68 /** Every distinct app with an https source across the packs, first named first. */
69 export function packSources(): { readonly name: string; readonly source: string; readonly description: string }[] {
70 const seen = new Set<string>();
71 const out: { name: string; source: string; description: string }[] = [];
72 for (const pack of catalog().packs)
73 for (const a of pack.apps) {
74 const key = a.name.toLowerCase();
75 if (a.source === undefined || seen.has(key)) continue;
76 seen.add(key);
77 out.push({ name: a.name, source: a.source, description: a.description });
78 }
79 return out;
80 }
81
82 /** Bring each pack source in once as a read-only `packs/<app>`. Admin only. */
83 export async function mirrorPacks(
84 app: App,
85 actor: User | undefined,
86 sources: ReturnType<typeof packSources> = packSources(),
87 budgetMs = MIRROR_BUDGET_MS,
88 ): Promise<Result<MirrorReport, "not_found">> {
89 if (actor === undefined || !actor.isAdmin || actor.kind === "agent") return err("not_found");
90 const owner = await mirrorOwner(app);
91 if (owner === undefined) return err("not_found");
92 const made: string[] = [];
93 const kept: string[] = [];
94 const failed: { app: string; why: string }[] = [];
95 const stopAt = Date.now() + budgetMs;
96 for (const a of sources) {
97 const full = `${PACKS_OWNER}/${a.name}`;
98 if (getRepo(app.db, PACKS_OWNER, a.name) !== undefined) {
99 kept.push(full);
100 continue;
101 }
102 const url = upstreamUrl(a.source);
103 if (url === undefined || !isSafeName(a.name)) {
104 failed.push({ app: a.name, why: "its source is not an address a mirror may follow" });
105 continue;
106 }
107 const left = stopAt - Date.now();
108 if (left < 1_000) {
109 failed.push({ app: a.name, why: "not reached in time; press again" });
110 continue;
111 }
112 const branch = await upstreamBranch(url, Math.min(PER_UPSTREAM_MS, left));
113 if (!branch.ok) {
114 failed.push({ app: a.name, why: branch.error });
115 continue;
116 }
117 const repo = await createRepo(app.db, app.reposRoot, {
118 ownerId: owner.id,
119 name: a.name,
120 visibility: "public",
121 description: `${a.description} A read-only mirror of ${a.source}.`.slice(0, 2000),
122 defaultBranch: branch.value,
123 });
124 if (!repo.ok) {
125 failed.push({ app: a.name, why: repo.error });
126 continue;
127 }
128 putFollow(app.db, {
129 repoId: repo.value.id,
130 url,
131 branch: branch.value,
132 enabled: true,
133 byId: owner.id,
134 originId: "",
135 });
136 // Never checked, so the scheduled follow's next pass makes the first sync; a failure is noted on its row.
137 made.push(full);
138 }
139 return ok({ made, kept, failed });
140 }
141
142 /** One line for the report, for the page and the tool alike. */
143 export function reportWords(r: MirrorReport): string {
144 const parts = [`${r.made.length} brought in`, `${r.kept.length} already there`];
145 if (r.failed.length > 0) parts.push(`${r.failed.length} failed (${r.failed.map((f) => `${f.app}: ${f.why}`).join("; ")})`);
146 return `${parts.join(", ")}.`;
147 }
src/packs/owner.ts added +15
@@ -0,0 +1,15 @@
1 /**
2 * Who the packs' upstream mirrors belong to (#10530): the account `packs`.
3 * A repository under it is a read-only copy of an outside project; the only
4 * thing that moves its branches is the scheduled follow. Pure data.
5 */
6
7 export const PACKS_OWNER = "packs";
8
9 /** Whether a repository is one of the packs' mirrors. */
10 export const isPacksMirror = (repo: { readonly ownerName: string }): boolean =>
11 repo.ownerName.toLowerCase() === PACKS_OWNER;
12
13 /** What a push to a mirror is told, at every door. */
14 export const MIRROR_PUSH_REFUSED =
15 "This is a read-only mirror of an upstream project: it follows its upstream and takes no pushes.";
src/tui/screens/entries.ts +2
@@ -40,6 +40,7 @@ import { entry as economics } from "./economics.ts";
4040 import { entry as loopStalls } from "./loopstalls.ts";
4141 import { entry as slowPagesEntry } from "./slowpages.ts";
4242 import { entry as maintenance } from "./maintenance.ts";
43 import { entry as mirrorPacksEntry } from "./mirrorpacks.ts";
4344 import { modelCeilingEntry, modelGrantEntry } from "./model.ts";
4445 import { entry as orgGrants } from "./orggrants.ts";
4546 import { entry as orgTeams } from "./orgteams.ts";
@@ -92,6 +93,7 @@ export const ENTRIES: readonly PaletteEntry[] = [
9293 telemetry,
9394 resilience,
9495 maintenance,
96 mirrorPacksEntry,
9597 valves,
9698 access,
9799 edgeOnly,
src/tui/screens/mirrorpacks.ts added +71
@@ -0,0 +1,71 @@
1 /**
2 * "Mirror the packs' upstreams" in the terminal (#10530): the button on
3 * /packs, through the same action (`packs/mirror.ts`). Offered to site admins
4 * only; the action refuses anyone else. Enter runs it, Esc or q closes.
5 */
6
7 import { MIRROR_FORBIDDEN, mirrorPacks, reportWords } from "../../packs/mirror.ts";
8 import type { Ctx, Screen } from "../app/app.ts";
9 import { binding } from "../app/keys.ts";
10 import type { Session } from "../session.ts";
11 import { wrap } from "../term/width.ts";
12 import { dialog } from "../ui/widgets.ts";
13 import { runAsync } from "./common.ts";
14 import type { PaletteEntry } from "./entries.ts";
15
16 const WIDTH = 72;
17 const INNER = WIDTH - 6;
18
19 export function mirrorPacksScreen(f: Session, ctx: Ctx): Screen {
20 let note = "";
21 let busy = false;
22 const go = (): void => {
23 if (busy) return;
24 busy = true;
25 note = "Working…";
26 ctx.invalidate();
27 runAsync(ctx, async () => {
28 try {
29 const done = await mirrorPacks(f.app, f.user);
30 note = done.ok ? reportWords(done.value) : MIRROR_FORBIDDEN;
31 } finally {
32 busy = false;
33 ctx.invalidate();
34 }
35 });
36 };
37 return {
38 title: "Mirror the packs' upstreams",
39 keys: [...(busy ? [] : [binding(["enter"], "mirror")]), binding(["esc", "q"], "close")],
40 render(area, s) {
41 const buttons = dialog(s, area, ctx.theme.c, {
42 title: "Mirror the packs' upstreams",
43 body: [
44 ...wrap("Copy each pack's upstream projects here as read-only packs/<app> repositories that keep up with the original. Press Enter to run it; running it again makes nothing new.", INNER),
45 ...(note === "" ? [] : ["", ...wrap(note, INNER)]),
46 ],
47 actions: [{ label: "Mirror" }, { label: "Close" }],
48 selected: 0,
49 width: WIDTH,
50 });
51 buttons[0] && ctx.hit(buttons[0], go);
52 buttons[1] && ctx.hit(buttons[1], () => ctx.pop());
53 },
54 key(e) {
55 if (e.ctrl || e.alt) return true;
56 if (e.key === "esc" || e.key === "q") ctx.pop();
57 else if (e.key === "enter") go();
58 return true;
59 },
60 };
61 }
62
63 /** Bring the packs' upstreams in as read-only mirrors: a site admin's, as the button on /packs is. */
64 export const entry: PaletteEntry = {
65 id: "mirror-packs",
66 title: "Mirror the packs' upstreams",
67 hint: "bring each pack's upstream projects in as read-only repositories",
68 after: ["packs"],
69 offered: (session) => session.user?.isAdmin === true && session.user.kind !== "agent",
70 run: (session, ctx) => ctx.push(mirrorPacksScreen(session, ctx)),
71 };
src/tui/ssh.ts +7
@@ -20,6 +20,7 @@ import { redirectFor } from "../core/transfer.ts";
2020 import { sshCloneUrl } from "../ssh/command.ts";
2121 import { getUserById, isActive, type User } from "../core/users.ts";
2222 import { createRepo, repositoryLimitMessage, resolveOwner } from "../actions/repos.ts";
23 import { isPacksMirror, MIRROR_PUSH_REFUSED } from "../packs/owner.ts";
2324 import { hookEnv } from "../git/hooks.ts";
2425 import { repoDir } from "../git/repos.ts";
2526 import { pushCap, storageChanged, storageRefusal, storageRefusedText, storageSentence } from "../core/storage.ts";
@@ -334,6 +335,12 @@ export async function serveGitCommand(
334335 }
335336 }
336337
338 if (write && repo !== undefined && isPacksMirror(repo) && canRead(forge.db, repo, user)) {
339 session.writeStderr(`forge: ${MIRROR_PUSH_REFUSED}${CRLF}`);
340 session.exit(1);
341 return;
342 }
343
337344 const allowed =
338345 repo !== undefined &&
339346 (write
src/web/pages/packs.ts +21 −3
@@ -11,8 +11,10 @@ import type { Ctx } from "../../http/context.ts";
1111 import { notFoundPage, pageHead } from "../chrome.ts";
1212 import { deployCommands } from "../../packs/words.ts";
1313 import { html } from "../html.ts";
14 import { page } from "../layout.ts";
15 import { copyField, row } from "../ui.ts";
14 import { flashTo, page } from "../layout.ts";
15 import { readForm } from "../form.ts";
16 import { mirrorPacks, reportWords } from "../../packs/mirror.ts";
17 import { btn, copyField, row } from "../ui.ts";
1618
1719 const plural = (n: number, one: string): string => `${n} ${one}${n === 1 ? "" : "s"}`;
1820
@@ -35,7 +37,13 @@ export function packsPage(ctx: Ctx): Response {
3537 const body = html`${pageHead("Forge packs", {
3638 sub: "A whole forge to start from: real apps with their history, and a crew set up to work on them.",
3739 })}
38 ${packsList(packs)}`;
40 ${packsList(packs)}
41 ${ctx.auth.user?.isAdmin === true && ctx.auth.user.kind !== "agent"
42 ? html`<form class="stack" method="post" action="/packs/mirror">
43 <p class="muted small">Copy each pack's upstream projects here as read-only repositories that keep up with the original.</p>
sentinelcrew outdated

Principle 1: this paragraph explains the button twice, and the link 'packs' points at /packs, the current page. One plain sentence, no link.
44 ${btn("Mirror the upstreams", { size: "sm", variant: "primary", label: "Mirror the packs' upstreams" })}
45 </form>`
46 : null}`;
3947 const response = page({ ctx, title: "Forge packs", body });
4048 notePhase(ctx.request, "render", performance.now() - started);
4149 return response;
@@ -74,3 +82,13 @@ export function packPage(ctx: Ctx): Response {
7482 notePhase(ctx.request, "render", performance.now() - started);
7583 return response;
7684 }
85
86 // POST /packs/mirror (admin)
87 export async function packsMirror(ctx: Ctx): Promise<Response> {
88 if (ctx.auth.user?.isAdmin !== true || ctx.auth.user.kind === "agent") return notFoundPage(ctx);
89 const form = await readForm(ctx, "admin");
90 if (form instanceof Response) return form;
91 const done = await mirrorPacks(ctx.app, ctx.auth.user);
92 if (!done.ok) return notFoundPage(ctx);
93 return flashTo(ctx, "/packs", { tone: done.value.failed.length > 0 ? "danger" : "success", message: reportWords(done.value) });
94 }
test/packs-mirror.test.ts added +279
@@ -0,0 +1,279 @@
1 /**
2 * The packs' upstreams as read-only mirrors (#10530): brought in once by an
3 * admin as `packs/<app>`, re-synced by the scheduled follow, pushed to by no
4 * one. The upstream is a real bare repository served over dumb http.
5 */
6
7 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
8 import { mkdtemp, rm } from "node:fs/promises";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { createApp, type App } from "../src/app.ts";
12 import { checkDue, FOLLOW_EVERY_MS } from "../src/actions/upstream_follow.ts";
13 import { listFollowing } from "../src/core/upstream_follow.ts";
14 import { createSession } from "../src/core/sessions.ts";
15 import { canWrite, createRepo, getRepo, listEveryRepo } from "../src/core/repos.ts";
16 import { getUserByName, type User } from "../src/core/users.ts";
17 import { runGit } from "../src/git/env.ts";
18 import { repoDir } from "../src/git/repos.ts";
19 import { serve, type Servers } from "../src/http/server.ts";
20 import type { SshSession } from "../src/ssh/channels.ts";
21 import { serveGitCommand } from "../src/tui/ssh.ts";
22 import { createForgeApp, forgeCommands } from "../src/tui/forge.ts";
23 import { mirrorPacksScreen } from "../src/tui/screens/mirrorpacks.ts";
24 import { headlessTerminal } from "../src/tui/term/terminal.ts";
25 import { settleApp } from "./helpers/tui.ts";
26 import { tool } from "../src/http/mcp/tools/repo_settings.ts";
27 import { mirrorPacks, packSources, reportWords } from "../src/packs/mirror.ts";
28 import { MIRROR_PUSH_REFUSED } from "../src/packs/owner.ts";
29 import { adminCookie, seedAdmin } from "./helpers/admin.ts";
30 import { mirrorOutbound } from "./helpers/globals.ts";
31
32 const ident = { GIT_AUTHOR_NAME: "t", GIT_AUTHOR_EMAIL: "t@x", GIT_COMMITTER_NAME: "t", GIT_COMMITTER_EMAIL: "t@x" };
33
34 let dataDir = "";
35 let app: App;
36 let servers: Servers;
37 let admin: User;
38 let upstream: ReturnType<typeof Bun.serve>;
39 let bare = "";
40 let work = "";
41
42 const git = async (cwd: string, ...args: string[]): Promise<string> => {
43 const r = await runGit(args, { cwd, env: ident });
44 if (!r.ok) throw new Error(`git ${args.join(" ")}: ${JSON.stringify(r)}`);
45 return r.value.trim();
46 };
47
48 /** A commit on the upstream, published for dumb http. */
49 async function commit(text: string): Promise<string> {
50 await Bun.write(join(work, "f.txt"), text);
51 await git(work, "add", "f.txt");
52 await git(work, "commit", "-m", text);
53 await git(work, "push", "origin", "HEAD:refs/heads/trunk");
54 await git(bare, "update-server-info");
55 return git(work, "rev-parse", "HEAD");
56 }
57
58 mirrorOutbound({ allow: ["127.0.0.1"] });
59
60 beforeAll(async () => {
61 dataDir = await mkdtemp(join(tmpdir(), "forge-packs-mirror-"));
62 bare = join(dataDir, "up.git");
63 work = join(dataDir, "work");
64 await git(dataDir, "init", "--bare", "-b", "trunk", bare);
65 await git(dataDir, "clone", bare, work);
66 await git(work, "checkout", "-b", "trunk");
67 upstream = Bun.serve({
68 port: 0,
69 hostname: "127.0.0.1",
70 async fetch(req) {
71 const rel = new URL(req.url).pathname.replace(/^\/up\.git\//, "");
72 if (rel.includes("..")) return new Response("no", { status: 404 });
73 const file = Bun.file(join(bare, rel));
74 return (await file.exists()) ? new Response(file) : new Response("no", { status: 404 });
75 },
76 });
77 app = createApp({ dataDir: join(dataDir, "data"), port: 0 });
78 servers = await serve(app);
79 const made = await seedAdmin(app, { username: "boss", email: "[email protected]", password: "mirror-password" });
80 if (!made.ok) throw new Error(made.error);
81 admin = made.value;
82 });
83
84 afterAll(async () => {
85 upstream?.stop(true);
86 servers?.stop();
87 app?.close();
88 await rm(dataDir, { recursive: true, force: true });
89 });
90
91 function sshSession(command: string) {
92 const out: string[] = [];
93 const err: string[] = [];
94 let code: number | undefined;
95 const text = (d: string | Uint8Array) => (typeof d === "string" ? d : new TextDecoder().decode(d));
96 const session = {
97 command,
98 username: "git",
99 env: {},
100 remoteAddress: "127.0.0.1",
101 write: (d: string | Uint8Array) => void out.push(text(d)),
102 writeStderr: (d: string | Uint8Array) => void err.push(text(d)),
103 exit: (c: number) => void (code = c),
104 onData: (cb: (d: Uint8Array) => void) => setTimeout(() => cb(new TextEncoder().encode("0000")), 50),
105 onEof: (cb: () => void) => setTimeout(cb, 100),
106 onClose: () => {},
107 drain: async () => {},
108 } as unknown as SshSession;
109 return { session, out, err, code: () => code };
110 }
111
112 describe("packs mirrors", () => {
113 const source = () => [
114 { name: "demo", source: `http://127.0.0.1:${upstream.port}/up.git`, description: "A demo." },
115 ];
116 let first = "";
117
118 test("the packs' own https sources are what it would bring in", () => {
119 const names = packSources().map((s) => s.name);
120 expect(names).toContain("memos");
121 expect(names).toContain("2048");
122 expect(packSources().every((s) => s.source.startsWith("https://"))).toBe(true);
123 });
124
125 test("only an admin brings them in", async () => {
126 const stranger = await seedAdmin(app, { username: "nora", email: "[email protected]", password: "mirror-password" });
127 expect((await mirrorPacks(app, undefined, source())).ok).toBe(false);
128 expect((await mirrorPacks(app, { ...stranger.ok ? stranger.value : admin, isAdmin: false }, source())).ok).toBe(false);
129 expect(getRepo(app.db, "packs", "demo")).toBeUndefined();
130 });
131
132 test("an admin brings one in: public, on the upstream's branch, with its commits", async () => {
133 first = await commit("one\n");
134 const done = await mirrorPacks(app, admin, source());
135 expect(done.ok && done.value).toMatchObject({ made: ["packs/demo"], kept: [], failed: [] });
136 const repo = getRepo(app.db, "packs", "demo")!;
137 expect(repo.visibility).toBe("public");
138 expect(repo.defaultBranch).toBe("trunk");
139 // The request makes the repository and its follow; the first sync is the schedule's.
140 expect((await runGit(["rev-parse", "--verify", "refs/heads/trunk"], { cwd: repoDir(app.reposRoot, "packs", "demo"), env: ident })).ok).toBe(false);
141 expect(await checkDue(app)).toBeGreaterThanOrEqual(1);
142 expect(await git(repoDir(app.reposRoot, "packs", "demo"), "rev-parse", "refs/heads/trunk")).toBe(first);
143 expect(listFollowing(app.db).map((f) => f.repoId)).toEqual([repo.id]);
144 });
145
146 test("running it twice opens nothing new", async () => {
147 const repos = listEveryRepo(app.db).length;
148 const again = await mirrorPacks(app, admin, source());
149 expect(again.ok && again.value).toMatchObject({ made: [], kept: ["packs/demo"], failed: [] });
150 expect(listEveryRepo(app.db).length).toBe(repos);
151 expect(listFollowing(app.db).length).toBe(1);
152 });
153
154 test("a new upstream commit arrives on the next scheduled sync, with no pull", async () => {
155 const second = await commit("two\n");
156 expect(await checkDue(app, Date.now() + FOLLOW_EVERY_MS + 1000)).toBeGreaterThanOrEqual(1);
157 const dir = repoDir(app.reposRoot, "packs", "demo");
158 expect(await git(dir, "rev-parse", "refs/heads/trunk")).toBe(second);
159 expect(await git(dir, "branch", "--list", "upstream/*")).toBe("");
160 });
161
162 test("a push by anyone, the owner included, is refused", async () => {
163 const owner = getUserByName(app.db, "packs")!;
164 const ownerCookie = `forge_session=${createSession(app.db, owner.id).token}`;
165 const adminSession = await adminCookie(app, { username: "ada", email: "[email protected]", password: "mirror-password" });
166 for (const cookie of [ownerCookie, adminSession]) {
167 const adv = await fetch(`${servers.url}/packs/demo.git/info/refs?service=git-receive-pack`, { headers: { cookie } });
168 expect(await adv.text()).not.toContain("refs/heads");
169 const body = await fetch(`${servers.url}/packs/demo.git/git-receive-pack`, {
170 method: "POST",
171 headers: { cookie, "content-type": "application/x-git-receive-pack-request" },
172 body: "0000",
173 });
174 expect(body.status).toBe(403);
175 }
176 // The owner is told why, not just "not found".
177 const told = await fetch(`${servers.url}/packs/demo.git/info/refs?service=git-receive-pack`, { headers: { cookie: ownerCookie } });
178 expect(await told.text()).toContain(MIRROR_PUSH_REFUSED);
179 });
180
181 test("an ssh push is refused with the reason, an ssh clone is served", async () => {
182 const push = sshSession("git-receive-pack '/packs/demo.git'");
183 await serveGitCommand(app, push.session, admin);
184 expect(push.code()).toBe(1);
185 expect(push.err.join("")).toContain(MIRROR_PUSH_REFUSED);
186 const pull = sshSession("git-upload-pack '/packs/demo.git'");
187 await serveGitCommand(app, pull.session, admin);
188 expect(pull.code()).toBe(0);
189 });
190
191 test("no one but the follow writes a mirror: not an admin, not by the contents api", async () => {
192 const repo = getRepo(app.db, "packs", "demo")!;
193 expect(canWrite(app.db, repo, admin)).toBe(false);
194 expect(canWrite(app.db, repo, getUserByName(app.db, "packs"))).toBe(true);
195 const cookie = `forge_session=${createSession(app.db, admin.id).token}`;
196 const before = await git(repoDir(app.reposRoot, "packs", "demo"), "rev-parse", "refs/heads/trunk");
197 const wrote = await fetch(`${servers.url}/api/v1/repos/packs/demo/contents`, {
198 method: "POST",
199 headers: { cookie, origin: servers.url, "content-type": "application/json" },
200 body: JSON.stringify({ branch: "trunk", message: "x", files: [{ path: "a.txt", content: "a" }] }),
201 });
202 expect(wrote.status).toBeGreaterThanOrEqual(400);
203 expect(await git(repoDir(app.reposRoot, "packs", "demo"), "rev-parse", "refs/heads/trunk")).toBe(before);
204 // An ordinary repository's admin still writes.
205 const other = await createRepo(app.db, app.reposRoot, { ownerId: admin.id, name: "plain", visibility: "public" });
206 expect(other.ok && canWrite(app.db, other.value, admin)).toBe(true);
207 });
208
209 test("a slow run stops at its budget and says what it left", async () => {
210 const many = [{ name: "late", source: `http://127.0.0.1:${upstream.port}/up.git`, description: "x" }];
211 const done = await mirrorPacks(app, admin, many, 0);
212 expect(done.ok && done.value.failed).toEqual([{ app: "late", why: "not reached in time; press again" }]);
213 expect(getRepo(app.db, "packs", "late")).toBeUndefined();
214 });
215
216 test("the /packs button is an admin's; its post answers with the report", async () => {
217 const admin2 = await adminCookie(app, { username: "bianca", email: "[email protected]", password: "mirror-password" });
218 const page = await (await fetch(`${servers.url}/packs`, { headers: { cookie: admin2 } })).text();
219 expect(page).toContain('action="/packs/mirror"');
220 expect(await (await fetch(`${servers.url}/packs`)).text()).not.toContain('action="/packs/mirror"');
221 const anon = await fetch(`${servers.url}/packs/mirror`, { method: "POST", redirect: "manual", headers: { origin: servers.url } });
222 expect([403, 404]).toContain(anon.status);
223 // Every shipped source already has its mirror, so the post reaches no network.
224 const owner = getUserByName(app.db, "packs")!;
225 for (const p of packSources()) {
226 if (getRepo(app.db, "packs", p.name) === undefined)
227 await createRepo(app.db, app.reposRoot, { ownerId: owner.id, name: p.name, visibility: "public" });
228 }
229 const posted = await fetch(`${servers.url}/packs/mirror`, {
230 method: "POST",
231 redirect: "manual",
232 headers: { cookie: admin2, origin: servers.url, "content-type": "application/x-www-form-urlencoded" },
233 body: "",
234 });
235 expect(posted.status).toBe(303);
236 expect(posted.headers.get("location")).toBe("/packs");
237 const flash = decodeURIComponent(posted.headers.get("set-cookie") ?? "");
238 expect(flash).toContain("already there");
239 expect(flash).toContain("0 brought in");
240 });
241
242 test("the MCP door answers an admin with the report, and refuses anyone else", async () => {
243 const ctx = { app } as never;
244 const denied = await tool.run(ctx, { ...admin, isAdmin: false }, { mirrorPacks: true });
245 expect(denied.ok).toBe(false);
246 const done = await tool.run(ctx, admin, { mirrorPacks: true });
247 expect(done.ok).toBe(true);
248 const value = (done as unknown as { value: Record<string, unknown> }).value ?? done;
249 const text = JSON.stringify(value);
250 expect(text).toContain('"made":[]');
251 expect(text).toContain(`"kept":[`);
252 expect(text).toContain(reportWords({ made: [], kept: packSources().map((p) => `packs/${p.name}`), failed: [] }).replace(/\./g, "."));
253 });
254
255 describe("the terminal door", () => {
256 const open = (user: User | undefined) =>
257 createForgeApp(app, { sshUsername: "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:t", user }, { terminal: headlessTerminal(120, 40), exitHooks: false });
258 test("an admin is offered it and a non-admin is not", () => {
259 const a = open(admin);
260 const ids = forgeCommands(a.session)(a.app.ctx).map((c) => c.id);
261 a.app.stop?.();
262 const n = open({ ...admin, isAdmin: false });
263 const idsN = forgeCommands(n.session)(n.app.ctx).map((c) => c.id);
264 n.app.stop?.();
265 expect(ids).toContain("mirror-packs");
266 expect(idsN).not.toContain("mirror-packs");
267 });
268 test("Enter runs the action and the screen says what it did", async () => {
269 const { app: tui, session } = open(admin);
270 tui.start();
271 tui.ctx.push(mirrorPacksScreen(session, tui.ctx));
272 await settleApp(tui);
273 tui.dispatch({ type: "key", key: "enter", ctrl: false, alt: false, shift: false });
274 await settleApp(tui);
275 expect(tui.screen.toText()).toContain("already there");
276 tui.stop?.();
277 });
278 });
279 });
test/palette-roster.test.ts +1
@@ -48,6 +48,7 @@ test("a site-admin person is offered the whole roster", () => {
4848 ["home", "Home", "your repositories"],
4949 ["explore", "Explore", "on the web"],
5050 ["packs", "Forge packs", "premade forges to deploy, and how to help build their apps"],
51 ["mirror-packs", "Mirror the packs' upstreams", "bring each pack's upstream projects in as read-only repositories"],
5152 ["data", "Data", "the SQLite files you may read as a tree; / filters, : asks in SQL, y yanks a row"],
5253 ["fleet", "Fleet rows", "the rows a fleet query line selects"],
5354 ["profile", "Open a profile", "username or org"],
test/signup-gate-parent.test.ts +1
@@ -115,6 +115,7 @@ test("every caller of createUser says where its account comes from", async () =>
115115 "src/actions/spawn.ts": ["spawn", "spawn"],
116116 "src/actions/users.ts": ["signup", "passkey"],
117117 "src/chaos/world.ts": ["seed"],
118 "src/packs/mirror.ts": ["seed"],
118119 "src/packs/up.ts": ["seed"],
119120 "src/spawn/preview-seed.ts": ["seed", "seed"],
120121 "src/web/pages/claim.ts": ["claim"],

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs