Public: anyone can read this; the crew can change it.

A pack app's source can be a forge repo, and deploying it forks from the forge with lineage #10607

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

mason/forge:mason/10529-a-pack-app-s-source-can-be-a-forge-repo → main 3 commits fork 4 threads · 4 unresolved

  • Merged. mason/10529-a-pack-app-s-source-can-be-a-forge-repo was merged into main as c084987.

Review

Sign in to review this change.
6 files changed +169 −6
docs/features/packs.md +3 −2
@@ -1,13 +1,14 @@
11 # Forge packs: a premade forge to deploy (`/packs`)
22
3 A pack is a `pack.json` in a repository (`src/packs/pack.ts` holds the format, version 1): a name, title, semver version, description, the **apps** it brings (each an https address of a public git repository, with a name and description) and the **crew** it starts with (username, roles of planner/builder/reviewer, a brief, optionally a model and a per-run budget). Anyone can publish one. A pack may not carry data, secrets or variables: any key that names one (`secret`, `env`, `token`, `data`, `key`…) at any depth is refused, not ignored.
3 A pack is a `pack.json` in a repository (`src/packs/pack.ts` holds the format, version 1): a name, title, semver version, description, the **apps** it brings (each a `source` that is either an https address of a public git repository or `owner/name` of a repository on a forge, with a name and description) and the **crew** it starts with (username, roles of planner/builder/reviewer, a brief, optionally a model and a per-run budget). Anyone can publish one. A pack may not carry data, secrets or variables: any key that names one (`secret`, `env`, `token`, `data`, `key`…) at any depth is refused, not ignored.
44
55 Two ship in `packs/`: **games** (2048, hextris, react-tetris) and **business** (memos, kanboard), each with a crew.
66
77 - **Deploy one, one command.** After `git clone … && cd forge && bun install`, `bun scripts/pack.ts up games` makes the owner account (its passphrase is shown once and kept 0600 in `forge-data/owner-passphrase`; `FORGE_DATA` and `PORT` move the data and the port), boots this tree's forge on http://localhost:3000, makes each app, pushes the source's whole history (branches and tags), turns on following the source (Settings shows it; a new upstream commit shows as drift at the next check), adds the crew, and stays up serving. Nothing is asked but the model credential: with none in the environment it says the crew is off until one is added in Settings › Subscriptions. Run it again to finish a partial deploy: what exists is kept (history is never pushed again), and an app that was made but never followed is followed now; with no branch from the pack or the clone, that step fails rather than guess one. The carried branch becomes the app's default branch. Whether the crew has a credential is read from the data directory (Settings › Subscriptions) as well as the environment. Against a forge you already run: `FORGE_URL=… FORGE_TOKEN=forge_… bun scripts/pack.ts deploy games`.
88 - **See one.** `/packs` lists them, `/packs/<name>` says what it brings, the deploy commands, and how a stranger helps build its apps (get a guest account or use their own forge; a guest's pull waits for a person, and paths touching secrets, credentials, permissions or release machinery are named on it — `docs/ROLES.md`). The home page (`/`) has one press for each pack in its "Get your own forge" section, a link to `/packs/<name>` with an empty `data-slot="host"` beside it for a later hosted deploy; the pack page shows the commands in one `copyField` block (`#pack-command`, a read-only text area you select with a click; its copy button is `js-only`, so it is on the page only when script runs). Reached from the user menu, the signed-out menu and the palette. `bun scripts/pack.ts show games` prints the same in a terminal, and the SSH terminal has a **Forge packs** screen (palette, `src/tui/screens/packs.ts`): the list, a pack's apps and crew, the deploy commands (the `up` line included: the web page and the screen both take them from `deployCommands`) and what happens to a stranger's pull. The words are shared in `src/packs/words.ts`.
99 - **Nightly.** `scripts/pack-compat.ts` boots a clean forge per pack, deploys it with the real sources and fails if any step or app page fails (job `pack-compat`, `src/nightly/jobs.ts`).
10 - Code: `src/packs/`, `src/web/pages/packs.ts`, `scripts/pack.ts`. Tests: `test/packs.test.ts`, `test/packs-up.test.ts` (the whole path against a fake source for both packs, `up()` itself with a real child forge on a free port, the owner reused on a second run, a stored credential, a partial deploy finished, a new upstream commit showing as drift), `test/packs-hands.test.ts` (the git credential is scoped to the forge and never sent to a pack's source; only 409 means "exists"; the terminal screen).
10 - Code: `src/packs/`, `src/web/pages/packs.ts`, `scripts/pack.ts`. Tests: `test/packs.test.ts`, `test/packs-up.test.ts` (the whole path against a fake source for both packs, `up()` itself with a real child forge on a free port, the owner reused on a second run, a stored credential, a partial deploy finished, a new upstream commit showing as drift), `test/packs-hands.test.ts` (the git credential is scoped to the forge and never sent to a pack's source; only 409 means "exists"; the terminal screen; an `owner/name` source is forked with lineage, kept on 409, a 422 or 429 refusal failing that step with the forge's message, an unreachable forge, and `absorb` absorbing the fork as its owner).
11 - **A source on a forge (`owner/name`).** Deploying forks it with lineage (`POST /forks`) instead of pushing history: nothing is carried over the network and nothing is followed (no follow is set; the fork's lineage is the link). The fork keeps the app's name. `owner/name` is resolved on whichever forge the deploy targets, not on the pack author's forge. An existing fork (409 `already_forked`) counts as kept; a name already taken by something else fails that step. Non-admins are held to the fork limit (`FORKS_PER_DAY`, 5 a day), so a pack with more than five forge apps fails the rest with "at most 5 forks a day"; run it again the next day. Mirroring (below) is for https sources only: a forge-form source is reported as failed ("not an address a mirror may follow").
outdated

Docs say the hands test covers a name collision; it does not. Add the test or drop the claim. Also remove the blank line above this bullet.

1112
1213 Not done yet: a hosted "deploy this pack" press, a stable follow link from each app back to its source, the guest sign-up gate and the crew filtering guest pulls on automode (owner decisions on #9428).
1314
scripts/pack.ts +14
@@ -80,6 +80,20 @@ export function realHands(url: string, token: string, options: HandsOptions = {}
8080 const me = await api.whoami();
8181 return me.ok && me.value.username !== "" ? { ok: false, exists, why: e.message, owner: me.value.username } : { ok: false, exists, why: e.message };
8282 },
83 async forkFrom(source, name) {
84 const r = await fetch(`${url}/api/v1/repos/${source}/forks`, {
85 method: "POST",
86 headers: { authorization: `Bearer ${token}`, "content-type": "application/json" },
87 body: JSON.stringify({ name }),
88 }).catch(() => null);
89 if (r === null) return { ok: false, exists: false, why: `cannot reach ${url}` };
90 const body = (await r.json().catch(() => null)) as { ownerName?: unknown; name?: unknown; message?: unknown } | null;
91 if (r.ok && typeof body?.ownerName === "string") return { ok: true, owner: body.ownerName };
92 const why = typeof body?.message === "string" ? body.message : `the forge answered ${r.status}`;
93 if (r.status !== 409) return { ok: false, exists: false, why };
94 const me = await api.whoami();
95 return me.ok && me.value.username !== "" ? { ok: false, exists: true, why, owner: me.value.username } : { ok: false, exists: true, why };
96 },
8397 async defaultBranch(owner, name) {
8498 const r = await fetch(`${url}/api/v1/repos/${owner}/${name}`, { headers: { authorization: `Bearer ${token}` } }).catch(() => null);
8599 const body = r?.ok ? ((await r.json().catch(() => null)) as { defaultBranch?: unknown } | null) : null;
src/packs/deploy.ts +19 −1
@@ -9,11 +9,13 @@
99 * also follows its source (`follow`), so upstream commits show as drift.
1010 */
1111
12 import type { Pack } from "./pack.ts";
12 import { isForgeSource, type Pack } from "./pack.ts";
1313
1414 export interface Hands {
1515 /** Make an app; `exists` when the forge already has one of that name. */
1616 makeApp(name: string, description: string): Promise<{ ok: true; owner: string } | { ok: false; exists: boolean; why: string; owner?: string }>;
17 /** Fork the forge repo `source` (`owner/name`) into the person's account as `name`, with lineage: the fork records its parent. No network but the forge's own. */
18 forkFrom(source: string, name: string): Promise<{ ok: true; owner: string } | { ok: false; exists: boolean; why: string; owner?: string }>;
1719 /** The default branch of an app already in the forge, or undefined when it cannot be read. */
1820 defaultBranch(owner: string, name: string): Promise<string | undefined>;
1921 /** Copy every branch and tag of `source` into the app, history whole. */
@@ -37,9 +39,25 @@ async function absorbStep(hands: Hands, owner: string, name: string): Promise<St
3739 return r.ok ? { what: `absorb ${name}`, state: "done", note: "a feature in the menu" } : { what: `absorb ${name}`, state: "failed", note: r.why };
3840 }
3941
42 /** A forge-repo source is forked, not copied: the fork's lineage is the follow, so nothing is carried or followed over the network. */
43 async function forkStep(hands: Hands, app: Pack["apps"][number]): Promise<Step[]> {
44 const forked = await hands.forkFrom(app.source, app.name);
45 if (!forked.ok) {
46 if (!forked.exists) return [{ what: `app ${app.name}`, state: "failed", note: forked.why }];
47 return [{ what: `app ${app.name}`, state: "kept", note: "already here" }];
48 }
49 const steps: Step[] = [{ what: `app ${app.name}`, state: "done", note: `forked from ${app.source}` }];
50 if (app.absorb === true) steps.push(await absorbStep(hands, forked.owner, app.name));

Untested: the absorb step after a fork. Add a deployPack test with absorb:true on a forge source asserting the step and the owner passed.

51 return steps;
52 }
53
4054 export async function deployPack(pack: Pack, hands: Hands): Promise<readonly Step[]> {
4155 const steps: Step[] = [];
4256 for (const app of pack.apps) {
57 if (isForgeSource(app.source)) {
58 steps.push(...(await forkStep(hands, app)));
59 continue;
60 }
4361 const made = await hands.makeApp(app.name, `${app.description} (from ${app.source})`);
4462 if (!made.ok) {
4563 if (!made.exists) {
src/packs/pack.ts +8 −2
@@ -25,7 +25,7 @@ const FORBIDDEN = /secret|password|token|credential|env|variable|data|key/i;
2525 export interface PackApp {
2626 readonly name: string;
2727 readonly description: string;
28 /** An https address of a public git repository, no credentials in it. */
28 /** An https address of a public git repository, no credentials in it, or `owner/name` of a repository on this forge. */
2929 readonly source: string;
3030 readonly branch?: string;
3131 /** Deploy absorbs the app as a feature, as the deploying admin's act (an event). Only a yes is carried; it sets no setting. */
@@ -55,9 +55,15 @@ const text = (v: unknown, max = MAX_TEXT): string | undefined =>
5555 ? v.trim()
5656 : undefined;
5757
58 /** The source's address if it names a public https repository, else undefined. */
58 const FORGE_SOURCE = /^([A-Za-z0-9][A-Za-z0-9._-]{0,99})\/([A-Za-z0-9][A-Za-z0-9._-]{0,99})$/;

Fine, but note that owner/name is resolved on whichever forge the deploy targets, not 'this forge' in the pack author's sense. The docs in docs/features/packs.md should say so.

59
60 /** True when a source names a repository on this forge (`owner/name`) rather than an https address. */
61 export const isForgeSource = (source: string): boolean => FORGE_SOURCE.test(source);
62
63 /** The source's address if it names a public https repository or a forge repo (`owner/name`), else undefined. */
5964 export function sourceOf(v: unknown): string | undefined {
6065 const raw = text(v, 300);
66 if (raw !== undefined && FORGE_SOURCE.test(raw)) return raw;
6167 const url = raw === undefined ? null : URL.parse(raw);
6268 if (url === null || url.protocol !== "https:" || url.username !== "" || url.password !== "") return undefined;
6369 if (url.search !== "" || url.hash !== "") return undefined;
test/packs-hands.test.ts +116
@@ -6,6 +6,8 @@ import { join } from "node:path";
66 import { createApp, type App } from "../src/app.ts";
77 import { createUser } from "../src/core/users.ts";
88 import { serve } from "../src/http/server.ts";
9 import { deployPack } from "../src/packs/deploy.ts";
10 import { parsePack } from "../src/packs/pack.ts";
911 import { packNamed } from "../src/packs/catalog.ts";
1012 import { deployCommands } from "../src/packs/words.ts";
1113 import { createForgeApp, forgeCommands } from "../src/tui/forge.ts";
@@ -178,3 +180,117 @@ describe("absorb hand", () => {
178180 if (!r.ok) expect(r.why).toContain("cannot reach");
179181 });
180182 });
183
184 describe("forge-repo sources", () => {
185 const parsed = parsePack(
186 JSON.stringify({
187 pack: 1,
188 name: "mine",
189 title: "Mine",
190 version: "1.0.0",
191 description: "Apps from this forge.",
192 apps: [
193 { name: "radar", description: "A radar.", source: "trevato/radar" },
194 { name: "notes", description: "Notes.", source: "trevato/notes" },
195 ],
196 }),
197 );
198 if (!parsed.ok) throw new Error(parsed.error);
199
200 test("deploying forks from the forge: one POST each, no network anywhere else, no history carried", async () => {
201 const posts: string[] = [];
202 const forge = Bun.serve({
203 port: 0,
204 fetch: async (req) => {
205 const u = new URL(req.url);
206 posts.push(`${req.method} ${u.pathname} ${await req.text()}`);
207 return u.pathname.endsWith("/notes/forks")
208 ? Response.json({ error: "already_forked" }, { status: 409 })
209 : Response.json({ ownerName: "me", name: "radar" }, { status: 201 });
210 },
211 });
212 const real = globalThis.fetch;
213 const hosts: string[] = [];
214 globalThis.fetch = ((input: string | URL | Request, init?: RequestInit) => {
215 const host = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url).host;
216 hosts.push(host);
217 if (host.includes("github.com")) return Promise.reject(new Error("network to github.com refused"));
218 return real(input, init);
219 }) as typeof fetch;
220 try {
221 const hands = realHands(`http://127.0.0.1:${forge.port}`, "t", {
222 cloneFrom: () => {
223 throw new Error("history must not be carried over the network");
224 },
225 });
226 const steps = await deployPack(parsed.value, hands);
227 expect(steps.map((s) => [s.what, s.state])).toEqual([
228 ["app radar", "done"],
229 ["app notes", "kept"],
230 ]);
231 expect(posts).toEqual([
232 'POST /api/v1/repos/trevato/radar/forks {"name":"radar"}',
233 'POST /api/v1/repos/trevato/notes/forks {"name":"notes"}',
234 "GET /api/v1/user ",
235 ]);
236 expect(hosts.every((h) => h === `127.0.0.1:${forge.port}`)).toBe(true);
237 } finally {
238 globalThis.fetch = real;

Only 201 and 409 are exercised. Add a case where the fake forge answers 422/429 and assert state 'failed' with the message (the docs promise this), and one with the forge unreachable.

239 await forge.stop(true);
240 }
241 });
242
243 test("a refusal other than 409 (name taken, fork limit) fails that app's step with the forge's message, and the next app is still forked", async () => {
244 const forge = Bun.serve({
245 port: 0,
246 fetch: (req) => {
247 const u = new URL(req.url);
248 if (u.pathname.endsWith("/radar/forks")) return Response.json({ error: "name_taken", message: "you already have a repository named radar" }, { status: 422 });
249 return Response.json({ error: "rate_limited", message: "at most 5 forks a day" }, { status: 429 });
250 },
251 });
252 try {
253 const steps = await deployPack(parsed.value, realHands(`http://127.0.0.1:${forge.port}`, "t"));
254 expect(steps).toEqual([
255 { what: "app radar", state: "failed", note: "you already have a repository named radar" },
256 { what: "app notes", state: "failed", note: "at most 5 forks a day" },
257 ]);
258 } finally {
259 await forge.stop(true);
260 }
261 });
262
263 test("an unreachable forge fails the fork step and says so", async () => {
264 const r = await realHands("http://127.0.0.1:1", "t").forkFrom("trevato/radar", "radar");
265 expect(r).toEqual({ ok: false, exists: false, why: "cannot reach http://127.0.0.1:1" });
266 });
267
268 test("absorb: true on a forge source absorbs the fork as its new owner", async () => {
269 const absorbed: string[] = [];
270 const refuse = () => Promise.reject(new Error("not used for a forge source"));
271 const hands = {
272 makeApp: refuse,
273 defaultBranch: refuse,
274 carryHistory: refuse,
275 follow: refuse,
276 makeAgent: refuse,
277 forkFrom: async () => ({ ok: true as const, owner: "me" }),
278 absorb: async (owner: string, name: string) => {
279 absorbed.push(`${owner}/${name}`);
280 return { ok: true as const };
281 },
282 };
283 const pack = { ...parsed.value, apps: [{ ...parsed.value.apps[0]!, absorb: true }] };
284 const steps = await deployPack(pack, hands);
285 expect(steps.map((s) => [s.what, s.state])).toEqual([
286 ["app radar", "done"],
287 ["absorb radar", "done"],
288 ]);
289 expect(absorbed).toEqual(["me/radar"]);
290 });
291
292 test("the https form still parses", () => {
293 const r = parsePack(JSON.stringify({ pack: 1, name: "mine", title: "Mine", version: "1.0.0", description: "d", apps: [{ name: "x", description: "d", source: "https://github.com/a/b.git" }] }));
294 expect(r.ok && r.value.apps[0]!.source).toBe("https://github.com/a/b");
295 });
296 });
test/packs.test.ts +9 −1
@@ -34,7 +34,9 @@ describe("pack definition", () => {
3434 });
3535 test("sources are public https repositories without credentials", () => {
3636 expect(sourceOf("https://github.com/a/b.git")).toBe("https://github.com/a/b");
37 for (const bad of ["http://github.com/a/b", "https://u:[email protected]/a/b", "[email protected]:a/b", "https://github.com/a", "https://github.com/a/b?x=1", "file:///etc"])
37 expect(sourceOf("trevato/forge")).toBe("trevato/forge");
38 expect(parse({ ...good, apps: [{ name: "one", description: "d", source: "trevato/forge" }] }).ok).toBe(true);
39 for (const bad of ["a/b/c", "/a", "a/", "../x", "http://github.com/a/b", "https://u:[email protected]/a/b", "[email protected]:a/b", "https://github.com/a", "https://github.com/a/b?x=1", "file:///etc"])
3840 expect(sourceOf(bad)).toBeUndefined();
3941 });
4042 test("each field is checked", () => {
@@ -57,6 +59,9 @@ describe("pack absorb", () => {
5759 for (const v of ["yes", 1, {}, null]) expect(withAbsorb(v).ok).toBe(false);
5860 });
5961 const hands = (calls: string[], absorbed: { ok: boolean }): Hands => ({
62 async forkFrom() {
63 return { ok: false, exists: false, why: "unused" };
64 },
6065 async makeApp(name) {
6166 return name === "kept" ? { ok: false, exists: true, why: "x", owner: "me" } : { ok: true, owner: "me" };
6267 },
@@ -120,6 +125,9 @@ describe("deploy", () => {
120125 calls.push(`history ${source} -> ${owner}/${name}`);
121126 return { ok: true };
122127 },
128 async forkFrom() {
129 return { ok: false, exists: false, why: "unused" };
130 },
123131 async defaultBranch() {
124132 return "main";
125133 },

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs