Public: anyone can read this; the crew can change it.
Platform inventory: every app the owner has on the platform, what it holds, and its migration state #13200
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/646-platform-inventory-every-app-the-owner-h → main 8 commits fork 3 threads · 3 unresolved
21 files
changed +1114
−25
docs/MCP.md +1 −1
| @@ -169,7 +169,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`; | ||
| 169 | 169 | | `org_teams` | `create`, `add_member`, `members`, `metrics`, `remove_member`, `remove` | An org's teams: create, add member with a role, list members, remove | 682 | |
| 170 | 170 | | `orgs` | — | Your orgs; with org, its teams | 170 | |
| 171 | 171 | | `origin_apps` | — | Your origin OAuth apps: list, add, remove | 290 | |
| 172 | | `origins` | `list`, `repos`, `sync`, `hosts` | Origins: list, repos, sync, hosts | 253 | | |
| 172 | | `origins` | `list`, `repos`, `inventory`, `sync`, `hosts` | Origins: list, repos, inventory, sync, hosts | 276 | | |
| 173 | 173 | | `outsiders` | — | Whether non-writers may direct an app's crew (off by default) | 261 | |
| 174 | 174 | | `parent_unlink` | — | Unlink parent-forge sign-in | 117 | |
| 175 | 175 | | `person` | — | A person's profile and repos you may see | 291 | |
docs/design/audit.md +1
| @@ -91,6 +91,7 @@ Machine routes are not pages and have no entry: `/audit.csv`, | ||
| 91 | 91 | - `/apps/open` — `pages/apps.ts` (`appsOpen`) · a door from a name to a running app · `/apps` · colour 1. |
| 92 | 92 | - `/organizations/new` — `pages/org.ts` · make an organization · none · none seen. |
| 93 | 93 | - `/migrate` — `pages/migrate.ts` · bring a repository in from elsewhere · `/settings/origins` (mirror jobs) · none seen. |
| 94 | - `/migrate/inventory` — `pages/inventory.ts` · what is left to bring over from connected origins · `/migrate` · none seen. | |
| 94 | 95 | |
| 95 | 96 | ### Forges (the family) |
| 96 | 97 |
docs/features/bring.md +25
| @@ -13,3 +13,28 @@ the `+` menu and the palette. | ||
| 13 | 13 | link to connect it; there is no anonymous fetch yet. |
| 14 | 14 | - Code: `src/origins/bring.ts` (link, host, needs), `src/actions/bring.ts`, |
| 15 | 15 | `src/web/pages/bring.ts`. Test: `test/bring.test.ts`. |
| 16 | ||
| 17 | ## What is left to bring over (`/migrate/inventory`) | |
| 18 | ||
| 19 | Connect the account your apps live on (`/migrate`), then open | |
| 20 | `/migrate/inventory`: every repository of every connected origin, apps first. | |
| 21 | ||
| 22 | Each row says what the repository holds, read from the repository itself: an | |
| 23 | app (`forge.json` or the platform's deploy file), a Postgres database, domains, | |
| 24 | and the **names** of its secrets, never a value (`.env` is never fetched). The | |
| 25 | badge is this forge's own record of the move: not started, migrating, in the | |
| 26 | forge, at parity, cut over. | |
| 27 | ||
| 28 | A row not yet begun offers **Bring into this forge** (the migrate form with the | |
| 29 | repository chosen) and **Into a new forge** (the new-forge form named for it). A | |
| 30 | row already here offers **Open here**. The header's one primary button brings | |
| 31 | the next app over. | |
| 32 | ||
| 33 | An origin that could not be reached is named above the list, so an empty list is | |
| 34 | never mistaken for an empty account. At most 40 repositories are read per visit, | |
| 35 | most recently pushed first; the rest say "Not read yet" and are read on a later | |
| 36 | visit. What was read is kept until the repository is pushed. | |
| 37 | ||
| 38 | - Terminal: the palette's "What is left to bring over"; `b` brings the row in, | |
| 39 | `n` starts a new forge, `g` reads again. | |
| 40 | - MCP: the `origins` tool, `action: inventory`, `id` an origin: the same rows. |
src/actions/inventory.ts added +72
| @@ -0,0 +1,72 @@ | ||
| 1 | /** | |
| 2 | * The inventory of one connected origin (#646): its repositories, what each | |
| 3 | * holds and where its move stands, with the two ways to act on a row. The | |
| 4 | * page and the MCP's `origins` tool both answer from here. | |
| 5 | */ | |
| 6 | ||
| 7 | import type { App } from "../app.ts"; | |
| 8 | import { getOriginFor } from "../core/origins.ts"; | |
| 9 | import type { User } from "../core/users.ts"; | |
| 10 | import { err, ok, type Result } from "../kernel/result.ts"; | |
| 11 | import { providerForOrigin } from "../origins/credential.ts"; | |
| 12 | import { inventoryRows, type InventoryRow } from "../origins/inventory.ts"; | |
| 13 | import { slugFor } from "../core/apps.ts"; | |
| 14 | import { listRemoteRepos } from "./origins.ts"; | |
| 15 | import { requireUser } from "./access.ts"; | |
| 16 | ||
| 17 | export interface InventoryItem extends InventoryRow { | |
| 18 | /** Bring it into this forge: the migrate page with the repository chosen. */ | |
| 19 | readonly bring: string; | |
| 20 | /** Start a new forge named for it. */ | |
| 21 | readonly newForge: string; | |
| 22 | } | |
| 23 | ||
| 24 | const hostOf = (webUrl: string): string => URL.parse(webUrl)?.host.toLowerCase() ?? ""; | |
| 25 | ||
| 26 | export async function inventoryFor( | |
| 27 | app: App, | |
| 28 | actor: User | undefined, | |
| 29 | originId: string, | |
| 30 | ): Promise<Result<InventoryItem[], "unauthorized" | "not_found" | "unreachable">> { | |
| 31 | const user = requireUser(actor); | |
| 32 | if (!user.ok) return user; | |
| 33 | const origin = getOriginFor(app.db, user.value.id, originId); | |
| 34 | if (origin === undefined) return err("not_found"); | |
| 35 | const listed = await listRemoteRepos(app, user.value, origin.id); | |
| 36 | if (!listed.ok) return listed; | |
| 37 | const provider = providerForOrigin(app, origin); | |
| 38 | if (!provider.ok) return err("unreachable"); | |
| 39 | const rows = await inventoryRows(app.db, user.value, provider.value, listed.value); | |
| 40 | const host = hostOf(origin.webUrl); | |
| 41 | return ok( | |
| 42 | rows.map((row) => ({ | |
| 43 | ...row, | |
| 44 | bring: `/migrate?from=${encodeURIComponent(`${host}/${row.fullName}`).replaceAll("%2F", "/").replaceAll("%3A", ":")}`, | |
| 45 | newForge: `/new?name=${encodeURIComponent(slugFor(row.fullName.split("/").pop() ?? ""))}`, | |
| 46 | })), | |
| 47 | ); | |
| 48 | } | |
| 49 | ||
| 50 | /** What a row holds, in one line. */ | |
| 51 | export function holds(item: InventoryItem): string { | |
| 52 | if (!item.scanned) return "Not read yet"; | |
| 53 | return [ | |
| 54 | item.app === null ? "No app" : item.app === "forge" ? "App (forge.json)" : "App (platform deploy file)", | |
| 55 | ...(item.postgres ? ["Postgres"] : []), | |
| 56 | `${item.domains.length} domain${item.domains.length === 1 ? "" : "s"}`, | |
| 57 | `${item.secrets.length} secret${item.secrets.length === 1 ? "" : "s"}`, | |
| 58 | ].join(" · "); | |
| 59 | } | |
| 60 | ||
| 61 | /** Several origins read at once: their rows, and the logins of those that did not answer. */ | |
| 62 | export async function inventoryOfOrigins( | |
| 63 | app: App, | |
| 64 | user: User, | |
| 65 | origins: readonly { id: string; login: string }[], | |
| 66 | ): Promise<{ items: InventoryItem[]; unreachable: string[] }> { | |
| 67 | const read = await Promise.all(origins.map((o) => inventoryFor(app, user, o.id))); | |
| 68 | return { | |
| 69 | items: read.flatMap((r) => (r.ok ? r.value : [])), | |
| 70 | unreachable: origins.filter((_, i) => !read[i]!.ok).map((o) => o.login), | |
| 71 | }; | |
| 72 | } |
src/checks/looks.ts +1 −1
| @@ -31,7 +31,7 @@ const SAFE = /^[0-9A-Za-z]{1,64}$/; | ||
| 31 | 31 | |
| 32 | 32 | const MAX_BYTES = 1024 * 1024; |
| 33 | 33 | // The forge's own preview takes ~55 looks (`PREVIEW_OWNER_LOOKS`); every one must fit. |
| 34 | const MAX_LOOKS = 72; | |
| 34 | const MAX_LOOKS = 76; | |
| 35 | 35 | const MAX_WORDS = 80; |
| 36 | 36 | |
| 37 | 37 | const words = (v: unknown): string | null => |
src/core/provenance.ts +15 −1
| @@ -141,7 +141,9 @@ function parseParents(text: string): string[] { | ||
| 141 | 141 | } |
| 142 | 142 | } |
| 143 | 143 | |
| 144 | /** The record, or `undefined` for a repository that was not migrated (or whose record is unusable). */ | |
| 144 | /** | |
| 145 | * The record, or `undefined` for a repository that was not migrated (or whose record is unusable). | |
| 146 | */ | |
| 145 | 147 | export function getProvenance( |
| 146 | 148 | db: Database, |
| 147 | 149 | repoId: string, |
| @@ -210,3 +212,15 @@ export function setProvenanceShown( | ||
| 210 | 212 | .run(repoId, shown ? 1 : 0).changes > 0 |
| 211 | 213 | ); |
| 212 | 214 | } |
| 215 | ||
| 216 | /** The repositories migrated from the page at `url`, newest first: what an inventory row asks. */ | |
| 217 | export function repoIdsMigratedFrom(db: Database, url: string): string[] { | |
| 218 | const kept = provenanceUrl(url); | |
| 219 | if (kept === undefined) return []; | |
| 220 | return db | |
| 221 | .query<{ repo_id: string }, [string]>( | |
| 222 | "SELECT repo_id FROM repo_provenance WHERE lower(source_url) = lower(?1) ORDER BY migrated_at DESC", | |
| 223 | ) | |
| 224 | .all(kept) | |
| 225 | .map((r) => r.repo_id); | |
| 226 | } |
src/http/mcp/tools/origins.ts +14 −5
| @@ -1,4 +1,5 @@ | ||
| 1 | 1 | import { listRemoteRepos, listUserOrigins, syncOriginNow } from "../../../actions/origins.ts"; |
| 2 | import { inventoryFor } from "../../../actions/inventory.ts"; | |
| 2 | 3 | import { originJson, remoteRepoJson } from "../../api.ts"; |
| 3 | 4 | import { ORIGIN_UNREACHABLE } from "../../../actions/origin_app_words.ts"; |
| 4 | 5 | import { scopeAllows } from "../../../core/tokens.ts"; |
| @@ -7,13 +8,13 @@ import { good, bad, type Tool } from "../shared.ts"; | ||
| 7 | 8 | |
| 8 | 9 | export const tool: Tool = { |
| 9 | 10 | name: "origins", |
| 10 | doc: "`{ action, id?, hosts? }`: `list` your connected origins, `repos` the repositories one holds (`id`; `mirroredAs` says which are already here), `sync` one now, or `hosts`: with no `hosts`, read the internal hosts an origin may live on (a forge admin, `fromEnv` read-only), with `hosts` (one per line) replace them as Settings → Connected accounts does, a rejected line named in the error; the same as `GET /api/v1/user/origins`, `…/:id/repos` and `POST …/:id/sync`, for you only, an admin token as there; never a credential in an answer", | |
| 11 | doc: "`{ action, id?, hosts? }`: `list` your connected origins, `repos` the repositories one holds (`id`; `mirroredAs` says which are already here), `inventory` the same with what each holds (app, Postgres, domains, secret names, never values) and its migration state, `sync` one now, or `hosts`: with no `hosts`, read the internal hosts an origin may live on (a forge admin, `fromEnv` read-only), with `hosts` (one per line) replace them as Settings → Connected accounts does, a rejected line named in the error; the same as `GET /api/v1/user/origins`, `…/:id/repos` and `POST …/:id/sync`, for you only, an admin token as there; never a credential in an answer", | |
| 11 | 12 | description: |
| 12 | "Origins: list, repos, sync, hosts.", | |
| 13 | "Origins: list, repos, inventory, sync, hosts.", | |
| 13 | 14 | inputSchema: { |
| 14 | 15 | type: "object", |
| 15 | 16 | properties: { |
| 16 | action: { type: "string", enum: ["list", "repos", "sync", "hosts"] }, | |
| 17 | action: { type: "string", enum: ["list", "repos", "inventory", "sync", "hosts"] }, | |
| 17 | 18 | id: { type: "string" }, |
| 18 | 19 | hosts: { type: "string" }, |
| 19 | 20 | }, |
| @@ -25,8 +26,8 @@ export const tool: Tool = { | ||
| 25 | 26 | const action = args["action"]; |
| 26 | 27 | const id = args["id"]; |
| 27 | 28 | if (action === "hosts") return trustedHosts(ctx, user, args); |
| 28 | if (action !== "list" && action !== "repos" && action !== "sync") | |
| 29 | return bad("bad_action", "`action` is list, repos, sync or hosts"); | |
| 29 | if (action !== "list" && action !== "repos" && action !== "inventory" && action !== "sync") | |
| 30 | return bad("bad_action", "`action` is list, repos, inventory, sync or hosts"); | |
| 30 | 31 | if (action !== "list" && (typeof id !== "string" || id === "")) |
| 31 | 32 | return bad("invalid", "send the origin's `id`, from list"); |
| 32 | 33 | if (!scopeAllows(ctx.auth.scope ?? "ro", "admin")) |
| @@ -38,6 +39,14 @@ export const tool: Tool = { | ||
| 38 | 39 | const synced = await syncOriginNow(ctx.app, user, origin); |
| 39 | 40 | return synced.ok ? good(synced.value) : bad("not_found", "no such origin"); |
| 40 | 41 | } |
| 42 | if (action === "inventory") { | |
| 43 | const rows = await inventoryFor(ctx.app, user, origin); | |
| 44 | return rows.ok | |
| 45 | ? good(rows.value) | |
| 46 | : rows.error === "unreachable" | |
| 47 | ? bad("origin_unreachable", ORIGIN_UNREACHABLE) | |
| 48 | : bad("not_found", "no such origin"); | |
| 49 | } | |
| 41 | 50 | const listed = await listRemoteRepos(ctx.app, user, origin); |
| 42 | 51 | if (!listed.ok) |
| 43 | 52 | return listed.error === "unreachable" |
src/http/server.ts +11 −5
| @@ -316,6 +316,7 @@ import { funnelSubmit } from "./funnel.ts"; | ||
| 316 | 316 | import { pricingDone, pricingOrder, pricingPage, resendPage, resendSubmit } from "../web/pages/pricing.ts"; |
| 317 | 317 | import { startForgeSubmit } from "../web/pages/start-forge.ts"; |
| 318 | 318 | import { redirect } from "../web/layout.ts"; |
| 319 | import { inventoryPage } from "../web/pages/inventory.ts"; | |
| 319 | 320 | import { newForgePage, newForgeSubmit } from "../web/pages/new-forge.ts"; |
| 320 | 321 | import { askedPage } from "../web/studio/asked_page.ts"; |
| 321 | 322 | import { notesOverrule, notesSave, notesView } from "../web/pages/notes.ts"; |
| @@ -912,7 +913,8 @@ export function buildRouter(): Router<Handler> { | ||
| 912 | 913 | router.add("GET", "/", home); |
| 913 | 914 | // The home is `/`; its old address goes there (web/pages/home_flow.ts). Literal, above /:owner. |
| 914 | 915 | router.add("GET", "/home", homeRedirect); |
| 915 | // The new home's action bar: preview or apply one bulk action over the rows picked (web/pages/home_table.ts). | |
| 916 | // The new home's action bar: preview or apply one bulk action over the rows picked | |
| 917 | // (web/pages/home_table.ts). | |
| 916 | 918 | router.add("POST", "/home/act", homeActPost); |
| 917 | 919 | router.add("POST", "/desk", deskLeave); |
| 918 | 920 | router.add("POST", "/desk/clear", deskClear); |
| @@ -1044,7 +1046,8 @@ export function buildRouter(): Router<Handler> { | ||
| 1044 | 1046 | router.add("GET", "/new/issue", newIssueAnywhere); |
| 1045 | 1047 | router.add("GET", "/repos", myReposPage); |
| 1046 | 1048 | router.add("GET", "/displays", displaysPage); |
| 1047 | // A repository's display, read only (`display.ts`): 404 without a display.json or without read access. | |
| 1049 | // A repository's display, read only (`display.ts`): 404 without a display.json or without read | |
| 1050 | // access. | |
| 1048 | 1051 | router.add("GET", "/:owner/:name/display", displayPage); |
| 1049 | 1052 | router.add("POST", "/:owner/:name/display", displayChoose); |
| 1050 | 1053 | router.add("GET", "/now/:stage", nowPage); |
| @@ -1077,11 +1080,13 @@ export function buildRouter(): Router<Handler> { | ||
| 1077 | 1080 | // the clone. Literal routes, above /:owner for the same reason. |
| 1078 | 1081 | router.add("GET", "/migrate", migratePage); |
| 1079 | 1082 | router.add("POST", "/migrate", migrateSubmit); |
| 1083 | router.add("GET", "/migrate/inventory", inventoryPage); | |
| 1080 | 1084 | router.add("GET", "/migrate/jobs/:job", migrationJobPage); |
| 1081 | 1085 | // Every app on one canvas; a literal route, so it must sit above /:owner. |
| 1082 | 1086 | router.add("GET", "/map", globalMapPage); |
| 1083 | // The pipeline moved to a panel on /boards (#2774); the old address follows it. Literal, above /:owner. | |
| 1084 | // /flow is the week's valves (#8698); a query string is an old pipeline drill, which went to /boards. | |
| 1087 | // The pipeline moved to a panel on /boards (#2774); the old address follows it. Literal, above | |
| 1088 | // /:owner. /flow is the week's valves (#8698); a query string is an old pipeline drill, which | |
| 1089 | // went to /boards. | |
| 1085 | 1090 | router.add("GET", "/flow", flowPage); |
| 1086 | 1091 | router.add("POST", "/flow/valve", valveSet); |
| 1087 | 1092 | router.add("POST", "/flow/backup/retry", backupRetry); |
| @@ -2733,7 +2738,8 @@ export async function serve(app: App): Promise<Servers> { | ||
| 2733 | 2738 | }); |
| 2734 | 2739 | // The minute ledger's history: one write a minute, never on a request. |
| 2735 | 2740 | const trafficWriter = createTrafficWriter(app.db, liveOf(app).perfRecord.ledger, app.log); |
| 2736 | // Page views are buffered by the beacon doors and written here; GitHub's traffic is pulled daily when a token is set. | |
| 2741 | // Page views are buffered by the beacon doors and written here; GitHub's traffic is pulled daily | |
| 2742 | // when a token is set. | |
| 2737 | 2743 | const viewWriter = createViewWriter(app.db, app.log); |
| 2738 | 2744 | const githubPull = startGithubPull(app.db, app.log); |
| 2739 | 2745 | const featureSchedule = startFeatureSchedule(app); |
src/origins/inventory.ts added +168
| @@ -0,0 +1,168 @@ | ||
| 1 | /** | |
| 2 | * The inventory: every repository of a connected origin, what it holds, and | |
| 3 | * where its move here stands. One row per repository, plain data out; the | |
| 4 | * page (`web/pages/inventory.ts`) and the MCP's `origins` tool (`inventory`) | |
| 5 | * draw the same rows. | |
| 6 | * | |
| 7 | * What it holds is read from the origin with a few bounded reads per | |
| 8 | * repository (`inventory_detect.ts`), cached until the repository is pushed | |
| 9 | * to again. Where it stands is the forge's own records, never a guess: | |
| 10 | * | |
| 11 | * not_started no repository here came from it | |
| 12 | * migrating a migration of it is running now | |
| 13 | * in_forge a repository here came from it | |
| 14 | * at_parity …and the mirror back to it has pushed what the forge holds | |
| 15 | * cut_over …and a domain of its app has had its old DNS written down | |
| 16 | * and has been verified: the name points here | |
| 17 | * | |
| 18 | * Secret values are never read: see `READ_FILES`. | |
| 19 | */ | |
| 20 | ||
| 21 | import type { Database } from "bun:sqlite"; | |
| 22 | import { getAppByRepo } from "../core/apps.ts"; | |
| 23 | import { listDomains } from "../core/domains.ts"; | |
| 24 | import { getMirrorTarget } from "../core/migrate_mirror.ts"; | |
| 25 | import type { OriginRepoRow } from "../core/origins.ts"; | |
| 26 | import { repoIdsMigratedFrom } from "../core/provenance.ts"; | |
| 27 | import { canAdmin, getRepoById } from "../core/repos.ts"; | |
| 28 | import type { User } from "../core/users.ts"; | |
| 29 | import { runningJob } from "./jobs.ts"; | |
| 30 | import { detect, filesToRead, NOTHING, type Detected } from "./inventory_detect.ts"; | |
| 31 | import type { Provider } from "./types.ts"; | |
| 32 | ||
| 33 | export const MIGRATION_STATES = ["not_started", "migrating", "in_forge", "at_parity", "cut_over"] as const; | |
| 34 | export type MigrationState = (typeof MIGRATION_STATES)[number]; | |
| 35 | ||
| 36 | export const STATE_WORDS: Readonly<Record<MigrationState, string>> = { | |
| 37 | not_started: "Not started", | |
| 38 | migrating: "Migrating", | |
| 39 | in_forge: "In the forge", | |
| 40 | at_parity: "At parity", | |
| 41 | cut_over: "Cut over", | |
| 42 | }; | |
| 43 | ||
| 44 | export interface InventoryRow extends Detected { | |
| 45 | readonly originId: string; | |
| 46 | readonly fullName: string; | |
| 47 | readonly htmlUrl: string; | |
| 48 | readonly private: boolean; | |
| 49 | /** False when the cap on reads per request left this one for the next. */ | |
| 50 | readonly scanned: boolean; | |
| 51 | readonly state: MigrationState; | |
| 52 | /** `/owner/name` of the repository here that came from it, when there is one. */ | |
| 53 | readonly local: string | null; | |
| 54 | } | |
| 55 | ||
| 56 | /** Repositories read per request, most recently pushed first; the rest say so. */ | |
| 57 | export const SCAN_LIMIT = 40; | |
| 58 | const CONCURRENCY = 6; | |
| 59 | const FILE_CAP = 64 * 1024; | |
| 60 | const CACHE_MAX = 500; | |
| 61 | ||
| 62 | /** What was read, per forge (its database), so two forges never share a read. */ | |
| 63 | const caches = new WeakMap<object, Map<string, Detected>>(); | |
| 64 | function cacheOf(db: object): Map<string, Detected> { | |
| 65 | let cache = caches.get(db); | |
| 66 | if (cache === undefined) caches.set(db, (cache = new Map())); | |
| 67 | return cache; | |
| 68 | } | |
| 69 | ||
| 70 | /** | |
| 71 | * Keep what was read of a repository, as a scan would: a preview's seed holds no origin to read. | |
| 72 | */ | |
| 73 | export function rememberRead(db: Database, row: Pick<OriginRepoRow, "originId" | "fullName" | "pushedAt">, read: Detected): void { | |
| 74 | cacheOf(db).set(`${row.originId}:${row.fullName}:${row.pushedAt ?? ""}`, read); | |
| 75 | } | |
| 76 | ||
| 77 | const TEXT = new TextDecoder(); | |
| 78 | ||
| 79 | /** One repository's holdings: its root tree, then only the files `filesToRead` names. */ | |
| 80 | export async function scanRepo(provider: Provider, row: OriginRepoRow): Promise<Detected | undefined> { | |
| 81 | const ref = row.defaultBranch === "" ? "HEAD" : row.defaultBranch; | |
| 82 | const tree = await provider.listTree(row.fullName, ref, ""); | |
| 83 | if (!tree.ok) return undefined; | |
| 84 | const wanted = filesToRead(tree.value.filter((e) => e.kind === "blob").map((e) => e.name)); | |
| 85 | const files: Record<string, string> = {}; | |
| 86 | await Promise.all( | |
| 87 | wanted.map(async (name) => { | |
| 88 | const blob = await provider.getBlob(row.fullName, ref, name); | |
| 89 | if (blob.ok && !blob.value.truncated && blob.value.size <= FILE_CAP) files[name] = TEXT.decode(blob.value.content); | |
| 90 | }), | |
| 91 | ); | |
| 92 | return detect(files); | |
| 93 | } | |
| 94 | ||
| 95 | async function scanAll(db: Database, provider: Provider, rows: readonly OriginRepoRow[]): Promise<Map<string, Detected>> { | |
| 96 | const cache = cacheOf(db); | |
| 97 | const found = new Map<string, Detected>(); | |
| 98 | const todo: OriginRepoRow[] = []; | |
| 99 | for (const row of rows.slice(0, SCAN_LIMIT)) { | |
| 100 | const key = `${row.originId}:${row.fullName}:${row.pushedAt ?? ""}`; | |
| 101 | const hit = cache.get(key); | |
| 102 | if (hit !== undefined) found.set(row.fullName, hit); | |
| 103 | else todo.push(row); | |
| 104 | } | |
| 105 | let next = 0; | |
| 106 | const worker = async (): Promise<void> => { | |
| 107 | for (let row = todo[next++]; row !== undefined; row = todo[next++]) { | |
| 108 | const detected = await scanRepo(provider, row).catch(() => undefined); | |
| 109 | if (detected === undefined) continue; | |
| 110 | if (cache.size >= CACHE_MAX) cache.delete(cache.keys().next().value as string); | |
| 111 | cache.set(`${row.originId}:${row.fullName}:${row.pushedAt ?? ""}`, detected); | |
| 112 | found.set(row.fullName, detected); | |
| 113 | } | |
| 114 | }; | |
| 115 | await Promise.all(Array.from({ length: Math.min(CONCURRENCY, todo.length) }, worker)); | |
| 116 | return found; | |
| 117 | } | |
| 118 | ||
| 119 | /** Where a repository's move stands, from this forge's records. */ | |
| 120 | export function stateOf( | |
| 121 | db: Database, | |
| 122 | user: User, | |
| 123 | row: Pick<OriginRepoRow, "originId" | "fullName" | "htmlUrl" | "mirroredAs">, | |
| 124 | ): { state: MigrationState; local: string | null } { | |
| 125 | const ids = [...(row.mirroredAs === null ? [] : [row.mirroredAs]), ...repoIdsMigratedFrom(db, row.htmlUrl)]; | |
| 126 | for (const id of ids) { | |
| 127 | const repo = getRepoById(db, id); | |
| 128 | if (repo === undefined || !canAdmin(db, repo, user)) continue; | |
| 129 | const local = `/${repo.ownerName}/${repo.name}`; | |
| 130 | const app = getAppByRepo(db, repo.id); | |
| 131 | if (app !== undefined && listDomains(db, app.id).some((d) => d.verifiedAt !== null && d.oldRecords.length > 0)) | |
| 132 | return { state: "cut_over", local }; | |
| 133 | const mirror = getMirrorTarget(db, repo.id); | |
| 134 | if (mirror !== undefined && mirror.pushedAt !== "" && mirror.lastError === "") return { state: "at_parity", local }; | |
| 135 | return { state: "in_forge", local }; | |
| 136 | } | |
| 137 | const running = | |
| 138 | runningJob(row.originId, row.fullName, "migrate", user.id) ?? runningJob("", row.fullName, "migrate", user.id); | |
| 139 | return { state: running === undefined ? "not_started" : "migrating", local: null }; | |
| 140 | } | |
| 141 | ||
| 142 | /** Every repository of one origin as an inventory row, apps first, then by name. */ | |
| 143 | export async function inventoryRows( | |
| 144 | db: Database, | |
| 145 | user: User, | |
| 146 | provider: Provider, | |
| 147 | repos: readonly OriginRepoRow[], | |
| 148 | ): Promise<InventoryRow[]> { | |
| 149 | const recent = [...repos].sort((a, b) => (b.pushedAt ?? "").localeCompare(a.pushedAt ?? "")); | |
| 150 | const found = await scanAll(db, provider, recent); | |
| 151 | return repos | |
| 152 | .map((r): InventoryRow => { | |
| 153 | const seen = found.get(r.fullName); | |
| 154 | return { | |
| 155 | ...(seen ?? NOTHING), | |
| 156 | originId: r.originId, | |
| 157 | fullName: r.fullName, | |
| 158 | htmlUrl: r.htmlUrl, | |
| 159 | private: r.private, | |
| 160 | scanned: seen !== undefined, | |
| 161 | ...stateOf(db, user, r), | |
| 162 | }; | |
| 163 | }) | |
| 164 | .sort( | |
| 165 | (a, b) => | |
| 166 | Number(b.app !== null) - Number(a.app !== null) || a.fullName.localeCompare(b.fullName, "en", { sensitivity: "base" }), | |
| 167 | ); | |
| 168 | } |
src/origins/inventory_detect.ts added +135
| @@ -0,0 +1,135 @@ | ||
| 1 | /** | |
| 2 | * What a repository holds, read from a handful of its root files: is it an | |
| 3 | * app, does it want a Postgres database, which domains does it name and which | |
| 4 | * secrets does it need. Pure: file names and file text in, plain data out. | |
| 5 | * | |
| 6 | * Secrets are NAMES. Only files that carry names are ever asked for | |
| 7 | * (`READ_FILES`: examples and templates, never `.env`), and every parser here | |
| 8 | * keeps the name and drops what follows it. | |
| 9 | */ | |
| 10 | ||
| 11 | import { isHostname } from "../core/domains.ts"; | |
| 12 | ||
| 13 | /** The files that say "this is an app", and which kind of app. */ | |
| 14 | export const DEPLOY_FILES: Readonly<Record<string, "forge" | "platform">> = { | |
| 15 | "forge.json": "forge", | |
| 16 | "platform.yaml": "platform", | |
| 17 | "platform.yml": "platform", | |
| 18 | ".platform.yaml": "platform", | |
| 19 | ".platform.app.yaml": "platform", | |
| 20 | }; | |
| 21 | ||
| 22 | /** The only root files ever fetched. `.env` is not here, and must never be. */ | |
| 23 | export const READ_FILES: readonly string[] = [ | |
| 24 | ...Object.keys(DEPLOY_FILES), | |
| 25 | ".env.example", | |
| 26 | ".env.sample", | |
| 27 | ".env.template", | |
| 28 | "docker-compose.yml", | |
| 29 | "docker-compose.yaml", | |
| 30 | "package.json", | |
| 31 | ]; | |
| 32 | ||
| 33 | export interface Detected { | |
| 34 | readonly app: "forge" | "platform" | null; | |
| 35 | readonly postgres: boolean; | |
| 36 | readonly domains: readonly string[]; | |
| 37 | readonly secrets: readonly string[]; | |
| 38 | } | |
| 39 | ||
| 40 | export const NOTHING: Detected = { app: null, postgres: false, domains: [], secrets: [] }; | |
| 41 | ||
| 42 | const MAX_NAMES = 30; | |
| 43 | const SECRET_NAME = /^[A-Za-z_][A-Za-z0-9_]{0,63}$/; | |
| 44 | const POSTGRES = /postgres(?:ql)?:\/\/|\bpostgres(?:ql)?\b|DATABASE_URL|\bPGHOST\b|\bPGDATABASE\b|"pg"\s*:/i; | |
| 45 | ||
| 46 | /** The root files worth fetching, given the names in the root tree. */ | |
| 47 | export const filesToRead = (rootNames: readonly string[]): string[] => | |
| 48 | READ_FILES.filter((name) => rootNames.includes(name)); | |
| 49 | ||
| 50 | /** `NAME` of every `NAME=value` line of an env example; the value is never kept. */ | |
| 51 | function envNames(text: string): string[] { | |
| 52 | const out: string[] = []; | |
| 53 | for (const line of text.split("\n")) { | |
| 54 | const m = /^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=/.exec(line); | |
| 55 | if (m?.[1] !== undefined) out.push(m[1]); | |
| 56 | } | |
| 57 | return out; | |
| 58 | } | |
| 59 | ||
| 60 | /** The items of an indented block under `key:` in simple YAML: `- x`, `- name: x`, or `x: …`. */ | |
| 61 | function yamlBlock(text: string, key: string): string[] { | |
| 62 | const lines = text.split("\n"); | |
| 63 | const out: string[] = []; | |
| 64 | const at = lines.findIndex((l) => new RegExp(`^\\s*${key}\\s*:\\s*$`).test(l)); | |
| 65 | if (at < 0) return out; | |
| 66 | const base = /^\s*/.exec(lines[at]!)![0].length; | |
| 67 | for (const line of lines.slice(at + 1)) { | |
| 68 | if (line.trim() === "" || line.trim().startsWith("#")) continue; | |
| 69 | if (/^\s*/.exec(line)![0].length <= base) break; | |
| 70 | const m = /^\s*(?:-\s*)?(?:name\s*:\s*)?["']?([^\s"':#]+)["']?\s*(?::|$|#)/.exec(line); | |
| 71 | if (m?.[1] !== undefined) out.push(m[1]); | |
| 72 | } | |
| 73 | return out; | |
| 74 | } | |
| 75 | ||
| 76 | /** `host: x`, `hostname: x`, `domain: x` on one line. */ | |
| 77 | function yamlHosts(text: string): string[] { | |
| 78 | const out: string[] = []; | |
| 79 | for (const m of text.matchAll(/^\s*-?\s*(?:host|hostname|domain)\s*:\s*["']?([^\s"'#]+)/gim)) | |
| 80 | if (m[1] !== undefined) out.push(m[1]); | |
| 81 | return out; | |
| 82 | } | |
| 83 | ||
| 84 | const parseJson = (text: string): Record<string, unknown> => { | |
| 85 | try { | |
| 86 | const v: unknown = JSON.parse(text); | |
| 87 | return v !== null && typeof v === "object" && !Array.isArray(v) ? (v as Record<string, unknown>) : {}; | |
| 88 | } catch { | |
| 89 | return {}; | |
| 90 | } | |
| 91 | }; | |
| 92 | ||
| 93 | const namesOf = (value: unknown): string[] => | |
| 94 | Array.isArray(value) | |
| 95 | ? value.filter((v): v is string => typeof v === "string") | |
| 96 | : value !== null && typeof value === "object" | |
| 97 | ? Object.keys(value) | |
| 98 | : []; | |
| 99 | ||
| 100 | const unique = (items: readonly string[]): string[] => [...new Set(items)]; | |
| 101 | ||
| 102 | /** | |
| 103 | * What the files hold. `files` maps a root file name to its text; a file not | |
| 104 | * in `READ_FILES` is ignored, so even a caller that passes `.env` gets | |
| 105 | * nothing out of it. | |
| 106 | */ | |
| 107 | export function detect(files: Readonly<Record<string, string>>): Detected { | |
| 108 | let app: Detected["app"] = null; | |
| 109 | const hosts: string[] = []; | |
| 110 | const secrets: string[] = []; | |
| 111 | let postgres = false; | |
| 112 | for (const name of READ_FILES) { | |
| 113 | const text = files[name]; | |
| 114 | if (text === undefined) continue; | |
| 115 | const kind = DEPLOY_FILES[name]; | |
| 116 | if (kind !== undefined && app === null) app = kind; | |
| 117 | if (POSTGRES.test(text)) postgres = true; | |
| 118 | if (name === "forge.json") { | |
| 119 | const json = parseJson(text); | |
| 120 | hosts.push(...namesOf(json["domains"]).filter((v) => typeof v === "string")); | |
| 121 | secrets.push(...namesOf(json["secrets"])); | |
| 122 | } else if (kind === "platform") { | |
| 123 | hosts.push(...yamlBlock(text, "domains"), ...yamlBlock(text, "hosts"), ...yamlHosts(text)); | |
| 124 | secrets.push(...yamlBlock(text, "secrets")); | |
| 125 | } else if (name.startsWith(".env.")) { | |
| 126 | secrets.push(...envNames(text)); | |
| 127 | } | |
| 128 | } | |
| 129 | return { | |
| 130 | app, | |
| 131 | postgres, | |
| 132 | domains: unique(hosts.map((h) => h.toLowerCase()).filter((h) => h.includes(".") && isHostname(h))).slice(0, 10), | |
| 133 | secrets: unique(secrets.filter((s) => SECRET_NAME.test(s))).slice(0, MAX_NAMES), | |
| 134 | }; | |
| 135 | } |
src/spawn/preview-inventory.ts added +70
| @@ -0,0 +1,70 @@ | ||
| 1 | /** | |
| 2 | * The preview's inventory (#646): an origin of the preview admin with three | |
| 3 | * repositories, what each holds already read, so `/migrate/inventory` has rows | |
| 4 | * to draw without an origin to read from. Nothing here reaches a network: the | |
| 5 | * origin's host is `.invalid`, and the reads are kept as a scan would keep them. | |
| 6 | */ | |
| 7 | ||
| 8 | import type { App } from "../app.ts"; | |
| 9 | import { cacheOriginRepos, listOriginRepos, listOrigins, createOrigin } from "../core/origins.ts"; | |
| 10 | import type { User } from "../core/users.ts"; | |
| 11 | import { err, ok, type Result } from "../kernel/result.ts"; | |
| 12 | import type { Detected } from "../origins/inventory_detect.ts"; | |
| 13 | import { rememberRead } from "../origins/inventory.ts"; | |
| 14 | import type { RemoteRepo } from "../origins/types.ts"; | |
| 15 | ||
| 16 | const API = "https://old-platform.invalid/api/v1"; | |
| 17 | const WEB = "https://old-platform.invalid"; | |
| 18 | const LOGIN = "preview-old"; | |
| 19 | ||
| 20 | const REPOS: readonly { name: string; read: Detected }[] = [ | |
| 21 | { name: "shop", read: { app: "forge", postgres: true, domains: ["shop.example.com"], secrets: ["DATABASE_URL", "STRIPE_KEY"] } }, | |
| 22 | { name: "blog", read: { app: "platform", postgres: false, domains: ["blog.example.org", "www.example.org"], secrets: ["SMTP_PASSWORD"] } }, | |
| 23 | { name: "notes", read: { app: null, postgres: false, domains: [], secrets: [] } }, | |
| 24 | ]; | |
| 25 | ||
| 26 | const remote = (name: string, id: number): RemoteRepo => ({ | |
| 27 | remoteId: String(id), | |
| 28 | fullName: `${LOGIN}/${name}`, | |
| 29 | owner: LOGIN, | |
| 30 | name, | |
| 31 | description: "", | |
| 32 | private: false, | |
| 33 | fork: false, | |
| 34 | parent: null, | |
| 35 | defaultBranch: "main", | |
| 36 | cloneUrl: `${WEB}/${LOGIN}/${name}.git`, | |
| 37 | sshUrl: "", | |
| 38 | htmlUrl: `${WEB}/${LOGIN}/${name}`, | |
| 39 | pushedAt: `2026-01-0${String(id)}T00:00:00Z`, | |
| 40 | permissions: { push: false, admin: false }, | |
| 41 | openIssues: 0, | |
| 42 | openPulls: 0, | |
| 43 | openIssuesIncludesPulls: false, | |
| 44 | sizeKb: 0, | |
| 45 | }); | |
| 46 | ||
| 47 | /** Once per boot: the origin and its rows are kept in the database, the reads in memory. */ | |
| 48 | export function seedInventory(app: App, admin: User): Result<void, string> { | |
| 49 | let origin = listOrigins(app.db, admin.id).find((o) => o.login === LOGIN); | |
| 50 | if (origin === undefined) { | |
| 51 | const made = createOrigin(app.db, app.config.dataDir, { | |
| 52 | userId: admin.id, | |
| 53 | kind: "forgejo", | |
| 54 | apiUrl: API, | |
| 55 | webUrl: WEB, | |
| 56 | login: LOGIN, | |
| 57 | token: "preview-sample-not-a-credential", | |
| 58 | readOnly: true, | |
| 59 | }); | |
| 60 | if (!made.ok) return err(`the preview origin: ${made.error}`); | |
| 61 | origin = made.value; | |
| 62 | } | |
| 63 | if (listOriginRepos(app.db, origin.id).length === 0) | |
| 64 | cacheOriginRepos(app.db, origin.id, REPOS.map((r, i) => remote(r.name, i + 1))); | |
| 65 | for (const row of listOriginRepos(app.db, origin.id)) { | |
| 66 | const known = REPOS.find((r) => row.fullName === `${LOGIN}/${r.name}`); | |
| 67 | if (known !== undefined) rememberRead(app.db, row, known.read); | |
| 68 | } | |
| 69 | return ok(undefined); | |
| 70 | } |
src/spawn/preview-looks.ts +12 −6
| @@ -1,11 +1,10 @@ | ||
| 1 | 1 | /** |
| 2 | 2 | * The looks every preview of the forge takes as its owner (#8974): home |
| 3 | 3 | * with its pins, /flow, the seeded repository's head, its pull's page and |
| 4 | * its Iterations page (#9182), the seeded app's page (#9782), the Later gate's proposal (#9212), its | |
| 5 | * milestones (#9326) and its waves (#9325), each at desk and phone width. | |
| 6 | * The looks service adds them to whatever forge.json lists for a preview | |
| 7 | * of the forge's own repository (`FORGE_REPO`), so no edit of forge.json can | |
| 8 | * drop one. | |
| 4 | * its Iterations page (#9182), the seeded app's page (#9782), the Later gate's proposal (#9212), | |
| 5 | * its milestones (#9326) and its waves (#9325), each at desk and phone width. The looks service | |
| 6 | * adds them to whatever forge.json lists for a preview of the forge's own repository | |
| 7 | * (`FORGE_REPO`), so no edit of forge.json can drop one. | |
| 9 | 8 | * |
| 10 | 9 | * Plain data and no imports beyond a type: the seed (`preview-seed.ts`) |
| 11 | 10 | * makes what these pages show, and the looks service reads the list. |
| @@ -112,7 +111,10 @@ export const PREVIEW_ISSUES_PAGE = `${PREVIEW_REPO_HOME}/issues`; | ||
| 112 | 111 | /** The pulls list, whose long titles must wrap at a 390 px phone (#9346). */ |
| 113 | 112 | export const PREVIEW_PULLS_PAGE = `${PREVIEW_REPO_HOME}/pulls`; |
| 114 | 113 | |
| 115 | /** The owner's resilience page, whose protections and test paths must wrap at a 390 px phone (#9328). */ | |
| 114 | /** | |
| 115 | * The owner's resilience page, whose protections and test paths must wrap at a 390 px phone | |
| 116 | * (#9328). | |
| 117 | */ | |
| 116 | 118 | export const PREVIEW_RESILIENCE_PAGE = "/resilience"; |
| 117 | 119 | |
| 118 | 120 | /** The changelog, whose long landing titles must wrap at a 390 px phone (#9349). */ |
| @@ -187,6 +189,9 @@ export const PREVIEW_NEWCOMER_PAGE = "/?as=newcomer"; | ||
| 187 | 189 | */ |
| 188 | 190 | export const PREVIEW_SIGNED_OUT_PAGE = "/?as=signedout"; |
| 189 | 191 | |
| 192 | /** What is left to bring over, with the seeded origin's three repositories (#646). */ | |
| 193 | export const PREVIEW_INVENTORY_PAGE = "/migrate/inventory"; | |
| 194 | ||
| 190 | 195 | export interface OwnerLook { |
| 191 | 196 | readonly page: string; |
| 192 | 197 | readonly kind: LookKind; |
| @@ -227,6 +232,7 @@ export const PREVIEW_OWNER_LOOKS: readonly OwnerLook[] = [ | ||
| 227 | 232 | ...PREVIEW_SETTINGS_PAGES, |
| 228 | 233 | "/settings", |
| 229 | 234 | "/forges", |
| 235 | PREVIEW_INVENTORY_PAGE, | |
| 230 | 236 | ].flatMap((page) => |
| 231 | 237 | (["desktop", "phone"] as const).map((kind): OwnerLook => |
| 232 | 238 | page === PREVIEW_NEWCOMER_PAGE |
src/spawn/preview-seed.ts +15 −4
| @@ -19,6 +19,7 @@ import { keepNotice, noticeOf } from "../billing/notice.ts"; | ||
| 19 | 19 | import { putEntitlement } from "../billing/store.ts"; |
| 20 | 20 | import { join } from "node:path"; |
| 21 | 21 | import type { App } from "../app.ts"; |
| 22 | import { seedInventory } from "./preview-inventory.ts"; | |
| 22 | 23 | import { createOrg } from "../core/orgs.ts"; |
| 23 | 24 | import type { Ctx } from "../http/context.ts"; |
| 24 | 25 | import type { Issue } from "../core/issues.ts"; |
| @@ -338,6 +339,8 @@ export async function seedPreview( | ||
| 338 | 339 | const newcomer = await seedNewcomer(app); |
| 339 | 340 | if (!newcomer.ok) return newcomer; |
| 340 | 341 | seedGrace(app); |
| 342 | const inventory = seedInventory(app, admin); | |
| 343 | if (!inventory.ok) return inventory; | |
| 341 | 344 | return ok({ seeded: false }); |
| 342 | 345 | } |
| 343 | 346 | const world = await seedWorld(app, admin); |
| @@ -356,6 +359,8 @@ export async function seedPreview( | ||
| 356 | 359 | const newcomer = await seedNewcomer(app); |
| 357 | 360 | if (!newcomer.ok) return newcomer; |
| 358 | 361 | seedGrace(app); |
| 362 | const inventory = seedInventory(app, admin); | |
| 363 | if (!inventory.ok) return inventory; | |
| 359 | 364 | return ok({ seeded: true }); |
| 360 | 365 | } |
| 361 | 366 | |
| @@ -399,7 +404,8 @@ function seedGrace(app: App): void { | ||
| 399 | 404 | */ |
| 400 | 405 | async function seedSettingsPages(app: App, admin: User): Promise<Result<void, string>> { |
| 401 | 406 | if (getUserSetting(app.db, admin.id, SETTINGS_SEEDED) !== undefined) return ok(undefined); |
| 402 | // Guarded like the agent and the subscription: a refused seed retries on the next boot and adds no second token. | |
| 407 | // Guarded like the agent and the subscription: a refused seed retries on the next boot and adds | |
| 408 | // no second token. | |
| 403 | 409 | if (!listTokens(app.db, admin.id).some((t) => t.name === PREVIEW_TOKEN)) |
| 404 | 410 | createToken(app.db, admin.id, PREVIEW_TOKEN, "ro"); |
| 405 | 411 | if (getAgentByName(app.db, PREVIEW_AGENT) === undefined) { |
| @@ -594,7 +600,8 @@ async function seedWorld(app: App, admin: User): Promise<Result<void, string>> { | ||
| 594 | 600 | if (!waved.ok) return waved; |
| 595 | 601 | const bets = seedBets(app, admin, repo.value, open); |
| 596 | 602 | if (!bets.ok) return bets; |
| 597 | // Before the forks, so the merge moves no fork; after the numbered issues, so they keep their numbers. | |
| 603 | // Before the forks, so the merge moves no fork; after the numbered issues, so they keep their | |
| 604 | // numbers. | |
| 598 | 605 | const landed = await seedLanding(app, admin, repo.value); |
| 599 | 606 | if (!landed.ok) return landed; |
| 600 | 607 | const family = await seedForkFamily(app, admin, repo.value, start.value.commitOid); |
| @@ -765,7 +772,8 @@ function seedBets(app: App, admin: User, repo: Repo, open: readonly Issue[]): Re | ||
| 765 | 772 | ); |
| 766 | 773 | const backed = bet("Show the week", 2); |
| 767 | 774 | bet("Price the list", 4); |
| 768 | // The worst case for the phone: a long name, an unbroken word, more journeys than fit at rest (#8970). | |
| 775 | // The worst case for the phone: a long name, an unbroken word, more journeys than fit at rest | |
| 776 | // (#8970). | |
| 769 | 777 | bet( |
| 770 | 778 | "Make every long and winding bet name wrap instead of pushing the page sideways", |
| 771 | 779 | 0, |
| @@ -878,7 +886,10 @@ async function seedApp(app: App, admin: User): Promise<Result<void, string>> { | ||
| 878 | 886 | return answer.status === 200 ? ok() : err(`${PREVIEW_APP_PAGE} would answer ${String(answer.status)}, not 200`); |
| 879 | 887 | } |
| 880 | 888 | |
| 881 | /** Ten recorded pull runs for the app page's Adapter section (#10158): one failed with a reason of the longest allowed, an unbroken word in it. */ | |
| 889 | /** | |
| 890 | * Ten recorded pull runs for the app page's Adapter section (#10158): one failed with a reason of | |
| 891 | * the longest allowed, an unbroken word in it. | |
| 892 | */ | |
| 882 | 893 | export const PREVIEW_LONG_REASON = `upstream refused the pull: ${"x".repeat(60)} ${"https://api.example.invalid/v2/orders?cursor=".repeat(3)}`.slice(0, 200); |
| 883 | 894 | function seedPullRuns(app: App, appId: string): void { |
| 884 | 895 | for (let i = 0; i < 10; i++) { |
src/tui/screens/entries.ts +5 −1
| @@ -13,6 +13,7 @@ import { entry as accountAccess } from "./account_access.ts"; | ||
| 13 | 13 | import { entry as agents } from "./agents.ts"; |
| 14 | 14 | import { entry as askData } from "./ask_data.ts"; |
| 15 | 15 | import { entry as archived } from "./archived.ts"; |
| 16 | import { entry as inventory } from "./inventory.ts"; | |
| 16 | 17 | import { entry as audit } from "./audit.ts"; |
| 17 | 18 | import { entry as appTemplates } from "./app_templates.ts"; |
| 18 | 19 | import { entry as dotfiles } from "./dotfiles.ts"; |
| @@ -186,6 +187,7 @@ export const ENTRIES: readonly PaletteEntry[] = [ | ||
| 186 | 187 | orgGrants, |
| 187 | 188 | agents, |
| 188 | 189 | archived, |
| 190 | inventory, | |
| 189 | 191 | invites, |
| 190 | 192 | billingLinks, |
| 191 | 193 | planRow, |
| @@ -225,7 +227,9 @@ export const ENTRIES: readonly PaletteEntry[] = [ | ||
| 225 | 227 | newIssue, |
| 226 | 228 | ]; |
| 227 | 229 | |
| 228 | /** Put each entry the session is offered into `commands`, after the command it names or at the end. */ | |
| 230 | /** | |
| 231 | * Put each entry the session is offered into `commands`, after the command it names or at the end. | |
| 232 | */ | |
| 229 | 233 | export function addEntries(commands: Command[], session: Session, ctx: Ctx): void { |
| 230 | 234 | const placed = new Map<string, number>(); |
| 231 | 235 | for (const e of ENTRIES) { |
src/tui/screens/inventory.ts added +97
| @@ -0,0 +1,97 @@ | ||
| 1 | /** | |
| 2 | * /migrate/inventory in the terminal: every repository of the connected | |
| 3 | * origins, what it holds and where its move stands. The rows and the words | |
| 4 | * are `actions/inventory.ts`'s, the page's and the MCP's. `b` brings the | |
| 5 | * repository in (the migrate picker), `n` starts a new forge named for it, | |
| 6 | * `o` opens it here once it is. Offered to whoever is signed in, as the page. | |
| 7 | */ | |
| 8 | ||
| 9 | import { holds, inventoryOfOrigins, type InventoryItem } from "../../actions/inventory.ts"; | |
| 10 | import { listUserOrigins } from "../../actions/origins.ts"; | |
| 11 | import { STATE_WORDS } from "../../origins/inventory.ts"; | |
| 12 | import type { Ctx, Screen } from "../app/app.ts"; | |
| 13 | import { binding } from "../app/keys.ts"; | |
| 14 | import { style } from "../term/style.ts"; | |
| 15 | import { truncate } from "../term/width.ts"; | |
| 16 | import type { Session } from "../session.ts"; | |
| 17 | import { fill, len, split } from "../ui/layout.ts"; | |
| 18 | import { emptyState } from "../ui/widgets.ts"; | |
| 19 | import { ListState, drawList, runAsync, type ListRow } from "./common.ts"; | |
| 20 | import type { PaletteEntry } from "./entries.ts"; | |
| 21 | import { migrateScreen } from "./migrate.ts"; | |
| 22 | import { newForgeScreen } from "./newforge.ts"; | |
| 23 | ||
| 24 | export function inventoryScreen(f: Session, ctx: Ctx): Screen { | |
| 25 | const user = f.user; | |
| 26 | const state = new ListState(); | |
| 27 | let items: InventoryItem[] = []; | |
| 28 | let unreachable: string[] = []; | |
| 29 | let loading = true; | |
| 30 | const selected = (): InventoryItem | undefined => items[state.index]; | |
| 31 | ||
| 32 | const load = (): void => { | |
| 33 | if (user === undefined) return; | |
| 34 | loading = true; | |
| 35 | runAsync(ctx, async () => { | |
| 36 | const got = await inventoryOfOrigins(f.app, user, listUserOrigins(f.app, user)); | |
| 37 | items = got.items; | |
| 38 | unreachable = got.unreachable; | |
| 39 | loading = false; | |
| 40 | state.clamp(items.length); | |
| 41 | ctx.invalidate(); | |
| 42 | }); | |
| 43 | }; | |
| 44 | load(); | |
| 45 | ||
| 46 | return { | |
| 47 | title: "What is left to bring over", | |
| 48 | keys: [binding(["j", "k"], "move cursor"), binding(["b"], "bring in"), binding(["n"], "new forge"), binding(["g"], "read again")], | |
| 49 | render(area, s) { | |
| 50 | const c = ctx.theme.c; | |
| 51 | const [head, body] = split(area, "v", [len(1), fill(1)]); | |
| 52 | const said = | |
| 53 | unreachable.length === 0 ? "b brings it in · n starts a new forge" : `Could not reach ${unreachable.join(", ")}`; | |
| 54 | s.text(head!.x, head!.y, truncate(said, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w); | |
| 55 | if (items.length === 0) { | |
| 56 | emptyState(s, body!, c, { title: loading ? "Reading…" : unreachable.length > 0 ? "Nothing could be read." : "Nothing listed." }); | |
| 57 | return; | |
| 58 | } | |
| 59 | const rows: ListRow[] = items.map((i) => ({ | |
| 60 | title: i.fullName, | |
| 61 | secondary: STATE_WORDS[i.state], | |
| 62 | meta: holds(i), | |
| 63 | })); | |
| 64 | drawList(s, body!, c, state, rows); | |
| 65 | }, | |
| 66 | key(e) { | |
| 67 | if (state.key(e, items.length)) return true; | |
| 68 | if (e.ctrl || e.alt) return false; | |
| 69 | if (e.key === "g") { | |
| 70 | load(); | |
| 71 | return true; | |
| 72 | } | |
| 73 | const item = selected(); | |
| 74 | if (item === undefined) return false; | |
| 75 | if (e.key === "b" && item.state === "not_started") { | |
| 76 | ctx.push(migrateScreen(f, ctx)); | |
| 77 | return true; | |
| 78 | } | |
| 79 | if (e.key === "n" && item.state === "not_started") { | |
| 80 | const name = new URL(item.newForge, "http://x").searchParams.get("name") ?? ""; | |
| 81 | ctx.push(newForgeScreen(f, ctx, { name })); | |
| 82 | return true; | |
| 83 | } | |
| 84 | return false; | |
| 85 | }, | |
| 86 | }; | |
| 87 | } | |
| 88 | ||
| 89 | /** What is left on the connected origins, as /migrate/inventory has it. */ | |
| 90 | export const entry: PaletteEntry = { | |
| 91 | id: "inventory", | |
| 92 | title: "What is left to bring over", | |
| 93 | hint: "every repository on your connected accounts, what it holds, where its move stands", | |
| 94 | after: ["migrate"], | |
| 95 | offered: (session) => session.user !== undefined, | |
| 96 | run: (session, ctx) => ctx.push(inventoryScreen(session, ctx)), | |
| 97 | }; |
src/web/pages/inventory.ts added +95
| @@ -0,0 +1,95 @@ | ||
| 1 | /** | |
| 2 | * GET /migrate/inventory — what is left on the origins you connected: every | |
| 3 | * repository, what it holds (app, Postgres, domains, secret NAMES), where its | |
| 4 | * move here stands, and one action. The owner who wants to retire the old | |
| 5 | * platform reads it top down: the primary action brings the next app over. | |
| 6 | * The rows are `actions/inventory.ts`'s, the same the MCP answers. | |
| 7 | */ | |
| 8 | ||
| 9 | import { holds, inventoryOfOrigins, type InventoryItem } from "../../actions/inventory.ts"; | |
| 10 | import { listUserOrigins } from "../../actions/origins.ts"; | |
| 11 | import { MIGRATION_STATES, STATE_WORDS, type MigrationState } from "../../origins/inventory.ts"; | |
| 12 | import { notePhase, takeSessionTime, timePhaseAsync } from "../../core/route_timing.ts"; | |
| 13 | import type { Ctx } from "../../http/context.ts"; | |
| 14 | import { notFoundPage, pageHead, signInFirst } from "../chrome.ts"; | |
| 15 | import { html, type Html } from "../html.ts"; | |
| 16 | import { page } from "../layout.ts"; | |
| 17 | import { badge, btn, empty, row, type Tone } from "../ui.ts"; | |
| 18 | ||
| 19 | const TONE: Record<MigrationState, Tone> = { | |
| 20 | not_started: "default", | |
| 21 | migrating: "attn", | |
| 22 | in_forge: "default", | |
| 23 | at_parity: "success", | |
| 24 | cut_over: "success", | |
| 25 | }; | |
| 26 | ||
| 27 | function inventoryRow(item: InventoryItem): Html { | |
| 28 | const started = item.state !== "not_started" && item.state !== "migrating"; | |
| 29 | const actions = | |
| 30 | item.local !== null && started | |
| 31 | ? [btn("Open here", { href: item.local, size: "sm", variant: "outline" })] | |
| 32 | : item.state === "migrating" | |
| 33 | ? [] | |
| 34 | : [ | |
| 35 | btn("Bring into this forge", { href: item.bring, size: "sm", variant: "outline" }), | |
| 36 | btn("Into a new forge", { href: item.newForge, size: "sm", variant: "ghost" }), | |
| 37 | ]; | |
| 38 | return row({ | |
| 39 | title: item.fullName, | |
| 40 | meta: html`${holds(item)}${item.private ? " · private" : ""}`, | |
| 41 | facts: [ | |
| 42 | ...(item.domains.length > 0 ? [{ label: "Domains", value: item.domains.join(", ") }] : []), | |
| 43 | ...(item.secrets.length > 0 ? [{ label: "Secret names", value: item.secrets.join(", ") }] : []), | |
| 44 | ], | |
| 45 | trail: html`${badge(STATE_WORDS[item.state], { tone: TONE[item.state] })} ${actions}`, | |
| 46 | data: { state: item.state, repo: item.fullName }, | |
| 47 | }); | |
| 48 | } | |
| 49 | ||
| 50 | /** The count line: how many are in each state, in the order a move goes. */ | |
| 51 | function tally(items: readonly InventoryItem[]): string { | |
| 52 | return MIGRATION_STATES.map((s) => [s, items.filter((i) => i.state === s).length] as const) | |
| 53 | .filter(([, n]) => n > 0) | |
| 54 | .map(([s, n]) => `${n} ${STATE_WORDS[s].toLowerCase()}`) | |
| 55 | .join(" · "); | |
| 56 | } | |
| 57 | ||
| 58 | export async function inventoryPage(ctx: Ctx): Promise<Response> { | |
| 59 | if (!ctx.auth.user) return signInFirst(ctx); | |
| 60 | const user = ctx.auth.user; | |
| 61 | notePhase(ctx.request, "session", takeSessionTime(ctx.request)); | |
| 62 | const origins = listUserOrigins(ctx.app, user); | |
| 63 | const asked = ctx.url.searchParams.get("origin") ?? ""; | |
| 64 | const chosen = asked === "" ? origins : origins.filter((o) => o.id === asked); | |
| 65 | if (asked !== "" && chosen.length === 0) return notFoundPage(ctx); | |
| 66 | const { items, unreachable } = await timePhaseAsync(ctx.request, "inventory", () => | |
| 67 | inventoryOfOrigins(ctx.app, user, chosen), | |
| 68 | ); | |
| 69 | const next = items.find((i) => i.app !== null && i.state === "not_started"); | |
| 70 | const rendering = performance.now(); | |
| 71 | const body = | |
| 72 | origins.length === 0 | |
| 73 | ? empty({ | |
| 74 | icon: "download", | |
| 75 | title: "No account connected", | |
| 76 | note: "Connect the account the apps live on, and what is left there is listed here.", | |
| 77 | action: btn("Connect an account", { href: "/migrate", variant: "primary" }), | |
| 78 | }) | |
| 79 | : html`${pageHead("What is left to bring over", { | |
| 80 | sub: items.length === 0 && unreachable.length === 0 ? "Nothing listed." : tally(items), | |
| 81 | actions: | |
| 82 | next === undefined | |
| 83 | ? undefined | |
| 84 | : btn(`Bring ${next.fullName}`, { href: next.bring, variant: "primary", icon: "download" }), | |
| 85 | })} | |
| 86 | ${unreachable.length === 0 | |
| 87 | ? null | |
| 88 | : html`<p class="muted small mb" id="inventory-unreachable"> | |
| 89 | Could not reach ${unreachable.join(", ")}. | |
| 90 | </p>`} | |
| 91 | <ul class="list" id="inventory">${items.map(inventoryRow)}</ul>`; | |
| 92 | const response = page({ ctx, title: "Inventory", body }); | |
| 93 | notePhase(ctx.request, "render", performance.now() - rendering); | |
| 94 | return response; | |
| 95 | } |
src/web/pages/migrate.ts +5
| @@ -454,6 +454,11 @@ function migrateForm( | ||
| 454 | 454 | ${icon("globe")} ${refused.join(" ")} |
| 455 | 455 | </p>`} |
| 456 | 456 | ${accounts(ctx, origins)} |
| 457 | ${origins.length === 0 | |
| 458 | ? null | |
| 459 | : html`<p class="mb" id="migrate-inventory-link"> | |
| 460 | ${btn("See what is left to bring over", { href: "/migrate/inventory", variant: "outline", icon: "download" })} | |
outdated
Principle 5: a second primary button competes with the form's own primary action on this page. Make this one a plain link (variant outline or ghost). Its explanation repeats the label (principle 1). |
||
| 461 | </p>`} | |
| 457 | 462 | <form method="get" action="/migrate" id="migrate-search"></form> |
| 458 | 463 | <form method="post" action="/migrate"> |
| 459 | 464 | <h2 class="mb">Source</h2> |
test/origin-inventory-edges.test.ts added +153
| @@ -0,0 +1,153 @@ | ||
| 1 | /** | |
| 2 | * The inventory's edges (#646): a move under way, the cut at SCAN_LIMIT, the | |
| 3 | * cache, an origin that does not answer, an unknown origin through the MCP, | |
| 4 | * and the terminal screen. | |
| 5 | */ | |
| 6 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 7 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { createApp, type App } from "../src/app.ts"; | |
| 11 | import { createToken } from "../src/core/tokens.ts"; | |
| 12 | import { getUserByName } from "../src/core/users.ts"; | |
| 13 | import { serve, type Servers } from "../src/http/server.ts"; | |
| 14 | import { createJob, resetJobs, advance } from "../src/origins/jobs.ts"; | |
| 15 | import { SCAN_LIMIT } from "../src/origins/inventory.ts"; | |
| 16 | import { createForgeApp } from "../src/tui/forge.ts"; | |
| 17 | import { headlessTerminal } from "../src/tui/term/terminal.ts"; | |
| 18 | import { newFakeState, serveFakeForge, type FakeForge } from "./fake-origin.ts"; | |
| 19 | import { seedAdmin } from "./helpers/admin.ts"; | |
| 20 | import { quietLogs } from "./helpers/globals.ts"; | |
| 21 | import { settleApp } from "./helpers/tui.ts"; | |
| 22 | import { proveAll } from "./helpers/verified.ts"; | |
| 23 | ||
| 24 | quietLogs(); | |
| 25 | ||
| 26 | const PASSWORD = "alice-login-secret"; | |
| 27 | const TOKEN = "octo-edge-secret-1c3d"; | |
| 28 | const auth = `Basic ${Buffer.from(`alice:${PASSWORD}`).toString("base64")}`; | |
| 29 | const MANY = SCAN_LIMIT + 5; | |
| 30 | ||
| 31 | let dir = ""; | |
| 32 | let app: App; | |
| 33 | let servers: Servers; | |
| 34 | let fake: FakeForge; | |
| 35 | let gone: FakeForge; | |
| 36 | let originId = ""; | |
| 37 | let goneId = ""; | |
| 38 | ||
| 39 | async function connect(host: string, token: string): Promise<string> { | |
| 40 | const res = await fetch(`${servers.url}/api/v1/user/origins`, { | |
| 41 | method: "POST", | |
| 42 | headers: { authorization: auth, "content-type": "application/json" }, | |
| 43 | body: JSON.stringify({ host, token }), | |
| 44 | }); | |
| 45 | const id = String(((await res.json()) as { id: string }).id); | |
| 46 | await fetch(`${servers.url}/api/v1/user/origins/${id}/repos?refresh=1`, { headers: { authorization: auth } }); | |
| 47 | return id; | |
| 48 | } | |
| 49 | ||
| 50 | beforeAll(async () => { | |
| 51 | dir = await mkdtemp(join(tmpdir(), "forge-inventory-edges-")); | |
| 52 | const state = newFakeState("forgejo", "octo", TOKEN); | |
| 53 | for (let i = 1; i <= MANY; i++) { | |
| 54 | const n = String(i).padStart(2, "0"); | |
| 55 | state.repos.push({ id: i, fullName: `octo/r${n}`, cloneUrl: `http://127.0.0.1:1/octo/r${n}.git`, defaultBranch: "main", pushedAt: `2026-01-01T00:${n}:00Z` }); | |
| 56 | state.files.set(`octo/r${n}`, new Map([["forge.json", "{}"]])); | |
| 57 | } | |
| 58 | fake = serveFakeForge(state); | |
| 59 | app = createApp({ dataDir: dir, port: 0 }); | |
| 60 | servers = await serve(app); | |
| 61 | await seedAdmin(app, { username: "alice", email: "[email protected]", password: PASSWORD }); | |
| 62 | proveAll(app); | |
| 63 | originId = await connect(fake.url, TOKEN); | |
| 64 | const other = newFakeState("forgejo", "pat", "pat-secret-7"); | |
| 65 | other.repos.push({ id: 1, fullName: "pat/lost", cloneUrl: "http://127.0.0.1:1/pat/lost.git", defaultBranch: "main", pushedAt: "2026-01-01T00:00:00Z" }); | |
| 66 | gone = serveFakeForge(other); | |
| 67 | goneId = await connect(gone.url, "pat-secret-7"); | |
| 68 | }); | |
| 69 | ||
| 70 | afterAll(async () => { | |
| 71 | fake.stop(); | |
| 72 | gone.stop(); | |
| 73 | resetJobs(); | |
| 74 | await servers.stop(); | |
| 75 | app.close(); | |
| 76 | await rm(dir, { recursive: true, force: true }); | |
| 77 | }); | |
| 78 | ||
| 79 | const page = async (q = ""): Promise<string> => | |
| 80 | (await fetch(`${servers.url}/migrate/inventory${q}`, { headers: { authorization: auth } })).text(); | |
| 81 | ||
| 82 | const treeReads = (): number => fake.state.requests.filter((r) => r.path.endsWith("/contents") || r.path.includes("/contents?")).length; | |
| 83 | ||
| 84 | test("only SCAN_LIMIT repositories are read, newest push first; the rest say Not read yet", async () => { | |
| 85 | const body = await page(`?origin=${originId}`); | |
| 86 | expect((body.match(/Not read yet/g) ?? []).length).toBe(MANY - SCAN_LIMIT); | |
| 87 | expect(body).toContain("octo/r01"); | |
| 88 | expect(body).toContain("App (forge.json)"); | |
| 89 | }); | |
| 90 | ||
| 91 | test("a second visit is answered from what was read: the origin's trees are not asked again", async () => { | |
| 92 | const before = treeReads(); | |
| 93 | await page(`?origin=${originId}`); | |
| 94 | expect(treeReads()).toBe(before); | |
| 95 | }); | |
| 96 | ||
| 97 | test("an origin that does not answer is named, not read as an empty account", async () => { | |
| 98 | gone.stop(); | |
| 99 | app.db.query("DELETE FROM origin_repo WHERE origin_id = ?1").run(goneId); | |
| 100 | const body = await page(`?origin=${goneId}`); | |
| 101 | expect(body).toContain("Could not reach"); | |
| 102 | expect(body).not.toContain("Nothing listed."); | |
| 103 | const all = await page(); | |
| 104 | expect(all).toContain("octo/r01"); | |
| 105 | expect(all).toContain("Could not reach"); | |
| 106 | }); | |
| 107 | ||
| 108 | test("a row can go into a new forge as well as into this one", async () => { | |
| 109 | const body = await page(`?origin=${originId}`); | |
| 110 | expect(body).toContain("Bring into this forge"); | |
| 111 | expect(body).toContain("Into a new forge"); | |
| 112 | expect(body).toContain('href="/new?name=r01"'); | |
| 113 | }); | |
| 114 | ||
| 115 | test("a migration under way shows as migrating and offers no second start", async () => { | |
| 116 | const alice = getUserByName(app.db, "alice")!; | |
| 117 | const job = createJob(originId, "octo/r01", { kind: "migrate", actorId: alice.id }); | |
| 118 | advance(job.id, { stage: "fetching" }); | |
| 119 | const body = await page(`?origin=${originId}`); | |
| 120 | const at = body.indexOf('data-repo="octo/r01"'); | |
| 121 | const next = body.slice(body.lastIndexOf("<li", at), body.indexOf("</li>", at)); | |
| 122 | expect(next).toContain('data-state="migrating"'); | |
| 123 | expect(next).not.toContain("Bring into this forge"); | |
| 124 | }); | |
| 125 | ||
| 126 | test("the MCP says not_found for an origin that is not yours", async () => { | |
| 127 | const token = createToken(app.db, getUserByName(app.db, "alice")!.id, "edges", "admin").token; | |
| 128 | const res = await fetch(`${servers.url}/mcp`, { | |
| 129 | method: "POST", | |
| 130 | headers: { "content-type": "application/json", accept: "application/json, text/event-stream", authorization: `Bearer ${token}` }, | |
| 131 | body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "origins", arguments: { action: "inventory", id: "nope" } } }), | |
| 132 | }); | |
| 133 | const text = JSON.stringify(await res.json()); | |
| 134 | expect(text).toContain("not_found"); | |
| 135 | }); | |
| 136 | ||
| 137 | test("the terminal lists the same rows, and says an origin could not be reached", async () => { | |
| 138 | const terminal = headlessTerminal(140, 32); | |
| 139 | const identity = { sshUsername: "t", remoteAddress: "127.0.0.1", user: getUserByName(app.db, "alice")! }; | |
| 140 | const { app: tui } = createForgeApp(app, identity, { terminal, exitHooks: false }); | |
| 141 | tui.start(); | |
| 142 | const key = (k: string, ctrl = false) => tui.dispatch({ type: "key", key: k, ctrl, alt: false, shift: false }); | |
| 143 | key("k", true); | |
| 144 | for (const ch of "What is left to bring over") key(ch); | |
| 145 | key("enter"); | |
| 146 | await settleApp(tui); | |
| 147 | const text = tui.screen.toText(); | |
| 148 | expect(text).not.toContain("Nothing listed."); | |
| 149 | expect(text).toContain("octo/r"); | |
| 150 | expect(text).toContain("Could not reach"); | |
| 151 | expect(text).toContain("App (forge.json)"); | |
| 152 | tui.stop(); | |
| 153 | }); |
test/origin-inventory.test.ts added +168
| @@ -0,0 +1,168 @@ | ||
| 1 | /** | |
| 2 | * The inventory (#646): detection on fixtures, the five states from the | |
| 3 | * forge's own records, and secret values never read — not requested from the | |
| 4 | * origin, not in the page, not in the MCP's answer. | |
| 5 | */ | |
| 6 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 7 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 8 | import { tmpdir } from "node:os"; | |
| 9 | import { join } from "node:path"; | |
| 10 | import { createRepo } from "../src/actions/repos.ts"; | |
| 11 | import { createApp, type App } from "../src/app.ts"; | |
| 12 | import { setMirrorTarget } from "../src/core/migrate_mirror.ts"; | |
| 13 | import { createApp as makeApp } from "../src/core/apps.ts"; | |
| 14 | import { addDomain, recordCheck, setOldRecords } from "../src/core/domains.ts"; | |
| 15 | import { recordProvenance } from "../src/core/provenance.ts"; | |
| 16 | import { createToken } from "../src/core/tokens.ts"; | |
| 17 | import { serve, type Servers } from "../src/http/server.ts"; | |
| 18 | import { stateOf } from "../src/origins/inventory.ts"; | |
| 19 | import { detect, filesToRead } from "../src/origins/inventory_detect.ts"; | |
| 20 | import { newFakeState, serveFakeForge, type FakeForge } from "./fake-origin.ts"; | |
| 21 | import { seedAdmin } from "./helpers/admin.ts"; | |
| 22 | import { quietLogs } from "./helpers/globals.ts"; | |
| 23 | import { proveAll } from "./helpers/verified.ts"; | |
| 24 | ||
| 25 | quietLogs(); | |
| 26 | ||
| 27 | const PASSWORD = "alice-login-secret"; | |
| 28 | const TOKEN = "octo-origin-secret-9f2a"; | |
| 29 | const VALUE = "hunter2-the-secret-value"; | |
| 30 | const auth = `Basic ${Buffer.from(`alice:${PASSWORD}`).toString("base64")}`; | |
| 31 | ||
| 32 | test("detection: a forge.json app with Postgres, domains and secret names", () => { | |
| 33 | const got = detect({ | |
| 34 | "forge.json": JSON.stringify({ run: "bun src/main.ts", domains: ["shop.example.com", "not a host"], secrets: ["STRIPE_KEY"] }), | |
| 35 | "package.json": '{"dependencies":{"pg":"^8"}}', | |
| 36 | ".env.example": `DATABASE_URL=postgres://u:${VALUE}@db/x\nSESSION_SECRET=${VALUE}\n# comment\n`, | |
| 37 | }); | |
| 38 | expect(got.app).toBe("forge"); | |
| 39 | expect(got.postgres).toBe(true); | |
| 40 | expect(got.domains).toEqual(["shop.example.com"]); | |
| 41 | expect(got.secrets).toEqual(["STRIPE_KEY", "DATABASE_URL", "SESSION_SECRET"]); | |
| 42 | expect(JSON.stringify(got)).not.toContain(VALUE); | |
| 43 | }); | |
| 44 | ||
| 45 | test("detection: the platform's deploy file, its hosts and secrets block", () => { | |
| 46 | const got = detect({ | |
| 47 | "platform.yaml": "name: blog\ndomains:\n - blog.example.org\n - www.example.org\nsecrets:\n - name: SMTP_PASSWORD\n - API_TOKEN\nresources:\n - postgres\n", | |
| 48 | }); | |
| 49 | expect(got.app).toBe("platform"); | |
| 50 | expect(got.postgres).toBe(true); | |
| 51 | expect(got.domains).toEqual(["blog.example.org", "www.example.org"]); | |
| 52 | expect(got.secrets).toEqual(["SMTP_PASSWORD", "API_TOKEN"]); | |
| 53 | }); | |
| 54 | ||
| 55 | test("detection: no deploy file is no app, and .env is never one of the files read", () => { | |
| 56 | expect(detect({ "README.md": "hi", "package.json": "{}" })).toMatchObject({ app: null, postgres: false }); | |
| 57 | expect(detect({ ".env": `TOKEN=${VALUE}` }).secrets).toEqual([]); | |
| 58 | expect(filesToRead([".env", ".env.local", "forge.json", ".env.example"])).toEqual(["forge.json", ".env.example"]); | |
| 59 | }); | |
| 60 | ||
| 61 | let dir = ""; | |
| 62 | let app: App; | |
| 63 | let servers: Servers; | |
| 64 | let fake: FakeForge; | |
| 65 | let originId = ""; | |
| 66 | ||
| 67 | beforeAll(async () => { | |
| 68 | dir = await mkdtemp(join(tmpdir(), "forge-inventory-")); | |
| 69 | const state = newFakeState("forgejo", "octo", TOKEN); | |
| 70 | const repo = (id: number, name: string, files: Record<string, string>) => { | |
| 71 | state.repos.push({ id, fullName: `octo/${name}`, cloneUrl: `http://127.0.0.1:1/octo/${name}.git`, defaultBranch: "main", pushedAt: `2026-01-0${id}T00:00:00Z` }); | |
| 72 | state.files.set(`octo/${name}`, new Map(Object.entries(files))); | |
| 73 | }; | |
| 74 | repo(1, "shop", { | |
| 75 | "forge.json": JSON.stringify({ domains: ["shop.example.com"] }), | |
| 76 | ".env.example": `DATABASE_URL=postgres://x:${VALUE}@db/x\nSTRIPE_KEY=${VALUE}\n`, | |
| 77 | ".env": `STRIPE_KEY=${VALUE}\n`, | |
| 78 | }); | |
| 79 | repo(2, "blog", { "platform.yaml": "name: blog\nsecrets:\n - SMTP_PASSWORD\n" }); | |
| 80 | repo(3, "notes", { "README.md": "just notes" }); | |
| 81 | fake = serveFakeForge(state); | |
| 82 | app = createApp({ dataDir: dir, port: 0 }); | |
| 83 | servers = await serve(app); | |
| 84 | await seedAdmin(app, { username: "alice", email: "[email protected]", password: PASSWORD }); | |
| 85 | proveAll(app); | |
| 86 | const connected = await fetch(`${servers.url}/api/v1/user/origins`, { | |
| 87 | method: "POST", | |
| 88 | headers: { authorization: auth, "content-type": "application/json" }, | |
| 89 | body: JSON.stringify({ host: fake.url, token: TOKEN }), | |
| 90 | }); | |
| 91 | originId = String(((await connected.json()) as { id: string }).id); | |
| 92 | await fetch(`${servers.url}/api/v1/user/origins/${originId}/repos?refresh=1`, { headers: { authorization: auth } }); | |
| 93 | }); | |
| 94 | ||
| 95 | afterAll(async () => { | |
| 96 | fake.stop(); | |
| 97 | await servers.stop(); | |
| 98 | app.close(); | |
| 99 | await rm(dir, { recursive: true, force: true }); | |
| 100 | }); | |
| 101 | ||
| 102 | async function mcp(args: Record<string, unknown>): Promise<any> { | |
| 103 | const token = createToken(app.db, (app.db.query("SELECT id FROM user WHERE username = 'alice'").get() as { id: string }).id, "inventory", "admin").token; | |
| 104 | const res = await fetch(`${servers.url}/mcp`, { | |
| 105 | method: "POST", | |
| 106 | headers: { "content-type": "application/json", accept: "application/json, text/event-stream", authorization: `Bearer ${token}` }, | |
| 107 | body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "origins", arguments: args } }), | |
| 108 | }); | |
| 109 | const body = (await res.json()) as any; | |
| 110 | return JSON.parse(body.result.content[0].text); | |
| 111 | } | |
| 112 | ||
| 113 | test("the MCP inventory returns the rows: apps first, names only, and .env is never requested", async () => { | |
| 114 | const rows = await mcp({ action: "inventory", id: originId }); | |
| 115 | expect(rows.map((r: any) => r.fullName)).toEqual(["octo/blog", "octo/shop", "octo/notes"]); | |
| 116 | const shop = rows.find((r: any) => r.fullName === "octo/shop"); | |
| 117 | expect(shop).toMatchObject({ app: "forge", postgres: true, domains: ["shop.example.com"], secrets: ["DATABASE_URL", "STRIPE_KEY"], state: "not_started", scanned: true }); | |
| 118 | expect(shop.bring).toContain("/migrate?from="); | |
| 119 | expect(shop.newForge).toBe("/new?name=shop"); | |
| 120 | expect(rows.find((r: any) => r.fullName === "octo/notes").app).toBeNull(); | |
| 121 | expect(JSON.stringify(rows)).not.toContain(VALUE); | |
| 122 | expect(fake.state.requests.some((r) => /\/\.env(?!\.)/.test(r.path))).toBe(false); | |
| 123 | }); | |
| 124 | ||
| 125 | test("the page lists three apps and one primary next action, with no secret value", async () => { | |
| 126 | const res = await fetch(`${servers.url}/migrate/inventory`, { headers: { authorization: auth } }); | |
| 127 | const body = await res.text(); | |
| 128 | expect(res.status).toBe(200); | |
| 129 | for (const name of ["octo/shop", "octo/blog", "octo/notes"]) expect(body).toContain(name); | |
| 130 | expect(body).toContain("Bring octo/blog"); | |
| 131 | expect(body).toContain("STRIPE_KEY"); | |
| 132 | expect(body).not.toContain(VALUE); | |
| 133 | const out = await (await fetch(`${servers.url}/migrate/inventory`, { redirect: "manual" })).text(); | |
| 134 | expect(out).not.toContain("octo/shop"); | |
| 135 | expect((await fetch(`${servers.url}/migrate/inventory?origin=nope`, { headers: { authorization: auth } })).status).toBe(404); | |
| 136 | }); | |
| 137 | ||
| 138 | test("the migrate page links to the inventory once an account is connected", async () => { | |
| 139 | const res = await fetch(`${servers.url}/migrate`, { headers: { authorization: auth } }); | |
| 140 | expect(await res.text()).toContain('href="/migrate/inventory"'); | |
| 141 | }); | |
| 142 | ||
| 143 | test("states come from the forge's own records", async () => { | |
| 144 | const user = app.db.query("SELECT * FROM user WHERE username = 'alice'").get() as any; | |
| 145 | const { getUserById } = await import("../src/core/users.ts"); | |
| 146 | const alice = getUserById(app.db, user.id)!; | |
| 147 | const row = { originId, fullName: "octo/shop", htmlUrl: `${fake.url}/octo/shop`, mirroredAs: null }; | |
| 148 | expect(stateOf(app.db, alice, row)).toEqual({ state: "not_started", local: null }); | |
| 149 | ||
| 150 | const made = await createRepo(app, alice, { name: "shop", visibility: "private" }); | |
| 151 | if (!made.ok) throw new Error(String(made.error)); | |
| 152 | recordProvenance(app.db, made.value.id, { url: row.htmlUrl, parents: [] }); | |
| 153 | expect(stateOf(app.db, alice, row)).toEqual({ state: "in_forge", local: `/alice/shop` }); | |
| 154 | ||
| 155 | setMirrorTarget(app.db, { repoId: made.value.id, originId, fullName: row.fullName, cloneUrl: "http://x/y.git", tips: {} }); | |
| 156 | expect(stateOf(app.db, alice, row).state).toBe("in_forge"); | |
| 157 | app.db.query("UPDATE migrate_mirror SET pushed_at = ?1 WHERE repo_id = ?2").run("2026-02-01T00:00:00Z", made.value.id); | |
| 158 | expect(stateOf(app.db, alice, row).state).toBe("at_parity"); | |
| 159 | ||
| 160 | const made2 = makeApp(app.db, { repoId: made.value.id, slug: "shop", access: "private" }); | |
| 161 | expect(made2.ok).toBe(true); | |
| 162 | const appId = made2.ok ? made2.value.id : ""; | |
| 163 | const added = addDomain(app.db, { host: "shop.example.com", appId, createdBy: alice.id }, { forgeHost: "forge.test" }); | |
| 164 | expect(added.ok).toBe(true); | |
| 165 | recordCheck(app.db, "shop.example.com", { verified: true, said: "ok" }); | |
| 166 | setOldRecords(app.db, "shop.example.com", [{ type: "A", name: "shop.example.com", value: "203.0.113.1" }]); | |
| 167 | expect(stateOf(app.db, alice, row).state).toBe("cut_over"); | |
| 168 | }); |
test/palette-roster.test.ts +5 −1
| @@ -89,6 +89,7 @@ test("a site-admin person is offered the whole roster", () => { | ||
| 89 | 89 | ["bring-app", "Bring an app", "paste a link from another forge; it runs here"], |
| 90 | 90 | ["new-org", "New organization", "an organization you own, for a client's apps"], |
| 91 | 91 | ["migrate", "Migrate a repository", "bring one in from an origin or a URL, pick its branch, put a crew on it"], |
| 92 | ["inventory", "What is left to bring over", "every repository on your connected accounts, what it holds, where its move stands"], | |
| 92 | 93 | ["new-forge", "New forge", "spawn a client's forge and get its claim link"], |
| 93 | 94 | ["asked", "Asked", "your thoughts and what became of them"], |
| 94 | 95 | ["thought-notes", "Notes", "write an idea and watch it become shipped work"], |
| @@ -256,6 +257,7 @@ test("a site-admin agent is offered the roster without the person-only entries", | ||
| 256 | 257 | ["settings", "Repository settings", "owner/name"], |
| 257 | 258 | ["new-repo", "New repository", undefined], |
| 258 | 259 | ["migrate", "Migrate a repository", "bring one in from an origin or a URL, pick its branch, put a crew on it"], |
| 260 | ["inventory", "What is left to bring over", "every repository on your connected accounts, what it holds, where its move stands"], | |
| 259 | 261 | ["new-forge", "New forge", "spawn a client's forge and get its claim link"], |
| 260 | 262 | ["asked", "Asked", "your thoughts and what became of them"], |
| 261 | 263 | ["thought-notes", "Notes", "write an idea and watch it become shipped work"], |
| @@ -354,6 +356,7 @@ test("a member person is offered the signed-in roster without the admin entries" | ||
| 354 | 356 | ["bring-app", "Bring an app", "paste a link from another forge; it runs here"], |
| 355 | 357 | ["new-org", "New organization", "an organization you own, for a client's apps"], |
| 356 | 358 | ["migrate", "Migrate a repository", "bring one in from an origin or a URL, pick its branch, put a crew on it"], |
| 359 | ["inventory", "What is left to bring over", "every repository on your connected accounts, what it holds, where its move stands"], | |
| 357 | 360 | ["new-forge", "New forge", "spawn a client's forge and get its claim link"], |
| 358 | 361 | ["asked", "Asked", "your thoughts and what became of them"], |
| 359 | 362 | ["thought-notes", "Notes", "write an idea and watch it become shipped work"], |
| @@ -539,7 +542,8 @@ describe("on a pull's Files step the palette lists what the page offers this rea | ||
| 539 | 542 | for (const [tab, path] of [["Conversation", thread], ["Files", `${thread}/files`]] as const) { |
| 540 | 543 | test(`on the ${tab} tab a ${who}'s action rows are exactly the page's forms`, async () => { |
| 541 | 544 | const { rows, forms } = await read(path, kind()); |
| 542 | // The Files tab has no comment form of its own: its Comment row links to the Conversation's. | |
| 545 | // The Files tab has no comment form of its own: its Comment row links to the | |
| 546 | // Conversation's. | |
| 543 | 547 | const acts = rows.map((r) => r.t).filter((t) => !READING.includes(t) && !(tab === "Files" && t === "Comment")); |
| 544 | 548 | const same = (t: string) => (t === "Bring up to date" ? "Ship" : t); |
| 545 | 549 | expect(acts.map(same).sort()).toEqual([...new Set(forms)].sort()); |
test/preview-inventory.test.ts added +46
| @@ -0,0 +1,46 @@ | ||
| 1 | /** | |
| 2 | * The preview's inventory (#646): the owner looks include /migrate/inventory at | |
| 3 | * desk and phone, and the seeded origin draws three repositories, read, at once. | |
| 4 | */ | |
| 5 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 6 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 7 | import { tmpdir } from "node:os"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import { createApp, type App } from "../src/app.ts"; | |
| 10 | import { createSession } from "../src/core/sessions.ts"; | |
| 11 | import { forgeHttp } from "../src/http/server.ts"; | |
| 12 | import { PREVIEW_INVENTORY_PAGE, PREVIEW_OWNER_LOOKS } from "../src/spawn/preview-looks.ts"; | |
| 13 | import { previewAdmin, seedPreview } from "../src/spawn/preview-seed.ts"; | |
| 14 | ||
| 15 | let dir = ""; | |
| 16 | let app: App; | |
| 17 | ||
| 18 | beforeAll(async () => { | |
| 19 | dir = await mkdtemp(join(tmpdir(), "forge-preview-inventory-")); | |
| 20 | app = createApp({ dataDir: dir, port: 0 }); | |
| 21 | const seeded = await seedPreview(app, { FORGE_APP_ENV: "preview" }); | |
| 22 | if (!seeded.ok) throw new Error(seeded.error); | |
| 23 | }, 60_000); | |
| 24 | ||
| 25 | afterAll(async () => { | |
| 26 | app.close(); | |
| 27 | await rm(dir, { recursive: true, force: true }); | |
| 28 | }); | |
| 29 | ||
| 30 | test("the owner looks take the inventory at both widths", () => { | |
| 31 | for (const kind of ["desktop", "phone"] as const) | |
| 32 | expect(PREVIEW_OWNER_LOOKS).toContainEqual({ page: PREVIEW_INVENTORY_PAGE, kind }); | |
| 33 | }); | |
| 34 | ||
| 35 | test("the seeded inventory draws three repositories, read, and offers both ways in", async () => { | |
| 36 | const http = forgeHttp(app); | |
| 37 | const session = createSession(app.db, previewAdmin(app)!.id); | |
| 38 | const res = await http.handle(new Request(`http://localhost${PREVIEW_INVENTORY_PAGE}`, { headers: { cookie: `forge_session=${session.token}` } })); | |
| 39 | const body = await res.text(); | |
| 40 | http.close(); | |
| 41 | expect(res.status).toBe(200); | |
| 42 | for (const name of ["preview-old/shop", "preview-old/blog", "preview-old/notes"]) expect(body).toContain(name); | |
| 43 | expect(body).toContain("STRIPE_KEY"); | |
| 44 | expect(body).not.toContain("Not read yet"); | |
| 45 | expect(body).toContain("Into a new forge"); | |
| 46 | }); |
new above
offline
Principle 8: plain words. 'Not read yet — the list is long; open this page again' explains itself. Say 'Not read yet'.