Public: anyone can read this; the crew can change it.
Forgify a business: origins mirror its repos in, extract pulls them apart, the crew writes the sync and pull adapters, and the result ships as its own forge #10032
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/390-forgify-a-business-origins-mirror-its-re → main 11 commits fork 9 threads · 9 unresolved
5 files
changed +202
−18
docs/ADAPTERS.md +16 −10
| @@ -10,8 +10,9 @@ answer and records each pull as a run. | ||
| 10 | 10 | |
| 11 | 11 | ## Status |
| 12 | 12 | |
| 13 | This is the contract, not a description of running code. Today the forge does | |
| 14 | not read the `adapter` block and does not call `POST /pull`. The answer's | |
| 13 | This is the contract, not a description of running code. Today the forge | |
| 14 | reads the `adapter` block (`src/apps/adapter.ts`) and does not call | |
| 15 | `POST /pull`. The answer's | |
| 15 | 16 | checks and the pull run record exist (`src/apps/pull.ts`: `parsePullAnswer`, |
| 16 | 17 | `recordPull`, kept in the `pull_run` table), but nothing calls them yet. An |
| 17 | 18 | adapter written to this page is an ordinary app until that work lands. The rest of the forge's app rules apply to it now: see |
| @@ -34,10 +35,9 @@ Anyone else gets 401 or 403, and the source is not read, so a public app or | ||
| 34 | 35 | a preview cannot be made to pull by whoever reaches it. When the forge |
| 35 | 36 | schedules pulls, its call has to carry that signed identity. |
| 36 | 37 | |
| 37 | The manifest parser (`src/apps/adapter.ts`) reads only `adapter.schedule` | |
| 38 | today. So the template names its source hosts in `src/source.ts` and opens | |
| 39 | them with `egress`, and its `adapter` block holds only `schedule` until the | |
| 40 | parser accepts `source`. | |
| 38 | The template names its source hosts in `src/source.ts` and opens them with | |
| 39 | `egress`. Its `adapter` block may also name them under `source`, which the | |
| 40 | parser reads; each must then be in `egress` as well. | |
| 41 | 41 | |
| 42 | 42 | ## The `adapter` block in forge.json |
| 43 | 43 | |
| @@ -56,11 +56,17 @@ An app declares itself an adapter with an optional top-level `adapter` block: | ||
| 56 | 56 | |
| 57 | 57 | - `schedule.every` is the interval between pulls, in seconds. It is an |
| 58 | 58 | integer from 60 to 86400, which is one minute to one day. |
| 59 | - `source.name` is the external system's name, as people read it. | |
| 60 | - `source.hosts` lists the hosts the adapter reads from. | |
| 59 | - `source` is optional: `source.name` is the external system's name, as | |
| 60 | people read it, and `source.hosts` lists the 1 to 8 hosts the adapter reads | |
| 61 | from, each a bare lowercase host name. | |
| 62 | - `config` is optional: the settings the adapter reads, each named by the | |
| 63 | environment variable it comes from, with a `type` of `string`, `number`, | |
| 64 | `boolean` or `secret`, and `required` or a `default`. A `secret` names a | |
| 65 | variable and never holds a value. A field may not name a variable the | |
| 66 | forge sets itself. | |
| 61 | 67 | |
| 62 | A block that breaks any of these rules is invalid, and the forge does not | |
| 63 | schedule the app. | |
| 68 | Any other key, at any level, is refused. A block that breaks any of these | |
| 69 | rules is invalid, and the forge does not schedule the app. | |
| 64 | 70 | |
| 65 | 71 | ## Source hosts and egress |
| 66 | 72 |
docs/APPS.md +4 −2
| @@ -546,9 +546,11 @@ Only `text/html` responses the app sends uncompressed with a `Content-Length` | ||
| 546 | 546 | |
| 547 | 547 | `adapter`, at the top level, says the app is an adapter the forge pulls |
| 548 | 548 | from on a schedule (#437): `{ "adapter": { "schedule": { "every": 900 } } }`. |
| 549 | `every` is whole seconds, 60 to 86400; any other key, at either level, fails | |
| 549 | `every` is whole seconds, 60 to 86400. `source` (the system's name and the | |
| 550 | hosts it reads, each also in `egress`) and `config` (the settings it reads, | |
| 551 | by environment variable) are optional; any other key, at any level, fails | |
| 550 | 552 | the manifest with the field named (`src/apps/adapter.ts`). Absent, the app is |
| 551 | not an adapter. | |
| 553 | not an adapter. The contract is [](ADAPTERS.md). | |
| 552 | 554 | |
| 553 | 555 | Two more fields say what the app's code may use (rule 9): |
| 554 | 556 |
src/apps/adapter.ts +113 −5
| @@ -1,22 +1,50 @@ | ||
| 1 | 1 | /** |
| 2 | 2 | * `adapter` (top level of `forge.json`): the app is an adapter, an app the |
| 3 | * forge pulls from on a schedule (#437). The block says how often: | |
| 3 | * forge pulls from on a schedule (#437, docs/ADAPTERS.md). The block says how | |
| 4 | * often, and may say what it pulls from and which settings it reads: | |
| 4 | 5 | * |
| 5 | * { "adapter": { "schedule": { "every": 900 } } } | |
| 6 | * { "adapter": { | |
| 7 | * "schedule": { "every": 900 }, | |
| 8 | * "source": { "name": "Shop API", "hosts": ["api.shop.example"] }, | |
| 9 | * "config": { "token": { "type": "secret", "env": "SHOP_TOKEN", "required": true } } } } | |
| 6 | 10 | * |
| 7 | 11 | * `every` is whole seconds, 60 (one minute) to 86400 (one day). Pure: plain |
| 8 | 12 | * data in, plain data or one sentence naming the field out. Unknown keys are |
| 9 | 13 | * refused at every level, so a misspelt `scheduel` fails the manifest instead |
| 10 | 14 | * of silently meaning "no schedule". |
| 15 | * | |
| 16 | * Config carries names, never values: a secret field points at an operator | |
| 17 | * variable (`core/app_env.ts`), so whoever reads the code reads no credential. | |
| 11 | 18 | */ |
| 12 | 19 | |
| 20 | import { isReservedAppEnvName } from "../core/app_env.ts"; | |
| 13 | 21 | import { err, ok, type Result } from "../kernel/result.ts"; |
| 14 | 22 | |
| 23 | export type AdapterFieldType = "string" | "number" | "boolean" | "secret"; | |
| 24 | ||
| 25 | export interface AdapterField { | |
| 26 | readonly name: string; | |
| 27 | readonly type: AdapterFieldType; | |
| 28 | /** The environment variable the adapter reads it from. */ | |
| 29 | readonly env: string; | |
| 30 | readonly required: boolean; | |
| 31 | /** Absent on a secret, and wherever none is declared. */ | |
| 32 | readonly default?: string | number | boolean | undefined; | |
| 33 | } | |
| 34 | ||
| 35 | export interface AdapterSource { | |
| 36 | readonly name: string; | |
| 37 | /** The hosts it reads from; each must also be in `egress` (`manifest.ts`). */ | |
| 38 | readonly hosts: readonly string[]; | |
| 39 | } | |
| 40 | ||
| 15 | 41 | export interface AdapterSpec { |
| 16 | 42 | readonly schedule: { |
| 17 | 43 | /** Seconds between pulls, an integer in [EVERY_MIN, EVERY_MAX]. */ |
| 18 | 44 | readonly every: number; |
| 19 | 45 | }; |
| 46 | readonly source?: AdapterSource | undefined; | |
| 47 | readonly config?: readonly AdapterField[] | undefined; | |
| 20 | 48 | } |
| 21 | 49 | |
| 22 | 50 | /** One minute: more often than this is a poll loop, not a schedule. */ |
| @@ -24,17 +52,31 @@ export const EVERY_MIN = 60; | ||
| 24 | 52 | /** One day. */ |
| 25 | 53 | export const EVERY_MAX = 86_400; |
| 26 | 54 | |
| 55 | const HOSTS_MAX = 8; | |
| 56 | const FIELDS_MAX = 32; | |
| 57 | const HOST = /^(?=.{1,253}$)([a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/; | |
| 58 | const FIELD_NAME = /^[a-z][a-z0-9_]{0,39}$/; | |
| 59 | const ENV_NAME = /^[A-Z][A-Z0-9_]{0,63}$/; | |
| 60 | const TYPES: readonly string[] = ["string", "number", "boolean", "secret"]; | |
| 61 | ||
| 27 | 62 | type Json = Record<string, unknown>; |
| 28 | 63 | |
| 29 | 64 | const isObject = (value: unknown): value is Json => |
| 30 | 65 | typeof value === "object" && value !== null && !Array.isArray(value); |
| 31 | 66 | |
| 67 | const bad = (message: string): Result<never, string> => | |
| 68 | err(`forge.json: adapter.${message}`); | |
| 69 | ||
| 32 | 70 | /** The keys `allowed` does not name, quoted, for the error sentence. */ |
| 33 | 71 | function unknownKeys(value: Json, allowed: readonly string[]): string | undefined { |
| 34 | 72 | const stray = Object.keys(value).filter((k) => !allowed.includes(k)); |
| 35 | 73 | return stray.length === 0 ? undefined : stray.map((k) => JSON.stringify(k)).join(", "); |
| 36 | 74 | } |
| 37 | 75 | |
| 76 | /** A bare lowercase hostname: no scheme, port, path, wildcard or address. */ | |
| 77 | const isSourceHost = (value: string): boolean => | |
| 78 | HOST.test(value) && !/^[0-9.]+$/.test(value); | |
| 79 | ||
| 38 | 80 | /** |
| 39 | 81 | * The parsed `adapter` block, `undefined` when it is absent. `raw` is the |
| 40 | 82 | * value of the top-level `adapter` key exactly as `JSON.parse` gave it. |
| @@ -42,9 +84,9 @@ function unknownKeys(value: Json, allowed: readonly string[]): string | undefine | ||
| 42 | 84 | export function parseAdapter(raw: unknown): Result<AdapterSpec | undefined, string> { |
| 43 | 85 | if (raw === undefined) return ok(undefined); |
| 44 | 86 | if (!isObject(raw)) return err("forge.json: adapter must be an object"); |
| 45 | const strayTop = unknownKeys(raw, ["schedule"]); | |
| 87 | const strayTop = unknownKeys(raw, ["schedule", "source", "config"]); | |
| 46 | 88 | if (strayTop !== undefined) |
| 47 | return err(`forge.json: adapter takes only schedule, not ${strayTop}`); | |
| 89 | return err(`forge.json: adapter takes only schedule, source and config, not ${strayTop}`); | |
| 48 | 90 | const schedule = raw["schedule"]; |
| 49 | 91 | if (schedule === undefined) return err("forge.json: adapter.schedule is required"); |
| 50 | 92 | if (!isObject(schedule)) return err("forge.json: adapter.schedule must be an object"); |
| @@ -61,5 +103,71 @@ export function parseAdapter(raw: unknown): Result<AdapterSpec | undefined, stri | ||
| 61 | 103 | return err( |
| 62 | 104 | `forge.json: adapter.schedule.every must be a whole number of seconds, ${EVERY_MIN}–${EVERY_MAX}`, |
| 63 | 105 | ); |
| 64 | return ok({ schedule: { every } }); | |
| 106 | const source = parseSource(raw["source"]); | |
| 107 | if (!source.ok) return source; | |
| 108 | const config = parseConfig(raw["config"]); | |
| 109 | if (!config.ok) return config; | |
| 110 | return ok({ | |
| 111 | schedule: { every }, | |
| 112 | ...(source.value === undefined ? {} : { source: source.value }), | |
| 113 | ...(config.value === undefined ? {} : { config: config.value }), | |
| 114 | }); | |
| 115 | } | |
| 116 | ||
| 117 | function parseSource(raw: unknown): Result<AdapterSource | undefined, string> { | |
| 118 | if (raw === undefined) return ok(undefined); | |
| 119 | if (!isObject(raw)) return bad("source must be an object"); | |
| 120 | const stray = unknownKeys(raw, ["name", "hosts"]); | |
| 121 | if (stray !== undefined) return bad(`source takes only name and hosts, not ${stray}`); | |
| 122 | const name = raw["name"]; | |
| 123 | if (typeof name !== "string" || !/^[\w .-]{1,60}$/.test(name)) | |
| 124 | return bad("source.name must be 1–60 letters, digits, spaces, dots or dashes"); | |
| 125 | const hosts = raw["hosts"]; | |
| 126 | if (!Array.isArray(hosts) || hosts.length === 0 || hosts.length > HOSTS_MAX) | |
| 127 | return bad(`source.hosts must list 1–${HOSTS_MAX} hostnames`); | |
| 128 | for (const host of hosts) | |
| 129 | if (typeof host !== "string" || !isSourceHost(host)) | |
| 130 | return bad("source.hosts entries must be bare lowercase hostnames (no scheme, port or path)"); | |
| 131 | return ok({ name, hosts: hosts as string[] }); | |
| 132 | } | |
| 133 | ||
| 134 | function parseConfig(raw: unknown): Result<AdapterField[] | undefined, string> { | |
| 135 | if (raw === undefined) return ok(undefined); | |
| 136 | if (!isObject(raw)) return bad("config must be an object"); | |
| 137 | const names = Object.keys(raw); | |
| 138 | if (names.length > FIELDS_MAX) return bad(`config may name at most ${FIELDS_MAX} fields`); | |
| 139 | const fields: AdapterField[] = []; | |
| 140 | for (const fieldName of names) { | |
| 141 | const f = raw[fieldName]; | |
| 142 | if (!FIELD_NAME.test(fieldName)) | |
| 143 | return bad(`config.${fieldName}: names are lowercase letters, digits and _`); | |
| 144 | if (!isObject(f)) return bad(`config.${fieldName} must be an object`); | |
| 145 | const stray = unknownKeys(f, ["type", "env", "required", "default"]); | |
| 146 | if (stray !== undefined) | |
| 147 | return bad(`config.${fieldName} takes only type, env, required and default, not ${stray}`); | |
| 148 | const type = f["type"]; | |
| 149 | if (typeof type !== "string" || !TYPES.includes(type)) | |
| 150 | return bad(`config.${fieldName}.type must be one of ${TYPES.join(", ")}`); | |
| 151 | const env = f["env"]; | |
| 152 | if (typeof env !== "string" || !ENV_NAME.test(env) || isReservedAppEnvName(env)) | |
| 153 | return bad(`config.${fieldName}.env must name an operator variable (A–Z, 0–9, _) the forge does not own`); | |
| 154 | const def = f["default"]; | |
| 155 | if (type === "secret") { | |
| 156 | if (def !== undefined) | |
| 157 | return bad(`config.${fieldName} is a secret: it names a variable, it never holds a value`); | |
| 158 | } else if (def !== undefined && typeof def !== type) | |
| 159 | return bad(`config.${fieldName}.default must be a ${type}`); | |
| 160 | if (typeof def === "string" && def.length > 200) | |
| 161 | return bad(`config.${fieldName}.default is too long`); | |
| 162 | if (f["required"] !== undefined && typeof f["required"] !== "boolean") | |
| 163 | return bad(`config.${fieldName}.required must be true or false`); | |
| 164 | fields.push({ | |
| 165 | name: fieldName, | |
| 166 | type: type as AdapterFieldType, | |
| 167 | env, | |
| 168 | required: f["required"] === true, | |
| 169 | ...(def === undefined ? {} : { default: def as string | number | boolean }), | |
| 170 | }); | |
| 171 | } | |
| 172 | return ok(fields); | |
| 65 | 173 | } |
src/apps/manifest.ts +7
| @@ -176,6 +176,13 @@ export function parseManifest( | ||
| 176 | 176 | return invalid("forge.json: toolbar must be true or false"); |
| 177 | 177 | const adapter = parseAdapter(parsed.value["adapter"]); |
| 178 | 178 | if (!adapter.ok) return invalid(adapter.error); |
| 179 | // Only `egress` opens a host (docs/ADAPTERS.md): a source host it does | |
| 180 | // not list would be refused by the proxy, so the block is invalid. | |
| 181 | const unopened = adapter.value?.source?.hosts.find( | |
| 182 | (h) => !(egress.value ?? []).some((e) => e.toLowerCase() === h), | |
| 183 | ); | |
| 184 | if (unopened !== undefined) | |
| 185 | return invalid(`forge.json: adapter.source.hosts names ${unopened}, which egress does not list`); | |
| 179 | 186 | const kind = app["kind"]; |
| 180 | 187 | if (kind !== undefined && kind !== "web" && kind !== "service") |
| 181 | 188 | return invalid('forge.json: app.kind must be "web" or "service"'); |
test/apps-adapter.test.ts +62 −1
| @@ -47,7 +47,7 @@ describe("parseAdapter", () => { | ||
| 47 | 47 | |
| 48 | 48 | test("unknown keys are refused at each level", () => { |
| 49 | 49 | expect(refused({ schedule: { every: 900 }, url: "/pull" })).toBe( |
| 50 | 'forge.json: adapter takes only schedule, not "url"', | |
| 50 | 'forge.json: adapter takes only schedule, source and config, not "url"', | |
| 51 | 51 | ); |
| 52 | 52 | expect(refused({ schedule: { every: 900, jitter: 5 } })).toBe( |
| 53 | 53 | 'forge.json: adapter.schedule takes only every, not "jitter"', |
| @@ -86,3 +86,64 @@ describe("the manifest carries the adapter", () => { | ||
| 86 | 86 | }); |
| 87 | 87 | }); |
| 88 | 88 | }); |
| 89 | ||
| 90 | const good = { | |
| 91 | source: { name: "Shop API", hosts: ["api.shop.example"] }, | |
| 92 | schedule: { every: 900 }, | |
| 93 | config: { | |
| 94 | token: { type: "secret", env: "SHOP_TOKEN", required: true }, | |
| 95 | page: { type: "number", env: "SHOP_PAGE", default: 50 }, | |
| 96 | }, | |
| 97 | }; | |
| 98 | ||
| 99 | describe("source and config (#437)", () => { | |
| 100 | test("a block with source and config parses to plain data", () => { | |
| 101 | const r = parseAdapter(good); | |
| 102 | expect(r.ok && r.value?.source).toEqual({ name: "Shop API", hosts: ["api.shop.example"] }); | |
| 103 | expect(r.ok && r.value?.config?.[0]?.required).toBe(true); | |
| 104 | expect(r.ok && r.value?.config?.[1]?.default).toBe(50); | |
| 105 | }); | |
| 106 | ||
| 107 | test("source and config are optional beside schedule, and their stray keys are refused", () => { | |
| 108 | expect(parseAdapter({ schedule: { every: 900 } })).toEqual({ ok: true, value: { schedule: { every: 900 } } }); | |
| 109 | expect(parseAdapter({ ...good, source: { ...good.source, url: "x" } }).ok).toBe(false); | |
| 110 | expect(parseAdapter({ ...good, config: { t: { type: "string", env: "T", value: "v" } } }).ok).toBe(false); | |
| 111 | }); | |
| 112 | ||
| 113 | test("hosts are bare hostnames, never URLs, ports or addresses", () => { | |
| 114 | for (const h of ["https://a.example", "a.example:8080", "a.example/x", "127.0.0.1", "*.example.com", "localhost", "A.Example"]) | |
| 115 | expect(parseAdapter({ ...good, source: { name: "x", hosts: [h] } }).ok).toBe(false); | |
| 116 | expect(parseAdapter({ ...good, source: { name: "x", hosts: [] } }).ok).toBe(false); | |
| 117 | }); | |
| 118 | ||
| 119 | test("a secret names a variable and never holds a value", () => { | |
| 120 | const cfg = { token: { type: "secret", env: "T", default: "hunter2" } }; | |
| 121 | expect(parseAdapter({ ...good, config: cfg }).ok).toBe(false); | |
| 122 | }); | |
| 123 | ||
| 124 | test("a config field may not point at a variable the forge owns", () => { | |
| 125 | const cfg = { k: { type: "string", env: "FORGE_TOKEN" } }; | |
| 126 | expect(parseAdapter({ ...good, config: cfg }).ok).toBe(false); | |
| 127 | }); | |
| 128 | ||
| 129 | test("a default must match its type", () => { | |
| 130 | const cfg = { n: { type: "number", env: "N", default: "ten" } }; | |
| 131 | expect(parseAdapter({ ...good, config: cfg }).ok).toBe(false); | |
| 132 | }); | |
| 133 | ||
| 134 | test("the manifest carries it, and an invalid block fails the manifest", () => { | |
| 135 | const m = parseManifest(JSON.stringify({ app: { run: "bun x.ts" }, egress: ["api.shop.example"], adapter: good }), undefined); | |
| 136 | expect(m.ok && m.value.adapter?.source?.name).toBe("Shop API"); | |
| 137 | const plain = parseManifest(JSON.stringify({ app: { run: "bun x.ts" } }), undefined); | |
| 138 | expect(plain.ok && plain.value.adapter).toBeUndefined(); | |
| 139 | expect(parseManifest(JSON.stringify({ app: { run: "bun x.ts" }, adapter: { source: 1 } }), undefined).ok).toBe(false); | |
| 140 | }); | |
| 141 | ||
| 142 | test("a source host egress does not list makes the block invalid: only egress opens a host", () => { | |
| 143 | const m = parseManifest(JSON.stringify({ app: { run: "bun x.ts" }, egress: ["cdn.example"], adapter: good }), undefined); | |
| 144 | expect(m.ok).toBe(false); | |
| 145 | expect(!m.ok && m.error.kind === "invalid" && m.error.message).toContain("api.shop.example"); | |
| 146 | const opened = parseManifest(JSON.stringify({ app: { run: "bun x.ts" }, egress: ["cdn.example", "API.shop.example"], adapter: good }), undefined); | |
| 147 | expect(opened.ok && opened.value.egress).toEqual(["cdn.example", "api.shop.example"]); | |
| 148 | }); | |
| 149 | }); |
new above
offline
Dead in the forge: nothing outside test/adapter.test.ts calls this, and it duplicates examples/adapter-template/src/guard.ts. FORGE_ADAPTER_HOSTS is only advisory (the adapter must opt in to guardedFetch); docs/ADAPTERS.md says hosts outside the list are refused, which is not true of the forge. Either enforce it through the egress proxy (src/apps/egress.ts) or reword the docs to say it is a declaration the template honours.