trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns10g rEventsFlows

An example adapter app answers an idempotent POST /pull and refuses hosts outside its source #10059

Merged masoncrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/10049-an-example-adapter-app-answers-an-idempo → main 3 commits fork 2 threads · 2 unresolved

  • Merged. mason/10049-an-example-adapter-app-answers-an-idempo was merged into main as 4783e9c.

Review

Sign in to review this change.
11 files changed +702 −0
docs/ADAPTERS.md +22
@@ -16,6 +16,28 @@ pull runs. An adapter written to this page is an ordinary app until that work
1616 lands. The rest of the forge's app rules apply to it now: see
1717 [`docs/APPS.md`](APPS.md).
1818
19 ## The template
20
21 [`examples/adapter-template/`](../examples/adapter-template/) is a working
22 adapter to copy. It has no dependencies, answers `POST /pull`, keeps its
23 records and cursor in bun:sqlite under `FORGE_APP_DATA`, and refuses any host
24 outside its source. Its README says what to change. Its tests prove that a
25 replay writes nothing new and that another host is refused. The forge's check
26 runs them through `test/adapter-template.test.ts`, which also parses the
27 template's `forge.json` and checks that each source host is in `egress`.
28
29 The template accepts `POST /pull` only from an `admin` the forge signed for:
30 `x-forge-identity` and `x-forge-signature`, checked against
31 `FORGE_APP_IDENTITY_KEY` (see "Who is there" in [`docs/APPS.md`](APPS.md)).
32 Anyone else gets 401 or 403, and the source is not read, so a public app or
33 a preview cannot be made to pull by whoever reaches it. When the forge
34 schedules pulls, its call has to carry that signed identity.
35
36 The manifest parser (`src/apps/adapter.ts`) reads only `adapter.schedule`
37 today. So the template names its source hosts in `src/source.ts` and opens
38 them with `egress`, and its `adapter` block holds only `schedule` until the
39 parser accepts `source`.
40
1941 ## The `adapter` block in forge.json
2042
2143 An app declares itself an adapter with an optional top-level `adapter` block:
examples/adapter-template/README.md added +65
@@ -0,0 +1,65 @@
1 # adapter-template
2
3 A working adapter to copy: it pulls records from one external system into its
4 own bun:sqlite file and answers the forge's `POST /pull` with
5 `{pulled, skipped, cursor, errors}`. The contract it follows is
6 [`docs/ADAPTERS.md`](../../docs/ADAPTERS.md). Zero dependencies.
7
8 ## Files
9
10 | file | what it holds |
11 | --------------- | -------------------------------------------------------------------- |
12 | `forge.json` | how the forge runs it: `egress` opens the source host, `adapter.schedule` sets the interval |
13 | `src/source.ts` | the source: its name and hosts, the only `fetch` the adapter uses, and how a page and a record are read |
14 | `src/store.ts` | records keyed by the source's id, and the cursor, under `FORGE_APP_DATA/adapter.sqlite` |
15 | `src/pull.ts` | one pull: read the page after the cursor, write what is new, move the cursor, count |
16 | `src/server.ts` | `POST /pull` (admins only), `GET /` (what is held, to admins), `GET /healthz` |
17 | `src/who.ts` | who is asking, from the forge's signed `x-forge-identity` headers |
18 | `test/` | a replay writes nothing new; a host outside the source is refused; only a signed admin may pull |
19
20 ## Make it yours
21
22 1. In `src/source.ts`, set `SOURCE.name` and `SOURCE.hosts`, the URL in
23 `readPage`, and `parseRecord` for your system's records. Keep each
24 record's key the source's own id: that is what makes a replay a no-op.
25 2. In `forge.json`, put the same hosts in `egress`, and pick
26 `adapter.schedule.every` (60 to 86400 seconds).
27 3. `bun test`.
28
29 ## Rules it keeps
30
31 - **Idempotent.** A record is written under its source id and only when it is
32 new or changed; records and the cursor land in one transaction. A pull from
33 the same cursor counts what it already holds as `skipped`.
34 - **Only its source.** `sourceFetch` refuses any host not in `SOURCE.hosts`,
35 any scheme but `https`, and does not follow redirects. `egress` is the
36 forge's lock on the same door; this is the adapter's own.
37 - **Only an admin pulls.** A pull reaches out to the source and writes, so
38 `POST /pull` first checks the forge's signed `x-forge-identity` and
39 `x-forge-signature` against `FORGE_APP_IDENTITY_KEY` (`src/who.ts`: HMAC,
40 constant-time compare, expiry in the future). No headers, a bad signature or
41 an expired one answers 401; a signed `user` answers 403. Neither reads the
42 source. Without the key nobody may pull.
43 - **A failed pull moves nothing.** The answer is a fixed 502 `pull failed`
44 and the cursor stays; the detail, which can name the source's host or URL,
45 goes to the log only.
46 - **The page tells visitors only the adapter's name.** `GET /` shows the
47 cursor, record count and last pull to admins only, since a cursor can be a
48 source's page token.
49 - **One pull at a time.** A second `POST /pull` waits for the first.
50 - A browser `POST /pull` from another origin is refused; every response sets
51 `Content-Security-Policy`, `X-Content-Type-Options` and `Referrer-Policy`.
52
53 ## Try it
54
55 ```sh
56 key=$(openssl rand -hex 16)
57 FORGE_APP_DATA=$(mktemp -d) FORGE_APP_IDENTITY_KEY=$key PORT=3000 bun run src/server.ts &
58 id="me|me|admin|$(( $(date +%s) * 1000 + 600000 ))"
59 sig=$(printf %s "$id" | openssl dgst -sha256 -hmac "$key" -binary | base64 | tr '+/' '-_' | tr -d '=')
60 curl -X POST -H "x-forge-identity: $id" -H "x-forge-signature: $sig" localhost:3000/pull
61 ```
62
63 Run under the forge, the forge signs those headers itself. `api.example.com`
64 is a placeholder, so the pull answers 502 until `src/source.ts` names a real
65 source.
examples/adapter-template/forge.json added +12
@@ -0,0 +1,12 @@
1 {
2 "egress": ["api.example.com"],
3 "app": {
4 "run": "bun run src/server.ts",
5 "install": "",
6 "health": "/healthz",
7 "test": "bun test"
8 },
9 "adapter": {
10 "schedule": { "every": 3600 }
11 }
12 }
examples/adapter-template/package.json added +6
@@ -0,0 +1,6 @@
1 {
2 "name": "adapter-template",
3 "private": true,
4 "type": "module",
5 "scripts": { "start": "bun run src/server.ts", "test": "bun test" }
6 }
examples/adapter-template/src/pull.ts added +45
@@ -0,0 +1,45 @@
1 /**
2 * One pull: read the page after the stored cursor, write what is new, move
3 * the cursor, and answer the four counts the forge records
4 * (docs/ADAPTERS.md).
5 *
6 * Records and the cursor are written in one transaction, so the cursor
7 * never runs ahead of the records it covers. A pull from the same cursor
8 * reads the same page, finds every record held, and counts them `skipped`.
9 */
10
11 import { isCursor, parseRecord, type Page } from "./source.ts";
12 import type { Store } from "./store.ts";
13
14 export interface PullResult {
15 readonly pulled: number;
16 readonly skipped: number;
17 readonly cursor: string;
18 readonly errors: number;
19 }
20
21 /** The upper bound the forge accepts for each count. */
22 const COUNT_MAX = 1_000_000;
23
24 export async function pull(
25 store: Store,
26 read: (cursor: string) => Promise<Page>,
27 ): Promise<PullResult> {
28 const from = store.cursor();
29 const page = await read(from);
30 const records = page.records.slice(0, COUNT_MAX);
31 const cursor = isCursor(page.cursor) ? page.cursor : from;
32 return store.atomically(() => {
33 let pulled = 0;
34 let skipped = 0;
35 let errors = 0;
36 for (const raw of records) {
37 const record = parseRecord(raw);
38 if (record === undefined) errors++;
39 else if (store.put(record)) pulled++;
40 else skipped++;
41 }
42 store.setCursor(cursor);
43 return { pulled, skipped, cursor, errors };
44 });
45 }
examples/adapter-template/src/server.ts added +126
@@ -0,0 +1,126 @@
1 /**
2 * The adapter's HTTP face:
3 *
4 * POST /pull one pull, for an admin only; answers {pulled, skipped, cursor, errors}
5 * GET / the adapter's name; to an admin, also its cursor and counts
6 * GET /healthz
7 *
8 * Who is asking comes from the forge's signed headers (`who.ts`): a pull
9 * reaches out to the source and writes, so only an `admin` (one who may
10 * write the app's repository) may start one. Anyone else is refused before
11 * the source is read.
12 *
13 * The forge hands the process `PORT`, `FORGE_APP_DATA` and
14 * `FORGE_APP_IDENTITY_KEY` (docs/APPS.md). Without the key nobody is an
15 * admin, so every pull is refused. Zero dependencies; `bun run src/server.ts`.
16 */
17
18 import { pull, type PullResult } from "./pull.ts";
19 import { readPage, SOURCE, type Page } from "./source.ts";
20 import { openStore, type Store } from "./store.ts";
21 import { who } from "./who.ts";
22
23 const HEADERS = {
24 "content-security-policy": "default-src 'none'; frame-ancestors 'none'",
25 "x-content-type-options": "nosniff",
26 "referrer-policy": "no-referrer",
27 } as const;
28
29 const escape = (text: string): string =>
30 text.replace(
31 /[&<>"']/g,
32 (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[c]!,
33 );
34
35 const json = (body: unknown, status = 200): Response =>
36 new Response(JSON.stringify(body), {
37 status,
38 headers: { ...HEADERS, "content-type": "application/json" },
39 });
40
41 /**
42 * A mutation from a browser must come from this origin. The forge's own
43 * call carries no `Origin`; a cross-site form or fetch does, and is refused.
44 */
45 function sameOrigin(req: Request): boolean {
46 if (req.headers.get("sec-fetch-site") === "cross-site") return false;
47 const origin = req.headers.get("origin");
48 return origin === null || origin === new URL(req.url).origin;
49 }
50
51 /** The page. The cursor and counts are an admin's to see, like the pull. */
52 function page(store: Store, last: PullResult | undefined, admin: boolean): string {
53 const lastLine =
54 last === undefined
55 ? "No pull since this process started."
56 : `Last pull: ${last.pulled} pulled, ${last.skipped} skipped, ${last.errors} errors.`;
57 const detail = admin
58 ? `<p>${store.count()} records held. Cursor: <code>${escape(store.cursor() || "(none yet)")}</code></p>
59 <p>${escape(lastLine)}</p>`
60 : "<p>Pulls and their cursor are shown to the app's admins.</p>";
61 return `<!doctype html><html lang="en"><head><meta charset="utf-8">
62 <meta name="viewport" content="width=device-width, initial-scale=1">
63 <title>${escape(SOURCE.name)} adapter</title></head><body><main>
64 <h1>${escape(SOURCE.name)} adapter</h1>
65 ${detail}
66 </main></body></html>`;
67 }
68
69 export interface HandlerOptions {
70 /** Reads the page after a cursor; the real source by default. */
71 readonly read?: (cursor: string) => Promise<Page>;
72 /** `FORGE_APP_IDENTITY_KEY`; empty means nobody is an admin. */
73 readonly key: string;
junipercrew outdated

No identity check before the pull. Verify the forge's signed headers (admin) here, before queueing, and refuse with 401/403. Add the tests listed in the review body.
74 /** The clock signed identities are checked against. */
75 readonly now?: () => number;
76 }
77
78 /** The request handler, over a store and the options above. */
79 export function makeHandler(
80 store: Store,
81 { read = (cursor) => readPage(cursor), key, now = Date.now }: HandlerOptions,
82 ): (req: Request) => Promise<Response> {
junipercrew outdated

error.message can leak the source host or URL to the caller. Answer a fixed message.
83 let last: PullResult | undefined;
84 // One pull at a time: a second request waits for the first, then starts
85 // from the cursor the first left.
86 let queue: Promise<unknown> = Promise.resolve();
87 return async (req) => {
88 const { pathname } = new URL(req.url);
89 const caller = who(req, key, now());
90 if (pathname === "/pull") {
91 if (req.method !== "POST") return json({ error: "POST only" }, 405);
92 if (!sameOrigin(req)) return json({ error: "cross-origin request refused" }, 403);
93 if (caller.role === "anon") return json({ error: "sign in as an admin of this app" }, 401);
94 if (caller.role !== "admin") return json({ error: "only an admin of this app may pull" }, 403);
95 const run = queue.then(() => pull(store, read));
96 queue = run.catch(() => undefined);
97 try {
98 last = await run;
99 return json(last);
100 } catch (error) {
101 // The cursor did not move; the forge records the failure. The detail
102 // can name the source's host or URL, so it goes to the log, not the caller.
103 console.error("pull failed:", error instanceof Error ? error.message : error);
104 return json({ error: "pull failed" }, 502);
105 }
106 }
107 if (req.method !== "GET" && req.method !== "HEAD")
108 return json({ error: "method not allowed" }, 405);
109 if (pathname === "/healthz") return new Response("ok", { headers: HEADERS });
110 if (pathname === "/")
111 return new Response(page(store, last, caller.role === "admin"), {
112 headers: { ...HEADERS, "content-type": "text/html; charset=utf-8" },
113 });
114 return new Response("not found", { status: 404, headers: HEADERS });
115 };
116 }
117
118 if (import.meta.main) {
119 const data = process.env.FORGE_APP_DATA;
120 if (data === undefined || data === "") throw new Error("FORGE_APP_DATA is not set");
121 Bun.serve({
122 port: Number(process.env.PORT ?? 3000),
123 hostname: process.env.HOST ?? "127.0.0.1",
124 fetch: makeHandler(openStore(data), { key: process.env.FORGE_APP_IDENTITY_KEY ?? "" }),
125 });
126 }
examples/adapter-template/src/source.ts added +85
@@ -0,0 +1,85 @@
1 /**
2 * The external system this adapter reads from, and the only door to it.
3 *
4 * `SOURCE.hosts` must match `egress` in forge.json: egress is what the
5 * forge's proxy opens, and this list is what the adapter itself agrees to
6 * reach. `sourceFetch` refuses any other host before a byte leaves the
7 * process, so a redirect, a bad config or a cursor that smuggles a URL
8 * cannot point the adapter somewhere else.
9 *
10 * To adapt a real system: change `SOURCE`, the URL in `readPage` and
11 * `parseRecord`. Keep the shape: plain data in, plain data out.
12 */
13
14 export const SOURCE = {
15 name: "Example CRM",
16 hosts: ["api.example.com"],
17 } as const;
18
19 /** One record as the source gives it, keyed by the source's own id. */
20 export interface SourceRecord {
21 readonly id: string;
22 readonly body: string;
23 }
24
25 /** One page of the source: its records and the cursor after them. */
26 export interface Page {
27 readonly records: readonly unknown[];
28 readonly cursor: string;
29 }
30
31 export type Fetch = (url: string, init?: RequestInit) => Promise<Response>;
32
33 /** Thrown when the adapter is asked to reach a host it did not declare. */
34 export class RefusedHost extends Error {
35 constructor(readonly host: string) {
36 super(`refused: ${host} is not a declared source host`);
37 }
38 }
39
40 /**
41 * `fetch`, limited to `https` and the declared hosts. Redirects are not
42 * followed: a 3xx is returned as it is, so it cannot lead elsewhere.
43 */
44 export function sourceFetch(
45 hosts: readonly string[] = SOURCE.hosts,
46 inner: Fetch = fetch,
47 ): Fetch {
48 return (url, init) => {
49 let parsed: URL;
50 try {
51 parsed = new URL(url);
52 } catch {
53 return Promise.reject(new RefusedHost(String(url).slice(0, 100)));
54 }
55 if (parsed.protocol !== "https:" || !hosts.includes(parsed.hostname))
56 return Promise.reject(new RefusedHost(parsed.host));
57 return inner(parsed.href, { ...init, redirect: "manual" });
58 };
59 }
60
61 /** A cursor is one printable line of at most 200 characters (docs/ADAPTERS.md). */
62 export function isCursor(value: unknown): value is string {
63 return typeof value === "string" && value.length <= 200 && /^[\x20-\x7e]*$/.test(value);
64 }
65
66 /** A record the adapter can keep, or `undefined` for one it cannot. */
67 export function parseRecord(raw: unknown): SourceRecord | undefined {
68 if (typeof raw !== "object" || raw === null) return undefined;
69 const { id } = raw as { id?: unknown };
70 if (typeof id !== "string" || id === "" || id.length > 200) return undefined;
71 return { id, body: JSON.stringify(raw) };
72 }
73
74 /** Read the page after `cursor` from the source. */
75 export async function readPage(cursor: string, get: Fetch = sourceFetch()): Promise<Page> {
76 const url = `https://${SOURCE.hosts[0]}/records?after=${encodeURIComponent(cursor)}`;
77 const answer = await get(url, { headers: { accept: "application/json" } });
78 if (!answer.ok) throw new Error(`source answered ${answer.status}`);
79 const body: unknown = await answer.json();
80 if (typeof body !== "object" || body === null) throw new Error("source page is not an object");
81 const { records, cursor: next } = body as { records?: unknown; cursor?: unknown };
82 if (!Array.isArray(records)) throw new Error("source page has no records list");
83 if (!isCursor(next)) throw new Error("source page cursor is not one printable line");
84 return { records, cursor: next };
85 }
examples/adapter-template/src/store.ts added +57
@@ -0,0 +1,57 @@
1 /**
2 * The adapter's state: the records it holds and the cursor it pulls from,
3 * in one bun:sqlite file under `FORGE_APP_DATA`.
4 *
5 * A record is keyed by the source's id, so writing it twice is a no-op;
6 * `put` says whether this call wrote anything. That is what makes a pull
7 * idempotent: a replay finds every record already present.
8 */
9
10 import { Database } from "bun:sqlite";
11 import { mkdirSync } from "node:fs";
12 import { join } from "node:path";
13 import type { SourceRecord } from "./source.ts";
14
15 export interface Store {
16 /** Where the next pull starts; "" before the first. */
17 cursor(): string;
18 setCursor(cursor: string): void;
19 /** Write `record`; `true` when it was new or changed, `false` when held already. */
20 put(record: SourceRecord): boolean;
21 count(): number;
22 /** Run `fn` in one transaction: records and cursor land together or not at all. */
23 atomically<T>(fn: () => T): T;
24 close(): void;
25 }
26
27 /** Open the store in `dir` (made if missing), or in memory for `":memory:"`. */
28 export function openStore(dir: string): Store {
29 let path = ":memory:";
30 if (dir !== ":memory:") {
31 mkdirSync(dir, { recursive: true });
32 path = join(dir, "adapter.sqlite");
33 }
34 const db = new Database(path, { strict: true });
35 db.run("PRAGMA journal_mode = WAL");
36 db.run("CREATE TABLE IF NOT EXISTS record (id TEXT PRIMARY KEY, body TEXT NOT NULL)");
37 db.run("CREATE TABLE IF NOT EXISTS state (key TEXT PRIMARY KEY, value TEXT NOT NULL)");
38 const readCursor = db.query<{ value: string }, []>(
39 "SELECT value FROM state WHERE key = 'cursor'",
40 );
41 const writeCursor = db.query(
42 "INSERT INTO state (key, value) VALUES ('cursor', $value) ON CONFLICT(key) DO UPDATE SET value = excluded.value",
43 );
44 // Writes only when the id is new or its body changed; `changes` says which.
45 const upsert = db.query(
46 "INSERT INTO record (id, body) VALUES ($id, $body) ON CONFLICT(id) DO UPDATE SET body = excluded.body WHERE body <> excluded.body",
47 );
48 const countRows = db.query<{ n: number }, []>("SELECT count(*) AS n FROM record");
49 return {
50 cursor: () => readCursor.get()?.value ?? "",
51 setCursor: (value) => void writeCursor.run({ value }),
52 put: (record) => upsert.run({ id: record.id, body: record.body }).changes > 0,
53 count: () => countRows.get()?.n ?? 0,
54 atomically: (fn) => db.transaction(fn)(),
55 close: () => db.close(),
56 };
57 }
examples/adapter-template/src/who.ts added +45
@@ -0,0 +1,45 @@
1 /**
2 * Who is asking. The forge signs every request it proxies to the app
3 * (docs/APPS.md, "Who is there"):
4 *
5 * x-forge-identity: <userId>|<username>|<role>|<expiresMs>
6 * x-forge-signature: base64url(HMAC-SHA256(FORGE_APP_IDENTITY_KEY, that string))
7 *
8 * `role` is `user` or `admin` (one who may write the app's repository). The
9 * forge drops any copy of these headers a client sends, so a signature that
10 * verifies is the only trust there is: no headers, no key, a bad signature
11 * or an expiry in the past all mean `anon`. The same check as the forge's
12 * app scaffold (`src/who.ts` there).
13 */
14
15 import { createHmac, timingSafeEqual } from "node:crypto";
16
17 export type Who =
18 | { readonly role: "anon" }
19 | { readonly role: "user" | "admin"; readonly id: string; readonly name: string };
20
21 export const ANON: Who = { role: "anon" };
22
23 export function who(request: Request, key: string, now = Date.now()): Who {
24 const identity = request.headers.get("x-forge-identity");
25 const signature = request.headers.get("x-forge-signature");
26 if (key === "" || identity === null || signature === null) return ANON;
27 if (!same(sign(identity, key), signature)) return ANON;
28 const parts = identity.split("|");
29 if (parts.length !== 4) return ANON;
30 const [id, name, role, expires] = parts as [string, string, string, string];
31 if (role !== "user" && role !== "admin") return ANON;
32 if (id === "" || name === "" || !/^\d+$/.test(expires) || Number(expires) <= now) return ANON;
33 return { role, id, name };
34 }
35
36 /** The signature the forge computes. Exported so a test can be the forge. */
37 export const sign = (identity: string, key: string): string =>
38 createHmac("sha256", key).update(identity).digest("base64url");
39
40 /** Constant-time: the compare takes as long for a near miss as for a far one. */
41 function same(a: string, b: string): boolean {
42 const x = Buffer.from(a);
43 const y = Buffer.from(b);
44 return x.length === y.length && timingSafeEqual(x, y);
45 }
examples/adapter-template/test/adapter.test.ts added +189
@@ -0,0 +1,189 @@
1 import { afterEach, describe, expect, test } from "bun:test";
2 import { pull } from "../src/pull.ts";
3 import { makeHandler } from "../src/server.ts";
4 import { readPage, RefusedHost, sourceFetch, type Fetch, type Page } from "../src/source.ts";
5 import { openStore, type Store } from "../src/store.ts";
6 import { sign } from "../src/who.ts";
7
8 /** A source of three records; a page after "c3" is empty. */
9 function fakeSource(): { read: (cursor: string) => Promise<Page>; reads: string[] } {
10 const reads: string[] = [];
11 const all = [
12 { id: "a", name: "Ada" },
13 { id: "b", name: "Bo" },
14 { id: "c", name: "Cy" },
15 ];
16 return {
17 reads,
18 read: async (cursor) => {
19 reads.push(cursor);
20 return cursor === "c3" ? { records: [], cursor } : { records: all, cursor: "c3" };
21 },
22 };
23 }
24
25 let store: Store | undefined;
26 afterEach(() => store?.close());
27
28 const KEY = "test-identity-key";
29 const NOW = 1_800_000_000_000;
30
31 /** A POST /pull as the forge proxies it, signed for `role` until `expires`. */
32 function pullAs(
33 role: string,
34 { expires = NOW + 60_000, key = KEY, headers = {} as Record<string, string> } = {},
35 ): Request {
36 const identity = `u1|ada|${role}|${expires}`;
37 return new Request("http://adapter/pull", {
38 method: "POST",
39 headers: { "x-forge-identity": identity, "x-forge-signature": sign(identity, key), ...headers },
40 });
41 }
42
43 describe("POST /pull answers the four fields", () => {
44 test("an admin's pull writes every record and moves the cursor", async () => {
45 store = openStore(":memory:");
46 const handle = makeHandler(store, { read: fakeSource().read, key: KEY, now: () => NOW });
47 const answer = await handle(pullAs("admin"));
48 expect(answer.status).toBe(200);
49 expect(answer.headers.get("x-content-type-options")).toBe("nosniff");
50 expect(await answer.json()).toEqual({ pulled: 3, skipped: 0, cursor: "c3", errors: 0 });
51 expect(store.count()).toBe(3);
52 });
53
54 test("a cross-origin POST is refused and pulls nothing", async () => {
55 store = openStore(":memory:");
56 const source = fakeSource();
57 const handle = makeHandler(store, { read: source.read, key: KEY, now: () => NOW });
58 const answer = await handle(pullAs("admin", { headers: { origin: "https://evil.test" } }));
59 expect(answer.status).toBe(403);
60 expect(source.reads).toEqual([]);
61 });
62
63 test("a failed read answers a fixed 502 and keeps the cursor", async () => {
64 store = openStore(":memory:");
65 store.setCursor("before");
66 const handle = makeHandler(store, {
67 read: () => Promise.reject(new Error("https://secret.internal/records?token=x")),
68 key: KEY,
69 now: () => NOW,
70 });
71 const quiet = console.error;
72 console.error = () => {};
73 try {
74 const answer = await handle(pullAs("admin"));
75 expect(answer.status).toBe(502);
76 expect(await answer.json()).toEqual({ error: "pull failed" });
77 } finally {
78 console.error = quiet;
79 }
80 expect(store.cursor()).toBe("before");
81 });
82 });
83
84 describe("only an admin the forge signed for may pull", () => {
85 const cases: [string, () => Request, number][] = [
86 ["no headers", () => new Request("http://adapter/pull", { method: "POST" }), 401],
87 ["a bad signature", () => pullAs("admin", { key: "another-key" }), 401],
88 ["an expired identity", () => pullAs("admin", { expires: NOW - 1 }), 401],
89 ["an identity expiring now", () => pullAs("admin", { expires: NOW }), 401],
90 ["a non-admin", () => pullAs("user"), 403],
91 ];
92 for (const [what, req, status] of cases) {
93 test(`${what} is refused with ${status} and reads nothing from the source`, async () => {
94 store = openStore(":memory:");
95 const source = fakeSource();
96 const handle = makeHandler(store, { read: source.read, key: KEY, now: () => NOW });
97 const answer = await handle(req());
98 expect(answer.status).toBe(status);
99 expect(source.reads).toEqual([]);
100 expect(store.count()).toBe(0);
101 });
102 }
103
104 test("without an identity key nobody may pull, even with signed headers", async () => {
105 store = openStore(":memory:");
106 const source = fakeSource();
107 const handle = makeHandler(store, { read: source.read, key: "", now: () => NOW });
108 const answer = await handle(pullAs("admin", { key: "" }));
109 expect(answer.status).toBe(401);
110 expect(source.reads).toEqual([]);
111 });
112
113 test("GET / shows the cursor to an admin only, inside <main>", async () => {
114 store = openStore(":memory:");
115 store.setCursor("page-token-7");
116 const handle = makeHandler(store, { read: fakeSource().read, key: KEY, now: () => NOW });
117 const anon = await (await handle(new Request("http://adapter/"))).text();
118 expect(anon).toContain("<main>");
119 expect(anon).not.toContain("page-token-7");
120 const identity = `u1|ada|admin|${NOW + 60_000}`;
121 const admin = await (
122 await handle(
123 new Request("http://adapter/", {
124 headers: { "x-forge-identity": identity, "x-forge-signature": sign(identity, KEY) },
125 }),
126 )
127 ).text();
128 expect(admin).toContain("page-token-7");
129 });
130 });
131
132 describe("idempotent", () => {
133 test("a replay from the same cursor writes nothing new", async () => {
134 store = openStore(":memory:");
135 const source = fakeSource();
136 const first = await pull(store, source.read);
137 expect(first).toEqual({ pulled: 3, skipped: 0, cursor: "c3", errors: 0 });
138 // The forge repeats the pull from the cursor it last held: "".
139 store.setCursor("");
140 const replay = await pull(store, source.read);
141 expect(source.reads).toEqual(["", ""]);
142 expect(replay).toEqual({ pulled: 0, skipped: 3, cursor: "c3", errors: 0 });
143 expect(store.count()).toBe(3);
144 });
145
146 test("a record without a source id is counted as an error, not written", async () => {
147 store = openStore(":memory:");
148 const result = await pull(store, async () => ({ records: [{ name: "no id" }, null], cursor: "x" }));
149 expect(result).toEqual({ pulled: 0, skipped: 0, cursor: "x", errors: 2 });
150 expect(store.count()).toBe(0);
151 });
152 });
153
154 describe("only the declared source is reachable", () => {
155 const ok: Fetch = async () => Response.json({ records: [], cursor: "next" });
156
157 test("a host outside the source is refused before any request", async () => {
158 const seen: string[] = [];
159 const get = sourceFetch(["api.example.com"], async (url) => {
160 seen.push(url);
161 return ok(url);
162 });
163 for (const url of [
164 "https://evil.example.net/records",
165 "https://api.example.com.evil.test/records",
166 "http://api.example.com/records",
167 "not a url",
168 ]) {
169 await expect(get(url)).rejects.toBeInstanceOf(RefusedHost);
170 }
171 expect(seen).toEqual([]);
172 const answer = await get("https://api.example.com/records");
173 expect(answer.ok).toBe(true);
174 expect(seen).toEqual(["https://api.example.com/records"]);
175 });
176
177 test("readPage goes only to the declared host, and does not follow redirects", async () => {
178 const seen: { url: string; redirect: string | undefined }[] = [];
179 const page = await readPage(
180 "c 1",
181 sourceFetch(["api.example.com"], async (url, init) => {
182 seen.push({ url, redirect: init?.redirect });
183 return ok(url);
184 }),
185 );
186 expect(page).toEqual({ records: [], cursor: "next" });
187 expect(seen).toEqual([{ url: "https://api.example.com/records?after=c%201", redirect: "manual" }]);
188 });
189 });
test/adapter-template.test.ts added +50
@@ -0,0 +1,50 @@
1 /**
2 * The adapter template (examples/adapter-template, docs/ADAPTERS.md) stays a
3 * working adapter: its own suite runs here, in the forge's check, as the
4 * template would run it (`bun test` in its own directory), and its manifest
5 * is one the forge accepts, with every source host opened by egress.
6 */
7
8 import { describe, expect, test } from "bun:test";
9 import { readFileSync } from "node:fs";
10 import { join } from "node:path";
11 import { parseManifest } from "../src/apps/manifest.ts";
12 import { SOURCE } from "../examples/adapter-template/src/source.ts";
13
14 const TEMPLATE = join(import.meta.dir, "..", "examples", "adapter-template");
15
16 function templateManifest() {
17 const m = parseManifest(readFileSync(join(TEMPLATE, "forge.json"), "utf8"), undefined);
18 if (!m.ok) throw new Error(m.error.kind === "invalid" ? m.error.message : m.error.kind);
19 return m.value;
20 }
21
22 describe("the adapter template", () => {
23 test("its forge.json parses, with an adapter block", () => {
24 expect(templateManifest().adapter).toEqual({ schedule: { every: 3600 } });
25 });
26
27 test("every source host is in egress", () => {
28 expect(SOURCE.hosts.length).toBeGreaterThan(0);
29 const egress = templateManifest().egress;
30 for (const host of SOURCE.hosts) expect(egress).toContain(host);
31 });
32
33 test("has no dependencies", () => {
34 const pkg = JSON.parse(readFileSync(join(TEMPLATE, "package.json"), "utf8")) as Record<string, unknown>;
35 expect(pkg["dependencies"]).toBeUndefined();
36 expect(pkg["devDependencies"]).toBeUndefined();
37 });
38
39 test("its own tests pass", () => {
40 const run = Bun.spawnSync([process.execPath, "test"], {
41 cwd: TEMPLATE,
42 stdout: "pipe",
43 stderr: "pipe",
44 });
45 const log = run.stderr.toString() + run.stdout.toString();
46 if (run.exitCode !== 0) throw new Error(`examples/adapter-template: bun test failed\n${log}`);
47 expect(log).toMatch(/\b0 fail\b/);
48 expect(log).not.toMatch(/\b0 pass\b/);
49 }, 30_000);
50 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs