Public: anyone can read this; the crew can change it.
A feature's scheduled tool is called the way its hand-run tool is: the unified timer broke the morning paper's print with a header its app refuses #14218
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/14216-a-feature-s-scheduled-tool-is-called-the → main 7 commits fork
12 files
changed +321
−22
docs/features/feature-schedule.md +21
| @@ -30,6 +30,27 @@ A feature's `schedule` is sugar for top-level `schedules` (#13533): `src/apps/fe | ||
| 30 | 30 | |
| 31 | 31 | A name used by both a `schedules` entry and a feature tool is refused by the manifest. A run due while the forge was away fires within an hour (`catchUp`) and says it was late. **Only while the app is absorbed:** the forge calls a feature's tool only for an app an admin has absorbed. The timer exists from the commit (so its row shows), but when it comes due for an app that is not absorbed it is dropped unfired, with no run recorded and no call made (`permits` in `timer_fire.ts`, set by `timer_service.ts`); Run now on the features page and the terminal's `r` refuse it in words, arming nothing. Un-absorbing therefore stops the schedule at its next due time; absorbing again starts it again, at the next due time, never at once. |
| 32 | 32 | |
| 33 | ## Calling a tool: one contract (#14216) | |
| 34 | ||
| 35 | A call of `POST /forge/tools/<tool>` carries the same headers whoever starts it: the clock, Run now, a person or an agent running the tool by hand (the `features` tool), or a probe. Both paths build them with `callHeaders` in `src/apps/timer_fire.ts`; `test/feature-call-contract.test.ts` holds it. | |
| 36 | ||
| 37 | | Header | Is | | |
| 38 | |---|---| | |
| 39 | | `x-forge-schedule` | the tool's name | | |
| 40 | | `x-forge-scheduled-at` | the instant the call is for (a hand's call: now) | | |
| 41 | | `x-forge-attempt` | `1`, or the try count of a delivery | | |
| 42 | | `idempotency-key` | `<app>:<name>:<scheduled time>` for the clock, `manual:<tool>:<uuid>` for a hand | | |
| 43 | | `x-forge-identity` | `forge-timer\|<idempotency key>\|timer\|<expiry ms>` | | |
| 44 | | `x-forge-signature` | base64url HMAC-SHA256 of `x-forge-identity` under the app's identity key | | |
| 45 | ||
| 46 | **A hand's call is signed as a timer's, deliberately.** The identity of Run now, of a person's or an agent's call and of a probe is `forge-timer|manual:<tool>:<uuid>|timer|<expiry>`: one check (`fromForge`) accepts every call the forge makes to a tool, and the app does not branch on who started it. The `manual:` prefix of the idempotency key is what tells a hand's call from the clock's, if an app wants to know. Only the forge's key signs it; a visitor cannot. | |
| 47 | ||
| 48 | **Verifying "this is my forge".** The key is `FORGE_APP_IDENTITY_KEY` in the app's environment, the same key that verifies a visitor's `x-forge-identity`. A request is the forge's own when `x-forge-identity` starts with `forge-timer|`, has not expired and its signature verifies. Do not guess from the absence of headers: the proxy puts `x-forge-identity` on a visitor's relayed request too, and a guard that treats any such header as a visitor's refuses the clock's call. The app template's `fromForge` (`src/once.ts`) is this check; a tool route should call it. | |
| 49 | ||
| 50 | **Probe.** A tool declared `"probe": true` promises to answer 200 to a call carrying `x-forge-probe: 1` without doing its work. When the forge reads a commit's schedules (once per commit, so also when a mechanism change re-reads every app), it sends each such scheduled tool one probe, after every app's schedules are read and within a 30 s budget for the pass. Only production is probed: a preview's schedules do not fire. An app that refuses it gets a failed run on the schedule, with the answer's status, and its owner is asked at once. A tool without `probe` is never called in advance: a call would do the work. | |
| 51 | ||
| 52 | **Proven.** `/api/v1/health` `schedules.ran` counts a schedule only once a fired run was answered with a 2xx (200, 202, 204 alike); a refused probe never counts. | |
| 53 | ||
| 33 | 54 | ## What the forge records |
| 34 | 55 | |
| 35 | 56 | The runs are the timers' (`timer_run`, `src/core/timers.ts`): when each started, how many attempts, the HTTP status, the first 2 KB of the app's answer, the error sentence. A hand's call (the `features` tool, `mcp`) is the one thing still kept in `feature_fire` and `feature_run` (`src/core/feature_fires.ts`); un-absorbing or deleting the app clears those. |
src/actions/feature_tools.ts +4 −2
| @@ -8,6 +8,7 @@ | ||
| 8 | 8 | |
| 9 | 9 | import type { App } from "../app.ts"; |
| 10 | 10 | import { runRecorded } from "../apps/feature_schedule.ts"; |
| 11 | import { identityKey } from "../apps/proxy.ts"; | |
| 11 | 12 | import { callTool } from "../apps/feature_tools.ts"; |
| 12 | 13 | import { appsRuntime } from "../apps/runtime.ts"; |
| 13 | 14 | import { getAbsorbed } from "../core/features.ts"; |
| @@ -36,6 +37,7 @@ export async function callFeatureTool( | ||
| 36 | 37 | if (spec.writes && !canWrite(app.db, found.repo, actor)) return refuse("forbidden", "A tool that changes things is for someone who can write this app."); |
| 37 | 38 | const port = appsRuntime(app)?.envStatus(found.record, "production")?.port; |
| 38 | 39 | if (port === undefined) return refuse("not_running", "The app is not running."); |
| 40 | const key = identityKey(app.config.dataDir, found.record.id); | |
| 39 | 41 | // A plan is told only once the call could be made: allowed, absorbed, running. |
| 40 | 42 | if (spec.writes && dryRun) return ok({ planned: `Would call ${tool} on ${found.record.slug}: ${spec.description} Send dryRun: false to do it.` }); |
| 41 | 43 | // A tool the schedule names is the same recorded run as the schedule's and the page's, so it |
| @@ -43,10 +45,10 @@ export async function callFeatureTool( | ||
| 43 | 45 | // recorded. |
| 44 | 46 | const scheduled = declared.ok && declared.value?.schedule.some((e) => e.tool === tool) === true; |
| 45 | 47 | if (!scheduled) { |
| 46 | const answer = await callTool(port, tool, args); | |
| 48 | const answer = await callTool(port, tool, args, { key }); | |
| 47 | 49 | return answer.ok ? ok({ answer: answer.value }) : refuse("failed", answer.error); |
| 48 | 50 | } |
| 49 | const ran = await runRecorded(app.db, found.record.id, tool, port, args, "mcp"); | |
| 51 | const ran = await runRecorded(app.db, found.record.id, tool, port, args, key, "mcp"); | |
| 50 | 52 | if (!ran.ran) return refuse("failed", `${tool} is already running.`); |
| 51 | 53 | return ran.answer.ok ? ok({ answer: ran.answer.value }) : refuse("failed", ran.answer.error); |
| 52 | 54 | } |
src/apps/feature_schedule.ts +2 −1
| @@ -29,6 +29,7 @@ export async function runRecorded( | ||
| 29 | 29 | tool: string, |
| 30 | 30 | port: number, |
| 31 | 31 | args: Record<string, unknown>, |
| 32 | identity: string, | |
| 32 | 33 | how: How, |
| 33 | 34 | now: () => number = Date.now, |
| 34 | 35 | ): Promise<{ ran: true; answer: Result<unknown, string> } | { ran: false }> { |
| @@ -39,7 +40,7 @@ export async function runRecorded( | ||
| 39 | 40 | const started = now(); |
| 40 | 41 | // callTool is bounded (WAIT_MS) and answers an error rather than throwing; a throw is still a |
| 41 | 42 | // recorded failure. |
| 42 | const answer = await callTool(port, tool, args).catch((): Result<unknown, string> => err("the app did not answer")); | |
| 43 | const answer = await callTool(port, tool, args, { key: identity, now }).catch((): Result<unknown, string> => err("the app did not answer")); | |
| 43 | 44 | recordFire(db, appId, { tool, at: started, ok: answer.ok, error: answer.ok ? null : answer.error, durationMs: Math.max(0, now() - started) }, how); |
| 44 | 45 | return { ran: true, answer }; |
| 45 | 46 | } finally { |
src/apps/feature_tools.ts +22 −4
| @@ -12,12 +12,18 @@ | ||
| 12 | 12 | */ |
| 13 | 13 | |
| 14 | 14 | import { err, ok, type Result } from "../kernel/result.ts"; |
| 15 | import { callHeaders } from "./timer_fire.ts"; | |
| 15 | 16 | |
| 16 | 17 | export interface ToolSpec { |
| 17 | 18 | readonly name: string; |
| 18 | 19 | readonly description: string; |
| 19 | 20 | /** Whether a call changes anything: a write plans first, like the forge's own. */ |
| 20 | 21 | readonly writes: boolean; |
| 22 | /** | |
| 23 | * Whether the tool answers a probe: a call carrying `x-forge-probe: 1` that checks the forge's | |
| 24 | * headers are accepted and does none of the work. Only such a tool is probed. | |
| 25 | */ | |
| 26 | readonly probe?: true; | |
| 21 | 27 | } |
| 22 | 28 | |
| 23 | 29 | const NAME = /^[a-z][a-z0-9_]{0,39}$/; |
| @@ -37,22 +43,34 @@ export function parseTools(raw: unknown): Result<readonly ToolSpec[], string> { | ||
| 37 | 43 | const t = (raw as Record<string, unknown>)[name]; |
| 38 | 44 | if (!NAME.test(name)) return err(`feature: tool ${JSON.stringify(name)}: names are lowercase words with _, at most 40 characters`); |
| 39 | 45 | if (typeof t !== "object" || t === null || Array.isArray(t)) return err(`feature: tool ${name} must be an object`); |
| 40 | const { description, writes, ...extra } = t as Record<string, unknown>; | |
| 46 | const { description, writes, probe, ...extra } = t as Record<string, unknown>; | |
| 41 | 47 | if (typeof description !== "string" || description.trim() === "" || description.length > DESCRIPTION_MAX) |
| 42 | 48 | return err(`feature: tool ${name}: description is text of 1 to ${DESCRIPTION_MAX} characters`); |
| 43 | 49 | if (writes !== undefined && typeof writes !== "boolean") return err(`feature: tool ${name}: writes is true or false`); |
| 50 | if (probe !== undefined && typeof probe !== "boolean") return err(`feature: tool ${name}: probe is true or false`); | |
| 44 | 51 | if (Object.keys(extra).length > 0) return err(`feature: tool ${name}: unknown field ${Object.keys(extra)[0]}`); |
| 45 | out.push({ name, description: description.trim(), writes: writes !== false }); | |
| 52 | out.push({ name, description: description.trim(), writes: writes !== false, ...(probe === true ? { probe: true as const } : {}) }); | |
| 46 | 53 | } |
| 47 | 54 | return ok(out); |
| 48 | 55 | } |
| 49 | 56 | |
| 50 | 57 | /** Call a declared tool on the app listening on `port`: its JSON answer, or why not. */ |
| 51 | export async function callTool(port: number, tool: string, args: Record<string, unknown>): Promise<Result<unknown, string>> { | |
| 58 | export async function callTool( | |
| 59 | port: number, | |
| 60 | tool: string, | |
| 61 | args: Record<string, unknown>, | |
| 62 | opts: { readonly key: string; readonly probe?: boolean; readonly now?: () => number }, | |
| 63 | ): Promise<Result<unknown, string>> { | |
| 64 | const now = (opts.now ?? Date.now)(); | |
| 52 | 65 | try { |
| 53 | 66 | const res = await fetch(`http://127.0.0.1:${port}/forge/tools/${tool}`, { |
| 54 | 67 | method: "POST", |
| 55 | headers: { "content-type": "application/json", accept: "application/json" }, | |
| 68 | headers: { | |
| 69 | "content-type": "application/json", | |
| 70 | accept: "application/json", | |
| 71 | ...callHeaders(opts.key, tool, now, 1, now + WAIT_MS, `manual:${tool}:${crypto.randomUUID()}`), | |
| 72 | ...(opts.probe === true ? { "x-forge-probe": "1" } : {}), | |
| 73 | }, | |
| 56 | 74 | body: JSON.stringify({ args }), |
| 57 | 75 | redirect: "manual", |
| 58 | 76 | signal: AbortSignal.timeout(WAIT_MS), |
src/apps/schedule_probe.ts added +51
| @@ -0,0 +1,51 @@ | ||
| 1 | /** | |
| 2 | * A dry run of a feature's scheduled tools, once, when the forge reads them from a commit | |
| 3 | * (#14216): a schedule that moved to a new mechanism, or a new commit that changed what an app's | |
| 4 | * guard accepts, is found refused the same hour, not at the next 07:15. | |
| 5 | * | |
| 6 | * The call is a tool call like any other (`callTool`: the headers of a delivery) plus | |
| 7 | * `x-forge-probe: 1`. Only a tool that declares `"probe": true` is probed: it promises to answer | |
| 8 | * 200 to a probe and do none of its work, so a probe never prints the paper. A refusal is recorded | |
| 9 | * as a failed run of the schedule, which the app's owner is asked about like any failed run; | |
| 10 | * an answer of 200 records nothing, because a schedule is proven only by a fired run the app | |
| 11 | * answered 200 (`overdueSchedules`). | |
| 12 | */ | |
| 13 | ||
| 14 | import type { Database } from "bun:sqlite"; | |
| 15 | import type { AppRecord } from "../core/apps.ts"; | |
| 16 | import { recordRun, scheduleTimers, type TimerEnv } from "../core/timers.ts"; | |
| 17 | import { callTool, type ToolSpec } from "./feature_tools.ts"; | |
| 18 | import { toolPath } from "./feature.ts"; | |
| 19 | import type { Reach } from "./timer_fire.ts"; | |
| 20 | ||
| 21 | /** Probe each scheduled tool of `app` in `env` that answers probes; the tools refused. */ | |
| 22 | export async function probeSchedules( | |
| 23 | db: Database, | |
| 24 | reach: Reach, | |
| 25 | app: AppRecord, | |
| 26 | env: TimerEnv, | |
| 27 | tools: readonly ToolSpec[], | |
| 28 | now: () => number = Date.now, | |
| 29 | ): Promise<string[]> { | |
| 30 | const refused: string[] = []; | |
| 31 | for (const t of scheduleTimers(db, app.id)) { | |
| 32 | if (t.env !== env || !t.path.startsWith(toolPath(""))) continue; | |
| 33 | const name = t.path.slice(toolPath("").length); | |
| 34 | if (tools.find((x) => x.name === name)?.probe !== true) continue; | |
| 35 | if (reach.permits?.(app, t) === false) continue; | |
| 36 | const port = await reach.ensure(app, env).catch(() => undefined); | |
| 37 | if (port === undefined) continue; | |
| 38 | const started = now(); | |
| 39 | const answer = await callTool(port, name, {}, { key: reach.identityKey(app.id), probe: true, now }); | |
| 40 | if (answer.ok) continue; | |
| 41 | refused.push(name); | |
| 42 | const status = /^the app answered (\d{3})$/.exec(answer.error)?.[1]; | |
| 43 | recordRun(db, { | |
| 44 | appId: app.id, env, name: t.name, idemKey: `probe:${t.name}:${started}`, scheduledAt: started, startedAt: started, finishedAt: now(), | |
| 45 | attempts: 1, status: "failed", httpStatus: status === undefined ? null : Number(status), response: null, | |
| 46 | error: `A probe of ${name} was refused: ${answer.error}. The schedule would fail the same way; the app's guard must accept the forge's signed headers (docs/features/feature-schedule.md, "Calling a tool").`, | |
| 47 | lateMs: 0, lateReason: null, durationMs: Math.max(0, now() - started), | |
| 48 | }); | |
| 49 | } | |
| 50 | return refused; | |
| 51 | } |
src/apps/timer_fire.ts +15 −3
| @@ -59,15 +59,27 @@ export function idempotencyKey(slug: string, t: Pick<Timer, "name" | "key" | "sc | ||
| 59 | 59 | * `forge-timer|<idempotency key>|timer|<expiry ms>`. |
| 60 | 60 | */ |
| 61 | 61 | export function deliveryHeaders(key: string, t: Timer, attempt: number, expires: number, idemKey: string): Record<string, string> { |
| 62 | return { | |
| 63 | ...callHeaders(key, t.name === "" ? t.key : t.name, t.scheduledAt, attempt, expires, idemKey), | |
| 64 | ...(t.body === null ? {} : { "content-type": t.contentType ?? "application/octet-stream" }), | |
| 65 | }; | |
| 66 | } | |
| 67 | ||
| 68 | /** | |
| 69 | * The one set of headers the forge sends when it calls an app's route, whoever starts the call: the | |
| 70 | * clock, Run now, a person or an agent running a feature's tool by hand, a probe (docs/features/ | |
| 71 | * feature-schedule.md, "Calling a tool"). The signature covers | |
| 72 | * `forge-timer|<idempotency key>|timer|<expiry ms>`. | |
| 73 | */ | |
| 74 | export function callHeaders(key: string, name: string, scheduledAt: number, attempt: number, expires: number, idemKey: string): Record<string, string> { | |
| 62 | 75 | const identity = `forge-timer|${idemKey}|timer|${expires}`; |
| 63 | 76 | return { |
| 64 | "x-forge-schedule": t.name === "" ? t.key : t.name, | |
| 65 | "x-forge-scheduled-at": iso(t.scheduledAt), | |
| 77 | "x-forge-schedule": name, | |
| 78 | "x-forge-scheduled-at": iso(scheduledAt), | |
| 66 | 79 | "x-forge-attempt": String(attempt), |
| 67 | 80 | "idempotency-key": idemKey, |
| 68 | 81 | "x-forge-identity": identity, |
| 69 | 82 | "x-forge-signature": createHmac("sha256", key).update(identity).digest("base64url"), |
| 70 | ...(t.body === null ? {} : { "content-type": t.contentType ?? "application/octet-stream" }), | |
| 71 | 83 | }; |
| 72 | 84 | } |
| 73 | 85 |
src/apps/timer_service.ts +39 −4
| @@ -15,7 +15,8 @@ import { getRepoById } from "../core/repos.ts"; | ||
| 15 | 15 | import { dropGoneEnvs, syncSchedules, syncedCommit, type Timer } from "../core/timers.ts"; |
| 16 | 16 | import { readFileAt } from "./checkout.ts"; |
| 17 | 17 | import { identityKey } from "./proxy.ts"; |
| 18 | import { toolPath } from "./feature.ts"; | |
| 18 | import { parseFeature, toolPath } from "./feature.ts"; | |
| 19 | import { probeSchedules } from "./schedule_probe.ts"; | |
| 19 | 20 | import { parseManifest } from "./manifest.ts"; |
| 20 | 21 | import { appsRuntime } from "./runtime.ts"; |
| 21 | 22 | import type { EnvName } from "./data.ts"; |
| @@ -25,9 +26,12 @@ import { during } from "../perf/jobs.ts"; | ||
| 25 | 26 | |
| 26 | 27 | /** |
| 27 | 28 | * Bumped when what a commit's manifest means for schedules changes, so every environment is read |
| 28 | * again (2: a feature's `schedule` is a schedule, #13533). | |
| 29 | * again (2: a feature's `schedule` is a schedule, #13533; 3: each is probed once, #14216). | |
| 29 | 30 | */ |
| 30 | const SYNC_REVISION = "#2"; | |
| 31 | /** The most one pass of the sync spends probing, all apps together. */ | |
| 32 | const PROBE_BUDGET_MS = 30_000; | |
| 33 | ||
| 34 | const SYNC_REVISION = "#3"; | |
| 31 | 35 | |
| 32 | 36 | /** |
| 33 | 37 | * A feature's tool is called only while an admin has absorbed the app (docs/features/absorb.md). |
| @@ -52,9 +56,12 @@ export function runtimeReach(app: App): Reach { | ||
| 52 | 56 | } |
| 53 | 57 | |
| 54 | 58 | /** Read each app's declared schedules at the commit each environment serves, once per commit. */ |
| 55 | export async function syncAll(app: App, now: () => number = Date.now): Promise<void> { | |
| 59 | export async function syncAll(app: App, now: () => number = Date.now, reach: Reach = runtimeReach(app)): Promise<void> { | |
| 56 | 60 | const rt = appsRuntime(app); |
| 57 | 61 | if (rt === undefined) return; |
| 62 | // Probes call apps (waking one, waiting on its answer), so they run after every app's schedules | |
| 63 | // are read: a slow app delays no other app's read. | |
| 64 | const toProbe: { record: AppRecord; text: string }[] = []; | |
| 58 | 65 | for (const record of listApps(app.db)) { |
| 59 | 66 | const live = new Set<string>(); |
| 60 | 67 | for (const st of rt.status(record)) { |
| @@ -69,9 +76,19 @@ export async function syncAll(app: App, now: () => number = Date.now): Promise<v | ||
| 69 | 76 | // A forge.json that does not parse is the deploy's to report; it declares no schedule, so the |
| 70 | 77 | // rows are dropped. |
| 71 | 78 | syncSchedules(app.db, record.id, st.env, manifest?.ok ? (manifest.value.schedules ?? []) : [], sync, now()); |
| 79 | // Only production is probed: a preview's schedules do not fire (they are read for the page), | |
| 80 | // so a refusal there would be reported for a run that never comes. | |
| 81 | if (st.env === "production" && text !== undefined) toProbe.push({ record, text }); | |
| 72 | 82 | } |
| 73 | 83 | dropGoneEnvs(app.db, record.id, live); |
| 74 | 84 | } |
| 85 | // Bounded in all: probes still waiting when the budget is spent are left; they are not retried, | |
| 86 | // for a schedule is proven by a run, not by its probe. | |
| 87 | const deadline = Date.now() + PROBE_BUDGET_MS; | |
| 88 | for (const { record, text } of toProbe) { | |
| 89 | if (Date.now() >= deadline) break; | |
| 90 | await Promise.race([probeFeature(app.db, reach, record, text, now), Bun.sleep(deadline - Date.now())]); | |
| 91 | } | |
| 75 | 92 | } |
| 76 | 93 | |
| 77 | 94 | export function startTimers(app: App, tickMs = 1000): { stop(): void } | undefined { |
| @@ -103,3 +120,21 @@ export function startTimers(app: App, tickMs = 1000): { stop(): void } | undefin | ||
| 103 | 120 | tick.unref(); |
| 104 | 121 | return { stop: () => clearInterval(tick) }; |
| 105 | 122 | } |
| 123 | ||
| 124 | /** | |
| 125 | * Probe the scheduled tools a commit's `forge.json` declares, once; a text that does not parse has | |
| 126 | * none. | |
| 127 | */ | |
| 128 | async function probeFeature(db: Database, reach: Reach, record: AppRecord, text: string, now: () => number): Promise<void> { | |
| 129 | let raw: unknown; | |
| 130 | try { | |
| 131 | raw = JSON.parse(text); | |
| 132 | } catch { | |
| 133 | return; | |
| 134 | } | |
| 135 | if (typeof raw !== "object" || raw === null || Array.isArray(raw)) return; | |
| 136 | const feature = parseFeature(raw as Record<string, unknown>); | |
| 137 | if (feature.ok && feature.value !== undefined && feature.value.schedule.length > 0) { | |
| 138 | await probeSchedules(db, reach, record, "production", feature.value.tools, now).catch(() => undefined); | |
| 139 | } | |
| 140 | } |
src/core/timers.ts +5 −2
| @@ -119,7 +119,10 @@ export interface ScheduleCounts { | ||
| 119 | 119 | readonly oldestSeconds: number; |
| 120 | 120 | /** Unpaused schedules in all. */ |
| 121 | 121 | readonly total: number; |
| 122 | /** Of those, how many have at least one recorded run. */ | |
| 122 | /** | |
| 123 | * Of those, how many have a fired run the app answered 2xx (a refused probe, or a run it answered | |
| 124 | * otherwise, proves nothing). | |
| 125 | */ | |
| 123 | 126 | readonly ran: number; |
| 124 | 127 | } |
| 125 | 128 | |
| @@ -137,7 +140,7 @@ export function overdueSchedules(db: Database, now: number): ScheduleCounts { | ||
| 137 | 140 | .get(now - OVERDUE_AFTER_MS); |
| 138 | 141 | const all = db |
| 139 | 142 | .query<{ total: number; ran: number | null }, []>( |
| 140 | `SELECT COUNT(*) AS total, SUM(EXISTS (SELECT 1 FROM timer_run r WHERE r.app_id = t.app_id AND r.name = t.name AND r.env = t.env)) AS ran | |
| 143 | `SELECT COUNT(*) AS total, SUM(EXISTS (SELECT 1 FROM timer_run r WHERE r.app_id = t.app_id AND r.name = t.name AND r.env = t.env AND r.status = 'ok' AND r.http_status BETWEEN 200 AND 299)) AS ran | |
| 141 | 144 | FROM timer t WHERE t.kind = 'schedule' AND t.paused = 0`, |
| 142 | 145 | ) |
| 143 | 146 | .get(); |
test/feature-call-contract.test.ts added +123
| @@ -0,0 +1,123 @@ | ||
| 1 | /** | |
| 2 | * One contract for calling a feature's tool (#14216): the hand's call and the clock's carry the | |
| 3 | * same headers, an app can verify them by signature, and a schedule whose app refuses them is | |
| 4 | * flagged at once by a probe, not at its next firing. The app here refuses anything it cannot | |
| 5 | * verify. | |
| 6 | */ | |
| 7 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 8 | import { createHmac } from "node:crypto"; | |
| 9 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 10 | import { tmpdir } from "node:os"; | |
| 11 | import { join } from "node:path"; | |
| 12 | import { createApp, type App } from "../src/app.ts"; | |
| 13 | import { createApp as createAppRecord, getAppById, type AppRecord } from "../src/core/apps.ts"; | |
| 14 | import { createRepo } from "../src/core/repos.ts"; | |
| 15 | import { createUser } from "../src/core/users.ts"; | |
| 16 | import { overdueSchedules, recordRun, runsOf, failingNow, scheduleTimer, syncSchedules } from "../src/core/timers.ts"; | |
| 17 | import { scheduleSpecs } from "../src/apps/feature.ts"; | |
| 18 | import { callTool } from "../src/apps/feature_tools.ts"; | |
| 19 | import { probeSchedules } from "../src/apps/schedule_probe.ts"; | |
| 20 | import { deliveryHeaders, type Reach } from "../src/apps/timer_fire.ts"; | |
| 21 | import type { ToolSpec } from "../src/apps/feature_tools.ts"; | |
| 22 | ||
| 23 | const KEY = "the-apps-identity-key"; | |
| 24 | let dataDir: string; | |
| 25 | let app: App; | |
| 26 | let record: AppRecord; | |
| 27 | const seen: Record<string, string>[] = []; | |
| 28 | let probes = 0; | |
| 29 | let works = 0; | |
| 30 | ||
| 31 | /** | |
| 32 | * The documented verification: a valid signature over x-forge-identity under the app's key, not yet | |
| 33 | * expired. | |
| 34 | */ | |
| 35 | function fromForge(h: Headers): boolean { | |
| 36 | const identity = h.get("x-forge-identity") ?? ""; | |
| 37 | const sig = h.get("x-forge-signature") ?? ""; | |
| 38 | const [kind, , , expires] = identity.split("|"); | |
| 39 | return kind === "forge-timer" && Number(expires) > Date.now() && sig === createHmac("sha256", KEY).update(identity).digest("base64url"); | |
| 40 | } | |
| 41 | ||
| 42 | const server = Bun.serve({ | |
| 43 | port: 0, | |
| 44 | hostname: "127.0.0.1", | |
| 45 | fetch(req) { | |
| 46 | // An app that refuses what it cannot verify, and any header it does not know. | |
| 47 | if (!fromForge(req.headers)) return new Response("nope", { status: 404 }); | |
| 48 | seen.push(Object.fromEntries(req.headers)); | |
| 49 | if (req.headers.get("x-forge-probe") === "1") { | |
| 50 | probes++; | |
| 51 | return Response.json({ probe: true }); | |
| 52 | } | |
| 53 | works++; | |
| 54 | return Response.json({ printed: true }); | |
| 55 | }, | |
| 56 | }); | |
| 57 | ||
| 58 | beforeAll(async () => { | |
| 59 | dataDir = await mkdtemp(join(tmpdir(), "forge-call-contract-")); | |
| 60 | app = createApp({ dataDir, port: 0, disk: () => ({ freeBytes: 95, totalBytes: 100 }) }); | |
| 61 | const owner = await createUser(app.db, { origin: "seed", trusted: true, username: "paper-owner", email: "[email protected]", password: "a-good-passphrase" }); | |
| 62 | if (!owner.ok) throw new Error(owner.error); | |
| 63 | const repo = await createRepo(app.db, app.reposRoot, { ownerId: owner.value.id, name: "paper" }); | |
| 64 | if (!repo.ok) throw new Error(repo.error); | |
| 65 | const made = createAppRecord(app.db, { repoId: repo.value.id, slug: "paper", access: "private" }); | |
| 66 | if (!made.ok) throw new Error(made.error); | |
| 67 | record = getAppById(app.db, made.value.id)!; | |
| 68 | const specs = scheduleSpecs([{ tool: "print_today", at: "07:15", tz: "America/Chicago" }]); | |
| 69 | syncSchedules(app.db, record.id, "production", specs, "c1", Date.now()); | |
| 70 | }); | |
| 71 | afterAll(async () => { | |
| 72 | server.stop(true); | |
| 73 | app.close(); | |
| 74 | await rm(dataDir, { recursive: true, force: true }); | |
| 75 | }); | |
| 76 | ||
| 77 | const reach = (key: string): Reach => ({ ensure: async () => server.port, identityKey: () => key, fetch }); | |
| 78 | const tool = (probe: boolean): ToolSpec => ({ name: "print_today", description: "Print", writes: true, ...(probe ? { probe: true as const } : {}) }); | |
| 79 | ||
| 80 | test("a person's call carries the same headers as the clock's, and the app verifies them", async () => { | |
| 81 | const out = await callTool(server.port!, "print_today", {}, { key: KEY }); | |
| 82 | expect(out).toEqual({ ok: true, value: { printed: true } }); | |
| 83 | const timer = scheduleTimer(app.db, record.id, "print_today", "production")!; | |
| 84 | const clock = Object.keys(deliveryHeaders(KEY, timer, 1, Date.now() + 1000, "k")).filter((h) => h !== "content-type"); | |
| 85 | const hand = seen.at(-1)!; | |
| 86 | for (const h of clock) expect(hand[h]).toBeDefined(); | |
| 87 | // Signed under another key, the app refuses it as it would a visitor's relayed request. | |
| 88 | expect(await callTool(server.port!, "print_today", {}, { key: "someone-else" })).toEqual({ ok: false, error: "the app answered 404" }); | |
| 89 | }); | |
| 90 | ||
| 91 | test("a probe of a tool the app refuses is a failed run the owner is asked about; the schedule is not proven", async () => { | |
| 92 | const before = overdueSchedules(app.db, Date.now()).ran; | |
| 93 | const refused = await probeSchedules(app.db, reach("a-wrong-key"), record, "production", [tool(true)]); | |
| 94 | expect(refused).toEqual(["print_today"]); | |
| 95 | const run = runsOf(app.db, record.id, "print_today", 5)[0]!; | |
| 96 | expect(run.status).toBe("failed"); | |
| 97 | expect(run.httpStatus).toBe(404); | |
| 98 | expect(run.error).toContain("refused"); | |
| 99 | expect(failingNow(app.db, record.id).map((r) => r.name)).toEqual(["print_today"]); | |
| 100 | expect(overdueSchedules(app.db, Date.now()).ran).toBe(before); | |
| 101 | }); | |
| 102 | ||
| 103 | test("a probe the app accepts records nothing and does none of the work; a tool that does not answer probes is never called", async () => { | |
| 104 | const runs = runsOf(app.db, record.id).length; | |
| 105 | const worked = works; | |
| 106 | expect(await probeSchedules(app.db, reach(KEY), record, "production", [tool(true)])).toEqual([]); | |
| 107 | expect(probes).toBe(1); | |
| 108 | expect(works).toBe(worked); | |
| 109 | expect(runsOf(app.db, record.id).length).toBe(runs); | |
| 110 | expect(await probeSchedules(app.db, reach("a-wrong-key"), record, "production", [tool(false)])).toEqual([]); | |
| 111 | expect(runsOf(app.db, record.id).length).toBe(runs); | |
| 112 | }); | |
| 113 | ||
| 114 | test("a schedule whose app answers 204 to its run is proven; a 404 is not", () => { | |
| 115 | const timer = scheduleTimer(app.db, record.id, "print_today", "production")!; | |
| 116 | const base = { appId: record.id, env: "production" as const, name: timer.name, scheduledAt: 1, startedAt: 1, finishedAt: 2, attempts: 1, response: null, error: null, lateMs: 0, lateReason: null, durationMs: 1 }; | |
| 117 | const proven = () => overdueSchedules(app.db, Date.now()).ran; | |
| 118 | const before = proven(); | |
| 119 | recordRun(app.db, { ...base, idemKey: "r-404", status: "failed", httpStatus: 404 }); | |
| 120 | expect(proven()).toBe(before); | |
| 121 | recordRun(app.db, { ...base, idemKey: "r-204", status: "ok", httpStatus: 204 }); | |
| 122 | expect(proven()).toBe(1); | |
| 123 | }); |
test/feature-schedule-one.test.ts +2 −1
| @@ -134,6 +134,7 @@ describe("the morning paper's print", () => { | ||
| 134 | 134 | const runs = runsOf(db, id, "print_today"); |
| 135 | 135 | expect(runs.map((r) => r.status)).toEqual(["failed"]); |
| 136 | 136 | expect(runs[0]?.error).toContain("the app did not answer"); |
| 137 | expect(overdueSchedules(db, now)).toMatchObject({ total: 1, ran: 1 }); | |
| 137 | // A failed run proves nothing (#14216): only a fired run the app answered 200 counts as ran. | |
| 138 | expect(overdueSchedules(db, now)).toMatchObject({ total: 1, ran: 0 }); | |
| 138 | 139 | }); |
| 139 | 140 | }); |
test/feature-tools.test.ts +5 −5
| @@ -47,7 +47,7 @@ describe("calling a tool on the app", () => { | ||
| 47 | 47 | return Response.json({ found: 3 }); |
| 48 | 48 | }); |
| 49 | 49 | try { |
| 50 | const out = await callTool(s.port!, "scan", { depth: 2 }); | |
| 50 | const out = await callTool(s.port!, "scan", { depth: 2 }, { key: "k" }); | |
| 51 | 51 | expect(out).toEqual({ ok: true, value: { found: 3 } }); |
| 52 | 52 | expect(seen).toBe('POST /forge/tools/scan {"args":{"depth":2}}'); |
| 53 | 53 | } finally { |
| @@ -63,13 +63,13 @@ describe("calling a tool on the app", () => { | ||
| 63 | 63 | return new Response("x".repeat(70 * 1024)); |
| 64 | 64 | }); |
| 65 | 65 | try { |
| 66 | expect((await callTool(bad.port!, "boom", {})).ok).toBe(false); | |
| 67 | expect((await callTool(bad.port!, "text", {})).ok).toBe(false); | |
| 68 | const big = await callTool(bad.port!, "big", {}); | |
| 66 | expect((await callTool(bad.port!, "boom", {}, { key: "k" })).ok).toBe(false); | |
| 67 | expect((await callTool(bad.port!, "text", {}, { key: "k" })).ok).toBe(false); | |
| 68 | const big = await callTool(bad.port!, "big", {}, { key: "k" }); | |
| 69 | 69 | expect(!big.ok && big.error).toContain("64 KB"); |
| 70 | 70 | } finally { |
| 71 | 71 | bad.stop(true); |
| 72 | 72 | } |
| 73 | expect((await callTool(1, "scan", {})).ok).toBe(false); | |
| 73 | expect((await callTool(1, "scan", {}, { key: "k" })).ok).toBe(false); | |
| 74 | 74 | }); |
| 75 | 75 | }); |
test/timers-sync.test.ts +32
| @@ -10,6 +10,7 @@ import { createApp, type App } from "../src/app.ts"; | ||
| 10 | 10 | import { AppsRuntime } from "../src/apps/runtime.ts"; |
| 11 | 11 | import { newApp } from "../src/actions/apps.ts"; |
| 12 | 12 | import { commitToBranch } from "../src/actions/repos.ts"; |
| 13 | import type { Reach } from "../src/apps/timer_fire.ts"; | |
| 13 | 14 | import { syncAll } from "../src/apps/timer_service.ts"; |
| 14 | 15 | import { getAppByRepo } from "../src/core/apps.ts"; |
| 15 | 16 | import { getRepo } from "../src/core/repos.ts"; |
| @@ -66,3 +67,34 @@ test("the schedules a production commit declares become rows once per commit; a | ||
| 66 | 67 | await syncAll(app); |
| 67 | 68 | expect(scheduleTimers(app.db, record.id)[0]!.paused).toBe(true); |
| 68 | 69 | }, 30_000); |
| 70 | ||
| 71 | test("a new commit's scheduled tool is probed once by the sync pass, and not again on the next", async () => { | |
| 72 | const alice = getUserByName(app.db, "alice")!; | |
| 73 | const made = await newApp(app, alice, { name: "gazette", visibility: "private" }); | |
| 74 | if (!made.ok) throw new Error(String(made.error)); | |
| 75 | const repo = getRepo(app.db, "alice", "gazette")!; | |
| 76 | const record = getAppByRepo(app.db, repo.id)!; | |
| 77 | const forgeJson = JSON.stringify({ | |
| 78 | app: { run: "bun run src/server.ts" }, | |
| 79 | feature: { name: "Gazette", tools: { print_today: { description: "Print", probe: true } }, schedule: [{ tool: "print_today", at: "07:15", tz: "America/Chicago" }] }, | |
| 80 | }); | |
| 81 | const committed = await commitToBranch(app, alice, repo, { message: "feature", files: [{ path: "forge.json", content: forgeJson }] }); | |
| 82 | if (!committed.ok) throw new Error(JSON.stringify(committed.error)); | |
| 83 | const tip = (await resolveOid(repoDir(app.reposRoot, "alice", "gazette"), `refs/heads/${repo.defaultBranch}`))!; | |
| 84 | const production = () => runtime.status(record).find((s) => s.env === "production"); | |
| 85 | for (let i = 0; i < 100 && production()?.commitOid !== tip; i++) { | |
| 86 | await Bun.sleep(100); // real-timer: waits for the runtime to deploy the commit | |
| 87 | } | |
| 88 | expect(production()?.commitOid).toBe(tip); | |
| 89 | let probes = 0; | |
| 90 | const stub = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: (req) => (req.headers.get("x-forge-probe") === "1" ? (probes++, new Response(null, { status: 204 })) : new Response("no", { status: 404 })) }); | |
| 91 | try { | |
| 92 | const reach: Reach = { ensure: async () => stub.port, identityKey: () => "k", fetch }; | |
| 93 | await syncAll(app, Date.now, reach); | |
| 94 | expect(probes).toBe(1); | |
| 95 | await syncAll(app, Date.now, reach); | |
| 96 | expect(probes).toBe(1); | |
| 97 | } finally { | |
| 98 | stub.stop(true); | |
| 99 | } | |
| 100 | }, 30_000); |
new above
offline