2b44f54 · ca83f60 · eb13b5d · 41318f0 · 73d9b66 · 1a9a4cf
@@ -1279,6 +1279,12 @@
Declare schedules in `forge.json`:
A fired job is an ordinary request to the app's port and counts toward its own limits. How to try it: `docs/features/timers.md`.
### Jobs
A **durable job** is code in the app that reads top to bottom, waits an hour or a week with no process running, survives a deploy, and never repeats a step that finished (`src/jobs.ts` in the app template; no dependency). The forge's part is only the timer above: `sleep` and `waitFor` save where the run is in the app's own database and arm a timer with key `job:<run>:<wake>` that `POST`s `/jobs/run` with `{ run, wake }`; the function then runs again from the top and finished steps answer from their saved results. A step must be safe to retry until it returns (it gets an `idempotencyKey`). A step that throws is retried with backoff (default 3 retries) and then the run is failed: the app answers `422` with `x-forge-no-retry: 1`, which the forge records as failed at once (no redelivery) and asks the owner once, by run. A step or job name that no longer matches what a run did fails that run plainly; `version` keeps an old path for old runs. A step result is JSON up to 256 KB; unfinished runs per app are `JOBS_ACTIVE_MAX` (default 1000) and pending timers `FORGE_TIMERS_PER_APP`.
- **Seen and steered.** The runs live in the app, so the forge asks it (waking it), signed as a timer's delivery is: `GET /jobs/runs`, `POST /jobs/runs/:id/cancel|retry`. The app's page links to `/:owner/:name/app/jobs` while it holds a job timer: name, key, started, where it is now (running, sleeping until, waiting for, done, failed), **Cancel** and **Retry from the failed step**. `GET /api/v1/repos/:owner/:name/jobs`, `POST …/jobs/:id/(cancel|retry)` and the `app_deploy` MCP tool (`jobs`, `cancel_job`, `retry_job`) answer the same rows (`src/actions/jobs.ts`). A writer's: a run's error and result are the app's own text.
- How to try it: `docs/features/jobs.md`.
## Where state lives
Git is truth for code. The app's serving configuration (slug, access, the
@@ -49,7 +49,7 @@
bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
| `accounts` | — | Accounts, newest first: role, verified, suspended; `cursor` pages | 264 |
| `agent` | — | Admin: create\|update\|remove an agent | 371 |
| `agents` | — | Agents you may see, as GET /api/v1/agents | 160 |
| `app_deploy` | `redeploy`, `stop`, `pull`, `schedules`, `run_schedule`, `pause_schedule`, `resume_schedule` | Redeploy or stop production or a pull-N preview, pull an adapter now, or list and steer the app's schedules | 419 |
| `app_deploy` | `redeploy`, `stop`, `pull`, `schedules`, `run_schedule`, `pause_schedule`, `resume_schedule`, `jobs`, `cancel_job`, `retry_job` | Redeploy or stop an env, pull, list and steer schedules and jobs | 408 |
| `app_domains` | — | App domains: list, add, check, remove | 201 |
| `app_env` | `list`, `set`, `unset` | An app's environment variables: list names, set or unset by scope; write-only | 393 |
| `app_members` | — | People a private app names, who may enter it and its previews | 297 |
@@ -0,0 +1,13 @@
# Durable jobs
A job is code that waits (`sleep`, `waitFor`) without a process, survives restarts and deploys, and runs each step once. The rules are the "Jobs" part of "Time" in `docs/APPS.md`; the library and a worked example are in every new app (`src/jobs.ts`, `src/reminder.ts`, README "Jobs").
1. In a new app, open a note signed in and press **Remind me tomorrow** (`POST /notes/:id/remind`): a run of the `remind` job starts, loads the note, and waits a day for the note to be removed.
2. Open the app on the forge: while the run holds a timer, its page has **Jobs → See the job runs** (`/:owner/:name/app/jobs`). The row says `waiting for removed:<id>, until <time> at the latest`. **Cancel** stops it; a failed run offers **Retry from the failed step**.
3. The same from a script: `curl -H "authorization: Bearer $TOKEN" $FORGE_URL/api/v1/repos/<owner>/<name>/jobs`, `curl -X POST …/jobs/<run>/cancel`; the `app_deploy` MCP tool does it with `action: "jobs"`, `"cancel_job"` or `"retry_job"` (`name` is the run id).
4. The terminal: with the app's repository open, the palette offers **Job runs** (a writer's, while the app holds a job timer); `c` cancels the run under the cursor, `t` retries a failed one, `g` reads again (`src/tui/screens/jobs.ts`).
5. A run that fails for good (a step out of retries, or a changed job) asks the owner once.
Where the code is: the runtime `src/jobs.ts` in the app template (`templates/app-template.json`; its tests, `src/jobs.test.ts`, run by `test/template-jobs.test.ts`), the forge's rows and steering `src/actions/jobs.ts`, `src/http/api/jobs.ts`, `src/web/studio/jobs.ts`, the no-retry answer in `src/apps/timer_fire.ts`, and the ask in `scheduleLines` (`src/actions/schedules.ts`). Tests: `test/jobs-doors.test.ts`.
What the forge could not do: `trevato/app-template` is a separate repository, which this change cannot push to. The bundle carries the new files, so a forge without that repository hands them out; a forge that reads the repository needs the same files (`src/jobs.ts`, `src/reminder.ts`, `src/jobs.test.ts`, the routes and button, the README section) landed there, and `scripts/take-template.ts` then replaces the bundle.
@@ -0,0 +1,154 @@
/**
 * An app's durable job runs, seen and steered (#11596). The runs live in the
 * app's own data (`src/jobs.ts` of the app template), so the forge asks the app, waking it if
 * it sleeps: a signed `GET /jobs/runs`, and a signed `POST /jobs/runs/:id/cancel`
 * or `…/retry`, signed as a timer's delivery is (`controlHeaders`). The page,
 * `GET …/jobs`, the MCP tool and the terminal read these functions, so they
 * say the same words. Writers only: a run's error and result are the app's
 * own text. Production's runs; a preview keeps its own in its own data.
 */
import type { App } from "../app.ts";
import { runtimeReach } from "../apps/timer_service.ts";
import { controlHeaders, type Reach } from "../apps/timer_fire.ts";
import { getAppByRepo } from "../core/apps.ts";
import type { Repo } from "../core/repos.ts";
import type { User } from "../core/users.ts";
import { err, ok, type Result } from "../kernel/result.ts";
import { requireWriter } from "./access.ts";
/** How the forge reaches the app: the real runtime, unless a test hands in its own. */
export const jobReach: { override: Reach | undefined } = { override: undefined };
const reachOf = (app: App): Reach => jobReach.override ?? runtimeReach(app);
export const JOB_STATUSES = ["running", "sleeping", "waiting", "done", "failed", "cancelled"] as const;
export type JobStatus = (typeof JOB_STATUSES)[number];
export interface JobRunRow {
  readonly id: string;
  readonly job: string;
  readonly key: string;
  readonly status: JobStatus;
  /** ms epoch: when a sleeping run wakes, or a waiting run's timeout. */
  readonly until: number | null;
  readonly event: string | null;
  readonly error: string | null;
  readonly failedStep: string | null;
  readonly started: number;
  readonly updated: number;
  /** A finished run's result, as text, cut at 500 characters. */
  readonly result: string | null;
  /** Where it is now, in words. */
  readonly now: string;
}
export type JobError = "not_found" | "unauthorized" | "forbidden" | "unreachable" | "not_now" | "invalid";
export const JOB_WORDS: Record<"unreachable" | "not_now" | "invalid", string> = {
  unreachable: "The app did not answer, so its job runs cannot be read or changed now. It may be asleep and failing to start; look at its deploy.",
  not_now: "That run is not in a state that allows it: only an unfinished run can be cancelled and only a failed one retried.",
  invalid: "The action is cancel or retry, on a run id.",
};
const iso = (ms: number): string => new Date(ms).toISOString().replace("T", " ").slice(0, 16) + " UTC";
const str = (v: unknown, max: number): string | null => (typeof v === "string" ? v.slice(0, max).replace(/[\x00-\x08\x0b\x0c\x0e-\x1f]/g, "?") : null);
const num = (v: unknown): number | null => (typeof v === "number" && Number.isFinite(v) ? v : null);
/** Where a run is now: the words the page, the tool and the terminal share. */
export function nowWords(r: Pick<JobRunRow, "status" | "until" | "event" | "error" | "failedStep">): string {
  switch (r.status) {
    case "sleeping":
      return r.until === null ? "sleeping" : `sleeping until ${iso(r.until)}${r.error === null ? "" : " (retrying a step)"}`;
    case "waiting":
      return `waiting for ${r.event ?? "an event"}${r.until === null ? "" : `, until ${iso(r.until)} at the latest`}`;
    case "failed":
      return `failed${r.failedStep === null ? "" : ` at ${r.failedStep}`}`;
    default:
      return r.status;
  }
}
/** One row of the app's answer, or null when it is not one: the app's text is never trusted for its shape. */
function rowOf(v: unknown): JobRunRow | null {
  if (typeof v !== "object" || v === null) return null;
  const o = v as Record<string, unknown>;
  const status = JOB_STATUSES.find((s) => s === o["status"]);
  const id = str(o["id"], 40);
  if (status === undefined || id === null || !/^[A-Za-z0-9-]{1,40}$/.test(id)) return null;
  const base = {
    id, job: str(o["job"], 60) ?? "", key: str(o["key"], 100) ?? "", status, until: num(o["until"]), event: str(o["event"], 200),
    error: str(o["error"], 500), failedStep: str(o["failedStep"], 100), started: num(o["started"]) ?? 0, updated: num(o["updated"]) ?? 0,
  };
  const result = o["result"] === null || o["result"] === undefined ? null : (str(typeof o["result"] === "string" ? o["result"] : JSON.stringify(o["result"]), 500));
  return { ...base, result, now: nowWords(base) };
}
/** What the app may answer with: a hundred rows of capped text fit well inside it. */
export const BODY_MAX = 1_000_000;
/** The body read as a stream with a byte cap; undefined (and the stream cancelled) once it passes the cap, so a large answer is never held in memory. */
export async function readCapped(res: Response, max: number): Promise<string | undefined> {
  const reader = res.body?.getReader();
  if (reader === undefined) return "";
  const chunks: Uint8Array[] = [];
  let n = 0;
  for (;;) {
    const { done, value } = await reader.read();
    if (done || value === undefined) break;
    n += value.length;
    if (n > max) {
      void reader.cancel().catch(() => undefined);
      return undefined;
    }
    chunks.push(value);
  }
  return new TextDecoder().decode(Buffer.concat(chunks));
}
async function ask(app: App, repo: Repo, method: "GET" | "POST", path: string, reach: Reach): Promise<Result<unknown, "not_found" | "unreachable" | "not_now">> {
  const record = getAppByRepo(app.db, repo.id);
  if (record === undefined) return err("not_found");
  const port = await reach.ensure(record, "production").catch(() => undefined);
  if (port === undefined) return err("unreachable");
  try {
    const res = await reach.fetch(`http://127.0.0.1:${port}${path}`, {
      method, headers: controlHeaders(reach.identityKey(record.id), Date.now()), redirect: "manual", signal: AbortSignal.timeout(10_000),
    });
    if (res.status === 409) return err("not_now");
    if (!res.ok) return err("unreachable");
    const body = await readCapped(res, BODY_MAX);
    return body === undefined ? err("unreachable") : ok(JSON.parse(body) as unknown);
  } catch {
    return err("unreachable");
  }
}
/** The runs, newest first. */
export async function jobRows(app: App, actor: User | undefined, repo: Repo, reach: Reach = reachOf(app)): Promise<Result<JobRunRow[], JobError>> {
  const writer = requireWriter(app, actor, repo);
  if (!writer.ok) return err(writer.error);
  const got = await ask(app, repo, "GET", "/jobs/runs?limit=100", reach);
  if (!got.ok) return err(got.error);
  const runs = (got.value as { runs?: unknown } | null)?.runs;
  return ok(Array.isArray(runs) ? runs.slice(0, 100).map(rowOf).filter((r): r is JobRunRow => r !== null) : []);
}
export type JobAction = "cancel" | "retry";
export async function steerJob(app: App, actor: User | undefined, repo: Repo, id: string, action: string, reach: Reach = reachOf(app)): Promise<Result<true, JobError>> {
  const writer = requireWriter(app, actor, repo);
  if (!writer.ok) return err(writer.error);
  if ((action !== "cancel" && action !== "retry") || !/^[A-Za-z0-9-]{1,40}$/.test(id)) return err("invalid");
  const done = await ask(app, repo, "POST", `/jobs/runs/${id}/${action}`, reach);
  if (!done.ok) return err(done.error);
  // The same event a schedule's steering is, so every door already listening redraws.
  app.bus.append({ kind: "schedule.steered", repoId: repo.id, actorId: writer.value.id, payload: { name: `job:${id}`, env: "production", action } });
  return ok(true);
}
/** How many job timers the app holds: what says an app has jobs without waking it. */
export function jobTimers(app: App, repo: Repo): number {
  const record = getAppByRepo(app.db, repo.id);
  if (record === undefined) return 0;
  return app.db.query<{ n: number }, [string]>("SELECT COUNT(*) AS n FROM timer WHERE app_id = ?1 AND kind = 'once' AND key LIKE 'job:%'").get(record.id)?.n ?? 0;
}
@@ -10,7 +10,7 @@
import type { App } from "../app.ts";
import { getAppByRepo } from "../core/apps.ts";
import { canAdmin, listReposWritableTo, type Repo } from "../core/repos.ts";
import type { User } from "../core/users.ts";
import { failingNow, runNow, runsOf, scheduleTimer, scheduleTimers, setPaused, type TimerRun } from "../core/timers.ts";
import { failedJobs, failingNow, runNow, runsOf, scheduleTimer, scheduleTimers, setPaused, type TimerRun } from "../core/timers.ts";
import { err, ok, type Result } from "../kernel/result.ts";
import { requireReader, requireWriter } from "./access.ts";
@@ -108,7 +108,7 @@
export function steerSchedule(app: App, actor: User | undefined, repo: Repo, nam
 * with the reason. The item is the schedule, not the run, so a schedule that
 * keeps failing is one ask, not one a day (`crew/notify.ts` says only what is new).
 */
export function scheduleLines(app: App, user: User): { line: string; items: string[]; link: string }[] {
export function scheduleLines(app: App, user: User, now = Date.now()): { line: string; items: string[]; link: string }[] {
  if (user.kind !== "user") return [];
  const out: { line: string; items: string[]; link: string }[] = [];
  for (const repo of listReposWritableTo(app.db, user, { limit: 200, withoutArchived: true })) {
@@ -118,6 +118,14 @@
export function scheduleLines(app: App, user: User): { line: string; items: stri
      const why = run.status === "failed" ? `Schedule ${run.name} failed: ${run.error}` : run.error;
      out.push({ line: `${repo.name}: ${why}`, items: [`schedule:${record.id}:${run.env}:${run.name}`], link: `/${repo.ownerName}/${repo.name}#schedules` });
    }
    // A durable job that failed for good is one ask, named by its run (`job:<run>:<wake>`), never one per wake.
    const seen = new Set<string>();
    for (const run of failedJobs(app.db, record.id, now)) {
      const id = run.name.split(":")[1] ?? run.name;
      if (seen.has(id)) continue;
      seen.add(id);
      out.push({ line: `${repo.name}: a job run failed: ${run.error ?? "no reason given"}`, items: [`job:${record.id}:${id}`], link: `/${repo.ownerName}/${repo.name}/app/jobs` });
    }
  }
  return out;
}
@@ -63,6 +63,23 @@
export function deliveryHeaders(key: string, t: Timer, attempt: number, expires:
  };
}
/** The `error` an app's JSON answer names, when it names one. */
function failureText(head: string): string | undefined {
  try {
    const e = (JSON.parse(head) as { error?: unknown }).error;
    return typeof e === "string" && e !== "" ? e : undefined;
  } catch {
    return undefined;
  }
}
/** The headers of a request the forge makes of an app outside a timer (listing or steering its jobs): the same signature, an idempotency key of its own. */
export function controlHeaders(key: string, now: number): Record<string, string> {
  const idem = `control:${crypto.randomUUID()}`;
  const identity = `forge-timer|${idem}|timer|${now + 60_000}`;
  return { "idempotency-key": idem, "x-forge-identity": identity, "x-forge-signature": createHmac("sha256", key).update(identity).digest("base64url") };
}
async function readHead(res: Response): Promise<string> {
  const reader = res.body?.getReader();
  if (reader === undefined) return "";
@@ -78,7 +95,7 @@
async function readHead(res: Response): Promise<string> {
  return new TextDecoder().decode(Buffer.concat(chunks)).slice(0, RESPONSE_KEPT).replace(/[\x00-\x08\x0b\x0c\x0e-\x1f]/g, "?");
}
interface Attempt { readonly ok: boolean; readonly status: number | null; readonly response: string | null; readonly error: string | null }
interface Attempt { readonly final?: boolean; readonly ok: boolean; readonly status: number | null; readonly response: string | null; readonly error: string | null }
async function attemptOnce(reach: Reach, app: AppRecord, t: Timer, attempt: number, now: () => number): Promise<Attempt> {
  const port = await reach.ensure(app, t.env).catch(() => undefined);
@@ -94,6 +111,8 @@
async function attemptOnce(reach: Reach, app: AppRecord, t: Timer, attempt: numb
    });
    const head = await readHead(res);
    if (res.status >= 200 && res.status < 300) return { ok: true, status: res.status, response: head, error: null };
    // An app that says `x-forge-no-retry` has decided: delivering again would only repeat it (a durable job that failed, #11596).
    if (res.headers.get("x-forge-no-retry") === "1") return { ok: false, final: true, status: res.status, response: head, error: failureText(head) ?? `the app answered ${res.status}` };
    return { ok: false, status: res.status, response: head, error: `the app answered ${res.status}` };
  } catch (e) {
    const timedOut = e instanceof Error && (e.name === "TimeoutError" || e.name === "AbortError");
@@ -153,10 +172,10 @@
export async function fireTimer(db: Database, reach: Reach, t: Timer, now: () =>
      finishTimer(db, t, next);
      return;
    }
    if (attempt <= t.retries) return retryTimer(db, t.id, finished + backoffFor(t.attempt), attempt);
    if (result.final !== true && attempt <= t.retries) return retryTimer(db, t.id, finished + backoffFor(t.attempt), attempt);
    recordRun(db, {
      ...base, startedAt: started, finishedAt: finished, attempts: attempt, status: "failed", httpStatus: result.status, response: result.response,
      error: `${result.error ?? "failed"}, after ${attempt} ${attempt === 1 ? "attempt" : "attempts"}`, ...late, durationMs: finished - started,
      error: result.final === true ? (result.error ?? "failed") : `${result.error ?? "failed"}, after ${attempt} ${attempt === 1 ? "attempt" : "attempts"}`, ...late, durationMs: finished - started,
    });
    finishTimer(db, t, next);
    if (t.kind === "schedule" && failingDays(runsOf(db, t.appId, t.name, 30).filter((r) => r.env === t.env)) >= PAUSE_AFTER_DAYS) {
@@ -0,0 +1,9 @@
import { REPO, type Route } from "../shared.ts";
export const route: Route = {
  method: "get",
  path: `${REPO}/jobs`,
  summary: "An app's durable job runs, newest first: job, key, status (running, sleeping, waiting, done, failed, cancelled), where it is now, when it wakes, the failed step and error, a finished run's result; read from the app itself, waking it; a writer's; 404 for anyone else, 502 when the app does not answer",
  auth: true,
  role: "member",
};
@@ -162,6 +162,7 @@
export { route as getApiV1ReposOwnerNameIssuesNumberDependencies } from "./getAp
export { route as getApiV1ReposOwnerNameIssuesNumberPlan } from "./getApiV1ReposOwnerNameIssuesNumberPlan.ts";
export { route as getApiV1ReposOwnerNameIssuesNumberTeams } from "./getApiV1ReposOwnerNameIssuesNumberTeams.ts";
export { route as getApiV1ReposOwnerNameIterations } from "./getApiV1ReposOwnerNameIterations.ts";
export { route as getApiV1ReposOwnerNameJobs } from "./getApiV1ReposOwnerNameJobs.ts";
export { route as getApiV1ReposOwnerNameLabels } from "./getApiV1ReposOwnerNameLabels.ts";
export { route as getApiV1ReposOwnerNameMap } from "./getApiV1ReposOwnerNameMap.ts";
export { route as getApiV1ReposOwnerNameMilestones } from "./getApiV1ReposOwnerNameMilestones.ts";
@@ -343,6 +344,7 @@
export { route as postApiV1ReposOwnerNameIssuesNumberComments } from "./postApiV
export { route as postApiV1ReposOwnerNameIssuesNumberDependencies } from "./postApiV1ReposOwnerNameIssuesNumberDependencies.ts";
export { route as postApiV1ReposOwnerNameIssuesNumberDependenciesBlockerConfirm } from "./postApiV1ReposOwnerNameIssuesNumberDependenciesBlockerConfirm.ts";
export { route as postApiV1ReposOwnerNameIssuesNumberPrivate } from "./postApiV1ReposOwnerNameIssuesNumberPrivate.ts";
export { route as postApiV1ReposOwnerNameJobsIdAction } from "./postApiV1ReposOwnerNameJobsIdAction.ts";
export { route as postApiV1ReposOwnerNameLabels } from "./postApiV1ReposOwnerNameLabels.ts";
export { route as postApiV1ReposOwnerNameMakeItYours } from "./postApiV1ReposOwnerNameMakeItYours.ts";
export { route as postApiV1ReposOwnerNameMilestonesNumberLimb } from "./postApiV1ReposOwnerNameMilestonesNumberLimb.ts";
@@ -0,0 +1,9 @@
import { REPO, type Route } from "../shared.ts";
export const route: Route = {
  method: "post",
  path: `${REPO}/jobs/{id}/{action}`,
  summary: "Steer a job run: cancel an unfinished one, or retry a failed one from its failed step (rw scope, writer); 422 when the run is not in a state that allows it or another action, 502 when the app does not answer",
  auth: true,
  role: "member",
};
@@ -9,6 +9,7 @@
const ALIAS: Readonly<Record<string, string>> = {
  app: "apps",
  timers: "apps",
  schedules: "apps",
  jobs: "apps",
  "app-templates": "apps",
  display: "builds",
  following: "forks",
@@ -247,6 +247,18 @@
export function failingNow(db: Database, appId: string): TimerRun[] {
  return out;
}
/**
 * Job runs (#11596) the app failed for good in the last `days` days: one per run, newest first.
 * A durable job's timers are named `job:<run>:<wake>`; the app answers a failed run with
 * `x-forge-no-retry`, which records it as failed at once.
 */
export function failedJobs(db: Database, appId: string, now: number, days = 3, limit = 20): TimerRun[] {
  const rows = db
    .query<Row, [string, number, number]>("SELECT * FROM timer_run WHERE app_id = ?1 AND name LIKE 'job:%' AND status = 'failed' AND finished_at > ?2 ORDER BY id DESC LIMIT ?3")
    .all(appId, now - days * 86_400_000, limit);
  return rows.map(toRun);
}
/** Drop the schedule rows of environments that are gone (a closed pull's preview). */
export function dropGoneEnvs(db: Database, appId: string, live: ReadonlySet<string>): void {
  for (const t of scheduleTimers(db, appId)) if (!live.has(t.env)) db.query("DELETE FROM timer WHERE id = ?1").run(t.id);
@@ -354,6 +354,7 @@
import { passportRoutes } from "./passport_edge.ts";
import { egressGrantRoutes } from "./api/egress_grants.ts";
import { printRoutes } from "./api/print.ts";
import { timersRoutes } from "./api/timers.ts";
import { jobsRoutes } from "./api/jobs.ts";
import { schedulesRoutes } from "./api/schedules.ts";
import { forgeRoutes } from "./api/forges.ts";
import { packRoutes } from "./api/packs.ts";
@@ -3862,6 +3863,7 @@
export function apiRoutes(router: Router<Handler>): void {
  printRoutes(router);
  timersRoutes(router);
  schedulesRoutes(router);
  jobsRoutes(router);
  promptRoutes(router);
@@ -0,0 +1,33 @@
/**
 * `GET /api/v1/repos/:owner/:name/jobs` and `POST …/jobs/:id/(cancel|retry)`
 * (#11596): an app's durable job runs and the two things a writer may do to
 * one, the same answers as the app's page and the MCP tool (`actions/jobs.ts`).
 */
import { JOB_WORDS, jobRows, steerJob } from "../../actions/jobs.ts";
import { loadRepo } from "../api.ts";
import type { Handler } from "../context.ts";
import { apiError, json, notFound, requireScope, respondAccess } from "../respond.ts";
import type { Router } from "../router.ts";
export function jobsRoutes(router: Router<Handler>): void {
  router.add("GET", "/api/v1/repos/:owner/:name/jobs", async (ctx) => {
    const repo = loadRepo(ctx);
    if (!repo) return notFound();
    const rows = await jobRows(ctx.app, ctx.auth.user, repo);
    if (!rows.ok) return rows.error === "not_found" ? notFound() : rows.error === "unreachable" ? apiError("unreachable", 502, JOB_WORDS.unreachable) : respondAccess(rows.error, 422);
    return json({ items: rows.value, next_cursor: null });
  });
  router.add("POST", "/api/v1/repos/:owner/:name/jobs/:id/:action", async (ctx) => {
    const denied = requireScope(ctx, "rw");
    if (denied) return denied;
    const repo = loadRepo(ctx);
    if (!repo) return notFound();
    const done = await steerJob(ctx.app, ctx.auth.user, repo, ctx.params["id"] ?? "", ctx.params["action"] ?? "");
    if (done.ok) return json({ run: ctx.params["id"], action: ctx.params["action"] }, 200);
    if (done.error === "not_found") return notFound();
    if (done.error === "unreachable") return apiError("unreachable", 502, JOB_WORDS.unreachable);
    if (done.error === "not_now" || done.error === "invalid") return apiError(done.error, 422, JOB_WORDS[done.error]);
    return respondAccess(done.error, 422);
  });
}
@@ -1,21 +1,22 @@
import { envRefusalText } from "../../../actions/env_words.ts";
import { pullNow, PULL_NOW_REFUSALS } from "../../../actions/app_pull.ts";
import { redeployEnv, stopEnv } from "../../../actions/apps.ts";
import { JOB_WORDS, jobRows, steerJob } from "../../../actions/jobs.ts";
import { scheduleRows, steerSchedule } from "../../../actions/schedules.ts";
import { good, bad, type Tool, str, REPO_ARG, repoArg, isAnswer } from "../shared.ts";
export const tool: Tool = {
  name: "app_deploy",
  doc: "`{ repo, env, action }`: `redeploy` or `stop` an app's `production` or a pull's `pull-N` preview; a writer's, queued as `POST …/app/:env/redeploy` and `…/stop` are; answers `{ queued, run }` or `{ stopped }`. `{ repo, action: \"pull\" }` runs an adapter's pull now as `POST …/app/pull` does; answers `{ run }`. `{ repo, action: \"schedules\" }` lists the app's schedules (`{ items }`, as `GET …/schedules`); `run_schedule`, `pause_schedule` or `resume_schedule` with `name` steers one, a writer's, as `POST …/schedules/:name/:action` does",
  doc: "`{ repo, env, action }`: `redeploy` or `stop` an app's `production` or a pull's `pull-N` preview; a writer's, queued as `POST …/app/:env/redeploy` and `…/stop` are; answers `{ queued, run }` or `{ stopped }`. `{ repo, action: \"pull\" }` runs an adapter's pull now as `POST …/app/pull` does; answers `{ run }`. `{ repo, action: \"schedules\" }` lists the app's schedules (`{ items }`, as `GET …/schedules`); `run_schedule`, `pause_schedule` or `resume_schedule` with `name` steers one, a writer's, as `POST …/schedules/:name/:action` does. `jobs` lists the durable job runs (`{ items }`, as `GET …/jobs`); `cancel_job` or `retry_job` with `name` the run id steers one, as `POST …/jobs/:id/:action` does",
  description:
    "Redeploy or stop production or a pull-N preview, pull an adapter now, or list and steer the app's schedules.",
    "Redeploy or stop an env, pull, list and steer schedules and jobs.",
  inputSchema: {
    type: "object",
    properties: {
      repo: REPO_ARG,
      env: { type: "string" },
      name: { type: "string" },
      action: { type: "string", enum: ["redeploy", "stop", "pull", "schedules", "run_schedule", "pause_schedule", "resume_schedule"] },
      action: { type: "string", enum: ["redeploy", "stop", "pull", "schedules", "run_schedule", "pause_schedule", "resume_schedule", "jobs", "cancel_job", "retry_job"] },
    },
    required: ["repo", "action"],
  },
@@ -33,12 +34,20 @@
export const tool: Tool = {
      const rows = scheduleRows(ctx.app, user, repo);
      return rows.ok ? good({ items: rows.value }) : bad("not_found", "No such app.");
    }
    if (action === "jobs") {
      const rows = await jobRows(ctx.app, user, repo);
      return rows.ok ? good({ items: rows.value }) : bad(rows.error, rows.error === "unreachable" ? JOB_WORDS.unreachable : "No such app, or you may not read its jobs.");
    }
    if (action === "cancel_job" || action === "retry_job") {
      const done = await steerJob(ctx.app, user, repo, str(args["name"]) ?? "", action.slice(0, 5));
      return done.ok ? good({ done: true }) : bad(done.error, done.error === "unreachable" || done.error === "not_now" || done.error === "invalid" ? JOB_WORDS[done.error] : "No such app, or you may not steer its jobs.");
    }
    if (action === "run_schedule" || action === "pause_schedule" || action === "resume_schedule") {
      const done = steerSchedule(ctx.app, user, repo, str(args["name"]) ?? "", env || "production", action.slice(0, action.indexOf("_")) as "run" | "pause" | "resume");
      return done.ok ? good({ done: true }) : bad(done.error, done.error === "forbidden" ? "A writer steers a schedule." : "No such schedule.");
    }
    if (action !== "redeploy" && action !== "stop")
      return bad("invalid_action", "action is redeploy, stop, pull, schedules, run_schedule, pause_schedule or resume_schedule");
      return bad("invalid_action", "action is redeploy, stop, pull, schedules, run_schedule, pause_schedule, resume_schedule, jobs, cancel_job or retry_job");
    if (action === "stop") {
      const stopped = await stopEnv(ctx.app, user, repo, env);
      return stopped.ok
@@ -240,6 +240,7 @@
import {
  newIssueSubmit,
} from "../web/pages/issues.ts";
import { adapterPull } from "../web/studio/adapter.ts";
import { jobsPage, steerJobPage } from "../web/studio/jobs.ts";
import { steerSchedulePage } from "../web/studio/schedules.ts";
import { noteModerate } from "../web/studio/notes.ts";
import { tellPage } from "../web/studio/tell.ts";
@@ -1410,6 +1411,8 @@
export function buildRouter(): Router<Handler> {
  router.add("GET", "/apps/open", appsOpen);
  router.add("POST", "/:owner/:name/app/notes", noteModerate);
  router.add("POST", "/:owner/:name/app/pull", adapterPull);
  router.add("GET", "/:owner/:name/app/jobs", jobsPage);
  router.add("POST", "/:owner/:name/app/jobs/:id/:action", steerJobPage);
  router.add("POST", "/:owner/:name/app/schedules/:schedule/:action", steerSchedulePage);
  router.add("POST", "/:owner/:name/app/:env/redeploy", appRedeploy);
  router.add("POST", "/:owner/:name/app/:env/stop", appStop);
@@ -0,0 +1,87 @@
/**
 * An app's durable job runs, in the terminal (#11596): the page's rows
 * (`web/studio/jobs.ts`) from the same actions (`actions/jobs.ts`), gated the
 * same way: a writer reads and steers. The runs live in the app itself, so
 * the screen asks it (waking it) on open and after each write, and keeps the
 * answer; `c` cancels the run under the cursor, `t` retries a failed one from
 * its failed step, `g` reads again. A write is an event on the bus, so an
 * open web page redraws.
 */
import { requireWriter } from "../../actions/access.ts";
import { JOB_WORDS, jobRows, jobTimers, steerJob, type JobError, type JobRunRow } from "../../actions/jobs.ts";
import type { Repo } from "../../core/repos.ts";
import type { Ctx, Screen } from "../app/app.ts";
import type { Command } from "../app/palette.ts";
import { binding } from "../app/keys.ts";
import { style } from "../term/style.ts";
import { truncate } from "../term/width.ts";
import type { Session } from "../session.ts";
import { fill, len, split } from "../ui/layout.ts";
import { ListState, drawList, type ListRow } from "./common.ts";
export const JOB_TUI_WORDS: Record<JobError, string> = {
  ...JOB_WORDS,
  not_found: "No such app.",
  unauthorized: "Sign in to see job runs.",
  forbidden: "A writer reads and steers job runs.",
};
export function jobsScreen(f: Session, ctx: Ctx, repo: Repo): Screen {
  const state = new ListState();
  let rows: JobRunRow[] = [];
  let note = "Reading the runs from the app…";
  const read = async (): Promise<void> => {
    const got = await jobRows(f.app, f.user, repo);
    rows = got.ok ? got.value : [];
    note = got.ok ? (rows.length === 0 ? "This app has no job runs." : "c cancels an unfinished run, t retries a failed one, g reads again.") : JOB_TUI_WORDS[got.error];
    ctx.invalidate();
  };
  const steer = async (action: "cancel" | "retry"): Promise<void> => {
    const at = rows[state.index];
    if (at === undefined) return;
    const done = await steerJob(f.app, f.user, repo, at.id, action);
    ctx.toast(done.ok ? `${at.job} ${at.key}: ${action === "cancel" ? "cancelled" : "retrying"}.` : JOB_TUI_WORDS[done.error], done.ok ? undefined : "danger");
    await read();
  };
  return {
    title: `${repo.ownerName}/${repo.name} — Job runs`,
    keys: [binding(["j", "k"], "move cursor"), binding(["c"], "cancel"), binding(["t"], "retry from the failed step"), binding(["g"], "read again")],
    async load() {
      await read();
    },
    render(area, s) {
      const c = ctx.theme.c;
      const [head, list] = split(area, "v", [len(1), fill(1)]);
      s.text(head!.x, head!.y, truncate(note, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w);
      const lines: ListRow[] = rows.map((r) => ({
        title: `${r.job} ${r.key}`,
        meta: r.now,
        ...(r.error === null ? {} : { note: r.error }),
      }));
      drawList(s, list!, c, state, lines);
    },
    key(e) {
      if (state.key(e, rows.length)) return true;
      if (e.ctrl || e.alt) return false;
      if (e.key === "g") void read();
      else if (e.key === "c") void steer("cancel");
      else if (e.key === "t") void steer("retry");
      else return false;
      return true;
    },
  };
}
/** The palette's command while a repository whose app holds job timers is open; a writer's only. */
export function jobsCommands(f: Session, ctx: Ctx, repo: Repo | undefined): Command[] {
  if (repo === undefined || !requireWriter(f.app, f.user, repo).ok || jobTimers(f.app, repo) === 0) return [];
  return [
    {
      id: "jobs-app",
      title: `${repo.ownerName}/${repo.name} — Job runs`,
      hint: "this app's durable job runs: cancel one, retry a failed one",
      run: () => ctx.push(jobsScreen(f, ctx, repo)),
    },
  ];
}
@@ -171,6 +171,7 @@
import { autonomyCommands } from "./autonomy.ts";
import { trafficCommands } from "./trafficview.ts";
import { chaptersCommands } from "./chapters.ts";
import { schedulesCommands } from "./schedules.ts";
import { jobsCommands } from "./jobs.ts";
import { queueCommands } from "./queuegrowth.ts";
import { blockedCommands } from "./blocked.ts";
import { statementCommands } from "./statement.ts";
@@ -1996,6 +1997,7 @@
export function workspaceScreen(
      ...autonomyCommands(f, ctx, repo),
      ...trafficCommands(f, ctx, repo),
      ...schedulesCommands(f, ctx, repo),
      ...jobsCommands(f, ctx, repo),
      ...chaptersCommands(f, ctx, repo),
      ...queueCommands(f, ctx, repo),
      ...blockedCommands(f, ctx, repo),
@@ -60,6 +60,7 @@
import { nowCard, nowRegion } from "./now.ts";
import { repoAsks } from "./asks.ts";
import { notesCard } from "./notes.ts";
import { adapterCard } from "./adapter.ts";
import { jobsLink } from "./jobs.ts";
import { schedulesCard } from "./schedules.ts";
import { appPanels } from "./previews.ts";
import { noCrewLine, pausedWords, servingHistory, type WordCtx } from "./words.ts";
@@ -495,7 +496,7 @@
export async function studioApp(
  const adapter = ownerFacts
    ? await timePhaseAsync(req, "adapter", () => adapterCard(ctx, record, repo))
    : null;
  const schedules = ownerFacts ? schedulesCard(ctx, repo) : null;
  const schedules = ownerFacts ? html`${schedulesCard(ctx, repo)}${jobsLink(ctx, repo)}` : null;
  const rendering = performance.now();
  const tellRegion = html`${tell}
        ${ownerFacts && ctx.auth.user ? answersRegion(ctx, repo, ctx.auth.user) : null}`;
@@ -0,0 +1,86 @@
/**
 * An app's durable job runs on the forge (#11596): `/:owner/:name/app/jobs`
 * lists each run (name, key, started, where it is now: running, sleeping until,
 * waiting for, done, failed) and, for a writer, **Cancel** on an unfinished run
 * and **Retry from the failed step** on a failed one. The words are `jobRows`
 * in `actions/jobs.ts`, shared with the API and the MCP tool; this module adds
 * markup. The runs are read from the app itself, so the page wakes it; the app
 * page only links here, and only when the app holds a job timer. A run's text
 * is the app's own and goes through `html`, which escapes it. The forms POST to
 * `/:owner/:name/app/jobs/:id/:action`, same-origin checked by `readForm`.
 */
import { requireWriter } from "../../actions/access.ts";
import { JOB_WORDS, jobRows, jobTimers, steerJob, type JobRunRow } from "../../actions/jobs.ts";
import type { Repo } from "../../core/repos.ts";
import { notePhase, takeSessionTime, timePhaseAsync } from "../../core/route_timing.ts";
import type { Ctx } from "../../http/context.ts";
import { crumbsFor, loadRepo, notFoundPage, pageHead, repoBase, signInFirst } from "../chrome.ts";
import { readForm } from "../form.ts";
import { html, type Html } from "../html.ts";
import { page, redirect } from "../layout.ts";
import { badge, btn, empty, time, type Tone } from "../ui.ts";
const TONE: Record<JobRunRow["status"], Tone> = { running: "attn", sleeping: "default", waiting: "default", done: "success", failed: "danger", cancelled: "default" };
/** A line on the app page, when the app has job timers pending: where to see the runs. */
export function jobsLink(ctx: Ctx, repo: Repo): Html | null {
  if (!requireWriter(ctx.app, ctx.auth.user, repo).ok) return null;
  const n = jobTimers(ctx.app, repo);
  if (n === 0) return null;
  return html`<section class="jobs" id="jobs"><h2>Jobs</h2>
    <p>${String(n)} ${n === 1 ? "run is" : "runs are"} waiting on a timer. <a href="${repoBase(repo)}/app/jobs">See the job runs</a>.</p></section>`;
}
function runRow(base: string, r: JobRunRow): Html {
  const act = (action: string, label: string): Html => html`<form method="post" action="${`${base}/app/jobs/${r.id}/${action}`}">${btn(label, { size: "sm", variant: "outline" })}</form>`;
  return html`<tr>
    <td><code>${r.job}</code></td>
    <td class="break"><code>${r.key}</code></td>
    <td>${r.started === 0 ? null : time(new Date(r.started).toISOString())}</td>
    <td>${badge(r.status, { tone: TONE[r.status] })} <span class="break">${r.now}</span>
      ${r.error === null ? null : html`<br><span class="muted break">${r.error}</span>`}
      ${r.result === null ? null : html`<br><span class="muted break">${r.result}</span>`}</td>
    <td>${r.status === "failed" ? act("retry", "Retry from the failed step") : r.status === "running" || r.status === "sleeping" || r.status === "waiting" ? act("cancel", "Cancel") : null}</td>
  </tr>`;
}
// GET /:owner/:name/app/jobs
export async function jobsPage(ctx: Ctx): Promise<Response> {
  const repo = loadRepo(ctx);
  if (!repo) return notFoundPage(ctx);
  if (ctx.auth.user === undefined) return signInFirst(ctx);
  notePhase(ctx.request, "session", takeSessionTime(ctx.request));
  const rows = await timePhaseAsync(ctx.request, "read", () => jobRows(ctx.app, ctx.auth.user!, repo));
  if (!rows.ok && (rows.error === "not_found" || rows.error === "forbidden" || rows.error === "unauthorized")) {
    notePhase(ctx.request, "render", 0);
    return notFoundPage(ctx);
  }
  const base = repoBase(repo);
  const body = !rows.ok
    ? empty({ icon: "alert", title: "The job runs could not be read", note: rows.error === "unreachable" ? JOB_WORDS.unreachable : "Try again in a moment." })
    : rows.value.length === 0
      ? empty({ icon: "clock", title: "No job runs yet", note: "A run appears here when the app starts one with `job(…).start(…)`." })
      : html`<table class="table"><thead><tr><th>Job</th><th>Key</th><th>Started</th><th>Where it is now</th><th></th></tr></thead>
          <tbody>${rows.value.map((r) => runRow(base, r))}</tbody></table>`;
  const rendering = performance.now();
  const out = page({
    ctx,
    title: `Jobs · ${repo.name}`,
    crumbs: crumbsFor(repo),
    body: html`${pageHead("Job runs")}<section data-live="schedule.steered,run.finished" data-live-repo="${repo.ownerName}/${repo.name}">${body}</section>`,
  });
  notePhase(ctx.request, "render", performance.now() - rendering);
  return out;
}
// POST /:owner/:name/app/jobs/:id/:action — then back to the list.
export async function steerJobPage(ctx: Ctx): Promise<Response> {
  const repo = loadRepo(ctx);
  if (!repo || !requireWriter(ctx.app, ctx.auth.user, repo).ok) return notFoundPage(ctx);
  const form = await readForm(ctx, "rw");
  if (form instanceof Response) return form;
  const done = await steerJob(ctx.app, ctx.auth.user, repo, ctx.params["id"] ?? "", ctx.params["action"] ?? "");
  const why = done.ok ? undefined : done.error === "unreachable" || done.error === "not_now" || done.error === "invalid" ? JOB_WORDS[done.error] : "Not allowed.";
  return redirect(`${repoBase(repo)}/app/jobs`, { url: ctx.url, forgeUrl: ctx.app.config.url, ...(why === undefined ? {} : { flash: { tone: "danger", message: why } }) });
}
@@ -0,0 +1,22 @@
/**
 * The app template's files, as a new app gets them (`templates/app-template.json`),
 * written to a directory so the forge's tests can import and run the very text it
 * hands out (#11596: the job runtime and its own tests live in the template).
 */
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
const bundle = JSON.parse(readFileSync(new URL("../../templates/app-template.json", import.meta.url), "utf8")) as { files: Record<string, string> };
/** The template written under a fresh directory; the path of that directory. */
export function writeTemplate(): string {
  const dir = mkdtempSync(join(process.env["TMPDIR"] ?? tmpdir(), "forge-tpl-"));
  for (const [path, body] of Object.entries(bundle.files)) {
    mkdirSync(dirname(join(dir, path)), { recursive: true });
    writeFileSync(join(dir, path), body);
  }
  return dir;
}
export const TEMPLATE_DIR = writeTemplate();
@@ -0,0 +1,259 @@
/**
 * Durable job runs through the forge's doors (#11596): the page, the API and
 * the MCP tool read the runs from the app itself, signed as a timer's delivery
 * is, and steer them with Cancel and Retry; a writer only. The app here is the
 * template's own runtime (`src/jobs.ts` in `templates/app-template.json`) behind a fake reach that
 * checks the signature the way `once.ts`'s `fromForge` does. A job that failed
 * for good asks the owner once, however many wakes it had.
 */
import { createHmac } from "node:crypto";
import { Database } from "bun:sqlite";
import { afterAll, beforeAll, describe, expect, test } from "bun:test";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createApp, type App } from "../src/app.ts";
import { createRepo } from "../src/actions/repos.ts";
import { JOB_WORDS, BODY_MAX, jobReach, jobRows, readCapped, steerJob } from "../src/actions/jobs.ts";
import { scheduleLines } from "../src/actions/schedules.ts";
import { fireTimer, type Reach } from "../src/apps/timer_fire.ts";
import { TEMPLATE_DIR } from "./helpers/template-app.ts";
import { createApp as insertApp } from "../src/core/apps.ts";
import type { Repo } from "../src/core/repos.ts";
import { createToken } from "../src/core/tokens.ts";
import { armTimer, dueTimers } from "../src/core/timers.ts";
import { createUser, type User } from "../src/core/users.ts";
import { serve, type Servers } from "../src/http/server.ts";
import { jobsCommands, jobsScreen } from "../src/tui/screens/jobs.ts";
import type { Ctx as TuiCtx } from "../src/tui/app/app.ts";
import type { Session } from "../src/tui/session.ts";
import { promote } from "./helpers/members.ts";
// The app's runtime is the template's own file, as a new app carries it; tsc cannot see into it, so what these tests use is named here.
interface Run { id: string; key: string; status: string }
interface Jobs {
  job(name: string, fn: (ctx: any, input: any) => Promise<unknown>): unknown;
  start(name: string, input: unknown, key?: string): Promise<Run>;
  deliver(id: string, wake?: number): Promise<string>;
  get(id: string): Run | undefined;
  list(): Run[];
}
const { Jobs, jobsRoute } = (await import(`${TEMPLATE_DIR}/src/jobs.ts`)) as { Jobs: new (db: Database, o: unknown) => Jobs; jobsRoute: (j: Jobs, r: Request, u: URL, good: boolean) => Promise<Response> };
const KEY = "an-identity-key";
const DAY = 86_400_000;
describe("job runs over the page, the API and the MCP tool", () => {
  let dataDir: string;
  let app: App;
  let servers: Servers;
  let cy: User;
  let dee: User;
  let repo: Repo;
  let appId: string;
  let ownerRw: string;
  let readerRo: string;
  let cookie: string;
  let jobs: Jobs;
  let appAsleep = false;
  const clock = Date.parse("2026-11-12T00:00:00Z");
  // The app: the template's jobs behind the signature check, no process.
  const reach: Reach = {
    ensure: async () => (appAsleep ? undefined : 4321),
    identityKey: () => KEY,
    fetch: (async (input: string | URL, init?: RequestInit) => {
      const req = new Request(String(input), init);
      const identity = req.headers.get("x-forge-identity") ?? "";
      const [who, signed, , expires] = identity.split("|");
      const good = createHmac("sha256", KEY).update(identity).digest("base64url") === req.headers.get("x-forge-signature") && who === "forge-timer" && signed === req.headers.get("idempotency-key") && Number(expires) > clock;
      return jobsRoute(jobs, req, new URL(req.url), good);
    }) as typeof fetch,
  };
  const call = (method: string, path: string, token: string | undefined, body?: unknown) =>
    fetch(`${servers.url}${path}`, { method, headers: { connection: "close", ...(token ? { authorization: `Bearer ${token}` } : {}), ...(body === undefined ? {} : { "content-type": "application/json" }) }, ...(body === undefined ? {} : { body: JSON.stringify(body) }) });
  beforeAll(async () => {
    dataDir = await mkdtemp(join(tmpdir(), "forge-jobs-"));
    app = createApp({ dataDir, port: 0, host: "forge.test" });
    servers = await serve(app);
    const make = async (username: string, adminIfFirst = false) => {
      const made = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst });
      if (!made.ok) throw new Error(made.error);
      return made.value;
    };
    await make("ada", true);
    cy = await make("cy");
    promote(app.db, cy);
    dee = await make("dee");
    const made = await createRepo(app, cy, { name: "shop", visibility: "public", description: "" });
    if (!made.ok) throw new Error(String(made.error));
    repo = made.value;
    const row = insertApp(app.db, { repoId: repo.id, slug: "shop", access: "public" });
    if (!row.ok) throw new Error(row.error);
    appId = row.value.id;
    ownerRw = createToken(app.db, cy.id, "me", "rw").token;
    readerRo = createToken(app.db, dee.id, "me", "ro").token;
    const r = await fetch(`${servers.url}/signin`, { method: "POST", body: new URLSearchParams({ username: "cy", password: "a-good-passphrase" }), redirect: "manual" });
    cookie = `${(r.headers.get("set-cookie") ?? "").split(";")[0]!}; forge_mode=dev`;
    jobReach.override = reach;
    // The app's runs: one sleeping, one waiting, one failed for good.
    jobs = new Jobs(new Database(":memory:"), { now: () => clock, timers: { arm: async () => undefined, cancel: async () => undefined } });
    jobs.job("remind", async (ctx) => {
      await ctx.step("load", () => 1);
      await ctx.sleep("wait", { days: 3 });
    });
    jobs.job("order", async (ctx) => ctx.waitFor("reply", { event: "reply:1" }));
    jobs.job("pay", async (ctx) => ctx.step("charge", () => { throw new Error("card <b>declined</b>"); }, { retries: 0 }));
  }, 60_000);
  afterAll(async () => {
    jobReach.override = undefined;
    servers?.stop();
    app?.close();
    await rm(dataDir, { recursive: true, force: true });
  });
  test("a writer reads each run and where it is now; a reader and a guest are refused", async () => {
    const a = await jobs.start("remind", {}, "remind:1");
    const b = await jobs.start("order", {}, "order:1");
    const c = await jobs.start("pay", {}, "pay:1");
    for (const r of [a, b, c]) await jobs.deliver(r.id);
    const rows = await jobRows(app, cy, repo);
    expect(rows.ok && rows.value.map((r) => [r.key, r.now])).toEqual([
      ["pay:1", "failed at charge"],
      ["order:1", "waiting for reply:1"],
      ["remind:1", "sleeping until 2026-11-15 00:00 UTC"],
    ]);
    expect(await jobRows(app, dee, repo)).toEqual({ ok: false, error: "forbidden" });
    expect(await jobRows(app, undefined, repo)).toMatchObject({ ok: false });
    // The API says the same, to a writer, and 404s or refuses the rest.
    const api = await call("GET", "/api/v1/repos/cy/shop/jobs", ownerRw);
    expect(api.status).toBe(200);
    expect(((await api.json()) as { items: { key: string }[] }).items.map((i) => i.key)).toEqual(["pay:1", "order:1", "remind:1"]);
    expect((await call("GET", "/api/v1/repos/cy/shop/jobs", readerRo)).status).toBeGreaterThanOrEqual(403);
  });
  test("the page lists the runs with their errors escaped, Cancel on an unfinished run and Retry on a failed one", async () => {
    const page = await (await fetch(`${servers.url}/cy/shop/app/jobs`, { headers: { cookie, connection: "close" } })).text();
    expect(page).toContain("waiting for reply:1");
    expect(page).toContain("sleeping until 2026-11-15 00:00 UTC");
    expect(page).toContain("card &lt;b&gt;declined&lt;/b&gt;");
    expect(page).not.toContain("card <b>declined");
    expect(page).toContain("Retry from the failed step");
    expect(page.match(/>\s*Cancel\s*</g)?.length).toBe(2);
    // The app page links to it once a job holds a timer.
    expect(await (await fetch(`${servers.url}/cy/shop/app`, { headers: { cookie, connection: "close" } })).text()).not.toContain("See the job runs");
    armTimer(app.db, { appId, env: "production", key: "job:abc:1", path: "/jobs/run", at: clock + DAY, timeoutMs: 1000, retries: 0 }, clock);
    expect(await (await fetch(`${servers.url}/cy/shop/app`, { headers: { cookie, connection: "close" } })).text()).toContain("See the job runs");
  });
  test("Cancel and Retry steer the run through the API, the MCP tool and the page's form; a state that does not allow it is 422", async () => {
    const [pay, order] = [jobs.list().find((r) => r.key === "pay:1")!, jobs.list().find((r) => r.key === "order:1")!];
    expect((await call("POST", `/api/v1/repos/cy/shop/jobs/${order.id}/cancel`, readerRo)).status).toBeGreaterThanOrEqual(403);
    expect(jobs.get(order.id)?.status).toBe("waiting");
    expect((await call("POST", `/api/v1/repos/cy/shop/jobs/${order.id}/cancel`, ownerRw)).status).toBe(200);
    expect(jobs.get(order.id)?.status).toBe("cancelled");
    expect((await call("POST", `/api/v1/repos/cy/shop/jobs/${order.id}/cancel`, ownerRw)).status).toBe(422);
    expect((await call("POST", `/api/v1/repos/cy/shop/jobs/${order.id}/explode`, ownerRw)).status).toBe(422);
    const heard: string[] = [];
    const off = app.bus.subscribe((e) => heard.push(e.kind));
    const mcp = await call("POST", "/mcp", ownerRw, { jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "app_deploy", arguments: { repo: "cy/shop", action: "retry_job", name: pay.id } } });
    expect(JSON.stringify(await mcp.json())).not.toContain("isError\":true");
    expect(jobs.get(pay.id)?.status).toBe("running");
    off();
    expect(heard).toContain("schedule.steered");
    const listed = await call("POST", "/mcp", ownerRw, { jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "app_deploy", arguments: { repo: "cy/shop", action: "jobs" } } });
    expect(JSON.stringify(await listed.json())).toContain("remind:1");
    // The form on the page: the same-origin check holds and the run changes.
    const form = await fetch(`${servers.url}/cy/shop/app/jobs/${jobs.list().find((r) => r.key === "remind:1")!.id}/cancel`, { method: "POST", headers: { cookie, connection: "close", origin: servers.url }, redirect: "manual" });
    expect(form.status).toBe(303);
    expect(jobs.list().find((r) => r.key === "remind:1")?.status).toBe("cancelled");
    const foreign = await fetch(`${servers.url}/cy/shop/app/jobs/${pay.id}/cancel`, { method: "POST", headers: { cookie, connection: "close", origin: "https://evil.example" }, redirect: "manual" });
    expect(foreign.status).not.toBe(303);
  });
  test("Cancel and Retry are plain POST forms, so the page works with JavaScript off", async () => {
    const page = await (await fetch(`${servers.url}/cy/shop/app/jobs`, { headers: { cookie, connection: "close" } })).text();
    const forms = page.match(/<form method="post" action="[^"]*\/app\/jobs\/[^"]+\/(?:cancel|retry)">/g) ?? [];
    expect(forms.length).toBeGreaterThan(0);
    // A guest never sees the steering forms.
    expect(await (await fetch(`${servers.url}/cy/shop/app/jobs`, { headers: { connection: "close" } })).text()).not.toContain("/app/jobs/");
  });
  test("an answer past the byte cap is read as a stream, stopped, and called unreachable", async () => {
    const big = new Response(new ReadableStream({ pull: (c) => c.enqueue(new Uint8Array(400_000).fill(32)) }));
    expect(await readCapped(big, BODY_MAX)).toBeUndefined();
    expect(await readCapped(new Response('{"runs":[]}'), BODY_MAX)).toBe('{"runs":[]}');
    const saved = reach.fetch;
    (reach as { fetch: typeof fetch }).fetch = (async () => new Response(new ReadableStream({ pull: (c) => c.enqueue(new Uint8Array(400_000).fill(32)) }))) as unknown as typeof fetch;
    try {
      expect(await jobRows(app, cy, repo)).toEqual({ ok: false, error: "unreachable" });
    } finally {
      (reach as { fetch: typeof fetch }).fetch = saved;
    }
  });
  test("the terminal door lists the runs and cancels or retries them for a writer only", async () => {
    const session = (user: User | undefined) => ({ user, db: app.db, app }) as unknown as Session;
    const toasts: string[] = [];
    const ctx = { toast: (t: string) => toasts.push(t), invalidate() {}, push() {}, theme: {} } as unknown as TuiCtx;
    // The palette offers it to a writer once the app holds a job timer, and to no one else.
    expect(jobsCommands(session(cy), ctx, repo)).toHaveLength(1);
    expect(jobsCommands(session(dee), ctx, repo)).toEqual([]);
    expect(jobsCommands(session(undefined), ctx, repo)).toEqual([]);
    expect(jobsCommands(session(cy), ctx, undefined)).toEqual([]);
    const run = await jobs.start("remind", {}, "remind:tui");
    await jobs.deliver(run.id);
    expect(jobs.get(run.id)?.status).toBe("sleeping");
    const screen = jobsScreen(session(cy), ctx, repo);
    await screen.load?.(new AbortController().signal);
    // The newest run is the cursor's: c cancels it through the same action the page uses.
    screen.key?.({ key: "c" } as never);
    await new Promise((r) => setTimeout(r, 50));
    expect(jobs.get(run.id)?.status).toBe("cancelled");
    expect(toasts.at(-1)).toBe("remind remind:tui: cancelled.");
    // A cancelled run cannot be retried: the app's refusal is said in words.
    screen.key?.({ key: "t" } as never);
    await new Promise((r) => setTimeout(r, 50));
    expect(toasts.at(-1)).toBe(JOB_WORDS.not_now);
    // A reader's screen reads nothing and steers nothing.
    const readerToasts: string[] = [];
    const readerCtx = { toast: (t: string) => readerToasts.push(t), invalidate() {}, push() {}, theme: {} } as unknown as TuiCtx;
    const reader = jobsScreen(session(dee), readerCtx, repo);
    await reader.load?.(new AbortController().signal);
    reader.key?.({ key: "c" } as never);
    await new Promise((r) => setTimeout(r, 50));
    expect(readerToasts.at(-1)).toBeUndefined();
  });
  test("an app that does not answer is said so, not guessed at", async () => {
    appAsleep = true;
    expect(await jobRows(app, cy, repo)).toEqual({ ok: false, error: "unreachable" });
    expect(await steerJob(app, cy, repo, "abc", "cancel")).toEqual({ ok: false, error: "unreachable" });
    expect((await call("GET", "/api/v1/repos/cy/shop/jobs", ownerRw)).status).toBe(502);
    appAsleep = false;
  });
  test("a run that failed for good is recorded at once, not delivered again, and asks the owner once", async () => {
    const run = await jobs.start("pay", {}, "pay:2");
    const body = JSON.stringify({ run: run.id, wake: 1 });
    armTimer(app.db, { appId, env: "production", key: `job:${run.id}:1`, path: "/jobs/run", at: clock, body: new TextEncoder().encode(body), timeoutMs: 1000, retries: 3 }, clock);
    const [t] = dueTimers(app.db, clock).filter((x) => x.key === `job:${run.id}:1`);
    // The app would sign this delivery itself; here the forge's own signer is the one under test.
    await fireTimer(app.db, reach, t!, () => clock);
    expect(dueTimers(app.db, clock + 10 * DAY).some((x) => x.key === `job:${run.id}:1`)).toBe(false);
    const lines = scheduleLines(app, cy, clock);
    const mine = lines.filter((l) => l.items[0]!.startsWith("job:"));
    expect(mine).toHaveLength(1);
    expect(mine[0]!.line).toContain('Step "charge" failed after 1 attempt: card <b>declined</b>');
    expect(scheduleLines(app, cy, clock)[0]!.items).toEqual(lines[0]!.items);
    expect(scheduleLines(app, dee, clock)).toEqual([]);
  });
});
@@ -0,0 +1,4 @@
// The template's own job tests (src/jobs.test.ts in a new app), run here too, on the files as the bundle carries them.
import { TEMPLATE_DIR } from "./helpers/template-app.ts";
await import(`${TEMPLATE_DIR}/src/jobs.test.ts`);
		@@ -1279,6 +1279,12 @@ Declare schedules in `forge.json`:
1279	1279
1280	1280	A fired job is an ordinary request to the app's port and counts toward its own limits. How to try it: `docs/features/timers.md`.
1281	1281
	1282	### Jobs
	1283
	1284	A durable job is code in the app that reads top to bottom, waits an hour or a week with no process running, survives a deploy, and never repeats a step that finished (`src/jobs.ts` in the app template; no dependency). The forge's part is only the timer above: `sleep` and `waitFor` save where the run is in the app's own database and arm a timer with key `job:<run>:<wake>` that `POST`s `/jobs/run` with `{ run, wake }`; the function then runs again from the top and finished steps answer from their saved results. A step must be safe to retry until it returns (it gets an `idempotencyKey`). A step that throws is retried with backoff (default 3 retries) and then the run is failed: the app answers `422` with `x-forge-no-retry: 1`, which the forge records as failed at once (no redelivery) and asks the owner once, by run. A step or job name that no longer matches what a run did fails that run plainly; `version` keeps an old path for old runs. A step result is JSON up to 256 KB; unfinished runs per app are `JOBS_ACTIVE_MAX` (default 1000) and pending timers `FORGE_TIMERS_PER_APP`.
	1285	- Seen and steered. The runs live in the app, so the forge asks it (waking it), signed as a timer's delivery is: `GET /jobs/runs`, `POST /jobs/runs/:id/cancel\|retry`. The app's page links to `/:owner/:name/app/jobs` while it holds a job timer: name, key, started, where it is now (running, sleeping until, waiting for, done, failed), Cancel and Retry from the failed step. `GET /api/v1/repos/:owner/:name/jobs`, `POST …/jobs/:id/(cancel\|retry)` and the `app_deploy` MCP tool (`jobs`, `cancel_job`, `retry_job`) answer the same rows (`src/actions/jobs.ts`). A writer's: a run's error and result are the app's own text.
	1286	- How to try it: `docs/features/jobs.md`.
	1287
1282	1288	## Where state lives
1283	1289
1284	1290	Git is truth for code. The app's serving configuration (slug, access, the
		@@ -49,7 +49,7 @@ bytes in `tools/list`. After changing a tool run `bun scripts/mcp-table.ts`;
49	49	\| `accounts` \| — \| Accounts, newest first: role, verified, suspended; `cursor` pages \| 264 \|
50	50	\| `agent` \| — \| Admin: create\\|update\\|remove an agent \| 371 \|
51	51	\| `agents` \| — \| Agents you may see, as GET /api/v1/agents \| 160 \|
52		\| `app_deploy` \| `redeploy`, `stop`, `pull`, `schedules`, `run_schedule`, `pause_schedule`, `resume_schedule` \| Redeploy or stop production or a pull-N preview, pull an adapter now, or list and steer the app's schedules \| 419 \|
	52	\| `app_deploy` \| `redeploy`, `stop`, `pull`, `schedules`, `run_schedule`, `pause_schedule`, `resume_schedule`, `jobs`, `cancel_job`, `retry_job` \| Redeploy or stop an env, pull, list and steer schedules and jobs \| 408 \|
53	53	\| `app_domains` \| — \| App domains: list, add, check, remove \| 201 \|
54	54	\| `app_env` \| `list`, `set`, `unset` \| An app's environment variables: list names, set or unset by scope; write-only \| 393 \|
55	55	\| `app_members` \| — \| People a private app names, who may enter it and its previews \| 297 \|
		@@ -0,0 +1,13 @@
	1	# Durable jobs
	2
	3	A job is code that waits (`sleep`, `waitFor`) without a process, survives restarts and deploys, and runs each step once. The rules are the "Jobs" part of "Time" in `docs/APPS.md`; the library and a worked example are in every new app (`src/jobs.ts`, `src/reminder.ts`, README "Jobs").
	4
	5	1. In a new app, open a note signed in and press Remind me tomorrow (`POST /notes/:id/remind`): a run of the `remind` job starts, loads the note, and waits a day for the note to be removed.
	6	2. Open the app on the forge: while the run holds a timer, its page has Jobs → See the job runs (`/:owner/:name/app/jobs`). The row says `waiting for removed:<id>, until <time> at the latest`. Cancel stops it; a failed run offers Retry from the failed step.
	7	3. The same from a script: `curl -H "authorization: Bearer $TOKEN" $FORGE_URL/api/v1/repos/<owner>/<name>/jobs`, `curl -X POST …/jobs/<run>/cancel`; the `app_deploy` MCP tool does it with `action: "jobs"`, `"cancel_job"` or `"retry_job"` (`name` is the run id).
	8	4. The terminal: with the app's repository open, the palette offers Job runs (a writer's, while the app holds a job timer); `c` cancels the run under the cursor, `t` retries a failed one, `g` reads again (`src/tui/screens/jobs.ts`).
	9	5. A run that fails for good (a step out of retries, or a changed job) asks the owner once.
	10
	11	Where the code is: the runtime `src/jobs.ts` in the app template (`templates/app-template.json`; its tests, `src/jobs.test.ts`, run by `test/template-jobs.test.ts`), the forge's rows and steering `src/actions/jobs.ts`, `src/http/api/jobs.ts`, `src/web/studio/jobs.ts`, the no-retry answer in `src/apps/timer_fire.ts`, and the ask in `scheduleLines` (`src/actions/schedules.ts`). Tests: `test/jobs-doors.test.ts`.
	12
	13	What the forge could not do: `trevato/app-template` is a separate repository, which this change cannot push to. The bundle carries the new files, so a forge without that repository hands them out; a forge that reads the repository needs the same files (`src/jobs.ts`, `src/reminder.ts`, `src/jobs.test.ts`, the routes and button, the README section) landed there, and `scripts/take-template.ts` then replaces the bundle.
		@@ -0,0 +1,154 @@
	1	/**
	2	* An app's durable job runs, seen and steered (#11596). The runs live in the
	3	* app's own data (`src/jobs.ts` of the app template), so the forge asks the app, waking it if
	4	* it sleeps: a signed `GET /jobs/runs`, and a signed `POST /jobs/runs/:id/cancel`
	5	* or `…/retry`, signed as a timer's delivery is (`controlHeaders`). The page,
	6	* `GET …/jobs`, the MCP tool and the terminal read these functions, so they
	7	* say the same words. Writers only: a run's error and result are the app's
	8	* own text. Production's runs; a preview keeps its own in its own data.
	9	*/
	10
	11	import type { App } from "../app.ts";
	12	import { runtimeReach } from "../apps/timer_service.ts";
	13	import { controlHeaders, type Reach } from "../apps/timer_fire.ts";
	14	import { getAppByRepo } from "../core/apps.ts";
	15	import type { Repo } from "../core/repos.ts";
	16	import type { User } from "../core/users.ts";
	17	import { err, ok, type Result } from "../kernel/result.ts";
	18	import { requireWriter } from "./access.ts";
	19
	20	/** How the forge reaches the app: the real runtime, unless a test hands in its own. */
	21	export const jobReach: { override: Reach \| undefined } = { override: undefined };
	22	const reachOf = (app: App): Reach => jobReach.override ?? runtimeReach(app);
	23
	24	export const JOB_STATUSES = ["running", "sleeping", "waiting", "done", "failed", "cancelled"] as const;
	25	export type JobStatus = (typeof JOB_STATUSES)[number];
	26
	27	export interface JobRunRow {
	28	readonly id: string;
	29	readonly job: string;
	30	readonly key: string;
	31	readonly status: JobStatus;
	32	/** ms epoch: when a sleeping run wakes, or a waiting run's timeout. */
	33	readonly until: number \| null;
	34	readonly event: string \| null;
	35	readonly error: string \| null;
	36	readonly failedStep: string \| null;
	37	readonly started: number;
	38	readonly updated: number;
	39	/** A finished run's result, as text, cut at 500 characters. */
	40	readonly result: string \| null;
	41	/** Where it is now, in words. */
	42	readonly now: string;
	43	}
	44
	45	export type JobError = "not_found" \| "unauthorized" \| "forbidden" \| "unreachable" \| "not_now" \| "invalid";
	46
	47	export const JOB_WORDS: Record<"unreachable" \| "not_now" \| "invalid", string> = {
	48	unreachable: "The app did not answer, so its job runs cannot be read or changed now. It may be asleep and failing to start; look at its deploy.",
	49	not_now: "That run is not in a state that allows it: only an unfinished run can be cancelled and only a failed one retried.",
	50	invalid: "The action is cancel or retry, on a run id.",
	51	};
	52
	53	const iso = (ms: number): string => new Date(ms).toISOString().replace("T", " ").slice(0, 16) + " UTC";
	54	const str = (v: unknown, max: number): string \| null => (typeof v === "string" ? v.slice(0, max).replace(/[\x00-\x08\x0b\x0c\x0e-\x1f]/g, "?") : null);
	55	const num = (v: unknown): number \| null => (typeof v === "number" && Number.isFinite(v) ? v : null);
	56
	57	/** Where a run is now: the words the page, the tool and the terminal share. */
	58	export function nowWords(r: Pick<JobRunRow, "status" \| "until" \| "event" \| "error" \| "failedStep">): string {
	59	switch (r.status) {
	60	case "sleeping":
	61	return r.until === null ? "sleeping" : `sleeping until ${iso(r.until)}${r.error === null ? "" : " (retrying a step)"}`;
	62	case "waiting":
	63	return `waiting for ${r.event ?? "an event"}${r.until === null ? "" : `, until ${iso(r.until)} at the latest`}`;
	64	case "failed":
	65	return `failed${r.failedStep === null ? "" : ` at ${r.failedStep}`}`;
	66	default:
	67	return r.status;
	68	}
	69	}
	70
	71	/** One row of the app's answer, or null when it is not one: the app's text is never trusted for its shape. */
	72	function rowOf(v: unknown): JobRunRow \| null {
	73	if (typeof v !== "object" \|\| v === null) return null;
	74	const o = v as Record<string, unknown>;
	75	const status = JOB_STATUSES.find((s) => s === o["status"]);
	76	const id = str(o["id"], 40);
	77	if (status === undefined \|\| id === null \|\| !/^[A-Za-z0-9-]{1,40}$/.test(id)) return null;
	78	const base = {
	79	id, job: str(o["job"], 60) ?? "", key: str(o["key"], 100) ?? "", status, until: num(o["until"]), event: str(o["event"], 200),
	80	error: str(o["error"], 500), failedStep: str(o["failedStep"], 100), started: num(o["started"]) ?? 0, updated: num(o["updated"]) ?? 0,
	81	};
	82	const result = o["result"] === null \|\| o["result"] === undefined ? null : (str(typeof o["result"] === "string" ? o["result"] : JSON.stringify(o["result"]), 500));
	83	return { ...base, result, now: nowWords(base) };
	84	}
	85
	86	/** What the app may answer with: a hundred rows of capped text fit well inside it. */
	87	export const BODY_MAX = 1_000_000;
	88
	89	/** The body read as a stream with a byte cap; undefined (and the stream cancelled) once it passes the cap, so a large answer is never held in memory. */
	90	export async function readCapped(res: Response, max: number): Promise<string \| undefined> {
	91	const reader = res.body?.getReader();
	92	if (reader === undefined) return "";
	93	const chunks: Uint8Array[] = [];
	94	let n = 0;
	95	for (;;) {
	96	const { done, value } = await reader.read();
	97	if (done \|\| value === undefined) break;
	98	n += value.length;
	99	if (n > max) {
	100	void reader.cancel().catch(() => undefined);
	101	return undefined;
	102	}
	103	chunks.push(value);
	104	}
	105	return new TextDecoder().decode(Buffer.concat(chunks));
	106	}
	107
	108	async function ask(app: App, repo: Repo, method: "GET" \| "POST", path: string, reach: Reach): Promise<Result<unknown, "not_found" \| "unreachable" \| "not_now">> {
	109	const record = getAppByRepo(app.db, repo.id);
	110	if (record === undefined) return err("not_found");
	111	const port = await reach.ensure(record, "production").catch(() => undefined);
	112	if (port === undefined) return err("unreachable");
	113	try {
	114	const res = await reach.fetch(`http://127.0.0.1:${port}${path}`, {
	115	method, headers: controlHeaders(reach.identityKey(record.id), Date.now()), redirect: "manual", signal: AbortSignal.timeout(10_000),
	116	});
	117	if (res.status === 409) return err("not_now");
	118	if (!res.ok) return err("unreachable");
	119	const body = await readCapped(res, BODY_MAX);
	120	return body === undefined ? err("unreachable") : ok(JSON.parse(body) as unknown);
	121	} catch {
	122	return err("unreachable");
	123	}
	124	}
	125
	126	/** The runs, newest first. */
	127	export async function jobRows(app: App, actor: User \| undefined, repo: Repo, reach: Reach = reachOf(app)): Promise<Result<JobRunRow[], JobError>> {
	128	const writer = requireWriter(app, actor, repo);
	129	if (!writer.ok) return err(writer.error);
	130	const got = await ask(app, repo, "GET", "/jobs/runs?limit=100", reach);
	131	if (!got.ok) return err(got.error);
	132	const runs = (got.value as { runs?: unknown } \| null)?.runs;
	133	return ok(Array.isArray(runs) ? runs.slice(0, 100).map(rowOf).filter((r): r is JobRunRow => r !== null) : []);
	134	}
	135
	136	export type JobAction = "cancel" \| "retry";
	137
	138	export async function steerJob(app: App, actor: User \| undefined, repo: Repo, id: string, action: string, reach: Reach = reachOf(app)): Promise<Result<true, JobError>> {
	139	const writer = requireWriter(app, actor, repo);
	140	if (!writer.ok) return err(writer.error);
	141	if ((action !== "cancel" && action !== "retry") \|\| !/^[A-Za-z0-9-]{1,40}$/.test(id)) return err("invalid");
	142	const done = await ask(app, repo, "POST", `/jobs/runs/${id}/${action}`, reach);
	143	if (!done.ok) return err(done.error);
	144	// The same event a schedule's steering is, so every door already listening redraws.
	145	app.bus.append({ kind: "schedule.steered", repoId: repo.id, actorId: writer.value.id, payload: { name: `job:${id}`, env: "production", action } });
	146	return ok(true);
	147	}
	148
	149	/** How many job timers the app holds: what says an app has jobs without waking it. */
	150	export function jobTimers(app: App, repo: Repo): number {
	151	const record = getAppByRepo(app.db, repo.id);
	152	if (record === undefined) return 0;
	153	return app.db.query<{ n: number }, [string]>("SELECT COUNT(*) AS n FROM timer WHERE app_id = ?1 AND kind = 'once' AND key LIKE 'job:%'").get(record.id)?.n ?? 0;
	154	}
		@@ -10,7 +10,7 @@ import type { App } from "../app.ts";
10	10	import { getAppByRepo } from "../core/apps.ts";
11	11	import { canAdmin, listReposWritableTo, type Repo } from "../core/repos.ts";
12	12	import type { User } from "../core/users.ts";
13		import { failingNow, runNow, runsOf, scheduleTimer, scheduleTimers, setPaused, type TimerRun } from "../core/timers.ts";
	13	import { failedJobs, failingNow, runNow, runsOf, scheduleTimer, scheduleTimers, setPaused, type TimerRun } from "../core/timers.ts";
14	14	import { err, ok, type Result } from "../kernel/result.ts";
15	15	import { requireReader, requireWriter } from "./access.ts";
16	16
		@@ -108,7 +108,7 @@ export function steerSchedule(app: App, actor: User \| undefined, repo: Repo, nam
108	108	* with the reason. The item is the schedule, not the run, so a schedule that
109	109	* keeps failing is one ask, not one a day (`crew/notify.ts` says only what is new).
110	110	*/
111		export function scheduleLines(app: App, user: User): { line: string; items: string[]; link: string }[] {
	111	export function scheduleLines(app: App, user: User, now = Date.now()): { line: string; items: string[]; link: string }[] {
112	112	if (user.kind !== "user") return [];
113	113	const out: { line: string; items: string[]; link: string }[] = [];
114	114	for (const repo of listReposWritableTo(app.db, user, { limit: 200, withoutArchived: true })) {
		@@ -118,6 +118,14 @@ export function scheduleLines(app: App, user: User): { line: string; items: stri
118	118	const why = run.status === "failed" ? `Schedule ${run.name} failed: ${run.error}` : run.error;
119	119	out.push({ line: `${repo.name}: ${why}`, items: [`schedule:${record.id}:${run.env}:${run.name}`], link: `/${repo.ownerName}/${repo.name}#schedules` });
120	120	}
	121	// A durable job that failed for good is one ask, named by its run (`job:<run>:<wake>`), never one per wake.
	122	const seen = new Set<string>();
	123	for (const run of failedJobs(app.db, record.id, now)) {
	124	const id = run.name.split(":")[1] ?? run.name;
	125	if (seen.has(id)) continue;
	126	seen.add(id);
	127	out.push({ line: `${repo.name}: a job run failed: ${run.error ?? "no reason given"}`, items: [`job:${record.id}:${id}`], link: `/${repo.ownerName}/${repo.name}/app/jobs` });
	128	}
121	129	}
122	130	return out;
123	131	}
		@@ -63,6 +63,23 @@ export function deliveryHeaders(key: string, t: Timer, attempt: number, expires:
63	63	};
64	64	}
65	65
	66	/** The `error` an app's JSON answer names, when it names one. */
	67	function failureText(head: string): string \| undefined {
	68	try {
	69	const e = (JSON.parse(head) as { error?: unknown }).error;
	70	return typeof e === "string" && e !== "" ? e : undefined;
	71	} catch {
	72	return undefined;
	73	}
	74	}
	75
	76	/** The headers of a request the forge makes of an app outside a timer (listing or steering its jobs): the same signature, an idempotency key of its own. */
	77	export function controlHeaders(key: string, now: number): Record<string, string> {
	78	const idem = `control:${crypto.randomUUID()}`;
	79	const identity = `forge-timer\|${idem}\|timer\|${now + 60_000}`;
	80	return { "idempotency-key": idem, "x-forge-identity": identity, "x-forge-signature": createHmac("sha256", key).update(identity).digest("base64url") };
	81	}
	82
66	83	async function readHead(res: Response): Promise<string> {
67	84	const reader = res.body?.getReader();
68	85	if (reader === undefined) return "";
		@@ -78,7 +95,7 @@ async function readHead(res: Response): Promise<string> {
78	95	return new TextDecoder().decode(Buffer.concat(chunks)).slice(0, RESPONSE_KEPT).replace(/[\x00-\x08\x0b\x0c\x0e-\x1f]/g, "?");
79	96	}
80	97
81		interface Attempt { readonly ok: boolean; readonly status: number \| null; readonly response: string \| null; readonly error: string \| null }
	98	interface Attempt { readonly final?: boolean; readonly ok: boolean; readonly status: number \| null; readonly response: string \| null; readonly error: string \| null }
82	99
83	100	async function attemptOnce(reach: Reach, app: AppRecord, t: Timer, attempt: number, now: () => number): Promise<Attempt> {
84	101	const port = await reach.ensure(app, t.env).catch(() => undefined);
		@@ -94,6 +111,8 @@ async function attemptOnce(reach: Reach, app: AppRecord, t: Timer, attempt: numb
94	111	});
95	112	const head = await readHead(res);
96	113	if (res.status >= 200 && res.status < 300) return { ok: true, status: res.status, response: head, error: null };
	114	// An app that says `x-forge-no-retry` has decided: delivering again would only repeat it (a durable job that failed, #11596).
	115	if (res.headers.get("x-forge-no-retry") === "1") return { ok: false, final: true, status: res.status, response: head, error: failureText(head) ?? `the app answered ${res.status}` };
97	116	return { ok: false, status: res.status, response: head, error: `the app answered ${res.status}` };
98	117	} catch (e) {
99	118	const timedOut = e instanceof Error && (e.name === "TimeoutError" \|\| e.name === "AbortError");
		@@ -153,10 +172,10 @@ export async function fireTimer(db: Database, reach: Reach, t: Timer, now: () =>
153	172	finishTimer(db, t, next);
154	173	return;
155	174	}
156		if (attempt <= t.retries) return retryTimer(db, t.id, finished + backoffFor(t.attempt), attempt);
	175	if (result.final !== true && attempt <= t.retries) return retryTimer(db, t.id, finished + backoffFor(t.attempt), attempt);
157	176	recordRun(db, {
158	177	...base, startedAt: started, finishedAt: finished, attempts: attempt, status: "failed", httpStatus: result.status, response: result.response,
159		error: `${result.error ?? "failed"}, after ${attempt} ${attempt === 1 ? "attempt" : "attempts"}`, ...late, durationMs: finished - started,
	178	error: result.final === true ? (result.error ?? "failed") : `${result.error ?? "failed"}, after ${attempt} ${attempt === 1 ? "attempt" : "attempts"}`, ...late, durationMs: finished - started,
160	179	});
161	180	finishTimer(db, t, next);
162	181	if (t.kind === "schedule" && failingDays(runsOf(db, t.appId, t.name, 30).filter((r) => r.env === t.env)) >= PAUSE_AFTER_DAYS) {
		@@ -0,0 +1,9 @@
	1	import { REPO, type Route } from "../shared.ts";
	2
	3	export const route: Route = {
	4	method: "get",
	5	path: `${REPO}/jobs`,
	6	summary: "An app's durable job runs, newest first: job, key, status (running, sleeping, waiting, done, failed, cancelled), where it is now, when it wakes, the failed step and error, a finished run's result; read from the app itself, waking it; a writer's; 404 for anyone else, 502 when the app does not answer",
	7	auth: true,
	8	role: "member",
	9	};
		@@ -162,6 +162,7 @@ export { route as getApiV1ReposOwnerNameIssuesNumberDependencies } from "./getAp
162	162	export { route as getApiV1ReposOwnerNameIssuesNumberPlan } from "./getApiV1ReposOwnerNameIssuesNumberPlan.ts";
163	163	export { route as getApiV1ReposOwnerNameIssuesNumberTeams } from "./getApiV1ReposOwnerNameIssuesNumberTeams.ts";
164	164	export { route as getApiV1ReposOwnerNameIterations } from "./getApiV1ReposOwnerNameIterations.ts";
	165	export { route as getApiV1ReposOwnerNameJobs } from "./getApiV1ReposOwnerNameJobs.ts";
165	166	export { route as getApiV1ReposOwnerNameLabels } from "./getApiV1ReposOwnerNameLabels.ts";
166	167	export { route as getApiV1ReposOwnerNameMap } from "./getApiV1ReposOwnerNameMap.ts";
167	168	export { route as getApiV1ReposOwnerNameMilestones } from "./getApiV1ReposOwnerNameMilestones.ts";
		@@ -343,6 +344,7 @@ export { route as postApiV1ReposOwnerNameIssuesNumberComments } from "./postApiV
343	344	export { route as postApiV1ReposOwnerNameIssuesNumberDependencies } from "./postApiV1ReposOwnerNameIssuesNumberDependencies.ts";
344	345	export { route as postApiV1ReposOwnerNameIssuesNumberDependenciesBlockerConfirm } from "./postApiV1ReposOwnerNameIssuesNumberDependenciesBlockerConfirm.ts";
345	346	export { route as postApiV1ReposOwnerNameIssuesNumberPrivate } from "./postApiV1ReposOwnerNameIssuesNumberPrivate.ts";
	347	export { route as postApiV1ReposOwnerNameJobsIdAction } from "./postApiV1ReposOwnerNameJobsIdAction.ts";
346	348	export { route as postApiV1ReposOwnerNameLabels } from "./postApiV1ReposOwnerNameLabels.ts";
347	349	export { route as postApiV1ReposOwnerNameMakeItYours } from "./postApiV1ReposOwnerNameMakeItYours.ts";
348	350	export { route as postApiV1ReposOwnerNameMilestonesNumberLimb } from "./postApiV1ReposOwnerNameMilestonesNumberLimb.ts";
		@@ -9,6 +9,7 @@ const ALIAS: Readonly<Record<string, string>> = {
9	9	app: "apps",
10	10	timers: "apps",
11	11	schedules: "apps",
	12	jobs: "apps",
12	13	"app-templates": "apps",
13	14	display: "builds",
14	15	following: "forks",
		@@ -247,6 +247,18 @@ export function failingNow(db: Database, appId: string): TimerRun[] {
247	247	return out;
248	248	}
249	249
	250	/**
	251	* Job runs (#11596) the app failed for good in the last `days` days: one per run, newest first.
	252	* A durable job's timers are named `job:<run>:<wake>`; the app answers a failed run with
	253	* `x-forge-no-retry`, which records it as failed at once.
	254	*/
	255	export function failedJobs(db: Database, appId: string, now: number, days = 3, limit = 20): TimerRun[] {
	256	const rows = db
	257	.query<Row, [string, number, number]>("SELECT * FROM timer_run WHERE app_id = ?1 AND name LIKE 'job:%' AND status = 'failed' AND finished_at > ?2 ORDER BY id DESC LIMIT ?3")
	258	.all(appId, now - days * 86_400_000, limit);
	259	return rows.map(toRun);
	260	}
	261
250	262	/** Drop the schedule rows of environments that are gone (a closed pull's preview). */
251	263	export function dropGoneEnvs(db: Database, appId: string, live: ReadonlySet<string>): void {
252	264	for (const t of scheduleTimers(db, appId)) if (!live.has(t.env)) db.query("DELETE FROM timer WHERE id = ?1").run(t.id);