Public: anyone can read this; the crew can change it.
An admin redelivers a failed webhook delivery from the terminal's Deliveries screen with `r` #8084
Check · Review · Land · Live done — Landed and live in production.
torque/forge:torque/8075-an-admin-redelivers-a-failed-webhook-del → main 5 commits fork 2 threads · 2 unresolved
6 files
changed +216
−6
docs/MCP.md +2
| @@ -202,6 +202,8 @@ for a tunnel and a forge that sleeps. | ||
| 202 | 202 | |
| 203 | 203 | <!-- tools:end --> |
| 204 | 204 | |
| 205 | Sending a failed webhook delivery again (`redeliver`, #8075) is the terminal's `r` on a hook's Deliveries screen only; the `hooks` tool has no action for it yet. | |
| 206 | ||
| 205 | 207 | ### Adding a tool |
| 206 | 208 | |
| 207 | 209 | One tool, one file (#5924): `src/http/mcp/tools/<name>.ts` exports `tool`, |
docs/features/webhook-redeliver.md added +9
| @@ -0,0 +1,9 @@ | ||
| 1 | # Redeliver a failed webhook delivery | |
| 2 | ||
| 3 | A webhook delivery that ran out of attempts stays failed. A repository admin can send it again from the terminal. | |
| 4 | ||
| 5 | - **Key.** Open a hook's Deliveries screen in the terminal and press `r` on a row. The screen is a repository admin's, and so is the key. | |
| 6 | - **Which rows.** Only a `failed` row. On a pending or delivered row `r` does nothing. | |
| 7 | - **What happens.** `redeliver` (`src/core/webhooks.ts`) puts the same row back to `pending` with its attempts reset to zero and its last status and error cleared; there is no new row and no new event. It reads, checks and updates the row in one transaction. The worker tries it on its next pass with the same event's payload; on a paused hook (`p`) it waits, pending, until the hook is resumed. | |
| 8 | - **Sealed reports.** The delivery of a sealed report stays refused, as it is when the event first fires: `r` leaves the row failed. | |
| 9 | - **Terminal only, for now.** There is no page button and no MCP action yet. The terminal was the issue's ask (#8075), and the read-only deliveries list on the web (`GET …/hooks/:id/deliveries`, MCP `hooks` with `action: deliveries`) has no control to put it on. |
src/core/webhooks.ts +52
| @@ -187,6 +187,58 @@ export function enqueueForEvent(db: Database, event: ForgeEvent): number { | ||
| 187 | 187 | return queued; |
| 188 | 188 | } |
| 189 | 189 | |
| 190 | export type RedeliverError = "not_found" | "not_failed" | "sealed"; | |
| 191 | ||
| 192 | /** | |
| 193 | * Send a failed delivery again: the same row goes back to `pending` with | |
| 194 | * its attempts reset and its claim cleared, and `deliverDue` picks it up on | |
| 195 | * its next pass. No new row, no new event. Only a `failed` delivery of this | |
| 196 | * hook moves, and in one statement, so a row a sender holds (it is | |
| 197 | * `pending` while claimed) is never reset under it. A sealed report's | |
| 198 | * delivery stays refused, as `enqueueForEvent` refuses it. The read, the | |
| 199 | * sealed check and the update share one immediate transaction, so a report | |
| 200 | * sealed in between cannot slip through. | |
| 201 | */ | |
| 202 | export function redeliver( | |
| 203 | db: Database, | |
| 204 | deliveryId: string, | |
| 205 | webhookId: string, | |
| 206 | ): Result<true, RedeliverError> { | |
| 207 | return db.transaction(() => redeliverIn(db, deliveryId, webhookId)).immediate(); | |
| 208 | } | |
| 209 | ||
| 210 | function redeliverIn(db: Database, deliveryId: string, webhookId: string): Result<true, RedeliverError> { | |
| 211 | const row = db | |
| 212 | .query< | |
| 213 | { state: string; repo_id: string | null; payload: string }, | |
| 214 | [string, string] | |
| 215 | >( | |
| 216 | `SELECT d.state, e.repo_id, e.payload | |
| 217 | FROM webhook_delivery d JOIN event e ON e.id = d.event_id | |
| 218 | WHERE d.id = ?1 AND d.webhook_id = ?2`, | |
| 219 | ) | |
| 220 | .get(deliveryId, webhookId); | |
| 221 | if (row === null) return err("not_found"); | |
| 222 | if (row.state !== "failed") return err("not_failed"); | |
| 223 | let payload: Record<string, unknown> = {}; | |
| 224 | try { | |
| 225 | payload = JSON.parse(row.payload) as Record<string, unknown>; | |
| 226 | } catch { | |
| 227 | // an unreadable payload is treated as not sealed (it names no issue); the | |
| 228 | // event table's JSON check makes this unreachable, so no test can drive it | |
| 229 | } | |
| 230 | if (eventSealed(db, { repoId: row.repo_id, payload })) return err("sealed"); | |
| 231 | const moved = db | |
| 232 | .query( | |
| 233 | `UPDATE webhook_delivery | |
| 234 | SET state = 'pending', attempts = 0, next_try_at = NULL, | |
| 235 | last_status = NULL, last_error = NULL | |
| 236 | WHERE id = ?1 AND webhook_id = ?2 AND state = 'failed'`, | |
| 237 | ) | |
| 238 | .run(deliveryId, webhookId); | |
| 239 | return moved.changes === 1 ? ok(true) : err("not_failed"); | |
| 240 | } | |
| 241 | ||
| 190 | 242 | // ---- the worker ---- |
| 191 | 243 | |
| 192 | 244 | interface DueRow { |
src/tui/screens/hooks.ts +23 −6
| @@ -1,10 +1,11 @@ | ||
| 1 | 1 | /** |
| 2 | 2 | * A repository's webhooks in the terminal, an admin's: each hook's url, |
| 3 | 3 | * kinds and last delivery, never its secret. `enter` opens its last 20 |
| 4 | * deliveries. `a` asks for url, kinds and a secret (typed masked, 16–200 | |
| 5 | * characters) and adds it through `createWebhook`; `p` pauses or resumes | |
| 6 | * the selected one through `pauseWebhook` (a paused row reads `paused` and | |
| 7 | * gets no deliveries); `d` removes the selected one after a confirm through | |
| 4 | * deliveries; `r` there sends a failed one again through `redeliver`. | |
outdated
Rewrap this comment; it is over-long and splits the sentence about |
||
| 5 | * `a` asks for url, kinds and a secret (typed masked, 16–200 characters) | |
| 6 | * and adds it through `createWebhook`; `p` pauses or resumes the selected | |
| 7 | * one through `pauseWebhook` (a paused row reads `paused` and gets no | |
| 8 | * deliveries); `d` removes the selected one after a confirm through | |
| 8 | 9 | * `deleteWebhook`. The words of a refusal are MCP `hooks`'s. |
| 9 | 10 | */ |
| 10 | 11 | |
| @@ -14,7 +15,7 @@ import { | ||
| 14 | 15 | pauseWebhook, |
| 15 | 16 | } from "../../actions/repos.ts"; |
| 16 | 17 | import { canAdmin, type Repo } from "../../core/repos.ts"; |
| 17 | import { listWebhooks, recentDeliveries, type DeliveryInfo, type Webhook } from "../../core/webhooks.ts"; | |
| 18 | import { listWebhooks, recentDeliveries, redeliver, type DeliveryInfo, type Webhook } from "../../core/webhooks.ts"; | |
| 18 | 19 | import type { Ctx, Screen } from "../app/app.ts"; |
| 19 | 20 | import type { Command } from "../app/palette.ts"; |
| 20 | 21 | import { binding } from "../app/keys.ts"; |
| @@ -57,7 +58,14 @@ function deliveriesScreen(f: Session, ctx: Ctx, hook: Webhook, allowed: () => bo | ||
| 57 | 58 | let all: DeliveryInfo[] = []; |
| 58 | 59 | return { |
| 59 | 60 | title: "Deliveries", |
| 60 | keys: [binding(["j", "k"], "move"), binding(["esc", "q"], "back")], | |
| 61 | get keys() { | |
| 62 | const failed = all[state.index]?.state === "failed"; | |
| 63 | return [ | |
| 64 | binding(["j", "k"], "move"), | |
| 65 | binding(["r"], "redeliver", { disabled: !failed }), | |
| 66 | binding(["esc", "q"], "back"), | |
| 67 | ]; | |
| 68 | }, | |
| 61 | 69 | render(area, s) { |
| 62 | 70 | if (!allowed()) { |
| 63 | 71 | ctx.pop(); |
| @@ -83,6 +91,15 @@ function deliveriesScreen(f: Session, ctx: Ctx, hook: Webhook, allowed: () => bo | ||
| 83 | 91 | ctx.pop(); |
| 84 | 92 | return true; |
| 85 | 93 | } |
| 94 | if (e.key === "r") { | |
| 95 | const d = all[state.index]; | |
| 96 | if (d === undefined || d.state !== "failed") return false; | |
| 97 | const done = redeliver(f.db, d.id, hook.id); | |
| 98 | if (done.ok) ctx.toast("queued again", "success"); | |
| 99 | else ctx.toast("that delivery cannot be sent again", "danger"); | |
| 100 | ctx.invalidate(); | |
| 101 | return true; | |
| 102 | } | |
| 86 | 103 | return state.key(e, all.length); |
| 87 | 104 | }, |
| 88 | 105 | }; |
test/tui-hooks-deliveries.test.ts +37
| @@ -137,3 +137,40 @@ test("losing admin closes the deliveries instead of showing them", async () => { | ||
| 137 | 137 | expect(h.text()).not.toContain("refused: private address"); |
| 138 | 138 | h.stop(); |
| 139 | 139 | }); |
| 140 | ||
| 141 | const states = (): string[] => | |
| 142 | app.db | |
| 143 | .query<{ state: string }, []>( | |
| 144 | "SELECT d.state FROM webhook_delivery d JOIN event e ON e.id = d.event_id WHERE d.webhook_id = (SELECT id FROM webhook ORDER BY rowid LIMIT 1 OFFSET 1) ORDER BY d.rowid DESC", | |
| 145 | ) | |
| 146 | .all() | |
| 147 | .map((r) => r.state); | |
| 148 | ||
| 149 | test("r on a failed row queues it again; on a row that is not failed it does nothing", async () => { | |
| 150 | const h = open(); | |
| 151 | await h.feed("j", "enter"); | |
| 152 | // Newest first: issue.opened (failed) is on top, push (ok) below it. | |
| 153 | await h.feed("j", "r"); | |
| 154 | expect(h.text()).toMatch(/push · ok · 204 · 1/); | |
| 155 | expect(h.text()).not.toContain("queued again"); | |
| 156 | expect(states()).toEqual(["failed", "ok"]); | |
| 157 | await h.feed("k", "r"); | |
| 158 | const t = h.text(); | |
| 159 | expect(t).toContain("queued again"); | |
| 160 | expect(t).toMatch(/issue\.opened · pending · — · 0/); | |
| 161 | expect(t).not.toContain("refused: private address"); | |
| 162 | await h.feed("r"); | |
| 163 | expect(h.text()).toMatch(/issue\.opened · pending/); | |
| 164 | h.stop(); | |
| 165 | }); | |
| 166 | ||
| 167 | test("a person who lost admin cannot redeliver with r: the row stays failed", async () => { | |
| 168 | app.db.run("UPDATE webhook_delivery SET state = 'failed' WHERE state = 'pending'"); | |
| 169 | const h = open(); | |
| 170 | await h.feed("j", "enter"); | |
| 171 | (h.session as { user: User }).user = bob; | |
| 172 | await h.feed("r"); | |
| 173 | expect(h.text()).not.toContain("queued again"); | |
| 174 | expect(app.db.query("SELECT 1 FROM webhook_delivery WHERE state = 'pending'").get()).toBeNull(); | |
| 175 | h.stop(); | |
| 176 | }); |
test/webhook-redeliver.test.ts added +93
| @@ -0,0 +1,93 @@ | ||
| 1 | import { afterAll, beforeAll, expect, test } from "bun:test"; | |
| 2 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 3 | import { tmpdir } from "node:os"; | |
| 4 | import { join } from "node:path"; | |
| 5 | import { createApp, type App } from "../src/app.ts"; | |
| 6 | import { createIssue } from "../src/core/issues.ts"; | |
| 7 | import { createRepo } from "../src/core/repos.ts"; | |
| 8 | import { createUser } from "../src/core/users.ts"; | |
| 9 | import { createWebhook, enqueueForEvent, recentDeliveries, redeliver } from "../src/core/webhooks.ts"; | |
| 10 | ||
| 11 | let dir: string; | |
| 12 | let app: App; | |
| 13 | let repoId: string; | |
| 14 | let userId: string; | |
| 15 | ||
| 16 | const hook = (url: string): string => { | |
| 17 | const r = createWebhook(app.db, { repoId, url, kinds: "*", secret: "s3cret-signing-key-0123456789" }); | |
| 18 | if (!r.ok) throw new Error(r.error); | |
| 19 | return r.value.webhook.id; | |
| 20 | }; | |
| 21 | const queue = (webhookId: string, state: string): string => { | |
| 22 | const ev = app.bus.append({ kind: "push", repoId, actorId: userId, payload: {} }); | |
| 23 | enqueueForEvent(app.db, ev); | |
| 24 | const row = app.db | |
| 25 | .query<{ id: string }, [string, string]>("SELECT id FROM webhook_delivery WHERE webhook_id = ?1 AND event_id = ?2") | |
| 26 | .get(webhookId, ev.id)!; | |
| 27 | app.db.run( | |
| 28 | "UPDATE webhook_delivery SET state = ?2, attempts = 6, last_status = 500, next_try_at = '2999-01-01T00:00:00.000Z' WHERE id = ?1", | |
| 29 | [row.id, state], | |
| 30 | ); | |
| 31 | return row.id; | |
| 32 | }; | |
| 33 | const row = (id: string) => | |
| 34 | app.db | |
| 35 | .query<{ state: string; attempts: number; next_try_at: string | null }, [string]>( | |
| 36 | "SELECT state, attempts, next_try_at FROM webhook_delivery WHERE id = ?1", | |
| 37 | ) | |
| 38 | .get(id)!; | |
| 39 | const count = (): number => app.db.query<{ n: number }, []>("SELECT count(*) n FROM webhook_delivery").get()!.n; | |
| 40 | ||
| 41 | beforeAll(async () => { | |
| 42 | dir = await mkdtemp(join(tmpdir(), "forge-redeliver-")); | |
| 43 | app = createApp({ dataDir: dir, port: 0 }); | |
| 44 | const u = await createUser(app.db, { | |
| 45 | origin: "seed", trusted: true, username: "alice", email: "[email protected]", | |
| 46 | password: "a-good-passphrase", adminIfFirst: true, | |
| 47 | }); | |
| 48 | if (!u.ok) throw new Error(u.error); | |
| 49 | userId = u.value.id; | |
| 50 | const r = await createRepo(app.db, app.reposRoot, { ownerId: userId, name: "lobby" }); | |
| 51 | if (!r.ok) throw new Error(r.error); | |
| 52 | repoId = r.value.id; | |
| 53 | }); | |
| 54 | afterAll(async () => { | |
| 55 | app.db.close(); | |
| 56 | await rm(dir, { recursive: true, force: true }); | |
| 57 | }); | |
| 58 | ||
| 59 | test("a failed delivery goes back to pending, attempts 0, claim cleared, same row", () => { | |
| 60 | const w = hook("https://example.com/a"); | |
| 61 | const d = queue(w, "failed"); | |
| 62 | const before = count(); | |
| 63 | expect(redeliver(app.db, d, w).ok).toBe(true); | |
| 64 | expect(row(d)).toEqual({ state: "pending", attempts: 0, next_try_at: null }); | |
| 65 | expect(count()).toBe(before); | |
| 66 | expect(recentDeliveries(app.db, w)[0]!.state).toBe("pending"); | |
| 67 | }); | |
| 68 | ||
| 69 | test("ok, pending and another hook's delivery are refused and unchanged", () => { | |
| 70 | const w = hook("https://example.com/b"); | |
| 71 | const other = hook("https://example.com/c"); | |
| 72 | for (const state of ["ok", "pending"]) { | |
| 73 | const d = queue(w, state); | |
| 74 | expect(redeliver(app.db, d, w)).toEqual({ ok: false, error: "not_failed" }); | |
| 75 | expect(row(d).attempts).toBe(6); | |
| 76 | } | |
| 77 | const d = queue(w, "failed"); | |
| 78 | expect(redeliver(app.db, d, other)).toEqual({ ok: false, error: "not_found" }); | |
| 79 | expect(redeliver(app.db, "nope", w)).toEqual({ ok: false, error: "not_found" }); | |
| 80 | expect(row(d).state).toBe("failed"); | |
| 81 | }); | |
| 82 | ||
| 83 | test("a sealed report's delivery stays refused", () => { | |
| 84 | const w = hook("https://example.com/d"); | |
| 85 | const issue = createIssue(app.db, { repoId, authorId: userId, title: "Hole", body: "x", sealed: true }); | |
| 86 | if (!issue.ok) throw new Error(issue.error); | |
| 87 | const ev = app.bus.append({ | |
| 88 | kind: "issue.opened", repoId, actorId: userId, payload: { number: issue.value.number }, | |
| 89 | }); | |
| 90 | app.db.run("INSERT INTO webhook_delivery (id, webhook_id, event_id, state) VALUES ('sealed1', ?1, ?2, 'failed')", [w, ev.id]); | |
| 91 | expect(redeliver(app.db, "sealed1", w)).toEqual({ ok: false, error: "sealed" }); | |
| 92 | expect(row("sealed1").state).toBe("failed"); | |
| 93 | }); |
new above
offline
Empty catch: fine for intent, but the comment should say an unreadable payload is treated as not sealed, and a test should cover it.