trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

Each release tick leaves a heartbeat: release_keeper answers when the tick last ran, what it did and why #11677

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/11674-each-release-tick-leaves-a-heartbeat-rel → main 6 commits fork 3 threads · 3 unresolved

  • Merged. scout/11674-each-release-tick-leaves-a-heartbeat-rel was merged into main as 87b2403.

Review

Sign in to review this change.
8 files changed +149 −3
docs/REFERENCE.md +1
@@ -593,6 +593,7 @@ The release keeper (`src/keeper/`, #7557) cuts, promotes and verifies releases o
593593 - **Where to see it:** the **Release keeper** card on `/releases`, `Release keeper` in the terminal's palette, the **Maintenance** board under `/boards`, MCP `release_keeper` and `GET /api/v1/keeper` (JSON; 403 for a caller who may not read releases), each gated as `/releases` is: a guest sees none of it.
594594
595595 How it works and how to turn it on: [`docs/features/release-keeper.md`](features/release-keeper.md).
596 The release tick's heartbeat (`heartbeat`, and `POST /api/v1/release/heartbeat` for a release path outside the forge): [`docs/features/release-heartbeat.md`](features/release-heartbeat.md).
596597
597598 `/changelog` lists the last 30 releases of `FORGE_REPO` (public only) with their pulls, for anyone; the terminal's palette has `Changelog` for the same list. In the terminal, `/` takes a word and keeps the releases whose version or pull titles contain it (`esc` clears it); Enter on a release lists its pulls, and Enter on a pull opens it (`docs/features/changelog.md`).
598599
docs/features/release-heartbeat.md +12
@@ -11,3 +11,15 @@ Where to see it:
1111 The flow alarm `main_ahead` (`src/flow/alarm_main_ahead.ts`) fires when main's last finished check passed, its head is not the commit production runs, and main has been ahead for more than 90 minutes. Change the limit with the `release.ahead_minutes` row of `forge_setting` (a whole number, 1 to 10080; anything else is 90). It names production's version and commit, the commits and pulls ahead, how long, and when the tick was last heard from. It is an ask for the owner (home) and one Sentinel issue (`src/crew/sentinel_main_ahead.ts`), which closes itself at the next hunt after production catches up, unless a person commented on it. The issue names the tick's last outcome and time, never its reason: the issue may be public, and the reason is shown only to readers of /releases.
1212
1313 `/api/v1/telemetry/home`: `live` is the version and commit this process runs (null when unstamped); `main` is the head of the forge's own default branch and `behind` what main holds that live does not (#11676).
14
15 ## Recording a tick from outside (#11674)
16
17 A release path that runs outside the forge (a workflow, a script) leaves the same heartbeat, replacing the last one:
18
19 ```sh
20 curl -X POST "$FORGE_URL/api/v1/release/heartbeat" \
21 -H "authorization: Bearer $ADMIN_TOKEN" -H "content-type: application/json" \
22 -d '{"outcome":"skipped","reason":"main has not moved"}'
23 ```
24
25 It needs a token with the `admin` scope belonging to a site administrator (401 without a token, 403 otherwise). `outcome` is `cut`, `skipped` or `failed`; `reason` is 1 to 300 characters with no control characters (422 otherwise). The answer is the recorded beat, `201`.
src/contract/openapi/routes/index.ts +1
@@ -299,6 +299,7 @@ export { route as postApiV1Migrate } from "./postApiV1Migrate.ts";
299299 export { route as postApiV1Orgs } from "./postApiV1Orgs.ts";
300300 export { route as postApiV1OrgsOrgTeams } from "./postApiV1OrgsOrgTeams.ts";
301301 export { route as postApiV1OrgsOrgTeamsTeamMembers } from "./postApiV1OrgsOrgTeamsTeamMembers.ts";
302 export { route as postApiV1ReleaseHeartbeat } from "./postApiV1ReleaseHeartbeat.ts";
302303 export { route as postApiV1Repos } from "./postApiV1Repos.ts";
303304 export { route as postApiV1ReposOwnerNameApp } from "./postApiV1ReposOwnerNameApp.ts";
304305 export { route as postApiV1ReposOwnerNameAppDomains } from "./postApiV1ReposOwnerNameAppDomains.ts";
src/contract/openapi/routes/postApiV1ReleaseHeartbeat.ts added +14
@@ -0,0 +1,14 @@
1 import type { Route } from "../shared.ts";
2
3 export const route: Route = {
4 method: "post",
5 path: "/api/v1/release/heartbeat",
6 summary: "A release path outside the forge records one tick: outcome cut, skipped or failed, and a reason of at most 300 characters",
7 auth: true,
8 role: "admin",
9 refusals: {
10 forbidden: "a release heartbeat is recorded by the forge's administrators only",
sentinelcrew outdated

This route is admin only; readers of the repository are refused, so the text is wrong.
11 invalid_outcome: "outcome is cut, skipped or failed",
12 invalid_reason: "reason is plain words, 1 to 300 characters",
13 },
14 };
src/contract/tags.ts +1
@@ -43,6 +43,7 @@ const ALIAS: Readonly<Record<string, string>> = {
4343 health: "meta",
4444 version: "meta",
4545 releases: "meta",
46 release: "meta",
4647 changelog: "meta",
4748 flags: "meta",
4849 here: "meta",
src/http/api/keeper.ts +22 −1
@@ -2,16 +2,24 @@
22 * `GET /api/v1/keeper`: the release keeper's releases, holds and what blocks
33 * the next cut, for whoever may read /releases — MCP `release_keeper` over
44 * HTTP. The answer is `keeper/landed.ts`'s `keeperAnswer`.
5 *
6 * `POST /api/v1/release/heartbeat` (#11674): a release path outside the forge
7 * leaves the same heartbeat the in-process keeper's tick does, `{outcome,
8 * reason}`, for an administrator's admin token; anyone else is refused.
59 */
sentinelcrew outdated

Mis-indented comment terminator.
610
711 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
12 import { isOutcome, REASON_MAX, recordBeat } from "../../keeper/beat.ts";
813 import { keeperAnswer } from "../../keeper/landed.ts";
914 import { mayRead } from "../../release/gate.ts";
1015 import type { Handler } from "../context.ts";
1116 import { holds, weakEtag } from "../etag.ts";
12 import { apiError } from "../respond.ts";
17 import { apiError, body, json, requireScope } from "../respond.ts";
1318 import type { Router } from "../router.ts";
1419
20 /** A reason reaches terminals and pages: no control characters (escape sequences, newlines). */
21 const CONTROL = /[\u0000-\u001f\u007f-\u009f]/;
22
1523 export function keeperRoutes(router: Router<Handler>): void {
1624 router.add("GET", "/api/v1/keeper", (ctx) => {
1725 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
@@ -25,4 +33,17 @@ export function keeperRoutes(router: Router<Handler>): void {
2533 if (holds(ctx.request, etag)) return new Response(null, { status: 304, headers });
2634 return new Response(text, { headers: { "content-type": "application/json", ...headers } });
2735 });
36
37 router.add("POST", "/api/v1/release/heartbeat", async (ctx) => {
38 const denied = requireScope(ctx, "admin");
39 if (denied) return denied;
40 if (ctx.auth.user?.isAdmin !== true)
41 return apiError("forbidden", 403, "a release heartbeat is recorded by the forge's administrators only");
42 const input = await body(ctx.request);
43 if (!isOutcome(input.outcome)) return apiError("invalid_outcome", 422, "outcome is cut, skipped or failed");
44 const reason = input.reason;
45 if (typeof reason !== "string" || reason.trim() === "" || reason.length > REASON_MAX || CONTROL.test(reason))
46 return apiError("invalid_reason", 422, `reason is plain words, 1 to ${REASON_MAX} characters`);
47 return json(recordBeat(ctx.app.db, input.outcome, reason), 201);
48 });
2849 }
src/keeper/beat.ts +4 −2
@@ -9,11 +9,13 @@
99
1010 import type { Database } from "bun:sqlite";
1111
12 export type Outcome = "cut" | "skipped" | "failed";
12 export const OUTCOMES = ["cut", "skipped", "failed"] as const;
13 export type Outcome = (typeof OUTCOMES)[number];
14 export const isOutcome = (v: unknown): v is Outcome => typeof v === "string" && (OUTCOMES as readonly string[]).includes(v);
1315
1416 /** No heartbeat for this long is itself said. */
1517 export const SILENT_MS = 60 * 60_000;
16 const REASON_MAX = 300;
18 export const REASON_MAX = 300;
1719
1820 export interface Beat {
1921 readonly at: string;
test/keeper-heartbeat.test.ts added +94
@@ -0,0 +1,94 @@
1 /** A release path outside the forge leaves the release tick's heartbeat (#11674) through POST /api/v1/release/heartbeat. */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { createToken } from "../src/core/tokens.ts";
8 import { createUser } from "../src/core/users.ts";
9 import { serve, type Servers } from "../src/http/server.ts";
10 import { lastBeat, recordBeat } from "../src/keeper/beat.ts";
11 import { keeperAnswer } from "../src/keeper/landed.ts";
12
13 describe("release heartbeat", () => {
14 let dataDir: string;
15 let app: App;
16 let servers: Servers;
17 let admin: string;
18 let bob: string;
19 let aliceRw: string;
20 const post = (token: string, data: unknown) =>
21 fetch(`${servers.url}/api/v1/release/heartbeat`, {
22 method: "POST",
23 headers: { "content-type": "application/json", authorization: `Bearer ${token}` },
24 body: JSON.stringify(data),
25 });
26
27 beforeAll(async () => {
28 dataDir = await mkdtemp(join(tmpdir(), "forge-heartbeat-"));
29 app = createApp({ dataDir, port: 0 });
30 servers = await serve(app);
31 const mk = async (username: string, first = false) => {
32 const u = await createUser(app.db, {
33 origin: "seed", trusted: true, username,
34 email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst: first,
35 });
36 if (!u.ok) throw new Error(u.error);
37 return u.value;
38 };
39 const alice = await mk("alice", true);
40 admin = createToken(app.db, alice.id, "a", "admin").token;
41 aliceRw = createToken(app.db, alice.id, "a-rw", "rw").token;
42 bob = createToken(app.db, (await mk("bob")).id, "b", "admin").token;
43 }, 30_000);
44
45 afterAll(async () => {
46 servers.stop();
47 app.close();
48 await rm(dataDir, { recursive: true, force: true });
49 });
50
51 test("no tick yet reads null", () => {
52 expect(keeperAnswer(app.db).heartbeat).toBeNull();
53 });
54
55 test("a posted heartbeat replaces the keeper's own and is what GET /api/v1/keeper answers", async () => {
56 recordBeat(app.db, "skipped", "main has not moved", new Date());
57 const res = await post(admin, { outcome: "failed", reason: " the workflow lost its runner " });
58 expect(res.status).toBe(201);
59 expect(await res.json()).toMatchObject({ outcome: "failed", reason: "the workflow lost its runner" });
60 const last = keeperAnswer(app.db).heartbeat!;
61 expect(last).toMatchObject({ outcome: "failed", reason: "the workflow lost its runner", silent: false });
62 const got: any = await (await fetch(`${servers.url}/api/v1/keeper`, { headers: { authorization: `Bearer ${admin}` } })).json();
63 expect(got.heartbeat).toMatchObject({ at: last.at, outcome: "failed", reason: last.reason });
64 });
65
66 test("a non-admin is refused as /releases refuses them, and nothing is written", async () => {
67 const before = lastBeat(app.db);
68 expect((await post(bob, { outcome: "cut", reason: "sneaky" })).status).toBe(403);
69 expect(lastBeat(app.db)).toEqual(before);
70 });
71
72 test("no token is a 401, and an admin's token without the admin scope is a 403; nothing is written", async () => {
73 const before = lastBeat(app.db);
74 const anon = await fetch(`${servers.url}/api/v1/release/heartbeat`, {
75 method: "POST",
76 headers: { "content-type": "application/json" },
77 body: JSON.stringify({ outcome: "cut", reason: "anon" }),
78 });
79 expect(anon.status).toBe(401);
80 expect((await post(aliceRw, { outcome: "cut", reason: "rw" })).status).toBe(403);
81 expect(lastBeat(app.db)).toEqual(before);
82 });
83
84 test("a reason carrying a control character is a 422", async () => {
85 expect((await post(admin, { outcome: "failed", reason: "red\u001b[2Jwiped" })).status).toBe(422);
86 expect((await post(admin, { outcome: "failed", reason: "two\nlines" })).status).toBe(422);
87 });
88
89 test("a bad outcome or reason is a 422", async () => {
90 expect((await post(admin, { outcome: "maybe", reason: "x" })).status).toBe(422);
91 expect((await post(admin, { outcome: "cut", reason: "x".repeat(301) })).status).toBe(422);
92 expect((await post(admin, { outcome: "cut" })).status).toBe(422);
93 });
94 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs