Public: anyone can read this; the crew can change it.

The forge alarms when main is green and ahead of production for more than 90… #11592

Parked opened this
Closed without merging

scout/forge:scout/11507-the-forge-alarms-when-main-is-green-and → main 11 commits fork

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
31 files changed +1167 −46
docs/features/lane-stopped.md added +10
@@ -0,0 +1,10 @@
1 # The lane has stopped (#11507)
2
3 On 2026-11-12 nothing merged for 12 h 51 min and nothing said so: every check was being stopped for CPU, and main was not ahead of production, so the main-ahead alarm stayed quiet. The flow alarm `lane_stopped` (`src/flow/alarm_lane.ts`) fires for the forge's own repository in two states, as one alarm:
4
5 - **The lane has stopped:** pulls are open (or work is ready), the crew is not at its ceiling, and no pull has merged for 90 minutes. Change the limit with the `flow.lane_minutes` row of `forge_setting` (a whole number, 1 to 10080; anything else is 90). It names how long, how many checks failed since the last merge and the commonest failure line verbatim.
6 - **One cause failing many pulls:** the same failure line (numbers masked) failed checks on five different pulls within an hour and since the last merge. The line is the first line of a check's summary, for a check that named no failing test; a named test is `src/checks/shared_red.ts`'s. /flow says it is the forge's failure; `check_failed` is not sent to builders for it (`waitOnMain` in `src/crew/dispatch.ts`) and the stale-red sweep neither parks nor reworks its pulls, only checks them again.
7
8 It is an ask for the owner and one Sentinel issue (`src/crew/sentinel_lane.ts`), closed at the next hunt after a pull merges and the cause is gone, unless a person commented on it. `land_past_gate` treats the alarm as landing stopped (`ownLandingStopped`).
9
10 Left out: the Sentinel issue's evidence carries the failure count, so it is commented again as that count moves.
docs/features/release-heartbeat.md added +13
@@ -0,0 +1,13 @@
1 # Release heartbeat and the main-ahead alarm (#11507)
2
3 The release tick records a heartbeat each time it runs (`src/keeper/beat.ts`, table `keeper_beat`): when, whether it cut, skipped or failed, and why. The outcome is the keeper's own (`tickStep` in `src/keeper/keeper.ts` says it where it decides), never read back from the line's words; a tick that throws is recorded as failed (`stepWithBeat`). None for 60 minutes is said plainly ("No heartbeat from the release tick for …").
4
5 Where to see it:
6
7 - `/releases`, in the release keeper section: the last heartbeat (`#keeper-heartbeat`) and, while main is green and ahead of production, how far and for how long (`#keeper-ahead`).
8 - The terminal: the palette's **Release keeper** screen, gated as /releases is, shows the same two lines under what holds the next cut.
9 - `release_keeper` (MCP) and `GET /api/v1/keeper`: `heartbeat` (at, outcome, reason, silent, says).
10
11 The flow alarm `main_ahead` (`src/flow/alarm_main_ahead.ts`) fires when main's last finished check passed, its head is not the commit production runs, and main has been ahead for more than 90 minutes. Change the limit with the `release.ahead_minutes` row of `forge_setting` (a whole number, 1 to 10080; anything else is 90). It names production's version and commit, the commits and pulls ahead, how long, and when the tick was last heard from. It is an ask for the owner (home) and one Sentinel issue (`src/crew/sentinel_main_ahead.ts`), which closes itself at the next hunt after production catches up, unless a person commented on it. The issue names the tick's last outcome and time, never its reason: the issue may be public, and the reason is shown only to readers of /releases.
12
13 `/api/v1/telemetry/home`: `live` is unchanged (main's newest deploy). `production` is the version and commit this process runs (null when the commit is unknown); `main` is main's newest deploy, the same as `live`, under its own name.
src/checks/gate_stopped.ts +7 −1
@@ -12,6 +12,7 @@
1212
1313 import type { Database } from "bun:sqlite";
1414 import { getRepo } from "../core/repos.ts";
15 import { laneAlarm } from "../flow/alarm_lane.ts";
1516 import { ownRepoPath } from "../kernel/own_repo.ts";
1617
1718 /** How many gates in a row, red on one base, make landing stopped. */
@@ -66,7 +67,12 @@ function oneCause(gates: readonly { reason: string | null; tests: string }[]): b
6667 export function ownLandingStopped(db: Database, repo: { readonly id: string }): Stopped | undefined {
6768 const [owner, name] = ownRepoPath().split("/");
6869 const own = owner && name ? getRepo(db, owner, name) : undefined;
69 return own !== undefined && own.id === repo.id ? landingStopped(db, own.id) : undefined;
70 if (own === undefined || own.id !== repo.id) return undefined;
71 const gates = landingStopped(db, own.id);
72 if (gates !== undefined) return gates;
73 // The lane itself stopped (#11507): nothing merging for the limit, or one cause failing many pulls.
74 const lane = laneAlarm(db, new Date());
75 return lane.fired ? { since: lane.since!, gates: 0 } : undefined;
7076 }
7177
7278 /** The one sentence home and the pipeline carry. */
src/contract/openapi/routes/getApiV1TelemetryHome.ts +1 −1
@@ -1,3 +1,3 @@
11 import type { Route } from "../shared.ts";
22
3 export const route: Route = { method: "get", path: "/api/v1/telemetry/home", summary: "Read the figures the home is scored against: landed in 24 h, live version, builders and why idle, spend against the ceiling, what waits on the caller; the owner or an agent holding telemetry:read, with a ro token; every agent read is logged", auth: true, role: "member" };
3 export const route: Route = { method: "get", path: "/api/v1/telemetry/home", summary: "Read the figures the home is scored against: landed in 24 h, live version (live and main: main's newest deploy; production: the version and commit this process runs, null when unknown), builders and why idle, spend against the ceiling, what waits on the caller; the owner or an agent holding telemetry:read, with a ro token; every agent read is logged", auth: true, role: "member" };
src/core/migration-pins/20261111170000-keeper-beat.pin added +1
@@ -0,0 +1,1 @@
1 a57e110db188bf5db091597675159f303ad60ac496f0c841a66d62e09a6df833
src/core/migrations/20261111170000-keeper-beat.ts added +10
@@ -0,0 +1,10 @@
1 // The release tick's last heartbeat (#11507): one row, replaced each time the tick
2 // reports in — when, what it did (cut, skipped, failed) and why. Additive.
3 export default `
4 CREATE TABLE keeper_beat (
5 id INTEGER PRIMARY KEY CHECK (id = 1),
6 at TEXT NOT NULL,
7 outcome TEXT NOT NULL CHECK (outcome IN ('cut', 'skipped', 'failed')),
8 reason TEXT NOT NULL
9 );
10 `;
src/crew/dispatch.ts +6
@@ -47,6 +47,7 @@ import { appendRunLog } from "../runs/log.ts";
4747 import { noteHeld, type HoldCause } from "../runs/held.ts";
4848 import { decideAndRecord } from "./ledger.ts";
4949 import { bugsFirst } from "../flow/bugs.ts";
50 import { forgeFailed } from "../flow/alarm_lane.ts";
5051 import { byDials } from "./team_share.ts";
5152 import { peopleFirst } from "./people_first.ts";
5253 import { inMaintenance } from "../maintenance/state.ts";
@@ -1094,6 +1095,11 @@ function waitOnMain(
10941095 });
10951096 return true;
10961097 }
1098 // One failure line on five pulls within the hour is the forge's own (#11507): no builder is sent, no pull parked.
1099 if (forgeFailed(app.db, run, Date.now())) {
1100 log.info("check failed on a line five pulls share; the forge's failure, not the pull's", { issue: issue.number });
1101 return true;
1102 }
10971103 const main = sharedWithMain(redMain(app.db, repo.id), run);
10981104 if (!main && !sharedWithPulls(app.db, run)) return false;
10991105 if (typeof commitOid === "string" && typeof env === "string") {
src/crew/person_spoke.ts added +13
@@ -0,0 +1,13 @@
1 /** Whether a person, not an agent, has commented on an issue: then they decide, not the sentinel. */
2
3 import type { Database } from "bun:sqlite";
4
5 export function personSpoke(db: Database, issueId: string): boolean {
6 const spoke = db
7 .query<{ n: number }, [string]>(
8 `SELECT COUNT(*) AS n FROM comment c JOIN user u ON u.id = c.author_id
9 WHERE c.issue_id = ?1 AND u.is_agent = 0 AND u.kind = 'user'`,
10 )
11 .get(issueId);
12 return spoke !== null && spoke.n > 0;
13 }
src/crew/sentinel.ts +16 −17
@@ -47,6 +47,10 @@ import {
4747 } from "../checks/shared_red.ts";
4848 import { readRunLogTail } from "../runs/log.ts";
4949 import { clip } from "../text/clip.ts";
50 import { forgeCeiling } from "../flow/ceiling.ts";
51 import { personSpoke } from "./person_spoke.ts";
52 import { closeCaughtUp, mainAhead } from "./sentinel_main_ahead.ts";
53 import { closeLaneMoving, laneStopped } from "./sentinel_lane.ts";
5054 import type { FindingKind } from "../hunt/finding.ts";
5155 import { addLabels, kindLabels } from "../hunt/label.ts";
5256
@@ -400,7 +404,7 @@ export function repeatedErrors(now: number): Finding[] {
400404 * Each cause whose spans total more than ten minutes in the day ending at
401405 * `now`, quoting its sentence. Shorter totals stay in the history.
402406 */
403 export function degradedCauses(db: Database, now: number, forgeRepoId?: string, dataDir?: string): Finding[] {
407 export function degradedCauses(db: Database, now: number, forgeRepoId?: string, dataDir?: string, ceilingUsd: number | null = null): Finding[] {
404408 const from = now - DAY_MS;
405409 const byCause = new Map<
406410 string,
@@ -479,7 +483,7 @@ export function ownCause(cause: string, forgeRepoId: string | undefined): boolea
479483 * at once, not when the day has added up. The finding shares its key with
480484 * `degradedCauses`, so one cause is one issue.
481485 */
482 export function budgetBreaches(db: Database, now: number, forgeRepoId?: string, dataDir?: string): Finding[] {
486 export function budgetBreaches(db: Database, now: number, forgeRepoId?: string, dataDir?: string, ceilingUsd: number | null = null): Finding[] {
483487 const out: Finding[] = [];
484488 for (const s of spans(db, now - DAY_MS, now)) {
485489 if (s.closedAt !== null || !BREACH_CAUSE.test(s.cause) || !ownCause(s.cause, forgeRepoId)) continue;
@@ -569,8 +573,10 @@ function sharedLines(
569573 }
570574
571575 /** Everything the sentinel reads, in one pass. */
572 export function findFaults(db: Database, now = Date.now(), forgeRepoId?: string, dataDir?: string): Finding[] {
576 export function findFaults(db: Database, now = Date.now(), forgeRepoId?: string, dataDir?: string, ceilingUsd: number | null = null): Finding[] {
573577 const found = [
578 ...mainAhead(db, now),
579 ...laneStopped(db, now, ceilingUsd),
574580 ...budgetBreaches(db, now, forgeRepoId, dataDir),
575581 ...heldSlots(db, now),
576582 ...strayTriggers(db),
@@ -770,17 +776,6 @@ function sinceFix(app: App, repo: Repo, actor: User, f: Finding, evidence: strin
770776 return 1;
771777 }
772778
773 /** Whether a person, not an agent, has commented on the issue: then they decide, not the sentinel. */
774 function personSpoke(db: Database, issueId: string): boolean {
775 const spoke = db
776 .query<{ n: number }, [string]>(
777 `SELECT COUNT(*) AS n FROM comment c JOIN user u ON u.id = c.author_id
778 WHERE c.issue_id = ?1 AND u.is_agent = 0 AND u.kind = 'user'`,
779 )
780 .get(issueId);
781 return spoke !== null && spoke.n > 0;
782 }
783
784779 /** The test a shared finding names, read back from its evidence. */
785780 const testOf = (evidence: readonly string[]): string | undefined =>
786781 evidence.find((e) => e.startsWith("test: "))?.slice(6);
@@ -1031,7 +1026,7 @@ export function hunt(
10311026 let commented = 0;
10321027 const [owner = "", name = ""] = repoPath.split("/");
10331028 const repo = getRepo(app.db, owner, name);
1034 const findings = find(app.db, now, repo?.id, app.config.dataDir);
1029 const findings = find(app.db, now, repo?.id, app.config.dataDir, forgeCeiling(app.config));
10351030 const actor = sentinelActor(app.db);
10361031 if (repo !== undefined && actor !== undefined) {
10371032 settleShared(app, repo, actor, now);
@@ -1106,8 +1101,12 @@ export function hunt(
11061101 }
11071102 }
11081103 }
1109 if (repo !== undefined && actor !== undefined)
1110 closeHeld(app, repo, actor, new Set(findings.map((f) => f.key)), now);
1104 if (repo !== undefined && actor !== undefined) {
1105 const active = new Set(findings.map((f) => f.key));
1106 closeHeld(app, repo, actor, active, now);
1107 closeCaughtUp(app, repo, actor, active, now);
1108 closeLaneMoving(app, repo, actor, active, now);
1109 }
11111110 // A hunt with nowhere to file is recorded as such, never as a quiet one.
11121111 const unfiled = repo === undefined || actor === undefined;
11131112 app.db
src/crew/sentinel_lane.ts added +34
@@ -0,0 +1,34 @@
1 /**
2 * The sentinel's half of the lane-stopped alarm (#11507): nothing merging for
3 * the limit, or one cause failing many pulls, is one finding; its issue closes
4 * at the next hunt after the lane moves, unless a person spoke on it.
5 */
6
7 import type { Database } from "bun:sqlite";
8 import type { App } from "../app.ts";
9 import type { Repo } from "../core/repos.ts";
10 import type { User } from "../core/users.ts";
11 import { laneAlarm } from "../flow/alarm_lane.ts";
12 import { closeCaughtUp } from "./sentinel_main_ahead.ts";
13 import type { Finding } from "./sentinel.ts";
14
15 export const LANE_KEY = "lane-stopped";
16
17 export function laneStopped(db: Database, now: number, ceilingUsd: number | null): Finding[] {
18 const a = laneAlarm(db, new Date(now), { ceilingUsd });
19 if (!a.fired) return [];
20 return [{
21 key: LANE_KEY,
22 title: a.forge === true ? "One failure is stopping many pulls: the forge's, not theirs" : "The lane has stopped: nothing has merged and nothing said so",
23 evidence: a.facts ?? [a.says],
24 pillar: "P0 Trust: the forge says when it is not well, and is well again",
25 kind: "defect",
26 lookFor: "A pull merged since, and no one failure line failing five pulls within an hour.",
27 urgent: true,
28 }];
29 }
30
31 /** Close the open lane issue once the lane moves; `active` is this hunt's finding keys. */
32 export function closeLaneMoving(app: App, repo: Repo, actor: User, active: ReadonlySet<string>, now: number): void {
33 closeCaughtUp(app, repo, actor, active, now, LANE_KEY, "The lane moved: a pull merged and no one failure is stopping many");
34 }
src/crew/sentinel_main_ahead.ts added +65
@@ -0,0 +1,65 @@
1 /**
2 * The sentinel's half of the main-ahead alarm (#11507): main green and ahead
3 * of production past the setting is one finding, and its issue closes itself
4 * at the next hunt after production catches up — unless a person spoke on it,
5 * then it is theirs to close.
6 */
7
8 import type { Database } from "bun:sqlite";
9 import type { App } from "../app.ts";
10 import { addComment, setIssueState } from "../actions/issues.ts";
11 import { getIssueById } from "../core/issues.ts";
12 import type { Repo } from "../core/repos.ts";
13 import type { User } from "../core/users.ts";
14 import { mainAheadAlarm } from "../flow/alarm_main_ahead.ts";
15 import { createLog } from "../kernel/log.ts";
16 import { personSpoke } from "./person_spoke.ts";
17 import type { Finding } from "./sentinel.ts";
18
19 const log = createLog("sentinel");
20
21 export const MAIN_AHEAD_KEY = "main-ahead";
22
23 export function mainAhead(db: Database, now: number): Finding[] {
24 const a = mainAheadAlarm(db, new Date(now));
25 if (!a.fired) return [];
26 return [{
27 key: MAIN_AHEAD_KEY,
28 title: "Main is green and ahead of production: the release tick has not cut",
29 evidence: a.facts ?? [a.says],
30 pillar: "P0 Trust: the forge says when it is not well, and is well again",
31 kind: "defect",
32 lookFor: "Production runs main's head, or main has been ahead of it for under the limit.",
33 urgent: true,
34 }];
35 }
36
37 /** Close the open main-ahead issue once its finding is gone; `active` is this hunt's finding keys. */
38 export function closeCaughtUp(
39 app: App,
40 repo: Repo,
41 actor: User,
42 active: ReadonlySet<string>,
43 now: number,
44 key: string = MAIN_AHEAD_KEY,
45 cleared: string = "Production caught up with main",
46 ): void {
47 if (active.has(key)) return;
48 const open = app.db
49 .query<{ issue_id: string }, [string]>(
50 `SELECT t.issue_id FROM sentinel_finding t JOIN issue i ON i.id = t.issue_id WHERE i.state = 'open' AND t.key = ?1`,
51 )
52 .all(key);
53 for (const t of open) {
54 try {
55 if (personSpoke(app.db, t.issue_id)) continue;
56 const issue = getIssueById(app.db, t.issue_id);
57 if (issue === undefined || issue.repoId !== repo.id) continue;
58 const said = addComment(app, actor, repo, issue, `${cleared} at ${new Date(now).toISOString()}. Closing as done; if it falls behind again it is filed again.`);
59 if (!said.ok) continue;
60 setIssueState(app, actor, repo, issue, "closed", { reason: "done", system: true });
61 } catch (error) {
62 log.error("sentinel close failed", { key, error: String(error) });
63 }
64 }
65 }
src/crew/stale_reds.ts +5 −1
@@ -28,6 +28,7 @@ import { closesFromBody, listOpenPulls, pullDir, type Pull } from "../core/pulls
2828 import type { Repo } from "../core/repos.ts";
2929 import { automaticRunsSince, getRun, INFRA_SUMMARY, type Run } from "../core/runs.ts";
3030 import { getUserById } from "../core/users.ts";
31 import { forgeFailed } from "../flow/alarm_lane.ts";
3132 import { QUIET_MS, SWEEP_REASON } from "../flow/stale_reds.ts";
3233 import { mergeable, resolveOid } from "../git/merge.ts";
3334 import { createLog } from "../kernel/log.ts";
@@ -159,7 +160,9 @@ export async function sweepStaleReds(
159160 }
160161
161162 const since = lastHumanWord(app, issue, agent);
162 if (automaticRunsSince(app.db, issue.id, since) >= CHECK_FAILED_LIMIT) {
163 // The forge's failure (#11507) is never a reason to park: the pull is checked again, not reworked.
164 const forge = forgeFailed(app.db, red.run, now.getTime());
165 if (!forge && automaticRunsSince(app.db, issue.id, since) >= CHECK_FAILED_LIMIT) {
163166 const why = `the check stayed red through ${CHECK_FAILED_LIMIT} automatic tries: ${failingLine(red.run)}`;
164167 if (await parkPullTold(app, repo, pull, why, since)) moves.push({ pull: pull.number, did: "parked" });
165168 continue;
@@ -178,6 +181,7 @@ export async function sweepStaleReds(
178181 continue;
179182 }
180183 if (again.state !== "failed" || again.summary.startsWith(INFRA_SUMMARY)) continue;
184 if (forgeFailed(app.db, again, now.getTime())) continue;
181185 // Said as the builder, not as a person: a person's word restarts the tries.
182186 addComment(
183187 app,
src/flow/alarm_lane.ts added +188
@@ -0,0 +1,188 @@
1 /**
2 * The "lane has stopped" flow alarm (#11507). On 2026-11-12 nothing merged for
3 * 12 h 51 min and nothing said so: main was not ahead of production, so the
4 * main-ahead alarm stayed quiet, and every check was being stopped for CPU.
5 * Two states raise this one alarm for the forge's own repository:
6 *
7 * - the lane has stopped: pulls are open or work is ready, the crew is not
8 * at its ceiling, and nothing has merged for `flow.lane_minutes` (90);
9 * - one cause fails many pulls: the same failure line (numbers masked) failed
10 * checks on {@link FORGE_PULLS} different pulls within an hour. That is the
11 * forge's failure, not the builders': `forgeFailed` is what the crew asks
12 * before it reworks or parks a pull for it.
13 *
14 * A failure line is the first line of a check's summary, for a check that
15 * named no failing test (a stop for CPU, a budget, a lost machine); a named
16 * test is `checks/shared_red.ts`'s. Pure of the clock.
17 */
18
19 import type { Database } from "bun:sqlite";
20 import { spentTodayEverywhere } from "../crew/autonomy.ts";
21 import type { Run } from "../core/runs.ts";
22 import { getRepo } from "../core/repos.ts";
23 import { agoWords } from "../keeper/beat.ts";
24 import { ownRepoPath } from "../kernel/own_repo.ts";
25 import { ceilingReached } from "./ceiling.ts";
26 import { readPipeline } from "./pipeline.ts";
27
28 export const LANE_SETTING = "flow.lane_minutes";
29 export const LANE_DEFAULT_MIN = 90;
30 /** Different pulls failing one line within {@link FORGE_WINDOW_MS} make it the forge's. */
31 export const FORGE_PULLS = 5;
32 export const FORGE_WINDOW_MS = 3_600_000;
33 const LINE_MAX = 200;
34
35 export interface LaneAlarm {
36 readonly fired: boolean;
37 /** When the lane last merged, or the first failure of the shared line; set when fired. */
38 readonly since?: string;
39 readonly says: string;
40 /** The commonest failure line, verbatim, when one exists. */
41 readonly line?: string | undefined;
42 /** Whether one cause is failing many pulls. */
43 readonly forge?: boolean;
44 readonly facts?: readonly string[];
45 }
46
47 /** The minutes without a merge before the alarm; a whole number from 1 to 10080 is the owner's, else 90. */
48 export function laneMinutes(db: Database): number {
49 const row = db.query<{ value: string }, [string]>("SELECT value FROM forge_setting WHERE key = ?1").get(LANE_SETTING);
50 const n = row === null ? NaN : Number(row.value);
51 return Number.isInteger(n) && n >= 1 && n <= 10_080 ? n : LANE_DEFAULT_MIN;
52 }
53
54 /** The same line with every number as `#`, so `16.0 cores for 30s` and `15.2 cores for 31s` are one cause. */
55 export const maskLine = (line: string): string => line.replace(/\d+(?:\.\d+)?/g, "#");
56
57 interface Failure {
58 readonly issue: string;
59 readonly at: string;
60 readonly line: string;
61 }
62
63 /** A check's failure line, or "" when it named a failing test (not this alarm's) or said nothing. */
64 function lineOf(summary: string, detail: string): string {
65 try {
66 const names = (JSON.parse(detail) as { names?: unknown } | null)?.names;
67 if (Array.isArray(names) && names.length > 0) return "";
68 } catch {
69 // An unreadable detail names no test.
70 }
71 const first = summary.trim().split("\n")[0] ?? "";
72 return first.replace(/^check failed: /, "").slice(0, LINE_MAX);
73 }
74
75 /** Failed pull checks of the repository finished since `since`, newest first, bounded. */
76 function failuresSince(db: Database, repoId: string, since: string): Failure[] {
77 return db
78 .query<{ issue_id: string | null; finished_at: string; summary: string; check_detail: string }, [string, string]>(
79 `SELECT issue_id, finished_at, summary, check_detail FROM run
80 WHERE repo_id = ?1 AND kind = 'check' AND state = 'failed' AND finished_at >= ?2
81 ORDER BY finished_at DESC LIMIT 2000`,
82 )
83 .all(repoId, since)
84 .map((r) => ({ issue: r.issue_id ?? "", at: r.finished_at, line: lineOf(r.summary, r.check_detail) }));
85 }
86
87 interface Group {
88 readonly masked: string;
89 /** The newest instance, verbatim. */
90 readonly line: string;
91 count: number;
92 readonly pulls: Set<string>;
93 since: string;
94 }
95
96 function groups(failures: readonly Failure[]): Group[] {
97 const by = new Map<string, Group>();
98 for (const f of failures) {
99 if (f.line === "") continue;
100 const m = maskLine(f.line);
101 const g = by.get(m) ?? { masked: m, line: f.line, count: 0, pulls: new Set<string>(), since: f.at };
102 g.count++;
103 if (f.issue !== "") g.pulls.add(f.issue);
104 if (f.at < g.since) g.since = f.at;
105 by.set(m, g);
106 }
107 return [...by.values()].sort((a, b) => b.count - a.count || (a.masked < b.masked ? -1 : 1));
108 }
109
110 /** When a pull of the repository last merged. */
111 function lastMerge(db: Database, repoId: string): string | null {
112 return (
113 db
114 .query<{ at: string | null }, [string]>(`SELECT MAX(p.merged_at) AS at FROM pull p JOIN issue i ON i.id = p.issue_id WHERE i.repo_id = ?1`)
115 .get(repoId)?.at ?? null
116 );
117 }
118
119 /** The failure lines that failed {@link FORGE_PULLS} pulls' checks within the hour and since the last merge (a merge proves the lane): the forge's own. */
120 function forgeGroups(db: Database, repoId: string, now: number): Group[] {
121 const hour = new Date(now - FORGE_WINDOW_MS).toISOString();
122 const merged = lastMerge(db, repoId);
123 const since = merged !== null && merged > hour ? merged : hour;
124 return groups(failuresSince(db, repoId, since)).filter((g) => g.pulls.size >= FORGE_PULLS);
125 }
126
127 /** Whether this failed check's line is one the forge failed on many pulls: the crew neither reworks nor parks for it. */
128 export function forgeFailed(db: Database, run: Run | undefined, now: number): boolean {
129 if (run === undefined || run.kind !== "check" || run.state !== "failed") return false;
130 const line = lineOf(run.summary, run.checkDetail === null || run.checkDetail === undefined ? "{}" : JSON.stringify(run.checkDetail));
131 if (line === "") return false;
132 const m = maskLine(line);
133 return forgeGroups(db, run.repoId, now).some((g) => g.masked === m);
134 }
135
136 function ownRepoId(db: Database): string | undefined {
137 const [owner, name] = ownRepoPath().split("/");
138 return owner && name ? getRepo(db, owner, name)?.id : undefined;
139 }
140
141 const count = (n: number, noun: string) => `${n} ${noun}${n === 1 ? "" : "s"}`;
142
143 export function laneAlarm(db: Database, now: Date, forge: { readonly ceilingUsd: number | null } = { ceilingUsd: null }): LaneAlarm {
144 const quiet = { fired: false, says: "Pulls are landing, or there is nothing to land, or the crew is paused on purpose." } as const;
145 const repoId = ownRepoId(db);
146 if (repoId === undefined) return quiet;
147 const at = now.getTime();
148 const shared = forgeGroups(db, repoId, at)[0];
149 const last = lastMerge(db, repoId);
150 const open = db
151 .query<{ n: number }, [string]>(`SELECT COUNT(*) AS n FROM issue WHERE repo_id = ?1 AND kind = 'pull' AND state = 'open'`)
152 .get(repoId)?.n ?? 0;
153 const limit = laneMinutes(db) * 60_000;
154 let stopped = false;
155 let wait = 0;
156 if (last !== null && (at - Date.parse(last) > limit)) {
157 wait = at - Date.parse(last);
158 const ready = open > 0 ? open : (readPipeline(db, now).find((s) => s.key === "ready")?.waiting ?? 0);
159 stopped = ready > 0 && !ceilingReached(spentTodayEverywhere(db, now), forge.ceilingUsd);
160 }
161 if (!stopped && shared === undefined) return quiet;
162
163 const parts: string[] = [];
164 const facts: string[] = [];
165 let line: string | undefined;
166 let since: string;
167 if (stopped) {
168 const failed = failuresSince(db, repoId, last!);
169 const top = groups(failed)[0];
170 line = top?.line;
171 since = last!;
172 parts.push(
173 `Nothing has merged for ${agoWords(wait).replace(" ago", "")} (since ${last}) with ${count(open, "pull")} open; ${failed.length} ${failed.length === 1 ? "check" : "checks"} failed in that span` +
174 (top === undefined ? "." : `, ${top.count} of them with: ${top.line}`),
175 );
176 facts.push(`last merge at ${last}`, `${count(open, "pull")} open`, `${failed.length} checks failed since the last merge`);
177 if (top !== undefined) facts.push(`commonest failure, ${top.count} times: ${top.line}`);
178 } else since = shared!.since;
179 if (shared !== undefined) {
180 line = line ?? shared.line;
181 if (stopped && shared.masked !== maskLine(line)) facts.push(`failed ${count(shared.pulls.size, "pull")} within the hour: ${shared.line}`);
182 parts.push(
183 `One cause is failing ${count(shared.pulls.size, "pull")} within the hour: ${shared.line}. That is the forge's failure, not the builders': their reworks and parks for it are held`,
184 );
185 facts.push(`forge failure on ${count(shared.pulls.size, "pull")} within the hour since ${shared.since}: ${shared.line}`);
186 }
187 return { fired: true, since, says: `${parts.join(". ")}.`.replace(/\.\.$/, "."), line, forge: shared !== undefined, facts };
188 }
src/flow/alarm_main_ahead.ts added +91
@@ -0,0 +1,91 @@
1 /**
2 * The "main is ahead of production" flow alarm (#11507): the forge's own main
3 * head is proven green (its last finished check passed), differs from the
4 * commit production runs, and has for longer than the setting
5 * `release.ahead_minutes` (default 90). On 2026-11-11 production stood still
6 * for 5 h 50 min with main green and nothing said so. Names production's
7 * version and commit, how far main is ahead (commits and pulls), how long, and
8 * the last time the release tick was heard from. Pure of the clock.
9 */
10
11 import type { Database } from "bun:sqlite";
12 import { getRepo } from "../core/repos.ts";
13 import { agoWords, lastBeat } from "../keeper/beat.ts";
14 import { ownRepoPath } from "../kernel/own_repo.ts";
15 import { running } from "../kernel/version.ts";
16 import { waitingRows } from "../release/landings.ts";
17
18 export const AHEAD_SETTING = "release.ahead_minutes";
19 export const AHEAD_DEFAULT_MIN = 90;
20
21 export interface MainAheadAlarm {
22 readonly fired: boolean;
23 /** When main first moved past production; while it is ahead. */
24 readonly since?: string;
25 readonly says: string;
26 /** Facts for the issue and the ask; only when fired. */
27 readonly facts?: readonly string[];
28 }
29
30 /** The minutes main may be ahead and green before the alarm; a whole number of 1 or more is the owner's, else 90. */
31 export function aheadMinutes(db: Database): number {
32 const row = db.query<{ value: string }, [string]>("SELECT value FROM forge_setting WHERE key = ?1").get(AHEAD_SETTING);
33 const n = row === null ? NaN : Number(row.value);
34 return Number.isInteger(n) && n >= 1 && n <= 10_080 ? n : AHEAD_DEFAULT_MIN;
35 }
36
37 /** The commit of the last finished check of the forge's default branch, when that check passed. */
38 function greenHead(db: Database): string | undefined {
39 const [owner, name] = ownRepoPath().split("/");
40 const repo = owner && name ? getRepo(db, owner, name) : undefined;
41 if (repo === undefined) return undefined;
42 const last = db
43 .query<{ state: string; commit_oid: string }, [string, string]>(
44 `SELECT state, commit_oid FROM run
45 WHERE repo_id = ?1 AND kind = 'check' AND issue_id IS NULL AND env = ?2 AND state IN ('ok', 'failed')
46 ORDER BY id DESC LIMIT 1`,
47 )
48 .get(repo.id, repo.defaultBranch);
49 return last?.state === "ok" ? last.commit_oid : undefined;
50 }
51
52 export function mainAheadAlarm(db: Database, now: Date): MainAheadAlarm {
53 const quiet = { fired: false, says: "Production runs main's head, or main's head is not proven green." } as const;
54 // The cheap checks first: this runs on every /flow and home request, and the waiting pulls are read only once main's green head is not production's.
55 const prod = running();
56 if (prod.commit === null) return quiet;
57 const head = greenHead(db);
58 if (head === undefined || head.startsWith(prod.commit) || prod.commit.startsWith(head)) return quiet;
59 const waiting = waitingRows(db);
60 const first = waiting[0];
61 if (first === undefined) return quiet;
62 const wait = now.getTime() - Date.parse(first.at);
63 const limit = aheadMinutes(db) * 60_000;
64 if (!(wait > limit)) return { fired: false, since: first.at, says: `Main is ahead of production ${agoWords(wait).replace(" ago", "")}; not yet due.` };
65 const commits = new Set(waiting.map((r) => r.oid).filter((o) => o !== "")).size || waiting.length;
66 const pulls = waiting.length;
67 const beat = lastBeat(db);
68 const tick = beat === null ? "never" : agoWords(now.getTime() - Date.parse(beat.at));
69 const how = `${commits} ${commits === 1 ? "commit" : "commits"} and ${pulls} ${pulls === 1 ? "pull" : "pulls"}`;
70 const forHow = agoWords(wait).replace(" ago", "");
71 const facts = [
72 `production runs ${prod.version} at ${prod.commit.slice(0, 7)}`,
73 `main is green at ${head.slice(0, 7)}, ${how} ahead`,
74 `ahead since ${first.at}`,
75 // Absolute times: the evidence is commented again only when it changes, not as the clock moves.
76 // The tick's reason stays out: these facts are a Sentinel issue's, which may be public; /releases shows it to its readers.
77 `the release tick was last heard from ${beat === null ? "never" : `${beat.at} (${beat.outcome})`}`,
78 ];
79 return {
80 fired: true,
81 since: first.at,
82 says: `Main is green and ${how} ahead of production (${prod.version} at ${prod.commit.slice(0, 7)}) for ${forHow}; the release tick was last heard from ${tick}.`,
83 facts,
84 };
85 }
86
87 /** Main's state against production as one sentence for /releases and the terminal; undefined when main is not ahead and green. */
88 export function aheadLine(db: Database, now: Date): string | undefined {
89 const a = mainAheadAlarm(db, now);
90 return a.fired || a.since !== undefined ? a.says : undefined;
91 }
src/flow/alarms.ts +7 −1
@@ -16,6 +16,8 @@ import { alarmBackup } from "./alarm_backup.ts";
1616 import { alarmFromApp } from "./alarm_from_app.ts";
1717 import { alarmPaidOrder } from "./alarm_paid_order.ts";
1818 import { alarmIdleCrew } from "./alarm_idle_crew.ts";
19 import { laneAlarm } from "./alarm_lane.ts";
20 import { mainAheadAlarm } from "./alarm_main_ahead.ts";
1921 import { noReleaseAlarm } from "./alarm_no_release.ts";
2022 import { redMainAlarm } from "./alarm_red_main.ts";
2123 import { alarmRepeat } from "./alarm_repeat.ts";
@@ -23,7 +25,7 @@ import { alarmSilent } from "./alarm_silent.ts";
2325 import { alarmStuck } from "./alarm_stuck.ts";
2426
2527 export interface FiredAlarm {
26 readonly key: "landing_stopped" | "red_main" | "stuck" | "idle_crew" | "no_release" | "bug_clock" | "silent" | "repeat" | "from_app" | "unreachable" | "backup" | "paid_order";
28 readonly key: "landing_stopped" | "red_main" | "stuck" | "idle_crew" | "no_release" | "main_ahead" | "lane_stopped" | "bug_clock" | "silent" | "repeat" | "from_app" | "unreachable" | "backup" | "paid_order";
2729 readonly says: string;
2830 readonly since: string;
2931 /** A same-origin path to the item the alarm names. */
@@ -70,6 +72,10 @@ export function firedAlarms(
7072 }
7173 const release = noReleaseAlarm(db, now);
7274 if (release.fired) out.push({ key: "no_release", says: release.says, since: release.since!, href: "/releases" });
75 const ahead = mainAheadAlarm(db, now);
76 if (ahead.fired) out.push({ key: "main_ahead", says: ahead.says, since: ahead.since!, href: "/releases" });
77 const lane = laneAlarm(db, now, { ceilingUsd: opts.ceilingUsd ?? null });
78 if (lane.fired) out.push({ key: "lane_stopped", says: lane.says, since: lane.since!, href: "/flow" });
7379 const bug = bugClockAlarm(db, now, opts.publicOnly ? { publicOnly: true } : {});
7480 if (bug.fired) out.push({ key: "bug_clock", says: bug.says, since: bug.since!, href: bug.href });
7581 return out;
src/http/mcp/tools/release_keeper.ts +1 −1
@@ -4,7 +4,7 @@ import { bad, good, type Tool } from "../shared.ts";
44
55 export const tool: Tool = {
66 name: "release_keeper",
7 doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, commit (short sha), stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason, landedToLiveSeconds — from the merge of its pull to the new version answering, null until it has (#8042)) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; for whoever may read /releases",
7 doc: "`{}`: what the release keeper (#7557) did: its releases newest first (version, commit (short sha), stage — rolling, verifying, verified or rolled_back —, the known-good version it would roll back to, roll probes and non-200s, probes and pages under load, worst ms, seconds dark, reason, landedToLiveSeconds — from the merge of its pull to the new version answering, null until it has (#8042)) and its holds (kind failed or owner, version, reason, probe numbers, the pull that said `release hold N`, lifted); `blocked` says what stops the next cut; `heartbeat` is the release tick's last report (at, outcome cut/skipped/failed, reason, silent when over an hour old, #11507); for whoever may read /releases",
88 description: "The release keeper's releases, holds and what blocks the next cut.",
99 inputSchema: { type: "object", properties: {} },
1010 writes: false,
src/keeper/beat.ts added +85
@@ -0,0 +1,85 @@
1 /**
2 * The release tick's heartbeat (#11507): each time the tick runs it reports
3 * what it did — cut, skipped and why, or failed and why — and the forge keeps
4 * the last. The tick runs outside the forge's own judgment, so when it stops
5 * this is the only trace; `release_keeper`, /releases and the terminal's
6 * release keeper screen show it, and a beat older
7 * than {@link SILENT_MS} is said plainly. Plain rows in, plain rows out.
8 */
9
10 import type { Database } from "bun:sqlite";
11
12 export type Outcome = "cut" | "skipped" | "failed";
13
14 /** No heartbeat for this long is itself said. */
15 export const SILENT_MS = 60 * 60_000;
16 const REASON_MAX = 300;
17
18 export interface Beat {
19 readonly at: string;
20 readonly outcome: Outcome;
21 readonly reason: string;
22 }
23
24 export function recordBeat(db: Database, outcome: Outcome, reason: string, at: Date = new Date()): Beat {
25 const beat: Beat = { at: at.toISOString(), outcome, reason: reason.replace(/\s+/g, " ").trim().slice(0, REASON_MAX) };
26 db.query(
27 `INSERT INTO keeper_beat (id, at, outcome, reason) VALUES (1, ?1, ?2, ?3)
28 ON CONFLICT (id) DO UPDATE SET at = excluded.at, outcome = excluded.outcome, reason = excluded.reason`,
29 ).run(beat.at, beat.outcome, beat.reason);
30 return beat;
31 }
32
33 export function lastBeat(db: Database): Beat | null {
34 return db.query<Beat, []>("SELECT at, outcome, reason FROM keeper_beat WHERE id = 1").get() ?? null;
35 }
36
37 /** What one tick did, as the keeper says it: the line and its outcome, never read back from the words. */
38 export interface Step {
39 readonly line: string;
40 readonly outcome: Outcome;
41 }
42
43 /** Run one tick and record its beat; a tick that throws is recorded as failed and the error goes on. */
44 export async function stepWithBeat(db: Database, step: () => Promise<Step>, now: () => Date = () => new Date()): Promise<string> {
45 let done: Step;
46 try {
47 done = await step();
48 } catch (error) {
49 writeBeat(db, "failed", `step failed: ${error instanceof Error ? error.message : String(error)}`, now());
50 throw error;
51 }
52 writeBeat(db, done.outcome, done.line, now());
53 return done.line;
54 }
55
56 /** The beat is a trace, not the tick: a write that fails is said and never turns a good tick into a failed one. */
57 function writeBeat(db: Database, outcome: Outcome, reason: string, at: Date): void {
58 try {
59 recordBeat(db, outcome, reason, at);
60 } catch (error) {
61 console.error(`keeper heartbeat not recorded: ${error instanceof Error ? error.message : String(error)}`);
62 }
63 }
64
65 /** "never", "4 min ago", "2 h 5 min ago". */
66 export function agoWords(ms: number): string {
67 const min = Math.max(0, Math.floor(ms / 60_000));
68 if (min < 60) return `${min} min ago`;
69 return `${Math.floor(min / 60)} h ${min % 60} min ago`;
70 }
71
72 /** The heartbeat as one sentence, saying plainly when there is none or it is old. */
73 export function beatLine(beat: Beat | null, now: Date): string {
74 if (beat === null) return "The release tick has never reported a heartbeat.";
75 const age = now.getTime() - Date.parse(beat.at);
76 const said = `The release tick last reported ${agoWords(age)} (${beat.outcome}): ${beat.reason}`;
77 return age > SILENT_MS ? `No heartbeat from the release tick for ${agoWords(age).replace(" ago", "")}. ${said}` : said;
78 }
79
80 /** The heartbeat as a `dl` row's value: outcome, age and reason; "never reported" before the first. */
81 export function beatFact(beat: Beat | null, now: Date): string {
82 if (beat === null) return "never reported";
83 const age = now.getTime() - Date.parse(beat.at);
84 return `${beat.outcome}, ${agoWords(age)}${age > SILENT_MS ? " (silent)" : ""}: ${beat.reason}`;
85 }
src/keeper/keeper.ts +23 −11
@@ -37,6 +37,7 @@ import { stallReason, STALL_HOLD_MS } from "../perf/boot_stall.ts";
3737 import { countRollProbe, dropCut, finish, getKeeperRelease, inFlight, keepStall, keepTally, listKeeperReleases, markLive, releaseBefore, recordCut, type KeeperRelease } from "./record.ts";
3838 import { rollBackToNamed } from "./rollback.ts";
3939 import type { Probe, Tally, Verdict } from "./verify.ts";
40 import type { Step } from "./beat.ts";
4041
4142 /** A release not live this long after its cut never rolled: 3 h (the tag's check may take 2 h, the build some more). */
4243 export const ROLL_DEADLINE_MS = 3 * 3_600_000;
@@ -116,8 +117,8 @@ function fileDark(p: KeeperPorts, r: KeeperRelease): number | undefined {
116117 const verifying = new WeakMap<KeeperPorts, number>();
117118
118119 /** The verify of an in-flight release this process is the new version of, to its end. */
119 async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<string> {
120 if (verifying.has(p)) return `verifying ${r.version}`;
120 async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<Step> {
121 if (verifying.has(p)) return { line: `verifying ${r.version}`, outcome: "skipped" };
121122 verifying.set(p, r.id);
122123 try {
123124 const verdict = await p.verify((t) => keepTally(p.db, r.id, t, r.rollNon200));
@@ -131,9 +132,9 @@ async function verifyAndDecide(p: KeeperPorts, r: KeeperRelease): Promise<string
131132 const lifted = liftFixed(p.db, r.version, p.now());
132133 p.bus.append({ kind: "release.verified_live", repoId: p.repo.id, payload: { ...numbers, holdsLifted: lifted } });
133134 fileDark(p, getKeeperRelease(p.db, r.id)!);
134 return `${r.version} verified live: ${kept.probes} probes, worst ${kept.worstMs} ms, ${kept.secondsDark} s dark${stalled === undefined ? "" : `; ${stalled}`}`;
135 return { line: `${r.version} verified live: ${kept.probes} probes, worst ${kept.worstMs} ms, ${kept.secondsDark} s dark${stalled === undefined ? "" : `; ${stalled}`}`, outcome: "cut" };
135136 }
136 return await rollBack(p, kept, verdict.reason);
137 return { line: await rollBack(p, kept, verdict.reason), outcome: "failed" };
137138 } finally {
138139 verifying.delete(p);
139140 }
@@ -171,6 +172,16 @@ async function rollBack(p: KeeperPorts, r: KeeperRelease, reason: string): Promi
171172
172173 /** One step of the keeper; the line says what it did or why it did nothing. */
173174 export async function tick(p: KeeperPorts): Promise<string> {
175 return (await tickStep(p)).line;
176 }
177
178 /**
179 * One step with its outcome, said where it is decided (#11507): `cut` when it
180 * moved a release (cut one, or verified one live), `failed` when something
181 * refused or broke (a verify, a tag, the deployer's admit, main unreadable),
182 * else `skipped` with the reason.
183 */
184 export async function tickStep(p: KeeperPorts): Promise<Step> {
174185 const holds = standingHolds(p.db);
175186 findCauses(p, holds);
176187 const flying = inFlight(p.db);
@@ -182,14 +193,14 @@ export async function tick(p: KeeperPorts): Promise<string> {
182193 // A cut recorded but never tagged (the process died between the two): tag it now, or drop it.
183194 if (flying.stage === "rolling" && (await p.newest())?.version !== flying.version) {
184195 const refused = await tagged(p, flying);
185 if (refused !== undefined) return refused;
196 if (refused !== undefined) return { line: refused, outcome: "failed" };
186197 }
187198 if (flying.stage === "rolling" && p.now() - Date.parse(flying.cutAt) > ROLL_DEADLINE_MS)
188 return rollBack(p, flying, `${flying.version} was not live ${ROLL_DEADLINE_MS / 3_600_000} h after its cut`);
199 return { line: await rollBack(p, flying, `${flying.version} was not live ${ROLL_DEADLINE_MS / 3_600_000} h after its cut`), outcome: "failed" };
189200 // A promote that did not take (the API refused, or something set the image back) is sent again.
190201 const want = refFor(p, flying.knownGood.image, flying.version, flying.sha);
191202 if (flying.stage === "rolling" && (await p.deployer.image()) !== want) await promote(p, flying.version, want);
192 return `${flying.version} is ${flying.stage}`;
203 return { line: `${flying.version} is ${flying.stage}`, outcome: "skipped" };
193204 }
194205 const main = await p.main();
195206 const newest = await p.newest();
@@ -207,19 +218,20 @@ export async function tick(p: KeeperPorts): Promise<string> {
207218 now: p.now(),
208219 ...(p.minIntervalMs === undefined ? {} : { minIntervalMs: p.minIntervalMs }),
209220 });
210 if (!decision.cut || main === undefined) return decision.cut ? "main is unreadable" : decision.why;
221 if (!decision.cut) return { line: decision.why, outcome: "skipped" };
222 if (main === undefined) return { line: "main is unreadable", outcome: "failed" };
211223 // A tree the deployer cannot release (dependencies changed, preflight red) is not cut: no tag, no row.
212224 const refused = await p.deployer.admits?.(main);
213 if (refused !== undefined) return `not cut: ${refused}`;
225 if (refused !== undefined) return { line: `not cut: ${refused}`, outcome: "failed" };
214226 const image = await p.deployer.image();
215227 const knownGood = { version: p.live(), image };
216228 // The row first, the tag second: a crash between them leaves a `rolling` row
217229 // whose tag the next tick sends (`tagged`), never a tag the keeper has no row for.
218230 const r = recordCut(p.db, { version: decision.version, sha: main, knownGood, at: p.now() });
219231 const tagRefused = await tagged(p, r);
220 if (tagRefused !== undefined) return tagRefused;
232 if (tagRefused !== undefined) return { line: tagRefused, outcome: "failed" };
221233 const sent = await promote(p, r.version, refFor(p, image, r.version, main));
222 return `cut ${r.version} at ${main.slice(0, 12)}; ${sent}; known good: ${knownGood.version}`;
234 return { line: `cut ${r.version} at ${main.slice(0, 12)}; ${sent}; known good: ${knownGood.version}`, outcome: "cut" };
223235 }
224236
225237 const refFor = (p: KeeperPorts, image: string, version: string, sha: string): string =>
src/keeper/landed.ts +15 −1
@@ -7,6 +7,7 @@
77 */
88
99 import type { Database } from "bun:sqlite";
10 import { beatLine, lastBeat, SILENT_MS, type Beat } from "./beat.ts";
1011 import { blocking, listHolds, standingHolds, type Hold } from "./hold.ts";
1112 import { listKeeperReleases, type KeeperRelease } from "./record.ts";
1213
@@ -55,11 +56,24 @@ export function withLandedToLive(db: Database, releases: readonly KeeperRelease[
5556 return releases.map((r) => ({ ...r, commit: r.sha.slice(0, 7), landedToLiveSeconds: landedToLive(db, r) ?? null }));
5657 }
5758
59 /** The tick's last heartbeat with its sentence and whether it has gone quiet (#11507); null before the first. */
60 export type Heartbeat = Beat & { readonly silent: boolean; readonly says: string };
61
62 export function heartbeatOf(db: Database, now: Date = new Date()): Heartbeat | null {
63 const beat = lastBeat(db);
64 if (beat === null) return null;
65 return { ...beat, silent: now.getTime() - Date.parse(beat.at) > SILENT_MS, says: beatLine(beat, now) };
66 }
67
5868 /** What the keeper did and what stops the next cut: one answer for MCP `release_keeper` and `GET /api/v1/keeper`. */
59 export function keeperAnswer(db: Database): { blocked: string | null; releases: LandedRelease[]; holds: Hold[] } {
69 export function keeperAnswer(
70 db: Database,
71 now: Date = new Date(),
72 ): { blocked: string | null; releases: LandedRelease[]; holds: Hold[]; heartbeat: Heartbeat | null } {
6073 return {
6174 blocked: blocking(standingHolds(db)) ?? null,
6275 releases: withLandedToLive(db, listKeeperReleases(db)),
6376 holds: listHolds(db),
77 heartbeat: heartbeatOf(db, now),
6478 };
6579 }
src/main.ts +3 −2
@@ -97,7 +97,8 @@ import { proveRunning } from "./release/prove.ts";
9797 import { CHECK_EVERY_MS, setBehind, watchBehind } from "./release/behind.ts";
9898 import { readLandings } from "./release/landings.ts";
9999 import { ownTagsReader } from "./release/own_tags.ts";
100 import { rollProbe, tick as keeperTick } from "./keeper/keeper.ts";
100 import { rollProbe, tickStep as keeperTick } from "./keeper/keeper.ts";
101 import { stepWithBeat } from "./keeper/beat.ts";
101102 import { keeperEnv, keeperPorts } from "./keeper/wire.ts";
102103 import { snapshotPlace } from "./actions/health.ts";
103104 import { capacityDefault, egressEnvIgnored } from "./core/capacity.ts";
@@ -824,7 +825,7 @@ if (keeperOn.on) {
824825 else {
825826 let said = "";
826827 const step = () =>
827 void keeperTick(ports)
828 void stepWithBeat(app.db, () => keeperTick(ports))
828829 .then((line) => {
829830 if (line !== said) app.log.info("release keeper", { line });
830831 said = line;
src/release/landings.ts +8
@@ -134,6 +134,14 @@ export function waitingSince(db: Database): string | undefined {
134134 return onItsWay.at(-1)?.at;
135135 }
136136
137 /** The merged pulls not yet in the running commit, oldest first, with their merge commits; empty when nothing was read (#11507). */
138 export function waitingRows(db: Database): MergedRow[] {
139 if (kept === undefined) return [];
140 const rows = mergedRows(db);
141 const way = new Set(split(rows, kept).onItsWay.map((l) => `${l.repo}#${l.number}`));
142 return rows.filter((r) => way.has(`${r.repo}#${r.number}`)).reverse();
143 }
144
137145 /** How long a miss is believed: a repository can turn public, or merge its first pull, after boot. */
138146 const MISS_MS = 60_000;
139147
src/system/home_truth.ts +6
@@ -9,6 +9,7 @@ import type { App } from "../app.ts";
99 import { listReposVisibleTo } from "../core/repos.ts";
1010 import type { User } from "../core/users.ts";
1111 import { forgeCeiling } from "../flow/ceiling.ts";
12 import { running } from "../kernel/version.ts";
1213 import { readKpis } from "../flow/kpis.ts";
1314 import { waitingOn } from "../web/studio/asks.ts";
1415
@@ -24,9 +25,14 @@ export function homeTruth(app: App, user: User, now: Date = new Date()) {
2425 const k = readKpis(app.db, now, { ceilingUsd: forgeCeiling(app.config) });
2526 const asks = waitingOn(app.db, user, listReposVisibleTo(app.db, user, { limit: 200, withoutArchived: true }));
2627 const oldest = asks.reduce<string | null>((o, a) => (o === null || Date.parse(a.since) < Date.parse(o) ? a.since : o), null);
28 // `live` stays the newest deploy of main, as readers know it; `production` names the commit this process runs
29 // and `main` repeats `live` under its own name (#11507).
30 const prod = running();
2731 return {
2832 landed: k.landed,
2933 live: k.live,
34 production: prod.commit === null ? null : { version: prod.version, sha: prod.commit },
35 main: k.live,
3036 builders: { ...k.builders, idle: idleReason(k.builders, k.spend) },
3137 spend: k.spend,
3238 waiting: { count: asks.length, oldestAt: oldest },
src/system/telemetry.ts +5 −1
@@ -11,6 +11,7 @@ import type { App } from "../app.ts";
1111 import { SOURCES } from "../boards/sources.ts";
1212 import { conflictedPulls, conflictedSays, rebaseLane, rebaseLaneSays } from "../flow/conflicted.ts";
1313 import { firedAlarms } from "../flow/alarms.ts";
14 import { forgeCeiling } from "../flow/ceiling.ts";
1415 import { running } from "../kernel/version.ts";
1516 import { readTalkTurns, talkTurnsWords } from "../flow/talk_turns.ts";
1617 import { readTalkResumes, talkResumesWords } from "../flow/talk_resumes.ts";
@@ -18,6 +19,7 @@ import { readVitals, vitalsFor, vitalsLine, vitalsMisses } from "../flow/vitals.
1819 import { parkedSays, readParked } from "../flow/parked.ts";
1920 import { readDeclines } from "../flow/declines.ts";
2021 import { readKeeper } from "../flow/keeper.ts";
22 import { heartbeatOf } from "../keeper/landed.ts";
2123 import { readPipeline, readStepStats, statWords, verdict } from "../flow/pipeline.ts";
2224 import { type QueueReading, readQueues } from "../flow/queue.ts";
2325 import { reviewersBusy, reviewingSays } from "../flow/reviewing.ts";
@@ -103,6 +105,8 @@ function flowBody(x: {
103105 steps: x.steps.map((m) => ({ ...m, says: statWords(m) })),
104106 // The cause binding landings now and the keeper's last act (#7645).
105107 keeper: readKeeper(app.db, reader.kind === "owner"),
108 // The release tick's last report, reason and all, for the owner only (#11507): the reason may name private work.
109 heartbeat: reader.kind === "owner" ? heartbeatOf(app.db, now) : null,
106110 // How long each main has gone unverified after landing (#7786); private repositories for the owner only.
107111 verified: x.verified.map((row) => ({ ...row, says: verifiedSays(row, now.getTime()) })),
108112 // Issues the crew parked in the last 24 h and 7 d, the newest three named (#6471).
@@ -117,7 +121,7 @@ function flowBody(x: {
117121 // Declined build runs of the last 24 h by reason, each with the runs behind it.
118122 declines: readDeclines(app.db, now, reader.kind === "agent"),
119123 // The four alarms (#5930) that are firing now; empty when none is.
120 alarms: firedAlarms(app.db, now, { publicOnly: reader.kind === "agent", dataDir: app.config.dataDir }),
124 alarms: firedAlarms(app.db, now, { publicOnly: reader.kind === "agent", dataDir: app.config.dataDir, ceilingUsd: forgeCeiling(app.config) }),
121125 // Review runs in flight per reviewer, cut to public repositories for an agent (#4733).
122126 reviewing: { says: reviewingSays(busy), reviewers: busy },
123127 conflicted: { says: conflictedSays(conflicted, now.getTime()), pulls: conflicted },
src/tui/forge.ts +2
@@ -82,6 +82,7 @@ import { readHandoverLists } from "../flow/handover_read.ts";
8282 import { readDrifts } from "../flow/alarm_drift.ts";
8383 import { readHeld } from "../flow/held.ts";
8484 import { answerKeeper, readKeeper } from "../flow/keeper.ts";
85 import { beatLine, lastBeat } from "../keeper/beat.ts";
8586 import { readVerified } from "../flow/verified.ts";
8687 import { memoryScreen } from "./screens/system.ts";
8788 import { noteRead, telemetryReader } from "../actions/telemetry.ts";
@@ -500,6 +501,7 @@ export function forgeCommands(session: Session): (ctx: Ctx) => Command[] {
500501 readVerified(session.app, flowReader.kind === "owner"),
501502 keeper,
502503 session.app.config.dataDir,
504 flowReader.kind === "owner" ? beatLine(lastBeat(session.db), new Date()) : undefined,
503505 ), keeper.ask && user ? { ask: keeper.ask, answer: () => answerKeeper(session.app, keeper.ask!.id, user.id) } : undefined,
504506 mayDrift(session)
505507 ? { read: async () => readHandoverLists(session.db, await readDrifts(session.app), Date.now()), update: (n) => askUpdate(session, n) }
src/tui/screens/flow.ts +3
@@ -93,6 +93,8 @@ export function flowLines(
9393 keeper?: KeeperRead,
9494 /** Where the backup outcome files are kept; without it the off-box backup alarm cannot be read (the terminal passes the app's). */
9595 dataDir: string | null = null,
96 /** The release tick's heartbeat line, an owner's only (#11507). */
97 heartbeat?: string,
9698 ): string[] {
9799 const stages = readPipeline(db, now);
98100 const flow = readThroughput(db, stages.map((s) => s.key), now);
@@ -115,6 +117,7 @@ export function flowLines(
115117 return [
116118 ...wrap(verdict(stages, now.getTime()), inner),
117119 ...(keeper === undefined || keeperLine(keeper) === "" ? [] : ["", ...wrap(keeperLine(keeper), inner)]),
120 ...(heartbeat === undefined ? [] : ["", ...wrap(heartbeat, inner)]),
118121 ...(keeper?.ask == null ? [] : wrap(`${keeper.ask.question} Press l to ${keeper.ask.button.toLowerCase()}.`, inner)),
119122 "",
120123 ...wrap(vitalsLine(vitalsFor(readVitals(db, now, agent !== undefined), agent === undefined)), inner),
src/tui/screens/keeper.ts +21 −4
@@ -1,6 +1,7 @@
11 /**
22 * The release keeper in the terminal (#7557), the /releases card's read:
3 * what holds the next cut, then each release the keeper cut with its stage,
3 * what holds the next cut, the release tick's heartbeat and whether main is
4 * green and ahead of production (#11507), then each release the keeper cut with its stage,
45 * its short commit, landed to live (#8042), the known-good version it rolls
56 * back to and what the probes counted.
67 * Gated by `release/gate.ts` as the page is. `h` holds releases with a
@@ -8,6 +9,8 @@
89 * functions the card and MCP `release_hold` call.
910 */
1011
12 import { aheadLine } from "../../flow/alarm_main_ahead.ts";
13 import { beatLine, lastBeat } from "../../keeper/beat.ts";
1114 import { stallWords } from "../../perf/boot_stall.ts";
1215 import { blocking, holdAs, liftAs, standingHolds } from "../../keeper/hold.ts";
1316 import { secondsWords, withLandedToLive, type LandedRelease } from "../../keeper/landed.ts";
@@ -32,13 +35,23 @@ const STAGE: Record<Stage, { label: string; tone: Tone }> = {
3235 export interface KeeperRead {
3336 /** What holds the next cut, or the line that says nothing does. */
3437 readonly blocked: string;
38 /** The release tick's last heartbeat as a sentence, said plainly when there is none or it is old. */
39 readonly heartbeat: string;
40 /** Main green and ahead of production, and for how long; absent when it is not. */
41 readonly ahead?: string;
3542 readonly releases: readonly LandedRelease[];
3643 }
3744
3845 /** The card's content for this person; undefined for anyone the page would 404. */
39 export function readKeeper(f: Session): KeeperRead | undefined {
46 export function readKeeper(f: Session, now: Date = new Date()): KeeperRead | undefined {
4047 if (!mayRead(f.app, f.user)) return undefined;
41 return { blocked: blocking(standingHolds(f.db)) ?? "Nothing holds the next cut.", releases: withLandedToLive(f.db, listKeeperReleases(f.db)) };
48 const ahead = aheadLine(f.db, now);
49 return {
50 blocked: blocking(standingHolds(f.db)) ?? "Nothing holds the next cut.",
51 heartbeat: beatLine(lastBeat(f.db), now),
52 ...(ahead === undefined ? {} : { ahead }),
53 releases: withLandedToLive(f.db, listKeeperReleases(f.db)),
54 };
4255 }
4356
4457 export function keeperRows(releases: readonly LandedRelease[]): ListRow[] {
@@ -90,7 +103,11 @@ export function keeperScreen(f: Session, ctx: Ctx): Screen {
90103 return;
91104 }
92105 // What holds the cut ends with what to land ("release hold N"): wrapped, up to three lines, never cut short of it.
93 const lines = wrap(read.blocked, area.w).slice(0, 3);
106 const lines = [
107 ...wrap(read.blocked, area.w).slice(0, 3),
108 ...wrap(read.heartbeat, area.w).slice(0, 2),
109 ...(read.ahead === undefined ? [] : wrap(read.ahead, area.w).slice(0, 2)),
110 ];
94111 const [head, body] = split(area, "v", [len(lines.length + 1), fill(1)]);
95112 lines.forEach((l, i) => s.text(head!.x, head!.y + i, truncate(l, head!.w), style(c.mutedFg, c.bg), head!.x + head!.w));
96113 if (read.releases.length === 0) {
src/web/pages/flow-panel.ts +4 −1
@@ -53,6 +53,7 @@ import type { App } from "../../app.ts";
5353 import { html, type Html } from "../html.ts";
5454 import { doctorCard } from "./boards-doctor.ts";
5555 import { keeperRow } from "../studio/keeper_row.ts";
56 import { beatFact, lastBeat } from "../../keeper/beat.ts";
5657 import { gatesSection } from "./boards-gates.ts";
5758 import { heldCard } from "./boards-held.ts";
5859 import { badge, card, meter } from "../ui.ts";
@@ -127,7 +128,9 @@ export function flowPanel(app: App, reader: Reader, url: URL, base: string): Htm
127128 ["Verified after landing", html`A repository that declares <span class="mono">check.verify</span> in forge.json has its main verified after landing.`],
128129 ["Elsewhere", html`Limits are set on <a href="/settings/capacity">Capacity</a>. How the forge stands against its threats is on <a href="/resilience">Resilience</a>; where an outside agent can help is on <a href="/contribute">Contribute</a>.`],
129130 ], helpAsked(url));
130 return html`${keeperRow(app.db, reader.kind === "owner")}${help}${card(
131 return html`${keeperRow(app.db, reader.kind === "owner")}${reader.kind === "owner"
132 ? html`<dl class="status-rows" id="keeper-heartbeat-row"><div class="status-row"><dt>Release tick</dt><dd id="keeper-heartbeat">${beatFact(lastBeat(app.db), new Date(now))}</dd></div></dl>`
133 : ""}${help}${card(
131134 html`<p id="bottleneck" data-stage="${worst?.key ?? ""}">
132135 <strong>${verdict(stages, now)}</strong>
133136 </p>
src/web/pages/keeper-panel.ts +12 −3
@@ -1,11 +1,14 @@
11 /**
22 * The release keeper on /releases (#7557): what stops the next cut, and
33 * each release it cut with its stage, the known-good version it rolls back
4 * to, and what the probes counted. Nothing when it has done nothing.
4 * to, and what the probes counted; the release tick's heartbeat and whether
5 * main is green and ahead of production (#11507). Nothing when it has done nothing.
56 * A plain section under an h2, no card (#9998).
67 */
78
89 import type { Database } from "bun:sqlite";
10 import { aheadLine } from "../../flow/alarm_main_ahead.ts";
11 import { beatFact, lastBeat } from "../../keeper/beat.ts";
912 import { blocking, MAX_REASON, standingHolds } from "../../keeper/hold.ts";
1013 import { secondsWords, withLandedToLive } from "../../keeper/landed.ts";
1114 import { listKeeperReleases, type Stage } from "../../keeper/record.ts";
@@ -21,11 +24,13 @@ const STAGE: Record<Stage, { label: string; tone: Tone }> = {
2124 };
2225
2326 /** `mayHold` is `mayRollBack`: the section then offers the owner's hold and its lift. */
24 export function keeperPanel(db: Database, mayHold = false): Html | undefined {
27 export function keeperPanel(db: Database, mayHold = false, now: Date = new Date()): Html | undefined {
2528 const releases = withLandedToLive(db, listKeeperReleases(db));
2629 const holds = standingHolds(db);
2730 const blocked = blocking(holds);
28 if (releases.length === 0 && blocked === undefined && !mayHold) return undefined;
31 const beat = lastBeat(db);
32 const ahead = aheadLine(db, now);
33 if (releases.length === 0 && blocked === undefined && beat === null && ahead === undefined && !mayHold) return undefined;
2934 const rows = releases.map((r, i) => {
3035 const s = STAGE[r.stage];
3136 return html`<tr>
@@ -43,6 +48,10 @@ export function keeperPanel(db: Database, mayHold = false): Html | undefined {
4348 return html`<section class="settings-block" id="keeper">
4449 <h2>Release keeper</h2>
4550 <p class="small ${blocked === undefined ? "muted" : ""}" id="keeper-blocked">${blocked ?? "Nothing holds the next cut."}</p>
51 <dl class="status-rows" id="keeper-facts">
52 <div class="status-row"><dt>Release tick</dt><dd id="keeper-heartbeat">${beatFact(beat, now)}</dd></div>
53 ${when(ahead !== undefined, () => html`<div class="status-row"><dt>Main vs production</dt><dd id="keeper-ahead">${ahead}</dd></div>`)}
54 </dl>
4655 ${when(rows.length > 0, () => html`<div class="table-scroll" role="region" tabindex="0" aria-label="Keeper releases"><table class="table" data-sticky-lead>
4756 <thead><tr><th>Version</th><th>Cut</th><th>Landed to live</th><th>Known good</th><th>Roll non-200</th><th>Probes / pages</th><th>Worst</th><th>Stall, first 5 min</th><th>Dark</th></tr></thead>
4857 <tbody>${rows}</tbody>
test/flow-alarm-lane.test.ts added +192
@@ -0,0 +1,192 @@
1 /** The lane has stopped (#11507): ten open pulls, 91 minutes without a merge and one repeated failure line raise exactly one alarm; a merge clears it. */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { mkdtempSync, rmSync } from "node:fs";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { createRepo, getRepo } from "../src/core/repos.ts";
8 import { createUser } from "../src/core/users.ts";
9 import { getRun } from "../src/core/runs.ts";
10 import { ownLandingStopped } from "../src/checks/gate_stopped.ts";
11 import { hunt } from "../src/crew/sentinel.ts";
12 import { firedAlarms } from "../src/flow/alarms.ts";
13 import { forgeFailed, laneAlarm, laneMinutes, maskLine } from "../src/flow/alarm_lane.ts";
14 import { id } from "../src/kernel/id.ts";
15
16 const NOW = new Date("2026-11-12T17:11:00.000Z");
17 const ago = (m: number) => new Date(NOW.getTime() - m * 60_000).toISOString();
18 const LINE = "stopped: used 16.0 cores for 30s, over its limit of 16";
19
20 let root: string;
21 let app: App;
22 let repoId: string;
23 let userId: string;
24 let n = 0;
25 const saved = { sentinel: process.env["FORGE_SENTINEL_REPO"], commit: process.env["FORGE_COMMIT"] };
26
27 beforeAll(async () => {
28 process.env["FORGE_SENTINEL_REPO"] = "trevato/forge";
29 process.env["FORGE_COMMIT"] = "abcdef1234567890abcdef1234567890abcdef12";
30 root = mkdtempSync(join(tmpdir(), "forge-lane-"));
31 app = createApp({ dataDir: root, port: 0 });
32 const u = await createUser(app.db, { username: "trevato", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true, adminIfFirst: true });
33 if (!u.ok) throw new Error(u.error);
34 userId = u.value.id;
35 const repo = await createRepo(app.db, app.reposRoot, { ownerId: u.value.id, name: "forge", visibility: "public", description: "f" });
36 if (!repo.ok) throw new Error(repo.error);
37 repoId = repo.value.id;
38 });
39 afterAll(() => {
40 app.close();
41 rmSync(root, { recursive: true, force: true });
42 for (const [k, v] of [["FORGE_SENTINEL_REPO", saved.sentinel], ["FORGE_COMMIT", saved.commit]] as const)
43 if (v === undefined) delete process.env[k];
44 else process.env[k] = v;
45 });
46
47 function pull(state: "open" | "closed", mergedMinAgo?: number): string {
48 const issueId = id();
49 n++;
50 app.db.query(`INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at) VALUES (?1, ?2, ?3, 'pull', 'P', '', ?4, ?5, ?6)`).run(issueId, repoId, 2000 + n, userId, state, ago(600));
51 app.db
52 .query(`INSERT INTO pull (issue_id, base_branch, head_branch, base_oid, head_oid, head_repo_id, merged_at, merged_oid) VALUES (?1, 'main', ?2, '', '', ?3, ?4, ?5)`)
53 .run(issueId, `b/${n}`, repoId, mergedMinAgo === undefined ? null : ago(mergedMinAgo), mergedMinAgo === undefined ? "" : "9".repeat(40));
54 return issueId;
55 }
56 function failed(issueId: string, minAgo: number, summary = LINE, detail = "{}"): string {
57 const runId = `${String(++n).padStart(4, "0")}${id()}`;
58 app.db
59 .query(`INSERT INTO run (id, repo_id, issue_id, kind, env, state, summary, check_detail, commit_oid, created_at, finished_at) VALUES (?1, ?2, ?3, 'check', 'pull-1', 'failed', ?4, ?5, 'c', ?6, ?6)`)
60 .run(runId, repoId, issueId, summary, detail, ago(minAgo));
61 return runId;
62 }
63 const reset = () => {
64 app.db.query(`DELETE FROM run`).run();
65 app.db.query(`DELETE FROM pull`).run();
66 app.db.query(`DELETE FROM issue`).run();
67 };
68 const lane = () => firedAlarms(app.db, NOW, { dataDir: null }).filter((a) => a.key === "lane_stopped");
69
70 /** Ten open pulls and a last merge `since` minutes ago; one failed check apiece after it. */
71 function stalled(since: number): string[] {
72 reset();
73 pull("closed", since);
74 const pulls = Array.from({ length: 10 }, () => pull("open"));
75 pulls.forEach((p, i) => {
76 failed(p, 75 + i, LINE.replace("16.0", `1${i}.0`));
77 });
78 return pulls;
79 }
80
81 describe("the lane has stopped", () => {
82 test("ten open pulls, 91 minutes without a merge and one repeated line raise exactly one alarm with that line", () => {
83 stalled(91);
84 const a = lane();
85 expect(a).toHaveLength(1);
86 expect(a[0]!.says).toContain("Nothing has merged for 1 h 31 min");
87 expect(a[0]!.says).toContain("10 pulls open");
88 expect(a[0]!.says).toContain("10 checks failed");
89 expect(a[0]!.since).toBe(ago(91));
90 // The commonest line, verbatim as the forge holds it (the newest instance of the masked group).
91 expect(laneAlarm(app.db, NOW).line).toBe(LINE.replace("16.0", "10.0"));
92 });
93
94 test("the line is quoted as recorded, with its count in the span", () => {
95 reset();
96 pull("closed", 120);
97 const p = pull("open");
98 for (let i = 0; i < 3; i++) failed(p, 100 - i);
99 failed(p, 50, "check failed: install broke");
100 const a = laneAlarm(app.db, NOW);
101 expect(a.fired).toBe(true);
102 expect(a.says).toContain(`4 checks failed in that span, 3 of them with: ${LINE}`);
103 });
104
105 test("89 minutes is quiet, and so is a setting that moves the limit", () => {
106 stalled(89);
107 expect(lane()).toHaveLength(0);
108 app.db.query(`INSERT INTO forge_setting (key, value, updated_by, updated_at) VALUES ('flow.lane_minutes', '60', 't', 'now')`).run();
109 expect(laneMinutes(app.db)).toBe(60);
110 expect(lane()).toHaveLength(1);
111 app.db.query(`UPDATE forge_setting SET value = 'abc' WHERE key = 'flow.lane_minutes'`).run();
112 expect(laneMinutes(app.db)).toBe(90);
113 app.db.query(`DELETE FROM forge_setting WHERE key = 'flow.lane_minutes'`).run();
114 });
115
116 test("nothing open or ready, or the crew at its ceiling, is quiet", () => {
117 reset();
118 pull("closed", 300);
119 expect(lane()).toHaveLength(0);
120 pull("open");
121 expect(firedAlarms(app.db, NOW, { dataDir: null, ceilingUsd: null }).filter((a) => a.key === "lane_stopped")).toHaveLength(1);
122 app.db
123 .query(`INSERT INTO run (id, repo_id, kind, state, cost_usd, created_at, summary, check_detail) VALUES (?1, ?2, 'agent', 'ok', 50, ?3, '', '')`)
124 .run(id(), repoId, new Date(NOW.getTime() - 60_000).toISOString());
125 expect(firedAlarms(app.db, NOW, { dataDir: null, ceilingUsd: 20 }).filter((a) => a.key === "lane_stopped")).toHaveLength(0);
126 });
127
128 test("a merge clears it", () => {
129 stalled(91);
130 expect(lane()).toHaveLength(1);
131 pull("closed", 1);
132 expect(lane()).toHaveLength(0);
133 });
134
135 test("it reaches the owner's ask and one Sentinel issue, which closes when a pull merges", () => {
136 stalled(91);
137 const owner = app.db.query<{ id: string }, []>(`SELECT id FROM user WHERE username = 'trevato'`).get()!;
138 expect(owner.id).toBe(userId);
139 hunt(app, "trevato/forge", NOW.getTime());
140 hunt(app, "trevato/forge", NOW.getTime() + 60_000);
141 const open = () => app.db.query<{ body: string }, []>(`SELECT body FROM issue WHERE state = 'open' AND title LIKE 'Sentinel: %lane has stopped%'`).all();
142 expect(open()).toHaveLength(1);
143 expect(open()[0]!.body).toContain("stopped: used");
144 expect(open()[0]!.body).toContain("checks failed since the last merge");
145 pull("closed", 1);
146 hunt(app, "trevato/forge", NOW.getTime() + 120_000);
147 expect(open()).toHaveLength(0);
148 });
149
150 test("land_past_gate's guard recognises it as stopped", () => {
151 stalled(91);
152 const repo = getRepo(app.db, "trevato", "forge")!;
153 expect(ownLandingStopped(app.db, repo)).toBeDefined();
154 reset();
155 expect(ownLandingStopped(app.db, repo)).toBeUndefined();
156 });
157 });
158
159 describe("one cause failing many pulls", () => {
160 test("five pulls in an hour on one masked line are the forge's: alarm names it, the check is the forge's", () => {
161 reset();
162 pull("closed", 10);
163 const pulls = Array.from({ length: 5 }, () => pull("open"));
164 const runs = pulls.map((p, i) => failed(p, 5 + i, LINE.replace("16.0", `${i + 11}.5`)));
165 const a = lane();
166 expect(a).toHaveLength(1);
167 expect(a[0]!.says).toContain("the forge's failure, not the builders'");
168 expect(a[0]!.says).toContain("5 pulls");
169 expect(laneAlarm(app.db, NOW).forge).toBe(true);
170 expect(forgeFailed(app.db, getRun(app.db, runs[0]!), NOW.getTime())).toBe(true);
171 });
172
173 test("four pulls, or five over two hours, or a named test, are the pull's own", () => {
174 reset();
175 pull("closed", 10);
176 const pulls = Array.from({ length: 5 }, () => pull("open"));
177 const runs = pulls.slice(0, 4).map((p, i) => failed(p, 5 + i));
178 expect(lane()).toHaveLength(0);
179 expect(forgeFailed(app.db, getRun(app.db, runs[0]!), NOW.getTime())).toBe(false);
180 failed(pulls[4]!, 130);
181 expect(forgeFailed(app.db, getRun(app.db, runs[0]!), NOW.getTime())).toBe(false);
182 reset();
183 pull("closed", 10);
184 const named = Array.from({ length: 5 }, () => pull("open")).map((p, i) => failed(p, 5 + i, "1 test failed", JSON.stringify({ names: ["a > b"] })));
185 expect(lane()).toHaveLength(0);
186 expect(forgeFailed(app.db, getRun(app.db, named[0]!), NOW.getTime())).toBe(false);
187 });
188
189 test("numbers are masked", () => {
190 expect(maskLine(LINE)).toBe("stopped: used # cores for #s, over its limit of #");
191 });
192 });
test/flow-alarm-main-ahead.test.ts added +319
@@ -0,0 +1,319 @@
1 /** Main green and ahead of production (#11507): one alarm past 90 min with the facts, none at 89, closes on catch-up; the heartbeat shows. */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { mkdtempSync, rmSync } from "node:fs";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { createRepo } from "../src/core/repos.ts";
8 import { createUser, type User } from "../src/core/users.ts";
9 import { addComment } from "../src/actions/issues.ts";
10 import { getIssueById } from "../src/core/issues.ts";
11 import { getRepo } from "../src/core/repos.ts";
12 import { hunt } from "../src/crew/sentinel.ts";
13 import { firedAlarms } from "../src/flow/alarms.ts";
14 import { aheadMinutes, mainAheadAlarm } from "../src/flow/alarm_main_ahead.ts";
15 import { lastBeat, recordBeat, stepWithBeat } from "../src/keeper/beat.ts";
16 import { keeperPanel } from "../src/web/pages/keeper-panel.ts";
17 import { keeperAnswer } from "../src/keeper/landed.ts";
18 import { tickStep } from "../src/keeper/keeper.ts";
19 import { keeperForge } from "./helpers/keeper.ts";
20 import { id } from "../src/kernel/id.ts";
21 import { forgetLive, keepLive } from "../src/release/landings.ts";
22 import { alarmLines } from "../src/crew/notify.ts";
23 import { homeTruth } from "../src/system/home_truth.ts";
24 import { Session } from "../src/tui/forge.ts";
25 import { keeperScreen, readKeeper } from "../src/tui/screens/keeper.ts";
26 import { createForgeApp } from "../src/tui/forge.ts";
27 import { headlessTerminal } from "../src/tui/term/terminal.ts";
28 import { settleApp } from "./helpers/tui.ts";
29 import { flowFor } from "../src/system/telemetry.ts";
30 import { flowPanel } from "../src/web/pages/flow-panel.ts";
31
32 const NOW = new Date("2026-11-11T12:00:00.000Z");
33 const ago = (m: number) => new Date(NOW.getTime() - m * 60_000).toISOString();
34 const PROD = "abcdef1234567890abcdef1234567890abcdef12";
35 const HEAD = "1111111111111111111111111111111111111111";
36
37 let root: string;
38 let app: App;
39 let repoId: string;
40 let branch: string;
41 let userId: string;
42 let owner: User;
43 let n = 0;
44 const saved = { sentinel: process.env["FORGE_SENTINEL_REPO"], commit: process.env["FORGE_COMMIT"], version: process.env["FORGE_VERSION"] };
45
46 beforeAll(async () => {
47 process.env["FORGE_SENTINEL_REPO"] = "trevato/forge";
48 process.env["FORGE_COMMIT"] = PROD;
49 process.env["FORGE_VERSION"] = "0.7.535";
50 root = mkdtempSync(join(tmpdir(), "forge-main-ahead-"));
51 app = createApp({ dataDir: root, port: 0 });
52 const u = await createUser(app.db, { username: "trevato", email: "[email protected]", password: "a-good-passphrase", origin: "seed", trusted: true, adminIfFirst: true });
53 if (!u.ok) throw new Error(u.error);
54 userId = u.value.id;
55 owner = u.value;
56 const repo = await createRepo(app.db, app.reposRoot, { ownerId: u.value.id, name: "forge", visibility: "public", description: "f" });
57 if (!repo.ok) throw new Error(repo.error);
58 repoId = repo.value.id;
59 branch = repo.value.defaultBranch;
60 });
61 afterAll(() => {
62 app.close();
63 rmSync(root, { recursive: true, force: true });
64 for (const [k, v] of [["FORGE_SENTINEL_REPO", saved.sentinel], ["FORGE_COMMIT", saved.commit], ["FORGE_VERSION", saved.version]] as const)
65 if (v === undefined) delete process.env[k];
66 else process.env[k] = v;
67 forgetLive();
68 });
69
70 function check(state: "ok" | "failed", oid: string): void {
71 app.db
72 .query(`INSERT INTO run (id, repo_id, kind, env, state, summary, check_detail, commit_oid, created_at, finished_at) VALUES (?1, ?2, 'check', ?3, ?4, '', '{}', ?5, ?6, ?6)`)
73 .run(`${String(++n).padStart(4, "0")}${id()}`, repoId, branch, state, oid, ago(1));
74 }
75 function merge(minAgo: number, oid: string): void {
76 const issueId = id();
77 n++;
78 app.db.query(`INSERT INTO issue (id, repo_id, number, kind, title, body, author_id, state, created_at) VALUES (?1, ?2, ?3, 'pull', 'P', '', ?4, 'closed', ?5)`).run(issueId, repoId, 1000 + n, userId, ago(minAgo));
79 app.db.query(`INSERT INTO pull (issue_id, base_branch, head_branch, base_oid, head_oid, head_repo_id, merged_at, merged_oid) VALUES (?1, ?2, ?3, '', '', ?4, ?5, ?6)`).run(issueId, branch, `b/${n}`, repoId, ago(minAgo), oid);
80 }
81 const reset = () => {
82 app.db.query(`DELETE FROM run`).run();
83 app.db.query(`DELETE FROM pull`).run();
84 keepLive("trevato/forge", [], NOW.getTime() - 600 * 60_000);
85 };
86
87 describe("main ahead alarm", () => {
88 test("none at 89 minutes, one at 91 with the facts", () => {
89 reset();
90 check("ok", HEAD);
91 merge(89, "2".repeat(40));
92 expect(mainAheadAlarm(app.db, NOW).fired).toBe(false);
93 expect(firedAlarms(app.db, NOW, { dataDir: null }).filter((a) => a.key === "main_ahead")).toHaveLength(0);
94 reset();
95 check("ok", HEAD);
96 merge(91, "2".repeat(40));
97 merge(30, "3".repeat(40));
98 recordBeat(app.db, "skipped", "no cut: a hold stands", new Date(NOW.getTime() - 61 * 60_000));
99 const fired = firedAlarms(app.db, NOW, { dataDir: null }).filter((a) => a.key === "main_ahead");
100 expect(fired).toHaveLength(1);
101 const says = fired[0]!.says;
102 expect(says).toContain("0.7.535");
103 expect(says).toContain("abcdef1");
104 expect(says).toContain("2 commits and 2 pulls");
105 expect(says).toContain("1 h 31 min");
106 expect(says).toContain("1 h 1 min ago");
107 expect(fired[0]!.since).toBe(ago(91));
108 });
109
110 test("a red head, or production on the head, does not fire", () => {
111 reset();
112 merge(200, "2".repeat(40));
113 check("failed", HEAD);
114 expect(mainAheadAlarm(app.db, NOW).fired).toBe(false);
115 check("ok", PROD);
116 expect(mainAheadAlarm(app.db, NOW).fired).toBe(false);
117 });
118
119 test("the limit is a setting", () => {
120 reset();
121 check("ok", HEAD);
122 merge(50, "2".repeat(40));
123 expect(mainAheadAlarm(app.db, NOW).fired).toBe(false);
124 app.db.query(`INSERT INTO forge_setting (key, value, updated_by, updated_at) VALUES ('release.ahead_minutes', '45', 'test', 'now') ON CONFLICT (key) DO UPDATE SET value = '45'`).run();
125 expect(mainAheadAlarm(app.db, NOW).fired).toBe(true);
126 app.db.query(`DELETE FROM forge_setting WHERE key = 'release.ahead_minutes'`).run();
127 });
128
129 test("it reaches the owner's ask and one Sentinel issue, which closes when production catches up", () => {
130 reset();
131 check("ok", HEAD);
132 merge(120, "2".repeat(40));
133 expect(owner.isAdmin).toBe(true);
134 const lines = alarmLines(app, owner, NOW.getTime());
135 expect(lines.some((l) => l.id.startsWith("alarm:main_ahead@"))).toBe(true);
136 hunt(app, "trevato/forge", NOW.getTime());
137 hunt(app, "trevato/forge", NOW.getTime() + 60_000);
138 const open = () => app.db.query<{ title: string }, []>(`SELECT title FROM issue WHERE state = 'open' AND title LIKE 'Sentinel: Main is green and ahead%'`).all();
139 expect(open()).toHaveLength(1);
140 // Production catches up: it runs the head and the pull is in the running commit.
141 keepLive("trevato/forge", ["2".repeat(40)], NOW.getTime());
142 hunt(app, "trevato/forge", NOW.getTime() + 120_000);
143 expect(open()).toHaveLength(0);
144 });
145 });
146
147 describe("heartbeat", () => {
148 test("release_keeper shows the last heartbeat and its reason, and says when it is quiet", () => {
149 app.db.query(`DELETE FROM keeper_beat`).run();
150 expect(keeperAnswer(app.db, NOW).heartbeat).toBeNull();
151 recordBeat(app.db, "skipped", "not cut: a hold stands", new Date(NOW.getTime() - 5 * 60_000));
152 const h = keeperAnswer(app.db, NOW).heartbeat!;
153 expect(h).toMatchObject({ outcome: "skipped", reason: "not cut: a hold stands", silent: false });
154 recordBeat(app.db, "failed", "tag refused", new Date(NOW.getTime() - 61 * 60_000));
155 const quiet = keeperAnswer(app.db, NOW).heartbeat!;
156 expect(quiet.silent).toBe(true);
157 expect(quiet.says).toContain("No heartbeat from the release tick for 1 h 1 min");
158 expect(lastBeat(app.db)!.reason).toBe("tag refused");
159 });
160 });
161
162 describe("home telemetry", () => {
163 test("production's commit is named apart from main's", () => {
164 reset();
165 app.db.query(`INSERT INTO run (id, repo_id, kind, env, state, commit_oid, created_at, serving_at) VALUES (?1, ?2, 'deploy', 'production', 'ok', ?3, ?4, ?4)`).run(id(), repoId, HEAD, ago(5));
166 const t = homeTruth(app, owner, NOW);
167 expect(t.production).toEqual({ version: "0.7.535", sha: PROD });
168 expect(t.main?.sha).toBe(HEAD);
169 // `live` keeps its meaning: main's newest deploy.
170 expect(t.live?.sha).toBe(HEAD);
171 });
172
173 test("with the running commit unknown, production is null and live is untouched", () => {
174 const commit = process.env["FORGE_COMMIT"];
175 delete process.env["FORGE_COMMIT"];
176 try {
177 const t = homeTruth(app, owner, NOW);
178 expect(t.production).toBeNull();
179 expect(t.live?.sha).toBe(HEAD);
180 expect(t.main).toEqual(t.live);
181 } finally {
182 process.env["FORGE_COMMIT"] = commit;
183 }
184 });
185 });
186
187 describe("the main-ahead issue", () => {
188 const open = () => app.db.query<{ id: string; body: string }, []>(`SELECT id, body FROM issue WHERE state = 'open' AND title LIKE 'Sentinel: Main is green and ahead%'`).all();
189
190 test("keeps the tick's reason out of the issue, and stays open once a person spoke on it", async () => {
191 reset();
192 check("ok", HEAD);
193 merge(120, "4".repeat(40));
194 recordBeat(app.db, "skipped", "private words from the tick", new Date(NOW.getTime() - 5 * 60_000));
195 hunt(app, "trevato/forge", NOW.getTime() + 600_000);
196 const filed = open();
197 expect(filed).toHaveLength(1);
198 expect(filed[0]!.body).toContain("skipped");
199 expect(filed[0]!.body).not.toContain("private words");
200 const repo = getRepo(app.db, "trevato", "forge")!;
201 const said = addComment(app, owner, repo, getIssueById(app.db, filed[0]!.id)!, "Looking at the tick now.");
202 expect(said.ok).toBe(true);
203 keepLive("trevato/forge", ["4".repeat(40)], NOW.getTime());
204 hunt(app, "trevato/forge", NOW.getTime() + 720_000);
205 expect(open()).toHaveLength(1);
206 });
207 });
208
209 describe("the limit setting", () => {
210 const set = (v: string) =>
211 app.db.query(`INSERT INTO forge_setting (key, value, updated_by, updated_at) VALUES ('release.ahead_minutes', ?1, 'test', 'now') ON CONFLICT (key) DO UPDATE SET value = ?1`).run(v);
212 test("a value out of range or not a whole number is 90", () => {
213 for (const bad of ["0", "20000", "abc", "1.5", ""]) {
214 set(bad);
215 expect(aheadMinutes(app.db)).toBe(90);
216 }
217 set("10080");
218 expect(aheadMinutes(app.db)).toBe(10080);
219 app.db.query(`DELETE FROM forge_setting WHERE key = 'release.ahead_minutes'`).run();
220 expect(aheadMinutes(app.db)).toBe(90);
221 });
222 });
223
224 describe("the tick's beat", () => {
225 test("a step's outcome is recorded as the keeper said it, whatever its words", async () => {
226 const at = new Date(NOW.getTime() - 60_000);
227 const line = await stepWithBeat(app.db, async () => ({ line: "not cut: a failed hold stands", outcome: "skipped" }), () => at);
228 expect(line).toBe("not cut: a failed hold stands");
229 expect(lastBeat(app.db)).toEqual({ at: at.toISOString(), outcome: "skipped", reason: "not cut: a failed hold stands" });
230 });
231
232 test("a step that throws is recorded as failed, and the error goes on", async () => {
233 const step = stepWithBeat(app.db, async () => { throw new Error("git is gone"); }, () => NOW);
234 await expect(step).rejects.toThrow("git is gone");
235 expect(lastBeat(app.db)).toEqual({ at: NOW.toISOString(), outcome: "failed", reason: "step failed: git is gone" });
236 });
237 });
238
239 describe("a beat that cannot be written", () => {
240 test("a good tick is still reported good", async () => {
241 const broken = { query: () => { throw new Error("disk full"); } } as never;
242 const line = await stepWithBeat(broken, async () => ({ line: "cut 0.7.9", outcome: "cut" }), () => NOW);
243 expect(line).toBe("cut 0.7.9");
244 });
245 });
246
247 describe("both doors", () => {
248 test("/releases and the terminal's keeper screen show the heartbeat and that main is ahead", () => {
249 reset();
250 check("ok", HEAD);
251 merge(120, "5".repeat(40));
252 recordBeat(app.db, "skipped", "<b>hold</b> stands", new Date(NOW.getTime() - 3 * 60_000));
253 const page = String(keeperPanel(app.db, false, NOW));
254 expect(page).toContain('id="keeper-heartbeat"');
255 expect(page).toContain("skipped, 3 min ago: ");
256 expect(page).toContain('<dt>Release tick</dt>');
257 expect(page).toContain("&lt;b&gt;hold&lt;/b&gt;");
258 expect(page).not.toContain("<b>hold");
259 expect(page).toMatch(/<dt>Main vs production<\/dt><dd id="keeper-ahead">Main is green and 1 commit and 1 pull ahead/);
260 const read = readKeeper(new Session(app, { sshUsername: owner.username, remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user: owner }), NOW)!;
261 expect(read.heartbeat).toContain("3 min ago (skipped)");
262 expect(read.ahead).toContain("for 2 h 0 min");
263 });
264
265 test("with main on production, no ahead line, and the heartbeat says when it is quiet", () => {
266 reset();
267 check("ok", PROD);
268 recordBeat(app.db, "cut", "cut 0.7.536", new Date(NOW.getTime() - 90 * 60_000));
269 const read = readKeeper(new Session(app, { sshUsername: owner.username, remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user: owner }), NOW)!;
270 expect(read.ahead).toBeUndefined();
271 expect(read.heartbeat).toContain("No heartbeat from the release tick for 1 h 30 min");
272 expect(String(keeperPanel(app.db, false, NOW))).toContain("cut, 1 h 30 min ago (silent): cut 0.7.536");
273 expect(String(keeperPanel(app.db, false, NOW))).not.toContain("keeper-ahead");
274 });
275 });
276
277 describe("the terminal screen and /flow", () => {
278 test("the keeper screen draws the heartbeat and the ahead line", async () => {
279 reset();
280 check("ok", HEAD);
281 merge(120, "5".repeat(40));
282 recordBeat(app.db, "failed", "tag refused", new Date());
283 const id = { sshUsername: owner.username, remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user: owner };
284 const { app: tui, session } = createForgeApp(app, id, { terminal: headlessTerminal(160, 20), exitHooks: false });
285 tui.start();
286 tui.ctx.push(keeperScreen(session, tui.ctx));
287 await settleApp(tui);
288 const text = tui.screen.toText().replace(/\s+/g, " ");
289 tui.stop?.();
290 expect(text).toContain("The release tick last reported 0 min ago (failed): tag refused");
291 expect(text).toContain("Main is green and 1 commit and 1 pull ahead");
292 });
293
294 test("/flow shows the heartbeat with its reason to the owner and never to an agent", () => {
295 reset();
296 recordBeat(app.db, "skipped", "a private reason", new Date(NOW.getTime() - 5 * 60_000));
297 expect(flowFor(app, { kind: "owner" }, NOW).heartbeat?.reason).toBe("a private reason");
298 expect(flowFor(app, { kind: "agent" }, NOW).heartbeat).toBeNull();
299 expect(JSON.stringify(flowFor(app, { kind: "agent" }, NOW))).not.toContain("a private reason");
300 const url = new URL("http://forge.test/flow");
301 expect(String(flowPanel(app, { kind: "owner" }, url, "/flow"))).toContain("skipped, ");
302 expect(String(flowPanel(app, { kind: "agent" }, url, "/flow"))).not.toContain("a private reason");
303 });
304 });
305
306 describe("the keeper's own outcome", () => {
307 test("a cut, a failed verify, then a skip because the failed hold stands", async () => {
308 const f = await keeperForge();
309 expect((await tickStep(f.ports)).outcome).toBe("cut");
310 f.world.live = "0.7.284";
311 f.world.healthz = (n) => (n >= 5 ? { status: 503, ms: 2 } : { status: 200, ms: 2 });
312 const failed = await tickStep(f.ports);
313 expect(failed.outcome).toBe("failed");
314 expect(failed.line).toContain("failed its verify");
315 const held = await tickStep(f.ports);
316 expect(held.line).toContain("failed");
317 expect(held.outcome).toBe("skipped");
318 });
319 });
test/tui-release-keeper.test.ts +1 −1
@@ -56,6 +56,6 @@ describe("the release keeper screen", () => {
5656 test("with nothing cut, it says nothing holds the next cut and that no releases are kept yet", async () => {
5757 const f = await keeperForge();
5858 const read = readKeeper(new Session(f.app, identity(f.user)));
59 expect(read).toEqual({ blocked: "Nothing holds the next cut.", releases: [] });
59 expect(read).toEqual({ blocked: "Nothing holds the next cut.", heartbeat: "The release tick has never reported a heartbeat.", releases: [] });
6060 });
6161 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs