trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns3g rEventsFlows

Rule watcher: gate-only reds, rule breakers with exact fixes, flaky and load-sensitive budget tests #11260

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/7560-rule-watcher-gate-only-reds-rule-breaker → main 7 commits fork 3 threads · 3 unresolved

  • Merged. mason/7560-rule-watcher-gate-only-reds-rule-breaker was merged into main as 676fd12.

Review

Sign in to review this change.
24 files changed +921 −12
docs/features/rule-watcher.md added +12
@@ -0,0 +1,12 @@
1 # The rule watcher (#7560)
2
3 The maintenance duty `rule-watcher` (`src/maintain/duties/rule_watcher.ts`) runs every 15 minutes and reads the check record. Each read is idempotent: one open issue per title, one comment per text.
4
5 - **Gate-only reds** (`src/rules/gate_only.ts`): a test red in a merge-queue gate whose pulls were all green on their own latest check. Two such gates in a week file one issue ("Gate-only red: …") naming the test, the gates, and whether its file is in the pull check's rule set (`RULE_TESTS`, the #7138 class). One files none.
6 - **Rule breakers** (`src/rules/breaker.ts`, table in `src/rules/table.ts`): an open pull whose latest check is red on a rule test gets one comment with the rule, `file:line` and the fix. The same text is never posted twice; a red outside the table gets no comment. A new rule is one entry in `RULES`.
7 - **Flake trend** (`src/rules/flake_trend.ts`): once a week, the week's `check_flake` list ordered by the gate minutes each flake delayed; the worst three get an issue (the same `Flaky test:` title the check files, so it is not doubled). Quarantine is not allowed.
8 - **Budget headroom** (`src/rules/budget_rerun.ts`): a test whose name says it holds a speed, re-run green three times in a week, files "Budget headroom: …" with the p95 of its failed attempts (`check_flake.ms`, #7354).
9
10 Each run's findings and probes (each probe names the real query, to run again) are on `/maintenance` and, in the terminal, the "maintenance duty runs" screen (`maintenance-runs`, `src/tui/screens/maintainruns.ts`), which lists every duty including this one. The flake trend is said once a week, on the first run of the week, or when it files an issue.
11
12 Rule breaker comments are posted by the `rule-watcher` agent account (`src/rules/bot.ts`), never as the repository's owner. Issues the watcher files still go through the shared `fileIssueOnce`, authored by the owner as the check's own flake issues are.
src/checks/flake.ts +21 −5
@@ -36,8 +36,20 @@ const NAME_MAX = 300;
3636 export interface Flaky {
3737 readonly file: string;
3838 readonly name: string;
39 /** How long the failed attempt took, when the log said (#7560: the budget watcher's p95). */
40 readonly ms?: number;
3941 }
4042
43 const FAIL_TIME = /^\(fail\) .+? \[([0-9.]+)(m?s)\]/;
44 /** The milliseconds a `(fail) name [1.2ms]` line says, or undefined. */
45 export function failMs(line: string): number | undefined {
46 const m = FAIL_TIME.exec(line);
47 if (m === null) return undefined;
48 const n = Number(m[1]) * (m[2] === "s" ? 1000 : 1);
49 return Number.isFinite(n) ? Math.round(n) : undefined;
50 }
51 const withMs = (t: Flaky, ms: number | undefined): Flaky => (ms === undefined ? t : { ...t, ms });
52
4153 /**
4254 * The tests a failed check can re-run alone: at most {@link MAX_RERUN}
4355 * failures, every one named with the file whose report printed it. Anything
@@ -50,6 +62,7 @@ export function rerunnable(
5062 if (detail === undefined || detail.failed < 1 || detail.failed > MAX_RERUN) return undefined;
5163 if (detail.names.length !== detail.failed) return undefined;
5264 const found = new Map<string, string>();
65 const took = new Map<string, number>();
5366 let file: string | undefined;
5467 for (const line of lines) {
5568 const header = HEADER.exec(line);
@@ -58,8 +71,11 @@ export function rerunnable(
5871 continue;
5972 }
6073 const name = FAIL.exec(line)?.[1];
61 if (name !== undefined && detail.names.includes(name) && !found.has(name) && file !== undefined)
74 if (name !== undefined && detail.names.includes(name) && !found.has(name) && file !== undefined) {
6275 found.set(name, file);
76 const ms = failMs(line);
77 if (ms !== undefined) took.set(name, ms);
78 }
6379 }
6480 const out: Flaky[] = [];
6581 for (const name of detail.names) {
@@ -73,7 +89,7 @@ export function rerunnable(
7389 name.includes("(timeout)")
7490 )
7591 return undefined;
76 out.push({ file: at, name: name });
92 out.push(withMs({ file: at, name: name }, took.get(name)));
7793 }
7894 return out;
7995 }
@@ -232,8 +248,8 @@ export function recordFlakes(
232248 const opened: number[] = [];
233249 for (const t of tests) {
234250 db.query(
235 "INSERT INTO check_flake (id, repo_id, run_id, commit_oid, test, file, at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
236 ).run(id(), repo.id, ref.runId, ref.commit, t.name, t.file, new Date(now).toISOString());
251 "INSERT INTO check_flake (id, repo_id, run_id, commit_oid, test, file, at, ms) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
252 ).run(id(), repo.id, ref.runId, ref.commit, t.name, t.file, new Date(now).toISOString(), t.ms ?? null);
237253 const since = new Date(now - WINDOW_MS).toISOString();
238254 const count = db
239255 .query<{ n: number }, [string, string, string]>(
@@ -307,7 +323,7 @@ export function failedTests(lines: readonly string[]): Flaky[] {
307323 if (name === undefined || file === undefined || !PATH.test(file)) continue;
308324 if (name.length > NAME_MAX || CONTROL.test(name) || seen.has(`${file}\0${name}`)) continue;
309325 seen.add(`${file}\0${name}`);
310 out.push({ file, name });
326 out.push(withMs({ file, name }, failMs(line)));
311327 }
312328 return out;
313329 }
src/core/migration-pins/20261111120000-check-flake-ms.pin added +1
@@ -0,0 +1,1 @@
1 98a3fb7e1e4b46c3f12fa4dc4798741657a2b6169bfdeba7a67164950313e979
src/core/migrations/20261111120000-check-flake-ms.ts added +6
@@ -0,0 +1,6 @@
1 // How long the failed attempt of a flaked test took (#7560): the rule watcher
2 // reads the p95 of a budget test's attempts to say how much headroom it lacks.
3 // Null for rows written before this column. Additive.
4 export default `
5 ALTER TABLE check_flake ADD COLUMN ms INTEGER;
6 `;
src/maintain/duties/rule_watcher.ts added +28
@@ -0,0 +1,28 @@
1 /**
2 * The rule watcher (#7560): four reads of the check record, once an interval.
3 * The reads live in `src/rules/watch.ts`; this file only calls them in turn.
4 * Findings and actions are on `/maintenance` and the terminal's "maintenance
5 * duty runs" screen (`src/tui/screens/maintainruns.ts`), as for every duty.
6 */
7
8 import type { Database } from "bun:sqlite";
9 import type { Repo } from "../../core/repos.ts";
10 import { breakers, budgets, flakeWeek, gateOnly, type Watch } from "../../rules/watch.ts";
11 import type { Duty } from "../duty.ts";
12 import { lastStarted } from "../runs.ts";
13
14 function reposOf(db: Database): Map<string, Pick<Repo, "id" | "ownerId">> {
15 const rows = db.query<{ id: string; owner_id: string }, []>("SELECT id, owner_id FROM repo").all();
16 return new Map(rows.map((r) => [r.id, { id: r.id, ownerId: r.owner_id }]));
17 }
18
19 export const duty: Duty = {
20 name: "rule-watcher",
junipercrew outdated

FLAKE_SQL is not a real query; a probe must be reproducible. Likewise the LIKE-on-findings week de-dupe at ~line 40 is fragile.
21 intervalMs: 15 * 60_000,
22 run({ db, bus, now }) {
23 const w: Watch = { db, bus, repos: reposOf(db), now };
24 // The runner records this run after it returns, so the last start on record is the previous run's.
25 const reads = [breakers(w), gateOnly(w), flakeWeek(w, lastStarted(db, "rule-watcher")), budgets(w)];
26 return { findings: reads.flatMap((r) => r.findings), actions: reads.flatMap((r) => r.actions) };
27 },
28 };
src/maintain/duty.ts +3
@@ -1,6 +1,7 @@
11 /** What a maintenance duty is (#11204): a name, an interval and a handler; plain data in, plain data out. */
22
33 import type { Database } from "bun:sqlite";
4 import type { EventBus } from "../core/events.ts";
45 import type { App } from "../app.ts";
56
67 /** The number a finding rests on and when it was read: a query's result or a measurement. */
@@ -26,6 +27,8 @@ export interface DutyResult {
2627
2728 export interface DutyContext {
2829 readonly db: Database;
30 /** The forge's event bus, for a duty that files an issue. */
31 readonly bus: EventBus;
2932 readonly now: Date;
3033 /** The running app, for a duty that files or comments through the same actions a person uses. */
3134 readonly app?: App | undefined;
src/maintain/registry.ts +2 −1
@@ -2,7 +2,8 @@
22
33 import type { Duty } from "./duty.ts";
44 import { duty as pullsWaiting } from "./duties/stale_pulls.ts";
5 import { duty as ruleWatcher } from "./duties/rule_watcher.ts";
56 import { duty as efficiencyWatcher } from "./duties/efficiency_watcher.ts";
67 import { duty as flowKeeper } from "./duties/flow_keeper.ts";
78
8 export const duties: readonly Duty[] = [pullsWaiting, efficiencyWatcher, flowKeeper];
9 export const duties: readonly Duty[] = [pullsWaiting, ruleWatcher, efficiencyWatcher, flowKeeper];
src/maintain/runner.ts +1 −1
@@ -43,7 +43,7 @@ export function startMaintain(app: App, options: { duties?: readonly Duty[]; eve
4343 let outcome = "ok";
4444 try {
4545 // The duty keeps its `running` slot until it really settles, so a timed-out duty never overlaps itself.
46 const work = Promise.resolve().then(() => duty.run({ db: app.db, now, app }));
46 const work = Promise.resolve().then(() => duty.run({ db: app.db, bus: app.bus, now, app }));
4747 work.then(() => running.delete(duty.name), () => running.delete(duty.name));
4848 let timer: ReturnType<typeof setTimeout> | undefined;
4949 const deadline = new Promise<never>((_, reject) => {
src/rules/bot.ts added +20
@@ -0,0 +1,20 @@
1 /**
2 * The rule watcher's own account (#7560): an agent user with no password, so a
3 * comment it posts shows as the watcher's and never as the repository owner's
4 * words. Made on first use; the same row is found afterwards.
5 */
6
7 import type { Database } from "bun:sqlite";
8 import { id } from "../kernel/id.ts";
9
10 export const BOT_NAME = "rule-watcher";
11
12 export function watcherId(db: Database): string {
13 const found = db.query<{ id: string }, [string]>("SELECT id FROM user WHERE username = ?1 COLLATE NOCASE").get(BOT_NAME);
14 if (found !== null) return found.id;
15 const made = id();
16 db.query(
17 "INSERT OR IGNORE INTO user (id, username, email, password_hash, display_name, is_admin, is_agent) VALUES (?1, ?2, ?3, '', 'Rule watcher', 0, 1)",
18 ).run(made, BOT_NAME, `${BOT_NAME}@agents.invalid`);
19 return db.query<{ id: string }, [string]>("SELECT id FROM user WHERE username = ?1 COLLATE NOCASE").get(BOT_NAME)!.id;
20 }
src/rules/breaker.ts added +69
@@ -0,0 +1,69 @@
1 /**
2 * Rule breakers (#7560): when a pull's own check is red on a rule test, the
3 * pull gets one comment with the rule, the file and line, and the fix, from
4 * `table.ts`. A comment whose text a pull already carries is not repeated, so
5 * pushing again with the same breach says nothing new. A red that breaks no
6 * rule in the table gets no comment: never a vague "tests failed".
7 */
8
9 import type { Database } from "bun:sqlite";
10 import { addComment } from "../core/issues.ts";
11 import { watcherId } from "./bot.ts";
12 import { breachOf, type Breach } from "./table.ts";
13
14 /** How far back a failed check is read. */
15 export const LOOKBACK_MS = 24 * 3_600_000;
16
17 const MARK = "_Posted by the rule watcher, not written by a person._";
18
19 export function breakerComment(breaches: readonly Breach[]): string {
20 const lines = breaches.map(
21 (b) => `- **${b.rule}** at \`${b.file}${b.line === undefined ? "" : `:${b.line}`}\`\n Fix: ${b.fix}.`,
22 );
23 return `${MARK}\n\nThis pull's check is red on ${breaches.length === 1 ? "a repo rule" : "repo rules"}:\n\n${lines.join("\n")}`;
24 }
25
26 interface Failed {
27 readonly pull: string;
28 readonly repo: string;
29 readonly names: readonly string[];
30 }
31
32 /** The read the breaker rests on (`?1` = the earliest a failed check may be): each open pull's latest check, when failed. */
33 export const FAILED_CHECKS_SQL = `SELECT c.issue_id, c.repo_id, c.check_detail, c.id
34 FROM run c
35 JOIN issue i ON i.id = c.issue_id AND i.kind = 'pull' AND i.state = 'open'
36 WHERE c.kind = 'check' AND c.state = 'failed' AND c.created_at >= ?1 AND c.check_detail <> ''
37 AND c.id = (SELECT l.id FROM run l WHERE l.issue_id = c.issue_id AND l.kind = 'check' ORDER BY l.created_at DESC, l.id DESC LIMIT 1)
38 ORDER BY c.id`;
39
40 function failedChecks(db: Database, since: string): Failed[] {
41 const rows = db.query<{ issue_id: string; repo_id: string; check_detail: string; id: string }, [string]>(FAILED_CHECKS_SQL).all(since);
42 const out: Failed[] = [];
43 for (const r of rows) {
44 try {
45 const names = (JSON.parse(r.check_detail) as { names?: unknown }).names;
46 if (Array.isArray(names)) out.push({ pull: r.issue_id, repo: r.repo_id, names: names.filter((n): n is string => typeof n === "string") });
47 } catch {
48 // A detail this build cannot read says nothing.
49 }
50 }
51 return out;
52 }
53
54 /** Comment on each open pull whose failed check broke a rule; returns the pull ids commented, once per distinct text. */
55 export function commentBreakers(db: Database, now: Date): string[] {
56 const posted: string[] = [];
57 const bot = watcherId(db);
58 for (const f of failedChecks(db, new Date(now.getTime() - LOOKBACK_MS).toISOString())) {
59 const found = f.names.flatMap((n) => breachOf(n) ?? []);
60 if (found.length === 0) continue;
61 const body = breakerComment(found);
62 const same = db
63 .query<{ n: number }, [string, string]>("SELECT COUNT(*) AS n FROM comment WHERE issue_id = ?1 AND body = ?2")
64 .get(f.pull, body)!.n;
65 if (same > 0) continue;
66 if (addComment(db, { issueId: f.pull, authorId: bot, body }).ok) posted.push(f.pull);
junipercrew outdated

authorId is the repo owner: the comment appears as the owner's. Use a bot/system author.
67 }
68 return posted;
69 }
src/rules/budget_rerun.ts added +78
@@ -0,0 +1,78 @@
1 /**
2 * Budget tests under load (#7560): a test that holds a speed and failed in a
3 * full run, then passed when run alone, three times in a week has no headroom
4 * under the gate's load. One issue per such test, with the p95 of the failed
5 * attempts' measured times (`check_flake.ms`); #7354 is the first of its kind.
6 */
7
8 import type { Database } from "bun:sqlite";
9 import type { EventBus } from "../core/events.ts";
10 import type { Repo } from "../core/repos.ts";
11 import { fileIssueOnce } from "../checks/flake.ts";
12
13 export const WINDOW_MS = 7 * 24 * 3_600_000;
14 export const FILE_AT = 3;
15
16 /**
17 * A test whose name says it holds a speed: the word budget, a percentile, or
18 * "under"/"within" a stated time. A bare "N ms" or "answers in" is not enough:
19 * plenty of tests mention a time without being held to one.
20 */
21 export const BUDGET_NAME = /\b(budget|p9[059]|(under|within) \d[\d_.]* ?m?s)\b/i;
22
23 export interface BudgetRerun {
24 readonly repoId: string;
25 readonly test: string;
26 readonly file: string;
27 readonly passes: number;
28 /** Nearest-rank p95 of the attempts' ms; undefined when none was measured. */
29 readonly p95Ms: number | undefined;
30 }
31
32 export function p95(values: readonly number[]): number | undefined {
33 if (values.length === 0) return undefined;
34 const sorted = [...values].sort((a, b) => a - b);
35 return sorted[Math.max(0, Math.ceil(0.95 * sorted.length) - 1)];
36 }
37
38 /** The read behind the budget rows: `?1` = the week's start (ISO); names are then filtered by {@link BUDGET_NAME}. */
39 export const BUDGET_SQL = "SELECT repo_id, test, file, ms FROM check_flake WHERE at >= ?1 ORDER BY at";
40
41 export function budgetReruns(db: Database, now: number): BudgetRerun[] {
42 const rows = db
43 .query<{ repo_id: string; test: string; file: string; ms: number | null }, [string]>(BUDGET_SQL)
44 .all(new Date(now - WINDOW_MS).toISOString());
45 const by = new Map<string, { repoId: string; test: string; file: string; passes: number; ms: number[] }>();
46 for (const r of rows) {
47 if (!BUDGET_NAME.test(r.test)) continue;
48 const key = `${r.repo_id}\0${r.test}`;
49 const at = by.get(key) ?? { repoId: r.repo_id, test: r.test, file: r.file, passes: 0, ms: [] };
50 at.passes += 1;
51 if (r.file !== "") at.file = r.file;
52 if (r.ms !== null) at.ms.push(r.ms);
53 by.set(key, at);
54 }
55 return [...by.values()].map((b) => ({ repoId: b.repoId, test: b.test, file: b.file, passes: b.passes, p95Ms: p95(b.ms) }));
56 }
57
58 export const headroomTitle = (test: string): string => `Budget headroom: ${test.slice(0, 200)}`;
59
60 /** One headroom issue per budget test re-run green {@link FILE_AT} times in the week; returns the numbers opened. */
61 export function fileHeadroom(db: Database, bus: EventBus, repos: ReadonlyMap<string, Pick<Repo, "id" | "ownerId">>, now: number): number[] {
62 const opened: number[] = [];
63 for (const b of budgetReruns(db, now)) {
64 const repo = repos.get(b.repoId);
65 if (repo === undefined || b.passes < FILE_AT || b.file === "") continue;
66 const filed = fileIssueOnce(db, bus, repo, {
67 title: headroomTitle(b.test),
68 body:
69 `_Filed by the rule watcher from the flake record, not written by a person._\n\n` +
70 `\`${b.test}\` in \`${b.file}\` failed in a full run and passed when re-run alone ${b.passes} times this week: ` +
71 `its budget leaves no headroom under the gate's load (see #7354). ` +
72 `Measured p95 of the failed attempts: ${b.p95Ms === undefined ? "not measured (older rows carry no time)" : `${b.p95Ms} ms`}.\n\n` +
73 `Count work instead of milliseconds (\`expectWork\`), or give the budget room above that p95; do not quarantine it.\n\nFiles: ${b.file}`,
74 });
75 if (filed.opened) opened.push(filed.number);
76 }
77 return opened;
78 }
src/rules/flake_trend.ts added +79
@@ -0,0 +1,79 @@
1 /**
2 * The flake trend (#7560): the week's flaky tests from `check_flake`, ordered
3 * by the gate minutes each cost. A flake costs the minutes of the check it
4 * delayed: the gate's `duration_ms` when it failed in a gate, else the
5 * check run's own. The worst three get an issue for the crew to build (the
6 * same title the check's own filing uses, so one is never doubled). Quarantine
7 * is not an option the issue offers: the fix is a deterministic or
8 * budget-aware test.
9 */
10
11 import type { Database } from "bun:sqlite";
12 import type { EventBus } from "../core/events.ts";
13 import type { Repo } from "../core/repos.ts";
14 import { fileIssueOnce, flakeTitle } from "../checks/flake.ts";
15
16 export const WINDOW_MS = 7 * 24 * 3_600_000;
17 export const WORST = 3;
18
19 export interface Trend {
20 readonly repoId: string;
21 readonly repo: string;
22 readonly test: string;
23 readonly file: string;
24 readonly count: number;
25 /** Gate minutes the flakes delayed, one decimal. */
26 readonly minutes: number;
27 }
28
29 /** The read behind the trend: `?1` = the week's start (ISO), `?2` = the row limit. */
30 export const FLAKE_TREND_SQL = `SELECT f.repo_id AS repoId, u.username || '/' || r.name AS repo, f.test AS test, MAX(f.file) AS file,
31 COUNT(*) AS count,
32 ROUND(SUM(COALESCE(
33 (SELECT g.duration_ms FROM gate_result g WHERE g.run_id = f.run_id AND g.repo_id = f.repo_id LIMIT 1),
34 (SELECT MAX(0, (julianday(c.finished_at) - julianday(c.started_at)) * 86400000) FROM run c WHERE c.id = f.run_id),
35 0)) / 60000.0, 1) AS minutes
36 FROM check_flake f
37 JOIN repo r ON r.id = f.repo_id
38 JOIN user u ON u.id = r.owner_id
39 WHERE f.at >= ?1
40 GROUP BY f.repo_id, f.test
41 ORDER BY minutes DESC, count DESC, test
42 LIMIT ?2`;
43
44 /** Most minutes first; then most flakes; then the name, so the order is whole. */
45 export function flakeTrend(db: Database, now: number, limit = 20): Trend[] {
46 const rows = db
47 .query<Trend, [string, number]>(FLAKE_TREND_SQL)
48 .all(new Date(now - WINDOW_MS).toISOString(), limit);
49 return rows;
50 }
51
52 /** The week's list as lines for a summary: `test ×n, m min (repo)`. */
53 export const trendLines = (list: readonly Trend[]): string[] => list.map((t) => `${t.test} ×${t.count}, ${t.minutes} min (${t.repo})`);
54
55 /** An ISO-ish label for the week `now` falls in: the Monday's date. */
56 export function weekOf(now: number): string {
57 const d = new Date(now);
58 const back = (d.getUTCDay() + 6) % 7;
59 return new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate() - back)).toISOString().slice(0, 10);
60 }
61
62 /** An issue for each of the worst {@link WORST} that names a file; returns the numbers opened. */
63 export function fileWorst(db: Database, bus: EventBus, repos: ReadonlyMap<string, Pick<Repo, "id" | "ownerId">>, list: readonly Trend[]): number[] {
64 const opened: number[] = [];
65 for (const t of list.slice(0, WORST)) {
66 const repo = repos.get(t.repoId);
67 // No file, no "Files:" line to route it to a builder: no issue.
68 if (repo === undefined || t.file === "") continue;
69 const filed = fileIssueOnce(db, bus, repo, {
70 title: flakeTitle(t.test),
71 body:
72 `_Filed by the rule watcher from the week's flake list, not written by a person._\n\n` +
73 `\`${t.test}\` in \`${t.file}\` failed and passed alone ${t.count} time${t.count === 1 ? "" : "s"} this week, delaying checks by about ${t.minutes} minutes. ` +
74 `It is among the three costliest flakes. Make it deterministic or budget-aware; quarantine is not allowed.\n\nFiles: ${t.file}`,
75 });
76 if (filed.opened) opened.push(filed.number);
77 }
78 return opened;
79 }
src/rules/gate_only.ts added +126
@@ -0,0 +1,126 @@
1 /**
2 * Gate-only reds (#7560): a test that failed in a merge-queue gate while the
3 * pull the gate carried had passed its own check. Two such gates in a week
4 * file one issue naming the test and what differs; one files none. The pull's
5 * own check is its latest finished `check` run: green (`ok`) means every test
6 * it ran passed there. Reads `gate_result` (kept a week) and `run`.
7 */
8
9 import type { Database } from "bun:sqlite";
10 import type { EventBus } from "../core/events.ts";
11 import type { Repo } from "../core/repos.ts";
12 import { fileIssueOnce } from "../checks/flake.ts";
13 import { RULE_TESTS } from "../checks/tiers.ts";
14
15 export const WINDOW_MS = 7 * 24 * 3_600_000;
16 export const FILE_AT = 2;
17
18 export interface GateOnly {
19 readonly repoId: string;
20 readonly test: string;
21 readonly file: string;
22 /** Gates that failed it while their pulls' own checks were green. */
23 readonly gates: readonly { readonly id: string; readonly pulls: readonly number[] }[];
24 }
25
26 interface GateRow {
27 id: string;
28 repo_id: string;
29 pulls: string;
30 tests: string;
31 }
32
33 const numbers = (raw: string): number[] => {
34 try {
35 const v = JSON.parse(raw) as unknown;
36 return Array.isArray(v) ? v.filter((n): n is number => Number.isInteger(n)) : [];
37 } catch {
38 return [];
39 }
40 };
41
42 function failing(raw: string): { name: string; file: string }[] {
43 try {
44 const v = JSON.parse(raw) as unknown;
45 if (!Array.isArray(v)) return [];
46 return v.flatMap((t: { name?: unknown; file?: unknown }) =>
47 typeof t?.name === "string" ? [{ name: t.name, file: typeof t.file === "string" ? t.file : "" }] : [],
48 );
49 } catch {
50 return [];
51 }
52 }
53
54 /** The red gates of the week: `?1` = the week's start (ISO). */
55 export const GATE_RED_SQL = "SELECT id, repo_id, pulls, tests FROM gate_result WHERE state = 'red' AND finished_at >= ?1 ORDER BY id";
56
57 /**
58 * The latest finished own check of every pull the gates carried, in one read:
59 * `?1` = the repository ids, `?2` = the pull numbers, both as JSON arrays.
60 * Keyed `repoId#number`; true when it was green.
61 */
62 export const OWN_CHECK_SQL = `SELECT i.repo_id AS repo_id, i.number AS number,
63 (SELECT c.state FROM run c WHERE c.issue_id = i.id AND c.kind = 'check' AND c.state IN ('ok','failed')
64 ORDER BY c.created_at DESC, c.id DESC LIMIT 1) AS state
65 FROM issue i
66 WHERE i.kind = 'pull' AND i.repo_id IN (SELECT value FROM json_each(?1)) AND i.number IN (SELECT value FROM json_each(?2))`;
67
68 function ownChecksGreen(db: Database, repoIds: readonly string[], numbers: readonly number[]): Set<string> {
69 if (numbers.length === 0) return new Set();
70 const rows = db
71 .query<{ repo_id: string; number: number; state: string | null }, [string, string]>(OWN_CHECK_SQL)
72 .all(JSON.stringify(repoIds), JSON.stringify(numbers));
73 return new Set(rows.filter((r) => r.state === "ok").map((r) => `${r.repo_id}#${r.number}`));
74 }
75
76 /** The tests that are gate-only red at least once in the week, with the gates that showed it. */
77 export function gateOnlyReds(db: Database, now: number): GateOnly[] {
78 const rows = db.query<GateRow, [string]>(GATE_RED_SQL).all(new Date(now - WINDOW_MS).toISOString());
79 const carried = rows.map((g) => ({ g, pulls: numbers(g.pulls) }));
80 const green = ownChecksGreen(
81 db,
82 [...new Set(rows.map((g) => g.repo_id))],
83 [...new Set(carried.flatMap((c) => c.pulls))],
84 );
85 const by = new Map<string, { repoId: string; test: string; file: string; gates: { id: string; pulls: number[] }[] }>();
86 for (const { g, pulls } of carried) {
87 // A gate whose pulls were not all green on their own: its red is a pull's, not the gate's.
88 if (pulls.length === 0 || !pulls.every((n) => green.has(`${g.repo_id}#${n}`))) continue;
junipercrew outdated

pull: green[0] names the first pull for every test in the gate; wrong for multi-pull gates. Also ownCheckGreen above is an N+1 query per tick.
89 for (const t of failing(g.tests)) {
90 const key = `${g.repo_id}\0${t.name}`;
91 const at = by.get(key) ?? { repoId: g.repo_id, test: t.name, file: t.file, gates: [] };
92 at.gates.push({ id: g.id, pulls });
93 by.set(key, at);
94 }
95 }
96 return [...by.values()];
97 }
98
99 export const gateOnlyTitle = (test: string): string => `Gate-only red: ${test.slice(0, 200)}`;
100
101 /** Say what the pull's check did not run, from the rule set it always runs. */
102 function differs(file: string): string {
103 if (file !== "" && RULE_TESTS.includes(file))
104 return `\`${file}\` is in the pull check's rule set (the #7138 class) yet passed there: look for what the gate adds, the other pulls' changes in one candidate tree, or its load and file order.`;
105 return `${file === "" ? "The test's file is unknown" : `\`${file}\` is not in the pull check's rule set`}: the pull's fast check runs the tests its diff reaches, so a repo-wide scan, an ordering or a load-sensitive step shows only in the gate (#7138, #7354).`;
106 }
107
108 /** File one issue per test that was gate-only red {@link FILE_AT} times or more in the week; returns the numbers opened. */
109 export function fileGateOnly(db: Database, bus: EventBus, repos: ReadonlyMap<string, Pick<Repo, "id" | "ownerId">>, now: number): number[] {
110 const opened: number[] = [];
111 for (const g of gateOnlyReds(db, now)) {
112 const repo = repos.get(g.repoId);
113 if (repo === undefined || g.gates.length < FILE_AT) continue;
114 const shown = g.gates.slice(0, 5).map((x) => `gate ${x.id} (${x.pulls.length === 1 ? "pull" : "pulls"} ${x.pulls.map((n) => `#${n}`).join(", ")})`).join(", ");
115 const filed = fileIssueOnce(db, bus, repo, {
116 title: gateOnlyTitle(g.test),
117 body:
118 `_Filed by the rule watcher from the gate record, not written by a person._\n\n` +
119 `\`${g.test}\` failed in ${g.gates.length} merge-queue gates this week while each carried pull's own check was green: ${shown}.\n\n` +
120 `${differs(g.file)}\n\nMake it deterministic or budget-aware; do not quarantine it.` +
121 (g.file === "" ? "" : `\n\nFiles: ${g.file}`),
122 });
123 if (filed.opened) opened.push(filed.number);
124 }
125 return opened;
126 }
src/rules/table.ts added +66
@@ -0,0 +1,66 @@
1 /**
2 * The repo-wide rules and the fix for each (#7560): one entry per rule test.
3 * A failing rule test is named `<describe> > <breach>`, the breach being
4 * `path:line: source` (`test/helpers/rule.ts`) or `path is not pinned: …`.
5 * Plain text in, plain data out.
6 */
7
8 export interface RuleFix {
9 /** The rule, as its describe says it. */
10 readonly rule: string;
11 /** What the describe's name starts with. */
12 readonly describe: RegExp;
13 /** The fix, one sentence. */
14 readonly fix: string;
15 }
16
17 export const RULES: readonly RuleFix[] = [
18 {
19 rule: "no raw wall-clock budget in a test (#4429)",
20 describe: /^no raw wall-clock budget in a test \(#4429\)/,
21 fix: "hold the speed by counting work with `expectWork(measured, { max })` from test/helpers/work.ts (SQL statements, bytes, git spawns), not by milliseconds; a hang guard above 1 s may stay a plain expect",
22 },
23 {
24 rule: "no tight wall-clock bound in a test",
25 describe: /^no tight wall-clock bound in a test/,
26 fix: "assert a speed relative to a baseline measured in the same run, or bound it at 200 ms or more (WALL_CLOCK_FLOOR_MS in scripts/flake.ts) so only the failure the test names crosses it",
27 },
28 {
29 rule: "no wait that runs out of time and carries on (#9017)",
30 describe: /^a wait that runs out of time and carries on/,
31 fix: "wait on the event itself, or throw when the deadline passes instead of falling through to the next line",
32 },
33 {
34 rule: "every migration is pinned by its own pull (#783)",
35 describe: /^a migration on main is never edited/,
36 fix: "run `bun scripts/pin-migrations.ts` and commit the one `.pin` file it writes beside the migration; never edit a migration that is already pinned, add a new one",
37 },
38 ];
39
40 export interface Breach {
41 readonly rule: string;
42 readonly fix: string;
43 readonly file: string;
44 readonly line?: number;
45 /** The breach as the test named it, cut. */
46 readonly said: string;
47 }
48
49 const AT = /^([\w@./+-]{1,200}):(\d+): /;
50 const FILE_ONLY = /^([\w@./+-]{1,200}) is not pinned/;
51 const SAID_MAX = 200;
52
53 /** The rule a failing test's name breaks, with the place it names; undefined when it is no rule's. */
54 export function breachOf(name: string): Breach | undefined {
55 const cut = name.indexOf(" > ");
56 if (cut < 0) return undefined;
57 const entry = RULES.find((r) => r.describe.test(name.slice(0, cut)));
58 if (entry === undefined) return undefined;
59 const said = name.slice(cut + 3);
60 const at = AT.exec(said);
61 if (at !== null) return { rule: entry.rule, fix: entry.fix, file: at[1]!, line: Number(at[2]), said: said.slice(0, SAID_MAX) };
62 const file = FILE_ONLY.exec(said)?.[1];
63 // A rule test red with no place (its aggregate test) names no file: no exact comment.
64 if (file === undefined) return undefined;
65 return { rule: entry.rule, fix: entry.fix, file, said: said.slice(0, SAID_MAX) };
66 }
src/rules/watch.ts added +82
@@ -0,0 +1,82 @@
1 /**
2 * The rule watcher's four reads, each one function returning its findings and
3 * actions (#7560). The duty calls them in turn; each is idempotent (one open
4 * issue per title, one comment per text). A probe names the real query the
5 * finding rests on, so anyone can run it again.
6 */
7
8 import type { Database } from "bun:sqlite";
9 import type { EventBus } from "../core/events.ts";
10 import type { Repo } from "../core/repos.ts";
11 import type { Finding } from "../maintain/duty.ts";
12 import { BUDGET_SQL, budgetReruns, fileHeadroom, FILE_AT as BUDGET_AT } from "./budget_rerun.ts";
13 import { commentBreakers, FAILED_CHECKS_SQL } from "./breaker.ts";
14 import { FLAKE_TREND_SQL, fileWorst, flakeTrend, trendLines, weekOf } from "./flake_trend.ts";
15 import { fileGateOnly, GATE_RED_SQL, gateOnlyReds, FILE_AT as GATE_AT, OWN_CHECK_SQL } from "./gate_only.ts";
16
17 export interface Read {
18 readonly findings: Finding[];
19 readonly actions: string[];
20 }
21
22 export interface Watch {
23 readonly db: Database;
24 readonly bus: EventBus;
25 readonly repos: ReadonlyMap<string, Pick<Repo, "id" | "ownerId">>;
26 readonly now: Date;
27 }
28
29 const plural = (n: number, one: string, many = `${one}s`): string => `${n} ${n === 1 ? one : many}`;
30 const probe = (w: Watch, what: string, number: number) => ({ kind: "query" as const, what, number, at: w.now.toISOString() });
31 const numbered = (list: readonly number[]): string => `#${list.join(", #")}`;
32
33 export function breakers(w: Watch): Read {
34 const posted = commentBreakers(w.db, w.now);
35 if (posted.length === 0) return { findings: [], actions: [] };
36 return {
37 findings: [{ summary: `${plural(posted.length, "pull")} red on a repo rule`, probe: probe(w, FAILED_CHECKS_SQL, posted.length) }],
38 actions: [`commented the rule and its fix on ${posted.length} pull(s)`],
39 };
40 }
41
42 export function gateOnly(w: Watch): Read {
43 const at = w.now.getTime();
44 const reds = gateOnlyReds(w.db, at).filter((g) => g.gates.length >= GATE_AT);
45 const filed = fileGateOnly(w.db, w.bus, w.repos, at);
46 const findings: Finding[] = [];
47 if (reds.length > 0)
48 findings.push({
49 summary: `${plural(reds.length, "test")} red in the gate and green on the pull, ${GATE_AT}+ times this week`,
50 probe: probe(w, `${GATE_RED_SQL}; ${OWN_CHECK_SQL}`, reds.length),
51 });
52 return { findings, actions: filed.length > 0 ? [`filed gate-only issue(s) ${numbered(filed)}`] : [] };
53 }
54
55 /** `previousStart`: when the duty last started before this run; the week's trend is said on the first run of a week. */
56 export function flakeWeek(w: Watch, previousStart: number | undefined): Read {
57 const at = w.now.getTime();
58 const trend = flakeTrend(w.db, at);
59 if (trend.length === 0) return { findings: [], actions: [] };
60 const label = weekOf(at);
61 const worst = fileWorst(w.db, w.bus, w.repos, trend);
62 const firstOfWeek = previousStart === undefined || previousStart < Date.parse(label);
63 const actions = worst.length > 0 ? [`filed flake issue(s) ${numbered(worst)}`] : [];
64 if (!firstOfWeek && worst.length === 0) return { findings: [], actions };
65 return {
66 findings: [{ summary: `flake trend ${label}: ${trendLines(trend.slice(0, 5)).join("; ")}`, probe: probe(w, FLAKE_TREND_SQL, trend.length) }],
67 actions,
68 };
69 }
70
71 export function budgets(w: Watch): Read {
72 const at = w.now.getTime();
73 const reruns = budgetReruns(w.db, at).filter((b) => b.passes >= BUDGET_AT);
74 const filed = fileHeadroom(w.db, w.bus, w.repos, at);
75 const findings: Finding[] = [];
76 if (reruns.length > 0)
77 findings.push({
78 summary: `${plural(reruns.length, "budget test")} re-run green ${BUDGET_AT}+ times this week`,
79 probe: probe(w, BUDGET_SQL, reruns.length),
80 });
81 return { findings, actions: filed.length > 0 ? [`filed headroom issue(s) ${numbered(filed)}`] : [] };
82 }
test/check-flake.test.ts +1 −1
@@ -24,7 +24,7 @@ const report = (names: string[], file = "t/a.test.ts"): string[] => [
2424
2525 test("one or two named failures can be re-run; more, or a hook, cannot", () => {
2626 const one = report(["grp > flaky"]);
27 expect(rerunnable(one, parseBunTestFailure(one))).toEqual([{ file: "t/a.test.ts", name: "grp > flaky" }]);
27 expect(rerunnable(one, parseBunTestFailure(one))).toEqual([{ file: "t/a.test.ts", name: "grp > flaky", ms: 1 }]);
2828 const three = report(["a", "b", "c"]);
2929 expect(rerunnable(three, parseBunTestFailure(three))).toBeUndefined();
3030 const hook = ["t/a.test.ts:", "(fail) t/a.test.ts > beforeEach (timeout)", " 1 pass", " 1 fail"];
test/full-suite-report.test.ts +3 −3
@@ -58,8 +58,8 @@ const BUN_LOG = [
5858 describe("failedTests", () => {
5959 test("bun's closing summary gives no failure to the last file printed", () => {
6060 expect(failedTests(BUN_LOG)).toEqual([
61 { file: "test/a.test.ts", name: "a > one" },
62 { file: "test/b.test.ts", name: "b > two" },
61 { file: "test/a.test.ts", name: "a > one", ms: 0 },
62 { file: "test/b.test.ts", name: "b > two", ms: 0 },
6363 ]);
6464 expect(failedFiles(BUN_LOG)).toEqual(["test/a.test.ts", "test/b.test.ts"]);
6565 });
@@ -84,7 +84,7 @@ describe("failedTests", () => {
8484 "(fail) b > two",
8585 ];
8686 expect(failedTests(lines)).toEqual([
87 { file: "test/a.test.ts", name: "a > one" },
87 { file: "test/a.test.ts", name: "a > one", ms: 3 },
8888 { file: "test/b.test.ts", name: "b > two" },
8989 ]);
9090 });
test/helpers/keeper-seed.ts +1 −1
@@ -82,5 +82,5 @@ export async function seed(names: string[] = ["ada", "bo"]): Promise<Seed> {
8282 export async function runKeeper(s: Seed, now: Date, deps: import("../../src/maintain/duties/flow_keeper.ts").Deps & { withApp?: boolean } = {}) {
8383 const { flowKeeper } = await import("../../src/maintain/duties/flow_keeper.ts");
8484 const { withApp, ...rest } = deps;
85 return flowKeeper({ builders: () => undefined, ...(rest.vantages === undefined && !withApp ? { vantages: () => [] } : {}), ...rest }).run({ db: s.db, now, ...(withApp ? { app: s.app } : {}) });
85 return flowKeeper({ builders: () => undefined, ...(rest.vantages === undefined && !withApp ? { vantages: () => [] } : {}), ...rest }).run({ db: s.db, bus: s.app.bus, now, ...(withApp ? { app: s.app } : {}) });
8686 }
test/helpers/rules-app.ts added +39
@@ -0,0 +1,39 @@
1 /** An app with one repo owned by alice, for the rule watcher's tests (#7560). */
2 import { afterAll, beforeAll } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createRepo } from "../../src/actions/repos.ts";
7 import { createApp, type App } from "../../src/app.ts";
8 import { createUser } from "../../src/core/users.ts";
9 import { quietLogs } from "./globals.ts";
10
11 export interface RulesWorld {
12 app: App;
13 repoId: string;
14 ownerId: string;
15 repos: Map<string, { id: string; ownerId: string }>;
16 titles(): string[];
17 }
18
19 export function rulesWorld(): RulesWorld {
20 quietLogs();
21 let dir = "";
22 const world = {} as RulesWorld;
23 beforeAll(async () => {
24 dir = await mkdtemp(join(tmpdir(), "rules-"));
25 world.app = createApp({ dataDir: dir, port: 0, host: "forge.test" });
26 const u = await createUser(world.app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
27 if (!u.ok) throw new Error(u.error);
28 const made = await createRepo(world.app, u.value, { name: "rules", visibility: "public" });
29 if (!made.ok) throw new Error("repo");
30 world.repoId = made.value.id;
31 world.ownerId = u.value.id;
32 world.repos = new Map([[made.value.id, { id: made.value.id, ownerId: u.value.id }]]);
33 world.titles = () => world.app.db.query<{ title: string }, [string]>("SELECT title FROM issue WHERE repo_id = ?1 AND kind = 'issue'").all(world.repoId).map((r) => r.title);
34 });
35 afterAll(async () => {
36 await rm(dir, { recursive: true, force: true });
37 });
38 return world;
39 }
test/helpers/rules-seed.ts added +45
@@ -0,0 +1,45 @@
1 /** Rows the rule watcher reads (#7560): a repo with pulls, their check runs, gate results and flakes, written straight. */
2 import type { Database } from "bun:sqlite";
3 import { createIssue } from "../../src/core/issues.ts";
4 import { createRun } from "../../src/core/runs.ts";
5 import { id } from "../../src/kernel/id.ts";
6
7 let last = 0;
8 /** A clock that never repeats, so two runs seeded in one millisecond keep their order. */
9 export const tick = (): number => (last = Math.max(Date.now(), last + 1));
10
11 export const iso = (ms: number): string => new Date(ms).toISOString();
12
13 export function seedPull(db: Database, repoId: string, authorId: string, title = "a pull"): { id: string; number: number } {
14 const made = createIssue(db, { repoId, authorId, title, body: "", kind: "pull" });
15 if (!made.ok) throw new Error(made.error);
16 return { id: made.value.id, number: made.value.number };
17 }
18
19 /** A finished pull check: `ok`, or `failed` naming the failing tests. */
20 export function seedCheck(db: Database, repoId: string, pullId: string, state: "ok" | "failed", names: string[] = [], at = tick()): string {
21 const run = createRun(db, { repoId, kind: "check", issueId: pullId });
22 db.query("UPDATE run SET state = ?2, check_detail = ?3, created_at = ?4, started_at = ?4, finished_at = ?5 WHERE id = ?1").run(
23 run.id,
24 state,
25 state === "failed" ? JSON.stringify({ passed: 1, failed: names.length, names }) : "",
26 iso(at),
27 iso(at + 120_000),
28 );
29 return run.id;
30 }
31
32 /** Keeps one red gate; returns its check-lane run id, which a flake in that gate carries. */
33 export function seedGate(db: Database, repoId: string, pulls: number[], failing: { name: string; file?: string }[], at = Date.now(), minutes = 10): string {
34 const gate = id();
35 const runId = `gate-${gate}`;
36 db.query(
37 `INSERT INTO gate_result (id, repo_id, run_id, tip_oid, pulls, state, failed, tests, started_at, finished_at, duration_ms)
38 VALUES (?1, ?2, ?3, 'abc', ?4, 'red', ?5, ?6, ?7, ?8, ?9)`,
39 ).run(gate, repoId, runId, JSON.stringify(pulls), failing.length, JSON.stringify(failing.map((f) => ({ ...f, errors: [] }))), iso(at - minutes * 60_000), iso(at), minutes * 60_000);
40 return runId;
41 }
42
43 export function seedFlake(db: Database, repoId: string, test: string, file: string, at = Date.now(), ms: number | null = null, runId = id()): void {
44 db.query("INSERT INTO check_flake (id, repo_id, run_id, commit_oid, test, file, at, ms) VALUES (?1, ?2, ?3, 'abc', ?4, ?5, ?6, ?7)").run(id(), repoId, runId, test, file, iso(at), ms);
45 }
test/rules-breaker-comment.test.ts added +69
@@ -0,0 +1,69 @@
1 /** Rule breakers (#7560): one comment with rule, place and fix; a repeat says nothing; a plain red gets none. */
2 import { expect, test } from "bun:test";
3 import { commentBreakers } from "../src/rules/breaker.ts";
4 import { breachOf } from "../src/rules/table.ts";
5 import { rulesWorld } from "./helpers/rules-app.ts";
6 import { seedCheck, seedPull } from "./helpers/rules-seed.ts";
7
8 const w = rulesWorld();
9 const RAW = "no raw wall-clock budget in a test (#4429) > test/x.test.ts:4: expect(performance.now() - t0).toBe" + "LessThan(5);";
10 const PIN = "a migration on main is never edited (#783) > src/core/migrations/2026-x.ts is not pinned: run bun scripts/pin-migrations.ts";
11 const comments = (pullId: string) => w.app.db.query<{ body: string }, [string]>("SELECT body FROM comment WHERE issue_id = ?1").all(pullId).map((c) => c.body);
12
13 test("the table names the rule, the file, the line and the fix", () => {
14 expect(breachOf(RAW)).toMatchObject({ file: "test/x.test.ts", line: 4, rule: expect.stringContaining("#4429"), fix: expect.stringContaining("expectWork") });
15 expect(breachOf("a migration on main is never edited (#783) > src/core/migrations/2026-x.ts is not pinned: run it")).toMatchObject({ file: "src/core/migrations/2026-x.ts", fix: expect.stringContaining("pin-migrations") });
16 expect(breachOf("some other > test")).toBeUndefined();
17 expect(breachOf("no raw wall-clock budget in a test (#4429) > clean")).toBeUndefined();
18 });
19
20 test("a pull red on a rule gets one comment; pushing again with the same breach adds none", () => {
21 const p = seedPull(w.app.db, w.repoId, w.ownerId);
22 seedCheck(w.app.db, w.repoId, p.id, "failed", [RAW]);
23 expect(commentBreakers(w.app.db, new Date())).toEqual([p.id]);
24 const [body] = comments(p.id);
25 expect(comments(p.id)).toHaveLength(1);
26 expect(body).toContain("test/x.test.ts:4");
27 expect(body).toContain("expectWork");
28 expect(body).toContain("#4429");
29
30 seedCheck(w.app.db, w.repoId, p.id, "failed", [RAW]);
31 expect(commentBreakers(w.app.db, new Date())).toEqual([]);
32 expect(commentBreakers(w.app.db, new Date())).toEqual([]);
33 expect(comments(p.id)).toHaveLength(1);
34 });
35
36 test("a different breach is a new comment; a red that breaks no rule, or a green check, gets none", () => {
37 const p = seedPull(w.app.db, w.repoId, w.ownerId);
38 seedCheck(w.app.db, w.repoId, p.id, "failed", [RAW]);
39 commentBreakers(w.app.db, new Date());
40 seedCheck(w.app.db, w.repoId, p.id, "failed", [PIN]);
41 expect(commentBreakers(w.app.db, new Date())).toEqual([p.id]);
42 expect(comments(p.id)).toHaveLength(2);
43
44 const plain = seedPull(w.app.db, w.repoId, w.ownerId);
45 seedCheck(w.app.db, w.repoId, plain.id, "failed", ["grp > ordinary failure"]);
46 const green = seedPull(w.app.db, w.repoId, w.ownerId);
47 seedCheck(w.app.db, w.repoId, green.id, "ok");
48 expect(commentBreakers(w.app.db, new Date())).toEqual([]);
49 expect(comments(plain.id)).toEqual([]);
50 expect(comments(green.id)).toEqual([]);
51 });
52
53 test("only a pull's latest check speaks: a breach already fixed is not commented", () => {
54 const p = seedPull(w.app.db, w.repoId, w.ownerId);
55 seedCheck(w.app.db, w.repoId, p.id, "failed", [RAW.replace("x.test", "fixed.test")]);
56 seedCheck(w.app.db, w.repoId, p.id, "ok");
57 expect(commentBreakers(w.app.db, new Date())).toEqual([]);
58 });
59
60 test("the comment is the rule watcher's, never the repository owner's", () => {
61 const p = seedPull(w.app.db, w.repoId, w.ownerId);
62 seedCheck(w.app.db, w.repoId, p.id, "failed", [RAW]);
63 commentBreakers(w.app.db, new Date());
64 const row = w.app.db
65 .query<{ author_id: string; username: string; is_agent: number }, [string]>("SELECT c.author_id, u.username, u.is_agent FROM comment c JOIN user u ON u.id = c.author_id WHERE c.issue_id = ?1")
66 .get(p.id)!;
67 expect(row.author_id).not.toBe(w.ownerId);
68 expect(row).toMatchObject({ username: "rule-watcher", is_agent: 1 });
69 });
test/rules-budget-rerun-count.test.ts added +57
@@ -0,0 +1,57 @@
1 /** Budget tests under load (#7560): re-run green three times in a week files a headroom issue with its p95. */
2 import { expect, test } from "bun:test";
3 import { duty } from "../src/maintain/duties/rule_watcher.ts";
4 import { budgetReruns, fileHeadroom, headroomTitle, p95 } from "../src/rules/budget_rerun.ts";
5 import { failMs, recordFlakes } from "../src/checks/flake.ts";
6 import { rulesWorld } from "./helpers/rules-app.ts";
7 import { seedFlake } from "./helpers/rules-seed.ts";
8
9 const w = rulesWorld();
10 const now = Date.now();
11 const BUDGET = "feed > the feed answers in under 50" + " ms p95 budget";
12
13 test("p95 is the nearest rank; the failed attempt's time is read from its line", () => {
14 expect(p95([])).toBeUndefined();
15 expect(p95([10, 20, 30, 40, 50, 60, 70, 80, 90, 100])).toBe(100);
16 expect(p95([5])).toBe(5);
17 expect(failMs("(fail) a > b [123.45ms]")).toBe(123);
18 expect(failMs("(fail) a > b [1.5s]")).toBe(1500);
19 expect(failMs("(fail) a > b")).toBeUndefined();
20 });
21
22 test("two re-run passes file nothing; the third files one issue with the measured p95", () => {
23 seedFlake(w.app.db, w.repoId, BUDGET, "test/feed.test.ts", now - 3000, 60);
24 seedFlake(w.app.db, w.repoId, BUDGET, "test/feed.test.ts", now - 2000, 90);
25 seedFlake(w.app.db, w.repoId, "feed > ordinary flake", "test/feed.test.ts", now - 2000);
26 seedFlake(w.app.db, w.repoId, "feed > ordinary flake", "test/feed.test.ts", now - 1000);
27 seedFlake(w.app.db, w.repoId, "feed > ordinary flake", "test/feed.test.ts", now - 500);
28 expect(fileHeadroom(w.app.db, w.app.bus, w.repos, now)).toEqual([]);
29 seedFlake(w.app.db, w.repoId, BUDGET, "test/feed.test.ts", now - 1000, 120);
30 expect(budgetReruns(w.app.db, now).map((b) => [b.test, b.passes, b.p95Ms])).toEqual([[BUDGET, 3, 120]]);
31 expect(fileHeadroom(w.app.db, w.app.bus, w.repos, now)).toHaveLength(1);
32 const body = w.app.db.query<{ body: string }, [string]>("SELECT body FROM issue WHERE title = ?1").get(headroomTitle(BUDGET))!.body;
33 expect(body).toContain("p95 of the failed attempts: 120 ms");
34 expect(body).toContain("#7354");
35 expect(fileHeadroom(w.app.db, w.app.bus, w.repos, now)).toEqual([]);
36 });
37
38 test("recordFlakes keeps the time the failed attempt took", () => {
39 recordFlakes(w.app.db, w.app.bus, w.repos.get(w.repoId)!, [{ file: "test/t.test.ts", name: "t > timed", ms: 77 }], { runId: "r1", commit: "abc" });
40 expect(w.app.db.query<{ ms: number }, []>("SELECT ms FROM check_flake WHERE test = 't > timed'").get()!.ms).toBe(77);
41 });
42
43 test("the duty runs all four reads and reports each finding on a probe", async () => {
44 const result = await duty.run({ db: w.app.db, bus: w.app.bus, now: new Date(now) });
45 expect(result.findings.some((f) => f.summary.includes("budget test"))).toBe(true);
46 expect(result.findings.some((f) => f.summary.startsWith("flake trend "))).toBe(true);
47 for (const f of result.findings) expect(f.probe?.what).toBeTruthy();
48 });
49
50 test("a name that only mentions a time is not a budget test", async () => {
51 const { BUDGET_NAME } = await import("../src/rules/budget_rerun.ts");
52 expect(BUDGET_NAME.test("the feed answers within the latency budget")).toBe(true);
53 expect(BUDGET_NAME.test(["the feed answers in", "under 50", "ms"].join(" "))).toBe(true);
54 expect(BUDGET_NAME.test("p95 holds")).toBe(true);
55 expect(BUDGET_NAME.test("retries after 500 ms of silence")).toBe(false);
56 expect(BUDGET_NAME.test("the page answers in json")).toBe(false);
57 });
test/rules-flake-trend.test.ts added +49
@@ -0,0 +1,49 @@
1 /** The flake trend (#7560): ordered by gate minutes delayed; the worst three get issues, once. */
2 import { expect, test } from "bun:test";
3 import { fileWorst, flakeTrend, trendLines, weekOf } from "../src/rules/flake_trend.ts";
4 import { rulesWorld } from "./helpers/rules-app.ts";
5 import { seedFlake, seedGate } from "./helpers/rules-seed.ts";
6
7 const w = rulesWorld();
8 const now = Date.now();
9
10 test("the list is ordered by wasted gate minutes, not by count", () => {
11 // Many flakes in short gates, few in long ones.
12 for (let i = 0; i < 4; i++) seedFlake(w.app.db, w.repoId, "busy > cheap", "test/cheap.test.ts", now - i * 1000, null, seedGate(w.app.db, w.repoId, [], [], now - i * 1000, 0.25));
13 for (let i = 0; i < 2; i++) seedFlake(w.app.db, w.repoId, "slow > costly", "test/costly.test.ts", now - i * 1000, null, seedGate(w.app.db, w.repoId, [], [], now - i * 1000, 30));
14 seedFlake(w.app.db, w.repoId, "mid > one", "test/mid.test.ts", now, null, seedGate(w.app.db, w.repoId, [], [], now, 12));
15 seedFlake(w.app.db, w.repoId, "tiny > one", "test/tiny.test.ts", now, null, seedGate(w.app.db, w.repoId, [], [], now, 2));
16 const list = flakeTrend(w.app.db, now);
17 expect(list.map((t) => t.test)).toEqual(["slow > costly", "mid > one", "tiny > one", "busy > cheap"]);
18 expect(list[0]).toMatchObject({ count: 2, minutes: 60 });
19 expect(trendLines(list)[0]).toBe("slow > costly ×2, 60 min (alice/rules)");
20 });
21
22 test("the worst three get one issue each, with no quarantine; a repeat files none; the fourth gets none", () => {
23 const list = flakeTrend(w.app.db, now);
24 expect(fileWorst(w.app.db, w.app.bus, w.repos, list)).toHaveLength(3);
25 expect(w.titles()).toEqual(expect.arrayContaining(["Flaky test: slow > costly", "Flaky test: mid > one", "Flaky test: tiny > one"]));
26 expect(w.titles()).not.toContain("Flaky test: busy > cheap");
27 expect(fileWorst(w.app.db, w.app.bus, w.repos, list)).toEqual([]);
28 const body = w.app.db.query<{ body: string }, []>("SELECT body FROM issue WHERE title = 'Flaky test: slow > costly'").get()!.body;
29 expect(body).toContain("quarantine is not allowed");
30 expect(body).toContain("60 minutes");
31 });
32
33 test("the week is labelled by its Monday", () => {
34 expect(weekOf(Date.parse("2026-11-11T10:00:00Z"))).toBe("2026-11-09");
35 expect(weekOf(Date.parse("2026-11-15T23:59:00Z"))).toBe("2026-11-09");
36 expect(weekOf(Date.parse("2026-11-16T00:00:00Z"))).toBe("2026-11-16");
37 });
38
39 test("the trend is said on the first run of a week, and its probe is the real query", async () => {
40 const { flakeWeek } = await import("../src/rules/watch.ts");
41 const { FLAKE_TREND_SQL } = await import("../src/rules/flake_trend.ts");
42 const watch = { db: w.app.db, bus: w.app.bus, repos: w.repos, now: new Date(now) };
43 const monday = Date.parse(weekOf(now));
44 const first = flakeWeek(watch, monday - 1000);
45 expect(first.findings).toHaveLength(1);
46 expect(first.findings[0]!.probe).toMatchObject({ what: FLAKE_TREND_SQL });
47 expect(() => w.app.db.query(FLAKE_TREND_SQL).all("2000-01-01T00:00:00Z", 5)).not.toThrow();
48 expect(flakeWeek(watch, monday + 1000).findings).toEqual([]);
49 });
test/rules-gate-only.test.ts added +63
@@ -0,0 +1,63 @@
1 /** Gate-only reds (#7560): two in a week file one issue; one files none; a red the pull shared is not gate-only. */
2 import { expect, test } from "bun:test";
3 import { fileGateOnly, gateOnlyTitle } from "../src/rules/gate_only.ts";
4 import { rulesWorld } from "./helpers/rules-app.ts";
5 import { seedCheck, seedGate, seedPull } from "./helpers/rules-seed.ts";
6
7 const w = rulesWorld();
8 const NAME = "sweep > repo-wide scan";
9 const now = Date.now();
10
11 test("a test red once in the gate with its pull green files nothing; twice files one issue, once more", () => {
12 const p1 = seedPull(w.app.db, w.repoId, w.ownerId);
13 const p2 = seedPull(w.app.db, w.repoId, w.ownerId);
14 seedCheck(w.app.db, w.repoId, p1.id, "ok");
15 seedCheck(w.app.db, w.repoId, p2.id, "ok");
16 seedGate(w.app.db, w.repoId, [p1.number], [{ name: NAME, file: "test/unreached.test.ts" }], now - 1000);
17 expect(fileGateOnly(w.app.db, w.app.bus, w.repos, now)).toEqual([]);
18 expect(w.titles()).not.toContain(gateOnlyTitle(NAME));
19
20 seedGate(w.app.db, w.repoId, [p2.number], [{ name: NAME, file: "test/unreached.test.ts" }], now - 500);
21 const opened = fileGateOnly(w.app.db, w.app.bus, w.repos, now);
22 expect(opened).toHaveLength(1);
23 expect(w.titles().filter((t) => t === gateOnlyTitle(NAME))).toHaveLength(1);
24 const body = w.app.db.query<{ body: string }, [string]>("SELECT body FROM issue WHERE title = ?1").get(gateOnlyTitle(NAME))!.body;
25 expect(body).toContain("pull #");
26 expect(body).toContain("Files: test/unreached.test.ts");
27 expect(body).toContain("do not quarantine");
28
29 expect(fileGateOnly(w.app.db, w.app.bus, w.repos, now)).toEqual([]);
30 });
31
32 test("a red whose pull was red on its own check is the pull's, not the gate's", () => {
33 const a = seedPull(w.app.db, w.repoId, w.ownerId);
34 const b = seedPull(w.app.db, w.repoId, w.ownerId);
35 seedCheck(w.app.db, w.repoId, a.id, "failed", ["x > y"]);
36 seedCheck(w.app.db, w.repoId, b.id, "failed", ["x > y"]);
37 seedGate(w.app.db, w.repoId, [a.number], [{ name: "x > shared red", file: "test/x.test.ts" }], now - 300);
38 seedGate(w.app.db, w.repoId, [b.number], [{ name: "x > shared red", file: "test/x.test.ts" }], now - 200);
39 expect(fileGateOnly(w.app.db, w.app.bus, w.repos, now)).toEqual([]);
40 });
41
42 test("a gate older than the week counts for nothing", () => {
43 const a = seedPull(w.app.db, w.repoId, w.ownerId);
44 seedCheck(w.app.db, w.repoId, a.id, "ok");
45 const old = now - 8 * 86_400_000;
46 seedGate(w.app.db, w.repoId, [a.number], [{ name: "old > red", file: "test/o.test.ts" }], old);
47 seedGate(w.app.db, w.repoId, [a.number], [{ name: "old > red", file: "test/o.test.ts" }], old + 1000);
48 expect(fileGateOnly(w.app.db, w.app.bus, w.repos, now)).toEqual([]);
49 });
50
51 test("a gate that carried several pulls names every one of them, not the first", () => {
52 const a = seedPull(w.app.db, w.repoId, w.ownerId);
53 const b = seedPull(w.app.db, w.repoId, w.ownerId);
54 seedCheck(w.app.db, w.repoId, a.id, "ok");
55 seedCheck(w.app.db, w.repoId, b.id, "ok");
56 const name = "sweep > multi-pull scan";
57 seedGate(w.app.db, w.repoId, [a.number, b.number], [{ name, file: "test/m.test.ts" }], now - 900);
58 seedGate(w.app.db, w.repoId, [b.number], [{ name, file: "test/m.test.ts" }], now - 800);
59 fileGateOnly(w.app.db, w.app.bus, w.repos, now);
60 const body = w.app.db.query<{ body: string }, [string]>("SELECT body FROM issue WHERE title = ?1").get(gateOnlyTitle(name))!.body;
61 expect(body).toContain(`pulls #${a.number}, #${b.number}`);
62 expect(body).toContain(`pull #${b.number})`);
63 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs