Public: anyone can read this; the crew can change it.

A failed production probe files one defect carrying the URL, status, bytes and time, once per day #13273

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13265-a-failed-production-probe-files-one-defe → main 1 commit fork

  • Merged. scout/13265-a-failed-production-probe-files-one-defe was merged into main as 6f34bd9.

Review

Sign in to review this change.
3 files changed +156 −60
docs/features/v1-proven.md +15 −10
@@ -10,16 +10,21 @@ A weekly maintenance duty, `v1-proven` (#12989), runs the production probes of `
1010 (`2026-11-15 fail #123 <item>`), a failing probe is also a finding. Read them with
1111 `GET /api/v1/maintenance/runs` (admin token) or in the terminal's "Maintenance duty runs" screen. The web
1212 `/maintenance` page is the power-off preparation and does not list duty runs.
13 - **A defect issue.** A failing probe opens one issue in the forge's own repository (`FORGE_REPO`, e.g. `owner/forge`),
14 titled `Defect of #<probe issue>: <item>`, whose body quotes the failing line, carries the stable line
15 `Probe key: v1-proven/<probe issue>` and says to fix the cause and close it. With no `FORGE_REPO` nothing is filed and
16 the run says "no issue filed".
17 - **A repeat failure.** The open defect of a probe is found by that `Probe key:` line in the body, so renaming the title
18 loses nothing; a defect filed before the key existed is found by its `Defect of #<probe issue>: ` title. While it is
19 open, the same probe failing again adds a comment ("Still failing on <date>") and no second issue. Two probes failing
20 in one run open two issues. Once the issue is closed, a new failure opens a new one.
21 - **A pass.** A probe that passes while its defect is open adds one comment ("Passing again on <date>") and leaves the
22 defect open: a person closes it. Repeated passes add nothing; a failure after a pass comments again, and the next pass
13 - **A defect issue.** A run with failing probes opens one issue in the forge's own repository (`FORGE_REPO`, e.g.
14 `owner/forge`), not one per probe (#13265), titled `Defect of the production proof: <n> lines fail`. Its body lists each
15 failing line, each with the evidence block of what that probe read (URL, status, bytes, ms, time), carries the stable
16 line `Probe key: v1-proven/run` and says to fix the cause and close it. With no `FORGE_REPO` nothing is filed and the
17 run says "no issue filed".
18 - **Cannot reach.** Probes whose read got no HTTP answer at all fold into one line, `cannot reach <origin>: <k> of <n>
19 probes got no answer`; when that is the only failure the title is `Defect of the production proof: cannot reach
20 <origin>`. An unreachable forge files one line, not one per probe.
21 - **A repeat failure.** The open defect is found by its `Probe key:` line, so renaming the title loses nothing. While it
22 is open, a run that fails again (a second run the same day, or next week's) adds one comment ("Still failing on
23 <date>") listing that run's failing lines, and no second issue. Once it is closed, a new failure opens a new one. A
24 per-probe defect filed before #13265 (`Probe key: v1-proven/<probe issue>` or the title `Defect of #<probe issue>: `)
25 still takes its own probe's comments while it is open.
26 - **A pass.** A run with no failing line while the defect is open adds one comment ("Passing again on <date>") and leaves
27 it open: a person closes it. Repeated passes add nothing; a failure after a pass comments again, and the next pass
2328 notes again. A pass with no open defect says nothing.
2429 - **Limits.** Runs at most twice a day, weekly by default, and spends no money. The findings only land on the run record;
2530 the runner files no notice of its own.
src/maintain/proven.ts +87 −38
@@ -1,8 +1,11 @@
11 /**
22 * v1.0.0 stays proven (#12989): the production probes of `scripts/v1-proven.ts` (the one list,
3 * `core/v1_proof.ts`) run once a week against the forge's own public origin, GET only. A failing
4 * probe opens one issue `Defect of #<probe issue>: …` quoting its line; a probe that fails again
5 * while that issue is open adds a comment instead. Plain data in, plain data out.
3 * `core/v1_proof.ts`) run once a week against the forge's own public origin, GET only. A run
4 * that finds failing probes opens one issue `Defect of the production proof: …` listing each
5 * failing line with what it read (#13265); a run while that issue is open (a second run the same
6 * day, or next week's) adds a comment instead; once closed, a new failure is a new issue. Probes
7 * that got no HTTP answer at all are one line, `cannot reach <origin>`, not one per probe. Plain
8 * data in, plain data out.
69 */
710
811 import type { Database } from "bun:sqlite";
@@ -11,58 +14,104 @@ import { line, prove, type Proof, type Result } from "../core/v1_proof.ts";
1114 import { watcherId } from "../rules/bot.ts";
1215 import { evidenceBlock } from "./evidence.ts";
1316
14 export const defectTitle = (r: Pick<Result, "issue" | "item">): string => `Defect of #${r.issue}: ${r.item}`;
17 const DEFECT_PREFIX = "Defect of the production proof: ";
1518
1619 /** The proof a week runs: the probes over fetch, at the forge's own origin and no other host. */
1720 export const weeklyProof: Proof = (base) => prove(base, fetch);
1821
19 /** The stable key a defect carries in its body; the title may be edited, this line is not. */
20 export const probeKey = (probeIssue: number): string => `Probe key: v1-proven/${probeIssue}`;
22 /** The stable key a run's defect carries in its body; the title may be edited, this line is not. */
23 const RUN_KEY = "Probe key: v1-proven/run";
24 /** The key a per-probe defect filed before #13265 carries. */
25 const legacyKey = (probeIssue: number): string => `Probe key: v1-proven/${probeIssue}`;
2126
22 /** The open defect issue for a probe: the newest open issue of the repository whose body
23 * carries its key, or, for a defect filed before the key existed, whose title is the one this
24 * duty gave it (`Defect of #N: `). */
25 function openDefect(db: Database, repoId: string, probeIssue: number): string | undefined {
26 const row = db
27 .query<{ id: string }, [string, string, string]>("SELECT id FROM issue WHERE repo_id = ?1 AND state = 'open' AND (instr(body, ?2) > 0 OR substr(title, 1, length(?3)) = ?3) ORDER BY created_at DESC, id DESC LIMIT 1")
28 .get(repoId, probeKey(probeIssue), `Defect of #${probeIssue}: `);
29 return row?.id;
27 const one = (db: Database, sql: string, ...args: string[]): string | undefined =>
28 db.query<{ id: string }, string[]>(`${sql} ORDER BY created_at DESC, id DESC LIMIT 1`).get(...args)?.id;
29
30 /** The run's open defect: the newest open issue whose body carries the run key. */
31 const runDefect = (db: Database, repoId: string): string | undefined =>
32 one(db, "SELECT id FROM issue WHERE repo_id = ?1 AND state = 'open' AND instr(body, ?2) > 0", repoId, RUN_KEY);
33
34 /** A per-probe defect filed before one issue per run: by its body key or the title it was given. */
35 function legacyDefect(db: Database, repoId: string, probeIssue: number): string | undefined {
36 return one(db, "SELECT id FROM issue WHERE repo_id = ?1 AND state = 'open' AND (instr(body, ?2) > 0 OR substr(title, 1, length(?3)) = ?3)", repoId, legacyKey(probeIssue), `Defect of #${probeIssue}: `);
3037 }
3138
39 /** A probe whose read got no HTTP answer (`prove` marks it status 0): the forge could not be
40 * reached, not a claim found false. */
41 const unreached = (r: Result): boolean => r.status === "fail" && r.seen !== undefined && r.seen.status === 0;
42
43 const originOf = (r: Result): string => {
44 try {
45 return new URL(r.seen?.url ?? "").origin;
46 } catch {
47 return "the forge";
48 }
49 };
50
3251 const passingNote = (day: string): string => `Passing again on ${day}. Close this defect when you are satisfied; the probe will not close it.`;
3352
34 /** Files, comments or notes passing on each result with an open defect; returns one sentence
35 * per thing done. */
53 /** Notes passing once on an open defect whose last word was a failure. */
54 function notePassing(db: Database, issueId: string, bot: string, day: string): boolean {
55 const last = listComments(db, issueId).at(-1);
56 if (last?.body.startsWith("Passing again on ")) return false; // said once
57 return addComment(db, { issueId, authorId: bot, body: passingNote(day) }).ok;
58 }
59
60 /** The failing lines of a run, each quoted with the evidence of what it read; every probe that
61 * reached nothing folds into one `cannot reach` line. */
62 function failingLines(results: readonly Result[], at: Date): { title: string; text: string; count: number } {
63 const fails = results.filter((r) => r.status === "fail");
64 const lost = fails.filter(unreached);
65 const parts: string[] = [];
66 const quote = (text: string, r: Result): string =>
67 `> ${text.replace(/\r?\n/g, " ")}\n\n${evidenceBlock({ ...(r.seen === undefined ? {} : { where: r.seen.url, status: r.seen.status, bytes: r.seen.bytes, ms: r.seen.ms }), at: at.toISOString() })}`;
68 const first = lost[0];
69 if (first !== undefined) parts.push(quote(`cannot reach ${originOf(first)}: ${lost.length} of ${results.length} probes got no answer (${first.detail})`, first));
70 for (const r of fails) if (!unreached(r)) parts.push(quote(line(r), r));
71 const answered = fails.length - lost.length;
72 const title =
73 first !== undefined && answered === 0
74 ? `${DEFECT_PREFIX}cannot reach ${originOf(first)}`
75 : `${DEFECT_PREFIX}${parts.length} ${parts.length === 1 ? "line fails" : "lines fail"}`;
76 return { title, text: parts.join("\n\n"), count: parts.length };
77 }
78
79 /** Files one issue for a run's failures, or comments on the run's defect; notes passing on the
80 * open defects whose probes pass again. Returns one sentence per thing done. */
3681 export function reportResults(db: Database, repoId: string, results: readonly Result[], at: Date): string[] {
3782 const said: string[] = [];
3883 const bot = watcherId(db);
3984 const day = at.toISOString().slice(0, 10);
85 const rest: Result[] = [];
4086 for (const r of results) {
41 if (r.status === "unproven") continue;
42 const open = openDefect(db, repoId, r.issue);
87 const legacy = r.status === "unproven" ? undefined : legacyDefect(db, repoId, r.issue);
88 if (legacy === undefined) {
89 rest.push(r);
90 continue;
91 }
4392 if (r.status === "pass") {
44 if (open === undefined) continue;
45 const last = listComments(db, open).at(-1);
46 if (last?.body.startsWith("Passing again on ")) continue; // said once
47 const c = addComment(db, { issueId: open, authorId: bot, body: passingNote(day) });
48 if (c.ok) said.push(`commented on the open defect of #${r.issue}: passing again`);
49 continue;
93 if (notePassing(db, legacy, bot, day)) said.push(`commented on the open defect of #${r.issue}: passing again`);
94 } else if (addComment(db, { issueId: legacy, authorId: bot, body: `Still failing on ${day}:\n\n${failingLines([r], at).text}` }).ok) {
95 said.push(`commented on the open defect of #${r.issue}: still failing`);
5096 }
51 const quote = `> ${line(r).replace(/\r?\n/g, " ")}\n\n${evidenceBlock({ ...(r.seen === undefined ? {} : { where: r.seen.url, status: r.seen.status, bytes: r.seen.bytes, ms: r.seen.ms }), at: at.toISOString() })}`;
52 if (open !== undefined) {
53 const c = addComment(db, { issueId: open, authorId: bot, body: `Still failing on ${day}:\n\n${quote}` });
54 if (c.ok) said.push(`commented on the open defect of #${r.issue}: still failing`);
55 continue;
56 }
57 const made = createIssue(db, {
58 repoId,
59 authorId: bot,
60 title: defectTitle(r),
61 body: `The weekly production probe (\`scripts/v1-proven.ts\`) failed on ${day}:\n\n${quote}\n\n${probeKey(r.issue)}\n\nFix the cause, then close this issue; a probe that fails again meanwhile is added here as a comment.`,
62 });
63 if (!made.ok) continue;
64 said.push(`opened issue #${made.value.number}: ${defectTitle(r)}`);
6597 }
98 const failed = failingLines(rest, at);
99 const run = runDefect(db, repoId);
100 if (failed.count === 0) {
101 if (run !== undefined && notePassing(db, run, bot, day)) said.push("commented on the open defect of the production proof: passing again");
102 return said;
103 }
104 if (run !== undefined) {
105 if (addComment(db, { issueId: run, authorId: bot, body: `Still failing on ${day}:\n\n${failed.text}` }).ok) said.push("commented on the defect of the production proof: still failing");
106 return said;
107 }
108 const made = createIssue(db, {
109 repoId,
110 authorId: bot,
111 title: failed.title,
112 body: `The weekly production probe (\`scripts/v1-proven.ts\`) failed on ${day}:\n\n${failed.text}\n\n${RUN_KEY}\n\nFix the cause, then close this issue; a run that fails again meanwhile is added here as a comment.`,
113 });
114 if (made.ok) said.push(`opened issue #${made.value.number}: ${failed.title}`);
66115 return said;
67116 }
68117
test/maintain-v1-proven.test.ts +54 −12
@@ -1,5 +1,6 @@
11 /**
2 * v1.0.0 stays proven (#12989): all pass, no issue; one fails, one issue; again, a comment.
2 * v1.0.0 stays proven (#12989): all pass, no issue; a run that fails, one issue listing every
3 * failing line (#13265); again, or again the same day, a comment.
34 */
45 import { afterAll, beforeAll, expect, test } from "bun:test";
56 import { mkdtemp, rm } from "node:fs/promises";
@@ -43,7 +44,7 @@ afterAll(async () => {
4344
4445 const run = (url = "https://forge.example/", forgeRepo = "boss/forge", proof = fake) =>
4546 v1Proven(proof).run({ db: app.db, bus: app.bus, now: NOW, app: { config: { url, forgeRepo } } as unknown as App });
46 const defects = () => app.db.query<{ title: string; id: string }, [string]>("SELECT id, title FROM issue WHERE repo_id = ?1 AND title LIKE 'Defect of #%'").all(repoId);
47 const defects = () => app.db.query<{ title: string; id: string }, [string]>("SELECT id, title FROM issue WHERE repo_id = ?1 AND (title LIKE 'Defect of %' OR instr(body, 'Probe key: v1-proven/') > 0)").all(repoId);
4748
4849 test("every probe passes: no issue, one line per probe on the record, the public origin read", async () => {
4950 const r = await run();
@@ -61,7 +62,7 @@ test("one fails: one issue quoting its line; the same fails again: a comment, no
6162 expect(first.findings).toHaveLength(1);
6263 const found = defects();
6364 expect(found).toHaveLength(1);
64 expect(found[0]!.title.startsWith(`Defect of #${bad.issue}`)).toBe(true);
65 expect(found[0]!.title).toBe("Defect of the production proof: 1 line fails");
6566 const body = app.db.query<{ body: string }, [string]>("SELECT body FROM issue WHERE id = ?1").get(found[0]!.id)!.body;
6667 expect(body).toContain(`fail #${bad.issue} ${bad.item}: / answers 500, not 200`);
6768
@@ -97,15 +98,18 @@ test("no forgeRepo: the failure is recorded and found, nothing is filed", async
9798 expect(r.actions.some((a) => a.includes("no issue filed"))).toBe(true);
9899 });
99100
100 test("two probes failing in one run open two issues", async () => {
101 app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND title LIKE 'Defect of #%'").run(repoId);
101 test("two probes failing in one run open one issue listing both lines", async () => {
102 app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND (title LIKE 'Defect of %' OR instr(body, 'Probe key: v1-proven/') > 0)").run(repoId);
102103 const before = defects().length;
103104 const a = probes[0]!.issue, b = probes[1]!.issue;
104105 const r = await run(undefined, undefined, mixed({ [a]: "fail", [b]: "fail" }));
105106 expect(r.findings).toHaveLength(2);
106107 const found = defects().slice(before);
107 expect(found).toHaveLength(2);
108 expect(found.map((d) => d.title.match(/#(\d+)/)![1]).sort()).toEqual([String(a), String(b)].sort());
108 expect(found).toHaveLength(1);
109 expect(found[0]!.title).toBe("Defect of the production proof: 2 lines fail");
110 const body = app.db.query<{ body: string }, [string]>("SELECT body FROM issue WHERE id = ?1").get(found[0]!.id)!.body;
111 expect(body).toContain(`fail #${a} `);
112 expect(body).toContain(`fail #${b} `);
109113 });
110114
111115 test("no public origin: nothing is called", async () => {
@@ -120,7 +124,7 @@ test("the duty is registered, weekly", () => {
120124 });
121125
122126 test("a passing probe notes it once on its open defect and leaves it open; the defect is found by its body key, not its title", async () => {
123 app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND title LIKE 'Defect of #%'").run(repoId);
127 app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND (title LIKE 'Defect of %' OR instr(body, 'Probe key: v1-proven/') > 0)").run(repoId);
124128 const bad = probes[2]!;
125129 await run(undefined, undefined, mixed({ [bad.issue]: "fail" }));
126130 const open = defects().at(-1)!;
@@ -131,8 +135,8 @@ test("a passing probe notes it once on its open defect and leaves it open; the d
131135 expect(total()).toBe(before);
132136 expect(listComments(app.db, open.id).map((c) => c.body.split(" ")[0])).toEqual(["Still"]);
133137
134 await run();
135 await run();
138 await run(undefined, undefined, mixed({}));
139 await run(undefined, undefined, mixed({}));
136140 const notes = listComments(app.db, open.id).filter((c) => c.body.startsWith("Passing again"));
137141 expect(notes).toHaveLength(1);
138142 const state = app.db.query<{ state: string }, [string]>("SELECT state FROM issue WHERE id = ?1").get(open.id)!;
@@ -140,7 +144,7 @@ test("a passing probe notes it once on its open defect and leaves it open; the d
140144 });
141145
142146 test("a filed defect and its still-failing comment carry the evidence block of what the probe read (#13033)", async () => {
143 app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND title LIKE 'Defect of #%'").run(repoId);
147 app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND (title LIKE 'Defect of %' OR instr(body, 'Probe key: v1-proven/') > 0)").run(repoId);
144148 const before = defects().length;
145149 const bad = probes[0]!;
146150 const seen = { url: "https://forge.example/", status: 500, bytes: 12, ms: 41.5 };
@@ -158,7 +162,7 @@ test("a filed defect and its still-failing comment carry the evidence block of w
158162 expect(readEvidence(comment.body)).toMatchObject({ url: seen.url, status: "500", bytes: "12" });
159163 });
160164
161 const closeDefects = () => app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND title LIKE 'Defect of #%'").run(repoId);
165 const closeDefects = () => app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND (title LIKE 'Defect of %' OR instr(body, 'Probe key: v1-proven/') > 0)").run(repoId);
162166 const total = () => app.db.query<{ c: number }, [string]>("SELECT count(*) AS c FROM issue WHERE repo_id = ?1").get(repoId)!.c;
163167
164168 test("a pass with no open defect comments nothing and files nothing", async () => {
@@ -193,3 +197,41 @@ test("a legacy open defect with no body key is found by its title: a failure com
193197 expect(total()).toBe(before);
194198 expect(listComments(app.db, legacy.value.id).map((c) => c.body.split(" ")[0])).toEqual(["Still"]);
195199 });
200
201 test("one run with several failures, run twice the same day: one issue, then one comment listing them all", async () => {
202 closeDefects();
203 const before = total();
204 const a = probes[0]!.issue, b = probes[1]!.issue, c = probes[2]!.issue;
205 const proof = mixed({ [a]: "fail", [b]: "fail", [c]: "fail" });
206 await run(undefined, undefined, proof);
207 await run(undefined, undefined, proof);
208 expect(total()).toBe(before + 1);
209 const made = defects().at(-1)!;
210 const comments = listComments(app.db, made.id);
211 expect(comments).toHaveLength(1);
212 for (const n of [a, b, c]) expect(comments[0]!.body).toContain(`fail #${n} `);
213 });
214
215 test("an unreachable forge files one 'cannot reach' line, not one per probe", async () => {
216 closeDefects();
217 const before = total();
218 const proof = async (): Promise<Result[]> =>
219 probes.map((p) => ({ issue: p.issue, item: p.item, status: "fail", detail: "probe error: TypeError: Unable to connect", seen: { url: "https://forge.example/healthz", status: 0, bytes: 0, ms: 0 } }));
220 const r = await run(undefined, undefined, proof);
221 expect(total()).toBe(before + 1);
222 const made = defects().at(-1)!;
223 expect(made.title).toBe("Defect of the production proof: cannot reach https://forge.example");
224 const body = app.db.query<{ body: string }, [string]>("SELECT body FROM issue WHERE id = ?1").get(made.id)!.body;
225 expect(body).toContain(`cannot reach https://forge.example: ${probes.length} of ${probes.length} probes got no answer`);
226 expect(body).not.toContain("fail #");
227 expect(r.actions.filter((x) => x.startsWith("opened issue"))).toHaveLength(1);
228 });
229
230 test("a run defect notes passing once when every probe passes again", async () => {
231 closeDefects();
232 await run(undefined, undefined, mixed({ [probes[0]!.issue]: "fail" }));
233 const made = defects().at(-1)!;
234 await run(undefined, undefined, mixed({}));
235 await run(undefined, undefined, mixed({}));
236 expect(listComments(app.db, made.id).map((c) => c.body.split(" ")[0])).toEqual(["Passing"]);
237 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs