trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

The full suite runs as a scheduled report that files what it finds and never blocks a landing #8247

Merged masoncrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/7780-the-full-suite-runs-as-a-scheduled-repor → main 7 commits fork 4 threads · 4 unresolved

  • Merged. mason/7780-the-full-suite-runs-as-a-scheduled-repor was merged into main as 4566ee3.

Review

Sign in to review this change.
11 files changed +367 −25
README.md +1 −1
@@ -410,7 +410,7 @@ work here.
410410
411411 ## What works today
412412
413 Every pull says which of four steps it is at — check, review, land, live — and why it waits there: a line under its title on both tabs (linked to `/pipeline`, a public page that explains the four steps), `step: { step, why, done }` in the MCP `pull` tool, and `step: …` in the terminal pull screen's header; the same word and sentence sit on the pull's post in the feed, its row on the pulls list and on `/review` (each linked to `/pipeline`), and in the terminal's feed, pulls list and on-you list (`Review — No review yet.` under the row). See [`docs/features/pipeline.md`](docs/features/pipeline.md).
413 Every pull says which of four steps it is at — check, review, land, live — and why it waits there: a line under its title on both tabs (linked to `/pipeline`, a public page that explains the four steps), `step: { step, why, done }` in the MCP `pull` tool, and `step: …` in the terminal pull screen's header; the same word and sentence sit on the pull's post in the feed, its row on the pulls list and on `/review` (each linked to `/pipeline`), and in the terminal's feed, pulls list and on-you list (`Review — No review yet.` under the row). The whole suite is a report, not a gate: after a landing it runs against the default branch and each failing test file gets one issue labelled `full-suite` (listed from `/pipeline`; the terminal's Flow dialog counts the open ones and the issue list filters by label with `l`). See [`docs/features/pipeline.md`](docs/features/pipeline.md).
414414
415415 A public app of a public repository tells an agent that lands on it what it can
416416 do: its HTML pages, and the "not running" page, carry two `Link` headers —
docs/features/pipeline.md +21 −2
@@ -45,8 +45,27 @@ The step is looked up per row (a few queries each); batching it is left for when
4545 the lists measure slow.
4646 `changeHead` computes "who acts next" once and hands it to `stepOfPull`.
4747
48 ## The full suite as a report
49
50 A verification after landing (`src/checks/verify.ts`) runs every shard against
51 the default branch's tip. When it is red, `src/checks/report.ts` files one
52 issue per failing test file through `fileIssueOnce` (`src/checks/flake.ts`),
53 titled `Full suite red: <file>` and labelled `full-suite`; a later red adds a
54 comment to the open issue instead of filing again. Nothing that decides a
55 pull, a batch or a release reads a verification, so the report holds none.
56
57 How it is scheduled: the checks service looks every minute (`VERIFY_SWEEP_MS`)
58 for a repository whose default branch has moved since its last verification
59 and verifies it, at most once per `VERIFY_EVERY_MS` (30 minutes, set in
60 `src/main.ts`) and never two at once. So the report runs after a landing, not
61 on a daily clock: a quiet main is not re-run, a busy one at most every half hour.
62
63 `/pipeline` links the label (in the forge's own repository, `ownRepoPath()`)
64 in the card under "Where to look next". The terminal's Flow dialog ends with
65 a line counting the open report issues; the issue list there filters by
66 label with `l`.
67
4868 ## Not here yet
4969
50 The rest of #7703 is filed as its own issues: remove the proof (#7779), run
51 the full suite as a scheduled report (#7780), re-run a shard that crashed
70 The rest of #7703 is filed as its own issues: remove the proof (#7779), re-run a shard that crashed
5271 outside any test (#7781), reuse a verdict for the same tree (#7782), and measure each step (#7784).
src/checks/flake.ts +52 −20
@@ -10,7 +10,7 @@
1010 */
1111
1212 import type { Database } from "bun:sqlite";
13 import { createIssue } from "../core/issues.ts";
13 import { createIssue, setIssueLabels } from "../core/issues.ts";
1414 import type { Repo } from "../core/repos.ts";
1515 import type { EventBus } from "../core/events.ts";
1616 import { id } from "../kernel/id.ts";
@@ -241,35 +241,67 @@ export function recordFlakes(
241241 .get(repo.id, t.name, since)!.n;
242242 // No issue without a file: the "Files:" line is what routes it to a builder.
243243 if (count < FILE_AT || t.file === "") continue;
244 const title = flakeTitle(t.name);
245 const open = db
246 .query<{ id: string }, [string, string]>(
247 "SELECT id FROM issue WHERE repo_id = ?1 AND title = ?2 AND state = 'open' LIMIT 1",
248 )
249 .get(repo.id, title);
250 if (open !== null) continue;
251 const made = createIssue(db, {
252 repoId: repo.id,
253 authorId: repo.ownerId,
254 title,
244 const filed = fileIssueOnce(db, bus, repo, {
245 title: flakeTitle(t.name),
255246 body:
256247 `_Filed by the check runner from a check log, not written by a person._\n\n` +
257248 `\`${t.name}\` in \`${t.file}\` failed in ${where} and passed when run alone, ${count} times in 7 days ` +
258249 `(latest: check run ${ref.runId}, ${ref.commit.slice(0, 7)}). The check stayed green each time. ` +
259250 `Find what it depends on: time, load, order, or shared state.\n\nFiles: ${t.file}`,
260251 });
261 if (!made.ok) continue;
262 bus.append({
263 kind: "issue.opened",
264 repoId: repo.id,
265 actorId: repo.ownerId,
266 payload: { number: made.value.number },
267 });
268 opened.push(made.value.number);
252 if (filed.opened) opened.push(filed.number);
269253 }
270254 return opened;
271255 }
272256
257 /**
258 * One open issue per title: the open one when there is one (`opened` false),
259 * else a new one, announced on the bus. Number 0 when it could not be made.
260 */
261 export function fileIssueOnce(
262 db: Database,
263 bus: EventBus,
264 repo: Pick<Repo, "id" | "ownerId">,
265 issue: { title: string; body: string; labelId?: string },
266 ): { number: number; opened: boolean } {
267 const open = db
268 .query<{ number: number }, [string, string]>(
269 "SELECT number FROM issue WHERE repo_id = ?1 AND title = ?2 AND state = 'open' LIMIT 1",
270 )
271 .get(repo.id, issue.title);
272 if (open !== null) return { number: open.number, opened: false };
273 const made = createIssue(db, { repoId: repo.id, authorId: repo.ownerId, title: issue.title, body: issue.body });
274 if (!made.ok) return { number: 0, opened: false };
275 if (issue.labelId !== undefined) setIssueLabels(db, made.value.id, [issue.labelId]);
276 bus.append({
277 kind: "issue.opened",
278 repoId: repo.id,
279 actorId: repo.ownerId,
280 payload: { number: made.value.number },
281 });
282 return { number: made.value.number, opened: true };
283 }
284
285 /** Every failing test of a `bun test` log with the file whose report printed it, once each, in order. */
286 export function failedTests(lines: readonly string[]): Flaky[] {
287 const out: Flaky[] = [];
288 const seen = new Set<string>();
289 let file: string | undefined;
290 for (const line of lines) {
291 const header = HEADER.exec(line);
292 if (header) {
293 file = header[1]!.replace(/^\.\//, "");
294 continue;
295 }
296 const name = FAIL.exec(line)?.[1];
297 if (name === undefined || file === undefined || !PATH.test(file)) continue;
298 if (name.length > NAME_MAX || CONTROL.test(name) || seen.has(`${file}\0${name}`)) continue;
299 seen.add(`${file}\0${name}`);
300 out.push({ file, name });
301 }
302 return out;
303 }
304
273305 /** What a red queue verdict says of its tests: each failing one with its file, and the files it learned. */
274306 export interface QueueRed {
275307 readonly tests?: { readonly failing: readonly { readonly name: string; readonly file?: string }[] };
src/checks/report.ts added +96
@@ -0,0 +1,96 @@
1 /**
2 * The full suite as a scheduled report (#7780). A verification after landing
3 * (`checks/verify.ts`) runs every shard against main's tip; when it is red,
4 * each failing test file gets one issue, labelled {@link REPORT_LABEL} so it
5 * is found: the open one is added to when a later run fails the same file,
6 * and none is made twice. The report only files. Nothing reads a verification
7 * when it decides whether a pull, a batch or a release may go ahead (those
8 * look at the default branch's own check), so a red report holds none of them.
9 */
10
11 import type { App } from "../app.ts";
12 import type { Database } from "bun:sqlite";
13 import { addComment, createLabel, getLabel, listIssues, type Label } from "../core/issues.ts";
14 import { getRepo, type Repo } from "../core/repos.ts";
15 import { ownRepoPath } from "../kernel/own_repo.ts";
16 import type { CheckDetail, Run } from "../core/runs.ts";
17 import { failedTests, fileIssueOnce } from "./flake.ts";
18
19 /** The label every report issue carries. */
20 export const REPORT_LABEL = "full-suite";
21 const REPORT_COLOR = "d93f0b";
22 /** The most failing tests a run carries for the report. */
23 export const MAX_FAILURES = 40;
24 /** The most tests one issue names. */
25 const NAMED = 10;
26
27 export const reportTitle = (file: string): string => `Full suite red: ${file}`;
28
29 /** A red verification's detail with the failing tests under its failing files (at most {@link MAX_FAILURES}). */
30 export function redDetail(
31 detail: CheckDetail | undefined,
32 files: readonly string[],
33 lines: readonly string[],
34 ): CheckDetail | undefined {
35 if (files.length === 0) return detail;
36 return {
37 ...(detail ?? { passed: 0, failed: files.length, names: [] }),
38 files,
39 failures: failedTests(lines).filter((t) => files.includes(t.file)).slice(0, MAX_FAILURES),
40 };
41 }
42
43 function labelOf(app: App, repo: Repo): Label | undefined {
44 const made = createLabel(app.db, repo.id, REPORT_LABEL, REPORT_COLOR);
45 return made.ok ? made.value : getLabel(app.db, repo.id, REPORT_LABEL);
46 }
47
48 /** The lines for a red verification's log: one per failing file, filed or added to. Empty when green. */
49 export function reportRed(app: App, repo: Repo, run: Run): string[] {
50 if (run.state !== "failed") return [];
sentinelcrew outdated

'and failed again when run alone' is not established by anything here. State only what the run recorded.
51 const files = run.checkDetail?.files ?? [];
52 const failures = run.checkDetail?.failures ?? [];
53 const label = files.length > 0 ? labelOf(app, repo) : undefined;
54 const lines: string[] = [];
55 for (const file of files) {
56 const names = failures.filter((f) => f.file === file).map((f) => f.name);
57 const said = names.length === 0 ? "" : `\n\n${names.slice(0, NAMED).map((n) => `- \`${n}\``).join("\n")}`;
58 const more = names.length > NAMED ? `\n- and ${names.length - NAMED} more` : "";
59 const where = `check run ${run.id}, ${run.commitOid.slice(0, 7)}`;
60 const title = reportTitle(file);
61 const filed = fileIssueOnce(app.db, app.bus, repo, {
62 title,
63 body:
64 `_Filed by the scheduled full-suite report, not written by a person. It holds no pull, batch or release._\n\n` +
65 `\`${file}\` failed in the full suite on ${repo.defaultBranch} (${where}).` +
66 `${said}${more}\n\nFiles: ${file}`,
67 ...(label === undefined ? {} : { labelId: label.id }),
68 });
69 if (filed.number === 0) lines.push(`report: ${file} not filed`);
70 else if (filed.opened) lines.push(`report: filed #${filed.number} for ${file}`);
71 else {
72 const open = app.db
73 .query<{ id: string }, [string, number]>("SELECT id FROM issue WHERE repo_id = ?1 AND number = ?2")
74 .get(repo.id, filed.number);
75 if (open !== null)
76 addComment(app.db, {
77 issueId: open.id,
78 authorId: repo.ownerId,
79 body: `Still red in the full suite (${where}).${said}${more}`,
80 });
81 lines.push(`report: added to #${filed.number} for ${file}`);
82 }
83 }
84 return lines;
85 }
86
87 /** Where the report's issues are listed: the forge's own repository, filtered by the label. */
88 export const reportPath = (): string => `/${ownRepoPath()}/issues?label=${REPORT_LABEL}`;
89
90 /** One line for the terminal: how many report issues are open and where they are listed. */
91 export function reportLine(db: Database): string {
92 const [owner = "", name = ""] = ownRepoPath().split("/");
93 const repo = getRepo(db, owner, name);
94 const open = repo === undefined ? 0 : listIssues(db, repo.id, { label: REPORT_LABEL }).length;
95 return `Full suite report: ${open} open issue${open === 1 ? "" : "s"} labelled ${REPORT_LABEL} in ${owner}/${name} (the terminal's issue list, key l).`;
96 }
src/checks/service.ts +4 −1
@@ -64,6 +64,7 @@ import { proposeRevert } from "./revert_red.ts";
6464 import { rebasesFirst } from "./finishing_first.ts";
6565 import { IMPACT_FILE, keepImpact, seedImpact } from "./impact-store.ts";
6666 import { afterVerify } from "./culprit.ts";
67 import { redDetail, reportRed } from "./report.ts";
6768 import {
6869 bisectOrigin,
6970 failedFiles,
@@ -858,6 +859,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
858859 log.error("revert proposal failed", { error: String(error) }),
859860 );
860861 // A red verification after landing names its culprit (#7786).
862 if (done && run.env === VERIFY_ENV)
863 reportRed(app, repo, done).forEach(write);
861864 if (done && inVerifyLane(run.env))
862865 await afterVerify(app, repo, done, (r, env, oid, reason) => request(r, env, oid, reason))
863866 .then((lines) => lines.forEach(write))
@@ -1396,7 +1399,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13961399 return {
13971400 ok: false,
13981401 error: killed ? ran.error : summary,
1399 detail: files.length > 0 ? { ...(detail ?? { passed: 0, failed: files.length, names: [] }), files } : detail,
1402 detail: redDetail(detail, files, checkLines),
14001403 stage: checkStage(ran.error),
14011404 ...(killed ? { killed } : {}),
14021405 };
src/core/runs.ts +2
@@ -47,6 +47,8 @@ export interface CheckDetail {
4747 readonly names: readonly string[];
4848 /** The test files the failures printed under, when a verification after landing failed (#7786). */
4949 readonly files?: readonly string[];
50 /** The failing tests under those files, at most forty, for the report that files them (#7780). */
51 readonly failures?: readonly { readonly file: string; readonly name: string }[];
5052 /** The check machine's free disk share (0..1) when a red finished (#5453). */
5153 readonly diskFree?: number;
5254 }
src/tui/screens/flow.ts +6 −1
@@ -13,12 +13,15 @@
1313 * changes requested waiting on a builder and on a reviewer (`flow/requested.ts`, #2172); then each
1414 * autonomous repository's ready list, startable beside the count per
1515 * filter that holds the rest (`flow/held.ts`, #2244), as the flow panel draws it;
16 * last, how long each main has gone unverified after landing (`flow/verified.ts`, #7786).
16 * then how long each main has gone unverified after landing (`flow/verified.ts`, #7786);
17 * then one line for the full-suite report's open issues (`checks/report.ts`, #7780);
18 * last, the duplicate lines (`duplicateLines`).
1719 * Read-only; Enter, Esc or q closes it.
1820 */
1921
2022 import type { Database } from "bun:sqlite";
2123 import { recentGates, type KeptResult } from "../../checks/gate_result.ts";
24 import { reportLine } from "../../checks/report.ts";
2225 import { HOLDS } from "../../crew/held.ts";
2326 import { pullCounts } from "../../crew/hot_files.ts";
2427 import { warmingLine } from "../../crew/hot_history.ts";
@@ -139,6 +142,8 @@ export function flowLines(
139142 ? []
140143 : ["", ...verified.flatMap((r) => wrap(`${r.repo}: ${verifiedSays(r, now.getTime())}`, inner))]),
141144 "",
145 ...wrap(reportLine(db), inner),
146 "",
142147 ...duplicateLines(dups, inner),
143148 ];
144149 }
src/web/pages/pipeline.ts +6
@@ -12,6 +12,7 @@ import { cachedStepStats, statParts, STEP_WINDOW_DAYS, type StepStat } from "../
1212 import { notePhase, takeSessionTime } from "../../core/route_timing.ts";
1313 import { html } from "../html.ts";
1414 import { page } from "../layout.ts";
15 import { reportPath } from "../../checks/report.ts";
1516 import { card, row, stat } from "../ui.ts";
1617
1718 /** What a pull waits on at each step, what only that step catches, and the parts inside it. */
@@ -110,6 +111,11 @@ export function pipelinePage(ctx: Ctx): Response {
110111 and the rest of the path runs lint and types. An admin sees how many changes sit at each step right now on
111112 <a href="/boards">Boards</a>.
112113 </p>
114 <p class="small" id="full-suite">
115 The whole suite still runs, as a report: after a landing, on a schedule, against the default branch's tip.
116 Each test file it finds failing gets one issue labelled <span class="mono">full-suite</span>, listed
117 <a href="${reportPath()}">here</a>. The report holds no pull, batch or release.
118 </p>
113119 <p class="muted small">For agents: the MCP tool <span class="mono">pull</span> answers <span class="mono">step: { step, why, done }</span> with these four words.</p>
114120 </div>`,
115121 { head: html`<h2>Where to look next</h2>` },
test/check-verify.test.ts +8
@@ -98,6 +98,14 @@ test("a red verification that fails alone too names the push that broke it in on
9898 const run = await until(() => finished(VERIFY_ENV, oid));
9999 expect(run.state).toBe("failed");
100100 expect(getRun(app.db, run.id)?.checkDetail?.files).toEqual(["t/a.test.ts"]);
101 // The service filled the failures and filed the report for the failing file.
102 expect(getRun(app.db, run.id)?.checkDetail?.failures).toEqual([{ file: "t/a.test.ts", name: "grp > works" }]);
103 const reported = await until(() =>
104 app.db
105 .query<{ body: string }, [string]>("SELECT body FROM issue WHERE repo_id = ?1 AND title = 'Full suite red: t/a.test.ts'")
106 .get(repo.id) ?? undefined,
107 );
108 expect(reported.body).toContain("grp > works");
101109 const issue = await until(() =>
102110 app.db
103111 .query<{ title: string; body: string }, [string]>("SELECT title, body FROM issue WHERE repo_id = ?1 AND kind = 'issue' AND title LIKE 'Red after landing:%'")
test/full-suite-report.test.ts added +159
@@ -0,0 +1,159 @@
1 /**
2 * The full suite as a scheduled report (#7780): a red verification files one
3 * labelled issue per failing file, a rerun adds to it instead of filing
4 * again, and the red holds nothing.
5 */
6
7 import { afterEach, beforeEach, describe, expect, test } from "bun:test";
8 import { mkdtemp, rm } from "node:fs/promises";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { createApp, type App } from "../src/app.ts";
12 import { createRepo } from "../src/actions/repos.ts";
13 import { flowLines } from "../src/tui/screens/flow.ts";
14 import { MAX_FAILURES, redDetail, reportRed, reportTitle, REPORT_LABEL } from "../src/checks/report.ts";
15 import { failedTests } from "../src/checks/flake.ts";
16 import { heldByRed, redMain } from "../src/checks/red_main.ts";
17 import { VERIFY_ENV } from "../src/checks/verify.ts";
18 import { listIssues, listComments, labelsFor } from "../src/core/issues.ts";
19 import type { Repo } from "../src/core/repos.ts";
20 import { createRun, getRun, type Run } from "../src/core/runs.ts";
21 import { createUser } from "../src/core/users.ts";
22
23 describe("failedTests", () => {
24 test("pairs each failure with the file that printed it, once", () => {
25 const lines = [
26 "test/a.test.ts:",
27 "(pass) ok",
28 "(fail) a > one [3.00ms]",
29 "(fail) a > one [3.00ms]",
30 "test/b.test.ts:",
31 "(fail) b > two",
32 ];
33 expect(failedTests(lines)).toEqual([
34 { file: "test/a.test.ts", name: "a > one" },
35 { file: "test/b.test.ts", name: "b > two" },
36 ]);
37 });
38 });
39
40 describe("the report", () => {
41 let dataDir: string;
42 let app: App;
43 let repo: Repo;
44
45 beforeEach(async () => {
46 dataDir = await mkdtemp(join(tmpdir(), "forge-report-"));
47 app = createApp({ dataDir, port: 0 });
48 const created = await createUser(app.db, {
49 origin: "seed",
50 trusted: true,
51 username: "alice",
52 email: "[email protected]",
53 password: "a-good-passphrase",
54 adminIfFirst: true,
55 });
56 if (!created.ok) throw new Error(created.error);
57 const made = await createRepo(app, created.value, { name: "app", visibility: "public" });
58 if (!made.ok) throw new Error(String(made.error));
59 repo = made.value;
60 });
61
62 afterEach(async () => {
63 app.close();
64 await rm(dataDir, { recursive: true, force: true });
65 });
66
67 const red = (env: string, commit: string): Run => {
68 const run = createRun(app.db, { repoId: repo.id, kind: "check", env, commitOid: commit, reason: "schedule" });
69 const detail = {
70 passed: 10,
71 failed: 3,
72 names: ["a > one", "a > two", "b > three"],
73 files: ["test/a.test.ts", "test/b.test.ts"],
74 failures: [
75 { file: "test/a.test.ts", name: "a > one" },
76 { file: "test/a.test.ts", name: "a > two" },
77 { file: "test/b.test.ts", name: "b > three" },
78 ],
79 };
80 app.db
81 .query("UPDATE run SET state = 'failed', check_detail = ?2, finished_at = ?3 WHERE id = ?1")
82 .run(run.id, JSON.stringify(detail), new Date().toISOString());
83 return getRun(app.db, run.id)!;
84 };
85
86 const reports = () => listIssues(app.db, repo.id, { label: REPORT_LABEL, state: "all" });
87
88 test("one labelled issue per failing file, naming its failing tests", () => {
89 const lines = reportRed(app, repo, red(VERIFY_ENV, "a".repeat(40)));
90 expect(lines).toHaveLength(2);
91 const issues = reports();
92 expect(issues.map((i) => i.title).sort()).toEqual([
93 reportTitle("test/a.test.ts"),
94 reportTitle("test/b.test.ts"),
95 ]);
96 const a = issues.find((i) => i.title === reportTitle("test/a.test.ts"))!;
97 expect(a.body).toContain("a > one");
98 expect(a.body).toContain("a > two");
99 expect(a.body).not.toContain("b > three");
100 expect(a.body).toContain("Files: test/a.test.ts");
101 expect(labelsFor(app.db, a.id).map((l) => l.name)).toEqual([REPORT_LABEL]);
102 });
103
104 test("a rerun files nothing new: it adds to the open issues", () => {
105 reportRed(app, repo, red(VERIFY_ENV, "a".repeat(40)));
106 const again = reportRed(app, repo, red(VERIFY_ENV, "b".repeat(40)));
107 expect(again.every((l) => l.startsWith("report: added to #"))).toBe(true);
108 expect(reports()).toHaveLength(2);
109 const a = reports().find((i) => i.title === reportTitle("test/a.test.ts"))!;
110 expect(listComments(app.db, a.id)).toHaveLength(1);
111 });
112
113 test("a closed issue is filed anew when the file fails again", () => {
114 reportRed(app, repo, red(VERIFY_ENV, "a".repeat(40)));
115 app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1").run(repo.id);
116 reportRed(app, repo, red(VERIFY_ENV, "b".repeat(40)));
117 expect(reports()).toHaveLength(4);
118 });
119
120 test("a green verification files nothing", () => {
121 const run = createRun(app.db, { repoId: repo.id, kind: "check", env: VERIFY_ENV, commitOid: "c".repeat(40) });
122 app.db.query("UPDATE run SET state = 'ok' WHERE id = ?1").run(run.id);
123 expect(reportRed(app, repo, getRun(app.db, run.id)!)).toEqual([]);
124 expect(reports()).toHaveLength(0);
125 });
126
127 test("a red report holds nothing: main is not red, and a pull's check is not held", () => {
128 reportRed(app, repo, red(VERIFY_ENV, "a".repeat(40)));
129 const main = redMain(app.db, repo.id);
130 expect(main).toBeUndefined();
131 const check = { kind: "check" as const, issueId: "x", reason: "", createdAt: new Date().toISOString() };
132 expect(heldByRed(check, main, 20 * 60_000, Date.now())).toBe(false);
133 });
134
sentinelcrew outdated

Tautology: it asserts a constant equals itself. Test the cap on real input.
135 test("a run carries at most forty failures, only under its failing files", () => {
136 const lines = ["test/a.test.ts:", ...Array.from({ length: 60 }, (_, i) => `(fail) a > t${i}`), "test/c.test.ts:", "(fail) c > x"];
137 const detail = redDetail(undefined, ["test/a.test.ts"], lines)!;
138 expect(detail.failures).toHaveLength(MAX_FAILURES);
139 expect(detail.failures!.every((f) => f.file === "test/a.test.ts")).toBe(true);
140 expect(detail.files).toEqual(["test/a.test.ts"]);
141 expect(redDetail(undefined, [], lines)).toBeUndefined();
142 });
143
144 test("the issue says only what the run recorded", () => {
145 reportRed(app, repo, red(VERIFY_ENV, "a".repeat(40)));
146 expect(reports()[0]!.body).not.toContain("run alone");
147 });
148
149 test("the terminal's Flow dialog counts the open report issues", () => {
150 process.env.FORGE_SENTINEL_REPO = "alice/app";
151 try {
152 expect(flowLines(app.db).join("\n")).toContain("Full suite report: 0 open issues");
153 reportRed(app, repo, red(VERIFY_ENV, "a".repeat(40)));
154 expect(flowLines(app.db).join("\n")).toContain("Full suite report: 2 open issues labelled full-suite");
155 } finally {
156 delete process.env.FORGE_SENTINEL_REPO;
157 }
158 });
159 });
test/pipeline-path.test.ts +12
@@ -206,3 +206,15 @@ test("a merged pull into an app is landed until production has served a deploy s
206206 expect(stepOfPull(app.db, repo, pull, live)).toEqual({ step: "live", why: "Landed and live in production.", done: true });
207207 expect(await read(`/alice/shipped/pulls/${opened.value.number}`)).toContain('data-path="live"');
208208 });
209
210 test("/pipeline links the full-suite report's issues, and that link answers", async () => {
211 process.env["FORGE_SENTINEL_REPO"] = "alice/open";
212 try {
213 const body = await (await fetch(`${servers.url}/pipeline`)).text();
214 expect(body).toContain('id="full-suite"');
215 expect(body).toContain('href="/alice/open/issues?label=full-suite"');
216 expect((await fetch(`${servers.url}/alice/open/issues?label=full-suite`)).status).toBe(200);
217 } finally {
218 delete process.env["FORGE_SENTINEL_REPO"];
219 }
220 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs