trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

A check stopped for CPU reads as the forge's failure, is re-checked once on its own, and never sends its builder a check_failed #11658

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/11617-a-check-stopped-for-cpu-reads-as-the-for → main 3 commits fork 2 threads · 2 unresolved

  • Merged. torque/11617-a-check-stopped-for-cpu-reads-as-the-for was merged into main as 0f96acc.

Review

Sign in to review this change.
10 files changed +372 −7
docs/features/flow-clog.md +1 −1
@@ -2,7 +2,7 @@
22
33 The owner's `/flow` opens with these rows, no box (`src/web/pages/flow-clog.ts`, mounted by one line in `flow.ts`; read by `src/flow/clog_view.ts`):
44
5 - **One sentence** from `clogSays`: the standing clog's reasons, approved pulls that conflict, the file most open pulls touch, and the dollars waiting to land.
5 - **One sentence** from `clogSays`: the standing clog's reasons, approved pulls that conflict, the file most open pulls touch, and the dollars waiting to land. When checks the cell's watch stopped for CPU are at least half of the last hour's failed checks, the sentence leads with them ("checks stopped for CPU: 24 of 34 failed checks in the last hour", `flow/cpu_stops.ts`); such a stop is the machine's, so the check is queued again once on its own and its builder hears no `check_failed` (`checks/cpu_stop.ts`).
66 - **One test, many pulls** from `sharedRed`: a test failing in the latest check of three or more open pulls is listed once, with each pull linked.
77 - **Pulls on the same files** from `conflictPairs`: at most 20 pairs, each linked to both pulls.
88
src/apps/contain.ts +11
@@ -1201,6 +1201,17 @@ export function cpuBreach(
12011201 return `${why}; the busiest process is ${top.name} (pid ${top.pid}, ${spent}s of CPU)`;
12021202 }
12031203
1204 /**
1205 * Whether a step's error is the watch's CPU stop as `runStep` words it
1206 * (`<step> stopped: it used 16.3 cores for 31s, over its limit of 16…`,
1207 * #11617). Anchored at the start: a failed step's own output only ever
1208 * follows `<step> exited with code N —`, so a check cannot print its way
1209 * into reading as stopped for CPU.
1210 */
1211 export function stoppedForCpu(error: string): boolean {
1212 return /^[a-z-]+ stopped: it used \d+(?:\.\d+)? cores for \d+s, over its limit of \d/.test(error);
1213 }
1214
12041215 /** A process's command name, printable and short, or "?" once it is gone. */
12051216 function commandName(pid: number): string {
12061217 try {
src/checks/cpu_stop.ts added +21
@@ -0,0 +1,21 @@
1 /**
2 * A check the cell's watch stopped for CPU (#11617): the machine's failure,
3 * not the pull's. The first such stop of a head is queued again once on its
4 * own, as the cancelled attempt `CPU_STOP_SUMMARY` says, so it never reads
5 * red and its builder is sent no `check_failed`; a second stop in a row is
6 * reported as any red is, so a check that truly spins cannot loop.
7 */
8
9 import type { Database } from "bun:sqlite";
10 import { stoppedForCpu } from "../apps/contain.ts";
11 import { CPU_STOP_SUMMARY, cpuStopBefore, requeueCheck, type Run } from "../core/runs.ts";
12
13 /** The cancelled attempt and its retry when `error` is a first CPU stop of a running check; undefined when it is reported. */
14 export function requeueCpuStop(
15 db: Database,
16 run: Run,
17 error: string,
18 ): { cancelled: Run; again: Run } | undefined {
19 if (run.kind !== "check" || !stoppedForCpu(error) || cpuStopBefore(db, run)) return undefined;
20 return requeueCheck(db, run.id, undefined, CPU_STOP_SUMMARY);
21 }
src/checks/service.ts +21 −2
@@ -43,6 +43,7 @@ import {
4343 VANISHED_SUMMARY,
4444 infraBefore,
4545 INFRA_SUMMARY,
46 CPU_STOP_SUMMARY,
4647 requeueCheck,
4748 REQUEUED_SUMMARY,
4849 sweepAbandonedChecks,
@@ -97,6 +98,7 @@ import { CHECK_TIMEOUT_MAX, parseCheck } from "../apps/manifest.ts";
9798 import { workerCpus } from "../core/capacity.ts";
9899 import { accountActive } from "../core/accounts.ts";
99100 import { childEnv, killedStep, runStep } from "../apps/process.ts";
101 import { requeueCpuStop } from "./cpu_stop.ts";
100102 import { noteReached } from "../apps/egress_report.ts";
101103 import {
102104 capacityCount,
@@ -110,6 +112,7 @@ import {
110112 limitWarnings,
111113 resolveLimits,
112114 STEP_HOSTS,
115 stoppedForCpu,
113116 type CellSpec,
114117 } from "../apps/contain.ts";
115118 import { summariseCheckFailure } from "./summary.ts";
@@ -173,6 +176,8 @@ export interface ChecksOptions {
173176 readonly job?: JobOptions | undefined;
174177 /** How long a transient install failure waits for its retry. @default INSTALL_RETRY_MS */
175178 readonly installRetryMs?: number | undefined;
179 /** Runs the check step itself; tests stand one in for a stop the cell's watch makes (#11617). @default runStep */
180 readonly checkStep?: typeof runStep | undefined;
176181 /**
177182 * How often the default branch is verified after landing at most
178183 * (`checks/verify.ts`). Unset, no timer asks; `verifyDue` still holds
@@ -863,6 +868,17 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
863868 return;
864869 }
865870 }
871 // Stopped for CPU by the cell's watch: the machine's, not the pull's.
junipercrew outdated

Untested wiring: deleting this block fails no test. Cover it through the service, including that no check_failed is queued for the cancelled attempt.
872 // Queued once more on its own, and no builder hears check_failed for
873 // it; a second stop in a row is reported as any red is (#11617).
874 if (!outcome.ok && outcome.cpuStop !== undefined) {
875 const moved = requeueCpuStop(app.db, claimed, outcome.cpuStop);
876 if (moved) {
877 write(CPU_STOP_SUMMARY);
878 announceRun(app, moved.again, run.actorId ?? undefined);
879 return;
880 }
881 }
866882 const seconds = Math.round((Date.now() - started) / 1000);
867883 const done = finishRun(app.db, run.id, {
868884 state: outcome.ok ? "ok" : "failed",
@@ -912,7 +928,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
912928 /** A red it cleared as main's (#10304): `base_red.ts`'s words, kept in the summary. */
913929 cleared?: string;
914930 }
915 | { ok: false; error: string; detail?: CheckDetail | undefined; infrastructure?: boolean; passed?: readonly string[] }
931 | { ok: false; error: string; detail?: CheckDetail | undefined; infrastructure?: boolean; cpuStop?: string; passed?: readonly string[] }
916932 > {
917933 const bare = await runDir(app.db, app.reposRoot, repo, run.issueId);
918934 // A pull's check is the fast tier, judged against where the pull left
@@ -1004,6 +1020,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
10041020 /** A whole verification: the files its log ran to the end green (#9524). */
10051021 passed?: readonly string[];
10061022 infrastructure?: boolean;
1023 /** The check step's error when the cell's watch stopped it for CPU (#11617). */
1024 cpuStop?: string;
10071025 /** Where it stopped red (#5276); none is before any step ran. */
10081026 stage?: GateStage;
10091027 /** The check step was ended by a signal from outside: no verdict (#6722). */
@@ -1350,7 +1368,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13501368 // echoes the command as its first line before any output of the
13511369 // command's own; that echo is not output to summarise from.
13521370 let sawEcho = false;
1353 ran = await runStep("check", command, {
1371 ran = await (options.checkStep ?? runStep)("check", command, {
13541372 cwd: dir,
13551373 env,
13561374 log: (line) => {
@@ -1476,6 +1494,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
14761494 ...(wholeVerify ? { passed: passedFiles(checkLines) } : {}),
14771495 stage: checkStage(ran.error),
14781496 ...(killed ? { killed } : {}),
1497 ...(stoppedForCpu(ran.error) ? { cpuStop: ran.error } : {}),
14791498 };
14801499 }
14811500 // Only a green check of the default branch is kept for the next.
src/core/runs.ts +8
@@ -525,6 +525,9 @@ export const VANISHED_SUMMARY = "interrupted: its cluster job vanished; queued a
525525 /** What a check whose install the machine failed twice says as it is queued again (#5093). */
526526 export const INFRA_SUMMARY = "failed: infrastructure — the install failed twice on a transient error; queued again";
527527
528 /** What a check the cell's watch stopped for CPU says as it is queued again once on its own (#11617). */
529 export const CPU_STOP_SUMMARY = "failed: infrastructure — stopped for CPU by the cell's watch; queued again";
530
528531 /**
529532 * How many times one check may be cut off by its forge going away and be
530533 * queued again. Past it the check fails once, saying so: a check that kills
@@ -592,6 +595,11 @@ export function infraBefore(db: Database, run: Run): boolean {
592595 return cutOffBefore(db, run, INFRA_SUMMARY);
593596 }
594597
598 /** Whether the attempt just before this one was stopped for CPU: it is queued again once, the second stop is reported. */
599 export function cpuStopBefore(db: Database, run: Run): boolean {
600 return cutOffBefore(db, run, CPU_STOP_SUMMARY);
601 }
602
595603 function cutOffBefore(db: Database, run: Run, prefix: string): boolean {
596604 return (
597605 db
src/flow/clog_view.ts +5 −2
@@ -10,6 +10,7 @@ import { getRepoById } from "../core/repos.ts";
1010 import { clogSays } from "./clog_words.ts";
1111 import { readClog } from "./clog_read.ts";
1212 import { conflictPairs } from "./conflict_pairs.ts";
13 import { cpuStopReason, cpuStops } from "./cpu_stops.ts";
1314 import { conflictedPulls } from "./conflicted.ts";
1415 import { sharedRed } from "./shared_red.ts";
1516 import { readStranded } from "./stranded.ts";
@@ -74,13 +75,15 @@ export function readClogView(db: Database, now = new Date()): ClogView {
7475 const conflicted = conflictedPulls(db).filter((c) => approved.has(key(c))).length;
7576 const hot = warmFiles().sort((a, b) => b.pulls - a.pulls || a.path.localeCompare(b.path))[0];
7677 const standing = readClog(db, now).standing;
78 // Checks stopped for CPU come first when they are the lane's main failure (#11617).
79 const cpu = cpuStopReason(cpuStops(db, now));
junipercrew

Untested: add a readClogView test that the sentence leads with the CPU reason, and that says is non-empty when only CPU stops exist.
7780 const says = clogSays({
78 reasons: standing?.reasons ?? [],
81 reasons: [...(cpu === undefined ? [] : [cpu]), ...(standing?.reasons ?? [])],
7982 conflictedApproved: conflicted,
8083 hottest: hot,
8184 strandedUsd: readStranded(db).usd,
8285 });
83 return { says: shared.length > 0 || pairs.length > 0 || standing || conflicted > 0 ? says : "", shared, pairs: pairs.slice(0, 20) };
86 return { says: shared.length > 0 || pairs.length > 0 || standing || conflicted > 0 || cpu !== undefined ? says : "", shared, pairs: pairs.slice(0, 20) };
8487 }
8588
8689
src/flow/cpu_stops.ts added +47
@@ -0,0 +1,47 @@
1 /**
2 * Checks stopped for CPU in the last hour (#11617), for /flow's one sentence:
3 * named only when they are the lane's main failure, at least as many as every
4 * other failed check together. A stop counts whether it was queued again
5 * (cancelled, `CPU_STOP_SUMMARY`) or reported red (a second stop in a row).
6 * Walks `run_kind_finished` (kind, finished_at): one hour of finished checks.
7 */
8
9 import type { Database } from "bun:sqlite";
10 import { CPU_STOP_SUMMARY } from "../core/runs.ts";
11
12 const HOUR_MS = 3_600_000;
13
14 /** How the watch's stop reads in a red check's summary: `check failed: stopped: used 16.3 cores for 31s, over its limit of 16…`. */
15 const CPU_RED = /stopped: (?:it )?used \d+(?:\.\d+)? cores for \d+s, over its limit of \d/;
16
17 export interface CpuStops {
18 /** Checks stopped for CPU in the hour. */
19 readonly stopped: number;
20 /** Every check that failed in the hour, the stops among them. */
21 readonly failed: number;
22 }
23
24 export function cpuStops(db: Database, now = new Date()): CpuStops {
25 const since = new Date(now.getTime() - HOUR_MS).toISOString();
26 const rows = db
27 .query<{ state: string; summary: string }, [string]>(
28 `SELECT state, summary FROM run
29 WHERE kind = 'check' AND finished_at IS NOT NULL AND finished_at >= ?1
30 AND state IN ('failed', 'cancelled')`,
31 )
32 .all(since);
33 let stopped = 0;
34 let failed = 0;
35 for (const r of rows) {
36 const cpu = r.state === "cancelled" ? r.summary === CPU_STOP_SUMMARY : CPU_RED.test(r.summary);
37 if (r.state === "failed" || cpu) failed++;
38 if (cpu) stopped++;
39 }
40 return { stopped, failed };
41 }
42
43 /** The reason /flow's sentence carries, or undefined when CPU stops are not the main failure. */
44 export function cpuStopReason(s: CpuStops): string | undefined {
45 if (s.stopped === 0 || s.stopped * 2 < s.failed) return undefined;
46 return `checks stopped for CPU: ${s.stopped} of ${s.failed} failed ${s.failed === 1 ? "check" : "checks"} in the last hour`;
47 }
test/check-cpu-stop.test.ts added +134
@@ -0,0 +1,134 @@
1 /**
2 * A check stopped for CPU (#11617): the cell's watch line reads as the
3 * machine's failure, the first stop of a head is queued again once on its own
4 * (cancelled, never red, so no builder hears check_failed), the second is
5 * reported, and /flow names the stops when they are the hour's main failure.
6 */
7 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
8 import { mkdtemp, rm } from "node:fs/promises";
9 import { tmpdir } from "node:os";
10 import { join } from "node:path";
11 import { createApp, type App } from "../src/app.ts";
12 import { createRepo } from "../src/actions/repos.ts";
13 import { cpuBreach, stoppedForCpu } from "../src/apps/contain.ts";
14 import { requeueCpuStop } from "../src/checks/cpu_stop.ts";
15 import type { Repo } from "../src/core/repos.ts";
16 import { claimRun, CPU_STOP_SUMMARY, createRun, finishRun, getRun, listRuns } from "../src/core/runs.ts";
17 import { createUser } from "../src/core/users.ts";
18 import { cpuStopReason, cpuStops } from "../src/flow/cpu_stops.ts";
19 import { readClogView } from "../src/flow/clog_view.ts";
20
21 /** The error `runStep` returns when the watch stops a check for CPU, from the watch's own words. */
22 const STOP = `check stopped: it ${cpuBreach(16.3, 31, 16, { pid: 52062, ticks: 46_200, name: "sh" })}`;
23
24 describe("stoppedForCpu", () => {
25 test("reads runStep's CPU stop, and nothing a check could print", () => {
26 expect(STOP).toBe("check stopped: it used 16.3 cores for 31s, over its limit of 16; the busiest process is sh (pid 52062, 462s of CPU)");
27 expect(stoppedForCpu(STOP)).toBe(true);
28 expect(stoppedForCpu(`check stopped: it ${cpuBreach(3, 300, 2, undefined)}`)).toBe(true);
29 // The check's own output follows "exited with code": it cannot pose as the watch.
30 expect(stoppedForCpu("check exited with code 1 — stopped: used 16.3 cores for 31s, over its limit of 16")).toBe(false);
31 expect(stoppedForCpu("check stopped: it used 900 MiB of memory, over its 512 MiB limit (default)")).toBe(false);
32 expect(stoppedForCpu("check took longer than 600s")).toBe(false);
33 });
34 });
35
36 describe("a check stopped for CPU", () => {
37 let dataDir: string;
38 let app: App;
39 let repo: Repo;
40
41 beforeAll(async () => {
42 dataDir = await mkdtemp(join(tmpdir(), "forge-cpu-stop-"));
43 app = createApp({ dataDir, port: 0 });
44 const a = await createUser(app.db, {
45 origin: "seed",
46 trusted: true,
47 username: "alice",
48 email: "[email protected]",
49 password: "a-good-passphrase",
50 adminIfFirst: true,
51 });
52 if (!a.ok) throw new Error(a.error);
53 const made = await createRepo(app, a.value, { name: "engine", visibility: "private" });
54 if (!made.ok) throw new Error(String(made.error));
55 repo = made.value;
56 });
57 afterAll(async () => {
58 app.close();
59 await rm(dataDir, { recursive: true, force: true });
60 });
61
62 test("is queued again once, not red; a second stop is reported", () => {
63 const first = createRun(app.db, { repoId: repo.id, kind: "check", env: "pull-7", commitOid: "a".repeat(40), reason: "push" });
64 const running = claimRun(app.db, first.id)!;
65 const moved = requeueCpuStop(app.db, running, STOP)!;
66 expect(moved).toBeDefined();
67 // Cancelled, not failed: nothing announces a red, so no check_failed is queued for it.
68 expect(getRun(app.db, first.id)!.state).toBe("cancelled");
69 expect(getRun(app.db, first.id)!.summary).toBe(CPU_STOP_SUMMARY);
70 expect(moved.again.state).toBe("queued");
71 expect(moved.again.attempt).toBe(1);
72 expect(moved.again.commitOid).toBe("a".repeat(40));
73 expect(moved.again.env).toBe("pull-7");
74
75 const second = claimRun(app.db, moved.again.id)!;
76 expect(requeueCpuStop(app.db, second, STOP)).toBeUndefined();
77 expect(getRun(app.db, second.id)!.state).toBe("running");
78 finishRun(app.db, second.id, { state: "failed", summary: `check failed: ${STOP.replace("check stopped: it ", "stopped: ")}` });
79 const runs = listRuns(app.db, repo.id, { kind: "check" });
80 expect(runs.map((r) => r.state).sort()).toEqual(["cancelled", "failed"]);
81 });
82
83 test("a red that is not a CPU stop is not queued again", () => {
84 const run = createRun(app.db, { repoId: repo.id, kind: "check", env: "pull-8", commitOid: "b".repeat(40), reason: "push" });
85 const running = claimRun(app.db, run.id)!;
86 expect(requeueCpuStop(app.db, running, "check exited with code 1 — 2 tests failed")).toBeUndefined();
87 expect(getRun(app.db, run.id)!.state).toBe("running");
88 finishRun(app.db, run.id, { state: "failed", summary: "check failed: 2 tests failed" });
89 });
90
91 test("/flow names CPU stops when they are the hour's main failure", () => {
92 // The hour so far: the queued-again stop, the reported stop, one ordinary red.
93 expect(cpuStops(app.db)).toEqual({ stopped: 2, failed: 3 });
94 expect(cpuStopReason(cpuStops(app.db))).toBe("checks stopped for CPU: 2 of 3 failed checks in the last hour");
95 // An hour later they have left the window.
96 expect(cpuStops(app.db, new Date(Date.now() + 3_700_000))).toEqual({ stopped: 0, failed: 0 });
97 expect(cpuStopReason({ stopped: 1, failed: 3 })).toBeUndefined();
98 expect(cpuStopReason({ stopped: 0, failed: 0 })).toBeUndefined();
99 expect(cpuStopReason({ stopped: 1, failed: 1 })).toBe("checks stopped for CPU: 1 of 1 failed check in the last hour");
100 });
101
102 test("/flow's sentence leads with the CPU stops, and is said when they are the only clog", () => {
103 // Nothing else clogs this forge: no shared red, no pairs, no standing clog, no conflicts.
104 const view = readClogView(app.db);
105 expect(view.says.startsWith("Clogged: checks stopped for CPU: 2 of 3 failed checks in the last hour")).toBe(true);
106 expect(view.shared).toEqual([]);
107 expect(view.pairs).toEqual([]);
108 // An hour later the stops have left the window, and the sentence with them.
109 expect(readClogView(app.db, new Date(Date.now() + 3_700_000)).says).not.toContain("stopped for CPU");
110 });
111
112 test("the hour's checks are read from run_kind_finished, not a scan", () => {
113 const db = app.db;
114 const original = db.query.bind(db);
115 const sqls: string[] = [];
116 (db as unknown as { query: unknown }).query = (sql: string) => {
117 sqls.push(sql);
118 return original(sql);
119 };
120 try {
121 cpuStops(db);
122 } finally {
123 (db as unknown as { query: unknown }).query = original;
124 }
125 expect(sqls.length).toBe(1);
126 const plan = original(`EXPLAIN QUERY PLAN ${sqls[0]!}`)
127 .all("x")
128 .map((r) => (r as { detail: string }).detail)
129 .join("\n");
130 expect(plan).toContain("run_kind_finished");
131 expect(plan).not.toMatch(/^SCAN (?!.*USING (COVERING )?INDEX)/m);
132 expect(plan).not.toContain("TEMP B-TREE");
133 });
134 });
test/crew-check-cpu-stop.test.ts added +116
@@ -0,0 +1,116 @@
1 /**
2 * A check stopped for CPU, through the checks service and the crew (#11617):
3 * the first stop of a pull's head is cancelled and queued again, its builder
4 * hears nothing; the second stop in a row is red and sends check_failed.
5 * The check step is stood in (`ChecksOptions.checkStep`): the stop is the
6 * cell watch's own words, which no check's output can produce.
7 */
8
9 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
10 import { existsSync } from "node:fs";
11 import { join } from "node:path";
12 import type { App } from "../src/app.ts";
13 import type { User } from "../src/core/users.ts";
14 import { addToCrew } from "../src/actions/crew.ts";
15 import { assignIssue, openIssue } from "../src/actions/issues.ts";
16 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
17 import { cpuBreach } from "../src/apps/contain.ts";
18 import type { runStep } from "../src/apps/process.ts";
19 import { CPU_STOP_SUMMARY, listRuns } from "../src/core/runs.ts";
20 import { err, ok } from "../src/kernel/result.ts";
21 import { startStandInCrew, stopStandInCrew, type StandInCrew } from "./helpers/crew-standin.ts";
22
23 /** The error `runStep` returns when the watch stops a check for CPU. */
24 const STOP = `check stopped: it ${cpuBreach(16.3, 31, 16, { pid: 52062, ticks: 46_200, name: "sh" })}`;
25
26 describe("a pull's check stopped for CPU, through the service", () => {
27 let standIn: StandInCrew;
28 let app: App;
29 let alice: User;
30 let bot: User;
31 /** Each stop the stand-in step made, in order. */
32 const stops: string[] = [];
33 /** The second stop waits for this, so the first is seen alone. */
34 let release!: () => void;
35 const released = new Promise<void>((r) => (release = r));
36 /** Whether the service announced a red check while only the first stop had happened. */
37 let redBeforeSecond = false;
38
39 // The pull's checkout holds CREW.md (the stand-in agent's work); main's does not and passes.
40 // The pull's first two checks are stopped for CPU; any later one passes.
41 const checkStep: typeof runStep = async (_name, command, options) => {
42 options.log(`$ ${command}`);
43 if (!existsSync(join(options.cwd, "CREW.md")) || stops.length >= 2) return ok();
44 if (stops.length === 1) await released;
45 stops.push(STOP);
46 options.log(STOP);
47 return err(STOP);
48 };
49
50 beforeAll(async () => {
51 standIn = await startStandInCrew({ tickMs: 50, checkStep });
52 ({ app, alice, bot } = standIn);
53 app.bus.subscribe((e) => {
54 if (e.kind === "run.finished" && (e.payload as { kind?: unknown; state?: unknown }).kind === "check"
55 && (e.payload as { state?: unknown }).state === "failed" && stops.length < 2) redBeforeSecond = true;
56 });
57 }, 30_000);
58
59 afterAll(() => stopStandInCrew(standIn));
60
61 const until = async <T>(find: () => T | undefined, timeoutMs = 20_000): Promise<T> => {
62 const deadline = Date.now() + timeoutMs;
63 while (Date.now() < deadline) {
64 const found = find();
65 if (found !== undefined) return found;
66 await Bun.sleep(20);
67 }
68 throw new Error("did not happen in time");
69 };
70
71 test("the first stop is queued again with no check_failed; the second is red and sends one", async () => {
72 const made = await createRepo(app, alice, { name: "kiln", visibility: "private" });
73 if (!made.ok) throw new Error(String(made.error));
74 const kiln = made.value;
75 const seeded = await commitToBranch(app, alice, kiln, {
76 message: "Start",
77 files: [
78 { path: "README.md", content: "# kiln\n" },
79 { path: "forge.json", content: '{"check": {"run": "exit 0", "install": ""}}\n' },
80 ],
81 });
82 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
83 expect(addToCrew(app, alice, kiln, "bot").ok).toBe(true);
84 const issue = openIssue(app, alice, kiln, { title: "Fire the kiln", body: "Slowly." });
85 if (!issue.ok) throw new Error(String(issue.error));
86 expect(assignIssue(app, alice, kiln, issue.value, bot).ok).toBe(true);
87
88 const built = await until(() =>
89 listRuns(app.db, kiln.id, { kind: "agent", limit: 50 }).find((r) => r.reason === "assigned" && r.state === "ok"),
90 );
91 const pullNumber = Number(built.summary.match(/#(\d+)/)?.[1]);
92 const pullChecks = () => listRuns(app.db, kiln.id, { kind: "check", env: `pull-${pullNumber}`, limit: 50 });
93 const reworks = () => listRuns(app.db, kiln.id, { kind: "agent", limit: 50 }).filter((r) => r.reason === "check_failed");
94
95 // The first stop: cancelled as the machine's, and its retry queued for the same head.
96 const first = await until(() => pullChecks().find((r) => r.attempt === 0 && r.state === "cancelled"));
97 expect(first.summary).toBe(CPU_STOP_SUMMARY);
98 const retry = await until(() => pullChecks().find((r) => r.attempt === 1));
99 expect(retry.commitOid).toBe(first.commitOid);
100 expect(stops).toHaveLength(1);
101 // The builder hears nothing of it: give dispatch time to answer an event, were there one.
102 await Bun.sleep(500);
103 expect(reworks()).toHaveLength(0);
104 expect(redBeforeSecond).toBe(false);
105
106 // The second stop in a row is reported red, and the builder is sent check_failed.
107 release();
108 const red = await until(() => pullChecks().find((r) => r.attempt === 1 && r.state === "failed"));
109 expect(red.summary).toContain("used 16.3 cores for 31s, over its limit of 16");
110 expect(stops).toHaveLength(2);
111 await until(() => (reworks().length > 0 ? true : undefined));
112 expect(reworks()).toHaveLength(1);
113 // Never a third attempt of that head: the second stop was not queued again.
114 expect(pullChecks().filter((r) => r.commitOid === first.commitOid).map((r) => r.attempt).sort()).toEqual([0, 1]);
115 }, 30_000);
116 });
test/helpers/crew-standin.ts +8 −2
@@ -18,7 +18,7 @@ import { createApp, type App } from "../../src/app.ts";
1818 import { addToCrew, createAgent } from "../../src/actions/crew.ts";
1919 import { commitToBranch, createRepo } from "../../src/actions/repos.ts";
2020 import { startCrew, type CrewHandle } from "../../src/crew/dispatch.ts";
21 import { startChecks, type ChecksHandle } from "../../src/checks/service.ts";
21 import { startChecks, type ChecksHandle, type ChecksOptions } from "../../src/checks/service.ts";
2222 import type { Repo } from "../../src/core/repos.ts";
2323 import { listRuns, type Run } from "../../src/core/runs.ts";
2424 import { createUser, type User } from "../../src/core/users.ts";
@@ -53,6 +53,8 @@ export async function startStandInCrew(
5353 readonly installRunner?: InstallRunner;
5454 /** Files committed to main beside the README. */
5555 readonly files?: { path: string; content: string }[];
56 /** Stands in for the check step (`ChecksOptions.checkStep`). */
57 readonly checkStep?: ChecksOptions["checkStep"];
5658 } = {},
5759 ): Promise<StandInCrew> {
5860 const dataDir = await mkdtemp(join(tmpdir(), "forge-crew-"));
@@ -198,7 +200,11 @@ echo '{"type":"result","subtype":"success","is_error":false,"result":"Added CREW
198200 `,
199201 );
200202 chmodSync(fake, 0o755);
201 const checks = startChecks(app, { forgeUrl: servers.url, concurrency: 1 });
203 const checks = startChecks(app, {
204 forgeUrl: servers.url,
205 concurrency: 1,
206 ...(options.checkStep === undefined ? {} : { checkStep: options.checkStep }),
207 });
202208 const crew = await startCrew(app, {
203209 forgeUrl: servers.url,
204210 localUrl: servers.url,

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs