trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

A check that reaches its CPU limit is slowed, not failed: no check is stopped for using the cores it was given #11575

Parked torquecrew opened this in answer to trevatoowner
ConversationFiles
Closed without merging

torque/forge:torque/11573-a-check-that-reaches-its-cpu-limit-is-sl → main 5 commits fork

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
14 files changed +346 −26
docs/REFERENCE.md +1 −1
@@ -214,7 +214,7 @@ Which repository is the forge's own source can also be set by an admin, without
214214
215215 **Keeping a domain drill**: `POST /api/v1/drills` (a site admin person; an agent is refused with 403, and a session needs a same-origin request) takes `{domain, seen:{health, board}, backSeconds}`: a domain key from `docs/DOMAINS.md`, what the drill read from health and the board (text, at most 8000 characters each), and whole seconds until the forge was well again (0 to 31536000). The body is capped at 32000 bytes (413). It answers 201 `{id, domain, backSeconds, green, why}`: the verdict is judged against the domain's own words and kept whether green or red. An unknown domain, bad `seen` or bad `backSeconds` is 422. Kept drills are read back on **Settings → Backups**, in its **Failure domains** card (`docs/SNAPSHOT.md`). To run a drill by hand, `FORGE_TOKEN=… bun scripts/drills/run.ts <domain> --forge <url>` (plain `http` is refused unless the host is loopback or you pass `--insecure`) prompts for the fault, watches health and the board until the forge is back, and posts this for you (`docs/DOMAINS.md`).
216216
217 **Re-checking a shared red**: `POST /api/v1/repos/:owner/:name/checks/recheck` (a repository admin, `admin` token scope; anyone else gets 404, and a session needs a same-origin request) takes `{test}`: the name of one failing test exactly as the check log's failing names show it, 1 to 300 characters (otherwise 400). It asks again, through the ordinary check queue, for every open pull whose latest check on its current head failed on that test, and answers `{queued: [pull numbers]}`. At most 50 pulls are queued per call; when more were red, the answer adds `more` (how many were left) and a `message`, and a second call takes the rest. A pull waiting for a writer's approval, or whose head declares no check, is not in `queued`.
217 **Re-checking a shared red**: `POST /api/v1/repos/:owner/:name/checks/recheck` (a repository admin, `admin` token scope; anyone else gets 404, and a session needs a same-origin request) takes `{test}`: the name of one failing test exactly as the check log's failing names show it, 1 to 300 characters (otherwise 400). It asks again, through the ordinary check queue, for every open pull whose latest check on its current head failed on that test, and answers `{queued: [pull numbers]}`. At most 50 pulls are queued per call; when more were red, the answer adds `more` (how many were left) and a `message`, and a second call takes the rest. A pull waiting for a writer's approval, or whose head declares no check, is not in `queued`. Sent `{cause: "cpu"}` in place of `{test}`, it asks again for every open pull whose latest check on its head went red only because the CPU watch stopped it (`stopped: used … cores …`, #11573); any other `cause` is a 400 `invalid_cause`. A check the CPU watch stops is itself queued again once, as infrastructure, before it can go red: the watch stops a cell only past 1.5× its cores held for two minutes, and never where the kernel holds its cores (`cpu.max`).
218218
219219 If your shell already has `FORGE_URL` set — a crew workspace sets it to the
220220 forge you work for — this forge takes it as its own canonical address and
docs/features/flow-clog.md +1 −1
@@ -2,7 +2,7 @@
22
33 The owner's `/flow` opens with these rows, no box (`src/web/pages/flow-clog.ts`, mounted by one line in `flow.ts`; read by `src/flow/clog_view.ts`):
44
5 - **One sentence** from `clogSays`: the standing clog's reasons, approved pulls that conflict, the file most open pulls touch, and the dollars waiting to land.
5 - **One sentence** from `clogSays`: the open pulls red only because the CPU watch stopped their check, first, when they are at least half the red and more than one (#11573); the standing clog's reasons, approved pulls that conflict, the file most open pulls touch, and the dollars waiting to land.
66 - **One test, many pulls** from `sharedRed`: a test failing in the latest check of three or more open pulls is listed once, with each pull linked.
77 - **Pulls on the same files** from `conflictPairs`: at most 20 pairs, each linked to both pulls.
88
src/apps/contain.ts +51 −3
@@ -857,6 +857,35 @@ function readMountinfo(): string {
857857
858858 /** How often the process table is read, and over how long CPU is averaged. */
859859 export const WATCH = { tickMs: 1_000, cpuWindowMs: 30_000, diskEveryTicks: 15 };
860 /**
861 * When the watch stops a cell for CPU, where the kernel does not hold its
862 * cores (#11573): its windowed average past `factor` × its limit, held for
863 * `sustainMs`. A cell using the cores it was given — 16.1 of 16, the
864 * sampling's own jitter — is never over; a runaway at 1.5× for two minutes is.
865 */
866 export const CPU_STOP = { factor: 1.5, sustainMs: 120_000 };
867
868 /** Whether `cores` (averaged over the window) is past the stop line for a limit of `cpus`: strictly over `factor` × `cpus`. */
869 export function cpuOver(cores: number, cpus: number): boolean {
870 return cores > cpus * CPU_STOP.factor;
871 }
872
873 /**
874 * One tick of the CPU watch: when the cell has been over the stop line since
875 * (`since`, ms, or undefined when it is not over now) and whether that has
876 * held for `CPU_STOP.sustainMs`. Plain data in and out, so a test can run
877 * three minutes of ticks without a process.
878 */
879 export function cpuTick(
880 since: number | undefined,
881 cores: number,
882 cpus: number,
883 now: number,
884 ): { since: number | undefined; stop: boolean } {
885 if (!cpuOver(cores, cpus)) return { since: undefined, stop: false };
886 const from = since ?? now;
887 return { since: from, stop: now - from >= CPU_STOP.sustainMs };
888 }
860889 /** A cell's process limit: the watch's every tick, and the kernel's `pids.max` where pids is delegated. */
861890 export const PROCESSES_MAX = 512;
862891 const CLK_TCK = 100;
@@ -875,6 +904,8 @@ interface Watched {
875904 readonly cap: MemoryCap;
876905 /** The tree's CPU ticks so far and resident KiB, at the last tick. */
877906 last?: { ticks: number; kib: number };
907 /** Since when (ms) its windowed cores have been past the stop line (`cpuTick`). */
908 cpuOverSince?: number | undefined;
878909 }
879910
880911 const watched = new Set<Watched>();
@@ -1149,12 +1180,25 @@ export function cpuBreach(
11491180 cpus: number,
11501181 top: { pid: number; ticks: number; name: string } | undefined,
11511182 ): string {
1152 const why = `used ${cores.toFixed(1)} cores for ${Math.round(seconds)}s, over its limit of ${cpus}`;
1183 const why = `used ${cores.toFixed(1)} cores for ${Math.round(seconds)}s, over ${CPU_STOP.factor}× its limit of ${cpus}`;
11531184 if (top === undefined) return why;
11541185 const spent = Math.round(top.ticks / CLK_TCK);
11551186 return `${why}; the busiest process is ${top.name} (pid ${top.pid}, ${spent}s of CPU)`;
11561187 }
11571188
1189 /**
1190 * Whether a step's error is the watch's CPU stop: `runStep`'s "<step>
1191 * stopped: it " and `cpuBreach`'s line. Anchored at the start, where only
1192 * the runner writes — a step's own output is quoted after "exited with
1193 * code", so no pull can print its way into a re-check. A stop for CPU is
1194 * the machine's verdict on a sampled average, not the code's: a check
1195 * reads it as infrastructure (#11573).
1196 */
1197 export function cpuStopped(error: string): boolean {
1198 return CPU_STOP_LINE.test(error);
1199 }
1200 const CPU_STOP_LINE = /^[a-z]+ stopped: it used \d+(?:\.\d+)? cores for \d+s, over /;
1201
11581202 /** A process's command name, printable and short, or "?" once it is gone. */
11591203 function commandName(pid: number): string {
11601204 try {
@@ -1246,11 +1290,15 @@ function tick(): void {
12461290 if (oldest !== undefined && now - oldest.at >= WATCH.cpuWindowMs) {
12471291 const cores =
12481292 (ticks - oldest.ticks) / CLK_TCK / ((now - oldest.at) / 1000);
1249 if (cores > limits.cpus) {
1293 const verdict = cpuTick(entry.cpuOverSince, cores, limits.cpus, now);
1294 entry.cpuOverSince = verdict.since;
1295 if (verdict.stop) {
12501296 // Read only now, at the stop: no tick pays for naming a process.
12511297 const top = busiest(tree, (pid) => rows.get(pid)?.ticks ?? 0);
1298 // Over since the window that first crossed the line began.
1299 const seconds = (now - (verdict.since ?? now) + WATCH.cpuWindowMs) / 1000;
12521300 entry.breach(
1253 cpuBreach(cores, (now - oldest.at) / 1000, limits.cpus, top === undefined ? undefined : { ...top, name: commandName(top.pid) }),
1301 cpuBreach(cores, seconds, limits.cpus, top === undefined ? undefined : { ...top, name: commandName(top.pid) }),
12541302 );
12551303 continue;
12561304 }
src/checks/recheck.ts +35 −5
@@ -1,7 +1,8 @@
11 /**
22 * Every open pull whose latest check on its current head failed on one
33 * named test, asked for again (#6445): once a shared red is fixed on main,
4 * the pulls it held are handed back to the check queue in one call.
4 * the pulls it held are handed back to the check queue in one call. And
5 * every open pull red only because the CPU watch stopped it (#11573).
56 */
67
78 import type { App } from "../app.ts";
@@ -40,16 +41,45 @@ export async function recheckNamed(
4041 test: string,
4142 max = RECHECK_MAX,
4243 ): Promise<Rechecked> {
43 const red = listOpenPulls(app.db, repo.id).filter((pull) => {
44 return recheckWhere(app, user, repo, (run) => run.checkDetail?.names.includes(test) ?? false, max);
45 }
46
47 /**
48 * A failed check's summary that is the CPU watch's stop, not a test's red
49 * (#11573): "check failed: stopped: used 16.3 cores for 30s, over its limit
50 * of 16; …" as the pulls of 2026-11-11 read, or a stop since.
51 */
52 export function cpuStopSummary(summary: string): boolean {
53 return CPU_STOP_SUMMARY_LINE.test(summary);
54 }
55 const CPU_STOP_SUMMARY_LINE = /^check failed: (?:[a-z]+ )?stopped: (?:it )?used \d+(?:\.\d+)? cores for \d+s, over /;
56
57 /**
58 * Every open pull whose latest check on its head went red only because the
59 * CPU watch stopped it, asked for again (#11573): the pulls a stop at 16.1
60 * of 16 cores failed before the watch learned that is not over.
61 */
62 export async function recheckCpuStopped(app: App, user: User, repo: Repo, max = RECHECK_MAX): Promise<Rechecked> {
63 return recheckWhere(app, user, repo, (run) => cpuStopSummary(run.summary ?? ""), max);
64 }
65
66 async function recheckWhere(
67 app: App,
68 user: User,
69 repo: Repo,
70 red: (run: Run) => boolean,
71 max: number,
72 ): Promise<Rechecked> {
73 const pulls = listOpenPulls(app.db, repo.id).filter((pull) => {
4474 if (pull.headOid === "") return false;
4575 const run = latestCheck(app, pull);
46 return run?.state === "failed" && (run.checkDetail?.names.includes(test) ?? false);
76 return run?.state === "failed" && red(run);
4777 });
4878 const queued: number[] = [];
49 for (const pull of red.slice(0, max)) {
79 for (const pull of pulls.slice(0, max)) {
5080 // The ordinary request: the check queue's own slots and order apply.
5181 const asked = await askCheck(app, user, repo, pull);
5282 if (asked.ok) queued.push(pull.number);
5383 }
54 return { queued, left: Math.max(0, red.length - max) };
84 return { queued, left: Math.max(0, pulls.length - max) };
5585 }
src/checks/service.ts +18 −1
@@ -43,6 +43,8 @@ import {
4343 VANISHED_SUMMARY,
4444 infraBefore,
4545 INFRA_SUMMARY,
46 cpuStopBefore,
47 CPU_STOP_SUMMARY,
4648 requeueCheck,
4749 REQUEUED_SUMMARY,
4850 sweepAbandonedChecks,
@@ -106,6 +108,7 @@ import {
106108 } from "../core/capacity.ts";
107109 import {
108110 cellsAvailable,
111 cpuStopped,
109112 limitWarnings,
110113 resolveLimits,
111114 STEP_HOSTS,
@@ -862,6 +865,16 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
862865 return;
863866 }
864867 }
868 // The CPU watch stopped it: the machine's stop, not the pull's red.
869 // Checked once more on its own, and its builder hears nothing (#11573).
870 if (!outcome.ok && outcome.cpuStop === true && !cpuStopBefore(app.db, claimed)) {
871 const moved = requeueCheck(app.db, run.id, undefined, CPU_STOP_SUMMARY);
872 if (moved) {
873 write(CPU_STOP_SUMMARY);
874 announceRun(app, moved.again, run.actorId ?? undefined);
875 return;
876 }
877 }
865878 const seconds = Math.round((Date.now() - started) / 1000);
866879 const done = finishRun(app.db, run.id, {
867880 state: outcome.ok ? "ok" : "failed",
@@ -905,7 +918,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
905918 signal: AbortSignal,
906919 ): Promise<
907920 | { ok: true; proven?: string; passed?: readonly string[] }
908 | { ok: false; error: string; detail?: CheckDetail | undefined; infrastructure?: boolean; passed?: readonly string[] }
921 | { ok: false; error: string; detail?: CheckDetail | undefined; infrastructure?: boolean; cpuStop?: boolean; passed?: readonly string[] }
909922 > {
910923 const bare = await runDir(app.db, app.reposRoot, repo, run.issueId);
911924 // A pull's check is the fast tier, judged against where the pull left
@@ -991,6 +1004,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
9911004 /** A whole verification: the files its log ran to the end green (#9524). */
9921005 passed?: readonly string[];
9931006 infrastructure?: boolean;
1007 /** The CPU watch stopped the check step (#11573): queued once more, not red. */
1008 cpuStop?: boolean;
9941009 /** Where it stopped red (#5276); none is before any step ran. */
9951010 stage?: GateStage;
9961011 /** The check step was ended by a signal from outside: no verdict (#6722). */
@@ -1351,6 +1366,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13511366 });
13521367 }
13531368 if (!ran.ok && ran.error === JOB_VANISHED) return { ok: false, error: JOB_VANISHED };
1369 // Stopped for CPU: no test failed, so nothing is re-run alone or filed.
1370 if (!ran.ok && cpuStopped(ran.error)) return { ok: false, error: ran.error, cpuStop: true, stage: checkStage(ran.error) };
13541371 // A file the gate cleared because it passed alone is still a failure
13551372 // to track: the cell cannot reach the forge, so it is filed from here.
13561373 try {
src/contract/openapi/routes/postApiV1ReposOwnerNameChecksRecheck.ts +2 −1
@@ -3,12 +3,13 @@ import { REPO, type Route } from "../shared.ts";
33 export const route: Route = {
44 method: "post",
55 path: `${REPO}/checks/recheck`,
6 summary: "Ask again for every open pull whose latest check on its head failed on the test named in the body { test } (admin scope, a repository admin's; at most 50 per call); answers { queued: [numbers] }, with more when some were left",
6 summary: "Ask again for every open pull whose latest check on its head failed on the test named in the body { test }, or, with { cause: \"cpu\" }, was stopped by the CPU watch (admin scope, a repository admin's; at most 50 per call); answers { queued: [numbers] }, with more when some were left",
77 auth: true,
88 role: "member",
99 status: 200,
1010 reply: { queued: "array" },
1111 refusals: {
1212 invalid_test: "test is a name of 1 to 300 characters (400)",
13 invalid_cause: 'cause is "cpu", the one cause named (400)',
1314 },
1415 };
src/core/runs.ts +12
@@ -525,6 +525,18 @@ export const VANISHED_SUMMARY = "interrupted: its cluster job vanished; queued a
525525 /** What a check whose install the machine failed twice says as it is queued again (#5093). */
526526 export const INFRA_SUMMARY = "failed: infrastructure — the install failed twice on a transient error; queued again";
527527
528 /**
529 * What a check the CPU watch stopped says as it is queued again (#11573):
530 * the machine's stop, not the pull's red, so it is checked once more and
531 * its builder hears nothing.
532 */
533 export const CPU_STOP_SUMMARY = "failed: infrastructure — stopped for CPU by the watch, not by its code; queued again";
534
535 /** Whether the attempt just before this one was stopped for CPU: it is queued again once. */
536 export function cpuStopBefore(db: Database, run: Run): boolean {
537 return cutOffBefore(db, run, CPU_STOP_SUMMARY);
538 }
539
528540 /**
529541 * How many times one check may be cut off by its forge going away and be
530542 * queued again. Past it the check fails once, saying so: a check that kills
src/flow/clog_view.ts +31 −4
@@ -1,7 +1,8 @@
11 /**
2 * What clogs the lane (#11454), read once for /flow and its terminal screen: one sentence (`clogSays`), a test failing on three or more open
3 * pulls shown once (`sharedRed`), and pairs of open pulls touching the same files (`conflictPairs`, at most 20). Read from the runs table and
4 * the pull file lists `crew/hot_files.ts` keeps in memory; no git per request. Quiet is an empty sentence and empty lists.
2 * What clogs the lane (#11454), read once for /flow and its terminal screen: one sentence (`clogSays`, naming the CPU watch's stops
3 * when they are most of the red, #11573), a test failing on three or more open pulls shown once (`sharedRed`), and pairs of open pulls
4 * touching the same files (`conflictPairs`, at most 20). Read from the runs table and the pull file lists `crew/hot_files.ts` keeps in
5 * memory; no git per request. Quiet is an empty sentence and empty lists.
56 */
67
78 import type { Database } from "bun:sqlite";
@@ -14,6 +15,7 @@ import { conflictedPulls } from "./conflicted.ts";
1415 import { sharedRed } from "./shared_red.ts";
1516 import { readStranded } from "./stranded.ts";
1617 import { approvedRows } from "../web/pages/flow-waiting.ts";
18 import { cpuStopSummary } from "../checks/recheck.ts";
1719
1820 export interface ClogView {
1921 readonly says: string;
@@ -39,6 +41,28 @@ function failingRows(db: Database): { repo: string; number: number; tests: strin
3941 });
4042 }
4143
44 /**
45 * How many open pulls' latest check failed, and how many of those only
46 * because the CPU watch stopped it (#11573): the summaries of the latest
47 * failed check of each open pull, read once.
48 */
49 export function redCauses(db: Database): { failed: number; cpu: number } {
50 const rows = db
51 .query<{ summary: string | null }, []>(
52 `SELECT u.summary AS summary
53 FROM pull p JOIN issue i ON i.id = p.issue_id AND i.state = 'open'
54 JOIN run u ON u.id = (SELECT n.id FROM run n WHERE n.issue_id = i.id AND n.kind = 'check' ORDER BY n.id DESC LIMIT 1)
55 WHERE p.merged_at IS NULL AND u.state = 'failed'`,
56 )
57 .all();
58 return { failed: rows.length, cpu: rows.filter((r) => cpuStopSummary(r.summary ?? "")).length };
59 }
60
61 /** The CPU stop is the lane's main failure: at least half its red pulls, and more than one. */
62 export function cpuIsMain(causes: { failed: number; cpu: number }): boolean {
63 return causes.cpu >= 2 && causes.cpu * 2 >= causes.failed;
64 }
65
4266 export function readClogView(db: Database, now = new Date()): ClogView {
4367 const failing = failingRows(db);
4468 const key = (r: Ref): string => `${r.repo}#${r.number}`;
@@ -74,13 +98,16 @@ export function readClogView(db: Database, now = new Date()): ClogView {
7498 const conflicted = conflictedPulls(db).filter((c) => approved.has(key(c))).length;
7599 const hot = warmFiles().sort((a, b) => b.pulls - a.pulls || a.path.localeCompare(b.path))[0];
76100 const standing = readClog(db, now).standing;
101 const causes = redCauses(db);
102 const cpu = cpuIsMain(causes) ? causes.cpu : 0;
77103 const says = clogSays({
78104 reasons: standing?.reasons ?? [],
79105 conflictedApproved: conflicted,
80106 hottest: hot,
81107 strandedUsd: readStranded(db).usd,
108 cpuStopped: cpu,
82109 });
83 return { says: shared.length > 0 || pairs.length > 0 || standing || conflicted > 0 ? says : "", shared, pairs: pairs.slice(0, 20) };
110 return { says: shared.length > 0 || pairs.length > 0 || standing || conflicted > 0 || cpu > 0 ? says : "", shared, pairs: pairs.slice(0, 20) };
84111 }
85112
86113
src/flow/clog_words.ts +5
@@ -5,6 +5,8 @@ export type ClogFacts = {
55 conflictedApproved?: number | null;
66 hottest?: { path: string; pulls: number } | undefined;
77 strandedUsd?: number | null;
8 /** Open pulls red only because the CPU watch stopped their check (#11573), given when that is the lane's main failure. */
9 cpuStopped?: number | null;
810 };
911
1012 const MAX = 160;
@@ -17,8 +19,11 @@ export function clogSays(f: ClogFacts): string {
1719 const conflicts = count(f.conflictedApproved);
1820 const hot = f.hottest && count(f.hottest.pulls) > 0 && f.hottest.path ? f.hottest : undefined;
1921 const dollars = Math.round(f.strandedUsd ?? 0);
22 const cpu = count(f.cpuStopped);
2023 const build = (reasons: readonly string[], withHot: boolean): string => {
2124 const parts = reasons.filter((r) => r.trim() !== "").map((r) => r.trim());
25 // The lane's main failure leads: it is not the pulls' code (#11573).
26 if (cpu) parts.unshift(`${pulls(cpu, "open")} went red only because the CPU watch stopped ${cpu === 1 ? "its check" : "their checks"}`);
2227 if (conflicts) parts.push(`${pulls(conflicts, "approved")} ${conflicts === 1 ? "conflicts" : "conflict"}`);
2328 if (hot && withHot) parts.push(`${pulls(count(hot.pulls), "open")} touch ${hot.path}`);
2429 const head = parts.length ? `Clogged: ${parts.join("; ")}.` : "";
src/http/checks_recheck.ts +14 −6
@@ -1,8 +1,9 @@
11 /**
2 * The admin's one call to re-check every open pull red on one named test.
2 * The admin's one call to re-check every open pull red on one named test,
3 * or (`{cause: "cpu"}`) every open pull the CPU watch stopped (#11573).
34 */
45
5 import { RECHECK_MAX, recheckNamed, TEST_NAME_MAX } from "../checks/recheck.ts";
6 import { RECHECK_MAX, recheckCpuStopped, recheckNamed, TEST_NAME_MAX } from "../checks/recheck.ts";
67 import { canAdmin, getRepo } from "../core/repos.ts";
78 import type { Handler } from "./context.ts";
89 import { apiError, body, json, notFound, requireScope, str } from "./respond.ts";
@@ -14,10 +15,17 @@ export function checksRecheckRoutes(router: Router<Handler>): void {
1415 if (denied) return denied;
1516 const repo = getRepo(ctx.app.db, ctx.params.owner!, ctx.params.name!);
1617 if (!repo || !canAdmin(ctx.app.db, repo, ctx.auth.user)) return notFound();
17 const test = str((await body(ctx.request)).test);
18 if (test.length < 1 || test.length > TEST_NAME_MAX)
19 return apiError("invalid_test", 400, `test is a name of 1 to ${TEST_NAME_MAX} characters`);
20 const done = await recheckNamed(ctx.app, ctx.auth.user!, repo, test);
18 const input = await body(ctx.request);
19 let done;
20 if (input.cause !== undefined) {
21 if (input.cause !== "cpu") return apiError("invalid_cause", 400, 'cause is "cpu", the one cause named');
22 done = await recheckCpuStopped(ctx.app, ctx.auth.user!, repo);
23 } else {
24 const test = str(input.test);
25 if (test.length < 1 || test.length > TEST_NAME_MAX)
26 return apiError("invalid_test", 400, `test is a name of 1 to ${TEST_NAME_MAX} characters`);
27 done = await recheckNamed(ctx.app, ctx.auth.user!, repo, test);
28 }
2129 return json(
2230 done.left > 0
2331 ? { queued: done.queued, more: done.left, message: `at most ${RECHECK_MAX} per call; ${done.left} left` }
test/checks-recheck.test.ts +29
@@ -114,3 +114,32 @@ test("past the cap, the rest are left and the answer says how many", async () =>
114114 const next = await recheckNamed(app, alice, repo, "capped", 1);
115115 expect(next).toEqual({ queued: [pulls[1]!.number], left: 0 });
116116 });
117
118 /** A red the CPU watch made, not a test (#11573): its summary as the pulls of 2026-11-11 read. */
119 function stoppedForCpu(pull: Pull, summary: string): void {
120 const run = createRun(app.db, { repoId: repo.id, kind: "check", env: `pull-${pull.number}`, commitOid: pull.headOid, issueId: pull.id });
121 claimRun(app.db, run.id);
122 finishRun(app.db, run.id, { state: "failed", summary });
123 }
124
125 test("{cause: \"cpu\"} queues the pulls the CPU watch stopped, and not one red on a test", async () => {
126 stoppedForCpu(pulls[0]!, "check failed: stopped: used 16.3 cores for 30s, over its limit of 16; the busiest process is sh (pid 52062, 462s of CPU)");
127 stoppedForCpu(pulls[1]!, "check failed: check stopped: it used 30.0 cores for 150s, over 1.5× its limit of 16");
128 redOn(pulls[2]!, "used 16.3 cores");
129 const before = asked().length;
130 const res = await call({ cause: "cpu" });
131 expect(res.status).toBe(200);
132 expect(await res.json()).toEqual({ queued: [pulls[0]!.number, pulls[1]!.number] });
133 expect(asked().length - before).toBe(2);
134 });
135
136 test("any other cause is a 400 and queues nothing", async () => {
137 const before = asked().length;
138 expect((await call({ cause: "memory" })).status).toBe(400);
139 expect((await call({ cause: 1 })).status).toBe(400);
140 expect(asked().length - before).toBe(0);
141 });
142
143 test("a non-admin asking by cause gets 404", async () => {
144 expect((await call({ cause: "cpu" }, {}, bobCookie)).status).toBe(404);
145 });
test/contain.test.ts +8 −1
@@ -27,6 +27,8 @@ import {
2727 mountMode,
2828 secretMounts,
2929 WATCH,
30 CPU_STOP,
31 cpuStopped,
3032 type CellSpec,
3133 } from "../src/apps/contain.ts";
3234 import { envDir } from "../src/apps/data.ts";
@@ -786,10 +788,12 @@ describe.skipIf(noCells)("cells: limits stop a step, and say which", () => {
786788 let dir: string;
787789 let hidden: string;
788790 const saved = { ...WATCH };
791 const savedStop = { ...CPU_STOP };
789792
790793 beforeAll(async () => {
791794 dir = await mkdtemp(join(tmpdir(), "forge-cell-limits-"));
792795 hidden = await mkdtemp(join(tmpdir(), "forge-cell-hidden-"));
796 CPU_STOP.sustainMs = 1_000;
793797 WATCH.tickMs = 200;
794798 WATCH.cpuWindowMs = 3_000;
795799 WATCH.diskEveryTicks = 1;
@@ -797,6 +801,7 @@ describe.skipIf(noCells)("cells: limits stop a step, and say which", () => {
797801
798802 afterAll(async () => {
799803 Object.assign(WATCH, saved);
804 Object.assign(CPU_STOP, savedStop);
800805 await rm(dir, { recursive: true, force: true });
801806 await rm(hidden, { recursive: true, force: true });
802807 });
@@ -840,8 +845,10 @@ describe.skipIf(noCells)("cells: limits stop a step, and say which", () => {
840845 expect(result.ok).toBe(false);
841846 if (!result.ok)
842847 expect(result.error).toMatch(
843 /^check stopped: it used [\d.]+ cores for \d+s, over its limit of 0\.02; the busiest process is \S+ \(pid \d+, \d+s of CPU\)$/,
848 /^check stopped: it used [\d.]+ cores for \d+s, over 1\.5× its limit of 0\.02; the busiest process is \S+ \(pid \d+, \d+s of CPU\)$/,
844849 );
850 // …and a check reads that stop as infrastructure, not as its red (#11573).
851 if (!result.ok) expect(cpuStopped(result.error)).toBe(true);
845852 }, 30_000);
846853
847854 test("over its disk limit", async () => {
test/cpu-breach.test.ts +80 −3
@@ -3,7 +3,8 @@
33 * (#11509), and only when one has spent any.
44 */
55 import { describe, expect, test } from "bun:test";
6 import { busiest, cpuBreach } from "../src/apps/contain.ts";
6 import { cpuStopSummary } from "../src/checks/recheck.ts";
7 import { busiest, cpuBreach, cpuOver, cpuStopped, cpuTick, CPU_STOP } from "../src/apps/contain.ts";
78
89 describe("a cell stopped for CPU", () => {
910 test("the busiest process is the one with the most ticks in the tree", () => {
@@ -20,11 +21,87 @@ describe("a cell stopped for CPU", () => {
2021 // 9_300 ticks over 30 s at 100 ticks a second is 3.1 cores.
2122 const cores = 9_300 / 100 / 30;
2223 expect(cpuBreach(cores, 30.4, 1, { pid: 4242, ticks: 9_150, name: "chrome" })).toBe(
23 "used 3.1 cores for 30s, over its limit of 1; the busiest process is chrome (pid 4242, 92s of CPU)",
24 "used 3.1 cores for 30s, over 1.5× its limit of 1; the busiest process is chrome (pid 4242, 92s of CPU)",
2425 );
2526 });
2627
2728 test("with no process to name, the line is the cores alone", () => {
28 expect(cpuBreach(3.1, 30, 1, undefined)).toBe("used 3.1 cores for 30s, over its limit of 1");
29 expect(cpuBreach(3.1, 30, 1, undefined)).toBe("used 3.1 cores for 30s, over 1.5× its limit of 1");
30 });
31 });
32
33 /**
34 * A cell is stopped for CPU only past 1.5× its limit, held two minutes
35 * (#11573): a check using the 16 cores it was given is never stopped.
36 * Ticks are the watch's own, one a second (`WATCH.tickMs`).
37 */
38 describe("when the watch stops a cell for CPU", () => {
39 /** Run `seconds` of one-second ticks at `cores(s)`; the second it stops at, or undefined. */
40 const run = (cores: (s: number) => number, cpus: number, seconds: number): number | undefined => {
41 let since: number | undefined;
42 for (let s = 0; s <= seconds; s++) {
43 const v = cpuTick(since, cores(s), cpus, s * 1_000);
44 since = v.since;
45 if (v.stop) return s;
46 }
47 return undefined;
48 };
49
50 test("the stop line is 1.5× the limit, for two minutes", () => {
51 expect(CPU_STOP).toEqual({ factor: 1.5, sustainMs: 120_000 });
52 });
53
54 test("using exactly the limit is not over it", () => {
55 expect(cpuOver(16, 16)).toBe(false);
56 expect(cpuTick(undefined, 16, 16, 0)).toEqual({ since: undefined, stop: false });
57 });
58
59 test("16.1 and 17.5 of 16 — the pulls #11546 and #11547 were stopped at — are not over", () => {
60 expect(cpuOver(16.1, 16)).toBe(false);
61 expect(cpuOver(17.5, 16)).toBe(false);
62 // 1.5 × 16 = 24: at the line is not over it, a hair past is.
63 expect(cpuOver(24, 16)).toBe(false);
64 expect(cpuOver(24.01, 16)).toBe(true);
65 });
66
67 test("a check using all of its 16 cores for three minutes is never stopped", () => {
68 expect(run(() => 16, 16, 180)).toBeUndefined();
69 // Sampling jitter just above the limit, every tick, for three minutes.
70 expect(run((s) => 16 + (s % 3) * 0.6, 16, 180)).toBeUndefined();
71 });
72
73 test("a runaway at twice its limit is stopped once it has held 120 s", () => {
74 expect(run(() => 32, 16, 180)).toBe(120);
75 });
76
77 test("a spike past the line that falls back starts the count again", () => {
78 // Over for 100 s, back at the limit for 1 s, over again: 100 + 120.
79 const cores = (s: number): number => (s === 100 ? 16 : 30);
80 expect(run(cores, 16, 300)).toBe(221);
81 });
82 });
83
84 describe("a check stopped for CPU reads as infrastructure (#11573)", () => {
85 test("the runner's stop line is a CPU stop", () => {
86 expect(cpuStopped(`check stopped: it ${cpuBreach(30, 150, 16, { pid: 7, ticks: 49_400, name: "sh" })}`)).toBe(true);
87 expect(cpuStopped("check stopped: it used 3.1 cores for 150s, over 1.5× its limit of 1")).toBe(true);
88 });
89
90 test("a memory or disk stop, a test's red, and a step printing the line are not", () => {
91 expect(cpuStopped("check stopped: it used 300 MiB of memory, over its 128 MiB limit (the default)")).toBe(false);
92 expect(cpuStopped("check stopped: it wrote 200 MiB to disk, over its 64 MiB limit")).toBe(false);
93 expect(cpuStopped("check exited with code 1 — 2 fail")).toBe(false);
94 expect(
95 cpuStopped("check exited with code 1 — check stopped: it used 30.0 cores for 150s, over 1.5× its limit of 16"),
96 ).toBe(false);
97 });
98 });
99
100 describe("a check's summary that is a CPU stop (#11573)", () => {
101 test("the 2026-11-11 summaries and a stop since are; a test's red is not", () => {
102 expect(cpuStopSummary("check failed: stopped: used 16.1 cores for 31s, over its limit of 16 …")).toBe(true);
103 expect(cpuStopSummary("check failed: check stopped: it used 30.0 cores for 150s, over 1.5× its limit of 16")).toBe(true);
104 expect(cpuStopSummary("check failed: 2 tests failed")).toBe(false);
105 expect(cpuStopSummary("check failed: stopped: used 3000 MiB of memory, over its 2048 MiB limit")).toBe(false);
29106 });
30107 });
test/flow-clog-cpu.test.ts added +59
@@ -0,0 +1,59 @@
1 /**
2 * /flow's one sentence names the CPU watch's stops when they are the lane's
3 * main failure (#11573): the pulls are red for the machine, not their code.
4 */
5 import { beforeAll, describe, expect, test } from "bun:test";
6 import type { Database } from "bun:sqlite";
7 import { cpuIsMain, readClogView, redCauses } from "../src/flow/clog_view.ts";
8 import { clogSays } from "../src/flow/clog_words.ts";
9 import { migrations } from "../src/core/schema.ts";
10 import { openDb } from "../src/kernel/db.ts";
11 import { id } from "../src/kernel/id.ts";
12
13 const STOPPED = "check failed: stopped: used 16.3 cores for 30s, over its limit of 16; the busiest process is sh (pid 52062, 462s of CPU)";
14
15 let db: Database;
16 let n = 0;
17 /** An open pull whose latest check failed with `summary`. */
18 const redPull = (summary: string): void => {
19 const issueId = id();
20 n += 1;
21 db.query(`INSERT INTO issue (id, repo_id, number, kind, title, author_id, state) VALUES (?1, 'r1', ?2, 'pull', 'P', 'u1', 'open')`).run(issueId, n);
22 db.query(`INSERT INTO pull (issue_id, base_branch, head_branch, base_oid, head_oid, head_repo_id) VALUES (?1, 'main', ?2, '', ?3, 'r1')`).run(issueId, `b${n}`, `o${n}`);
23 db.query(`INSERT INTO run (id, repo_id, kind, issue_id, state, summary) VALUES (?1, 'r1', 'check', ?2, 'failed', ?3)`).run(id(), issueId, summary);
24 };
25
26 beforeAll(() => {
27 db = openDb(":memory:", migrations);
28 db.run("INSERT INTO user (id, username, email, password_hash) VALUES ('u1', 'trevato', 't@x', '')");
29 db.run("INSERT INTO repo (id, owner_id, name) VALUES ('r1', 'u1', 'forge')");
30 });
31
32 describe("the clog sentence and the CPU stop", () => {
33 test("the sentence leads with the CPU stop when it is given", () => {
34 expect(clogSays({ reasons: [], cpuStopped: 3 })).toBe("Clogged: 3 open pulls went red only because the CPU watch stopped their checks.");
35 expect(clogSays({ reasons: [], cpuStopped: 1, conflictedApproved: 2 })).toBe(
36 "Clogged: 1 open pull went red only because the CPU watch stopped its check; 2 approved pulls conflict.",
37 );
38 expect(clogSays({ reasons: [] })).toBe("");
39 });
40
41 test("main means at least half the red, and more than one pull", () => {
42 expect(cpuIsMain({ failed: 34, cpu: 24 })).toBe(true);
43 expect(cpuIsMain({ failed: 4, cpu: 2 })).toBe(true);
44 expect(cpuIsMain({ failed: 5, cpu: 2 })).toBe(false);
45 expect(cpuIsMain({ failed: 1, cpu: 1 })).toBe(false);
46 });
47
48 test("one stop among test reds is not named; when stops are most of the red, /flow says so", () => {
49 redPull("check failed: 2 tests failed");
50 redPull("check failed: 1 test failed");
51 redPull(STOPPED);
52 expect(redCauses(db)).toEqual({ failed: 3, cpu: 1 });
53 expect(readClogView(db).says).not.toContain("CPU watch");
54 redPull(STOPPED);
55 redPull("check failed: stopped: used 17.5 cores for 31s, over its limit of 16");
56 expect(redCauses(db)).toEqual({ failed: 5, cpu: 3 });
57 expect(readClogView(db).says).toContain("3 open pulls went red only because the CPU watch stopped their checks");
58 });
59 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs