trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns13g rEventsFlows

The merge queue re-runs a batch's budget-only reds once, serially, before calling it red — budget tests under gate load are sinking 8-pull batches #7357

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/7354-the-merge-queue-re-runs-a-batch-s-budget → main 2 commits fork 1 thread · 1 unresolved

  • Merged. torque/7354-the-merge-queue-re-runs-a-batch-s-budget was merged into main as e15cf9e.

Review

Sign in to review this change.
9 files changed +607 −3
docs/features/gate-budget-retry.md added +35
@@ -0,0 +1,35 @@
1 # Budget reds in a merge-queue gate (#7354)
2
3 A budget test (`expectBudget`, `test/helpers/budget.ts`) can fail only under the
4 gate's own load, while main and the pull pass it alone. When a red merge-queue
5 gate is red **only** on budget tests, the gate re-runs those test files once,
6 serially, before the batch counts as red (`src/checks/budget_retry.ts`, wired
7 in `checks/service.ts`).
8
9 - **Which reds count.** `expectBudget` throws `over budget (expectBudget): …`
10 when a time is over its limit, and bun prints a test's error just above its
11 `(fail)` line. A gate is a budget red when every `(fail)` comes after that
12 error, each has a file, the fail counts add up, and the log has no error
13 outside a test, no shard that named no test, and no skipped shards. At most
14 four files. Anything else is a functional red, searched at once with no re-run.
15 - **The re-run.** `bun test --max-concurrency=1 './file' …` in the same checkout
16 and cell, with `FORGE_TEST_QUIET=1`, so each budget is held to its absolute ms.
17 It passes: the gate is **green**, the batch lands, and each test is a row in
18 `check_flake` (the flake list; twice in a week files an issue, as
19 [check-flake](check-flake.md)). It fails: **red, as before**.
20 - **Only the test files re-run.** A green re-run turns the whole check green,
21 as main's flake re-run does. That is sound for this forge: `check.run` ends
22 with `test-shards`. A repository whose gate runs more steps after its tests
23 (`… && test-shards … && more`) has those steps skipped on a budget red, and a
24 green re-run lands the batch without them ever running.
25 - **The shards run on.** `scripts/test-shards.ts` stops the other shards at the
26 first red (fail-fast), except a shard red only on budget tests: the gate can
27 only be called green when every other file ran too.
28 - **The tick line** says `budgetRetries="P passed/F failed"` for the last hour's
29 re-runs when there were any. Re-runs that keep failing mean a budget is truly
30 regressed.
31
32 Left out: the main and pull checks (main has its own flake re-run), cluster
33 Jobs (the re-run needs the check's own cell), and a `budget` tag in a test's
34 name. Only `expectBudget`'s own error counts, so a raw `toBeLessThan` or a test
35 that only says "budget" in its name is never re-run.
scripts/shards.ts +11
@@ -8,6 +8,7 @@
88 */
99
1010 import { nameUnnamed } from "../src/checks/bun-test.ts";
11 import { budgetReds } from "../src/checks/budget_retry.ts";
1112
1213 /**
1314 * No more than this many `bun test` processes at once, whatever the host:
@@ -367,6 +368,16 @@ export interface ShardResult {
367368 readonly seconds?: number;
368369 }
369370
371 /**
372 * Whether a finished shard stops the shards not yet done (fail-fast): any
373 * red does, except one whose every failure is a budget test (#7354). The
374 * forge re-runs those once, serially, after the gate — which it can only
375 * call green when every other file ran too.
376 */
377 export function stopsTheRest(result: Pick<ShardResult, "code" | "output">): boolean {
378 return result.code !== 0 && budgetReds(result.output.split("\n")) === undefined;
379 }
380
370381 /** What the log ends with, and the exit code of the whole run. */
371382 export interface Tally {
372383 readonly code: number;
scripts/test-shards.ts +5 −2
@@ -23,7 +23,8 @@
2323 * finishes, so the log reads one shard at a time. Either way the tail
2424 * names every failing test from every shard. The first red shard stops
2525 * the rest (fail-fast) so a known fail does not burn the remaining wall
26 * clock. Exits non-zero if any shard does.
26 * clock — unless every red in it is a budget test, which the forge re-runs
27 * once after the gate (#7354). Exits non-zero if any shard does.
2728 *
2829 * The whole gate, unsliced (`--gate` on the full tier, as `check.run`),
2930 * runs one file per `bun test`, heaviest first into whichever worker is
@@ -63,6 +64,7 @@ import {
6364 besideOthers,
6465 failingFiles,
6566 onlyTestFailures,
67 stopsTheRest,
6668 type ShardResult,
6769 } from "./shards.ts";
6870 import { gateFiles, gateRefusal, presentTests } from "./gate.ts";
@@ -561,7 +563,8 @@ async function pool(width: number): Promise<Timed[]> {
561563 const index = next++;
562564 if (stop) break;
563565 results[index] = await runShard(slices[index]!, index, width === 1);
564 if (results[index]!.code !== 0) {
566 // A shard red only on budget tests lets the rest run (#7354).
567 if (stopsTheRest(results[index]!)) {
565568 stop = true;
566569 halted = true;
567570 for (const child of children) child.kill("SIGTERM");
src/checks/budget_retry.ts added +149
@@ -0,0 +1,149 @@
1 /**
2 * A merge-queue gate whose only reds are budget tests re-runs those test
3 * files once, serially, before the batch counts as red (#7354). A budget
4 * test fails only under the gate's own load — eight pulls' worth of shards
5 * beside it — and passes on main and on the pull alone; a red one cost a
6 * batch of eight and thirteen minutes. Passing the re-run, the batch is
7 * green and the tests are recorded as flakes (`flake.ts`); failing it, the
8 * batch is red as before. A functional red is never re-run: it is searched
9 * at once.
10 *
11 * A budget red is one `expectBudget` (`test/helpers/budget.ts`) threw: its
12 * error starts with {@link BUDGET_MARK}, and bun prints a test's error just
13 * above its `(fail)` line. Plain lines in, files and a command out. The
14 * names and files come from the repository's own output, so each is checked
15 * before it reaches a shell, and quoted there.
16 */
17
18 import { nameUnnamed } from "./bun-test.ts";
19 import type { Flaky } from "./flake.ts";
20
21 /** What `expectBudget`'s error starts with: the gate reads it to know a budget red. */
22 export const BUDGET_MARK = "over budget (expectBudget)";
23
24 const MARKED = /^error: over budget \(expectBudget\): /;
25 const HEADER = /^(\S+\.test\.[cm]?[jt]sx?):$/;
26 const FAIL = /^\(fail\) (.+?)(?: \[[0-9.]+m?s\])?( \[shard [0-9]+\])?$/;
27 const RESULT = /^\((?:pass|fail|skip|todo)\) /;
28 const FAIL_COUNT = /^\s*(\d+) fail$/;
29 /** Lines that say a red was not (only) a failed test: an error outside one, a shard that named none, shards never run. */
30 const NOT_A_TEST = [
31 /^# Unhandled error/,
32 /^\s*[1-9][0-9]* errors?\s*$/,
33 /^shard [0-9]+ (?:killed after|exited [0-9]+ without naming)/,
34 /^test-shards: fail-fast/,
35 ];
36 const PATH = /^[\w@./+-]{1,200}$/;
37 // eslint-disable-next-line no-control-regex
38 const CONTROL = /[\u0000-\u001f\u007f-\u009f]/;
39 const NAME_MAX = 300;
40
41 /** The most files one re-run takes: more budget reds than this is a slow change, not a loaded gate. */
42 export const MAX_BUDGET_FILES = 4;
43
44 /**
45 * The tests of a red check, each with its file, when every red in it is a
46 * budget red; undefined otherwise — a functional failure, a failure outside
47 * any test, a test bun did not name, a shard skipped, or no test failed.
48 * The sharded gate echoes each `(fail)` line with ` [shard N]` at its end:
49 * an echo names a test again, it is not a second failure.
50 */
51 export function budgetReds(lines: readonly string[]): Flaky[] | undefined {
52 const named = nameUnnamed(lines);
53 if (named.some((line) => NOT_A_TEST.some((re) => re.test(line)))) return undefined;
54 const found = new Map<string, Flaky>();
55 const echoed = new Set<string>();
56 let file: string | undefined;
57 let marked = false;
58 let failures = 0;
59 let counted = 0;
60 for (const line of named) {
61 const header = HEADER.exec(line);
62 if (header) {
63 file = header[1];
64 marked = false;
65 continue;
66 }
67 if (MARKED.test(line)) marked = true;
68 const count = FAIL_COUNT.exec(line);
69 if (count) counted += Number(count[1]);
70 const fail = FAIL.exec(line);
71 if (fail) {
72 const name = fail[1]!;
73 if (fail[2] !== undefined) echoed.add(name);
74 else {
75 failures++;
76 if (!marked || file === undefined) return undefined;
77 if (!PATH.test(file) || name.length > NAME_MAX || CONTROL.test(name)) return undefined;
78 found.set(`${file}\0${name}`, { file, name });
79 }
80 }
81 if (RESULT.test(line)) marked = false;
82 }
83 if (failures === 0 || failures !== counted) return undefined;
84 const names = new Set([...found.values()].map((t) => t.name));
85 for (const name of echoed) if (!names.has(name)) return undefined;
86 const tests = [...found.values()];
87 if (budgetFiles(tests).length > MAX_BUDGET_FILES) return undefined;
88 return tests;
89 }
90
91 /** The files the tests are in, each once, in the order first named. */
92 export function budgetFiles(tests: readonly Flaky[]): string[] {
93 return [...new Set(tests.map((t) => t.file))];
94 }
95
96 const quote = (s: string): string => `'${s.replace(/'/g, `'\\''`)}'`;
97
98 /**
99 * `bun test` of exactly these files (a `./` path, never a filter that also
100 * matches a sibling), one test at a time. The caller sets `FORGE_TEST_QUIET=1`
101 * in its environment: the machine is the gate's alone again by then.
102 */
103 export function budgetRetryCommand(files: readonly string[]): string {
104 const paths = files.map((f) => (f.startsWith("/") || f.startsWith("./") ? f : `./${f}`));
105 return `bun test --max-concurrency=1 ${paths.map(quote).join(" ")}`;
106 }
107
108 /** Whether the re-run's own output says tests ran and none failed. */
109 export function retryPassed(lines: readonly string[]): boolean {
110 let pass = 0;
111 let fail = 0;
112 for (const line of lines) {
113 const p = /^\s*(\d+) pass$/.exec(line);
114 if (p) pass += Number(p[1]);
115 const f = FAIL_COUNT.exec(line);
116 if (f) fail += Number(f[1]);
117 }
118 return pass > 0 && fail === 0;
119 }
120
121 const HOUR_MS = 60 * 60_000;
122
123 /** Each repository's re-runs in the last hour: when, and whether it passed. */
124 const retries = new Map<string, { at: number; passed: boolean }[]>();
125
126 /** A gate re-ran its budget reds: passed or not. */
127 export function noteBudgetRetry(repoId: string, passed: boolean, now = Date.now()): void {
128 const kept = (retries.get(repoId) ?? []).filter((r) => now - r.at < HOUR_MS);
129 kept.push({ at: now, passed });
130 retries.set(repoId, kept);
131 }
132
133 /** The last hour's budget re-runs of a repository's gates. */
134 export function budgetRetries(repoId: string, now = Date.now()): { passed: number; failed: number } {
135 const kept = (retries.get(repoId) ?? []).filter((r) => now - r.at < HOUR_MS);
136 retries.set(repoId, kept);
137 const passed = kept.filter((r) => r.passed).length;
138 return { passed, failed: kept.length - passed };
139 }
140
141 /**
142 * The tick line's words for the last hour's budget re-runs:
143 * `2 passed/1 failed`. Undefined when there were none: a budget that
144 * keeps failing its re-run is truly regressed, and the steward sees it here.
145 */
146 export function budgetRetryWords(repoId: string, now = Date.now()): string | undefined {
147 const { passed, failed } = budgetRetries(repoId, now);
148 return passed + failed === 0 ? undefined : `${passed} passed/${failed} failed`;
149 }
src/checks/service.ts +38
@@ -58,6 +58,7 @@ import { keepCache, restoreCache } from "./cache.ts";
5858 import { MAX_FILED, clearedFiles, fileOrderDependent } from "./order_dependent.ts";
5959 import { parseBunTestFailure } from "./bun-test.ts";
6060 import { passedAlone, recordFlakes, rerunCommand, rerunnable } from "./flake.ts";
61 import { budgetFiles, budgetReds, budgetRetryCommand, noteBudgetRetry, retryPassed } from "./budget_retry.ts";
6162 import { diskRecovered, withDisk } from "./disk_red.ts";
6263 import { ASKED, redMain, unheld } from "./red_main.ts";
6364 import { proposeRevert } from "./revert_red.ts";
@@ -743,6 +744,8 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
743744 readonly fileOrderDependent?: boolean;
744745 /** The default branch's own run: a failure of one or two tests is re-run alone before it counts as red. */
745746 readonly isMain?: boolean;
747 /** A merge-queue gate: a red only on budget tests re-runs those files once, serially (#7354). */
748 readonly gate?: boolean;
746749 readonly write: (line: string) => void;
747750 readonly signal: AbortSignal;
748751 }
@@ -1098,6 +1101,40 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
10981101 write(`flake re-run failed: ${String(error)}`);
10991102 }
11001103 }
1104 // A gate's red only on budget tests is the gate's own load, not the
1105 // batch's code (#7354): those files run once more, one test at a
1106 // time, held to their absolute budgets. Passing, the batch is green
1107 // and the tests are recorded as flakes; failing, it is red as before.
1108 if (!ran.ok && job === undefined && x.gate === true) {
1109 try {
1110 const tests = budgetReds(checkLines);
1111 if (tests !== undefined) {
1112 const files = budgetFiles(tests);
1113 write(`re-running ${files.length} file(s) red only on budget tests, serially: ${files.join(" ")}`);
1114 const out: string[] = [];
1115 const again = await runStep("budget", budgetRetryCommand(files), {
1116 cwd: dir,
1117 env: { ...env, FORGE_TEST_QUIET: "1" },
1118 log: (line) => {
1119 out.push(line);
1120 write(line);
1121 },
1122 signal,
1123 timeoutMs: Math.min(manifest.timeout, RERUN_TIMEOUT_S) * 1000,
1124 cell,
1125 });
1126 const passed = again.ok && retryPassed(out);
1127 if (!signal.aborted) noteBudgetRetry(repo.id, passed);
1128 if (passed) {
1129 recordFlakes(app.db, app.bus, repo, tests, { runId: run.id, commit: run.commitOid });
1130 write(`budget retry passed: ${tests.map((t) => `"${t.name}"`).join(", ")} held their budgets alone; recorded, the gate is green`);
1131 ran = { ok: true, value: undefined };
1132 } else write("budget retry failed: the gate is red");
1133 }
1134 } catch (error) {
1135 write(`budget retry failed to run: ${String(error)}`);
1136 }
1137 }
11011138 if (!ran.ok) {
11021139 const { summary, detail } = summariseCheckFailure(
11031140 checkLines,
@@ -1336,6 +1373,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13361373 commitOid,
13371374 pull,
13381375 tier: "full",
1376 gate: true,
13391377 write: (line) => lines.push(line),
13401378 signal: controller.signal,
13411379 });
src/crew/autonomy.ts +9
@@ -130,6 +130,7 @@ import { approvedOnHead, finishApproved, finishFields } from "./finishing.ts";
130130 import { recheckRedApproved } from "./red_approved.ts";
131131 import { keepBatch } from "../flow/red_batches.ts";
132132 import { recordQueueFlakes } from "../checks/flake.ts";
133 import { budgetRetryWords } from "../checks/budget_retry.ts";
133134 import { readVitals, vitalsMisses, vitalsWords } from "../flow/vitals.ts";
134135 import { moveRequested, tallyRequested, tallyWords } from "./requested.ts";
135136 import { reviewersOf, sendToReview, sweepReviews } from "./review.ts";
@@ -1348,6 +1349,8 @@ export async function autonomyTick(
13481349 batch: batchWords(current.id, queueBatch(app.db)),
13491350 // Pulls the queue found red on top of others of their candidate (#4235).
13501351 clashes: queueClashes(current.id),
1352 // The last hour's gates re-run red only on budget tests, passed/failed (#7354).
1353 ...budgetRetriesField(current.id, now.getTime()),
13511354 spent: `${status.spentTodayUsd.toFixed(2)}/${status.ceilingUsd}`,
13521355 ...(status.paused === null ? {} : { paused: status.paused }),
13531356 // Starts after the first, and how many followed a crash (#3708).
@@ -2121,6 +2124,12 @@ function queueOptions(app: App, repo: Repo): QueueOptions {
21212124 };
21222125 }
21232126
2127 /** `budgetRetries: "2 passed/1 failed"` on the tick line when a gate re-ran its budget reds this hour; nothing otherwise. */
sentinelcrew outdated

This JSDoc belongs to gateFor, but it now sits above budgetRetriesField's doc block, and gateFor has no doc. Move budgetRetriesField (with its comment) above this line, or next to the other tick-line helpers.
2128 function budgetRetriesField(repoId: string, now: number): { budgetRetries?: string } {
2129 const words = budgetRetryWords(repoId, now);
2130 return words === undefined ? {} : { budgetRetries: words };
2131 }
2132
21242133 /** The queue's gate: the full tier on the candidate's tip, asked as its first pull. */
21252134 function gateFor(app: App, repo: Repo): GateCandidate {
21262135 return (candidate, signal) =>
test/budget-retry.test.ts added +171
@@ -0,0 +1,171 @@
1 /**
2 * The gate tells a budget red from a functional one (#7354): every red an
3 * `expectBudget` threw is re-run; anything else in the log keeps it red.
4 */
5 import { afterAll, beforeAll, expect, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import {
10 BUDGET_MARK,
11 budgetFiles,
12 budgetReds,
13 budgetRetries,
14 budgetRetryCommand,
15 budgetRetryWords,
16 noteBudgetRetry,
17 retryPassed,
18 } from "../src/checks/budget_retry.ts";
19 import { stopsTheRest } from "../scripts/shards.ts";
20
21 const budgetError = `error: ${BUDGET_MARK}: 81.0 ms is not under 50.0 ms (budget 50 ms, baseline 2.0 ms)`;
22
23 /** One one-file shard as test-shards writes it: bun's report under its header. */
24 const shard = (n: number, file: string, fails: { name: string; error: string }[], passes = 1): string[] => [
25 `── shard ${n}/3 (exit ${fails.length > 0 ? 1 : 0}, 2.0s): ${file}`,
26 "bun test v1.3.14",
27 "",
28 `${file}:`,
29 ...Array.from({ length: passes }, (_, i) => `(pass) ok ${i} [1.00ms]`),
30 ...fails.flatMap((f) => ["12 | expectBudget(dur, { ms: 50, against });", " ^", f.error, " at <anonymous> (/x/y.ts:12:3)", `(fail) ${f.name} [81.00ms]`]),
31 "",
32 ` ${passes} pass`,
33 ` ${fails.length} fail`,
34 `Ran ${passes + fails.length} tests across 1 file. [90.00ms]`,
35 ];
36
37 const tail = (names: string[]): string[] => [
38 "",
39 `${names.length === 0 ? 0 : 1} of 3 shards failed:`,
40 ...names.map((n) => `(fail) ${n} [81.00ms] [shard 1]`),
41 `Ran 9 tests across 3 files in 4.0s; ${names.length} failed.`,
42 ];
43
44 test("a gate red only on budget tests names each with its file, echoes and all", () => {
45 const lines = [
46 ...shard(1, "test/branches-many.test.ts", [{ name: "200 branches hold the 50 ms budget", error: budgetError }]),
47 ...shard(2, "test/map-seed-timing.test.ts", [{ name: "warm > the seed's map answers within its budget", error: budgetError }]),
48 ...shard(3, "test/quiet.test.ts", []),
49 ...tail(["200 branches hold the 50 ms budget", "warm > the seed's map answers within its budget"]),
50 ];
51 const tests = budgetReds(lines)!;
52 expect(tests).toEqual([
53 { file: "test/branches-many.test.ts", name: "200 branches hold the 50 ms budget" },
54 { file: "test/map-seed-timing.test.ts", name: "warm > the seed's map answers within its budget" },
55 ]);
56 expect(budgetFiles(tests)).toEqual(["test/branches-many.test.ts", "test/map-seed-timing.test.ts"]);
57 });
58
59 test("one functional red beside a budget red keeps the gate red: nothing is re-run", () => {
60 const functional = "error: expect(received).toBe(expected)";
61 const lines = [
62 ...shard(1, "test/a.test.ts", [{ name: "slow", error: budgetError }]),
63 ...shard(2, "test/b.test.ts", [{ name: "wrong", error: functional }]),
64 ...tail(["slow", "wrong"]),
65 ];
66 expect(budgetReds(lines)).toBeUndefined();
67 expect(budgetReds([...shard(1, "test/b.test.ts", [{ name: "wrong", error: functional }]), ...tail(["wrong"])])).toBeUndefined();
68 });
69
70 test("a budget error printed for one test does not cover the next test's failure", () => {
71 const lines = [
72 "test/a.test.ts:",
73 budgetError,
74 "(fail) slow [81.00ms]",
75 "error: expect(received).toBe(expected)",
76 "(fail) wrong [1.00ms]",
77 " 0 pass",
78 " 2 fail",
79 ];
80 expect(budgetReds(lines)).toBeUndefined();
81 });
82
83 test("a red that is not only failed tests is never a budget red", () => {
84 const budget = shard(1, "test/a.test.ts", [{ name: "slow", error: budgetError }]);
85 expect(budgetReds(budget)).toHaveLength(1);
86 // An error between tests, a shard that named none, shards skipped, a hook.
87 expect(budgetReds([...budget, "# Unhandled error between tests"])).toBeUndefined();
88 expect(budgetReds([...budget, " 1 error"])).toBeUndefined();
89 expect(budgetReds([...budget, "shard 2 exited 1 without naming a test: test/b.test.ts"])).toBeUndefined();
90 expect(budgetReds([...budget, "test-shards: fail-fast — skipped 4 remaining shards after a red exit"])).toBeUndefined();
91 expect(budgetReds(["test/a.test.ts:", "(fail) (unnamed) [1.00ms]", " ^ a beforeEach hook timed out", " 0 pass", " 1 fail"])).toBeUndefined();
92 // A shard that crashed after its budget red never counted it: not every red was counted.
93 expect(budgetReds(budget.filter((l) => l !== " 1 fail"))).toBeUndefined();
94 // An echo of a test no shard showed failing with a budget error.
95 expect(budgetReds([...budget, "(fail) other [1.00ms] [shard 2]"])).toBeUndefined();
96 // Type errors, no tests at all, a green log.
97 expect(budgetReds(["src/a.ts(1,1): error TS2322"])).toBeUndefined();
98 expect(budgetReds(shard(1, "test/a.test.ts", []))).toBeUndefined();
99 });
100
101 test("a file or name that could become shell is never re-run", () => {
102 expect(budgetReds(shard(1, "test/$(rm -rf).test.ts", [{ name: "slow", error: budgetError }]))).toBeUndefined();
103 expect(budgetReds(shard(1, "test/a.test.ts", [{ name: "slow\u0007", error: budgetError }]))).toBeUndefined();
104 });
105
106 test("more budget files than a loaded gate explains is a slow change: red", () => {
107 const lines = [1, 2, 3, 4, 5].flatMap((n) => shard(n, `test/t${n}.test.ts`, [{ name: `slow ${n}`, error: budgetError }]));
108 expect(budgetReds(lines)).toBeUndefined();
109 expect(budgetReds(lines.slice(0, lines.length * (4 / 5)))).toHaveLength(4);
110 });
111
112 test("the re-run is exactly those files, quoted, one test at a time", () => {
113 expect(budgetRetryCommand(["test/a.test.ts", "test/it's.test.ts"])).toBe(
114 `bun test --max-concurrency=1 './test/a.test.ts' './test/it'\\''s.test.ts'`,
115 );
116 });
117
118 test("a re-run passes only when tests ran and none failed", () => {
119 expect(retryPassed([" 7 pass", " 0 fail"])).toBe(true);
120 expect(retryPassed([" 6 pass", " 1 fail"])).toBe(false);
121 expect(retryPassed([" 0 pass", " 0 fail"])).toBe(false);
122 expect(retryPassed(["error: module not found"])).toBe(false);
123 });
124
125 test("a fail-fast pool lets a budget-only red shard run on, and stops at any other", () => {
126 const budget = shard(1, "test/a.test.ts", [{ name: "slow", error: budgetError }]).join("\n");
127 const functional = shard(1, "test/a.test.ts", [{ name: "wrong", error: "error: expect(received).toBe(expected)" }]).join("\n");
128 expect(stopsTheRest({ code: 1, output: budget })).toBe(false);
129 expect(stopsTheRest({ code: 1, output: functional })).toBe(true);
130 expect(stopsTheRest({ code: 137, output: "" })).toBe(true);
131 expect(stopsTheRest({ code: 0, output: "" })).toBe(false);
132 });
133
134 test("the tick line counts the last hour's re-runs, passed and failed", () => {
135 const repo = "budget-words";
136 const t = 1_800_000_000_000;
137 expect(budgetRetryWords(repo, t)).toBeUndefined();
138 noteBudgetRetry(repo, true, t);
139 noteBudgetRetry(repo, true, t + 1_000);
140 noteBudgetRetry(repo, false, t + 2_000);
141 expect(budgetRetries(repo, t + 3_000)).toEqual({ passed: 2, failed: 1 });
142 expect(budgetRetryWords(repo, t + 3_000)).toBe("2 passed/1 failed");
143 // An hour on, the first two are gone.
144 expect(budgetRetryWords(repo, t + 3_601_500)).toBe("0 passed/1 failed");
145 expect(budgetRetryWords(repo, t + 7_200_000)).toBeUndefined();
146 });
147
148 let dir: string;
149 beforeAll(async () => {
150 dir = await mkdtemp(join(tmpdir(), "budget-retry-"));
151 });
152 afterAll(async () => {
153 await rm(dir, { recursive: true, force: true });
154 });
155
156 test("what a real expectBudget prints when it fails is read as a budget red; a plain expect is not", async () => {
157 const helper = join(import.meta.dir, "helpers", "budget.ts");
158 await Bun.write(
159 join(dir, "t", "a.test.ts"),
160 `import { expect, test } from "bun:test";\nimport { expectBudget } from ${JSON.stringify(helper)};\n` +
161 `test("slow page", () => { expectBudget(300, { ms: 50, against: 1 }); });\n` +
162 `test("no phase", () => { expectBudget(Number.NaN, { ms: 50, against: 1 }); });\n` +
163 `test("fine", () => { expectBudget(1, { ms: 50, against: 1 }); });\n`,
164 );
165 const run = Bun.spawnSync(["bun", "test", "./t/a.test.ts", "-t", "slow page|fine"], { cwd: dir, env: { ...process.env, FORGE_TEST_QUIET: "" } });
166 const out = (run.stdout.toString() + run.stderr.toString()).split("\n");
167 expect(budgetReds(out)).toEqual([{ file: "t/a.test.ts", name: "slow page" }]);
168 // A timing that is missing is a functional red: the phase was never named.
169 const nan = Bun.spawnSync(["bun", "test", "./t/a.test.ts", "-t", "no phase"], { cwd: dir, env: { ...process.env, FORGE_TEST_QUIET: "" } });
170 expect(budgetReds((nan.stdout.toString() + nan.stderr.toString()).split("\n"))).toBeUndefined();
171 }, 30_000);
test/helpers/budget.ts +17 −1
@@ -9,8 +9,15 @@
99 * a quiet machine; no lane sets it yet). Otherwise the limit is relative to the
1010 * baseline but never above `MAX_RELAXATION` times `ms`, so a 50 ms budget
1111 * cannot drift past 200 ms however slow the baseline.
12 *
13 * A time over its limit throws an error that starts with `BUDGET_MARK`, so
14 * the merge queue's gate can tell a budget red from a functional one and
15 * re-run it once, serially, before it sinks a batch (#7354,
16 * `src/checks/budget_retry.ts`). A time that is not a number is a missing
17 * phase, not a slow one: that fails as any `expect` does.
1218 */
1319 import { expect } from "bun:test";
20 import { BUDGET_MARK } from "../../src/checks/budget_retry.ts";
1421
1522 /** How many times slower than the same-run baseline a page may be under load. */
1623 export const BASELINE_FACTOR = 10;
@@ -29,7 +36,16 @@ export function budgetLimit(ms: number, against: number): number {
2936
3037 /** Asserts `measured` (ms) fits the budget `ms`, judged against `against` (ms, same run) unless quiet. */
3138 export function expectBudget(measured: number, opts: { ms: number; against: number }): void {
32 expect(measured).toBeLessThan(budgetLimit(opts.ms, opts.against));
39 if (!Number.isFinite(measured)) {
40 expect(measured).toBeLessThan(budgetLimit(opts.ms, opts.against));
41 return;
42 }
43 const limit = budgetLimit(opts.ms, opts.against);
44 if (measured < limit) return;
45 throw new Error(
46 `${BUDGET_MARK}: ${measured.toFixed(1)} ms is not under ${limit.toFixed(1)} ms ` +
47 `(budget ${opts.ms} ms, baseline ${opts.against.toFixed(1)} ms${quiet() ? ", quiet" : ""})`,
48 );
3349 }
3450
3551 /** The best wall-clock ms of a few loopback requests to `/healthz`: what a request costs right now. */
test/merge-queue-budget-retry.test.ts added +172
@@ -0,0 +1,172 @@
1 /**
2 * The merge queue, driven through the real check lane (#7354): a batch whose
3 * only red is a budget test re-runs that file once, serially, and lands
4 * when it passes — the test recorded as a flake. Red again on the re-run,
5 * or red on a functional test, the batch is red and searched as before:
6 * a functional red is never re-run.
7 */
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { createApp, type App } from "../src/app.ts";
13 import { openPull } from "../src/actions/pulls.ts";
14 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
15 import { budgetRetries } from "../src/checks/budget_retry.ts";
16 import { flakeList } from "../src/checks/flake.ts";
17 import { gateCandidate } from "../src/checks/landing.ts";
18 import { type Candidate, enqueue, type Expect, type Queued } from "../src/checks/queue.ts";
19 import { startChecks, type ChecksHandle } from "../src/checks/service.ts";
20 import { getPull } from "../src/core/pulls.ts";
21 import type { Repo } from "../src/core/repos.ts";
22 import { createUser, type User } from "../src/core/users.ts";
23 import { repoDir } from "../src/git/repos.ts";
24 import { quietLogs } from "./helpers/globals.ts";
25
26 quietLogs();
27
28 /** What the gate's sharded run prints for one red test in `t/a.test.ts`, then exits 1. */
29 const gateLog = (error: string): string =>
30 [
31 "cat <<'EOF'",
32 "── shard 1/2 (exit 1, 2.0s): t/a.test.ts",
33 "t/a.test.ts:",
34 error,
35 "(fail) grp > page [81.00ms]",
36 " 0 pass",
37 " 1 fail",
38 "Ran 1 test across 1 file. [90.00ms]",
39 "── shard 2/2 (exit 0, 1.0s): t/b.test.ts",
40 "t/b.test.ts:",
41 "(pass) fine [1.00ms]",
42 " 1 pass",
43 " 0 fail",
44 "1 of 2 shards failed:",
45 "(fail) grp > page [81.00ms] [shard 1]",
46 "EOF",
47 "exit 1",
48 "",
49 ].join("\n");
50
51 const BUDGET = "error: over budget (expectBudget): 81.0 ms is not under 50.0 ms (budget 50 ms, baseline 2.0 ms)";
52 const FUNCTIONAL = "error: expect(received).toBe(expected)";
53
54 describe("the merge queue re-runs a batch's budget-only reds once", () => {
55 let dataDir: string;
56 let app: App;
57 let checks: ChecksHandle;
58 let alice: User;
59 const cells = process.env["FORGE_CELLS"];
60
61 beforeAll(async () => {
62 process.env["FORGE_CELLS"] = "off";
63 dataDir = await mkdtemp(join(tmpdir(), "forge-budget-retry-"));
64 app = createApp({ dataDir, port: 0 });
65 checks = startChecks(app, { forgeUrl: "http://127.0.0.1:1", concurrency: 2 });
66 const a = await createUser(app.db, {
67 origin: "seed",
68 trusted: true,
69 username: "alice",
70 email: "[email protected]",
71 password: "a-good-passphrase",
72 adminIfFirst: true,
73 });
74 if (!a.ok) throw new Error(a.error);
75 alice = a.value;
76 });
77
78 afterAll(async () => {
79 await checks.stop({ drain: false });
80 app.close();
81 if (cells === undefined) delete process.env["FORGE_CELLS"];
82 else process.env["FORGE_CELLS"] = cells;
83 await rm(dataDir, { recursive: true, force: true });
84 });
85
86 /**
87 * A repository whose gate prints `error` above its one red test, whose
88 * `t/a.test.ts` does `body` when re-run, and two pulls, queued as a batch.
89 */
90 async function queueWith(name: string, error: string, body: string) {
91 const made = await createRepo(app, alice, { name, visibility: "public" });
92 if (!made.ok) throw new Error(String(made.error));
93 const repo: Repo = made.value;
94 const seeded = await commitToBranch(app, alice, repo, {
95 message: "Start",
96 files: [
97 { path: "forge.json", content: JSON.stringify({ check: { run: "true", fast: "true", full: "sh gate.sh" } }) },
98 { path: "gate.sh", content: gateLog(error) },
99 { path: "t/a.test.ts", content: `import { describe, expect, test } from "bun:test";\ndescribe("grp", () => { test("page", () => { ${body} }); });\n` },
100 ],
101 });
102 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
103 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
104 const git = (...args: string[]) => Bun.spawnSync(["git", "-C", dir, ...args]).stdout.toString().trim();
105 const handedBack: number[] = [];
106 const entries: Queued[] = [];
107 for (const n of [1, 2]) {
108 const branched = await commitToBranch(app, alice, repo, {
109 branch: `f${n}`,
110 message: `Add f${n}`,
111 files: [{ path: `f${n}.txt`, content: `${n}\n` }],
112 });
113 if (!branched.ok) throw new Error(JSON.stringify(branched.error));
114 const opened = await openPull(app, alice, repo, { title: `F${n}`, baseBranch: "main", headBranch: `f${n}` });
115 if (!opened.ok) throw new Error(String(opened.error));
116 const pull = getPull(app.db, repo.id, opened.value.number)!;
117 entries.push({
118 pull,
119 dir,
120 land: async (want: Expect) => {
121 if (git("rev-parse", "refs/heads/main") !== want.baseOid) return undefined;
122 const tip = git("-c", "user.name=t", "-c", "user.email=t@t", "commit-tree", want.treeOid, "-p", want.baseOid, "-m", `#${pull.number}`);
123 git("update-ref", "refs/heads/main", tip);
124 return tip;
125 },
126 release: () => {},
127 handBack: () => handedBack.push(pull.number),
128 });
129 }
130 const gates: number[][] = [];
131 const gate = (c: Candidate, signal: AbortSignal) => {
132 gates.push(c.steps.map((s) => s.entry.pull.number));
133 return gateCandidate(app, repo, c.steps[0]!.entry.pull, c.dir, c.tipOid, signal);
134 };
135 const landed = await enqueue(repo.id, entries, gate, { depth: 1, room: () => 0 });
136 return { repo, landed, gates, handedBack, numbers: entries.map((e) => e.pull.number) };
137 }
138
139 test("a batch red only on a budget test that passes on the re-run lands whole, and the test is a flake", async () => {
140 const { repo, landed, gates, handedBack, numbers } = await queueWith("budget-green", BUDGET, "expect(1).toBe(1);");
141 expect(landed).toEqual([true, true]);
142 expect(gates).toEqual([numbers]);
143 expect(handedBack).toEqual([]);
144 expect(budgetRetries(repo.id)).toEqual({ passed: 1, failed: 0 });
145 expect(flakeList(app.db, Date.now()).find((f) => f.repo === "alice/budget-green")).toMatchObject({
146 test: "grp > page",
147 file: "t/a.test.ts",
148 count: 1,
149 });
150 }, 90_000);
151
152 test("a budget test red on the re-run too keeps the batch red, and the search goes on", async () => {
153 const { repo, landed, gates, handedBack, numbers } = await queueWith("budget-red", BUDGET, "expect(1).toBe(2);");
154 expect(landed).toEqual([false, false]);
155 expect(gates[0]).toEqual(numbers);
156 expect(gates.length).toBeGreaterThan(1);
157 expect(handedBack.length).toBeGreaterThan(0);
158 // Each gate re-ran once and failed; none passed.
159 expect(budgetRetries(repo.id)).toEqual({ passed: 0, failed: gates.length });
160 expect(flakeList(app.db, Date.now()).some((f) => f.repo === "alice/budget-red")).toBe(false);
161 }, 90_000);
162
163 test("a functional red is searched at once, never re-run, though its file would pass", async () => {
164 const { repo, landed, gates, handedBack, numbers } = await queueWith("functional", FUNCTIONAL, "expect(1).toBe(1);");
165 expect(landed).toEqual([false, false]);
166 // The batch, then its front half at once: bisected, not gated again whole.
167 expect(gates.slice(0, 2)).toEqual([numbers, [numbers[0]!]]);
168 expect(handedBack.length).toBeGreaterThan(0);
169 expect(budgetRetries(repo.id)).toEqual({ passed: 0, failed: 0 });
170 expect(flakeList(app.db, Date.now()).some((f) => f.repo === "alice/functional")).toBe(false);
171 }, 90_000);
172 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs