trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns12g rEventsFlows

Timing tests stop failing because the machine is busy #7822

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/7789-timing-tests-stop-failing-because-the-ma → main 4 commits fork 1 thread · 1 unresolved

  • Merged. torque/7789-timing-tests-stop-failing-because-the-ma was merged into main as 626bb21.

Review

Sign in to review this change.
16 files changed +218 −605
.forgejo/workflows/timing.yml added +39
@@ -0,0 +1,39 @@
1 name: timing
2 # Nightly: the wall-clock budgets (#7789). No check the forge runs, a pull's or
3 # the merge queue's gate, enforces them: it only warns of a time
4 # over its budget (test/helpers/budget.ts); here, one test file at a time on an
5 # otherwise idle runner, each budget is held to its absolute ms. A newly red
6 # file opens one `Red: test/….test.ts` issue and a green one closes it, as
7 # outside-gate does (scripts/flake-hunt.ts timing). Needs secret FORGE_TOKEN and
8 # variable FORGE_URL.
9 on:
10 schedule:
11 - cron: "23 4 * * *"
12 workflow_dispatch: {}
13
14 jobs:
15 timing:
16 runs-on: isolated
17 timeout-minutes: 30
18 env:
19 REGISTRY: forgejo-http.forgejo.svc.cluster.local:3000
20 FORGE_TEST_TIMING: "1"
21 FORGE_TEST_QUIET: "1"
22 steps:
23 - name: clone main
24 env:
25 TOKEN: ${{ github.token }}
26 run: |
27 set -eu
28 git clone -q "http://x-access-token:$TOKEN@$REGISTRY/$GITHUB_REPOSITORY.git" src
29 cd src
30 bun install --frozen-lockfile
31
32 - name: budget tests, serially
33 working-directory: src
34 env:
35 FORGE_API: ${{ vars.FORGE_URL }}/api/v1
36 FORGE_REPO: ${{ github.repository }}
37 FORGE_TOKEN: ${{ secrets.FORGE_TOKEN }}
38 FORGE_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_number }}
39 run: bun scripts/flake-hunt.ts timing --file-issues
CLAUDE.md +5 −2
@@ -108,8 +108,11 @@ bun run src/main.ts # boots on :3000, data in ./data
108108 timeout near the real duration, a wall-clock bound under 200 ms on a
109109 request or a spawn (`test/flake.test.ts` refuses one). A speed is bounded only
110110 with `expectBudget` (`test/helpers/budget.ts`): relative to a same-run
111 baseline, absolute only when `FORGE_TEST_QUIET=1` (set by hand on a quiet machine; no
112 lane sets it yet, so absolute budgets are not enforced anywhere);
111 baseline. No check the forge runs fails on it, a pull's or the merge queue's
112 gate: a time over budget is only warned. It is enforced only with
113 `FORGE_TEST_TIMING=1`, by the nightly `timing` workflow (one file at a time,
114 `FORGE_TEST_QUIET=1`, `Red:` issues). To fail a regression in a pull, count work
115 (statements, bytes, phases) or use a fake clock (#7789);
113116 `test/flake.test.ts` refuses a raw `under N ms` title or bound.
114117 - **Every non-trivial change ships with a test.** Prefer protocol-level tests
115118 that boot a real app (`createApp` + `serve`) over unit mocks. Adversarial
docs/features/gate-budget-retry.md deleted −35
@@ -1,35 +0,0 @@
1 # Budget reds in a merge-queue gate (#7354)
2
3 A budget test (`expectBudget`, `test/helpers/budget.ts`) can fail only under the
4 gate's own load, while main and the pull pass it alone. When a red merge-queue
5 gate is red **only** on budget tests, the gate re-runs those test files once,
6 serially, before the batch counts as red (`src/checks/budget_retry.ts`, wired
7 in `checks/service.ts`).
8
9 - **Which reds count.** `expectBudget` throws `over budget (expectBudget): …`
10 when a time is over its limit, and bun prints a test's error just above its
11 `(fail)` line. A gate is a budget red when every `(fail)` comes after that
12 error, each has a file, the fail counts add up, and the log has no error
13 outside a test, no shard that named no test, and no skipped shards. At most
14 four files. Anything else is a functional red, searched at once with no re-run.
15 - **The re-run.** `bun test --max-concurrency=1 './file' …` in the same checkout
16 and cell, with `FORGE_TEST_QUIET=1`, so each budget is held to its absolute ms.
17 It passes: the gate is **green**, the batch lands, and each test is a row in
18 `check_flake` (the flake list; twice in a week files an issue, as
19 [check-flake](check-flake.md)). It fails: **red, as before**.
20 - **Only the test files re-run.** A green re-run turns the whole check green,
21 as main's flake re-run does. That is sound for this forge: `check.run` ends
22 with `test-shards`. A repository whose gate runs more steps after its tests
23 (`… && test-shards … && more`) has those steps skipped on a budget red, and a
24 green re-run lands the batch without them ever running.
25 - **The shards run on.** `scripts/test-shards.ts` stops the other shards at the
26 first red (fail-fast), except a shard red only on budget tests: the gate can
27 only be called green when every other file ran too.
28 - **The tick line** says `budgetRetries="P passed/F failed"` for the last hour's
29 re-runs when there were any. Re-runs that keep failing mean a budget is truly
30 regressed.
31
32 Left out: the main and pull checks (main has its own flake re-run), cluster
33 Jobs (the re-run needs the check's own cell), and a `budget` tag in a test's
34 name. Only `expectBudget`'s own error counts, so a raw `toBeLessThan` or a test
35 that only says "budget" in its name is never re-run.
scripts/flake-hunt.ts +16 −8
@@ -15,6 +15,10 @@
1515 * `--file-issues` a newly red file opens one `Red: test/….test.ts`
1616 * issue naming its first failing test, a file red on another test
1717 * updates it, and a file green again closes it.
18 * bun scripts/flake-hunt.ts timing [--file-issues]
19 * The files that call `expectBudget`, one at a time, as `outside` does;
20 * the scheduled run sets FORGE_TEST_TIMING=1 FORGE_TEST_QUIET=1 so each
21 * budget is held to its absolute ms (#7789).
1822 * bun scripts/flake-hunt.ts quarantine
1923 * Puts every file an open `Flaky: test/….test.ts` issue the hunt filed
2024 * (author is the token's user, not untrusted) names into
@@ -39,6 +43,8 @@ import {
3943 redActions,
4044 redFile,
4145 redIssueFor,
46 type RedRun,
47 timingFiles,
4248 type OpenRed,
4349 quarantineFlaky,
4450 readRun,
@@ -235,9 +241,10 @@ async function openRedIssues(): Promise<Map<string, OpenRed>> {
235241 return found;
236242 }
237243
238 /** The files outside the gate, one at a time; red ones reported to the forge. */
239 async function outside(): Promise<number> {
240 const files = outsideFiles(presentTests());
244 /** The files outside the gate (or, for `timing`, the budget files), one at a time; red ones reported to the forge. */
245 async function scheduled(kind: RedRun = "outside"): Promise<number> {
246 const present = presentTests();
247 const files = kind === "timing" ? timingFiles(present, (file) => readFileSync(file, "utf8")) : outsideFiles(present);
241248 const fileIssues = rest.includes("--file-issues");
242249 const open = fileIssues ? await openRedIssues() : new Map<string, OpenRed>();
243250 const runs: FileRun[] = [];
@@ -250,12 +257,12 @@ async function outside(): Promise<number> {
250257 for (const action of redActions(runs, open)) {
251258 if (!fileIssues) continue;
252259 if (action.kind === "open") {
253 const res = await forgeJson("POST", "/issues", redIssueFor(action.run, where));
260 const res = await forgeJson("POST", "/issues", redIssueFor(action.run, where, kind));
254261 const made = (await res.json().catch(() => ({}))) as { number?: unknown };
255262 if (!res.ok) fail(`filing the issue for ${action.run.file}: ${res.status}`);
256263 console.log(` filed #${made.number} for ${action.run.file}`);
257264 } else if (action.kind === "update") {
258 const { body } = redIssueFor(action.run, where);
265 const { body } = redIssueFor(action.run, where, kind);
259266 const res = await forgeJson("PATCH", `/issues/${action.number}`, { body });
260267 if (!res.ok) {
261268 const note = await forgeJson("POST", `/issues/${action.number}/comments`, { body });
@@ -272,7 +279,7 @@ async function outside(): Promise<number> {
272279 }
273280 }
274281 const red = runs.filter((run) => !run.green).length;
275 console.log(`${files.length} files outside the gate, ${red} red`);
282 console.log(`${files.length} files ${kind === "timing" ? "holding wall-clock budgets" : "outside the gate"}, ${red} red`);
276283 return 0;
277284 }
278285
@@ -368,11 +375,12 @@ async function admit(): Promise<number> {
368375 return admitted(results) ? 0 : 1;
369376 }
370377
371 if (mode === "outside") process.exit(await outside());
378 if (mode === "outside") process.exit(await scheduled());
379 else if (mode === "timing") process.exit(await scheduled("timing"));
372380 else if (mode === "hunt") process.exit(await hunt());
373381 else if (mode === "quarantine") process.exit(await quarantineAll());
374382 else if (mode === "admit") process.exit(await admit());
375383 else
376384 fail(
377 "usage: flake-hunt.ts hunt [--runs N] [--shard N/M] [--file-issues] | outside [--file-issues] | quarantine | admit test/x.test.ts [--runs N]",
385 "usage: flake-hunt.ts hunt [--runs N] [--shard N/M] [--file-issues] | outside [--file-issues] | timing [--file-issues] | quarantine | admit test/x.test.ts [--runs N]",
378386 );
scripts/flake.ts +35 −3
@@ -357,20 +357,52 @@ export function firstFailing(run: FileRun): string {
357357 : "none named; the file failed outside a test";
358358 }
359359
360 /** The issue for a file outside the gate that is red. */
360 /** Files that test the budget helper itself, not a speed: never in the timing run. */
361 export const TIMING_EXCLUDED: readonly string[] = ["test/budget.test.ts"];
362
363 /** How a file says it bounds a wall-clock speed: it imports the budget helper and calls it. */
364 const BUDGET_IMPORT = /from\s+["']\.\/helpers\/budget\.ts["']/;
365 const BUDGET_CALL = /\bexpectBudget\(/;
366
367 /**
368 * The files the nightly timing run holds to their wall-clock budgets (#7789):
369 * each present test that imports `test/helpers/budget.ts` and calls
370 * `expectBudget`, less {@link TIMING_EXCLUDED}. A file that only mentions the
371 * name, in a string or a comment, is not one.
372 */
373 export function timingFiles(
374 present: readonly string[],
375 read: (file: string) => string,
376 ): string[] {
377 return present.filter((file) => {
378 if (TIMING_EXCLUDED.includes(file)) return false;
379 const text = read(file);
380 return BUDGET_IMPORT.test(text) && BUDGET_CALL.test(text);
381 });
382 }
383
384 /** Which scheduled run found a file red: the files outside the gate, or the wall-clock budgets. */
385 export type RedRun = "outside" | "timing";
386
387 /** The issue for a file the scheduled run found red. */
361388 export function redIssueFor(
362389 run: FileRun,
363390 where: string,
391 kind: RedRun = "outside",
364392 ): { title: string; body: string } {
365393 const fence = "```";
366394 return {
367395 title: `${RED_TITLE}${run.file}`,
368396 body: [
369 `\`${run.file}\` is outside the gate and was red in the scheduled run. This issue closes itself when the file passes again.`,
397 kind === "timing"
398 ? `\`${run.file}\` was over a wall-clock budget, or red, in the nightly timing run (one file at a time, absolute budgets). This issue closes itself when the file passes again.`
399 : `\`${run.file}\` is outside the gate and was red in the scheduled run. This issue closes itself when the file passes again.`,
370400 "",
371401 `- first failing test: ${firstFailing(run)}`,
372402 `- run: ${where || "(a local run)"}`,
373 `- reproduce: \`bun test --timeout 60000 ./${run.file}\``,
403 kind === "timing"
404 ? `- reproduce: \`FORGE_TEST_TIMING=1 FORGE_TEST_QUIET=1 bun test --timeout 60000 ./${run.file}\``
405 : `- reproduce: \`bun test --timeout 60000 ./${run.file}\``,
374406 "",
375407 fence,
376408 run.text || "(no failure text captured)",
scripts/shards.ts +2 −5
@@ -8,7 +8,6 @@
88 */
99
1010 import { nameUnnamed } from "../src/checks/bun-test.ts";
11 import { budgetReds } from "../src/checks/budget_retry.ts";
1211
1312 /**
1413 * No more than this many `bun test` processes at once, whatever the host:
@@ -370,12 +369,10 @@ export interface ShardResult {
370369
371370 /**
372371 * Whether a finished shard stops the shards not yet done (fail-fast): any
373 * red does, except one whose every failure is a budget test (#7354). The
374 * forge re-runs those once, serially, after the gate — which it can only
375 * call green when every other file ran too.
372 * red does. A wall-clock budget cannot be one in a check (#7789).
376373 */
377374 export function stopsTheRest(result: Pick<ShardResult, "code" | "output">): boolean {
378 return result.code !== 0 && budgetReds(result.output.split("\n")) === undefined;
375 return result.code !== 0;
379376 }
380377
381378 /** What the log ends with, and the exit code of the whole run. */
scripts/test-shards.ts +2 −4
@@ -24,8 +24,7 @@
2424 * finishes, so the log reads one shard at a time. Either way the tail
2525 * names every failing test from every shard. The first red shard stops
2626 * the rest (fail-fast) so a known fail does not burn the remaining wall
27 * clock — unless every red in it is a budget test, which the forge re-runs
28 * once after the gate (#7354). Exits non-zero if any shard does.
27 * clock. Exits non-zero if any shard does.
2928 *
3029 * `--keep-going` is the verification after landing (#7786): no shard stops
3130 * the rest, a file red on its own is run once more alone (green then, it
@@ -607,8 +606,7 @@ async function pool(width: number): Promise<Timed[]> {
607606 const index = next++;
608607 if (stop) break;
609608 results[index] = await runShard(slices[index]!, index, width === 1);
610 // A shard red only on budget tests lets the rest run (#7354);
611 // a verification after landing runs everything (#7786).
609 // A verification after landing runs everything (#7786).
612610 if (!keepGoing && stopsTheRest(results[index]!)) {
613611 stop = true;
614612 halted = true;
src/checks/budget_retry.ts deleted −149
@@ -1,149 +0,0 @@
1 /**
2 * A merge-queue gate whose only reds are budget tests re-runs those test
3 * files once, serially, before the batch counts as red (#7354). A budget
4 * test fails only under the gate's own load — eight pulls' worth of shards
5 * beside it — and passes on main and on the pull alone; a red one cost a
6 * batch of eight and thirteen minutes. Passing the re-run, the batch is
7 * green and the tests are recorded as flakes (`flake.ts`); failing it, the
8 * batch is red as before. A functional red is never re-run: it is searched
9 * at once.
10 *
11 * A budget red is one `expectBudget` (`test/helpers/budget.ts`) threw: its
12 * error starts with {@link BUDGET_MARK}, and bun prints a test's error just
13 * above its `(fail)` line. Plain lines in, files and a command out. The
14 * names and files come from the repository's own output, so each is checked
15 * before it reaches a shell, and quoted there.
16 */
17
18 import { nameUnnamed } from "./bun-test.ts";
19 import type { Flaky } from "./flake.ts";
20
21 /** What `expectBudget`'s error starts with: the gate reads it to know a budget red. */
22 export const BUDGET_MARK = "over budget (expectBudget)";
23
24 const MARKED = /^error: over budget \(expectBudget\): /;
25 const HEADER = /^(\S+\.test\.[cm]?[jt]sx?):$/;
26 const FAIL = /^\(fail\) (.+?)(?: \[[0-9.]+m?s\])?( \[shard [0-9]+\])?$/;
27 const RESULT = /^\((?:pass|fail|skip|todo)\) /;
28 const FAIL_COUNT = /^\s*(\d+) fail$/;
29 /** Lines that say a red was not (only) a failed test: an error outside one, a shard that named none, shards never run. */
30 const NOT_A_TEST = [
31 /^# Unhandled error/,
32 /^\s*[1-9][0-9]* errors?\s*$/,
33 /^shard [0-9]+ (?:killed after|exited [0-9]+ without naming)/,
34 /^test-shards: fail-fast/,
35 ];
36 const PATH = /^[\w@./+-]{1,200}$/;
37 // eslint-disable-next-line no-control-regex
38 const CONTROL = /[\u0000-\u001f\u007f-\u009f]/;
39 const NAME_MAX = 300;
40
41 /** The most files one re-run takes: more budget reds than this is a slow change, not a loaded gate. */
42 export const MAX_BUDGET_FILES = 4;
43
44 /**
45 * The tests of a red check, each with its file, when every red in it is a
46 * budget red; undefined otherwise — a functional failure, a failure outside
47 * any test, a test bun did not name, a shard skipped, or no test failed.
48 * The sharded gate echoes each `(fail)` line with ` [shard N]` at its end:
49 * an echo names a test again, it is not a second failure.
50 */
51 export function budgetReds(lines: readonly string[]): Flaky[] | undefined {
52 const named = nameUnnamed(lines);
53 if (named.some((line) => NOT_A_TEST.some((re) => re.test(line)))) return undefined;
54 const found = new Map<string, Flaky>();
55 const echoed = new Set<string>();
56 let file: string | undefined;
57 let marked = false;
58 let failures = 0;
59 let counted = 0;
60 for (const line of named) {
61 const header = HEADER.exec(line);
62 if (header) {
63 file = header[1];
64 marked = false;
65 continue;
66 }
67 if (MARKED.test(line)) marked = true;
68 const count = FAIL_COUNT.exec(line);
69 if (count) counted += Number(count[1]);
70 const fail = FAIL.exec(line);
71 if (fail) {
72 const name = fail[1]!;
73 if (fail[2] !== undefined) echoed.add(name);
74 else {
75 failures++;
76 if (!marked || file === undefined) return undefined;
77 if (!PATH.test(file) || name.length > NAME_MAX || CONTROL.test(name)) return undefined;
78 found.set(`${file}\0${name}`, { file, name });
79 }
80 }
81 if (RESULT.test(line)) marked = false;
82 }
83 if (failures === 0 || failures !== counted) return undefined;
84 const names = new Set([...found.values()].map((t) => t.name));
85 for (const name of echoed) if (!names.has(name)) return undefined;
86 const tests = [...found.values()];
87 if (budgetFiles(tests).length > MAX_BUDGET_FILES) return undefined;
88 return tests;
89 }
90
91 /** The files the tests are in, each once, in the order first named. */
92 export function budgetFiles(tests: readonly Flaky[]): string[] {
93 return [...new Set(tests.map((t) => t.file))];
94 }
95
96 const quote = (s: string): string => `'${s.replace(/'/g, `'\\''`)}'`;
97
98 /**
99 * `bun test` of exactly these files (a `./` path, never a filter that also
100 * matches a sibling), one test at a time. The caller sets `FORGE_TEST_QUIET=1`
101 * in its environment: the machine is the gate's alone again by then.
102 */
103 export function budgetRetryCommand(files: readonly string[]): string {
104 const paths = files.map((f) => (f.startsWith("/") || f.startsWith("./") ? f : `./${f}`));
105 return `bun test --max-concurrency=1 ${paths.map(quote).join(" ")}`;
106 }
107
108 /** Whether the re-run's own output says tests ran and none failed. */
109 export function retryPassed(lines: readonly string[]): boolean {
110 let pass = 0;
111 let fail = 0;
112 for (const line of lines) {
113 const p = /^\s*(\d+) pass$/.exec(line);
114 if (p) pass += Number(p[1]);
115 const f = FAIL_COUNT.exec(line);
116 if (f) fail += Number(f[1]);
117 }
118 return pass > 0 && fail === 0;
119 }
120
121 const HOUR_MS = 60 * 60_000;
122
123 /** Each repository's re-runs in the last hour: when, and whether it passed. */
124 const retries = new Map<string, { at: number; passed: boolean }[]>();
125
126 /** A gate re-ran its budget reds: passed or not. */
127 export function noteBudgetRetry(repoId: string, passed: boolean, now = Date.now()): void {
128 const kept = (retries.get(repoId) ?? []).filter((r) => now - r.at < HOUR_MS);
129 kept.push({ at: now, passed });
130 retries.set(repoId, kept);
131 }
132
133 /** The last hour's budget re-runs of a repository's gates. */
134 export function budgetRetries(repoId: string, now = Date.now()): { passed: number; failed: number } {
135 const kept = (retries.get(repoId) ?? []).filter((r) => now - r.at < HOUR_MS);
136 retries.set(repoId, kept);
137 const passed = kept.filter((r) => r.passed).length;
138 return { passed, failed: kept.length - passed };
139 }
140
141 /**
142 * The tick line's words for the last hour's budget re-runs:
143 * `2 passed/1 failed`. Undefined when there were none: a budget that
144 * keeps failing its re-run is truly regressed, and the steward sees it here.
145 */
146 export function budgetRetryWords(repoId: string, now = Date.now()): string | undefined {
147 const { passed, failed } = budgetRetries(repoId, now);
148 return passed + failed === 0 ? undefined : `${passed} passed/${failed} failed`;
149 }
src/checks/service.ts −38
@@ -58,7 +58,6 @@ import { keepCache, restoreCache } from "./cache.ts";
5858 import { MAX_FILED, clearedFiles, fileOrderDependent } from "./order_dependent.ts";
5959 import { parseBunTestFailure } from "./bun-test.ts";
6060 import { passedAlone, recordFlakes, rerunCommand, rerunnable, pullOwnsRed } from "./flake.ts";
61 import { budgetFiles, budgetReds, budgetRetryCommand, noteBudgetRetry, retryPassed } from "./budget_retry.ts";
6261 import { diskRecovered, withDisk } from "./disk_red.ts";
6362 import { ASKED, redMain, unheld } from "./red_main.ts";
6463 import { proposeRevert } from "./revert_red.ts";
@@ -938,8 +937,6 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
938937 readonly fileOrderDependent?: boolean;
939938 /** The default branch's own run: a failure of one or two tests is re-run alone before it counts as red. */
940939 readonly isMain?: boolean;
941 /** A merge-queue gate: a red only on budget tests re-runs those files once, serially (#7354). */
942 readonly gate?: boolean;
943940 readonly write: (line: string) => void;
944941 readonly signal: AbortSignal;
945942 }
@@ -1351,40 +1348,6 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
13511348 write(`flake re-run failed: ${String(error)}`);
13521349 }
13531350 }
1354 // A gate's red only on budget tests is the gate's own load, not the
1355 // batch's code (#7354): those files run once more, one test at a
1356 // time, held to their absolute budgets. Passing, the batch is green
1357 // and the tests are recorded as flakes; failing, it is red as before.
1358 if (!ran.ok && job === undefined && x.gate === true) {
1359 try {
1360 const tests = budgetReds(checkLines);
1361 if (tests !== undefined) {
1362 const files = budgetFiles(tests);
1363 write(`re-running ${files.length} file(s) red only on budget tests, serially: ${files.join(" ")}`);
1364 const out: string[] = [];
1365 const again = await runStep("budget", budgetRetryCommand(files), {
1366 cwd: dir,
1367 env: { ...env, FORGE_TEST_QUIET: "1" },
1368 log: (line) => {
1369 out.push(line);
1370 write(line);
1371 },
1372 signal,
1373 timeoutMs: Math.min(manifest.timeout, RERUN_TIMEOUT_S) * 1000,
1374 cell,
1375 });
1376 const passed = again.ok && retryPassed(out);
1377 if (!signal.aborted) noteBudgetRetry(repo.id, passed);
1378 if (passed) {
1379 recordFlakes(app.db, app.bus, repo, tests, { runId: run.id, commit: run.commitOid });
1380 write(`budget retry passed: ${tests.map((t) => `"${t.name}"`).join(", ")} held their budgets alone; recorded, the gate is green`);
1381 ran = { ok: true, value: undefined };
1382 } else write("budget retry failed: the gate is red");
1383 }
1384 } catch (error) {
1385 write(`budget retry failed to run: ${String(error)}`);
1386 }
1387 }
13881351 if (!ran.ok) {
13891352 const { summary, detail } = summariseCheckFailure(
13901353 checkLines,
@@ -1648,7 +1611,6 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
16481611 commitOid,
16491612 pull,
16501613 tier: "full",
1651 gate: true,
16521614 write: (line) => lines.push(line),
16531615 signal: controller.signal,
16541616 });
src/crew/autonomy.ts −9
@@ -135,7 +135,6 @@ import { approvedOnHead, finishApproved, finishFields } from "./finishing.ts";
135135 import { recheckRedApproved } from "./red_approved.ts";
136136 import { keepBatch } from "../flow/red_batches.ts";
137137 import { recordQueueFlakes } from "../checks/flake.ts";
138 import { budgetRetryWords } from "../checks/budget_retry.ts";
139138 import { readVitals, vitalsMisses, vitalsWords } from "../flow/vitals.ts";
140139 import { moveRequested, tallyRequested, tallyWords } from "./requested.ts";
141140 import { reviewersOf, sendToReview, sweepReviews } from "./review.ts";
@@ -1392,8 +1391,6 @@ export async function autonomyTick(
13921391 batch: batchWords(current.id, queueBatch(app.db)),
13931392 // Pulls the queue found red on top of others of their candidate (#4235).
13941393 clashes: queueClashes(current.id),
1395 // The last hour's gates re-run red only on budget tests, passed/failed (#7354).
1396 ...budgetRetriesField(current.id, now.getTime()),
13971394 spent: `${status.spentTodayUsd.toFixed(2)}/${status.ceilingUsd}`,
13981395 ...(status.paused === null ? {} : { paused: status.paused }),
13991396 // Starts after the first, and how many followed a crash (#3708).
@@ -2244,12 +2241,6 @@ function queueOptions(app: App, repo: Repo): QueueOptions {
22442241 };
22452242 }
22462243
2247 /** `budgetRetries: "2 passed/1 failed"` on the tick line when a gate re-ran its budget reds this hour; nothing otherwise. */
2248 function budgetRetriesField(repoId: string, now: number): { budgetRetries?: string } {
2249 const words = budgetRetryWords(repoId, now);
2250 return words === undefined ? {} : { budgetRetries: words };
2251 }
2252
22532244 /** The queue's gate: the full tier on the candidate's tip, asked as its first pull. */
22542245 function gateFor(app: App, repo: Repo): GateCandidate {
22552246 return (candidate, signal) =>
test/budget-retry.test.ts deleted −171
@@ -1,171 +0,0 @@
1 /**
2 * The gate tells a budget red from a functional one (#7354): every red an
3 * `expectBudget` threw is re-run; anything else in the log keeps it red.
4 */
5 import { afterAll, beforeAll, expect, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import {
10 BUDGET_MARK,
11 budgetFiles,
12 budgetReds,
13 budgetRetries,
14 budgetRetryCommand,
15 budgetRetryWords,
16 noteBudgetRetry,
17 retryPassed,
18 } from "../src/checks/budget_retry.ts";
19 import { stopsTheRest } from "../scripts/shards.ts";
20
21 const budgetError = `error: ${BUDGET_MARK}: 81.0 ms is not under 50.0 ms (budget 50 ms, baseline 2.0 ms)`;
22
23 /** One one-file shard as test-shards writes it: bun's report under its header. */
24 const shard = (n: number, file: string, fails: { name: string; error: string }[], passes = 1): string[] => [
25 `── shard ${n}/3 (exit ${fails.length > 0 ? 1 : 0}, 2.0s): ${file}`,
26 "bun test v1.3.14",
27 "",
28 `${file}:`,
29 ...Array.from({ length: passes }, (_, i) => `(pass) ok ${i} [1.00ms]`),
30 ...fails.flatMap((f) => ["12 | expectBudget(dur, { ms: 50, against });", " ^", f.error, " at <anonymous> (/x/y.ts:12:3)", `(fail) ${f.name} [81.00ms]`]),
31 "",
32 ` ${passes} pass`,
33 ` ${fails.length} fail`,
34 `Ran ${passes + fails.length} tests across 1 file. [90.00ms]`,
35 ];
36
37 const tail = (names: string[]): string[] => [
38 "",
39 `${names.length === 0 ? 0 : 1} of 3 shards failed:`,
40 ...names.map((n) => `(fail) ${n} [81.00ms] [shard 1]`),
41 `Ran 9 tests across 3 files in 4.0s; ${names.length} failed.`,
42 ];
43
44 test("a gate red only on budget tests names each with its file, echoes and all", () => {
45 const lines = [
46 ...shard(1, "test/branches-many.test.ts", [{ name: "200 branches hold the 50 ms budget", error: budgetError }]),
47 ...shard(2, "test/map-seed-timing.test.ts", [{ name: "warm > the seed's map answers within its budget", error: budgetError }]),
48 ...shard(3, "test/quiet.test.ts", []),
49 ...tail(["200 branches hold the 50 ms budget", "warm > the seed's map answers within its budget"]),
50 ];
51 const tests = budgetReds(lines)!;
52 expect(tests).toEqual([
53 { file: "test/branches-many.test.ts", name: "200 branches hold the 50 ms budget" },
54 { file: "test/map-seed-timing.test.ts", name: "warm > the seed's map answers within its budget" },
55 ]);
56 expect(budgetFiles(tests)).toEqual(["test/branches-many.test.ts", "test/map-seed-timing.test.ts"]);
57 });
58
59 test("one functional red beside a budget red keeps the gate red: nothing is re-run", () => {
60 const functional = "error: expect(received).toBe(expected)";
61 const lines = [
62 ...shard(1, "test/a.test.ts", [{ name: "slow", error: budgetError }]),
63 ...shard(2, "test/b.test.ts", [{ name: "wrong", error: functional }]),
64 ...tail(["slow", "wrong"]),
65 ];
66 expect(budgetReds(lines)).toBeUndefined();
67 expect(budgetReds([...shard(1, "test/b.test.ts", [{ name: "wrong", error: functional }]), ...tail(["wrong"])])).toBeUndefined();
68 });
69
70 test("a budget error printed for one test does not cover the next test's failure", () => {
71 const lines = [
72 "test/a.test.ts:",
73 budgetError,
74 "(fail) slow [81.00ms]",
75 "error: expect(received).toBe(expected)",
76 "(fail) wrong [1.00ms]",
77 " 0 pass",
78 " 2 fail",
79 ];
80 expect(budgetReds(lines)).toBeUndefined();
81 });
82
83 test("a red that is not only failed tests is never a budget red", () => {
84 const budget = shard(1, "test/a.test.ts", [{ name: "slow", error: budgetError }]);
85 expect(budgetReds(budget)).toHaveLength(1);
86 // An error between tests, a shard that named none, shards skipped, a hook.
87 expect(budgetReds([...budget, "# Unhandled error between tests"])).toBeUndefined();
88 expect(budgetReds([...budget, " 1 error"])).toBeUndefined();
89 expect(budgetReds([...budget, "shard 2 exited 1 without naming a test: test/b.test.ts"])).toBeUndefined();
90 expect(budgetReds([...budget, "test-shards: fail-fast — skipped 4 remaining shards after a red exit"])).toBeUndefined();
91 expect(budgetReds(["test/a.test.ts:", "(fail) (unnamed) [1.00ms]", " ^ a beforeEach hook timed out", " 0 pass", " 1 fail"])).toBeUndefined();
92 // A shard that crashed after its budget red never counted it: not every red was counted.
93 expect(budgetReds(budget.filter((l) => l !== " 1 fail"))).toBeUndefined();
94 // An echo of a test no shard showed failing with a budget error.
95 expect(budgetReds([...budget, "(fail) other [1.00ms] [shard 2]"])).toBeUndefined();
96 // Type errors, no tests at all, a green log.
97 expect(budgetReds(["src/a.ts(1,1): error TS2322"])).toBeUndefined();
98 expect(budgetReds(shard(1, "test/a.test.ts", []))).toBeUndefined();
99 });
100
101 test("a file or name that could become shell is never re-run", () => {
102 expect(budgetReds(shard(1, "test/$(rm -rf).test.ts", [{ name: "slow", error: budgetError }]))).toBeUndefined();
103 expect(budgetReds(shard(1, "test/a.test.ts", [{ name: "slow\u0007", error: budgetError }]))).toBeUndefined();
104 });
105
106 test("more budget files than a loaded gate explains is a slow change: red", () => {
107 const lines = [1, 2, 3, 4, 5].flatMap((n) => shard(n, `test/t${n}.test.ts`, [{ name: `slow ${n}`, error: budgetError }]));
108 expect(budgetReds(lines)).toBeUndefined();
109 expect(budgetReds(lines.slice(0, lines.length * (4 / 5)))).toHaveLength(4);
110 });
111
112 test("the re-run is exactly those files, quoted, one test at a time", () => {
113 expect(budgetRetryCommand(["test/a.test.ts", "test/it's.test.ts"])).toBe(
114 `bun test --max-concurrency=1 './test/a.test.ts' './test/it'\\''s.test.ts'`,
115 );
116 });
117
118 test("a re-run passes only when tests ran and none failed", () => {
119 expect(retryPassed([" 7 pass", " 0 fail"])).toBe(true);
120 expect(retryPassed([" 6 pass", " 1 fail"])).toBe(false);
121 expect(retryPassed([" 0 pass", " 0 fail"])).toBe(false);
122 expect(retryPassed(["error: module not found"])).toBe(false);
123 });
124
125 test("a fail-fast pool lets a budget-only red shard run on, and stops at any other", () => {
126 const budget = shard(1, "test/a.test.ts", [{ name: "slow", error: budgetError }]).join("\n");
127 const functional = shard(1, "test/a.test.ts", [{ name: "wrong", error: "error: expect(received).toBe(expected)" }]).join("\n");
128 expect(stopsTheRest({ code: 1, output: budget })).toBe(false);
129 expect(stopsTheRest({ code: 1, output: functional })).toBe(true);
130 expect(stopsTheRest({ code: 137, output: "" })).toBe(true);
131 expect(stopsTheRest({ code: 0, output: "" })).toBe(false);
132 });
133
134 test("the tick line counts the last hour's re-runs, passed and failed", () => {
135 const repo = "budget-words";
136 const t = 1_800_000_000_000;
137 expect(budgetRetryWords(repo, t)).toBeUndefined();
138 noteBudgetRetry(repo, true, t);
139 noteBudgetRetry(repo, true, t + 1_000);
140 noteBudgetRetry(repo, false, t + 2_000);
141 expect(budgetRetries(repo, t + 3_000)).toEqual({ passed: 2, failed: 1 });
142 expect(budgetRetryWords(repo, t + 3_000)).toBe("2 passed/1 failed");
143 // An hour on, the first two are gone.
144 expect(budgetRetryWords(repo, t + 3_601_500)).toBe("0 passed/1 failed");
145 expect(budgetRetryWords(repo, t + 7_200_000)).toBeUndefined();
146 });
147
148 let dir: string;
149 beforeAll(async () => {
150 dir = await mkdtemp(join(tmpdir(), "budget-retry-"));
151 });
152 afterAll(async () => {
153 await rm(dir, { recursive: true, force: true });
154 });
155
156 test("what a real expectBudget prints when it fails is read as a budget red; a plain expect is not", async () => {
157 const helper = join(import.meta.dir, "helpers", "budget.ts");
158 await Bun.write(
159 join(dir, "t", "a.test.ts"),
160 `import { expect, test } from "bun:test";\nimport { expectBudget } from ${JSON.stringify(helper)};\n` +
161 `test("slow page", () => { expectBudget(300, { ms: 50, against: 1 }); });\n` +
162 `test("no phase", () => { expectBudget(Number.NaN, { ms: 50, against: 1 }); });\n` +
163 `test("fine", () => { expectBudget(1, { ms: 50, against: 1 }); });\n`,
164 );
165 const run = Bun.spawnSync(["bun", "test", "./t/a.test.ts", "-t", "slow page|fine"], { cwd: dir, env: { ...process.env, FORGE_TEST_QUIET: "" } });
166 const out = (run.stdout.toString() + run.stderr.toString()).split("\n");
167 expect(budgetReds(out)).toEqual([{ file: "t/a.test.ts", name: "slow page" }]);
168 // A timing that is missing is a functional red: the phase was never named.
169 const nan = Bun.spawnSync(["bun", "test", "./t/a.test.ts", "-t", "no phase"], { cwd: dir, env: { ...process.env, FORGE_TEST_QUIET: "" } });
170 expect(budgetReds((nan.stdout.toString() + nan.stderr.toString()).split("\n"))).toBeUndefined();
171 }, 30_000);
test/budget.test.ts +12 −1
@@ -2,10 +2,13 @@
22 import { afterEach, expect, test } from "bun:test";
33 import { BASELINE_FACTOR, MAX_RELAXATION, budgetLimit, expectBudget } from "./helpers/budget.ts";
44
5 const savedTiming = process.env.FORGE_TEST_TIMING;
56 const saved = process.env.FORGE_TEST_QUIET;
67 afterEach(() => {
78 if (saved === undefined) delete process.env.FORGE_TEST_QUIET;
89 else process.env.FORGE_TEST_QUIET = saved;
10 if (savedTiming === undefined) delete process.env.FORGE_TEST_TIMING;
11 else process.env.FORGE_TEST_TIMING = savedTiming;
912 });
1013
1114 test("a quiet cell holds the absolute ms, whatever the baseline", () => {
@@ -32,8 +35,16 @@ test("only FORGE_TEST_QUIET=1 means quiet", () => {
3235 expect(budgetLimit(50, 1000)).toBe(50 * MAX_RELAXATION);
3336 });
3437
35 test("expectBudget fails a page that regressed to 150 ms beside a 5 ms baseline", () => {
38 test("expectBudget fails a page that regressed to 150 ms beside a 5 ms baseline, when timing is enforced", () => {
3639 delete process.env.FORGE_TEST_QUIET;
40 process.env.FORGE_TEST_TIMING = "1";
3741 expect(() => expectBudget(150, { ms: 50, against: 5 })).toThrow();
3842 expect(() => expectBudget(40, { ms: 50, against: 5 })).not.toThrow();
3943 });
44
45 test("without FORGE_TEST_TIMING a slow time never fails: load alone cannot turn a pull red (#7789)", () => {
46 delete process.env.FORGE_TEST_TIMING;
47 expect(() => expectBudget(5000, { ms: 50, against: 5 })).not.toThrow();
48 // A missing phase is not a slow one: it still fails.
49 expect(() => expectBudget(Number.NaN, { ms: 50, against: 5 })).toThrow();
50 });
test/check-tiers.test.ts +40
@@ -559,6 +559,46 @@ describe("the tiers in the forge's own checks service", () => {
559559 expect(readRunLog(dataDir, run!.id).lines.join("\n")).toContain(`CELL-CPUS=${want}`);
560560 });
561561
562 test("no check the forge runs holds wall-clock budgets: a pull's check and the queue's gate alike (#7789)", async () => {
563 // Set on the forge itself, so a leak through the check's environment shows.
564 const saved = process.env["FORGE_TEST_TIMING"];
565 process.env["FORGE_TEST_TIMING"] = "1";
566 try {
567 const probe = 'echo "TIMING=${FORGE_TEST_TIMING:-unset}"';
568 const seeded = await commitToBranch(app, alice, repo, {
569 message: "Probe the timing flag",
570 files: [{ path: "forge.json", content: JSON.stringify({ check: { run: "echo WHOLE", fast: probe, full: probe } }) }],
571 });
572 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
573 const branched = await commitToBranch(app, alice, repo, {
574 branch: "cpu-limit",
575 message: "Touch e",
576 files: [{ path: "e.txt", content: "e\n" }],
577 });
578 if (!branched.ok) throw new Error(JSON.stringify(branched.error));
579 const pull = listPulls(app.db, repo.id, { state: "open", limit: 20 }).find((p) => p.headBranch === "cpu-limit")!;
580 const deadline = Date.now() + 30_000;
581 let run: Run | undefined;
582 while (Date.now() < deadline) {
583 run = listRuns(app.db, repo.id, { kind: "check", limit: 20 }).find(
584 (r) => r.issueId === pull.id && r.commitOid === branched.value.commitOid,
585 );
586 if (run && (run.state === "ok" || run.state === "failed")) break;
587 await Bun.sleep(50);
588 }
589 expect(run?.state).toBe("ok");
590 const pullLog = readRunLog(dataDir, run!.id).lines.join("\n");
591 expect(pullLog).toContain("TIMING=unset");
592 const gated = await checks.gate(repo, pull, seeded.value.commitOid);
593 expect(gated.ok).toBe(true);
594 expect(gated.lines.join("\n")).toContain("TIMING=unset");
595 expect(gated.lines.join("\n")).not.toContain("TIMING=1");
596 } finally {
597 if (saved === undefined) delete process.env["FORGE_TEST_TIMING"];
598 else process.env["FORGE_TEST_TIMING"] = saved;
599 }
600 });
601
562602 test("a red full gate names its tests and the pull learns the files", async () => {
563603 const red = await commitToBranch(app, alice, repo, {
564604 message: "Full goes red",
test/flake.test.ts +43
@@ -26,6 +26,8 @@ import {
2626 redFile,
2727 redIssueFor,
2828 readRun,
29 TIMING_EXCLUDED,
30 timingFiles,
2931 retries,
3032 ROOT_CAUSES,
3133 shuffled,
@@ -425,3 +427,44 @@ describe("the scheduled run of the files outside the gate", () => {
425427 expect(redFile("Red: test/a.test.ts")).toBe("test/a.test.ts");
426428 });
427429 });
430
431 describe("the nightly timing run (#7789)", () => {
432 const run = (file: string, failing: string[] = []) =>
433 readRun(file, 0, failing.length ? 1 : 0, failing.map((f) => `(fail) ${f}`).join("\n"));
434 // Built, not written out, so this file is not itself a budget file.
435 const helper = ["./helpers", "budget.ts"].join("/");
436 const imports = `import { expectBudget } from "${helper}";\n`;
437 const files: Record<string, string> = {
438 "test/a-timing.test.ts": `${imports}expectBudget(dur, { ms: 50, against });\n`,
439 "test/b-mentions.test.ts": 'expect(rawBudgets("x", "expectBudget(dur, { ms: 50, against });")).toEqual([]);\n',
440 "test/c-imports-only.test.ts": `import { baselineMs } from "${helper}";\n`,
441 "test/budget.test.ts": `${imports}expectBudget(150, { ms: 50, against: 5 });\n`,
442 };
443
444 test("holds only files that import the budget helper and call expectBudget, less the helper's own test", () => {
445 expect(TIMING_EXCLUDED).toContain("test/budget.test.ts");
446 expect(timingFiles(Object.keys(files), (f) => files[f]!)).toEqual(["test/a-timing.test.ts"]);
447 });
448
449 test("the real selection names budget files and nothing that merely mentions the name", () => {
450 const picked = timingFiles(presentTests(), (f) => readFileSync(f, "utf8"));
451 expect(picked.length).toBeGreaterThan(0);
452 expect(picked).not.toContain("test/flake.test.ts");
453 for (const f of TIMING_EXCLUDED) expect(picked).not.toContain(f);
454 });
455
456 test("a timing Red: issue says it is a budget and reproduces with timing on", () => {
457 const red = run("test/a-timing.test.ts", ["page"]);
458 const issue = redIssueFor(red, "http://run/2", "timing");
459 expect(issue.title).toBe("Red: test/a-timing.test.ts");
460 expect(issue.body).toContain("wall-clock budget");
461 expect(issue.body).toContain("FORGE_TEST_TIMING=1 FORGE_TEST_QUIET=1 bun test");
462 expect(issue.body).not.toContain("outside the gate");
463 });
464
465 test("flake-hunt names the timing mode in its usage", () => {
466 const out = Bun.spawnSync(["bun", "scripts/flake-hunt.ts", "nope"]);
467 expect(out.exitCode).toBe(2);
468 expect(out.stderr.toString()).toContain("timing [--file-issues]");
469 });
470 });
test/helpers/budget.ts +24 −8
@@ -5,19 +5,28 @@
55 * the same code is 3 ms on a quiet machine and 80 ms beside six busy cells.
66 * `expectBudget` judges a measured time against a baseline taken in the same
77 * run, so load that slows the page slows the baseline with it. The absolute
8 * `ms` is asserted when the cell is quiet (`FORGE_TEST_QUIET=1`, set by hand on
9 * a quiet machine; no lane sets it yet). Otherwise the limit is relative to the
8 * `ms` is asserted when the cell is quiet (`FORGE_TEST_QUIET=1`, which the
9 * nightly `timing` workflow sets). Otherwise the limit is relative to the
1010 * baseline but never above `MAX_RELAXATION` times `ms`, so a 50 ms budget
1111 * cannot drift past 200 ms however slow the baseline.
1212 *
13 * A time over its limit throws an error that starts with `BUDGET_MARK`, so
14 * the merge queue's gate can tell a budget red from a functional one and
15 * re-run it once, serially, before it sinks a batch (#7354,
16 * `src/checks/budget_retry.ts`). A time that is not a number is a missing
17 * phase, not a slow one: that fails as any `expect` does.
13 * Wall-clock is judged only where it can be trusted (#7789): with
14 * `FORGE_TEST_TIMING=1`, set only by the nightly `timing` workflow. In every
15 * check the forge runs, a pull's and the merge queue's gate alike, a time over its limit is reported on stderr and the test
16 * passes, so the machine being busy can never turn a pull or a landing red; what a page
17 * costs is a matter for the performance watcher (#7559) reading real traffic,
18 * and a test that must fail a regression counts work (statements, bytes,
19 * phases) instead.
20 *
21 * With timing on, a time over its limit throws an error that starts with
22 * `BUDGET_MARK`, so the nightly's `Red:` issue says it is a budget. A time
23 * that is not a number is a missing phase, not a slow one: that fails as any
24 * `expect` does, timing on or off.
1825 */
1926 import { expect } from "bun:test";
20 import { BUDGET_MARK } from "../../src/checks/budget_retry.ts";
27
28 /** What a budget red's error starts with. */
29 export const BUDGET_MARK = "over budget (expectBudget)";
2130
2231 /** How many times slower than the same-run baseline a page may be under load. */
2332 export const BASELINE_FACTOR = 10;
@@ -28,6 +37,9 @@ export const MAX_RELAXATION = 4;
2837 /** Whether the cell is quiet enough to hold a number to its absolute ms. */
2938 export const quiet = (): boolean => process.env.FORGE_TEST_QUIET === "1";
3039
40 /** Whether wall-clock budgets are enforced: only the scheduled run says so (`FORGE_TEST_TIMING=1`). */
41 export const timing = (): boolean => process.env.FORGE_TEST_TIMING === "1";
42
3143 /** The ms a measured time may take: `ms` when quiet, else relative to `against`. */
3244 export function budgetLimit(ms: number, against: number): number {
3345 if (quiet()) return ms;
@@ -42,6 +54,10 @@ export function expectBudget(measured: number, opts: { ms: number; against: numb
4254 }
4355 const limit = budgetLimit(opts.ms, opts.against);
4456 if (measured < limit) return;
57 if (!timing()) {
58 console.warn(`over budget, not enforced (FORGE_TEST_TIMING unset): ${measured.toFixed(1)} ms vs ${limit.toFixed(1)} ms (budget ${opts.ms} ms)`);
59 return;
60 }
4561 throw new Error(
4662 `${BUDGET_MARK}: ${measured.toFixed(1)} ms is not under ${limit.toFixed(1)} ms ` +
4763 `(budget ${opts.ms} ms, baseline ${opts.against.toFixed(1)} ms${quiet() ? ", quiet" : ""})`,
test/merge-queue-budget-retry.test.ts deleted −172
@@ -1,172 +0,0 @@
1 /**
2 * The merge queue, driven through the real check lane (#7354): a batch whose
3 * only red is a budget test re-runs that file once, serially, and lands
4 * when it passes — the test recorded as a flake. Red again on the re-run,
5 * or red on a functional test, the batch is red and searched as before:
6 * a functional red is never re-run.
7 */
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { createApp, type App } from "../src/app.ts";
13 import { openPull } from "../src/actions/pulls.ts";
14 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
15 import { budgetRetries } from "../src/checks/budget_retry.ts";
16 import { flakeList } from "../src/checks/flake.ts";
17 import { gateCandidate } from "../src/checks/landing.ts";
18 import { type Candidate, enqueue, type Expect, type Queued } from "../src/checks/queue.ts";
19 import { startChecks, type ChecksHandle } from "../src/checks/service.ts";
20 import { getPull } from "../src/core/pulls.ts";
21 import type { Repo } from "../src/core/repos.ts";
22 import { createUser, type User } from "../src/core/users.ts";
23 import { repoDir } from "../src/git/repos.ts";
24 import { quietLogs } from "./helpers/globals.ts";
25
26 quietLogs();
27
28 /** What the gate's sharded run prints for one red test in `t/a.test.ts`, then exits 1. */
29 const gateLog = (error: string): string =>
30 [
31 "cat <<'EOF'",
32 "── shard 1/2 (exit 1, 2.0s): t/a.test.ts",
33 "t/a.test.ts:",
34 error,
35 "(fail) grp > page [81.00ms]",
36 " 0 pass",
37 " 1 fail",
38 "Ran 1 test across 1 file. [90.00ms]",
39 "── shard 2/2 (exit 0, 1.0s): t/b.test.ts",
40 "t/b.test.ts:",
41 "(pass) fine [1.00ms]",
42 " 1 pass",
43 " 0 fail",
44 "1 of 2 shards failed:",
45 "(fail) grp > page [81.00ms] [shard 1]",
46 "EOF",
47 "exit 1",
48 "",
49 ].join("\n");
50
51 const BUDGET = "error: over budget (expectBudget): 81.0 ms is not under 50.0 ms (budget 50 ms, baseline 2.0 ms)";
52 const FUNCTIONAL = "error: expect(received).toBe(expected)";
53
54 describe("the merge queue re-runs a batch's budget-only reds once", () => {
55 let dataDir: string;
56 let app: App;
57 let checks: ChecksHandle;
58 let alice: User;
59 const cells = process.env["FORGE_CELLS"];
60
61 beforeAll(async () => {
62 process.env["FORGE_CELLS"] = "off";
63 dataDir = await mkdtemp(join(tmpdir(), "forge-budget-retry-"));
64 app = createApp({ dataDir, port: 0 });
65 checks = startChecks(app, { forgeUrl: "http://127.0.0.1:1", concurrency: 2 });
66 const a = await createUser(app.db, {
67 origin: "seed",
68 trusted: true,
69 username: "alice",
70 email: "[email protected]",
71 password: "a-good-passphrase",
72 adminIfFirst: true,
73 });
74 if (!a.ok) throw new Error(a.error);
75 alice = a.value;
76 });
77
78 afterAll(async () => {
79 await checks.stop({ drain: false });
80 app.close();
81 if (cells === undefined) delete process.env["FORGE_CELLS"];
82 else process.env["FORGE_CELLS"] = cells;
83 await rm(dataDir, { recursive: true, force: true });
84 });
85
86 /**
87 * A repository whose gate prints `error` above its one red test, whose
88 * `t/a.test.ts` does `body` when re-run, and two pulls, queued as a batch.
89 */
90 async function queueWith(name: string, error: string, body: string) {
91 const made = await createRepo(app, alice, { name, visibility: "public" });
92 if (!made.ok) throw new Error(String(made.error));
93 const repo: Repo = made.value;
94 const seeded = await commitToBranch(app, alice, repo, {
95 message: "Start",
96 files: [
97 { path: "forge.json", content: JSON.stringify({ check: { run: "true", fast: "true", full: "sh gate.sh" } }) },
98 { path: "gate.sh", content: gateLog(error) },
99 { path: "t/a.test.ts", content: `import { describe, expect, test } from "bun:test";\ndescribe("grp", () => { test("page", () => { ${body} }); });\n` },
100 ],
101 });
102 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
103 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
104 const git = (...args: string[]) => Bun.spawnSync(["git", "-C", dir, ...args]).stdout.toString().trim();
105 const handedBack: number[] = [];
106 const entries: Queued[] = [];
107 for (const n of [1, 2]) {
108 const branched = await commitToBranch(app, alice, repo, {
109 branch: `f${n}`,
110 message: `Add f${n}`,
111 files: [{ path: `f${n}.txt`, content: `${n}\n` }],
112 });
113 if (!branched.ok) throw new Error(JSON.stringify(branched.error));
114 const opened = await openPull(app, alice, repo, { title: `F${n}`, baseBranch: "main", headBranch: `f${n}` });
115 if (!opened.ok) throw new Error(String(opened.error));
116 const pull = getPull(app.db, repo.id, opened.value.number)!;
117 entries.push({
118 pull,
119 dir,
120 land: async (want: Expect) => {
121 if (git("rev-parse", "refs/heads/main") !== want.baseOid) return undefined;
122 const tip = git("-c", "user.name=t", "-c", "user.email=t@t", "commit-tree", want.treeOid, "-p", want.baseOid, "-m", `#${pull.number}`);
123 git("update-ref", "refs/heads/main", tip);
124 return tip;
125 },
126 release: () => {},
127 handBack: () => handedBack.push(pull.number),
128 });
129 }
130 const gates: number[][] = [];
131 const gate = (c: Candidate, signal: AbortSignal) => {
132 gates.push(c.steps.map((s) => s.entry.pull.number));
133 return gateCandidate(app, repo, c.steps[0]!.entry.pull, c.dir, c.tipOid, signal);
134 };
135 const landed = await enqueue(repo.id, entries, gate, { depth: 1, room: () => 0 });
136 return { repo, landed, gates, handedBack, numbers: entries.map((e) => e.pull.number) };
137 }
138
139 test("a batch red only on a budget test that passes on the re-run lands whole, and the test is a flake", async () => {
140 const { repo, landed, gates, handedBack, numbers } = await queueWith("budget-green", BUDGET, "expect(1).toBe(1);");
141 expect(landed).toEqual([true, true]);
142 expect(gates).toEqual([numbers]);
143 expect(handedBack).toEqual([]);
144 expect(budgetRetries(repo.id)).toEqual({ passed: 1, failed: 0 });
145 expect(flakeList(app.db, Date.now()).find((f) => f.repo === "alice/budget-green")).toMatchObject({
146 test: "grp > page",
147 file: "t/a.test.ts",
148 count: 1,
149 });
150 }, 90_000);
151
152 test("a budget test red on the re-run too keeps the batch red, and the search goes on", async () => {
153 const { repo, landed, gates, handedBack, numbers } = await queueWith("budget-red", BUDGET, "expect(1).toBe(2);");
154 expect(landed).toEqual([false, false]);
155 expect(gates[0]).toEqual(numbers);
156 expect(gates.length).toBeGreaterThan(1);
157 expect(handedBack.length).toBeGreaterThan(0);
158 // Each gate re-ran once and failed; none passed.
159 expect(budgetRetries(repo.id)).toEqual({ passed: 0, failed: gates.length });
160 expect(flakeList(app.db, Date.now()).some((f) => f.repo === "alice/budget-red")).toBe(false);
161 }, 90_000);
162
163 test("a functional red is searched at once, never re-run, though its file would pass", async () => {
164 const { repo, landed, gates, handedBack, numbers } = await queueWith("functional", FUNCTIONAL, "expect(1).toBe(1);");
165 expect(landed).toEqual([false, false]);
166 // The batch, then its front half at once: bisected, not gated again whole.
167 expect(gates.slice(0, 2)).toEqual([numbers, [numbers[0]!]]);
168 expect(handedBack.length).toBeGreaterThan(0);
169 expect(budgetRetries(repo.id)).toEqual({ passed: 0, failed: 0 });
170 expect(flakeList(app.db, Date.now()).some((f) => f.repo === "alice/functional")).toBe(false);
171 }, 90_000);
172 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs