trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

A pull enters the merge queue only after the full suite passes on its current head merged with main (not the impact-mapped subset) #7620

Merged torquecrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/7616-a-pull-enters-the-merge-queue-only-after → main 2 commits fork

  • Merged. torque/7616-a-pull-enters-the-merge-queue-only-after was merged into main as accaa65.

Review

Sign in to review this change.
10 files changed +682 −20
docs/CREW.md +15 −3
@@ -167,9 +167,21 @@ the mark. Only the crew parks: the API and the page refuse `reason: "parked"`.
167167 In the database the pull's `close_reason` stays NULL (its CHECK predates the
168168 word) and `issue.parked = 1` carries it, so raw SQL never reads it as done.
169169
170 **The merge queue.** A green, approved pull in an autonomous repository joins
171 its repository's queue (`src/checks/queue.ts`). The queue builds one
172 candidate: main, then each queued pull's squash on top of the one before, in
170 **The merge queue.** A green, approved pull in an autonomous repository
171 joins its repository's queue (`src/checks/queue.ts`) once it is proved
172 (#7616, `src/checks/proof.ts`): when its check has a fast tier, the full
173 gate — the same command, environment and sharding — runs on its head
174 squashed onto main's tip first, using one check slot. Green admits it, kept
175 as a `proof` row in the decision ledger with the head and base. A red one is
176 held (`proof-red` in the tick's waits; `proving` while it runs), and the
177 pull gets a comment that names the failing tests, so a test the fast tier
178 never reached costs no batch. A proof goes stale when main moves more than
179 16 first-parent landings past it (`STALE_LANDINGS`). The pull is then proved
180 again before it joins, and a pull already in the line is left out of its
181 next batch until it is proved again. A proof that has not answered by the
182 gate's deadline (15 minutes) is cut off and no verdict is kept, so a hung
183 one never holds its pull or a check slot.
184 The queue builds one candidate: main, then each queued pull's squash on top of the one before, in
173185 queue order, leaving out any pull that conflicts or has nothing left to land.
174186 A candidate carries at most `queue.batch` pulls (Settings → Capacity,
175187 `FORGE_QUEUE_BATCH`, 1–32, default 8); the rest wait their turn. Each line
src/checks/landing.ts +34 −2
@@ -62,6 +62,38 @@ export async function gateCandidate(
6262 return gateVerdict(app, repo, commitOid, await checks.gate(repo, pull, commitOid, signal));
6363 }
6464
65 /**
66 * The proof before the line (#7616, `proof.ts`): the full gate on the
67 * pull's head squashed onto its base, run as the gate runs. Undefined when
68 * there is nothing to prove with — no checks service, a lane that cannot
69 * prove, or a check with no fast tier, whose pull check was already the
70 * whole suite.
71 */
72 export async function proveCandidate(
73 app: App,
74 repo: Repo,
75 pull: Pull,
76 dir: string,
77 commitOid: string,
78 signal?: AbortSignal,
79 ): Promise<LandVerdict | undefined> {
80 const prove = checksFor(app)?.prove;
81 if (prove === undefined || !(await provable(app, dir, commitOid))) return undefined;
82 // Not the forge's last gate: the Resilience page reads the queue's gates only.
83 return gateVerdict(app, repo, commitOid, await prove(repo, pull, commitOid, signal), false);
84 }
85
86 /** Whether a pull at `commitOid` is proved before the line: a lane that proves, and a check with a fast tier. */
87 export async function provable(app: App, dir: string, commitOid: string): Promise<boolean> {
88 if (checksFor(app)?.prove === undefined) return false;
89 const [forgeJson, packageJson] = await Promise.all([
90 readFileAt(dir, commitOid, "forge.json"),
91 readFileAt(dir, commitOid, "package.json"),
92 ]);
93 const declared = parseCheck(forgeJson, packageJson);
94 return declared.ok && declared.value !== undefined && declared.value.fast !== null;
95 }
96
6597 /** How long the queue waits between reads of a gate another forge runs (#6611). */
6698 export const ADOPT_POLL_MS = 5_000;
6799
@@ -99,10 +131,10 @@ export async function adoptGate(
99131 }
100132
101133 /** What the lane's answer on `commitOid` decides: its log kept, its red named. */
102 function gateVerdict(app: App, repo: Repo, commitOid: string, done: GateDone): LandVerdict {
134 function gateVerdict(app: App, repo: Repo, commitOid: string, done: GateDone, last = true): LandVerdict {
103135 // Cut off, not judged: no verdict, and not the gate log a red reads (#3155).
104136 if (done.interrupted === true) return { lands: false, interrupted: true };
105 keepForgeGate(app.config.dataDir, app.config.forgeRepo, `${repo.ownerName}/${repo.name}`, done.lines);
137 if (last) keepForgeGate(app.config.dataDir, app.config.forgeRepo, `${repo.ownerName}/${repo.name}`, done.lines);
106138 const slug = `${repo.ownerName}/${repo.name}`;
107139 const kept = keepGateLog(app.config.dataDir, slug, commitOid, done.ok, done.lines, Date.now());
108140 const failed = failuresOf(done.lines);
src/checks/proof.ts added +215
@@ -0,0 +1,215 @@
1 /**
2 * The proof before the line (#7616). A pull's own check is the fast tier —
3 * the tests its change reaches — while the merge queue's gate runs every
4 * test, so a pull that broke a test its change never reached was found in
5 * the queue, where one red costs a batch of up to eight. Before a pull
6 * joins the line it is proved: the full gate (the same command,
7 * environment and sharding — `ChecksHandle.prove` is the gate's own run)
8 * on its head squashed onto the base's tip, the tree a batch of one would
9 * gate. Green is kept in the decision ledger as a `proof` row with the head
10 * and the base it was proved on, and admits the pull; red is told to the
11 * pull with its failing tests named and holds it, at no cost to any batch.
12 *
13 * A proof goes stale once the base has moved more than `STALE_LANDINGS`
14 * first-parent commits past the base it was proved on, or off it: the pull
15 * is proved again before it joins, and one already waiting in the line is
16 * left out of its next batch (`Queued.proven`) and proved again. A red
17 * proof stands for its head until a push, or until the base has moved that
18 * far — a red that was main's may be gone from a newer main.
19 *
20 * A proof cut off (the forge stopping, the Job lost) is no verdict: nothing
21 * is kept, and the next sweep proves it again. One that has not answered by
22 * the gate's own deadline (`GATE_DEADLINE_MS`, #3430) is cut off the same
23 * way, its run aborted, so a hung proof never holds its pull or a check slot
24 * for good. One proof runs at a time per
25 * pull, and only one beyond the first while no check slot is spare.
26 */
27
28 import type { Database } from "bun:sqlite";
29 import { latestDecision, recordDecision } from "../core/decisions.ts";
30 import type { Pull } from "../core/pulls.ts";
31 import { runGit } from "../git/env.ts";
32 import { mergeable } from "../git/merge.ts";
33 import { GATE_DEADLINE_MS } from "./queue.ts";
34 import type { LandVerdict } from "./tiers.ts";
35
36 /**
37 * How far the base may move past a proof before it is proved again: two
38 * full batches (`MAX_CANDIDATE` is 8), so a pull proved as it joined is
39 * still proved while the two batches ahead of it land, and one that waited
40 * longer than that is proved on what main has become.
41 */
42 export const STALE_LANDINGS = 16;
43
44 /** What a proof found: the head it ran on, the base tip it was squashed onto, and whether it was green. */
45 export interface Proof {
46 readonly head: string;
47 readonly base: string;
48 readonly green: boolean;
49 }
50
51 /** Where a pull stands before the line. */
52 export type Standing = "proved" | "proving" | "red" | "unproved";
53
54 /** The proofs running now: pull id → the head being proved. */
55 const proving = new Map<string, string>();
56
57 /** How many proofs are running on this forge: they hold check slots as gates do. */
58 export function proofsInFlight(): number {
59 return proving.size;
60 }
61
62 /** The newest proof of a pull, read off the ledger; undefined when it has none. */
63 export function proofOf(db: Database, pullId: string): Proof | undefined {
64 const row = latestDecision(db, pullId, "proof");
65 if (row === undefined) return undefined;
66 const state = row.state;
67 if (typeof state !== "object" || state === null) return undefined;
68 const { head, base } = state as Record<string, unknown>;
69 if (typeof head !== "string" || typeof base !== "string") return undefined;
70 return { head, base, green: row.answer === "green" };
71 }
72
73 /** Keep a proof's verdict on the ledger. */
74 export function keepProof(db: Database, repoId: string, pullId: string, proof: Proof): void {
75 recordDecision(db, {
76 kind: "proof",
77 repoId,
78 issueId: pullId,
79 state: { head: proof.head, base: proof.base },
80 answer: proof.green ? "green" : "red",
81 p: 1,
82 by: "rule",
83 ms: 0,
84 });
85 }
86
87 /**
88 * First-parent commits from `from` to `to`: the landings since a proof.
89 * Undefined when `from` is not behind `to` (the base was rewritten) or git
90 * cannot say.
91 */
92 export async function landingsSince(dir: string, from: string, to: string): Promise<number | undefined> {
93 if (from === to) return 0;
94 const behind = await runGit(["merge-base", "--is-ancestor", from, to], { cwd: dir });
95 if (!behind.ok) return undefined;
96 const count = await runGit(["rev-list", "--count", "--first-parent", `${from}..${to}`], { cwd: dir });
97 if (!count.ok) return undefined;
98 const n = Number(count.value.trim());
99 return Number.isSafeInteger(n) && n >= 0 ? n : undefined;
100 }
101
102 /** A proof still holds on `tip`: the base has moved no more than `STALE_LANDINGS` past it. */
103 export async function current(dir: string, proof: Proof, tip: string): Promise<boolean> {
104 const moved = await landingsSince(dir, proof.base, tip);
105 return moved !== undefined && moved <= STALE_LANDINGS;
106 }
107
108 /** Where `pull` at its head stands on the base tip `tip`. */
109 export async function standing(db: Database, dir: string, pull: Pull, tip: string): Promise<Standing> {
110 if (proving.get(pull.id) === pull.headOid) return "proving";
111 const proof = proofOf(db, pull.id);
112 if (proof === undefined || proof.head !== pull.headOid) return "unproved";
113 if (!(await current(dir, proof, tip))) return "unproved";
114 return proof.green ? "proved" : "red";
115 }
116
117 /** The pull's head squashed onto `tip`, as a commit the gate can run on; undefined when it does not merge. */
118 export async function proofCommit(dir: string, tip: string, pull: Pull): Promise<string | undefined> {
119 const merged = await mergeable(dir, tip, pull.headOid);
120 if (!merged.ok) return undefined;
121 const commit = await runGit(
122 ["commit-tree", merged.value.treeOid, "-p", tip, "-m", `proof: #${pull.number}`],
123 { cwd: dir, env: IDENTITY },
124 );
125 return commit.ok ? commit.value.trim() : undefined;
126 }
127
128 const IDENTITY = {
129 GIT_AUTHOR_NAME: "forge",
130 GIT_AUTHOR_EMAIL: "forge@localhost",
131 GIT_COMMITTER_NAME: "forge",
132 GIT_COMMITTER_EMAIL: "forge@localhost",
133 };
134
135 /** What starting a proof needs: where, on what, how to run the gate, and what to do with its answer. */
136 export interface ProofRun {
137 readonly db: Database;
138 readonly repoId: string;
139 readonly pull: Pull;
140 readonly dir: string;
141 readonly tip: string;
142 /** The full gate on a commit; `signal` aborts at the deadline, and the run should stop. */
143 readonly run: (commitOid: string, signal: AbortSignal) => Promise<LandVerdict>;
144 /** Check slots spare now; a second proof starts only on one. */
145 readonly room: () => number;
146 /** The proof was judged and kept. */
147 readonly done: (proof: Proof, verdict: LandVerdict) => void;
148 /** How long the proof may run before it is cut off; the gate's deadline unless a test shortens it. */
149 readonly deadlineMs?: number;
150 }
151
152 /**
153 * Start proving `pull` on `tip` beside the caller, which does not wait.
154 * False when it did not start: a proof of this head runs already, no slot
155 * is free, or the head does not merge onto the tip.
156 */
157 export async function startProof(x: ProofRun): Promise<boolean> {
158 if (proving.has(x.pull.id)) return false;
159 if (proving.size > 0 && x.room() <= 0) return false;
160 proving.set(x.pull.id, x.pull.headOid);
161 let commit: string | undefined;
162 try {
163 commit = await proofCommit(x.dir, x.tip, x.pull);
164 } catch {
165 commit = undefined;
166 }
167 if (commit === undefined) {
168 proving.delete(x.pull.id);
169 return false;
170 }
171 const abandon = new AbortController();
172 let timer: ReturnType<typeof setTimeout> | undefined;
173 // A proof that never answers is cut off at the deadline: no verdict (#3430).
174 const cut = new Promise<LandVerdict>((resolve) => {
175 timer = setTimeout(() => {
176 abandon.abort();
177 resolve({ lands: false, interrupted: true });
178 }, x.deadlineMs ?? GATE_DEADLINE_MS);
179 });
180 void Promise.race([x.run(commit, abandon.signal), cut])
181 .then(
182 (verdict) => {
183 // Released before the answer is acted on: `done` may sweep, and must not read it as running.
184 clearTimeout(timer);
185 proving.delete(x.pull.id);
186 if ("interrupted" in verdict) return;
187 const proof: Proof = { head: x.pull.headOid, base: x.tip, green: verdict.lands };
188 keepProof(x.db, x.repoId, x.pull.id, proof);
189 x.done(proof, verdict);
190 },
191 () => {
192 clearTimeout(timer);
193 proving.delete(x.pull.id);
194 },
195 );
196 return true;
197 }
198
199 /** Forget the proofs running: a test's reset. */
200 export function resetProofsForTest(): void {
201 proving.clear();
202 }
203
204 /** What a red proof tells the pull: the base it was squashed onto, the tests that failed, and the gate's log. */
205 export function proofRedWords(branch: string, base: string, red: Extract<LandVerdict, { reopen: true }>): string {
206 const names = (red.tests?.failing ?? []).map((t) => `- ${t.name}`);
207 const more = (red.tests?.failed ?? names.length) - names.length;
208 const log = /The gate's log: (\S+)/.exec(red.comment)?.[1];
209 return [
210 `The full suite on this head squashed onto ${branch} (${base.slice(0, 7)}) was red, so the pull has not joined the merge queue.`,
211 names.length === 0 ? "It named no test (a type error, a timeout, a setup step)." : `Red tests:\n${names.join("\n")}${more > 0 ? `\n- and ${more} more` : ""}`,
212 ...(log === undefined ? [] : [`The gate's log: ${log}`]),
213 "It is proved again on a push.",
214 ].join("\n\n");
215 }
src/checks/queue.ts +15
@@ -57,6 +57,11 @@
5757 * with its pulls, counts and failing tests (#6739, `gate_result.ts`), read
5858 * by `GET …/queue/gates`.
5959 *
60 * A pull joins proved (#7616, `proof.ts`): the full suite was green on its
61 * head squashed onto its base. One whose proof the base has since moved too
62 * far past is left out of its batch, as a conflict is, and proved again
63 * before it joins once more.
64 *
6065 * What is queued is not stored: it is the pulls `landIfGreen` admitted, and
6166 * every sweep admits them again. The queue only orders them and remembers
6267 * what is at the gate and what its searches found, for the tick line and
@@ -102,6 +107,12 @@ export interface Queued {
102107 * that its red tree held: none when it was red on its own merge.
103108 */
104109 handBack(red: Red, baseOid: string, together: readonly number[]): void;
110 /**
111 * The pull's proof before the line still holds on this base (#7616,
112 * `proof.ts`). False leaves it out of the batch, answered `false`, to be
113 * proved again before it joins; absent, it always holds.
114 */
115 proven?(baseOid: string): Promise<boolean>;
105116 }
106117
107118 /** One pull's squash on top of the steps before it. */
@@ -1596,6 +1607,10 @@ async function build(batch: readonly Waiting[], on?: Candidate): Promise<Candida
15961607 let treeOid = baseTreeOid;
15971608 const steps: Step[] = [];
15981609 for (const { entry } of batch) {
1610 if (entry.proven !== undefined && !(await entry.proven(tipOid))) {
1611 log.info(`left out of the batch: #${entry.pull.number}'s proof is stale, proved again first`);
1612 continue;
1613 }
15991614 const merged = await mergeable(dir, tipOid, entry.pull.headOid);
16001615 if (!merged.ok) {
16011616 // Left out, never red (#5276): the rest gate without it, and once the
src/checks/service.ts +23 −12
@@ -97,6 +97,7 @@ import { placementFromEnv, type Placement } from "./placement.ts";
9797 import { JOB_VANISHED, runJobStep } from "./job_step.ts";
9898 import { checkInstallCache, installWithRetry, withLock } from "./install.ts";
9999 import { checkStage, type GateStage } from "./gate_reason.ts";
100 import { proofsInFlight } from "./proof.ts";
100101 import { extraGates, gateQueued, gateRunId, gatesInFlight, watchGates } from "./queue.ts";
101102 import type { JobOptions } from "../cluster/jobs.ts";
102103 import type { CheckDetail } from "../core/runs.ts";
@@ -173,6 +174,14 @@ export interface ChecksHandle {
173174 * `signal` aborting (the queue's deadline, #3430) cuts it off the same way.
174175 */
175176 gate(repo: Repo, pull: Pull, commitOid: string, signal?: AbortSignal): Promise<GateDone>;
177 /**
178 * The proof before the line (#7616, `checks/proof.ts`): the gate's own
179 * run — command, environment, sharding — on the pull's head squashed onto
180 * its base, cut off as a gate is when `signal` aborts (its deadline). A
181 * lane without it (a test's stand-in) admits on the pull's own check, as
182 * before.
183 */
184 prove?(repo: Repo, pull: Pull, commitOid: string, signal?: AbortSignal): Promise<GateDone>;
176185 /**
177186 * The gate of `commitOid` this forge is running now, to wait on rather
178187 * than start again (#6611); undefined when none is.
@@ -498,7 +507,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
498507 const limit = capacity();
499508 if (options.gate === undefined) gate.resize(limit);
500509 // Every merge-queue gate in flight holds a slot, a line's first too (#6376).
501 while (inFlight.size < limit - gatesInFlight()) {
510 while (inFlight.size < limit - gatesInFlight() - proofsInFlight()) {
502511 // A pull's check waits while its main is red (`checks/red_main.ts`).
503512 const queued = unheld(
504513 listActive(app.db, "check").filter(
@@ -517,7 +526,7 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
517526 const { rebases, rest: others } = rebasesFirst(app.db, queued);
518527 // While a gate is about to start, the last free slot is its: a pull's
519528 // check waits for it, a rebase's and a main check do not.
520 const held = holdsLastSlot(limit - gatesInFlight() - inFlight.size, gateQueued());
529 const held = holdsLastSlot(limit - gatesInFlight() - proofsInFlight() - inFlight.size, gateQueued());
521530 const next = startable(rebases) ?? startable(held ? others.filter((run) => run.issueId === null) : others);
522531 if (!next) return;
523532 const repo = getRepoById(app.db, next.repoId);
@@ -1418,22 +1427,24 @@ export function startChecks(app: App, options: ChecksOptions): ChecksHandle {
14181427 }
14191428 }
14201429
1430 /** A gate asked of a forge already stopping is not started: the forge that carries on asks it again. */
1431 function gateRun(repo: Repo, pull: Pull, commitOid: string, signal?: AbortSignal): Promise<GateDone> {
1432 if (stopped) return Promise.resolve({ ok: false, error: INTERRUPTED, lines: [], interrupted: true });
1433 const controller = new AbortController();
1434 const done = gateOnce(repo, pull, commitOid, controller, signal);
1435 gates.set(controller, { commitOid, done });
1436 return done;
1437 }
1438
14211439 const handle: ChecksHandle = {
14221440 request,
1423 gate: (repo, pull, commitOid, signal) => {
1424 // A gate asked of a forge already stopping is not started: the forge
1425 // that carries on asks it again.
1426 if (stopped) return Promise.resolve({ ok: false, error: INTERRUPTED, lines: [], interrupted: true });
1427 const controller = new AbortController();
1428 const done = gateOnce(repo, pull, commitOid, controller, signal);
1429 gates.set(controller, { commitOid, done });
1430 return done;
1431 },
1441 gate: gateRun,
1442 prove: gateRun,
14321443 adopt: (commitOid) => {
14331444 for (const g of gates.values()) if (g.commitOid === commitOid) return g.done;
14341445 return undefined;
14351446 },
1436 spareSlots: () => Math.max(0, capacity() - PULL_RESERVE - inFlight.size - extraGates()),
1447 spareSlots: () => Math.max(0, capacity() - PULL_RESERVE - inFlight.size - extraGates() - proofsInFlight()),
14371448 lane: gate,
14381449 stop: async (stopOptions) => {
14391450 stopped = true;
src/core/decisions.ts +7 −2
@@ -19,14 +19,19 @@ import { id as newId } from "../kernel/id.ts";
1919 * the summaries that read them keep the word.
2020 *
2121 * `hand` is a person's tap on "Let the decider" on a hand-over card
22 * (`crew/handover.ts`): its state is the curated context the decider saw. */
22 * (`crew/handover.ts`): its state is the curated context the decider saw.
23 *
24 * `proof` is the full suite on a pull's head squashed onto its base, before
25 * the pull joins the merge queue (`checks/proof.ts`): its state is the head
26 * and the base, its answer `green` or `red`. */
2327 export type DecisionKind =
2428 | "next"
2529 | "land"
2630 | "retry"
2731 | "lean"
2832 | "guard"
29 | "hand";
33 | "hand"
34 | "proof";
3035
3136 export interface Decision {
3237 readonly id: string;
src/crew/autonomy.ts +66 −1
@@ -23,7 +23,8 @@ import {
2323 setIssueState,
2424 } from "../actions/issues.ts";
2525 import type { AccessError } from "../actions/access.ts";
26 import { adoptGate, gateCandidate, releaseOnLand } from "../checks/landing.ts";
26 import { adoptGate, gateCandidate, provable, proveCandidate, releaseOnLand } from "../checks/landing.ts";
27 import { current, proofOf, proofRedWords, standing as proofStanding, startProof } from "../checks/proof.ts";
2728 import { queueStore } from "../checks/queue_store.ts";
2829 import { addLearned } from "../checks/learned.ts";
2930 import { batchWords, dropFromLine, enqueue, type GateCandidate, type Queued, type QueueOptions, queueClashes, queueWords, waitingPulls } from "../checks/queue.ts";
@@ -2016,6 +2017,14 @@ async function admit(
20162017 // not hold it for good; a red that reproduces stays red (#3155).
20172018 const told = latestDecision(app.db, pull.id, "land");
20182019 if (redOf(told)?.head === pull.headOid && !regateDue(told!, Date.now())) return err("red");
2020 // The proof before the line (#7616): the whole suite, green on this head
2021 // squashed onto the base's tip, or the pull does not join — a test its
2022 // fast tier never reached is found here, at no batch's cost.
2023 const proves = baseTip !== "" && (await provable(app, dir, pull.headOid));
2024 if (proves) {
2025 const wait = await proofWait(app, repo, by, pull, dir, baseTip);
2026 if (wait !== undefined) return err(wait);
2027 }
20192028 // Green only against a base that has moved since is admitted as it is:
20202029 // the candidate's full gate runs on the base tip with this pull merged,
20212030 // which is the proof a re-check of its own would give, and a red one
@@ -2056,9 +2065,65 @@ async function admit(
20562065 addLearned(app.config.dataDir, pull.id, red.learn);
20572066 if (!again) addComment(app, by, repo, getIssueById(app.db, pull.id) ?? pull, clashSaid(red.comment, together));
20582067 },
2068 // A proof the base has moved too far past is proved again before the batch.
2069 ...(proves ? { proven: (baseOid: string) => proofHolds(app, pull, dir, baseOid) } : {}),
20592070 });
20602071 }
20612072
2073 /**
2074 * Why a pull waits on its proof (#7616), or undefined when it is proved on
2075 * `baseTip`. An unproved pull's proof is started here, beside the sweep:
2076 * green admits it at once (`landIfGreen`), red is told to the pull with
2077 * its failing tests named.
2078 */
2079 async function proofWait(
2080 app: App,
2081 repo: Repo,
2082 by: User,
2083 pull: Pull,
2084 dir: string,
2085 baseTip: string,
2086 ): Promise<"proving" | "proof-red" | undefined> {
2087 const at = await proofStanding(app.db, dir, pull, baseTip);
2088 if (at === "proved") return undefined;
2089 if (at === "red") return "proof-red";
2090 if (at === "proving") return "proving";
2091 const started = await startProof({
2092 db: app.db,
2093 repoId: repo.id,
2094 pull,
2095 dir,
2096 tip: baseTip,
2097 run: async (commitOid, signal) => (await proveCandidate(app, repo, pull, dir, commitOid, signal)) ?? { lands: true },
2098 room: () => checksFor(app)?.spareSlots() ?? 0,
2099 done: (proof, verdict) => {
2100 log.info(`proof ${proof.green ? "green" : "red"}`, {
2101 repo: `${repo.ownerName}/${repo.name}`,
2102 pull: pull.number,
2103 base: proof.base.slice(0, 7),
2104 });
2105 const now = getPull(app.db, repo.id, pull.number);
2106 if (now === undefined || now.headOid !== proof.head || now.state !== "open") return;
2107 if (!verdict.lands && "reopen" in verdict) {
2108 addLearned(app.config.dataDir, pull.id, verdict.learn);
2109 addComment(app, by, repo, getIssueById(app.db, pull.id) ?? now, proofRedWords(pull.baseBranch, proof.base, verdict));
2110 return;
2111 }
2112 void landIfGreen(app, repo, now).catch((error) =>
2113 log.warn("could not land after its proof", { pull: pull.number, error: String(error) }),
2114 );
2115 },
2116 });
2117 if (started) log.info("proving before the line", { repo: `${repo.ownerName}/${repo.name}`, pull: pull.number, base: baseTip.slice(0, 7) });
2118 return "proving";
2119 }
2120
2121 /** The pull's green proof still holds on `baseOid` (#7616). */
2122 async function proofHolds(app: App, pull: Pull, dir: string, baseOid: string): Promise<boolean> {
2123 const proof = proofOf(app.db, pull.id);
2124 return proof !== undefined && proof.green && proof.head === pull.headOid && (await current(dir, proof, baseOid));
2125 }
2126
20622127 /**
20632128 * A red's comment, and when it was red on top of pulls of its candidate
20642129 * that landed first, a line naming them (#4235). The search does not gate
src/crew/waits.ts +2
@@ -21,6 +21,8 @@ export type Wait =
2121 | "carried"
2222 | "wall"
2323 | "review"
24 | "proving"
25 | "proof-red"
2426 | "red";
2527
2628 /** How many approved pulls waited on each reason. */
src/web/pages/board.ts +1
@@ -153,6 +153,7 @@ const KIND_WORDS: Record<DecisionKind, string> = {
153153 retry: "try again",
154154 lean: "which way",
155155 hand: "hand-over",
156 proof: "proved before the queue",
156157 // Never listed (`listDecisionsForRepos` leaves guards out); named for the type.
157158 guard: "not started",
158159 };
test/merge-queue-proof.test.ts added +304
@@ -0,0 +1,304 @@
1 /**
2 * The proof before the line (#7616). A pull whose own check was the fast
3 * tier joins the merge queue only once the full suite is green on its head
4 * squashed onto main. The lane here is a stand-in: its proof goes red on a
5 * tree holding a `breaks-*` file (a test the pull's change never reached),
6 * and its gate counts the candidates it was asked about.
7 */
8 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
9 import { mkdtemp, rm } from "node:fs/promises";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { createApp, type App } from "../src/app.ts";
13 import { openPull } from "../src/actions/pulls.ts";
14 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
15 import {
16 keepProof,
17 landingsSince,
18 proofOf,
19 proofRedWords,
20 proofsInFlight,
21 resetProofsForTest,
22 STALE_LANDINGS,
23 standing,
24 startProof,
25 } from "../src/checks/proof.ts";
26 import { enqueue, type Queued } from "../src/checks/queue.ts";
27 import { provideChecksForTest, type ChecksHandle } from "../src/checks/service.ts";
28 import { landVerdict } from "../src/checks/tiers.ts";
29 import { listComments } from "../src/core/issues.ts";
30 import { getPull, type Pull } from "../src/core/pulls.ts";
31 import { getRepoById, type Repo } from "../src/core/repos.ts";
32 import { claimRun, createRun, finishRun } from "../src/core/runs.ts";
33 import { createUser, type User } from "../src/core/users.ts";
34 import { sweepGreen } from "../src/crew/autonomy.ts";
35 import { repoDir } from "../src/git/repos.ts";
36 import { logLevel, setLogLevel } from "../src/kernel/log.ts";
37
38 /** A loaded check machine runs git eight times slower: the clock must not decide the verdict. */
39 const SLOW = 120_000;
40
41 describe("the proof before the merge queue", () => {
42 let dataDir: string;
43 let app: App;
44 let alice: User;
45 let level: ReturnType<typeof logLevel>;
46 /** Every proof the lane ran, and every candidate it gated. */
47 const proved: { repo: string; commit: string; ok: boolean }[] = [];
48 const gated: { repo: string; commit: string }[] = [];
49
50 const git = (repo: Repo, ...args: string[]) =>
51 Bun.spawnSync(["git", ...args], { cwd: repoDir(app.reposRoot, repo.ownerName, repo.name) })
52 .stdout.toString()
53 .trim();
54
55 beforeAll(async () => {
56 level = logLevel();
57 setLogLevel("error");
58 dataDir = await mkdtemp(join(tmpdir(), "forge-merge-queue-proof-"));
59 app = createApp({ dataDir, port: 0 });
60 const a = await createUser(app.db, {
61 origin: "seed",
62 trusted: true,
63 username: "alice",
64 email: "[email protected]",
65 password: "a-good-passphrase",
66 adminIfFirst: true,
67 });
68 if (!a.ok) throw new Error(a.error);
69 alice = a.value;
70 provideChecksForTest(app, {
71 gate: async (repo: Repo, _pull: Pull, commitOid: string) => {
72 gated.push({ repo: repo.id, commit: commitOid });
73 return { ok: true, error: "", lines: [] };
74 },
75 prove: async (repo: Repo, _pull: Pull, commitOid: string) => {
76 const files = git(repo, "ls-tree", "-r", "--name-only", commitOid).split("\n");
77 const ok = !files.some((f) => f.startsWith("breaks-"));
78 proved.push({ repo: repo.id, commit: commitOid, ok });
79 return ok
80 ? { ok, error: "", lines: [] }
81 : { ok, error: "red", lines: ["test/unrelated.test.ts:", "(fail) unrelated > still works [1.00ms]"] };
82 },
83 spareSlots: () => 0,
84 } as unknown as ChecksHandle);
85 });
86
87 afterAll(async () => {
88 resetProofsForTest();
89 setLogLevel(level);
90 await app.close?.();
91 await rm(dataDir, { recursive: true, force: true });
92 });
93
94 async function autonomousRepo(name: string, autonomous = true): Promise<Repo> {
95 const made = await createRepo(app, alice, { name, visibility: "private" });
96 if (!made.ok) throw new Error(String(made.error));
97 const seeded = await commitToBranch(app, alice, made.value, {
98 message: "Start",
99 files: [
100 { path: "README.md", content: `# ${name}\n` },
101 { path: "forge.json", content: JSON.stringify({ check: { run: "true", fast: "true" } }) },
102 ],
103 });
104 if (!seeded.ok) throw new Error(JSON.stringify(seeded.error));
105 if (autonomous) app.db.query("UPDATE repo SET autonomy = 'autonomous', autonomy_by = ?2 WHERE id = ?1").run(made.value.id, alice.id);
106 return getRepoById(app.db, made.value.id)!;
107 }
108
109 /** A pull whose own (fast) check passed on its head. */
110 async function greenPull(repo: Repo, path: string, content = `export const n = "${path}";\n`): Promise<Pull> {
111 const pushed = await commitToBranch(app, alice, repo, {
112 branch: `feature/${path}`,
113 base: "main",
114 message: path,
115 files: [{ path, content }],
116 });
117 if (!pushed.ok) throw new Error(JSON.stringify(pushed.error));
118 const opened = await openPull(app, alice, repo, { title: path, baseBranch: "main", headBranch: `feature/${path}` });
119 if (!opened.ok) throw new Error(String(opened.error));
120 const pull = getPull(app.db, repo.id, opened.value.number)!;
121 const check = createRun(app.db, {
122 repoId: repo.id,
123 kind: "check",
124 issueId: pull.id,
125 env: `pull-${pull.number}`,
126 commitOid: pull.headOid,
127 reason: "pull",
128 });
129 claimRun(app.db, check.id);
130 finishRun(app.db, check.id, { state: "ok", summary: "check passed" });
131 return pull;
132 }
133
134 const merged = (repo: Repo, pull: Pull) => getPull(app.db, repo.id, pull.number)!.mergedAt !== null;
135 const tree = (repo: Repo, rev: string) => git(repo, "rev-parse", `${rev}^{tree}`);
136 const mainOf = (repo: Repo) => git(repo, "rev-parse", "refs/heads/main");
137 const until = async (ok: () => boolean): Promise<void> => {
138 for (const end = Date.now() + SLOW; !ok() && Date.now() < end; ) await Bun.sleep(5);
139 };
140
141 test("a pull that breaks an unrelated test is never admitted, and is told the test by name", async () => {
142 const repo = await autonomousRepo("breaks-unrelated");
143 const pull = await greenPull(repo, "breaks-other.ts");
144 expect(await sweepGreen(app, repo)).toBe(0);
145 await until(() => proofOf(app.db, pull.id) !== undefined);
146 // Proved on its head squashed onto main, red, and nothing reached the gate.
147 expect(proved.filter((p) => p.repo === repo.id)).toHaveLength(1);
148 const [proof] = proved.filter((p) => p.repo === repo.id);
149 expect(git(repo, "rev-parse", `${proof!.commit}^`)).toBe(mainOf(repo));
150 expect(proofOf(app.db, pull.id)).toEqual({ head: pull.headOid, base: mainOf(repo), green: false });
151 await until(() => listComments(app.db, pull.id).length > 0);
152 const told = listComments(app.db, pull.id).map((c) => c.body).join("\n");
153 expect(told).toContain("has not joined the merge queue");
154 expect(told).toContain("- unrelated > still works");
155 // The next sweep neither admits it nor proves the same head again.
156 expect(await sweepGreen(app, repo)).toBe(0);
157 expect(proved.filter((p) => p.repo === repo.id)).toHaveLength(1);
158 expect(gated.filter((g) => g.repo === repo.id)).toHaveLength(0);
159 expect(merged(repo, pull)).toBe(false);
160 });
161
162 test("a green proof admits the pull, which lands on the tree it was proved on", async () => {
163 const repo = await autonomousRepo("proved-green");
164 const pull = await greenPull(repo, "fine.ts");
165 // Not admitted on its fast check alone: proved first, beside the sweep.
166 expect(await sweepGreen(app, repo)).toBe(0);
167 await until(() => merged(repo, pull));
168 expect(merged(repo, pull)).toBe(true);
169 const proofs = proved.filter((p) => p.repo === repo.id);
170 expect(proofs).toHaveLength(1);
171 expect(tree(repo, "refs/heads/main")).toBe(tree(repo, proofs[0]!.commit));
172 expect(gated.filter((g) => g.repo === repo.id)).toHaveLength(1);
173 });
174
175 test("a proof main has moved too far past is proved again on the new main before it joins", async () => {
176 const repo = await autonomousRepo("proof-stale");
177 const pull = await greenPull(repo, "late.ts");
178 const old = mainOf(repo);
179 keepProof(app.db, repo.id, pull.id, { head: pull.headOid, base: old, green: true });
180 for (let i = 0; i <= STALE_LANDINGS; i++) {
181 const moved = await commitToBranch(app, alice, repo, {
182 message: `landing ${i}`,
183 files: [{ path: `main-${i}.ts`, content: `${i}\n` }],
184 });
185 if (!moved.ok) throw new Error(JSON.stringify(moved.error));
186 }
187 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
188 expect(await landingsSince(dir, old, mainOf(repo))).toBe(STALE_LANDINGS + 1);
189 const now = mainOf(repo);
190 expect(await sweepGreen(app, repo)).toBe(0);
191 await until(() => merged(repo, pull));
192 expect(merged(repo, pull)).toBe(true);
193 const proofs = proved.filter((p) => p.repo === repo.id);
194 expect(proofs).toHaveLength(1);
195 expect(git(repo, "rev-parse", `${proofs[0]!.commit}^`)).toBe(now);
196 });
197
198 test("an admitted pull whose proof went stale in the line is left out of its batch, unrun", async () => {
199 const repo = await autonomousRepo("stale-in-line");
200 const pulls = [await greenPull(repo, "ok.ts"), await greenPull(repo, "stale.ts")];
201 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
202 const asked: string[] = [];
203 const entries: Queued[] = pulls.map((pull, i) => ({
204 pull,
205 dir,
206 land: async () => undefined,
207 release: () => {},
208 handBack: () => {},
209 proven: async (base) => {
210 asked.push(base);
211 return i === 0;
212 },
213 }));
214 const steps: number[][] = [];
215 const answers = await enqueue(repo.id, entries, async (candidate) => {
216 steps.push(candidate.steps.map((s) => s.entry.pull.number));
217 return { lands: true };
218 });
219 expect(answers).toEqual([false, false]);
220 // The stale one never reached a gate; the fresh one gated alone.
221 expect(steps).toEqual([[pulls[0]!.number]]);
222 expect(asked.length).toBeGreaterThanOrEqual(2);
223 });
224
225 test("a red proof's words name the base, every failing test and the log", () => {
226 const red = landVerdict(false, ["a > b"], ["test/a.test.ts"], "/api/v1/repos/o/r/gates/x/log", undefined, {
227 passed: 9,
228 failed: 3,
229 failing: [
230 { name: "a > b", errors: [] },
231 { name: "c", errors: [] },
232 ],
233 });
234 if (red.lands) throw new Error("green");
235 const words = proofRedWords("main", "0123456789abcdef", red);
236 expect(words).toContain("squashed onto main (0123456)");
237 expect(words).toContain("- a > b\n- c\n- and 1 more");
238 expect(words).toContain("The gate's log: /api/v1/repos/o/r/gates/x/log");
239 });
240
241 test("a proof that never answers is cut off at its deadline: its slot and its pull are let go, no verdict kept", async () => {
242 const repo = await autonomousRepo("proof-hangs", false);
243 const pull = await greenPull(repo, "hang.ts");
244 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
245 const tip = mainOf(repo);
246 let aborted: AbortSignal | undefined;
247 let answered = false;
248 const started = await startProof({
249 db: app.db,
250 repoId: repo.id,
251 pull,
252 dir,
253 tip,
254 // Hung: answers only once its signal aborts, and then too late to count.
255 run: (_commit, signal) =>
256 new Promise((resolve) => {
257 aborted = signal;
258 signal.addEventListener("abort", () => setTimeout(() => resolve({ lands: true }), 50), { once: true });
259 }),
260 room: () => 0,
261 done: () => {
262 answered = true;
263 },
264 deadlineMs: 100,
265 });
266 expect(started).toBe(true);
267 // Running: it holds a slot, and the pull stands `proving`.
268 expect(proofsInFlight()).toBe(1);
269 expect(await standing(app.db, dir, pull, tip)).toBe("proving");
270 await until(() => proofsInFlight() === 0);
271 expect(aborted?.aborted).toBe(true);
272 expect(proofsInFlight()).toBe(0);
273 expect(await standing(app.db, dir, pull, tip)).toBe("unproved");
274 await Bun.sleep(100);
275 expect(answered).toBe(false);
276 expect(proofOf(app.db, pull.id)).toBeUndefined();
277 });
278
279 test("a head that does not merge onto the tip is not proved, and holds no slot", async () => {
280 const repo = await autonomousRepo("proof-conflict", false);
281 const pull = await greenPull(repo, "README.md", "# theirs\n");
282 const moved = await commitToBranch(app, alice, repo, { message: "ours", files: [{ path: "README.md", content: "# ours\n" }] });
283 if (!moved.ok) throw new Error(JSON.stringify(moved.error));
284 const dir = repoDir(app.reposRoot, repo.ownerName, repo.name);
285 let ran = false;
286 const started = await startProof({
287 db: app.db,
288 repoId: repo.id,
289 pull,
290 dir,
291 tip: mainOf(repo),
292 run: async () => {
293 ran = true;
294 return { lands: true };
295 },
296 room: () => 1,
297 done: () => {},
298 });
299 expect(started).toBe(false);
300 expect(ran).toBe(false);
301 expect(proofsInFlight()).toBe(0);
302 expect(await standing(app.db, dir, pull, mainOf(repo))).toBe("unproved");
303 });
304 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs