Public: anyone can read this; the crew can change it.
Sentinel: queue:check is past its budget: 14 checks wait behind one lane; the oldest since 15:10; the lane runs #4445 for 4 min (limit 60). #4484
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/4459-sentinel-queue-check-is-past-its-budget → main 1 commit fork 1 thread · 1 unresolved
5 files
changed +117
−3
README.md +1 −1
| @@ -1381,7 +1381,7 @@ kill, which leave nothing in the log themselves). | ||
| 1381 | 1381 | mechanical rather than advisory. |
| 1382 | 1382 | |
| 1383 | 1383 | `docs/FLOW.md` is where every issue stands: `GET /api/v1/repos/:owner/:name/flow` returns each open issue's stage, since, stalled and unblock, plus what went live in the last 24 h. |
| 1384 | Where the whole forge waits is the owner's, on `/boards` (the old `/flow` redirects there, query kept; an agent granted `telemetry:read` reads the same from `/api/v1/telemetry/flow` and the MCP `flow` tool): the flow panel shows each stage from ready to live with what waits, the oldest wait and its capacity, under one sentence naming the bottleneck, and beside each what it let out in the last hour with a 24-hour sparkline, under a line of landings and the median ask→live — every count links to the items it counts. A stalled step is a card on the home (Studio mode) for anyone who can write to the repository: an open pull whose check was never asked for, or is red, past its stage's allowance, with its stage, how long it has been stuck, and one button — Run check or Re-run — that asks for a check at the head and returns to the home. The terminal's **Stalled steps** command (palette) lists the same steps, live, and Enter or `r` does the same re-run; both read `flow/stalled.ts` and act through `flow/unblock.ts`, and neither shows anything to someone who cannot write. A conflicted pull never sits: every tick the crew's open pulls that no longer merge get one bring-up-to-date run each (one in flight per pull); after 2 failed tries in the same conflict, or 24 h since the first, the pull is closed with the conflicting files named and its issue goes back to ready, unassigned, with no strike against the builder. Conflicted pulls are listed oldest first on `/boards`. The terminal's Flow command and the MCP `flow` tool read the same: the sentence, the landings line, and each stage's count out in the last hour and in 24 h. | |
| 1384 | Where the whole forge waits is the owner's, on `/boards` (the old `/flow` redirects there, query kept; an agent granted `telemetry:read` reads the same from `/api/v1/telemetry/flow` and the MCP `flow` tool): the flow panel shows each stage from ready to live with what waits, the oldest wait and its capacity, under one sentence naming the bottleneck, and beside each what it let out in the last hour with a 24-hour sparkline, under a line of landings and the median ask→live — every count links to the items it counts. A stalled step is a card on the home (Studio mode) for anyone who can write to the repository: an open pull whose check was never asked for, or is red, past its stage's allowance, with its stage, how long it has been stuck, and one button — Run check or Re-run — that asks for a check at the head and returns to the home. The terminal's **Stalled steps** command (palette) lists the same steps, live, and Enter or `r` does the same re-run; both read `flow/stalled.ts` and act through `flow/unblock.ts`, and neither shows anything to someone who cannot write. A conflicted pull never sits: every tick the crew's open pulls that no longer merge get one bring-up-to-date run each (one in flight per pull), oldest pull first and only as many as the check lane has slots for, since each ends in a push and a check: check slots minus the checks queued and running minus the runs already in flight, with one always allowed (`crew/conflict_room.ts`, #4459); after 2 failed tries in the same conflict, or 24 h since the first, the pull is closed with the conflicting files named and its issue goes back to ready, unassigned, with no strike against the builder. Conflicted pulls are listed oldest first on `/boards`. The terminal's Flow command and the MCP `flow` tool read the same: the sentence, the landings line, and each stage's count out in the last hour and in 24 h. | |
| 1385 | 1385 | |
| 1386 | 1386 | `/resilience` is a forge admin's too (a guest is sent to sign in, anyone else gets 404): each protection in `docs/THREATS.md` as one line, green, red, untested or "no test yet", judged by the test file it cites in the last full gate of the forge's own repository (`FORGE_REPO`; other repositories' gates never touch it). The terminal's **Resilience** command (palette, admins) lists the same lines. |
| 1387 | 1387 |
src/crew/autonomy.ts +9 −1
| @@ -26,6 +26,7 @@ import { addLearned } from "../checks/learned.ts"; | ||
| 26 | 26 | import { enqueue, type GateCandidate, type Queued, type QueueOptions, queueClashes, queueWords } from "../checks/queue.ts"; |
| 27 | 27 | import { checksFor } from "../checks/service.ts"; |
| 28 | 28 | import { queueDepth } from "../core/capacity.ts"; |
| 29 | import { readConflictRoom } from "./conflict_room.ts"; | |
| 29 | 30 | import { mergeable, resolveOid, treeOf } from "../git/merge.ts"; |
| 30 | 31 | import { repoDir } from "../git/repos.ts"; |
| 31 | 32 | import { CLOSED_NOTE } from "../core/done_on_main.ts"; |
| @@ -1554,11 +1555,14 @@ export function releaseConflicted( | ||
| 1554 | 1555 | * bring-up-to-date run (one in flight per pull). After two finished |
| 1555 | 1556 | * attempts, or 24 h since the first, it is closed and its issue is ready |
| 1556 | 1557 | * again. An approved pull is not this sweep's but `finishApproved`'s. |
| 1558 | * Runs start oldest pull first, only as many as the check lane has room | |
| 1559 | * for (`conflict_room.ts`); the rest wait for a later tick (#4459). | |
| 1557 | 1560 | * Returns how many pulls were acted on. |
| 1558 | 1561 | */ |
| 1559 | 1562 | export async function sweepConflicts(app: App, repo: Repo): Promise<number> { |
| 1560 | 1563 | if (repo.autonomy !== "autonomous") return 0; |
| 1561 | 1564 | let acted = 0; |
| 1565 | let room = readConflictRoom(app.db); | |
| 1562 | 1566 | // Forget pulls that are no longer open: closed, merged or released. |
| 1563 | 1567 | for (const id of new Set([...checked.keys(), ...cleanAt.keys()])) |
| 1564 | 1568 | if (getIssueById(app.db, id)?.state !== "open") { |
| @@ -1566,7 +1570,8 @@ export async function sweepConflicts(app: App, repo: Repo): Promise<number> { | ||
| 1566 | 1570 | cleanAt.delete(id); |
| 1567 | 1571 | asked.delete(id); |
| 1568 | 1572 | } |
| 1569 | for (const pull of listPulls(app.db, repo.id, { sealed: true, state: "open", limit: 50 })) { | |
| 1573 | const open = listPulls(app.db, repo.id, { sealed: true, state: "open", limit: 50 }); | |
| 1574 | for (const pull of open.sort((a, b) => a.number - b.number)) { | |
| 1570 | 1575 | if (!pull.authorIsAgent || pull.mergedAt !== null) continue; |
| 1571 | 1576 | if (getHold(app.db, pull.id) !== undefined) continue; |
| 1572 | 1577 | const dir = await pullDir(app.db, app.reposRoot, repo, pull); |
| @@ -1641,7 +1646,10 @@ export async function sweepConflicts(app: App, repo: Repo): Promise<number> { | ||
| 1641 | 1646 | // push must not be queued again on every tick. |
| 1642 | 1647 | const tried = asked.get(pull.id); |
| 1643 | 1648 | if (tried?.tip === baseTip && tried.n >= RUNS_PER_TIP) continue; |
| 1649 | // Its push is a check: none starts that the lane has no slot for (#4459). | |
| 1650 | if (room <= 0) continue; | |
| 1644 | 1651 | if (bringUpToDate(app, repo, pull, files, since) !== undefined) { |
| 1652 | room--; | |
| 1645 | 1653 | acted++; |
| 1646 | 1654 | asked.set(pull.id, { |
| 1647 | 1655 | tip: baseTip, |
src/crew/conflict_room.ts added +51
| @@ -0,0 +1,51 @@ | ||
| 1 | /** | |
| 2 | * How many bring-up-to-date runs the conflict sweep may start now (#4459). | |
| 3 | * | |
| 4 | * Each one ends in a push, and every push is a full check. When a landing | |
| 5 | * touches a file many open pulls share (docs/DESIGN.md §5d), it conflicts | |
| 6 | * all of them at once: one tick queued 17 conflict runs, and their 17 | |
| 7 | * checks went into the lane together, so the oldest waited past its | |
| 8 | * budget, and the load made the checks flake into retries. Main often | |
| 9 | * moved again before they finished, which conflicted the same pulls once | |
| 10 | * more. | |
| 11 | * | |
| 12 | * So the sweep starts a run only when the lane has a slot for the check | |
| 13 | * it will push: check slots, minus the checks queued and running, minus | |
| 14 | * the conflict runs already in flight (each of those will push one). One | |
| 15 | * run in flight is always allowed, so a busy lane slows the sweep but | |
| 16 | * never stops it. A pull left out waits for a later tick, and if main | |
| 17 | * moves meanwhile, one merge brings it past both landings. | |
| 18 | */ | |
| 19 | ||
| 20 | import type { Database } from "bun:sqlite"; | |
| 21 | import { capacityValue } from "../core/capacity.ts"; | |
| 22 | ||
| 23 | export function conflictRoom( | |
| 24 | /** Checks the forge runs at once (`checks.concurrency`). */ | |
| 25 | slots: number, | |
| 26 | /** Check runs queued or running. */ | |
| 27 | checks: number, | |
| 28 | /** Bring-up-to-date runs queued or running. */ | |
| 29 | inFlight: number, | |
| 30 | ): number { | |
| 31 | const room = slots - checks - inFlight; | |
| 32 | return Math.max(room, inFlight === 0 ? 1 : 0); | |
| 33 | } | |
| 34 | ||
| 35 | /** `conflictRoom` from what the forge holds now. */ | |
| 36 | export function readConflictRoom(db: Database): number { | |
| 37 | const row = db | |
| 38 | .query<{ checks: number; conflicts: number }, []>( | |
| 39 | `SELECT | |
| 40 | COALESCE(SUM(kind = 'check'), 0) AS checks, | |
| 41 | COALESCE(SUM(kind = 'agent' AND reason = 'conflict'), 0) AS conflicts | |
| 42 | FROM run WHERE state IN ('queued','running')`, | |
| 43 | ) | |
| 44 | .get(); | |
| 45 | const slots = Number(capacityValue(db, "checks.concurrency")); | |
| 46 | return conflictRoom( | |
| 47 | Number.isInteger(slots) && slots > 0 ? slots : 1, | |
| 48 | row?.checks ?? 0, | |
| 49 | row?.conflicts ?? 0, | |
| 50 | ); | |
| 51 | } |
test/conflict-room.test.ts added +19
| @@ -0,0 +1,19 @@ | ||
| 1 | import { describe, expect, test } from "bun:test"; | |
| 2 | import { conflictRoom } from "../src/crew/conflict_room.ts"; | |
| 3 | ||
| 4 | describe("conflictRoom: the sweep starts no more pushes than the check lane has slots for (#4459)", () => { | |
| 5 | test("an idle lane takes one run per slot", () => { | |
| 6 | expect(conflictRoom(4, 0, 0)).toBe(4); | |
| 7 | }); | |
| 8 | ||
| 9 | test("checks queued or running and runs in flight each take a slot", () => { | |
| 10 | expect(conflictRoom(4, 1, 0)).toBe(3); | |
| 11 | expect(conflictRoom(4, 1, 2)).toBe(1); | |
| 12 | expect(conflictRoom(4, 2, 2)).toBe(0); | |
| 13 | }); | |
| 14 | ||
| 15 | test("a full lane still lets one run through when none is in flight", () => { | |
| 16 | expect(conflictRoom(1, 14, 0)).toBe(1); | |
| 17 | expect(conflictRoom(4, 14, 1)).toBe(0); | |
| 18 | }); | |
| 19 | }); |
test/conflict_sweep.test.ts +37 −1
| @@ -15,7 +15,8 @@ import { openPull } from "../src/actions/pulls.ts"; | ||
| 15 | 15 | import { commitToBranch, createRepo } from "../src/actions/repos.ts"; |
| 16 | 16 | import { getIssueById } from "../src/core/issues.ts"; |
| 17 | 17 | import { getRepoById, type Repo } from "../src/core/repos.ts"; |
| 18 | import { claimRun, finishRun, listRuns } from "../src/core/runs.ts"; | |
| 18 | import { claimRun, createRun, finishRun, listRuns } from "../src/core/runs.ts"; | |
| 19 | import { setCapacity } from "../src/core/capacity.ts"; | |
| 19 | 20 | import { createUser, getUserByName, type User } from "../src/core/users.ts"; |
| 20 | 21 | import { parkedCount } from "../src/crew/parks.ts"; |
| 21 | 22 | import { createToken } from "../src/core/tokens.ts"; |
| @@ -356,3 +357,38 @@ test("the same main tip asks the crew at most twice", async () => { | ||
| 356 | 357 | expect(runs().length).toBe(2); |
| 357 | 358 | expect(getIssueById(app.db, pull.id)!.state).toBe("open"); |
| 358 | 359 | }); |
| 360 | ||
| 361 | test("one landing that conflicts many pulls starts only as many runs as the check lane has room for, oldest first (#4459)", async () => { | |
| 362 | // Only this test's pulls are open, so the slots are theirs alone. | |
| 363 | app.db.query("UPDATE issue SET state = 'closed' WHERE repo_id = ?1 AND kind = 'pull'").run(repo.id); | |
| 364 | const pulls = [await pullOn("lane-a.ts"), await pullOn("lane-b.ts"), await pullOn("lane-c.ts")]; | |
| 365 | const shared = await commitToBranch(app, alice, repo, { | |
| 366 | branch: "main", | |
| 367 | message: "One landing under all three", | |
| 368 | files: pulls.map((_, i) => ({ | |
| 369 | path: `lane-${"abc"[i]}.ts`, | |
| 370 | content: "export const v = 'main';\n", | |
| 371 | })), | |
| 372 | }); | |
| 373 | if (!shared.ok) throw new Error(JSON.stringify(shared.error)); | |
| 374 | setCapacity(app.db, "checks.concurrency", "3", alice.id); | |
| 375 | // A check already waits: two slots are left for the pushes to come. | |
| 376 | const waiting = createRun(app.db, { repoId: repo.id, kind: "check", env: "main" }); | |
| 377 | try { | |
| 378 | expect(await sweepConflicts(app, fresh())).toBe(2); | |
| 379 | expect(pulls.map((p) => p.runs().length)).toEqual([1, 1, 0]); | |
| 380 | // Nothing is free until a run or the check finishes. | |
| 381 | expect(await sweepConflicts(app, fresh())).toBe(0); | |
| 382 | claimRun(app.db, waiting.id); | |
| 383 | finishRun(app.db, waiting.id, { state: "ok", summary: "" }); | |
| 384 | expect(await sweepConflicts(app, fresh())).toBe(1); | |
| 385 | expect(pulls.map((p) => p.runs().length)).toEqual([1, 1, 1]); | |
| 386 | } finally { | |
| 387 | for (const p of pulls) | |
| 388 | for (const run of p.runs()) { | |
| 389 | claimRun(app.db, run.id); | |
| 390 | finishRun(app.db, run.id, { state: "ok", summary: "" }); | |
| 391 | } | |
| 392 | setCapacity(app.db, "checks.concurrency", null, alice.id); | |
| 393 | } | |
| 394 | }); |
new above
offline
Import is out of order (./conflict_room.ts between ../core and ../git); move it to the other ./ imports so the import-order lint stays quiet.