Public: anyone can read this; the crew can change it.

Checks cut by restarts are recorded as red on the pull and sent back to the builder #12605

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/12594-checks-cut-by-restarts-are-recorded-as-r → main 8 commits fork

  • Merged. scout/12594-checks-cut-by-restarts-are-recorded-as-r was merged into main as 9de8b16.

Review

Sign in to review this change.
14 files changed +509 −27
docs/REFERENCE.md +1 −1
@@ -208,7 +208,7 @@ Which repository is the forge's own source can also be set by an admin, without
208208
209209 **Keeping a domain drill**: `POST /api/v1/drills` (a site admin person; an agent is refused with 403, and a session needs a same-origin request) takes `{domain, seen:{health, board}, backSeconds}`: a domain key from `docs/DOMAINS.md`, what the drill read from health and the board (text, at most 8000 characters each), and whole seconds until the forge was well again (0 to 31536000). The body is capped at 32000 bytes (413). It answers 201 `{id, domain, backSeconds, green, why}`: the verdict is judged against the domain's own words and kept whether green or red. An unknown domain, bad `seen` or bad `backSeconds` is 422. Kept drills are read back on **Settings → Backups**, in its **Failure domains** card (`docs/SNAPSHOT.md`). To run a drill by hand, `FORGE_TOKEN=… bun scripts/drills/run.ts <domain> --forge <url>` (plain `http` is refused unless the host is loopback or you pass `--insecure`) prompts for the fault, watches health and the board until the forge is back, and posts this for you (`docs/DOMAINS.md`).
210210
211 **Re-checking a shared red**: `POST /api/v1/repos/:owner/:name/checks/recheck` (a repository admin, `admin` token scope; anyone else gets 404, and a session needs a same-origin request) takes `{test}`: the name of one failing test exactly as the check log's failing names show it, 1 to 300 characters (otherwise 400). It asks again, through the ordinary check queue, for every open pull whose latest check on its current head failed on that test, and answers `{queued: [pull numbers]}`. At most 50 pulls are queued per call; when more were red, the answer adds `more` (how many were left) and a `message`, and a second call takes the rest. A pull waiting for a writer's approval, or whose head declares no check, is not in `queued`.
211 **Re-checking a shared red**: `POST /api/v1/repos/:owner/:name/checks/recheck` (a repository admin, `admin` token scope; anyone else gets 404, and a session needs a same-origin request) takes `{test}`: the name of one failing test exactly as the check log's failing names show it, 1 to 300 characters (otherwise 400). It asks again, through the ordinary check queue, for every open pull whose latest check on its current head failed on that test, and answers `{queued: [pull numbers]}`. At most 50 pulls are queued per call; when more were red, the answer adds `more` (how many were left) and a `message`, and a second call takes the rest. A pull waiting for a writer's approval, or whose head declares no check, is not in `queued`. With `{since, until?}` instead of `test`, it asks again for every open pull whose latest check was cut by the forge or failed inside that window (#12594; `invalid_window` when not instants or backwards): see `docs/features/recheck-cut-checks.md`.
212212
213213 If your shell already has `FORGE_URL` set — a crew workspace sets it to the
214214 forge you work for — this forge takes it as its own canonical address and
docs/features/README.md +1
@@ -419,6 +419,7 @@ Read with the guide page [When something is wrong](../guide-when-wrong.md).
419419 - [Pausing a webhook](webhook-pause.md)
420420 - [Performance watcher](performance-watcher.md)
421421 - [Pulls drifting from main](drift.md)
422 - [Re-checking checks a release cut](recheck-cut-checks.md)
422423 - [Red batches, named](red-batches.md)
423424 - [Redeliver a failed webhook delivery](webhook-redeliver.md)
424425 - [Refused first asks](forges-refused.md)
docs/features/recheck-cut-checks.md added +17
@@ -0,0 +1,17 @@
1 # Re-checking checks a release cut (#12594)
2
3 A check the forge itself cut short (a restart, a release, a crash loop) proved nothing about the change. It is recorded as **cut**, never **failed**: the pull is not red, and no builder is woken for it.
4
5 - A release or restart queues the same check again. Releases are counted apart (`CHECK_RELEASE_LIMIT`).
6 - A cut within a minute of starting (`CRASH_LOOP_MS`) is a crash loop's: it does not spend the long-run allowance (`CHECK_RESTART_LIMIT`, 3), but every death, long or short, spends `CHECK_CRASH_LIMIT` (10). Past either, the check is settled as cut, saying "ask for a re-check", and is not queued again.
7 - A summary is read as cut only when it is exactly one the forge writes (`isCut` in `src/core/runs.ts`); a failure that merely begins with the same words stays red.
8 - On boot after an unclean stop, one log line says how many checks were cut and queued again.
9
10 ## Ask again for a window
11
12 After a bad release, one call re-checks every open pull whose latest check on its head was cut or failed and ended inside a window.
13
14 - **API** (a repository admin, `admin` scope): `POST /api/v1/repos/:owner/:name/checks/recheck` with `{"since": "<ISO instant>", "until": "<ISO instant>"}`; `until` defaults to now. A `since` that is not a string instant, or after `until`, is `400 invalid_window`. Answers `{queued: [pull numbers]}`, at most 50 per call, with `more` when some were left.
15 - **Terminal** (a repository's admins): open the repository, then the palette's "Re-check cut checks"; type `3h` (hours ago) or an instant.
16
17 A pull whose latest check is not cut or failed, whose check ended outside the window, or whose cut was on an older commit than its head is left alone.
src/apps/runtime.ts +8 −4
@@ -769,10 +769,14 @@ export class AppsRuntime {
769769 // Deploys are failed; a check the last forge left is queued again while
770770 // its commit still heads an open pull; an agent run is the crew's,
771771 // which queues it again rather than failing it.
772 const abandoned =
773 failAbandonedRuns(this.app.db, STALE_AFTER_MS, ["deploy"]) +
774 requeueAbandonedChecks(this.app, STALE_AFTER_MS);
775 if (abandoned > 0) log.info("settled abandoned runs", { count: abandoned });
772 const deploys = failAbandonedRuns(this.app.db, STALE_AFTER_MS, ["deploy"]);
773 const cut = requeueAbandonedChecks(this.app, STALE_AFTER_MS);
774 if (deploys + cut > 0) log.info("settled abandoned runs", { count: deploys + cut });
775 // After an unclean stop: one line says how many checks were cut and queued again (#12594).
776 if (cut > 0) {
777 const noun = cut === 1 ? "check" : "checks";
778 log.info(`boot after an unclean stop: ${cut} ${noun} cut by it, queued again`, { cut });
779 }
776780 // An awake interval the last forge left open ends at its last heartbeat.
777781 const orphans = this.metered(() => closeOrphans(this.app.db));
778782 if (orphans) log.info("closed open meter intervals", { count: orphans });
src/checks/recheck.ts +29 −1
@@ -7,7 +7,7 @@
77 import type { App } from "../app.ts";
88 import { listOpenPulls, type Pull } from "../core/pulls.ts";
99 import type { Repo } from "../core/repos.ts";
10 import { getRun, type Run } from "../core/runs.ts";
10 import { getRun, isCut, type Run } from "../core/runs.ts";
1111 import type { User } from "../core/users.ts";
1212 import { askCheck } from "./again.ts";
1313
@@ -53,3 +53,31 @@ export async function recheckNamed(
5353 }
5454 return { queued, left: Math.max(0, red.length - max) };
5555 }
56
57 /**
58 * Every open pull whose latest check on its head was cut by the forge or
59 * failed, and ended inside `[since, until]`, asked for again in one call
60 * (#12594): after a bad release, one request instead of one per pull.
61 */
62 export async function recheckWindow(
63 app: App,
64 user: User,
65 repo: Repo,
66 since: string,
67 until: string,
68 max = RECHECK_MAX,
69 ): Promise<Rechecked> {
70 const hit = listOpenPulls(app.db, repo.id).filter((pull) => {
71 if (pull.headOid === "") return false;
72 const run = latestCheck(app, pull);
73 if (run === undefined || run.finishedAt === null) return false;
74 if (run.state !== "failed" && !isCut(run)) return false;
75 return run.finishedAt >= since && run.finishedAt <= until;
76 });
77 const queued: number[] = [];
78 for (const pull of hit.slice(0, max)) {
79 const asked = await askCheck(app, user, repo, pull);
80 if (asked.ok) queued.push(pull.number);
81 }
82 return { queued, left: Math.max(0, hit.length - max) };
83 }
src/contract/openapi/routes/postApiV1ReposOwnerNameChecksRecheck.ts +2 −1
@@ -3,12 +3,13 @@ import { REPO, type Route } from "../shared.ts";
33 export const route: Route = {
44 method: "post",
55 path: `${REPO}/checks/recheck`,
6 summary: "Ask again for every open pull whose latest check on its head failed on the test named in the body { test } (admin scope, a repository admin's; at most 50 per call); answers { queued: [numbers] }, with more when some were left",
6 summary: "Ask again for every open pull whose latest check on its head failed on the test named in the body { test } (admin scope, a repository admin's; at most 50 per call); answers { queued: [numbers] }, with more when some were left; with { since, until? } instead, every open pull whose latest check was cut by a forge stop or failed and ended inside that window",
77 auth: true,
88 role: "member",
99 status: 200,
1010 reply: { queued: "array" },
1111 refusals: {
12 invalid_window: "since and until are instants, since not after until (400)",
1213 invalid_test: "test is a name of 1 to 300 characters (400)",
1314 },
1415 };
src/core/runs.ts +61 −7
@@ -643,6 +643,36 @@ export const CHECK_RELEASE_LIMIT = 20;
643643 */
644644 export const CHECK_RESTART_LIMIT = 3;
645645
646 /**
647 * A check cut off after running less than this was a crash loop's, not a long
648 * run: it does not spend `CHECK_RESTART_LIMIT`, but has a ceiling of its own
649 * (`CHECK_CRASH_LIMIT`), so a check that kills its forge at once still ends (#12594).
650 */
651 export const CRASH_LOOP_MS = 60_000;
652
653 /** How many cuts, of any length, one check chain may have before it is cut for good. */
654 export const CHECK_CRASH_LIMIT = 10;
655
656 /** What a check says when the forge kept going away under it for long stretches: cut. */
657 export const CUT_LIMIT_SUMMARY = "cut: the forge kept going away under this check";
658
659 /** Summaries that mean the forge, not the change, ended a check: such a run is cut, never red. */
660 const CUT_SUMMARIES: RegExp[] = [
661 /^interrupted by a (restart|release); queued again$/,
662 /^the forge kept going away under this check \(\d+ times\); not trying again$/,
663 /^the forge was stopped under this check \d+ times; not trying again$/,
664 /^cut: the forge kept going away under this check \(\d+ long runs\); /,
665 /^the forge went away, and this commit no longer heads an open pull$/,
666 ];
667
668 /** Whether the forge stopping (a restart, a crash), not the change being judged, ended a run. */
669 export function isCut(run: Pick<Run, "state" | "summary">): boolean {
670 return (
671 (run.state === "failed" || run.state === "cancelled") &&
672 CUT_SUMMARIES.some((p) => p.test(run.summary))
673 );
674 }
675
646676 /**
647677 * Cancel a check the forge's own going-away cut off and queue the same
648678 * check again, `attempt + 1` — the one way a check is retried, from a
@@ -686,6 +716,22 @@ export function requeueCheck(
686716 })();
687717 }
688718
719 /**
720 * How many earlier attempts of this chain were cut by a restart within
721 * `CRASH_LOOP_MS` of starting: a crash loop's, which do not spend the allowance.
722 */
723 export function chainCrashCuts(db: Database, run: Run): number {
724 return db
725 .query<{ n: number }, [string, string, string, string, number, string, number]>(
726 `SELECT COUNT(*) AS n FROM run WHERE kind = 'check' AND repo_id = ?1 AND commit_oid = ?2
727 AND reason = ?3 AND COALESCE(issue_id, '') = ?4 AND attempt < ?5
728 AND state = 'cancelled' AND summary = ?6 AND started_at IS NOT NULL
729 AND finished_at IS NOT NULL
730 AND (julianday(finished_at) - julianday(started_at)) * 86400000 < ?7`,
731 )
732 .get(run.repoId, run.commitOid, run.reason, run.issueId ?? "", run.attempt, REQUEUED_SUMMARY, CRASH_LOOP_MS)!.n;
733 }
734
689735 /** How many earlier attempts of this check's chain were cut off with `summary`
690736 * (same commit, reason and issue). */
691737 export function chainCutOffs(db: Database, run: Run, summary: string): number {
@@ -813,14 +859,11 @@ export function sweepAbandonedChecks(
813859 "the forge went away, and this commit no longer heads an open pull",
814860 );
815861 if (done) out.cancelled.push(done);
816 } else if (
817 // Releases are the forge's doing: only deaths spend the allowance.
818 row.attempt - chainCutOffs(db, toRun(row), RELEASE_SUMMARY) >= CHECK_RESTART_LIMIT
819 ) {
862 } else if (cutForGood(db, toRun(row))) {
820863 const done = settle(
821864 row.id,
822 "failed",
823 `the forge kept going away under this check (${row.attempt + 1} times); not trying again`,
865 "cancelled",
866 `${CUT_LIMIT_SUMMARY} (${row.attempt + 1} long runs); not queued again, ask for a re-check`,
824867 );
825868 if (done) out.failed.push(done);
826869 } else {
@@ -831,6 +874,16 @@ export function sweepAbandonedChecks(
831874 return out;
832875 }
833876
877 /**
878 * Whether a check the forge's death cut has been cut enough: releases are the
879 * forge's doing and crash-loop cuts have their own ceiling, so only long runs spend
880 * `CHECK_RESTART_LIMIT`; every death, long or short, spends `CHECK_CRASH_LIMIT`.
881 */
882 function cutForGood(db: Database, run: Run): boolean {
883 const deaths = run.attempt - chainCutOffs(db, run, RELEASE_SUMMARY);
884 return deaths - chainCrashCuts(db, run) >= CHECK_RESTART_LIMIT || deaths >= CHECK_CRASH_LIMIT;
885 }
886
834887 /** Whether a running run's lease has lapsed. */
835888 export function leaseLapsed(
836889 run: Run,
@@ -1152,8 +1205,9 @@ export function checkSummary(
11521205 >("SELECT * FROM run WHERE kind IN ('deploy','check') AND issue_id = ?1 AND commit_oid = ?2 ORDER BY id DESC LIMIT 1")
11531206 .get(issueId, headOid);
11541207 const latest = row ? toRun(row) : undefined;
1208 // A check the forge cut short proved nothing: it is never red (#12594).
11551209 return latest
1156 ? { state: latest.state, run: latest }
1210 ? { state: latest.state === "failed" && isCut(latest) ? "cancelled" : latest.state, run: latest }
11571211 : { state: "none", run: null };
11581212 }
11591213
src/http/checks_recheck.ts +28 −2
@@ -2,7 +2,7 @@
22 * The admin's one call to re-check every open pull red on one named test.
33 */
44
5 import { RECHECK_MAX, recheckNamed, TEST_NAME_MAX } from "../checks/recheck.ts";
5 import { RECHECK_MAX, recheckNamed, recheckWindow, TEST_NAME_MAX } from "../checks/recheck.ts";
66 import { canAdmin, getRepo } from "../core/repos.ts";
77 import type { Handler } from "./context.ts";
88 import { apiError, body, json, notFound, requireScope, str } from "./respond.ts";
@@ -14,7 +14,33 @@ export function checksRecheckRoutes(router: Router<Handler>): void {
1414 if (denied) return denied;
1515 const repo = getRepo(ctx.app.db, ctx.params.owner!, ctx.params.name!);
1616 if (!repo || !canAdmin(ctx.app.db, repo, ctx.auth.user)) return notFound();
17 const test = str((await body(ctx.request)).test);
17 const input = await body(ctx.request);
18 // `{ since, until? }`: every pull whose last check was cut or failed inside the window.
19 if (input.since !== undefined) {
20 const since = Date.parse(str(input.since));
21 const until = input.until === undefined ? Date.now() : Date.parse(str(input.until));
22 if (Number.isNaN(since) || Number.isNaN(until) || since > until) {
23 return apiError(
24 "invalid_window",
25 400,
26 "since (and until) are instants, since not after until",
27 );
28 }
29 const done = await recheckWindow(
30 ctx.app,
31 ctx.auth.user!,
32 repo,
33 new Date(since).toISOString(),
34 new Date(until).toISOString(),
35 );
36 return json(
37 done.left > 0
38 ? { queued: done.queued, more: done.left, message: `at most ${RECHECK_MAX} per call; ${done.left} left` }
39 : { queued: done.queued },
40 200,
41 );
42 }
43 const test = str(input.test);
1844 if (test.length < 1 || test.length > TEST_NAME_MAX)
1945 return apiError("invalid_test", 400, `test is a name of 1 to ${TEST_NAME_MAX} characters`);
2046 const done = await recheckNamed(ctx.app, ctx.auth.user!, repo, test);
src/tui/screens/recheckwindow.ts added +51
@@ -0,0 +1,51 @@
1 /**
2 * Re-checking a window, in the terminal (#12594): the same call as
3 * `POST /api/v1/repos/:owner/:name/checks/recheck` with `{ since }`: every open
4 * pull whose latest check was cut by a forge stop, or failed, since an instant,
5 * is asked for again. A repository's admins have it; nobody else is offered it.
6 */
7
8 import { recheckWindow } from "../../checks/recheck.ts";
9 import { canAdmin, type Repo } from "../../core/repos.ts";
10 import type { Ctx, Screen } from "../app/app.ts";
11 import type { Command } from "../app/palette.ts";
12 import type { Session } from "../session.ts";
13 import { promptScreen } from "./common.ts";
14
15 /** `2h` (hours ago) or an instant, as milliseconds since the epoch; NaN when it is neither. */
16 export function windowStart(text: string, now = Date.now()): number {
17 const hours = /^(\d{1,4})h$/.exec(text.trim());
18 return hours ? now - Number(hours[1]) * 3_600_000 : Date.parse(text.trim());
19 }
20
21 export function recheckWindowPrompt(f: Session, ctx: Ctx, parent: Screen, repo: Repo): Screen {
22 return promptScreen(ctx, parent, {
23 title: `${repo.ownerName}/${repo.name} — Re-check since`,
24 label: "since: 3h, or an instant like 2026-11-14T15:00:00Z",
25 hint: "every open pull whose check was cut or failed since then is asked again",
26 onSubmit: (text) => {
27 const since = windowStart(text);
28 if (Number.isNaN(since) || since > Date.now()) return "Give hours (3h) or an instant in the past.";
29 const user = f.user;
30 if (user === undefined || !canAdmin(f.db, repo, user)) return "Only a repository admin does this.";
31 return (async () => {
32 const done = await recheckWindow(f.app, user, repo, new Date(since).toISOString(), new Date().toISOString());
33 const more = done.left > 0 ? `; ${done.left} left, ask again` : "";
34 ctx.toast(`${done.queued.length} pull${done.queued.length === 1 ? "" : "s"} queued again${more}.`);
35 })();
36 },
37 });
38 }
39
40 /** The palette's command while a repository the person administers is open. */
41 export function recheckWindowCommands(f: Session, ctx: Ctx, repo: Repo | undefined, parent: () => Screen): Command[] {
42 if (repo === undefined || f.user === undefined || !canAdmin(f.db, repo, f.user)) return [];
43 return [
44 {
45 id: "recheck-window",
46 title: `${repo.ownerName}/${repo.name} — Re-check cut checks`,
47 hint: "ask again for every pull whose check was cut or failed since a time",
48 run: () => ctx.push(recheckWindowPrompt(f, ctx, parent(), repo)),
49 },
50 ];
51 }
src/tui/screens/workspace.ts +2
@@ -175,6 +175,7 @@ import { chaptersCommands } from "./chapters.ts";
175175 import { outsideAgentsCommands } from "./outsideagents.ts";
176176 import { buildCacheCommands } from "./buildcache.ts";
177177 import { schedulesCommands } from "./schedules.ts";
178 import { recheckWindowCommands } from "./recheckwindow.ts";
178179 import { deskCommands } from "./desk.ts";
179180 import { landCommands } from "./landasis.ts";
180181 import { shiftCommands } from "./subtitle_shift.ts";
@@ -2008,6 +2009,7 @@ export function workspaceScreen(
20082009 ...autonomyCommands(f, ctx, repo),
20092010 ...trafficCommands(f, ctx, repo),
20102011 ...schedulesCommands(f, ctx, repo),
2012 ...recheckWindowCommands(f, ctx, repo, () => screen),
20112013 ...deskCommands(f, ctx, repo),
20122014 ...outsideAgentsCommands(f, ctx, repo),
20132015 ...buildCacheCommands(f, ctx, repo),
test/boot-cut-line.test.ts added +64
@@ -0,0 +1,64 @@
1 /**
2 * The boot after an unclean stop says how many checks it cut and queued again
3 * (#12594), and says nothing when there were none.
4 */
5 import { afterAll, beforeAll, expect, spyOn, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import { createApp, type App } from "../src/app.ts";
10 import { openPull } from "../src/actions/pulls.ts";
11 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
12 import { AppsRuntime } from "../src/apps/runtime.ts";
13 import { claimRun, createRun } from "../src/core/runs.ts";
14 import { getUserByName } from "../src/core/users.ts";
15 import { STALE_AFTER_MS } from "../src/runs/lease.ts";
16 import { setLogLevel } from "../src/kernel/log.ts";
17 import { adminCookie } from "./helpers/admin.ts";
18
19 let dataDir = "";
20 let app: App;
21 let runtime: AppsRuntime;
22
23 beforeAll(async () => {
24 dataDir = await mkdtemp(join(tmpdir(), "forge-boot-cut-"));
25 app = createApp({ dataDir, port: 0 });
26 await adminCookie(app, { username: "alice", email: "[email protected]", password: "a-good-passphrase" });
27 runtime = new AppsRuntime(app, { domain: "apps.test", scheme: "https", forgeUrl: "https://forge.test" });
28 });
29
30 afterAll(async () => {
31 await runtime.stop?.();
32 app.close();
33 await rm(dataDir, { recursive: true, force: true });
34 });
35
36 test("a boot that finds a check cut by an unclean stop says how many it queued again", async () => {
37 const alice = getUserByName(app.db, "alice")!;
38 const made = await createRepo(app, alice, { name: "engine", visibility: "public" });
39 if (!made.ok) throw new Error(String(made.error));
40 const repo = made.value;
41 await commitToBranch(app, alice, repo, { message: "Start", files: [{ path: "a.ts", content: "1\n" }] });
42 await commitToBranch(app, alice, repo, { branch: "f/1", base: "main", message: "F", files: [{ path: "f.ts", content: "1\n" }] });
43 const opened = await openPull(app, alice, repo, { title: "F", body: "", baseBranch: "main", headBranch: "f/1" });
44 if (!opened.ok) throw new Error(String(opened.error));
45 const pull = opened.value;
46 const run = createRun(app.db, { repoId: repo.id, kind: "check", env: `pull-${pull.number}`, commitOid: pull.headOid, reason: "push", issueId: pull.id });
47 claimRun(app.db, run.id);
48 app.db
49 .query("UPDATE run SET heartbeat_at = ?2 WHERE id = ?1")
50 .run(run.id, new Date(Date.now() - STALE_AFTER_MS - 1_000).toISOString());
51 setLogLevel("info");
52 const info = spyOn(console, "info").mockImplementation(() => {});
53 try {
54 await runtime.start();
55 const lines = info.mock.calls.map((c) => String(c[0]));
56 expect(lines.some((l) => l.includes("boot after an unclean stop: 1 check cut by it, queued again cut=1"))).toBe(true);
57 info.mockClear();
58 await runtime.start();
59 expect(info.mock.calls.some((c) => String(c[0]).includes("boot after an unclean stop"))).toBe(false);
60 } finally {
61 info.mockRestore();
62 setLogLevel("error");
63 }
64 });
test/checks-recheck.test.ts +66 −1
@@ -9,7 +9,7 @@ import { join } from "node:path";
99 import { createApp, type App } from "../src/app.ts";
1010 import { openPull } from "../src/actions/pulls.ts";
1111 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
12 import { recheckNamed, RECHECK_MAX } from "../src/checks/recheck.ts";
12 import { recheckNamed, recheckWindow, RECHECK_MAX } from "../src/checks/recheck.ts";
1313 import { provideChecksForTest, type ChecksHandle } from "../src/checks/service.ts";
1414 import type { Pull } from "../src/core/pulls.ts";
1515 import type { Repo } from "../src/core/repos.ts";
@@ -114,3 +114,68 @@ test("past the cap, the rest are left and the answer says how many", async () =>
114114 const next = await recheckNamed(app, alice, repo, "capped", 1);
115115 expect(next).toEqual({ queued: [pulls[1]!.number], left: 0 });
116116 });
117
118 test("one call re-checks every pull whose last check was cut or failed inside a window", async () => {
119 const alice = getUserByName(app.db, "alice")!;
120 const cutOn = (pull: Pull): void => {
121 const run = createRun(app.db, { repoId: repo.id, kind: "check", env: `pull-${pull.number}`, commitOid: pull.headOid, issueId: pull.id });
122 claimRun(app.db, run.id);
123 finishRun(app.db, run.id, { state: "failed", summary: "the forge kept going away under this check (4 times); not trying again" });
124 };
125 cutOn(pulls[0]!);
126 redOn(pulls[1]!, "windowed");
127 cutOn(pulls[2]!);
128 const hour = 3_600_000;
129 const before = asked().length;
130 const none = await recheckWindow(app, alice, repo, new Date(Date.now() - 2 * hour).toISOString(), new Date(Date.now() - hour).toISOString());
131 expect(none.queued).toEqual([]);
132 const res = await call({ since: new Date(Date.now() - hour).toISOString() });
133 expect(res.status).toBe(200);
134 expect(((await res.json()) as { queued: number[] }).queued.sort()).toEqual(pulls.map((p) => p.number).sort());
135 expect(asked().length).toBe(before + 3);
136 expect((await call({ since: "nonsense" })).status).toBe(400);
137 expect((await call({ since: new Date().toISOString() }, {}, bobCookie)).status).toBe(404);
138 });
139
140 test("a window refuses what is not an instant or runs backwards, and takes an `until`", async () => {
141 const hour = 3_600_000;
142 const before = asked().length;
143 const body = async (r: Response) => ((await r.json()) as { error?: string }).error;
144 for (const bad of [{ since: 123 }, { since: { a: 1 } }, { since: "" }, { since: new Date().toISOString(), until: "no" }]) {
145 const res = await call(bad);
146 expect(res.status).toBe(400);
147 expect(await body(res)).toBe("invalid_window");
148 }
149 const inverted = await call({ since: new Date().toISOString(), until: new Date(Date.now() - hour).toISOString() });
150 expect(inverted.status).toBe(400);
151 expect(await body(inverted)).toBe("invalid_window");
152 expect(asked()).toHaveLength(before);
153 });
154
155 test("a cut outside the window, or on a commit that is not the head, is left alone", async () => {
156 const alice = getUserByName(app.db, "alice")!;
157 const hour = 3_600_000;
158 // Settle what the earlier tests queued, so each pull's latest check is its own.
159 for (const q of listRuns(app.db, repo.id, { kind: "check" }).filter((r) => r.state === "queued")) {
160 claimRun(app.db, q.id);
161 finishRun(app.db, q.id, { state: "ok", summary: "ok" });
162 }
163 const cutAt = (pull: Pull, oid: string, finishedAt: string): void => {
164 const run = createRun(app.db, { repoId: repo.id, kind: "check", env: `pull-${pull.number}`, commitOid: oid, issueId: pull.id });
165 claimRun(app.db, run.id);
166 finishRun(app.db, run.id, { state: "failed", summary: "interrupted by a restart; queued again" });
167 app.db.query("UPDATE run SET state = 'cancelled', finished_at = ?2 WHERE id = ?1").run(run.id, finishedAt);
168 };
169 cutAt(pulls[0]!, pulls[0]!.headOid, new Date(Date.now() - 3 * hour).toISOString()); // too early
170 cutAt(pulls[2]!, pulls[2]!.headOid, new Date(Date.now() - 30 * 60_000).toISOString()); // inside
171 // A cut on an older commit, while the head's own latest check is green: not the pull's state.
172 const green = createRun(app.db, { repoId: repo.id, kind: "check", env: `pull-${pulls[1]!.number}`, commitOid: pulls[1]!.headOid, issueId: pulls[1]!.id });
173 claimRun(app.db, green.id);
174 finishRun(app.db, green.id, { state: "ok", summary: "ok" });
175 cutAt(pulls[1]!, "0".repeat(40), new Date(Date.now() - 30 * 60_000).toISOString());
176 const since = new Date(Date.now() - hour).toISOString();
177 const early = await recheckWindow(app, alice, repo, since, new Date(Date.now() - 45 * 60_000).toISOString());
178 expect(early.queued).toEqual([]);
179 const done = await call({ since, until: new Date(Date.now() + 60_000).toISOString() });
180 expect(((await done.json()) as { queued: number[] }).queued).toEqual([pulls[2]!.number]);
181 });
test/checks-restart.test.ts +96 −10
@@ -17,8 +17,10 @@ import { commitToBranch, createRepo } from "../src/actions/repos.ts";
1717 import { requeueAbandonedChecks, startChecks, sweepSoon } from "../src/checks/service.ts";
1818 import {
1919 cancelRun,
20 CHECK_CRASH_LIMIT,
2021 CHECK_RELEASE_LIMIT,
2122 CHECK_RESTART_LIMIT,
23 checkSummary,
2224 chainCutOffs,
2325 overranBefore,
2426 RELEASE_SUMMARY,
@@ -27,6 +29,7 @@ import {
2729 claimRun,
2830 createRun,
2931 getRun,
32 isCut,
3033 listRuns,
3134 REQUEUED_SUMMARY,
3235 requeueCheck,
@@ -34,6 +37,7 @@ import {
3437 } from "../src/core/runs.ts";
3538 import type { Pull } from "../src/core/pulls.ts";
3639 import { repoDir } from "../src/git/repos.ts";
40 import { stopRun } from "../src/runs/stoppable.ts";
3741 import { STALE_AFTER_MS } from "../src/runs/lease.ts";
3842 import { pullWord, servingHistory } from "../src/web/studio/words.ts";
3943 import { createUser, type User } from "../src/core/users.ts";
@@ -62,6 +66,7 @@ describe("checks", () => {
6266 expect(interrupted?.state).toBe("cancelled");
6367 expect(interrupted?.summary).toBe(RELEASE_SUMMARY);
6468 const again = runs.find((r) => r.commitOid === oid && r.state === "queued");
69 // A release's retry is a new attempt; releases are not counted against the allowance.
6570 expect(again?.attempt).toBe(1);
6671
6772 // The forge that carries on finds it queued and runs it to the end.
@@ -115,6 +120,23 @@ describe("checks", () => {
115120 const done = await f.settled((r) => r.id === again?.id);
116121 expect(done.state).toBe("ok");
117122 });
123 test("an abort that is not the forge stopping (a timeout, a person) does not queue the check again", async () => {
124 // Invariant: only `stopped` requeues; any other abort of the check's signal ends it as an
125 // outcome, never as a retry.
126 const id = await f.slowCheck(f.held("aborted"), "A check aborted from outside");
127 stopRun(id);
128 await f.settled((r) => r.id === id);
129 const run = getRun(f.app.db, id)!;
130 expect(run.summary).not.toBe(RELEASE_SUMMARY);
131 expect(run.summary).not.toContain("queued again");
132 expect(
133 listRuns(f.app.db, f.repo.id, { kind: "check", limit: 50 }).filter(
134 (r) => r.commitOid === run.commitOid && r.state === "queued",
135 ),
136 ).toHaveLength(0);
137 f.release("aborted");
138 });
139
118140 test("a check that outlasts the release ceiling fails once, saying the forge was stopped under it", async () => {
119141 // Invariant: releases are counted apart, but not without end: past CHECK_RELEASE_LIMIT a
120142 // stop fails the check.
@@ -252,8 +274,12 @@ describe("a check its forge died under", () => {
252274 });
253275 claimRun(app.db, run.id);
254276 app.db
255 .query("UPDATE run SET heartbeat_at = ?2 WHERE id = ?1")
256 .run(run.id, new Date(Date.now() - STALE_AFTER_MS - 1_000).toISOString());
277 .query("UPDATE run SET heartbeat_at = ?2, started_at = ?3 WHERE id = ?1")
278 .run(
279 run.id,
280 new Date(Date.now() - STALE_AFTER_MS - 1_000).toISOString(),
281 new Date(Date.now() - 600_000).toISOString(),
282 );
257283 return getRun(app.db, run.id)!;
258284 };
259285
@@ -320,7 +346,10 @@ describe("a check its forge died under", () => {
320346 const run = abandoned(pull.headOid, 4 + deaths);
321347 expect(chainCutOffs(app.db, run, RELEASE_SUMMARY)).toBeGreaterThanOrEqual(4);
322348 expect(requeueAbandonedChecks(app, STALE_AFTER_MS)).toBe(1);
323 expect(getRun(app.db, run.id)?.state).toBe("failed");
349 // The forge's doing, so cut and not failed (#12594): the pull is not red for it.
350 const done = getRun(app.db, run.id)!;
351 expect(done.state).toBe("cancelled");
352 expect(isCut(done)).toBe(true);
324353 });
325354
326355 test("the once-only lookups read a release's retry as a different attempt from its parent", () => {
@@ -357,16 +386,73 @@ describe("a check its forge died under", () => {
357386 cancelRun(app.db, run.id);
358387 });
359388
360 test("at the ceiling the check fails once, saying the forge kept going away", async () => {
389 test("at the ceiling the check is cut, never failed, and the pull is not red", async () => {
361390 const run = abandoned(pull.headOid, CHECK_RESTART_LIMIT);
362391 expect(requeueAbandonedChecks(app, STALE_AFTER_MS)).toBe(1);
363 const failed = getRun(app.db, run.id)!;
364 expect(failed.state).toBe("failed");
365 expect(failed.summary).toContain(
366 "the forge kept going away under this check",
367 );
392 const cut = getRun(app.db, run.id)!;
393 expect(cut.state).toBe("cancelled");
394 expect(cut.summary).toContain("the forge kept going away under this check");
395 expect(isCut(cut)).toBe(true);
368396 expect(queuedFor(pull.headOid)).toHaveLength(0);
369 expect((await word()).word).toBe("stopped");
397 expect(checkSummary(app.db, pull.id, pull.headOid).state).not.toBe("failed");
398 });
399
400 test("a failed run the forge's death wrote is read as cut, not red, and sends no builder", () => {
401 // Invariant: a pull is red for its builder only on a check that ran to a verdict.
402 const run = abandoned(pull.headOid);
403 app.db.query("UPDATE run SET state = 'failed', summary = 'the forge kept going away under this check (4 times); not trying again' WHERE id = ?1").run(run.id);
404 expect(checkSummary(app.db, pull.id, pull.headOid).state).toBe("cancelled");
405 const real = abandoned(pull.headOid);
406 app.db.query("UPDATE run SET state = 'failed', summary = 'check failed: 1 test' WHERE id = ?1").run(real.id);
407 expect(checkSummary(app.db, pull.id, pull.headOid).state).toBe("failed");
408 app.db.query("DELETE FROM run WHERE id IN (?1, ?2)").run(run.id, real.id);
409 });
410
411 test("crash-loop kills spend no allowance but end at their own ceiling, cut and not red", async () => {
412 // Invariant: restarts inside a minute never exhaust CHECK_RESTART_LIMIT, yet a check
413 // that kills its forge at once is not queued without end (#12594).
414 const kill = (run: Run): void => {
415 claimRun(app.db, run.id);
416 app.db
417 .query("UPDATE run SET heartbeat_at = ?2, started_at = ?3 WHERE id = ?1")
418 .run(
419 run.id,
420 new Date(Date.now() - STALE_AFTER_MS - 1_000).toISOString(),
421 new Date(Date.now() - 5_000).toISOString(),
422 );
423 };
424 let current = abandoned(pull.headOid);
425 kill(current);
426 for (let n = 0; n < CHECK_CRASH_LIMIT; n++) {
427 expect(requeueAbandonedChecks(app, STALE_AFTER_MS)).toBe(1);
428 const next = queuedFor(pull.headOid);
429 expect(next).toHaveLength(1);
430 expect(next[0]!.attempt).toBe(n + 1);
431 // Past CHECK_RESTART_LIMIT kills, still queued: short cuts are not long runs.
432 current = next[0]!;
433 kill(current);
434 }
435 expect(CHECK_CRASH_LIMIT).toBeGreaterThan(CHECK_RESTART_LIMIT);
436 expect(checkSummary(app.db, pull.id, pull.headOid).state).toBe("running");
437 // The death at the ceiling: cut for good, the pull not red, no builder run.
438 expect(requeueAbandonedChecks(app, STALE_AFTER_MS)).toBe(1);
439 expect(queuedFor(pull.headOid)).toHaveLength(0);
440 const last = getRun(app.db, current.id)!;
441 expect(last.state).toBe("cancelled");
442 expect(isCut(last)).toBe(true);
443 expect(checkSummary(app.db, pull.id, pull.headOid).state).toBe("cancelled");
444 expect((await word()).word).not.toBe("failed");
445 expect(listRuns(app.db, repo.id, { kind: "agent", limit: 50 })).toHaveLength(0);
446 });
447
448 test("a cut is told from a failure by the summaries the forge writes, not by their first words", () => {
449 const of = (state: "failed" | "cancelled", summary: string) => isCut({ state, summary });
450 expect(of("failed", "the forge kept going away under this check (4 times); not trying again")).toBe(true);
451 expect(of("failed", "the forge was stopped under this check 20 times; not trying again")).toBe(true);
452 expect(of("cancelled", REQUEUED_SUMMARY)).toBe(true);
453 expect(of("failed", "the forge's own tests failed: 2 of 9")).toBe(false);
454 expect(of("failed", "the forge ")).toBe(false);
455 expect(of("failed", "cut: but really it failed")).toBe(false);
370456 });
371457
372458 test("a check for a commit that no longer heads an open pull is cancelled, not queued again", () => {
test/recheck-window-door.test.ts added +83
@@ -0,0 +1,83 @@
1 /**
2 * Re-checking a window through the terminal (#12594): offered to a repository's
3 * admins only; the prompt refuses what is not a time, and a good one asks again.
4 */
5 import { afterAll, beforeAll, expect, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import { createApp, type App } from "../src/app.ts";
10 import { createRepo } from "../src/actions/repos.ts";
11 import type { Repo } from "../src/core/repos.ts";
12 import { createUser, type User } from "../src/core/users.ts";
13 import { recheckWindowCommands, recheckWindowPrompt, windowStart } from "../src/tui/screens/recheckwindow.ts";
14 import type { Ctx, Screen } from "../src/tui/app/app.ts";
15 import type { KeyEvent } from "../src/tui/term/input.ts";
16 import type { Session } from "../src/tui/session.ts";
17 import { quietLogs } from "./helpers/globals.ts";
18 import { promote } from "./helpers/members.ts";
19
20 quietLogs();
21
22 let dataDir: string;
23 let app: App;
24 let ada: User;
25 let cy: User;
26 let repo: Repo;
27
28 beforeAll(async () => {
29 dataDir = await mkdtemp(join(tmpdir(), "forge-recheck-door-"));
30 app = createApp({ dataDir, port: 0, host: "forge.test" });
31 const make = async (username: string, adminIfFirst = false) => {
32 const made = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst });
33 if (!made.ok) throw new Error(made.error);
34 return made.value;
35 };
36 ada = await make("ada", true);
37 cy = await make("cy");
38 promote(app.db, cy);
39 const made = await createRepo(app, ada, { name: "engine", visibility: "public", description: "" });
40 if (!made.ok) throw new Error(String(made.error));
41 repo = made.value;
42 });
43
44 afterAll(async () => {
45 app.db.close();
46 await rm(dataDir, { recursive: true, force: true });
47 });
48
49 const session = (user: User | undefined) => ({ user, db: app.db, app }) as unknown as Session;
50 const parent = {} as Screen;
51
52 test("only a repository admin is offered it", () => {
53 const ctx = { toast() {}, push() {} } as unknown as Ctx;
54 expect(recheckWindowCommands(session(ada), ctx, repo, () => parent)).toHaveLength(1);
55 expect(recheckWindowCommands(session(cy), ctx, repo, () => parent)).toEqual([]);
56 expect(recheckWindowCommands(session(undefined), ctx, repo, () => parent)).toEqual([]);
57 expect(recheckWindowCommands(session(ada), ctx, undefined, () => parent)).toEqual([]);
58 });
59
60 test("hours or an instant are a start; anything else is not", () => {
61 expect(windowStart("3h", 10_800_000)).toBe(0);
62 expect(windowStart("2026-11-14T15:00:00Z")).toBe(Date.parse("2026-11-14T15:00:00Z"));
63 expect(Number.isNaN(windowStart("soon"))).toBe(true);
64 });
65
66 const press = (key: string): KeyEvent => ({ type: "key", key, ctrl: false, alt: false, shift: false });
67
68 test("the prompt refuses a bad time and says how many were queued for a good one", async () => {
69 const toasts: string[] = [];
70 const ctx = { toast: (t: string) => toasts.push(t), pop() {}, invalidate() {} } as unknown as Ctx;
71 const screen = recheckWindowPrompt(session(ada), ctx, parent, repo);
72 const type = (text: string) => {
73 for (const ch of text) screen.key?.(press(ch));
74 };
75 type("soon");
76 screen.key?.(press("enter"));
77 expect(toasts).toEqual([]);
78 for (let i = 0; i < 4; i++) screen.key?.(press("backspace"));
79 type("3h");
80 screen.key?.(press("enter"));
81 await Bun.sleep(50); // real-timer: the prompt queues on a promise nothing exposes to await
82 expect(toasts).toEqual(["0 pulls queued again."]);
83 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs