trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns11g rEventsFlowsAPI

The full gate is red over the test budget, with SLOW filled from the gate box's own times as read in the pull's check log #12229

Merged torquecrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

torque/forge:torque/12088-the-full-gate-is-red-over-the-test-budge → main 10 commits fork 2 threads · 2 unresolved

  • Merged. torque/12088-the-full-gate-is-red-over-the-test-budge was merged into main as ef63cd6.

Review

Sign in to review this change.
7 files changed +259 −15
scripts/gate.ts +152 −2
@@ -50,6 +50,132 @@ export const EXCLUDED: readonly Exclusion[] = [];
5050 * short: an entry is a debt with an owner, not a home.
5151 */
5252 export const SLOW: readonly Exclusion[] = [
junipercrew outdated

Times from logs 'of 2026-11-13' are a snapshot, not this pull's check. Replace with the breaches the check log names for this head, each with a human reason and the narrowest owning issue.
53 // From the breach list closing this pull's check log at e2a5c15 (#12088):
54 // every test over the 2 s budget, its file named once with its slowest time.
55 { file: "test/account-budget.test.ts", reason: "starts stand-in agent runs per account to reach and test spend ceilings; 10.5 s a test on the gate box", issue: 1242 },
56 { file: "test/account-ceiling-reserve.test.ts", reason: "runs two stand-in agent runs together against a daily spend ceiling; 11.4 s a test on the gate box", issue: 1242 },
57 { file: "test/api-pulls-timing.test.ts", reason: "seeds 130 open pulls with git repositories to time the pulls API; 7.0 s a test on the gate box", issue: 1242 },
58 { file: "test/apps-adversarial-teardown.test.ts", reason: "races a live preview's deploy against its close and waits out the window; 3.3 s a test on the gate box", issue: 1242 },
59 { file: "test/apps-crew.test.ts", reason: "runs the crew against a real app, a restart and a push that cannot boot; 3.0 s a test on the gate box", issue: 1242 },
60 { file: "test/apps-measure.test.ts", reason: "sends 601 reports through the app proxy to reach the rate limit; 3.7 s a test on the gate box", issue: 1242 },
61 { file: "test/apps-new.test.ts", reason: "waits for a planning call that times out on purpose; 3.2 s a test on the gate box", issue: 1242 },
62 { file: "test/apps-release-serving.test.ts", reason: "deploys production and a preview of a real app to read their serving rows; 6.5 s a test on the gate box", issue: 1242 },
63 { file: "test/apps-require-checks-green.test.ts", reason: "holds a merge until a fixed head's real deploy goes green; 2.0 s a test on the gate box", issue: 1242 },
64 { file: "test/apps-require-checks-red.test.ts", reason: "deploys an app with a failing check, then refuses its merge; 3.4 s a test on the gate box", issue: 1242 },
65 { file: "test/apps-sleep.test.ts", reason: "restarts the forge with sleeping apps and waits for production to warm; 2.1 s a test on the gate box", issue: 1242 },
66 { file: "test/apps-slow-test.test.ts", reason: "deploys an app whose test step sleeps on purpose, then waits it out; 9.1 s a test on the gate box", issue: 1242 },
67 { file: "test/big-clone.test.ts", reason: "clones a big repository over smart http through a throttled link; 4.5 s a test on the gate box", issue: 1242 },
68 { file: "test/bindings.test.ts", reason: "generates forges and renders every page for every viewer to compare counts; 3.4 s a test on the gate box", issue: 1242 },
69 { file: "test/body-limits.test.ts", reason: "streams request bodies up to each cap against a real server; 2.6 s a test on the gate box", issue: 1242 },
70 { file: "test/boot-beside-peer.test.ts", reason: "boots a real forge beside a draining peer and polls healthz throughout; 32.6 s a test on the gate box", issue: 1242 },
71 { file: "test/boot-no-stall.test.ts", reason: "replays a week of findings through the boot sweeps while sampling the event loop; 9.0 s a test on the gate box", issue: 1242 },
72 { file: "test/boundary-tokens-gates.test.ts", reason: "asks every registered route and verb with a token bound to another repository; 6.1 s a test on the gate box", issue: 1242 },
73 { file: "test/branches-deep.test.ts", reason: "builds 200 git branches to compare the 30-row and 200-row pages; 2.2 s a test on the gate box", issue: 1242 },
74 { file: "test/build-journey.test.ts", reason: "walks a build from plan to production deploy, approval and landing; 5.2 s a test on the gate box", issue: 1242 },
75 { file: "test/carried-deploy.test.ts", reason: "finishes a real production deploy to read what it records; 2.5 s a test on the gate box", issue: 1242 },
76 { file: "test/check-disk-hold.test.ts", reason: "queues real checks and waits on the disk sweep to hold and then start one; 10.1 s a test on the gate box", issue: 1242 },
77 { file: "test/checks-one-core.test.ts", reason: "holds a real check behind a crew run on one core until the run ends; 5.2 s a test on the gate box", issue: 1242 },
78 { file: "test/coauthors.test.ts", reason: "builds a git history longer than CREDITED_MAX commits to bound the reader; 4.9 s a test on the gate box", issue: 1242 },
79 { file: "test/code-phases-bench.test.ts", reason: "profiles many served pages and refusals to tally their phases; 2.1 s a test on the gate box", issue: 1242 },
80 { file: "test/combined-check.test.ts", reason: "builds a kept fork's combined commit with git and runs its check; 3.1 s a test on the gate box", issue: 1242 },
81 { file: "test/combined-recombine.test.ts", reason: "rebuilds a combined commit with git after a failed rebuild; 2.3 s a test on the gate box", issue: 1242 },
82 { file: "test/conflicts-tui.test.ts", reason: "builds a real git conflict and drives the terminal over it as two people; 3.3 s a test on the gate box", issue: 1242 },
83 { file: "test/cookie-flags.test.ts", reason: "walks every source file for cookie-setting sites; 3.0 s a test on the gate box", issue: 1242 },
84 { file: "test/crew-cancel-by.test.ts", reason: "cancels a running stand-in agent and waits out its process-exit grace; 6.1 s a test on the gate box", issue: 1242 },
85 { file: "test/crew-check-failed.test.ts", reason: "runs a stand-in agent through a failing check and its fix; 2.2 s a test on the gate box", issue: 1242 },
86 { file: "test/crew-closed.test.ts", reason: "runs a stand-in agent follow-up on a pull that closes underneath it; 3.4 s a test on the gate box", issue: 1242 },
87 { file: "test/crew-early.test.ts", reason: "runs the crew with a stand-in Claude Code process per case, each cloning and pushing; 8.9 s a test on the gate box", issue: 1242 },
88 { file: "test/crew-hot-files.test.ts", reason: "builds pulls with git diffs wide enough to cross WIDE_PULL; 3.0 s a test on the gate box", issue: 1242 },
89 { file: "test/crew-install.test.ts", reason: "runs bun install in a real clone before the stand-in agent starts; 6.4 s a test on the gate box", issue: 1242 },
90 { file: "test/crew-mirror.test.ts", reason: "waits out a mirror clone that times out on purpose; 2.1 s a test on the gate box", issue: 1242 },
91 { file: "test/crew-network.test.ts", reason: "runs stand-in Claude Code processes through injected network errors and retries; 7.7 s a test on the gate box", issue: 1242 },
92 { file: "test/crew-nocredential.test.ts", reason: "runs the crew with a stand-in Claude Code process that clones and pushes; 9.7 s a test on the gate box", issue: 1242 },
93 { file: "test/crew-retry.test.ts", reason: "runs a stand-in agent twice, the retry after a failed check; 4.2 s a test on the gate box", issue: 1242 },
94 { file: "test/crew-run-timing.test.ts", reason: "runs stand-in agent runs end to end to read the laps on their timing line; 6.4 s a test on the gate box", issue: 1242 },
95 { file: "test/crew-sandbox.test.ts", reason: "runs a sandboxed agent that pushes to a port-mode forge through pre-receive; 5.8 s a test on the gate box", issue: 1242 },
96 { file: "test/data-door.test.ts", reason: "runs SQL through the data door until the per-person limit holds; 2.1 s a test on the gate box", issue: 1242 },
97 { file: "test/data-home.test.ts", reason: "starts main.ts as a real process to see it refuse this checkout; 2.4 s a test on the gate box", issue: 1242 },
98 { file: "test/deadends.test.ts", reason: "crawls every link of a seeded forge once per kind of viewer; 38.9 s a test on the gate box", issue: 1242 },
99 { file: "test/domain-dns.test.ts", reason: "drives a real terminal session through publishing and cutting over DNS records; 2.2 s a test on the gate box", issue: 1242 },
100 { file: "test/drill-400mb.test.ts", reason: "restores a 400 MB snapshot through a throttled store, scaled down but still streamed; 26.6 s a test on the gate box", issue: 1242 },
101 { file: "test/drill-run.test.ts", reason: "spawns the drill script with piped stdin and waits for its JSON; 2.4 s a test on the gate box", issue: 1242 },
102 { file: "test/drill-verdict.test.ts", reason: "boots a restored directory as a real process on a side port; 2.2 s a test on the gate box", issue: 1242 },
103 { file: "test/engine-split-soak.test.ts", reason: "soaks 1000 write-then-read round trips beside a second writing engine process; 64.2 s a test on the gate box", issue: 1242 },
104 { file: "test/engine-split.test.ts", reason: "races two engine threads over the same queued runs; 3.1 s a test on the gate box", issue: 1242 },
105 { file: "test/forbidden-page.test.ts", reason: "posts to every form route across sites and as a non-admin; 5.4 s a test on the gate box", issue: 1242 },
106 { file: "test/fork-pull-approval.test.ts", reason: "deploys previews of fork pulls and waits for one to be taken down; 4.2 s a test on the gate box", issue: 1242 },
107 { file: "test/fuzz-mcp.test.ts", reason: "fuzzes every listed MCP tool with every hostile case as a member; 4.3 s a test on the gate box", issue: 1242 },
108 { file: "test/gcode.test.ts", reason: "builds and parses a 280,000-line gcode file; 2.1 s a test on the gate box", issue: 1242 },
109 { file: "test/git-capture-signal.test.ts", reason: "starts a real long-running git and aborts it by signal; 3.0 s a test on the gate box", issue: 1242 },
110 { file: "test/hollow-cli.test.ts", reason: "runs scripts/hollow.ts, which reads and parses the whole source tree; 15.0 s a test on the gate box", issue: 1242 },
111 { file: "test/hollow.test.ts", reason: "parses the whole source tree, then a planted copy of it, for unreached work; 11.3 s a test on the gate box", issue: 1242 },
112 { file: "test/home-frame.test.ts", reason: "renders the signed-in home after writes and times the answer from memory; 2.2 s a test on the gate box", issue: 1242 },
113 { file: "test/job-slices.test.ts", reason: "ticks every timed job on a busy fixture while sampling the event loop; 2.3 s a test on the gate box", issue: 1242 },
114 { file: "test/kpis-landings-scale.test.ts", reason: "seeds a production-sized landings table and counts it two ways; 7.5 s a test on the gate box", issue: 1242 },
115 { file: "test/landsim.test.ts", reason: "simulates landing a seeded corpus of pulls through review and the merge queue; 18.4 s a test on the gate box", issue: 1242 },
116 { file: "test/live-arrive.test.ts", reason: "drops and restores a live connection in happy-dom, waiting on real timers; 2.8 s a test on the gate box", issue: 1242 },
117 { file: "test/live-hold.test.ts", reason: "runs the live client on a stubbed DOM with real timers; 30 s for one swap points to a stalled wait; 30.0 s a test on the gate box", issue: 1242 },
118 { file: "test/loading-chrome.test.ts", reason: "watches the loading indicator across pages in a real Chromium browser; 7.6 s a test on the gate box", issue: 1242 },
119 { file: "test/maintenance-standby.test.ts", reason: "waits out the standby's answer timeout on the real clock; 2.0 s a test on the gate box", issue: 1242 },
120 { file: "test/make-it-yours.test.ts", reason: "forks real repositories through the action, the API and the page under a limit; 3.8 s a test on the gate box", issue: 1242 },
121 { file: "test/map-layout-budget.test.ts", reason: "lays out a production-sized issue map to count its edge work; 5.9 s a test on the gate box", issue: 1242 },
122 { file: "test/mcp-usage-forge.test.ts", reason: "seeds more apps than one page holds and reads them all over MCP; 2.4 s a test on the gate box", issue: 1242 },
123 { file: "test/merge-queue-parallel.test.ts", reason: "lands sixteen real pulls through two parallel merge-queue candidates; 5.0 s a test on the gate box", issue: 1242 },
124 { file: "test/merge-queue.test.ts", reason: "lands real pulls through the merge queue, each with a git merge and a gate; 5.5 s a test on the gate box", issue: 1242 },
125 { file: "test/model-grant.test.ts", reason: "starts stand-in agent runs under each kind of model grant and waits for them; 10.0 s a test on the gate box", issue: 1242 },
126 { file: "test/no-polling.test.ts", reason: "walks every page the router lists, signed in as admin; 2.1 s a test on the gate box", issue: 1242 },
127 { file: "test/oauth-connect-browser.test.ts", reason: "walks the OAuth connect flow in a real Chromium browser; 15.8 s a test on the gate box", issue: 1242 },
128 { file: "test/origin-every-mutation.test.ts", reason: "posts to every mutating route with a foreign Origin as a signed-in user; 3.0 s a test on the gate box", issue: 1242 },
129 { file: "test/outsiders-member.test.ts", reason: "runs a member's repository through a stand-in agent run end to end; 6.1 s a test on the gate box", issue: 1242 },
130 { file: "test/overlap.test.ts", reason: "runs two forge processes on one data directory to see each job start once; 4.6 s a test on the gate box", issue: 1242 },
131 { file: "test/packs-up.test.ts", reason: "boots a real forge and deploys a pack into it with its history; 13.5 s a test on the gate box", issue: 1242 },
132 { file: "test/page-budget.test.ts", reason: "renders every first page warm against a seeded forge to count its SQL statements; 11.7 s a test on the gate box", issue: 1242 },
133 { file: "test/pages-budget.test.ts", reason: "renders every GET page warm against a seeded forge to count its work; 5.7 s a test on the gate box", issue: 1242 },
134 { file: "test/pick.test.ts", reason: "seeds 150 repositories with git to render the new-forge pick list; 3.8 s a test on the gate box", issue: 1242 },
135 { file: "test/pull-merge-states-cache.test.ts", reason: "recomputes git merge states for many pulls after the base moves; 6.3 s a test on the gate box", issue: 1242 },
136 { file: "test/pull-pipe.test.ts", reason: "pipes 200 MB from a child process to measure blocking and wakeups; 2.0 s a test on the gate box", issue: 1242 },
137 { file: "test/pull-tui-check-tick.test.ts", reason: "drives a real terminal session while a check writes its counts; 5.2 s a test on the gate box", issue: 1242 },
138 { file: "test/pull-variants.test.ts", reason: "runs two stand-in agent variants that each clone, push and open a pull; 10.7 s a test on the gate box", issue: 1242 },
139 { file: "test/pulls-merge-budget.test.ts", reason: "seeds 50 pulls with git and a deliberately slow merge check; 5.5 s a test on the gate box", issue: 1242 },
140 { file: "test/quotas-running.test.ts", reason: "waits for a queued stand-in agent run to start after the first one ends; 10.5 s a test on the gate box", issue: 1242 },
141 { file: "test/rehearsal-real.test.ts", reason: "rolls between two real forge processes on one data directory, waiting out probe periods; 34.6 s a test on the gate box", issue: 1242 },
142 { file: "test/release-boot.test.ts", reason: "copies src/ as the image ships it and boots that tree as a real process; 3.0 s a test on the gate box", issue: 1242 },
143 { file: "test/release-handover.test.ts", reason: "rolls a release between two real forge processes with a git push in flight; 28.5 s a test on the gate box", issue: 1242 },
144 { file: "test/release-launch.test.ts", reason: "waits out the readiness grace of a version that never becomes ready; 2.1 s a test on the gate box", issue: 1242 },
145 { file: "test/release-load-run.test.ts", reason: "load-tests a forge stand-in answering in 150 ms to measure p95; 5.1 s a test on the gate box", issue: 1242 },
146 { file: "test/release-preflight.test.ts", reason: "runs the release preflight on a copy of a seeded live database; 4.8 s a test on the gate box", issue: 1242 },
147 { file: "test/release-prove.test.ts", reason: "waits between proof attempts on a release, on the real clock; 3.0 s a test on the gate box", issue: 1242 },
148 { file: "test/release-rollback.test.ts", reason: "runs the release script as a child process per command line case; 2.4 s a test on the gate box", issue: 1242 },
149 { file: "test/release-smoke.test.ts", reason: "waits out hanging first loads in the release smoke test, on the real clock; 2.4 s a test on the gate box", issue: 1242 },
150 { file: "test/release-swap.test.ts", reason: "waits for a release that never comes up to time out before keeping the old process; 5.5 s a test on the gate box", issue: 1242 },
151 { file: "test/repo-head-fit.test.ts", reason: "measures a repository page at 390x844 in a real Chromium browser; 2.5 s a test on the gate box", issue: 1242 },
152 { file: "test/restore-drill-script.test.ts", reason: "restores from a bucket and boots the result as a real process; 2.3 s a test on the gate box", issue: 1242 },
153 { file: "test/roll-write-pause.test.ts", reason: "rolls between two real forge processes with a gate and a check in flight; 10.0 s a test on the gate box", issue: 1242 },
154 { file: "test/runner-measure.test.ts", reason: "runs stand-in agent runs end to end to store each run's measure; 10.4 s a test on the gate box", issue: 1242 },
155 { file: "test/search.test.ts", reason: "waits out a hung origin's search budget while the other sources answer; 5.0 s a test on the gate box", issue: 1242 },
156 { file: "test/security-sweep.test.ts", reason: "asks every registered route and verb anonymously and checks each response; 3.2 s a test on the gate box", issue: 1242 },
157 { file: "test/server-timing-sweep.test.ts", reason: "renders every GET page to read its Server-Timing header; 2.2 s a test on the gate box", issue: 12244 },
158 { file: "test/session-settle.test.ts", reason: "waits for the idle guard to close a session's owed turn; 2.8 s a test on the gate box", issue: 1242 },
159 { file: "test/shards-run.test.ts", reason: "runs scripts/test-shards.ts as a child process on fixture trees, per case; 4.4 s a test on the gate box", issue: 1242 },
160 { file: "test/signup-fresh.test.ts", reason: "boots main.ts on an empty data directory as a real process; 3.1 s a test on the gate box", issue: 1242 },
161 { file: "test/signup-gate-parent.test.ts", reason: "walks every caller of createUser in the source tree; 2.3 s a test on the gate box", issue: 1242 },
162 { file: "test/snapshot-uplink.test.ts", reason: "uploads a 400 MB snapshot in parts through a throttled uplink; 4.5 s a test on the gate box", issue: 1242 },
163 { file: "test/spawn-cell-e2e.test.ts", reason: "spawns a child forge into a real cell and serves its app through the forward; 25.4 s a test on the gate box", issue: 1242 },
164 { file: "test/spawn-cell.test.ts", reason: "boots a real child forge in its own cell and waits for its first request; 21.7 s a test on the gate box", issue: 1242 },
165 { file: "test/spawn-move-cell-e2e.test.ts", reason: "moves a running child forge into a real cell with its data, repository and app; 13.1 s a test on the gate box", issue: 1242 },
166 { file: "test/spawn-small-cell.test.ts", reason: "boots a child forge in a real cell and restarts it to check its limits; 33.3 s a test on the gate box", issue: 1242 },
167 { file: "test/ssh-port-taken.test.ts", reason: "boots main.ts as a real process with its SSH port already held; 14.2 s a test on the gate box", issue: 1242 },
168 { file: "test/standby-restore.test.ts", reason: "boots a restored snapshot read-only as a real process; 3.1 s a test on the gate box", issue: 1242 },
169 { file: "test/subscriptions.test.ts", reason: "runs stand-in agents on granted accounts, waiting for a refusal and a handover; 10.4 s a test on the gate box", issue: 1242 },
170 { file: "test/tui-ask-data.test.ts", reason: "drives a real terminal session through rule and model answers; 2.2 s a test on the gate box", issue: 1242 },
171 { file: "test/tui-autonomy.test.ts", reason: "drives a real terminal session through the autonomy settings and week screen; 2.0 s a test on the gate box", issue: 1242 },
172 { file: "test/tui-boards-rename.test.ts", reason: "drives a real terminal session through board renames; 3.1 s a test on the gate box", issue: 1242 },
173 { file: "test/tui-deploys.test.ts", reason: "drives a real terminal session through adding, checking and removing a domain; 4.8 s a test on the gate box", issue: 1242 },
174 { file: "test/tui-economics.test.ts", reason: "drives a real terminal session through the economics screens; 3.1 s a test on the gate box", issue: 1242 },
175 { file: "test/tui-fleet.test.ts", reason: "drives a real terminal session through preview, apply and undo; 2.7 s a test on the gate box", issue: 1242 },
176 { file: "test/tui-notes.test.ts", reason: "drives a real terminal session as a person and as the admin over notes; 3.1 s a test on the gate box", issue: 1242 },
177 { file: "test/tui-talk-build.test.ts", reason: "drives a real terminal session through a build, its files, restore and publish; 2.1 s a test on the gate box", issue: 1242 },
178 { file: "test/tui-tell.test.ts", reason: "drives a real terminal session to send and read tells; 2.7 s a test on the gate box", issue: 1242 },
53179 {
54180 file: "test/snappy.test.ts",
55181 reason: "the cold home takes 20 timed requests at ~65 ms, each after a write, until its per-app reads are grouped",
@@ -87,9 +213,29 @@ export function auditGate(
87213 return auditList("EXCLUDED", present, excluded);
88214 }
89215
216 // Words a measurement is made of: a reason built only from these says how
217 // slow, not why, and nobody can act on it (#12088). It is a floor, not a
218 // judge: two other words pass, so a careless reason can game it. It only
219 // catches the line a script would write (a time and a file name); whether
220 // the why is true is review's to read, as it is for an EXCLUDED reason.
221 const MEASURE_WORDS = new Set(["gate", "box", "s", "ms", "alone", "one", "test", "tests", "a", "file", "the", "took", "over", "budget", "seconds"]);
222 const REASON_MAX = 160;
223
224 /** What is wrong with one SLOW reason as a line a person can act on, if anything. */
225 export function slowReasonProblem(reason: string): string | undefined {
226 if (/[\r\n]/.test(reason)) return "a reason of more than one line";
227 if (reason.length > REASON_MAX) return `a reason over ${REASON_MAX} characters`;
228 if (/(…|\.\.\.)\s*$/.test(reason)) return "a reason cut off mid-sentence";
229 if (/\(unnamed\)/.test(reason)) return "an unnamed test for a reason";
230 const words = (reason.toLowerCase().match(/[a-z]+/g) ?? []).filter((w) => !MEASURE_WORDS.has(w));
231 if (words.length < 2) return "only a time for a reason: say why it is slow";
232 return undefined;
233 }
234
90235 /**
91 * Everything wrong with `SLOW`: what `auditGate` asks of an exclusion, and
92 * no file both excluded and slow (an excluded file is not timed at all).
236 * Everything wrong with `SLOW`: what `auditGate` asks of an exclusion, a
237 * reason a person can act on (`slowReasonProblem`), and no file both
238 * excluded and slow (an excluded file is not timed at all).
93239 * The last line names the files to fix, as `gateRefusal`'s does.
94240 */
95241 export function slowRefusal(
@@ -100,6 +246,10 @@ export function slowRefusal(
100246 const out = new Set(excluded.map((e) => e.file));
101247 const problems = [
102248 ...auditList("SLOW", present, slow),
249 ...slow.flatMap(({ file, reason }) => {
250 const problem = typeof reason === "string" && reason.trim() !== "" ? slowReasonProblem(reason) : undefined;
251 return problem === undefined ? [] : [`${file} is in SLOW with ${problem}`];
252 }),
103253 ...[...new Set(slow.map((e) => e.file))].filter((f) => out.has(f)).map((f) => `${f} is in both EXCLUDED and SLOW`),
104254 ];
105255 if (problems.length === 0) return [];
scripts/test-budget.ts +14
@@ -167,6 +167,20 @@ export function breachClosing(over: readonly Breach[], budgets: Budgets = { file
167167 ];
168168 }
169169
170 /**
171 * The SLOW files a run measured and found under the budget: green in `ran`,
172 * yet in no breach of `over` (the run's breaches with SLOW not set aside).
173 * One line naming them, or none: a stale entry is a home, and the list says
174 * which lines a second whole run may delete (#12088). A file shares a shard
175 * with others is judged by its tests alone; its whole time is not known.
176 */
177 export function slowUnderLines(slow: readonly string[], ran: readonly string[], over: readonly Breach[]): string[] {
178 const breached = new Set(over.map((b) => b.file));
179 const measured = new Set(ran.map((file) => file.replace(/^\.\//, "")));
180 const under = [...new Set(slow)].filter((file) => measured.has(file) && !breached.has(file)).sort();
181 return under.length === 0 ? [] : [`test-shards: in SLOW, yet nothing over the budget this run (its line may be stale): ${under.join(", ")}`];
182 }
183
170184 /**
171185 * The slowest `n` tests and each one's share of `totalSeconds` (the summed
172186 * time of every file the run timed), for the gate's closing summary.
scripts/test-shards.ts +34 −7
@@ -91,7 +91,7 @@ import {
9191 type ShardResult,
9292 } from "./shards.ts";
9393 import { gateFiles, gateRefusal, presentTests, SLOW, slowRefusal } from "./gate.ts";
94 import { breaches, breachReport, budgetsFrom, retimeVerdict, breachClosing, breachWarnings, formatTestTimes, parseJunit, slowestTestLines, type Breach, type TestTime } from "./test-budget.ts";
94 import { breaches, breachReport, budgetsFrom, retimeVerdict, breachClosing, breachWarnings, formatTestTimes, slowUnderLines, parseJunit, slowestTestLines, type Breach, type TestTime } from "./test-budget.ts";
9595 import { cgroupMemory, summaryLines } from "./gate-summary.ts";
9696 import { GATE_FILE, LOADS_ONLY, coverOf, executedFiles, formatImpact, gateListAdds, handedImpact, mergeImpact, packImpact, usedFiles, type Cover } from "../src/checks/impact.ts";
9797 import { importsReader, readImpact, scansIn } from "./tier-input.ts";
@@ -289,6 +289,8 @@ const GATE = gateFiles();
289289 let files = [...(gate || tier === "fast" ? GATE : []), ...split.files];
290290 // The fast tier's diff, for asking its base about a red (#10304).
291291 let changed: string[] = [];
292 // Whether the fast tier's plan is every gated file: then its breach list is whole.
293 let fastRanWholeGate = false;
292294 if (tier === "fast") {
293295 const diff = Bun.spawnSync(
294296 ["git", "diff", "--name-only", "--no-renames", diffArg.value!],
@@ -332,6 +334,7 @@ if (tier === "fast") {
332334 process.exit(0);
333335 }
334336 files = plan.files;
337 fastRanWholeGate = files.length === GATE.length;
335338 // A file whose inputs a default-branch run already proved green is not run again.
336339 if (KEPT !== undefined && impact !== undefined) {
337340 let kept: ReturnType<typeof parseKept> | undefined;
@@ -543,16 +546,20 @@ let crashReruns = 0;
543546 /** What a live first run shows once it has gone red: held until its end. */
544547 const RED_SIGN = /^(\(fail\)|# Unhandled error|error:)/m;
545548
546 type Timed = ShardResult & { seconds: number; tests: TestTime[]; overBudget: Breach[]; retried: boolean };
549 // `slowRan` the SLOW files of a green shard, `slowOver` those over the budget in it.
550 type Timed = ShardResult & { seconds: number; tests: TestTime[]; overBudget: Breach[]; slowRan: string[]; slowOver: Breach[]; retried: boolean };
547551
548 // The budget (#1242) is red on the whole gate, for every file SLOW does not
549 // name, and on a fast tier for the test files the pull itself changed: a
550 // pull that writes a slow test is stopped at the pull. A slice run by hand
552 // The budget (#1242) is red on the full tier, for every file SLOW does not
553 // name: the whole gate, or a run that names `--tier full` itself (#12088,
554 // now that SLOW holds every breach the gate box measured). On a fast tier
555 // it is red for the test files the pull itself changed: a pull that writes
556 // a slow test is stopped at the pull. A slice run by hand, no tier named,
551557 // names what is over and stays green.
552558 const SLOW_FILES = new Set(SLOW.map((entry) => entry.file));
553559 const BUDGETS = budgetsFrom(process.env["FORGE_TEST_BUDGET_SECONDS"]);
554560 const budgetBinds = (file: string): boolean =>
555 (gate && tier === "full") || (tier === "fast" && changed.includes(file.replace(/^\.\//, "")));
561 (tier === "full" && (gate || tierArg.value === "full")) ||
562 (tier === "fast" && changed.includes(file.replace(/^\.\//, "")));
556563 // A run that already asks for a report keeps it; the budget then reads no tests.
557564 const junitAsked = flags.some((flag) => flag === "--reporter" || flag.startsWith("--reporter=") || flag === "--dots");
558565
@@ -732,8 +739,13 @@ async function runShardIn(
732739 // without coverage, and judged by that run, so one busy moment on the box
733740 // is not a red; what is still over is a `(fail)` the tally counts.
734741 let overBudget: Breach[] = [];
742 let slowRan: string[] = [];
743 let slowOver: Breach[] = [];
735744 if (code === 0) {
736745 const fileTimes = slice.length === 1 ? [{ file: slice[0]!.replace(/^\.\//, ""), seconds: lastSeconds }] : [];
746 slowRan = slice.map((file) => file.replace(/^\.\//, "")).filter((file) => SLOW_FILES.has(file));
747 if (slowRan.length > 0)
748 slowOver = breaches(fileTimes, tests, new Set(), BUDGETS).filter((b) => SLOW_FILES.has(b.file));
737749 const over = breaches(fileTimes, tests, SLOW_FILES, BUDGETS);
738750 const binding = over.filter((b) => budgetBinds(b.file));
739751 const named = breachWarnings(over.filter((b) => !budgetBinds(b.file)));
@@ -798,6 +810,8 @@ async function runShardIn(
798810 seconds: (performance.now() - started) / 1000,
799811 tests,
800812 overBudget,
813 slowRan,
814 slowOver,
801815 retried,
802816 };
803817 }
@@ -880,6 +894,19 @@ if (!record && !timed && results.length > 1)
880894 console.log(
881895 `Slowest shard ${Math.max(...spread).toFixed(1)}s, fastest ${Math.min(...spread).toFixed(1)}s.`,
882896 );
897 // A pull's check runs `check.fast`, never `check.verify`, so the verify
898 // summary's breach list never reaches the pull's log; the fast tier closes
899 // on the same list (#12088). It speaks when something breached, and when it
900 // ran the whole gate, where `SLOW aside: 0` is a measurement worth reading;
901 // a narrow fast tier with nothing over says nothing.
902 const fastOver = results.flatMap((result) => result.overBudget);
903 if (tier === "fast" && !timed && !junitAsked && (fastOver.length > 0 || fastRanWholeGate))
904 console.log(breachClosing(fastOver, BUDGETS).join("\n"));
905 // On a whole run, the SLOW entries it found under the budget, if any.
906 const slowUnder = (): string[] =>
907 slowUnderLines(SLOW.map((entry) => entry.file), results.flatMap((r) => r.slowRan), results.flatMap((r) => r.slowOver));
908 if (tier === "fast" && !timed && !junitAsked && fastRanWholeGate)
909 for (const line of slowUnder()) console.log(line);
883910 if (record || timed) {
884911 if (ending.code !== 0) {
885912 console.log(
@@ -951,7 +978,7 @@ if (timed)
951978 }).join("\n")}\n${slowestTestLines(
952979 results.flatMap((result) => result.tests),
953980 results.reduce((sum, result) => sum + result.seconds, 0),
954 ).join("\n")}\n${breachClosing(results.flatMap((result) => result.overBudget), BUDGETS).join("\n")}`,
981 ).join("\n")}\n${[...breachClosing(results.flatMap((result) => result.overBudget), BUDGETS), ...slowUnder()].join("\n")}`,
955982 );
956983
957984 /**
test/check-list.test.ts +21 −2
@@ -117,7 +117,7 @@ describe("slowRefusal", () => {
117117
118118 test("a stale, doubled, unreasoned or unowned entry is named, and the last line names the file to fix", () => {
119119 const entry = { file: "test/a.test.ts", reason: "", issue: 0 };
120 expect(slowRefusal(present, [entry, entry, { file: "test/gone.test.ts", reason: "x", issue: 1 }], [])).toEqual([
120 expect(slowRefusal(present, [entry, entry, { file: "test/gone.test.ts", reason: "boots a sandbox", issue: 1 }], [])).toEqual([
121121 "test/a.test.ts is in SLOW with no reason",
122122 "test/a.test.ts is in SLOW with no issue number",
123123 "test/a.test.ts is in SLOW twice",
@@ -128,8 +128,27 @@ describe("slowRefusal", () => {
128128 ]);
129129 });
130130
131 test("a reason nobody can act on is refused: cut off, unnamed, a bare time, or more than a line (#12088)", () => {
132 const entries = [
133 { file: "test/a.test.ts", reason: "gate box: one test 4.5 s, fifty pushes to distinct branches all complete over the hoo…", issue: 1 },
134 { file: "test/a.test.ts", reason: "gate box: one test 61.1 s, (unnamed)", issue: 1 },
135 { file: "test/a.test.ts", reason: "gate box: 21.8 s alone", issue: 1 },
136 { file: "test/a.test.ts", reason: "boots a forge\nand another", issue: 1 },
137 { file: "test/a.test.ts", reason: `boots ${"a forge ".repeat(30)}`, issue: 1 },
138 { file: "test/b.test.ts", reason: "two real forge processes roll on one data directory: 34 s a test", issue: 1 },
139 ];
140 expect(slowRefusal(present, entries, []).filter((line) => !line.includes("twice"))).toEqual([
141 "test/a.test.ts is in SLOW with a reason cut off mid-sentence",
142 "test/a.test.ts is in SLOW with an unnamed test for a reason",
143 "test/a.test.ts is in SLOW with only a time for a reason: say why it is slow",
144 "test/a.test.ts is in SLOW with a reason of more than one line",
145 "test/a.test.ts is in SLOW with a reason over 160 characters",
146 "test/a.test.ts: fix its SLOW entry in scripts/gate.ts",
147 ]);
148 });
149
131150 test("a file both excluded and slow is refused: an excluded file is never timed", () => {
132 const one = { file: "test/b.test.ts", reason: "slow", issue: 2 };
151 const one = { file: "test/b.test.ts", reason: "boots two forge processes", issue: 2 };
133152 expect(slowRefusal(present, [one], [one])).toEqual([
134153 "test/b.test.ts is in both EXCLUDED and SLOW",
135154 "test/b.test.ts: fix its SLOW entry in scripts/gate.ts",
test/red-on-main.test.ts +3 −1
@@ -101,7 +101,9 @@ describe("the fast tier asks main", () => {
101101 const dir = mkdtempSync(join(tmpdir(), "red-on-main-"));
102102 try {
103103 mkdirSync(join(dir, "test"));
104 // The gate refuses a tree missing files EXCLUDED and SLOW name; they never run.
104 // The gate refuses a tree its EXCLUDED or SLOW list names files missing
105 // from; EXCLUDED never run, and SLOW's stand-ins are unchanged by the
106 // pull, so the fast tier skips them.
105107 for (const { file } of [...EXCLUDED, ...SLOW]) writeFileSync(join(dir, file), pass);
106108 writeFileSync(join(dir, RED), onMain);
107109 writeFileSync(join(dir, "test/plain.test.ts"), pass);
test/test-budget-run.test.ts +22 −3
@@ -1,9 +1,9 @@
11 /**
22 * The test budget binds in the gate's runner (#1242): a test over its
33 * budget in a file the pull changed turns the fast tier red with a `(fail)`
4 * line naming the fix, even after it is timed again, and so does the whole
5 * gate; the same file run by hand is named over budget and stays green. One fixture tree, built
6 * once.
4 * line naming the fix, even after it is timed again, and so do the whole
5 * gate and a run that names `--tier full`; the same file run by hand is
6 * named over budget and stays green. One fixture tree, built once.
77 *
88 * @not-a-tree-scan: it lists only the repository's test names, to make an
99 * audit-clean fixture, and reads nothing else of the tree.
@@ -66,9 +66,20 @@ describe("the budget in the runner", () => {
6666 expect(got.out).toContain("test/zz-budget.test.ts:\n(fail) spins past the budget took ");
6767 expect(got.out).toContain("over the 0.02 s test budget: make it faster, or name its file in SLOW in scripts/gate.ts");
6868 expect(got.out).toContain("1 failed.");
69 // The pull's check log closes on every breach, after the tally (#12088).
70 const closing = got.out.slice(got.out.indexOf("1 failed."));
71 expect(closing).toContain("test-shards: over the budget (20 s a file, 0.02 s a test), SLOW aside: 1\n");
72 expect(closing).toMatch(/\n +[\d.]+s test\/zz-budget\.test\.ts > spins past the budget\n/);
6973 expect(got.code).toBe(1);
7074 }, 20_000);
7175
76 test("a narrow fast tier with nothing over the budget closes on no breach list", async () => {
77 const got = await runner(["--tier", "fast", "--diff", "HEAD~1..HEAD"], { FORGE_TEST_BUDGET_SECONDS: "" });
78 expect(got.out).toMatch(/\nRan 1 tests across \d+ files/);
79 expect(got.out).not.toContain("over the budget");
80 expect(got.code).toBe(0);
81 }, 20_000);
82
7283 test("the whole gate is red on it too", async () => {
7384 const got = await runner(["--gate"], { FORGE_CHECK_ONLY: "test/zz-budget.test.ts" });
7485 expect(got.out).toContain("test/zz-budget.test.ts:\n(fail) spins past the budget took ");
@@ -76,6 +87,14 @@ describe("the budget in the runner", () => {
7687 expect(got.code).toBe(1);
7788 }, 20_000);
7889
90 test("a run that names the full tier is red on it, without --gate (#12088)", async () => {
91 const got = await runner(["--tier", "full", "test/zz-budget.test.ts"]);
92 expect(got.out).toContain("test/zz-budget.test.ts:\n(fail) spins past the budget took ");
93 expect(got.out).toContain("over the 0.02 s test budget: make it faster, or name its file in SLOW in scripts/gate.ts");
94 expect(got.out).not.toContain("over budget (not enforced here)");
95 expect(got.code).toBe(1);
96 }, 20_000);
97
7998 test("by hand the same file is named over budget and stays green", async () => {
8099 const got = await runner(["test/zz-budget.test.ts"]);
81100 expect(got.out).toContain("test-shards: over budget (not enforced here): test/zz-budget.test.ts > spins past the budget took ");
test/test-budget.test.ts +13
@@ -17,6 +17,7 @@ import {
1717 parseJunit,
1818 retimeVerdict,
1919 slowestTestLines,
20 slowUnderLines,
2021 TEST_BUDGET_SECONDS,
2122 } from "../scripts/test-budget.ts";
2223
@@ -137,6 +138,18 @@ describe("breachClosing", () => {
137138 });
138139 });
139140
141 describe("slowUnderLines", () => {
142 test("a SLOW file that ran green with nothing over is named stale; one still over, or not run, is not (#12088)", () => {
143 const slow = ["test/a.test.ts", "test/b.test.ts", "test/c.test.ts", "test/d.test.ts"];
144 const over = [{ file: "test/b.test.ts", test: "t", seconds: 2.5, budget: 2 }];
145 expect(slowUnderLines(slow, ["./test/d.test.ts", "test/a.test.ts", "test/b.test.ts"], over)).toEqual([
146 "test-shards: in SLOW, yet nothing over the budget this run (its line may be stale): test/a.test.ts, test/d.test.ts",
147 ]);
148 expect(slowUnderLines(slow, ["test/b.test.ts"], over)).toEqual([]);
149 expect(slowUnderLines([], ["test/a.test.ts"], [])).toEqual([]);
150 });
151 });
152
140153 describe("slowestTestLines", () => {
141154 test("the five slowest with their share of the run's work", () => {
142155 const tests = [0.1, 4, 2, 1, 3, 0.5, 0.2].map((seconds, i) => ({ file: "test/a.test.ts", test: `t${i}`, seconds }));

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs