trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

The gate is every test/*.test.ts minus EXCLUDED: GATE is deleted and a pull adding a test touches no shared file #2084

Merged masoncrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/2071-the-gate-is-every-test-test-ts-minus-exc → main 8 commits fork

  • Merged. mason/2071-the-gate-is-every-test-test-ts-minus-exc was merged into main as b2dc787.

Review

Sign in to review this change.
15 files changed +331 −563
.forgejo/workflows/flake-hunt.yml +2 −2
@@ -3,7 +3,7 @@ name: flake-hunt
33 # run, on a loaded box (scripts/flake-hunt.ts). A file red in some runs and
44 # green in others is flaky: the hunt files an issue on the forge carrying
55 # the failure text, the seed and this run; the quarantine job then moves
6 # every file an open `Flaky: …` issue names from GATE to EXCLUDED and
6 # every file an open `Flaky: …` issue names into EXCLUDED and
77 # proposes that as a pull. Nobody hand-edits the list.
88 #
99 # Flakiness is judged per file across runs, so the gate splits into slices
@@ -83,4 +83,4 @@ jobs:
8383 push -q "${FORGE_URL}/${FORGE_REPO}.git" "HEAD:refs/heads/$branch"
8484 curl -fsS -X POST "$FORGE_API/repos/$FORGE_REPO/pulls" \
8585 -H "Authorization: Bearer $FORGE_TOKEN" -H "Content-Type: application/json" \
86 --data "$(BRANCH="$branch" bun -e 'const b = process.env.BRANCH; console.log(JSON.stringify({title: `Quarantine flaky tests (${b})`, head: b, base: "main", body: "Moved by `bun scripts/flake-hunt.ts quarantine`, not by hand: each file leaves GATE under the open `Flaky:` issue that carries its failure, seed and run."}))')"
86 --data "$(BRANCH="$branch" bun -e 'const b = process.env.BRANCH; console.log(JSON.stringify({title: `Quarantine flaky tests (${b})`, head: b, base: "main", body: "Moved by `bun scripts/flake-hunt.ts quarantine`, not by hand: each file leaves the gate under the open `Flaky:` issue that carries its failure, seed and run."}))')"
CLAUDE.md +6 −5
@@ -85,11 +85,12 @@ bun run src/main.ts # boots on :3000, data in ./data
8585 (the gate list), `test/check-list.test.ts`, `test/hollow.test.ts`. Never
8686 `bun scripts/test-shards.ts --gate` inside a run: the check on your push is
8787 that same gate, and running both doubles the load on a small machine.
88 It runs the files in `GATE` (`scripts/gate.ts`), not the whole suite; every
89 other test file is in `EXCLUDED` with its reason and owning issue. Add a new
90 test file to `GATE`, or the check goes red.
91 - **A flaky test is a red that teaches nothing.** A file joins `GATE` only
92 with ten green random-order runs under load in the pull
88 It runs every `test/*.test.ts` except the few in `EXCLUDED`
89 (`scripts/gate.ts`), each with its reason and owning issue. A new test file
90 is in the gate by existing: edit no list for it.
91 - **A flaky test is a red that teaches nothing.** An excluded file rejoins the
92 gate (its `EXCLUDED` entry deleted) only with ten green random-order runs
93 under load in the pull
9394 (`bun scripts/flake-hunt.ts admit test/x.test.ts`); the nightly hunt files
9495 a `Flaky:` issue and `flake-hunt.ts quarantine` moves the file to
9596 `EXCLUDED`, so never hand-edit that move. Never add `retry` — fix the
README.md +20 −11
@@ -802,7 +802,8 @@ bun run check:record # the same, and rewrites scripts/test-times.json
802802
803803 `bun run check` is the gate: the same command `forge.json` declares, so what
804804 it says here is what the deploy will say. It type-checks once, then runs the
805 test files `scripts/gate.ts` lists in `GATE` as parallel `bun test` shards
805 test files `scripts/gate.ts` admits — every `test/*.test.ts` not in its
806 `EXCLUDED` — as parallel `bun test` shards
806807 (`scripts/test-shards.ts`): one per CPU the process may use, no more than
807808 one per 512 MiB of the memory its cgroup allows (16 on the 16-CPU, 8 GiB
808809 check cell; the old ceiling of eight left half of it idle, #2038), and
@@ -823,8 +824,9 @@ enough to set the gate's floor alone: the studio's and the apps
823824 adversarial suites are split by subject into `test/studio*.test.ts` and
824825 `test/apps-adversarial*.test.ts`.
825826
826 The check's first step is the gate-list audit (`bun scripts/gate.ts`), so a
827 test file listed in neither `GATE` nor `EXCLUDED` fails in under a second,
827 The check's first step is the gate-list audit (`bun scripts/gate.ts`), so an
828 `EXCLUDED` entry for a file that is gone, or one with no reason or issue,
829 fails in under a second,
828830 before install-heavy work. `tsc` then runs `--incremental` with a 2 GB heap,
829831 its build info in `.forge-cache/` (made first with `mkdir -p`: tsc will not
830832 create the directory, and a checkout with no restored cache failed on it,
@@ -887,16 +889,22 @@ in that test. Two things it will not do:
887889 and names the tool (`test/helpers/tools.ts`). Install OpenSSH and they run
888890 again — the skip is conditional on the binary, never on anything else.
889891
890 Every `test/*.test.ts` is either in `GATE` or in `EXCLUDED` with a one-line
891 reason and the issue that owns admitting it; `test/check-list.test.ts` goes
892 red on a file in neither, in both, or named without existing, so a new test
893 file never sits outside the gate silently. A file joins the gate by moving
894 from `EXCLUDED` to `GATE` in a pull whose check is green and whose description
895 carries ten green runs in random order under load:
892 Every `test/*.test.ts` is in the gate unless `EXCLUDED` names it with a
893 one-line reason and the issue that owns admitting it, so a new test file is
894 gated the moment it exists and no pull edits a shared list for it (#2037).
895 `test/check-list.test.ts` goes red on an exclusion named twice, without a
896 reason or issue, or for a file that does not exist. An excluded file rejoins
897 the gate by deleting its `EXCLUDED` entry in a pull whose check is green and
898 whose description carries ten green runs in random order under load:
896899 `bun scripts/flake-hunt.ts admit test/x.test.ts` runs the file ten times, each
897900 its own `bun test --randomize` with a fresh seed beside busy loops, prints the
898901 record and exits non-zero unless all ten are green.
899902
903 A brand-new test file does not need that record: it joins the gate on its
904 own green check, not ten runs. That is accepted on purpose — a flaky new
905 test can turn the gate red once before anyone sees it — and the nightly
906 flake hunt below is the backstop that files it and quarantines it.
907
900908 **No flaky tests.** Nightly, `.forgejo/workflows/flake-hunt.yml` runs every
901909 gate file five times on a loaded box, each run a fresh seed that orders both
902910 the files and (through `--randomize`) the tests inside them, nothing stopped
@@ -907,8 +915,9 @@ dependence, a leaked server, a timeout set near the real duration, a tight
907915 wall-clock bound); then
908916 `bun scripts/flake-hunt.ts quarantine` moves every file an open `Flaky:`
909917 issue the hunt itself filed (its author is the token's user and the forge
910 has not marked it untrusted) names from `GATE` to `EXCLUDED` under that issue and proposes the move
911 as a pull. Nobody hand-edits the list. A file red in every run is broken,
918 has not marked it untrusted) into `EXCLUDED` under that issue and proposes
919 the move as a pull; a file the issue names that no longer exists is skipped,
920 since its exclusion would turn the audit red. Nobody hand-edits the list. A file red in every run is broken,
912921 not flaky, and the check already says so. The fix is the test's shape, never
913922 a retry: `test/flake.test.ts` fails on `--retry`, `--rerun-each`, a bunfig
914923 `retry` or a `{ retry: N }` anywhere the gate runs, and on any test that
scripts/compare-changed.ts +2 −2
@@ -18,6 +18,7 @@ import { tmpdir } from "node:os";
1818 import { junitFiles, redFiles, runsAll, verdict, missed, type Row } from "../src/checks/changed_compare.ts";
1919 import { GATE_FILE, gateListAdds, parseImpact } from "../src/checks/impact.ts";
2020 import { fastSet, importsOf } from "../src/checks/tiers.ts";
21 import { treeGate } from "./gate.ts";
2122
2223 const argv = process.argv.slice(2);
2324 const flag = (name: string, fallback: string): string => {
@@ -106,8 +107,7 @@ for (const pull of pulls) {
106107 if (!existsSync(tree)) git("worktree", "add", "--detach", "-f", tree, oid);
107108 else Bun.spawnSync(["git", "-C", tree, "checkout", "--detach", "--force", oid]);
108109 if (!existsSync(join(tree, "node_modules"))) symlinkSync(join(root, "node_modules"), join(tree, "node_modules"));
109 const gateModule = (await import(`${join(tree, "scripts/gate.ts")}?${oid}`)) as { GATE: string[] };
110 const gate = new Set<string>(gateModule.GATE);
110 const gate = new Set<string>(await treeGate(tree, oid));
111111 const cache = new Map<string, string[]>();
112112 const imports = (file: string): string[] => {
113113 let found = cache.get(file);
scripts/flake-hunt.ts +10 −13
@@ -10,8 +10,8 @@
1010 * seed and the run (FORGE_API, FORGE_REPO, FORGE_TOKEN; FORGE_RUN_URL
1111 * names the run). Exits 1 when it found one.
1212 * bun scripts/flake-hunt.ts quarantine
13 * Moves every file an open `Flaky: test/….test.ts` issue the hunt filed
14 * (author is the token's user, not untrusted) names from GATE to
13 * Puts every file an open `Flaky: test/….test.ts` issue the hunt filed
14 * (author is the token's user, not untrusted) names into
1515 * EXCLUDED under that issue, in scripts/gate.ts. The issue is
1616 * the record; this is the only hand that edits the list for a flake.
1717 * bun scripts/flake-hunt.ts admit test/x.test.ts [--runs 10]
@@ -30,12 +30,12 @@ import {
3030 huntedFile,
3131 HUNT_RUNS,
3232 issueFor,
33 quarantine,
33 quarantineFlaky,
3434 readRun,
3535 shuffled,
3636 type FileRun,
3737 } from "./flake.ts";
38 import { GATE } from "./gate.ts";
38 import { gateFiles, presentTests } from "./gate.ts";
3939 import { parseShardSpec, sliceFiles } from "./shards.ts";
4040
4141 const GATE_FILE = new URL("./gate.ts", import.meta.url).pathname;
@@ -175,7 +175,7 @@ async function hunt(): Promise<number> {
175175 const runs = runsOf(HUNT_RUNS);
176176 const spec = parseShardSpec(valueOf("--shard"));
177177 if (!spec.ok) fail(spec.error);
178 let files = [...GATE];
178 let files = gateFiles();
179179 if (!("none" in spec)) {
180180 const sliced = sliceFiles(files, spec.index, spec.total);
181181 if (!sliced.ok) fail(sliced.error);
@@ -233,14 +233,11 @@ async function hunt(): Promise<number> {
233233 }
234234
235235 async function quarantineAll(): Promise<number> {
236 let source = readFileSync(GATE_FILE, "utf8");
237 const moved: string[] = [];
238 for (const [file, issue] of await openFlakyIssues()) {
239 const next = quarantine(source, file, issue);
240 if (next === source) continue;
241 source = next;
242 moved.push(`${file} (#${issue})`);
243 }
236 const { source, moved } = quarantineFlaky(
237 readFileSync(GATE_FILE, "utf8"),
238 await openFlakyIssues(),
239 new Set(presentTests()),
240 );
244241 if (moved.length === 0) {
245242 console.log("nothing to quarantine");
246243 return 0;
scripts/flake.ts +35 −18
@@ -4,8 +4,8 @@
44 * A test file that is red in one run and green in another, over the same
55 * code, is flaky. `scripts/flake-hunt.ts` runs each gate file several times
66 * in random order and hands every run here; this module names the flaky
7 * files, writes the issue that carries the evidence, and moves a file from
8 * `GATE` to `EXCLUDED` in `scripts/gate.ts` — so nobody hand-edits the list.
7 * files, writes the issue that carries the evidence, and adds a file to
8 * `EXCLUDED` in `scripts/gate.ts` — so nobody hand-edits a quarantine.
99 *
1010 * The unit is the file, judged by `bun test`'s exit code: that is what the
1111 * gate judges, what quarantine moves, and the only verdict a crash between
@@ -173,7 +173,7 @@ export function issueFor(
173173 "",
174174 ...ROOT_CAUSES.map((cause) => `- [ ] ${cause.name} — ${cause.fix}`),
175175 "",
176 `To rejoin the gate: fix it, run \`bun scripts/flake-hunt.ts admit ${flake.file}\`, and put its ${ADMISSION_RUNS}-run record in the pull that moves the file back to \`GATE\`.`,
176 `To rejoin the gate: fix it, run \`bun scripts/flake-hunt.ts admit ${flake.file}\`, and put its ${ADMISSION_RUNS}-run record in the pull that deletes its \`EXCLUDED\` entry.`,
177177 ].join("\n"),
178178 };
179179 }
@@ -190,8 +190,8 @@ export function flakyFile(title: string): string | undefined {
190190 /**
191191 * The file an issue from the forge's list quarantines, or undefined. Only
192192 * an issue the hunt filed counts: its author is the hunt's own user and the
193 * forge has not marked it `untrusted`. Anyone else's `Flaky:` title moves
194 * nothing out of GATE.
193 * forge has not marked it `untrusted`. Anyone else's `Flaky:` title takes
194 * nothing out of the gate.
195195 */
196196 export function huntedFile(
197197 issue: { title?: unknown; authorId?: unknown; untrusted?: unknown },
@@ -213,25 +213,42 @@ const entryFor = (file: string, reason: string, issue: number): string =>
213213 "",
214214 ].join("\n");
215215
216 /** `scripts/gate.ts` with `file` moved from GATE to EXCLUDED under `issue`. */
216 /**
217 * `scripts/gate.ts` with `file` excluded under `issue`: one entry at the end
218 * of `EXCLUDED`. Unchanged when the file is excluded already.
219 */
217220 export function quarantine(
218221 source: string,
219222 file: string,
220223 issue: number,
221224 reason = "flaky: red in one hunt run and green in another",
222225 ): string {
223 const gateStart = source.indexOf("export const GATE");
224 const gateEnd = source.indexOf("\n];", gateStart);
225 const line = ` ${JSON.stringify(file)},\n`;
226 const at = source.indexOf(line, gateStart);
227 if (gateStart === -1 || at === -1 || at > gateEnd) return source;
228 const without = source.slice(0, at) + source.slice(at + line.length);
229 const end = without.indexOf("\n];", without.indexOf("export const EXCLUDED"));
230 return (
231 without.slice(0, end + 1) +
232 entryFor(file, reason, issue) +
233 without.slice(end + 1)
234 );
226 const start = source.indexOf("export const EXCLUDED");
227 const end = source.indexOf("\n];", start);
228 if (start === -1 || end === -1) return source;
229 if (source.slice(start, end).includes(`file: ${JSON.stringify(file)}`)) return source;
230 return source.slice(0, end + 1) + entryFor(file, reason, issue) + source.slice(end + 1);
231 }
232
233 /**
234 * `scripts/gate.ts` with every open hunt issue's file quarantined, and the
235 * `file (#issue)` lines that moved. A file no longer `present` is skipped:
236 * an exclusion for a file that is gone would turn the gate's audit red.
237 */
238 export function quarantineFlaky(
239 source: string,
240 flaky: Iterable<readonly [string, number]>,
241 present: ReadonlySet<string>,
242 ): { source: string; moved: string[] } {
243 const moved: string[] = [];
244 for (const [file, issue] of flaky) {
245 if (!present.has(file)) continue;
246 const next = quarantine(source, file, issue);
247 if (next === source) continue;
248 source = next;
249 moved.push(`${file} (#${issue})`);
250 }
251 return { source, moved };
235252 }
236253
237254 /** A retry anywhere in what the gate runs: the lines that carry one. */
scripts/gate.ts +44 −403
@@ -1,14 +1,16 @@
11 /**
2 * Which test files the gate runs, and why every other one does not.
2 * Which test files the gate runs, and why the few others do not.
33 *
4 * `GATE` is the list `bun scripts/test-shards.ts --gate` runs, so
5 * `forge.json`'s `check.run` names no test file itself. `EXCLUDED` is every
6 * other `test/*.test.ts`, each with the reason it is out and the issue that
7 * owns bringing it in. `test/check-list.test.ts` goes red when a file is in
8 * neither list, in both, or named here without existing — so a new test file
9 * cannot sit outside the gate silently.
4 * The gate is every `test/*.test.ts` minus `EXCLUDED` (`gateFiles`): the
5 * list `bun scripts/test-shards.ts --gate` runs, so `forge.json`'s
6 * `check.run` names no test file itself, and a new test file is gated the
7 * moment it exists — no shared list to edit, nothing to conflict on (#2037).
8 * `EXCLUDED` is the quarantine: each file with the reason it is out and the
9 * issue that owns bringing it back. `test/check-list.test.ts` goes red when
10 * an exclusion names a file that does not exist, twice, or without a reason
11 * or an issue.
1012 *
11 * A file joins the gate by moving from `EXCLUDED` to `GATE` in a pull whose
13 * A file rejoins the gate by deleting its `EXCLUDED` entry in a pull whose
1214 * check is green and whose description carries ten green runs in random
1315 * order under load (`bun scripts/flake-hunt.ts admit <file>`). A flaky file
1416 * leaves it only through `bun scripts/flake-hunt.ts quarantine`, under the
@@ -16,6 +18,7 @@
1618 */
1719
1820 import { readdirSync } from "node:fs";
21 import { join } from "node:path";
1922
2023 export type Exclusion = {
2124 /** Repository-relative path, `test/….test.ts`. */
@@ -26,382 +29,6 @@ export type Exclusion = {
2629 issue: number;
2730 };
2831
29 export const GATE: readonly string[] = [
30 "test/archive.test.ts",
31 "test/tui-archive.test.ts",
32 "test/archive-agent.test.ts",
33 "test/waves-page.test.ts",
34 "test/waves-order.test.ts",
35 "test/domain-drill.test.ts",
36 "test/conflicts.test.ts",
37 "test/lands-once.test.ts",
38 "test/merge-queue.test.ts",
39 "test/conflicts-tui.test.ts",
40 "test/accounts-retire.test.ts",
41 "test/releases-page.test.ts",
42 "test/client-home.test.ts",
43 "test/cloud-use.test.ts",
44 "test/accounts.test.ts",
45 "test/asks-mute.test.ts",
46 "test/feed-chips.test.ts",
47 "test/agents.test.ts",
48 "test/app-domains.test.ts",
49 "test/app.test.ts",
50 "test/apps-idle.test.ts",
51 "test/body-limits.test.ts",
52 "test/boards-query.test.ts",
53 "test/boards-page.test.ts",
54 "test/boards-store.test.ts",
55 "test/boards-sampler.test.ts",
56 "test/turnstile.test.ts",
57 "test/variants-deck.test.ts",
58 "test/variants-surfaces.test.ts",
59 "test/thoughts.test.ts",
60 "test/triage.test.ts",
61 "test/passkeys.test.ts",
62 "test/passkey-recovery.test.ts",
63 "test/signup-passkey.test.ts",
64 "test/webauthn.test.ts",
65 "test/chaos.test.ts",
66 "test/flow-stage.test.ts",
67 "test/resilience.test.ts",
68 "test/flow-card.test.ts",
69 "test/cluster-jobs.test.ts",
70 "test/checks-placement.test.ts",
71 "test/checks-job-vanish.test.ts",
72 "test/fuzz-found.test.ts",
73 "test/fuzz-parsers.test.ts",
74 "test/attach-look.test.ts",
75 "test/protection.test.ts",
76 "test/tui-limbs.test.ts",
77 "test/tui-onyou.test.ts",
78 "test/tui-waves.test.ts",
79 "test/tui-deploys.test.ts",
80 "test/tui-access.test.ts",
81 "test/tui-newforge.test.ts",
82 "test/tui-decider.test.ts",
83 "test/claim-link.test.ts",
84 "test/og-card.test.ts",
85 "test/og-route.test.ts",
86 "test/og-png.test.ts",
87 "test/og-designs.test.ts",
88 "test/router.test.ts",
89 "test/health.test.ts",
90 "test/client-address.test.ts",
91 "test/client-list.test.ts",
92 "test/family-heartbeat.test.ts",
93 "test/contents-api.test.ts",
94 "test/crew-agents-only.test.ts",
95 "test/crew-brief-audience.test.ts",
96 "test/run-log-audience.test.ts",
97 "test/crew.test.ts",
98 "test/crew-already-done.test.ts",
99 "test/crew-check-failed.test.ts",
100 "test/crew-continues.test.ts",
101 "test/crew-declines.test.ts",
102 "test/crew-network.test.ts",
103 "test/crew-pause.test.ts",
104 "test/crew-review.test.ts",
105 "test/crew-review-budget.test.ts",
106 "test/commit-identity.test.ts",
107 "test/enterable-apps.test.ts",
108 "test/event-order.test.ts",
109 "test/refresh-meta.test.ts",
110 "test/unfurl.test.ts",
111 "test/restore-card.test.ts",
112 "test/recovery-doc.test.ts",
113 "test/release-checkout.test.ts",
114 "test/release-preflight.test.ts",
115 "test/release-rollback.test.ts",
116 "test/release-swap.test.ts",
117 "test/release-handover.test.ts",
118 "test/threats-doc.test.ts",
119 "test/returns.test.ts",
120 "test/review-route.test.ts",
121 "test/review-lines.test.ts",
122 "test/review-gate.test.ts",
123 "test/review-from-check.test.ts",
124 "test/review-from-green.test.ts",
125 "test/themes.test.ts",
126 "test/waves.test.ts",
127 "test/cut-wave.test.ts",
128 "test/crew-wrapup.test.ts",
129 "test/crew-limits.test.ts",
130 "test/crew-parks.test.ts",
131 "test/crew-red-main.test.ts",
132 "test/crew-sandbox.test.ts",
133 "test/crew-remotes.test.ts",
134 "test/crew-brief-gate.test.ts",
135 "test/passport.test.ts",
136 "test/passport-edge.test.ts",
137 "test/crew-direction.test.ts",
138 "test/crew-direction-autonomy.test.ts",
139 "test/crew-guard.test.ts",
140 "test/review-waits.test.ts",
141 "test/scale.test.ts",
142 "test/sealed-private-work.test.ts",
143 "test/settings.test.ts",
144 "test/signin-throttle.test.ts",
145 "test/signins.test.ts",
146 "test/sole-owner-suspend.test.ts",
147 "test/standing-every-process.test.ts",
148 "test/steer.test.ts",
149 "test/sealed-git.test.ts",
150 "test/sealed-issues.test.ts",
151 "test/sealed-release.test.ts",
152 "test/sealed-preview.test.ts",
153 "test/session-settle.test.ts",
154 "test/landing.test.ts",
155 "test/decide.test.ts",
156 "test/ledger.test.ts",
157 "test/capacity.test.ts",
158 "test/access-log.test.ts",
159 "test/flow-pipeline.test.ts",
160 "test/flow-throughput.test.ts",
161 "test/limits.test.ts",
162 "test/guard-new-accounts.test.ts",
163 "test/checks.test.ts",
164 "test/dependencies.test.ts",
165 "test/map-closing.test.ts",
166 "test/parked-close.test.ts",
167 "test/autonomy.test.ts",
168 "test/autonomy_sweep.test.ts",
169 "test/conflict_sweep.test.ts",
170 "test/studio.test.ts",
171 "test/studio-changes.test.ts",
172 "test/studio-door.test.ts",
173 "test/studio-runs.test.ts",
174 "test/studio-shipping.test.ts",
175 "test/studio-search.test.ts",
176 "test/studio-words.test.ts",
177 "test/converge-settled.test.ts",
178 "test/following.test.ts",
179 "test/forks.test.ts",
180 "test/front.test.ts",
181 "test/fork-counts.test.ts",
182 "test/fork-flow.test.ts",
183 "test/approve-here.test.ts",
184 "test/fork-pull-approval.test.ts",
185 "test/cross-repo-pull.test.ts",
186 "test/pull-mergeable.test.ts",
187 "test/pull-variants.test.ts",
188 "test/pull-checks-line.test.ts",
189 "test/pulls.test.ts",
190 "test/api-reader-marks.test.ts",
191 "test/app-members.test.ts",
192 "test/apps.test.ts",
193 "test/apps-crew.test.ts",
194 "test/apps-identity.test.ts",
195 "test/apps-new.test.ts",
196 "test/apps-slow-test.test.ts",
197 "test/apps-tests-clause.test.ts",
198 "test/apps-adversarial.test.ts",
199 "test/apps-adversarial-teardown.test.ts",
200 "test/apps-adversarial-test-step.test.ts",
201 "test/apps-adversarial-grants.test.ts",
202 "test/apps-adversarial-rules.test.ts",
203 "test/apps-busy.test.ts",
204 "test/apps-token.test.ts",
205 "test/contain.test.ts",
206 "test/contract.test.ts",
207 "test/subscription-budget.test.ts",
208 "test/surfaces-adversarial.test.ts",
209 "test/orgs.test.ts",
210 "test/suspended-hidden.test.ts",
211 "test/verify-email.test.ts",
212 "test/web.test.ts",
213 "test/git-trust.test.ts",
214 "test/helpers.test.ts",
215 "test/shards.test.ts",
216 "test/gate-summary.test.ts",
217 "test/snapshot-drill.test.ts",
218 "test/drill-verdict.test.ts",
219 "test/drill-board-line.test.ts",
220 "test/snapshot-drill-page.test.ts",
221 "test/snapshot-store.test.ts",
222 "test/snapshot-mirror.test.ts",
223 "test/snapshot-mirror-page.test.ts",
224 "test/migrate.test.ts",
225 "test/schema-additive.test.ts",
226 "test/schema-migrations.test.ts",
227 "test/domains.test.ts",
228 "test/vitals.test.ts",
229 "test/vitals-degraded.test.ts",
230 "test/red-main.test.ts",
231 "test/sentinel.test.ts",
232 "test/tui-sentinel.test.ts",
233 "test/version.test.ts",
234 "test/watches.test.ts",
235 "test/guard.test.ts",
236 "test/asked.test.ts",
237 "test/thought-stream.test.ts",
238 "test/asks.test.ts",
239 "test/review-wait.test.ts",
240 "test/asks-bulk.test.ts",
241 "test/proposal-card.test.ts",
242 "test/asks-waiting-for.test.ts",
243 "test/asks-one-list.test.ts",
244 "test/hand-decider.test.ts",
245 "test/decisions-by-doors.test.ts",
246 "test/standing-rule.test.ts",
247 "test/asks-expiry.test.ts",
248 "test/asks-rank.test.ts",
249 "test/plan-pillars.test.ts",
250 "test/plan-route.test.ts",
251 "test/planner-declined.test.ts",
252 "test/feed.test.ts",
253 "test/feed-choose.test.ts",
254 "test/feed-page.test.ts",
255 "test/card-shell.test.ts",
256 "test/offline.test.ts",
257 "test/share.test.ts",
258 "test/cloud.test.ts",
259 "test/run-secrets.test.ts",
260 "test/reserved-names.test.ts",
261 "test/forge-cloudflare.test.ts",
262 "test/forge-cloudflare-celld.test.ts",
263 "test/branch_tell.test.ts",
264 "test/tell_from.test.ts",
265 "test/usage.test.ts",
266 "test/tokens.test.ts",
267 "test/credentials.test.ts",
268 "test/standing.test.ts",
269 "test/board.test.ts",
270 "test/decider-day.test.ts",
271 "test/spawn-plan.test.ts",
272 "test/spawn-anon.test.ts",
273 "test/spawn-host.test.ts",
274 "test/spawn.test.ts",
275 "test/spawn-api.test.ts",
276 "test/spawn-child-apps.test.ts",
277 "test/spawn-child-runs-apps.test.ts",
278 "test/spawn-first-boot.test.ts",
279 "test/spawn-fresh-forks.test.ts",
280 "test/spawn-keys.test.ts",
281 "test/web-apps.test.ts",
282 "test/web-ui.test.ts",
283 "test/fleet-ops.test.ts",
284 "test/meter.test.ts",
285 "test/told-by.test.ts",
286 "test/highlight.test.ts",
287 "test/diff-edge-cases.test.ts",
288 "test/diff.test.ts",
289 "test/edge-auth.test.ts",
290 "test/own-session.test.ts",
291 "test/edit-words.test.ts",
292 "test/edits-security.test.ts",
293 "test/edits.test.ts",
294 "test/event-audience.test.ts",
295 "test/everyday.test.ts",
296 "test/explore.test.ts",
297 "test/feeds.test.ts",
298 "test/first-login.test.ts",
299 "test/fleet-graph.test.ts",
300 "test/git-local.test.ts",
301 "test/git-ssh.test.ts",
302 "test/git.test.ts",
303 "test/hardening.test.ts",
304 "test/here-manifest.test.ts",
305 "test/image-gif.test.ts",
306 "test/image-graphics.test.ts",
307 "test/image-jpeg.test.ts",
308 "test/image-png.test.ts",
309 "test/issues.test.ts",
310 "test/kernel.test.ts",
311 "test/keys.test.ts",
312 "test/limb_order.test.ts",
313 "test/limbs.test.ts",
314 "test/live-pages.test.ts",
315 "test/live.test.ts",
316 "test/looks.test.ts",
317 "test/mail-caps.test.ts",
318 "test/make-it-yours.test.ts",
319 "test/map.test.ts",
320 "test/pillars.test.ts",
321 "test/carried.test.ts",
322 "test/carried-deploy.test.ts",
323 "test/milestones.test.ts",
324 "test/mcp.test.ts",
325 "test/name-marks.test.ts",
326 "test/new-forge.test.ts",
327 "test/notify.test.ts",
328 "test/onboarding.test.ts",
329 "test/org-owners.test.ts",
330 "test/origins-oauth.test.ts",
331 "test/origins-operator.test.ts",
332 "test/origins-reconnect.test.ts",
333 "test/origins-security-auth.test.ts",
334 "test/origins-security-provider.test.ts",
335 "test/origins-security-push.test.ts",
336 "test/origins-sync.test.ts",
337 "test/origins.test.ts",
338 "test/overlap.test.ts",
339 "test/outsiders-switch.test.ts",
340 "test/check-budget.test.ts",
341 "test/check-proven.test.ts",
342 "test/check-list.test.ts",
343 "test/check-land-main.test.ts",
344 "test/check-tiers.test.ts",
345 "test/changed-compare.test.ts",
346 "test/pull-check-again.test.ts",
347 "test/holds.test.ts",
348 "test/hollow.test.ts",
349 "test/flags.test.ts",
350 "test/home-feed.test.ts",
351 "test/home-walk.test.ts",
352 "test/oauth-discovery.test.ts",
353 "test/oauth-flow.test.ts",
354 "test/oidc.test.ts",
355 "test/data-home.test.ts",
356 "test/tree.test.ts",
357 "test/reports.test.ts",
358 "test/report-discovery.test.ts",
359 "test/apps-report.test.ts",
360 "test/apps-toolbar.test.ts",
361 "test/private-app.test.ts",
362 "test/view-source.test.ts",
363 "test/web-browse.test.ts",
364 "test/web-editor.test.ts",
365 "test/web-phone.test.ts",
366 "test/web-workflows.test.ts",
367 "test/webhooks.test.ts",
368 "test/actions.test.ts",
369 "test/app-env.test.ts",
370 "test/apps-ssh.test.ts",
371 "test/pacing.test.ts",
372 "test/openapi.test.ts",
373 "test/palette.test.ts",
374 "test/qr.test.ts",
375 "test/branches.test.ts",
376 "test/cli-sync.test.ts",
377 "test/code.test.ts",
378 "test/counts.test.ts",
379 "test/crew-loop.test.ts",
380 "test/crew-totals.test.ts",
381 "test/flake.test.ts",
382 "test/crew-locks.test.ts",
383 "test/ssh-site.test.ts",
384 "test/web-code.test.ts",
385 "test/web-origins.test.ts",
386 "test/roll.test.ts",
387 "test/rehearsal.test.ts",
388 "test/tui-tell.test.ts",
389 "test/tui-app.test.ts",
390 "test/tui-apps.test.ts",
391 "test/tui-editor.test.ts",
392 "test/tui-icons.test.ts",
393 "test/tui-input.test.ts",
394 "test/tui-local.test.ts",
395 "test/tui-mouse.test.ts",
396 "test/tui-render.test.ts",
397 "test/tui-screens.test.ts",
398 "test/tui-size.test.ts",
399 "test/tui-widgets.test.ts",
400 "test/tui-width.test.ts",
401 "test/tui-workflows.test.ts",
402 "test/tui-workspace.test.ts",
403 ];
404
40532 export const EXCLUDED: readonly Exclusion[] = [
40633 { file: "test/crew-sandbox-real.test.ts", reason: "landed unlisted and red locally; the builder admits it", issue: 1238 },
40734 { file: "test/api-errors.test.ts", reason: "not yet admitted", issue: 261 },
@@ -487,38 +114,26 @@ export const EXCLUDED: readonly Exclusion[] = [
487114 ];
488115
489116 /**
490 * Everything wrong with a gate list, against the test files that exist:
491 * one line per problem, none when every file is accounted for exactly once.
117 * Everything wrong with the quarantine, against the test files that exist:
118 * one line per problem, none when every exclusion is sound.
492119 */
493120 export function auditGate(
494121 present: readonly string[],
495 gate: readonly string[],
496122 excluded: readonly Exclusion[],
497123 ): string[] {
498124 const problems: string[] = [];
499125 const exists = new Set(present);
500 const gated = new Set<string>();
501 for (const file of gate) {
502 if (gated.has(file)) problems.push(`${file} is in GATE twice`);
503 gated.add(file);
504 if (!exists.has(file))
505 problems.push(`${file} is in GATE but does not exist`);
506 }
507126 const out = new Set<string>();
508127 for (const { file, reason, issue } of excluded) {
509128 if (out.has(file)) problems.push(`${file} is in EXCLUDED twice`);
510129 out.add(file);
511130 if (!exists.has(file))
512131 problems.push(`${file} is in EXCLUDED but does not exist`);
513 if (gated.has(file)) problems.push(`${file} is in both GATE and EXCLUDED`);
514132 if (typeof reason !== "string" || reason.trim() === "")
515133 problems.push(`${file} is excluded with no reason`);
516134 if (!Number.isInteger(issue) || issue <= 0)
517135 problems.push(`${file} is excluded with no issue number`);
518136 }
519 for (const file of present)
520 if (!gated.has(file) && !out.has(file))
521 problems.push(`${file} is in neither GATE nor EXCLUDED`);
522137 return problems;
523138 }
524139
@@ -534,24 +149,50 @@ export function presentTests(root = "."): string[] {
534149 }
535150 }
536151
152 /** What the gate runs: every present test file not excluded, in name order. */
153 export function gateFiles(
154 present: readonly string[] = presentTests(),
155 excluded: readonly Exclusion[] = EXCLUDED,
156 ): string[] {
157 const out = new Set(excluded.map((e) => e.file));
158 return present.filter((file) => !out.has(file));
159 }
160
537161 /**
538 * What the check prints when the list is wrong: every problem, then one
539 * last line naming the files to list. Empty when the list is right.
162 * The gate a checked-out tree's own `scripts/gate.ts` names, whichever shape
163 * it has: a tree from before #2071 lists it in `GATE`; a later one derives
164 * it with `gateFiles` from the tree's test files. `key` makes each tree's
165 * import its own module, so one process can read many commits.
166 */
167 export async function treeGate(tree: string, key: string): Promise<string[]> {
168 const module = (await import(`${join(tree, "scripts/gate.ts")}?${key}`)) as {
169 GATE?: readonly string[];
170 gateFiles?: (present: readonly string[]) => string[];
171 presentTests?: (root: string) => string[];
172 };
173 if (module.gateFiles && module.presentTests)
174 return module.gateFiles(module.presentTests(tree));
175 return [...(module.GATE ?? [])];
176 }
177
178 /**
179 * What the check prints when the quarantine is wrong: every problem, then
180 * one last line naming the files to fix. Empty when it is right.
540181 */
541182 export function gateRefusal(present: readonly string[]): string[] {
542 const problems = auditGate(present, GATE, EXCLUDED);
183 const problems = auditGate(present, EXCLUDED);
543184 if (problems.length === 0) return [];
544185 const named = [
545186 ...new Set(problems.map((p) => p.match(/^(test\/\S+\.test\.ts) /)?.[1])),
546187 ].filter((f): f is string => f !== undefined);
547188 return [
548189 ...problems,
549 `${named.join(", ")}: list it in GATE or EXCLUDED in scripts/gate.ts`,
190 `${named.join(", ")}: fix its EXCLUDED entry in scripts/gate.ts`,
550191 ];
551192 }
552193
553194 // `bun scripts/gate.ts`: the audit alone, first in `forge.json`'s check, so
554 // an unlisted file fails in under a second — not after install and tsc.
195 // a stale exclusion fails in under a second — not after install and tsc.
555196 if (import.meta.main) {
556197 const refusal = gateRefusal(presentTests());
557198 for (const line of refusal) console.error(`gate: ${line}`);
scripts/test-shards.ts +6 −4
@@ -8,7 +8,8 @@
88 * bun scripts/test-shards.ts --record [bun test flags…] test/a.test.ts …
99 * bun scripts/test-shards.ts [bun test flags…] -- a b … (test/a.test.ts …)
1010 *
11 * `--gate` runs the files `scripts/gate.ts` admits, which is how
11 * `--gate` runs the files `scripts/gate.ts` admits (every test file not
12 * excluded), which is how
1213 * `forge.json`'s `check.run` names them; files on the command line are for
1314 * running a slice by hand. `--shard N/M` (or `FORGE_CHECK_SHARD_INDEX` +
1415 * `FORGE_CHECK_SHARD_TOTAL`) keeps only the Nth contiguous slice of that
@@ -59,7 +60,7 @@ import {
5960 weigher,
6061 type ShardResult,
6162 } from "./shards.ts";
62 import { GATE, gateRefusal, presentTests } from "./gate.ts";
63 import { gateFiles, gateRefusal, presentTests } from "./gate.ts";
6364 import { cgroupMemory, summaryLines } from "./gate-summary.ts";
6465 import { GATE_FILE, executedFiles, formatImpact, gateListAdds, handedImpact, mergeImpact, parseImpact } from "../src/checks/impact.ts";
6566 import { fastSet, importsOf, duration } from "../src/checks/tiers.ts";
@@ -223,8 +224,8 @@ if (!split.ok) {
223224 process.exit(2);
224225 }
225226 const flags = split.flags;
226 // A test file in neither GATE nor EXCLUDED costs a whole gate run to find
227 // out; say so before anything spawns. Explicit files alone are not audited.
227 // A stale exclusion costs a whole gate run to find out; say so before
228 // anything spawns. Explicit files alone are not audited.
228229 if (gate || tier === "fast") {
229230 const refusal = gateRefusal(presentTests());
230231 if (refusal.length > 0) {
@@ -232,6 +233,7 @@ if (gate || tier === "fast") {
232233 process.exit(2);
233234 }
234235 }
236 const GATE = gateFiles();
235237 let files = [...(gate || tier === "fast" ? GATE : []), ...split.files];
236238 if (tier === "fast") {
237239 const diff = Bun.spawnSync(
src/checks/impact.ts +11 −11
@@ -91,15 +91,16 @@ export function formatImpact(map: ReadonlyMap<string, readonly string[]>): strin
9191 export const GATE_FILE = "scripts/gate.ts";
9292
9393 /**
94 * The test files a `git diff -U0 scripts/gate.ts` adds to GATE or EXCLUDED
95 * (a move counts as an add). `undefined` when any changed line is anything
96 * but a list entry — that is logic, and the map cannot bound it. Removed
97 * entries select nothing and are not logic.
94 * The test files a `git diff -U0 scripts/gate.ts` brings into the gate: the
95 * entries it takes out of EXCLUDED and does not put back (a move within the
96 * list admits nothing). `undefined` when any changed line is anything but an
97 * exclusion — that is logic, and the map cannot bound it. Added entries
98 * take a file out of the gate, which selects nothing and is not logic.
9899 */
99100 export function gateListAdds(diff: string): string[] | undefined {
100 const entry = /^"(test\/[A-Za-z0-9_.-]+\.test\.[cm]?[jt]sx?)",?(?:\s*\/\/.*)?$/;
101101 const exclusion = /^\{?\s*file:\s*"(test\/[A-Za-z0-9_.-]+\.test\.[cm]?[jt]sx?)",?/;
102102 const data = /^(\{|\},?|reason:\s*.*|issue:\s*\d+,?)$/;
103 const removed = new Set<string>();
103104 const added = new Set<string>();
104105 let inHunk = false;
105106 for (const line of diff.split("\n")) {
@@ -109,18 +110,17 @@ export function gateListAdds(diff: string): string[] | undefined {
109110 const sign = line[0];
110111 if (sign !== "+" && sign !== "-") continue;
111112 const text = line.slice(1).trim();
112 const match = entry.exec(text) ?? exclusion.exec(text);
113 if (match) {
114 if (sign === "+") added.add(match[1]!);
115 } else if (text !== "" && !data.test(text) && !/^\{\s*file:.*\},?$/.test(text)) return undefined;
113 const match = exclusion.exec(text);
114 if (match) (sign === "-" ? removed : added).add(match[1]!);
115 else if (text !== "" && !data.test(text)) return undefined;
116116 }
117 return [...added];
117 return [...removed].filter((file) => !added.has(file));
118118 }
119119
120120 /**
121121 * What the map cannot bound: manifests, scripts, the kernel. `gateAdds` is
122122 * the result of `gateListAdds` for this pull; with it, a change to
123 * scripts/gate.ts alone is bounded by the tests it lists.
123 * scripts/gate.ts alone is bounded by the tests it admits.
124124 */
125125 export function needsFull(
126126 changed: readonly string[],
src/checks/tiers.ts +2 −2
@@ -73,7 +73,7 @@ export interface FastInput {
7373 readonly tests: readonly string[];
7474 /** A file's relative imports; `[]` for one that cannot be read. */
7575 readonly imports: (file: string) => readonly string[];
76 /** The tests a diff of scripts/gate.ts adds (`gateListAdds`); `undefined` when it is logic or unread. */
76 /** The tests a diff of scripts/gate.ts admits (`gateListAdds`); `undefined` when it is logic or unread. */
7777 readonly gateAdds?: readonly string[];
7878 /** Seconds each test took last time (`scripts/test-times.json`). */
7979 readonly times: ReadonlyMap<string, number>;
@@ -118,7 +118,7 @@ export function fastSet(input: FastInput): FastPlan {
118118 for (const test of input.tests) why.set(test, `full: ${unbounded} changed`);
119119 return finish(`full tier, ${unbounded} changed`);
120120 }
121 // A gate.ts that only lists tests is those tests, nothing more.
121 // A gate.ts that only edits EXCLUDED is the tests it admits, nothing more.
122122 const diffed =
123123 input.gateAdds === undefined
124124 ? input.changed
test/check-list.test.ts +80 −41
@@ -1,13 +1,15 @@
11 /**
2 * Every test file is accounted for (`scripts/gate.ts`): in the gate, or
3 * excluded by name with a reason and the issue that owns admitting it. A
4 * new test file that is in neither list turns the check red, so it cannot
5 * sit outside the gate silently.
2 * The gate is every test file minus a short quarantine (`scripts/gate.ts`,
3 * #2037): a new test file is gated by existing, with no list to edit, and
4 * every exclusion names a file that exists, a reason, and the issue that
5 * owns admitting it.
66 */
77
88 import { describe, expect, test } from "bun:test";
9 import { readdirSync } from "node:fs";
10 import { auditGate, EXCLUDED, GATE } from "../scripts/gate.ts";
9 import { copyFileSync, mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";
10 import { tmpdir } from "node:os";
11 import { join } from "node:path";
12 import { auditGate, EXCLUDED, gateFiles, gateRefusal, treeGate } from "../scripts/gate.ts";
1113
1214 const root = new URL("..", import.meta.url).pathname;
1315 const present = readdirSync(`${root}test`)
@@ -16,12 +18,19 @@ const present = readdirSync(`${root}test`)
1618 .sort();
1719
1820 describe("the gate's list", () => {
19 test("every test/*.test.ts is in exactly one list, and every entry exists", () => {
20 expect(auditGate(present, GATE, EXCLUDED)).toEqual([]);
21 test("every exclusion is sound against the files that exist", () => {
22 expect(auditGate(present, EXCLUDED)).toEqual([]);
23 expect(gateRefusal(present)).toEqual([]);
2124 });
2225
23 test("this guard is itself in the gate", () => {
24 expect(GATE).toContain("test/check-list.test.ts");
26 test("the gate is every present test file not excluded", () => {
27 const out = new Set(EXCLUDED.map((e) => e.file));
28 expect(gateFiles()).toEqual(present.filter((f) => !out.has(f)));
29 expect(gateFiles().length + EXCLUDED.length).toBe(present.length);
30 });
31
32 test("this guard is itself in the gate: no exclusion names it", () => {
33 expect(EXCLUDED.map((e) => e.file)).not.toContain("test/check-list.test.ts");
2534 });
2635
2736 test("forge.json's check names no test file and runs the gate's list", async () => {
@@ -34,55 +43,85 @@ describe("the gate's list", () => {
3443 });
3544 });
3645
37 describe("auditGate", () => {
38 const files = ["test/a.test.ts", "test/b.test.ts"];
46 describe("gateFiles", () => {
47 const files = ["test/a.test.ts", "test/b.test.ts", "test/c.test.ts"];
3948
40 test("a file in neither list is named", () => {
41 expect(auditGate(files, ["test/a.test.ts"], [])).toEqual([
42 "test/b.test.ts is in neither GATE nor EXCLUDED",
49 test("a new file is in the gate with no list naming it", () => {
50 expect(gateFiles([...files, "test/x.test.ts"], [])).toEqual([
51 ...files,
52 "test/x.test.ts",
4353 ]);
4454 });
4555
46 test("a file in both lists is named", () => {
56 test("an excluded file is not", () => {
4757 expect(
48 auditGate(files, files, [
49 { file: "test/b.test.ts", reason: "slow", issue: 1 },
50 ]),
51 ).toEqual(["test/b.test.ts is in both GATE and EXCLUDED"]);
58 gateFiles(files, [{ file: "test/b.test.ts", reason: "slow", issue: 1 }]),
59 ).toEqual(["test/a.test.ts", "test/c.test.ts"]);
60 });
61 });
62
63 describe("auditGate", () => {
64 const files = ["test/a.test.ts", "test/b.test.ts"];
65
66 test("a file in no list is no problem", () => {
67 expect(auditGate(files, [])).toEqual([]);
5268 });
5369
5470 test("an exclusion with no reason or no issue is named", () => {
5571 expect(
56 auditGate(
57 files,
58 [],
59 [
60 { file: "test/a.test.ts", reason: " ", issue: 7 },
61 { file: "test/b.test.ts", reason: "slow", issue: 0 },
62 ],
63 ),
72 auditGate(files, [
73 { file: "test/a.test.ts", reason: " ", issue: 7 },
74 { file: "test/b.test.ts", reason: "slow", issue: 0 },
75 ]),
6476 ).toEqual([
6577 "test/a.test.ts is excluded with no reason",
6678 "test/b.test.ts is excluded with no issue number",
6779 ]);
6880 });
6981
70 test("an entry naming a file that does not exist is named", () => {
82 test("an exclusion naming a file that does not exist is named", () => {
7183 expect(
72 auditGate(
73 files,
74 [...files, "test/gone.test.ts"],
75 [{ file: "test/old.test.ts", reason: "slow", issue: 3 }],
76 ),
77 ).toEqual([
78 "test/gone.test.ts is in GATE but does not exist",
79 "test/old.test.ts is in EXCLUDED but does not exist",
80 ]);
84 auditGate(files, [{ file: "test/old.test.ts", reason: "slow", issue: 3 }]),
85 ).toEqual(["test/old.test.ts is in EXCLUDED but does not exist"]);
8186 });
8287
83 test("a file named twice in one list is named", () => {
84 expect(auditGate(files, [...files, "test/a.test.ts"], [])).toEqual([
85 "test/a.test.ts is in GATE twice",
88 test("a file excluded twice is named", () => {
89 const entry = { file: "test/a.test.ts", reason: "slow", issue: 3 };
90 expect(auditGate(files, [entry, entry])).toEqual([
91 "test/a.test.ts is in EXCLUDED twice",
8692 ]);
8793 });
8894 });
95
96 describe("treeGate: the gate an older commit's tree names", () => {
97 const tree = (gate: string, tests: string[]): string => {
98 const dir = mkdtempSync(join(tmpdir(), "forge-tree-gate-"));
99 mkdirSync(join(dir, "scripts"));
100 mkdirSync(join(dir, "test"));
101 writeFileSync(join(dir, "scripts/gate.ts"), gate);
102 for (const file of tests) writeFileSync(join(dir, file), "");
103 return dir;
104 };
105
106 test("a tree from before #2071 is read from its GATE list", async () => {
107 const dir = tree(
108 'export const GATE = ["test/a.test.ts"];\nexport const EXCLUDED = [];\n',
109 ["test/a.test.ts", "test/b.test.ts"],
110 );
111 try {
112 expect(await treeGate(dir, "old")).toEqual(["test/a.test.ts"]);
113 } finally {
114 rmSync(dir, { recursive: true, force: true });
115 }
116 });
117
118 test("a later tree derives it from its own test files minus EXCLUDED", async () => {
119 const dir = tree("", ["test/a.test.ts", "test/zz-new.test.ts", EXCLUDED[0]!.file]);
120 copyFileSync(`${root}scripts/gate.ts`, join(dir, "scripts/gate.ts"));
121 try {
122 expect(await treeGate(dir, "new")).toEqual(["test/a.test.ts", "test/zz-new.test.ts"]);
123 } finally {
124 rmSync(dir, { recursive: true, force: true });
125 }
126 });
127 });
test/check-tiers.test.ts +22 −13
@@ -198,16 +198,22 @@ describe("the fast set, by measured coverage", () => {
198198 expect(fastSet({ ...base, changed: ["scripts/test-times.json"] }).files).toEqual([]);
199199 });
200200
201 test("a gate.ts diff that only lists tests selects those tests; logic runs full", () => {
201 test("a gate.ts diff that only edits EXCLUDED selects the tests it admits; logic runs full", () => {
202202 const listed = [
203 "@@ -10,0 +11,2 @@",
204 '+ "test/new.test.ts",',
205 '+ { file: "test/other.test.ts", reason: "slow", issue: 5 },',
206 "@@ -20 +22,0 @@",
207 '- "test/old.test.ts",',
203 "@@ -10,2 +10,0 @@",
204 '- { file: "test/new.test.ts", reason: "slow", issue: 5 },',
205 "- {",
206 '- file: "test/other.test.ts",',
207 '- reason: "flaky",',
208 "- issue: 6,",
209 "- },",
210 "@@ -20,0 +18,2 @@",
211 '+ { file: "test/old.test.ts", reason: "flaky", issue: 7 },',
212 '+ { file: "test/other.test.ts", reason: "moved", issue: 6 },',
208213 ].join("\n");
209214 const adds = gateListAdds(listed);
210 expect(adds).toEqual(["test/new.test.ts", "test/other.test.ts"]);
215 // Quarantining old selects nothing; other only moved, so it stays out.
216 expect(adds).toEqual(["test/new.test.ts"]);
211217 const plan = fastSet({ ...base, changed: ["scripts/gate.ts"], gateAdds: adds! });
212218 expect(plan.line).not.toContain("full tier");
213219 expect(plan.files).not.toContain("test/web-phone.test.ts");
@@ -219,12 +225,15 @@ describe("the fast set, by measured coverage", () => {
219225 gateAdds: adds!,
220226 });
221227 expect(more.why.get("test/new.test.ts")).toBe("changed");
222 expect(gateListAdds('@@ -1 +1 @@\n+ "test/a.test.ts",\n+export const X = 1;')).toBeUndefined();
223 expect(gateListAdds('@@ -1 +1 @@\n- if (x) return;')).toBeUndefined();
224 const hunk = "@@ -1,0 +2,2 @@\n";
225 expect(gateListAdds(`${hunk}+ "test/a.test.ts",\n+++i;`)).toBeUndefined();
226 expect(gateListAdds(`${hunk}+ "test/a.test.ts",\n---n;`)).toBeUndefined();
227 expect(gateListAdds(`--- a/scripts/gate.ts\n+++ b/scripts/gate.ts\n${hunk}+ "test/a.test.ts",`)).toEqual(["test/a.test.ts"]);
228 const hunk = "@@ -1,2 +1,0 @@\n";
229 const entry = '- { file: "test/a.test.ts", reason: "slow", issue: 1 },';
230 expect(gateListAdds(`${hunk}${entry}\n+export const X = 1;`)).toBeUndefined();
231 expect(gateListAdds("@@ -1 +1 @@\n- if (x) return;")).toBeUndefined();
232 // A bare name is no list entry now: there is no GATE to put it in.
233 expect(gateListAdds('@@ -1 +1 @@\n+ "test/a.test.ts",')).toBeUndefined();
234 expect(gateListAdds(`${hunk}${entry}\n+++i;`)).toBeUndefined();
235 expect(gateListAdds(`${hunk}${entry}\n---n;`)).toBeUndefined();
236 expect(gateListAdds(`--- a/scripts/gate.ts\n+++ b/scripts/gate.ts\n${hunk}${entry}`)).toEqual(["test/a.test.ts"]);
228237 const logic = fastSet({ ...base, changed: ["scripts/gate.ts"] });
229238 expect(logic.files).toEqual(base.tests);
230239 });
test/flake.test.ts +40 −11
@@ -9,7 +9,7 @@ import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises";
99 import { readdirSync, readFileSync } from "node:fs";
1010 import { tmpdir } from "node:os";
1111 import { join } from "node:path";
12 import { auditGate, GATE, type Exclusion } from "../scripts/gate.ts";
12 import { auditGate, gateFiles, type Exclusion } from "../scripts/gate.ts";
1313 import {
1414 admissionRecord,
1515 admitted,
@@ -21,6 +21,7 @@ import {
2121 HUNT_RUNS,
2222 issueFor,
2323 quarantine,
24 quarantineFlaky,
2425 readRun,
2526 retries,
2627 ROOT_CAUSES,
@@ -164,30 +165,58 @@ describe("the issue", () => {
164165
165166 describe("quarantine", () => {
166167 const source = read("scripts/gate.ts");
167 const file = GATE[0]!;
168 const file = gateFiles()[0]!;
168169
169 test("moves the file from GATE to EXCLUDED under the issue", async () => {
170 test("takes the file out of the gate into EXCLUDED under the issue", async () => {
170171 const moved = quarantine(source, file, 4242);
171172 expect(moved).not.toBe(source);
172173 const dir = await scratch();
173174 await writeFile(join(dir, "gate.ts"), moved);
174175 const mod = (await import(join(dir, "gate.ts"))) as {
175 GATE: readonly string[];
176176 EXCLUDED: readonly Exclusion[];
177 gateFiles: (present?: readonly string[]) => string[];
177178 };
178179 const present = readdirSync(`${root}test`)
179180 .filter((name) => name.endsWith(".test.ts"))
180 .map((name) => `test/${name}`);
181 expect(auditGate(present, mod.GATE, mod.EXCLUDED)).toEqual([]);
182 expect(mod.GATE).not.toContain(file);
183 expect(mod.GATE).toHaveLength(GATE.length - 1);
181 .map((name) => `test/${name}`)
182 .sort();
183 expect(auditGate(present, mod.EXCLUDED)).toEqual([]);
184 const gate = mod.gateFiles(present);
185 expect(gate).not.toContain(file);
186 expect(gate).toHaveLength(gateFiles().length - 1);
184187 expect(mod.EXCLUDED.at(-1)).toMatchObject({ file, issue: 4242 });
185188 });
186189
187 test("a file not in GATE is left alone", async () => {
190 test("a file excluded already is left alone", async () => {
188191 const { EXCLUDED } = await import("../scripts/gate.ts");
189192 expect(quarantine(source, EXCLUDED[0]!.file, 1)).toBe(source);
190 expect(quarantine(source, "test/nope.test.ts", 1)).toBe(source);
193 expect(quarantine("no list here", file, 1)).toBe("no list here");
194 });
195 });
196
197 describe("quarantineFlaky", () => {
198 const source = read("scripts/gate.ts");
199 const file = gateFiles()[0]!;
200 const present = new Set(gateFiles());
201
202 test("an issue naming a missing file adds no EXCLUDED entry", () => {
203 const got = quarantineFlaky(source, [["test/gone.test.ts", 77]], present);
204 expect(got.source).toBe(source);
205 expect(got.moved).toEqual([]);
206 });
207
208 test("a present file is quarantined and named; one excluded already is not", () => {
209 const got = quarantineFlaky(
210 source,
211 [
212 ["test/gone.test.ts", 77],
213 [file, 78],
214 [file, 79],
215 ],
216 present,
217 );
218 expect(got.source).toBe(quarantine(source, file, 78));
219 expect(got.moved).toEqual([`${file} (#78)`]);
191220 });
192221 });
193222
@@ -268,7 +297,7 @@ describe("no retry in the gate", () => {
268297 (name) => `.forgejo/workflows/${name}`,
269298 ),
270299 // This file holds the detector's own examples.
271 ...GATE.filter((file) => file !== "test/flake.test.ts"),
300 ...gateFiles().filter((file) => file !== "test/flake.test.ts"),
272301 ];
273302 expect(named.flatMap((path) => retries(path, read(path)))).toEqual([]);
274303 });
test/hollow.test.ts +3 −3
@@ -6,7 +6,7 @@
66 */
77
88 import { describe, expect, test } from "bun:test";
9 import { GATE } from "../scripts/gate.ts";
9 import { EXCLUDED } from "../scripts/gate.ts";
1010 import { audit, hollow, readSources } from "../scripts/hollow.ts";
1111
1212 const root = new URL("..", import.meta.url).pathname;
@@ -54,8 +54,8 @@ describe("the repository", () => {
5454 WHOLE_TREE,
5555 );
5656
57 test("this guard is itself in the gate", () => {
58 expect(GATE).toContain("test/hollow.test.ts");
57 test("this guard is itself in the gate: no exclusion names it", () => {
58 expect(EXCLUDED.map((e) => e.file)).not.toContain("test/hollow.test.ts");
5959 });
6060
6161 test(
test/shards.test.ts +48 −24
@@ -27,7 +27,7 @@ import {
2727 type ShardResult,
2828 } from "../scripts/shards.ts";
2929 import { parseCheck } from "../src/apps/manifest.ts";
30 import { EXCLUDED, GATE } from "../scripts/gate.ts";
30 import { EXCLUDED, gateFiles, presentTests } from "../scripts/gate.ts";
3131 import { CHECK_CACHE } from "../src/checks/cache.ts";
3232
3333 const files = Array.from({ length: 23 }, (_, i) => `test/f${i}.test.ts`);
@@ -353,7 +353,7 @@ describe("splitArgs", () => {
353353 expect(split.ok).toBe(true);
354354 if (!split.ok) return;
355355 const files = [
356 ...(split.flags.includes("--gate") ? GATE : []),
356 ...(split.flags.includes("--gate") ? gateFiles() : []),
357357 ...split.files,
358358 ];
359359 expect(files.length).toBeGreaterThan(0);
@@ -464,11 +464,27 @@ test("says two things, a pause apart", async () => {
464464
465465 describe("the gate audit before any shard", () => {
466466 const script = new URL("../scripts/test-shards.ts", import.meta.url).pathname;
467 const listed = [...GATE, ...EXCLUDED.map((e) => e.file)];
468 const run = async (extra: string[], args: string[]) => {
467 const listed = presentTests();
468 const git = (dir: string, ...args: string[]) =>
469 Bun.spawnSync(["git", "-c", "user.name=t", "-c", "user.email=t@t", "-c", "commit.gpgsign=false", ...args], {
470 cwd: dir,
471 stdout: "ignore",
472 stderr: "ignore",
473 }).exitCode;
474 const run = async (extra: string[], args: string[], drop: string[] = [], commit = false) => {
469475 const dir = mkdtempSync(join(tmpdir(), "forge-audit-"));
470476 mkdirSync(join(dir, "test"));
471 for (const file of [...listed, ...extra]) writeFileSync(join(dir, file), "");
477 for (const file of listed) if (!drop.includes(file)) writeFileSync(join(dir, file), "");
478 if (commit) {
479 expect(git(dir, "init", "-q")).toBe(0);
480 expect(git(dir, "add", "-A")).toBe(0);
481 expect(git(dir, "commit", "-qm", "base")).toBe(0);
482 }
483 for (const file of extra) writeFileSync(join(dir, file), "");
484 if (commit) {
485 expect(git(dir, "add", "-A")).toBe(0);
486 expect(git(dir, "commit", "-qm", "add a test")).toBe(0);
487 }
472488 const started = performance.now();
473489 const child = Bun.spawn(["bun", script, ...args], {
474490 cwd: dir,
@@ -485,36 +501,43 @@ describe("the gate audit before any shard", () => {
485501 return { code, out, err, ms: performance.now() - started };
486502 };
487503
488 test("an unlisted test file is refused by name, last line, nothing spawned", async () => {
504 test("a stale exclusion is refused by name, last line, nothing spawned", async () => {
505 const gone = EXCLUDED[0]!.file;
489506 for (const args of [["--gate"], ["--tier", "fast", "--diff", "a..b"]]) {
490 const got = await run(["test/zz-new.test.ts"], args);
507 const got = await run([], args, [gone]);
491508 expect(got.code).toBe(2);
492509 const last = got.err.trim().split("\n").pop()!;
493 expect(last).toContain("test/zz-new.test.ts");
494 expect(last).toContain("list it in GATE or EXCLUDED in scripts/gate.ts");
510 expect(last).toBe(`test-shards: ${gone}: fix its EXCLUDED entry in scripts/gate.ts`);
495511 expect(got.out + got.err).not.toContain("start ");
496512 expect(got.out).not.toContain("── shard");
497513 expect(got.ms).toBeLessThan(1000);
498514 }
499515 });
500516
501 test("a listed set proceeds, and explicit files are not audited", async () => {
502 const last = String(GATE.length);
503 const got = await run([], ["--gate", "--shard", `${last}/${last}`]);
504 expect(got.err).not.toContain("list it in GATE");
505 expect(got.out).toContain(`matrix slice ${last}/${last}`);
506 const explicit = await run(["test/zz-new.test.ts"], ["test/zz-new.test.ts"]);
507 expect(explicit.err).not.toContain("list it in GATE");
517 test("a new file is in the gate with no list naming it, and explicit files are not audited", async () => {
518 const last = String(gateFiles().length + 1);
519 const got = await run(["test/zz-new.test.ts"], ["--gate", "--shard", `${last}/${last}`]);
520 expect(got.err).not.toContain("EXCLUDED");
521 expect(got.out).toContain(`matrix slice ${last}/${last} (1 of ${last} files)`);
522 const explicit = await run(["test/zz-new.test.ts"], ["test/zz-new.test.ts"], [listed[0]!]);
523 expect(explicit.err).not.toContain("EXCLUDED");
524 }, 30_000);
525
526 test("a pull that only adds test/x.test.ts is gated on it (#2037)", async () => {
527 const got = await run(["test/zz-new.test.ts"], ["--tier", "fast", "--diff", "HEAD~1..HEAD"], [], true);
528 expect(got.err).not.toContain("EXCLUDED");
529 expect(got.out).toContain("for 1 changed");
530 expect(got.out).toContain(" test/zz-new.test.ts: changed\n");
508531 }, 30_000);
509532 });
510533
511534 describe("the gate audit alone, first in the check", () => {
512535 const script = new URL("../scripts/gate.ts", import.meta.url).pathname;
513 const listed = [...GATE, ...EXCLUDED.map((e) => e.file)];
514 const run = async (extra: string[]) => {
536 const listed = presentTests();
537 const run = async (extra: string[], drop: string[] = []) => {
515538 const dir = mkdtempSync(join(tmpdir(), "forge-gate-"));
516539 mkdirSync(join(dir, "test"));
517 for (const file of [...listed, ...extra]) writeFileSync(join(dir, file), "");
540 for (const file of [...listed, ...extra]) if (!drop.includes(file)) writeFileSync(join(dir, file), "");
518541 const child = Bun.spawn(["bun", script], { cwd: dir, stdin: "ignore", stdout: "pipe", stderr: "pipe" });
519542 const err = await new Response(child.stderr).text();
520543 const code = await child.exited;
@@ -522,13 +545,14 @@ describe("the gate audit alone, first in the check", () => {
522545 return { code, err };
523546 };
524547
525 test("an unlisted file fails it by name; a listed set passes", async () => {
526 const red = await run(["test/zz-new.test.ts"]);
548 test("a stale exclusion fails it by name; a new unlisted file passes", async () => {
549 const gone = EXCLUDED[0]!.file;
550 const red = await run([], [gone]);
527551 expect(red.code).toBe(2);
528 expect(red.err.trim().split("\n").pop()).toContain(
529 "test/zz-new.test.ts: list it in GATE or EXCLUDED in scripts/gate.ts",
552 expect(red.err.trim().split("\n").pop()).toBe(
553 `gate: ${gone}: fix its EXCLUDED entry in scripts/gate.ts`,
530554 );
531 const green = await run([]);
555 const green = await run(["test/zz-new.test.ts"]);
532556 expect(green.code).toBe(0);
533557 expect(green.err).toBe("");
534558 }, 30_000);

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs