No flaky tests: ten green runs to join the gate, a nightly flake hunt in random order, mechanical quarantine #1524

Flaky tests now have a ten-run bar to join the gate, a nightly hunt in random order, and a quarantine that works from issues so nobody edits the list by hand. This redoes #1504 and adds fixes for juniper's three review points on it. tsc --noEmit, test/flake.test.ts and test/check-list.test.ts pass locally. I didn't run the whole check or a live hunt here.

Review fixes in this push (fb4cb8a0):

• Who can quarantine a file: before, any open issue titled Flaky: test/x.test.ts moved that file out of GATE. Now huntedFile (scripts/flake.ts) only counts an issue when its author is the token's own user (read from GET /api/v1/user) and the forge hasn't marked it untrusted. A new test covers someone else's issue, a missing author, and an untrusted issue.
• Quarantine job clone: it now clones through the internal REGISTRY with the job token, as the hunt jobs do, so a private repository works. It pushes its flake-hunt/<date> branch to the forge explicitly, because the forge is where it opens the pull.
• Admit test timeout: the admit test starts two bun test runs beside busy loops, so it now has a 120 s timeout instead of the 5 s default.

What the pull does:

• Admission: bun scripts/flake-hunt.ts admit test/x.test.ts runs the file ten times while busy loops take half the cores. Each run is its own bun test --randomize with a fresh seed. It prints the seeds as the record for the admitting pull and exits non-zero unless all ten are green. test/flake.test.ts joins GATE with its own record.
• Hunt: flake-hunt.ts hunt [--shard N/M] [--file-issues] runs every gate file five times under the same load. Each run's seed shuffles the order of the files and of the tests inside each file.

- A file that fails in some runs and passes in others is flaky. A file that fails in every run is reported as broken, not flaky. - With --file-issues it files Flaky: test/x.test.ts with the failure text, the seed with a command to replay it, and a link to the run. It skips files that already have an open issue.

• Quarantine: flake-hunt.ts quarantine moves each file named by an open Flaky: issue from GATE to EXCLUDED under that issue's number.
• Nightly: .forgejo/workflows/flake-hunt.yml runs at 02:17 UTC, split into 20 jobs because the runner stops a job at about 30 minutes. A final job runs quarantine and opens the pull.
• Root causes: the issue body lists shared port, real clock, order dependence, leaked server, a timeout near the real duration and a tight wall-clock bound, each as a tickbox with its fix. It says a retry is not a fix.
• No retry: test/flake.test.ts fails if --retry, --rerun-each, a bunfig retry = or { retry: N } appears anywhere the gate runs or is configured.
• No short time limits: the same test fails on any time limit under 200 ms in a test file. Because of that, test/spawn-first-boot.test.ts now allows 500 ms for its loopback median instead of 50 ms.

Not done here: the cases in test/claim-link.test.ts reported on the issue are left for the hunt to catch and file.

Worth a look: huntedFile and quarantine in scripts/flake.ts, and the workflow's quarantine job, which is the only thing that writes scripts/gate.ts. The workflow needs two settings on the mirror: secret FORGE_TOKEN and variable FORGE_URL.

To try it: bun scripts/flake-hunt.ts admit test/flake.test.ts, then bun scripts/flake-hunt.ts hunt --shard 1/20 --runs 2.

🤖 Generated with Claude Code

Closes #1243

mason only (reviewers: not for you): this pull is approved but no longer merges into main. Bring main in on this branch, resolve the conflict, and push; the approval stands if the change does not.

Flake tests can now only join the gate after ten green runs in random order. A nightly hunt runs the suite shuffled and under load. A quarantine job removes a flaky file from the gate, but only when the issue was filed by the hunt itself. Nothing in the gate may retry a test or bound one with a tight timeout.

This push only merges main into the branch, so the approved change is the same. The one conflict was in scripts/gate.ts: this branch adds test/flake.test.ts to GATE and main adds test/crew-locks.test.ts at the same spot, so both are now listed. After the merge, the type check passes, and test/flake.test.ts, test/check-list.test.ts (which checks the gate list) and test/crew-locks.test.ts pass locally.

What is still open from the review, left out so the approval holds:

• The admit test in test/flake.test.ts spawns two real bun test runs beside busy loops and has a 120 s timeout. That is the kind of test this pull is meant to catch. The next step is to make admit accept an injected runner, or to run bun scripts/flake-hunt.ts admit test/flake.test.ts on it, in a follow-up.
• The 20-slice nightly matrix has not been checked against the isolated runner's 30-minute limit. The first nightly will show whether it fits; if a slice times out, raise the slice count or lower HUNT_RUNS. The quarantine job runs with if: always(), so it will also act on a partial hunt.

The hunt has still not run end to end. To try it: bun scripts/flake-hunt.ts hunt --runs 3 --shard 1/20 runs one slice locally, and bun scripts/flake-hunt.ts admit test/x.test.ts prints the admission record for a file.

🤖 Generated with Claude Code