No flaky tests: ten green runs to join the gate, a nightly flake hunt in random order, mechanical quarantine #1243

The owner: "we need to be sure the tests aren't flaky."

Today's lane shows the cost: apps-ws (5 of 8 fail in the full run, pass alone), apps-idle (red once in seven), studio's preview case, signin-throttle (58 s alone, 60 s timeout), capacity, app-domains and sealed-release each needed a pull to make them green "on a loaded runner". A flaky test is a red that teaches nothing and a green that proves nothing; today each one cost a 20-minute lane slot and a person's attention.

Done looks like:

• [ ] Admission: a test file joins the gate only after ten green runs under the gate's own load, in random order, recorded in the admitting pull (today it is three).
• [ ] A nightly flake hunt (.forgejo/workflows/flake-hunt.yml or the forge's own routine): the whole suite five times with --randomize and a fresh seed each time, on a loaded box; a test that fails in any run and passes in another is flaky.
• [ ] Quarantine is mechanical: a flaky test is moved to EXCLUDED with an issue filed by the hunt that carries the failure text, the seed and the run that failed; nobody hand-edits the list.
• [ ] Root causes named in the issue template: a shared port, a real clock, order dependence, a leaked server, a timeout set near the real duration; the fix is the shape, not a retry.
• [ ] No retry on tests anywhere in the gate; a retried green is not a green.

Evidence: fork pulls #1165, #1167, #1168, #1174, #1186, #1191 from 2026-10-13.

Bun-native flags that bear on this (from the research brief on #1242): --parallel[=N] with BUN_TEST_WORKER_ID, --shard=i/n --timings=<file> --update-timings balanced by recorded duration, --randomize --seed --rerun-each N for the flake hunt, --coverage-reporter=lcov one file at a time for the impact map. Prefer them to our own sharding where they fit.

Parked by the crew: the check failed and the decider saw no point in another try (pull #1268 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

One for the hunt, caught on the upstream gate tonight (trevato-apps/forge PR #36, shard 47, 23:58Z): test/spawn-first-boot.test.ts "the child answers /healthz and / quickly once booted" asserts the median of five loopback fetches is under 50 ms; on the shared runner it measured 65.4 ms and went red on a change that never touched it. I raised the bound to 500 ms in that PR (a boot per request is seconds, which is what the test guards against). The pattern to hunt: any wall-clock bound under ~200 ms on a request or a spawn is a flake on a loaded runner; the assertion should be relative (vs. a baseline measured in the same run) or generous enough to only catch the failure mode it names.

Parked by the crew: the check stayed red through 2 automatic tries (pull #1323 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Flake on the upstream gate, 2026-10-15 08:21Z (release 0.7.84 held): test/claim-link.test.ts — "a replay of the re-issue under one Idempotency-Key never returns the link" got 409 instead of 201 (line 226), and "MCP claim_link keeps the link out of the conversation unless reveal is sent (#839)" got true instead of false (line 266). The same trunk passes the file 10/10 locally. Two cases in one describe failing together on a loaded runner points at shared state between the cases (one forge, one idempotency store) rather than timing; each case should make its own child and key.

Parked by the crew: the reviewer asked for changes and the builder's run ended without pushing any (pull #1504 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Load-sensitivity, 2026-10-15 22:20–22:40Z: with 9 checks × 6 shards running, 21 of 37 finished checks failed, the failures clustering in edits.test.ts (mirror pushes to origin), overlap.test.ts (two processes on one data dir), snapshot-drill.test.ts (restore that hangs), the fork flow — git/process-heavy tests. On main, alone, the same files pass (10/10, 4/4, 5/5). Shards lowered to 4. These tests need to hold under the gate's real concurrency: timeouts tuned from measured p99 under load, no fixed sleeps, no shared ports or temp paths between shards.