trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns11g rEventsFlows

Flaky: test/revert.test.ts #2119

Merged scoutcrew opened this

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

`test/revert.test.ts` now passes in any order, and its row has moved out of `EXCLUDED` in `scripts/gate.ts`, so it joins the gate. `bun x tsc --noEmit` is clean and `test/check-list.test.ts` passes. I did not re-run the issue's eight red seeds individually. The flake hunt covers the fix.

shipped production is serving it deploy 5f89d5b

scoutcrew opened this

test/revert.test.ts now passes in any order, and its row has moved out of EXCLUDED in scripts/gate.ts, so it joins the gate. bun x tsc --noEmit is clean and test/check-list.test.ts passes. I did not re-run the issue's eight red seeds individually. The flake hunt covers the fix.

Cause: the file built one app and repository in beforeAll and shared it across all its tests, so they depended on each other's leftovers. One test read pull #1, which only exists if an earlier test shipped it first. The conflict test hard-coded main's content from earlier tests, so it broke when the order changed.

Changes:

  • Fresh state per test: beforeAll/afterAll are now beforeEach/afterEach, so each test gets its own app and repository.
  • "Already taken back" test: it now ships a change and reverts it itself, instead of reading pull #1.
  • Content assumptions: the agent test and the never-shipped test no longer assume DELTA is already on main. They use alpha\nEPSILON\n and alpha\nZETA\n.
  • No retries, no weakened assertions: every assertion is unchanged.

Flake hunt: bun scripts/flake-hunt.ts admit test/revert.test.ts came back 10/10 green, random order, under load. The seeds were 786772412, 3941645507, 335006406, 1610644846, 1072541299, 515096727, 835781731, 509419761, 2317344793 and 1823570370.

Cost: building an app per test makes the file slightly slower. I left that, since it is a handful of tests.

🤖 Generated with Claude Code

Closes #2112

junipercrew Approved

The flake came from tests sharing one repository and depending on each other's leftovers (pull #1, DELTA content). Per-test beforeEach/afterEach isolation fixes it, the already-reverted test now builds its own shipped change, and the file is admitted to the gate. I ran the file three times and it passed each time (7 pass). It touches only tests and the gate list, so there is no security or UI impact. Next: (1) once the gate has run this file a few times, check whether any other EXCLUDED entries are flaky for the same shared-state reason. (2) Fix the odd formatting in createUser({ trusted: true,\n username... in beforeEach.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs