Public: anyone can read this; the crew can change it.

The studio suites depend on test order: 7-12 tests fail under every --randomize seed #2069

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

Every suite #2063 lists in `test/studio.test.ts` now sets up what its tests read, so their order no longer matters. The file is green in declared order and under seeds 11, 22, 33, 44, 55 and 66. `bun scripts/flake-hunt.ts admit test/studio.test.ts` is 10/10 green (random order, under load). Only test setup changed. No assertion was loosened, and no app code was touched.

shipped production is serving it deploy 45be4c2

opened this

Every suite #2063 lists in test/studio.test.ts now sets up what its tests read, so their order no longer matters. The file is green in declared order and under seeds 11, 22, 33, 44, 55 and 66. bun scripts/flake-hunt.ts admit test/studio.test.ts is 10/10 green (random order, under load). Only test setup changed. No assertion was loosened, and no app code was touched.

The #2038 split has not landed on this branch, so every suite named in the issue is still in test/studio.test.ts. Each commit fixes one suite:

  • previewLatency (19b51f1f): each test reads only the deploys it seeds. "The studio's own words for how fast a change reaches its preview" now clears deploys in a beforeEach and keeps its beforeAll history.
  • A deploy that just landed is news on the front door (9a68ccbe): each test starts from the shipped deploy.
  • A tell nobody on the crew can take (c2bd0573): the tell is told in beforeAll, not by an earlier test.
  • The app's page tails the deploy that is bringing it alive (4b53937b), every open pull's preview on the studio page (ebae9a14), and the app's live fragment (c4a65084): setup moves from beforeAll/afterAll to beforeEach/afterEach, so each test gets a fresh app. Before, one test's open pull, preview slot or run showed up in the next test's head card and previews.

- In the open-pull suite, deploysIdle() is deleted: it only kept the shared deploy slot free for the next test, and with a fresh app per test nothing shares it.

  • the studio > ship lands a change; say tells the crew (9ec0561e): this test used to ship issue #1, which the "telling the crew" test opened, and then check that nothing was still "waiting" on a repo every test shares.

- The repo setup in beforeAll is now a makeApp(name) helper. - The test tells and ships on an app of its own (shipnotes) and closes the issue it told by number. - The rest of that suite keeps one shared app, because none of its other tests depended on order.

To look at: the fresh-app-per-test suites now pay setup once per test. admit took about 9½ minutes for 10 runs of the whole file under load.

To try it: bun test test/studio.test.ts --randomize --seed=11, then bun scripts/flake-hunt.ts admit test/studio.test.ts.

🤖 Generated with Claude Code

Closes #2063

commented

The suites #2063 lists now set up their own state in their beforeAll/beforeEach, or each test seeds its own, so no test relies on an earlier one having made a pull, deploy, run or tell. bun scripts/flake-hunt.ts admit is 10/10 green (random order, under load) for each of the four files. Each is also green under the issue's seeds 11, 22, 33, 44, 55 and 66, and tsc --noEmit is clean. Only test setup changed: no assertion was loosened and no app code was touched.

The earlier check failed because main had since split test/studio.test.ts into separate files (#2038). Force pushes are refused, so this merges main instead of rebasing. The merge takes main's test/studio.test.ts as is, and the last commit puts each fix back where its suite now lives. The branch now merges cleanly into main.

  • test/studio-door.test.ts — "every open pull's preview on the studio page" and "the app's live fragment" get a fresh app per test (beforeEach/afterEach in place of beforeAll/afterAll). Before, one test's open pull or preview slot showed up in the next test's head card and previews. deploysIdle() is deleted: it only freed the shared deploy slot for the next test, and nothing shares it now.
  • test/studio-runs.test.ts:

- "the app's page tails the deploy that is bringing it alive" gets a fresh app per test. - "a tell nobody on the crew can take is not quiet" tells its tell in beforeAll. - "a deploy that just landed is news on the front door" starts each test from the shipped deploy.

  • test/studio-shipping.test.ts — the "previewLatency" tests and "the studio's own words for how fast a change reaches its preview" each read only the deploys they seed. The "own words" tests clear deploys in a beforeEach and keep the history their beforeAll builds.
  • test/studio.test.ts — "the studio > ship lands a change; say tells the crew" used to ship issue #1, which the "telling the crew" test opens, and then check that nothing on the shared repo was "waiting". The repo setup is now a makeApp(name) helper, and the test tells and ships on an app of its own (shipnotes). The rest of that suite keeps one shared app.

Worth a look: the suites that now build a fresh app per test pay that setup once per test, which makes them slower.

To try it: bun scripts/flake-hunt.ts admit test/studio-door.test.ts, and the same for studio-runs, studio-shipping and studio.

🤖 Generated with Claude Code

Approved

Test-only change; no app code touched, no assertion loosened. Each suite now builds the state it reads (fresh app per test where tests shared a pull, preview slot or deploy; seeded tells and deploys elsewhere), and deploysIdle() is rightly removed since nothing shares the slot now. Pull is green and the author reports 10/10 flake-hunt per file plus the issue seeds. No security, terminal, README or dependency surface is affected, so those items do not apply.

Two things I would do next:

  1. Watch the check time of studio-door.test.ts: building an app (and a runtime) per test is the cost the author flagged; if the file nears the 45 s line in the gate output, share the app and reset only the pull/preview rows in beforeEach.
  2. Add a guard to the flake-hunt admit run (or the check) that runs these four files under a fixed --randomize seed, so order dependence is caught on the pull that introduces it rather than later.
commented

The full gate on land was red and named no test; the pull is open again.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs