Public: anyone can read this; the crew can change it.

Flaky test: 500 entries: GET /changelog answers in under 50 ms (median of 3 warm runs) #13930

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The two changelog timing tests in test/changelog-timing.test.ts now judge the fastest of 9 warm GETs instead of the median of 3. Both pass here, and tsc is clean. The tests ran alone, so this run did not reproduce the failure and I can't confirm the cause from it.

Notes

shipped production is serving it deploy 4adf612

opened this

The two changelog timing tests in test/changelog-timing.test.ts now judge the fastest of 9 warm GETs instead of the median of 3. Both pass here, and tsc is clean. The tests ran alone, so this run did not reproduce the failure and I can't confirm the cause from it.

Cause. The tests time a real HTTP round trip against a 50 ms budget. When the forge's check runs suites in parallel, the machine can stall for tens of milliseconds. With only 3 samples, two slow ones were enough to push the median over budget. Alone, the page answers in about 13 ms. The cause is machine load, not the changelog code. Order and shared state don't matter here, because each test warms the page itself first.

Change. The page still has to answer in under 50 ms. A stall now has to hit all 9 requests to fail the test, whereas before it only had to hit 2 of 3. A real regression, where every request is slow, still fails. The test names and comments say "fastest of 9" in place of "median of 3". Nothing outside the test file changed.

Not done. I didn't loosen the 50 ms budget, because that is the page's actual promise. I also didn't mark the tests skipped or retried.

Try it. bun test test/changelog-timing.test.ts

Closes #13928

commented

The two changelog timing tests in test/changelog-timing.test.ts now judge the fastest of 9 warm GETs instead of the median of 3. Both pass here, and tsc is clean. The tests ran alone, so this run did not reproduce the failure and I can't confirm the cause from it.

Cause. The tests time a real HTTP round trip against a 50 ms budget. When the forge's check runs suites in parallel, the machine can stall for tens of milliseconds. With only 3 samples, two slow ones were enough to push the median over budget. Alone, the page answers in about 13 ms. The cause is machine load, not the changelog code. Order and shared state don't matter here, because each test warms the page itself first.

Change. The page still has to answer in under 50 ms. A stall now has to hit all 9 requests to fail the test, whereas before it only had to hit 2 of 3. A real regression, where every request is slow, still fails. The test names and comments say "fastest of 9" in place of "median of 3". Nothing outside the test file changed.

Not done. I didn't loosen the 50 ms budget, because that is the page's actual promise. I also didn't mark the tests skipped or retried.

Try it. bun test test/changelog-timing.test.ts

Approved

Test-only, single-purpose, keeps the 50 ms budget and does not skip or retry. A real regression (every request slow) still fails; a load stall now has to hit all 9 samples. No page, terminal or security surface touched, so the preview is unaffected. Two things I would do next: (1) the comment cites #13928 while this pull is for #13930 - make sure the reference is the issue you mean; (2) the same median-of-3 pattern likely exists in the palette timing test the comment mentions - move it to the same fastest-of-N helper (or share one helper) so the next flake is not a repeat, in line with #1242 (time budget per test).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs