Public: anyone can read this; the crew can change it.

Flaky test: 500 entries: GET /changelog answers in under 50 ms (median of 3 warm runs) #15250

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

The changelog timing test now measures CPU time per GET instead of wall-clock time, so machine load can no longer fail it. I ran it three times locally and it passed each time; tsc printed no errors. Wall-clock time was the flake.

Notes

shipped production is serving it deploy cfe12e4

opened this

The changelog timing test now measures CPU time per GET instead of wall-clock time, so machine load can no longer fail it. I ran it three times locally and it passed each time; tsc printed no errors.

Wall-clock time was the flake. Even taking the fastest of 9 runs, a stall from other suites sharing the machine could push every sample over the 50 ms budget. The server and client run in one process, so process.cpuUsage around each request captures the page's real work. A stall adds wall time but no CPU.

The budget is still 50 ms, for both the 500-entry test and the two-chapter test, and the page is not changed. Locally the fastest run is about 6–7 CPU-ms for the plain page and 7–12 CPU-ms with chapters. Both tests still check for a 200 status and the expected content. The change is only in test/changelog-timing.test.ts.

Not done: the test no longer catches a regression that is slow only because it waits (for example on I/O), since waiting costs no CPU. I judged that acceptable for a flake fix. I did not run the full suite; the forge runs it on push.

Closes #15248

commented

The changelog timing test now measures CPU time per GET instead of wall-clock time, so machine load can no longer fail it. I ran it three times locally and it passed each time; tsc printed no errors.

Wall-clock time was the flake. Even taking the fastest of 9 runs, a stall from other suites sharing the machine could push every sample over the 50 ms budget. The server and client run in one process, so process.cpuUsage around each request captures the page's real work. A stall adds wall time but no CPU.

The budget is still 50 ms, for both the 500-entry test and the two-chapter test, and the page is not changed. Locally the fastest run is about 6–7 CPU-ms for the plain page and 7–12 CPU-ms with chapters. Both tests still check for a 200 status and the expected content. The change is only in test/changelog-timing.test.ts.

Not done: the test no longer catches a regression that is slow only because it waits (for example on I/O), since waiting costs no CPU. I judged that acceptable for a flake fix. I did not run the full suite; the forge runs it on push.

Approved

Test-only flake fix; no page, no UI, no new dependency, no README change. Measuring CPU per GET removes the machine-load flake, which fits the direction (fast tests that never flake). The 50 ms budget is kept and the 200 status and content checks remain. Nothing here touches the preview experience, so there is nothing to see there.

Two things I would do next:

  1. Rename the two tests so they say cpu-ms. The titles still read "answers in under 50 ms", which now misstates what is measured. Update the issue number in the file header too.
  2. The test can no longer catch a page that is slow only because it waits (I/O, a lock). Keep one cheap wall-clock guard with a generous bound, for example a single GET under 1 s, so a hang still fails.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs