Public: anyone can read this; the crew can change it.

"warm home holds p95 under the stated budget" passes every time on a loaded check machine #5257

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

`warm home holds p95 under the stated budget` (`test/home-cache.test.ts`) now checks which statements a warm hit runs instead of the page's wall-clock time. The source is unchanged: the flake was in the test, not a slowdown.

shipped production is serving it deploy 4915f35

opened this

warm home holds p95 under the stated budget (test/home-cache.test.ts) now checks which statements a warm hit runs instead of the page's wall-clock time. The source is unchanged: the flake was in the test, not a slowdown.

Cause: from the flake list, a budget measured end to end instead of in the code it guards. The old test timed 30 full / requests against 200 ms p95, but the feed takes most of that time and the machine's load sets the rest. The kpis and panels phases the test is about can't be timed either: on a loaded machine a warm hit took 0.4–48 ms and a miss 0.8–43 ms, so no time limit tells them apart.

What the test checks now (it costs the same statements on any machine):

  • Baseline: it warms the page, empties the KPI and panel caches, and records the statements that miss runs and the warm page doesn't. That is the KPI reads and the pinned panel's own query, 9 statements on this fixture.
  • The 30 warm requests: each must show kpis and panels as desc="hit" in Server-Timing, and must run none of those statements.
  • Requests near a write: the route-timing flush runs on a timer and writes to the database, and the session's last-seen update writes too. Either moves the write clock, so the next request misses for a correct reason. A request is skipped when the write clock moved between the start of the request before it and its own end; at least 20 of the 30 must still be checked.
  • Feed expiry: the feed cache can expire mid-run and add statements, so the test looks only at the miss-only statements, not the page's total count.

docs/CACHING.md §Budgets now says this, and that the 50 ms figure is read off the bench end to end.

How it was checked:

  • bun test --rerun-each 30 test/home-cache.test.ts: 90 pass.
  • The same rerun with two CPU-spinning shells and three other test files running beside it: 90 pass.
  • bun test --rerun-each 30 test/home-cache.test.ts test/map-cache.test.ts test/counts.test.ts while a full bun test test/ ran in parallel: 540 pass.
  • bun x tsc --noEmit is clean.
  • With src/web/studio/home_cache.ts broken three ways, the test failed each time:

- both caches cleared on every call, so every warm request misses; - only the panel cache cleared that way; - the cache still hitting but readKpis also called on every request.

Not done: no change to the source, and no time bound kept or widened on / or on the two phases. The bench still reports the 50 ms figure.

No prompt injection was found in the issue or the repository.

🤖 Generated with Claude Code

Closes #5237

commented

warm home holds p95 under the stated budget (test/home-cache.test.ts) now checks which statements a warm hit runs instead of the page's wall-clock time. The source is unchanged: the flake was in the test, not a slowdown.

Cause: from the flake list, a budget measured end to end instead of in the code it guards. The old test timed 30 full / requests against 200 ms p95, but the feed takes most of that time and the machine's load sets the rest. The kpis and panels phases the test is about can't be timed either: on a loaded machine a warm hit took 0.4–48 ms and a miss 0.8–43 ms, so no time limit tells them apart.

What the test checks now (it costs the same statements on any machine):

  • Baseline: it warms the page, empties the KPI and panel caches, and records the statements that miss runs and the warm page doesn't. That is the KPI reads and the pinned panel's own query, 9 statements on this fixture.
  • The 30 warm requests: each must show kpis and panels as desc="hit" in Server-Timing, and must run none of those statements.
  • Requests near a write: the route-timing flush runs on a timer and writes to the database, and the session's last-seen update writes too. Either moves the write clock, so the next request misses for a correct reason. A request is skipped when the write clock moved between the start of the request before it and its own end; at least 20 of the 30 must still be checked.
  • Feed expiry: the feed cache can expire mid-run and add statements, so the test looks only at the miss-only statements, not the page's total count.

docs/CACHING.md §Budgets now says this, and that the 50 ms figure is read off the bench end to end.

How it was checked:

  • bun test --rerun-each 30 test/home-cache.test.ts: 90 pass.
  • The same rerun with two CPU-spinning shells and three other test files running beside it: 90 pass.
  • bun test --rerun-each 30 test/home-cache.test.ts test/map-cache.test.ts test/counts.test.ts while a full bun test test/ ran in parallel: 540 pass.
  • bun x tsc --noEmit is clean.
  • With src/web/studio/home_cache.ts broken three ways, the test failed each time:

- both caches cleared on every call, so every warm request misses; - only the panel cache cleared that way; - the cache still hitting but readKpis also called on every request.

Not done: no change to the source, and no time bound kept or widened on / or on the two phases. The bench still reports the 50 ms figure.

No prompt injection was found in the issue or the repository.

🤖 Generated with Claude Code

Approved

The test now checks what the cached code runs (warm hits of kpis/panels run none of the miss-only statements) instead of wall-clock time, so it no longer depends on machine load. It still fails if the cache is broken. Source is untouched, the CACHING.md note matches, and test/home-cache.test.ts passes here (3 pass). Two things next: (1) the miss-only assertion pins one literal SQL string, so a reworded panel query fails it with a confusing message; derive it from the pinned panel instead. (2) Add the three mutation checks from the description as a note in the doc or a comment so the next person knows what the test guards.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs