Public: anyone can read this; the crew can change it.

Performance watcher: Server-Timing budgets for home, boards and repo pages from real traffic, slow phase filed with numbers #7559

shipped production is serving it opened this has it
opened this

Part of the epic #7553. Blocked by the maintainer framework (#7554).

Problem

The 7 s home page (#7494: caches keyed on the write stamp never hit; readKpis 4.4 s) was found by the steward reading Server-Timing by hand. The forge emits Server-Timing (src/core/route_timing.ts, src/http/server.ts) but nothing aggregates it or compares it to a budget, and the 0.7.277 outage (an 8 s synchronous scan on every /boards render) only showed under live load.

Build (duty: performance-watcher)

  • A rolling record of real requests per route pattern: total ms and each Server-Timing phase, p50/p95, count, cache hit or miss where a phase says so (new files under src/perf/; capped memory; route patterns, never ids, query strings or personal data).
  • Budgets per route pattern in one file, starting with home /, /boards, repo pages, /explore, /api/v1/health, each in words ("home p95 under 800 ms warm") with the number beside it.
  • Fire when p95 is over budget for 15 min with enough real requests. The finding names the route, the slowest phase and its share, and the cache hit rate if the phase is a cache; one issue per route and phase.
  • A labelled synthetic probe (once a minute, as an admin) so a route with no traffic is still measured; its requests do not count as traffic.
  • Event-loop block detector: when src/kernel/loop_lag.ts p99 is over its limit, attribute it to the route active (how 0.7.277 would be named in minutes).

Done when

  • Seeded Server-Timing data where kpis takes 4.4 s on home files one issue naming route, phase, p95 and share; a fast route files nothing.
  • A cache that never hits (0 hits in 100 requests) is named in the finding.
  • Tests: test/perf-aggregate.test.ts, test/perf-budget-fire.test.ts, test/perf-no-pii.test.ts, test/perf-synthetic-labelled.test.ts, test/perf-loop-lag-attribution.test.ts.
  • Links: #7494 (open), #6273 and #6556 (Server-Timing phases on home), #5642.
commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

commented

Owner, 2026-11-01: "like the throughput subsystem, we should have deep metrics on user activity and response times to constantly improve efficiency and concistent speed." Widen this watcher: besides server phases, measure what people actually get: real browser navigation timings by page, p50/p95/p99 so consistency shows and not only the average, and which pages people use most, so the most used pages get fixed first. Record only timings and page shapes, no personal content. This sits beside the throughput keeper #7645.

commented

Owner, 2026-11-02, after home's server time fell from about 1,280 ms to 32 ms: "performance metrics are a key part to us tuning how good the actual forge is as a system." Top priority now; do not wait on the maintainer framework if #7554 is not done. Ship the smallest useful version first, then grow it.

First version:

  • Keep p50, p95 and p99 per page from Server-Timing, split by signed-in and signed-out, with each phase, over 24 h and 7 d.
  • Keep event-loop stalls per minute, with the worst ones, plus the statements holding the thread with their call site. Today's numbers to beat: 111 stalls in 10 minutes, max 5.4 s; #7874 is fixing the worst.
  • Add real browser timings, first byte and first paint, by page, with only timings and page shape recorded.
  • Show it as one board that says what is slowest right now and how it has moved since last week.
  • A release shows what it did to these numbers: compare before and after, per page, on /releases and the changelog.

A page or statement over budget files one issue with its numbers, once.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs