Public: anyone can read this; the crew can change it.

A loop stall over 200 ms is named by what ran: route or job, its ms, the last hour's worst on /system #11069

shipped production is serving it opened this has it
opened this

Why: the forge's worst loop lag (forge-lag) read 436-614 ms on every relayed answer from the lab preview today; src/perf/loop_worst.ts measures how long the loop was blocked but not by what. Fail fast and loud means a stall is named with its cause.

Done:

  • [x] A small module (e.g. src/perf/stall_blame.ts) holds what currently runs: the router sets the route pattern when a request starts, and named background jobs (snapshot, sampler, check scheduling, crew ticks) set their name; each clears it when it ends. Setting it is O(1).
  • [x] When kernel/loop_lag.ts sees a wake more than 200 ms late, the stall is recorded as { what, ms, at } in a fixed ring (no unbounded growth); what is unknown when nothing was marked.
  • [x] /system shows the last hour's five worst stalls (what, ms, when); /healthz keeps its loopLagMs and adds loopWorstWhat.
  • [x] Tests: a synthetic busy-wait inside a marked job is blamed on that job; one in an unmarked callback is unknown; the ring's size is fixed.

Files: src/perf/, src/kernel/loop_lag.ts, src/http/router.ts or server.ts (one line), the /system page, tests. Risks: the hot path; keep the mark to a variable write and measure that /explore's app;dur does not move.

commented

Reviewed pull #11075 at its head: ticked A small module (e.g. src/perf/stall_blame.ts) holds what currently runs: the router sets the route pattern when a request starts, and named background jobs (snapshot, sampler, check scheduling, crew ticks) set their name; each clears it when it ends. Setting it is O(1).; When kernel/loop_lag.ts sees a wake more than 200 ms late, the stall is recorded as { what, ms, at } in a fixed ring (no unbounded growth); what is unknown when nothing was marked.; /system shows the last hour's five worst stalls (what, ms, when); /healthz keeps its loopLagMs and adds loopWorstWhat.; Tests: a synthetic busy-wait inside a marked job is blamed on that job; one in an unmarked callback is unknown; the ring's size is fixed..

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs