The metrics page shows main-thread stall time by route, so a fix's before and after reads from the forge itself #9037

GET /metrics now shows how long the main thread stalled, broken down by route. Before and after numbers for a fix, like #8978's 69.6 s in 6 minutes on /api/v1/telemetry/flow, can now be read from the forge itself instead of the pod log.

• src/perf/loop.ts: the loop watch already names the routes running across each stall over 250 ms. It now also adds each stall to running totals by route since the process started, and keeps the last hour's stalls (bounded). The tally comes from tally(now). A stall is charged in full to every route that overlapped it, which is how the event loop blocked log lines read. So the sum over routes can be more than the wall time. A stall with no request running counts against route="none".
• src/metrics/stalls.ts (new), included in forgeMetrics, adds four series:

- forge_loop_stall_milliseconds_total{route} and forge_loop_stalls_total{route} count up from process start. - forge_loop_stall_hour_milliseconds{route} and forge_loop_stall_hour_count{route} cover the top 10 routes of the last hour, most first. - /api/v1/telemetry/flow is always in the hourly series, at 0 when it stalled nothing, so its figure can always be read.

• docs/METRICS.md lists the four series and says how to compare with #8978: about 700 s an hour at that rate, or increase(...[6m]) over the total for the same 6-minute window.
• test/loop-stall-by-route.test.ts: a stall during a route, on a clock the test moves, gets through the real watchStalls watcher and is counted against that route. It shows up as exactly 395 ms, so the test doesn't depend on wall-clock time. The file also covers charging every overlapping route and the none case, the hour dropping old stalls while the totals keep them, the top-10 cut with the flow route always listed, and a real request to /api/v1/health appearing in /metrics after a stall.

Not done:

• /metrics is Prometheus text, so "the metrics page lists the top routes" is met by the hourly series rather than a new HTML page.
• Stalls shorter than 250 ms are still not counted.
• A request still running when the watcher reports a stall isn't named. That limit was already there.
• I couldn't find the keep_plan tool in this session, so the issue has no plan.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. 134 tests pass across the related suites: flake, metrics-, perf-, loop-*, openmetrics, flow-cache, dependencies-named. scripts/fast.ts named 1,096 files, over its budget, so I chose those suites by hand and left the full suite to the forge's check.

To try it: read /metrics with an admin ro token on the preview and look for the forge_loop_stall_ series.

🤖 Generated with Claude Code

Closes #9016

GET /metrics now shows how long the main thread stalled, broken down by route. Before and after numbers for a fix, like #8978's 69.6 s in 6 minutes on /api/v1/telemetry/flow, can now be read from the forge itself instead of the pod log.

• src/perf/loop.ts: the loop watch already names the routes running across each stall over 250 ms. It now also adds each stall to running totals by route since the process started, and keeps the last hour's stalls (bounded). The tally comes from tally(now). A stall is charged in full to every route that overlapped it, which is how the event loop blocked log lines read. So the sum over routes can be more than the wall time. A stall with no request running counts against route="none".
• src/metrics/stalls.ts (new), included in forgeMetrics, adds four series:

- forge_loop_stall_milliseconds_total{route} and forge_loop_stalls_total{route} count up from process start. - forge_loop_stall_hour_milliseconds{route} and forge_loop_stall_hour_count{route} cover the top 10 routes of the last hour, most first. - /api/v1/telemetry/flow is always in the hourly series, at 0 when it stalled nothing, so its figure can always be read.

• docs/METRICS.md lists the four series and says how to compare with #8978: about 700 s an hour at that rate, or increase(...[6m]) over the total for the same 6-minute window.
• test/loop-stall-by-route.test.ts: a stall during a route, on a clock the test moves, gets through the real watchStalls watcher and is counted against that route. It shows up as exactly 395 ms, so the test doesn't depend on wall-clock time. The file also covers charging every overlapping route and the none case, the hour dropping old stalls while the totals keep them, the top-10 cut with the flow route always listed, and a real request to /api/v1/health appearing in /metrics after a stall.

Not done:

• /metrics is Prometheus text, so "the metrics page lists the top routes" is met by the hourly series rather than a new HTML page.
• Stalls shorter than 250 ms are still not counted.
• A request still running when the watcher reports a stall isn't named. That limit was already there.
• I couldn't find the keep_plan tool in this session, so the issue has no plan.

Checks: bun scripts/gate.ts and bun x tsc --noEmit pass. 134 tests pass across the related suites: flake, metrics-, perf-, loop-*, openmetrics, flow-cache, dependencies-named. scripts/fast.ts named 1,096 files, over its budget, so I chose those suites by hand and left the full suite to the forge's check.

To try it: read /metrics with an admin ro token on the preview and look for the forge_loop_stall_ series.

🤖 Generated with Claude Code

The metrics page now shows how long the main thread stalled, broken down by route. That data comes from the tally the loop watch already keeps. So a fix's before and after for /api/v1/telemetry/flow can be read from the forge itself instead of the pod log: #8978 measured 69.6 s in 6 minutes, about 700,000 ms an hour at that rate.

• The stall tally (src/perf/loop.ts): each stall over 250 ms is charged in full to every route pattern that ran across it, or to none when no request did. That is how the event loop blocked log lines read, so the sum over routes can be more than the wall time. It keeps totals since the process started and the stalls of the last hour.
• One selection, three readers (src/metrics/stalls.ts): stallHour() returns the hour's top 10 routes, most milliseconds first, plus /api/v1/telemetry/flow always (0 when it stalled nothing). These all read it:

- GET /metrics: four series. forge_loop_stall_milliseconds_total and forge_loop_stalls_total count since process start; forge_loop_stall_hour_milliseconds and forge_loop_stall_hour_count cover the last hour. - /system: a "Main-thread stalls, last hour" card listing route, stall ms and count. It is drawn once when the page loads, the route text is escaped, and it is gated like the rest of the page. - The terminal: : → "Main-thread stalls" (src/tui/screens/loopstalls.ts, entry in ENTRIES). Only a forge admin who is a person is offered it.

• Docs: docs/features/stalls-by-route.md is the how-to, including how to compare with #8978. docs/METRICS.md lists the four series.
• Tests (test/loop-stall-by-route.test.ts):

- A stall during a route, on a clock the test moves, goes through the real watchStalls watcher and is counted against that route as exactly 395 ms. - Charging every overlapping route, and the none case. - The hour dropping old stalls while the totals keep them. - The top-10 cut with the flow route always listed. - The /system card escapes route text; a member gets 404. - The terminal screen and its palette entry for an admin person, and for nobody else (member, admin agent, guest). - /metrics naming the route a stall ran across. That test feeds the process-wide loopWatch; a comment says so, and every assertion reads only route-specific lines.

• Changed test: test/palette-roster.test.ts gains the new palette entry. That test is a snapshot of every admin entry, so it was right to fail.

Not done: the /system card doesn't update live with the machine stream, since an hourly figure has no use for 2 s updates. Stalls under 250 ms aren't counted, and a request still running when the stall is detected isn't named; that limit was already there. I couldn't find the keep_plan tool in this session, so the issue has no plan.

Checks: bun scripts/gate.ts, bun x tsc --noEmit, bun scripts/design-lint.ts and scripts/hollow.ts pass. The covering suites pass: loop-stall-by-route, flake, metrics-, palette, system-*, tui-screens, tui-system, web-ui, readme-cap, perf-loop-lag-attribution. The full suite is left to the forge's check.

To try it: as an admin, open /system on the preview and look for the card, or the "Main-thread stalls" screen in the terminal.

🤖 Generated with Claude Code

The metrics page now shows how long the main thread stalled, broken down by route. Before and after numbers for a fix, like #8978's 69.6 s in 6 minutes on /api/v1/telemetry/flow, can now be read from the forge itself instead of the pod log.

• The stall tally (src/perf/loop.ts): each stall over 250 ms is charged in full to every route pattern that ran across it, or to none when no request did. That is how the event loop blocked log lines read, so the sum over routes can be more than the wall time. It keeps totals since the process started and the stalls of the last hour.
• One selection, three readers (src/metrics/stalls.ts): stallHour() returns the hour's top 10 routes, most milliseconds first, plus /api/v1/telemetry/flow always (0 when it stalled nothing). These all read it:

- GET /metrics: forge_loop_stall_milliseconds_total and forge_loop_stalls_total count since process start; forge_loop_stall_hour_milliseconds and forge_loop_stall_hour_count cover the last hour. - /system: a "Main-thread stalls, last hour" card listing route, stall ms and count. It is drawn once when the page loads, route text is escaped, and it is gated like the rest of the page. - The terminal: : → "Main-thread stalls" (src/tui/screens/loopstalls.ts). Only a forge admin who is a person is offered it.

• Docs: docs/features/stalls-by-route.md is the how-to, including how to compare with #8978 (about 700,000 ms an hour at that rate). docs/METRICS.md lists the four series.
• Tests (test/loop-stall-by-route.test.ts):

- A stall during a route, on a clock the test moves, goes through the real watchStalls watcher and is counted against that route as exactly 395 ms. - The hour dropping old stalls and the top-10 cut with the flow route always listed. - The /system card escapes route text; a member gets 404. - The terminal screen and its palette entry for an admin person, and for nobody else. - /metrics naming the route a stall ran across. That test feeds the process-wide watch, so its assertions read only route-specific lines, and a comment says why.

• Changed test: test/palette-roster.test.ts gains the new palette entry. That test is a snapshot of every admin entry, so it was right to fail.
• Check fix: the last push failed the gate because no commit message had a Try: or Why now: line. An empty commit now carries both; there is no code change since the previous push.

Not done: the /system card doesn't update live with the machine stream, since an hourly figure has no use for 2 s updates. Stalls under 250 ms aren't counted, and a request still running when the stall is detected isn't named; that limit was already there. I couldn't find the keep_plan tool in this session, so the issue has no plan.

Checks: bun scripts/gate.ts, bun x tsc --noEmit, design-lint and hollow pass. The covering suites passed on the previous push: loop-stall-by-route, flake, metrics-, palette, system-*, tui-screens, tui-system, web-ui, readme-cap, perf-loop-lag-attribution. The full suite is left to the forge's check.

To try it: as an admin, open /system on the preview and look for the card, or open "Main-thread stalls" in the terminal.

🤖 Generated with Claude Code