Observability 2: every panel's data as JSON/CSV, bucketed on the server - charts never stop at 1000 points #3713

Every board panel's data is now available as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=&format=csv. Charts are bucketed in SQL to the step and are no longer cut off at 1000 points, so /boards/usage over 24 h now runs right up to now.

Cause. runQuery ended every query with LIMIT 1000 and sorted oldest first, so a day of per-minute buckets kept the first 1000 minutes and lost the newest ~7 hours.

What changed

• src/boards/query.ts: a query grouped by a time bucket can take a step (seconds, 1 s to a year). Each point is rounded down to its bucket in UTC. Minute, hour and day buckets give the same labels as before, so the existing query tests pass unchanged. A time-grouped answer is never cut short: past 20 000 rows it is refused (tooMany, "ask for a coarser step"). Queries grouped by a column (like by state) still stop at 1000 rows, as before.
• src/boards/panel_data.ts (new): reads one panel over a window and step. runFor moved here from web/pages/boards.ts.

- It parses and checks from, to and step (300, 15s, 5m, 1h, 1d). - Without a step, a time panel picks the finest standard step that keeps it within 360 buckets: a 24 h board gets 5 minutes (288 buckets), 7 days gets 30 minutes. If an answer is still too large, it tries the next coarser step. - A step you ask for is kept, or refused if too fine — never truncated. Panels grouped by a column ignore step and report "step": null. - CSV output is group,value or group,series,value. A text cell starting with =, +, - or @ gets a leading ' so a spreadsheet won't run it as a formula.

• src/http/api.ts: the new route, admin people only, like the other boards routes. :id is the panel's position on the board, counting from 0. A bad window or format gets 422, a missing board or panel 404, a non-admin 403, signed out 401.
• Board page and home's pinned panels (boards-view.ts, studio/pinned.ts) read through the same module, so the charts on screen match the API. The pinned-panel cache now includes the step in its key.
• Docs: docs/APPS.md and README.md describe step, the new route and the row limits.

Not done: the explore page (/boards/data) has no step field yet; I added no MCP tool for panel data; and buckets with no data are left out, not filled with zeros.

Tests: test/boards-panel-data.test.ts covers the three checks from the issue:

• 24 h of 1-minute data comes back up to the newest minute (1440 rows; the old limit would return 1000).
• step=5m over an aligned day returns 288 buckets.
• The CSV rows equal the JSON rows.

It also covers the coarser-step retry, refusing a step that is too fine, formula escaping, and the error statuses. The type check is clean, and the boards, pins, home, usage, OpenAPI and auth suites pass (650 tests).

To try it: as an admin, open /boards/usage; the CPU charts should end at the current time. Then fetch /api/v1/boards/usage/panels/0?step=5m and the same address with &format=csv.

🤖 Generated with Claude Code

Closes #3691

Every board panel's data is now available as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=&format=csv. Charts are bucketed in SQL to the step and are no longer cut off at 1000 points, so /boards/usage over 24 h now runs right up to now.

Cause. runQuery ended every query with LIMIT 1000 and sorted oldest first, so a day of per-minute buckets kept the first 1000 minutes and lost the newest ~7 hours.

What changed

• src/boards/query.ts: a query grouped by a time bucket can take a step (seconds, 1 s to a year). Each point is rounded down to its bucket in UTC. Minute, hour and day buckets give the same labels as before, so the existing query tests pass unchanged. A time-grouped answer is never cut short: past 20 000 rows it is refused (tooMany, "ask for a coarser step"). Queries grouped by a column (like by state) still stop at 1000 rows, as before.
• src/boards/panel_data.ts (new): reads one panel over a window and step. runFor moved here from web/pages/boards.ts.

- It parses and checks from, to and step (300, 15s, 5m, 1h, 1d). - Without a step, a time panel picks the finest standard step that keeps it within 360 buckets: a 24 h board gets 5 minutes (288 buckets), 7 days gets 30 minutes. If an answer is still too large, it tries the next coarser step. - A step you ask for is kept, or refused if too fine — never truncated. Panels grouped by a column ignore step and report "step": null. - CSV output is group,value or group,series,value. A text cell starting with =, +, - or @ gets a leading ' so a spreadsheet won't run it as a formula.

• src/http/api.ts: the new route, admin people only, like the other boards routes. :id is the panel's position on the board, counting from 0. A bad window or format gets 422, a missing board or panel 404, a non-admin 403, signed out 401.
• Board page and home's pinned panels (boards-view.ts, studio/pinned.ts) read through the same module, so the charts on screen match the API. The pinned-panel cache now includes the step in its key.
• Docs: docs/APPS.md and README.md describe step, the new route and the row limits.

Not done: the explore page (/boards/data) has no step field yet; I added no MCP tool for panel data; and buckets with no data are left out, not filled with zeros.

Tests: test/boards-panel-data.test.ts covers the three checks from the issue:

• 24 h of 1-minute data comes back up to the newest minute (1440 rows; the old limit would return 1000).
• step=5m over an aligned day returns 288 buckets.
• The CSV rows equal the JSON rows.

It also covers the coarser-step retry, refusing a step that is too fine, formula escaping, and the error statuses. The type check is clean, and the boards, pins, home, usage, OpenAPI and auth suites pass (650 tests).

To try it: as an admin, open /boards/usage; the CPU charts should end at the current time. Then fetch /api/v1/boards/usage/panels/0?step=5m and the same address with &format=csv.

🤖 Generated with Claude Code

Every board panel's data is now available as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=&format=csv. Charts are bucketed in SQL to the step and are no longer cut off at 1000 points, so /boards/usage over 24 h now runs right up to now.

Cause. runQuery ended every query with LIMIT 1000 and sorted oldest first, so a day of per-minute buckets kept the first 1000 minutes and lost the newest ~7 hours.

What changed

• src/boards/query.ts: a query grouped by a time bucket can take a step (seconds, 1 s to a year). Each point is rounded down to its bucket in UTC. Minute, hour and day buckets give the same labels as before, so the existing query tests pass unchanged. A time-grouped answer is never cut short: past 20 000 rows it is refused (tooMany, "ask for a coarser step"). Queries grouped by a column (like by state) still stop at 1000 rows, as before.
• src/boards/panel_data.ts (new): reads one panel over a window and step. runFor moved here from web/pages/boards.ts.

- It parses and checks from, to and step (300, 15s, 5m, 1h, 1d). - Without a step, a time panel picks the finest standard step that keeps it within 360 buckets: a 24 h board gets 5 minutes (288 buckets), 7 days gets 30 minutes. If an answer is still too large, it tries the next coarser step. - A step you ask for is kept, or refused if too fine — never truncated. Panels grouped by a column ignore step and report "step": null. - CSV output is group,value or group,series,value. A text cell starting with =, +, - or @ gets a leading ' so a spreadsheet won't run it as a formula.

• src/http/api.ts: the new route, admin people only, like the other boards routes. :id is the panel's position on the board, counting from 0. A bad window or format gets 422, a missing board or panel 404, a non-admin 403, signed out 401.
• Board page and home's pinned panels (boards-view.ts, studio/pinned.ts) read through the same module, so the charts on screen match the API. The pinned-panel cache now includes the step in its key.
• Docs: docs/APPS.md and README.md describe step, the new route and the row limits.

Fix for the failed check: the test itself was wrong, not the code. It seeded 1440 per-minute samples, then asked for a window that starts 24 h before the request. When a minute boundary passed in between, the oldest sample fell outside the window and 1439 rows came back. The 1440-minute test and the CSV-matches-JSON test now give an explicit from at the oldest sample, so they no longer depend on the clock.

Not done: the explore page (/boards/data) has no step field yet; I added no MCP tool for panel data; and buckets with no data are left out, not filled with zeros.

Tests: test/boards-panel-data.test.ts covers the three checks from the issue:

• 24 h of 1-minute data comes back up to the newest minute (1440 rows; the old limit would return 1000).
• step=5m over an aligned day returns 288 buckets.
• The CSV rows equal the JSON rows.

It also covers the coarser-step retry, refusing a step that is too fine, formula escaping, and the error statuses. After the fix, that file passes (13 tests) and the type check is clean. The wider boards, pins, home and API suites passed on the previous push and were not re-run here.

To try it: as an admin, open /boards/usage; the CPU charts should end at the current time. Then fetch /api/v1/boards/usage/panels/0?step=5m and the same address with &format=csv.

🤖 Generated with Claude Code

Every board panel's data is available as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=&format=csv. Charts are now grouped into time buckets in SQL, so they no longer stop at 1000 points, and /boards/usage over 24 h runs right up to now. Following review, the board page, MCP and the terminal now all lead to that data.

Cause. runQuery ended every query with LIMIT 1000 and sorted oldest first. A day of per-minute buckets therefore kept the first 1000 minutes and dropped the newest ~7 hours.

What changed

• src/boards/query.ts: a query grouped by a time bucket can take a step (seconds, from 1 s to a year), and each point is rounded down to its bucket in UTC.

- Minute, hour and day buckets give the same labels as before. - A time-grouped answer is never cut short: past 20 000 rows it is refused (tooMany, "ask for a coarser step"). - Queries grouped by a column still stop at 1000 rows.

• src/boards/panel_data.ts (new): reads one panel over a window and a step. runFor moved here from web/pages/boards.ts.

- It checks from, to and step (300, 15s, 5m, 1h, 1d). - Without a step, a time panel picks the finest standard step that keeps it within 360 buckets (24 h gives 5 minutes). If the answer is still too large, it moves up to the next coarser step. - A step that was asked for is kept, or refused if it is too fine; it is never truncated. - CSV is group,value or group,series,value. A text cell starting with =, +, - or @ gets a leading ' so a spreadsheet won't run it as a formula.

• src/http/api.ts: the route, for admin people only. :id is the panel's position on the board, counting from 0. A bad window or format gets 422, a missing board or panel 404, a non-admin 403, and signed out 401.
• Panel cards (from review): each card on the board page and on home's pinned panels links json and csv beside explore. These are plain links, and csv downloads a file named <slug>-<n>.csv. drawPanel now takes the panel's { slug, index }.
• MCP (from review): instead of adding a separate board_panel tool, the existing boards tool takes panel (with optional from, to, step) next to slug. That reuses its admin-person gate. It returns {board, panel, title, chart, from, to, step, rows} from panelData, the same code the API uses.
• Terminal: the board screen shows each panel's latest reading: a time panel's last bucket and step, another panel's row count, or why it was refused.
• Docs: docs/APPS.md and README.md describe step, the route, the card links, the MCP option and the row limits.

Not done: the explore page (/boards/data) has no step field yet, and buckets with no data are left out rather than filled with zeros.

Tests:

• 24 h of 1-minute data reaches the newest minute (1440 rows).
• step=5m over an aligned day returns 288 buckets.
• The CSV rows equal the JSON rows.
• The board page links each panel's JSON and CSV.
• MCP boards with panel returns exactly what panelData returns, and refuses agents and non-admins.
• The terminal shows the last-bucket line.

The type check is clean, and the boards, pins, MCP, TUI, home and studio suites pass.

To try it: as an admin, open /boards/usage; the CPU charts should end at the current time. Click a card's csv, or fetch /api/v1/boards/usage/panels/0?step=5m.

🤖 Generated with Claude Code

Closes #3691

Every board panel's data is available as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=&format=csv. Charts are grouped into time buckets in SQL, so they no longer stop at 1000 points, and /boards/usage over 24 h runs right up to now. The board page, MCP and the terminal all lead to that data.

Cause. runQuery ended every query with LIMIT 1000 and sorted oldest first. A day of per-minute buckets therefore kept the first 1000 minutes and dropped the newest ~7 hours.

What changed

• src/boards/query.ts: a query grouped by a time bucket can take a step (seconds, from 1 s to a year), and each point is rounded down to its bucket in UTC.

- Minute, hour and day buckets give the same labels as before. - A time-grouped answer is never cut short: past 20 000 rows it is refused (tooMany, "ask for a coarser step"). - Queries grouped by a column still stop at 1000 rows.

• src/boards/panel_data.ts (new): reads one panel over a window and a step. runFor moved here from web/pages/boards.ts.

- It checks from, to and step (300, 15s, 5m, 1h, 1d). - Without a step, a time panel picks the finest standard step that keeps it within 360 buckets (24 h gives 5 minutes). If the answer is still too large, it moves up to the next coarser step. - A step that was asked for is kept, or refused if it is too fine; it is never truncated. - CSV is group,value or group,series,value. A text cell starting with =, +, - or @ gets a leading ' so a spreadsheet won't run it as a formula.

• src/http/api.ts: the route, for admin people only. :id is the panel's position on the board, counting from 0. A bad window or format gets 422, a missing board or panel 404, a non-admin 403, and signed out 401.
• Panel cards: each card on the board page and on home's pinned panels links json and csv beside explore. These are plain links, and csv downloads a file named <slug>-<n>.csv. drawPanel now takes the panel's { slug, index }.
• MCP: instead of adding a separate board_panel tool as the review suggested, the existing boards tool takes panel (with optional from, to, step) next to slug. That reuses its admin-person gate. It returns {board, panel, title, chart, from, to, step, rows} from panelData, the same code the API uses.
• Terminal: the board screen shows each panel's latest reading: a time panel's last bucket and step, another panel's row count, or why it was refused.
• Docs: docs/APPS.md and README.md describe step, the route, the card links, the MCP option and the row limits.

Test fix: the check failed because the test was wrong, not the code. The "aligned day" window in test/boards-panel-data.test.ts ended at the last 5-minute mark before the newest point. When the clock was 4 minutes past a mark, the window began a minute before the oldest point and returned 287 buckets. It now ends at the first 5-minute mark after the newest point. I checked all five possible minute positions and each gives 288 buckets.

Not done: the explore page (/boards/data) has no step field yet, and buckets with no data are left out rather than filled with zeros.

Tests:

• 24 h of 1-minute data reaches the newest minute (1440 rows).
• step=5m over an aligned day returns 288 buckets.
• The CSV rows equal the JSON rows.
• The board page links each panel's JSON and CSV.
• MCP boards with panel returns exactly what panelData returns, and refuses agents and non-admins.
• The terminal shows the last-bucket line.

After the test fix I re-ran only the panel-data suite, which passes. Before it, the type check was clean and the boards, pins, MCP, TUI, home and studio suites passed.

To try it: as an admin, open /boards/usage; the CPU charts should end at the current time. Click a card's csv, or fetch /api/v1/boards/usage/panels/0?step=5m.

🤖 Generated with Claude Code

Closes #3691

Every board panel's data is available as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=&format=csv. Charts are grouped into time buckets in SQL, so they no longer stop at 1000 points, and /boards/usage over 24 h runs right up to now. The board page, MCP and the terminal all lead to that data.

Cause. runQuery ended every query with LIMIT 1000 and sorted oldest first. A day of per-minute buckets therefore kept the first 1000 minutes and dropped the newest ~7 hours.

What changed

• src/boards/query.ts: a query grouped by a time bucket can take a step (seconds, from 1 s to a year), and each point is rounded down to its bucket in UTC.

- Minute, hour and day buckets give the same labels as before. - A time-grouped answer is never cut short: past 20 000 rows it is refused (tooMany, "ask for a coarser step"). - Queries grouped by a column still stop at 1000 rows. - The source, scope, window and filter checks moved out of runQuery into rowsOf. The new oldestTime uses them to read MIN(time) over the same rows the query reads.

• src/boards/panel_data.ts (new): reads one panel over a window and a step. runFor moved here from web/pages/boards.ts.

- It checks from, to and step (300, 15s, 5m, 1h, 1d). - Without a step, a time panel picks the finest standard step that keeps it within 360 buckets (24 h gives 5 minutes). If the answer is still too large, it moves up to the next coarser step. - When the window has no start (a board whose range is all), the span runs from the panel's oldest row, so the step still fits about 360 points. Before, such a panel ran at its own bucket width, up to 20 000 points, with a full re-scan for each coarser step it tried. - A step that was asked for is kept, or refused if it is too fine; it is never truncated. - CSV is group,value or group,series,value. A text cell starting with =, +, - or @ gets a leading ' so a spreadsheet won't run it as a formula.

• src/http/api.ts: the route, for admin people only. :id is the panel's position on the board, counting from 0. A bad window or format gets 422, a missing board or panel 404, a non-admin 403, and signed out 401.
• Panel cards: each card on the board page and on home's pinned panels links explore, json and csv. The csv link downloads a file named <slug>-<n>.csv.

- The three links sit in one .card-links class, defined in style/components.ts and named in docs/DESIGN.md §5. - Under (pointer: coarse) each link is a 44 px target, as .topbar-tools > a.small is. They stay in the card head rather than moving to .card-foot.

• MCP: instead of adding a separate board_panel tool, the existing boards tool takes panel (with optional from, to, step) next to slug. That reuses its admin-person gate. It returns {board, panel, title, chart, from, to, step, rows} from panelData, the same code the API uses.
• Terminal: the board screen shows each panel's latest reading: a time panel's last bucket and step, another panel's row count, or why it was refused.
• Docs: docs/APPS.md and README.md describe step, the route, the card links, the MCP option, the row limits and how an all board picks its step.

Not done: the explore page (/boards/data) has no step field yet, and buckets with no data are left out rather than filled with zeros.

Tests (test/boards-panel-data.test.ts unless noted):

• 24 h of 1-minute data reaches the newest minute (1440 rows).
• step=5m over an aligned day returns 288 buckets. An earlier version of this test depended on the clock and failed one minute in five. That was the test's fault, not the code's; it now holds on any minute.
• The CSV rows equal the JSON rows.
• An all board picks a 5-minute step for a day of data. This test fails without the fix, which stayed at the 60 s bucket.
• oldestTime respects the query's filters and window, and returns nothing for no rows, a refused query or an empty scope.
• The card head carries .card-links, and the stylesheet gives it the 44 px coarse-pointer rule.
• MCP boards with panel returns exactly what panelData returns, and refuses agents and non-admins (test/mcp.test.ts).
• The terminal shows each panel's latest reading (test/tui-boards.test.ts).

The type check is clean, and the boards, pins, MCP, TUI, home, components, usage and design-check suites pass.

To try it: as an admin, open /boards/usage; the CPU charts should end at the current time. On a phone, each of explore, json and csv should be a full-height tap target. Then fetch /api/v1/boards/usage/panels/0?step=5m, or click a card's csv.

🤖 Generated with Claude Code

Closes #3691

Every board panel's data is available as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=&format=csv. Charts are grouped into time buckets in SQL, so they no longer stop at 1000 points, and /boards/usage over 24 h runs right up to now. The board page, MCP and the terminal all lead to that data.

Cause. runQuery ended every query with LIMIT 1000 and sorted oldest first. A day of per-minute buckets therefore kept the first 1000 minutes and dropped the newest ~7 hours.

What changed

• src/boards/query.ts: a query grouped by a time bucket can take a step (seconds, from 1 s to a year), and each point is rounded down to its bucket in UTC.

- Minute, hour and day buckets give the same labels as before. - A time-grouped answer is never cut short: past 20 000 rows it is refused (tooMany, "ask for a coarser step"). - Queries grouped by a column still stop at 1000 rows. - The source, scope, window and filter checks moved out of runQuery into rowsOf. The new oldestTime uses them to read MIN(time) over the same rows the query reads.

• src/boards/panel_data.ts (new): reads one panel over a window and a step. runFor moved here from web/pages/boards.ts.

- It checks from, to and step (300, 15s, 5m, 1h, 1d). - Without a step, a time panel picks the finest standard step that keeps it within 360 buckets (24 h gives 5 minutes). If the answer is still too large, it moves up to the next coarser step. - When the window has no start (a board whose range is all), the span runs from the panel's oldest row, so the step still fits about 360 points. - A step that was asked for is kept, or refused if it is too fine; it is never truncated. - CSV is group,value or group,series,value. A text cell starting with =, +, - or @ gets a leading ' so a spreadsheet won't run it as a formula.

• src/http/api.ts: the route, for admin people only. :id is the panel's position on the board, counting from 0. A bad window or format gets 422, a missing board or panel 404, a non-admin 403, and signed out 401.
• Panel cards: each card on the board page and on home's pinned panels links explore, json and csv. The csv link downloads a file named <slug>-<n>.csv.

- The three links sit in one .card-links class, defined in style/components.ts and named in docs/DESIGN.md §5. - Under (pointer: coarse) each link is a 44 px target.

• MCP: instead of adding a separate board_panel tool, the existing boards tool takes panel (with optional from, to, step) next to slug. That reuses its admin-person gate. It returns {board, panel, title, chart, from, to, step, rows} from panelData, the same code the API uses.
• Terminal: the board screen shows each panel's latest reading: a time panel's last bucket and step, another panel's row count, or why it was refused.
• Docs: docs/APPS.md and README.md describe step, the route, the card links, the MCP option, the row limits and how an all board picks its step.
• Merge with main: the only conflict was docs/DESIGN.md §5. This branch added .card-links to the card row, and main added fold to the list row on the line below; both are kept.

Not done: the explore page (/boards/data) has no step field yet, and buckets with no data are left out rather than filled with zeros.

Tests (test/boards-panel-data.test.ts unless noted):

• 24 h of 1-minute data reaches the newest minute (1440 rows).
• step=5m over an aligned day returns 288 buckets on any minute of the clock.
• The CSV rows equal the JSON rows.
• An all board picks a 5-minute step for a day of data, not its 60 s bucket.
• oldestTime respects the query's filters and window, and returns nothing for no rows, a refused query or an empty scope.
• The card head carries .card-links, and the stylesheet gives it the 44 px coarse-pointer rule.
• MCP boards with panel returns exactly what panelData returns, and refuses agents and non-admins (test/mcp.test.ts).
• The terminal shows each panel's latest reading (test/tui-boards.test.ts).

After the merge, the type check is clean and the design-check, boards, MCP, components and TUI-boards suites pass. The home, pins and usage suites last passed before the merge.

To try it: as an admin, open /boards/usage; the CPU charts should end at the current time. On a phone, each of explore, json and csv should be a full-height tap target. Then fetch /api/v1/boards/usage/panels/0?step=5m, or click a card's csv.

🤖 Generated with Claude Code

Closes #3691

Every board panel's data can now be fetched as JSON or CSV at GET /api/v1/boards/:slug/panels/:id?from=&to=&step=. Charts are bucketed in SQL to the requested step, so they no longer stop at 1000 rows. A 24-hour panel of per-minute data now runs right up to the present, where before it ended about 7 hours early.

• Queries (3b801290): board queries take a step and bucket in SQL. If a time-grouped query would still be too large, it is refused rather than cut off.
• Endpoint (ee27c889, f6ee2c00): the panel picks the step unless the request names one. CSV holds the same rows as JSON. The tests use a fixed time window and check three things: 24 hours of 1-minute data reaches the newest point, step=5m gives 288 buckets, and CSV matches JSON.
• Panels with no start (82c1dc8e): an "all time" panel sizes its step from its oldest row, so it stays near 360 points.
• Cards (81147e21, be0ff9ee): each panel card links its JSON and a CSV download next to explore. The links share .card-links, and each is a 44 px target on touch screens.
• MCP and terminal (0203c606, 695c76af, 3756c7a3): the MCP boards tool with a slug and a panel returns that panel's data through panelData. The terminal board screen shows each panel's last bucket and step, and the docs list the new endpoint.

This push only merges four new commits from main. It merged cleanly, and nothing in #3691 changed. On the merged tree the type check passes, and the three test files this branch changes pass (54 tests). The issue still has the Later label, and the automatic triage note says it waits until someone names a pillar for it; that is the owner's call, so I left it alone.

To try it:

curl -H "authorization: Bearer $TOKEN" "$URL/api/v1/boards/usage/panels/<id>?step=5m"

Add format=csv, or use the CSV link on a panel card.

🤖 Generated with Claude Code