trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRuns14g rEventsFlows

Observability 2: every panel's data as JSON/CSV, bucketed on the server - charts never stop at 1000 points #3713

Merged masoncrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

mason/forge:mason/3691-observability-2-every-panel-s-data-as-js → main 11 commits fork 4 threads · 4 unresolved

  • Merged. mason/3691-observability-2-every-panel-s-data-as-js was merged into main as 577c2c4.

Review

Sign in to review this change.
15 files changed +632 −61
README.md +2 −2
@@ -1741,8 +1741,8 @@ work here.
17411741 - the `board_pins` MCP tool (no `op` lists; `op` pin, unpin or move changes). `pins` is therefore not a board slug.
17421742 The first boot also writes seven starting boards, **Usage**, **Crew**, **Releases**, **Apps**, **Security**, **Throughput** and **Forges**. Apps' per-app panels show repo ids until a source exposes a safe name; Security (sign-ins, failed sign-ins, tokens by scope, admin actions, from the admin-only `signins`, `tokens` and `adminActions` sources and `usage`) shows no token text, hash, payload or IP; Throughput is items entering each stage from issue to live, per hour, from the `stages` source. Each is written once: `.offered` in that repo records them, so an edit stays and a deletion sticks.
17431743 A live board (any `refresh` but `off`; the form offers off or on, and `10s` and `1m` are still accepted and mean the same) redraws its grid when a run, pull, check, issue or usage event arrives, through the live-region swap
1744 (`/boards/:slug/live`, etag'd, one fetch per burst). The same doors are in the API: `GET /api/v1/boards`, `GET|PUT|DELETE /api/v1/boards/:slug`
1745 (an agent or anyone but an admin is 403; a panel whose query cannot run is refused with 422). The same four doors are MCP tools (`boards`, `board_write`, `board_delete`; `boards` with a `slug` reads one) and, for the same person, a **Boards** screen in the terminal that lists boards and shows a board's panels as text. No dependency was added.
1744 (`/boards/:slug/live`, etag'd, one fetch per burst). The same doors are in the API: `GET /api/v1/boards`, `GET|PUT|DELETE /api/v1/boards/:slug`, and each panel's data as JSON or CSV at `GET /api/v1/boards/:slug/panels/:id?from=&to=&step=5m&format=csv`, bucketed in SQL to the step and never cut short (`docs/APPS.md`); each panel's card links `json` and `csv` (a plain download link) beside `explore`
1745 (an agent or anyone but an admin is 403; a panel whose query cannot run is refused with 422). The same four doors are MCP tools (`boards`, `board_write`, `board_delete`; `boards` with a `slug` reads one, and with a `panel` too that panel's data, with the same `from`, `to` and `step`) and, for the same person, a **Boards** screen in the terminal that lists boards and shows a board's panels as text, each with what its data holds now (a time panel's last bucket and step, another's row count). No dependency was added.
17461746 - **Issues** — labels, threaded comments, close/reopen, markdown.
17471747 - **Pull requests** — open from a branch, a **beautiful diff viewer** (syntax
17481748 highlighting composed with intra-line word highlighting, mobile-first,
docs/APPS.md +25 −1
@@ -1206,11 +1206,35 @@ a column holding a secret or free text is never listed).
12061206 - `filter`: up to 16 of `{column, op, value}`, `op` one of `eq ne lt lte gt gte`,
12071207 or `starts` (a prefix of a text or time column, matched literally).
12081208 - `groupBy`: a column, or `{"bucket": "minute" | "hour" | "day"}`.
1209 - `step`: with a bucket only, seconds per bucket in its place (1 to a
1210 year); each point is floored to its bucket in UTC.
12091211
1210 It answers `{"rows": [{"group", "value"}]}`, at most 1000 rows. A query the
1212 It answers `{"rows": [{"group", "value"}]}`. A query grouped by a column
1213 stops at 1000 rows; one grouped by time is never cut short: past 20000
1214 rows it is refused (`400 bad_query`, "ask for a coarser step"). A query the
12111215 module refuses is `400 bad_query` with its message; a body that is not JSON
12121216 is `400 bad_json`; a body over the cap is `413 too_large`.
12131217
1218 Each panel of a board is its own door, `GET
1219 /api/v1/boards/:slug/panels/:id` (`:id` its place from 0;
1220 `boards/panel_data.ts`), answering `{board, panel, title, chart, from, to,
1221 step, rows}`. `from` and `to` (dates) stand in for the panel's, and the
1222 board's range is the start when neither names one; `step` is seconds or a
1223 count with a unit (`300`, `15s`, `5m`, `1h`, `1d`). Without a step a panel
1224 grouped by time picks the finest of 1 min, 2, 5, 10, 15, 30 min, 1, 2, 3,
1225 6, 12 h, a day or a week that keeps it within 360 buckets (a 24 h board is
1226 5 min, 288 buckets; with no start, as on an `all` board, the span runs from
1227 the panel's oldest row), and steps coarser while an answer is too large; a step
1228 asked for is kept or refused, never cut. A panel grouped by a column
1229 ignores `step` (`"step": null`). `format=csv` is the same rows as
1230 `group,value` or `group,series,value`; a text cell that starts with `=`,
1231 `+`, `-` or `@` is prefixed with `'` so a spreadsheet does not run it. A bad
1232 window or format is `422`, a missing board or panel `404`. The board page
1233 and home's pinned panels draw through the same door, and each card links its
1234 `json` and `csv` beside `explore`; the `boards` MCP tool with `slug` and
1235 `panel` answers the same, and the terminal's board screen says each panel's
1236 last bucket and step.
1237
12141238 The `machine` source is what the machine itself used (`boards/sampler.ts`):
12151239 every 15 s the forge reads its pod's cgroup v2 (`cpu.stat` `usage_usec` as a
12161240 rate, `memory.current`), each labelled check and app cell's process tree
docs/DESIGN.md +1 −1
@@ -156,7 +156,7 @@ specification.
156156 | **progress bar** `progress()` (`pages/milestones.ts`) | `progress` (element rule in `style/components.ts`; every `<progress>` on the site except `.tests-bar`, the run page's own bar) | — | native `<progress max value>`, 4px, full width, an `aria-label` saying what is done; no width rule per value | `▓▓▓░░ 3 of 5` |
157157 | **meter** `meter()` | `meter` | `data-tone=attn` when the share is a limit reached | native `<meter min=0 max=100>`, a whole percentage, an `aria-label` saying what it is a share of; no width rule per value, so no inline style | `▓▓▓░░ 62%` |
158158 | **kbd** `kbd()` | `kbd` | — | — | a reverse-video key name |
159 | **card** `card()` / `listCard()` | `.card` (`.card-head`, `.card-body`, `.card-foot`) | `data-tone=success\|danger\|merged\|attn` | — | a box with a title bar |
159 | **card** `card()` / `listCard()` | `.card` (`.card-head`, `.card-links`, `.card-body`, `.card-foot`) | `data-tone=success\|danger\|merged\|attn` | — | a box with a title bar |
160160 | **list row** `row()` | `.row` (`.row-lead`, `.row-main`, `.row-title`, `.row-meta`, `.row-trail`, `.row-actions`; settings: `.setting-list`, `.row-fields`) | `data-state=open\|closed\|merged\|draft` — colours the **lead glyph**, so a list filtered by state carries no badge repeating the filter | hover, focus, `data-nav`; `fold` makes the row a `<details>`'s `<summary>`, its lead the caret turned when open (/map's lanes); `.row-meta` is one honest line: nowrap, ellipsis, `title` — except in a `.setting-list`, where it wraps | one line: glyph, title, right-aligned meta |
161161 | **edit row** | `.edit-row` | — | one line on a desktop, stacked below 720px | a form line |
162162 | **colour field** | `.color-field` | — | native `<input type=color>` plus its hex; works with no script | a hex entry |
src/boards/panel_data.ts added +141
@@ -0,0 +1,141 @@
1 /**
2 * A panel's data over a window, at a step: the one door a board page, home's
3 * pinned panels and `GET /api/v1/boards/:slug/panels/:id` read a panel
4 * through. A panel grouped by time is bucketed in SQL to the step — asked
5 * for, or else the finest that keeps the chart near `TARGET_POINTS` wide —
6 * and never cut short: an answer too large for its step is asked again at
7 * the next coarser one. Plain data in, plain data out; the caller has
8 * checked the asker is an admin.
9 */
10 import type { Database } from "bun:sqlite";
11 import { usesSeries, type Chart } from "./chart.ts";
12 import { BUCKETS, MAX_STEP, oldestTime, runQuery, type Answer, type Bucket, type Row } from "./query.ts";
13 import type { Board, Panel } from "./store.ts";
14
15 export const RANGE_MS = { "1h": 3_600_000, "24h": 86_400_000, "7d": 604_800_000, "30d": 2_592_000_000 } as const;
16 /** The steps a panel picks from on its own, in seconds: 1 min to a week. */
17 export const STEPS = [60, 120, 300, 600, 900, 1800, 3600, 7200, 10_800, 21_600, 43_200, 86_400, 604_800] as const;
18 /** About one point per two pixels of a chart's plot: more says nothing a reader can see. */
19 export const TARGET_POINTS = 360;
20 const UNIT = { s: 1, m: 60, h: 3600, d: 86_400 } as const;
21
22 /** The window asked for: ISO times and a step in seconds, each absent when not asked. */
23 export type Window = { from?: string; to?: string; step?: number };
24 export type PanelData =
25 | { ok: true; rows: Row[]; from: string | null; to: string | null; step: number | null }
26 | { ok: false; error: string };
27
28 /**
29 * Run `query` for `chart`: a chart that draws one value per group is asked
30 * without the query's series, so its groups are not split. The run is
31 * scoped to every repo — only an admin reaches a board.
32 */
33 export function runFor(db: Database, chart: Chart, query: Record<string, unknown>): Answer {
34 if (usesSeries(chart) || query.series === undefined) return runQuery(db, query, { repoIds: "all" });
35 const { series: _, ...rest } = query;
36 return runQuery(db, rest, { repoIds: "all" });
37 }
38
39 /** `300`, `15s`, `5m`, `1h` or `1d` as seconds; undefined for anything else or out of 1 s to a year. */
40 export function parseStep(s: string): number | undefined {
41 const m = /^(\d{1,8})([smhd]?)$/.exec(s);
42 if (!m) return undefined;
43 const n = Number(m[1]) * UNIT[(m[2] || "s") as keyof typeof UNIT];
44 return n >= 1 && n <= MAX_STEP ? n : undefined;
45 }
46
47 /** `from`, `to` and `step` from a URL, judged: the window, or why it is not one. */
48 export function parseWindow(search: URLSearchParams): { ok: true; window: Window } | { ok: false; error: string } {
49 const window: Window = {};
50 for (const key of ["from", "to"] as const) {
51 const v = search.get(key);
52 if (v === null || v === "") continue;
53 const t = v.length <= 64 ? Date.parse(v) : Number.NaN;
54 if (Number.isNaN(t)) return { ok: false, error: `${key} must be a date, like 2026-10-20T00:00:00Z` };
55 window[key] = new Date(t).toISOString();
56 }
57 const step = search.get("step");
58 if (step !== null && step !== "") {
59 const n = parseStep(step);
60 if (n === undefined) return { ok: false, error: "step must be seconds or a count of s, m, h or d, like 300 or 5m, up to a year" };
61 window.step = n;
62 }
63 if (window.from !== undefined && window.to !== undefined && window.from >= window.to) {
64 return { ok: false, error: "from must be before to" };
65 }
66 return { ok: true, window };
67 }
68
69 /** The finest step from `STEPS`, no finer than `natural`, that keeps `spanMs` within `TARGET_POINTS` buckets. */
70 export function stepFor(spanMs: number, natural: number): number {
71 const fit = STEPS.find((s) => s >= natural && (spanMs / 1000) / s <= TARGET_POINTS);
72 return fit ?? Math.max(natural, STEPS[STEPS.length - 1]!);
73 }
74
75 /** The panel's bucket width in seconds, or undefined when it is not grouped by time. */
76 function bucketOf(query: Record<string, unknown>): number | undefined {
77 const g = query.groupBy;
78 if (typeof g !== "object" || g === null) return undefined;
79 const b = (g as Record<string, unknown>).bucket;
80 return typeof b === "string" && Object.hasOwn(BUCKETS, b) ? BUCKETS[b as Bucket] : undefined;
81 }
82
83 /**
84 * `panel`'s rows over `window` at `now`. The window's `from` and `to` stand
85 * in for the query's, and the board's range is the start when neither
86 * names one; with no start at all, the span runs from the oldest row. A
87 * step asked for is kept, or refused when it would be cut short; a step not
88 * asked for fits the span and grows until the answer fits.
89 */
90 export function panelData(
91 db: Database,
92 board: Board,
93 panel: Panel,
94 window: Window = {},
95 now = Date.now(),
96 run: typeof runFor = runFor,
97 ): PanelData {
98 const query: Record<string, unknown> = { ...panel.query };
99 if (window.from !== undefined) query.from = window.from;
100 if (window.to !== undefined) query.to = window.to;
101 if (query.from === undefined && board.range !== "all") query.from = new Date(now - RANGE_MS[board.range]).toISOString();
102 const from = typeof query.from === "string" ? query.from : null;
103 const to = typeof query.to === "string" ? query.to : null;
104 const natural = bucketOf(query);
105 if (natural === undefined) {
106 const answer = run(db, panel.chart, query);
107 return answer.ok ? { ok: true, rows: answer.rows, from, to, step: null } : { ok: false, error: answer.error };
108 }
109 if (window.step !== undefined) {
110 const answer = run(db, panel.chart, { ...query, step: window.step });
111 return answer.ok ? { ok: true, rows: answer.rows, from, to, step: window.step } : { ok: false, error: answer.error };
112 }
113 // With no start (a board whose range is `all`), the span runs from the oldest row the panel reads.
114 const oldest = from === null ? oldestTime(db, query, { repoIds: "all" }) : from;
sentinelcrew outdated

Medium: with range: "all" and no from, start is NaN, so the step stays at the natural bucket. A minute panel then renders up to 20000 points on the board page and on home, with a full-scan retry for each coarser step. Take the span from the source's oldest time so stepFor caps it near TARGET_POINTS.
115 const start = oldest === null ? Number.NaN : Date.parse(oldest);
116 const end = to === null ? now : Date.parse(to);
117 let step = Number.isNaN(start) || Number.isNaN(end) ? natural : stepFor(end - start, natural);
118 for (;;) {
119 const answer = run(db, panel.chart, { ...query, step });
120 if (answer.ok) return { ok: true, rows: answer.rows, from, to, step };
121 const coarser = STEPS.find((s) => s > step);
122 if (answer.tooMany !== true || coarser === undefined) return { ok: false, error: answer.error };
123 step = coarser;
124 }
125 }
126
127 /** A text cell a spreadsheet will not read as a formula, quoted when it holds a comma, quote or newline. */
128 function csvCell(v: string | number | null | undefined): string {
129 if (v === null || v === undefined) return "";
130 if (typeof v === "number") return String(v);
131 const safe = /^[=+\-@\t\r]/.test(v) ? `'${v}` : v;
132 return /[",\r\n]/.test(safe) ? `"${safe.replace(/"/g, '""')}"` : safe;
133 }
134
135 /** Rows as CSV: `group,value`, or `group,series,value` when the rows are split by a series. */
136 export function rowsCsv(rows: readonly Row[]): string {
137 const split = rows.some((r) => r.series !== undefined);
138 const head = split ? ["group", "series", "value"] : ["group", "value"];
139 const lines = rows.map((r) => (split ? [r.group, r.series, r.value] : [r.group, r.value]).map(csvCell).join(","));
140 return [head.join(","), ...lines].join("\r\n") + "\r\n";
141 }
src/boards/query.ts +68 −19
@@ -8,18 +8,22 @@ import { SOURCES, type ColumnType } from "./sources.ts";
88
99 /** `starts` is a prefix match on a text or time column (`target` starts `cpu:`); the rest compare. */
1010 const OPS = { eq: "=", ne: "<>", lt: "<", lte: "<=", gt: ">", gte: ">=", starts: null } as const;
11 const BUCKETS = {
12 minute: "%Y-%m-%dT%H:%M:00Z",
13 hour: "%Y-%m-%dT%H:00:00Z",
14 day: "%Y-%m-%dT00:00:00Z",
15 } as const;
11 /** A bucket's width in seconds; a query's `step` replaces it. */
12 export const BUCKETS = { minute: 60, hour: 3600, day: 86_400 } as const;
1613 const AGGREGATES = ["count", "sum", "avg", "max"] as const;
1714 const MAX_ROWS = 1000;
15 /**
16 * A query grouped by time is never cut short: past this many rows it is
17 * refused (`tooMany`), so the caller asks again at a coarser step.
18 */
19 export const MAX_POINTS = 20_000;
20 /** The widest step, in seconds: a year. */
21 export const MAX_STEP = 31_536_000;
1822 const MAX_FILTERS = 16;
1923 const MAX_REPOS = 1000;
2024
2125 type Op = keyof typeof OPS;
22 type Bucket = keyof typeof BUCKETS;
26 export type Bucket = keyof typeof BUCKETS;
2327 type Aggregate = (typeof AGGREGATES)[number];
2428
2529 export type Query = {
@@ -28,6 +32,8 @@ export type Query = {
2832 to?: string;
2933 filter?: { column: string; op: string; value: string | number }[];
3034 groupBy?: string | { bucket: string };
35 /** Seconds per time bucket, in place of the bucket's own width; only with a `{bucket}` group. */
36 step?: number;
3137 aggregate: string;
3238 column?: string;
3339 /** A second grouping column: each group is split into one row per value (stacked area, heat strip). */
@@ -38,7 +44,8 @@ export type Query = {
3844 export type Scope = { repoIds: readonly string[] | "all" };
3945
4046 export type Row = { group: string | number | null; value: number | null; series?: string | number | null };
41 export type Refusal = { ok: false; error: string };
47 /** `tooMany`: a time-grouped answer has more rows than `MAX_POINTS`; a coarser step may fit. */
48 export type Refusal = { ok: false; error: string; tooMany?: true };
4249 export type Answer = { ok: true; rows: Row[] } | Refusal;
4350
4451 const refuse = (error: string): Refusal => ({ ok: false, error });
@@ -50,14 +57,18 @@ function valueFits(type: ColumnType, v: unknown): boolean {
5057 return typeof v === "string" && v.length <= 256;
5158 }
5259
53 /** Validate then run `q` against `db`, over the rows `scope` allows. Never throws on bad input. */
54 export function runQuery(db: Database, q: unknown, scope: Scope): Answer {
60 type Source = (typeof SOURCES)[string];
61 /** The rows `q` reads before grouping: its source, and the WHERE its scope, window and filters make. */
62 type Rows = { query: Record<string, unknown>; src: Source; from: string; where: string[]; params: (string | number)[] };
63
64 /** Validate `q`'s source, scope, window and filters; the rows they name, nothing (`empty`) or a refusal. */
65 function rowsOf(q: unknown, scope: Scope): Rows | Refusal | "empty" {
5566 const ids = scope?.repoIds;
5667 if (ids !== "all") {
5768 if (!Array.isArray(ids) || ids.length > MAX_REPOS || ids.some((i) => typeof i !== "string")) {
5869 return refuse("scope must list repos or be all");
5970 }
60 if (ids.length === 0) return { ok: true, rows: [] };
71 if (ids.length === 0) return "empty";
6172 }
6273 if (typeof q !== "object" || q === null || Array.isArray(q)) return refuse("query must be an object");
6374 const query = q as Record<string, unknown>;
@@ -69,7 +80,7 @@ export function runQuery(db: Database, q: unknown, scope: Scope): Answer {
6980 const cols = src.columns;
7081 const where: string[] = src.where ? [src.where] : [];
7182 const params: (string | number)[] = [];
72 if (ids !== "all" && src.owner === "admin") return { ok: true, rows: [] };
83 if (ids !== "all" && src.owner === "admin") return "empty";
7384 if (ids !== "all") {
7485 const marks = ids.map(() => "?").join(",");
7586 where.push(
@@ -102,6 +113,32 @@ export function runQuery(db: Database, q: unknown, scope: Scope): Answer {
102113 // instr(), not LIKE: a % or _ in the value is a character, never a wildcard.
103114 where.push(sqlOp === null ? `instr(${ident(column)}, ?) = 1` : `${ident(column)} ${sqlOp} ?`);
104115 }
116 const from = src.select === undefined ? ident(src.table) : `(${src.select}) AS ${ident(src.table)}`;
117 return { query, src, from, where, params };
118 }
119
120 const whereSql = (where: readonly string[]) => (where.length ? ` WHERE ${where.join(" AND ")}` : "");
121
122 /**
123 * The oldest time among the rows `q` reads, as stored, or null when there
124 * are none (or `q` is refused): what a window with no start spans from.
125 */
126 export function oldestTime(db: Database, q: unknown, scope: Scope): string | null {
127 const r = rowsOf(q, scope);
128 if (r === "empty" || !("src" in r)) return null;
129 const row = db.query(`SELECT MIN(${ident(r.src.time)}) AS "at" FROM ${r.from}${whereSql(r.where)}`).get(...r.params) as
130 | { at: string | null }
131 | null;
132 return row?.at ?? null;
133 }
134
135 /** Validate then run `q` against `db`, over the rows `scope` allows. Never throws on bad input. */
136 export function runQuery(db: Database, q: unknown, scope: Scope): Answer {
137 const r = rowsOf(q, scope);
138 if (r === "empty") return { ok: true, rows: [] };
139 if (!("src" in r)) return r;
140 const { query, src, from, where, params } = r;
141 const cols = src.columns;
105142
106143 const agg = query.aggregate;
107144 if (typeof agg !== "string" || !(AGGREGATES as readonly string[]).includes(agg)) {
@@ -116,7 +153,7 @@ export function runQuery(db: Database, q: unknown, scope: Scope): Answer {
116153 }
117154
118155 let groupSql = "NULL";
119 let groupParam: string | undefined;
156 let step: number | undefined;
120157 const g = query.groupBy;
121158 if (typeof g === "string") {
122159 if (!has(cols, g)) return refuse(`unknown column: ${JSON.stringify(g)}`);
@@ -124,8 +161,17 @@ export function runQuery(db: Database, q: unknown, scope: Scope): Answer {
124161 } else if (g !== undefined) {
125162 const b = typeof g === "object" && g !== null ? (g as Record<string, unknown>).bucket : undefined;
126163 if (typeof b !== "string" || !has(BUCKETS, b)) return refuse(`unknown bucket: ${JSON.stringify(b)}`);
127 groupSql = `strftime(?, ${ident(src.time)})`;
128 groupParam = BUCKETS[b as Bucket];
164 step = BUCKETS[b as Bucket];
165 // The bucket's start: its time in whole seconds, floored to the step, in UTC.
166 groupSql = `strftime('%Y-%m-%dT%H:%M:%SZ', CAST(strftime('%s', ${ident(src.time)}) AS INTEGER) / ? * ?, 'unixepoch')`;
167 }
168 if (query.step !== undefined) {
169 const n = query.step;
170 if (step === undefined) return refuse("step needs a group by a time bucket");
171 if (typeof n !== "number" || !Number.isInteger(n) || n < 1 || n > MAX_STEP) {
172 return refuse(`step must be a whole number of seconds, 1 to ${MAX_STEP}`);
173 }
174 step = n;
129175 }
130176
131177 let seriesSql = "";
@@ -136,12 +182,15 @@ export function runQuery(db: Database, q: unknown, scope: Scope): Answer {
136182 }
137183 const grouped = [groupSql === "NULL" ? "" : "1", seriesSql === "" ? "" : "3"].filter(Boolean).join(", ");
138184
139 const from = src.select === undefined ? ident(src.table) : `(${src.select}) AS ${ident(src.table)}`;
140185 const sql =
141186 `SELECT ${groupSql} AS "group", ${valueSql} AS "value"${seriesSql} FROM ${from}` +
142 (where.length ? ` WHERE ${where.join(" AND ")}` : "") +
187 whereSql(where) +
143188 (grouped === "" ? "" : ` GROUP BY ${grouped} ORDER BY ${grouped}`) +
144 ` LIMIT ${MAX_ROWS}`;
145 const bound = groupParam === undefined ? params : [groupParam, ...params];
146 return { ok: true, rows: db.query(sql).all(...bound) as Row[] };
189 ` LIMIT ${step === undefined ? MAX_ROWS : MAX_POINTS + 1}`;
190 const bound = step === undefined ? params : [step, step, ...params];
191 const rows = db.query(sql).all(...bound) as Row[];
192 if (step !== undefined && rows.length > MAX_POINTS) {
193 return { ok: false, error: `more than ${MAX_POINTS} points: ask for a coarser step or a shorter range`, tooMany: true };
194 }
195 return { ok: true, rows };
147196 }
src/http/api.ts +31
@@ -19,6 +19,7 @@ import { parentForgeFor } from "../core/parent_releases.ts";
1919 import { readFlow } from "../flow/read.ts";
2020 import { listPinned as listPinnedPanels, movePinned, pinPanel, unpinPanel, type PinError } from "../actions/board-pins.ts";
2121 import { deleteBoard, listAllBoards, showBoard, writeBoard, type BoardError } from "../actions/boards.ts";
22 import { panelData, parseWindow, rowsCsv } from "../boards/panel_data.ts";
2223 import { runQuery } from "../boards/query.ts";
2324 import { recordDrill } from "../core/domain_drills.ts";
2425 import { drillVerdict } from "../kernel/domain_drill.ts";
@@ -1549,6 +1550,36 @@ export function apiRoutes(router: Router<Handler>): void {
15491550 return shown.ok ? json({ slug: ctx.params.slug, ...shown.value }) : notFound();
15501551 });
15511552
1553 // One panel's data (`boards/panel_data.ts`): `:id` is its place on the
1554 // board from 0; `from`, `to` and `step` (300, 5m, 1h) the window;
sentinelcrew outdated

The door is correct here, but nothing on the board page or in MCP leads to it. See the review body: add a csv link on each panel card, and a boards/board_panel MCP door that goes through panelData.
1555 // `format=csv` the same rows as a table file.
1556 router.add("GET", "/api/v1/boards/:slug/panels/:id", async (ctx) => {
1557 const denied = boardsAdmin(ctx);
1558 if (denied) return denied;
1559 const shown = await showBoard(ctx.app, ctx.auth.user, ctx.params.slug!);
1560 const id = ctx.params.id!;
1561 const panel = shown.ok && /^\d{1,2}$/.test(id) ? shown.value.panels[Number(id)] : undefined;
1562 if (!shown.ok || !panel) return notFound();
1563 const format = ctx.url.searchParams.get("format") ?? "json";
1564 if (format !== "json" && format !== "csv") return apiError("invalid_format", 422, "format must be json or csv");
1565 const window = parseWindow(ctx.url.searchParams);
1566 if (!window.ok) return apiError("invalid_window", 422, window.error);
1567 const data = panelData(ctx.app.db, shown.value, panel, window.window);
1568 if (!data.ok) return apiError("bad_query", 400, data.error);
1569 const headers = { "cache-control": "private, no-store" };
1570 if (format === "csv") {
1571 return new Response(rowsCsv(data.rows), {
1572 headers: {
1573 ...headers,
1574 "content-type": "text/csv; charset=utf-8",
1575 "content-disposition": `attachment; filename="${ctx.params.slug}-${id}.csv"`,
1576 },
1577 });
1578 }
1579 const { ok: _, ...out } = data;
1580 return json({ board: ctx.params.slug, panel: Number(id), title: panel.title, chart: panel.chart, ...out }, 200, headers);
1581 });
1582
15521583 router.add("PUT", "/api/v1/boards/:slug", async (ctx) => {
15531584 const denied = boardsAdmin(ctx) ?? requireAccount(ctx, { adminToken: true });
15541585 if (denied) return denied;
src/http/mcp.ts +25 −3
@@ -200,6 +200,7 @@ import { manifestFor } from "../actions/here.ts";
200200 import { clientBucket } from "./peer.ts";
201201 import { listPinned as listPinnedPanels, movePinned, pinPanel, unpinPanel } from "../actions/board-pins.ts";
202202 import { deleteBoard, listAllBoards, showBoard, writeBoard, type BoardError } from "../actions/boards.ts";
203 import { panelData, parseWindow } from "../boards/panel_data.ts";
203204 import { fileReport } from "../actions/reports.ts";
204205 import { BODY_MAX, REQUEST_MAX, TITLE_MAX } from "../core/reports.ts";
205206 import { placementWords } from "../checks/placement.ts";
@@ -3179,10 +3180,16 @@ const TOOLS: readonly Tool[] = [
31793180 {
31803181 name: "boards",
31813182 description:
3182 "The forge's boards, for a forge admin who is a person (an agent is refused): each board's slug, title, range, refresh and panels. The same as /boards and GET /api/v1/boards. With `slug`, that one board.",
3183 "The forge's boards, for a forge admin who is a person (an agent is refused): each board's slug, title, range, refresh and panels. The same as /boards and GET /api/v1/boards. With `slug`, that one board. With `slug` and `panel` (its place from 0), that panel's data: {from, to, step, rows}, bucketed to `step` (300, 5m, 1h, 1d) or one the panel picks, never cut short; `from` and `to` (dates) stand in for the panel's. The same as GET /api/v1/boards/:slug/panels/:id.",
31833184 inputSchema: {
31843185 type: "object",
3185 properties: { slug: { type: "string", maxLength: 40, description: "One board, by slug." } },
3186 properties: {
3187 slug: { type: "string", maxLength: 40, description: "One board, by slug." },
3188 panel: { type: "integer", minimum: 0, maximum: 23, description: "One panel's data, by its place on the board from 0." },
3189 from: { type: "string", maxLength: 64, description: "With panel: the window's start, a date." },
3190 to: { type: "string", maxLength: 64, description: "With panel: the window's end, a date." },
3191 step: { type: "string", maxLength: 16, description: "With panel: seconds per bucket, or a count of s, m, h or d." },
3192 },
31863193 },
31873194 writes: false,
31883195 async run(ctx, user, args) {
@@ -3194,7 +3201,22 @@ const TOOLS: readonly Tool[] = [
31943201 : boardFailed(listed.error);
31953202 }
31963203 const shown = await showBoard(ctx.app, user, str(args["slug"]));
3197 return shown.ok ? good({ slug: str(args["slug"]), ...shown.value }) : boardFailed(shown.error);
3204 if (!shown.ok) return boardFailed(shown.error);
3205 if (args["panel"] === undefined) return good({ slug: str(args["slug"]), ...shown.value });
3206 const index = args["panel"];
3207 const panel = typeof index === "number" && Number.isInteger(index) ? shown.value.panels[index] : undefined;
3208 if (!panel) return bad("not_found", "that board has no panel there");
3209 const asked = new URLSearchParams();
3210 for (const k of ["from", "to", "step"]) {
3211 const v = args[k];
3212 if (v !== undefined) asked.set(k, typeof v === "number" ? String(v) : str(v));
3213 }
3214 const window = parseWindow(asked);
3215 if (!window.ok) return bad("invalid_window", window.error);
3216 const data = panelData(ctx.app.db, shown.value, panel, window.window);
3217 if (!data.ok) return bad("bad_query", data.error);
3218 const { ok: _, ...out } = data;
3219 return good({ board: str(args["slug"]), panel: index, title: panel.title, chart: panel.chart, ...out });
31983220 },
31993221 },
32003222 {
src/tui/screens/boards.ts +29 −5
@@ -4,10 +4,13 @@
44 * `listAllBoards` and `showBoard` the page and the API do, so the gate is the
55 * action's (an agent or anyone else is told nothing); the palette offers the
66 * door only to that person. Reading only, but for pins: a chart is not drawn, a panel is said
7 * as its title, chart, size and the query that feeds it. `[`/`]` mark a panel,
7 * as its title, chart, size and the query that feeds it, then what `panelData` reads for it
8 * (the page's and the API's door): a time panel's last bucket and its step, else its row count. `[`/`]` mark a panel,
89 * `p` pins or unpins it to home and `<`/`>` move a pinned one (`actions/board-pins.ts`).
910 */
1011
12 import { formatValue } from "../../boards/chart.ts";
13 import { panelData, type PanelData } from "../../boards/panel_data.ts";
1114 import type { Board } from "../../boards/store.ts";
1215 import { listPinned, movePinned, pinPanel, unpinPanel } from "../../actions/board-pins.ts";
1316 import { listAllBoards, showBoard } from "../../actions/boards.ts";
@@ -41,8 +44,24 @@ export async function boardRows(session: Session): Promise<BoardRow[]> {
4144 }));
4245 }
4346
44 /** A board, panel by panel, as lines of text; the panel at `at` is marked, and a pinned one says so. */
45 export function boardLines(board: Board, at = -1, pinned: readonly string[] = []): string[] {
47 /** A step in seconds as the API takes it: `5m`, `1h`, `1d`, or seconds. */
48 export const stepWords = (s: number): string =>
49 s % 86_400 === 0 ? `${s / 86_400}d` : s % 3600 === 0 ? `${s / 3600}h` : s % 60 === 0 ? `${s / 60}m` : `${s}s`;
50
51 /** What a panel's data says in one line: its last bucket and step, its row count, or why it was refused. */
52 export function dataLine(d: PanelData): string {
53 if (!d.ok) return `refused: ${plain(d.error)}`;
54 if (d.step === null) return plural(d.rows.length, "row");
55 const last = d.rows.at(-1);
56 const at = `step ${stepWords(d.step)} · ${plural(d.rows.length, "point")}`;
57 return last === undefined ? `no points · ${at}` : `last ${plain(String(last.group ?? "all"))}: ${formatValue(last.value ?? 0)} · ${at}`;
58 }
59
60 /**
61 * A board, panel by panel, as lines of text; the panel at `at` is marked, a
62 * pinned one says so, and each panel with its `data` read says what it holds.
63 */
64 export function boardLines(board: Board, at = -1, pinned: readonly string[] = [], data: readonly PanelData[] = []): string[] {
4665 const lines = [
4766 plain(board.title),
4867 `range ${board.range} · refresh ${board.refresh} · ${plural(board.panels.length, "panel")}`,
@@ -52,6 +71,8 @@ export function boardLines(board: Board, at = -1, pinned: readonly string[] = []
5271 .map(([k, v]) => `${plain(k)}=${plain(typeof v === "string" ? v : JSON.stringify(v))}`)
5372 .join(" ");
5473 lines.push("", `${i === at ? "▸ " : ""}${plain(p.title)}${pinned.includes(p.title) ? " [pinned to home]" : ""}`, ` ${p.chart} · ${p.w}×${p.h}`, ` ${query === "" ? "no query" : query}`);
74 const d = data[i];
75 if (d !== undefined) lines.push(` ${dataLine(d)}`);
5576 }
5677 return lines;
5778 }
@@ -60,11 +81,14 @@ function boardScreen(ctx: Ctx, f: Session, slug: string, board: Board): Screen {
6081 const scroll = new Scroll();
6182 let at = board.panels.length > 0 ? 0 : -1;
6283 let pinned: string[] = [];
84 let data: PanelData[] = [];
6385 let lines = boardLines(board, at, pinned);
6486 const read = async (): Promise<void> => {
6587 const listed = await listPinned(f.app, f.user);
6688 pinned = listed.ok ? listed.value.filter((p) => p.slug === slug).map((p) => p.title) : [];
67 lines = boardLines(board, at, pinned);
89 // Only an admin person reaches this screen (`showBoard` let them in), as `panelData` asks.
90 data = board.panels.map((p) => panelData(f.app.db, board, p));
91 lines = boardLines(board, at, pinned, data);
6892 ctx.invalidate();
6993 };
7094 /** Pin the marked panel, or unpin it when it is; `by` moves a pinned one in the home order. */
@@ -98,7 +122,7 @@ function boardScreen(ctx: Ctx, f: Session, slug: string, board: Board): Screen {
98122 if (scroll.key(e, lines.length)) return true;
99123 if (e.key === "[" || e.key === "]") {
100124 if (board.panels.length > 0) at = (at + (e.key === "]" ? 1 : board.panels.length - 1)) % board.panels.length;
101 lines = boardLines(board, at, pinned);
125 lines = boardLines(board, at, pinned, data);
102126 return true;
103127 }
104128 if (e.key === "p") return act(), true;
src/web/pages/boards-view.ts +11 −10
@@ -21,37 +21,38 @@ import { fragment, page } from "../layout.ts";
2121 import { btn, card, empty } from "../ui.ts";
2222 import { flowPanel, flowPanelCss } from "./flow-panel.ts";
2323 import { isPinned, listPins } from "../../boards/pins.ts";
24 import { cellsOf, chartBox, paramsOf, queryFields, readParams, runFor } from "./boards.ts";
24 import { panelData, runFor } from "../../boards/panel_data.ts";
25 import { cellsOf, chartBox, paramsOf, queryFields, readParams } from "./boards.ts";
2526
26 const RANGE_MS = { "1h": 3_600_000, "24h": 86_400_000, "7d": 604_800_000, "30d": 2_592_000_000 } as const;
2727 /** The events the panels' queries read (runs, pulls, checks, issues, usage): any of them may change a chart. */
2828 const READS = "run.queued,run.started,run.finished,pull.*,check.*,issue.*,usage.*";
2929
3030 const option = (value: string, selected: boolean) =>
3131 html`<option value="${value}" ${selected ? raw("selected") : null}>${value}</option>`;
3232
33 /** The panel's query as asked, with the board's range as its start when it names none. */
34 function ranged(query: Record<string, unknown>, range: Board["range"]): Record<string, unknown> {
35 if (range === "all" || query.from !== undefined) return query;
36 return { ...query, from: new Date(Date.now() - RANGE_MS[range]).toISOString() };
37 }
33 /** Where a panel lives: its board's slug and its place on it, which name its data door. */
34 export type PanelAt = { slug: string; index: number };
3835
3936 export function drawPanel(
4037 app: App,
4138 board: Board,
4239 panel: Panel,
40 at: PanelAt,
sentinelcrew outdated

Every card renders through panelData, so this is the place to also emit the panel's data link (/api/v1/boards/${slug}/panels/${i}?format=csv) beside explore. drawPanel needs the slug and the index passed in.
4341 aside: Html | null = null,
4442 run: typeof runFor = runFor,
4543 ): Html {
46 const answer = run(app.db, panel.chart, ranged(panel.query, board.range));
44 const answer = panelData(app.db, board, panel, {}, Date.now(), run);
4745 const q = paramsOf(panel.query, panel.chart);
4846 const body = !answer.ok
4947 ? html`<p class="muted small">${answer.error}</p>`
5048 : chartBox(panel.chart, cellsOf(answer.rows), panel.title);
5149 const explore = new URLSearchParams(Object.entries(q).filter(([k, v]) => v !== "" && k !== "rows"));
50 const data = `/api/v1/boards/${at.slug}/panels/${String(at.index)}`;
5251 return html`<div class="card bp-w${String(panel.w)} bp-h${String(panel.h)}">
5352 <div class="card-head"><h2>${panel.title}</h2>
54 <a class="muted small" href="/boards/data?${explore.toString()}">explore</a></div>
53 <span class="card-links muted small"><a href="/boards/data?${explore.toString()}">explore</a>
sentinelcrew outdated

High (DESIGN.md density): three 13px inline links with no coarse-pointer padding, so each is about 17px tall on a phone. Put them under one class with min-height:44px and padding under @media (pointer: coarse) (see .topbar-tools > a.small in style/base.ts), or move json/csv to .card-foot.
54 <a href="${data}" aria-label="${panel.title}: data as JSON">json</a>
55 <a href="${data}?format=csv" download="${at.slug}-${String(at.index)}.csv" aria-label="${panel.title}: data as CSV">csv</a></span></div>
5556 <div class="card-body">${body}</div>
5657 ${aside}
5758 </div>`;
@@ -68,7 +69,7 @@ function grid(app: App, slug: string, board: Board): Html {
6869 >
6970 ${board.panels.length === 0
7071 ? html`<p class="muted">No panels yet. Add one below.</p>`
71 : board.panels.map((p) => drawPanel(app, board, p))}
72 : board.panels.map((p, index) => drawPanel(app, board, p, { slug, index }))}
7273 </section>`;
7374 }
7475
src/web/pages/boards.ts +4 −15
@@ -7,9 +7,9 @@
77 * narrowed to that point's window and listed by the source's first column.
88 * Anyone else gets the not-found page.
99 */
10 import type { Database } from "bun:sqlite";
1110 import { CHART_NAMES, CHARTS, drawChart, usesSeries, type Cell, type Chart } from "../../boards/chart.ts";
12 import { runQuery, type Answer, type Query, type Row } from "../../boards/query.ts";
11 import { runFor } from "../../boards/panel_data.ts";
12 import { runQuery, type Query, type Row } from "../../boards/query.ts";
1313 import { gaps, SYSTEM } from "../../boards/sampler.ts";
1414 import { SOURCES } from "../../boards/sources.ts";
1515 import { listAllBoards } from "../../actions/boards.ts";
@@ -73,17 +73,6 @@ export function toQuery(p: Params): Query {
7373 return q;
7474 }
7575
76 /**
77 * Run `query` for `chart`: a chart that draws one value per group is asked
78 * without the query's series, so its groups are not split. The run is
79 * scoped to every repo — only an admin reaches a board.
80 */
81 export function runFor(db: Database, chart: Chart, query: Record<string, unknown>): Answer {
82 if (usesSeries(chart) || query.series === undefined) return runQuery(db, query, { repoIds: "all" });
83 const { series: _, ...rest } = query;
84 return runQuery(db, rest, { repoIds: "all" });
85 }
86
8776 /** Rows as the cells a chart draws: a null group or series reads "all"/"none". */
8877 export const cellsOf = (rows: readonly Row[], hrefs: readonly (string | null)[] = []): Cell[] =>
8978 rows.map((r, i) => ({
@@ -131,8 +120,8 @@ function behind(p: Params, row: Row): string | null {
131120
132121 /**
133122 * Ready-made queries over the machine's cores and thermometers, from `now`:
134 * the last hour per CPU, and twelve hours of hourly heat — 12 × the CPUs
135 * stays under the 1000-row cap up to 83 CPUs.
123 * the last hour per CPU (one row each, under the 1000-row cap of a column
124 * group), and twelve hours of hourly heat.
136125 */
137126 export function machineLinks(now: number): { label: string; href: string }[] {
138127 const ago = (h: number) => new Date(Math.floor(now / 60_000) * 60_000 - h * 3_600_000).toISOString();
src/web/studio/pinned.ts +3 −3
@@ -11,7 +11,7 @@ import type { User } from "../../core/users.ts";
1111 import { html, type Html } from "../html.ts";
1212 import { btn } from "../ui.ts";
1313 import { cachedPanel } from "./home_cache.ts";
14 import { runFor } from "../pages/boards.ts";
14 import { runFor } from "../../boards/panel_data.ts";
1515 import { drawPanel } from "../pages/boards-view.ts";
1616
1717 function controls(p: Pin): Html {
@@ -27,10 +27,10 @@ function controls(p: Pin): Html {
2727
2828 /** Admin only: the caller has checked. The answer is kept per panel definition (cache 5). */
2929 const cached = (p: Pin & { board: Board; panel: Panel }): typeof runFor => (db, chart, query) =>
30 cachedPanel(db, [p.slug, p.board.range, p.panel.title, chart, p.panel.query], () => runFor(db, chart, query));
30 cachedPanel(db, [p.slug, p.board.range, p.panel.title, chart, p.panel.query, query.step], () => runFor(db, chart, query));
3131
3232 export async function pinnedPanels(app: App, user: User): Promise<Html | null> {
3333 const listed = await listPinned(app, user);
3434 if (!listed.ok || listed.value.length === 0) return null;
35 return html`<section class="bgrid" aria-label="Pinned panels">${listed.value.map((p) => drawPanel(app, p.board, p.panel, controls(p), cached(p)))}</section>`;
35 return html`<section class="bgrid" aria-label="Pinned panels">${listed.value.map((p) => drawPanel(app, p.board, p.panel, { slug: p.slug, index: p.board.panels.indexOf(p.panel) }, controls(p), cached(p)))}</section>`;
3636 }
src/web/style/components.ts +4
@@ -307,6 +307,10 @@ a.sha:hover { color: var(--accent); }
307307 padding: 10px 14px; border-bottom: 1px solid var(--border); background: var(--muted);
308308 }
309309 .card-head h2, .card-head h3 { margin: 0; }
310 /* A card head's links: spaced apart, and each a 44px target on a thumb. */
311 .card-links { display: inline-flex; align-items: center; gap: 10px; }
312 .card-links > a { color: inherit; }
313 @media (pointer: coarse) { .card-links { gap: 0; } .card-links > a { display: inline-flex; align-items: center; min-height: 44px; padding: 0 8px; } }
310314 .card-body { padding: 14px; }
311315 .card-foot {
312316 display: flex; align-items: center; gap: 8px; flex-wrap: wrap;
test/boards-panel-data.test.ts added +234
@@ -0,0 +1,234 @@
1 /**
2 * A panel's data as JSON and CSV (#3691): bucketed in SQL to the step asked
3 * for, or one the panel picks, and never cut short — a day of per-minute
4 * points reaches now, where the old 1000-row cap stopped hours early.
5 */
6 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { writeBoard } from "../src/actions/boards.ts";
11 import { createApp } from "../src/app.ts";
12 import { panelData, parseStep, parseWindow, rowsCsv, stepFor, TARGET_POINTS, type runFor } from "../src/boards/panel_data.ts";
13 import { MAX_POINTS, oldestTime, runQuery } from "../src/boards/query.ts";
14 import type { Board, Panel } from "../src/boards/store.ts";
15 import { components } from "../src/web/style/components.ts";
16 import { createToken } from "../src/core/tokens.ts";
17 import { createUser } from "../src/core/users.ts";
18 import { serve } from "../src/http/server.ts";
19
20 let dir: string;
21 let app: ReturnType<typeof createApp>;
22 let servers: Awaited<ReturnType<typeof serve>>;
23 const tok: Record<string, string> = {};
24 /** The newest minute with a point: one per minute for the 24 h before it. */
25 const LAST = Math.floor(Date.now() / 60_000) * 60_000;
26 const minute = (t: number) => new Date(t).toISOString().slice(0, 16) + ":00Z";
27 const CPU = { source: "machine", aggregate: "avg", column: "cpu_cores", groupBy: { bucket: "minute" }, filter: [{ column: "target", op: "eq", value: "forge" }] };
28 const BOARD: Board = {
29 title: "Usage test",
30 range: "24h",
31 refresh: "off",
32 panels: [
33 { title: "Pod CPU", chart: "line", query: CPU, w: 2, h: 1 },
34 { title: "By target", chart: "bar", query: { source: "machine", aggregate: "count", groupBy: "target" }, w: 1, h: 1 },
35 ],
36 };
37 type Out = { board: string; panel: number; title: string; step: number | null; rows: { group: string; value: number }[] };
38
39 const api = (path: string, who = tok.admin) =>
40 fetch(servers.url + path, { headers: who ? { authorization: `Bearer ${who}` } : {} });
41
42 beforeAll(async () => {
43 dir = await mkdtemp(join(tmpdir(), "forge-panel-data-"));
44 app = createApp({ dataDir: dir, port: 0 });
45 servers = await serve(app);
46 const mk = async (username: string, adminIfFirst?: boolean) => {
47 const r = await createUser(app.db, {
48 origin: "seed", trusted: true, username, email: `${username}@forge.local`,
49 password: "a-good-passphrase", ...(adminIfFirst ? { adminIfFirst } : {}),
50 });
51 if (!r.ok) throw new Error(r.error);
52 return r.value;
53 };
54 const alice = await mk("alice", true);
55 const bob = await mk("bob");
56 tok.admin = createToken(app.db, alice.id, "t", "admin").token;
57 tok.bob = createToken(app.db, bob.id, "t", "admin").token;
58 const ins = app.db.prepare("INSERT INTO machine_sample (at, step, target, cpu_cores) VALUES (?, 15, ?, ?)");
59 app.db.transaction(() => {
60 for (let i = 0; i < 1440; i++) ins.run(new Date(LAST - i * 60_000).toISOString(), "forge", i % 7);
61 ins.run(new Date(LAST).toISOString(), "=HYPERLINK(\"x\")", 1);
62 })();
63 const made = await writeBoard(app, alice, "usage-test", BOARD);
64 if (!made.ok) throw new Error(JSON.stringify(made.error));
65 });
66
67 afterAll(async () => {
68 servers.stop();
69 app.close();
70 await rm(dir, { recursive: true, force: true });
71 });
72
73 describe("runQuery with a step", () => {
74 test("a day of per-minute buckets is every minute, up to the newest, not the first 1000", () => {
75 const a = runQuery(app.db, CPU, { repoIds: "all" });
76 if (!a.ok) throw new Error(a.error);
77 expect(a.rows.length).toBe(1440);
78 expect(a.rows.at(-1)!.group).toBe(minute(LAST));
79 });
80
81 test("a step floors each point to its bucket in UTC; minute, hour and day read as before", () => {
82 const at = (q: Record<string, unknown>) => {
83 const a = runQuery(app.db, { ...CPU, ...q }, { repoIds: "all" });
84 return a.ok ? a.rows.at(-1)!.group : a.error;
85 };
86 const five = Math.floor(LAST / 300_000) * 300_000;
87 expect(at({ step: 300 })).toBe(minute(five));
88 expect(at({ groupBy: { bucket: "hour" } })).toBe(new Date(Math.floor(LAST / 3_600_000) * 3_600_000).toISOString().slice(0, 13) + ":00:00Z");
89 expect(at({ groupBy: { bucket: "day" } })).toBe(new Date(LAST).toISOString().slice(0, 10) + "T00:00:00Z");
90 });
91
92 test("a bad step, or a step without a time bucket, is refused by name", () => {
93 for (const step of [0, -60, 1.5, "60", 31_536_001]) {
94 expect(runQuery(app.db, { ...CPU, step }, { repoIds: "all" })).toMatchObject({ ok: false, error: expect.stringContaining("step") });
95 }
96 expect(runQuery(app.db, { source: "machine", aggregate: "count", groupBy: "target", step: 60 }, { repoIds: "all" }))
97 .toEqual({ ok: false, error: "step needs a group by a time bucket" });
98 });
99 });
100
101 describe("panelData", () => {
102 test("a 24 h board picks a 5-minute step: 288 buckets or one more for the partial edge, the last one now", () => {
103 const d = panelData(app.db, BOARD, BOARD.panels[0]!, {}, LAST + 30_000);
104 if (!d.ok) throw new Error(d.error);
105 expect(d.step).toBe(300);
106 expect(d.rows.length).toBeGreaterThanOrEqual(288);
107 expect(d.rows.length).toBeLessThanOrEqual(289);
108 expect(d.rows.at(-1)!.group).toBe(minute(Math.floor(LAST / 300_000) * 300_000));
109 });
110
111 test("a board whose range is all fits the step to the span from the panel's oldest row, not its bucket", () => {
112 const all: Board = { ...BOARD, range: "all" };
113 const d = panelData(app.db, all, all.panels[0]!, {}, LAST + 30_000);
114 if (!d.ok) throw new Error(d.error);
115 expect(d).toMatchObject({ from: null, step: 300 });
116 expect(d.rows.length).toBeLessThanOrEqual(TARGET_POINTS + 1);
117 // The oldest row is the query's own: its filters and window count, and no rows means no span to fit.
118 expect(oldestTime(app.db, CPU, { repoIds: "all" })).toBe(new Date(LAST - 1439 * 60_000).toISOString());
119 const other = { ...CPU, filter: [{ column: "target", op: "eq", value: '=HYPERLINK("x")' }] };
120 expect(oldestTime(app.db, other, { repoIds: "all" })).toBe(new Date(LAST).toISOString());
121 expect(oldestTime(app.db, { ...CPU, to: "2000-01-01T00:00:00Z" }, { repoIds: "all" })).toBeNull();
122 expect(oldestTime(app.db, { source: "nope" }, { repoIds: "all" })).toBeNull();
123 expect(oldestTime(app.db, CPU, { repoIds: [] })).toBeNull();
124 const empty: Panel = { ...all.panels[0]!, query: { ...CPU, to: "2000-01-01T00:00:00Z" } };
125 expect(panelData(app.db, all, empty, {}, LAST)).toMatchObject({ ok: true, rows: [], step: 60 });
126 });
127
128 test("an answer too large for its step is asked again coarser, never cut", () => {
129 const steps: unknown[] = [];
130 const run: typeof runFor = (_db, _chart, q) => {
131 steps.push(q.step);
132 return (q.step as number) < 900 ? { ok: false, error: "too many", tooMany: true } : { ok: true, rows: [] };
133 };
134 expect(panelData(app.db, BOARD, BOARD.panels[0]!, {}, LAST, run)).toMatchObject({ ok: true, step: 900 });
135 expect(steps).toEqual([300, 600, 900]);
136 steps.length = 0;
137 expect(panelData(app.db, BOARD, BOARD.panels[0]!, { step: 60 }, LAST, run)).toEqual({ ok: false, error: "too many" });
138 expect(steps).toEqual([60]);
139 });
140
141 test("a step fits the span; a step is read as seconds or with a unit", () => {
142 expect([stepFor(3_600_000, 60), stepFor(86_400_000, 60), stepFor(604_800_000, 60), stepFor(2_592_000_000, 60)]).toEqual([60, 300, 1800, 7200]);
143 expect(stepFor(86_400_000, 86_400)).toBe(86_400);
144 expect(["300", "15s", "5m", "1h", "1d"].map(parseStep)).toEqual([300, 15, 300, 3600, 86_400]);
145 for (const bad of ["", "0", "5x", "-5m", "1.5h", "366d", "5 m"]) expect(parseStep(bad)).toBeUndefined();
146 expect(parseWindow(new URLSearchParams("from=2026-10-20T00:00:00Z&to=2026-10-19T00:00:00Z"))).toEqual({ ok: false, error: "from must be before to" });
147 expect(parseWindow(new URLSearchParams("from=yesterday")).ok).toBe(false);
148 });
149
150 test("CSV quotes what needs it and never starts a text cell with a formula", () => {
151 expect(rowsCsv([{ group: "a,b", value: -1.5 }, { group: null, value: null }])).toBe('group,value\r\n"a,b",-1.5\r\n,\r\n');
152 expect(rowsCsv([{ group: "=1+1", series: '"q"', value: 2 }])).toBe(`group,series,value\r\n'=1+1,"""q""",2\r\n`);
153 });
154 });
155
156 describe("GET /api/v1/boards/:slug/panels/:id", () => {
157 test("24 h of 1-minute data at the panel's own step reaches now", async () => {
158 const res = await api("/api/v1/boards/usage-test/panels/0");
159 expect(res.status).toBe(200);
160 const out = (await res.json()) as Out;
161 expect([out.board, out.panel, out.title, out.step]).toEqual(["usage-test", 0, "Pod CPU", 300]);
162 expect(out.rows.at(-1)!.group).toBe(minute(Math.floor(LAST / 300_000) * 300_000));
163 });
164
165 test("step=1m over 24 h is all 1440 minutes; step=5m over an aligned day is 288 buckets", async () => {
166 // From the oldest point, not the clock: a minute turning mid-test would drop it from a now-based window.
167 const oldest = new Date(LAST - 1439 * 60_000).toISOString();
168 const all = (await (await api(`/api/v1/boards/usage-test/panels/0?step=1m&from=${oldest}`)).json()) as Out;
169 expect(all.rows.length).toBe(1440);
170 expect(all.rows.at(-1)!.group).toBe(minute(LAST));
171 // The first 5-minute mark after LAST: the day before it starts within the data and its last bucket holds LAST,
172 // whatever minute the clock is on (ending at the mark before LAST could start a minute before the oldest point).
173 const to = Math.floor(LAST / 300_000) * 300_000 + 300_000;
174 const q = `from=${new Date(to - 86_400_000).toISOString()}&to=${new Date(to).toISOString()}&step=5m`;
175 const five = (await (await api(`/api/v1/boards/usage-test/panels/0?${q}`)).json()) as Out;
176 expect(five.step).toBe(300);
177 expect(five.rows.length).toBe(288);
178 });
179
180 test("the CSV is the JSON's rows", async () => {
181 // One fixed window for both: a now-based one could lose its oldest point between the two reads.
182 const q = `step=1h&from=${new Date(LAST - 1439 * 60_000).toISOString()}`;
183 const json = (await (await api(`/api/v1/boards/usage-test/panels/0?${q}`)).json()) as Out;
184 const res = await api(`/api/v1/boards/usage-test/panels/0?${q}&format=csv`);
185 expect(res.headers.get("content-type")).toBe("text/csv; charset=utf-8");
186 const [head, ...lines] = (await res.text()).trimEnd().split("\r\n");
187 expect(head).toBe("group,value");
188 expect(lines.map((l) => l.split(","))).toEqual(json.rows.map((r) => [r.group, String(r.value)]));
189 expect(lines.length).toBeGreaterThanOrEqual(24);
190 });
191
192 test("a panel not grouped by time ignores the step; a text cell is never a formula", async () => {
193 const out = (await (await api("/api/v1/boards/usage-test/panels/1?step=5m")).json()) as Out;
194 expect(out.step).toBeNull();
195 expect(out.rows.map((r) => r.group)).toEqual(['=HYPERLINK("x")', "forge"]);
196 const csv = await (await api("/api/v1/boards/usage-test/panels/1?format=csv")).text();
197 expect(csv).toBe(`group,value\r\n"'=HYPERLINK(""x"")",1\r\nforge,1440\r\n`);
198 });
199
200 test("a bad window or format is 422; a missing panel or board is 404; a non-admin 403, signed out 401", async () => {
201 expect((await api("/api/v1/boards/usage-test/panels/0?step=5x")).status).toBe(422);
202 expect((await api("/api/v1/boards/usage-test/panels/0?from=soon")).status).toBe(422);
203 expect((await api("/api/v1/boards/usage-test/panels/0?format=xml")).status).toBe(422);
204 expect((await api("/api/v1/boards/usage-test/panels/2")).status).toBe(404);
205 expect((await api("/api/v1/boards/usage-test/panels/x")).status).toBe(404);
206 expect((await api("/api/v1/boards/nope/panels/0")).status).toBe(404);
207 expect((await api("/api/v1/boards/usage-test/panels/0", tok.bob)).status).toBe(403);
208 expect((await api("/api/v1/boards/usage-test/panels/0", "")).status).toBe(401);
209 });
210
211 test("the board page links each panel's data as JSON and as a CSV download, plain links that need no script", async () => {
212 const page = await (await api("/boards/usage-test")).text();
213 for (const i of ["0", "1"]) {
214 expect(page).toContain(`href="/api/v1/boards/usage-test/panels/${i}"`);
215 expect(page).toContain(`href="/api/v1/boards/usage-test/panels/${i}?format=csv" download="usage-test-${i}.csv"`);
216 }
217 // One class holds a card head's links, and on a thumb each is a 44px target (docs/DESIGN.md).
218 expect(page.match(/<span class="card-links muted small"><a href="\/boards\/data\?[^"]*">explore<\/a>\s*<a href="\/api\/v1\/boards\/usage-test\/panels\/\d"/g)?.length).toBe(2);
219 expect(components).toMatch(/@media \(pointer: coarse\) \{[^}]*\}\s*\.card-links > a \{[^}]*min-height: 44px[^}]*\}/);
220 // The link answers what it names, to the same admin.
221 expect((await api("/api/v1/boards/usage-test/panels/1?format=csv")).headers.get("content-type")).toBe("text/csv; charset=utf-8");
222 });
223
224 test("a step finer than the cap allows is refused, not cut", async () => {
225 const ins = app.db.prepare("INSERT INTO machine_sample (at, step, target, cpu_cores) VALUES (?, 15, 'forge', 1)");
226 app.db.transaction(() => {
227 for (let i = 1; i <= MAX_POINTS; i++) ins.run(new Date(LAST - 86_400_000 - i * 1000).toISOString());
228 })();
229 const res = await api(`/api/v1/boards/usage-test/panels/0?step=1s&from=${new Date(LAST - 2 * 86_400_000).toISOString()}`);
230 expect(res.status).toBe(400);
231 expect(((await res.json()) as { message: string }).message).toContain("coarser step");
232 app.db.run("DELETE FROM machine_sample WHERE at < ?", [new Date(LAST - 86_400_000).toISOString()]);
233 });
234 });
test/mcp.test.ts +30
@@ -27,6 +27,8 @@ import { savePlacement } from "../src/core/placements.ts";
2727 import { serve, type Servers } from "../src/http/server.ts";
2828 import { limitChanges, limitValue } from "../src/settings/limits.ts";
2929 import { promote } from "./helpers/members.ts";
30 import { showBoard } from "../src/actions/boards.ts";
31 import { panelData } from "../src/boards/panel_data.ts";
3032
3133 describe("the forge over MCP", () => {
3234 let dataDir: string;
@@ -892,6 +894,34 @@ describe("the forge over MCP", () => {
892894 expect((await tool(rw, "board_delete", { slug: "runs" })).error).toContain("not_found");
893895 });
894896
897 test("boards with a panel: its data through panelData, the window judged; an agent and a non-admin are refused", async () => {
898 const query = { source: "runs", aggregate: "count", groupBy: { bucket: "hour" } };
899 const board = { title: "Hourly", range: "7d", refresh: "off", panels: [{ title: "Runs", chart: "line", query, w: 1, h: 1 }] };
900 expect((await tool(rw, "board_write", { slug: "hourly", board })).value).toMatchObject({ slug: "hourly" });
901 const from = "2026-01-01T00:00:00.000Z";
902 const to = "2027-01-01T00:00:00.000Z";
903 const got = (await tool(ro, "boards", { slug: "hourly", panel: 0, from, to, step: "1d" })).value;
904 const shown = await showBoard(app, alice, "hourly");
905 if (!shown.ok) throw new Error("no board");
906 const direct = panelData(app.db, shown.value, shown.value.panels[0]!, { from, to, step: 86_400 });
907 if (!direct.ok) throw new Error(direct.error);
908 expect(got).toEqual({ board: "hourly", panel: 0, title: "Runs", chart: "line", from, to, step: 86_400, rows: direct.rows });
909 expect((await tool(rw, "boards", { slug: "hourly", panel: 0, step: 300 })).value.step).toBe(300);
910 expect((await tool(rw, "boards", { slug: "hourly", panel: 0 })).value.step).toBe(3600);
911 expect((await tool(rw, "boards", { slug: "hourly", panel: 0, step: "5x" })).error).toContain("invalid_window");
912 expect((await tool(rw, "boards", { slug: "hourly", panel: 0, from: "soon" })).error).toContain("invalid_window");
913 expect((await tool(rw, "boards", { slug: "hourly", panel: 1 })).error).toContain("not_found");
914 expect((await tool(rw, "boards", { slug: "nope", panel: 0 })).error).toContain("not_found");
915 const made = await createAgent(app, alice, { username: "databot", brief: "x" });
916 if (!made.ok) throw new Error(String(made.error));
917 app.db.query("UPDATE user SET is_admin = 1 WHERE id = ?1").run(made.value.user.id);
918 const botToken = createToken(app.db, made.value.user.id, "bot", "rw").token;
919 for (const who of [bobRw, botToken]) {
920 expect((await tool(who, "boards", { slug: "hourly", panel: 0 })).error).toContain("forbidden");
921 }
922 await tool(rw, "board_delete", { slug: "hourly" });
923 });
924
895925 test("ui:// card resources are listed, read, and tied to their tools", async () => {
896926 const list = await call(rw, "resources/list");
897927 const uris = list.body.result.resources.map((r: any) => r.uri).sort();
test/tui-boards.test.ts +24 −2
@@ -8,13 +8,13 @@ import { mkdtemp, rm } from "node:fs/promises";
88 import { tmpdir } from "node:os";
99 import { join } from "node:path";
1010 import { listPinned, pinPanel, unpinPanel } from "../src/actions/board-pins.ts";
11 import { writeBoard } from "../src/actions/boards.ts";
11 import { deleteBoard, writeBoard } from "../src/actions/boards.ts";
1212 import { createAgent } from "../src/actions/crew.ts";
1313 import { createApp, type App } from "../src/app.ts";
1414 import { createUser, type User } from "../src/core/users.ts";
1515 import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
1616 import { dashboardScreen } from "../src/tui/screens/dashboard.ts";
17 import { boardRows, boardsScreen } from "../src/tui/screens/boards.ts";
17 import { boardRows, boardsScreen, dataLine, stepWords } from "../src/tui/screens/boards.ts";
1818 import { headlessTerminal } from "../src/tui/term/terminal.ts";
1919 import { quietLogs } from "./helpers/globals.ts";
2020 import { settleApp } from "./helpers/tui.ts";
@@ -101,6 +101,28 @@ describe("tui boards", () => {
101101 expect(text).toContain("source=runs");
102102 });
103103
104 test("each panel says what panelData reads for it: a time panel its last bucket and step, another its rows", async () => {
105 const wrote = await writeBoard(app, carol, "weekly", {
106 title: "Hourly runs",
107 range: "7d",
108 refresh: "off",
109 panels: [
110 { title: "Runs per hour", chart: "line", query: { source: "runs", aggregate: "count", groupBy: { bucket: "hour" } }, w: 1, h: 1 },
111 { title: "Runs by state", chart: "bar", query: { source: "runs", aggregate: "count", groupBy: "state" }, w: 1, h: 1 },
112 ],
113 });
114 if (!wrote.ok) throw new Error(JSON.stringify(wrote.error));
115 // The boards are listed by slug: runs, then weekly.
116 const text = await screenText(carol, "j\r");
117 expect(text).toContain("no points · step 1h · 0 points");
118 expect(text).toContain("0 rows");
119 expect(dataLine({ ok: true, rows: [{ group: "2026-10-20T21:45:00Z", value: 1500 }], from: null, to: null, step: 300 }))
120 .toBe("last 2026-10-20T21:45:00Z: 1500 · step 5m · 1 point");
121 expect(dataLine({ ok: false, error: "too\u001b[2Jmany" })).toBe("refused: too [2Jmany");
122 expect([15, 300, 3600, 86_400, 90].map(stepWords)).toEqual(["15s", "5m", "1h", "1d", "90s"]);
123 await deleteBoard(app, carol, "weekly");
124 });
125
104126 test("nobody else has rows or the door", async () => {
105127 for (const who of [bob, bot, undefined]) {
106128 expect(await boardRows(new Session(app, identity(who)))).toEqual([]);

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs