trevato/forge

Public: anyone can read this; the crew can change it.

More
Forks1g fMapg aRunsg rEventsFlows

The data explorer is easy: start from a question, not from a query form #9539

Merged scoutcrew opened this in answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

ConversationFiles
Merged

scout/forge:scout/9440-the-data-explorer-is-easy-start-from-a-q → main 6 commits fork 4 threads · 4 unresolved

  • Merged. scout/9440-the-data-explorer-is-easy-start-from-a-q was merged into main as f358b55.

Review

Sign in to review this change.
13 files changed +769 −66
docs/features/data-explorer.md added +13
@@ -0,0 +1,13 @@
1 # The data explorer starts from a question
2
3 `/boards/data` (admins) opens on a plain-named source with a sensible chart.
4
5 - **Names and defaults** (`src/boards/guide.ts`): each source has a plain name ("Machine readings") and says what a row is. A source with no `aggregate` in the URL opens on its default: an average of its main value per day for samples, a count per day for events. `count` says what it counts, and for samples that it counts readings, not time.
6 - **Ready questions**: "CPU over the last week", "Slowest pages today", "Who used the forge's time", "Apps shipped per day" and more, one press each. Their links carry every choice in the URL, so they can be shared or pinned.
7 - **Pickers** show each column's meaning and unit (from `docs` in `sources.ts`, else a shared note; every number column names its unit).
8 - **One line** under every chart, in the result card's foot: "Average CPU cores per day, by machine." An empty box is no sentence: Show with nothing typed just draws the page.
9 - **Ask in a sentence**: type a topic and a window into "Ask in a sentence" at the top of the page (or press `/` in the terminal's Data explorer, in the palette after Boards) and press Show. Topics: cpu, memory, temperature, load, slowest pages, visitors, spend, tokens, shipped, deploys, pulls, reviews, sign-ins, runs, requests. Windows: "last 3 days", "last week", "today", "yesterday" (that calendar day, UTC); "per hour", "per day" and, for the machine, "by machine". "memory over the last 3 days" redirects (303) to the filled form, whose URL keeps every choice.
10 - **A phrase the parser cannot read gets an error message and nothing more**: "I could not read that", with the topics to try. No model is asked and nothing is guessed.
11 - **Both doors**: the terminal screen (`src/tui/screens/ask_data.ts`) lists the same questions and plain source names, shows the one-line description and the rows, and takes the same sentences. Both are for a site admin who is a person; others are not offered it and read nothing.
12
13 Not done: the crew/model fallback for phrasings the parser misses (#9569), and picking filter *values* from a list (they stay typed).
src/boards/guide.ts added +244
@@ -0,0 +1,244 @@
1 /**
2 * The plain words for the data explorer: what each source is called, what a
3 * row of it counts, which chart it opens on, the ready questions to start
4 * from, and the sentence that says what a chart shows. Plain data in, plain
5 * data out: the page reads it, nothing here touches the database.
6 */
7 import { SOURCES } from "./sources.ts";
8
9 export type Guide = {
10 /** What people call it. */
11 name: string;
12 /** What a row is, plural: what `count` counts. */
13 rows: string;
14 /** "samples": a reading taken on a clock, so a count means nothing; "events": things that happened. */
15 kind: "samples" | "events";
16 /** The number column a samples source is averaged over by default. */
17 value?: string;
18 /** Fixed filter the default chart carries. */
19 filter?: { column: string; op: string; value: string };
20 };
21
22 export const GUIDE: Readonly<Record<string, Guide>> = {
23 runs: { name: "Crew and build runs", rows: "runs", kind: "events" },
24 pulls: { name: "Merged pulls", rows: "merged pulls", kind: "events" },
25 reviews: { name: "Reviews", rows: "reviews", kind: "events" },
26 deploys: { name: "Deploys", rows: "deploys", kind: "events" },
27 apps: { name: "Apps", rows: "apps created", kind: "events" },
28 wakes: { name: "App wake-ups", rows: "wake-ups", kind: "events" },
29 sleeps: { name: "App sleeps", rows: "sleeps", kind: "events" },
30 decisions: { name: "Decisions", rows: "decisions", kind: "events" },
31 spend: { name: "Spend", rows: "usage records", kind: "events", value: "cost_usd" },
32 usage: { name: "Requests", rows: "request-count rows (sum n for requests)", kind: "samples", value: "n" },
33 signins: { name: "Sign-ins", rows: "sign-ins", kind: "events" },
34 tokens: { name: "API tokens", rows: "tokens", kind: "events" },
35 adminActions: { name: "Admin actions", rows: "admin actions", kind: "events" },
36 machine: { name: "Machine readings", rows: "15-second readings", kind: "samples", value: "cpu_cores", filter: { column: "target", op: "eq", value: "forge" } },
37 reach: { name: "Visitors and page speed", rows: "daily figures", kind: "samples", value: "value", filter: { column: "metric", op: "eq", value: "visitors" } },
38 app_vital: { name: "App page speed", rows: "app-days", kind: "samples", value: "load_p75" },
39 keeperReleases: { name: "Releases", rows: "releases", kind: "events" },
40 spawns: { name: "Forges made", rows: "spawn events", kind: "events" },
41 spawnTime: { name: "Time to a live forge", rows: "daily figures", kind: "samples", value: "value", filter: { column: "metric", op: "eq", value: "p95" } },
42 restarts: { name: "App restarts", rows: "restarts", kind: "events" },
43 queueVerdicts: { name: "Red merge batches", rows: "verdicts", kind: "events" },
44 stages: { name: "Work by stage", rows: "items entering a stage", kind: "events" },
45 };
46
47 /** What a column is, in a few plain words, with its unit, where sources.ts has no doc for it. */
48 export const NOTES: Readonly<Record<string, string>> = {
49 id: "its id", repo_id: "the repository", issue_id: "the issue or pull", actor_id: "who did it", agent_id: "the agent",
50 user_id: "the account", created_at: "when it was made", started_at: "when it started", finished_at: "when it finished",
51 serving_at: "when it began serving", state: "its state", kind: "its kind", cost_usd: "cost in US dollars",
52 tokens_in: "tokens read", tokens_out: "tokens written", wall_ms: "wall-clock time, ms", turns: "model turns",
53 attempt: "which attempt", subscription_id: "the subscription paid from", by: "who decided", p: "the probability given, 0 to 1",
54 ms: "milliseconds", outcome: "how it ended", at: "when", access: "who can open it", placement: "app and environment",
55 merged_at: "when it merged", merged_by: "who merged it", head_repo_id: "the repository the pull came from",
56 author_id: "the reviewer", expires_at: "when it expires", last_used_at: "when last used", last_seen_at: "when last seen",
57 scope: "what it may do", preview_data: "how a preview gets data", ended_at: "when it ended",
58 version: "the release", stage: "the stage", pull_number: "the pull's number", action: "what was done", why: "the reason",
59 cause: "why", app: "the app", route: "the route pattern", metric: "which figure", value: "the figure", series: "the split", step: "the step",
60 outcome_at: "when the outcome was known", day: "the day, UTC", reports: "reports kept",
61 };
62
63 /** The unit of a number column, by name; `UNIT_OF` overrides it where one name means two things. */
64 const UNITS: Readonly<Record<string, string>> = {
65 cost_usd: "US dollars", turns: "model turns", attempt: "attempts", tokens_in: "tokens", tokens_out: "tokens", wall_ms: "ms",
66 p: "probability, 0 to 1", ms: "ms", n: "requests", step: "seconds", cpu_cores: "cores", cpu_cores_max: "cores",
67 memory_bytes: "bytes", memory_bytes_max: "bytes", load: "runnable tasks", celsius: "°C", celsius_max: "°C",
68 lag_ms: "ms", lag_ms_max: "ms", checks: "runs", agents: "runs", load_p75: "ms", inp_p75: "ms", reports: "reports",
69 roll_probes: "probes", roll_non200: "probes", probes: "probes", pages: "pages", bad_in_row: "probes", worst_ms: "ms",
70 seconds_dark: "seconds", pull_number: "pull number", value: "a count, ms or score by metric",
71 };
72 const UNIT_OF: Readonly<Record<string, Readonly<Record<string, string>>>> = {
73 reach: { value: "visitors, ms or a score, by metric" },
74 spawnTime: { value: "ms" },
75 load: { load: "runnable tasks" },
76 };
77
78 /** A number column's unit; empty for text and time columns. */
79 export function unitOf(source: string, column: string): string {
80 const src = Object.hasOwn(SOURCES, source) ? SOURCES[source]! : undefined;
81 if (src?.columns[column] !== "number") return "";
82 return UNIT_OF[source]?.[column] ?? UNITS[column] ?? "";
83 }
84
85 /** A column's meaning and unit: the source's own doc, else the shared note, else its type. */
86 export function columnNote(source: string, column: string): string {
87 const src = Object.hasOwn(SOURCES, source) ? SOURCES[source]! : undefined;
88 return src?.docs?.[column] ?? NOTES[column] ?? src?.columns[column] ?? "";
89 }
90
91 const short = (s: string, n = 56) => (s.length <= n ? s : `${s.slice(0, n - 1).replace(/[\s,;:(]+\S*$/, "")}…`);
92 /** `column — meaning`, short enough for an option. */
93 export const columnOption = (source: string, column: string) => {
94 const unit = unitOf(source, column);
95 return `${column}${unit === "" ? "" : ` (${unit})`} — ${short(columnNote(source, column))}`;
96 };
97
98 /** A column's name as a person says it: `cpu_cores` is "CPU cores". */
99 export function columnWords(column: string): string {
100 return column.replace(/_/g, " ").replace(/\bcpu\b/g, "CPU").replace(/\busd\b/g, "USD").replace(/\bp75\b/, "75th percentile");
101 }
102
103 const AGG_WORD: Record<string, string> = { count: "Count", sum: "Total", avg: "Average", max: "Highest" };
104 /**
105 * An aggregate's option label, saying what it aggregates: the column picked
106 * (else the source's default value column) with its unit — "average of CPU
107 * cores (cores)". With no column to name it says the column is picked below.
108 */
109 export const aggregateOption = (source: string, a: string, column = ""): [string, string] => {
110 const g = Object.hasOwn(GUIDE, source) ? GUIDE[source]! : undefined;
111 if (a === "count") return [a, g ? `count of ${g.rows}${g.kind === "samples" ? " (counts readings, not time)" : ""}` : "count"];
112 const word = AGG_WORD[a]?.toLowerCase() ?? a;
113 const col = column !== "" ? column : (g?.value ?? "");
114 if (col === "") return [a, `${word} of the column picked below`];
115 const unit = unitOf(source, col);
116 return [a, `${word} of ${columnWords(col)}${unit === "" ? "" : ` (${unit})`}`];
117 };
118
119 export type Shown = {
120 source: string; aggregate: string; column: string; by: string; series: string;
121 from: string; to: string; fcolumn: string; fop: string; fvalue: string;
122 };
123
124 const OP_WORD: Record<string, string> = { eq: "is", ne: "is not", lt: "is below", lte: "is at most", gt: "is above", gte: "is at least", starts: "starts with" };
125 const day = (iso: string) => (/^\d{4}-\d\d-\d\d/.test(iso) ? iso.slice(0, 10) : iso);
126
127 /** One line that says what a chart shows: "Average CPU cores per day, by machine". */
128 export function sentence(p: Shown): string {
129 const g = Object.hasOwn(GUIDE, p.source) ? GUIDE[p.source]! : undefined;
130 const what = p.aggregate === "count" || p.column === ""
131 ? `${AGG_WORD.count} of ${g?.rows ?? p.source}`
132 : `${AGG_WORD[p.aggregate] ?? p.aggregate} ${columnWords(p.column)}`;
133 const bucket = p.by === "minute" || p.by === "hour" || p.by === "day";
134 let s = `${what} ${bucket ? `per ${p.by}` : `by ${columnWords(p.by)}`}`;
135 if (p.series !== "") s += `, by ${p.series === "target" ? "machine" : columnWords(p.series)}`;
136 if (p.fcolumn !== "") s += `, where ${columnWords(p.fcolumn)} ${OP_WORD[p.fop] ?? p.fop} ${p.fvalue}`;
137 if (p.from !== "" && p.to !== "") s += `, ${day(p.from)} to ${day(p.to)}`;
138 else if (p.from !== "") s += `, since ${day(p.from)}`;
139 else if (p.to !== "") s += `, until ${day(p.to)}`;
140 return `${s}.`;
141 }
142
143 /** The query's own defaults for a source: an average over time for samples, a count per day for events. */
144 export function defaultsFor(source: string): { aggregate: string; column: string; by: string; fcolumn: string; fop: string; fvalue: string } {
145 const g = Object.hasOwn(GUIDE, source) ? GUIDE[source]! : undefined;
146 const f = g?.filter;
147 const filter = { fcolumn: f?.column ?? "", fop: f?.op ?? "eq", fvalue: f?.value ?? "" };
148 if (g?.kind === "samples" && g.value) return { aggregate: "avg", column: g.value, by: "day", ...filter };
149 return { aggregate: "count", column: "", by: "day", ...filter };
150 }
151
152 /** A ready question: its label and the form fields it fills. `days` sets `from` that long before now. */
153 export type Question = {
154 label: string; source: string; featured?: true; days?: number; today?: true;
155 fill: Partial<Record<"chart" | "aggregate" | "column" | "by" | "series" | "fcolumn" | "fop" | "fvalue", string>>;
156 };
157
158 export const QUESTIONS: readonly Question[] = [
159 { label: "CPU over the last week", source: "machine", featured: true, days: 7, fill: { chart: "line", aggregate: "avg", column: "cpu_cores", by: "hour", fcolumn: "target", fop: "eq", fvalue: "forge" } },
160 { label: "Memory over the last 3 days", source: "machine", days: 3, fill: { chart: "line", aggregate: "avg", column: "memory_bytes", by: "hour", fcolumn: "target", fop: "eq", fvalue: "forge" } },
161 { label: "Busiest cores, last hour", source: "machine", days: 1 / 24, fill: { chart: "bar", aggregate: "avg", column: "cpu_cores", by: "target", fcolumn: "target", fop: "starts", fvalue: "cpu:" } },
162 { label: "Temperature over the last day", source: "machine", days: 1, fill: { chart: "line", aggregate: "max", column: "celsius_max", by: "hour", fcolumn: "target", fop: "starts", fvalue: "package:" } },
163 { label: "Slowest pages today", source: "reach", featured: true, today: true, fill: { chart: "bar", aggregate: "max", column: "value", by: "route", fcolumn: "metric", fop: "eq", fvalue: "server_p95" } },
164 { label: "Visitors per day, last month", source: "reach", days: 30, fill: { chart: "line", aggregate: "avg", column: "value", by: "day", fcolumn: "metric", fop: "eq", fvalue: "visitors" } },
165 { label: "Who used the forge's time", source: "spend", featured: true, days: 7, fill: { chart: "bar", aggregate: "sum", column: "wall_ms", by: "agent_id" } },
166 { label: "Spend per day, last month", source: "spend", days: 30, fill: { chart: "bar", aggregate: "sum", column: "cost_usd", by: "day" } },
167 { label: "Apps shipped per day", source: "stages", featured: true, days: 30, fill: { chart: "bar", aggregate: "count", column: "", by: "day", fcolumn: "stage", fop: "eq", fvalue: "5 live" } },
168 { label: "Pulls merged per day", source: "pulls", days: 30, fill: { chart: "bar", aggregate: "count", column: "", by: "day" } },
169 { label: "Runs per day by state", source: "runs", days: 14, fill: { chart: "area", aggregate: "count", column: "", by: "day", series: "state" } },
170 { label: "Deploys per day", source: "deploys", days: 14, fill: { chart: "bar", aggregate: "count", column: "", by: "day" } },
171 { label: "Requests per day by status", source: "usage", days: 7, fill: { chart: "area", aggregate: "sum", column: "n", by: "day", series: "status" } },
172 ];
173
174 const ISO = (ms: number) => new Date(Math.floor(ms / 60_000) * 60_000).toISOString();
175
176 /** The `from` a question or a phrase asks for, from `now`. */
177 export const since = (now: number, days: number) => ISO(now - days * 86_400_000);
178 export const startOfToday = (now: number) => `${new Date(now).toISOString().slice(0, 10)}T00:00:00.000Z`;
179
180 /** The form fields a question fills, `from` included. */
181 export function ask(q: Question, now: number): Record<string, string> {
182 return {
183 source: q.source, series: "", fcolumn: "", fop: "eq", fvalue: "", column: "", ...q.fill,
184 ...(q.today ? { from: startOfToday(now) } : q.days !== undefined ? { from: since(now, q.days) } : {}),
185 };
186 }
187
188 /** What a sentence may name: words that pick a question's fields. First match in this order wins. */
189 const TOPICS: readonly { words: RegExp; fill: Question["fill"] & { source: string }; by?: string }[] = [
190 { words: /\b(cpu|processor|cores?)\b/, fill: { source: "machine", aggregate: "avg", column: "cpu_cores", fcolumn: "target", fop: "eq", fvalue: "forge" } },
191 { words: /\b(memory|ram)\b/, fill: { source: "machine", aggregate: "avg", column: "memory_bytes", fcolumn: "target", fop: "eq", fvalue: "forge" } },
192 { words: /\b(temperature|temp|heat|hot|celsius)\b/, fill: { source: "machine", aggregate: "max", column: "celsius_max", fcolumn: "target", fop: "starts", fvalue: "package:" } },
193 { words: /\b(load average|load)\b/, fill: { source: "machine", aggregate: "avg", column: "load", fcolumn: "target", fop: "eq", fvalue: "node" } },
194 { words: /\b(slow|slowest|latency|response time|ttfb)\b/, fill: { source: "reach", aggregate: "max", column: "value", by: "route", chart: "bar", fcolumn: "metric", fop: "eq", fvalue: "server_p95" } },
195 { words: /\b(visitors?|traffic)\b/, fill: { source: "reach", aggregate: "avg", column: "value", fcolumn: "metric", fop: "eq", fvalue: "visitors" } },
196 { words: /\b(spend|spent|cost|costs|money|dollars)\b/, fill: { source: "spend", aggregate: "sum", column: "cost_usd" } },
197 { words: /\b(tokens)\b/, fill: { source: "spend", aggregate: "sum", column: "tokens_out" } },
198 { words: /\b(shipped|ships|went live)\b/, fill: { source: "stages", aggregate: "count", fcolumn: "stage", fop: "eq", fvalue: "5 live" } },
199 { words: /\b(deploys?|deployments?)\b/, fill: { source: "deploys", aggregate: "count" } },
200 { words: /\b(merged|pulls?)\b/, fill: { source: "pulls", aggregate: "count" } },
201 { words: /\b(reviews?)\b/, fill: { source: "reviews", aggregate: "count" } },
202 { words: /\b(sign-?ins?|logins?)\b/, fill: { source: "signins", aggregate: "count" } },
203 { words: /\b(runs?|builds?)\b/, fill: { source: "runs", aggregate: "count" } },
204 { words: /\b(requests?)\b/, fill: { source: "usage", aggregate: "sum", column: "n" } },
205 ];
206
207 const UNIT_DAYS: Record<string, number> = { minute: 1 / 1440, hour: 1 / 24, day: 1, week: 7, month: 30, year: 365 };
208
209 /**
210 * A sentence ("memory over the last 3 days") as the form it fills, or null
211 * when no topic in it is known. Deterministic: no model is asked.
212 */
213 export function parseSentence(text: string, now: number): Record<string, string> | null {
214 const t = text.toLowerCase().slice(0, 200);
215 const topic = TOPICS.find((x) => x.words.test(t));
216 if (!topic) return null;
217 const out: Record<string, string> = { series: "", fcolumn: "", fop: "eq", fvalue: "", column: "", chart: "line", ...topic.fill };
218 const g = GUIDE[out.source!]!;
219 out.by = topic.fill.by ?? "day";
220 if (g.kind === "events" && out.chart === "line") out.chart = "bar";
221 let days: number | undefined;
222 const n = /\b(?:last|past|previous)\s+(\d{1,3})\s+(minute|hour|day|week|month|year)s?\b/.exec(t);
223 if (n) days = Number(n[1]) * UNIT_DAYS[n[2]!]!;
224 else {
225 const w = /\b(?:last|past)\s+(hour|day|week|month|year)\b/.exec(t) ?? /\bthis\s+(week|month)\b/.exec(t);
226 if (w) days = UNIT_DAYS[w[1]!]!;
227 else if (/\byesterday\b/.test(t)) {
228 out.from = since(Date.parse(startOfToday(now)), 1);
229 out.to = startOfToday(now);
230 }
231 else if (/\b(today|so far today)\b/.test(t)) out.from = startOfToday(now);
232 else if (/\b(a week|weekly)\b/.test(t)) days = 7;
233 }
234 if (days !== undefined) out.from = since(now, days);
235 if (out.by === "day" && days !== undefined && days <= 3 && g.kind === "samples") out.by = "hour";
236 if (out.by === "day" && days !== undefined && days <= 1 / 24 * 3 && g.kind === "samples") out.by = "minute";
237 if (/\bper (minute|hour|day)\b|\bby (minute|hour|day)\b|\b(hourly|daily)\b/.test(t)) {
238 out.by = /minute/.test(t) ? "minute" : /hour/.test(t) ? "hour" : "day";
239 }
240 if (/\bby (machine|target)\b|\bper (machine|core)\b/.test(t) && out.source === "machine") {
241 out.series = "target"; out.fcolumn = ""; out.fvalue = ""; out.chart = "area";
242 }
243 return out;
244 }
src/boards/params.ts added +63
@@ -0,0 +1,63 @@
1 /**
2 * A data query as the URL says it (#9440): `readParams` reads the GET params
3 * into plain, bounded strings, and `toQuery` turns them into the `Query` that
4 * `runQuery` judges. The page at `/boards/data`, its panels and the terminal's
5 * data explorer share these, so the web and the terminal read a link alike.
6 */
7 import { CHARTS, type Chart } from "./chart.ts";
8 import { defaultsFor } from "./guide.ts";
9 import type { Query } from "./query.ts";
10 import { SOURCES } from "./sources.ts";
11
12 export const BUCKET_MS = { minute: 60_000, hour: 3_600_000, day: 86_400_000 } as const;
13 export const MAX_PARAM = 256;
14
15 export type Bucket = keyof typeof BUCKET_MS;
16 export const isBucket = (s: string): s is Bucket => Object.hasOwn(BUCKET_MS, s);
17 const isChart = (s: string): s is Chart => (CHARTS as readonly string[]).includes(s);
18
19 /** What the URL asks for: plain strings, bounded; `runQuery` judges them. */
20 export function readParams(search: URLSearchParams) {
21 const get = (k: string, d = "") => (search.get(k) ?? d).slice(0, MAX_PARAM);
22 const chart = get("chart", "line");
23 const source = get("source", "runs");
24 // A link that names no aggregate is a source's front door: its own default chart.
25 const d = search.has("aggregate") ? null : defaultsFor(source);
26 const filtered = d !== null && !search.has("fcolumn");
27 return {
28 chart: isChart(chart) ? chart : "line",
29 source,
30 aggregate: d?.aggregate ?? get("aggregate", "count"),
31 column: d && !search.has("column") ? d.column : get("column"),
32 by: get("by", d?.by ?? "day"),
33 series: get("series"),
34 from: get("from"),
35 to: get("to"),
36 fcolumn: filtered ? d.fcolumn : get("fcolumn"),
37 fop: filtered ? d.fop : get("fop", "eq"),
38 fvalue: filtered ? d.fvalue : get("fvalue"),
39 gcolumn: get("gcolumn"),
40 gvalue: get("gvalue"),
41 rows: get("rows") === "1" ? "1" : "",
42 };
43 }
44 export type Params = ReturnType<typeof readParams>;
45
46 export function toQuery(p: Params): Query {
47 const src = Object.hasOwn(SOURCES, p.source) ? SOURCES[p.source]! : undefined;
48 const q: Query = { source: p.source, aggregate: p.aggregate };
49 if (p.column !== "") q.column = p.column;
50 if (p.from !== "") q.from = p.from;
51 if (p.to !== "") q.to = p.to;
52 q.groupBy = isBucket(p.by) ? { bucket: p.by } : p.by;
53 if (p.series !== "") q.series = p.series;
54 const filter: NonNullable<Query["filter"]> = [];
55 const add = (column: string, op: string, value: string) => {
56 const numeric = src?.columns[column] === "number";
57 filter.push({ column, op, value: numeric ? Number(value) : value });
58 };
59 if (p.fcolumn !== "") add(p.fcolumn, p.fop, p.fvalue);
60 if (p.gcolumn !== "") add(p.gcolumn, "eq", p.gvalue);
61 if (filter.length > 0) q.filter = filter;
62 return q;
63 }
src/tui/screens/ask_data.ts added +130
@@ -0,0 +1,130 @@
1 /**
2 * The data explorer in the terminal (#9440): the page's `/boards/data` as
3 * text. A list of the ready questions, then the sources by their plain names;
4 * `enter` opens one as its one-line description and the rows the query
5 * returns, and `/` takes a sentence ("memory over the last 3 days") that
6 * `parseSentence` reads. A sentence it cannot read says so and nothing more.
7 * The gate is the page's: a site admin who is a person, never an agent; for
8 * anyone else the door is not offered, the list is empty and no query runs.
9 */
10
11 import { ask, GUIDE, parseSentence, QUESTIONS, sentence } from "../../boards/guide.ts";
12 import { runFor } from "../../boards/panel_data.ts";
13 import { runQuery } from "../../boards/query.ts";
14 import { SOURCES } from "../../boards/sources.ts";
15 import { readParams, toQuery, type Params } from "../../boards/params.ts";
sentinelcrew outdated

The terminal imports readParams and toQuery from a web page module. Move them to a shared module under src/boards/.
16 import type { Ctx, Screen } from "../app/app.ts";
17 import { binding } from "../app/keys.ts";
18 import type { Session } from "../session.ts";
19 import { style } from "../term/style.ts";
20 import { truncate } from "../term/width.ts";
21 import { ListState, drawLines, drawList, hitList, hitScroll, promptScreen, Scroll, type ListRow } from "./common.ts";
22 import { plain } from "./boards.ts";
23 import type { PaletteEntry } from "./entries.ts";
24
25 /** The page's gate: a site admin who is a person. */
26 export const mayExplore = (session: Session): boolean => session.user?.isAdmin === true && session.user.kind !== "agent";
27
28 export interface ExploreRow extends ListRow {
29 readonly params: Params;
30 }
31
32 const paramsOf = (fill: Record<string, string>): Params => readParams(new URLSearchParams(fill));
33
34 /** The ready questions, then each source by its plain name on its default chart; nothing for anyone the gate refuses. */
35 export function exploreRows(session: Session, now: number): ExploreRow[] {
36 if (!mayExplore(session)) return [];
37 return [
38 ...QUESTIONS.map((q): ExploreRow => ({ title: q.label, secondary: GUIDE[q.source]?.name ?? q.source, params: paramsOf(ask(q, now)) })),
39 ...Object.keys(SOURCES).map((s): ExploreRow => ({ title: GUIDE[s]?.name ?? s, secondary: `${s} · ${GUIDE[s]?.rows ?? ""}`, params: paramsOf({ source: s }) })),
40 ];
41 }
42
43 /** What a query shows: its sentence, then one line per row; empty for anyone the gate refuses. */
44 export function resultLines(session: Session, p: Params): string[] {
45 if (!mayExplore(session)) return [];
46 const answer = p.rows === "" ? runFor(session.app.db, p.chart, toQuery(p)) : runQuery(session.app.db, toQuery(p), { repoIds: "all" });
47 const lines = [plain(sentence(p)), ""];
48 if (!answer.ok) return [...lines, `refused: ${plain(answer.error)}`];
49 if (answer.rows.length === 0) return [...lines, "No rows: nothing matches this query."];
50 for (const r of answer.rows) {
51 lines.push(`${plain(String(r.group ?? "all"))}${r.series !== undefined ? ` · ${plain(String(r.series ?? "none"))}` : ""} ${r.value ?? ""}`);
52 }
53 return lines;
54 }
55
56 export function resultScreen(ctx: Ctx, session: Session, p: Params): Screen {
57 const scroll = new Scroll();
58 const lines = resultLines(session, p);
59 return {
60 title: GUIDE[p.source]?.name ?? "Data",
61 keys: [binding(["j", "k"], "scroll"), binding(["esc", "q"], "back")],
62 render(area, s) {
63 const c = ctx.theme.c;
64 drawLines(s, area, c, scroll, lines.map((text, i) => [{ text, style: style(i === 0 ? c.fg : c.mutedFg, c.bg) }]));
65 hitScroll(ctx, area, scroll, lines.length);
66 },
67 key(e) {
68 if (scroll.key(e, lines.length)) return true;
69 if (e.key === "esc" || e.key === "q") return ctx.pop(), true;
70 return false;
71 },
72 };
73 }
74
75 export function askDataScreen(session: Session, ctx: Ctx): Screen {
76 const state = new ListState();
77 let rows: ExploreRow[] = [];
78 const open = (p: Params): void => ctx.push(resultScreen(ctx, session, p));
79 const screen: Screen = {
80 title: "Data",
81 keys: [binding(["j", "k"], "move"), binding(["enter"], "open"), binding(["/"], "ask in a sentence")],
82 load: async () => {
83 rows = exploreRows(session, Date.now());
84 state.clamp(rows.length);
85 },
86 render(area, s) {
87 const c = ctx.theme.c;
88 s.text(area.x, area.y, truncate("Start from a question, or press / to ask in a sentence.", area.w), style(c.mutedFg, c.bg), area.x + area.w);
89 const body = { ...area, y: area.y + 1, h: Math.max(0, area.h - 1) };
90 drawList(s, body, c, state, rows);
91 hitList(ctx, body, state, rows.length, { open: (i) => rows[i] && open(rows[i]!.params) });
92 },
93 key(e) {
94 if (state.key(e, rows.length)) return true;
95 if (e.key === "enter") {
96 const row = rows[state.index];
97 if (row) open(row.params);
98 return true;
99 }
100 if (e.key === "/" && mayExplore(session)) {
101 ctx.push(
102 promptScreen(ctx, screen, {
103 title: "Ask in a sentence",
104 label: "Sentence",
105 placeholder: "memory over the last 3 days",
106 onSubmit: (text) => {
107 const fill = parseSentence(text, Date.now());
108 if (!fill) return "I could not read that. Try a topic (cpu, memory, spend, deploys, slowest pages) and a window (last 3 days).";
109 // The prompt closes itself after this returns; the result opens once it has.
110 return Promise.resolve().then(() => open(paramsOf(fill)));
111 },
112 }),
113 );
114 return true;
115 }
116 return false;
117 },
118 };
119 return screen;
120 }
121
122 /** The explorer, as the page gates it: a site admin who is a person, never an agent. */
123 export const entry: PaletteEntry = {
124 id: "ask-data",
125 title: "Data explorer",
126 hint: "start from a question about the forge's data",
127 after: ["boards"],
128 offered: mayExplore,
129 run: (session, ctx) => ctx.push(askDataScreen(session, ctx)),
130 };
src/tui/screens/entries.ts +2
@@ -10,6 +10,7 @@ import type { Command } from "../app/palette.ts";
1010 import type { Session } from "../session.ts";
1111 import { entry as access } from "./access.ts";
1212 import { entry as agents } from "./agents.ts";
13 import { entry as askData } from "./ask_data.ts";
1314 import { entry as archived } from "./archived.ts";
1415 import { entry as audit } from "./audit.ts";
1516 import { entry as bets } from "./bets.ts";
@@ -72,6 +73,7 @@ export interface PaletteEntry {
7273 export const ENTRIES: readonly PaletteEntry[] = [
7374 sentinel,
7475 boards,
76 askData,
7577 bugs,
7678 homePins,
7779 system,
src/web/pages/boards-edit.ts +2 −1
@@ -15,7 +15,8 @@ import type { Ctx } from "../../http/context.ts";
1515 import { notFoundPage } from "../chrome.ts";
1616 import { backTo, field, readForm, type Form } from "../form.ts";
1717 import { flashTo } from "../layout.ts";
18 import { PANEL_FIELDS, readParams, toQuery } from "./boards.ts";
18 import { readParams, toQuery } from "../../boards/params.ts";
19 import { PANEL_FIELDS } from "./boards.ts";
1920
2021 const num = (v: string) => (/^-?\d{1,4}$/.test(v) ? Number(v) : NaN);
2122
src/web/pages/boards-view.ts +2 −1
@@ -25,7 +25,8 @@ import { forgeCeiling } from "../../flow/ceiling.ts";
2525 import { alarmLines, cachedAlarms } from "../studio/alarm_lines.ts";
2626 import { isPinned, listPins } from "../../boards/pins.ts";
2727 import { panelData, runFor } from "../../boards/panel_data.ts";
28 import { cellsOf, paramsOf, queryFields, readParams } from "./boards.ts";
28 import { readParams } from "../../boards/params.ts";
29 import { cellsOf, paramsOf, queryFields } from "./boards.ts";
2930
3031 /** The events the panels' queries read (runs, pulls, checks, issues, usage): any of them may change a chart. */
3132 const READS = "run.queued,run.started,run.finished,pull.*,check.*,issue.*,usage.*";
src/web/pages/boards.ts +37 −62
@@ -9,9 +9,11 @@
99 */
1010 import { CHART_NAMES, CHARTS, type Cell, type Chart } from "../../boards/chart.ts";
1111 import { runFor } from "../../boards/panel_data.ts";
12 import { runQuery, type Query, type Row } from "../../boards/query.ts";
12 import { runQuery, type Row } from "../../boards/query.ts";
1313 import { gaps, SYSTEM } from "../../boards/sampler.ts";
1414 import { SOURCES } from "../../boards/sources.ts";
15 import { BUCKET_MS, isBucket, MAX_PARAM, readParams, toQuery, type Bucket, type Params } from "../../boards/params.ts";
16 import { aggregateOption, ask, columnNote, columnOption, unitOf, GUIDE, parseSentence, QUESTIONS, sentence } from "../../boards/guide.ts";
1517 import { listAllBoards } from "../../actions/boards.ts";
1618 import type { Ctx } from "../../http/context.ts";
1719 import { notePhase, takeSessionTime, timePhase, timePhaseAsync } from "../../core/route_timing.ts";
@@ -20,59 +22,12 @@ import { html, when } from "../html.ts";
2022 import { page } from "../layout.ts";
2123 import { btn, card, chartBox, empty, field, selectField } from "../ui.ts";
2224
23 const BUCKET_MS = { minute: 60_000, hour: 3_600_000, day: 86_400_000 } as const;
2425 const AGGREGATES = ["count", "sum", "avg", "max"] as const;
2526 const OPS = ["eq", "ne", "lt", "lte", "gt", "gte", "starts"] as const;
26 const MAX_PARAM = 256;
27 const OP_LABEL = { eq: "eq — is", ne: "ne — is not", lt: "lt — below", lte: "lte — at most", gt: "gt — above", gte: "gte — at least", starts: "starts — begins with" } as const;
2728 /** The fields a panel is made from: its chart and its query. */
2829 export const PANEL_FIELDS = ["chart", "source", "aggregate", "column", "by", "series", "from", "to", "fcolumn", "fop", "fvalue"] as const;
2930
30 type Bucket = keyof typeof BUCKET_MS;
31 const isBucket = (s: string): s is Bucket => Object.hasOwn(BUCKET_MS, s);
32 const isChart = (s: string): s is Chart => (CHARTS as readonly string[]).includes(s);
33
34 /** What the URL asks for: plain strings, bounded; `runQuery` judges them. */
35 export function readParams(search: URLSearchParams) {
36 const get = (k: string, d = "") => (search.get(k) ?? d).slice(0, MAX_PARAM);
37 const chart = get("chart", "line");
38 return {
39 chart: isChart(chart) ? chart : "line",
40 source: get("source", "runs"),
41 aggregate: get("aggregate", "count"),
42 column: get("column"),
43 by: get("by", "day"),
44 series: get("series"),
45 from: get("from"),
46 to: get("to"),
47 fcolumn: get("fcolumn"),
48 fop: get("fop", "eq"),
49 fvalue: get("fvalue"),
50 gcolumn: get("gcolumn"),
51 gvalue: get("gvalue"),
52 rows: get("rows") === "1" ? "1" : "",
53 };
54 }
55 export type Params = ReturnType<typeof readParams>;
56
57 export function toQuery(p: Params): Query {
58 const src = Object.hasOwn(SOURCES, p.source) ? SOURCES[p.source]! : undefined;
59 const q: Query = { source: p.source, aggregate: p.aggregate };
60 if (p.column !== "") q.column = p.column;
61 if (p.from !== "") q.from = p.from;
62 if (p.to !== "") q.to = p.to;
63 q.groupBy = isBucket(p.by) ? { bucket: p.by } : p.by;
64 if (p.series !== "") q.series = p.series;
65 const filter: NonNullable<Query["filter"]> = [];
66 const add = (column: string, op: string, value: string) => {
67 const numeric = src?.columns[column] === "number";
68 filter.push({ column, op, value: numeric ? Number(value) : value });
69 };
70 if (p.fcolumn !== "") add(p.fcolumn, p.fop, p.fvalue);
71 if (p.gcolumn !== "") add(p.gcolumn, "eq", p.gvalue);
72 if (filter.length > 0) q.filter = filter;
73 return q;
74 }
75
7631 /** Rows as the cells a chart draws: a null group or series reads "all"/"none". */
7732 export const cellsOf = (rows: readonly Row[], hrefs: readonly (string | null)[] = []): Cell[] =>
7833 rows.map((r, i) => ({
@@ -151,15 +106,15 @@ export function queryFields(p: Params) {
151106 ];
152107 return html`
153108 ${sel("chart", "Chart", CHARTS.map((c) => [c, CHART_NAMES[c]]), p.chart)}
154 ${sel("source", "Source", Object.keys(SOURCES).map((s) => [s, s]), p.source)}
155 ${sel("aggregate", "Aggregate", AGGREGATES.map((a) => [a, a]), p.aggregate)}
156 ${sel("column", "Column", colOpts([["", "(rows)"]], (c) => c, (c) => src!.columns[c] === "number"), p.column, "Column to aggregate")}
157 ${sel("by", "Group by", [...(Object.keys(BUCKET_MS) as Bucket[]).map((b): [string, string] => [b, `per ${b}`]), ...colOpts([], (c) => `by ${c}`)], p.by)}
109 ${sel("source", "Source", Object.keys(SOURCES).map((s) => [s, GUIDE[s]?.name ?? s]), p.source)}
110 ${sel("aggregate", "Aggregate", AGGREGATES.map((a) => aggregateOption(p.source, a, p.column)), p.aggregate)}
111 ${sel("column", "Column", colOpts([["", "(rows)"]], (c) => columnOption(p.source, c), (c) => src!.columns[c] === "number"), p.column, "Column to aggregate")}
112 ${sel("by", "Group by", [...(Object.keys(BUCKET_MS) as Bucket[]).map((b): [string, string] => [b, `per ${b}`]), ...colOpts([], (c) => `by ${columnOption(p.source, c)}`)], p.by)}
158113 ${sel("series", "Series", colOpts([["", "(one series)"]], (c) => `split by ${c}`), p.series, "Stacked area, heat strip, table")}
159114 ${text("from", "From", p.from, "from 2026-10-01")}
160115 ${text("to", "To", p.to, "to 2026-10-08")}
161 ${sel("fcolumn", "Filter column", colOpts([["", "(no filter)"]], (c) => c), p.fcolumn)}
162 ${sel("fop", "Filter operator", OPS.map((o) => [o, o]), p.fop)}
116 ${sel("fcolumn", "Filter column", colOpts([["", "(no filter)"]], (c) => columnOption(p.source, c)), p.fcolumn)}
117 ${sel("fop", "Filter operator", OPS.map((o) => [o, OP_LABEL[o]]), p.fop)}
163118 ${text("fvalue", "Filter value", p.fvalue, "value")}
164119 `;
165120 }
@@ -189,6 +144,9 @@ export async function boardsPage(ctx: Ctx): Promise<Response> {
189144 const user = ctx.auth.user;
190145 if (user?.isAdmin !== true || user.kind === "agent") return notFoundPage(ctx);
191146 notePhase(ctx.request, "session", takeSessionTime(ctx.request));
147 const asked = ctx.url.searchParams.get("ask");
148 const fill = asked ? parseSentence(asked, Date.now()) : null;
149 if (fill) return new Response(null, { status: 303, headers: { location: link({ ...readParams(new URLSearchParams(fill)), ...fill } as Params) } });
192150 const p = readParams(ctx.url.searchParams);
193151 const src = Object.hasOwn(SOURCES, p.source) ? SOURCES[p.source]! : undefined;
194152 const answer = timePhase(ctx.request, "query", () =>
@@ -202,10 +160,10 @@ export async function boardsPage(ctx: Ctx): Promise<Response> {
202160 html`<ul class="list">
203161 ${Object.entries(SOURCES).map(
204162 ([name, s]) => html`<li class="row">
205 <a href="${link({ ...p, source: name, by: "day", column: "", series: "", fcolumn: "", fvalue: "", gcolumn: "", gvalue: "", rows: "" })}"><strong>${name}</strong></a>
206 <span class="muted small mono">${Object.entries(s.columns).map(([c, t]) => `${c}:${t}`).join(" ")}</span>
207 ${s.docs && name === p.source
208 ? html`<dl class="small">${Object.entries(s.docs).map(([c, d]) => html`<dt class="mono">${c}</dt><dd class="muted">${d}</dd>`)}</dl>`
163 <a href="${link({ ...p, source: name, aggregate: "", by: "", column: "", series: "", fcolumn: "", fop: "", fvalue: "", gcolumn: "", gvalue: "", rows: "" })}"><strong>${GUIDE[name]?.name ?? name}</strong></a>
164 <span class="muted small mono">${name}</span>
165 ${name === p.source
166 ? html`<dl class="small">${Object.entries(s.columns).map(([c, t]) => html`<dt class="mono">${c} <span class="muted">(${t})</span></dt><dd class="muted">${columnNote(name, c)}${unitOf(name, c) === "" ? "" : ` (${unitOf(name, c)})`}</dd>`)}</dl>`
209167 : null}
210168 </li>`,
211169 )}
@@ -213,6 +171,23 @@ export async function boardsPage(ctx: Ctx): Promise<Response> {
213171 { head: html`<h2>Sources</h2>`, pad: false },
214172 );
215173
174 const now = Date.now();
175 const askLink = (fill: Record<string, string>) => link({ ...readParams(new URLSearchParams(fill)), ...fill } as Params);
176 // A blank box (Show pressed with nothing typed) is no sentence, not an unreadable one.
177 const unread = asked !== null && asked.trim() !== "";
178 const questions = card(
179 html`<form method="get" action="/boards/data" class="bar">
180 ${field({ label: "Ask in a sentence", name: "ask", id: "f-ask", placeholder: "memory over the last 3 days", maxlength: 200, ...(unread ? { error: "I could not read that. Try a topic (cpu, memory, spend, deploys, slowest pages) and a window (last 3 days)." } : {}) })}
181 ${btn("Show", { variant: "ghost" })}
182 </form>
183 <ul class="list">
184 ${QUESTIONS.filter((q) => q.featured === true || q.source === p.source).map(
185 (q) => html`<li class="row"><a href="${askLink(ask(q, now))}"><strong>${q.label}</strong></a> <span class="muted small">${GUIDE[q.source]?.name ?? q.source}</span></li>`,
186 )}
187 </ul>`,
188 { head: html`<h2>Start from a question</h2>` },
189 );
190
216191 const form = card(
217192 html`<form method="get" action="/boards/data" class="bar">
218193 ${queryFields(p)}
@@ -232,7 +207,7 @@ export async function boardsPage(ctx: Ctx): Promise<Response> {
232207 const split = answer.rows[0]?.series !== undefined;
233208 const chart =
234209 p.rows === ""
235 ? chartBox(p.chart, points, `${p.aggregate} of ${p.source}`)
210 ? chartBox(p.chart, points, GUIDE[p.source]?.name ?? p.source)
236211 : html`<p class="muted small">The rows behind a point, as the values of ${cols[0] ?? ""} with how many rows have each. <a href="${link({ ...p, rows: "" })}">Back to the chart</a>.</p>`;
237212 result = html`${chart}
238213 ${when(p.rows !== "" || p.chart !== "table", () => html`<table class="table mt">
@@ -263,7 +238,7 @@ export async function boardsPage(ctx: Ctx): Promise<Response> {
263238 )}
264239 <input type="hidden" name="w" value="1" /><input type="hidden" name="h" value="1" />
265240 ${selectField({ label: "Board", name: "slug", id: "f-pin-slug", options: boards.map((b) => ({ value: b.slug, label: b.board.title })) })}
266 ${field({ label: "Panel title", name: "title", id: "f-pin-title", required: true, maxlength: 80, value: `${p.aggregate} of ${p.source}` })}
241 ${field({ label: "Panel title", name: "title", id: "f-pin-title", required: true, maxlength: 80, value: sentence(p).slice(0, 80) })}
267242 ${btn("Pin to board")}
268243 </form>`,
269244 { head: html`<h2>Pin this query</h2>` },
@@ -287,7 +262,7 @@ export async function boardsPage(ctx: Ctx): Promise<Response> {
287262 ctx,
288263 title: "Boards",
289264 body: html`<div class="page-head"><h1>Data</h1>${btn("All boards", { href: "/boards", variant: "ghost" })}</div>
290 ${form} ${absent} ${card(result, { head: html`<h2>${p.source}</h2>` })} ${pin} ${sources}`,
265 ${questions} ${card(result, { head: html`<h2>${GUIDE[p.source]?.name ?? p.source}</h2>`, foot: html`${sentence(p)}`, id: "says" })} ${form} ${absent} ${pin} ${sources}`,
291266 });
292267 notePhase(ctx.request, "render", Math.max(0, performance.now() - rendering - boardsMs));
293268 return response;
test/boards-guide.test.ts added +191
@@ -0,0 +1,191 @@
1 /** The data explorer starts from a question: plain names, defaults, ready questions, a sentence box. */
2 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp } from "../src/app.ts";
7 import { aggregateOption, ask, columnOption, defaultsFor, GUIDE, NOTES, parseSentence, QUESTIONS, sentence, unitOf } from "../src/boards/guide.ts";
8 import { runQuery } from "../src/boards/query.ts";
9 import { SOURCES } from "../src/boards/sources.ts";
10 import { createToken } from "../src/core/tokens.ts";
11 import { createUser } from "../src/core/users.ts";
12 import { serve } from "../src/http/server.ts";
13 import { readParams, toQuery } from "../src/boards/params.ts";
14
15 const NOW = Date.parse("2026-11-08T12:00:30Z");
16
17 describe("guide", () => {
18 test("every source has a plain name and a default that runs", () => {
19 expect(Object.keys(GUIDE).sort()).toEqual(Object.keys(SOURCES).sort());
20 for (const [s, g] of Object.entries(GUIDE)) {
21 if (g.value) expect(SOURCES[s]!.columns[g.value]).toBe("number");
22 if (g.filter) expect(SOURCES[s]!.columns[g.filter.column]).toBeDefined();
23 }
24 expect(GUIDE.machine!.name).toBe("Machine readings");
25 });
26 test("samples default to an average over time, events to a count per day", () => {
27 expect(defaultsFor("machine")).toMatchObject({ aggregate: "avg", column: "cpu_cores", by: "day" });
28 expect(defaultsFor("runs")).toMatchObject({ aggregate: "count", column: "", by: "day" });
29 const p = readParams(new URLSearchParams("source=machine"));
30 expect(p).toMatchObject({ aggregate: "avg", column: "cpu_cores", fcolumn: "target", fvalue: "forge" });
31 // an explicit aggregate is never overridden
32 expect(readParams(new URLSearchParams("source=machine&aggregate=count")).aggregate).toBe("count");
33 });
34 test("every ready question is a query the engine accepts", async () => {
35 const dir = await mkdtemp(join(tmpdir(), "forge-boards-questions-"));
36 const app = createApp({ dataDir: dir, port: 0 });
37 const db = app.db;
38 try {
39 for (const q of QUESTIONS) {
40 const f = ask(q, NOW);
41 const p = readParams(new URLSearchParams(f));
42 expect(runQuery(db, toQuery(p), { repoIds: "all" }).ok).toBe(true);
43 }
44 for (const label of ["CPU over the last week", "Slowest pages today", "Who used the forge's time", "Apps shipped per day"]) {
45 expect(QUESTIONS.some((q) => q.label === label && q.featured)).toBe(true);
sentinelcrew outdated

dataDir "" resolves to the cwd, so this writes app state into the checkout and the app is never closed. Use a mkdtemp dir and close it.
46 }
47 } finally {
48 app.close();
49 await rm(dir, { recursive: true, force: true });
50 }
51 });
52 test("a chart says what it shows in one line", () => {
53 const p = readParams(new URLSearchParams("source=machine&aggregate=avg&column=cpu_cores&by=day&series=target"));
54 expect(sentence({ ...p, fcolumn: "", fvalue: "" })).toBe("Average CPU cores per day, by machine.");
55 expect(sentence(readParams(new URLSearchParams("source=runs&by=state")))).toBe("Count of runs by state.");
56 });
57 test("the sentence says the window's bounds: from and to, since, until", () => {
58 const at = (q: string) => sentence(readParams(new URLSearchParams(`source=runs&by=state&${q}`)));
59 expect(at("from=2026-11-01T00:00:00Z&to=2026-11-08T00:00:00Z")).toBe("Count of runs by state, 2026-11-01 to 2026-11-08.");
60 expect(at("from=2026-11-01")).toBe("Count of runs by state, since 2026-11-01.");
61 expect(at("to=2026-11-08T00:00:00Z")).toBe("Count of runs by state, until 2026-11-08.");
62 });
63 test("sum, average and highest name the column they aggregate and its unit", () => {
64 expect(aggregateOption("machine", "avg", "memory_bytes")).toEqual(["avg", "average of memory bytes (bytes)"]);
65 expect(aggregateOption("spend", "sum")).toEqual(["sum", "total of cost USD (US dollars)"]);
66 expect(aggregateOption("machine", "max")[1]).toBe("highest of CPU cores (cores)");
67 expect(aggregateOption("runs", "sum")[1]).toBe("total of the column picked below");
68 });
69 test("a sentence fills the form without a model", () => {
70 const m = parseSentence("memory over the last 3 days", NOW)!;
71 expect(m).toMatchObject({ source: "machine", column: "memory_bytes", aggregate: "avg", by: "hour", from: "2026-11-05T12:00:00.000Z" });
72 expect(parseSentence("Slowest pages today", NOW)).toMatchObject({ source: "reach", by: "route", from: "2026-11-08T00:00:00.000Z" });
73 expect(parseSentence("how much did we spend last week", NOW)).toMatchObject({ source: "spend", aggregate: "sum", from: "2026-11-01T12:00:00.000Z" });
74 expect(parseSentence("deploys per hour", NOW)).toMatchObject({ source: "deploys", by: "hour" });
75 expect(parseSentence("the meaning of life", NOW)).toBeNull();
76 });
77 });
78
79 describe("meaning and unit", () => {
80 test("every number column of every source names its unit, and the picker shows it", () => {
81 for (const [s, src] of Object.entries(SOURCES)) {
82 for (const [c, t] of Object.entries(src.columns)) {
83 if (t !== "number") continue;
84 expect(unitOf(s, c), `${s}.${c}`).not.toBe("");
85 expect(columnOption(s, c), `${s}.${c}`).toContain(`(${unitOf(s, c)})`);
86 }
87 }
88 expect(columnOption("machine", "memory_bytes")).toContain("(bytes)");
89 expect(columnOption("spend", "cost_usd")).toContain("(US dollars)");
90 });
91 test("every shared note is for a column some source has", () => {
92 const all = new Set(Object.values(SOURCES).flatMap((s) => Object.keys(s.columns)));
93 for (const [c, note] of Object.entries(NOTES)) {
94 expect(all.has(c), c).toBe(true);
95 expect(note, c).not.toBe("");
96 }
97 });
98 test("yesterday is that calendar day, not a sliding window", () => {
99 expect(parseSentence("runs yesterday", NOW)).toMatchObject({ from: "2026-11-07T00:00:00.000Z", to: "2026-11-08T00:00:00.000Z" });
100 });
101 });
102
103 describe("/boards/data", () => {
104 let dir = "", app: ReturnType<typeof createApp>, servers: Awaited<ReturnType<typeof serve>>, bearer = "";
105 beforeAll(async () => {
106 dir = await mkdtemp(join(tmpdir(), "forge-boards-guide-"));
107 app = createApp({ dataDir: dir, port: 0 });
108 servers = await serve(app);
109 const r = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
110 if (!r.ok) throw new Error(r.error);
111 bearer = createToken(app.db, r.value.id, "t", "admin").token;
112 });
113 afterAll(async () => {
114 servers.stop();
115 app.close();
116 await rm(dir, { recursive: true, force: true });
117 });
118 const get = (path: string, init: RequestInit = {}) => fetch(servers.url + path, { ...init, headers: { authorization: `Bearer ${bearer}` } });
119
120 test("the page names sources plainly, lists ready questions and says what the chart shows", async () => {
121 const body = await (await get("/boards/data?source=machine")).text();
122 expect(body).toContain("Machine readings");
123 expect(body).toContain("Start from a question");
124 expect(body).toContain("CPU over the last week");
125 expect(body).toContain("Apps shipped per day");
126 expect(body).toContain("Average CPU cores per day, where target is forge.");
127 expect(body).toContain("counts readings, not time");
128 expect(body).toMatch(/cpu_cores \(cores\) — cores in use/);
129 });
130 test("count on a samples source says what it counts, in the rendered page", async () => {
131 const body = await (await get("/boards/data?source=machine")).text();
132 expect(body).toMatch(/<option[^>]*value="count"[^>]*>\s*count of 15-second readings \(counts readings, not time\)\s*<\/option>/);
133 const events = await (await get("/boards/data?source=runs")).text();
134 expect(events).toMatch(/value="count"[^>]*>\s*count of runs\s*</);
135 });
136 test("a source-list link lands on that source's default chart", async () => {
137 const page = await (await get("/boards/data?source=runs&aggregate=count&by=state")).text();
138 const href = page.match(/href="(\/boards\/data\?[^"]*)"><strong>Machine readings<\/strong>/)![1]!.replaceAll("&amp;", "&");
139 const landed = await (await get(href)).text();
140 expect(landed).toContain("Average CPU cores per day, where target is forge.");
141 expect(landed).not.toContain("refused");
142 });
143 test("the page has one primary button", async () => {
144 const body = await (await get("/boards/data?source=machine")).text();
145 expect(body.match(/data-variant="primary"/g)?.length).toBe(1);
146 });
147 test("a ready question opens a working chart from its link", async () => {
148 const page = await (await get("/boards/data?source=reach")).text();
149 const href = page.match(/href="(\/boards\/data\?[^"]*)"><strong>Slowest pages today/)![1]!.replaceAll("&amp;", "&");
150 const res = await get(href);
151 expect(res.status).toBe(200);
152 expect(await res.text()).not.toContain("refused");
153 });
154 test("a sentence redirects to a link that keeps every choice", async () => {
155 const res = await get("/boards/data?ask=memory+over+the+last+3+days", { redirect: "manual" });
156 expect(res.status).toBe(303);
157 const loc = new URL(res.headers.get("location")!, "http://x");
158 expect(loc.searchParams.get("source")).toBe("machine");
159 expect(loc.searchParams.get("column")).toBe("memory_bytes");
160 expect(loc.searchParams.get("ask")).toBeNull();
161 });
162 test("a sentence it cannot read says so and still draws the form", async () => {
163 const res = await get("/boards/data?ask=zzz+qqq");
164 expect(res.status).toBe(200);
165 expect(await res.text()).toContain("I could not read that");
166 });
167 test("an empty box is no sentence: the page draws without an error", async () => {
168 for (const q of ["ask=", "ask=+++"]) {
169 const res = await get(`/boards/data?${q}`);
170 expect(res.status).toBe(200);
171 expect(await res.text()).not.toContain("I could not read that");
172 }
173 });
174 test("the one-line sentence is said once, in the result card's foot", async () => {
175 const body = await (await get("/boards/data?source=machine")).text();
176 expect(body.split("Average CPU cores per day, where target is forge.").length - 1).toBe(1);
177 expect(body).toMatch(/<div class="card-foot">\s*Average CPU cores per day/);
178 });
179 test("a member who is not a site admin gets 404 for a sentence, and no redirect", async () => {
180 const r = await createUser(app.db, { origin: "seed", trusted: true, username: "bob", email: "[email protected]", password: "a-good-passphrase" });
181 if (!r.ok) throw new Error(r.error);
182 const member = createToken(app.db, r.value.id, "t", "admin").token;
183 for (const q of ["ask=memory+over+the+last+3+days", "ask=zzz"]) {
184 const res = await fetch(`${servers.url}/boards/data?${q}`, { redirect: "manual", headers: { authorization: `Bearer ${member}` } });
185 expect(res.status).toBe(404);
186 expect(res.headers.get("location")).toBeNull();
187 }
188 const anon = await fetch(`${servers.url}/boards/data?ask=memory`, { redirect: "manual" });
189 expect(anon.status).toBe(404);
190 });
191 });
test/boards-page.test.ts +1 −1
@@ -101,7 +101,7 @@ describe("/boards/data", () => {
101101 const body = await (await get("/boards/data?source=runs&by=day", "admin")).text();
102102 expect(body).not.toContain("boards-result");
103103 expect(body).toMatch(/<table\s+class="table mt"/);
104 expect(body).toContain("<h2>runs</h2>");
104 expect(body).toContain("<h2>Crew and build runs</h2>");
105105 expect(body).toContain('<th scope="col" class="num">');
106106 expect(body).toContain("2026-10-02");
107107 });
test/boards-selectors.test.ts +1 −1
@@ -53,7 +53,7 @@ test("every query selector has a visible label and sits outside .row-actions", a
5353 const res = await fetch(`${servers.url}/boards/data`, { headers: { authorization: `Bearer ${token}` } });
5454 expect(res.status).toBe(200);
5555 const html = await res.text();
56 const form = html.match(/<form method="get" action="\/boards\/data"[^>]*>([\s\S]*?)<\/form>/);
56 const form = [...html.matchAll(/<form method="get" action="\/boards\/data"[^>]*>([\s\S]*?)<\/form>/g)].find((m) => m[0].includes('name="chart"'));
5757 expect(form).not.toBeNull();
5858 expect(form![0]).not.toContain("row-actions");
5959 expect(form![0]).toContain('class="bar"');
test/palette-roster.test.ts +1
@@ -81,6 +81,7 @@ test("a site-admin person is offered the whole roster", () => {
8181 ["signins", "Signed in", "your browsers and tokens; d signs a browser out"],
8282 ["sentinel", "Sentinel", "the forge's open faults"],
8383 ["boards", "Boards", "the forge's boards and their panels"],
84 ["ask-data", "Data explorer", "start from a question about the forge's data"],
8485 ["bugs", "Bugs", "open bugs and the bug clock"],
8586 ["home-pins", "Pinned panels", "the panels pinned to home: reorder and unpin"],
8687 ["now", "Now", "what check, review, land and live count, with the rows"],
test/tui-ask-data.test.ts added +82
@@ -0,0 +1,82 @@
1 /** The data explorer in the terminal (#9440): the page's gate, questions, sentence and one-line description. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { createUser, type User } from "../src/core/users.ts";
8 import { createForgeApp, forgeCommands, Session } from "../src/tui/forge.ts";
9 import { exploreRows, resultLines } from "../src/tui/screens/ask_data.ts";
10 import { entry } from "../src/tui/screens/ask_data.ts";
11 import { readParams } from "../src/boards/params.ts";
12 import { headlessTerminal } from "../src/tui/term/terminal.ts";
13 import { quietLogs } from "./helpers/globals.ts";
14 import { settleApp } from "./helpers/tui.ts";
15
16 quietLogs();
17 let dir: string, app: App, carol: User, bob: User;
18 const identity = (user?: User) => ({ sshUsername: user?.username ?? "git", remoteAddress: "127.0.0.1", fingerprint: "SHA256:test", user });
19
20 beforeAll(async () => {
21 dir = await mkdtemp(join(tmpdir(), "forge-tui-askdata-"));
22 app = createApp({ dataDir: dir, port: 0 });
23 const mk = async (username: string, first = false) => {
24 const r = await createUser(app.db, { origin: "seed", trusted: true, username, email: `${username}@forge.local`, password: "a-good-passphrase", adminIfFirst: first });
25 if (!r.ok) throw new Error(r.error);
26 return r.value;
27 };
28 carol = await mk("carol", true);
29 bob = await mk("bob");
30 }, 30_000);
31 afterAll(async () => {
32 app.close();
33 await rm(dir, { recursive: true, force: true });
34 });
35
36 async function typed(user: User | undefined, keys: string[]): Promise<string> {
37 const terminal = headlessTerminal(120, 30);
38 const { app: tui, session } = createForgeApp(app, identity(user), { terminal, exitHooks: false });
39 tui.start();
40 entry.run(session, tui.ctx);
41 await settleApp(tui);
42 for (const k of keys) {
43 terminal.feed(k);
44 await settleApp(tui);
45 }
46 const text = tui.screen.toText();
47 tui.stop?.();
48 return text;
49 }
50
51 test("an admin person sees the plain names and ready questions", async () => {
52 const text = await typed(carol, []);
53 expect(text).toContain("CPU over the last week");
54 expect(text).toContain("Slowest pages today");
55 const rows = exploreRows(new Session(app, identity(carol)), Date.now());
56 expect(rows.some((r) => r.title === "Machine readings")).toBe(true);
57 });
58
59 test("enter opens a chart with its one-line description", async () => {
60 const text = await typed(carol, ["\r"]);
61 expect(text).toContain("Average CPU cores per hour, where target is forge");
62 });
63
64 test("a sentence is read by parseSentence, and one it cannot read is an error and nothing more", async () => {
65 const ok = await typed(carol, ["/", ..."memory over the last 3 days", "\r"]);
66 expect(ok).toContain("Average memory bytes per hour");
67 const bad = await typed(carol, ["/", ..."zzz qqq", "\r"]);
68 expect(bad).toContain("I could not read that");
69 });
70
71 test("nobody else has the door, the rows or a query", async () => {
72 for (const who of [bob, undefined]) {
73 const session = new Session(app, identity(who));
74 expect(exploreRows(session, Date.now())).toEqual([]);
75 expect(resultLines(session, readParams(new URLSearchParams("source=runs")))).toEqual([]);
76 const { app: tui, session: s2 } = createForgeApp(app, identity(who), { terminal: headlessTerminal(80, 24), exitHooks: false });
77 expect(forgeCommands(s2)(tui.ctx).map((c) => c.id)).not.toContain("ask-data");
78 tui.stop?.();
79 expect(await typed(who, [])).not.toContain("CPU over the last week");
80 }
81 expect(resultLines(new Session(app, identity(carol)), readParams(new URLSearchParams("source=runs"))).length).toBeGreaterThan(0);
82 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs