Public: anyone can read this; the crew can change it.

The data explorer offers a filter's values as a list to pick from #13118

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13114-the-data-explorer-offers-a-filter-s-valu → main 5 commits fork 6 threads · 6 unresolved

  • Merged. scout/13114-the-data-explorer-offers-a-filter-s-valu was merged into main as a7a3a16.

Review

Sign in to review this change.
6 files changed +266 −19
docs/features/data-explorer.md +2 −1
@@ -9,5 +9,6 @@
99 - **Ask in a sentence**: type a topic and a window into "Ask in a sentence" at the top of the page (or press `/` in the terminal's Data explorer, in the palette after Boards) and press Show. Topics: cpu, memory, temperature, load, slowest pages, visitors, spend, tokens, shipped, deploys, pulls, reviews, sign-ins, runs, requests. Windows: "last 3 days", "last week", "today", "yesterday" (that calendar day, UTC); "per hour", "per day" and, for the machine, "by machine". "memory over the last 3 days" redirects (303) to the filled form, whose URL keeps every choice.
1010 - **A phrase the parser cannot read gets an error message and nothing more**: "I could not read that", with the topics to try. No model is asked and nothing is guessed.
1111 - **Both doors**: the terminal screen (`src/tui/screens/ask_data.ts`) lists the same questions and plain source names, shows the one-line description and the rows, and takes the same sentences. Both are for a site admin who is a person; others are not offered it and read nothing.
12 - **Filter values from a list**: in the data browser, choosing a column (with no text typed) lists up to 20 of its commonest values as links, folded under the rows; one filters to exactly that value (an equality match, not "contains"). The terminal screen has it too: `v` on a table asks for a column and lists the same values (same mask, same gate). The page says when a filter is an equals. Each list spends one query from the same per-person budget as a SQL run (`sql_window.ts`), and is counted over the first 10,000 rows so one read stays cheap; a masked column offers nothing.
1213
13 Not done: the crew/model fallback for phrasings the parser misses (#9569), and picking filter *values* from a list (they stay typed).
14 Not done: the crew/model fallback for phrasings the parser misses (#9569).
outdated

This done-feature bullet is placed after the 'Not done:' line; move it above it.

src/data/browse.ts +53 −9
@@ -21,6 +21,8 @@ export interface RowFilter {
2121 /** Rows where any column (or just `column`) contains `text`. */
2222 readonly text: string;
2323 readonly column?: string;
24 /** With `column`: the whole value equals `text`, rather than containing it. */
25 readonly exact?: boolean;
2426 }
2527
2628 export interface RowsOptions {
@@ -144,15 +146,19 @@ export function rows(
144146 const searchable = (col === undefined ? columns : [col]).filter(
145147 (c) => !masked.has(c),
146148 );
147 const like = opts.filter.text.replace(/[\\%_]/g, (m) => `\\${m}`);
148 where.push(
149 searchable.length === 0
150 ? "0"
151 : searchable
152 .map((c) => `CAST(${quote(c)} AS TEXT) LIKE ? ESCAPE '\\'`)
153 .join(" OR "),
154 );
155 for (let i = 0; i < searchable.length; i++) args.push(`%${like}%`);
149 if (opts.filter.exact && col !== undefined) {
150 // A picked value: the whole cell equals it.
151 where.push(searchable.length === 0 ? "0" : `CAST(${quote(col)} AS TEXT) = ?`);
152 if (searchable.length > 0) args.push(opts.filter.text);
outdated

The else { body is not indented, and the block is hard to read. Re-indent it or restructure with an early branch.

153 } else {
154 const like = opts.filter.text.replace(/[\\%_]/g, (m) => `\\${m}`);
155 where.push(
156 searchable.length === 0
157 ? "0"
158 : searchable.map((c) => `CAST(${quote(c)} AS TEXT) LIKE ? ESCAPE '\\'`).join(" OR "),
159 );
160 for (let i = 0; i < searchable.length; i++) args.push(`%${like}%`);
161 }
156162 }
157163 let order = "";
158164 if (opts.order) {
@@ -188,3 +194,41 @@ export function rows(
188194 };
189195 });
190196 }
197
198 /** Most values the list offers. */
199 export const TOP_VALUES = 20;
200 /** Rows read to count them: the list is the commonest among the first, cheap on any table. */
201 export const TOP_SCAN_ROWS = 10_000;
202 /** Longest value offered; longer ones are not a pick-list's business. */
203 export const TOP_VALUE_CHARS = 80;
204
205 export interface TopValue {
206 readonly value: string;
207 readonly count: number;
208 }
209
210 /**
211 * Up to `TOP_VALUES` of the commonest values of `table.column`, among its first `TOP_SCAN_ROWS`
212 * rows. Undefined when the table or column does not exist, or the column is masked: a masked
213 * column offers nothing, since each value would be a secret shown.
214 */
215 export function topValues(source: Source, table: string, column: string): TopValue[] | undefined {
216 return withDb(source, (db) => {
217 if (!tableNames(db).includes(table)) return undefined;
218 const columns = db
219 .query<{ name: string }, []>(`PRAGMA table_info(${quote(table)})`)
220 .all()
221 .map((c) => c.name);
222 if (!columns.includes(column)) return undefined;
223 if (maskPlan(db, source.id).get(table)?.has(column)) return undefined;
224 const c = quote(column);
225 return db
226 .query<{ v: string; n: number }, []>(
227 `SELECT v, count(*) AS n FROM (SELECT CAST(${c} AS TEXT) AS v FROM ${quote(table)} LIMIT ${TOP_SCAN_ROWS})
228 WHERE v IS NOT NULL AND v <> '' AND length(v) <= ${TOP_VALUE_CHARS}
229 GROUP BY v ORDER BY n DESC, v LIMIT ${TOP_VALUES}`,
230 )
231 .all()
232 .map((r) => ({ value: r.v, count: r.n }));
233 });
234 }
src/tui/screens/data.ts +55 −5
@@ -14,7 +14,7 @@ import { query, type QueryResult } from "../../data/query.ts";
1414 import { cellLine, rowObject, type Cell } from "../../data/shape.ts";
1515 import { listSources, maySeeData } from "../../data/sources.ts";
1616 import { sqlWindow } from "../../data/sql_window.ts";
17 import { PAGE_SIZE } from "../../data/browse.ts";
17 import { PAGE_SIZE, topValues, type TopValue } from "../../data/browse.ts";
1818 import { exportCsv } from "../../data/export_csv.ts";
1919 import { filtered, newReads, opens, pageCount, parentKey, visible, type Node } from "../../data/tree.ts";
2020 import type { Ctx, Screen } from "../app/app.ts";
@@ -85,11 +85,45 @@ function resultScreen(ctx: Ctx, sql: string, r: Extract<QueryResult, { ok: true
8585 };
8686 }
8787
88 /** A column's commonest values and how often each occurs. */
89 function valuesScreen(ctx: Ctx, table: string, column: string, found: readonly TopValue[]): Screen {
90 let at = 0;
91 return {
92 title: "Common values",
93 keys: [binding(["j", "k"], "move"), binding("y", "yank value"), binding("esc", "back")],
94 render(area, s) {
95 const c = ctx.theme.c;
96 const note = `${table}.${column}: commonest of the first 10,000 rows`;
97 s.text(area.x + 1, area.y, truncate(note, area.w - 2), style(c.mutedFg, c.bg), area.x + area.w);
98 if (found.length === 0) s.text(area.x + 1, area.y + 2, "No values to offer.", style(c.mutedFg, c.bg), area.x + area.w);
99 const h = Math.max(1, area.h - 3);
100 const from = Math.max(0, Math.min(at - h + 1, found.length - h));
101 found.slice(from, from + h).forEach((v, i) => {
102 const on = from + i === at;
103 s.text(area.x + 1, area.y + 2 + i, truncate(`${on ? "> " : " "}${v.value} ${v.count}`, area.w - 2), style(on ? c.accent : c.fg, c.bg), area.x + area.w);
104 });
105 },
106 key(e) {
107 if (e.ctrl || e.alt) return false;
108 if (e.key === "esc" || e.key === "q") ctx.pop();
109 else if (e.key === "j" || e.key === "down") at = Math.min(Math.max(0, found.length - 1), at + 1);
110 else if (e.key === "k" || e.key === "up") at = Math.max(0, at - 1);
111 else if (e.key === "y") {
112 const v = found[at];
113 if (!v) return true;
114 ctx.copy(v.value);
115 ctx.toast("Value sent to your clipboard.");
116 } else return false;
117 return true;
118 },
119 };
120 }
121
88122 export function dataScreen(session: Session, ctx: Ctx): Screen {
89123 const open = new Set<string>();
90124 let at = 0;
91125 let filter = "";
92 let mode: "filter" | "sql" | undefined;
126 let mode: "filter" | "sql" | "values" | undefined;
93127 let line = typing(FILTER_MAX);
94128
95129 // Built once and rebuilt only when `open`, a page, the filter or a reload changes it.
@@ -145,6 +179,17 @@ export function dataScreen(session: Session, ctx: Ctx): Screen {
145179 );
146180 }
147181
182 /** The commonest values of `column` in the table under the cursor: the page's pick-list. */
183 function showValues(n: Node | undefined, column: string) {
184 const name = column.trim();
185 if (!n?.source || !n.table) return ctx.toast("Move onto a table or one of its rows first.", "danger");
186 const wait = session.user ? sqlWindow(session.app, session.user.id) : 1;
187 if (wait !== undefined) return ctx.toast(`Too many queries; try again in ${wait} s.`, "danger");
188 const found = name === "" ? undefined : topValues(n.source, n.table, name);
189 if (!found) return ctx.toast("No such column, or its values are sealed.", "danger");
190 ctx.push(valuesScreen(ctx, n.table, name, found));
191 }
192
148193 /**
149194 * The table under the cursor as CSV on the clipboard: the page's download, in the terminal's
150195 * size.
@@ -162,7 +207,7 @@ export function dataScreen(session: Session, ctx: Ctx): Screen {
162207 title: "Data",
163208 get keys() {
164209 return mode
165 ? [binding("enter", mode === "sql" ? "run" : "keep"), binding("esc", "cancel")]
210 ? [binding("enter", mode === "filter" ? "keep" : mode === "sql" ? "run" : "list"), binding("esc", "cancel")]
166211 : [
167212 binding(["j", "k"], "move"),
168213 binding(["h", "l"], "close / open"),
@@ -170,6 +215,7 @@ export function dataScreen(session: Session, ctx: Ctx): Screen {
170215 binding(":", "SQL"),
171216 binding("y", "yank row as JSON"),
172217 binding("e", "export table as CSV"),
218 binding("v", "common values of a column"),
173219 binding(["n", "p"], `page of ${PAGE_SIZE}`),
174220 binding("r", "reload"),
175221 binding("esc", "back"),
@@ -189,7 +235,7 @@ export function dataScreen(session: Session, ctx: Ctx): Screen {
189235 s.text(area.x + 1, area.y + 2 + i, truncate(`${" ".repeat(n.depth)}${mark}${n.label}`, area.w - 2), style(on ? c.accent : c.fg, c.bg), area.x + area.w);
190236 });
191237 if (mode) {
192 s.text(area.x + 1, area.y + area.h - 1, truncate(`${mode === "sql" ? ":" : "/"}${line.text}`, area.w - 2), style(c.fg, c.card), area.x + area.w);
238 s.text(area.x + 1, area.y + area.h - 1, truncate(`${mode === "sql" ? ":" : mode === "values" ? "column " : "/"}${line.text}`, area.w - 2), style(c.fg, c.card), area.x + area.w);
193239 }
194240 },
195241 key(e) {
@@ -204,7 +250,8 @@ export function dataScreen(session: Session, ctx: Ctx): Screen {
204250 filter = line.text;
205251 dirty();
206252 at = 0;
207 } else ask(line.text);
253 } else if (done === "values") showValues(current(), line.text);
254 else ask(line.text);
208255 } else if (!line.key(e)) return true;
209256 else if (mode === "filter") {
210257 filter = line.text;
@@ -234,6 +281,9 @@ export function dataScreen(session: Session, ctx: Ctx): Screen {
234281 } else if (e.key === "/") {
235282 mode = "filter";
236283 line = typing(FILTER_MAX, filter);
284 } else if (e.key === "v") {
285 mode = "values";
286 line = typing(FILTER_MAX);
237287 } else if (e.key === ":") {
238288 mode = "sql";
239289 line = typing(SQL_MAX, n?.table ? `SELECT * FROM "${n.table.replaceAll('"', '""')}" LIMIT 20` : "");
src/web/pages/data_view.ts +19 −4
@@ -7,7 +7,7 @@
77 * every view, SQL included, is an address that can be shared.
88 */
99
10 import { PAGE_SIZE, rows, tables, type RowsResult } from "../../data/browse.ts";
10 import { PAGE_SIZE, rows, tables, topValues, type RowsResult } from "../../data/browse.ts";
1111 import { query } from "../../data/query.ts";
1212 import type { Source } from "../../data/sources.ts";
1313 import { html, type Html } from "../html.ts";
@@ -130,12 +130,13 @@ export async function dataBody(view: DataView, q: URLSearchParams): Promise<Html
130130 const dir = q.get("dir") === "desc" ? "desc" : "asc";
131131 const text = (q.get("q") ?? "").slice(0, 200);
132132 const col = q.get("col") || undefined;
133 const exact = col !== undefined && q.get("exact") === "1";
133134 const pageNo = intOr0(q.get("page"));
134 const base = { source: source.id, table: tableName, sort, dir: sort ? dir : undefined, q: text, col };
135 const base = { source: source.id, table: tableName, sort, dir: sort ? dir : undefined, q: text, col, exact: exact ? "1" : undefined };
135136 let shown: RowsResult | undefined = rows(source, tableName, {
136137 page: pageNo,
137138 ...(sort ? { order: { column: sort, dir } } : {}),
138 ...(text !== "" ? { filter: { text, ...(col ? { column: col } : {}) } } : {}),
139 ...(text !== "" ? { filter: { text, ...(col ? { column: col } : {}), ...(exact ? { exact } : {}) } } : {}),
139140 });
140141 let note: string | null = null;
141142 if (!shown && (sort || text !== "")) {
@@ -145,6 +146,18 @@ export async function dataBody(view: DataView, q: URLSearchParams): Promise<Html
145146 }
146147 if (!shown) return undefined;
147148 const cols = shown.columns;
149 // The list is a read like a SQL run: it spends the same per-person budget (`sqlWait`).
150 const wanted = col !== undefined && text === "";
outdated

Principle 2: up to 20 link rows above the grid push the actual rows below the fold on a 360px phone. Put the list after the grid or fold it in a details element. Principle 1: the aria-label repeats the visible 'Common values of' line, so keep one of them. Also say that a picked value filters by equality, since the form's q box shows only the text and exact is lost on the next submit.

151 const wait = wanted ? view.sqlWait?.() : undefined;
152 const common = wanted && wait === undefined ? topValues(source, tableName, col) : undefined;
outdated

Link sets q=value, which is a substring filter: the shown count (exact) will not match the rows after clicking. Make it an exact match for picks or say 'contains'. Also the terminal data screen has no counterpart (both doors).

153 const picks =
154 common && common.length > 0
155 ? html`<details class="disclosure mt"><summary>Common values of ${col ?? ""}</summary><ul class="list">${common.map((v) =>
156 row({ href: url(path, { ...base, q: v.value, exact: "1" }), title: v.value, trail: html`${String(v.count)}` }),
157 )}</ul></details>`
158 : wait !== undefined
159 ? html`<p class="muted small">Too many queries to list common values; try again in ${String(wait)} s.</p>`
160 : null;
148161 const more = (shown.page + 1) * PAGE_SIZE < shown.total;
149162 const header = (c: string): Html => {
150163 const next = sort === c && dir === "asc" ? "desc" : "asc";
@@ -178,13 +191,15 @@ export async function dataBody(view: DataView, q: URLSearchParams): Promise<Html
178191 </div>
179192 </form>
180193 ${note ? html`<p class="muted small">${note}</p>` : null}
194 ${exact && text !== "" ? html`<p class="muted small">Rows where ${col ?? ""} equals “${text}”. Filtering again finds rows that contain the text.</p>` : null}
181195 ${shown.rows.length === 0 ? empty({ title: "No rows", note: text !== "" ? "Nothing matches that filter." : "This table is empty." }) : grid(cols, shown.rows, `${tableName} in ${source.label}`, header, true)}
182196 ${pager({
183197 note: `${shown.total} rows`,
184198 words: { prev: "Previous page", next: "Next page" },
185199 ...(shown.page > 0 ? { prev: url(path, { ...base, page: shown.page - 1 }) } : {}),
186200 ...(more ? { next: url(path, { ...base, page: shown.page + 1 }) } : {}),
187 })}`;
201 })}
202 ${picks}`;
188203 }
189204
190205 out = html`${out}
test/data-top-values.test.ts added +102
@@ -0,0 +1,102 @@
1 /**
2 * The explorer's pick-list of a column's commonest values:
3 * the list, the mask and the scan budget.
4 */
5 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
6 import { Database } from "bun:sqlite";
7 import { mkdtemp, rm } from "node:fs/promises";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { rows, TOP_SCAN_ROWS, TOP_VALUES, topValues } from "../src/data/browse.ts";
11 import type { Source } from "../src/data/sources.ts";
12 import { dataBody } from "../src/web/pages/data_view.ts";
13
14 describe("filter values as a list", () => {
15 let dir = "";
16 let source: Source;
17 beforeAll(async () => {
18 dir = await mkdtemp(join(tmpdir(), "forge-top-values-"));
19 const path = join(dir, "app.db");
20 const d = new Database(path);
21 d.exec("CREATE TABLE t (id INTEGER PRIMARY KEY, city TEXT, api_token TEXT, note TEXT)");
22 const ins = d.prepare("INSERT INTO t (city, api_token, note) VALUES (?, ?, ?)");
23 d.transaction(() => {
24 for (let i = 0; i < 30; i++) ins.run(`city-${String(i).padStart(2, "0")}`, `sekrit-${i}`, null);
25 for (let i = 0; i < 5; i++) ins.run("Paris", "sekrit-p", null);
26 for (let i = 0; i < 2; i++) ins.run("Rome", "sekrit-r", "x".repeat(200));
27 })();
28 d.close();
29 source = { id: "app:x:production:app.db", label: "Test", kind: "app", path } as unknown as Source;
30 });
31 afterAll(() => rm(dir, { recursive: true, force: true }));
32
33 test("lists at most 20 values, commonest first, ties by value", () => {
34 const v = topValues(source, "t", "city")!;
35 expect(v.length).toBe(TOP_VALUES);
36 expect(v[0]).toEqual({ value: "Paris", count: 5 });
37 expect(v[1]).toEqual({ value: "Rome", count: 2 });
38 });
39
40 test("nulls and over-long values are not offered", () => {
41 expect(topValues(source, "t", "note")).toEqual([]);
42 });
43
44 test("a masked column, an unknown column and an unknown table offer nothing", () => {
45 expect(topValues(source, "t", "api_token")).toBeUndefined();
46 expect(topValues(source, "t", "nope")).toBeUndefined();
47 expect(topValues(source, "nope", "city")).toBeUndefined();
48 });
49
50 test("only the first rows are counted, so the read stays inside its budget", async () => {
51 const path = join(dir, "big.db");
52 const d = new Database(path);
53 d.exec("CREATE TABLE t (v TEXT)");
54 const ins = d.prepare("INSERT INTO t VALUES (?)");
55 d.transaction(() => {
56 for (let i = 0; i < TOP_SCAN_ROWS; i++) ins.run("early");
57 for (let i = 0; i < TOP_SCAN_ROWS + 5; i++) ins.run("late");

Timing assertion flakes on a loaded machine; the count check below already proves the budget. Remove it.

58 })();
59 d.close();
60 const big = { ...source, path } as Source;
61 const v = topValues(big, "t", "v")!;
62 expect(v).toEqual([{ value: "early", count: TOP_SCAN_ROWS }]);
63 });
64
65 test("the page shows the links for a chosen column, and none for a masked one", async () => {
66 const view = { path: "/data", sources: [source] };
67 const page = String(await dataBody(view, new URLSearchParams({ source: source.id, table: "t", col: "city" })));
68 expect(page).toContain("Common values of city");
69 expect(page).toMatch(/q=Paris[^"]*exact=1/);
70 const sealed = String(await dataBody(view, new URLSearchParams({ source: source.id, table: "t", col: "api_token" })));
71 expect(sealed).not.toContain("sekrit");
72 expect(sealed).not.toContain("Common values");
73 const typed = String(await dataBody(view, new URLSearchParams({ source: source.id, table: "t", col: "city", q: "Par" })));
74 expect(typed).not.toContain("Common values");
75 });
76
77 test("a picked value filters by equality: its rows are exactly its count", async () => {
78 const path = join(dir, "near.db");
79 const d = new Database(path);
80 d.exec("CREATE TABLE t (city TEXT)");
81 for (const c of ["Paris", "Paris", "Parisian"]) d.run("INSERT INTO t VALUES (?)", [c]);
82 d.close();
83 const near = { ...source, path } as Source;
84 expect(rows(near, "t", { filter: { text: "Paris", column: "city" } })!.total).toBe(3);
85 expect(rows(near, "t", { filter: { text: "Paris", column: "city", exact: true } })!.total).toBe(2);
86 const page = String(await dataBody({ path: "/data", sources: [near] }, new URLSearchParams({ source: near.id, table: "t", col: "city", q: "Paris", exact: "1" })));
87 expect(page).not.toContain("Parisian</");
88 });
89
90 test("the list folds below the rows, says a pick means equals, and spends the query budget", async () => {
91 const view = { path: "/data", sources: [source] };
92 const page = String(await dataBody(view, new URLSearchParams({ source: source.id, table: "t", col: "city" })));
93 expect(page).toContain("<details");
94 expect(page.indexOf("<table")).toBeGreaterThan(-1);
95 expect(page.indexOf("Common values of city")).toBeGreaterThan(page.indexOf("<table"));
96 const picked = String(await dataBody(view, new URLSearchParams({ source: source.id, table: "t", col: "city", q: "Paris", exact: "1" })));
97 expect(picked).toContain("city equals “Paris”");
98 const refused = String(await dataBody({ ...view, sqlWait: () => 7 }, new URLSearchParams({ source: source.id, table: "t", col: "city" })));
99 expect(refused).not.toContain("Common values of city");
100 expect(refused).toContain("try again in 7 s");
101 });
102 });
test/tui-data.test.ts +35
@@ -240,6 +240,27 @@ test("a control byte in a cell cannot reach the terminal from a result, and y on
240240 expect(JSON.parse(copied[0]!)).toEqual({ v: "\u001b[31mred\u0007", n: 2 });
241241 tui.stop?.();
242242 });
243 test("v asks for a column and lists its common values, masked columns offer none", async () => {
244 const { tui, copied } = await fresh("alice");
245 await open_(tui, 3);
246 press(tui, "j"); // users
247 press(tui, "v");
248 typeText(tui, "name");
249 press(tui, "enter");
250 await settleApp(tui);
251 expect(tui.screen.toText()).toContain("ann-visible 1");
252 press(tui, "y");
253 expect(copied).toEqual(["ann-visible"]);
254 press(tui, "esc");
255 press(tui, "v");
256 typeText(tui, "api_token");
257 press(tui, "enter");
258 await settleApp(tui);
259 const text = tui.screen.toText();
260 expect(text).toContain("sealed");
261 expect(text).not.toContain("tok-secret-1");
262 tui.stop?.();
263 });
243264
244265 test(": is refused once the person's SQL budget is spent", async () => {
245266 const { tui, session } = await (async () => {
@@ -277,3 +298,17 @@ test("the prompts cut SQL at 4000 characters and the filter at 200, typed or pas
277298 filter.paste("bc");
278299 expect(filter.text).toBe(`${"a".repeat(FILTER_MAX - 1)}b`);
279300 });
301
302 test("v spends the query budget: once it is spent, no list is read", async () => {

Non-blocking: this test relies on the earlier budget-spending test having run first. Reordering or running it alone makes it fail. Spend the budget here so it stands alone.

303 const { tui } = await fresh("alice");
304 await open_(tui, 3);
305 press(tui, "j"); // users
306 press(tui, "v");
307 typeText(tui, "name");
308 press(tui, "enter");
309 await settleApp(tui);
310 const text = tui.screen.toText();
311 expect(text).toContain("Too many queries");
312 expect(text).not.toContain("ann-visible 1");
313 tui.stop?.();
314 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs