Public: anyone can read this; the crew can change it.

The data explorer offers a filter's values as a list to pick from #13118

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

The data browser now lists a column's commonest values as links, and picking one adds the filter. The new test file and the existing data-browser and data-door tests pass, `tsc`, `scripts/gate.ts` and `design-lint` are clean, and the branch is pushed. I did not run the page on a preview. I also did not call `keep_plan` or `estimate`; they weren't available in this session.

shipped production is serving it deploy 2f4eb9c

opened this

The data browser now lists a column's commonest values as links, and picking one adds the filter. The new test file and the existing data-browser and data-door tests pass, tsc, scripts/gate.ts and design-lint are clean, and the branch is pushed. I did not run the page on a preview. I also did not call keep_plan or estimate; they weren't available in this session.

  • The list: with a column chosen and no filter text typed, the page shows "Common values of <column>". It lists up to 20 values, commonest first, and ties go in alphabetical order. Each value links to the same view with q=<value>, so it is a shareable address. The code is topValues in src/data/browse.ts and the markup is in src/web/pages/data_view.ts.
  • Budget: topValues counts only the first 10,000 rows (TOP_SCAN_ROWS), so it is cheap on any table. It skips nulls, empty strings and values longer than 80 characters. The list is therefore the commonest among a table's first 10,000 rows. This is a row cap, not the SQL box's per-person budget (sql_window.ts), which I judged wrong for a page view.
  • Mask: topValues uses the same maskPlan as the rest of browse.ts. A masked column, whether forced, secret-named or a view that launders one, returns nothing, so the page shows no list for it.
  • Test: test/data-top-values.test.ts covers the 20-value cap and ordering, nulls and long values, masked, unknown-column and unknown-table cases, the scan cap with a timing bound, and the rendered page. The page check includes a masked column whose secret never appears and a typed filter that hides the list.
  • Docs: docs/features/data-explorer.md loses the "picking filter values from a list" part of its Not-done line and gains a bullet describing the list. I kept the other half of that line, the crew/model fallback for phrasings the parser misses (#9569), because that is still not built.
  • Left out: the list does not apply to the "Ask in a sentence" or Boards explorer, only to the data browser, which is where the filter box lives. It also offers no "contains" versus "equals" choice, so a picked value filters by substring like typed text does.

Try it: open /admin/data, choose a source and a table, pick a column in the filter, press Filter, and click a value.

Closes #13114

3 earlier comments · 2 events
commented

The data browser now lists a column's commonest values as links, and picking one adds the filter. The new test file and the existing data-browser and data-door tests pass, tsc, scripts/gate.ts and design-lint are clean, and the branch is pushed. I did not run the page on a preview. I also did not call keep_plan or estimate; they weren't available in this session.

  • The list: with a column chosen and no filter text typed, the page shows "Common values of <column>". It lists up to 20 values, commonest first, and ties go in alphabetical order. Each value links to the same view with q=<value>, so it is a shareable address. The code is topValues in src/data/browse.ts and the markup is in src/web/pages/data_view.ts.
  • Budget: topValues counts only the first 10,000 rows (TOP_SCAN_ROWS), so it is cheap on any table. It skips nulls, empty strings and values longer than 80 characters. The list is therefore the commonest among a table's first 10,000 rows. This is a row cap, not the SQL box's per-person budget (sql_window.ts), which I judged wrong for a page view.
  • Mask: topValues uses the same maskPlan as the rest of browse.ts. A masked column, whether forced, secret-named or a view that launders one, returns nothing, so the page shows no list for it.
  • Test: test/data-top-values.test.ts covers the 20-value cap and ordering, nulls and long values, masked, unknown-column and unknown-table cases, the scan cap with a timing bound, and the rendered page. The page check includes a masked column whose secret never appears and a typed filter that hides the list.
  • Docs: docs/features/data-explorer.md loses the "picking filter values from a list" part of its Not-done line and gains a bullet describing the list. I kept the other half of that line, the crew/model fallback for phrasings the parser misses (#9569), because that is still not built.
  • Left out: the list does not apply to the "Ask in a sentence" or Boards explorer, only to the data browser, which is where the filter box lives. It also offers no "contains" versus "equals" choice, so a picked value filters by substring like typed text does.

Try it: open /admin/data, choose a source and a table, pick a column in the filter, press Filter, and click a value.

commented

The data explorer now lists a column's commonest values as links, and picking one adds the filter. The first push failed the gate on two over-long comment lines and a missing look line, and I've fixed both. scripts/gate.ts is now clean and the new test file passes. I haven't seen the forge's re-run of the check on the new push.

  • Comments: I rewrapped src/data/browse.ts and test/data-top-values.test.ts to fit 100 columns.
  • Look: I added a "No look:" line to the commit message. The data browser is admin-only and shows each viewer's own databases, so app.looks has no page that holds it. If you'd rather have before/after pictures, name a page for a "Looks:" line and I'll swap it in.

What changed

  • List: with a column chosen and no filter text typed, the page shows "Common values of <column>". It lists up to 20 values, commonest first, and ties go in alphabetical order. Each value links to the same view with q=<value>.
  • Budget: values are counted over a table's first 10,000 rows, so the read stays cheap on any table. This is a row cap, not the SQL box's per-person budget. Nulls, empty values and values over 80 characters are skipped.
  • Mask: topValues uses the same maskPlan as the rest of src/data/browse.ts. A masked column offers no list, so no value can leak through it.
  • Test: test/data-top-values.test.ts covers the cap and ordering, the mask, the scan budget and the rendered page.
  • Docs: docs/features/data-explorer.md loses "picking filter values from a list" from its Not-done line. The crew/model fallback for phrasings the parser misses (#9569) stays there, because it is still not built.

Not done: the list is only in the data browser, not in "Ask in a sentence" or Boards. A picked value filters by substring, like typed text, not by exact match.

Try it: open /admin/data, choose a table, pick a column in the filter, press Filter, and click a value.

Changes requested — before the latest push

Core logic is sound (same maskPlan, 20 cap, 10k scan, escaped via html``). Fix in one round: (1) Both doors: the page gains a column value pick-list but src/tui/screens/data.ts has no equivalent; add the common values for a column there (via topValues, same mask, same admin gate) with a test. (2) A picked value filters by substring, so 'Paris' also matches 'Parisian' and the row count no longer equals the link's count; make the pick an exact match or label it honestly. (3) The 500 ms wall-clock assertion is a flake risk; assert the count (only first 10k rows) alone. (4) Docs: the new bullet sits under the 'Not done:' line; put it with the other done bullets.

commented

This pull lets you pick a filter value from a column's commonest values in the data explorer, on the page and in the terminal. This push addresses all four review points; the gate, design-lint and tsc are clean, and the three covering test files pass (40 tests).

  • Both doors: in the terminal Data screen, v on a table or one of its rows asks for a column and lists its common values with counts. It uses topValues (same mask, 20-value cap, 10,000-row scan) and the same admin/writer gate as the rest of that screen. A masked or unknown column says "No such column, or its values are sealed." and shows nothing. y copies the highlighted value. A test in test/tui-data.test.ts covers the list, the copy and the masked case.
  • Exact pick: a picked link now carries exact=1, and the filter compares CAST(col AS TEXT) = ? instead of LIKE %…%. "Paris" no longer matches "Parisian". The typed filter box is still a substring ("contains") search. A test shows 3 rows by substring and 2 by exact match, and that the exact page does not show "Parisian". Exactness is applied to a single named column only.
  • Timing assertion: removed from test/data-top-values.test.ts. The assertion that only the first 10,000 rows are counted stays.
  • Docs: the bullet in docs/features/data-explorer.md now sits above the "Not done:" line. It also says picks are an equality match and that the terminal has v.

Left out: the counts still come from the first 10,000 rows of a table, so on a larger table a value's link count can be lower than its true row count. The terminal list shows values to copy; it does not apply them as a filter, because its / filter works on tree lines rather than table rows.

Try it: open /admin/data, choose a table, choose a column in the filter and click a common value. In the terminal, open Data, move onto a table, press v and type a column name.

Changes requested — before the latest push

The list works on the page and in the terminal, with the mask, the cap and the how-to doc in place. Three things to fix in one round. (1) The 20-row list sits above the grid, so on a 360px phone the real rows are pushed below the fold (principle 2): put the list after the grid, or collapse it in a details element that works with JS off. (2) The picked filter is an exact match but nothing on the page says so, and editing the box drops exact silently: label the active filter 'equals' (or carry it in a hidden field) so the page does not mislead. (3) src/data/browse.ts has an else { branch that was not re-indented, so the block is misformatted: indent it or return early. Issue box 2 asks for the list to be read within the existing query budget. The pull instead adds its own 10,000-row cap, and the builder said so. Either route the read through the existing budget (sql_window.ts) or have the issue changed to say a row cap is the budget.

commented

The data explorer now offers a column's commonest values to pick a filter from, on the page and in the terminal. This round addresses @juniper's review; tsc, the gate and design-lint are clean, and the three covering test files pass (42 tests).

  • Below the rows (principle 2): on /data, /admin/data and an app's data tab, the list now comes after the grid and pager, inside a <details class="disclosure">. Its summary is "Common values of <column>". That works with JS off and is the list's only label, so the repeated aria-label is gone (principle 1).
  • Equals is said: a picked value shows "Rows where <column> equals “<value>”. Filtering again finds rows that contain the text." So the next submit dropping exact is stated on the page rather than silent. I chose not to keep exact in a hidden field: that would make every typed filter an equality match until you press Clear.
  • Query budget (issue box 2): each list read now spends one query from the per-person budget in src/data/sql_window.ts. On the page it goes through the door's existing sqlWait, and in the terminal v calls sqlWindow. When the budget is spent the page shows "Too many queries to list common values; try again in N s." and reads nothing; the terminal shows a toast saying the same. The 10,000-row scan stays as a limit on how much one read costs, alongside the budget rather than in place of it.
  • src/data/browse.ts: the substring branch of the row filter is re-indented under its else, and the exact-match branch has a comment.

Tests in test/data-top-values.test.ts check that the list comes after the <table> inside a <details>, that a pick shows "equals", and that a spent budget reads no list. test/tui-data.test.ts checks that a spent budget stops v. I moved the existing v test above the test that uses up the budget, because it runs after that one and would otherwise be refused. The new budget test relies on that same earlier test using up the budget.

docs/features/data-explorer.md now says the list is folded under the rows, that the page labels an equals filter, and that each list spends the query budget.

To try it: open /admin/data, choose a table and a column, press Filter, then open "Common values of …" below the rows and pick one. In the terminal, open Data, move onto a table, press v and type a column name.

🤖 Generated with Claude Code

Approved

Earlier asks, each met: (1) the list sits below the rows and pager inside a details disclosure, with no repeated aria-label, so it works with JS off; (2) a pick says "Rows where <col> equals ..." and the page states that filtering again means contains; (3) each list read spends the per-person query budget, on the page through sqlWait and in the terminal through sqlWindow, with a plain "try again in N s" message when it is spent; (4) the terminal has v, with the same mask and gate; (5) the docs bullet sits above Not done, and the README is untouched. Masked columns offer nothing, and the tests cover the list, the mask, the scan cap and the budget. I did not drive the preview, because the explorer is admin-only; this review rests on the diff, its tests and the green check.

Two things I would do next, both non-blocking:

  1. Non-blocking: the new TUI budget test depends on an earlier test having used up the budget, which is order-coupled and fragile. Spend the budget inside the test itself.
  2. Non-blocking: the link row shows only a bare count as its trail. A visually hidden "rows" label would let a screen reader say what the number means.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs