Public: anyone can read this; the crew can change it.

small: an app with no description shows the first sentence of its README in /explore #12439

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

An app with an empty description now shows the first sentence of its README.md in explore, on the page, the API and the MCP. I ran the new test, explore.test.ts, explore-parity.test.ts, explore-speed.test.ts, explore-budget.test.ts, mcp-size.test.ts and bun x tsc --noEmit, and all pass.

Notes

shipped production is serving it deploy d883f28

opened this

An app with an empty description now shows the first sentence of its README.md in explore, on the page, the API and the MCP.

I ran the new test, explore.test.ts, explore-parity.test.ts, explore-speed.test.ts, explore-budget.test.ts, mcp-size.test.ts and bun x tsc --noEmit, and all pass. I did not look at the output of bun scripts/gate.ts. scripts/fast.ts said the diff reaches 1961 of 3127 test files, so I hand-picked those suites and did not run the full suite. The forge runs the full check on the push.

What changed

  • src/core/readme_line.ts (new, pure): takes markdown and returns the first prose sentence as plain text. It skips headings, badges, tables, code fences and front matter, and strips links, emphasis, code and HTML. It cuts to 140 characters, ending in … if it had to cut.
  • src/actions/explore.ts:

- A new described() helper fills in description for app rows that have none. explore() and explorePage() both call it, so the page, the API and the MCP share it with no new tool and no MCP size change. - It reads README.md at the default branch's tip, at most 16 KB, and caches the result by repo and tip, so a request never re-reads an unchanged README. - A private repo, a missing README or any read error leaves the description empty.

  • test/explore-description.test.ts (new): covers the markdown stripping and the 140-character cut. It also checks the fallback, a missing README staying empty, an own description winning, and a private repo staying empty.

Left out

  • The explore text filter (q) still matches only the stored description, not the README sentence.
  • Only README.md is read, not README.markdown, .rst or .txt.
  • The sentence is plain text and is escaped by the page templates on output, not in the helper.

Try it: open /explore?q=kind:app or /api/v1/explore?q=kind:app and look at an app that has a README but no description.

🤖 Generated with Claude Code

Closes #12409

4 earlier comments · 2 events
commented

An app with an empty description now shows the first sentence of its README.md in explore, on the page, the API and the MCP.

I ran the new test, explore.test.ts, explore-parity.test.ts, explore-speed.test.ts, explore-budget.test.ts, mcp-size.test.ts and bun x tsc --noEmit, and all pass. I did not look at the output of bun scripts/gate.ts. scripts/fast.ts said the diff reaches 1961 of 3127 test files, so I hand-picked those suites and did not run the full suite. The forge runs the full check on the push.

What changed

  • src/core/readme_line.ts (new, pure): takes markdown and returns the first prose sentence as plain text. It skips headings, badges, tables, code fences and front matter, and strips links, emphasis, code and HTML. It cuts to 140 characters, ending in … if it had to cut.
  • src/actions/explore.ts:

- A new described() helper fills in description for app rows that have none. explore() and explorePage() both call it, so the page, the API and the MCP share it with no new tool and no MCP size change. - It reads README.md at the default branch's tip, at most 16 KB, and caches the result by repo and tip, so a request never re-reads an unchanged README. - A private repo, a missing README or any read error leaves the description empty.

  • test/explore-description.test.ts (new): covers the markdown stripping and the 140-character cut. It also checks the fallback, a missing README staying empty, an own description winning, and a private repo staying empty.

Left out

  • The explore text filter (q) still matches only the stored description, not the README sentence.
  • Only README.md is read, not README.markdown, .rst or .txt.
  • The sentence is plain text and is escaped by the page templates on output, not in the helper.

Try it: open /explore?q=kind:app or /api/v1/explore?q=kind:app and look at an app that has a README but no description.

🤖 Generated with Claude Code

commented

Merged main into the branch and pushed; the gate and the type check are clean. I ran the 26 explore-related test files (209 pass, 0 fail) but not the full suite, since scripts/fast.ts found the diff too broad to name tests for.

The conflict. src/actions/explore.ts conflicted in explorePage. main made the page size a limit parameter, and this branch wraps the items in described(). I kept both: items: await described(app, items) and nextCursor tested against limit.

A second problem from the merge. Three tests in test/explore-rank.test.ts started failing. A warm rank must not query the database, but described() was calling getRepo for every app with no description (60 queries per page). I added publicRepo() in src/actions/explore.ts, which keeps each repository lookup for 5 seconds. Non-public repositories are cached as absent, so their READMEs are still never read. A repository that goes private can therefore show its README sentence for up to 5 seconds. The 60-second rank cache already keeps showing the row for longer than that.

I also shortened two comment lines that the gate rejected for being over 100 columns.

To try it, open /explore?q=kind:app and look at an app that has no description. It should show the first sentence of its README.

Changes requested — before the latest push

The fallback works and stays public-only, but three things need fixing in one round.

  1. Test gap: explorePage (the API and MCP path) now calls described() and nothing tests it. The test only calls explore(), so deleting that line fails no test. Add a test through explorePage (or /api/v1/explore) that checks the README sentence is present. Also test the front-matter skip and the table/badge skip in readmeLine, and a repo with a 140+ character first sentence through the real path.
  2. Privacy: publicRepo() caches visibility for 5 s, so a repo that has just gone private still serves its README sentence. Private content leaks across a visibility change. Drop the cache and read visibility from the row or the already-loaded rank data. Or, if the cache stays, key it on the repo's updated-at, or invalidate it where visibility changes. A test should cover public then private.
  3. Cleanliness and speed: explore.ts is already large and now holds two module-level caches and git reads, so move the README lookup (described, publicRepo, readmeLines) into its own small module, e.g. src/core/readme_description.ts. Also remove the eslint-disable-next-line comment unless the repo lints with eslint. refTip is a file read per app row on every request, up to 60 per page; confirm it stays under the 50 ms budget warm, since the preview /explore took about 280 ms here.

Also note: the issue says the description is escaped on the page, the API and the MCP. The helper leaves escaping to the templates. Add a test that a README containing <script> or & renders escaped on the page.

commented

An app with no description now shows the first sentence of its README in /explore, the API and the MCP, for public repositories only. I pushed the review fixes as 35ada0dc0.

I ran the 26 explore test files (213 pass), the gate and tsc. I did not run the full suite.

  • Privacy: I dropped the 5-second publicRepo() cache. Each app row now carries its repo's id, branch and public flag, set when the rank reads the repository. The README lookup reads visibility from the row and makes no database query. A new test makes a repo public, then private, and its sentence disappears on the next call.
  • Module: described and its cache of sentences by repo and tip moved to src/core/readme_description.ts. src/actions/explore.ts only imports it. I removed the eslint-disable comment from src/core/readme_line.ts, since the repo has no eslint config.
  • New tests in test/explore-description.test.ts:

- explorePage returns the README sentence. This test would fail without the described() call there. - Front matter, a table and a badge are all skipped. - A sentence over 140 characters is cut with an ellipsis, through the real path. - The page escapes a README containing <script> and &.

  • Speed: I did not time a warm /explore with real READMEs. The 500-repo warm-rank budget test still passes, and the 60 refTip file reads on a page are the only new per-request work. The ~280 ms you saw on the preview is unexplained, and I did not profile it.

Rank rows were cached for up to 60 seconds before this change. The new public flag is read from that same cached row, not fetched fresh. In the public-then-private test the flag changes at once. I did not look at how the rank cache is invalidated, so I can't say whether that holds in every case.

To try it, open /explore?q=kind:app and look at an app that has no description.

Changes requested — before the latest push

The fallback works and stays public-only, but three changes are needed in one round.

  1. Both doors / docs: no docs/features/<name>.md describes the fallback (what it reads, 140 chars, public only, README.md only). Add it. Not in README.md. Also confirm the TUI explore screen shows the filled description (it should, via explore(); add a one-line test or note which screen).
  2. Correctness: readmeLine strips every _ and *, so snake_case words are mangled (my_app_name becomes myappname), and a setext title (Title over ===) becomes the sentence. Strip emphasis only at word boundaries (_x_ with non-word neighbours), and skip a paragraph followed by a ===/--- underline. The sentence cut at the first period also truncates e.g. and v1.0 style text; require the period to follow a non-abbreviation, or at least test it. Add tests for each.
  3. Speed: the preview /explore was reported ~280 ms and is unexplained; refTip is a file read per undescribed app, up to 60 per page. Measure a warm /explore with real READMEs and keep it under 50 ms, or only resolve the README for rows actually returned after the cursor slice (explore() already does; confirm everything() callers do not).

Boxes: all three are met in substance, but left open until the above lands.

commented

When an app in /explore has no description of its own, it shows the first sentence of its public README.md instead. This round fixes the three things from review, adds a feature doc and measures how fast a warm page is. It is merged with main.

Correctness (src/core/readme_line.ts)

  • Emphasis markers (x, __x__, _x_, ~~x~~) are removed only when paired at word boundaries, and inline code loses its backticks. my_app_name and snake_case_ stay as written, and x * y * z is left alone.
  • A paragraph sitting over a === or --- underline is a setext title, so it is skipped and the next paragraph is used.
  • A sentence ends at ., ! or ? followed by a space, but not when the next word starts lowercase. It also doesn't end after e.g., i.e., vs., cf. and a few others, or at a period inside a word like v1.0.
  • Each case has a test in test/explore-description.test.ts.
  • etc. is left off the abbreviation list on purpose. It often really does end a sentence ("…, Y, etc. Built with Bun."), and when it doesn't, the next word is usually lowercase, which the rule above already handles.

Docs

  • docs/features/readme-description.md covers:

- it reads only README.md on the default branch, at most 16 KB of it; - what gets skipped and stripped, and where the sentence ends; - the 140-character cap with …; - public repositories only; - the cost.

  • The terminal's explorer is the dashboard's / search with kind: or sort: (src/tui/screens/dashboard.ts). It calls explorePage and shows row.description as each row's second line, so it gets the filled description. The existing explorePage test covers that path.

Speed

  • Both callers of everything(), explore() and explorePage(), cut to the rows actually returned (after the cursor and limit) before described reads any README.
  • A warm request does one small file read per undescribed app to find the branch tip, and runs no git process. The sentence is kept in memory by repository and tip.
  • New test/explore-description-speed.test.ts puts 60 public undescribed apps, each with a README, behind a real server. Warm /explore?kind=app measured 9–16 ms median across runs, against the 50 ms target.
  • The first request took about 360 ms, because it reads each README with git once and ranks the page. That most likely accounts for the reported ~280 ms on the preview.
  • The test fails only above 250 ms, so a loaded CI machine doesn't fail it, and it prints the measured number on every run.

Checks run: bun scripts/gate.ts, bun x tsc --noEmit, and the explore-description, explore-description-speed, tui-dashboard-explore, tui-explore-paging, docs-link, readme-cap, both-doors, explore-parity and explore-numbers tests all pass. scripts/fast.ts asked for 1,988 test files because of the merge from main, so I ran these covering files instead of the full suite.

Try it: create a public app with an empty description and a README like Title\n=====\n\nRuns my_app_name, e.g. Bun. More. On /explore?kind=app it shows "Runs my_app_name, e.g. Bun."

🤖 Generated with Claude Code

commented

When an app has no description of its own, /explore now shows the first sentence of its public README.md instead. The terminal's explorer and the MCP explore tool show it too, and so does GET /api/v1/explore, which reads it through explorePage. This round fixes the three review points, and a later commit fixes what failed the check.

Correctness (src/core/readme_line.ts)

  • Emphasis markers (x, __x__, _x_, ~~x~~) are removed only when they come in pairs at word boundaries, so my_app_name and snake_case_ stay as written. Inline code just loses its backticks.
  • A paragraph sitting over a === or --- underline is a setext title: it is skipped, and the next paragraph is used.
  • A sentence ends at ., ! or ? followed by a space, unless the next word starts lowercase. It also doesn't end after e.g., i.e., vs., cf. and a few other abbreviations, or at a period inside a word like v1.0. etc. is left off that list on purpose: it often really does end a sentence.
  • Each case has a test in test/explore-description.test.ts.

Docs: docs/features/readme-description.md covers what is read, what is skipped, the 140-character cap and the public-only rule. It also names the terminal screen that shows the sentence: the dashboard's explorer lines in src/tui/screens/dashboard.ts.

Speed

  • Both callers of everything() (explore() and explorePage()) cut to the rows actually returned before any README is read.
  • A warm request reads one small ref file per undescribed app and starts no git process; the sentence is kept in memory by repository and branch tip.
  • The new test/explore-description-speed.test.ts puts 60 public apps, each with a README and no description, behind a real server. Warm /explore measured 9–16 ms median against the 50 ms target. The first request took about 360 ms because it reads each README once, which likely explains the ~280 ms reported on the preview. The test fails only above 250 ms so a loaded machine doesn't trip it, and it prints the measured time on every run.

The failed check

  • Two comment lines in the speed test were longer than 100 columns; they are rewrapped.
  • The gate's "pipeline first" rule wanted a Direction: line, because this pull adds a feature doc without a pipeline change. The rewrap commit carries one: this finishes the review of an open pull rather than starting new work.

Run locally on this push: bun scripts/gate.ts with FORGE_CHECK_BASE=main and bun x tsc --noEmit are clean, and both explore-description test files pass. On the previous push, the terminal-explorer, docs, both-doors and explore-parity tests also passed. I ran chosen test files rather than the full suite, because the merge from main made scripts/fast.ts ask for 1,988 of them.

To try it: make a public app with an empty description and a README of Title\n=====\n\nRuns my_app_name, e.g. Bun. More.. On /explore?kind=app it shows "Runs my_app_name, e.g. Bun."

🤖 Generated with Claude Code

Approved

All three earlier asks are met: markdown stripping, setext titles and sentence boundaries are fixed with tests; docs/features/readme-description.md names the terminal path (dashboard explorer rows via explorePage) and README.md is untouched; the speed test measures a warm page with 60 undescribed apps. The preview serves /api/v1/explore with the new row fields. Two non-blocking next steps: (1) repo.id, branch and public now ride out in the API and MCP rows; strip them at the serialiser, or confirm they are wanted. (2) A warm-time bound of 250 ms is loose against the 50 ms goal; tighten it once CI timing is known.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs