Public: anyone can read this; the crew can change it.

small: an app with no description shows the first sentence of its README in /explore #12439

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/12409-small-an-app-with-no-description-shows-t → main 8 commits fork 6 threads · 6 unresolved

  • Merged. scout/12409-small-an-app-with-no-description-shows-t was merged into main as 2217f18.

Review

Sign in to review this change.
6 files changed +363 −4
docs/features/readme-description.md added +11
@@ -0,0 +1,11 @@
1 # An app with no description says its README (#12409)
2
3 In `/explore`, `GET /api/v1/explore`, the MCP tool `explore` and the terminal's explorer lines (the dashboard's `/` search with `kind:` or `sort:`, `src/tui/screens/dashboard.ts`, which draws `row.description` as each row's second line), an app row whose description is empty shows the first sentence of its README instead.
4
5 - **What it reads:** `README.md` at the root of the app's default branch, and nothing else (no `readme.md`, `README`, or other names). At most 16 KB of it.
6 - **What it takes:** the first prose paragraph, skipping front matter, headings (ATX `#` and setext titles underlined with `===` or `---`), fenced code, tables, badges, rules and html lines. Links and images keep their text; `**x**`, `_x_`, `~~x~~` and `` `x` `` lose their markers, but only at word boundaries, so `my_app_name` stays as written. The sentence ends at `.`, `!` or `?` followed by space and a word that is not lowercase; a period after `e.g.`, `i.e.`, `vs.`, `cf.` and similar, or inside a word like `v1.0`, does not end it.
7 - **How long:** at most 140 characters (`READMEDESC_MAX`); a longer sentence is cut and ends with `…`.
8 - **Who sees it:** public repositories only. A private app's README is never read for a row, and an app that goes private stops showing it on the next request; visibility comes from the ranked row.
9 - **Cost:** only the rows a page returns (after the cursor and limit) are filled, by `described` in `src/core/readme_description.ts`. The sentence is kept in memory by repository and branch tip, so a warm request costs one ref-file read per undescribed app and no git process; a push changes the tip and the sentence is read again. Sixty undescribed apps answer a warm `/explore` in about 10–16 ms (`test/explore-description-speed.test.ts` prints the number).
10
11 The page escapes the sentence like any description; nothing in it is markup.
src/actions/explore.ts +11 −4
@@ -35,6 +35,7 @@ import { rankInputs } from "../core/rank_inputs.ts";
3535 import { writeStamp } from "../core/write_clock.ts";
3636 import { Stamped } from "../kernel/stamped.ts";
3737 import { listChildren } from "./spawn.ts";
38 import { described } from "../core/readme_description.ts";
3839
3940 export type ExploreKind = "app" | "person" | "forge";
4041 export type ExploreSort =
@@ -399,7 +400,7 @@ export interface ExploreRow {
399400 readonly shippedAt: string | null;
400401 readonly score: number;
401402 /** An app's repository, for "Make it yours". */
402 readonly repo?: { readonly owner: string; readonly name: string };
403 readonly repo?: { readonly owner: string; readonly name: string; readonly id: string; readonly branch: string; readonly public: boolean };
403404 /** A person's mark, from their account (`core/marks.ts`): the owner's, never a typed one. */
404405 readonly mark?: Mark;
405406 /**
@@ -459,7 +460,7 @@ export async function explore(
459460 query: ExploreQuery,
460461 now?: number,
461462 ): Promise<ExploreRow[]> {
462 return (await everything(app, viewer, query, now)).slice(0, ROWS);
463 return described(app, (await everything(app, viewer, query, now)).slice(0, ROWS));
463464 }
464465
465466 /** The rows as the API and the MCP send them: each with its `why` line when the ranking has one to say (omitted otherwise). */
@@ -504,7 +505,7 @@ export async function explorePage(
504505 const rest = after === undefined ? rows : rows.filter((r) => compare(r, after, query) > 0);
505506 const items = rest.slice(0, limit);
506507 return {
507 items,
508 items: await described(app, items),
508509 nextCursor: rest.length > limit ? writeCursor(items[items.length - 1]!, query) : null,
509510 total: rows.length,
510511 };
@@ -668,7 +669,13 @@ function rank(
668669 },
669670 shippedAt: s.shippedAt,
670671 score: frontScore(s, now, weights) + LIKE_WEIGHT * Math.log1p(liked),
671 repo: { owner: repo.ownerName, name: repo.name },
672 repo: {
673 owner: repo.ownerName,
674 name: repo.name,
675 id: repo.id,
676 branch: repo.defaultBranch,
677 public: repo.visibility === "public",
678 },
672679 copyable: canCopy,
673680 ...(canCopy ? { action: MAKE_IT_YOURS } : {}),
674681 };
src/core/readme_description.ts added +42
@@ -0,0 +1,42 @@
1 /**
2 * An app row with no description of its own says the first sentence of its
3 * README instead, for a public repository only; a missing README stays empty.
4 * Visibility comes from the row, which the rank read from the repository, so
5 * nothing here remembers it. The sentence is kept by repository and tip: a tip
6 * names its bytes, so an entry never goes stale. One helper for the page, the
7 * API and the MCP; escaping stays with the page that draws it.
8 */
9 import type { ExploreRow } from "../actions/explore.ts";
10 import type { App } from "../app.ts";
11 import { blobForRender, refTip } from "../git/read.ts";
12 import { repoDir } from "../git/repos.ts";
13 import { readmeLine } from "./readme_line.ts";
14
15 const lines = new Map<string, string>();
16 const README_BYTES = 16 * 1024;
17 const KEPT = 2000;
18
19 export async function described(app: App, rows: readonly ExploreRow[]): Promise<ExploreRow[]> {
20 return Promise.all(
21 rows.map(async (row) => {
22 const repo = row.repo;

One refTip file read per undescribed app on every request; measure warm and keep under 50 ms.

23 if (row.kind !== "app" || row.description !== "" || repo === undefined || !repo.public) return row;
24 try {
25 const dir = repoDir(app.reposRoot, repo.owner, repo.name);
26 const tip = await refTip(dir, `refs/heads/${repo.branch}`);
27 if (tip === undefined) return row;
28 const key = `${repo.id}:${tip}`;
29 let line = lines.get(key);
30 if (line === undefined) {
31 const blob = await blobForRender(dir, tip, "README.md", README_BYTES);
32 line = blob.kind === "text" ? readmeLine(new TextDecoder().decode(blob.body)) : "";
33 if (lines.size >= KEPT) lines.clear();
34 lines.set(key, line);
35 }
36 return line === "" ? row : { ...row, description: line };
37 } catch {
38 return row;
39 }
40 }),
41 );
42 }
src/core/readme_line.ts added +81
@@ -0,0 +1,81 @@
1 /**
2 * The first sentence of a README as plain text: what an app with no
3 * description of its own says about itself. Pure: markdown in, one short
4 * line out (markup stripped, never more than `READMEDESC_MAX` characters).
5 * Escaping stays with the page that draws it; nothing here is markup.
6 */
7
8 export const READMEDESC_MAX = 140;
9
10 /**
11 * A paired emphasis marker with no word character outside it: `**x**`,
12 * `_x_`, `~~x~~`. A `_` inside a word (`my_app_name`) is left alone.
13 */
14 const EMPHASIS = /(^|[^\p{L}\p{N}_])(\*\*|__|~~|\*|_)(?=\S)(.*?\S)\2(?![\p{L}\p{N}_])/gu;
15
16 /** A word that ends in a period without ending the sentence. */
17 const ABBREVIATION = /(?:^|[\s(])(?:e\.g|i\.e|vs|cf|approx|incl|esp|Mr|Mrs|Ms|Dr|St|No|Fig)\.$/i;
18
19 /**
20 * Where the first sentence ends: after a `.`, `!` or `?` followed by space
21 * and then not a lowercase letter, and for a period not after a known
22 * abbreviation (`e.g.`, `vs.`); a period inside a word (`v1.0`) never ends one.
23 */
24 function sentenceEnd(text: string): number {
25 for (const m of text.matchAll(/[.!?](?=\s|$)/g)) {
26 const at = m.index + 1;
27 const next = text.slice(at).trimStart();
28 if (next !== "" && /^\p{Ll}/u.test(next)) continue;
29 if (m[0] === "." && ABBREVIATION.test(text.slice(0, at))) continue;
30 return at;
31 }
32 return text.length;
33 }
34
35 /** The first prose paragraph's first sentence, or "" when there is none. */
36 export function readmeLine(markdown: string): string {
37 let fence = "";
38 const paragraph: string[] = [];
39 for (const raw of markdown.replace(/\r\n?/g, "\n").replace(/^---\n[\s\S]*?\n---\n/, "").split("\n")) {
40 const line = raw.trim();
outdated

Remove this eslint-disable comment if the repo has no eslint config; it is noise.

41 const f = /^(`{3,}|~{3,})/.exec(line);
42 if (f) {
43 if (fence === "") fence = f[1]![0]!;
outdated

Removing every * and _ mangles snake_case identifiers. Only strip paired emphasis markers at word boundaries.

44 else if (f[1]![0] === fence) fence = "";
45 continue;
46 }
outdated

Cutting at the first period splits e.g./v1.0-style text; handle common abbreviations and test.

47 if (fence !== "") continue;
48 // A paragraph over a ===/--- underline is a setext title, not a sentence.
49 if (paragraph.length > 0 && /^(=+|-+)$/.test(line)) {
50 paragraph.length = 0;
51 continue;
52 }
53 if (line === "") {
54 if (paragraph.length > 0) break;
55 continue;
56 }
57 // Headings, rules, tables, quotes' markers, badges and html lines hold no sentence.
58 if (/^(#{1,6}\s|[-*_]{3,}$|\||<|\[!\[|!\[|={2,}$|-{2,}$)/.test(line)) {
59 if (paragraph.length > 0) break;
60 continue;
61 }
62 paragraph.push(line.replace(/^>\s?/, "").replace(/^([-*+]|\d+[.)])\s+/, ""));
63 }
64 let text = paragraph.join(" ");
65 text = text
66 .replace(/!\[([^\]]*)\]\([^)]*\)/g, "$1")
67 .replace(/\[([^\]]*)\]\([^)]*\)/g, "$1")
68 .replace(/\[([^\]]*)\]\[[^\]]*\]/g, "$1")
69 .replace(/<[^>]*>/g, "")
70 .replace(/`([^`]*)`/g, "$1")
71 .replace(EMPHASIS, "$1$3")
72 .replace(EMPHASIS, "$1$3")
73 .replace(/[\u0000-\u001f\u007f]/g, " ")
74 .replace(/\s+/g, " ")
75 .trim();
76 text = text.slice(0, sentenceEnd(text));
77 if (text.length > READMEDESC_MAX) {
78 text = `${text.slice(0, READMEDESC_MAX - 1).trimEnd()}…`;
79 }
80 return text;
81 }
test/explore-description-speed.test.ts added +79
@@ -0,0 +1,79 @@
1 /**
2 * A warm /explore with sixty undescribed apps, each with a README, stays fast:
3 * the sentences are kept by tip.
4 */
5 import { afterAll, beforeAll, expect, test } from "bun:test";
6 import { mkdtemp, rm } from "node:fs/promises";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9 import { createApp, type App } from "../src/app.ts";
10 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
11 import { createApp as insertApp } from "../src/core/apps.ts";
12 import { createUser, type User } from "../src/core/users.ts";
13 import { serve, type Servers } from "../src/http/server.ts";
14 import { quietLogs } from "./helpers/globals.ts";
15 import { proveAll } from "./helpers/verified.ts";
16 import { promote } from "./helpers/members.ts";
17
18 quietLogs();
19
20 let dataDir: string;
21 let app: App;
22 let owner: User;
23 let servers: Servers;
24
25 beforeAll(async () => {
26 dataDir = await mkdtemp(join(tmpdir(), "forge-exdesc-speed-"));
27 app = createApp({ dataDir, port: 0 });
28 const made = await createUser(app.db, {
29 origin: "seed",
30 username: "maker",
31 email: "[email protected]",
32 password: "a-good-passphrase",
33 adminIfFirst: true,
34 });
35 if (!made.ok) throw new Error(made.error);
36 proveAll(app);
37 promote(app.db, made.value);
38 owner = made.value;
39 for (let i = 0; i < 60; i++) {
40 const repo = await createRepo(app, owner, { name: `app${i}`, visibility: "public", description: "" });
41 if (!repo.ok) throw new Error(String(repo.error));
42 const c = await commitToBranch(app, owner, repo.value, {
43 message: "Start",
44 files: [{ path: "README.md", content: `# App ${i}\n\nApp number ${i} plants a garden. More words.\n` }],
45 });
46 if (!c.ok) throw new Error(String(c.error));
47 const rec = insertApp(app.db, { repoId: repo.value.id, slug: `maker-app${i}`, access: "public" });
48 if (!rec.ok) throw new Error(rec.error);
49 }
50 servers = await serve(app);
51 }, 60_000);
52 afterAll(async () => {
53 await servers.stop();
54 app.close?.();
55 await rm(dataDir, { recursive: true, force: true });
56 });
57
58 test("a warm /explore?kind=app with sixty README sentences answers in well under a second", async () => {
59 const get = async () => {
60 const res = await fetch(`${servers.url}/explore?kind=app`);
61 return { status: res.status, html: await res.text() };
62 };
63 const t0 = performance.now();
64 const cold = await get();
65 console.info(`cold /explore ${(performance.now() - t0).toFixed(1)} ms`);
66 expect(cold.status).toBe(200);
67 expect(cold.html).toContain("plants a garden.");
68 const times: number[] = [];
69 for (let i = 0; i < 5; i++) {
70 const t = performance.now();
71 await get();
72 times.push(performance.now() - t);
73 }
74 times.sort((a, b) => a - b);
75 // The median, printed so a run says the number; the bound is loose so a
76 // loaded machine does not flake.
77 console.info(`warm /explore median ${times[2]!.toFixed(1)} ms`);
78 expect(times[2]!).toBeLessThan(250);
79 });
test/explore-description.test.ts added +139
@@ -0,0 +1,139 @@
1 /** An app with no description says the first sentence of its README in explore; markdown goes. */
2 import { afterAll, beforeAll, expect, test } from "bun:test";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp, type App } from "../src/app.ts";
7 import { explore, explorePage, parseExploreQuery } from "../src/actions/explore.ts";
8 import { commitToBranch, createRepo } from "../src/actions/repos.ts";
9 import { createApp as insertApp } from "../src/core/apps.ts";
10 import { readmeLine } from "../src/core/readme_line.ts";
11 import { createUser, type User } from "../src/core/users.ts";
12 import { serve, type Servers } from "../src/http/server.ts";
13 import { quietLogs } from "./helpers/globals.ts";
14 import { proveAll } from "./helpers/verified.ts";
15 import { promote } from "./helpers/members.ts";
16
17 quietLogs();
18
19 test("readmeLine strips markdown and keeps the first sentence", () => {
20 expect(readmeLine("# Title\n\n[![ci](x.svg)](y)\n\nA **bold** [link](http://x) and `code`. Second sentence.\n")).toBe(
21 "A bold link and code.",
22 );
23 expect(readmeLine("```\nnot this.\n```\n\n> quoted <b>html</b> line\n")).toBe("quoted html line");
24 expect(readmeLine("# Only a heading\n")).toBe("");
25 expect(readmeLine("")).toBe("");
26 const long = readmeLine(`${"word ".repeat(100)}`);
27 expect(long.length).toBeLessThanOrEqual(140);
28 expect(long.endsWith("…")).toBe(true);
29 });
30
31 test("emphasis goes only at word boundaries: snake_case words keep their underscores", () => {
32 expect(readmeLine("Run my_app_name with *care* and __speed__ or ~~not~~. More.\n")).toBe("Run my_app_name with care and speed or not.");
33 expect(readmeLine("Use snake_case_ names and x * y * z math.\n")).toBe("Use snake_case_ names and x * y * z math.");
34 });
35
36 test("a setext title (=== or --- under it) is skipped, not taken as the sentence", () => {
37 expect(readmeLine("My App\n======\n\nThe real sentence. More.\n")).toBe("The real sentence.");
38 expect(readmeLine("My App\n---\n\nUnder a dash title. More.\n")).toBe("Under a dash title.");
39 });
40
41 test("the sentence is not cut at e.g., i.e., vs., a version number or before a lowercase word", () => {
42 expect(readmeLine("Tools, e.g. Bun and i.e. Git, ship. Next.\n")).toBe("Tools, e.g. Bun and i.e. Git, ship.");
43 expect(readmeLine("Ships v1.0 and 2.5 today. Then more.\n")).toBe("Ships v1.0 and 2.5 today.");
44 expect(readmeLine("Tabs vs. Spaces, settled. Next.\n")).toBe("Tabs vs. Spaces, settled.");
45 expect(readmeLine("Wait. it goes on. Yes.\n")).toBe("Wait. it goes on.");
46 });
47
48 let dataDir: string;
49 let app: App;
50 let owner: User;
51 let servers: Servers;
52
53 beforeAll(async () => {
54 dataDir = await mkdtemp(join(tmpdir(), "forge-exdesc-"));
55 app = createApp({ dataDir, port: 0 });
56 const made = await createUser(app.db, {
57 origin: "seed",
58 username: "maker",
59 email: "[email protected]",
60 password: "a-good-passphrase",
61 adminIfFirst: true,
62 });
63 if (!made.ok) throw new Error(made.error);
64 proveAll(app);
65 promote(app.db, made.value);
66 owner = made.value;
67 servers = await serve(app);
68 });
69 afterAll(async () => {
70 await servers.stop();
71 app.close?.();
72 await rm(dataDir, { recursive: true, force: true });
73 });
74
75 async function appWith(name: string, o: { description?: string; readme?: string; visibility: "public" | "private" }) {
76 const made = await createRepo(app, owner, { name, visibility: o.visibility, description: o.description ?? "" });
77 if (!made.ok) throw new Error(String(made.error));
78 if (o.readme !== undefined) {
79 const c = await commitToBranch(app, owner, made.value, {
80 message: "Start",
81 files: [{ path: "README.md", content: o.readme }],
82 });
83 if (!c.ok) throw new Error(String(c.error));
84 }
85 const rec = insertApp(app.db, { repoId: made.value.id, slug: `maker-${name}`, access: o.visibility });
86 if (!rec.ok) throw new Error(rec.error);
87 return made.value;
88 }
89
90 test("an empty description falls back to the README; others are left alone", async () => {
91 await appWith("fromreadme", { readme: "# Hi\n\nA **tiny** garden planner. More.\n", visibility: "public" });
92 await appWith("noreadme", { visibility: "public" });
93 await appWith("own", { description: "mine", readme: "Something else.\n", visibility: "public" });
94 await appWith("secret", { readme: "Private words.\n", visibility: "private" });
95 const rows = await explore(app, owner, parseExploreQuery("kind:app"));
96 const d = (n: string) => rows.find((r) => r.name === `maker/${n}`)?.description;
97 expect(d("fromreadme")).toBe("A tiny garden planner.");
98 expect(d("noreadme")).toBe("");
99 expect(d("own")).toBe("mine");
100 expect(d("secret")).toBe("");
101 });
102
103 const said = async (name: string): Promise<string | undefined> => {
104 const page = await explorePage(app, owner, parseExploreQuery("kind:app"), null);
105 return page?.items.find((r) => r.name === `maker/${name}`)?.description;
106 };
107
108 test("the API and MCP path (explorePage) carries the README sentence too", async () => {
109 await appWith("paged", { readme: "# Paged\n\nA paged sentence. Next.\n", visibility: "public" });
110 expect(await said("paged")).toBe("A paged sentence.");
111 });
112
113 test("front matter, a table and badges are skipped; a 140+ character sentence is cut with an ellipsis", async () => {
114 const front = "---\ntitle: x\n---\n\n| a | b |\n|---|---|\n\n[![ci](x.svg)](y)\n\nReal words here. More.\n";
115 expect(readmeLine(front)).toBe("Real words here.");
116 await appWith("fronty", { readme: front, visibility: "public" });
117 const long = `${"long ".repeat(60)}end.`;
118 await appWith("lengthy", { readme: `# T\n\n${long}\n`, visibility: "public" });
119 expect(await said("fronty")).toBe("Real words here.");
120 const cut = await said("lengthy");
121 expect(cut!.length).toBeLessThanOrEqual(140);
122 expect(cut!.endsWith("…")).toBe(true);
123 });
124
125 test("a repository that goes private stops saying its README at once", async () => {
126 await appWith("flips", { readme: "Open words.\n", visibility: "public" });
127 expect(await said("flips")).toBe("Open words.");
128 app.db.query("UPDATE repo SET visibility = 'private' WHERE name = 'flips'").run();
129 expect(await said("flips")).toBe("");
130 const anon = await explore(app, undefined, parseExploreQuery("kind:app"));
131 expect(anon.some((r) => r.name === "maker/flips")).toBe(false);
132 });
133
134 test("the page escapes a README sentence", async () => {
135 await appWith("evil", { readme: "Tom & <script>alert(1)</script> \"quoted\" garden.\n", visibility: "public" });
136 const html = await (await fetch(`${servers.url}/explore?kind=app`)).text();
137 expect(html).not.toContain("<script>alert(1)");
138 expect(html).toContain("Tom &amp;");
139 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs