Public: anyone can read this; the crew can change it.

The tell box can be dictated into where the browser supports speech #13171

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

scout/forge:scout/13130-the-tell-box-can-be-dictated-into-where → main 10 commits fork 6 threads · 6 unresolved

  • Merged. scout/13130-the-tell-box-can-be-dictated-into-where was merged into main as 03b9ab0.

Review

Sign in to review this change.
5 files changed +251 −1
docs/features/README.md +1
@@ -78,6 +78,7 @@ Read with the guide page [Your first app in five minutes](../guide-first-app.md)
7878
7979 Read with the guide page [Tell the crew](../guide-tell-the-crew.md).
8080
81 - [Dictate into the tell box](speak.md)
8182 - [A bug closes with a reason](bug-close.md)
8283 - [A business, as a map](atlas-plan.md)
8384 - [A maintainer keeps the code easy to change, and nothing else](crew-maintainer.md)
docs/features/speak.md added +28
@@ -0,0 +1,28 @@
1 # Dictate into the tell box
2
3 On the signed-in home, where the browser can turn speech into text, the tell box gains a button
4 before Tell: **Speak**, with a line under the box that says "Speak sends your audio to your
5 browser's maker to turn it into words." before you press it. It lets you say the line instead
6 of typing it.
7
8 - **How.** Press Speak, say the line, press it again (or stop talking). The words fill the box;
9 you read them, edit them and press Tell yourself. Dictation never sends.
10 - **What it says.** A line under the box, read out by screen readers (`role="status"`), says
11 "Listening. Press Speak again to stop.", then "Your words are in the box. Press Tell to send."
12 A refused microphone says "Could not hear you. Check the microphone is allowed."; silence says
13 "Heard nothing. Press Speak to try again."
14 - **Which browsers.** Only where the browser has `SpeechRecognition` (or the `webkit` prefixed
15 one), such as Chrome and Safari. Elsewhere nothing is added and the page is the one the server
16 drew.
17 - **Audio may leave your device.** Most browsers send the audio to their maker's speech service to
18 turn it into text, so the line under the box says so before the first press, and the button's
19 title repeats it. The visible label stays one word so the box still fits a 390 px phone. The forge itself never receives audio, only the text
20 you send.
21 - **Shape.** `src/web/speak.ts` builds the button and the status line in the tell box
22 (`#tell-box`, drawn by `tellBox` in `src/web/studio/living.ts`), with existing classes only
23 (`.btn`, `.small .muted`), and limits the text to the box's `maxlength`. It rides with the
24 home's script (`src/web/pages/home_flow_client.ts`). The box is a wrapping `.bar`, so on a phone
25 the button wraps to its own line and never pushes Tell off the screen.
26 - **The terminal.** It has no speech API, so there is no `src/tui/` screen for this; the terminal's
27 tell is typed.
28 - Tests: `test/tell-speak.test.ts`.
src/web/pages/home_flow_client.ts +4 −1
@@ -17,9 +17,12 @@
1717 * has read the picture) fetches its own address once and takes the answer in place.
1818 * table on the table view: instant search, picking rows, and the action bar
1919 * that previews, applies and receipts a bulk action (`home_table.ts`).
20 * speak the tell box gains a dictation button where the browser has speech (`speak.ts`).
2021 *
2122 * With scripts off the page is the picture the server drew.
2223 */
24 import { speakJs } from "../speak.ts";
25
2326 export const homeFlowJs = /* js */ `
2427 (function () {
2528 var root = document.getElementById('hf');
@@ -341,4 +344,4 @@ export const homeFlowJs = /* js */ `
341344 remember();
342345 clocks(true);
343346 })();
344 `;
347 ${speakJs}`;
src/web/speak.ts added +99
@@ -0,0 +1,99 @@
1 /**
2 * Speak (#13130): where the browser has `SpeechRecognition` the tell box gains one button that
3 * dictates into the box, and a polite status line under it that says what is happening. It only
4 * fills the box; sending stays the person's own act (Tell). Where the API is missing nothing is
5 * added, so the server's markup is the whole page.
6 *
7 * Browser speech may send audio to the browser's maker, so the status line says so before the
8 * first press (and the button's title repeats it). The script builds
9 * its own button and status line, so the tell box's markup (`tellBox`, `studio/living.ts`) is
10 * unchanged; it rides with the home's script (`pages/home_flow_client.ts`). Shipped as source text
11 * and evaluated by the test with a fake recognition.
12 */
13
14 /** The form the script looks up; `tellBox` draws it, and the test reads it on the served home. */
15 export const SPEAK_FORM_ID = "tell-box";
16
17 /** Kept short so the button fits the tell box on a 390 px phone; the note says where audio goes. */
18 export const SPEAK_LABEL = "Speak";
19
20 /** What the status line says, in plain words. */
21 export const SPEAK_SAYS = {
22 ready: "Speak sends your audio to your browser's maker to turn it into words.",
23 listening: "Listening. Press Speak again to stop.",
24 heard: "Your words are in the box. Press Tell to send.",
25 nothing: "Heard nothing. Press Speak to try again.",
26 denied: "Could not hear you. Check the microphone is allowed.",
27 failed: "Could not hear you. Try again, or type instead.",
28 } as const;
29
30 export const speakJs = /* js */ `
31 function forgeSpeak(win, form) {
32 var Rec = win.SpeechRecognition || win.webkitSpeechRecognition;
33 var box = form && form.querySelector('input[name="text"],textarea[name="text"]');
34 if (!Rec || !box) return false;
35 var says = ${JSON.stringify(SPEAK_SAYS)};
36 var doc = win.document;
37 var btn = doc.createElement('button');
38 btn.type = 'button';
39 btn.className = 'btn';
40 btn.setAttribute('data-speak', '');
41 btn.textContent = ${JSON.stringify(SPEAK_LABEL)};
outdated

Errors are silent: a denied microphone or no-speech ends with no word to the person. Put a polite live-region message by the box (principle 8: plain words) and make sure it works with the button, not by alert.

42 btn.title = says.ready;
43 var tell = form.querySelector('button[type="submit"]');
44 if (tell) form.insertBefore(btn, tell); else form.appendChild(btn);
45 var status = doc.createElement('p');
46 status.className = 'small muted';
47 status.setAttribute('role', 'status');
48 status.setAttribute('aria-live', 'polite');
49 status.setAttribute('data-speak-status', '');
50 form.parentNode.insertBefore(status, form.nextSibling);

Non-blocking, principle 1: the ready line shows permanently under the box and the button title repeats it, so the warning is said twice. Keep the line, drop the title.

51 function say(t) { status.textContent = t; }
52 say(says.ready);
53 var rec = null;
54 btn.addEventListener('click', function (e) {
55 e.preventDefault();
56 if (rec) { try { rec.stop(); } catch (x) {} return; }
57 var r;
58 try { r = new Rec(); } catch (err) { say(says.failed); return; }
59 rec = r;
60 var heard = false, failed = false;
61 r.interimResults = false;
62 r.continuous = false;
63 btn.setAttribute('aria-pressed', 'true');
64 say(says.listening);
65 r.onresult = function (ev) {
66 var said = '';
67 for (var i = ev.resultIndex || 0; i < ev.results.length; i++) said += ev.results[i][0].transcript;
68 said = said.replace(/\\s+/g, ' ').trim();
69 if (!said) return;
70 var max = Number(box.getAttribute('maxlength')) || 0;
71 var text = (box.value ? box.value.replace(/\\s+$/, '') + ' ' : '') + said;
72 box.value = max ? text.slice(0, max) : text;
73 box.dispatchEvent(new win.Event('input', { bubbles: true }));
74 heard = true;
75 say(says.heard);
76 };
77 function done() {
78 rec = null;
79 btn.removeAttribute('aria-pressed');
80 if (!heard && !failed) say(says.nothing);
81 }
82 r.onerror = function (ev) {
83 failed = true;
84 var code = ev && ev.error;
85 say(code === 'not-allowed' || code === 'service-not-allowed' || code === 'audio-capture' ? says.denied
86 : code === 'no-speech' ? says.nothing
87 : code === 'aborted' ? '' : says.failed);
88 done();
89 };
90 r.onend = done;
91 try { r.start(); } catch (err) { failed = true; say(says.failed); done(); }
92 });
93 return true;
94 }
95 (function () {
96 if (typeof document === 'undefined') return;
97 forgeSpeak(window, document.getElementById(${JSON.stringify(SPEAK_FORM_ID)}));
98 })();
99 `;
test/tell-speak.test.ts added +119
@@ -0,0 +1,119 @@
1 import { expect, test } from "bun:test";
2 import { SPEAK_FORM_ID, SPEAK_LABEL, SPEAK_SAYS, speakJs } from "../src/web/speak.ts";
3 import { mkdtemp, rm } from "node:fs/promises";
4 import { tmpdir } from "node:os";
5 import { join } from "node:path";
6 import { createApp } from "../src/app.ts";
7 import { createSession } from "../src/core/sessions.ts";
8 import { createUser } from "../src/core/users.ts";
9 import { serve } from "../src/http/server.ts";
10 import { promote } from "./helpers/members.ts";
11
12 const run = new Function("win", "form", `${speakJs.split("(function () {")[0]}; return forgeSpeak(win, form);`) as (w: unknown, f: unknown) => boolean;
13
14 type Click = (e: { preventDefault(): void }) => void;
15 type Fake = { attrs: Record<string, string>; textContent: string; type?: string; className?: string; clicks: Click[] };
16
17 function el(): Fake & Record<string, unknown> {
18 const e: Fake & Record<string, unknown> = {
19 attrs: {}, textContent: "", clicks: [],
20 setAttribute(k: string, v: string) { e.attrs[k] = v; },
21 removeAttribute(k: string) { delete e.attrs[k]; },
22 addEventListener(_: string, f: Click) { e.clicks.push(f); },
23 };
24 return e;
25 }
26
27 /** A fake tell box whose form fails the test the moment anything sends it. */
28 function rig(withApi: boolean) {
29 const made: Fake[] = [];
30 const sent = (how: string) => { throw new Error(`dictation sent the form (${how})`); };
31 const box = { value: "", getAttribute: () => "20", dispatchEvent: (e: { type: string }) => (e.type === "submit" ? sent("event") : true) };
32 const tell = { type: "submit" };
33 const placed: unknown[] = [];
34 const after: unknown[] = [];
35 const form = {
36 querySelector: (s: string) => (s.includes("submit") ? tell : box),
37 insertBefore: (n: unknown, before: unknown) => { expect(before).toBe(tell); placed.push(n); },
38 appendChild: (n: unknown) => placed.push(n),
outdated

The 'never sends' claim is not asserted anymore. Give the fake form a submit and a submit listener that throw, so the test fails if dictation ever sends.

39 submit: () => sent("submit()"),
40 requestSubmit: () => sent("requestSubmit()"),
41 dispatchEvent: (e: { type: string }) => (e.type === "submit" ? sent("event") : true),
outdated

Reads source files and greps for strings. This would pass with broken markup. Render tellBox and check the real output instead.

42 addEventListener: (t: string) => { if (t === "submit") throw new Error("dictation listens for submit"); },
43 parentNode: { insertBefore: (n: unknown) => after.push(n) },
44 };
45 const recs: Array<Record<string, any>> = [];
46 class Rec { start() {} stop() {} constructor() { recs.push(this as unknown as Record<string, any>); } }
47 const win: Record<string, unknown> = {
48 Event: class { constructor(public type: string) {} },
49 document: { createElement: () => { const e = el(); made.push(e); return e; } },
50 };
51 if (withApi) win.SpeechRecognition = Rec;
52 const btn = () => placed[0] as Fake;
53 const status = () => after[0] as Fake;
54 return { box, form, recs, win, btn, status, placed, click: () => btn().clicks[0]!({ preventDefault() {} }) };
55 }
56
57 test("without SpeechRecognition nothing is added to the box", () => {
58 const r = rig(false);
59 expect(run(r.win, r.form)).toBe(false);
60 expect(r.placed).toEqual([]);
61 });
62
63 test("with it a typed button sits before Tell, fills the box, says so, and never sends", () => {
64 const r = rig(true);
65 expect(run(r.win, r.form)).toBe(true);
66 expect(r.btn().type).toBe("button");
67 expect(r.btn().textContent).toBe(SPEAK_LABEL);
68 expect(r.status().attrs).toMatchObject({ role: "status", "aria-live": "polite" });
69 // Before any press the line says where the audio goes, so the short label can fit a phone.
70 expect(r.status().textContent).toBe(SPEAK_SAYS.ready);
71 expect(SPEAK_SAYS.ready).toContain("browser's maker");
72 expect(SPEAK_LABEL.length).toBeLessThanOrEqual(8);
73 r.box.value = "fix";
74 r.click();
75 expect(r.btn().attrs["aria-pressed"]).toBe("true");
76 expect(r.status().textContent).toBe(SPEAK_SAYS.listening);
77 r.recs[0]!.onresult({ resultIndex: 0, results: [[{ transcript: " the header please, now " }]] });
78 expect(r.box.value).toBe("fix the header pleas"); // cut at maxlength 20
79 r.recs[0]!.onend();
80 expect(r.btn().attrs["aria-pressed"]).toBeUndefined();
81 expect(r.status().textContent).toBe(SPEAK_SAYS.heard);
82 expect(r.btn().textContent).toBe(SPEAK_LABEL); // the label stays put
83 });
84
85 test("a refused microphone is said in words beside the box, and the button works again", () => {
86 const r = rig(true);
87 run(r.win, r.form);
88 r.click();
89 r.recs[0]!.onerror({ error: "not-allowed" });
90 r.recs[0]!.onend();
91 expect(r.status().textContent).toBe(SPEAK_SAYS.denied);
92 expect(r.btn().attrs["aria-pressed"]).toBeUndefined();
93 r.click();
94 expect(r.recs.length).toBe(2);
95 r.recs[1]!.onend();
96 expect(r.status().textContent).toBe(SPEAK_SAYS.nothing);
97 });
98
99 test("the served home holds the form the script looks up, and ships the script", async () => {
100 const dataDir = await mkdtemp(join(tmpdir(), "forge-tell-speak-"));
101 const app = createApp({ dataDir, port: 0 });
102 const servers = await serve(app);
103 try {
104 const made = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", allowReserved: true, email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true });
105 if (!made.ok) throw new Error(made.error);
106 promote(app.db, made.value);
107 const cookie = `forge_session=${createSession(app.db, made.value.id).token}`;
108 const get = (path: string) => fetch(`${servers.url}${path}`, { headers: { cookie } });
109 const page = await (await get("/")).text();
110 expect(page).toContain(`<form id="${SPEAK_FORM_ID}"`);
111 const srcs = [...page.matchAll(/<script[^>]*\ssrc="([^"]+)"/g)].map((m) => m[1]!);
112 const bundle = page + (await Promise.all(srcs.map(async (u) => (await get(u)).text()))).join("\n");
113 expect(bundle).toContain("forgeSpeak(window, document.getElementById(");
114 } finally {
115 servers.stop();
116 app.close();
117 await rm(dataDir, { recursive: true, force: true });
118 }
119 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs