Public: anyone can read this; the crew can change it.
The tell box can be dictated into where the browser supports speech #13171
Check · Review · Land · Live done — Landed and live in production.
scout/forge:scout/13130-the-tell-box-can-be-dictated-into-where → main 10 commits fork 6 threads · 6 unresolved
5 files
changed +251
−1
docs/features/README.md +1
| @@ -78,6 +78,7 @@ Read with the guide page [Your first app in five minutes](../guide-first-app.md) | ||
| 78 | 78 | |
| 79 | 79 | Read with the guide page [](../guide-tell-the-crew.md). |
| 80 | 80 | |
| 81 | - [](speak.md) | |
| 81 | 82 | - [](bug-close.md) |
| 82 | 83 | - [](atlas-plan.md) |
| 83 | 84 | - [](crew-maintainer.md) |
docs/features/speak.md added +28
| @@ -0,0 +1,28 @@ | ||
| 1 | # Dictate into the tell box | |
| 2 | ||
| 3 | On the signed-in home, where the browser can turn speech into text, the tell box gains a button | |
| 4 | before Tell: **Speak**, with a line under the box that says "Speak sends your audio to your | |
| 5 | browser's maker to turn it into words." before you press it. It lets you say the line instead | |
| 6 | of typing it. | |
| 7 | ||
| 8 | - **How.** Press Speak, say the line, press it again (or stop talking). The words fill the box; | |
| 9 | you read them, edit them and press Tell yourself. Dictation never sends. | |
| 10 | - **What it says.** A line under the box, read out by screen readers (`role="status"`), says | |
| 11 | "Listening. Press Speak again to stop.", then "Your words are in the box. Press Tell to send." | |
| 12 | A refused microphone says "Could not hear you. Check the microphone is allowed."; silence says | |
| 13 | "Heard nothing. Press Speak to try again." | |
| 14 | - **Which browsers.** Only where the browser has `SpeechRecognition` (or the `webkit` prefixed | |
| 15 | one), such as Chrome and Safari. Elsewhere nothing is added and the page is the one the server | |
| 16 | drew. | |
| 17 | - **Audio may leave your device.** Most browsers send the audio to their maker's speech service to | |
| 18 | turn it into text, so the line under the box says so before the first press, and the button's | |
| 19 | title repeats it. The visible label stays one word so the box still fits a 390 px phone. The forge itself never receives audio, only the text | |
| 20 | you send. | |
| 21 | - **Shape.** `src/web/speak.ts` builds the button and the status line in the tell box | |
| 22 | (`#tell-box`, drawn by `tellBox` in `src/web/studio/living.ts`), with existing classes only | |
| 23 | (`.btn`, `.small .muted`), and limits the text to the box's `maxlength`. It rides with the | |
| 24 | home's script (`src/web/pages/home_flow_client.ts`). The box is a wrapping `.bar`, so on a phone | |
| 25 | the button wraps to its own line and never pushes Tell off the screen. | |
| 26 | - **The terminal.** It has no speech API, so there is no `src/tui/` screen for this; the terminal's | |
| 27 | tell is typed. | |
| 28 | - Tests: `test/tell-speak.test.ts`. |
src/web/pages/home_flow_client.ts +4 −1
| @@ -17,9 +17,12 @@ | ||
| 17 | 17 | * has read the picture) fetches its own address once and takes the answer in place. |
| 18 | 18 | * table on the table view: instant search, picking rows, and the action bar |
| 19 | 19 | * that previews, applies and receipts a bulk action (`home_table.ts`). |
| 20 | * speak the tell box gains a dictation button where the browser has speech (`speak.ts`). | |
| 20 | 21 | * |
| 21 | 22 | * With scripts off the page is the picture the server drew. |
| 22 | 23 | */ |
| 24 | import { speakJs } from "../speak.ts"; | |
| 25 | ||
| 23 | 26 | export const homeFlowJs = /* js */ ` |
| 24 | 27 | (function () { |
| 25 | 28 | var root = document.getElementById('hf'); |
| @@ -341,4 +344,4 @@ export const homeFlowJs = /* js */ ` | ||
| 341 | 344 | remember(); |
| 342 | 345 | clocks(true); |
| 343 | 346 | })(); |
| 344 | `; | |
| 347 | ${speakJs}`; |
src/web/speak.ts added +99
| @@ -0,0 +1,99 @@ | ||
| 1 | /** | |
| 2 | * Speak (#13130): where the browser has `SpeechRecognition` the tell box gains one button that | |
| 3 | * dictates into the box, and a polite status line under it that says what is happening. It only | |
| 4 | * fills the box; sending stays the person's own act (Tell). Where the API is missing nothing is | |
| 5 | * added, so the server's markup is the whole page. | |
| 6 | * | |
| 7 | * Browser speech may send audio to the browser's maker, so the status line says so before the | |
| 8 | * first press (and the button's title repeats it). The script builds | |
| 9 | * its own button and status line, so the tell box's markup (`tellBox`, `studio/living.ts`) is | |
| 10 | * unchanged; it rides with the home's script (`pages/home_flow_client.ts`). Shipped as source text | |
| 11 | * and evaluated by the test with a fake recognition. | |
| 12 | */ | |
| 13 | ||
| 14 | /** The form the script looks up; `tellBox` draws it, and the test reads it on the served home. */ | |
| 15 | export const SPEAK_FORM_ID = "tell-box"; | |
| 16 | ||
| 17 | /** Kept short so the button fits the tell box on a 390 px phone; the note says where audio goes. */ | |
| 18 | export const SPEAK_LABEL = "Speak"; | |
| 19 | ||
| 20 | /** What the status line says, in plain words. */ | |
| 21 | export const SPEAK_SAYS = { | |
| 22 | ready: "Speak sends your audio to your browser's maker to turn it into words.", | |
| 23 | listening: "Listening. Press Speak again to stop.", | |
| 24 | heard: "Your words are in the box. Press Tell to send.", | |
| 25 | nothing: "Heard nothing. Press Speak to try again.", | |
| 26 | denied: "Could not hear you. Check the microphone is allowed.", | |
| 27 | failed: "Could not hear you. Try again, or type instead.", | |
| 28 | } as const; | |
| 29 | ||
| 30 | export const speakJs = /* js */ ` | |
| 31 | function forgeSpeak(win, form) { | |
| 32 | var Rec = win.SpeechRecognition || win.webkitSpeechRecognition; | |
| 33 | var box = form && form.querySelector('input[name="text"],textarea[name="text"]'); | |
| 34 | if (!Rec || !box) return false; | |
| 35 | var says = ${JSON.stringify(SPEAK_SAYS)}; | |
| 36 | var doc = win.document; | |
| 37 | var btn = doc.createElement('button'); | |
| 38 | btn.type = 'button'; | |
| 39 | btn.className = 'btn'; | |
| 40 | btn.setAttribute('data-speak', ''); | |
| 41 | btn.textContent = ${JSON.stringify(SPEAK_LABEL)}; | |
| 42 | btn.title = says.ready; | |
| 43 | var tell = form.querySelector('button[type="submit"]'); | |
| 44 | if (tell) form.insertBefore(btn, tell); else form.appendChild(btn); | |
| 45 | var status = doc.createElement('p'); | |
| 46 | status.className = 'small muted'; | |
| 47 | status.setAttribute('role', 'status'); | |
| 48 | status.setAttribute('aria-live', 'polite'); | |
| 49 | status.setAttribute('data-speak-status', ''); | |
| 50 | form.parentNode.insertBefore(status, form.nextSibling); | |
Non-blocking, principle 1: the ready line shows permanently under the box and the button title repeats it, so the warning is said twice. Keep the line, drop the title. |
||
| 51 | function say(t) { status.textContent = t; } | |
| 52 | say(says.ready); | |
| 53 | var rec = null; | |
| 54 | btn.addEventListener('click', function (e) { | |
| 55 | e.preventDefault(); | |
| 56 | if (rec) { try { rec.stop(); } catch (x) {} return; } | |
| 57 | var r; | |
| 58 | try { r = new Rec(); } catch (err) { say(says.failed); return; } | |
| 59 | rec = r; | |
| 60 | var heard = false, failed = false; | |
| 61 | r.interimResults = false; | |
| 62 | r.continuous = false; | |
| 63 | btn.setAttribute('aria-pressed', 'true'); | |
| 64 | say(says.listening); | |
| 65 | r.onresult = function (ev) { | |
| 66 | var said = ''; | |
| 67 | for (var i = ev.resultIndex || 0; i < ev.results.length; i++) said += ev.results[i][0].transcript; | |
| 68 | said = said.replace(/\\s+/g, ' ').trim(); | |
| 69 | if (!said) return; | |
| 70 | var max = Number(box.getAttribute('maxlength')) || 0; | |
| 71 | var text = (box.value ? box.value.replace(/\\s+$/, '') + ' ' : '') + said; | |
| 72 | box.value = max ? text.slice(0, max) : text; | |
| 73 | box.dispatchEvent(new win.Event('input', { bubbles: true })); | |
| 74 | heard = true; | |
| 75 | say(says.heard); | |
| 76 | }; | |
| 77 | function done() { | |
| 78 | rec = null; | |
| 79 | btn.removeAttribute('aria-pressed'); | |
| 80 | if (!heard && !failed) say(says.nothing); | |
| 81 | } | |
| 82 | r.onerror = function (ev) { | |
| 83 | failed = true; | |
| 84 | var code = ev && ev.error; | |
| 85 | say(code === 'not-allowed' || code === 'service-not-allowed' || code === 'audio-capture' ? says.denied | |
| 86 | : code === 'no-speech' ? says.nothing | |
| 87 | : code === 'aborted' ? '' : says.failed); | |
| 88 | done(); | |
| 89 | }; | |
| 90 | r.onend = done; | |
| 91 | try { r.start(); } catch (err) { failed = true; say(says.failed); done(); } | |
| 92 | }); | |
| 93 | return true; | |
| 94 | } | |
| 95 | (function () { | |
| 96 | if (typeof document === 'undefined') return; | |
| 97 | forgeSpeak(window, document.getElementById(${JSON.stringify(SPEAK_FORM_ID)})); | |
| 98 | })(); | |
| 99 | `; |
test/tell-speak.test.ts added +119
| @@ -0,0 +1,119 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { SPEAK_FORM_ID, SPEAK_LABEL, SPEAK_SAYS, speakJs } from "../src/web/speak.ts"; | |
| 3 | import { mkdtemp, rm } from "node:fs/promises"; | |
| 4 | import { tmpdir } from "node:os"; | |
| 5 | import { join } from "node:path"; | |
| 6 | import { createApp } from "../src/app.ts"; | |
| 7 | import { createSession } from "../src/core/sessions.ts"; | |
| 8 | import { createUser } from "../src/core/users.ts"; | |
| 9 | import { serve } from "../src/http/server.ts"; | |
| 10 | import { promote } from "./helpers/members.ts"; | |
| 11 | ||
| 12 | const run = new Function("win", "form", `${speakJs.split("(function () {")[0]}; return forgeSpeak(win, form);`) as (w: unknown, f: unknown) => boolean; | |
| 13 | ||
| 14 | type Click = (e: { preventDefault(): void }) => void; | |
| 15 | type Fake = { attrs: Record<string, string>; textContent: string; type?: string; className?: string; clicks: Click[] }; | |
| 16 | ||
| 17 | function el(): Fake & Record<string, unknown> { | |
| 18 | const e: Fake & Record<string, unknown> = { | |
| 19 | attrs: {}, textContent: "", clicks: [], | |
| 20 | setAttribute(k: string, v: string) { e.attrs[k] = v; }, | |
| 21 | removeAttribute(k: string) { delete e.attrs[k]; }, | |
| 22 | addEventListener(_: string, f: Click) { e.clicks.push(f); }, | |
| 23 | }; | |
| 24 | return e; | |
| 25 | } | |
| 26 | ||
| 27 | /** A fake tell box whose form fails the test the moment anything sends it. */ | |
| 28 | function rig(withApi: boolean) { | |
| 29 | const made: Fake[] = []; | |
| 30 | const sent = (how: string) => { throw new Error(`dictation sent the form (${how})`); }; | |
| 31 | const box = { value: "", getAttribute: () => "20", dispatchEvent: (e: { type: string }) => (e.type === "submit" ? sent("event") : true) }; | |
| 32 | const tell = { type: "submit" }; | |
| 33 | const placed: unknown[] = []; | |
| 34 | const after: unknown[] = []; | |
| 35 | const form = { | |
| 36 | querySelector: (s: string) => (s.includes("submit") ? tell : box), | |
| 37 | insertBefore: (n: unknown, before: unknown) => { expect(before).toBe(tell); placed.push(n); }, | |
| 38 | appendChild: (n: unknown) => placed.push(n), | |
outdated
The 'never sends' claim is not asserted anymore. Give the fake form a submit and a submit listener that throw, so the test fails if dictation ever sends. |
||
| 39 | submit: () => sent("submit()"), | |
| 40 | requestSubmit: () => sent("requestSubmit()"), | |
| 41 | dispatchEvent: (e: { type: string }) => (e.type === "submit" ? sent("event") : true), | |
outdated
Reads source files and greps for strings. This would pass with broken markup. Render tellBox and check the real output instead. |
||
| 42 | addEventListener: (t: string) => { if (t === "submit") throw new Error("dictation listens for submit"); }, | |
| 43 | parentNode: { insertBefore: (n: unknown) => after.push(n) }, | |
| 44 | }; | |
| 45 | const recs: Array<Record<string, any>> = []; | |
| 46 | class Rec { start() {} stop() {} constructor() { recs.push(this as unknown as Record<string, any>); } } | |
| 47 | const win: Record<string, unknown> = { | |
| 48 | Event: class { constructor(public type: string) {} }, | |
| 49 | document: { createElement: () => { const e = el(); made.push(e); return e; } }, | |
| 50 | }; | |
| 51 | if (withApi) win.SpeechRecognition = Rec; | |
| 52 | const btn = () => placed[0] as Fake; | |
| 53 | const status = () => after[0] as Fake; | |
| 54 | return { box, form, recs, win, btn, status, placed, click: () => btn().clicks[0]!({ preventDefault() {} }) }; | |
| 55 | } | |
| 56 | ||
| 57 | test("without SpeechRecognition nothing is added to the box", () => { | |
| 58 | const r = rig(false); | |
| 59 | expect(run(r.win, r.form)).toBe(false); | |
| 60 | expect(r.placed).toEqual([]); | |
| 61 | }); | |
| 62 | ||
| 63 | test("with it a typed button sits before Tell, fills the box, says so, and never sends", () => { | |
| 64 | const r = rig(true); | |
| 65 | expect(run(r.win, r.form)).toBe(true); | |
| 66 | expect(r.btn().type).toBe("button"); | |
| 67 | expect(r.btn().textContent).toBe(SPEAK_LABEL); | |
| 68 | expect(r.status().attrs).toMatchObject({ role: "status", "aria-live": "polite" }); | |
| 69 | // Before any press the line says where the audio goes, so the short label can fit a phone. | |
| 70 | expect(r.status().textContent).toBe(SPEAK_SAYS.ready); | |
| 71 | expect(SPEAK_SAYS.ready).toContain("browser's maker"); | |
| 72 | expect(SPEAK_LABEL.length).toBeLessThanOrEqual(8); | |
| 73 | r.box.value = "fix"; | |
| 74 | r.click(); | |
| 75 | expect(r.btn().attrs["aria-pressed"]).toBe("true"); | |
| 76 | expect(r.status().textContent).toBe(SPEAK_SAYS.listening); | |
| 77 | r.recs[0]!.onresult({ resultIndex: 0, results: [[{ transcript: " the header please, now " }]] }); | |
| 78 | expect(r.box.value).toBe("fix the header pleas"); // cut at maxlength 20 | |
| 79 | r.recs[0]!.onend(); | |
| 80 | expect(r.btn().attrs["aria-pressed"]).toBeUndefined(); | |
| 81 | expect(r.status().textContent).toBe(SPEAK_SAYS.heard); | |
| 82 | expect(r.btn().textContent).toBe(SPEAK_LABEL); // the label stays put | |
| 83 | }); | |
| 84 | ||
| 85 | test("a refused microphone is said in words beside the box, and the button works again", () => { | |
| 86 | const r = rig(true); | |
| 87 | run(r.win, r.form); | |
| 88 | r.click(); | |
| 89 | r.recs[0]!.onerror({ error: "not-allowed" }); | |
| 90 | r.recs[0]!.onend(); | |
| 91 | expect(r.status().textContent).toBe(SPEAK_SAYS.denied); | |
| 92 | expect(r.btn().attrs["aria-pressed"]).toBeUndefined(); | |
| 93 | r.click(); | |
| 94 | expect(r.recs.length).toBe(2); | |
| 95 | r.recs[1]!.onend(); | |
| 96 | expect(r.status().textContent).toBe(SPEAK_SAYS.nothing); | |
| 97 | }); | |
| 98 | ||
| 99 | test("the served home holds the form the script looks up, and ships the script", async () => { | |
| 100 | const dataDir = await mkdtemp(join(tmpdir(), "forge-tell-speak-")); | |
| 101 | const app = createApp({ dataDir, port: 0 }); | |
| 102 | const servers = await serve(app); | |
| 103 | try { | |
| 104 | const made = await createUser(app.db, { origin: "seed", trusted: true, username: "alice", allowReserved: true, email: "[email protected]", password: "a-good-passphrase", adminIfFirst: true }); | |
| 105 | if (!made.ok) throw new Error(made.error); | |
| 106 | promote(app.db, made.value); | |
| 107 | const cookie = `forge_session=${createSession(app.db, made.value.id).token}`; | |
| 108 | const get = (path: string) => fetch(`${servers.url}${path}`, { headers: { cookie } }); | |
| 109 | const page = await (await get("/")).text(); | |
| 110 | expect(page).toContain(`<form id="${SPEAK_FORM_ID}"`); | |
| 111 | const srcs = [...page.matchAll(/<script[^>]*\ssrc="([^"]+)"/g)].map((m) => m[1]!); | |
| 112 | const bundle = page + (await Promise.all(srcs.map(async (u) => (await get(u)).text()))).join("\n"); | |
| 113 | expect(bundle).toContain("forgeSpeak(window, document.getElementById("); | |
| 114 | } finally { | |
| 115 | servers.stop(); | |
| 116 | app.close(); | |
| 117 | await rm(dataDir, { recursive: true, force: true }); | |
| 118 | } | |
| 119 | }); |
new above
offline
Errors are silent: a denied microphone or no-speech ends with no word to the person. Put a polite live-region message by the box (principle 8: plain words) and make sure it works with the button, not by alert.