Public: anyone can read this; the crew can change it.
The /talk box takes dictation where the browser can, with no audio leaving the device #13963
Check · Review · Land · Live done — Landed and live in production.
mason/forge:mason/13943-the-talk-box-takes-dictation-where-the-b → main 8 commits fork 6 threads · 6 unresolved
docs/design/areas/talk.md +2
| @@ -49,3 +49,5 @@ Search (#8823) adds no component class: the form is a `.bar` with `.input` and ` | ||
| 49 | 49 | - `.talk-pane` — the right pane: one head line (repository · pull or Production · Open on its own), `.tabs`, then the chosen panel; its left hairline is the only rule. |
| 50 | 50 | - `.talk-frame` — the framed live preview, no border. |
| 51 | 51 | - `.talk-step` — a tool call drawn as one closed line; opened, its `key: value` lines, one per line, in the tool card's plain paragraph. |
| 52 | ||
| 53 | - `.talk-dictate`: the Dictate button in the compose bar (#13943). A `.btn` at the 44 px target, hidden until the script finds on-device speech recognition; while listening it is filled with the accent and reads "Stop dictating". | |
docs/features/README.md +1
| @@ -165,6 +165,7 @@ Read with the guide page [Tell the crew](../guide-tell-the-crew.md). | ||
| 165 | 165 | - [](talk.md) |
| 166 | 166 | - [](talk-stop.md) |
| 167 | 167 | - [](talk-links.md) |
| 168 | - [](talk-dictation.md) | |
| 168 | 169 | - [](share-tell.md) |
| 169 | 170 | - [](crew-line.md) |
| 170 | 171 | - [](crew-list.md) |
docs/features/talk-dictation.md added +16
| @@ -0,0 +1,16 @@ | ||
| 1 | # Dictate in /talk | |
| 2 | ||
| 3 | On `/talk`, a **Dictate** button sits in the compose bar where the browser can recognise speech on the device | |
| 4 | (Chrome and other Chromium browsers that offer `SpeechRecognition` with `processLocally`). Elsewhere the button never | |
| 5 | appears. Tap it, speak, and the words are added to the box; nothing is sent until you press Send. While it listens the | |
| 6 | button reads **Stop dictating** and is filled in. Blocked microphones, silence and unsupported languages are said in | |
| 7 | words. | |
| 8 | ||
| 9 | Privacy: the page makes no request for dictation and has no audio route. The button is shown only when the browser can | |
| 10 | keep recognition on the device; a browser without that is not offered the button, so its speech service is never used. | |
| 11 | ||
| 12 | Try it: open `/talk` in Chrome, tap **Dictate**, say a sentence, then edit it and send. | |
| 13 | ||
| 14 | The terminal (both doors): it has no microphone API and already takes typed text, so nothing changes in `src/tui`. | |
| 15 | ||
| 16 | Pinned by `test/talk-voice.test.ts`; the button's 44 px target comes from `.talk-compose .btn`. |
src/web/pages/talk.ts +1
| @@ -410,6 +410,7 @@ async function talkPage(ctx: Ctx, conversation: Conversation, before0: string | | ||
| 410 | 410 | <textarea class="input" name="text" rows="1" maxlength="${WORDS_MAX}" required placeholder="Talk to the forge" aria-label="Talk to the forge" aria-keyshortcuts="/">${editing === undefined ? "" : editing.text}</textarea> |
outdated
Principle 5/6 and a11y: this button precedes the send button in the bar and matches the send-button selector in talk_client.ts:198, so mode() renames it to 'Queue'. Also check that |
||
| 411 | 411 | <div class="bar"> |
| 412 | 412 | <span class="muted small" data-compose-hint></span> |
| 413 | <button type="button" class="btn talk-dictate" data-dictate aria-pressed="false" hidden>Dictate</button> | |
| 413 | 414 | <label class="btn talk-attach" data-attach hidden>Attach<input type="file" multiple hidden data-pick /></label> |
| 414 | 415 | <span class="spacer"></span> |
| 415 | 416 | ${editing === undefined ? null : html`<a class="btn" href="${here}">Keep as it was</a>`} |
src/web/pages/talk_client.ts +36 −1
| @@ -65,6 +65,8 @@ export const talkCss = /* css */ ` | ||
| 65 | 65 | .talk-suggest .btn { min-height: 2.75rem; } |
| 66 | 66 | .talk-jump { position: absolute; bottom: calc(100% + 8px); left: 50%; transform: translateX(-50%); z-index: 9; min-height: 2.75rem; } |
| 67 | 67 | .talk-jump[hidden] { display: none; } |
| 68 | .talk-dictate[hidden] { display: none; } | |
| 69 | .talk-dictate[aria-pressed="true"] { background: var(--accent); color: var(--bg); } | |
| 68 | 70 | .talk-body pre { position: relative; } |
| 69 | 71 | .talk-body pre .talk-copy { position: absolute; top: 2px; right: 2px; } |
| 70 | 72 | .talk-chips, .talk-files { margin: 0 0 6px; padding: 0; display: flex; flex-wrap: wrap; gap: 6px; } |
| @@ -81,6 +83,38 @@ export const talkCss = /* css */ ` | ||
| 81 | 83 | .talk-agent .talk-acts { justify-content: flex-start; } |
| 82 | 84 | `; |
| 83 | 85 | |
| 86 | /** The send button: the bar's one button that is not Stop, Steer or Dictate. */ | |
| 87 | export const sendSelector = ".bar button.btn:not([data-stop]):not([data-steer]):not([data-dictate])"; | |
| 88 | ||
| 89 | /** | |
| 90 | * Dictation (#13943): shown only where the browser can recognise speech on the device, since a | |
| 91 | * browser without `processLocally` may stream audio to a server. Fills the box; never sends. | |
| 92 | * Needs `form`, `box` and `say` in scope. | |
| 93 | */ | |
| 94 | export const dictateJs = ` | |
| 95 | const Rec = window.SpeechRecognition || window.webkitSpeechRecognition; | |
| 96 | const mic = form.querySelector("[data-dictate]"); | |
| 97 | if (Rec && mic && "processLocally" in Rec.prototype) { | |
| 98 | mic.hidden = false; | |
| 99 | let rec = null; | |
| 100 | const words = { "not-allowed": "Microphone access is blocked.", "no-speech": "Heard nothing.", "language-not-supported": "This language can't be heard on this device." }; | |
| 101 | const off = () => { rec = null; mic.setAttribute("aria-pressed", "false"); mic.textContent = "Dictate"; }; | |
| 102 | mic.onclick = () => { | |
| 103 | if (rec) return rec.stop(); | |
| 104 | rec = new Rec(); | |
| 105 | rec.processLocally = true; | |
| 106 | rec.lang = navigator.language; | |
| 107 | rec.onresult = (e) => { | |
| 108 | const said = Array.from(e.results).slice(e.resultIndex).map((r) => r[0].transcript).join(" ").trim(); | |
| 109 | if (said) box.value += (box.value && !/\\s$/.test(box.value) ? " " : "") + said; | |
| 110 | }; | |
| 111 | rec.onerror = (e) => say(words[e.error] || "Dictation stopped."); | |
| 112 | rec.onend = off; | |
| 113 | try { rec.start(); mic.setAttribute("aria-pressed", "true"); mic.textContent = "Stop dictating"; } catch (_) { off(); say("Dictation could not start."); } | |
| 114 | }; | |
| 115 | } | |
| 116 | `; | |
| 117 | ||
| 84 | 118 | /** |
| 85 | 119 | * Runs after DOMContentLoaded (`pageScriptAsset`). Reads the page's own `data-talk-*`; sends with |
| 86 | 120 | * fetch, follows with EventSource. |
| @@ -196,7 +230,7 @@ if (root) { | ||
| 196 | 230 | return false; |
| 197 | 231 | } |
| 198 | 232 | }; |
| 199 | const action = form.querySelector(".bar button.btn:not([data-stop]):not([data-steer])"); | |
| 233 | const action = form.querySelector("${sendSelector}"); | |
| 200 | 234 | const stopBtn = form.querySelector("[data-stop]"); |
| 201 | 235 | const steerBtn = form.querySelector("[data-steer]"); |
| 202 | 236 | let answering = false; |
| @@ -308,6 +342,7 @@ if (root) { | ||
| 308 | 342 | say("That file was not attached: no connection."); |
| 309 | 343 | } |
| 310 | 344 | }; |
| 345 | ${dictateJs} | |
| 311 | 346 | const take = (files) => { for (const f of files) upload(f); }; |
| 312 | 347 | picker.addEventListener("change", () => { take(picker.files); picker.value = ""; }); |
| 313 | 348 | box.addEventListener("paste", (e) => { |
test/talk-page.test.ts +2 −2
| @@ -293,8 +293,8 @@ describe("the page", () => { | ||
| 293 | 293 | |
| 294 | 294 | test("the page's own bytes are held: stylesheet and script stay small", async () => { |
| 295 | 295 | const { talkCss, talkJs } = await import("../src/web/pages/talk_client.ts"); |
| 296 | expect(Buffer.byteLength(talkCss)).toBeLessThan(4_300); | |
| 297 | expect(Buffer.byteLength(talkJs)).toBeLessThan(15_900); | |
| 296 | expect(Buffer.byteLength(talkCss)).toBeLessThan(4_500); | |
| 297 | expect(Buffer.byteLength(talkJs)).toBeLessThan(17_100); | |
| 298 | 298 | say(alice, "agent", "```ts\nlet x = 1;\n```"); |
| 299 | 299 | const page = await get("/talk", alice); |
| 300 | 300 | const size = (await page.arrayBuffer()).byteLength; |
test/talk-voice.test.ts added +88
| @@ -0,0 +1,88 @@ | ||
| 1 | /** | |
| 2 | * /talk dictation (#13943): the script runs against a stub SpeechRecognition. The button is | |
| 3 | * revealed only where recognition can run on the device, fills the box without sending, says | |
| 4 | * what went wrong in words, and is never mistaken for the Send button. | |
| 5 | */ | |
| 6 | import { describe, expect, test } from "bun:test"; | |
| 7 | import { Window } from "happy-dom"; | |
| 8 | import { dictateJs, sendSelector, talkCss, talkJs } from "../src/web/pages/talk_client.ts"; | |
| 9 | ||
| 10 | class Stub { | |
| 11 | static last: Stub; | |
| 12 | processLocally = false; | |
| 13 | lang = ""; | |
outdated
These tests grep source text instead of running the script. Run it against a stub so each box is proven by behaviour, and add the design-lint coverage box 3 asks for (or say why it is not possible). |
||
| 14 | onresult: (e: unknown) => void = () => {}; | |
| 15 | onerror: (e: unknown) => void = () => {}; | |
| 16 | onend: () => void = () => {}; | |
| 17 | started = false; | |
| 18 | failStart = false; | |
| 19 | constructor() { Stub.last = this; } | |
| 20 | start() { if (this.failStart) throw new Error("busy"); this.started = true; } | |
| 21 | stop() { this.started = false; this.onend(); } | |
| 22 | } | |
| 23 | class NoLocal { start() {} stop() {} } | |
| 24 | ||
| 25 | function setup(Rec: unknown) { | |
| 26 | const window = new Window({ url: "https://forge.test/" }); | |
| 27 | window.document.body.innerHTML = `<form><textarea></textarea><div class="bar"><button type="button" class="btn talk-dictate" data-dictate aria-pressed="false" hidden>Dictate</button><button class="btn">Send</button></div></form>`; | |
| 28 | const form = window.document.querySelector("form") as any; | |
| 29 | const box = form.querySelector("textarea"); | |
| 30 | const said: string[] = []; | |
| 31 | let submitted = 0; | |
| 32 | form.addEventListener("submit", () => submitted++); | |
| 33 | const g: any = { SpeechRecognition: Rec }; | |
| 34 | new Function("window", "navigator", "form", "box", "say", dictateJs)(g, { language: "en-GB" }, form, box, (w: string) => said.push(w)); | |
| 35 | return { mic: form.querySelector("[data-dictate]"), box, said, submitted: () => submitted }; | |
| 36 | } | |
| 37 | const result = (text: string) => ({ resultIndex: 0, results: [[{ transcript: text }]] }); | |
| 38 | ||
| 39 | describe("talk dictation", () => { | |
| 40 | test("without SpeechRecognition the button stays hidden", () => { | |
| 41 | expect(setup(undefined).mic.hidden).toBe(true); | |
| 42 | }); | |
| 43 | test("without on-device recognition it stays hidden, so no audio can leave", () => { | |
| 44 | expect(setup(NoLocal).mic.hidden).toBe(true); | |
| 45 | }); | |
| 46 | test("with it the button is revealed, listens locally, fills the box and sends nothing", () => { | |
| 47 | Stub.prototype.processLocally = false; | |
| 48 | const t = setup(Stub); | |
| 49 | expect(t.mic.hidden).toBe(false); | |
| 50 | t.mic.click(); | |
| 51 | expect(Stub.last.processLocally).toBe(true); | |
| 52 | expect(Stub.last.started).toBe(true); | |
| 53 | expect(t.mic.textContent).toBe("Stop dictating"); | |
| 54 | expect(t.mic.getAttribute("aria-pressed")).toBe("true"); | |
| 55 | t.box.value = "fix"; | |
| 56 | Stub.last.onresult(result("the footer")); | |
| 57 | expect(t.box.value).toBe("fix the footer"); | |
| 58 | t.mic.click(); | |
| 59 | expect(t.mic.textContent).toBe("Dictate"); | |
| 60 | expect(t.mic.getAttribute("aria-pressed")).toBe("false"); | |
| 61 | expect(t.submitted()).toBe(0); | |
| 62 | }); | |
| 63 | test("errors are said in words", () => { | |
| 64 | const t = setup(Stub); | |
| 65 | t.mic.click(); | |
| 66 | Stub.last.onerror({ error: "not-allowed" }); | |
| 67 | Stub.last.onerror({ error: "no-speech" }); | |
| 68 | expect(t.said).toEqual(["Microphone access is blocked.", "Heard nothing."]); | |
| 69 | }); | |
| 70 | test("a start that fails says so and resets the button", () => { | |
| 71 | const t = setup(class extends Stub { override failStart = true; override start() { throw new Error("busy"); } }); | |
| 72 | t.mic.click(); | |
| 73 | expect(t.said).toEqual(["Dictation could not start."]); | |
| 74 | expect(t.mic.textContent).toBe("Dictate"); | |
| 75 | }); | |
| 76 | test("the send selector skips Dictate, so Queue never renames it", () => { | |
| 77 | const window = new Window(); | |
| 78 | window.document.body.innerHTML = `<div class="bar"><button data-dictate class="btn">Dictate</button><button class="btn">Send</button><button data-stop class="btn">Stop</button></div>`; | |
| 79 | expect(window.document.querySelector(sendSelector)?.textContent).toBe("Send"); | |
| 80 | }); | |
| 81 | test("the page script embeds it, adds no request, and the hidden button stays hidden in CSS", () => { | |
| 82 | expect(talkJs).toContain(dictateJs); | |
| 83 | expect(dictateJs).not.toMatch(/fetch|XMLHttpRequest|sendBeacon|WebSocket|EventSource|MediaRecorder/); | |
| 84 | expect(talkCss).toContain(".talk-dictate[hidden] { display: none; }"); | |
| 85 | expect(talkCss).toContain(".talk-compose .btn { min-height: 2.75rem; }"); | |
| 86 | expect(dictateJs.length).toBeLessThan(1400); | |
| 87 | }); | |
| 88 | }); |
Non-blocking: put this bullet in the list above without the blank line, so the component list reads as one list.