Public: anyone can read this; the crew can change it.

The /talk box takes dictation where the browser can, with no audio leaving the device #13963

Merged opened this

Check · Review · Land · Live done — Landed and live in production.

Merged

mason/forge:mason/13943-the-talk-box-takes-dictation-where-the-b → main 8 commits fork 6 threads · 6 unresolved

  • Merged. mason/13943-the-talk-box-takes-dictation-where-the-b was merged into main as bb2d0b8.

Review

Sign in to review this change.
7 files changed +146 −3
docs/design/areas/talk.md +2
@@ -49,3 +49,5 @@ Search (#8823) adds no component class: the form is a `.bar` with `.input` and `
4949 - `.talk-pane` — the right pane: one head line (repository · pull or Production · Open on its own), `.tabs`, then the chosen panel; its left hairline is the only rule.
5050 - `.talk-frame` — the framed live preview, no border.
5151 - `.talk-step` — a tool call drawn as one closed line; opened, its `key: value` lines, one per line, in the tool card's plain paragraph.
52
53 - `.talk-dictate`: the Dictate button in the compose bar (#13943). A `.btn` at the 44 px target, hidden until the script finds on-device speech recognition; while listening it is filled with the accent and reads "Stop dictating".

Non-blocking: put this bullet in the list above without the blank line, so the component list reads as one list.

docs/features/README.md +1
@@ -165,6 +165,7 @@ Read with the guide page [Tell the crew](../guide-tell-the-crew.md).
165165 - [Talk to the forge's agent](talk.md)
166166 - [Stopping a Talk answer](talk-stop.md)
167167 - [Talk answers link the issue and pull numbers they cite](talk-links.md)
168 - [Dictate a message into the /talk box, on the device](talk-dictation.md)
168169 - [Tell the crew from a share, with a look before it files](share-tell.md)
169170 - [The crew line on Home](crew-line.md)
170171 - [The crew list](crew-list.md)
docs/features/talk-dictation.md added +16
@@ -0,0 +1,16 @@
1 # Dictate in /talk
2
3 On `/talk`, a **Dictate** button sits in the compose bar where the browser can recognise speech on the device
4 (Chrome and other Chromium browsers that offer `SpeechRecognition` with `processLocally`). Elsewhere the button never
5 appears. Tap it, speak, and the words are added to the box; nothing is sent until you press Send. While it listens the
6 button reads **Stop dictating** and is filled in. Blocked microphones, silence and unsupported languages are said in
7 words.
8
9 Privacy: the page makes no request for dictation and has no audio route. The button is shown only when the browser can
10 keep recognition on the device; a browser without that is not offered the button, so its speech service is never used.
11
12 Try it: open `/talk` in Chrome, tap **Dictate**, say a sentence, then edit it and send.
13
14 The terminal (both doors): it has no microphone API and already takes typed text, so nothing changes in `src/tui`.
15
16 Pinned by `test/talk-voice.test.ts`; the button's 44 px target comes from `.talk-compose .btn`.
src/web/pages/talk.ts +1
@@ -410,6 +410,7 @@ async function talkPage(ctx: Ctx, conversation: Conversation, before0: string |
410410 <textarea class="input" name="text" rows="1" maxlength="${WORDS_MAX}" required placeholder="Talk to the forge" aria-label="Talk to the forge" aria-keyshortcuts="/">${editing === undefined ? "" : editing.text}</textarea>
outdated

Principle 5/6 and a11y: this button precedes the send button in the bar and matches the send-button selector in talk_client.ts:198, so mode() renames it to 'Queue'. Also check that hidden actually hides a .btn (the Jump button needed its own [hidden] rule).

411411 <div class="bar">
412412 <span class="muted small" data-compose-hint></span>
413 <button type="button" class="btn talk-dictate" data-dictate aria-pressed="false" hidden>Dictate</button>
413414 <label class="btn talk-attach" data-attach hidden>Attach<input type="file" multiple hidden data-pick /></label>
414415 <span class="spacer"></span>
415416 ${editing === undefined ? null : html`<a class="btn" href="${here}">Keep as it was</a>`}
src/web/pages/talk_client.ts +36 −1
@@ -65,6 +65,8 @@ export const talkCss = /* css */ `
6565 .talk-suggest .btn { min-height: 2.75rem; }
6666 .talk-jump { position: absolute; bottom: calc(100% + 8px); left: 50%; transform: translateX(-50%); z-index: 9; min-height: 2.75rem; }
6767 .talk-jump[hidden] { display: none; }
68 .talk-dictate[hidden] { display: none; }
69 .talk-dictate[aria-pressed="true"] { background: var(--accent); color: var(--bg); }
6870 .talk-body pre { position: relative; }
6971 .talk-body pre .talk-copy { position: absolute; top: 2px; right: 2px; }
7072 .talk-chips, .talk-files { margin: 0 0 6px; padding: 0; display: flex; flex-wrap: wrap; gap: 6px; }
@@ -81,6 +83,38 @@ export const talkCss = /* css */ `
8183 .talk-agent .talk-acts { justify-content: flex-start; }
8284 `;
8385
86 /** The send button: the bar's one button that is not Stop, Steer or Dictate. */
87 export const sendSelector = ".bar button.btn:not([data-stop]):not([data-steer]):not([data-dictate])";
88
89 /**
90 * Dictation (#13943): shown only where the browser can recognise speech on the device, since a
91 * browser without `processLocally` may stream audio to a server. Fills the box; never sends.
92 * Needs `form`, `box` and `say` in scope.
93 */
94 export const dictateJs = `
95 const Rec = window.SpeechRecognition || window.webkitSpeechRecognition;
96 const mic = form.querySelector("[data-dictate]");
97 if (Rec && mic && "processLocally" in Rec.prototype) {
98 mic.hidden = false;
99 let rec = null;
100 const words = { "not-allowed": "Microphone access is blocked.", "no-speech": "Heard nothing.", "language-not-supported": "This language can't be heard on this device." };
101 const off = () => { rec = null; mic.setAttribute("aria-pressed", "false"); mic.textContent = "Dictate"; };
102 mic.onclick = () => {
103 if (rec) return rec.stop();
104 rec = new Rec();
105 rec.processLocally = true;
106 rec.lang = navigator.language;
107 rec.onresult = (e) => {
108 const said = Array.from(e.results).slice(e.resultIndex).map((r) => r[0].transcript).join(" ").trim();
109 if (said) box.value += (box.value && !/\\s$/.test(box.value) ? " " : "") + said;
110 };
111 rec.onerror = (e) => say(words[e.error] || "Dictation stopped.");
112 rec.onend = off;
113 try { rec.start(); mic.setAttribute("aria-pressed", "true"); mic.textContent = "Stop dictating"; } catch (_) { off(); say("Dictation could not start."); }
114 };
115 }
116 `;
117
84118 /**
85119 * Runs after DOMContentLoaded (`pageScriptAsset`). Reads the page's own `data-talk-*`; sends with
86120 * fetch, follows with EventSource.
@@ -196,7 +230,7 @@ if (root) {
196230 return false;
197231 }
198232 };
199 const action = form.querySelector(".bar button.btn:not([data-stop]):not([data-steer])");
233 const action = form.querySelector("${sendSelector}");
200234 const stopBtn = form.querySelector("[data-stop]");
201235 const steerBtn = form.querySelector("[data-steer]");
202236 let answering = false;
@@ -308,6 +342,7 @@ if (root) {
308342 say("That file was not attached: no connection.");
309343 }
310344 };
345 ${dictateJs}
311346 const take = (files) => { for (const f of files) upload(f); };
312347 picker.addEventListener("change", () => { take(picker.files); picker.value = ""; });
313348 box.addEventListener("paste", (e) => {
test/talk-page.test.ts +2 −2
@@ -293,8 +293,8 @@ describe("the page", () => {
293293
294294 test("the page's own bytes are held: stylesheet and script stay small", async () => {
295295 const { talkCss, talkJs } = await import("../src/web/pages/talk_client.ts");
296 expect(Buffer.byteLength(talkCss)).toBeLessThan(4_300);
297 expect(Buffer.byteLength(talkJs)).toBeLessThan(15_900);
296 expect(Buffer.byteLength(talkCss)).toBeLessThan(4_500);
297 expect(Buffer.byteLength(talkJs)).toBeLessThan(17_100);
298298 say(alice, "agent", "```ts\nlet x = 1;\n```");
299299 const page = await get("/talk", alice);
300300 const size = (await page.arrayBuffer()).byteLength;
test/talk-voice.test.ts added +88
@@ -0,0 +1,88 @@
1 /**
2 * /talk dictation (#13943): the script runs against a stub SpeechRecognition. The button is
3 * revealed only where recognition can run on the device, fills the box without sending, says
4 * what went wrong in words, and is never mistaken for the Send button.
5 */
6 import { describe, expect, test } from "bun:test";
7 import { Window } from "happy-dom";
8 import { dictateJs, sendSelector, talkCss, talkJs } from "../src/web/pages/talk_client.ts";
9
10 class Stub {
11 static last: Stub;
12 processLocally = false;
13 lang = "";
outdated

These tests grep source text instead of running the script. Run it against a stub so each box is proven by behaviour, and add the design-lint coverage box 3 asks for (or say why it is not possible).

14 onresult: (e: unknown) => void = () => {};
15 onerror: (e: unknown) => void = () => {};
16 onend: () => void = () => {};
17 started = false;
18 failStart = false;
19 constructor() { Stub.last = this; }
20 start() { if (this.failStart) throw new Error("busy"); this.started = true; }
21 stop() { this.started = false; this.onend(); }
22 }
23 class NoLocal { start() {} stop() {} }
24
25 function setup(Rec: unknown) {
26 const window = new Window({ url: "https://forge.test/" });
27 window.document.body.innerHTML = `<form><textarea></textarea><div class="bar"><button type="button" class="btn talk-dictate" data-dictate aria-pressed="false" hidden>Dictate</button><button class="btn">Send</button></div></form>`;
28 const form = window.document.querySelector("form") as any;
29 const box = form.querySelector("textarea");
30 const said: string[] = [];
31 let submitted = 0;
32 form.addEventListener("submit", () => submitted++);
33 const g: any = { SpeechRecognition: Rec };
34 new Function("window", "navigator", "form", "box", "say", dictateJs)(g, { language: "en-GB" }, form, box, (w: string) => said.push(w));
35 return { mic: form.querySelector("[data-dictate]"), box, said, submitted: () => submitted };
36 }
37 const result = (text: string) => ({ resultIndex: 0, results: [[{ transcript: text }]] });
38
39 describe("talk dictation", () => {
40 test("without SpeechRecognition the button stays hidden", () => {
41 expect(setup(undefined).mic.hidden).toBe(true);
42 });
43 test("without on-device recognition it stays hidden, so no audio can leave", () => {
44 expect(setup(NoLocal).mic.hidden).toBe(true);
45 });
46 test("with it the button is revealed, listens locally, fills the box and sends nothing", () => {
47 Stub.prototype.processLocally = false;
48 const t = setup(Stub);
49 expect(t.mic.hidden).toBe(false);
50 t.mic.click();
51 expect(Stub.last.processLocally).toBe(true);
52 expect(Stub.last.started).toBe(true);
53 expect(t.mic.textContent).toBe("Stop dictating");
54 expect(t.mic.getAttribute("aria-pressed")).toBe("true");
55 t.box.value = "fix";
56 Stub.last.onresult(result("the footer"));
57 expect(t.box.value).toBe("fix the footer");
58 t.mic.click();
59 expect(t.mic.textContent).toBe("Dictate");
60 expect(t.mic.getAttribute("aria-pressed")).toBe("false");
61 expect(t.submitted()).toBe(0);
62 });
63 test("errors are said in words", () => {
64 const t = setup(Stub);
65 t.mic.click();
66 Stub.last.onerror({ error: "not-allowed" });
67 Stub.last.onerror({ error: "no-speech" });
68 expect(t.said).toEqual(["Microphone access is blocked.", "Heard nothing."]);
69 });
70 test("a start that fails says so and resets the button", () => {
71 const t = setup(class extends Stub { override failStart = true; override start() { throw new Error("busy"); } });
72 t.mic.click();
73 expect(t.said).toEqual(["Dictation could not start."]);
74 expect(t.mic.textContent).toBe("Dictate");
75 });
76 test("the send selector skips Dictate, so Queue never renames it", () => {
77 const window = new Window();
78 window.document.body.innerHTML = `<div class="bar"><button data-dictate class="btn">Dictate</button><button class="btn">Send</button><button data-stop class="btn">Stop</button></div>`;
79 expect(window.document.querySelector(sendSelector)?.textContent).toBe("Send");
80 });
81 test("the page script embeds it, adds no request, and the hidden button stays hidden in CSS", () => {
82 expect(talkJs).toContain(dictateJs);
83 expect(dictateJs).not.toMatch(/fetch|XMLHttpRequest|sendBeacon|WebSocket|EventSource|MediaRecorder/);
84 expect(talkCss).toContain(".talk-dictate[hidden] { display: none; }");
85 expect(talkCss).toContain(".talk-compose .btn { min-height: 2.75rem; }");
86 expect(dictateJs.length).toBeLessThan(1400);
87 });
88 });

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs