Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

workbenchthis machine

Mobile driving: pick a joint and drag one big slider (portrait), two thumb sticks (landscape); zero motion the instant a finger lifts #81

Parked opened this in answer to
Closed without merging

vesper/arm:vesper/50-mobile-driving-pick-a-joint-and-drag-one → main 2 commits fork

  • Closed without merging. Nothing was applied to main.

Review

Sign in to review this change.
15 files changed +658 −1175
Software/README.md +52 −53
@@ -168,7 +168,7 @@ arm (its own URDF and meshes, drag a joint and the arm follows), a slider and
168168 hold-to-jog buttons per joint that stream targets like a leader does, a
169169 button per saved pose plus save and delete, hold and release, record and
170170 play, one STOP button (pressed, not released; space or Esc) with a separate
171 two-tap **Resume driving** in the stopped banner (STOP is also how a pose
171 two-tap **Resume driving** in the stopped note (STOP is also how a pose
172172 move or a playback is ended early; there is no separate "End move" button,
173173 so driving again after it takes a Resume), fullscreen (F), a metrics panel, a settings
174174 panel for the safe zone, and a 3D-calibration panel that explains the vision
@@ -353,7 +353,7 @@ has Web Serial, not which browser it is.
353353 before your stream has closed.
354354 - iPhone and iPad browsers can't open USB ports (WebKit has no Web Serial).
355355 The button explains this there, as on Firefox and desktop Safari, and points
356 to the sliders, the − and + buttons, and dragging the 3D model.
356 to dragging the 3D model and the sliders.
357357
358358 The page reads the leader with the CLI's own bus code: `src/leader-arm.ts`
359359 puts `FeetechBus` on a Web Serial port, and the server hands it and the
@@ -510,67 +510,66 @@ appended to `admin.log.jsonl` next to it, with who did it. `/admin` shows
510510 the latest entries. Changes are accepted only from the owner, and only from
511511 the app's own page.
512512
513 **On a phone.** The page is laid out for one thumb, phone first. The view
514 (camera or 3D) sits on top at full width. Under it a status strip shows who
515 drives, your place in the line, the time left, the turn's one button, and
516 the red **Stop**. Stop is there whenever you drive, and for the owner. Below
517 that is a sheet with tabs at the bottom edge: **Chat**, **Queue**,
513 **On a phone.** The page is laid out for one thumb, phone first. Held
514 upright, the camera sits on top at full width (16:9) and the 3D model right
515 under it, both always on screen. Under them a status strip shows who drives,
516 your place in the line, the time left, the turn's one button, the notes pill
517 and the red **Stop**. Stop is there whenever you drive, and for the owner.
518 Below that is a tray with tabs at the bottom edge: **Chat**, **Queue**,
518519 **Controls** (only on your turn, and it opens by itself when your turn
519 starts) and **Admin** (the owner: the room, the safe zone, the trims,
520 metrics and 3D calibration).
520 starts), **Admin** (the owner: the room, the safe zone, the trims, metrics
521 and 3D calibration) and **Machine**. The grabber on top of the tray, or the
522 open tab tapped again, folds the tray away and gives the model the room.
521523
522 - Controls, phone upright: a row of chips (J1 base, J2 shoulder, J3 elbow,
523 J4 wrist, J5 roll, Grip) picks one joint. One tall lever drives it: push
524 up or down, and the further you push the faster it goes. Let go and it
525 stops. ▲/▼ beside the lever nudge one step per tap and jog while held.
526 Under them come the poses, Hold, Record and the rest. This layout is for
527 screens up to 640 px wide. A tablet held upright gets the joystick and
528 sliders of the wider screens. From a keyboard, arrows on the chips pick
529 the joint and ↑/↓ on the lever step it.
530 - Controls, phone on its side: the sheet tucks away when your turn starts,
531 and two thumb sticks sit over the picture. The left stick moves J1
532 sideways and J2 up/down. The right stick moves J4 sideways and J3 up/down,
533 and **Wrist** swaps it to J5 roll and the gripper. **Open** and **Close**
534 between the sticks are press-and-hold. **Show** brings the sheet back.
535 The sticks and the grip buttons take touch and a mouse only. With a
536 keyboard or a screen reader, **Show** brings back the sheet, and its joint
537 sliders and ±-buttons drive every joint, the gripper included.
538 - How touch driving works (`src/ui/drive.js`): a finger's deflection is a
539 speed, with a dead band in the middle, never a position. Each finger is
540 tracked by its own pointer, so two thumbs drive two sets of joints
541 independently. The instant a finger lifts, its joints' targets are sent
542 as they stand, without waiting for the next frame (`src/ui/frames.js`
543 sends whatever is still pending with it). The arm's governor
544 settles on them. The same happens on pointercancel, on a lost capture,
545 when the window loses focus, when the page is hidden, and when driving
546 stops being allowed: your turn ends, the arm is stopped, or it is
547 reconnecting (`canDrive` in `src/ui/drive.js`). Frames go out
548 at 25 Hz. A target never runs more than 0.6 s ahead of where the arm
549 last said it was, so a slow link or a speed cap never queues up motion.
550 The server's limits clamp every frame as before. Android buzzes on a
551 press and at the end of a joint's range. Every press also shows on
552 screen. The screen stays awake during your turn (Screen Wake Lock), and
553 the status line shows the round trip to the arm once it passes 250 ms.
554 On a phone the strip carries one stop button, the E-stop.
555 - Controls, wider screens: a joystick moves the gripper. Up reaches out,
556 sideways swings around the base, and ▲/▼ raise and lower it. The page
557 solves that onto pan, lift and elbow against the URDF
558 (`src/ui/reach.js`), and the server checks every frame against the zone
559 as usual. Below the joystick come the joint sliders with hold-to-jog −/+,
560 the poses, and the rest.
524 - Driving is touching the model (`src/ui/grab.js`). Put a finger on a part
525 of the arm and drag: that part's joint turns, and only that joint goes to
526 the arm, live, through the same latest-wins stream as the sliders (at most
527 one frame per 33 ms, `src/ui/frames.js`). A finger on empty space orbits
528 the view, and two fingers orbit and pinch to zoom. None of that sends
529 anything. The model is drawn where you asked the arm to go; a faint blue
530 ghost shows where the real arm is, whenever it lags behind by more than a
531 few ticks. Lifting the finger sends the joint's last target at once, not
532 at the next frame. So do pointercancel, a lost capture, a hidden page and
533 a window that loses focus. When it is not your turn, or the arm is
534 stopped or reconnecting, touching the model only orbits it. The mouse
535 works the same way on a wide screen.
536 - Controls tab: every joint's slider at once, one compact row each (name ·
537 slider · value). On each rail a tall blue mark shows your target and a
538 thin line shows where the arm is. Wider screens add hold-to-jog −/+
539 buttons per joint. Below the sliders come the poses, Hold, Record and
540 the rest.
541 - Notes never cover the camera or the model. The stop, the arm's warnings
542 (reconnecting, the simulation, a jam, the safe zone), the owner's
543 announcement and the page's replies become one pill in the strip ("2
544 notes", red for a stop or an error). Tap it to read them in a sheet,
545 where **Resume driving** and **Clear the stop** are too.
546 - Android buzzes when a finger takes a joint. The screen stays awake during
547 your turn (Screen Wake Lock). The status line shows the round trip to the
548 arm once it passes 250 ms. The server's limits clamp every frame as
549 before.
561550 - Chat: the page follows the visual viewport. The keyboard shrinks the
562551 sheet and the view, and never covers the box or the last message.
563552 Long-press a message (or right-click it) to delete it, or mute or ban its
564553 author.
565 - Phone on its side: the view fills the screen, the strip and the sheet
566 float over it, and **Hide** clears them away. A wide screen puts the
554 - Phone on its side: the camera and the model share the screen side by
555 side, the strip and the sheet float over them, and **Hide** clears them
556 away. A wide screen puts the
567557 sheet in a column beside the view.
568 - Every tap target is at least 44 px. The safe-area insets are respected,
558 - Checked in a browser: `src/layout.test.ts` loads the page in headless
559 Chromium (over the DevTools protocol, `src/browser.ts`, no dependency) at
560 390×844 and 412×915, light and dark. It checks that the camera is on top
561 at 16:9 with the model under it, that all six sliders fit in the tray,
562 and that no note overlaps either view. Without a Chromium that can start,
563 the suite is skipped. `bun bench/shots.ts [dir]` takes the screenshots:
564 both phones, upright and on their side, light and dark, with the tray
565 open, folded, stopped and with the notes sheet.
566 - Every button is at least 44 px. The tray's grabber is thinner, and
567 tapping the open tab does the same. The safe-area insets are respected,
569568 the page follows the system's light or dark mode, and nothing moves for
570569 people who ask for reduced motion. The bottom-right corner belongs to the
571570 forge pill, so no control sits under it.
572 - Mobile data: the first view needs no three.js. The model and the
573 joystick arrive after it. The relay gzips the page, three.js, the URDF
571 - Mobile data: the first view needs no three.js. The model and its touch
572 handling arrive after it. The relay gzips the page, three.js, the URDF
574573 and the meshes (`src/compress.ts`). The page's modules are cached for
575574 good under a name that changes with their content. On a slow link, a
576575 slow `/status` round trip or the browser's Save-Data, the camera drops
Software/so101/bench/shots.ts added +55
@@ -0,0 +1,55 @@
1 /**
2 * Screenshots of the page on phones, for a pull: iPhone (390x844) and Android (412x915), upright and on its side,
3 * light and dark, each with the tray open, the tray folded and the notes sheet open (after a Stop).
4 * bun bench/shots.ts [out-dir] (default ./shots; CHROMIUM=/path/to/chrome picks the browser)
5 * The arm is the simulated bus, the model the repository's URDF; there is no camera, so its pane is black.
6 */
7 import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
8 import { tmpdir } from "node:os";
9 import { join } from "node:path";
10 import { launch } from "../src/browser.ts";
11 import { FeetechBus } from "../src/bus.ts";
12 import { FakeBus } from "../src/fake-bus.ts";
13 import { startServer } from "../src/server.ts";
14
15 const out = process.argv[2] ?? "shots";
16 mkdirSync(out, { recursive: true });
17 const browser = await launch();
18 if (!browser) throw new Error("no headless Chromium could start: set CHROMIUM to one");
19 const fake = new FakeBus([1, 2, 3, 4, 5, 6]);
20 for (const id of [1, 2, 3, 4, 5, 6]) fake.servos.get(id)!.set([0x32, 0x03, 0x9f, 0x0d], 9);
21 const server = await startServer(new FeetechBus(fake), { host: "127.0.0.1", port: 0, posesFile: join(mkdtempSync(join(tmpdir(), "so101-shots-")), "poses.json") });
22 const base = `http://127.0.0.1:${server.port}`;
23
24 for (const [phone, w, h] of [["iphone", 390, 844], ["android", 412, 915]] as const) {
25 for (const [turn, width, height] of [["portrait", w, h], ["landscape", h, w]] as const) {
26 for (const dark of [false, true]) {
27 const page = await browser.page();
28 await page.view({ width, height, dark, scale: 2 });
29 await page.goto(`${base}/?stream=${encodeURIComponent(`${base}/no-camera/stream`)}`);
30 await page.until(`document.querySelectorAll("#joints .j").length === 6`);
31 await page.until(`document.querySelector("#pane3 canvas")`, 15_000).catch(() => {});
32 await Bun.sleep(1500); // the meshes
33 const name = (state: string) => join(out, `${phone}-${turn}-${dark ? "dark" : "light"}-${state}.png`);
34 writeFileSync(name("tray"), await page.shot());
35 if (turn === "portrait") {
36 await page.eval(`document.getElementById("fold").click()`);
37 await Bun.sleep(200);
38 writeFileSync(name("folded"), await page.shot());
39 await page.eval(`document.getElementById("fold").click()`);
40 }
41 await page.eval(`document.getElementById("stop").dispatchEvent(new PointerEvent("pointerdown", { button: 0, bubbles: true }))`);
42 await page.until(`!document.getElementById("notes").hidden`);
43 await Bun.sleep(400);
44 writeFileSync(name("stopped"), await page.shot());
45 await page.eval(`document.getElementById("notes").click()`);
46 await Bun.sleep(200);
47 writeFileSync(name("notes"), await page.shot());
48 await page.eval(`fetch("/resume", { method: "POST", body: "{}" })`);
49 console.log(`${phone} ${turn} ${dark ? "dark" : "light"}`);
50 }
51 }
52 }
53 await browser.close();
54 await Promise.race([server.close(), Bun.sleep(500)]);
55 console.log(`screenshots in ${out}`);
Software/so101/src/browser.ts added +125
@@ -0,0 +1,125 @@
1 /**
2 * A headless Chromium driven over the DevTools protocol, for the layout test (src/layout.test.ts) and the screenshots
3 * (bench/shots.ts); the app never loads it. No dependency: the browser the machine has, a WebSocket and a few commands.
4 * `launch` resolves null when there is no browser, or when it cannot start (a sandbox without sockets).
5 */
6 import { existsSync, mkdtempSync, rmSync } from "node:fs";
7 import { tmpdir } from "node:os";
8 import { join } from "node:path";
9
10 const BROWSERS = [process.env.CHROMIUM, "/usr/bin/chromium", "/usr/bin/chromium-browser", "/usr/bin/google-chrome", "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"];
11
12 export type View = { width: number; height: number; mobile?: boolean; dark?: boolean; scale?: number };
13
14 export type Page = {
15 /** The viewport: a phone (touch, portrait or landscape by its shape) or not, light or dark. */
16 view(v: View): Promise<void>;
17 /** Load a page and wait for its load event. */
18 goto(url: string): Promise<void>;
19 /** An expression's value, awaited when it is a promise; a thrown error comes back as one. */
20 eval<T>(expression: string): Promise<T>;
21 /** Wait until an expression is truthy. */
22 until(expression: string, timeoutMs?: number): Promise<void>;
23 /** A PNG of the viewport. */
24 shot(): Promise<Uint8Array>;
25 };
26
27 export type Browser = { page(): Promise<Page>; close(): Promise<void> };
28
29 type Message = { id?: number; method?: string; params?: Record<string, unknown>; sessionId?: string; result?: Record<string, unknown>; error?: { message: string } };
30
31 /** Where the browser says its DevTools listen; the rest of its stderr is drained so it never blocks on a full pipe. */
32 async function devtoolsUrl(stream: ReadableStream<Uint8Array>): Promise<string | null> {
33 const reader = stream.getReader(), decoder = new TextDecoder();
34 let text = "";
35 for (;;) {
36 const { value, done } = await reader.read();
37 if (done) return null;
38 text += decoder.decode(value, { stream: true });
39 const url = /DevTools listening on (ws:\/\/\S+)/.exec(text)?.[1];
40 if (url) {
41 void (async () => { while (!(await reader.read().catch(() => ({ done: true }))).done); })();
42 return url;
43 }
44 }
45 }
46
47 export async function launch(timeoutMs = 15_000): Promise<Browser | null> {
48 const bin = BROWSERS.find((p): p is string => Boolean(p) && existsSync(p!));
49 if (!bin) return null;
50 const profile = mkdtempSync(join(tmpdir(), "so101-chromium-"));
51 const proc = Bun.spawn([bin, "--headless=new", "--no-sandbox", "--no-first-run", "--no-default-browser-check", "--disable-dev-shm-usage", "--disable-crash-reporter",
52 "--hide-scrollbars", "--mute-audio", "--use-angle=swiftshader", "--enable-unsafe-swiftshader", `--user-data-dir=${profile}`, "--remote-debugging-port=0", "about:blank"],
53 { stdout: "ignore", stderr: "pipe", env: { ...process.env, HOME: profile } });
54 const gone = () => { try { proc.kill(); } catch {} rmSync(profile, { recursive: true, force: true }); };
55 const url = await Promise.race([devtoolsUrl(proc.stderr), Bun.sleep(timeoutMs).then(() => null)]);
56 if (!url) return gone(), null;
57 const ws = new WebSocket(url);
58 if (!(await new Promise<boolean>((resolve) => { ws.onopen = () => resolve(true); ws.onerror = () => resolve(false); }))) return gone(), null;
59 let next = 0;
60 const pending = new Map<number, { resolve: (r: Record<string, unknown>) => void; reject: (e: Error) => void }>();
61 const events: Array<(m: Message) => void> = [];
62 ws.onmessage = (e) => {
63 const m = JSON.parse(String(e.data)) as Message;
64 if (m.id !== undefined) {
65 const p = pending.get(m.id);
66 pending.delete(m.id);
67 if (m.error) p?.reject(new Error(m.error.message)); else p?.resolve(m.result ?? {});
68 } else for (const f of [...events]) f(m);
69 };
70 const send = (method: string, params: Record<string, unknown> = {}, sessionId?: string) => new Promise<Record<string, unknown>>((resolve, reject) => {
71 const id = ++next;
72 pending.set(id, { resolve, reject });
73 ws.send(JSON.stringify({ id, method, params, ...(sessionId ? { sessionId } : {}) }));
74 });
75
76 async function page(): Promise<Page> {
77 const { targetId } = await send("Target.createTarget", { url: "about:blank" });
78 const { sessionId } = (await send("Target.attachToTarget", { targetId, flatten: true })) as { sessionId: string };
79 const on = (method: string, params: Record<string, unknown> = {}) => send(method, params, sessionId);
80 await on("Page.enable");
81 await on("Runtime.enable");
82 const self: Page = {
83 async view({ width, height, mobile = true, dark = false, scale = 1 }) {
84 await on("Emulation.setDeviceMetricsOverride", { width, height, deviceScaleFactor: scale, mobile, screenOrientation: width > height ? { type: "landscapePrimary", angle: 90 } : { type: "portraitPrimary", angle: 0 } });
85 await on("Emulation.setTouchEmulationEnabled", { enabled: mobile, maxTouchPoints: mobile ? 5 : 0 });
86 await on("Emulation.setEmulatedMedia", { features: [{ name: "prefers-color-scheme", value: dark ? "dark" : "light" }] });
87 },
88 async goto(url) {
89 const loaded = new Promise<void>((resolve) => {
90 const f = (m: Message) => { if (m.sessionId === sessionId && m.method === "Page.loadEventFired") { events.splice(events.indexOf(f), 1); resolve(); } };
91 events.push(f);
92 });
93 await on("Page.navigate", { url });
94 await Promise.race([loaded, Bun.sleep(15_000)]);
95 },
96 async eval<T>(expression: string) {
97 const r = (await on("Runtime.evaluate", { expression, returnByValue: true, awaitPromise: true })) as { result: { value?: T }; exceptionDetails?: { text: string; exception?: { description?: string } } };
98 if (r.exceptionDetails) throw new Error(r.exceptionDetails.exception?.description ?? r.exceptionDetails.text);
99 return r.result.value as T;
100 },
101 async until(expression, timeoutMs = 10_000) {
102 const end = Date.now() + timeoutMs;
103 while (!(await self.eval<boolean>(`Boolean(${expression})`))) {
104 if (Date.now() > end) throw new Error(`timed out waiting for ${expression}`);
105 await Bun.sleep(50);
106 }
107 },
108 async shot() {
109 const { data } = (await on("Page.captureScreenshot", { format: "png" })) as { data: string };
110 return Buffer.from(data, "base64");
111 },
112 };
113 return self;
114 }
115
116 return {
117 page,
118 async close() {
119 await send("Browser.close").catch(() => {});
120 ws.close();
121 await Promise.race([proc.exited, Bun.sleep(2000)]);
122 gone();
123 },
124 };
125 }
Software/so101/src/layout.test.ts added +71
@@ -0,0 +1,71 @@
1 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
2 import { mkdtempSync } from "node:fs";
3 import { tmpdir } from "node:os";
4 import { join } from "node:path";
5 import { launch, type Browser } from "./browser.ts";
6 import { FeetechBus } from "./bus.ts";
7 import { FakeBus } from "./fake-bus.ts";
8 import { startServer, type RunningServer } from "./server.ts";
9
10 // The page in a real browser at a phone's size: where the camera, the model, the sliders and the notes end up.
11 // Without a Chromium that can start (none installed, or a sandbox without sockets) this suite is skipped.
12 const browser: Browser | null = await launch();
13 if (!browser) console.warn("layout.test.ts: skipped, no headless Chromium could start here");
14
15 type Box = { x: number; y: number; w: number; h: number };
16 const overlap = (a: Box, b: Box) => a.w > 0 && a.h > 0 && b.w > 0 && b.h > 0 && a.x < b.x + b.w - 0.5 && b.x < a.x + a.w - 0.5 && a.y < b.y + b.h - 0.5 && b.y < a.y + a.h - 0.5;
17 /** In the page: the boxes of the camera, the model, the strip, the tray, each slider row, and every note on screen. */
18 const BOXES = `(() => {
19 const box = (e) => { const r = e.getBoundingClientRect(); return { x: r.left, y: r.top, w: r.width, h: r.height }; };
20 const shown = (e) => e.checkVisibility ? e.checkVisibility() : e.getClientRects().length > 0;
21 return {
22 camera: box(document.querySelector(".pane.on:not(#pane3)")), model: box(document.getElementById("pane3")),
23 strip: box(document.getElementById("strip")), tray: box(document.getElementById("tab-controls")),
24 rows: [...document.querySelectorAll("#joints .j")].map(box),
25 notes: [...document.querySelectorAll("#notes, #banner, #msg, #notice, [role=alert], [role=status], dialog")].filter(shown).map((e) => ({ id: e.id, ...box(e) })),
26 };
27 })()`;
28
29 let fake: FakeBus;
30 let server: RunningServer;
31 let base: string;
32
33 describe.skipIf(!browser)("on a phone held upright", () => {
34 beforeAll(async () => {
35 fake = new FakeBus([1, 2, 3, 4, 5, 6]);
36 for (const id of [1, 2, 3, 4, 5, 6]) fake.servos.get(id)!.set([0x32, 0x03, 0x9f, 0x0d], 9); // limits 818-3487
37 server = await startServer(new FeetechBus(fake), { host: "127.0.0.1", port: 0, posesFile: join(mkdtempSync(join(tmpdir(), "so101-layout-")), "poses.json") });
38 base = `http://127.0.0.1:${server.port}`;
39 });
40 afterAll(async () => {
41 await browser?.close();
42 await Promise.race([server?.close(), Bun.sleep(500)]);
43 });
44
45 for (const [width, height] of [[390, 844], [412, 915]] as const) {
46 for (const dark of [false, true]) {
47 test(`${width}x${height} ${dark ? "dark" : "light"}: the camera on top, the model under it, every slider in the tray, and no note over either`, async () => {
48 const page = await browser!.page();
49 await page.view({ width, height, dark });
50 await page.goto(`${base}/?stream=${encodeURIComponent(`${base}/no-camera/stream`)}`);
51 await page.until(`document.querySelectorAll(".pane.on").length >= 2 && document.querySelectorAll("#joints .j").length === 6`);
52 let b = await page.eval<{ camera: Box; model: Box; strip: Box; tray: Box; rows: Box[]; notes: Array<Box & { id: string }> }>(BOXES);
53 expect(b.camera.y).toBeLessThan(1);
54 expect(b.camera.w).toBeCloseTo(width, 0);
55 expect(Math.abs(b.camera.h - (width * 9) / 16)).toBeLessThan(2);
56 expect(b.model.y).toBeGreaterThanOrEqual(b.camera.y + b.camera.h - 0.5); // under it, both whole
57 expect(b.model.h).toBeGreaterThan(150);
58 expect(b.strip.y).toBeGreaterThanOrEqual(b.model.y + b.model.h - 0.5);
59 expect(b.rows.length).toBe(6);
60 for (const row of b.rows) expect(row.y + row.h).toBeLessThanOrEqual(b.tray.y + b.tray.h + 0.5); // all six at once, none scrolled away
61 // A stop raises notes: the held banner and the reply. They show as one pill in the strip, never over the view.
62 await page.eval(`document.getElementById("stop").dispatchEvent(new PointerEvent("pointerdown", { button: 0, bubbles: true }))`);
63 await page.until(`!document.getElementById("notes").hidden`);
64 b = await page.eval(BOXES);
65 expect(b.notes.map((n) => n.id)).toContain("notes");
66 for (const n of b.notes) for (const view of [b.camera, b.model]) expect({ note: n.id, overlaps: overlap(n, view) }).toEqual({ note: n.id, overlaps: false });
67 await page.eval(`fetch("/resume", { method: "POST", body: "{}" })`);
68 });
69 }
70 }
71 });
Software/so101/src/relay.test.ts +3 −3
@@ -161,17 +161,17 @@ describe("relay", () => {
161161 expect(page.headers.get("content-encoding")).toBe("gzip");
162162 expect(page.headers.get("x-content-type-options")).toBe("nosniff");
163163 const html = gunzipSync(Buffer.from(await page.arrayBuffer())).toString();
164 const module = /\/ui\/[0-9a-f]{12}\/reach\.js/.exec(html)?.[0];
164 const module = /\/ui\/[0-9a-f]{12}\/grab\.js/.exec(html)?.[0];
165165 expect(module).toBeDefined();
166166 const js = await raw(module!);
167167 expect(js.status).toBe(200);
168168 expect(js.headers.get("cache-control")).toContain("immutable");
169169 expect(js.headers.get("content-encoding")).toBe("gzip");
170 expect(gunzipSync(Buffer.from(await js.arrayBuffer())).toString()).toContain("export function reachStep");
170 expect(gunzipSync(Buffer.from(await js.arrayBuffer())).toString()).toContain("export class Grab");
171171 // A client that asks for nothing compressed gets the plain bytes.
172172 const plain = await fetch(`${base}${module}`, { headers: { "accept-encoding": "identity" }, decompress: false } as RequestInit);
173173 expect(plain.headers.get("content-encoding")).toBeNull();
174 expect(await plain.text()).toContain("export function reachStep");
174 expect(await plain.text()).toContain("export class Grab");
175175 });
176176
177177 test("set: status, poses and /follow go to the arm server; the camera's bytes come through", async () => {
Software/so101/src/server.test.ts +26 −8
@@ -160,7 +160,7 @@ describe("viewer page", () => {
160160 }
161161 expect(bytes).toBeLessThan(50_000);
162162 // three.js, the leader's mapping, the calibration panel and the Machine panel are imported where they are used, not up front.
163 for (const lazy of ["three", "/leader-map.js", "calibration.js", "reach.js", "machine-panel.js", "machine.js"]) expect([...upfront].some((p) => p === lazy || p.endsWith(`/${lazy}`))).toBe(false);
163 for (const lazy of ["three", "/leader-map.js", "calibration.js", "grab.js", "machine-panel.js", "machine.js"]) expect([...upfront].some((p) => p === lazy || p.endsWith(`/${lazy}`))).toBe(false);
164164 expect(script).toContain('import("/leader-map.js")');
165165 expect(script).toMatch(/import\("\/ui\/[0-9a-f]{12}\/calibration\.js"\)/);
166166 expect(script).toMatch(/import\("\/ui\/[0-9a-f]{12}\/machine-panel\.js"\)/);
@@ -176,7 +176,7 @@ describe("viewer page", () => {
176176 expect(r.headers.get("content-type")).toBe("text/javascript");
177177 expect(r.headers.get("cache-control")).toContain("immutable");
178178 const version = path!.split("/")[2];
179 expect((await fetch(`${base}/ui/${version}/reach.test.ts`)).status).toBe(404);
179 expect((await fetch(`${base}/ui/${version}/grab.test.ts`)).status).toBe(404);
180180 expect((await fetch(`${base}/ui/${version}/nothing.js`)).status).toBe(404);
181181 expect((await fetch(`${base}/ui/${version}/..%2Fserver.ts`)).status).toBe(404);
182182 });
@@ -191,19 +191,37 @@ describe("viewer page", () => {
191191 expect(page).toMatch(/#tabs \{[^}]*var\(--pill\)/);
192192 });
193193
194 test("serves the touch-driving modules, and the phone layout keeps the browser's gestures off the controls", async () => {
194 test("notes never cover the camera or the model: they sit in a sheet the strip's pill opens", async () => {
195195 const page = await (await fetch(`${base}/`)).text();
196 for (const name of ["drive", "frames"]) {
196 const view = page.slice(page.indexOf('<div id="app">'), page.indexOf('<section id="strip"'));
197 const sheet = page.slice(page.indexOf('<dialog id="notesDlg"'), page.indexOf("</dialog>", page.indexOf('<dialog id="notesDlg"')));
198 for (const id of ["banner", "msg", "notice"]) {
199 expect(view).not.toContain(`id="${id}"`);
200 expect(sheet).toContain(`id="${id}"`);
201 }
202 expect(view).not.toMatch(/role="(alert|status)"/);
203 expect(page.slice(page.indexOf('<section id="strip"'), page.indexOf("</section>", page.indexOf('<section id="strip"')))).toContain('<button id="notes"');
204 // A phone held upright stacks them: the camera at 16:9 on top, the model under it.
205 const upright = page.slice(page.indexOf("@media (max-width: 640px) and (orientation: portrait)"));
206 expect(upright).toMatch(/#stage \{ flex-direction: column; \}/);
207 expect(upright).toMatch(/\.pane:not\(#pane3\) \{ flex: none; aspect-ratio: 16 \/ 9; \}/);
208 });
209
210 test("drives by touching the model: its module is served, the canvas keeps the browser's gestures, and no joystick remains", async () => {
211 const page = await (await fetch(`${base}/`)).text();
212 for (const name of ["frames", "grab"]) {
197213 const path = new RegExp(`/ui/[0-9a-f]{12}/${name}\\.js`).exec(page)?.[0];
198214 expect(path).toBeDefined();
199215 const r = await fetch(`${base}${path}`);
200216 expect(r.status).toBe(200);
201217 expect(r.headers.get("cache-control")).toContain("immutable");
202218 }
203 expect(page).toMatch(/#lever \{[^}]*touch-action: none/);
204 expect(page).toMatch(/\.stick \{[^}]*touch-action: none/);
205 expect(page).toMatch(/#grips button \{[^}]*min-height: (7[2-9]|[89]\d)px/);
206 expect(page).toMatch(/#chips button small \{[^}]*white-space: nowrap/); // a long label never wraps and shifts the row
219 expect(page).toContain('canvas.style.touchAction = "none"');
220 expect(page).toMatch(/import\("\/ui\/[0-9a-f]{12}\/grab\.js"\)/); // with three.js, after the first view
221 // The joysticks are gone, on a phone and on a wide screen alike, and so are the one-joint lever and its chips.
222 for (const gone of ['id="sticks"', 'class="stick"', 'id="pad"', 'id="reach"', 'id="lever"', 'id="chips"', "drive.js", "reach.js", "joystick"]) expect(page).not.toContain(gone);
223 // Every joint has its slider at once.
224 expect(page).toContain('<div id="joints"></div>');
207225 // One STOP in every layout: one stop control in the markup, and no layout hides it. Resume is its own control.
208226 expect(page.match(/<button[^>]*data-stop/g)?.length).toBe(1);
209227 expect(page).not.toMatch(/#stop \{[^}]*display: none/);
Software/so101/src/ui/calibration.js +1 −1
@@ -35,7 +35,7 @@ const ARTICLE = `
3535 <p>This page drives an <a href="https://github.com/TheRobotStudio/SO-ARM100" target="_blank" rel="noopener">SO-101</a>, the open robot arm by The Robot Studio, with the calibration conventions of <a href="https://huggingface.co/docs/lerobot/so101" target="_blank" rel="noopener">Hugging Face LeRobot</a>. The 3D view uses the arm's own URDF through <a href="https://github.com/gkjohnson/urdf-loaders" target="_blank" rel="noopener">urdf-loader</a> and three.js. The code, the build notes and the quest are in the repository's <code>Software/</code> folder.</p>
3636 </article>`;
3737
38 /** The page's document: this module only runs there (the type check has no DOM, as for src/ui/drive.js). @type {any} */
38 /** The page's document: this module only runs there (the type check has no DOM, as for src/ui/intent.js). @type {any} */
3939 const doc = /** @type {any} */ (globalThis).document;
4040
4141 /** @param {string} tag @param {Record<string, unknown>} [props] @param {...any} children @returns {any} */
Software/so101/src/ui/drive.js deleted −356
@@ -1,356 +0,0 @@
1 /**
2 * Touch driving: a finger's deflection is a joint's velocity, never its position. Plain ESM, served to the page at
3 * /ui/ and imported by the tests. The wire stays what /follow always took, a target in ticks per joint: each held
4 * joint's target runs ahead at the deflection's share of the jog speed, and the moment the last finger on it lifts
5 * the target stops where it is and goes out at once. That frame is the zero: the arm's governor settles on it.
6 * The server's limits still clamp every frame; this only keeps the page from asking for more.
7 */
8
9 /** The tick direction that raises the gripper, swings the base right, rolls the wrist or opens the jaw (from the URDF). */
10 export const UP = /** @type {Record<number, number>} */ ({ 1: 1, 2: -1, 3: -1, 4: -1, 5: 1, 6: 1 });
11
12 /**
13 * A raw deflection (-1..1) through a dead band and a gentle curve: nothing near the middle, fine control
14 * a little way out, full speed at the edge.
15 * @param {number} v
16 * @param {number} [dead]
17 */
18 export function shape(v, dead = 0.12) {
19 if (!Number.isFinite(v)) return 0;
20 const a = Math.min(1, Math.abs(v));
21 if (a <= dead) return 0;
22 return Math.sign(v) * ((a - dead) / (1 - dead)) ** 1.5;
23 }
24
25 /** @typedef {Record<number, { min: number, max: number }>} Limits */
26 /** @typedef {Record<number, number>} Frame */
27
28 /**
29 * Whether a finger may drive now: it is this person's turn, the arm is not stopped and not reconnecting, and its
30 * limits are known. The page asks before a press starts and on every status poll, letting go of whatever is held
31 * the moment the answer turns false.
32 * @param {{ mayDrive: boolean, estop?: unknown, held?: unknown, reconnecting?: unknown, limits?: Limits | null }} state
33 */
34 export function canDrive({ mayDrive, estop, held, reconnecting, limits }) {
35 return Boolean(mayDrive) && !estop && !held && !reconnecting && Object.keys(limits ?? {}).length > 0;
36 }
37
38 /**
39 * The velocity integrator. Each holder (a pointer, a key) holds some joints at some deflections; a joint's velocity is
40 * the sum of its holders', capped at 1. `step` advances the held joints and returns their targets; `release` drops a
41 * holder and returns the targets of the joints it was moving, which is what stops them.
42 */
43 export class Drive {
44 /** @param {{ margin?: number, lead?: number }} [options] */
45 constructor(options = {}) {
46 /** Ticks kept clear of each end of a joint's range, as the page's sliders always did. */
47 this.margin = options.margin ?? 30;
48 /** How far a target may run ahead of where the arm last said it was: a slow arm (a speed cap) never builds up a backlog. */
49 this.lead = options.lead ?? 0.6;
50 /** @type {Map<string, Map<number, number>>} holder → joint → deflection */
51 this.holders = new Map();
52 /** @type {Map<number, number>} the targets of joints being driven, in fractional ticks */
53 this.targets = new Map();
54 /** @type {Map<number, number>} where the arm last said each joint was */
55 this.present = new Map();
56 }
57
58 /** Where the arm is. A joint nobody holds starts from here when it is next held. @param {Iterable<readonly [number, number]>} positions */
59 seed(positions) {
60 for (const [id, p] of positions) if (Number.isFinite(p)) this.present.set(id, p);
61 }
62
63 /** Whether any joint is held off the middle. */
64 get moving() {
65 for (const axes of this.holders.values()) for (const v of axes.values()) if (v !== 0) return true;
66 return false;
67 }
68
69 /** The joints held, by anyone. */
70 held() {
71 const out = new Set();
72 for (const axes of this.holders.values()) for (const id of axes.keys()) out.add(id);
73 return out;
74 }
75
76 /**
77 * A holder's joints and deflections, replacing what it held before; a joint it lets go of this way stops like a release.
78 * @param {string} holder
79 * @param {Record<number, number>} axes joint id → deflection, -1..1, already shaped
80 * @returns {Frame | null} targets for joints that stopped, to send at once
81 */
82 hold(holder, axes) {
83 const before = this.holders.get(holder);
84 const next = new Map();
85 for (const [k, v] of Object.entries(axes)) {
86 const id = Number(k);
87 if (!Number.isInteger(id) || !Number.isFinite(v)) continue;
88 if (!this.targets.has(id)) {
89 const p = this.present.get(id);
90 if (p === undefined) continue; // no position yet: nothing to run ahead of
91 this.targets.set(id, p);
92 }
93 next.set(id, Math.max(-1, Math.min(1, v)));
94 }
95 this.holders.set(holder, next);
96 const dropped = before ? [...before.keys()].filter((id) => !next.has(id)) : [];
97 return this.#stop(dropped);
98 }
99
100 /** @param {string} holder @returns {Frame | null} the zero: the targets of the joints this holder moved, now still */
101 release(holder) {
102 const axes = this.holders.get(holder);
103 if (!axes) return null;
104 this.holders.delete(holder);
105 return this.#stop([...axes.keys()]);
106 }
107
108 /** Every holder at once: the window lost focus, the page was hidden. @returns {Frame | null} */
109 releaseAll() {
110 const ids = [...this.held()];
111 this.holders.clear();
112 return this.#stop(ids);
113 }
114
115 /**
116 * One small step of one joint, for a tap on − or +.
117 * @param {number} id @param {number} ticks @param {Limits} limits
118 * @returns {Frame | null}
119 */
120 nudge(id, ticks, limits) {
121 const from = this.targets.get(id) ?? this.present.get(id);
122 if (from === undefined || !limits[id]) return null;
123 const t = this.#clamp(id, from + ticks, limits);
124 if (!this.held().has(id)) this.targets.delete(id); else this.targets.set(id, t);
125 this.present.set(id, t); // the next nudge goes on from here, not from a stale reading
126 return { [id]: Math.round(t) };
127 }
128
129 /**
130 * Advance every held joint by `dt` seconds at up to `speed` ticks/s.
131 * @param {number} dt @param {number} speed @param {Limits} limits
132 * @returns {{ frame: Frame | null, stops: number[] }} the targets to send, and joints that just met an end of their range
133 */
134 step(dt, speed, limits) {
135 /** @type {Map<number, number>} */
136 const v = new Map();
137 for (const axes of this.holders.values()) for (const [id, d] of axes) v.set(id, (v.get(id) ?? 0) + d);
138 /** @type {Frame} */
139 const frame = {};
140 const stops = [];
141 let any = false;
142 for (const [id, sum] of v) {
143 const t0 = this.targets.get(id);
144 if (t0 === undefined || !limits[id]) continue;
145 const vel = Math.max(-1, Math.min(1, sum));
146 if (vel === 0) continue;
147 const want = t0 + vel * speed * dt;
148 let t = this.#clamp(id, want, limits);
149 const ended = t !== want;
150 // Never more than `lead` seconds of travel ahead of the arm: a slow arm (a speed cap) is waited for, not queued
151 // up behind. Only further travel is held back; a target never steps back toward a stale reading.
152 const p = this.present.get(id);
153 if (p !== undefined) {
154 const room = Math.max(speed * this.lead, 20);
155 t = vel > 0 ? Math.min(t, Math.max(t0, p + room)) : Math.max(t, Math.min(t0, p - room));
156 }
157 if (t === t0) continue; // nowhere to go: nothing to send
158 if (ended) stops.push(id);
159 this.targets.set(id, t);
160 frame[id] = Math.round(t);
161 any = true;
162 }
163 return { frame: any ? frame : null, stops };
164 }
165
166 /** @param {number} id @param {number} t @param {Limits} limits */
167 #clamp(id, t, limits) {
168 const l = limits[id];
169 if (!l) return t;
170 return Math.max(l.min + this.margin, Math.min(l.max - this.margin, t));
171 }
172
173 /** @param {number[]} ids */
174 #stop(ids) {
175 const still = this.held();
176 /** @type {Frame} */
177 const frame = {};
178 let any = false;
179 for (const id of ids) {
180 if (still.has(id)) continue; // another finger still drives it
181 const t = this.targets.get(id);
182 this.targets.delete(id);
183 if (t === undefined) continue;
184 frame[id] = Math.round(t);
185 this.present.set(id, t); // until the arm says otherwise, it is where it was told to stop
186 any = true;
187 }
188 return any ? frame : null;
189 }
190 }
191
192 /**
193 * The page's driver: a Drive, a clock at the send rate while anything is held, and the ways a press ends.
194 * `send` takes frames at the clock's rate (the page may coalesce them); `flush` must put a frame on the wire now,
195 * because it is the zero; `end` says nothing is held any more.
196 */
197 export class Driver {
198 /**
199 * @param {{
200 * send: (frame: Frame) => void,
201 * flush: (frame: Frame) => void,
202 * end: () => void,
203 * begin?: () => boolean | void,
204 * limits: () => Limits,
205 * speed: () => number,
206 * hz?: number,
207 * onStop?: (id: number) => void,
208 * timers?: { every: (fn: () => void, ms: number) => unknown, cancel: (handle: unknown) => void, now: () => number },
209 * }} options
210 */
211 constructor(options) {
212 this.o = options;
213 this.drive = new Drive();
214 this.hz = Math.max(20, Math.min(30, options.hz ?? 25));
215 this.timers = options.timers ?? { every: (fn, ms) => setInterval(fn, ms), cancel: (h) => clearInterval(/** @type {any} */ (h)), now: () => performance.now() };
216 /** @type {unknown} */
217 this.clock = null;
218 this.last = 0;
219 }
220
221 /** Whether a finger (or a key) is down on anything. */
222 get active() {
223 return this.clock !== null;
224 }
225
226 /** @param {Iterable<readonly [number, number]>} positions */
227 seed(positions) {
228 this.drive.seed(positions);
229 }
230
231 /** @param {string} holder @param {Record<number, number>} axes @returns {boolean} false when driving is refused */
232 hold(holder, axes) {
233 if (!this.clock) {
234 if (this.o.begin?.() === false) return false;
235 this.last = this.timers.now();
236 this.clock = this.timers.every(() => this.tick(), 1000 / this.hz);
237 }
238 const stopped = this.drive.hold(holder, axes);
239 if (stopped) this.o.flush(stopped);
240 return true;
241 }
242
243 /** A holder lets go: its joints' zero goes out now, not at the next tick. @param {string} holder */
244 release(holder) {
245 const frame = this.drive.release(holder);
246 if (frame) this.o.flush(frame);
247 if (this.clock && this.drive.holders.size === 0) this.#idle();
248 }
249
250 /** Everything lets go at once: the window lost focus, the page was hidden. */
251 releaseAll() {
252 const frame = this.drive.releaseAll();
253 if (frame) this.o.flush(frame);
254 if (this.clock) this.#idle();
255 }
256
257 /** One step of one joint, for a tap. @param {number} id @param {number} ticks signed */
258 nudge(id, ticks) {
259 if (this.o.begin?.() === false) return;
260 const frame = this.drive.nudge(id, ticks, this.o.limits());
261 if (frame) this.o.flush(frame);
262 if (!this.clock) this.o.end();
263 }
264
265 tick() {
266 const now = this.timers.now();
267 const dt = Math.min(0.1, Math.max(0, (now - this.last) / 1000)); // a late tick never turns into a jump
268 this.last = now;
269 const { frame, stops } = this.drive.step(dt, this.o.speed(), this.o.limits());
270 if (frame) this.o.send(frame);
271 for (const id of stops) this.o.onStop?.(id);
272 }
273
274 #idle() {
275 this.timers.cancel(this.clock);
276 this.clock = null;
277 this.o.end();
278 }
279
280 /**
281 * Every way a hold can end without a pointerup: the window loses focus, the page is hidden or put away.
282 * @param {EventTarget} win @param {EventTarget & { hidden?: boolean }} doc
283 */
284 watch(win, doc) {
285 const all = () => this.releaseAll();
286 win.addEventListener("blur", all);
287 win.addEventListener("pagehide", all);
288 doc.addEventListener("visibilitychange", () => { if (doc.hidden) all(); });
289 }
290 }
291
292 /**
293 * @typedef {EventTarget & {
294 * setPointerCapture?: (id: number) => void,
295 * getBoundingClientRect: () => { left: number, top: number, width: number, height: number },
296 * classList: { toggle: (name: string, on?: boolean) => unknown },
297 * }} Surface
298 * @typedef {Event & { pointerId: number, clientX: number, clientY: number }} Pointer
299 */
300
301 /**
302 * One finger on a surface, tracked by its pointerId: `place` maps where it is to joint deflections, every way the
303 * finger can leave ends the hold, and a second finger on the same surface is ignored until the first one lifts.
304 * Two surfaces are two holders, so two thumbs drive two sets of joints independently.
305 * @param {Surface} el
306 * @param {Driver} driver
307 * @param {string} holder
308 * @param {(x: number, y: number) => Record<number, number> | null} place x, y from -1 to 1 across the surface, y up
309 * @param {{ show?: (x: number, y: number) => void, haptic?: () => void }} [view]
310 * @returns {{ end: () => void }}
311 */
312 export function bindSurface(el, driver, holder, place, view = {}) {
313 /** @type {number | null} */
314 let pointer = null;
315 /** @param {Pointer} e */
316 const at = (e) => {
317 const r = el.getBoundingClientRect();
318 const x = r.width ? ((e.clientX - r.left) / r.width) * 2 - 1 : 0;
319 const y = r.height ? 1 - ((e.clientY - r.top) / r.height) * 2 : 0;
320 return [Math.max(-1, Math.min(1, x)), Math.max(-1, Math.min(1, y))];
321 };
322 /** @param {Pointer} e */
323 const apply = (e) => {
324 const [x = 0, y = 0] = at(e);
325 view.show?.(x, y);
326 const axes = place(x, y);
327 if (!axes) return false;
328 return driver.hold(holder, axes);
329 };
330 const end = () => {
331 if (pointer === null) return;
332 pointer = null;
333 el.classList.toggle("held", false);
334 view.show?.(0, 0);
335 driver.release(holder);
336 };
337 el.addEventListener("pointerdown", (ev) => {
338 const e = /** @type {Pointer} */ (ev);
339 if (pointer !== null) return;
340 e.preventDefault();
341 pointer = e.pointerId;
342 try { el.setPointerCapture?.(e.pointerId); } catch { /* a synthetic pointer */ }
343 el.classList.toggle("held", true);
344 view.haptic?.();
345 if (apply(e) === false) end();
346 });
347 el.addEventListener("pointermove", (ev) => {
348 const e = /** @type {Pointer} */ (ev);
349 if (e.pointerId === pointer) apply(e);
350 });
351 for (const type of ["pointerup", "pointercancel", "lostpointercapture"]) {
352 el.addEventListener(type, (ev) => { if (/** @type {Pointer} */ (ev).pointerId === pointer) end(); });
353 }
354 el.addEventListener("contextmenu", (e) => e.preventDefault());
355 return { end };
356 }
Software/so101/src/ui/drive.test.ts deleted −293
@@ -1,293 +0,0 @@
1 import { describe, expect, test } from "bun:test";
2 import { bindSurface, canDrive, Drive, Driver, shape } from "./drive.js";
3
4 const LIMITS = { 1: { min: 1000, max: 3000 }, 2: { min: 1000, max: 3000 }, 3: { min: 1000, max: 3000 }, 4: { min: 1000, max: 3000 }, 6: { min: 2000, max: 3500 } };
5 const AT = [[1, 2000], [2, 2000], [3, 2000], [4, 2000], [6, 2500]] as const;
6
7 /** A driver on a hand-wound clock that records what goes on the wire, and when. */
8 function rig() {
9 const wire: Array<{ kind: "send" | "flush" | "end"; frame?: Record<number, number>; at: number }> = [];
10 let now = 0;
11 let tick: (() => void) | null = null;
12 const driver = new Driver({
13 send: (frame) => wire.push({ kind: "send", frame, at: now }),
14 flush: (frame) => wire.push({ kind: "flush", frame, at: now }),
15 end: () => wire.push({ kind: "end", at: now }),
16 limits: () => LIMITS,
17 speed: () => 300,
18 timers: { every: (fn) => ((tick = fn), 1), cancel: () => (tick = null), now: () => now },
19 });
20 driver.seed(AT);
21 /** Advance the clock by one 40 ms frame and run the tick, when one is running. */
22 const frame = () => {
23 now += 40;
24 tick?.();
25 };
26 return { driver, wire, frame, running: () => tick !== null };
27 }
28
29 /** A surface 100 px square at the origin, built on EventTarget like the page's elements. */
30 function surface() {
31 const el = Object.assign(new EventTarget(), {
32 captured: [] as number[],
33 held: false,
34 setPointerCapture(id: number) { this.captured.push(id); },
35 getBoundingClientRect: () => ({ left: 0, top: 0, width: 100, height: 100 }),
36 classList: { toggle: (_: string, on?: boolean) => (el.held = Boolean(on)) },
37 });
38 return el;
39 }
40 const pointer = (type: string, pointerId: number, clientX = 50, clientY = 50) => Object.assign(new Event(type, { cancelable: true }), { pointerId, clientX, clientY });
41
42 describe("the stick's shape", () => {
43 test("a dead band in the middle, the full speed at the edge, never more", () => {
44 expect(shape(0.05)).toBe(0);
45 expect(shape(-0.12)).toBe(0);
46 expect(shape(1)).toBe(1);
47 expect(shape(-3)).toBe(-1);
48 expect(shape(Number.NaN)).toBe(0);
49 expect(shape(0.5)).toBeGreaterThan(0);
50 expect(shape(0.5)).toBeLessThan(0.5); // fine control a little way out
51 expect(shape(-0.5)).toBe(-shape(0.5));
52 });
53 });
54
55 describe("the velocity integrator", () => {
56 test("a held joint runs ahead at the deflection's share of the speed, from where the arm is", () => {
57 const d = new Drive();
58 d.seed(AT);
59 d.hold("a", { 1: 0.5 });
60 expect(d.step(0.1, 300, LIMITS).frame).toEqual({ 1: 2015 });
61 expect(d.step(0.1, 300, LIMITS).frame).toEqual({ 1: 2030 });
62 });
63
64 test("the target stops at the range's margin and says so once", () => {
65 const d = new Drive();
66 d.seed([[1, 2960]]);
67 d.hold("a", { 1: 1 });
68 expect(d.step(0.1, 300, LIMITS)).toEqual({ frame: { 1: 2970 }, stops: [1] });
69 expect(d.step(0.1, 300, LIMITS)).toEqual({ frame: null, stops: [] });
70 });
71
72 test("never runs more than the lead ahead of an arm that lags, and never steps back toward a stale reading", () => {
73 const d = new Drive({ lead: 0.5 });
74 d.seed([[1, 2000]]);
75 d.hold("a", { 1: 1 });
76 for (let i = 0; i < 20; i++) d.step(0.1, 300, LIMITS);
77 expect(d.targets.get(1)).toBe(2150); // 0.5 s of 300 ticks/s ahead of where the arm said it was
78 d.seed([[1, 1900]]); // a reading older than the target
79 expect(d.step(0.1, 300, LIMITS).frame).toBeNull();
80 expect(d.targets.get(1)).toBe(2150);
81 });
82
83 test("a joint without a position is not driven", () => {
84 const d = new Drive();
85 d.hold("a", { 5: 1 });
86 expect(d.step(0.1, 300, LIMITS).frame).toBeNull();
87 expect(d.release("a")).toBeNull();
88 });
89
90 test("a tap nudges from the last target, inside the range", () => {
91 const d = new Drive();
92 d.seed([[6, 3450]]);
93 expect(d.nudge(6, 20, LIMITS)).toEqual({ 6: 3470 });
94 expect(d.nudge(6, 20, LIMITS)).toEqual({ 6: 3470 });
95 expect(d.nudge(6, -20, LIMITS)).toEqual({ 6: 3450 });
96 });
97 });
98
99 describe("the driver", () => {
100 test("lifting the finger sends zero within one frame: the target as it stands, flushed at once, then nothing", () => {
101 const { driver, wire, frame, running } = rig();
102 driver.hold("lever", { 2: 1 });
103 frame(); frame();
104 const last = wire.at(-1)!;
105 expect(last.kind).toBe("send");
106 const sentAt = wire.length;
107 driver.release("lever");
108 // Before any further tick of the clock: the zero is already on the wire.
109 expect(wire.slice(sentAt)).toEqual([{ kind: "flush", frame: last.frame, at: last.at }, { kind: "end", at: last.at }]);
110 expect(running()).toBe(false);
111 frame(); frame();
112 expect(wire.length).toBe(sentAt + 2); // and nothing after it
113 });
114
115 test("two thumbs drive independent joints: lifting one stops only its joints", () => {
116 const { driver, wire, frame } = rig();
117 driver.hold("left", { 1: 1, 2: 0 });
118 driver.hold("right", { 3: -1 });
119 frame();
120 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 1: 2012, 3: 1988 }, at: 40 });
121 driver.release("left");
122 expect(wire.at(-1)).toEqual({ kind: "flush", frame: { 1: 2012, 2: 2000 }, at: 40 });
123 frame();
124 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 3: 1976 }, at: 80 }); // the right thumb goes on
125 driver.release("right");
126 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 3: 1976 }, at: 80 }, { kind: "end", at: 80 }]);
127 });
128
129 test("two fingers on one joint: it stops only when both have lifted", () => {
130 const { driver, wire, frame } = rig();
131 driver.hold("lever", { 4: 1 });
132 driver.hold("plus", { 4: 1 });
133 frame();
134 driver.release("plus");
135 expect(wire.at(-1)!.kind).toBe("send");
136 frame();
137 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 4: 2024 }, at: 80 });
138 driver.release("lever");
139 expect(wire.at(-1)!.kind).toBe("end");
140 });
141
142 test("blur, pagehide and a hidden page stop every joint at once", () => {
143 for (const how of ["blur", "pagehide", "hidden"] as const) {
144 const { driver, wire, frame, running } = rig();
145 const win = new EventTarget(), doc = Object.assign(new EventTarget(), { hidden: false });
146 driver.watch(win, doc);
147 driver.hold("left", { 1: 1 });
148 driver.hold("right", { 3: 1 });
149 frame();
150 if (how === "hidden") { doc.hidden = true; doc.dispatchEvent(new Event("visibilitychange")); }
151 else win.dispatchEvent(new Event(how));
152 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 1: 2012, 3: 2012 }, at: 40 }, { kind: "end", at: 40 }]);
153 expect(running()).toBe(false);
154 }
155 });
156
157 test("a hidden page that was not driving sends nothing", () => {
158 const { driver, wire } = rig();
159 const win = new EventTarget(), doc = Object.assign(new EventTarget(), { hidden: true });
160 driver.watch(win, doc);
161 doc.dispatchEvent(new Event("visibilitychange"));
162 win.dispatchEvent(new Event("blur"));
163 expect(wire).toEqual([]);
164 });
165
166 test("refused by the page (not your turn), a press starts nothing", () => {
167 const wire: unknown[] = [];
168 const driver = new Driver({ send: (f) => wire.push(f), flush: (f) => wire.push(f), end: () => wire.push("end"), begin: () => false, limits: () => LIMITS, speed: () => 300 });
169 driver.seed(AT);
170 expect(driver.hold("x", { 1: 1 })).toBe(false);
171 expect(driver.active).toBe(false);
172 driver.nudge(1, 20);
173 expect(wire).toEqual([]);
174 });
175
176 test("a tap nudges and ends at once", () => {
177 const { driver, wire } = rig();
178 driver.nudge(6, -20);
179 expect(wire).toEqual([{ kind: "flush", frame: { 6: 2480 }, at: 0 }, { kind: "end", at: 0 }]);
180 });
181 });
182
183 describe("a surface under a finger", () => {
184 const lever = (y: number) => ({ 2: y });
185
186 test("pointerup sends the zero before the next frame, and the surface lets go", () => {
187 const { driver, wire, frame } = rig();
188 const el = surface();
189 bindSurface(el, driver, "lever", (_, y) => lever(y));
190 el.dispatchEvent(pointer("pointerdown", 7, 50, 0)); // the top edge: full speed up
191 expect(el.captured).toEqual([7]);
192 expect(el.held).toBe(true);
193 frame();
194 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 2: 2012 }, at: 40 });
195 el.dispatchEvent(pointer("pointerup", 7));
196 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 2: 2012 }, at: 40 }, { kind: "end", at: 40 }]);
197 expect(el.held).toBe(false);
198 });
199
200 for (const type of ["pointercancel", "lostpointercapture"]) {
201 test(`${type} ends the hold like a lift`, () => {
202 const { driver, wire, frame, running } = rig();
203 const el = surface();
204 bindSurface(el, driver, "lever", (_, y) => lever(y));
205 el.dispatchEvent(pointer("pointerdown", 3, 50, 100));
206 frame();
207 el.dispatchEvent(pointer(type, 3));
208 expect(wire.at(-1)!.kind).toBe("end");
209 expect(running()).toBe(false);
210 });
211 }
212
213 test("a second finger on the same surface neither steers it nor ends it", () => {
214 const { driver, wire, frame } = rig();
215 const el = surface();
216 bindSurface(el, driver, "lever", (_, y) => lever(y));
217 el.dispatchEvent(pointer("pointerdown", 1, 50, 0));
218 el.dispatchEvent(pointer("pointerdown", 2, 50, 100));
219 el.dispatchEvent(pointer("pointermove", 2, 50, 100));
220 el.dispatchEvent(pointer("pointerup", 2));
221 frame();
222 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 2: 2012 }, at: 40 });
223 });
224
225 test("two sticks under two thumbs, each with its own pointer, drive their own joints", () => {
226 const { driver, wire, frame } = rig();
227 const left = surface(), right = surface();
228 bindSurface(left, driver, "left", (x, y) => ({ 1: x, 2: y }));
229 bindSurface(right, driver, "right", (x, y) => ({ 4: x, 3: y }));
230 left.dispatchEvent(pointer("pointerdown", 11, 100, 50)); // full right
231 right.dispatchEvent(pointer("pointerdown", 12, 50, 100)); // full down
232 frame();
233 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 1: 2012, 3: 1988 }, at: 40 });
234 right.dispatchEvent(pointer("pointerup", 11)); // the other thumb's id on this stick: ignored
235 left.dispatchEvent(pointer("pointerup", 11));
236 expect(wire.at(-1)).toEqual({ kind: "flush", frame: { 1: 2012, 2: 2000 }, at: 40 });
237 frame();
238 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 3: 1976 }, at: 80 });
239 });
240
241 test("a press the page refuses lets go of the surface at once", () => {
242 const driver = new Driver({ send: () => {}, flush: () => {}, end: () => {}, begin: () => false, limits: () => LIMITS, speed: () => 300 });
243 const el = surface();
244 bindSurface(el, driver, "lever", (_, y) => lever(y));
245 el.dispatchEvent(pointer("pointerdown", 1, 50, 0));
246 expect(el.held).toBe(false);
247 });
248
249 test("the arm drops while a thumb is held: the poll's releaseAll flushes the zero, and the still-pressed stick sends nothing more", () => {
250 const wire: Array<{ kind: string; frame?: Record<number, number> }> = [];
251 let tick: (() => void) | null = null, now = 0;
252 const state = { mayDrive: true, estop: false, reconnecting: false, limits: LIMITS };
253 const driver = new Driver({
254 send: (frame) => wire.push({ kind: "send", frame }),
255 flush: (frame) => wire.push({ kind: "flush", frame }),
256 end: () => wire.push({ kind: "end" }),
257 begin: () => canDrive(state),
258 limits: () => LIMITS,
259 speed: () => 300,
260 timers: { every: (fn) => ((tick = fn), 1), cancel: () => (tick = null), now: () => now },
261 });
262 driver.seed(AT);
263 const stick = surface();
264 bindSurface(stick, driver, "left", (x, y) => ({ 1: x, 2: y }));
265 stick.dispatchEvent(pointer("pointerdown", 7, 100, 50));
266 now += 40; tick!();
267 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 1: 2012 } });
268 // The status poll: reconnecting, so driving is refused and everything held lets go.
269 state.reconnecting = true;
270 if (!canDrive(state)) driver.releaseAll();
271 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 1: 2012, 2: 2000 } }, { kind: "end" }]);
272 expect(tick).toBeNull();
273 const after = wire.length;
274 stick.dispatchEvent(pointer("pointermove", 7, 100, 0)); // the thumb is still down and moving
275 expect(wire.length).toBe(after);
276 expect(tick).toBeNull();
277 });
278 });
279
280 describe("canDrive", () => {
281 const ok = { mayDrive: true, estop: false, reconnecting: false, limits: LIMITS };
282 test("drives on your turn, with the arm up and its limits known", () => {
283 expect(canDrive(ok)).toBe(true);
284 });
285 test("refuses when it is not your turn, the arm is stopped, the arm is reconnecting, or no limits have come", () => {
286 expect(canDrive({ ...ok, mayDrive: false })).toBe(false);
287 expect(canDrive({ ...ok, estop: true })).toBe(false);
288 expect(canDrive({ ...ok, held: { by: "ann", at: "2026-10-20T22:05:00Z" } })).toBe(false); // STOP, until someone resumes
289 expect(canDrive({ ...ok, reconnecting: true })).toBe(false);
290 expect(canDrive({ ...ok, limits: {} })).toBe(false);
291 expect(canDrive({ ...ok, limits: null })).toBe(false);
292 });
293 });
Software/so101/src/ui/grab.js added +106
@@ -0,0 +1,106 @@
1 /**
2 * Driving by touching the 3D model. One finger (or the mouse) that lands on a part of the arm takes that part's
3 * joint, and only that joint goes to the arm while it is held; a finger that lands on empty space, a second finger,
4 * a pinch, are the view's (orbit and zoom) and never send anything. Lifting the finger sends the joint's last target
5 * at once, and so do pointercancel, a lost capture, a hidden page and a blurred window. Plain ESM, served at /ui/
6 * and imported by the tests: the page hands it plain pointer data and the model's hooks.
7 */
8
9 /** @typedef {{ id: number, x: number, y: number }} Point */
10 /** @typedef {Record<number, number>} Frame */
11
12 export class Grab {
13 /**
14 * @param {{
15 * allowed: () => boolean,
16 * pick: (x: number, y: number) => string | null,
17 * drag: (x: number, y: number) => void,
18 * drop: (x: number, y: number) => void,
19 * ticks: (joint: string, angle: number) => [number, number] | null,
20 * send: (frame: Frame) => void,
21 * flush: (frame: Frame) => void,
22 * begin?: (joint: string) => void,
23 * end?: (joint: string) => void,
24 * }} io allowed: whether this page may drive now; pick: the joint under a point, taken (null: none); drag/drop: the
25 * held finger moved/lifted there; ticks: a joint's angle as [servo id, target ticks], or null when it cannot go
26 */
27 constructor(io) {
28 this.io = io;
29 /** @type {number | null} the pointer holding a joint */
30 this.pointer = null;
31 /** @type {string | null} the joint it holds */
32 this.joint = null;
33 /** @type {Frame | null} the last target sent for it */
34 this.last = null;
35 /** @type {[number, number]} */
36 this.at = [0, 0];
37 }
38
39 /**
40 * A finger or the mouse comes down. True when the model keeps it: it took a joint, or a joint is already held and
41 * this second finger must neither orbit nor drive. False: the view's, to orbit or zoom.
42 * @param {Point} p
43 */
44 down(p) {
45 if (this.pointer !== null) return true;
46 if (!this.io.allowed()) return false;
47 const joint = this.io.pick(p.x, p.y);
48 if (!joint) return false;
49 this.pointer = p.id; this.joint = joint; this.last = null; this.at = [p.x, p.y];
50 this.io.begin?.(joint);
51 return true;
52 }
53
54 /** @param {Point} p */
55 move(p) {
56 if (p.id !== this.pointer) return false;
57 this.at = [p.x, p.y];
58 this.io.drag(p.x, p.y);
59 return true;
60 }
61
62 /** The finger lifted, or was cancelled or lost. @param {Point} p */
63 up(p) {
64 if (p.id !== this.pointer) return false;
65 this.at = [p.x, p.y];
66 this.release();
67 return true;
68 }
69
70 /** Let go of the held joint, wherever the finger is: its last target goes now, not at the next slot. */
71 release() {
72 if (this.pointer === null) return;
73 const joint = /** @type {string} */ (this.joint);
74 this.io.drop(...this.at); // the model's last turn of the joint still goes out, as the held joint's
75 this.pointer = null; this.joint = null;
76 if (this.last) this.io.flush(this.last);
77 this.last = null;
78 this.io.end?.(joint);
79 }
80
81 /**
82 * The model turned a joint. Only the held one is sent; anything else (the arm's own pose, a mirrored leader) is not.
83 * @param {string} joint @param {number} angle
84 * @returns {[number, number] | null} the [id, ticks] sent
85 */
86 angle(joint, angle) {
87 if (this.joint === null || joint !== this.joint) return null;
88 const t = this.io.ticks(joint, angle);
89 if (!t) return null;
90 this.last = { [t[0]]: t[1] };
91 this.io.send(this.last);
92 return t;
93 }
94
95 /**
96 * Blur, pagehide and a hidden page let go.
97 * @param {{ addEventListener: (type: string, f: () => void) => void }} win
98 * @param {{ addEventListener: (type: string, f: () => void) => void, hidden?: boolean }} doc
99 */
100 watch(win, doc) {
101 const all = () => this.release();
102 win.addEventListener("blur", all);
103 win.addEventListener("pagehide", all);
104 doc.addEventListener("visibilitychange", () => { if (doc.hidden) all(); });
105 }
106 }
Software/so101/src/ui/grab.test.ts added +103
@@ -0,0 +1,103 @@
1 import { describe, expect, test } from "bun:test";
2 import { Grab } from "./grab.js";
3
4 const IDS: Record<string, number> = { shoulder_pan: 1, shoulder_lift: 2, elbow_flex: 3 };
5
6 /**
7 * A stand-in for the 3D model: the elbow's link fills x 0..100, the shoulder's x 100..200, and the rest is empty
8 * space. A held finger turns its joint by its sideways travel (0.01 rad a pixel), and every turn of any joint is an
9 * angle-change, as the URDF element fires them, which the page hands to the grab. The wire records what goes out.
10 */
11 function rig(allowed = true) {
12 const wire: Array<{ kind: "send" | "flush" | "begin" | "end"; frame?: Record<number, number>; joint?: string }> = [];
13 const angles: Record<string, number> = { shoulder_pan: 0, shoulder_lift: 0, elbow_flex: 0 };
14 let held: { joint: string; x: number } | null = null;
15 const turn = (joint: string, angle: number) => { angles[joint] = angle; grab.angle(joint, angle); };
16 const grab: Grab = new Grab({
17 allowed: () => allowed,
18 pick: (x) => {
19 const joint = x < 0 || x >= 200 ? null : x < 100 ? "elbow_flex" : "shoulder_lift";
20 if (joint) held = { joint, x };
21 return joint;
22 },
23 drag: (x) => { if (held) { turn(held.joint, angles[held.joint]! + (x - held.x) * 0.01); held.x = x; } },
24 drop: (x) => { if (held) { turn(held.joint, angles[held.joint]! + (x - held.x) * 0.01); held = null; } },
25 ticks: (joint, angle) => (IDS[joint] ? [IDS[joint]!, Math.round(2048 + angle * 100)] : null),
26 send: (frame) => wire.push({ kind: "send", frame }),
27 flush: (frame) => wire.push({ kind: "flush", frame }),
28 begin: (joint) => wire.push({ kind: "begin", joint }),
29 end: (joint) => wire.push({ kind: "end", joint }),
30 });
31 return { grab, wire, turn, frames: () => wire.filter((w) => w.kind === "send" || w.kind === "flush").map((w) => w.frame!) };
32 }
33
34 describe("touching the model", () => {
35 test("a touch-drag on a joint's link streams targets for that joint only", () => {
36 const { grab, wire, turn, frames } = rig();
37 expect(grab.down({ id: 7, x: 150, y: 40 })).toBe(true);
38 expect(wire[0]).toEqual({ kind: "begin", joint: "shoulder_lift" });
39 grab.move({ id: 7, x: 160, y: 40 });
40 turn("shoulder_pan", 0.4); // the arm's own pose arriving meanwhile is drawn, never sent
41 turn("elbow_flex", -0.2);
42 grab.move({ id: 7, x: 180, y: 90 });
43 expect(frames()).toEqual([{ 2: 2058 }, { 2: 2078 }]);
44 expect(frames().every((f) => Object.keys(f).join() === "2")).toBe(true);
45 });
46
47 test("lifting the finger sends its last target at once, then nothing", () => {
48 const { grab, wire, turn, frames } = rig();
49 grab.down({ id: 1, x: 50, y: 0 });
50 grab.move({ id: 1, x: 60, y: 0 });
51 expect(grab.up({ id: 1, x: 70, y: 0 })).toBe(true);
52 // The model's last turn at the lift point, then that same target flushed (not left for the next slot), then the end.
53 expect(wire.slice(-3)).toEqual([{ kind: "send", frame: { 3: 2068 } }, { kind: "flush", frame: { 3: 2068 } }, { kind: "end", joint: "elbow_flex" }]);
54 const sent = wire.length;
55 turn("elbow_flex", 1);
56 grab.move({ id: 1, x: 90, y: 0 });
57 expect(wire.length).toBe(sent);
58 expect(frames().at(-1)).toEqual({ 3: 2068 });
59 });
60
61 for (const how of ["blur", "pagehide", "hidden"] as const) {
62 test(`${how} lets go of a held joint like a lift`, () => {
63 const { grab, wire } = rig();
64 const win = new EventTarget(), doc = Object.assign(new EventTarget(), { hidden: false });
65 grab.watch(win, doc);
66 grab.down({ id: 3, x: 120, y: 0 });
67 grab.move({ id: 3, x: 130, y: 0 });
68 if (how === "hidden") { doc.hidden = true; doc.dispatchEvent(new Event("visibilitychange")); }
69 else win.dispatchEvent(new Event(how));
70 expect(wire.at(-1)).toEqual({ kind: "end", joint: "shoulder_lift" });
71 expect(wire.at(-2)!.kind).toBe("flush");
72 expect(grab.joint).toBeNull();
73 });
74 }
75
76 test("orbit and pinch do not send commands: fingers on empty space are the view's", () => {
77 const { grab, wire } = rig();
78 expect(grab.down({ id: 1, x: 300, y: 10 })).toBe(false); // one finger on empty space: orbit
79 expect(grab.down({ id: 2, x: 400, y: 200 })).toBe(false); // the second: a pinch or a two-finger orbit
80 for (const id of [1, 2]) expect(grab.move({ id, x: 350, y: 120 })).toBe(false);
81 expect(grab.up({ id: 1, x: 0, y: 0 })).toBe(false);
82 expect(wire).toEqual([]);
83 });
84
85 test("a second finger while a joint is held neither orbits nor drives, and its lift ends nothing", () => {
86 const { grab, wire, frames } = rig();
87 grab.down({ id: 1, x: 150, y: 0 });
88 expect(grab.down({ id: 2, x: 50, y: 0 })).toBe(true); // kept from the view, and the elbow is not taken
89 expect(grab.move({ id: 2, x: 90, y: 0 })).toBe(false);
90 expect(grab.up({ id: 2, x: 90, y: 0 })).toBe(false);
91 grab.move({ id: 1, x: 160, y: 0 });
92 expect(frames()).toEqual([{ 2: 2058 }]);
93 expect(wire.filter((w) => w.kind === "end")).toEqual([]);
94 });
95
96 test("not your turn: touching a part orbits instead, and nothing is picked or sent", () => {
97 let picked = 0;
98 const grab = new Grab({ allowed: () => false, pick: () => (picked++, "elbow_flex"), drag() {}, drop() {}, ticks: () => [3, 1], send: () => { throw new Error("sent"); }, flush() {} });
99 expect(grab.down({ id: 1, x: 50, y: 0 })).toBe(false);
100 expect(picked).toBe(0);
101 expect(grab.angle("elbow_flex", 1)).toBeNull();
102 });
103 });
Software/so101/src/ui/machine-panel.js +1 −1
@@ -7,7 +7,7 @@
77 import { copyText, entryLine, jointRows, linkRows, stateRows } from "./machine.js";
88
99 const KEEP = 500;
10 /** The page's document and navigator: this module only runs there (the type check has no DOM, as for src/ui/drive.js). @type {any} */
10 /** The page's document and navigator: this module only runs there (the type check has no DOM, as for src/ui/intent.js). @type {any} */
1111 const doc = /** @type {any} */ (globalThis).document;
1212 const nav = /** @type {any} */ (globalThis).navigator;
1313 const STYLE = `
Software/so101/src/ui/reach.js deleted −79
@@ -1,79 +0,0 @@
1 /**
2 * The touch joystick's arithmetic: move the gripper, not a joint. Plain ESM, served to the page at /ui/ and
3 * imported by the tests. The page owns the arm's geometry (the URDF in three.js) and hands it in as `tip`:
4 * joint angles in, the gripper's position in the base frame (metres, z up) out.
5 */
6
7 /**
8 * The joystick and the height buttons as a displacement of the gripper, in the base frame.
9 * Up on the pad reaches away from the base, sideways swings around it; `lift` is -1, 0 or 1.
10 * `tip` is where the gripper is now; the step is `speed` m/s for `dt` seconds at full deflection.
11 * @param {{ x: number, y: number, lift: number }} stick
12 * @param {number[]} tip
13 * @param {number} speed
14 * @param {number} dt
15 * @returns {number[]}
16 */
17 export function stickToDelta(stick, tip, speed, dt) {
18 const [tx = 0, ty = 0] = tip;
19 const r = Math.hypot(tx, ty);
20 // Straight above the base there is no "away": take the base's own forward, +x.
21 const fx = r > 1e-3 ? tx / r : 1;
22 const fy = r > 1e-3 ? ty / r : 0;
23 const x = clampUnit(stick.x), y = clampUnit(stick.y), lift = clampUnit(stick.lift);
24 const step = speed * dt;
25 // Sideways is the forward direction turned a quarter to the right, seen from above.
26 return [(fx * y + fy * x) * step, (fy * y - fx * x) * step, lift * step];
27 }
28
29 /**
30 * One damped-least-squares step: the joint changes that move the tip by `delta`.
31 * The Jacobian comes from `tip` by finite differences, so any chain the page can evaluate works;
32 * the damping keeps the step small near a stretched-out or folded-up arm instead of letting it whip.
33 * @param {(angles: number[]) => number[]} tip
34 * @param {number[]} angles
35 * @param {number[]} delta
36 * @param {{ h?: number, damping?: number, maxStep?: number }} [options]
37 * @returns {number[]}
38 */
39 export function reachStep(tip, angles, delta, options = {}) {
40 const h = options.h ?? 1e-4, damping = options.damping ?? 0.02, maxStep = options.maxStep ?? 0.08;
41 const here = tip(angles);
42 const n = angles.length;
43 /** @type {number[][]} J[row][joint] */
44 const J = [[], [], []];
45 for (let j = 0; j < n; j++) {
46 const moved = angles.slice();
47 moved[j] = (moved[j] ?? 0) + h;
48 const there = tip(moved);
49 for (let r = 0; r < 3; r++) /** @type {number[]} */ (J[r])[j] = ((there[r] ?? 0) - (here[r] ?? 0)) / h;
50 }
51 // dq = Jᵀ (J Jᵀ + λ² I)⁻¹ delta
52 const A = [0, 1, 2].map((r) => [0, 1, 2].map((c) => dot(/** @type {number[]} */ (J[r]), /** @type {number[]} */ (J[c])) + (r === c ? damping * damping : 0)));
53 const w = solve3(A, delta);
54 const dq = angles.map((_, j) => w.reduce((s, wr, r) => s + wr * (/** @type {number[]} */ (J[r])[j] ?? 0), 0));
55 // Never more than maxStep radians on any joint in one step.
56 const biggest = Math.max(...dq.map(Math.abs));
57 return biggest > maxStep ? dq.map((d) => (d * maxStep) / biggest) : dq;
58 }
59
60 /** @param {number} v */
61 const clampUnit = (v) => (Number.isFinite(v) ? Math.max(-1, Math.min(1, v)) : 0);
62 /** @param {number[]} a @param {number[]} b */
63 const dot = (a, b) => a.reduce((s, v, i) => s + v * (b[i] ?? 0), 0);
64
65 /**
66 * Cramer's rule for a 3×3 system; a singular matrix gives no motion.
67 * @param {number[][]} A
68 * @param {number[]} b
69 */
70 function solve3(A, b) {
71 /** @param {number[][]} m */
72 const det = (m) => {
73 const [[a = 0, b = 0, c = 0] = [], [d = 0, e = 0, f = 0] = [], [g = 0, h = 0, i = 0] = []] = m;
74 return a * (e * i - f * h) - b * (d * i - f * g) + c * (d * h - e * g);
75 };
76 const d = det(A);
77 if (!Number.isFinite(d) || Math.abs(d) < 1e-12) return [0, 0, 0];
78 return [0, 1, 2].map((c) => det(A.map((row, r) => row.map((v, k) => (k === c ? b[r] ?? 0 : v)))) / d);
79 }
Software/so101/src/ui/reach.test.ts deleted −58
@@ -1,58 +0,0 @@
1 import { describe, expect, test } from "bun:test";
2 import { reachStep, stickToDelta } from "./reach.js";
3
4 // A yaw joint and a two-link planar arm on top of it: the SO-101's pan, shoulder lift and elbow, roughly.
5 const L1 = 0.116, L2 = 0.135;
6 const tip = ([pan = 0, lift = 0, elbow = 0]: number[]) => {
7 const reach = L1 * Math.cos(lift) + L2 * Math.cos(lift + elbow);
8 const z = L1 * Math.sin(lift) + L2 * Math.sin(lift + elbow);
9 return [reach * Math.cos(pan), reach * Math.sin(pan), z];
10 };
11 const distance = (a: number[], b: number[]) => Math.hypot(...a.map((v, i) => v - b[i]!));
12
13 describe("the joystick as a displacement", () => {
14 test("up on the pad reaches away from the base, whichever way the arm faces", () => {
15 const [dx, dy, dz] = stickToDelta({ x: 0, y: 1, lift: 0 }, [0, 0.2, 0.1], 0.05, 0.1);
16 expect(dx).toBeCloseTo(0);
17 expect(dy).toBeCloseTo(0.005);
18 expect(dz).toBe(0);
19 });
20
21 test("right on the pad swings clockwise seen from above, and lift is straight up", () => {
22 const [dx, dy, dz] = stickToDelta({ x: 1, y: 0, lift: 1 }, [0.2, 0, 0.1], 0.05, 0.1);
23 expect(dx).toBeCloseTo(0);
24 expect(dy).toBeCloseTo(-0.005);
25 expect(dz).toBeCloseTo(0.005);
26 });
27
28 test("a deflection past the edge or a broken number is clamped, never amplified", () => {
29 const [dx, dy, dz] = stickToDelta({ x: 0, y: 9, lift: Number.NaN }, [0.2, 0, 0], 0.05, 0.1);
30 expect(dx).toBeCloseTo(0.005);
31 expect(dy).toBeCloseTo(0);
32 expect(dz).toBe(0);
33 });
34 });
35
36 describe("a reach step", () => {
37 test("moves the tip the way it was asked, and the error shrinks step by step", () => {
38 let angles = [0.3, 0.6, -1.2];
39 const start = tip(angles);
40 const goal = [start[0]! + 0.03, start[1]! - 0.02, start[2]! + 0.02];
41 let before = distance(start, goal);
42 for (let i = 0; i < 20; i++) {
43 const here = tip(angles);
44 const delta = goal.map((g, k) => g - here[k]!);
45 angles = angles.map((a, k) => a + reachStep(tip, angles, delta)[k]!);
46 const after = distance(tip(angles), goal);
47 expect(after).toBeLessThanOrEqual(before + 1e-9);
48 before = after;
49 }
50 expect(before).toBeLessThan(0.001);
51 });
52
53 test("a stretched-out arm asked to reach further barely moves instead of whipping", () => {
54 const angles = [0, 0, 0]; // straight out: no joint can take the tip further away
55 const dq = reachStep(tip, angles, [0.05, 0, 0]);
56 for (const d of dq) expect(Math.abs(d)).toBeLessThanOrEqual(0.08);
57 });
58 });
Software/so101/src/viewer.html +115 −323
@@ -18,7 +18,6 @@
1818 } }
1919 </script>
2020 <link rel="modulepreload" href="/ui/__VERSION__/camera-plan.js">
21 <link rel="modulepreload" href="/ui/__VERSION__/drive.js">
2221 <link rel="modulepreload" href="/ui/__VERSION__/frames.js">
2322 <style>
2423 /* Phone portrait first: the view on top, a status strip with the E-stop under it, a sheet of tabs below that.
@@ -59,10 +58,8 @@
5958 /* the view: cameras and the 3D model, always on black */
6059 #app { grid-area: view; position: relative; min-height: 0; background: #000; color: #e8eaed; overflow: hidden; }
6160 #stage { position: absolute; inset: 0; display: flex; gap: 2px; }
62 /* the emergency stop's held state, and the banner that says who pressed it */
61 /* the emergency stop's held state */
6362 .estop:disabled { opacity: .5; cursor: not-allowed; } .estop.on { background: #7d1410; box-shadow: 0 0 0 3px var(--bad); }
64 #banner.stop { display: block; border: 2px solid var(--bad); background: rgba(80,10,8,.92); color: #fff; font-weight: 600; }
65 #banner button { margin-left: 10px; min-height: 32px; padding: 2px 10px; }
6663 .pane { flex: 1; min-width: 0; position: relative; background: #000; display: none; }
6764 .pane.on { display: block; }
6865 .pane img { width: 100%; height: 100%; object-fit: contain; display: block; }
@@ -75,10 +72,6 @@
7572 #top .info { display: inline-flex; gap: 6px; align-items: center; background: rgba(0,0,0,.6); padding: 0 10px; min-height: 36px; border-radius: 8px; max-width: 100%; }
7673 #link { color: #9aa4ae; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: min(36ch, 46vw); } #link.on { color: #3fb950; } #link.warn { color: #d29922; } #link.bad { color: #f85149; font-weight: 700; }
7774 .dot { width: 8px; height: 8px; border-radius: 50%; background: #f85149; display: inline-block; flex: none; } .dot.on { background: #3fb950; } .dot.warn { background: #d29922; }
78 #banner { position: absolute; top: 50px; left: 50%; transform: translateX(-50%); z-index: 5; width: max-content; max-width: min(640px, calc(100% - 16px)); padding: 8px 12px; border-radius: 8px; font-size: 13px; background: rgba(0,0,0,.8); border: 1px solid #30363d; display: none; }
79 #banner.bad { display: block; border-color: #f85149; color: #ffb3ad; } #banner.warn { display: block; border-color: #d29922; color: #f2cc60; }
80 #msg { position: absolute; left: calc(8px + var(--safe-l)); right: calc(8px + var(--safe-r)); bottom: 8px; z-index: 6; margin: 0 auto; width: max-content; max-width: calc(100% - 16px); padding: 8px 12px; border-radius: 8px; font-size: 13px; background: rgba(13,17,23,.9); color: #e8eaed; border: 1px solid #30363d; }
81 #msg:empty { display: none; } #msg.bad { border-color: #f85149; color: #ffb3ad; }
8275
8376 /* the status strip: who drives, your place, time left, the turn's action and the E-stop */
8477 #strip { grid-area: strip; display: flex; align-items: center; gap: 10px; padding: 8px calc(12px + var(--safe-r)) 8px calc(12px + var(--safe-l)); background: var(--surface); border-bottom: 1px solid var(--line); min-height: 64px; }
@@ -124,7 +117,15 @@
124117 #tabs .badge { display: inline-block; min-width: 18px; margin-left: 4px; padding: 0 5px; border-radius: 9px; background: var(--blue); color: var(--on-blue); font-size: 11px; line-height: 18px; }
125118 #tabs .badge:empty { display: none; }
126119
127 #notice { flex: none; padding: 10px calc(12px + var(--safe-r)) 10px calc(12px + var(--safe-l)); border-bottom: 1px solid var(--line); border-left: 3px solid var(--blue); background: var(--blue-soft); font-size: 14px; overflow-wrap: anywhere; }
120 /* notes: the stop, the arm's warnings, the announcement and the page's replies. Never over the camera or the model:
121 one pill in the strip says how many, and opens them. */
122 #notes { flex: none; padding: 0 12px; border-radius: 999px; font-size: 13px; font-weight: 650; white-space: nowrap; }
123 #notes.warn { border-color: var(--warn); color: var(--warn); } #notes.bad { border-color: var(--bad); color: var(--bad); background: var(--bad-soft); }
124 #banner, #msg, #notice { margin: 8px 0 0; padding: 10px 12px; border-radius: 10px; border: 1px solid var(--line); border-left: 3px solid var(--blue); font-size: 14px; overflow-wrap: anywhere; }
125 #banner:not(.bad, .warn, .stop), #msg:empty { display: none; }
126 #banner.bad, #banner.stop, #msg.bad { border-color: var(--bad); background: var(--bad-soft); } #banner.warn { border-color: var(--warn); } #banner.stop { font-weight: 600; }
127 .sr { position: absolute; width: 1px; height: 1px; overflow: hidden; clip-path: inset(50%); white-space: nowrap; }
128 #banner button { margin: 8px 8px 0 0; }
128129
129130 /* chat */
130131 #tab-chat { padding: 0; gap: 0; }
@@ -149,14 +150,6 @@
149150 #forgeSignin { color: var(--link); font-size: 13px; align-self: flex-start; }
150151
151152 /* controls */
152 #reach { display: grid; grid-template-columns: auto 1fr; gap: 12px; align-items: center; }
153 #pad { position: relative; width: min(46vw, 180px); aspect-ratio: 1; border-radius: 50%; background: radial-gradient(circle, var(--btn) 0 58%, var(--raised) 59%); border: 1px solid var(--line); touch-action: none; -webkit-user-select: none; user-select: none; }
154 #pad::before, #pad::after { content: ""; position: absolute; background: var(--line); left: 50%; top: 12%; bottom: 12%; width: 1px; } #pad::after { left: 12%; right: 12%; top: 50%; bottom: auto; width: auto; height: 1px; }
155 #knob { position: absolute; left: 50%; top: 50%; width: 38%; aspect-ratio: 1; margin: -19% 0 0 -19%; border-radius: 50%; background: var(--blue); box-shadow: 0 2px 8px rgba(0,0,0,.35); transition: transform .12s ease-out; pointer-events: none; z-index: 1; }
156 #pad.held #knob { transition: none; }
157 #reach .side { display: grid; gap: 8px; }
158 #reach .side button { width: 100%; touch-action: none; -webkit-user-select: none; user-select: none; }
159 #reach .hint { grid-column: 1 / -1; margin: 0; color: var(--dim); font-size: 12px; }
160153 #joints { display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: 6px 14px; }
161154 .j { display: grid; grid-template-columns: 1fr auto; grid-template-areas: "name val" "slide slide"; align-items: center; column-gap: 6px; padding: 4px 6px; border-radius: 10px; }
162155 .j.jam { background: var(--bad-soft); } .j.drag { background: var(--blue-soft); }
@@ -169,40 +162,8 @@
169162 .pose { display: inline-flex; gap: 4px; align-items: center; }
170163 #readout { color: var(--dim); font-size: 13px; display: flex; gap: 14px; flex-wrap: wrap; } #readout b { color: var(--fg); font-weight: 550; }
171164
172 /* touch driving (src/ui/drive.js): on a phone held upright, pick a joint and push one big lever; let go and it stops */
173 #touch { display: none; gap: 10px; }
174 #chips { display: grid; grid-template-columns: repeat(6, minmax(0, 1fr)); gap: 4px; }
175 #chips button { min-height: 50px; padding: 0 2px; border-radius: 10px; display: grid; place-content: center; gap: 1px; line-height: 1.1; }
176 #chips button b { font-size: 15px; } #chips button small { font-size: 10.5px; color: var(--dim); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
177 #chips button[aria-checked=true] { background: var(--blue); border-color: var(--blue); color: var(--on-blue); } #chips button[aria-checked=true] small { color: inherit; opacity: .85; }
178 #one { display: grid; grid-template-columns: minmax(0, 1fr) 84px minmax(0, 1fr); gap: 12px; height: clamp(210px, 36svh, 360px); }
179 #one .nudges { display: grid; grid-template-rows: 1fr 1fr; gap: 10px; }
180 #one .nudges button { display: grid; place-content: center; gap: 2px; font-size: 22px; line-height: 1; touch-action: none; -webkit-user-select: none; user-select: none; }
181 #one .nudges small { font-size: 12px; color: var(--dim); }
182 #lever { position: relative; border-radius: 42px; background: var(--btn); border: 1px solid var(--line); touch-action: none; -webkit-user-select: none; user-select: none; overflow: hidden; cursor: ns-resize; }
183 #lever::before { content: ""; position: absolute; left: 16px; right: 16px; top: 50%; height: 2px; margin-top: -1px; background: var(--line); }
184 #leverAt { position: absolute; left: 0; bottom: 0; width: 5px; height: 100%; background: var(--ok); transform-origin: bottom; transform: scaleY(var(--at, .5)); }
185 #leverKnob { position: absolute; left: 7px; right: 7px; top: 50%; height: 68px; margin-top: -34px; border-radius: 34px; background: var(--blue); color: var(--on-blue); display: grid; place-content: center; font-size: 22px; box-shadow: 0 2px 10px rgba(0,0,0,.3); transition: transform .14s ease-out; pointer-events: none; }
186 #lever.held { border-color: var(--blue); background: var(--blue-soft); } #lever.held #leverKnob { transition: none; }
187 #lever.end, .stick.end { box-shadow: 0 0 0 3px var(--warn); }
188 #one .about { display: grid; align-content: center; gap: 4px; min-width: 0; }
189 #oneName { font-size: 20px; font-weight: 700; } #onePos { font-size: 28px; font-weight: 650; font-variant-numeric: tabular-nums; }
190 #oneHint { color: var(--dim); font-size: 12px; }
191165 #lag { color: #f2cc60; font-variant-numeric: tabular-nums; }
192166 #chat .earlier { display: block; margin: 2px auto 8px; font-size: 13px; }
193 /* on a phone on its side: two thumb sticks over the picture, the gripper between them */
194 #sticks { display: none; position: absolute; inset: 0; z-index: 6; pointer-events: none; --stick: min(40vh, 164px); }
195 #sticks > * { pointer-events: auto; }
196 .stick { position: absolute; bottom: calc(14px + var(--safe-b)); width: var(--stick); height: var(--stick); border-radius: 50%; background: radial-gradient(circle, rgba(255,255,255,.08) 0 54%, rgba(255,255,255,.03) 55%); border: 1px solid rgba(255,255,255,.24); touch-action: none; -webkit-user-select: none; user-select: none; }
197 #stickL { left: calc(18px + var(--safe-l)); } #stickR { right: calc(var(--pill) + var(--safe-r)); }
198 .stick .knob { position: absolute; left: 50%; top: 50%; width: 42%; height: 42%; margin: -21% 0 0 -21%; border-radius: 50%; background: rgba(56,139,253,.8); box-shadow: 0 2px 10px rgba(0,0,0,.45); transition: transform .14s ease-out; pointer-events: none; }
199 .stick.held { border-color: #388bfd; } .stick.held .knob { background: #388bfd; transition: none; }
200 .stick .cap { position: absolute; left: -20px; right: -20px; bottom: 100%; margin-bottom: 4px; text-align: center; font-size: 11px; color: #e8eaed; text-shadow: 0 1px 2px #000; pointer-events: none; white-space: nowrap; }
201 #sticks button { background: rgba(0,0,0,.55); border-color: rgba(255,255,255,.24); color: #e8eaed; touch-action: none; -webkit-user-select: none; user-select: none; }
202 #sticks button.held { background: #388bfd; border-color: #388bfd; }
203 #swap { position: absolute; right: calc(var(--pill) + var(--safe-r)); bottom: calc(var(--stick) + 40px + var(--safe-b)); min-height: 40px; font-size: 13px; }
204 #grips { position: absolute; left: 50%; bottom: calc(14px + var(--safe-b)); transform: translateX(-50%); display: flex; gap: 10px; }
205 #grips button { min-width: 80px; min-height: 72px; font-weight: 650; }
206167
207168 /* admin */
208169 .card { display: grid; gap: 10px; padding: 12px; border: 1px solid var(--line); border-radius: 12px; background: var(--raised); }
@@ -276,18 +237,21 @@
276237 #shell.bare #sheet { transform: translateX(100%); visibility: hidden; transition: transform .2s ease-out, visibility 0s .2s; }
277238 #top { top: auto; bottom: 6px; justify-content: flex-start; right: calc(min(380px, 48vw) + 12px + var(--safe-r)); }
278239 #shell.bare #top { right: calc(var(--pill) + var(--safe-r)); }
279 #banner { top: 64px; }
280 #msg { bottom: 52px; }
281240 .tab { padding-top: 8px; }
282 #pad { width: min(30vh, 150px); }
283241 }
284 @media (max-width: 640px) and (orientation: portrait) { #touch { display: grid; } #reach, #joints { display: none !important; } }
285 /* One STOP in every layout; it holds until Resume. */
286 @media (orientation: landscape) and (max-height: 500px) {
287 #shell.driving.bare #sticks { display: block; }
288 #shell.driving.bare #top { top: 6px; bottom: auto; left: auto; right: calc(6px + var(--safe-r)); max-width: 42%; justify-content: flex-end; }
289 #shell.driving.bare #strip { max-width: calc(58% - 18px - var(--safe-l)); }
290 #shell.driving.bare #msg { bottom: auto; top: 110px; }
242 /* A phone held upright: the camera on top at 16:9 and the 3D model under it, both always there; the model is the
243 main control. The sheet is a tray under them: the grabber, or the open tab tapped again, folds it away. */
244 #fold { display: none; flex: none; width: 100%; min-height: 28px; border: 0; border-radius: 0; background: none; padding: 0; }
245 #fold::before { content: ""; display: block; width: 40px; height: 4px; margin: 0 auto; border-radius: 2px; background: var(--line); }
246 @media (max-width: 640px) and (orientation: portrait) {
247 #shell, #shell.typing { grid-template-rows: minmax(0, 1fr) auto var(--tray, min(42%, 350px)); }
248 #shell.folded { --tray: auto; } #shell.folded .tab { display: none; }
249 #stage { flex-direction: column; }
250 .pane:not(#pane3) { flex: none; aspect-ratio: 16 / 9; } #stage:not(:has(#pane3.on)) .pane { flex: 1; aspect-ratio: auto; }
251 #fold { display: block; } .tab { padding-top: 0; gap: 8px; }
252 #joints { grid-template-columns: 100%; gap: 0; }
253 .j { grid-template-columns: 6.6em minmax(0, 1fr) 3.2em; grid-template-areas: "name slide val"; padding: 0 2px; }
254 .j .jog, .j small { display: none; } .j .v { justify-content: flex-end; } .j input[type=range] { height: 36px; }
291255 }
292256 @media (max-width: 420px) { #top .label { display: none; } #link { max-width: 30vw; } }
293257 @media (prefers-reduced-motion: reduce) { *, *::before, *::after { transition: none !important; animation: none !important; scroll-behavior: auto !important; } }
@@ -306,26 +270,19 @@
306270 <button id="fs" title="F">Fullscreen</button>
307271 </div>
308272 <div id="driverTag" aria-hidden="true"></div>
309 <div id="banner" role="alert"></div>
310 <div id="msg" role="status"></div>
311 <div id="sticks" role="group" aria-label="Thumb sticks">
312 <div class="stick" id="stickL" role="group" aria-label="Left stick: sideways turns the base, up and down lifts the shoulder"><span class="cap">J1 base ↔ · J2 shoulder ↕</span><span class="knob"></span></div>
313 <div class="stick" id="stickR" role="group" aria-label="Right stick"><span class="cap" id="capR"></span><span class="knob"></span></div>
314 <button id="swap" type="button" aria-pressed="false">Wrist</button>
315 <div id="grips" role="group" aria-label="Gripper"><button id="gripOpen" type="button">Open</button><button id="gripClose" type="button">Close</button></div>
316 </div>
317273 </div>
318274
319275 <section id="strip" aria-label="Who drives" aria-live="polite">
320276 <div id="ring" hidden aria-hidden="true"><span id="ringN"></span></div>
321277 <div class="who"><div id="driverLine" class="none">Connecting…</div><div id="clock" aria-live="off"></div><div id="capLine"></div></div>
322278 <div id="turnActions"></div>
279 <button id="notes" type="button" hidden aria-haspopup="dialog" aria-controls="notesDlg"></button><span id="said" class="sr" role="status"></span>
323280 <button id="sheetToggle" hidden aria-controls="sheet" aria-expanded="true">Hide</button>
324281 <button id="stop" class="estop" data-stop title="Stop (space or Esc): halt everything, hold with torque, until someone resumes" aria-label="Stop the arm">Stop</button>
325282 </section>
326283
327284 <section id="sheet" aria-label="Chat, queue and controls">
328 <div id="notice" role="status" hidden></div>
285 <button id="fold" type="button" aria-controls="sheet" aria-expanded="true" aria-label="Fold the tray away"></button>
329286 <div class="tab" id="tab-chat" role="tabpanel" aria-labelledby="t-chat">
330287 <div id="chat" role="log" aria-label="Chat"></div>
331288 <form id="say" hidden><input id="sayText" type="text" maxlength="500" autocomplete="off" enterkeyhint="send" placeholder="Say something" aria-label="Message"><button class="go">Send</button></form>
@@ -347,24 +304,6 @@
347304 <p class="note">Plug an SO-101 leader into this computer or Android phone. The 3D model mirrors it while you wait; switch Teleop on and, on your turn, the arm follows it live. Hiding this tab or unplugging the leader stops the arm where it is.</p>
348305 </div>
349306 <div class="tab" id="tab-controls" role="tabpanel" aria-labelledby="t-controls" hidden>
350 <section id="touch" aria-label="Drive one joint">
351 <div id="chips" role="radiogroup" aria-label="Joint"></div>
352 <div id="one">
353 <div class="nudges"><button type="button" data-nudge="1" id="nudgeUp">▲<small></small></button><button type="button" data-nudge="-1" id="nudgeDown">▼<small></small></button></div>
354 <div id="lever" role="slider" tabindex="0" aria-orientation="vertical" aria-valuemin="-100" aria-valuemax="100" aria-valuenow="0"><span id="leverAt"></span><span id="leverKnob" aria-hidden="true">⇕</span></div>
355 <div class="about"><span id="oneName"></span><span id="onePos">—</span><span id="oneHint">Push the lever: further is faster. Let go and it stops.</span></div>
356 </div>
357 </section>
358 <div id="reach" hidden>
359 <div id="pad" role="application" aria-label="Move the gripper: up reaches out, down pulls in, sideways swings around the base"><span id="knob"></span></div>
360 <div class="side">
361 <button data-lift="1" aria-label="Gripper up">▲ Up</button>
362 <button data-lift="-1" aria-label="Gripper down">▼ Down</button>
363 <button data-grip="1">Open</button>
364 <button data-grip="-1">Close</button>
365 </div>
366 <p class="hint">Hold the pad to move the gripper: up reaches out, sideways swings around the base. The jog speed sets how fast.</p>
367 </div>
368307 <p id="why" role="status"></p>
369308 <div id="joints"></div>
370309 <div class="row" id="poses"></div>
@@ -467,6 +406,13 @@
467406 </article>
468407 </div>
469408
409 <dialog id="notesDlg" aria-labelledby="notesTitle">
410 <h3 id="notesTitle">Notes</h3>
411 <div id="banner"></div>
412 <div id="notice" hidden></div>
413 <div id="msg"></div>
414 <form method="dialog" class="actions"><button class="go">Close</button></form>
415 </dialog>
470416 <dialog id="saveDlg">
471417 <form method="dialog">
472418 <h3>Save the current pose</h3>
@@ -515,7 +461,6 @@
515461 </div>
516462 <script type="module">
517463 import { cameraPlan, QUALITIES, samePlan } from "/ui/__VERSION__/camera-plan.js";
518 import { bindSurface, canDrive, Driver, shape, UP } from "/ui/__VERSION__/drive.js";
519464 import { Frames } from "/ui/__VERSION__/frames.js";
520465
521466 const $ = (id) => document.getElementById(id);
@@ -525,6 +470,20 @@
525470 $("msg").textContent = text; $("msg").className = bad ? "bad" : "";
526471 clearTimeout(msgTimer); if (text) msgTimer = setTimeout(() => ($("msg").textContent = ""), bad ? 9000 : 5000);
527472 };
473 // Notes never cover the camera or the model: the stop, the arm's warnings, the announcement and the page's replies sit
474 // in a sheet, and one pill in the strip says how many (red for a stop or an error) and opens it.
475 const NOTES = ["banner", "notice", "msg"].map($);
476 new MutationObserver((changes) => {
477 const shown = NOTES.filter((n) => (n.id === "banner" ? n.className : n.id === "notice" ? !n.hidden : n.textContent)), pill = $("notes");
478 pill.hidden = !shown.length;
479 pill.textContent = `${shown.length} note${shown.length === 1 ? "" : "s"}`;
480 pill.className = shown.some((n) => /bad|stop/.test(n.className)) ? "bad" : shown.some((n) => n.className === "warn") ? "warn" : "";
481 pill.title = shown.map((n) => n.textContent).join(" · ");
482 const latest = NOTES.find((n) => n.contains(changes.at(-1).target)), said = latest && shown.includes(latest) ? latest.textContent : "";
483 if (said && $("said").textContent !== said) $("said").textContent = said; // read out, once
484 if (!shown.length && $("notesDlg").open) $("notesDlg").close();
485 }).observe($("notesDlg"), { childList: true, characterData: true, subtree: true, attributeFilter: ["class", "hidden"] });
486 $("notes").onclick = () => $("notesDlg").showModal();
528487 const api = async (path, body, method = body === undefined ? "GET" : "POST") => {
529488 const r = await fetch(path, { method, body: body === undefined ? undefined : JSON.stringify(body), cache: "no-store" });
530489 const data = await r.json().catch(() => ({}));
@@ -532,8 +491,8 @@
532491 };
533492 const el = (tag, props = {}, ...children) => { const e = Object.assign(document.createElement(tag), props); e.append(...children); return e; };
534493 const NAMES = { 1: "shoulder_pan", 2: "shoulder_lift", 3: "elbow_flex", 4: "wrist_flex", 5: "wrist_roll", 6: "gripper" };
494 const SHORT = { 1: "J1 base", 2: "J2 shoulder", 3: "J3 elbow", 4: "J4 wrist", 5: "J5 roll", 6: "Grip" };
535495 const IDS = Object.fromEntries(Object.entries(NAMES).map(([id, n]) => [n, Number(id)]));
536 const narrow = matchMedia("(max-width: 760px), (max-height: 500px)");
537496
538497 // ---------------------------------------------------------------- the screen: the keyboard, the sheet, the tabs
539498 // The shell follows the visual viewport, so the on-screen keyboard shrinks the sheet instead of covering the chat.
@@ -546,7 +505,6 @@
546505 $("sheetToggle").textContent = bare ? "Show" : "Hide"; $("sheetToggle").setAttribute("aria-expanded", String(!bare));
547506 };
548507 $("sheetToggle").onclick = () => setBare(!shell.classList.contains("bare"));
549 const sideways = matchMedia("(orientation: landscape) and (max-height: 500px)");
550508 let tab = "chat";
551509 /** The Machine tab's panel, once it is loaded (below); here, because leaving any tab closes it. */
552510 const machine = { panel: null };
@@ -564,7 +522,10 @@
564522 if (name === "admin") loadSettings();
565523 if (name === "machine") machineOpen(); else machineClose();
566524 }
567 for (const b of $("tabs").querySelectorAll("[role=tab]")) b.onclick = () => showTab(b.dataset.tab);
525 /** The tray (a phone held upright): folded, only the tabs show and the model has the room. */
526 const fold = (on) => { shell.classList.toggle("folded", on); $("fold").setAttribute("aria-expanded", String(!on)); $("fold").ariaLabel = on ? "Unfold the tray" : "Fold the tray away"; };
527 $("fold").onclick = () => fold(!shell.classList.contains("folded"));
528 for (const b of $("tabs").querySelectorAll("[role=tab]")) b.onclick = () => { fold(b.dataset.tab === tab && !shell.classList.contains("folded")); showTab(b.dataset.tab); };
568529 $("tabs").addEventListener("keydown", (e) => {
569530 if (e.key !== "ArrowLeft" && e.key !== "ArrowRight") return;
570531 const shown = [...$("tabs").querySelectorAll("[role=tab]:not([hidden])")];
@@ -623,10 +584,10 @@
623584 plan = cameraPlan(navigator.connection, rtts, "stream", { quality, throttle: relayed });
624585 /** The stream's address under the plan: through the relay, ?fps= holds it to fewer frames. */
625586 const streamUrl = (c) => fresh(plan.mode === "stream" && plan.fps && relayed ? `${c.url}?fps=${plan.fps}` : c.url);
626 const saved = (() => { try { const s = JSON.parse(localStorage.getItem("so101.panes") || "null"); return Array.isArray(s) ? s : null; } catch { return null; } })();
587 const saved = (() => { try { const s = JSON.parse(localStorage.getItem("so101.views") || "null"); return Array.isArray(s) ? s : null; } catch { return null; } })();
627588 // Pane keys: "cam:<id>" and "pane3". Until the startup below knows the model and the link, cameras are listed but not shown.
628589 let chosen = saved ? new Set(saved) : null, viewsReady = false, cameraFirst = true;
629 const keep = () => { try { localStorage.setItem("so101.panes", JSON.stringify([...document.querySelectorAll(".pane.on")].map((p) => p.dataset.key))); } catch {} };
590 const keep = () => { try { localStorage.setItem("so101.views", JSON.stringify([...document.querySelectorAll(".pane.on")].map((p) => p.dataset.key))); } catch {} };
630591 const shown = () => document.querySelectorAll(".pane.on").length;
631592 const paneOn = (c) => c.pane.classList.contains("on");
632593 async function snap(c) {
@@ -655,22 +616,20 @@
655616 if (on) load3D();
656617 } else { const c = cams.find((x) => x.pane.dataset.key === key); if (c) showCam(c, on); }
657618 }
658 /** A view button: on or off; a phone shows one view at a time, so picking one there puts the other away. */
619 /** A view button: on or off, keeping at least one. */
659620 function pick(key) {
660621 const pane = key === "pane3" ? $("pane3") : cams.find((x) => x.pane.dataset.key === key)?.pane;
661622 if (!pane) return;
662623 const on = !pane.classList.contains("on");
663624 if (!on && shown() === 1) return; // keep one pane
664 if (on && narrow.matches) for (const other of document.querySelectorAll(".pane.on")) setPane(other.dataset.key, false);
665625 setPane(key, on);
666626 keep(); viewer.recenter?.();
667627 }
668 /** Whether a view is on when it first appears: the viewer's last choice, else the camera when there is a real one, and the model beside it on a wide screen. */
628 /** Whether a view is on when it first appears: the viewer's last choice, else the camera when there is a real one and the model with it (under it on a phone). */
669629 function wanted(key) {
670 if (narrow.matches && shown() >= 1) return false;
671630 if (key === "pane3" && !modelInfo?.ok) return false;
672631 if (chosen) return chosen.has(key);
673 if (key === "pane3") return !narrow.matches || !cameraFirst || !cams.length;
632 if (key === "pane3") return true;
674633 return cameraFirst && key === cams[0]?.pane.dataset.key;
675634 }
676635 /** At least one view on: the first camera, else the model. */
@@ -780,7 +739,7 @@
780739 $("v3").onclick = () => pick("pane3");
781740
782741 // ---------------------------------------------------------------- the stream: the page as a leader
783 // Opened while a slider, a jog button, the joystick or a model joint is held; frames go at most every 33 ms; closed shortly after letting go.
742 // Opened while a slider, a jog button or a model joint is held; frames go at most every 33 ms; closed shortly after letting go.
784743 // Stamped targets (src/ui/frames.js): while the socket opens only the newest waits, and goes only if still fresh.
785744 const stream = {
786745 ws: null, closeTimer: null, halted: false, waiting: null,
@@ -872,8 +831,7 @@
872831 // STOP: drop what this page would still send, then the server halts and holds the arm until Resume.
873832 async function stop() {
874833 stream.halt();
875 driver.releaseAll();
876 reachEnd();
834 grab?.release();
877835 stopLeaderDrive("stopped");
878836 const r = await api("/stop", {});
879837 msg(r.ok ? `STOPPED${r.stopped ? ` ${r.stopped}` : ""}${r.saved ? `; saved recording ${r.saved}` : ""}: the arm holds until you resume` : r.error, true);
@@ -1040,7 +998,7 @@
1040998 async function connectLeader() {
1041999 let lib, tele;
10421000 try { [lib, tele] = await Promise.all([import("/app/leader-arm.ts"), import("/app/teleop.ts")]); }
1043 catch { return explain("This server could not hand the page the leader arm's code. Meanwhile you can drive with the sliders, the − and + buttons, or by dragging a joint of the 3D model."); }
1001 catch { return explain("This server could not hand the page the leader arm's code. Meanwhile you can drive by dragging a part of the 3D model, or with the sliders."); }
10441002 const support = lib.leaderSupport({ serial: "serial" in navigator, secure: isSecureContext, userAgent: navigator.userAgent, touchPoints: navigator.maxTouchPoints });
10451003 if (!support.ok) return explain(support.why);
10461004 const port = await navigator.serial.requestPort({ filters: lib.BOARD_FILTERS }).catch(() => null);
@@ -1203,28 +1161,54 @@
12031161 document.addEventListener("visibilitychange", () => teleop?.update());
12041162 navigator.serial?.addEventListener("disconnect", (e) => { if (e.target === leader?.port) disconnectLeader("the leader was unplugged"); });
12051163
1206 // ---------------------------------------------------------------- the 3D model
1164 // ---------------------------------------------------------------- the 3D model: the main control
12071165 // three.js is most of what the page downloads, so it comes after the first view: the room and the camera work without it.
1166 // A finger (or the mouse) on a part of the arm turns that part's joint, live (src/ui/grab.js); on empty space, two
1167 // fingers or a pinch, it orbits and zooms the view and sends nothing. The model shows your target; a ghost shows the arm.
12081168 const viewer = $("model");
1209 let THREE = null, model3D = null, dragging = null, table = null, marker = null;
1169 let THREE = null, model3D = null, dragging = null, table = null, marker = null, grab = null, ghost = null, may = false;
12101170 function load3D() {
12111171 if (!modelInfo?.ok) return Promise.resolve(false);
12121172 return (model3D ??= (async () => {
1213 const [three, { STLLoader }, { default: URDFManipulator }, reach] = await Promise.all([
1214 import("three"), import("three/examples/jsm/loaders/STLLoader.js"), import("urdf-loader/manipulator"), import("/ui/__VERSION__/reach.js"),
1173 const [three, { STLLoader }, { default: URDFManipulator }, { Grab }] = await Promise.all([
1174 import("three"), import("three/examples/jsm/loaders/STLLoader.js"), import("urdf-loader/manipulator"), import("/ui/__VERSION__/grab.js"),
12151175 ]);
1216 THREE = three; reachMath = reach;
1176 THREE = three;
12171177 if (!customElements.get("urdf-viewer")) customElements.define("urdf-viewer", URDFManipulator);
12181178 viewer.loadMeshFunc = (path, manager, material, done) => {
12191179 new STLLoader(manager).load(path, (geometry) => done(new THREE.Mesh(geometry, material ?? new THREE.MeshPhongMaterial({ color: 0x9aa4ae, shininess: 30 }))), undefined, (err) => done(null, err));
12201180 };
12211181 const canvas = viewer.renderer.domElement;
12221182 canvas.style.touchAction = "none";
1223 const asMouse = (type, t) => canvas.dispatchEvent(new MouseEvent(type, { clientX: t.clientX, clientY: t.clientY, bubbles: true }));
1224 canvas.addEventListener("touchstart", (e) => { if (e.touches.length === 1) asMouse("mousedown", e.touches[0]); }, { passive: true });
1225 canvas.addEventListener("touchmove", (e) => { if (e.touches.length === 1 && dragging) { e.preventDefault(); asMouse("mousemove", e.touches[0]); } }, { passive: false });
1226 canvas.addEventListener("touchend", (e) => { if (dragging) asMouse("mouseup", e.changedTouches[0]); }, { passive: true });
1227 canvas.addEventListener("touchcancel", (e) => { if (dragging) asMouse("mouseup", e.changedTouches[0]); }, { passive: true });
1183 viewer.controls.touches = { ONE: THREE.TOUCH.ROTATE, TWO: THREE.TOUCH.DOLLY_ROTATE };
1184 // The element's own mouse handling would turn a joint whoever touches it: every real mouse event stops here, and
1185 // the grab feeds it made-up ones only while it holds a joint for someone who may drive.
1186 const as = (type, x, y) => canvas.dispatchEvent(new MouseEvent(type, { clientX: x, clientY: y }));
1187 for (const type of ["mousedown", "mousemove", "mouseup"]) viewer.addEventListener(type, (e) => { if (e.isTrusted) e.stopPropagation(); }, true);
1188 let touch = false;
1189 grab = new Grab({
1190 allowed: () => may && Boolean(viewer.robot) && !calib && !leaderStreaming(),
1191 pick: (x, y) => { as("mousedown", x, y); return viewer.dragControls.manipulating?.name ?? null; },
1192 drag: (x, y) => as("mousemove", x, y),
1193 drop: (x, y) => { as("mouseup", x, y); if (touch) as("mousemove", -1e5, -1e5); }, // a finger leaves no hover behind
1194 ticks: (name, angle) => { const id = IDS[name]; return id && limits[id] ? [id, clampTicks(id, toTicks(id, angle))] : null; },
1195 send: (f) => stream.send(f),
1196 flush: (f) => stream.flush(f),
1197 begin: (name) => { dragging = name; stream.open(); navigator.vibrate?.(8); },
1198 end: (name) => { dragging = null; intent.release(IDS[name], performance.now()); stream.end(); if (status) applyModel(status); },
1199 });
1200 grab.watch(window, document);
1201 viewer.addEventListener("pointerdown", (e) => {
1202 if (e.pointerType === "mouse" && e.button !== 0) return; // the right button pans
1203 touch = e.pointerType !== "mouse";
1204 if (grab.down({ id: e.pointerId, x: e.clientX, y: e.clientY })) { e.stopPropagation(); try { canvas.setPointerCapture(e.pointerId); } catch {} }
1205 }, true);
1206 viewer.addEventListener("pointermove", (e) => { if (!grab.move({ id: e.pointerId, x: e.clientX, y: e.clientY }) && e.pointerType === "mouse" && grab.joint === null) as("mousemove", e.clientX, e.clientY); });
1207 for (const type of ["pointerup", "pointercancel", "lostpointercapture"]) viewer.addEventListener(type, (e) => grab.up({ id: e.pointerId, x: e.clientX, y: e.clientY }));
1208 viewer.addEventListener("angle-change", (e) => {
1209 const t = viewer.robot && grab.angle(e.detail, viewer.robot.joints[e.detail].angle);
1210 if (t) intent.set(t[0], t[1], performance.now());
1211 });
12281212 applyLock();
12291213 viewer.urdf = modelInfo.urdf;
12301214 return true;
@@ -1236,6 +1220,12 @@
12361220 table.position.z = modelInfo.floorZ - 0.0005;
12371221 const grid = new THREE.GridHelper(0.9, 18, 0x4a5058, 0x30363d); grid.rotation.x = Math.PI / 2; grid.position.z = modelInfo.floorZ;
12381222 marker = new THREE.Mesh(new THREE.SphereGeometry(0.012, 16, 12), new THREE.MeshBasicMaterial({ color: 0xf85149 })); marker.visible = false;
1223 // The ghost: the real arm, faint, wherever your target has run ahead of it. It shares the meshes and is never picked.
1224 ghost = viewer.robot.clone();
1225 const faint = new THREE.MeshBasicMaterial({ color: 0x8ab4f8, transparent: true, opacity: 0.28, depthWrite: false });
1226 ghost.traverse((o) => { o.raycast = () => {}; if (o.isMesh) o.material = faint; if (o.isURDFJoint) o.ignoreLimits = true; });
1227 ghost.visible = false;
1228 viewer.robot.parent.add(ghost);
12391229 for (const o of [table, grid, marker]) o.raycast = () => {};
12401230 viewer.world.add(table, grid, marker);
12411231 viewer.plane.visible = false; // the element's own shadow plane sits under the lowest part, which would be below the table
@@ -1244,26 +1234,19 @@
12441234 viewer.controls.target.set(0.08, 0.1, 0);
12451235 viewer.controls.update();
12461236 viewer.recenter();
1247 $("reach").hidden = !tipLink();
1248 });
1249 viewer.addEventListener("manipulate-start", (e) => { dragging = e.detail; stream.open(); });
1250 viewer.addEventListener("manipulate-end", () => { const id = IDS[dragging]; if (id) intent.release(id, performance.now()); dragging = null; stream.end(); });
1251 viewer.addEventListener("angle-change", (e) => {
1252 if (dragging !== e.detail || !viewer.robot) return;
1253 const id = IDS[e.detail]; if (!id || !limits[id]) return;
1254 const ticks = clampTicks(id, toTicks(id, viewer.robot.joints[e.detail].angle));
1255 intent.set(id, ticks, performance.now());
1256 stream.send({ [id]: ticks });
12571237 });
12581238 function applyModel(d) {
12591239 if (!viewer.robot || calib) return; // while calibrating, the model shows the pose to copy
1240 let apart = false;
12601241 for (const j of leaderMirrored() ? [] : d.joints) {
12611242 const name = NAMES[j.id];
1262 if (!name || j.position == null || name === dragging || !limits[j.id]) continue;
1263 if (reachState.timer && j.id <= 3) continue; // the joystick is ahead of the arm; snapping back would make it stutter
1264 const want = intent.target(j.id, j.position, performance.now());
1265 viewer.setJointValue(name, toRad(j.id, want ?? j.position));
1243 if (!name || j.position == null || !limits[j.id]) continue;
1244 ghost?.joints[name]?.setJointValue(toRad(j.id, j.position));
1245 const want = name === dragging ? toTicks(j.id, viewer.robot.joints[name].angle) : intent.target(j.id, j.position, performance.now());
1246 if (want != null && Math.abs(want - j.position) > 15) apart = true;
1247 if (name !== dragging) viewer.setJointValue(name, toRad(j.id, want ?? j.position));
12661248 }
1249 if (ghost && (apart || ghost.visible)) { ghost.visible = apart; viewer.redraw(); }
12671250 if (marker) {
12681251 const v = d.blocked ?? d.violation;
12691252 marker.visible = Boolean(v);
@@ -1271,193 +1254,6 @@
12711254 }
12721255 }
12731256
1274 // ---------------------------------------------------------------- the joystick: move the gripper, not a joint
1275 // The pad's deflection becomes a displacement of the gripper (src/ui/reach.js), solved onto pan, lift and elbow.
1276 let reachMath = null;
1277 const REACH = [1, 2, 3];
1278 const SPEED = { 100: 0.02, 300: 0.05, 800: 0.1 }; // m/s at full deflection, by jog speed
1279 const reachState = { stick: { x: 0, y: 0, lift: 0 }, angles: null, timer: null, holders: 0 };
1280 const tipLink = () => viewer.robot?.links?.gripper_frame_link ?? viewer.robot?.links?.gripper_link ?? null;
1281 /** Where the gripper would be with pan, lift and elbow at `angles`, in the arm's base frame: the model is posed, read and put back. */
1282 function tipAt(angles) {
1283 const robot = viewer.robot, link = tipLink();
1284 const saved = REACH.map((id) => robot.joints[NAMES[id]].jointValue[0]);
1285 REACH.forEach((id, i) => robot.joints[NAMES[id]].setJointValue(angles[i]));
1286 robot.updateMatrixWorld(true);
1287 const p = robot.worldToLocal(link.getWorldPosition(new THREE.Vector3()));
1288 REACH.forEach((id, i) => robot.joints[NAMES[id]].setJointValue(saved[i]));
1289 robot.updateMatrixWorld(true);
1290 return [p.x, p.y, p.z];
1291 }
1292 function reachBegin() {
1293 if (reachState.timer) { reachState.holders++; return; }
1294 if (!viewer.robot || !reachMath || !tipLink()) return msg("the 3D model is still loading", true);
1295 const at = REACH.map((id) => status?.joints.find((j) => j.id === id)?.position);
1296 if (at.some((p) => p == null) || REACH.some((id) => !limits[id])) return msg("no position for the arm yet", true);
1297 reachState.angles = REACH.map((id, i) => toRad(id, at[i]));
1298 reachState.holders = 1;
1299 stream.open();
1300 reachState.timer = setInterval(() => {
1301 const { stick } = reachState;
1302 if (!stick.x && !stick.y && !stick.lift) return;
1303 const delta = reachMath.stickToDelta(stick, tipAt(reachState.angles), SPEED[$("rate").value] ?? 0.05, 0.05);
1304 const dq = reachMath.reachStep(tipAt, reachState.angles, delta);
1305 const frame = {};
1306 REACH.forEach((id, i) => { frame[id] = clampTicks(id, toTicks(id, reachState.angles[i] + dq[i])); });
1307 reachState.angles = REACH.map((id) => toRad(id, frame[id])); // a joint at its limit stays there
1308 REACH.forEach((id, i) => viewer.setJointValue(NAMES[id], reachState.angles[i]));
1309 stream.send(frame);
1310 }, 50);
1311 }
1312 function reachEnd(all = true) {
1313 if (!reachState.timer) return;
1314 if (!all && --reachState.holders > 0) return;
1315 clearInterval(reachState.timer); reachState.timer = null; reachState.holders = 0;
1316 reachState.stick = { x: 0, y: 0, lift: 0 };
1317 stream.end();
1318 }
1319 {
1320 const pad = $("pad"), knob = $("knob");
1321 let pointer = null;
1322 const place = (e) => {
1323 const r = pad.getBoundingClientRect(), radius = r.width / 2;
1324 let x = (e.clientX - r.left - radius) / radius, y = -(e.clientY - r.top - radius) / radius;
1325 const length = Math.hypot(x, y);
1326 if (length > 1) { x /= length; y /= length; }
1327 reachState.stick.x = Math.abs(x) < 0.12 ? 0 : x; reachState.stick.y = Math.abs(y) < 0.12 ? 0 : y; // a dead zone around the middle
1328 knob.style.transform = `translate(${x * radius * 0.62}px, ${-y * radius * 0.62}px)`;
1329 };
1330 const release = () => {
1331 if (pointer === null) return;
1332 pointer = null; pad.classList.remove("held"); knob.style.transform = "";
1333 reachState.stick.x = reachState.stick.y = 0; reachEnd(false);
1334 };
1335 pad.addEventListener("pointerdown", (e) => {
1336 if (pointer !== null) return;
1337 e.preventDefault(); pointer = e.pointerId;
1338 try { pad.setPointerCapture(e.pointerId); } catch {}
1339 pad.classList.add("held"); place(e); reachBegin();
1340 });
1341 pad.addEventListener("pointermove", (e) => { if (e.pointerId === pointer) place(e); });
1342 for (const ev of ["pointerup", "pointercancel", "lostpointercapture"]) pad.addEventListener(ev, (e) => { if (e.pointerId === pointer) release(); });
1343 addEventListener("blur", release);
1344 for (const b of document.querySelectorAll("#reach [data-lift]")) {
1345 let held = false;
1346 const up = () => { if (!held) return; held = false; reachState.stick.lift = 0; reachEnd(false); };
1347 b.addEventListener("pointerdown", (e) => { e.preventDefault(); if (held) return; held = true; try { b.setPointerCapture(e.pointerId); } catch {} reachState.stick.lift = Number(b.dataset.lift); reachBegin(); });
1348 for (const ev of ["pointerup", "pointercancel", "lostpointercapture"]) b.addEventListener(ev, up);
1349 b.addEventListener("contextmenu", (e) => e.preventDefault());
1350 }
1351 for (const b of document.querySelectorAll("#reach [data-grip]")) jog(b, 6, () => Number(b.dataset.grip));
1352 }
1353
1354 // ---------------------------------------------------------------- touch driving (src/ui/drive.js)
1355 // Upright: a chip picks the joint, the lever's deflection is its speed, ▲ and ▼ nudge it and jog when held.
1356 const driver = new Driver({
1357 send: (f) => stream.send(f),
1358 flush: (f) => stream.flush(f),
1359 end: () => stream.end(),
1360 begin: () => {
1361 if (!canDrive({ mayDrive: mayDrive(), estop: status?.estop, held: status?.held, reconnecting: status?.reconnecting, limits })) return false;
1362 stream.open();
1363 return true;
1364 },
1365 limits: () => limits,
1366 speed: () => Number($("rate").value) || 300,
1367 onStop: (id) => {
1368 navigator.vibrate?.(30);
1369 for (const e of [$("lever"), $("stickL"), $("stickR")]) { e.classList.add("end"); setTimeout(() => e.classList.remove("end"), 400); }
1370 msg(`${LABEL[id]?.[1] ?? id} is at the end of its range`);
1371 },
1372 });
1373 driver.watch(window, document);
1374 // Turning the phone swaps the controls under the thumbs: whatever was held stops, and on its side the sticks come up.
1375 sideways.addEventListener("change", () => { driver.releaseAll(); if (sideways.matches && shell.classList.contains("driving")) setBare(true); });
1376 const tap = () => navigator.vibrate?.(8);
1377 const LABEL = { 1: ["J1", "Base"], 2: ["J2", "Shoulder"], 3: ["J3", "Elbow"], 4: ["J4", "Wrist"], 5: ["J5", "Roll"], 6: ["Grip", "Gripper"] };
1378 const ENDS = { 1: ["right", "left"], 2: ["up", "down"], 3: ["up", "down"], 4: ["up", "down"], 5: ["roll +", "roll −"], 6: ["open", "close"] };
1379 let picked = (() => { try { const n = Number(localStorage.getItem("so101.joint")); return LABEL[n] ? n : 2; } catch { return 2; } })();
1380 const lever = $("lever");
1381 function pickJoint(id) {
1382 if (driver.drive.holders.has("lever")) driver.hold("lever", { [id]: 0 }); // the joint that was moving stops; the finger now drives this one
1383 picked = id;
1384 try { localStorage.setItem("so101.joint", String(id)); } catch {}
1385 for (const b of $("chips").children) { const on = Number(b.dataset.id) === id; b.setAttribute("aria-checked", String(on)); b.tabIndex = on ? 0 : -1; }
1386 $("oneName").textContent = LABEL[id][1];
1387 $("nudgeUp").querySelector("small").textContent = ENDS[id][0]; $("nudgeDown").querySelector("small").textContent = ENDS[id][1];
1388 $("nudgeUp").setAttribute("aria-label", `${LABEL[id][1]} a step ${ENDS[id][0]}`); $("nudgeDown").setAttribute("aria-label", `${LABEL[id][1]} a step ${ENDS[id][1]}`);
1389 lever.setAttribute("aria-label", `${LABEL[id][1]}: push ${ENDS[id][0]} or ${ENDS[id][1]}, let go to stop`);
1390 if (status) renderTouch(status);
1391 }
1392 $("chips").replaceChildren(...Object.entries(LABEL).map(([id, [short, long]]) => {
1393 const b = el("button", { type: "button", onclick: () => { tap(); pickJoint(Number(id)); } }, el("b", { textContent: short }), el("small", { textContent: long }));
1394 b.dataset.id = id; b.setAttribute("role", "radio"); b.setAttribute("aria-label", `${short} ${long}`);
1395 return b;
1396 }));
1397 $("chips").addEventListener("keydown", (e) => {
1398 const d = e.key === "ArrowRight" || e.key === "ArrowDown" ? 1 : e.key === "ArrowLeft" || e.key === "ArrowUp" ? -1 : 0;
1399 if (!d) return;
1400 e.preventDefault(); pickJoint(((picked - 1 + d + 6) % 6) + 1); $("chips").children[picked - 1]?.focus();
1401 });
1402 pickJoint(picked);
1403 /** Where the picked joint is: degrees from the middle of its range, the gripper as how far open; the green bar on the lever likewise. */
1404 function renderTouch(d) {
1405 const j = d.joints.find((x) => x.id === picked), l = limits[picked];
1406 if (!j || j.position == null || !l) { $("onePos").textContent = "—"; return; }
1407 const share = (j.position - l.min) / (l.max - l.min);
1408 const up = UP[picked] > 0 ? share : 1 - share; // the bar fills toward the lever's top end
1409 lever.style.setProperty("--at", String(Math.max(0, Math.min(1, up)).toFixed(3)));
1410 $("onePos").textContent = picked === 6 ? `${Math.round(share * 100)}% open` : `${(((j.position - mid(picked)) * 360) / 4096).toFixed(0)}°`;
1411 }
1412 bindSurface(lever, driver, "lever", (_, y) => ({ [picked]: shape(y) * UP[picked] }), {
1413 haptic: tap,
1414 show: (_, y) => {
1415 $("leverKnob").style.transform = `translateY(${(-y * (lever.clientHeight / 2 - 38)).toFixed(1)}px)`;
1416 lever.setAttribute("aria-valuenow", String(Math.round(y * 100)));
1417 },
1418 });
1419 // ▲ ▼: a tap is one step, a hold jogs at half speed; arrow keys on the lever step too.
1420 const step = () => Math.max(5, Math.round((Number($("rate").value) || 300) / 15));
1421 for (const b of [$("nudgeUp"), $("nudgeDown")]) {
1422 let pointer = null, timer = null;
1423 const dir = () => Number(b.dataset.nudge) * UP[picked];
1424 b.addEventListener("pointerdown", (e) => {
1425 if (pointer !== null) return;
1426 e.preventDefault(); pointer = e.pointerId;
1427 try { b.setPointerCapture(e.pointerId); } catch {}
1428 tap(); b.classList.add("held");
1429 driver.nudge(picked, dir() * step());
1430 timer = setTimeout(() => { timer = null; if (pointer !== null) driver.hold("nudge", { [picked]: dir() * 0.5 }); }, 350);
1431 });
1432 const up = (e) => { if (e.pointerId !== pointer) return; pointer = null; clearTimeout(timer); b.classList.remove("held"); driver.release("nudge"); };
1433 for (const ev of ["pointerup", "pointercancel", "lostpointercapture"]) b.addEventListener(ev, up);
1434 b.addEventListener("keydown", (e) => { if (e.key === "Enter" || e.key === " ") { e.preventDefault(); e.stopPropagation(); driver.nudge(picked, dir() * step()); } });
1435 b.addEventListener("contextmenu", (e) => e.preventDefault());
1436 }
1437 lever.addEventListener("keydown", (e) => {
1438 const d = e.key === "ArrowUp" ? 1 : e.key === "ArrowDown" ? -1 : 0;
1439 if (d) { e.preventDefault(); driver.nudge(picked, d * UP[picked] * step()); }
1440 });
1441 // The sticks: a disc, so a corner is no faster than an edge; each stick has its own finger.
1442 const disc = (x, y) => { const r = Math.hypot(x, y); return r > 1 ? [x / r, y / r] : [x, y]; };
1443 const knob = (stick) => (x, y) => {
1444 const [dx, dy] = disc(x, y), r = stick.clientWidth * 0.3;
1445 stick.querySelector(".knob").style.transform = `translate(${(dx * r).toFixed(1)}px, ${(-dy * r).toFixed(1)}px)`;
1446 };
1447 bindSurface($("stickL"), driver, "left", (x, y) => { const [dx, dy] = disc(x, y); return { 1: shape(dx) * UP[1], 2: shape(dy) * UP[2] }; }, { show: knob($("stickL")), haptic: tap });
1448 let wrist = false;
1449 const right = () => (wrist ? { x: 5, y: 6 } : { x: 4, y: 3 });
1450 const rightStick = bindSurface($("stickR"), driver, "right", (x, y) => { const [dx, dy] = disc(x, y), m = right(); return { [m.x]: shape(dx) * UP[m.x], [m.y]: shape(dy) * UP[m.y] }; }, { show: knob($("stickR")), haptic: tap });
1451 const labelRight = () => {
1452 $("capR").textContent = wrist ? "J5 roll ↔ · grip ↕" : "J4 wrist ↔ · J3 elbow ↕";
1453 $("stickR").setAttribute("aria-label", wrist ? "Right stick: sideways rolls the wrist, up opens the gripper" : "Right stick: sideways bends the wrist, up and down lifts the elbow");
1454 $("swap").textContent = wrist ? "Arm" : "Wrist"; $("swap").setAttribute("aria-pressed", String(wrist));
1455 };
1456 $("swap").onclick = () => { rightStick.end(); wrist = !wrist; tap(); labelRight(); };
1457 labelRight();
1458 bindSurface($("gripOpen"), driver, "open", () => ({ 6: UP[6] }), { haptic: tap });
1459 bindSurface($("gripClose"), driver, "close", () => ({ 6: -UP[6] }), { haptic: tap });
1460
14611257 // Keep the screen on while it is your turn: a phone that dims mid-drive is a hand off the controls.
14621258 let wakeLock = null, waking = false, wantAwake = false;
14631259 async function wake(on) {
@@ -1571,22 +1367,21 @@
15711367 $("t-controls").hidden = !mayDrive();
15721368 $("t-admin").hidden = !owner;
15731369 $("t-chat").hidden = $("t-queue").hidden = !relayed;
1574 for (const id of ["joints", "poses", "drive", "reach", "touch", "sticks"]) $(id).inert = !can;
1370 for (const id of ["joints", "poses", "drive"]) $(id).inert = !can;
15751371 shell.classList.toggle("driving", can);
1576 if (!can) driver.releaseAll();
1372 may = can;
1373 if (!can) grab?.release();
15771374 wake(can);
15781375 $("stop").disabled = !(mayDrive() || owner);
15791376 $("stop").title = $("stop").disabled ? "Stop: the driver and the admins can press it" : "Stop (space or Esc): halt everything, hold with torque, until someone resumes";
15801377 // The server takes settings, the safety limits among them, from admins only.
15811378 for (const id of ["zoneRow", "trims", "safety", "jointLimits"]) $(id).inert = !owner;
1582 viewer.toggleAttribute("disable-dragging", !can);
1583 if (!can) reachEnd();
15841379 teleop?.update(); // the turn ended: teleop stops at once (the server closes it too; the page need not wait for that)
15851380 // Your turn just came: open the controls. It just ended: back to the chat.
15861381 if (mayDrive() !== drove) {
15871382 drove = mayDrive();
15881383 if (relayed) showTab(drove ? "controls" : "chat");
1589 if (drove && sideways.matches) setBare(true); // on its side, a turn starts on the thumb sticks
1384 if (drove) fold(false);
15901385 }
15911386 else if ($(`t-${tab}`).hidden) showTab(tab);
15921387 }
@@ -1875,7 +1670,7 @@
18751670 const div = document.createElement("div"); div.className = "j";
18761671 div.innerHTML = `<b></b><div class="v"><span class="pos"></span><small class="ld"></small><small class="t"></small></div>
18771672 <div class="slide"><span class="jog"><button data-d="-1" aria-label="less">−</button></span><span class="rail"><input type="range" min="${lim.min + 30}" max="${lim.max - 30}" step="1"></span><span class="jog"><button data-d="1" aria-label="more">+</button></span></div>`;
1878 div.querySelector("b").textContent = j.joint ?? String(j.id);
1673 Object.assign(div.querySelector("b"), { textContent: SHORT[j.id] ?? j.joint ?? String(j.id), title: j.joint ?? "" });
18791674 const range = div.querySelector("input");
18801675 range.setAttribute("aria-label", j.joint ?? `joint ${j.id}`);
18811676 range.addEventListener("pointerdown", () => { sliding = j.id; stream.open(); });
@@ -1943,10 +1738,7 @@
19431738 {
19441739 const was = $("why").textContent;
19451740 status = d;
1946 driver.seed(d.joints.filter((j) => j.position != null).map((j) => [j.id, j.position]));
19471741 if ((explainLock() ?? "") !== was) applyLock();
1948 // Stopped, reconnecting or no longer this person's turn: a held thumb lets go, and its zero goes out now.
1949 if (!canDrive({ mayDrive: mayDrive(), estop: d.estop, held: d.held, reconnecting: d.reconnecting, limits })) driver.releaseAll();
19501742 // The round trip to the arm, which every command also makes: shown once it is long enough to feel.
19511743 const lag = [...rtts].sort((a, b) => a - b)[rtts.length >> 1] ?? 0;
19521744 $("lag").hidden = rtts.length < 3 || lag <= 250; $("lag").textContent = `· ${Math.round(lag)} ms lag`;
@@ -1960,7 +1752,7 @@
19601752 $("stop").classList.toggle("on", Boolean(d.estop || d.held)); $("stop").textContent = d.estop || d.held ? "Stopped" : "Stop";
19611753 $("record").textContent = d.driving === "record" ? "■ Stop recording" : "● Record";
19621754 for (const id of ["hold", "releaseBtn", "play"]) $(id).disabled = d.driving !== null;
1963 if (tab === "controls") { render(d); renderTouch(d); }
1755 if (tab === "controls") render(d);
19641756 banner(d); applyModel(d);
19651757 $("readout").innerHTML = `<span>lowest part <b>${Number.isFinite(d.lowestMm) ? d.lowestMm : "—"} mm</b> above the table</span><span>zone <b>${d.zone === "enforce" || d.zone === "warn" || d.zone === "off" ? d.zone : "—"}</b></span>`;
19661758 }
@@ -1998,7 +1790,7 @@
19981790 if (early) applyStatus(early);
19991791 poll();
20001792 if (relayed) pollLink();
2001 // The model is fetched once the first view is up, whether or not it is on screen: the joystick needs it.
1793 // The model is fetched once the first view is up, whether or not it is on screen: it is the main control.
20021794 if (modelInfo.ok) (window.requestIdleCallback ?? setTimeout)(() => load3D(), { timeout: 3000 });
20031795 })();
20041796 </script>

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs