Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

workbenchthis machine

Mobile driving: pick a joint and drag one big slider (portrait), two thumb sticks (landscape); zero motion the instant a finger lifts #60

Merged opened this in answer to

Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.

Merged

quill/arm:quill/50-mobile-driving-pick-a-joint-and-drag-one → main 4 commits fork

  • Merged. quill/50-mobile-driving-pick-a-joint-and-drag-one was merged into main as 058825d.

Review

Sign in to review this change.
7 files changed +1020 −22
Software/README.md +39 −5
@@ -408,11 +408,45 @@ that is a sheet with tabs at the bottom edge: **Chat**, **Queue**,
408408 starts) and **Admin** (the owner: the room, the safe zone, the trims,
409409 metrics and 3D calibration).
410410
411 - Controls: a joystick moves the gripper. Up reaches out, sideways swings
412 around the base, and ▲/▼ raise and lower it. The page solves that onto
413 pan, lift and elbow against the URDF (`src/ui/reach.js`), and the server
414 checks every frame against the zone as usual. Below the joystick come the
415 joint sliders with hold-to-jog −/+, the poses, and the rest.
411 - Controls, phone upright: a row of chips (J1 base, J2 shoulder, J3 elbow,
412 J4 wrist, J5 roll, Grip) picks one joint. One tall lever drives it: push
413 up or down, and the further you push the faster it goes. Let go and it
414 stops. ▲/▼ beside the lever nudge one step per tap and jog while held.
415 Under them come the poses, Hold, Record and the rest. This layout is for
416 screens up to 640 px wide. A tablet held upright gets the joystick and
417 sliders of the wider screens. From a keyboard, arrows on the chips pick
418 the joint and ↑/↓ on the lever step it.
419 - Controls, phone on its side: the sheet tucks away when your turn starts,
420 and two thumb sticks sit over the picture. The left stick moves J1
421 sideways and J2 up/down. The right stick moves J4 sideways and J3 up/down,
422 and **Wrist** swaps it to J5 roll and the gripper. **Open** and **Close**
423 between the sticks are press-and-hold. **Show** brings the sheet back.
424 The sticks and the grip buttons take touch and a mouse only. With a
425 keyboard or a screen reader, **Show** brings back the sheet, and its joint
426 sliders and ±-buttons drive every joint, the gripper included.
427 - How touch driving works (`src/ui/drive.js`): a finger's deflection is a
428 speed, with a dead band in the middle, never a position. Each finger is
429 tracked by its own pointer, so two thumbs drive two sets of joints
430 independently. The instant a finger lifts, its joints' targets are sent
431 as they stand, without waiting for the next frame (`src/ui/frames.js`
432 sends whatever is still pending with it). The arm's governor
433 settles on them. The same happens on pointercancel, on a lost capture,
434 when the window loses focus, when the page is hidden, and when driving
435 stops being allowed: your turn ends, the arm is stopped, or it is
436 reconnecting (`canDrive` in `src/ui/drive.js`). Frames go out
437 at 25 Hz. A target never runs more than 0.6 s ahead of where the arm
438 last said it was, so a slow link or a speed cap never queues up motion.
439 The server's limits clamp every frame as before. Android buzzes on a
440 press and at the end of a joint's range. Every press also shows on
441 screen. The screen stays awake during your turn (Screen Wake Lock), and
442 the status line shows the round trip to the arm once it passes 250 ms.
443 On a phone the strip carries one stop button, the E-stop.
444 - Controls, wider screens: a joystick moves the gripper. Up reaches out,
445 sideways swings around the base, and ▲/▼ raise and lower it. The page
446 solves that onto pan, lift and elbow against the URDF
447 (`src/ui/reach.js`), and the server checks every frame against the zone
448 as usual. Below the joystick come the joint sliders with hold-to-jog −/+,
449 the poses, and the rest.
416450 - Chat: the page follows the visual viewport. The keyboard shrinks the
417451 sheet and the view, and never covers the box or the last message.
418452 Long-press a message (or right-click it) to delete it, or mute or ban its
Software/so101/src/server.test.ts +16
@@ -158,6 +158,22 @@ describe("viewer page", () => {
158158 expect(page).toContain("prefers-reduced-motion: reduce");
159159 expect(page).toMatch(/#tabs \{[^}]*var\(--pill\)/);
160160 });
161
162 test("serves the touch-driving modules, and the phone layout keeps the browser's gestures off the controls", async () => {
163 const page = await (await fetch(`${base}/`)).text();
164 for (const name of ["drive", "frames"]) {
165 const path = new RegExp(`/ui/[0-9a-f]{12}/${name}\\.js`).exec(page)?.[0];
166 expect(path).toBeDefined();
167 const r = await fetch(`${base}${path}`);
168 expect(r.status).toBe(200);
169 expect(r.headers.get("cache-control")).toContain("immutable");
170 }
171 expect(page).toMatch(/#lever \{[^}]*touch-action: none/);
172 expect(page).toMatch(/\.stick \{[^}]*touch-action: none/);
173 expect(page).toMatch(/#grips button \{[^}]*min-height: (7[2-9]|[89]\d)px/);
174 expect(page).toMatch(/#chips button small \{[^}]*white-space: nowrap/); // a long label never wraps and shifts the row
175 expect(page).toMatch(/@media \(max-width: 760px\), \(max-height: 500px\) \{ #stop \{ display: none/); // the E-stop is the one stop on a phone
176 });
161177 });
162178
163179 describe("moves over HTTP", () => {
Software/so101/src/ui/drive.js added +356
@@ -0,0 +1,356 @@
1 /**
2 * Touch driving: a finger's deflection is a joint's velocity, never its position. Plain ESM, served to the page at
3 * /ui/ and imported by the tests. The wire stays what /follow always took, a target in ticks per joint: each held
4 * joint's target runs ahead at the deflection's share of the jog speed, and the moment the last finger on it lifts
5 * the target stops where it is and goes out at once. That frame is the zero: the arm's governor settles on it.
6 * The server's limits still clamp every frame; this only keeps the page from asking for more.
7 */
8
9 /** The tick direction that raises the gripper, swings the base right, rolls the wrist or opens the jaw (from the URDF). */
10 export const UP = /** @type {Record<number, number>} */ ({ 1: 1, 2: -1, 3: -1, 4: -1, 5: 1, 6: 1 });
11
12 /**
13 * A raw deflection (-1..1) through a dead band and a gentle curve: nothing near the middle, fine control
14 * a little way out, full speed at the edge.
15 * @param {number} v
16 * @param {number} [dead]
17 */
18 export function shape(v, dead = 0.12) {
19 if (!Number.isFinite(v)) return 0;
20 const a = Math.min(1, Math.abs(v));
21 if (a <= dead) return 0;
22 return Math.sign(v) * ((a - dead) / (1 - dead)) ** 1.5;
23 }
24
25 /** @typedef {Record<number, { min: number, max: number }>} Limits */
26 /** @typedef {Record<number, number>} Frame */
27
28 /**
29 * Whether a finger may drive now: it is this person's turn, the arm is not stopped and not reconnecting, and its
30 * limits are known. The page asks before a press starts and on every status poll, letting go of whatever is held
31 * the moment the answer turns false.
32 * @param {{ mayDrive: boolean, estop?: unknown, reconnecting?: unknown, limits?: Limits | null }} state
33 */
34 export function canDrive({ mayDrive, estop, reconnecting, limits }) {
35 return Boolean(mayDrive) && !estop && !reconnecting && Object.keys(limits ?? {}).length > 0;
36 }
37
38 /**
39 * The velocity integrator. Each holder (a pointer, a key) holds some joints at some deflections; a joint's velocity is
40 * the sum of its holders', capped at 1. `step` advances the held joints and returns their targets; `release` drops a
41 * holder and returns the targets of the joints it was moving, which is what stops them.
42 */
43 export class Drive {
44 /** @param {{ margin?: number, lead?: number }} [options] */
45 constructor(options = {}) {
46 /** Ticks kept clear of each end of a joint's range, as the page's sliders always did. */
47 this.margin = options.margin ?? 30;
48 /** How far a target may run ahead of where the arm last said it was: a slow arm (a speed cap) never builds up a backlog. */
49 this.lead = options.lead ?? 0.6;
50 /** @type {Map<string, Map<number, number>>} holder → joint → deflection */
51 this.holders = new Map();
52 /** @type {Map<number, number>} the targets of joints being driven, in fractional ticks */
53 this.targets = new Map();
54 /** @type {Map<number, number>} where the arm last said each joint was */
55 this.present = new Map();
56 }
57
58 /** Where the arm is. A joint nobody holds starts from here when it is next held. @param {Iterable<readonly [number, number]>} positions */
59 seed(positions) {
60 for (const [id, p] of positions) if (Number.isFinite(p)) this.present.set(id, p);
61 }
62
63 /** Whether any joint is held off the middle. */
64 get moving() {
65 for (const axes of this.holders.values()) for (const v of axes.values()) if (v !== 0) return true;
66 return false;
67 }
68
69 /** The joints held, by anyone. */
70 held() {
71 const out = new Set();
72 for (const axes of this.holders.values()) for (const id of axes.keys()) out.add(id);
73 return out;
74 }
75
76 /**
77 * A holder's joints and deflections, replacing what it held before; a joint it lets go of this way stops like a release.
78 * @param {string} holder
79 * @param {Record<number, number>} axes joint id → deflection, -1..1, already shaped
80 * @returns {Frame | null} targets for joints that stopped, to send at once
81 */
82 hold(holder, axes) {
83 const before = this.holders.get(holder);
84 const next = new Map();
85 for (const [k, v] of Object.entries(axes)) {
86 const id = Number(k);
87 if (!Number.isInteger(id) || !Number.isFinite(v)) continue;
88 if (!this.targets.has(id)) {
89 const p = this.present.get(id);
90 if (p === undefined) continue; // no position yet: nothing to run ahead of
91 this.targets.set(id, p);
92 }
93 next.set(id, Math.max(-1, Math.min(1, v)));
94 }
95 this.holders.set(holder, next);
96 const dropped = before ? [...before.keys()].filter((id) => !next.has(id)) : [];
97 return this.#stop(dropped);
98 }
99
100 /** @param {string} holder @returns {Frame | null} the zero: the targets of the joints this holder moved, now still */
101 release(holder) {
102 const axes = this.holders.get(holder);
103 if (!axes) return null;
104 this.holders.delete(holder);
105 return this.#stop([...axes.keys()]);
106 }
107
108 /** Every holder at once: the window lost focus, the page was hidden. @returns {Frame | null} */
109 releaseAll() {
110 const ids = [...this.held()];
111 this.holders.clear();
112 return this.#stop(ids);
113 }
114
115 /**
116 * One small step of one joint, for a tap on − or +.
117 * @param {number} id @param {number} ticks @param {Limits} limits
118 * @returns {Frame | null}
119 */
120 nudge(id, ticks, limits) {
121 const from = this.targets.get(id) ?? this.present.get(id);
122 if (from === undefined || !limits[id]) return null;
123 const t = this.#clamp(id, from + ticks, limits);
124 if (!this.held().has(id)) this.targets.delete(id); else this.targets.set(id, t);
125 this.present.set(id, t); // the next nudge goes on from here, not from a stale reading
126 return { [id]: Math.round(t) };
127 }
128
129 /**
130 * Advance every held joint by `dt` seconds at up to `speed` ticks/s.
131 * @param {number} dt @param {number} speed @param {Limits} limits
132 * @returns {{ frame: Frame | null, stops: number[] }} the targets to send, and joints that just met an end of their range
133 */
134 step(dt, speed, limits) {
135 /** @type {Map<number, number>} */
136 const v = new Map();
137 for (const axes of this.holders.values()) for (const [id, d] of axes) v.set(id, (v.get(id) ?? 0) + d);
138 /** @type {Frame} */
139 const frame = {};
140 const stops = [];
141 let any = false;
142 for (const [id, sum] of v) {
143 const t0 = this.targets.get(id);
144 if (t0 === undefined || !limits[id]) continue;
145 const vel = Math.max(-1, Math.min(1, sum));
146 if (vel === 0) continue;
147 const want = t0 + vel * speed * dt;
148 let t = this.#clamp(id, want, limits);
149 const ended = t !== want;
150 // Never more than `lead` seconds of travel ahead of the arm: a slow arm (a speed cap) is waited for, not queued
151 // up behind. Only further travel is held back; a target never steps back toward a stale reading.
152 const p = this.present.get(id);
153 if (p !== undefined) {
154 const room = Math.max(speed * this.lead, 20);
155 t = vel > 0 ? Math.min(t, Math.max(t0, p + room)) : Math.max(t, Math.min(t0, p - room));
156 }
157 if (t === t0) continue; // nowhere to go: nothing to send
158 if (ended) stops.push(id);
159 this.targets.set(id, t);
160 frame[id] = Math.round(t);
161 any = true;
162 }
163 return { frame: any ? frame : null, stops };
164 }
165
166 /** @param {number} id @param {number} t @param {Limits} limits */
167 #clamp(id, t, limits) {
168 const l = limits[id];
169 if (!l) return t;
170 return Math.max(l.min + this.margin, Math.min(l.max - this.margin, t));
171 }
172
173 /** @param {number[]} ids */
174 #stop(ids) {
175 const still = this.held();
176 /** @type {Frame} */
177 const frame = {};
178 let any = false;
179 for (const id of ids) {
180 if (still.has(id)) continue; // another finger still drives it
181 const t = this.targets.get(id);
182 this.targets.delete(id);
183 if (t === undefined) continue;
184 frame[id] = Math.round(t);
185 this.present.set(id, t); // until the arm says otherwise, it is where it was told to stop
186 any = true;
187 }
188 return any ? frame : null;
189 }
190 }
191
192 /**
193 * The page's driver: a Drive, a clock at the send rate while anything is held, and the ways a press ends.
194 * `send` takes frames at the clock's rate (the page may coalesce them); `flush` must put a frame on the wire now,
195 * because it is the zero; `end` says nothing is held any more.
196 */
197 export class Driver {
198 /**
199 * @param {{
200 * send: (frame: Frame) => void,
201 * flush: (frame: Frame) => void,
202 * end: () => void,
203 * begin?: () => boolean | void,
204 * limits: () => Limits,
205 * speed: () => number,
206 * hz?: number,
207 * onStop?: (id: number) => void,
208 * timers?: { every: (fn: () => void, ms: number) => unknown, cancel: (handle: unknown) => void, now: () => number },
209 * }} options
210 */
211 constructor(options) {
212 this.o = options;
213 this.drive = new Drive();
214 this.hz = Math.max(20, Math.min(30, options.hz ?? 25));
215 this.timers = options.timers ?? { every: (fn, ms) => setInterval(fn, ms), cancel: (h) => clearInterval(/** @type {any} */ (h)), now: () => performance.now() };
216 /** @type {unknown} */
217 this.clock = null;
218 this.last = 0;
219 }
220
221 /** Whether a finger (or a key) is down on anything. */
222 get active() {
223 return this.clock !== null;
224 }
225
226 /** @param {Iterable<readonly [number, number]>} positions */
227 seed(positions) {
228 this.drive.seed(positions);
229 }
230
231 /** @param {string} holder @param {Record<number, number>} axes @returns {boolean} false when driving is refused */
232 hold(holder, axes) {
233 if (!this.clock) {
234 if (this.o.begin?.() === false) return false;
235 this.last = this.timers.now();
236 this.clock = this.timers.every(() => this.tick(), 1000 / this.hz);
237 }
238 const stopped = this.drive.hold(holder, axes);
239 if (stopped) this.o.flush(stopped);
240 return true;
241 }
242
243 /** A holder lets go: its joints' zero goes out now, not at the next tick. @param {string} holder */
244 release(holder) {
245 const frame = this.drive.release(holder);
246 if (frame) this.o.flush(frame);
247 if (this.clock && this.drive.holders.size === 0) this.#idle();
248 }
249
250 /** Everything lets go at once: the window lost focus, the page was hidden. */
251 releaseAll() {
252 const frame = this.drive.releaseAll();
253 if (frame) this.o.flush(frame);
254 if (this.clock) this.#idle();
255 }
256
257 /** One step of one joint, for a tap. @param {number} id @param {number} ticks signed */
258 nudge(id, ticks) {
259 if (this.o.begin?.() === false) return;
260 const frame = this.drive.nudge(id, ticks, this.o.limits());
261 if (frame) this.o.flush(frame);
262 if (!this.clock) this.o.end();
263 }
264
265 tick() {
266 const now = this.timers.now();
267 const dt = Math.min(0.1, Math.max(0, (now - this.last) / 1000)); // a late tick never turns into a jump
268 this.last = now;
269 const { frame, stops } = this.drive.step(dt, this.o.speed(), this.o.limits());
270 if (frame) this.o.send(frame);
271 for (const id of stops) this.o.onStop?.(id);
272 }
273
274 #idle() {
275 this.timers.cancel(this.clock);
276 this.clock = null;
277 this.o.end();
278 }
279
280 /**
281 * Every way a hold can end without a pointerup: the window loses focus, the page is hidden or put away.
282 * @param {EventTarget} win @param {EventTarget & { hidden?: boolean }} doc
283 */
284 watch(win, doc) {
285 const all = () => this.releaseAll();
286 win.addEventListener("blur", all);
287 win.addEventListener("pagehide", all);
288 doc.addEventListener("visibilitychange", () => { if (doc.hidden) all(); });
289 }
290 }
291
292 /**
293 * @typedef {EventTarget & {
294 * setPointerCapture?: (id: number) => void,
295 * getBoundingClientRect: () => { left: number, top: number, width: number, height: number },
296 * classList: { toggle: (name: string, on?: boolean) => unknown },
297 * }} Surface
298 * @typedef {Event & { pointerId: number, clientX: number, clientY: number }} Pointer
299 */
300
301 /**
302 * One finger on a surface, tracked by its pointerId: `place` maps where it is to joint deflections, every way the
303 * finger can leave ends the hold, and a second finger on the same surface is ignored until the first one lifts.
304 * Two surfaces are two holders, so two thumbs drive two sets of joints independently.
305 * @param {Surface} el
306 * @param {Driver} driver
307 * @param {string} holder
308 * @param {(x: number, y: number) => Record<number, number> | null} place x, y from -1 to 1 across the surface, y up
309 * @param {{ show?: (x: number, y: number) => void, haptic?: () => void }} [view]
310 * @returns {{ end: () => void }}
311 */
312 export function bindSurface(el, driver, holder, place, view = {}) {
313 /** @type {number | null} */
314 let pointer = null;
315 /** @param {Pointer} e */
316 const at = (e) => {
317 const r = el.getBoundingClientRect();
318 const x = r.width ? ((e.clientX - r.left) / r.width) * 2 - 1 : 0;
319 const y = r.height ? 1 - ((e.clientY - r.top) / r.height) * 2 : 0;
320 return [Math.max(-1, Math.min(1, x)), Math.max(-1, Math.min(1, y))];
321 };
322 /** @param {Pointer} e */
323 const apply = (e) => {
324 const [x = 0, y = 0] = at(e);
325 view.show?.(x, y);
326 const axes = place(x, y);
327 if (!axes) return false;
328 return driver.hold(holder, axes);
329 };
330 const end = () => {
331 if (pointer === null) return;
332 pointer = null;
333 el.classList.toggle("held", false);
334 view.show?.(0, 0);
335 driver.release(holder);
336 };
337 el.addEventListener("pointerdown", (ev) => {
338 const e = /** @type {Pointer} */ (ev);
339 if (pointer !== null) return;
340 e.preventDefault();
341 pointer = e.pointerId;
342 try { el.setPointerCapture?.(e.pointerId); } catch { /* a synthetic pointer */ }
343 el.classList.toggle("held", true);
344 view.haptic?.();
345 if (apply(e) === false) end();
346 });
347 el.addEventListener("pointermove", (ev) => {
348 const e = /** @type {Pointer} */ (ev);
349 if (e.pointerId === pointer) apply(e);
350 });
351 for (const type of ["pointerup", "pointercancel", "lostpointercapture"]) {
352 el.addEventListener(type, (ev) => { if (/** @type {Pointer} */ (ev).pointerId === pointer) end(); });
353 }
354 el.addEventListener("contextmenu", (e) => e.preventDefault());
355 return { end };
356 }
Software/so101/src/ui/drive.test.ts added +292
@@ -0,0 +1,292 @@
1 import { describe, expect, test } from "bun:test";
2 import { bindSurface, canDrive, Drive, Driver, shape } from "./drive.js";
3
4 const LIMITS = { 1: { min: 1000, max: 3000 }, 2: { min: 1000, max: 3000 }, 3: { min: 1000, max: 3000 }, 4: { min: 1000, max: 3000 }, 6: { min: 2000, max: 3500 } };
5 const AT = [[1, 2000], [2, 2000], [3, 2000], [4, 2000], [6, 2500]] as const;
6
7 /** A driver on a hand-wound clock that records what goes on the wire, and when. */
8 function rig() {
9 const wire: Array<{ kind: "send" | "flush" | "end"; frame?: Record<number, number>; at: number }> = [];
10 let now = 0;
11 let tick: (() => void) | null = null;
12 const driver = new Driver({
13 send: (frame) => wire.push({ kind: "send", frame, at: now }),
14 flush: (frame) => wire.push({ kind: "flush", frame, at: now }),
15 end: () => wire.push({ kind: "end", at: now }),
16 limits: () => LIMITS,
17 speed: () => 300,
18 timers: { every: (fn) => ((tick = fn), 1), cancel: () => (tick = null), now: () => now },
19 });
20 driver.seed(AT);
21 /** Advance the clock by one 40 ms frame and run the tick, when one is running. */
22 const frame = () => {
23 now += 40;
24 tick?.();
25 };
26 return { driver, wire, frame, running: () => tick !== null };
27 }
28
29 /** A surface 100 px square at the origin, built on EventTarget like the page's elements. */
30 function surface() {
31 const el = Object.assign(new EventTarget(), {
32 captured: [] as number[],
33 held: false,
34 setPointerCapture(id: number) { this.captured.push(id); },
35 getBoundingClientRect: () => ({ left: 0, top: 0, width: 100, height: 100 }),
36 classList: { toggle: (_: string, on?: boolean) => (el.held = Boolean(on)) },
37 });
38 return el;
39 }
40 const pointer = (type: string, pointerId: number, clientX = 50, clientY = 50) => Object.assign(new Event(type, { cancelable: true }), { pointerId, clientX, clientY });
41
42 describe("the stick's shape", () => {
43 test("a dead band in the middle, the full speed at the edge, never more", () => {
44 expect(shape(0.05)).toBe(0);
45 expect(shape(-0.12)).toBe(0);
46 expect(shape(1)).toBe(1);
47 expect(shape(-3)).toBe(-1);
48 expect(shape(Number.NaN)).toBe(0);
49 expect(shape(0.5)).toBeGreaterThan(0);
50 expect(shape(0.5)).toBeLessThan(0.5); // fine control a little way out
51 expect(shape(-0.5)).toBe(-shape(0.5));
52 });
53 });
54
55 describe("the velocity integrator", () => {
56 test("a held joint runs ahead at the deflection's share of the speed, from where the arm is", () => {
57 const d = new Drive();
58 d.seed(AT);
59 d.hold("a", { 1: 0.5 });
60 expect(d.step(0.1, 300, LIMITS).frame).toEqual({ 1: 2015 });
61 expect(d.step(0.1, 300, LIMITS).frame).toEqual({ 1: 2030 });
62 });
63
64 test("the target stops at the range's margin and says so once", () => {
65 const d = new Drive();
66 d.seed([[1, 2960]]);
67 d.hold("a", { 1: 1 });
68 expect(d.step(0.1, 300, LIMITS)).toEqual({ frame: { 1: 2970 }, stops: [1] });
69 expect(d.step(0.1, 300, LIMITS)).toEqual({ frame: null, stops: [] });
70 });
71
72 test("never runs more than the lead ahead of an arm that lags, and never steps back toward a stale reading", () => {
73 const d = new Drive({ lead: 0.5 });
74 d.seed([[1, 2000]]);
75 d.hold("a", { 1: 1 });
76 for (let i = 0; i < 20; i++) d.step(0.1, 300, LIMITS);
77 expect(d.targets.get(1)).toBe(2150); // 0.5 s of 300 ticks/s ahead of where the arm said it was
78 d.seed([[1, 1900]]); // a reading older than the target
79 expect(d.step(0.1, 300, LIMITS).frame).toBeNull();
80 expect(d.targets.get(1)).toBe(2150);
81 });
82
83 test("a joint without a position is not driven", () => {
84 const d = new Drive();
85 d.hold("a", { 5: 1 });
86 expect(d.step(0.1, 300, LIMITS).frame).toBeNull();
87 expect(d.release("a")).toBeNull();
88 });
89
90 test("a tap nudges from the last target, inside the range", () => {
91 const d = new Drive();
92 d.seed([[6, 3450]]);
93 expect(d.nudge(6, 20, LIMITS)).toEqual({ 6: 3470 });
94 expect(d.nudge(6, 20, LIMITS)).toEqual({ 6: 3470 });
95 expect(d.nudge(6, -20, LIMITS)).toEqual({ 6: 3450 });
96 });
97 });
98
99 describe("the driver", () => {
100 test("lifting the finger sends zero within one frame: the target as it stands, flushed at once, then nothing", () => {
101 const { driver, wire, frame, running } = rig();
102 driver.hold("lever", { 2: 1 });
103 frame(); frame();
104 const last = wire.at(-1)!;
105 expect(last.kind).toBe("send");
106 const sentAt = wire.length;
107 driver.release("lever");
108 // Before any further tick of the clock: the zero is already on the wire.
109 expect(wire.slice(sentAt)).toEqual([{ kind: "flush", frame: last.frame, at: last.at }, { kind: "end", at: last.at }]);
110 expect(running()).toBe(false);
111 frame(); frame();
112 expect(wire.length).toBe(sentAt + 2); // and nothing after it
113 });
114
115 test("two thumbs drive independent joints: lifting one stops only its joints", () => {
116 const { driver, wire, frame } = rig();
117 driver.hold("left", { 1: 1, 2: 0 });
118 driver.hold("right", { 3: -1 });
119 frame();
120 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 1: 2012, 3: 1988 }, at: 40 });
121 driver.release("left");
122 expect(wire.at(-1)).toEqual({ kind: "flush", frame: { 1: 2012, 2: 2000 }, at: 40 });
123 frame();
124 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 3: 1976 }, at: 80 }); // the right thumb goes on
125 driver.release("right");
126 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 3: 1976 }, at: 80 }, { kind: "end", at: 80 }]);
127 });
128
129 test("two fingers on one joint: it stops only when both have lifted", () => {
130 const { driver, wire, frame } = rig();
131 driver.hold("lever", { 4: 1 });
132 driver.hold("plus", { 4: 1 });
133 frame();
134 driver.release("plus");
135 expect(wire.at(-1)!.kind).toBe("send");
136 frame();
137 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 4: 2024 }, at: 80 });
138 driver.release("lever");
139 expect(wire.at(-1)!.kind).toBe("end");
140 });
141
142 test("blur, pagehide and a hidden page stop every joint at once", () => {
143 for (const how of ["blur", "pagehide", "hidden"] as const) {
144 const { driver, wire, frame, running } = rig();
145 const win = new EventTarget(), doc = Object.assign(new EventTarget(), { hidden: false });
146 driver.watch(win, doc);
147 driver.hold("left", { 1: 1 });
148 driver.hold("right", { 3: 1 });
149 frame();
150 if (how === "hidden") { doc.hidden = true; doc.dispatchEvent(new Event("visibilitychange")); }
151 else win.dispatchEvent(new Event(how));
152 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 1: 2012, 3: 2012 }, at: 40 }, { kind: "end", at: 40 }]);
153 expect(running()).toBe(false);
154 }
155 });
156
157 test("a hidden page that was not driving sends nothing", () => {
158 const { driver, wire } = rig();
159 const win = new EventTarget(), doc = Object.assign(new EventTarget(), { hidden: true });
160 driver.watch(win, doc);
161 doc.dispatchEvent(new Event("visibilitychange"));
162 win.dispatchEvent(new Event("blur"));
163 expect(wire).toEqual([]);
164 });
165
166 test("refused by the page (not your turn), a press starts nothing", () => {
167 const wire: unknown[] = [];
168 const driver = new Driver({ send: (f) => wire.push(f), flush: (f) => wire.push(f), end: () => wire.push("end"), begin: () => false, limits: () => LIMITS, speed: () => 300 });
169 driver.seed(AT);
170 expect(driver.hold("x", { 1: 1 })).toBe(false);
171 expect(driver.active).toBe(false);
172 driver.nudge(1, 20);
173 expect(wire).toEqual([]);
174 });
175
176 test("a tap nudges and ends at once", () => {
177 const { driver, wire } = rig();
178 driver.nudge(6, -20);
179 expect(wire).toEqual([{ kind: "flush", frame: { 6: 2480 }, at: 0 }, { kind: "end", at: 0 }]);
180 });
181 });
182
183 describe("a surface under a finger", () => {
184 const lever = (y: number) => ({ 2: y });
185
186 test("pointerup sends the zero before the next frame, and the surface lets go", () => {
187 const { driver, wire, frame } = rig();
188 const el = surface();
189 bindSurface(el, driver, "lever", (_, y) => lever(y));
190 el.dispatchEvent(pointer("pointerdown", 7, 50, 0)); // the top edge: full speed up
191 expect(el.captured).toEqual([7]);
192 expect(el.held).toBe(true);
193 frame();
194 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 2: 2012 }, at: 40 });
195 el.dispatchEvent(pointer("pointerup", 7));
196 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 2: 2012 }, at: 40 }, { kind: "end", at: 40 }]);
197 expect(el.held).toBe(false);
198 });
199
200 for (const type of ["pointercancel", "lostpointercapture"]) {
201 test(`${type} ends the hold like a lift`, () => {
202 const { driver, wire, frame, running } = rig();
203 const el = surface();
204 bindSurface(el, driver, "lever", (_, y) => lever(y));
205 el.dispatchEvent(pointer("pointerdown", 3, 50, 100));
206 frame();
207 el.dispatchEvent(pointer(type, 3));
208 expect(wire.at(-1)!.kind).toBe("end");
209 expect(running()).toBe(false);
210 });
211 }
212
213 test("a second finger on the same surface neither steers it nor ends it", () => {
214 const { driver, wire, frame } = rig();
215 const el = surface();
216 bindSurface(el, driver, "lever", (_, y) => lever(y));
217 el.dispatchEvent(pointer("pointerdown", 1, 50, 0));
218 el.dispatchEvent(pointer("pointerdown", 2, 50, 100));
219 el.dispatchEvent(pointer("pointermove", 2, 50, 100));
220 el.dispatchEvent(pointer("pointerup", 2));
221 frame();
222 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 2: 2012 }, at: 40 });
223 });
224
225 test("two sticks under two thumbs, each with its own pointer, drive their own joints", () => {
226 const { driver, wire, frame } = rig();
227 const left = surface(), right = surface();
228 bindSurface(left, driver, "left", (x, y) => ({ 1: x, 2: y }));
229 bindSurface(right, driver, "right", (x, y) => ({ 4: x, 3: y }));
230 left.dispatchEvent(pointer("pointerdown", 11, 100, 50)); // full right
231 right.dispatchEvent(pointer("pointerdown", 12, 50, 100)); // full down
232 frame();
233 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 1: 2012, 3: 1988 }, at: 40 });
234 right.dispatchEvent(pointer("pointerup", 11)); // the other thumb's id on this stick: ignored
235 left.dispatchEvent(pointer("pointerup", 11));
236 expect(wire.at(-1)).toEqual({ kind: "flush", frame: { 1: 2012, 2: 2000 }, at: 40 });
237 frame();
238 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 3: 1976 }, at: 80 });
239 });
240
241 test("a press the page refuses lets go of the surface at once", () => {
242 const driver = new Driver({ send: () => {}, flush: () => {}, end: () => {}, begin: () => false, limits: () => LIMITS, speed: () => 300 });
243 const el = surface();
244 bindSurface(el, driver, "lever", (_, y) => lever(y));
245 el.dispatchEvent(pointer("pointerdown", 1, 50, 0));
246 expect(el.held).toBe(false);
247 });
248
249 test("the arm drops while a thumb is held: the poll's releaseAll flushes the zero, and the still-pressed stick sends nothing more", () => {
250 const wire: Array<{ kind: string; frame?: Record<number, number> }> = [];
251 let tick: (() => void) | null = null, now = 0;
252 const state = { mayDrive: true, estop: false, reconnecting: false, limits: LIMITS };
253 const driver = new Driver({
254 send: (frame) => wire.push({ kind: "send", frame }),
255 flush: (frame) => wire.push({ kind: "flush", frame }),
256 end: () => wire.push({ kind: "end" }),
257 begin: () => canDrive(state),
258 limits: () => LIMITS,
259 speed: () => 300,
260 timers: { every: (fn) => ((tick = fn), 1), cancel: () => (tick = null), now: () => now },
261 });
262 driver.seed(AT);
263 const stick = surface();
264 bindSurface(stick, driver, "left", (x, y) => ({ 1: x, 2: y }));
265 stick.dispatchEvent(pointer("pointerdown", 7, 100, 50));
266 now += 40; tick!();
267 expect(wire.at(-1)).toEqual({ kind: "send", frame: { 1: 2012 } });
268 // The status poll: reconnecting, so driving is refused and everything held lets go.
269 state.reconnecting = true;
270 if (!canDrive(state)) driver.releaseAll();
271 expect(wire.slice(-2)).toEqual([{ kind: "flush", frame: { 1: 2012, 2: 2000 } }, { kind: "end" }]);
272 expect(tick).toBeNull();
273 const after = wire.length;
274 stick.dispatchEvent(pointer("pointermove", 7, 100, 0)); // the thumb is still down and moving
275 expect(wire.length).toBe(after);
276 expect(tick).toBeNull();
277 });
278 });
279
280 describe("canDrive", () => {
281 const ok = { mayDrive: true, estop: false, reconnecting: false, limits: LIMITS };
282 test("drives on your turn, with the arm up and its limits known", () => {
283 expect(canDrive(ok)).toBe(true);
284 });
285 test("refuses when it is not your turn, the arm is stopped, the arm is reconnecting, or no limits have come", () => {
286 expect(canDrive({ ...ok, mayDrive: false })).toBe(false);
287 expect(canDrive({ ...ok, estop: true })).toBe(false);
288 expect(canDrive({ ...ok, reconnecting: true })).toBe(false);
289 expect(canDrive({ ...ok, limits: {} })).toBe(false);
290 expect(canDrive({ ...ok, limits: null })).toBe(false);
291 });
292 });
Software/so101/src/ui/frames.js added +47
@@ -0,0 +1,47 @@
1 // @ts-check
2 /**
3 * Frames on their way to /follow: `send` coalesces them into one message per slot, joint by joint, the newest target
4 * winning; `flush` puts what is pending and the given frame on the wire now, because it is where a finger let go.
5 * `write` gets the JSON; whether a socket is open is the page's business.
6 * @typedef {Record<number, number>} Frame
7 */
8 export class Frames {
9 /**
10 * @param {(data: string) => void} write
11 * @param {{ wait?: number, later?: (fn: () => void, ms: number) => unknown, cancel?: (handle: unknown) => void }} [options]
12 */
13 constructor(write, options = {}) {
14 this.write = write;
15 this.wait = options.wait ?? 33;
16 this.later = options.later ?? ((fn, ms) => setTimeout(fn, ms));
17 this.cancel = options.cancel ?? ((h) => clearTimeout(/** @type {any} */ (h)));
18 /** @type {Frame | null} */
19 this.pending = null;
20 /** @type {unknown} */
21 this.timer = null;
22 }
23
24 /** @param {Frame} frame */
25 send(frame) {
26 this.pending = { ...(this.pending ?? {}), ...frame };
27 if (this.timer !== null) return;
28 this.timer = this.later(() => {
29 this.timer = null;
30 const data = this.pending; this.pending = null;
31 if (data) this.write(JSON.stringify(data));
32 }, this.wait);
33 }
34
35 /** Now, not at the next slot, with anything pending under it. @param {Frame} frame */
36 flush(frame) {
37 const data = { ...(this.pending ?? {}), ...frame };
38 this.drop();
39 this.write(JSON.stringify(data));
40 }
41
42 /** Forget what is pending: the stream ended. */
43 drop() {
44 if (this.timer !== null) this.cancel(this.timer);
45 this.timer = null; this.pending = null;
46 }
47 }
Software/so101/src/ui/frames.test.ts added +48
@@ -0,0 +1,48 @@
1 import { describe, expect, test } from "bun:test";
2 import { Frames } from "./frames.js";
3
4 /** Frames on a hand-run timer, writing to an array. */
5 function rig() {
6 const wire: Array<Record<string, number>> = [];
7 let due: (() => void) | null = null;
8 const frames = new Frames((data) => wire.push(JSON.parse(data)), { later: (fn) => ((due = fn), 1), cancel: () => (due = null) });
9 const slot = () => { const fn = due; due = null; fn?.(); };
10 return { frames, wire, slot, waiting: () => due !== null };
11 }
12
13 describe("Frames", () => {
14 test("sends in one slot coalesce joint by joint, the newest target winning", () => {
15 const { frames, wire, slot } = rig();
16 frames.send({ 1: 2000, 2: 2000 });
17 frames.send({ 2: 2010, 3: 1990 });
18 expect(wire).toEqual([]);
19 slot();
20 expect(wire).toEqual([{ 1: 2000, 2: 2010, 3: 1990 }]);
21 slot();
22 expect(wire.length).toBe(1);
23 });
24
25 test("a flush goes out now, carrying what was pending under it, and the slot it cancels sends nothing", () => {
26 const { frames, wire, slot, waiting } = rig();
27 frames.send({ 1: 2012, 3: 1988 });
28 frames.flush({ 1: 2020, 2: 2000 });
29 expect(wire).toEqual([{ 1: 2020, 2: 2000, 3: 1988 }]);
30 expect(waiting()).toBe(false);
31 slot();
32 expect(wire.length).toBe(1);
33 });
34
35 test("a flush with nothing pending is the frame alone", () => {
36 const { frames, wire } = rig();
37 frames.flush({ 6: 2500 });
38 expect(wire).toEqual([{ 6: 2500 }]);
39 });
40
41 test("drop forgets the pending frame", () => {
42 const { frames, wire, slot } = rig();
43 frames.send({ 1: 2000 });
44 frames.drop();
45 slot();
46 expect(wire).toEqual([]);
47 });
48 });
Software/so101/src/viewer.html +222 −17
@@ -18,6 +18,8 @@
1818 } }
1919 </script>
2020 <link rel="modulepreload" href="/ui/__VERSION__/camera-plan.js">
21 <link rel="modulepreload" href="/ui/__VERSION__/drive.js">
22 <link rel="modulepreload" href="/ui/__VERSION__/frames.js">
2123 <link rel="modulepreload" href="/leader-map.js">
2224 <style>
2325 /* Phone portrait first: the view on top, a status strip with the E-stop under it, a sheet of tabs below that.
@@ -148,6 +150,40 @@
148150 .pose { display: inline-flex; gap: 4px; align-items: center; }
149151 #readout { color: var(--dim); font-size: 13px; display: flex; gap: 14px; flex-wrap: wrap; } #readout b { color: var(--fg); font-weight: 550; }
150152
153 /* touch driving (src/ui/drive.js): on a phone held upright, pick a joint and push one big lever; let go and it stops */
154 #touch { display: none; gap: 10px; }
155 #chips { display: grid; grid-template-columns: repeat(6, minmax(0, 1fr)); gap: 4px; }
156 #chips button { min-height: 50px; padding: 0 2px; border-radius: 10px; display: grid; place-content: center; gap: 1px; line-height: 1.1; }
157 #chips button b { font-size: 15px; } #chips button small { font-size: 10.5px; color: var(--dim); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
158 #chips button[aria-checked=true] { background: var(--blue); border-color: var(--blue); color: var(--on-blue); } #chips button[aria-checked=true] small { color: inherit; opacity: .85; }
159 #one { display: grid; grid-template-columns: minmax(0, 1fr) 84px minmax(0, 1fr); gap: 12px; height: clamp(210px, 36svh, 360px); }
160 #one .nudges { display: grid; grid-template-rows: 1fr 1fr; gap: 10px; }
161 #one .nudges button { display: grid; place-content: center; gap: 2px; font-size: 22px; line-height: 1; touch-action: none; -webkit-user-select: none; user-select: none; }
162 #one .nudges small { font-size: 12px; color: var(--dim); }
163 #lever { position: relative; border-radius: 42px; background: var(--btn); border: 1px solid var(--line); touch-action: none; -webkit-user-select: none; user-select: none; overflow: hidden; cursor: ns-resize; }
164 #lever::before { content: ""; position: absolute; left: 16px; right: 16px; top: 50%; height: 2px; margin-top: -1px; background: var(--line); }
165 #leverAt { position: absolute; left: 0; bottom: 0; width: 5px; height: 100%; background: var(--ok); transform-origin: bottom; transform: scaleY(var(--at, .5)); }
166 #leverKnob { position: absolute; left: 7px; right: 7px; top: 50%; height: 68px; margin-top: -34px; border-radius: 34px; background: var(--blue); color: var(--on-blue); display: grid; place-content: center; font-size: 22px; box-shadow: 0 2px 10px rgba(0,0,0,.3); transition: transform .14s ease-out; pointer-events: none; }
167 #lever.held { border-color: var(--blue); background: var(--blue-soft); } #lever.held #leverKnob { transition: none; }
168 #lever.end, .stick.end { box-shadow: 0 0 0 3px var(--warn); }
169 #one .about { display: grid; align-content: center; gap: 4px; min-width: 0; }
170 #oneName { font-size: 20px; font-weight: 700; } #onePos { font-size: 28px; font-weight: 650; font-variant-numeric: tabular-nums; }
171 #oneHint { color: var(--dim); font-size: 12px; }
172 #lag { color: #f2cc60; font-variant-numeric: tabular-nums; }
173 /* on a phone on its side: two thumb sticks over the picture, the gripper between them */
174 #sticks { display: none; position: absolute; inset: 0; z-index: 6; pointer-events: none; --stick: min(40vh, 164px); }
175 #sticks > * { pointer-events: auto; }
176 .stick { position: absolute; bottom: calc(14px + var(--safe-b)); width: var(--stick); height: var(--stick); border-radius: 50%; background: radial-gradient(circle, rgba(255,255,255,.08) 0 54%, rgba(255,255,255,.03) 55%); border: 1px solid rgba(255,255,255,.24); touch-action: none; -webkit-user-select: none; user-select: none; }
177 #stickL { left: calc(18px + var(--safe-l)); } #stickR { right: calc(var(--pill) + var(--safe-r)); }
178 .stick .knob { position: absolute; left: 50%; top: 50%; width: 42%; height: 42%; margin: -21% 0 0 -21%; border-radius: 50%; background: rgba(56,139,253,.8); box-shadow: 0 2px 10px rgba(0,0,0,.45); transition: transform .14s ease-out; pointer-events: none; }
179 .stick.held { border-color: #388bfd; } .stick.held .knob { background: #388bfd; transition: none; }
180 .stick .cap { position: absolute; left: -20px; right: -20px; bottom: 100%; margin-bottom: 4px; text-align: center; font-size: 11px; color: #e8eaed; text-shadow: 0 1px 2px #000; pointer-events: none; white-space: nowrap; }
181 #sticks button { background: rgba(0,0,0,.55); border-color: rgba(255,255,255,.24); color: #e8eaed; touch-action: none; -webkit-user-select: none; user-select: none; }
182 #sticks button.held { background: #388bfd; border-color: #388bfd; }
183 #swap { position: absolute; right: calc(var(--pill) + var(--safe-r)); bottom: calc(var(--stick) + 40px + var(--safe-b)); min-height: 40px; font-size: 13px; }
184 #grips { position: absolute; left: 50%; bottom: calc(14px + var(--safe-b)); transform: translateX(-50%); display: flex; gap: 10px; }
185 #grips button { min-width: 80px; min-height: 72px; font-weight: 650; }
186
151187 /* admin */
152188 .card { display: grid; gap: 10px; padding: 12px; border: 1px solid var(--line); border-radius: 12px; background: var(--raised); }
153189 .card h3 { margin: 0; font-size: 15px; }
@@ -225,6 +261,15 @@
225261 .tab { padding-top: 8px; }
226262 #pad { width: min(30vh, 150px); }
227263 }
264 @media (max-width: 640px) and (orientation: portrait) { #touch { display: grid; } #reach, #joints { display: none !important; } }
265 /* One stop on a phone: the E-stop. Letting go stops the joints; Hold in Controls ends a move or a playback. */
266 @media (max-width: 760px), (max-height: 500px) { #stop { display: none !important; } }
267 @media (orientation: landscape) and (max-height: 500px) {
268 #shell.driving.bare #sticks { display: block; }
269 #shell.driving.bare #top { top: 6px; bottom: auto; left: auto; right: calc(6px + var(--safe-r)); max-width: 42%; justify-content: flex-end; }
270 #shell.driving.bare #strip { max-width: calc(58% - 18px - var(--safe-l)); }
271 #shell.driving.bare #msg { bottom: auto; top: 110px; }
272 }
228273 @media (max-width: 420px) { #top .label { display: none; } #link { max-width: 30vw; } }
229274 @media (prefers-reduced-motion: reduce) { *, *::before, *::after { transition: none !important; animation: none !important; scroll-behavior: auto !important; } }
230275 </style>
@@ -237,11 +282,17 @@
237282 </div>
238283 <div id="top">
239284 <span class="seg" role="group" aria-label="Views"><button id="v3" aria-pressed="false">3D</button></span>
240 <span class="info"><span id="link" hidden></span><span id="live" class="label"></span><span class="dot" id="dot"></span><span id="state">…</span></span>
285 <span class="info"><span id="link" hidden></span><span id="live" class="label"></span><span class="dot" id="dot"></span><span id="state">…</span><span id="lag" hidden></span></span>
241286 <button id="fs" title="F">Fullscreen</button>
242287 </div>
243288 <div id="banner" role="alert"></div>
244289 <div id="msg" role="status"></div>
290 <div id="sticks" role="group" aria-label="Thumb sticks">
291 <div class="stick" id="stickL" role="group" aria-label="Left stick: sideways turns the base, up and down lifts the shoulder"><span class="cap">J1 base ↔ · J2 shoulder ↕</span><span class="knob"></span></div>
292 <div class="stick" id="stickR" role="group" aria-label="Right stick"><span class="cap" id="capR"></span><span class="knob"></span></div>
293 <button id="swap" type="button" aria-pressed="false">Wrist</button>
294 <div id="grips" role="group" aria-label="Gripper"><button id="gripOpen" type="button">Open</button><button id="gripClose" type="button">Close</button></div>
295 </div>
245296 </div>
246297
247298 <section id="strip" aria-label="Who drives" aria-live="polite">
@@ -273,6 +324,14 @@
273324 <p class="note">Plug an SO-101 leader into this computer or Android phone. The 3D model mirrors it while you wait; on your turn, Drive with leader in Controls moves the arm with it.</p>
274325 </div>
275326 <div class="tab" id="tab-controls" role="tabpanel" aria-labelledby="t-controls" hidden>
327 <section id="touch" aria-label="Drive one joint">
328 <div id="chips" role="radiogroup" aria-label="Joint"></div>
329 <div id="one">
330 <div class="nudges"><button type="button" data-nudge="1" id="nudgeUp">▲<small></small></button><button type="button" data-nudge="-1" id="nudgeDown">▼<small></small></button></div>
331 <div id="lever" role="slider" tabindex="0" aria-orientation="vertical" aria-valuemin="-100" aria-valuemax="100" aria-valuenow="0"><span id="leverAt"></span><span id="leverKnob" aria-hidden="true">⇕</span></div>
332 <div class="about"><span id="oneName"></span><span id="onePos">—</span><span id="oneHint">Push the lever: further is faster. Let go and it stops.</span></div>
333 </div>
334 </section>
276335 <div id="reach" hidden>
277336 <div id="pad" role="application" aria-label="Move the gripper: up reaches out, down pulls in, sideways swings around the base"><span id="knob"></span></div>
278337 <div class="side">
@@ -458,6 +517,8 @@
458517 </div>
459518 <script type="module">
460519 import { cameraPlan } from "/ui/__VERSION__/camera-plan.js";
520 import { bindSurface, canDrive, Driver, shape, UP } from "/ui/__VERSION__/drive.js";
521 import { Frames } from "/ui/__VERSION__/frames.js";
461522 import { mapLeaderFrame, parseCalibration } from "/leader-map.js";
462523
463524 const $ = (id) => document.getElementById(id);
@@ -483,10 +544,12 @@
483544 const vv = window.visualViewport;
484545 const fit = () => { if (!vv) return; shell.style.setProperty("--vv-h", `${vv.height}px`); shell.style.setProperty("--vv-top", `${vv.offsetTop}px`); };
485546 vv?.addEventListener("resize", fit); vv?.addEventListener("scroll", fit); fit();
486 $("sheetToggle").onclick = () => {
487 const bare = shell.classList.toggle("bare");
547 const setBare = (bare) => {
548 shell.classList.toggle("bare", bare);
488549 $("sheetToggle").textContent = bare ? "Show" : "Hide"; $("sheetToggle").setAttribute("aria-expanded", String(!bare));
489550 };
551 $("sheetToggle").onclick = () => setBare(!shell.classList.contains("bare"));
552 const sideways = matchMedia("(orientation: landscape) and (max-height: 500px)");
490553 let tab = "chat";
491554 function showTab(name, focus = false) {
492555 if ($(`t-${name}`).hidden) name = ["controls", "chat", "admin"].find((t) => !$(`t-${t}`).hidden) ?? "chat";
@@ -694,7 +757,13 @@
694757 // ---------------------------------------------------------------- the stream: the page as a leader
695758 // Opened while a slider, a jog button, the joystick or a model joint is held; frames go at most every 33 ms; closed shortly after letting go.
696759 const stream = {
697 ws: null, pending: null, timer: null, closeTimer: null, halted: false,
760 ws: null, closeTimer: null, halted: false,
761 // one message per slot to the socket that is open, or the one about to be
762 frames: new Frames((data) => {
763 const ws = stream.ws;
764 if (ws?.readyState === 1) ws.send(data);
765 else if (ws?.readyState === 0) ws.addEventListener("open", () => ws.send(data), { once: true });
766 }),
698767 open() {
699768 if (!mayDrive()) return; // the server would refuse it: only the person whose turn it is streams
700769 clearTimeout(this.closeTimer);
@@ -711,16 +780,13 @@
711780 send(frame) {
712781 if (this.halted) return;
713782 this.open();
714 if (!this.ws) return;
715 this.pending = { ...(this.pending ?? {}), ...frame };
716 if (this.timer) return;
717 this.timer = setTimeout(() => {
718 this.timer = null;
719 const data = JSON.stringify(this.pending); this.pending = null;
720 if (!this.ws) return;
721 if (this.ws.readyState === 1) this.ws.send(data);
722 else if (this.ws.readyState === 0) this.ws.addEventListener("open", () => this.ws?.send(data), { once: true });
723 }, 33);
783 if (this.ws) this.frames.send(frame);
784 },
785 /** Now, not at the next slot, with anything pending: a finger lifted and this is where its joints stop. */
786 flush(frame) {
787 if (this.halted) return;
788 this.open();
789 if (this.ws) this.frames.flush(frame);
724790 },
725791 end() { clearTimeout(this.closeTimer); this.closeTimer = setTimeout(() => { const ws = this.ws; this.ws = null; this.halted = false; ws?.close(); }, 250); },
726792 };
@@ -771,6 +837,7 @@
771837 $("stop").onclick = stop;
772838 // The emergency stop: the server halts whatever drives the arm and holds it with torque on until an admin clears it.
773839 async function eStop() {
840 driver.releaseAll();
774841 const r = await api("/estop", {});
775842 msg(r.ok ? "EMERGENCY STOP: the arm holds; an admin must clear it" : r.error, true);
776843 if (stream.ws) stream.ws.close();
@@ -1196,6 +1263,131 @@
11961263 for (const b of document.querySelectorAll("#reach [data-grip]")) jog(b, 6, () => Number(b.dataset.grip));
11971264 }
11981265
1266 // ---------------------------------------------------------------- touch driving (src/ui/drive.js)
1267 // Upright: a chip picks the joint, the lever's deflection is its speed, ▲ and ▼ nudge it and jog when held.
1268 // On its side: two thumb sticks over the picture (left J1/J2, right J3/J4 or J5/gripper) and the gripper between them.
1269 // Every finger is its own holder; the instant one lifts, its joints' targets go out as they stand and the arm settles there.
1270 const driver = new Driver({
1271 send: (f) => stream.send(f),
1272 flush: (f) => stream.flush(f),
1273 end: () => stream.end(),
1274 begin: () => {
1275 if (!canDrive({ mayDrive: mayDrive(), estop: status?.estop, reconnecting: status?.reconnecting, limits })) return false;
1276 stream.open();
1277 return true;
1278 },
1279 limits: () => limits,
1280 speed: () => Number($("rate").value) || 300,
1281 onStop: (id) => {
1282 navigator.vibrate?.(30);
1283 for (const e of [$("lever"), $("stickL"), $("stickR")]) { e.classList.add("end"); setTimeout(() => e.classList.remove("end"), 400); }
1284 msg(`${LABEL[id]?.[1] ?? id} is at the end of its range`);
1285 },
1286 });
1287 driver.watch(window, document);
1288 // Turning the phone swaps the controls under the thumbs: whatever was held stops, and on its side the sticks come up.
1289 sideways.addEventListener("change", () => { driver.releaseAll(); if (sideways.matches && shell.classList.contains("driving")) setBare(true); });
1290 const tap = () => navigator.vibrate?.(8);
1291 const LABEL = { 1: ["J1", "Base"], 2: ["J2", "Shoulder"], 3: ["J3", "Elbow"], 4: ["J4", "Wrist"], 5: ["J5", "Roll"], 6: ["Grip", "Gripper"] };
1292 const ENDS = { 1: ["right", "left"], 2: ["up", "down"], 3: ["up", "down"], 4: ["up", "down"], 5: ["roll +", "roll −"], 6: ["open", "close"] };
1293 let picked = (() => { try { const n = Number(localStorage.getItem("so101.joint")); return LABEL[n] ? n : 2; } catch { return 2; } })();
1294 const lever = $("lever");
1295 function pickJoint(id) {
1296 if (driver.drive.holders.has("lever")) driver.hold("lever", { [id]: 0 }); // the joint that was moving stops; the finger now drives this one
1297 picked = id;
1298 try { localStorage.setItem("so101.joint", String(id)); } catch {}
1299 for (const b of $("chips").children) { const on = Number(b.dataset.id) === id; b.setAttribute("aria-checked", String(on)); b.tabIndex = on ? 0 : -1; }
1300 $("oneName").textContent = LABEL[id][1];
1301 $("nudgeUp").querySelector("small").textContent = ENDS[id][0]; $("nudgeDown").querySelector("small").textContent = ENDS[id][1];
1302 $("nudgeUp").setAttribute("aria-label", `${LABEL[id][1]} a step ${ENDS[id][0]}`); $("nudgeDown").setAttribute("aria-label", `${LABEL[id][1]} a step ${ENDS[id][1]}`);
1303 lever.setAttribute("aria-label", `${LABEL[id][1]}: push ${ENDS[id][0]} or ${ENDS[id][1]}, let go to stop`);
1304 if (status) renderTouch(status);
1305 }
1306 $("chips").replaceChildren(...Object.entries(LABEL).map(([id, [short, long]]) => {
1307 const b = el("button", { type: "button", onclick: () => { tap(); pickJoint(Number(id)); } }, el("b", { textContent: short }), el("small", { textContent: long }));
1308 b.dataset.id = id; b.setAttribute("role", "radio"); b.setAttribute("aria-label", `${short} ${long}`);
1309 return b;
1310 }));
1311 $("chips").addEventListener("keydown", (e) => {
1312 const d = e.key === "ArrowRight" || e.key === "ArrowDown" ? 1 : e.key === "ArrowLeft" || e.key === "ArrowUp" ? -1 : 0;
1313 if (!d) return;
1314 e.preventDefault(); pickJoint(((picked - 1 + d + 6) % 6) + 1); $("chips").children[picked - 1]?.focus();
1315 });
1316 pickJoint(picked);
1317 /** Where the picked joint is: degrees from the middle of its range, the gripper as how far open; the green bar on the lever likewise. */
1318 function renderTouch(d) {
1319 const j = d.joints.find((x) => x.id === picked), l = limits[picked];
1320 if (!j || j.position == null || !l) { $("onePos").textContent = "—"; return; }
1321 const share = (j.position - l.min) / (l.max - l.min);
1322 const up = UP[picked] > 0 ? share : 1 - share; // the bar fills toward the lever's top end
1323 lever.style.setProperty("--at", String(Math.max(0, Math.min(1, up)).toFixed(3)));
1324 $("onePos").textContent = picked === 6 ? `${Math.round(share * 100)}% open` : `${(((j.position - mid(picked)) * 360) / 4096).toFixed(0)}°`;
1325 }
1326 bindSurface(lever, driver, "lever", (_, y) => ({ [picked]: shape(y) * UP[picked] }), {
1327 haptic: tap,
1328 show: (_, y) => {
1329 $("leverKnob").style.transform = `translateY(${(-y * (lever.clientHeight / 2 - 38)).toFixed(1)}px)`;
1330 lever.setAttribute("aria-valuenow", String(Math.round(y * 100)));
1331 },
1332 });
1333 // ▲ ▼: a tap is one step, a hold jogs at half speed; arrow keys on the lever step too.
1334 const step = () => Math.max(5, Math.round((Number($("rate").value) || 300) / 15));
1335 for (const b of [$("nudgeUp"), $("nudgeDown")]) {
1336 let pointer = null, timer = null;
1337 const dir = () => Number(b.dataset.nudge) * UP[picked];
1338 b.addEventListener("pointerdown", (e) => {
1339 if (pointer !== null) return;
1340 e.preventDefault(); pointer = e.pointerId;
1341 try { b.setPointerCapture(e.pointerId); } catch {}
1342 tap(); b.classList.add("held");
1343 driver.nudge(picked, dir() * step());
1344 timer = setTimeout(() => { timer = null; if (pointer !== null) driver.hold("nudge", { [picked]: dir() * 0.5 }); }, 350);
1345 });
1346 const up = (e) => { if (e.pointerId !== pointer) return; pointer = null; clearTimeout(timer); b.classList.remove("held"); driver.release("nudge"); };
1347 for (const ev of ["pointerup", "pointercancel", "lostpointercapture"]) b.addEventListener(ev, up);
1348 b.addEventListener("keydown", (e) => { if (e.key === "Enter" || e.key === " ") { e.preventDefault(); e.stopPropagation(); driver.nudge(picked, dir() * step()); } });
1349 b.addEventListener("contextmenu", (e) => e.preventDefault());
1350 }
1351 lever.addEventListener("keydown", (e) => {
1352 const d = e.key === "ArrowUp" ? 1 : e.key === "ArrowDown" ? -1 : 0;
1353 if (d) { e.preventDefault(); driver.nudge(picked, d * UP[picked] * step()); }
1354 });
1355 // The sticks: a disc, so a corner is no faster than an edge; each stick has its own finger.
1356 const disc = (x, y) => { const r = Math.hypot(x, y); return r > 1 ? [x / r, y / r] : [x, y]; };
1357 const knob = (stick) => (x, y) => {
1358 const [dx, dy] = disc(x, y), r = stick.clientWidth * 0.3;
1359 stick.querySelector(".knob").style.transform = `translate(${(dx * r).toFixed(1)}px, ${(-dy * r).toFixed(1)}px)`;
1360 };
1361 bindSurface($("stickL"), driver, "left", (x, y) => { const [dx, dy] = disc(x, y); return { 1: shape(dx) * UP[1], 2: shape(dy) * UP[2] }; }, { show: knob($("stickL")), haptic: tap });
1362 let wrist = false;
1363 const right = () => (wrist ? { x: 5, y: 6 } : { x: 4, y: 3 });
1364 const rightStick = bindSurface($("stickR"), driver, "right", (x, y) => { const [dx, dy] = disc(x, y), m = right(); return { [m.x]: shape(dx) * UP[m.x], [m.y]: shape(dy) * UP[m.y] }; }, { show: knob($("stickR")), haptic: tap });
1365 const labelRight = () => {
1366 $("capR").textContent = wrist ? "J5 roll ↔ · grip ↕" : "J4 wrist ↔ · J3 elbow ↕";
1367 $("stickR").setAttribute("aria-label", wrist ? "Right stick: sideways rolls the wrist, up opens the gripper" : "Right stick: sideways bends the wrist, up and down lifts the elbow");
1368 $("swap").textContent = wrist ? "Arm" : "Wrist"; $("swap").setAttribute("aria-pressed", String(wrist));
1369 };
1370 $("swap").onclick = () => { rightStick.end(); wrist = !wrist; tap(); labelRight(); };
1371 labelRight();
1372 bindSurface($("gripOpen"), driver, "open", () => ({ 6: UP[6] }), { haptic: tap });
1373 bindSurface($("gripClose"), driver, "close", () => ({ 6: -UP[6] }), { haptic: tap });
1374
1375 // Keep the screen on while it is your turn: a phone that dims mid-drive is a hand off the controls.
1376 let wakeLock = null, waking = false, wantAwake = false;
1377 async function wake(on) {
1378 wantAwake = on;
1379 if (!on) { const l = wakeLock; wakeLock = null; await l?.release().catch(() => {}); return; }
1380 if (wakeLock || waking || document.hidden || !navigator.wakeLock) return;
1381 waking = true;
1382 try {
1383 const l = await navigator.wakeLock.request("screen");
1384 if (!wantAwake) await l.release();
1385 else { wakeLock = l; l.addEventListener("release", () => { if (wakeLock === l) wakeLock = null; }); }
1386 } catch { /* refused (battery saver): driving works anyway */ }
1387 waking = false;
1388 }
1389 document.addEventListener("visibilitychange", () => { if (!document.hidden && wantAwake) wake(true); });
1390
11991391 // ---------------------------------------------------------------- screen
12001392 const app = $("app");
12011393 const fullscreen = () => (document.fullscreenElement ? document.exitFullscreen() : shell.requestFullscreen());
@@ -1284,7 +1476,10 @@
12841476 $("t-controls").hidden = !mayDrive();
12851477 $("t-admin").hidden = !owner;
12861478 $("t-chat").hidden = $("t-queue").hidden = !relayed;
1287 for (const id of ["joints", "poses", "drive", "reach"]) $(id).inert = !can;
1479 for (const id of ["joints", "poses", "drive", "reach", "touch", "sticks"]) $(id).inert = !can;
1480 shell.classList.toggle("driving", can);
1481 if (!can) driver.releaseAll();
1482 wake(can);
12881483 $("stop").hidden = !(can || owner);
12891484 $("estop").disabled = !(mayDrive() || owner);
12901485 $("estop").title = $("estop").disabled ? "Emergency stop: the driver and the admins can press it" : "Emergency stop (Esc): halt, hold with torque, until an admin clears it";
@@ -1294,7 +1489,11 @@
12941489 if (!can) reachEnd();
12951490 if (!can && leaderStreaming()) stopLeaderDrive(); // the server closes it too; the page need not wait for that
12961491 // Your turn just came: open the controls. It just ended: back to the chat.
1297 if (mayDrive() !== drove) { drove = mayDrive(); if (relayed) showTab(drove ? "controls" : "chat"); }
1492 if (mayDrive() !== drove) {
1493 drove = mayDrive();
1494 if (relayed) showTab(drove ? "controls" : "chat");
1495 if (drove && sideways.matches) setBare(true); // on its side, a turn starts on the thumb sticks
1496 }
12981497 else if ($(`t-${tab}`).hidden) showTab(tab);
12991498 }
13001499 const serverNow = () => Date.now() + lobby.offset;
@@ -1595,13 +1794,19 @@
15951794 const wasStopped = Boolean(status?.estop);
15961795 rtts.push(performance.now() - started); if (rtts.length > 10) rtts.shift(); // the link's round trip, for the camera plan
15971796 status = d;
1797 driver.seed(d.joints.filter((j) => j.position != null).map((j) => [j.id, j.position]));
15981798 if (Boolean(d.estop) !== wasStopped) applyLock();
1799 // Stopped, reconnecting or no longer this person's turn: a held thumb lets go, and its zero goes out now.
1800 if (!canDrive({ mayDrive: mayDrive(), estop: d.estop, reconnecting: d.reconnecting, limits })) driver.releaseAll();
1801 // The round trip to the arm, which every command also makes: shown once it is long enough to feel.
1802 const lag = [...rtts].sort((a, b) => a - b)[rtts.length >> 1] ?? 0;
1803 $("lag").hidden = rtts.length < 3 || lag <= 250; $("lag").textContent = `· ${Math.round(lag)} ms lag`;
15991804 $("dot").className = d.reconnecting ? "dot warn" : "dot on";
16001805 $("state").textContent = d.reconnecting ? "reconnecting…" : d.driving === null ? "idle" : d.driving === "record" ? `recording ${d.recording}` : leaderStreaming() ? `leader · ${leader.frames} frames` : d.driving;
16011806 $("estop").classList.toggle("on", Boolean(d.estop)); $("estop").textContent = d.estop ? "Stopped" : "Stop";
16021807 $("record").textContent = d.driving === "record" ? "■ Stop recording" : "● Record";
16031808 for (const id of ["hold", "releaseBtn", "play"]) $(id).disabled = d.driving !== null;
1604 if (tab === "controls") render(d);
1809 if (tab === "controls") { render(d); renderTouch(d); }
16051810 banner(d); applyModel(d);
16061811 $("readout").innerHTML = `<span>lowest part <b>${Number.isFinite(d.lowestMm) ? d.lowestMm : "—"} mm</b> above the table</span><span>zone <b>${d.zone === "enforce" || d.zone === "warn" || d.zone === "off" ? d.zone : "—"}</b></span>`;
16071812 } catch {

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs