Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

workbenchthis machine

/precision reports browser→relay and read-back→browser hops on a clock the page aligns with the relay #498

Merged opened this

Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.

Merged

smith/arm:smith/482-precision-reports-browser-relay-and-read → main 2 commits fork 2 threads · 2 unresolved

  • Merged. smith/482-precision-reports-browser-relay-and-read was merged into main as 286a4d9.

Review

Sign in to review this change.
19 files changed +387 −14
Software/so101/README.md +14
@@ -24,3 +24,17 @@ It is off in previews, without `ARM_PI_SSH_KEY`, with no arm server, and when th
2424 On the Pi: a `crew` user whose `authorized_keys` holds the key's public half, with write access to `<dir>`'s parent, and a sudoers entry for the restart command with no password, e.g. `crew ALL=(root) NOPASSWD: /bin/systemctl restart so101`. The host key is accepted on first use and kept in `FORGE_APP_DATA/pi-known-hosts`.
2525
2626 Who sees what: a visitor's page says only "Pi update pending" while the commits differ. The owner and moderators see both commits and the last sync outcome on `/admin` (Versions) and with `/versions` in the terminal.
27
28 ## Reading the latency hops
29
30 `GET /precision` has `hops`, each with `p50`, `p95` and `max` in ms, over the last ten minutes and per drive session; on `/admin` (Precision) and with `/precision` in the terminal (owner and moderators) they are listed as "latency by hop", p50 then p95. Each is one crossing, timed on clocks aligned by a four-timestamp ping, so none is skewed by the machines' clocks:
31
32 | Hop | From → to, timed by |
33 | --- | --- |
34 | `browser→relay` | the driver's page stamps a command with `t`; the relay corrects it by the page's offset (`src/ui/clock-sync.js`, over `/live`) |
35 | `relay→Pi` | the relay stamps the command it forwards; the arm server compares on arrival |
36 | `bus→move` | the arm server's own: a `Goal_Position` write to the first read-back past 3 ticks |
37 | `read-back→relay` | the arm server stamps each status with its read time; the relay compares on arrival |
38 | `read-back→browser` | the same stamp, carried to the page (on the relay's clock), compared there |
39
40 `latencyP95Ms` is the sum of the p95s that were measured. The link hops are timed against the real arm only, and only the current driver's page reports its clock (at most once a second); with the simulation, or nobody driving, the list is empty and says so.
Software/so101/bench/precision.ts +1 −1
@@ -72,7 +72,7 @@ export async function bench(script: readonly Step[] = gesture()): Promise<Card>
7272 const ws = new WebSocket(`${arm.url.replace("http", "ws")}/follow`);
7373 await new Promise((r, j) => (ws.once("open", r), ws.once("error", j)));
7474 // Loopback is its own clock: the stamp is the send time, and a watcher reports each status's age as the relay would.
75 const watch = new WebSocket(`ws://127.0.0.1:${server.port}/watch`);
75 const watch = new WebSocket(`${arm.url.replace("http", "ws")}/watch`);
7676 await new Promise((r, j) => (watch.once("open", r), watch.once("error", j)));
7777 watch.on("message", (data) => {
7878 const { readAt } = JSON.parse(String(data)) as { readAt?: number };
Software/so101/src/lobby.security.test.ts +28
@@ -151,4 +151,32 @@ describe("lobby security", () => {
151151 owner.send({ type: "end", id: "u-alice" });
152152 await bob.next((m) => m.type === "state" && m.driver === null);
153153 });
154
155 test("only the driver's page may report a clock, and a socket is answered a sync or taken a clock once a second", async () => {
156 const port = await start();
157 const bob = await join(port, BOB);
158 const alice = await join(port, ALICE);
159 const guest = await join(port, {});
160 bob.send({ type: "join" });
161 await bob.next((m) => m.type === "state" && m.driver?.name === "bob");
162 const heard: number[][] = [];
163 const off = lobby!.onReadBack((ms) => heard.push(ms));
164 for (const c of [alice, guest]) {
165 c.send({ type: "clock", offsetMs: 5, readBack: [1, 2] });
166 await settle(c);
167 }
168 expect(heard).toEqual([]);
169 expect(lobby!.pageOffset("u-alice")).toBeNull();
170 bob.send({ type: "clock", offsetMs: 7000, readBack: [1, 2] });
171 bob.send({ type: "clock", offsetMs: 1, readBack: [9] }); // the same second: not taken
172 await settle(bob);
173 expect(heard).toEqual([[1, 2]]);
174 expect(lobby!.pageOffset("u-bob")).toBe(7000);
175 // sync: answered to anyone, at most once a second
176 guest.send({ type: "sync", t0: 5 });
177 guest.send({ type: "sync", t0: 6 });
178 await settle(guest);
179 expect(guest.heard.filter((m) => m.type === "sync").map((m) => m.t0)).toEqual([5]);
180 off();
181 });
154182 });
Software/so101/src/lobby.ts +42
@@ -27,6 +27,8 @@
2727 * {"type": "report", "id": 12} signed in, not banned: report a chat line; answered to you alone with {"type": "reported", "id"}, a repeat too
2828 * {"type": "report", …} anyone: the page's own timings (src/viewer-reports.ts), every 5 s (no "id")
2929 * {"type": "who"} anyone: who is here, answered to that socket alone, at most once a second (people: name, colour?, guest, role)
30 * {"type": "sync", "t0": ms} the page's clock ping, answered with {"type": "sync", t0, t1, t2} (src/page-clock.ts)
31 * {"type": "clock", "offsetMs", "readBack"} the page's offset to the relay and the read-back hops it saw (src/page-clock.ts)
3032 * {"type": "pong"} the answer to {"type": "ping"}
3133 * {"type": "history", "before": 12} anyone: up to HISTORY_PAGE lines older than that one
3234 * Server → client: hello (who you are, and whether you may moderate: delete a line), you (an anonymous socket only: its guest name and colour, once), state (the turn, the line, pause, the announcement and the speed cap),
@@ -57,6 +59,8 @@ import { identify, LOCAL, type Person, type Who } from "./identity.ts";
5759 import { VisitorCounts } from "./visitor-counts.ts";
5860 import { Turns, type TurnEvent } from "./turns.ts";
5961 import { parseReport, ViewerReports } from "./viewer-reports.ts";
62 import { wallMs } from "./clock.ts";
63 import { parseClock } from "./page-clock.ts";
6064
6165 /** How long a mute lasts, by the name the page and the terminal send; null is for good. */
6266 export const MUTE_FOR: Record<string, number | null> = { "10m": 600_000, "1h": 3_600_000, "24h": 86_400_000, forever: null };
@@ -108,6 +112,10 @@ interface Seat {
108112 guest?: { id: string; name: GuestName };
109113 /** When this socket's last "who" was answered. */
110114 whoAt?: number;
115 /** The relay's clock minus this page's, as the page last reported it. */
116 offsetMs?: number;
117 /** When this socket's last sync and clock frames were taken. */
118 clockAt?: { sync?: number; clock?: number };
111119 }
112120
113121 const MAX_FRAME = 4096;
@@ -140,6 +148,7 @@ export class Lobby {
140148 private readonly seats = new Set<Seat>();
141149 private readonly changeListeners = new Set<() => void>();
142150 private readonly endListeners = new Set<(who: Person) => void>();
151 private readonly readBackListeners = new Set<(ms: number[]) => void>();
143152 private readonly arriveListeners = new Set<(tell: (text: string) => void) => void>();
144153 private readonly pinger: ReturnType<typeof setInterval>;
145154 private readonly silentMs: number;
@@ -300,6 +309,18 @@ export class Lobby {
300309 return () => this.changeListeners.delete(listener);
301310 }
outdated

The comment "Called with whoever's turn just ended" belongs on onTurnEnd; it now documents onReadBack. Move onReadBack/pageOffset above it.

302311
312 /** Called with the read-back→browser hops the driver's page reported. */
313 onReadBack(listener: (ms: number[]) => void): () => void {
314 this.readBackListeners.add(listener);
315 return () => this.readBackListeners.delete(listener);
316 }
317
318 /** The relay's clock minus that person's page's, ms; null until a page of theirs has reported one. */
319 pageOffset(id: string): number | null {
320 for (const s of this.seats) if (s.who?.id === id && s.offsetMs !== undefined) return s.offsetMs;
321 return null;
322 }
323
303324 /** Called with whoever's turn just ended, for whatever reason: their streams and moves must stop. */
304325 onTurnEnd(listener: (who: Person) => void): () => void {
305326 this.endListeners.add(listener);
@@ -408,6 +429,27 @@ export class Lobby {
408429 const m = message as Record<string, unknown>;
outdated

No identity or rate gate on sync/clock: any guest can inject hops into the scorecard. Gate on a signed-in, non-banned person (ideally the driver) and limit to about 1 per second per socket.

409430 const who = seat.who;
410431 if (m.type === "pong") return;
432 if (m.type === "sync" || m.type === "clock") {
433 // Cheap to answer but not free, and the hops feed the scorecard: at most one a second per kind from a socket, and a clock only from the driver's page.
434 const at = this.now();
435 const last = (seat.clockAt ??= {});
436 const kind = m.type;
437 if (last[kind] !== undefined && at - last[kind]! < 1000) return;
438 last[kind] = at;
439 }
440 if (m.type === "sync") {
441 const t1 = wallMs();
442 if (typeof m.t0 === "number" && Number.isFinite(m.t0)) this.send(seat, { type: "sync", t0: m.t0, t1, t2: wallMs() });
443 return;
444 }
445 if (m.type === "clock") {
446 if (!who || this.barred(who, seat.from) || !(this.local || this.turns.isDriver(who.id))) return;
447 const clock = parseClock(m);
448 if (!clock) return this.send(seat, { type: "error", error: "that clock report is numbers in range" });
449 seat.offsetMs = clock.offsetMs;
450 if (clock.readBack.length) for (const listener of this.readBackListeners) listener(clock.readBack);
451 return;
452 }
411453 if (m.type === "report" && "id" in m) return this.reportLine(seat, m.id);
412454 if (m.type === "report") {
413455 const report = parseReport(m);
Software/so101/src/page-clock.test.ts added +14
@@ -0,0 +1,14 @@
1 import { expect, test } from "bun:test";
2 import { parseClock } from "./page-clock.ts";
3
4 test("an offset and up to forty hops in range; anything else is refused, not clamped", () => {
5 expect(parseClock({ offsetMs: -7000.5, readBack: [0, 3.2] })).toEqual({ offsetMs: -7000.5, readBack: [0, 3.2] });
6 expect(parseClock({ offsetMs: 12 })).toEqual({ offsetMs: 12, readBack: [] });
7 expect(parseClock({ offsetMs: "1" })).toBeNull();
8 expect(parseClock({ offsetMs: Infinity })).toBeNull();
9 expect(parseClock({ offsetMs: 1e15 })).toBeNull();
10 expect(parseClock({ offsetMs: 1, readBack: [-1] })).toBeNull();
11 expect(parseClock({ offsetMs: 1, readBack: [60_001] })).toBeNull();
12 expect(parseClock({ offsetMs: 1, readBack: "3" })).toBeNull();
13 expect(parseClock({ offsetMs: 1, readBack: new Array(41).fill(1) })).toBeNull();
14 });
Software/so101/src/page-clock.ts added +28
@@ -0,0 +1,28 @@
1 /**
2 * What a page says about its clock over /live, so the relay can time the hops either side of it:
3 * {"type": "sync", "t0": ms} answered with {"type": "sync", t0, t1, t2}: the relay's receive and send times (the four-timestamp ping, src/hops.ts)
4 * {"type": "clock", "offsetMs": ms, "readBack": [ms…]} the page's offset to the relay (relay minus page, from src/ui/clock-sync.js) and the read-back→browser hops it saw
5 * Only finite numbers in range are taken; anything else is refused, never clamped.
6 */
7
8 import { MAX_HOP_MS } from "./precision.ts";
9
10 /** The most read-back hops one frame carries. */
11 export const MAX_READ_BACK = 40;
12 /** An offset is a wall clock minus a page clock: a few billion ms at most, in either direction. */
13 const MAX_OFFSET_MS = 1e14;
14
15 export interface PageClock {
16 offsetMs: number;
17 readBack: number[];
18 }
19
20 /** The clock frame's numbers, or null when it is not exactly an offset and up to MAX_READ_BACK hops in range. */
21 export function parseClock(m: Record<string, unknown>): PageClock | null {
22 const { offsetMs, readBack } = m;
23 if (typeof offsetMs !== "number" || !Number.isFinite(offsetMs) || Math.abs(offsetMs) > MAX_OFFSET_MS) return null;
24 if (readBack === undefined) return { offsetMs, readBack: [] };
25 if (!Array.isArray(readBack) || readBack.length > MAX_READ_BACK) return null;
26 if (!readBack.every((v) => typeof v === "number" && Number.isFinite(v) && v >= 0 && v <= MAX_HOP_MS)) return null;
27 return { offsetMs, readBack: readBack as number[] };
28 }
Software/so101/src/precision-panel.test.ts +8
@@ -21,6 +21,14 @@ describe("reading /precision", () => {
2121 expect(readPrecision(scorecard())!.worstJoint).toBe("elbow_flex");
2222 });
2323
24 test("the hops that were timed are listed with p50 and p95, and an untimed one is left out", () => {
25 const hops = { "browser→relay": { n: 3, p50: 4, p95: 9, max: 9 }, "relay→Pi": { n: 0, p50: null, p95: null, max: null } };
26 const p = readPrecision(scorecard({ hops }))!;
27 expect(p.hops).toEqual([{ name: "browser→relay", p50: 4, p95: 9 }]);
28 expect(precisionSection(p, esc)).toContain("browser→relay");
29 expect(readPrecision(scorecard())!.hops).toEqual([]);
30 });
31
2432 test("anything that is not a scorecard is null; a field of the wrong type is a gap, never an error", () => {
2533 expect(readPrecision("nope")).toBeNull();
2634 expect(readPrecision({ commit: 1, sessions: [] })).toBeNull();
Software/so101/src/precision-panel.ts +22 −1
@@ -20,6 +20,8 @@ export interface PrecisionPanel {
2020 worstJoint: string | null;
2121 sessions: number;
2222 headline: Headline[];
23 /** Each link hop with a measurement, as the window (else the last session) has it. */
24 hops: Array<{ name: string; p50: number | null; p95: number | null }>;
2325 }
2426
2527 const num = (v: unknown): number | null => (typeof v === "number" && Number.isFinite(v) ? v : null);
@@ -39,7 +41,13 @@ export function readPrecision(body: unknown): PrecisionPanel | null {
3941 const snaps = num(r.snapBacks), releases = num(r.releases);
4042 return snaps !== null && releases ? Number(((snaps / releases) * 100).toFixed(1)) : null;
4143 };
44 const hops = Object.entries(obj(window.hops)).flatMap(([name, h]) => {
45 const spread = obj(h);
46 const n = text(name, 24);
47 return n && num(spread.p95) !== null ? [{ name: n, p50: num(spread.p50), p95: num(spread.p95) }] : [];
48 });
4249 return {
50 hops,
4351 commit: text(card.commit) ?? "unknown",
4452 worstJoint: text(window.worstJoint) ?? text(last.worstJoint),
4553 sessions: rows.length,
@@ -75,5 +83,18 @@ export function precisionSection(p: PrecisionPanel | null | undefined, esc: (s:
7583 return `<li><span>${esc(h.label)}</span><span><b>${esc(value)}</b>${spark ? ` <span aria-hidden="true">${spark}</span>` : ""}</span></li>`;
7684 })
7785 .join("");
78 return `${head}<p>Worst joint: <b>${esc(p.worstJoint ?? "—")}</b> · last 10 minutes, else the last session · trend over ${p.sessions} session${p.sessions === 1 ? "" : "s"} · commit ${esc(p.commit)}</p><ol class="bars precision" aria-label="Precision, headline numbers">${items}</ol></section>`;
86 return `${head}<p>Worst joint: <b>${esc(p.worstJoint ?? "—")}</b> · last 10 minutes, else the last session · trend over ${p.sessions} session${p.sessions === 1 ? "" : "s"} · commit ${esc(p.commit)}</p><ol class="bars precision" aria-label="Precision, headline numbers">${items}</ol>${hopList(p.hops, esc)}</section>`;
87 }
88
89 /** What is said where the hops would be, when none was timed: the simulation has no link to time. */
90 export const NO_HOPS = "No link hops timed: they are measured against the real arm only, while someone drives it.";
91
92 /** One line per timed hop: name, then p50 and p95 in ms. */
93 export const hopLines = (hops: PrecisionPanel["hops"]): string[] => hops.map((h) => `${h.name} p50 ${h.p50 ?? "—"} · p95 ${h.p95} ms`);
94
95 /** The hops that were timed, each with its p50 and p95; nothing when none was. */
96 function hopList(hops: PrecisionPanel["hops"], esc: (s: string) => string): string {
97 if (!hops.length) return `<p class="empty">${esc(NO_HOPS)}</p>`;
98 const rows = hops.map((h) => `<li><span>${esc(h.name)}</span><span><b>${esc(`${h.p50 ?? "—"} / ${h.p95} ms`)}</b></span></li>`).join("");
99 return `<h3>Latency by hop, p50 / p95</h3><ol class="bars precision" aria-label="Latency by hop">${rows}</ol>`;
79100 }
Software/so101/src/precision.ts +5 −1
@@ -49,6 +49,10 @@ export function parseSnapBack(body: unknown): { cause: Cause; sizeDeg: number }
4949 /** The hops timed across the link, by name: a duration is only counted when it is from 0 to MAX_HOP_MS. */
5050 export const RELAY_HOP = "relay→Pi";
5151 export const READ_BACK_HOP = "read-back→relay";
52 /** The two the page's clock lets the relay time: a frame's `t` against its arrival, and a status's read time against its arrival at the page. */
53 export const BROWSER_HOP = "browser→relay";
54 export const PAGE_HOP = "read-back→browser";
55 export const LINK_HOPS: readonly string[] = [RELAY_HOP, READ_BACK_HOP, BROWSER_HOP, PAGE_HOP];
5256 export const MAX_HOP_MS = 60_000;
5357
5458 export type Kind = "error" | "arrive" | "overshoot" | "settle" | "busToMove";
@@ -144,7 +148,7 @@ export function summarize(records: Records, since = -Infinity): Card {
144148 for (const s of snaps) byCause[s.cause]++;
145149 const releases = records.releases.filter((at) => at >= since).length;
146150 const worst = joints.filter((j) => j.errorDeg.p95 !== null).sort((a, b) => b.errorDeg.p95! - a.errorDeg.p95!)[0];
147 const hops: Record<string, Spread> = { [RELAY_HOP]: spread(after(records.hops.get(RELAY_HOP) ?? []), 2), [READ_BACK_HOP]: spread(after(records.hops.get(READ_BACK_HOP) ?? []), 2), "bus→move": spread(allBus, 0) };
151 const hops: Record<string, Spread> = { ...Object.fromEntries(LINK_HOPS.map((name) => [name, spread(after(records.hops.get(name) ?? []), 2)])), "bus→move": spread(allBus, 0) };
148152 const measured = Object.values(hops).filter((h) => h.p95 !== null);
149153 return {
150154 trackingP95Deg: round(percentile(allErrors.map((t) => t / TICKS_PER_DEGREE), 95)),
Software/so101/src/relay.test.ts +51
@@ -11,6 +11,8 @@ import { CameraHub } from "./cameras.ts";
1111 import { wallMs } from "./clock.ts";
1212 import { stamper } from "./command-gate.ts";
1313 import { FakeBus } from "./fake-bus.ts";
14 import { Lobby } from "./lobby.ts";
15 import { ClockSync } from "./ui/clock-sync.js";
1416 import { OFFLINE_FRAME } from "./offline-frame.ts";
1517 import { parseRemote, sameOrigin, startRelay, type Remote, type RunningRelay } from "./relay.ts";
1618 import { startServer, type RunningServer } from "./server.ts";
@@ -447,6 +449,55 @@ describe("relay", () => {
447449 }
448450 });
449451
452 test("a page whose clock is 7 s behind has its hops timed on the relay's clock: browser→relay from a frame's t, read-back→browser from the status it got", async () => {
453 const SKEW = 7000;
454 const armClock = () => wallMs() + 5000;
455 const pageNow = () => wallMs() - SKEW;
456 const hops: Record<string, number[]> = {};
457 const wss = new WebSocketServer({ noServer: true });
458 camera = createServer((_req, res) => res.writeHead(404).end());
459 camera.on("upgrade", (req, socket, head) =>
460 wss.handleUpgrade(req, socket, head, (ws) => {
461 ws.on("message", (d) => {
462 const body = JSON.parse(String(d)) as { ping?: number; hop?: string; ms?: number[] };
463 if (typeof body.ping === "number") ws.send(JSON.stringify({ pong: { t0: body.ping, t1: armClock(), t2: armClock() } }));
464 if (body.hop) (hops[body.hop] ??= []).push(...body.ms!);
465 });
466 }),
467 );
468 const pump = setInterval(() => wss.clients.forEach((c) => c.send(JSON.stringify({ readAt: armClock() - 2, joints: [] }))), 10);
469 const sync = new ClockSync(pageNow);
470 const lobby = new Lobby({ local: true });
471 try {
472 relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(await listening(camera)), lobby, link: { ...FAST, staleMs: 500, syncEveryMs: 20 } });
473 const live = new WebSocket(`ws://127.0.0.1:${relay.port}/live`);
474 live.on("message", (d) => {
475 const m = JSON.parse(String(d));
476 if (m.type === "sync") sync.pong(m);
477 if (m.type === "status" && m.status) sync.status(m.status.readAt);
478 });
479 await new Promise((r, j) => (live.once("open", r), live.once("error", j)));
480 for (let i = 0; i < 4; i++) (live.send(JSON.stringify(sync.ping())), await sleep(20));
481 expect(Math.abs(sync.offset! - SKEW)).toBeLessThan(20);
482 await sleep(100);
483 live.send(JSON.stringify(sync.report()));
484 const ws = new WebSocket(`ws://127.0.0.1:${relay.port}/follow`);
485 await new Promise((r, j) => (ws.once("open", r), ws.once("error", j)));
486 await sleep(50);
487 ws.send(stamper(pageNow)({ 1: 2050 }));
488 await sleep(150);
489 for (const name of ["browser→relay", "read-back→browser"]) {
490 expect(hops[name]?.length).toBeGreaterThan(0);
491 for (const ms of hops[name]!) expect(ms).toBeGreaterThan(-5), expect(ms).toBeLessThan(100); // not the 7000 the clocks differ by
492 }
493 ws.close();
494 live.close();
495 } finally {
496 clearInterval(pump);
497 for (const c of wss.clients) c.terminate();
498 }
499 });
500
450501 test("a stream on the real arm goes quiet while the link reconnects, and flows again once it is back", async () => {
451502 // An arm server whose /watch can be silenced while its /follow stays up: what the relay passes on is counted.
452503 let watching = true;
Software/so101/src/relay.ts +16 −3
@@ -65,7 +65,7 @@ import { wallMs } from "./clock.ts";
6565 import { ArmLink, type LinkTimings } from "./arm-link.ts";
6666 import { MachineFeed } from "./machine-feed.ts";
6767 import { refusalBatch } from "./refusals.ts";
68 import { parseSnapBack } from "./precision.ts";
68 import { BROWSER_HOP, PAGE_HOP, parseSnapBack } from "./precision.ts";
6969 import { perServo } from "./machine-log.ts";
7070 import { parseFrame } from "./safety.ts";
7171 import { acceptsGzip, GzipCache, worthCompressing } from "./compress.ts";
@@ -89,7 +89,7 @@ import { proxyFor, whyNot } from "./egress.ts";
8989 import { CalibrationError } from "./leader-map.js";
9090 import type { Lobby } from "./lobby.ts";
9191 import { capBody, FrameCap } from "./speed-cap.ts";
92 import { Clock, CommandGate, HELD_CLOSE, isRelease } from "./command-gate.ts";
92 import { Clock, CommandGate, HELD_CLOSE, isRelease, parseCommand } from "./command-gate.ts";
9393 import { FLOOD_CLOSE, FrameRate, frameBytes, MAX_FRAME_BYTES, TOO_BIG_CLOSE } from "./frame-limit.ts";
9494
9595 export interface Remote {
@@ -456,7 +456,11 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
456456 const last = armLink!.last;
457457 const reconnecting = mode() === "reconnecting";
458458 if (!last) return { code: 503, body: { error: `${RECONNECTING}…`, source: source(), reconnecting } };
459 return { code: 200, body: { ...last, held: gate.held ?? last.held ?? null, source: source(), reconnecting, ...(reconnecting ? { seenAt: new Date(armLink!.lastAt).toISOString() } : {}) } };
459 // `readAt` is on the arm server's clock; the page compares it with its own aligned to ours, so it goes out on ours (or not at all).
460 const { readAt, ...rest } = last;
461 const offset = armLink!.offsetMs;
462 const aligned = typeof readAt === "number" && offset !== null ? { readAt: Number((readAt - offset).toFixed(3)) } : {};
463 return { code: 200, body: { ...rest, ...aligned, held: gate.held ?? last.held ?? null, source: source(), reconnecting, ...(reconnecting ? { seenAt: new Date(armLink!.lastAt).toISOString() } : {}) } };
460464 }
461465 const answer = await fetch(simulation + path, { cache: "no-store", signal: AbortSignal.timeout(readTimeoutMs) });
462466 const body = (await answer.json()) as Record<string, unknown>;
@@ -920,6 +924,10 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
920924 if (lobby && !lobby.turns.isDriver(driver) && !lobby.local) return void machine.note("clamp", "not passed on", { why: "the turn ended" });
921925 lobby?.activity(driver);
922926 }
927 // The page stamps `t` on its own clock and reports its offset to ours: browser→relay, for the arm server's scorecard.
928 const offset = driver !== null && real ? lobby?.pageOffset(driver) ?? null : null;
929 const sent = offset === null ? null : parseCommand(String(raw))?.t ?? null;
930 if (sent !== null) armLink?.report(BROWSER_HOP, wallMs() - (sent + offset!));
923931 const verdict = admission.admit(String(raw));
924932 if (!verdict.ok) return void machine.note("clamp", "not passed on", { why: verdict.drop });
925933 if (arm.readyState !== WebSocket.OPEN) return;
@@ -1049,6 +1057,10 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
10491057 // Its streams close (the arm server holds the arm when a stream ends) and a move, playback or recording stops.
10501058 const settingsChanged = lobby?.onSettingsChange(follow);
10511059 const arrived = lobby?.onArrive((tell) => (pushed ? tell(pushed) : void push()));
1060 // The read-back→browser hops a page measured go to the arm server's scorecard beside the relay's own.
1061 const readBacks = lobby?.onReadBack((ms) => {
1062 if (remote && onArm()) for (const one of ms) armLink?.report(PAGE_HOP, one);
1063 });
10521064 const turnEnded = lobby?.onTurnEnd((who) => {
10531065 for (const [browser, id] of streams) if (id === who.id) browser.close(1000, "your turn ended");
10541066 const base = remote && onArm() ? remote.http : simulation;
@@ -1085,6 +1097,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
10851097 clearInterval(summary);
10861098 settingsChanged?.();
10871099 turnEnded?.();
1100 readBacks?.();
10881101 arrived?.();
10891102 armLink?.close();
10901103 simLink?.close();
Software/so101/src/server.ts +3 −2
@@ -82,7 +82,7 @@ import { clamp, type Frame } from "./mapping.ts";
8282 import { MachineLog, perServo } from "./machine-log.ts";
8383 import { Metrics } from "./metrics.ts";
8484 import { wallMs } from "./clock.ts";
85 import { ARRIVE_TICKS, parseSnapBack, READ_BACK_HOP, RELAY_HOP, type Cause, type Tracker } from "./precision.ts";
85 import { ARRIVE_TICKS, LINK_HOPS, parseSnapBack, READ_BACK_HOP, RELAY_HOP, type Cause, type Tracker } from "./precision.ts";
8686 import { deployedCommit, PrecisionHistory } from "./precision-history.ts";
8787 import { MotionError, moveToPose, playTrajectory, recordTrajectory, STALL_SPAN_TICKS } from "./motion.ts";
8888 import { assertPoseName, parsePoseFile, serializePoseFile, type PoseFile } from "./poses.ts";
@@ -1538,7 +1538,7 @@ export async function startServer(bus: FeetechBus, options: ServerOptions = {}):
15381538 let watchedAt = 0;
15391539 /** What a watcher may say: {"ping": t0} is answered with {"pong": {t0, t1, t2}} (the four-timestamp ping, src/hops.ts); {"readBack": [ms…]} is the relay's read-back hop. Anything else is ignored. */
15401540 const watcherMessage = (text: string, t1: number): string | null => {
1541 let body: { ping?: unknown; readBack?: unknown } | null;
1541 let body: { ping?: unknown; readBack?: unknown; hop?: unknown; ms?: unknown } | null;
15421542 try {
15431543 body = JSON.parse(text);
15441544 } catch {
@@ -1546,6 +1546,7 @@ export async function startServer(bus: FeetechBus, options: ServerOptions = {}):
15461546 }
15471547 if (!body || typeof body !== "object") return null;
15481548 if (typeof body.ping === "number" && Number.isFinite(body.ping)) return JSON.stringify({ pong: { t0: body.ping, t1, t2: wallMs() } });
1549 if (typeof body.hop === "string" && LINK_HOPS.includes(body.hop) && Array.isArray(body.ms)) for (const ms of body.ms.slice(0, 64)) if (typeof ms === "number") precision.hop(body.hop, ms);
15491550 if (Array.isArray(body.readBack)) for (const ms of body.readBack.slice(0, 64)) if (typeof ms === "number") precision.hop(READ_BACK_HOP, ms);
15501551 return null;
15511552 };
Software/so101/src/terminal.test.ts +7 −1
@@ -210,7 +210,8 @@ describe("terminal table", () => {
210210 test("/precision prints the scorecard /admin shows, line by line as text, or the refusal; without a reader it says so", async () => {
211211 expect(typed(emptyRoom(), "/precision")).toEqual({ precision: true });
212212 const row = (trackingP95Deg: number) => ({ trackingP95Deg, arriveP95Ms: 200, snapBacks: 1, releases: 4, latencyP95Ms: 40, worstJoint: "elbow_flex" });
213 const body = { commit: "abc1234", window: { worstJoint: "wrist\x1b[31m_roll" }, sessions: [row(1), row(3)] };
213 const hops = { "browser→relay": { n: 4, p50: 3.5, p95: 8 }, "read-back→browser": { n: 2, p50: 6, p95: 11.25 }, "relay→Pi": { n: 0, p50: null, p95: null } };
214 const body = { commit: "abc1234", window: { worstJoint: "wrist\x1b[31m_roll", hops }, sessions: [row(1), row(3)] };
214215 const asked: Array<[string, unknown]> = [];
215216 const fetcher = (async (url: string, init: RequestInit) => (asked.push([url, init.headers]), Response.json(body))) as unknown as typeof fetch;
216217 const card = await cardReader({ PORT: "8102" }, fetcher)();
@@ -218,6 +219,11 @@ describe("terminal table", () => {
218219 const lines = precisionLines(card.panel!);
219220 expect(lines[0]).toMatch(/^Tracking error, p95 +3° +▁█$/);
220221 expect(lines).toContain("worst joint wrist[31m_roll · last 10 minutes, else the last session · trend over 2 sessions · commit abc1234");
222 expect(lines).toContain("latency by hop:");
223 expect(lines).toContain(" browser→relay p50 3.5 · p95 8 ms");
224 expect(lines).toContain(" read-back→browser p50 6 · p95 11.25 ms");
225 expect(lines.join("\n")).not.toContain("relay→Pi");
226 expect(precisionLines({ ...card.panel!, hops: [] }).at(-1)).toContain("real arm only");
221227 const refused = await cardReader({ PORT: "8102" }, (async () => Response.json({ error: "no such route" }, { status: 404 })) as unknown as typeof fetch)();
222228 expect(refused).toEqual({ status: 404, error: "precision is for the owner and moderators", panel: null });
223229 const printed: string[] = [];
Software/so101/src/terminal.ts +4 −2
@@ -46,7 +46,7 @@ import WebSocket from "ws";
4646 import { UNPRINTABLE } from "./chat.ts";
4747 import { ID, NAME, sign } from "./identity.ts";
4848 import { readServos, type ServoPanel } from "./servo-panel.ts";
49 import { headlineValue, readPrecision, sparkline, type PrecisionPanel } from "./precision-panel.ts";
49 import { headlineValue, hopLines, NO_HOPS, readPrecision, sparkline, type PrecisionPanel } from "./precision-panel.ts";
5050 import { presenceText } from "./ui/lobby-panel.js";
5151 import { entryLine, jointRows, linkRows, stateRows } from "./ui/machine.js";
5252
@@ -188,13 +188,15 @@ export function cardReader(env: Env, fetcher: typeof fetch = fetch): ReadCard {
188188 };
189189 }
190190
191 /** The panel's lines: each headline number with its sparkline, then the worst joint and the commit. */
191 /** The panel's lines: each headline number with its sparkline, the worst joint and the commit, then each link hop with its p50 and p95. */
192192 export function precisionLines(p: PrecisionPanel): string[] {
193193 if (!p.sessions && p.headline.every((h) => h.value === null)) return [plain(`nothing measured yet: drive the arm and the numbers appear · commit ${p.commit}`)];
194194 const width = Math.max(...p.headline.map((h) => h.label.length));
195195 return [
196196 ...p.headline.map((h) => plain(`${h.label.padEnd(width)} ${headlineValue(h)} ${sparkline(h.trend)}`).trimEnd()),
197197 plain(`worst joint ${p.worstJoint ?? "—"} · last 10 minutes, else the last session · trend over ${p.sessions} session${p.sessions === 1 ? "" : "s"} · commit ${p.commit}`),
198 "latency by hop:",
199 ...(p.hops.length ? hopLines(p.hops).map((l) => plain(` ${l}`)) : [` ${NO_HOPS}`]),
198200 ];
199201 }
200202
Software/so101/src/ui/clock-sync.js added +85
@@ -0,0 +1,85 @@
1 // @ts-check
2 /**
3 * The page's clock against the relay's, over /live: a {"type": "sync", "t0"} ping every SYNC_MS, answered with the relay's
4 * receive and send times (t1, t2); the offset is the relay's clock minus this page's performance.now(), from the
5 * lowest-delay of the last WINDOW answers (the four-timestamp ping of src/hops.ts). The relay times a command's browser→relay hop
6 * from its `t` and this offset; the page times read-back→browser from a status's `readAt` (already on the relay's clock)
7 * and sends both with the offset in a {"type": "clock"} frame every REPORT_MS.
8 */
9
10 export const SYNC_MS = 3000;
11 export const REPORT_MS = 5000;
12 const WINDOW = 8;
13 /** Read-back hops kept per report: the newest, a sample of the hop. */
14 const KEEP = 40;
15
16 /** @typedef {{ t0: number, t1: number, t2: number, t3: number }} Exchange */
17
18 /**
19 * The offset and delay of one exchange, or null when it is malformed.
20 * @param {Exchange} x
21 */
22 export function exchange(x) {
23 if (![x.t0, x.t1, x.t2, x.t3].every(Number.isFinite)) return null;
24 const delay = x.t3 - x.t0 - (x.t2 - x.t1);
25 return delay < 0 ? null : { offset: (x.t1 - x.t0 + (x.t2 - x.t3)) / 2, delay };
26 }
27
28 export class ClockSync {
29 /** @param {() => number} [now] */
30 constructor(now = () => performance.now()) {
31 this.now = now;
32 /** @type {Exchange[]} */ this.exchanges = [];
33 /** @type {number[]} */ this.readBack = [];
34 }
35
36 /** The frame to send for a ping. */
37 ping() { return { type: "sync", t0: this.now() }; }
38
39 /** The relay's answer to a ping, received now. @param {{ t0?: unknown, t1?: unknown, t2?: unknown }} m */
40 pong(m) {
41 const { t0, t1, t2 } = m;
42 if (typeof t0 !== "number" || typeof t1 !== "number" || typeof t2 !== "number") return;
43 this.exchanges.push({ t0, t1, t2, t3: this.now() });
44 if (this.exchanges.length > WINDOW) this.exchanges.shift();
45 }
46
47 /** The relay's clock minus this page's, ms; null until a ping is answered. */
48 get offset() {
49 let best = null;
50 for (const x of this.exchanges) {
51 const s = exchange(x);
52 if (s && (!best || s.delay < best.delay)) best = s;
53 }
54 return best ? best.offset : null;
55 }
56
57 /** A status arrived whose arm read it at `readAt` on the relay's clock. @param {unknown} readAt */
58 status(readAt) {
59 const offset = this.offset;
60 if (typeof readAt !== "number" || offset === null) return;
61 this.readBack.push(Number((this.now() + offset - readAt).toFixed(2)));
62 if (this.readBack.length > KEEP) this.readBack.shift();
63 }
64
65 /** The frame to send: the offset and the hops since the last, which it starts over; null until there is an offset. */
66 report() {
67 const offsetMs = this.offset;
68 if (offsetMs === null) return null;
69 const readBack = this.readBack.filter((ms) => ms >= 0);
70 this.readBack = [];
71 return { type: "clock", offsetMs: Number(offsetMs.toFixed(3)), readBack };
72 }
73 }
74
75 /**
76 * A ClockSync pinging and reporting through `say`; the lobby's frames reach it as "clock-sync" events (src/ui/lobby-panel.js).
77 * @param {(m: Record<string, unknown>) => void} say
78 */
79 export function start(say) {
80 const clock = new ClockSync();
81 const ping = () => say(clock.ping());
82 ping();
83 const timers = [setInterval(ping, SYNC_MS), setInterval(() => { const r = clock.report(); if (r) say(r); }, REPORT_MS)];
84 return Object.assign(clock, { stop: () => timers.forEach(clearInterval) });
85 }
Software/so101/src/ui/clock-sync.test.ts added +44
@@ -0,0 +1,44 @@
1 import { expect, test } from "bun:test";
2 import { ClockSync, exchange, start } from "./clock-sync.js";
3
4 test("an exchange gives the offset (relay minus page) and the delay, whatever the skew", () => {
5 // page clock 1000 behind the relay; 10 ms out, 2 ms on the relay, 10 ms back
6 expect(exchange({ t0: 0, t1: 1010, t2: 1012, t3: 22 })).toEqual({ offset: 1000, delay: 20 });
7 expect(exchange({ t0: 0, t1: 1, t2: 2, t3: NaN })).toBeNull();
8 expect(exchange({ t0: 10, t1: 0, t2: 0, t3: 5 })).toBeNull();
9 });
10
11 test("the offset is the lowest-delay answer of the last eight; none until one is answered", () => {
12 let now = 0;
13 const c = new ClockSync(() => now);
14 expect(c.offset).toBeNull();
15 expect(c.report()).toBeNull();
16 const answer = (out: number, back: number) => { now = 0; const { t0 } = c.ping(); now = back + out; c.pong({ t0, t1: 1000 + out, t2: 1000 + out }); };
17 answer(10, 10); // symmetric: offset 1000
18 answer(40, 2); // queued out: skewed, and slower
19 expect(c.offset).toBe(1000);
20 for (let i = 0; i < 8; i++) answer(10, 10);
21 expect(c.exchanges).toHaveLength(8);
22 c.pong({ t0: "x" });
23 expect(c.exchanges).toHaveLength(8);
24 });
25
26 test("a status's read time on the relay's clock gives read-back→browser; a report carries the offset and the hops, then starts over", () => {
27 let now = 0;
28 const c = new ClockSync(() => now);
29 c.status(5); // no offset yet: nothing to compare
30 c.pong({ t0: 0, t1: 1000, t2: 1000 });
31 now = 0;
32 c.status(1000 - 0 - 3.5); // page now 0 is relay 1000: read 3.5 ms before
33 c.status("x");
34 expect(c.report()).toEqual({ type: "clock", offsetMs: 1000, readBack: [3.5] });
35 expect(c.report()).toEqual({ type: "clock", offsetMs: 1000, readBack: [] });
36 });
37
38 test("start pings at once and reports through say", () => {
39 const sent: Array<Record<string, unknown>> = [];
40 const c = start((m) => sent.push(m));
41 expect(sent[0]).toMatchObject({ type: "sync" });
42 expect(c).toBeInstanceOf(ClockSync);
43 c.stop();
44 });
Software/so101/src/ui/lobby-panel.js +4 −3
@@ -62,8 +62,8 @@ export function whoList(el, m) {
6262 /** @param {number} attempt 1 for the first retry @returns {number} ms to wait: 100, 200, 400, 800, 1600, then 2000 at most: a restart costs a blink */
6363 export const backoff = (attempt) => Math.min(100 * 2 ** Math.max(0, attempt - 1), 2000);
6464
65 /** Tell the page's health meter (src/ui/health-meter.js) what happened. @param {string} what */
66 const tell = (what) => /** @type {any} */ (globalThis).dispatchEvent?.(new Event(what));
65 /** Tell the page (src/ui/health-meter.js, clock-sync.js) what happened. @param {string} what @param {unknown} [detail] */
66 const tell = (what, detail) => /** @type {any} */ (globalThis).dispatchEvent?.(detail === undefined ? new Event(what) : new CustomEvent(what, { detail }));
6767
6868 /**
6969 * @param {any} page what the page owns: said(lines) for new chat lines, $, el, relayed, tab(), showTab, msg, applyLock, applyStatus, noStatus, words, admin, note, refused
@@ -354,10 +354,11 @@ export function createRoom(page) {
354354 if (lobby.retry) { lobby.retry = 0; $("reconnect").classList.remove("on"); } // the first message: it is back
355355 const m = JSON.parse(e.data);
356356 if (m.type === "ping") return say({ type: "pong" });
357 if (m.type === "sync") return tell("clock-sync", m);
357358 if (m.type === "presence") { const t = presenceText(m); if (t !== null) { const b = $("presenceLine"); b.textContent = t; b.disabled = false; } return; }
358359 if (m.type === "who") return showWho(m);
359360 if (m.type === "refused") return page.refused?.(m);
360 if (m.type === "status") { tell("pose"); lobby.statusSeen = true; return m.status ? page.applyStatus(m.status) : page.noStatus(); }
361 if (m.type === "status") { tell("pose"); if (m.status?.readAt !== undefined) tell("clock-status", m.status.readAt); lobby.statusSeen = true; return m.status ? page.applyStatus(m.status) : page.noStatus(); }
361362 if (m.type === "hello") lobby.hello = m;
362363 else if (m.type === "state") { lobby.offset = m.now - Date.now(); lobby.state = m; }
363364 else if (m.type === "chat") return addChat(m.messages, true), showEarlier(m.more);
Software/so101/src/viewer.html +1
@@ -1361,6 +1361,7 @@
13611361 });
13621362 const { lobby, mayDrive, isOwner, say, serverNow, lobbyConnect, chatBox, renderTurn, button } = room;
13631363 if (relayed) import("/ui/__VERSION__/health-meter.js").then((m) => m.start(say, stream.frames));
1364 if (relayed) import("/ui/__VERSION__/clock-sync.js").then((m) => { const c = m.start(say); addEventListener("clock-sync", (e) => c.pong(e.detail)); addEventListener("clock-status", (e) => c.status(e.detail)); });
13641365 let drove = false;
13651366 function explainLock() {
13661367 const me = lobby.hello?.you, down = linkNow === "offline", st = typeof status?.source === "string" && status.source.startsWith("simulated");

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs