Public: anyone can read this; the crew can change it.
Migrated from github.com/Trevato/SO-ARM100
workbenchthis machine
The live arm never flaps to the simulation on one slow answer: keep the connection, retry, and say "reconnecting" #59
Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.
quill/arm:quill/48-the-live-arm-never-flaps-to-the-simulati → main 1 commit fork
10 files
changed +689
−114
Software/README.md +23 −10
| @@ -133,8 +133,9 @@ The API, all JSON (the full list with bodies is at the top of | ||
| 133 | 133 | (explicit `{"targets": {"3": 2048}}`), `/grip`, `/torque`, `/stop`, |
| 134 | 134 | `/estop`, `/estop/clear`, `/record/start`, `/record/stop`, |
| 135 | 135 | `/recording/delete`, `/play`; |
| 136 | `PUT /calibration/camera|handeye`; and the WebSocket `/follow` taking | |
| 137 | `{"1": ticks, ...}` frames. One thing drives the arm at a time. Every move | |
| 136 | `PUT /calibration/camera|handeye`; the WebSocket `/follow` taking | |
| 137 | `{"1": ticks, ...}` frames; and the read-only WebSocket `/watch`, which sends | |
| 138 | the `/status` body every 250 ms, one bus read shared by every watcher. One thing drives the arm at a time. Every move | |
| 138 | 139 | goes through the joint supervisor; `/stop` ends a move, a playback, a |
| 139 | 140 | recording, or a stream, and the arm holds. The stream is clamped to the |
| 140 | 141 | joint limits, eases over to the first frame, drops jumps no hand makes, is |
| @@ -217,16 +218,28 @@ destination, or can't reach it itself, the top bar and `/link` give its | ||
| 217 | 218 | reason, e.g. **the forge's egress proxy answered 502: 100.89.95.68:8101 did |
| 218 | 219 | not answer**. That means the forge's machine has no route to the Pi, which |
| 219 | 220 | is not something the app can fix. Every `/status` says which arm answered: |
| 220 | `"source": "live <host:port>"` or `"source": "simulated: <why>"`. | |
| 221 | `"source": "live <host:port>"`, `"source": "reconnecting to the arm: <why>"` | |
| 222 | or `"source": "simulated: <why>"`. | |
| 221 | 223 | |
| 222 | 224 | With `ARM_SERVER` unset, the app runs the simulated arm and says |
| 223 | **Simulated**. When it is set but the server doesn't answer `/status` within | |
| 224 | 1.5 s, the top bar says **Simulated · \<host\> unreachable (why)**, and the | |
| 225 | page runs on the simulation instead of waiting on the dead link. The app | |
| 226 | checks again every 5 s and switches back by itself. The page reloads on | |
| 227 | either switch, because the other arm has its own limits and poses. A move | |
| 228 | that was in flight to the real arm when the link dropped comes back as an | |
| 229 | error; it is not replayed on the simulation. Changes (`POST`, `PUT`, the | |
| 225 | **Simulated**. With it set, the app holds one WebSocket to the arm server's | |
| 226 | `/watch` (`src/arm-link.ts`), which pushes its status four times a second, and | |
| 227 | every viewer's `/status` is answered from that. The Pi is asked once, however | |
| 228 | many people are watching. An arm server from before `/watch` is polled over | |
| 229 | HTTP instead. One slow or dropped answer changes nothing. The link only counts | |
| 230 | as dropped when the connection closes, or brings no status for 2 s. | |
| 231 | ||
| 232 | A dropped link is never swapped for the simulation. The top bar says | |
| 233 | **Reconnecting to the arm…**, the page keeps showing the last known pose, and | |
| 234 | the app retries after 0.5, 1, 2, then every 4 s. While it reconnects, driving | |
| 235 | is paused: commands come back as 503, and a stream's frames are not passed on. | |
| 236 | The arm server's dead-man sees the silence and holds the arm. Stop and the | |
| 237 | emergency stop are still sent. When the arm answers again the page carries on | |
| 238 | without reloading. Only after 60 s down does the app run the page on the | |
| 239 | simulation, and then the top bar says **OFFLINE · simulated arm** in red, with | |
| 240 | a banner saying that this is not the real arm. The retries go on, and the page | |
| 241 | reloads onto the real arm when it comes back. A move that was in flight to the | |
| 242 | real arm when the link dropped comes back as an error; it is not replayed. Changes (`POST`, `PUT`, the | |
| 230 | 243 | stream) are only accepted from the app's own page, so another site can't |
| 231 | 244 | drive the arm through it. |
| 232 | 245 |
Software/so101/bench/fake-serve.ts +3 −2
| @@ -4,7 +4,8 @@ | ||
| 4 | 4 | * Goals become positions at once, so every move "arrives"; poses and recordings go to a temp dir. |
| 5 | 5 | * PORT, HOST and FORGE_APP_DATA override the port, the 127.0.0.1 bind and the temp dir (how the forge runs it). |
| 6 | 6 | * ARM_SERVER=ws://<host>:8101 puts a real arm's `so101 serve` behind the same page: the relay in src/relay.ts |
| 7 | * sends the arm API, /follow and the cameras there while it answers, and to the simulation when it does not. | |
| 7 | * sends the arm API, /follow and the cameras there. A dropped link says "reconnecting" and retries (src/arm-link.ts); | |
| 8 | * only after 60 s down does the page go over to the simulation, labelled offline. | |
| 8 | 9 | * The owner lists the cameras on /admin/camera-list (kept in cameras.json under the data dir); the first one comes from ARM_SERVER. |
| 9 | 10 | * Anyone may watch; only the person whose turn it is may drive (src/lobby.ts). People join with a name picked here |
| 10 | 11 | * (src/accounts.ts, kept in accounts.sqlite); the forge signs who its own signed-in people are with FORGE_APP_IDENTITY_KEY. |
| @@ -60,4 +61,4 @@ const cameras = new CameraHub({ file: join(dir, "cameras.json"), seed: remote ? | ||
| 60 | 61 | const leaders = new LeaderCalibrations({ file: join(dir, "leader-calibrations.json") }); |
| 61 | 62 | const relay = await startRelay({ host, port, simulation: `http://127.0.0.1:${simulation.port}`, remote, problem, lobby, cameras, leaders }); |
| 62 | 63 | const { mode, host: armHost } = relay.link(); |
| 63 | console.log(`arm on http://${host}:${relay.port} (${mode === "live" ? `live arm ${armHost}` : "simulated"}; data in ${dir}${local ? "; local: every visitor drives as the owner" : ""})`); | |
| 64 | console.log(`arm on http://${host}:${relay.port} (${mode === "simulated" ? "simulated" : `arm ${armHost}: ${mode}`}; data in ${dir}${local ? "; local: every visitor drives as the owner" : ""})`); |
Software/so101/src/admin-relay.test.ts +3 −2
| @@ -76,7 +76,7 @@ test("the arm server set on /admin is the one the relay uses, at once; a bad one | ||
| 76 | 76 | expect(await bad.text()).not.toContain("<script>x"); |
| 77 | 77 | await fetch(`${base}/admin`, form(owner, { action: "save", armServer: "http://127.0.0.1:1" })); |
| 78 | 78 | expect(relay!.link().host).toBe("127.0.0.1:1"); |
| 79 | expect(relay!.link().mode).toBe("simulated"); | |
| 79 | expect(relay!.link().mode).toBe("reconnecting"); // a named arm server that does not answer is retried, not swapped for the simulation | |
| 80 | 80 | await fetch(`${base}/admin`, form(owner, { action: "save", armServer: "" })); |
| 81 | 81 | expect(relay!.link().host).toBeNull(); |
| 82 | 82 | }); |
| @@ -97,7 +97,8 @@ test("reading /link never changes anything; the arm server switches when the set | ||
| 97 | 97 | const base = await boot(); |
| 98 | 98 | const owner = as("trevato", "admin"); |
| 99 | 99 | await fetch(`${base}/admin`, form(owner, { action: "save", armServer: "http://127.0.0.1:1" })); |
| 100 | const after = await (await fetch(`${base}/link`)).json(); | |
| 100 | await new Promise((r) => setTimeout(r, 200)); // the first attempt at the new address is over: what /link says now is settled | |
| 101 | const after = (await (await fetch(`${base}/link`)).json()) as { host: string }; | |
| 101 | 102 | expect(after.host).toBe("127.0.0.1:1"); |
| 102 | 103 | expect(await (await fetch(`${base}/link`)).json()).toEqual(after); |
| 103 | 104 | }); |
Software/so101/src/arm-link.test.ts added +194
| @@ -0,0 +1,194 @@ | ||
| 1 | import { afterEach, expect, test } from "bun:test"; | |
| 2 | import { createServer, type Server } from "node:http"; | |
| 3 | import WebSocket, { WebSocketServer } from "ws"; | |
| 4 | import { ArmLink, parseStatus, type LinkState, type LinkTimings } from "./arm-link.ts"; | |
| 5 | ||
| 6 | const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); | |
| 7 | /** The issue's timings at 1/100: retries after 5, 10, 20, 40 ms, offline after 600 ms (60 s), 10 s down is 100 ms. */ | |
| 8 | const FAST: LinkTimings = { backoffMs: [5, 10, 20, 40], staleMs: 60, offlineAfterMs: 600, connectTimeoutMs: 50, pollEveryMs: 10 }; | |
| 9 | const status = (n: number) => ({ joints: [{ id: 1, position: n }], driving: null }); | |
| 10 | ||
| 11 | /** | |
| 12 | * An arm server with a /watch to steer: "up" sends a status every 10 ms, "silent" keeps the connection and sends | |
| 13 | * nothing, "down" cuts every connection and refuses new ones. `watch: false` is an arm server from before /watch. | |
| 14 | */ | |
| 15 | async function fakeArm({ watch = true } = {}) { | |
| 16 | let state: "up" | "silent" | "down" = "up"; | |
| 17 | let n = 0; | |
| 18 | let skip = 0; | |
| 19 | let failNext = 0; | |
| 20 | const wss = new WebSocketServer({ noServer: true }); | |
| 21 | const http: Server = createServer((req, res) => { | |
| 22 | if (state === "down") return void req.socket.destroy(); | |
| 23 | if (req.url !== "/status") return void res.writeHead(404).end(); | |
| 24 | if (failNext > 0) return void (failNext--, res.writeHead(500).end("busy")); | |
| 25 | res.writeHead(200, { "content-type": "application/json" }).end(JSON.stringify(status(++n))); | |
| 26 | }); | |
| 27 | http.on("upgrade", (req, socket, head) => { | |
| 28 | if (state === "down" || !watch || req.url !== "/watch") return void socket.destroy(); | |
| 29 | wss.handleUpgrade(req, socket, head, () => {}); | |
| 30 | }); | |
| 31 | const pump = setInterval(() => { | |
| 32 | if (state !== "up") return; | |
| 33 | if (skip > 0) return void skip--; | |
| 34 | for (const c of wss.clients) c.send(JSON.stringify(status(++n))); | |
| 35 | }, 10); | |
| 36 | const port = await new Promise<number>((r) => http.listen(0, "127.0.0.1", () => r((http.address() as { port: number }).port))); | |
| 37 | return { | |
| 38 | port, | |
| 39 | remote: { http: `http://127.0.0.1:${port}`, ws: `ws://127.0.0.1:${port}`, host: `127.0.0.1:${port}` }, | |
| 40 | set(s: typeof state) { | |
| 41 | state = s; | |
| 42 | if (s === "down") for (const c of wss.clients) c.terminate(); | |
| 43 | }, | |
| 44 | /** Leave out the next `k` status frames, or fail the next `k` GET /status. */ | |
| 45 | drop(k: number) { | |
| 46 | skip = k; | |
| 47 | failNext = k; | |
| 48 | }, | |
| 49 | watchers: () => wss.clients.size, | |
| 50 | close() { | |
| 51 | clearInterval(pump); | |
| 52 | for (const c of wss.clients) c.terminate(); | |
| 53 | http.closeAllConnections(); | |
| 54 | http.close(); | |
| 55 | }, | |
| 56 | }; | |
| 57 | } | |
| 58 | ||
| 59 | const open = (url: string, timeoutMs: number) => | |
| 60 | new Promise<WebSocket>((resolve, reject) => { | |
| 61 | const ws = new WebSocket(url); | |
| 62 | const timer = setTimeout(() => (ws.terminate(), reject(new Error(`no answer in ${timeoutMs} ms`))), timeoutMs); | |
| 63 | ws.once("open", () => (clearTimeout(timer), resolve(ws))); | |
| 64 | ws.on("error", (err) => (clearTimeout(timer), reject(err))); | |
| 65 | }); | |
| 66 | ||
| 67 | let link: ArmLink | undefined; | |
| 68 | let arm: Awaited<ReturnType<typeof fakeArm>> | undefined; | |
| 69 | afterEach(() => { | |
| 70 | link?.close(); | |
| 71 | arm?.close(); | |
| 72 | link = arm = undefined; | |
| 73 | }); | |
| 74 | ||
| 75 | /** Watch the link's state every 2 ms, so a flap shorter than a poll is still caught. */ | |
| 76 | function record(l: ArmLink) { | |
| 77 | const seen: LinkState[] = [l.state]; | |
| 78 | const t = setInterval(() => seen.at(-1) !== l.state && seen.push(l.state), 2); | |
| 79 | return { seen, stop: () => clearInterval(t) }; | |
| 80 | } | |
| 81 | ||
| 82 | test("a status frame is a JSON object with joints; anything else is not one", () => { | |
| 83 | expect(parseStatus('{"joints":[]}')).toEqual({ joints: [] }); | |
| 84 | expect(parseStatus("[]")).toBeNull(); | |
| 85 | expect(parseStatus('{"error":"x"}')).toBeNull(); | |
| 86 | expect(parseStatus("not json")).toBeNull(); | |
| 87 | }); | |
| 88 | ||
| 89 | test("one connection: the status arrives over /watch, and one dropped answer keeps it live", async () => { | |
| 90 | arm = await fakeArm(); | |
| 91 | link = new ArmLink({ remote: arm.remote, open, timings: FAST }); | |
| 92 | await link.ready; | |
| 93 | expect(link.state).toBe("live"); | |
| 94 | expect(arm.watchers()).toBe(1); | |
| 95 | const r = record(link); | |
| 96 | arm.drop(1); // one frame never comes: 20 ms between two, well inside the 60 ms a link may go quiet | |
| 97 | await sleep(100); | |
| 98 | r.stop(); | |
| 99 | expect(r.seen).toEqual(["live"]); | |
| 100 | expect(link.problem).toBeNull(); | |
| 101 | expect((link.last!.joints as Array<{ position: number }>)[0]!.position).toBeGreaterThan(1); | |
| 102 | expect(arm.watchers()).toBe(1); // the same connection all along | |
| 103 | }); | |
| 104 | ||
| 105 | test("10 s down (at 1/100 time): reconnecting with the last pose kept, then live again on its own", async () => { | |
| 106 | arm = await fakeArm(); | |
| 107 | const changes: string[] = []; | |
| 108 | link = new ArmLink({ remote: arm.remote, open, timings: FAST, onChange: (s, p) => changes.push(`${p}→${s}`) }); | |
| 109 | await link.ready; | |
| 110 | await sleep(30); | |
| 111 | arm.set("down"); | |
| 112 | await sleep(20); | |
| 113 | expect(link.state).toBe("reconnecting"); | |
| 114 | expect(link.problem).toContain(`${arm.remote.host} unreachable`); | |
| 115 | const kept = link.last; | |
| 116 | expect(kept).not.toBeNull(); | |
| 117 | await sleep(80); // 10 s | |
| 118 | expect(link.state).toBe("reconnecting"); | |
| 119 | expect(link.last).toBe(kept); | |
| 120 | arm.set("up"); | |
| 121 | await sleep(80); // the next retry is at most 40 ms (4 s) away | |
| 122 | expect(link.state).toBe("live"); | |
| 123 | expect(link.last).not.toBe(kept); | |
| 124 | expect(changes).toEqual(["reconnecting→live", "live→reconnecting", "reconnecting→live"]); | |
| 125 | }); | |
| 126 | ||
| 127 | test("a connection that stays open but goes quiet is taken for dead and made again", async () => { | |
| 128 | arm = await fakeArm(); | |
| 129 | link = new ArmLink({ remote: arm.remote, open, timings: FAST }); | |
| 130 | await link.ready; | |
| 131 | arm.set("silent"); | |
| 132 | await sleep(90); | |
| 133 | expect(link.state).toBe("reconnecting"); | |
| 134 | expect(link.problem).toContain("no status in 60 ms"); | |
| 135 | arm.set("up"); | |
| 136 | await sleep(60); | |
| 137 | expect(link.state).toBe("live"); | |
| 138 | }); | |
| 139 | ||
| 140 | test("60 s down (at 1/100 time): offline, and the retries go on until it answers", async () => { | |
| 141 | arm = await fakeArm(); | |
| 142 | link = new ArmLink({ remote: arm.remote, open, timings: FAST }); | |
| 143 | await link.ready; | |
| 144 | arm.set("down"); | |
| 145 | await sleep(400); | |
| 146 | expect(link.state).toBe("reconnecting"); | |
| 147 | await sleep(300); | |
| 148 | expect(link.state).toBe("offline"); | |
| 149 | expect(link.problem).toContain("unreachable"); | |
| 150 | arm.set("up"); | |
| 151 | await sleep(80); | |
| 152 | expect(link.state).toBe("live"); | |
| 153 | }); | |
| 154 | ||
| 155 | test("an arm that never answered starts reconnecting, and is offline after 60 s (at 1/100 time)", async () => { | |
| 156 | arm = await fakeArm(); | |
| 157 | arm.set("down"); | |
| 158 | link = new ArmLink({ remote: arm.remote, open, timings: FAST }); | |
| 159 | await link.ready; | |
| 160 | expect(link.state).toBe("reconnecting"); | |
| 161 | expect(link.last).toBeNull(); | |
| 162 | await sleep(650); | |
| 163 | expect(link.state).toBe("offline"); | |
| 164 | }); | |
| 165 | ||
| 166 | test("an arm server from before /watch is polled over HTTP, and one failed poll keeps it live", async () => { | |
| 167 | arm = await fakeArm({ watch: false }); | |
| 168 | link = new ArmLink({ remote: arm.remote, open, timings: FAST }); | |
| 169 | await link.ready; | |
| 170 | expect(link.state).toBe("live"); | |
| 171 | expect(arm.watchers()).toBe(0); | |
| 172 | const r = record(link); | |
| 173 | arm.drop(1); | |
| 174 | await sleep(60); | |
| 175 | r.stop(); | |
| 176 | expect(r.seen).toEqual(["live"]); | |
| 177 | arm.set("down"); | |
| 178 | await sleep(120); | |
| 179 | expect(link.state).toBe("reconnecting"); | |
| 180 | arm.set("up"); | |
| 181 | await sleep(80); | |
| 182 | expect(link.state).toBe("live"); | |
| 183 | }); | |
| 184 | ||
| 185 | test("the retries back off: 5, 10, 20, then every 40 ms", async () => { | |
| 186 | let attempts = 0; | |
| 187 | const counting = (url: string, ms: number) => (attempts++, open(url, ms)); | |
| 188 | arm = await fakeArm(); | |
| 189 | arm.set("down"); | |
| 190 | link = new ArmLink({ remote: arm.remote, open: counting, timings: FAST }); | |
| 191 | await sleep(200); // 0, 5, 15, 35, 75, 115, 155, 195 ms (plus each attempt's own time) | |
| 192 | expect(attempts).toBeGreaterThanOrEqual(5); | |
| 193 | expect(attempts).toBeLessThanOrEqual(8); | |
| 194 | }); |
Software/so101/src/arm-link.ts added +193
| @@ -0,0 +1,193 @@ | ||
| 1 | /** | |
| 2 | * The relay's one connection to a real arm server, and what it knows through it. | |
| 3 | * | |
| 4 | * It holds a WebSocket to the arm server's /watch, which pushes the /status body a few times a second; every viewer's | |
| 5 | * /status is answered from the last one, so the Pi is asked once however many people watch. An arm server too old to | |
| 6 | * have /watch is polled over HTTP instead (one GET /status at a time), and the watch is tried again after any drop. | |
| 7 | * | |
| 8 | * live a status came in within `staleMs`. One late or dropped answer does not end it. | |
| 9 | * reconnecting the connection is gone: retried after 0.5, 1, 2, 4, 4… s, the last known status kept for the page. | |
| 10 | * This is where a link starts, until the first status comes in. | |
| 11 | * offline still gone `offlineAfterMs` after it went (60 s); the relay may fall back to the simulation now, | |
| 12 | * and says so. The retries go on, and the first status that comes in makes it live again. | |
| 13 | */ | |
| 14 | ||
| 15 | import type WebSocket from "ws"; | |
| 16 | import { whyNot } from "./egress.ts"; | |
| 17 | ||
| 18 | export type LinkState = "live" | "reconnecting" | "offline"; | |
| 19 | ||
| 20 | export interface LinkTimings { | |
| 21 | /** The waits between attempts after the connection goes, ms; the last one repeats. */ | |
| 22 | backoffMs: readonly number[]; | |
| 23 | /** With no status for this long, a connection that looks open is taken for dead. */ | |
| 24 | staleMs: number; | |
| 25 | /** Reconnecting this long makes the link offline. */ | |
| 26 | offlineAfterMs: number; | |
| 27 | /** How long an attempt (a WebSocket handshake, or a GET /status) may take. */ | |
| 28 | connectTimeoutMs: number; | |
| 29 | /** How often an arm server without /watch is asked for its status. */ | |
| 30 | pollEveryMs: number; | |
| 31 | } | |
| 32 | ||
| 33 | export const DEFAULT_TIMINGS: LinkTimings = { backoffMs: [500, 1000, 2000, 4000], staleMs: 2000, offlineAfterMs: 60_000, connectTimeoutMs: 1500, pollEveryMs: 250 }; | |
| 34 | ||
| 35 | export interface ArmLinkOptions { | |
| 36 | /** The arm server's HTTP and WebSocket bases, and its host:port for messages. */ | |
| 37 | remote: { http: string; ws: string; host: string }; | |
| 38 | /** Opens a WebSocket (through the egress proxy when there is one); rejects on failure or after the timeout. */ | |
| 39 | open: (url: string, timeoutMs: number) => Promise<WebSocket>; | |
| 40 | timings?: Partial<LinkTimings>; | |
| 41 | /** Called on every change of state, after it is made. */ | |
| 42 | onChange?: (state: LinkState, previous: LinkState) => void; | |
| 43 | } | |
| 44 | ||
| 45 | export type Status = Record<string, unknown>; | |
| 46 | ||
| 47 | const reason = (err: unknown) => { | |
| 48 | const e = err as { message?: string; cause?: { code?: string }; code?: string }; | |
| 49 | return e?.cause?.code ?? e?.code ?? e?.message ?? String(err); | |
| 50 | }; | |
| 51 | ||
| 52 | /** A status frame: a JSON object with a joints list. Anything else is ignored. */ | |
| 53 | export function parseStatus(text: string): Status | null { | |
| 54 | try { | |
| 55 | const body = JSON.parse(text) as unknown; | |
| 56 | return body && typeof body === "object" && !Array.isArray(body) && Array.isArray((body as Status).joints) ? (body as Status) : null; | |
| 57 | } catch { | |
| 58 | return null; | |
| 59 | } | |
| 60 | } | |
| 61 | ||
| 62 | export class ArmLink { | |
| 63 | state: LinkState = "reconnecting"; | |
| 64 | /** Why the link is not live; null while it is. */ | |
| 65 | problem: string | null; | |
| 66 | /** The last status the arm server sent, kept through a drop. */ | |
| 67 | last: Status | null = null; | |
| 68 | /** When the last status came in, and when the link last stopped being live (or started). */ | |
| 69 | lastAt = 0; | |
| 70 | downSince = Date.now(); | |
| 71 | readonly ready: Promise<void>; | |
| 72 | private readonly t: LinkTimings; | |
| 73 | private attempt = 0; | |
| 74 | private polling = false; | |
| 75 | private closed = false; | |
| 76 | private ws: WebSocket | null = null; | |
| 77 | private retry: ReturnType<typeof setTimeout> | undefined; | |
| 78 | private stale: ReturnType<typeof setTimeout> | undefined; | |
| 79 | private offline: ReturnType<typeof setTimeout> | undefined; | |
| 80 | private settle!: () => void; | |
| 81 | private readonly o: ArmLinkOptions; | |
| 82 | ||
| 83 | constructor(o: ArmLinkOptions) { | |
| 84 | this.o = o; | |
| 85 | this.t = { ...DEFAULT_TIMINGS, ...o.timings }; | |
| 86 | this.problem = `connecting to ${o.remote.host}`; | |
| 87 | this.ready = new Promise((resolve) => (this.settle = resolve)); | |
| 88 | this.offline = setTimeout(() => this.goOffline(), this.t.offlineAfterMs); | |
| 89 | void this.connect(); | |
| 90 | } | |
| 91 | ||
| 92 | close() { | |
| 93 | this.closed = true; | |
| 94 | clearTimeout(this.retry); | |
| 95 | clearTimeout(this.stale); | |
| 96 | clearTimeout(this.offline); | |
| 97 | this.ws?.terminate(); | |
| 98 | this.ws = null; | |
| 99 | this.settle(); | |
| 100 | } | |
| 101 | ||
| 102 | private async connect() { | |
| 103 | if (this.closed) return; | |
| 104 | if (this.polling) return this.poll(); | |
| 105 | let ws: WebSocket; | |
| 106 | try { | |
| 107 | ws = await this.o.open(`${this.o.remote.ws}/watch`, this.t.connectTimeoutMs); | |
| 108 | } catch { | |
| 109 | // Why the watch failed is not something Bun's client says; a plain GET does, and finds an arm server without /watch. | |
| 110 | return this.poll(); | |
| 111 | } | |
| 112 | if (this.closed) return void ws.terminate(); | |
| 113 | this.ws = ws; | |
| 114 | let why = "the connection closed"; | |
| 115 | const wait = () => { | |
| 116 | clearTimeout(this.stale); | |
| 117 | this.stale = setTimeout(() => ((why = `no status in ${this.t.staleMs} ms`), ws.terminate()), this.t.staleMs); | |
| 118 | }; | |
| 119 | wait(); | |
| 120 | ws.on("message", (data) => { | |
| 121 | const body = parseStatus(String(data)); | |
| 122 | if (!body) return; | |
| 123 | wait(); | |
| 124 | this.seen(body); | |
| 125 | }); | |
| 126 | ws.on("error", (err) => ((why = reason(err)), ws.terminate())); | |
| 127 | ws.on("close", () => { | |
| 128 | if (this.ws !== ws) return; | |
| 129 | this.ws = null; | |
| 130 | clearTimeout(this.stale); | |
| 131 | this.lost(why); | |
| 132 | }); | |
| 133 | } | |
| 134 | ||
| 135 | private async poll() { | |
| 136 | try { | |
| 137 | const r = await fetch(`${this.o.remote.http}/status`, { signal: AbortSignal.timeout(this.t.connectTimeoutMs), cache: "no-store" }); | |
| 138 | if (!r.ok) throw new Error(await whyNot(r, "/status")); | |
| 139 | const body = parseStatus(await r.text()); | |
| 140 | if (!body) throw new Error("/status did not answer an arm's status"); | |
| 141 | if (this.closed) return; | |
| 142 | this.polling = true; | |
| 143 | this.seen(body); | |
| 144 | this.after(this.t.pollEveryMs); | |
| 145 | } catch (err) { | |
| 146 | if (this.closed) return; | |
| 147 | const why = (err as Error)?.name === "TimeoutError" ? `no answer in ${this.t.connectTimeoutMs} ms` : reason(err); | |
| 148 | // One dropped answer is not a dropped link. | |
| 149 | if (this.polling && this.state === "live" && Date.now() - this.lastAt < this.t.staleMs) return this.after(this.t.pollEveryMs); | |
| 150 | this.lost(why); | |
| 151 | } | |
| 152 | } | |
| 153 | ||
| 154 | private after(ms: number) { | |
| 155 | clearTimeout(this.retry); | |
| 156 | if (!this.closed) this.retry = setTimeout(() => void this.connect(), ms); | |
| 157 | } | |
| 158 | ||
| 159 | private seen(body: Status) { | |
| 160 | this.last = body; | |
| 161 | this.lastAt = Date.now(); | |
| 162 | this.attempt = 0; | |
| 163 | clearTimeout(this.offline); | |
| 164 | this.problem = null; | |
| 165 | this.change("live"); | |
| 166 | this.settle(); | |
| 167 | } | |
| 168 | ||
| 169 | private lost(why: string) { | |
| 170 | if (this.closed) return; | |
| 171 | this.problem = `${this.o.remote.host} unreachable (${why})`; | |
| 172 | this.polling = false; // the next attempt tries the watch again | |
| 173 | if (this.state === "live") { | |
| 174 | this.downSince = Date.now(); | |
| 175 | this.offline = setTimeout(() => this.goOffline(), this.t.offlineAfterMs); | |
| 176 | this.change("reconnecting"); | |
| 177 | } | |
| 178 | this.settle(); | |
| 179 | const waits = this.t.backoffMs; | |
| 180 | this.after(waits[Math.min(this.attempt++, waits.length - 1)] ?? 1000); | |
| 181 | } | |
| 182 | ||
| 183 | private goOffline() { | |
| 184 | if (this.state === "reconnecting" && !this.closed) this.change("offline"); | |
| 185 | } | |
| 186 | ||
| 187 | private change(state: LinkState) { | |
| 188 | const previous = this.state; | |
| 189 | if (previous === state) return; | |
| 190 | this.state = state; | |
| 191 | this.o.onChange?.(state, previous); | |
| 192 | } | |
| 193 | } |
Software/so101/src/relay.test.ts +96 −19
| @@ -5,7 +5,7 @@ import { connect, createServer as createTcpServer, type Server as TcpServer } fr | ||
| 5 | 5 | import { tmpdir } from "node:os"; |
| 6 | 6 | import { join } from "node:path"; |
| 7 | 7 | import { gunzipSync } from "node:zlib"; |
| 8 | import WebSocket from "ws"; | |
| 8 | import WebSocket, { WebSocketServer } from "ws"; | |
| 9 | 9 | import { FeetechBus } from "./bus.ts"; |
| 10 | 10 | import { FakeBus } from "./fake-bus.ts"; |
| 11 | 11 | import { OFFLINE_FRAME } from "./offline-frame.ts"; |
| @@ -20,7 +20,7 @@ const listening = (server: Server | TcpServer) => new Promise<number>((r) => ser | ||
| 20 | 20 | /** An arm server on a fake bus; `ids` tells the simulation (six joints) and the "real arm" (three) apart. */ |
| 21 | 21 | async function arm(ids: number[]): Promise<RunningServer> { |
| 22 | 22 | const fake = new FakeBus(ids); |
| 23 | return startServer(new FeetechBus(fake), { host: "127.0.0.1", port: 0, ids, posesFile: join(mkdtempSync(join(tmpdir(), "so101-relay-")), "poses.json"), watchdogMs: 60 }); | |
| 23 | return startServer(new FeetechBus(fake), { host: "127.0.0.1", port: 0, ids, posesFile: join(mkdtempSync(join(tmpdir(), "so101-relay-")), "poses.json"), watchdogMs: 60, watchEveryMs: 10 }); | |
| 24 | 24 | } |
| 25 | 25 | const remoteAt = (port: number, camera?: number): Remote => parseRemote({ ARM_SERVER: `ws://127.0.0.1:${port}`, ...(camera ? { ARM_CAMERA_1: `http://127.0.0.1:${camera}/stream` } : {}) })!; |
| 26 | 26 | const JPEG = Buffer.from([0xff, 0xd8, 0x4a, 0x50, 0x45, 0x47, 0xff, 0xd9]); |
| @@ -48,6 +48,9 @@ const joints = async (base: string) => ((await (await fetch(`${base}/status`)).j | ||
| 48 | 48 | const source = async (base: string) => ((await (await fetch(`${base}/status`)).json()) as { source: string }).source; |
| 49 | 49 | const link = async (base: string) => (await (await fetch(`${base}/link`)).json()) as { mode: string; host: string | null; problem: string | null }; |
| 50 | 50 | |
| 51 | /** The link's timings at 1/100: retries after 5–40 ms, offline after 600 ms (60 s). */ | |
| 52 | const FAST = { backoffMs: [5, 10, 20, 40], staleMs: 60, offlineAfterMs: 600, connectTimeoutMs: 50, pollEveryMs: 10 }; | |
| 53 | ||
| 51 | 54 | let simulation: RunningServer; |
| 52 | 55 | let simBase: string; |
| 53 | 56 | beforeAll(async () => { |
| @@ -198,19 +201,25 @@ describe("relay", () => { | ||
| 198 | 201 | expect(refused.status).toBe(403); |
| 199 | 202 | }); |
| 200 | 203 | |
| 201 | test("unreachable: the simulation answers within 3 s and the page is told why", async () => { | |
| 204 | test("unreachable from the start: reconnecting, never the simulation, until 60 s pass (at 1/100 time)", async () => { | |
| 202 | 205 | hang = createTcpServer(() => {}); // accepts and never answers |
| 203 | 206 | const port = await listening(hang); |
| 204 | 207 | const started = Date.now(); |
| 205 | relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(port) }); | |
| 208 | relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(port), link: FAST }); | |
| 206 | 209 | const base = `http://127.0.0.1:${relay.port}`; |
| 207 | expect(await joints(base)).toBe(6); | |
| 208 | const l = await link(base); | |
| 209 | 210 | expect(Date.now() - started).toBeLessThan(3000); |
| 210 | expect(l.mode).toBe("simulated"); | |
| 211 | const l = await link(base); | |
| 212 | expect(l.mode).toBe("reconnecting"); | |
| 211 | 213 | expect(l.host).toBe(`127.0.0.1:${port}`); |
| 212 | 214 | expect(l.problem).toContain("unreachable"); |
| 213 | expect(await source(base)).toBe(`simulated: ${l.problem}`); | |
| 215 | const s = await fetch(`${base}/status`); | |
| 216 | expect(s.status).toBe(503); | |
| 217 | expect(await s.json()).toMatchObject({ reconnecting: true, source: expect.stringContaining(`reconnecting to the arm: 127.0.0.1:${port}`) }); | |
| 218 | expect((await fetch(`${base}/move`, { method: "POST", body: '{"targets":{"1":2000}}' })).status).toBe(503); | |
| 219 | await sleep(650); | |
| 220 | expect((await link(base)).mode).toBe("offline"); | |
| 221 | expect(await joints(base)).toBe(6); | |
| 222 | expect(await source(base)).toStartWith("simulated: offline, "); | |
| 214 | 223 | }); |
| 215 | 224 | |
| 216 | 225 | test("the forge's egress proxy cannot reach the arm: its reason is the one the page and /status give", async () => { |
| @@ -218,9 +227,9 @@ describe("relay", () => { | ||
| 218 | 227 | relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(await listening(camera)) }); |
| 219 | 228 | const base = `http://127.0.0.1:${relay.port}`; |
| 220 | 229 | const l = await link(base); |
| 230 | expect(l.mode).toBe("reconnecting"); | |
| 221 | 231 | expect(l.problem).toContain("the forge's egress proxy answered 502: 100.89.95.68:8101 did not answer"); |
| 222 | expect(await source(base)).toBe(`simulated: ${l.problem}`); | |
| 223 | expect(await joints(base)).toBe(6); | |
| 232 | expect(await source(base)).toEndWith(l.problem!); | |
| 224 | 233 | }); |
| 225 | 234 | |
| 226 | 235 | test("inside a cell, /follow reaches the arm server through the egress proxy", async () => { |
| @@ -253,7 +262,8 @@ describe("relay", () => { | ||
| 253 | 262 | const ws = new WebSocket(`ws://127.0.0.1:${relay.port}/follow`); |
| 254 | 263 | await new Promise((r, j) => (ws.once("open", r), ws.once("error", j))); |
| 255 | 264 | await sleep(50); |
| 256 | expect(tunnelled).toEqual([`127.0.0.1:${remote.port}`]); | |
| 265 | // Both connections to the arm go through it: the link's /watch, and the stream. | |
| 266 | expect(tunnelled).toEqual([`127.0.0.1:${remote.port}`, `127.0.0.1:${remote.port}`]); | |
| 257 | 267 | expect(((await (await fetch(`http://127.0.0.1:${remote.port}/status`)).json()) as { driving: string | null }).driving).toBe("stream"); |
| 258 | 268 | ws.close(); |
| 259 | 269 | } finally { |
| @@ -264,20 +274,87 @@ describe("relay", () => { | ||
| 264 | 274 | } |
| 265 | 275 | }); |
| 266 | 276 | |
| 267 | test("an arm server that goes away mid-session falls back on the next request, /follow included", async () => { | |
| 277 | test("one connection to the arm server: every viewer's /status comes from it, and one slow answer keeps it live", async () => { | |
| 268 | 278 | remote = await arm([1, 2, 6]); |
| 269 | relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(remote.port), probeEveryMs: 60_000 }); | |
| 279 | relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(remote.port), link: FAST }); | |
| 280 | const base = `http://127.0.0.1:${relay.port}`; | |
| 281 | const asked: number[] = []; | |
| 282 | for (let i = 0; i < 20; i++) { | |
| 283 | asked.push(await joints(base)); | |
| 284 | await sleep(5); | |
| 285 | } | |
| 286 | expect(asked.every((n) => n === 3)).toBe(true); | |
| 287 | expect((await link(base)).mode).toBe("live"); | |
| 288 | }); | |
| 289 | ||
| 290 | test("an arm server that goes away: reconnecting with the last pose, driving suspended, then live again on the same arm", async () => { | |
| 291 | remote = await arm([1, 2, 6]); | |
| 292 | const port = remote.port; | |
| 293 | // A stopped server closes the watch at once; the patience for a quiet one is long here, so a move's busy bus is no drop. | |
| 294 | relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(port), link: { ...FAST, staleMs: 1000 } }); | |
| 270 | 295 | const base = `http://127.0.0.1:${relay.port}`; |
| 271 | 296 | expect(await joints(base)).toBe(3); |
| 272 | 297 | await shut(remote); |
| 273 | 298 | remote = undefined; |
| 274 | const started = Date.now(); | |
| 275 | expect(await joints(base)).toBe(6); | |
| 276 | expect(Date.now() - started).toBeLessThan(3000); | |
| 277 | expect((await link(base)).mode).toBe("simulated"); | |
| 299 | await sleep(80); | |
| 300 | expect((await link(base)).mode).toBe("reconnecting"); | |
| 301 | const s = await fetch(`${base}/status`); | |
| 302 | expect(s.status).toBe(200); | |
| 303 | const body = (await s.json()) as { joints: unknown[]; reconnecting: boolean; source: string }; | |
| 304 | expect(body.joints.length).toBe(3); // the last known pose, not the simulation's six joints | |
| 305 | expect(body.reconnecting).toBe(true); | |
| 306 | expect(body.source).toStartWith(`reconnecting to the arm: 127.0.0.1:${port}`); | |
| 307 | const move = await fetch(`${base}/move`, { method: "POST", body: '{"targets":{"1":2000}}' }); | |
| 308 | expect(move.status).toBe(503); | |
| 309 | expect(((await move.json()) as { error: string }).error).toContain("reconnecting to the arm"); | |
| 278 | 310 | const ws = new WebSocket(`ws://127.0.0.1:${relay.port}/follow`); |
| 279 | await new Promise((r, j) => (ws.once("open", r), ws.once("error", j))); | |
| 280 | ws.close(); | |
| 311 | const closed = await new Promise<{ code: number; why: string }>((r) => ws.on("close", (code, why) => r({ code, why: String(why) }))); | |
| 312 | expect(closed).toEqual({ code: 1013, why: "reconnecting to the arm…" }); | |
| 313 | ||
| 314 | remote = await startServer(new FeetechBus(new FakeBus([1, 2, 6])), { host: "127.0.0.1", port, ids: [1, 2, 6], posesFile: join(mkdtempSync(join(tmpdir(), "so101-relay-")), "poses.json"), watchEveryMs: 10 }); | |
| 315 | await sleep(120); | |
| 316 | expect((await link(base)).mode).toBe("live"); | |
| 317 | expect(await source(base)).toBe(`live 127.0.0.1:${port}`); | |
| 318 | expect((await fetch(`${base}/move`, { method: "POST", body: '{"targets":{"1":2000}}' })).status).toBe(200); | |
| 319 | }); | |
| 320 | ||
| 321 | test("a stream on the real arm goes quiet while the link reconnects, and flows again once it is back", async () => { | |
| 322 | // An arm server whose /watch can be silenced while its /follow stays up: what the relay passes on is counted. | |
| 323 | let watching = true; | |
| 324 | const got: string[] = []; | |
| 325 | const wss = new WebSocketServer({ noServer: true }); | |
| 326 | camera = createServer((_req, res) => res.writeHead(404).end()); | |
| 327 | camera.on("upgrade", (req, socket, head) => | |
| 328 | wss.handleUpgrade(req, socket, head, (ws) => { | |
| 329 | if (req.url === "/follow") ws.on("message", (d) => got.push(String(d))); | |
| 330 | }), | |
| 331 | ); | |
| 332 | const pump = setInterval(() => watching && wss.clients.forEach((c) => c.send('{"joints":[]}')), 10); | |
| 333 | try { | |
| 334 | relay = await startRelay({ port: 0, simulation: simBase, remote: remoteAt(await listening(camera)), link: FAST }); | |
| 335 | const ws = new WebSocket(`ws://127.0.0.1:${relay.port}/follow`); | |
| 336 | await new Promise((r, j) => (ws.once("open", r), ws.once("error", j))); | |
| 337 | ws.send('{"1": 2050}'); | |
| 338 | await sleep(30); | |
| 339 | expect(got).toEqual(['{"1": 2050}']); | |
| 340 | watching = false; | |
| 341 | await sleep(100); | |
| 342 | expect(relay.link().mode).toBe("reconnecting"); | |
| 343 | ws.send('{"1": 2060}'); | |
| 344 | await sleep(30); | |
| 345 | expect(got).toEqual(['{"1": 2050}']); // the arm server's dead-man sees silence, and holds | |
| 346 | expect(ws.readyState).toBe(WebSocket.OPEN); | |
| 347 | watching = true; | |
| 348 | await sleep(80); | |
| 349 | expect(relay.link().mode).toBe("live"); | |
| 350 | ws.send('{"1": 2070}'); | |
| 351 | await sleep(30); | |
| 352 | expect(got).toEqual(['{"1": 2050}', '{"1": 2070}']); | |
| 353 | ws.close(); | |
| 354 | } finally { | |
| 355 | clearInterval(pump); | |
| 356 | for (const c of wss.clients) c.terminate(); | |
| 357 | } | |
| 281 | 358 | }); |
| 282 | 359 | |
| 283 | 360 | test("a bad setting is reported and the arm is simulated", async () => { |
Software/so101/src/relay.ts +85 −69
| @@ -2,8 +2,11 @@ | ||
| 2 | 2 | * The hosted app's front door: one origin for the browser, whichever arm stands behind it. |
| 3 | 3 | * |
| 4 | 4 | * With ARM_SERVER unset, everything goes to the simulated arm. With it set (ws://<host>:8101, http://…, https://…), |
| 5 | * the arm API, the WebSocket /follow and the cameras go to that `so101 serve`, as long as it answers; when it does | |
| 6 | * not, the relay says so on /link and answers from the simulation instead, so the page never hangs on a dead link. | |
| 5 | * the arm API, the WebSocket /follow and the cameras go to that `so101 serve`. The relay holds one connection to it | |
| 6 | * (src/arm-link.ts) and answers every /status from it. When that connection drops, the relay never jumps to the | |
| 7 | * simulation: it says "reconnecting" and keeps showing the last known pose while it retries. Driving is suspended | |
| 8 | * (commands are refused and stream frames are not passed on, so the arm server's dead-man holds the arm); a stop | |
| 9 | * still goes through. Only after 60 s down does the page go over to the simulation, labelled offline. | |
| 7 | 10 | * The page itself, the 3D model and the vendored scripts always come from here. |
| 8 | 11 | * Inside a forge cell the only way out is the egress proxy (src/egress.ts): fetch finds it on its own, the WebSocket |
| 9 | 12 | * to the arm server is told it, and when the proxy refuses or cannot dial, its reason is the one /link and /status give. |
| @@ -22,8 +25,10 @@ | ||
| 22 | 25 | * POST /join {"name": "…"}: become someone here, no forge account needed (src/accounts.ts); the answer |
| 23 | 26 | * sets the HttpOnly session cookie and names you: {"you": {"id", "name", "owner"}} |
| 24 | 27 | * GET /admin, POST /admin the owner's page: pause, end a turn, take control, the settings, the log |
| 25 | * GET /link {"mode": "live"|"simulated", "host": "<host:port>"|null, "problem": "…"|null} | |
| 26 | * GET /status the arm's own /status, and which arm said it: "source": "live <host:port>" or "simulated: <why>" | |
| 28 | * GET /link {"mode": "live"|"reconnecting"|"offline"|"simulated", "host": "<host:port>"|null, "problem": "…"|null} | |
| 29 | * GET /status the arm's own /status, and which arm said it: "source": "live <host:port>", "reconnecting to the arm: | |
| 30 | * <why>" (the last known status, with "reconnecting": true and "seenAt"; 503 before | |
| 31 | * there is one), or "simulated: <why>" ("simulated: offline, <why>" after 60 s down) | |
| 27 | 32 | * GET /cameras the listed cameras: [{"id", "name", "online"}] |
| 28 | 33 | * GET /camera/<id>/<stream|state|snapshot> one listed camera, one upstream connection shared by all its viewers (src/cameras.ts, src/camera-feed.ts); |
| 29 | 34 | * the "camera offline" picture when its source is down |
| @@ -39,6 +44,7 @@ import type { Duplex } from "node:stream"; | ||
| 39 | 44 | import { Readable } from "node:stream"; |
| 40 | 45 | import { pipeline } from "node:stream/promises"; |
| 41 | 46 | import WebSocket, { WebSocketServer } from "ws"; |
| 47 | import { ArmLink, type LinkTimings } from "./arm-link.ts"; | |
| 42 | 48 | import { acceptsGzip, GzipCache, worthCompressing } from "./compress.ts"; |
| 43 | 49 | import { AccountError, sessionCookie, visitor } from "./accounts.ts"; |
| 44 | 50 | import { AdminError, changeFromForm } from "./admin.ts"; |
| @@ -64,7 +70,8 @@ export interface Remote { | ||
| 64 | 70 | } |
| 65 | 71 | |
| 66 | 72 | export interface Link { |
| 67 | mode: "live" | "simulated"; | |
| 73 | /** "simulated" only when no arm server is named; "offline" when one is, and has been gone 60 s. */ | |
| 74 | mode: "live" | "reconnecting" | "offline" | "simulated"; | |
| 68 | 75 | /** The arm server the setting names, live or not. */ |
| 69 | 76 | host: string | null; |
| 70 | 77 | /** Why a configured arm server is not in use right now. */ |
| @@ -81,11 +88,11 @@ export interface RelayOptions { | ||
| 81 | 88 | cameras?: CameraHub; |
| 82 | 89 | /** A setting that could not be used (shown on /link while simulating). */ |
| 83 | 90 | problem?: string | null; |
| 84 | /** How long the arm server has to answer /status before it counts as unreachable. */ | |
| 85 | probeTimeoutMs?: number; | |
| 86 | /** How often the arm server is checked, whichever way the last check went. */ | |
| 87 | probeEveryMs?: number; | |
| 88 | /** How long a read through to the arm server may wait for its answer before the simulation answers instead. */ | |
| 91 | /** The link's retries and its patience (src/arm-link.ts). */ | |
| 92 | link?: Partial<LinkTimings>; | |
| 93 | /** How often the admin's arm server setting is looked at again. */ | |
| 94 | followEveryMs?: number; | |
| 95 | /** How long a read through to the arm server may wait for its answer. */ | |
| 89 | 96 | readTimeoutMs?: number; |
| 90 | 97 | /** Who may command the arm; without one, anyone who reaches the relay may (a run on one's own machine). */ |
| 91 | 98 | lobby?: Lobby | null; |
| @@ -120,6 +127,9 @@ const LEADER = "/leader/calibration"; | ||
| 120 | 127 | const ADMIN_ONLY = new Set(["POST /estop/clear", "PUT /settings"]); |
| 121 | 128 | /** The driver's, and any admin's at any time. */ |
| 122 | 129 | const ADMIN_TOO = new Set(["POST /stop", "POST /estop"]); |
| 130 | /** What still goes to the arm server while the link is reconnecting: stopping it. */ | |
| 131 | const STOPS = new Set(["POST /stop", "POST /estop"]); | |
| 132 | const RECONNECTING = "reconnecting to the arm"; | |
| 123 | 133 | /** The close code that tells the arm server its driver was lost, not done: it parks the arm. */ |
| 124 | 134 | export const LOST_CLOSE = 4001; |
| 125 | 135 | const MAX_JOIN_BODY = 1024; |
| @@ -189,8 +199,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 189 | 199 | let named = ""; |
| 190 | 200 | const simulation = options.simulation.replace(/\/+$/, ""); |
| 191 | 201 | const simulationWs = simulation.replace(/^http/, "ws"); |
| 192 | const probeTimeoutMs = options.probeTimeoutMs ?? 1500; | |
| 193 | const probeEveryMs = options.probeEveryMs ?? 5000; | |
| 202 | const followEveryMs = options.followEveryMs ?? 5000; | |
| 194 | 203 | const readTimeoutMs = options.readTimeoutMs ?? 2500; |
| 195 | 204 | const lobby = options.lobby ?? null; |
| 196 | 205 | const heartbeatMs = options.heartbeatMs ?? 1000; |
| @@ -198,51 +207,48 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 198 | 207 | const leaders = options.leaders ?? new LeaderCalibrations(); |
| 199 | 208 | |
| 200 | 209 | // ------------------------------------------------------------------ the link |
| 201 | let live = false; | |
| 210 | let armLink: ArmLink | null = null; | |
| 202 | 211 | let problem: string | null = options.problem ?? null; |
| 203 | /** Requests in flight to the arm server: cut when it is found gone, so nothing waits on a dead link. */ | |
| 212 | /** Requests in flight to the arm server: cut when the link drops, so nothing waits on a dead one. */ | |
| 204 | 213 | const inFlight = new Set<AbortController>(); |
| 205 | const up = () => { | |
| 206 | if (!live && remote) console.log(`arm server ${remote.host}: live`); | |
| 207 | live = true; | |
| 208 | problem = null; | |
| 209 | }; | |
| 210 | const down = (why: string) => { | |
| 214 | const connect = () => { | |
| 215 | armLink?.close(); | |
| 216 | armLink = null; | |
| 211 | 217 | if (!remote) return; |
| 212 | if (live || problem === null) console.error(`arm server ${remote.host}: ${why}; simulating`); | |
| 213 | live = false; | |
| 214 | problem = `${remote.host} unreachable (${why})`; | |
| 215 | for (const c of inFlight) c.abort(new Error(why)); | |
| 218 | const host = remote.host; | |
| 219 | armLink = new ArmLink({ | |
| 220 | remote, | |
| 221 | open: (url, timeoutMs) => open(url, timeoutMs), | |
| 222 | timings: options.link, | |
| 223 | onChange: (state, previous) => { | |
| 224 | if (state === "live") console.log(`arm server ${host}: live`); | |
| 225 | else console.error(`arm server ${host}: ${state === "offline" ? "offline; simulating" : "reconnecting"} (${armLink?.problem})`); | |
| 226 | if (previous === "live") for (const c of inFlight) c.abort(new Error(armLink?.problem ?? "the link dropped")); | |
| 227 | }, | |
| 228 | }); | |
| 216 | 229 | }; |
| 217 | let probing: Promise<void> | null = null; | |
| 218 | const probe = () => | |
| 219 | (probing ??= (async () => { | |
| 220 | try { | |
| 221 | const r = await fetch(`${remote!.http}/status`, { signal: AbortSignal.timeout(probeTimeoutMs), cache: "no-store" }); | |
| 222 | if (r.ok) { | |
| 223 | await r.body?.cancel(); | |
| 224 | up(); | |
| 225 | } else down(await whyNot(r, "/status")); | |
| 226 | } catch (err) { | |
| 227 | down((err as Error)?.name === "TimeoutError" ? `no answer in ${probeTimeoutMs} ms` : reason(err)); | |
| 228 | } finally { | |
| 229 | probing = null; | |
| 230 | } | |
| 231 | })()); | |
| 232 | /** Follow the admin's arm server setting: a change cuts what is in flight and probes the new address at once. */ | |
| 230 | /** Follow the admin's arm server setting: a change cuts what is in flight and connects to the new address at once. */ | |
| 233 | 231 | const follow = () => { |
| 234 | 232 | if (!lobby) return; |
| 235 | 233 | const address = lobby.admin.settings.armServer; |
| 236 | 234 | if (address === named) return; |
| 237 | 235 | named = address; |
| 238 | 236 | remote = address ? parseRemote({ ARM_SERVER: address }) : environment; |
| 239 | live = false; | |
| 240 | 237 | problem = null; |
| 241 | 238 | for (const c of inFlight) c.abort(new Error("the arm server was changed")); |
| 242 | if (remote) void probe(); | |
| 239 | connect(); | |
| 240 | }; | |
| 241 | const mode = (): Link["mode"] => (remote && armLink ? armLink.state : "simulated"); | |
| 242 | /** Whether the arm server, not the simulation, is the arm right now: live, or only briefly gone. */ | |
| 243 | const onArm = () => mode() === "live" || mode() === "reconnecting"; | |
| 244 | const link = (): Link => ({ mode: mode(), host: remote?.host ?? null, problem: armLink ? armLink.problem : problem }); | |
| 245 | const source = () => { | |
| 246 | const m = mode(); | |
| 247 | if (m === "live") return `live ${remote!.host}`; | |
| 248 | if (m === "reconnecting") return `${RECONNECTING}: ${armLink!.problem}`; | |
| 249 | if (m === "offline") return `simulated: offline, ${armLink!.problem}`; | |
| 250 | return `simulated: ${problem ?? "ARM_SERVER is not set"}`; | |
| 243 | 251 | }; |
| 244 | const link = (): Link => ({ mode: live ? "live" : "simulated", host: remote?.host ?? null, problem }); | |
| 245 | const source = () => (live && remote ? `live ${remote.host}` : `simulated: ${problem ?? "ARM_SERVER is not set"}`); | |
| 246 | 252 | |
| 247 | 253 | // ------------------------------------------------------------------ HTTP |
| 248 | 254 | const json = (res: ServerResponse, status: number, body: unknown) => { |
| @@ -303,18 +309,18 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 303 | 309 | } |
| 304 | 310 | }; |
| 305 | 311 | |
| 306 | /** The arm's /status, from the live arm while it answers and the simulation otherwise, saying which it is. */ | |
| 312 | /** | |
| 313 | * The arm's /status, saying which arm it is: from the link while it is live, the last one it had while reconnecting, | |
| 314 | * and from the simulation otherwise. | |
| 315 | */ | |
| 307 | 316 | const status = async (res: ServerResponse, path: string) => { |
| 308 | let answer: Response | undefined; | |
| 309 | if (live && remote) { | |
| 310 | try { | |
| 311 | answer = await fetch(remote.http + path, { signal: AbortSignal.timeout(readTimeoutMs), cache: "no-store" }); | |
| 312 | if (!answer.ok) down(await whyNot(answer, "/status")); | |
| 313 | } catch (err) { | |
| 314 | down((err as Error)?.name === "TimeoutError" ? `no answer in ${readTimeoutMs} ms` : reason(err)); | |
| 315 | } | |
| 317 | if (onArm()) { | |
| 318 | const last = armLink!.last; | |
| 319 | const reconnecting = mode() === "reconnecting"; | |
| 320 | if (!last) return json(res, 503, { error: `${RECONNECTING}…`, source: source(), reconnecting }); | |
| 321 | return json(res, 200, { ...last, source: source(), reconnecting, ...(reconnecting ? { seenAt: new Date(armLink!.lastAt).toISOString() } : {}) }); | |
| 316 | 322 | } |
| 317 | if (!live || !answer?.ok) answer = await fetch(simulation + path, { cache: "no-store" }); | |
| 323 | const answer = await fetch(simulation + path, { cache: "no-store" }); | |
| 318 | 324 | const body = (await answer.json()) as Record<string, unknown>; |
| 319 | 325 | return json(res, answer.status, { ...body, source: source() }); |
| 320 | 326 | }; |
| @@ -533,14 +539,15 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 533 | 539 | // Who stopped or cleared the arm is the relay's to say, never the page's; every other command is held to the admin's speed cap. |
| 534 | 540 | const raw = reading || route === "POST /estop" || route === "POST /estop/clear" ? undefined : await readBody(req); |
| 535 | 541 | const body = reading ? undefined : !raw ? Buffer.from(JSON.stringify({ by })) : lobby ? capBody(url.pathname, raw, lobby.maxSpeedPercent) : raw; |
| 536 | if (LOCAL.test(url.pathname) || !live || !remote) return await pass(res, await ask(simulation + path, req, body, null, false)); | |
| 542 | if (LOCAL.test(url.pathname) || !onArm() || !remote) return await pass(res, await ask(simulation + path, req, body, null, false)); | |
| 543 | if (mode() === "reconnecting" && !reading && !STOPS.has(route)) { | |
| 544 | return json(res, 503, { error: `${RECONNECTING} at ${remote.host}…; driving resumes when it answers`, reconnecting: true }); | |
| 545 | } | |
| 537 | 546 | try { |
| 538 | // A read must answer fast or fall back; a move or a playback answers when the arm arrives, and is only cut when the link is. | |
| 547 | // A read must answer fast; a move or a playback answers when the arm arrives, and is only cut when the link drops. | |
| 539 | 548 | return await pass(res, await ask(remote.http + path, req, body, reading ? readTimeoutMs : null, true)); |
| 540 | 549 | } catch (err) { |
| 541 | down(reason(err)); | |
| 542 | if (!reading) return json(res, 502, { error: `the arm server at ${remote.host} did not answer (${reason(err)}); the page is now on the simulation` }); | |
| 543 | return await pass(res, await ask(simulation + path, req, body, null, false)); | |
| 550 | return json(res, mode() === "reconnecting" ? 503 : 502, { error: `the arm server at ${remote.host} did not answer (${reason(err)})`, reconnecting: mode() === "reconnecting" }); | |
| 544 | 551 | } |
| 545 | 552 | } catch (err) { |
| 546 | 553 | if (!res.headersSent) json(res, 502, { error: reason(err) }); |
| @@ -564,17 +571,21 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 564 | 571 | clearTimeout(timer); |
| 565 | 572 | resolve(ws); |
| 566 | 573 | }); |
| 567 | ws.once("error", (err) => { | |
| 574 | ws.on("error", (err) => { | |
| 568 | 575 | clearTimeout(timer); |
| 569 | reject(err); | |
| 576 | reject(err); // the first one settles it; a later one (Bun's client can send two) must still find a listener | |
| 570 | 577 | }); |
| 571 | 578 | }); |
| 572 | 579 | /** Close codes a server may send on; the rest (1005, 1006, 1015) only describe a connection that died. */ |
| 573 | 580 | const sendable = (code: number) => (code >= 1000 && code <= 1003) || (code >= 1007 && code <= 1014) || (code >= 3000 && code <= 4999); |
| 574 | 581 | /** Each open /follow stream and whose it is, so a turn that ends can close it. */ |
| 575 | 582 | const streams = new Map<WebSocket, string>(); |
| 576 | /** `base` is the arm server's HTTP side, asked where the joints are so the speed cap starts from there. */ | |
| 583 | /** | |
| 584 | * `base` is the arm server's HTTP side, asked where the joints are so the speed cap starts from there. On the real arm, | |
| 585 | * frames are not passed on while the link reconnects: the stream goes quiet, and the arm server's dead-man holds the arm. | |
| 586 | */ | |
| 577 | 587 | const bridge = (browser: WebSocket, arm: WebSocket, base: string, driver: string | null) => { |
| 588 | const real = base !== simulation; | |
| 578 | 589 | if (driver !== null) { |
| 579 | 590 | streams.set(browser, driver); |
| 580 | 591 | browser.on("close", () => streams.delete(browser)); |
| @@ -606,7 +617,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 606 | 617 | if (lobby && !lobby.turns.isDriver(driver) && !lobby.local) return; |
| 607 | 618 | lobby?.activity(driver); |
| 608 | 619 | } |
| 609 | if (arm.readyState !== WebSocket.OPEN) return; | |
| 620 | if (arm.readyState !== WebSocket.OPEN || (real && mode() !== "live")) return; | |
| 610 | 621 | const percent = lobby?.maxSpeedPercent ?? 100; |
| 611 | 622 | if (percent >= 100) { |
| 612 | 623 | if (lobby) cap.cap(String(data), percent); // keeps each joint's last target, so a cap set mid-stream starts from there |
| @@ -651,12 +662,15 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 651 | 662 | void (async () => { |
| 652 | 663 | let arm: WebSocket; |
| 653 | 664 | let base: string; |
| 665 | if (remote && mode() === "reconnecting") { | |
| 666 | return wss.handleUpgrade(req, socket, head, (browser) => browser.close(1013, `${RECONNECTING}…`)); | |
| 667 | } | |
| 654 | 668 | try { |
| 655 | if (!remote || !live) throw null; | |
| 669 | if (!remote || mode() !== "live") throw null; | |
| 656 | 670 | arm = await open(`${remote.ws}/follow`, readTimeoutMs); |
| 657 | 671 | base = remote.http; |
| 658 | 672 | } catch (err) { |
| 659 | if (err) down(reason(err)); | |
| 673 | if (err) return wss.handleUpgrade(req, socket, head, (browser) => browser.close(1013, `the arm server at ${remote!.host} did not take the stream (${reason(err)})`)); | |
| 660 | 674 | try { |
| 661 | 675 | arm = await open(`${simulationWs}/follow`, readTimeoutMs); |
| 662 | 676 | base = simulation; |
| @@ -674,7 +688,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 674 | 688 | const settingsChanged = lobby?.onSettingsChange(follow); |
| 675 | 689 | const turnEnded = lobby?.onTurnEnd((who) => { |
| 676 | 690 | for (const [browser, id] of streams) if (id === who.id) browser.close(1000, "your turn ended"); |
| 677 | const base = remote && live ? remote.http : simulation; | |
| 691 | const base = remote && onArm() ? remote.http : simulation; | |
| 678 | 692 | fetch(`${base}/stop`, { method: "POST", signal: AbortSignal.timeout(readTimeoutMs) }).then( |
| 679 | 693 | (r) => r.body?.cancel(), |
| 680 | 694 | (err: unknown) => console.error(`stop after ${who.name}'s turn: ${reason(err)}`), |
| @@ -682,9 +696,10 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 682 | 696 | }); |
| 683 | 697 | |
| 684 | 698 | // ------------------------------------------------------------------ start |
| 685 | follow(); | |
| 686 | if (remote) await probe(); // bounded by the probe timeout: the first page already knows which arm it has | |
| 687 | const timer = remote || lobby ? setInterval(() => (follow(), remote && void probe()), probeEveryMs) : undefined; | |
| 699 | if (lobby) follow(); | |
| 700 | if (!armLink) connect(); | |
| 701 | await (armLink as ArmLink | null)?.ready; // bounded by the first attempt's timeout: the first page already knows which arm it has | |
| 702 | const timer = lobby ? setInterval(follow, followEveryMs) : undefined; | |
| 688 | 703 | timer?.unref?.(); |
| 689 | 704 | await new Promise<void>((resolve) => http.listen(options.port ?? 8102, options.host ?? "127.0.0.1", resolve)); |
| 690 | 705 | const address = http.address(); |
| @@ -696,6 +711,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 696 | 711 | clearInterval(timer); |
| 697 | 712 | settingsChanged?.(); |
| 698 | 713 | turnEnded?.(); |
| 714 | armLink?.close(); | |
| 699 | 715 | if (!options.cameras) cameras.close(); |
| 700 | 716 | for (const c of inFlight) c.abort(); |
| 701 | 717 | for (const client of wss.clients) client.terminate(); |
Software/so101/src/server.test.ts +24
| @@ -80,6 +80,30 @@ describe("so101 serve", () => { | ||
| 80 | 80 | const p = await fetch(`${base}/pose/go`, { method: "POST", body: JSON.stringify({ name: "nope" }) }); |
| 81 | 81 | expect(p.status).toBe(404); // no poses file at all: nothing saved yet |
| 82 | 82 | }); |
| 83 | test("/watch pushes the status to every watcher without taking the arm; any other path is not a WebSocket", async () => { | |
| 84 | const watch = () => | |
| 85 | new Promise<{ ws: WebSocket; first: { joints: Array<{ id: number }>; driving: unknown } }>((resolve, reject) => { | |
| 86 | const ws = new WebSocket(`${base.replace("http", "ws")}/watch`); | |
| 87 | ws.once("message", (d) => resolve({ ws, first: JSON.parse(String(d)) })); | |
| 88 | ws.on("error", reject); | |
| 89 | }); | |
| 90 | const [a, b] = await Promise.all([watch(), watch()]); | |
| 91 | expect(a.first.joints.map((j) => j.id)).toEqual([1, 2, 6]); | |
| 92 | expect(b.first.driving).toBeNull(); | |
| 93 | let more = 0; | |
| 94 | a.ws.on("message", () => more++); | |
| 95 | await sleep(600); // pushed every 250 ms | |
| 96 | expect(more).toBeGreaterThanOrEqual(1); | |
| 97 | a.ws.close(); | |
| 98 | b.ws.close(); | |
| 99 | const refused = await new Promise<string>((resolve) => { | |
| 100 | const ws = new WebSocket(`${base.replace("http", "ws")}/nope`); | |
| 101 | ws.on("open", () => resolve("open")); | |
| 102 | ws.on("error", () => resolve("refused")); | |
| 103 | ws.on("close", () => resolve("refused")); | |
| 104 | }); | |
| 105 | expect(refused).toBe("refused"); | |
| 106 | }); | |
| 83 | 107 | }); |
| 84 | 108 | |
| 85 | 109 | describe("leader mapping of a whole frame", () => { |
Software/so101/src/server.ts +46 −2
| @@ -32,6 +32,8 @@ | ||
| 32 | 32 | * GET /app/<file> the modules the page imports itself (the leader arm over Web Serial), as JavaScript |
| 33 | 33 | * GET /printables the sheets to print for the vision quest (board, cube markers, gripper marker); GET /printables/<file> |
| 34 | 34 | * WS /follow client sends {"1": ticks, "2": ticks, ...} frames; the server drives the arm |
| 35 | * WS /watch read-only: the server sends the /status body every watchEveryMs, one bus read shared by every watcher; | |
| 36 | * the hosted app's relay keeps one open, so viewers see the arm without each of them asking the Pi | |
| 35 | 37 | * |
| 36 | 38 | * One thing drives the arm at a time. Every target is pulled inside the joint limits and every motion held to the speed and |
| 37 | 39 | * acceleration limits (src/safety.ts); a stream that goes silent holds, one whose connection is lost parks. Every target, whether a move or a stream frame, is run through |
| @@ -43,6 +45,7 @@ | ||
| 43 | 45 | import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, statSync, unlinkSync, writeFileSync } from "node:fs"; |
| 44 | 46 | import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; |
| 45 | 47 | import { dirname, extname, join, resolve } from "node:path"; |
| 48 | import type { Duplex } from "node:stream"; | |
| 46 | 49 | import { createHash } from "node:crypto"; |
| 47 | 50 | import { fileURLToPath } from "node:url"; |
| 48 | 51 | import { WebSocketServer, type WebSocket } from "ws"; |
| @@ -81,6 +84,8 @@ export interface ServerOptions { | ||
| 81 | 84 | easeSpeed?: number; |
| 82 | 85 | /** How often the stream's jam watch samples position and load, ms. */ |
| 83 | 86 | jamSampleMs?: number; |
| 87 | /** How often each /watch connection is sent the status, ms. */ | |
| 88 | watchEveryMs?: number; | |
| 84 | 89 | } |
| 85 | 90 | |
| 86 | 91 | export interface RunningServer { |
| @@ -892,7 +897,7 @@ export async function startServer(bus: FeetechBus, options: ServerOptions = {}): | ||
| 892 | 897 | // Frames are merged down to the command rate; each joint's target is pulled inside its bounds, judged by the zone, |
| 893 | 898 | // and handed to the governor, which moves the goals toward it within the speed and acceleration limits. |
| 894 | 899 | // Silence makes the arm hold (the dead-man); a connection that is lost rather than closed makes it park. |
| 895 | const wss = new WebSocketServer({ server: http, path: "/follow", maxPayload: 4096 }); | |
| 900 | const wss = new WebSocketServer({ noServer: true, maxPayload: 4096 }); | |
| 896 | 901 | wss.on("connection", (ws: WebSocket) => { |
| 897 | 902 | if (estop) { |
| 898 | 903 | ws.close(ESTOP_CLOSE, stopped(estop)); |
| @@ -1142,6 +1147,43 @@ export async function startServer(bus: FeetechBus, options: ServerOptions = {}): | ||
| 1142 | 1147 | }); |
| 1143 | 1148 | }); |
| 1144 | 1149 | |
| 1150 | // ------------------------------------------------------------------ the watch | |
| 1151 | // Read-only: the status, pushed. One timer reads the bus while anyone watches; a watcher's messages are ignored. | |
| 1152 | const watchers = new WebSocketServer({ noServer: true, maxPayload: 1024 }); | |
| 1153 | const watchEveryMs = options.watchEveryMs ?? 250; | |
| 1154 | let watching: ReturnType<typeof setInterval> | undefined; | |
| 1155 | let reading = false; | |
| 1156 | const broadcast = async () => { | |
| 1157 | if (reading) return; // a slow bus read is not stacked behind another: the next tick asks again | |
| 1158 | reading = true; | |
| 1159 | try { | |
| 1160 | const text = JSON.stringify(await status()); | |
| 1161 | for (const w of watchers.clients) if (w.readyState === w.OPEN) w.send(text); | |
| 1162 | } catch (err) { | |
| 1163 | console.error(`watch: ${err instanceof Error ? err.message : String(err)}`); | |
| 1164 | } finally { | |
| 1165 | reading = false; | |
| 1166 | } | |
| 1167 | }; | |
| 1168 | watchers.on("connection", (ws: WebSocket) => { | |
| 1169 | watching ??= setInterval(() => void broadcast(), watchEveryMs); | |
| 1170 | void broadcast(); | |
| 1171 | ws.on("error", () => ws.terminate()); | |
| 1172 | ws.on("close", () => { | |
| 1173 | if (watchers.clients.size > 0 || !watching) return; | |
| 1174 | clearInterval(watching); | |
| 1175 | watching = undefined; | |
| 1176 | }); | |
| 1177 | }); | |
| 1178 | ||
| 1179 | http.on("upgrade", (req: IncomingMessage, socket: Duplex, head: Buffer) => { | |
| 1180 | const path = new URL(req.url ?? "/", "http://x").pathname; | |
| 1181 | const to = path === "/follow" ? wss : path === "/watch" ? watchers : null; | |
| 1182 | if (!to) return void socket.end("HTTP/1.1 404 Not Found\r\nconnection: close\r\ncontent-length: 0\r\n\r\n"); | |
| 1183 | socket.on("error", () => socket.destroy()); | |
| 1184 | to.handleUpgrade(req, socket, head, (ws) => to.emit("connection", ws, req)); | |
| 1185 | }); | |
| 1186 | ||
| 1145 | 1187 | await new Promise<void>((resolve) => http.listen(options.port ?? 8101, options.host ?? "0.0.0.0", resolve)); |
| 1146 | 1188 | const address = http.address(); |
| 1147 | 1189 | const port = typeof address === "object" && address ? address.port : options.port ?? 8101; |
| @@ -1149,8 +1191,10 @@ export async function startServer(bus: FeetechBus, options: ServerOptions = {}): | ||
| 1149 | 1191 | port, |
| 1150 | 1192 | close: () => |
| 1151 | 1193 | new Promise<void>((resolve) => { |
| 1152 | for (const client of wss.clients) client.terminate(); | |
| 1194 | clearInterval(watching); | |
| 1195 | for (const client of [...wss.clients, ...watchers.clients]) client.terminate(); | |
| 1153 | 1196 | wss.close(); |
| 1197 | watchers.close(); | |
| 1154 | 1198 | http.close(() => resolve()); |
| 1155 | 1199 | http.closeAllConnections(); // the viewer keeps idle keep-alive connections open; close() alone would wait for them |
| 1156 | 1200 | }), |
Software/so101/src/viewer.html +22 −10
| @@ -72,8 +72,8 @@ | ||
| 72 | 72 | #top button { background: rgba(0,0,0,.6); border-color: rgba(255,255,255,.18); color: #e8eaed; min-height: 36px; padding: 0 10px; border-radius: 8px; font-size: 13px; } |
| 73 | 73 | #top .seg { display: inline-flex; flex-wrap: wrap; gap: 2px; } #top .seg button { max-width: 40vw; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } #top .seg button.on { background: #388bfd; border-color: #388bfd; color: #fff; } |
| 74 | 74 | #top .info { display: inline-flex; gap: 6px; align-items: center; background: rgba(0,0,0,.6); padding: 0 10px; min-height: 36px; border-radius: 8px; max-width: 100%; } |
| 75 | #link { color: #9aa4ae; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: min(36ch, 46vw); } #link.on { color: #3fb950; } #link.warn { color: #d29922; } | |
| 76 | .dot { width: 8px; height: 8px; border-radius: 50%; background: #f85149; display: inline-block; flex: none; } .dot.on { background: #3fb950; } | |
| 75 | #link { color: #9aa4ae; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: min(36ch, 46vw); } #link.on { color: #3fb950; } #link.warn { color: #d29922; } #link.bad { color: #f85149; font-weight: 700; } | |
| 76 | .dot { width: 8px; height: 8px; border-radius: 50%; background: #f85149; display: inline-block; flex: none; } .dot.on { background: #3fb950; } .dot.warn { background: #d29922; } | |
| 77 | 77 | #banner { position: absolute; top: 50px; left: 50%; transform: translateX(-50%); z-index: 5; width: max-content; max-width: min(640px, calc(100% - 16px)); padding: 8px 12px; border-radius: 8px; font-size: 13px; background: rgba(0,0,0,.8); border: 1px solid #30363d; display: none; } |
| 78 | 78 | #banner.bad { display: block; border-color: #f85149; color: #ffb3ad; } #banner.warn { display: block; border-color: #d29922; color: #f2cc60; } |
| 79 | 79 | #msg { position: absolute; left: calc(8px + var()); right: calc(8px + var()); bottom: 8px; z-index: 6; margin: 0 auto; width: max-content; max-width: calc(100% - 16px); padding: 8px 12px; border-radius: 8px; font-size: 13px; background: rgba(13,17,23,.9); color: #e8eaed; border: 1px solid #30363d; } |
| @@ -514,20 +514,27 @@ | ||
| 514 | 514 | // ---------------------------------------------------------------- which arm |
| 515 | 515 | // Behind the hosted app's relay (/link answers), the arm is a live `so101 serve` or the simulation, and the cameras come through the relay. |
| 516 | 516 | const relayed = await fetch("/link", { cache: "no-store" }).then((r) => r.ok, () => false); |
| 517 | let linkMode = null; | |
| 517 | // A real arm that drops is "reconnecting": the same arm, its last pose kept, driving paused. Only after a minute down | |
| 518 | // does the relay hand the page the simulation, and then it says OFFLINE in red. | |
| 519 | let linkMode = null, linkNow = null; | |
| 520 | const realArm = (mode) => mode === "live" || mode === "reconnecting"; | |
| 518 | 521 | async function pollLink() { |
| 519 | 522 | const l = await api("/link").catch(() => null); |
| 520 | 523 | if (l?.ok) { |
| 524 | linkNow = l.mode; | |
| 521 | 525 | const el = $("link"); |
| 522 | 526 | el.hidden = false; |
| 523 | el.textContent = l.mode === "live" ? `Live arm · ${l.host}` : l.problem ? `Simulated · ${l.problem}` : "Simulated"; | |
| 527 | el.textContent = l.mode === "live" ? `Live arm · ${l.host}` : l.mode === "reconnecting" ? `Reconnecting to the arm… · ${l.host}` | |
| 528 | : l.mode === "offline" ? `OFFLINE · simulated arm · ${l.problem}` : l.problem ? `Simulated · ${l.problem}` : "Simulated"; | |
| 524 | 529 | el.title = el.textContent; |
| 525 | el.className = l.mode === "live" ? "on" : l.problem ? "warn" : ""; | |
| 530 | el.className = l.mode === "live" ? "on" : l.mode === "offline" ? "bad" : l.mode === "reconnecting" || l.problem ? "warn" : ""; | |
| 526 | 531 | // Another arm has other limits, poses and joints: start the page over on it, once nothing on the page is driving. |
| 527 | if (linkMode !== null && l.mode !== linkMode && !stream.ws && !leader) return location.reload(); | |
| 532 | // So does a page that opened while reconnecting and never got the arm's limits. | |
| 533 | const other = linkMode !== null && realArm(l.mode) !== realArm(linkMode); | |
| 534 | if ((other || (l.mode === "live" && !limits?.ok)) && linkMode !== null && !stream.ws && !leader) return location.reload(); | |
| 528 | 535 | if (linkMode === null) linkMode = l.mode; |
| 529 | 536 | } |
| 530 | setTimeout(pollLink, 3000); | |
| 537 | setTimeout(pollLink, l?.mode === "live" ? 3000 : 1000); | |
| 531 | 538 | } |
| 532 | 539 | |
| 533 | 540 | // ---------------------------------------------------------------- cameras |
| @@ -1573,6 +1580,8 @@ | ||
| 1573 | 1580 | if (isOwner()) b.append(el("button", { textContent: "Clear the stop", className: "danger", onclick: clearEStop })); |
| 1574 | 1581 | return; |
| 1575 | 1582 | } |
| 1583 | if (d.reconnecting) { b.className = "warn"; b.textContent = `Reconnecting to the arm… showing where it was at ${new Date(d.seenAt).toLocaleTimeString()}. Driving is paused, and the arm holds where it is.`; return; } | |
| 1584 | if (typeof d.source === "string" && d.source.startsWith("simulated: offline")) { b.className = "bad"; b.textContent = "The arm has been unreachable for over a minute. This is the SIMULATION, not the real arm."; return; } | |
| 1576 | 1585 | if (d.jammed?.length) { b.className = "bad"; b.textContent = `jam: ${d.jammed.map((id) => NAMES[id] ?? id).join(", ")} pushed against something and stopped — back the leader off`; } |
| 1577 | 1586 | else if (d.blocked) { b.className = d.zone === "enforce" ? "bad" : "warn"; b.textContent = `${d.zone === "enforce" ? "held at the boundary" : "warning"}: ${say(d.blocked)}`; } |
| 1578 | 1587 | else if (d.violation) { b.className = "warn"; b.textContent = `the arm is ${say(d.violation)} (zone: ${d.zone})`; } |
| @@ -1587,15 +1596,18 @@ | ||
| 1587 | 1596 | rtts.push(performance.now() - started); if (rtts.length > 10) rtts.shift(); // the link's round trip, for the camera plan |
| 1588 | 1597 | status = d; |
| 1589 | 1598 | if (Boolean(d.estop) !== wasStopped) applyLock(); |
| 1590 | $("dot").className = "dot on"; | |
| 1591 | $("state").textContent = d.driving === null ? "idle" : d.driving === "record" ? `recording ${d.recording}` : leaderStreaming() ? `leader · ${leader.frames} frames` : d.driving; | |
| 1599 | $("dot").className = d.reconnecting ? "dot warn" : "dot on"; | |
| 1600 | $("state").textContent = d.reconnecting ? "reconnecting…" : d.driving === null ? "idle" : d.driving === "record" ? `recording ${d.recording}` : leaderStreaming() ? `leader · ${leader.frames} frames` : d.driving; | |
| 1592 | 1601 | $("estop").classList.toggle("on", Boolean(d.estop)); $("estop").textContent = d.estop ? "Stopped" : "Stop"; |
| 1593 | 1602 | $("record").textContent = d.driving === "record" ? "■ Stop recording" : "● Record"; |
| 1594 | 1603 | for (const id of ["hold", "releaseBtn", "play"]) $(id).disabled = d.driving !== null; |
| 1595 | 1604 | if (tab === "controls") render(d); |
| 1596 | 1605 | banner(d); applyModel(d); |
| 1597 | 1606 | $("readout").innerHTML = `<span>lowest part <b>${Number.isFinite(d.lowestMm) ? d.lowestMm : "—"} mm</b> above the table</span><span>zone <b>${d.zone === "enforce" || d.zone === "warn" || d.zone === "off" ? d.zone : "—"}</b></span>`; |
| 1598 | } catch { $("dot").className = "dot"; $("state").textContent = "no arm"; } | |
| 1607 | } catch { | |
| 1608 | const reconnecting = linkNow === "reconnecting"; | |
| 1609 | $("dot").className = reconnecting ? "dot warn" : "dot"; $("state").textContent = reconnecting ? "reconnecting…" : "no arm"; | |
| 1610 | } | |
| 1599 | 1611 | // Hidden, the page keeps the arm's state fresh at a walk: nobody is looking, and a phone's battery and data matter. |
| 1600 | 1612 | setTimeout(poll, document.hidden ? 2000 : 200); |
| 1601 | 1613 | } |
new above
offline