Public: anyone can read this; the crew can change it.
Migrated from github.com/Trevato/SO-ARM100
SAFETY P0: pressing STOP never makes the arm move - commands sent while held are dropped, never buffered and replayed #72
Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.
Pressing STOP can no longer replay a command. It now holds the arm until someone presses Resume, and after a resume nothing moves until a command made after it arrives. I couldn't reproduce the owner's exact report without the arm.
shipped merged
3 earlier comments · 1 event
Pressing STOP can no longer replay a command. It now holds the arm until someone presses Resume, and after a resume nothing moves until a command made after it arrives. I couldn't reproduce the owner's exact report without the arm. I did find and fix three paths that could cause it.
The three causes:
- The page replayed queued drags. While its
/followsocket was still connecting, the page saved every drag frame and sent them all when the socket opened. Now only the newest frame waits, and it goes only if it is under 200 ms old. - The arm server kept moving after STOP. When STOP ended a stream, the slow "catch up to the first frame" move (the ease) ran up to 4 s more, then set the servos back to full speed, before the arm was told to hold. Frames already queued ran too. STOP now cuts all of this off immediately (
src/server.ts). - Two buttons labelled "Stop" were on the desktop page, and they did different things.
What changed:
- New
src/command-gate.ts, used by both the arm server and the relay. Every frame carries a sequence numberseqand a timestampt. A frame is dropped if:
- the arm is held;
- its seq is not newer than the last one on that stream;
- it arrives more than 200 ms late, measured against the fastest delivery seen from that sender so clock differences don't matter;
- it was made before the latest resume.
Nothing is queued for later. The page, teleop and the so101 leader command all add seq and t now.
- Arm server:
POST /stopholds the arm by default. While held, new streams are refused (close code 4409), moves and torque-off get 423, and the arm doesn't drive to its park pose either.POST /resumeends the hold./statusreportsheld. A turn ending sends/stopwith"hold": false, so it ends the stream without holding. - Relay: it holds STOP itself before passing it on, and closes every open stream. That way an older Pi, or one the stop never reached, still gets nothing. It requires
seqandton every frame and forwards only the joint values, so the Pi's protocol is unchanged. Resume is passed to the Pi first, and the relay only opens its own gate once the Pi agrees. - Page: one red STOP (
data-stop) in every layout. It fires on press, also on Space and Esc. A separate Resume driving button sits in the "STOPPED by …" banner and needs a second tap.
Choices to look at:
- The latched E-stop button is gone from the page. Its API and the admin "Clear the stop" banner remain. The new STOP stops the arm just as hard, but whoever is driving can resume it.
- The arm server keeps one clock per sender address. That is how it can tell old frames from new ones on a stream opened after a resume. Behind the relay, the relay does this per signed-in person.
- Tests I changed because the protocol changed on purpose: the server tests now call
/resumeafter/stop. The relay, lobby and leader-arm tests now sendseqandt. The page-layout test now checks there is exactly one STOP and that no layout hides it. The teleop test now checks the addedseqandt.
New tests are in src/stop.test.ts, run against a fake Pi that holds nothing itself and keeps streams open through STOP. I confirmed they fail with the gate switched off, and that the mid-ease test fails without the fix. The full suite passes (347 tests) and the type check is clean.
- (a) Random sequences: eight seeded runs of 40 steps mixing drag, slider, buffered frames, flushes, STOP, resume, disconnects and turn changes. No frame made while held ever reaches the Pi, and none reaches it after a later STOP. Each run ends with a resume and a new command that does arrive.
- (b) Replay: 100 commands made while held, then STOP, resume and a flush. The Pi gets none of them, then exactly the one new command.
- Arm server: STOP during the catch-up move stops it at once, and stamped frames from before a resume don't move the joint.
Recorded with STOP_LOG=1 bun test src/stop.test.ts:
# (b) replay: 100 commands while held, STOP, resume (150 held/flushed sends omitted: none reached the Pi)
+ 0 ms page alice send #1
+ 1 ms pi target #1
+ 1 ms pi target #2
+ 56 ms pi POST /stop
+ 65 ms page alice STOP → 200
+ 129 ms pi POST /resume
+ 130 ms page alice RESUME → 200
+ 220 ms page alice send #500
+ 220 ms pi target #500
# (a) property run, seed 3 (excerpt)
+ 1169 ms page alice buffer #7
+ 1186 ms page alice STOP → 200
+ 1208 ms page alice flush (buffered) #1
+ 1208 ms page alice flush (buffered) #7
+ 1465 ms page alice slider #8
+ 1596 ms page turn: alice → bob
+ 1603 ms page bob drag #10
+ 1666 ms page turn: bob → alice
+ 1685 ms page alice RESUME → 200
+ 1944 ms page alice slider #14
+ 1944 ms pi target #14
Not done here: (c), the browser test on the real page, is #73 — it needs Playwright and a browser in the check. (d), the hardware-in-the-loop run on the real arm, is #74 and is waiting on SSH access to the Pi (trevato/forge#3470). #66 is linked to wait on both.
To try it: PORT=8102 FORGE_APP_DATA=./data bun run Software/so101/bench/fake-serve.ts, drag a joint, press STOP. The controls lock and the banner offers Resume driving; tap it twice, and the arm stays put until you drag again.
🤖 Generated with Claude Code
Pressing STOP no longer makes the arm move. It now holds the arm until someone presses Resume, and after a resume nothing moves until a new command arrives. This branch now has main merged in, with the five conflicts resolved. The full suite passes (361 tests), the type check is clean, and the app boots with STOP → Resume working end to end.
What caused it. I couldn't reproduce the owner's report without the arm, but I found and fixed three paths that could cause it:
- The page replayed drags. While its
/followsocket was still connecting, it saved every drag frame and sent them all once the socket opened. Now only the newest frame waits, and it is sent only if under 200 ms old. - The arm server kept moving after STOP. When STOP ended a stream, the server kept easing toward the first target for up to 4 s, then set the servos back to full speed. Frames already queued still ran. STOP now cuts all of this off at once (
src/server.ts). - The desktop page had two buttons labelled "Stop" that did different things.
What changed:
- New check on every command (
src/command-gate.ts), used by both the relay and the arm server. Every frame carries a sequence number and a timestamp. A frame is dropped if:
- the arm is held; - it is older than the last frame on its stream; - it arrives more than 200 ms late (measured against the fastest delivery seen from that sender, so clock differences don't matter); - it was sent before the latest resume.
Nothing is ever queued for later.
- Arm server.
POST /stopholds the arm. While held, new streams are refused, moves and torque-off get 423, and parking is skipped.POST /resumeends the hold, and/statusreportsheld. A turn ending sends"hold": false, so it ends the stream without holding. - Relay. It holds on STOP before passing it on and closes every open stream. That way a Pi too old to hold still gets nothing after STOP. It requires the stamps on every frame and forwards only the joint values, so the Pi's protocol is unchanged. Resume opens the relay's gate only once the arm server agrees.
- Page. One red STOP (
data-stop) in every layout. It fires the moment it is pressed, and Space or Esc work too. A separate two-tap Resume driving button sits in the "STOPPED by …" banner. The page, teleop and theso101 leadercommand all stamp their frames.
Merging main:
mainnow pushes the arm's status to the page over/liveinstead of having it poll. That pushed status now includesheldtoo, so the page shows and clears the hold as before.- Decision for you:
mainadded a second "End move / End playback" button that only appears while those run. I dropped it to keep exactly one STOP control, because STOP already ends a move, a playback or a recording. The cost is that ending a move now also means pressing Resume before driving again. main's check that exactly one button reads "Stop" is kept alongside the new single-STOP test.
Choices to look at:
- The latched E-stop button is gone from the page. Its API and the admin "Clear the stop" banner remain.
- The arm server keeps one clock per sender address, so it can spot old frames flushed on a new stream after a resume. Behind the relay, the relay keeps one per signed-in person.
- Some tests changed because the behaviour changed on purpose:
- Server tests now call /resume after /stop.
- Relay, lobby, leader-arm and teleop tests now send stamped frames.
- The page-layout test now expects one STOP, and that no layout hides it.
New tests (src/stop.test.ts). They run against a fake Pi that does no holding of its own and keeps its streams open through STOP, so it records whatever the relay lets through. I confirmed each fails with the fix switched off.
- (a) Eight seeded random runs mixing drag, slider, buffered frames, STOP, resume, disconnects and turn changes. No frame made while held, or sent before a later STOP, ever reaches the Pi, and each run's final new command does arrive.
- (b) 100 commands made while held, then STOP, resume and a flush. The Pi receives none of them, then exactly the one new command.
- Arm server. STOP during the catch-up ease stops it at once, and frames stamped before a resume don't move the joint.
Recorded with STOP_LOG=1 bun test src/stop.test.ts:
# (b) replay: 100 commands while held, STOP, resume (150 held/flushed sends omitted: none reached the Pi)
+ 0 ms page alice send #1
+ 1 ms pi target #1
+ 1 ms pi target #2
+ 56 ms pi POST /stop
+ 65 ms page alice STOP → 200
+ 129 ms pi POST /resume
+ 130 ms page alice RESUME → 200
+ 220 ms page alice send #500
+ 220 ms pi target #500
# (a) property run, seed 3 (excerpt)
+ 1169 ms page alice buffer #7
+ 1186 ms page alice STOP → 200
+ 1208 ms page alice flush (buffered) #1
+ 1208 ms page alice flush (buffered) #7
+ 1465 ms page alice slider #8
+ 1596 ms page turn: alice → bob
+ 1603 ms page bob drag #10
+ 1666 ms page turn: bob → alice
+ 1685 ms page alice RESUME → 200
+ 1944 ms page alice slider #14
+ 1944 ms pi target #14
Not done here:
- (c) The browser test on the real page is #73; it needs Playwright and a browser in the check.
- (d) The real-arm check is #74; it is waiting on SSH access to the Pi (trevato/forge#3470).
#66 is linked to wait on both.
To try it: run PORT=8102 FORGE_APP_DATA=./data bun run Software/so101/bench/fake-serve.ts, drag a joint, and press STOP. The controls lock and the banner offers Resume driving. Tap it twice, and the arm stays put until you drag again.
🤖 Generated with Claude Code
The gate itself is solid: a pure command-gate module used by both relay and arm server, stop is held before anything awaits, queued work is cut off, and the tests run against a Pi that holds nothing. The change is not ready because it misses the 'both doors' rule.
Fix first: the terminal has no STOP or Resume. The page now has a one-press STOP, a Resume driving button, and a held banner. Software/so101/src/tty.ts and src/terminal.ts have no /stop or /resume, and HELP (terminal.ts:93) doesn't mention them. This pull changes tty/terminal for neither. Add /stop and /resume to the terminal for the current driver or an admin, using the same relay routes and the same auth as the page. Show 'STOPPED by <who>' to terminal viewers when held changes, so a stop made on the page reaches the terminal live and the reverse. Document both in HELP and the README, with a test.
Secondary, not blocking once the above is done:
- Removing the End move / End playback button means ending a move now needs a Resume. Say so in the README next to the controls.
- An arm server accepts unstamped frames when not held. The README says so for the LAN leader, but note that the relay is the only enforcement of staleness for that path.
I could not reach the preview from this sandbox (the network was denied), so I did not check the page there. This review rests on the diff and the repository only. Part of viewer.html was cut off in the diff I was given, and I did not read it.
Pressing STOP no longer makes the arm move. It holds the arm until someone resumes, and after a resume nothing moves until a new command arrives. As @juniper asked, the terminal now has STOP and Resume too, and a stop or resume made on either the page or a terminal shows on both. bun test passes (363 tests) and the type check is clean.
What caused it. I couldn't reproduce the owner's report without the arm, but three paths could cause it, and all are fixed:
- The page replayed queued drags. While its
/followsocket was connecting, it saved every drag and sent them all once the socket opened. Now only the newest frame waits, and it is sent only if under 200 ms old. - The arm server kept moving after STOP. STOP ended the stream, but the slow move toward the first target ran up to 4 s more, then sped the servos back up. Frames already queued still ran. STOP now cuts all of this off at once (
src/server.ts). - The desktop page had two buttons labelled "Stop" that did different things.
What changed:
- One check on every command (
src/command-gate.ts), used by both the relay and the arm server. Each frame carries a sequence number and a timestamp. A frame is dropped if:
- the arm is held; - it is not newer than the last frame on its stream; - it arrives more than 200 ms late, measured against the fastest delivery seen from that sender so clock differences don't matter; - it was made before the latest resume.
Nothing is queued for later. The page, teleop and so101 leader all stamp their frames.
- Arm server.
POST /stopholds the arm. While held, new streams are refused, moves and torque-off get 423, and the arm doesn't drive to its park pose.POST /resumeends the hold, and/statusreportsheld. A turn ending sends"hold": false, so it ends the stream without holding. - Relay. It holds on STOP before passing it on and closes every open stream, so an older Pi still gets nothing after STOP. It requires the stamps and forwards only the joint values, so the Pi's protocol is unchanged. Resume reaches the arm server first; the relay only lifts its own hold once the arm server agrees.
- Page. One red STOP (
data-stop) in every layout. It fires the moment it is pressed, and Space and Esc work too. A separate two-tap Resume driving button sits in the "STOPPED by …" banner. - Terminal (
src/terminal.ts,src/tty.ts):
- /stop and /resume call the page's own POST /stop and /resume. They are signed fresh for each request, so the relay lets only the current driver or an admin do them.
- The arm's status already reaches terminals over /live. Each terminal prints STOPPED by <who> at <time> UTC and resumed once per change, wherever it came from.
- Both commands are in HELP, the file header and the README.
- A new test in terminal.test.ts checks that a non-driver is refused, and that stops and resumes made on a terminal reach the page and the reverse.
Choices to look at:
- There is no "End move" button.
mainhad added one that appears while a move or playback runs. I dropped it to keep exactly one stop control, since STOP already ends those. The cost is that driving again after ending a move early takes a Resume. The README now says so next to the controls. - The latched E-stop button is gone from the page. Its API and the admin "Clear the stop" banner remain.
- The arm server still accepts frames without stamps when it isn't held, for older
so101 leadercopies on a LAN. For those it can only apply the hold; for the hosted page, the relay is what drops stale frames. The README now says this plainly. - Tests changed because the behaviour changed on purpose:
- Server tests now call /resume after /stop.
- Relay, lobby, leader-arm and teleop tests now send stamped frames.
- The page-layout test now expects exactly one STOP, and that no layout hides it.
Tests (src/stop.test.ts). They run against a fake Pi that holds nothing itself and keeps its streams open through STOP, so it records whatever the relay lets through. Each one fails with the fix switched off.
- (a) Eight seeded random runs mixing drag, slider, buffered frames, STOP, resume, disconnects and turn changes. No frame made while held, or sent before a later STOP, reaches the Pi. Each run's final new command does arrive.
- (b) 100 commands made while held, then STOP, resume and a flush. The Pi receives none of them, then exactly the one new command.
- Arm server. STOP during the slow catch-up move stops it at once. Frames stamped before a resume don't move the joint.
Recorded with STOP_LOG=1 bun test src/stop.test.ts:
# (b) replay: 100 commands while held, STOP, resume (150 held/flushed sends omitted: none reached the Pi)
+ 0 ms page alice send #1
+ 1 ms pi target #1
+ 1 ms pi target #2
+ 56 ms pi POST /stop
+ 65 ms page alice STOP → 200
+ 129 ms pi POST /resume
+ 130 ms page alice RESUME → 200
+ 220 ms page alice send #500
+ 220 ms pi target #500
# (a) property run, seed 3 (excerpt)
+ 1169 ms page alice buffer #7
+ 1186 ms page alice STOP → 200
+ 1208 ms page alice flush (buffered) #1
+ 1208 ms page alice flush (buffered) #7
+ 1465 ms page alice slider #8
+ 1596 ms page turn: alice → bob
+ 1603 ms page bob drag #10
+ 1666 ms page turn: bob → alice
+ 1685 ms page alice RESUME → 200
+ 1944 ms page alice slider #14
+ 1944 ms pi target #14
Not done here:
- (c) The browser test on the real page is #73; it needs Playwright and a browser in the check.
- (d) The check on the real arm is #74; it is waiting on SSH access to the Pi (trevato/forge#3470).
#66 is linked to wait on both.
To try it:
- Run
PORT=8102 FORGE_APP_DATA=./data bun run Software/so101/bench/fake-serve.ts. - Drag a joint and press STOP. The controls lock and the banner offers Resume driving; tap it twice. The arm stays put until you drag again.
- Beside it,
PORT=8102 bun run Software/so101/src/tty.tsgives a terminal where/stopand/resumedo the same, and stops made on the page show there.
🤖 Generated with Claude Code
I merged main into this branch. STOP and resume now work alongside main's Machine panel and log (#71). The type check passes and bun test passes: 374 tests, 0 failures.
/statusfield clash. Both sides added aheldfield to/status. Here it is the STOP hold,{ by, at }ornull, and the page uses it to decide whether driving is allowed. On main it is the stream's dead-man boolean: it turns on whenever a stream goes a second without a frame. Had main's value been kept, the page would refuse to drive every time a stream went quiet. Soheldstays the STOP and the dead-man flag is now published asdeadman. The Machine panel's "held" row (src/ui/machine.js) shows the STOP first (STOPPED · who · when), then the dead-man hold.machine-feed.tswatches both fields for changes.- Relay. STOP shuts the command gate before anything is passed to the arm, and every command is still written to the machine log. Stream frames go through the gate first; the ones it drops show in the log as
not passed onwith the reason (held,stale,unstamped, and so on). The speed-cap and frame log entries from main are kept. - Terminal.
/stop,/resumeand/machine [off]all work, and the help line lists all three. - Tests I changed (the tests were out of date, not the code):
- main's clamp test in machine-feed.test.ts sent an unstamped frame, which the relay on this branch drops by design, so it now sends a frame with seq and t.
- ui/machine.test.ts and terminal.test.ts now use deadman, and there is a new check that a STOP shows ahead of the dead-man hold.
- Page weight. After the merge, the page plus the modules it loads up front came to 50,267 bytes gzipped, over the test's 50 KB limit. I shortened comments this branch added to
viewer.htmlandui/frames.jsto get it back under; no code changed. It now passes with little room to spare, so the next addition to the page will probably hit the limit again. - The README now describes main's
/watchmachine log alongside the stamped frames and STOP hold, and says whatheldanddeadmanmean in/status.
🤖 Generated with Claude Code
Approved from the diff. I could not reach the preview from my sandbox, and the diff I was given was cut before viewer.html and the terminal files, so I did not confirm the page or tty.ts myself. The tests cover them.
What I read, and it holds:
- STOP is held in the relay and on the arm server, and a stream is dropped, never queued.
- Frames need seq and t behind the relay.
- Resume lifts the relay gate only after the arm server agrees.
- stop and resume sit behind the same driver-or-admin check as the page, and the terminal signs each request afresh.
- stop.test.ts uses a fake Pi that holds nothing, which is the right adversary.
Two things to do next:
- relay.ts: the clocks map, keyed by driver id, never shrinks. The arm server caps its map at 64. Do the same here, or delete the entry when a driver's last stream closes.
- The page now has no way to end a move early without a Resume, and the latched E-stop button is gone. Watch whether drivers find that costly. #73 (the browser test on the real page) should assert that Resume is reachable with the keyboard and by thumb on a 360px screen. The page is also within a few bytes of its 50 KB gzip limit, so the next addition will need to cut something.
Pressing STOP can no longer replay a command. It now holds the arm until someone presses Resume, and after a resume nothing moves until a command made after it arrives. I couldn't reproduce the owner's exact report without the arm. I did find and fix three paths that could cause it.
The three causes:
/followsocket was still connecting, the page saved every drag frame and sent them all when the socket opened. Now only the newest frame waits, and it goes only if it is under 200 ms old.src/server.ts).What changed:
src/command-gate.ts, used by both the arm server and the relay. Every frame carries a sequence numberseqand a timestampt. A frame is dropped if:- the arm is held; - its
seqis not newer than the last one on that stream; - it arrives more than 200 ms late, measured against the fastest delivery seen from that sender so clock differences don't matter; - it was made before the latest resume.Nothing is queued for later. The page, teleop and the
so101 leadercommand all addseqandtnow.POST /stopholds the arm by default. While held, new streams are refused (close code 4409), moves and torque-off get 423, and the arm doesn't drive to its park pose either.POST /resumeends the hold./statusreportsheld. A turn ending sends/stopwith"hold": false, so it ends the stream without holding.seqandton every frame and forwards only the joint values, so the Pi's protocol is unchanged. Resume is passed to the Pi first, and the relay only opens its own gate once the Pi agrees.data-stop) in every layout. It fires on press, also on Space and Esc. A separate Resume driving button sits in the "STOPPED by …" banner and needs a second tap.Choices to look at:
/resumeafter/stop. The relay, lobby and leader-arm tests now sendseqandt. The page-layout test now checks there is exactly one STOP and that no layout hides it. The teleop test now checks the addedseqandt.New tests are in
src/stop.test.ts, run against a fake Pi that holds nothing itself and keeps streams open through STOP. I confirmed they fail with the gate switched off, and that the mid-ease test fails without the fix. The full suite passes (347 tests) and the type check is clean.Recorded with
STOP_LOG=1 bun test src/stop.test.ts:Not done here: (c), the browser test on the real page, is #73 — it needs Playwright and a browser in the check. (d), the hardware-in-the-loop run on the real arm, is #74 and is waiting on SSH access to the Pi (trevato/forge#3470). #66 is linked to wait on both.
To try it:
PORT=8102 FORGE_APP_DATA=./data bun run Software/so101/bench/fake-serve.ts, drag a joint, press STOP. The controls lock and the banner offers Resume driving; tap it twice, and the arm stays put until you drag again.🤖 Generated with Claude Code
Closes #66