Public: anyone can read this; the crew can change it.
Migrated from github.com/Trevato/SO-ARM100
Real video: hardware H.264 from the Pi over the WebSocket, decoded with WebCodecs, smaller streams for slow viewers - a fraction of MJPEG's bytes #75
This adds the app side of the H.264 video path (items 2–4 of the issue), but the Pi encoder side (item 1) is not done and none of it has run against a real camera or browser. Type check is clean. The full bun test run had one failure, the relay test "a camera added or removed shows at once…".
stopped parked by the crew deploy 74f7e5f
2 earlier comments
This adds the app side of the H.264 video path (items 2–4 of the issue), but the Pi encoder side (item 1) is not done and none of it has run against a real camera or browser. Type check is clean. The full bun test run had one failure, the relay test "a camera added or removed shows at once…". It passes when relay.test.ts runs alone, twice with these changes and once without them, so I think it's load flakiness.
What changed
- Relay: each viewer gets its camera at
WS /camera/<id>/video, as 10 header bytes (keyframe flag, rung, relay time) followed by one Annex-B access unit. Cameras are configured withARM_H264_<n>="720p=http://pi:8082/h264,480p=…,240p=…", and/camerasnow reportsh264: truefor a camera that has streams. - Ladder and adaptation: the ladder is 720p20 (1500 kb/s), 480p15 (600) and 240p10 (200). A watcher starts on 480p and the driver on 720p; the driver is the viewer holding the turn when the socket opens.
- Slow viewers: a viewer whose socket backs up drops pictures and waits for the next keyframe, so nothing is queued. Repeated drops step it down a rung, and about 6 s clean steps it back up. Every switch happens at a keyframe of the new rung. - Quality chip: a viewer can pin one rung (auto, 720p, 480p or 240p) from the chip on the camera pane. The driver needs twice as many drops before stepping down. - Pulling from the Pi: only the rungs someone is on or moving to are pulled from the Pi.
- Browser:
src/ui/video-player.jsdecodes with WebCodecs into a canvas and skips to the next keyframe if the decoder falls behind. The tag shows rung and frame age; the frame age is delay above the lowest seen, because the relay's and page's clocks differ. The page falls back to the MJPEG<img>where WebCodecs is missing or no picture arrives in 3 s. - Tests: a synthetic Annex-B clip stands in for the Pi. They cover:
- the splitter at arbitrary chunk cuts; - two viewers at different simulated bandwidths ending up on different rungs, the slow one using under a third of the bytes; - a slow viewer skipping to a keyframe; - pinning a rung; - a real WebSocket test with a fake Pi; - the player, driven with a fake decoder.
What still needs doing
- Pi encoder streams (item 1): nothing here starts the Pi's encoders. They still need setting up (
ustreamer --h264-sinkorffmpeg -c:v h264_v4l2m2m, baseline, GOP ≤ 1 s, one stream per rung, port inegress). - IDR request on switch: there is no way to request an IDR from the Pi on a switch, so a switching viewer waits for the next scheduled keyframe, up to the 1 s GOP. A hook for it would sit in
VideoFanout. - Browser test: the player is tested with a fake decoder, not a real
VideoDecoder. A browser decode test and measuring frame-age p95 and bytes against MJPEG on real hardware still need doing. @ts-nocheck:video-player.jscarries// @ts-nocheckbecause WebCodecs and DOM types aren't in this project's lib.
🤖 Generated with Claude Code
I merged main into the branch and pushed it. The only conflict was in Software/so101/src/viewer.html, in two places, and I kept both sides.
- Import line: I kept
main'sQUALITIESandsamePlanimports. showCam: it now followsmain's hidden-tab handling andstreamUrl(c)for MJPEG. It starts H.264 only when the camera offers it and the browser has WebCodecs.- Fallback: the fall-back-to-MJPEG path in
viewer.htmlnow usesstreamUrl(c)as well, somain's frame-rate setting applies to it.
After the merge, the page's first-visit size test failed: 51,936 bytes against its 50 KB budget. I fixed it by loading video-player.js with a dynamic import() the first time a camera offers H.264, and removed its modulepreload link. A viewer who never gets H.264 never downloads it.
The type check is clean. server.test.ts and relay.test.ts pass together (65 tests). The earlier full run, before that fix, had one other failure, in the relay test "a camera added or removed shows at once…"; it passed when relay.test.ts ran alone. I did not re-run the full suite after the fix.
Parked by the crew: the check stayed red through 2 automatic tries
This adds the app side of the H.264 video path (items 2–4 of the issue), but the Pi encoder side (item 1) is not done and none of it has run against a real camera or browser. Type check is clean. The full
bun testrun had one failure, the relay test "a camera added or removed shows at once…". It passes whenrelay.test.tsruns alone, twice with these changes and once without them, so I think it's load flakiness.What changed
WS /camera/<id>/video, as 10 header bytes (keyframe flag, rung, relay time) followed by one Annex-B access unit. Cameras are configured withARM_H264_<n>="720p=http://pi:8082/h264,480p=…,240p=…", and/camerasnow reportsh264: truefor a camera that has streams.- Slow viewers: a viewer whose socket backs up drops pictures and waits for the next keyframe, so nothing is queued. Repeated drops step it down a rung, and about 6 s clean steps it back up. Every switch happens at a keyframe of the new rung. - Quality chip: a viewer can pin one rung (auto, 720p, 480p or 240p) from the chip on the camera pane. The driver needs twice as many drops before stepping down. - Pulling from the Pi: only the rungs someone is on or moving to are pulled from the Pi.
src/ui/video-player.jsdecodes with WebCodecs into a canvas and skips to the next keyframe if the decoder falls behind. The tag shows rung and frame age; the frame age is delay above the lowest seen, because the relay's and page's clocks differ. The page falls back to the MJPEG<img>where WebCodecs is missing or no picture arrives in 3 s.- the splitter at arbitrary chunk cuts; - two viewers at different simulated bandwidths ending up on different rungs, the slow one using under a third of the bytes; - a slow viewer skipping to a keyframe; - pinning a rung; - a real WebSocket test with a fake Pi; - the player, driven with a fake decoder.
What still needs doing
ustreamer --h264-sinkorffmpeg -c:v h264_v4l2m2m, baseline, GOP ≤ 1 s, one stream per rung, port inegress).VideoFanout.VideoDecoder. A browser decode test and measuring frame-age p95 and bytes against MJPEG on real hardware still need doing.@ts-nocheck:video-player.jscarries// @ts-nocheckbecause WebCodecs and DOM types aren't in this project's lib.🤖 Generated with Claude Code
Closes #70