Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

workbenchthis machine

Real video: hardware H.264 from the Pi over the WebSocket, decoded with WebCodecs, smaller streams for slow viewers - a fraction of MJPEG's bytes #70

Closed opened this has it

Pull #207 was closed · 2 runs

opened this

Owner, 2026-10-20 22:45Z: "cant we do smart things about res and streaming a smaller packet? like youtube prob has some great tech for this".

Probe (steward, the Pi): Raspberry Pi 4 Model B, 4 cores, 7.7 GB; hardware H.264 encoder at /dev/video11 (bcm2835-codec-encode) plus ISP /dev/video12; the USB camera offers MJPG and YUYV; today so101-camera@1 runs ustreamer --resolution=1280x720 --format=MJPEG --desired-fps=20 --drop-same-frames=3 on :8080. MJPEG sends every frame as a full JPEG — roughly 5-10x the bytes of H.264 at the same quality.

YouTube's DASH/HLS adaptive streaming buffers 2-10 s — right for watching, wrong for driving. The low-latency equivalent (what cloud gaming and WebRTC do) is a short-GOP H.264 stream with per-viewer adaptation; through the Cloudflare edge (TCP only) the practical transport is the app's one WebSocket (#54) + WebCodecs in the browser.

Done:

  1. Pi: H.264 from the hardware encoder — ustreamer's H.264 sink (--h264-sink, M2M encoder) or ffmpeg -c:v h264_v4l2m2m; baseline/main profile, GOP ≤ 1 s, no B-frames, ~1.5 Mb/s at 720p20. Keep MJPEG as the fallback. (Pi changes go through the crew's SSH once trevato/forge#3470 lands; until then the steward applies them on request.)
  2. App: one upstream H.264 stream from the Pi, fanned out over the viewer WebSocket as Annex-B access units with timestamps; the browser decodes with WebCodecs VideoDecoder (Chrome, Edge, Safari 16.4+, Firefox 130+) into a canvas; falls back to the MJPEG relay where WebCodecs is missing.
  3. Smaller packets for slow viewers — a ladder: 720p20 (~1.5 Mb/s), 480p15 (~600 kb/s), 240p10 (~200 kb/s), via the Pi's ISP/encoder or a server-side transcode only if needed; each viewer is switched by measured throughput and frame age (switch at a keyframe; request an IDR on switch); a "quality" chip lets the viewer pin one; the driver gets priority on the best rendition.
  4. Latency first: never buffer — a slow viewer skips to the next keyframe instead of queueing; show frame age (capture → shown) live (#54 metrics). Targets: frame age p95 < 250 ms on home wifi, < 500 ms on 4G, 0 freezes in 10 min; bytes per viewer reported before/after vs MJPEG.
  5. Tests: a fake H.264 source (pre-recorded clip) — two viewers at different simulated bandwidths get different renditions; a slow viewer skips to keyframes; a browser test decodes frames via WebCodecs.
commented

Parked by the crew: the check stayed red through 2 automatic tries (pull #75 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

commented

Split into single-pull pieces, all reusing pull #207's branch:

  • #257: Annex-B splitter and per-viewer fanout
  • #258: the WebCodecs player
  • #259: the relay route, /cameras h264, egress and README
  • #260: the page pane, the quality select and the fallback
  • #262: the Pi H.264 recipe
  • #263: frame age and bytes measured against the targets

Pull #207 was handed back after three change requests, and this issue had already been parked once (pull #75). Every point in this issue's done list is in one of the pieces.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs