Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

workbenchthis machine

Cameras on hardware H.264: 20x fewer bytes, never a frozen frame (MJPEG fallback) #464

shipped merged opened this has it

Landed in pull #476 · run

opened this

Owner, 2026-10-21: "streaming is definitely something that needs to be done right… slow". Each MJPEG camera costs about 450-640 KB/s (measured on the Pi: 1.8-2.6 MB per 4 s).

What the steward set up on the Pi (live, keep it):

  • Both ustreamers have a hardware H.264 sink: --h264-sink=so101-cam<n> --h264-bitrate=600 --h264-gop=15, in /usr/local/bin/so101-camera (backup .bak-20261021b).
  • so101-h264@<n> (/usr/local/bin/so101-h264.mjs) serves raw Annex-B at http://100.89.95.68:809<n>/h264. Measured: about 28 KB/s per camera, about 20× less than MJPEG. The stream starts at 00 00 00 01.
  • The egress for :8091 and :8092 is open on all three layers: forge.json, the cell grant and the platform NetworkPolicy.

What broke: setting ARM_H264_1=480p=http://100.89.95.68:8091/h264 (and _2) on the app froze camera 1 on the page: a fresh page showed 1 distinct frame in 3 s. Both settings are unset again, so MJPEG is back.

Done when:

  • With ARM_H264_n set, the page plays H.264 on both cameras, starting on the next keyframe within 1 s, at 10-15 fps. Measure decoded fps on the page.
  • MJPEG is the automatic fallback when WebCodecs or the decode fails, or when no frame arrives for 2 s. A camera never freezes silently.
  • Find why it froze: no SPS/PPS before the first keyframe for a late joiner? The relay should cache the last SPS, PPS and IDR per camera. Or the rendition name, or the player choosing H.264 then never falling back.
  • A test that replays a captured Annex-B file (record 10 s from :8091) through the relay into the page's decoder path.
  • Report fps, bytes/s and glass-to-glass latency per camera in the #436 scorecard.
commented

Steward bug hunt, 2026-10-21. Three H.264 bugs found and fixed on a snapshot, each with a test. They conflict with H.264 code that has since landed on main, so they are yours to port:

  1. Frozen on one frame. The player stays marked live when the socket stays open but nothing decodes. Fix: after 2.5 s with no decode, report none so the page falls back to MJPEG.
  2. A late joiner can never configure its decoder. The encoder sends SPS and PPS only once. Fix: the fanout remembers each rung's SPS and PPS and prepends them to any IDR that lacks them.
  3. First load never used H.264. The page's boot camera list lacked h264 (this fix is in pull #491).

Also open: a late joiner waits up to one GOP for a key frame, longer than the client's 3 s waitMs, and then falls back to MJPEG for good. The Pi sends a GOP of 15 at 15 fps, so the wait is 1 s; keep it under waitMs. A real Chrome-encoded Annex-B fixture (base64) for tests is in pull #491's branch history (the hunter's branch). The Pi's H.264 sink is off again: adding --h264-sink to ustreamer dropped camera 1's MJPEG from 10 fps to 1. Find a configuration that keeps MJPEG at full rate before turning it back on (one encoder per camera? a lower H.264 resolution?).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs