Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

Cameras on hardware H.264: 20x fewer bytes, never a frozen frame (MJPEG fallback) #476

Merged opened this in answer to

Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.

Merged

With ARMH264n set, cameras now play H.264, and MJPEG takes over by itself whenever H.264 stops showing pictures. Nothing here has been checked against the real Pi: this sandbox is refused egress to 100.89.95.68, so I couldn't record a capture or measure fps, bytes/s or latency on live hardware. Why camera 1 froze.

Notes

shipped merged

opened this

With ARM_H264_n set, cameras now play H.264, and MJPEG takes over by itself whenever H.264 stops showing pictures. Nothing here has been checked against the real Pi: this sandbox is refused egress to 100.89.95.68, so I couldn't record a capture or measure fps, bytes/s or latency on live hardware.

Why camera 1 froze. Once the player had drawn its first picture it never fell back. A decoder error, or skipping ahead when it fell behind, left it waiting for a keyframe it could configure from. If the stream only carries SPS/PPS at its head, no such keyframe comes, so one picture stays on screen forever. I couldn't check whether the Pi's IDRs carry SPS/PPS, so all of these are fixed:

  • Relay (src/h264.ts, src/video-fanout.ts):

- Each camera's last SPS and PPS are cached and put back on any keyframe that arrives without them. - A viewer who joins gets the last keyframe straight away if it is under 2 s old, then plays from the next one. - A picture split into several slices now stays one unit. - The cache is cleared when a source drops.

  • Player (src/ui/video-player.js): it switches to MJPEG and says why when:

- no picture is drawn for 2 s, at the start or later; - the decoder fails 3 times with no picture between; - the browser can't configure the codec; - the socket closes.

A decoder that fails is rebuilt at the next keyframe. Timestamps always go up, so two pictures in the same millisecond don't collide.

  • Measurement:

- Once a second the page sends decoded fps, frame age and latency, and the live line shows live · H.264 14 fps. - The relay adds the bytes/s it pulls from the Pi. - Per camera, over the last 10 minutes: a new Video section on /admin and video in GET /precision. - Latency runs from the relay receiving a picture to the page drawing it, using a clock offset from pings. The hop from the camera through the Pi's encoder isn't included, because raw Annex-B carries no capture time.

  • Replay test (src/h264-replay.test.ts): a capture goes through a fake Pi at 15 fps, the real relay and the page's real player, into a fake decoder that rejects what a browser's would.

- Without a real capture it uses a synthetic stream shaped like the Pi's, with SPS/PPS only at the head. - It checks that a late joiner starts within 1 s at 10–15 fps with nothing rejected, and that a stopped stream switches to MJPEG within 2 s. - With the cache removed, the head-only case fails. - ARM_H264_CAPTURE=<file> runs it against a real capture; the README has the curl line to record one.

Choices I made:

  • After a fallback, H.264 stays off for that camera until the page is reloaded. Retrying would freeze the picture for 2 s each time it failed.
  • The video numbers live in memory, so a restart clears them.
  • The splitter still holds each picture until the next one starts, which adds one frame of delay.

Tests: 361 pass across the video, relay, admin, UI and README suites, plus the new route test. The type check shows the same 33 errors as main and none new. I changed two existing tests because the behaviour changed: the player's once-a-second report now includes fps, and the splitter needs one more byte before it releases a picture.

To try it: set ARM_H264_1=480p=http://100.89.95.68:8091/h264 and ARM_H264_2=480p=http://100.89.95.68:8092/h264 on the preview. Watch the live line and the Video section on /admin, then stop so101-h264@1 on the Pi: camera 1 should switch to MJPEG within about 2 s.

🤖 Generated with Claude Code

Closes #464

commented

With ARM_H264_n set, cameras now play H.264, and MJPEG takes over by itself whenever H.264 stops showing pictures. Nothing here has been checked against the real Pi: this sandbox is refused egress to 100.89.95.68, so I couldn't record a capture or measure fps, bytes/s or latency on live hardware.

Why camera 1 froze. Once the player had drawn its first picture it never fell back. A decoder error, or skipping ahead when it fell behind, left it waiting for a keyframe it could configure from. If the stream only carries SPS/PPS at its head, no such keyframe comes, so one picture stays on screen forever. I couldn't check whether the Pi's IDRs carry SPS/PPS, so all of these are fixed:

  • Relay (src/h264.ts, src/video-fanout.ts):

- Each camera's last SPS and PPS are cached and put back on any keyframe that arrives without them. - A viewer who joins gets the last keyframe straight away if it is under 2 s old, then plays from the next one. - A picture split into several slices now stays one unit. - The cache is cleared when a source drops.

  • Player (src/ui/video-player.js): it switches to MJPEG and says why when:

- no picture is drawn for 2 s, at the start or later; - the decoder fails 3 times with no picture between; - the browser can't configure the codec; - the socket closes.

A decoder that fails is rebuilt at the next keyframe. Timestamps always go up, so two pictures in the same millisecond don't collide.

  • Measurement:

- Once a second the page sends decoded fps, frame age and latency, and the live line shows live · H.264 14 fps. - The relay adds the bytes/s it pulls from the Pi. - Per camera, over the last 10 minutes: a new Video section on /admin and video in GET /precision. - Latency runs from the relay receiving a picture to the page drawing it, using a clock offset from pings. The hop from the camera through the Pi's encoder isn't included, because raw Annex-B carries no capture time.

  • Replay test (src/h264-replay.test.ts): a capture goes through a fake Pi at 15 fps, the real relay and the page's real player, into a fake decoder that rejects what a browser's would.

- Without a real capture it uses a synthetic stream shaped like the Pi's, with SPS/PPS only at the head. - It checks that a late joiner starts within 1 s at 10–15 fps with nothing rejected, and that a stopped stream switches to MJPEG within 2 s. - With the cache removed, the head-only case fails. - ARM_H264_CAPTURE=<file> runs it against a real capture; the README has the curl line to record one.

Choices I made:

  • After a fallback, H.264 stays off for that camera until the page is reloaded. Retrying would freeze the picture for 2 s each time it failed.
  • The video numbers live in memory, so a restart clears them.
  • The splitter still holds each picture until the next one starts, which adds one frame of delay.

Tests: 361 pass across the video, relay, admin, UI and README suites, plus the new route test. The type check shows the same 33 errors as main and none new. I changed two existing tests because the behaviour changed: the player's once-a-second report now includes fps, and the splitter needs one more byte before it releases a picture.

To try it: set ARM_H264_1=480p=http://100.89.95.68:8091/h264 and ARM_H264_2=480p=http://100.89.95.68:8092/h264 on the preview. Watch the live line and the Video section on /admin, then stop so101-h264@1 on the Pi: camera 1 should switch to MJPEG within about 2 s.

🤖 Generated with Claude Code

commented

The relay now caches each camera's last SPS, PPS and keyframe, so a viewer who joins late plays from the next keyframe. The player switches to MJPEG instead of freezing, and decoded fps, bytes/s and latency for each camera appear on the admin scorecard. Nothing here has been checked against the real Pi: this sandbox is refused egress to 100.89.95.68, so I couldn't record a capture or measure fps, bytes/s or latency on live hardware.

Why camera 1 froze. Once the player had drawn its first picture it never fell back. After a decoder error, or after skipping ahead when it fell behind, it waited for a keyframe it could configure from. If the stream carries SPS/PPS only at its head, that keyframe never comes and one picture stays on screen. I couldn't check whether the Pi's IDRs carry SPS/PPS, so all of these are fixed.

  • Relay (src/h264.ts, src/video-fanout.ts):

- Each camera's last SPS and PPS are cached and put back on any keyframe that arrives without them. - A viewer who joins gets the last keyframe straight away if it is under 2 s old, then plays from the next one. - A picture split into several slices stays one access unit. - The cache is cleared when a source drops.

  • Player (src/ui/video-player.js): it switches to MJPEG and says why in any of these cases:

- no picture is drawn for 2 s, at the start or later; - the decoder fails 3 times with no picture in between; - the browser can't configure the codec; - the socket closes.

A decoder that fails is rebuilt at the next keyframe. Timestamps always go up, so two pictures in the same millisecond don't collide.

  • Measurement:

- Once a second the page reports decoded fps, frame age and latency. The live line shows live · H.264 14 fps. - The relay adds the bytes/s it pulls from the Pi. - Per camera, over the last 10 minutes, the numbers appear in a new Video section on /admin and under video in GET /precision. - Latency runs from the relay receiving a picture to the page drawing it, using a clock offset taken from pings over the same socket. The hop from the camera through the Pi's encoder isn't included, because raw Annex-B carries no capture time.

  • Replay test (src/h264-replay.test.ts): a capture goes through a fake Pi at 15 fps, the real relay and the page's real player, into a fake decoder that rejects what a browser's decoder would.

- Without a real capture it uses a synthetic stream shaped like the Pi's, with SPS/PPS only at the head. - It checks that a late joiner starts within 1 s at 10–15 fps with nothing rejected, and that a stopped stream switches to MJPEG within 2 s. - With the cache removed, the head-only case fails. - ARM_H264_CAPTURE=<file> runs it against a real capture; the README has the curl line to record one.

Merge with main: the conflicts in admin-page.ts, admin-routes.ts and relay.ts keep both sides. The owner-only servo table from main and the Video section both render on /admin. askVideo comes after askServos, so main's callers don't change.

Choices I made:

  • After a fallback, H.264 stays off for that camera until the page is reloaded. Retrying would freeze the picture for 2 s each time it failed.
  • The video numbers are kept in memory, so a restart clears them.
  • The splitter still holds each picture until the next one starts, which adds one frame of delay.

Tests: after the merge, bun test passes 797 with 0 failing. The type check shows the same 33 errors as before; none are in the code this pull changes. Two existing tests changed because the behaviour changed:

  • the player's once-a-second report now includes fps;
  • the splitter needs one more byte before it releases a picture.

To try it: set ARM_H264_1=480p=http://100.89.95.68:8091/h264 and ARM_H264_2=480p=http://100.89.95.68:8092/h264 on the preview. Watch the live line and the Video section on /admin, then stop so101-h264@1 on the Pi: camera 1 should switch to MJPEG within about 2 s.

🤖 Generated with Claude Code

Approved

Approved. The player now leaves H.264 for MJPEG on 2 s of silence, three decoder failures, an unconfigurable codec or a closed socket, so it no longer holds a frozen frame. It states the reason, and the relay re-adds SPS/PPS and replays a fresh keyframe to late joiners. Page reports are validated in the fan-out tests, with the fallback reason whitelisted and reports rate-limited. /precision and the new Video section on /admin stay owner/moderator only: the preview answers 404 to an anonymous /precision, and / answers 200. Tests cover each new behaviour, including a replay through the real relay and player. I could not read src/video-stats.ts, which is cut off from my view of the diff, nor run the suite.

Two things to do next:

  1. Record a real capture from the Pi and run h264-replay.test.ts against it with ARM_H264_CAPTURE, because only a synthetic stream has been checked.
  2. Retry H.264 after a fallback, for example on the next keyframe-bearing reconnect, instead of staying on MJPEG until reload. A single transient stall currently costs the 20x saving for the whole session.
Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs