Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

workbenchthis machine

Cameras on hardware H.264: 20x fewer bytes, never a frozen frame (MJPEG fallback) #476

Merged opened this in answer to

Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.

Merged

vesper/arm:vesper/464-cameras-on-hardware-h-264-20x-fewer-byte → main 4 commits fork

  • Merged. vesper/464-cameras-on-hardware-h-264-20x-fewer-byte was merged into main as 97878ae.

Review

Sign in to review this change.
18 files changed +911 −63
Software/README.md +45 −5
@@ -247,10 +247,40 @@ rendition per viewer. A viewer whose connection backs up steps down the ladder
247247 and, rather than queueing, skips to the next keyframe; the **quality selector**
248248 on the picture lists the configured rungs and pins one, or leaves it on Auto. The page decodes with
249249 WebCodecs, loading the player only when `/cameras` reports `"h264": true` for
250 the camera. When the browser has no WebCodecs, or no picture arrives in time,
251 the page falls back to MJPEG (`/camera/<id>/stream`) by itself. The video
252 socket is open to guests, like `/stream`; the driver is given the best
253 rendition. Control messages from the viewer are capped at 200 bytes.
250 the camera. The page falls back to MJPEG (`/camera/<id>/stream`) by itself,
251 and never holds a frozen picture: when the browser has no WebCodecs or cannot
252 configure the codec, when no picture is drawn for 2 s (at the start or at any
253 time after), when the decoder fails three times with no picture between, or
254 when the socket closes. The live line under the picture shows the decoded
255 frame rate (`live · H.264 14 fps`). The video socket is open to guests, like
256 `/stream`; the driver is given the best rendition. Control messages from the
257 viewer are capped at 200 bytes.
258
259 *Starting fast, and starting at all.* The relay keeps each rendition's last
260 SPS and PPS and puts them on every keyframe that comes without them, so a
261 viewer who joins late, or a decoder that has to start over after an error,
262 can always configure from the next keyframe; an encoder that sends them only
263 at the head of its stream used to leave a late joiner on one picture. A viewer
264 who joins is also sent the rendition's last keyframe at once (when it is under
265 2 s old), so a picture is up before the next keyframe comes. Pictures cut into
266 several slices stay one access unit (`src/h264.ts`).
267
268 *Measured.* Once a second each page tells the relay the frames it drew, the
269 frame age and the latency from the relay to the glass (on the relay's clock:
270 the page keeps the offset from pings over the same socket); a page that falls
271 back says why. The relay adds the bytes a second it pulls from the Pi and
272 shows all of it per camera, over the last 10 minutes, in the **Video** section
273 of `/admin` beside the precision scorecard, and under `video` in
274 `GET /precision`. The hop from the camera's glass through the Pi's encoder to
275 the relay is not in the latency: raw Annex-B carries no capture time. The
276 numbers are kept in memory, so a restart starts them over.
277
278 *Tested against a capture.* `src/h264-replay.test.ts` replays an Annex-B file
279 through a fake Pi, the relay and the page's own player into a decoder that
280 refuses what a browser's would. It uses a synthetic stream of the Pi's shape
281 unless `ARM_H264_CAPTURE` names a capture, recorded on the Pi's network with
282 `curl -s --max-time 10 http://100.89.95.68:8091/h264 > cam1.h264`, then
283 `ARM_H264_CAPTURE=$PWD/cam1.h264 bun test src/h264-replay.test.ts`.
254284
255285 *Not in this pull.* The Pi's encoder side: something on the Pi must serve each
256286 rendition as an H.264 stream at the URL you name (hardware encoder, one port
@@ -258,9 +288,19 @@ per rung). Without it the relay's fetch fails and viewers stay on MJPEG.
258288
259289 *The forge's egress list.* The relay reaches the Pi through the forge's egress
260290 proxy (`proxyFor` in `src/egress.ts`, the same as the other camera paths), so every rendition's `host:port` must be in `forge.json`'s `egress`.
261 It lists 100.89.95.68 on 8080, 8081, 8101 and 8082 to 8084 (the example ports above);
291 It lists 100.89.95.68 on 8080, 8081, 8101, 8082 to 8084 (the example ports above)
292 and 8091 and 8092 (ustreamer's own H.264 sink, one per camera, below);
262293 if your encoder listens elsewhere, add those ports.
263294
295 *ustreamer's H.264 sink, both cameras.* On the Pi each ustreamer also feeds a
296 hardware H.264 sink (`--h264-sink=so101-cam<n> --h264-bitrate=600
297 --h264-gop=15`), served as raw Annex-B by `so101-h264@<n>` at
298 `http://100.89.95.68:809<n>/h264`: about 28 KB/s a camera against 450-640 KB/s
299 for MJPEG. It is one rendition, so it goes on the 480p rung:
300
301 ARM_H264_1=480p=http://100.89.95.68:8091/h264
302 ARM_H264_2=480p=http://100.89.95.68:8092/h264
303
264304 *The terminal (`ssh`) shows no video, and has no quality choice:* both are page-only.
265305
266306 **In the browser.** Behind the relay the page draws each camera from
Software/so101/src/admin-page.ts +4
@@ -13,6 +13,7 @@ import type { Snapshot } from "./turns.ts";
1313 import type { ViewerRow } from "./viewer-reports.ts";
1414 import { precisionSection, type PrecisionPanel } from "./precision-panel.ts";
1515 import { servosSection, type ServoPanel } from "./servo-panel.ts";
16 import { videoSection, type VideoCard } from "./video-panel.ts";
1617
1718 export const esc = (text: string): string => text.replace(/[&<>"']/g, (c) => `&#${c.charCodeAt(0)};`);
1819
@@ -158,6 +159,8 @@ export interface AdminPage {
158159 precision?: PrecisionPanel | null;
159160 /** Owner only: the servos' control registers (src/servo-panel.ts); null when the arm did not answer, absent when not asked. */
160161 servos?: ServoPanel | null;
162 /** The cameras' H.264 numbers, beside it (src/video-panel.ts); absent when not asked. */
163 video?: readonly VideoCard[];
161164 done?: Done | null;
162165 error?: string | null;
163166 }
@@ -283,6 +286,7 @@ ${p.canEdit ? `<section aria-labelledby="table"><h2 id="table">Table height</h2>
283286 ${p.canEdit && p.visitors ? visitorsSection(p.visitors) : ""}
284287 ${p.precision !== undefined ? precisionSection(p.precision, esc) : ""}
285288 ${p.canEdit && p.servos !== undefined ? servosSection(p.servos, esc) : ""}
289 ${p.video !== undefined ? videoSection(p.video, esc) : ""}
286290 <section aria-labelledby="health"><h2 id="health">Health</h2>${health(p.health ?? [])}</section>
287291 <section aria-labelledby="mods"><h2 id="mods">Moderators</h2>${mods}</section>
288292 <section aria-labelledby="moderation"><h2 id="moderation">Moderation</h2>${inForce("mute", p.muted ?? [], canLift)}${inForce("ban", p.banned ?? [], canLift)}${actions}</section>
Software/so101/src/admin-routes.ts +8 −4
@@ -7,9 +7,12 @@ import { sameOrigin } from "./relay.ts";
77 import { adminCsp, DONE, renderAdmin, renderRefusal, type Done } from "./admin-page.ts";
88 import type { PrecisionPanel } from "./precision-panel.ts";
99 import type { ServoPanel } from "./servo-panel.ts";
10 import type { VideoCard } from "./video-panel.ts";
1011
1112 /** Asks the arm for its precision scorecard; only called for someone allowed on the page. */
1213 export type AskPrecision = () => Promise<PrecisionPanel | null>;
14 /** The relay's own numbers for its H.264 cameras (src/video-stats.ts). */
15 export type AskVideo = () => readonly VideoCard[];
1316
1417 /** Asks the arm for its servos' control registers; only called for the owner. */
1518 export type AskServos = () => Promise<ServoPanel | null>;
@@ -28,7 +31,7 @@ export interface AdminSession {
2831 * Answer 404 to anyone not allowed, and the page itself to GET and HEAD. Returns the session for any other
2932 * method by an allowed caller, and null when the response has been sent.
3033 */
31 export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, askPrecision?: AskPrecision, askServos?: AskServos): Promise<AdminSession | null> {
34 export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo): Promise<AdminSession | null> {
3235 const who = lobby.who(req);
3336 const html = (status: number, text: string, edit = false) => {
3437 res.writeHead(status, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store", "content-security-policy": adminCsp(edit) });
@@ -53,8 +56,9 @@ export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: Se
5356 }
5457 const precision = (req.method === "GET" || req.method === "HEAD") && askPrecision ? await askPrecision() : undefined;
5558 const servos = canEdit && (req.method === "GET" || req.method === "HEAD") && askServos ? await askServos() : undefined;
59 const video = askVideo?.();
5660 const show = (status: number, done: Done | null, error: string | null) =>
57 html(status, renderAdmin({ precision, servos, who, canEdit, settings: lobby.admin.settings, snapshot: lobby.turns.snapshot(), log: lobby.admin.log, removed: lobby.chat.removed, muted: lobby.chat.mutedList(), banned: lobby.accounts.bans(), health: lobby.reports.rows(), reports: lobby.lineReports.list(), visitors: canEdit ? { peak: lobby.visitors.peakToday(), hours: lobby.visitors.joinsPerHour() } : undefined, done, error }), canEdit);
61 html(status, renderAdmin({ precision, servos, video, who, canEdit, settings: lobby.admin.settings, snapshot: lobby.turns.snapshot(), log: lobby.admin.log, removed: lobby.chat.removed, muted: lobby.chat.mutedList(), banned: lobby.accounts.bans(), health: lobby.reports.rows(), reports: lobby.lineReports.list(), visitors: canEdit ? { peak: lobby.visitors.peakToday(), hours: lobby.visitors.joinsPerHour() } : undefined, done, error }), canEdit);
5862 if (req.method === "GET" || req.method === "HEAD") {
5963 const done = url.searchParams.get("done");
6064 show(200, done && Object.hasOwn(DONE, done) ? (done as Done) : null, null);
@@ -73,8 +77,8 @@ const REPORT_ACTIONS = new Set(["report-remove", "report-mute", "report-dismiss"
7377 type ReadBody = (req: IncomingMessage, limit?: number) => Promise<Buffer>;
7478
7579 /** The owner's page. Anyone else is told so; a change is a form post, answered with a redirect back. */
76 export async function handleAdmin(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, readBody: ReadBody, askPrecision?: AskPrecision, askServos?: AskServos): Promise<void> {
77 const session = await serveAdminPage(lobby, req, res, url, askPrecision, askServos);
80 export async function handleAdmin(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, readBody: ReadBody, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo): Promise<void> {
81 const session = await serveAdminPage(lobby, req, res, url, askPrecision, askServos, askVideo);
7882 if (!session) return;
7983 const { who, canEdit, show } = session;
8084 if (req.method !== "POST") {
Software/so101/src/h264-clip.ts +19 −6
@@ -1,16 +1,29 @@
11 /** A stand-in for the Pi's encoder in tests: a short Annex-B clip with the shape of the real one (SPS, PPS, IDR, then P pictures; no B-frames). */
22
3 const nal = (type: number, size: number, fill: number) => Buffer.concat([Buffer.from([0, 0, 0, 1, type | 0x60]), Buffer.alloc(size, fill)]);
3 // A slice's payload opens with first_mb_in_slice = 0, coded as the single bit 1: the top bit of its first byte.
4 const nal = (type: number, size: number, fill: number) => Buffer.concat([Buffer.from([0, 0, 0, 1, type | 0x60, 0x80 | fill]), Buffer.alloc(Math.max(0, size - 1), fill)]);
45 // SPS body: profile 66 (baseline), constraints 0xc0, level 31, then filler that contains no start code.
56 const SPS = Buffer.concat([Buffer.from([0, 0, 0, 1, 0x67, 66, 0xc0, 31]), Buffer.alloc(6, 0x11)]);
67 const PPS = nal(8, 3, 0x22);
8 const AUD = Buffer.from([0, 0, 0, 1, 0x09, 0xf0]);
79
8 /** One picture: `key` pictures carry the parameter sets; `size` is the slice's payload in bytes. */
9 export function picture(key: boolean, size: number): Buffer {
10 return key ? Buffer.concat([SPS, PPS, nal(5, size, 0x33)]) : nal(1, size, 0x44);
10 export interface Shape {
11 /** Parameter sets on every keyframe (the default), or only on the first picture of the stream. */
12 sets?: "every-key" | "head";
13 /** An access unit delimiter before every picture. */
14 aud?: boolean;
15 }
16
17 /** One picture: `key` pictures carry the parameter sets unless `sets` is false; `size` is the slice's payload in bytes. */
18 export function picture(key: boolean, size: number, sets = true, aud = false): Buffer {
19 const body = key ? Buffer.concat([...(sets ? [SPS, PPS] : []), nal(5, size, 0x33)]) : nal(1, size, 0x44);
20 return aud ? Buffer.concat([AUD, body]) : body;
1121 }
1222
1323 /** `seconds` of video at `fps` with a keyframe every `gop` pictures; keyframes are 8x a P picture's size. */
14 export function clip(seconds: number, fps: number, gop: number, deltaBytes: number): Buffer[] {
15 return Array.from({ length: Math.round(seconds * fps) }, (_, i) => picture(i % gop === 0, i % gop === 0 ? deltaBytes * 8 : deltaBytes));
24 export function clip(seconds: number, fps: number, gop: number, deltaBytes: number, shape: Shape = {}): Buffer[] {
25 return Array.from({ length: Math.round(seconds * fps) }, (_, i) => {
26 const key = i % gop === 0;
27 return picture(key, key ? deltaBytes * 8 : deltaBytes, shape.sets === "head" ? i === 0 : true, shape.aud === true);
28 });
1629 }
Software/so101/src/h264-replay.test.ts added +173
@@ -0,0 +1,173 @@
1 /**
2 * A captured Annex-B stream replayed through the relay into the page's decoder path: a Pi that serves the file at
3 * 15 pictures a second, the relay as it runs (startRelay with ARM_H264_1), and the page's own VideoPlayer
4 * (src/ui/video-player.js) on a real WebSocket, decoding into a strict stand-in for WebCodecs that refuses what a
5 * browser's decoder would: a first keyframe without SPS and PPS, a delta before any keyframe, half a picture, a
6 * timestamp that does not rise.
7 *
8 * The capture: ARM_H264_CAPTURE=<file> names one, recorded on the Pi's network with
9 * curl -s --max-time 10 http://100.89.95.68:8091/h264 > cam1.h264
10 * Without it, a synthetic one of the same shape is used: 480p-sized pictures at about 28 KB/s, a keyframe every 15,
11 * an AUD before each, and SPS and PPS only at the head of the stream, the case that froze a late joiner.
12 */
13 import { afterEach, describe, expect, test } from "bun:test";
14 import { mkdtempSync, readFileSync, rmSync } from "node:fs";
15 import { createServer, type Server } from "node:http";
16 import { tmpdir } from "node:os";
17 import { join } from "node:path";
18 import WebSocket from "ws";
19 import { CameraHub } from "./cameras.ts";
20 import { clip, type Shape } from "./h264-clip.ts";
21 import { AccessUnitSplitter, nalsOf } from "./h264.ts";
22 import { Lobby } from "./lobby.ts";
23 import { startRelay, type RunningRelay } from "./relay.ts";
24 import { VideoPlayer } from "./ui/video-player.js";
25
26 const FPS = 15;
27 const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
28
29 /** The capture's pictures, in order: the file cut into access units, the last one included. */
30 function pictures(shape: Shape): Buffer[] {
31 const file = process.env.ARM_H264_CAPTURE;
32 if (!file) return clip(10, FPS, 15, 1000, shape);
33 const data = readFileSync(file);
34 const units = new AccessUnitSplitter().push(Buffer.concat([data, Buffer.from([0, 0, 0, 1, 0x09, 0xf0])])); // an AUD closes the last
35 return units.map((u) => u.data);
36 }
37
38 let dir = "";
39 let relay: RunningRelay | undefined;
40 let lobby: Lobby | undefined;
41 let pi: Server | undefined;
42 const players: VideoPlayer[] = [];
43 afterEach(async () => {
44 players.splice(0).forEach((p) => p.stop());
45 await Promise.race([relay?.close(), sleep(500)]);
46 lobby?.close();
47 pi?.closeAllConnections();
48 pi?.close();
49 if (dir) rmSync(dir, { recursive: true, force: true });
50 dir = "";
51 relay = lobby = pi = undefined;
52 });
53
54 /** The Pi: each connection gets the capture from `from`, a picture every 1/FPS s, looping; `stop()` ends every stream. */
55 async function boot(list: Buffer[], from = 0) {
56 const open = new Set<() => void>();
57 pi = createServer((req, res) => {
58 if (req.url !== "/h264") return void res.writeHead(404).end();
59 res.writeHead(200, { "content-type": "video/h264" });
60 let i = from;
61 const timer = setInterval(() => res.write(list[i++ % list.length]!), 1000 / FPS);
62 const end = () => (clearInterval(timer), res.destroy());
63 open.add(end);
64 res.on("close", () => (clearInterval(timer), open.delete(end)));
65 });
66 await new Promise<void>((r) => pi!.listen(0, "127.0.0.1", r));
67 const port = (pi.address() as { port: number }).port;
68 dir = mkdtempSync(join(tmpdir(), "so101-replay-"));
69 lobby = new Lobby({ identityKey: "k".repeat(64), adminFile: join(dir, "admin.json") });
70 const cameras = new CameraHub({ seed: [{ name: "One", source: "http://127.0.0.1:9/stream" }] });
71 relay = await startRelay({ port: 0, simulation: "http://127.0.0.1:1", lobby, cameras, video: { 1: { "480p": `http://127.0.0.1:${port}/h264` } }, env: {} });
72 return { url: `ws://127.0.0.1:${relay.port}/camera/1/video`, stop: () => [...open].forEach((end) => end()) };
73 }
74
75 /** A decoder that holds the stream to what a browser's would accept, and draws each picture at once. */
76 function strictDecoder() {
77 const seen = { pictures: 0, refused: [] as string[], configs: [] as string[], drawnAt: [] as number[] };
78 class Decoder {
79 decodeQueueSize = 0;
80 private sets = false;
81 private key = false;
82 private last = -Infinity;
83 private readonly init: { output: (f: unknown) => void; error: (e: unknown) => void };
84 constructor(init: { output: (f: unknown) => void; error: (e: unknown) => void }) {
85 this.init = init;
86 }
87 configure(c: { codec: string }) {
88 if (!/^avc1\.[0-9a-f]{6}$/.test(c.codec)) throw new TypeError(`codec ${c.codec}`);
89 seen.configs.push(c.codec);
90 }
91 decode(chunk: { type: string; timestamp: number; data: Uint8Array }) {
92 const types = nalsOf(chunk.data).map((n) => n.type);
93 const why =
94 chunk.timestamp <= this.last ? "timestamp did not rise"
95 : chunk.type === "key" && !this.sets && !(types.includes(7) && types.includes(8)) ? "first keyframe without SPS and PPS"
96 : chunk.type === "delta" && !this.key ? "delta before a keyframe"
97 : types.filter((t) => t === 1 || t === 5).length !== 1 ? "not one picture"
98 : null;
99 if (why) return (seen.refused.push(why), this.init.error(new Error(why)));
100 this.last = chunk.timestamp;
101 this.sets ||= types.includes(7);
102 this.key ||= chunk.type === "key";
103 seen.pictures++;
104 seen.drawnAt.push(Date.now());
105 this.init.output({ displayWidth: 854, displayHeight: 480, timestamp: chunk.timestamp, close() {} });
106 }
107 close() {}
108 }
109 return { Decoder, seen };
110 }
111
112 /** The page's player on `url`, with the strict decoder; its states, and when it started. */
113 function watch(url: string, quietMs?: number) {
114 const { Decoder, seen } = strictDecoder();
115 const states: { state: string; why?: string }[] = [];
116 const canvas = { width: 0, height: 0, getContext: () => ({ drawImage() {} }) };
117 const player = new VideoPlayer({ url, canvas, quietMs, onStatus: (s) => states.push(s), deps: { WebSocket, VideoDecoder: Decoder, EncodedVideoChunk: class { constructor(init: object) { Object.assign(this, init); } } } });
118 const started = Date.now();
119 player.start();
120 players.push(player);
121 return { player, seen, states, started };
122 }
123
124 /** Pictures a second over the drawn times after the first. */
125 const fps = (at: number[]) => (at.length < 2 ? 0 : ((at.length - 1) * 1000) / (at.at(-1)! - at[0]!));
126
127 describe("a capture replayed through the relay into the page's decoder", () => {
128 for (const [name, shape, from] of [
129 ["SPS and PPS only at the head of the stream", { sets: "head", aud: true }, 0],
130 ["SPS and PPS on every keyframe, the relay joining the Pi's stream mid-GOP", { sets: "every-key" }, 7],
131 ] as const) {
132 test(`${name}: a late joiner plays from the next keyframe within 1 s, at 10-15 fps, nothing refused`, async () => {
133 const list = pictures(shape);
134 const { url } = await boot(list, from);
135 const first = watch(url); // pulls the Pi
136 await sleep(1300); // the second joins mid-GOP, long after the head of the stream went by
137 const late = watch(url);
138 await sleep(2200);
139 for (const v of [first, late]) {
140 expect(v.seen.refused).toEqual([]);
141 expect(v.states.map((s) => s.state)).toEqual(["live"]);
142 expect(v.seen.pictures).toBeGreaterThan(15);
143 expect(fps(v.seen.drawnAt.slice(1))).toBeGreaterThanOrEqual(10); // the first may be the cached keyframe, drawn before the stream catches up
144 expect(fps(v.seen.drawnAt.slice(1))).toBeLessThanOrEqual(16);
145 }
146 expect(late.seen.drawnAt[0]! - late.started).toBeLessThan(1000);
147 expect(late.seen.configs).toEqual(["avc1.42c01f"]);
148 // What the pages measured reaches the scorecard: decoded fps and the bytes from the Pi.
149 late.player.report();
150 await sleep(100);
151 const card = relay!.videoCards()[0]!;
152 expect(card.camera).toBe(1);
153 expect(card.viewers).toBe(2);
154 expect(card.fpsP50).toBeGreaterThanOrEqual(10);
155 expect(card.bytesPerS).toBeGreaterThan(5_000);
156 }, 10_000);
157 }
158
159 test("when the Pi's stream stops, the page goes back to MJPEG within 2 s instead of holding the last picture", async () => {
160 const { url, stop } = await boot(pictures({ sets: "head", aud: true }));
161 const v = watch(url, 600);
162 await sleep(800);
163 expect(v.states.map((s) => s.state)).toEqual(["live"]);
164 pi!.close(); // and it does not come back
165 stop();
166 const stopped = Date.now();
167 while (v.states.length < 2 && Date.now() - stopped < 2500) await sleep(20);
168 expect(v.states.at(-1)).toEqual(expect.objectContaining({ state: "none", why: "quiet" }));
169 expect(Date.now() - stopped).toBeLessThan(2000);
170 await sleep(100);
171 expect(relay!.videoCards()[0]!.fallbacks.quiet).toBe(1);
172 }, 10_000);
173 });
Software/so101/src/h264.test.ts +46 −4
@@ -1,6 +1,6 @@
11 import { describe, expect, test } from "bun:test";
2 import { clip } from "./h264-clip.ts";
3 import { AccessUnitSplitter } from "./h264.ts";
2 import { clip, picture } from "./h264-clip.ts";
3 import { AccessUnitSplitter, nalsOf, ParameterSets } from "./h264.ts";
44
55 describe("access units", () => {
66 const pictures = clip(1, 10, 5, 100);
@@ -20,10 +20,25 @@ describe("access units", () => {
2020 }
2121 });
2222
23 test("a picture is out as soon as the next one begins, not when it ends", () => {
23 test("a picture is out as soon as the next one begins (its start code, header and first byte), not when it ends", () => {
2424 const splitter = new AccessUnitSplitter();
2525 expect(splitter.push(pictures[0]!)).toEqual([]);
26 expect(splitter.push(pictures[1]!.subarray(0, 5))).toHaveLength(1);
26 expect(splitter.push(pictures[1]!.subarray(0, 5))).toEqual([]);
27 expect(splitter.push(pictures[1]!.subarray(5, 6))).toHaveLength(1);
28 });
29
30 test("a picture cut into several slices stays one unit: only a slice with first_mb_in_slice 0 opens a picture", () => {
31 const second = Buffer.from([0, 0, 0, 1, 0x61, 0x20, 0x44, 0x44]); // a slice of the same P picture: first_mb_in_slice > 0
32 const stream = Buffer.concat([pictures[0]!, pictures[1]!, second, pictures[2]!, pictures[3]!]);
33 const units = new AccessUnitSplitter().push(stream);
34 expect(units.map((u) => u.data.length)).toEqual([pictures[0]!.length, pictures[1]!.length + second.length, pictures[2]!.length]);
35 });
36
37 test("access unit delimiters open a picture too", () => {
38 const aud = clip(1, 10, 5, 50, { aud: true });
39 const units = new AccessUnitSplitter().push(Buffer.concat(aud));
40 expect(units.map((u) => u.data.equals(aud[units.indexOf(u)]!))).toEqual(aud.slice(0, -1).map(() => true));
41 expect(units.map((u) => u.key)).toEqual(aud.slice(0, -1).map((_, i) => i % 5 === 0));
2742 });
2843
2944 test("refuses a stream that never ends a unit", () => {
@@ -33,3 +48,30 @@ describe("access units", () => {
3348 }).toThrow();
3449 });
3550 });
51
52 describe("parameter sets", () => {
53 const types = (b: Buffer) => nalsOf(b).map((n) => n.type);
54
55 test("a keyframe without SPS and PPS gets the last ones seen, after its AUD; one that has them is left alone", () => {
56 const sets = new ParameterSets();
57 const head = picture(true, 80, true, true);
58 expect(sets.complete({ key: true, data: head }).data.equals(head)).toBe(true);
59 expect(sets.known).toBe(true);
60 const bare = picture(true, 80, false, true);
61 expect(types(bare)).toEqual([9, 5]);
62 const done = sets.complete({ key: true, data: bare });
63 expect(types(done.data)).toEqual([9, 7, 8, 5]);
64 expect(done.data.equals(head)).toBe(true);
65 const delta = picture(false, 20);
66 expect(sets.complete({ key: false, data: delta }).data).toBe(delta);
67 });
68
69 test("before any SPS a bare keyframe goes as it is; new sets replace the old", () => {
70 const sets = new ParameterSets();
71 const bare = picture(true, 80, false);
72 expect(sets.complete({ key: true, data: bare }).data).toBe(bare);
73 const other = Buffer.concat([Buffer.from([0, 0, 0, 1, 0x67, 100, 0, 40, 0x11]), Buffer.from([0, 0, 0, 1, 0x68, 0xee]), picture(true, 10, false)]);
74 sets.complete({ key: true, data: other });
75 expect([...sets.complete({ key: true, data: bare }).data.subarray(0, 8)]).toEqual([0, 0, 0, 1, 0x67, 100, 0, 40]);
76 });
77 });
Software/so101/src/h264.ts +62 −8
@@ -1,9 +1,11 @@
11 /**
22 * H.264 in Annex-B (start-code) form, cut into access units as the bytes arrive.
33 *
4 * An access unit is one picture: optional parameter sets (SPS/PPS), then its slices. The encoder on the Pi sends one
5 * slice per picture and no B-frames, so a picture ends the moment the next NAL that opens one begins: the unit
6 * leaves as soon as the next start code and its header byte are seen, never waiting for the next picture to finish.
4 * An access unit is one picture: optional parameter sets (SPS/PPS), then its slices. The encoder on the Pi sends no
5 * B-frames, so a picture ends the moment the next NAL that opens one begins: an AUD, SEI or parameter set, or a slice
6 * whose first_mb_in_slice is 0 (the top bit of its first byte), so a picture cut into several slices stays one unit.
7 * The unit leaves as soon as the next start code and the byte after its header are seen, never waiting for the next
8 * picture to finish.
79 */
810
911 export interface AccessUnit {
@@ -14,7 +16,8 @@ export interface AccessUnit {
1416 }
1517
1618 /** NAL types that open a new access unit once a picture's slice has been seen. */
17 const OPENS = new Set([1, 5, 6, 7, 8, 9]);
19 const OPENS = new Set([6, 7, 8, 9]);
20 const isSlice = (type: number) => type === 1 || type === 5;
1821 /** No unit is larger than this; a stream that never finds a start code is cut off by throwing. */
1922 export const MAX_UNIT = 4 << 20;
2023
@@ -31,23 +34,24 @@ export class AccessUnitSplitter {
3134 this.buf = this.buf.length ? Buffer.concat([this.buf, chunk]) : Buffer.from(chunk);
3235 const out: AccessUnit[] = [];
3336 const b = this.buf;
34 for (let i = this.scan; i + 3 < b.length; i++) {
37 for (let i = this.scan; i + 4 < b.length; i++) {
3538 if (b[i + 2]! > 1) i += 2; // cannot be inside a start code
3639 else if (b[i] === 0 && b[i + 1] === 0 && b[i + 2] === 1) {
3740 const type = b[i + 3]! & 0x1f;
3841 const start = i > this.from && b[i - 1] === 0 ? i - 1 : i;
39 if (this.picture && OPENS.has(type) && start > this.from) {
42 const opens = OPENS.has(type) || (isSlice(type) && (b[i + 4]! & 0x80) !== 0);
43 if (this.picture && opens && start > this.from) {
4044 out.push({ key: this.key, data: Buffer.from(b.subarray(this.from, start)) });
4145 this.from = start;
4246 this.picture = this.key = false;
4347 }
44 if (type === 1 || type === 5) this.picture = true;
48 if (isSlice(type)) this.picture = true;
4549 if (type === 5) this.key = true;
4650 i += 3;
4751 this.scan = i + 1;
4852 }
4953 }
50 this.scan = Math.max(this.scan, b.length - 3);
54 this.scan = Math.max(this.scan, b.length - 4);
5155 if (this.from > 0) {
5256 this.buf = Buffer.from(b.subarray(this.from));
5357 this.scan -= this.from;
@@ -57,3 +61,53 @@ export class AccessUnitSplitter {
5761 return out;
5862 }
5963 }
64
65 export interface Nal {
66 type: number;
67 /** Where its start code begins, and where the next one does (or the end). */
68 start: number;
69 end: number;
70 }
71
72 /** The NAL units of one Annex-B access unit, in order. */
73 export function nalsOf(data: Uint8Array): Nal[] {
74 const out: Nal[] = [];
75 for (let i = 0; i + 3 < data.length; i++) {
76 if (data[i + 2]! > 1) i += 2;
77 else if (data[i] === 0 && data[i + 1] === 0 && data[i + 2] === 1) {
78 const start = i > 0 && data[i - 1] === 0 ? i - 1 : i;
79 if (out.length) out.at(-1)!.end = start;
80 out.push({ type: data[i + 3]! & 0x1f, start, end: data.length });
81 i += 3;
82 }
83 }
84 return out;
85 }
86
87 /**
88 * One rendition's parameter sets, kept from wherever the encoder last sent them, so that every keyframe a viewer can
89 * start on carries an SPS and a PPS. An encoder that sends them only at the head of its stream (or a server that
90 * passes a late joiner the stream from mid-way) otherwise leaves a viewer that joined later, or a decoder that had to
91 * start over after an error, with IDR pictures it cannot configure from: one picture, then a frozen one.
92 */
93 export class ParameterSets {
94 private sps: Buffer | null = null;
95 private pps: Buffer | null = null;
96
97 get known(): boolean {
98 return this.sps !== null && this.pps !== null;
99 }
100
101 /** Learn from `unit` and, when it is a keyframe without both of its own, give it the last ones seen. */
102 complete(unit: AccessUnit): AccessUnit {
103 const nals = nalsOf(unit.data);
104 const sps = nals.filter((n) => n.type === 7), pps = nals.filter((n) => n.type === 8);
105 const take = (list: Nal[]) => Buffer.concat(list.map((n) => unit.data.subarray(n.start, n.end)));
106 if (sps.length) this.sps = take(sps);
107 if (pps.length) this.pps = take(pps);
108 if (!unit.key || (sps.length && pps.length) || !this.known) return unit;
109 // AUD first, then SPS and PPS, then the rest: the order a decoder reads them in.
110 const part = (keep: (type: number) => boolean) => take(nals.filter((n) => keep(n.type)));
111 return { key: true, data: Buffer.concat([part((t) => t === 9), this.sps!, this.pps!, part((t) => t !== 9 && t !== 7 && t !== 8)]) };
112 }
113 }
Software/so101/src/relay-video.test.ts +16
@@ -6,6 +6,7 @@ import { tmpdir } from "node:os";
66 import { join } from "node:path";
77 import WebSocket from "ws";
88 import { CameraHub } from "./cameras.ts";
9 import { sign } from "./identity.ts";
910 import { clip } from "./h264-clip.ts";
1011 import { Lobby } from "./lobby.ts";
1112 import { startRelay, type RunningRelay } from "./relay.ts";
@@ -91,3 +92,18 @@ test("a camera with no source, and a cross-origin upgrade, are refused; the same
9192 expect(await upgrade(host, "/camera/1/video", { Origin: "https://evil.example" })).not.toContain("101");
9293 expect(await upgrade(host, "/camera/1/video", { Origin: `http://${host}` })).toContain("101");
9394 });
95
96 test("the owner's GET /precision carries each H.264 camera's numbers under video, even with the arm away, and /admin shows them", async () => {
97 const host = await boot();
98 const identity = `id-trevato|trevato|admin|${Date.now() + 600_000}`;
99 const owner = { "x-forge-identity": identity, "x-forge-signature": sign(identity, "k".repeat(64)) };
100 const answer = await fetch(`http://${host}/precision`, { headers: owner });
101 expect(answer.status).toBe(502); // the simulation at :1 does not answer
102 const body = (await answer.json()) as { video: { camera: number; viewers: number; fallbacks: Record<string, number> }[] };
103 expect(body.video.map((c) => c.camera)).toEqual([1]);
104 expect(body.video[0]!.fallbacks).toEqual({ quiet: 0, decode: 0, unsupported: 0, closed: 0 });
105 expect((await fetch(`http://${host}/precision`)).status).toBe(404);
106 const html = await (await fetch(`http://${host}/admin`, { headers: owner })).text();
107 expect(html).toContain('<h2 id="video">Video</h2>');
108 expect(html).toContain("<h3>Camera 1</h3>");
109 });
Software/so101/src/relay.ts +13 −5
@@ -36,7 +36,8 @@
3636 * POST /join {"name": "…"}: become someone here, no forge account needed (src/accounts.ts); the answer
3737 * sets the HttpOnly session cookie and names you: {"you": {"id", "name", "owner"}}
3838 * GET /admin, POST /admin the owner's page: pause, end a turn, take control, the settings, the log
39 * GET /precision?hours=1-720 owner and moderators: the arm's precision scorecard (src/precision-history.ts); 404 to anyone else
39 * GET /precision?hours=1-720 owner and moderators: the arm's precision scorecard (src/precision-history.ts), with the relay's
40 * H.264 numbers per camera under "video" (src/video-stats.ts); 404 to anyone else
4041 * POST /precision/snap-back the driver's page: {cause, sizeDeg} for a snap-back only the page saw (cause from src/precision.ts), 400 for any other
4142 * GET /link {"mode": "live"|"reconnecting"|"offline"|"simulated", "host": "<host:port>"|null, "problem": "…"|null}
4243 * GET /status the arm's own /status, and which arm said it: "source": "live <host:port>", "reconnecting to the arm:
@@ -78,6 +79,7 @@ import { readServos } from "./servo-panel.ts";
7879 import { parseFps, type FeedOptions } from "./camera-feed.ts";
7980 import { CameraError, CameraHub } from "./cameras.ts";
8081 import { VideoFeed, type VideoSources } from "./video-feed.ts";
82 import type { VideoCard } from "./video-panel.ts";
8183 import { LADDER } from "./video-fanout.ts";
8284 import { LOCAL as LOCAL_PERSON } from "./identity.ts";
8385 import { LeaderCalibrations } from "./leader-calibrations.ts";
@@ -149,6 +151,8 @@ export interface RelayOptions {
149151 export interface RunningRelay {
150152 readonly port: number;
151153 link(): Link;
154 /** Each H.264 camera's numbers for the scorecard, as GET /precision gives them under `video`. */
155 videoCards(): VideoCard[];
152156 close(): Promise<void>;
153157 }
154158
@@ -472,7 +476,9 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
472476 return null;
473477 }
474478 };
475 /** GET /precision: the scorecard as JSON, for the owner and moderators, who see it on /admin; to anyone else there is no such route. */
479 /** Each H.264 camera's numbers for the scorecard: decoded fps, bytes from the Pi, relay-to-glass latency, fallbacks. */
480 const videoCards = (): VideoCard[] => [...videos].filter(([, feed]) => feed.available).map(([camera, feed]) => ({ camera, ...feed.summary() }));
481 /** GET /precision: the scorecard as JSON (the arm's, with the relay's `video` beside it), for the owner and moderators, who see it on /admin; to anyone else there is no such route. */
476482 const precisionRead = async (req: IncomingMessage, res: ServerResponse, url: URL) => {
477483 if (!lobby || !lobby.who(req) || !lobby.moderates(req)) return json(res, 404, { error: "no such route" });
478484 if (req.method !== "GET") {
@@ -483,9 +489,10 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
483489 if (hours !== null && !(/^\d{1,3}$/.test(hours) && Number(hours) >= 1 && Number(hours) <= 720)) return json(res, 400, { error: "hours must be 1-720" });
484490 try {
485491 const answer = await armPrecision(hours);
486 return json(res, answer.status, await answer.json());
492 const body = (await answer.json()) as unknown;
493 return json(res, answer.status, body && typeof body === "object" && !Array.isArray(body) ? { ...body, video: videoCards() } : body);
487494 } catch (err) {
488 return json(res, 502, { error: `the arm did not answer /precision (${reason(err)})` });
495 return json(res, 502, { error: `the arm did not answer /precision (${reason(err)})`, video: videoCards() });
489496 }
490497 };
491498 /** POST /precision/snap-back (the driver's page: {cause, sizeDeg}): checked here, then added to the arm's scorecard beside its own. */
@@ -689,7 +696,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
689696 const cameraAdmin = ADMIN_CAMERA.exec(url.pathname);
690697 if (cameraAdmin) return await administer(req, res, cameraAdmin[1] === undefined ? null : Number(cameraAdmin[1]));
691698 if (lobby && url.pathname === "/admin/lift") return await handleLift(lobby, req, res, readBody);
692 if (lobby && (url.pathname === "/admin" || url.pathname === "/admin/health")) return await handleAdmin(lobby, req, res, url, readBody, askPrecision, askServos);
699 if (lobby && (url.pathname === "/admin" || url.pathname === "/admin/health")) return await handleAdmin(lobby, req, res, url, readBody, askPrecision, askServos, videoCards);
693700 if (url.pathname === "/precision") return await precisionRead(req, res, url);
694701 if (url.pathname === LEADER) return await leaderCalibration(req, res);
695702 if (url.pathname === "/join") {
@@ -1044,6 +1051,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> {
10441051 return {
10451052 port: typeof address === "object" && address ? address.port : options.port ?? 8102,
10461053 link,
1054 videoCards,
10471055 close: () =>
10481056 new Promise<void>((resolve) => {
10491057 clearInterval(timer);
Software/so101/src/ui/video-player.js +108 −22
@@ -5,10 +5,22 @@
55 * 10 header bytes (keyframe flag, rendition index, relay time float64) then one Annex-B access unit.
66 *
77 * Nothing is queued: a picture that arrives while the decoder is behind is dropped, and so is every one after it
8 * until the next keyframe. `onStatus` hears "live", "none" (no picture in `waitMs`: use MJPEG) and the frame age.
8 * until the next keyframe. A decoder that fails is made again at the next keyframe (the relay puts SPS and PPS on
9 * every one). `onStatus` hears "live", and "none" with why when the page should use MJPEG instead: no picture drawn
10 * for QUIET_MS (at the start or at any time after), the decoder failing DECODE_ERRORS times without a picture between,
11 * a codec this browser cannot configure, or the socket closing. It never stops on a frozen picture.
12 *
13 * Once a second it tells the relay what it measured: the frame age, the frames a second it drew, and the latency from
14 * the relay to the glass, on the relay's clock, the offset taken from pings over the same socket.
915 */
1016
1117 export const HEADER = 10;
18 /** No picture drawn for this long and the page goes back to MJPEG. */
19 export const QUIET_MS = 2000;
20 /** Decoder failures in a row, with no picture between, before giving up on H.264. */
21 export const DECODE_ERRORS = 3;
22 /** How often the clock offset to the relay is measured again. */
23 export const PING_MS = 5000;
1224
1325 /** @param {ArrayBuffer | Uint8Array} data */
1426 export function parseFrame(data) {
@@ -30,10 +42,16 @@ export function codecOf(unit) {
3042
3143 export const canDecode = () => typeof VideoDecoder === "function" && typeof EncodedVideoChunk === "function" && typeof WebSocket === "function";
3244
45 /** The middle of some numbers, or null. @param {number[]} v */
46 const middle = (v) => (v.length ? [...v].sort((a, b) => a - b)[v.length >> 1] : null);
47
3348 export class VideoPlayer {
3449 /**
35 * @param {{ url: string, canvas: { width: number, height: number, getContext(k: string): any }, onStatus?: (s: { state: string, age?: number, rendition?: string, ladder?: any[] }) => void, waitMs?: number,
50 * @param {{ url: string, canvas: { width: number, height: number, getContext(k: string): any },
51 * onStatus?: (s: { state: string, why?: string, age?: number, fps?: number, latency?: number | null, rendition?: string, ladder?: any[] }) => void,
52 * waitMs?: number, quietMs?: number,
3653 * deps?: { WebSocket?: any, VideoDecoder?: any, EncodedVideoChunk?: any, now?: () => number } }} o
54 * waitMs: for the first picture (default quietMs); quietMs: between pictures (default QUIET_MS)
3755 */
3856 constructor(o) {
3957 this.o = o;
@@ -43,25 +61,46 @@ export class VideoPlayer {
4361 this.Chunk = d.EncodedVideoChunk ?? (typeof EncodedVideoChunk === "undefined" ? undefined : EncodedVideoChunk);
4462 this.now = d.now ?? (() => Date.now());
4563 this.ctx = o.canvas.getContext("2d");
64 this.quietMs = o.quietMs ?? QUIET_MS;
4665 this.decoder = null;
4766 this.configured = false;
4867 this.needKey = true;
4968 this.lowest = Infinity;
5069 this.age = 0;
5170 this.frames = 0;
71 this.errors = 0;
72 this.lastTs = -Infinity;
73 /** The relay's clock minus the page's, from the ping with the shortest round trip; null until one comes back. */
74 this.offset = null;
75 this.bestRtt = Infinity;
76 /** Since the last report: pictures drawn, their latencies, and when it began. */
77 this.drawn = 0;
78 this.latencies = [];
79 this.since = this.now();
80 this.fps = 0;
81 this.latency = null;
5282 this.ladder = [];
5383 this.closed = false;
5484 this.ws = null;
5585 this.timers = [];
86 this.watchdog = null;
5687 }
5788
5889 start() {
5990 const ws = (this.ws = new this.WS(this.o.url));
6091 ws.binaryType = "arraybuffer";
6192 ws.onmessage = (e) => (typeof e.data === "string" ? this.control(e.data) : this.frame(e.data));
62 ws.onclose = () => this.status("none");
63 this.timers.push(setTimeout(() => this.frames || this.status("none"), this.o.waitMs ?? 3000));
93 ws.onclose = () => this.status("none", "closed");
94 ws.onopen = () => this.ping();
95 this.quiet(this.o.waitMs ?? this.quietMs);
6496 this.timers.push(setInterval(() => this.report(), 1000));
97 this.timers.push(setInterval(() => this.ping(), PING_MS));
98 }
99
100 /** (Re)arm the watchdog: no picture drawn in `ms` and the page goes back to MJPEG. */
101 quiet(ms = this.quietMs) {
102 clearTimeout(this.watchdog);
103 if (!this.closed) this.watchdog = setTimeout(() => this.status("none", "quiet"), ms);
65104 }
66105
67106 /** Pin a rung by name, or "auto". @param {string} quality */
@@ -69,28 +108,57 @@ export class VideoPlayer {
69108 if (this.ws?.readyState === 1) this.ws.send(JSON.stringify({ quality }));
70109 }
71110
111 ping() {
112 if (this.ws?.readyState === 1) this.ws.send(JSON.stringify({ ping: this.now() }));
113 }
114
72115 stop() {
73116 this.closed = true;
117 clearTimeout(this.watchdog);
74118 this.timers.forEach((t) => (clearTimeout(t), clearInterval(t)));
75119 try { this.ws?.close(); } catch {}
76120 try { this.decoder?.close(); } catch {}
77121 }
78122
79 status(state) {
80 if (state === "none") this.stop(); // giving up releases the socket and the decoder
81 this.o.onStatus?.({ state, age: this.age, rendition: this.rendition, ladder: this.ladder });
123 /** @param {string} state @param {string} [why] */
124 status(state, why) {
125 if (state === "none") {
126 if (this.closed) return; // said once
127 if (this.ws?.readyState === 1) try { this.ws.send(JSON.stringify({ fallback: why })); } catch {}
128 this.stop(); // giving up releases the socket and the decoder
129 }
130 this.o.onStatus?.({ state, why, age: this.age, fps: this.fps, latency: this.latency, rendition: this.rendition, ladder: this.ladder });
82131 }
83132
84133 control(text) {
85134 let m;
86135 try { m = JSON.parse(text); } catch { return; }
136 if (typeof m !== "object" || m === null) return;
137 if (typeof m.pong === "number" && typeof m.at === "number") return this.pong(m.pong, m.at);
87138 if (Array.isArray(m.ladder)) this.ladder = m.ladder;
88139 if (typeof m.rendition === "string") this.rendition = m.rendition;
89140 if (this.frames) this.status("live");
90141 }
91142
143 /** A ping came back: the relay's time `at` is taken as the page's halfway through the round trip. */
144 pong(sent, at) {
145 const rtt = this.now() - sent;
146 if (!(rtt >= 0) || rtt >= this.bestRtt) return;
147 this.bestRtt = rtt;
148 this.offset = at - (sent + rtt / 2);
149 }
150
92151 report() {
93 if (this.ws?.readyState === 1 && this.age > 0) this.ws.send(JSON.stringify({ age: Math.round(this.age) }));
152 const now = this.now();
153 const seconds = (now - this.since) / 1000;
154 if (seconds <= 0) return;
155 this.fps = Number((this.drawn / seconds).toFixed(1));
156 const latency = middle(this.latencies);
157 this.latency = latency === null ? null : Math.round(latency);
158 this.drawn = 0;
159 this.latencies = [];
160 this.since = now;
161 if (this.ws?.readyState === 1 && this.frames) this.ws.send(JSON.stringify({ age: Math.round(this.age), fps: this.fps, ...(this.latency === null ? {} : { latency: this.latency }) }));
94162 }
95163
96164 frame(buffer) {
@@ -104,29 +172,47 @@ export class VideoPlayer {
104172 if (this.needKey && !f.key) return;
105173 if (f.key && !this.configured && !this.configure(f.data)) return;
106174 this.needKey = false;
175 // Timestamps only go forward: two pictures the relay got in the same millisecond still get their own.
176 const ts = (this.lastTs = Math.max(this.lastTs + 1, Math.round(f.sentAt * 1000)));
107177 try {
108 this.decoder.decode(new this.Chunk({ type: f.key ? "key" : "delta", timestamp: Math.round(f.sentAt * 1000), data: f.data }));
178 this.decoder.decode(new this.Chunk({ type: f.key ? "key" : "delta", timestamp: ts, data: f.data }));
109179 } catch {
110 this.needKey = true;
111 this.configured = false;
180 this.failed();
112181 }
113182 }
114183
184 /** The decoder failed: it is made again at the next keyframe, until it has failed too often without a picture. */
185 failed() {
186 this.needKey = true;
187 this.configured = false;
188 if (++this.errors >= DECODE_ERRORS) this.status("none", "decode");
189 }
190
115191 configure(unit) {
116192 const codec = codecOf(unit);
117193 if (!codec || !this.Decoder) return false;
118194 try { this.decoder?.close(); } catch {}
119 this.decoder = new this.Decoder({
120 output: (frame) => {
121 const c = this.o.canvas;
122 if (c.width !== frame.displayWidth || c.height !== frame.displayHeight) (c.width = frame.displayWidth), (c.height = frame.displayHeight);
123 this.ctx.drawImage(frame, 0, 0);
124 frame.close();
125 if (!this.frames++) this.status("live");
126 },
127 error: () => ((this.needKey = true), (this.configured = false)),
128 });
129 this.decoder.configure({ codec, optimizeForLatency: true });
195 try {
196 this.decoder = new this.Decoder({
197 output: (frame) => {
198 if (this.closed) return frame.close();
199 const c = this.o.canvas;
200 if (c.width !== frame.displayWidth || c.height !== frame.displayHeight) (c.width = frame.displayWidth), (c.height = frame.displayHeight);
201 this.ctx.drawImage(frame, 0, 0);
202 if (this.offset !== null) this.latencies.push(Math.max(0, this.now() + this.offset - frame.timestamp / 1000));
203 frame.close();
204 this.errors = 0;
205 this.drawn++;
206 this.quiet();
207 if (!this.frames++) this.status("live");
208 },
209 error: () => this.failed(),
210 });
211 this.decoder.configure({ codec, optimizeForLatency: true });
212 } catch {
213 this.status("none", "unsupported");
214 return false;
215 }
130216 return (this.configured = true);
131217 }
132218 }
Software/so101/src/ui/video-player.test.ts +126 −1
@@ -22,6 +22,7 @@ class FakeSocket {
2222 sent: string[] = [];
2323 onmessage: ((e: { data: unknown }) => void) | null = null;
2424 onclose: (() => void) | null = null;
25 onopen: (() => void) | null = null;
2526 url: string;
2627 constructor(url: string) {
2728 this.url = url;
@@ -157,8 +158,9 @@ describe("video player", () => {
157158 expect(t.decoded).toHaveLength(2);
158159 t.ws.onmessage!({ data: frame(true, 0, 9_850, pics[5]!) }); // 150 ms away; the best seen is 0 by now
159160 expect(t.decoded).toHaveLength(3);
161 t.tick(1000);
160162 t.player.report();
161 expect(JSON.parse(t.ws.sent.at(-1)!)).toEqual({ age: 150 });
163 expect(JSON.parse(t.ws.sent.at(-1)!)).toEqual({ age: 150, fps: 3 }); // three pictures drawn in the second; no pong yet, so no latency
162164 t.player.quality("240p");
163165 expect(JSON.parse(t.ws.sent.at(-1)!)).toEqual({ quality: "240p" });
164166 t.player.stop();
@@ -193,3 +195,126 @@ describe("video player", () => {
193195 expect(t.decoders[0]!.closes).toBe(1);
194196 });
195197 });
198
199 describe("never a frozen picture", () => {
200 const pics = clip(10, 5, 100);
201 /** A decoder that draws each chunk at once with its timestamp, or fails while `failing` is set; `throws` makes configure throw. */
202 function rig(o: { quietMs?: number; throws?: boolean } = {}) {
203 const states: { state: string; why?: string; fps?: number; latency?: number | null }[] = [];
204 const decoders: any[] = [];
205 const chunks: { type: string; timestamp: number }[] = [];
206 let now = 50_000;
207 const ctl = { failing: false };
208 class Decoder {
209 decodeQueueSize = 0;
210 closed = false;
211 init: { output: (f: unknown) => void; error: (e: unknown) => void };
212 constructor(init: { output: (f: unknown) => void; error: (e: unknown) => void }) {
213 this.init = init;
214 decoders.push(this);
215 }
216 configure() {
217 if (o.throws) throw new TypeError("unsupported codec");
218 }
219 decode(chunk: { type: string; timestamp: number }) {
220 chunks.push(chunk);
221 if (ctl.failing) return this.init.error(new Error("decode"));
222 this.init.output({ displayWidth: 854, displayHeight: 480, timestamp: chunk.timestamp, close() {} });
223 }
224 close() {
225 this.closed = true;
226 }
227 }
228 const canvas = { width: 0, height: 0, getContext: () => ({ drawImage() {} }) };
229 const player = new VideoPlayer({ url: "ws://x/video", canvas, quietMs: o.quietMs ?? 10_000, onStatus: (s) => states.push(s), deps: { WebSocket: FakeSocket, VideoDecoder: Decoder, EncodedVideoChunk: class { constructor(init: object) { Object.assign(this, init); } }, now: () => now } });
230 player.start();
231 const ws = FakeSocket.last;
232 const send = (i: number, sentAt = now) => ws.onmessage!({ data: frame(i % 5 === 0, 0, sentAt, pics[i]!) });
233 return { player, ws, states, decoders, chunks, ctl, send, tick: (ms: number) => (now += ms), get now() { return now; } };
234 }
235 const sent = (ws: FakeSocket) => ws.sent.map((s) => JSON.parse(s));
236
237 test("pictures that stop after it went live send the page back to MJPEG, saying why", async () => {
238 const r = rig({ quietMs: 30 });
239 r.send(0);
240 r.send(1);
241 expect(r.states.map((s) => s.state)).toEqual(["live"]);
242 await Bun.sleep(15);
243 r.send(2); // a picture re-arms the watchdog
244 await Bun.sleep(20);
245 expect(r.states.map((s) => s.state)).toEqual(["live"]);
246 await Bun.sleep(40);
247 expect(r.states.map((s) => [s.state, s.why])).toEqual([["live", undefined], ["none", "quiet"]]);
248 expect(sent(r.ws)).toContainEqual({ fallback: "quiet" });
249 expect(r.ws.closes).toBe(1);
250 expect(r.decoders[0].closed).toBe(true);
251 });
252
253 test("a decoder that fails is made again at the next keyframe; failing three times with no picture between gives up", () => {
254 const r = rig();
255 r.send(0);
256 r.ctl.failing = true;
257 r.send(1); // fails: waits for a key
258 r.ctl.failing = false;
259 r.send(2);
260 r.send(3);
261 expect(r.chunks).toHaveLength(2);
262 r.send(5); // a keyframe: a new decoder, and it plays on
263 r.send(6);
264 expect(r.decoders).toHaveLength(2);
265 expect(r.chunks.slice(2).map((c) => c.type)).toEqual(["key", "delta"]);
266 expect(r.states.map((s) => s.state)).toEqual(["live"]);
267 r.ctl.failing = true;
268 r.send(7);
269 r.send(0);
270 r.send(5);
271 expect(r.states.map((s) => [s.state, s.why]).at(-1)).toEqual(["none", "decode"]);
272 expect(sent(r.ws)).toContainEqual({ fallback: "decode" });
273 });
274
275 test("a codec the browser cannot configure gives up at once", () => {
276 const r = rig({ throws: true });
277 r.send(0);
278 expect(r.states.map((s) => [s.state, s.why])).toEqual([["none", "unsupported"]]);
279 });
280
281 test("the socket closing after pictures is a fallback too, said once", () => {
282 const r = rig();
283 r.send(0);
284 r.ws.onclose!();
285 r.ws.onclose!();
286 expect(r.states.map((s) => [s.state, s.why])).toEqual([["live", undefined], ["none", "closed"]]);
287 });
288
289 test("pictures the relay got in the same millisecond still get rising timestamps", () => {
290 const r = rig();
291 r.send(0, 1000);
292 r.send(1, 1000);
293 r.send(2, 999.9995);
294 const ts = r.chunks.map((c) => c.timestamp);
295 expect(ts[0]).toBe(1_000_000);
296 expect(ts[1]! > ts[0]! && ts[2]! > ts[1]!).toBe(true);
297 });
298
299 test("latency to the glass is on the relay's clock, from the ping with the shortest round trip; fps is what was drawn", () => {
300 const r = rig();
301 r.ws.onopen!();
302 const first = sent(r.ws).at(-1);
303 expect(first).toEqual({ ping: r.now });
304 // The relay's clock is 5 s ahead of the page's; this round trip took 40 ms.
305 r.tick(40);
306 r.ws.onmessage!({ data: JSON.stringify({ pong: first.ping, at: first.ping + 5000 + 20 }) });
307 // A slower round trip later is not taken.
308 r.player.ping();
309 const second = sent(r.ws).at(-1);
310 r.tick(400);
311 r.ws.onmessage!({ data: JSON.stringify({ pong: second.ping, at: second.ping + 5000 + 900 }) });
312 expect(r.player.offset).toBe(5000);
313 r.tick(560); // 1 s since the start
314 // Got by the relay 90 ms ago on its clock, drawn now.
315 for (let i = 0; i < 12; i++) r.send(i % 10, r.now + 5000 - 90);
316 r.player.report();
317 expect(sent(r.ws).at(-1)).toEqual({ age: 0, fps: 12, latency: 90 });
318 expect(r.player.latency).toBe(90);
319 });
320 });
Software/so101/src/video-fanout.test.ts +68
@@ -1,5 +1,6 @@
11 import { describe, expect, test } from "bun:test";
22 import { clip } from "./h264-clip.ts";
3 import { nalsOf } from "./h264.ts";
34 import { HEADER, LADDER, VideoFanout, type VideoSink } from "./video-fanout.ts";
45
56 function sink(): VideoSink & { sent: Buffer[]; texts: any[]; queued: number } {
@@ -113,3 +114,70 @@ describe("renditions and viewers", () => {
113114 expect(f.wanted().size).toBe(0);
114115 });
115116 });
117
118 describe("late joiners", () => {
119 // The worst case for a late joiner: the encoder sends SPS and PPS once, at the head of its stream.
120 const head = clip(4, 15, 15, 120, { sets: "head", aud: true });
121 const one = [{ name: "480p", width: 854, height: 480, fps: 15, kbps: 600 }];
122 const types = (m: Buffer) => nalsOf(m.subarray(HEADER)).map((n) => n.type);
123
124 test("every keyframe a viewer is sent carries SPS and PPS, even when the encoder sent them only once", () => {
125 const f = new VideoFanout({ ladder: one });
126 const early = sink();
127 f.join(early);
128 for (let i = 0; i < 20; i++) f.push("480p", { key: i % 15 === 0, data: head[i]! });
129 const late = sink();
130 f.join(late); // the cached keyframe (picture 15) is fresh: it comes at once, with its sets
131 for (let i = 20; i < 40; i++) f.push("480p", { key: i % 15 === 0, data: head[i]! });
132 for (const s of [early, late]) {
133 const keys = s.sent.filter((m) => (m[0]! & 1) === 1);
134 expect(keys.length).toBeGreaterThan(1);
135 for (const k of keys) expect(types(k)).toEqual([9, 7, 8, 5]);
136 }
137 // late: the cached key, nothing until the next live key at 30, then every picture
138 expect(frames(late).map((x) => x.key)).toEqual([true, true, ...Array(9).fill(false)]);
139 });
140
141 test("a stale keyframe is not sent, and a source that went away is forgotten", () => {
142 let t = 0;
143 const f = new VideoFanout({ ladder: one, now: () => t });
144 f.push("480p", { key: true, data: head[0]! });
145 t += 2500;
146 const s = sink();
147 f.join(s);
148 expect(s.sent).toHaveLength(0);
149 f.push("480p", { key: true, data: head[0]! });
150 f.forget("480p");
151 const after = sink();
152 f.join(after);
153 expect(after.sent).toHaveLength(0);
154 f.push("480p", { key: true, data: head[15]! }); // no sets known since the source went: it goes as it is
155 expect(types(after.sent[0]!)).toEqual([9, 5]);
156 });
157 });
158
159 describe("what the pages report", () => {
160 test("pings are answered on the relay's clock; reports are taken once a second; a fallback once per viewer", () => {
161 let t = 1000;
162 const f = new VideoFanout({ now: () => t });
163 const s = sink();
164 const v = f.join(s);
165 v.message('{"ping":42.5}');
166 v.message('{"ping":43}'); // too soon
167 expect(s.texts.filter((x) => "pong" in x)).toEqual([{ pong: 42.5, at: 1000 }]);
168 v.message('{"age":10,"fps":14,"latency":120}');
169 v.message('{"age":10,"fps":1,"latency":9000}'); // too soon: ignored
170 t += 1000;
171 v.message('{"age":10,"fps":12.04,"latency":80.6}');
172 t += 1000;
173 v.message('{"fps":-1}');
174 v.message('{"fps":500}');
175 v.message('{"fps":10,"latency":"x"}');
176 v.message('{"fallback":"quiet"}');
177 v.message('{"fallback":"quiet"}');
178 v.message('{"fallback":"<script>"}');
179 const card = f.stats.summary(t, f.watching);
180 expect(card).toMatchObject({ viewers: 1, reports: 3, fpsP50: 12, fpsMin: 10, latencyP50Ms: 81, latencyP95Ms: 120 });
181 expect(card.fallbacks).toEqual({ quiet: 1, decode: 0, unsupported: 0, closed: 0 });
182 });
183 });
Software/so101/src/video-fanout.ts +53 −6
@@ -6,11 +6,15 @@
66 * the ladder, one that went a while clean steps back up, each switch at a keyframe of the new rendition. A viewer can
77 * pin one rendition. The driver is patient (it steps down only after more drops) so the best picture stays with it.
88 *
9 * Every keyframe goes out with an SPS and a PPS (the last ones seen when the encoder left them off: src/h264.ts), and
10 * a viewer that joins is sent the rung's last keyframe at once when it is fresh, so it has a picture before the next one.
11 *
912 * A binary message is HEADER bytes then the access unit: byte 0 bit 0 = keyframe, byte 1 = the rendition's index in
1013 * the ladder, bytes 2..9 = the time the relay got the picture (ms, float64 big-endian). Text messages are JSON.
1114 */
1215
13 import type { AccessUnit } from "./h264.ts";
16 import { ParameterSets, type AccessUnit } from "./h264.ts";
17 import { isFallback, parseReport, VideoStats } from "./video-stats.ts";
1418
1519 export interface Rendition {
1620 name: string;
@@ -46,6 +50,8 @@ export interface FanoutOptions {
4650 cleanMs?: number;
4751 /** A frame age the viewer reports above this also counts as a drop. */
4852 maxAgeMs?: number;
53 /** A rung's last keyframe younger than this is sent to a viewer as it joins. */
54 keyFreshMs?: number;
4955 now?: () => number;
5056 }
5157
@@ -64,6 +70,17 @@ interface Viewer {
6470 bytes: number;
6571 frames: number;
6672 skipped: number;
73 /** When its last report and ping were taken, and whether it has said it gave up. */
74 reportedAt: number;
75 pingedAt: number;
76 fellBack: boolean;
77 }
78
79 /** What the fanout keeps per rung: its parameter sets, and its last keyframe as a ready message. */
80 interface Rung {
81 sets: ParameterSets;
82 key: Buffer | null;
83 keyAt: number;
6784 }
6885
6986 export interface ViewerStats {
@@ -76,11 +93,14 @@ export interface ViewerStats {
7693 export class VideoFanout {
7794 readonly ladder: readonly Rendition[];
7895 private readonly viewers = new Set<Viewer>();
96 private readonly rungs = new Map<number, Rung>();
97 /** What the viewers' pages report, and the bytes the feed pulls (src/video-stats.ts). */
98 readonly stats = new VideoStats();
7999 private readonly o: Required<Omit<FanoutOptions, "ladder">>;
80100
81101 constructor(options: FanoutOptions = {}) {
82102 this.ladder = options.ladder ?? LADDER;
83 this.o = { maxBacklog: 64 << 10, dropsToStepDown: 3, cleanMs: 6000, maxAgeMs: 500, now: Date.now, ...options };
103 this.o = { maxBacklog: 64 << 10, dropsToStepDown: 3, cleanMs: 6000, maxAgeMs: 500, keyFreshMs: 2000, now: Date.now, ...options };
84104 }
85105
86106 get watching(): number {
@@ -98,9 +118,16 @@ export class VideoFanout {
98118 join(sink: VideoSink, driver = false): { leave(): void; message(text: string): void; stats(): ViewerStats } {
99119 const start = driver ? 0 : Math.min(1, this.ladder.length - 1);
100120 const now = this.o.now();
101 const v: Viewer = { sink, driver, current: start, target: start, pinned: null, needKey: true, drops: 0, lastDrop: now, lastStep: now, bytes: 0, frames: 0, skipped: 0 };
121 const v: Viewer = { sink, driver, current: start, target: start, pinned: null, needKey: true, drops: 0, lastDrop: now, lastStep: now, bytes: 0, frames: 0, skipped: 0, reportedAt: -Infinity, pingedAt: -Infinity, fellBack: false };
102122 this.viewers.add(v);
103123 sink.text(JSON.stringify({ ladder: this.ladder, rendition: this.ladder[start]!.name }));
124 // A picture now, from the last keyframe; it still waits for the next one to go on, as the pictures between depend on it.
125 const rung = this.rungs.get(start);
126 if (rung?.key && now - rung.keyAt < this.o.keyFreshMs) {
127 sink.send(rung.key);
128 v.bytes += rung.key.length;
129 v.frames++;
130 }
104131 return {
105132 leave: () => void this.viewers.delete(v),
106133 message: (text) => this.message(v, text),
@@ -108,10 +135,15 @@ export class VideoFanout {
108135 };
109136 }
110137
111 /** A viewer's JSON: {"quality":"auto"|<name>} to pin one, {"age":ms} for the frame age it measures. Anything else is ignored. */
138 /**
139 * A viewer's JSON: {"quality":"auto"|<name>} to pin one; once a second {"age":ms, "fps":n, "latency":ms} for the frame
140 * age, decoded frames a second and relay-to-glass latency it measures; {"ping":t} answered {"pong":t, "at":relay ms}
141 * for its clock offset; {"fallback":why} as it gives up for MJPEG. More than one report a second, or a ping every
142 * 200 ms, is ignored, so one page cannot fill the scorecard. Anything else is ignored.
143 */
112144 private message(v: Viewer, text: string) {
113145 if (text.length > 200) return;
114 let m: { quality?: unknown; age?: unknown };
146 let m: { quality?: unknown; age?: unknown; ping?: unknown; fallback?: unknown };
115147 try {
116148 m = JSON.parse(text);
117149 } catch {
@@ -124,18 +156,32 @@ export class VideoFanout {
124156 else if (at >= 0) (v.pinned = at), (v.target = at);
125157 }
126158 if (typeof m.age === "number" && Number.isFinite(m.age) && m.age > this.o.maxAgeMs) this.drop(v);
159 const now = this.o.now();
160 const report = parseReport(m as Record<string, unknown>);
161 if (report && now - v.reportedAt >= 900) ((v.reportedAt = now), this.stats.report(now, report));
162 if (typeof m.ping === "number" && Number.isFinite(m.ping) && now - v.pingedAt >= 200) ((v.pingedAt = now), v.sink.text(JSON.stringify({ pong: m.ping, at: now })));
163 if (isFallback(m.fallback) && !v.fellBack) ((v.fellBack = true), this.stats.fallback(now, m.fallback));
164 }
165
166 /** A rung's source went away: what it said about its stream no longer holds for the next one. */
167 forget(rendition: string): void {
168 this.rungs.delete(this.ladder.findIndex((r) => r.name === rendition));
127169 }
128170
129171 /** The next picture of a rendition arrives from the Pi. */
130 push(rendition: string, unit: AccessUnit): void {
172 push(rendition: string, picture: AccessUnit): void {
131173 const at = this.ladder.findIndex((r) => r.name === rendition);
132174 if (at < 0) return;
133175 const now = this.o.now();
176 let rung = this.rungs.get(at);
177 if (!rung) this.rungs.set(at, (rung = { sets: new ParameterSets(), key: null, keyAt: 0 }));
178 const unit = rung.sets.complete(picture);
134179 const message = Buffer.allocUnsafe(HEADER + unit.data.length);
135180 message[0] = unit.key ? 1 : 0;
136181 message[1] = at;
137182 message.writeDoubleBE(now, 2);
138183 unit.data.copy(message, HEADER);
184 if (unit.key && rung.sets.known) ((rung.key = message), (rung.keyAt = now));
139185 for (const v of this.viewers) {
140186 if (v.target === at && v.current !== at && unit.key) {
141187 v.current = at;
@@ -193,5 +239,6 @@ export class VideoFanout {
193239
194240 close(): void {
195241 this.viewers.clear();
242 this.rungs.clear();
196243 }
197244 }
Software/so101/src/video-feed.ts +8
@@ -10,6 +10,7 @@
1010 import { proxyFor } from "./egress.ts";
1111 import { AccessUnitSplitter } from "./h264.ts";
1212 import { LADDER, VideoFanout, type FanoutOptions, type VideoSink } from "./video-fanout.ts";
13 import type { VideoSummary } from "./video-stats.ts";
1314
1415 export interface VideoSources {
1516 /** Rendition name → the URL of its Annex-B stream. */
@@ -68,6 +69,11 @@ export class VideoFeed {
6869 this.tick();
6970 }
7071
72 /** The scorecard's numbers for this camera (src/video-stats.ts). */
73 summary(now = Date.now()): VideoSummary {
74 return this.fanout.stats.summary(now, this.fanout.watching);
75 }
76
7177 close(): void {
7278 this.closed = true;
7379 this.fanout.close();
@@ -100,11 +106,13 @@ export class VideoFeed {
100106 if (!response.ok || !response.body) throw new Error(`answered ${response.status}`);
101107 const splitter = new AccessUnitSplitter();
102108 for await (const chunk of response.body as unknown as AsyncIterable<Uint8Array>) {
109 this.fanout.stats.received(Date.now(), chunk.length);
103110 for (const unit of splitter.push(chunk)) this.fanout.push(name, unit);
104111 }
105112 } catch {
106113 // down or cut off: the next tick tries again while the rung is wanted
107114 } finally {
115 this.fanout.forget(name); // the next stream may come from an encoder that started over
108116 if (this.pulls.get(name) === controller) this.pulls.delete(name);
109117 }
110118 })();
Software/so101/src/video-panel.test.ts added +24
@@ -0,0 +1,24 @@
1 import { expect, test } from "bun:test";
2 import { videoSection, type VideoCard } from "./video-panel.ts";
3
4 const esc = (s: string) => s.replace(/[&<>"']/g, (c) => `&#${c.charCodeAt(0)};`);
5 const card = (over: Partial<VideoCard> = {}): VideoCard => ({
6 camera: 1, viewers: 2, fpsP50: 14, fpsMin: 11, bytesPerS: 28_672, latencyP50Ms: 85, latencyP95Ms: 140, reports: 120,
7 fallbacks: { quiet: 1, decode: 0, unsupported: 0, closed: 2 }, ...over,
8 });
9
10 test("each H.264 camera shows decoded fps, KB/s from the Pi, relay-to-glass latency and its fallbacks with why", () => {
11 const html = videoSection([card(), card({ camera: 2, fpsP50: null, fpsMin: null, latencyP50Ms: null, latencyP95Ms: null, reports: 1, fallbacks: { quiet: 0, decode: 0, unsupported: 0, closed: 0 } })], esc);
12 expect(html).toContain("<h3>Camera 1</h3>");
13 expect(html).toContain("<b>14 fps (11 fps)</b>");
14 expect(html).toContain("<b>28.0 KB/s</b>");
15 expect(html).toContain("<b>85 ms / 140 ms</b>");
16 expect(html).toContain("<b>quiet 1, closed 2</b>");
17 expect(html).toContain("1 page report in");
18 expect(html).toContain("<b>— / —</b>");
19 expect(html).toContain("<b>none</b>");
20 });
21
22 test("no H.264 camera says how to have one", () => {
23 expect(videoSection([], esc)).toContain("ARM_H264_&lt;n&gt;");
24 });
Software/so101/src/video-panel.ts added +33
@@ -0,0 +1,33 @@
1 /**
2 * The Video part of the precision scorecard on /admin: per camera with H.264, what its viewers' pages decoded, what the
3 * relay pulled from the Pi, the latency from the relay to the glass, and how often a page fell back to MJPEG. The
4 * numbers are the relay's own (src/video-stats.ts), from the last ten minutes; plain text, no script.
5 */
6
7 import type { VideoSummary } from "./video-stats.ts";
8
9 export interface VideoCard extends VideoSummary {
10 camera: number;
11 }
12
13 const or = (v: number | null, unit: string) => (v === null ? "—" : `${v} ${unit}`);
14
15 /** The section; `esc` is the page's own escaping. An empty list (no camera has H.264) says so. */
16 export function videoSection(cards: readonly VideoCard[], esc: (s: string) => string): string {
17 const head = `<section aria-labelledby="video"><h2 id="video">Video</h2>`;
18 if (!cards.length) return `${head}<p class="empty">No camera has H.264: set <code>ARM_H264_&lt;n&gt;</code> and the cameras play it, with MJPEG as the fallback.</p></section>`;
19 const rows = cards
20 .map((c) => {
21 const falls = Object.entries(c.fallbacks).filter(([, n]) => n > 0).map(([why, n]) => `${why} ${n}`).join(", ") || "none";
22 const items: Array<[string, string]> = [
23 ["Decoded, p50 (lowest)", c.fpsP50 === null ? "—" : `${c.fpsP50} fps (${c.fpsMin} fps)`],
24 ["From the Pi", `${(c.bytesPerS / 1024).toFixed(1)} KB/s`],
25 ["Relay → glass, p50 / p95", `${or(c.latencyP50Ms, "ms")} / ${or(c.latencyP95Ms, "ms")}`],
26 ["Fell back to MJPEG", falls],
27 ];
28 const list = items.map(([label, value]) => `<li><span>${esc(label)}</span><span><b>${esc(value)}</b></span></li>`).join("");
29 return `<h3>Camera ${c.camera}</h3><p>${c.viewers} watching · ${c.reports} page report${c.reports === 1 ? "" : "s"} in the last 10 minutes</p><ol class="bars precision" aria-label="Camera ${c.camera}, video">${list}</ol>`;
30 })
31 .join("");
32 return `${head}${rows}<p>Relay → glass is from the relay getting a picture to the page drawing it; the hop before, from the camera through the Pi's encoder, carries no capture time and is not in it.</p></section>`;
33 }
Software/so101/src/video-stats.ts added +102
@@ -0,0 +1,102 @@
1 /**
2 * One camera's H.264 as its viewers see it, for the precision scorecard: the frames a second their pages decoded, the
3 * bytes a second the relay pulled from the Pi, the latency from the relay to the glass, and how often a page gave up
4 * on H.264 for MJPEG, with why. Plain data in (each page's once-a-second report, the bytes as they come), plain data
5 * out (a summary of the last WINDOW_MS).
6 *
7 * The latency is from the moment the relay got a picture to the moment the page drew it, on the relay's clock (the
8 * page keeps an estimate of the offset, from pings over its socket). The hop before, from the camera's glass through
9 * the Pi's encoder to the relay, is not in it: raw Annex-B carries no capture time.
10 */
11
12 /** The span the summary covers, ms. */
13 export const WINDOW_MS = 10 * 60_000;
14 /** The span bytes a second are averaged over, ms. */
15 export const RATE_MS = 10_000;
16 /** Most reports kept; the oldest go first. */
17 const KEEP = 20_000;
18
19 /** Why a page left H.264 for MJPEG: a fixed list, so every count means the same thing release to release. */
20 export const FALLBACKS = ["quiet", "decode", "unsupported", "closed"] as const;
21 export type Fallback = (typeof FALLBACKS)[number];
22 export const isFallback = (v: unknown): v is Fallback => typeof v === "string" && (FALLBACKS as readonly string[]).includes(v);
23
24 /** The most a page may report: frames a second, and milliseconds of latency. */
25 export const MAX_FPS = 120;
26 export const MAX_LATENCY_MS = 60_000;
27
28 export interface VideoReport {
29 fps: number;
30 /** Relay to glass, ms; null while the page has no clock offset yet. */
31 latencyMs: number | null;
32 }
33
34 /** A page's report, checked: fps from 0 to MAX_FPS, latency from 0 to MAX_LATENCY_MS (or left out); null when it is neither. */
35 export function parseReport(m: Record<string, unknown>): VideoReport | null {
36 const { fps, latency } = m;
37 if (typeof fps !== "number" || !Number.isFinite(fps) || fps < 0 || fps > MAX_FPS) return null;
38 const latencyMs = typeof latency === "number" && Number.isFinite(latency) && latency >= 0 && latency <= MAX_LATENCY_MS ? Math.round(latency) : null;
39 return { fps: Number(fps.toFixed(1)), latencyMs };
40 }
41
42 export interface VideoSummary {
43 /** Pages watching now. */
44 viewers: number;
45 /** Decoded frames a second, over the pages' reports in the window. */
46 fpsP50: number | null;
47 fpsMin: number | null;
48 /** From the Pi to the relay, over the last RATE_MS. */
49 bytesPerS: number;
50 latencyP50Ms: number | null;
51 latencyP95Ms: number | null;
52 /** Reports in the window. */
53 reports: number;
54 fallbacks: Record<Fallback, number>;
55 }
56
57 /** Nearest-rank percentile of unsorted values; null for none. */
58 function percentile(values: readonly number[], p: number): number | null {
59 if (!values.length) return null;
60 const sorted = [...values].sort((a, b) => a - b);
61 return sorted[Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1))]!;
62 }
63
64 export class VideoStats {
65 private readonly reports: Array<{ at: number } & VideoReport> = [];
66 private readonly falls: Array<{ at: number; why: Fallback }> = [];
67 private readonly bytes: Array<readonly [number, number]> = [];
68
69 report(at: number, r: VideoReport): void {
70 this.reports.push({ at, ...r });
71 if (this.reports.length > KEEP) this.reports.shift();
72 }
73
74 fallback(at: number, why: Fallback): void {
75 this.falls.push({ at, why });
76 if (this.falls.length > KEEP) this.falls.shift();
77 }
78
79 received(at: number, n: number): void {
80 this.bytes.push([at, n]);
81 while (this.bytes.length && this.bytes[0]![0] < at - RATE_MS) this.bytes.shift();
82 }
83
84 summary(now: number, viewers: number): VideoSummary {
85 const recent = this.reports.filter((r) => r.at > now - WINDOW_MS);
86 const fps = recent.map((r) => r.fps);
87 const latency = recent.flatMap((r) => (r.latencyMs === null ? [] : [r.latencyMs]));
88 const fallbacks = Object.fromEntries(FALLBACKS.map((f) => [f, 0])) as Record<Fallback, number>;
89 for (const f of this.falls) if (f.at > now - WINDOW_MS) fallbacks[f.why]++;
90 const inRate = this.bytes.filter(([at]) => at > now - RATE_MS).reduce((sum, [, n]) => sum + n, 0);
91 return {
92 viewers,
93 fpsP50: percentile(fps, 50),
94 fpsMin: fps.length ? Math.min(...fps) : null,
95 bytesPerS: Math.round(inRate / (RATE_MS / 1000)),
96 latencyP50Ms: percentile(latency, 50),
97 latencyP95Ms: percentile(latency, 95),
98 reports: recent.length,
99 fallbacks,
100 };
101 }
102 }
Software/so101/src/viewer.html +3 −2
@@ -642,7 +642,7 @@
642642 // Behind the relay the camera is frames over WS /camera/<id>/ws (src/ui/video.js, imported when first needed); on the machine itself, the plain <img> stream.
643643 let video = null;
644644 const stopFeed = (c) => { c.epoch = (c.epoch ?? 0) + 1; c.feed?.close?.(); c.feed = null; };
645 // A camera the relay flags h264 plays through ui/video-pane.js and video-player.js, only where WebCodecs exists; no picture in 3 s and it goes back to the frames.
645 // A camera the relay flags h264 plays through ui/video-pane.js and video-player.js, only where WebCodecs exists; no picture drawn for 2 s (at the start or later), a decoder that keeps failing or a closed socket and it goes back to the frames.
646646 async function playH264(c, epoch) {
647647 let pane, player;
648648 try { [pane, player] = await Promise.all([import("/ui/__VERSION__/video-pane.js"), import("/ui/__VERSION__/video-player.js")]); } catch { return false; }
@@ -789,7 +789,8 @@
789789 }
790790 const why = quality === "saver" ? "data saver" : "slow link";
791791 const slow = plan.mode === "snapshot" ? ` · ${(1000 / plan.everyMs).toFixed(plan.everyMs > 1000 ? 1 : 0)} fps, ${why}` : plan.fps ? ` · ${plan.fps} fps, ${why}` : "";
792 $("live").textContent = cams.filter(paneOn).map((c) => (c.live ? `live${slow}` : c.online === false ? "offline" : "waiting")).join(" · ");
792 const h264 = (c) => (c.feed?.h264 && c.live ? ` · H.264 ${Math.round(c.feed.player?.fps ?? 0)} fps` : "");
793 $("live").textContent = cams.filter(paneOn).map((c) => (c.live ? `live${h264(c) || slow}` : c.online === false ? "offline" : "waiting")).join(" · ");
793794 }, 1000);
794795 if (boot) { reconcile(boot.cameras); setTimeout(pollCameras, 3000); }
795796 else if (relayed) pollCameras();

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs