Public: anyone can read this; the crew can change it.
Migrated from github.com/Trevato/SO-ARM100
workbenchthis machine
Cameras on hardware H.264: 20x fewer bytes, never a frozen frame (MJPEG fallback) #476
Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.
vesper/arm:vesper/464-cameras-on-hardware-h-264-20x-fewer-byte → main 4 commits fork
18 files
changed +911
−63
Software/README.md +45 −5
| @@ -247,10 +247,40 @@ rendition per viewer. A viewer whose connection backs up steps down the ladder | ||
| 247 | 247 | and, rather than queueing, skips to the next keyframe; the **quality selector** |
| 248 | 248 | on the picture lists the configured rungs and pins one, or leaves it on Auto. The page decodes with |
| 249 | 249 | WebCodecs, loading the player only when `/cameras` reports `"h264": true` for |
| 250 | the camera. When the browser has no WebCodecs, or no picture arrives in time, | |
| 251 | the page falls back to MJPEG (`/camera/<id>/stream`) by itself. The video | |
| 252 | socket is open to guests, like `/stream`; the driver is given the best | |
| 253 | rendition. Control messages from the viewer are capped at 200 bytes. | |
| 250 | the camera. The page falls back to MJPEG (`/camera/<id>/stream`) by itself, | |
| 251 | and never holds a frozen picture: when the browser has no WebCodecs or cannot | |
| 252 | configure the codec, when no picture is drawn for 2 s (at the start or at any | |
| 253 | time after), when the decoder fails three times with no picture between, or | |
| 254 | when the socket closes. The live line under the picture shows the decoded | |
| 255 | frame rate (`live · H.264 14 fps`). The video socket is open to guests, like | |
| 256 | `/stream`; the driver is given the best rendition. Control messages from the | |
| 257 | viewer are capped at 200 bytes. | |
| 258 | ||
| 259 | *Starting fast, and starting at all.* The relay keeps each rendition's last | |
| 260 | SPS and PPS and puts them on every keyframe that comes without them, so a | |
| 261 | viewer who joins late, or a decoder that has to start over after an error, | |
| 262 | can always configure from the next keyframe; an encoder that sends them only | |
| 263 | at the head of its stream used to leave a late joiner on one picture. A viewer | |
| 264 | who joins is also sent the rendition's last keyframe at once (when it is under | |
| 265 | 2 s old), so a picture is up before the next keyframe comes. Pictures cut into | |
| 266 | several slices stay one access unit (`src/h264.ts`). | |
| 267 | ||
| 268 | *Measured.* Once a second each page tells the relay the frames it drew, the | |
| 269 | frame age and the latency from the relay to the glass (on the relay's clock: | |
| 270 | the page keeps the offset from pings over the same socket); a page that falls | |
| 271 | back says why. The relay adds the bytes a second it pulls from the Pi and | |
| 272 | shows all of it per camera, over the last 10 minutes, in the **Video** section | |
| 273 | of `/admin` beside the precision scorecard, and under `video` in | |
| 274 | `GET /precision`. The hop from the camera's glass through the Pi's encoder to | |
| 275 | the relay is not in the latency: raw Annex-B carries no capture time. The | |
| 276 | numbers are kept in memory, so a restart starts them over. | |
| 277 | ||
| 278 | *Tested against a capture.* `src/h264-replay.test.ts` replays an Annex-B file | |
| 279 | through a fake Pi, the relay and the page's own player into a decoder that | |
| 280 | refuses what a browser's would. It uses a synthetic stream of the Pi's shape | |
| 281 | unless `ARM_H264_CAPTURE` names a capture, recorded on the Pi's network with | |
| 282 | `curl -s --max-time 10 http://100.89.95.68:8091/h264 > cam1.h264`, then | |
| 283 | `ARM_H264_CAPTURE=$PWD/cam1.h264 bun test src/h264-replay.test.ts`. | |
| 254 | 284 | |
| 255 | 285 | *Not in this pull.* The Pi's encoder side: something on the Pi must serve each |
| 256 | 286 | rendition as an H.264 stream at the URL you name (hardware encoder, one port |
| @@ -258,9 +288,19 @@ per rung). Without it the relay's fetch fails and viewers stay on MJPEG. | ||
| 258 | 288 | |
| 259 | 289 | *The forge's egress list.* The relay reaches the Pi through the forge's egress |
| 260 | 290 | proxy (`proxyFor` in `src/egress.ts`, the same as the other camera paths), so every rendition's `host:port` must be in `forge.json`'s `egress`. |
| 261 | It lists 100.89.95.68 on 8080, 8081, 8101 and 8082 to 8084 (the example ports above); | |
| 291 | It lists 100.89.95.68 on 8080, 8081, 8101, 8082 to 8084 (the example ports above) | |
| 292 | and 8091 and 8092 (ustreamer's own H.264 sink, one per camera, below); | |
| 262 | 293 | if your encoder listens elsewhere, add those ports. |
| 263 | 294 | |
| 295 | *ustreamer's H.264 sink, both cameras.* On the Pi each ustreamer also feeds a | |
| 296 | hardware H.264 sink (`--h264-sink=so101-cam<n> --h264-bitrate=600 | |
| 297 | --h264-gop=15`), served as raw Annex-B by `so101-h264@<n>` at | |
| 298 | `http://100.89.95.68:809<n>/h264`: about 28 KB/s a camera against 450-640 KB/s | |
| 299 | for MJPEG. It is one rendition, so it goes on the 480p rung: | |
| 300 | ||
| 301 | ARM_H264_1=480p=http://100.89.95.68:8091/h264 | |
| 302 | ARM_H264_2=480p=http://100.89.95.68:8092/h264 | |
| 303 | ||
| 264 | 304 | *The terminal (`ssh`) shows no video, and has no quality choice:* both are page-only. |
| 265 | 305 | |
| 266 | 306 | **In the browser.** Behind the relay the page draws each camera from |
Software/so101/src/admin-page.ts +4
| @@ -13,6 +13,7 @@ import type { Snapshot } from "./turns.ts"; | ||
| 13 | 13 | import type { ViewerRow } from "./viewer-reports.ts"; |
| 14 | 14 | import { precisionSection, type PrecisionPanel } from "./precision-panel.ts"; |
| 15 | 15 | import { servosSection, type ServoPanel } from "./servo-panel.ts"; |
| 16 | import { videoSection, type VideoCard } from "./video-panel.ts"; | |
| 16 | 17 | |
| 17 | 18 | export const esc = (text: string): string => text.replace(/[&<>"']/g, (c) => `&#${c.charCodeAt(0)};`); |
| 18 | 19 | |
| @@ -158,6 +159,8 @@ export interface AdminPage { | ||
| 158 | 159 | precision?: PrecisionPanel | null; |
| 159 | 160 | /** Owner only: the servos' control registers (src/servo-panel.ts); null when the arm did not answer, absent when not asked. */ |
| 160 | 161 | servos?: ServoPanel | null; |
| 162 | /** The cameras' H.264 numbers, beside it (src/video-panel.ts); absent when not asked. */ | |
| 163 | video?: readonly VideoCard[]; | |
| 161 | 164 | done?: Done | null; |
| 162 | 165 | error?: string | null; |
| 163 | 166 | } |
| @@ -283,6 +286,7 @@ ${p.canEdit ? `<section aria-labelledby="table"><h2 id="table">Table height</h2> | ||
| 283 | 286 | ${p.canEdit && p.visitors ? visitorsSection(p.visitors) : ""} |
| 284 | 287 | ${p.precision !== undefined ? precisionSection(p.precision, esc) : ""} |
| 285 | 288 | ${p.canEdit && p.servos !== undefined ? servosSection(p.servos, esc) : ""} |
| 289 | ${p.video !== undefined ? videoSection(p.video, esc) : ""} | |
| 286 | 290 | <section aria-labelledby="health"><h2 id="health">Health</h2>${health(p.health ?? [])}</section> |
| 287 | 291 | <section aria-labelledby="mods"><h2 id="mods">Moderators</h2>${mods}</section> |
| 288 | 292 | <section aria-labelledby="moderation"><h2 id="moderation">Moderation</h2>${inForce("mute", p.muted ?? [], canLift)}${inForce("ban", p.banned ?? [], canLift)}${actions}</section> |
Software/so101/src/admin-routes.ts +8 −4
| @@ -7,9 +7,12 @@ import { sameOrigin } from "./relay.ts"; | ||
| 7 | 7 | import { adminCsp, DONE, renderAdmin, renderRefusal, type Done } from "./admin-page.ts"; |
| 8 | 8 | import type { PrecisionPanel } from "./precision-panel.ts"; |
| 9 | 9 | import type { ServoPanel } from "./servo-panel.ts"; |
| 10 | import type { VideoCard } from "./video-panel.ts"; | |
| 10 | 11 | |
| 11 | 12 | /** Asks the arm for its precision scorecard; only called for someone allowed on the page. */ |
| 12 | 13 | export type AskPrecision = () => Promise<PrecisionPanel | null>; |
| 14 | /** The relay's own numbers for its H.264 cameras (src/video-stats.ts). */ | |
| 15 | export type AskVideo = () => readonly VideoCard[]; | |
| 13 | 16 | |
| 14 | 17 | /** Asks the arm for its servos' control registers; only called for the owner. */ |
| 15 | 18 | export type AskServos = () => Promise<ServoPanel | null>; |
| @@ -28,7 +31,7 @@ export interface AdminSession { | ||
| 28 | 31 | * Answer 404 to anyone not allowed, and the page itself to GET and HEAD. Returns the session for any other |
| 29 | 32 | * method by an allowed caller, and null when the response has been sent. |
| 30 | 33 | */ |
| 31 | export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, askPrecision?: AskPrecision, askServos?: AskServos): Promise<AdminSession | null> { | |
| 34 | export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo): Promise<AdminSession | null> { | |
| 32 | 35 | const who = lobby.who(req); |
| 33 | 36 | const html = (status: number, text: string, edit = false) => { |
| 34 | 37 | res.writeHead(status, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store", "content-security-policy": adminCsp(edit) }); |
| @@ -53,8 +56,9 @@ export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: Se | ||
| 53 | 56 | } |
| 54 | 57 | const precision = (req.method === "GET" || req.method === "HEAD") && askPrecision ? await askPrecision() : undefined; |
| 55 | 58 | const servos = canEdit && (req.method === "GET" || req.method === "HEAD") && askServos ? await askServos() : undefined; |
| 59 | const video = askVideo?.(); | |
| 56 | 60 | const show = (status: number, done: Done | null, error: string | null) => |
| 57 | html(status, renderAdmin({ precision, servos, who, canEdit, settings: lobby.admin.settings, snapshot: lobby.turns.snapshot(), log: lobby.admin.log, removed: lobby.chat.removed, muted: lobby.chat.mutedList(), banned: lobby.accounts.bans(), health: lobby.reports.rows(), reports: lobby.lineReports.list(), visitors: canEdit ? { peak: lobby.visitors.peakToday(), hours: lobby.visitors.joinsPerHour() } : undefined, done, error }), canEdit); | |
| 61 | html(status, renderAdmin({ precision, servos, video, who, canEdit, settings: lobby.admin.settings, snapshot: lobby.turns.snapshot(), log: lobby.admin.log, removed: lobby.chat.removed, muted: lobby.chat.mutedList(), banned: lobby.accounts.bans(), health: lobby.reports.rows(), reports: lobby.lineReports.list(), visitors: canEdit ? { peak: lobby.visitors.peakToday(), hours: lobby.visitors.joinsPerHour() } : undefined, done, error }), canEdit); | |
| 58 | 62 | if (req.method === "GET" || req.method === "HEAD") { |
| 59 | 63 | const done = url.searchParams.get("done"); |
| 60 | 64 | show(200, done && Object.hasOwn(DONE, done) ? (done as Done) : null, null); |
| @@ -73,8 +77,8 @@ const REPORT_ACTIONS = new Set(["report-remove", "report-mute", "report-dismiss" | ||
| 73 | 77 | type ReadBody = (req: IncomingMessage, limit?: number) => Promise<Buffer>; |
| 74 | 78 | |
| 75 | 79 | /** The owner's page. Anyone else is told so; a change is a form post, answered with a redirect back. */ |
| 76 | export async function handleAdmin(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, readBody: ReadBody, askPrecision?: AskPrecision, askServos?: AskServos): Promise<void> { | |
| 77 | const session = await serveAdminPage(lobby, req, res, url, askPrecision, askServos); | |
| 80 | export async function handleAdmin(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, readBody: ReadBody, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo): Promise<void> { | |
| 81 | const session = await serveAdminPage(lobby, req, res, url, askPrecision, askServos, askVideo); | |
| 78 | 82 | if (!session) return; |
| 79 | 83 | const { who, canEdit, show } = session; |
| 80 | 84 | if (req.method !== "POST") { |
Software/so101/src/h264-clip.ts +19 −6
| @@ -1,16 +1,29 @@ | ||
| 1 | 1 | /** A stand-in for the Pi's encoder in tests: a short Annex-B clip with the shape of the real one (SPS, PPS, IDR, then P pictures; no B-frames). */ |
| 2 | 2 | |
| 3 | const nal = (type: number, size: number, fill: number) => Buffer.concat([Buffer.from([0, 0, 0, 1, type | 0x60]), Buffer.alloc(size, fill)]); | |
| 3 | // A slice's payload opens with first_mb_in_slice = 0, coded as the single bit 1: the top bit of its first byte. | |
| 4 | const nal = (type: number, size: number, fill: number) => Buffer.concat([Buffer.from([0, 0, 0, 1, type | 0x60, 0x80 | fill]), Buffer.alloc(Math.max(0, size - 1), fill)]); | |
| 4 | 5 | // SPS body: profile 66 (baseline), constraints 0xc0, level 31, then filler that contains no start code. |
| 5 | 6 | const SPS = Buffer.concat([Buffer.from([0, 0, 0, 1, 0x67, 66, 0xc0, 31]), Buffer.alloc(6, 0x11)]); |
| 6 | 7 | const PPS = nal(8, 3, 0x22); |
| 8 | const AUD = Buffer.from([0, 0, 0, 1, 0x09, 0xf0]); | |
| 7 | 9 | |
| 8 | /** One picture: `key` pictures carry the parameter sets; `size` is the slice's payload in bytes. */ | |
| 9 | export function picture(key: boolean, size: number): Buffer { | |
| 10 | return key ? Buffer.concat([SPS, PPS, nal(5, size, 0x33)]) : nal(1, size, 0x44); | |
| 10 | export interface Shape { | |
| 11 | /** Parameter sets on every keyframe (the default), or only on the first picture of the stream. */ | |
| 12 | sets?: "every-key" | "head"; | |
| 13 | /** An access unit delimiter before every picture. */ | |
| 14 | aud?: boolean; | |
| 15 | } | |
| 16 | ||
| 17 | /** One picture: `key` pictures carry the parameter sets unless `sets` is false; `size` is the slice's payload in bytes. */ | |
| 18 | export function picture(key: boolean, size: number, sets = true, aud = false): Buffer { | |
| 19 | const body = key ? Buffer.concat([...(sets ? [SPS, PPS] : []), nal(5, size, 0x33)]) : nal(1, size, 0x44); | |
| 20 | return aud ? Buffer.concat([AUD, body]) : body; | |
| 11 | 21 | } |
| 12 | 22 | |
| 13 | 23 | /** `seconds` of video at `fps` with a keyframe every `gop` pictures; keyframes are 8x a P picture's size. */ |
| 14 | export function clip(seconds: number, fps: number, gop: number, deltaBytes: number): Buffer[] { | |
| 15 | return Array.from({ length: Math.round(seconds * fps) }, (_, i) => picture(i % gop === 0, i % gop === 0 ? deltaBytes * 8 : deltaBytes)); | |
| 24 | export function clip(seconds: number, fps: number, gop: number, deltaBytes: number, shape: Shape = {}): Buffer[] { | |
| 25 | return Array.from({ length: Math.round(seconds * fps) }, (_, i) => { | |
| 26 | const key = i % gop === 0; | |
| 27 | return picture(key, key ? deltaBytes * 8 : deltaBytes, shape.sets === "head" ? i === 0 : true, shape.aud === true); | |
| 28 | }); | |
| 16 | 29 | } |
Software/so101/src/h264-replay.test.ts added +173
| @@ -0,0 +1,173 @@ | ||
| 1 | /** | |
| 2 | * A captured Annex-B stream replayed through the relay into the page's decoder path: a Pi that serves the file at | |
| 3 | * 15 pictures a second, the relay as it runs (startRelay with ARM_H264_1), and the page's own VideoPlayer | |
| 4 | * (src/ui/video-player.js) on a real WebSocket, decoding into a strict stand-in for WebCodecs that refuses what a | |
| 5 | * browser's decoder would: a first keyframe without SPS and PPS, a delta before any keyframe, half a picture, a | |
| 6 | * timestamp that does not rise. | |
| 7 | * | |
| 8 | * The capture: ARM_H264_CAPTURE=<file> names one, recorded on the Pi's network with | |
| 9 | * curl -s --max-time 10 http://100.89.95.68:8091/h264 > cam1.h264 | |
| 10 | * Without it, a synthetic one of the same shape is used: 480p-sized pictures at about 28 KB/s, a keyframe every 15, | |
| 11 | * an AUD before each, and SPS and PPS only at the head of the stream, the case that froze a late joiner. | |
| 12 | */ | |
| 13 | import { afterEach, describe, expect, test } from "bun:test"; | |
| 14 | import { mkdtempSync, readFileSync, rmSync } from "node:fs"; | |
| 15 | import { createServer, type Server } from "node:http"; | |
| 16 | import { tmpdir } from "node:os"; | |
| 17 | import { join } from "node:path"; | |
| 18 | import WebSocket from "ws"; | |
| 19 | import { CameraHub } from "./cameras.ts"; | |
| 20 | import { clip, type Shape } from "./h264-clip.ts"; | |
| 21 | import { AccessUnitSplitter, nalsOf } from "./h264.ts"; | |
| 22 | import { Lobby } from "./lobby.ts"; | |
| 23 | import { startRelay, type RunningRelay } from "./relay.ts"; | |
| 24 | import { VideoPlayer } from "./ui/video-player.js"; | |
| 25 | ||
| 26 | const FPS = 15; | |
| 27 | const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); | |
| 28 | ||
| 29 | /** The capture's pictures, in order: the file cut into access units, the last one included. */ | |
| 30 | function pictures(shape: Shape): Buffer[] { | |
| 31 | const file = process.env.ARM_H264_CAPTURE; | |
| 32 | if (!file) return clip(10, FPS, 15, 1000, shape); | |
| 33 | const data = readFileSync(file); | |
| 34 | const units = new AccessUnitSplitter().push(Buffer.concat([data, Buffer.from([0, 0, 0, 1, 0x09, 0xf0])])); // an AUD closes the last | |
| 35 | return units.map((u) => u.data); | |
| 36 | } | |
| 37 | ||
| 38 | let dir = ""; | |
| 39 | let relay: RunningRelay | undefined; | |
| 40 | let lobby: Lobby | undefined; | |
| 41 | let pi: Server | undefined; | |
| 42 | const players: VideoPlayer[] = []; | |
| 43 | afterEach(async () => { | |
| 44 | players.splice(0).forEach((p) => p.stop()); | |
| 45 | await Promise.race([relay?.close(), sleep(500)]); | |
| 46 | lobby?.close(); | |
| 47 | pi?.closeAllConnections(); | |
| 48 | pi?.close(); | |
| 49 | if (dir) rmSync(dir, { recursive: true, force: true }); | |
| 50 | dir = ""; | |
| 51 | relay = lobby = pi = undefined; | |
| 52 | }); | |
| 53 | ||
| 54 | /** The Pi: each connection gets the capture from `from`, a picture every 1/FPS s, looping; `stop()` ends every stream. */ | |
| 55 | async function boot(list: Buffer[], from = 0) { | |
| 56 | const open = new Set<() => void>(); | |
| 57 | pi = createServer((req, res) => { | |
| 58 | if (req.url !== "/h264") return void res.writeHead(404).end(); | |
| 59 | res.writeHead(200, { "content-type": "video/h264" }); | |
| 60 | let i = from; | |
| 61 | const timer = setInterval(() => res.write(list[i++ % list.length]!), 1000 / FPS); | |
| 62 | const end = () => (clearInterval(timer), res.destroy()); | |
| 63 | open.add(end); | |
| 64 | res.on("close", () => (clearInterval(timer), open.delete(end))); | |
| 65 | }); | |
| 66 | await new Promise<void>((r) => pi!.listen(0, "127.0.0.1", r)); | |
| 67 | const port = (pi.address() as { port: number }).port; | |
| 68 | dir = mkdtempSync(join(tmpdir(), "so101-replay-")); | |
| 69 | lobby = new Lobby({ identityKey: "k".repeat(64), adminFile: join(dir, "admin.json") }); | |
| 70 | const cameras = new CameraHub({ seed: [{ name: "One", source: "http://127.0.0.1:9/stream" }] }); | |
| 71 | relay = await startRelay({ port: 0, simulation: "http://127.0.0.1:1", lobby, cameras, video: { 1: { "480p": `http://127.0.0.1:${port}/h264` } }, env: {} }); | |
| 72 | return { url: `ws://127.0.0.1:${relay.port}/camera/1/video`, stop: () => [...open].forEach((end) => end()) }; | |
| 73 | } | |
| 74 | ||
| 75 | /** A decoder that holds the stream to what a browser's would accept, and draws each picture at once. */ | |
| 76 | function strictDecoder() { | |
| 77 | const seen = { pictures: 0, refused: [] as string[], configs: [] as string[], drawnAt: [] as number[] }; | |
| 78 | class Decoder { | |
| 79 | decodeQueueSize = 0; | |
| 80 | private sets = false; | |
| 81 | private key = false; | |
| 82 | private last = -Infinity; | |
| 83 | private readonly init: { output: (f: unknown) => void; error: (e: unknown) => void }; | |
| 84 | constructor(init: { output: (f: unknown) => void; error: (e: unknown) => void }) { | |
| 85 | this.init = init; | |
| 86 | } | |
| 87 | configure(c: { codec: string }) { | |
| 88 | if (!/^avc1\.[0-9a-f]{6}$/.test(c.codec)) throw new TypeError(`codec ${c.codec}`); | |
| 89 | seen.configs.push(c.codec); | |
| 90 | } | |
| 91 | decode(chunk: { type: string; timestamp: number; data: Uint8Array }) { | |
| 92 | const types = nalsOf(chunk.data).map((n) => n.type); | |
| 93 | const why = | |
| 94 | chunk.timestamp <= this.last ? "timestamp did not rise" | |
| 95 | : chunk.type === "key" && !this.sets && !(types.includes(7) && types.includes(8)) ? "first keyframe without SPS and PPS" | |
| 96 | : chunk.type === "delta" && !this.key ? "delta before a keyframe" | |
| 97 | : types.filter((t) => t === 1 || t === 5).length !== 1 ? "not one picture" | |
| 98 | : null; | |
| 99 | if (why) return (seen.refused.push(why), this.init.error(new Error(why))); | |
| 100 | this.last = chunk.timestamp; | |
| 101 | this.sets ||= types.includes(7); | |
| 102 | this.key ||= chunk.type === "key"; | |
| 103 | seen.pictures++; | |
| 104 | seen.drawnAt.push(Date.now()); | |
| 105 | this.init.output({ displayWidth: 854, displayHeight: 480, timestamp: chunk.timestamp, close() {} }); | |
| 106 | } | |
| 107 | close() {} | |
| 108 | } | |
| 109 | return { Decoder, seen }; | |
| 110 | } | |
| 111 | ||
| 112 | /** The page's player on `url`, with the strict decoder; its states, and when it started. */ | |
| 113 | function watch(url: string, quietMs?: number) { | |
| 114 | const { Decoder, seen } = strictDecoder(); | |
| 115 | const states: { state: string; why?: string }[] = []; | |
| 116 | const canvas = { width: 0, height: 0, getContext: () => ({ drawImage() {} }) }; | |
| 117 | const player = new VideoPlayer({ url, canvas, quietMs, onStatus: (s) => states.push(s), deps: { WebSocket, VideoDecoder: Decoder, EncodedVideoChunk: class { constructor(init: object) { Object.assign(this, init); } } } }); | |
| 118 | const started = Date.now(); | |
| 119 | player.start(); | |
| 120 | players.push(player); | |
| 121 | return { player, seen, states, started }; | |
| 122 | } | |
| 123 | ||
| 124 | /** Pictures a second over the drawn times after the first. */ | |
| 125 | const fps = (at: number[]) => (at.length < 2 ? 0 : ((at.length - 1) * 1000) / (at.at(-1)! - at[0]!)); | |
| 126 | ||
| 127 | describe("a capture replayed through the relay into the page's decoder", () => { | |
| 128 | for (const [name, shape, from] of [ | |
| 129 | ["SPS and PPS only at the head of the stream", { sets: "head", aud: true }, 0], | |
| 130 | ["SPS and PPS on every keyframe, the relay joining the Pi's stream mid-GOP", { sets: "every-key" }, 7], | |
| 131 | ] as const) { | |
| 132 | test(`${name}: a late joiner plays from the next keyframe within 1 s, at 10-15 fps, nothing refused`, async () => { | |
| 133 | const list = pictures(shape); | |
| 134 | const { url } = await boot(list, from); | |
| 135 | const first = watch(url); // pulls the Pi | |
| 136 | await sleep(1300); // the second joins mid-GOP, long after the head of the stream went by | |
| 137 | const late = watch(url); | |
| 138 | await sleep(2200); | |
| 139 | for (const v of [first, late]) { | |
| 140 | expect(v.seen.refused).toEqual([]); | |
| 141 | expect(v.states.map((s) => s.state)).toEqual(["live"]); | |
| 142 | expect(v.seen.pictures).toBeGreaterThan(15); | |
| 143 | expect(fps(v.seen.drawnAt.slice(1))).toBeGreaterThanOrEqual(10); // the first may be the cached keyframe, drawn before the stream catches up | |
| 144 | expect(fps(v.seen.drawnAt.slice(1))).toBeLessThanOrEqual(16); | |
| 145 | } | |
| 146 | expect(late.seen.drawnAt[0]! - late.started).toBeLessThan(1000); | |
| 147 | expect(late.seen.configs).toEqual(["avc1.42c01f"]); | |
| 148 | // What the pages measured reaches the scorecard: decoded fps and the bytes from the Pi. | |
| 149 | late.player.report(); | |
| 150 | await sleep(100); | |
| 151 | const card = relay!.videoCards()[0]!; | |
| 152 | expect(card.camera).toBe(1); | |
| 153 | expect(card.viewers).toBe(2); | |
| 154 | expect(card.fpsP50).toBeGreaterThanOrEqual(10); | |
| 155 | expect(card.bytesPerS).toBeGreaterThan(5_000); | |
| 156 | }, 10_000); | |
| 157 | } | |
| 158 | ||
| 159 | test("when the Pi's stream stops, the page goes back to MJPEG within 2 s instead of holding the last picture", async () => { | |
| 160 | const { url, stop } = await boot(pictures({ sets: "head", aud: true })); | |
| 161 | const v = watch(url, 600); | |
| 162 | await sleep(800); | |
| 163 | expect(v.states.map((s) => s.state)).toEqual(["live"]); | |
| 164 | pi!.close(); // and it does not come back | |
| 165 | stop(); | |
| 166 | const stopped = Date.now(); | |
| 167 | while (v.states.length < 2 && Date.now() - stopped < 2500) await sleep(20); | |
| 168 | expect(v.states.at(-1)).toEqual(expect.objectContaining({ state: "none", why: "quiet" })); | |
| 169 | expect(Date.now() - stopped).toBeLessThan(2000); | |
| 170 | await sleep(100); | |
| 171 | expect(relay!.videoCards()[0]!.fallbacks.quiet).toBe(1); | |
| 172 | }, 10_000); | |
| 173 | }); |
Software/so101/src/h264.test.ts +46 −4
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | import { describe, expect, test } from "bun:test"; |
| 2 | import { clip } from "./h264-clip.ts"; | |
| 3 | import { AccessUnitSplitter } from "./h264.ts"; | |
| 2 | import { clip, picture } from "./h264-clip.ts"; | |
| 3 | import { AccessUnitSplitter, nalsOf, ParameterSets } from "./h264.ts"; | |
| 4 | 4 | |
| 5 | 5 | describe("access units", () => { |
| 6 | 6 | const pictures = clip(1, 10, 5, 100); |
| @@ -20,10 +20,25 @@ describe("access units", () => { | ||
| 20 | 20 | } |
| 21 | 21 | }); |
| 22 | 22 | |
| 23 | test("a picture is out as soon as the next one begins, not when it ends", () => { | |
| 23 | test("a picture is out as soon as the next one begins (its start code, header and first byte), not when it ends", () => { | |
| 24 | 24 | const splitter = new AccessUnitSplitter(); |
| 25 | 25 | expect(splitter.push(pictures[0]!)).toEqual([]); |
| 26 | expect(splitter.push(pictures[1]!.subarray(0, 5))).toHaveLength(1); | |
| 26 | expect(splitter.push(pictures[1]!.subarray(0, 5))).toEqual([]); | |
| 27 | expect(splitter.push(pictures[1]!.subarray(5, 6))).toHaveLength(1); | |
| 28 | }); | |
| 29 | ||
| 30 | test("a picture cut into several slices stays one unit: only a slice with first_mb_in_slice 0 opens a picture", () => { | |
| 31 | const second = Buffer.from([0, 0, 0, 1, 0x61, 0x20, 0x44, 0x44]); // a slice of the same P picture: first_mb_in_slice > 0 | |
| 32 | const stream = Buffer.concat([pictures[0]!, pictures[1]!, second, pictures[2]!, pictures[3]!]); | |
| 33 | const units = new AccessUnitSplitter().push(stream); | |
| 34 | expect(units.map((u) => u.data.length)).toEqual([pictures[0]!.length, pictures[1]!.length + second.length, pictures[2]!.length]); | |
| 35 | }); | |
| 36 | ||
| 37 | test("access unit delimiters open a picture too", () => { | |
| 38 | const aud = clip(1, 10, 5, 50, { aud: true }); | |
| 39 | const units = new AccessUnitSplitter().push(Buffer.concat(aud)); | |
| 40 | expect(units.map((u) => u.data.equals(aud[units.indexOf(u)]!))).toEqual(aud.slice(0, -1).map(() => true)); | |
| 41 | expect(units.map((u) => u.key)).toEqual(aud.slice(0, -1).map((_, i) => i % 5 === 0)); | |
| 27 | 42 | }); |
| 28 | 43 | |
| 29 | 44 | test("refuses a stream that never ends a unit", () => { |
| @@ -33,3 +48,30 @@ describe("access units", () => { | ||
| 33 | 48 | }).toThrow(); |
| 34 | 49 | }); |
| 35 | 50 | }); |
| 51 | ||
| 52 | describe("parameter sets", () => { | |
| 53 | const types = (b: Buffer) => nalsOf(b).map((n) => n.type); | |
| 54 | ||
| 55 | test("a keyframe without SPS and PPS gets the last ones seen, after its AUD; one that has them is left alone", () => { | |
| 56 | const sets = new ParameterSets(); | |
| 57 | const head = picture(true, 80, true, true); | |
| 58 | expect(sets.complete({ key: true, data: head }).data.equals(head)).toBe(true); | |
| 59 | expect(sets.known).toBe(true); | |
| 60 | const bare = picture(true, 80, false, true); | |
| 61 | expect(types(bare)).toEqual([9, 5]); | |
| 62 | const done = sets.complete({ key: true, data: bare }); | |
| 63 | expect(types(done.data)).toEqual([9, 7, 8, 5]); | |
| 64 | expect(done.data.equals(head)).toBe(true); | |
| 65 | const delta = picture(false, 20); | |
| 66 | expect(sets.complete({ key: false, data: delta }).data).toBe(delta); | |
| 67 | }); | |
| 68 | ||
| 69 | test("before any SPS a bare keyframe goes as it is; new sets replace the old", () => { | |
| 70 | const sets = new ParameterSets(); | |
| 71 | const bare = picture(true, 80, false); | |
| 72 | expect(sets.complete({ key: true, data: bare }).data).toBe(bare); | |
| 73 | const other = Buffer.concat([Buffer.from([0, 0, 0, 1, 0x67, 100, 0, 40, 0x11]), Buffer.from([0, 0, 0, 1, 0x68, 0xee]), picture(true, 10, false)]); | |
| 74 | sets.complete({ key: true, data: other }); | |
| 75 | expect([...sets.complete({ key: true, data: bare }).data.subarray(0, 8)]).toEqual([0, 0, 0, 1, 0x67, 100, 0, 40]); | |
| 76 | }); | |
| 77 | }); |
Software/so101/src/h264.ts +62 −8
| @@ -1,9 +1,11 @@ | ||
| 1 | 1 | /** |
| 2 | 2 | * H.264 in Annex-B (start-code) form, cut into access units as the bytes arrive. |
| 3 | 3 | * |
| 4 | * An access unit is one picture: optional parameter sets (SPS/PPS), then its slices. The encoder on the Pi sends one | |
| 5 | * slice per picture and no B-frames, so a picture ends the moment the next NAL that opens one begins: the unit | |
| 6 | * leaves as soon as the next start code and its header byte are seen, never waiting for the next picture to finish. | |
| 4 | * An access unit is one picture: optional parameter sets (SPS/PPS), then its slices. The encoder on the Pi sends no | |
| 5 | * B-frames, so a picture ends the moment the next NAL that opens one begins: an AUD, SEI or parameter set, or a slice | |
| 6 | * whose first_mb_in_slice is 0 (the top bit of its first byte), so a picture cut into several slices stays one unit. | |
| 7 | * The unit leaves as soon as the next start code and the byte after its header are seen, never waiting for the next | |
| 8 | * picture to finish. | |
| 7 | 9 | */ |
| 8 | 10 | |
| 9 | 11 | export interface AccessUnit { |
| @@ -14,7 +16,8 @@ export interface AccessUnit { | ||
| 14 | 16 | } |
| 15 | 17 | |
| 16 | 18 | /** NAL types that open a new access unit once a picture's slice has been seen. */ |
| 17 | const OPENS = new Set([1, 5, 6, 7, 8, 9]); | |
| 19 | const OPENS = new Set([6, 7, 8, 9]); | |
| 20 | const isSlice = (type: number) => type === 1 || type === 5; | |
| 18 | 21 | /** No unit is larger than this; a stream that never finds a start code is cut off by throwing. */ |
| 19 | 22 | export const MAX_UNIT = 4 << 20; |
| 20 | 23 | |
| @@ -31,23 +34,24 @@ export class AccessUnitSplitter { | ||
| 31 | 34 | this.buf = this.buf.length ? Buffer.concat([this.buf, chunk]) : Buffer.from(chunk); |
| 32 | 35 | const out: AccessUnit[] = []; |
| 33 | 36 | const b = this.buf; |
| 34 | for (let i = this.scan; i + 3 < b.length; i++) { | |
| 37 | for (let i = this.scan; i + 4 < b.length; i++) { | |
| 35 | 38 | if (b[i + 2]! > 1) i += 2; // cannot be inside a start code |
| 36 | 39 | else if (b[i] === 0 && b[i + 1] === 0 && b[i + 2] === 1) { |
| 37 | 40 | const type = b[i + 3]! & 0x1f; |
| 38 | 41 | const start = i > this.from && b[i - 1] === 0 ? i - 1 : i; |
| 39 | if (this.picture && OPENS.has(type) && start > this.from) { | |
| 42 | const opens = OPENS.has(type) || (isSlice(type) && (b[i + 4]! & 0x80) !== 0); | |
| 43 | if (this.picture && opens && start > this.from) { | |
| 40 | 44 | out.push({ key: this.key, data: Buffer.from(b.subarray(this.from, start)) }); |
| 41 | 45 | this.from = start; |
| 42 | 46 | this.picture = this.key = false; |
| 43 | 47 | } |
| 44 | if (type === 1 || type === 5) this.picture = true; | |
| 48 | if (isSlice(type)) this.picture = true; | |
| 45 | 49 | if (type === 5) this.key = true; |
| 46 | 50 | i += 3; |
| 47 | 51 | this.scan = i + 1; |
| 48 | 52 | } |
| 49 | 53 | } |
| 50 | this.scan = Math.max(this.scan, b.length - 3); | |
| 54 | this.scan = Math.max(this.scan, b.length - 4); | |
| 51 | 55 | if (this.from > 0) { |
| 52 | 56 | this.buf = Buffer.from(b.subarray(this.from)); |
| 53 | 57 | this.scan -= this.from; |
| @@ -57,3 +61,53 @@ export class AccessUnitSplitter { | ||
| 57 | 61 | return out; |
| 58 | 62 | } |
| 59 | 63 | } |
| 64 | ||
| 65 | export interface Nal { | |
| 66 | type: number; | |
| 67 | /** Where its start code begins, and where the next one does (or the end). */ | |
| 68 | start: number; | |
| 69 | end: number; | |
| 70 | } | |
| 71 | ||
| 72 | /** The NAL units of one Annex-B access unit, in order. */ | |
| 73 | export function nalsOf(data: Uint8Array): Nal[] { | |
| 74 | const out: Nal[] = []; | |
| 75 | for (let i = 0; i + 3 < data.length; i++) { | |
| 76 | if (data[i + 2]! > 1) i += 2; | |
| 77 | else if (data[i] === 0 && data[i + 1] === 0 && data[i + 2] === 1) { | |
| 78 | const start = i > 0 && data[i - 1] === 0 ? i - 1 : i; | |
| 79 | if (out.length) out.at(-1)!.end = start; | |
| 80 | out.push({ type: data[i + 3]! & 0x1f, start, end: data.length }); | |
| 81 | i += 3; | |
| 82 | } | |
| 83 | } | |
| 84 | return out; | |
| 85 | } | |
| 86 | ||
| 87 | /** | |
| 88 | * One rendition's parameter sets, kept from wherever the encoder last sent them, so that every keyframe a viewer can | |
| 89 | * start on carries an SPS and a PPS. An encoder that sends them only at the head of its stream (or a server that | |
| 90 | * passes a late joiner the stream from mid-way) otherwise leaves a viewer that joined later, or a decoder that had to | |
| 91 | * start over after an error, with IDR pictures it cannot configure from: one picture, then a frozen one. | |
| 92 | */ | |
| 93 | export class ParameterSets { | |
| 94 | private sps: Buffer | null = null; | |
| 95 | private pps: Buffer | null = null; | |
| 96 | ||
| 97 | get known(): boolean { | |
| 98 | return this.sps !== null && this.pps !== null; | |
| 99 | } | |
| 100 | ||
| 101 | /** Learn from `unit` and, when it is a keyframe without both of its own, give it the last ones seen. */ | |
| 102 | complete(unit: AccessUnit): AccessUnit { | |
| 103 | const nals = nalsOf(unit.data); | |
| 104 | const sps = nals.filter((n) => n.type === 7), pps = nals.filter((n) => n.type === 8); | |
| 105 | const take = (list: Nal[]) => Buffer.concat(list.map((n) => unit.data.subarray(n.start, n.end))); | |
| 106 | if (sps.length) this.sps = take(sps); | |
| 107 | if (pps.length) this.pps = take(pps); | |
| 108 | if (!unit.key || (sps.length && pps.length) || !this.known) return unit; | |
| 109 | // AUD first, then SPS and PPS, then the rest: the order a decoder reads them in. | |
| 110 | const part = (keep: (type: number) => boolean) => take(nals.filter((n) => keep(n.type))); | |
| 111 | return { key: true, data: Buffer.concat([part((t) => t === 9), this.sps!, this.pps!, part((t) => t !== 9 && t !== 7 && t !== 8)]) }; | |
| 112 | } | |
| 113 | } |
Software/so101/src/relay-video.test.ts +16
| @@ -6,6 +6,7 @@ import { tmpdir } from "node:os"; | ||
| 6 | 6 | import { join } from "node:path"; |
| 7 | 7 | import WebSocket from "ws"; |
| 8 | 8 | import { CameraHub } from "./cameras.ts"; |
| 9 | import { sign } from "./identity.ts"; | |
| 9 | 10 | import { clip } from "./h264-clip.ts"; |
| 10 | 11 | import { Lobby } from "./lobby.ts"; |
| 11 | 12 | import { startRelay, type RunningRelay } from "./relay.ts"; |
| @@ -91,3 +92,18 @@ test("a camera with no source, and a cross-origin upgrade, are refused; the same | ||
| 91 | 92 | expect(await upgrade(host, "/camera/1/video", { Origin: "https://evil.example" })).not.toContain("101"); |
| 92 | 93 | expect(await upgrade(host, "/camera/1/video", { Origin: `http://${host}` })).toContain("101"); |
| 93 | 94 | }); |
| 95 | ||
| 96 | test("the owner's GET /precision carries each H.264 camera's numbers under video, even with the arm away, and /admin shows them", async () => { | |
| 97 | const host = await boot(); | |
| 98 | const identity = `id-trevato|trevato|admin|${Date.now() + 600_000}`; | |
| 99 | const owner = { "x-forge-identity": identity, "x-forge-signature": sign(identity, "k".repeat(64)) }; | |
| 100 | const answer = await fetch(`http://${host}/precision`, { headers: owner }); | |
| 101 | expect(answer.status).toBe(502); // the simulation at :1 does not answer | |
| 102 | const body = (await answer.json()) as { video: { camera: number; viewers: number; fallbacks: Record<string, number> }[] }; | |
| 103 | expect(body.video.map((c) => c.camera)).toEqual([1]); | |
| 104 | expect(body.video[0]!.fallbacks).toEqual({ quiet: 0, decode: 0, unsupported: 0, closed: 0 }); | |
| 105 | expect((await fetch(`http://${host}/precision`)).status).toBe(404); | |
| 106 | const html = await (await fetch(`http://${host}/admin`, { headers: owner })).text(); | |
| 107 | expect(html).toContain('<h2 id="video">Video</h2>'); | |
| 108 | expect(html).toContain("<h3>Camera 1</h3>"); | |
| 109 | }); |
Software/so101/src/relay.ts +13 −5
| @@ -36,7 +36,8 @@ | ||
| 36 | 36 | * POST /join {"name": "…"}: become someone here, no forge account needed (src/accounts.ts); the answer |
| 37 | 37 | * sets the HttpOnly session cookie and names you: {"you": {"id", "name", "owner"}} |
| 38 | 38 | * GET /admin, POST /admin the owner's page: pause, end a turn, take control, the settings, the log |
| 39 | * GET /precision?hours=1-720 owner and moderators: the arm's precision scorecard (src/precision-history.ts); 404 to anyone else | |
| 39 | * GET /precision?hours=1-720 owner and moderators: the arm's precision scorecard (src/precision-history.ts), with the relay's | |
| 40 | * H.264 numbers per camera under "video" (src/video-stats.ts); 404 to anyone else | |
| 40 | 41 | * POST /precision/snap-back the driver's page: {cause, sizeDeg} for a snap-back only the page saw (cause from src/precision.ts), 400 for any other |
| 41 | 42 | * GET /link {"mode": "live"|"reconnecting"|"offline"|"simulated", "host": "<host:port>"|null, "problem": "…"|null} |
| 42 | 43 | * GET /status the arm's own /status, and which arm said it: "source": "live <host:port>", "reconnecting to the arm: |
| @@ -78,6 +79,7 @@ import { readServos } from "./servo-panel.ts"; | ||
| 78 | 79 | import { parseFps, type FeedOptions } from "./camera-feed.ts"; |
| 79 | 80 | import { CameraError, CameraHub } from "./cameras.ts"; |
| 80 | 81 | import { VideoFeed, type VideoSources } from "./video-feed.ts"; |
| 82 | import type { VideoCard } from "./video-panel.ts"; | |
| 81 | 83 | import { LADDER } from "./video-fanout.ts"; |
| 82 | 84 | import { LOCAL as LOCAL_PERSON } from "./identity.ts"; |
| 83 | 85 | import { LeaderCalibrations } from "./leader-calibrations.ts"; |
| @@ -149,6 +151,8 @@ export interface RelayOptions { | ||
| 149 | 151 | export interface RunningRelay { |
| 150 | 152 | readonly port: number; |
| 151 | 153 | link(): Link; |
| 154 | /** Each H.264 camera's numbers for the scorecard, as GET /precision gives them under `video`. */ | |
| 155 | videoCards(): VideoCard[]; | |
| 152 | 156 | close(): Promise<void>; |
| 153 | 157 | } |
| 154 | 158 | |
| @@ -472,7 +476,9 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 472 | 476 | return null; |
| 473 | 477 | } |
| 474 | 478 | }; |
| 475 | /** GET /precision: the scorecard as JSON, for the owner and moderators, who see it on /admin; to anyone else there is no such route. */ | |
| 479 | /** Each H.264 camera's numbers for the scorecard: decoded fps, bytes from the Pi, relay-to-glass latency, fallbacks. */ | |
| 480 | const videoCards = (): VideoCard[] => [...videos].filter(([, feed]) => feed.available).map(([camera, feed]) => ({ camera, ...feed.summary() })); | |
| 481 | /** GET /precision: the scorecard as JSON (the arm's, with the relay's `video` beside it), for the owner and moderators, who see it on /admin; to anyone else there is no such route. */ | |
| 476 | 482 | const precisionRead = async (req: IncomingMessage, res: ServerResponse, url: URL) => { |
| 477 | 483 | if (!lobby || !lobby.who(req) || !lobby.moderates(req)) return json(res, 404, { error: "no such route" }); |
| 478 | 484 | if (req.method !== "GET") { |
| @@ -483,9 +489,10 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 483 | 489 | if (hours !== null && !(/^\d{1,3}$/.test(hours) && Number(hours) >= 1 && Number(hours) <= 720)) return json(res, 400, { error: "hours must be 1-720" }); |
| 484 | 490 | try { |
| 485 | 491 | const answer = await armPrecision(hours); |
| 486 | return json(res, answer.status, await answer.json()); | |
| 492 | const body = (await answer.json()) as unknown; | |
| 493 | return json(res, answer.status, body && typeof body === "object" && !Array.isArray(body) ? { ...body, video: videoCards() } : body); | |
| 487 | 494 | } catch (err) { |
| 488 | return json(res, 502, { error: `the arm did not answer /precision (${reason(err)})` }); | |
| 495 | return json(res, 502, { error: `the arm did not answer /precision (${reason(err)})`, video: videoCards() }); | |
| 489 | 496 | } |
| 490 | 497 | }; |
| 491 | 498 | /** POST /precision/snap-back (the driver's page: {cause, sizeDeg}): checked here, then added to the arm's scorecard beside its own. */ |
| @@ -689,7 +696,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 689 | 696 | const cameraAdmin = ADMIN_CAMERA.exec(url.pathname); |
| 690 | 697 | if (cameraAdmin) return await administer(req, res, cameraAdmin[1] === undefined ? null : Number(cameraAdmin[1])); |
| 691 | 698 | if (lobby && url.pathname === "/admin/lift") return await handleLift(lobby, req, res, readBody); |
| 692 | if (lobby && (url.pathname === "/admin" || url.pathname === "/admin/health")) return await handleAdmin(lobby, req, res, url, readBody, askPrecision, askServos); | |
| 699 | if (lobby && (url.pathname === "/admin" || url.pathname === "/admin/health")) return await handleAdmin(lobby, req, res, url, readBody, askPrecision, askServos, videoCards); | |
| 693 | 700 | if (url.pathname === "/precision") return await precisionRead(req, res, url); |
| 694 | 701 | if (url.pathname === LEADER) return await leaderCalibration(req, res); |
| 695 | 702 | if (url.pathname === "/join") { |
| @@ -1044,6 +1051,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 1044 | 1051 | return { |
| 1045 | 1052 | port: typeof address === "object" && address ? address.port : options.port ?? 8102, |
| 1046 | 1053 | link, |
| 1054 | videoCards, | |
| 1047 | 1055 | close: () => |
| 1048 | 1056 | new Promise<void>((resolve) => { |
| 1049 | 1057 | clearInterval(timer); |
Software/so101/src/ui/video-player.js +108 −22
| @@ -5,10 +5,22 @@ | ||
| 5 | 5 | * 10 header bytes (keyframe flag, rendition index, relay time float64) then one Annex-B access unit. |
| 6 | 6 | * |
| 7 | 7 | * Nothing is queued: a picture that arrives while the decoder is behind is dropped, and so is every one after it |
| 8 | * until the next keyframe. `onStatus` hears "live", "none" (no picture in `waitMs`: use MJPEG) and the frame age. | |
| 8 | * until the next keyframe. A decoder that fails is made again at the next keyframe (the relay puts SPS and PPS on | |
| 9 | * every one). `onStatus` hears "live", and "none" with why when the page should use MJPEG instead: no picture drawn | |
| 10 | * for QUIET_MS (at the start or at any time after), the decoder failing DECODE_ERRORS times without a picture between, | |
| 11 | * a codec this browser cannot configure, or the socket closing. It never stops on a frozen picture. | |
| 12 | * | |
| 13 | * Once a second it tells the relay what it measured: the frame age, the frames a second it drew, and the latency from | |
| 14 | * the relay to the glass, on the relay's clock, the offset taken from pings over the same socket. | |
| 9 | 15 | */ |
| 10 | 16 | |
| 11 | 17 | export const HEADER = 10; |
| 18 | /** No picture drawn for this long and the page goes back to MJPEG. */ | |
| 19 | export const QUIET_MS = 2000; | |
| 20 | /** Decoder failures in a row, with no picture between, before giving up on H.264. */ | |
| 21 | export const DECODE_ERRORS = 3; | |
| 22 | /** How often the clock offset to the relay is measured again. */ | |
| 23 | export const PING_MS = 5000; | |
| 12 | 24 | |
| 13 | 25 | /** @param {ArrayBuffer | Uint8Array} data */ |
| 14 | 26 | export function parseFrame(data) { |
| @@ -30,10 +42,16 @@ export function codecOf(unit) { | ||
| 30 | 42 | |
| 31 | 43 | export const canDecode = () => typeof VideoDecoder === "function" && typeof EncodedVideoChunk === "function" && typeof WebSocket === "function"; |
| 32 | 44 | |
| 45 | /** The middle of some numbers, or null. @param {number[]} v */ | |
| 46 | const middle = (v) => (v.length ? [...v].sort((a, b) => a - b)[v.length >> 1] : null); | |
| 47 | ||
| 33 | 48 | export class VideoPlayer { |
| 34 | 49 | /** |
| 35 | * @param {{ url: string, canvas: { width: number, height: number, getContext(k: string): any }, onStatus?: (s: { state: string, age?: number, rendition?: string, ladder?: any[] }) => void, waitMs?: number, | |
| 50 | * @param {{ url: string, canvas: { width: number, height: number, getContext(k: string): any }, | |
| 51 | * onStatus?: (s: { state: string, why?: string, age?: number, fps?: number, latency?: number | null, rendition?: string, ladder?: any[] }) => void, | |
| 52 | * waitMs?: number, quietMs?: number, | |
| 36 | 53 | * deps?: { WebSocket?: any, VideoDecoder?: any, EncodedVideoChunk?: any, now?: () => number } }} o |
| 54 | * waitMs: for the first picture (default quietMs); quietMs: between pictures (default QUIET_MS) | |
| 37 | 55 | */ |
| 38 | 56 | constructor(o) { |
| 39 | 57 | this.o = o; |
| @@ -43,25 +61,46 @@ export class VideoPlayer { | ||
| 43 | 61 | this.Chunk = d.EncodedVideoChunk ?? (typeof EncodedVideoChunk === "undefined" ? undefined : EncodedVideoChunk); |
| 44 | 62 | this.now = d.now ?? (() => Date.now()); |
| 45 | 63 | this.ctx = o.canvas.getContext("2d"); |
| 64 | this.quietMs = o.quietMs ?? QUIET_MS; | |
| 46 | 65 | this.decoder = null; |
| 47 | 66 | this.configured = false; |
| 48 | 67 | this.needKey = true; |
| 49 | 68 | this.lowest = Infinity; |
| 50 | 69 | this.age = 0; |
| 51 | 70 | this.frames = 0; |
| 71 | this.errors = 0; | |
| 72 | this.lastTs = -Infinity; | |
| 73 | /** The relay's clock minus the page's, from the ping with the shortest round trip; null until one comes back. */ | |
| 74 | this.offset = null; | |
| 75 | this.bestRtt = Infinity; | |
| 76 | /** Since the last report: pictures drawn, their latencies, and when it began. */ | |
| 77 | this.drawn = 0; | |
| 78 | this.latencies = []; | |
| 79 | this.since = this.now(); | |
| 80 | this.fps = 0; | |
| 81 | this.latency = null; | |
| 52 | 82 | this.ladder = []; |
| 53 | 83 | this.closed = false; |
| 54 | 84 | this.ws = null; |
| 55 | 85 | this.timers = []; |
| 86 | this.watchdog = null; | |
| 56 | 87 | } |
| 57 | 88 | |
| 58 | 89 | start() { |
| 59 | 90 | const ws = (this.ws = new this.WS(this.o.url)); |
| 60 | 91 | ws.binaryType = "arraybuffer"; |
| 61 | 92 | ws.onmessage = (e) => (typeof e.data === "string" ? this.control(e.data) : this.frame(e.data)); |
| 62 | ws.onclose = () => this.status("none"); | |
| 63 | this.timers.push(setTimeout(() => this.frames || this.status("none"), this.o.waitMs ?? 3000)); | |
| 93 | ws.onclose = () => this.status("none", "closed"); | |
| 94 | ws.onopen = () => this.ping(); | |
| 95 | this.quiet(this.o.waitMs ?? this.quietMs); | |
| 64 | 96 | this.timers.push(setInterval(() => this.report(), 1000)); |
| 97 | this.timers.push(setInterval(() => this.ping(), PING_MS)); | |
| 98 | } | |
| 99 | ||
| 100 | /** (Re)arm the watchdog: no picture drawn in `ms` and the page goes back to MJPEG. */ | |
| 101 | quiet(ms = this.quietMs) { | |
| 102 | clearTimeout(this.watchdog); | |
| 103 | if (!this.closed) this.watchdog = setTimeout(() => this.status("none", "quiet"), ms); | |
| 65 | 104 | } |
| 66 | 105 | |
| 67 | 106 | /** Pin a rung by name, or "auto". @param {string} quality */ |
| @@ -69,28 +108,57 @@ export class VideoPlayer { | ||
| 69 | 108 | if (this.ws?.readyState === 1) this.ws.send(JSON.stringify({ quality })); |
| 70 | 109 | } |
| 71 | 110 | |
| 111 | ping() { | |
| 112 | if (this.ws?.readyState === 1) this.ws.send(JSON.stringify({ ping: this.now() })); | |
| 113 | } | |
| 114 | ||
| 72 | 115 | stop() { |
| 73 | 116 | this.closed = true; |
| 117 | clearTimeout(this.watchdog); | |
| 74 | 118 | this.timers.forEach((t) => (clearTimeout(t), clearInterval(t))); |
| 75 | 119 | try { this.ws?.close(); } catch {} |
| 76 | 120 | try { this.decoder?.close(); } catch {} |
| 77 | 121 | } |
| 78 | 122 | |
| 79 | status(state) { | |
| 80 | if (state === "none") this.stop(); // giving up releases the socket and the decoder | |
| 81 | this.o.onStatus?.({ state, age: this.age, rendition: this.rendition, ladder: this.ladder }); | |
| 123 | /** @param {string} state @param {string} [why] */ | |
| 124 | status(state, why) { | |
| 125 | if (state === "none") { | |
| 126 | if (this.closed) return; // said once | |
| 127 | if (this.ws?.readyState === 1) try { this.ws.send(JSON.stringify({ fallback: why })); } catch {} | |
| 128 | this.stop(); // giving up releases the socket and the decoder | |
| 129 | } | |
| 130 | this.o.onStatus?.({ state, why, age: this.age, fps: this.fps, latency: this.latency, rendition: this.rendition, ladder: this.ladder }); | |
| 82 | 131 | } |
| 83 | 132 | |
| 84 | 133 | control(text) { |
| 85 | 134 | let m; |
| 86 | 135 | try { m = JSON.parse(text); } catch { return; } |
| 136 | if (typeof m !== "object" || m === null) return; | |
| 137 | if (typeof m.pong === "number" && typeof m.at === "number") return this.pong(m.pong, m.at); | |
| 87 | 138 | if (Array.isArray(m.ladder)) this.ladder = m.ladder; |
| 88 | 139 | if (typeof m.rendition === "string") this.rendition = m.rendition; |
| 89 | 140 | if (this.frames) this.status("live"); |
| 90 | 141 | } |
| 91 | 142 | |
| 143 | /** A ping came back: the relay's time `at` is taken as the page's halfway through the round trip. */ | |
| 144 | pong(sent, at) { | |
| 145 | const rtt = this.now() - sent; | |
| 146 | if (!(rtt >= 0) || rtt >= this.bestRtt) return; | |
| 147 | this.bestRtt = rtt; | |
| 148 | this.offset = at - (sent + rtt / 2); | |
| 149 | } | |
| 150 | ||
| 92 | 151 | report() { |
| 93 | if (this.ws?.readyState === 1 && this.age > 0) this.ws.send(JSON.stringify({ age: Math.round(this.age) })); | |
| 152 | const now = this.now(); | |
| 153 | const seconds = (now - this.since) / 1000; | |
| 154 | if (seconds <= 0) return; | |
| 155 | this.fps = Number((this.drawn / seconds).toFixed(1)); | |
| 156 | const latency = middle(this.latencies); | |
| 157 | this.latency = latency === null ? null : Math.round(latency); | |
| 158 | this.drawn = 0; | |
| 159 | this.latencies = []; | |
| 160 | this.since = now; | |
| 161 | if (this.ws?.readyState === 1 && this.frames) this.ws.send(JSON.stringify({ age: Math.round(this.age), fps: this.fps, ...(this.latency === null ? {} : { latency: this.latency }) })); | |
| 94 | 162 | } |
| 95 | 163 | |
| 96 | 164 | frame(buffer) { |
| @@ -104,29 +172,47 @@ export class VideoPlayer { | ||
| 104 | 172 | if (this.needKey && !f.key) return; |
| 105 | 173 | if (f.key && !this.configured && !this.configure(f.data)) return; |
| 106 | 174 | this.needKey = false; |
| 175 | // Timestamps only go forward: two pictures the relay got in the same millisecond still get their own. | |
| 176 | const ts = (this.lastTs = Math.max(this.lastTs + 1, Math.round(f.sentAt * 1000))); | |
| 107 | 177 | try { |
| 108 | this.decoder.decode(new this.Chunk({ type: f.key ? "key" : "delta", timestamp: Math.round(f.sentAt * 1000), data: f.data })); | |
| 178 | this.decoder.decode(new this.Chunk({ type: f.key ? "key" : "delta", timestamp: ts, data: f.data })); | |
| 109 | 179 | } catch { |
| 110 | this.needKey = true; | |
| 111 | this.configured = false; | |
| 180 | this.failed(); | |
| 112 | 181 | } |
| 113 | 182 | } |
| 114 | 183 | |
| 184 | /** The decoder failed: it is made again at the next keyframe, until it has failed too often without a picture. */ | |
| 185 | failed() { | |
| 186 | this.needKey = true; | |
| 187 | this.configured = false; | |
| 188 | if (++this.errors >= DECODE_ERRORS) this.status("none", "decode"); | |
| 189 | } | |
| 190 | ||
| 115 | 191 | configure(unit) { |
| 116 | 192 | const codec = codecOf(unit); |
| 117 | 193 | if (!codec || !this.Decoder) return false; |
| 118 | 194 | try { this.decoder?.close(); } catch {} |
| 119 | this.decoder = new this.Decoder({ | |
| 120 | output: (frame) => { | |
| 121 | const c = this.o.canvas; | |
| 122 | if (c.width !== frame.displayWidth || c.height !== frame.displayHeight) (c.width = frame.displayWidth), (c.height = frame.displayHeight); | |
| 123 | this.ctx.drawImage(frame, 0, 0); | |
| 124 | frame.close(); | |
| 125 | if (!this.frames++) this.status("live"); | |
| 126 | }, | |
| 127 | error: () => ((this.needKey = true), (this.configured = false)), | |
| 128 | }); | |
| 129 | this.decoder.configure({ codec, optimizeForLatency: true }); | |
| 195 | try { | |
| 196 | this.decoder = new this.Decoder({ | |
| 197 | output: (frame) => { | |
| 198 | if (this.closed) return frame.close(); | |
| 199 | const c = this.o.canvas; | |
| 200 | if (c.width !== frame.displayWidth || c.height !== frame.displayHeight) (c.width = frame.displayWidth), (c.height = frame.displayHeight); | |
| 201 | this.ctx.drawImage(frame, 0, 0); | |
| 202 | if (this.offset !== null) this.latencies.push(Math.max(0, this.now() + this.offset - frame.timestamp / 1000)); | |
| 203 | frame.close(); | |
| 204 | this.errors = 0; | |
| 205 | this.drawn++; | |
| 206 | this.quiet(); | |
| 207 | if (!this.frames++) this.status("live"); | |
| 208 | }, | |
| 209 | error: () => this.failed(), | |
| 210 | }); | |
| 211 | this.decoder.configure({ codec, optimizeForLatency: true }); | |
| 212 | } catch { | |
| 213 | this.status("none", "unsupported"); | |
| 214 | return false; | |
| 215 | } | |
| 130 | 216 | return (this.configured = true); |
| 131 | 217 | } |
| 132 | 218 | } |
Software/so101/src/ui/video-player.test.ts +126 −1
| @@ -22,6 +22,7 @@ class FakeSocket { | ||
| 22 | 22 | sent: string[] = []; |
| 23 | 23 | onmessage: ((e: { data: unknown }) => void) | null = null; |
| 24 | 24 | onclose: (() => void) | null = null; |
| 25 | onopen: (() => void) | null = null; | |
| 25 | 26 | url: string; |
| 26 | 27 | constructor(url: string) { |
| 27 | 28 | this.url = url; |
| @@ -157,8 +158,9 @@ describe("video player", () => { | ||
| 157 | 158 | expect(t.decoded).toHaveLength(2); |
| 158 | 159 | t.ws.onmessage!({ data: frame(true, 0, 9_850, pics[5]!) }); // 150 ms away; the best seen is 0 by now |
| 159 | 160 | expect(t.decoded).toHaveLength(3); |
| 161 | t.tick(1000); | |
| 160 | 162 | t.player.report(); |
| 161 | expect(JSON.parse(t.ws.sent.at(-1)!)).toEqual({ age: 150 }); | |
| 163 | expect(JSON.parse(t.ws.sent.at(-1)!)).toEqual({ age: 150, fps: 3 }); // three pictures drawn in the second; no pong yet, so no latency | |
| 162 | 164 | t.player.quality("240p"); |
| 163 | 165 | expect(JSON.parse(t.ws.sent.at(-1)!)).toEqual({ quality: "240p" }); |
| 164 | 166 | t.player.stop(); |
| @@ -193,3 +195,126 @@ describe("video player", () => { | ||
| 193 | 195 | expect(t.decoders[0]!.closes).toBe(1); |
| 194 | 196 | }); |
| 195 | 197 | }); |
| 198 | ||
| 199 | describe("never a frozen picture", () => { | |
| 200 | const pics = clip(10, 5, 100); | |
| 201 | /** A decoder that draws each chunk at once with its timestamp, or fails while `failing` is set; `throws` makes configure throw. */ | |
| 202 | function rig(o: { quietMs?: number; throws?: boolean } = {}) { | |
| 203 | const states: { state: string; why?: string; fps?: number; latency?: number | null }[] = []; | |
| 204 | const decoders: any[] = []; | |
| 205 | const chunks: { type: string; timestamp: number }[] = []; | |
| 206 | let now = 50_000; | |
| 207 | const ctl = { failing: false }; | |
| 208 | class Decoder { | |
| 209 | decodeQueueSize = 0; | |
| 210 | closed = false; | |
| 211 | init: { output: (f: unknown) => void; error: (e: unknown) => void }; | |
| 212 | constructor(init: { output: (f: unknown) => void; error: (e: unknown) => void }) { | |
| 213 | this.init = init; | |
| 214 | decoders.push(this); | |
| 215 | } | |
| 216 | configure() { | |
| 217 | if (o.throws) throw new TypeError("unsupported codec"); | |
| 218 | } | |
| 219 | decode(chunk: { type: string; timestamp: number }) { | |
| 220 | chunks.push(chunk); | |
| 221 | if (ctl.failing) return this.init.error(new Error("decode")); | |
| 222 | this.init.output({ displayWidth: 854, displayHeight: 480, timestamp: chunk.timestamp, close() {} }); | |
| 223 | } | |
| 224 | close() { | |
| 225 | this.closed = true; | |
| 226 | } | |
| 227 | } | |
| 228 | const canvas = { width: 0, height: 0, getContext: () => ({ drawImage() {} }) }; | |
| 229 | const player = new VideoPlayer({ url: "ws://x/video", canvas, quietMs: o.quietMs ?? 10_000, onStatus: (s) => states.push(s), deps: { WebSocket: FakeSocket, VideoDecoder: Decoder, EncodedVideoChunk: class { constructor(init: object) { Object.assign(this, init); } }, now: () => now } }); | |
| 230 | player.start(); | |
| 231 | const ws = FakeSocket.last; | |
| 232 | const send = (i: number, sentAt = now) => ws.onmessage!({ data: frame(i % 5 === 0, 0, sentAt, pics[i]!) }); | |
| 233 | return { player, ws, states, decoders, chunks, ctl, send, tick: (ms: number) => (now += ms), get now() { return now; } }; | |
| 234 | } | |
| 235 | const sent = (ws: FakeSocket) => ws.sent.map((s) => JSON.parse(s)); | |
| 236 | ||
| 237 | test("pictures that stop after it went live send the page back to MJPEG, saying why", async () => { | |
| 238 | const r = rig({ quietMs: 30 }); | |
| 239 | r.send(0); | |
| 240 | r.send(1); | |
| 241 | expect(r.states.map((s) => s.state)).toEqual(["live"]); | |
| 242 | await Bun.sleep(15); | |
| 243 | r.send(2); // a picture re-arms the watchdog | |
| 244 | await Bun.sleep(20); | |
| 245 | expect(r.states.map((s) => s.state)).toEqual(["live"]); | |
| 246 | await Bun.sleep(40); | |
| 247 | expect(r.states.map((s) => [s.state, s.why])).toEqual([["live", undefined], ["none", "quiet"]]); | |
| 248 | expect(sent(r.ws)).toContainEqual({ fallback: "quiet" }); | |
| 249 | expect(r.ws.closes).toBe(1); | |
| 250 | expect(r.decoders[0].closed).toBe(true); | |
| 251 | }); | |
| 252 | ||
| 253 | test("a decoder that fails is made again at the next keyframe; failing three times with no picture between gives up", () => { | |
| 254 | const r = rig(); | |
| 255 | r.send(0); | |
| 256 | r.ctl.failing = true; | |
| 257 | r.send(1); // fails: waits for a key | |
| 258 | r.ctl.failing = false; | |
| 259 | r.send(2); | |
| 260 | r.send(3); | |
| 261 | expect(r.chunks).toHaveLength(2); | |
| 262 | r.send(5); // a keyframe: a new decoder, and it plays on | |
| 263 | r.send(6); | |
| 264 | expect(r.decoders).toHaveLength(2); | |
| 265 | expect(r.chunks.slice(2).map((c) => c.type)).toEqual(["key", "delta"]); | |
| 266 | expect(r.states.map((s) => s.state)).toEqual(["live"]); | |
| 267 | r.ctl.failing = true; | |
| 268 | r.send(7); | |
| 269 | r.send(0); | |
| 270 | r.send(5); | |
| 271 | expect(r.states.map((s) => [s.state, s.why]).at(-1)).toEqual(["none", "decode"]); | |
| 272 | expect(sent(r.ws)).toContainEqual({ fallback: "decode" }); | |
| 273 | }); | |
| 274 | ||
| 275 | test("a codec the browser cannot configure gives up at once", () => { | |
| 276 | const r = rig({ throws: true }); | |
| 277 | r.send(0); | |
| 278 | expect(r.states.map((s) => [s.state, s.why])).toEqual([["none", "unsupported"]]); | |
| 279 | }); | |
| 280 | ||
| 281 | test("the socket closing after pictures is a fallback too, said once", () => { | |
| 282 | const r = rig(); | |
| 283 | r.send(0); | |
| 284 | r.ws.onclose!(); | |
| 285 | r.ws.onclose!(); | |
| 286 | expect(r.states.map((s) => [s.state, s.why])).toEqual([["live", undefined], ["none", "closed"]]); | |
| 287 | }); | |
| 288 | ||
| 289 | test("pictures the relay got in the same millisecond still get rising timestamps", () => { | |
| 290 | const r = rig(); | |
| 291 | r.send(0, 1000); | |
| 292 | r.send(1, 1000); | |
| 293 | r.send(2, 999.9995); | |
| 294 | const ts = r.chunks.map((c) => c.timestamp); | |
| 295 | expect(ts[0]).toBe(1_000_000); | |
| 296 | expect(ts[1]! > ts[0]! && ts[2]! > ts[1]!).toBe(true); | |
| 297 | }); | |
| 298 | ||
| 299 | test("latency to the glass is on the relay's clock, from the ping with the shortest round trip; fps is what was drawn", () => { | |
| 300 | const r = rig(); | |
| 301 | r.ws.onopen!(); | |
| 302 | const first = sent(r.ws).at(-1); | |
| 303 | expect(first).toEqual({ ping: r.now }); | |
| 304 | // The relay's clock is 5 s ahead of the page's; this round trip took 40 ms. | |
| 305 | r.tick(40); | |
| 306 | r.ws.onmessage!({ data: JSON.stringify({ pong: first.ping, at: first.ping + 5000 + 20 }) }); | |
| 307 | // A slower round trip later is not taken. | |
| 308 | r.player.ping(); | |
| 309 | const second = sent(r.ws).at(-1); | |
| 310 | r.tick(400); | |
| 311 | r.ws.onmessage!({ data: JSON.stringify({ pong: second.ping, at: second.ping + 5000 + 900 }) }); | |
| 312 | expect(r.player.offset).toBe(5000); | |
| 313 | r.tick(560); // 1 s since the start | |
| 314 | // Got by the relay 90 ms ago on its clock, drawn now. | |
| 315 | for (let i = 0; i < 12; i++) r.send(i % 10, r.now + 5000 - 90); | |
| 316 | r.player.report(); | |
| 317 | expect(sent(r.ws).at(-1)).toEqual({ age: 0, fps: 12, latency: 90 }); | |
| 318 | expect(r.player.latency).toBe(90); | |
| 319 | }); | |
| 320 | }); |
Software/so101/src/video-fanout.test.ts +68
| @@ -1,5 +1,6 @@ | ||
| 1 | 1 | import { describe, expect, test } from "bun:test"; |
| 2 | 2 | import { clip } from "./h264-clip.ts"; |
| 3 | import { nalsOf } from "./h264.ts"; | |
| 3 | 4 | import { HEADER, LADDER, VideoFanout, type VideoSink } from "./video-fanout.ts"; |
| 4 | 5 | |
| 5 | 6 | function sink(): VideoSink & { sent: Buffer[]; texts: any[]; queued: number } { |
| @@ -113,3 +114,70 @@ describe("renditions and viewers", () => { | ||
| 113 | 114 | expect(f.wanted().size).toBe(0); |
| 114 | 115 | }); |
| 115 | 116 | }); |
| 117 | ||
| 118 | describe("late joiners", () => { | |
| 119 | // The worst case for a late joiner: the encoder sends SPS and PPS once, at the head of its stream. | |
| 120 | const head = clip(4, 15, 15, 120, { sets: "head", aud: true }); | |
| 121 | const one = [{ name: "480p", width: 854, height: 480, fps: 15, kbps: 600 }]; | |
| 122 | const types = (m: Buffer) => nalsOf(m.subarray(HEADER)).map((n) => n.type); | |
| 123 | ||
| 124 | test("every keyframe a viewer is sent carries SPS and PPS, even when the encoder sent them only once", () => { | |
| 125 | const f = new VideoFanout({ ladder: one }); | |
| 126 | const early = sink(); | |
| 127 | f.join(early); | |
| 128 | for (let i = 0; i < 20; i++) f.push("480p", { key: i % 15 === 0, data: head[i]! }); | |
| 129 | const late = sink(); | |
| 130 | f.join(late); // the cached keyframe (picture 15) is fresh: it comes at once, with its sets | |
| 131 | for (let i = 20; i < 40; i++) f.push("480p", { key: i % 15 === 0, data: head[i]! }); | |
| 132 | for (const s of [early, late]) { | |
| 133 | const keys = s.sent.filter((m) => (m[0]! & 1) === 1); | |
| 134 | expect(keys.length).toBeGreaterThan(1); | |
| 135 | for (const k of keys) expect(types(k)).toEqual([9, 7, 8, 5]); | |
| 136 | } | |
| 137 | // late: the cached key, nothing until the next live key at 30, then every picture | |
| 138 | expect(frames(late).map((x) => x.key)).toEqual([true, true, ...Array(9).fill(false)]); | |
| 139 | }); | |
| 140 | ||
| 141 | test("a stale keyframe is not sent, and a source that went away is forgotten", () => { | |
| 142 | let t = 0; | |
| 143 | const f = new VideoFanout({ ladder: one, now: () => t }); | |
| 144 | f.push("480p", { key: true, data: head[0]! }); | |
| 145 | t += 2500; | |
| 146 | const s = sink(); | |
| 147 | f.join(s); | |
| 148 | expect(s.sent).toHaveLength(0); | |
| 149 | f.push("480p", { key: true, data: head[0]! }); | |
| 150 | f.forget("480p"); | |
| 151 | const after = sink(); | |
| 152 | f.join(after); | |
| 153 | expect(after.sent).toHaveLength(0); | |
| 154 | f.push("480p", { key: true, data: head[15]! }); // no sets known since the source went: it goes as it is | |
| 155 | expect(types(after.sent[0]!)).toEqual([9, 5]); | |
| 156 | }); | |
| 157 | }); | |
| 158 | ||
| 159 | describe("what the pages report", () => { | |
| 160 | test("pings are answered on the relay's clock; reports are taken once a second; a fallback once per viewer", () => { | |
| 161 | let t = 1000; | |
| 162 | const f = new VideoFanout({ now: () => t }); | |
| 163 | const s = sink(); | |
| 164 | const v = f.join(s); | |
| 165 | v.message('{"ping":42.5}'); | |
| 166 | v.message('{"ping":43}'); // too soon | |
| 167 | expect(s.texts.filter((x) => "pong" in x)).toEqual([{ pong: 42.5, at: 1000 }]); | |
| 168 | v.message('{"age":10,"fps":14,"latency":120}'); | |
| 169 | v.message('{"age":10,"fps":1,"latency":9000}'); // too soon: ignored | |
| 170 | t += 1000; | |
| 171 | v.message('{"age":10,"fps":12.04,"latency":80.6}'); | |
| 172 | t += 1000; | |
| 173 | v.message('{"fps":-1}'); | |
| 174 | v.message('{"fps":500}'); | |
| 175 | v.message('{"fps":10,"latency":"x"}'); | |
| 176 | v.message('{"fallback":"quiet"}'); | |
| 177 | v.message('{"fallback":"quiet"}'); | |
| 178 | v.message('{"fallback":"<script>"}'); | |
| 179 | const card = f.stats.summary(t, f.watching); | |
| 180 | expect(card).toMatchObject({ viewers: 1, reports: 3, fpsP50: 12, fpsMin: 10, latencyP50Ms: 81, latencyP95Ms: 120 }); | |
| 181 | expect(card.fallbacks).toEqual({ quiet: 1, decode: 0, unsupported: 0, closed: 0 }); | |
| 182 | }); | |
| 183 | }); |
Software/so101/src/video-fanout.ts +53 −6
| @@ -6,11 +6,15 @@ | ||
| 6 | 6 | * the ladder, one that went a while clean steps back up, each switch at a keyframe of the new rendition. A viewer can |
| 7 | 7 | * pin one rendition. The driver is patient (it steps down only after more drops) so the best picture stays with it. |
| 8 | 8 | * |
| 9 | * Every keyframe goes out with an SPS and a PPS (the last ones seen when the encoder left them off: src/h264.ts), and | |
| 10 | * a viewer that joins is sent the rung's last keyframe at once when it is fresh, so it has a picture before the next one. | |
| 11 | * | |
| 9 | 12 | * A binary message is HEADER bytes then the access unit: byte 0 bit 0 = keyframe, byte 1 = the rendition's index in |
| 10 | 13 | * the ladder, bytes 2..9 = the time the relay got the picture (ms, float64 big-endian). Text messages are JSON. |
| 11 | 14 | */ |
| 12 | 15 | |
| 13 | import type { AccessUnit } from "./h264.ts"; | |
| 16 | import { ParameterSets, type AccessUnit } from "./h264.ts"; | |
| 17 | import { isFallback, parseReport, VideoStats } from "./video-stats.ts"; | |
| 14 | 18 | |
| 15 | 19 | export interface Rendition { |
| 16 | 20 | name: string; |
| @@ -46,6 +50,8 @@ export interface FanoutOptions { | ||
| 46 | 50 | cleanMs?: number; |
| 47 | 51 | /** A frame age the viewer reports above this also counts as a drop. */ |
| 48 | 52 | maxAgeMs?: number; |
| 53 | /** A rung's last keyframe younger than this is sent to a viewer as it joins. */ | |
| 54 | keyFreshMs?: number; | |
| 49 | 55 | now?: () => number; |
| 50 | 56 | } |
| 51 | 57 | |
| @@ -64,6 +70,17 @@ interface Viewer { | ||
| 64 | 70 | bytes: number; |
| 65 | 71 | frames: number; |
| 66 | 72 | skipped: number; |
| 73 | /** When its last report and ping were taken, and whether it has said it gave up. */ | |
| 74 | reportedAt: number; | |
| 75 | pingedAt: number; | |
| 76 | fellBack: boolean; | |
| 77 | } | |
| 78 | ||
| 79 | /** What the fanout keeps per rung: its parameter sets, and its last keyframe as a ready message. */ | |
| 80 | interface Rung { | |
| 81 | sets: ParameterSets; | |
| 82 | key: Buffer | null; | |
| 83 | keyAt: number; | |
| 67 | 84 | } |
| 68 | 85 | |
| 69 | 86 | export interface ViewerStats { |
| @@ -76,11 +93,14 @@ export interface ViewerStats { | ||
| 76 | 93 | export class VideoFanout { |
| 77 | 94 | readonly ladder: readonly Rendition[]; |
| 78 | 95 | private readonly viewers = new Set<Viewer>(); |
| 96 | private readonly rungs = new Map<number, Rung>(); | |
| 97 | /** What the viewers' pages report, and the bytes the feed pulls (src/video-stats.ts). */ | |
| 98 | readonly stats = new VideoStats(); | |
| 79 | 99 | private readonly o: Required<Omit<FanoutOptions, "ladder">>; |
| 80 | 100 | |
| 81 | 101 | constructor(options: FanoutOptions = {}) { |
| 82 | 102 | this.ladder = options.ladder ?? LADDER; |
| 83 | this.o = { maxBacklog: 64 << 10, dropsToStepDown: 3, cleanMs: 6000, maxAgeMs: 500, now: Date.now, ...options }; | |
| 103 | this.o = { maxBacklog: 64 << 10, dropsToStepDown: 3, cleanMs: 6000, maxAgeMs: 500, keyFreshMs: 2000, now: Date.now, ...options }; | |
| 84 | 104 | } |
| 85 | 105 | |
| 86 | 106 | get watching(): number { |
| @@ -98,9 +118,16 @@ export class VideoFanout { | ||
| 98 | 118 | join(sink: VideoSink, driver = false): { leave(): void; message(text: string): void; stats(): ViewerStats } { |
| 99 | 119 | const start = driver ? 0 : Math.min(1, this.ladder.length - 1); |
| 100 | 120 | const now = this.o.now(); |
| 101 | const v: Viewer = { sink, driver, current: start, target: start, pinned: null, needKey: true, drops: 0, lastDrop: now, lastStep: now, bytes: 0, frames: 0, skipped: 0 }; | |
| 121 | const v: Viewer = { sink, driver, current: start, target: start, pinned: null, needKey: true, drops: 0, lastDrop: now, lastStep: now, bytes: 0, frames: 0, skipped: 0, reportedAt: -Infinity, pingedAt: -Infinity, fellBack: false }; | |
| 102 | 122 | this.viewers.add(v); |
| 103 | 123 | sink.text(JSON.stringify({ ladder: this.ladder, rendition: this.ladder[start]!.name })); |
| 124 | // A picture now, from the last keyframe; it still waits for the next one to go on, as the pictures between depend on it. | |
| 125 | const rung = this.rungs.get(start); | |
| 126 | if (rung?.key && now - rung.keyAt < this.o.keyFreshMs) { | |
| 127 | sink.send(rung.key); | |
| 128 | v.bytes += rung.key.length; | |
| 129 | v.frames++; | |
| 130 | } | |
| 104 | 131 | return { |
| 105 | 132 | leave: () => void this.viewers.delete(v), |
| 106 | 133 | message: (text) => this.message(v, text), |
| @@ -108,10 +135,15 @@ export class VideoFanout { | ||
| 108 | 135 | }; |
| 109 | 136 | } |
| 110 | 137 | |
| 111 | /** A viewer's JSON: {"quality":"auto"|<name>} to pin one, {"age":ms} for the frame age it measures. Anything else is ignored. */ | |
| 138 | /** | |
| 139 | * A viewer's JSON: {"quality":"auto"|<name>} to pin one; once a second {"age":ms, "fps":n, "latency":ms} for the frame | |
| 140 | * age, decoded frames a second and relay-to-glass latency it measures; {"ping":t} answered {"pong":t, "at":relay ms} | |
| 141 | * for its clock offset; {"fallback":why} as it gives up for MJPEG. More than one report a second, or a ping every | |
| 142 | * 200 ms, is ignored, so one page cannot fill the scorecard. Anything else is ignored. | |
| 143 | */ | |
| 112 | 144 | private message(v: Viewer, text: string) { |
| 113 | 145 | if (text.length > 200) return; |
| 114 | let m: { quality?: unknown; age?: unknown }; | |
| 146 | let m: { quality?: unknown; age?: unknown; ping?: unknown; fallback?: unknown }; | |
| 115 | 147 | try { |
| 116 | 148 | m = JSON.parse(text); |
| 117 | 149 | } catch { |
| @@ -124,18 +156,32 @@ export class VideoFanout { | ||
| 124 | 156 | else if (at >= 0) (v.pinned = at), (v.target = at); |
| 125 | 157 | } |
| 126 | 158 | if (typeof m.age === "number" && Number.isFinite(m.age) && m.age > this.o.maxAgeMs) this.drop(v); |
| 159 | const now = this.o.now(); | |
| 160 | const report = parseReport(m as Record<string, unknown>); | |
| 161 | if (report && now - v.reportedAt >= 900) ((v.reportedAt = now), this.stats.report(now, report)); | |
| 162 | if (typeof m.ping === "number" && Number.isFinite(m.ping) && now - v.pingedAt >= 200) ((v.pingedAt = now), v.sink.text(JSON.stringify({ pong: m.ping, at: now }))); | |
| 163 | if (isFallback(m.fallback) && !v.fellBack) ((v.fellBack = true), this.stats.fallback(now, m.fallback)); | |
| 164 | } | |
| 165 | ||
| 166 | /** A rung's source went away: what it said about its stream no longer holds for the next one. */ | |
| 167 | forget(rendition: string): void { | |
| 168 | this.rungs.delete(this.ladder.findIndex((r) => r.name === rendition)); | |
| 127 | 169 | } |
| 128 | 170 | |
| 129 | 171 | /** The next picture of a rendition arrives from the Pi. */ |
| 130 | push(rendition: string, unit: AccessUnit): void { | |
| 172 | push(rendition: string, picture: AccessUnit): void { | |
| 131 | 173 | const at = this.ladder.findIndex((r) => r.name === rendition); |
| 132 | 174 | if (at < 0) return; |
| 133 | 175 | const now = this.o.now(); |
| 176 | let rung = this.rungs.get(at); | |
| 177 | if (!rung) this.rungs.set(at, (rung = { sets: new ParameterSets(), key: null, keyAt: 0 })); | |
| 178 | const unit = rung.sets.complete(picture); | |
| 134 | 179 | const message = Buffer.allocUnsafe(HEADER + unit.data.length); |
| 135 | 180 | message[0] = unit.key ? 1 : 0; |
| 136 | 181 | message[1] = at; |
| 137 | 182 | message.writeDoubleBE(now, 2); |
| 138 | 183 | unit.data.copy(message, HEADER); |
| 184 | if (unit.key && rung.sets.known) ((rung.key = message), (rung.keyAt = now)); | |
| 139 | 185 | for (const v of this.viewers) { |
| 140 | 186 | if (v.target === at && v.current !== at && unit.key) { |
| 141 | 187 | v.current = at; |
| @@ -193,5 +239,6 @@ export class VideoFanout { | ||
| 193 | 239 | |
| 194 | 240 | close(): void { |
| 195 | 241 | this.viewers.clear(); |
| 242 | this.rungs.clear(); | |
| 196 | 243 | } |
| 197 | 244 | } |
Software/so101/src/video-feed.ts +8
| @@ -10,6 +10,7 @@ | ||
| 10 | 10 | import { proxyFor } from "./egress.ts"; |
| 11 | 11 | import { AccessUnitSplitter } from "./h264.ts"; |
| 12 | 12 | import { LADDER, VideoFanout, type FanoutOptions, type VideoSink } from "./video-fanout.ts"; |
| 13 | import type { VideoSummary } from "./video-stats.ts"; | |
| 13 | 14 | |
| 14 | 15 | export interface VideoSources { |
| 15 | 16 | /** Rendition name → the URL of its Annex-B stream. */ |
| @@ -68,6 +69,11 @@ export class VideoFeed { | ||
| 68 | 69 | this.tick(); |
| 69 | 70 | } |
| 70 | 71 | |
| 72 | /** The scorecard's numbers for this camera (src/video-stats.ts). */ | |
| 73 | summary(now = Date.now()): VideoSummary { | |
| 74 | return this.fanout.stats.summary(now, this.fanout.watching); | |
| 75 | } | |
| 76 | ||
| 71 | 77 | close(): void { |
| 72 | 78 | this.closed = true; |
| 73 | 79 | this.fanout.close(); |
| @@ -100,11 +106,13 @@ export class VideoFeed { | ||
| 100 | 106 | if (!response.ok || !response.body) throw new Error(`answered ${response.status}`); |
| 101 | 107 | const splitter = new AccessUnitSplitter(); |
| 102 | 108 | for await (const chunk of response.body as unknown as AsyncIterable<Uint8Array>) { |
| 109 | this.fanout.stats.received(Date.now(), chunk.length); | |
| 103 | 110 | for (const unit of splitter.push(chunk)) this.fanout.push(name, unit); |
| 104 | 111 | } |
| 105 | 112 | } catch { |
| 106 | 113 | // down or cut off: the next tick tries again while the rung is wanted |
| 107 | 114 | } finally { |
| 115 | this.fanout.forget(name); // the next stream may come from an encoder that started over | |
| 108 | 116 | if (this.pulls.get(name) === controller) this.pulls.delete(name); |
| 109 | 117 | } |
| 110 | 118 | })(); |
Software/so101/src/video-panel.test.ts added +24
| @@ -0,0 +1,24 @@ | ||
| 1 | import { expect, test } from "bun:test"; | |
| 2 | import { videoSection, type VideoCard } from "./video-panel.ts"; | |
| 3 | ||
| 4 | const esc = (s: string) => s.replace(/[&<>"']/g, (c) => `&#${c.charCodeAt(0)};`); | |
| 5 | const card = (over: Partial<VideoCard> = {}): VideoCard => ({ | |
| 6 | camera: 1, viewers: 2, fpsP50: 14, fpsMin: 11, bytesPerS: 28_672, latencyP50Ms: 85, latencyP95Ms: 140, reports: 120, | |
| 7 | fallbacks: { quiet: 1, decode: 0, unsupported: 0, closed: 2 }, ...over, | |
| 8 | }); | |
| 9 | ||
| 10 | test("each H.264 camera shows decoded fps, KB/s from the Pi, relay-to-glass latency and its fallbacks with why", () => { | |
| 11 | const html = videoSection([card(), card({ camera: 2, fpsP50: null, fpsMin: null, latencyP50Ms: null, latencyP95Ms: null, reports: 1, fallbacks: { quiet: 0, decode: 0, unsupported: 0, closed: 0 } })], esc); | |
| 12 | expect(html).toContain("<h3>Camera 1</h3>"); | |
| 13 | expect(html).toContain("<b>14 fps (11 fps)</b>"); | |
| 14 | expect(html).toContain("<b>28.0 KB/s</b>"); | |
| 15 | expect(html).toContain("<b>85 ms / 140 ms</b>"); | |
| 16 | expect(html).toContain("<b>quiet 1, closed 2</b>"); | |
| 17 | expect(html).toContain("1 page report in"); | |
| 18 | expect(html).toContain("<b>— / —</b>"); | |
| 19 | expect(html).toContain("<b>none</b>"); | |
| 20 | }); | |
| 21 | ||
| 22 | test("no H.264 camera says how to have one", () => { | |
| 23 | expect(videoSection([], esc)).toContain("ARM_H264_<n>"); | |
| 24 | }); |
Software/so101/src/video-panel.ts added +33
| @@ -0,0 +1,33 @@ | ||
| 1 | /** | |
| 2 | * The Video part of the precision scorecard on /admin: per camera with H.264, what its viewers' pages decoded, what the | |
| 3 | * relay pulled from the Pi, the latency from the relay to the glass, and how often a page fell back to MJPEG. The | |
| 4 | * numbers are the relay's own (src/video-stats.ts), from the last ten minutes; plain text, no script. | |
| 5 | */ | |
| 6 | ||
| 7 | import type { VideoSummary } from "./video-stats.ts"; | |
| 8 | ||
| 9 | export interface VideoCard extends VideoSummary { | |
| 10 | camera: number; | |
| 11 | } | |
| 12 | ||
| 13 | const or = (v: number | null, unit: string) => (v === null ? "—" : `${v} ${unit}`); | |
| 14 | ||
| 15 | /** The section; `esc` is the page's own escaping. An empty list (no camera has H.264) says so. */ | |
| 16 | export function videoSection(cards: readonly VideoCard[], esc: (s: string) => string): string { | |
| 17 | const head = `<section aria-labelledby="video"><h2 id="video">Video</h2>`; | |
| 18 | if (!cards.length) return `${head}<p class="empty">No camera has H.264: set <code>ARM_H264_<n></code> and the cameras play it, with MJPEG as the fallback.</p></section>`; | |
| 19 | const rows = cards | |
| 20 | .map((c) => { | |
| 21 | const falls = Object.entries(c.fallbacks).filter(([, n]) => n > 0).map(([why, n]) => `${why} ${n}`).join(", ") || "none"; | |
| 22 | const items: Array<[string, string]> = [ | |
| 23 | ["Decoded, p50 (lowest)", c.fpsP50 === null ? "—" : `${c.fpsP50} fps (${c.fpsMin} fps)`], | |
| 24 | ["From the Pi", `${(c.bytesPerS / 1024).toFixed(1)} KB/s`], | |
| 25 | ["Relay → glass, p50 / p95", `${or(c.latencyP50Ms, "ms")} / ${or(c.latencyP95Ms, "ms")}`], | |
| 26 | ["Fell back to MJPEG", falls], | |
| 27 | ]; | |
| 28 | const list = items.map(([label, value]) => `<li><span>${esc(label)}</span><span><b>${esc(value)}</b></span></li>`).join(""); | |
| 29 | return `<h3>Camera ${c.camera}</h3><p>${c.viewers} watching · ${c.reports} page report${c.reports === 1 ? "" : "s"} in the last 10 minutes</p><ol class="bars precision" aria-label="Camera ${c.camera}, video">${list}</ol>`; | |
| 30 | }) | |
| 31 | .join(""); | |
| 32 | return `${head}${rows}<p>Relay → glass is from the relay getting a picture to the page drawing it; the hop before, from the camera through the Pi's encoder, carries no capture time and is not in it.</p></section>`; | |
| 33 | } |
Software/so101/src/video-stats.ts added +102
| @@ -0,0 +1,102 @@ | ||
| 1 | /** | |
| 2 | * One camera's H.264 as its viewers see it, for the precision scorecard: the frames a second their pages decoded, the | |
| 3 | * bytes a second the relay pulled from the Pi, the latency from the relay to the glass, and how often a page gave up | |
| 4 | * on H.264 for MJPEG, with why. Plain data in (each page's once-a-second report, the bytes as they come), plain data | |
| 5 | * out (a summary of the last WINDOW_MS). | |
| 6 | * | |
| 7 | * The latency is from the moment the relay got a picture to the moment the page drew it, on the relay's clock (the | |
| 8 | * page keeps an estimate of the offset, from pings over its socket). The hop before, from the camera's glass through | |
| 9 | * the Pi's encoder to the relay, is not in it: raw Annex-B carries no capture time. | |
| 10 | */ | |
| 11 | ||
| 12 | /** The span the summary covers, ms. */ | |
| 13 | export const WINDOW_MS = 10 * 60_000; | |
| 14 | /** The span bytes a second are averaged over, ms. */ | |
| 15 | export const RATE_MS = 10_000; | |
| 16 | /** Most reports kept; the oldest go first. */ | |
| 17 | const KEEP = 20_000; | |
| 18 | ||
| 19 | /** Why a page left H.264 for MJPEG: a fixed list, so every count means the same thing release to release. */ | |
| 20 | export const FALLBACKS = ["quiet", "decode", "unsupported", "closed"] as const; | |
| 21 | export type Fallback = (typeof FALLBACKS)[number]; | |
| 22 | export const isFallback = (v: unknown): v is Fallback => typeof v === "string" && (FALLBACKS as readonly string[]).includes(v); | |
| 23 | ||
| 24 | /** The most a page may report: frames a second, and milliseconds of latency. */ | |
| 25 | export const MAX_FPS = 120; | |
| 26 | export const MAX_LATENCY_MS = 60_000; | |
| 27 | ||
| 28 | export interface VideoReport { | |
| 29 | fps: number; | |
| 30 | /** Relay to glass, ms; null while the page has no clock offset yet. */ | |
| 31 | latencyMs: number | null; | |
| 32 | } | |
| 33 | ||
| 34 | /** A page's report, checked: fps from 0 to MAX_FPS, latency from 0 to MAX_LATENCY_MS (or left out); null when it is neither. */ | |
| 35 | export function parseReport(m: Record<string, unknown>): VideoReport | null { | |
| 36 | const { fps, latency } = m; | |
| 37 | if (typeof fps !== "number" || !Number.isFinite(fps) || fps < 0 || fps > MAX_FPS) return null; | |
| 38 | const latencyMs = typeof latency === "number" && Number.isFinite(latency) && latency >= 0 && latency <= MAX_LATENCY_MS ? Math.round(latency) : null; | |
| 39 | return { fps: Number(fps.toFixed(1)), latencyMs }; | |
| 40 | } | |
| 41 | ||
| 42 | export interface VideoSummary { | |
| 43 | /** Pages watching now. */ | |
| 44 | viewers: number; | |
| 45 | /** Decoded frames a second, over the pages' reports in the window. */ | |
| 46 | fpsP50: number | null; | |
| 47 | fpsMin: number | null; | |
| 48 | /** From the Pi to the relay, over the last RATE_MS. */ | |
| 49 | bytesPerS: number; | |
| 50 | latencyP50Ms: number | null; | |
| 51 | latencyP95Ms: number | null; | |
| 52 | /** Reports in the window. */ | |
| 53 | reports: number; | |
| 54 | fallbacks: Record<Fallback, number>; | |
| 55 | } | |
| 56 | ||
| 57 | /** Nearest-rank percentile of unsorted values; null for none. */ | |
| 58 | function percentile(values: readonly number[], p: number): number | null { | |
| 59 | if (!values.length) return null; | |
| 60 | const sorted = [...values].sort((a, b) => a - b); | |
| 61 | return sorted[Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1))]!; | |
| 62 | } | |
| 63 | ||
| 64 | export class VideoStats { | |
| 65 | private readonly reports: Array<{ at: number } & VideoReport> = []; | |
| 66 | private readonly falls: Array<{ at: number; why: Fallback }> = []; | |
| 67 | private readonly bytes: Array<readonly [number, number]> = []; | |
| 68 | ||
| 69 | report(at: number, r: VideoReport): void { | |
| 70 | this.reports.push({ at, ...r }); | |
| 71 | if (this.reports.length > KEEP) this.reports.shift(); | |
| 72 | } | |
| 73 | ||
| 74 | fallback(at: number, why: Fallback): void { | |
| 75 | this.falls.push({ at, why }); | |
| 76 | if (this.falls.length > KEEP) this.falls.shift(); | |
| 77 | } | |
| 78 | ||
| 79 | received(at: number, n: number): void { | |
| 80 | this.bytes.push([at, n]); | |
| 81 | while (this.bytes.length && this.bytes[0]![0] < at - RATE_MS) this.bytes.shift(); | |
| 82 | } | |
| 83 | ||
| 84 | summary(now: number, viewers: number): VideoSummary { | |
| 85 | const recent = this.reports.filter((r) => r.at > now - WINDOW_MS); | |
| 86 | const fps = recent.map((r) => r.fps); | |
| 87 | const latency = recent.flatMap((r) => (r.latencyMs === null ? [] : [r.latencyMs])); | |
| 88 | const fallbacks = Object.fromEntries(FALLBACKS.map((f) => [f, 0])) as Record<Fallback, number>; | |
| 89 | for (const f of this.falls) if (f.at > now - WINDOW_MS) fallbacks[f.why]++; | |
| 90 | const inRate = this.bytes.filter(([at]) => at > now - RATE_MS).reduce((sum, [, n]) => sum + n, 0); | |
| 91 | return { | |
| 92 | viewers, | |
| 93 | fpsP50: percentile(fps, 50), | |
| 94 | fpsMin: fps.length ? Math.min(...fps) : null, | |
| 95 | bytesPerS: Math.round(inRate / (RATE_MS / 1000)), | |
| 96 | latencyP50Ms: percentile(latency, 50), | |
| 97 | latencyP95Ms: percentile(latency, 95), | |
| 98 | reports: recent.length, | |
| 99 | fallbacks, | |
| 100 | }; | |
| 101 | } | |
| 102 | } |
Software/so101/src/viewer.html +3 −2
| @@ -642,7 +642,7 @@ | ||
| 642 | 642 | // Behind the relay the camera is frames over WS /camera/<id>/ws (src/ui/video.js, imported when first needed); on the machine itself, the plain <img> stream. |
| 643 | 643 | let video = null; |
| 644 | 644 | const stopFeed = (c) => { c.epoch = (c.epoch ?? 0) + 1; c.feed?.close?.(); c.feed = null; }; |
| 645 | // A camera the relay flags h264 plays through ui/video-pane.js and video-player.js, only where WebCodecs exists; no picture in 3 s and it goes back to the frames. | |
| 645 | // A camera the relay flags h264 plays through ui/video-pane.js and video-player.js, only where WebCodecs exists; no picture drawn for 2 s (at the start or later), a decoder that keeps failing or a closed socket and it goes back to the frames. | |
| 646 | 646 | async function playH264(c, epoch) { |
| 647 | 647 | let pane, player; |
| 648 | 648 | try { [pane, player] = await Promise.all([import("/ui/__VERSION__/video-pane.js"), import("/ui/__VERSION__/video-player.js")]); } catch { return false; } |
| @@ -789,7 +789,8 @@ | ||
| 789 | 789 | } |
| 790 | 790 | const why = quality === "saver" ? "data saver" : "slow link"; |
| 791 | 791 | const slow = plan.mode === "snapshot" ? ` · ${(1000 / plan.everyMs).toFixed(plan.everyMs > 1000 ? 1 : 0)} fps, ${why}` : plan.fps ? ` · ${plan.fps} fps, ${why}` : ""; |
| 792 | $("live").textContent = cams.filter(paneOn).map((c) => (c.live ? `live${slow}` : c.online === false ? "offline" : "waiting")).join(" · "); | |
| 792 | const h264 = (c) => (c.feed?.h264 && c.live ? ` · H.264 ${Math.round(c.feed.player?.fps ?? 0)} fps` : ""); | |
| 793 | $("live").textContent = cams.filter(paneOn).map((c) => (c.live ? `live${h264(c) || slow}` : c.online === false ? "offline" : "waiting")).join(" · "); | |
| 793 | 794 | }, 1000); |
| 794 | 795 | if (boot) { reconcile(boot.cameras); setTimeout(pollCameras, 3000); } |
| 795 | 796 | else if (relayed) pollCameras(); |
new above
offline