Public: anyone can read this; the crew can change it.
Migrated from github.com/Trevato/SO-ARM100
workbenchthis machine
Grow cameras: a saved camera profile per Pi camera, re-applied at every start, and a Tune panel with the box drawn once and colour correction under grow light #591
Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.
vesper/arm:vesper/584-grow-cameras-see-the-plants-a-saved-came → main 4 commits fork
13 files
changed +1390
−15
Software/README.md +84
| @@ -900,6 +900,90 @@ The vision kit in `Software/vision` works against this server and stream: | ||
| 900 | 900 | `SO101_ARM=http://<pi>:8101 SO101_CAMERA=http://<pi>:8080/stream`, then the |
| 901 | 901 | `just vision-*` recipes. See `Software/vision/QUEST.md`. |
| 902 | 902 | |
| 903 | ### Camera profile: seeing the plants under a grow light | |
| 904 | ||
| 905 | A webcam meters on the brightest thing it sees. In the grow box that is the | |
| 906 | light, so on auto exposure the box comes out as a white blob on blue. The fix | |
| 907 | is a fixed exposure and white balance. A webcam forgets both on every re-plug | |
| 908 | and reboot, so the Pi applies a saved **profile** each time the camera starts. | |
| 909 | ||
| 910 | **Until a profile is saved, each camera gets the owner's own choice** from | |
| 911 | 2026-10-26 (`OWNER_DEFAULTS` in `src/camera-profile.ts`). Camera 1, the side | |
| 912 | XWF-1080P on :8080, gets auto exposure, auto white balance and backlight | |
| 913 | compensation 1. Camera 2, the overhead eMeet C960 on :8081, gets brightness | |
| 914 | 115, contrast 120 and saturation 150. The Tune panel starts from these, and | |
| 915 | `so101 camera-profile <n>` applies them on the Pi when nothing has been saved, | |
| 916 | so they survive a re-plug from the first deploy on. | |
| 917 | ||
| 918 | **Tune it on /admin → Cameras → Tune.** The panel shows the camera once a | |
| 919 | second and has a slider for exposure, white balance, gain, focus, backlight | |
| 920 | compensation, brightness, contrast and saturation. Each control can be *leave* (the camera's own value), *auto* or | |
| 921 | *fixed*. A slider goes on the camera at once (`POST /admin/cameras/<id>/try`), | |
| 922 | and **Save** keeps it (`PUT /admin/cameras/<id>`). *Grow-box settings* loads | |
| 923 | what worked on 2026-10-26: exposure 180 (×100 µs), white balance 6500 K, | |
| 924 | backlight compensation 0. | |
| 925 | ||
| 926 | **Mark the box.** Drag on the picture to mark the box. The region is kept in | |
| 927 | fractions of the frame, so a region drawn on the 640×360 stream crops a | |
| 928 | 1920×1080 still the same way (`cropRect` in `src/ui/still-frame.js`). The | |
| 929 | check under the pictures only looks at that region. When the region is mostly | |
| 930 | blown out or black, it says: *Camera can't see the plants: move it inside or | |
| 931 | beside the box, below the light.* That means the camera is in the wrong place. | |
| 932 | The overhead camera, which looks down at the top of the light panel, is the | |
| 933 | case it was written for. | |
| 934 | ||
| 935 | **Colour.** A blue/violet grow light leaves little red and green, so colour is | |
| 936 | never true under it. *Measure colour* takes gray-world gains over the region, | |
| 937 | ignoring blown-out and black pixels, and caps each channel at ×4. The right | |
| 938 | picture shows the box with the gains applied, labelled "corrected". The gains | |
| 939 | are saved with the camera. `whiteCardGains` takes them from a shot of a white | |
| 940 | card under the light instead. | |
| 941 | ||
| 942 | **On the Pi**, the arm server keeps camera n's profile in | |
| 943 | `camera-profiles/<n>.json` beside its poses, and `so101 camera-profile <n>` | |
| 944 | puts it on the camera. The camera's device is read from `/etc/so101/camera<n>`: | |
| 945 | the `/dev/v4l/by-id` path alone, or as `DEVICE=<path>`. Each control is one | |
| 946 | `v4l2-ctl --set-ctrl` call made with an argument list, never a shell. A | |
| 947 | control the camera lacks (the XWF-1080P has no focus) is reported and the rest | |
| 948 | still go on. With no profile saved, it applies the owner's default for that camera, if | |
| 949 | there is one, and otherwise leaves the camera alone. It never stops a stream | |
| 950 | from starting. | |
| 951 | ||
| 952 | sh | |
| 953 | sudo apt-get install v4l-utils | |
| 954 | v4l2-ctl -d /dev/v4l/by-id/<camera>-video-index0 --list-ctrls # the names and ranges this camera has | |
| 955 | ls /dev/v4l/by-id/ | sudo tee /etc/so101/camera2 # then keep only camera 2's -video-index0 line | |
| 956 | sudo systemctl edit so101-camera@.service | |
| 957 | ||
| 958 | ||
| 959 | ini | |
| 960 | [Service] | |
| 961 | ExecStartPre=-/usr/bin/node <so101 tree>/bin/so101 camera-profile %i --file <the poses.json so101 serve uses> | |
| 962 | ||
| 963 | ||
| 964 | Use the arm server's tree (the one `so101.service` runs) and the same | |
| 965 | `--file` as its `so101 serve`, given as an absolute path, so both read the | |
| 966 | same `camera-profiles/`. The leading `-` keeps the stream starting even if the | |
| 967 | command fails. To re-apply on a re-plug, restart the unit when its camera | |
| 968 | appears: | |
| 969 | ||
| 970 | sh | |
| 971 | echo 'ACTION=="add", SUBSYSTEM=="video4linux", ATTR{index}=="0", RUN+="/usr/bin/systemctl restart --no-block [email protected] [email protected]"' \ | |
| 972 | | sudo tee /etc/udev/rules.d/90-so101-camera.rules | |
| 973 | sudo udevadm control --reload | |
| 974 | ||
| 975 | ||
| 976 | **Tuning is page-only on purpose.** An exposure is judged by looking at the | |
| 977 | picture, which neither the app's terminal (`tty`) nor the CLI can show. So | |
| 978 | `so101 camera-profile` only re-applies what the panel saved. On a Pi with no | |
| 979 | browser at hand, `v4l2-ctl --set-ctrl` tries a value by hand, and the panel | |
| 980 | keeps it. | |
| 981 | ||
| 982 | The relay never passes `/camera-profile/` through to the arm server. Only the | |
| 983 | owner's two routes above reach it. The simulation has no camera to tune: Save | |
| 984 | keeps the look in the relay's camera list and says the Pi did not take it. | |
| 985 | The arm server must be running on the Pi for a profile to reach the camera. | |
| 986 | ||
| 903 | 987 | ### H.264 renditions on 8082-8084 |
| 904 | 988 | |
| 905 | 989 | Beside the MJPEG stream, the Pi 4's hardware encoder (`/dev/video11`, the |
Software/so101/src/camera-control.ts added +90
| @@ -0,0 +1,90 @@ | ||
| 1 | /** | |
| 2 | * The Pi's side of a camera profile (src/camera-profile.ts): where camera n is, where its saved profile is kept, and | |
| 3 | * running `v4l2-ctl` to put a profile on it. The arm server calls it when the owner tries or saves a profile on /admin; | |
| 4 | * `so101 camera-profile <n>` calls it from the camera service's ExecStartPre, so the profile survives a re-plug or a reboot. | |
| 5 | * | |
| 6 | * Runs under Node as well as Bun (the CLI is Node): nothing here reaches bun:sqlite. v4l2-ctl is run with execFile, an | |
| 7 | * argument list built from checked numbers, never a shell. | |
| 8 | */ | |
| 9 | ||
| 10 | import { execFile } from "node:child_process"; | |
| 11 | import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; | |
| 12 | import { dirname, join } from "node:path"; | |
| 13 | import { controls, parseDevice, parseProfile, ProfileError, v4l2Calls, type CameraProfile } from "./camera-profile.ts"; | |
| 14 | ||
| 15 | /** The camera services are so101-camera@1 … @8. */ | |
| 16 | export const MAX_UNIT = 8; | |
| 17 | ||
| 18 | export function parseUnit(text: unknown): number { | |
| 19 | const n = typeof text === "number" ? text : typeof text === "string" && /^\d{1,2}$/.test(text) ? Number(text) : NaN; | |
| 20 | if (!Number.isInteger(n) || n < 1 || n > MAX_UNIT) throw new ProfileError(`a camera on the Pi is numbered 1 to ${MAX_UNIT}, like so101-camera@1`); | |
| 21 | return n; | |
| 22 | } | |
| 23 | ||
| 24 | /** | |
| 25 | * Camera n's device, from /etc/so101/camera<n>: the file holds the /dev/v4l/by-id path, alone or as DEVICE=<path> | |
| 26 | * (so the same file can be the service's EnvironmentFile). | |
| 27 | */ | |
| 28 | export function cameraDevice(unit: number, etc = "/etc/so101"): string { | |
| 29 | const file = join(etc, `camera${parseUnit(unit)}`); | |
| 30 | if (!existsSync(file)) throw new ProfileError(`${file} is missing: write camera ${unit}'s /dev/v4l/by-id path there`); | |
| 31 | const lines = readFileSync(file, "utf8").split("\n").map((l) => l.trim()).filter((l) => l && !l.startsWith("#")); | |
| 32 | const line = lines.find((l) => l.startsWith("DEVICE=")) ?? lines.find((l) => l.startsWith("/dev/")); | |
| 33 | if (!line) throw new ProfileError(`${file} names no device: write camera ${unit}'s /dev/v4l/by-id path there`); | |
| 34 | return parseDevice(line.replace(/^DEVICE=/, "").replace(/^["']|["']$/g, "")); | |
| 35 | } | |
| 36 | ||
| 37 | /** Where camera n's saved profile is kept, beside the arm server's poses. */ | |
| 38 | export function profileFile(dataDir: string, unit: number): string { | |
| 39 | return join(dataDir, "camera-profiles", `${parseUnit(unit)}.json`); | |
| 40 | } | |
| 41 | ||
| 42 | /** The saved profile, or null when none was saved (or the file is unreadable, which is said on stderr). */ | |
| 43 | export function loadProfile(file: string): CameraProfile | null { | |
| 44 | if (!existsSync(file)) return null; | |
| 45 | try { | |
| 46 | return parseProfile(JSON.parse(readFileSync(file, "utf8"))); | |
| 47 | } catch (err) { | |
| 48 | console.error(`camera profile: ${file} unreadable (${(err as Error).message})`); | |
| 49 | return null; | |
| 50 | } | |
| 51 | } | |
| 52 | ||
| 53 | export function saveProfile(file: string, profile: CameraProfile): void { | |
| 54 | mkdirSync(dirname(file), { recursive: true }); | |
| 55 | const tmp = `${file}.tmp`; | |
| 56 | writeFileSync(tmp, JSON.stringify(parseProfile(profile))); | |
| 57 | renameSync(tmp, file); | |
| 58 | } | |
| 59 | ||
| 60 | /** Runs one `v4l2-ctl` call; resolves with its error text, or "" when it worked. */ | |
| 61 | export type RunV4l2 = (args: string[]) => Promise<string>; | |
| 62 | ||
| 63 | export const runV4l2: RunV4l2 = (args) => | |
| 64 | new Promise((resolve) => { | |
| 65 | execFile("v4l2-ctl", args, { timeout: 3000 }, (err, _stdout, stderr) => { | |
| 66 | if (!err) return resolve(""); | |
| 67 | const code = (err as NodeJS.ErrnoException).code; | |
| 68 | resolve(code === "ENOENT" ? "v4l2-ctl is not installed (sudo apt-get install v4l-utils)" : (stderr || err.message).trim().split("\n")[0]!.slice(0, 200)); | |
| 69 | }); | |
| 70 | }); | |
| 71 | ||
| 72 | export interface Applied { | |
| 73 | /** The controls the camera took, as name=value. */ | |
| 74 | set: string[]; | |
| 75 | /** The ones it refused, and why: a control this camera lacks is the usual one. */ | |
| 76 | failed: { control: string; error: string }[]; | |
| 77 | } | |
| 78 | ||
| 79 | /** Put a profile on the camera at `device`, one control at a time; a refused control is reported, never thrown. */ | |
| 80 | export async function applyProfile(device: string, profile: CameraProfile, run: RunV4l2 = runV4l2): Promise<Applied> { | |
| 81 | const calls = v4l2Calls(device, profile); | |
| 82 | const names = controls(profile).map((c) => `${c.name}=${c.value}`); | |
| 83 | const out: Applied = { set: [], failed: [] }; | |
| 84 | for (const [i, args] of calls.entries()) { | |
| 85 | const error = await run(args); | |
| 86 | if (error) out.failed.push({ control: names[i]!, error }); | |
| 87 | else out.set.push(names[i]!); | |
| 88 | } | |
| 89 | return out; | |
| 90 | } |
Software/so101/src/camera-list.html +210 −4
| @@ -9,14 +9,27 @@ | ||
| 9 | 9 | body { font: 15px/1.5 system-ui, sans-serif; max-width: 42rem; margin: 2rem auto; padding: 0 1rem; } |
| 10 | 10 | li { display: flex; gap: .75rem; align-items: baseline; padding: .4rem 0; border-bottom: 1px solid var(); list-style: none; } |
| 11 | 11 | ul { padding: 0; } li span.src { color: var(); flex: 1; overflow-wrap: anywhere; } |
| 12 | form { display: flex; gap: .5rem; flex-wrap: wrap; } input, button { font: inherit; padding: .3rem .5rem; } li button { padding: .5rem .9rem; } label { align-self: center; } #msg { color: #c33; min-height: 1.5em; } | |
| 12 | form { display: flex; gap: .5rem; flex-wrap: wrap; } input, button, select { font: inherit; padding: .3rem .5rem; } li button { padding: .5rem .9rem; } label { align-self: center; } #msg { color: #c33; min-height: 1.5em; } | |
| 13 | #tune { border-top: 2px solid var(); margin-top: 1.5rem; padding-top: .5rem; } | |
| 14 | .views { display: grid; grid-template-columns: 1fr 1fr; gap: .5rem; } @media (max-width: 34rem) { .views { grid-template-columns: 1fr; } } | |
| 15 | figure { margin: 0; } figcaption { color: var(); font-size: 13px; } | |
| 16 | canvas { display: block; width: 100%; aspect-ratio: 16 / 9; object-fit: contain; background: #0002; touch-action: none; } | |
| 17 | #view { cursor: crosshair; } | |
| 18 | .ctl { display: grid; grid-template-columns: 7.5rem 6.5rem 1fr 4rem; gap: .5rem; align-items: center; margin: .3rem 0; } | |
| 19 | .ctl output { font-variant-numeric: tabular-nums; text-align: right; } .ctl input[type=range]:disabled { opacity: .35; } | |
| 20 | /* a phone: the label and its mode on one line, the slider and its value under them, the slider as wide as the screen */ | |
| 21 | @media (max-width: 34rem) { .ctl { grid-template-columns: 1fr auto; row-gap: .15rem; margin: .6rem 0; } .ctl input[type=range] { min-height: 44px; } } | |
| 22 | #box { border: 1px solid var(); border-radius: 4px; margin: .5rem 0; padding: .3rem .6rem .5rem; display: flex; gap: .5rem 1rem; flex-wrap: wrap; } | |
| 23 | #box label { display: flex; gap: .3rem; align-items: center; } #box input { width: 4.5rem; } | |
| 24 | #seen { min-height: 1.5em; } #seen.bad { color: #c33; font-weight: 600; } | |
| 25 | .row { display: flex; gap: .5rem; flex-wrap: wrap; margin: .5rem 0; } #note { min-height: 1.5em; } #note.bad { color: #c33; } | |
| 13 | 26 | </style> |
| 14 | 27 | </head> |
| 15 | 28 | <body> |
| 16 | 29 | <p><a href="/">← the arm</a> · <a href="/admin">Admin</a></p> |
| 17 | 30 | <h1>Cameras</h1> |
| 18 | 31 | <p>Each camera is streamed once from its source and shared by everyone watching. Viewers see a change within a few seconds.</p> |
| 19 | <noscript><p>This page needs JavaScript to list and change cameras.</p></noscript> | |
| 32 | <noscript><p>This page needs JavaScript to list and change cameras, and to tune them.</p></noscript> | |
| 20 | 33 | <ul id="list"></ul> |
| 21 | 34 | <form id="add"> |
| 22 | 35 | <label for="name">Name</label> |
| @@ -26,7 +39,34 @@ | ||
| 26 | 39 | <button>Add camera</button> |
| 27 | 40 | </form> |
| 28 | 41 | <p id="msg" role="status"></p> |
| 29 | <script> | |
| 42 | <section id="tune" hidden aria-labelledby="tuneTitle"> | |
| 43 | <h2 id="tuneTitle">Tune</h2> | |
| 44 | <p>Drag on the picture, or type the box's edges below in percent of the frame, to mark the box: stills are cropped to it, and the colour correction and the check below look only there. Sliders go on the camera at once; <b>Save</b> keeps them, and the Pi puts them back each time the camera starts.</p> | |
| 45 | <div class="views"> | |
| 46 | <figure><canvas id="view" width="640" height="360" role="img" aria-label="The camera's picture, one snapshot a second, with the box marked in yellow" aria-describedby="boxSays"></canvas><figcaption id="viewCap">as the camera sees it</figcaption></figure> | |
| 47 | <figure><canvas id="fixed" width="640" height="360" role="img" aria-label="Only the box, with the colour correction applied when there is one" aria-describedby="fixedCap"></canvas><figcaption id="fixedCap">the box</figcaption></figure> | |
| 48 | </div> | |
| 49 | <fieldset id="box"> | |
| 50 | <legend id="boxSays">The box: the whole frame</legend> | |
| 51 | <label>Left <input id="bx" type="number" min="0" max="98" step="1" inputmode="numeric">%</label> | |
| 52 | <label>Top <input id="by" type="number" min="0" max="98" step="1" inputmode="numeric">%</label> | |
| 53 | <label>Width <input id="bw" type="number" min="2" max="100" step="1" inputmode="numeric">%</label> | |
| 54 | <label>Height <input id="bh" type="number" min="2" max="100" step="1" inputmode="numeric">%</label> | |
| 55 | </fieldset> | |
| 56 | <p id="seen" role="status"></p> | |
| 57 | <div id="ctls"></div> | |
| 58 | <div class="row"> | |
| 59 | <button type="button" id="grow">Grow-box settings</button> | |
| 60 | <button type="button" id="measure">Measure colour</button> | |
| 61 | <button type="button" id="uncorrect">No correction</button> | |
| 62 | <button type="button" id="whole">Whole frame</button> | |
| 63 | <label for="unit">Pi camera</label><select id="unit"><option value="">not the Pi's</option></select> | |
| 64 | </div> | |
| 65 | <div class="row"><button type="button" id="save">Save</button><button type="button" id="closeTune">Close</button></div> | |
| 66 | <p id="note" role="status" aria-live="polite"></p> | |
| 67 | </section> | |
| 68 | <script type="module"> | |
| 69 | import { CANT_SEE, applyGains, cropRect, grayWorldGains, regionFromDrag, regionFromPercent, regionToPercent, visibility } from "/ui/__VERSION__/still-frame.js"; | |
| 30 | 70 | const $ = (id) => document.getElementById(id); |
| 31 | 71 | const say = (t) => { $("msg").textContent = t; }; |
| 32 | 72 | async function call(method, path, body) { |
| @@ -42,9 +82,11 @@ | ||
| 42 | 82 | const li = document.createElement("li"); |
| 43 | 83 | const name = document.createElement("b"); name.textContent = c.name; |
| 44 | 84 | const src = document.createElement("span"); src.className = "src"; src.textContent = c.source; |
| 85 | const tune = document.createElement("button"); tune.textContent = "Tune"; tune.setAttribute("aria-label", `Tune ${c.name}`); | |
| 86 | tune.onclick = () => open(c); | |
| 45 | 87 | const rm = document.createElement("button"); rm.textContent = "Remove"; rm.setAttribute("aria-label", `Remove ${c.name}`); |
| 46 | 88 | rm.onclick = () => call("DELETE", `/admin/cameras/${c.id}`).then(load, (e) => say(e.message)); |
| 47 | li.append(name, src, rm); | |
| 89 | li.append(name, src, tune, rm); | |
| 48 | 90 | return li; |
| 49 | 91 | })); |
| 50 | 92 | say(""); |
| @@ -55,6 +97,170 @@ | ||
| 55 | 97 | call("POST", "/admin/cameras", { name: $("name").value, source: $("source").value }).then(() => { $("add").reset(); load(); }, (err) => say(err.message)); |
| 56 | 98 | }; |
| 57 | 99 | load(); |
| 100 | ||
| 101 | // ---------------------------------------------------------------- Tune | |
| 102 | /** Each control: its profile field, its label, its range, and whether the camera can run it itself. */ | |
| 103 | const CONTROLS = [ | |
| 104 | { key: "exposure", label: "Exposure", min: 1, max: 2000, step: 1, auto: true, unit: "×100 µs" }, | |
| 105 | { key: "whiteBalance", label: "White balance", min: 2800, max: 6500, step: 100, auto: true, unit: "K" }, | |
| 106 | { key: "gain", label: "Gain", min: 0, max: 255, step: 1, auto: false, unit: "" }, | |
| 107 | { key: "focus", label: "Focus", min: 0, max: 1023, step: 5, auto: true, unit: "" }, | |
| 108 | { key: "backlight", label: "Backlight comp.", min: 0, max: 8, step: 1, auto: false, unit: "" }, | |
| 109 | { key: "brightness", label: "Brightness", min: -64, max: 255, step: 1, auto: false, unit: "" }, | |
| 110 | { key: "contrast", label: "Contrast", min: 0, max: 255, step: 1, auto: false, unit: "" }, | |
| 111 | { key: "saturation", label: "Saturation", min: 0, max: 255, step: 1, auto: false, unit: "" }, | |
| 112 | ]; | |
| 113 | const GROW = { exposure: 180, whiteBalance: 6500, backlight: 0 }; | |
| 114 | for (let n = 1; n <= 8; n++) $("unit").append(new Option(`so101-camera@${n}`, String(n))); | |
| 115 | /** @type {{ id: number, name: string } | null} */ let cam = null; | |
| 116 | let region = null, gains = null, timer = 0, trying = 0, frame = null, drag = null; | |
| 117 | const view = $("view"), fixed = $("fixed"); | |
| 118 | ||
| 119 | for (const c of CONTROLS) { | |
| 120 | const row = document.createElement("div"); row.className = "ctl"; | |
| 121 | const label = document.createElement("label"); label.textContent = c.label; label.htmlFor = `r-${c.key}`; | |
| 122 | const mode = document.createElement("select"); mode.id = `m-${c.key}`; mode.setAttribute("aria-label", `${c.label}: leave, auto or fixed`); | |
| 123 | mode.append(new Option("leave", "leave"), ...(c.auto ? [new Option("auto", "auto")] : []), new Option("fixed", "fixed")); | |
| 124 | const range = Object.assign(document.createElement("input"), { type: "range", id: `r-${c.key}`, min: c.min, max: c.max, step: c.step }); | |
| 125 | const out = document.createElement("output"); out.id = `o-${c.key}`; | |
| 126 | const shown = () => { range.disabled = mode.value !== "fixed"; out.textContent = mode.value === "fixed" ? `${range.value}${c.unit ? " " + c.unit : ""}` : mode.value; }; | |
| 127 | mode.onchange = () => { shown(); tryLater(); }; | |
| 128 | range.oninput = () => { mode.value = "fixed"; shown(); tryLater(); }; | |
| 129 | row.append(label, mode, range, out); | |
| 130 | $("ctls").append(row); | |
| 131 | c.show = shown; | |
| 132 | } | |
| 133 | ||
| 134 | function profile() { | |
| 135 | const p = {}; | |
| 136 | for (const c of CONTROLS) { | |
| 137 | const mode = $(`m-${c.key}`).value; | |
| 138 | if (mode === "auto") p[c.key] = "auto"; | |
| 139 | else if (mode === "fixed") p[c.key] = Number($(`r-${c.key}`).value); | |
| 140 | } | |
| 141 | return p; | |
| 142 | } | |
| 143 | function showProfile(p) { | |
| 144 | for (const c of CONTROLS) { | |
| 145 | const v = p[c.key]; | |
| 146 | $(`m-${c.key}`).value = v === undefined ? "leave" : v === "auto" ? "auto" : "fixed"; | |
| 147 | if (typeof v === "number") $(`r-${c.key}`).value = String(v); | |
| 148 | c.show(); | |
| 149 | } | |
| 150 | } | |
| 151 | /** The panel's one status line: what the Pi did, and every error, in the same live region. */ | |
| 152 | const tell = (text, bad = false) => { $("note").textContent = text; $("note").className = bad ? "bad" : ""; }; | |
| 153 | const sayPi = (pi) => { | |
| 154 | if (!pi) return tell(""); | |
| 155 | if (pi.error) return tell(`The Pi did not take it: ${pi.error}`, true); | |
| 156 | tell(`Camera ${pi.saved ? "kept" : "set"}: ${pi.set.join(", ") || "nothing"}${pi.failed.length ? `. It has no ${pi.failed.map((f) => f.control.split("=")[0]).join(", ")}` : ""}.`); | |
| 157 | }; | |
| 158 | function tryLater() { | |
| 159 | clearTimeout(timer); | |
| 160 | timer = setTimeout(async () => { | |
| 161 | if (!cam) return; | |
| 162 | const mine = ++trying; | |
| 163 | try { const r = await call("POST", `/admin/cameras/${cam.id}/try`, { profile: profile() }); if (mine === trying) sayPi(r.pi); } | |
| 164 | catch (e) { tell(e.message, true); } | |
| 165 | }, 300); | |
| 166 | } | |
| 167 | ||
| 168 | async function open(c) { | |
| 169 | const all = await call("GET", "/admin/cameras").catch((e) => (say(e.message), [])); | |
| 170 | const full = all.find((x) => x.id === c.id); | |
| 171 | if (!full) return; | |
| 172 | cam = { id: full.id, name: full.name }; | |
| 173 | region = full.look.region; gains = full.look.gains; | |
| 174 | $("tuneTitle").textContent = `Tune: ${full.name}`; | |
| 175 | $("unit").value = full.look.unit === null ? "" : String(full.look.unit); | |
| 176 | showProfile(full.look.profile); | |
| 177 | sayPi(null); | |
| 178 | showBox(); | |
| 179 | $("tune").hidden = false; | |
| 180 | $("tune").scrollIntoView({ behavior: "smooth", block: "start" }); | |
| 181 | grab(); | |
| 182 | } | |
| 183 | $("closeTune").onclick = () => { cam = null; $("tune").hidden = true; }; | |
| 184 | ||
| 185 | /** The camera now, one snapshot a second while the panel is open. */ | |
| 186 | async function grab() { | |
| 187 | const id = cam?.id; | |
| 188 | if (id === undefined) return; | |
| 189 | try { | |
| 190 | const r = await fetch(`/camera/${id}/snapshot?t=${Date.now()}`, { cache: "no-store" }); | |
| 191 | if (!r.ok) throw new Error(`the camera did not answer (${r.status})`); | |
| 192 | const bitmap = await createImageBitmap(await r.blob()); | |
| 193 | const c = document.createElement("canvas"); c.width = bitmap.width; c.height = bitmap.height; | |
| 194 | const g = c.getContext("2d", { willReadFrequently: true }); g.drawImage(bitmap, 0, 0); | |
| 195 | frame = { canvas: c, pixels: g.getImageData(0, 0, c.width, c.height) }; | |
| 196 | $("viewCap").textContent = `as the camera sees it, ${c.width}×${c.height}`; | |
| 197 | paint(); | |
| 198 | } catch (e) { $("viewCap").textContent = e.message; } | |
| 199 | if (cam?.id === id) setTimeout(grab, 1000); | |
| 200 | } | |
| 201 | ||
| 202 | /** The number inputs and the legend follow the region, however it was set. */ | |
| 203 | function showBox() { | |
| 204 | const p = regionToPercent(region); | |
| 205 | for (const k of ["x", "y", "w", "h"]) { const el = $(`b${k}`); if (document.activeElement !== el) el.value = String(p[k]); } | |
| 206 | $("boxSays").textContent = region ? `The box: ${p.w}% wide and ${p.h}% tall, ${p.x}% from the left and ${p.y}% from the top` : "The box: the whole frame"; | |
| 207 | } | |
| 208 | for (const k of ["x", "y", "w", "h"]) { | |
| 209 | $(`b${k}`).onchange = () => { | |
| 210 | try { | |
| 211 | region = regionFromPercent({ x: $("bx").value, y: $("by").value, w: $("bw").value, h: $("bh").value }); | |
| 212 | tell(""); showBox(); paint(); | |
| 213 | } catch (e) { tell(e.message, true); } | |
| 214 | }; | |
| 215 | } | |
| 216 | ||
| 217 | function paint() { | |
| 218 | showBox(); | |
| 219 | if (!frame) return; | |
| 220 | const { canvas, pixels } = frame; | |
| 221 | view.width = canvas.width; view.height = canvas.height; | |
| 222 | const v = view.getContext("2d"); | |
| 223 | v.drawImage(canvas, 0, 0); | |
| 224 | const rect = cropRect(region, canvas.width, canvas.height); | |
| 225 | if (region) { | |
| 226 | v.fillStyle = "#0008"; | |
| 227 | v.fillRect(0, 0, view.width, rect.y); v.fillRect(0, rect.y + rect.h, view.width, view.height - rect.y - rect.h); | |
| 228 | v.fillRect(0, rect.y, rect.x, rect.h); v.fillRect(rect.x + rect.w, rect.y, view.width - rect.x - rect.w, rect.h); | |
| 229 | v.strokeStyle = "#ffd400"; v.lineWidth = Math.max(2, view.width / 320); v.strokeRect(rect.x, rect.y, rect.w, rect.h); | |
| 230 | } | |
| 231 | const box = new ImageData(rect.w, rect.h); | |
| 232 | for (let row = 0; row < rect.h; row++) box.data.set(pixels.data.subarray(((rect.y + row) * pixels.width + rect.x) * 4, ((rect.y + row) * pixels.width + rect.x + rect.w) * 4), row * rect.w * 4); | |
| 233 | const seen = visibility(pixels.data, pixels.width, rect); | |
| 234 | if (gains) applyGains(box.data, gains); | |
| 235 | fixed.width = rect.w; fixed.height = rect.h; | |
| 236 | fixed.getContext("2d").putImageData(box, 0, 0); | |
| 237 | $("fixedCap").textContent = `${region ? "the box" : "the whole frame"}, ${gains ? `corrected (×${gains.r} ×${gains.g} ×${gains.b})` : "not corrected"}`; | |
| 238 | const pct = (f) => `${Math.round(f * 100)}%`; | |
| 239 | $("seen").className = seen.verdict === "ok" ? "" : "bad"; | |
| 240 | $("seen").textContent = seen.verdict === "ok" ? `The camera can see the box: ${pct(seen.blown)} blown out, ${pct(seen.dark)} black.` : `${CANT_SEE} (${pct(seen.blown)} blown out, ${pct(seen.dark)} black)`; | |
| 241 | } | |
| 242 | ||
| 243 | const at = (e) => { const b = view.getBoundingClientRect(); return { x: (e.clientX - b.left) / b.width, y: (e.clientY - b.top) / b.height }; }; | |
| 244 | view.onpointerdown = (e) => { drag = at(e); view.setPointerCapture(e.pointerId); }; | |
| 245 | view.onpointermove = (e) => { if (!drag) return; const r = regionFromDrag(drag, at(e)); if (r.w > 0.02 && r.h > 0.02) { region = r; paint(); } }; | |
| 246 | view.onpointerup = () => { drag = null; }; | |
| 247 | $("whole").onclick = () => { region = null; paint(); }; | |
| 248 | $("grow").onclick = () => { showProfile(GROW); tryLater(); }; | |
| 249 | $("uncorrect").onclick = () => { gains = null; paint(); }; | |
| 250 | $("measure").onclick = () => { | |
| 251 | if (!frame) return; | |
| 252 | const found = grayWorldGains(frame.pixels.data, frame.pixels.width, cropRect(region, frame.canvas.width, frame.canvas.height)); | |
| 253 | if (!found) return tell("Too little of the box is neither blown out nor black to judge its colour.", true); | |
| 254 | gains = found; paint(); | |
| 255 | }; | |
| 256 | $("save").onclick = async () => { | |
| 257 | if (!cam) return; | |
| 258 | const round = (r) => r && { x: +r.x.toFixed(4), y: +r.y.toFixed(4), w: +r.w.toFixed(4), h: +r.h.toFixed(4) }; | |
| 259 | try { | |
| 260 | const r = await call("PUT", `/admin/cameras/${cam.id}`, { unit: $("unit").value ? Number($("unit").value) : null, region: round(region), gains, profile: profile() }); | |
| 261 | if (r.pi) sayPi(r.pi); else tell("Saved."); | |
| 262 | } catch (e) { tell(e.message, true); } | |
| 263 | }; | |
| 58 | 264 | </script> |
| 59 | 265 | </body> |
| 60 | 266 | </html> |
Software/so101/src/camera-look.ts added +66
| @@ -0,0 +1,66 @@ | ||
| 1 | /** | |
| 2 | * How a listed camera should look, kept with it in the camera list (src/cameras.ts): its profile on the Pi | |
| 3 | * (src/camera-profile.ts), which so101-camera@<unit> it is there, the region of the frame that is the grow box, and the | |
| 4 | * colour correction its stills get. Plain data, checked here at the boundary. | |
| 5 | */ | |
| 6 | ||
| 7 | import { defaultProfile, parseProfile, ProfileError, type CameraProfile } from "./camera-profile.ts"; | |
| 8 | import { parseUnit } from "./camera-control.ts"; | |
| 9 | import { parseGains, parseRegion, type Gains, type Rect } from "./ui/still-frame.js"; | |
| 10 | ||
| 11 | export interface CameraLook { | |
| 12 | profile: CameraProfile; | |
| 13 | /** so101-camera@<unit> on the Pi; null when the camera is not one of the Pi's. */ | |
| 14 | unit: number | null; | |
| 15 | /** The part of the frame that is the box, in fractions; null for the whole frame. */ | |
| 16 | region: Rect | null; | |
| 17 | /** The colour correction for stills; null for none. */ | |
| 18 | gains: Gains | null; | |
| 19 | } | |
| 20 | ||
| 21 | const KEYS = new Set(["profile", "unit", "region", "gains"]); | |
| 22 | ||
| 23 | /** | |
| 24 | * The Pi serves so101-camera@n on port 8079 + n (8080 for the first): the unit a camera's source is, when it is one of those. | |
| 25 | * Only a guess for a new camera; the owner can set the unit on the Tune panel. | |
| 26 | */ | |
| 27 | export function unitOf(source: string): number | null { | |
| 28 | try { | |
| 29 | const port = Number(new URL(source).port); | |
| 30 | return port >= 8080 && port <= 8087 ? port - 8079 : null; | |
| 31 | } catch { | |
| 32 | return null; | |
| 33 | } | |
| 34 | } | |
| 35 | ||
| 36 | /** The look a camera has before anyone tunes it: the owner's chosen profile for its unit (src/camera-profile.ts). */ | |
| 37 | export function defaultLook(source: string): CameraLook { | |
| 38 | const unit = unitOf(source); | |
| 39 | return { profile: defaultProfile(unit), unit, region: null, gains: null }; | |
| 40 | } | |
| 41 | ||
| 42 | /** A change to a look → the look after it; fields left out keep their value. Throws a ProfileError saying what is wrong. */ | |
| 43 | export function changeLook(now: CameraLook, input: unknown): CameraLook { | |
| 44 | if (input === null || typeof input !== "object" || Array.isArray(input)) throw new ProfileError("send {profile?, unit?, region?, gains?}"); | |
| 45 | const o = input as Record<string, unknown>; | |
| 46 | for (const key of Object.keys(o)) if (!KEYS.has(key)) throw new ProfileError(`a camera's look has no "${key.slice(0, 40)}"`); | |
| 47 | const next = { ...now }; | |
| 48 | if (Object.hasOwn(o, "profile")) next.profile = parseProfile(o.profile); | |
| 49 | if (Object.hasOwn(o, "unit")) next.unit = o.unit === null ? null : parseUnit(o.unit); | |
| 50 | try { | |
| 51 | if (Object.hasOwn(o, "region")) next.region = parseRegion(o.region); | |
| 52 | if (Object.hasOwn(o, "gains")) next.gains = parseGains(o.gains); | |
| 53 | } catch (err) { | |
| 54 | throw new ProfileError((err as Error).message); | |
| 55 | } | |
| 56 | return next; | |
| 57 | } | |
| 58 | ||
| 59 | /** A look read back from the camera list's file: whatever does not check out is the default. */ | |
| 60 | export function readLook(saved: unknown, source: string): CameraLook { | |
| 61 | try { | |
| 62 | return changeLook(defaultLook(source), saved ?? {}); | |
| 63 | } catch { | |
| 64 | return defaultLook(source); | |
| 65 | } | |
| 66 | } |
Software/so101/src/camera-profile.test.ts added +149
| @@ -0,0 +1,149 @@ | ||
| 1 | import { afterEach, describe, expect, test } from "bun:test"; | |
| 2 | import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; | |
| 3 | import { tmpdir } from "node:os"; | |
| 4 | import { join } from "node:path"; | |
| 5 | import { defaultProfile, GROW_BOX, parseDevice, parseProfile, ProfileError, v4l2Calls } from "./camera-profile.ts"; | |
| 6 | import { defaultLook } from "./camera-look.ts"; | |
| 7 | import { applyProfile, cameraDevice, loadProfile, parseUnit, profileFile, saveProfile } from "./camera-control.ts"; | |
| 8 | import { main } from "./cli.ts"; | |
| 9 | ||
| 10 | const XWF = "/dev/v4l/by-id/usb-XWF-1080P-video-index0"; | |
| 11 | const dirs: string[] = []; | |
| 12 | const temp = () => { | |
| 13 | const dir = mkdtempSync(join(tmpdir(), "camera-profile-")); | |
| 14 | dirs.push(dir); | |
| 15 | return dir; | |
| 16 | }; | |
| 17 | afterEach(() => { | |
| 18 | for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true }); | |
| 19 | }); | |
| 20 | ||
| 21 | describe("a camera profile", () => { | |
| 22 | test("the grow box's hand-tuned settings become the v4l2-ctl calls the owner typed, modes first", () => { | |
| 23 | expect(v4l2Calls(XWF, GROW_BOX)).toEqual([ | |
| 24 | ["-d", XWF, "--set-ctrl=auto_exposure=1"], | |
| 25 | ["-d", XWF, "--set-ctrl=white_balance_automatic=0"], | |
| 26 | ["-d", XWF, "--set-ctrl=exposure_time_absolute=180"], | |
| 27 | ["-d", XWF, "--set-ctrl=white_balance_temperature=6500"], | |
| 28 | ["-d", XWF, "--set-ctrl=backlight_compensation=0"], | |
| 29 | ]); | |
| 30 | }); | |
| 31 | ||
| 32 | test("auto hands a control back to the camera; a field left out is left alone", () => { | |
| 33 | expect(v4l2Calls("/dev/video0", { exposure: "auto", focus: "auto", gain: 40 }).map((a) => a[2])).toEqual([ | |
| 34 | "--set-ctrl=auto_exposure=3", | |
| 35 | "--set-ctrl=focus_automatic_continuous=1", | |
| 36 | "--set-ctrl=gain=40", | |
| 37 | ]); | |
| 38 | expect(v4l2Calls("/dev/video0", {})).toEqual([]); | |
| 39 | expect(v4l2Calls("/dev/video0", { focus: 300 }).map((a) => a[2])).toEqual(["--set-ctrl=focus_automatic_continuous=0", "--set-ctrl=focus_absolute=300"]); | |
| 40 | }); | |
| 41 | ||
| 42 | test("anything but a checked number or auto is refused at the boundary", () => { | |
| 43 | for (const bad of [null, [], "x", { exposure: "180" }, { exposure: 1.5 }, { exposure: 0 }, { gain: "auto" }, { gain: 256 }, { whiteBalance: 100 }, { colour: 1 }, { exposure: "180; rm -rf /" }]) { | |
| 44 | expect(() => parseProfile(bad)).toThrow(ProfileError); | |
| 45 | } | |
| 46 | expect(parseProfile({ exposure: null, gain: 0 })).toEqual({ gain: 0 }); | |
| 47 | }); | |
| 48 | ||
| 49 | test("a device is a path under /dev, nothing that climbs out or carries a flag", () => { | |
| 50 | expect(parseDevice(` ${XWF} `)).toBe(XWF); | |
| 51 | for (const bad of ["/dev/../etc/passwd", "video0", "-d /dev/video0", "/dev/video0 --all", "", 3]) expect(() => parseDevice(bad)).toThrow(ProfileError); | |
| 52 | }); | |
| 53 | }); | |
| 54 | ||
| 55 | describe("the owner's chosen settings (#584, 2026-10-26)", () => { | |
| 56 | test("each camera starts from what the owner set by hand, as the v4l2-ctl calls they typed", () => { | |
| 57 | expect(v4l2Calls("/dev/video2", defaultProfile(2)).map((a) => a[2])).toEqual(["--set-ctrl=brightness=115", "--set-ctrl=contrast=120", "--set-ctrl=saturation=150"]); | |
| 58 | expect(v4l2Calls(XWF, defaultProfile(1)).map((a) => a[2])).toEqual(["--set-ctrl=auto_exposure=3", "--set-ctrl=white_balance_automatic=1", "--set-ctrl=backlight_compensation=1"]); | |
| 59 | expect(defaultProfile(3)).toEqual({}); | |
| 60 | expect(defaultProfile(null)).toEqual({}); | |
| 61 | }); | |
| 62 | ||
| 63 | test("a new camera on the Pi's ports starts from its unit's default; one elsewhere from nothing", () => { | |
| 64 | expect(defaultLook("http://100.89.95.68:8081/stream").profile).toEqual({ brightness: 115, contrast: 120, saturation: 150 }); | |
| 65 | expect(defaultLook("http://100.89.95.68:8080/stream").profile).toEqual({ exposure: "auto", whiteBalance: "auto", backlight: 1 }); | |
| 66 | expect(defaultLook("http://cam.local:9000/stream").profile).toEqual({}); | |
| 67 | defaultLook("http://100.89.95.68:8081/stream").profile.saturation = 1; // a copy: the defaults themselves never change | |
| 68 | expect(defaultProfile(2).saturation).toBe(150); | |
| 69 | }); | |
| 70 | ||
| 71 | test("brightness, contrast and saturation are checked like the rest", () => { | |
| 72 | expect(parseProfile({ brightness: -10, contrast: 0, saturation: 255 })).toEqual({ brightness: -10, contrast: 0, saturation: 255 }); | |
| 73 | for (const bad of [{ brightness: 300 }, { contrast: -1 }, { saturation: "auto" }, { saturation: 1.5 }]) expect(() => parseProfile(bad)).toThrow(ProfileError); | |
| 74 | }); | |
| 75 | }); | |
| 76 | ||
| 77 | describe("the Pi's side", () => { | |
| 78 | test("a saved profile round-trips through the file the camera service reads, into the calls it runs", async () => { | |
| 79 | const data = temp(); | |
| 80 | const file = profileFile(data, 1); | |
| 81 | expect(loadProfile(file)).toBeNull(); | |
| 82 | saveProfile(file, GROW_BOX); | |
| 83 | const etc = temp(); | |
| 84 | writeFileSync(join(etc, "camera1"), `# the side camera\nDEVICE=${XWF}\n`); | |
| 85 | const ran: string[][] = []; | |
| 86 | const applied = await applyProfile(cameraDevice(1, etc), loadProfile(file)!, async (args) => (ran.push(args), args[2]!.includes("backlight") ? "unknown control 'backlight_compensation'" : "")); | |
| 87 | expect(ran).toEqual(v4l2Calls(XWF, GROW_BOX)); | |
| 88 | expect(applied.set).toEqual(["auto_exposure=1", "white_balance_automatic=0", "exposure_time_absolute=180", "white_balance_temperature=6500"]); | |
| 89 | expect(applied.failed).toEqual([{ control: "backlight_compensation=0", error: "unknown control 'backlight_compensation'" }]); | |
| 90 | }); | |
| 91 | ||
| 92 | test("camera n's device comes from /etc/so101/camera<n>, a bare path or DEVICE=; a missing one says what to write", () => { | |
| 93 | const etc = temp(); | |
| 94 | writeFileSync(join(etc, "camera2"), "/dev/video2\n"); | |
| 95 | expect(cameraDevice(2, etc)).toBe("/dev/video2"); | |
| 96 | expect(() => cameraDevice(3, etc)).toThrow(/camera3 is missing/); | |
| 97 | writeFileSync(join(etc, "camera4"), "DEVICE=/dev/../etc/shadow\n"); | |
| 98 | expect(() => cameraDevice(4, etc)).toThrow(ProfileError); | |
| 99 | }); | |
| 100 | ||
| 101 | test("units are 1 to 8; an unreadable profile file is no profile", () => { | |
| 102 | expect(parseUnit("2")).toBe(2); | |
| 103 | for (const bad of ["0", "9", "../1", 1.5, "1/../../x"]) expect(() => parseUnit(bad)).toThrow(ProfileError); | |
| 104 | const data = temp(); | |
| 105 | mkdirSync(join(data, "camera-profiles")); | |
| 106 | writeFileSync(profileFile(data, 1), "{nope"); | |
| 107 | expect(loadProfile(profileFile(data, 1))).toBeNull(); | |
| 108 | }); | |
| 109 | }); | |
| 110 | ||
| 111 | describe("so101 camera-profile <n>, the camera service's ExecStartPre", () => { | |
| 112 | const quiet = async (argv: string[]) => { | |
| 113 | const [log, err] = [console.log, process.stderr.write.bind(process.stderr)]; | |
| 114 | const said: string[] = []; | |
| 115 | console.log = (...a: unknown[]) => void said.push(a.join(" ")); | |
| 116 | process.stderr.write = ((t: string) => (said.push(String(t)), true)) as typeof process.stderr.write; | |
| 117 | try { | |
| 118 | return { code: await main(argv), said: said.join("\n") }; | |
| 119 | } finally { | |
| 120 | console.log = log; | |
| 121 | process.stderr.write = err; | |
| 122 | } | |
| 123 | }; | |
| 124 | ||
| 125 | test("with no profile saved and no owner's default it leaves the camera alone and lets the stream start", async () => { | |
| 126 | const data = temp(); | |
| 127 | const { code, said } = await quiet(["camera-profile", "3", "--file", join(data, "poses.json")]); | |
| 128 | expect(code).toBe(0); | |
| 129 | expect(said).toContain("no profile saved for camera 3"); | |
| 130 | expect(said).toContain("leaving the camera as it is"); | |
| 131 | }); | |
| 132 | ||
| 133 | test("with no profile saved, the overhead camera gets the owner's 2026-10-26 settings at its start", async () => { | |
| 134 | const data = temp(); | |
| 135 | const { code, said } = await quiet(["camera-profile", "2", "--file", join(data, "poses.json"), "--device", "/dev/null-camera"]); | |
| 136 | expect(code).toBe(0); // a control the camera refuses (here: no such device) is reported, never a failed start | |
| 137 | expect(said).toContain("using the owner's default"); | |
| 138 | for (const c of ["brightness=115", "contrast=120", "saturation=150"]) expect(said).toContain(c); | |
| 139 | }); | |
| 140 | ||
| 141 | test("a bad unit or device is refused before anything runs", async () => { | |
| 142 | const data = temp(); | |
| 143 | saveProfile(profileFile(data, 1), GROW_BOX); | |
| 144 | expect((await quiet(["camera-profile", "9"])).code).toBe(1); | |
| 145 | const bad = await quiet(["camera-profile", "1", "--file", join(data, "poses.json"), "--device", "/dev/video0;reboot"]); | |
| 146 | expect(bad.code).toBe(1); | |
| 147 | expect(bad.said).toContain("a camera device is a path under /dev"); | |
| 148 | }); | |
| 149 | }); |
Software/so101/src/camera-profile.ts added +121
| @@ -0,0 +1,121 @@ | ||
| 1 | /** | |
| 2 | * A camera's profile: the few UVC controls that decide whether it sees anything under a grow light (exposure, white | |
| 3 | * balance, gain, focus, backlight compensation, brightness, contrast, saturation), as plain data, and the `v4l2-ctl` invocations that put it on the camera. | |
| 4 | * | |
| 5 | * Every field is optional: one left out is left as the camera has it. A webcam forgets all of this on a re-plug or a | |
| 6 | * reboot, so the Pi applies the saved profile each time its camera service starts (README, "Camera profile"). | |
| 7 | * | |
| 8 | * The invocations are argument lists for execFile, never a shell line: each value is a number checked here, and each | |
| 9 | * control goes in its own call, so a control one camera lacks (the XWF-1080P has no focus) does not stop the others. | |
| 10 | * Modes come before values: a UVC camera refuses an exposure time while auto exposure is on. | |
| 11 | */ | |
| 12 | ||
| 13 | export interface CameraProfile { | |
| 14 | /** "auto", or a fixed exposure time in 100 µs units (exposure_time_absolute). */ | |
| 15 | exposure?: "auto" | number; | |
| 16 | /** "auto", or a fixed colour temperature in kelvin. */ | |
| 17 | whiteBalance?: "auto" | number; | |
| 18 | gain?: number; | |
| 19 | /** "auto" (continuous autofocus), or a fixed focus position. */ | |
| 20 | focus?: "auto" | number; | |
| 21 | /** backlight_compensation: 0 keeps a bright box from darkening everything else. */ | |
| 22 | backlight?: number; | |
| 23 | brightness?: number; | |
| 24 | contrast?: number; | |
| 25 | saturation?: number; | |
| 26 | } | |
| 27 | ||
| 28 | export class ProfileError extends Error {} | |
| 29 | ||
| 30 | /** The range each number may take: wide enough for any UVC webcam, narrow enough that a typo is caught. */ | |
| 31 | export const RANGES = { | |
| 32 | exposure: [1, 10_000], | |
| 33 | whiteBalance: [2000, 10_000], | |
| 34 | gain: [0, 255], | |
| 35 | focus: [0, 1023], | |
| 36 | backlight: [0, 8], | |
| 37 | // UVC cameras disagree on these: -64..64 on some, 0..255 on others (the eMeet C960 is 0..255) | |
| 38 | brightness: [-255, 255], | |
| 39 | contrast: [0, 255], | |
| 40 | saturation: [0, 255], | |
| 41 | } as const satisfies Record<keyof CameraProfile, readonly [number, number]>; | |
| 42 | ||
| 43 | /** What the owner tuned by hand on the grow box's side camera on 2026-10-26 (#584): the inside of the dome and the tray show. */ | |
| 44 | export const GROW_BOX: CameraProfile = { exposure: 180, whiteBalance: 6500, backlight: 0 }; | |
| 45 | ||
| 46 | /** | |
| 47 | * What the owner chose by hand on the workshop's Pi on 2026-10-26 (#584), per so101-camera unit: each camera's profile | |
| 48 | * until one is saved. Camera 1 is the side XWF-1080P on :8080 ("decent" on auto, with backlight compensation); camera 2 | |
| 49 | * the overhead eMeet C960 on :8081 ("saturation up a little … looked good"). | |
| 50 | */ | |
| 51 | export const OWNER_DEFAULTS: Readonly<Record<number, CameraProfile>> = { | |
| 52 | 1: { exposure: "auto", whiteBalance: "auto", backlight: 1 }, | |
| 53 | 2: { brightness: 115, contrast: 120, saturation: 150 }, | |
| 54 | }; | |
| 55 | ||
| 56 | /** Camera `unit`'s profile before anyone saves one: the owner's choice where there is one, else nothing set. */ | |
| 57 | export function defaultProfile(unit: number | null): CameraProfile { | |
| 58 | return unit !== null && Object.hasOwn(OWNER_DEFAULTS, unit) ? { ...OWNER_DEFAULTS[unit] } : {}; | |
| 59 | } | |
| 60 | ||
| 61 | const AUTO_ALLOWED = new Set<keyof CameraProfile>(["exposure", "whiteBalance", "focus"]); | |
| 62 | ||
| 63 | /** Any JSON → a profile, or a ProfileError saying which field is wrong. Unknown fields are refused, not dropped. */ | |
| 64 | export function parseProfile(input: unknown): CameraProfile { | |
| 65 | if (input === null || typeof input !== "object" || Array.isArray(input)) throw new ProfileError("a camera profile is an object"); | |
| 66 | const out: CameraProfile = {}; | |
| 67 | for (const [key, value] of Object.entries(input)) { | |
| 68 | if (!Object.hasOwn(RANGES, key)) throw new ProfileError(`a camera profile has no "${key.slice(0, 40)}"`); | |
| 69 | const field = key as keyof CameraProfile; | |
| 70 | if (value === undefined || value === null) continue; | |
| 71 | if (value === "auto" && AUTO_ALLOWED.has(field)) { | |
| 72 | (out as Record<string, unknown>)[field] = "auto"; | |
| 73 | continue; | |
| 74 | } | |
| 75 | const [lo, hi] = RANGES[field]; | |
| 76 | if (typeof value !== "number" || !Number.isInteger(value) || value < lo || value > hi) { | |
| 77 | throw new ProfileError(`${field} is ${AUTO_ALLOWED.has(field) ? '"auto" or ' : ""}a whole number from ${lo} to ${hi}`); | |
| 78 | } | |
| 79 | (out as Record<string, unknown>)[field] = value; | |
| 80 | } | |
| 81 | return out; | |
| 82 | } | |
| 83 | ||
| 84 | /** One control to set, by its V4L2 name, in the order they must be set. */ | |
| 85 | export interface Control { | |
| 86 | name: string; | |
| 87 | value: number; | |
| 88 | } | |
| 89 | ||
| 90 | /** The controls a profile sets, modes first. */ | |
| 91 | export function controls(profile: CameraProfile): Control[] { | |
| 92 | const modes: Control[] = []; | |
| 93 | const values: Control[] = []; | |
| 94 | const { exposure, whiteBalance, focus, gain, backlight, brightness, contrast, saturation } = profile; | |
| 95 | // auto_exposure: 1 is manual, 3 is aperture priority (the webcam's "auto") | |
| 96 | if (exposure === "auto") modes.push({ name: "auto_exposure", value: 3 }); | |
| 97 | else if (exposure !== undefined) modes.push({ name: "auto_exposure", value: 1 }), values.push({ name: "exposure_time_absolute", value: exposure }); | |
| 98 | if (whiteBalance === "auto") modes.push({ name: "white_balance_automatic", value: 1 }); | |
| 99 | else if (whiteBalance !== undefined) modes.push({ name: "white_balance_automatic", value: 0 }), values.push({ name: "white_balance_temperature", value: whiteBalance }); | |
| 100 | if (focus === "auto") modes.push({ name: "focus_automatic_continuous", value: 1 }); | |
| 101 | else if (focus !== undefined) modes.push({ name: "focus_automatic_continuous", value: 0 }), values.push({ name: "focus_absolute", value: focus }); | |
| 102 | if (backlight !== undefined) values.push({ name: "backlight_compensation", value: backlight }); | |
| 103 | if (gain !== undefined) values.push({ name: "gain", value: gain }); | |
| 104 | if (brightness !== undefined) values.push({ name: "brightness", value: brightness }); | |
| 105 | if (contrast !== undefined) values.push({ name: "contrast", value: contrast }); | |
| 106 | if (saturation !== undefined) values.push({ name: "saturation", value: saturation }); | |
| 107 | return [...modes, ...values]; | |
| 108 | } | |
| 109 | ||
| 110 | /** A device path: /dev/video0 or /dev/v4l/by-id/usb-…-video-index0, nothing that climbs out of /dev. */ | |
| 111 | export function parseDevice(text: unknown): string { | |
| 112 | const path = typeof text === "string" ? text.trim() : ""; | |
| 113 | if (!/^\/dev\/[A-Za-z0-9._:+\/-]{1,200}$/.test(path) || path.split("/").includes("..")) throw new ProfileError("a camera device is a path under /dev, like /dev/v4l/by-id/<camera>-video-index0"); | |
| 114 | return path; | |
| 115 | } | |
| 116 | ||
| 117 | /** The argument list of each `v4l2-ctl` call that puts the profile on the camera at `device`, one control per call. */ | |
| 118 | export function v4l2Calls(device: string, profile: CameraProfile): string[][] { | |
| 119 | const dev = parseDevice(device); | |
| 120 | return controls(parseProfile(profile)).map((c) => ["-d", dev, `--set-ctrl=${c.name}=${c.value}`]); | |
| 121 | } |
Software/so101/src/camera-tune.test.ts added +192
| @@ -0,0 +1,192 @@ | ||
| 1 | import { afterEach, describe, expect, test } from "bun:test"; | |
| 2 | import { mkdtempSync, readFileSync } from "node:fs"; | |
| 3 | import { tmpdir } from "node:os"; | |
| 4 | import { join } from "node:path"; | |
| 5 | import { FeetechBus } from "./bus.ts"; | |
| 6 | import { FakeBus } from "./fake-bus.ts"; | |
| 7 | import { CameraHub } from "./cameras.ts"; | |
| 8 | import { GROW_BOX, v4l2Calls, type CameraProfile } from "./camera-profile.ts"; | |
| 9 | import { loadProfile, profileFile } from "./camera-control.ts"; | |
| 10 | import { sign } from "./identity.ts"; | |
| 11 | import { Lobby } from "./lobby.ts"; | |
| 12 | import { startRelay, type RunningRelay } from "./relay.ts"; | |
| 13 | import { startServer, type RunningServer } from "./server.ts"; | |
| 14 | ||
| 15 | // The Tune panel's path, end to end: /admin/cameras/<id> on the relay → PUT /camera-profile/<unit> on the arm server → | |
| 16 | // the profile on the camera and in the file `so101 camera-profile <n>` reads at the camera service's next start. | |
| 17 | ||
| 18 | const owner = { "content-type": "application/json", "sec-fetch-site": "same-origin" }; | |
| 19 | const XWF = "/dev/v4l/by-id/usb-XWF-1080P-video-index0"; | |
| 20 | let arm: RunningServer | null = null; | |
| 21 | let relay: RunningRelay | null = null; | |
| 22 | let hub: CameraHub | null = null; | |
| 23 | afterEach(async () => { | |
| 24 | await relay?.close(); | |
| 25 | await arm?.close(); | |
| 26 | hub?.close(); | |
| 27 | relay = arm = hub = null; | |
| 28 | }); | |
| 29 | ||
| 30 | /** An arm server whose camera control records what it was asked, and a relay with one camera on the Pi's port 8080. */ | |
| 31 | async function setup(control = true) { | |
| 32 | const dataDir = mkdtempSync(join(tmpdir(), "so101-tune-")); | |
| 33 | const asked: { unit: number; profile: CameraProfile }[] = []; | |
| 34 | arm = await startServer(new FeetechBus(new FakeBus([1, 2, 6])), { | |
| 35 | host: "127.0.0.1", | |
| 36 | port: 0, | |
| 37 | ids: [1, 2, 6], | |
| 38 | posesFile: join(dataDir, "poses.json"), | |
| 39 | cameraControl: control | |
| 40 | ? async (unit, profile) => { | |
| 41 | asked.push({ unit, profile }); | |
| 42 | return { set: v4l2Calls(XWF, profile).map((a) => a[2]!.slice("--set-ctrl=".length)), failed: [] }; | |
| 43 | } | |
| 44 | : undefined, | |
| 45 | }); | |
| 46 | const camerasFile = join(dataDir, "cameras.json"); | |
| 47 | hub = new CameraHub({ file: camerasFile, seed: [{ name: "Side", source: "http://127.0.0.1:8080/stream" }] }); | |
| 48 | relay = await startRelay({ port: 0, simulation: `http://127.0.0.1:${arm.port}`, cameras: hub }); | |
| 49 | const base = `http://127.0.0.1:${relay.port}`; | |
| 50 | return { base, dataDir, camerasFile, asked }; | |
| 51 | } | |
| 52 | ||
| 53 | describe("tuning a camera", () => { | |
| 54 | test("Save keeps the look with the camera and the profile on the Pi, where the camera service's next start reads it", async () => { | |
| 55 | const { base, dataDir, camerasFile, asked } = await setup(); | |
| 56 | const region = { x: 0.3, y: 0.25, w: 0.4, h: 0.5 }; | |
| 57 | const r = await fetch(`${base}/admin/cameras/1`, { method: "PUT", headers: owner, body: JSON.stringify({ profile: GROW_BOX, region, gains: { r: 2, g: 1.5, b: 0.6 } }) }); | |
| 58 | expect(r.status).toBe(200); | |
| 59 | const body = (await r.json()) as { look: unknown; pi: { saved: boolean; set: string[] } }; | |
| 60 | expect(body.look).toEqual({ profile: GROW_BOX, unit: 1, region, gains: { r: 2, g: 1.5, b: 0.6 } }); | |
| 61 | expect(body.pi.saved).toBe(true); | |
| 62 | expect(body.pi.set).toContain("exposure_time_absolute=180"); | |
| 63 | expect(asked).toEqual([{ unit: 1, profile: GROW_BOX }]); | |
| 64 | ||
| 65 | // the relay's list keeps the look across a restart | |
| 66 | expect(JSON.parse(readFileSync(camerasFile, "utf8")).cameras[0].look).toEqual(body.look); | |
| 67 | expect(new CameraHub({ file: camerasFile }).look(1)).toEqual(body.look as never); | |
| 68 | // the Pi keeps the profile where `so101 camera-profile 1` finds it, and it turns into the calls the owner typed by hand | |
| 69 | expect(v4l2Calls(XWF, loadProfile(profileFile(dataDir, 1))!).map((a) => a[2])).toEqual([ | |
| 70 | "--set-ctrl=auto_exposure=1", | |
| 71 | "--set-ctrl=white_balance_automatic=0", | |
| 72 | "--set-ctrl=exposure_time_absolute=180", | |
| 73 | "--set-ctrl=white_balance_temperature=6500", | |
| 74 | "--set-ctrl=backlight_compensation=0", | |
| 75 | ]); | |
| 76 | }); | |
| 77 | ||
| 78 | test("a slider's try goes on the camera but is not kept; a region alone is not sent to the Pi", async () => { | |
| 79 | const { base, dataDir, asked } = await setup(); | |
| 80 | const tried = await fetch(`${base}/admin/cameras/1/try`, { method: "POST", headers: owner, body: JSON.stringify({ profile: { exposure: 300 } }) }); | |
| 81 | expect(tried.status).toBe(200); | |
| 82 | expect(((await tried.json()) as { pi: { saved: boolean } }).pi.saved).toBe(false); | |
| 83 | expect(asked).toEqual([{ unit: 1, profile: { exposure: 300 } }]); | |
| 84 | expect(loadProfile(profileFile(dataDir, 1))).toBeNull(); | |
| 85 | const region = await fetch(`${base}/admin/cameras/1`, { method: "PUT", headers: owner, body: JSON.stringify({ region: null }) }); | |
| 86 | expect(((await region.json()) as { pi: unknown }).pi).toBeNull(); | |
| 87 | expect(asked.length).toBe(1); | |
| 88 | }); | |
| 89 | ||
| 90 | test("bad input is refused at the boundary and the camera is left as it was", async () => { | |
| 91 | const { base, asked } = await setup(); | |
| 92 | const put = (body: string, path = "/admin/cameras/1") => fetch(`${base}${path}`, { method: "PUT", headers: owner, body }); | |
| 93 | expect((await put("{nope")).status).toBe(400); | |
| 94 | expect((await put(JSON.stringify({ profile: { exposure: "180; reboot" } }))).status).toBe(400); | |
| 95 | expect((await put(JSON.stringify({ region: { x: 0.9, y: 0, w: 0.5, h: 0.5 } }))).status).toBe(400); | |
| 96 | expect((await put(JSON.stringify({ gains: { r: 9, g: 1, b: 1 } }))).status).toBe(400); | |
| 97 | expect((await put(JSON.stringify({ unit: 12 }))).status).toBe(400); | |
| 98 | expect((await put(JSON.stringify({ colour: 1 }))).status).toBe(400); | |
| 99 | expect((await put("{}", "/admin/cameras/9")).status).toBe(404); | |
| 100 | expect((await fetch(`${base}/admin/cameras/1/try`, { method: "DELETE", headers: owner })).status).toBe(405); | |
| 101 | expect(hub!.has(1)).toBe(true); | |
| 102 | expect((await fetch(`${base}/admin/cameras/1`, { method: "PUT", headers: { ...owner, "sec-fetch-site": "cross-site" }, body: "{}" })).status).toBe(403); | |
| 103 | expect(asked).toEqual([]); | |
| 104 | expect(hub!.look(1)).toEqual({ profile: { exposure: "auto", whiteBalance: "auto", backlight: 1 }, unit: 1, region: null, gains: null }); | |
| 105 | }); | |
| 106 | ||
| 107 | test("nobody reaches the arm server's camera profiles through the relay, not even with a good body", async () => { | |
| 108 | const { base, asked } = await setup(); | |
| 109 | expect((await fetch(`${base}/camera-profile/1`, { method: "PUT", headers: owner, body: JSON.stringify(GROW_BOX) })).status).toBe(404); | |
| 110 | expect((await fetch(`${base}/camera-profile/1`)).status).toBe(404); | |
| 111 | expect(asked).toEqual([]); | |
| 112 | }); | |
| 113 | ||
| 114 | test("the simulation has no camera to tune: the look is kept and the panel is told why the Pi did not take it", async () => { | |
| 115 | const { base } = await setup(false); | |
| 116 | const r = await fetch(`${base}/admin/cameras/1`, { method: "PUT", headers: owner, body: JSON.stringify({ profile: GROW_BOX }) }); | |
| 117 | expect(r.status).toBe(200); | |
| 118 | expect(((await r.json()) as { pi: { error: string } }).pi.error).toContain("simulation"); | |
| 119 | expect(hub!.look(1)!.profile).toEqual(GROW_BOX); | |
| 120 | }); | |
| 121 | ||
| 122 | test("only the owner tunes a camera", async () => { | |
| 123 | hub = new CameraHub({ seed: [{ name: "Side", source: "http://127.0.0.1:8080/stream" }] }); | |
| 124 | const key = "k".repeat(32); | |
| 125 | const lobby = new Lobby({ identityKey: key }); | |
| 126 | relay = await startRelay({ port: 0, simulation: "http://127.0.0.1:1", cameras: hub, lobby }); | |
| 127 | const later = Date.now() + 60_000; | |
| 128 | const as = (identity: string | null) => { | |
| 129 | const headers: Record<string, string> = { ...owner }; | |
| 130 | if (identity) Object.assign(headers, { "x-forge-identity": identity, "x-forge-signature": sign(identity, key) }); | |
| 131 | return fetch(`http://127.0.0.1:${relay!.port}/admin/cameras/1`, { method: "PUT", headers, body: JSON.stringify({ region: { x: 0, y: 0, w: 0.5, h: 0.5 } }) }); | |
| 132 | }; | |
| 133 | expect((await as(null)).status).toBe(401); | |
| 134 | expect((await as(`u1|alice|user|${later}`)).status).toBe(403); | |
| 135 | expect(hub.look(1)!.region).toBeNull(); | |
| 136 | expect((await as(`u2|trevato|admin|${later}`)).status).toBe(200); | |
| 137 | expect(hub.look(1)!.region).toEqual({ x: 0, y: 0, w: 0.5, h: 0.5 }); | |
| 138 | lobby.close(); | |
| 139 | }); | |
| 140 | ||
| 141 | test("the cameras page loads the still-frame module from a path the arm server serves", async () => { | |
| 142 | const { base } = await setup(); | |
| 143 | const page = await (await fetch(`${base}/admin/camera-list`)).text(); | |
| 144 | const path = /"(\/ui\/[0-9a-f]{12}\/still-frame\.js)"/.exec(page)?.[1]; | |
| 145 | expect(path).toBeDefined(); | |
| 146 | const module = await fetch(`${base}${path}`); | |
| 147 | expect(module.status).toBe(200); | |
| 148 | expect(await module.text()).toContain("export function grayWorldGains"); | |
| 149 | }); | |
| 150 | }); | |
| 151 | ||
| 152 | describe("the Tune panel, read rather than rendered (there is no browser here)", () => { | |
| 153 | const page = readFileSync(new URL("./camera-list.html", import.meta.url), "utf8"); | |
| 154 | ||
| 155 | test("on a 360 px phone each control stacks: label and mode on one line, a full-width slider and its value under them", () => { | |
| 156 | const wide = /\.ctl \{[^}]*grid-template-columns: ([^;]+);/.exec(page)?.[1]; | |
| 157 | expect(wide).toBe("7.5rem 6.5rem 1fr 4rem"); | |
| 158 | const narrow = /@media \(max-width: (\d+)rem\) \{ \.ctl \{[^}]*grid-template-columns: ([^;]+);/.exec(page); | |
| 159 | expect(Number(narrow?.[1]) * 16).toBeGreaterThanOrEqual(360); | |
| 160 | // two columns, filled in source order (label, mode, range, output): the slider gets the whole 1fr of its own row | |
| 161 | expect(narrow?.[2]).toBe("1fr auto"); | |
| 162 | expect(page.indexOf('row.append(label, mode, range, out)')).toBeGreaterThan(0); | |
| 163 | }); | |
| 164 | ||
| 165 | test("the box can be set without a pointer, and the pictures say what they are", () => { | |
| 166 | for (const [id, name] of [["bx", "Left"], ["by", "Top"], ["bw", "Width"], ["bh", "Height"]]) { | |
| 167 | expect(page).toMatch(new RegExp(`<label>${name} <input id="${id}" type="number"`)); | |
| 168 | } | |
| 169 | expect(page).toContain("regionFromPercent({ x: $(\"bx\").value"); | |
| 170 | for (const id of ["view", "fixed"]) expect(page).toMatch(new RegExp(`<canvas id="${id}"[^>]*role="img" aria-label="[^"]{20,}"`)); | |
| 171 | }); | |
| 172 | ||
| 173 | test("the panel has one live status line for results and errors alike, and says it needs JavaScript", () => { | |
| 174 | const panel = page.slice(page.indexOf('<section id="tune"'), page.indexOf("</section>")); | |
| 175 | expect(panel.match(/role="status"/g)?.length).toBe(2); // #seen (can it see the box) and #note (everything else) | |
| 176 | expect(page).not.toContain('$("pi")'); | |
| 177 | expect(page).toMatch(/<noscript>[^<]*<p>[^<]*tune them/); | |
| 178 | }); | |
| 179 | }); | |
| 180 | ||
| 181 | describe("the arm server's camera profiles", () => { | |
| 182 | test("a unit outside 1-8 or a bad profile is refused before anything runs", async () => { | |
| 183 | const { asked } = await setup(); | |
| 184 | const at = `http://127.0.0.1:${arm!.port}`; | |
| 185 | expect((await fetch(`${at}/camera-profile/9`, { method: "PUT", body: "{}" })).status).toBe(400); | |
| 186 | expect((await fetch(`${at}/camera-profile/..%2F1`, { method: "PUT", body: "{}" })).status).toBe(400); | |
| 187 | expect((await fetch(`${at}/camera-profile/1`, { method: "PUT", body: JSON.stringify({ gain: -1 }) })).status).toBe(400); | |
| 188 | expect(asked).toEqual([]); | |
| 189 | expect(await (await fetch(`${at}/camera-profile/1`)).json()).toEqual({ unit: 1, saved: false, profile: { exposure: "auto", whiteBalance: "auto", backlight: 1 } }); | |
| 190 | expect(await (await fetch(`${at}/camera-profile/3`)).json()).toEqual({ unit: 3, saved: false, profile: {} }); | |
| 191 | }); | |
| 192 | }); |
Software/so101/src/cameras.ts +23 −3
| @@ -9,12 +9,15 @@ | ||
| 9 | 9 | import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; |
| 10 | 10 | import { dirname } from "node:path"; |
| 11 | 11 | import { CameraFeed, type FeedOptions } from "./camera-feed.ts"; |
| 12 | import { changeLook, defaultLook, readLook, type CameraLook } from "./camera-look.ts"; | |
| 12 | 13 | |
| 13 | 14 | export interface CameraSpec { |
| 14 | 15 | id: number; |
| 15 | 16 | name: string; |
| 16 | 17 | /** The stream's URL, normalised: scheme, host, port, path. */ |
| 17 | 18 | source: string; |
| 19 | /** Its profile on the Pi, its region and its colour correction (src/camera-look.ts). */ | |
| 20 | look: CameraLook; | |
| 18 | 21 | } |
| 19 | 22 | |
| 20 | 23 | export interface CameraInfo extends CameraSpec { |
| @@ -69,7 +72,9 @@ export class CameraHub { | ||
| 69 | 72 | if (this.file && existsSync(this.file)) { |
| 70 | 73 | try { |
| 71 | 74 | const saved = JSON.parse(readFileSync(this.file, "utf8")) as { next?: number; cameras?: CameraSpec[] }; |
| 72 | this.specs = (saved.cameras ?? []).filter((c) => Number.isSafeInteger(c.id) && typeof c.name === "string" && typeof c.source === "string"); | |
| 75 | this.specs = (saved.cameras ?? []) | |
| 76 | .filter((c) => Number.isSafeInteger(c.id) && typeof c.name === "string" && typeof c.source === "string") | |
| 77 | .map((c) => ({ id: c.id, name: c.name, source: c.source, look: readLook(c.look, c.source) })); | |
| 73 | 78 | this.next = Math.max(saved.next ?? 1, ...this.specs.map((c) => c.id + 1)); |
| 74 | 79 | } catch (err) { |
| 75 | 80 | console.error(`cameras: ${this.file} unreadable (${(err as Error).message}); starting from the default`); |
| @@ -81,7 +86,7 @@ export class CameraHub { | ||
| 81 | 86 | private seed(seed: HubOptions["seed"]): void { |
| 82 | 87 | this.specs = []; |
| 83 | 88 | this.next = 1; |
| 84 | for (const s of seed ?? []) this.specs.push({ id: this.next++, name: s.name, source: s.source }); | |
| 89 | for (const s of seed ?? []) this.specs.push({ id: this.next++, name: s.name, source: s.source, look: defaultLook(s.source) }); | |
| 85 | 90 | this.save(); |
| 86 | 91 | } |
| 87 | 92 | |
| @@ -98,12 +103,27 @@ export class CameraHub { | ||
| 98 | 103 | |
| 99 | 104 | add(name: unknown, source: unknown): CameraSpec { |
| 100 | 105 | if (this.specs.length >= MAX_CAMERAS) throw new CameraError(`at most ${MAX_CAMERAS} cameras`); |
| 101 | const spec = { id: this.next++, name: parseName(name), source: parseSource(source) }; | |
| 106 | const at = parseSource(source); | |
| 107 | const spec = { id: this.next++, name: parseName(name), source: at, look: defaultLook(at) }; | |
| 102 | 108 | this.specs.push(spec); |
| 103 | 109 | this.save(); |
| 104 | 110 | return spec; |
| 105 | 111 | } |
| 106 | 112 | |
| 113 | /** Change a camera's look and keep it; null for an unknown camera. Throws a ProfileError for a bad change. */ | |
| 114 | setLook(id: number, change: unknown): CameraLook | null { | |
| 115 | const spec = this.specs.find((c) => c.id === id); | |
| 116 | if (!spec) return null; | |
| 117 | spec.look = changeLook(spec.look, change); | |
| 118 | this.save(); | |
| 119 | return spec.look; | |
| 120 | } | |
| 121 | ||
| 122 | /** A listed camera's look, or null when unknown. */ | |
| 123 | look(id: number): CameraLook | null { | |
| 124 | return this.specs.find((c) => c.id === id)?.look ?? null; | |
| 125 | } | |
| 126 | ||
| 107 | 127 | remove(id: number): boolean { |
| 108 | 128 | const at = this.specs.findIndex((c) => c.id === id); |
| 109 | 129 | if (at < 0) return false; |
Software/so101/src/cli.ts +25 −1
| @@ -1,4 +1,7 @@ | ||
| 1 | 1 | import { closeSync, openSync, readFileSync, renameSync, writeFileSync } from "node:fs"; |
| 2 | import { dirname } from "node:path"; | |
| 3 | import { applyProfile, cameraDevice, loadProfile, parseUnit, profileFile } from "./camera-control.ts"; | |
| 4 | import { defaultProfile, parseDevice } from "./camera-profile.ts"; | |
| 2 | 5 | import { parseArgs } from "node:util"; |
| 3 | 6 | import { SerialPort } from "serialport"; |
| 4 | 7 | import { FeetechBus, range } from "./bus.ts"; |
| @@ -47,6 +50,9 @@ usage: so101 <command> [options] | ||
| 47 | 50 | pose list | pose delete <name> |
| 48 | 51 | serve [--listen 0.0.0.0:8101] [--file poses.json] [--model <dir with so101_new_calib.urdf>] |
| 49 | 52 | HTTP API + WebSocket target stream for this arm (run where the follower is) |
| 53 | camera-profile <n> [--file poses.json] [--device /dev/v4l/by-id/…] | |
| 54 | put the profile saved on /admin on so101-camera@n's camera (its ExecStartPre); | |
| 55 | the device is read from /etc/so101/camera<n> unless given | |
| 50 | 56 | leader --to ws://host:8101 [--hz 60] |
| 51 | 57 | read this leader arm and drive a remote follower through its serve |
| 52 | 58 | setup [--dry-run] guided: find both arms, reuse calibration, then drive |
| @@ -85,6 +91,7 @@ const OPTIONS = { | ||
| 85 | 91 | to: { type: "string" }, |
| 86 | 92 | model: { type: "string" }, |
| 87 | 93 | ssh: { type: "string" }, |
| 94 | device: { type: "string" }, | |
| 88 | 95 | } as const; |
| 89 | 96 | |
| 90 | 97 | interface Values { |
| @@ -111,6 +118,7 @@ interface Values { | ||
| 111 | 118 | model?: string; |
| 112 | 119 | to?: string; |
| 113 | 120 | ssh?: string; |
| 121 | device?: string; | |
| 114 | 122 | } |
| 115 | 123 | |
| 116 | 124 | type Command = (values: Values, positionals: string[]) => Promise<void>; |
| @@ -522,7 +530,8 @@ const COMMANDS: Record<string, Command> = { | ||
| 522 | 530 | const port = bounded(integer(portText ?? "8101", "--listen port"), 1, 65535, "--listen port"); |
| 523 | 531 | return withBus(values, async (bus) => { |
| 524 | 532 | const { startServer } = await import("./server.ts"); // lazily: it reaches bun:sqlite, which Node lacks |
| 525 | const server = await startServer(bus, { host, port, posesFile: values.file ?? "poses.json", modelDir: values.model, gainSyncMs: 2000 }); | |
| 533 | const cameraControl = (unit: number, profile: Parameters<typeof applyProfile>[1]) => applyProfile(cameraDevice(unit), profile); | |
| 534 | const server = await startServer(bus, { host, port, posesFile: values.file ?? "poses.json", modelDir: values.model, gainSyncMs: 2000, cameraControl }); | |
| 526 | 535 | console.log(`serving ${bus.path} on http://${host}:${server.port} — open it in a browser; the API is listed in src/server.ts`); |
| 527 | 536 | console.log("ctrl-c to stop"); |
| 528 | 537 | await new Promise<void>((resolve) => process.once("SIGINT", () => resolve())); |
| @@ -532,6 +541,21 @@ const COMMANDS: Record<string, Command> = { | ||
| 532 | 541 | }); |
| 533 | 542 | }, |
| 534 | 543 | |
| 544 | "camera-profile": async (values, [unitText]) => { | |
| 545 | if (unitText === undefined) throw new Error("usage: so101 camera-profile <n> [--file poses.json] [--device /dev/v4l/by-id/…]"); | |
| 546 | const unit = parseUnit(unitText); | |
| 547 | const file = profileFile(dirname(values.file ?? "poses.json"), unit); | |
| 548 | const saved = loadProfile(file); | |
| 549 | // Until the owner saves one, the profile they chose by hand on 2026-10-26 (#584); a camera with neither streams as it likes. | |
| 550 | const profile = saved ?? defaultProfile(unit); | |
| 551 | if (!Object.keys(profile).length) return console.log(`no profile saved for camera ${unit} (${file}); leaving the camera as it is`); | |
| 552 | if (!saved) console.log(`no profile saved for camera ${unit} (${file}); using the owner's default`); | |
| 553 | const device = values.device === undefined ? cameraDevice(unit) : parseDevice(values.device); | |
| 554 | const { set, failed } = await applyProfile(device, profile); | |
| 555 | console.log(`camera ${unit} (${device}): set ${set.join(" ") || "nothing"}`); | |
| 556 | for (const f of failed) console.log(` not set: ${f.control} (${f.error})`); | |
| 557 | }, | |
| 558 | ||
| 535 | 559 | leader: (values) => { |
| 536 | 560 | if (!values.to) throw new Error("usage: so101 leader --to ws://host:8101 [--hz 60]"); |
| 537 | 561 | const hz = bounded(integer(values.hz ?? "60", "--hz"), 1, 120, "--hz"); |
Software/so101/src/relay.ts +68 −7
| @@ -53,6 +53,10 @@ | ||
| 53 | 53 | * GET /story, /story/arm.webp the story of the arm for anyone, docs/story.md with the camera and the week's scorecard; its preview photo (src/story.ts) |
| 54 | 54 | * GET /admin/camera-list the owner's page for the camera list |
| 55 | 55 | * POST /admin/cameras {"name", "source"}, DELETE /admin/cameras/<id> the owner changes the list; viewers see it within seconds |
| 56 | * PUT /admin/cameras/<id> {"profile"?, "unit"?, "region"?, "gains"?} the owner's Tune panel saves a camera's look (src/camera-look.ts); | |
| 57 | * a profile goes on to the Pi's camera and is kept there (the arm server's PUT /camera-profile/<unit>) | |
| 58 | * POST /admin/cameras/<id>/try {"profile"} put a profile on the Pi's camera without keeping it: the panel's sliders, live | |
| 59 | * (the arm server's own /camera-profile/ is never passed through: only these two reach it, for the owner) | |
| 56 | 60 | * Resting (the owner closed it, or a real arm unheard for two minutes; src/opening.ts): / is the resting card |
| 57 | 61 | * (src/resting-page.ts), /status answers {"open": false, "reason", "note", "reopenAt", "lastOpenAt"} without asking the |
| 58 | 62 | * arm, every turn has ended, the cameras and /follow are refused (and those open are closed), and nothing goes to the arm |
| @@ -64,6 +68,7 @@ | ||
| 64 | 68 | * turn or no turn: GET answers it or null, PUT {"joints": {...}} stores it |
| 65 | 69 | */ |
| 66 | 70 | |
| 71 | import { createHash } from "node:crypto"; | |
| 67 | 72 | import { readFileSync } from "node:fs"; |
| 68 | 73 | import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; |
| 69 | 74 | import type { Duplex } from "node:stream"; |
| @@ -97,6 +102,7 @@ import { readTuning } from "./tuning-panel.ts"; | ||
| 97 | 102 | import { readServos } from "./servo-panel.ts"; |
| 98 | 103 | import { parseFps, type FeedOptions } from "./camera-feed.ts"; |
| 99 | 104 | import { CameraError, CameraHub } from "./cameras.ts"; |
| 105 | import { parseProfile, ProfileError, type CameraProfile } from "./camera-profile.ts"; | |
| 100 | 106 | import { VideoFeed, type VideoSources } from "./video-feed.ts"; |
| 101 | 107 | import type { VideoCard } from "./video-panel.ts"; |
| 102 | 108 | import { LADDER } from "./video-fanout.ts"; |
| @@ -189,8 +195,15 @@ const MAX_BODY = 1_000_000; | ||
| 189 | 195 | const VIDEO = /^\/camera\/(\d{1,9})\/video$/; |
| 190 | 196 | const CAMERA_WS = /^\/camera\/(\d{1,9})\/ws$/; |
| 191 | 197 | const CAMERA = /^\/camera\/(\d{1,9})\/(stream|state|snapshot)$/; |
| 192 | const ADMIN_CAMERA = /^\/admin\/cameras(?:\/(\d{1,9}))?$/; | |
| 193 | const CAMERA_PAGE = readFileSync(new URL("./camera-list.html", import.meta.url), "utf8"); | |
| 198 | const ADMIN_CAMERA = /^\/admin\/cameras(?:\/(\d{1,9})(\/try)?)?$/; | |
| 199 | /** The cameras page, with the still-frame module under a name that changes with its content (the arm server's /ui/ takes any). */ | |
| 200 | const STILL_FRAME = new URL("./ui/still-frame.js", import.meta.url); | |
| 201 | const CAMERA_PAGE = readFileSync(new URL("./camera-list.html", import.meta.url), "utf8").replaceAll( | |
| 202 | "/ui/__VERSION__/", | |
| 203 | `/ui/${createHash("sha256").update(readFileSync(STILL_FRAME)).digest("hex").slice(0, 12)}/`, | |
| 204 | ); | |
| 205 | /** The arm server's camera profiles: the relay asks for them itself, for the owner, and passes nobody's request on. */ | |
| 206 | const CAMERA_PROFILE = /^\/camera-profile(?:\/|$)/; | |
| 194 | 207 | /** Served from this process whichever arm is live: they describe the app, not the arm. */ |
| 195 | 208 | const LOCAL = /^\/(?:index\.html|leader-map\.js)?$|^\/(?:model|vendor|printables|ui|app)(?:\/|$)/; |
| 196 | 209 | /** The page itself, which the relay writes the first view's facts into. */ |
| @@ -695,11 +708,57 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 695 | 708 | res.end(out); |
| 696 | 709 | }; |
| 697 | 710 | |
| 698 | /** The owner's changes to the camera list. */ | |
| 699 | const administer = async (req: IncomingMessage, res: ServerResponse, id: number | null) => { | |
| 711 | /** | |
| 712 | * Put a profile on so101-camera@<unit>'s camera through the arm server, kept there for the camera's next start unless only | |
| 713 | * tried: what it set and what the camera refused, or why it could not. | |
| 714 | */ | |
| 715 | const toPi = async (unit: number | null, profile: CameraProfile, keep: boolean): Promise<Record<string, unknown>> => { | |
| 716 | if (unit === null) return { error: "this camera is not one of the Pi's (no so101-camera unit): set its unit on the Tune panel" }; | |
| 717 | try { | |
| 718 | const answer = await fetch(`${remote && onArm() ? remote.http : simulation}/camera-profile/${unit}${keep ? "" : "?try=1"}`, { | |
| 719 | method: "PUT", | |
| 720 | headers: { "content-type": "application/json" }, | |
| 721 | body: JSON.stringify(profile), | |
| 722 | signal: AbortSignal.timeout(Math.max(readTimeoutMs, 5000)), | |
| 723 | cache: "no-store", | |
| 724 | }); | |
| 725 | const body = (await answer.json().catch(() => ({}))) as Record<string, unknown>; | |
| 726 | if (answer.status === 404) return { error: "the arm server on the Pi is older than camera profiles: deploy it" }; | |
| 727 | if (!answer.ok) return { error: typeof body.error === "string" ? body.error.slice(0, 200) : `the arm server answered ${answer.status}` }; | |
| 728 | const list = (v: unknown) => (Array.isArray(v) ? v.slice(0, 16) : []); | |
| 729 | return { saved: body.saved === true, set: list(body.set).filter((c) => typeof c === "string"), failed: list(body.failed) }; | |
| 730 | } catch (err) { | |
| 731 | return { error: `the arm server did not answer (${reason(err)})` }; | |
| 732 | } | |
| 733 | }; | |
| 734 | ||
| 735 | /** The owner's changes to the camera list, and to each camera's look. */ | |
| 736 | const administer = async (req: IncomingMessage, res: ServerResponse, id: number | null, trying: boolean) => { | |
| 700 | 737 | const who = lobby ? lobby.who(req) : null; |
| 701 | 738 | if (lobby && !who?.owner) return json(res, who ? 403 : 401, { error: "only the owner may change the cameras" }); |
| 702 | 739 | if (req.method === "GET" && id === null) return json(res, 200, cameras.list()); |
| 740 | if (id !== null && (trying ? req.method === "POST" : req.method === "PUT")) { | |
| 741 | const look = cameras.look(id); | |
| 742 | if (!look) return json(res, 404, { error: "no such camera" }); | |
| 743 | let input: Record<string, unknown>; | |
| 744 | try { | |
| 745 | input = JSON.parse((await readBody(req)).toString("utf8") || "{}"); | |
| 746 | } catch { | |
| 747 | return json(res, 400, { error: "send JSON" }); | |
| 748 | } | |
| 749 | try { | |
| 750 | if (trying) { | |
| 751 | if (input === null || typeof input !== "object") throw new ProfileError("send {\"profile\"}"); | |
| 752 | return json(res, 200, { pi: await toPi(look.unit, parseProfile(input.profile), false) }); | |
| 753 | } | |
| 754 | const saved = cameras.setLook(id, input)!; | |
| 755 | const pi = input !== null && typeof input === "object" && Object.hasOwn(input, "profile") ? await toPi(saved.unit, saved.profile, true) : null; | |
| 756 | return json(res, 200, { look: saved, pi }); | |
| 757 | } catch (err) { | |
| 758 | if (err instanceof ProfileError) return json(res, 400, { error: err.message }); | |
| 759 | throw err; | |
| 760 | } | |
| 761 | } | |
| 703 | 762 | if (req.method === "POST" && id === null) { |
| 704 | 763 | let input: { name?: unknown; source?: unknown }; |
| 705 | 764 | try { |
| @@ -714,8 +773,9 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 714 | 773 | throw err; |
| 715 | 774 | } |
| 716 | 775 | } |
| 717 | if (req.method === "DELETE" && id !== null) return cameras.remove(id) ? json(res, 200, { ok: true }) : json(res, 404, { error: "no such camera" }); | |
| 718 | return json(res, 405, { error: "GET or POST /admin/cameras, DELETE /admin/cameras/<id>" }); | |
| 776 | if (req.method === "DELETE" && id !== null && !trying) return cameras.remove(id) ? json(res, 200, { ok: true }) : json(res, 404, { error: "no such camera" }); | |
| 777 | if (trying) return json(res, 405, { error: "POST /admin/cameras/<id>/try" }); | |
| 778 | return json(res, 405, { error: "GET or POST /admin/cameras, PUT or DELETE /admin/cameras/<id>" }); | |
| 719 | 779 | }; |
| 720 | 780 | |
| 721 | 781 | /** A person's own calibration: signed in is enough, it never touches the arm. */ |
| @@ -805,7 +865,8 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 805 | 865 | return res.end(CAMERA_PAGE); |
| 806 | 866 | } |
| 807 | 867 | const cameraAdmin = ADMIN_CAMERA.exec(url.pathname); |
| 808 | if (cameraAdmin) return await administer(req, res, cameraAdmin[1] === undefined ? null : Number(cameraAdmin[1])); | |
| 868 | if (cameraAdmin) return await administer(req, res, cameraAdmin[1] === undefined ? null : Number(cameraAdmin[1]), cameraAdmin[2] !== undefined); | |
| 869 | if (CAMERA_PROFILE.test(url.pathname)) return json(res, 404, { error: "no such route" }); | |
| 809 | 870 | if (lobby && url.pathname === "/admin/lift") return await handleLift(lobby, req, res, readBody); |
| 810 | 871 | if (lobby && (url.pathname === "/admin" || url.pathname === "/admin/health")) return await handleAdmin(lobby, req, res, url, readBody, askPrecision, askServos, videoCards, options.piSync ? () => options.piSync!.versions() : undefined, tuneP); |
| 811 | 872 | if (serveStory(req, res, url.pathname, { cameras: () => (restingNow() ? [] : cameras.list()), numbers: storyNumbers })) return; |
Software/so101/src/server.ts +40
| @@ -29,6 +29,10 @@ | ||
| 29 | 29 | * POST /play {"name": "wave", "rate"?: 1, "torque"?: 70, "release"?: false} |
| 30 | 30 | * GET /calibration {"camera": {...}|null, "handeye": {...}|null, ...} what the vision kit published |
| 31 | 31 | * PUT /calibration/<kind> store that JSON next to the poses (kind: camera, camera2, handeye, ...) |
| 32 | * GET /camera-profile/<n> {"unit", "saved", "profile"}: camera n's saved profile (src/camera-profile.ts), else the owner's default for it | |
| 33 | * PUT /camera-profile/<n>[?try=1] {"exposure"?, "whiteBalance"?, "gain"?, "focus"?, "backlight"?, "brightness"?, "contrast"?, "saturation"?} put it on so101-camera@n's | |
| 34 | * camera now, one control at a time, and keep it for the camera service's next start (?try=1: only put it on); | |
| 35 | * {"saved", "set", "failed"}. Without camera control (the simulation) nothing runs: 501, and nothing is kept | |
| 32 | 36 | * GET /settings the safe zone: {"zone": "enforce"|"warn"|"off", "floorMarginMm", "baseMarginMm", "jointOffsetsDeg"}, |
| 33 | 37 | * and the safety limits of src/safety.ts (speed, acceleration, rate, dead-man, park, gripper force, joint limits) |
| 34 | 38 | * PUT /settings change any of those; "servoGains" takes partial per-joint patches ({"2": {"p": 80}}), joints 1-5, 0-254 |
| @@ -78,6 +82,8 @@ import { GainSync } from "./servo-gain-sync.ts"; | ||
| 78 | 82 | import { TUNABLE, ParamsError, factoryValues, readServoParams, recordFactory, validateWrite, writeServoParams } from "./servo-params.ts"; |
| 79 | 83 | import { WebSocketServer, type WebSocket } from "ws"; |
| 80 | 84 | import { browserModule } from "./browser-modules.ts"; |
| 85 | import { defaultProfile, parseProfile, ProfileError, type CameraProfile } from "./camera-profile.ts"; | |
| 86 | import { loadProfile, parseUnit, profileFile, saveProfile, type Applied } from "./camera-control.ts"; | |
| 81 | 87 | import { Clock, CommandGate, HELD_CLOSE, isRelease, sentAtOf } from "./command-gate.ts"; |
| 82 | 88 | import { FLOOD_CLOSE, FrameRate, frameBytes, MAX_FRAME_BYTES, TOO_BIG_CLOSE } from "./frame-limit.ts"; |
| 83 | 89 | import type { FeetechBus } from "./bus.ts"; |
| @@ -136,6 +142,8 @@ export interface ServerOptions { | ||
| 136 | 142 | now?: () => number; |
| 137 | 143 | /** Longest /watch goes without sending an unchanged status, ms (200): keep it inside the relay link's staleMs. */ |
| 138 | 144 | watchQuietMs?: number; |
| 145 | /** Puts a profile on so101-camera@<unit>'s camera (src/camera-control.ts); only `so101 serve` on the Pi gives one, so the simulation never runs v4l2-ctl. */ | |
| 146 | cameraControl?: (unit: number, profile: CameraProfile) => Promise<Applied>; | |
| 139 | 147 | } |
| 140 | 148 | |
| 141 | 149 | export interface RunningServer { |
| @@ -827,6 +835,38 @@ export async function startServer(bus: FeetechBus, options: ServerOptions = {}): | ||
| 827 | 835 | res.writeHead(200, { "content-type": "text/javascript; charset=utf-8", "cache-control": "no-cache", etag: served.etag }); |
| 828 | 836 | return res.end(served.js); |
| 829 | 837 | } |
| 838 | const cameraProfile = /^\/camera-profile\/([^/]*)$/.exec(url.pathname); | |
| 839 | if (cameraProfile) { | |
| 840 | let unit: number; | |
| 841 | try { | |
| 842 | unit = parseUnit(cameraProfile[1]); | |
| 843 | } catch (err) { | |
| 844 | throw new HttpError(400, (err as Error).message); | |
| 845 | } | |
| 846 | if (req.method === "GET") { | |
| 847 | const saved = loadProfile(profileFile(dataDir, unit)); | |
| 848 | return json(res, 200, { unit, saved: saved !== null, profile: saved ?? defaultProfile(unit) }); | |
| 849 | } | |
| 850 | if (req.method !== "PUT") throw new HttpError(405, "GET or PUT /camera-profile/<n>"); | |
| 851 | let profile: CameraProfile; | |
| 852 | try { | |
| 853 | profile = parseProfile(await readBody(req)); | |
| 854 | } catch (err) { | |
| 855 | if (err instanceof ProfileError) throw new HttpError(400, err.message); | |
| 856 | throw err; | |
| 857 | } | |
| 858 | if (!options.cameraControl) return json(res, 501, { error: "no camera control here: this is the simulation, not the Pi", saved: false, set: [], failed: [] }); | |
| 859 | let applied: Applied; | |
| 860 | try { | |
| 861 | applied = await options.cameraControl(unit, profile); | |
| 862 | } catch (err) { | |
| 863 | if (err instanceof ProfileError) return json(res, 409, { error: err.message, saved: false, set: [], failed: [] }); | |
| 864 | throw err; | |
| 865 | } | |
| 866 | const keep = url.searchParams.get("try") !== "1"; | |
| 867 | if (keep) saveProfile(profileFile(dataDir, unit), profile); | |
| 868 | return json(res, 200, { saved: keep, ...applied }); | |
| 869 | } | |
| 830 | 870 | if (req.method === "PUT" && url.pathname.startsWith("/calibration/")) { |
| 831 | 871 | const kind = url.pathname.slice("/calibration/".length); |
| 832 | 872 | if (!/^[a-z][a-z0-9_-]{0,31}$/.test(kind)) throw new HttpError(400, "calibration kind: lowercase letters, digits, - and _"); |
Software/so101/src/ui/still-frame.js added +201
| @@ -0,0 +1,201 @@ | ||
| 1 | /** | |
| 2 | * What a still of the grow box needs before anyone looks at it: cropped to the region the owner drew (the box, not the | |
| 3 | * room), colour-corrected for a blue/violet grow light, and checked for whether the plants can be seen at all. | |
| 4 | * Plain ESM on RGBA pixels (a canvas's ImageData), served at /ui/ for the cameras page and imported by the tests and the relay. | |
| 5 | * | |
| 6 | * The region is kept in fractions of the frame, so one drawn on the 640×360 stream crops a 1920×1080 still the same way. | |
| 7 | */ | |
| 8 | ||
| 9 | /** @typedef {{ x: number, y: number, w: number, h: number }} Rect */ | |
| 10 | /** @typedef {{ r: number, g: number, b: number }} Gains */ | |
| 11 | ||
| 12 | /** The hint the grow card and the Tune panel show when the region is mostly blown out or black. */ | |
| 13 | export const CANT_SEE = "Camera can't see the plants: move it inside or beside the box, below the light."; | |
| 14 | ||
| 15 | /** A corrected channel is never pushed more than this: past it a blue-only light gives noise, not colour. */ | |
| 16 | export const MAX_GAIN = 4; | |
| 17 | export const MIN_GAIN = 0.25; | |
| 18 | ||
| 19 | /** Luma at or above this is blown out, at or below DARK is black. */ | |
| 20 | export const BLOWN = 245; | |
| 21 | export const DARK = 16; | |
| 22 | ||
| 23 | /** | |
| 24 | * Any JSON → a region in fractions of the frame, or null for "the whole frame". Throws an Error naming what is wrong. | |
| 25 | * @param {unknown} input | |
| 26 | * @returns {Rect | null} | |
| 27 | */ | |
| 28 | export function parseRegion(input) { | |
| 29 | if (input === null || input === undefined) return null; | |
| 30 | if (typeof input !== "object" || Array.isArray(input)) throw new Error("a region is {x, y, w, h}, fractions of the frame"); | |
| 31 | const o = /** @type {Record<string, unknown>} */ (input); | |
| 32 | const keys = Object.keys(o).sort().join(","); | |
| 33 | if (keys !== "h,w,x,y") throw new Error("a region is {x, y, w, h}, fractions of the frame"); | |
| 34 | const num = (/** @type {unknown} */ v) => (typeof v === "number" && Number.isFinite(v) ? v : NaN); | |
| 35 | const [x, y, w, h] = [num(o.x), num(o.y), num(o.w), num(o.h)]; | |
| 36 | if (![x, y, w, h].every((v) => v >= 0 && v <= 1)) throw new Error("a region's x, y, w and h are each from 0 to 1"); | |
| 37 | if (w < 0.02 || h < 0.02 || x + w > 1.0001 || y + h > 1.0001) throw new Error("a region lies inside the frame and is at least 2% wide and tall"); | |
| 38 | const r4 = (/** @type {number} */ v) => Math.round(v * 10_000) / 10_000; | |
| 39 | return { x: r4(x), y: r4(y), w: r4(Math.min(w, 1 - x)), h: r4(Math.min(h, 1 - y)) }; | |
| 40 | } | |
| 41 | ||
| 42 | /** | |
| 43 | * Two corners dragged on the picture, in fractions, → a region (either drag direction). | |
| 44 | * @param {{ x: number, y: number }} a | |
| 45 | * @param {{ x: number, y: number }} b | |
| 46 | * @returns {Rect} | |
| 47 | */ | |
| 48 | export function regionFromDrag(a, b) { | |
| 49 | const clamp = (/** @type {number} */ v) => Math.min(1, Math.max(0, v)); | |
| 50 | const [x0, x1] = [Math.min(clamp(a.x), clamp(b.x)), Math.max(clamp(a.x), clamp(b.x))]; | |
| 51 | const [y0, y1] = [Math.min(clamp(a.y), clamp(b.y)), Math.max(clamp(a.y), clamp(b.y))]; | |
| 52 | return { x: x0, y: y0, w: x1 - x0, h: y1 - y0 }; | |
| 53 | } | |
| 54 | ||
| 55 | /** | |
| 56 | * A region → the four whole percentages the Tune panel's number inputs show (the whole frame for null). | |
| 57 | * @param {Rect | null} region | |
| 58 | * @returns {{ x: number, y: number, w: number, h: number }} | |
| 59 | */ | |
| 60 | export function regionToPercent(region) { | |
| 61 | const r = region ?? { x: 0, y: 0, w: 1, h: 1 }; | |
| 62 | const pct = (/** @type {number} */ v) => Math.round(v * 100); | |
| 63 | return { x: pct(r.x), y: pct(r.y), w: pct(r.w), h: pct(r.h) }; | |
| 64 | } | |
| 65 | ||
| 66 | /** | |
| 67 | * The number inputs' percentages → a region, kept inside the frame (a box pushed past an edge is cut at it), or null | |
| 68 | * when they are the whole frame. Throws an Error naming what is wrong, like parseRegion. | |
| 69 | * @param {{ x: unknown, y: unknown, w: unknown, h: unknown }} pct | |
| 70 | * @returns {Rect | null} | |
| 71 | */ | |
| 72 | export function regionFromPercent(pct) { | |
| 73 | const [x, y, w, h] = [pct.x, pct.y, pct.w, pct.h].map((v) => (typeof v === "number" ? v : typeof v === "string" && v.trim() ? Number(v) : NaN)); | |
| 74 | if (![x, y, w, h].every((v) => Number.isFinite(v))) throw new Error("the box's left, top, width and height are each a number of percent"); | |
| 75 | const fx = Math.min(1, Math.max(0, /** @type {number} */ (x) / 100)); | |
| 76 | const fy = Math.min(1, Math.max(0, /** @type {number} */ (y) / 100)); | |
| 77 | const fw = Math.min(1 - fx, /** @type {number} */ (w) / 100); | |
| 78 | const fh = Math.min(1 - fy, /** @type {number} */ (h) / 100); | |
| 79 | if (fx === 0 && fy === 0 && fw >= 1 && fh >= 1) return null; | |
| 80 | return parseRegion({ x: fx, y: fy, w: fw, h: fh }); | |
| 81 | } | |
| 82 | ||
| 83 | /** | |
| 84 | * A region (fractions, or null for all of it) → whole pixels of a width×height frame, inside it and at least 1×1. | |
| 85 | * @param {Rect | null} region | |
| 86 | * @param {number} width | |
| 87 | * @param {number} height | |
| 88 | * @returns {Rect} | |
| 89 | */ | |
| 90 | export function cropRect(region, width, height) { | |
| 91 | if (!region) return { x: 0, y: 0, w: width, h: height }; | |
| 92 | const x = Math.min(width - 1, Math.max(0, Math.round(region.x * width))); | |
| 93 | const y = Math.min(height - 1, Math.max(0, Math.round(region.y * height))); | |
| 94 | const right = Math.min(width, Math.max(x + 1, Math.round((region.x + region.w) * width))); | |
| 95 | const bottom = Math.min(height, Math.max(y + 1, Math.round((region.y + region.h) * height))); | |
| 96 | return { x, y, w: right - x, h: bottom - y }; | |
| 97 | } | |
| 98 | ||
| 99 | /** | |
| 100 | * Visit each pixel of `rect` in RGBA `data` of a frame `width` pixels wide. | |
| 101 | * @param {ArrayLike<number>} data | |
| 102 | * @param {number} width | |
| 103 | * @param {Rect} rect | |
| 104 | * @param {(r: number, g: number, b: number) => void} visit | |
| 105 | */ | |
| 106 | function each(data, width, rect, visit) { | |
| 107 | for (let row = rect.y; row < rect.y + rect.h; row++) { | |
| 108 | for (let col = rect.x, i = (row * width + col) * 4; col < rect.x + rect.w; col++, i += 4) visit(data[i] ?? 0, data[i + 1] ?? 0, data[i + 2] ?? 0); | |
| 109 | } | |
| 110 | } | |
| 111 | ||
| 112 | const luma = (/** @type {number} */ r, /** @type {number} */ g, /** @type {number} */ b) => 0.2126 * r + 0.7152 * g + 0.0722 * b; | |
| 113 | ||
| 114 | /** | |
| 115 | * Gray-world: the scene averages to gray, so the gains that make each channel's mean the same are the light's colour | |
| 116 | * undone. Blown-out and black pixels say nothing about the light and are left out. Null when too little is left to judge. | |
| 117 | * @param {ArrayLike<number>} data RGBA | |
| 118 | * @param {number} width | |
| 119 | * @param {Rect} rect | |
| 120 | * @returns {Gains | null} | |
| 121 | */ | |
| 122 | export function grayWorldGains(data, width, rect) { | |
| 123 | let [r, g, b, n] = [0, 0, 0, 0]; | |
| 124 | each(data, width, rect, (pr, pg, pb) => { | |
| 125 | const top = Math.max(pr, pg, pb); | |
| 126 | if (top >= 250 || top <= DARK) return; | |
| 127 | r += pr; | |
| 128 | g += pg; | |
| 129 | b += pb; | |
| 130 | n++; | |
| 131 | }); | |
| 132 | if (n < Math.max(16, rect.w * rect.h * 0.05) || !r || !g || !b) return null; | |
| 133 | const gray = (r + g + b) / 3; | |
| 134 | const gain = (/** @type {number} */ sum) => Math.round(Math.min(MAX_GAIN, Math.max(MIN_GAIN, gray / sum)) * 1000) / 1000; | |
| 135 | return { r: gain(r), g: gain(g), b: gain(b) }; | |
| 136 | } | |
| 137 | ||
| 138 | /** | |
| 139 | * Gains from a white card shot under the grow light: the card's mean, made white. | |
| 140 | * @param {ArrayLike<number>} data RGBA | |
| 141 | * @param {number} width | |
| 142 | * @param {Rect} card the card's pixels | |
| 143 | * @returns {Gains | null} | |
| 144 | */ | |
| 145 | export function whiteCardGains(data, width, card) { | |
| 146 | let [r, g, b, n] = [0, 0, 0, 0]; | |
| 147 | each(data, width, card, (pr, pg, pb) => ((r += pr), (g += pg), (b += pb), n++)); | |
| 148 | if (!n || !r || !g || !b) return null; | |
| 149 | const top = Math.max(r, g, b); | |
| 150 | const gain = (/** @type {number} */ sum) => Math.round(Math.min(MAX_GAIN, Math.max(MIN_GAIN, top / sum)) * 1000) / 1000; | |
| 151 | return { r: gain(r), g: gain(g), b: gain(b) }; | |
| 152 | } | |
| 153 | ||
| 154 | /** | |
| 155 | * Any JSON → gains, or null for "uncorrected". Throws an Error naming what is wrong. | |
| 156 | * @param {unknown} input | |
| 157 | * @returns {Gains | null} | |
| 158 | */ | |
| 159 | export function parseGains(input) { | |
| 160 | if (input === null || input === undefined) return null; | |
| 161 | const o = /** @type {Record<string, unknown>} */ (input); | |
| 162 | if (typeof input !== "object" || Array.isArray(input) || Object.keys(o).sort().join(",") !== "b,g,r") throw new Error("a colour correction is {r, g, b}"); | |
| 163 | const ok = (/** @type {unknown} */ v) => typeof v === "number" && v >= MIN_GAIN && v <= MAX_GAIN; | |
| 164 | if (!ok(o.r) || !ok(o.g) || !ok(o.b)) throw new Error(`each colour gain is from ${MIN_GAIN} to ${MAX_GAIN}`); | |
| 165 | return { r: /** @type {number} */ (o.r), g: /** @type {number} */ (o.g), b: /** @type {number} */ (o.b) }; | |
| 166 | } | |
| 167 | ||
| 168 | /** | |
| 169 | * Multiply each pixel's channels by the gains, in place; alpha is left alone. | |
| 170 | * @param {Uint8ClampedArray | number[]} data RGBA | |
| 171 | * @param {Gains} gains | |
| 172 | */ | |
| 173 | export function applyGains(data, gains) { | |
| 174 | const clamp = (/** @type {number} */ v) => (v > 255 ? 255 : v < 0 ? 0 : Math.round(v)); | |
| 175 | for (let i = 0; i + 3 < data.length; i += 4) { | |
| 176 | data[i] = clamp((data[i] ?? 0) * gains.r); | |
| 177 | data[i + 1] = clamp((data[i + 1] ?? 0) * gains.g); | |
| 178 | data[i + 2] = clamp((data[i + 2] ?? 0) * gains.b); | |
| 179 | } | |
| 180 | } | |
| 181 | ||
| 182 | /** | |
| 183 | * How much of the region is blown out or black, and whether the plants could be seen there. | |
| 184 | * @param {ArrayLike<number>} data RGBA | |
| 185 | * @param {number} width | |
| 186 | * @param {Rect} rect | |
| 187 | * @returns {{ blown: number, dark: number, verdict: "ok" | "blown" | "dark" }} | |
| 188 | */ | |
| 189 | export function visibility(data, width, rect) { | |
| 190 | let [blown, dark, n] = [0, 0, 0]; | |
| 191 | each(data, width, rect, (r, g, b) => { | |
| 192 | const y = luma(r, g, b); | |
| 193 | if (y >= BLOWN) blown++; | |
| 194 | else if (y <= DARK) dark++; | |
| 195 | n++; | |
| 196 | }); | |
| 197 | const [fb, fd] = n ? [blown / n, dark / n] : [0, 1]; | |
| 198 | const round = (/** @type {number} */ v) => Math.round(v * 1000) / 1000; | |
| 199 | const verdict = fb >= 0.5 || (fb + fd >= 0.7 && fb >= fd) ? "blown" : fd >= 0.6 || fb + fd >= 0.7 ? "dark" : "ok"; | |
| 200 | return { blown: round(fb), dark: round(fd), verdict }; | |
| 201 | } |
Software/so101/src/ui/still-frame.test.ts added +121
| @@ -0,0 +1,121 @@ | ||
| 1 | import { describe, expect, test } from "bun:test"; | |
| 2 | import { applyGains, CANT_SEE, cropRect, grayWorldGains, parseGains, parseRegion, regionFromDrag, regionFromPercent, regionToPercent, visibility, whiteCardGains } from "./still-frame.js"; | |
| 3 | ||
| 4 | /** A width×height RGBA frame painted by `at(x, y)`. */ | |
| 5 | function frame(width: number, height: number, at: (x: number, y: number) => [number, number, number]): Uint8ClampedArray { | |
| 6 | const data = new Uint8ClampedArray(width * height * 4); | |
| 7 | for (let y = 0; y < height; y++) for (let x = 0; x < width; x++) data.set([...at(x, y), 255], (y * width + x) * 4); | |
| 8 | return data; | |
| 9 | } | |
| 10 | ||
| 11 | /** | |
| 12 | * The grow box as the side camera saw it on 2026-10-26: a blown-out light panel along the top, the box in the middle | |
| 13 | * (gray tray, green leaves) lit blue/violet, and the dark room around it. 64×36, a tenth of the 640×360 stream. | |
| 14 | */ | |
| 15 | const W = 64; | |
| 16 | const H = 36; | |
| 17 | const BOX = { x: 24, y: 12, w: 20, h: 16 }; | |
| 18 | const growBox = frame(W, H, (x, y) => { | |
| 19 | if (y < 6) return [255, 255, 255]; // the panel | |
| 20 | const inBox = x >= BOX.x && x < BOX.x + BOX.w && y >= BOX.y && y < BOX.y + BOX.h; | |
| 21 | if (!inBox) return [8, 8, 12]; // the room | |
| 22 | const leaf = (x + y) % 3 === 0; | |
| 23 | // what a blue/violet light makes of a gray tray and green leaves: little red, little green, a lot of blue | |
| 24 | return leaf ? [30, 70, 160] : [50, 40, 190]; | |
| 25 | }); | |
| 26 | ||
| 27 | describe("the crop", () => { | |
| 28 | test("a region in fractions crops the stream and a full-resolution still to the same part of the scene", () => { | |
| 29 | const region = { x: 0.375, y: 1 / 3, w: 0.3125, h: 4 / 9 }; | |
| 30 | expect(cropRect(region, 64, 36)).toEqual(BOX); | |
| 31 | expect(cropRect(region, 640, 360)).toEqual({ x: 240, y: 120, w: 200, h: 160 }); | |
| 32 | expect(cropRect(region, 1920, 1080)).toEqual({ x: 720, y: 360, w: 600, h: 480 }); | |
| 33 | expect(cropRect(null, 1920, 1080)).toEqual({ x: 0, y: 0, w: 1920, h: 1080 }); | |
| 34 | }); | |
| 35 | ||
| 36 | test("a crop never leaves the frame and is never empty", () => { | |
| 37 | expect(cropRect({ x: 0.99, y: 0.99, w: 0.01, h: 0.01 }, 10, 10)).toEqual({ x: 9, y: 9, w: 1, h: 1 }); | |
| 38 | expect(cropRect({ x: 0.5, y: 0, w: 0.6, h: 1 }, 100, 50)).toEqual({ x: 50, y: 0, w: 50, h: 50 }); | |
| 39 | }); | |
| 40 | ||
| 41 | test("a drag in either direction is the same region", () => { | |
| 42 | expect(regionFromDrag({ x: 0.7, y: 0.8 }, { x: 0.2, y: 0.1 })).toEqual(regionFromDrag({ x: 0.2, y: 0.1 }, { x: 0.7, y: 0.8 })); | |
| 43 | expect(regionFromDrag({ x: -1, y: 0.5 }, { x: 2, y: 0.5 })).toEqual({ x: 0, y: 0.5, w: 1, h: 0 }); | |
| 44 | }); | |
| 45 | ||
| 46 | test("the keyboard's four percentages and a drawn region are the same box", () => { | |
| 47 | expect(regionFromPercent({ x: "30", y: "25", w: "40", h: "50" })).toEqual({ x: 0.3, y: 0.25, w: 0.4, h: 0.5 }); | |
| 48 | expect(regionToPercent({ x: 0.3, y: 0.25, w: 0.4, h: 0.5 })).toEqual({ x: 30, y: 25, w: 40, h: 50 }); | |
| 49 | expect(regionToPercent(null)).toEqual({ x: 0, y: 0, w: 100, h: 100 }); | |
| 50 | expect(regionFromPercent({ x: 0, y: 0, w: 100, h: 100 })).toBeNull(); | |
| 51 | expect(regionFromPercent({ x: 80, y: 0, w: 50, h: 100 })).toEqual({ x: 0.8, y: 0, w: 0.2, h: 1 }); // cut at the edge | |
| 52 | for (const bad of [{ x: "", y: 0, w: 10, h: 10 }, { x: "a", y: 0, w: 10, h: 10 }, { x: 0, y: 0, w: 1, h: 10 }, { x: 100, y: 0, w: 10, h: 10 }]) expect(() => regionFromPercent(bad)).toThrow(); | |
| 53 | }); | |
| 54 | ||
| 55 | test("a region from outside is checked: inside the frame, at least 2% each way, nothing else in it", () => { | |
| 56 | expect(parseRegion({ x: 0.25, y: 0.25, w: 0.5, h: 0.5 })).toEqual({ x: 0.25, y: 0.25, w: 0.5, h: 0.5 }); | |
| 57 | expect(parseRegion(null)).toBeNull(); | |
| 58 | for (const bad of [[], "x", { x: 0, y: 0, w: 1 }, { x: 0, y: 0, w: 1, h: 1, z: 1 }, { x: -0.1, y: 0, w: 0.5, h: 0.5 }, { x: 0.6, y: 0, w: 0.6, h: 0.5 }, { x: 0, y: 0, w: 0.01, h: 0.5 }, { x: "0", y: 0, w: 1, h: 1 }]) { | |
| 59 | expect(() => parseRegion(bad)).toThrow(); | |
| 60 | } | |
| 61 | }); | |
| 62 | }); | |
| 63 | ||
| 64 | describe("colour under the grow light", () => { | |
| 65 | test("gray-world on the box takes the blue cast out: the corrected tray is near gray and the leaves turn greener than the tray", () => { | |
| 66 | const gains = grayWorldGains(growBox, W, BOX)!; | |
| 67 | expect(gains.b).toBeLessThan(1); | |
| 68 | expect(gains.r).toBeGreaterThan(1); | |
| 69 | const tray = [50, 40, 190]; | |
| 70 | const leaf = [30, 70, 160]; | |
| 71 | const corrected = [...tray, 255, ...leaf, 255]; | |
| 72 | applyGains(corrected, gains); | |
| 73 | const [tr, tg, tb] = corrected; | |
| 74 | expect(Math.max(tr!, tg!, tb!) - Math.min(tr!, tg!, tb!)).toBeLessThan(Math.max(...tray) - Math.min(...tray)); // closer to gray | |
| 75 | expect(corrected[5]! / corrected[6]!).toBeGreaterThan(tg! / tb!); // the leaf's green stands out from the tray | |
| 76 | expect(corrected[3]).toBe(255); // alpha untouched | |
| 77 | }); | |
| 78 | ||
| 79 | test("the panel and the room do not steer the correction; with nothing left to judge there is none", () => { | |
| 80 | expect(grayWorldGains(growBox, W, { x: 0, y: 0, w: W, h: H })).toEqual(grayWorldGains(growBox, W, BOX)); | |
| 81 | expect(grayWorldGains(growBox, W, { x: 0, y: 0, w: W, h: 6 })).toBeNull(); | |
| 82 | }); | |
| 83 | ||
| 84 | test("no gain passes the cap, however little red a violet light leaves", () => { | |
| 85 | const violet = frame(8, 8, () => [1, 20, 240]); | |
| 86 | const gains = grayWorldGains(violet, 8, { x: 0, y: 0, w: 8, h: 8 })!; | |
| 87 | expect(gains.r).toBe(4); | |
| 88 | expect(gains.b).toBeGreaterThanOrEqual(0.25); | |
| 89 | }); | |
| 90 | ||
| 91 | test("a white card under the light becomes white", () => { | |
| 92 | const card = frame(4, 4, () => [80, 100, 200]); | |
| 93 | const gains = whiteCardGains(card, 4, { x: 0, y: 0, w: 4, h: 4 })!; | |
| 94 | expect(gains).toEqual({ r: 2.5, g: 2, b: 1 }); | |
| 95 | applyGains(card, gains); | |
| 96 | expect([...card.slice(0, 3)]).toEqual([200, 200, 200]); | |
| 97 | }); | |
| 98 | ||
| 99 | test("gains from outside are checked", () => { | |
| 100 | expect(parseGains({ r: 1.2, g: 1, b: 0.5 })).toEqual({ r: 1.2, g: 1, b: 0.5 }); | |
| 101 | expect(parseGains(undefined)).toBeNull(); | |
| 102 | for (const bad of [{ r: 5, g: 1, b: 1 }, { r: 1, g: 1 }, { r: 1, g: 1, b: 1, a: 1 }, [1, 1, 1], { r: "1", g: 1, b: 1 }]) expect(() => parseGains(bad)).toThrow(); | |
| 103 | }); | |
| 104 | }); | |
| 105 | ||
| 106 | describe("can the camera see the plants", () => { | |
| 107 | test("the overhead camera's view, the top of the light panel, is flagged as blown out", () => { | |
| 108 | const overhead = frame(W, H, (x, y) => (Math.hypot(x - 32, y - 18) < 30 ? [255, 250, 255] : [120, 90, 200])); | |
| 109 | expect(visibility(overhead, W, { x: 0, y: 0, w: W, h: H }).verdict).toBe("blown"); | |
| 110 | }); | |
| 111 | ||
| 112 | test("the side camera's region on the box is fine; the whole frame of the dark room is not", () => { | |
| 113 | expect(visibility(growBox, W, BOX)).toEqual({ blown: 0, dark: 0, verdict: "ok" }); | |
| 114 | const night = frame(W, H, () => [6, 6, 10]); | |
| 115 | expect(visibility(night, W, cropRect(null, W, H)).verdict).toBe("dark"); | |
| 116 | }); | |
| 117 | ||
| 118 | test("the hint says what to do", () => { | |
| 119 | expect(CANT_SEE).toBe("Camera can't see the plants: move it inside or beside the box, below the light."); | |
| 120 | }); | |
| 121 | }); |
new above
offline