Public: anyone can read this; the crew can change it.
Migrated from github.com/Trevato/SO-ARM100
workbenchthis machine
The Pi runs what main ships: each deploy updates the arm server on the Pi (swap, restart while idle, roll back if unhealthy) #471
Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.
smith/arm:smith/467-the-pi-runs-what-main-ships-each-deploy → main 7 commits fork 1 thread · 1 unresolved
13 files
changed +543
−16
Software/so101/README.md added +26
| @@ -0,0 +1,26 @@ | ||
| 1 | # so101: the arm server and the hosted app | |
| 2 | ||
| 3 | `bench/fake-serve.ts` is the hosted app (the page, the room, the relay); `so101 serve` is the arm server on the Pi. | |
| 4 | ||
| 5 | ## Keeping the Pi at the deployed commit | |
| 6 | ||
| 7 | After each production deploy the app asks the Pi's `GET /version`. If it differs from the app's commit, and nobody holds a turn, the app: | |
| 8 | ||
| 9 | 1. copies `Software/so101` over ssh into `<dir>.new` on the Pi (the live `node_modules` is copied across; the Pi installs nothing); | |
| 10 | 2. moves `<dir>` to `<dir>.prev` and `<dir>.new` to `<dir>` (if the second move fails, the first is undone); | |
| 11 | 3. runs the restart command and polls the Pi's `/status` for 10 s. | |
| 12 | ||
| 13 | If `/status` is not live in time, `<dir>` goes to `<dir>.failed`, `<dir>.prev` comes back, and the server is restarted on it. The sync status on `/admin` and in the terminal's `/versions` then says `rolled-back`; that commit is not tried again until the app's commit changes. If the old tree does not come up either, it says `rollback failed: the old tree did not come up either`: someone must go to the Pi. One previous tree is kept (`<dir>.prev`), replaced at the next deploy. | |
| 14 | ||
| 15 | It is off in previews, without `ARM_PI_SSH_KEY`, with no arm server, and when the app does not know its own commit. | |
| 16 | ||
| 17 | | Variable | Meaning | | |
| 18 | | --- | --- | | |
| 19 | | `ARM_PI_SSH_KEY` | Required. The private key (the secret's text) for the Pi's `crew` user; written to `FORGE_APP_DATA/pi-key`, mode 0600. | | |
| 20 | | `ARM_PI_DIR` | The live tree on the Pi. Default `/home/dietpi/so101`. | | |
| 21 | | `ARM_PI_USER` | The ssh login. Default `crew`. | | |
| 22 | | `ARM_PI_RESTART` | The restart command, run through `sh -c` on the Pi. Default `sudo systemctl restart so101`; **that default is an assumption, not confirmed against the Pi**: set it to what actually restarts the server there. | | |
| 23 | ||
| 24 | On the Pi: a `crew` user whose `authorized_keys` holds the key's public half, with write access to `<dir>`'s parent, and a sudoers entry for the restart command with no password, e.g. `crew ALL=(root) NOPASSWD: /bin/systemctl restart so101`. The host key is accepted on first use and kept in `FORGE_APP_DATA/pi-known-hosts`. | |
| 25 | ||
| 26 | Who sees what: a visitor's page says only "Pi update pending" while the commits differ. The owner and moderators see both commits and the last sync outcome on `/admin` (Versions) and with `/versions` in the terminal. |
Software/so101/bench/fake-serve.ts +3 −1
| @@ -23,6 +23,7 @@ import { LeaderCalibrations } from "../src/leader-calibrations.ts"; | ||
| 23 | 23 | import { Lobby } from "../src/lobby.ts"; |
| 24 | 24 | import { Moderator, modelFromEnv } from "../src/moderator.ts"; |
| 25 | 25 | import { parseRemote, parseVideo, startRelay, type Remote } from "../src/relay.ts"; |
| 26 | import { piSyncFromEnv } from "../src/pi-deploy.ts"; | |
| 26 | 27 | import { startServer } from "../src/server.ts"; |
| 27 | 28 | |
| 28 | 29 | const fake = new FakeBus([1, 2, 3, 4, 5, 6]); |
| @@ -68,6 +69,7 @@ try { | ||
| 68 | 69 | } catch (err) { |
| 69 | 70 | console.error(`${(err as Error).message}; cameras stay MJPEG`); |
| 70 | 71 | } |
| 71 | const relay = await startRelay({ host, port, simulation: `http://127.0.0.1:${simulation.port}`, remote, problem, lobby, cameras, leaders, video, summaryEveryMs: 60_000, holdFile: join(dir, "hold.json"), baselineFile: join(dir, "servo-factory.json") }); | |
| 72 | const piSync = piSyncFromEnv(process.env, remote?.http ?? null, remote?.host ?? null, () => lobby.turns.snapshot().driver === null, dir); | |
| 73 | const relay = await startRelay({ piSync: piSync ?? undefined, host, port, simulation: `http://127.0.0.1:${simulation.port}`, remote, problem, lobby, cameras, leaders, video, summaryEveryMs: 60_000, holdFile: join(dir, "hold.json"), baselineFile: join(dir, "servo-factory.json") }); | |
| 72 | 74 | const { mode, host: armHost } = relay.link(); |
| 73 | 75 | console.log(`arm on http://${host}:${relay.port} (${mode === "simulated" ? "simulated" : `arm ${armHost}: ${mode}`}; data in ${dir}${local ? "; local: every visitor drives as the owner" : ""})`); |
Software/so101/src/admin-page.ts +11
| @@ -11,6 +11,7 @@ import type { Report } from "./reports.ts"; | ||
| 11 | 11 | import type { Person, Who } from "./identity.ts"; |
| 12 | 12 | import type { Snapshot } from "./turns.ts"; |
| 13 | 13 | import type { ViewerRow } from "./viewer-reports.ts"; |
| 14 | import type { Versions } from "./pi-deploy.ts"; | |
| 14 | 15 | import { precisionSection, type PrecisionPanel } from "./precision-panel.ts"; |
| 15 | 16 | import { servosSection, type ServoPanel } from "./servo-panel.ts"; |
| 16 | 17 | import { videoSection, type VideoCard } from "./video-panel.ts"; |
| @@ -157,6 +158,8 @@ export interface AdminPage { | ||
| 157 | 158 | visitors?: { peak: number; hours: readonly number[] }; |
| 158 | 159 | /** The arm's precision scorecard (src/precision-panel.ts); null when the arm did not answer, absent when not asked. */ |
| 159 | 160 | precision?: PrecisionPanel | null; |
| 161 | /** The app's and the Pi's commits; a difference is a warning. */ | |
| 162 | versions?: Versions; | |
| 160 | 163 | /** Owner only: the servos' control registers (src/servo-panel.ts); null when the arm did not answer, absent when not asked. */ |
| 161 | 164 | servos?: ServoPanel | null; |
| 162 | 165 | /** The cameras' H.264 numbers, beside it (src/video-panel.ts); absent when not asked. */ |
| @@ -239,6 +242,13 @@ const visitorsSection = (v: { peak: number; hours: readonly number[] }): string | ||
| 239 | 242 | return `<section aria-labelledby="visitors"><h2 id="visitors">Visitors</h2><p>Peak viewers today: <b>${v.peak}</b></p><ol class="bars" aria-label="Joins per hour, last 24 hours, oldest first">${bars}</ol></section>`; |
| 240 | 243 | }; |
| 241 | 244 | |
| 245 | /** The two commits side by side; a mismatch says so, with what the last attempt to fix it came to. */ | |
| 246 | function versionsSection(v: Versions): string { | |
| 247 | const same = v.pi === v.app; | |
| 248 | const warn = same ? "" : `<p class="flash bad" role="alert">The Pi runs ${v.pi === null ? "something unknown (it did not answer)" : esc(v.pi)}, not ${esc(v.app)}. Last attempt: ${esc(v.sync)}.</p>`; | |
| 249 | return `<section aria-labelledby="versions"><h2 id="versions">Versions</h2><p>App <code>${esc(v.app)}</code> · Pi <code>${esc(v.pi ?? "?")}</code></p>${warn}</section>`; | |
| 250 | } | |
| 251 | ||
| 242 | 252 | export function renderAdmin(p: AdminPage): string { |
| 243 | 253 | const s = p.settings; |
| 244 | 254 | const d = p.snapshot.driver; |
| @@ -284,6 +294,7 @@ ${p.canEdit ? `<section aria-labelledby="table"><h2 id="table">Table height</h2> | ||
| 284 | 294 | <div class="save"><button type="button" class="go" id="table-save">Save</button> <button type="button" id="table-here">The arm rests on the table now</button> <span id="table-note" role="status"></span></div> |
| 285 | 295 | </section>` : ""} |
| 286 | 296 | ${p.canEdit && p.visitors ? visitorsSection(p.visitors) : ""} |
| 297 | ${p.versions ? versionsSection(p.versions) : ""} | |
| 287 | 298 | ${p.precision !== undefined ? precisionSection(p.precision, esc) : ""} |
| 288 | 299 | ${p.canEdit && p.servos !== undefined ? servosSection(p.servos, esc) : ""} |
| 289 | 300 | ${p.video !== undefined ? videoSection(p.video, esc) : ""} |
Software/so101/src/admin-routes.ts +5 −4
| @@ -5,6 +5,7 @@ import type { Lobby } from "./lobby.ts"; | ||
| 5 | 5 | import { AdminError, changeFromForm } from "./admin.ts"; |
| 6 | 6 | import { sameOrigin } from "./relay.ts"; |
| 7 | 7 | import { adminCsp, DONE, renderAdmin, renderRefusal, type Done } from "./admin-page.ts"; |
| 8 | import type { Versions } from "./pi-deploy.ts"; | |
| 8 | 9 | import type { PrecisionPanel } from "./precision-panel.ts"; |
| 9 | 10 | import type { ServoPanel } from "./servo-panel.ts"; |
| 10 | 11 | import type { VideoCard } from "./video-panel.ts"; |
| @@ -31,7 +32,7 @@ export interface AdminSession { | ||
| 31 | 32 | * Answer 404 to anyone not allowed, and the page itself to GET and HEAD. Returns the session for any other |
| 32 | 33 | * method by an allowed caller, and null when the response has been sent. |
| 33 | 34 | */ |
| 34 | export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo): Promise<AdminSession | null> { | |
| 35 | export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo, versions?: () => Versions): Promise<AdminSession | null> { | |
| 35 | 36 | const who = lobby.who(req); |
| 36 | 37 | const html = (status: number, text: string, edit = false) => { |
| 37 | 38 | res.writeHead(status, { "content-type": "text/html; charset=utf-8", "cache-control": "no-store", "content-security-policy": adminCsp(edit) }); |
| @@ -58,7 +59,7 @@ export async function serveAdminPage(lobby: Lobby, req: IncomingMessage, res: Se | ||
| 58 | 59 | const servos = canEdit && (req.method === "GET" || req.method === "HEAD") && askServos ? await askServos() : undefined; |
| 59 | 60 | const video = askVideo?.(); |
| 60 | 61 | const show = (status: number, done: Done | null, error: string | null) => |
| 61 | html(status, renderAdmin({ precision, servos, video, who, canEdit, settings: lobby.admin.settings, snapshot: lobby.turns.snapshot(), log: lobby.admin.log, removed: lobby.chat.removed, muted: lobby.chat.mutedList(), banned: lobby.accounts.bans(), health: lobby.reports.rows(), reports: lobby.lineReports.list(), visitors: canEdit ? { peak: lobby.visitors.peakToday(), hours: lobby.visitors.joinsPerHour() } : undefined, done, error }), canEdit); | |
| 62 | html(status, renderAdmin({ precision, servos, video, versions: versions?.(), who, canEdit, settings: lobby.admin.settings, snapshot: lobby.turns.snapshot(), log: lobby.admin.log, removed: lobby.chat.removed, muted: lobby.chat.mutedList(), banned: lobby.accounts.bans(), health: lobby.reports.rows(), reports: lobby.lineReports.list(), visitors: canEdit ? { peak: lobby.visitors.peakToday(), hours: lobby.visitors.joinsPerHour() } : undefined, done, error }), canEdit); | |
| 62 | 63 | if (req.method === "GET" || req.method === "HEAD") { |
| 63 | 64 | const done = url.searchParams.get("done"); |
| 64 | 65 | show(200, done && Object.hasOwn(DONE, done) ? (done as Done) : null, null); |
| @@ -77,8 +78,8 @@ const REPORT_ACTIONS = new Set(["report-remove", "report-mute", "report-dismiss" | ||
| 77 | 78 | type ReadBody = (req: IncomingMessage, limit?: number) => Promise<Buffer>; |
| 78 | 79 | |
| 79 | 80 | /** The owner's page. Anyone else is told so; a change is a form post, answered with a redirect back. */ |
| 80 | export async function handleAdmin(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, readBody: ReadBody, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo): Promise<void> { | |
| 81 | const session = await serveAdminPage(lobby, req, res, url, askPrecision, askServos, askVideo); | |
| 81 | export async function handleAdmin(lobby: Lobby, req: IncomingMessage, res: ServerResponse, url: URL, readBody: ReadBody, askPrecision?: AskPrecision, askServos?: AskServos, askVideo?: AskVideo, versions?: () => Versions): Promise<void> { | |
| 82 | const session = await serveAdminPage(lobby, req, res, url, askPrecision, askServos, askVideo, versions); | |
| 82 | 83 | if (!session) return; |
| 83 | 84 | const { who, canEdit, show } = session; |
| 84 | 85 | if (req.method !== "POST") { |
Software/so101/src/pi-deploy.test.ts added +142
| @@ -0,0 +1,142 @@ | ||
| 1 | import { afterEach, expect, test } from "bun:test"; | |
| 2 | import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; | |
| 3 | import { tmpdir } from "node:os"; | |
| 4 | import { join } from "node:path"; | |
| 5 | import { askPiCommit, deployToPi, packTree, PiSync, piSyncFromEnv, type Target } from "./pi-deploy.ts"; | |
| 6 | ||
| 7 | const roots: string[] = []; | |
| 8 | afterEach(() => roots.splice(0).forEach((r) => rmSync(r, { recursive: true, force: true }))); | |
| 9 | ||
| 10 | /** A fake Pi: the live tree holds a COMMIT and a node_modules; scripts run in a local shell. */ | |
| 11 | function fake() { | |
| 12 | const root = mkdtempSync(join(tmpdir(), "pi-")); | |
| 13 | roots.push(root); | |
| 14 | const live = join(root, "so101"); | |
| 15 | mkdirSync(join(live, "node_modules"), { recursive: true }); | |
| 16 | writeFileSync(join(live, "COMMIT"), "old000000000\n"); | |
| 17 | writeFileSync(join(live, "node_modules", "dep"), "x"); | |
| 18 | const src = join(root, "src-tree"); | |
| 19 | mkdirSync(src); | |
| 20 | writeFileSync(join(src, "server.ts"), "new code"); | |
| 21 | const target: Target = { | |
| 22 | async exec(script, stdin) { | |
| 23 | const p = Bun.spawn(["sh", "-c", script], { stdin: stdin ?? "ignore", stdout: "pipe", stderr: "pipe" }); | |
| 24 | const [out, err, code] = await Promise.all([new Response(p.stdout).text(), new Response(p.stderr).text(), p.exited]); | |
| 25 | return { code, out: out + err }; | |
| 26 | }, | |
| 27 | }; | |
| 28 | const restarts = join(root, "restarts"); | |
| 29 | const options = (over: object = {}) => ({ commit: "abc123456789", dir: live, restart: `echo r >> ${restarts}`, target, pack: () => packTree(src), idle: () => true, healthy: async () => true, patienceMs: 300, pollMs: 20, ...over }); | |
| 30 | /** Live only once the old tree is back: the new one never comes up. */ | |
| 31 | const oldOnly = async () => readFileSync(join(live, "COMMIT"), "utf8").trim() === "old000000000"; | |
| 32 | return { live, restarts, options, oldOnly }; | |
| 33 | } | |
| 34 | ||
| 35 | test("a new commit is copied, swapped in with node_modules kept, and the previous tree stays", async () => { | |
| 36 | const f = fake(); | |
| 37 | expect(await deployToPi(f.options())).toBe("deployed"); | |
| 38 | expect(readFileSync(join(f.live, "COMMIT"), "utf8").trim()).toBe("abc123456789"); | |
| 39 | expect(readFileSync(join(f.live, "server.ts"), "utf8")).toBe("new code"); | |
| 40 | expect(existsSync(join(f.live, "node_modules", "dep"))).toBe(true); | |
| 41 | expect(readFileSync(join(`${f.live}.prev`, "COMMIT"), "utf8").trim()).toBe("old000000000"); | |
| 42 | expect(readFileSync(f.restarts, "utf8")).toBe("r\n"); | |
| 43 | }); | |
| 44 | ||
| 45 | test("nothing is touched while someone holds a turn, nor when a turn starts during the copy", async () => { | |
| 46 | const f = fake(); | |
| 47 | expect(await deployToPi(f.options({ idle: () => false }))).toBe("waiting"); | |
| 48 | let calls = 0; | |
| 49 | expect(await deployToPi(f.options({ idle: () => ++calls === 1 }))).toBe("waiting"); | |
| 50 | expect(readFileSync(join(f.live, "COMMIT"), "utf8").trim()).toBe("old000000000"); | |
| 51 | expect(existsSync(`${f.live}.new`)).toBe(false); | |
| 52 | expect(existsSync(f.restarts)).toBe(false); | |
| 53 | }); | |
| 54 | ||
| 55 | test("a server that is not live in time is rolled back and restarted on the old tree", async () => { | |
| 56 | const f = fake(); | |
| 57 | expect(await deployToPi(f.options({ healthy: f.oldOnly }))).toBe("rolled-back"); | |
| 58 | expect(readFileSync(join(f.live, "COMMIT"), "utf8").trim()).toBe("old000000000"); | |
| 59 | expect(existsSync(join(`${f.live}.failed`, "server.ts"))).toBe(true); | |
| 60 | expect(readFileSync(f.restarts, "utf8")).toBe("r\nr\n"); | |
| 61 | }); | |
| 62 | ||
| 63 | test("rejects a commit or path that is not plain", async () => { | |
| 64 | const f = fake(); | |
| 65 | await expect(deployToPi(f.options({ commit: "x; rm -rf /" }))).rejects.toThrow(); | |
| 66 | await expect(deployToPi(f.options({ dir: "/a b" }))).rejects.toThrow(); | |
| 67 | }); | |
| 68 | ||
| 69 | test("PiSync deploys on a mismatch, leaves a match alone, and does not retry a rolled-back commit", async () => { | |
| 70 | let pi: string | null = "old"; | |
| 71 | let outcome: "deployed" | "rolled-back" = "rolled-back"; | |
| 72 | const asked: string[] = []; | |
| 73 | const sync = new PiSync({ app: "new", pi: async () => pi, deploy: async (c) => (asked.push(c), outcome) }); | |
| 74 | await sync.check(); | |
| 75 | expect(sync.versions()).toEqual({ app: "new", pi: "old", sync: "rolled-back" }); | |
| 76 | await sync.check(); | |
| 77 | expect(asked).toEqual(["new"]); | |
| 78 | pi = null; | |
| 79 | await sync.check(); | |
| 80 | expect(sync.versions().sync).toBe("the Pi did not answer"); | |
| 81 | pi = "new"; | |
| 82 | await sync.check(); | |
| 83 | expect(sync.versions().sync).toBe("in step"); | |
| 84 | const fresh = new PiSync({ app: "n2", pi: async () => pi, deploy: async () => { pi = "n2"; return "deployed"; } }); | |
| 85 | await fresh.check(); | |
| 86 | expect(fresh.versions()).toEqual({ app: "n2", pi: "n2", sync: "deployed" }); | |
| 87 | }); | |
| 88 | ||
| 89 | test("if the second move fails the live tree is put back and nothing is restarted", async () => { | |
| 90 | const f = fake(); | |
| 91 | const o = f.options(); | |
| 92 | const target: Target = { | |
| 93 | exec: async (script, stdin) => { | |
| 94 | if (script.includes(".prev") && script.includes("&&")) rmSync(`${f.live}.new`, { recursive: true }); // the tree to swap in vanishes | |
| 95 | return o.target.exec(script, stdin); | |
| 96 | }, | |
| 97 | }; | |
| 98 | expect(await deployToPi({ ...o, target })).toBe("failed"); | |
| 99 | expect(readFileSync(join(f.live, "COMMIT"), "utf8").trim()).toBe("old000000000"); | |
| 100 | expect(existsSync(join(f.live, "node_modules", "dep"))).toBe(true); | |
| 101 | expect(existsSync(f.restarts)).toBe(false); | |
| 102 | }); | |
| 103 | ||
| 104 | test("a rollback whose old tree does not come up either is reported as such", async () => { | |
| 105 | const f = fake(); | |
| 106 | const sync = new PiSync({ app: "new", pi: async () => "old", deploy: async () => deployToPi(f.options({ healthy: async () => false })) }); | |
| 107 | await sync.check(); | |
| 108 | expect(sync.versions().sync).toBe("rollback failed: the old tree did not come up either"); | |
| 109 | expect(await deployToPi(f.options({ healthy: async () => false }))).toBe("rollback-failed"); | |
| 110 | }); | |
| 111 | ||
| 112 | test("askPiCommit: the commit, 'unknown' from an old server without /version, null when it is not live", async () => { | |
| 113 | const serve = (routes: Record<string, () => Response>) => Bun.serve({ port: 0, fetch: (r) => routes[new URL(r.url).pathname]?.() ?? new Response("no", { status: 404 }) }); | |
| 114 | const up = serve({ "/status": () => Response.json({}), "/version": () => Response.json({ commit: "abc123456789" }) }); | |
| 115 | const old = serve({ "/status": () => Response.json({}) }); | |
| 116 | const down = serve({}); | |
| 117 | const odd = serve({ "/status": () => Response.json({}), "/version": () => Response.json({ commit: "x; rm" }) }); | |
| 118 | try { | |
| 119 | expect(await askPiCommit(`http://127.0.0.1:${up.port}`)).toBe("abc123456789"); | |
| 120 | expect(await askPiCommit(`http://127.0.0.1:${old.port}`)).toBe("unknown"); | |
| 121 | expect(await askPiCommit(`http://127.0.0.1:${odd.port}`)).toBe("unknown"); | |
| 122 | expect(await askPiCommit(`http://127.0.0.1:${down.port}`)).toBeNull(); | |
| 123 | expect(await askPiCommit("http://127.0.0.1:1")).toBeNull(); | |
| 124 | } finally { | |
| 125 | for (const s of [up, old, down, odd]) s.stop(true); | |
| 126 | } | |
| 127 | }); | |
| 128 | ||
| 129 | test("piSyncFromEnv is off without a key, in a preview, or with no Pi; on otherwise, and the key file is owner-only", () => { | |
| 130 | const root = mkdtempSync(join(tmpdir(), "pi-env-")); | |
| 131 | roots.push(root); | |
| 132 | const on = { ARM_PI_SSH_KEY: "key", FORGE_APP_COMMIT: "abc1234567890" }; | |
| 133 | const make = (env: Record<string, string>, http: string | null = "http://pi:8101", host: string | null = "pi") => piSyncFromEnv(env, http, host, () => true, root); | |
| 134 | expect(make({ FORGE_APP_COMMIT: "abc1234567890" })).toBeNull(); | |
| 135 | expect(make({ ...on, FORGE_APP_ENV: "preview" })).toBeNull(); | |
| 136 | expect(make(on, null, null)).toBeNull(); | |
| 137 | writeFileSync(join(root, "pi-key"), "stale", { mode: 0o644 }); | |
| 138 | const sync = make(on); | |
| 139 | expect(sync?.versions()).toEqual({ app: "abc123456789", pi: null, sync: "idle" }); | |
| 140 | expect(statSync(join(root, "pi-key")).mode & 0o777).toBe(0o600); | |
| 141 | expect(readFileSync(join(root, "pi-key"), "utf8")).toBe("key\n"); | |
| 142 | }); |
Software/so101/src/pi-deploy.ts added +182
| @@ -0,0 +1,182 @@ | ||
| 1 | /** | |
| 2 | * The Pi runs what production runs. After a deploy the hosted app asks the Pi's GET /version; when it differs, and nobody | |
| 3 | * holds a turn, the app copies Software/so101 to a new directory beside the live one, swaps the two, restarts the arm | |
| 4 | * server and, if /status is not live within the patience, puts the previous tree back. One previous tree is kept. | |
| 5 | * The transport is a `Target` (ssh to the Pi in production, a local shell over a fake directory in the test). | |
| 6 | */ | |
| 7 | import { chmodSync, writeFileSync } from "node:fs"; | |
| 8 | import { join } from "node:path"; | |
| 9 | import { deployedCommit } from "./precision-history.ts"; | |
| 10 | ||
| 11 | /** Runs a shell script on the Pi, with `stdin` as its input. */ | |
| 12 | export interface Target { | |
| 13 | exec(script: string, stdin?: Uint8Array): Promise<{ code: number; out: string }>; | |
| 14 | } | |
| 15 | ||
| 16 | export interface DeployOptions { | |
| 17 | commit: string; | |
| 18 | /** The live tree on the Pi; the new one is built at `<dir>.new`, the previous kept at `<dir>.prev`. */ | |
| 19 | dir: string; | |
| 20 | restart: string; | |
| 21 | target: Target; | |
| 22 | /** The source tree as a tar stream. */ | |
| 23 | pack: () => Promise<Uint8Array>; | |
| 24 | /** Nobody holds a turn. */ | |
| 25 | idle: () => boolean; | |
| 26 | /** Whether the arm server answers /status. */ | |
| 27 | healthy: () => Promise<boolean>; | |
| 28 | patienceMs?: number; | |
| 29 | pollMs?: number; | |
| 30 | } | |
| 31 | ||
| 32 | /** "rolled-back" put the old tree back and it is live; "rollback-failed" means the Pi may be down and a person must look. */ | |
| 33 | export type Outcome = "deployed" | "waiting" | "rolled-back" | "rollback-failed" | "failed"; | |
| 34 | ||
| 35 | const PATH = /^\/[\w./-]{1,120}$/; | |
| 36 | const COMMIT = /^[0-9a-f]{7,40}$/; | |
| 37 | /** Single-quote for a POSIX shell. */ | |
| 38 | export const quote = (s: string) => `'${s.replaceAll("'", `'\\''`)}'`; | |
| 39 | ||
| 40 | export async function deployToPi(o: DeployOptions): Promise<Outcome> { | |
| 41 | if (!PATH.test(o.dir) || !COMMIT.test(o.commit)) throw new Error("bad deploy target or commit"); | |
| 42 | const dir = quote(o.dir); | |
| 43 | const fresh = quote(`${o.dir}.new`); | |
| 44 | const prev = quote(`${o.dir}.prev`); | |
| 45 | if (!o.idle()) return "waiting"; | |
| 46 | const put = await o.target.exec( | |
| 47 | `set -e; rm -rf ${fresh}; mkdir ${fresh}; tar -x -C ${fresh}; if [ -d ${dir}/node_modules ]; then cp -a ${dir}/node_modules ${fresh}/node_modules; fi; printf '%s\\n' ${o.commit} > ${fresh}/COMMIT`, | |
| 48 | await o.pack(), | |
| 49 | ); | |
| 50 | if (put.code !== 0) return "failed"; | |
| 51 | if (!o.idle()) { | |
| 52 | await o.target.exec(`rm -rf ${fresh}`); | |
| 53 | return "waiting"; | |
| 54 | } | |
| 55 | // if the second mv fails the live tree is moved back, so the Pi is never left without a server directory | |
| 56 | const swap = await o.target.exec(`rm -rf ${prev} && mv ${dir} ${prev} && { mv ${fresh} ${dir} || { mv ${prev} ${dir}; exit 1; }; }`); | |
| 57 | if (swap.code !== 0) return "failed"; | |
| 58 | const up = async () => { | |
| 59 | if ((await o.target.exec(o.restart)).code !== 0) return false; | |
| 60 | return live(); | |
| 61 | }; | |
| 62 | const live = async () => { | |
| 63 | const until = Date.now() + (o.patienceMs ?? 10_000); | |
| 64 | while (Date.now() < until) { | |
| 65 | if (await o.healthy().catch(() => false)) return true; | |
| 66 | await Bun.sleep(o.pollMs ?? 500); | |
| 67 | } | |
| 68 | return false; | |
| 69 | }; | |
| 70 | if (await up()) return "deployed"; | |
| 71 | await o.target.exec(`set -e; rm -rf ${dir}.failed; mv ${dir} ${dir}.failed; mv ${prev} ${dir}`); | |
| 72 | return (await o.target.exec(o.restart)).code === 0 && (await live()) ? "rolled-back" : "rollback-failed"; | |
| 73 | } | |
| 74 | ||
| 75 | /** ssh to the Pi as `user`; the key is a file of the caller's keeping. */ | |
| 76 | export function sshTarget(o: { host: string; user: string; keyFile: string; knownHosts: string }): Target { | |
| 77 | return { | |
| 78 | async exec(script, stdin) { | |
| 79 | const proc = Bun.spawn( | |
| 80 | ["ssh", "-i", o.keyFile, "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", "-o", "StrictHostKeyChecking=accept-new", "-o", `UserKnownHostsFile=${o.knownHosts}`, `${o.user}@${o.host}`, `sh -c ${quote(script)}`], | |
| 81 | { stdin: stdin ?? "ignore", stdout: "pipe", stderr: "pipe" }, | |
| 82 | ); | |
| 83 | const [out, err, code] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text(), proc.exited]); | |
| 84 | return { code, out: out + err }; | |
| 85 | }, | |
| 86 | }; | |
| 87 | } | |
| 88 | ||
| 89 | /** Software/so101 as a tar stream, without node_modules (the Pi keeps its own). */ | |
| 90 | export async function packTree(root: string): Promise<Uint8Array> { | |
| 91 | const proc = Bun.spawn(["tar", "-c", "--exclude=./node_modules", "-C", root, "."], { stdout: "pipe" }); | |
| 92 | const bytes = new Uint8Array(await new Response(proc.stdout).arrayBuffer()); | |
| 93 | if ((await proc.exited) !== 0) throw new Error("tar failed"); | |
| 94 | return bytes; | |
| 95 | } | |
| 96 | ||
| 97 | export interface Versions { | |
| 98 | app: string; | |
| 99 | /** The Pi's commit; null when it did not answer. */ | |
| 100 | pi: string | null; | |
| 101 | /** What the last attempt came to. */ | |
| 102 | sync: string; | |
| 103 | } | |
| 104 | ||
| 105 | export interface SyncOptions { | |
| 106 | /** The commit this app was deployed at. */ | |
| 107 | app: string; | |
| 108 | /** What the Pi's GET /version says; null when unreachable. */ | |
| 109 | pi: () => Promise<string | null>; | |
| 110 | deploy: (commit: string) => Promise<Outcome>; | |
| 111 | } | |
| 112 | ||
| 113 | /** Keeps the Pi at the app's commit: `check()` is called on a timer. A commit that rolled back is not tried again. */ | |
| 114 | export class PiSync { | |
| 115 | private pi: string | null = null; | |
| 116 | private sync = "idle"; | |
| 117 | private rejected = ""; | |
| 118 | private running = false; | |
| 119 | private readonly o: SyncOptions; | |
| 120 | constructor(o: SyncOptions) { | |
| 121 | this.o = o; | |
| 122 | } | |
| 123 | ||
| 124 | /** Whether the Pi is known to run something other than the app; all a visitor is told. */ | |
| 125 | differs(): boolean { | |
| 126 | return this.pi !== null && this.pi !== this.o.app; | |
| 127 | } | |
| 128 | ||
| 129 | versions(): Versions { | |
| 130 | return { app: this.o.app, pi: this.pi, sync: this.sync }; | |
| 131 | } | |
| 132 | ||
| 133 | async check(): Promise<void> { | |
| 134 | if (this.running) return; | |
| 135 | this.running = true; | |
| 136 | try { | |
| 137 | this.pi = await this.o.pi(); | |
| 138 | if (this.pi === null) return void (this.sync = "the Pi did not answer"); | |
| 139 | if (this.pi === this.o.app) return void (this.sync = "in step"); | |
| 140 | if (this.rejected === this.o.app) return void (this.sync = `${this.o.app} rolled back; not retried`); | |
| 141 | const outcome = await this.o.deploy(this.o.app); | |
| 142 | if (outcome === "rolled-back" || outcome === "rollback-failed") this.rejected = this.o.app; | |
| 143 | this.sync = outcome === "waiting" ? "waiting for the turn to end" : outcome === "rollback-failed" ? "rollback failed: the old tree did not come up either" : outcome; | |
| 144 | if (outcome === "deployed") this.pi = await this.o.pi(); | |
| 145 | } catch (err) { | |
| 146 | this.sync = `failed: ${(err as Error).message}`; | |
| 147 | } finally { | |
| 148 | this.running = false; | |
| 149 | } | |
| 150 | } | |
| 151 | } | |
| 152 | ||
| 153 | /** The Pi's commit as the arm server reports it; "unknown" from one that has no /version yet. */ | |
| 154 | export async function askPiCommit(http: string): Promise<string | null> { | |
| 155 | try { | |
| 156 | const live = await fetch(`${http}/status`, { signal: AbortSignal.timeout(2500), cache: "no-store" }); | |
| 157 | if (!live.ok) return null; | |
| 158 | const answer = await fetch(`${http}/version`, { signal: AbortSignal.timeout(2500), cache: "no-store" }); | |
| 159 | const commit = answer.ok ? ((await answer.json()) as { commit?: unknown }).commit : null; | |
| 160 | return typeof commit === "string" && COMMIT.test(commit) ? commit : "unknown"; | |
| 161 | } catch { | |
| 162 | return null; | |
| 163 | } | |
| 164 | } | |
| 165 | ||
| 166 | /** The deploy wiring from the environment; null unless this is production with a key and a Pi to reach. */ | |
| 167 | export function piSyncFromEnv(env: Record<string, string | undefined>, http: string | null, host: string | null, idle: () => boolean, dataDir: string): PiSync | null { | |
| 168 | const key = env.ARM_PI_SSH_KEY; | |
| 169 | const app = deployedCommit(env); | |
| 170 | if (!key || !http || !host || app === "unknown" || env.FORGE_APP_ENV === "preview") return null; | |
| 171 | const dir = env.ARM_PI_DIR ?? "/home/dietpi/so101"; | |
| 172 | const keyFile = join(dataDir, "pi-key"); | |
| 173 | writeFileSync(keyFile, key.endsWith("\n") ? key : `${key}\n`, { mode: 0o600 }); | |
| 174 | chmodSync(keyFile, 0o600); // a leftover file keeps its old mode otherwise | |
| 175 | const target = sshTarget({ host, user: env.ARM_PI_USER ?? "crew", keyFile, knownHosts: join(dataDir, "pi-known-hosts") }); | |
| 176 | const root = join(import.meta.dir, ".."); | |
| 177 | return new PiSync({ | |
| 178 | app, | |
| 179 | pi: () => askPiCommit(http), | |
| 180 | deploy: (commit) => deployToPi({ commit, dir, restart: env.ARM_PI_RESTART ?? "sudo systemctl restart so101", target, pack: () => packTree(root), idle, healthy: async () => (await fetch(`${http}/status`, { signal: AbortSignal.timeout(2000) })).ok }), | |
| 181 | }); | |
| 182 | } |
Software/so101/src/pi-versions.test.ts added +95
| @@ -0,0 +1,95 @@ | ||
| 1 | import { afterEach, expect, test } from "bun:test"; | |
| 2 | import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; | |
| 3 | import { tmpdir } from "node:os"; | |
| 4 | import { join } from "node:path"; | |
| 5 | import { renderAdmin } from "./admin-page.ts"; | |
| 6 | import { Admin } from "./admin.ts"; | |
| 7 | import { sign } from "./identity.ts"; | |
| 8 | import { Lobby } from "./lobby.ts"; | |
| 9 | import { deployedCommit } from "./precision-history.ts"; | |
| 10 | import { PiSync } from "./pi-deploy.ts"; | |
| 11 | import { startRelay, type RunningRelay } from "./relay.ts"; | |
| 12 | import { Door, emptyRoom, typed, versionLines, versionsReader } from "./terminal.ts"; | |
| 13 | import { Turns } from "./turns.ts"; | |
| 14 | ||
| 15 | const KEY = "k".repeat(64); | |
| 16 | const later = Date.now() + 600_000; | |
| 17 | const as = (name: string, role: "user" | "admin") => { | |
| 18 | const identity = `id-${name}|${name}|${role}|${later}`; | |
| 19 | return { "x-forge-identity": identity, "x-forge-signature": sign(identity, KEY) }; | |
| 20 | }; | |
| 21 | const dirs: string[] = []; | |
| 22 | const closers: Array<() => unknown> = []; | |
| 23 | afterEach(async () => { | |
| 24 | for (const c of closers.splice(0)) await c(); | |
| 25 | dirs.splice(0).forEach((d) => rmSync(d, { recursive: true, force: true })); | |
| 26 | }); | |
| 27 | ||
| 28 | test("deployedCommit reads the COMMIT file of a copied tree, and ignores a malformed one", () => { | |
| 29 | const tree = mkdtempSync(join(tmpdir(), "tree-")); | |
| 30 | dirs.push(tree); | |
| 31 | writeFileSync(join(tree, "COMMIT"), "abcdef0123456789abcdef\n"); | |
| 32 | expect(deployedCommit({}, "/nowhere", tree)).toBe("abcdef012345"); | |
| 33 | writeFileSync(join(tree, "COMMIT"), "not a commit\n"); | |
| 34 | expect(deployedCommit({}, "/nowhere", tree)).toBe("unknown"); | |
| 35 | }); | |
| 36 | ||
| 37 | const sync = (pi: string | null) => new PiSync({ app: "abc123456789", pi: async () => pi, deploy: async () => "deployed" }); | |
| 38 | ||
| 39 | test("the admin page shows both commits, and a mismatch as a warning, escaped", () => { | |
| 40 | const base = { who: { id: "o", name: "owner", owner: true }, canEdit: true, settings: new Admin().settings, snapshot: new Turns().snapshot(), log: [] }; | |
| 41 | const same = renderAdmin({ ...base, versions: { app: "abc123456789", pi: "abc123456789", sync: "in step" } }); | |
| 42 | expect(same).toContain("Pi <code>abc123456789</code>"); | |
| 43 | expect(same).not.toContain("The Pi runs"); | |
| 44 | const bad = renderAdmin({ ...base, versions: { app: "abc123456789", pi: "<script>x</script>", sync: "failed: <b>" } }); | |
| 45 | expect(bad).toContain("The Pi runs <script>x</script>, not abc123456789"); | |
| 46 | expect(bad).toContain("Last attempt: failed: <b>"); | |
| 47 | expect(bad).not.toContain("<script>x</script>"); | |
| 48 | expect(renderAdmin({ ...base, versions: { app: "a", pi: null, sync: "the Pi did not answer" } })).toContain("it did not answer"); | |
| 49 | expect(renderAdmin(base)).not.toContain('id="versions"'); | |
| 50 | }); | |
| 51 | ||
| 52 | test("GET /admin/versions is for the owner and moderators; a visitor is only told, in /link and the page, that the Pi is behind", async () => { | |
| 53 | const dir = mkdtempSync(join(tmpdir(), "so101-ver-")); | |
| 54 | dirs.push(dir); | |
| 55 | const lobby = new Lobby({ identityKey: KEY, adminFile: join(dir, "admin.json") }); | |
| 56 | closers.push(() => lobby.close()); | |
| 57 | const arm = Bun.serve({ port: 0, fetch: () => new Response("<html><head></head><body></body></html>", { headers: { "content-type": "text/html" } }) }); | |
| 58 | closers.push(() => arm.stop(true)); | |
| 59 | const piSync = sync("old000000000"); | |
| 60 | await piSync.check(); | |
| 61 | const relay: RunningRelay = await startRelay({ port: 0, simulation: `http://127.0.0.1:${arm.port}`, lobby, piSync }); | |
| 62 | closers.push(() => relay.close()); | |
| 63 | const base = `http://127.0.0.1:${relay.port}`; | |
| 64 | expect((await fetch(`${base}/admin/versions`)).status).toBe(404); | |
| 65 | expect((await fetch(`${base}/admin/versions`, { headers: as("eve", "user") })).status).toBe(404); | |
| 66 | const seen = await fetch(`${base}/admin/versions`, { headers: as("trevato", "admin") }); | |
| 67 | expect(seen.status).toBe(200); | |
| 68 | expect(await seen.json()).toEqual({ app: "abc123456789", pi: "old000000000", sync: "deployed" }); | |
| 69 | const home = await (await fetch(`${base}/`)).text(); | |
| 70 | const told = (await (await fetch(`${base}/link`)).json()) as { piBehind?: boolean }; | |
| 71 | expect(told.piBehind).toBe(true); | |
| 72 | expect(JSON.stringify(told)).not.toContain("old000000000"); | |
| 73 | expect(home).toContain('"piBehind":true'); | |
| 74 | expect(home).not.toContain("old000000000"); | |
| 75 | expect(home).not.toContain("abc123456789"); | |
| 76 | }); | |
| 77 | ||
| 78 | test("/versions in the terminal prints the commits and the last sync, or the refusal", async () => { | |
| 79 | expect(typed(emptyRoom(), "/versions")).toEqual({ versions: true }); | |
| 80 | const body = { app: "abc", pi: "de\x1bf", sync: "in step" }; | |
| 81 | expect(versionLines(body)).toEqual(["app abc · Pi def · DIFFER", "last sync: in step"]); | |
| 82 | expect(versionLines({ app: "abc", pi: "abc", sync: "in step" })[0]).toBe("app abc · Pi abc"); | |
| 83 | expect(versionLines(null)).toHaveLength(1); | |
| 84 | const asked: string[] = []; | |
| 85 | const ok = await versionsReader({ PORT: "8102" }, (async (url: string) => (asked.push(url), Response.json(body))) as unknown as typeof fetch)(); | |
| 86 | expect(asked[0]).toBe("http://127.0.0.1:8102/admin/versions"); | |
| 87 | const refused = await versionsReader({}, (async () => Response.json({ error: "no such route" }, { status: 404 })) as unknown as typeof fetch)(); | |
| 88 | expect(refused.error).toBe("versions are for the owner and moderators"); | |
| 89 | const printed: string[] = []; | |
| 90 | new Door("ws://127.0.0.1:1/live", {}, (l) => printed.push(l), null, null, null, null, async () => ok).input("/versions"); | |
| 91 | new Door("ws://127.0.0.1:1/live", {}, (l) => printed.push(l), null, null, null, null, async () => refused).input("/versions"); | |
| 92 | new Door("ws://127.0.0.1:1/live", {}, (l) => printed.push(l)).input("/versions"); | |
| 93 | await Bun.sleep(50); | |
| 94 | expect([...printed].sort()).toEqual(["app abc · Pi def · DIFFER", "last sync: in step", "error: versions are for the owner and moderators", "error: versions are not available here"].sort()); | |
| 95 | }); |
Software/so101/src/precision-history.ts +7 −1
| @@ -10,9 +10,15 @@ import { join } from "node:path"; | ||
| 10 | 10 | import { emptyRecords, summarize, trim, Tracker, WINDOW_MS, type Card, type Cause } from "./precision.ts"; |
| 11 | 11 | |
| 12 | 12 | /** The deployed commit: the forge's or CI's variable when set, else the checkout's HEAD, else "unknown". Never a shell. */ |
| 13 | export function deployedCommit(env: Record<string, string | undefined> = process.env, root = join(import.meta.dir, "../../..")): string { | |
| 13 | export function deployedCommit(env: Record<string, string | undefined> = process.env, root = join(import.meta.dir, "../../.."), tree = join(import.meta.dir, "..")): string { | |
| 14 | 14 | const named = env.FORGE_APP_COMMIT ?? env.FORGE_COMMIT ?? env.GIT_COMMIT ?? env.SOURCE_COMMIT; |
| 15 | 15 | if (named && /^[0-9a-f]{7,40}$/i.test(named)) return named.slice(0, 12).toLowerCase(); |
| 16 | try { | |
| 17 | const shipped = readFileSync(join(tree, "COMMIT"), "utf8").trim(); // a copied tree (the Pi's) carries its commit | |
| 18 | if (/^[0-9a-f]{7,40}$/.test(shipped)) return shipped.slice(0, 12); | |
| 19 | } catch { | |
| 20 | /* not a copied tree */ | |
| 21 | } | |
| 16 | 22 | try { |
| 17 | 23 | const git = join(root, ".git"); |
| 18 | 24 | const head = readFileSync(join(git, "HEAD"), "utf8").trim(); |
Software/so101/src/relay.ts +23 −3
| @@ -73,6 +73,7 @@ import { AccountError, sessionCookie, visitor } from "./accounts.ts"; | ||
| 73 | 73 | import { CameraSocket } from "./camera-socket.ts"; |
| 74 | 74 | import { OFFLINE_FRAME } from "./offline-frame.ts"; |
| 75 | 75 | import { ServoBaseline } from "./servo-baseline.ts"; |
| 76 | import type { PiSync } from "./pi-deploy.ts"; | |
| 76 | 77 | import { handleAdmin, handleLift } from "./admin-routes.ts"; |
| 77 | 78 | import { readPrecision } from "./precision-panel.ts"; |
| 78 | 79 | import { readServos } from "./servo-panel.ts"; |
| @@ -111,6 +112,9 @@ export interface Link { | ||
| 111 | 112 | } |
| 112 | 113 | |
| 113 | 114 | export interface RelayOptions { |
| 115 | /** Keeps the Pi at the app's commit (src/pi-deploy.ts); checked every `piEveryMs`. */ | |
| 116 | piSync?: PiSync; | |
| 117 | piEveryMs?: number; | |
| 114 | 118 | host?: string; |
| 115 | 119 | port?: number; |
| 116 | 120 | /** The simulated arm's HTTP base, e.g. http://127.0.0.1:41234. */ |
| @@ -328,6 +332,8 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 328 | 332 | const mode = (): Link["mode"] => (remote && armLink ? armLink.state : "simulated"); |
| 329 | 333 | /** Whether the arm server, not the simulation, is the arm right now: live, or only briefly gone. */ |
| 330 | 334 | const onArm = () => mode() === "live" || mode() === "reconnecting"; |
| 335 | /** What a visitor's page is told of the link: only whether the Pi is behind the app, never the commits. */ | |
| 336 | const linkBody = () => ({ ...link(), ...(options.piSync?.differs() ? { piBehind: true } : {}) }); | |
| 331 | 337 | const link = (): Link => ({ mode: mode(), host: remote?.host ?? null, problem: armLink ? armLink.problem : problem }); |
| 332 | 338 | // ------------------------------------------------------------------ the Machine panel's feed |
| 333 | 339 | const shown = () => (onArm() ? armLink : simLink); |
| @@ -467,6 +473,15 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 467 | 473 | return null; |
| 468 | 474 | } |
| 469 | 475 | }; |
| 476 | /** GET /admin/versions: the app's and the Pi's commits and the last sync, for the owner and moderators; to anyone else there is no such route. */ | |
| 477 | const versionsRead = (req: IncomingMessage, res: ServerResponse) => { | |
| 478 | if (!lobby || !lobby.who(req) || !lobby.moderates(req)) return json(res, 404, { error: "no such route" }); | |
| 479 | if (req.method !== "GET") { | |
| 480 | res.setHeader("allow", "GET"); | |
| 481 | return json(res, 405, { error: "GET" }); | |
| 482 | } | |
| 483 | return json(res, 200, options.piSync?.versions() ?? null); | |
| 484 | }; | |
| 470 | 485 | /** The arm's servo registers for the owner's admin page; null when the arm does not answer in time. */ |
| 471 | 486 | const askServos = async () => { |
| 472 | 487 | try { |
| @@ -579,7 +594,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 579 | 594 | */ |
| 580 | 595 | const page = async (req: IncomingMessage, res: ServerResponse, { response }: Awaited<ReturnType<typeof ask>>) => { |
| 581 | 596 | const html = await response.text(); |
| 582 | const boot = JSON.stringify({ link: link(), cameras: cameras.list().map(({ id, name, online }) => ({ id, name, online })) }).replace(/</g, "\\u003c"); | |
| 597 | const boot = JSON.stringify({ link: linkBody(), cameras: cameras.list().map(({ id, name, online }) => ({ id, name, online })) }).replace(/</g, "\\u003c"); | |
| 583 | 598 | const body = Buffer.from(response.status === 200 ? html.replace("</head>", `<script type="application/json" id="boot">${boot}</script>\n</head>`) : html); |
| 584 | 599 | const headers: Record<string, string> = { "content-type": response.headers.get("content-type") ?? "text/html; charset=utf-8", "cache-control": "no-store", vary: "accept-encoding" }; |
| 585 | 600 | const browser = browserCookie(req.headers.cookie); |
| @@ -696,7 +711,8 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 696 | 711 | const cameraAdmin = ADMIN_CAMERA.exec(url.pathname); |
| 697 | 712 | if (cameraAdmin) return await administer(req, res, cameraAdmin[1] === undefined ? null : Number(cameraAdmin[1])); |
| 698 | 713 | if (lobby && url.pathname === "/admin/lift") return await handleLift(lobby, req, res, readBody); |
| 699 | if (lobby && (url.pathname === "/admin" || url.pathname === "/admin/health")) return await handleAdmin(lobby, req, res, url, readBody, askPrecision, askServos, videoCards); | |
| 714 | if (lobby && (url.pathname === "/admin" || url.pathname === "/admin/health")) return await handleAdmin(lobby, req, res, url, readBody, askPrecision, askServos, videoCards, options.piSync ? () => options.piSync!.versions() : undefined); | |
| 715 | if (url.pathname === "/admin/versions") return versionsRead(req, res); | |
| 700 | 716 | if (url.pathname === "/precision") return await precisionRead(req, res, url); |
| 701 | 717 | if (url.pathname === LEADER) return await leaderCalibration(req, res); |
| 702 | 718 | if (url.pathname === "/join") { |
| @@ -731,7 +747,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 731 | 747 | res.writeHead(200, { "content-type": "text/plain; version=0.0.4; charset=utf-8", "cache-control": "no-store" }); |
| 732 | 748 | return res.end(metrics.prometheus() + (lobby?.reports.prometheus() ?? "")); |
| 733 | 749 | } |
| 734 | if (req.method === "GET" && url.pathname === "/link") return json(res, 200, link()); | |
| 750 | if (req.method === "GET" && url.pathname === "/link") return json(res, 200, linkBody()); | |
| 735 | 751 | // Public on purpose: a deploy asks whether it would cut a turn short. No names, no cookie. |
| 736 | 752 | if (req.method === "GET" && url.pathname === "/deploy-ok") { |
| 737 | 753 | const driver = lobby?.turns.snapshot().driver; |
| @@ -1043,6 +1059,9 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 1043 | 1059 | await (armLink as ArmLink | null)?.ready; // bounded by the first attempt's timeout: the first page already knows which arm it has |
| 1044 | 1060 | const timer = lobby ? setInterval(follow, followEveryMs) : undefined; |
| 1045 | 1061 | timer?.unref?.(); |
| 1062 | const piTimer = options.piSync ? setInterval(() => void options.piSync!.check(), options.piEveryMs ?? 60_000) : undefined; | |
| 1063 | piTimer?.unref?.(); | |
| 1064 | if (options.piSync) setTimeout(() => void options.piSync!.check(), 3000).unref?.(); | |
| 1046 | 1065 | const pusher = lobby ? setInterval(() => void push(), statusEveryMs) : undefined; |
| 1047 | 1066 | pusher?.unref?.(); |
| 1048 | 1067 | const summary = options.summaryEveryMs ? setInterval(() => console.log(metrics.summaryLine()), options.summaryEveryMs) : undefined; |
| @@ -1057,6 +1076,7 @@ export async function startRelay(options: RelayOptions): Promise<RunningRelay> { | ||
| 1057 | 1076 | new Promise<void>((resolve) => { |
| 1058 | 1077 | clearInterval(timer); |
| 1059 | 1078 | clearInterval(pusher); |
| 1079 | clearInterval(piTimer); | |
| 1060 | 1080 | clearInterval(summary); |
| 1061 | 1081 | settingsChanged?.(); |
| 1062 | 1082 | turnEnded?.(); |
Software/so101/src/server.ts +4 −1
| @@ -4,6 +4,7 @@ | ||
| 4 | 4 | * |
| 5 | 5 | * GET / the browser app: cameras, 3D model, live joints, poses, jog, record and play, metrics |
| 6 | 6 | * GET /status every joint: position, load, temperature, torque; what drives the arm; zone state and the table (tableZ) |
| 7 | * GET /version {"commit"}: the commit this server runs (the COMMIT file of a copied tree, else the checkout) | |
| 7 | 8 | * GET /limits each servo's calibrated Min/Max_Position_Limit |
| 8 | 9 | * GET /servo-params each servo's control registers (delay, P/D/I, startup force, dead zones, velocity loop), read-only; 409 while something drives |
| 9 | 10 | * PUT /servo-params {"id": n, "P"?, "D"?, "I"?, "cwDeadZone"?, "ccwDeadZone"?, "minStartupForce"?} write one joint's gains within safe ranges; 409 while anything drives, 423 on E-stop or hold |
| @@ -81,7 +82,7 @@ import { clamp, type Frame } from "./mapping.ts"; | ||
| 81 | 82 | import { MachineLog, perServo } from "./machine-log.ts"; |
| 82 | 83 | import { Metrics } from "./metrics.ts"; |
| 83 | 84 | import { ARRIVE_TICKS, parseSnapBack, type Cause, type Tracker } from "./precision.ts"; |
| 84 | import { PrecisionHistory } from "./precision-history.ts"; | |
| 85 | import { deployedCommit, PrecisionHistory } from "./precision-history.ts"; | |
| 85 | 86 | import { MotionError, moveToPose, playTrajectory, recordTrajectory, STALL_SPAN_TICKS } from "./motion.ts"; |
| 86 | 87 | import { assertPoseName, parsePoseFile, serializePoseFile, type PoseFile } from "./poses.ts"; |
| 87 | 88 | import { MAX_ZERO_OFFSET_DEG, zeroOffsetDeg } from "./joint-zero.ts"; |
| @@ -775,6 +776,8 @@ export async function startServer(bus: FeetechBus, options: ServerOptions = {}): | ||
| 775 | 776 | return file(res, join(here, "leader-map.js"), "no-cache"); |
| 776 | 777 | case "GET /status": |
| 777 | 778 | return json(res, 200, await status()); |
| 779 | case "GET /version": | |
| 780 | return json(res, 200, { commit: deployedCommit() }); | |
| 778 | 781 | case "GET /limits": |
| 779 | 782 | return json(res, 200, Object.fromEntries(limits)); |
| 780 | 783 | case "GET /servo-params": { |
Software/so101/src/terminal.ts +41 −3
| @@ -33,6 +33,7 @@ | ||
| 33 | 33 | * /table <mm> admin: set the table offset, -100..100 mm (PUT /settings tableOffsetMm) |
| 34 | 34 | * /zero <joint> [deg] admin: make the model read deg (default 0) for that joint at the pose it is in now (POST /settings/joint-zero) |
| 35 | 35 | * /health owner or moderator: each open page's link as it last reported it (GET /admin/health, the page's Admin -> Health) |
| 36 | * /versions owner or moderator: the app's and the Pi's commits and the last sync (GET /admin/versions) | |
| 36 | 37 | * /precision owner or moderator: the precision scorecard /admin shows (GET /precision, src/precision-panel.ts) |
| 37 | 38 | * /servos owner: each joint's servo control registers /admin shows (GET /servo-params, src/servo-panel.ts); 404 to anyone else |
| 38 | 39 | * /machine [off] the Machine panel (/machine, src/machine-feed.ts): the joints, the link and the server as they are, |
| @@ -118,6 +119,24 @@ export function reader(env: Env, fetcher: typeof fetch = fetch): Read { | ||
| 118 | 119 | }; |
| 119 | 120 | } |
| 120 | 121 | |
| 122 | /** What GET /admin/versions answered: the lines to print, or why not (a guest is told there is no such route). */ | |
| 123 | export type Seen = { error: string | null; lines: string[] }; | |
| 124 | export type ReadVersions = () => Promise<Seen>; | |
| 125 | ||
| 126 | /** GET the commits as /admin shows them, signed afresh; the server decides who may. */ | |
| 127 | export function versionsReader(env: Env, fetcher: typeof fetch = fetch): ReadVersions { | |
| 128 | return async () => { | |
| 129 | try { | |
| 130 | const r = await fetcher(`${appUrl(env)}/admin/versions`, { headers: terminalHeaders(env), signal: AbortSignal.timeout(5000) }); | |
| 131 | const body = (await r.json().catch(() => null)) as Record<string, unknown> | null; | |
| 132 | if (r.ok) return { error: null, lines: versionLines(body) }; | |
| 133 | return { error: r.status === 404 ? "versions are for the owner and moderators" : plain(body?.error ?? `status ${r.status}`), lines: [] }; | |
| 134 | } catch (err) { | |
| 135 | return { error: plain((err as Error).message), lines: [] }; | |
| 136 | } | |
| 137 | }; | |
| 138 | } | |
| 139 | ||
| 121 | 140 | /** What GET /servo-params answered, read as the admin panel reads it; or why not (anyone but the owner is told there is no such route). */ |
| 122 | 141 | export type Servos = { status: number; error: string | null; panel: ServoPanel | null }; |
| 123 | 142 | export type ReadServos = () => Promise<Servos>; |
| @@ -136,6 +155,13 @@ export function servosReader(env: Env, fetcher: typeof fetch = fetch): ReadServo | ||
| 136 | 155 | }; |
| 137 | 156 | } |
| 138 | 157 | |
| 158 | /** The commits and the last sync outcome, one line each; a mismatch is flagged. */ | |
| 159 | export function versionLines(v: Record<string, unknown> | null): string[] { | |
| 160 | if (!v) return ["the Pi is not kept in step from here (no deploy key, or not production)"]; | |
| 161 | const pi = v.pi === null ? "?" : plain(v.pi); | |
| 162 | return [plain(`app ${v.app} · Pi ${pi}${v.pi === v.app ? "" : " · DIFFER"}`), plain(`last sync: ${v.sync}`)]; | |
| 163 | } | |
| 164 | ||
| 139 | 165 | /** One line per joint: its registers, then its error if the servo did not answer; or the one sentence. */ |
| 140 | 166 | export function servosLines(p: ServoPanel): string[] { |
| 141 | 167 | if ("notice" in p) return [plain(p.notice)]; |
| @@ -239,7 +265,7 @@ const said = (room: Room, m: unknown): string => { | ||
| 239 | 265 | }; |
| 240 | 266 | |
| 241 | 267 | const ZERO_USAGE = "usage: /zero <joint, e.g. shoulder_pan> [degrees the model should read, default 0]"; |
| 242 | export const HELP = "type to chat · /join /leave /ready /extend · /stop /resume (driver, owner) · /end [who] /table here|touch 1|2|3|<mm> /zero <joint> [deg] (owner) · /mute <who> /unmute <who> (owner, moderator) · /delete <who|#id> (owner, moderator) · /report <who|#id> · /reports /dismiss #<id> (owner, moderator) · /removed /restore #<id> /visitors (owner) · /health /precision (owner, moderator) · /servos (owner) · /machine [off] · /who · /quit"; | |
| 268 | export const HELP = "type to chat · /join /leave /ready /extend · /stop /resume (driver, owner) · /end [who] /table here|touch 1|2|3|<mm> /zero <joint> [deg] (owner) · /mute <who> /unmute <who> (owner, moderator) · /delete <who|#id> (owner, moderator) · /report <who|#id> · /reports /dismiss #<id> (owner, moderator) · /removed /restore #<id> /visitors (owner) · /health /precision /versions (owner, moderator) · /servos (owner) · /machine [off] · /who · /quit"; | |
| 243 | 269 | |
| 244 | 270 | /** The Machine feed's socket beside the lobby's: same app, same port. */ |
| 245 | 271 | export const machineUrl = (live: string): string => live.replace(/\/live$/, "/machine"); |
| @@ -382,7 +408,7 @@ function target(room: Room, word: string): string | null { | ||
| 382 | 408 | return ID.test(word) ? word : null; |
| 383 | 409 | } |
| 384 | 410 | |
| 385 | export type Typed = { send: Frame } | { post: "/stop" | "/resume" } | { table: "here" | number | { touch: 1 | 2 | 3 } } | { zero: { joint: string; referenceDeg: number } } | { print: string } | { lines: string[] } | { machine: boolean } | { health: true } | { precision: true } | { servos: true } | { quit: true } | null; | |
| 411 | export type Typed = { send: Frame } | { post: "/stop" | "/resume" } | { table: "here" | number | { touch: 1 | 2 | 3 } } | { zero: { joint: string; referenceDeg: number } } | { print: string } | { lines: string[] } | { machine: boolean } | { health: true } | { precision: true } | { servos: true } | { versions: true } | { quit: true } | null; | |
| 386 | 412 | |
| 387 | 413 | /** What a typed line asks for. Validation the server repeats anyway; this only saves a round trip on a typo. */ |
| 388 | 414 | export function typed(room: Room, input: string): Typed { |
| @@ -461,6 +487,8 @@ export function typed(room: Room, input: string): Typed { | ||
| 461 | 487 | return { precision: true }; |
| 462 | 488 | case "servos": |
| 463 | 489 | return { servos: true }; |
| 490 | case "versions": | |
| 491 | return { versions: true }; | |
| 464 | 492 | case "machine": |
| 465 | 493 | if (arg && arg.toLowerCase() !== "off" && arg.toLowerCase() !== "on") return { print: "usage: /machine [off]" }; |
| 466 | 494 | return { machine: arg.toLowerCase() !== "off" }; |
| @@ -489,9 +517,11 @@ export class Door { | ||
| 489 | 517 | private readonly read: Read | null; |
| 490 | 518 | private readonly readCard: ReadCard | null; |
| 491 | 519 | private readonly readServos: ReadServos | null; |
| 520 | private readonly readVersions: ReadVersions | null; | |
| 492 | 521 | |
| 493 | 522 | /** `post` reaches the app's /stop and /resume, `read` its /admin/health, `readCard` its /precision; without them, the door says it cannot. */ |
| 494 | constructor(url: string, headers: Record<string, string>, print: (line: string) => void, post: Post | null = null, read: Read | null = null, readCard: ReadCard | null = null, readServos: ReadServos | null = null) { | |
| 523 | constructor(url: string, headers: Record<string, string>, print: (line: string) => void, post: Post | null = null, read: Read | null = null, readCard: ReadCard | null = null, readServos: ReadServos | null = null, readVersions: ReadVersions | null = null) { | |
| 524 | this.readVersions = readVersions; | |
| 495 | 525 | this.read = read; |
| 496 | 526 | this.readCard = readCard; |
| 497 | 527 | this.readServos = readServos; |
| @@ -539,6 +569,7 @@ export class Door { | ||
| 539 | 569 | else if ("health" in t) void this.health(); |
| 540 | 570 | else if ("precision" in t) void this.precision(); |
| 541 | 571 | else if ("servos" in t) void this.servos(); |
| 572 | else if ("versions" in t) void this.versions(); | |
| 542 | 573 | else if ("machine" in t) this.watchMachine(t.machine); |
| 543 | 574 | else this.send(t.send); |
| 544 | 575 | return true; |
| @@ -560,6 +591,13 @@ export class Door { | ||
| 560 | 591 | for (const line of r.error ? [`error: ${r.error}`] : healthLines(r.rows)) this.print(line); |
| 561 | 592 | } |
| 562 | 593 | |
| 594 | /** The server decides who may; a refusal is printed, else the commits and the last sync outcome. */ | |
| 595 | private async versions(): Promise<void> { | |
| 596 | if (!this.readVersions) return this.print("error: versions are not available here"); | |
| 597 | const r = await this.readVersions(); | |
| 598 | for (const line of r.error ? [`error: ${r.error}`] : r.lines) this.print(line); | |
| 599 | } | |
| 600 | ||
| 563 | 601 | /** The server decides who may; a refusal is printed, else the scorecard. */ |
| 564 | 602 | private async precision(): Promise<void> { |
| 565 | 603 | if (!this.readCard) return this.print("error: precision is not available here"); |
Software/so101/src/tty.ts +2 −2
| @@ -4,7 +4,7 @@ | ||
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | import { createInterface, type Interface } from "node:readline"; |
| 7 | import { cardReader, Door, HELP, liveUrl, poster, reader, servosReader, terminalHeaders } from "./terminal.ts"; | |
| 7 | import { cardReader, Door, HELP, liveUrl, poster, reader, servosReader, terminalHeaders, versionsReader } from "./terminal.ts"; | |
| 8 | 8 | |
| 9 | 9 | const env = process.env; |
| 10 | 10 | const where = env.FORGE_APP_ENV === "preview" ? `preview of #${env.FORGE_APP_PULL ?? "?"}` : (env.FORGE_APP_ENV ?? "local"); |
| @@ -17,7 +17,7 @@ const print = (line: string) => { | ||
| 17 | 17 | rl?.prompt(true); |
| 18 | 18 | }; |
| 19 | 19 | |
| 20 | const door = new Door(liveUrl(env), terminalHeaders(env), print, poster(env), reader(env), cardReader(env), servosReader(env)); | |
| 20 | const door = new Door(liveUrl(env), terminalHeaders(env), print, poster(env), reader(env), cardReader(env), servosReader(env), versionsReader(env)); | |
| 21 | 21 | print(`\x1b[1mSO-101 arm\x1b[0m · ${where} · ${HELP}`); |
| 22 | 22 | try { |
| 23 | 23 | await door.open(() => { |
Software/so101/src/viewer.html +2 −1
| @@ -585,8 +585,9 @@ | ||
| 585 | 585 | el.hidden = false; |
| 586 | 586 | el.textContent = l.mode === "live" ? `Live arm · ${l.host}` : l.mode === "reconnecting" ? `Reconnecting to the arm… · ${l.host}` |
| 587 | 587 | : l.mode === "offline" ? `OFFLINE · simulated arm · ${l.problem}` : l.problem ? `Simulated · ${l.problem}` : "Simulated"; |
| 588 | if (l.piBehind) el.textContent += " · Pi update pending"; // the arm server is older than the app; it is being updated | |
| 588 | 589 | el.title = el.textContent; |
| 589 | el.className = l.mode === "live" ? "on" : l.mode === "offline" ? "bad" : l.mode === "reconnecting" || l.problem ? "warn" : ""; | |
| 590 | el.className = l.piBehind ? "warn" : l.mode === "live" ? "on" : l.mode === "offline" ? "bad" : l.mode === "reconnecting" || l.problem ? "warn" : ""; | |
| 590 | 591 | // Another arm has other limits, poses and joints: start the page over on it, once nothing on the page is driving. |
| 591 | 592 | // So does a page that opened while reconnecting and never got the arm's limits. |
| 592 | 593 | const other = linkMode !== null && realArm(l.mode) !== realArm(linkMode); |
new above
offline
A 'failed' outcome is not remembered, so a persistent ssh or copy failure re-sends the whole tree every 60 s. Back off after repeated failures.