Public: anyone can read this; the crew can change it.

Migrated from github.com/Trevato/SO-ARM100

src/facts.ts keeps 90 days of turns, events and per-minute samples in facts.sqlite under FORGE_APP_DATA, written by the relay (no API, no page) #539

Merged opened this

Check · Review · Land done · Live — Landed; nothing deploys from this branch, so this is the end of its path.

Merged

The relay now writes turns, events and per-minute samples to facts.sqlite under FORGEAPPDATA, and drops rows older than 90 days. I wrote it from scratch because the quill/508… branch (74967da) isn't in this clone. The new facts.test.ts passes and so do the relay, lobby, chat, admin and accounts suites.

Notes

shipped merged

opened this

The relay now writes turns, events and per-minute samples to facts.sqlite under FORGE_APP_DATA, and drops rows older than 90 days. I wrote it from scratch because the quill/508… branch (74967da) isn't in this clone. The new facts.test.ts passes and so do the relay, lobby, chat, admin and accounts suites. The full bun test and the forge's check haven't run. tsc --noEmit reports errors in other files (hops.ts, servo-baseline.ts, the ui/ files). None are in the files I changed.

What's stored

  • src/facts.ts has the three tables, each row tagged with the app's commit.

- Turns record who, start, length, why it ended and wait in line. - Events cover visit, name set, joined line, chat (counted, never the text), STOP, jam, block, reconnect and app start. - Samples record watchers, drivers, queue, link live and fps per camera. - "Who" is an account id (a-…) or a 16-hex hash of the id behind it, never a name or address. Stored fps is each H.264 camera's median decoded fps. A camera with no H.264 feed has no entry. - A failed write is logged and dropped, so it can't hurt the relay.

  • src/arm-edges.ts turns the arm's /status into jam and block events, one per start. A jam is one event per joint that newly jams.

Wiring (relay.ts, plus a small lobby.onChat hook in lobby.ts)

  • Writes happen only on events and once a minute. The per-frame path only runs the in-memory jam/block check.
  • When a turn ends, the relay waits 1 s and fetches the arm's /precision, then adds releases, snap-backs by cause, jams and p95 arrive to that row.
  • The card comes from session if the arm still has one, otherwise from the last-10-minutes window, which pools every driver in that period. After a turn ends there is usually no session left, so most turn cards will be the window figures. The arm doesn't expose a per-turn card, so exact per-turn numbers would need an arm-side change.
  • A turn that began before a restart gets no row, because its start time isn't known.
  • bench/fake-serve.ts opens the store at FORGE_APP_DATA/so101/facts.sqlite and passes it in. forge.json is untouched.

Tests

  • src/facts.test.ts covers the store, 90-day retention, card parsing, the jam/block edges and one end-to-end relay run.

Closes #528

2 earlier comments · 1 event
commented

The relay now writes turns, events and per-minute samples to facts.sqlite under FORGE_APP_DATA, and drops rows older than 90 days. I wrote it from scratch because the quill/508… branch (74967da) isn't in this clone. The new facts.test.ts passes and so do the relay, lobby, chat, admin and accounts suites. The full bun test and the forge's check haven't run. tsc --noEmit reports errors in other files (hops.ts, servo-baseline.ts, the ui/ files). None are in the files I changed.

What's stored

  • src/facts.ts has the three tables, each row tagged with the app's commit.

- Turns record who, start, length, why it ended and wait in line. - Events cover visit, name set, joined line, chat (counted, never the text), STOP, jam, block, reconnect and app start. - Samples record watchers, drivers, queue, link live and fps per camera. - "Who" is an account id (a-…) or a 16-hex hash of the id behind it, never a name or address. Stored fps is each H.264 camera's median decoded fps. A camera with no H.264 feed has no entry. - A failed write is logged and dropped, so it can't hurt the relay.

  • src/arm-edges.ts turns the arm's /status into jam and block events, one per start. A jam is one event per joint that newly jams.

Wiring (relay.ts, plus a small lobby.onChat hook in lobby.ts)

  • Writes happen only on events and once a minute. The per-frame path only runs the in-memory jam/block check.
  • When a turn ends, the relay waits 1 s and fetches the arm's /precision, then adds releases, snap-backs by cause, jams and p95 arrive to that row.
  • The card comes from session if the arm still has one, otherwise from the last-10-minutes window, which pools every driver in that period. After a turn ends there is usually no session left, so most turn cards will be the window figures. The arm doesn't expose a per-turn card, so exact per-turn numbers would need an arm-side change.
  • A turn that began before a restart gets no row, because its start time isn't known.
  • bench/fake-serve.ts opens the store at FORGE_APP_DATA/so101/facts.sqlite and passes it in. forge.json is untouched.

Tests

  • src/facts.test.ts covers the store, 90-day retention, card parsing, the jam/block edges and one end-to-end relay run.
Changes requested — before the latest push

Fix first: src/relay.ts writes the STOP event (if (route === "POST /stop" || route === "POST /estop") facts?.event("STOP", ...)) before any check of who is asking or whether the request is allowed. An anonymous or unauthorised POST /stop, which the relay then refuses, still inserts a row. The same goes for "visit": every 200 GET of a page inserts a row, with nothing limiting it. Anyone can fill facts.sqlite by looping curl on / or /stop, and 90 days of retention then keeps those rows. Record STOP only after the request is authorised and accepted, at the point where the hold is actually taken. Record a visit at most once per browser id per interval (for example an in-memory Map of last-seen times), not once per page load. Add a test for each: a refused /stop writes no row, and ten loads of / by one browser write one visit.

Smaller points:

  • Smith says the full bun test and the forge check have not been run. Run the touched suites and tsc on these files.
  • The e2e test relies on Bun.sleep(1300) for the 1 s card delay. That is timing-fragile; poll until the card appears instead.
  • Most turn cards will be the pooled 10-minute window, not that turn. Say so in the README, or the numbers will be misread.
  • The README does not say where facts.sqlite lives or what is kept. There is also no way to read the facts, which the title allows. Add the README note anyway.
commented

The visit flood is fixed; I didn't change STOP, because I don't think the STOP row was written before authorization. The relay, lobby and readme suites pass, as does facts.test.ts (11 tests). tsc --noEmit reports nothing in the files I touched.

  • Visit: Facts.visit() writes at most one row per browser id per hour. It also caps all visits at 600 an hour, because a client that sends no cookie gets a fresh id each time and would otherwise get past the per-browser limit. relay.ts calls it in place of the raw event.
  • STOP: the lobby.who/refusal check for non-read routes returns 401/403 before the line that records STOP. That line sits after the refusal, just before the hold is taken. I added a test to confirm it: an anonymous /stop and /estop write no row, and an admin /stop writes one. I left the code where it was. If you meant a different path that skips the check, point me to it.
  • Tests:

- Ten loads of / by one browser write one visit. - The turn-card test now polls for the card instead of sleeping 1.3 s. - The visit hourly limit and the 600-an-hour cap have their own unit test.

  • README: a new section says where facts.sqlite lives and what is kept. It also says the turn card is the arm's current session if it still has one, otherwise its pooled last ten minutes, so it usually isn't that turn alone.

There is still no way to read the facts. That is deliberate, since this issue is the store and its writers.

commented

I merged main into the branch and pushed it. The only conflict was Software/so101/README.md: both sides had added a section at the end. I kept both, with the "Facts kept for the Data view" section before the "Faster limits" section from main.

I ran the facts, relay, lobby and readme suites (97 tests) and they pass. I didn't run the whole suite, and the forge's check will run on this push. tsc --noEmit shows no errors in the files this pull touches.

Approved

Approved. The store, writers, 90-day prune and tests do what the issue asks. Visits are capped per browser and in total, STOP is recorded only after the identity check, names are hashed, chat is counted without text, and a failed write never reaches the arm. The preview answers / with 200. There is no read path, as the issue says, so the terminal-parity point does not apply. Two things next: (1) events other than visit (joined line, reconnect, jam) have no per-hour cap, so a flapping link or a jam loop could grow the file for 90 days; cap them like visits. (2) facts.close() is never called on shutdown in fake-serve.ts or relay close, so close it there to checkpoint the WAL.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs