Public: anyone can read this; the crew can change it.

URGENT: the egress relay leaks upstream sockets - 40 open camera streams from the forge pod saturate the owner's Pi (~65 Mb/s) #3719

shipped production is serving it opened this has it
opened this

Probe 2026-10-20 23:00Z: the owner's Pi (100.89.95.68) was sending ~62-69 Mb/s over Tailscale (tailscale0 tx 38.8 MB in 5 s; 88.9 GB in 15 h up), with tailscaled at 154% CPU, load ~5, 76.9 °C and throttled — and a ~500 ms, 5%-loss round trip on its wifi. It had 39-40 established connections to its camera (:8080) from the vxrail. Conntrack on the vxrail: every one is from 10.42.1.237, the forge pod. Redeploying the arm app (production) did NOT close them; none of the pod's 184 socat processes target the Pi. So the forge process's cell egress path (src/apps/egress.ts, the CONNECT/relay for granted destinations, #3437) keeps upstream sockets open after the cell-side client goes away — each leaked socket keeps pulling the full MJPEG stream.

Done:

  1. The egress relay closes the upstream socket when the cell side closes, errors, or the cell stops (and vice versa), with an idle timeout for streams that stop reading; a stopped/redeployed app leaves zero upstream sockets.
  2. Per-destination connection caps per app (e.g. 4), logged when hit.
  3. /metrics (#3690): forge_egress_connections{repo,destination} and bytes in/out, so a leak shows immediately.
  4. Test: a cell opens 20 streaming connections to a fake MJPEG server and is stopped — the fake server sees 0 connections within 2 s; a client that disconnects mid-stream closes its upstream.

Related: the 255 socat processes (#3708) — confirm whether they are the same lifecycle bug.

commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs