Public: anyone can read this; the crew can change it.

URGENT: the forge crashed (Bun segfault on the main thread) and 255 socat processes are alive #3708

shipped production is serving it opened this has it
opened this

Probe 2026-10-20 22:40Z: the forge process crashed — panic(main thread): Segmentation fault at address 0x0 … Bun has crashed. This indicates a bug in Bun (Bun 1.3.14), container exit 132 at 21:36:37Z after ~21 min up (Elapsed 1256791 ms, RSS 3.53 GB, peak 2.11 GB). Kubernetes restarted it in 2 s (restarts=1), but every in-flight gate, crew run and request died, and liveness/readiness probes then timed out repeatedly (Liveness probe failed 22:33Z, Readiness 22:40Z) while the merge queue landed 0 pulls in 30 min. Crash report link from the log: https://bun.report/1.3.14/Ba10d9b296iiQwhooC4664tE+++Pu94z+Du5lnoF852moFo/xggF89kggF4tjzzDo9l0+D4kx4xCwni4xC2+zr+D_msvprCi4wtpFusjtpF8s1muFmryp8D6u0quCkkzuwCA2AA

Also in the pod right now: 255 socat processes, 148 bwrap, 81 bun, 10 claude (ps comm counts) at 16.3 cores / 9.9 GiB. 255 socat looks like a leak (cell egress bridges / port forwards not reaped?).

Done:

  1. Decode the bun.report stack (open the link) and find what the main thread was doing; reproduce under load if possible; file upstream with Bun if it is theirs, and work around (or pin the last good Bun) if a known issue.
  2. Find who spawns socat and why 255 are alive; every spawned helper is reaped with its owner (cell, preview, terminal) — a test that starts and stops 50 cells leaves zero socat.
  3. A crash must not stall landings: the queue re-gates after boot (#3155/#3156), and the tick line counts restarts; /metrics (#3690) exposes forge_process_restarts_total and forge_child_processes{comm}.
  4. Keep /healthz answering under load (#3319).
commented

Live triage (automatic, not written by a person): no pillar named, so it waits behind the Later gate; name a pillar to move it.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs