URGENT: the forge crashed (Bun segfault on the main thread) and 255 socat processes are alive #3724

Crew runs could leave socat leaking for good: a session's process group was only killed on timeout or cancel. The fix kills whatever a session leaves in its group however the session ends. Restarts and crashes are now counted, and the process table by name is logged once a minute. The other parts of the issue are not done: the crash report couldn't be decoded from this run (split to #3726), and the /metrics series wait for #3690 (split to #3727).

Where the socat came from. Claude Code starts its sandbox's HTTP and SOCKS bridges as socat UNIX-LISTEN:…,fork children inside its own process group, and each connection forks another socat. A session that exited normally or crashed left them re-parented to pid 1. reapOrphans couldn't find them either: it looks for processes holding files in the clone, and the bridges' sockets are in the run's temp directory.

What changed

• src/crew/claude.ts: runClaude calls a new reapGroup once the session exits, for every ending. It SIGKILLs anything left in the group and refuses pgid ≤ 1 or the forge's own pid.
• src/crew/leftovers.ts: a backstop sweep. It kills any socat whose parent is pid 1 and whose command line names a path under the crew directory, by its process group. Nothing else on the machine matches.
• src/kernel/census.ts and a minute processes line in src/main.ts: logs count=, top="socat=… bwrap=…", restarts=, and reapedBridges=N when the sweep killed something.
• src/kernel/boots.ts: boots.json under the data directory counts every start and marks each one clean on an orderly exit. A start that never got that mark is counted as unclean (a crash or OOM kill). A release's old process draining beside the new one drops out of that count once it exits. The boot line and every autonomy tick line say restarts=N, plus unclean=N when there are any. Started here, one process killed mid-drain then one stopped with SIGINT gave restarts=1 unclean=1, and the second was marked clean.

Not done here

• The Bun segfault: the sandbox refused the connection to bun.report, so the crash stack is not decoded and Bun is not pinned. #3726 covers decoding it, then pinning or upgrading Bun (the Dockerfile's oven/bun:1.3-debian floats) or reporting it upstream.
• /metrics: the endpoint (#3690) is still open, so #3727 waits on it for forge_process_restarts_total and forge_child_processes{comm}. #3708 waits on both new issues.
• Re-gating after boot: #3155/#3156 are already closed, so nothing changed there.
• /healthz under load: that is #3319's work, which is still open.

Tests

• test/session-reap.test.ts: a stand-in Claude starts a sleeper in its group and then finishes cleanly or exits 3; either way the sleeper is gone. Both tests fail with reapGroup removed.
• test/boots-census.test.ts: the ledger (crash, release overlap, more starts than it keeps, corrupt file, read-only directory), parsing /proc/<pid>/stat, and that the sweep only picks orphaned crew bridges.
• test/cell-relays-reaped.test.ts: starts and stops 50 cells and expects zero socat left, with a first case showing the count can see a running cell's relays. It is skipped in this sandbox (no unix sockets), so its first run is the forge's check on this push.
• Type check is clean; the new files, session-settle, steer, crew-watch, the autonomy tests and the five cross files all pass. / and /healthz answer 200 on a local boot.

To try it, boot with bun run src/main.ts and watch for processes … top="…" once a minute and restarts= on forge is up. Kill the process with kill -9 and boot again: the line then says unclean=1.

🤖 Generated with Claude Code

Closes #3708

Crew runs could leave socat leaking for good: a session's process group was only killed on timeout or cancel. The fix kills whatever a session leaves in its group however the session ends. Restarts and crashes are now counted, and the process table by name is logged once a minute. The other parts of the issue are not done: the crash report couldn't be decoded from this run (split to #3726), and the /metrics series wait for #3690 (split to #3727).

Where the socat came from. Claude Code starts its sandbox's HTTP and SOCKS bridges as socat UNIX-LISTEN:…,fork children inside its own process group, and each connection forks another socat. A session that exited normally or crashed left them re-parented to pid 1. reapOrphans couldn't find them either: it looks for processes holding files in the clone, and the bridges' sockets are in the run's temp directory.

What changed

• src/crew/claude.ts: runClaude calls a new reapGroup once the session exits, for every ending. It SIGKILLs anything left in the group and refuses pgid ≤ 1 or the forge's own pid.
• src/crew/leftovers.ts: a backstop sweep. It kills any socat whose parent is pid 1 and whose command line names a path under the crew directory, by its process group. Nothing else on the machine matches.
• src/kernel/census.ts and a minute processes line in src/main.ts: logs count=, top="socat=… bwrap=…", restarts=, and reapedBridges=N when the sweep killed something.
• src/kernel/boots.ts: boots.json under the data directory counts every start and marks each one clean on an orderly exit. A start that never got that mark is counted as unclean (a crash or OOM kill). A release's old process draining beside the new one drops out of that count once it exits. The boot line and every autonomy tick line say restarts=N, plus unclean=N when there are any. Started here, one process killed mid-drain then one stopped with SIGINT gave restarts=1 unclean=1, and the second was marked clean.

Not done here

• The Bun segfault: the sandbox refused the connection to bun.report, so the crash stack is not decoded and Bun is not pinned. #3726 covers decoding it, then pinning or upgrading Bun (the Dockerfile's oven/bun:1.3-debian floats) or reporting it upstream.
• /metrics: the endpoint (#3690) is still open, so #3727 waits on it for forge_process_restarts_total and forge_child_processes{comm}. #3708 waits on both new issues.
• Re-gating after boot: #3155/#3156 are already closed, so nothing changed there.
• /healthz under load: that is #3319's work, which is still open.

Tests

• test/session-reap.test.ts: a stand-in Claude starts a sleeper in its group and then finishes cleanly or exits 3; either way the sleeper is gone. Both tests fail with reapGroup removed.
• test/boots-census.test.ts: the ledger (crash, release overlap, more starts than it keeps, corrupt file, read-only directory), parsing /proc/<pid>/stat, and that the sweep only picks orphaned crew bridges.
• test/cell-relays-reaped.test.ts: starts and stops 50 cells and expects zero socat left, with a first case showing the count can see a running cell's relays. It is skipped in this sandbox (no unix sockets), so its first run is the forge's check on this push.
• Type check is clean; the new files, session-settle, steer, crew-watch, the autonomy tests and the five cross files all pass. / and /healthz answer 200 on a local boot.

To try it, boot with bun run src/main.ts and watch for processes … top="…" once a minute and restarts= on forge is up. Kill the process with kill -9 and boot again: the line then says unclean=1.

🤖 Generated with Claude Code

This pull cleans up the crew-session child processes that the issue found leaking: the 255 live socat processes came from crew runs that ended without being cleaned up. It also counts restarts, so a crash shows up in the forge's own logs. The newest comment on the issue is the automatic triage note asking for a pillar label. It needs no code change, so this push only merges main into the branch, which merged without conflicts.

What the branch changes (from 498d04fe, rebased onto current main):

• Where the socat came from: Claude Code starts its sandbox's socat bridges in its own process group. The forge only signalled that group when a session timed out or was cancelled. Any other way a session ended left its bridges orphaned under pid 1.
• The fix: src/crew/claude.ts now kills whatever is left in the group after every session exits.
• A once-a-minute sweep (src/kernel/census.ts, src/crew/leftovers.ts) counts the process table by name and kills any orphaned bridge running under the crew directory, so leftovers from before this fix also get cleaned up.
• Restart counting: src/kernel/boots.ts records each start, and each start that followed an unclean exit, in boots.json under the data directory. The boot line and the autonomy tick line now print restarts=N unclean=N.

Not in this pull:

• The Bun segfault (item 1): not investigated. The sandbox can't reach bun.report, so the stack hasn't been decoded and nothing has been reported to Bun.
• The /metrics series (item 3): forge_process_restarts_total and forge_child_processes{comm} are left for #3690, since /metrics doesn't exist yet. The census and boot counts are already in a form it can read.
• Re-gating the queue after a crash (item 3): left to #3155/#3156.
• /healthz under load (item 4): left to #3319.
• The test the issue asks for: I haven't confirmed that starting and stopping 50 cells leaves zero socat. test/cell-relays-reaped.test.ts is the test aimed at this, but some of these tests can't run in this sandbox (only the totals are shown), so check that the forge's check runs them.

Checks: on the merged branch, the type check passes and test/boots-census.test.ts, test/session-reap.test.ts and test/cell-relays-reaped.test.ts give 17 pass, 4 skipped, 0 fail.

To try it: after a crew run finishes, ps -eo comm | grep -c socat should stay flat, and the tick line should show restarts= and unclean=.

🤖 Generated with Claude Code