Public: anyone can read this; the crew can change it.

Two processes at once: each run queued once, each webhook sent once, streams drained with a resume hint #909

Merged opened this in answer to

Check · Review · Land · Live done — Landed and live in production.

Merged

Closes #875 together with #902 (part a, additive migrations). This is part b: two processes on one data dir.

shipped production is serving it for issue #875 — its issue is already done deploy fc82d1a

opened this

Closes #875 together with #902 (part a, additive migrations). This is part b: two processes on one data dir.

Shared functions that change:

  • queueAgentRun (crew/dispatch.ts): its look and its insert are now one write transaction.
  • The checks service's request: the same.
  • deliverDue (core/webhooks.ts): it claims each delivery before sending it.
  • eventStream (http/sse.ts): it registers every open stream, and the new drainStreams() ends them all.

What changed

  • Each run queued once. During a roll, two forge processes share the database. A run that one process looked for and the other inserted was queued twice. test/overlap.test.ts shows it with two real processes racing to queue the crew on 400 issues: before this change, issues got two runs each. The crew's queueing and the checks' queueing now happen inside transact (BEGIN IMMEDIATE), so each issue ends with exactly one run.
  • Each webhook sent once. Both processes saw the same pending deliveries, and both sent them. deliverDue now claims a row with a 60-second lease on next_try_at before sending it. Only the claimant sends; a sender that dies mid-send is retried when the lease runs out.
  • Draining. A draining process ends every open event stream with retry: 1000 and event: drain, so browsers reconnect, to the new process, from the last id they saw. They no longer hold the old socket until it exits. main.ts's shutdown calls drainStreams() first.
  • The contract written down. docs/ARCHITECTURE.md has a new section, "Two processes at once": what the old process does when it drains, and a table of every place both processes could act, with the compare-and-set or lease that makes it happen once. That covers claimRun, queueAgentRun, the checks' request, run leases, webhook claims, claimPromptRun and the reconciler.

Tests (test/overlap.test.ts, in the gate; each failed before its change):

  • Two processes queue the crew on 400 issues at the same moment: one run per issue, and the counts add up to 400.
  • Two processes send 60 due deliveries: the receiver gets each exactly once.
  • A draining process ends an open /events stream with the resume hint.

Not here: a minute-long soak with two full forges is the rehearsal on a copy (#873, part 4). This test holds each shared write path instead. App environments stay per-process by design, and the section says so.

🤖 Generated with Claude Code

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs