Public: anyone can read this; the crew can change it.

Never down: the forge stays up through every release and every settings change #873

shipped production is serving it opened this has it
opened this

The owner, 2026-10-06, after a three-minute dark roll: "that should never happen … i should have the option to have it never go down. imagine healthcare."

Today a roll on the 8 GB droplet goes dark because a second forge pod cannot be scheduled (memory requests at 97%, the pod at 3.5 to 4 GiB). Two dark rolls happened on 2026-10-05/06: the v0.7.52 release (3m33s) and a settings-only roll for two concurrency values (about 3 minutes).

Done means: a release or a settings change never returns 503 to anyone. The pieces, each its own issue blocked by this one:

  1. Headroom for two forge pods during a roll (infrastructure: a larger droplet or a second node; the owner decides). The release agent refuses a roll that would go dark unless the owner said yes to that roll.
  2. Capacity settings change live: crew concurrency, check concurrency, check shards, cells egress mode, read from the settings store on each use or on a signal, with the admin page and MCP as the door, audited. A restart is never needed for a number.
  3. Two forge processes overlap safely on one data dir during a roll: the drain contract (the old process finishes what it holds, the new one takes new work), SQLite WAL and busy timeouts, the reconciler and the warden idempotent across two processes, and expand/contract migrations: a migration the old process can keep serving under (add before remove; rename in two releases).
  4. A rehearsal that proves it: the release checklist rolls a copy with two pods overlapping and shows zero refused requests.

Pillar: trust first (#586).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs