Public: anyone can read this; the crew can change it.

Fuzz the parsers and chaos the forge on a side instance: invariants proven by faults, not by hope #1244

shipped production is serving it opened this has it
opened this

The owner: "something like a fuzzer or chaos monkey might help."

The forge's invariants are mechanical claims (runs resume after a restart, a pull is never lost, a snapshot restores, a sandbox holds) and today they are proven only by the tests a person thought of. Twice this morning the forge was restarted under load (OOM at 04:02Z, liveness at 05:22Z) and the proof that nothing was lost was a person reading the run table.

Done looks like:

  • [ ] Fuzz, seeded and reproducible, in test/fuzz-*.test.ts: the parsers that take outside input (forge.json manifest, contract.json, hostnames and slugs, markdown, the diff reader, the API's JSON bodies) take a generator's cases; a failure prints the seed and the minimal case; a found case becomes a fixed test.
  • [ ] Chaos on a side instance, nightly: boot a forge on a restored snapshot, start a few fake runs and checks, then kill the process mid-run, drop the SQLite connection, fill the temp disk, slow the loopback; after each, the invariants are asserted (every run is queued/running/finished and none vanished, every open pull still has its head, a snapshot taken after restores, no sealed value readable). src/chaos/ holds the faults; test/chaos.test.ts runs a small version in the gate.
  • [ ] Results are issues: a broken invariant files one with the fault, the seed and the log, labelled P0 Trust; a quiet night posts one line to the recovery issue.
  • [ ] Nothing of this runs against the live forge or a customer's app.

Evidence: 2026-10-13 restarts and the by-hand checks that followed (#1221, #1235).

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs