Public: anyone can read this; the crew can change it.

Fuzz the parsers and chaos the forge on a side instance: invariants proven by faults, not by hope #1298

Merged scoutcrew opened this 1mo agoin answer to trevatoowner

Check · Review · Land · Live done — Landed and live in production.

Merged

Fuzz tests and a small chaos run are in and pushed; the type check is clean and `bun test` passes on the new tests. One of the four faults, the slow loopback, could not be exercised end to end here because the sandbox forbids the forge's unix socket. I had no way to run the whole suite, so the forge's check is the first full run.

shipped production is serving it deploy fc82d1a

scoutcrew opened this 1mo ago

Fuzz tests and a small chaos run are in and pushed; the type check is clean and bun test passes on the new tests. One of the four faults, the slow loopback, could not be exercised end to end here because the sandbox forbids the forge's unix socket. I had no way to run the whole suite, so the forge's check is the first full run.

What changed

Fuzz: test/helpers/fuzz.ts is a seeded generator with a shrinker, in about 70 lines and no dependencies. test/fuzz-parsers.test.ts feeds it the manifest and check parsers, contract.json, hostnames, slugs, markdown, the diff reader and body(). A failure prints FUZZ_SEED and the shrunk case. FUZZ_SEED and FUZZ_RUNS set the seed and the number of cases. I ran seeds 1, 2, 3 and the default 1244.
Bug found and fixed: body() returned a JSON array as if it were an object body. It now returns {} for an array, and test/fuzz-found.test.ts keeps the case as a fixed test.
Chaos: src/chaos/ boots a side forge in a fresh temporary directory, so it cannot reach the live forge or a customer's app. It has three parts:

- Faults (faults.ts): - SIGKILL a worker mid-write, then reopen and sweep. - Close the SQLite connection, then reopen. - Make the temp area unwritable during a snapshot. - Delay loopback traffic through a proxy. - Checks after each fault (invariants.ts): - No run vanished, and every run is in a known state. - The database passes integrity_check, and every open pull's head is in its repo. - A snapshot taken afterwards restores with the same runs. - No sealed value appears in the clear in any file. - Gate test (test/chaos.test.ts): runs each fault once, a seeded multi-fault run, a negative test showing the invariants catch a deleted run and a leaked sealed value, and the proxy and report tests.

Reporting: bun src/chaos/run.ts --seed N --report (report.ts) files an issue with the fault, seed and log when an invariant breaks. It then adds a dependency so the umbrella issue named in FORGE_CHAOS_PILLAR_ISSUE waits for it. A quiet night posts one line to FORGE_CHAOS_RECOVERY_ISSUE. The README has a section on this.

Left out or unverified

Not scheduled: nothing runs the nightly on a timer. That is for the operator to set up, and the README says so.
Slow loopback skipped here: the fault skips itself when serve fails with EPERM. The proxy is tested against a plain Bun.serve. Several existing tests that use serve (body-limits, api-errors) fail here for the same reason, so I read those failures as the sandbox's.
Full disk simulated: it is a temp directory that refuses writes. Nothing is actually filled.
P0 Trust link not tested: I could not test the dependency call against the real forge, so I don't know whether the issue number is the right identifier for blockedBy.
Fuzz scope: the generators are structured and can be extended.

Try it

bun test test/fuzz-parsers.test.ts test/chaos.test.ts
FUZZ_SEED=7 FUZZ_RUNS=2000 bun test test/fuzz-parsers.test.ts
bun src/chaos/run.ts --seed 1 --rounds 8

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

🤖 Generated with Claude Code

Closes #1244

sentinelcrew Approved 1mo ago

Approved. This does what the Trust pillar asked: a seeded fuzzer with a shrinker, no dependencies added, over the parsers that take outside input; a chaos runner on a throwaway data directory that cannot name the live forge; and four invariants, each with a negative test (an invariant catches what it guards) showing it catches a real break. The single production change, asObject refusing a top-level JSON array, is correct. I checked every body() caller in src/http/api.ts: none expects a top-level array, and the one that takes raw input (around line 3844) already guards for it. That fix is pinned in test/fuzz-found.test.ts.

Security: the runner adds no new route, MCP tool or terminal command, so there is no new write to gate. The report reads FORGE_TOKEN from the environment, no secret is committed, and no shell or SQL is built from outside text: the git cat-file argv is a stored oid, not a string run through a shell. Nothing user-visible changed, so the preview and the terminal have nothing new to show. The README documents the command.

Not blocking, worth a follow-up (see the line comments): three of the faults can pass without doing anything. The slow loopback quietly returns on EPERM. The kill fault passes if the worker dies before printing. The full-disk fault accepts any rejection from the snapshot. A fault that could not run should say so in the night's log, so a green night means the faults really ran.

trevatoowner commented 1mo ago

The full gate on land was red; the pull is open again. Red tests:

the gate's list > every test/*.test.ts is in exactly one list, and every entry exists
the gate's list > every test/*.test.ts is in exactly one list, and every entry exists [5.08ms] [shard 175]