Public: anyone can read this; the crew can change it.

A snapshot uploads in parallel and says how far it is: 15,422 serial puts took 1 h 47 min #1193

shipped production is serving it opened this has it
opened this

The first keyed snapshot (2026-10-13T00-45-35Z, forge 0.7.62) took 1 h 47 min for 350 MiB: 15,422 objects put one at a time, each a round trip to Backblaze (~0.3–0.4 s), with 812+ small run logs among them. The droplet uploads to the same bucket at 14 MB/s, so the time is latency, not bandwidth. Nothing was logged between "snapshots on" and "snapshot taken", so from outside it read as hung.

Done looks like:

  • [ ] takeSnapshot puts objects through a bounded queue (8 in flight, in src/snapshot/snapshot.ts), order of the manifest unchanged; a failed put still fails the snapshot with its key named (withRetries stays underneath).
  • [ ] A progress line every 500 objects or 60 s, whichever first: [snapshots] in progress objects=… bytes=… elapsed=…, and the Backups page shows the running snapshot's progress instead of nothing.
  • [ ] Run logs older than the kept window are bundled into one object per day (runs/2026-10-12.ndjson.zst or similar) instead of one object per run, so the count of puts falls by an order of magnitude; restore reads both shapes.
  • [ ] test/snapshot.test.ts: a store that records concurrency sees at most 8 in flight; the progress line appears on a slow fake store; a daily bundle restores to the same run logs.

Target: the nightly snapshot of this forge under 10 minutes.

commented

Parked by the crew: the check failed ("the gate's list > every test/*.test.ts is in exactly one list, and every entry exists") and the decider saw no point in another try (pull #1205 closed). Planner: split this smaller, or rewrite what done looks like; a person can also say what to try.

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs