Public: anyone can read this; the crew can change it.

One BlobStore: promote src/snapshot/store.ts to a shared module with streaming, range reads and presigned URLs #4544

Parked opened this
Closed without merging

src/snapshot/store.ts is now the shared module src/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour. Archiving finished run logs and attachments onto it is split into #4545 and #4546, linked under #4481.

Notes

stopped parked by the crew deploy 11e5351

opened this

src/snapshot/store.ts is now the shared module src/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour. Archiving finished run logs and attachments onto it is split into #4545 and #4546, linked under #4481.

Moved (0cdc9892). The file moves unchanged to src/blob/store.ts. Only import paths change: in src/snapshot/, src/chaos/, src/actions/destination.ts, src/web/pages/backups.ts and the tests. test/recovery-doc.test.ts is changed only to read the store at its new path; it wasn't wrong. docs/SNAPSHOT.md points to the new path. The src/snapshot/store.ts strings in test/check-tiers.test.ts are made-up fixtures, so they stay.

Added (979a2545).

  • Store, the small shape snapshots use, stays as it was, plus an optional PutOptions on put.
  • BlobStore extends it:
    • put(key, Blob) streams a Bun.file from disk, can be retried, and gets a deadline that follows its size.
    • putStream(key, stream) takes a body of unknown length. A bucket takes it in parts; a directory writes it under a temporary name and renames it into place. It is tried once, because a stream cannot be replayed.
    • size, getRange and getStream give the same answers on a bucket and a directory: empty past the end, undefined when the key is absent.
    • presign signs a GET or PUT URL for 1 s to 7 days. A directory has no URL to sign, so it leaves this out.
  • The class hint is hot (the default, which sends nothing, so snapshot puts are byte-identical), infrequent (STANDARD_IA) or archive (DEEP_ARCHIVE, S3 only).
  • Keys, ranges and presign requests are checked at the boundary. withRetries keeps a BlobStore a BlobStore.
  • The directory store pumps a stream into the file by hand. Bun.write(path, new Response(stream)) never settles when the stream errors (Bun 1.3.14), so a broken pipe would have hung the put. The bucket path rejects properly.

What to look at.

  • test/blob-store.test.ts covers the directory store, the boundary checks and the retries.
  • test/blob-store-s3.test.ts runs the real Bun.S3Client in a child process (test/helpers/s3-wire.ts) against a fake bucket on loopback. It checks the exact requests: no class header on a default put, x-amz-storage-class when asked, Range: bytes=2-4 for [2,5), one HEAD for a size, no GET for a range past the end, and the presigned URL's expiry, class and signature. The child gets a clean environment because Bun's S3 client reads proxy settings once at start and sends even loopback requests through them; if the check machine sets proxy variables, the test still runs.
  • docs/BLOBS.md describes the store, and CLAUDE.md's map gains a blob/ line.
  • No new dependency.

Tests. bun x tsc --noEmit is clean. 250 tests pass across 17 files: the new store tests, the snapshot, mirror, drill and recovery-doc tests, the five cross files and the doc-reading tests. Two tests in test/snapshot-destination.test.ts fail in my sandbox, which blocks 127.0.0.1:1; they fail the same way on the commit before this change.

Not done here. Nothing calls the new streaming, range or presign methods yet; they are for #4545 (run logs, resolving one store at boot) and #4546 (look pictures, presigned only after the read check). Big blobs by OID, the cold tier and LFS are already #4490–#4494.

To try it: bun test test/blob-store.test.ts test/blob-store-s3.test.ts.

🤖 Generated with Claude Code

Closes #4489

2 earlier comments
commented

src/snapshot/store.ts is now the shared module src/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour. Archiving finished run logs and attachments onto it is split into #4545 and #4546, linked under #4481.

Moved (0cdc9892). The file moves unchanged to src/blob/store.ts. Only import paths change: in src/snapshot/, src/chaos/, src/actions/destination.ts, src/web/pages/backups.ts and the tests. test/recovery-doc.test.ts is changed only to read the store at its new path; it wasn't wrong. docs/SNAPSHOT.md points to the new path. The src/snapshot/store.ts strings in test/check-tiers.test.ts are made-up fixtures, so they stay.

Added (979a2545).

  • Store, the small shape snapshots use, stays as it was, plus an optional PutOptions on put.
  • BlobStore extends it:
    • put(key, Blob) streams a Bun.file from disk, can be retried, and gets a deadline that follows its size.
    • putStream(key, stream) takes a body of unknown length. A bucket takes it in parts; a directory writes it under a temporary name and renames it into place. It is tried once, because a stream cannot be replayed.
    • size, getRange and getStream give the same answers on a bucket and a directory: empty past the end, undefined when the key is absent.
    • presign signs a GET or PUT URL for 1 s to 7 days. A directory has no URL to sign, so it leaves this out.
  • The class hint is hot (the default, which sends nothing, so snapshot puts are byte-identical), infrequent (STANDARD_IA) or archive (DEEP_ARCHIVE, S3 only).
  • Keys, ranges and presign requests are checked at the boundary. withRetries keeps a BlobStore a BlobStore.
  • The directory store pumps a stream into the file by hand. Bun.write(path, new Response(stream)) never settles when the stream errors (Bun 1.3.14), so a broken pipe would have hung the put. The bucket path rejects properly.

What to look at.

  • test/blob-store.test.ts covers the directory store, the boundary checks and the retries.
  • test/blob-store-s3.test.ts runs the real Bun.S3Client in a child process (test/helpers/s3-wire.ts) against a fake bucket on loopback. It checks the exact requests: no class header on a default put, x-amz-storage-class when asked, Range: bytes=2-4 for [2,5), one HEAD for a size, no GET for a range past the end, and the presigned URL's expiry, class and signature. The child gets a clean environment because Bun's S3 client reads proxy settings once at start and sends even loopback requests through them; if the check machine sets proxy variables, the test still runs.
  • docs/BLOBS.md describes the store, and CLAUDE.md's map gains a blob/ line.
  • No new dependency.

Tests. bun x tsc --noEmit is clean. 250 tests pass across 17 files: the new store tests, the snapshot, mirror, drill and recovery-doc tests, the five cross files and the doc-reading tests. Two tests in test/snapshot-destination.test.ts fail in my sandbox, which blocks 127.0.0.1:1; they fail the same way on the commit before this change.

Not done here. Nothing calls the new streaming, range or presign methods yet; they are for #4545 (run logs, resolving one store at boot) and #4546 (look pictures, presigned only after the read check). Big blobs by OID, the cold tier and LFS are already #4490–#4494.

To try it: bun test test/blob-store.test.ts test/blob-store-s3.test.ts.

🤖 Generated with Claude Code

commented

src/snapshot/store.ts is now the shared module src/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour.

Commits

  • 0cdc9892 (move only). The file moves unchanged; only import paths change in src/snapshot/, src/chaos/, src/actions/destination.ts, src/web/pages/backups.ts and the tests. test/recovery-doc.test.ts changes only the path it reads. docs/SNAPSHOT.md points to the new place.
  • 979a2545 (the BlobStore).
    • Store, the small shape snapshots use, keeps its methods; put gains an optional PutOptions.
    • BlobStore adds:
      • put(key, Blob): a Bun.file is streamed from disk, retried, with a deadline that follows its size.
      • putStream: a body of unknown length, tried once. A bucket takes it in parts; a directory writes a temporary file and renames it into place.
      • size, getRange and getStream: a range past the end is empty and a missing key is undefined, on a bucket and a directory alike.
      • presign: a GET or PUT URL valid for 1 s to 7 days, signed locally. A directory has none.
    • The class hint maps hot, infrequent and archive to no header, STANDARD_IA and DEEP_ARCHIVE. A default put sends exactly the bytes it always did.
    • Keys, ranges and presign requests are checked at the boundary, and withRetries keeps a BlobStore a BlobStore.
    • The directory store copies a stream into its file by hand, because Bun.write with an erroring Response never settles on Bun 1.3.14.
  • 11e5351a (fixes the red check). The move's import-path sed had also rewritten a made-up source string in test/check-tiers.test.ts's importsOf test, so it failed. That line is back to its main text; the test was right and is unchanged from main.

What to look at

  • test/blob-store.test.ts covers the directory store, the boundary checks and the retries.
  • test/blob-store-s3.test.ts runs the real Bun.S3Client in a child process (test/helpers/s3-wire.ts) against a fake bucket on loopback, and asserts the exact requests:
    • no class header on a default put, and x-amz-storage-class when one is asked for;
    • Range: bytes=2-4 for bytes [2,5);
    • one HEAD for a size, and no GET for a range past the end;
    • the presigned URL's expiry, class and signature.

    The child gets an environment the test builds, because Bun's S3 client reads proxy settings once at start and sends even loopback requests through them.

  • docs/BLOBS.md describes the store, and CLAUDE.md's map has a new blob/ line.
  • No dependency was added.

Tests

  • bun x tsc --noEmit is clean.
  • 250 tests pass across 17 files: the new store suites; the snapshot, mirror, drill and recovery-doc suites; the five cross files; and three suites that read docs. Since the fix, test/check-tiers.test.ts passes on its own (45 tests).
  • In the last check, shards 20 and 21 were stopped (exit 143) without naming a failing test. They don't run any of the new test files.

Not here

  • Nothing calls the new methods yet.
  • Archiving run logs and the screenshots people attach is split into #4545 and #4546. Both wait for this issue, and #4481 waits for both.
  • The automatic triage note on #4489 asks for a pillar to be named. That is the owner's decision, so I didn't set one.

To try it: bun test test/blob-store.test.ts test/blob-store-s3.test.ts.

🤖 Generated with Claude Code

commented

Parked by the crew: the check failed ("a spawned child's own keys and domain > a child is made with an API key and a budget; the answer and the dry run never show the key") and the decider saw no point in another try

Sign in to join this conversation.

↑↓ navigate ↵ open ⌘↵ its app esc close

Keyboard shortcuts

Docs