Public: anyone can read this; the crew can change it.
One BlobStore: promote src/snapshot/store.ts to a shared module with streaming, range reads and presigned URLs #4544
src/snapshot/store.ts is now the shared module src/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour. Archiving finished run logs and attachments onto it is split into #4545 and #4546, linked under #4481.
stopped parked by the crew deploy 11e5351
2 earlier comments
src/snapshot/store.ts is now the shared module src/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour. Archiving finished run logs and attachments onto it is split into #4545 and #4546, linked under #4481.
Moved (0cdc9892). The file moves unchanged to src/blob/store.ts. Only import paths change: in src/snapshot/, src/chaos/, src/actions/destination.ts, src/web/pages/backups.ts and the tests. test/recovery-doc.test.ts is changed only to read the store at its new path; it wasn't wrong. docs/SNAPSHOT.md points to the new path. The src/snapshot/store.ts strings in test/check-tiers.test.ts are made-up fixtures, so they stay.
Added (979a2545).
Store, the small shape snapshots use, stays as it was, plus an optionalPutOptionsonput.BlobStoreextends it:put(key, Blob)streams aBun.filefrom disk, can be retried, and gets a deadline that follows its size.putStream(key, stream)takes a body of unknown length. A bucket takes it in parts; a directory writes it under a temporary name and renames it into place. It is tried once, because a stream cannot be replayed.size,getRangeandgetStreamgive the same answers on a bucket and a directory: empty past the end,undefinedwhen the key is absent.presignsigns a GET or PUT URL for 1 s to 7 days. A directory has no URL to sign, so it leaves this out.
- The class hint is
hot(the default, which sends nothing, so snapshot puts are byte-identical),infrequent(STANDARD_IA) orarchive(DEEP_ARCHIVE, S3 only). - Keys, ranges and presign requests are checked at the boundary.
withRetrieskeeps aBlobStoreaBlobStore. - The directory store pumps a stream into the file by hand.
Bun.write(path, new Response(stream))never settles when the stream errors (Bun 1.3.14), so a broken pipe would have hung the put. The bucket path rejects properly.
What to look at.
test/blob-store.test.tscovers the directory store, the boundary checks and the retries.test/blob-store-s3.test.tsruns the realBun.S3Clientin a child process (test/helpers/s3-wire.ts) against a fake bucket on loopback. It checks the exact requests: no class header on a default put,x-amz-storage-classwhen asked,Range: bytes=2-4for[2,5), one HEAD for a size, no GET for a range past the end, and the presigned URL's expiry, class and signature. The child gets a clean environment because Bun's S3 client reads proxy settings once at start and sends even loopback requests through them; if the check machine sets proxy variables, the test still runs.docs/BLOBS.mddescribes the store, andCLAUDE.md's map gains ablob/line.- No new dependency.
Tests. bun x tsc --noEmit is clean. 250 tests pass across 17 files: the new store tests, the snapshot, mirror, drill and recovery-doc tests, the five cross files and the doc-reading tests. Two tests in test/snapshot-destination.test.ts fail in my sandbox, which blocks 127.0.0.1:1; they fail the same way on the commit before this change.
Not done here. Nothing calls the new streaming, range or presign methods yet; they are for #4545 (run logs, resolving one store at boot) and #4546 (look pictures, presigned only after the read check). Big blobs by OID, the cold tier and LFS are already #4490–#4494.
To try it: bun test test/blob-store.test.ts test/blob-store-s3.test.ts.
🤖 Generated with Claude Code
src/snapshot/store.ts is now the shared module src/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour.
Commits
0cdc9892(move only). The file moves unchanged; only import paths change insrc/snapshot/,src/chaos/,src/actions/destination.ts,src/web/pages/backups.tsand the tests.test/recovery-doc.test.tschanges only the path it reads.docs/SNAPSHOT.mdpoints to the new place.979a2545(theBlobStore).Store, the small shape snapshots use, keeps its methods;putgains an optionalPutOptions.BlobStoreadds:put(key, Blob): aBun.fileis streamed from disk, retried, with a deadline that follows its size.putStream: a body of unknown length, tried once. A bucket takes it in parts; a directory writes a temporary file and renames it into place.size,getRangeandgetStream: a range past the end is empty and a missing key isundefined, on a bucket and a directory alike.presign: a GET or PUT URL valid for 1 s to 7 days, signed locally. A directory has none.
- The class hint maps
hot,infrequentandarchiveto no header,STANDARD_IAandDEEP_ARCHIVE. A default put sends exactly the bytes it always did. - Keys, ranges and presign requests are checked at the boundary, and
withRetrieskeeps aBlobStoreaBlobStore. - The directory store copies a stream into its file by hand, because
Bun.writewith an erroringResponsenever settles on Bun 1.3.14.
11e5351a(fixes the red check). The move's import-pathsedhad also rewritten a made-up source string intest/check-tiers.test.ts'simportsOftest, so it failed. That line is back to itsmaintext; the test was right and is unchanged frommain.
What to look at
test/blob-store.test.tscovers the directory store, the boundary checks and the retries.test/blob-store-s3.test.tsruns the realBun.S3Clientin a child process (test/helpers/s3-wire.ts) against a fake bucket on loopback, and asserts the exact requests:- no class header on a default put, and
x-amz-storage-classwhen one is asked for; Range: bytes=2-4for bytes[2,5);- one HEAD for a size, and no GET for a range past the end;
- the presigned URL's expiry, class and signature.
The child gets an environment the test builds, because Bun's S3 client reads proxy settings once at start and sends even loopback requests through them.
- no class header on a default put, and
docs/BLOBS.mddescribes the store, andCLAUDE.md's map has a newblob/line.- No dependency was added.
Tests
bun x tsc --noEmitis clean.- 250 tests pass across 17 files: the new store suites; the snapshot, mirror, drill and recovery-doc suites; the five cross files; and three suites that read docs. Since the fix,
test/check-tiers.test.tspasses on its own (45 tests). - In the last check, shards 20 and 21 were stopped (exit 143) without naming a failing test. They don't run any of the new test files.
Not here
- Nothing calls the new methods yet.
- Archiving run logs and the screenshots people attach is split into #4545 and #4546. Both wait for this issue, and #4481 waits for both.
- The automatic triage note on #4489 asks for a pillar to be named. That is the owner's decision, so I didn't set one.
To try it: bun test test/blob-store.test.ts test/blob-store-s3.test.ts.
🤖 Generated with Claude Code
Parked by the crew: the check failed ("a spawned child's own keys and domain > a child is made with an API key and a budget; the answer and the dry run never show the key") and the decider saw no point in another try
src/snapshot/store.tsis now the shared modulesrc/blob/store.ts, with streamed puts, byte ranges, presigned URLs and a storage-class hint. Snapshots use it with no change in behaviour. Archiving finished run logs and attachments onto it is split into #4545 and #4546, linked under #4481.Moved (
0cdc9892). The file moves unchanged tosrc/blob/store.ts. Only import paths change: insrc/snapshot/,src/chaos/,src/actions/destination.ts,src/web/pages/backups.tsand the tests.test/recovery-doc.test.tsis changed only to read the store at its new path; it wasn't wrong.docs/SNAPSHOT.mdpoints to the new path. Thesrc/snapshot/store.tsstrings intest/check-tiers.test.tsare made-up fixtures, so they stay.Added (
979a2545).Store, the small shape snapshots use, stays as it was, plus an optionalPutOptionsonput.BlobStoreextends it:put(key, Blob)streams aBun.filefrom disk, can be retried, and gets a deadline that follows its size.putStream(key, stream)takes a body of unknown length. A bucket takes it in parts; a directory writes it under a temporary name and renames it into place. It is tried once, because a stream cannot be replayed.size,getRangeandgetStreamgive the same answers on a bucket and a directory: empty past the end,undefinedwhen the key is absent.presignsigns a GET or PUT URL for 1 s to 7 days. A directory has no URL to sign, so it leaves this out.hot(the default, which sends nothing, so snapshot puts are byte-identical),infrequent(STANDARD_IA) orarchive(DEEP_ARCHIVE, S3 only).withRetrieskeeps aBlobStoreaBlobStore.Bun.write(path, new Response(stream))never settles when the stream errors (Bun 1.3.14), so a broken pipe would have hung the put. The bucket path rejects properly.What to look at.
test/blob-store.test.tscovers the directory store, the boundary checks and the retries.test/blob-store-s3.test.tsruns the realBun.S3Clientin a child process (test/helpers/s3-wire.ts) against a fake bucket on loopback. It checks the exact requests: no class header on a default put,x-amz-storage-classwhen asked,Range: bytes=2-4for[2,5), one HEAD for a size, no GET for a range past the end, and the presigned URL's expiry, class and signature. The child gets a clean environment because Bun's S3 client reads proxy settings once at start and sends even loopback requests through them; if the check machine sets proxy variables, the test still runs.docs/BLOBS.mddescribes the store, andCLAUDE.md's map gains ablob/line.Tests.
bun x tsc --noEmitis clean. 250 tests pass across 17 files: the new store tests, the snapshot, mirror, drill and recovery-doc tests, the five cross files and the doc-reading tests. Two tests intest/snapshot-destination.test.tsfail in my sandbox, which blocks127.0.0.1:1; they fail the same way on the commit before this change.Not done here. Nothing calls the new streaming, range or presign methods yet; they are for #4545 (run logs, resolving one store at boot) and #4546 (look pictures, presigned only after the read check). Big blobs by OID, the cold tier and LFS are already #4490–#4494.
To try it:
bun test test/blob-store.test.ts test/blob-store-s3.test.ts.🤖 Generated with Claude Code
Closes #4489