Add cross-provider ranged, parallel downloads#340
Merged
Conversation
BucketObject.download_to_file fetches objects larger than the configured threshold as ranged reads of part_size bytes, up to max_concurrency parts in parallel, so large downloads are not bound to a single connection and the whole object is never held in memory. AWS delegates to boto3's TransferManager and Azure to azure-storage-blob's concurrent downloader; GCP and OpenStack Swift use the generic driver, which fetches ranges in parallel via cloned providers (memory bounded to ~concurrency * part_size; the partial file is removed on failure). A new BucketObjectService.download_range primitive is also usable directly for partial reads. Since the same threshold/part-size/concurrency knobs now tune transfers in both directions, UploadConfig (shipped only in 4.2.x, unused downstream) is renamed to TransferConfig with no alias kept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Downloads get the same treatment uploads got in #333.
BucketObject.download_to_file(path, config)fetches objects larger than the configured threshold as ranged reads ofpart_sizebytes, up tomax_concurrencyparts in parallel — large downloads are no longer bound to a single connection, and the whole object is never held in memory (notably fixing GCP, whoseiter_contentbuffered the entire object).download_filedriven by CloudBridge's knobs)download_blob(max_concurrency=…))get_mediareads)The generic driver mirrors the upload driver: parts fetched across a bounded thread pool of cloned providers (no shared SDK state), each worker writing at its offset into a preallocated file, memory bounded to ~
concurrency × part_size, and the partial file removed on any failure. A newBucketObjectService.download_range(bucket, object_name, offset, length)primitive backs the driver and is usable directly for partial reads.UploadConfig→TransferConfig: the same threshold/part-size/concurrency knobs now tune both directions (pass a different instance per call to tune each direction separately, mirroring boto3's singleTransferConfig).UploadConfigonly shipped in 4.2.x and has no downstream users, so it is renamed outright with no alias.Tests
tests/test_download_driver.py: in-memory-fake suite for the generic driver (exact range tiling with short tail, single-shot below threshold, clone-per-worker isolation + real overlap, per-call config precedence, partial-file cleanup on failure, part-size validation)tests/test_object_store_service.py: small roundtrip, large ranged roundtrip with a position-sensitive byte pattern (all providers), and boto3TransferConfigwiring assertions (AWS/mock)Verification
tox -e py3.13-mock: full suite greentox -e mypy: clean (interface strict bar enforces the new signatures on all providers)tox -e lint: cleanLive provider runs via the
safe-to-testlabel will exercise the ranged roundtrip against real Azure/GCP/Swift.