Skip to content

Add bounded multipart upload parallelism#228

Open
jiuker wants to merge 2 commits into
minio:mainfrom
jiuker:fix-issue-216
Open

Add bounded multipart upload parallelism#228
jiuker wants to merge 2 commits into
minio:mainfrom
jiuker:fix-issue-216

Conversation

@jiuker

@jiuker jiuker commented Jun 5, 2026

Copy link
Copy Markdown

fix: #216

Add bounded multipart upload parallelism

PutObject now supports concurrent UploadPart calls via the new PutObjectArgs::max_inflight_parts field (default 1 = sequential behavior preserved).

Changes

File Change
include/miniocpp/args.h Added unsigned int max_inflight_parts = 1 to PutObjectArgs
src/args.cc Validate max_inflight_parts > 0
src/client.cc Refactored PutObject into sequential + parallel branches
tests/tests.cc Added PutObjectWithInflight() — MD5 integrity test

src/client.cc highlights

  • Sequential path (max_inflight_parts <= 1): Original single-buffer loop, unchanged.
  • Parallel path (max_inflight_parts > 1):
    • N page-aligned buffers allocated via AlignedBuffer pool.
    • Producer/consumer model: main thread reads stream, std::async dispatches UploadPart.
    • Bounded inflight cap via std::deque + drain_one lambda.
    • FIFO ordering preserved by part number — CompleteMultipartUpload doesn't care about completion order.
  • Bug fix: Corrected one_byte read-ahead indexing in the unknown-size path (buf[part_size] instead of buf[part_size + 1]).

Testing

PutObjectWithInflight() generates 10MB of random data, computes the original MD5, uploads with max_inflight_parts = 1, 2, 4, downloads each object, and verifies MD5 matches the original.

Add bounded multipart upload parallelism
@jiuker jiuker requested a review from harshavardhana June 5, 2026 09:44
lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: parallel multipart UploadPart

1 participant