Skip to content

Fix transfermanager multipart uploads for seekable bodies - #3468

Open
goutamadwant wants to merge 1 commit into
aws:mainfrom
goutamadwant:fix-transfermanager-seekable-upload
Open

Fix transfermanager multipart uploads for seekable bodies#3468
goutamadwant wants to merge 1 commit into
aws:mainfrom
goutamadwant:fix-transfermanager-seekable-upload

Conversation

@goutamadwant

Copy link
Copy Markdown

Fixes #3453

This updates UploadObject so multipart uploads for seekable bodies use section readers for each part instead of copying every part through the transfermanager buffer pool.

Seekable bodies are detected when the input implements both io.ReaderAt and io.Seeker. In that path, each multipart part is served with io.NewSectionReader, including when the body starts from a non-zero current offset. The existing buffered path is still used for non-seekable readers.

Validation:

  • Added regression coverage for seekable multipart uploads using section readers and preserving part contents from a non-zero start offset.
  • go test . -run TestUploadOrderMultiSeekableBodyUsesSectionReaders -count=1
  • go test ./... -count=1

Changelog

No changelog entry added. Maintainers will handle the changelog entry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature/s3/transfermanager: UploadObject buffers seekable bodies (*os.File) instead of streaming — peak memory regression vs feature/s3/manager

1 participant