fix(lib-storage): error early when full-object Checksum* is supplied with multipart Upload (#6742) - #7990
Open
mohanrajvenkatesan23-04 wants to merge 1 commit into
Conversation
…with multipart Upload (aws#6742) When a caller passes a precomputed full-object checksum (ChecksumSHA256, ChecksumSHA1, ChecksumCRC32, ChecksumCRC32C, ChecksumCRC64NVME) to lib-storage's Upload for a body larger than partSize, Upload performs a multipart upload. S3 then computes a composite per-part checksum and rejects the request with a confusing BadDigest error after the full network roundtrip. Add a fail-fast client-side validation in __createMultipartUpload() that throws a clear Error naming the offending param, explaining why it cannot work with multipart, and listing the supported workarounds (use ChecksumAlgorithm, or call PutObject directly). ChecksumAlgorithm is intentionally NOT rejected since it is the supported way to use checksums on multipart uploads. The single-part PutObject path is unaffected. Refs aws#6742
6 tasks
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Fixes #6742 —
[lib-storage] Upload fails if application provides checksum for >5 MB file.Description
When a caller passes a precomputed full-object checksum (
ChecksumSHA256,ChecksumSHA1,ChecksumCRC32,ChecksumCRC32C,ChecksumCRC64NVME) intolib-storage'sUploadfor a body that exceedspartSize,Uploadperforms a multipart upload. S3 then computes a composite per-part checksum and rejects the upload with a confusingBadDigesterror only after the full network roundtrip — which is especially painful for large objects.The proper service-side fix needs the new
x-amz-checksum-typeheader (tracked upstream by the S3 service team). Until that lands, this PR is a fail-fast client-side improvement: it converts the confusing post-upload server error into a clear pre-upload client error.What changed
__validateChecksumForMultipart()onUpload(lib/lib-storage/src/Upload.ts) is invoked as the first statement of__createMultipartUpload()— so it only fires on the multipart code path. Single-partPutObjectcallers are unaffected.Checksum*params is present, it throws anErrorthat:ChecksumAlgorithminstead, or callPutObjectdirectly),ChecksumAlgorithmis intentionally not rejected — it is the supported way to use checksums on multipart uploads.This is purely additive. No currently-working configuration is altered.
Testing
Added 4 tests in
lib/lib-storage/src/Upload.spec.ts:ChecksumSHA256rejects with a message matching/ChecksumSHA256/and/issues\/6742/, and the mocked S3 client never receivesCreateMultipartUploadCommand/UploadPartCommand/CompleteMultipartUploadCommand.it.each) — same rejection check forChecksumSHA1,ChecksumCRC32,ChecksumCRC32C,ChecksumCRC64NVME.ChecksumAlgorithm: \"SHA256\"succeeds — happy path unchanged.ChecksumSHA256is unaffected (validation does not fire on thePutObjectpath).Local test status (transparency)
tsc --noEmitpassed for the changed files; no new TypeScript errors.dist-*/build outputs (@aws-sdk/client-s3cannot be resolved from a fresh clone). CI on this PR will validate test execution.Checklist
lib/lib-storage/src/Upload.tsandlib/lib-storage/src/Upload.spec.tsare touched.Fixes #6742).Checksum*params) and positive (ChecksumAlgorithm, single-partPutObject) cases.generated by AI tools, and reviewed by Mohanraj Venkatesan