Skip to content

fix(lib-storage): error early when full-object Checksum* is supplied with multipart Upload (#6742) - #7990

Open
mohanrajvenkatesan23-04 wants to merge 1 commit into
aws:mainfrom
mohanrajvenkatesan23-04:fix/issue-6742-lib-storage-checksum-multipart-validation
Open

fix(lib-storage): error early when full-object Checksum* is supplied with multipart Upload (#6742)#7990
mohanrajvenkatesan23-04 wants to merge 1 commit into
aws:mainfrom
mohanrajvenkatesan23-04:fix/issue-6742-lib-storage-checksum-multipart-validation

Conversation

@mohanrajvenkatesan23-04

Copy link
Copy Markdown

Issue

Fixes #6742[lib-storage] Upload fails if application provides checksum for >5 MB file.

Description

When a caller passes a precomputed full-object checksum (ChecksumSHA256, ChecksumSHA1, ChecksumCRC32, ChecksumCRC32C, ChecksumCRC64NVME) into lib-storage's Upload for a body that exceeds partSize, Upload performs a multipart upload. S3 then computes a composite per-part checksum and rejects the upload with a confusing BadDigest error only after the full network roundtrip — which is especially painful for large objects.

The proper service-side fix needs the new x-amz-checksum-type header (tracked upstream by the S3 service team). Until that lands, this PR is a fail-fast client-side improvement: it converts the confusing post-upload server error into a clear pre-upload client error.

What changed

  • New private __validateChecksumForMultipart() on Upload (lib/lib-storage/src/Upload.ts) is invoked as the first statement of __createMultipartUpload() — so it only fires on the multipart code path. Single-part PutObject callers are unaffected.
  • When any of the five full-object Checksum* params is present, it throws an Error that:
    • names the offending param,
    • explains why it cannot work with multipart,
    • lists the two supported workarounds (use ChecksumAlgorithm instead, or call PutObject directly),
    • links the issue.
  • ChecksumAlgorithm is intentionally not rejected — it is the supported way to use checksums on multipart uploads.

This is purely additive. No currently-working configuration is altered.

Note: the proper fix is blocked on the S3 service-side x-amz-checksum-type header support. This PR is a fail-fast client-side guard until that lands.

Testing

Added 4 tests in lib/lib-storage/src/Upload.spec.ts:

  1. Negative — multipart upload with ChecksumSHA256 rejects with a message matching /ChecksumSHA256/ and /issues\/6742/, and the mocked S3 client never receives CreateMultipartUploadCommand / UploadPartCommand / CompleteMultipartUploadCommand.
  2. Negative (parametrized via it.each) — same rejection check for ChecksumSHA1, ChecksumCRC32, ChecksumCRC32C, ChecksumCRC64NVME.
  3. Positive — multipart with ChecksumAlgorithm: \"SHA256\" succeeds — happy path unchanged.
  4. Positive — single-part upload with precomputed ChecksumSHA256 is unaffected (validation does not fire on the PutObject path).

Local test status (transparency)

  • tsc --noEmit passed for the changed files; no new TypeScript errors.
  • Local Vitest run was blocked by the known Yarn-on-Windows + missing workspace dist-*/ build outputs (@aws-sdk/client-s3 cannot be resolved from a fresh clone). CI on this PR will validate test execution.

Checklist

  • Scoped change: only lib/lib-storage/src/Upload.ts and lib/lib-storage/src/Upload.spec.ts are touched.
  • No public API additions or breaking changes — only converts a previously-broken runtime path into a clear client-side error.
  • Conventional Commits title with module scope.
  • Issue linked (Fixes #6742).
  • Tests added covering both negative (all five Checksum* params) and positive (ChecksumAlgorithm, single-part PutObject) cases.
  • Pre-commit hooks (lint-staged + lint:versions + lint:dependencies + lint:api) ran and passed; hooks were not skipped.

generated by AI tools, and reviewed by Mohanraj Venkatesan

…with multipart Upload (aws#6742)

When a caller passes a precomputed full-object checksum (ChecksumSHA256,
ChecksumSHA1, ChecksumCRC32, ChecksumCRC32C, ChecksumCRC64NVME) to
lib-storage's Upload for a body larger than partSize, Upload performs a
multipart upload. S3 then computes a composite per-part checksum and
rejects the request with a confusing BadDigest error after the full
network roundtrip.

Add a fail-fast client-side validation in __createMultipartUpload() that
throws a clear Error naming the offending param, explaining why it
cannot work with multipart, and listing the supported workarounds
(use ChecksumAlgorithm, or call PutObject directly). ChecksumAlgorithm
is intentionally NOT rejected since it is the supported way to use
checksums on multipart uploads. The single-part PutObject path is
unaffected.

Refs aws#6742
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[lib-storage] Upload fails if application provides checksum for >5 MB file

1 participant