feat(core,services/s3): support supplying a precomputed checksum on write#7830
Open
wolfv wants to merge 1 commit into
Open
feat(core,services/s3): support supplying a precomputed checksum on write#7830wolfv wants to merge 1 commit into
wolfv wants to merge 1 commit into
Conversation
…rite Add a public `Checksum` type and `ChecksumAlgorithm` enum to opendal-core, a `WriteOptions::checksum` field, an `OpWrite::with_checksum` accessor, and a `write_with(...).checksum(...)` builder method. A new `Capability::write_with_checksum` flag advertises support. The S3 service consumes a supplied checksum on the single-shot PutObject path, emitting the matching `x-amz-checksum-*` header from the raw digest bytes instead of computing one from the body. Because a whole-object checksum cannot be mapped onto individual parts, S3 returns Unsupported when a checksum is combined with append or multipart writes rather than silently dropping the integrity check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dosubot
Bot
added
size:L
Member
|
Maybe it worths to take a look at #6817 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
Users often already know an object's checksum ahead of time (for example a SHA-256 they computed when producing the data) and want the storage service to verify the upload against that value, rather than having OpenDAL recompute one from the body. There is currently no way to pass a known checksum through a write operation.
What changes are included in this PR?
A new, additive public API in
opendal-core:Checksumtype andChecksumAlgorithmenum (Checksum::sha256(bytes),crc32c,sha1,crc64nvme). The value holds the raw digest bytes; services encode it as their wire format requires.WriteOptions::checksumfield,OpWrite::with_checksum/OpWrite::checksumaccessors, and aOperator::write_with(...).checksum(...)builder method (plus the same onwriter_with).Capability::write_with_checksumto advertise support.S3 service support:
PutObjectpath, a supplied checksum is sent as the matchingx-amz-checksum-*header (base64-encoded from the raw digest bytes) instead of computing one from the body.Unsupportedwhen a checksum is combined with append or multipart writes, rather than silently dropping the integrity check.Unit tests cover the header mapping, and doctests cover the new public API.
Are there any user-facing changes?
Yes, all additive (no breaking changes):
opendal::Checksum,opendal::ChecksumAlgorithm,WriteOptions::checksum,Operator::write_with(...).checksum(...),Capability::write_with_checksum.AI Usage Statement
This PR was prepared with the assistance of Claude Code (Claude Opus 4.8). All changes were reviewed by the author.