Backport #1097 to v1.5-variegata: write delete manifests with content='deletes'#1100
Merged
Tishj merged 1 commit intoJun 22, 2026
Conversation
Tishj
marked this pull request as draft
June 22, 2026 18:23
Member
|
Please remove the comment 👍 |
WriteToFile hardcoded the Avro file-metadata 'content' key to "data" for every
manifest, including delete manifests. The Iceberg spec requires this key to
reflect what the manifest holds ("data" or "deletes"); spec-strict readers
cross-check it against the manifest list entry's content type and reject a
delete manifest mislabeled as "data". Derive it from the manifest's content.
Adds a test asserting a v3 DELETE's delete manifest carries content='deletes'
(and not the mislabeled 'data') in its Avro file metadata.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Raghvendra Singh <raghav@cashify.in>
raghav-reglobe
force-pushed
the
backport-delete-manifest-content-v1.5-variegata
branch
from
June 22, 2026 20:28
891840e to
f343c51
Compare
Contributor
Author
|
Done 👍 |
raghav-reglobe
marked this pull request as ready for review
June 22, 2026 20:31
Member
|
Thanks! |
Tmonster
reviewed
Jun 23, 2026
| SELECT | ||
| count(*) FILTER (WHERE position('0e636f6e74656e740e64656c65746573' IN lower(hex(content))) > 0) AS has_deletes, | ||
| count(*) FILTER (WHERE position('0e636f6e74656e740864617461' IN lower(hex(content))) > 0) AS has_data | ||
| FROM read_blob(getvariable('delete_manifest')); |
Member
There was a problem hiding this comment.
Why does read_avro(getvariable('delete_manifest')) not work here?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This backports #1097 (merged to main) onto the v1.5-variegata release branch.
Without it, a v3 DELETE on this branch writes its delete manifest's Avro
contentfile metadata asdatainstead ofdeletes— it's hardcoded inWriteToFile. Readers that cross-check that key against the manifest-list entry's content type then reject the mismatch. The change just makes the metadata follow what the manifest actually holds (DATA -> "data", else"deletes").Clean cherry-pick of #1097 with its test — figured the release branch could use it too. Happy to reshape if you'd prefer.