Skip to content

Write delete manifests with content='deletes' file metadata#1097

Merged
Tishj merged 2 commits into
duckdb:mainfrom
raghav-reglobe:fix-delete-manifest-content
Jun 22, 2026
Merged

Write delete manifests with content='deletes' file metadata#1097
Tishj merged 2 commits into
duckdb:mainfrom
raghav-reglobe:fix-delete-manifest-content

Conversation

@raghav-reglobe

Copy link
Copy Markdown
Contributor

Problem

WriteToFile in src/core/metadata/manifest/iceberg_manifest.cpp hardcodes the Avro
file-metadata content key to "data" for every manifest it writes, including delete
manifests:

metadata_values.emplace_back("content", "data");

Per the Iceberg spec, a manifest's content metadata must reflect what it holds — "data"
for a data manifest, "deletes" for a delete manifest. So a v3 DELETE (which writes a
deletion-vector delete manifest) produces an Avro file whose content metadata says
data, contradicting the manifest list entry, which correctly records it as a delete
manifest.

Readers that take a manifest's content from the manifest list entry tolerate this (so it
does not affect query results in duckdb-iceberg itself), but spec-strict readers cross-check
the manifest file's own content metadata against the list entry and reject the mismatch.
For example, iceberg-go's ReadManifest errors with:

manifest file's 'content' metadata indicates "data", but entry from manifest list indicates "deletes"

Fix

Derive the value from the manifest's content type (the IcebergManifestFile the function
already receives):

metadata_values.emplace_back("content",
                             manifest_file.content == IcebergManifestContentType::DATA ? "data" : "deletes");

Testing

test/sql/local/catalog_test_config_setup/catalog_agnostic/delete/test_delete_manifest_content_type.test:
after a v3 DELETE, resolves the delete manifest via iceberg_metadata and asserts its
Avro file metadata carries content='deletes' (and not the mislabeled content='data').

Found alongside #1095 — both are v3 deletion-vector write-correctness fixes.

WriteToFile hardcoded the Avro file-metadata 'content' key to "data" for every
manifest, including delete manifests. The Iceberg spec requires this key to
reflect what the manifest holds ("data" or "deletes"); spec-strict readers
cross-check it against the manifest list entry's content type and reject a
delete manifest mislabeled as "data". Derive it from the manifest's content.

Adds a test asserting a v3 DELETE's delete manifest carries content='deletes'
(and not the mislabeled 'data') in its Avro file metadata.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Raghvendra Singh <raghav@cashify.in>

@Tishj Tishj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍

@Tishj
Tishj merged commit d9fffd0 into duckdb:main Jun 22, 2026
18 checks passed
Tishj added a commit that referenced this pull request Jun 22, 2026
…-content-v1.5-variegata

Backport #1097 to v1.5-variegata: write delete manifests with content='deletes'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants