Fix duplicate deletion vectors when filter pushdown clones the multi file list#1054
Merged
Tishj merged 8 commits intoJun 13, 2026
Merged
Conversation
NiclasHaderer
added a commit
to motherduckdb/duckdb-iceberg
that referenced
this pull request
Jun 11, 2026
Member
|
This might be a different solution to #1038 Could you include the test from that PR here as well to verify? |
NiclasHaderer
force-pushed
the
nh/dynamic-pushdown-deletion-bug
branch
from
June 11, 2026 11:20
9f37fff to
c05091a
Compare
Contributor
Author
|
Just added the test and ran it locally. Seems to work |
NiclasHaderer
marked this pull request as ready for review
June 11, 2026 15:16
Tishj
reviewed
Jun 13, 2026
| filtered_list->have_bound = true; | ||
| //! Share the delete state: DML operators hold a reference to the original list of the scan's bind data, | ||
| //! while the scan itself can read through a filtered copy created by (dynamic) filter pushdown. | ||
| filtered_list->positional_delete_data = positional_delete_data; |
Member
There was a problem hiding this comment.
I feel like there might be a lifetime issue here, because the map eventually references a IcebergManifestEntry entry, which can be owned by this initial instance and isn't properly transferred
I think we just need to rework this a bit to group all of the gathered metadata into a shared instance.
The member that is of significance to this issue is likely the mutable vector<IcebergManifestListEntry> committed_delete_manifests;
lnkuiper
added a commit
to duckdb/duckdb
that referenced
this pull request
Jun 15, 2026
Includes duckdb/duckdb-iceberg#1054 as well
krlmlr
pushed a commit
to krlmlr/duckdb
that referenced
this pull request
Jun 21, 2026
Includes duckdb/duckdb-iceberg#1054 as well
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some Iceberg DML operators (
IcebergDelete) capture a reference to theIcebergMultiFileListin the scan's bind data at planning time. If filters are pushed into the scan afterwards (e.g. runtime join min/max filters duringUPDATE ... FROM),PushdownInternalcreates a clone with empty delete-state maps, and the scan reads existing deletes into the clone.FlushDeletesthen consults the original's empty map, misses the table's existing deletion vector, and neither invalidates its manifest entry nor merges its positions. The commit ends up with two DVs for the same data file, and the next read fails withTable is corrupt, two or more deletion vectors exist for the same referenced_data_file.Fix: keep
positional_delete_dataandequality_delete_datain ashared_ptrand alias them inPushdownInternal, so a list and all its pushdown clones share one delete state regardless of which instance scans and which one flushes. Filter state stays per-instance. Sharing needs no additional locking because the accesses are ordered by the pipeline: deletes are written while the scan runs, andFlushDeletesonly reads them in the sink's Finalize, which the scheduler fires after the scan has completed.