Skip to content

Pb/iceberg partitioning aware cardinality estimation#1023

Open
Tishj wants to merge 10 commits into
duckdb:mainfrom
Tishj:pb/iceberg-partitioning-aware-cardinality-estimation
Open

Pb/iceberg partitioning aware cardinality estimation#1023
Tishj wants to merge 10 commits into
duckdb:mainfrom
Tishj:pb/iceberg-partitioning-aware-cardinality-estimation

Conversation

@Tishj

@Tishj Tishj commented Jun 2, 2026

Copy link
Copy Markdown
Member

This PR supersedes #778

I've made several changes to the branch, to reduce code duplication and make use of the data that's already present.

I've had to leave part of the fix commented out however, as we need global_columns and global_column_ids to call ProcessDeletes
To reduce future merge conflicts I've left that untouched for now, but likely we have to split that logic into a part that can be done without those, to then follow it up with a method that finalizes the ProcessDeletes logic (which looks to be purely equality-delete focused).

ywelsch and others added 6 commits February 23, 2026 09:17
Implements IcebergGetPartitionStats for iceberg_scan, wiring it into DuckDB's partition statistics API. For each data file the function now surfaces the per-file row count together with per-column min/max bounds (read from the Iceberg manifest's lower_bounds/upper_bounds maps) via a new IcebergPartitionRowGroup. This allows the planner to prune partitions using column-level predicates and to produce accurate per-partition cardinality estimates, rather than falling back to a single table-level estimate.

For tables with delete files the function scans the delete manifests independently to compute per-data-file net row counts. Positional delete entries that carry referenced_data_file (V3 puffin deletion vectors and optimised V2 positional deletes) are resolved exactly — the deleted count is subtracted from the data file's gross record_count and the result is reported as COUNT_EXACT. Equality deletes and V2 positional deletes without a specific target file remain COUNT_APPROXIMATE, since their impact cannot be determined per-file from manifest metadata alone.

Tests are added to verify that EXPLAIN cardinality reflects partition pruning (~5,000 full scan → ~1,000 with a single-partition filter on filtering_on_partition_bounds) and that deletion vectors produce correct net estimates (deletion_vectors: EXPLAIN ~50,000, actual count(*) = 50,000). A new data generator and test (partitioned_deletion_vectors) specifically exercises the combination of partition pruning and deletion vectors, showing the per-file net count (~500) rather than the gross count (~1,000) after applying a partition filter.
@Tishj

Tishj commented Jun 2, 2026

Copy link
Copy Markdown
Member Author
		// if the above code didn't find an equality filter (i.e country_code = "[us]")
		// and there are other table filters (i.e cost > 50), use default selectivity.
		bool has_equality_filter = (cardinality_after_filters != base_table_cardinality);
		if (!has_equality_filter && has_non_optional_filters) {
			cardinality_after_filters = MaxValue<idx_t>(
			    LossyNumericCast<idx_t>(double(base_table_cardinality) * RelationStatisticsHelper::DEFAULT_SELECTIVITY),
			    1U);
		}

This is causing the estimated cardinality to be reduced from 500 to 100, get_partition_stats doesn't even get hit 😬

@Tishj
Tishj marked this pull request as ready for review June 2, 2026 14:59
@Tishj

Tishj commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

Actually, I think this should be using this instead of reading all the manifest files/data files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants