Enable file pruning on nested columns

[jvansanten]

After thinking more carefully about what I was trying to achieve in #766, I realized that most of my underlying problem was related to the way FilterCombiner treats nested columns. Given a struct_extract expression, it wraps the inner filter in a StructFilter and pushes it into the filter set as a filter on the containing column. This has two unhelpful side-effects:

• Table filters can only address top-level columns. This probably makes sense for a row-oriented database where the containing struct needs to be deserialized to get at its members, but not for a column-oriented database. In particular, struct columns have no bounds in the manifest.
• Filters on different fields of a struct are always combined into a single ConjunctionAndFilter.

This PR attempts to address both of these by inverting the combination and wrapping that FilterCombiner does to create a decomposed TableFilterSet that refers to individual columns. This is then used both for partition and range pruning in FileMatchesFilter().

I suspect it would have been cleaner to apply this transformation once to IcebergMultiFileList::table_filters rather than for every single data file, but there appears to be some tight coupling between IcebergMultiFileList::table_filters and IcebergMultiFileReader that I can't quite wrap my head around yet. Suggestions welcome!

[Tishj]

Changes look great on the outset, but this needs some tests still

[Tishj]

Thanks for the tests, can we extend this to cover:

• multiple types
• deeper nesting
• lists
• maps
• IS NULL / IS NOT NULL on child columns
• IS NULL / IS NOT NULL on structs/lists/maps (not elements)
• conjunction AND on combinations of the above items

[jvansanten]

I've added a few new tests in test/sql/local/irc_any_catalog/reads/test_nested_column_pruning.test. Files are not currently pruned when filtering on elements of lists or maps, as PushDownFilterIntoExpr() leaves these as generic ExpressionFilters, nor with IS NULL or IS NOT NULL predicates, as FilterCombiner::TryPushdownExpression() does not treat them.

[jvansanten]

Ahoy, is there anything else that needs to happen before this can be merged?

[Tishj]

I'm very sorry to ask this, but the alternative is to close your PR: can you rebase with updated main?

There's been a decent sized shift upstream (in duckdb/duckdb) that makes these changes hard to merge.
We've moved away from TableFilter, instead only using Expression. With the only remaining TableFilter type being ExpressionTableFilter.

[jvansanten]

Thanks for coming back to this!

I see that a lot has changed under the feet of this PR, which is just what happens when one lives for too long. I will check to what degree its approach is still relevant/feasible/advisable and then rebase or close as appropriate.

[jvansanten]

@Tishj After digging around the current main, it looks to me like the built-in RemoveUnusedColumns optimizer now pushes down struct_extract calls, such that the filters IcebergMultiFileList::ComplexPushdown() receives already refer to individual columns. PushdownInternal() then discards this information by grouping them by top-level column id, e.g. the filter (struct_extract(struct_extract(deepstruct, 'struct'), 'int') <= 1) is optimized to deepstruct.struct.int <= 1, then applied as deepstruct <= 1 in IcebergMultiFileList::FileMatchesFilter() (which doesn't apply, because the struct column doesn't have bounds, and couldn't be compared to an int even if it did). It seems like the simplest way forward would be to group filters by ColumnIndex rather than by top-level column position. For ColumnIndex with index_type FULL_READ (i.e that refer to top-level columns) this should be equivalent to existing behavior, while for PUSHDOWN_EXTRACT it should allow pruning on nested fields.

Is there a reason not to do this? Otherwise I would try to implement it.

[Tishj]

I think you're right, and I don't think there's a reason not to do this, it's just not implemented yet