Skip to content

Read hive partitions and the filename option in read_csv and read_parquet - #2993

Merged
tamnd merged 1 commit into
mainfrom
d5-hive
Oct 10, 2026
Merged

tamnd merged 1 commit into
mainfrom
d5-hive

Conversation

@tamnd

@tamnd tamnd commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Part of #1446.

read_csv and read_parquet now take hive_partitioning, hive_types, hive_types_autocast and filename, and a replacement scan over a partitioned directory finds the partitions on its own, which is what the pin does by default.

A path segment of the form key=value adds a column key whose value is the segment's value, following the pin's parse: only directories count, a segment with two = or a ? is not a partition, and a key that appears twice keeps its first value. Partitioning is switched on without being asked when the first file has partitions and every file has the same keys. When it is asked for and the files do not agree, the call is refused with the pin's "Hive partition mismatch" binder errors.

The type of a key is guessed from every file's value the way the pin guesses it, trying DATE, then TIMESTAMP, then BIGINT, and falling back to VARCHAR when a value fits none of them or the files disagree. NULL and __HIVE_DEFAULT_PARTITION__ are skipped in the guess and read back as nulls, %XX escapes are decoded, and hive_types sets a type by hand, with a value that does not convert refused at read time with the pin's "Unable to cast" error. hive_types_autocast=false makes every key VARCHAR.

filename=true adds a column called filename with each row's path, and a string names the column. It goes after the file's own columns and after file_row_number, and the partition keys follow. A key that has the name of a file column takes that column over, type and value both, which is how the pin reads a directory written with the key kept in the files. The binder then drops the statistics it had for that column, since the file's numbers no longer describe it. The refusals for a filename column that clashes with a file column or a key, a hive_types that is not a struct of strings, a hive_types key that is not a partition, hive_types together with hive_partitioning=false and a null argument are the pin's words, all checked on v2.0.0-dev84237.

The binder writes what it decided into the plan's options, so the executor reads the decision and works the values out per file. The values are constant vectors, one per file, and a filter on an added column is kept above the scan rather than pushed into the reader. The late materialization pass leaves these scans alone for now.

Three tests in crates/rudb/tests/csv.rs cover the guess, the null markers and decoding, and the refusals, and one in crates/rudb/tests/parquet.rs covers a key that takes over a file column next to file_row_number and filename. COPY ... (PARTITION_BY ...) and the JSON readers are not part of this change.

@tamnd tamnd added kind/compat A gap against DuckDB, on any of the six surfaces. area/duckdb-compat The DuckDB compatibility surfaces. labels Oct 10, 2026
@tamnd
tamnd merged commit e922b0b into main Oct 10, 2026
1 check failed
@tamnd
tamnd deleted the d5-hive branch October 10, 2026 03:14
@tamnd tamnd mentioned this pull request Oct 10, 2026
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/duckdb-compat The DuckDB compatibility surfaces. kind/compat A gap against DuckDB, on any of the six surfaces.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant