Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1446.
read_csvandread_parquetnow takehive_partitioning,hive_types,hive_types_autocastandfilename, and a replacement scan over a partitioned directory finds the partitions on its own, which is what the pin does by default.A path segment of the form
key=valueadds a columnkeywhose value is the segment's value, following the pin's parse: only directories count, a segment with two=or a?is not a partition, and a key that appears twice keeps its first value. Partitioning is switched on without being asked when the first file has partitions and every file has the same keys. When it is asked for and the files do not agree, the call is refused with the pin's "Hive partition mismatch" binder errors.The type of a key is guessed from every file's value the way the pin guesses it, trying DATE, then TIMESTAMP, then BIGINT, and falling back to VARCHAR when a value fits none of them or the files disagree.
NULLand__HIVE_DEFAULT_PARTITION__are skipped in the guess and read back as nulls,%XXescapes are decoded, andhive_typessets a type by hand, with a value that does not convert refused at read time with the pin's "Unable to cast" error.hive_types_autocast=falsemakes every key VARCHAR.filename=trueadds a column calledfilenamewith each row's path, and a string names the column. It goes after the file's own columns and afterfile_row_number, and the partition keys follow. A key that has the name of a file column takes that column over, type and value both, which is how the pin reads a directory written with the key kept in the files. The binder then drops the statistics it had for that column, since the file's numbers no longer describe it. The refusals for a filename column that clashes with a file column or a key, ahive_typesthat is not a struct of strings, ahive_typeskey that is not a partition,hive_typestogether withhive_partitioning=falseand a null argument are the pin's words, all checked on v2.0.0-dev84237.The binder writes what it decided into the plan's options, so the executor reads the decision and works the values out per file. The values are constant vectors, one per file, and a filter on an added column is kept above the scan rather than pushed into the reader. The late materialization pass leaves these scans alone for now.
Three tests in crates/rudb/tests/csv.rs cover the guess, the null markers and decoding, and the refusals, and one in crates/rudb/tests/parquet.rs covers a key that takes over a file column next to
file_row_numberandfilename.COPY ... (PARTITION_BY ...)and the JSON readers are not part of this change.