Repository navigation
Write COPY TO with PARTITION_BY as hive partitioned directories - #2995
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1446.
COPY ... TOnow takesPARTITION_BY. The rows are split by the values of the partition columns into one directory per value, a level per column, namedcolumn=valuewith both sides percent encoded, and each of the deepest directories gets one file in the chosen format. A NULL goes in__HIVE_DEFAULT_PARTITION__, and a value that would read as that name has its first character encoded so it gets a directory of its own. The partition columns are left out of the files unlessWRITE_PARTITION_COLUMNSis set. All of this is what the pin writes, checked file by file on v2.0.0-dev84237. Before this, 244 records in copy/partitioned failed with "partition_by is not supported".The options around it work too.
OVERWRITEremoves every file under the directory first and leaves the directories,OVERWRITE_OR_IGNOREwrites over a file of the same name, andAPPENDadds files named with a UUID. Without any of them a directory that already holds a file is refused with the pin's error.FILENAME_PATTERNtakes{i},{uuid},{uuidv4}and{uuidv7}, a pattern of plain text gets the number put after it, andFILE_EXTENSIONsets the extension.HIVE_FILE_PATTERN falseputs every file side by side in the one directory, numbered by the order of the partition values, which is the order the pin numbers them in.PARTITION_BYtakes a list in parentheses or a single name, bare or quoted either way, matched without regard to case, and*names every column. The refusals are the pin's and come in the pin's order: a column the query does not have, a column named twice,APPENDwithout a UUID in the pattern, more than one of the three overwrite options,USE_TMP_FILEorPER_THREAD_OUTPUTwith a partition, and a partition that leaves no column to write.Without
PARTITION_BY,OVERWRITE,OVERWRITE_OR_IGNORE,APPEND,FILENAME_PATTERN,FILE_EXTENSION,HIVE_FILE_PATTERNandUSE_TMP_FILEare taken and have nothing to act on, as on the pin for one file, where they were refused as not supported before. AFILENAME_PATTERNwithout a UUID besideAPPENDis still refused, since the pin checks that either way.PARTITION_PATH,PER_THREAD_OUTPUTon its own andFILE_SIZE_BYTESare still refused and come next.The options are read in a new crates/rudb-bind/src/partition.rs before the format sees the rest, and crates/rudb/src/export.rs groups the rows and hands each partition to the writer it already had. Two tests in crates/rudb/tests/copy_to.rs cover the layouts, the overwrite options, the flat numbering, the read back through
hive_partitioningand the refusals.