Skip to content

Write COPY TO with PARTITION_BY as hive partitioned directories - #2995

Merged
tamnd merged 1 commit into
mainfrom
d5-partition
Oct 10, 2026
Merged

tamnd merged 1 commit into
mainfrom
d5-partition

Conversation

@tamnd

@tamnd tamnd commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Part of #1446.

COPY ... TO now takes PARTITION_BY. The rows are split by the values of the partition columns into one directory per value, a level per column, named column=value with both sides percent encoded, and each of the deepest directories gets one file in the chosen format. A NULL goes in __HIVE_DEFAULT_PARTITION__, and a value that would read as that name has its first character encoded so it gets a directory of its own. The partition columns are left out of the files unless WRITE_PARTITION_COLUMNS is set. All of this is what the pin writes, checked file by file on v2.0.0-dev84237. Before this, 244 records in copy/partitioned failed with "partition_by is not supported".

The options around it work too. OVERWRITE removes every file under the directory first and leaves the directories, OVERWRITE_OR_IGNORE writes over a file of the same name, and APPEND adds files named with a UUID. Without any of them a directory that already holds a file is refused with the pin's error. FILENAME_PATTERN takes {i}, {uuid}, {uuidv4} and {uuidv7}, a pattern of plain text gets the number put after it, and FILE_EXTENSION sets the extension. HIVE_FILE_PATTERN false puts every file side by side in the one directory, numbered by the order of the partition values, which is the order the pin numbers them in.

PARTITION_BY takes a list in parentheses or a single name, bare or quoted either way, matched without regard to case, and * names every column. The refusals are the pin's and come in the pin's order: a column the query does not have, a column named twice, APPEND without a UUID in the pattern, more than one of the three overwrite options, USE_TMP_FILE or PER_THREAD_OUTPUT with a partition, and a partition that leaves no column to write.

Without PARTITION_BY, OVERWRITE, OVERWRITE_OR_IGNORE, APPEND, FILENAME_PATTERN, FILE_EXTENSION, HIVE_FILE_PATTERN and USE_TMP_FILE are taken and have nothing to act on, as on the pin for one file, where they were refused as not supported before. A FILENAME_PATTERN without a UUID beside APPEND is still refused, since the pin checks that either way. PARTITION_PATH, PER_THREAD_OUTPUT on its own and FILE_SIZE_BYTES are still refused and come next.

The options are read in a new crates/rudb-bind/src/partition.rs before the format sees the rest, and crates/rudb/src/export.rs groups the rows and hands each partition to the writer it already had. Two tests in crates/rudb/tests/copy_to.rs cover the layouts, the overwrite options, the flat numbering, the read back through hive_partitioning and the refusals.

@tamnd tamnd added kind/compat A gap against DuckDB, on any of the six surfaces. area/duckdb-compat The DuckDB compatibility surfaces. labels Oct 10, 2026
@tamnd
tamnd merged commit f9492fd into main Oct 10, 2026
1 check failed
@tamnd
tamnd deleted the d5-partition branch October 10, 2026 03:15
@tamnd tamnd mentioned this pull request Oct 10, 2026
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/duckdb-compat The DuckDB compatibility surfaces. kind/compat A gap against DuckDB, on any of the six surfaces.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant