Skip to content

Rotate COPY TO files by FILE_SIZE_BYTES and ROW_GROUPS_PER_FILE and share PER_THREAD_OUTPUT over the threads - #2996

Merged
tamnd merged 1 commit into
mainfrom
d5-rotate
Oct 10, 2026
Merged

tamnd merged 1 commit into
mainfrom
d5-rotate

Conversation

@tamnd

@tamnd tamnd commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Part of #1446.

COPY ... TO now takes FILE_SIZE_BYTES, ROW_GROUPS_PER_FILE (and its other name BATCHES_PER_FILE), PER_THREAD_OUTPUT and WRITE_EMPTY_FILE. With any of the first three the path becomes a directory of data_{i} files, named by FILENAME_PATTERN and FILE_EXTENSION when they are given, and it follows the same rules as PARTITION_BY for a directory that already holds files.

The rows are cut into batches the way the pin cuts them. For CSV and JSON a batch is 2048 rows. For Parquet a batch is as many of those as it takes to fill a row group. Before each batch the current file is closed if it has reached the size or the batch count, and the next file is opened. Everything here was checked against v2.0.0-dev84237 by line and byte counts. For example, 5000 rows with FILE_SIZE_BYTES 1.5e3 give files of 2048, 2048 and 904 rows, and 10000 rows written to Parquet with ROW_GROUP_SIZE 2000, ROW_GROUPS_PER_FILE 2 give 4096, 4096 and 1808. A partitioned copy rotates inside each partition directory, and with HIVE_FILE_PATTERN false the numbers carry on across partitions.

There is one deliberate difference. The pin checks the size before writing a batch, even on a file it has just opened, so a size below that of an empty file (the CSV header, say) makes it open empty files until the disk is full. That is tamnd/duckdb#49. Here a file is only closed once it holds a batch.

FILE_SIZE_BYTES is read like the pin's ParseBytesArg. A string goes through the same size parser as memory_limit, which moves into rudb-common so the binder can use it, and anything else is cast to UBIGINT. The options that cannot go together are refused in the pin's order and in its words, and that check now counts USE_TMP_FILE as set even when it is false. Setting two of OVERWRITE, OVERWRITE_OR_IGNORE and APPEND is refused even when one of them is false, as the pin does. JSON still refuses ROW_GROUPS_PER_FILE as an unknown option, like the pin. WRITE_EMPTY_FILE false writes no file at all when there are no rows.

PER_THREAD_OUTPUT on its own shares the batches out over one file per thread in the pool, so with SET threads = 4 the rows land in up to four files, which is what per_thread_output.test checks for. With no rows it still writes one empty file.

The writers in export.rs are now one sink per file that takes a batch at a time and counts the bytes it has written, so the single-file path and the rotating path share the same code.

@tamnd tamnd added kind/compat A gap against DuckDB, on any of the six surfaces. area/duckdb-compat The DuckDB compatibility surfaces. labels Oct 10, 2026
@tamnd
tamnd merged commit d27f88d into main Oct 10, 2026
1 check failed
@tamnd
tamnd deleted the d5-rotate branch October 10, 2026 03:30
@tamnd tamnd mentioned this pull request Oct 10, 2026
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/duckdb-compat The DuckDB compatibility surfaces. kind/compat A gap against DuckDB, on any of the six surfaces.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant