Repository navigation
Merged
Conversation
…hare PER_THREAD_OUTPUT over the threads
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1446.
COPY ... TOnow takesFILE_SIZE_BYTES,ROW_GROUPS_PER_FILE(and its other nameBATCHES_PER_FILE),PER_THREAD_OUTPUTandWRITE_EMPTY_FILE. With any of the first three the path becomes a directory ofdata_{i}files, named byFILENAME_PATTERNandFILE_EXTENSIONwhen they are given, and it follows the same rules asPARTITION_BYfor a directory that already holds files.The rows are cut into batches the way the pin cuts them. For CSV and JSON a batch is 2048 rows. For Parquet a batch is as many of those as it takes to fill a row group. Before each batch the current file is closed if it has reached the size or the batch count, and the next file is opened. Everything here was checked against v2.0.0-dev84237 by line and byte counts. For example, 5000 rows with
FILE_SIZE_BYTES 1.5e3give files of 2048, 2048 and 904 rows, and 10000 rows written to Parquet withROW_GROUP_SIZE 2000, ROW_GROUPS_PER_FILE 2give 4096, 4096 and 1808. A partitioned copy rotates inside each partition directory, and withHIVE_FILE_PATTERN falsethe numbers carry on across partitions.There is one deliberate difference. The pin checks the size before writing a batch, even on a file it has just opened, so a size below that of an empty file (the CSV header, say) makes it open empty files until the disk is full. That is tamnd/duckdb#49. Here a file is only closed once it holds a batch.
FILE_SIZE_BYTESis read like the pin's ParseBytesArg. A string goes through the same size parser asmemory_limit, which moves into rudb-common so the binder can use it, and anything else is cast to UBIGINT. The options that cannot go together are refused in the pin's order and in its words, and that check now countsUSE_TMP_FILEas set even when it is false. Setting two ofOVERWRITE,OVERWRITE_OR_IGNOREandAPPENDis refused even when one of them is false, as the pin does. JSON still refusesROW_GROUPS_PER_FILEas an unknown option, like the pin.WRITE_EMPTY_FILE falsewrites no file at all when there are no rows.PER_THREAD_OUTPUTon its own shares the batches out over one file per thread in the pool, so withSET threads = 4the rows land in up to four files, which is what per_thread_output.test checks for. With no rows it still writes one empty file.The writers in export.rs are now one sink per file that takes a batch at a time and counts the bytes it has written, so the single-file path and the rotating path share the same code.