I need to concatenate about 160 tables stored in a MonetDBLite database into a single table. I used to do so with the SQLite backend to dplyr, like this: Reduce(union_all, tables_list). Unfortunately, this doesn't work with MonetDBLite: the database grows from 3.5GB to 70GB, and the merge fails after there's no free disk space left.
I've eventually found a workaround by splitting the operation in smaller parts:
indiv1 <- compute(Reduce(union_all, indl[1:50]))
indiv2 <- compute(Reduce(union_all, indl[51:100]))
indiv3 <- compute(Reduce(union_all, indl[101:150]))
indiv4 <- compute(Reduce(union_all, indl[151:length(indl)]))
indiv <- Reduce(union_all, list(indiv1, indiv2, indiv3, indiv4))
In the end, the database only takes 8.5GB (including original small tables and the big concatenated table).
Is this expected? FWIW, I've checked that the SQL commands generated by dplyr are very clean, i.e. a series of (SELECT * FROM TABLE1) UNION ALL .... I was wondering whether some temporary files were not freed as they should in the middle of the operation.
I need to concatenate about 160 tables stored in a MonetDBLite database into a single table. I used to do so with the SQLite backend to
dplyr, like this:Reduce(union_all, tables_list). Unfortunately, this doesn't work with MonetDBLite: the database grows from 3.5GB to 70GB, and the merge fails after there's no free disk space left.I've eventually found a workaround by splitting the operation in smaller parts:
In the end, the database only takes 8.5GB (including original small tables and the big concatenated table).
Is this expected? FWIW, I've checked that the SQL commands generated by dplyr are very clean, i.e. a series of
(SELECT * FROM TABLE1) UNION ALL .... I was wondering whether some temporary files were not freed as they should in the middle of the operation.