Currently, peak memory is at ~2x the array final size.
This happens because we first create all the chunks in the lapply() loop, and then put them in the correct position in the final array:
|
chunk_selections <- withCallingHandlers( |
|
lapply( |
|
# We skip missing chunks here since they will just be filled with the fill value |
|
# when initializing the consolidated array. |
|
existing_idx, |
|
function(i) { |
|
.extract_elements( |
|
current_chunk_index = required_chunks[i, ], |
|
current_chunk_path = chunk_paths[i], |
|
metadata = metadata, |
|
index = index, |
|
s3_client = s3_client, |
|
chunk_idx = chunk_idx |
|
) |
|
} |
|
), |
|
warning = function(w) { |
|
warnings <<- c(warnings, list(w)) # nolint: undesirable_operator_linter. |
|
invokeRestart("muffleWarning") |
|
} |
|
) |
This makes sense for a speed point of view but we could also reduce the peak memory usage by not keeping all the chunks in memory but putting them in the correct position as they are created.
I expect different users might have different needs so it probably needs to be customizable or optional.
Currently, peak memory is at ~2x the array final size.
This happens because we first create all the chunks in the
lapply()loop, and then put them in the correct position in the final array:Rarr/R/read_data.R
Lines 145 to 165 in 6f05b27
This makes sense for a speed point of view but we could also reduce the peak memory usage by not keeping all the chunks in memory but putting them in the correct position as they are created.
I expect different users might have different needs so it probably needs to be customizable or optional.