A few steps in ONEFlux assume there is at least some data records for a variables being provided as input, e.g., the new metadata file generation (BIF) routine. Removing data columns that are completely empty before starting the execution could be a clean way to avoid this issue.
Possible implementation: load all individual files in 01_qc_visual/qcv_files/*, merge, check empty variables, save new versions in 01_qc_visual/qcv_files without empty variables.
Consider the checks of required variables throughout the pipeline.
A few steps in ONEFlux assume there is at least some data records for a variables being provided as input, e.g., the new metadata file generation (BIF) routine. Removing data columns that are completely empty before starting the execution could be a clean way to avoid this issue.
Possible implementation: load all individual files in 01_qc_visual/qcv_files/*, merge, check empty variables, save new versions in 01_qc_visual/qcv_files without empty variables.
Consider the checks of required variables throughout the pipeline.