The pipeline seems to create an enormous amount of output. Therefore, with only a few sample inputs, the pipeline fails because of no disk space left (1.5 TB) ~20 samples a ~8Gb per Read (2x)
-
Idea: Is it possible to remove intermediate results earlier? I, for example, noticed that fastq.gz files get unzipped and remain unzipped throughout a specific part of the pipeline. Also, once the fastq.gz files are trimmed, can't the untrimmed files be removed? I bet there are a lot of similar space optimization options.
-
Idea: Would it be possible for the pipeline to estimate the largest amount of output at any given moment, with respect to the input datasets? The user could then decide if a reduction of input would be useful
-
Idea: Could there be a mode implemented (and maybe also defaulted) where disk space is treated dynamically in a way that only so many samples are processed at the same time, so that at every point, not more than 80% diskspace is used?
The pipeline seems to create an enormous amount of output. Therefore, with only a few sample inputs, the pipeline fails because of no disk space left (1.5 TB) ~20 samples a ~8Gb per Read (2x)
Idea: Is it possible to remove intermediate results earlier? I, for example, noticed that fastq.gz files get unzipped and remain unzipped throughout a specific part of the pipeline. Also, once the fastq.gz files are trimmed, can't the untrimmed files be removed? I bet there are a lot of similar space optimization options.
Idea: Would it be possible for the pipeline to estimate the largest amount of output at any given moment, with respect to the input datasets? The user could then decide if a reduction of input would be useful
Idea: Could there be a mode implemented (and maybe also defaulted) where disk space is treated dynamically in a way that only so many samples are processed at the same time, so that at every point, not more than 80% diskspace is used?