Multigroup analysis

[knokknok]

Hello,
My experimental design includes more than two groups between which I'd want to identify DMRs.
Is this possible without restarting the pipeline from scratch?
Thanks!

[JakeLehle]

Can you tell me more about your specific groups?

If I remember correctly the DMR identification is a pairwise comparison so it likely will require you to set up multiple pipeline runs for each comparison.

While you might have more than two groups could you consolidate them down into "control" vs "various treatment/experimental conditions" groups?

[knokknok]

Dear @JakeLehle

Thanks for your reply!

I have one factor with 3 modalities and would like to compare each pair (so 3 comparisons).
I have another factor with 2 modalities (different annotations of the same samples) and would like to compare the two.

If I remember correctly the DMR identification is a pairwise comparison so it likely will require you to set up multiple pipeline runs for each comparison.

That was my thought and I have a config file where all samples are listed and I define Group1 and Group2 to reference the subset of samples for 1 comparison of interest.
When the first comparison is over, if I edit the config file to change only the Group1 and Group2 informations, will the pipeline start from scratch and erase the previous computations or will it use the previous calling information?

[JakeLehle]

Hey @knokknok

Yes, if you edit the config.yaml and rerun the pipeline, it will overwrite the files in your selected result folder, which you also set the path for in the config.yaml.

So since you have control over the name of the config.yaml and the output folder why not make one for each comparison?

As a quick solution, why not make 4 configs each doing the pairwise comparison 3 for the first modalities and 1 more for the last two modalities you have, and then run the pipeline 4 times with output being directed to new results folder which you can name modality_X_vs_Y or whatever you like. You should ultimately end up with 4 big tables with each comparison listing the DMRs and UMRs/LMRs which you can compare between each for similarities and differences.

Let me know if you have any follow-up questions!

[knokknok]

Dear @JakeLehle

The issue is that I'd rather not wait a week (which is the time I estimate will be needed to map & call the samples) for each comparison...

[JakeLehle]

Yes, the pipeline takes a while to process whole genome sequencing files especially if you have a large number of samples.

Okay, let's speed that up.

New questions.
-Are you maxing out your available cores for parallel computing? What do you have the cores set to in the config.yaml?
-Are you running this at a university on a server or on a personal computer?

I would hope on a server since you would have to have a pretty advanced personal computer to run WGBS files.
If you have access to a server just run all 4 jobs at once.

[knokknok]

-Are you maxing out your available cores for parallel computing? What do you have the cores set to in the config.yaml?

No, I set it to 8 while the server has 20 (40 hyperthreads). I did this because of this sentence in the article:
"When running the pipeline, we limited the core utilization to only 8 of the 80 available cores, which we found was nearing a plateau in multithreading efficiency for alignment speed"

I imagine running the 4 simultaneously would not get anywhere near a 4x and would run into I/O bottlenecks?

Isn't snakemake supposed to be able to start from an intermediate state if the required files are available?
What would happen if I moved the dmr directory (and maybe others?) out of the folder, modified the group1/group2 information in the config file and ran again using the same output directory?

Again, thanks for the time you are devoting to this!

[JakeLehle]

Hello @knokknok yeah I do remember adding that line to the manuscript. However, It would probably be a good idea to bump up the number to the max for your system for your sake since your dataset is much larger.

If I remember correctly for the manuscript, I wanted to set the core number to something small enough that any user would have access to. So 2-8 seemed like a core number that your average low/midrange setup would have access to. However I believe the dataset that I calculated the supplemental 4 section on was the tester dataset that came with the wg-blimp installation which is on the smaller side. That's why the run time in the supplemental section 4 figure was only 50 mins which is what I remember the sample dataset takes. I did see a plateau in the computing time that was not accelerated by adding more cores with smaller datasets so I didn't want to increase it more up to 80 (which is definitely beyond an average setup) and then likely over thread the small dataset analysis but I don't think I ever checked the much larger dataset and I wanted to keep the core number consistent as I moved onto testing the larger dataset size that was still in the range of an average user.

For your purpose with a much larger dataset I think setting it to the max for your system is a good idea to see if that can get you some time saving.

I'm not sure on the I/O bottleneck but I don't think that would be an issue. Most servers (and I'm just guessing at your setup) have you checkout a node and if your run each node at the max cores possible then that should segregate their processes. It would be 4 separate computers running things independent of each other. Also if you set up the outputs into 4 different directories there shouldn't be any issues with one process trying to use files that are locked during runtime in the same dir.

So to your final point. Yes you totally could move the some of the DMR directories out and then change the config and rerun the pipeline and get snakemake to pickup and try to remake the new DMR directory to fulfill the target files needed to finish but you are kinda making your life extra hard because then if you want to use the shiny server to view things easily you have to move the DMR back into the right dir in your results folder and give it the right config file to the shiny to run it and output the files correctly. Simple solution is just have each comparison have it's own config and output into 4 distinct dirs which will save you time and help with organization in my opinion.

Let me know how everything goes. Very happy to see people still getting some use out of this pipeline. Let me know if you hit any other issues and I can help out.

[knokknok]

Dear @JakeLehle,

I only have access to one server so I am thinking I will try to create different directories with symbolic links to "trick" snakemake into doing the different analyses once the first one is over.
It is currently spending a relatively long time on the mark_duplicates steps which unfortunately do not seem to be parallelized...
I will let you know how it goes.

Very happy to see people still getting some use out of this pipeline.

To be honest, I was initially looking at msPipe since it included some additional gene set analyses and allowed multiple analyses in the same run...
But given the time it required, I looked into wg-blimp 😉

Best,
Knokknok.