infer mixture crashes with 47 mixtures

[CGWallace]

Hi Eric:

This is the first time I am running more than 15 mixtures, In fact I have 47 - This is not unusual for us.
I have a work around where I split the mixture and run in blocks of 10. then combine the reports.

"infer_mixture" seems to complete the analysis for all mixtures but in the final phase (data amalgamation I assume) it crashes. Here is the final few lines of output.

Working on mixture collection: WCVI_9 with 22 individuals
calculating log-likelihoods of the mixture individuals. time: 0.14 seconds
performing 5000 burn-in and 25000 more sweeps of method "MCMC" time: 4.09 seconds
tidying output into a tibble. time: 0.61 seconds
Error: cannot allocate vector of size 2.1 Gb

[krshedd]

Sounds like you ran out of memory. Have you tried thinning the trace output with the sample_int_Pi argument? If that doesn't help then you will need to either:

1. keep using your workaround of doing a few mixtures at a time,
2. free up some memory by cleaning out unused objects from your workspace, or
3. get a computer with more RAM.

[CGWallace]

Kyle: Thanks for getting back to me. Just thought you should be aware of this issue. I did run out of memory I had to break up the mixture file into groups of no more than 15. Colin Wallace Molecular Genetics Lab From: Kyle Shedd [mailto:notifications@github.com] Sent: May 8, 2018 2:51 PM To: eriqande/rubias Cc: Wallace, Colin; Author Subject: Re: [eriqande/rubias] infer mixture crashes with 47 mixtures (#18) Sounds like you ran out of memory. Have you tried thinning the trace output with the sample_int_Pi argument? If that doesn't help then you will need to either: 1. keep using your workaround of doing a few mixtures at a time, 2. free up some memory by cleaning out unused objects from your workspace, or 3. get a computer with more RAM. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#18 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af8e68B4Z5Ym8gQWxsn4fEPrn4VrDqPYks5twhMygaJpZM4Tq-01>.

[krshedd]

Did you guys have any luck trying to thin the trace output? We've been setting sample_in_Pi = 10 to save space.

[CGWallace]

Kyle: I will give that a go. Could that effect the calculation of Standard deviation? Thanks. Colin Wallace Molecular Genetics Lab From: Kyle Shedd [mailto:notifications@github.com] Sent: May 8, 2018 3:32 PM To: eriqande/rubias Cc: Wallace, Colin; Author Subject: Re: [eriqande/rubias] infer mixture crashes with 47 mixtures (#18) Did you guys have any luck trying to thin the trace output? We've been setting sample_in_Pi = 10 to save space. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#18 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af8e66kpKBlnjS1a0eLGz3RJZwFhnHYlks5twhzegaJpZM4Tq-01>.

[krshedd]

Hi Collin, no, it shouldn't substantially change the calculation of standard deviations or credibility intervals. I saw in your first post that you are running 25,000 sweeps with a burn-in of 5,000 sweeps. If you thin by 10 your trace will still have 2,000 sweeps to characterize stock comp distributions.

[CGWallace]

Kyle: I have a new issue that has shown up today. I had a few SNPs fail in my run so there are no values the complete mixture. What I have done is erase them from both the mixture and the baseline. This could be done within Rubias as an option. Not sure why I haven’t seen this before today as I have run this 10’s of times if not 100’s of times.

All_mix.GMA <- infer_mixture(bco_base, new_all_mix,

+ gen_start_col = 6, reps = 25000, burn_in = 5000) Scoring locus Oki_RAD13015_68 as haploid Scoring locus Oki_RAD70461_61 as haploid Scoring locus Oki_RAD95743_39 as haploid Scoring locus Ots_118938_311 as haploid Scoring locus Ots_96500_124 as haploid Scoring locus Ots_Cath_D141_134 as haploid Scoring locus Ots_crRAD18336_135 as haploid Scoring locus Ots_crRAD21752_87 as haploid Scoring locus Ots_crRAD2806_95 as haploid Scoring locus Ots_parp3_45 as haploid Scoring locus Ots_stk6_521 as haploid Error in infer_mixture(bco_base, new_all_mix, gen_start_col = 6, reps = 25000, : Ploidy mismatch in reference and mixture data sets at loci 2099141257262265271273276294298

Colin Wallace Molecular Genetics Lab From: Kyle Shedd [mailto:notifications@github.com] Sent: May 8, 2018 3:41 PM To: eriqande/rubias Cc: Wallace, Colin; Author Subject: Re: [eriqande/rubias] infer mixture crashes with 47 mixtures (#18) Hi Collin, no, it shouldn't substantially change the calculation of standard deviations or credibility intervals. I saw in your first post that you are running 25,000 sweeps with a burn-in of 5,000 sweeps. If you thin by 10 your trace will still have 2,000 sweeps to characterize stock comp distributions. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#18 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af8e6_U0egO3bXHTJsJnpsH4tkg6NfW1ks5twh7ugaJpZM4Tq-01>.

[krshedd]

Hi Collin, my guess is that since you had SNPs fail in your mixture then both columns were NA, and since rubias assumes that a SNP is haploid if the second column for a SNP just NA, it threw the ploidy mismatch error (because you have diploid data in the baseline, but it assumed haploid in the mixture). I suppose Eric could add some code to pull out SNPs that have just NA in the first column of a mixture, but your solution of creating a new baseline file seems pretty simple.

[CGWallace]

Thanks Kyle. Colin Wallace Molecular Genetics Lab From: Kyle Shedd [mailto:notifications@github.com] Sent: May 8, 2018 4:18 PM To: eriqande/rubias Cc: Wallace, Colin; Author Subject: Re: [eriqande/rubias] infer mixture crashes with 47 mixtures (#18) Hi Collin, my guess is that since you had SNPs fail in your mixture then both columns were NA, and since rubias assumes that a SNP is haploid if the second column for a SNP just NA, it threw the ploidy mismatch error (because you have diploid data in the baseline, but it assumed haploid in the mixture). I suppose Eric could add some code to pull out SNPs that have just NA in the first column of a mixture, but your solution of creating a new baseline file seems pretty simple. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#18 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af8e6x_bKDltjGRPlb4VT0jENqR9XEbgks5twieZgaJpZM4Tq-01>.

[eriqande]

Hey Guys!

It is totally awesome seeing you engaging on GitHub. Thanks! Sorry I have been a little out of touch in this whole conversation. Somewhat crazy times here...

Kyle, great recommendation to thin things for less memory use.

I'll put up an issue to make it not throw a ploidy error when everything is missing at a locus.

[CGWallace]

Eric: The error was very informative to me as I am looking into the possibility that there is an issue with our latest primer pool. I was thinking that making this an option (like ignore.ploidy.error = FALSE) in the infer.mixture call might be best. Where the default would be to stop and report the problem. I still haven’t gone back to that and tried the thinning as I am trying to complete all of our mix stock analysis. May try that later this week. Colin Wallace Molecular Genetics Lab From: Eric C. Anderson [mailto:notifications@github.com] Sent: May 9, 2018 10:34 AM To: eriqande/rubias Cc: Wallace, Colin; Author Subject: Re: [eriqande/rubias] infer mixture crashes with 47 mixtures (#18) Hey Guys! It is totally awesome seeing you engaging on GitHub. Thanks! Sorry I have been a little out of touch in this whole conversation. Somewhat crazy times here... Kyle, great recommendation to thin things for less memory use. I'll put up an issue to make it not throw a ploidy error when everything is missing at a locus. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#18 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af8e6zmPMsWPphdOJOTd7yxFeDqkdo7oks5twyiSgaJpZM4Tq-01>.

[eriqande]

Cool. Good idea!

[CGWallace]

I noticed at various spots in this post there is a spelling error: sample_in_Pi should be sample_int_Pi.
Colin