conda version of hifiasm slower than manually built version

[gconcepcion]

Hi Haoyu,

We've noticed internally that running a version of hifiasm compiled outside of conda with gcc/11.1.0 or gcc/11.3.0 runs significantly faster in terms of CPU time than a version of hifiasm installed / compiled using conda.

For a small 10Mb test dataset, the difference is roughly 3-4 times faster for the manually built version than with the conda built version.
hifiasm compiled w/ gcc/11.3.0 :
[M::main] Real time: 121.909 sec; CPU: 4144.494 sec; Peak RSS: 16.353 GB

hifiasm installed / compiled using conda:
[M::main] Real time: 393.322 sec; CPU: 15529.166 sec; Peak RSS: 16.353 GB

The time differential is even more significant when running a full human sized dataset.

Can you think of anything about the conda installed version that would result in a binary that takes longer to compute the assembly graph? Could this have something to do with compiler flags?

Thanks!

[chhylp123]

I have no idea about that... Could you please share the 10M dataset with us? I will have a try to reproduce this issue.

[gconcepcion]

Just sent you an email with a link to a 10Mb region of HG005

[chhylp123]

Thanks a lot!

[xzhoubayer]

quay.io/biocontainers/hifiasm:0.18.5--h5b5514e_0 is 5-fold slower than the 0.15.2 version compiled outside the container, when they were used to assemble a 2.5 Gb plant genome

[chhylp123]

@xzhoubayer Is it a public dataset? Can I reproduce this issue on my side?

[xzhoubayer]

It is not a public dataset. I cannot share. I think there are some public data sets of corn genomes

[xzhoubayer]

0.18.5 should not be slower than 0.15.2, right?

[chhylp123]

Might be a conda issue, instead of hifiasm itself.

[chhylp123]

Hi @gconcepcion, I'm wondering are previous conda versions of hifiasm also several times slower?

[williamrowell]

I switched from building my own docker with hifiasm to using the conda hifiasm sometime between July and October 2020 (I think you added the bioconda recipe around this time), and I've been using the conda version regularly since then. Based on the ~500 human samples I've assembled with the conda build, my expectation has always been roughly 18 to 36 hours on 48 threads for 20-30x depth, and I haven't really seen that change.

[gconcepcion]

I personally never noticed the issue because I always manually build hifiasm and it's always very fast. @williamrowell and I discovered this issue recently because he has been telling me for the past two years how slow it was in his human-wgs pipeline (which was contrary to my experience) so I finally decided to dig in to figure out what was going on and realized there is some unknown discrepancy between conda and a manually built version.

[hkeward]

I ran a small test sample using the conda-based and a manually built version of hifiasm (version 0.15.5).
Using the conda environment:

User	System	Elapsed	%CPU	Maxresident memory
441484.13	1035.65	2:59:40	4104%CPU	29.50G

Using a manually built version of hifiasm:

User	System	Elapsed	%CPU	Maxresident memory
166349.91	1198.05	1:11:50	3887%CPU	29.22G

Using the manually built version ran ~2.5x faster.