Hi, thanks for your work developing this tool. I am trying to run Deepurify on 10 medium-quality MAGs, using 1 GPU with compute capability of at least 6.0. My conda session info is at the bottom. The program halted but could not re-run using temporary files. Is there a way to continue the software where it left off? Or to automatically delete incomplete runs? (my goal would be to run on 1000+ MAGs) ``` (deepurify)[zrwerbin@scc-c09 ~]$ deepurify clean -i /projectnb/frpmars/soil_microbe_db/input_genomes/smag_decontam_spire/ -o /projectnb/frpmars/soil_microbe_db/input_genomes/smag_decontam_spire_out/ --bin_suffix fna --gpu_num 1 ################################## ### WELCOME TO USE DEEPURIFY ### ################################## Estimating Taxonomic Similarity by Labeling Lineage for Each Contig in the MAG. Processer 0 has 10 files in device cuda:0. PROCESSER 1638892, 11.7% complete for spire_mag_00037037. (Current / Total) --> (10 / 10) (deepurify)[zrwerbin@scc-c09 ~]$ ``` When I re-run the command, I get a slew of errors: ``` Starting Filter Contaminations and Separate Bins... Process Process-19:_00037053.fna, 1 / 10 Process Process-20: Traceback (most recent call last): Traceback (most recent call last): Process Process-24: File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 325, in subProcessFilter filterContaminationOneBin( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 259, in filterContaminationOneBin q = res[binNamePro] ~~~^^^^^^^^^^^^ File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() KeyError: 'spire_mag_00037053' File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 325, in subProcessFilter filterContaminationOneBin( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 259, in filterContaminationOneBin q = res[binNamePro] ~~~^^^^^^^^^^^^ KeyError: 'spire_mag_00037053' Process Process-22: Process Process-21: Traceback (most recent call last): Process Process-23: File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 325, in subProcessFilter filterContaminationOneBin( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 259, in filterContaminationOneBin q = res[binNamePro] ~~~^^^^^^^^^^^^ KeyError: 'spire_mag_00037053' Traceback (most recent call last): Traceback (most recent call last): File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 325, in subProcessFilter filterContaminationOneBin( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 259, in filterContaminationOneBin q = res[binNamePro] ~~~^^^^^^^^^^^^ KeyError: 'spire_mag_00037053' File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 325, in subProcessFilter filterContaminationOneBin( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 259, in filterContaminationOneBin q = res[binNamePro] ~~~^^^^^^^^^^^^ KeyError: 'spire_mag_00037053' Traceback (most recent call last): File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 325, in subProcessFilter filterContaminationOneBin( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/FilterBinsTools/FilterUtils.py", line 259, in filterContaminationOneBin q = res[binNamePro] ~~~^^^^^^^^^^^^ KeyError: 'spire_mag_00037053' Starting Run CheckM... Starting Gather Result... Traceback (most recent call last): File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/bin/deepurify", line 8, in <module> sys.exit(cli()) ^^^^^ File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/cli.py", line 181, in cli cleanMAGs( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/clean_func.py", line 196, in cleanMAGs runLabelFilterSplitBins( File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/funcs.py", line 265, in runLabelFilterSplitBins outInfo = findBestBinsAfterFiltering( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/SelectMAGsTools/SelectionUitls.py", line 145, in findBestBinsAfterFiltering root = buildTreeForBin(binFileName, deepurifyFolderPath, sevenFilteredChekcMList) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify/lib/python3.12/site-packages/Deepurify/SelectMAGsTools/SelectionUitls.py", line 84, in buildTreeForBin root = Tree(binName, "", sevenFilteredChekcMList[0][binName], [True]) ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ KeyError: 'spire_mag_00037053' (deepurify)[zrwerbin@scc-c09 ~]$ (deepurify)[zrwerbin@scc-c09 ~]$ (deepurify)[zrwerbin@scc-c09 ~]$ ``` Session info: ``` # Loaded modules first... module load prodigal/2.6.3 module load hmmer/3.3.2 module load miniconda module load cuda/11.3 conda activate deepurify # (deepurify)[zrwerbin@scc-c09 ~]$ conda list # packages in environment at /projectnb/talbot-lab-data/zrwerbin/.conda/envs/deepurify: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge anyio 4.2.0 pypi_0 pypi argon2-cffi 23.1.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi arrow 1.3.0 pypi_0 pypi asttokens 2.4.1 pypi_0 pypi async-lru 2.0.4 pypi_0 pypi attrs 23.2.0 pypi_0 pypi babel 2.14.0 pypi_0 pypi beautifulsoup4 4.12.3 pypi_0 pypi blas 1.0 mkl conda-forge bleach 6.1.0 pypi_0 pypi brotli-python 1.1.0 py312h30efb56_1 conda-forge bzip2 1.0.8 hd590300_5 conda-forge ca-certificates 2024.2.2 hbcca054_0 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge cffi 1.16.0 pypi_0 pypi charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge checkm-genome 1.2.2 pypi_0 pypi comm 0.2.1 pypi_0 pypi contourpy 1.2.0 pypi_0 pypi cuda-cudart 12.1.105 0 nvidia cuda-cupti 12.1.105 0 nvidia cuda-libraries 12.1.0 0 nvidia cuda-nvrtc 12.1.105 0 nvidia cuda-nvtx 12.1.105 0 nvidia cuda-opencl 12.3.101 0 nvidia cuda-runtime 12.1.0 0 nvidia cycler 0.12.1 pypi_0 pypi debugpy 1.8.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi deepurify 1.2.4 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi dendropy 4.6.1 pypi_0 pypi executing 2.0.1 pypi_0 pypi fastjsonschema 2.19.1 pypi_0 pypi ffmpeg 4.3 hf484d3e_0 pytorch filelock 3.13.1 pyhd8ed1ab_0 conda-forge fonttools 4.47.2 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi freetype 2.12.1 h267a509_2 conda-forge gmp 6.3.0 h59595ed_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge h11 0.14.0 pypi_0 pypi httpcore 1.0.2 pypi_0 pypi httpx 0.26.0 pypi_0 pypi icu 73.2 h59595ed_0 conda-forge idna 3.6 pyhd8ed1ab_0 conda-forge intel-openmp 2023.1.0 hdb19cb5_46306 ipykernel 6.29.2 pypi_0 pypi ipython 8.21.0 pypi_0 pypi ipywidgets 8.1.1 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jedi 0.19.1 pypi_0 pypi jinja2 3.1.3 pyhd8ed1ab_0 conda-forge joblib 1.3.2 pypi_0 pypi jpeg 9e h166bdaf_2 conda-forge json5 0.9.14 pypi_0 pypi jsonpointer 2.4 pypi_0 pypi jsonschema 4.21.1 pypi_0 pypi jsonschema-specifications 2023.12.1 pypi_0 pypi jupyter 1.0.0 pypi_0 pypi jupyter-client 8.6.0 pypi_0 pypi jupyter-console 6.6.3 pypi_0 pypi jupyter-core 5.7.1 pypi_0 pypi jupyter-events 0.9.0 pypi_0 pypi jupyter-lsp 2.2.2 pypi_0 pypi jupyter-server 2.12.5 pypi_0 pypi jupyter-server-terminals 0.5.2 pypi_0 pypi jupyterlab 4.1.0 pypi_0 pypi jupyterlab-pygments 0.3.0 pypi_0 pypi jupyterlab-server 2.25.2 pypi_0 pypi jupyterlab-widgets 3.0.9 pypi_0 pypi kiwisolver 1.4.5 pypi_0 pypi lame 3.100 h166bdaf_1003 conda-forge lcms2 2.15 hfd0df8a_0 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libblas 3.9.0 1_h86c2bf4_netlib conda-forge libcblas 3.9.0 5_h92ddd45_netlib conda-forge libcublas 12.1.0.26 0 nvidia libcufft 11.0.2.4 0 nvidia libcufile 1.8.1.2 0 nvidia libcurand 10.3.4.107 0 nvidia libcusolver 11.4.4.55 0 nvidia libcusparse 12.0.2.55 0 nvidia libdeflate 1.17 h0b41bf4_0 conda-forge libexpat 2.5.0 hcb278e6_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.2.0 h807b86a_5 conda-forge libgfortran-ng 13.2.0 h69a702a_5 conda-forge libgfortran5 13.2.0 ha4646dd_5 conda-forge libgomp 13.2.0 h807b86a_5 conda-forge libhwloc 2.9.3 default_h554bfaf_1009 conda-forge libiconv 1.17 hd590300_2 conda-forge libjpeg-turbo 2.0.0 h9bf148f_0 pytorch liblapack 3.9.0 5_h92ddd45_netlib conda-forge libnpp 12.0.2.50 0 nvidia libnsl 2.0.1 hd590300_0 conda-forge libnvjitlink 12.1.105 0 nvidia libnvjpeg 12.1.1.14 0 nvidia libpng 1.6.42 h2797004_0 conda-forge libsqlite 3.44.2 h2797004_0 conda-forge libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge libtiff 4.5.0 h6adf6a1_2 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libwebp-base 1.3.2 hd590300_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxml2 2.12.5 h232c23b_0 conda-forge libzlib 1.2.13 hd590300_5 conda-forge llvm-openmp 15.0.7 h0cdce71_0 conda-forge markupsafe 2.1.5 py312h98912ed_0 conda-forge matplotlib 3.8.2 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mistune 3.0.2 pypi_0 pypi mkl 2023.1.0 h213fc3f_46344 mpmath 1.3.0 pyhd8ed1ab_0 conda-forge nbclient 0.9.0 pypi_0 pypi nbconvert 7.16.0 pypi_0 pypi nbformat 5.9.2 pypi_0 pypi ncurses 6.4 h59595ed_2 conda-forge nest-asyncio 1.6.0 pypi_0 pypi nettle 3.6 he412f7d_0 conda-forge networkx 3.2.1 pyhd8ed1ab_0 conda-forge notebook 7.0.7 pypi_0 pypi notebook-shim 0.2.3 pypi_0 pypi numpy 1.26.4 py312heda63a1_0 conda-forge openh264 2.1.1 h780b84a_0 conda-forge openjpeg 2.5.0 hfec8fc6_2 conda-forge openssl 3.2.1 hd590300_0 conda-forge overrides 7.7.0 pypi_0 pypi packaging 23.2 pypi_0 pypi pandocfilters 1.5.1 pypi_0 pypi parso 0.8.3 pypi_0 pypi pexpect 4.9.0 pypi_0 pypi pillow 10.2.0 py312h5eee18b_0 pip 24.0 pyhd8ed1ab_0 conda-forge platformdirs 4.2.0 pypi_0 pypi pplacer 1.1.alpha19 h9ee0642_2 bioconda prometheus-client 0.19.0 pypi_0 pypi prompt-toolkit 3.0.43 pypi_0 pypi psutil 5.9.8 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pycparser 2.21 pypi_0 pypi pygments 2.17.2 pypi_0 pypi pyparsing 3.1.1 pypi_0 pypi pysam 0.22.0 pypi_0 pypi pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.12.1 hab00c5b_1_cpython conda-forge python-dateutil 2.8.2 pypi_0 pypi python-json-logger 2.0.7 pypi_0 pypi python_abi 3.12 4_cp312 conda-forge pytorch 2.2.0 py3.12_cuda12.1_cudnn8.9.2_0 pytorch pytorch-cuda 12.1 ha16c6d3_5 pytorch pytorch-mutex 1.0 cuda pytorch pyyaml 6.0.1 py312h98912ed_1 conda-forge pyzmq 25.1.2 pypi_0 pypi qtconsole 5.5.1 pypi_0 pypi qtpy 2.4.1 pypi_0 pypi readline 8.2 h8228510_1 conda-forge referencing 0.33.0 pypi_0 pypi requests 2.31.0 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi rpds-py 0.17.1 pypi_0 pypi scikit-learn 1.4.0 pypi_0 pypi scipy 1.12.0 pypi_0 pypi send2trash 1.8.2 pypi_0 pypi setuptools 69.0.3 pyhd8ed1ab_0 conda-forge six 1.16.0 pypi_0 pypi sniffio 1.3.0 pypi_0 pypi soupsieve 2.5 pypi_0 pypi stack-data 0.6.3 pypi_0 pypi sympy 1.12 pyh04b8f61_3 conda-forge tbb 2021.11.0 h00ab1b0_1 conda-forge terminado 0.18.0 pypi_0 pypi threadpoolctl 3.2.0 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tk 8.6.13 noxft_h4845f30_101 conda-forge torchaudio 2.2.0 py312_cu121 pytorch torchvision 0.17.0 py312_cu121 pytorch tornado 6.4 pypi_0 pypi tqdm 4.66.1 pypi_0 pypi traitlets 5.14.1 pypi_0 pypi types-python-dateutil 2.8.19.20240106 pypi_0 pypi typing_extensions 4.9.0 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge uri-template 1.3.0 pypi_0 pypi urllib3 2.2.0 pyhd8ed1ab_0 conda-forge wcwidth 0.2.13 pypi_0 pypi webcolors 1.13 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.7.0 pypi_0 pypi wheel 0.42.0 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.9 pypi_0 pypi xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge zlib 1.2.13 hd590300_5 conda-forge zstd 1.5.5 hfc55251_0 conda-forge ```