genome-seek cache error

[yingchen69]

Hi,

I installed genome-seek through conda:

mamba create -c conda-forge -c bioconda -p /mycondaEnv/snakemake_singularity snakemake singularity
git clone https://github.com/OpenOmics/genome-seek.git
cd genome-seek
mamba activate /mycondaEnv/snakemake_singularity
./genome-seek --version
genome-seek 0.3.3-alpha
snakemake --version
7.25.0
singularity --version
singularity version 3.8.6

But when I tried to run: /genome-seek/genome-seek cache --sif-cache /sif-cache, I got the following error:

genome-seek (0.3.3-alpha)
Image will be pulled from "/data/OpenOmics/SIFs/ccbr_wes_base_v0.1.0.sif".
Image will be pulled from "/data/OpenOmics/SIFs/deepvariant_1.3.0-gpu.sif".
Image will be pulled from "/data/OpenOmics/SIFs/glnexus_v1.4.1.sif".
Image will be pulled from "/data/OpenOmics/SIFs/ncbr_opencravat_latest.sif".
Image will be pulled from "/data/OpenOmics/SIFs/ncbr_octopus_v0.1.0.sif".
Image will be pulled from "/data/OpenOmics/SIFs/ncbr_sigprofiler_v0.1.0.sif".
Image will be pulled from "/data/OpenOmics/SIFs/ncbr_vcf2maf_v0.1.0.sif".
/projectsp/foran/yc790/apps/genome-seek/src/cache.sh: line 210: SLURM_JOB_ID: unbound variable
/projectsp/foran/yc790/apps/genome-seek/src/cache.sh: line 210: SLURM_JOB_ID: unbound variable
WARNING: Failed to run 'set -euo pipefail; /genome-seek/src/cache.sh local  -s '/sif-cache'  -i '/data/OpenOmics/SIFs/ccbr_wes_base_v0.1.0.sif,/data/OpenOmics/SIFs/deepvariant_1.3.0-gpu.sif,/data/OpenOmics/SIFs/glnexus_v1.4.1.sif,/data/OpenOmics/SIFs/ncbr_opencravat_latest.sif,/data/OpenOmics/SIFs/ncbr_octopus_v0.1.0.sif,/data/OpenOmics/SIFs/ncbr_sigprofiler_v0.1.0.sif,/data/OpenOmics/SIFs/ncbr_vcf2maf_v0.1.0.sif'  -t '/sif-cache/yc790/.singularity/' ' command!
        └── Command returned a non-zero exitcode of '1'.
Fatal: Failed to pull all containers. Please try again!

It seems that the image sif files are missing and the SLURM_JOB_ID in cache.sh is not defined. Is there a way to get around?

Thanks a lot!

Ying

[skchronicles]

Hello @yingchen69,

I hope you are having a great day! I just have a really quick question.

1. Are you running this pipeline on the NIH's Biowulf cluster?

2. Are you running this pipeline on a SLURM cluster?

I am just wondering because if you are running the pipeline on Biowulf, you will not need to cache any of the references/software containers? Also, at the current moment, this pipeline is set up to run on a cluster-- specifically a SLURM cluster. Other job schedulers can/will be added in the future on a per-need basis, but this is the current backend for execution.

Please let me know what you think.

[yingchen69]

Hi Skyler, I am not with NIH. Justin leads me to the OpenOmics and I am trying to set it here at Rutgers to run it on our cluster Amarel. My guess is that the dockers images are on your local srever's /data. Thanks, Ying

…

________________________________ From: Skyler Kuhn ***@***.***> Sent: Monday, March 27, 2023 1:42 PM To: OpenOmics/genome-seek ***@***.***> Cc: yingchen69 ***@***.***>; Mention ***@***.***> Subject: Re: [OpenOmics/genome-seek] genome-seek cache error (Issue #14) Hello @yingchen69<https://github.com/yingchen69>, I hope you are having a great day! I just have a really quick question. 1. Are you running this pipeline on the NIH's Biowulf cluster? 2. Are you running this pipeline on a SLURM cluster? I am just wondering because if you are running the pipeline on Biowulf, you will not need to cache any of the references/software containers? Also, at the current moment, this pipeline is set up to run on a cluster-- specifically a SLURM cluster. Other job schedulers can/will be added in the future on a per-need basis, but this is the current backend for execution. Please let me know what you think. — Reply to this email directly, view it on GitHub<#14 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABQ767Q3OPMEDGHR6NPM5PDW6HGRDANCNFSM6AAAAAAWIVJSPU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[yingchen69]

Hi Skyler,

Yes we use slurm on Amarel.

Thanks,

Ying

[skchronicles]

Hey @yingchen69,

I would need to make a few changes on my side if you would like to run this outside the NIH. Right now, it is pointing to local SIFs, but I can update that to use DockerHub URIs. There is also a pretty large resource bundle that you will need to download too to run the pipeline. I was planning on adding those changes with the next major release, but I haven't had time. This pipeline is still a WIP, but I will set aside some time later this week and next week to make those updates and a few other updates that have been proposed.

I will reach out to you when everything is completed.

Best,
@skchronicles

[yingchen69]

Hi Skyler, Thanks a lot! Ying

…

________________________________ From: Skyler Kuhn ***@***.***> Sent: Monday, March 27, 2023 6:05 PM To: OpenOmics/genome-seek ***@***.***> Cc: yingchen69 ***@***.***>; Mention ***@***.***> Subject: Re: [OpenOmics/genome-seek] genome-seek cache error (Issue #14) Hey @yingchen69<https://github.com/yingchen69>, I would need to make a few changes on my side if you would like to run this outside the NIH. Right now, it is pointing to local SIFs, but I can update that to use DockerHub URIs. There is also a pretty large resource bundle that you will need to download too to run the pipeline. I was planning on adding those changes with the next major release, but I haven't had time. This pipeline is still a WIP, but I will set aside some time later this week and next week to make those updates and a few other updates that have been proposed. I will reach out to you when everything is completed. Best, @yingchen69<https://github.com/yingchen69> — Reply to this email directly, view it on GitHub<#14 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABQ767UDPZGWQQBJWQOHMBTW6IFLHANCNFSM6AAAAAAWIVJSPU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[yingchen69]

Hi Skyler,

Sorry to bother you. Is there any update?

Thanks,

Ying

[moritzschaefer]

@skchronicles This pipeline is absolutely fantastic!

Is here a way to obtain the resource-bundle for setting this up on non-biowulf compte environments?

All the best,
Moritz