Download SRA Files in Parallel with Slurm and Conda

This repository provides tools and configuration to efficiently download SRA (Sequence Read Archive) accessions in parallel on an HPC (High-Performance Computing) environment using the Slurm workload manager.
SRA Toolkit dependencies are managed via a dedicated Conda environment.

---

📁 Repository Contents

• download_sra_slurm.sh
  Main Bash script for submitting jobs to Slurm.

  • Reads a list of SRA accessions and manages parallel download and conversion.
  • Activates the Conda environment and runs prefetch and fastq-dump (via SRA Toolkit).
  • Handles temporary file cleanup.

• sra_accessions.txt
  Input file listing SRA accessions to download.

  • One accession per line.
  • Lines starting with # are treated as comments and ignored.

• SRA-tool.yml
  Conda environment definition file.

  • Contains the required packages and tools for the script to work (primarily the SRA Toolkit).

---

🚀 Usage Guide

1. Conda Environment Setup

1. Access your HPC cluster:

   ssh your_username@cluster_address

2. Clone this repository:

   git clone https://github.com/AndGob/SRA-download  # Replace with actual URL
   cd your_repo

3. Create the Conda environment:

   conda env create -f SRA-tool.yml

4. (Optional) Verify installation:

   conda activate SRA-tool
   fastq-dump --version
   conda deactivate

---

2. Prepare the SRA Accessions File

Edit the sra_accessions.txt file and add one accession ID per line. Example: NOTE: Current example refear to data from 10.1038/s41467-022-31142-5 publication

SRR1234567
SRR1234568
SRR1234569
# This is a test accession that will not be processed
SRR1234570

⚠️ Make sure there are no trailing spaces or hidden characters.

---

3. Launching the Slurm Job

1. Open download_sra_slurm.sh in your favorite editor.

2. Update this line:

   #SBATCH --array=0-3%4

   • 0-3: Replace 3 with the number of accessions - 1.
   • %4: Max number of parallel jobs (adjust to your cluster limits, e.g., %10).

3. Make the script executable:

   chmod +x download_sra_slurm.sh

4. Submit the job:

   sbatch download_sra_slurm.sh

---

🧭 Monitoring and Output

• Check job status:

  squeue -u your_username

• Log files: Slurm generates one .out and .err file per task:

  ls -l slurm-*.out slurm-*.err
  cat slurm-1234567_0.out
  less slurm-1234567_0.err

• Downloaded files: FASTQ files will be saved in the SRA_Downloads/ directory (automatically created by the script).

---

🛑 Ignoring Generated Files (.gitignore)

To keep your Git repo clean, use the following .gitignore:

# Ignore the directory with downloaded FASTQ files
SRA_Downloads/

# Ignore temporary .sra files
*.sra

# Ignore folders like SRRxxxxxx/
SRR*/

# Ignore Slurm output logs
slurm-*.out
slurm-*.err

If already tracked by Git, untrack and commit:

git rm -r --cached SRA_Downloads/
git commit -m "Ignore generated SRA files and logs"

---

📬 Contact

For questions, feel free to open an issue or contact the repository maintainer.