Skip to content

Test data run fails under SLURM configuration but succeeds with TORQUE settings #682

Open
Open
Test data run fails under SLURM configuration but succeeds with TORQUE settings#682

Description

@h20gg702
h20gg702
opened on Apr 19, 2025

I encountered an issue when running HiC-Pro 3.1.0 on our SLURM-based cluster.
The installation process succeeded, but the test data run failed under SLURM configuration.
After switching to TORQUE configuration, the test data run completed successfully.

Is there any difference in HIC-pro between TORQUE and SLURM?
Mine is SLURM system so I wanna use HIC-pro under SLURM configuration

make configure
make -f ./scripts/install/Makefile CONFIG_SYS=./config-install.txt
make[1]: Entering directory '/home1/HiC-Pro'
./scripts/install/install_dependencies.sh -c ./config-install.txt -p /home1/HiC-Pro_3.1.0 -o /home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0 -q
Make sure internet connection works for your shell prompt under current user's privilege ...
Starting HiC-Pro installation !
export /apps/spack/2406/apps/linux-rocky8-x86_64_v3/gcc-13.3.0/bowtie2-2.5.2-m4m67z7/bin/bowtie2 in PATH
export /apps/spack/2406/apps/linux-rocky8-x86_64_v3/gcc-13.3.0/samtools-1.19.2-h5qi25w/bin/samtools in PATH
export /apps/spack/2406/apps/linux-rocky8-x86_64_v3/gcc-13.3.0/r-4.4.3-iupdrxg/bin/R in PATH
export ~/miniconda3/bin/python in PATH
Checking dependencies
- Python libraries ...OK
- R installation ...OK
- Bowtie2 installation ...OK
- Samtools installation ...OK

Checking HiC-Pro configuration
- Configuration for SLURM system ...OK

done !
make[1]: Leaving directory '/home1/HiC-Pro'

make install
(g++ -Wall -O2 -std=c++0x -o build_matrix /home1/HiC-Pro/scripts/src/build_matrix.cpp; mv build_matrix /home1/HiC-Pro/scripts)
In file included from /apps/generic/gcc/13.3.0/include/c++/13.3.0/cstdlib:79,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/ext/string_conversions.h:43,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/bits/basic_string.h:4109,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/string:54,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/bits/locale_classes.h:40,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/bits/ios_base.h:41,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/ios:44,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/ostream:40,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/iostream:41,
                 from /home1/HiC-Pro/scripts/src/build_matrix.cpp:7:
In function 'int atoi(const char*)',
    inlined from 'int interaction_parse(char*, char*&, chrsize_t&, char*&, chrsize_t&)' at /home1/HiC-Pro/scripts/src/build_matrix.cpp:702:20,
    inlined from 'int build_matrix(int, chrsize_t, const char*, const char*, const char*, const char*, Format, const std::string&, bool, MatrixFormat, chrsize_t, bool, const char*, const char*)' at /home1/HiC-Pro/scripts/src/build_matrix.cpp:895:22:
/usr/include/stdlib.h:363:23: warning: 'str' may be used uninitialized [-Wmaybe-uninitialized]
  363 |   return (int) strtol (__nptr, (char **) NULL, 10);
      |                ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home1/HiC-Pro/scripts/src/build_matrix.cpp: In function 'int build_matrix(int, chrsize_t, const char*, const char*, const char*, const char*, Format, const std::string&, bool, MatrixFormat, chrsize_t, bool, const char*, const char*)':
/home1/HiC-Pro/scripts/src/build_matrix.cpp:683:9: note: 'str' was declared here
  683 |   char* str;
      |         ^~~
(g++ -Wall -O2 -std=c++0x -o cutsite_trimming /home1/HiC-Pro/scripts/src/cutsite_trimming.cpp; mv cutsite_trimming /home1/HiC-Pro/scripts)
cp -Ri /home1/HiC-Pro /home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0
HiC-Pro installed in /home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0 !

/home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0/bin/HiC-Pro \
> -i /project/HIC_CRC/test_data/test_data \
> -o /project/HIC_CRC/test_data/hicpro_latest_test \
> -c /project/HIC_CRC/test_data/config_test_latest.txt

Run HiC-Pro 3.1.0
--------------------------------------------
Fri Apr 18 18:06:51 PDT 2025
Bowtie2 alignment step1 ...
Logs: logs/dixon_2M/mapping_step1.log
Logs: logs/dixon_2M_2/mapping_step1.log

--------------------------------------------
Fri Apr 18 18:07:21 PDT 2025
Bowtie2 alignment step2 ...
Logs: logs/dixon_2M/mapping_step2.log
Logs: logs/dixon_2M_2/mapping_step2.log

--------------------------------------------
Fri Apr 18 18:07:35 PDT 2025
Combine R1/R2 alignment files ...
Logs: logs/dixon_2M/mapping_combine.log
Logs: logs/dixon_2M_2/mapping_combine.log

--------------------------------------------
Fri Apr 18 18:07:39 PDT 2025
Mapping statistics for R1 and R2 tags ...
Logs: logs/dixon_2M/mapping_stats.log
Logs: logs/dixon_2M_2/mapping_stats.log

--------------------------------------------
Fri Apr 18 18:07:40 PDT 2025
Pairing of R1 and R2 tags ...
Logs: logs/dixon_2M/mergeSAM.log
make: *** [/home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0/bin/../scripts//Makefile:144: bowtie_pairing] Error 1

Then I switched to TORQUE when I make configure on SLURM.

make configure
make -f ./scripts/install/Makefile CONFIG_SYS=./config-install.txt
make[1]: Entering directory '/home1/HiC-Pro'
./scripts/install/install_dependencies.sh -c ./config-install.txt -p /home1/HiC-Pro_3.1.0 -o /home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0 -q
Make sure internet connection works for your shell prompt under current user's privilege ...
Starting HiC-Pro installation !
export /apps/spack/2406/apps/linux-rocky8-x86_64_v3/gcc-13.3.0/bowtie2-2.5.2-m4m67z7/bin/bowtie2 in PATH
export /apps/spack/2406/apps/linux-rocky8-x86_64_v3/gcc-13.3.0/samtools-1.19.2-h5qi25w/bin/samtools in PATH
export /apps/spack/2406/apps/linux-rocky8-x86_64_v3/gcc-13.3.0/r-4.4.3-iupdrxg/bin/R in PATH
export ~/miniconda3/bin/python in PATH
Checking dependencies
- Python libraries ...OK
- R installation ...OK
- Bowtie2 installation ...OK
- Samtools installation ...OK

Checking HiC-Pro configuration
- Configuration for TORQUE/PBS system ...OK

done !
make[1]: Leaving directory '/home1/HiC-Pro'

make install
(g++ -Wall -O2 -std=c++0x -o build_matrix /home1/HiC-Pro/scripts/src/build_matrix.cpp; mv build_matrix /home1/HiC-Pro/scripts)
In file included from /apps/generic/gcc/13.3.0/include/c++/13.3.0/cstdlib:79,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/ext/string_conversions.h:43,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/bits/basic_string.h:4109,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/string:54,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/bits/locale_classes.h:40,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/bits/ios_base.h:41,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/ios:44,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/ostream:40,
                 from /apps/generic/gcc/13.3.0/include/c++/13.3.0/iostream:41,
                 from /home1/HiC-Pro/scripts/src/build_matrix.cpp:7:
In function 'int atoi(const char*)',
    inlined from 'int interaction_parse(char*, char*&, chrsize_t&, char*&, chrsize_t&)' at /home1/HiC-Pro/scripts/src/build_matrix.cpp:702:20,
    inlined from 'int build_matrix(int, chrsize_t, const char*, const char*, const char*, const char*, Format, const std::string&, bool, MatrixFormat, chrsize_t, bool, const char*, const char*)' at /home1/HiC-Pro/scripts/src/build_matrix.cpp:895:22:
/usr/include/stdlib.h:363:23: warning: 'str' may be used uninitialized [-Wmaybe-uninitialized]
  363 |   return (int) strtol (__nptr, (char **) NULL, 10);
      |                ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home1/HiC-Pro/scripts/src/build_matrix.cpp: In function 'int build_matrix(int, chrsize_t, const char*, const char*, const char*, const char*, Format, const std::string&, bool, MatrixFormat, chrsize_t, bool, const char*, const char*)':
/home1/HiC-Pro/scripts/src/build_matrix.cpp:683:9: note: 'str' was declared here
  683 |   char* str;
      |         ^~~
(g++ -Wall -O2 -std=c++0x -o cutsite_trimming /home1/HiC-Pro/scripts/src/cutsite_trimming.cpp; mv cutsite_trimming /home1/HiC-Pro/scripts)
cp -Ri /home1/HiC-Pro /home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0
HiC-Pro installed in /home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0 !

/home1/HiC-Pro_3.1.0/HiC-Pro_3.1.0/bin/HiC-Pro \
> -i /project/HIC_CRC/test_data/test_data \
> -o /project/HIC_CRC/test_data/hicpro_latest_test \
> -c /project/HIC_CRC/test_data/config_test_latest.txt

Run HiC-Pro 3.1.0
--------------------------------------------
Fri Apr 18 14:51:46 PDT 2025
Bowtie2 alignment step1 ...
Logs: logs/dixon_2M/mapping_step1.log
Logs: logs/dixon_2M_2/mapping_step1.log

--------------------------------------------
Fri Apr 18 14:52:39 PDT 2025
Bowtie2 alignment step2 ...
Logs: logs/dixon_2M/mapping_step2.log
Logs: logs/dixon_2M_2/mapping_step2.log

--------------------------------------------
Fri Apr 18 14:52:53 PDT 2025
Combine R1/R2 alignment files ...
Logs: logs/dixon_2M/mapping_combine.log
Logs: logs/dixon_2M_2/mapping_combine.log

--------------------------------------------
Fri Apr 18 14:52:57 PDT 2025
Mapping statistics for R1 and R2 tags ...
Logs: logs/dixon_2M/mapping_stats.log
Logs: logs/dixon_2M_2/mapping_stats.log

--------------------------------------------
Fri Apr 18 14:52:58 PDT 2025
Pairing of R1 and R2 tags ...
Logs: logs/dixon_2M/mergeSAM.log
Logs: logs/dixon_2M_2/mergeSAM.log

--------------------------------------------
Fri Apr 18 14:53:01 PDT 2025
Assign alignments to restriction fragments ...
Logs: logs/dixon_2M/mapped_2hic_fragments.log
Logs: logs/dixon_2M_2/mapped_2hic_fragments.log

--------------------------------------------
Fri Apr 18 14:55:49 PDT 2025
Merge chunks from the same sample ...
Logs: logs/dixon_2M/merge_valid_interactions.log
Logs: logs/dixon_2M_2/merge_valid_interactions.log

--------------------------------------------
Fri Apr 18 14:55:50 PDT 2025
Merge stat files per sample ...
Logs: logs/dixon_2M/merge_stats.log
Logs: logs/dixon_2M_2/merge_stats.log

--------------------------------------------
Fri Apr 18 14:55:51 PDT 2025
Run quality checks for all samples ...
Logs: logs/dixon_2M/make_Rplots.log
Logs: logs/dixon_2M_2/make_Rplots.log

--------------------------------------------
Fri Apr 18 14:55:58 PDT 2025
Generate binned matrix files ...
Logs: logs/dixon_2M/build_raw_maps.log
Logs: logs/dixon_2M_2/build_raw_maps.log

--------------------------------------------
Fri Apr 18 14:55:59 PDT 2025
Run ICE Normalization ...
Logs: logs/dixon_2M/ice_500000.log
Logs: logs/dixon_2M/ice_1000000.log
Logs: logs/dixon_2M_2/ice_500000.log
Logs: logs/dixon_2M_2/ice_1000000.log

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions