Problem at step "REMOVING UNTRUSTED K - MERS"

[sashulkaSh]

Hello!
I run:

reckoner -genome 4500000000 -prefix ${p} -threads 128 -kmcmemory 900 -longkmer -verbose -reuse ${r1} ${r2}

RECKONER ver. 2.1                                                                                                                                                                                          
                                                                                                                                                                                                           
##################################################                                                                                                                                                         
INPUT DATA SUMMARY                                                                                                                                                                                         
##################################################                                                                                                                                                         
Read file 1                                                                                                                                                                                                
      Name                    :  1.trim.fq                                                                                                                   
      Compressed              : no                                                                                                                                                                         
Read file 2                                                                                                                                                                                                
      Name                    : 2.trim.fq                                                                                                                   
      Compressed              : no                                                                                                                                                                         
Log file name                 : 1.log                                                                                                                        
K-mer length                  : not defined (will be determined automatically)                                                                                                                             
Long k-mer length             : not defined (will be determined automatically)                                                                                                                             
Number of threads             : 128                                                                                                                                                                        
                                                                                                                                                                                                           
##################################################                                                                                                                                                         
CHECKING READS                                                                                                                                                                                             
##################################################                                                                                                                                                         
1.trim.fq                                                                                                                                                   
     Quality score offset          : 33                                                                                                                                                                    
2.trim.fq                                                                                                                                                   
     Quality score offset          : 33                                                                                                                                                                    
                                                                                                                                                                                                           
##################################################                                                                                                                                                         
DETERMINING PARAMETERS                                                                                                                                                                                     
##################################################                                                                                                                                                         
Determined k-mer length: 32                                                                                                                                                                                
Determined long k-mer length: 48                                                                                                                                                                           
                                                                                                                                                                                                           
##################################################                                                                                                                                                         
K - MER COUNTING                                                                                                                                                                                           
##################################################                                                                                                                                                         
Error: some error while reading suffix file                                                                                                                                                                
At least one of the KMC databases does not exist. Both of them will be created.                                                                                                                            
                                                                                                                                                                                                           
***********************************************************************************************************************************************************************************************************
*********************************************************************                                                                                                                                      
Stage 1: 100%                                                                                                                                                                                              
Stage 2: 100%
1st stage: 285.866s                               
2nd stage: 310.003s   
Total    : 595.869s                         
Tmp size : 244856MB                                                                                  
                                                                                                     
Stats: 
   No. of k-mers below min. threshold :  31749099169
   No. of k-mers above max. threshold :            0                                                                                                                                                       
   No. of unique k-mers               :  37137077307                                                                                                                                                       
   No. of unique counted k-mers       :   5387978138                                                                                                                                                       
   Total no. of k-mers                : 250142374748                                                                                                                                                       
   Total no. of reads                 :   1524476080
   Total no. of super-k-mers          :  20070135337
***********************************************************************************************************************************************************************************************************
*********************************************************************
Stage 1: 43%



Stage 1: 44%
Stage 1: 100%
Stage 2: 100%
1st stage: 281.839s
2nd stage: 367.553s
Total    : 649.392s
Tmp size : 207319MB

Stats:
   No. of k-mers below min. threshold :            0
   No. of k-mers above max. threshold :            0
   No. of unique k-mers               :  45200870921
   No. of unique counted k-mers       :  45200870921
   Total no. of k-mers                : 225731535323
   Total no. of reads                 :   1524476080
   Total no. of super-k-mers          :  11519864474

################################################## 
DETERMINING CUTOFF THRESHOLD
################################################## 
Cutoff: 5

################################################## 
REMOVING UNTRUSTED K - MERS
################################################## 
in1: 0% 

then RECKONER does nothing, I left the program for a few days, nothing changed

In the top (or htop), I see that the program does not use resources

what is my problem? pls, help!

(files, that were created:
1.kmc_pre
1.kmc_suf
1.log
1.long.kmc_pre
1.long.kmc_suf
1.lst
1.tmp.kmc_pre
1.tmp.kmc_suf
)

[maciejdlugosz]

@sashulkaSh
Thank you for reporting the problem. Please, give a size of created kmc_pre/kmc_suf files, size of the input FASTQ files and amount of free HDD space.

[sashulkaSh]

4 1.kmc_pre
4 1.kmc_suf
258M 1.long.kmc_pre
464G 1.long.kmc_suf
258M 1.tmp.kmc_pre
36G 1.tmp.kmc_suf
PE reads - 319G *2
64T free space
RAM 1T

[maciejdlugosz]

@sashulkaSh
I'm not able to infer, what is the reason of the problem. Can you give asccession number? And, as I see, the reads are trimmed - what is a strategy of trimming?

[YuanfengZhang]

❯ ../resources/RECKONER/bin/reckoner -genome 3200000000 -read ./r1.fq.gz -read ./r2.fq.gz -prefix reckoner -thr
eads 64
RECKONER ver. 2.1

##################################################
INPUT DATA SUMMARY
##################################################
Read file 1
      Name                    : ./r1.fq.gz
      Compressed              : yes (GZIP)
Read file 2
      Name                    : ./r2.fq.gz
      Compressed              : yes (GZIP)
Log file name                 : reckoner/r1.log
K-mer length                  : not defined (will be determined automatically)
Number of threads             : 64

##################################################
CHECKING READS
##################################################
./r1.fq.gz
     Quality score offset          : 33
./r2.fq.gz
     Quality score offset          : 33

##################################################
DETERMINING PARAMETERS
##################################################
Determined k-mer length: 31

##################################################
K - MER COUNTING
##################################################
**************************************************************************************
Stage 1: 100%
Stage 2: 100%
1st stage: 9.20011s
2nd stage: 0.542666s
Total    : 9.74277s
Tmp size : 21MB

Stats:
   No. of k-mers below min. threshold :     12807786
   No. of k-mers above max. threshold :            0
   No. of unique k-mers               :     14732449
   No. of unique counted k-mers       :      1924663
   Total no. of k-mers                :     19412693
   Total no. of reads                 :       162732
   Total no. of super-k-mers          :      1884800

##################################################
DETERMINING CUTOFF THRESHOLD
##################################################
Cutoff: 5

##################################################
REMOVING UNTRUSTED K - MERS
##################################################
in1: 75%

and,

❯ ll reckoner
总计 16M
-rw-rw-r-- 1 zhangyuanfeng zhangyuanfeng    4  5月 25 12:15 r1.kmc_pre
-rw-rw-r-- 1 zhangyuanfeng zhangyuanfeng    4  5月 25 12:15 r1.kmc_suf
-rw-rw-r-- 1 zhangyuanfeng zhangyuanfeng 1.3K  5月 25 12:15 r1.log
-rw-rw-r-- 1 zhangyuanfeng zhangyuanfeng 1.3M  5月 25 12:15 r1.tmp.kmc_pre
-rw-rw-r-- 1 zhangyuanfeng zhangyuanfeng  15M  5月 25 12:15 r1.tmp.kmc_suf

the process background:

❯ ps aux | rg 'kmc'             
zhangyu+   31314  0.0  0.0   8456  3840 pts/22   S+   12:57   0:00 rg kmc
zhangyu+ 4092368 99.9  0.0 17165304 71680 pts/15 Sl+  12:15  41:54 /home/zhangyuanfeng/project/ngs/resources/RECKONER/bin/kmc_tools transform reckoner/r1.tmp reduce reckoner/r1 -ci5

It seems that the kmc_tools transform is taking a very long time (>= 1h) and the {threads} argument is not fully utilized there. It's just not unpractical... because:

❯ zcat r1.fq.gz | wc -l | awk '{print $1 / 4}'
4246210

❯ zcat r1.fq.gz | sed -n '2p' | wc -m
151

each of the fastq files I used only contains 4246210 paired-end 150bp reads (~30x, chr22 only). It seems that RECKONER would take years to calibrate a whole genome sequencing fastq pair. Is there any chance that the parallelization get refined? Thanks.

[YuanfengZhang]

[maciejdlugosz]

@YuanfengZhang
Thank you for the information. It seems that it is the same problem as reported by @sashulkaSh, however we performed a set of tests and still are not able to reproduce the problem. It seems, that in some cases one of supporting programs, kmc_tools, hangs and is not able to generate the result, what looks like it needs long time.

As you run your experiments on quite small FASTQs, is it possible to share them with us?

[YuanfengZhang]

Give me your email address, please.

[maciejdlugosz]

maciej.dlugosz@polsl.pl
And, if it is possible, could you give results of lscpu command?

[YuanfengZhang]

lscpu output:

❯ lscpu                 
架构：                    x86_64
  CPU 运行模式：          32-bit, 64-bit
  Address sizes:          46 bits physical, 57 bits virtual
  字节序：                Little Endian
CPU:                      160
  在线 CPU 列表：         0-159
厂商 ID：                 GenuineIntel
  型号名称：              Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz
    CPU 系列：            6
    型号：                106
    每个核的线程数：      2
    每个座的核数：        40
    座：                  2
    步进：                6
    CPU 最大 MHz：        3400.0000
    CPU 最小 MHz：        800.0000
    BogoMIPS：            4600.00
    标记：                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss
                           ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonsto
                          p_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
                           dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch
                           cpuid_fault epb cat_l3 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsba
                          se tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflusho
                          pt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_m
                          bm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts vnmi avx512vbmi umip pku ospke avx5
                          12_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig
                           flush_l1d arch_capabilities
Virtualization features:  
  虚拟化：                VT-x
Caches (sum of all):      
  L1d:                    3.8 MiB (80 instances)
  L1i:                    2.5 MiB (80 instances)
  L2:                     100 MiB (80 instances)
  L3:                     120 MiB (2 instances)
NUMA:                     
  NUMA 节点：             2
  NUMA 节点0 CPU：        0-39,80-119
  NUMA 节点1 CPU：        40-79,120-159
Vulnerabilities:          
  Gather data sampling:   Mitigation; Microcode
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, K
                          VM SW loop
  Srbds:                  Not affected
  Tsx async abort:        Not affected

[maciejdlugosz]

@YuanfengZhang
We reproduced the problem and working on a fix. Temporarily, I attach a version with a workaround.

RECKONER21_workaround-Linux.tar.gz

[YuanfengZhang]

Thx, the binary files work well. log:

❯ zcat test.R1.fq.gz | wc -l | awk '{print $1 / 4}' 
8245830
❯ zcat test.R2.fq.gz | wc -l | awk '{print $1 / 4}' 
8245830

❯ time ../../../../resources/RECKONER/bin/reckoner -genome 3200000000 -read test.R1.fq.gz -read test.R2.fq.gz -prefix ./reckoner -threads 24
RECKONER ver. 2.1

##################################################
INPUT DATA SUMMARY
##################################################
Read file 1
      Name                    : test.R1.fq.gz
      Compressed              : yes (GZIP)
Read file 2
      Name                    : test.R2.fq.gz
      Compressed              : yes (GZIP)
Log file name                 : ./reckoner/test.R1.log
K-mer length                  : not defined (will be determined automatically)
Number of threads             : 24

##################################################
CHECKING READS
##################################################
test.R1.fq.gz
     Quality score offset          : 33
test.R2.fq.gz
     Quality score offset          : 33

##################################################
DETERMINING PARAMETERS
##################################################
Determined k-mer length: 31

##################################################
K - MER COUNTING
##################################################
*****************************************************************
Stage 1: 100%
Stage 2: 100%
1st stage: 7.35178s
2nd stage: 19.9324s
Total    : 27.2841s
Tmp size : 1992MB

Stats:
   No. of k-mers below min. threshold :    180938127
   No. of k-mers above max. threshold :            0
   No. of unique k-mers               :    253211672
   No. of unique counted k-mers       :     72273545
   Total no. of k-mers                :   1919568554
   Total no. of reads                 :     16491660
   Total no. of super-k-mers          :    169428796

##################################################
DETERMINING CUTOFF THRESHOLD
##################################################
Cutoff: 5

##################################################
REMOVING UNTRUSTED K - MERS
##################################################
in1: 100% 

##################################################
CORRECTING ERRORS
##################################################
test.R1.fq.gz
     Number of corrected reads                  : 7694819
     Number of corrected substitutions          : 24282932
     Number of corrected insertions             : 1909700
     Number of corrected deletions              : 168403
test.R2.fq.gz
     Number of corrected reads                  : 7715032
     Number of corrected substitutions          : 28540331
     Number of corrected insertions             : 1717352
     Number of corrected deletions              : 159787

##################################################
RUNNING TIME
##################################################
Checking reads
     test.R1.fq.gz
          Start: Tue Jun  3 16:09:24 2025
          End  : Tue Jun  3 16:09:24 2025
     test.R2.fq.gz
          Start: Tue Jun  3 16:09:24 2025
          End  : Tue Jun  3 16:09:24 2025
Determining parameters
     Start: Tue Jun  3 16:09:24 2025
     End  : Tue Jun  3 16:09:24 2025
Counting k-mers
     Start: Tue Jun  3 16:09:24 2025
     End  : Tue Jun  3 16:09:52 2025
Determining cutoff threshold
     Start: Tue Jun  3 16:09:52 2025
     End  : Tue Jun  3 16:09:53 2025
Removing untrusted k-mers
     Start: Tue Jun  3 16:09:53 2025
     End  : Tue Jun  3 16:09:56 2025
Correcting errors in reads
     test.R1.fq.gz
          Start: Tue Jun  3 16:09:57 2025
          End  : Tue Jun  3 16:10:44 2025
     test.R2.fq.gz
          Start: Tue Jun  3 16:10:44 2025
          End  : Tue Jun  3 16:11:41 2025

The program is successfully completed

 -genome 3200000000 -read test.R1.fq.gz -read test.R2.fq.gz -prefix    2481.68s user 16.83s system 1829% cpu 2:16.58 total