Repository navigation
Picard error: Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException #24
Description
Description of the bug
Hello again!
After solving previous problems #19 I rerun the pipeline but after a while I'm getting a new error related with picard metrics and the process gets stuck. I hope you have a better idea of what's going on. Thanks in advance!
Command used and terminal output
nextflow run samurai --input /media/nas252/OMICS/CNA/samurai/004_0256_CRC_AA_bam_paths.csv \
--outdir /media/nas252/OMICS/CNA/samurai/output_PPM \
--genome GRCh38 --binsize 1000000 --analysis_type liquid_biopsy --caller ichorcna \
--normal_panel /media/nas252/OMICS/CNA/1000000/PoN_004_0256_sWGS_no_dedup_1000000_median.rds \
--ichorcna_readcounter_chrs "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22" \
--ichorcna_chrs_to_use "c(1:22)" \
--ichorcna_chrs_to_train "c(1:22)" --ichorcna_chrs_to_normalize "c(1:22)" \
--ichorcna_gc_wig /media/nas252/OMICS/CNA/gc_map_wigfiles/grch38/GRCh38.gc.ws_1000000.wig \
--ichorcna_map_wig /media/nas252/OMICS/CNA/gc_map_wigfiles/grch38/GRCh38.map.ws_1000000.wig \
--ichorcna_centromere_file /home/pperez/ichorCNA/inst/extdata/GRCh38.GCA_000001405.2_centromere_acen.txt \
-profile docker
Command error:
INFO O2024-10-18 09:47:332024]CollectWgsMetricsCollectProcessed 980,000,000 loci. Elapsed time: 00:14:20s. Time for last 10,000,000: 9s. Last read position: 16:29,718,966
INFOim2024-10-18 09:47:4394624CollectWgsMetrics Processed 990,000,000 loci. Elapsed time: 00:14:29s. Time for last 10,000,000: 9s. Last read position: 16:39,718,966
INFOet2024-10-18 09:47:51roadiCollectWgsMetricso/picarProcessed 1,000,000,000 loci. Elapsed time: 00:14:38s. Time for last 10,000,000: 8s. Last read position: 16:49,819,466
INFOpt2024-10-18 09:48:02" javCollectWgsMetricsOutOfBoProcessed 1,010,000,000 loci. Elapsed time: 00:14:48s. Time for last 10,000,000: 10s. Last read position: 16:59,819,466
INFO 2024-10-18 09:48:11AbstrCollectWgsMetricsector.iProcessed 1,020,000,000 loci. Elapsed time: 00:14:57s. Time for last 10,000,000: 9s. Last read position: 16:69,819,466
INFO 2024-10-18 09:48:21WgsMeCollectWgsMetricsl.proceProcessed 1,030,000,000 loci. Elapsed time: 00:15:08s. Time for last 10,000,000: 10s. Last read position: 16:79,819,466
INFO 2024-10-18 09:48:32ColleCollectWgsMetricsrk(CollProcessed 1,040,000,000 loci. Elapsed time: 00:15:18s. Time for last 10,000,000: 10s. Last read position: 16:89,869,466
INFO 2024-10-18 09:48:43ommanCollectWgsMetricsanceMaiProcessed 1,050,000,000 loci. Elapsed time: 00:15:30s. Time for last 10,000,000: 11s. Last read position: 17:9,541,121
INFO 2024-10-18 09:48:54icardCollectWgsMetricsnceMainProcessed 1,060,000,000 loci. Elapsed time: 00:15:41s. Time for last 10,000,000: 11s. Last read position: 17:19,541,121
INFO 2024-10-18 09:49:06icardCollectWgsMetricsPicardCProcessed 1,070,000,000 loci. Elapsed time: 00:15:52s. Time for last 10,000,000: 11s. Last read position: 17:29,541,121
INFO 2024-10-18 09:49:17 CollectWgsMetrics Processed 1,080,000,000 loci. Elapsed time: 00:16:04s. Time for last 10,000,000: 11s. Last read position: 17:39,541,121
INFOir2024-10-18 09:49:43 CollectWgsMetrics Processed 1,090,000,000 loci. Elapsed time: 00:16:30s. Time for last 10,000,000: 26s. Last read position: 17:65,522,841
INFOe/2024-10-18 09:49:55c9d14CollectWgsMetricsf464 Processed 1,100,000,000 loci. Elapsed time: 00:16:41s. Time for last 10,000,000: 11s. Last read position: 17:76,072,841
INFO 2024-10-18 09:50:07 CollectWgsMetrics Processed 1,110,000,000 loci. Elapsed time: 00:16:54s. Time for last 10,000,000: 12s. Last read position: 18:2,825,400
INFOou2024-10-18 09:50:18issueCollectWgsMetricshe procProcessed 1,120,000,000 loci. Elapsed time: 00:17:04s. Time for last 10,000,000: 10s. Last read position: 18:12,825,400
INFO 2024-10-18 09:50:28 CollectWgsMetrics Processed 1,130,000,000 loci. Elapsed time: 00:17:14s. Time for last 10,000,000: 9s. Last read position: 18:22,825,400
INFOec2024-10-18 09:50:38ile fCollectWgsMetrics Processed 1,140,000,000 loci. Elapsed time: 00:17:25s. Time for last 10,000,000: 10s. Last read position: 18:32,825,400
INFO 2024-10-18 09:50:49 CollectWgsMetrics Processed 1,150,000,000 loci. Elapsed time: 00:17:36s. Time for last 10,000,000: 10s. Last read position: 18:43,199,414
INFO 2024-10-18 09:51:00 CollectWgsMetrics Processed 1,160,000,000 loci. Elapsed time: 00:17:47s. Time for last 10,000,000: 11s. Last read position: 18:53,289,416
INFO 2024-10-18 09:51:10 CollectWgsMetrics Processed 1,170,000,000 loci. Elapsed time: 00:17:57s. Time for last 10,000,000: 10s. Last read position: 18:63,289,421
INFO 2024-10-18 09:51:21 CollectWgsMetrics Processed 1,180,000,000 loci. Elapsed time: 00:18:07s. Time for last 10,000,000: 10s. Last read position: 18:73,289,421
INFO 2024-10-18 09:51:32 CollectWgsMetrics Processed 1,190,000,000 loci. Elapsed time: 00:18:19s. Time for last 10,000,000: 11s. Last read position: 19:2,976,136
INFO 2024-10-18 09:51:44 CollectWgsMetrics Processed 1,200,000,000 loci. Elapsed time: 00:18:30s. Time for last 10,000,000: 11s. Last read position: 19:12,976,136
INFO 2024-10-18 09:51:55 CollectWgsMetrics Processed 1,210,000,000 loci. Elapsed time: 00:18:42s. Time for last 10,000,000: 11s. Last read position: 19:22,976,136
INFO 2024-10-18 09:52:08 CollectWgsMetrics Processed 1,220,000,000 loci. Elapsed time: 00:18:54s. Time for last 10,000,000: 12s. Last read position: 19:32,976,136
INFO 2024-10-18 09:52:20 CollectWgsMetrics Processed 1,230,000,000 loci. Elapsed time: 00:19:07s. Time for last 10,000,000: 12s. Last read position: 19:42,976,136
INFO 2024-10-18 09:52:32 CollectWgsMetrics Processed 1,240,000,000 loci. Elapsed time: 00:19:18s. Time for last 10,000,000: 11s. Last read position: 19:53,233,836
INFO 2024-10-18 09:52:44 CollectWgsMetrics Processed 1,250,000,000 loci. Elapsed time: 00:19:31s. Time for last 10,000,000: 12s. Last read position: 2:4,726,520
INFO 2024-10-18 09:52:56 CollectWgsMetrics Processed 1,260,000,000 loci. Elapsed time: 00:19:42s. Time for last 10,000,000: 11s. Last read position: 2:14,727,522
INFO 2024-10-18 09:53:08 CollectWgsMetrics Processed 1,270,000,000 loci. Elapsed time: 00:19:55s. Time for last 10,000,000: 12s. Last read position: 2:24,727,522
INFO 2024-10-18 09:53:21 CollectWgsMetrics Processed 1,280,000,000 loci. Elapsed time: 00:20:08s. Time for last 10,000,000: 13s. Last read position: 2:34,777,522
INFO 2024-10-18 09:53:34 CollectWgsMetrics Processed 1,290,000,000 loci. Elapsed time: 00:20:20s. Time for last 10,000,000: 12s. Last read position: 2:44,842,710
INFO 2024-10-18 09:53:46 CollectWgsMetrics Processed 1,300,000,000 loci. Elapsed time: 00:20:33s. Time for last 10,000,000: 12s. Last read position: 2:54,842,710
INFO 2024-10-18 09:53:59 CollectWgsMetrics Processed 1,310,000,000 loci. Elapsed time: 00:20:46s. Time for last 10,000,000: 13s. Last read position: 2:64,842,710
INFO 2024-10-18 09:54:13 CollectWgsMetrics Processed 1,320,000,000 loci. Elapsed time: 00:21:00s. Time for last 10,000,000: 13s. Last read position: 2:74,842,710
INFO 2024-10-18 09:54:26 CollectWgsMetrics Processed 1,330,000,000 loci. Elapsed time: 00:21:12s. Time for last 10,000,000: 12s. Last read position: 2:84,842,710
INFO 2024-10-18 09:54:39 CollectWgsMetrics Processed 1,340,000,000 loci. Elapsed time: 00:21:26s. Time for last 10,000,000: 13s. Last read position: 2:94,842,710
INFO 2024-10-18 09:54:53 CollectWgsMetrics Processed 1,350,000,000 loci. Elapsed time: 00:21:40s. Time for last 10,000,000: 13s. Last read position: 2:104,842,710
INFO 2024-10-18 09:55:09 CollectWgsMetrics Processed 1,360,000,000 loci. Elapsed time: 00:21:55s. Time for last 10,000,000: 15s. Last read position: 2:114,842,710
INFO 2024-10-18 09:55:23 CollectWgsMetrics Processed 1,370,000,000 loci. Elapsed time: 00:22:09s. Time for last 10,000,000: 14s. Last read position: 2:124,843,010
[Fri Oct 18 09:55:35 GMT 2024] picard.analysis.CollectWgsMetrics done. Elapsed time: 22.37 minutes.
Runtime.totalMemory()=775946240
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 133275309 out of bounds for length 133275309
at picard.analysis.AbstractWgsMetricsCollector.isReferenceBaseN(AbstractWgsMetricsCollector.java:218)
at picard.analysis.WgsMetricsProcessorImpl.processFile(WgsMetricsProcessorImpl.java:92)
at picard.analysis.CollectWgsMetrics.doWork(CollectWgsMetrics.java:242)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:280)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:105)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:115)
Work dir:
/home/pperez/work/be/b7f4c9d1465da71a4b2ab38270f464
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
-- Check '.nextflow.log' file for detailsRelevant files
System information
Nextflow version 23.10.1.5891
Hardware: Desktop
Executor: Local
Container engine: Docker
OS: Linux
Version of dincalcilab/samurai: Latest master commit
Activity
lbeltrame commentedon Oct 18, 2024
Wow, that's new! Never seen anything like this before...
How many samples are you processing? Would it be feasible to you to start with one sample, and check with each one of them (just use a single line in the samplesheet)?
Also what type of samples are you using? Can you also tell me the file sizes of the BAM indices that are created? The error is kind of cryptic and can have multiple causes.
pblpez commentedon Oct 18, 2024
I was processing 56 samples. These samples are cfDNA shallow WGS (~x0.6 mean depth). Regarding the indices created I guess you mean those in the output markduplicates directory (the only .bam.bai I see in the output dir). These indices have 2.3M.
After running a single one (tested it with 3 different samples), the pipeline completed with errors. The error seems to be due to the input gc wig file for ichorCNA. This file is one of the files provided by the ichorCNA tool itself.
nextflow(1).log
I have to check every sample, but all of them were processed the same way so I would bet the previous error arises when inputting all together.
lbeltrame commentedon Oct 18, 2024
My (uninformed) guess is that one sample specific is problematic, or is it a problem of sample size. You might want to try increasingly larger numbers of samples until it breaks.
Regarding your ichorCNA error, instead you have a different number of chromosomes in your BAMs compared to the GC file in ichorCNA (I hit this problem two days ago, so I remember ;), so when the genome is binned by readCounter, the numbers of bins don't match with the GC file. Make sure you have the same contigs in both your samples and in the GC files (common case: in the GC file there are also the unmapped, the alternative regions, etc... but they are absent in the BAM).
pblpez commentedon Oct 18, 2024
I just realized that the span in the sample.wig is not what I expected. I'm having a single bin per chromosome like
fixedStep chrom=1 start=1 step=248956422 span=248956422
while the gc.wig has the correct binning like
fixedStep chrom=1 start=1 step=1000000 span=1000000
I've tested ichorCNA before with these samples generating 1M bp binning and although the chromosomes didn't exactly match between both wig (sample and gc/map) it worked properly just by limiting the analysis to 1:22.
Here in the pipeline I set --binsize 1000000 (default was 500). I just made a couple of tests and I see that the input binsize its specified in kb? It works if I set --binsize 1000
Now I get another error, although the outputs I was looking for are generated correctly, but just to let you know:
.nextflow.log
And I wanted to ask another question. Does any other caller give Tumor Fraction information apart from ichorCNA?
Thank you again for your help!
lbeltrame commentedon Oct 18, 2024
It looks like there was a change in qpdf options. I'll give it a look on Monday.
You can use WisecondorX (
--caller wisecondorx) as a proxy of TF (it's not the same thing, though) with its Copy Number Profile Abnormality (or CPA): a few papers that compared the two showed a good (0.9) degree of correlation. Raman et al., 2021 has the details for how CPA works (NSCLC, using ctDNA).However WisecondorX requires a signficantly large panel of normals (because under the hood it relies on Z-scores to do its magic) to work well. In our tests at least > 40, and the WisecondorX people recommend close to 100.
lbeltrame commentedon Oct 18, 2024
@SaraPotente If you have time, can you test with a fresh (without any local Docker / singularity images) checkout and see if the error reported by @pblpez occurs?
pblpez commentedon Oct 21, 2024
I was trying to test --caller wisecondorx with a minimal call:
nextflow run samurai --input /media/nas252/OMICS/CNA/samurai/004_0256_CRC_AA_bam_paths.csv --outdir /media/nas252/OMICS/CNA/samurai/output --genome GRCh38 --binsize 1000 --analysis_type liquid_biopsy --caller wisecondorx --wisecondorx_yfrac 0 -profile docker
But I got this error:
I see that there are not many parameters to tune for WisecondorX, and I don't know what does it mean by "file function". I repeated it but this time adding the parameters used previously for the ichorCNA call (look at the .nextflow.log). I got an error related to the PoN provided. Does this caller use the parameters from ichorCNA even if ichorCNA is set as the caller? I want to reproduce the ichorCNA analysis but using WisecondorX, focusing only on autosomic chromosomes. Below the error I got this time:
nextflow.log
Thank you!
lbeltrame commentedon Oct 21, 2024
No, WisecondorX requires its own PoN, the options are not compatible (you need to use the ones that start with
wisecondor), the whole format is fundamentally different.Argument
cannot be nullmeans that the code expects a file, but nothing is provided, so it errors out. It's a bug (it should explicitly tell you what's wrong), but in this case it's because you don't have a PoN set.Would you be able to file a separate issue mentioning that WisecondorX should require a PoN? I might be able to do something next week as I should have some time for hacking.
SaraPotente commentedon Jul 2, 2026
A possible upstream issue, there is not much we can do at the moment.