mapper IndexError on some variants

[kmoad]

@kmoad
Sorry to bother you again, but I still get positions with errors. It looks like the mapper is missing something in the reference (the algorithm in python is getting out of range? or the reference is inadequate? - it's unclear from the error).
I was running 6 wes for a project, and I got multiple variants excluded from mapping and are absent from the error log and the annotation. The errors are posted only in the terminal. The explanation of the error is identical for each position, so I posted only 2 examples with full detail, but I included all positions. The error appears for each sample, when the variant is present.
Unless there is a systematic cause of the error which could be fixed, I believe we will keep discovering new "faulty" positions as they are present in various samples.
Again, using most update version 2.13 with "properly" updated HG38 1.47.1 under Win10, Chrome for gui. Will switch on Ubuntu to see if it's OS related, but I doubt this.

Best regards,
Consuel

So, here are the errors:

Traceback (most recent call last):
File "C:\Program Files\OpenCRAVAT\python\lib\site-packages\cravat\base_mapper.py", line 307, in run_as_slave
crx_data = self.map(crv_data)
File "F:\open-cravat\modules\mappers\hg38\hg38.py", line 1037, in map
so, achange, cchange, coding = self._get_snv_map_data(
File "F:\open-cravat\modules\mappers\hg38\hg38.py", line 1252, in _get_snv_map_data
so, ref_aanum, alt_aanum = self._get_svn_cds_so(
File "F:\open-cravat\modules\mappers\hg38\hg38.py", line 5788, in _get_svn_cds_so
basebits = (seq[seqbyteno] >> (6 - seqbitno)) & 0b00000011
IndexError: index out of range

LINE:4612
INPUT:289742 chr7 56081717 G A
ERROR:index out of range

Traceback (most recent call last):
File "C:\Program Files\OpenCRAVAT\python\lib\site-packages\cravat\base_mapper.py", line 307, in run_as_slave
crx_data = self.map(crv_data)
File "F:\open-cravat\modules\mappers\hg38\hg38.py", line 1037, in map
so, achange, cchange, coding = self._get_snv_map_data(
File "F:\open-cravat\modules\mappers\hg38\hg38.py", line 1252, in _get_snv_map_data
so, ref_aanum, alt_aanum = self._get_svn_cds_so(
File "F:\open-cravat\modules\mappers\hg38\hg38.py", line 5788, in _get_svn_cds_so
basebits = (seq[seqbyteno] >> (6 - seqbitno)) & 0b00000011
IndexError: index out of range

LINE:24728
INPUT:323759 chr8 133060118 G T
ERROR:index out of range

LINE:20176
INPUT:618237 chr19 52801310 T C
ERROR:index out of range

LINE:26709
INPUT:618790 chr20 13072370 T G
ERROR:index out of range

LINE:26905
INPUT:888011 chr21 37165716 T C
ERROR:index out of range

Originally posted by @clinicalngs in #345

[kmoad]

I expect that I can find the pattern of when this is occurring. Thanks for providing some examples. I'll reach out if more are needed.

[EugeneEA]

@kmoad Is there any update on that issue? It seems that I have somesing simmilar (using the latest cravat through docker image (karchinlab/opencravat:2.16.0) and install all base and ncessary modules to the mounted directory ):

[cariaso]

I notice this in a log, and wonder if it might be related

2025/11/09 11:54:29 cravat.mapper mapper database: /mnt/modules/mappers/hg38/data/gene_33_10000.sqlite
2025/11/09 11:54:30 cravat.mapper started: Sun Nov 9 11:54:30 2025 | 0
2025/11/09 12:10:11 cravat.mapper Traceback (most recent call last):
File "/srv/venv/lib64/python3.13/site-packages/cravat/base_mapper.py", line 301, in run_as_slave
crx_data = self.map(crv_data)
File "/mnt/modules/mappers/hg38/hg38.py", line 886, in map
tr_map_starts = self._get_tr_map_data(chrom, gpos)
File "/mnt/modules/mappers/hg38/hg38.py", line 4936, in _get_tr_map_data
self.c.execute(q)
~~~~~~~~~~~~~~^^^
sqlite3.OperationalError: near "-": syntax error
2025/11/09 12:10:11 cravat.mapper finished: Sun Nov 9 12:10:11 2025 | 0
2025/11/09 12:10:11 cravat.mapper runtime: 941.753

[kmoad]

This one looks to be using the older sqlite gene_33_10000.sqlite

What version of mapper is this on?

[cariaso]

I see this:
cravat started: Mon Nov 10 17:32:46 2025
cravat input files: /tmp/opencravat-open-cravat-1762795963l3y8bo97/opencravat-input.txt.retroheader.txt
cravat input assembly: hg19
cravat version: open-cravat 2.17.0 /srv/venv/lib64/python3.13/site-packages/cravat
cravat version: clinvar 2023.02.01 /mnt/modules/annotators/clinvar
cravat version: clingen 1.0.1 /mnt/modules/annotators/clingen
cravat version: loftool 3.6.1 /mnt/modules/annotators/loftool
cravat version: dbsnp 154.0.2 /mnt/modules/annotators/dbsnp
cravat version: gnomad3 1.1.0 /mnt/modules/annotators/gnomad3
cravat version: omim 1.0.0 /mnt/modules/annotators/omim
cravat version: pharmgkb 2.3.8 /mnt/modules/annotators/pharmgkb
cravat version: phdsnpg 0.0.9 /mnt/modules/annotators/phdsnpg
cravat version: sift 1.2.0 /mnt/modules/annotators/sift
cravat version: thousandgenomes 4.1.0 /mnt/modules/annotators/thousandgenomes
cravat version: clingen_allele_registry 1.0.0 /mnt/modules/annotators/clingen_allele_registry
cravat version: hg38 1.10.4 /mnt/modules/mappers/hg38
cravat.converter started: Mon Nov 10 17:32:46 2025
cravat.converter Input file(s): /tmp/opencravat-open-cravat-1762795963l3y8bo97/opencravat-input.txt.retroheader.txt
cravat.converter liftover from hg19
cravat.converter input format: 23andme
cravat.converter error lines: 159
cravat.converter finished: Mon Nov 10 17:36:48 2025
cravat.converter num input lines: 160815
cravat.converter runtime: 228.355
cravat num_workers: 1
cravat input line chunksize=160655 total number of input lines=160655 number of chunks=1
cravat.mapper input file: /tmp/opencravat-open-cravat-1762795963l3y8bo97/outdir/opencravat-input.txt.retroheader.txt.crv
cravat.mapper mapper database: /mnt/modules/mappers/hg38/data/gene_33_10000.sqlite
cravat.mapper started: Mon Nov 10 17:37:47 2025 | 0
cravat.mapper finished: Mon Nov 10 17:37:58 2025 | 0
cravat.mapper runtime: 11.064
cravat num_workers: 1
cravat.gnomad3 started: Mon Nov 10 17:37:59 2025
cravat.gnomad3 opening sqlite via connect_uri='file:/mnt/modules/annotators/gnomad3/data/gnomad3.sqlite?immutable=1'
cravat.gnomad3 finished: Mon Nov 10 19:29:35 2025
cravat.gnomad3 runtime: 6696.445s
cravat.clingen started: Mon Nov 10 19:29:35 2025
cravat.clingen finished: Mon Nov 10 19:29:35 2025
cravat.clingen runtime: 0.130s
cravat.dbsnp started: Mon Nov 10 19:29:35 2025
cravat.dbsnp opening sqlite via connect_uri='file:/mnt/modules/annotators/dbsnp/data/dbsnp.sqlite?immutable=1'

[DCGenomics]

@kmoad @cariaso -- flagging this as important

[cariaso]

I've had the 'aha' that in my case, the error message mentions
/mnt/modules/mappers/hg38/hg38.py
which was built quite a while ago, via opencravat 2.3.0, and then saved as an aws snapshot. That is then reused with runs, which have gradually moved to newer releases of opencravat. But based on the '.py' filename, that isn't just data it's also code. So it seem very plausible that /mnt/modules/mappers/hg38/hg38.py is quite old, and being used with bleeding edge code.

I'm now freshening up to a latest & greatest version of that snapshot. I'll update this when I learn more.

While doing so, I've noted these errors

/usr/local/bin/oc module install -y ftdna-converter
ERROR
argument of type 'Requirement' is not iterable
Repeat command with --debug for more details
/usr/local/bin/oc module install -y ancestrydna-converter
ERROR
argument of type 'Requirement' is not iterable
Repeat command with --debug for more details

[cariaso]

super fresh build of modules. I see a bit more traceback info

Traceback (most recent call last):
  File "/srv/venv/lib64/python3.13/site-packages/cravat/base_mapper.py", line 301, in run_as_slave
    crx_data = self.map(crv_data)
  File "/mnt/modules/mappers/hg38/hg38.py", line 1037, in map
    so, achange, cchange, coding = self._get_snv_map_data(
                                   ~~~~~~~~~~~~~~~~~~~~~~^
        tid,
        ^^^^
    ...<18 lines>...
        exonno,
        ^^^^^^^
    )
    ^
  File "/mnt/modules/mappers/hg38/hg38.py", line 1252, in _get_snv_map_data
    so, ref_aanum, alt_aanum = self._get_svn_cds_so(
                               ~~~~~~~~~~~~~~~~~~~~^
        tid, cpos, cstart, tpos, tstart, tr_alt_base, apos
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/mnt/modules/mappers/hg38/hg38.py", line 5783, in _get_svn_cds_so
    basebits = (seq[seqbyteno] >> (6 - seqbitno)) & 0b00000011
                ~~~^^^^^^^^^^^
IndexError: index out of range
2025/11/20 21:40:53 cravat.mapper        Traceback (most recent call last):
  File "/srv/venv/lib64/python3.13/site-packages/cravat/base_mapper.py", line 301, in run_as_slave
    crx_data = self.map(crv_data)
  File "/mnt/modules/mappers/hg38/hg38.py", line 889, in map
    tr_map_starts = self._get_tr_map_data(chrom, gpos)
  File "/mnt/modules/mappers/hg38/hg38.py", line 4948, in _get_tr_map_data
    self.c.execute(q)
    ~~~~~~~~~~~~~~^^^
sqlite3.OperationalError: near "-": syntax error

[cariaso]

The sqlite errors are due to mishandling of exotic contig names:

sqlite error no such table: transcript_frags_chr22_KI270736v1_random (q='\n select * from transcript_frags_chr22_KI270736v1_random\n where binno=15 and start<=151279 and end>=151279 order by tid\n ')

sqlite error near "-": syntax error (q='\n select * from transcript_frags_chrHLA-DRB1*13:02:01\n where binno=0 and start<=4429 and end>=4429 order by tid\n ')

The index errors are related to handling SNVs, here are 2 examples

vcf src

chr7 56081717 . G A 266 PASS SNVHPOL=3;MQ=60 GT:GQ:GQX:DP:DPF:AD:ADF:ADR:SB:FT:PL 0/1:225:30:47:0:20,27:10,13:10,14:-33.1:PASS:301,0,222 .:.:.:.:.::::.::.,.,.

triggers

IndexError during ( tid=446428422 cpos=352 cstart=314 tpos=483 tstart=445 tr_ref_base='C' tr_alt_base='T' strand=-1 kind=8 apos=118 strand_gpos=56081717 start=56081717 end=56081755 chrom='chr7' fragno=11 lenref=1 lenalt=1 prevcont=0 nextcont=0 exonno=4

vcf src

chr20 13072370 . T G 554 PASS SNVHPOL=3;MQ=60 GT:GQ:GQX:DP:DPF:AD:ADF:ADR:SB:FT:PL 1/1:120:30:41:1:0,41:0,22:0,19:-64.3:PASS:370,123,0 .:.:.:.:.::::.::.,.,.

triggers

IndexError during ( tid=434210263 cpos=418 cstart=304 tpos=571 tstart=457 tr_ref_base='T' tr_alt_base='G' strand=1 kind=8 apos=140 strand_gpos=13072370 start=13072256 end=13072371 chrom='chr20' fragno=15 lenref=1 lenalt=1 prevcont=0 nextcont=0 exonno=3

[cariaso]

cravat/cravat_class.py

line 1372 needs a pool.join()

as now found at
#377