DeepSomatic 1.10.0 make_examples_somatic is OOM-killed on a 25 kb ONT tumor-only region due to unbounded read materialization

[VasLem]

make_examples_somatic reproducibly fails with Linux OOM kill on a single 25 kb ONT tumor-only region:
chr16:46375000-46400000

The subset from this bam (available for 7 days) is approximately 1.88 GiB and contains:
135704 alignments from samtools view -c
71477 primary reads from samtools stats
741038496 mapped bases from CIGAR
average read length around 9469
maximum read length 1482710

Environment

Observed with DeepSomatic / DeepVariant 1.10.0.
The make_examples_somatic.zip payload reports:
DEEP_VARIANT_VERSION = '1.10.0'
The DeepVariant image inspected was:
gcr.io/deepvariant-docker/deepvariant:gh1060
Digest:
sha256:960bc2c307f9cbc64a9ae56bb21f62fd3964af1f079397f92bbcbe88cb233122
Relevant command configuration
The failing run uses ONT tumor-only make_examples_somatic with:
--model_type=ONT_TUMOR_ONLY
--regions chr16:46375000-46400000
--num_shards=1
--make_examples_task=0
--steps_to_run=make_examples
--partition_size=25000
--max_reads_per_partition=0
--phase_reads=true
--enable_methylation_calling=true
--track_ref_reads=true
--realign_reads=false
--sort_by_haplotypes=true
--trim_reads_for_pileup=true

Observed behavior

The process logs:
Processing chr16:46374999-46399999
It is then killed by the kernel with exit code 247.
On a 32 GiB machine:
peak memory: 33099112448
cgroup memory event: oom_kill 1
A reproduction using the same 25 kb BAM subset on a 128 GiB high-memory machine also failed:
exit code: 247
peak memory: 133914116096
cgroup memory event: oom_kill 1
No Python exception, Abseil fatal message, or C++ assertion is emitted before the process is killed.
Where the failure appears to occur
The last emitted DeepVariant debug line is the Processing chr16:46374999-46399999 message immediately before RegionProcessor.process(...).
In the shipped make_examples_core.py, the no-realignment path queries reads overlapping the region and then materializes them with list(reads). Read sampling is only applied when max_reads_per_partition > 0 or a dynamic base cap is active.
For this ONT tumor-only configuration, max_reads_per_partition=0, so all reads overlapping the 25 kb interval are materialized. With track_ref_reads=true, the reads are then also passed through allele-counter logic. phase_reads=true and enable_methylation_calling=true both require track_ref_reads=true, so this appears to be a default high-memory path for ONT tumor-only calling in this configuration.

Expected behavior

DeepSomatic should either:
complete processing of the 25 kb region without requiring more than 128 GiB RAM, or
fail early with a clear diagnostic that the region exceeds supported read-count or read-bases limits.

Actual behavior

make_examples_somatic grows memory until the kernel OOM-kills the process, with no actionable DeepVariant/DeepSomatic error message.

Request

Please add a bounded-read or bounded-bases safeguard in make_examples_somatic for ONT tumor-only mode when max_reads_per_partition=0, especially with track_ref_reads, methylation calling, and read phasing enabled.
At minimum, it would be useful for make_examples_somatic to detect this condition before materializing all overlapping reads and emit a clear diagnostic containing the region, read count, total read bases, and the relevant cap/flag settings.

[kishwarshafin]

@VasLem can you please provide a few more questions:

• What ONT chemistry and basecaller you are using for this data?
• Why are you setting: --max_reads_per_partition=0? Is there any specific reason for this?

[VasLem]

Hi @kishwarshafin , sure:

• ONT Chemistry: R10.4.1
• Basecaller: Dorado v2.1.0
• Due to observed imbalanced coverage, we are using mosdepth to compute coverage in 1Mb bins, and split each chromosome into equal coverage chunks, and therefore cause all the parallelized per-chunk VMs to get similar amount of reads and have similar computational needs. The max_reads_per_partition was incorporated so that no information is missed in dense areas, accepting variable variant representation as a trade-off.

Having said that and accepting that I can just stop supplying that setting, a 1.88 GB BAM pointing at a rather small 25kb region cannot be crashing a 120GB VM, right?

[pichuan]

Hi @VasLem , if you're setting max_reads_per_partition to 0, that means there is no constraint on the number of reads loading into memory in the step where it needs.
With that, I'm not surprised that you're running OOM :-/

If you want to set max_reads_per_partition=0, you'll need to set max_reads_for_dynamic_bases_per_region.

For example, can you try something like:
max_reads_per_partition=0,max_reads_for_dynamic_bases_per_region=300

If you want to be conservative, you could try max_reads_per_partition=0,max_reads_for_dynamic_bases_per_region=3000 (but this will use more memory on a high coverage region)

[VasLem]

Hi @pichuan , could you please elaborate more on the usage of max_reads_for_dynamic_bases_per_region when I am not supplying candidate positions, only regions? From the documentation, it does not seem it is active, so it would not affect the analysis if set. If I enable use_candidate_partition I could potentially make use of it. Would that mean that instead of 25k bps partition size, I would get a partition containing at most max_reads_for_dynamic_bases_per_region reads?

[pichuan]

max_reads_for_dynamic_bases_per_region is not tied to candidate_positions. It works directly with regular --regions too.

When set to a value > 0, it computes a bases-based cap: max_bases_to_cover = max_reads_for_dynamic_bases_per_region × region_length. Reads are then reservoir-sampled until the total bases overlapping the region reach that cap.

So with --max_reads_per_partition=0 --max_reads_for_dynamic_bases_per_region=300 on your 25 kb region, it would allow up to 300 × 25,000 = 7.5M bases before downsampling kicks in. This is a bases budget, not a read count -- it naturally adapts to read length and region size, which is why it's particularly useful for long-read ONT data.

You do not need use_candidate_partition to use this flag.

If you can give it a try and let me know whether it works for you, that will be great. Thank you!