Skip to content

Region processing speedup - #125

Open
thekugelmeister wants to merge 4 commits into
mainfrom
region-processing-speedup
Open

Region processing speedup#125
thekugelmeister wants to merge 4 commits into
mainfrom
region-processing-speedup

Conversation

@thekugelmeister

Copy link
Copy Markdown
Collaborator

Quick fixes to speed up regions_to_list, by omitting parallelization infrastructure when running with a single core.

Updates at a glance

  • split *_process_chunk functions into a processing function and a higher-level parallelization function, for both pileup counts and vectors.
  • updated the user-facing counts and vector functions to skip initializing worker pools and shared memory when running with a single core
  • For running regions_to_list on pileup_counts_from_bedmethyl, this results in ~300x speedup

Other findings

While implementing these fixes, a few things have become clear through testing:

  • Initializing worker pools and shared memory has significant overhead in the current implementation
  • Reopening tabix files with pysam has significant overhead

Below is a plot comparing processing speeds for a set of simplified tests (blue) and different versions of pileup counting methods (orange). In addition to identifying the main cause of the regions_to_list slowdown, this visualization suggests that there may be a ~2x speedup available if we avoid reopening tabix files as much as possible.

This change will be implemented as part of a future effort.
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant