feat: add multithreading/multiprocessing acceleration - #3
Closed
Zac-lzh wants to merge 2 commits into
Closed
Conversation
Add CPU-based parallelization for compute-intensive operations: - New parallel.py module with ProcessPoolExecutor/ThreadPoolExecutor utilities - Vectorize Spearman ranking in compute_correlation_matrix (1.9x speedup) - Parallelize test_soft_powers power-value loop using threads - Parallelize construct_tf_network per-gene XGBoost loop using threads - Vectorize regulon_scores per-cell ranking loop (eliminates Python loop) - Wire n_threads parameter through construct_network to compute_tom/compute_kme - Add n_threads parameter to HDWGCNA class wrapper methods Addresses issue #1: multithreading support for hdWGCNA. Changes: - py_hdWGCNA/parallel.py (new): parallelization utilities - py_hdWGCNA/utils.py: vectorized Spearman, n_threads for TOM/kME/correlation - py_hdWGCNA/network.py: parallel power testing, n_threads wiring - py_hdWGCNA/tf_network.py: parallel TF network, vectorized regulon scores - py_hdWGCNA/hdWGCNA.py: n_threads parameter in wrapper methods
Add _kernels.py module with optimized implementations: - Use numpy BLAS for matrix multiply (already multi-threaded SIMD) - Use vectorized scipy.stats.rankdata for Spearman (1.7-2.1x speedup) - Graceful fallback if numba available but use numpy BLAS path Benchmark results (2000 genes): - Spearman: 1.7x speedup via vectorized rankdata - Power testing: 2.4x speedup via ThreadPoolExecutor - TOM: numpy BLAS already optimal (adj @ adj) - Full pipeline: 1.1-1.5x speedup Note: The main bottleneck (adj @ adj matrix multiply) is already handled by numpy's BLAS library (MKL/OpenBLAS) which uses multiple threads and SIMD instructions internally.
Collaborator
Author
|
Closed by author. The speedup from Python-level parallelization is limited since numpy BLAS (MKL/OpenBLAS) already uses multiple threads internally for matrix multiply. Significant acceleration would require C++ extensions with a compiler or GPU (cupy). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses issue #1: adds CPU-based multithreading/multiprocessing acceleration for compute-intensive operations in py-hdWGCNA.
Changes
parallel.pymodule: ProcessPoolExecutor/ThreadPoolExecutor utilitiesscipy.stats.rankdata(axis=1)(~1.9x speedup)np.argsortn_threadsparameter: Flows throughconstruct_network→compute_tom/compute_kme/compute_correlation_matrixn_threadsparameter totest_soft_powers,construct_network,construct_tf_network,regulon_scoresApproach
concurrent.futures.ThreadPoolExecutorfor numpy-heavy operations (numpy releases GIL)concurrent.futures.ProcessPoolExecutorfor CPU-bound Python codeOMP_NUM_THREADS/MKL_NUM_THREADSenvironment variablesconcurrent.futures)Verification
n_threadsparameter defaults toNone(auto-detect CPU count)Test plan
python -m pytest tests/ -v(20/20 pass)