Parallelize clustering distance calculation - #279
Open
tmsincomb wants to merge 4 commits into
Open
Conversation
Add deterministic Hypothesis differential coverage, bounded exhaustive cases, and threaded boundary checks. Preserve the legacy kernel for unsupported values so optimized clustering keeps exact behavior.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR accelerates clustering by replacing Python-level pairwise Levenshtein computations with RapidFuzz’s native (optionally threaded) distance matrix computation, while preserving legacy behavior—including somatic padding semantics and output equivalence.
Changes:
- Reworks
Cluster._get_distance_dfto use RapidFuzzcdist(threaded above a row-count threshold) with a guarded legacy fallback for edge/malformed inputs. - Implements somatic-padding subtraction via an inverted mutation→row index and clamps distances to preserve legacy non-negativity behavior.
- Expands unit coverage with focused regression tests and a large differential/property-based suite; adds
greedymarker and Hypothesis dependency/ignores.
Reviewed changes
Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/sadie/cluster/cluster.py | Adds RapidFuzz-based fast path for distance matrices with somatic-padding support and a legacy fallback. |
| tests/unit/cluster/test_cluster.py | Adds targeted regression tests for threaded distance behavior and linked somatic padding/output preservation. |
| tests/unit/cluster/test_cluster_differential.py | Adds extensive differential/property-based equivalence testing vs the legacy kernel. |
| pytest.ini | Adds greedy marker and ignores .hypothesis artifacts. |
| pyproject.toml | Adds RapidFuzz runtime dependency and Hypothesis dev dependency. |
| poetry.lock | Locks added Hypothesis (and transitive) dependencies. |
| .gitignore | Ignores .hypothesis/ directory. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+152
to
+171
| stable_types = (str, bytes, bool, int, float, complex, type(None)) | ||
| use_fast_path = len(df) > 0 and all( | ||
| type(index) in stable_types or index is pd.NA or index is pd.NaT for index in df.index | ||
| ) | ||
| if self.pad_somatic and len(self.pad_somatic_values) not in (1, 2): | ||
| use_fast_path = False | ||
| if use_fast_path: | ||
| use_fast_path = all( | ||
| type(row[metric]) in stable_types or row[metric] is pd.NA or row[metric] is pd.NaT | ||
| for row in df_lookup.values() | ||
| for metric in self.lookup | ||
| ) | ||
| if use_fast_path and self.pad_somatic: | ||
| use_fast_path = all( | ||
| type(mutations) in (list, tuple, np.ndarray) | ||
| and not (type(mutations) is np.ndarray and mutations.ndim != 1) | ||
| and all(type(mutation) in (str, np.str_) and "\0" not in mutation for mutation in mutations) | ||
| for row in df_lookup.values() | ||
| for mutations in (row[metric] for metric in self.pad_somatic_values) | ||
| ) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pad_somaticPerformance
Validation
Closes #278