Repository navigation
Release 4.0.2: vectorize reciprocal ANI averaging (issue #308) - #309
Merged
Merged
Conversation
add_avani looped over every Ndb row with iterrows() into a string-keyed dict. On a single large primary cluster (e.g. 28k same-species genomes) that is ~811M rows, taking hours and far too much memory. Encode each (reference, querry) pair as an integer and look up reverse pairs in bulk instead. Output is identical, including duplicate and NaN handling and the KeyError on a missing reverse pair; clustering a 2,000-genome primary cluster goes from 133s to 6s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013UpcPFwkh6QMusnD6ApThL
Bump the version and date the changelog. The fix is the vectorized reciprocal ANI averaging that makes large single-species primary clusters cluster ~25-30x faster (issue #308). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013UpcPFwkh6QMusnD6ApThL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up secondary clustering on large primary clusters. This matters most for same-species datasets, where everything falls into one primary cluster (#308). Also bumps the version to 4.0.2.
add_avani(drep/d_cluster/external.py) averages the two directions of each ANI comparison before clustering. It used to calliterrows()on every row of Ndb and build a string-keyed dict. For one 28k-genome primary cluster that's ~811M rows: hours of runtime and far too much memory. It now encodes each (reference, querry) pair as an integer and looks up the reverse pairs in bulk.KeyErrorraised when a reverse pair is missing.drep/VERSIONgoes to 4.0.2, and the changelog gets a dated 4.0.2 entry.Performance
_cluster_Ndbon synthetic single-cluster edge tables:build_ndb_from_edgesis now the bottleneck on this path. It took 43 s and ~7 GB at 4k genomes, and both grow quadratically. Making 28k same-species genomes truly tractable would need secondary clustering to build its distance matrix directly instead of the long-form Ndb. That's out of scope here.Testing
add_avani: gave identicalav_anion a 3,630-row table that included duplicate pairs, a NaN, and mixed path/bare genome names.dRep dereplicateon 400 simulated same-species genomes completes. Re-clustering its Ndb with the original settings reproduces the original clusters and winners exactly.tests/tests/test_cluster.py: passes excepttest_goANIandtest_goANI2, which need nsimscan (not installed in the test environment).tests/tests: 67 passed, 11 skipped, 2 failed. Neither failure comes from this change:test_dereplicate_3needs nsimscan.test_drep_scaffold_levelfails the same way with the oldadd_avani, and that script doesn't call it.🤖 Generated with Claude Code
https://claude.ai/code/session_013UpcPFwkh6QMusnD6ApThL
Generated by Claude Code