Fix dedupe() crash when an item is emptied by the processor - #95
Open
Sanjays2402 wants to merge 1 commit into
Open
Fix dedupe() crash when an item is emptied by the processor#95Sanjays2402 wants to merge 1 commit into
Sanjays2402 wants to merge 1 commit into
Conversation
process.dedupe() calls extractBests() for each item and then max() over the returned matches. When an item's processed form is empty (e.g. "###", which utils.full_process reduces to ""), every comparison scores 0 and extractBests() returns no matches at the default threshold of 70 - not even the item itself. max() is then called on an empty sequence and raises "ValueError: max() iterable argument is empty", crashing dedupe() on otherwise valid input. The function already warns that a processor-emptied query makes all comparisons score 0, but still called max() unconditionally. An item with no matches has no fuzzy duplicates, so keep it as its own cluster (add the item itself to the deduped set) instead of crashing. This leaves the result for all non-empty items unchanged. Add a regression test that reproduces the crash on dedupe(["###", "apple", "apple pie"]) and asserts the empty-processed item is preserved while the genuine duplicates still collapse. Fixes seatgeek#94
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #94
The bug
process.dedupe()crashes withValueError: max() iterable argument is emptyon otherwise valid input when one of the items is reduced to an empty string by the default processor:(
utils.full_process('###')yields''.)Root cause
dedupe()callsextractBests()per item and thenmax()over the returned matches:When an item's processed form is empty, every comparison scores 0, so at the default
threshold=70extractBests()returns[]— not even the item itself matches.max([])then raises. The code even emits a warning that a processor-emptied query makes all comparisons score 0, but still callsmax()unconditionally.Confirmed with the real code:
The fix
An item with no matches has no fuzzy duplicates, so keep it as its own cluster (add the item itself to the
dedupedset) instead of crashing. This is exactly the behavior the issue asks for, and leaves the result for every non-empty item unchanged.Before/after:
dedupe(['###', 'apple', 'apple pie'])ValueError['###', 'apple pie']dedupe(['###', '!!!', '@@@'])ValueError['###', '!!!', '@@@'](each kept)dedupe(['Frodo Baggin', 'Frodo Baggins', 'F. Baggins', ...])dedupe(['Tom', 'Dick', 'Harry'])(no dupes)Regression test
Added
test_dedupe_with_empty_processed_item, which reproduces the crash and asserts the empty-processed item is preserved while the genuine duplicates still collapse. Proof it guards the bug:FAILED ... ValueError: max() iterable argument is emptyatthefuzz/process.py:441.1 passed.Full suite green:
50 passedintest_thefuzz.py(including theTestCodeFormatpycodestyle gate) andtest_thefuzz_pytest.pypasses. The change adds no new pycodestyle errors inthefuzz/(baseline and post-change both 0 non-E501). Diff is +30/−0 acrossthefuzz/process.pyandtest_thefuzz.py.