Skip to content

Fix dedupe() crash when an item is emptied by the processor - #95

Open
Sanjays2402 wants to merge 1 commit into
seatgeek:masterfrom
Sanjays2402:fix/dedupe-empty-processed-item
Open

Fix dedupe() crash when an item is emptied by the processor#95
Sanjays2402 wants to merge 1 commit into
seatgeek:masterfrom
Sanjays2402:fix/dedupe-empty-processed-item

Conversation

@Sanjays2402

Copy link
Copy Markdown

Fixes #94

The bug

process.dedupe() crashes with ValueError: max() iterable argument is empty on otherwise valid input when one of the items is reduced to an empty string by the default processor:

from thefuzz.process import dedupe
dedupe(['###', 'apple', 'apple pie'])
# ValueError: max() iterable argument is empty

(utils.full_process('###') yields ''.)

Root cause

dedupe() calls extractBests() per item and then max() over the returned matches:

for item in contains_dupes:
    matches = extractBests(item, contains_dupes, scorer=scorer, score_cutoff=threshold, limit=None)
    deduped.add(max(matches, key=lambda x: (len(x[0]), x[0]))[0])

When an item's processed form is empty, every comparison scores 0, so at the default threshold=70 extractBests() returns [] — not even the item itself matches. max([]) then raises. The code even emits a warning that a processor-emptied query makes all comparisons score 0, but still calls max() unconditionally.

Confirmed with the real code:

>>> extractBests('###', ['###', 'apple', 'apple pie'], score_cutoff=70, limit=None)
[]                       # -> max([]) raises
>>> extractBests('apple', ['###', 'apple', 'apple pie'], score_cutoff=70, limit=None)
[('apple', 100), ('apple pie', 100)]

The fix

An item with no matches has no fuzzy duplicates, so keep it as its own cluster (add the item itself to the deduped set) instead of crashing. This is exactly the behavior the issue asks for, and leaves the result for every non-empty item unchanged.

matches = extractBests(item, contains_dupes, scorer=scorer, score_cutoff=threshold, limit=None)
if not matches:
    deduped.add(item)
    continue
deduped.add(max(matches, key=lambda x: (len(x[0]), x[0]))[0])

Before/after:

call before after
dedupe(['###', 'apple', 'apple pie']) ValueError ['###', 'apple pie']
dedupe(['###', '!!!', '@@@']) ValueError ['###', '!!!', '@@@'] (each kept)
dedupe(['Frodo Baggin', 'Frodo Baggins', 'F. Baggins', ...]) works unchanged
dedupe(['Tom', 'Dick', 'Harry']) (no dupes) returns original list unchanged

Regression test

Added test_dedupe_with_empty_processed_item, which reproduces the crash and asserts the empty-processed item is preserved while the genuine duplicates still collapse. Proof it guards the bug:

  • Without the source fix (test present): FAILED ... ValueError: max() iterable argument is empty at thefuzz/process.py:441.
  • With the fix: 1 passed.

Full suite green: 50 passed in test_thefuzz.py (including the TestCodeFormat pycodestyle gate) and test_thefuzz_pytest.py passes. The change adds no new pycodestyle errors in thefuzz/ (baseline and post-change both 0 non-E501). Diff is +30/−0 across thefuzz/process.py and test_thefuzz.py.

process.dedupe() calls extractBests() for each item and then max() over
the returned matches. When an item's processed form is empty (e.g. "###",
which utils.full_process reduces to ""), every comparison scores 0 and
extractBests() returns no matches at the default threshold of 70 - not even
the item itself. max() is then called on an empty sequence and raises
"ValueError: max() iterable argument is empty", crashing dedupe() on
otherwise valid input.

The function already warns that a processor-emptied query makes all
comparisons score 0, but still called max() unconditionally.

An item with no matches has no fuzzy duplicates, so keep it as its own
cluster (add the item itself to the deduped set) instead of crashing. This
leaves the result for all non-empty items unchanged.

Add a regression test that reproduces the crash on
dedupe(["###", "apple", "apple pie"]) and asserts the empty-processed item
is preserved while the genuine duplicates still collapse.

Fixes seatgeek#94
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dedupe() crashes with max() empty when an item reduces to an empty string under the default processor

1 participant