Skip to content

fix: hash-based encoding for object-dtype join/groupby keys (v2.2.5) - #32

Merged
dsfulf merged 1 commit into
mainfrom
fix/object-dtype-mixed-types
May 26, 2026
Merged

fix: hash-based encoding for object-dtype join/groupby keys (v2.2.5)#32
dsfulf merged 1 commit into
mainfrom
fix/object-dtype-mixed-types

Conversation

@dsfulf

@dsfulf dsfulf commented May 26, 2026

Copy link
Copy Markdown
Member

Summary

  • Fix: left_join / inner_join / group_by on object-dtype key columns containing heterogeneous Python types (e.g., str and int across sides, or mixed within one side) no longer raise TypeError: '<' not supported between instances of ....
  • Root cause: the pure-Python encoding fallback in _encode_columns and _encode_columns_paired calls np.unique(..., return_inverse=True), which internally argsorts the values. Object arrays with mixed Python types fail cross-type < comparison. The C accelerator (_c_encode_strings) handles this correctly via hash equality but only triggers at >= 50k rows; _validate_dtypes lets both sides through because _dtypes='object' on both sides is an equality match.
  • Fix: added GroupSet._hash_encode_object — a dict-based codebook used for object-dtype arrays in the pure-Python path. Semantics match _c_encode_strings (hash equality, not ordering).
  • Also fixes a pre-existing mypy return-value error in Tafra.to_pandas via cast(DataFrame, ...) — the protocol is intentionally a "fake class to satisfy typing of a pandas.DataFrame without a dependency" and can't structurally match pandas's actual signatures.
  • Documents the three version-bump files (pyproject.toml, recipe/meta.yaml, docs/changelog.md) in CLAUDE.md.

Test plan

  • Three new regression tests in TestStringDtype:
    • test_join_on_object_dtype_mixed_python_types_across_sides
    • test_join_on_object_dtype_mixed_within_side
    • test_group_by_object_dtype_mixed_python_types
  • Full suite: 204 passed
  • ruff check tafra — clean
  • mypy tafra — clean (was 1 pre-existing error, now resolved)

Generated with Claude Code

np.unique's internal argsort raises TypeError on object arrays mixing
Python types (e.g., str and int) because cross-type `<` is unsupported.
The pure-Python encoding paths in _encode_columns and _encode_columns_paired
now use a dict-based codebook for object dtype, matching the C accelerator's
hash-equality semantics. Fixes joins/groupbys that previously failed when
both sides had _dtypes='object' with heterogeneous Python values.

Also fixes a pre-existing mypy return-type error in to_pandas via cast(),
and documents the three version-bump files in CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes TypeError raised when left_join / inner_join / group_by operate on object-dtype key columns containing mixed Python types (e.g., str and int). The pure-Python encoding fallback used np.unique(..., return_inverse=True), which internally argsorts and fails on cross-type <. A new dict-based hash codebook (GroupSet._hash_encode_object) is added with semantics matching the C accelerator (_c_encode_strings), and used for object arrays in both single-column (_encode_columns) and paired (_encode_columns_paired) paths.

Changes:

  • Add GroupSet._hash_encode_object() and route object-dtype arrays through it in _encode_columns and _encode_columns_paired.
  • Wrap Tafra.to_pandas() return in cast(DataFrame, ...) to silence a pre-existing mypy return-value error against the lightweight DataFrame Protocol.
  • Bump version to 2.2.5 across pyproject.toml, recipe/meta.yaml, docs/changelog.md; document the three bump locations in CLAUDE.md.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tafra/group.py New hash-based object encoder; used in single-column and paired encoding paths below the C threshold
tafra/base.py cast(DataFrame, ...) around pd.DataFrame(...) return in to_pandas
test/test_tafra.py Three regression tests for mixed-type object-dtype joins and group_by
docs/changelog.md 2.2.5 release notes
pyproject.toml Version bump 2.2.4 → 2.2.5
recipe/meta.yaml Conda recipe version bump 2.2.4 → 2.2.5
CLAUDE.md Documents the three version-bump locations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@dsfulf
dsfulf merged commit 390157d into main May 26, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants