Repository navigation
[Python] Handle NaN in DataFrame drop_duplicates, duplicated and nunique - #40571
anishmehta24 wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40571 +/- ##
=============================================
+ Coverage 56.16% 79.93% +23.76%
=============================================
Files 1124 488 -636
Lines 178336 82550 -95786
Branches 1489 0 -1489
=============================================
- Hits 100170 65983 -34187
+ Misses 75645 16567 -59078
+ Partials 2521 0 -2521
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| @frame_base.args_to_kwargs(pd.Series) | ||
| @frame_base.populate_defaults(pd.Series) | ||
| def nunique(self, **kwargs): | ||
| def nunique(self, dropna, **kwargs): |
There was a problem hiding this comment.
Looks like we changed the api signature here. Is it going to cause a compatibility issue?
There was a problem hiding this comment.
No, callers see the same signature as pandas, Series.nunique(dropna=True). args_to_kwargs maps arguments by the pandas signature, and populate_defaults only fills defaults for parameters the method names, so dropna has to be listed to get its default of True. Before, it went into **kwargs and was ignored. nunique(), nunique(False), nunique(True) and nunique(dropna=False) all match pandas (checked with _run_test). It's the same pattern as quantile(self, q, **kwargs).
| * (Python) Fixed stateful DoFns with side inputs sometimes taking the timer key coder from a side input instead of the main input, which could make the worker fail to decode timer keys with `Unknown type tag` ([#40374](https://github.com/apache/beam/issues/40374)). | ||
| * (Python) `Duration` built from float seconds now rounds to the nearest microsecond instead of truncating, which could lose a microsecond ([#40263](https://github.com/apache/beam/issues/40263)). | ||
| * (Python) `ReadFromText` with `escapechar` no longer skips a delimiter that directly follows an escaped delimiter, which merged two records into one ([#40459](https://github.com/apache/beam/issues/40459)). | ||
| <<<<<<< ours |
There was a problem hiding this comment.
Conflict in CHANGES.md needs to be resolved.
There was a problem hiding this comment.
Sorry, conflict markers slipped in when I rebased. Rebased on master and fixed in 4d80dbe.
37f569f to
4d80dbe
Compare
2 participants
Fixes #40570
drop_duplicatesandduplicatedcolocated duplicates withgroupby, which drops NaN keys (anddropna=Falsedoesn't work with multiple columns, #21014). They now add a hash of the compared columns as an index level and partition on that, so duplicates still end up together. -0.0 is normalized first since pandas treats it as a duplicate of 0.0 but it hashes differently.Series.nuniquenow honoursdropna.Added tests to frames_test.py (they fail on master with
KeyError) and a CHANGES.md entry. frames_test.py and pandas_doctests_test.py pass.Written with help from Claude.