Hm, this is my take this far (can update the issue with this):
We have five type cases:
-
cases when pd.Series.dtype equality check works (what we have now implemented). It is quite liberally implemented on the numpy side, so this is easy to extend, with np.int8, np.int16, np.float, np.uint, ...
-
str types wrapped in np.object:s. (For the general case see (5)) Here one can use the speedy pd.Series.str accessor as I suggested earlier.
-
pandas series dtypes, CategoricalDype . Sort-of works with pd.Series.dtype but need special attention, i.e., for handling bare and parametrised version (failing test includes one each).
-
Index dtypes, needing a isinstance check
-
other np.object dtypes: i.e., anything. Basically cases where one need to check elements “manually and individually” (costly).
So requirements and importance (IMO),
-
(1) should work out of the box
-
(2) is pretty important and requires a simple and cheap function to be checked df.column_a.str.len().notnull.all()
-
(3) is very good to have (and one should use categorical both for typing and reduce memory footprint). However, I don’t use it anywhere at present, so not high prio for me 🙂
-
(4) is very good to have, I’d say. But that would entail that we revert to treat the column and index cases separately again.
-
(5) This I would say covers use-cases not super-common (i.e., when one puts dicts or lists or functions into the df) and is ok if not supported I think.
Implementation hardness:
-
What I think would make sense is a mapping type : check_function, like we have for the logical checks.
-
Likely one then need the distinction between index and columns
-
Given a ok structure of the two above, adding any new types should be very simple.
string check
just some notes:
-
seems hard to do cheaply without explit checks on the indicvidual values
-
We have pd.StringDtype, but it warns: "StringDtype is considered experimental. The implementation and
parts of the API may change without warning." SO nothing we should use in prod...
-
Not sure that the suggested accessor check pd.Series.str is actually C-vectorized... Also the suggested str.len() method actually seem to work with any objects (e.g. list and dict) with a len() defined... which points to that this method is just an expensive python iteration, and not a low-level thing.
Hm, this is my take this far (can update the issue with this):
We have five type cases:
cases when pd.Series.dtype equality check works (what we have now implemented). It is quite liberally implemented on the numpy side, so this is easy to extend, with np.int8, np.int16, np.float, np.uint, ...
str types wrapped in np.object:s. (For the general case see (5)) Here one can use the speedy pd.Series.str accessor as I suggested earlier.
pandas series dtypes, CategoricalDype . Sort-of works with pd.Series.dtype but need special attention, i.e., for handling bare and parametrised version (failing test includes one each).
Index dtypes, needing a isinstance check
other np.object dtypes: i.e., anything. Basically cases where one need to check elements “manually and individually” (costly).
So requirements and importance (IMO),
(1) should work out of the box
(2) is pretty important and requires a simple and cheap function to be checked df.column_a.str.len().notnull.all()
(3) is very good to have (and one should use categorical both for typing and reduce memory footprint). However, I don’t use it anywhere at present, so not high prio for me 🙂
(4) is very good to have, I’d say. But that would entail that we revert to treat the column and index cases separately again.
(5) This I would say covers use-cases not super-common (i.e., when one puts dicts or lists or functions into the df) and is ok if not supported I think.
Implementation hardness:
What I think would make sense is a mapping type : check_function, like we have for the logical checks.
Likely one then need the distinction between index and columns
Given a ok structure of the two above, adding any new types should be very simple.
string check
just some notes:
seems hard to do cheaply without explit checks on the indicvidual values
We have
pd.StringDtype, but it warns: "StringDtype is considered experimental. The implementation andparts of the API may change without warning." SO nothing we should use in prod...
Not sure that the suggested accessor check
pd.Series.stris actually C-vectorized... Also the suggestedstr.len()method actually seem to work with any objects (e.g. list and dict) with alen()defined... which points to that this method is just an expensive python iteration, and not a low-level thing.