How do I parameterize the index data type of a Series?

[gandhis1]

If I have a Series that has a DateTimeIndex and a float data type, is there a syntax I can use to represent that?

[Dr-Irv]

If you do s = pd.Series([1,3,4], index=pd.date_range("1/1/2022", "1/3/2022"), dtype=float), then the type that is tracked by the stubs will be Series[float]. We don't currently track the type of the supporting index.

Note that you can't do s: pd.Series[float] = pd.Series([1,2,3]]) While the type checker will be OK with that, the code won't execute because pandas doesn't use generic types on Series

[twoertwein]

from __future__ import annotations should make s: pd.Series[float] = pd.Series([1,2,3]]) work as python will not evaluate the annotation at runtime.

[jorenham]

I'm aware of that, but using from __future__ import annotations effectively stringifies all annotations. Static typecheckers probably won't mind, but the problem still persists at runtime:

>>> from __future__ import annotations
>>> import pandas as pd
>>>
>>> def get_series() -> pd.Series[float]:
...     return pd.Series([1 / 137])
... 
>>> get_series.__annotations__
{'return': 'pd.Series[float]'}
>>>
>>> from typing import get_type_hints
>>> get_type_hints(get_series)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/typing.py", line 2246, in get_type_hints
    hints[name] = _eval_type(value, globalns, localns)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/typing.py", line 400, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/typing.py", line 907, in _evaluate
    eval(self.__forward_code__, globalns, localns),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
TypeError: type 'Series' is not subscriptable

But without the __future__ import and the monkeypatch, this is the result:

>>> import pandas as pd
>>>
>>> pd.Series.__class_getitem__ = classmethod(lambda cls, arg: cls)
>>> def get_series() -> pd.Series[float]:
...     return pd.Series([1 / 137])
... 
>>> from typing import get_type_hints
>>> get_type_hints(get_series)
{'return': <class 'pandas.core.series.Series'>}
>>> 
>>> def get_other_series() -> 'pd.Series[str]':
...     return pd.Series(['1 / 137'])
>>> get_type_hints(get_other_series)
{'return': <class 'pandas.core.series.Series'>}

So this is still not an ideal solutions, because the type argument disappears. That's why I suggested the use of Annotated earlier.

I'm afraid that PEP 649 isn't going to help with this either. All it will do is defer the evaluation. Once the annotations are accessed, the __class_getitem__ method will be called either way. But it could also fail faster than that; I haven't tried it yet.

Anyway, to conclude: Both approaches hide the underlying problem, one a bit better than the other. So I suppose that the complete solution would involve making pd.Series fundamentally generic, i.e. in pandas, not in pandas-stubs.

[jorenham]

on a semi-related note, I just found out that typing.get_type_hints will evaluate any stringified expression:

>>> def evil() -> 'print("666")': ...
... 
>>> get_type_hints(evil)
666
{'return': <class 'NoneType'>}

[twoertwein]

I'm not sure whether you want to evaluate all pandas-stubs annotations as pandas-stubs uses some non-existing helper classes.

I suppose that the complete solution would involve making pd.Series fundamentally generic, i.e. in pandas, not in pandas-stubs.

I agree, but don't expect that to happen anytime soon. It would require that all internal pandas code also uses the dtype (that is a lot of work and in many cases it would be Series[Any] - I'm quite sure some people wouldn't like the idea of littering that all over the internal pandas code). Otherwise, mypy would complain(?) everywhere within pandas about missing type annotations.

[jorenham]

I'm not sure whether you want to evaluate all pandas-stubs annotations as pandas-stubs uses some non-existing helper classes.

I guess I'm trying to say that, when someone is using the documented runtime typing API, on some pandas-stubs annotations, that it shouldn't crash. Otherwise it won't be possible to use tools like beartype.

It would require that all internal pandas code also uses the dtype [...]

The quick 'n dirty fix would be to have a pandas.Series.__class_getitem__ that returns a typing._GenericAlias, so that it'll work as expected when used in typing.get_type_hints & co.