[Feature Request]: Support pandas 3+ in the dataframes API #40709

Activity

1. jrmccluskey

   added

   new feature

   awaiting triage

   and removed

   awaiting triage

   on Oct 21, 2026
2. jrmccluskey

   self-assigned this

   on Oct 21, 2026
3. github-actions

   added

   python

   P2

   on Oct 21, 2026
4. alisha-1000 commented on Oct 23, 2026

   

   alisha-1000

   on Oct 23, 2026

   Contributor

   More actions

   Hi @jrmccluskey,

   I'd be interested in investigating the feasibility of supporting pandas 3+ in Beam's DataFrames API.

   I plan to start by identifying the current pandas version constraints, reviewing the DataFrames API for deprecated or incompatible pandas usage, and running the relevant test suites against pandas 3.x to identify concrete compatibility failures. Based on those findings, I can assess the scope of the required changes and outline a backward-compatible implementation approach.

   Would this investigation be helpful? I'm happy to share the findings before proceeding with any broader changes.
5. dhanusharer commented on Oct 24, 2026

   

   dhanusharer

   on Oct 24, 2026

   More actions

   .take-issue
6. github-actions

   assigned

   dhanusharer

   on Oct 24, 2026
7. dhanusharer commented on Oct 29, 2026

   

   dhanusharer

   on Oct 29, 2026

   More actions

   Pandas 3.0+ Scoping Analysis & Decision Matrix

   Following up on the upgrade scoping for Pandas 3.0+, here is the inventory of breaking changes, Beam touchpoints, behavioral decisions, and rollout strategy:

   1. Inventory of Pandas 3.0/3.1 Changes & Beam Touchpoints

   • Python Versioning: Pandas 3.0 drops Python 3.10 (requires 3.11+). When the cap is raised, PEP 508 environment markers (python_version < "3.11" vs >= "3.11") will partition the dependency in setup.py.
   • String Dtype (future.infer_string=True): Default string columns infer as pd.StringDtype (PyArrow or python-backed) with <NA> instead of object with np.nan.
     • Touchpoints: apache_beam/typehints/pandas_type_compatibility.py and apache_beam/dataframe/schemas.py.
     • Decision: Use pd.api.types.is_string_dtype(dtype) to map all string dtypes to Optional[str], while keeping dtype.kind == 'O' mapped to Any.
   • Copy-on-Write (CoW): Permanently enforced; copy=True deprecated or ignored.
     • Touchpoints: apache_beam/dataframe/frame_base.py (@populate_defaults).
     • Decision: Set copy=None under Pandas 3.0+ to prevent deprecation warnings. Beam DataFrames already operate with immutable stage boundaries.
   • Removed Methods (applymap, first, last, GroupBy.fillna):
     • Touchpoints: DeferredDataFrame, DeferredSeries, _DeferredGroupByCols.
     • Decision: Guard definitions with hasattr(base_type, '...') to prevent import-time reflection crashes. In Pandas 3.0, users migrate to .map() and .ffill() / .bfill().
   • Removed Arguments (method in fillna, axis in groupby):
     • Touchpoints: frames.py decorators (@args_to_kwargs, @populate_defaults).
     • Decision: Add removed_args=['method'] and avoid passing method=None to df.fillna(). Raise clean NotImplementedError for groupby(axis=1) under Pandas 3.0+.
   • Default Containers & Dependencies:
     • Touchpoints: Automated weekly dependency bumps update release containers (apache/beam_python3.11_sdk, etc.). Keeping the cap below 3.0 in preparation PRs prevents prematurely shifting worker container runtimes.

   2. Rollout Plan

   1. Phase 1 (PR Support pandas 3+ in the dataframes API (#40709) #40768): Land all backward-compatible reflection hardening, hasattr() guards, and dtype conversions while keeping the cap below 3.0 (pandas>=2.2.2,<2.4). Validate with a new py311-pandas-23future tox environment running with PANDAS_MODE__COPY_ON_WRITE=1 and PANDAS_FUTURE__INFER_STRING=1.
   2. Phase 2: Run experimental CI suites against Pandas 3.0 pre-releases to verify edge-case doctest formatting and signature parity.
   3. Phase 3: Raise the cap (pandas<3.2) for Python 3.11+ in a dedicated PR with an entry in CHANGES.md.
8. jrmccluskey commented on Nov 3, 2026

   

   jrmccluskey

   on Nov 3, 2026

   ContributorAuthor

   More actions

   Partitioning the dependency based on Python 3.10 is a no-op since Beam Python has already dropped 3.10 support. You have some options based on the approach here, but since this blocks 3.15 support in the Python SDK you're better off not partitioning at all. Post-3.15 support we can make the decision to allow 3.11-3.14 to use pandas 3+ or not.

   Separately, phase 2 is not particularly well-planned. Why run against 3.0 pre-releases instead of the full 3.0+ releases we're looking to support? Doing that in CI is also something you'll need a Beam committer to help with, since they'll need to open a branch on the main repo to configure those changes and run modified workflows.

   I would also be interested in if there are any new features in pandas 3+ that would be worth surfacing via the dataframes API. This has been largely focused on making everything that already exists within the SDK work, but we should consider benefits to pandas 3+ as well.