Repository navigation
[#40250] Remove support for Python 3.10 - #40413
Conversation
|
Assigning reviewers: R: @shunping for label python. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
R: @jrmccluskey Could you please take a look? |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
Ack, just to set expectations this will be reviewed as early as tomorrow |
|
I'll step back from this review since @jrmccluskey will be handling it. Thanks! |
8ac3237 to
a2f9cb0
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #40413 +/- ##
============================================
+ Coverage 59.04% 59.78% +0.74%
+ Complexity 15624 13922 -1702
============================================
Files 2797 2586 -211
Lines 280750 260764 -19986
Branches 12488 11243 -1245
============================================
- Hits 165764 155898 -9866
+ Misses 108540 98855 -9685
+ Partials 6446 6011 -435
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jrmccluskey
left a comment
jrmccluskey
left a comment
There was a problem hiding this comment.
Should also make the changes outlined in #36774
I'm a little nervous about the workflow changes here being untested in the repo (actions configured strictly via the workflow yaml files do not run the new version of the workflow for security reasons.) My main concern is the ML test dependency set, but the others also have the inherent issue of not necessarily being known to be green until merging unless a PR is opened within the Beam repo directly. Verification of local test runs via gradle commands for workflows not requiring secrets would suffice.
| @@ -26,191 +26,4 @@ applyPythonNature() | |||
| // Required to setup a Python 3 virtualenv and task names. | |||
| pythonVersion = '3.10' | |||
|
|
|||
| def posargs = project.findProperty("posargs") ?: "" | |||
|
|
|||
| apply from: "../common.gradle" | |||
|
|
|||
| toxTask "testPy310CloudCoverage", "py310-cloudcoverage", "${posargs}" | |||
| test.dependsOn "testPy310CloudCoverage" | |||
| project.tasks.register("preCommitPyCoverage") { | |||
| dependsOn = ["testPy310CloudCoverage"] | |||
| } | |||
|
|
|||
| // Dep Postcommit runs test suites that evaluate compatibility of particular | |||
| // dependencies. Each suite is exercised on at most one python version. | |||
| // | |||
| // Should still leave at least one version in PreCommit unless the marked tests | |||
| // are also exercised by existing PreCommit | |||
| // e.g. pyarrow and pandas also run on PreCommit Dataframe and Coverage | |||
| project.tasks.register("postCommitPyDep") {} | |||
|
|
|||
| // Create a test task for supported major versions of pyarrow. | |||
| // Keep in sync with [testenv:py{310,311}-pyarrow-...] in tox.ini | |||
| // (versions released in roughly the last 1-2 years). | |||
|
|
|||
| toxTask "testPy310pyarrow-17", "py310-pyarrow-17", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-17" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-17" | |||
|
|
|||
| toxTask "testPy310pyarrow-18", "py310-pyarrow-18", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-18" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-18" | |||
|
|
|||
| toxTask "testPy310pyarrow-19", "py310-pyarrow-19", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-19" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-19" | |||
|
|
|||
| toxTask "testPy310pyarrow-20", "py310-pyarrow-20", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-20" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-20" | |||
|
|
|||
| toxTask "testPy310pyarrow-21", "py310-pyarrow-21", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-21" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-21" | |||
|
|
|||
| toxTask "testPy310pyarrow-22", "py310-pyarrow-22", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-22" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-22" | |||
|
|
|||
| toxTask "testPy310pyarrow-23", "py310-pyarrow-23", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-23" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-23" | |||
|
|
|||
| toxTask "testPy310pyarrow-24", "py310-pyarrow-24", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-24" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-24" | |||
|
|
|||
| toxTask "testPy310pyarrow-25", "py310-pyarrow-25", "${posargs}" | |||
| test.dependsOn "testPy310pyarrow-25" | |||
| postCommitPyDep.dependsOn "testPy310pyarrow-25" | |||
|
|
|||
| // Create a test task for each supported minor version of pandas | |||
| toxTask "testPy310pandas-14", "py310-pandas-14", "${posargs}" | |||
| test.dependsOn "testPy310pandas-14" | |||
| postCommitPyDep.dependsOn "testPy310pandas-14" | |||
|
|
|||
| toxTask "testPy310pandas-15", "py310-pandas-15", "${posargs}" | |||
| test.dependsOn "testPy310pandas-15" | |||
| postCommitPyDep.dependsOn "testPy310pandas-15" | |||
|
|
|||
| toxTask "testPy310pandas-20", "py310-pandas-20", "${posargs}" | |||
| test.dependsOn "testPy310pandas-20" | |||
| postCommitPyDep.dependsOn "testPy310pandas-20" | |||
|
|
|||
| // TODO(https://github.com/apache/beam/issues/31192): Add below suites | |||
| // after dependency compat tests suite switches to Python 3.9 or we add | |||
| // Python 2.2 support. | |||
|
|
|||
| // toxTask "testPy310pandas-21", "py310-pandas-21", "${posargs}" | |||
| // test.dependsOn "testPy310pandas-21" | |||
| // postCommitPyDep.dependsOn "testPy310pandas-21" | |||
|
|
|||
| // toxTask "testPy310pandas-22", "py310-pandas-22", "${posargs}" | |||
| // test.dependsOn "testPy310pandas-22" | |||
| // postCommitPyDep.dependsOn "testPy310pandas-22" | |||
|
|
|||
| // TODO(https://github.com/apache/beam/issues/30908): Revise what are we testing | |||
|
|
|||
| // Create a test task for each minor version of pytorch | |||
|
|
|||
| toxTask "testPy310pytorch-111", "py310-pytorch-111", "${posargs}" | |||
| test.dependsOn "testPy310pytorch-111" | |||
| postCommitPyDep.dependsOn "testPy310pytorch-111" | |||
|
|
|||
| toxTask "testPy310pytorch-112", "py310-pytorch-112", "${posargs}" | |||
| test.dependsOn "testPy310pytorch-112" | |||
| postCommitPyDep.dependsOn "testPy310pytorch-112" | |||
|
|
|||
| toxTask "testPy310pytorch-113", "py310-pytorch-113", "${posargs}" | |||
| test.dependsOn "testPy310pytorch-113" | |||
| postCommitPyDep.dependsOn "testPy310pytorch-113" | |||
|
|
|||
| // run on precommit | |||
| toxTask "testPy310pytorch-200", "py310-pytorch-200", "${posargs}" | |||
| test.dependsOn "testPy310pytorch-200" | |||
| postCommitPyDep.dependsOn "testPy310pytorch-200" | |||
|
|
|||
| toxTask "testPy310tft-113", "py310-tft-113", "${posargs}" | |||
| test.dependsOn "testPy310tft-113" | |||
| postCommitPyDep.dependsOn "testPy310tft-113" | |||
|
|
|||
| // TODO(https://github.com/apache/beam/issues/25796) - uncomment onnx tox task once onnx supports protobuf 4.x.x | |||
| // Create a test task for each minor version of onnx | |||
| // toxTask "testPy310onnx-113", "py310-onnx-113", "${posargs}" | |||
| // test.dependsOn "testPy310onnx-113" | |||
| // postCommitPyDep.dependsOn "testPy310onnx-113" | |||
|
|
|||
| // Create a test task for each minor version of tensorflow | |||
| toxTask "testPy310tensorflow-212", "py310-tensorflow-212", "${posargs}" | |||
| test.dependsOn "testPy310tensorflow-212" | |||
| postCommitPyDep.dependsOn "testPy310tensorflow-212" | |||
|
|
|||
| // Create a test task for each minor version of transformers | |||
| toxTask "testPy310transformers-428", "py310-transformers-428", "${posargs}" | |||
| test.dependsOn "testPy310transformers-428" | |||
| postCommitPyDep.dependsOn "testPy310transformers-428" | |||
|
|
|||
| toxTask "testPy310transformers-447", "py310-transformers-447", "${posargs}" | |||
| test.dependsOn "testPy310transformers-447" | |||
| postCommitPyDep.dependsOn "testPy310transformers-447" | |||
|
|
|||
| toxTask "testPy310transformers-latest", "py310-transformers-latest", "${posargs}" | |||
| test.dependsOn "testPy310transformers-latest" | |||
| postCommitPyDep.dependsOn "testPy310transformers-latest" | |||
|
|
|||
| toxTask "testPy310embeddingsMLTransform", "py310-embeddings", "${posargs}" | |||
| test.dependsOn "testPy310embeddingsMLTransform" | |||
| postCommitPyDep.dependsOn "testPy310embeddingsMLTransform" | |||
|
|
|||
| // Part of MLTransform embeddings test suite but requires tensorflow hub, which we need to test on | |||
| // mutliple versions so keeping this suite separate. | |||
| toxTask "testPy310TensorflowHubEmbeddings-014", "py310-TFHubEmbeddings-014", "${posargs}" | |||
| test.dependsOn "testPy310TensorflowHubEmbeddings-014" | |||
| postCommitPyDep.dependsOn "testPy310TensorflowHubEmbeddings-014" | |||
|
|
|||
| toxTask "testPy310TensorflowHubEmbeddings-015", "py310-TFHubEmbeddings-015", "${posargs}" | |||
| test.dependsOn "testPy310TensorflowHubEmbeddings-015" | |||
| postCommitPyDep.dependsOn "testPy310TensorflowHubEmbeddings-015" | |||
|
|
|||
| toxTask "whitespacelint", "whitespacelint", "${posargs}" | |||
|
|
|||
| task archiveFilesToLint(type: Zip) { | |||
| archiveFileName = "files-to-whitespacelint.zip" | |||
| destinationDirectory = file("$buildDir/dist") | |||
|
|
|||
| from ("$rootProject.projectDir") { | |||
| include "**/*.md" | |||
| include "**/build.gradle" | |||
| include '**/build.gradle.kts' | |||
| exclude '**/build/**' // intermediate build directory | |||
| exclude 'website/www/site/themes/docsy/**' // fork to google/docsy | |||
| exclude "**/node_modules/*" | |||
| exclude "**/.gogradle/*" | |||
| } | |||
| } | |||
|
|
|||
| task unpackFilesToLint(type: Copy) { | |||
| from zipTree("$buildDir/dist/files-to-whitespacelint.zip") | |||
| into "$buildDir/files-to-whitespacelint" | |||
| } | |||
|
|
|||
| whitespacelint.dependsOn archiveFilesToLint, unpackFilesToLint | |||
| unpackFilesToLint.dependsOn archiveFilesToLint | |||
| archiveFilesToLint.dependsOn cleanPython | |||
|
|
|||
| toxTask "jest", "jest", "${posargs}" | |||
|
|
|||
| toxTask "eslint", "eslint", "${posargs}" | |||
|
|
|||
| task copyTsSource(type: Copy) { | |||
| from ("$rootProject.projectDir") { | |||
| include "sdks/python/apache_beam/runners/interactive/extensions/**/*" | |||
| exclude "sdks/python/apache_beam/runners/interactive/extensions/**/lib/*" | |||
| exclude "sdks/python/apache_beam/runners/interactive/extensions/**/node_modules/*" | |||
| } | |||
| into "$buildDir/ts" | |||
| } | |||
|
|
|||
| jest.dependsOn copyTsSource | |||
| eslint.dependsOn copyTsSource | |||
| copyTsSource.dependsOn cleanPython | |||
| apply from: "../common.gradle" No newline at end of file | |||
There was a problem hiding this comment.
This can just be deleted outright, but in a follow-up PR. Leave the configuration here for now so PR checks using the old workflows can still execute properly here
| 'p311_ml_test': [ | ||
| 'datatable', |
There was a problem hiding this comment.
This isn't particularly useful without an additional run-through evaluating what ML dependencies we can install on 3.11 and test
| ## Breaking Changes | ||
|
|
||
| * X behavior was changed ([#X](https://github.com/apache/beam/issues/X)). | ||
| * Support for Python 3.10 was removed following its end-of-life ([#40250](https://github.com/apache/beam/issues/40250)). |
There was a problem hiding this comment.
Slide this down to deprecations
a2f9cb0 to
f449812
Compare
|
Hi @jrmccluskey, Thank you for the thorough review and guidance! I have addressed all requested changes in commit
Please let me know if anything else needs adjusting! |
There was a problem hiding this comment.
We're also missing the updates in https://github.com/apache/beam/tree/master/.github/actions (see the 3.9 PR for the specific changes.) BeamModulePluginGroovy also needs updates that mirror the old PR, I think that's the root cause of the setupVirtualenv failures on some configured workflows.
f449812 to
30e4d33
Compare
|
Hi @jrmccluskey, Thank you for pinpointing the exact root cause! I have updated both
All updates have been bundled into the signed commit |
|
I see that the Python 3.11 ML test suite is actually looking for a p311_ml_test extras configuration, duplicate the p310 definition for now and I will just take an action item forward to audit what can be moved. Ideally the ml_test extras definition would be more universal, but that is a problem to deal with later. |
30e4d33 to
2e174bf
Compare
|
Hi @jrmccluskey, Done! Duplicated the |
|
Thank you @jrmccluskey for the review, guidance, and for merging! Really enjoyed working through this with you. Looking forward to contributing more to Beam! 🚀 |
4 participants
Fixes #40250
Motivation & Context
Python 3.10 officially reached End-of-Life (EOL) on October 1, 2026.
Following Apache Beam's established policy of dropping EOL Python runtimes,
this PR removes all support for Python 3.10 and shifts the minimum supported
Python version to 3.11.
This mirrors the pattern established by PR #36665 (Python 3.9 removal).
Changes Made
SDK Core:
sdks/python/setup.py: Updatedpython_requires='>=3.11', removed3.10PyPI classifier, renamed
p310_ml_testtop311_ml_testsdks/python/apache_beam/__init__.py: Updated runtime version warningthreshold to
<= 10(now warns for Python 3.10 and below)sdks/python/pyproject.toml: Updatedtarget-version = "py311"for Ruff/BlackBuild & Gradle Infrastructure:
gradle.properties: Updatedpython_versions=3.11,3.12,3.13,3.14sdks/python/build.gradle: Removed 3.10-specificcibuildwheelversionconditional (unified to
3.4.0)sdks/python/tox.ini: Removedpy310fromenvlistand all test targetsbuild.gradle.kts(root): Removedpy310PreCommit/PostCommit task blocks;migrated integration tests and coverage to
py311sdks/python/test-suites/tox/py311/build.gradle: Migrated all dependencycompatibility tasks from the
py310suiteDataflow Runner:
apiclient.py: Updated_PYTHON_VERSIONS_SUPPORTED_BY_DATAFLOWto['3.11', '3.12', '3.13', '3.14']apiclient_test.py: Updated tests to verify 3.10 is now rejected and 3.11is the new minimum accepted version on Dataflow
GitHub Actions CI/CD (~60+ workflow files):
3.10matrix entries with3.11pythonVersionto3.11-PpythonVersion=3.11Documentation:
CHANGES.md: Added breaking change entry under[2.78.0] - Unreleasedwebsite/.../environments.md: Updated container image tag references to 3.11website/.../python-sdk-image-build.md: Updated SDK container tag listingsVerification
git grepnothing to commitChecklist
fixes #40250).CHANGES.mdwith noteworthy changes.Individual Contributor License Agreement.