Skip to content

[Python] Finish async bundles after processing outputs - #40000

Open
bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/python-async-bundle-lifecycle
Open

bvolpato wants to merge 3 commits into
apache:masterfrom
bvolpato:bvolpato/python-async-bundle-lifecycle

Conversation

@bvolpato

@bvolpato bvolpato commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

On the thread-pool path, AsyncWrapper calls finish_bundle() before consuming the wrapped process() result. A buffered transform can flush an empty bundle while the runner sees the input as complete.

This change consumes process outputs before finalization and flattens iterable returns to match the DoFn contract. If iteration fails, finalization is skipped and the input remains available for retry. Strings and bytes keep their existing scalar behavior. Returned lists and tuples now emit their elements individually, so code relying on the previous extra nesting will see a different output shape.

Local regression validation passed. A DirectRunner pipeline with a SQLite sink loses its buffered write with the unfixed implementation and preserves it with this change; the asyncio path passes as a control.

Run the Python unit suite with:

./gradlew :sdks:python:test-suites:tox:py312:testPython312

  • If this contribution is large, please file an Apache Individual Contributor License Agreement.
  • Replace these instructions with a description of your change and why you have made it.
  • Mention the appropriate issue in your description (for example: addresses #123 or fixes #456).
  • Update CHANGES.md with noteworthy changes.

See CONTRIBUTING.md for contribution guidelines.

@github-actions github-actions Bot added the python label Sep 3, 2026
@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Run Python_ML PreCommit 3.12

@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Run Python_ML PreCommit 3.13

@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Run Python_ML PreCommit 3.12

1 similar comment
@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Run Python_ML PreCommit 3.12

@bvolpato
bvolpato force-pushed the bvolpato/python-async-bundle-lifecycle branch from 1c69a4d to f4b06ef Compare October 10, 2026 22:33
@github-actions github-actions Bot added the core label Oct 10, 2026
@codecov

codecov Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 56.25%. Comparing base (76f6e6d) to head (388aacb).
⚠️ Report is 143 commits behind head on master.

Files with missing lines Patch % Lines
sdks/python/apache_beam/transforms/async_dofn.py 88.88% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40000      +/-   ##
============================================
+ Coverage     56.16%   56.25%   +0.08%     
  Complexity     2288     2288              
============================================
  Files          1124     1124              
  Lines        178336   178378      +42     
  Branches       1489     1489              
============================================
+ Hits         100170   100343     +173     
+ Misses        75645    75514     -131     
  Partials       2521     2521              
Flag Coverage Δ
python 79.72% <88.88%> (+0.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bvolpato
bvolpato marked this pull request as ready for review October 11, 2026 01:26
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @tvalentyn for label python.

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Move the entry out of the commented changelog template.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant