Skip to content

[Python] Keep BigQuery load jobs distinct for table aliases - #40001

Draft
bvolpato wants to merge 4 commits into
apache:masterfrom
bvolpato:bvolpato/python-bigquery-destination-job-ids
Draft

bvolpato wants to merge 4 commits into
apache:masterfrom
bvolpato:bvolpato/python-bigquery-destination-job-ids

Conversation

@bvolpato

@bvolpato bvolpato commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Python BigQuery FILE_LOADS can silently omit files when dynamic destinations use different spellings of the same table, such as project:dataset.table and dataset.table. File grouping keeps those strings separate, but load-job IDs hash the resolved table and collide. BigQuery then treats a later submission as an existing job.

This change uses the destination identity from file grouping to generate load-job IDs. Each group gets its own load job and temporary table, while retries remain deterministic. Callback arguments and copy write-disposition handling are preserved. The tox change applies Beam's existing GRPC_ENABLE_FORK_SUPPORT=0 workaround before Python imports gRPC.

The FILE_LOADS unit suite passed after merging master, using fake jobs and stubbed credential discovery. No live BigQuery service was used. Run the Python cloud unit suite with:

./gradlew :sdks:python:test-suites:tox:py312:testPy312Cloud

Job IDs change for noncanonical destination spellings. Replaying unfinished work across an in-place SDK update can resubmit completed work and duplicate rows. Canonical project:dataset.table destinations keep their existing IDs, and retries within either version remain stable.


  • If this contribution is large, please file an Apache Individual Contributor License Agreement.
  • Replace these instructions with a description of your change and why you have made it.
  • Mention the appropriate issue in your description (for example: addresses #123 or fixes #456).
  • Update CHANGES.md with noteworthy changes.

See CONTRIBUTING.md for contribution guidelines.

@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

I checked the macOS Python 3.14 failure. JavaJarServerTest.test_classpath_jar fails because the Java subprocess exits with SIGABRT; that test and its implementation are unchanged by this PR. The FILE_LOADS tests are skipped in that CI lane because it has no GCP dependencies. All 38 FILE_LOADS unit tests pass locally with those dependencies installed.

Could someone with rerun access retry the macOS job? My fork account cannot rerun upstream Actions. Thanks!

@codecov

codecov Bot commented Oct 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.75%. Comparing base (76f6e6d) to head (48a3f65).
⚠️ Report is 143 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40001       +/-   ##
=============================================
+ Coverage     56.16%   79.75%   +23.59%     
=============================================
  Files          1124      487      -637     
  Lines        178336    82280    -96056     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65626    -34544     
+ Misses        75645    16654    -58991     
+ Partials       2521        0     -2521     
Flag Coverage Δ
go ?
java ?
python 79.75% <ø> (+0.20%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant