add WRITE_TRUNCATE_DATA write_disposition to the bigquery python sdk

[GaryPWhite]

I've noticed when using the python SDK for Dataflow pipelines, we can't use the WRITE_TRUNCATE_DATA write disposition. I believe this change is as simple as adding the disposition for use, adding it to validate_write and the streaming insert prevention list.

I've also added the change to CHANGES.md. AFAIK there's no issue open for this right now.

LMK if there's anything missing.

[github-actions]

📋 Review Summary

This PR adds BigQueryDisposition.WRITE_TRUNCATE_DATA to the Python BigQuery I/O connector. However, BigQueryBatchFileLoads in bigquery_file_loads.py was not updated to handle WRITE_TRUNCATE_DATA when copying from temporary tables or updating destination schemas, which can lead to data loss on multi-partition loads and broken schema_update_options behavior.

🔍 General Feedback

• Critical (bigquery_file_loads.py): In BigQueryBatchFileLoads._load_data (sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:970), if self.write_disposition in ('WRITE_EMPTY', 'WRITE_TRUNCATE'): groups temporary table load job IDs by destination table so TriggerCopyJobs truncates the destination table only once on the first copy job and uses WRITE_APPEND for the rest. Because 'WRITE_TRUNCATE_DATA' is missing from this check, copy jobs from multiple temp tables will run in parallel as single-element batches across workers—each issuing WRITE_TRUNCATE_DATA and overwriting data copied by previous jobs (see #24535).
• Schema updates (bigquery_file_loads.py): In UpdateDestinationSchema.process (sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:368), self._write_disposition not in ('WRITE_TRUNCATE', 'WRITE_APPEND') will cause UpdateDestinationSchema to return early for WRITE_TRUNCATE_DATA, skipping schemaUpdateOptions on the destination table prior to copying temp tables.
• Storage Write API (bigquery.py): WriteToBigQuery with method=STORAGE_WRITE_API delegates to the Java cross-language expansion service, where BigQueryIO.Write.WriteDisposition only defines WRITE_TRUNCATE, WRITE_APPEND, and WRITE_EMPTY. Passing WRITE_TRUNCATE_DATA with STORAGE_WRITE_API will fail with a Java IllegalArgumentException during expansion; consider validating this upfront in Python.
• Testing: Please add unit tests covering WRITE_TRUNCATE_DATA (e.g., adding param(write_disposition=BigQueryDisposition.WRITE_TRUNCATE_DATA) to TestBigQueryFileLoads.test_multiple_partition_files_write_dispositions in bigquery_file_loads_test.py and testing BigQueryDisposition.validate_write / BigQueryWriteFn in bigquery_test.py).

---

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

[github-actions]

🔴 Critical: Allowing WRITE_TRUNCATE_DATA here enables it for BigQueryBatchFileLoads, but sdks/python/apache_beam/io/gcp/bigquery_file_loads.py only checks for ('WRITE_EMPTY', 'WRITE_TRUNCATE') in BigQueryBatchFileLoads._load_data (line 970) and ('WRITE_TRUNCATE', 'WRITE_APPEND') in UpdateDestinationSchema.process (line 368).

Without including 'WRITE_TRUNCATE_DATA' in BigQueryBatchFileLoads._load_data, loads that spill into multiple temporary tables will not be grouped by destination before TriggerCopyJobs. As a result, parallel copy jobs will each execute with WRITE_TRUNCATE_DATA and overwrite one another's data. Please update both checks in bigquery_file_loads.py.

[github-actions]

🟡 Medium: bigquery_v2_messages.py is an autogenerated Apitools client file (# NOTE: This file is autogenerated and should not be edited by hand.), and writeDisposition is an unvalidated _messages.StringField(8), so editing this docstring has no runtime effect. In addition, lines 5445–5450 introduce trailing whitespace that will fail lint checks (table.* WRITE_APPEND: ). Consider reverting manual edits to this file or removing the trailing whitespace.

[github-actions]

🟢 Low: This line is 98 characters long (and line 2133 is 105 characters), exceeding the 80-character line length limit enforced by lint checks.

[github-actions]

Assigning reviewers:

R: @damccorm for label python.
Backup reviewer: tvalentyn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[GaryPWhite]

still coming back to this with some of gem/copilots comments folks, putting it down for the night :)

[damccorm]

Sounds good, converting to draft for now - please comment when it is ready

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.92%. Comparing base (76f6e6d) to head (9335f60).
⚠️ Report is 200 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #40622       +/-   ##
=============================================
+ Coverage     56.16%   79.92%   +23.75%     
=============================================
  Files          1124      488      -636     
  Lines        178336    82539    -95797     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65972    -34198     
+ Misses        75645    16567    -59078     
+ Partials       2521        0     -2521     

Flag	Coverage Δ
go	?
java	?
python	79.92% <100.00%> (+0.37%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.