Skip to content

Port BigQueryIOWriteTest off DoFnTester - #40871

Open
Rishavbhattarai wants to merge 1 commit into
apache:masterfrom
Rishavbhattarai:port-bigqueryiowritetest-off-dofntester
Open

Rishavbhattarai wants to merge 1 commit into
apache:masterfrom
Rishavbhattarai:port-bigqueryiowritetest-off-dofntester

Conversation

@Rishavbhattarai

Copy link
Copy Markdown

Fixes #18689.

Ports BigQueryIOWriteTest off the deprecated DoFnTester. The only use was in testWritePartition.

Unlike the Datastore and SDF tests, nothing here blocks TestPipeline: WritePartition is serializable and holds no mocks. So the test now runs it through the class's existing p:

  • the file list goes in as one Iterable element, the way BatchLoads hands it over, so there's one bundle
  • the temp file prefix is a real side input instead of setSideInput
  • both outputs get KvCoder.of(ShardedKeyCoder.of(TableDestinationCoder.of()), WritePartition.ResultCoder.INSTANCE), matching BatchLoads
  • the existing checks moved into PAssert.satisfies on the expected tag, unchanged except that the empty-table check uses File.length() (no checked exception inside the lambda)

enableAbandonedNodeEnforcement(false) is gone since the pipeline now runs. The two Creates got names to keep transform names stable. The empty singleton file is still deleted after the check, because tearDown asserts the temp dir has no files.

Testing: ./gradlew :sdks:java:io:google-cloud-platform:test --tests 'org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest*' passes (220 run, 98 skipped, 0 failures), and spotlessCheck is clean. To make sure the PAssert actually runs, I changed the expected partition count in testWritePartitionLargeFileSize from 4 to 5 and both parameterizations failed.

testWritePartition now runs WritePartition through the test's
TestPipeline, with the temp file prefix as a real side input and the
same output coders BatchLoads sets. The existing checks move into a
PAssert.satisfies on the expected output tag.

Fixes apache#18689
@github-actions github-actions Bot added java io gcp size/m M: 50-249 lines changed labels Oct 28, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly migrates the testWritePartition helper in BigQueryIOWriteTest away from the deprecated DoFnTester to a standard Beam pipeline execution with ParDo and PAssert. The change removes the need to disable abandoned node enforcement and eliminates the remaining DoFnTester usage in the file.

🔍 General Feedback

  • The pipeline construction, side input wiring, multi-output tagging, and PAssert verification accurately preserve all original test assertions.
  • Consider also asserting that the unselected output tag (singlePartitionTag or multiPartitionsTag) is empty via PAssert.that(...).empty() for slightly stronger coverage, though the current migration is already equivalent to the previous DoFnTester behavior.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.
Backup reviewer: damccorm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port BigQueryIOWriteTest off DoFnTester

1 participant