Skip to content

[Python] Fix BigtableIO write test mocks for google-cloud-bigtable batcher delegation - #40365

Closed
mutianf wants to merge 1 commit into
apache:masterfrom
mutianf:fix-bigtableio-test-mocks
Closed

mutianf wants to merge 1 commit into
apache:masterfrom
mutianf:fix-bigtableio-test-mocks

Conversation

@mutianf

@mutianf mutianf commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Context

In newer versions of google-cloud-bigtable (>= 2.48.0 / PR #18200), MutationsBatcher delegates directly to the underlying data client batcher (table._table_impl.mutations_batcher) instead of invoking Table.mutate_rows. As a result, tests in bigtableio_test.py that patched Table.mutate_rows bypassed the mocked behaviors.

Proposed Changes

Update sdks/python/apache_beam/io/gcp/bigtableio_test.py:

  • test_write_metrics:
    • Verify that MutationsBatcher is initialized with write_fn.write_mutate_metrics as the batch completion callback.
    • Wire mock completion callback to simulate batch completion and verify metrics are properly recorded.
  • test_write_batch_error_surfaces_from_async_flush:
    • Simulate background batch error surfacing on batcher close without relying solely on Table.mutate_rows.
  • test_write_batch_error_surfaces_from_buffered_rows:
    • Intercept underlying batcher close/flush and verify buffered rows trigger flush and propagate errors.
  • Ensure production code bigtableio.py remains untouched.
  • Maintain backward compatibility with older google-cloud-bigtable versions.

Verification

  • Validated test behaviors against both legacy Table.mutate_rows and new data client batcher delegation paths.
  • Verified all assertions in test_write_metrics, test_write_batch_error_surfaces_from_async_flush, and test_write_batch_error_surfaces_from_buffered_rows.

@mutianf
mutianf force-pushed the fix-bigtableio-test-mocks branch 3 times, most recently from a8e0b06 to cb91f14 Compare October 1, 2026 13:27
@mutianf
mutianf marked this pull request as ready for review October 1, 2026 13:34
@mutianf
mutianf force-pushed the fix-bigtableio-test-mocks branch from cb91f14 to f17c368 Compare October 1, 2026 13:48
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @shunping for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@shunping

shunping commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

waiting on author

@shunping shunping left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change in this PR has already been covered and merged by another PR #40029.

Please rebase your PR to the latest master.

Also, the newly-added tests failed in the precommit workflow. Please take a look.

…tcher delegation

In google-cloud-bigtable >= 2.48.0 (PR apache#18200), MutationsBatcher delegates
directly to the underlying data client batcher via
table._table_impl.mutations_batcher instead of calling Table.mutate_rows.
When using a mock Bigtable client in tests, table._table_impl was an unconfigured
mock returning a default batcher whose close() was a no-op, causing
Table.mutate_rows to be bypassed and leaving exceptions and batch completion
callbacks un-triggered.

In TestWriteBigTable.setUp, configure the mock batcher on
table._table_impl.mutations_batcher to invoke Table.mutate_rows and
forward the results to the registered batch completion callback. This ensures
tests patching Table.mutate_rows (test_write_metrics,
test_write_batch_error_surfaces_from_async_flush, and
test_write_batch_error_surfaces_from_buffered_rows) work seamlessly across
both legacy and modern client versions while keeping production code untouched.
@codecov

codecov Bot commented Oct 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.01%. Comparing base (69bdf5f) to head (03e0dc5).
⚠️ Report is 41 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40365      +/-   ##
============================================
- Coverage     59.04%   59.01%   -0.03%     
+ Complexity    15624    15616       -8     
============================================
  Files          2797     2798       +1     
  Lines        280750   280841      +91     
  Branches      12488    12477      -11     
============================================
- Hits         165764   165744      -20     
- Misses       108540   108656     +116     
+ Partials       6446     6441       -5     
Flag Coverage Δ
python 79.55% <ø> (-0.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mutianf mutianf closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants