feat(import): add support for multiple hbase snapshot imports by tianlei2 · Pull Request #4600 · googleapis/java-bigtable-hbase

tianlei2 · 2026-04-28T18:41:41Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

b/429250716

This is the first PR that incorporates changes from https://github.com/jhambleton/java-bigtable-hbase/commits/dataflow-v2-v2.15.6 and some fixes to make it pass the tests.

Fixed Test Isolation Issues
SnapshotUtilsTest.testGetHbaseConfiguration was failing because the static configuration field SnapshotUtils.hbaseConfiguration cached state between test cases, leaking stale data into subsequent tests.
- Solution: Added a @before setup method to reset the static field to null via reflection before every test run.
Fixed Timestamp Formatting Tests
SnapshotUtilsTest.testAppendCurrentTimestamp was throwing a NumberFormatException because the return value contained a UUID suffix (timestamp-UUID), but the test attempted to parse the entire string directly as a Long.
- Solution: Updated the test to split the string using the "-" character to extract and correctly parse just the timestamp prefix.
Resolved Classpath and SPI Conflicts (dnsjava)
Integration tests failed on Java 8 and 11 in Kokoro because of unshaded transitive dependency conflicts (com.google.protobuf.LiteralByteString NoClassDefFoundError).
- Solution: Reverted back to the shaded hbase-shaded-mapreduce dependency, ensuring proper compatibility across all Java versions.
Uncommented and Fixed Tests in ImportJobFromHbaseSnapshotTest
Several useful unit tests were commented out in ImportJobFromHbaseSnapshotTest because mockito-core lacked the ability to mock static methods.
- Solution:
  Switched from mockito-core to mockito-inline in the pom.xml to allow static mocking.
  Uncommented the code and restored the original formatting to prevent any lint errors, enabling JUnit to verify correct configuration parsing.
ComputeAndValidateHashFromBigtableDoFnTest.java was accidentally deleted, adding back
Cleanups on unused comments

tianlei2 · 2026-04-29T17:39:52Z

Integration test is run and passing:
https://fusion2.corp.google.com/invocations/33f68247-794b-45c2-8294-1b97ff42c5d0/artifacts/github%2Fjava-bigtable-hbase%2Fbigtable-dataflow-parent%2Fbigtable-beam-import%2Ftarget%2Ffailsafe-reports%2Fintegration-beam%2Fcom.google.cloud.bigtable.beam.hbasesnapshots.EndToEndIT;config=default

…ndToEndIT

…sjava SPI conflict

…bFromHbaseSnapshotTest

…lsTest

…tationsFn

tianlei2 · 2026-05-28T03:55:53Z

/gemini review

…m/google/cloud/bigtable/beam/hbasesnapshots/HBaseSnapshotRestoreTool.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…gtableMutationsFn

…utationsFn

…nalApi

…ing jar

…ce ImportConfigTest

…rver

mutianf

Some final nits.

mutianf · 2026-06-02T15:10:55Z

+            importConfig.getSourcepath(),
+            importConfig.getRestorepath());
+    DataflowPipelineDebugOptions debugOptions = options.as(DataflowPipelineDebugOptions.class);
+    debugOptions.setGCThrashingPercentagePerPeriod(100.00);


Sg. Can we add a comment?

mutianf · 2026-06-02T15:12:13Z

/gemini review

…Config#validate() and remove duplicate checks from callers

…nterException when sourcepath is null

tianlei2 · 2026-06-03T17:46:25Z

/gemini review

…s for offline restore tool, and add unit tests

vermas2012 · 2026-06-15T15:18:22Z

+      LOG.info(
+          String.format(
+              "Restore path %s already exists, deleting it for idempotency", restorePath));
+      fileSystem.delete(restorePath, true);


I understand why we do this. But the CL introduces many ways to run concurrent jobs, we need to make sure that in no case can the parallel shards delete the restored path.

I think we can do 2 things,

Document this contract explicitly and loudly (Run the restore step first, only after completion lauch the shards).

Make sure that this delete does not happen when skipRestore is set. The current code does not reach here when skipResotre is set but it sets the invariant that we don't want to delete any restore path when skipRestore is set.
@mutianf what do you think?

vermas2012 · 2026-06-15T15:20:44Z

+
+/** Tests the {@link CreateBigtableMutationsFn} functionality. */
+@RunWith(JUnit4.class)
+public class CreateBigtableMutationsFnTest {


No assertion on the silent fully-dropped-row case. When every cell in a row exceeds filterLargeCellThresholdBytes, the method returns an empty list and emits nothing — but only droppedCells is incremented, never droppedRows. No test pins this behavior, and it's arguably a bug.

vermas2012 · 2026-06-15T15:22:39Z

+    SnapshotConfig snapshotConfig = SnapshotTestHelper.newSnapshotConfig("my-restore-path");
+
+    TableName tableName = TableName.valueOf("my-table");
+    RegionInfo regionInfo = RegionInfoBuilder.newBuilder(tableName).build();


We are only roundtripping with empty regions with "" start and end keys. Lets test both empty keys and populated keys. We can have start and end keys that are raw bytes, right?

vermas2012 · 2026-06-15T15:26:48Z

shading is under-documented (the highest long-term cost). The revert to shaded hbase-shaded-mapreduce, the hbase-shaded-client exclusion, and especially the new shade exclude of META-INF/services/java.net.spi.InetAddressResolverProvider are all there to defeat the dnsjava SPI/LiteralByteString conflict — but none carry an inline comment saying so. The next person who bumps HBase or Beam will "clean these up" and reintroduce the NoClassDefFoundError. Add a comment at each of those three spots referencing the conflict. The provided-scope annotations are now justified in-thread and fine.

Shading workarounds need inline rationale: pom.xml — three separate comments: (a) the hbase-shaded-client , (b) the META-INF/services/java.net.spi.InetAddressResolverProvider shade , (c) the hbase-shaded-mapreduce dependency.

tianlei2 requested a review from a team as a code owner April 28, 2026 18:41

product-auto-label Bot added size: xl Pull request size is extra large. api: bigtable Issues related to the googleapis/java-bigtable-hbase API. labels Apr 28, 2026

tianlei2 force-pushed the dataflow-import branch 3 times, most recently from f5324bd to f8b5932 Compare April 28, 2026 19:22

tianlei2 changed the title ~~Dataflow import~~ feat(import): add support for multiple hbase snapshot imports Apr 28, 2026

tianlei2 force-pushed the dataflow-import branch 3 times, most recently from 13f65dc to a3a6c7f Compare April 28, 2026 20:17

tianlei2 marked this pull request as draft April 28, 2026 20:42

tianlei2 marked this pull request as ready for review April 28, 2026 22:41

tianlei2 marked this pull request as draft April 28, 2026 22:42

tianlei2 added the kokoro:run Add this label to force Kokoro to re-run the tests. label Apr 29, 2026

tianlei2 self-assigned this Apr 29, 2026

yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Apr 29, 2026

tianlei2 added 4 commits April 29, 2026 18:18

feat(import): add support for multiple hbase snapshot imports

26e20e4

fix(import): add fork to compiler plugin and optimize GCS search in E…

58cb66e

…ndToEndIT

fix(import): format non-complying files for Google Java Style compliance

66659a8

fix(import): exclude hbase-shaded-client from mapreduce to prevent dn…

a1f9b04

…sjava SPI conflict

tianlei2 force-pushed the dataflow-import branch 4 times, most recently from 12f1c11 to d511a61 Compare April 29, 2026 19:46

tianlei2 added 2 commits April 29, 2026 19:55

test(import): switch to mockito-inline and fix unit tests in ImportJo…

49c3e28

…bFromHbaseSnapshotTest

test(import): fix test isolation and timestamp parsing in SnapshotUti…

5ec8dc1

…lsTest

tianlei2 force-pushed the dataflow-import branch from d511a61 to 5ec8dc1 Compare April 29, 2026 19:57

tianlei2 requested a review from vermas2012 April 29, 2026 20:27

googleapis deleted a comment from google-cla Bot Apr 29, 2026

tianlei2 added 5 commits May 28, 2026 03:28

Clean up imports and use short names in ReadRegionsTest

2aa610d

Clean up imports and use short names in ReadSnapshotRegionFnTest

4b75b2f

Use estimatedSerializedSizeOf instead of heapSize in CreateBigtableMu…

0b71307

…tationsFn

Commit SnapshotUtils fixes and formatted CreateBigtableMutationsFnTest

11f7768

Reformat CreateBigtableMutationsFn.java with google-java-format 1.35.0

a2a5b93

This comment was marked as resolved.

Sign in to view

tianlei2 and others added 13 commits May 28, 2026 09:46

Update bigtable-dataflow-parent/bigtable-beam-import/src/main/java/co…

e06e3f4

…m/google/cloud/bigtable/beam/hbasesnapshots/HBaseSnapshotRestoreTool.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Apply Gemini bot recommendations for null checks and resource leaks

42592c3

Add Javadocs to SnapshotConfig methods and update comment in CreateBi…

a8e1db5

…gtableMutationsFn

Mark SnapshotConfig and HBaseRegionScanner as InternalApi

aa338e8

Commit SnapshotKey annotation and detailed comment in CreateBigtableM…

7fc3131

…utationsFn

Add detailed comments in CreateBigtableMutationsFn and reformat

228d196

Mark HBaseSnapshotInputConfigBuilder as InternalApi

f86d079

Simplify null checks in processElement and mark ImportConfig as Inter…

9f0fcc8

…nalApi

Apply user edits and reformat CreateBigtableMutationsFn.java

463c2c5

Reformat HBaseSnapshotInputConfigBuilder and SnapshotConfig with work…

6fb4c65

…ing jar

Add comment explaining GC thrashing disable in ImportJob

3001b61

Add validation logic for backoff settings in ImportConfig and introdu…

221163e

…ce ImportConfigTest

Add Javadoc detailing deferred limit validation to Bigtable client/se…

c5c47a4

…rver

mutianf reviewed Jun 2, 2026

View reviewed changes

mutianf approved these changes Jun 2, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

tianlei2 added 2 commits June 3, 2026 16:00

Centralize null checks for sourcepath and snapshotInfos inside Import…

8716a51

…Config#validate() and remove duplicate checks from callers

Fix order of validation and restore path setup to throw clean NullPoi…

dd8eb05

…nterException when sourcepath is null

This comment was marked as resolved.

Sign in to view

Refactor bulk mutation timeout null safety, support local file system…

fd7fee4

…s for offline restore tool, and add unit tests

vermas2012 approved these changes Jun 15, 2026

View reviewed changes

Conversation

tianlei2 commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tianlei2 commented Apr 29, 2026

Uh oh!

tianlei2 commented May 28, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

mutianf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mutianf Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

mutianf commented Jun 2, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

tianlei2 commented Jun 3, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

vermas2012 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

vermas2012 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

vermas2012 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

vermas2012 Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tianlei2 commented Apr 28, 2026 •

edited

Loading