DO NOT MERGE: test CI for aerospike_bigtable_bigration_tools#7
Closed
prawilny wants to merge 57 commits into
Closed
DO NOT MERGE: test CI for aerospike_bigtable_bigration_tools#7prawilny wants to merge 57 commits into
prawilny wants to merge 57 commits into
Conversation
Co-authored-by: Kajetan Boroszko <kajetan@unoperate.com>
…ble-migration-tools
…sink .jar it requires
| The project consists of the following Maven submodules: | ||
|
|
||
| ### Adapter | ||
| Defines a transformation of Aerospike entities as Bigtable entities and provides utilities for doing the conversion. |
There was a problem hiding this comment.
Defines the transformation of Aerospike entities into Bigtable entities and provides utilities for performing it.
Collaborator
Author
There was a problem hiding this comment.
Done (s/bigtable/cloud bigtable/)
|
|
||
| The important classes are: | ||
| - [`RowBuilder`](adapter/src/main/java/com/google/cloud/aerospike/RowBuilder.java): The definition of the transformation of Aerospike entities into Bigtable ones. | ||
| - [`BigtableMutationBuilder`](adapter/src/main/java/com/google/cloud/aerospike/BigtableMutationBuilder.java): A builder allowing the user to create Bigtable mutations transforming input Aerospike records as Bigtable rows, |
There was a problem hiding this comment.
Come to think of it, the whole sentence needs a rewrite:
A builder for creating Bigtable mutations from Aerospike records.
Collaborator
Author
There was a problem hiding this comment.
I did it slightly differently, but close enough, I hope
| ### Backup Loader | ||
| Implements [`BackupReader`](backup-loader/src/main/java/com/google/cloud/aerospike/BackupReader.java), a reader of Aerospike backups. | ||
|
|
||
| Note that it is implemented by interfacing with the official [`aerospike-tools-backup`](https://github.com/aerospike/aerospike-tools-backup) library using Java Native Interface. |
There was a problem hiding this comment.
Note that it is implemented by using the official aerospike-tools-backup library via Java Native Interface.
| #### Dependencies | ||
| The Java `BackupReader` class requires a compiled native shared library to be present on the host (see [Dockerfile](Dockerfile)'s `dataflow-worker` target for an example how to satisfy this requirement). | ||
|
|
||
| The build process of that library is quite involved (see [Dockerfile](Dockerfile) and [backup-loader's pom.xml](backup-loader/pom.xml) for details) and the compilation of `backup-loader` will fail if the requirements are not met. |
There was a problem hiding this comment.
First part is not needed. Let's do:
For the details of the build process see...
There was a problem hiding this comment.
Again, come to think of it:
'backup-loader' should be built in the Docker container of the project, for details see...
So let's get rid of the next sentence as well
| The build process of that library is quite involved (see [Dockerfile](Dockerfile) and [backup-loader's pom.xml](backup-loader/pom.xml) for details) and the compilation of `backup-loader` will fail if the requirements are not met. | ||
| In practice, `backup-loader` can only be built in the Docker container of the project. | ||
|
|
||
| All the other modules are pure Java and easy to build in any environment with supported Java and Maven versions. |
There was a problem hiding this comment.
'and easy to build' -> 'and can be built'
| All the other modules are pure Java and easy to build in any environment with supported Java and Maven versions. | ||
|
|
||
| ### Backup Loader Examples | ||
| Contains runnable snippets exemplifying use of `BackupReader` and some scripts and config files useful for generating the backups needed for testing. |
There was a problem hiding this comment.
Contains runnable examples of BackupReader usage, along with scripts and config files for generating test backups.
| It also documents how to build both the project and the dependencies. | ||
|
|
||
| ## Workflow | ||
| [Justfile](./Justfile) is an index of interesting commands and shortcuts to running them. |
|
|
||
| #### Build and stage the template | ||
| In this step we build and publish (into buckets and registries configured by the arguments): | ||
| - Fat jar of the Dataflow template |
| ```bash | ||
| mvn clean package -pl adapter,replicator -DskipUnitTests | ||
| ``` | ||
| Copy the fat .jar from `replicator/target`. |
|
|
||
| Also mind the `--dns` flag described in [Authentication](#authentication) section. | ||
|
|
||
| Clone the `DataflowTemplates` repo somewhere and start the container with: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change is