Feature/gcp support

[orenshk]

Support for storing files in Google Cloud Storage.

• Added GCPRemoteStorage class that implements RemoteStorage
• Added support in lazydata CLI for adding a remote, but not for creating credentials. I feel this is job best left done to Google's client, but I'm happy to discuss further.

[rstojnic]

Hey, thanks very much for this - looks very good!

Re: setting up credentials. Agreed, I've only added it for AWS because it was easy to do, but I appreciate that there are more steps than simply providing the keys.

Will review in more details in the evening and hoping to merge soon.

[rstojnic]

The tests seem to be failing - could you have a look, thanks!

[rstojnic]

These dependencies should be added to setup.py as well, so they are installed together with the library when installing with pip.

[orenshk]

oh but isn't that why you were using lazy_module? Because then having these in setup.py will force non-gcp users to install the client as well.

I can add an extras_require to setup.py, so users can do
pip install lazydata["gcp"] to install the gcp requirements.

[rstojnic]

The primary reason for lazy_import was because boto3 has a really long load time, so that adds to startup time when using the lazydata CLI.

I've just looked at these two libraries - the two of them with all the dependencies are about 4MB to download. So, wondering how much hassle it is to have to install them additionally vs having to download them on first install? What do you think?

[orenshk]

oops, forgot to update circleci, will make another commit.

[orenshk]

Hey @rstojnic, to fix circleci tests I'll need to pass some google cloud related environment variables to the build, including a secret. Let me know how you want me to pass them to you.

made changes to PR to highlight the env variables needed

[rstojnic]

Hey @rstojnic, to fix circleci tests I'll need to pass some google cloud related environment variables to the build, including a secret. Let me know how you want me to pass them to you.

I'm not sure if I'm comfortable with taking your credentials! I think there is a way of passing them securely into circleci, but I'm worried they might get leaked out accidentally at some point.

My suggestion would be to maybe somehow mark these tests as requiring extra setup so we can exclude them from the circleci run. One idea is to use the @pytest.mark.skipif decorator to detect a CircleCI run through an environment variable, but open to other suggestions!

I'll run all tests locally on my machine with my credentials when I do releases to make sure nothing is broken for a release. I'll have to do something similar for AWS anyways.

[orenshk]

Fair enough, I created credentials specifically for this project that only have access to a single bucket, but I agree that it's a little unnerving. I'll mark them as skipif instead.