Hydro data updates - #62
Conversation
| args: | ||
| urlpath: "gs://pangeo-data/soilgrids/{{ variable }}_250m.tif" | ||
| chunks: {'y': 5120, 'x': 5120} | ||
| storage_options: {'anon': True} # TODO: remove, see GH#61 |
There was a problem hiding this comment.
@rabernat and @davidbrochart, wondering if you have suggestions on how to do this. This catalog loads fine in ocean.pangeo.io and fails on travis with the following error:
E rasterio._err.CPLE_AppDefinedError: Fetching OAuth2 access code from auth code failed.
The bucket is currently public so I'm not sure why I need to do any auth here. Thoughts?
There was a problem hiding this comment.
Actually this is a different issue. This one is more similar to intake/intake-xarray#31. I made a PR here: https://github.com/jhamman/pangeo-datastore/pull/1
…ngeo-datastore into hydro_data_updates
|
Hi @charlesbluca - any chance you could look into this? I'm curious how we will go about testing things like this now that http://catalog.pangeo.io/ is up. I'm pretty sure the failures here are just related to the CI system we're using in this repo but it would be good to get your thoughts on them. |
|
It looks like the failures are rooted to a bad response from |
|
The CI is broken here because we don’t have credentials in Travis to open requester pays bukets. Our options are:
I favor temporarily disabling the xarray opening, also because it’s slow. It doesn’t scale to open every dataset in the catalog every time CI is run. An even better option would be to look at the diff and only open the new datasets in each PR. That would require some git ninja skills that I don’t have. |
|
FWIW: I am in favor of disabling the dataset open here and just testing / linting the catalog. |
|
If we want to leverage a solution that allows for additional credentials, we could consider a Cloud Function that tests updated catalogs (or just all catalogs on a trigger). This is currently how I've been updating the build badges for the catalog. |
|
@jhamman could you just rebase this on master? The tests should go green. |
…ore into hydro_data_updates
4 participants
Adding two new datasets to the Pangeo catalog. Do not merge until I move these to their new bucket.