Add UK-NRFA fetcher, #34

[kratzert]

• Implements UK NRFA fetcher as a separate fetcher class.
• Includes unit tests and test_data
• Includes metadata download
• Includes cached sites file

Todo before merge

• Currently only includes streamflow (gdf). Which other variables do we want to include, see https://nrfaapps.ceh.ac.uk/nrfa/nrfa-api.html#parameter-data-type and what are the global names we want to use for them?
• I have not yet added parsing for metadata column names. I think it makes no sense to have a global name for all of them, since there will be different info for different providers. So what we should agree on is the set of metadata column names we want to add to constants.py and then translate the country specific columns accordingly (e.g. station/location == gauge_name, river, area, latitude, longitude).

@simonmoulds any thoughts on the first point?

@simonmoulds and @thiagovmdon any thoughts on the second point?

[kratzert]

Maybe

# to constants.py
LATITUDE = "latitude"
LONGITUDE = "longitude"
AREA = "area"
LOCATION = "location"
RIVER = "river"

And the dataframe is indexed by "gauge_id" (i.e. constants.GAUGE_ID). Related to #15

[thiagovmdon]

RIVER = "river"

I agree. I think having location and river names is essential to make sure we can filter potential duplicates in the future.

[simonmoulds]

I would suggest including gauged daily flow(gdf), gauged monthly flow (gmf), catchment daily rainfall (cdr) and catchment monthly rainfall (cmr). I don't think that naturalised flow is available for many catchments and I don't really know how this is estimated, so I would suggest leaving for now. For global variable names it probably depends a bit on the style we want to adopt. It could be discharge and precipitation, or Q and P (note: I think in this instance rainfall is synonymous with precipitation). However, as we are also thinking about retrieving data at different time resolutions (i.e. #3), do we want to apply a naming convention that details the resolution and aggregation method? SEPA do something like: '15minute' [i.e. instantaneous], 'CalendarYear.Mean', 'Month.Min', for example.

[simonmoulds]

For the metadata columns, I think it makes sense to choose a subset which will be provided by the vast majority of providers. In OHDB [@thiagovmdon - this is a streamflow database I compiled with colleagues at Oxford], we had:

STATION_NAME = 'station_name'
RIVER = 'river'
COUNTRY = 'country' 
SOURCE = 'source'
SOURCE_ID = 'source_id'
SUBSOURCE = 'subsource'
SUBSOURCE_ID = 'subsource_id'
LATITUDE = 'latitude'
LONGITUDE = 'longitude' 
ALTITUDE = 'altitude'
AREA = 'area'
START_DATE = 'start_date' 
END_DATE = 'end_date'

We found that providing information about the subsource was helpful when dealing with providers like GRDC or NRFA, which pull data from other sources (e.g. EA, SEPA in the case of NRFA). Start and end dates could either be taken from the original metadata, or calculated from the timeseries directly.

[kratzert]

For discussion on attribute names, see #39

For NRFA specific data vars:

• I am in favor of verbose names (i.e. "discharge" over "Q").
• Is there a need to download gauged monthly flow or could this also just be gdf and then resampled to monthly mean? Same for rainfall and monthly rainfall.
  • If we want to keep all of them, I think we need to come up with some names that we hope we can re-use for other fetchers.
  • I think I would be against something like "discharge_daily", "discharge_hourly", "discharge_15min" or whateverr is supported. If we return the data indexed by time, the resolution is clear and we can just keep calling it "discharge" (and so on?).
  • There is also gauging-stage and gauging-flow available, however I was not quite sure what this is. For the station I tested it, the data was not available. Is this instantaneous data?
• Another option is to have one set of variables (e.g. "discharge", "precipitation" and then accept a second argument related to temporal resolution. Then we could either a) download the specified temporal resolution if this is supported by the data provider (e.g. monthly in this case) or b) download the highest resolution available and then resample.

Not entirely sure what I prefer

[simonmoulds]

I should think the gauging-stage and gauging-flow are a series of spot gaugings used to create the rating curve. I tested it on 2001 and I get around 400 values at irregular intervals, and the same number of values for gauging-stage and gauging-flow. I imagine this will only be available for rated sections, and not where a control section is used.

I agree the time resolution can be inferred from the timeseries itself. But the method of aggregation can't (e.g. mean, min, max, instantaneous). So we might want to document this information somehow. It could be done in a separate column in the timeseries dataframe, or through a naming convention.

[thiagovmdon]

I would suggest including gauged daily flow(gdf), gauged monthly flow (gmf), catchment daily rainfall (cdr) and catchment monthly rainfall (cmr). I don't think that naturalised flow is available for many catchments and I don't really know how this is estimated, so I would suggest leaving for now. For global variable names it probably depends a bit on the style we want to adopt. It could be discharge and precipitation, or Q and P (note: I think in this instance rainfall is synonymous with precipitation). However, as we are also thinking about retrieving data at different time resolutions (i.e. #3), do we want to apply a naming convention that details the resolution and aggregation method? SEPA do something like: '15minute' [i.e. instantaneous], 'CalendarYear.Mean', 'Month.Min', for example.

Perhaps also including level would be better? Also, is there temperature for those UK stations? if yes, I would also be in favor of including.

[thiagovmdon]

For discussion on attribute names, see #39

For NRFA specific data vars:

• I am in favor of verbose names (i.e. "discharge" over "Q").

• Is there a need to download gauged monthly flow or could this also just be gdf and then resampled to monthly mean? Same for rainfall and monthly rainfall.

  • If we want to keep all of them, I think we need to come up with some names that we hope we can re-use for other fetchers.
  • I think I would be against something like "discharge_daily", "discharge_hourly", "discharge_15min" or whateverr is supported. If we return the data indexed by time, the resolution is clear and we can just keep calling it "discharge" (and so on?).
  • There is also gauging-stage and gauging-flow available, however I was not quite sure what this is. For the station I tested it, the data was not available. Is this instantaneous data?

• Another option is to have one set of variables (e.g. "discharge", "precipitation" and then accept a second argument related to temporal resolution. Then we could either a) download the specified temporal resolution if this is supported by the data provider (e.g. monthly in this case) or b) download the highest resolution available and then resample.

Not entirely sure what I prefer

• Agreed. discharge over Q.
• I think we can keep only the daily (and sub-hourly when needed). Other scales can be resampled.

[kratzert]

I would suggest including gauged daily flow(gdf), gauged monthly flow (gmf), catchment daily rainfall (cdr) and catchment monthly rainfall (cmr). I don't think that naturalised flow is available for many catchments and I don't really know how this is estimated, so I would suggest leaving for now. For global variable names it probably depends a bit on the style we want to adopt. It could be discharge and precipitation, or Q and P (note: I think in this instance rainfall is synonymous with precipitation). However, as we are also thinking about retrieving data at different time resolutions (i.e. #3), do we want to apply a naming convention that details the resolution and aggregation method? SEPA do something like: '15minute' [i.e. instantaneous], 'CalendarYear.Mean', 'Month.Min', for example.

Perhaps also including level would be better? Also, is there temperature for those UK stations? if yes, I would also be in favor of including.

They don't have stage/level as time series. At least not a daily time series. For precipitation, should the name be "catchment_precipitation" to make it clear that this is precipitation over the catchment area and not at the gauge station?

[kratzert]

I agree the time resolution can be inferred from the timeseries itself. But the method of aggregation can't (e.g. mean, min, max, instantaneous). So we might want to document this information somehow. It could be done in a separate column in the timeseries dataframe, or through a naming convention.

Okay, still not sure what to do with this. I don't want to over-complicate things. The main purpose of the library is to facilitate access to data from the provider. We return time-indexed data that can be saved to csv/netCDF/zarr and can be processed with any of the standard libraries (pandas, xarray). I tend a little bit towards not even supporting custom aggregations to different resolutions, at least not at the fetcher level. Because this is not really the task of a class that is responsible for fetching data.

We can add utilities to resample the data, but with pandas and a time-indexed dataframe it is literally as simple as

df_monthly_mean = df.resample('M').mean()
df_monthly_max = df.resample('M').max()
df_monthly_min = df.resample('M').min() 

Not really sure if we need to provide a function to wrap these one-lines or if we can assume that someone who bulk-downloads streamflow data is able to resample data.

The only thing I see is that it might make sense to separate between daily values and hourly (or instantaneous). I would say most modeling tasks are usually done with daily data, but I am personally highly interested in sub-daily data.

We could give it a different variable name (not sure if we then need multiple for e.g. hourly and 15-min or whatever frequencies we encounter), or if we still return this data under the variable "discharge" and there is some other mechanism to say I want daily, or whatever sub-daily data is available.

[simonmoulds]

I would suggest including gauged daily flow(gdf), gauged monthly flow (gmf), catchment daily rainfall (cdr) and catchment monthly rainfall (cmr). I don't think that naturalised flow is available for many catchments and I don't really know how this is estimated, so I would suggest leaving for now. For global variable names it probably depends a bit on the style we want to adopt. It could be discharge and precipitation, or Q and P (note: I think in this instance rainfall is synonymous with precipitation). However, as we are also thinking about retrieving data at different time resolutions (i.e. #3), do we want to apply a naming convention that details the resolution and aggregation method? SEPA do something like: '15minute' [i.e. instantaneous], 'CalendarYear.Mean', 'Month.Min', for example.

Perhaps also including level would be better? Also, is there temperature for those UK stations? if yes, I would also be in favor of including.

They don't have stage/level as time series. At least not a daily time series. For precipitation, should the name be "catchment_precipitation" to make it clear that this is precipitation over the catchment area and not at the gauge station?

Yes, I think so

[simonmoulds]

I agree the time resolution can be inferred from the timeseries itself. But the method of aggregation can't (e.g. mean, min, max, instantaneous). So we might want to document this information somehow. It could be done in a separate column in the timeseries dataframe, or through a naming convention.

Okay, still not sure what to do with this. I don't want to over-complicate things. The main purpose of the library is to facilitate access to data from the provider. We return time-indexed data that can be saved to csv/netCDF/zarr and can be processed with any of the standard libraries (pandas, xarray). I tend a little bit towards not even supporting custom aggregations to different resolutions, at least not at the fetcher level. Because this is not really the task of a class that is responsible for fetching data.

We can add utilities to resample the data, but with pandas and a time-indexed dataframe it is literally as simple as

df_monthly_mean = df.resample('M').mean()
df_monthly_max = df.resample('M').max()
df_monthly_min = df.resample('M').min() 

Not really sure if we need to provide a function to wrap these one-lines or if we can assume that someone who bulk-downloads streamflow data is able to resample data.

The only thing I see is that it might make sense to separate between daily values and hourly (or instantaneous). I would say most modeling tasks are usually done with daily data, but I am personally highly interested in sub-daily data.

We could give it a different variable name (not sure if we then need multiple for e.g. hourly and 15-min or whatever frequencies we encounter), or if we still return this data under the variable "discharge" and there is some other mechanism to say I want daily, or whatever sub-daily data is available.

I agree with the point about not needing to provide custom aggregations - that is best left to the user and it is straightforward. I think we should be returning data that is as close to that received from the API as possible, since that is the main purpose of the library.

The only difficulty is when the API allows you to request aggregated data (e.g. SEPA allows you to request the daily mean/min/max streamflow). It is much quicker for the user to request the daily aggregations than request the 15-minute instantaneous values only to aggregate them themselves.

What about adding an argument (e.g. value_type or timeseries_type) to the get_data method that would allow the user to specify whether they wanted to retrieve the daily mean/max/min, instantaneous etc. Note this would not imply doing any aggregation within the get_data method - only allowing the user to specify what data to retrieve from the API, if supported. Then the returned dataframe would just need to have the time and discharge or stage columns.

[thiagovmdon]

I agree the time resolution can be inferred from the timeseries itself. But the method of aggregation can't (e.g. mean, min, max, instantaneous). So we might want to document this information somehow. It could be done in a separate column in the timeseries dataframe, or through a naming convention.

Okay, still not sure what to do with this. I don't want to over-complicate things. The main purpose of the library is to facilitate access to data from the provider. We return time-indexed data that can be saved to csv/netCDF/zarr and can be processed with any of the standard libraries (pandas, xarray). I tend a little bit towards not even supporting custom aggregations to different resolutions, at least not at the fetcher level. Because this is not really the task of a class that is responsible for fetching data.
We can add utilities to resample the data, but with pandas and a time-indexed dataframe it is literally as simple as

df_monthly_mean = df.resample('M').mean()
df_monthly_max = df.resample('M').max()
df_monthly_min = df.resample('M').min() 

Not really sure if we need to provide a function to wrap these one-lines or if we can assume that someone who bulk-downloads streamflow data is able to resample data.
The only thing I see is that it might make sense to separate between daily values and hourly (or instantaneous). I would say most modeling tasks are usually done with daily data, but I am personally highly interested in sub-daily data.
We could give it a different variable name (not sure if we then need multiple for e.g. hourly and 15-min or whatever frequencies we encounter), or if we still return this data under the variable "discharge" and there is some other mechanism to say I want daily, or whatever sub-daily data is available.

I agree with the point about not needing to provide custom aggregations - that is best left to the user and it is straightforward. I think we should be returning data that is as close to that received from the API as possible, since that is the main purpose of the library.

The only difficulty is when the API allows you to request aggregated data (e.g. SEPA allows you to request the daily mean/min/max streamflow). It is much quicker for the user to request the daily aggregations than request the 15-minute instantaneous values only to aggregate them themselves.

What about adding an argument (e.g. value_type or timeseries_type) to the get_data method that would allow the user to specify whether they wanted to retrieve the daily mean/max/min, instantaneous etc. Note this would not imply doing any aggregation within the get_data method - only allowing the user to specify what data to retrieve from the API, if supported. Then the returned dataframe would just need to have the time and discharge or stage columns.

I also agree with not doing resamples ourselves, and that the final df will speak by itself. I think this is a nice suggestion @simonmoulds. Just one query: is instantaneous the best "naming" when we have like 15-min, hourly etc? "subdaily" would not make more sense? (talking as a general for all countries). So, sub daily (or any other naming we agree on) would be the highest resolution available, and daily (mean, max, min). No resamplings

[kratzert]

The only difficulty is when the API allows you to request aggregated data (e.g. SEPA allows you to request the daily mean/min/max streamflow). It is much quicker for the user to request the daily aggregations than request the 15-minute instantaneous values only to aggregate them themselves.

What about adding an argument (e.g. value_type or timeseries_type) to the get_data method that would allow the user to specify whether they wanted to retrieve the daily mean/max/min, instantaneous etc. Note this would not imply doing any aggregation within the get_data method - only allowing the user to specify what data to retrieve from the API, if supported. Then the returned dataframe would just need to have the time and discharge or stage columns.

Well, I partially agree. Sure, it is faster to get daily min/max directly if that is provided rather than getting hourly and aggregating by min/max operation. That being said, it is the first time I hear some data provider provides something else than daily mean. And from my personal understanding/use of this library, I would not assume people go and download the hourly data over and over again. Instead you let this run once and then save the downloaded data, after which it doesn't really matter?

That being said, I also don't have strong opinions about including daily_min/daily_max as two additional variables (I think in that case it is easier to just have two additional variable names that in that case might only be supported by one fetcher) rather than changing the get_data implementation of all other fetchers. Sounds reasonable?

But we can discuss more on this in #32

For this PR, I will stick to daily discharge and daily catchment precipitation for now (still need to push that) and will also start using the global attribute names added in #39 for the metadata.