Initial research

[nikola-rados]

We need to figure out how to break down the RVIC tool into usable wps processes. Explore the code and try to come to up with a few pathways for us to follow. Once we have sorted some of our ideas out we can create new issues and begin work.

Please post any notes/thoughts/discussion in this issue thread.

[corviday]

In order to generate routed streamflow, RVIC has to be run twice.

The first run is the "parameters" run - it looks at the watershed uphill from the point you care about and calculates how long it takes rain that falls on each grid square to reach the one you care about.

The second run is the "convolution" run - it starts with the equations created in the parameters run, and applies them to rainfall data to calculate the amount of water passing through the grid cell you're asking about at each time point.

You could probably make those two runs separate processes. The parameter process usually takes about ten seconds. The convolution process usually takes 10-20 minutes (it can be stopped and restarted, but I've never done that and don't know a huge amount about it). So splitting them up doesn't divide the work evenly.

---

RVIC takes a large number of arguments in the form of a configuration file. (There's a separate one for each of the two processes). You could also have a process to generate the configuration files, or a separate process to generate each one.

RVIC theoretically can be called from a python script and passed a dictionary of values, instead of gettng all its arguments from config files, but this isn't well documented and we've never done it here at PCIC. Might be the most convenient approach for this project, though.

[eyvorchuk]

What does the convert run do?

[nikola-rados]

Thanks for that rundown @corviday!

You could probably make those two runs separate processes.

Is there ever a time where you would only want to run just one of these processes? Or is it always one then the other to produce the ouptut you want? If the latter is the case I think it would make more sense to have it a single process (at least to start).

You could also have a process to generate the configuration files, or a separate process to generate each one.

This seems like a pretty good idea. Is there code in RVIC that already does this? Or would it be from scratch?

RVIC theoretically can be called from a python script and passed a dictionary of values

You are right in saying that this is probably the best approach for the project. I think taking in a collection of kwargs then either building a config or dict to pass into RVIC would be a good approach. Although, you also said it was a large number of args so perhaps that would be messy.

[corviday]

What does the convert run do?

I think it converts data from some other format into a format that can be used by RVIC. I don't know much about it, I've never needed to use it.

Is there code in RVIC that already does this [generate config files]? Or would it be from scratch?

It would be from scratch. The config files are just a list of argument: value pairs for all the arguments to the RVIC program. Most of the arguments are the same for all PCIC runs.

Is there ever a time where you would only want to run just one of these processes? Or is it always one then the other to produce the output you want? If the latter is the case I think it would make more sense to have it a single process (at least to start).

Every convolution run to output routed streamflow data needs a parameter run.

I think if you want to generate routed streamflow for the same point, but different climate model outputs - to compare CanESM2 and ACCESS1-0 or something - they could use the same parameter run. The elevations and soil types and all those things that go into the parameter run don't change over time or between different climate models. (On the other hand, it is pretty quick to run parameter, I don't know that there's really a good case for saving its results.)

[sum1lim]

RVIC takes a large number of arguments in the form of a configuration file. (There's a separate one for each of the two processes). You could also have a process to generate the configuration files, or a separate process to generate each one.

It looks like we do not necessarily have to write a process to write a .cfg file(looks kind of redundant to me). Instead, both convolution_init and gen_uh_init can take dictionary input.

So, we can write a process that allows users to input both:

1. an already written config file(this case we only have the call the function)
2. string inputs that need to be generated into a dictionary.

[sum1lim]

I believe this issue has not been resolved in the most recent version of RVIC. This might be problematic in our future processes. Correct me if I am wrong.

[corviday]

You are correct; that issue has not been resolved, as far as I know. A fix I used is here.

[sum1lim]

Also, I got this error:

slim@devel2:~/Desktop/hydro_data$ rvic parameters sample_parameter_config.cfg
-->> Could not load xarray!! <<--
INFO:init_logger>> -------------------- INITIALIZED RVIC LOG ------------------
INFO:init_logger>> LOG LEVEL: DEBUG
INFO:init_logger>> Logging To Console: True
INFO:init_logger>> LOG FILE: ./routing/sample/logs/RVIC-20200714-171433.log
INFO:init_logger>> ----------------------------------------------------------

INFO:gen_uh_init>> plots directory is ./routing/sample/plots
INFO:gen_uh_init>> logs directory is ./routing/sample/logs
INFO:gen_uh_init>> params directory is ./routing/sample/params
INFO:gen_uh_init>> inputs directory is ./routing/sample/inputs
INFO:gen_uh_init>> aggregated directory is ./routing/sample/temp/aggregated
INFO:gen_uh_init>> remapped directory is ./routing/sample/temp/remapped
INFO:gen_uh_init>> Opened Pour Points File: ./routing/sample/inputs/sample_pour.txt
ERROR:gen_uh_init>> Error opening pour points file: ./routing/sample/inputs/sample_pour.txt
ERROR:gen_uh_init>> 'DataFrame' object has no attribute 'ix'
Traceback (most recent call last):
  File "/home/slim/.local/lib/python3.7/site-packages/rvic/parameters.py", line 188, in gen_uh_init
    pour_points.ix[i, 'names'] = strip_invalid_char(name)
  File "/home/slim/.local/lib/python3.7/site-packages/pandas/core/generic.py", line 5274, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'ix'
ERROR:write>> Traceback (most recent call last):
ERROR:write>>   File "/home/slim/.local/bin/rvic", line 61, in <module>
ERROR:write>> main()
ERROR:write>>   File "/home/slim/.local/bin/rvic", line 43, in main
ERROR:write>> parameters(args.config_file, numofproc=args.numofproc)
ERROR:write>>   File "/home/slim/.local/lib/python3.7/site-packages/rvic/parameters.py", line 51, in parameters
ERROR:write>> outlets, config_dict, directories = gen_uh_init(config_file)
ERROR:write>>   File "/home/slim/.local/lib/python3.7/site-packages/rvic/parameters.py", line 188, in gen_uh_init
ERROR:write>> pour_points.ix[i, 'names'] = strip_invalid_char(name)
ERROR:write>>   File "/home/slim/.local/lib/python3.7/site-packages/pandas/core/generic.py", line 5274, in __getattr__
ERROR:write>> return object.__getattribute__(self, name)
ERROR:write>> AttributeError
ERROR:write>> :
ERROR:write>> 'DataFrame' object has no attribute 'ix'

I had to change this line from .ix[i, 'names'] to .loc[i, 'names]:

ERROR:write>>   File "/home/slim/.local/lib/python3.7/site-packages/rvic/parameters.py", line 188, in gen_uh_init
ERROR:write>> pour_points.ix[i, 'names'] = strip_invalid_char(name)

[eyvorchuk]

@sum1lim Where'd you get that config file? It's not in samples/configs.