Examples

We provide some examples for the Extended Sobol' Sensitivity Analysis xSSA method that is not only able to estimate parametric sensitivities but also the sensitivity of process options and processes. We use the examples used in our publication:

1. Benchmark models (Setup A and B in Figure 1)
2. Salmon River catchment using RAVEN hydrologic modelling framework (Setup C in Figure 1)

Figure 1: The three model setups used in this study. The first two serves as a artificial benchmark model since sensitivities of parameters, process options and processes can be derived analytically. The benchmark examples consists of three processes: A, B, and C (D, E, and F). The three processes are connected through A * B + C (D * E + F) to obtain the hypothetical model outputs. Processes A (D) and C (F) have two options, process B (E) has three. The parameters x_i required for each process option are listed right of the respective process options. Each parameter of the disjoint-parameter benchmark model (A) appears in exactly one process option (disjoint parameters) while in the more advanced benchmark model (B) parameters x₁ and x₃ appear in multiple process options and processes. The process formulations can be found in Eq.s 3 to 9 and Eq.s 10 to 16. (C) The third setup is used for a sensitivity analysis of the hydrologic modeling framework Raven. The three options M_i are used for the infiltration process, three options N_i for quickflow, two options O_i for evaporation, two options P_i for baseflow, and three options Q_i for snow balance. All other processes needed for the model are used with one fixed option, i.e., convolution for surface and delayed runoff R_i and S_i, respectively, potential melt T_i, percolation U_i, rain-snow partitioning V_i, and precipitation correction W_i. The remaining processes also have only one option but none of them contains tunable parameters. They are merged to a "remaining" process X_i. The Raven model parameters x₁ to x₃₅ are listed right of the process options. Details about the chosen process options can be found in Appendix C Table C1, and details on the model parameters and their ranges in Appendix C Table C2 of our publication.

Benchmark models

The two benchmark models with disjoint parameters (Figure 1A) and shared parameters (Figure 1B) are using three artificial processes with 2 to 3 process options. The example codes are here. The parameter ranges are given in examples/data_in/data_model/benchmark/parameters.txt.

To run the Extended Sobol' Sensitivity Analysis xSSA you just need to execute the script __run_benchmark_DVM.py or __run_benchmark_xSSA.py. The first will run the analysis using the approach of the Discrete Value Method (DVM; Baroni and Tarantola, 2014) while the second one is the method proposed here (xSSA; Mai et al. 2020). Both will run the analysis using a range of Sobol' reference parameter sets K (50, 100, 200, 500, 1000, 2000, 5000, 10000, 20000, 50000, 100000). The results will be saved in a Pickle file in the folder examples/data_out/benchmark/results_<setup>-benchmark-model_<method>.pkl where <setup> is either "disjoint" (Figure 1A) or "shared" (Figure 1B) and <method> is either "DVM" or "xSSA".

python __run_benchmark_DVM.py 
python __run_benchmark_xSSA.py 

The numerical Sobol' indexes can then be compared to the theoretical values that were derived using Mathematica. The error is converging to zero with increasing numbers of reference parameter sets K (specified through option -n).

Figure 2: Error between xSSA approximated and analytically-derived theoretical Sobol' sensitivity index estimates S_i^(appr) and S_i^(theo), respectively. The errors are derived for the main Sobol indexes S_i and the total Sobol indexes ST_i^(appr). The analyses were performed for (A,B) model parameters, (C,D) process options, and (E,F) processes of the shared-parameter benchmark model (Figure 1B). The analyses are performed using increasing numbers of Sobol' references sets K. Note that the x-axis is in logarithmic scale.

Salmon River catchment using RAVEN hydrologic modeling framework

As a real world example we analyzed the hydrologic modelling framework RAVEN with its flexibility regarding model structure. The example catchment we used is the Salmon River in British Columbia, Canada (Figure 3). The Salmon River catchment located in the Canadian Rocky Mountains in British Columbia is selected as the study watershed. The domain is depicted in Figure 3A and was chosen only for the purpose of demonstrating the proposed method. The catchment drains towards a Water Survey Canada (WSC) streamflow gauge station near Prince George (WSC ID 08KC001; latitude 54.09639°N, latitude -122.67972°W, elevation 606m) and has continuous data since 1953. The 4230 km² large, low-human impacted catchment is mainly evergreen needleleaf forested (83% of whole domain) on a loamy (63%) and loamy sandy (25%) soil (Fig. 3C and Fig. 3D).

Figure 3: (A) Location of the Salmon River catchment (red polygon) in British Columbia, Canada. The watershed is 4230 km² and located around 700 km north of Vancouver. It is located in the Rocky Mountains with an elevation of 606 m above sea level at the streamflow gauge station of the Salmon River (08KC001). (B) The average monthly mean temperatures (red line) and average monthly precipitation is divided into rain (dark blue) and snow (light blue). Maps of (C) the four soil types based on the Harmonized World Soil Data (HWSD; 30'') (Nachtergaele et al., 2010) and (D) four land cover types based on the MCD12Q1 MODIS/Terra+Aqua Land Cover (500m) (Friedl and Sulla-Menashe, 2015) of the Salmon River catchment are provided. The colors indicate different soil and land use classes.

We selected several options for different processes in RAVEN for this study (Figure 1C). In total 108 discrete models can be setup with these options while the choice of the first option for each process resembles the HMETS model. The Raven setup is using three artificial processes with 2 to 3 process options (see Figure 1C). The example codes are here. The parameter ranges are given in examples/data_in/data_model/08KC001/parameters.txt. Raven needs five setup files (*.rvi, *.rvp, *.rvt, *.rvh, and *.rvc) that contain parameters and derived parameters. The template files are encoded in raven_templates.py and will be used to fill in sampled parameters. The script raven_common.py contains helper routines needed to run the model.

To run the Extended Sobol' Sensitivity Analysis xSSA you just need to execute the script __run_xSSA.py. It will run the analysis using 1000 Sobol' reference parameter sets if not otherwise specified (option -n). If you want to save the model runs in a Pickle file, use option -o and set it to "pkl". If you want to save the model runs in a NetCDF file, use option -o and set it to "nc". The user also needs to assign a basin ID (-i) which is primarily used to create unique folder names in the temporary directory but also to obtain basin metadata from the provided file basin_physical_characteristics.txt.

# takes about 24 hours (results used for publication)
python __run_xSSA.py -n 1000 -o "nc"  -i "08KC001"

# takes about 10 min (good for some testing but do not rely on Sobol' indexes derived)
python __run_xSSA.py -n 10 -o "nc"  -i "08KC001"

The results will be stored in a Pickle or NetCDF file (depending on what you set for -o). The file will be located in examples/data_out/08KC001/results_nsets1000.nc.

Please note:

• You need to provide a Raven executable in the setup folder examples/data_in/data_model/08KC001/Raven.exe.
• The analysis using K=1000 Sobol' reference sets takes about 24 hours. Maybe start with setting -n 10.

Figure 4: Results of the Sobol' sensitivity analysis of the hydrologic modeling framework Raven. (A) The sensitivities of 35 model parameters (see Table C.2 in our publication) and 8 parameters r_i that are used to determine the weights of process options are estimated. The Sobol' sensitivity index estimates are determines also for (B) 19 process options and (C) the 11 processes. The different colors indicate the association of parameters and process options to the eleven processes. Parameters x₂₉ and x₃₀ are associated with several process options and are not colored but gray. The Sobol' main and total effects are shown (dark and light colored bars, respectively). All sensitivity index estimates shown are originally time-dependent and are aggregated as variance-weighted averages (Eq.s 15 and 16 of our publication). The average weights over the course of the year are shown in Figure 5.

Figure 5: Results of the Sobol' sensitivity analysis of the hydrologic modeling framework Raven. The sensitivities of eleven processes are shown as their averages per day of the year (colored bars). The simulation period is 1991 to 2010. The sensitivities are normalized such that they sum up to 1.0 at every day of the year (Eq. 25 of our publication). The sensitivities are variance-weighted averages (Eq. 24 of our publication). The (average) weight of each day of the year is shown as a black line. The weights for every time step are determined by the average simulated discharge at this time step (V(t) in Eq. 24 of our publication). The time-aggregated sensitivity index estimates of the eleven processes is shown in Figure 4C.