Merging of multiple available priors for one variable

[tramsauer]

I think, this should be done in the prior_engine itself.
Also, I'm thinking of an OrderedDict of the prior config, to keep the order of entered priors.. is that suitable? does this work with YAML - imported config files?

[TonioF]

One remark to begin with: I would like to the modules ending on prior in the prior_engine to prior_creator (or prior_file_creator). Prior would become PriorCreator, VegetationPrior VegetationPriorCreator and so on. The rationale behind this is that the term "prior" is used for these modules (and the classes within them), the output files, the variables, and the information within these files. I always need some time to find which of these is meant when we talk about a prior. So, for the sake of clarity, I propose to rename the modules (and accordingly the classes) to prior_creator.
Please let me know if this is a terrible idea or if the term prior_file_creator would be better in your opinion.

I fully agree that the merging (i.e., the building or updating of the .vrt-file) should take place in the prior engine itself. Ideally, the prior engine would decide on its own how to merge. One prior_creator would be the default for a variable; the user_prior would always be preferred if it is set.

[JorisTimmermans]

I am fine with the change. cheers Joris

…

On Thu, May 31, 2018 at 1:22 PM, Tonio Fincke ***@***.***> wrote: One remark to begin with: I would like to the modules ending on prior in the prior_engine to prior_creator (or prior_file_creator). Prior would become PriorCreator, VegetationPrior VegetationPriorCreator and so on. The rationale behind this is that the term "prior" is used for these modules (and the classes within them), the output files, the variables, and the information within these files. I always need some time to find which of these is meant when we talk about a prior. So, for the sake of clarity, I propose to rename the modules (and accordingly the classes) to prior_creator. Please let me know if this is a terrible idea or if the term prior_file_creator would be better in your opinion. I fully agree that the merging (i.e., the building or updating of the .vrt-file) should take place in the prior engine itself. Ideally, the prior engine would decide on its own how to merge. One prior_creator would be the default for a variable; the user_prior would always be preferred if it is set. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOWbjEtUYadTCZChZygPjQm1EnFxqkEYks5t39KRgaJpZM4UTd5s> .

[tramsauer]

@TonioF I fully agree with your ideas here.
following questions concerning your statement

One prior_creator would be the default for a variable; the user_prior would always be preferred if it is set.

arises:
How to keep things sorted if multiple user priors are added? Which one of multiple possibly added user prior files gets the lead?

Two ideas:

• Use the OrderedDict somehow when reading and creating the prior configuration file and let the user know that his/her order of user prior data input matters.
• Add some sort of 'importance level' property to user priors when being added (which could replace the default sorting and make the usage of an OrderedDict obsolete) and let the user optionally set these levels when adding/importing their user prior files.

While writing this, I think I'm in favor of the second option.
What do both of you think?

[TonioF]

I actually prefer the first solution. Users should define an identifier for their prior data and specify this when retrieving the prior info. This could be the prior type, as this has to be entered anyway. Does this make sense?

The problem with the second approach is that you'd need to know in advance how important your prior is. At some point you might like to re-schedule that importance and you'd like to take the top priority away from a prior.

[tramsauer]

I actually prefer the first solution. Users should define an identifier for their prior data and specify this when retrieving the prior info. This could be the prior type, as this has to be entered anyway. Does this make sense?

There is already automatically an identifier specified when adding an user prior:

• User prior entries in the config file are already labeled "user_1", "user_2", .. as prior types with the numbering being based on existance of such " user_* " entries in the config file.

Still one would have to decide on the (subjective) importance of the priors at some point. Options could be:

• sorting by order of entry (which would mainly apply for the user priors, because otherwise climatology would always be first, unless reverse sorting.. 🤔 )
• adding a function before merging the prior files for one variable, which lets the user interactively select the order and type (replacing? based on uncertainty?) of merging.. this could also be set optional. So, having the order of entry in the config file matter by default and provide a switch in the config, which would then allow the user to explicitly set merging options.. (while writing, this is my preferred solution..)

This would support @TonioF s idea here:

The problem with the second approach is that you'd need to know in advance how important your prior is. At some point you might like to re-schedule that importance and you'd like to take the top priority away from a prior.

What do you think @JorisTimmermans (and maybe @TonioF ) ?

[JorisTimmermans]

My question is why we would need user_1, user_2 etc... to specify user priors. Are 'configuration'/'request' files not unique per user/run? In my opinion, user specified priors always get a priority over regular priors. cheers Joris

…

On Fri, Jun 29, 2018 at 11:48 AM, RaT0M ***@***.***> wrote: I actually prefer the first solution. Users should define an identifier for their prior data and specify this when retrieving the prior info. This could be the prior type, as this has to be entered anyway. Does this make sense? There is already automatically an identifier specified when adding an user prior: - User prior entries in the config file are already labeled "user_1", "user_2", .. as *prior types* with the numbering being based on existance of such " *user_** " entries in the config file. Still one would have to decide on the (subjective) importance of the priors at some point. Options could be: - sorting by order of entry (which would mainly apply for the user priors, because otherwise climatology would always be first, unless reverse sorting.. 🤔 ) - adding a function before merging the prior files for one variable, which lets the user interactively select the order and type (replacing? based on uncertainty?) of merging.. this could also be set optional. So, having the order of entry in the config file matter by default and provide a switch in the config, which would then allow the user to explicitly set merging options.. (while writing, this is my preferred solution..) This would support @TonioF <https://github.com/TonioF> s idea here: The problem with the second approach is that you'd need to know in advance how important your prior is. At some point you might like to re-schedule that importance and you'd like to take the top priority away from a prior. What do you think @JorisTimmermans <https://github.com/JorisTimmermans> (and maybe @TonioF <https://github.com/TonioF> ) ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOWbjChq2VSBFfljIfR6IIdV-cLpUhh4ks5uBff7gaJpZM4UTd5s> .

[tramsauer]

Are 'configuration'/'request' files not unique per user/run?

My thoughts:
Yes, config file is unique per user/run. I thought of the user_1, user_2 to account for multiple user defined priors for one single variable. So the configuration after adding user defined prior data could e.g. look like:

.....

Prior:
  General:
    directory_data: /home/joris/Data/Prior_Engine/
  sm:
    climatology:
      dir: ./aux_data/Climatology/SoilMoisture/
    coarse:
      dir: ./aux_data/Coarse/SoilMoisture/
    user1:
      file: "path/to/some/file"
    user2:
      dir: "path/to/some/other/file"

  lai:
    database: 'some/db'
    dynamic:
        dir: 'path/directory'
    static:
        file: 'path/to/static'
    user1:
        file: 'path/to/file'

I agree, that

user specified priors always get a priority over regular priors.

at least as default.

But, should the other values -climatology, database,... - be omitted completely (by default if a user defined prior is added)?

In the future, an option to chose the order and method of merging multiple priors for one variable could be added..?

[JorisTimmermans]

why not specify: user: file1: "path/to/some/file" dir2: "path/to/some/other/file" If a person want's to use his prior, there are 2 possibilities: either the default-priors (pixels) are overwritten, or only part of the priors are overwritten (trait-priors are overwritten, and remapped over study area) cheers Joris

…

On Fri, Jun 29, 2018 at 2:34 PM, RaT0M ***@***.***> wrote: Are 'configuration'/'request' files not unique per user/run? My thoughts: Yes, config file is unique per user/run. I thought of the *user_1*, *user_2* to account for multiple user defined priors for *one single variable*. So the configuration after adding user defined prior data could e.g. look like: ..... Prior: General: directory_data: /home/joris/Data/Prior_Engine/ sm: climatology: dir: ./aux_data/Climatology/SoilMoisture/ coarse: dir: ./aux_data/Coarse/SoilMoisture/ user1: file: "path/to/some/file" user2: dir: "path/to/some/other/file" lai: database: 'some/db' dynamic: dir: 'path/directory' static: file: 'path/to/static' user1: file: 'path/to/file' I agree, that user specified priors always get a priority over regular priors. at least as default. But, should the other values -climatology, database,... - be omitted completely (by default if a user defined prior is added)? In the future, an option to chose the order and method of merging multiple priors for one variable could be added..? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOWbjIu-WMaFemXeBVvTv_1b7KHNWfYOks5uBh7pgaJpZM4UTd5s> .

[tramsauer]

For now, the Prior Engine calls for every variable and every ptype the specific 'subengine', e.g. soilmoisture_prior_creator. Also, we decided that merging of multiple priors should take place inside the prior engine itself.

Therefore, having multiple user ptypes (as it is already implemented: code) would just bring the files to the prior engine for merging.
Personally, I could also live with your approach, but we would have to merge in the user_prior_creator already, too? and sort out the order and method of merging there as well..

I do have a routine in the sm_prior_creator for merging - for now only mean of rasters - and others (e.g. get a single file and check for compliance) which could/should be stored in the prior_creator.py base class as well, so they are available to all instances. I will create another branch for that.

[TonioF]

We wouldn't necessarily have to overwrite data. We could still leave data we calculated once and have then the .vrt-file reference the correct .tiff. Also note that we can have overlapping .tiff files (useful, when user data is only available for a certain region): "If there is some amount of spatial overlapping between files, the order of files appearing in the list of source matter: files that are listed at the end are the ones from which the content will be fetched." ( http://www.gdal.org/gdalbuildvrt.html )