What is the output product?

[TonioF]

What is the output product? What is its relationship to orchestration code? Is it a time-series or an updated product? Which formats do we support? GeoTiff? NetCDF? Something else?

[jgomezdans]

The simplest thing to have is to have a bunch of single band files per parameter per time step (e.g. LAI_20180215.tif, LAI20180216.tif etc). They can then be put together in post-processing and saved to arbitrary formats. It also allows for convenient mosaicking. Storing uncertainty can be more complex, and I think two options should be given:

1. Estimate of the parameter variance per time step (so sigma_LAI_20180215.tif, sigma_LAI20180216.tif, etc)
2. Estimate of full, per-pixel covariance matrix per timestep

The latter is a pain. It's a symmetric square matrix, so if you have N parameters, I think you need to store 0.5*(N+1)*N. So for 10 parameters, that's 55 elements per pixel (!!).

There are some important assumptions made here: all parameters belonging to one pixel are independent, and this would be in transformed parameter space.

[JorisTimmermans]

I agree with Jose that that would be a huge pain in the bumm.. At the moment the (flattened) uncertainty is saved as the 2nd layer in the LAI_tiff. Does this need to be changed? cheers Joris <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virusvrij. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

…

On Fri, Feb 16, 2018 at 8:05 PM, José Gómez-Dans ***@***.***> wrote: The simplest thing to have is to have a bunch of single band files per parameter per time step (e.g. LAI_20180215.tif, LAI20180216.tif etc). They can then be put together in post-processing and saved to arbitrary formats. It also allows for convenient mosaicking. Storing uncertainty can be more complex, and I think two options should be given: 1. Estimate of the parameter variance per time step (so sigma_LAI_20180215.tif, sigma_LAI20180216.tif, etc) 2. Estimate of full, per-pixel covariance matrix per timestep The latter is a pain. It's a symmetric square matrix, so if you have N parameters, I think you need to store 0.5*(N+1)*N. So for 10 parameters, that's 55 elements per pixel (!!). There are some important assumptions made here: all parameters belonging to one pixel are independent, and this would be in transformed parameter space. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOWbjIwoIcq4w-aLVK8Bj3xaZMLue9FYks5tVdFwgaJpZM4SIlx7> .

[jgomezdans]

Hi,
Further to option 2... We can think of Option 1 as a "compressed" version of option 2. Option 2 would build up on Option 1 by storing the off-diagonal correlations rather than covariances. It would then be possible to reconstruct the covariance matrix directly from this. The advantage would be that it ought to be possible to store the correlations as a signed Byte, as the correlation would be -1 <= rho <= 1, and a quantisation step of ~0.01 would be OK. This would reduce file sizes: take a 10 element state vector, and that's ~180 bytes per pixel for the straightforward option. With the byte option, storage would ~78 bytes or thereabouts (you still need a float32 for the main diagonal terms, meaning 40 bytes are already taken up by storing option 1).

One further thing to note is that parameter names and ordering ought to also be stored.

[TonioF]

I prefer the idea of having one file per time step, too. Having one file per variable probably also makes sense, though this introduces quite a lot of files. We have the Data Access Component to deal with that, though :) .
About the uncertainties: As José has correctly pointed out in his last sentence, when we store the uncertainty for a single variable in some place, we would need to make sure that it is clear what are the relationships to other variables (which are not in this file). One might argue, that having covariance matrices when there is only one variable makes no sense. In #2 though, we stated that we want to save intermediate results, which would then also be the uncertainties. So, we should pursue option 2, but create one single uncertainty file for all (in José's example: ten) variables. We can use his approach to reduce storage cost by scaling, but we would benefit most from some method to generalize the information. If the pixel information is independent (as is assumed), we can't really do something here.