Implement the Raman water vapor processing

[martin-rdz]

General strategy

• Use the aggregated cloud free profiles together with the ECMWF model mixing ratio
• Later implement the reading for radiosondes (and MWR integrated water vapor)
• Store to sqlite database (new table)
• Calculate the high resolution wvmr similar to the attbsc (using the calibration constant from the profiles).

[martin-rdz]

Some starting point for the implementation of the profile-wise calibration:
Met Data

time_slices = [data_cube.retrievals_highres['time64'][grp] for grp in data_cube.clFreeGrps]
data_cube.met.get_mean_profiles(time_slices)

That returns a list of xarray.Dataset that are averaged to the clFreeGrps. Each contains the height, temperature, pressure, relative humidity and water vapor mixing ratio q interpolated to the Polly vertical resolution. In principal all the variables from the cloudnet ecmwf file are available, but not all are averaged (see io/readMeteo.py for details). That sippet is for example also used at PicassoProc.calcMolecular

Note: within the class in PicassoProc.py, the name data_cube has to replaced by self.

Molecular profiles

i = 0
molExt_387 = data_cube.mol_profiles[f'mExt_387'][i, :].copy()
molExt_407 = data_cube.mol_profiles[f'mExt_407'][i, :].copy()

height = data_cube.retrievals_highres['range']
molOD_387 = np.nancumsum(molExt_387 * np.concatenate(([height[0]], np.diff(height))))
...
trans_387 = np.exp(-2 * (molOD_387)) # neglecting the aerOD

Where also i is the index of the clFreeGrps
rho_air is once calculated at molecular.moist_air_density, but is never stored. Maybe we can adapt molecular.calc_profiles to store rho_air similar to the the number_density.

Background corrected signal

cldFree = data_cube.clFreeGrps[0]  # e.g. [0, 113]
cldFree = cldFree[0], cldFree[1] + 1 # for consistency with matlab
sigBGCor_387 = np.nanmean(np.squeeze(
    data_cube.retrievals_highres['sigBGCor'][slice(*cldFree), :, data_cube.gf('387', 'total', 'FR')]
    ), axis=0)
...

The data_cube.gf is a shorthand that gives a true false array to slice the channels of the (time, height, channel) array.

Structure of the WVC dictionary
The structure of the storage for the water vapor calibration constant should follow the data_cube.pol_cali or data_cube.LC.
Basically the first key is the method name (model_profile? and later mwr_iwv or radiosonde_profile - but open for discussion).
Then the channel. maybe we can omit that or have 407_FR for consistency. And finally a list of dictionaries with the actual calibration constant, the standard deviation (or other measure of accuracy, the start and end time (status is optional).
As an example:

>> data_cube.pol_cali
{'D90': {'355_FR': [{'eta': 47.74241805778966,
    'eta_std': 1.0396617649540694,
    'time_start': 1724207790,
    'time_end': 1724208000,
    'status': 1},
   {'eta': 48.418225044530985,
    'eta_std': 2.1214941769359217,
    'time_start': 1724222190,
    'time_end': 1724222400,
    'status': 1}],
  '532_FR': [{'eta': 12.380587648929659,
    'eta_std': 0.13602867159593476,
    'time_start': 1724207790,
    'time_end': 1724208000,
    'status': 1},
   {'eta': 12.130505397817474,
    'eta_std': 0.6827165364591472,
    'time_start': 1724222190,
    'time_end': 1724222400,
    'status': 1}]}}

The start and end unix timestamp of a cldFreeGrp can be read out with.

cldFreeTime = np.array(data_cube.retrievals_highres['time'])[cldFree]

With the key 'time64', it will provide a np.datetime64.

The code for the matlab version is online as well here and here
Hope I did not overlook anything very important.
The adaption of the database is a bit more tedious, maybe we should do that together.