VAC notebook

[gschwend]

Write an e2e notebook to serve as the user interface to create VACs on the LIneA OPen OnDemand platform. The bulk code should be hidden in ancillary Python modules. The notebook must be as clean as possible and mostly used to get parameters from the user.

Preparation steps outside the notebook - 1st time

• create conda env
• create jupyter kernel

Preparation steps outside the notebook - before every new process

• prepare pz-compute process dir inside user's scratch area (for provenance tracking) (it should include a copy of the VAC notebook inside the process dir)

Steps inside the notebook

• make footprint map from the combination of survey conditions maps available (optional, default=False)
• create temporary skinny tables in lustre T0 scratch area as the result of region selection + object selection + apply photometric corrections
• make input data QA plots
• run pipeline TSM from pzserver lib to create train/test sets
• run RAIL inform on training set (optional, if False, reuse existing .pkl file)
• run RAIL estimate on test set
• run RAIL evaluate on test results
• run pipeline pz-compute on the skinny tables
• make output data QA plots
• create wazp input files = FITS files with skinny table columns + pz PDF percentiles partitioned by healpix pixels of fixed resolution, maglim file, pz metrics file
• delete temporary skinny tables
• export notebook to HTML
• save a copy of notebook (.ipynb without outputs and .HTML with outputs) inside the process dir with process_id and timestamp prefixes
• update process_info.yaml with notebook filenames
• if production mode, move process dir to wazp project shared directory on lustre T1