threiten · maxgalli · Oct 27, 2020 · Feb 9, 2021 · Oct 27, 2020 · Oct 27, 2020
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,59 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+# C extensions
+*.so
+
+# Distribution / packaging
+bin/
+build/
+develop-eggs/
+dist/
+eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+.tox/
+.coverage
+.cache
+nosetests.xml
+coverage.xml
+
+# Translations
+*.mo
+
+# Mr Developer
+.mr.developer.cfg
+.project
+.pydevproject
+
+# Rope
+.ropeproject
+
+# Django stuff:
+*.log
+*.pot
+
+# Sphinx documentation
+docs/_build/
+
+# vim
+*.swp
+*.swo
+
+# config files and scripts
+*.sh
+*.yml
+*.yaml
diff --git a/README.md b/README.md
@@ -1,77 +1,28 @@
-# Chained quantile regression
-This repo contains the code to do data/MC correction using chained quantile regression and stochastic matching. 
-The class `quantileRegression_chain` can be used to correct a set of continious variables differentially and 
-while keeping their correlations. The class `quantileRegression_chain_disc` can be used to correct discontinious variables.
-## Training BDTs for quantiles
-To train the BDTs that will be used to extract the conditional pdf the functions `trainOnData` for data and `trainOnMC` 
-for MC have to be used. For example:
-```python
-import quantileRegression_chain as qRegC
-qRC = qRegC.quantileRegression_chain(year,EBEE,workDir,variables)
-qRC.loadDataDF(df_name,0,stop,rsh,columns)
-qRC.trainOnData(variable,weightsDir)
-```
-# Scripts for training
-The strategy to train on a large dataset is the following
-1. Train on data
-
-	To train on data use `scripts/run_qRC_training.sh`
-	```bash
-	./run_qRC_training.sh <config_file_ShowerShapes>.yaml <config_file_PhotonIso>.yaml <config_file_ChargedIsos>.yaml <n_evts> <EB/EE>
-	```
-	This will submit one job per quantile per variable to the SGE queue via qub. BEWARE: There is a hard coded path in this script. Change it accordingly
-
-2. Train Shower Shapes on MC
-
-   To train the shower shape correction for MC use `training/train_qRC_MC.py`. Before starting the training on MC, the training on Data needs to be finished completely
-   ```bash
-   python train_qRC_MC.py  -c <config_file_ShowerShapes>.yaml -N <n_evts> -E <EB/EE> -B <cluster_profile> -i <cluster_id>
-   ```
-
-3. Train Isolations on MC
-
-	To train the shower shape correction for MC use `training/train_qRC_MC.py`
-   ```bash
-   python train_qRC_I_MC.py  -c <config_file_(PhotonIso/ChargedIsos)>.yaml -N <n_evts> -E <EB/EE> -B <cluster_profile> -i <cluster_id>
-   ```
-
-## Final corrections training
-
-After validating the initial training, one can train the final single regressors that can be used to apply the corrections to the simulation in production. To do so, follow these steps:
-
-1. Train the final shower shape corrections
-
-	To train the final shower shape correction use `training/train_final_Reg_SS.py`
-	```bash
-	python train_final_Reg_SS.py  -c <config_file_ShowerShapes>.yaml -N <n_evts> -E <EB/EE> -B <ipython_cluster_profile> -i <cluster_id> -n 21
-	```
-2. Train final charged Iso corrections
-
-	To train the final correction for the charged isolations use `training/train_final_Reg_Iso.py`
-	```bash
-	python train_final_Reg_Iso.py  -c <config_file_(ChargedIsos)>.yaml -N <n_evts> -E <EB/EE> -B <ipython_cluster_profile> -i <cluster_id> -n 21
-	```
-
-3. Train final photon Iso corrections
-
-	To train the final correction for the photon isolation use `training/train_final_Reg_Iso.py`
-	```bash
-	python train_final_Reg_Iso.py  -c <config_file_(PhotonIso)>.yaml -N <n_evts> -E <EB/EE> -B <ipython_cluster_profile> -i <cluster_id> -n 21
-	```
-
-	The only difference between the command for charged and photon Iso are the config files
-
-### Note on config files
-
-In general the config files for the training for data and simulation for the initial and final training have the same format. Examples can be found in `examples`. The following keywords should be used
-
-| Keyword    | Used for                                                                   |
-|------------|----------------------------------------------------------------------------|
-| Dataframes | `(data/mc)_(EB/EE)` for the dataframe for data/MC in EB/EE                 |
-| variables  | The list of variables to be corrected. The order here is important         |
-| year       | The year of data-taking the relevant datasets are from                     |
-| workDir    | the path to the working dir, dataframes and weightsDir need to be in there |
-| weightsDir | directory to store the weights. Create before training                     |
-| outDir     | directory to store the final weight. Create before training                |
+# Quantile Regression Chain for Data/MC Corrections
 
+## Motivation
+TODO
 
+## Installation
+After cloning the repository and entering it, run:
+```bash
+$ python setup.py install
+```
+to install the package.
+All the dependencies should be handled by the ```setup.py``` script itself.
+
+## Run
+The process is rather long and involves submission of jobs on a cluster running [SLURM](https://slurm.schedmd.com/) and internal parallelization that can be performed either with [Ray](https://ray.io/) or [IPyParallel](https://ipyparallel.readthedocs.io/en/latest/#). 
+The ```utils``` directory contains directories numbered from 1 to 9 with the code and instructions to run, plus directory (```setup_ray```) with the scripts to setup a Ray cluster for the internal parallelization needed in some steps.
+
+Following are the main steps:
+
+1. TagAndProbe NTuples production
+2.  Pandas dataframes production
+3. Data training
+4. Montecarlo training
+5. Bayesian optimization
+6. Final training and IdMVA computation
+7. Plot
+8. Systematic uncertainties 
+9. Regressors conversion
diff --git a/__init__.py b/__init__.py
diff --git a/legacy/README.md b/legacy/README.md
@@ -0,0 +1,77 @@
+# Chained quantile regression
+This repo contains the code to do data/MC correction using chained quantile regression and stochastic matching. 
+The class `quantileRegression_chain` can be used to correct a set of continious variables differentially and 
+while keeping their correlations. The class `quantileRegression_chain_disc` can be used to correct discontinious variables.
+## Training BDTs for quantiles
+To train the BDTs that will be used to extract the conditional pdf the functions `trainOnData` for data and `trainOnMC` 
+for MC have to be used. For example:
+```python
+import quantileRegression_chain as qRegC
+qRC = qRegC.quantileRegression_chain(year,EBEE,workDir,variables)
+qRC.loadDataDF(df_name,0,stop,rsh,columns)
+qRC.trainOnData(variable,weightsDir)
+```
+# Scripts for training
+The strategy to train on a large dataset is the following
+1. Train on data
+
+	To train on data use `scripts/run_qRC_training.sh`
+	```bash
+	./run_qRC_training.sh <config_file_ShowerShapes>.yaml <config_file_PhotonIso>.yaml <config_file_ChargedIsos>.yaml <n_evts> <EB/EE>
+	```
+	This will submit one job per quantile per variable to the SGE queue via qub. BEWARE: There is a hard coded path in this script. Change it accordingly
+
+2. Train Shower Shapes on MC
+
+   To train the shower shape correction for MC use `training/train_qRC_MC.py`. Before starting the training on MC, the training on Data needs to be finished completely
+   ```bash
+   python train_qRC_MC.py  -c <config_file_ShowerShapes>.yaml -N <n_evts> -E <EB/EE> -B <cluster_profile> -i <cluster_id>
+   ```
+
+3. Train Isolations on MC
+
+	To train the shower shape correction for MC use `training/train_qRC_MC.py`
+   ```bash
+   python train_qRC_I_MC.py  -c <config_file_(PhotonIso/ChargedIsos)>.yaml -N <n_evts> -E <EB/EE> -B <cluster_profile> -i <cluster_id>
+   ```
+
+## Final corrections training
+
+After validating the initial training, one can train the final single regressors that can be used to apply the corrections to the simulation in production. To do so, follow these steps:
+
+1. Train the final shower shape corrections
+
+	To train the final shower shape correction use `training/train_final_Reg_SS.py`
+	```bash
+	python train_final_Reg_SS.py  -c <config_file_ShowerShapes>.yaml -N <n_evts> -E <EB/EE> -B <ipython_cluster_profile> -i <cluster_id> -n 21
+	```
+2. Train final charged Iso corrections
+
+	To train the final correction for the charged isolations use `training/train_final_Reg_Iso.py`
+	```bash
+	python train_final_Reg_Iso.py  -c <config_file_(ChargedIsos)>.yaml -N <n_evts> -E <EB/EE> -B <ipython_cluster_profile> -i <cluster_id> -n 21
+	```
+
+3. Train final photon Iso corrections
+
+	To train the final correction for the photon isolation use `training/train_final_Reg_Iso.py`
+	```bash
+	python train_final_Reg_Iso.py  -c <config_file_(PhotonIso)>.yaml -N <n_evts> -E <EB/EE> -B <ipython_cluster_profile> -i <cluster_id> -n 21
+	```
+
+	The only difference between the command for charged and photon Iso are the config files
+
+### Note on config files
+
+In general the config files for the training for data and simulation for the initial and final training have the same format. Examples can be found in `examples`. The following keywords should be used
+
+| Keyword    | Used for                                                                   |
+|------------|----------------------------------------------------------------------------|
+| Dataframes | `(data/mc)_(EB/EE)` for the dataframe for data/MC in EB/EE                 |
+| variables  | The list of variables to be corrected. The order here is important         |
+| year       | The year of data-taking the relevant datasets are from                     |
+| workDir    | the path to the working dir, dataframes and weightsDir need to be in there |
+| weightsDir | directory to store the weights. Create before training                     |
+| outDir     | directory to store the final weight. Create before training                |
+
+
diff --git a/apply/make_compl_df_qRC.py → legacy/apply/make_compl_df_qRC.py b/apply/make_compl_df_qRC.py → legacy/apply/make_compl_df_qRC.py
diff --git a/apply/make_corr_df.py → legacy/apply/make_corr_df.py b/apply/make_corr_df.py → legacy/apply/make_corr_df.py
diff --git a/...t/gpu/#output_bayOpt_gpu_probePhoIso.txt# → ...t/gpu/#output_bayOpt_gpu_probePhoIso.txt# b/...t/gpu/#output_bayOpt_gpu_probePhoIso.txt# → ...t/gpu/#output_bayOpt_gpu_probePhoIso.txt#
diff --git a/bayOpt/gpu/#output_bayOpt_gpu_probeR9.txt# → ...ayOpt/gpu/#output_bayOpt_gpu_probeR9.txt# b/bayOpt/gpu/#output_bayOpt_gpu_probeR9.txt# → ...ayOpt/gpu/#output_bayOpt_gpu_probeR9.txt#
diff --git a/...t/gpu/.#output_bayOpt_gpu_probePhoIso.txt → ...t/gpu/.#output_bayOpt_gpu_probePhoIso.txt b/...t/gpu/.#output_bayOpt_gpu_probePhoIso.txt → ...t/gpu/.#output_bayOpt_gpu_probePhoIso.txt
diff --git a/bayOpt/gpu/doBayOpt_qRC.py → legacy/bayOpt/gpu/doBayOpt_qRC.py b/bayOpt/gpu/doBayOpt_qRC.py → legacy/bayOpt/gpu/doBayOpt_qRC.py
diff --git a/...pt/gpu/output_bayOpt_gpu_probeChIso03.txt → ...pt/gpu/output_bayOpt_gpu_probeChIso03.txt b/...pt/gpu/output_bayOpt_gpu_probeChIso03.txt → ...pt/gpu/output_bayOpt_gpu_probeChIso03.txt
diff --git a/...tput_bayOpt_gpu_probeChIso03_ReReco16.txt → ...tput_bayOpt_gpu_probeChIso03_ReReco16.txt b/...tput_bayOpt_gpu_probeChIso03_ReReco16.txt → ...tput_bayOpt_gpu_probeChIso03_ReReco16.txt
diff --git a/...u/output_bayOpt_gpu_probeChIso03worst.txt → ...u/output_bayOpt_gpu_probeChIso03worst.txt b/...u/output_bayOpt_gpu_probeChIso03worst.txt → ...u/output_bayOpt_gpu_probeChIso03worst.txt
diff --git a/...bayOpt_gpu_probeChIso03worst_ReReco16.txt → ...bayOpt_gpu_probeChIso03worst_ReReco16.txt b/...bayOpt_gpu_probeChIso03worst_ReReco16.txt → ...bayOpt_gpu_probeChIso03worst_ReReco16.txt
diff --git a/...output_bayOpt_gpu_probeCovarianceIeIp.txt → ...output_bayOpt_gpu_probeCovarianceIeIp.txt b/...output_bayOpt_gpu_probeCovarianceIeIp.txt → ...output_bayOpt_gpu_probeCovarianceIeIp.txt
diff --git a/..._gpu_probeCovarianceIetaIphi_ReReco16.txt → ..._gpu_probeCovarianceIetaIphi_ReReco16.txt b/..._gpu_probeCovarianceIetaIphi_ReReco16.txt → ..._gpu_probeCovarianceIetaIphi_ReReco16.txt
diff --git a/...t/gpu/output_bayOpt_gpu_probeEtaWidth.txt → ...t/gpu/output_bayOpt_gpu_probeEtaWidth.txt b/...t/gpu/output_bayOpt_gpu_probeEtaWidth.txt → ...t/gpu/output_bayOpt_gpu_probeEtaWidth.txt
diff --git a/...put_bayOpt_gpu_probeEtaWidth_ReReco16.txt → ...put_bayOpt_gpu_probeEtaWidth_ReReco16.txt b/...put_bayOpt_gpu_probeEtaWidth_ReReco16.txt → ...put_bayOpt_gpu_probeEtaWidth_ReReco16.txt
diff --git a/...t/gpu/output_bayOpt_gpu_probePhiWidth.txt → ...t/gpu/output_bayOpt_gpu_probePhiWidth.txt b/...t/gpu/output_bayOpt_gpu_probePhiWidth.txt → ...t/gpu/output_bayOpt_gpu_probePhiWidth.txt
diff --git a/...put_bayOpt_gpu_probePhiWidth_ReReco16.txt → ...put_bayOpt_gpu_probePhiWidth_ReReco16.txt b/...put_bayOpt_gpu_probePhiWidth_ReReco16.txt → ...put_bayOpt_gpu_probePhiWidth_ReReco16.txt
diff --git a/bayOpt/gpu/output_bayOpt_gpu_probePhoIso.txt → ...Opt/gpu/output_bayOpt_gpu_probePhoIso.txt b/bayOpt/gpu/output_bayOpt_gpu_probePhoIso.txt → ...Opt/gpu/output_bayOpt_gpu_probePhoIso.txt
diff --git a/...utput_bayOpt_gpu_probePhoIso_ReReco16.txt → ...utput_bayOpt_gpu_probePhoIso_ReReco16.txt b/...utput_bayOpt_gpu_probePhoIso_ReReco16.txt → ...utput_bayOpt_gpu_probePhoIso_ReReco16.txt
diff --git a/bayOpt/gpu/output_bayOpt_gpu_probeR9.txt → .../bayOpt/gpu/output_bayOpt_gpu_probeR9.txt b/bayOpt/gpu/output_bayOpt_gpu_probeR9.txt → .../bayOpt/gpu/output_bayOpt_gpu_probeR9.txt
diff --git a/...pu/output_bayOpt_gpu_probeR9_ReReco16.txt → ...pu/output_bayOpt_gpu_probeR9_ReReco16.txt b/...pu/output_bayOpt_gpu_probeR9_ReReco16.txt → ...pu/output_bayOpt_gpu_probeR9_ReReco16.txt
diff --git a/bayOpt/gpu/output_bayOpt_gpu_probeS4.txt → .../bayOpt/gpu/output_bayOpt_gpu_probeS4.txt b/bayOpt/gpu/output_bayOpt_gpu_probeS4.txt → .../bayOpt/gpu/output_bayOpt_gpu_probeS4.txt
diff --git a/...pu/output_bayOpt_gpu_probeS4_ReReco16.txt → ...pu/output_bayOpt_gpu_probeS4_ReReco16.txt b/...pu/output_bayOpt_gpu_probeS4_ReReco16.txt → ...pu/output_bayOpt_gpu_probeS4_ReReco16.txt
diff --git a/.../gpu/output_bayOpt_gpu_probeSigmaIeIe.txt → .../gpu/output_bayOpt_gpu_probeSigmaIeIe.txt b/.../gpu/output_bayOpt_gpu_probeSigmaIeIe.txt → .../gpu/output_bayOpt_gpu_probeSigmaIeIe.txt
diff --git a/...ut_bayOpt_gpu_probeSigmaIeIe_ReReco16.txt → ...ut_bayOpt_gpu_probeSigmaIeIe_ReReco16.txt b/...ut_bayOpt_gpu_probeSigmaIeIe_ReReco16.txt → ...ut_bayOpt_gpu_probeSigmaIeIe_ReReco16.txt
diff --git a/bayOpt/gpu/run_bayOpt_gpu.sh → legacy/bayOpt/gpu/run_bayOpt_gpu.sh b/bayOpt/gpu/run_bayOpt_gpu.sh → legacy/bayOpt/gpu/run_bayOpt_gpu.sh
diff --git a/bayOpt/gpu/sub_bayOpt_gpu.sh → legacy/bayOpt/gpu/sub_bayOpt_gpu.sh b/bayOpt/gpu/sub_bayOpt_gpu.sh → legacy/bayOpt/gpu/sub_bayOpt_gpu.sh
diff --git a/convert/xgboost2tmva.ipynb → legacy/convert/xgboost2tmva.ipynb b/convert/xgboost2tmva.ipynb → legacy/convert/xgboost2tmva.ipynb
diff --git a/examples/config_apply_all_corr_5M.yaml → ...cy/examples/config_apply_all_corr_5M.yaml b/examples/config_apply_all_corr_5M.yaml → ...cy/examples/config_apply_all_corr_5M.yaml
diff --git a/...ples/config_apply_all_corr_syst_spl1.yaml → ...ples/config_apply_all_corr_syst_spl1.yaml b/...ples/config_apply_all_corr_syst_spl1.yaml → ...ples/config_apply_all_corr_syst_spl1.yaml
diff --git a/...ples/config_apply_all_corr_syst_spl2.yaml → ...ples/config_apply_all_corr_syst_spl2.yaml b/...ples/config_apply_all_corr_syst_spl2.yaml → ...ples/config_apply_all_corr_syst_spl2.yaml
diff --git a/examples/config_qRC_training_5M.yaml → legacy/examples/config_qRC_training_5M.yaml b/examples/config_qRC_training_5M.yaml → legacy/examples/config_qRC_training_5M.yaml
diff --git a/examples/config_qRC_training_ChI_5M.yaml → .../examples/config_qRC_training_ChI_5M.yaml b/examples/config_qRC_training_ChI_5M.yaml → .../examples/config_qRC_training_ChI_5M.yaml
diff --git a/examples/config_qRC_training_PhI_5M.yaml → .../examples/config_qRC_training_PhI_5M.yaml b/examples/config_qRC_training_PhI_5M.yaml → .../examples/config_qRC_training_PhI_5M.yaml
diff --git a/examples/finalRegression_settings.yaml → ...cy/examples/finalRegression_settings.yaml b/examples/finalRegression_settings.yaml → ...cy/examples/finalRegression_settings.yaml
diff --git a/scripts/job_train_qRC_I_data.sh → legacy/scripts/job_train_qRC_I_data.sh b/scripts/job_train_qRC_I_data.sh → legacy/scripts/job_train_qRC_I_data.sh
diff --git a/scripts/job_train_qRC_MC.sh → legacy/scripts/job_train_qRC_MC.sh b/scripts/job_train_qRC_MC.sh → legacy/scripts/job_train_qRC_MC.sh
diff --git a/scripts/job_train_qRC_data.sh → legacy/scripts/job_train_qRC_data.sh b/scripts/job_train_qRC_data.sh → legacy/scripts/job_train_qRC_data.sh
diff --git a/scripts/run_qRC_I_training.sh → legacy/scripts/run_qRC_I_training.sh b/scripts/run_qRC_I_training.sh → legacy/scripts/run_qRC_I_training.sh
diff --git a/scripts/run_qRC_training.sh → legacy/scripts/run_qRC_training.sh b/scripts/run_qRC_training.sh → legacy/scripts/run_qRC_training.sh
diff --git a/scripts/run_qRC_training_local.sh → legacy/scripts/run_qRC_training_local.sh b/scripts/run_qRC_training_local.sh → legacy/scripts/run_qRC_training_local.sh
diff --git a/scripts/test_qRC_Batch.sh → legacy/scripts/test_qRC_Batch.sh b/scripts/test_qRC_Batch.sh → legacy/scripts/test_qRC_Batch.sh
diff --git a/syst/produce_systematics.py → legacy/syst/produce_systematics.py b/syst/produce_systematics.py → legacy/syst/produce_systematics.py
diff --git a/syst/qRC_systematics.py → legacy/syst/qRC_systematics.py b/syst/qRC_systematics.py → legacy/syst/qRC_systematics.py
diff --git a/tmva/applyCorrection_tmva_ChIso.py → legacy/tmva/applyCorrection_tmva_ChIso.py b/tmva/applyCorrection_tmva_ChIso.py → legacy/tmva/applyCorrection_tmva_ChIso.py
diff --git a/tmva/computeCorrection_tmva_PhoIso.py → legacy/tmva/computeCorrection_tmva_PhoIso.py b/tmva/computeCorrection_tmva_PhoIso.py → legacy/tmva/computeCorrection_tmva_PhoIso.py
diff --git a/tmva/eleIDMVAonthefly.C → legacy/tmva/eleIDMVAonthefly.C b/tmva/eleIDMVAonthefly.C → legacy/tmva/eleIDMVAonthefly.C
diff --git a/tmva/phoIDMVAonthefly.C → legacy/tmva/phoIDMVAonthefly.C b/tmva/phoIDMVAonthefly.C → legacy/tmva/phoIDMVAonthefly.C
diff --git a/tmva/qRC_applyFinalRegression_tmva.py → legacy/tmva/qRC_applyFinalRegression_tmva.py b/tmva/qRC_applyFinalRegression_tmva.py → legacy/tmva/qRC_applyFinalRegression_tmva.py
diff --git a/tmva/qRC_computer.py → legacy/tmva/qRC_computer.py b/tmva/qRC_computer.py → legacy/tmva/qRC_computer.py
diff --git a/tmva/qRC_computer_PhoIso.py → legacy/tmva/qRC_computer_PhoIso.py b/tmva/qRC_computer_PhoIso.py → legacy/tmva/qRC_computer_PhoIso.py
diff --git a/tmva/qRC_xmlReader.C → legacy/tmva/qRC_xmlReader.C b/tmva/qRC_xmlReader.C → legacy/tmva/qRC_xmlReader.C
diff --git a/tmva/qRC_xmlReader_ChIso.C → legacy/tmva/qRC_xmlReader_ChIso.C b/tmva/qRC_xmlReader_ChIso.C → legacy/tmva/qRC_xmlReader_ChIso.C
diff --git a/tmva/qRC_xmlReader_PhoIso.C → legacy/tmva/qRC_xmlReader_PhoIso.C b/tmva/qRC_xmlReader_PhoIso.C → legacy/tmva/qRC_xmlReader_PhoIso.C
diff --git a/tmva/xgboost2tmva.py → legacy/tmva/xgboost2tmva.py b/tmva/xgboost2tmva.py → legacy/tmva/xgboost2tmva.py
@@ -4,6 +4,10 @@
 import xml.etree.cElementTree as ET
 regex_float_pattern = r'[-+]?(\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?'
 
+import logging
+logger = logging.getLogger(__name__)
+
+
 def build_tree(xgtree, base_xml_element, var_indices):
     parent_element_dict = {'0':base_xml_element}
     pos_dict = {'0':'s'}
@@ -13,7 +17,7 @@ def build_tree(xgtree, base_xml_element, var_indices):
             #leaf node
             result = re.match(r'(\t*)(\d+):leaf=({0})$'.format(regex_float_pattern), line)
             if not result:
-                print(line)
+                logger.info(line)
             depth = result.group(1).count('\t')
             inode = result.group(2)
             res = result.group(3)
@@ -23,7 +27,7 @@ def build_tree(xgtree, base_xml_element, var_indices):
             #\t\t3:[var_topcand_mass<138.19] yes=7,no=8,missing=7
             result = re.match(r'(\t*)([0-9]+):\[(?P<var>.+)<(?P<cut>{0})\]\syes=(?P<yes>\d+),no=(?P<no>\d+)'.format(regex_float_pattern),line)
             if not result:
-                print(line)
+                logger.info(line)
             depth = result.group(1).count('\t')
             inode = result.group(2)
             var = result.group('var')
@@ -37,12 +41,12 @@ def build_tree(xgtree, base_xml_element, var_indices):
                                              cType="1", res="0.0e+00", rms="0.0e+00", purity="0.0e+00", nType="0")
             parent_element_dict[lnode] = node_elementTree
             parent_element_dict[rnode] = node_elementTree
-            
+
 def convert_model(model, input_variables, output_xml):
     NTrees = len(model)
     var_list = input_variables
     var_indices = {}
-    
+
     # <MethodSetup>
     MethodSetup = ET.Element("MethodSetup", Method="BDT::BDT")
 
@@ -52,8 +56,8 @@ def convert_model(model, input_variables, output_xml):
         name = val[0]
         var_type = val[1]
         var_indices[name] = ind
-        Variable = ET.SubElement(Variables, "Variable", VarIndex=str(ind), Type=val[1], 
-            Expression=name, Label=name, Title=name, Unit="", Internal=name, 
+        Variable = ET.SubElement(Variables, "Variable", VarIndex=str(ind), Type=val[1],
+            Expression=name, Label=name, Title=name, Unit="", Internal=name,
             Min="0.0e+00", Max="0.0e+00")
 
     # <GeneralInfo>
@@ -65,18 +69,18 @@ def convert_model(model, input_variables, output_xml):
     Options = ET.SubElement(MethodSetup, "Options")
     Option_NodePurityLimit = ET.SubElement(Options, "Option", name="NodePurityLimit", modified="No").text = "5.00e-01"
     Option_BoostType = ET.SubElement(Options, "Option", name="BoostType", modified="Yes").text = "Grad"
-    
+
     # <Weights>
     Weights = ET.SubElement(MethodSetup, "Weights", NTrees=str(NTrees), AnalysisType="1")
-    
+
     for itree in range(NTrees):
         BinaryTree = ET.SubElement(Weights, "BinaryTree", type="DecisionTree", boostWeight="1.0e+00", itree=str(itree))
         build_tree(model[itree], BinaryTree, var_indices)
-        
+
     tree = ET.ElementTree(MethodSetup)
     tree.write(output_xml)
     # format it with 'xmllint --format'
-    
+
 # example
 # bst = xgb.train( param, d_train, num_round, watchlist );
 # model = bst.get_dump()

diff --git a/training/make_dataframes.py → legacy/training/make_dataframes.py b/training/make_dataframes.py → legacy/training/make_dataframes.py
diff --git a/training/split_dataframes.py → legacy/training/split_dataframes.py b/training/split_dataframes.py → legacy/training/split_dataframes.py
diff --git a/training/train_final_Reg_Iso.py → legacy/training/train_final_Reg_Iso.py b/training/train_final_Reg_Iso.py → legacy/training/train_final_Reg_Iso.py
diff --git a/training/train_final_Reg_SS.py → legacy/training/train_final_Reg_SS.py b/training/train_final_Reg_SS.py → legacy/training/train_final_Reg_SS.py
diff --git a/training/train_qRC_I_MC.py → legacy/training/train_qRC_I_MC.py b/training/train_qRC_I_MC.py → legacy/training/train_qRC_I_MC.py
diff --git a/training/train_qRC_I_data.py → legacy/training/train_qRC_I_data.py b/training/train_qRC_I_data.py → legacy/training/train_qRC_I_data.py
diff --git a/training/train_qRC_MC.py → legacy/training/train_qRC_MC.py b/training/train_qRC_MC.py → legacy/training/train_qRC_MC.py
diff --git a/training/train_qRC_data.py → legacy/training/train_qRC_data.py b/training/train_qRC_data.py → legacy/training/train_qRC_data.py
diff --git a/python/__init__.py b/python/__init__.py
diff --git a/python/Corrector.py → quantile_regression_chain/Corrector.py b/python/Corrector.py → quantile_regression_chain/Corrector.py
@@ -1,27 +1,27 @@
 import numpy as np
 
 class Corrector:
-   
+
    # store regressors
    def __init__(self,mcclf,dataclf,X,Y,diz=False):
       self.diz=diz #Flag for distribution with discrete 0, i.e. Isolation
       self.mcqtls   = np.array([clf.predict(X) for clf in mcclf])
       self.dataqtls = np.array([clf.predict(X) for clf in dataclf])
 
       self.Y = Y
-      
+
    # correction is actually done here
    def correctEvent(self,iev):
 
       mcqtls = self.mcqtls[:,iev]
       dataqtls = self.dataqtls[:,iev]
       Y = self.Y[iev]
-      
+
       if self.diz and Y == 0.:
          return 0.
 
       qmc =0
-      
+
       while qmc < len(mcqtls): # while + if, to avoid bumping the range
          if mcqtls[qmc] < Y:
             qmc+=1
@@ -38,11 +38,11 @@ def correctEvent(self,iev):
          qmc_low,qdata_low   = mcqtls[qmc-1],dataqtls[qmc-1]
          qmc_high,qdata_high = mcqtls[len(mcqtls)-1]*1.2,dataqtls[len(dataqtls)-1]*1.2
          # to set the value for the highest quantile 20% higher
-                                                                       
+
       return (qdata_high-qdata_low)/(qmc_high-qmc_low) * (Y - qmc_low) + qdata_low
 
    def __call__(self):
-      return np.array([ self.correctEvent(iev) for iev in xrange(self.Y.size) ]).ravel()
+      return np.array([ self.correctEvent(iev) for iev in range(self.Y.size) ]).ravel()
 
 def applyCorrection(mcclf,dataclf,X,Y,diz=False):
    return Corrector(mcclf,dataclf,X,Y,diz)()