Calibration parameter pkl file generation

[Georg-va]

Hi,
great work! I trained your model on my own dataset and tried to evaluate the newly trained weights on another dataset. However, for calling the evaluation.py with the "--calibrate" parameter, I require a .pkl file containing the calibration parameters.

Since these pkl files are not created automatically I wanted to ask if you can share how you created them in your experiments.

Thanks a lot in advance!
Georg

[luotingzhuang]

Hi Georg,

Thank you for your interest in our work and for pointing out the missing piece in the repo. We will update the GitHub repository later, but you can find the calibration implementation below. You will need to run inference on your validation sets because the calibration model will be fitted on them. val below is a dataframe with fold, probs, and label columns.

import joblib
from betacal import BetaCalibration
n_folds = 5
for i in range(n_folds):
    cal = BetaCalibration()
    cal.fit(val[val.fold==i].probs, val[val.fold==i].label)
    joblib.dump(cal, f"./calibration/cal_{i}.pkl")

Please don't hesitate to reply or reach out if you have any additional questions!
My email is luotingzhuang@g.ucla.edu.

Thanks,
Luoting (Lottie)