Add model calibration

[gbeane]

Overview: Probability Calibration via CalibratedClassifierCV

What it is
CalibratedClassifierCV wraps any classifier (RF/GBT/XGBoost, etc.) and learns a mapping from raw model scores to well-calibrated probabilities. It does this with an internal cross-validation loop: in each fold, it fits the base model on train-fold data and learns a calibration function on the fold’s held-out data. Two calibration methods are supported:

• isotonic (non-parametric, flexible; best with enough data)
• sigmoid (Platt scaling; smoother, works better on smaller data)

Why calibration matters
Tree models often output overconfident probabilities (lots of ~0.0 or ~1.0). Calibration fixes this so that predicted probabilities reflect reality (e.g., among samples with p≈0.7, ~70% are positive). Better calibration improves:

• Thresholding
• Loss-based metrics: log-loss / Brier score reflect actual probability quality.
• User trust: fewer “certain-but-wrong” predictions in JABS.

How we use it in JABS

• Added optional settings, which are saved in the project.json file: calibrate_probabilities: bool, calibration_method: "isotonic"|"sigmoid", calibration_cv: int.
• During training (including LOGO cross-validation), probability calibration is fit separately inside each fold. The calibrator is trained only on that fold’s training data and never sees the validation data, which prevents data leakage and keeps validation metrics honest.
• For feature importance, when calibrated we aggregate importances across the calibrated folds’ base estimators
• UI: a JABS Settings dialog lets users toggle calibration and choose the method/CV. The settings are persisted in the JABS project.json file.

Practical guidance

• Reasonable Default: calibrate_probabilities=True, method="isotonic", calibration_cv=3. (however, this PR does not change current behavior, so calibrate_probabilities defaults to False)
• Use "sigmoid" if folds are small; isotonic needs more data.
• Avoid very high calibration_cv—typically 3–5 is enough.
• Always calibrate during validation if you’ll deploy a calibrated final model (metrics should reflect deployed classifier).

Trade-offs

• Extra compute (fits base model multiple times).
• Slight variance increase; mitigated by sensible CV (3–5).
• For extremely imbalanced or tiny folds, prefer "sigmoid" or reduce CV.

Net impact for JABS

• More honest probabilities -> cleaner threshold selection for behaviors / improved search for low-confidence predictions.
• Better UX and trust in “confidence” displays.
• Fewer brittle 0/1 outputs; improved stability across datasets and sessions.

See Also

• https://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html
• https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration.html#sphx-glr-auto-examples-calibration-plot-calibration-py

User Guide

I'm going to hold off on updating the user guide until I'm sure we're going to merge these changes

Settings Dialog Screenshots

[ptuan5]

Changes in the scripts overall look good to me.
Cannot comment on the changes to the ui part because it's beyond my comprehension.

[ptuan5]

you use clean_features here for the calibrated branch, but not for the old, original branch?

[ptuan5]

for brier_score, accepting both 1D and 2D arrays, but here you only accept 2D arrays. Is there a reason for this? Should we enforce 2D altogether?

[ptuan5]

if we need both positive and negative for this, should we either raise an error or return early?

[gbeane]

closed because this has been hanging around as a draft forever. I'm going to reevaluate this and possibly open as a new PR based off the current main branch.