Add CATE model selection via surrogate metrics (R-score, DR-loss)

[aman-coder03]

Is your feature request related to a problem? Please describe.
when using causalml in practice, i will often train S-learner, T-learner, X-learner, and DR-learner on the same data and have no principled way to pick between them. Standard cross validation does not work here because you never observe both potential outcomes for the same unit, so there's no ground truth to score against.

R-loss that powers the R-learner's training loop (rlearner.py lines 152, 495, 636) is exactly the right idea, but it's only used internally and never exposed as a standalone scoring function. EconML users already have this via econml.score.RScorer . Causalml users currently don't have an equivalent

Describe the solution you'd like
add a CATEModelSelector to causalml.metrics that scores and compares fitted meta-learners using observed-data surrogate metrics, without needing access to true counterfactual outcomes...

metrics worth shipping in a first pass,

1. R-score (Nie and Wager, 2021): fits residual models for outcome and treatment, then measures how much CATE variance the model explains. Same R-loss already inside BaseRLearner.fit() , just exposed as a standalone function
2. DR pseudo-outcome loss (Kennedy 2023): constructs doubly-robust pseudo-outcomes and scores the CATE estimator against them. Mahajan et al. (2024) showed DR-based metrics dominate across 78 datasets
3. Plug-in T-score: uses a T-learner on held-out data as a CATE proxy. Mahajan et al. found this is never dominated across datasets despite being a simple baseline

rough API sketch

from causalml.metrics import CATEModelSelector

selector = CATEModelSelector(
    models={'s': s_learner, 't': t_learner, 'x': x_learner, 'dr': dr_learner},
    metric='dr_loss',  # or 'r_score', 'plug_in_t'
    cv=5,
)
selector.fit(X, treatment, y)

selector.scores_       # {'s': 0.31, 't': 0.47, 'x': 0.61, 'dr': 0.58}
selector.best_model()  # 'x'
selector.plot_scores()

no new heavy dependencies needed. Nuisance models reuse causalml's existing propensity module and sklearn

Describe alternatives you've considered
using EconML's RScorer directly works but requires fitting nuisance components twice across two libraries with no integration between them.
picking by held-out outcome MSE is what most people default to, but outcome MSE doesn't measure CATE accuracy. A model can have low outcome MSE while getting the treatment effect completely wrong.

Additional context
happy to start with just the R-score as a standalone function in causalml.metrics and build the selector wrapper from there, if that's a better way to scope an initial PR.

[jeongyoonlee]

Thanks for the detailed writeup — real gap, I'd like to take it. Two framing points before a PR:

1. Skip the CATEModelSelector class; add standalone score functions instead. Our evaluation API is already standalone functions over a predictions DataFrame (rate_score, auuc_score, qini_score), and they compose fine for model selection ("score each model, pick the best" is a one-liner). A stateful selector would introduce a second API shape and isn't needed — each score is more useful standalone (you can score a single model too). For the nuisance reuse a selector would have given you, follow the rate_score() precedent: expose a helper that builds DR/AIPW pseudo-outcomes once, then have the score functions consume them (rate_score already asks callers to pass AIPW pseudo-outcomes in as treatment_effect_col, so factor that out and share it).

2. What to add, and priority. We already cover the targeting/ranking axis via rate_score() (RATE — rank-invariant). What's missing is the CATE-accuracy axis. Your own cited benchmark (Mahajan et al. 2024) has DR-loss dominating and plug-in-T never dominated, with R-score not the standout — so lead with those:

• First PR — dr_score() (Kennedy 2023 pseudo-outcomes) + the shared pseudo-outcome helper, and plug_in_t_score(). Reuse rate.py's half-sample-bootstrap CI/p-value pattern.
• Second PR — an R-score function for EconML RScorer parity (factor the R-loss out of BaseRLearner.fit(), nuisances via the propensity module). Please avoid the name r_score — too close to rate_score; something like r_loss/rlearner_score reads more clearly.

Document that RATE = targeting quality while these new scores = effect-magnitude accuracy, so users know which to reach for. Be explicit about which carry inference; don't double-fit nuisances; include tests + a Methodology entry.