Wrap any trained classifier so its predictions come with a guaranteed coverage, a prediction set that contains the true label at least (say) 90% of the time, with a distribution-free, finite-sample proof. No retraining, no calibration assumptions. And a direct demonstration that the thing people reach for instead, the model's softmax "confidence", does not give you that guarantee.
conformal # coverage table: conformal vs naive softmax
conformal --json
Interactive/exportable version: docs/assets/architecture.html.
Split-conformal prediction (LAC) needs one held-out calibration set the model never trained
on. The nonconformity score of the true label is s = 1 − p[true]. Take the finite-sample
(1−α) quantile q̂ of those scores; for a new input, the prediction set is every class with
p_y ≥ 1 − q̂. That set satisfies, for any model and any data distribution:
P(true label ∈ set) ≥ 1 − α
The naive alternative, "predict the top class and trust its softmax probability", has no such guarantee. When the model is over-confident (here a Random Forest, 81% accurate on a noisy 6-class problem), its confidence is a story, not a coverage rate.
conformal, 1500 test points, RandomForest @ 81.2% accuracy:
| target coverage | conformal coverage | conformal set size | naive coverage | naive set size |
|---|---|---|---|---|
| 80% | 81.9% ✓ | 1.05 | 81.2% | 1.00 |
| 90% | 90.7% ✓ | 1.58 | 81.2% (−8.8) | 1.00 |
| 95% | 94.9% ✓ | 3.21 | 81.2% (−13.8) | 1.00 |
Two things to read off this table:
- Conformal hits every target; naive softmax plateaus at the model's accuracy. "Predict the argmax and trust the confidence" delivers ~81% coverage no matter what target you ask for, it literally cannot give you 90% or 95%, because it never grows the set. Ask for 95% coverage and you silently get 81%. That 14-point gap is the model's miscalibration, made concrete.
- Conformal buys coverage with set size, honestly. To guarantee 80% it returns ~1 class; to guarantee 95% it returns ~3.2, the set grows on the hard inputs where the model is genuinely uncertain. You get the coverage you asked for and an honest signal of where the model is unsure, instead of a confident single guess that's wrong 1 time in 5.
The only difference between the two columns is the threshold: conformal uses the calibrated
q̂, naive uses the model's assumed confidence. Calibration is the whole story.
Prediction sets with a real coverage guarantee are how you ship a classifier into a
decision-critical loop (triage, moderation, anything with a human reviewer) and make an honest
promise about its error rate, without assuming the softmax is calibrated, which it almost never
is. The method is model-agnostic: swap the Random Forest for any predict_proba estimator and the
guarantee holds.
pip install -e ".[dev]"
pytest -q # 5 passed — incl. the marginal-coverage guarantee on held-out dataThe key test asserts the guarantee directly: conformal coverage ≥ 1 − α on the test set for α ∈ {0.20, 0.10, 0.05}.
NumPy + scikit-learn. Split-conformal (LAC) with the finite-sample quantile correction, from the formula; Random Forest as a deliberately over-confident base model; coverage/set-size scoring on a held-out split.
MIT