Question on Penalized Scoring Rules

[shaochenze]

Hi @Ruhallah93, thank you for sharing this interesting work. I’d like to discuss a conceptual point regarding the Penalized Brier Score (PBS) and Penalized Log Loss (PLL).

In my understanding, scoring rules—by definition—evaluate probabilistic forecasts based solely on the predictive distribution P and the observed outcome i. However, PBS and PLL appear to deviate from this framework because their penalty terms explicitly incorporate knowledge of the ground-truth distribution Q. This seems inconsistent with the standard definition of scoring rules, which should not depend on Q directly.

Furthermore, the deterministic classification setting discussed in the paper (where Q is a degenerate one-hot distribution) diverges from the core use case of scoring rules. Scoring rules are designed for ​probabilistic forecasting​ scenarios (e.g., weather prediction, economic forecasting) where we aim to reward honest uncertainty quantification. In such contexts, the ground-truth Q is never directly observable - we only ever see individual realizations i~Q.

I’d appreciate your perspective on this. Please correct me if I’ve misunderstood any part of your approach.

Best regards,
Chenze

[Ruhallah93]

Thank you for this thoughtful question, @shaochenze. You've identified a crucial nuance that deserves careful clarification. Let me address this in two parts:

---

1. Relationship to Proper Scoring Rules

You're correct that proper scoring rules should only depend on the predictive distribution P and the observed outcome i. This is a fundamental requirement we strictly maintain. Let me clarify the apparent paradox:

• Key Insight: In multi-class classification with one-hot encoded labels, the "ground truth distribution" Q is degenerate (Dirac delta at class i). This allows us to:

  1. Maintain dependence only on P and i (via Q = δ_i)
  2. Compute the penalty using only observable information (argmax P vs argmax Q = i)

• Mathematical Formulation:
  The penalty term:

  $$\mathbb{1}_{\{\arg\max P \neq \arg\max Q\}} = \mathbb{1}_{\{\arg\max P \neq i\}}$$

  depends only on the observed outcome i, not on any hidden properties of Q. This keeps PBS/PLL within the strict definition of proper scoring rules.

---

2. Application Context

Your observation about probabilistic forecasting vs classification is astute. PBS/PLL are specifically designed for single-label multi-class classification, where:

• The true distribution Q is always observable (through one-hot labels)
• Decision-making often depends on predicted class (argmax P)

Why This Matters:
In high-stakes classification tasks (medical diagnosis, autonomous systems), we want metrics that:

1. Reward calibration (properness)
2. Simultaneously ensure correct class prediction (superiority)

Traditional proper scores achieve (1) but fail (2). PBS/PLL achieves both without requiring additional information beyond (P, i).

---

3. Extension to Probabilistic Forecasting

For general probabilistic forecasting where Q is non-degenerate, we agree PBS/PLL wouldn't apply. However, in classification:

• Observed i ≡ Full Knowledge of Q
  (Since Q = δ_i in supervised learning)

• Practical Alignment
  Most ML practitioners evaluate classifiers using both:

  • Calibration (via proper scores)
  • Accuracy (via argmax decisions)
    PBS/PLL combine these requirements into a single metric.

---

4. Properness Proof Sketch

For any true distribution Q = δ_i and predictive distribution P:

1. Expected Score:

   $$S(P,Q) = \mathbb{E}_{i \sim Q}[s(P,i)]$$

2. PBS/PLL Remain Proper:
   The penalty term:

   $$\mathbb{1}_{\{\arg\max P \neq i\}} \cdot C$$

   Creates a discontinuity at P = Q, but:

   • The unpenalized base score (Brier/LogLoss) is minimized at P=Q
   • The penalty increases the score for P ≠ Q
     Therefore, Q remains the unique minimizer.

---

5. Empirical Validation

Our experiments show PBS/PLL:

• Maintain ranking equivalence with Brier/LogLoss on calibrated models
• Penalize uncalibrated models that make incorrect class predictions
• Preserve properness (as measured by reliability diagrams)

---

This formulation allows practitioners to simultaneously optimize for:

• Calibration (via properness)
• Correct class prediction (via superiority)

without needing additional information beyond standard supervised learning setups. I appreciate you raising this important conceptual question – it helps clarify the nuanced but intentional design choices in PBS/PLL.

[shaochenze]

Thank you for your explanation! It makes sense if you’re specifically considering the single-label multi-class classification scenario. I now understand your point—by using scoring rules while incorporating appropriate misclassification penalties, we can effectively combine calibration and accuracy into a unified metric.

This is a really interesting approach—thanks again for sharing your work!