Update `plot_precision_recall_curve_with_thresholds_annotations` to Match sklearn's `plot_chance_level` Behavior

[idanmoradarthas]

Summary

Update the plot_precision_recall_curve_with_thresholds_annotations function to align with scikit-learn's PrecisionRecallDisplay.plot_chance_level parameter, replacing the current add_random_classifier_line parameter with more sophisticated behavior that plots a horizontal line representing the prevalence of the positive class.

Background

Current Implementation

Currently, our function has an add_random_classifier_line parameter that, when set to True, plots a simple diagonal line from (1, 0) to (0, 1) representing a "random classifier". This is incorrect for Precision-Recall curves.

Current code (lines 452-455 in metrics.py):

if add_random_classifier_line:  # Add dashed line for random classifier
    fig.add_trace(
        go.Scatter(
            x=[1, 0], y=[0, 1], mode="lines", line=dict(dash="dash", color="black"), name="Random Classifier"
        )
    )

sklearn's Implementation

In scikit-learn's PrecisionRecallDisplay (introduced in version 1.3), the plot_chance_level parameter works differently:

1. It plots a horizontal line, not a diagonal line
2. The y-value is the prevalence of the positive class (proportion of positive samples): prevalence = count(positive_class) / total_samples
3. It requires knowledge of the actual labels (y_true) to calculate prevalence
4. The prevalence is stored as an attribute prevalence_pos_label in the display object
5. Styling can be customized via a chance_level_kw parameter

Key insight from sklearn documentation:

"The chance level is the prevalence of the positive label computed from the data passed during from_estimator or from_predictions call."

Example from sklearn:

from collections import Counter

prevalence_pos_label = Counter(y_test.ravel())[1] / y_test.size

display = PrecisionRecallDisplay(
    recall=recall,
    precision=precision,
    average_precision=average_precision,
    prevalence_pos_label=prevalence_pos_label,
)
display.plot(plot_chance_level=True, despine=True)

The horizontal line at y = prevalence represents the performance of a classifier that always predicts the positive class, which is the actual baseline/chance level for Precision-Recall curves.

Motivation

1. Correctness: A diagonal line from (1,0) to (0,1) is incorrect for Precision-Recall curves. The chance level is a horizontal line at the prevalence value.
2. Alignment with sklearn: Our users familiar with sklearn will expect this behavior.
3. Better interpretation: The prevalence line helps users understand the baseline performance and whether their model beats random guessing.

Proposed Changes

1. Parameter Renaming and Behavior Change

In plot_precision_recall_curve_with_thresholds_annotations function:

Old signature:

def plot_precision_recall_curve_with_thresholds_annotations(
    y_true: np.ndarray,
    classifiers_names_and_scores_dict: Dict[str, np.ndarray],
    *,
    positive_label: Optional[Union[int, float, bool, str]] = None,
    sample_weight: Optional[np.ndarray] = None,
    drop_intermediate: bool = True,
    fig: Optional[go.Figure] = None,
    mode: Optional[str] = "lines+markers",
    add_random_classifier_line: bool = False,  # <-- CHANGE THIS
    show_legend: bool = True,
    **kwargs,
) -> go.Figure:

New signature:

def plot_precision_recall_curve_with_thresholds_annotations(
    y_true: np.ndarray,
    classifiers_names_and_scores_dict: Dict[str, np.ndarray],
    *,
    positive_label: Optional[Union[int, float, bool, str]] = None,
    sample_weight: Optional[np.ndarray] = None,
    drop_intermediate: bool = True,
    fig: Optional[go.Figure] = None,
    mode: Optional[str] = "lines+markers",
    plot_chance_level: bool = False,  # <-- NEW PARAMETER NAME
    chance_level_kw: Optional[Dict] = None,  # <-- NEW PARAMETER FOR STYLING
    show_legend: bool = True,
    **kwargs,
) -> go.Figure:

2. Implementation Details

Replace the current random classifier line logic with:

def plot_precision_recall_curve_with_thresholds_annotations(
    y_true: np.ndarray,
    classifiers_names_and_scores_dict: Dict[str, np.ndarray],
    *,
    positive_label: Optional[Union[int, float, bool, str]] = None,
    sample_weight: Optional[np.ndarray] = None,
    drop_intermediate: bool = True,
    fig: Optional[go.Figure] = None,
    mode: Optional[str] = "lines+markers",
    plot_chance_level: bool = False,
    chance_level_kw: Optional[Dict] = None,
    show_legend: bool = True,
    **kwargs,
) -> go.Figure:
    """Plot Precision-Recall curves with threshold annotations for multiple classifiers.

    :param y_true: array-like of shape (n_samples,). True binary labels.
    :param classifiers_names_and_scores_dict: mapping from classifier name to classifier's score.
    :param positive_label: int, float, bool or str, default=None. The label of the positive class.
    :param sample_weight: array-like of shape (n_samples,), default=None. Sample weights.
    :param drop_intermediate: bool, default=True. Whether to drop some suboptimal thresholds that would not appear on
                              a plotted Precision-Recall curve.
    :param fig: plotly's Figure object, optional. The figure to plot on.
    :param mode: str, default='lines+markers'. Determines the drawing mode for this scatter trace.
    :param plot_chance_level: bool, default=False. Whether to plot the chance level line. The chance level is a
                              horizontal line at the prevalence of the positive class, representing the precision
                              of a classifier that always predicts the positive class.
    :param chance_level_kw: dict, default=None. Keyword arguments to be passed to plotly's Scatter for rendering
                            the chance level line (e.g., line color, style).
    :param show_legend: bool, default=True. Whether to display legend in the plot.
    :param kwargs: additional keyword arguments to be passed to the plot function.
    :return: The Figure object with the plot drawn onto it.
    :raises ValueError: If the input data is invalid or inconsistent.
    """
    if fig is None:
        fig = go.Figure()

    for classifier_name, y_scores in classifiers_names_and_scores_dict.items():
        if y_true.shape != y_scores.shape:
            raise ValueError(
                f"Shape mismatch: y_true {y_true.shape} and y_scores {y_scores.shape} for classifier {classifier_name}"
            )

        try:
            precision_array, recall_array, thresholds = precision_recall_curve(
                y_true,
                y_scores,
                pos_label=positive_label,
                sample_weight=sample_weight,
                drop_intermediate=drop_intermediate,
            )
        except ValueError as e:
            raise ValueError(f"Error calculating Precision-Recall curve for classifier {classifier_name}: {str(e)}")

        fig.add_trace(
            go.Scatter(
                x=recall_array,
                y=precision_array,
                mode=mode,
                text=[
                    f"Prob: {threshold:.2f}<br>Precision: {precision:.2f}<br>Recall: {recall:.2f}"
                    for precision, recall, threshold in zip(precision_array, recall_array, thresholds)
                ],
                name=classifier_name,
                **kwargs,
            )
        )

    if plot_chance_level:
        # Calculate prevalence of positive class
        if positive_label is None:
            # Determine positive label (same logic as sklearn)
            unique_labels = np.unique(y_true)
            if len(unique_labels) != 2:
                raise ValueError("y_true must be binary for plotting chance level")
            # Use the convention that the larger label is positive
            positive_label = unique_labels[1]
        
        if sample_weight is not None:
            prevalence = np.sum(sample_weight[y_true == positive_label]) / np.sum(sample_weight)
        else:
            prevalence = np.sum(y_true == positive_label) / len(y_true)
        
        # Default styling for chance level line
        default_chance_level_kw = {
            "line": dict(dash="dash", color="gray"),
            "name": "Chance level (Prevalence)"
        }
        
        if chance_level_kw is not None:
            default_chance_level_kw.update(chance_level_kw)
        
        # Plot horizontal line at prevalence
        fig.add_trace(
            go.Scatter(
                x=[0, 1],
                y=[prevalence, prevalence],
                mode="lines",
                **default_chance_level_kw,
            )
        )

    fig.update_layout(xaxis_title="Recall", yaxis_title="Precision", showlegend=show_legend)

    return fig

3. Documentation Updates

Update the docstring to:

• Explain that plot_chance_level plots a horizontal line at the prevalence
• Clarify that this represents the precision of an "always positive" classifier
• Document the chance_level_kw parameter for customization
• Note the difference from ROC curves where the chance level is diagonal

4. Backward Compatibility

Breaking change: The parameter name change from add_random_classifier_line to plot_chance_level is a breaking change.

Options:

1. Deprecation period (RECOMMENDED): Keep add_random_classifier_line as a deprecated parameter for 1-2 releases with a warning
2. Direct breaking change: Document in release notes and CHANGELOG

Recommended approach:

def plot_precision_recall_curve_with_thresholds_annotations(
    y_true: np.ndarray,
    classifiers_names_and_scores_dict: Dict[str, np.ndarray],
    *,
    positive_label: Optional[Union[int, float, bool, str]] = None,
    sample_weight: Optional[np.ndarray] = None,
    drop_intermediate: bool = True,
    fig: Optional[go.Figure] = None,
    mode: Optional[str] = "lines+markers",
    add_random_classifier_line: Optional[bool] = None,  # Deprecated
    plot_chance_level: bool = False,
    chance_level_kw: Optional[Dict] = None,
    show_legend: bool = True,
    **kwargs,
) -> go.Figure:
    """..."""
    
    # Handle deprecated parameter
    if add_random_classifier_line is not None:
        warnings.warn(
            "Parameter 'add_random_classifier_line' is deprecated and will be removed in version X.Y.Z. "
            "Use 'plot_chance_level=True' instead. Note: the behavior has changed - the chance level "
            "is now a horizontal line at the prevalence, not a diagonal line.",
            DeprecationWarning,
            stacklevel=2
        )
        # Don't automatically convert old behavior to new, force users to update
        if add_random_classifier_line and not plot_chance_level:
            raise ValueError(
                "The 'add_random_classifier_line' parameter is deprecated. "
                "Please use 'plot_chance_level=True' instead."
            )
    
    # ... rest of implementation

Test Updates

Tests to Update

1. Update existing test test_plot_precision_recall_curve_with_thresholds_annotations:

   • Change add_random_classifier_line to plot_chance_level
   • Update baseline images to show horizontal line instead of diagonal
   • Verify the line is at the correct prevalence value

2. Add new test for prevalence calculation:

def test_plot_precision_recall_curve_chance_level_prevalence(plotly_models_dict):
    """Test that chance level line is plotted at correct prevalence."""
    y_true = np.array(plotly_models_dict["y_true"])
    classifiers_names_and_scores_dict = {
        name: np.array(data["y_scores"]) 
        for name, data in plotly_models_dict.items() 
        if name != "y_true"
    }
    
    # Calculate expected prevalence
    prevalence = np.sum(y_true == 1) / len(y_true)
    
    # Mock the precision_recall_curve to avoid need for full data
    def _mock_precision_recall_curve(y_true, y_score, **kwargs):
        for classifier, scores in classifiers_names_and_scores_dict.items():
            if np.array_equal(scores, y_score):
                data = plotly_models_dict[classifier]["precision_recall_curve"]
                return np.array(data["precision_array"]), np.array(data["recall_array"]), np.array(data["thresholds"])
    
    mocker.patch("ds_utils.metrics.precision_recall_curve", side_effect=_mock_precision_recall_curve)
    
    fig = plot_precision_recall_curve_with_thresholds_annotations(
        y_true, 
        classifiers_names_and_scores_dict, 
        plot_chance_level=True
    )
    
    # Find the chance level trace
    chance_level_trace = None
    for trace in fig.data:
        if "Chance level" in trace.name or "Prevalence" in trace.name:
            chance_level_trace = trace
            break
    
    assert chance_level_trace is not None, "Chance level line not found in plot"
    
    # Verify it's a horizontal line at prevalence
    assert np.allclose(chance_level_trace.x, [0, 1])
    assert np.allclose(chance_level_trace.y, [prevalence, prevalence])

3. Add test for chance_level_kw customization:

def test_plot_precision_recall_curve_chance_level_styling(plotly_models_dict):
    """Test that chance_level_kw parameter works for styling."""
    y_true = np.array(plotly_models_dict["y_true"])
    classifiers_names_and_scores_dict = {
        name: np.array(data["y_scores"]) 
        for name, data in plotly_models_dict.items() 
        if name != "y_true"
    }
    
    custom_style = {
        "line": dict(dash="dot", color="red", width=3),
        "name": "Custom Chance Level Name"
    }
    
    # ... mock precision_recall_curve ...
    
    fig = plot_precision_recall_curve_with_thresholds_annotations(
        y_true, 
        classifiers_names_and_scores_dict, 
        plot_chance_level=True,
        chance_level_kw=custom_style
    )
    
    # Find the chance level trace and verify styling
    chance_level_trace = None
    for trace in fig.data:
        if "Custom Chance Level Name" in trace.name:
            chance_level_trace = trace
            break
    
    assert chance_level_trace is not None
    assert chance_level_trace.line.dash == "dot"
    assert chance_level_trace.line.color == "red"

4. Add test for sample weights:

def test_plot_precision_recall_curve_chance_level_with_sample_weights(plotly_models_dict):
    """Test that prevalence is correctly calculated with sample weights."""
    y_true = np.array(plotly_models_dict["y_true"])
    sample_weight = np.random.random(len(y_true))
    
    # ... rest of test

5. Update parametrized test test_plot_precision_recall_curve_with_thresholds_annotations:

@pytest.mark.parametrize("plot_chance_level", [False, True], ids=["default", "with_chance_level"])
def test_plot_precision_recall_curve_with_thresholds_annotations(
    mocker, request, plot_chance_level, plotly_models_dict
):
    # ... existing test code but replace add_random_classifier_line with plot_chance_level

Test Files to Update

• tests/test_metrics.py - Update the main test cases
• Baseline images in tests/baseline_images/test_metrics/ - Regenerate with new horizontal lines

Migration Guide for Users

Include in release notes:

### Breaking Changes in `plot_precision_recall_curve_with_thresholds_annotations`

**Parameter renamed:** `add_random_classifier_line` → `plot_chance_level`

**Behavior changed:** The chance level line is now correctly plotted as a **horizontal line** at the prevalence of the positive class, not a diagonal line.

**Migration:**

Before:
```python
fig = plot_precision_recall_curve_with_thresholds_annotations(
    y_true, 
    scores_dict,
    add_random_classifier_line=True
)

After:

fig = plot_precision_recall_curve_with_thresholds_annotations(
    y_true, 
    scores_dict,
    plot_chance_level=True,
    chance_level_kw={"line": dict(dash="dash", color="gray")}  # optional styling
)

Why this change?
The previous implementation incorrectly plotted a diagonal line for the "random classifier". In Precision-Recall curves, the chance level is a horizontal line at the prevalence (proportion of positive samples), representing a classifier that always predicts the positive class.

Implementation Checklist

• Update function signature
• Implement prevalence calculation logic
• Add chance level line plotting with horizontal line
• Add chance_level_kw parameter for styling
• Update docstring with detailed explanation
• Add deprecation warning for old parameter (optional, if going with deprecation approach)
• Update existing tests
• Add new tests for prevalence calculation
• Add new tests for chance_level_kw styling
• Add tests for sample_weight support
• Regenerate baseline images
• Update CHANGELOG.md
• Update documentation/user guide if exists
• Add migration guide to release notes

References

• sklearn PrecisionRecallDisplay Documentation
• sklearn PR #26019 - Implementation of plot_chance_level
• sklearn Precision-Recall Example

Notes

• The chance level represents a classifier that always predicts the positive class, which achieves precision equal to the prevalence
• This is different from ROC curves where the chance level is a diagonal line (representing random guessing)
• The prevalence should be calculated respecting sample_weight if provided
• The positive_label parameter should be used to determine which class is positive for prevalence calculation