Add Regression Error Characteristic (REC) Curve and Regression AUC

[idanmoradarthas]

Summary

Add support for Regression Error Characteristic (REC) curves with Area Over the Curve (AOC) / Regression AUC calculation to the metrics module. This will provide a visual and quantitative way to compare regression model performance, similar to how ROC curves work for classification.

Motivation

Currently, the ds_utils.metrics module provides comprehensive visualization tools for classification models (confusion matrices, ROC curves, precision-recall curves) but lacks equivalent tools for regression model evaluation and comparison. The REC curve fills this gap by:

1. Visual comparison: Enables side-by-side comparison of multiple regression models
2. Error distribution insight: Shows the cumulative distribution of prediction errors
3. Quantitative metric: Provides a single metric (AOC/Regression AUC) for model comparison
4. Sklearn compatibility: Follows familiar sklearn API patterns for easy adoption

Background

The Regression Error Characteristic (REC) curve is the regression analog of the ROC curve. It plots the error tolerance (x-axis) against the accuracy (y-axis), where accuracy is the proportion of predictions within the error tolerance. The Area Over the Curve (AOC) provides a single score for model comparison - lower values indicate better performance.

Reference: Bi, J., & Bennett, K. P. (2003). Regression error characteristic curves. In Twentieth International Conference on Machine Learning (ICML-2003).

Proposed API

1. REC Curve Plotting Function

def plot_rec_curve_with_annotations(
    y_true: np.ndarray,
    regressors_names_and_predictions_dict: Dict[str, np.ndarray],
    *,
    sample_weight: Optional[np.ndarray] = None,
    normalize_auc: bool = True,
    fig: Optional[go.Figure] = None,
    mode: Optional[str] = "lines+markers",
    show_legend: bool = True,
    **kwargs,
) -> go.Figure:
    """Plot Regression Error Characteristic (REC) curves with AUC annotations.
    
    The REC curve shows the cumulative distribution of absolute errors, allowing
    comparison of regression model performance. The Area Over the Curve (AOC) is
    calculated and displayed in the legend for each regressor.
    
    :param y_true: array-like of shape (n_samples,). True target values.
    :param regressors_names_and_predictions_dict: mapping from regressor name to predictions.
    :param sample_weight: array-like of shape (n_samples,), default=None. Sample weights.
    :param normalize_auc: bool, default=True. If True, normalize AOC by maximum possible error
                         to get a score in [0, 1] range.
    :param fig: plotly's Figure object, optional. The figure to plot on.
    :param mode: str, default='lines+markers'. Determines the drawing mode for this scatter trace.
    :param show_legend: bool, default=True. Whether to display legend in the plot.
    :param kwargs: additional keyword arguments to be passed to the plot function.
    :return: The Figure object with the plot drawn onto it.
    :raises ValueError: If the input data is invalid or inconsistent.
    
    Example:
        >>> from sklearn.linear_model import LinearRegression
        >>> from sklearn.tree import DecisionTreeRegressor
        >>> lr = LinearRegression().fit(X_train, y_train)
        >>> dt = DecisionTreeRegressor().fit(X_train, y_train)
        >>> predictions = {
        ...     'Linear Regression': lr.predict(X_test),
        ...     'Decision Tree': dt.predict(X_test)
        ... }
        >>> fig = plot_rec_curve_with_annotations(y_test, predictions)
        >>> fig.show()
    """

2. Standalone Regression AUC Function

def regression_auc_score(
    y_true: np.ndarray,
    y_pred: np.ndarray,
    sample_weight: Optional[np.ndarray] = None,
    normalize: bool = True,
) -> float:
    """Calculate Area Over the REC Curve (AOC) / Regression AUC.
    
    This is the standalone version of the AUC calculation used in 
    plot_rec_curve_with_annotations. Lower values indicate better model performance.
    
    The AOC is calculated as the area between the REC curve and the y-axis. When
    normalized, it's divided by the maximum possible error to give a score in [0, 1].
    
    :param y_true: array-like of shape (n_samples,). True target values.
    :param y_pred: array-like of shape (n_samples,). Predicted target values.
    :param sample_weight: array-like of shape (n_samples,), default=None. Sample weights.
    :param normalize: bool, default=True. If True, normalize by maximum possible error
                     to get a score in [0, 1] range.
    :return: The regression AUC score. Lower is better (0 = perfect, 1 = worst possible).
    
    Example:
        >>> from sklearn.linear_model import LinearRegression
        >>> lr = LinearRegression().fit(X_train, y_train)
        >>> y_pred = lr.predict(X_test)
        >>> auc = regression_auc_score(y_test, y_pred)
        >>> print(f"Regression AUC: {auc:.4f}")
    """

Suggested Implementation

Helper Function: Calculate REC Curve

def _calculate_rec_curve(
    y_true: np.ndarray,
    y_pred: np.ndarray,
    sample_weight: Optional[np.ndarray] = None,
    n_points: int = 1000,
) -> Tuple[np.ndarray, np.ndarray]:
    """Calculate REC curve coordinates.
    
    :param y_true: True target values.
    :param y_pred: Predicted target values.
    :param sample_weight: Sample weights.
    :param n_points: Number of points to calculate on the curve.
    :return: Tuple of (error_tolerances, accuracies).
    """
    # Calculate absolute errors
    errors = np.abs(y_true - y_pred)
    
    # Apply sample weights if provided
    if sample_weight is not None:
        # Normalize weights
        sample_weight = np.array(sample_weight)
        sample_weight = sample_weight / sample_weight.sum()
    else:
        sample_weight = np.ones(len(errors)) / len(errors)
    
    # Create error tolerance range from 0 to max error
    max_error = np.max(errors)
    error_tolerances = np.linspace(0, max_error, n_points)
    
    # Calculate accuracy at each tolerance level
    accuracies = np.array([
        sample_weight[errors <= tolerance].sum()
        for tolerance in error_tolerances
    ])
    
    return error_tolerances, accuracies

Implementation: regression_auc_score

def regression_auc_score(
    y_true: np.ndarray,
    y_pred: np.ndarray,
    sample_weight: Optional[np.ndarray] = None,
    normalize: bool = True,
) -> float:
    """Calculate Area Over the REC Curve (AOC) / Regression AUC."""
    # Input validation
    y_true = np.asarray(y_true)
    y_pred = np.asarray(y_pred)
    
    if y_true.shape != y_pred.shape:
        raise ValueError(
            f"Shape mismatch: y_true {y_true.shape} and y_pred {y_pred.shape}"
        )
    
    # Calculate REC curve
    error_tolerances, accuracies = _calculate_rec_curve(
        y_true, y_pred, sample_weight
    )
    
    # Calculate area over the curve using trapezoidal rule
    # AOC is the area between the curve and y=1 line
    aoc = np.trapz(1 - accuracies, error_tolerances)
    
    # Normalize if requested
    if normalize:
        max_error = np.max(np.abs(y_true - y_pred))
        # Maximum possible AOC is when all predictions are worst possible
        max_aoc = max_error * 1.0  # Area of rectangle
        if max_aoc > 0:
            aoc = aoc / max_aoc
    
    return float(aoc)

Implementation: plot_rec_curve_with_annotations

def plot_rec_curve_with_annotations(
    y_true: np.ndarray,
    regressors_names_and_predictions_dict: Dict[str, np.ndarray],
    *,
    sample_weight: Optional[np.ndarray] = None,
    normalize_auc: bool = True,
    fig: Optional[go.Figure] = None,
    mode: Optional[str] = "lines+markers",
    show_legend: bool = True,
    **kwargs,
) -> go.Figure:
    """Plot Regression Error Characteristic (REC) curves with AUC annotations."""
    if fig is None:
        fig = go.Figure()
    
    y_true = np.asarray(y_true)
    
    for regressor_name, y_pred in regressors_names_and_predictions_dict.items():
        y_pred = np.asarray(y_pred)
        
        # Validate shapes
        if y_true.shape != y_pred.shape:
            raise ValueError(
                f"Shape mismatch: y_true {y_true.shape} and y_pred {y_pred.shape} "
                f"for regressor {regressor_name}"
            )
        
        try:
            # Calculate REC curve
            error_tolerances, accuracies = _calculate_rec_curve(
                y_true, y_pred, sample_weight
            )
        except ValueError as e:
            raise ValueError(
                f"Error calculating REC curve for regressor {regressor_name}: {str(e)}"
            )
        
        try:
            # Calculate AUC
            auc_score = regression_auc_score(
                y_true, y_pred, sample_weight=sample_weight, normalize=normalize_auc
            )
        except ValueError as e:
            raise ValueError(
                f"Error calculating AUC score for regressor {regressor_name}: {str(e)}"
            )
        
        # Create hover text
        hover_text = [
            f"Tolerance: {tol:.4f}<br>Accuracy: {acc:.4f}"
            for tol, acc in zip(error_tolerances, accuracies)
        ]
        
        # Add trace to figure
        fig.add_trace(
            go.Scatter(
                x=error_tolerances,
                y=accuracies,
                mode=mode,
                text=hover_text,
                name=f"{regressor_name} (AOC = {auc_score:.4f})",
                **kwargs,
            )
        )
    
    # Update layout
    fig.update_layout(
        xaxis_title="Error Tolerance",
        yaxis_title="Accuracy (Proportion within tolerance)",
        showlegend=show_legend,
    )
    
    return fig

Testing Requirements

Unit Tests

def test_regression_auc_score_perfect_predictions():
    """Test that perfect predictions give AUC of 0."""
    y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
    y_pred = y_true.copy()
    auc = regression_auc_score(y_true, y_pred)
    assert auc == pytest.approx(0.0, abs=1e-6)


def test_regression_auc_score_with_errors():
    """Test AUC calculation with known errors."""
    y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
    y_pred = np.array([1.5, 2.5, 3.5, 4.5, 5.5])
    auc = regression_auc_score(y_true, y_pred)
    assert 0 < auc < 1  # Should be between 0 and 1 when normalized


def test_regression_auc_score_shape_mismatch():
    """Test that shape mismatch raises ValueError."""
    y_true = np.array([1.0, 2.0, 3.0])
    y_pred = np.array([1.0, 2.0])
    with pytest.raises(ValueError, match="Shape mismatch"):
        regression_auc_score(y_true, y_pred)


def test_regression_auc_score_with_sample_weights():
    """Test AUC calculation with sample weights."""
    y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
    y_pred = np.array([1.5, 2.5, 3.5, 4.5, 5.5])
    weights = np.array([1.0, 1.0, 1.0, 0.5, 0.5])
    auc_weighted = regression_auc_score(y_true, y_pred, sample_weight=weights)
    auc_unweighted = regression_auc_score(y_true, y_pred)
    assert auc_weighted != auc_unweighted


@pytest.mark.parametrize("normalize_auc", [True, False])
def test_plot_rec_curve_with_annotations(mocker, normalize_auc):
    """Test REC curve plotting with different normalization settings."""
    y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0] * 20)
    predictions = {
        'Model A': y_true + np.random.normal(0, 0.5, len(y_true)),
        'Model B': y_true + np.random.normal(0, 1.0, len(y_true)),
    }
    
    fig = plot_rec_curve_with_annotations(
        y_true, predictions, normalize_auc=normalize_auc
    )
    
    # Verify figure has correct number of traces
    assert len(fig.data) == 2
    
    # Verify legend contains AOC scores
    assert "AOC" in fig.data[0].name
    assert "AOC" in fig.data[1].name
    
    # Verify axis labels
    assert fig.layout.xaxis.title.text == "Error Tolerance"
    assert fig.layout.yaxis.title.text == "Accuracy (Proportion within tolerance)"


def test_plot_rec_curve_existing_figure(mocker):
    """Test plotting REC curve on existing figure."""
    fig = go.Figure()
    fig.update_layout(title="Custom REC Curves")
    
    y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0] * 20)
    predictions = {
        'Model A': y_true + np.random.normal(0, 0.5, len(y_true)),
    }
    
    fig = plot_rec_curve_with_annotations(y_true, predictions, fig=fig)
    
    assert fig.layout.title.text == "Custom REC Curves"
    assert len(fig.data) == 1


def test_plot_rec_curve_shape_mismatch():
    """Test that shape mismatch raises ValueError."""
    y_true = np.array([1.0, 2.0, 3.0])
    predictions = {
        'Model A': np.array([1.0, 2.0]),
    }
    
    with pytest.raises(ValueError, match=r"Shape mismatch.*Model A"):
        plot_rec_curve_with_annotations(y_true, predictions)

Visual Regression Tests

Create baseline images in tests/baseline_images/test_metrics/ for:

• test_plot_rec_curve_basic.png - Basic REC curve with 2 models
• test_plot_rec_curve_normalized.png - REC curve with normalized AUC
• test_plot_rec_curve_with_weights.png - REC curve with sample weights

Documentation Requirements

1. Docstring examples: Add comprehensive examples to both functions
2. Module-level documentation: Update ds_utils/metrics.py module docstring
3. README: Add REC curve example to package README
4. Tutorial notebook (optional): Create Jupyter notebook demonstrating use cases

Implementation Checklist

• Implement _calculate_rec_curve() helper function
• Implement regression_auc_score() function
• Implement plot_rec_curve_with_annotations() function
• Add unit tests for regression_auc_score()
• Add unit tests for plot_rec_curve_with_annotations()
• Add visual regression tests with baseline images
• Update docstrings with examples
• Update module imports in __init__.py
• Update README with new functionality
• Add type hints and ensure mypy compliance
• Run full test suite and ensure all tests pass
• Update CHANGELOG

Additional Notes

• Plotly consistency: Follows the same pattern as plot_roc_curve_with_thresholds_annotations() and plot_precision_recall_curve_with_thresholds_annotations()
• Error handling: Includes proper validation and informative error messages
• Sample weights: Supports sklearn's sample_weight parameter for consistency
• Flexibility: Allows plotting on existing figures for multi-plot dashboards
• Performance: Uses vectorized NumPy operations for efficiency

References

• Bi, J., & Bennett, K. P. (2003). Regression error characteristic curves. Proceedings of the Twentieth International Conference on Machine Learning (ICML-2003), 43-50.
• sklearn metrics API