Enhance Regression Metrics Module with REC Curves and AUC Calculation

README.md

@@ -27,6 +27,7 @@ The API of the package is built to work with the Scikit-Learn API and Matplotlib
 The metrics module is organized into focused submodules:
 - **confusion_matrix** - Confusion matrix visualization and analysis
 - **curves** - ROC and Precision-Recall curves
+- **regression** - Regression Error Characteristic (REC) curves and Regression AUC
 - **time_series** - Time-series and forecasting directional metrics
 - **learning_curves** - Learning curve visualization
 - **probability_analysis** - Probability calibration and accuracy analysis
@@ -159,6 +160,38 @@ Directional Accuracy: 100.00%
 Directional Bias: 1.00
 ```
 
+### Regression Error Characteristic (REC) Curve
+
+Plot REC curves with AUC (Area Over the Curve) annotations for multiple regression models. REC curves plot the error tolerance on the x-axis against the accuracy (proportion of predictions within that tolerance) on the y-axis. The AOC provides a normalized metric in [0, 1] where 0 is perfect prediction.
+
+```python
+from ds_utils.metrics.regression import plot_rec_curve_with_annotations, regression_auc_score
+import numpy as np
+
+# Generate dummy data
+y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0])
+predictions = {
+    "Good Model": np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]),
+    "Bad Model": np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0]),
+}
+
+# Plot REC curves
+fig = plot_rec_curve_with_annotations(y_true, predictions)
+fig.show()
+
+# Get standalone AOC score
+good_aoc = regression_auc_score(y_true, predictions["Good Model"])
+print(f"Good Model AOC: {good_aoc:.4f}")
+```
+
+Output:
+```
+Good Model AOC: 0.1000
+```
+
+![plot rec curve with annotations](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png)
+
+
 ### Plot Error Analysis Chart
 
 This method automates the creation of an error analysis chart by computing the error type (correct, false_positive, false_negative) for each

docs/source/metrics/index.rst

@@ -14,3 +14,4 @@ The module of metrics contains methods that help to calculate and/or visualize e
    error_analysis
    learning_curves
    probability_analysis
+   regression

docs/source/metrics/regression.rst

@@ -0,0 +1,84 @@
+Regression Metrics
+==================
+
+Regression Error Characteristic (REC) curves provide a powerful framework for evaluating and comparing regression models, generalizing the Receiver Operating Characteristic (ROC) curve used in classification.
+
+Background and Theory
+-------------------
+
+The REC curve was introduced by **Bi & Bennett (2003)** in *"Regression error characteristic curves"* (Proceedings of the Twentieth International Conference on Machine Learning, pp. 43-50) as a method to visualize the cumulative distribution of errors in regression modeling. 
+
+Unlike classification where predictions are binary, regression predictions fall along a continuous scale. The REC curve handles this by plotting **Error Tolerance** on the x-axis against **Accuracy** on the y-axis. Here, "accuracy" is defined as the proportion of predictions that fall within the specified error tolerance.
+
+This framework was further extended into the Regression ROC (RROC) space by **Hernández-Orallo (2013)** in *"ROC curves for regression"* (Pattern Recognition, 46(12), 3395-3411, doi:10.1016/j.patcog.2013.06.014). This work demonstrated that the area under/over the curve relates deeply to the variance and expected magnitude of regression errors.
+
+### Interpreting the Area Over the Curve (AOC)
+
+In classification ROC curves, a larger Area Under the Curve (AUC) is better. For REC curves, we look at the **Area Over the Curve (AOC)**:
+
+* **0.0 is Perfect:** A perfect model has zero error for every prediction. Its REC curve jumps to 1.0 (100% accuracy) instantly at an error tolerance of 0, meaning there is zero area *over* the curve.
+* **Lower is Better:** The smaller the AOC, the closer the predictions are to the true values.
+* **Normalized Score:** The AOC is typically normalized by dividing by the maximum absolute error observed, yielding a bounded metric in [0, 1]. This makes it easier to compare performance across different datasets and scales.
+
+Regression AUC Score
+--------------------
+
+.. autofunction:: ds_utils.metrics.regression.regression_auc_score
+
+Code Example
+~~~~~~~~~~~~
+
+.. code-block:: python
+
+    import numpy as np
+    from ds_utils.metrics.regression import regression_auc_score
+
+    # Generate dummy data
+    y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0])
+    y_pred_good = np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0])
+    y_pred_bad = np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0])
+
+    # Calculate AOC scores (normalized)
+    good_aoc = regression_auc_score(y_true, y_pred_good)
+    bad_aoc = regression_auc_score(y_true, y_pred_bad)
+
+    print(f"Good Model AOC: {good_aoc:.4f}")
+    print(f"Bad Model AOC:  {bad_aoc:.4f}")
+
+.. code-block:: text
+
+    Output:
+    Good Model AOC: 0.1000
+    Bad Model AOC:  0.4200
+
+Plot REC Curve with Annotations
+-------------------------------
+
+.. autofunction:: ds_utils.metrics.regression.plot_rec_curve_with_annotations
+
+Code Example
+~~~~~~~~~~~~
+
+We can use the `plot_rec_curve_with_annotations` function to visually compare the two models from the previous example. The Plotly backend allows for interactive exploration of the error tolerances.
+
+.. code-block:: python
+
+    import numpy as np
+    from ds_utils.metrics.regression import plot_rec_curve_with_annotations
+
+    # Generate dummy data
+    y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0])
+    predictions = {
+        "Good Model": np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]),
+        "Bad Model": np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0]),
+    }
+
+    # Plot REC curves
+    fig = plot_rec_curve_with_annotations(y_true, predictions)
+    fig.show()
+
+And the following interactive graph will be shown:
+
+.. image:: ../../../tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png
+    :align: center
+    :alt: Regression Error Characteristic (REC) Curve

ds_utils/__init__.py

@@ -1,3 +1,3 @@
 """Data Science Utilities package."""
 
-__version__ = "1.10.0rc11"
+__version__ = "1.10.0rc12"

ds_utils/metrics/__init__.py

@@ -3,6 +3,7 @@
 This package provides functions for evaluating and visualizing model performance:
 - confusion_matrix - Confusion matrix visualization and classification metrics
 - curves - ROC and Precision-Recall curves with threshold annotations
+- regression - Regression Error Characteristic (REC) curves and Regression AUC
 - error_analysis - Tabular error-analysis report generation
 - learning_curves - Learning curve visualization
 - probability_analysis - Probability calibration and accuracy analysis

ds_utils/metrics/regression.py

@@ -0,0 +1,165 @@
+"""Regression evaluation metrics and visualization.
+
+This module provides the Regression Error Characteristic (REC) curve and
+the associated Area Over the Curve (AOC) metric for evaluating and
+comparing regression models.
+"""
+
+from typing import Dict, Optional, Tuple
+
+import numpy as np
+from plotly import graph_objects as go
+from sklearn.metrics import auc
+
+
+def _calculate_rec_curve(
+    y_true: np.ndarray,
+    y_pred: np.ndarray,
+    sample_weight: Optional[np.ndarray] = None,
+    n_points: int = 1000,
+) -> Tuple[np.ndarray, np.ndarray]:
+    """Calculate REC curve coordinates.
+
+    :param y_true: True target values.
+    :param y_pred: Predicted target values.
+    :param sample_weight: Sample weights.
+    :param n_points: Number of points to calculate on the curve.
+    :return: Tuple of (error_tolerances, accuracies).
+    """
+    errors = np.abs(y_true - y_pred)
+
+    if sample_weight is not None:
+        sample_weight = np.array(sample_weight)
+        sample_weight = sample_weight / sample_weight.sum()
+    else:
+        sample_weight = np.ones(len(errors)) / len(errors)
+
+    max_error = np.max(errors)
+    error_tolerances = np.linspace(0, max_error, n_points)
+
+    accuracies = np.array([sample_weight[errors <= tolerance].sum() for tolerance in error_tolerances])
+
+    return error_tolerances, accuracies
+
+
+def regression_auc_score(
+    y_true: np.ndarray,
+    y_pred: np.ndarray,
+    sample_weight: Optional[np.ndarray] = None,
+    normalize: bool = True,
+) -> float:
+    """Calculate Area Over the REC Curve (AOC) / Regression AUC.
+
+    This is the standalone version of the AUC calculation used in
+    :func:`plot_rec_curve_with_annotations`. Lower values indicate
+    better model performance. The AOC is calculated as the area between
+    the REC curve and the y=1 line.
+
+    When normalized, it is divided by the maximum absolute error to give
+    a score in [0, 1].
+
+    :param y_true: array-like of shape (n_samples,). True target values.
+    :param y_pred: array-like of shape (n_samples,). Predicted target values.
+    :param sample_weight: array-like of shape (n_samples,), default=None. Sample weights.
+    :param normalize: bool, default=True. If True, normalize by maximum absolute error
+                      to get a score in [0, 1] range.
+    :return: The regression AUC score. Lower is better (0 = perfect, 1 = worst possible).
+    :raises ValueError: If shapes of y_true and y_pred do not match.
+    """
+    y_true = np.asarray(y_true)
+    y_pred = np.asarray(y_pred)
+
+    if y_true.shape != y_pred.shape:
+        raise ValueError(f"Shape mismatch: y_true {y_true.shape} and y_pred {y_pred.shape}")
+
+    error_tolerances, accuracies = _calculate_rec_curve(y_true, y_pred, sample_weight)
+
+    max_error = np.max(np.abs(y_true - y_pred))
+
+    # Area under the REC curve using sklearn.metrics.auc
+    auc_under_curve = auc(error_tolerances, accuracies)
+
+    # AOC = total rectangle area (max_error * 1.0) minus area under curve
+    aoc = max_error * 1.0 - auc_under_curve
+
+    if normalize:
+        if max_error > 0:
+            aoc = aoc / max_error
+
+    return float(aoc)
+
+
+def plot_rec_curve_with_annotations(
+    y_true: np.ndarray,
+    regressors_names_and_predictions_dict: Dict[str, np.ndarray],
+    *,
+    sample_weight: Optional[np.ndarray] = None,
+    normalize_auc: bool = True,
+    fig: Optional[go.Figure] = None,
+    mode: Optional[str] = "lines+markers",
+    show_legend: bool = True,
+    **kwargs,
+) -> go.Figure:
+    """Plot Regression Error Characteristic (REC) curves with AUC annotations.
+
+    The REC curve shows the cumulative distribution of absolute errors,
+    allowing comparison of regression model performance. The Area Over
+    the Curve (AOC) is calculated and displayed in the legend for each
+    regressor.
+
+    :param y_true: array-like of shape (n_samples,). True target values.
+    :param regressors_names_and_predictions_dict: mapping from regressor name to predictions.
+    :param sample_weight: array-like of shape (n_samples,), default=None. Sample weights.
+    :param normalize_auc: bool, default=True. If True, normalize AOC by maximum absolute error
+                          to get a score in [0, 1] range.
+    :param fig: plotly's Figure object, optional. The figure to plot on.
+    :param mode: str, default='lines+markers'. Determines the drawing mode for this scatter trace.
+    :param show_legend: bool, default=True. Whether to display legend in the plot.
+    :param kwargs: additional keyword arguments to be passed to the plot function.
+    :return: The Figure object with the plot drawn onto it.
+    :raises ValueError: If the input data is invalid or inconsistent.
+    """
+    if fig is None:
+        fig = go.Figure()
+
+    y_true = np.asarray(y_true)
+
+    for regressor_name, y_pred in regressors_names_and_predictions_dict.items():
+        y_pred = np.asarray(y_pred)
+
+        if y_true.shape != y_pred.shape:
+            raise ValueError(
+                f"Shape mismatch: y_true {y_true.shape} and y_pred {y_pred.shape} for regressor {regressor_name}"
+            )
+
+        try:
+            error_tolerances, accuracies = _calculate_rec_curve(y_true, y_pred, sample_weight)
+        except ValueError as e:
+            raise ValueError(f"Error calculating REC curve for regressor {regressor_name}: {str(e)}")
+
+        try:
+            auc_score = regression_auc_score(y_true, y_pred, sample_weight=sample_weight, normalize=normalize_auc)
+        except ValueError as e:
+            raise ValueError(f"Error calculating AUC score for regressor {regressor_name}: {str(e)}")
+
+        hover_text = [f"Tolerance: {tol:.4f}<br>Accuracy: {acc:.4f}" for tol, acc in zip(error_tolerances, accuracies)]
+
+        fig.add_trace(
+            go.Scatter(
+                x=error_tolerances,
+                y=accuracies,
+                mode=mode,
+                text=hover_text,
+                hoverinfo="text",
+                name=f"{regressor_name} (AOC = {auc_score:.4f})",
+                **kwargs,
+            )
+        )
+
+    fig.update_layout(
+        xaxis_title="Error Tolerance",
+        yaxis_title="Accuracy (Proportion within tolerance)",
+        showlegend=show_legend,
+    )
+
+    return fig