diff --git a/README.md b/README.md index 76ceedd..52f5458 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ The API of the package is built to work with the Scikit-Learn API and Matplotlib The metrics module is organized into focused submodules: - **confusion_matrix** - Confusion matrix visualization and analysis - **curves** - ROC and Precision-Recall curves +- **regression** - Regression Error Characteristic (REC) curves and Regression AUC - **time_series** - Time-series and forecasting directional metrics - **learning_curves** - Learning curve visualization - **probability_analysis** - Probability calibration and accuracy analysis @@ -159,6 +160,38 @@ Directional Accuracy: 100.00% Directional Bias: 1.00 ``` +### Regression Error Characteristic (REC) Curve + +Plot REC curves with AUC (Area Over the Curve) annotations for multiple regression models. REC curves plot the error tolerance on the x-axis against the accuracy (proportion of predictions within that tolerance) on the y-axis. The AOC provides a normalized metric in [0, 1] where 0 is perfect prediction. + +```python +from ds_utils.metrics.regression import plot_rec_curve_with_annotations, regression_auc_score +import numpy as np + +# Generate dummy data +y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]) +predictions = { + "Good Model": np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]), + "Bad Model": np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0]), +} + +# Plot REC curves +fig = plot_rec_curve_with_annotations(y_true, predictions) +fig.show() + +# Get standalone AOC score +good_aoc = regression_auc_score(y_true, predictions["Good Model"]) +print(f"Good Model AOC: {good_aoc:.4f}") +``` + +Output: +``` +Good Model AOC: 0.1000 +``` + +![plot rec curve with annotations](https://raw.githubusercontent.com/idanmoradarthas/DataScienceUtils/master/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png) + + ### Plot Error Analysis Chart This method automates the creation of an error analysis chart by computing the error type (correct, false_positive, false_negative) for each diff --git a/docs/source/metrics/index.rst b/docs/source/metrics/index.rst index db970df..bae2564 100644 --- a/docs/source/metrics/index.rst +++ b/docs/source/metrics/index.rst @@ -14,3 +14,4 @@ The module of metrics contains methods that help to calculate and/or visualize e error_analysis learning_curves probability_analysis + regression diff --git a/docs/source/metrics/regression.rst b/docs/source/metrics/regression.rst new file mode 100644 index 0000000..521e367 --- /dev/null +++ b/docs/source/metrics/regression.rst @@ -0,0 +1,84 @@ +Regression Metrics +================== + +Regression Error Characteristic (REC) curves provide a powerful framework for evaluating and comparing regression models, generalizing the Receiver Operating Characteristic (ROC) curve used in classification. + +Background and Theory +------------------- + +The REC curve was introduced by **Bi & Bennett (2003)** in *"Regression error characteristic curves"* (Proceedings of the Twentieth International Conference on Machine Learning, pp. 43-50) as a method to visualize the cumulative distribution of errors in regression modeling. + +Unlike classification where predictions are binary, regression predictions fall along a continuous scale. The REC curve handles this by plotting **Error Tolerance** on the x-axis against **Accuracy** on the y-axis. Here, "accuracy" is defined as the proportion of predictions that fall within the specified error tolerance. + +This framework was further extended into the Regression ROC (RROC) space by **Hernández-Orallo (2013)** in *"ROC curves for regression"* (Pattern Recognition, 46(12), 3395-3411, doi:10.1016/j.patcog.2013.06.014). This work demonstrated that the area under/over the curve relates deeply to the variance and expected magnitude of regression errors. + +### Interpreting the Area Over the Curve (AOC) + +In classification ROC curves, a larger Area Under the Curve (AUC) is better. For REC curves, we look at the **Area Over the Curve (AOC)**: + +* **0.0 is Perfect:** A perfect model has zero error for every prediction. Its REC curve jumps to 1.0 (100% accuracy) instantly at an error tolerance of 0, meaning there is zero area *over* the curve. +* **Lower is Better:** The smaller the AOC, the closer the predictions are to the true values. +* **Normalized Score:** The AOC is typically normalized by dividing by the maximum absolute error observed, yielding a bounded metric in [0, 1]. This makes it easier to compare performance across different datasets and scales. + +Regression AUC Score +-------------------- + +.. autofunction:: ds_utils.metrics.regression.regression_auc_score + +Code Example +~~~~~~~~~~~~ + +.. code-block:: python + + import numpy as np + from ds_utils.metrics.regression import regression_auc_score + + # Generate dummy data + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]) + y_pred_good = np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]) + y_pred_bad = np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0]) + + # Calculate AOC scores (normalized) + good_aoc = regression_auc_score(y_true, y_pred_good) + bad_aoc = regression_auc_score(y_true, y_pred_bad) + + print(f"Good Model AOC: {good_aoc:.4f}") + print(f"Bad Model AOC: {bad_aoc:.4f}") + +.. code-block:: text + + Output: + Good Model AOC: 0.1000 + Bad Model AOC: 0.4200 + +Plot REC Curve with Annotations +------------------------------- + +.. autofunction:: ds_utils.metrics.regression.plot_rec_curve_with_annotations + +Code Example +~~~~~~~~~~~~ + +We can use the `plot_rec_curve_with_annotations` function to visually compare the two models from the previous example. The Plotly backend allows for interactive exploration of the error tolerances. + +.. code-block:: python + + import numpy as np + from ds_utils.metrics.regression import plot_rec_curve_with_annotations + + # Generate dummy data + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]) + predictions = { + "Good Model": np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]), + "Bad Model": np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0]), + } + + # Plot REC curves + fig = plot_rec_curve_with_annotations(y_true, predictions) + fig.show() + +And the following interactive graph will be shown: + +.. image:: ../../../tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png + :align: center + :alt: Regression Error Characteristic (REC) Curve diff --git a/ds_utils/__init__.py b/ds_utils/__init__.py index 5640c46..f8aed37 100644 --- a/ds_utils/__init__.py +++ b/ds_utils/__init__.py @@ -1,3 +1,3 @@ """Data Science Utilities package.""" -__version__ = "1.10.0rc11" +__version__ = "1.10.0rc12" diff --git a/ds_utils/metrics/__init__.py b/ds_utils/metrics/__init__.py index 43fc8dd..2d48bd9 100644 --- a/ds_utils/metrics/__init__.py +++ b/ds_utils/metrics/__init__.py @@ -3,6 +3,7 @@ This package provides functions for evaluating and visualizing model performance: - confusion_matrix - Confusion matrix visualization and classification metrics - curves - ROC and Precision-Recall curves with threshold annotations +- regression - Regression Error Characteristic (REC) curves and Regression AUC - error_analysis - Tabular error-analysis report generation - learning_curves - Learning curve visualization - probability_analysis - Probability calibration and accuracy analysis diff --git a/ds_utils/metrics/regression.py b/ds_utils/metrics/regression.py new file mode 100644 index 0000000..0eb5280 --- /dev/null +++ b/ds_utils/metrics/regression.py @@ -0,0 +1,165 @@ +"""Regression evaluation metrics and visualization. + +This module provides the Regression Error Characteristic (REC) curve and +the associated Area Over the Curve (AOC) metric for evaluating and +comparing regression models. +""" + +from typing import Dict, Optional, Tuple + +import numpy as np +from plotly import graph_objects as go +from sklearn.metrics import auc + + +def _calculate_rec_curve( + y_true: np.ndarray, + y_pred: np.ndarray, + sample_weight: Optional[np.ndarray] = None, + n_points: int = 1000, +) -> Tuple[np.ndarray, np.ndarray]: + """Calculate REC curve coordinates. + + :param y_true: True target values. + :param y_pred: Predicted target values. + :param sample_weight: Sample weights. + :param n_points: Number of points to calculate on the curve. + :return: Tuple of (error_tolerances, accuracies). + """ + errors = np.abs(y_true - y_pred) + + if sample_weight is not None: + sample_weight = np.array(sample_weight) + sample_weight = sample_weight / sample_weight.sum() + else: + sample_weight = np.ones(len(errors)) / len(errors) + + max_error = np.max(errors) + error_tolerances = np.linspace(0, max_error, n_points) + + accuracies = np.array([sample_weight[errors <= tolerance].sum() for tolerance in error_tolerances]) + + return error_tolerances, accuracies + + +def regression_auc_score( + y_true: np.ndarray, + y_pred: np.ndarray, + sample_weight: Optional[np.ndarray] = None, + normalize: bool = True, +) -> float: + """Calculate Area Over the REC Curve (AOC) / Regression AUC. + + This is the standalone version of the AUC calculation used in + :func:`plot_rec_curve_with_annotations`. Lower values indicate + better model performance. The AOC is calculated as the area between + the REC curve and the y=1 line. + + When normalized, it is divided by the maximum absolute error to give + a score in [0, 1]. + + :param y_true: array-like of shape (n_samples,). True target values. + :param y_pred: array-like of shape (n_samples,). Predicted target values. + :param sample_weight: array-like of shape (n_samples,), default=None. Sample weights. + :param normalize: bool, default=True. If True, normalize by maximum absolute error + to get a score in [0, 1] range. + :return: The regression AUC score. Lower is better (0 = perfect, 1 = worst possible). + :raises ValueError: If shapes of y_true and y_pred do not match. + """ + y_true = np.asarray(y_true) + y_pred = np.asarray(y_pred) + + if y_true.shape != y_pred.shape: + raise ValueError(f"Shape mismatch: y_true {y_true.shape} and y_pred {y_pred.shape}") + + error_tolerances, accuracies = _calculate_rec_curve(y_true, y_pred, sample_weight) + + max_error = np.max(np.abs(y_true - y_pred)) + + # Area under the REC curve using sklearn.metrics.auc + auc_under_curve = auc(error_tolerances, accuracies) + + # AOC = total rectangle area (max_error * 1.0) minus area under curve + aoc = max_error * 1.0 - auc_under_curve + + if normalize: + if max_error > 0: + aoc = aoc / max_error + + return float(aoc) + + +def plot_rec_curve_with_annotations( + y_true: np.ndarray, + regressors_names_and_predictions_dict: Dict[str, np.ndarray], + *, + sample_weight: Optional[np.ndarray] = None, + normalize_auc: bool = True, + fig: Optional[go.Figure] = None, + mode: Optional[str] = "lines+markers", + show_legend: bool = True, + **kwargs, +) -> go.Figure: + """Plot Regression Error Characteristic (REC) curves with AUC annotations. + + The REC curve shows the cumulative distribution of absolute errors, + allowing comparison of regression model performance. The Area Over + the Curve (AOC) is calculated and displayed in the legend for each + regressor. + + :param y_true: array-like of shape (n_samples,). True target values. + :param regressors_names_and_predictions_dict: mapping from regressor name to predictions. + :param sample_weight: array-like of shape (n_samples,), default=None. Sample weights. + :param normalize_auc: bool, default=True. If True, normalize AOC by maximum absolute error + to get a score in [0, 1] range. + :param fig: plotly's Figure object, optional. The figure to plot on. + :param mode: str, default='lines+markers'. Determines the drawing mode for this scatter trace. + :param show_legend: bool, default=True. Whether to display legend in the plot. + :param kwargs: additional keyword arguments to be passed to the plot function. + :return: The Figure object with the plot drawn onto it. + :raises ValueError: If the input data is invalid or inconsistent. + """ + if fig is None: + fig = go.Figure() + + y_true = np.asarray(y_true) + + for regressor_name, y_pred in regressors_names_and_predictions_dict.items(): + y_pred = np.asarray(y_pred) + + if y_true.shape != y_pred.shape: + raise ValueError( + f"Shape mismatch: y_true {y_true.shape} and y_pred {y_pred.shape} for regressor {regressor_name}" + ) + + try: + error_tolerances, accuracies = _calculate_rec_curve(y_true, y_pred, sample_weight) + except ValueError as e: + raise ValueError(f"Error calculating REC curve for regressor {regressor_name}: {str(e)}") + + try: + auc_score = regression_auc_score(y_true, y_pred, sample_weight=sample_weight, normalize=normalize_auc) + except ValueError as e: + raise ValueError(f"Error calculating AUC score for regressor {regressor_name}: {str(e)}") + + hover_text = [f"Tolerance: {tol:.4f}
Accuracy: {acc:.4f}" for tol, acc in zip(error_tolerances, accuracies)] + + fig.add_trace( + go.Scatter( + x=error_tolerances, + y=accuracies, + mode=mode, + text=hover_text, + hoverinfo="text", + name=f"{regressor_name} (AOC = {auc_score:.4f})", + **kwargs, + ) + ) + + fig.update_layout( + xaxis_title="Error Tolerance", + yaxis_title="Accuracy (Proportion within tolerance)", + showlegend=show_legend, + ) + + return fig diff --git a/skills/metrics/SKILL.md b/skills/metrics/SKILL.md index f2cd89f..79945d5 100644 --- a/skills/metrics/SKILL.md +++ b/skills/metrics/SKILL.md @@ -32,6 +32,7 @@ from ds_utils.metrics.curves import plot_precision_recall_curve_with_thresholds_ from ds_utils.metrics.probability_analysis import plot_error_analysis_chart from ds_utils.metrics.error_analysis import generate_error_analysis_report from ds_utils.metrics.time_series import directional_accuracy_score, directional_bias_score +from ds_utils.metrics.regression import regression_auc_score, plot_rec_curve_with_annotations ``` --- @@ -425,7 +426,76 @@ Directional Bias: 1.00 --- -## Typical Workflow +## regression_auc_score + +Calculates the Area Over the REC Curve (AOC) / Regression AUC for evaluating continuous predictions. Lower values indicate better model performance. The AOC is calculated as the area between the REC curve and the y=1 line. + +```python +from ds_utils.metrics.regression import regression_auc_score +import numpy as np + +# complete usage example +y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) +y_pred = np.array([1.5, 2.5, 3.5, 4.5, 5.5]) +auc_score = regression_auc_score(y_true, y_pred) +print(f"Regression AOC: {auc_score:.4f}") +``` + +Output: +```text +Regression AOC: 0.1000 +``` + +**Parameters:** +- `y_true` — array-like of shape (n_samples,). True target values. +- `y_pred` — array-like of shape (n_samples,). Predicted target values. +- `sample_weight` — array-like of shape (n_samples,), default=None. Sample weights. +- `normalize` — bool, default=True. If True, normalize by maximum absolute error to get a score in [0, 1] range. + +**Returns:** float in [0, 1] if `normalize=True`. Lower is better (0 = perfect, 1 = worst possible). + +**Common mistakes:** +- Comparing unnormalized AOC across models with different test sets. Always use `normalize=True` for cross-dataset comparisons. +- Assuming higher is better (like classification ROC AUC). For REC curves, **Area Over the Curve (AOC)** is used, so **lower is better**. + +--- + +## plot_rec_curve_with_annotations + +Plots Regression Error Characteristic (REC) curves with AUC annotations using Plotly. + +```python +from ds_utils.metrics.regression import plot_rec_curve_with_annotations +import numpy as np + +# complete usage example +y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) +predictions = { + 'Model A': np.array([1.1, 2.2, 3.3, 4.4, 5.5]), + 'Model B': np.array([1.5, 2.5, 3.5, 4.5, 5.5]), +} +fig = plot_rec_curve_with_annotations(y_true, predictions) +fig.show() +``` + +**Parameters:** +- `y_true` — array-like of shape (n_samples,). True target values. +- `regressors_names_and_predictions_dict` — dict, mapping from regressor name to predictions. +- `sample_weight` — array-like of shape (n_samples,), default=None. Sample weights. +- `normalize_auc` — bool, default=True. If True, normalize AOC by maximum absolute error to get a score in [0, 1] range. +- `fig` — plotly's Figure object, optional. The figure to plot on. +- `mode` — str, default='lines+markers'. Determines the drawing mode for this scatter trace. +- `show_legend` — bool, default=True. Whether to display legend in the plot. + +**Returns:** Plotly Figure. + +**Common mistakes:** +- Passing individual predictions directly; the function takes a dict of `{name: predictions_array}`. +- Returning a matplotlib plot; the function returns a **Plotly Figure**, which requires `fig.show()`. + +--- + +## Classification Workflow ```python import matplotlib.pyplot as plt @@ -502,3 +572,33 @@ da = directional_accuracy_score(y_forecast_true, y_forecast_pred) bias = directional_bias_score(y_forecast_true, y_forecast_pred) print(f"DA: {da:.2%}, Bias: {bias:.2f}") ``` + +--- + +## Regression Workflow + +```python +import numpy as np +from sklearn.ensemble import RandomForestRegressor +from ds_utils.metrics.regression import plot_rec_curve_with_annotations, regression_auc_score + +# 1. Train models and get predictions +rf_reg = RandomForestRegressor(random_state=42) +rf_reg.fit(X_train, y_train) + +y_pred_rf = rf_reg.predict(X_test) +y_pred_baseline = np.full_like(y_test, y_train.mean()) # Baseline model predicting the mean + +predictions = { + "Random Forest": y_pred_rf, + "Mean Baseline": y_pred_baseline +} + +# 2. Get standalone AOC score for a model +rf_aoc = regression_auc_score(y_test, y_pred_rf) +print(f"Random Forest AOC: {rf_aoc:.4f}") + +# 3. Plot REC curve comparing models +fig = plot_rec_curve_with_annotations(y_test, predictions) +fig.show() +``` diff --git a/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png b/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png new file mode 100644 index 0000000..2473fcb Binary files /dev/null and b/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_basic.png differ diff --git a/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_existing_figure.png b/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_existing_figure.png new file mode 100644 index 0000000..67a21b9 Binary files /dev/null and b/tests/baseline_images/test_metrics/test_regression/test_plot_rec_curve_existing_figure.png differ diff --git a/tests/test_metrics/test_curves.py b/tests/test_metrics/test_curves.py index 16f76c6..b73059a 100644 --- a/tests/test_metrics/test_curves.py +++ b/tests/test_metrics/test_curves.py @@ -4,7 +4,6 @@ from pathlib import Path from typing import Any, Dict -from matplotlib import pyplot as plt import numpy as np from plotly import graph_objects as go import pytest @@ -13,6 +12,7 @@ plot_precision_recall_curve_with_thresholds_annotations, plot_roc_curve_with_thresholds_annotations, ) +from tests.utils import save_plotly_figure_and_return_matplot BASELINE_DIR = Path(__file__).parents[1] / "baseline_images" / Path(__file__).parent.name / Path(__file__).stem RESULT_DIR = Path(__file__).parents[1] / "result_images" / Path(__file__).parent.name / Path(__file__).stem @@ -35,16 +35,6 @@ def plotly_models_pr_curve_dict() -> Dict[str, Any]: return json.load(file) -def save_plotly_figure_and_return_matplot(fig: go.Figure, path_to_save: Path) -> plt.Figure: - """Save plotly figure and convert to a matplotlib figure for comparison.""" - fig.write_image(str(path_to_save)) - img = plt.imread(path_to_save) - figure, ax = plt.subplots() - ax.imshow(img) - ax.axis("off") - return figure - - @pytest.mark.mpl_image_compare(baseline_dir=BASELINE_DIR, tolerance=18) @pytest.mark.parametrize( ("add_random_classifier_line", "random_classifier_line_kw"), diff --git a/tests/test_metrics/test_regression.py b/tests/test_metrics/test_regression.py new file mode 100644 index 0000000..b2b40f5 --- /dev/null +++ b/tests/test_metrics/test_regression.py @@ -0,0 +1,148 @@ +"""Tests for regression metrics module.""" + +from pathlib import Path + +import numpy as np +from plotly import graph_objects as go +import pytest + +from ds_utils.metrics.regression import ( + plot_rec_curve_with_annotations, + regression_auc_score, +) +from tests.utils import save_plotly_figure_and_return_matplot + +BASELINE_DIR = Path(__file__).parents[1] / "baseline_images" / Path(__file__).parent.name / Path(__file__).stem +RESULT_DIR = Path(__file__).parents[1] / "result_images" / Path(__file__).parent.name / Path(__file__).stem + +RESULT_DIR.mkdir(exist_ok=True, parents=True) + + +# --------------------------------------------------------------------------- +# regression_auc_score tests +# --------------------------------------------------------------------------- + + +def test_regression_auc_score_perfect_predictions(): + """Test that perfect predictions give AUC of 0.""" + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) + y_pred = y_true.copy() + auc_val = regression_auc_score(y_true, y_pred) + assert auc_val == pytest.approx(0.0, abs=1e-6) + + +def test_regression_auc_score_with_errors_normalized(): + """Test AUC calculation with known errors (normalized).""" + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) + y_pred = np.array([1.5, 2.5, 3.5, 4.5, 5.5]) + auc_val = regression_auc_score(y_true, y_pred) + assert 0 < auc_val < 1 + + +def test_regression_auc_score_with_errors_unnormalized(): + """Test AUC calculation with known errors (unnormalized).""" + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) + y_pred = np.array([1.5, 2.5, 3.5, 4.5, 5.5]) + auc_normalized = regression_auc_score(y_true, y_pred, normalize=True) + auc_raw = regression_auc_score(y_true, y_pred, normalize=False) + # Raw AOC should be in error-scale units, not [0,1] + assert auc_raw != auc_normalized + + +def test_regression_auc_score_shape_mismatch(): + """Test that shape mismatch raises ValueError.""" + y_true = np.array([1.0, 2.0, 3.0]) + y_pred = np.array([1.0, 2.0]) + with pytest.raises(ValueError, match="Shape mismatch"): + regression_auc_score(y_true, y_pred) + + +def test_regression_auc_score_with_sample_weights(): + """Test AUC calculation with sample weights.""" + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) + # Non-uniform errors: [0.1, 0.3, 0.5, 0.7, 1.0] + y_pred = np.array([1.1, 2.3, 3.5, 4.7, 6.0]) + # Weight heavily the samples with large errors + weights = np.array([1.0, 1.0, 1.0, 5.0, 5.0]) + auc_weighted = regression_auc_score(y_true, y_pred, sample_weight=weights) + auc_unweighted = regression_auc_score(y_true, y_pred) + assert auc_weighted != auc_unweighted + + +def test_regression_auc_score_worse_model_has_higher_aoc(): + """Test that a worse model produces a higher AOC.""" + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0]) + # Good model: varied small errors [0.1, 0.0, 0.2, 0.1, 0.3] + y_pred_good = np.array([1.1, 2.0, 3.2, 4.1, 5.3]) + # Bad model: varied large errors [1.0, 2.0, 0.5, 1.5, 2.0] + y_pred_bad = np.array([2.0, 4.0, 2.5, 5.5, 7.0]) + auc_good = regression_auc_score(y_true, y_pred_good, normalize=False) + auc_bad = regression_auc_score(y_true, y_pred_bad, normalize=False) + assert auc_good < auc_bad + + +# --------------------------------------------------------------------------- +# plot_rec_curve_with_annotations tests +# --------------------------------------------------------------------------- + + +@pytest.mark.mpl_image_compare(baseline_dir=BASELINE_DIR, tolerance=18) +def test_plot_rec_curve_basic(request): + """Test basic REC curve plotting with 2 models.""" + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]) + predictions = { + "Good Model": np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]), + "Bad Model": np.array([2.0, 3.5, 1.5, 5.5, 3.0, 8.0, 5.5, 9.5, 7.0, 12.0]), + } + fig = plot_rec_curve_with_annotations(y_true, predictions) + + return save_plotly_figure_and_return_matplot(fig, RESULT_DIR / f"{request.node.name}.png") + + +@pytest.mark.mpl_image_compare(baseline_dir=BASELINE_DIR, tolerance=18) +def test_plot_rec_curve_existing_figure(request): + """Test plotting REC curve on an existing figure.""" + fig = go.Figure() + fig.update_layout(title="Custom REC Curves") + + y_true = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]) + predictions = { + "Model A": np.array([1.1, 2.2, 2.8, 4.1, 5.0, 5.9, 7.2, 7.8, 9.1, 10.0]), + } + fig = plot_rec_curve_with_annotations(y_true, predictions, fig=fig) + + return save_plotly_figure_and_return_matplot(fig, RESULT_DIR / f"{request.node.name}.png") + + +def test_plot_rec_curve_shape_mismatch(): + """Test that shape mismatch raises ValueError with regressor name.""" + y_true = np.array([1.0, 2.0, 3.0]) + predictions = { + "Model A": np.array([1.0, 2.0]), + } + with pytest.raises(ValueError, match=r"Shape mismatch.*Model A"): + plot_rec_curve_with_annotations(y_true, predictions) + + +def test_plot_rec_curve_calc_failure(mocker): + """Test that REC curve calculation failure is re-raised with regressor name.""" + mocker.patch( + "ds_utils.metrics.regression._calculate_rec_curve", + side_effect=ValueError("test error"), + ) + y_true = np.array([1.0, 2.0, 3.0]) + predictions = {"Model A": np.array([1.5, 2.5, 3.5])} + with pytest.raises(ValueError, match="Error calculating REC curve for regressor Model A:"): + plot_rec_curve_with_annotations(y_true, predictions) + + +def test_plot_rec_curve_auc_score_calc_failure(mocker): + """Test that AUC score calculation failure is re-raised with regressor name.""" + mocker.patch( + "ds_utils.metrics.regression.regression_auc_score", + side_effect=ValueError("test error"), + ) + y_true = np.array([1.0, 2.0, 3.0]) + predictions = {"Model A": np.array([1.5, 2.5, 3.5])} + with pytest.raises(ValueError, match="Error calculating AUC score for regressor Model A:"): + plot_rec_curve_with_annotations(y_true, predictions) diff --git a/tests/test_version.py b/tests/test_version.py index 92f6083..f8879c4 100644 --- a/tests/test_version.py +++ b/tests/test_version.py @@ -5,4 +5,4 @@ def test_version(): """Test that the package version is correct.""" - assert ds_utils.__version__ == "1.10.0rc11" + assert ds_utils.__version__ == "1.10.0rc12" diff --git a/tests/utils.py b/tests/utils.py new file mode 100644 index 0000000..50d0214 --- /dev/null +++ b/tests/utils.py @@ -0,0 +1,16 @@ +"""Testing utility functions.""" + +from pathlib import Path + +from matplotlib import pyplot as plt +from plotly import graph_objects as go + + +def save_plotly_figure_and_return_matplot(fig: go.Figure, path_to_save: Path) -> plt.Figure: + """Save plotly figure and convert to a matplotlib figure for comparison.""" + fig.write_image(str(path_to_save)) + img = plt.imread(path_to_save) + figure, ax = plt.subplots() + ax.imshow(img) + ax.axis("off") + return figure