refactor(test-cases): consolidate diagnostic execution and drift-check integration tests

changelog/789.trivial.md

@@ -0,0 +1,3 @@
+Consolidated the diagnostic test-case execution path so the provider integration tests re-run each diagnostic and verify its committed regression bundle has not drifted, rather than only asserting the run succeeded.
+The per-provider integration test modules now share a single implementation via `climate_ref.testing.create_no_drift_test`,
+and the unused `run_test_case` pytest fixture was removed (use `TestCaseRunner` directly in custom tests).

docs/api-surface.md

@@ -113,7 +113,6 @@ The primary module providers interact with.
 | `TestCase`                 | Class (attrs) | A single test case definition                   |
 | `TestDataSpecification`    | Class (attrs) | Collection of test cases for a diagnostic       |
 | `TestCasePaths`            | Class (attrs) | Path resolver for test case data                |
-| `RegressionValidator`      | Class (attrs) | Validate outputs from stored regression data    |
 | `validate_cmec_bundles`    | Function      | Validate CMEC metric/output bundles             |
 | `collect_test_case_params` | Function      | Collect pytest parametrize params from provider |
 | `load_datasets_from_yaml`  | Function      | Load ExecutionDatasetCollection from YAML       |

docs/how-to-guides/testing-diagnostics.md

@@ -334,42 +334,84 @@ ref test-cases run --provider my-provider --diagnostic my-diagnostic --output-di
 ref test-cases run --provider my-provider --diagnostic my-diagnostic --force-regen
 ```
 
-## Writing Pytest Tests
+## Pytest Integration
 
-The CLI provides a standard harness for running simple test cases.
-Sometimes additional custom tests maybe required which require writing tests via `pytest`.
+### The generated per-provider test
 
-### Using the `run_test_case` Fixture
+Every provider package wires its test cases into pytest with a single line in
+`tests/integration/test_diagnostics.py`:
 
-The `run_test_case` fixture automatically handles missing data by skipping tests:
+```python
+from climate_ref_example import provider
+
+from climate_ref.testing import create_no_drift_test
+
+test_run_test_cases = create_no_drift_test(provider)
+```
+
+`create_no_drift_test` generates one parameterized pytest case per test case,
+identified as `{diagnostic}/{test_case}` and marked `slow` and `test_cases`.
+Each case re-executes the diagnostic against the fetched catalog
+using the same execute/build stages as `ref test-cases run`,
+then asserts the freshly rebuilt committed bundle
+matches the tracked `regression/` baseline within tolerance.
+Cases are skipped when the fetched catalog or the committed baseline is missing,
+so the suite stays green for contributors who haven't fetched data for that provider.
+
+Run them with the `--slow` flag:
+
+```bash
+# All test cases for a provider
+uv run pytest packages/climate-ref-example/tests/integration/test_diagnostics.py --slow
+
+# A single test case, selected by its id
+uv run pytest --slow \
+    "packages/climate-ref-example/tests/integration/test_diagnostics.py::test_run_test_cases[global-mean-timeseries/default]"
+```
+
+### Writing custom tests
+
+For behaviour the standard drift check doesn't cover,
+run a test case directly with `TestCaseRunner`.
+The `config` fixture (an isolated per-test configuration)
+comes from the `climate_ref.conftest_plugin` pytest plugin,
+which is auto-discovered when `climate-ref` is installed.
 
 ```python
 import pytest
+from my_provider import provider
+
+from climate_ref.testing import TestCaseRunner
+from climate_ref_core.testing import TestCasePaths, load_datasets_from_yaml
+
+
+@pytest.mark.slow
+def test_my_diagnostic(config, tmp_path):
+    diagnostic = provider.get("my-diagnostic")
+    diagnostic.provider.configure(config)
 
-def test_my_diagnostic(run_test_case):
-    from my_provider import MyDiagnostic
+    paths = TestCasePaths.from_diagnostic(diagnostic, "default")
+    if paths is None or not paths.catalog.exists():
+        pytest.skip("Test data not fetched; run `ref test-cases fetch` first")
 
-    diagnostic = MyDiagnostic()
-    result = run_test_case.run(diagnostic, "default")
+    runner = TestCaseRunner(config=config, datasets=load_datasets_from_yaml(paths.catalog))
+    result = runner.run(diagnostic, "default", output_dir=tmp_path / "output")
 
     assert result.successful
-    assert result.metric_bundle_filename is not None
 ```
 
-In future, tests will be generated for each of the test cases.
+### Markers
 
-### Marking Tests
-
-Use pytest markers for test categorization:
+Mark custom tests so they run in the right CI tier:
 
 ```python
 @pytest.mark.slow
-def test_full_resolution(run_test_case):
+def test_full_resolution(config):
     """Test with full-resolution ESGF data (slow)."""
     ...
 
 @pytest.mark.requires_esgf_data
-def test_requires_fetched_data(run_test_case):
+def test_requires_fetched_data(config):
     """Test that requires fetched ESGF data."""
     ...
 ```
@@ -380,6 +422,9 @@ Run specific test categories:
 # Include slow tests
 pytest --slow
 
+# Only the generated test-case drift tests
+pytest --slow -m test_cases
+
 # Skip tests requiring ESGF data
 pytest -m "not requires_esgf_data"
 ```
@@ -526,6 +571,62 @@ If you hold credentials locally, `ref test-cases mint` does the same from your m
    - The derived test data directory path
    - Whether catalogs/regression directories are being created
 
+### Debugging a failing test case
+
+When a pytest drift test (or the CI replay gate) reports that a committed bundle has drifted,
+the fastest loop is usually through the CLI rather than pytest,
+because the outputs stay in a predictable place.
+
+1. **Re-run the test case via the CLI** and inspect what it produced:
+
+   ```bash
+   ref --verbose test-cases run --provider my-provider --diagnostic my-diagnostic
+   ```
+
+   Every run repopulates the gitignored `output/latest/` slot in the test case directory
+   with the executed native files and, under `output/latest/regression/`,
+   the rebuilt committed bundle.
+   Pass `--output-directory ./scratch` to also keep the raw execution directory
+   (logs, intermediate files) somewhere convenient.
+
+2. **Diff the rebuilt bundle against the tracked baseline**:
+
+   ```bash
+   cd packages/climate-ref-my-provider/tests/test-data/my-diagnostic/default
+   git diff --no-index regression/ output/latest/regression/
+   ```
+
+   The drift assertion also prints a per-file summary of what moved beyond tolerance.
+
+3. **Compare two runs side by side** using named output slots —
+   `latest` is overwritten on every run, but a named slot persists:
+
+   ```bash
+   ref test-cases run ... --label before
+   # ... change the code ...
+   ref test-cases run ... --label after
+   diff -r .../output/before/regression .../output/after/regression
+   ```
+
+4. **Separate "my code changed the results" from "the baseline is stale"** with
+   `ref test-cases replay --provider my-provider`.
+
+   Replay rebuilds the bundle from the stored native baseline *without executing the diagnostic*,
+   so if replay passes but `run` drifts, your change moved the results;
+   if replay also fails, the committed baseline itself is inconsistent
+   (regenerate with `run --force-regen` and bump `test_case_version`).
+
+5. **Debug through pytest** when you need a debugger at the point of failure:
+
+   ```bash
+   uv run pytest --slow --pdb \
+       "packages/climate-ref-my-provider/tests/integration/test_diagnostics.py::test_run_test_cases[my-diagnostic/default]"
+   ```
+
+   The pytest run works in the test's `tmp_path`
+   (retained for the most recent runs under `/tmp/pytest-of-<user>/`):
+   the execution directory is `exec/` and the rebuilt bundle is `slot/regression/`.
+
 ## Other useful links
 
 - [Adding Custom Diagnostics](adding_custom_diagnostics.md)

packages/climate-ref-core/src/climate_ref_core/diagnostics.py

@@ -610,7 +610,7 @@ def provider(self) -> DiagnosticProvider:
     def provider(self, value: DiagnosticProvider) -> None:
         self._provider = value
 
-    def run(self, definition: ExecutionDefinition) -> ExecutionResult:
+    def run(self, definition: ExecutionDefinition, *, capture_regression: bool = False) -> ExecutionResult:
         """
         Run the diagnostic on the given configuration.
 
@@ -620,24 +620,32 @@ def run(self, definition: ExecutionDefinition) -> ExecutionResult:
         ----------
         definition
             The configuration to run the diagnostic on.
+        capture_regression
+            When True, call :meth:`prepare_regression_output` between execute
+            and build so the native output is normalised for deterministic regression capture.
+
+            Used only by the test-case runner.
         """
         # Execute the diagnostic
         # This may be run in a separate process (or python environment)
         self.execute(definition)
 
+        if capture_regression:
+            self.prepare_regression_output(definition)
+
         # Build the result from the output bundle
         return self.build_execution_result(definition)
 
     def prepare_regression_output(self, definition: ExecutionDefinition) -> None:
         """
         Normalise native output for deterministic regression capture.
 
-        Hook called by the test-case regression runner *between*
+        Hook called by :meth:`run` (only when ``capture_regression=True``) *between*
         :meth:`execute` and :meth:`build_execution_result`,
         allowing a provider to remove avoidable non-determinism (e.g. timestamped directory names)
         before the output bundle is built and captured as a baseline.
 
-        This is **not** called during normal execution (:meth:`run`),
+        Normal execution (``capture_regression=False``, the default) does not call it,
         so it must only be used for regression-capture concerns.
 
         The default implementation does nothing.

packages/climate-ref-core/src/climate_ref_core/testing.py

@@ -4,14 +4,12 @@
 This module provides:
 - TestCase and TestDataSpecification for defining test scenarios
 - YAML serialization for dataset catalogs (with paths stored separately)
-- RegressionValidator for validating pre-stored outputs
-- Utilities for CMEC bundle validation
+- Utilities for CMEC bundle and series regression validation
 """
 
 from __future__ import annotations
 
 import json
-import shutil
 import sys
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
@@ -27,10 +25,10 @@
     Selector,
     SourceDatasetType,
 )
-from climate_ref_core.diagnostics import ExecutionDefinition, ExecutionResult
+from climate_ref_core.diagnostics import ExecutionResult
 from climate_ref_core.esgf.base import ESGFRequest
 from climate_ref_core.metric_values.typing import SeriesMetricValue
-from climate_ref_core.output_files import PlaceholderMap, ordered_replacements
+from climate_ref_core.output_files import ordered_replacements
 from climate_ref_core.paths import safe_path
 from climate_ref_core.pycmec.metric import CMECMetric
 from climate_ref_core.pycmec.output import CMECOutput
@@ -651,93 +649,6 @@ def _sorted_dump(series: list[SeriesMetricValue]) -> list[dict[str, Any]]:
     )
 
 
-@frozen
-class RegressionValidator:
-    """
-    Validate diagnostic outputs from pre-stored regression data.
-
-    Loads regression outputs and validates CMEC bundles without
-    running the diagnostic. Suitable for fast CI validation.
-
-    The regression data is expected at:
-    test_data_dir/{diagnostic}/{test_case}/regression/
-    """
-
-    diagnostic: Diagnostic
-    test_case_name: str
-    test_data_dir: Path
-
-    @property
-    def paths(self) -> TestCasePaths:
-        """Get paths for this test case."""
-        return TestCasePaths.from_test_data_dir(self.test_data_dir, self.diagnostic.slug, self.test_case_name)
-
-    @property
-    def _baseline_placeholders(self) -> PlaceholderMap:
-        """
-        Base placeholder map for this verification context.
-
-        Declared once and reused by both :meth:`load_regression_definition` (hydrate) and
-        :meth:`validate` (series replacements) so the two sides cannot drift to different token sets.
-        This context does not know the shared-software root, so the map omits ``<SOFTWARE_ROOT_DIR>``;
-        each caller binds the per-execution ``<OUTPUT_DIR>`` via :meth:`PlaceholderMap.with_output`.
-        """
-        return PlaceholderMap.for_baseline(test_data_dir=self.test_data_dir)
-
-    def has_regression_data(self) -> bool:
-        """Check if regression data exists for this test case."""
-        regression_path = self.paths.regression
-        return regression_path.exists() and (regression_path / "diagnostic.json").exists()
-
-    def load_regression_definition(self, tmp_dir: Path) -> ExecutionDefinition:
-        """
-        Load regression data and create an ExecutionDefinition.
-
-        Copies regression data to tmp_dir and replaces path placeholders.
-        """
-        regression_path = self.paths.regression
-        catalog_path = self.paths.catalog
-
-        if not catalog_path.exists():
-            raise FileNotFoundError(
-                f"No catalog file at {catalog_path} for test case datasets. Run `ref test-cases fetch` first."
-            )
-        if not regression_path.exists():
-            raise FileNotFoundError(
-                f"No regression data at {regression_path}. Run 'ref test-cases run --force-regen' first."
-            )
-
-        output_dir = tmp_dir / "output"
-        output_dir.mkdir(parents=True, exist_ok=True)
-        shutil.copytree(regression_path, output_dir, dirs_exist_ok=True)
-
-        self._baseline_placeholders.with_output(output_dir).hydrate(output_dir)
-
-        datasets: ExecutionDatasetCollection = load_datasets_from_yaml(catalog_path)
-
-        return ExecutionDefinition(
-            diagnostic=self.diagnostic,
-            key=f"test-{self.test_case_name}",
-            datasets=datasets,
-            output_directory=output_dir,
-            root_directory=tmp_dir,
-        )
-
-    def validate(self, definition: ExecutionDefinition) -> None:
-        """Validate CMEC bundles and series in the regression output."""
-        result = self.diagnostic.build_execution_result(definition)
-        result.to_output_path("out.log").touch()  # Log file not tracked in regression
-        validate_cmec_bundles(self.diagnostic, result)
-        validate_series_regression(
-            expected_path=self.paths.regression / "series.json",
-            actual_path=definition.output_directory / "series.json",
-            slug=self.diagnostic.slug,
-            replacements=self._baseline_placeholders.with_output(
-                definition.output_directory
-            ).as_replacements(),
-        )
-
-
 def collect_test_case_params(provider: DiagnosticProvider) -> list[ParameterSet]:
     """
     Collect all diagnostic/test_case pairs from a provider for parameterized testing.