Add a test suite based on the upstream reference energies, and run it in CI (#5)

* Add a test suite based on the upstream reference energies

Each molecule directory in the vendored upstream data carries a README with a table
of determinant-selection thresholds and the electronic energy a diagonalization over
those determinants should produce. Those tables make a good first test: the
determinants are fixed, so the answer is deterministic and independent of sampling,
and the energies came from filtering a full CI calculation.

Eight cases are transcribed, from the h2o and n2 tables. The two cheapest run by
default in about 16 seconds; the rest are marked slow, following the --run-slow
pattern used in qiskit-addon-cutting so that they are reported as skipped with a
reason rather than silently deselected. Two of the slow cases were run to confirm the
transcription: h2o-1em4 (2.38e6 determinants) and n2-3em4 (4.12e5) reach their
published energies in about 16 minutes together.

One test is marked `mpi` and checks that the energy does not depend on how many
processes the determinants were spread across. `tox -e mpi` runs it under mpirun with
two processes, overridable with SBD_TEST_NPROCS.

The tests pin bit_length to 64. SBD packs determinants into words of that many bits
and freezes the resulting word count for the process -- det_vector::_elem_size is an
inline static that throws "elem_size mismatch" on a later, different value -- so h2o
(24 orbitals) and n2 (18) could not otherwise share a process. A bit_length of 64
gives both a single word. It does not affect the result: verified across 8, 20, 32,
48 and 64, spanning word counts 6 down to 1, with the h2o energy identical to ten
digits each time.

`sbd` is imported at module scope rather than through importorskip, so a build where
it is missing fails collection instead of reporting a suite that passed. The vendored
data is a submodule, so its absence does skip, and a backend named in SBD_TEST_DEVICE
that was not compiled skips too, since both are facts about the checkout rather than
failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Run the test suite in CI

Adapted from the qiskit-addon-utils workflow, with three changes this repo needs.

The checkout recurses submodules: the reference energies the tests check against
come from data in the vendored upstream checkout, and setup.py needs its headers to
compile at all.

MPI and BLAS are installed with apt, since the extension links against both.
setup.py defaults to looking for openblas on the system path and falls through to
mpicc detection, so libopenmpi-dev and libopenblas-dev are enough and no environment
variables are needed. That is also why the matrix is Linux-only for now: those
package names are Debian/Ubuntu specific, and macOS would additionally need libomp
because Apple clang ships without OpenMP.

Both tox environments run in one invocation rather than two. With `package = wheel`
and a shared wheel_build_env, one `tox -e py,mpi` compiles the extension once and
installs the resulting wheel into both environments; two separate commands would
compile it twice. Verified from a clean .tox: one build_wheel, two install_package,
36 seconds for the pair.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Report why tests were skipped

Most of this suite skips by default, so a bare count of skips said very little: it
did not distinguish a case being slow from a backend not being built or the reference
data being absent. Adding -rs to addopts, as qiskit-addon-cutting does, lists each
skip with its reason:

    SKIPPED [6] skipping slow test, as --run-slow was not provided
    SKIPPED [1] need --with-mpi option to run

--durations=10 comes along with it, since the cases differ by orders of magnitude in
cost and it is useful to see where the time went.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Report which backends were built and which is under test

setup.py builds the GPU backends only when nvc++ is present, so the same command
tests CPU alone on one machine and CPU plus GPU on another. Nothing in the output
said which, meaning a green run left the reader to deduce what had actually been
exercised -- and on CI, where the build messages are suppressed, to deduce it from
the absence of NVHPC on the runner.

A pytest header line now states it:

    sbd 1.6.1: backends built ['cpu'], testing default

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: garrison

.github/workflows/test_latest_versions.yml

@@ -0,0 +1,56 @@
+name: Tests
+
+on:
+  push:
+    branches:
+      - main
+      - 'stable/**'
+  pull_request:
+    branches:
+      - main
+      - 'stable/**'
+  schedule:
+    - cron: '0 1 * * *'
+
+jobs:
+  tests:
+    name: latest version tests (${{ matrix.os }}, ${{ matrix.python-version }})
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 30
+    strategy:
+      max-parallel: 4
+      matrix:
+        # Only Linux for now: the extension modules must compile against an MPI
+        # implementation and a BLAS, and the apt packages installed below are
+        # Debian/Ubuntu specific. macOS would additionally need libomp, since Apple
+        # clang ships without OpenMP support.
+        os: [ubuntu-latest]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          # The reference energies the tests check against come from data in the
+          # vendored upstream checkout, and setup.py needs its headers to compile.
+          submodules: recursive
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v7
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install MPI and BLAS
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libopenmpi-dev openmpi-bin libopenblas-dev
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install tox
+      - name: Test using tox environments
+        shell: bash
+        # One invocation rather than two, so that the extension is compiled once and
+        # the resulting wheel is installed into both environments. Running `tox -e py`
+        # and `tox -e mpi` as separate commands would compile it twice.
+        #
+        # Not --parallel: the two environments would then race to install the same
+        # wheel, and the MPI tests want the runner's cores to themselves.
+        run: |
+          tox -e py,mpi

pyproject.toml

@@ -30,6 +30,12 @@ dependencies = [
     "numpy>=1.19.0",
 ]
 
+[project.optional-dependencies]
+test = [
+    "pytest>=8.0",
+    "pytest-mpi>=0.6",
+]
+
 [project.urls]
 Homepage = "https://github.com/Qiskit/sbd-eigensolver-python"
 Documentation = "https://github.com/Qiskit/sbd-eigensolver-python"
@@ -41,4 +47,12 @@ Issues = "https://github.com/Qiskit/sbd-eigensolver-python/issues"
 # imperatively in setup.py; everything else lives in [project] above.
 packages = ["sbd"]
 package-dir = {sbd = "python"}
-zip-safe = false
+zip-safe = false
+
+[tool.pytest.ini_options]
+testpaths = ["test"]
+# -rs lists the skipped tests and why, which matters here because most of the suite
+# skips by default: a bare count of skips would not distinguish "this case is slow" or
+# "this backend was not built" from "the reference data is missing". --durations shows
+# where the time goes, the cases differing by orders of magnitude in cost.
+addopts = "-rs --durations=10"

test/conftest.py

@@ -0,0 +1,95 @@
+# This code is a Qiskit project.
+#
+# (C) Copyright IBM 2026.
+#
+# This code is licensed under the Apache License, Version 2.0. You may
+# obtain a copy of this license in the LICENSE.txt file in the root directory
+# of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
+#
+# Any modifications or derivative works of this code must retain this
+# copyright notice, and modified files need to carry a notice indicating
+# that they have been altered from the originals.
+
+"""Shared fixtures and helpers for the test suite."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import pytest
+
+# The reference molecules live in the vendored upstream SBD checkout, which is a git
+# submodule. It is not present in an sdist or a fresh clone until the submodule is
+# initialized, so tests that need it are skipped rather than failing.
+DATA_DIR = Path(__file__).resolve().parents[1] / "vendor" / "sbd-upstream" / "data"
+
+
+# Slow tests are opt-in through a command-line flag rather than excluded by default,
+# so that they are reported as skipped with a reason instead of silently deselected.
+# https://docs.pytest.org/en/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option
+
+
+# pylint: disable=missing-function-docstring
+def pytest_addoption(parser):
+    parser.addoption(
+        "--run-slow",
+        action="store_true",
+        default=False,
+        help="run slow tests",
+    )
+
+
+def pytest_configure(config):
+    config.addinivalue_line("markers", "slow: mark test as slow to run")
+
+
+def pytest_collection_modifyitems(config, items):
+    if not config.getoption("--run-slow"):
+        marker = pytest.mark.skip(reason="skipping slow test, as --run-slow was not provided")
+        for item in items:
+            if "slow" in item.keywords:
+                item.add_marker(marker)
+
+
+@pytest.fixture(scope="session")
+def data_dir() -> Path:
+    """Path to the vendored reference data, skipping the test if it is absent."""
+    if not DATA_DIR.is_dir():
+        pytest.skip(
+            f"reference data not found at {DATA_DIR}; "
+            "run 'git submodule update --init --recursive'"
+        )
+    return DATA_DIR
+
+
+def pytest_report_header():
+    """Record which backends were compiled, and which of them is under test.
+
+    Without this, a run gives no indication of what it actually exercised: setup.py
+    builds the GPU backends only when nvc++ is present, so the same command tests CPU
+    only on one machine and CPU plus GPU on another. A passing run should say which.
+    """
+    import sbd
+
+    available = sbd.available_backends()
+    requested = os.environ.get("SBD_TEST_DEVICE") or "default"
+    return f"sbd {sbd.__version__}: backends built {available}, testing {requested}"
+
+
+@pytest.fixture(scope="session")
+def backend():
+    """The SBD backend to test, honoring SBD_TEST_DEVICE if it is set.
+
+    A backend named in ``SBD_TEST_DEVICE`` that was not compiled into this build is
+    reported as a skip, since which backends exist depends on how the package was
+    built. That a backend is missing is a fact about the build; that ``sbd`` itself is
+    missing is a failure, and is left to raise.
+    """
+    import sbd
+
+    device = os.environ.get("SBD_TEST_DEVICE")
+    available = sbd.available_backends()
+    if device is not None and device not in available:
+        pytest.skip(f"backend {device!r} was not built; available: {available}")
+    return sbd.get_backend(device)

test/test_reference_energies.py

@@ -0,0 +1,139 @@
+# This code is a Qiskit project.
+#
+# (C) Copyright IBM 2026.
+#
+# This code is licensed under the Apache License, Version 2.0. You may
+# obtain a copy of this license in the LICENSE.txt file in the root directory
+# of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
+#
+# Any modifications or derivative works of this code must retain this
+# copyright notice, and modified files need to carry a notice indicating
+# that they have been altered from the originals.
+
+"""Check SBD against the reference energies published with the upstream test data.
+
+Each molecule directory under ``vendor/sbd-upstream/data`` carries a README with a
+table of determinant-selection thresholds and the electronic energy that a
+diagonalization over the corresponding determinants should produce. Those tables are
+the closest thing to ground truth available here: the determinants are fixed, so the
+answer is deterministic and independent of sampling, and the energies were obtained
+by filtering a full CI calculation.
+
+The cases are the rows of those tables. Only the cheapest of them run by default;
+the rest are marked ``slow`` because they take minutes to hours and, at the far end,
+more memory than a workstation has.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+import sbd
+
+# (molecule, alpha determinant file, expected electronic energy, is_slow)
+#
+# Transcribed from the tables in vendor/sbd-upstream/data/<molecule>/README.md. The
+# rows beyond the first of each molecule are marked slow: the determinant count grows
+# by roughly an order of magnitude per row, and compute time is reported upstream to
+# scale as (determinants)**1.23. The largest rows of each table are omitted entirely,
+# needing hundreds of gigabytes.
+REFERENCE_ENERGIES = [
+    # H2O, cc-pvdz, 24 orbitals, 10 electrons. FCI: -76.24377680
+    ("h2o", "h2o-1em3-alpha.txt", -76.23594663, False),
+    ("h2o", "h2o-1em4-alpha.txt", -76.24295848, True),
+    ("h2o", "h2o-1em5-alpha.txt", -76.24373504, True),
+    # N2, 6-31g, 18 orbitals, 14 electrons. FCI: -109.04874199
+    ("n2", "1em3-alpha.txt", -109.04162110, False),
+    ("n2", "3em4-alpha.txt", -109.04697304, True),
+    ("n2", "1em4-alpha.txt", -109.04835269, True),
+    ("n2", "3em5-alpha.txt", -109.04864315, True),
+    ("n2", "1em5-alpha.txt", -109.04871934, True),
+]
+
+
+def _case_id(case) -> str:
+    molecule, det_file, _, _ = case
+    # e.g. "h2o-1em3": the threshold is the informative part of the file name.
+    threshold = det_file.replace(f"{molecule}-", "").replace("-alpha.txt", "")
+    return f"{molecule}-{threshold}"
+
+
+# SBD packs a determinant into words of ``bit_length`` bits, and the resulting word
+# count is a process-wide constant: ``det_vector::_elem_size`` is an inline static that
+# throws "det_vector: elem_size mismatch" if a later diagonalization needs a different
+# one. Two molecules can therefore share a process only if they agree on it.
+#
+# The word count is ``ceil(2 * norb / bit_length)``, so a ``bit_length`` of 64 keeps it
+# at 1 for every reference molecule up to 32 orbitals: h2o (24), n2 (18) and nh3 (29).
+# The two larger ones, c2h2 (38) and c4h4 (44), would need 2 words and so cannot share
+# a process with these; adding them means a separate module, or forking per test.
+#
+# ``bit_length`` does not affect the result. Verified across 8, 20, 32, 48 and 64,
+# which span word counts 6 down to 1: the h2o energy was identical to ten digits.
+BIT_LENGTH = 64
+
+
+def _diagonalize(backend, fcidump, det_file, **overrides):
+    """Diagonalize over the determinants in ``det_file`` and return the energy."""
+    sbd_data = backend.TPB_SBD()
+    # A tolerance well below the precision the reference energies are quoted to, so
+    # that a disagreement means a wrong answer rather than an unconverged one.
+    sbd_data.eps = 1e-10
+    sbd_data.max_it = 200
+    sbd_data.bit_length = BIT_LENGTH
+    for name, value in overrides.items():
+        setattr(sbd_data, name, value)
+    results = sbd.tpb_diag_from_files(str(fcidump), str(det_file), sbd_data)
+    return results["energy"]
+
+
+@pytest.mark.parametrize(
+    "molecule,det_file,expected",
+    [
+        pytest.param(
+            molecule,
+            det_file,
+            expected,
+            marks=pytest.mark.slow if is_slow else (),
+            id=_case_id((molecule, det_file, expected, is_slow)),
+        )
+        for molecule, det_file, expected, is_slow in REFERENCE_ENERGIES
+    ],
+)
+def test_reference_energy(data_dir, backend, molecule, det_file, expected):
+    """The energy matches the value published for these determinants.
+
+    The reference energies are quoted to eight decimal places, so they are compared to
+    that precision rather than to the solver's own convergence tolerance.
+    """
+    molecule_dir = data_dir / molecule
+    energy = _diagonalize(
+        backend, molecule_dir / "fcidump.txt", molecule_dir / det_file
+    )
+    assert energy == pytest.approx(expected, abs=1e-8)
+
+
+@pytest.mark.mpi
+def test_energy_does_not_depend_on_process_count(data_dir, backend):
+    """Splitting the determinants across processes does not change the answer.
+
+    Run under ``mpirun``, this diagonalizes the same subspace over however many
+    processes were launched and compares against the published energy. A result that
+    depends on the process count would mean the distribution itself is wrong, which is
+    the failure this guards against; it is also why the comparison is against the
+    reference rather than against another run.
+    """
+    from mpi4py import MPI
+
+    comm = MPI.COMM_WORLD
+    molecule_dir = data_dir / "h2o"
+    energy = _diagonalize(
+        backend,
+        molecule_dir / "fcidump.txt",
+        molecule_dir / "h2o-1em3-alpha.txt",
+        adet_comm_size=comm.Get_size(),
+    )
+
+    # Only rank 0 receives the energy; the others are given a placeholder.
+    if comm.Get_rank() == 0:
+        assert energy == pytest.approx(-76.23594663, abs=1e-8)

tox.ini

@@ -0,0 +1,60 @@
+[tox]
+minversion = 4.4.3
+envlist = py{310,311,312,313,314}, mpi
+isolated_build = True
+
+[testenv]
+# Build a wheel once and install it into every environment, rather than the default
+# sdist which each environment would compile for itself. Compiling the extension is
+# the slowest part of a run, so doing it once matters.
+package = wheel
+wheel_build_env = .pkg
+# The extension modules are compiled against whatever MPI and BLAS the environment
+# provides, so the build cannot be isolated from it: pass the variables setup.py
+# consults through to the build and to the tests.
+passenv =
+    MPI_HOME
+    BLAS_LIB_PATH
+    BLAS_LIBS
+    NVHPC_HOME
+    SBD_GPU_ARCH
+    SBD_BUILD_BACKEND
+    SBD_TEST_DEVICE
+    SBD_TEST_NPROCS
+    CC
+    CXX
+    OMP_NUM_THREADS
+    PATH
+    LD_LIBRARY_PATH
+extras =
+    test
+commands =
+    pytest {posargs}
+
+[testenv:mpi]
+# Run the tests marked `mpi` under a launcher, and only those: the rest of the suite
+# is single-process, and running it under mpirun would just repeat each test once per
+# rank. `--only-mpi` selects them; note that it must be used *without* `--with-mpi`,
+# since pytest-mpi checks the two in an if/elif and the latter takes precedence,
+# which would quietly run everything.
+#
+# Two processes is enough to tell a genuinely distributed code path from one that only
+# works on a single rank, so that is the default. Override it to use more:
+#
+#     SBD_TEST_NPROCS=8 tox -e mpi
+#
+# Note that `mpirun` is not installed by tox, so the environment must already provide
+# it -- on a cluster that usually means loading a module first.
+allowlist_externals =
+    mpirun
+setenv =
+    SBD_TEST_NPROCS = {env:SBD_TEST_NPROCS:2}
+commands =
+    mpirun -n {env:SBD_TEST_NPROCS} pytest --only-mpi {posargs}
+
+[testenv:slow]
+# The reference cases grow by roughly an order of magnitude in determinant count per
+# row of the upstream tables, and compute time scales worse than linearly in that, so
+# these are opt-in rather than part of the default run.
+commands =
+    pytest --run-slow {posargs}