From 9e6249f612bf9f3fb9e95b4d149cbee158b9097f Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 13:11:25 -0400 Subject: [PATCH 01/10] Add the performance dashboard page to the documentation The page is the mbirjax original, updated for mbirtorch: the dashboard is shared, and mbirtorch's measurements appear on the cpu-torch and gpu-torch platforms beside mbirjax's, so the two can be compared. The operator list is the one the torch nightly records. This restores the toctree entry and the Developer Docs card link, both of which were held out with PENDING(dashboard) markers. Co-Authored-By: Claude Fable 5 --- docs/source/dev_performance_dashboard.rst | 45 +++++++++++++++++++++++ docs/source/index.rst | 9 +---- 2 files changed, 47 insertions(+), 7 deletions(-) create mode 100644 docs/source/dev_performance_dashboard.rst diff --git a/docs/source/dev_performance_dashboard.rst b/docs/source/dev_performance_dashboard.rst new file mode 100644 index 0000000..5e5994c --- /dev/null +++ b/docs/source/dev_performance_dashboard.rst @@ -0,0 +1,45 @@ +Performance Dashboard +===================== + +MBIRTorch's reconstruction performance — run time, peak memory, and **correctness** — is tracked +automatically over time, on both CPU and GPU, by a companion project, +`mbirjax_metrics `__. When a tracked branch changes, a +scheduled job re-measures it and publishes an interactive dashboard: + +**Live dashboard:** https://gbuzzard.github.io/mbirjax_metrics/ + +MBIRTorch and MBIRJAX share this dashboard: MBIRTorch's measurements appear on the ``cpu-torch`` +and ``gpu-torch`` platforms next to MBIRJAX's, so the two implementations can be compared directly. +The dashboard rebuilds and republishes automatically whenever new measurements are pushed, so that +link is always current; you do not need to run anything to read it. + +What it measures +---------------- + +For each tracked branch, the job runs MBIRTorch's reconstruction operators — the +direct-reconstruction filter, forward projection, back projection, and the iterative VCD +reconstruction — across a range of problem sizes and device counts, on both CPU and GPU. For every +configuration it records: + +- **run time** (the minimum over repeated trials), +- **peak memory**, and +- a numeric **fingerprint** of the output, used to detect correctness changes. + +How to read it +-------------- + +The dashboard explains itself: open the live page and expand the **"How to read this dashboard"** +panel at the top. It walks through the tiles, the red correctness banner, the History and Scaling +views, and the colors & marks. + +That reading guide is authored *inside* the dashboard and ships in the page itself, so it can never +drift from the UI it describes — which is why this page links to it rather than duplicating it. + +Running it yourself +------------------- + +The dashboard is a single self-contained page generated from a YAML time series; no server is needed. +The measurement engine, the nightly harness, and the build script all live in the +`mbirjax_metrics `__ repository — see its ``README`` and +the ``action_scripts/`` and ``tooling/`` guides there for how runs are measured, gated, and scheduled, +and how to build the dashboard locally. diff --git a/docs/source/index.rst b/docs/source/index.rst index df4df4d..ec23d2a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -45,9 +45,6 @@ MBIRTorch: High-performance tomographic reconstruction Based on PyTorch_, MBIRTorch can easily run on CPU or GPU. -.. PENDING(dashboard): point the Developer Docs card below back at - dev_performance_dashboard when that page lands (held at Greg's request). - .. grid:: 3 .. grid-item-card:: :material-regular:`rocket_launch;2em` Getting Started @@ -65,7 +62,7 @@ MBIRTorch: High-performance tomographic reconstruction .. grid-item-card:: :material-regular:`laptop_chromebook;2em` Developer Docs :class-card: developer-docs :columns: 12 6 6 4 - :link: dev_sharding_overview + :link: dev_performance_dashboard :link-type: doc @@ -96,13 +93,11 @@ MBIRTorch: High-performance tomographic reconstruction :maxdepth: 4 :caption: Developer Guide + dev_performance_dashboard dev_sharding_overview dev_projector_kernels dev_api dev_maintenance -.. PENDING(dashboard): restore dev_performance_dashboard to the toctree above - when that page lands (held at Greg's request). - .. _PyTorch: https://pytorch.org/docs/stable/index.html From e4c99bf6c7538b109ae6c5b126b43855220202d2 Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 13:36:09 -0400 Subject: [PATCH 02/10] Start CUDA before any test fakes the device count torch checks every device in range(device_count()) at the first CUDA use. Tests that exercise the device-selection policy fake device_count, so a first CUDA use inside one of them asked for a device that does not exist. The failure surfaced in an unrelated test later in the same worker: on a two-GPU node it turned the Triton kernels off, failing the four kernel-times-sharding tests. Co-Authored-By: Claude Fable 5 --- tests/conftest.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/conftest.py b/tests/conftest.py index 654cfec..7110b93 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -3,6 +3,13 @@ import pytest import torch +# Start CUDA before any test runs. torch checks every device in +# range(device_count()) at the first CUDA use, and tests below fake +# device_count, so a first use inside one of them asks for a device that does +# not exist and fails. +if torch.cuda.is_available(): + torch.zeros(1, device="cuda") + def available_devices(): devices = ["cpu"] From 2ae5b97d8d1f8b031f38f3343b94ec1969bfb468 Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 13:59:35 -0400 Subject: [PATCH 03/10] Update maintenance docs --- docs/source/dev_maintenance.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/dev_maintenance.rst b/docs/source/dev_maintenance.rst index 94f78d1..3fb561b 100644 --- a/docs/source/dev_maintenance.rst +++ b/docs/source/dev_maintenance.rst @@ -23,11 +23,11 @@ Releasing a New Version This is only available for registered maintainers. -1. Update ``__version__`` in ``mbirtorch/__init__.py`` and merge to - ``prerelease``. This is the only place the version number is written. +1. Update the version number by editing the value of ``__version__`` to be ``X.Y.Z`` in ``mbirtorch/__init__.py``. +Commit this to the ``prerelease`` branch. -2. On GitHub, draft a new release: tag ``vX.Y.ZrcN``, target ``prerelease``, - check "Set as a pre-release", and publish. This uploads to TestPyPI. +2. On GitHub, draft a new release: tag ``vX.Y.ZrcN`` in the ``prerelease`` branch. +Check "Set as a pre-release", and publish. This will automatically upload the release to TestPyPI. 3. Check the TestPyPI upload:: From a18c56a9d7171eeb9c8c2e593fee562e4adb3479 Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 14:06:55 -0400 Subject: [PATCH 04/10] Add the release workflow Publishing a GitHub release now uploads the package: a pre-release goes to TestPyPI, a full release goes to PyPI after approval of the pypi environment. Uploads use Trusted Publishing, and the tag must equal v + __version__. The release instructions now carry a worked example (0.2.0), with the rc versions in __version__ so the tag check and TestPyPI versioning work. Co-Authored-By: Claude Fable 5 --- .github/workflows/release.yml | 66 +++++++++++++++++++++++++++++++++ docs/source/dev_maintenance.rst | 24 +++++++----- 2 files changed, 81 insertions(+), 9 deletions(-) create mode 100644 .github/workflows/release.yml diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..1074ad4 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,66 @@ +name: Release +# Publishing a GitHub release uploads the package: a pre-release goes to +# TestPyPI, a full release goes to PyPI after approval of the `pypi` +# environment. Releasing a New Version in dev_maintenance.rst is the +# step-by-step procedure. + +on: + release: + types: [published] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Check that the tag matches the package version + run: | + TAG="${{ github.event.release.tag_name }}" + VERSION=$(python -c "import re; print(re.search(r'__version__\s*=\s*[\"\x27]([^\"\x27]+)', open('mbirtorch/__init__.py').read()).group(1))") + if [ "$TAG" != "v$VERSION" ]; then + echo "tag $TAG does not match __version__ $VERSION (expected v$VERSION)" + exit 1 + fi + - name: Build the sdist and wheel + run: | + pip install build + python -m build + - uses: actions/upload-artifact@v4 + with: + name: dist + path: dist/ + + publish-testpypi: + # Pre-releases (tag vX.Y.ZrcN from prerelease) go to TestPyPI. + if: github.event.release.prerelease + needs: build + runs-on: ubuntu-latest + environment: testpypi + permissions: + id-token: write # Trusted Publishing; no token is stored. + steps: + - uses: actions/download-artifact@v4 + with: + name: dist + path: dist/ + - uses: pypa/gh-action-pypi-publish@release/v1 + with: + repository-url: https://test.pypi.org/legacy/ + + publish-pypi: + # Full releases (tag vX.Y.Z on main) go to PyPI after approval. + if: "!github.event.release.prerelease" + needs: build + runs-on: ubuntu-latest + environment: pypi + permissions: + id-token: write # Trusted Publishing; no token is stored. + steps: + - uses: actions/download-artifact@v4 + with: + name: dist + path: dist/ + - uses: pypa/gh-action-pypi-publish@release/v1 diff --git a/docs/source/dev_maintenance.rst b/docs/source/dev_maintenance.rst index 3fb561b..e1399e2 100644 --- a/docs/source/dev_maintenance.rst +++ b/docs/source/dev_maintenance.rst @@ -23,26 +23,32 @@ Releasing a New Version This is only available for registered maintainers. -1. Update the version number by editing the value of ``__version__`` to be ``X.Y.Z`` in ``mbirtorch/__init__.py``. -Commit this to the ``prerelease`` branch. +The tag must always equal ``v`` + ``__version__``; the upload fails otherwise. +The example below releases version 0.2.0. -2. On GitHub, draft a new release: tag ``vX.Y.ZrcN`` in the ``prerelease`` branch. -Check "Set as a pre-release", and publish. This will automatically upload the release to TestPyPI. +1. In ``mbirtorch/__init__.py``, set ``__version__ = "0.2.0rc1"``. + Commit this to the ``prerelease`` branch. + +2. On GitHub, draft a new release: tag ``v0.2.0rc1``, target ``prerelease``. + Check "Set as a pre-release", and publish. This uploads to TestPyPI. 3. Check the TestPyPI upload:: - dev_scripts/check_published_wheel.sh --testpypi --version X.Y.ZrcN + dev_scripts/check_published_wheel.sh --testpypi --version 0.2.0rc1 + + If it fails, fix the problem, set ``__version__ = "0.2.0rc2"``, and repeat + from step 2. -4. Open a pull request from ``prerelease`` to ``main`` and merge it when the - checks pass. +4. Set ``__version__ = "0.2.0"`` and commit to ``prerelease``. Open a pull + request from ``prerelease`` to ``main`` and merge it when the checks pass. -5. Draft a new release: tag ``vX.Y.Z``, target ``main``, and publish. Then +5. Draft a new release: tag ``v0.2.0``, target ``main``, and publish. Then approve the ``pypi`` environment on the workflow run page. This uploads to PyPI. 6. Check the PyPI upload:: - dev_scripts/check_published_wheel.sh --version X.Y.Z + dev_scripts/check_published_wheel.sh --version 0.2.0 The documentation rebuilds automatically: ``latest`` follows ``main``, and ``stable`` follows the highest release tag. From a89636a3477c5d0bf8ec882c587e6b90170b9afa Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 14:15:26 -0400 Subject: [PATCH 05/10] Update test.pypi version number --- mbirtorch/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mbirtorch/__init__.py b/mbirtorch/__init__.py index 6b2f8b9..b227d32 100644 --- a/mbirtorch/__init__.py +++ b/mbirtorch/__init__.py @@ -5,7 +5,7 @@ kept for API compatibility; here it means "return the device tensor"). """ -__version__ = "0.0.1" +__version__ = "0.0.1rc1" # ── persistent torch.compile cache ──────────────────────────────────────────── # The inductor cache directory defaults to /tmp/torchinductor_, which the From 34e6c94054d7f795c204af15bffa1a9ba1428509 Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 14:45:13 -0400 Subject: [PATCH 06/10] Split the release instructions into TestPyPI and PyPI stages Co-Authored-By: Claude Fable 5 --- docs/source/dev_maintenance.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/source/dev_maintenance.rst b/docs/source/dev_maintenance.rst index e1399e2..0722617 100644 --- a/docs/source/dev_maintenance.rst +++ b/docs/source/dev_maintenance.rst @@ -26,6 +26,9 @@ This is only available for registered maintainers. The tag must always equal ``v`` + ``__version__``; the upload fails otherwise. The example below releases version 0.2.0. +Releasing to TestPyPI ++++++++++++++++++++++ + 1. In ``mbirtorch/__init__.py``, set ``__version__ = "0.2.0rc1"``. Commit this to the ``prerelease`` branch. @@ -39,6 +42,9 @@ The example below releases version 0.2.0. If it fails, fix the problem, set ``__version__ = "0.2.0rc2"``, and repeat from step 2. +Releasing to PyPI ++++++++++++++++++ + 4. Set ``__version__ = "0.2.0"`` and commit to ``prerelease``. Open a pull request from ``prerelease`` to ``main`` and merge it when the checks pass. From cdfac8a17405400955ccae4647b09d0b50204aa2 Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 14:48:10 -0400 Subject: [PATCH 07/10] Update pypi version number --- mbirtorch/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mbirtorch/__init__.py b/mbirtorch/__init__.py index b227d32..6b2f8b9 100644 --- a/mbirtorch/__init__.py +++ b/mbirtorch/__init__.py @@ -5,7 +5,7 @@ kept for API compatibility; here it means "return the device tensor"). """ -__version__ = "0.0.1rc1" +__version__ = "0.0.1" # ── persistent torch.compile cache ──────────────────────────────────────────── # The inductor cache directory defaults to /tmp/torchinductor_, which the From 11e317d443778b7edb96965a60b8cc1b6d4ebd5e Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 15:33:39 -0400 Subject: [PATCH 08/10] Add release.sh: one command per release stage release.sh X.Y.ZrcN publishes a release candidate to TestPyPI; release.sh X.Y.Z opens the pull request to main; release.sh X.Y.Z --publish creates the release on main after the merge. The instructions now use the script, with the manual procedure kept as a note. Co-Authored-By: Claude Fable 5 --- dev_scripts/release.sh | 59 +++++++++++++++++++++++++++++++++ docs/source/dev_maintenance.rst | 42 +++++++++++++---------- 2 files changed, 84 insertions(+), 17 deletions(-) create mode 100755 dev_scripts/release.sh diff --git a/dev_scripts/release.sh b/dev_scripts/release.sh new file mode 100755 index 0000000..3f48e1c --- /dev/null +++ b/dev_scripts/release.sh @@ -0,0 +1,59 @@ +#!/bin/bash +# Run one stage of the release procedure in dev_maintenance.rst. +# +# dev_scripts/release.sh 0.2.0rc1 # rc: publish a pre-release to TestPyPI +# dev_scripts/release.sh 0.2.0 # final: open the PR from prerelease to main +# dev_scripts/release.sh 0.2.0 --publish # after the PR merges: publish to PyPI +# +# Requires the gh CLI, logged in. Uploads still need approval of the pypi +# environment on the workflow run page. +set -euo pipefail + +cd "$(dirname "$0")/.." +VERSION="${1:?usage: release.sh X.Y.Z[rcN] [--publish]}" +PUBLISH="${2:-}" +INIT=mbirtorch/__init__.py + +case "$VERSION" in + *rc*) STAGE=rc ;; + *) STAGE=final ;; +esac +if [[ "$PUBLISH" == "--publish" && "$STAGE" == "rc" ]]; then + echo "--publish is for a final version; an rc publishes on its own" >&2 + exit 2 +fi + +if [[ "$PUBLISH" == "--publish" ]]; then + # The tag must point at main, so main must already carry this version. + git fetch -q origin main + if ! git show origin/main:$INIT | grep -q "__version__ = \"$VERSION\""; then + echo "main does not have __version__ = \"$VERSION\"; merge the PR first" >&2 + exit 1 + fi + gh release create "v$VERSION" --target main --title "MBIRTorch v$VERSION" \ + --generate-notes + echo "Release v$VERSION created. Approve the pypi environment on the" + echo "workflow run page, then check with:" + echo " dev_scripts/check_published_wheel.sh --version $VERSION" + exit 0 +fi + +git checkout -q prerelease +git pull -q origin prerelease +sed -i '' "s/^__version__ = \".*\"/__version__ = \"$VERSION\"/" $INIT +grep -q "__version__ = \"$VERSION\"" $INIT +git add $INIT +git commit -q -m "Set version to $VERSION" +git push -q origin prerelease + +if [[ "$STAGE" == "rc" ]]; then + gh release create "v$VERSION" --target prerelease --prerelease \ + --title "MBIRTorch v$VERSION" --generate-notes + echo "Pre-release v$VERSION created; TestPyPI upload is running. Check with:" + echo " dev_scripts/check_published_wheel.sh --testpypi --version $VERSION" +else + gh pr create --base main --head prerelease --title "Release $VERSION" \ + --body "Merges prerelease into main for the $VERSION release." + echo "Merge the PR when the checks pass, then run:" + echo " dev_scripts/release.sh $VERSION --publish" +fi diff --git a/docs/source/dev_maintenance.rst b/docs/source/dev_maintenance.rst index 0722617..c813ad5 100644 --- a/docs/source/dev_maintenance.rst +++ b/docs/source/dev_maintenance.rst @@ -21,41 +21,49 @@ The same tests run automatically on every push and pull request. Releasing a New Version ----------------------- -This is only available for registered maintainers. - -The tag must always equal ``v`` + ``__version__``; the upload fails otherwise. -The example below releases version 0.2.0. +This is only available for registered maintainers. It requires the ``gh`` +command, logged in to GitHub. The example below releases version 0.2.0. Releasing to TestPyPI +++++++++++++++++++++ -1. In ``mbirtorch/__init__.py``, set ``__version__ = "0.2.0rc1"``. - Commit this to the ``prerelease`` branch. +1. Publish a release candidate to TestPyPI:: -2. On GitHub, draft a new release: tag ``v0.2.0rc1``, target ``prerelease``. - Check "Set as a pre-release", and publish. This uploads to TestPyPI. + dev_scripts/release.sh 0.2.0rc1 -3. Check the TestPyPI upload:: +2. Check the TestPyPI upload:: dev_scripts/check_published_wheel.sh --testpypi --version 0.2.0rc1 - If it fails, fix the problem, set ``__version__ = "0.2.0rc2"``, and repeat - from step 2. + If it fails, fix the problem and repeat from step 1 with ``0.2.0rc2``. Releasing to PyPI +++++++++++++++++ -4. Set ``__version__ = "0.2.0"`` and commit to ``prerelease``. Open a pull - request from ``prerelease`` to ``main`` and merge it when the checks pass. +3. Open the release pull request:: + + dev_scripts/release.sh 0.2.0 + + Merge it on GitHub when the checks pass. -5. Draft a new release: tag ``v0.2.0``, target ``main``, and publish. Then - approve the ``pypi`` environment on the workflow run page. This uploads - to PyPI. +4. Publish the release:: -6. Check the PyPI upload:: + dev_scripts/release.sh 0.2.0 --publish + + Then approve the ``pypi`` environment on the workflow run page. This + uploads to PyPI. + +5. Check the PyPI upload:: dev_scripts/check_published_wheel.sh --version 0.2.0 +Each ``release.sh`` stage sets ``__version__`` in ``mbirtorch/__init__.py``, +commits, and creates the matching ``v``-prefixed tag; the upload fails if the +tag and ``__version__`` ever disagree. The manual procedure behind the +script: edit ``__version__``, commit to ``prerelease``, and draft a GitHub +release with tag ``v`` + ``__version__`` — target ``prerelease`` with "Set as +a pre-release" checked for an rc, target ``main`` for a final version. + The documentation rebuilds automatically: ``latest`` follows ``main``, and ``stable`` follows the highest release tag. From 2fea0cc1106bbb710fcf19672b6033306a226e98 Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 15:37:57 -0400 Subject: [PATCH 09/10] Upgrade pip before the RTD torch install RTD seeds its build environment with pip 23.1, which rejects the new typing_extensions wheel over the underscore spelling in its metadata, falls back to the sdist, and fails: the build dependencies are not on the torch index. Co-Authored-By: Claude Fable 5 --- .readthedocs.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.readthedocs.yaml b/.readthedocs.yaml index ee0f9cf..05b1798 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -17,6 +17,11 @@ build: # from PyPI, which on Linux is the CUDA build -- several GB of nvidia-* wheels # that exceed RTD's build disk and time limits. Autodoc only needs to import # the package, so the CPU wheel is sufficient. + # + # RTD seeds the environment with an old pip that rejects wheels whose + # metadata spells the name with an underscore (typing_extensions broke + # the 2026-08-12 build), so upgrade pip first. + - pip install --upgrade pip - pip install torch --index-url https://download.pytorch.org/whl/cpu # Build the Sphinx docs. From ccf9a8f08659149834b12763d386e719f57ab121 Mon Sep 17 00:00:00 2001 From: Charles Bouman Date: Wed, 12 Aug 2026 15:41:04 -0400 Subject: [PATCH 10/10] Set version to 0.0.1rc2 --- mbirtorch/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mbirtorch/__init__.py b/mbirtorch/__init__.py index 6b2f8b9..f19783a 100644 --- a/mbirtorch/__init__.py +++ b/mbirtorch/__init__.py @@ -5,7 +5,7 @@ kept for API compatibility; here it means "return the device tensor"). """ -__version__ = "0.0.1" +__version__ = "0.0.1rc2" # ── persistent torch.compile cache ──────────────────────────────────────────── # The inductor cache directory defaults to /tmp/torchinductor_, which the