shader-slang · ccummingsNV · Jul 17, 2026 · Jul 17, 2026 · Jul 17, 2026 · Jul 17, 2026
@@ -15,8 +15,9 @@ SlangPy has a custom benchmark framework built on pytest. Benchmarks live in `sl
 | `slangpy/benchmarks/` | Benchmark test files (`test_benchmark_*.py`) and Slang shader files |
 | `slangpy/benchmarks/conftest.py` | Auto-imports benchmark fixtures and registers plugins |
 | `slangpy/testing/benchmark/fixtures.py` | Pytest fixtures: `BenchmarkSlangFunction`, `BenchmarkPythonFunction`, `BenchmarkComputeKernel`, `ReportFixture` |
-| `slangpy/testing/benchmark/plugin.py` | Pytest plugin adding `--benchmark-save`, `--benchmark-compare`, `--benchmark-upload` CLI options |
-| `slangpy/testing/benchmark/report.py` | `BenchmarkReport` / `Report` TypedDicts, serialization, MongoDB upload |
+| `slangpy/testing/benchmark/plugin.py` | Pytest plugin adding local report and authenticated BenchView submission options |
+| `slangpy/testing/benchmark/benchview.py` | Native BenchView payload construction, batching, and authenticated HTTP submission |
+| `slangpy/testing/benchmark/report.py` | Legacy local report serialization and comparison data |
 | `slangpy/testing/benchmark/table.py` | Terminal table display with color-coded deltas |
 | `slangpy/testing/benchmark/utils.py` | Machine/GPU/commit info collection, JSON datetime helpers |
 
@@ -90,8 +91,49 @@ pytest slangpy/benchmarks -v --benchmark-compare my_run
 
 # List saved runs
 pytest slangpy/benchmarks --benchmark-list-runs
+
+# Submit to a local BenchView server (PowerShell)
+$env:BENCHVIEW_API_KEY = "<write key>"
+pytest slangpy/benchmarks -v --benchmark-submit <request-id> --benchmark-api-url http://localhost:3000
+
+# Run the CI wrapper against the nested hosted deployment
+$env:BENCHVIEW_API_KEY = "<write key>"
+python tools/ci.py benchmark-python --run-id <request-id> --api-url http://rtrci.nvidia.com/benchview
+```
+
+`BENCHVIEW_API_URL` may supply the base URL when direct pytest commands omit `--benchmark-api-url` or the CI wrapper omits `--api-url`. The URL is the root or nested BenchView application base, not the full submission endpoint. The write key is accepted only through `BENCHVIEW_API_KEY`; it is never a command-line option or printed by the benchmark plugin. `--benchmark-upload` remains an alias for `--benchmark-submit` for existing direct pytest scripts, but it now uses only the HTTP API and never connects to MongoDB.
+
+The ordinary `.github/workflows/ci-benchmark.yml` workflow is manual and is also dispatched by the nightly scheduler. With no `revision` input it checks out and benchmarks the selected branch tip. With an exact 40-character `revision` it checks out that future-compatible commit while retaining the workflow branch name for BenchView. It always builds the selected source and invokes the same `tools/ci.py benchmark-python` command shown above on the Windows and Linux performance workers. Configure repository variable `BENCHVIEW_API_URL` with the root or nested BenchView base URL and repository secret `BENCHVIEW_API_KEY` with its write key before enabling the workflow.
+
+## Benchmark action scheduling
+
+The nightly `.github/workflows/schedule-benchmarks.yml` workflow runs once per day. It uses the authenticated GitHub API client supplied by `actions/github-script`, so it needs no checkout, Python environment, GitHub CLI, or extra secret. It examines one fixed 24-hour UTC interval and dispatches the ordinary workflow once for every `main` commit in that interval, oldest first, through the `revision` input. It deliberately does not inspect or suppress existing runs; manually triggering the scheduler repeats the complete 24-hour interval.
+
+The local historical backfill controller still uses the official GitHub CLI because it is a resumable operator process rather than a GitHub-hosted workflow. It never accepts a token argument. Verify its prerequisite before any preview or dispatch:
+
+```powershell
+gh --version
+gh auth status
+```
+
+Historical commits use the separate manual `.github/workflows/backfill-benchmark.yml` workflow. Its inclusive supported floor is `f3ad0fd91d8cf4eeb2be3b505765b43482aa952a` from 2 September 2025; older revisions are rejected before setup or build. Each matrix job uses ordinary Git commands to create and synchronize a unique normal recursive clone below the runner's temporary directory, builds the untouched historical checkout, and only then overlays the current `tools/ci.py`, `tools/gpu_clock.py`, and `slangpy/testing/benchmark/` reporting harness. Native PowerShell and Bash cleanup steps validate the resolved clone path before removing it. Submitted observations explicitly identify the historical SHA and branch `main`.
+
+Preview the supported inventory without creating state or dispatching:
+
+```powershell
+python tools/backfill_benchmarks.py --dry-run
+```
+
+After the boundary and later historical pilot workflows have passed, start or resume the bounded scheduler with:
+
+```powershell
+python tools/backfill_benchmarks.py
 ```
 
+The scheduler stores only commit and workflow-run state in `.temp/benchmark-backfill-state.json`. It publishes `dispatching` state before each request, records GitHub's returned run ID afterward, dispatches at most one oldest commit per minute, and never permits more than four active backfill workflows. Ctrl+C exits with code 130 after the latest atomic state replacement; running the same command again reconciles deterministic `backfill-benchmark: <SHA>` titles and continues without duplicating accepted requests. `--once` performs at most one scheduling iteration for a controlled trial.
+
+Never run two scheduler processes against the same state file. If the scheduler reports incompatible or corrupt state, leave it untouched, archive it manually, and rerun so deterministic GitHub titles can reconstruct already requested commits.
+
 ## Writing New Benchmarks
 
 1. Create a `test_benchmark_*.py` file in `slangpy/benchmarks/`.
@@ -118,3 +160,7 @@ Reports are JSON files stored in `.benchmarks/`. Each benchmark entry includes:
 - `cpu_time` — total wall-clock time including warmup
 
 The terminal summary table shows color-coded deltas when comparing: green for >5% improvement, red for >5% regression.
+
+Local report files retain this legacy shape. In parallel, fixtures accumulate native BenchView observations. Tests whose original pytest function name contains `_cpu` submit `cpu_time`; every other benchmark submits `gpu_time`. Both use milliseconds. This matches the existing imported history, including Python wrappers that synchronize GPU work. The stable test ID is the normalized source file plus the original pytest function, while pytest parameters keep their legacy string values as case dimensions and `DeviceType.cuda` is shortened to `cuda`. Project, commit, machine, OS, CPU, and GPU information use BenchView's dedicated run and environment fields.
+
+One benchmark process submits observations in API-sized batches. Independent device or machine processes at the same Git revision and build configuration derive the same logical run key. A new benchmark execution uses fresh observation timestamps and execution identity, so it replaces matching test cases normally; retrying an unchanged request body is idempotent. Transient connection and gateway failures use five total attempts with exponential backoff while preserving the exact payload and idempotency key.
@@ -0,0 +1,153 @@
+name: backfill-benchmark
+run-name: "backfill-benchmark: ${{ inputs.target_sha }}"
+
+on:
+  workflow_dispatch:
+    inputs:
+      target_sha:
+        description: "Exact historical main commit to benchmark"
+        required: true
+        type: string
+
+permissions:
+  contents: read
+  checks: write
+  id-token: write
+
+jobs:
+  build:
+    runs-on: ${{ matrix.runs-on }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [windows, linux]
+        config: [Release]
+        python: ["3.10"]
+        include:
+          - { os: windows, platform: x86_64, compiler: msvc, config: Release, flags: "benchmark", runs-on: { labels: [Windows, X64, nvrgfx-perf-kernelvm-bridge] } }
+          - { os: linux, platform: x86_64, compiler: gcc, config: Release, flags: "benchmark", runs-on: { labels: [Linux, X64, nvrgfx-perf-kernelvm-bridge] } }
+
+    env:
+      CI_OS: ${{ matrix.os }}
+      CI_PLATFORM: ${{ matrix.platform }}
+      CI_COMPILER: ${{ matrix.compiler }}
+      CI_CONFIG: ${{ matrix.config }}
+      CI_PYTHON: ${{ matrix.python }}
+      CI_FLAGS: ${{ matrix.flags }}
+      BACKFILL_TARGET_SHA: ${{ inputs.target_sha }}
+      BACKFILL_FLOOR_SHA: f3ad0fd91d8cf4eeb2be3b505765b43482aa952a
+      BACKFILL_CLONE_DIR: ${{ runner.temp }}/slangpy-backfill-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.os }}
+      BENCHVIEW_API_URL: ${{ vars.BENCHVIEW_API_URL }}
+      BENCHVIEW_API_KEY: ${{ secrets.BENCHVIEW_API_KEY }}
+      BENCHVIEW_BENCHMARK_REF: ${{ inputs.target_sha }}
+      BENCHVIEW_BENCHMARK_BRANCH: main
+
+    steps:
+      - name: Setup Python ${{ matrix.python }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Setup MSVC
+        uses: step-security/msvc-dev-cmd@v1
+
+      - name: Setup CMake/Ninja
+        uses: lukka/get-cmake@latest
+
+      - name: Clone historical source
+        run: |
+          git clone --recursive "https://github.com/${{ github.repository }}.git" "${{ env.BACKFILL_CLONE_DIR }}"
+          git -C "${{ env.BACKFILL_CLONE_DIR }}" checkout --detach "${{ inputs.target_sha }}"
+
+      - name: Validate supported history boundary
+        run: |
+          echo "Validating target ${{ inputs.target_sha }} against earliest supported commit ${{ env.BACKFILL_FLOOR_SHA }}"
+          git -C "${{ env.BACKFILL_CLONE_DIR }}" merge-base --is-ancestor "${{ env.BACKFILL_FLOOR_SHA }}" "${{ inputs.target_sha }}"
+
+      - name: Synchronize historical submodules and LFS
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: |
+          git submodule sync --recursive
+          git submodule update --init --recursive
+          git lfs pull
+
+      - name: Setup historical Python environment
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: |
+          python -m pip install -r requirements-dev.txt
+          python -m pip install -r samples/requirements.txt
+          python -m pip install pytest-github-actions-annotate-failures
+
+      - name: Setup PyTorch environment
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: |
+          python -m pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
+
+      - name: Historical setup
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: python tools/ci.py setup
+
+      - name: Historical configure
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: python tools/ci.py configure
+
+      - name: Historical build
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: python tools/ci.py build
+
+      - name: Overlay current BenchView benchmark harness
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: git checkout "${{ github.sha }}" -- tools/ci.py tools/gpu_clock.py slangpy/testing/benchmark
+
+      - name: Install slangpy-torch bridge when present
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: python tools/ci.py install-slangpy-torch
+
+      - name: Benchmark historical source (Windows)
+        if: runner.os == 'Windows'
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: python tools/ci.py benchmark-python --run-id "${{ github.run_id }}" --api-url "${{ env.BENCHVIEW_API_URL }}" --lock-gpu-clocks
+
+      - name: Benchmark historical source (Linux)
+        if: runner.os == 'Linux'
+        working-directory: ${{ env.BACKFILL_CLONE_DIR }}
+        run: python tools/ci.py benchmark-python --run-id "${{ github.run_id }}" --api-url "${{ env.BENCHVIEW_API_URL }}" --lock-gpu-clocks
+
+      - name: Uninstall slangpy-torch
+        if: always()
+        run: python -m pip uninstall slangpy-torch -y
+        continue-on-error: true
+
+      - name: Safely remove temporary clone (Windows)
+        if: always() && runner.os == 'Windows'
+        shell: pwsh
+        run: |
+          $candidate = [IO.Path]::GetFullPath($env:BACKFILL_CLONE_DIR).TrimEnd([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar)
+          $runnerTemp = [IO.Path]::GetFullPath($env:RUNNER_TEMP).TrimEnd([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar)
+          $parent = [IO.Directory]::GetParent($candidate)
+          $name = [IO.Path]::GetFileName($candidate)
+          if ($null -eq $parent -or -not [StringComparer]::OrdinalIgnoreCase.Equals($parent.FullName, $runnerTemp)) {
+            throw "Refusing to remove clone outside runner.temp: $candidate"
+          }
+          if (-not $name.StartsWith("slangpy-backfill-", [StringComparison]::Ordinal)) {
+            throw "Refusing to remove unexpected directory name: $name"
+          }
+          if (Test-Path -LiteralPath $candidate) {
+            Remove-Item -LiteralPath $candidate -Recurse -Force
+          }
+
+      - name: Safely remove temporary clone (Linux)
+        if: always() && runner.os == 'Linux'
+        shell: bash
+        run: |
+          candidate="$(realpath -m -- "$BACKFILL_CLONE_DIR")"
+          runner_temp="$(realpath -m -- "$RUNNER_TEMP")"
+          if [[ "$(dirname -- "$candidate")" != "$runner_temp" ]]; then
+            echo "Refusing to remove clone outside runner.temp: $candidate" >&2
+            exit 1
+          fi
+          if [[ "$(basename -- "$candidate")" != slangpy-backfill-* ]]; then
+            echo "Refusing to remove unexpected directory name: $(basename -- "$candidate")" >&2
+            exit 1
+          fi
+          rm -rf -- "$candidate"
@@ -1,9 +1,13 @@
 name: ci-benchmark
+run-name: "ci-benchmark: ${{ inputs.revision || github.sha }}"
 
 on:
-  schedule:
-    - cron: '0 */4 * * *' # run every 4 hours
   workflow_dispatch:
+    inputs:
+      revision:
+        description: "Optional exact revision to benchmark; defaults to the selected branch tip"
+        required: false
+        type: string
 
 permissions:
   contents: read
@@ -32,10 +36,15 @@ jobs:
       CI_CONFIG: ${{ matrix.config }}
       CI_PYTHON: ${{ matrix.python }}
       CI_FLAGS: ${{ matrix.flags }}
+      BENCHVIEW_API_URL: ${{ vars.BENCHVIEW_API_URL }}
+      BENCHVIEW_API_KEY: ${{ secrets.BENCHVIEW_API_KEY }}
+      BENCHVIEW_BENCHMARK_REF: ${{ inputs.revision || github.sha }}
+      BENCHVIEW_BENCHMARK_BRANCH: ${{ github.ref_name }}
 
     steps:
       - uses: actions/checkout@v6
         with:
+          ref: ${{ inputs.revision || github.sha }}
           submodules: recursive
           lfs: true
 
@@ -59,7 +68,7 @@ jobs:
 
       # Setup PyTorch environment
       - name: Setup PyTorch environment
-        if: runner.os != 'macos' && contains(matrix.flags, 'unit-test')
+        if: runner.os != 'macos' && contains(matrix.flags, 'benchmark')
         run: |
           python -m pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
 
@@ -108,11 +117,22 @@ jobs:
       - name: Build
         run: python tools/ci.py build
 
+      # Install slangpy-torch extension (requires the completed SlangPy build and PyTorch).
+      - name: Install slangpy-torch
+        if: runner.os != 'macos' && contains(matrix.flags, 'benchmark')
+        run: python tools/ci.py install-slangpy-torch
+
       # Benchmark (Python)
       - name: Benchmark (Python, Windows, GPU Clock Locked)
         if: contains(matrix.flags, 'benchmark') && runner.os == 'Windows'
-        run: python tools/ci.py benchmark-python --run-id "${{ github.run_id }}" --mongodb-connection-string "${{ secrets.BENCHMARK_MONGODB_CONNECTION_STRING }}" --mongodb-database-name "nvr-ci" --lock-gpu-clocks
+        run: python tools/ci.py benchmark-python --run-id "${{ github.run_id }}" --api-url "${{ env.BENCHVIEW_API_URL }}" --lock-gpu-clocks
 
-      - name: Benchmark (Python, Linux, GPU Clock Unlocked)
+      - name: Benchmark (Python, Linux, GPU Clock Locked)
         if: contains(matrix.flags, 'benchmark') && runner.os == 'Linux'
-        run: python tools/ci.py benchmark-python --run-id "${{ github.run_id }}" --mongodb-connection-string "${{ secrets.BENCHMARK_MONGODB_CONNECTION_STRING }}" --mongodb-database-name "nvr-ci"
+        run: python tools/ci.py benchmark-python --run-id "${{ github.run_id }}" --api-url "${{ env.BENCHVIEW_API_URL }}" --lock-gpu-clocks
+
+      # Cleanup slangpy-torch from persistent self-hosted runners.
+      - name: Uninstall slangpy-torch
+        if: always()
+        run: python -m pip uninstall slangpy-torch -y
+        continue-on-error: true
@@ -0,0 +1,52 @@
+name: schedule-benchmarks
+
+on:
+  schedule:
+    - cron: "0 2 * * *"
+  workflow_dispatch:
+
+concurrency:
+  group: schedule-benchmarks
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+  actions: write
+
+jobs:
+  schedule:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Dispatch recent main commits
+        uses: actions/github-script@v9
+        with:
+          script: |
+            const workflow = "ci-benchmark.yml";
+            const branch = "main";
+            const until = new Date();
+            const since = new Date(until.getTime() - 24 * 60 * 60 * 1000);
+            const common = {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+            };
+            const commits = await github.paginate(github.rest.repos.listCommits, {
+              ...common,
+              sha: branch,
+              since: since.toISOString(),
+              until: until.toISOString(),
+              per_page: 100,
+            });
+            core.info(
+              `Nightly interval ${since.toISOString()} through ${until.toISOString()}: ` +
+                `${commits.length} commit(s) to dispatch.`,
+            );
+            for (const commit of commits.reverse()) {
+              const revision = commit.sha;
+              await github.rest.actions.createWorkflowDispatch({
+                ...common,
+                workflow_id: workflow,
+                ref: branch,
+                inputs: { revision },
+              });
+              core.info(`Dispatched ${workflow} for ${revision} from ${branch}.`);
+            }