[proof-of-concept] benchmarks dashboard

.github/workflows/perf-dashboard.yml

@@ -0,0 +1,113 @@
+name: Perf dashboard
+
+# Runs the microbenchmarks on a self-hosted AMD GPU runner, appends the results
+# to the per-family CSV shards, and publishes the static dashboard to gh-pages.
+# Prerequisite: the `gh-pages` branch must already exist (seed it once with the
+# dashboard files + an empty data/ dir).
+#
+# Opt-in only -- the expensive GPU benchmarks do NOT fire on every push/PR. The
+# `bench` job runs when explicitly requested:
+#   * a PR carries the `ci-perf-test` label, or
+#   * a push's head commit title contains `[ci-perf-test]`, or
+#   * a manual `workflow_dispatch`.
+#
+# TODO(perf-dashboard): prune PR shards on close. PR runs write
+# perf-<family>-pr<N>.csv and currently accumulate indefinitely. Add a companion
+# job triggered on `pull_request: closed` that deletes perf-*-pr<N>.csv and
+# removes its rows from index.csv (e.g. a `dashboard_prune.py --pr <N>`), then
+# commits gh-pages -- so merged/closed PRs don't leave dead data behind.
+
+on:
+  # Triggers are intentionally broad; the `bench` job's `if:` is the opt-in gate.
+  push:
+    branches: [dev]
+    paths:
+      - 'transformer_engine/**'
+      - 'benchmarks/microbenchmarks/**'
+  pull_request:
+    # `labeled` so that adding the `ci-perf-test` label triggers a run; no path
+    # filter so the label works on any PR.
+    types: [opened, synchronize, reopened, labeled]
+  workflow_dispatch:
+
+permissions:
+  contents: write            # push commits to gh-pages
+
+# Serialize every run globally: they all push to the single mutable gh-pages
+# branch, so we must not push concurrently.
+concurrency:
+  group: perf-dashboard
+  cancel-in-progress: false
+
+jobs:
+  bench:
+    # Opt-in gate: only spend a GPU runner when a perf run was explicitly asked
+    # for (PR label, [ci-perf-test] commit tag, or manual dispatch). A skipped
+    # job consumes no runner.
+    if: >-
+      github.event_name == 'workflow_dispatch'
+      || (github.event_name == 'pull_request'
+          && contains(github.event.pull_request.labels.*.name, 'ci-perf-test'))
+      || (github.event_name == 'push'
+          && contains(github.event.head_commit.message, '[ci-perf-test]'))
+    runs-on: [self-hosted, linux, gfx950]      # your AMD GPU runner label(s)
+    # GitHub's hosted runners have no AMD GPU; a self-hosted runner is required.
+    # If TE isn't in the runner's base env, run the job inside your TE container:
+    # container:
+    #   image: <your-rocm-te-image>
+    #   options: --device=/dev/kfd --device=/dev/dri --group-add video --shm-size 16g
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Checkout published site (gh-pages)
+        uses: actions/checkout@v4
+        with:
+          ref: gh-pages
+          path: site
+
+      - name: Run microbenchmarks
+        working-directory: benchmarks/microbenchmarks
+        run: |
+          # Keep going if one benchmark fails; publish whatever produced a CSV.
+          for b in gemm gemm_fp8 casting normalization grouped_gemm; do
+            python "benchmark_${b}.py" --csv || echo "::warning::benchmark_${b} failed"
+          done
+
+      - name: Ingest into per-family CSV shards
+        working-directory: benchmarks/microbenchmarks
+        run: |
+          set -euo pipefail
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
+            python dashboard_ingest.py benchmark_*.csv \
+              --out-dir "$GITHUB_WORKSPACE/site/data" \
+              --pr "${{ github.event.number }}" \
+              --commit "${{ github.event.pull_request.head.sha }}" \
+              --arch gfx950 --runner "${{ runner.name }}"
+          else
+            python dashboard_ingest.py benchmark_*.csv \
+              --out-dir "$GITHUB_WORKSPACE/site/data" \
+              --ref dev \
+              --commit "$GITHUB_SHA" \
+              --arch gfx950 --runner "${{ runner.name }}"
+          fi
+
+      - name: Refresh the static front-end (keep data/ as-is)
+        run: |
+          rsync -a --exclude=data \
+            benchmarks/microbenchmarks/dashboard/ "$GITHUB_WORKSPACE/site/"
+
+      - name: Commit & push to gh-pages
+        working-directory: site
+        run: |
+          set -euo pipefail
+          git config user.name  "perf-bot"
+          git config user.email "perf-bot@users.noreply.github.com"
+          git add -A
+          if git diff --cached --quiet; then
+            echo "no changes to publish"; exit 0
+          fi
+          git commit -m "perf: ${GITHUB_SHA::8} (${{ github.event_name }}) $(date -u +%FT%TZ)"
+          # Serialized by `concurrency`, but rebase-then-push is cheap insurance.
+          git pull --rebase origin gh-pages || true
+          git push origin gh-pages

benchmarks/microbenchmarks/.gitignore

@@ -0,0 +1,10 @@
+# Generated microbenchmark result CSVs (wide format; ingested into the dashboard,
+# then discarded). Not source -- never commit these.
+benchmark_*.csv
+
+# Generated single-file bundle (dashboard/dist/dashboard.html embeds the CSV data).
+dashboard/dist/
+
+# Local gh-pages checkout used by dashboard_redeploy.sh / build_bundle.py.
+# It's a separate repo clone full of generated data -- keep it out of TE.
+te-dash/

benchmarks/microbenchmarks/build_bundle.py

@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+###############################################################################
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved.
+#
+# See LICENSE for license information.
+###############################################################################
+"""Bundle the dashboard into a single self-contained ``dashboard.html``.
+
+Inlines styles.css, Chart.js and app.js, and embeds each data shard as a
+``<script type="text/csv" data-file="...">`` block that app.js reads via
+``getText()`` instead of fetching (fetch is blocked on ``file://``). The result
+opens offline by double-click -- no server, no network -- so it can be shared as
+a single file (email/Teams/etc.).
+
+Usage:
+  build_bundle.py [--data-dir DIR] [--out FILE]
+
+Defaults: --data-dir dashboard/data   --out dashboard/dist/dashboard.html
+For the published demo snapshot, point at the gh-pages checkout:
+  build_bundle.py --data-dir te-dash/data
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+DASH = Path(__file__).resolve().parent / "dashboard"
+
+
+def _read(path):
+    return Path(path).read_text(encoding="utf-8")
+
+
+def _inline_safe(text, tag):
+    r"""Neutralize a literal ``</tag>`` inside inlined code so it can't close the
+    surrounding ``<script>``/``<style>`` early. ``<\/tag`` is equivalent in JS
+    (string/regex/comment) and harmless in CSS."""
+    return re.sub(r"</(" + tag + r")", lambda m: "<\\/" + m.group(1), text, flags=re.IGNORECASE)
+
+
+def _replace_once(text, old, new, label):
+    n = text.count(old)
+    if n != 1:
+        sys.exit(f"expected exactly one '{label}' marker in index.html, found {n}")
+    return text.replace(old, new)
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--data-dir", default=str(DASH / "data"),
+                        help="dir with index.csv + perf-*.csv (default: dashboard/data)")
+    parser.add_argument("--out", default=str(DASH / "dist" / "dashboard.html"),
+                        help="output HTML path (default: dashboard/dist/dashboard.html)")
+    args = parser.parse_args()
+
+    html = _read(DASH / "index.html")
+    css = _inline_safe(_read(DASH / "styles.css"), "style")
+    chart = _inline_safe(_read(DASH / "vendor" / "chart.umd.min.js"), "script")
+    app = _inline_safe(_read(DASH / "app.js"), "script")
+
+    data_dir = Path(args.data_dir)
+    if not (data_dir / "index.csv").exists():
+        sys.exit(f"no index.csv in {data_dir} -- ingest some data there first")
+    # index.csv first (the catalog), then every shard it can reference.
+    names = ["index.csv"] + sorted(p.name for p in data_dir.glob("perf-*.csv"))
+    blocks = []
+    for name in names:
+        text = _read(data_dir / name)
+        if "</script" in text.lower():
+            sys.exit(f"{name} contains '</script' which would break embedding; abort")
+        blocks.append(f'<script type="text/csv" data-file="{name}">\n{text}\n</script>')
+    data_blocks = "\n".join(blocks)
+
+    html = _replace_once(html, '<link rel="stylesheet" href="./styles.css">',
+                         f"<style>\n{css}\n</style>", "styles.css link")
+    # Data blocks + inline Chart.js go where the vendored <script> was, so they
+    # exist before app.js runs; app.js replaces its own external <script>.
+    html = _replace_once(html, '<script src="./vendor/chart.umd.min.js"></script>',
+                         f"{data_blocks}\n<script>\n{chart}\n</script>", "chart.js script")
+    html = _replace_once(html, '<script src="./app.js"></script>',
+                         f"<script>\n{app}\n</script>", "app.js script")
+
+    out = Path(args.out)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(html, encoding="utf-8")
+    print(f"wrote {out}  ({out.stat().st_size / 1024:.0f} KB, {len(names)} data files: {', '.join(names)})")
+
+
+if __name__ == "__main__":
+    main()

benchmarks/microbenchmarks/dashboard/README.md

@@ -0,0 +1,98 @@
+# TransformerEngine microbenchmark dashboard
+
+A small static dashboard that tracks TE microbenchmark performance over time and
+flags regressions using a **run-to-run noise band** (a drop counts only if it
+exceeds `max(3% gate, 2σ)` of the kernel's own run-to-run variation). It renders
+entirely client-side — no server or build step — from per-family CSV "shards".
+
+Adapted from the [ROCm/FlyDSL](https://github.com/ROCm/FlyDSL) CI dashboard
+(Apache-2.0); Chart.js is vendored under `vendor/` (MIT).
+
+## Layout
+
+```
+dashboard/
+  index.html            landing (Health), All Benchmarks, PR Check tabs
+  app.js                all logic (vanilla JS; no framework)
+  styles.css
+  vendor/chart.umd.min.js   Chart.js (trend charts)
+  data/                 generated CSV shards (git-ignored)
+    index.csv           catalog: file,family,ref,pr
+    perf-<family>-<ref>.csv   long-format rows, appended per run
+../dashboard_ingest.py    wide benchmark CSV -> per-family shards (stdlib only)
+../run_all_benchmarks.sh  run the suite (+ optional ingest / bundle)
+../dashboard_redeploy.sh  publish the front-end (+ optional data / bundle) to gh-pages
+../build_bundle.py        emit a single self-contained dashboard.html
+```
+
+Each shard row is `ts,commit,run_id,arch,runner,op,shape,dtype,metric,value,time_ms,pr`.
+Shards are **append-only**; every ingest call is one run (unique `run_id`).
+
+## Quickstart
+
+Run benchmarks and ingest one run (GPU + TE + torch required):
+
+```bash
+cd benchmarks/microbenchmarks
+./run_all_benchmarks.sh --ingest --out-dir dashboard/data       # one run
+./run_all_benchmarks.sh --ingest --out-dir dashboard/data --runs 5   # a fresh baseline (needs >=4)
+```
+
+The arch (`gfx942` / `gfx950` / `gfx1250`) is auto-detected via `rocminfo`.
+
+View it locally (no server dependency other than a static file server, because
+the front-end `fetch()`es the shards):
+
+```bash
+cd dashboard && python3 -m http.server 8000     # http://localhost:8000
+```
+
+## Share as a single file
+
+Bundle everything (front-end + Chart.js + the CSV data) into one offline HTML you
+can email/Teams:
+
+```bash
+python3 build_bundle.py --data-dir dashboard/data     # -> dashboard/dist/dashboard.html
+# or in one shot:
+./run_all_benchmarks.sh --ingest --out-dir dashboard/data --bundle
+```
+
+Open by double-click — no server, no network. (Some orgs quarantine `.html`
+attachments; zip it if needed.)
+
+## Publish to GitHub Pages (optional)
+
+`dashboard_redeploy.sh` copies the front-end into a **gh-pages checkout** you own
+and pushes it. Point it at your own Pages repo:
+
+```bash
+git clone -b gh-pages <your-gh-pages-repo-url> /tmp/te-dash
+export TE_DASH_DST=/tmp/te-dash
+# ingest directly into the checkout, then publish front-end + data (+ a bundle):
+./run_all_benchmarks.sh --ingest --out-dir "$TE_DASH_DST/data"
+./dashboard_redeploy.sh --bundle
+```
+
+> GitHub Pages sites are public even from a private repo (private Pages needs
+> Enterprise Cloud). For internal-only use, prefer the single-file bundle or a
+> local/internal static server instead.
+
+## CI (opt-in)
+
+`.github/workflows/perf-dashboard.yml` runs the suite on a self-hosted GPU runner
+and publishes, but only **on demand** — it skips normal pushes/PRs. Trigger it by:
+
+- adding the **`ci-perf-test`** label to a PR, or
+- a push whose head commit title contains **`[ci-perf-test]`**, or
+- a manual `workflow_dispatch`.
+
+PR runs ingest as `--pr <N>` (isolated from the `dev` baseline) and surface in the
+**PR Check** tab; dev runs build the baseline shown in **Health** / **Trends**.
+
+## Adding an arch
+
+Arch is data-driven from `CFG.archOrder` in `app.js`. To add one: append it to
+`archOrder`, add a `--gfx<arch>` color + `.arch-chip[data-arch="gfx<arch>"]` rule
+in `styles.css`, and (if it's a new alias) allow it in `dashboard_ingest.py`'s
+`ALLOWED_ARCHES`.