fix(healers): runtime flow-UUID discovery — commit pgm-heal + de-hardcode thumbs (R2)

docs/COMPARISON-switchframe.md

@@ -0,0 +1,64 @@
+# Comparison: our MXL switcher vs. thiagopnts/switchframe
+
+A reference comparison (Oct 2026). **SwitchFrame** (github.com/thiagopnts/switchframe) is an
+independent project with the **same north star as ours**: a browser-controlled, server-side
+live production switcher built on the EBU **MXL** shared-memory transport. It is far more
+built-out than our prototype, so it is a valuable **source of ideas** — not a competitor to
+adopt (2 stars / 0 forks: it's one person's deep solo engine, not a standard).
+
+## What SwitchFrame is
+- **Go 1.25 engine + Svelte 5 control UI**, single binary, WebTransport/QUIC (HTTP-3) control.
+- Sources decode to **YUV420 server-side**; all switching/mixing/keying/compositing happens on
+  the server; browsers are pure control surfaces (identical philosophy to our Core/Skin split).
+- **Same substrate as us:** MXL (V210 video / float32 audio shared memory), SRT in/out
+  (listener+caller), MPEG-TS recording. Plus MoQ ingest, ST 2110, multi-dest SRT.
+- Scale: ~3,000 commits, 291 REST endpoints, 59 GPU/SIMD kernels (Metal + CUDA), AI captions
+  (whisper.cpp → CEA-608), AI background removal (TensorRT RVM), 3D-LUT color grade, 8-layer
+  DSK, instant replay, multi-operator roles + Opus comms.
+
+## Where it is ahead of us, and why
+| Area | SwitchFrame | Us (this repo) |
+|---|---|---|
+| Core engine | one Go binary, lock-free hot path (<1µs/frame), **atomic graph swap** (zero-drop reconfig) | orchestrated containers: CBC Rust **input-selector** + Python ingests + gst pipelines |
+| **"grain too late" wedge** | reader **re-syncs to `head-2` and continues** (transient, not fatal) | CBC selector **wedges/dies** on it → our PTZ-over-relay bug |
+| GPU | Metal/CUDA, 59 kernels, actually used | GTX 970 idle; v210 has no HW path; CEF forces SwiftShader (upstream image) |
+| v210 pack/unpack | **SIMD** (`v210_10bit_kernels_{amd64,arm64}.s`) | gst `videoconvert` in software (~1 core/active fanout before our tsparse fix) |
+| Timing | dedicated `clock` + `stmap` modules, independent monotonic PTS per stream | hand-tuned restamp in `contribution_core.py`, fragile under relay jitter |
+
+## The single most transferable lesson (applied)
+SwitchFrame's MXL reader (`server/mxl/reader.go`):
+> *"On 'too late' errors, re-sync to the current write head instead of dying."*
+> *"Timeout and too-early are transient — don't count as fatal errors."*
+It jumps to `head-2` and continues; only ~50 **consecutive** fatals stop it.
+
+This is the exact cure for our recurring **PTZ selector wedge** (`read source grain N: Out of
+range - too late`). We hit it because our stack uses CBC's Rust input-selector, which **dies**
+on that error where SwitchFrame's own reader **re-syncs**. We can't patch CBC's Rust, so we
+applied the same principle at the layer we control: **`mxl-selfheal` now detects the wedge and
+re-cuts to a verified-advancing slot**, forcing the router to re-grab from the current head
+(commit `feat(selfheal): re-sync a selector wedged on a grain 'too late' error`). That's a
+watchdog-level re-sync vs. SwitchFrame's in-reader re-sync — same idea, our architecture.
+
+## Ideas worth borrowing next (ranked)
+1. **Re-sync reader (DONE at watchdog layer).** The "transient, not fatal; re-grab head" rule
+   is the biggest reliability win and is now in selfheal. If we ever own the selector, bake it
+   into the reader itself (the proper place) instead of healing after the fact.
+2. **Role-based discovery (already aligned).** Their `discovery.go` ≈ our new `tools/mxl-flows.sh`
+   (resolve flows by stable ROLE, not baked UUIDs). We arrived at the same design independently.
+3. **SIMD v210** (`v210_10bit_kernels`). Our fanout/conform is the CPU hog (profile: ~1 core
+   each). A SIMD v210 pack/unpack (or moving conform to a Go/Rust helper) would reclaim cores
+   that gst `videoconvert` burns — cheaper than the GPU path (v210 has no GPU HW path anyway).
+4. **Atomic graph swap** for zero-drop reconfig. Our re-wires interrupt PGM (why we built
+   never-interrupt + stable-flow pre-wiring). Their atomic swap is the clean general answer; our
+   stable-flow approach is the pragmatic version on a selector we don't own.
+5. **GPU, selectively** (deferred). Their 59 kernels need a Go/CUDA engine. For us the only
+   realistic near-term GPU win is NVENC on the mxl2webrtc x264 encode — a container rebuild,
+   lower priority than #3.
+
+## Strategic read
+SwitchFrame validates our architecture (MXL + server-side switch + browser control is the right
+shape) and shows the mature endpoint. But it's a from-scratch Go engine; our value is being the
+**adoptable, MXL-native, container-orchestrated** path that runs on commodity hardware a facility
+already has. Borrow their proven algorithms (re-sync reader, SIMD v210, atomic swap); don't chase
+feature parity with a 3,000-commit solo engine. Keep our edge: quickstart-able, portable, honest
+about what runs on a $400 box.

tests/test_thumb_slots.py

@@ -0,0 +1,88 @@
+"""mxl_thumbs.py discovers its multiview slots from the live domain (reviewer R2:
+no hardcoded, facility-specific UUIDs). These tests pin the discovery contract
+against a synthetic domain of flow_def.json files — no GStreamer, no threads.
+
+discover_slots() is imported from the probe module; the run loop is guarded under
+__main__, and Gst.init happens there, so importing for the test starts nothing.
+"""
+import importlib
+import json
+import sys
+from pathlib import Path
+
+import pytest
+
+REPO = Path(__file__).resolve().parent.parent
+TOOLS = REPO / "tools"
+
+pytest.importorskip("gi")  # the probe imports GStreamer bindings at module top
+
+
+@pytest.fixture(scope="module")
+def mod():
+    sys.path.insert(0, str(TOOLS))
+    try:
+        m = importlib.import_module("mxl_thumbs")
+    finally:
+        sys.path.pop(0)
+    return m
+
+
+def _flow(domain, uuid, fmt, grouphint=None, label=""):
+    d = domain / f"{uuid}.mxl-flow"
+    d.mkdir()
+    body = {"id": uuid, "format": f"urn:x-nmos:format:{fmt}", "label": label}
+    if grouphint:
+        body["tags"] = {"urn:x-nmos:tag:grouphint/v1.0": [grouphint]}
+    (d / "flow_def.json").write_text(json.dumps(body))
+
+
+def test_discovers_video_sources_by_role(mod, tmp_path):
+    _flow(tmp_path, "d3e15194-6d1f-5955-8d52-52e7076b7a99", "video", "Pattern:Video", "Pattern Video")
+    _flow(tmp_path, "9998da48-9041-5ae1-b730-d0a918563107", "video", "Playout:Video", "Clip Video")
+    _flow(tmp_path, "9e111e00-aaaa-4bbb-8ccc-000000000001", "video", "Guest1:Video", "Guest 1")
+    slots = mod.discover_slots(str(tmp_path))
+    assert slots == {
+        "pattern": "d3e15194-6d1f-5955-8d52-52e7076b7a99",
+        "playout": "9998da48-9041-5ae1-b730-d0a918563107",
+        "guest1": "9e111e00-aaaa-4bbb-8ccc-000000000001",
+    }
+
+
+def test_audio_flows_are_skipped(mod, tmp_path):
+    _flow(tmp_path, "d3e15194-6d1f-5955-8d52-52e7076b7a99", "video", "Pattern:Video")
+    _flow(tmp_path, "7c1dd9ba-ca2c-5737-889b-abdd23c06020", "audio", "Pattern:Audio")
+    slots = mod.discover_slots(str(tmp_path))
+    assert list(slots) == ["pattern"]  # the audio flow produced no tile
+
+
+def test_program_outputs_excluded(mod, tmp_path):
+    """selector/keyer are the program bus, not selectable source tiles."""
+    _flow(tmp_path, "5a617f19-7c48-598c-9e94-4611c6316273", "video", "Selector:Video", "Selector PGM")
+    _flow(tmp_path, "86efffa4-9b4e-558c-8816-7e200e20cf17", "video", "Keyer:Video", "Keyer PGM")
+    _flow(tmp_path, "9e333e00-aaaa-4bbb-8ccc-000000000001", "video", "Guest3:Video", "Guest 3")
+    slots = mod.discover_slots(str(tmp_path))
+    assert list(slots) == ["guest3"]
+
+
+def test_stable_collapses_to_friendly_guest_name(mod, tmp_path):
+    """A never-interrupt slot's GuestNStable flow shows under the friendly 'guestN'."""
+    _flow(tmp_path, "57ab5e00-aaaa-4bbb-8ccc-000000000001", "video", "Guest5Stable:Video", "Guest 5 Stable")
+    slots = mod.discover_slots(str(tmp_path))
+    assert slots == {"guest5": "57ab5e00-aaaa-4bbb-8ccc-000000000001"}
+
+
+def test_env_force_overrides_discovery(mod, tmp_path):
+    forced = "cam=aaa,guest1=bbb"
+    slots = mod.discover_slots(str(tmp_path), forced=forced)
+    assert slots == {"cam": "aaa", "guest1": "bbb"}
+
+
+def test_missing_domain_is_empty_not_error(mod, tmp_path):
+    assert mod.discover_slots(str(tmp_path / "does-not-exist")) == {}
+
+
+def test_untagged_flow_falls_back_to_uuid_prefix(mod, tmp_path):
+    _flow(tmp_path, "abcd1234-6d1f-5955-8d52-52e7076b7a99", "video")  # no grouphint
+    slots = mod.discover_slots(str(tmp_path))
+    assert slots == {"abcd1234": "abcd1234-6d1f-5955-8d52-52e7076b7a99"}

tools/mxl-flows.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env sh
+# mxl-flows.sh — runtime flow-UUID discovery for the MXL switcher healers.
+#
+# WHY: pgm-heal and selfheal used to carry HARDCODED flow UUIDs ("G1=9e111e00…",
+# "PATTERN=d3e15194…"). Those UUIDs are box-specific — they are whatever the
+# quickstart happened to mint on THIS facility — so a script full of them cannot
+# be committed to the shared repo (reviewer issue #2): on any other facility the
+# UUIDs are wrong, the selector gets wired to flows that do not exist, and the
+# down-selector rebuild wires only pattern/playout while 400ing on the phantom
+# guest UUIDs (HW Oct 10).
+#
+# The facility is already self-describing. `mxl-info -d DOMAIN -l` prints, per
+# device group, a stable ROLE name and the flows under it:
+#
+#     Guest1: mxl:///mxl-domain?id=9e111e00-…
+#         Video : 9e111e00-… - Guest 1
+#     Pattern: mxl:///mxl-domain?id=d3e15194-…&id=7c1dd9ba-…
+#         Video : d3e15194-… - Pattern Video
+#         Audio : 7c1dd9ba-… - Pattern Audio
+#
+# The group names (Pattern, Playout, Guest1..GuestN, Guest5Stable, Keyer,
+# Selector) are the role identity; the UUID under each is discovered at runtime.
+# So the healers name ROLES, never UUIDs, and work on any facility.
+#
+# Sourced, not executed. POSIX sh. Requires: DOMAIN set, DOCKER set (the caller's
+# `docker` or `sudo docker`), SEL_CTR set (the selector container name).
+
+# mxl_flows_dump — cache a single `mxl-info -l` snapshot in MXL_FLOWS_CACHE so
+# repeated lookups in one heal pass cost one docker exec, not one per role.
+# Call mxl_flows_refresh at the top of each pass to invalidate it.
+MXL_FLOWS_CACHE=""
+mxl_flows_refresh(){ MXL_FLOWS_CACHE=""; }
+mxl_flows_dump(){
+  [ -n "$MXL_FLOWS_CACHE" ] && { printf '%s' "$MXL_FLOWS_CACHE"; return 0; }
+  MXL_FLOWS_CACHE=$($DOCKER exec "$SEL_CTR" sh -c "/opt/mxl/tools/mxl-info/mxl-info -d $DOMAIN -l" 2>/dev/null)
+  [ -n "$MXL_FLOWS_CACHE" ] || return 1
+  printf '%s' "$MXL_FLOWS_CACHE"
+}
+
+# flow_by_role ROLE [MEDIA]  — UUID of the flow for a device group ($1, e.g.
+# "Guest1", "Pattern", "Guest5Stable") and media kind ($2, Video|Audio, default
+# Video). Empty string + rc1 if that role/media is not present in the domain.
+#
+#   The group header line is "ROLE: mxl://…"; the flow lines under it are
+#   "<ws>Video : <uuid> - <label>". We locate the group block, then the first
+#   matching media line within it. Portable awk — no bashisms.
+flow_by_role(){
+  _role="$1"; _media="${2:-Video}"
+  mxl_flows_dump | awk -v role="$_role" -v media="$_media" '
+    $0 ~ "^"role":" { inblk=1; next }
+    /^[A-Za-z0-9]+:/  { inblk=0 }           # next group header ends the block
+    inblk && $1==media && $2==":" { print $3; exit }
+  '
+}
+
+# flow_present ROLE [MEDIA]  — true if the role/media flow exists right now.
+flow_present(){ [ -n "$(flow_by_role "$1" "${2:-Video}")" ]; }
+
+# all_video_roles — every device group that currently offers a Video flow, one
+# role per line, in mxl-info order. Lets a healer enumerate "whatever sources
+# exist" instead of hardcoding a slot list.
+all_video_roles(){
+  mxl_flows_dump | awk '
+    /^[A-Za-z0-9]+:/ { role=$1; sub(":","",role); next }
+    $1=="Video" && $2==":" && role!="" { print role; role="" }
+  '
+}