Detection: crossing performers swap identity, and re-acquisition guesses

[Pahegi]

Problem

The tracker binds detections to tracklets using geometry only – IoU against the
Kalman-predicted box plus a centre-distance rescue (video/tracking.py:303-364).
Nothing in the pipeline looks at what a person looks like. When two performers
cross, their boxes overlap and geometry alone cannot say which detection belongs to
which track. That produces two failures, and the worse one is silent.

1. ID swap inside the association. Both tracks stay alive; the greedy assignment
   at video/tracking.py:355-360 hands each track the wrong detection. _pinned_id
   is still "valid", so tracked_detection never enters its re-acquire branch. The
   marker quietly starts following the other performer. Nothing logs, nothing errors.
   Every other failure in this project fails closed; this one fails open.
2. Re-acquisition guesses. When a track does die, tracked_detection
   (video/detection.py:717-731) takes the detection nearest the last-followed
   centre within _REACQUIRE_MAX_CENTER_DIST = 0.15. At a crossing both performers
   are inside that gate. When neither is, it falls back to max(results, key=area) –
   the largest box, which is the person nearest the camera, not the one being
   followed. A performer who walks behind a truss for two seconds and reappears
   across the frame is currently unrecoverable.

Goal

Give each track a persistent appearance descriptor and consult it in both places,
without ever letting appearance override geometry. Stage lighting is adversarial
for appearance matching in ways surveillance re-ID benchmarks do not cover: colour
washes flatten everyone to the wash colour, backlight silhouettes them, and an
ensemble in identical costumes defeats any descriptor. A design where appearance can
admit or reject a match would regress shows that work today.

Proposed approach

Admission stays purely geometric. Appearance only re-ranks, and only within a
bounded margin. Two descriptors ship: a colour histogram (cv2 only, always
available) and an optional ONNX re-ID embedding.

Association (_associate, video/tracking.py:303-364). Split the current
score into iou_td (admission, unchanged) and rank (sorting). Rank becomes
iou_td + weight * (sim - 0.5) with weight = 0.15 and sim calibrated to [0,1].
Pairs still sort (rank, -centre_distance); the greedy loop is untouched. Feeding
the blend into the admission test instead would be a real bug: same-scene cosine sits
at 0.7-0.99, so the constant offset makes iou_td >= gate unconditionally true and
_HIGH_IOU_GATE silently stops existing.

A similarity matrix pre-filled with the neutral 0.5 wherever either side lacks a
descriptor makes the bonus exactly zero, so no descriptor means behaviour
bit-identical to today – no model, first frame, crop under the size floor, low-band
box. That is the load-bearing safety property and it gets its own regression test.

One narrow exception: a pair admitted only by the distance rescue (IoU 0) is
vetoed below sim = 0.25. That is the case video/tracking.py:45-47 already worries
about, and the downside is bounded – the track stays lost one more frame and keeps
gliding on the Kalman prediction, rather than spawning a new id. Keeping the veto
floor below the 0.5 neutral is what lets one matrix serve both the re-rank and the
veto with no validity mask.

Rejected: appearance as a general veto (a wrong reject sends the detection to the
spawn-a-new-track branch at video/tracking.py:295, costing an ID break plus a
spurious track – worse than today's swap); appearance as a third sort key (-dist is
continuous and never ties, so it is unreachable).

Stage 2 gets no appearance. _LOW_IOU_GATE = 0.5 is already near-locked, and the
low band is the dim/occluded band that produces the worst descriptors in the system.

Re-acquisition (video/detection.py:677-736). Keep the pinned track's vector.
Inside the existing gate, rank by proximity plus appearance instead of proximity
alone. Then replace the largest-box fallback: when a stored vector exists, take the
best appearance match above a strict floor, and only otherwise fall back to largest.
This is the biggest user-visible win and the riskiest element in the design – a false
positive re-pins the followspot onto the wrong performer – so it runs only when
today's alternative is already arbitrary, and the whole feature is one switch away
from off.

Cost. The histogram is cheap enough to run every frame on the top max_persons
high-band boxes. The ONNX descriptor is not (2-4 ms per crop on a Pi, in a 67 ms
budget), so it runs on a slow refresh cadence plus whenever the frame is ambiguous:
two high boxes overlapping, or any track currently lost. Both tests are geometric and
run before any crop is taken.

Descriptors read the pre-CLAHE frame. _preprocess (video/detection.py:1096)
equalises the LAB L channel per tile and round-trips through RGB, which shifts the
same person's hue histogram as unrelated content enters frame. It is also further
from an ONNX model's training distribution. Capture the frame before that line.

Implementation checklist

• openfollow/video/appearance.py (new) – AppearanceDescriptor Protocol
  (batched describe(frame, boxes), returning L2-normalised rows and None per
  unusable crop); ColorHistogramDescriptor (HSV, three horizontal bands, hue
  weighted by saturation, L1 per sub-histogram then L2 overall); cosine_matrix;
  calibrate; _crop_for_box with a minimum-size floor.
• openfollow/video/tracking.py – STrack.feature seeded in __init__ and
  EMA-folded in update (never while lost: update is only reached on a match);
  _associate gains an appearance keyword, the similarity matrix, the re-rank and
  the rescue veto; stage 1 passes it, stage 2 does not.
• openfollow/video/detection.py – DetectionBox.descriptor; pre-CLAHE frame
  capture; _track(raw, frame=None) so the default reproduces today; describe
  budget (high band, mask-filtered, top max_persons, above the size floor);
  ambiguity predicate as a pure helper; track feature copied onto output boxes;
  _pinned_feature and the re-acquire fusion; a _describer slot with its own
  lock, mirroring _load_backend's publish-only-on-success and close-off-lock
  discipline (video/detection.py:506-556); rebuild arm in
  _drain_pending_config (video/detection.py:858).
• openfollow/video/detection.py – OnnxReidDescriptor, reusing
  _select_providers (video/detection.py:250-263) via a _make_ort_session
  helper extracted from _OnnxBackend.__init__, with load-time input/output rank
  validation so a detection model dropped in the wrong folder fails loudly.
• openfollow/configuration.py – reid_mode (_coerce_choice, off/color/
  model, default off) and reid_model (_coerce_str). Every threshold and the
  weight stay module constants: they have no operator meaning.
• openfollow/web/ – parser-identical twins in routes.py:2196 and
  validation.py:568 (reuse _validate_model); the picker in the Advanced models
  block of partials/detection.tpl; context key in both _render_detection
  (routes.py:5360) and the index render (routes.py:4103); help/detection.md.
• Model storage – <storage>/models/reid/. This needs no discovery
  changes: _discover_storage_models (routes.py:526) filters on p.is_file()
  and seed_bundled_models (model_seed.py:39) globs non-recursively, so a
  subdirectory is already invisible to the detection picker.
• scripts/export_reid_onnx.py + a torchreid entry in the workstation-only
  export extra (pyproject.toml:54) and scripts/install-detection.sh:294. The
  existing export_onnx.py is YOLO(...).export(...) and the route hardcodes the
  .onnx to .pt derivation (routes.py:5493), so this needs a sibling script
  and a branch in _build_export_argv (routes.py:697).
• Observability – reid_mode, reid_available, reid_describe_avg_ms,
  reid_describe_count (its ratio to inference_count is the duty cycle) and
  reid_vetoes in performance_stats; matching keys in the zero-fallback dict in
  services.py; rows in partials/statistics.tpl:123+; track_id in the overlay
  box label (runtime/overlay_draw_scene.py:209), which is the only way to watch
  an ID switch happen on device.
• scripts/hw_validation/reid_descriptor_probe.py + README row – cost mode
  (ms per crop on the DUT, which validates or kills the always-on/gated split),
  separability mode (emits the calibration constants from labelled footage rather
  than guessing them), and a CLAHE ablation.

Suggested phasing, each independently revertable: (1) descriptor module + gallery,
inert; (2) re-acquire fusion; (3) association fusion; (4) ONNX tier + download;
(5) observability, probe, and the default flip once there are on-device numbers.

Tests (required – every change ships with tests)

• tests/test_appearance.py (new, pure): crop clipping at each edge, degenerate
  and undersized crops; dim and L2 norm; same-colour beats different-colour;
  red-top/blue-bottom vs blue-top/red-bottom scores low (proves the band split
  earns its place); zero-band L1 guard; cosine_matrix neutral fill on missing
  rows/cols; calibrate below/above/mid/degenerate.
• tests/test_tracking.py: feature seeded and EMA-folded; untouched while lost;
  re-rank flips a crossing at IoU 0.42 vs 0.40; re-rank cannot flip 0.8 vs 0.2
  with inverted appearance (pins the bound); identical descriptors reproduce
  today; no descriptors reproduce today bit-for-bit; rescue veto fires on an
  IoU-0 pair, not on an IoU-cleared one, not on an unknown similarity; stage 2
  recovers despite a mismatched descriptor.
• tests/test_detection*.py: frame=None keeps today's path; describe budget
  caps at max_persons from 60 raw boxes; the describer receives the pre-CLAHE
  frame (via the existing _FakeClahe); ambiguity predicate parametrised over
  overlap / lost / cadence; _load_describer and the config-drain rebuild across
  success, failure-keeps-prior, and no-change; re-acquire prefers the matching
  farther candidate, and the appearance fallback fires only above its floor.
• tests/test_configuration.py: reid_mode wrong type / unknown choice / None
  to "off"; reid_model non-str to "".
• tests/test_validation.py / tests/test_web_detection_structure.py: rule
  bounds match __post_init__; a reid_mode=bogus POST coerces; traversal
  rejected.

cv2 is monkeypatched via the existing _FakeCv2 seam rather than importorskip,
extended with a real pure-NumPy RGB to HSV so the colour assertions are not vacuous.
Two docs/COVERAGE.md pragma rows for the Protocol method bodies, matching the
existing _InferenceBackend.predict row.

Out of scope (follow-up)

• Assist mode (runtime/services_detection_pin.py:390). The operator's manual
  anchor is a better identity signal than any descriptor, and appearance would fight
  the documented retarget-by-nudging gesture. Assist still benefits for free: it
  re-seeds its whole smoothing chain on every track-id change, so a more stable
  tracker means fewer visible discontinuities. No code.
• An operator-facing appearance-weight control. Ship the constant; add a knob only
  if a venue demonstrates the need.
• Identical costumes and heavy colour washes. No descriptor solves these. The
  bonus cancels and behaviour degrades to today's, which is the correct failure mode.
• Hungarian assignment. Greedy plus a bounded perturbation agrees with optimal at
  N,M under 10, and SciPy is barred by the offline-runtime contract. Revisit only if
  the probe measures real divergence.

Hardware note

Crops come from the detection appsink at input_resolution, 640x480 at defaults, so
a distant performer is roughly 30x80 px and carries little identity signal. Re-ID
quality is bounded by inference_size; a higher-resolution crop tap is a separate
pipeline change. Calibration constants must be measured on real stage footage with
the probe, not guessed. The multi-person occlusion case cannot be reproduced on the
bench.

Verification

• poetry run pytest on the new and touched suites.
• make ci-remote (Pi-first, make ci fallback) before merge.
• Manual: two people crossing on a live feed with show_labels on, watching the
  track_id in the box label. Confirm ids survive the crossing, and that
  reid_mode = "off" restores today's behaviour exactly.