ci: compare metrics with TrackEval and deploy

[mikel-brostrom]

Summary

• add a direct parity test between py-motmetrics and TrackEval 1.3.0
• compare per-sequence and combined HOTA, CLEAR, and Identity results on the bundled TUD sequences
• cover every standard HOTA alpha threshold
• add a dedicated TrackEval parity CI job and document the check

Why

The existing HOTA tests assert copied TrackEval golden values, but they do not execute the reference implementation. Running both implementations over the same bundled data makes metric or aggregation drift visible in CI.

Impact

The normal Python test matrix remains unchanged. TrackEval is installed only in the dedicated Python 3.11 parity job, and the comparison test is skipped when that optional test dependency is absent.

Testing

• uv run --no-project pytest -q motmetrics/tests/test_trackeval_parity.py — 1 passed
• uv run --no-project pytest -q motmetrics/tests — 64 passed
• uv run --no-project ruff check motmetrics/tests/test_trackeval_parity.py — passed
• git diff --check — passed

[mikel-brostrom]

Parity pipeline results

Run uv run --no-project pytest -q -s motmetrics/tests/test_trackeval_parity.py
  uv run --no-project pytest -q -s motmetrics/tests/test_trackeval_parity.py
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.15/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.15/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.15/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.15/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.15/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.15/x64/lib
    UV_PYTHON_INSTALL_DIR: /home/runner/work/_temp/uv-python-dir
    UV_CACHE_DIR: /home/runner/work/_temp/setup-uv-cache

TrackEval parity (absolute tolerance: 1e-06)
HOTA, DetA, and AssA values are alpha means; their difference is the maximum over all alphas.
Dataset        | Metric | py-motmetrics  | TrackEval      | max abs diff | Status
---------------+--------+----------------+----------------+--------------+-------
TUD-Campus     | HOTA   | 0.391397437845 | 0.391397437845 | 0.000e+00    | PASS  
TUD-Campus     | DetA   | 0.418047030143 | 0.418047030143 | 0.000e+00    | PASS  
TUD-Campus     | AssA   | 0.369120681208 | 0.369120681208 | 0.000e+00    | PASS  
TUD-Campus     | MOTA   | 0.526462395543 | 0.526462395543 | 0.000e+00    | PASS  
TUD-Campus     | MOTP   | 0.722798915361 | 0.722798915361 | 3.331e-16    | PASS  
TUD-Campus     | CLR_Re | 0.58217270195  | 0.58217270195  | 0.000e+00    | PASS  
TUD-Campus     | CLR_Pr | 0.941441441441 | 0.941441441441 | 0.000e+00    | PASS  
TUD-Campus     | CLR_TP | 209            | 209            | 0.000e+00    | PASS  
TUD-Campus     | CLR_FN | 150            | 150            | 0.000e+00    | PASS  
TUD-Campus     | CLR_FP | 13             | 13             | 0.000e+00    | PASS  
TUD-Campus     | IDSW   | 7              | 7              | 0.000e+00    | PASS  
TUD-Campus     | MT     | 1              | 1              | 0.000e+00    | PASS  
TUD-Campus     | PT     | 6              | 6              | 0.000e+00    | PASS  
TUD-Campus     | ML     | 1              | 1              | 0.000e+00    | PASS  
TUD-Campus     | Frag   | 7              | 7              | 0.000e+00    | PASS  
TUD-Campus     | IDF1   | 0.557659208262 | 0.557659208262 | 0.000e+00    | PASS  
TUD-Campus     | IDR    | 0.451253481894 | 0.451253481894 | 0.000e+00    | PASS  
TUD-Campus     | IDP    | 0.72972972973  | 0.72972972973  | 0.000e+00    | PASS  
TUD-Campus     | IDTP   | 162            | 162            | 0.000e+00    | PASS  
TUD-Campus     | IDFN   | 197            | 197            | 0.000e+00    | PASS  
TUD-Campus     | IDFP   | 60             | 60             | 0.000e+00    | PASS  
TUD-Stadtmitte | HOTA   | 0.397849016993 | 0.397849016993 | 0.000e+00    | PASS  
TUD-Stadtmitte | DetA   | 0.392267572369 | 0.392267572369 | 0.000e+00    | PASS  
TUD-Stadtmitte | AssA   | 0.408840751811 | 0.408840751811 | 0.000e+00    | PASS  
TUD-Stadtmitte | MOTA   | 0.56401384083  | 0.56401384083  | 0.000e+00    | PASS  
TUD-Stadtmitte | MOTP   | 0.654095704456 | 0.654095704456 | 3.331e-16    | PASS  
TUD-Stadtmitte | CLR_Re | 0.608996539792 | 0.608996539792 | 0.000e+00    | PASS  
TUD-Stadtmitte | CLR_Pr | 0.939919893191 | 0.939919893191 | 0.000e+00    | PASS  
TUD-Stadtmitte | CLR_TP | 704            | 704            | 0.000e+00    | PASS  
TUD-Stadtmitte | CLR_FN | 452            | 452            | 0.000e+00    | PASS  
TUD-Stadtmitte | CLR_FP | 45             | 45             | 0.000e+00    | PASS  
TUD-Stadtmitte | IDSW   | 7              | 7              | 0.000e+00    | PASS  
TUD-Stadtmitte | MT     | 5              | 5              | 0.000e+00    | PASS  
TUD-Stadtmitte | PT     | 4              | 4              | 0.000e+00    | PASS  
TUD-Stadtmitte | ML     | 1              | 1              | 0.000e+00    | PASS  
TUD-Stadtmitte | Frag   | 6              | 6              | 0.000e+00    | PASS  
TUD-Stadtmitte | IDF1   | 0.644619422572 | 0.644619422572 | 0.000e+00    | PASS  
TUD-Stadtmitte | IDR    | 0.531141868512 | 0.531141868512 | 0.000e+00    | PASS  
TUD-Stadtmitte | IDP    | 0.819759679573 | 0.819759679573 | 0.000e+00    | PASS  
TUD-Stadtmitte | IDTP   | 614            | 614            | 0.000e+00    | PASS  
TUD-Stadtmitte | IDFN   | 542            | 542            | 0.000e+00    | PASS  
TUD-Stadtmitte | IDFP   | 135            | 135            | 0.000e+00    | PASS  
OVERALL        | HOTA   | 0.399957091288 | 0.399957091288 | 0.000e+00    | PASS  
OVERALL        | DetA   | 0.397683291242 | 0.397683291242 | 0.000e+00    | PASS  
OVERALL        | AssA   | 0.412449529845 | 0.412449529845 | 0.000e+00    | PASS  
OVERALL        | MOTA   | 0.555115511551 | 0.555115511551 | 0.000e+00    | PASS  
OVERALL        | MOTP   | 0.669822945506 | 0.669822945506 | 2.220e-16    | PASS  
OVERALL        | CLR_Re | 0.602640264026 | 0.602640264026 | 0.000e+00    | PASS  
OVERALL        | CLR_Pr | 0.940267765191 | 0.940267765191 | 0.000e+00    | PASS  
OVERALL        | CLR_TP | 913            | 913            | 0.000e+00    | PASS  
OVERALL        | CLR_FN | 602            | 602            | 0.000e+00    | PASS  
OVERALL        | CLR_FP | 58             | 58             | 0.000e+00    | PASS  
OVERALL        | IDSW   | 14             | 14             | 0.000e+00    | PASS  
OVERALL        | MT     | 6              | 6              | 0.000e+00    | PASS  
OVERALL        | PT     | 10             | 10             | 0.000e+00    | PASS  
OVERALL        | ML     | 2              | 2              | 0.000e+00    | PASS  
OVERALL        | Frag   | 13             | 13             | 0.000e+00    | PASS  
OVERALL        | IDF1   | 0.624296057924 | 0.624296057924 | 0.000e+00    | PASS  
OVERALL        | IDR    | 0.512211221122 | 0.512211221122 | 0.000e+00    | PASS  
OVERALL        | IDP    | 0.799176107106 | 0.799176107106 | 0.000e+00    | PASS  
OVERALL        | IDTP   | 776            | 776            | 0.000e+00    | PASS  
OVERALL        | IDFN   | 739            | 739            | 0.000e+00    | PASS  
OVERALL        | IDFP   | 195            | 195            | 0.000e+00    | PASS  
.
1 passed in 7.00s

[mikel-brostrom]

I changed the release pipeline. For a release containing HOTA you will need to configure some stuff on your side, but no worries, just follow:

https://github.com/mikel-brostrom/py-motmetrics/blob/fb1703110388dbcfff41801476e5f46e9f3af194/Release.md

When everything is set up and we have a new release we just delete this file as well 🚀 . Also I cannot try the release pipeline as I am not owner of the repo. So there might be some problems that I am unaware of. Notice that this branch have to be merged for you to try this functionality.