Skip to content

perf: speed up metric calculation#213

Draft
mikel-brostrom wants to merge 12 commits into
cheind:developfrom
mikel-brostrom:perf/metric-calculation-speedups
Draft

perf: speed up metric calculation#213
mikel-brostrom wants to merge 12 commits into
cheind:developfrom
mikel-brostrom:perf/metric-calculation-speedups

Conversation

@mikel-brostrom

@mikel-brostrom mikel-brostrom commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a reusable PreparedSequence representation shared by CLEAR, Identity, and HOTA
  • prepare sequences with one array extraction per DataFrame and frame-boundary slicing instead of per-frame pandas groups
  • compute CLEAR bookkeeping with dense identity-code arrays
  • compute all HOTA alpha thresholds directly from compact NumPy state
  • add a finite-cost fast path to the SciPy LAP backend while preserving solver selection and missing-edge behavior
  • add typed SequenceResult, BatchResult, and HOTAResult outputs through evaluate_sequence / evaluate_many
  • preserve the legacy MOTAccumulator API with lazy event replay for CLEAR and HOTA
  • vectorize legacy Identity occurrence and pair counting
  • default the compact evaluator to serial execution because thread contention regresses small and medium workloads; explicit n_jobs remains available and is capped by sequence count
  • split TrackEval CI timing into end-to-end and prepared metric-kernel measurements
  • document the batch API and add compact/legacy/parallel/unsorted-input equivalence tests

This PR is stacked on cheind/py-motmetrics#212 (agent/add-trackeval-parity-ci). It should be reviewed and merged after #212; once #212 lands in develop, GitHub will reduce this PR to the performance commits.

Root cause

The original path repeatedly constructed pandas frame groups and converted each group to NumPy, independently prepared CLEAR and HOTA inputs, updated generic event accumulators for every HOTA threshold, and later materialized/scanned their event tables. Small LAP matrices also paid more for finite-edge sanitization and result filtering than for SciPy's assignment itself.

The optimized evaluator extracts each DataFrame once, slices frames by sorted array boundaries, shares similarities across metric families, uses dense identity codes for CLEAR state, and aggregates directly into compact count matrices. Detailed event DataFrames are replayed only when a legacy caller accesses .events; tests compare replayed tables exactly with eager MOTAccumulator output.

Profiling also showed that the compact Python frame loops contend on the GIL: on the bundled workload two threads were slower than serial, and an eight-sequence replicated workload was roughly twice as slow with eight threads. The evaluator therefore runs serially by default and keeps threading opt-in.

Performance

On the bundled TUD-Campus and TUD-Stadtmitte parity workload (three warmed local runs):

  • end-to-end: py-motmetrics 0.018340 s vs TrackEval 0.123189 s — py-motmetrics was 6.72x faster
  • prepared metric kernel: py-motmetrics 0.011383 s vs TrackEval 0.029706 s — py-motmetrics was 2.61x faster

The completed GitHub-hosted Ubuntu runner reports:

  • end-to-end: py-motmetrics 0.041533 s vs TrackEval 0.314130 s — py-motmetrics was 7.56x faster
  • prepared metric kernel: py-motmetrics 0.026130 s vs TrackEval 0.063304 s — py-motmetrics was 2.42x faster

The CI job prints both timing groups in the log and GitHub job summary. Timing remains informational; correctness is the gate.

Correctness

Every per-sequence and combined HOTA, CLEAR, and Identity value matches TrackEval within the required 1e-6 tolerance. The latest local run's largest absolute difference was 2.220e-16.

Cached compact statistics are used only with the library's standard metric implementations. Custom metrics and mutable event access fall back to the normal event-based path.

Testing

  • .venv/bin/pytest -q — 80 passed
  • .venv/bin/pytest -q -s motmetrics/tests/test_trackeval_parity.py — exact parity, 1 passed
  • .venv/bin/ruff check --select E9,F63,F7,F82 motmetrics — passed
  • uv build --no-sources — wheel and source distribution built
  • wheel installed into a clean temporary venv; motmetrics 1.4.0 and evaluate_many imported successfully

@mikel-brostrom

mikel-brostrom commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author
End-to-end execution time (3 warmed runs; lower is better)
Evaluator     | Run 1     | Run 2     | Run 3     | Median   
--------------+-----------+-----------+-----------+----------
py-motmetrics | 0.042877s | 0.041533s | 0.041332s | 0.041533s
TrackEval     | 0.314251s | 0.314130s | 0.311473s | 0.314130s
py-motmetrics was 7.56x faster than TrackEval

Metric kernel execution time (3 warmed runs; lower is better)
Evaluator     | Run 1     | Run 2     | Run 3     | Median   
--------------+-----------+-----------+-----------+----------
py-motmetrics | 0.026234s | 0.026130s | 0.025840s | 0.026130s
TrackEval     | 0.063521s | 0.063304s | 0.063187s | 0.063304s
py-motmetrics was 2.42x faster than TrackEval
.
1 passed in 4.66s

@mikel-brostrom mikel-brostrom force-pushed the perf/metric-calculation-speedups branch from affba59 to b5db891 Compare July 14, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant