perf: speed up metric calculation#213
Draft
mikel-brostrom wants to merge 12 commits into
Draft
Conversation
Contributor
Author
End-to-end execution time (3 warmed runs; lower is better)
Evaluator | Run 1 | Run 2 | Run 3 | Median
--------------+-----------+-----------+-----------+----------
py-motmetrics | 0.042877s | 0.041533s | 0.041332s | 0.041533s
TrackEval | 0.314251s | 0.314130s | 0.311473s | 0.314130s
py-motmetrics was 7.56x faster than TrackEval
Metric kernel execution time (3 warmed runs; lower is better)
Evaluator | Run 1 | Run 2 | Run 3 | Median
--------------+-----------+-----------+-----------+----------
py-motmetrics | 0.026234s | 0.026130s | 0.025840s | 0.026130s
TrackEval | 0.063521s | 0.063304s | 0.063187s | 0.063304s
py-motmetrics was 2.42x faster than TrackEval
.
1 passed in 4.66s |
affba59 to
b5db891
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PreparedSequencerepresentation shared by CLEAR, Identity, and HOTASequenceResult,BatchResult, andHOTAResultoutputs throughevaluate_sequence/evaluate_manyMOTAccumulatorAPI with lazy event replay for CLEAR and HOTAn_jobsremains available and is capped by sequence countThis PR is stacked on cheind/py-motmetrics#212 (
agent/add-trackeval-parity-ci). It should be reviewed and merged after #212; once #212 lands indevelop, GitHub will reduce this PR to the performance commits.Root cause
The original path repeatedly constructed pandas frame groups and converted each group to NumPy, independently prepared CLEAR and HOTA inputs, updated generic event accumulators for every HOTA threshold, and later materialized/scanned their event tables. Small LAP matrices also paid more for finite-edge sanitization and result filtering than for SciPy's assignment itself.
The optimized evaluator extracts each DataFrame once, slices frames by sorted array boundaries, shares similarities across metric families, uses dense identity codes for CLEAR state, and aggregates directly into compact count matrices. Detailed event DataFrames are replayed only when a legacy caller accesses
.events; tests compare replayed tables exactly with eagerMOTAccumulatoroutput.Profiling also showed that the compact Python frame loops contend on the GIL: on the bundled workload two threads were slower than serial, and an eight-sequence replicated workload was roughly twice as slow with eight threads. The evaluator therefore runs serially by default and keeps threading opt-in.
Performance
On the bundled TUD-Campus and TUD-Stadtmitte parity workload (three warmed local runs):
0.018340 svs TrackEval0.123189 s— py-motmetrics was 6.72x faster0.011383 svs TrackEval0.029706 s— py-motmetrics was 2.61x fasterThe completed GitHub-hosted Ubuntu runner reports:
0.041533 svs TrackEval0.314130 s— py-motmetrics was 7.56x faster0.026130 svs TrackEval0.063304 s— py-motmetrics was 2.42x fasterThe CI job prints both timing groups in the log and GitHub job summary. Timing remains informational; correctness is the gate.
Correctness
Every per-sequence and combined HOTA, CLEAR, and Identity value matches TrackEval within the required
1e-6tolerance. The latest local run's largest absolute difference was2.220e-16.Cached compact statistics are used only with the library's standard metric implementations. Custom metrics and mutable event access fall back to the normal event-based path.
Testing
.venv/bin/pytest -q— 80 passed.venv/bin/pytest -q -s motmetrics/tests/test_trackeval_parity.py— exact parity, 1 passed.venv/bin/ruff check --select E9,F63,F7,F82 motmetrics— passeduv build --no-sources— wheel and source distribution builtmotmetrics 1.4.0andevaluate_manyimported successfully