fix(byte_tracker): stop unbounded growth of removed_stracks

[GrumpyChubbyCat]

Dear Kadu,

Thank you for jamtrack-rs! We build on the ByteTrack backend in a cross-hardware streaming/inference SDK of our own — jamtrack-rs powers its tracking path — and while running long-lived pipelines we hit a progressive slowdown that we traced to a single spot in ByteTracker::update(). Our SDK isn't public yet — it will be open-sourced once it's ready — but we wanted to contribute the fix back upstream in the meantime.

ByteTracker::update() folded every newly-removed track into a permanent removed_stracks vector:

self.removed_stracks = Self::joint_stracks(&self.removed_stracks,
                                           &current_removed_stracks);

removed_stracks is only read once per frame, to drop just-removed tracks from lost_stracks. Because track ids are monotonic and a removed track is already absent from both tracked_stracks and lost_stracks, it can never reappear there, so keeping the full history serves no purpose — a one-frame window is sufficient.

Accumulating it meant joint_stracks/sub_stracks cloned an ever-growing vector of STrack (each carrying Kalman state) on every frame. On a long-running stream this leaks memory and makes the per-frame cost grow with the total number of tracks ever seen (progressive FPS/RSS degradation).

Fix: keep only the tracks removed on the current frame. Tracking output is unchanged (the rest of update() is untouched); added a regression test that runs many spawn/despawn cycles and asserts removed_stracks stays bounded while a persistent object keeps a stable track id. Full suite: 150 passed.

Happy to adjust anything to match your preferences — thanks again for the library!

[kadu-v]

Sorry, I completely missed it!

Thanks for the PR!

I’ll review it tonight.

[kadu-v]

Thank you for the detailed explanation and regression test. I compared this change against the original ByteTrack implementation, the Roboflow implementation, and the ByteTrack paper.

Original ByteTrack implementation

The original Python implementation retains every removed track indefinitely:

self.lost_stracks = sub_stracks(self.lost_stracks, self.removed_stracks)
self.removed_stracks.extend(removed_stracks)

Source:
https://github.com/FoundationVision/ByteTrack/blob/d1bf0191adff59bc8fcfeaa0b33d3d1642552a99/yolox/tracker/byte_tracker.py#L278-L285

The current jamtrack-rs implementation follows the same update order, while using joint_stracks() to deduplicate IDs. However, removed_stracks still grows with the total number of tracks removed during the lifetime of the tracker.

removed_stracks is only used to subtract removed IDs from lost_stracks. Since track IDs are allocated monotonically and are not reused, retaining the complete removal history is unnecessary.

Roboflow implementation

Roboflow's ByteTrack implementation does not maintain a removed-track history. It keeps a single live track list and replaces it with the tracklets that still satisfy the lifecycle conditions:

self.tracks = _get_alive_tracklets(...)

Sources:

• https://github.com/roboflow/trackers/blob/59a86b003a1807e607f0cc1b32c869173d613e86/src/trackers/core/bytetrack/tracker.py#L229-L235
• https://github.com/roboflow/trackers/blob/59a86b003a1807e607f0cc1b32c869173d613e86/src/trackers/core/bytetrack/utils.py#L48-L64

Roboflow differs from the original ByteTrack implementation in other lifecycle and association details, so it is not a drop-in behavioral reference. Nevertheless, its lifecycle management confirms that a permanent removed-track history is not required.

ByteTrack paper

I also checked Algorithm 1 and the track-rebirth explanation in the ByteTrack paper:

https://arxiv.org/pdf/2110.06864

The paper describes the lifecycle as follows:

1. Unmatched tracks are moved to T_lost.
2. Lost tracks are retained for a limited number of frames—30 in the paper's default setting—so they can be re-associated.
3. Once that limit is exceeded, the track is deleted from the track set T.
4. Lost tracks are not included in the output.

The paper does not define or require a permanent collection of removed tracks. Once a track exceeds the lost-track buffer and is removed from the active/lost track set, its state no longer needs to be retained.

Conclusion

This is a valid minimal fix. Keeping only the latest removed tracks preserves the current behavior because their IDs are consumed by the next update, while preventing unbounded memory and cloning costs.

All tests passed: 150 unit tests and 5 doc tests.

One non-blocking suggestion: since removed_stracks is used on the next update, "tracks removed during this update" would be more precise than "just-removed tracks above." Also, "unbounded retention" is technically more accurate than "memory leak."

Looks good to merge. Thank you!

[kadu-v]

@GrumpyChubbyCat Thank you for your contributions again. I have released v0.5.1. Please check it!