Robustness: docstrings define as "failures divided by total frames"

[0xfedcafe]

toolkit/vot/analysis/supervised.py

Line 105 in 295d629

Robustness is defined as a number of failures divided by the total number of frames.

toolkit/vot/analysis/supervised.py

Line 172 in 295d629

burnin period. Robustness is defined as a number of failures divided by the total

But I can't find that division anywhere. Per sequence, subcompute just averages raw failure counts over repetitions (L155–L163).

    def subcompute(self, experiment: Experiment, tracker: Tracker, sequence: Sequence, dependencies: List[Grid]) -> Tuple[Any]:
        """Computes accuracy and robustness of a tracker on a given sequence.

        :param experiment: Experiment.
        :type experiment: Experiment
        :param tracker: Tracker.
        :type tracker: Tracker
        :param sequence: Sequence.
        :type sequence: Sequence
        :param dependencies: Dependencies.
        :type dependencies: List[Grid]

        :returns: Accuracy, robustness, AR, number of frames.
        :rtype: Tuple[Any]"""
        trajectories = experiment.gather(tracker, sequence)

        if len(trajectories) == 0:
            raise MissingResultsException()

        accuracy = 0
        failures = 0
        for trajectory in trajectories:
            failures += count_failures(trajectory.regions())[0]
            accuracy += compute_accuracy(trajectory.regions(), sequence, self.burnin, self.ignore_unknown, self.bounded)[0]

        failures /= len(trajectories)
        accuracy /= len(trajectories)

        ar = (math.exp(- (float(failures) / len(sequence)) * self.sensitivity), accuracy)

Then aggregate averages those counts weighted by sequence length (L218–L231):

    def aggregate(self, tracker: Tracker, sequences: List[Sequence], results: Grid):
        """Aggregates results of the analysis.

        :param tracker: Tracker.
        :type tracker: Tracker
        :param sequences: List of sequences.
        :type sequences: List[Sequence]
        :param results: Results of the analysis.
        :type results: Grid

        :returns: Accuracy, robustness, AR, number of frames.
        :rtype: Tuple[Any]"""

        failures = 0
        accuracy = 0
        weight_total = 0

        for a, f, _, w in results:
            failures += f * w
            accuracy += a * w
            weight_total += w

        failures /= weight_total
        accuracy /= weight_total
        length = weight_total / len(results)

        ar = (math.exp(- (failures / length) * self.analysis.sensitivity), accuracy)

        return accuracy, failures, ar, length

where weights are sequence lengths. So it's actually a sequence-length-weighted mean of per-sequence failure counts
$n = sequence\_count$
$f_i = failure\_count_i$
$l_i = length_i$

$$\frac{\sum_{i=0}^{n - 1}{f_i \cdot l_i}}{\sum_{i=0}^{n - 1}{l_i}}$$

and not as the docstring states

$$\frac{\sum_{i=0}^{n - 1}{f_i}}{\sum_{i=0}^{n - 1}{l_i}}$$

This weighted mean of raw failure counts matches the legacy MATLAB toolkit's default weighted_mean averaging, so the computation looks intentional:

result.robustness.weighted_mean_values = sum(result.robustness.values(useable, :) ...
    .* repmat(result.lengths(useable), 1, length(trackers)), 1) ./ sum(result.lengths(useable));

The docstring's formula does match what the legacy AR plot used - overall failures per frame - so it may be describing that term. The Python AR coordinate computes this slightly differently and only agrees when all sequences have equal length.

But in a paper "A modular toolkit for visual tracking performance evaluation" it's written, "The robustness is the number of times the tracker failed, i.e., drifted from the target, and had to be reinitialized."
So it also doesn't match the paper description.

[lukacu]

So, the question is whether the code is correct or the docstring? Here, the code is correct; we are talking about the weighted average. We will correct the docstring typo.

As for the paper, are you citing the right one? I have checked, and I can't find this statement in the SoftwareX paper. Are you referring to one of the previous methodological papers?

[0xfedcafe]

I'm sorry about the wrong citation pasted, missclick. The right one is Novel Performance Evaluation Methodology for Single-Target Trackers.