Expected Average Overlap on failing runs

[0xfedcafe]

In the paper "The Visual Object Tracking VOT2015 challenge results," the Expected Average Overlap metric is introduced as follows:
"All segments shorter than $N_s$ frames that did not finish with a failure are removed and the remaining segments are converted into $N_s$ frames long tracking outputs. The segments are either trimmed or padded with zero overlaps to the size $N_s$. An average overlap is computed for each segment, and the average across all segments is the estimate of $Φ^{N_s}$. Repeating this computation for different values of $N_s$ produces an estimate of the expected average overlap curve."

The computation of EAO is defined like this:

toolkit/vot/analysis/supervised.py

Line 74 in 295d629

def compute_eao_curve(overlaps: List, weights: List[float], success: List[bool]):

def compute_eao_curve(overlaps: List, weights: List[float], success: List[bool]):
    """Computes EAO curve from a list of overlaps, weights and success flags."""
    max_length = max([len(el) for el in overlaps]) 
    total_runs = len(overlaps)
    
    overlaps_array = np.zeros((total_runs, max_length), dtype=np.float32)
    mask_array = np.zeros((total_runs, max_length), dtype=np.float32)  # mask out frames which are not considered in EAO calculation
    weights_vector = np.reshape(np.array(weights, dtype=np.float32), (len(weights), 1))  # weight of each run

    for i, (o, success) in enumerate(zip(overlaps, success)):
        overlaps_array[i, :len(o)] = np.array(o)
        if not success:
            # tracker has failed during this run - fill zeros until the end of the run
            mask_array[i, :] = 1
        else:
            # tracker has successfully tracked to the end - consider only this part of the sequence
            mask_array[i, :len(o)] = 1

    overlaps_array_sum = overlaps_array.copy()
    for j in range(1, overlaps_array_sum.shape[1]):
        overlaps_array_sum[:, j] = np.mean(overlaps_array[:, 1:j+1], axis=1)
    
    return np.sum(weights_vector * overlaps_array_sum * mask_array, axis=0) / np.sum(mask_array * weights_vector, axis=0).tolist()

and EAOScore.compute:

toolkit/vot/analysis/supervised.py

Line 357 in 295d629

return dependencies[0].foreach(lambda x, i, j: (float(np.mean(x[0][self.low:self.high + 1])), ) )

return dependencies[0].foreach(lambda x, i, j: (float(np.mean(x[0][self.low:self.high + 1])), ) )

But in Python, it's not an error to take a slice more than the length of a sequence; it would just get truncated to the sequence length.

For weak trackers, based on the current code, if the longest segment is shorter than high + 1, the result will be biased due to the lack of padding at the end, contradicting the paper. The mean is taken over fewer $N_s$ values with a smaller denominator, the dropped values are the smallest in the curve, so EAO is biased upward, selectively for frequently-failing trackers.

Minimal reproduction:

import numpy as np
from vot.analysis.supervised import compute_eao_curve

# one run: init frame + 32 tracked frames at overlap 0.8, ending in failure
overlaps = [[1.0] + [0.8] * 32]
curve = compute_eao_curve(overlaps, [1.0], [False])

print(len(curve))                 # 33 - curve stops at the longest segment
print(np.mean(curve[20:51]))      # 0.800 - reported EAO for low=20, high=50

# By the definition, Φ(Ns) = 25.6/Ns for Ns in 33..50 (zero-padded failed segment),
# so the correct mean over [20, 50] is:
ext = [25.6 / j for j in range(33, 51)]
print(np.mean(np.concatenate([curve[20:], ext])))   # 0.699

Also, if the curve is shorter than low, the slice is empty and np.mean([]) returns NaN with only a RuntimeWarning.

This can be fixed by changing the compute_eao_curve:

max_length = max(max([len(el) for el in overlaps]), min_length)

And EAOCurve would pass the min_length to compute_eao_curve.