Evaluation of Point Tracking Tasks

[Mysophobias]

"In the evaluate function, when calling the compute_tapvid_metrics function, the variable pred_occluded is set to pred_occluded[0, track_idx] = 0. Is there a problem with this line of code? Should it be pred_occluded[0, track_idx][pred_track['frame_ids']] = 0
Looking forward to your response.

[sgjheywa]

Hi,

Thanks for noticing this, your code suggestion is correct in the general sense, this is how we processed challenge submissions that have predicted frame IDs, however in our static baseline we assume that we are only given the first point to track and that the point is static and visible in every frame so the code works as is. Does that make sense?

I will look at making it clearer in the code.

[Mysophobias]

Thank you for your response, and I apologize for my delayed reply as well.

Static baseline indeed assumes that the first point is visible in all subsequent frames. However, when considering this issue: google-deepmind/tapnet#34
evaluation_points incorrectly included more points that shouldn't be evaluated when query_mode = 'first'.
When employing pred_occluded[0, track_idx][pred_track['frame_ids']] = 0, the static baseline can eliminate a significant number of incorrect false_positives in the second and third lines of code below, without causing an inflation of false_positives due to incorrectly calculated evaluation_points. And using pred_occluded[0, track_idx] = 0 will ultimately result in a smaller Jaccard.

After correcting the evaluation_points, although usingpred_occluded[0, track_idx] = 0, the static baseline indeed can achieve the correct Jaccard through the fourth line of code below. However, when evaluating models that can predict occlusion, such as TAP-Net, more incorrect false_positives may be computed in the second and third lines, and these incorrect false_positives may still be included in the evaluation_points and won't be removed in the fourth line below.

I'm not sure if my understanding is correct, and I'm looking forward to your response.

gt_positives = np.sum(visible & evaluation_points, axis=(1, 2))
false_positives = (~visible) & pred_visible
false_positives = false_positives | ((~within_dist) & pred_visible)
false_positives = np.sum(false_positives & evaluation_points, axis=(1, 2))
jaccard = true_positives / (gt_positives + false_positives)

Hi,

Thanks for noticing this, your code suggestion is correct in the general sense, this is how we processed challenge submissions that have predicted frame IDs, however in our static baseline we assume that we are only given the first point to track and that the point is static and visible in every frame so the code works as is. Does that make sense?

I will look at making it clearer in the code.