[Benchmark] VedioEvalPro

[ssiq]

Reproduction setup

• Model: Qwen3-VL-32B-Instruct
• Frames per video: 16
• Sampling: official rand sampling
• Base random seed: 0
• Image resize: shortest edge 256, longest edge 480
• Inference failures: 0 for both evaluation tasks
• Samples: 1,289/1,289 completed by both implementations

MCQ is scored by exact option matching. OpenEnded answers are scored with the
Judge model.

Fixed-seed reproduction results

Task	Official VideoEval-Pro	VLMEvalKit	Comparison
MCQ	548/1289 (42.51%)	548/1289 (42.51%)	Identical
OpenEnded	205/1289 (15.90%)	200/1289 (15.52%)	Difference of 5 samples

The 465 videos were also compared at the frame level:

• Frame-index mismatches: 0
• Resized-dimension mismatches: 0
• JPEG-content mismatches: 0
• Missing frames: 0
• Total frames checked: 7,440

Therefore, both implementations use the same seed, frame indices, resized
dimensions, and frame contents. The MCQ scores are fully identical. The
remaining five OpenEnded differences are attributable to client-side request
packaging or image-encoding details, rather than random sampling or video
frame extraction.

The final recorded reproduction used gpt-4o-mini for Judge scoring. The
current code default has since been restored to the earlier official value,
gpt-4o-0806;