Verse-Bench repdroduction

[JennySeidenschwarz]

Hi guys!
Thanks for the great work :) Unfortunately I'm struggling to reproduce the results on Verse-Bench, any intuition here on why this could be the case? I only achieve 6.7 instead of 7.9.
Thanks in acvance!
Jenny

[robingg1]

Could you clarify which specific metric you are referring to?

To help you reproduce our results, we have updated and released our rewritten captions, which you can use directly. Please also note that differences in specific operator versions or inference settings can lead to slight deviations in the final results.

To ensure our setups are aligned, could you also provide the evaluation results of other baseline models tested under this exact same evaluation repository?

[JennySeidenschwarz]

Thanks for getting back so quick! Here some more details on my VerseBench experiment:

NAVA-official (inference_nava --use_sp, NAVA's Chinese) | Sync-C 6.84 | WER 0.131 | 516/600 sample coverage

I realized that 84 samples from set2 are on bilibili which I can't access outside of China. Any idea on how to access them? But from those samples actually only 4 contain speech so it should not really influence WER / Sync-C.

Did you see large fluctuations in performance with different operator versions?

I will get results on some other baseline as well!

[JennySeidenschwarz]

Ok I just got those results for the single sets, so can reproduce the results on set3. Could you confirm that the numbers on the paper are on the full set?

[robingg1]

Apologies for the confusion. To clarify, Sync-C, Sync-D, and WER are reported exclusively on the single-speaker subset (set3), where lip-sync is well-defined. All other metrics are averaged across the three subsets. This evaluation protocol follows previous models (like MOVA) and ensures a more accurate assessment. We have updated the README for clarity.

[JennySeidenschwarz]

Ok perfect, thanks a lot for clarifying this :) I assume Ovi and the other papers are doing the same, right?