About preprocess data to inference

[melon-bubble]

Hi! Thank you for this excellent work!

I am trying to inference on my own data. Would it be possible to share the preprocessing pipeline code or project? Any guidance on data preparation would be greatly appreciated.

Thanks in advance!

[GalaxyCong]

Hi! Thanks for your interest in our work, and sorry for the very late reply.

• In EmoDubber, we use an external aligner (MFA) to obtain phoneme durations.

• For the visual modality, the lip features (512-d) can be extracted following the pipeline from this project:
  https://github.com/mpc001/Lipreading_using_Temporal_Convolutional_Networks

Alternatively, you may also consider using AV-HuBERT (1024-d) as a replacement in your job.

• The facial expression features are extracted using EmoFAN:
  https://github.com/face-analysis/emonet

If you run into issues, feel free to let me know, I’d be happy to discuss.

Best regards!