VoiceDiT

This is a repository for the paper, VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis, ICASSP 2025.

VoiceDiT is a multi-modal generative model for producing environment-aware speech and audio from text and visual prompts.

🔧 Installation

Install from source

git clone https://github.com/kaistmm/VoiceDiT.git
cd VoiceDiT
pip install -r requirements.txt

📖 Usage

• Generate audio with description prompt and content prompt:

python generate.py --desc_prompt "She is talking in a park." --cont_prompt "Good morning! How are you feeling today?"

• Generate audio with audio prompt and content prompt:

python generate.py --modality "audio" --desc_prompt "assets/bird_chirping.wav" --cont_prompt "Good morning! How are you feeling today?"

• Generate audio with image prompt and content prompt:

python generate.py --modality "image" --desc_prompt "assets/park.jpg" --cont_prompt "Good morning! How are you feeling today?"

• Text-to-Speech Example:

python generate.py --desc_prompt "clean speech" --cont_prompt "Good morning! How are you feeling today?" --desc_guidance_scale 1 --cont_guidance_scale 9

• Text-to-Audio Example:

python generate.py --desc_prompt "trumpet" --cont_prompt "_" --desc_guidance_scale 9 --cont_guidance_scale 1

• Image-to-Audio Example:

python generate.py --desc_prompt "assets/tiger.png" --cont_prompt "_" --v2a_guidance_scale 2 --desc_guidance_scale 9 --cont_guidance_scale 1

Generated audios will be saved at the default output folder ./outputs.

⚙️ Full List of Options

View the full list of options with the following command:

python generate.py -h

🙏 Acknowledgements

This work would not have been possible without the following repositories:

VoiceLDM

PixArt-alpha

HuggingFace Diffusers

HuggingFace Transformers

AudioLDM

naturalspeech

audioldm_eval

AudioLDM2