🎙️ Real-Time Voice Cloning (SV2TTS)

Developed by: Muzammal Ikhlaq

Real-Time Voice Cloning is an advanced deep learning framework implementing SV2TTS (Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis). This state-of-the-art system generates a high-fidelity digital clone of any target speaker's voice using only 5 to 10 seconds of reference audio, allowing real-time text-to-speech synthesis in that cloned voice.

---

📌 Table of Contents

1. Key Features
2. SV2TTS Architecture & Workflow
3. Deep Learning Components
4. Repository Structure
5. Quick Start Guide
6. Usage Modes
   • GUI Toolbox
   • Command Line (CLI)
   • REST API Server
7. REST API Documentation
8. Performance & Inference Speed
9. Developer Information & License

---

🌟 Key Features

• ⚡ 5-Second Voice Cloning: Zero-shot voice cloning capability from just a 5-10 second audio clip.
• 🧠 3-Stage SV2TTS Neural Pipeline:
  1. Speaker Encoder: Extract a 256-dimensional fixed-size embedding vector from input audio.
  2. Tacotron Synthesizer: Convert text + speaker embedding into sequence mel-spectrograms.
  3. Wave-RNN Vocoder: Convert mel-spectrograms into high-fidelity 16kHz audio waveforms.
• 🖥️ Interactive PyQt5 GUI: Full graphical dashboard for recording audio, visualizing speaker embedding space (UMAP), and synthesizing audio interactively.
• ⚡ Command-Line Interface (CLI): Streamlined CLI for fast batch or terminal voice cloning.
• 🌐 RESTful API Server: Flask-based backend endpoint (/clone) for web, mobile, and third-party app integration.
• 📦 Pre-Trained Weights Included: Pre-loaded PyTorch model checkpoints (encoder.pt, synthesizer.pt, vocoder.pt).

---

📐 SV2TTS Architecture & Workflow

Note

SV2TTS decouples speaker identity verification from speech synthesis, enabling real-time zero-shot voice cloning without requiring any model retraining for new speakers.

flowchart LR
    subgraph Input ["Input Layer"]
        A["Reference Audio Sample (.wav, .mp3)"]
        B["Target Text Prompt"]
    end

    subgraph Pipeline ["SV2TTS Neural Pipeline"]
        A --> C["1. Speaker Encoder<br/>(ResNet/LSTM Model)"]
        C --> D["256d Speaker Embedding"]
        B --> E["2. Tacotron Synthesizer<br/>(Attention-based TTS)"]
        D --> E
        E --> F["Mel-Spectrogram"]
        F --> G["3. Wave-RNN Vocoder<br/>(Neural Audio Synthesis)"]
    end

    subgraph Output ["Output Layer"]
        G --> H["Cloned Speech Waveform (.wav)"]
    end

Loading

---

🧠 Deep Learning Components

Component	Model Type	Checkpoint Size	Function	Output
Speaker Encoder	3-Layer LSTM / ResNet	~16.3 MB (encoder.pt)	Extracts fixed 256d speaker identity vector from short audio	256d Vector
Synthesizer	Tacotron2 Architecture	~353.4 MB (synthesizer.pt)	Synthesizes mel-spectrogram from text conditioned on speaker vector	80-band Mel-Spectrogram
Vocoder	Wave-RNN / Autoregressive	~51.4 MB (vocoder.pt)	Reconstructs 16kHz time-domain audio waveform from mel-spectrogram	16kHz PCM Waveform

---

📁 Repository Structure

Real Time Voice Cloning Master/
│
├── saved_models/              # Pre-trained Neural Network Weights
│   └── default/
│       ├── encoder.pt          # Speaker Encoder PyTorch Checkpoint (~16 MB)
│       ├── synthesizer.pt      # Tacotron Synthesizer PyTorch Checkpoint (~353 MB)
│       └── vocoder.pt          # Wave-RNN Vocoder PyTorch Checkpoint (~51 MB)
│
├── api_server.py               # Flask REST API Server for HTTP Audio Requests
├── demo_cli.py                 # Command-Line Interface for Terminal Inference
├── demo_toolbox.py             # PyQt5 Graphical Interface Toolbox
│
├── encoder/                    # Speaker Encoder Module (Embedding generation)
├── synthesizer/                # Tacotron Synthesizer Module (Text-to-Spectrogram)
├── vocoder/                    # Wave-RNN Vocoder Module (Spectrogram-to-Audio)
├── toolbox/                    # GUI Frontend Components & Visualizers (PyQt5)
├── utils/                      # Audio Signal Processing & Helper Utilities
│
├── requirements.txt            # Python Package Dependencies
├── README.md                   # Primary Repository Documentation
└── DEPLOYMENT_GUIDE.md         # Production Hosting & Deployment Manual

---

🚀 Quick Start Guide

1. System Requirements

• OS: Windows 10/11, Linux (Ubuntu/Debian), macOS
• Python: Version 3.8 to 3.13
• FFmpeg: Required for decoding non-WAV formats (.mp3, .m4a, .flac)

2. Environment Setup

# Clone Repository
git clone https://github.com/Muzammal-Ikhlaq/VoiceClone-AI-Studio.git
cd VoiceClone-AI-Studio

# Create Python Virtual Environment
python -m venv voice_env

# Activate Environment
# Windows (PowerShell):
.\voice_env\Scripts\Activate.ps1
# Linux / macOS:
source voice_env/bin/activate

# Install Dependencies
pip install -r requirements.txt

---

🎮 Usage Modes

Option A: PyQt5 GUI Toolbox

Launch the interactive graphical user interface:

python demo_toolbox.py

Tip

Use the GUI toolbox to record reference audio directly from your microphone, browse preset voices, visualize speaker embedding clusters, and listen to synthesized audio in real time.

Option B: Command Line Interface (CLI)

Run inference interactively from your terminal:

python demo_cli.py

Available Flags:

• --cpu: Force CPU inference even if GPU/CUDA is available.
• --no_sound: Disable audio playback during test run.

Option C: Flask REST API Server

Start the HTTP REST API backend server:

python api_server.py

API Server listens by default on http://0.0.0.0:5000.

---

🔌 REST API Documentation

Endpoint: POST /clone

Accepts a reference audio file (.wav, .mp3, .m4a) and target text string, returning synthesized cloned audio .wav.

Request Parameters (Multipart Form-Data)

Field	Type	Required	Description
audio	File	Yes	Speaker reference audio file (5-10 sec recommended)
text	String	Yes	Sentence/text to be synthesized in target voice

---

Request Code Examples

1. cURL Request:

curl -X POST http://localhost:5000/clone \
  -F "audio=@speaker_sample.wav" \
  -F "text=Hello! This is a real-time voice cloning API demonstration." \
  --output cloned_output.wav

2. Python Client Request:

import requests

url = "http://localhost:5000/clone"
files = {"audio": open("speaker_sample.wav", "rb")}
data = {"text": "Hello world! Real-Time Voice Cloning is fully working."}

response = requests.post(url, files=files, data=data)

if response.status_code == 200:
    with open("cloned_output.wav", "wb") as f:
        f.write(response.content)
    print("Cloned audio saved to cloned_output.wav")
else:
    print("Error:", response.json())

3. Node.js / JavaScript Client Request:

const fs = require('fs');
const axios = require('axios');
const FormData = require('form-data');

async function generateVoice() {
    const form = new FormData();
    form.append('audio', fs.createReadStream('speaker_sample.wav'));
    form.append('text', 'Hello from Node.js backend integration!');

    const res = await axios.post('http://localhost:5000/clone', form, {
        headers: form.getHeaders(),
        responseType: 'arraybuffer'
    });

    fs.writeFileSync('cloned_output.wav', res.data);
    console.log('Audio saved successfully!');
}

generateVoice();

---

⚡ Performance & Inference Speed

Environment	Encoder Speed	Synthesizer Speed	Vocoder Speed	Total Real-Time Factor (RTF)
NVIDIA GPU (CUDA)	< 0.05 sec	~ 0.3 sec	~ 0.5 sec	~ 0.8x (Faster than real-time)
Multi-Core CPU	< 0.1 sec	~ 1.2 sec	~ 10-20 sec	~ 2.5x

---

👨‍💻 Developer Information

Developed by: Muzammal Ikhlaq

---

📖 Deployment & Production Hosting

For detailed production hosting instructions (Linux VPS, Nginx, Gunicorn, Systemd service, Docker containerization, and SSL setup), refer to DEPLOYMENT_GUIDE.md.

---

📄 License

This project is licensed under the MIT License.