Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ The project code is MIT. Model weights carry their own licenses:
Echo-TTS CC-BY-NC-SA-4.0 No
Fish Audio S2 Pro Fish Audio Research License No
Dots TTS Apache-2.0 Yes
Audio8 TTS Apache-2.0 Yes
DramaBox LTX-2 Community License Conditional
OmniVoice Apache-2.0 Yes
MOSS-TTS Apache-2.0 Yes
Expand Down
12 changes: 7 additions & 5 deletions PROJECT_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,15 @@
**Key architectural rules:**
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
- Runtime routing happens through `ModelLoadConfig.runtime_mode` + `runtime_profile`, not ad-hoc subprocess calls
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. Engines that support the modern stack run natively in the main Transformers 5 environment.
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, Higgs Audio 2, and Audio8 TTS. Engines that support the modern stack run natively in the main Transformers 5 environment.
- YAML (`docs/Dev reports/tts_audio_suite_engines.yaml`) is source of truth for engine doc tables → run `python3 scripts/generate_engine_tables.py --readme` to regenerate
- Auxiliary YAML (`docs/Dev reports/tts_audio_suite_aux_models.yaml`) is source of truth for helper/post-process model docs → run `python3 scripts/generate_aux_model_docs.py`
- All models download to `ComfyUI/models/TTS/<model-name>/`
- Engine registry: `utils/models/engine_registry.py`

## Engines

19 engines follow the pattern above:
20 engines follow the pattern above:

| Engine | Adapter | Processor | SRT Processor | Engine Node |
|--------|---------|-----------|---------------|-------------|
Expand All @@ -44,13 +44,14 @@
| Echo-TTS | `echo_tts_adapter.py` | `nodes/echo_tts/echo_tts_processor.py` | `echo_tts_srt_processor.py` | `echo_tts_engine_node.py` |
| Fish Audio S2 Pro | `fish_audio_s2_adapter.py` | `nodes/fish_audio_s2/fish_audio_s2_processor.py` | `fish_audio_s2_srt_processor.py` | `fish_audio_s2_engine_node.py` |
| Dots TTS | `dots_tts_adapter.py` | `nodes/dots_tts/dots_tts_processor.py` | `dots_tts_srt_processor.py` | `dots_tts_engine_node.py` |
| Audio8 TTS | `audio8_tts_adapter.py` | `nodes/audio8_tts/audio8_tts_processor.py` | `audio8_tts_srt_processor.py` | `audio8_tts_engine_node.py` |
| DramaBox | `dramabox_adapter.py` | `nodes/dramabox/dramabox_processor.py` | `dramabox_srt_processor.py` | `dramabox_engine_node.py` |
| OmniVoice | `omnivoice_adapter.py` | `nodes/omnivoice/omnivoice_processor.py` | `omnivoice_srt_processor.py` | `omnivoice_engine_node.py` |
| MOSS-SoundEffect v2 | `moss_soundeffect_v2_adapter.py` | — | — | `moss_soundeffect_v2_engine_node.py` |
| RVC | — | `engines/rvc/` | — | `rvc_engine_node.py` |

**Engine implementations live in:**
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/audio8_tts/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`

## Documentation Files

Expand Down Expand Up @@ -145,13 +146,14 @@
- `launcher.py` - runtime bootstrap, venv creation, Windows toolchain env setup
- `session.py`, `protocol.py` - JSONL worker transport and message protocol
- `bootstrap.py` - shared runtime bootstrap helpers
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py` - parent-process proxies
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py`, `audio8_tts_proxy.py` - parent-process proxies
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, and Audio8 TTS
- Current shared legacy T4 runtime profile is reused by:
- VibeVoice / Kugel
- Qwen3-TTS
- Qwen3-ASR and Granite's optional Qwen forced aligner
- Higgs Audio 2
- Audio8 TTS

### Audio (`utils/audio/`)
- `processing.py` - Tensor manipulation, normalization, format conversion
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu

<!-- ENGINE_COMPARISON_START -->

## Quick Engine Comparison — 19 Engines
## Quick Engine Comparison — 20 Engines

| Engine | Languages | Size | Key Features |
|--------|-----------|------|--------------|
Expand All @@ -41,6 +41,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
| **Echo-TTS** | 🇺🇸 | ~5.3GB + ~1.8GB | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) |
| **Fish Audio S2 Pro** | 🌐 80+ languages | ~10.3GB / ~8.0GB | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references |
| **Dots TTS** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +13 | ~6GB | Official auto language detect / language control, SOAR and MeanFlow distilled variants |
| **Audio8 TTS** | 🌐 11 recommended languages | ~2.39 GiB | Reference-free TTS + zero-shot/cross-lingual cloning, 44.1kHz output with sampling or greedy decoding |
| **DramaBox** | 🇺🇸 | ~16.4GB | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting |
| **OmniVoice** | 🌐 600+ languages | ~3.7GB | Inline non-verbal tags and pronunciation overrides, Reference-free voice design |
| **MOSS-TTS** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +18 | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue |
Expand Down Expand Up @@ -1527,6 +1528,7 @@ For offline/manual setup:
| Granite ASR | `ComfyUI/models/TTS/granite_asr/` | ✅ | Granite ASR models; plus adds native diarization/timestamps, optional Qwen forced aligner reused lazily for timestamps/SRT fallback |
| Echo-TTS | `ComfyUI/models/TTS/echo-tts-base/` | ✅ | ~7.1GB total (base + dac); CC-BY-NC-SA |
| Dots TTS | `ComfyUI/models/TTS/dots_tts/` | ✅ | Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder |
| Audio8 TTS | `ComfyUI/models/TTS/audio8_tts/Audio8-TTS-Preview-0.6b/` | ✅ | ~2.39 GiB official 0.6B Preview checkpoint; shared Transformers 4.57.3 runtime required because main 5.10.2 collapses voice cloning |
| DramaBox | `ComfyUI/models/TTS/dramabox/DramaBox/` | ✅ | ~16.4GB download; fast mode roughly 24GB VRAM; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved; conditional LTX-2 Community License |
| Fish Audio S2 Pro | `ComfyUI/models/TTS/fish_audio_s2_pro/` | ✅ | Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License |
| OmniVoice | `ComfyUI/models/TTS/omnivoice/` | ✅ | Official OmniVoice model. Voice cloning in this suite requires explicit reference text. |
Expand Down
97 changes: 95 additions & 2 deletions docs/Dev reports/tts_audio_suite_engines.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1339,6 +1339,77 @@ engines:
speed_performance: { supported: "partial", notes: "Moderate; mf variant is faster" }
reference_free_tts: { supported: true, notes: "(default speaker)" }

- id: audio8_tts
name: Audio8 TTS
models: "Audio8 TTS Preview 0.6B"
size: "~2.39 GiB"
license: "Apache-2.0"
commercial: true
language_summary_full: "11 recommended languages (including Cantonese)"
language_summary_compact: "🌐 11 recommended languages"

capabilities:
tts: true
srt: true
vc: false
asr: false
training: false

runtime_isolation:
default_mode: "shared_runtime"
main_environment: false
shared_runtime: true
dedicated_runtime: false
runtime_profile: "vibevoice_transformers4_shared"
notes: "Required for feature-complete Audio8 inference: main Transformers 5.10.2 collapsed the official voice-clone demo to 0.325s, while the shared Transformers 4.57.3 runtime generated 10.495s for the published 10.54s demo"

readme_key_features:
- "Reference-free TTS + zero-shot/cross-lingual cloning"
- "44.1kHz output with sampling or greedy decoding"

special_features:
- "44.1kHz reference-free TTS and zero-shot/cross-lingual voice cloning"
- "Exact reference transcript required when voice cloning"
- "Greedy or sampled decoding with temperature, top-p, top-k, seed, and token-limit controls"
- "Suite text chunking, character switching, and SRT timing/assembly"
- "No explicit language selector; multilingual behavior is text-driven"
- "No native emotion/style/speed controls, streaming, timestamps, dialogue/multi-speaker mode, or presets"
- "Not compatible with Voice Designer: the official model has no instruction-conditioned voice-design mode"
- "Unified TTS/SRT integration only; no special node or suite training integration"

model_sources:
- component: "Audio8 TTS Preview 0.6B"
source_name: "Audio8/Audio8-TTS-Preview-0.6b"
source_url: "https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b"
size: "~2.39 GiB"
auto_download: true
notes: "Official 0.6B Preview checkpoint with bundled 44.1kHz codec; suite uses the shared Transformers 4.57.3 runtime because voice cloning is not compatible with main Transformers 5.10.2"

languages:
en: { supported: true, flag: "🇺🇸", notes: "" }
zh: { supported: true, flag: "🇨🇳", notes: "(Chinese; upstream separately recommends Cantonese, folded into this matrix row)" }
de: { supported: true, flag: "🇩🇪", notes: "" }
es: { supported: true, flag: "🇪🇸", notes: "" }
fr: { supported: true, flag: "🇫🇷", notes: "" }
it: { supported: true, flag: "🇮🇹", notes: "" }
ja: { supported: true, flag: "🇯🇵", notes: "" }
ko: { supported: true, flag: "🇰🇷", notes: "" }
pl: { supported: true, flag: "🇵🇱", notes: "" }
nl: { supported: true, flag: "🇳🇱", notes: "" }

features:
voice_cloning: { supported: true, notes: "Zero-shot and cross-lingual; reference audio must be paired with its exact transcript" }
reference_transcript: { requirement: conditional }
native_multi_speaker: { supported: false, notes: "No native dialogue or multi-speaker interface; suite character switching generates speakers as separate segments" }
voice_conversion: { supported: false, notes: "No separate voice-conversion mode" }
asr_transcribe: { supported: false, notes: "" }
emotion_control: { supported: false, notes: "No native emotion, style, or speed controls" }
native_long_form: { supported: false, notes: "(uses suite text chunking and SRT assembly)" }
community_finetunes: { supported: false, notes: "No community model or upstream SFT/training integration in the suite" }
vram_efficient: { supported: "partial", notes: "~1.75 GiB model-plus-codec parameter memory at fp16/bf16 or ~3.5 GiB at float32/CPU, before KV cache, activations, and framework overhead; shared runtime reuses the main environment's PyTorch" }
speed_performance: { supported: "partial", notes: "Live shared-runtime validation: 6.36s reference-free audio in 39.3s including load; 9.99s cloned audio in 36.7s warm; exact cache reuse in 56ms" }
reference_free_tts: { supported: true, notes: "Reference audio is optional" }

- id: dramabox
name: DramaBox
models: "DramaBox 3.3B"
Expand Down Expand Up @@ -1903,15 +1974,17 @@ table_notes:
reference_transcript: >-
Conditional means the transcript is required only for the specific mode:
CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker
dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text
when provided but do not require it.
dialogue, and Audio8 TTS zero-shot/cross-lingual cloning. Higgs Audio 2,
Higgs Audio v3, and Dots TTS accept matching text when provided but do not
require it.
language_support:
- "**CosyVoice3 Chinese**: Includes 18+ dialects (Cantonese, Sichuan, Dongbei, Shanghai, etc.)"
- "**Higgs Audio 2**: Trained on EN, ZH (Mandarin), KO, DE, ES (English majority) - 10M hours AudioVerse dataset"
- "**Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages."
- "**IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary."
- "**MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix."
- "**OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset."
- "**Audio8 TTS**: The Preview recommends Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, and Spanish. It has no explicit language selector; language behavior is text-driven. Cantonese is folded into the Chinese row in this matrix."
- "**RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row."
- "Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT."

Expand Down Expand Up @@ -1980,6 +2053,10 @@ readme_model_download_table:
primary_model_path: "ComfyUI/models/TTS/dots_tts/"
auto_download: "✅"
notes: "Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder"
- engine: "Audio8 TTS"
primary_model_path: "ComfyUI/models/TTS/audio8_tts/Audio8-TTS-Preview-0.6b/"
auto_download: "✅"
notes: "~2.39 GiB official 0.6B Preview checkpoint; shared Transformers 4.57.3 runtime required because main 5.10.2 collapses voice cloning"
- engine: "DramaBox"
primary_model_path: "ComfyUI/models/TTS/dramabox/DramaBox/"
auto_download: "✅"
Expand Down Expand Up @@ -2423,6 +2500,22 @@ model_layouts_markdown: |
- Native sample rate is 48kHz.
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.

## Audio8 TTS

```text
ComfyUI/models/TTS/audio8_tts/
└── Audio8-TTS-Preview-0.6b/
└── (13 required official runtime files, including codec weights)
```

Notes:

- The official `Audio8/Audio8-TTS-Preview-0.6b` checkpoint is about 2.39 GiB and includes the 44.1kHz neural codec.
- The 0.6B model plus codec use about 1.75 GiB for parameters at fp16/bf16 or about 3.5 GiB at float32/CPU, before KV cache, activations, and framework overhead.
- The suite uses the existing `vibevoice_transformers4_shared` runtime with Transformers 4.57.3. Main Transformers 5.10.2 is not compatible with Audio8 voice cloning: the official demo collapsed to 0.325s, while the shared runtime generated 10.495s for the published 10.54s demo.
- Live shared-runtime validation produced 6.36s of reference-free audio in 39.3s including model load, 9.99s of cloned audio in 36.7s warm, and exact cache reuse in 56ms.
- The engine has unified TTS and SRT support only: no Voice Designer support, special node, training integration, native streaming/timestamps/dialogue, or built-in presets.

## OmniVoice

```text
Expand Down
Loading