Official repository for the paper "Synthetic Speech Augmentation for Low-Resource European ASR: Comparing Parakeet-TDT and Whisper, submitted to Interspeech 2026. A link to the paper will be added here once it becomes available.
The paper studies Estonian (et) and Slovenian (sl) — the two target low-resource European languages for the cross-architecture comparison (Whisper-large-v3 vs Parakeet-TDT-0.6B-v3).
Add-on experiments (not part of the paper): after submission, the same training pipeline was applied to Dutch (nl), Portuguese (pt), and Polish (pl) for Parakeet, and to Portuguese (pt) (with Dutch in progress) for Qwen3-ASR-1.7B. Published model cards document per-language results; those models are provided as community artifacts and are outside the scope of the paper's claims.
This project investigates TTS-based data augmentation for ASR in two low-resource European languages: Estonian (Finno-Ugric, 14 noun cases, agglutinative) and Slovenian (Slavic, 6 cases + dual number). We compare how two fundamentally different ASR architectures respond to identical synthetic augmentation:
- Whisper-large-v3 (1.5B params): encoder-decoder attention, 680K hrs multilingual pre-training
- Parakeet-TDT-0.6B-v3 (600M params): FastConformer + TDT transducer, 670K hrs pre-training (includes ET/SL via MOSEL corpus)
- First TTS augmentation study for Estonian and Slovenian ASR
- Cross-architecture comparison (Whisper vs Parakeet/FastConformer) under identical augmentation
- Morphology-aware text diversification with LLM-as-judge validation
- Open-source release of all synthetic datasets, fine-tuned models, and code on HuggingFace
| Dataset | Estonian (et) | Slovenian (sl) |
|---|---|---|
| CommonVoice 17 train | ~3,157 utterances (~46 hrs validated) | ~19,639 utterances (~145 hrs validated) |
| CommonVoice 17 test | 2,653 utterances | 3,236 utterances |
| FLEURS test | ~350 sentences | ~350 sentences |
- CV17 source:
fixie-ai/common_voice_17_0(HuggingFace) - FLEURS source:
google/fleurswith configset_eeandsl_si
~6,000 sentences per language, ~12,000 total, across 3 categories:
- Paraphrase (2,000/lang): Rewrites of CV17 sentences for lexical/syntactic diversity
- Domain Expansion (2,000/lang): New sentences across 10 underrepresented domains
- Morphological Diversity (2,000/lang): Sentences targeting rare grammatical forms
Text generation: GPT-5-mini with low reasoning Text validation: LLM-as-judge with rejection sampling (PASS/FAIL + regeneration on failure) Speech synthesis: OpenAI gpt-4o-mini-tts (6 voices cycled deterministically)
Main experiments (12): 2 models x 2 languages x 3 conditions (zero-shot, CV-only, CV+Synth) Ablation (4): CV+Synth WITHOUT morphological category (both models x both languages)
| # | Model | Language | Condition |
|---|---|---|---|
| 1-3 | Whisper-large-v3 | Estonian | Zero-shot / CV-only / CV+Synth |
| 4-6 | Whisper-large-v3 | Slovenian | Zero-shot / CV-only / CV+Synth |
| 7-9 | Parakeet-TDT-0.6B-v3 | Estonian | Zero-shot / CV-only / CV+Synth |
| 10-12 | Parakeet-TDT-0.6B-v3 | Slovenian | Zero-shot / CV-only / CV+Synth |
| 13-14 | Whisper-large-v3 | ET / SL | CV+Synth (no morph) |
| 15-16 | Parakeet-TDT-0.6B-v3 | ET / SL | CV+Synth (no morph) |
Multi-seed: 8 fine-tuning experiments (CV-only + CV+Synth) run with 3 random seeds each. Ablation with 1 seed. Total training runs: 28. Total GPU time: ~65 hours on H100.
- Metrics: WER + CER via
jiwer - Test sets: CommonVoice 17 test (in-domain) + FLEURS test (out-of-domain)
- Statistical tests: Paired bootstrap resampling (10K iterations), Wilcoxon signed-rank, McNemar's test
- Correction: Holm-Bonferroni for multiple comparisons
All training uses full fine-tuning at bf16 precision (no LoRA).
Whisper-large-v3: batch=32, lr=1e-5, warmup=500, max_steps=4000, bf16, SDPA attention Parakeet-TDT-0.6B-v3: batch=32, lr=5e-5, warmup=10%, max_epochs=100 (early stopping patience=10), bf16-mixed, CosineAnnealing (NeMo)
Full tables (validation + test, raw + normalized, per-condition, plus paired-bootstrap significance) live in results/README.md. Headline numbers below are raw WER on the CV17 and FLEURS test splits.
| Language | Condition | CV17 Test WER | CV17 Test CER | FLEURS Test WER |
|---|---|---|---|---|
| Estonian | Zero-shot | 27.19 | 6.42 | 39.14 |
| Estonian | CV only | 22.35 | 4.90 | 36.60 |
| Estonian | CV + Synth No Morph | 21.24 | 4.73 | 35.41 |
| Estonian | CV + Synth All | 21.03 | 4.64 | 35.29 |
| Slovenian | Zero-shot | 50.23 | 24.51 | 40.18 |
| Slovenian | CV only | 14.08 | 3.45 | 38.57 |
| Slovenian | CV + Synth No Morph | 12.22 | 2.93 | 35.05 |
| Slovenian | CV + Synth All | 11.56 | 2.87 | 34.71 |
| Language | Condition | CV17 Test WER | FLEURS Test WER |
|---|---|---|---|
| Estonian | Zero-shot | 34.40 | 40.73 |
| Estonian | CV only | 29.38 | 38.51 |
| Estonian | CV + Synth No Morph | 26.50 | 36.54 |
| Estonian | CV + Synth All | 26.46 | 36.56 |
| Slovenian | Zero-shot | 21.20 | 37.02 |
| Slovenian | CV only | 19.31 | 46.79 |
| Slovenian | CV + Synth No Morph | 15.65 | 40.46 |
| Slovenian | CV + Synth All | 16.40 | 41.11 |
Headline finding: the 0.6B-parameter Parakeet-TDT outperforms the 1.55B-parameter Whisper on both Estonian and Slovenian CV17 test WER after fine-tuning, despite Whisper having a large zero-shot advantage on Slovenian. On FLEURS Slovenian, Whisper fine-tuning causes out-of-domain degradation (WER 37.02 → 41.11), whereas Parakeet generalizes (40.18 → 34.71).
Raw WER/CER on CV17 test; training recipes are in the model cards on HuggingFace.
| Language | Model | Zero-shot WER | Fine-tuned WER | CER | Δ |
|---|---|---|---|---|---|
| Dutch | parakeet-tdt-0.6b-dutch | 5.99 | 5.33 | 1.46 | -0.66 pp |
| Portuguese | parakeet-tdt-0.6b-portuguese | 15.86 | 10.71 | 2.69 | -5.15 pp |
| Polish* | parakeet-tdt-0.6b-polish | 9.72 | 11.81 | 2.72 | +2.09 pp |
*Polish: fine-tune underperformed zero-shot (BIGOS train data vs CV17 test domain mismatch). Archived on HF but use the base nvidia/parakeet-tdt-0.6b-v3 for Polish inference.
syntts_asr/
├── README.md # This file
├── CLAUDE.md # Context for Claude Code assistant sessions
├── pyproject.toml # Python dependencies (uv-managed)
├── uv.lock
├── src/
│ ├── data_pipeline/ # Synthetic data generation + HF dataset prep
│ │ ├── download_data.py
│ │ ├── text_diversification.py # 3-category LLM generation + LLM-as-judge validation
│ │ ├── tts_synthesis.py # OpenAI TTS synthesis with voice cycling
│ │ └── prepare_and_push_dataset.py
│ ├── training/ # Model fine-tuning
│ │ ├── train_parakeet.py # Parakeet-TDT NeMo fine-tuning (et, sl, nl, pt, pl)
│ │ └── train_whisper.py # Whisper-large-v3 full fine-tuning
│ └── evaluation/ # WER/CER evaluation
│ ├── evaluate.py # Zero-shot + fine-tuned on CV17 + FLEURS
│ ├── significance.py # Paired bootstrap significance testing
│ └── report_normalized.py # Normalized WER/CER comparison
├── scripts/
│ ├── train/ # Training entrypoints (one per language)
│ │ ├── parakeet.sh # Generic template (et/sl paper experiments)
│ │ ├── parakeet_nl.sh # Dutch Parakeet add-on
│ │ ├── parakeet_pt.sh # Portuguese Parakeet add-on
│ │ ├── qwen_pt.sh # Qwen3-ASR-1.7B Portuguese (Docker)
│ │ ├── qwen_nl.sh # Qwen3-ASR-1.7B Dutch (Docker)
│ │ └── whisper.sh
│ ├── docker/ # Containerised workflow for Qwen3-ASR
│ │ ├── up.sh / down.sh / shell.sh / attach.sh / logs.sh
│ │ ├── train.sh # tmux-launched training in container
│ │ └── README.md
│ ├── publish/ # HF Hub upload + model cards + announcements
│ │ ├── parakeet_nl.py
│ │ ├── parakeet_pt.py
│ │ ├── parakeet_pl.py
│ │ ├── qwen_pt.py # Qwen3-ASR-1.7B-PT model card
│ │ ├── update_readmes.py
│ │ ├── create_experiments_repo.py
│ │ └── hf_post_parakeet.md # Draft announcement text
│ ├── evaluate/ # Batch evaluation + significance testing
│ │ ├── whisper_all.py
│ │ ├── whisper_significance.py
│ │ ├── whisper_significance_normalized.py
│ │ └── qwen_pt_zero_shot_baseline.py # apples-to-apples baseline for the Qwen card
│ ├── data/ # Data prep / model downloads
│ │ ├── download_and_convert.py
│ │ └── download_whisper_models.py
│ ├── setup.sh # One-time environment setup
│ └── README.md # How to run each workflow
├── Dockerfile # CUDA 12.8 + flash-attn for Qwen3-ASR runs
├── prompts/
│ ├── paraphrase.txt
│ ├── domain.txt
│ ├── morphological_et.txt
│ ├── morphological_sl.txt
│ └── validator.txt
├── data/ # Downloaded + generated data (gitignored)
└── results/ # Checkpoints, .nemo files, JSON metrics, wandb logs
See scripts/README.md for how to run each workflow end-to-end.
The text diversification approach is based on findings from multiple ASR augmentation papers:
- Text diversity >> speaker diversity (arXiv:2410.16726, "Enhancing Low-Resource ASR through Versatile TTS", 2024)
- LLMs lack morphological compositional generalization for agglutinative languages (arXiv:2410.12656)
- Domain expansion improves out-of-domain generalization (Hard-Synth, arXiv:2411.13159)
Category 1 - Paraphrase: Rewrite existing CV17 train sentences with different vocabulary and sentence structures. Increases lexical and syntactic diversity of existing content.
Category 2 - Domain Expansion: Generate new sentences across 10 domains underrepresented in CommonVoice (medical, legal, sports, technology, cooking, travel, weather, education, finance, daily conversation). Broadens vocabulary coverage.
Category 3 - Morphological Diversity: Explicitly generate sentences featuring rare grammatical forms. For Estonian: illative, inessive, elative, allative, adessive, translative, comitative, terminative cases. For Slovenian: dual number (all cases) + rare singular/plural case forms (instrumental, locative).
Every generated sentence is validated by a second LLM call that checks:
- Grammatical correctness
- Naturalness (would a native speaker say this?)
- Appropriate length (5-20 words)
- No code-switching (entirely in target language)
On FAIL: sentence is discarded and regenerated with the failure reason as corrective feedback. Max 2 regeneration attempts. This implements rejection sampling with LLM feedback.
- Deduplication: exact match + fuzzy (Jaccard > 0.85)
- Test set leakage check: dedup against CV17 test + dev + FLEURS test
- Character set validation: only valid characters for target language
These are published Estonian / Slovenian ASR numbers from other groups — not produced by this project. Reported on FLEURS with their own evaluation settings (often normalized WER), so they are not directly comparable to our raw-WER numbers in the Results section above.
| Model | Estonian FLEURS WER | Slovenian FLEURS WER |
|---|---|---|
| Parakeet-TDT-0.6B-v3 (zero-shot, NVIDIA) | 17.73% | 24.03% |
| Parakeet-TDT-0.6B-v3 (CoVoST, NVIDIA) | 22.04% | 31.80% |
| EuroSpeech Whisper-v3-Turbo (zero-shot) | 18.4% | 20.5% |
| EuroSpeech Whisper-v3-Turbo (fine-tuned) | 9.9% | 13.0% |
| TalTechNLP whisper-large-et (broadcast) | 6.9% | — |
| samolego/whisper-small-slovenian (ARTUR) | — | 11.0% |
- "Enhancing Low-Resource ASR through Versatile TTS" (2024, arXiv:2410.16726)
- "Hard-Synth: Synthesizing Diverse Hard Samples for ASR" (2024, arXiv:2411.13159)
- "Frustratingly Easy Data Augmentation for Low-Resource ASR" (2025, arXiv:2509.15373)
- "An Exhaustive Evaluation of TTS/VC Data Augmentation" (2025, arXiv:2503.08954)
- "When Whisper Meets TTS" (2023, TSD)
- "Scaling Laws for Synthetic Speech" (Interspeech 2025)
- "Making More of Little Data" (ACL 2023)
- "Evaluating Morphological Compositional Generalization in LLMs" (2024, arXiv:2410.12656)
- Parakeet-TDT (NVIDIA, arXiv:2509.14128)
- Whisper (Radford et al., 2023)
All artifacts are published on HuggingFace under yuriyvnv.
| Model | CV17 Test WER | Status |
|---|---|---|
| yuriyvnv/parakeet-tdt-0.6b-estonian | 21.03% | Published |
| yuriyvnv/parakeet-tdt-0.6b-slovenian | 11.56% | Published |
Parakeet-TDT-0.6B-v3 fine-tunes:
| Model | CV17 Test WER | Status |
|---|---|---|
| yuriyvnv/parakeet-tdt-0.6b-dutch | 5.33% | Published |
| yuriyvnv/parakeet-tdt-0.6b-portuguese | 10.71% | Published |
[yuriyvnv/parakeet-tdt-0.6b-polish] |
11.81% (worse than 9.72% zero-shot) | Internal only — not recommended |
Qwen3-ASR-1.7B fine-tunes (apples-to-apples with the zero-shot base; both eval'd via the same evaluate_model path with normalised refs):
| Model | Test set | Zero-shot WER | Fine-tuned WER | Δ rel | Status |
|---|---|---|---|---|---|
| yuriyvnv/Qwen3-ASR-1.7B-PT | CV17-pt test | 12.63% | 8.40% | -33.5% | Published |
| yuriyvnv/Qwen3-ASR-1.7B-PT | CV22-pt test | 12.91% | 8.72% | -32.5% | Published |
yuriyvnv/Qwen3-ASR-1.7B-NL |
— | — | — | — | Training in progress |
The Qwen3-ASR runs use the Docker workflow (scripts/docker/) — flash-attn 2 and matched CUDA 12.8 nvcc are required and not present on the host.
Datasets: yutiyvnv/synthetic_transcript_nl and yutiyvnvsynthetic_transcript_pt for the add-on languages available on Hugging Face.
Relevant Paper @article{perezhohin2024enhancing, title={Enhancing automatic speech recognition: effects of semantic audio filtering on models performance}, author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro}, journal={IEEE Access}, volume={12}, pages={155136--155150}, year={2024}, publisher={IEEE} }