Can a text-pretrained multilingual model be taught to translate speech to speech, for a language pair with almost no parallel speech data?
We built the whole pipeline to find out: generate a synthetic parallel-speech corpus, encode it to neural audio tokens, and adapt a Cohere2 backbone — driving a frozen Moshi depth decoder over Mimi codes — with a text "inner-monologue" running alongside the audio stream.
The full-corpus run covered 76,250 steps / 2.07 epochs on a Cloud TPU v6e-16. Capability arrived in a clear order — language identity → text translation → audio synthesis — and the run bought the first two:
measured @best, 500 held-out rows, greedy |
hi→tr | tr→hi |
|---|---|---|
| free-run text chrF++ — the inner-monologue | 25.7 | 25.1 |
| generated-audio ASR-chrF++ | 3.7 | 9.6 |
| ground-truth-audio ceiling, same pipeline | 92.1 | 86.6 |
The model learns the translation mapping; intelligible audio synthesis is the frontier. That ceiling was measured through the same codec and ASR, so the gap is not a broken harness — it is the frozen depth decoder. We publish every score next to its ceiling, because a low number without one is unreadable.
📊 Full evaluation report · ✍️ Write-up · 📈 Training run · 🤗 Models & datasets
| repo | what it is |
|---|---|
model |
Start here. Stage-2 training, the 8-stage evaluation harness, and the TPU launch tooling. Dual-backend: the same code runs on GPU or TPU. |
codec-finetuning |
Fine-tunes Mimi / DualCodec / Kanade on Turkish and Hindi across 8 optimizers, with Bayesian sweeps and bootstrap error bars — asking whether a language-specific codec beats the stock multilingual one. |
sound-quality-check |
4-stage speech-dataset quality control: round-trip ASR, MOS, SNR, VAD. This is what filtered the synthetic corpus. |
hindi-tts-probe |
5 Hindi TTS × 3 ASR round-trip benchmark. It picked the Hindi ASR judge used in the v0.3 evaluation. |
tinyaya-moshi-backbone |
The proof-of-concept that validated Tiny Aya as a Moshi backbone — prior art for everything above. |
dhruv |
Exploratory work and contributions. |
Some repos in this organization are private (data-pipeline,
tiny-aya-deliverables, simultaneous-translation) and are intentionally not
linked — the corpora they produced are published on
🤗 Hugging Face instead.
tr-hi-parallel-text text triples (en pivot -> tr / hi)
| TTS
tr-hi-parallel-speech-v2 synthetic speech + round-trip ASR quality signals
| Mimi encode
tr-hi-mimi-encoded 8 codebooks @ 12.5 Hz + word-level alignments
| Stage-2 training
tr-hi-s2st-v0.3 the released model
Every dataset in that chain is public, as is the released model with its ~89-checkpoint suite — so the whole trajectory is inspectable, not just the endpoint.
Training speech is entirely synthetic TTS, and the model is measurably distribution-bound: on real human speech the text score collapses from ~25 to ~8 chrF++. Translation references are machine-translated and carry their own ceiling. These are research artifacts for low-resource speech translation, not production translators. Earlier versions are kept with their mistakes disclosed — including two releases that trained audio-only because a filename check wrongly concluded the corpus had no text alignments. It ships 840,426.
Code is Apache-2.0. Released model weights are CC-BY-NC-4.0 —
non-commercial, inherited from
CohereLabs/tiny-aya-base.
Corpus text derives from FLORES (CC BY-SA 4.0 — share-alike propagates),
OPUS-100 and TTS output. Check the individual dataset cards before redistributing.
Compute for the v0.3 run was provided by Google's TPU Research Cloud (TRC).