Translate mode: x→y targets and two-way interpreting - #33
Draft
fikrikarim wants to merge 3 commits into
Draft
Conversation
Kokoro voice names carry their language as a prefix; tts.py never forwarded it, so every non-English voice was G2P'd as American English — audible as a heavy wrong accent (measured in translatebench). MLX gets lang_code, ONNX the espeak language map. The default English voice is byte-identical in behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Head captures targets and pairs 13/14 (languages array; the miss one repeat of a loose 'interpret between us' phrasing), rendering 12/12 one-way across Spanish/French/Japanese and 5/6 two-way with the direction right every time it rendered — the miss a transcript-only reply, the failure shape production's fallback line already covers. Voice probe writes per-language WAVs for ear checks; es fixtures synthesize on the Spanish voice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The decider reports the language(s) involved (one = one-way target, two = interpreting pair); translate_mode() parameterizes the mode with its own voice per one-way target and the English voice for pairs — Kokoro's strongest, and the decided policy when English is one side. The pair prompt lets the model pick the direction per utterance from the language it hears. A sentence the voice can't phonemize (Japanese through the English fallback) now costs its audio, never the turn. The chip reads 'Translating → Spanish' / 'Interpreting English ↔ Spanish'. Retargeting mid-translate still requires exiting first — the decider only sanctions exits while translating. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fikrikarim
force-pushed
the
two-way-translation
branch
from
August 2, 2026 16:45
a1c6ef5 to
2ec4795
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's here
Translation stops hardcoding English. "Translate everything I say into Spanish" renders into Spanish with the Spanish voice; "translate between English and Spanish" is a two-way interpreter — the model picks the direction per utterance from the language it hears, and the English voice reads both directions (decided policy: English in the pair anchors the voice).
actions.py: the head reports the language(s) involved — one entry is a one-way target, two an interpreting pair (measured 13/14,benchmarks/translatebench.py).modes.py:translate_mode(languages)parameterizes the mode; per-language Kokoro voices for one-way targets (en/es/fr/it/pt/hi — the ones whose G2P works on a stock install), English voice for pairs and unmapped targets.server.py: parameterizedTRANSLATE_PROMPT(12/12 across es/fr/ja) +TWO_WAY_PROMPT(5/6; the miss a transcript-only reply the fallback line covers, never a wrong direction).tts.py: Kokoro voices now route through their own language pipeline (lang_codefrom the voice prefix) — previously all non-English voices were G2P'd as English.pipeline.py: a sentence the voice can't phonemize costs its audio, never the turn.Draft because: TTS
Kokoro's non-English output is the weak link — we tried it and it doesn't seem great. Before this merges we should explore a TTS with real multilingual support, ideally with automatic language detection so the voice follows the output language per utterance (which would also remove the English-voice-reads-Spanish compromise in pair mode).
src/parlor/tts.pyis a small interface, so a swap is contained. Known gaps regardless: Japanese/Chinese need themisaki[ja]/misaki[zh]extras; retargeting mid-translate requires exiting first.🤖 Generated with Claude Code