Frontend generates incorrect german/multilingual audio

[djmaze]

When using a german reference voice (german_voice in this example), the following API request creates a perfect German sentence:

curl -X POST http://localhost:4123/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{"input": "Letzten Monat haben wir einen neuen Meilenstein erreicht: zwei Milliarden Aufrufe auf unserem YouTube-Kanal.", "voice": "german_voice",  "exaggeration": 0.5, "cfg_weight": 0.5, "temperature": 0.8}' --output output.wav

Uploading the same voice, using the same text and parameters in the frontend, many words sound like they were spoken by an English or other, non-native speaking person. Something seems to be off in the request to the backend.

[travisvn]

When using a german reference voice (german_voice in this example), the following API request creates a perfect German sentence:

curl -X POST http://localhost:4123/v1/audio/speech
-H "Content-Type: application/json"
-d '{"input": "Letzten Monat haben wir einen neuen Meilenstein erreicht: zwei Milliarden Aufrufe auf unserem YouTube-Kanal.", "voice": "german_voice", "exaggeration": 0.5, "cfg_weight": 0.5, "temperature": 0.8}' --output output.wav
Uploading the same voice, using the same text and parameters in the frontend, many words sound like they were spoken by an English or other, non-native speaking person. Something seems to be off in the request to the backend.

@djmaze Can you post the logs for your Docker build so that we can see what is happening on the backend when those frontend requests are made?

[djmaze]

I should add that this seems only occur in "Standard" mode. When choosing "Streaming", the output seems correct and very similar to the direct API output.

[djmaze]

Here is the matching backend log:

Using uploaded voice file: german_voice.wav (1,349,678 bytes)
📊 Request #42 - Initial memory: CPU 4203.5MB, GPU 3410.2MB allocated
Processing 1 text chunks with uploaded file and parameters:
  - Exaggeration: 0.5
  - CFG Weight: 0.3
  - Temperature: 0.8
Generating audio for chunk 1/1: 'Letzten Monat haben wir einen neuen Meilenstein er...'
INFO:     10.0.6.11:47660 - "GET /v1/info HTTP/1.0" 200 OK
INFO:     10.0.6.11:47672 - "GET /audio/speech/long-history/stats HTTP/1.0" 200 OK
INFO:     10.0.6.11:47682 - "GET /v1/info HTTP/1.0" 200 OK
INFO:     10.0.6.11:47690 - "GET /audio/speech/long-history?is_archived=false&sort=completed_desc&limit=20&offset=0 HTTP/1.0" 200 OK
INFO:     10.0.6.11:47706 - "GET /v1/health HTTP/1.0" 200 OK
INFO:     10.0.6.11:47722 - "GET /v1/status/progress HTTP/1.0" 200 OK
49.39it/s]WARNING:chatterbox.models.t3.inference.alignment_stream_analyzer:🚨 Detected 2x repetition of token 6486
WARNING:chatterbox.models.t3.inference.alignment_stream_analyzer:forcing EOS token, long_tail=tensor(False), alignment_repetition=tensor(False), token_repetition=True
50.75it/s]
INFO:     10.0.6.11:47726 - "GET /audio/speech/long?limit=10 HTTP/1.0" 200 OK
INFO:     10.0.6.11:47734 - "GET /v1/status/progress HTTP/1.0" 200 OK
✓ Audio generation completed. Size: 328,398 bytes
📊 Request #42 - Final memory: CPU 4198.5MB, GPU 3410.6MB allocated
📈 Memory change: CPU -5.0MB, GPU +0.4MB
🗑 Cleaned up temporary voice file: /tmp/voice_sample_s9qbjw4l.wav

[travisvn]

@djmaze Ok can you try it in Standard mode with a sample text that is short? Something that doesn't cause it to switch to "long mode"

It seems the problem here is limited to /audio/speech/long, which has different logic than the standard /v1/audio/speech

[travisvn]

@djmaze Alright I just spent a half hour investigating about an hour investigating.

I believe the problem has to do with the language retrieval when running a LongText request on the backend
https://github.com/travisvn/chatterbox-tts-api/blob/main/app/core/background_tasks.py#L166

I'd like to know:

• How you're running this project (in Docker? if so, which docker-compose file?)
  • Are you using CUDA?
• What does your .env file look like? I'm specifically thinking about the voice library directory location
• What is your API endpoint set to in the frontend?

I'd also like if you could:

• Upload a German voice again in the frontend, making sure to use the usual upload dialog and selecting "German" from the drop down
• Ensure that this German voice persisted in the voice library directory

I've gone in circles through the code trying to find the exact point that differentiates the long format from the regular endpoint, and answering those questions thoroughly should yield the answer.

Long text winds up here: https://github.com/travisvn/chatterbox-tts-api/blob/main/app/api/endpoints/speech.py#L245
Regular path winds up here: https://github.com/travisvn/chatterbox-tts-api/blob/main/app/api/endpoints/speech.py#L691

Seems like they pass the language_id in a slightly different format, which may be the problem. But until we've covered the stuff earlier (environment check, voice library file(s), etc.), we wont know for sure.

[djmaze]

First of all, thanks for your work!

@djmaze Alright I just spent a half hour investigating about an hour investigating.

• How you're running this project (in Docker? if so, which docker-compose file?)

Yes, I used docker-compose.uv.gpu.yml, although I adapted it to docker swarm mode, but it is essentially the same configuration. (The problem also occurred with the original compose setup.)

• Are you using CUDA?

Yes.

• What does your .env file look like? I'm specifically thinking about the voice library directory location

I did not change anything important from the example env file. Essentially I commented out the VOICE_SAMPLE_PATH and changed MODEL_CACHE_DIR to /cache and VOICE_LIBRARY_DIR to /voices.

• What is your API endpoint set to in the frontend?

It is https://hostname-of-the-backend/v1.

I'd also like if you could:

• Upload a German voice again in the frontend, making sure to use the usual upload dialog and selecting "German" from the drop down

I tried with several German voices, even with a shorter version of the original voice. All exhibit the same problem.

• Ensure that this German voice persisted in the voice library directory

All voices are persisted.

Seems like they pass the language_id in a slightly different format, which may be the problem. But until we've covered the stuff earlier (environment check, voice library file(s), etc.), we wont know for sure.

Maybe it's that..

[travisvn]

@djmaze Ok can you try it in Standard mode with a sample text that is short? Something that doesn't cause it to switch to "long mode"

It seems the problem here is limited to /audio/speech/long, which has different logic than the standard /v1/audio/speech

@djmaze Ok thanks for those responses but this is question number 1 I need to verify.

Please try just a sentence or something in the frontend and verify that it works.

Also, It is https://hostname-of-the-backend/v1. doesn't help much... And you really have https setup for this?

[djmaze]

I tried an even shorter sentence and I can see in the web developer console that the /audio/speech/long endpoint is used both for short and for long texts.

Also, It is https://hostname-of-the-backend/v1. doesn't help much... And you really have https setup for this?

Yes, I am hosting this on a private server which is available on the internet, so I don't want to post the real URL here. But that should not make a difference, should it?

[Lustralis]

I've the same issue with the Italian language, in the frontend I load an Italian speech and I pick the Italian language from the menu, the frontend shows it listed as Janis.wav (but I tried also flac and mp3) "IT Italian" tag, BUT in Sillytavern among the "Available Voices" it says clearly it's "en-US Janis" and If I try to generate an audio from an italian text, it's an english speech, the metadata are not correct.
I'm using Docker Compose for Nvidia Blackwell GPU.

[travisvn]

I've the same issue with the Italian language, in the frontend I load an Italian speech and I pick the Italian language from the menu, the frontend shows it listed as Janis.wav (but I tried also flac and mp3) "IT Italian" tag, BUT in Sillytavern among the "Available Voices" it says clearly it's "en-US Janis" and If I try to generate an audio from an italian text, it's an english speech, the metadata are not correct. I'm using Docker Compose for Nvidia Blackwell GPU.

@Lustralis Anything related to Sillytavern you're going to have to refer to their repo as support for any quirks they might have is out of scope.

I would assume the Italian (or any non-English) version should work if the right voice is specified, as it doesn't go through the "long audio" route that the frontend uses when the text exceeds a certain length.

This project aims to support OpenAI's Speech API endpoint such that in situations where you can provide the API URL like some self-hosting frontends (Open WebUI), you should be able to use chatterbox.

[Lustralis]

I've the same issue with the Italian language, in the frontend I load an Italian speech and I pick the Italian language from the menu, the frontend shows it listed as Janis.wav (but I tried also flac and mp3) "IT Italian" tag, BUT in Sillytavern among the "Available Voices" it says clearly it's "en-US Janis" and If I try to generate an audio from an italian text, it's an english speech, the metadata are not correct. I'm using Docker Compose for Nvidia Blackwell GPU.

@Lustralis Anything related to Sillytavern you're going to have to refer to their repo as support for any quirks they might have is out of scope.

I would assume the Italian (or any non-English) version should work if the right voice is specified, as it doesn't go through the "long audio" route that the frontend uses when the text exceeds a certain length.

This project aims to support OpenAI's Speech API endpoint such that in situations where you can provide the API URL like some self-hosting frontends (Open WebUI), you should be able to use chatterbox.

Sillytavern is just a check but we can ignore it.
I've uploaded a wav file with 22050 Hz, mono channel, in the webui (http://localhost:4321/) I click on "add voice" upload my wav file with italian speech and selected "IT Italian (italiano)", the webui tags my voice as IT Italian in its library, but if I generate a speech from an italian text, it generates an english pronounciation of the italian text, and doesn't work in "Streaming" mode neither.
I've tried to upload with the API call:
curl -X POST http://localhost:4123/voices -F "voice_file=@janis_it.wav" -F "voice_name=janis_it" -F "language=it"
The webui shows the voice added also in this way, but the voice is still english although it's tagged as Italian, it doesn't, it still generates a speech that is not italian.

[bluemoehre]

I'm facing the same situation here. All multi-language inferences got an English accent - no matter the length of the text.

EDIT: the issue is related to the frontend. Using Postman / API you get the desired results.

Fix:
The voice file is added to the request each time. These blocks have to be removed:

chatterbox-tts-api/frontend/src/services/tts.ts

Lines 25 to 27 in a5f4661

	if (request.voice_file) {
	formData.append('voice_file', request.voice_file);
	}

chatterbox-tts-api/frontend/src/services/tts.ts

Lines 157 to 159 in a5f4661

	if (request.voice_file) {
	formData.append('voice_file', request.voice_file);
	}

[Arjenlodder]

Same problem here!

[bluemoehre]

@Arjenlodder you may pull my fork and check if it's working for you. Also please add a review/comment to the PR, so we can fix this.