Questions about generation

[DePasqualeOrg]

Thanks for creating this! I'm testing it out, and I'm encountering some unexpected behavior when generating audio from text composed of multiple sentences. Often it generates only the first sentence, with a long silence after it. Sometimes it produces an empty audio file. Is this expected? If so, maybe you could provide more information on this in the readme.

[senstella]

Hello, there! Thank you for reporting the issue. While I only created the csm-mlx inference repository, not the underlying model itself, it's definitely interesting issue to investigate. Do you mind sharing more about the sentences and the sampling parameters you've used? I reckon it's mostly sampling problem.

[DePasqualeOrg]

I just used the default settings from the example in the readme, with a text comprised of two sentences as input. Have you tried this?

[senstella]

I believe there seems to be a problem with the repetition, as far as I've investigated. I'll try to implement a logits processor in this implementation as a workaround.

[senstella]

I've implemented a logits processor and updated the readme with better defaults accordingly! Let me know if this improves the issue in your case.

[DePasqualeOrg]

Thanks! That works now. I'll investigate this more when I have time, but it seems like it's generating sentences in isolation and concatenating the output, since the transitions between sentences don't seem very natural. Also, when I tried 4-bit quantization, I noticed that the audio started out fine but started to degrade after about 20 seconds.

Ideally I'd like to use this to convert longer texts to audio, but the model seems to have some properties that might need to be worked around for this use case.

[swolf0]

In my experience, text needs to be split into short sentences, but the model seems to do well stitching those separate generations together in a believable way. Typographic characters need to be removed too, and I've also stopped loading the entire conversation audio into the context. I have a clone voice sample, my request, then the entire chunked generation for that turn. I've found that the model still sounds conversational just using my latest turn as a request (I guess I have become the cache for the tone).

Kokoro's Heart voice works well as a base. Sometimes it sounds like Sesame's Maya. For long text, I'd recommend using Kokoro completely, but I understand if you're trying to make better dialog, CSM could be preferable!

[cryptonicsurfer]

Does anyone know how/if mlx could be used to finetune either csm-1b or kokoro to other languages, and if so how would this be done? I would hope to be able to use either model but tuned to Swedish. thanks!

[senstella]

Regarding to fine-tuning CSM models, there's a PR in #10. I'm trying to fix the loss calculation and structure of the PR. Once it's done, it could be easily done in CLI or a simple Python script, I believe!

However, considering the CSM model is trained only on English, it would likely be difficult to simply tune the model to Swedish without a large amount of data.

For Kokoro, it's a relatively small model. So I think it's possible to teach the model Swedish with less data (around a hundred hours).