Hi @senstella,
First, thank you for creating the MLX version of Parakeet. I was working on an MLX version of Canary for a project and encountered several issues. To avoid getting too deep into debugging and to move forward with my actual project, I switched to Parakeet and discovered your excellent contribution to the MLX community.
Context
My project uses speech-to-text (STT) in unconventional ways, including with non-speech signals. Because of this unique use case, I'm uncertain whether the issues I've identified affect all users, though they do relate to hallucinations in regular speech-to-text processing among others.
Some findings
- My current solution reduces hallucinations by approximately 25%
- Given the model's overall quality, this reduction may not be immediately noticeable to all users
- I've run several benchmarks using synthetic signals (which allow precise control and repeatability) to confirm the issues
Note on Implementation
The scope of the issues required me to completely rebuild of the code to identify and confirm them. For this reason, a traditional PR would not be feasible. I'm happy to discuss these findings in more detail if you're interested.
I will also publish a public repository with additional features for non-speech signals that will include fixes applicable to regular speech-to-text use cases.
Community Feedback Request
If you don't mind, I would like to ask other Parakeet MLX users to report any issues they have encountered with hallucinations and other bugs. Specifically, it would be helpful if they could benchmark the original Parakeet against Parakeet MLX to determine whether the issues are specific to the MLX implementation or related to the base model itself. For instance, if you have white noise in your background, this implementation may produce false positives, and you might also notice alignment issues. Please explain the problem you observed. For example, if you are converting audio that may contain background noise, and the resulting text has hallucinations or other error patterns, share a snippet of the audio file. If possible, test it with the Org model using spaces at nvidia/parakeet-tdt-0.6b-v2 and nvidia/parakeet-tdt-0.6b-v3.
FYI
I want to be clear that this post is not intended as criticism of your (@senstella) work. These issues actually taught me a lot, and I appreciate the opportunity to contribute back if it has any interest.
Hi @senstella,
First, thank you for creating the MLX version of Parakeet. I was working on an MLX version of Canary for a project and encountered several issues. To avoid getting too deep into debugging and to move forward with my actual project, I switched to Parakeet and discovered your excellent contribution to the MLX community.
Context
My project uses speech-to-text (STT) in unconventional ways, including with non-speech signals. Because of this unique use case, I'm uncertain whether the issues I've identified affect all users, though they do relate to hallucinations in regular speech-to-text processing among others.
Some findings
Note on Implementation
The scope of the issues required me to completely rebuild of the code to identify and confirm them. For this reason, a traditional PR would not be feasible. I'm happy to discuss these findings in more detail if you're interested.
I will also publish a public repository with additional features for non-speech signals that will include fixes applicable to regular speech-to-text use cases.
Community Feedback Request
If you don't mind, I would like to ask other Parakeet MLX users to report any issues they have encountered with hallucinations and other bugs. Specifically, it would be helpful if they could benchmark the original Parakeet against Parakeet MLX to determine whether the issues are specific to the MLX implementation or related to the base model itself. For instance, if you have white noise in your background, this implementation may produce false positives, and you might also notice alignment issues. Please explain the problem you observed. For example, if you are converting audio that may contain background noise, and the resulting text has hallucinations or other error patterns, share a snippet of the audio file. If possible, test it with the Org model using spaces at nvidia/parakeet-tdt-0.6b-v2 and nvidia/parakeet-tdt-0.6b-v3.
FYI
I want to be clear that this post is not intended as criticism of your (@senstella) work. These issues actually taught me a lot, and I appreciate the opportunity to contribute back if it has any interest.