Insights on porting parakeet-mlx real time transcription on rpi 5 or other embedded devices

[gkiri]

Thank you for Great work @senstella
I wanted to port and run realtime parakeet models on rpi 5 or other embedded devices .
can you provide some insights on how to do it or if its feasible theoretically or GPU is must for this ?

[senstella]

Hello! I believe it's theoretically feasible. The reason I wrote a custom kernel is that MLX's smart indexing created a memory bottleneck and slowed down large inputs. So it would likely function without it if you convert the original NeMo implementation to ONNX.

But before trying that, you might want to test the performance of the target device first. Perhaps try running something like onnx-asr to ensure inference performance exceeds real-time.

If it satisfies your requirements, the next step would be exporting the model after setting local attention like this:

model.change_attention_model("rel_pos_local_attn", [256,256])

However, I'm not much familar with ONNX, so I'm not sure how you can export the both convolution & attention KV cache interface and feed the cache. That would be the tricky part. The crucial part of streaming is KV cache, and I believe it would require writing or modifying an existing exporter. If you figure that out, you can implement streaming inference like this.

---

Or to keep it simple, you can do normal ONNX inference as the audio comes in and merge the results with various merge algorithms, but this can be unstable sometimes. It pretty much depends on your use case and accuracy tolerance!

[gkiri]

Thank you very much. let me start out on this , will update if I make progress.