This is a pretty sweet repo, been using it a couple times a week recently. Faster-whisper lets you actually run the large model in real time with good latency on a 3090. Actually it's even more insane, I run TWO LARGE MODELS AT THE SAME TIME, two stream-translators, so that I can have dual subtitles: one transcribed, one translated. It works fine 3090 as long as you are just doing normal desktop stuff! Wild.
But when streams have a lot of background noise (music, game sounds) I found you NEED to add some decent real-time noise reduction or Whisper just faceplants over and over.
Mainly I've used the Nvidia Broadcast tool to do this in real time, with a virtual cable if needed to get the audio routed correctly. Whisper is back at full power if I do this. But since stream-translator streams the audio directly I have to use something else instead.
If this could take in mic/speaker device audio alternative to streamlink, that would do it. Using this option loses the simplicity and latency benefits of streaming directly, but the alternative is Whisper collapsing in confusion on some streams. I know other repos already take in direct audio but ideally I want to stop bouncing between them...
Maybe there's a more elegant way to accept direct audio input that doesn't require a wacky virtual-cable or whatever? OBS studio integrates NVIDIA noise reduction via the broadcast SDK. Or there could be a good open source solution. I tried a couple but none of the real-times were close to good enough, compared to the NVIDIA version.
This is a pretty sweet repo, been using it a couple times a week recently. Faster-whisper lets you actually run the large model in real time with good latency on a 3090. Actually it's even more insane, I run TWO LARGE MODELS AT THE SAME TIME, two stream-translators, so that I can have dual subtitles: one transcribed, one translated. It works fine 3090 as long as you are just doing normal desktop stuff! Wild.
But when streams have a lot of background noise (music, game sounds) I found you NEED to add some decent real-time noise reduction or Whisper just faceplants over and over.
Mainly I've used the Nvidia Broadcast tool to do this in real time, with a virtual cable if needed to get the audio routed correctly. Whisper is back at full power if I do this. But since stream-translator streams the audio directly I have to use something else instead.
If this could take in mic/speaker device audio alternative to streamlink, that would do it. Using this option loses the simplicity and latency benefits of streaming directly, but the alternative is Whisper collapsing in confusion on some streams. I know other repos already take in direct audio but ideally I want to stop bouncing between them...
Maybe there's a more elegant way to accept direct audio input that doesn't require a wacky virtual-cable or whatever? OBS studio integrates NVIDIA noise reduction via the broadcast SDK. Or there could be a good open source solution. I tried a couple but none of the real-times were close to good enough, compared to the NVIDIA version.