Windows: faster-whisper GPU (CUDA) never works — CTranslate2 can't find the CUDA DLLs, silently falls back to CPU

[techmik]

The pyproject.toml comment says "Every other platform keeps faster-whisper, which already uses CUDA." On Windows that isn't true. With stt_device set to cuda (or auto on an NVIDIA machine), WhisperModel(...) constructs fine, warm() logs "model ready", and startup reports healthy — then the first real utterance dies with Library cublas64_12.dll is not found or cannot be loaded (or cudnn), and either errors or drops to CPU. The CUDA runtime isn't loaded until first inference, so the failure surfaces long after startup, disconnected from any setting.

Two things are missing on Windows:

1. The CUDA libraries aren't declared. A pip/uv install has no CUDA unless the user separately installed the CUDA Toolkit. mlx-whisper is already declared for Apple Silicon; the NVIDIA equivalent isn't declared for anyone.
2. Even with the wheels installed, Windows can't find their DLLs. They land in site-packages/nvidia/<pkg>/bin, which Windows never adds to the DLL search path (a real CUDA Toolkit install puts itself on PATH; the wheels don't). os.add_dll_directory() alone doesn't fix it — CTranslate2's native loader falls back to the plain PATH env var. Confirmed by testing both against the cublas64_12.dll failure.

Observed on: Windows 11 Home 10.0.26200, Python 3.12.10, NVIDIA GPU, current main.

Reproduction

Windows + NVIDIA machine, fresh uv sync, set "stt_device": "cuda" in backtalk.json, launch and speak. First utterance: RuntimeError: Library cublas64_12.dll is not found or cannot be loaded.

Fix

Two parts.

pyproject.toml — declare the CUDA runtime, Windows only (Linux + NVIDIA generally resolves CUDA through system packages or the user's own extras; on Mac these are ~1 GB of dead weight):

"nvidia-cublas-cu12>=12.9.2.10 ; sys_platform == 'win32'",
"nvidia-cuda-runtime-cu12>=12.9.79 ; sys_platform == 'win32'",
"nvidia-cudnn-cu12>=9.24.0.43 ; sys_platform == 'win32'",

ears.py — prepend the wheels' bin dirs to PATH before constructing WhisperModel. Needs import os at the top of the file.

def _add_nvidia_dll_dirs():
    """Windows only. pip-installed nvidia-cublas-cu12 / nvidia-cudnn-cu12 /
    nvidia-cuda-runtime-cu12 drop their DLLs under site-packages/nvidia/
    <pkg>/bin, which Windows never adds to the DLL search path on its own.
    os.add_dll_directory() alone isn't enough: CTranslate2's native loader
    falls back to the plain PATH env var. Without this, CUDA fails at first
    inference even though the packages are installed."""
    if sys.platform != "win32":
        return
    import importlib.util
    dirs = []
    for pkg in ("cublas", "cudnn", "cuda_runtime"):
        spec = importlib.util.find_spec(f"nvidia.{pkg}")
        if not spec or not spec.submodule_search_locations:
            continue
        bin_dir = os.path.join(spec.submodule_search_locations[0], "bin")
        if os.path.isdir(bin_dir):
            dirs.append(bin_dir)
    if dirs:
        os.environ["PATH"] = os.pathsep.join(dirs) + os.pathsep + os.environ["PATH"]

Call it in warm() immediately before _model = WhisperModel(...) in the faster-whisper branch. It composes with the existing _probe() CPU fallback: _probe() catches a genuinely broken GPU early; this makes a working GPU actually load so the fallback doesn't fire on a machine that's fine.