tabby_server_5050.log
Environment
- OS: Windows 11
- GPUs: 2x RTX 5070 Ti, 16 GB each, no NVLink (PCIe 5.0 x8 per card, connected via CPU lanes, not chipset)
- exllamav3: 0.0.43+cu132.torch2.11.0 (latest on PyPI at time of report, confirmed via
pip index versions exllamav3)
- CUDA: 13.2 / PyTorch: 2.11.0 (from the build tag above)
- Backend: TabbyAPI (OpenAI-compatible server on top of exllamav3)
- Model: Qwen3-VL-32B-Instruct-exl3 (vision-language, 33 vision modules)
- Launch command:
python main.py --host 127.0.0.1 --port 5050 --model-dir C:\tabbyAPI\models --model-name Qwen3-VL-32B-Instruct-exl3 --vision True --inline-model-loading True --max-seq-len 98304 --cache-size 98304 --cache-mode 6,4 --tensor-parallel True --tensor-parallel-backend native
What happened
The model loads successfully with tensor_parallel enabled — all layers and all 33 vision modules load without error, and two additional worker processes are spawned (one per GPU), confirming TP is active (each logs "Redirects are currently not supported in Windows").
The crash happens on the very first inference request (a single image + text prompt). The server dies with a GPU assert:
CUDA error: an illegal memory access was encountered
GPU assert at exllamav3_ext/parallel/barrier.cu:47
raised inside the call chain prefill_tp -> prepare_for_device -> x.to(self.device). The process aborts entirely (server exits, no HTTP error is returned — connection is simply reset), so every subsequent request fails with a connection error until the server is manually restarted.
With tensor_parallel disabled (standard layer-split across the two GPUs), the exact same model, prompt, and image work correctly with no errors. This is fully reproducible — every run with TP enabled crashes on the first request, every run with TP disabled works.
Suspected cause
My understanding is that the native TP backend synchronizes GPUs across separate OS processes using CUDA IPC / peer-to-peer memory access. On Windows, GeForce cards run under the WDDM driver model, where CUDA IPC/P2P between processes is not available (this is a platform-level limitation, not specific to this app) — so the very first cross-GPU barrier faults.
Questions
- Is Tensor Parallel currently expected to work on Windows at all, or is it effectively Linux-only given the CUDA IPC/P2P dependency?
- Is a host-staged fallback (routing inter-GPU activation exchange through pinned host memory instead of direct IPC) on the roadmap, or considered feasible?
- Anything else I should try (e.g. a different
tensor_parallel_backend value, an older exllamav3 build, or a different CUDA/PyTorch combination) before ruling this out?
Attached: full tabby_server_*.log from this run, covering model load through the crash.
Happy to test any patches, debug builds, or alternate CUDA/PyTorch combinations on this hardware (2x RTX 5070 Ti, Windows 11) if that's useful — this seems like a configuration that might not be well represented among your existing testers.
Thanks for the great project!
tabby_server_5050.log
Environment
pip index versions exllamav3)What happened
The model loads successfully with tensor_parallel enabled — all layers and all 33 vision modules load without error, and two additional worker processes are spawned (one per GPU), confirming TP is active (each logs "Redirects are currently not supported in Windows").
The crash happens on the very first inference request (a single image + text prompt). The server dies with a GPU assert:
raised inside the call chain
prefill_tp -> prepare_for_device -> x.to(self.device). The process aborts entirely (server exits, no HTTP error is returned — connection is simply reset), so every subsequent request fails with a connection error until the server is manually restarted.With
tensor_paralleldisabled (standard layer-split across the two GPUs), the exact same model, prompt, and image work correctly with no errors. This is fully reproducible — every run with TP enabled crashes on the first request, every run with TP disabled works.Suspected cause
My understanding is that the native TP backend synchronizes GPUs across separate OS processes using CUDA IPC / peer-to-peer memory access. On Windows, GeForce cards run under the WDDM driver model, where CUDA IPC/P2P between processes is not available (this is a platform-level limitation, not specific to this app) — so the very first cross-GPU barrier faults.
Questions
tensor_parallel_backendvalue, an older exllamav3 build, or a different CUDA/PyTorch combination) before ruling this out?Attached: full
tabby_server_*.logfrom this run, covering model load through the crash.Happy to test any patches, debug builds, or alternate CUDA/PyTorch combinations on this hardware (2x RTX 5070 Ti, Windows 11) if that's useful — this seems like a configuration that might not be well represented among your existing testers.
Thanks for the great project!