Skip to content

Add configurable ONNX provider selection - #23

Merged
HanClinto merged 8 commits into
mainfrom
provider-auto-gpu-issue-22
Jul 10, 2026
Merged

Add configurable ONNX provider selection#23
HanClinto merged 8 commits into
mainfrom
provider-auto-gpu-issue-22

Conversation

@hanclintoclaw-pixel

@hanclintoclaw-pixel hanclintoclaw-pixel commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes #22.

This adds a small provider option to both neural runtime classes:

  • provider="auto" is the default and prefers installed accelerator providers before CPU
  • provider="cpu" forces CPU for predictable behavior
  • provider="gpu" requires an accelerated ONNX Runtime provider and raises a clear error if none are available

The implementation keeps the public API intentionally small (auto / cpu / gpu) and hides ONNX Runtime provider names behind an internal resolver. That matches CollectorVision's simple quickstart-oriented API while still making GPU acceleration possible for users with the right runtime installed.

CollectorVision no longer declares ONNX Runtime as a dependency. Users install exactly one ONNX Runtime backend appropriate for their machine (onnxruntime, onnxruntime-gpu, DirectML/ROCm variants, etc.), and CollectorVision gives an actionable error the first time neural inference is used without a backend installed. It also warns if both onnxruntime and onnxruntime-gpu distributions are present because they provide the same Python module and can conflict.

Rationale

Before this change, NeuralCornerDetector and NeuralEmbedder both hardcoded providers=["CPUExecutionProvider"], so installing onnxruntime-gpu was not enough to use GPU acceleration through CollectorVision. The new default behavior is accelerator-first because that is what most users expect when a capable provider is installed, but it still falls back to CPU if automatic accelerator session creation fails.

Explicit gpu does not fall back silently because a user forcing acceleration should get a visible setup error if no accelerator provider is available or the runtime/drivers are not correct. ONNX Runtime can still use CPU as a graph fallback behind an accelerator provider when only part of a model is supported.

I did not add broad ONNX Runtime provider-list plumbing to the main API because it would expose too many magic strings for a library that is intended to stay easy to use. gpu is the meaningful user-facing choice across CoreML, CUDA, DirectML, ROCm, and future accelerator providers; we can add advanced provider-list support later if a concrete use case needs it.

Testing on train4070 exposed two packaging/runtime issues that shaped this final approach:

  • Extras cannot cleanly replace a base onnxruntime dependency, so GPU installs can accidentally install both onnxruntime and onnxruntime-gpu, causing the CPU wheel to win and hide CUDA.
  • Latest onnxruntime-gpu 1.27 requires CUDA 13 runtime libraries on train4070, while 1.26 plus CUDA/cuDNN Python runtime packages works on CUDA 12. CollectorVision should not globally pin that decision because newer CUDA users should be able to use newer ONNX Runtime builds.

Validation

  • uv run --with pytest python -m pytest tests/test_onnx_providers.py tests/test_interfaces.py
  • uv run --with ruff ruff check collector_vision/onnx_providers.py collector_vision/detectors/neural.py collector_vision/embedders/neural.py tests/test_onnx_providers.py
  • Smoke-tested NeuralEmbedder and NeuralCornerDetector with provider="cpu", provider="gpu" on CoreML, and default provider="auto"
  • Fresh install with no ONNX Runtime backend imports collector_vision successfully and raises actionable backend-install guidance when NeuralEmbedder() is constructed
  • Warns once per process when both onnxruntime and onnxruntime-gpu distributions are installed
  • On train4070 Linux/CUDA/RTX 4070, fresh pip install -e . followed by manual pip install "onnxruntime-gpu<1.27" nvidia-cudnn-cu12 nvidia-cuda-runtime-cu12 exposed CUDAExecutionProvider, passed targeted tests, and ran real detector + embedder inference with active providers ["CUDAExecutionProvider", "CPUExecutionProvider"]

Add a simple provider option for NeuralCornerDetector and NeuralEmbedder. The default auto mode tries available accelerators first and falls back to CPU, while cpu and cuda allow callers to force predictable behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
Replace the public cuda provider value with gpu so callers can request any available accelerator without learning ONNX Runtime provider names. Keep CUDA as an install/runtime detail behind the gpu extra and provider resolver.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
Move ONNX Runtime out of required dependencies so the gpu extra can install the accelerator runtime without the CPU wheel overriding it. Add a cpu extra for the normal install path and keep gpu as the public accelerator install path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
Constrain the Linux GPU extra to onnxruntime-gpu<1.27 after testing on train4070 showed 1.27 requires libcudart.so.13 while the CUDA 12 runtime is still common. This keeps provider='gpu' installable on the CUDA 12 host used for validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
Preload NVIDIA runtime libraries when CUDA or TensorRT providers are requested, add CUDA runtime dependencies to the gpu extra, and reject provider='gpu' sessions that silently initialize CPU-only. This matches the public contract that gpu means acceleration is required.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
Leave the default gpu extra free to install the latest ONNX Runtime GPU wheel so newer CUDA systems are not pinned back. Add gpu-cu12 as the compatibility path for CUDA 12 hosts like train4070.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
Remove ONNX Runtime extras so CollectorVision does not choose or conflict between CPU and GPU runtime wheels. Keep import lightweight, and raise actionable neural-inference guidance when no backend is installed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
Detect when both onnxruntime and onnxruntime-gpu distributions are installed and warn once per process. This catches the common conflict where both distributions provide the same onnxruntime module and one can hide GPU providers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0a8ed24a-df35-426a-80b6-e40c4f5f7308
@HanClinto
HanClinto merged commit ffabe92 into main Jul 10, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add GPU support

2 participants