feat(dev): add local vLLM model helper

.gitignore

@@ -59,6 +59,7 @@ uv.lock.bak
 # Environments
 .env
 .env.*
+.mise.local.toml
 env/
 venv/
 ENV/

README.md

@@ -154,6 +154,21 @@ anonymizer --help         # CLI usage
 make install-pre-commit   # Install pre-commit hooks
 ```
 
+### Local vLLM debugging
+
+On a Linux GPU host, install the optional local-model dependency group and use
+`tools/vllm_debug.py` to start or probe an OpenAI-compatible vLLM server:
+
+```bash
+uv sync --group dev --group local-models
+uv run python tools/vllm_debug.py models --cached --json
+uv run python tools/vllm_debug.py serve /path/to/cached/snapshot --served-model-name anonymizer-local
+uv run python tools/vllm_debug.py models --endpoint http://127.0.0.1:8000/v1
+```
+
+The helper does not prefetch model weights. Use `--vllm-python /path/to/python`
+when vLLM is installed in another virtual environment. The [local vLLM guide](docs/concepts/local-vllm.md) covers GPU-host requirements, Anonymizer configuration, verification, and internal-network access.
+
 ---
 
 ## Requirements

docs/concepts/local-vllm.md

@@ -0,0 +1,131 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# Run Local vLLM Models
+
+This guide is for contributors who operate an Anonymizer source checkout on a
+Linux host with NVIDIA GPUs. It starts a local, OpenAI-compatible [vLLM](https://docs.vllm.ai/)
+endpoint for development, evaluation, or an internal deployment. It does not
+download a model or configure production networking for you.
+
+The helper script, [`tools/vllm_debug.py`](https://github.com/NVIDIA-NeMo/Anonymizer/blob/main/tools/vllm_debug.py), is a source-tree development tool. It is not included in the `nemo-anonymizer` package.
+
+## Host Requirements
+
+Before installing the local-model dependencies, confirm that the host has:
+
+- Linux, Python 3.11 or later, and an NVIDIA GPU that has enough free VRAM for the selected model and its context length.
+- An NVIDIA driver that can communicate with the GPU. Verify this with `nvidia-smi`.
+- Access to the model weights, either through the Hugging Face cache or a local model directory. Observe the model license and access controls.
+
+The dependency group installs vLLM and its CUDA-compatible dependencies. Driver, GPU, model-size, and CUDA compatibility remain properties of the host. Start with a small model and a short context length before adopting a model for a benchmark or workflow.
+
+## Install the Local-Model Environment
+
+From a source checkout, install the development and local-model groups:
+
+```bash
+uv sync --group dev --group local-models
+nvidia-smi
+uv run python -c 'import torch; assert torch.cuda.is_available(); print(torch.cuda.get_device_name(0))'
+```
+
+If vLLM is kept in a separate virtual environment, the helper can use that interpreter with `--vllm-python /path/to/python`. This is useful when the model-serving environment has different CUDA constraints from the Anonymizer development environment.
+
+## Select a Cached Model
+
+The helper discovers snapshots already in the Hugging Face hub cache. It does not fetch model weights:
+
+```bash
+uv run python tools/vllm_debug.py models --cached --json
+```
+
+Pass a `snapshot_path` from that output to `serve`. A Hugging Face model ID is also accepted by vLLM, but may trigger a download.
+
+## Start and Verify a Server
+
+Run the server from a terminal that will remain open:
+
+```bash
+uv run python tools/vllm_debug.py serve /path/to/cached/snapshot \
+  --served-model-name anonymizer-local \
+  --gpu-memory-utilization 0.85 \
+  --max-model-len 8192
+```
+
+The default bind address, `127.0.0.1:8000`, keeps the endpoint local to the GPU host. `--served-model-name` gives clients a stable model ID, independent of the cache directory name. Stop the server with `Ctrl-C`.
+
+From another terminal, verify both the model registration and one completion:
+
+```bash
+uv run python tools/vllm_debug.py models
+uv run python tools/vllm_debug.py call \
+  --model anonymizer-local \
+  --prompt 'Reply with the word ready.' \
+  --timeout-seconds 120
+```
+
+Use `--dry-run` with `serve` to print the vLLM command before reserving GPU memory. For multi-GPU models, add `--tensor-parallel-size N`. For a LoRA adapter, add `--adapter /path/to/adapter` and, if needed, `--adapter-name NAME`.
+
+## Connect Anonymizer
+
+vLLM presents an OpenAI-compatible endpoint. Add it to a custom provider file:
+
+```yaml title="providers.yaml"
+providers:
+  - name: local-vllm
+    endpoint: http://127.0.0.1:8000/v1
+    provider_type: openai
+    api_key: EMPTY  # vLLM has no API key unless one is configured below
+```
+
+In a custom `models.yaml`, set each local model configuration's `provider` to `local-vllm` and its `model` to the server's served-model name, `anonymizer-local` in the example above. Then pass both files to `Anonymizer`:
+
+```python
+from anonymizer import Anonymizer
+
+anonymizer = Anonymizer(
+    model_providers="providers.yaml",
+    model_configs="models.yaml",
+)
+```
+
+`model_configs` replaces Anonymizer's entire bundled model pool. Copy the bundled [`models.yaml`](https://github.com/NVIDIA-NeMo/Anonymizer/blob/main/src/anonymizer/config/default_model_configs/models.yaml), retain every alias required by the roles you use, and change only the models that should route to vLLM. See [Custom models](models.md#custom-models) for the role map and validation command.
+
+Use `anonymizer.validate_config(config)` before processing data. A successful HTTP probe only confirms that the server responds. It does not establish that a model is suitable for detection, replacement, or privacy-preserving rewrite quality.
+
+## Access From Another Host
+
+Keep the default loopback binding whenever Anonymizer and vLLM run on the same machine. If a trusted internal client needs the endpoint, use a network policy or firewall as well as a server API key:
+
+```bash
+# .env.local is ignored by this repository. Do not commit it.
+LOCAL_VLLM_API_KEY='replace-with-a-long-random-secret'
+
+# Load it in the shell that starts the server and the client.
+set -a; source .env.local; set +a
+
+uv run python tools/vllm_debug.py serve /path/to/cached/snapshot \
+  --host 0.0.0.0 \
+  --served-model-name anonymizer-local \
+  --api-key-env LOCAL_VLLM_API_KEY
+```
+
+Set the same secret's environment-variable name in the Anonymizer provider configuration:
+
+```yaml title="providers.yaml"
+providers:
+  - name: local-vllm
+    endpoint: http://gpu-host.internal:8000/v1
+    provider_type: openai
+    api_key: LOCAL_VLLM_API_KEY
+```
+
+Use an ignored `.env.local` file, or an ignored `.mise.local.toml` file when your checkout uses Mise, to keep local endpoint credentials out of version control. Load the secret only in the shell or task runner that starts the server and client. Do not expose a raw vLLM endpoint to the public internet. For production access, place it behind your organization's authenticated TLS-enabled network boundary.
+
+## Operating Notes
+
+- List the server's registered model IDs with `models` after every model or adapter change. Client `model` values must match one of those IDs.
+- Start conservatively with `--gpu-memory-utilization` and `--max-model-len`; increase them only after observing stable GPU memory use and latency.
+- Keep GLiNER separate from the LLM when GPU memory is constrained. The [self-hosted GLiNER guide](self-hosting-gliner.md) describes the detection endpoint.
+- Treat local-model output as untrusted until it has passed the same Anonymizer preview, evaluation, and privacy review used for any other provider.

docs/concepts/models.md

@@ -35,6 +35,8 @@ Each pipeline stage has a **role** mapped to one of these aliases. See the full
 
 Pass `model_providers` when you need a non-default endpoint — for example OpenAI, OpenRouter, a local GLiNER server, or an internal inference deployment. Plain `Anonymizer()` already uses bundled [build.nvidia.com](https://build.nvidia.com) settings; override only when your models point at a different provider name or URL.
 
+For a GPU-hosted OpenAI-compatible LLM endpoint, see [Run local vLLM models](local-vllm.md). That guide covers the source-tree helper, optional dependencies, and a `local-vllm` provider configuration.
+
 Set your API keys first:
 
 ```bash

mkdocs.yml

@@ -161,6 +161,7 @@ nav:
       - Choosing a Strategy: concepts/choosing-a-strategy.md
       - Evaluation: concepts/evaluation.md
       - Self-hosting GLiNER: concepts/self-hosting-gliner.md
+      - Run Local vLLM Models: concepts/local-vllm.md
   - Troubleshooting: troubleshooting.md
   - Tutorials:
       - Overview: tutorials/index.md

pyproject.toml

@@ -53,6 +53,9 @@ notebooks = [
     "jupytext>=1.16.0,<2",
     "pillow>=12.0.0,<13",
 ]
+local-models = [
+    "vllm==0.20.0; sys_platform == 'linux'",
+]
 
 [build-system]
 requires = ["hatchling", "uv-dynamic-versioning"]

tests/tools/test_vllm_debug.py

@@ -0,0 +1,184 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Behavior tests for the local vLLM debug helper."""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+from types import ModuleType
+from typing import Any
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+TOOL_PATH = REPO_ROOT / "tools" / "vllm_debug.py"
+
+
+def load_tool() -> ModuleType:
+    spec = importlib.util.spec_from_file_location("vllm_debug_tool", TOOL_PATH)
+    assert spec is not None
+    assert spec.loader is not None
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def test_build_serve_command_includes_lora_and_gpu_options() -> None:
+    tool = load_tool()
+
+    command = tool.build_serve_command(
+        tool.ServeRequest(
+            model="HuggingFaceTB/SmolLM3-3B",
+            host="127.0.0.1",
+            port=8000,
+            served_model_name="anonymizer-local",
+            api_key="test-token",
+            adapter=Path("/models/adapter"),
+            adapter_name="anonymizer",
+            tensor_parallel_size=2,
+            gpu_memory_utilization=0.8,
+            max_model_len=4096,
+            eager=True,
+        )
+    )
+
+    assert command == [
+        sys.executable,
+        "-m",
+        "vllm.entrypoints.openai.api_server",
+        "--model",
+        "HuggingFaceTB/SmolLM3-3B",
+        "--host",
+        "127.0.0.1",
+        "--port",
+        "8000",
+        "--served-model-name",
+        "anonymizer-local",
+        "--api-key",
+        "test-token",
+        "--enable-lora",
+        "--lora-modules",
+        "anonymizer=/models/adapter",
+        "--tensor-parallel-size",
+        "2",
+        "--gpu-memory-utilization",
+        "0.8",
+        "--max-model-len",
+        "4096",
+        "--enforce-eager",
+    ]
+
+
+def test_build_serve_command_can_use_a_separate_vllm_interpreter() -> None:
+    tool = load_tool()
+
+    command = tool.build_serve_command(
+        tool.ServeRequest(
+            model="local-model",
+            python_executable=Path("/opt/vllm/bin/python"),
+        )
+    )
+
+    assert command[:5] == [
+        "/opt/vllm/bin/python",
+        "-m",
+        "vllm.entrypoints.openai.api_server",
+        "--model",
+        "local-model",
+    ]
+
+
+def test_render_serve_command_redacts_api_key() -> None:
+    tool = load_tool()
+
+    rendered = tool.render_serve_command(
+        tool.build_serve_command(tool.ServeRequest(model="local-model", api_key="test-token"))
+    )
+
+    assert "test-token" not in rendered
+    assert "--api-key '<redacted>'" in rendered
+
+
+def test_resolve_api_key_reads_a_named_environment_variable(monkeypatch: Any) -> None:
+    tool = load_tool()
+    monkeypatch.setenv("LOCAL_VLLM_API_KEY", "test-token")
+
+    assert tool.resolve_api_key(None, "LOCAL_VLLM_API_KEY") == "test-token"
+
+
+def test_discover_cached_models_returns_snapshot_paths(tmp_path: Path) -> None:
+    tool = load_tool()
+    snapshot = tmp_path / "models--HuggingFaceTB--SmolLM3-3B" / "snapshots" / "abc123"
+    snapshot.mkdir(parents=True)
+    (snapshot / "config.json").write_text("{}", encoding="utf-8")
+
+    models = tool.discover_cached_models(tmp_path)
+
+    assert models == [
+        tool.CachedModel(
+            repository="HuggingFaceTB/SmolLM3-3B",
+            snapshot_path=snapshot,
+        )
+    ]
+
+
+def test_fetch_models_uses_openai_models_endpoint(monkeypatch: Any) -> None:
+    tool = load_tool()
+    calls: list[str] = []
+
+    class Response:
+        def raise_for_status(self) -> None:
+            return None
+
+        def json(self) -> dict[str, object]:
+            return {"data": [{"id": "local-model"}]}
+
+    def fake_get(url: str, *, timeout: float) -> Response:
+        calls.append(url)
+        assert timeout == 10.0
+        return Response()
+
+    monkeypatch.setattr(tool.httpx, "get", fake_get)
+
+    assert tool.fetch_models("http://127.0.0.1:8000/v1", timeout_seconds=10.0) == ["local-model"]
+    assert calls == ["http://127.0.0.1:8000/v1/models"]
+
+
+def test_call_chat_sends_prompt_and_returns_content(monkeypatch: Any) -> None:
+    tool = load_tool()
+    request_body: dict[str, object] = {}
+
+    class Response:
+        def raise_for_status(self) -> None:
+            return None
+
+        def json(self) -> dict[str, object]:
+            return {
+                "choices": [{"message": {"content": "hello"}}],
+                "usage": {"prompt_tokens": 3, "completion_tokens": 1},
+            }
+
+    def fake_post(url: str, *, json: dict[str, object], timeout: float, headers: dict[str, str]) -> Response:
+        request_body.update(json)
+        assert url == "http://127.0.0.1:8000/v1/chat/completions"
+        assert timeout == 15.0
+        assert headers == {}
+        return Response()
+
+    monkeypatch.setattr(tool.httpx, "post", fake_post)
+
+    result = tool.call_chat(
+        endpoint="http://127.0.0.1:8000/v1",
+        model="local-model",
+        prompt="Say hello",
+        timeout_seconds=15.0,
+        api_key=None,
+    )
+
+    assert request_body == {
+        "model": "local-model",
+        "messages": [{"role": "user", "content": "Say hello"}],
+    }
+    assert result.content == "hello"
+    assert result.usage == {"prompt_tokens": 3, "completion_tokens": 1}