Inferencing on HP Z8 Workstations

This repository contains instructions for serving and benchmarking Large Language Models with vLLM. The content focuses primarily on NVIDIA GPU-based inference, with deployment paths for local serving, container-based serving, benchmarking, and Kubernetes-based model serving.

Choose a Deployment Path

Choose the section that matches your deployment model:

• setup-local-serving/: Run vLLM directly on a single machine.
• setup-docker-serving/: Run vLLM in a containerized environment.
• benchmarking/: Measure throughput, latency, and load behavior.
• setup-kubernetes-serving/: Build a Kubernetes cluster for multi-node model-serving workflows.

If you are new to this repository:

• Use the local or Docker instructions for a quick evaluation on one machine.
• Use the Kubernetes instructions if you need a cluster-based deployment with observability, routing, and production-style serving workflows.

Prerequisites

Linux

vLLM officially supports Linux. For development or evaluation on Windows, you can use:

• Docker
• Windows Subsystem for Linux (WSL)

WSL creates virtualization and memory overhead that impacts vLLM's performance

NVIDIA Driver

Check whether the NVIDIA driver is already installed:

nvidia-smi

If the driver is not installed yet, start with the repository guide at install-nvidia-drivers/README.md. That document walks through NVIDIA driver installation, CUDA Toolkit setup, cuDNN installation, and validation steps for Ubuntu-based systems.

For vendor documentation, you can also refer to NVIDIA's Driver Installation Guide and look up the latest driver for your GPU at NVIDIA Driver Downloads.

Python

At the time of writing, vLLM supports Python 3.10 through 3.13. Check your Python version with:

python3 --version

If Python is not installed, install it with:

sudo apt update
sudo apt install python3.12 -y

Virtual environment (optional)

Although optional, using a virtual environment such as Miniconda or Python's built-in venv is strongly recommended. Miniconda installation documentation can be found here, and you can create and activate an environment with:

conda create -n myenv python=3.12 -y
conda activate myenv

You can create and activate a Python virtual environment with:

python3 -m venv .venv
source .venv/bin/activate

• Use either Miniconda or Python's virtual environment, not both.
• myenv is a placeholder name for the environment.

Install vLLM

You can install and verify vLLM on your machine with:

pip install --upgrade uv
uv pip install vllm --torch-backend=auto

uv is an open-source package manager that can help resolve dependency conflicts and select compatible package versions.

Verify your installation with:

vllm --version

Choose a Model

When choosing a model, keep in mind that vLLM needs GPU memory for both model weights and the KV cache. As a starting point, choose a model that fits within about 70% of the available GPU memory. To browse models, use Hugging Face and review the size information in the Files and versions tab.

Download Models

You can download models using the Hugging Face CLI:

hf download path/to/model

By default, models are cached in /home/user/.cache/huggingface/hub. To avoid repeated downloads during local serving, pass the absolute path to the downloaded model snapshot. You can also change the Hugging Face cache directory:

export HF_HOME="/path/to/your/new/cache"

Kubernetes Cluster Setup

If you want to deploy model serving on a Kubernetes cluster, start with setup-kubernetes-serving/README.md.

That guide explains the recommended order for the cluster setup workflow:

1. Configure the control plane nodes.
2. Add worker nodes.
3. Install Loki.
4. Install Grafana.
5. Deploy vLLM.
6. Configure KGateway.

The Kubernetes setup path is intended for customers who need one or more of the following:

• Multi-node GPU serving
• HTTPS ingress and request routing
• Centralized logging and monitoring
• Cluster-managed model-serving operations

Contents

• setup-local-serving: Model serving using a single node (machine).
• setup-docker-serving: Container-based vLLM deployment examples.
• benchmarking: Load testing and performance benchmarking instructions.
• setup-kubernetes-serving: End-to-end Kubernetes cluster setup and platform integration guides.