KubeVirt GPU passthrough on Kubernetes

Ansible automation that turns a bare Ubuntu host into a Kubernetes node running virtual machines with a physically passed-through GPU — including a Windows guest reached over RDP.

Containers are the easy case. Some workloads are not containerisable: a Windows application, something that needs a real GPU with the vendor driver inside the guest, or software that simply predates all of this. KubeVirt runs those as VMs under the same scheduler and API as everything else, and this repository automates the awkward part — getting a physical GPU out of the host and into the guest, and proving it actually arrived.

What it builds

Ubuntu host with an IOMMU-isolated GPU bound to vfio-pci
  └── kubeadm single-node Kubernetes  (pinned 1.35.x)
        ├── containerd + Calico CNI
        ├── local-path storage
        ├── MinIO            object store for VM disk images
        ├── CDI              imports a qcow2 into a PVC
        ├── KubeVirt         VM lifecycle  (v1.8.4, HostDevices gate)
        ├── MetalLB          LoadBalancer addresses  (optional)
        └── Windows VM       the GPU, passed through, reachable over RDP

The part that is actually hard

Passing a GPU into a VM is a chain where every link fails silently. This automation checks each one rather than assuming it:

The card must be bound to vfio-pci, and must stay bound. Install the NVIDIA driver on the host and the card leaves vfio-pci and can never be handed to a VM again. The role re-confirms binding before and after every operation.

KubeVirt must be told the device is permitted. permittedHostDevices maps a PCI vendor/device selector to a resource name; without it the GPU is invisible to the scheduler no matter what the host says.

The resource must actually be advertised. The role waits for the node to publish the GPU resource rather than proceeding hopefully.

And then it proves it. Before touching a real workload, 06-gpu.yml spins up a temporary Linux validation VM, asserts the VMI requests the device, asserts the virt-launcher pod really received one, tears the VM down, and re-confirms vfio-pci ownership. If passthrough is broken you find out in a disposable VM, not in production.

Two failures worth knowing about

MetalLB announces from nothing, silently. kubeadm labels control-plane nodes node.kubernetes.io/exclude-from-external-load-balancers, and MetalLB refuses to announce from a node carrying it. On a single-node cluster that means nothing announces — while the Service shows an EXTERNAL-IP, the pod is Running, endpoints are ready, ARP responders exist, and no component logs an error. ARP requests simply go unanswered. The metallb role removes the label and then asserts at least one node is eligible, so it cannot pass silently again.

Windows reads the virtual RTC as local time; KubeVirt presents UTC. The guest therefore corrects its clock backwards at every boot. Repeated backward jumps are what flag a Windows licensing store as tampered — after which it refuses both activation and rearm. The VM template pins the RTC timezone, and it must match the timezone set inside Windows.

The pipeline

Each stage is a numbered playbook and can be run alone.

Playbook	Does
01-preflight.yml	Verifies OS release, IOMMU groups, GPU count and PCI addresses, vfio-pci binding; refuses to continue over a residual k3s install
02-kubernetes.yml	containerd, kubeadm init, Calico CNI, pinned package versions
03-storage.yml	local-path provisioner as the default StorageClass
04-kubevirt.yml	KubeVirt operator and CR, HostDevices feature gate, virtctl
05-minio-cdi.yml	MinIO plus CDI, and uploads the VM image if one is configured
06-gpu.yml	permittedHostDevices, waits for the resource, runs the disposable validation VM
07-windows-vm.yml	Imports the qcow2 via CDI and boots the Windows guest with the GPU
08-validation.yml	End-to-end assertions over the whole stack
09-metrics-server.yml	metrics-server
10-windows-vm-runtime.yml	Runtime checks against a running guest
11-lifecycle-validation.yml	Repeated stop/start cycles, asserting graceful shutdown and GPU re-attachment. Opt-in
13-metallb.yml	MetalLB and an address pool. Optional
14-windows-vm-reimport.yml	Destructive rebuild from the golden image. Requires an explicit confirm flag

playbooks/site.yml runs 01–08. Stop after 06-gpu.yml to get a working cluster with proven GPU passthrough and no guest image at all.

Requirements

• Ubuntu 24.04 host with sudo.
• IOMMU enabled in firmware and on the kernel command line, with the GPU isolated in its own IOMMU group and bound to vfio-pci. Preflight verifies all of this and stops if it is not true.
• No vendor GPU driver on the host. The driver belongs inside the guest.
• Ansible Core on the controller; ansible.utils from requirements.yml.
• For a Windows guest: a prepared qcow2 already on the host. Ansible deliberately does not move multi-gigabyte images through copy.

Quick start

git clone https://github.com/mdalgitsis/kubevirt-gpu-passthrough.git
cd kubevirt-gpu-passthrough
make collections

cp -r inventories/example inventories/mysite
$EDITOR inventories/mysite/hosts.yml            # your host
$EDITOR inventories/mysite/group_vars/all.yml   # GPU PCI id, versions, VM shape

# Cluster + KubeVirt + proven GPU passthrough, no guest image needed
ansible-playbook -i inventories/mysite/hosts.yml playbooks/01-preflight.yml
ansible-playbook -i inventories/mysite/hosts.yml playbooks/02-kubernetes.yml
# ... through 06-gpu.yml

# Or the whole thing, once windows_vm_image_remote_path points at your qcow2
ansible-playbook -i inventories/mysite/hosts.yml playbooks/site.yml

Set expected_gpu_pci_id, gpu_pci_vendor_selector and gpu_resource_name for your card. The defaults describe an NVIDIA L40S (10de:26b9).

Validating

make lint       # yamllint + ansible-lint
make syntax     # syntax-check every playbook
make inventory  # render the inventory graph

CI runs all of these. ansible-lint passes at the production profile.

Known limitations

• Single node. kubeadm init with no join flow; the control-plane node also runs the VMs.
• One GPU, whole-card passthrough. No MIG, no vGPU, no time-slicing — the card goes to one VM entirely.
• Ubuntu 24.04 only. Preflight asserts the release rather than pretending to support others.
• The Windows guest is assumed prepared. Building the image — installing the driver, enabling RDP, setting the timezone — is a manual step this automation does not perform.
• No idempotent reversal. There is a destructive reimport, but no clean "undo the cluster".

Acknowledgements

Developed for the UNITY-6G project, in a proof of concept carried out with CTTC (Centre Tecnològic de Telecomunicacions de Catalunya).

The automation here is the generic, reusable part. Partner infrastructure details, the specific application that ran inside the Windows guest, and the orchestration platform used around it are deliberately not included — this repository stands alone and needs nothing beyond Ansible and the host.

Built on KubeVirt, CDI, MinIO, Calico and MetalLB, each the work of its own maintainers.

License

Apache License 2.0.