Skip to content

CPUBreakdown.md

Latest commit

 

History

History
118 lines (100 loc) · 10.5 KB

CPUBreakdown.md

File metadata and controls

118 lines (100 loc) · 10.5 KB
Raw

Recommended Arguments for Gaming and Workstations

CPU Arguments ideal for both workstations and gaming, ensures the virtual machine cannot tell it is not running on baremetal:

-cpu 'host,host-cache-info=on,l3-cache=on,topoext=on,+invtsc,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_crash,hv_ipi,hv_vendor_id=AMDVENDOR,host-phys-bits=true,hypervisor=off,kvm=off'

Arguments for full CPU passthrough - ideal for workstations and creative works (7970X):

-smp '64,cores=8,threads=2,sockets=4,maxcpus=64'

Arguments for partial CPU passthrough - ideal for gaming:

-smp '16,cores=16,threads=1,sockets=1,maxcpus=16'

The main goal with creating the gaming system is to prevent L3 thrashing, to do this you ideally will have less than 2 NUMA nodes and will limit the total amount of cores as well. Hyperthreads/SMT are not as effective as the main cores (1->0.5 effectively), disabling these and providing a max of two CCDs with primary cores will provide the best results.

To get the performance using a multi NUMA node/CCD system you need to use the CCD attached to the GPU PCI lane and only use the memory attached to that CCD.

Breakdown of args: lines

1. SMP Topology (-smp …)

Parameter 16-core VM (Gaming) 64-core VM (Workstation) What it is Why it helps
-smp 16 64 Total number of vCPUs presented to the guest Matches the guest’s CPU count to the host for max performance
cores 16 8 Cores per socket Defines how many “real” cores each virtual socket has—aligns with host cache topology
threads 1 2 Threads per core (SMT/hyperthreading) More threads can boost throughput on SMT-capable CPUs
sockets 1 4 Virtual CPU sockets Splitting into multiple sockets can help NUMA-aware OSes optimize locality
maxcpus 16 64 Maximum hot-plug-able vCPUs Lets you add vCPUs at runtime up to this limit (must match total vCPU count)

Splitting the standard configuration of the NPS1 on a Threadripper system to NPS4 (or forcing the VM to make the assumption) significantly helps the operating system best perform as it understands it's environment. The Infinity Fabric is fast but not as fast as the IPC on a NUMA Node communicating to itself. Forcing a 7970X to assume it is effectively 4x 7700X CPUs will provide better performance.

2. CPU Feature Flags (-cpu …)

Flag What it is Why it helps
host Use the host’s exact CPU model Exposes every feature your physical CPU has, minimizing emulation overhead
host-cache-info=on Expose host cache hierarchy (sizes, sharing) Allows guest OS to optimize cache usage exactly like on bare-metal
l3-cache=on Specifically expose L3 cache topology Improves scheduling decisions for L3-sensitive workloads
topoext=on Enable topology extensions Lets the guest see extended CPU topology (e.g. dies, CCDs)
+invtsc Invariant TSC (time stamp counter) Provides a stable clock source across P-states, C-states, and VM migrations
+kvm_pv_unhalt Paravirtualized halt/unhalt Speeds up CPU idle and wake-up paths
+kvm_pv_eoi Paravirtualized End-Of-Interrupt Fast EOI handling—lowers interrupt exit latency
hv_spinlocks=0x1fff Hypervisor-spinlock mask Optimizes guest spin-lock loops to reduce VM-exit storms
hv_vapic Virtualized local APIC Faster interrupt routing inside the VM
hv_time Paravirtualized clock High-precision, low-overhead time keeping
hv_reset Hypervisor-driven reset interface Allows the host to reset the VM quickly
hv_vpindex Virtual processor index Guest can read its vCPU ID directly
hv_runtime Paravirtualized runtime interface Provides a generic hypervisor-guest API for improved sync
hv_relaxed Relaxed VM exit conditions Reduces exits on RDTSC and other timing instructions
hv_crash Crash notification Propagates host crashes to the guest OS immediately
hv_ipi Paravirtualized inter-processor interrupts Low-latency IPIs between vCPUs
hv_vendor_id=AMDVENDOR Override CPUID vendor string Masks “KVM” vendor, useful to bypass hypervisor checks (EAC, anti-cheat)
host-phys-bits=true Expose full host physical address width Lets the guest address all RAM on large-memory systems
hypervisor=off Hide hypervisor bit in CPUID Makes the guest believe it’s on bare-metal
kvm=off Hide KVM presence Further masks virtualization, avoiding certain software checks

NUMA Nodes and CPU Affinity

To check your system to query which CPUs are ideal for passing/utilizing, use the following command:

numactl --hardware

Output:

available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 31723 MB
node 0 free: 2760 MB
node 1 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 1 size: 32250 MB
node 1 free: 3747 MB
node 2 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 2 size: 32250 MB
node 2 free: 4960 MB
node 3 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 3 size: 32169 MB
node 3 free: 4378 MB
node distances:
node   0   1   2   3 
  0:  10  12  12  12 
  1:  12  10  12  12 
  2:  12  12  10  12 
  3:  12  12  12  10

Gaming VM

On this system (7970X with 4x 32GB of memory), you will want to pass specific CPUs/cores for certain scenarios for the best performance. From testing the best result for gaming was the following:

affinity: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
numa: 1
numa0: cpus=0-7,hostnodes=0,memory=24576,policy=bind
numa1: cpus=8-15,hostnodes=1,memory=24576,policy=bind

This utilizes two NUMA nodes, allocating the main cores of CCD0 (0-7) and CCD1 (8-15) to the VM. For memory, binding provides the most consistent results. This requires you to minimize the cross node communication for memory access. To do this it is best to use only the memory available on each NUMA node while leaving some available for the host. In my case, I allocated 24,576MiB from both CCD0 and CCD1. Enabling NUMA also ensures that the guest VM understands the layout and can schedule/allocate accordingly.

Workstation VM

For the workstation VM, you can allocate both SMT/HT as well as primary cores. The typical ordering is to use the main core/hyperthread, so on. For Node 0 on the numactl command, you can see that 0-7 are primary cores and 32-39 are HT/SMT, these must be interleaved for the best performance as it is what the guest VM will expect for ordering.

affinity: 0,32,1,33,2,34,3,35,4,36,5,37,6,38,7,39,8,40,9,41,10,42,11,43,12,44,13,45,14,46,15,47,16,48,17,49,18,50,19,51,20,52,21,53,22,54,23,55,24,56,25,57,26,58,27,59,28,60,29,61,30,62,31,63
numa: 1
numa0: cpus=0-15,hostnodes=0,memory=24576,policy=bind
numa1: cpus=16-31,hostnodes=1,memory=24576,policy=bind
numa2: cpus=32-47,hostnodes=2,memory=24576,policy=bind
numa3: cpus=48-63,hostnodes=3,memory=24576,policy=bind

Since this VM has all four NUMA nodes available, you can bind memory from all 4 NUMA domains. Make sure to leave some for the host as ballooning should be disabled.