Skip to content

Add minikube performance tests - #179

Draft
nirs wants to merge 2 commits into
mainfrom
minikube
Draft

Add minikube performance tests#179
nirs wants to merge 2 commits into
mainfrom
minikube

Conversation

@nirs

@nirs nirs commented Feb 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add test/minikube/test.py to compare network performance of minikube drivers
    (krunkit, vfkit) using iperf3 benchmarks
  • Test all combinations of driver, network (host/pod), iperf3 location (host/vm),
    and direction (tx/rx/bidir) — 24 tests total
  • Generate line plots and markdown report from results
  • Include initial results from M2 Max with krunkit offloading enabled

Usage

% cd test/minikube
% ./test.py run --minikube ~/src/minikube/out/minikube
% ./test.py analyze

Initial findings (M2 Max, krunkit with offloading)

  • Host network: krunkit 3-5x faster than vfkit (~30 Gbits/s tx,
    ~47 Gbits/s rx vs ~7-10 Gbits/s)
  • Pod network from host: krunkit still faster (~13 Gbits/s tx,
    ~50 Gbits/s rx)
  • Pod network from VM: krunkit broken (~0 Gbits/s) — offloading
    and pod network don't work together from inside the VM
  • Bidir fairness: krunkit shows periodic TX/RX oscillation not
    seen in vfkit

Issues

  • Add krunkit results without offloading
    • compare vfkit, krunkit, krunkit offloading in every test?
  • Merge rx and tx plots (rx positive, tx negative)?
  • Add mixed clusters
    • vfkit client, krunkit server
    • krunkit client, vfkit server
  • Add podman/gvproxy server?
    • rootless
    • rootfull
    • provider applehv
    • provider libkrun
  • Test on newer machine
    • M3
    • M4
    • M5

nirs added a commit that referenced this pull request Apr 1, 2026
Add --enable-tso and --enable-checksum-offload options to vmnet-run,
passed through to vmnet-helper. These options were previously only
available when running vmnet-helper directly.

Since macOS 26.2 offloading works well and can significantly improve
network performance. For example, with krunkit on M2 Max, host network
throughput is 3-5x faster with offloading (~30 Gbits/s tx, ~47 Gbits/s
rx vs ~7-10 Gbits/s without offloading).

Note that offloading does not work for all use cases. For example,
accessing the VM from pod network does not work with offloading
enabled (see #179).

Example run:

    % ./example server --driver krunkit --cpus 4 --connection=runner --enable-offloading
    [   0.052] INFO Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
    [   0.059] INFO Starting 'krunkit' virtual machine 'server' with mac address '76:d4:d3:83:fc:14'
    [   6.079] INFO Virtual machine IP address: 192.168.105.2

    % iperf3 -c server-vmnet-helper.local
    ...
    [ ID] Interval           Transfer     Bitrate
    [  7]   0.00-10.01  sec  43.2 GBytes  37.1 Gbits/sec                  sender
    [  7]   0.00-10.01  sec  43.2 GBytes  37.1 Gbits/sec                  receiver

    % iperf3 -c server-vmnet-helper.local -R
    ...
    [ ID] Interval           Transfer     Bitrate         Retr
    [  7]   0.00-10.00  sec  52.7 GBytes  45.3 Gbits/sec    0            sender
    [  7]   0.00-10.00  sec  52.7 GBytes  45.3 Gbits/sec                  receiver

Fixes #177
nirs added a commit that referenced this pull request Apr 1, 2026
Add --enable-tso and --enable-checksum-offload options to vmnet-run,
passed through to vmnet-helper. These options were previously only
available when running vmnet-helper directly.

Since macOS 26.2 offloading works well and can significantly improve
network performance. For example, with krunkit on M2 Max, host network
throughput is 3-5x faster with offloading (~30 Gbits/s tx, ~47 Gbits/s
rx vs ~7-10 Gbits/s without offloading).

Note that offloading does not work for all use cases. For example,
accessing the VM from pod network does not work with offloading
enabled (see #179).

Example run:

    % ./example server --driver krunkit --cpus 4 --connection=runner --enable-offloading
    [   0.052] INFO Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
    [   0.059] INFO Starting 'krunkit' virtual machine 'server' with mac address '76:d4:d3:83:fc:14'
    [   6.079] INFO Virtual machine IP address: 192.168.105.2

    % iperf3 -c server-vmnet-helper.local
    ...
    [ ID] Interval           Transfer     Bitrate
    [  7]   0.00-10.01  sec  43.2 GBytes  37.1 Gbits/sec                  sender
    [  7]   0.00-10.01  sec  43.2 GBytes  37.1 Gbits/sec                  receiver

    % iperf3 -c server-vmnet-helper.local -R
    ...
    [ ID] Interval           Transfer     Bitrate         Retr
    [  7]   0.00-10.00  sec  52.7 GBytes  45.3 Gbits/sec    0            sender
    [  7]   0.00-10.00  sec  52.7 GBytes  45.3 Gbits/sec                  receiver

Fixes #177
nirs added a commit that referenced this pull request Apr 1, 2026
Add --enable-tso and --enable-checksum-offload options to vmnet-run,
passed through to vmnet-helper. These options were previously only
available when running vmnet-helper directly.

Since macOS 26.2 offloading works well and can significantly improve
network performance. For example, with krunkit on M2 Max, host network
throughput is 3-5x faster with offloading (~30 Gbits/s tx, ~47 Gbits/s
rx vs ~7-10 Gbits/s without offloading).

Note that offloading does not work for all use cases. For example,
accessing the VM from pod network does not work with offloading
enabled (see #179).

Example run:

    % ./example server --driver krunkit --cpus 4 --connection=runner --enable-offloading
    [   0.052] INFO Creating cloud-init iso '/Users/nir/.vmnet-helper/vms/server/cidata.iso'
    [   0.059] INFO Starting 'krunkit' virtual machine 'server' with mac address '76:d4:d3:83:fc:14'
    [   6.079] INFO Virtual machine IP address: 192.168.105.2

    % iperf3 -c server-vmnet-helper.local
    ...
    [ ID] Interval           Transfer     Bitrate
    [  7]   0.00-10.01  sec  43.2 GBytes  37.1 Gbits/sec                  sender
    [  7]   0.00-10.01  sec  43.2 GBytes  37.1 Gbits/sec                  receiver

    % iperf3 -c server-vmnet-helper.local -R
    ...
    [ ID] Interval           Transfer     Bitrate         Retr
    [  7]   0.00-10.00  sec  52.7 GBytes  45.3 Gbits/sec    0            sender
    [  7]   0.00-10.00  sec  52.7 GBytes  45.3 Gbits/sec                  receiver

Fixes #177
nirs and others added 2 commits May 9, 2026 20:38
Add a test script to compare network performance of minikube drivers
(krunkit, vfkit) using iperf3 benchmarks.

The test creates two minikube clusters (server and client) for each
driver, deploys iperf3 using kustomize overlays, and runs benchmarks
across all combinations of:

- Drivers: krunkit, vfkit
- Networks: host (direct VM IP), pod (Kubernetes NodePort)
- Where: host (iperf3 on macOS), vm (iperf3 inside VM)
- Tests: tx, rx, bidir

Command line arguments allow testing specific combinations for
debugging:

    % ./test.py run -d krunkit -n host --tests tx rx

The analyze command generates line plots and a markdown report from
the results:

    % ./test.py analyze

Example:

    % cd test/minikube
    % ./test.py run --minikube ~/src/minikube/out/minikube -t 60
    Setup krunkit server
    Setup krunkit client
    Deploying host
    Testing krunkit-host-host-tx...
    ...
    Cleanup vfkit
    % ./test.py analyze
    Created out/plots/host-host.png
    Created out/plots/host-vm.png
    Created out/plots/pod-host.png
    Created out/plots/pod-vm.png
    Created out/report.md

Co-authored-by: Cursor <cursoragent@cursor.com>
Results from minikube iperf3 benchmarks comparing krunkit (with
offloading) and vfkit drivers on M2 Max.

Key findings:
- Host network: krunkit 3-5x faster than vfkit (~30 Gbits/s tx,
  ~47 Gbits/s rx vs ~7-10 Gbits/s) thanks to TSO offloading.
- Pod network from host: krunkit still faster (~13 Gbits/s tx,
  ~50 Gbits/s rx).
- Pod network from VM: krunkit broken (~0 Gbits/s) - offloading
  and pod network don't work together from inside the VM.
- Krunkit bidir shows periodic TX/RX oscillation not seen in vfkit,
  likely a fairness issue in libkrun's virtio-net.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant