Cannot utilize 100gbps on Mellanox ConnectX-5

[antiuser]

Hi there! Who can explain me how to utilize 100gbps on Mellanox Technologies MT27800 Family [ConnectX-5]
I can utilize around ~70gbps /usr/src/iperf-3.21/src# ./iperf3 -c 10.192.46.24 -P20 -t360 on E5-2699 v4 @ 2.20GHz (same server/client).

[SUM]   0.00-360.00 sec  2.72 TBytes  66.4 Gbits/sec  19545340             sender
[SUM]   0.00-360.00 sec  2.72 TBytes  66.4 Gbits/sec                  receiver

Tuning network stack as

/sbin/ethtool -G enp4s0np0 tx 8192 rx 8192
/sbin/ethtool -A enp4s0np0 autoneg off rx off tx off
/sbin/ethtool -K enp4s0np0 tx on sg on tso on gro on gso on
/sbin/ethtool -C enp4s0np0 adaptive-rx off adaptive-tx off
/usr/bin/ip link set dev enp4s0np0 txqueuelen 10000
/usr/sbin/set_irq_affinity.sh enp4s0np0

Also, when Flow Control is ON - i have zero retries, but when Flow Control os OFF - no retries at all. But this server should run BGP (thats why i dont need Flow Control turned on).

# uname  -r
5.10.0-34-amd64

# modinfo mlx5_core
filename:       /lib/modules/5.10.0-34-amd64/updates/dkms/mlx5_core.ko
alias:          auxiliary:mlx5_core.eth-rep
alias:          auxiliary:mlx5_core.eth
basedon:        Korg 6.8-rc4
version:        24.10-4.1.4
license:        Dual BSD/GPL
description:    Mellanox 5th generation network adapters (ConnectX series) core driver
author:         Eli Cohen <eli@mellanox.com>
srcversion:     F3AE215F1A13333F82677B5
alias:          pci:v000015B3d0000A2DFsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2DCsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2D6sv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2D3sv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2D2sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001025sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001023sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001021sv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Fsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Esv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Dsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Csv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Bsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Asv*sd*bc*sc*i*
alias:          pci:v000015B3d00001019sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001018sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001017sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001016sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001015sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001014sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001013sv*sd*bc*sc*i*
depends:        mlxdevm,auxiliary,mlx_compat,mlxfw,pci-hyperv-intf,ptp,psample
retpoline:      Y
name:           mlx5_core
vermagic:       5.10.0-34-amd64 SMP mod_unload modversions 
parm:           num_of_groups:Eswitch offloads number of big groups in FDB table. Valid range 1 - 1024. Default 15 (uint)
parm:           debug_mask:debug mask: 1 = dump cmd data, 2 = dump cmd exec time, 3 = both. Default=0 (uint)
parm:           prof_sel:profile selector. Valid range 0 - 4 (uint)

Thank you for advices and sorry for poor English.
Have a nice day :)

[davidBar-On]

Hi, I am not familiar with this machine, but I see that you turn on gso/gro. To use gso/gro with iperf3 you should set the (new) --gsro option.

[antiuser]

Hi, I am not familiar with this machine, but I see that you turn on gso/gro. To use gso/gro with iperf3 you should set the (new) --gsro option.

Thank you for your response, but this not changed...
./iperf3 -c 10.192.46.24 -P22 -t 360 --gsro
This is 22 Core processor working on 2.8GHz, may be we should use more powerfull processor to archive 100gpbs?
Also about memory

Handle 0x1100, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x1000
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: 1
	Locator: A1
	Bank Locator: Not Specified
	Type: DDR4
	Type Detail: Synchronous Registered (Buffered)
	Speed: 2400 MT/s
	Manufacturer: 019800B300AD
	Serial Number: B00A0289
	Asset Tag: 00182730
	Part Number: HP24D4R7D4HAI-32  
	Rank: 2
	Configured Memory Speed: 2400 MT/s
	Minimum Voltage: 1.2 V
	Maximum Voltage: 1.2 V
	Configured Voltage: 1.2 V

Handle 0x1101, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x1000
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 32 GB
	Form Factor: DIMM
	Set: 1
	Locator: A2
	Bank Locator: Not Specified
	Type: DDR4
	Type Detail: Synchronous Registered (Buffered)
	Speed: 2400 MT/s
	Manufacturer: 019800B300AD
	Serial Number: B20A0189
	Asset Tag: 00182730
	Part Number: HP24D4R7D4HAI-32  
	Rank: 2
	Configured Memory Speed: 2400 MT/s
	Minimum Voltage: 1.2 V
	Maximum Voltage: 1.2 V
	Configured Voltage: 1.2 V

On localhost tests (where client and server running on one machine) i can get about

[SUM]   0.00-12.21  sec   162 GBytes   114 Gbits/sec  110             sender
[SUM]   0.00-12.21  sec  0.00 Bytes  0.00 bits/sec                  receiver

[bmah888]

On iperf3, --gsro is only effective for UDP tests, not TCP, so this won't help you.

There are a lot of factors behind the performance of iperf3 (or any other network program).

In our environment, we sometimes need to make sure that network-bound processes run on CPU cores on sockets that are associated with the NICs, so something like numactl(8) can be helpful. I was not clear whether you have a single-socket or multi-socket computer or not.

If you have the client and server on the same host, and the client is connecting to the server on the network address of the local network interface, I don't even think the packets physically leave the host, they're basically treated as going over the loopback interface lo. The performance should be fairly similar if the client runs against localhost.

Have you tried varying the number of parallel connections? If you try larger and larger values for -P you'll reach a point where the threads are thrashing among the different CPU cores and the throughput will actually go down as you add more threads. According to Intel documentation at https://www.intel.com/content/www/us/en/products/sku/91317/intel-xeon-processor-e52699-v4-55m-cache-2-20-ghz/specifications.html your CPU does indeed have 22 cores and 44 threads, I am guessing this is why you did -P 22? Need to remember however that some of these are necessary for the OS, the main iperf3 thread that manages the tests, various other processes in the host, etc.