Context
- Version of iperf3:
NA
- Hardware:
NA
- Operating system (and distribution, if any):
NA
Suggested documentation or FAQ Enhancement
Based on recommendations given for solving issue #1021 about throughput drop to almost zero, following is a suggested addition to the iperf3 documentation or FAQ for handling such issues. Although this is not iperf3 SW issue, the recommendation can help iperf3 users to solve similar issues. For example, it seems that issue #703 and maybe #453, #303 happened because of similar problems.
Note that there is a related pull request #921 for iperf3 to support UDP Generic Segmentation Offload (GSO) on Linux (see also https://man7.org/linux/man-pages/man3/cmsg.3.html).
The suggested addition to the documentation:
TCP Offload issues or what to do when throughput drops to almost zero
Throughput drop to almost zero, except maybe for the first reported interval(s), may be related to problems in NIC TCP Off load, which is used to offload TCP functionality to the NIC (see https://en.wikipedia.org/wiki/TCP_offload_engine). TCP Offload target is to save main CPU performance, mainly of un/segmentation of large packets and checksum computation.
When TCP packets are sent with "Don't Fragment" flag set, which is the recommended settings, segmentation is done by the TCP stack based on the reported next hope MSS in the ICMP Fragmentation Needed message. With TCP Offload active segmentation is done by the NIC of the sending side which is known as TCP Segmentation offload (TSO) or in Windows as Large Send Offload (LSO). It seems that there are TSO/LSO implementations which for some reason ignore the reported MSS and therefore don’t perform segmentation. In these cases, when large packets are sent, e.g. the default iperf3 128KB (131,072 bytes), iperf3 will show that data was sent in the first interval, but since the packets don’t get to the server, no ack is received and therefore no data is sent in the following intervals. It may happen that after certain timeout the main CPU will re-send the packet by re-segmenting it, and in these cases data will get to the server after a while. However, it seems that segmentation is not automatically continued with the next packet, so data transfer rate be very low.
The recommended solution in such case is to disable TSO/LSO, at least on the relevant port. See for example: https://atomicit.ca/kb/articles/slow-network-speed-windows-10/. If that doesn’t help then "Don't Fragment" TCP flag may be disabled. See for example: https://support.microsoft.com/en-us/help/900926/recommended-tcp-ip-settings-for-wan-links-with-a-mtu-size-of-less-than. However, note that disabling the “Don’t Fragment” flag may cause other issue.
To test whether TSO/LSO may be the problem, do the following:
• If different machine configurations are used for the client and server, try the iperf3 reverse mode (-R). With TSO/LSO issue only on the client machine this test should succeed.
• Reduce buffer length to a small value that should not require segmentation, using the iperf3 -l option, e.g. "-l 512". It may also help to reduce the MTU by using the iperf3 -M option, e.g. "-M 1460". With TSO/LSO issue this test should succeed.
• Using tools like Wireshark identify the required MSS in the ICMP Fragmentation Needed messages (if reported). Run tests with the “-l” value set to 2 times the MSS and the 4 times, 6 times, etc. With TSO/LSO issue in each test the throughput should be reduced more. It may help to increase these tests time beyond the default 10 seconds to better see the behavior (iperf3 -t option).
Context
NA
NA
NA
Suggested documentation or FAQ Enhancement
Based on recommendations given for solving issue #1021 about throughput drop to almost zero, following is a suggested addition to the iperf3 documentation or FAQ for handling such issues. Although this is not iperf3 SW issue, the recommendation can help iperf3 users to solve similar issues. For example, it seems that issue #703 and maybe #453, #303 happened because of similar problems.
Note that there is a related pull request #921 for iperf3 to support UDP Generic Segmentation Offload (GSO) on Linux (see also https://man7.org/linux/man-pages/man3/cmsg.3.html).
The suggested addition to the documentation:
TCP Offload issues or what to do when throughput drops to almost zero
Throughput drop to almost zero, except maybe for the first reported interval(s), may be related to problems in NIC TCP Off load, which is used to offload TCP functionality to the NIC (see https://en.wikipedia.org/wiki/TCP_offload_engine). TCP Offload target is to save main CPU performance, mainly of un/segmentation of large packets and checksum computation.
When TCP packets are sent with "Don't Fragment" flag set, which is the recommended settings, segmentation is done by the TCP stack based on the reported next hope MSS in the ICMP Fragmentation Needed message. With TCP Offload active segmentation is done by the NIC of the sending side which is known as TCP Segmentation offload (TSO) or in Windows as Large Send Offload (LSO). It seems that there are TSO/LSO implementations which for some reason ignore the reported MSS and therefore don’t perform segmentation. In these cases, when large packets are sent, e.g. the default iperf3 128KB (131,072 bytes), iperf3 will show that data was sent in the first interval, but since the packets don’t get to the server, no ack is received and therefore no data is sent in the following intervals. It may happen that after certain timeout the main CPU will re-send the packet by re-segmenting it, and in these cases data will get to the server after a while. However, it seems that segmentation is not automatically continued with the next packet, so data transfer rate be very low.
The recommended solution in such case is to disable TSO/LSO, at least on the relevant port. See for example: https://atomicit.ca/kb/articles/slow-network-speed-windows-10/. If that doesn’t help then "Don't Fragment" TCP flag may be disabled. See for example: https://support.microsoft.com/en-us/help/900926/recommended-tcp-ip-settings-for-wan-links-with-a-mtu-size-of-less-than. However, note that disabling the “Don’t Fragment” flag may cause other issue.
To test whether TSO/LSO may be the problem, do the following:
• If different machine configurations are used for the client and server, try the iperf3 reverse mode (-R). With TSO/LSO issue only on the client machine this test should succeed.
• Reduce buffer length to a small value that should not require segmentation, using the iperf3 -l option, e.g. "-l 512". It may also help to reduce the MTU by using the iperf3 -M option, e.g. "-M 1460". With TSO/LSO issue this test should succeed.
• Using tools like Wireshark identify the required MSS in the ICMP Fragmentation Needed messages (if reported). Run tests with the “-l” value set to 2 times the MSS and the 4 times, 6 times, etc. With TSO/LSO issue in each test the throughput should be reduced more. It may help to increase these tests time beyond the default 10 seconds to better see the behavior (iperf3 -t option).