iperf3: snd-timeout not works as expected

[zokl]

affected version: iperf3.16, iperf3.17, iperf3.18 till last commit
unaffected version: iperf3.15

kernel: Linux FTW1 5.10.221 #0 SMP Mon Jul 22 22:56:41 2024 x86_64 GNU/Linux
system: OpenWRT 22.03

iperf3 build info

root@FTW1:~# iperf3 -v
iperf 3.18+ (cJSON 1.7.15)
Linux FTW1 5.10.221 #0 SMP Mon Jul 22 22:56:41 2024 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, bind to device, support IPv4 don't fragment, POSIX threads

If the snd-timeout parameter is set, I expect that when the interface is dropped after the timeout the iperf on the data sending side will be terminated. This does not happen.

If I restart the interface during the measurement, the server side is terminated, but the client side is not terminated.

Replication:

Server side:

root@FTServer-DK:~# iperf3 -s --rcv-timeout 3000 --snd-timeout 3000 -p 9999
-----------------------------------------------------------
Server listening on 9999 (test #1)
-----------------------------------------------------------

Accepted connection from 78.80.18.18, port 26762
[  5] local 147.32.211.31 port 9999 connected to 78.80.18.18 port 26852
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec
[  5]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec
[  5]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec
[  5]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                                                    <-----------------  mobile network goes down
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
iperf3: error - idle timeout for receiving data
-----------------------------------------------------------
Server listening on 9999 (test #2)
-----------------------------------------------------------

Client side:

root@FTW1:~# iperf3 -c <IP> -p 9999 --snd-timeout 3000 --rcv-timeout 3000 -b 1M -t 600
Connecting to host 147.32.211.31, port 9999
[  5] local 100.65.203.250 port 40958 connected to 147.32.211.31 port 9999
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    0   62.4 KBytes
[  5]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec    0   62.4 KBytes
[  5]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec    0   98.9 KBytes
[  5]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec    0    104 KBytes
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec    0    101 KBytes
[  5]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec    0    101 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes                           <-----------------  mobile network goes down
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  11.00-12.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  15.00-16.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  17.00-18.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  18.00-19.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
[  5]  19.00-20.00  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes
^C[  5]  20.00-20.12  sec  0.00 Bytes  0.00 bits/sec    0   1.30 KBytes

Right function in version 3.15

iperf3 3.15 build info

root@FTW1:~# iperf3 -v
iperf 3.15 (cJSON 1.7.15)
Linux FTW1 5.10.221 #0 SMP Mon Jul 22 22:56:41 2024 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, bind to device, support IPv4 don't fragment

root@FTW1:~# iperf3 -c 147.32.211.31 -p 9999 --snd-timeout 3000 --rcv-timeout 3000 -b 1M -t 600
Connecting to host 147.32.211.31, port 9999
[  5] local 100.115.99.196 port 47568 connected to 147.32.211.31 port 9999
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    0   74.2 KBytes
[  5]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec    0    107 KBytes
[  5]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec    0    129 KBytes
[  5]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec    0    176 KBytes
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec    0    176 KBytes
[  5]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec    0    176 KBytes
[  5]   6.00-7.00   sec   128 KBytes  1.05 Mbits/sec    0    176 KBytes
[  5]   7.00-8.00   sec   128 KBytes  1.05 Mbits/sec    0    176 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   4.50 KBytes                          <-----------------  mobile network goes down
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes
iperf3: error - unable to write to stream socket: Network unreachable

[davidBar-On]

It may be that this because iperf3 main thread is not tracking failures of the send/receive threads, the issue that PR #1654 tries to fix. It will help if you can check whether this is the problem. This can be done (on the Client side) by either:

1. Check whether the "sending" thread was terminated after more than the 3 seconds of not sending. E.g. in Linux, this may be done using top -H.
2. Build the client from the code of PR Stop a test if one of the threads terminated because of an error #1654 and see whether it solves the problem. (Note that this code is not up to date, but it is good enough for this test.)

[zokl]

Hello @davidBar-On

You describe the problem very well. Here is an output from your fixed version of iperf3. It works as expected.

root@FTWireless-VoNR:~# iperf3 -c 147.32.211.31 -p 8989 --rcv-timeout 3000 --snd-timeout 3000 -t 600
Connecting to host 147.32.211.31, port 8989
[  5] local 10.66.91.211 port 32864 connected to 147.32.211.31 port 8989
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.50 MBytes  12.6 Mbits/sec    0    165 KBytes
[  5]   1.00-2.00   sec  1.62 MBytes  13.6 Mbits/sec    0    182 KBytes
[  5]   2.00-3.00   sec  1.50 MBytes  12.6 Mbits/sec    0    113 KBytes
[  5]   3.00-4.00   sec  1.88 MBytes  15.7 Mbits/sec    0    104 KBytes
[  5]   4.00-5.00   sec  1.25 MBytes  10.5 Mbits/sec    0    154 KBytes
[  5]   5.00-6.00   sec  1.62 MBytes  13.6 Mbits/sec    0    110 KBytes
[  5]   6.00-7.00   sec  1.62 MBytes  13.6 Mbits/sec    0    113 KBytes
[  5]   7.00-8.00   sec  1.00 MBytes  8.39 Mbits/sec    1   1.38 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
iperf3: Number of running threads is 0 but expected 1
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
iperf3: error - a thread stopped running unexpectedly:

[davidBar-On]

Thanks for testing. I have now re-based PR #1654 to catch up with the latest iperf3 mainline (master) branch.

Since in the above output I see that there is no error message from a thread about its termination, I added such error message. I also submitted PR #1861 which adds similar error messages, since I am not sure if or when PR #1654 will be merged to the mainline.

[zokl]

Thanks for testing. I have now re-based PR #1654 to catch up with the latest iperf3 mainline (master) branch.

Since in the above output I see that there is no error message from a thread about its termination, I added such error message. I also submitted PR #1861 which adds similar error messages, since I am not sure if or when PR #1654 will be merged to the mainline.

Here is an output from your modification:

root@FTWireless-VoNR:~# iperf3 -c 147.32.211.31 -p 8989 --snd-timeout 3000 --rcv-timeout 3000 -t 600
Connecting to host 147.32.211.31, port 8989
[  5] local 10.0.69.233 port 43154 connected to 147.32.211.31 port 8989
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.50 MBytes  12.6 Mbits/sec    0    124 KBytes
[  5]   1.00-2.00   sec  1.50 MBytes  12.6 Mbits/sec    0    214 KBytes
[  5]   2.00-3.00   sec  1.50 MBytes  12.6 Mbits/sec    0    126 KBytes
[  5]   3.00-4.00   sec  1.38 MBytes  11.5 Mbits/sec    0    182 KBytes
[  5]   4.00-5.00   sec  1.12 MBytes  9.44 Mbits/sec    0    157 KBytes
[  5]   5.00-6.00   sec  1.62 MBytes  13.6 Mbits/sec    0    168 KBytes
[  5]   6.00-7.00   sec  1.50 MBytes  12.6 Mbits/sec    0    116 KBytes
[  5]   7.00-8.00   sec   256 KBytes  2.10 Mbits/sec    1   1.38 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
iperf3: Client Worker Thread 1 FD 5 failed - unable to write to stream socket: Network unreachable, with errno Network unreachable (101)
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
iperf3: Number of running threads is 0 but expected 1
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec    0   1.38 KBytes
iperf3: error - a thread stopped running unexpectedly:

Here is an output from server:

root@FTServer-DK:~# iperf3 -s --snd-timeout 3000 --rcv-timeout 3000 -p 8989
-----------------------------------------------------------
Server listening on 8989 (test #1)
-----------------------------------------------------------
Accepted connection from 46.135.16.40, port 11757
[  5] local 147.32.211.31 port 8989 connected to 46.135.16.40 port 11758
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  31.0 MBytes   260 Mbits/sec    0   2.98 MBytes
[  5]   1.00-2.00   sec  35.0 MBytes   294 Mbits/sec    0   1.90 MBytes
[  5]   2.00-3.00   sec  29.4 MBytes   246 Mbits/sec    0   2.05 MBytes
[  5]   3.00-4.00   sec  24.5 MBytes   206 Mbits/sec    0   2.31 MBytes
[  5]   4.00-5.00   sec  31.6 MBytes   265 Mbits/sec    0   3.38 MBytes
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    1   1.38 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.38 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    1   1.38 KBytes
iperf3: Server Worker Thread 1 FD 5 failed - unable to write to stream socket: Operation timed out, with errno Operation timed out (110)
iperf3: Number of running threads is 0 but expected 1
iperf3: error - select failed: Bad file descriptor
-----------------------------------------------------------
Server listening on 8989 (test #2)
-----------------------------------------------------------

[zokl]

@davidBar-On , thank you very much for fixing this issue.

[davidBar-On]

@zokl, thank you for testing! I see that by mistake errno value text is printed twice, so I now removed the second.