Stream throughput benchmark

[catkira]

@pcercuei
I compared stream throughput of libiio 0.x vs 1.x on the device, the host which is connected via 1G ethernet is running libiio 0.23. The benchmark was done with SoapySDRUtil using this command
SoapySDRUtil --rate="100000000" --direction=RX --args="driver=plutosdr,hostname=192.168.137.2"

with libiio 0.23 I get
11.1751 Msps 44.7002 MBps

with libiio 1.0 I get
15.863 Msps 63.4521 MBps

When I also install libiio 1.0 on the host and then perform a benchmark with this command:

Its interesting that there is already a significant speed improvement even when upgrading only on the device side
iio_rwdev -u ip:192.168.137.2 -B -b 65768 cf-ad9361-lpc

I get around Throughput: 52 MiB/s. Its interesting that this is slower than the 63 MB/s using libiio 0.23 on the host side.

[pcercuei]

Can you try with iio_readdev / iio_rwdev in every case, with the same parameters?

[catkira]

Edit: aaah shit, samplerate was set to low again (this really needs to be automated somehow!), now again...

device 0.x, host 0.x

benja@DESKTOP:/mnt/d/git/libiio/build$ iio_readdev -u ip:192.168.137.2 -B -b 65768 cf-ad9361-lpc
Throughput: 48 MiB/s

device 1.0, host 0.x

benja@DESKTOP:/mnt/d/git/libiio/build$ iio_readdev -u ip:192.168.137.2 -B -b 65768 cf-ad9361-lpc
Throughput: 52 MiB/s

device 1.0, host 1.0

benja@DESKTOP:/mnt/d/git/SoapyPlutoSDR/build$ iio_rwdev -u ip:192.168.137.2 -B -b 65768 cf-ad9361-lpc
Throughput: 48 MiB/s^

So here again, the combination 1.0 on device and 0.x on host is the fastest combination. There must be something to optimize in 1.0, but its quite close so it also might be nothing. But I am not seeing a speedup with libiio 1.0.

[pcercuei]

Well, 48 MiB/s is already about as fast as you can have on USB 2.0. You'd need a USB 3.0 Pluto to really benefit from the speedups.

[catkira]

I am using 1G ethernet on antsdr

[catkira]

I am adding iperf3 to the buildroot images of my pluto and antsdr so that I can run some benchmarks of the network performance

[pcercuei]

Better check CPU performance. Unless you are using beefier hardware the limiting factor will be the CPU. That's why I tested it on a ZCU702.

[catkira]

When I do top on the device during benchmark, CPU usage is at around 50%. But its only an average, there might be moments with higher CPU usage.
Are You using the DMABUF interface? That might also reduce CPU usage?
It would be nice to have an optional debug output in iiod that notifies about buffer overflows (network interface cannot take all data from AD9361). That way one could easily tell if its a network or CPU issue.

Eventually the user should get noticed if overflows happen in hdl (DMA interface too slow) or in the device CPU (network interface too slow).

@pcercuei Outputting if iiod is waiting for the network interface should be fairly easy, can You do that?

[pcercuei]

Does 'top' account for the time spent in the kernel? I know 'htop' does.

[catkira]

I dont know, have to check, but I will also add htop the my buildroot.

[catkira]

I managed to get htop running ony my ANTSDR. It was a little struggle, because I had to fix the ncurses package in buildroot first ^^ (https://github.com/catkira/buildroot/commit/66b2b6d6d34f918099860f8ac4a431d852a2e3bf)
The CPU usage shown by htop is around 2x 100%, while top shows only 1x 50%. Quite a big difference!
@pcercuei is it correct that there are 2 iiod processes?


(this is with 0.24 on device and host)

[pcercuei]

There should be only one iiod process - but there can be many threads. You can enable the tree view in htop with F5.

[catkira]

F5 does not change anything for me (other function keys are working).
This is how it looks with 1.0 on host and device:

With iio_rwdev I get 72 MiB/s most of the time, but there are periods with 51-55 MiB/s.
But I think its already clear that the CPU is the limitation. Do You think there is room for optimization?

[pcercuei]

There is not much to optimize - the CPU cycles are wasted reading and writing sockets.

There's a plan to introduce a zero-copy mechanism, but that's a very difficult task touching sensible parts of the Linux kernel, so it won't be here anytime soon.

[catkira]

I saw that the cross compilation is done without optimization (-Os) and only for armv7.
I am going to try with -march=armv8+simd and -O3.
The Xilinx cross compiler does not support armv9 yet.
Maybe copying the samples can be speeded up using simd instructions...
Maybe the memcpy can be optimized like this
https://stackoverflow.com/questions/30107385/replacing-memcpy-with-neon-intrinsics/30109183#30109183

Does the device cpu load go down a lot when You use the dmabuf interface? I guess thats what You mean by zero-copy right?

[catkira]

-march=armv8+simd and -O3 generated illegal instructions, but -march=armv8+simd and -O3 worked. However, there is no change in the speed of iio_rwdev benchmark. I guess its because the actual copy routine is not inside the iiolib code.
@pcercuei
The iiod code is a bit too complicated for me to quickly see where the copy operation happens or is called. Can You show me the code line (for the rx case as the iio_rwdev benchmark does)?
I want to try to look at the asm code and see if that can be optimized.

[pcercuei]

https://github.com/analogdevicesinc/libiio/blob/master/iiod/ops.c#L316

[catkira]

it seems like send is a system call with only a short wrapper in glibc. So its probably in the linux kernel. I am trying to compile it with optimized settings and see if it helps.

[catkira]

@pcercuei its a bit faster when compiling the kernel with optimization for speed. Between 58-68 MiB/s now, but I think thats abou the end of what is possible. I want to try the DMABUF interface. Do You think its hard to merge Your kernel patch into the adi linux?

[pcercuei]

The DMABUF interface is basically the same as the mmap interface, just a bit different, so it won't give you much.

The kernel patch would be quite difficult to merge into ADI's linux master branch, yes.

[catkira]

ah, I thought the DMABUF interface was the one used for zero-copy.
How do You test the zero-copy feature on hardware?

[pcercuei]

As I wrote above:

There's a plan to introduce a zero-copy mechanism, but that's a very difficult task touching sensible parts of the Linux kernel, so it won't be here anytime soon.

We don't test zero-copy on hardware because it's not implemented yet.

[catkira]

doesnt it make sense to have something running on hardware before You try to upstream it?

[pcercuei]

We're trying to upstream a kernel <-> userspace interface. It may support zero-copy later, but that's still TBD.

[catkira]

ok, but I mean even for the interface. Some requirements You only become aware of when You have a working example. But ok, I understand that this is nothing to wait for, because it probably takes >1year until zero-copy arrives. When it arrives it could eliminate the need to have PL usb3 or network interfaces for high bandwidth applications and enable new use cases for iio.

[catkira]

noooiiiiice, after optimizing the compilation parameters, increasing the blocksize gave another boost

Now I get stable 82-84 MiB/s (with libiio 1.0 on device and host via 1G ethernet). The CPUs are going to their limits

[pcercuei]

But then your latency is most likely off the charts...

[hz12opensource]

I hit the same performance limitation of libiio on LibreSDR (a PlutoSDR clone) with a Gigabit Ethernet port. The stock firmware can only run at about 47 MB/s and iiod CPU utilization was close to 100%. I tried to squeeze as much bandwidth as possible on this SDR:

• Kernel optimization is helpful, as @catkira pointed out. I compiled the kernel with -O3 flag. (@catkira what flags did you use to get best results?)
• CPU binding is important - on dual core SoC like Zynq 7020, pinning iiod process to core 1 helps a lot, because core 0 will be busy with handling interrupts.
• Overclocking the Zynq SoC and DDR also help a lot. I was able to overclock my SDR to 1100 MHz CPU and 750 MHz DDR and it runs quite stable. (See my repo for details on overclocking)

I achieved about 108 MB/s using the iio_readdev benchmark (libiio v0.25 on both client and server side). I monitored Ethernet interface and it consumes about 900Mbit/s bandwidth, close to the Gigabit Ethernet limit.

perf shows that a large portion of CPU cycles are spent on copying data from memory to TCP socket (arm_copy_from_user). If we can use zero copy on this, libiio performance can be improved significantly.

[catkira]

I also used some neon flags, but it depends on your vitis version which flags are available. However copying data into a socket is even less efficient than doing a memcpy, so I dont know if the compiler can infer SIMD instructions there.
@pcercuei is currently developing a zero-copy kernel interface for libiio, but I believe it will first work with the usb interface and eth maybe later.
I don't have a problem with the throughput at the moment, because I do some processing on the fpga and therefore my data rate is lower. I mostly hope that new features don't break old features now ;)

[hz12opensource]

@catkira Thank you! I also tried some neon flags like -march=armv7-a+simd -mfpu=neon but it seems it is not helpful (I am using arm-linux-gnueabihf-gcc 13.2.0, which seems to be the latest version to date). Also it seems tricky to use Neon code in kernel mode.

@pcercuei Linux actually supports zerocopied socket buffers with the MSG_ZEROCOPY flag, so technically using zero copy on socket buffers might be easier than on USB. I tried to modify libiio to use zero copy for TCP, but it is incompatible with existing buffers in libiio created using mmap. Some additional tricks like this are needed that are beyond my capability.

[pcercuei]

MSG_ZEROCOPY is probably possible now with the new dmabuf kernel interface, because the buffers are created in userspace there (they are memfds) and backed by pages.

[hz12opensource]

Thank you @pcercuei ! It will be fantastic if we have both dmabuf and MSG_ZEROCOPY in libiio eventually, so the library will be perfect for high speed data acquisition.

Do you have any instructions on how to try the new dmabuf kernel interface? I saw the dev-new-dmabuf-api branch and there are also some patchsets being submitted to the kernel, so it is no upstreamed yet?

[atmospher3]

Hi! I'm also experiencing similar performance limitations on a ZynqMP based AD9361 custom board. Getting about 55MB/s throughput over ethernet at close to 95% CPU usage.

I tried compiling the kernel with the -O3 flag, adjusting smp_affinity for gem interrupts, and setting CPU binding for the IIO daemon. Still no noticeable improvement in throughput.

Do you have any further suggestions? @hz12opensource

[mhennerich]

We're currently testing the cache coherency interconnect on MPSoC and it looks quite promising.

• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842098/Zynq+UltraScale+MPSoC+Cache+Coherency
• Ultrascale DMA via HPC hdl#925

Most of the CPU cycles are wasted for cache flush/invalidation.
Using the coherency interconnect doubles the throughput and reduces IIOD cpu usage significantly.

[atmospher3]

After I changed to connecting the DMA to the ps8 via the HPC port, the throughput improved significantly.

Now I can get 108MB/s over a Gigabit Ethernet.

Thank you @mhennerich !

[catkira]

DMA via HPC is a nice PR!
Unfortunately ZYNQ devices don't have HPC, but they have ACP. I wonder if using ACP could give a speedup there.

https://j-marjanovic.io/exploring-the-ps-pl-axi-interfaces-on-zynq-ultrascale-mpsoc.html
This analysis shows that ACP is faster than HCP for small transfer sizes.

Note: This improvement would not be useful for Pluto as USB 2.0 is the bottleneck already. But other devices with gigabit ethernet like AntSDR could benefit.

[pcercuei]

@catkira ideally we'd use the same port as the one the USB controller is connected to (and the Ethernet controller - I can only assume it's the same one) so that the respective DMAs are coherent between each other. That way, we can pass buffers around in a zero-copy fashion without the overhead of having to sync/flush the cpu caches.

[pbeltram]

I have pure embedded LX solution (LX kernel v5.15.36 and v6.6.40) with no overclocking, but it is bound to Zynq-7000 GEM.
No special FPGA IP is required, what is required is that source data in FPGA is streamed as an AXI stream via Xilinx DMA IP core.
I can continuously stream out data at 119Mbyte/sec and with additional Ethernet packets header/tail data the speed on wire is cca 970Mbits/sec on 1Gb Eth link..
During data transfer the Ethernet link is normally usable for other Ethernet protocols (ssh, http, ...), of course within remaining cca 30Mbits/sec bandwidth.
CPU load on Zynq-7000 during data transfer is cca 4% on CPU0 and cca 40% on CPU1.
It is not libiio bound, but at least it shows that solution to stream out data at wire speed on Zynq-7010 is possible, without overclocking and still have plenty of CPU power free.
Implementation is using UDP data transfer with addition that during data capturing it will request re-transmission of missing UDP packets.
Result is continuous data capturing without data lost.
The result of capturing depends a lot on PC and its NIC. Both must be powerful enough to handle this data transfer rate.
For PC NIC it is required that it supports jumbo frames.
I have demo for RedPiatya and ZyboZ7-20 (sampling 32 bit DIO) that can be download from here: https://github.com/pbeltram/npapi_test/tree/main
Pictures and additional descriptions are there.