xfcp + 10G on VCU118

[np84]

Hi Alex,

I have organized some 10G hardware (QSFP28->SFP+ converters, 10G SFP+ modules, fibre cables). Your verilog-ethernet VCU118 10G loopback runs like charm. However, there is no 10G version of xfcp. Is there any technical reason for that or should it work to simply combine the VCU118 10G loopback example with the VCU118 1G xfcp example?
Best,
Nico

[alexforencich]

So, the XFCP codebase is really designed around a fixed 8 bit datapath, as the design goals are more about flexibility than high performance. Therefore, running the whole thing at 10G would require a pretty significant overhaul. However, that's probably not necessary if you don't actually need all that bandwidth, what you can do instead is simply connect width converters between the 10G MAC and the XFCP UDP interface module and it should work just fine. And incidentally, I may do just that on a board that I need to do some transceiver characterization on that only has QSFP and PCIe.

[alexforencich]

Actually, the 10G MAC should already contain width converters as it should use the axis_fifo_adapter modules. So, you should be able to simply set the interface width on the 10G MAC to 8 bits. (This feature is intended for when you want to run with a wider interface and slower clock as is done in Corundum when operating at 25 Gbps (MAC is 64 at 390 MHz, but core logic runs at 250 MHz, so it uses a 128 bit internal interface), but you can certainly go narrower too if you want).

[np84]

I think that is exactly what I need. I will try this out soon. You are right that I do not need 10 Gbps but having it slightly faster is not bad at all. Am I correct that using the 8 bit 10G MAC interface and connecting the MAC clock (390 MHz) to xfcp should result in ~3.1 Gbps then? Is that possible or should I use another clock for xfcp?

[alexforencich]

At 10G, the clock is 156.25 MHz. It's 390 MHz at 25 Gbps. I like to run the XFCP logic at 125 MHz so I know all of the cycle counters and such increment at a consistent rate, but TBH it doesn't really make that much difference and running everything at 156 should be fine. However, running at 390 MHz could cause issues with timing closure.

[np84]

xfcp over QSFP/SFP+ actually works perfectly fine with 156.25 MHz. I'm not sure if this is exactly what you suggested, but I am using the "axis_adapter" module to convert between 8 bit and 64 bit. If you do not have any further suggestions, we can close this issue. Thanks!

[alexforencich]

Nice. I just added a stratix 10 example that does this. Since the MAC uses fifo adapter modules, all I had to do was set the data width to 8 bits. Take a look at: https://github.com/alexforencich/xfcp/blob/master/example/S10MX_DK/fpga/rtl/fpga_core.v#L151 . No additional modules required. That one is set up to use a core clock of 125 MHz to be consistent with the other designs, but if you want to run at 156.25 that works too.

[np84]

I did that and also clocked it to 125 MHz---everything works as expected. However, it turns out that the bandwidth is very low. I do not understand what is going on, but I cannot achieve more than 700 Kbps (yes, Kbps) write speed. I measured this by writing a 32 bit int (x) via interface.write(0,x.to_bytes(4,'little')) with 10^6 repetitions to your "xfcp_mod_wb" MemoryNode. It is clear that I will not achieve the full 1Gbps (8bit at 125MHz), but ~500Mbps would be nice. Do you have any idea? Can you verify this on your side?

[mpkopec]

Hi,
Since this discussion is interesting also for the XFCP implementation in one of the projects I am working on, I have made some calculations and would appreciate if Alex confirmed, denied or corrected them.

First of all, I assumed you used write() from MemoryNode. If so, note that saving one, 32-bit value takes a request and a response from the XFCP node, thus you send a packet with a value, wait for the response and based on that you know if the value is written. Consider this (see my notes for calculations), you have an Ethernet frame, then IPv4 frame in it, then UDP and lastly XFCP. From my rough calculations (I don't know the structure of the memory packet, since I don't use it, but I assumed best case - short packet - with only XFCP routing and 32 bits of your data) all of that takes up 73 bytes plus your data, which already yields 50 Mbps, because you use only 4 bytes out of total 77.

If you add the delay between the request and response equal to 10 us and the response packet with the same length, you efectively use only 4 bytes out of two packets and 10 us of time. Since 10 us is 10000 bits with 1 Gbps link, this adds up to the bandwidth being only 3.15 Mbps, which is close to your value, but not yet there. We are still off by a factor of around 5. The calculations do not however account for Python code execution, etc.

To inspect the delay between the request and response you can use Wireshark, depending on the connection (point-to-point, through a single switch, through more networking equipment) you can get the time delta as low as <1us as far as I remember.

The conclusion is - it all depends on how you design your node and how much data at once you send. For raw Ethernet frame payloads you can achieve 960 Mbps real data bandwidth if you fill up the whole frame and you keep up with keeping the link busy all the time. Also, for measuring bandwidth you could use a simple node that would send some number of full UDP over Ethernet packets using XFCP (the length considering the protocol stack structure should be calculated). After reception of the first packet you would save the timestamp, and after last packet save another. The number of packets should be, imho, large enough for at least a few seconds of constant traffic and the connection should be point-to-point to eliminate the network equipment from the equation.

[alexforencich]

Sounds about right. The memory packet header carries the address and transfer length, so it will be a handful of bytes. But, that overhead will be dwarfed by the other headers. It's going to be a lot more efficient if you transfer a block of data in one shot, writing to adjacent addresses.

However,the biggest issue is how the python code always waits for the response packet, at least for memory operations. It may make sense to rework the python code to mitigate this issue by supporting multiple in-flight operations at the same time, though I'm not sure what the best approach to that would be.

[mpkopec]

Thanks for the confirmation.

Just to clarify, I made a mistake in my last sum, I added bytes to bits, but this has little impact on the end number, which already is an estimate.

[np84]

Ok, combining multiple blocks into one bigger transfer raised the throughput to ~37Mbps for my own xfcp_mod (xfcp_mod_wb should be even faster). This is "ok" for now, but might be a problem for me in the future... :) maybe some day we can test how this can be improved by optimizing the python code.

[alexforencich]

Another thing that you can do is open multiple connections. The setup is smart enough to handle that correctly so you can have multiple scripts or even multiple hosts talking to the same design at the same time. Obviously you need to make sure that the different programs do not interfere with each other, but it definitely works and I have done this before several times, for instance with one control script and one passive monitoring script.

I will also say that one of the design goals of XFCP was to keep things as simple and as easy to use as possible. So, if it's possible to make changes to improve throughput without vastly increasing the code size and interface complexity, then it's worth looking in to.

[np84]

I appreciate the simplicity of XFCP and I'm very happy with it. My downstream task is to "offload" computation to the fpga (actual computation can take minutes). So it does not really matter if offloading takes a couple of seconds (which is what I have now with the ~40 Mbps). There is a chance that the offloaded tasks will become 4-16 times larger in the future but the time horizon for this is >1 year. So I am good for now. Thus, the following question is just because of my curiosity: is there a conceptual issue with 64-bit-xfcp? I.e., is there more to do beyond expanding all relevant ports from 8 to 64 bits?

[mpkopec]

Actually, I am currently working on a data upstream node for XFCP and plan to do some throughput testing, I can report on my findings if you like.

The simplicity of XFCP is really great and I think you can achieve a lot with proper design of the nodes and thinking ahead about the Python interface.

[alexforencich]

You know, now that I think about it, doing 64-bit ports is probably less of an issue than I originally thought. The reason for this is that the switches always have to add or remove a single byte. Yes, this is slightly more annoying with a 64-bit interface than with an 8-bit interface, but since it's always 1 byte then you don't need a full barrel shifter or something. I will put parametrizing the interface width on my to-do list. The harder module to parametrize is the COBS encoder/decoder, but that's only used for the serial port, so I don't think that will be a problem.