XDMA poor performance and high latency spikes explained and fixed

[dmitrym1]

Fixes #332

[ynt1988]

@dmitrym1
Thank you for your work on the XDMA optimization.
On a PCIe 3.0 x16 platform in AXI-ST mode, after applying your patch and following your guidance to set poll_mode=15, binding the cmpl thread to CPU1, and also binding the userspace thread to CPU1, I achieved a C2H speed of 10320 MB/s, but H2C only reached 8500 MB/s.

I compared the CPU utilization during H2C and C2H operations. During C2H, the kernel cmpl thread and userspace thread utilization were 70% and 30%, respectively. During H2C, they were 80% and 20%, respectively. Compared to C2H, the userspace thread utilization during H2C is 10% lower. I wonder if this is causing the performance difference. How can I improve the H2C speed?

[dmitrym1]

@ynt1988 Thank you for using my work. I agree with your conclusion about CPU usage. Since in poll mode the driver actively polls the Card for completion, having higher CPU utilization by kernel thread (i.e. more time in busy wait) and lower CPU utilization by userspace thread (i.e. less time in copying data) clearly leads to lower H2C speed. Which could mean that either the kernel driver does something extra in H2C transfer, or the transfer itself is slower. You could try to fix the first part using XDMA by @Prandr https://github.com/Prandr/dma_ip_drivers . He has significantly reworked the driver and could have fixed some potential issues. About the second part, make sure your Card completes the transfer as fast as possible. Other possible reason is hardware limitation.

[ynt1988]

@dmitrym1 Thank you for your reply. I have tried downloading and using @Prandr https://github.com/Prandr/dma_ip_drivers, but I don't see the "/dev/xdma_c2h_" and "/dev/xdma_h2c_" device files. Do I need to regenerate a new Block Design and bitstream for the FPGA?

[MauSilvestrini]

Hi @dmitrym1, many thanks for your work on XDMA driver. I replied to you the last month in this conversation :

#110 (comment)

At the moment I still have the same issue (high latency spike up to 500 us btw two consecutive read calls), and I ve already tried the @Prandr reworked driver version. However, I encountered the same issue as @ynt1988.

Would it be possible to stay in touch with you to further discuss this issue?

[Prandr]

@ynt1988 @MauSilvestrini
If you had a problem you could open an issue on my repo. I suspect it is the same as Prandr#11
I will add clear mentioning of the default IDs to the docs

You really try to make a dead horse to win a race with the driver here.

[ynt1988]

@Prandr You are right, that's exactly the issue I encountered. Thank you for the correction. My apologies, I should have carefully studied the make help output.

[MauSilvestrini]

@Prandr many thanks for your clarification ! I ll update the default DID in my FW design. I ll keep you updated.

[MauSilvestrini]

Hi @Prandr, I ve chosen to modify the xdma_mod.c structure (named as pci_ids) by adding my default DID that is 0x8018.
At this point I finally see both the /dev/xdma_c2h and /dev/xdma_h2c devices. However, I m getting an error while trying to open both the devices with the following:

fpga_fd = open(devname, O_RDWR);

Do I need to perform any additional configuration ?

[Prandr]

@MauSilvestrini
First of all you need to read documentation that I linked. Then you would have known that you don't need to modify the source code of the driver nor your design. And you would also learn the answer to your problem.