Building DPDK 22.11 and Using TRex with ConnectX-3 Pro (mlx4) on Ubuntu 22.04

Building DPDK 22.11

It seems the DPDK (22.11.1) documentation for building the mlx4 driver on Ubuntu 22.04 is not accurate or enough.

In the DPDK documentation, it says libibverbs and libmlx4 is needed and they are provided by rdma-core package. On Ubuntu 22.04, it does not seem to be the case. Instead, libibverbs1 package contains the first (libibverbs), and ibverbs-providers package contains the second (libmlx4). So instead of rdma-core, I install these two packages.

After this, when you execute meson setup, it still complains that it cannot find dependencies for mlx4:

drivers:
    net/mlx4:	missing dependency, "mlx4"

The reason seems to be the naming so I create two symbolic links to corresponding libraries:

sudo ln -s /usr/lib/x86_64-linux-gnu/libmlx4.so.1 /usr/lib/x86_64-linux-gnu/libmlx4.so
sudo ln -s /usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so

Then meson setup does not complain and mlx4 is included in the drivers to be build. However, when ninja is run, it gives an error during build:

fatal error: infiniband/verbs.h: No such file or directory

This header file is provided by libibverbs-dev package, so I install also that, and then the build finishes successfully.

This also enables mlx5 driver, so it is also built.

A bash script I use for this purpose can be found on github.

Note: If you add an option to mlx4_core module as described in the documentation, do not forget to run update-initramfs -u, otherwise the option is not applied.

Using Cisco TRex Traffic Generator

ConnectX-3 Pro is shown as a single PCI interface even if it has dual ports. When I tried using ConnectX-3 Pro with the latest (3.0) TRex release by specifiying the subinterface with / e.g. ["01:00.0/0", "01:00.0/0"], it did not work saying interface could not be found. I tried to trace this error in the source code and I realized actually trex mlx4 library (not sure what it is but there are for other NICs including mlx5) is not built anymore. I find it very strange that this is not mentioned in the release notes, so I checked every tag in the source code and I found v2.87 is the latest that includes mlx4 driver. So I downloaded this version, and it works without any error. Keep in mind Trex does not officially support ConnectX-3 series, hence it is not surprising mlx4 is removed in newer releases.

If not using the official OFED drivers, you can run trex with --no-ofed-check argument. Official OFED driver is available for Ubuntu 20.04 but not for 22.04. Strangely, even on Ubuntu 20.04, OFED drivers do not install libmlx4.so which TRex requires. I did not manage to solve this and I am not using OFED drivers anyway.

After these, TRex was still giving error cannot attach flow rules. I set the following driver option to fix this on /etc/modprobe.d/mlx4.conf:

options mlx4_core log_num_mgm_entry_size=-1

You might need to run update-initramfs -u aftrer this.

Finally, after configuring /etc/trex_cfg.yml, a large packet imix test run with sudo ./t-rex-64 -f cap2/imix_1518.yaml -m 400 --no-ofed-check --no-flow-control-change outputs the following on an 8 core/16GB virtual machine.

-Per port stats table 
      ports |               0 |               1 
 -----------------------------------------------------------------------------------------
   opackets |       276163904 |               0 
     obytes |    419216806272 |               0 
   ipackets |               0 |       276141434 
     ibytes |               0 |    419182696812 
    ierrors |               0 |               0 
    oerrors |               0 |               0 
      Tx Bw |      38.86 Gbps |       0.00  bps 

-Global stats enabled 
 Cpu Utilization : 50.3  %  154.4 Gb/core 
 Platform_factor : 1.0  
 Total-Tx        :      38.86 Gbps  
 Total-Rx        :      38.85 Gbps  
 Total-PPS       :       3.20 Mpps  
 Total-CPS       :       0.00  cps  

 Expected-PPS    :       3.20 Mpps  
 Expected-CPS    :       3.20 Mcps  
 Expected-BPS    :      38.86 Gbps  

 Active-flows    :     1600  Clients :      255   Socket-util : 0.0100 %    
 Open-flows      :     1600  Servers :    65535   Socket :     1600 Socket/Clients :  6.3 
 Total_queue_full : 19264         
 drop-rate       :       0.00  bps   
 current time    : 87.5 sec  
 test duration   : 12.5 sec