Building DPDK 22.11
It seems the DPDK (22.11.1) documentation for building the mlx4 driver on Ubuntu 22.04 is not accurate or enough.
In the DPDK documentation, it says
libmlx4 is needed and they are provided by
rdma-core package. On Ubuntu 22.04, it does not seem to be the case. Instead,
libibverbs1 package contains the first (libibverbs), and
ibverbs-providers package contains the second (libmlx4). So instead of
rdma-core, I install these two packages.
After this, when you execute meson setup, it still complains that it cannot find dependencies for mlx4:
drivers: net/mlx4: missing dependency, "mlx4"
The reason seems to be the naming so I create two symbolic links to corresponding libraries:
sudo ln -s /usr/lib/x86_64-linux-gnu/libmlx4.so.1 /usr/lib/x86_64-linux-gnu/libmlx4.so sudo ln -s /usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so
Then meson setup does not complain and mlx4 is included in the drivers to be build. However, when ninja is run, it gives an error during build:
fatal error: infiniband/verbs.h: No such file or directory
This header file is provided by
libibverbs-dev package, so I install also that, and then the build finishes successfully.
This also enables mlx5 driver, so it is also built.
A bash script I use for this purpose can be found on github.
Note: If you add an option to mlx4_core module as described in the documentation, do not forget to run
update-initramfs -u, otherwise the option is not applied.
Using Cisco TRex Traffic Generator
ConnectX-3 Pro is shown as a single PCI interface even if it has dual ports. When I tried using ConnectX-3 Pro with the latest (3.0) TRex release by specifiying the subinterface with
["01:00.0/0", "01:00.0/0"], it did not work saying interface could not be found. I tried to trace this error in the source code and I realized actually trex mlx4 library (not sure what it is but there are for other NICs including mlx5) is not built anymore. I find it very strange that this is not mentioned in the release notes, so I checked every tag in the source code and I found v2.87 is the latest that includes mlx4 driver. So I downloaded this version, and it works without any error. Keep in mind Trex does not officially support ConnectX-3 series, hence it is not surprising mlx4 is removed in newer releases.
If not using the official OFED drivers, you can run trex with
--no-ofed-check argument. Official OFED driver is available for Ubuntu 20.04 but not for 22.04. Strangely, even on Ubuntu 20.04, OFED drivers do not install libmlx4.so which TRex requires. I did not manage to solve this and I am not using OFED drivers anyway.
After these, TRex was still giving error
cannot attach flow rules. I set the following driver option to fix this on
options mlx4_core log_num_mgm_entry_size=-1
You might need to run
update-initramfs -u aftrer this.
Finally, after configuring
/etc/trex_cfg.yml, a large packet imix test run with
sudo ./t-rex-64 -f cap2/imix_1518.yaml -m 400 --no-ofed-check --no-flow-control-change outputs the following on an 8 core/16GB virtual machine.
-Per port stats table ports | 0 | 1 ----------------------------------------------------------------------------------------- opackets | 276163904 | 0 obytes | 419216806272 | 0 ipackets | 0 | 276141434 ibytes | 0 | 419182696812 ierrors | 0 | 0 oerrors | 0 | 0 Tx Bw | 38.86 Gbps | 0.00 bps -Global stats enabled Cpu Utilization : 50.3 % 154.4 Gb/core Platform_factor : 1.0 Total-Tx : 38.86 Gbps Total-Rx : 38.85 Gbps Total-PPS : 3.20 Mpps Total-CPS : 0.00 cps Expected-PPS : 3.20 Mpps Expected-CPS : 3.20 Mcps Expected-BPS : 38.86 Gbps Active-flows : 1600 Clients : 255 Socket-util : 0.0100 % Open-flows : 1600 Servers : 65535 Socket : 1600 Socket/Clients : 6.3 Total_queue_full : 19264 drop-rate : 0.00 bps current time : 87.5 sec test duration : 12.5 sec
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.