DPDK 22.11 LTS and TRex on PC Engines apu4d4 running Ubuntu 22.04 LTS

This is an update to my previous post. The instructions are still mostly the same but there are a few changes. Rather than updating the old post, I decided to post a new one.

PC Engines apu4d4 has 4 Intel i211AT based LAN ports. DPDK supports Intel i211AT (with igb driver).

I assume Ubuntu 22.04 LTS is installed on apu4d4.

These are the instructions for building, configuring and testing the installation of DPDK 22.11.1 LTS on this hardware.

Building DPDK

First download the dpdk source archive and extract it:

$ wget http://fast.dpdk.org/rel/dpdk-22.11.1.tar.xz
$ tar xJf dpdk-22.11.1.tar.xz
$ cd dpdk-22.11.1

Then install the required packages for build.

$ sudo apt update
$ sudo apt install build-essential meson ninja-build pkg-config libpcap-dev libnuma-dev python3-pyelftools

Attention: Later you may configure CPU core isolation for DPDK, and when you do that, the isolated cores will not be used for build as well. So if you want to re-build again, pay attention that all cores are enabled for the OS. Otherwise, the build process will take even longer.

Then configure the build. This process will take a minute or so. This creates the build folder (you can delete it and re-run this for a new and clean start) and configures this folder accordingly. The options below builds only the igb driver (otherwise all drivers are built if dependencies are satisfied), builds all examples (if not interested in the examples, do not use this) and does not build tests. By default, the build type is release and platform is native. At the end of this configuration run, it shows what is going to be build.

In the previous post, I also added HPET support. I think it is not necessary, so it is omitted here.

meson -Denable_drivers=igb -Dexamples=all -Dtests=false build

Now change to build folder and start the build. The build process is going to take some time, approximately 80 minutes.

$ cd build
$ ninja

It is a good idea to install it system-wide. This completes in a few seconds.

$ sudo ninja install
$ sudo ldconfig

The installation is completed. DPDK utilities and libraries are installed system-wide, but the examples are still in build folder.

Before using DPDK, some system-wide configuration is needed to be done.

Configuring DPDK

The use of Intel i211 with DPDK requires the ethernet port to be unbound from its OS driver (also called igb) and bound to Virtual Function I/O (vfio) driver. In other and technical words, because OS igb driver is not a bifurcated driver, VFIO should be used. Then a DPDK application can access this ethernet port (and OS cannot, so it cannot be shared between OS and DPDK application).

In order to do this, normally IOMMU has to be enabled but the embedded processor on apu4d4 does not have IOMMU. So a special configuration vfio.enable_unsafe_noiommu_mode=1 has to be added to kernel cmdline in GRUB configuration (GRUB_CMDLINE_LINUX_DEFAULT key in /etc/default/grub file).

Because DPDK uses huge memory pages (hugepages), of 2MB or larger, you should also add something like hugepages=32 to cmdline as well. This reserves 32 2MB hugepages.

Not a must but for consistent and higher performance, the CPU cores that will be used with a DPDK application should also be isolated (not used by Linux) and their power management should be disabled, so they always stay at power state C1. For this purpose, following can be used again to be added to GRUB_CMDLINE_LINUX_DEFAULT:

• isolcpus=2,3: DPDK will use CPU cores 2 and 3.
• nohz_full=2,3: Omit scheduling-clock ticks from CPU cores 2 and 3, since there will be only 1 task on these cores. You may not want this if you plan to run multiple tasks.
• irqaffinity=0,1: IRQs will only go to cores 0 and 1.
• processor.max_cstate=1: Limit CPU c-state to max 1, meaning all CPU cores will stay at C1, effectively disabling idle power management. Be aware that this will increase the power consumption and heat dissipation. This will not disable the dynamic frequency scaling, CPU cores can still decrease or increase their frequency based on the load. If you keep

Run sudo update-grub after these changes.

To use VFIO, vfio-pci module must be loaded, and to load it at boot you can add vfio-pci to /etc/modules.

Reboot after making these changes.

Binding Ethernet Ports to VFIO

Now the ethernet ports can be bound to vfio-pci.

$ sudo dpdk-devbind.py -b vfio-pci 0000:02:00.0

Checking the status:

$ dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 'I211 Gigabit Network Connection 1539' drv=vfio-pci unused=igb

Network devices using kernel driver
===================================
0000:01:00.0 'I211 Gigabit Network Connection 1539' if=enp1s0 drv=igb unused=vfio-pci *Active*
0000:03:00.0 'I211 Gigabit Network Connection 1539' if=enp3s0 drv=igb unused=vfio-pci *Active*
0000:04:00.0 'I211 Gigabit Network Connection 1539' if=enp4s0 drv=igb unused=vfio-pci *Active*

...

Active means this port is being actively used by the OS, e.g. it has an IP address. You probably want to check this and unassign the IP so you are sure it is not used for something else.

Checking Hugepages and number of Memory Channels and Rank

It is good to know these before running a test program.

Run dpdk-hugepages.py -s to display the status of hugepages configuration, e.g. it outputs:

$ dpdk-hugepages.py -s
Node Pages Size Total
0    32    2Mb    64Mb 

Hugepages mounted on /dev/hugepages

This is as configured above in GRUB settings.

apu4d4 has one memory channel and this can be checked with dmidecode -t memory. So -n EAL option to DPDK applications can be set to 1 or omitted.

Test

If you have finished building and configuring DPDK, updated the GRUB settings and rebooted, and then binded ethernet ports to DPDK, now a test program can be run.

$ sudo dpdk-test -l 2-3

EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_e1000_igb (8086:1539) device: 0000:02:00.0 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
APP: HPET is not enabled, using TSC as default timer
RTE>>

it seems to be working fine. -l specifies the CPU cores to run on.

TRex

Because Intel i211 does not have enough queues, you need to use --software command line option with TRex.