Install and Configure DPDK on PC Engines apu4d4

I want to check Data Plane Development Kit (DPDK) and I was thinking on which computer/platform I should do that. I have a few PC Engines apu4d4 system boards which has 4 Intel i211AT based LAN ports. DPDK supports Intel i211AT, so this seems to be a reasonable platform. Now, how to install and configure DPDK.

I assume Ubuntu 20.04 is installed on apu4d4.

Download, Compile and Install DPDK

Actually Ubuntu repo has DPDK, but I want to compile it myself and use a particular version. I selected the Long Term Stable (LTS) version 20.11.2. So first step is to download it from here and uncompress and change into that directory.

Install some packages to be able to compile:

sudo apt install build-essential meson ninja-build pkg-config libpcap-dev libnuma-dev

Now I can build it:

meson -Dexamples=all -Duse_hpet=true build
cd build
ninja
sudo ninja install
sudo ldconfig

meson (configure) finishes in less than 2 minutes but ninja (compile) takes a long time, approx. 54 minutes. I compile all of the examples and also HPET (High Precision Event Timer) support.

This is not enough to run a test program, some system configuration is needed.

Configuration for DPDK

GRUB Configuration

Not all of these need to be done at GRUB, but I decided to do so. Basically I need to set 4 kernel parameters (update GRUB_CMDLINE_LINUX_DEFAULT):

• set vfio.enable_unsafe_noiommu_mode=1. This is needed because the embedded processor on apu4d4 does not have IOMMU support. If this is not done, trying to use vfio-pci results error -22 (which can be seen by dmesg).

• set hugepages=512. Naturally 512 is to be set according to particular needs but I use 512 for the moment. This parameter sets up 512x 2MB hugepages (normally Linux uses 4K pages). DPDK only uses hugepages, so this is a must.

• set hpet_mmap=1. DPDK uses HPET through mmap, so this has to be set if DPDK HPET support is required.

• set isolcpus=2,3. This is not a must. It basically removes processor core 2 and 3 (starting from 0, so the two cores out of four) from Linux scheduler, so these cores are not used by Linux. DPDK alone can use these cores.

Run sudo update-grub after these changes and reboot.

Bind NICs to DPDK

GRUB configuration will be applied automatically but Network Interfaces (NICs) should be binded manually to DPDK everytime after a reboot. There is a DPDK utility for this: dpdk-devbind.py. I am using the last two interfaces (enp3s0 and enp4s0).

sudo dpdk-devbind.py -b vfio-pci enp3s0
sudo dpdk-devbind.py -b vfio-pci enp4s0

These should finish without an error. If there is an error, you have to fix that first, check dmesg.

Checking the Hugepages configuration, status of NICs and number of Memory Channels and Rank

It is good to know these before running a test program.

Run dpdk-hugepages.py -s to display the status of hugepages configuration, e.g. it outputs:

$ dpdk-hugepages.py -s
Node Pages Size Total
0    512   2Mb    1Gb

Hugepages mounted on /dev/hugepages

Run dpdk-devbind.py -s to display the status of NICs, e.g. it outputs:

$ dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'I211 Gigabit Network Connection 1539' drv=vfio-pci unused=igb
0000:04:00.0 'I211 Gigabit Network Connection 1539' drv=vfio-pci unused=igb

Network devices using kernel driver
===================================
0000:01:00.0 'I211 Gigabit Network Connection 1539' if=enp1s0 drv=igb unused=vfio-pci *Active*
0000:02:00.0 'I211 Gigabit Network Connection 1539' if=enp2s0 drv=igb unused=vfio-pci

No 'Baseband' devices detected
==============================

Crypto devices using kernel driver
==================================
0000:00:08.0 'Kabini/Mullins PSP-Platform Security Processor 1537' drv=ccp unused=vfio-pci

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

As you see, enp1s0 and enp2s0 are using the kernel driver (they are binded to kernel), while others are binded to DPDK driver (vfio-pci).

Finally, it is good to provide the number of memory channels to be used by DPDK and if you do not know this information it can be checked by dmidecode, e.g. it outputs:

$ sudo dmidecode -t memory
## dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.

Handle 0x0009, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 4 GB
	Error Information Handle: Not Provided
	Number Of Devices: 1

Handle 0x000A, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0009
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 4096 MB
	Form Factor: SODIMM
	Set: None
	Locator: DIMM 0
	Bank Locator: CHANNEL A
	Type: DDR3
	Type Detail: Synchronous Unbuffered (Unregistered)
	Speed: 1333 MT/s
	Manufacturer: Not Specified
	Serial Number: 00000000
	Asset Tag: Not Specified
	Part Number: Not Specified
	Rank: 1
	Configured Memory Speed: 1333 MT/s
	Minimum Voltage: 1.5 V
	Maximum Voltage: 1.5 V
	Configured Voltage: Unknown

It shows apu4d4 has one memory channel.

Test

If you have finished the build and install of DPDK successfully, updated the GRUB settings and rebooted, and binded NICs to DPDK, now a test program can be run. Arguments used:

• -l: list of cores to run on
• -n: number of memory channels to use

$ sudo dpdk-test -l 2-3 -n 1

EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   Invalid NUMA socket, default to 0
EAL:   Invalid NUMA socket, default to 0
EAL:   Invalid NUMA socket, default to 0
EAL:   using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_e1000_igb (8086:1539) device: 0000:03:00.0 (socket 0)
EAL:   Invalid NUMA socket, default to 0
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_e1000_igb (8086:1539) device: 0000:04:00.0 (socket 0)
EAL: No legacy callbacks, legacy socket not created
EAL: HPET frequency is ~14318 kHz
RTE>>

it seems to be working fine.

Because many system resources are accessed by a DPDK application and this requires super user access privileges, sudo is needed for application invocations.