A new version of this post for DPDK 22.11 and Ubuntu 22.04 is here.
I want to check Data Plane Development Kit (DPDK) and I was thinking on which computer/platform I should do that. I have a few PC Engines apu4d4 system boards which has 4 Intel i211AT based LAN ports. DPDK supports Intel i211AT, so this seems to be a reasonable platform. Now, how to install and configure DPDK.
I assume Ubuntu 20.04 is installed on apu4d4.
Download, Compile and Install DPDK
Actually Ubuntu repo has DPDK, but I want to compile it myself and use a particular version. I selected the Long Term Stable (LTS) version 20.11.2. So first step is to download it from here and uncompress and change into that directory.
Install some packages to be able to compile:
sudo apt install build-essential meson ninja-build pkg-config libpcap-dev libnuma-dev
Now I can build it:
meson -Dexamples=all -Duse_hpet=true build cd build ninja sudo ninja install sudo ldconfig
meson (configure) finishes in less than 2 minutes but ninja (compile) takes a long time, approx. 54 minutes. I compile all of the examples and also HPET (High Precision Event Timer) support.
This is not enough to run a test program, some system configuration is needed.
Configuration for DPDK
Not all of these need to be done at GRUB, but I decided to do so. Basically I need to set 4 kernel parameters (update
vfio.enable_unsafe_noiommu_mode=1. This is needed because the embedded processor on apu4d4 does not have IOMMU support. If this is not done, trying to use vfio-pci results error -22 (which can be seen by dmesg).
hugepages=512. Naturally 512 is to be set according to particular needs but I use 512 for the moment. This parameter sets up 512x 2MB hugepages (normally Linux uses 4K pages). DPDK only uses hugepages, so this is a must.
hpet_mmap=1. DPDK uses HPET through mmap, so this has to be set if DPDK HPET support is required.
isolcpus=2,3. This is not a must. It basically removes processor core 2 and 3 (starting from 0, so the two cores out of four) from Linux scheduler, so these cores are not used by Linux. DPDK alone can use these cores.
sudo update-grub after these changes and reboot.
Bind NICs to DPDK
GRUB configuration will be applied automatically but Network Interfaces (NICs) should be binded manually to DPDK everytime after a reboot. There is a DPDK utility for this:
dpdk-devbind.py. I am using the last two interfaces (enp3s0 and enp4s0).
sudo dpdk-devbind.py -b vfio-pci enp3s0 sudo dpdk-devbind.py -b vfio-pci enp4s0
These should finish without an error. If there is an error, you have to fix that first, check dmesg.
Checking the Hugepages configuration, status of NICs and number of Memory Channels and Rank
It is good to know these before running a test program.
dpdk-hugepages.py -s to display the status of hugepages configuration, e.g. it outputs:
$ dpdk-hugepages.py -s Node Pages Size Total 0 512 2Mb 1Gb Hugepages mounted on /dev/hugepages
dpdk-devbind.py -s to display the status of NICs, e.g. it outputs:
$ dpdk-devbind.py -s Network devices using DPDK-compatible driver ============================================ 0000:03:00.0 'I211 Gigabit Network Connection 1539' drv=vfio-pci unused=igb 0000:04:00.0 'I211 Gigabit Network Connection 1539' drv=vfio-pci unused=igb Network devices using kernel driver =================================== 0000:01:00.0 'I211 Gigabit Network Connection 1539' if=enp1s0 drv=igb unused=vfio-pci *Active* 0000:02:00.0 'I211 Gigabit Network Connection 1539' if=enp2s0 drv=igb unused=vfio-pci No 'Baseband' devices detected ============================== Crypto devices using kernel driver ================================== 0000:00:08.0 'Kabini/Mullins PSP-Platform Security Processor 1537' drv=ccp unused=vfio-pci No 'Eventdev' devices detected ============================== No 'Mempool' devices detected ============================= No 'Compress' devices detected ============================== No 'Misc (rawdev)' devices detected =================================== No 'Regex' devices detected ===========================
As you see, enp1s0 and enp2s0 are using the kernel driver (they are binded to kernel), while others are binded to DPDK driver (vfio-pci).
Finally, it is good to provide the number of memory channels to be used by DPDK and if you do not know this information it can be checked by
dmidecode, e.g. it outputs:
$ sudo dmidecode -t memory # dmidecode 3.2 Getting SMBIOS data from sysfs. SMBIOS 3.0 present. Handle 0x0009, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 4 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x000A, DMI type 17, 40 bytes Memory Device Array Handle: 0x0009 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 64 bits Size: 4096 MB Form Factor: SODIMM Set: None Locator: DIMM 0 Bank Locator: CHANNEL A Type: DDR3 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 1333 MT/s Manufacturer: Not Specified Serial Number: 00000000 Asset Tag: Not Specified Part Number: Not Specified Rank: 1 Configured Memory Speed: 1333 MT/s Minimum Voltage: 1.5 V Maximum Voltage: 1.5 V Configured Voltage: Unknown
It shows apu4d4 has one memory channel.
If you have finished the build and install of DPDK successfully, updated the GRUB settings and rebooted, and binded NICs to DPDK, now a test program can be run. Arguments used:
-l: list of cores to run on
-n: number of memory channels to use
$ sudo dpdk-test -l 2-3 -n 1 EAL: Detected 4 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: Invalid NUMA socket, default to 0 EAL: Invalid NUMA socket, default to 0 EAL: using IOMMU type 8 (No-IOMMU) EAL: Ignore mapping IO port bar(2) EAL: Probe PCI driver: net_e1000_igb (8086:1539) device: 0000:03:00.0 (socket 0) EAL: Invalid NUMA socket, default to 0 EAL: Ignore mapping IO port bar(2) EAL: Probe PCI driver: net_e1000_igb (8086:1539) device: 0000:04:00.0 (socket 0) EAL: No legacy callbacks, legacy socket not created EAL: HPET frequency is ~14318 kHz RTE>>
it seems to be working fine.
Because many system resources are accessed by a DPDK application and this requires super user access privileges, sudo is needed for application invocations.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.